Those who like to live on the bleeding edge might like to download and test the latest revision of the Akismet WordPress plugin from Subversion:

http://plugins.svn.wordpress.org/akismet/trunk

If you don’t know what Subversion is or how to use it, I’d suggest waiting for the next official Akismet release, which won’t be far away.

The new revision includes a new diagnostic feature on the Akismet Configuration tab that’s intended to address a problem with some web hosts.

We’ve known for a while that some web hosts and servers have firewall rules that block outgoing connections — including connections to the akismet.com API servers, which are necessary for the Akismet plugin to work. Usually the host administrators will add some firewall rules to permit the Akismet plugin to connect to akismet.com. But recently we’ve discovered that some hosts have created incomplete firewall rules, with the result that some Akismet connections succeed, but some fail. This caused Akismet to seem like it was working, when in fact only some spam was checked, and only some reports ever made it back to Akismet.com.

The problem is caused by the host’s firewall rules – it’s not something that Akismet can fix. We can detect the problem, however, which is exactly what the new revision does.

The new feature adds a Server Connectivity section to the Akismet Configuration tab. The new section will check for any problems connecting to any Akismet servers, including the partial firewall problem, and recommend a course of action if there is an issue.

There’s more code in the new revision than we would typically add in an Akismet update, so testing and feedback are welcome.

Some technical details for those who are interested:

Akismet uses round-robin DNS and load balancers to distribute the work of checking comments for spam across multiple servers. A DNS lookup of rest.akismet.com (the domain used for Akismet API calls) returns not one but several IP addresses, corresponding to the multiple servers. When your blog uses Akismet to check a comment, it will (more or less) randomly choose an Akismet server IP to use. Each time it checks a new comment, it might use a different Akismet server IP.

If a web host blocks all connections to all Akismet server IPs, the Akismet plugin can easily detect the problem and report an error. The plugin has always included a connectivity check as part of its configuration process, that will warn the user if Akismet servers are unreachable.

If a web host allows connections to some Akismet server IPs, but blocks connections to the others, the connectivity check might succeed (if it happens to connect to one of the allowed IPs), while some subsequent connections will fail (when they try to connect to one of the blocked IPs). Result: the plugin appears to be working fine, but in fact only some API calls are working. Users often won’t notice anything wrong, because some spam is caught.

The new revision expands the connectivity check to include all known Akismet servers. It displays a status message or warning to the user if some servers are unreachable. It also stores a list of server IPs and their status, with a 24 hour expiry. That list of IPs is used when checking and reporting spam, to ensure that only servers that are known to be reachable are used. The list will be refreshed after 24 hours so as to adapt to any changes in availability – such as when a web host changes their firewall and blocks or unblocks an Akismet server. (Which is something that a few hosts have been known to do without warning, unfortunately).

The FTC has launched legal action against a Californian web hosting service it says is responsible for botnets, malware, credit card theft and of course spam. The provider has been disconnected and its operators now face a lawsuit.

The FTC alleges that Pricewert/3FN operates as a “‘rogue’ or ‘black hat’ Internet service provider that recruits, knowingly hosts, and actively participates in the distribution of illegal, malicious, and harmful content,” including botnet control servers, child pornography and rogue antivirus products. 3FN also operates by the names APS Telecom and APX Telecom.

The provider is known as a frequent host of “scraper” or autoblog sites — fake blogs that re-publish unauthorized copies of content taken from other blogs, often sending spam pingbacks and trackbacks in the process.

Our sources indicate the network also provided services to several of the major forum and comment spammers. In particular, web sites owned by the developers of several spambot programs have been shut down (though we expect they will resurface elsewhere before long).

Of course if you’re using Akismet you won’t notice much of a difference: Akismet has long been highly effective at catching spam produced by their spambots and autoblogs.

Our stats suggest a significant and immediate drop in overall spam levels coinciding with the FTC’s action – on the order of about a 20% reduction (in spam that was or would be successfully caught by Akismet).

Akismet version 2.2.4 is now available. WordPress users can upgrade using the automatic plugin update feature.

Changed in this version:

• Fixed a key problem affecting the stats feature in WordPress MU
• Provide additional blog information in Akismet API calls

The extra blog information passed to the Akismet API will help Akismet to better adapt and provide results that are tailored to your specific blog.

Support: Please use the Akismet support form to ask questions or report possible problems. Support questions posted in comments will be removed.

Update: the problem has now been fixed.

A bug in development versions of WordPress is causing some comments to be incorrectly caught as spam. The problem is in WordPress, not Akismet, and there’s no way for Akismet to prevent it from happening.

The problem only affects WordPress blogs running current development code. It does not affect other applications that use Akismet.

Technically-minded users can read the specifics in the WordPress Trac system.

We expect to have a fix in WordPress core soon, and deployed to WordPress.com shortly after.

This is probably a good opportunity to remind readers that Akismet is not necessarily the reason a comment gets caught as spam. The majority of complaints we receive about real comments being caught as spam were in fact not caused by Akismet at all – but by other spam filtering plugins or features. (Unfortunately WordPress doesn’t show the reason a comment was moved to the Spam filter, so there’s no easy way to tell which ones were put there by Akismet and which ones by something else).

In particular, the WordPress Comment Blacklist feature (Settings / Discussion) trips up some users. It lets administrators provide a list of words, IP or email addresses that should be blocked as spam. Any comments that match words in the blacklist will automatically be moved to the Spam filter – regardless of whether or not Akismet considers it to be spam. The blacklist matches within words, meaning that if you add a short string such as “ru”, it will block any comment containing the words “truth” or “fruit”, or any other word with the letters “ru”. And, since the blacklist takes precedence over Akismet, reporting those comments to Akismet as false positives won’t stop them from being caught.

If you think the Comment Blacklist feature might be catching legitimate comments as spam, the tw-blacklight plugin might help your diagnosis.

We’ll update the Akismet blog when the WordPress bug has been fixed.

One of the most common forms of comment and pingback spam right now is the relatively subtle, ambiguous kind — short phrases or questions that are not obviously spam, at least on face value. Since we last posted about this, the more sophisticated spammers have progressed from old standbys like “nice post” and “great blog”, to more cunning things like questions (”where can I download your theme?”) and appeals to your helpful nature (”I’m having trouble subscribing to your RSS feed”).

Akismet almost always catches these kinds of bogus comments.

The tip-off of course is that they often include a link to a site that’s advertising dubious or sleazy merchandise (or worse, a web site that harms the viewer’s computer). But it’s easy to forget to look at the link before approving a comment, or give the comment author the benefit of the doubt without checking closely. And spammers have recently learned to post several comments over time, the first of which contains no link or obvious clue. (We call these precursor spams).

Anyway, a comment is a comment, right, so what’s the harm in approving a few tame platitudes, even if they were posted by spammers?

Unfortunately it is harmful, and most of the damage is to your own site.

By moving these comments out of your spam folder and publishing them on your blog, you’re doing three things, all of them bad:

1. You are undermining your site’s SEO.

The spammer’s web site might seem inoffensive on face value. But the black-hat SEO and spam methods used by its promoter are not. That same spammer is busy building backlinks from anywhere they can find them, including some of the web’s worst neighbourhoods. By regularly publishing links to spammers’ web sites, you’re giving Google and other search engines a hint that links from your blog are poor quality.

Now it’s true that Google will try not to penalize a web site for inadvertently linking to a bad neighbourhood. But even if they don’t, you are weakening the value of each of the other links from your blog – “diluting your GoogleJuice”, if you like – and helping to validate the spammer’s web site. In some cases you might even find that you are helping the spammer overtake your blog in search engine results.

2. You are attracting more spammers.

Less skilled spammers will deliberately seek out blogs that other spammers have successfully spammed, because they know they are easy targets. Organized spammers circulate lists of such blogs (for a small fee of course). And professionals keep their own lists of previous victims, because they know future spam is even more likely to be approved there. By letting some spam through – even seemingly harmless ones – you are providing a signal to spammers that your blog is a profitable target. (Experienced bloggers will be familiar with this phenomenon: you accidentally approve one seemingly unremarkable spam comment, and a big batch of ugly spam follows soon after).

WordPress and many other blog applications have a feature, independent of Akismet, where regular users who have had at least one comment approved, will automatically skip the moderation queue next time and have their comments published right away. Spammers know this, and they’ll come back to take advantage of it. Often they’ll link to a harmless looking site in their first comment (or include no link at all), but link to progressively more blatant spam in subsequent comments.

3. You are damaging your reputation.

You might not click on the links in all the comments on your blog, but some of your readers will. And some of those links will go to sites that are sleazy, offensive, or harmful.

Worse still, a spam tactic that is becoming more popular is to first post a small number of spam comments on innocent blogs; then send a large volume of spam to other web sites linking to the blog post that contains those comments. (They do this to try to get around spam filters and blacklist that recognize and catch links to their own site).

If you do publish spam comments on your blog, you might discover later that thousands of other blogs and forums have been spammed with links to your blog.

So what should you do about it?

Akismet will almost always catch these comments and put them in your Spam folder. Usually you don’t need to do anything; just don’t approve them for publication.

We have a real-time view of spam activity on millions of blogs around the world, so we can detect patterns in behaviour that can’t be seen by looking at any one single comment. If a bland, generic comment turns up in your spam folder, you should be suspicious of it – Akismet flagged it for a reason. Think twice before approving it for publication. Unless you know the author, it almost certainly is spam — or a subtle precursor to it.

Also, keep an eye out for forthcoming Akismet updates. In addition to our usual work behind the scenes monitoring and adapting to new spam techniques, we’re developing some new features designed specifically to help protect against the potential harm done by spammers.

Back in the early days of blogging and when comment spamming was still fairly small scale (compared to today) one method people used to stop a spambot was to use a CAPTCHA. The idea was that a comment spambot could not read the image and so the comment would fail and you would not get any spam. Obviously the spammers really did not like these CAPTCHA’s so they devoted resources to get around them.

In January 2004 — 5 years ago — Cory Doctorow blogged about pornography being used to get humans to solve captchas for spammers and there were scripts which could defeat different CAPTCHA’s. It is because of those scripts that you can now be faced with images that you struggle to get right. All you want to is leave a comment right?

Companies still believe in the power of the CAPTCHA and they are now very wrong to do so. Where there is a demand  by those wanting to spam there is supply – and it’s less than a cent to spam your blog. At ZDNet’s Security blog they report on an industry which can solve a quarter of a million CAPTCHA’s a day.

You write a post and you would like comments. Using a CAPTCHA to stop a spammer is not going to work. If someone is writing a reply to your post why make them solve some image with distorted letters? They want to think about your post not whether that is an 8 or a B, a 1 or I or l. If people have previously been annoyed by these things they may not even bother trying to leave a comment. You lose here — your blog lost a comment.

Putting visible obstacles to commenting irritates readers and gives spammers something to overcome and the more information spammers have the more likely they are to spam your blog. This is why Akismet works as it does — keep the spammers guessing but let people comment freely. Commenting should be as easy as blogging because that is how to keep your conversations going.

We’ve improved the accuracy of the Akismet stats feature that was introduced in Akismet 2.2.1. You might notice a slight change in the total spam, comments and accuracy figures reported at the top of your stats page, and the percentages shown in the pie chart. The Historical Stats figures and time graphs are unaffected – these were correct all along.

There’s no plugin upgrade required.

Our overall spam statistics remain unchanged, and Akismet is as accurate as ever.

Just a bit after a year after we welcomed Defensio to the market it’s just been announced that that they’re being swallowed by the public security company Websense.

Given the size of Defensio’s team and user base it probably wasn’t a huge acquisition, but this is still an important validation of the  space in general as traditional email anti-spam and security companies have been completely blind to web spam thus far, and in my opinion it’s the fastest growing threat to most businesses today.

(Think about it: if you get a spam in your email, it’s annoying but not the end of the world; if you get a spam on your website it’s immediately visible to your customers who probably won’t be impressed by “buy viagra’s” contribution to the conversation when thinking about your products or services.)

2009 is shaping up to be a pretty interesting year in our world. We’ll definitely be keeping an eye on the ever-increasing players in this market, but more importantly we’ll be listening to you guys and do our best to be as invisible and effective as possible. (Especially as we approach 10 billion spams blocked!)

Michael Manfre has created a plugin for Textpattern . You can download the plugin from his website and have protection from the spam. Thanks Michael!

The latest release of Gallery (2.3) now has Akismet support built in. Thanks to Chris for letting us know.

Kevin at BIGACE CMS has developed a commenting extension which uses Akismet. 

And lastly but by no means least the latest version of vBulletin blog also has Akismet support by default.