DNS Outage

Early in the morning there was a DNS error which caused many clients to be unable to resolve our addresses. I was asleep at the time but the Akismet team fixed the error shortly after it was reported, but unfortunately due to the nature of DNS some clients appear to still be caching the “not found” errors, even 6-7 hours after it’s been fixed.

This sucks, and any time there is a problem like this we have a thousand ideas about how to prevent anything like this from happening again. Over the next day or two we’ll be deciding Because of the nature of Akismet it hurts so much when there are any system problems, which is why we’ve been so fanatical about performance and reliabilit through several doublings of the volume of spam.

Now might be a good time to point out that we have a new version of the Akismet plugin for WordPress that adds a “Recheck queue” button to the moderation page. That way if something happens and you have a lot of comments stuck in moderation, with a single click Akismet can recheck them and clear out any known spam.

18 responses on “DNS Outage

  1. i got about 12 spams in that brief moment.

    Absence makes the heart grow fonder for akismet :-)

    No problem about the outage – just keep doing the good job you’ve been doing.

  2. Thanks for the great plugin, it’s kept me spam-free! I woke up this morning and saw that my moderation queue was abuzz and am glad to know why.

  3. The biggest problem is that some of the clients fail horribly when the server goes down. The Python client I am using tends to start raise uncaught exceptions — in their library or in modules they have imported. Letting spam through, apparently what the WP client does, is the correct thing to do the. We could manage damage but auditing code or encouraging those who implement the API to fail gracefully in this way.

  4. Do you guys have any clue into when, why and from whom spikes in blog spam come? I just notice these weird cycles (for instance today) where my site gets hammered by comment spam from one or two IPs, whereas at other times the spam is rather quiescent. I can glean some basic info from the Spam Zeitgeist, but I’d like to know more.

Comments are closed.