Tech Tips Blog

Combating the Dark Side

Anyone running a website that involves blogging, forums or filling out email forms knows that spammers are a major menace (and not a phantom menace). If you allow your users to make posts or comments, chances are you get more spam than legitimate content. It often feels like you don't have full control, as if you're Han Solo... but you're sealed in carbonite. Until now we've done okay in this battle with the Dark Side spammers, but we wanted to do better, and we did indeed find something much better. First, I need to give you a little back story.

A long time ago, in a galaxy far, far away...

The existing rebel base

So how did we defend against spammers before now? Fighting spammers is an ongoing battle against a moving target so the best thing to do is to fight them using tools that are evolving to match them. The first is the most obvious, namely The Force CAPTCHA. Specifically, we use reCAPTCHA, a constantly evolving service so good that Google purchased them a few years ago. You can see an example in the image below... or at the bottom of this page. These are the "annoying" challenges we all see on most web forms these days, but they are not nearly as annoying as seeing pages full of distasteful spam comments.

Not using CAPTCHA would mean that staff here would have to delete spam manually. Until recently, we were averaging about 70 spam comments blocked by CAPTCHA per day, and most of the attempts are not made by real people, but rather are made by stormtroopers bots, automated software that attempts to post spam all over the web. Just think of the amount of work involved to delete those comments if you decide to pit your staff hours against automated spam bots. Some days we peaked at hundreds of attempts blocked, so think of some poor padawan worker having to review several hundred spam messages, just in case one of them is not spam, or maybe is even another help request from Princess Leia a legit website visitor.

Worse, email notifications go out to the author of the post and to every commenter that asked to be notified of other comments. In other words, if we did not use CAPTCHA we would be helping the bots to spam our staff and our site visitors... which is totally uncool! It would be like we had turned to the Dark Side.

Since light sabers have not yet been invented, the only practical option is to use CAPTCHA.

Further, did you notice that the CAPTCHA in the example image includes a photograph? This is a more recent upgrade made to reCAPTCHA that is confusing bots everywhere. What did I have to do to get this new feature? Nothing! The entire captcha check was retrieved from reCAPTCHA, so when they updated their software it just appeared on our page... like magic. We get the benefit of all of their ongoing work fighting smarter and smarter spam bots, without lifting a finger. It's like having Master Yoda on your side.

Try not. Do or do not. There is no try.

So, is fighting spam a losing battle? Fact is, much like getting choked by Lord Vader from the other side of the room, an annoying amount of spam still got past CAPTCHA, despite the fact that we take several other measures to block spam from the website. We highlighted much of it in the SPAM section of our detailed site recipe. There are also things done elsewhere on the site and on the server side by our resident Jedi Master systems administrator to block spam bots from the site before they can even try to post spam. Still, we found ourselves regularly dealing with spam. We needed something new.

Named must your fear be before banish it you can.

Clearly we needed another strategy. A new trend is sweeping the web, closing comments on older posts. So much of the spam getting posted to our site was on blog entries that were several years old and concerned long dead issues. By closing comments you're shoring up your defenses and giving the bots fewer chinks in your armor to find, leaving them impotent, like headless stormtroopers. After a couple of bad bot break-ins, we decided to go this route. The results were immediately evident. We went from averaging 70 CAPTCHA-blocked attempts a day to just 9 per day. Further, before about 65% of these were attempts to post comments, but now very few spam bot attempts are on comments.

Help me, Obi-Wan Kenobi; you're my only hope!

So how did we do it? Closing comments manually is both tedious and prone to human error. We needed an automated solution. Since this is a Drupal site, we added the Comment Closer (commentcloser) module. This fairly straightforward tool lets us automatically close all commenting on posts of a certain age (we picked a 3 month period). Getting slightly more specific, automation is handled during regular cron jobs and the module will let you set different timelines for different content types (blogs, news, forum entries, etc). For example, if we wanted to we could also have set forums to 9 months, blog posts to just 3 months and left comments unaltered for stories.

We also turned on an option that initially tells visitors when the comments will be closing (per the example above), and, then will inform users that comments are closed, per the example below.

Use the Force, Luke!

Should You Do This? Strategically you may have reasons to keep comments open indefinitely, but if you don't then I'd strongly advise using this strategy. It has really helped us. Most CMSs out there already have a module, plugin or add-on for this purpose. Use it, or at worst, do it manually. It will save you a lot of time and stress.

Either that or get a job working for us rebels on anti-spam measures. Fighting the dark side's spam seems like a growth industry likely to provide many employment opportunities.

May the Force be with you!

Commenting on this Blog entry is closed.