Fri 22 Sep 2006
In Part I and Part II I have described to you how Spam gets to you and why Spammers send it. Legislation and regulation have failed to stop it. In Part III we discuss using technology to try and stop it.
So we have to turn to technology to stop the Spammers. Using technology (AntiSpam software etc) to stop what is essentially a human problem (people on the internet with the ethics of a microbe) is a bad approach but unfortunately the only one we have.
AntiSpam software mostly uses inspection techniques to stop Spam. The AntiSpam software inspects incoming email and compares the content with a known profile of Spam - it asks “Does this look like Spam?” - and then makes a weighted decision to allow or reject the email. At its best this technique stops most of the Spam with a few getting through.
The inspection technique is a balancing act. Set the rules too loose and you get lots of Spam email in your Inbox. Set the rules too tight and you stop all the Spam but also you start stopping legitimate email. Incorrectly stopped legitimate email is called a false positive. If you are getting false positives you will need to keep reading the Spam email so that you can find these legitimate email messages and this defeats the purpose of AntiSpam software.
A big problem with content inspection is that filters that what works today will have a shelf life. If there is a war on Spam then the enemy are the Spammers and they see everything we do to stop them. After AntiSpam software started being widely used Spammers started crafting their Spam messages to get around the filters. This is why you will see V1agra and other misspellings. There are a number of sophisticated techniques to make an email message look completely different to a human than how it is read by the computer’s Antispam software.
This leads to the insane situation where Spammers craft their email messages so that they will get through email filters to people they know don’t want their messages by virtue of having those filters.
Bayesian filters were advancement on the first generation inspection techniques. Previously the content inspection was only a weighted measurement of how a particular email message measured up against a profile of how Spam generally looks. Bayesian filters inspected your own Inbox to build a profile of Ham (legitimate email that you want to get). The decision to accept and reject was now balanced between how much an email looked like Spam (reject) and how much the message looked like Ham (accept).
Bayesian filters were initially very successful but once again our Spammer friends responded. This is why at the bottom of the Viagra offer you will see half a page from a Dickens novel. The Spammers are trying to bring the Ham weighting up for the message so it could get through. This has lead to some very odd looking emails. More than once I have looked at Spam that has got through this way and had no idea what was on offer. It clearly does not confuse, or put off the Schmucks.
At LANcom Technology, we have experienced the negative affects of this kind of Spammer fight-back in the last twelve months. Twelve months ago our Spam Server was very successful at stopping Spam by using standard content inspection techniques combined with Bayesian logic. Spammers have since adapted their email and we had to find a better solution as that one no longer worked (we have! – more at the end)
There have been other techniques tried and discarded. Whitelisting (only allowing e-mail from a defined list of servers) and blacklisting (blocking e-mail using a dynamic list of Spam sending email servers) have both failed because the administrative overhead is too great.
There have been some more technical orientated solutions offered. Micro charging and Sender Policy Framework were two that were championed.
With micro charging everybody pays a ‘central organisation’ a tiny fraction of a cent per mail email for the right to send email. Even big companies would only pay $10 - $20 per month but those sending 10 Million email messages a month (i.e. spammers) would be charged many thousands of dollars making their operation uneconomical.
Sender Policy Framework elegantly leveraged the existing Domain Name System (DNS) to ensure you could cross reference the identity of who was sending you mail. Both ideas have stalled for political reasons beyond the scope of this piece.
Our current solution from IronPort systems was deployed two months a go and has been very successful. It uses a technology Ironport call a reputation filter and it is essentially a highbred blacklist. Spam has a very distinct traffic profile that is very identifiable if you can see the traffic going to more than a few of the target servers. I blogged about how ISPs could identify spammers easily and why they don’t earlier this month.
Ironport is one of the major antispam software vendors and therefore it has thousands of anti spam servers live on the Internet. Each server uses standard content inspection techniques that are tuned to avoid false positives but will also consequently allow some Spam through.
The extra step that Ironport does is that their servers report suspicious email to a central location where it is stored and compared with other reports. Because a Spammer is sending millions of emails in each session they can be quickly identified and a bulletin sent out in real time to all Ironport servers that suspicious email from that server is in fact Spam (hence the term reputation filter).
The algorithms to make that call are sophisticated and get more accurate as more people join the network. Email can be compared without content being shared so there are no privacy issues and because one organisation controls it all the huge administrative hurdles of other similar solutions are avoided.
So in a 10 million email run Spammers might get the first hundred thousand away but then the reputation filter will close them down. 100000 emails is not enough to reach our one-in-ten-million schmuck and the Spammer will fail.
The best thing about the Ironport solution is that because the solution is not dependant on content inspection Spammers are having a very hard time getting around it. Spammers have to send to many, many email servers to make money and this easily identifies them to the reputation filter. Happy days indeed.