LANcom Technology Blog

I have been struck down by the flu in the last week and have spent a number of days at home getting better (and not infecting my workmates).

I have remote access to the office from home. Thanks to this I have been able send and receive email, reschedule appointments, clear my voicemail and write a couple of essential documents - all from home.

LANcom will be producing a report next month on what you need to be able to do this yourself. It will explain the ins and outs of getting home access and on-the-road access to your work computers.

If you would like a copy just send an email to info@lancom.co.nz asking for one.

In Part I and Part II I have described to you how Spam gets to you and why Spammers send it. Legislation and regulation have failed to stop it. In Part III we discuss using technology to try and stop it.

So we have to turn to technology to stop the Spammers. Using technology (AntiSpam software etc) to stop what is essentially a human problem (people on the internet with the ethics of a microbe) is a bad approach but unfortunately the only one we have.

AntiSpam software mostly uses inspection techniques to stop Spam. The AntiSpam software inspects incoming email and compares the content with a known profile of Spam -  it asks “Does this look like Spam?” - and then makes a weighted decision to allow or reject the email. At its best this technique stops most of the Spam with a few getting through.

The inspection technique is a balancing act. Set the rules too loose and you get lots of Spam email in your Inbox. Set the rules too tight and you stop all the Spam but also you start stopping legitimate email. Incorrectly stopped legitimate email is called a false positive. If you are getting false positives you will need to keep reading the Spam email so that you can find these legitimate email messages and this defeats the purpose of AntiSpam software.

A big problem with content inspection is that filters that what works today will have a shelf life. If there is a war on Spam then the enemy are the Spammers and they see everything we do to stop them. After AntiSpam software started being widely used Spammers started crafting their Spam messages to get around the filters. This is why you will see V1agra and other misspellings. There are a number of sophisticated techniques to make an email message look completely different to a human than how it is read by the computer’s Antispam software.

This leads to the insane situation where Spammers craft their email messages so that they will get through email filters to people they know don’t want their messages by virtue of having those filters.

Bayesian filters were advancement on the first generation inspection techniques. Previously the content inspection was only a weighted measurement of how a particular email message measured up against a profile of how Spam generally looks. Bayesian filters inspected your own Inbox to build a profile of Ham (legitimate email that you want to get). The decision to accept and reject was now balanced between how much an email looked like Spam (reject) and how much the message looked like Ham (accept).

Bayesian filters were initially very successful but once again our Spammer friends responded. This is why at the bottom of the Viagra offer you will see half a page from a Dickens novel. The Spammers are trying to bring the Ham weighting up for the message so it could get through. This has lead to some very odd looking emails. More than once I have looked at Spam that has got through this way and had no idea what was on offer. It clearly does not confuse, or put off the Schmucks.

At LANcom Technology, we have experienced the negative affects of this kind of Spammer fight-back in the last twelve months. Twelve months ago our Spam Server was very successful at stopping Spam by using standard content inspection techniques combined with Bayesian logic. Spammers have since adapted their email and we had to find a better solution as that one no longer worked (we have! – more at the end)

There have been other techniques tried and discarded. Whitelisting (only allowing e-mail from a defined list of servers) and blacklisting (blocking e-mail using a dynamic list of Spam sending email servers) have both failed because the administrative overhead is too great.

There have been some more technical orientated solutions offered. Micro charging and Sender Policy Framework were two that were championed.

With micro charging everybody pays a ‘central organisation’ a tiny fraction of a cent per mail email for the right to send email. Even big companies would only pay $10 - $20 per month but those sending 10 Million email messages a month (i.e. spammers) would be charged many thousands of dollars making their operation uneconomical.

Sender Policy Framework elegantly leveraged the existing Domain Name System (DNS) to ensure you could cross reference the identity of who was sending you mail. Both ideas have stalled for political reasons beyond the scope of this piece.

Our current solution from IronPort systems was deployed two months a go and has been very successful. It uses a technology Ironport call a reputation filter and it is essentially a highbred blacklist. Spam has a very distinct traffic profile that is very identifiable if you can see the traffic going to more than a few of the target servers. I blogged about how ISPs could identify spammers easily and why they don’t earlier this month.

Ironport is one of the major antispam software vendors and therefore it has thousands of anti spam servers live on the Internet. Each server uses standard content inspection techniques that are tuned to avoid false positives but will also consequently allow some Spam through.

The extra step that Ironport does is that their servers report suspicious email to a central location where it is stored and compared with other reports. Because a Spammer is sending millions of emails in each session they can be quickly identified and a bulletin sent out in real time to all Ironport servers that suspicious email from that server is in fact Spam (hence the term reputation filter).

The algorithms to make that call are sophisticated and get more accurate as more people join the network. Email can be compared without content being shared so there are no privacy issues and because one organisation controls it all the huge administrative hurdles of other similar solutions are avoided.

So in a 10 million email run Spammers might get the first hundred thousand away but then the reputation filter will close them down. 100000 emails is not enough to reach our one-in-ten-million schmuck and the Spammer will fail.

The best thing about the Ironport solution is that because the solution is not dependant on content inspection Spammers are having a very hard time getting around it. Spammers have to send to many, many email servers to make money and this easily identifies them to the reputation filter. Happy days indeed.

However the spammers get your mailbox address, once one spammer has it, they will send you Spam.

Spammers spam to make money. Spammers make money by using two methodologies.

1. Selling crappy, fraudulent and sometimes non existent services or products. (e.g. Viagra, pyramid schemes, fake Rolexes, pirated software and movies etc)
2. ‘Pump and dump schemes’ where the Spammer releases anonymous stock tips for penny stocks that the spammer owns  in the hope that a schmuck acts on the tip that ‘serendipity’ sent him and surges the price.

The theory of Spam is to use Internet email to contact 10 million people at a time so that you can do business with the one-in-10-million schmuck who will actually buy your crappy product.

Without the schmuck there would be no Spam (I have already blogged on this) but out of 10 Million people you are always likely to get one or two.

That Spammers can get this kind of reach shows the absolute genius of the Internet’s designers. The designers of the Internet brilliantly built a system that could connect millions of computers but they spent little time planning for malicious use. If they had it would be unlikely that the Internet would be anywhere as powerful as it is today.

Most of the core Internet protocols that computers use to talk to each other (e.g. HTTP used for web browsing and SMTP used for email) are anonymous by default. Your email server will assume that any email server that connects to it over the Internet is giving its name accurately and that the email is legitimate. The sending email server doesn’t log onto your server in any way and doesn’t have to provide credentials to you. Spammers abuse the trust that is inherent in the Internet to send you their rubbish.

The core Internet protocols are not about to change in a hurry so we need other ways to stop Spam.

There have been a number of legislative attempts but these have been spectacularly unsuccessful. Geography is a problem. Where do you prosecute a US based spammer sending email from a Russian server selling products shipped from China to mailboxes in New Zealand?

Spammers also use Trojan techniques to highjack consumer PCs and get them to send their Spam. If Mr and Mrs Jones have out of date antivirus software their computer could be sending millions of Spam messages on behalf of the Spammer without their knowledge. These people would be caught up in any comprehensive Spam legislation.

Lazy marketers are also a problem for legislators. The email that Paul Richardson from TotalSalesPartners (nobody I know) sent to me yesterday had an opt-out option which was kind of funny as I had never opted-in. It was sent to info@lancom.co.nz and while it was not as vile as some of the other Spam we get it was Spam. These people should know better. These people would be caught up in any comprehensive Spam legislation.

Apple has moved it’s flagship Mac computer to run on the same Intel chip that PCs do. One of the outcomes of this is that the Apple iMac can now run Windows XP from bootup if you choose.

We shipped about 20 Apple iMacs last week. They were all running Windows XP. The client was a school who wanted their students to get a great experience using the school’s computer. They felt that the experience would feel better for the students if they sat down to a Mac even though the actual learning applications ran on Windows and could run on any Windows PC.

It is a nice story because it is what Apple has been teaching the personal computer industry for over 20 years - that the user experience is everything. The most important outcome LANcom should target from it’s work is that every time one of our customers gets up from working on their computer they should have had a satisfying experience.

Everything else is contributory. Apple got this years ago and it has been the cornerstone of this mavericks survival in the difficult years and prosperity in the buoyant years.

Spam is generally defined as e-mail received that is unsolicited and undesired. This definition is kind of a shame because while the unsolicited part of the definition is fairly easy to establish factually most spammers take the convenient position that because they thought you desired their email it is not Spam. Spammers – the people who send Spam – are a people with no integrity.  You probably already know that.

It is a serious problem. Not only do Spammers waste the time of millions of people to make each dollar they are compromising the Internet itself. By most estimates Spam is more than 60% of all email sent on the Internet.  This creates a huge clog that impinges on legitimate email (to any spammer reading legitimate email means email that it is solicited and desired).

Spammers get your email address in three ways. One way is to take your publicly listed domain name (e.g. lancom.co.nz for LANcom Technology) and guess mailbox names like info (info@lancom.co.nz) or sales (sales@lancom.co.nz) or accounts (accounts@lancom.co.nz).

The other way is to harvest your mail address from the Internet. Spammers have programs that search web pages looking for email addresses. If your web site has staff profiles and if those staff profiles have email addresses embedded in them those addresses will be harvested by the Spammers and your people will be sent Spam.

The third way is by buying addresses. This can by purchasing lists of email addresses from unscrupulous companies. For example, an insurance company who has decided that they can make money by selling your information. Legislation has been effective in limiting this practice. Some spammers illegally bribe employees of companies with these lists to get a copy.

Š

We’re in the middle of revolutionising our corporate manuals and knowledge base. Like all revolutions the change is fundamental but also subtle as it is primarily a reworking of what we already do. We have the same information but we are organising it in a much better way.

Like many of our customers LANcom Technology has built up a library of company manuals, procedures, how-tos and other documents in a swarm of Word documents. Word is a good tool to create documents but the classic silo for these documents is poor. If you are like LANcom Technology (and any typical Kiwi company) these Word documents are buried in a labyrinth of directories.

This labyrinth makes these documents hard to find for anyone except the person who created them. After a few months even the person who created the document can be struggling. Keeping the information current is difficult, finding the right information when needed (among 200 possible Word documents) is almost impossible. It is such a waste of effort and opportunity.

We are putting all our company information page by page into a Wiki. A Wiki is basically a web site where every page has an edit button. So that whenever you see a page from the company manual that needs a small change you click edit and make the change and save. All versions of the page are automatically archived with information of who did it and when. You can easily see who made what changes and when and you can roll it back when necessary.

Each page can be attached to many different categories. It’s like being able to place the Word document in many different folders – e.g. A knowledgebase document about the problem with one customer’s email server can be stored in the customer’s ‘folder’ but also in the ‘Email problems’ folder. Every word on every page is indexed automatically. It is like having Google for your entire company documentation. Just search on ‘email’ and you will get a list of pages with ‘email’ in the text that includes this knowledgebase article. Search for ‘email’ and the customer‘s name and you will likely get the exact correct page. Each search takes less than a second.

The web pages can be edited like a Word document with tables and fonts etc so they are as easy to be read as Word documents. Every page can be converted into a PDF document for printing and/or emailing. Multiple pages in a category can be linked together to create a PDF book. For example all pages tagged to the Engineering Practice category could be linked so that a PDF version of the Engineer’s Handbook could be generated at anytime. Some of these pages may appear in other books as necessary.

It works well for three reasons:

1. Creating documents is easier because you don’t necessarily need to know where a particular document should filed. Just create it and save it. It will be easily searchable and can be categorised anytime later.
2. Information is kept up to date. It’s easy to update documents and its easy to see who has updated and when.
3. Most importantly information is easy to find. There’s only one place to go for any information that the company has in its manuals etc. You start Internet Explorer and browse the home page of the Wiki. If you have a good knowledge of the document you are after you can drill down any number of category paths to find it. If you have no idea just use search to quickly find your document. Once found you can bookmark it as a favourite

The great thing is that success in finding documents feeds the energy to create new documents and the system naturally gets better and better

Š

We get asked this a lot when customers are purchasing new servers.

The answer is 10-15 years if you are asking how long it will be before the hardware dies.

Most of the HP LC3s that we sold in 1996 would still fire up if anyone bothered to turn them on. The problem is that their hard disks (4Gb) would be way too small and their processors way too slow to run today’s software.

More importantly for any mission critical server (like an email server) is that you could not buy spare parts. The hard drives, RAM memory and Processors that you buy today as spare parts would not work in the LC3 and the original hard drives, RAM memory and Processors have not been made for many years.

The life span we aim for is 3-5 years. Three years if the demands of software change quickly (like they did through the 90s) and up to five years if the new server software is not so demanding.

Generally you cannot get warranty extended beyond five years and no critical server should be run without a current warranty.

Š

Why spend on Information technology? What will you get out of your investment?

At LANcom we can demonstrate what IT does for you and what may be the benefits to your business (you decide if they’re real). We are also in a great position to see IT investment across a whole range of kiwi businesses. 

There are three reasons why LANcom customers invest in Information Technology.

•  Decreasing your costs
•  Increasing your service levels to your customers
•  Making life easier for your employees

Ten years ago the dominant reason to buy IT was to save money. “I am spending $4000 per month faxing Japan. If I start using e-mail that cost will drop to $400 per month. Spending $5000 on an e-mail server is an easy decision”. Naturally there was lot’s of talk about return on investment (ROI) at the time as this kind of investment shows a direct return that is easy to quantify.

More recently the dominant reason has become increasing customer satisfaction. “If I set up a web site customers will be able to check out the status of their orders 24/7″. There is no immediate cost saving to balance the extra cost of the new technology (in this case a web site with customers orders on it). It will cost you more money than before. The return is from keeping customers and getting new customers by getting a service edge on the market. Of course your competitors catch up and soon deploy the same technology and soon having a web site to service customers is just part of the cost of being in the game.

The third reason is obvious. If your staff have quality equipment to work with they will be happier and more efficient. Nobody needs a bigger screen than they have already. It just makes it easier to work.

This break from straight cost-saving investment has made the investment and return equation complex. Often quantitative analysis is not done at all and the decision maker makes a gut decision. As Kiwi companies get more and more dependent on IT and as their future becomes more and more dependent on quality IT investments these ROI calculations will have to be made. We will all have to up our game in making quality ROI estimates.

This was a chapter from an Internet security backgrounder report I wrote about two years ago.

I wrote a Spam backgrounder today (posted soon) and it made me think of this.

It’s still a valid question and it most often gets answered in slogans (Microsoft = bad etc) so I thought I would post it.

In the last 20 years the two most influential and dominant forces in computing have been Microsoft software and the Internet. If you know a little of the history of each you will better understand why there are tens of thousands of viruses and security that seems so poor.

Microsoft Software

In 1990 Microsoft was an important, but by no means the only dominant player in the incredibly fast growing personal computer market. Although Personal computers had been around since 1978 (the Apple II) there were still very few true cross vendor standards.

The keystrokes to print in the market leading Spreadsheet – Lotus 123 – were completely different to those needed to print in the market leading word processor – WordPerfect. There were competing network standards and it was rare for computers to be able to connect to similar computers and almost unheard of for connection between dissimilar Personal Computers.

In 1990 Microsoft released Windows 3.0 which was about to change the landscape in usability. To print was now File/Print for any application running in Windows. Within a couple of years Windows for workgroups made it simple for a user to connect their PC to another without extra software or expert help. Applications were enhanced so that they could cooperate – Excel spreadsheets could be embedded in a Word document.

In 1990 just getting computers to work for ordinary people was the biggest challenge. The next 10 years of development at Microsoft was focussed primarily on making computers as easy to use as possible. Software was designed to be as obliging as possible because the number one problem was complexity and before the Internet the only people who could connect to your computer were in you company. Why would the Sales Department start hacking into the Accounts Department?

The Internet

In 1973, the U.S. Defense Advanced Research Projects Agency (DARPA) initiated a research program to investigate techniques and technologies for interlinking packet networks of various kinds. This project developed the TCP and IP protocols that were to become the backbone of the Internet.

By the late 70s the project had been handed over to the University community who used it to communicate between their sites. Development was fast but growth of the network much smaller so that most of the Internet’s major protocols were designed on a network where there was almost no anonymity –  In 1984 when the Internet standardised on TCP/IP for its communication protocol there were only 1000 sites on the system and everybody knew everybody else. Today there are over 250 Million computers connected and your connection can be reached by any one of these

Microsoft software on the Internet – eager to please software on a trusting network

So when Microsoft software started working with the burgeoning Internet it was, in hindsight, a predicable debacle. The core protocols (TCP/IP for computer connections connecting, SMTP for e-mail, HTTP for web browsing) of the Internet allow anonymous interaction and fundamentally trust that the other end of the communication does not have malicious intent.

The trusting Internet delivers these connections to Microsoft software that is way too eager to help out. From its early days Microsoft Outlook has been designed not only to help people construct and send e-mail but also any other program on the computer. When you click File/Send in Excel it is outlook that steps up and sends the Spreadsheet file on behalf of Excel. The problem has been that until the latest two versions Outlook did not discriminate who it helped out.  So if a virus got on your computer Outlook would give it the list of all the contacts you had in you contact folder and send virus infected e-mail to these people when requested by the virus program.

Microsoft has been widely criticised for its approach to security. The reasons why they got caught can be explained but they certainly have been very remiss until recently in facing the problems. They have spent most of their time on the Internet lagging behind security threats.

Microsoft was late to the Internet party itself but managed to turn around a sizable handicap into technology leadership on the Internet. Hopefully they can focus the same energy to their Internet security problems. Recent releases are encouraging.

One of the design outcomes of the Internet was anonymous access. It is unlikely that this was a design goal but rather identification and authentication seen as unnecessary on a Network where everybody was accountable. For many of the services on the Internet malicious and unaccountable users make anonymous access less and less feasible. The Spam epidemic is the best example of this. You can expect to reveal your identity (explicitly and implicitly) more and more often when using the Internet.

When we talk about upload and download speeds with broadband, what are we talking about and why are they important?

When you connect to the Internet data is flowing in two directions all the time to and from the Internet.

Whenever you browse to a web page your Browser sends a small requesting message to the web server involved to ask it to send back a web page that it can display. This is is uploading data (although a small upload). When the Web server sends back the page to the browser, this is downloading (usually a bigger load than the request).

The average user will download more data (web pages and files) than they will upload (request for these web pages and files). E-mail has similar ratios. Most users receive more e-mail than they send and therefore download more data from the Internet (receiving mail) than they upload to the Internet (sending)

The most common form of broadband in NZ is ADSL. ‘A’ stands for asymmetric and means that the download speed will be faster than the upload speed.

For a web server the upload (large web page) and download (small request) is the opposite of a user. For this reason the crippling of upload ADSL speed to a miserly 128kb/s was a big issue for our customers with web servers.