Current Spam Filter Statistics

Configuring your Barracuda Account

Introduction to Spam

Spam is a rampant problem on the Internet, and increases daily. Fighting it is an arms race, as spammers find ways around defenses, requiring new defenses, in a never ending cycle. The latest trick you may have noticed is to put the spam message in an image with enough "noise" to foil automatic attempts to read the message for analysis (taking a lesson from the "Captcha" techniques many websites now use to avoid automatic account creation by bad guys). The spam fighters are still working on ways to deal with this problem... Unfortunately, it's so cheap to send spam that it only takes a very small response rate from scam victims to make it highly profitable.

Judging from questions we get frequently, there are some misconceptions about mail and spam that need to be dispelled before continuing.

In short:

  1. Email is trivial to forge --- you can't trust what a message says, either about who it's from or who it's to (or anything else for that matter!).


  2. *NEVER* click on a link in email and trust the result. If you think a message requesting confidential information, or that you login to a web site containing confidential information, is legitimate, manually enter the URL in the location field in your browser, or select a previously saved known good URL from your bookmarks (though I've heard of viruses hacking your bookmarks --- while rare, you should be careful there too: know your URLs and be suspicious!).


The key thing to remember is that email is trivial to forge, and spam and viruses almost always are. So if you get bounces or spam reports where it looks like you've been sending spam when you haven't, it just means that someone got ahold of your email address, not that your account has been compromised. Think of email like a letter in an envelope. What's written on the envelope is what's used to deliver the letter. In the email world, the "envelope" refers to the commands sent when the message is transmitted. What you see in your email is the letter inside, which can have anything on it the sender wants. That also explains why you get spam that doesn't appear to be addressed to you --- the envelope had your address, but the letter didn't.

250 mail02.peak.org Hello cvo-sr1-off.peak.org [69.59.192.10], pleased to meet you
mail From: santa@north.pole.com
250 2.1.0 santa@north.pole.com... Sender ok
rcpt To: joeblow@peak.org
250 2.1.5 joeblow@peak.org... Recipient ok
Example of SMTP "envelope"

The reasons for doing it this way are technical, but involve such things as making sure mail isn't duplicated and that mail list mail isn't cluttered with potentially thousands of addresses.

One of the reasons that email is trivial to forge is that defining just what is and isn't "forgery" is tricky in the online world --- how does someone tell the difference between you sending mail from a friend's house or Internet Cafe vs an imposter? Because it has become such a problem however, there is a new standard that is being adopted to fight it. This is called Sender Policy Framework (SPF, http://www.openspf.org/). Basically, domain owners publish a list of mail servers authorized to send mail using From addresses in that domain. Recipient mail servers can then check the server sending the mail against that list to decide whether or not to accept it.

Unfortunately, this can cause problems for people traveling and wishing to send legitimate mail from non-standard locations. The solution to *that* is for such people to configure their system to use their own ISP's mail server when sending mail, using authentication to identify themselves as a legitimate user, and not a spammer trying to cover his tracks by relaying through a third-party mail server. Another option is to use their ISP's webmail interface.

There is still, however, an issue with people using mail forwarders, which can be anything from having one address setup to forward to another, to mail lists that forward mail with the original sender's From address rather than using the list address as the From address. Dealing with these issues is enough of a problem that on an ISP wide basis, we can't use it to definitively block "forged" mail, however we do publish a list of peak.org mail servers so that mail servers can be properly suspicious of forged mail.

Another option that *should* be used more often, but isn't, is digitally signed email. This uses cryptographic techniques to ensure that the message is from who it says it is. Unfortunately, making sure the signatures are trustworthy is a complex task, with the result that no one has made it easy to get and use the technology in outgoing mail yet. Even so, there is no excuse for organizations to not use it in their official communications, which would eliminate a lot of phishing opportunities (mail that is specifically forged to appear to be from an institution, often financial, in order to trick you into revealing sensitive and confidential information to bad guys). Most modern mail programs will validate signed mail (Outlook shows a ribbon to the right of the message headers if a signed message is valid; Thunderbird shows a little pencil icon in the lower right corner).

To reiterate:

  1. Email is trivial to forge --- you can't trust what a message says, either about who it's from or who it's to (or anything else for that matter!).

  2. *NEVER* click on a link in email and trust the result. If you think a message requesting confidential information, or that you login to a web site containing confidential information, is legitimate, manually enter the URL in the location field in your browser, or select a previously saved known good URL from your bookmarks (though I've heard of viruses hacking your bookmarks --- while rare, you should be careful there too: know your URLs and be suspicious!).

What We're Doing About It

At Peak, we have a spam fighting appliance called "Barracuda" from Barracuda Networks (actually, we have two setup in parallel so that if one has problems, the other will keep things flowing, but they are configured to look like one single system for most purposes). The people at Barracuda constantly monitor trends in spam and viruses, and the appliances update themselves hourly with the latest information.

While not perfect, it does a very good job: In the last year or so, it has processed over 300 million messages. Of those, 270 million were blocked (600,000 of which were viruses), 5.5 million were quarantined as suspicious, and only 30 million were passed as legitimate mail. Currently, we're running at about 1.4 million messages/day, of which only about 8% are legitimate.

Email statistics from one of the barracudas
Email statistics from one of the barracudas

You will almost certainly have gotten one of the daily Spam Quarantine Summary notices from the barracuda. Here's some info to help manage and control how the barracuda processes your mail:

First, note that the barracuda is an independent appliance, and account management and email systems are pretty much different for every organization. As a result, the barracuda has no link to the mail servers other than via standard delivery protocols. That means it doesn't know which addresses are aliases pointing to the same mailbox, so as far as it's concerned each one is completely separate. Here at peak, because of the history of the company, it's common to, for example, have "joe@peak.org", "joe@casco.net" and "joe@pioneer.net" all point to the "joe" mailbox. If Joe has his own domain, he might have "joe@joe.com", "postmaster@joe.com" and "webmaster@joe.com" pointed there as well. To the barracuda, these are 6 different entities.

Spam Quarantine Summary example
Spam Quarantine Summary example
As a result, if you're getting these summary messages for addresses you don't use, the first thing to do is to call our support line and request that the unused addresses be removed from your account. Any mail to them will then be rejected and not waste your time.

When you do get a summary message, it will have a list of messages with an "Actions" column that allows you to individually Deliver, Whitelist or Delete the messages.

First off, note that this is a static email message: clicking an appropriate Action link will not result in any change to the email message, as it's already in your inbox.
Second, if you look down at the bottom of the message, it will say:

To view your entire quarantine inbox or manage your preferences, click here.

where "click here" is an active link. This is the best place to go to manage your quarantine, because that actually takes you to the barracuda's live web interface. You will need to use a recent such message, as the "click here" link contains a randomly generated code that is only valid for about a day or so for security reasons.

  Spam Quarantine Inbox example
Spam Quarantine Inbox example

Once there, you will be viewing your current Quarantine Inbox. This interface has a very nice feature that makes it easy to clear the spam, but first, scan the list to see if there is anything that is legitimate. If so, check the boxes on the left of the legitimate messages, and then rather than clicking on one of the Action links to the right, click on the "Classify as Not Spam" button at the top. This allows you to process all the legitimate messages at once. More importantly, it causes the barracuda to analyze the selected messages and learn the patterns in them to reduce the likelihood of similar messages in the future being marked as spam. Then it delivers the messages and removes them from the quarantine.

When any and all legitimate messages have been processed, check the check box at the top of the checkbox column, to the left of "Time Received". This causes all the boxes on the page to get checked, making it really easy to click "Classify as Spam" and have it process all the messages as spam (learn the patterns and delete them) in one fell swoop.

If you're unsure about a message, you can click on it and the message will pop up in a separate browser window for you to take a look at safely. For those who might be interested, doing so will also let you see the patterns in the message and how they affect the spam scoring (click on "View Bayesian Breakdown").

Tweaking Your Barracuda Settings

If you like, you can fine-tune your barracuda settings to see if you can improve its spam filtering abilities for your specific mail patterns. The easiest way to get to the control interface is to use one of the Spam Quarantine Summary messages and the "click here" link. If you don't have one handy, and the address you want to manage is in the peak.org domain, then you can just login with your regular login (we were able to tie it into the main authentication system that much).

Unfortunately, if the address is not in the peak.org domain, the barracuda makes it more difficult: you'll have to call into the support line here at Peak and ask to have your password set to something known. We're working with Barracuda to fix this issue...

Once you get in, click on the Preferences tab, which gives you access to Whitelist/Blacklist, Quarantine Settings and Spam Settings option tabs.

  White/Blacklist Inbox example
White/Blacklist example

The Whitelist/Blacklist is pretty basic and obvious: enter addresses into the whitelist if you don't want them blocked for any reason, and the blacklist for those you never want to see mail from again. You can white or black list an entire domain by just putting in the domain name. There is no pattern matching, however (i.e. you can't say "joe*@somewhere.com" to match any addresses that start with "joe").

Quarantine Settings example  
Quarantine Settings example

In the Quarantine Settings tab, you can enable or disable the quarantine entirely. This is where the barracuda holds messages for up to a month that look like spam, but might not be. We highly recommend using the quarantine, as it's only from here that you can train it as to what looks like spam and what doesn't, from your specific perspective. As a result, you shouldn't need to make any changes at all in this tab.

  Spam Settings example
Spam Settings example

The main place for making adjustments is in the Spam Settings tab. If you find that you just don't get along with the barracuda at all, you can turn off spam filtering entirely here. Sorta: unfortunately, some of the spam filtering techniques are used before the recipient address is known, and in a few cases, the barracuda just does them anyway. This generally isn't a problem, but if it is in your case, you can get your own private domain that does not get routed through the barracuda at all. Since mail is routed by domain name, not specific address, that is the only way to avoid it.

Most people will have "Use System Defaults" set in the Spam Scoring section, but if that's the case, you probably haven't gotten this far anyhow :-) If you have, this is is the spot where you can adjust the levels at which various choices are made by the barracuda. In order to do so, you need to click the "No" button on "Use System Defaults" and "Save Changes". That enables the custom setting sliders.

Custom Settings example  
Custom Settings example

When a message comes in, the barracuda runs various tests on the message, such as "Is the sending server on a black list?" and "Does the message look like spam or ham?" (ham is non-spam --- that's really what it's called in spam fighting circles!) It then assigns a score noting the likelihood that the message is spam. Negative numbers don't look anything like spam to the barracuda, while anything above about 6-7 looks very much like spam.

The actions available are Tag, Quarantine and Block:

When a message is Tagged, the Subject of the message has "[SPAM?]" prepended to it, then the message is delivered. This marks the message as suspicious, and you can have your email program's filtering functions take appropriate action.

Messages exceeding the Quarantine threshold are held on the barracuda as described above.

Those exceeding the Block threshold are discarded.

The default is to disable tagging, to quarantine anything scoring above 4, and discard anything over 7. This gets rid of the most obvious spam, while allowing you to train the barracuda about those messages falling in the grey area.

Why Did I Get This Message?

All spam filters are imperfect --- it's the nature of the business. Some people like to know more about why messages did or didn't get through though. For this reason, the barracuda adds information about how the message was scored into the message headers. Most mail clients don't show you this information by default --- you'll have to ask for it. In Outlook, you have to right-click on the message, then Properties/Details (it's easier to read if you continue on to Message Source). In Thunderbird, you just click View/Message Source, or type Ctrl-U (Cmd-U for Macs).

What you get by doing that is some very cryptic text that includes something like this:

X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0000
X-Barracuda-Virus-Scanned: by Peak Internet Spam Firewall at peak.org
X-Barracuda-Spam-Score: 2.26
X-Barracuda-Spam-Status: No, SCORE=2.26 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=3.0 KILL_LEVEL=4.0 
	tests=BAD_ENC_HEADER, HTML_MESSAGE, MIME_HTML_ONLY
X-Barracuda-Spam-Report: Code version 3.1, rules version 3.1.8660
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	2.25 BAD_ENC_HEADER         Message has bad MIME encoding in the header
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.00 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts

The useful stuff in this is:

X-Barracuda-Bayes: FLAG DATABASE PROBABILITY CONFIDENCE SCORE
FLAG: Indicates whether a message was Spam or Not-Spam
DATABASE: Indicates whether a user or global database was used.
PROBABILITY: Chance the message is Spam.
CONFIDENCE: Chance the scoring was accurate based on tokens in use.
SCORE: Amount added to the overall score
X-Barracuda-Spam-Score:
Overall spam score for the message. This is then compared with the Spam Setting values to determine what to do with the message.
X-Barracuda-Spam-Status:
This is the overall summary of how the message scored, compared with your spam settings. The tests that matched are listed also. Note that a setting of "1000.0" corresponds to "10" on the slider, and means that action is disabled (e.g. in the example above, Tagging is disabled --- if the message is delivered at all, the Subject line will not be modified).
X-Barracuda-Spam-Report:
Finally, the Report gives more detail on the matching tests: how the test affected the score and more detail about what it actually means

For More Information

That's pretty much all there is to it. If you want more information, you can get the full Barracuda User's Manual from their website, and of course, our support people are always happy to answer any questions you may have.