Last Updated Fri Nov 21 10:20:34 2008

[problem] [history] [technical] [return addresses] [resolution]

Problems with Barracuda blacklist

Barracuda Networks may make good email filters, but their public relations and support leaves a lot to be desired. Our primary mailserver was repeatedly blacklisted by their appliances (which are installed at two of our major business partners, causing rejection of legitimate email). They offer no explanation beyond a vague "we have seen spam from it" - no evidence was produced despite repeated requests. Calls to Barracuda helpdesk were referred to a webpage which promised to investigate "within 2 business days" (i.e. up to 96 hours).

The Problem

The problem is that Barracuda will not explain why they are blacklisting us. They have not responded to email or voicemail for technical support, and their "Intent Manager", apparently the only person with any authority, is variously away all week, busy, at lunch, or off sick. Se we are left to guess, and to complain about Barracuda Networks to anyone that will listen in the hope that someone might know something. Since we are a research organization, not some big corporation with an aggressive marketing division, we don't have lawyers on speed dial. Besides, this is Canada.

History

On 22 October 2007, a user reported that emails sent to 3 people bounced, with a DSN like
----- The following addresses had permanent fatal errors -----
<sally@example.com>
----- Transcript of session follows -----
... while talking to shield.example.com.:
>>> DATA
<<< 554 Service unavailable; Client host [142.90.100.150] blocked using Barracuda Reputation; http://bbl.barracudacentral.com/q.cgi?ip=142.90.100.150
<<< 554 Error: no valid recipients
My eye hit on the last line, and as this was a non-academic (i.e. for us, non-work-related) recipient, I misdiagnosed it as a problem at the far end (user account changed, etc.). Sendmail logs showed something like:
Oct 22 11:11:36 trmail sendmail[13930]: l9MIBZUI013926: to=<sally@example.com>, ctladdr=<someone@triumf.ca> (368/100),
  delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=387670, shield.example.com. [192.168.22.33], dsn=5.0.0, stat=Service unavailable

The next day we had more bounces, and it was apparent that work-related mail to partners such as SFU, MDS Nordion and UCLA was being rejected. Investigation showed that connections from our mailer to the mailserver at certain institutions were getting a "554" SMTP error status at intial connection (before HELO/EHLO).

We immediately used the Barracuda web form to request delisting from the "reputation list", and were somewhat surprised to see a "2 business day" resolution time as compared to, say, 1 hour for the Composite Blocking List or instantaneous for our in-house DNSBL. We then phoned the Barracuda support line to try and find the reason for listing, as the lookup page gave no indication. The helpdesk staff were polite but unable to help, and a tech support person said, basically, that their methods were proprietary and that the customer support line was for customers (not victims).

Technical

TRIUMF has been sending email internationally since about 1984 (HEPnet/DECnet and Bitnet, for the historically curious). I don't recall offhand any third party blocking our mail like this. We've had the odd visitor's laptop doing direct-to-MX mail; we get reports from SpamCop with full headers, verify from traffic logs, and deal with it. No problem.

So with no evidence to go on, we tried to guess why Barracuda might be listing us.
It was fairly clear that we were not actually sending spam through our mailserver. The server logs showed no evidence of a traffic increase, and we had no complaints whatsoever from end users, SpamCop, other DNSBLs etc.
Possibilities seemed to be

We have occasionally had TRIUMF addresses used in forged return addresses, and coincidentally during this period a couple of users reported Barracuda-related delivery failure messages - ones with no TRIUMF data except in the return address. But Barracuda deny that they are looking at return addresses, and it does seem unlikely for anyone nowadays to do this.

We moved our mailing lists to use a different outbound server, but this did not result in the second server being listed and the original server was subsequently re-listed. The list software (mailman) is in any case supposed to automatically unsubscribe dead addresses and our lists are typically small and members-only.

We historically forward a high proportion of user accounts - our users go on sabbatical, or work both here and at another institution, so we have used the traditional Unix mail forwarding scheme extensively. Since the spring of 2007, when we implemented a sendmail SpamAssassin milter, we have been filtering most of this mail before it is forwarded offsite. But in general it is a user's responsibility to decide what to forward to their account elsewhere. We looked at these forwarded accounts but could see no specific issues, e.g. dead offsite accounts. However, since our filters cannot exactly match filters at the target organization, there will always be cases where we forward mail that they label as spam (and vice-versa).

We looked at delivery failures, as reported to "postmaster". Some years ago, these were in general real problems that an administrator could fix. But more recently "postmaster" has been the target of direct spam, and the rest of the mail is mostly non-delivery from forged return addresses. We had got out of the habit of looking at this mail, assuming that users will handle genuine non-delivery events. On examining these records, we found a few specific problems:

I hypothesize the following: If we receive spam which has the forged return address of a Barracuda customer, then generate a DSN because we cannot forward this email for any reason (e.g. filter rejection, mailbox full, account deleted), then the spam is sent to the Barracuda appliance. We have recently changed our configuration so that the mail body is no longer returned in a DSN, which might help.

This is still conjecture, but it seems a plausible mechanism. In fact, we are doing exactly the same thing - analyzing mail and adding repeat spamming addresses to an in-house blacklist. The differences being that we don't distribute this list to hundreds of other sites, that our delisting page takes immediate effect, and that we would if asked show evidence and examples of such spam.

Barracuda's attitude, as stated in their website and auto-generated email, seems to be that they only sell appliances and that the responsibility for blocking our mail lies with their customers, so we should complain to them, not Barracuda. However, the reason that these customers have bought appliances in the first place is that they are non-technical or don't want to deal with their own mail. So trying to complain to them is like yelling into the wind. Besides, there are dozens of them and even contacting the IT department at an occasional correspondant can be difficult.
We did manage to get whitelisted quite quickly at SFU. At another company, who had outsourced their IT services, it was a question of asking an employee to generate a trouble ticket and escalate it through the system. When we did make contact, it took some time to explain the problem and then they had finger trouble whitelisting us in the appliance - it seems that there is both a name-based whitelist and a numeric whitelist, and only the numeric list takes precedence over the intent blacklist. The whole process, from initially contacting the employee to finally getting whitelisted, took a couple of weeks. Having to repeat this process at some 40-odd companies did not seem reasonable.

Return Addresses

Email has, at minimum, a return address and a recipient address. These are usually the same as the "From" and "To" addresses which the user sees, but need not be in e.g. mailing list mail or blind-copy mail. The return address (from the SMTP "MAIL FROM:" sequence) generally shows in email headers as "Return-Path". "From:" and "Reply-To:" which are visible to the user are independant values extracted from the message body.

In recent spam, the return address is not generally correct - the spammer does not want to receive replies, or even DSNs. However, since one long-standing mail filter requires that the return address has at least a valid domain name, it is often the real address of an innocent third party. In polymorphic spam, it will change between each message of batch of messages to avoid creating a recognizable pattern. My suspicion is that the spam engine simply re-uses the recipient list as a source of forged return addresses.

Resolution

Essentially none.

We still have no answer from Barracuda as to exactly why were listed, and hence no expectation that we will not be listed in the future. We have gotten two partners to whitelist us around their appliance using the numeric whitelist.

Some people have suggested we should threaten legal action, or actually take Barracuda to court. But this is Canada; the best we can do so far is a strongly worded note.

But gone away..

The following measures seem to have made the problem disappear on our sendmail-based system: [problem] [history] [technical] [return addresses] [resolution]
A.Daviel