Possibly Historical Document - Last Updated Sun Feb 20 02:22:36 2005

An analysis of online UBE filters

(from November 2000, I think)

See also Spam statistics (2002), Blacklists Compared (Jeff Makey, ongoing), Spam statistics (2003).

In the last few years, UBE - unsolicited bulk email, commonly known as spam - has become a fact of life for many of us. For some, it is open warfare between the senders, who see it as a low-cost advertising medium and an expression of free speech, and the unwilling recipients, who see it as an invasion of privacy and a hidden cost of being online, since they may pay bandwidth or dialup charges to receive it.

Each side in the battle tries to outwit the other. The recipients devise filters and the senders try to find a way around to get their message across. In the early days filters were simple - refuse any mail from AOL or CyberPromo. Later, the number of senders increased dramatically, while domains like AOL and Hotmail began to be used by mainstream users and could no longer be blocked. Maintaining filters manually became an impossible task. Around this time, the senders discovered SMTP relay - they could route and amplify their messages through another server elsewhere, often without the permission of the owner.

In response to this, a few organizations developed online databases of known spammers and open relays. These tools make use of the innate ability of Unix "sendmail" to resolve Internet addresses by allowing the database to be queried using Domain Name Service.

To determine the effectiveness of four of these tools, I used mail from some of my own folders. I analyzed 24,000 messages from 3,500 unique senders, using personal mail and mail sent to the BugTraq security mailing list, a VRML mailing list, the Vancouver Linux User Group and an Internet robots mailing list. The four tools were RBL, DUL and RSS, all at MAPS (mail-abuse.com), plus ORBS (orbs.org).

BugTraq is a respected mailing list, yet its listserver is blocked by one of these tools. In testing, this block was ignored and only the original senders and relays were checked.

Procedure

The headers of each mail message were examined. Each email relay or server in a Received: header was looked up in the following databases using the DNS-based method (gethostbyname): In the case of the DUL list, only the first Received header was checked (DUL checks for mail sent directly from a dialup address).

A log was made of the results of each lookup, together with a personal assessment of whether the mail was in fact UBE or not. In the case of Bugtraq, the list is known to be well moderated and it was assumed that there was no UBE. In the case of VANLUG, posting to the list is restricted to members and experience has shown that there is little or no UBE. The remainder of the mail was assigned a Yes/No tag based on the subject line and in some cases examination of the content.

Some correspondants send a large amount of mail; an attempt was made to count unique addresses ("From" + mail relay). The analysis is not 100% rigorous; the idea was to get a rough idea how reliable the filtering services are.

Note that the database checks were not made at the time the mail was received, but all at once during analysis. Some untagged mail would in fact have been blocked, and vice-versa, since the database contents change over time.

Results

Note: This survey was performed in 2000. ORBS is no longer in operation, and it was considered too aggressive for use at TRIUMF. These results do not represent current (2003) performance of anti-spam blacklists.

Total Messages: 24836
Unique Senders:  3592

                  ORBS   RBL   DUL    mail-abuse   Total
Messages Tagged   8329   95    52     610          8940
False Positive    6492   31    10     352
Hosts Tagged      424    34    34     120
False Positive    268    12    6      40
UBE Missed        42     55    24     51              6
Conclusion: If you filter using all 4 domain-based systems, you will receive very little UBE. You may also lose a significant amount of legitimate mail. Domains blocked include:

Jan Krüger's sendmail patch (below) offers an option to not block suspected UBE, but to add extra headers to it. This would allow end users to use their own filters.

References

Sendmail related links
mail-abuse.com (MAPS)
ORBS
check_local by Jan Krüger

See also

Andrew Daviel <>

Odysseus the Wanderer has nothing to do with this page. Do not send him mail! interim

interim
interim
interim