Possibly Historical Document - Last Updated Fri Jun 15 15:27:46 2007

* Filters on TRmail * Filters in Netscape * Filters in Thunderbird * SpamAssassin * Typical spam (annotated) *
Reporting spam * Preventing spam * False Positives * New Stuff (2004) *

Filtering out Spam

May 2007

Further to the spam rejection based on sender ip address detailed below (March 2004), we will now reject spam based on content. Prior to June 2007, we accepted spam, tagged it, then either discarded it (with a score S>14) or passed it to the user, with the option of server-side filtering.

There is now a threshold above which spam will be rejected up front. The default value is 14, the same number at which spam was discarded for those with server-side filtering. This number is configurable at Set Spam Rejection Score in User Config.

There are now two user-configurable values - the rejection threshold, above which spam is rejected, and the tag threshold, above which it is labelled as spam and may be filtered into a separate folder (often 'junk') by server or client-based filters. The current defaults (14 and 6) are shown, but users may change both. If the rejection threshold is set equal to or below the tag threshold, users will "see no spam".

For individual mail, the scoring is identical to the filter/tagging already performed - user whitelists act identically to pass commercial newsletters and personal mail.

For mail addressed to more than one recipient - old-style mailing lists and personal mail addressed e.g. "To: John.Doe@triumf.ca, Jill.Poe@triumf.ca", a default threshold and whitelists will be used. Until we gain more experience with this, we will probably re-score mail for each recipient - it is possible, though unlikely, that a newsletter sent to two people may be scored at 8, hence tagged, but the recipients may whitelist the sender and remove the tag.

A note on rejection: the difference between "reject" and "discard" is the difference between refusing to sign for a FedEx package, and accepting it then dropping it in the trash. If the courier is legitimate, they will notify the sender that it could not be delivered. A legitimate sender would see a delivery notification like this:

  Final-Recipient: RFC822; Jill.Doe@trmail.triumf.ca
  Action: failed
  Diagnostic-Code: SMTP; 550 5.7.1 Rejected; see
  http://trmail.triumf.ca/cgi-bin/SA?16.70+10.00
  Last-Attempt-Date: Wed, 2 May 2007 15:32:27 -0700

Another change is that forwarded mail will now be filtered; if a user has a .forward file to forward mail to another institution, we will no longer forward spam with a score above threshold.
As mentioned below, we whitelist partner institutions, and mail sent direct from these will not be rejected. Please send requests to add new partner institutions to Andrew Daviel or to Computing Services.

Technical: The new filtering is implemented using a milter in sendmail (spamass-milter-0.2.0, with some modifications). The mail flow is show diagramattically in Old Flow, New Flow (PDF).

Also new (May 2007): See advanced filters in Filters on TRmail

March 2004

TRIUMF is now adopting a more aggressive attitude to spam in response to the rise in volume (virus-sent spam, etc.)
We are now rejecting some mail up front with an error message. The message contains a "get out of jail" card (from the board game "Monopoly"..)

Current Features:

Email coming from an ip address listed in cbl.abuseat.org, rbl2.triumf.ca, or list.dsbl.org is rejected, unless it is whitelisted.
CBL lists mostly PCs infected with virus-borne trojans that let them relay spam. List.dsbl.org lists machines with an open proxy or mail relay. If a system is on one of these lists, it should be disinfected or repaired. There is a well-defined mechanism to get off the lists.
rbl2.triumf.ca lists machines that have sent us spam (it is a destination for TRIUMF spamtraps and the spam@triumf.ca mailbox). It may occasionally acquire legitimate addresses, but there is an immediate get-out mecahnism, and entries are purged on a regular basis.

Senders see a rejection message such as:

550 5.0.0 <visitor@triumf.ca>... Rejected; see http://cbl.triumf.ca/62.135.123.51
550 5.0.0 <enigma@triumf.ca>... Rejected; see http://rbl2.triumf.ca/218.48.222.72
550 5.0.0 <steve@triumf.ca>... Rejected; see http://trmail.triumf.ca/cgi-bin/dsbl?62.103.113.68
There is also static whitelisting of collaborating institutions; see whitelist.cf
Users may add their own entries using the Web configuration tool

Statistics:

Junk email is variously known as "spam", unsolicited bulk email (UBE) or unsolicited commercial email (UCE). "Spam" is actually canned luncheon meat, a trademarked product of Hormel Foods, made famous by the Monty Python Spam Sketch

previous information:

There are a number of organizations that maintain a database of Internet sites that either send spam themselves, or allow spam to be relayed and amplified through their mail servers. These databases may be queried online using the DNS lookup mechanism; see the Spamalyzer to check a particular address. See UBE filter analysis and spam stats for a comparison of these databases.

The TRIUMF mail server (trmail) now adds extra headers X-Spam-List and X-Spam-Warn to email messages if the sender is listed in one of these databases. This allows users to separate or discard suspected spam by using mail filters. Currently several databases are queried; these change from time to time:

Spamcop and XBL are more aggressive (may flag more legitimate mail) and generate a Spam-Warn header.

Note: These headers are only added to mail coming directly from a listed site. They are not added to forwarded mail, e.g. from another institution or computer. So in this case you may see UCE that would otherwise be filtered.

Note: Filters are not perfect. You will get some junkmail that is not filtered. You may also get some legitimate mail that is filtered, because the sender is unwittingly running an open relay, or is sharing a relay or ISP with people that are sending junkmail.
You are advised to check the junk folder for legitimate mail occasionally;

Filtering Spam on TRmail

May 2007: Advanced Filtering:

In addition to the predefined "canned" filters listed below, users can now build more complex server-side filters using a Web tool. This tool may be found under "Filter mail/spam (advanced)" in "User Config" (Mail Settings) on trmail.triumf.ca.

The tool builds a sequence of decisions as follows:

If a condition is met, then the following conditions are not evaluated. The following kind of filters may be built: Note that as of June 2007, "discard" is deprecated in favour of "reject" - which must happen earlier in the process than filtering, and is controlled by SpamAssassin scores (blacklists)

Regular Filtering

Some "canned" filters have been added on the TRIUMF mailserver trmail. You can select one of these using the Web configuration tool

When you have selected a filter that uses the "junk" or "lists" folders, you may have to use the "subscribe" feature in your mail client to subscribe to them before you can see them. In Netscape Messenger 4, this is in the "File" menu from the toolbar.

Filtering Spam by signature

Providing that the CPU load is low, mail from offsite is being checked by the Razor filtering service. This filters individual messages based on an MD5 checksum generated by spamtraps and samples submitted by users (see next paragraph).

Reporting Spam

Please bounce spam immediately to spam@triumf.ca. In Pine, the "bounce" command B is easiest (may need to be enabled in setup/config). In Netscape or Outlook Express, please forward the message as an attachment *. This will enter it in the Razor database, rbl2.triumf.ca and possibly DSBL; the quicker it is submitted the more people will be protected. For other mail tools, please contact Andrew Daviel
You may also submit spam to spamcop.net (requires registration), or to spam@uce.gov

If the message is more than a day or so old, just delete it.
Please be sure that the message really is spam, and not a message from a mailing list you forgot you subscribed to, or an antivirus warning, or membership reminder. False positives on personal messages are harmless, but false positives on legitimate mailouts can inconvenience many people.

* In Netscape on Windows, click-and-hold the Forward button, then select "Attachment".
In Outlook Express, from the toolbar select Message -> Forward as attachment
In Netscape on Linux, either

or change the default action of the Forward button (if you report spam more often than you forward mail): You may also forward a message with full headers enabled (specifically, X-Razor-SHA1).
(Normal forwarding does not work, because Netscape reformats messages, changing their signature)

Reporting Spam to Authorities

It is highly unlikely that this is useful - it is just too difficult to trace where mail really came from. Nowadays messages are probably sent via a proxy on someone's home PC, installed by a virus or trojan horse program. Such proxies don't keep incriminating logs, and the owner of the PC probably has no idea it is running.

However, there are cases where you may wish to report an email:

In these cases, please:

Filtering Spam on Unix machines

For alph04, lin00 etc. see procmail strategies

SpamAssassin

SpamAssassin is a rule-based system for scoring mail messages based on their "spamminess". Many spam messages use tricks to avoid content filters and to disguise their real origin, such as using invalid Hotmail addresses, splitting keywords with spaces or with HTML comments, and SpamAssassin catches these. It also adds points for things such as EXCESSIVE SHOUTING, phrases such as "this is not spam", "sent in accordance with US Senate Bill xxxx" etc., and subtracts points for things like a PGP signature. Messages which score over a certain threshold are flagged as spam in the subject line and also with an X-Status-Flag header. If you have junkmail filtering enabled, they will then be sent to the junk folder along with messages from open relays.

SpamAssassin is highly configurable. The configuration tool allows users to adjust their personal spam threshold, and to whitelist certain addresses such as commercial newsletters sent as HTML.

Filtering Spam in Netscape4

In Netscape, select Messenger.
Select File --> New Subfolder. Create a new folder named e.g. "spam" as a subfolder of "trmail".

Select Edit --> Message Filters.
Select "Filters for 'Inbox'". Select "New".
Name the new filter "spam".
(In Netscape for Windows, select "Advanced"). Select "Customize Headers...".
Create a new custom header "X-Spam-List" (and if desired "X-Spam-Warn").
Select "contains" "." (a dot) Set the action "Move to folder" "spam". Click OK.
This filter should appear before the default filter (if used); select the filter then use the up/down arrows to re-order it.

If you wish, you can check each database separately by matching (part of) dul.maps.vix.com, relays.orbs.org, relays.mail-abuse.org,rbl.maps.vix.com.

Filtering mail in Thunderbird

(June 2007)

Mozilla Thunderbird has two tools to filter mail, "Message Filters" similar to that described for Netscape below, and "Junk Mail Controls" which is an adaptive spam filter. This operates independantly from the server-side filtering based on SpamAssassin. It can, however, be set to trust the results from SpamAssassin as shown below

Thunderbird also uses message labelling in IMAP, and the server-side filtering has been adapted to set the "Junk" label.

Filtering List mail in Netscape4

For users using Netscape 4 to read mail, it is possible to use a mail filter to automatically move list mail to a separate folder. The procedure described below will move all mail not specifically directed to you to a separate folder. Note that this will include mail send to you using blind-carbon-copy (Bcc), and also mail from legitimate mailing lists, so should be used with caution.

Some (most) junkmail is sent to a list of people, rather than individually. This means that your name is not in the To: field. Netscape allows mail to be automatically filtered into a separate mailbox in such cases.

In Netscape, select Messenger.
Select File --> New Subfolder. Create a new folder named e.g. "junk" as a subfolder of "trmail".

Select Edit --> Message Filters.
Select "Filters for 'Inbox'". Select "New".
Create a filter named "junk" with "to or Cc:" doesn't contain your email address. Check "More". Check "match all". Create a second criterion with "to or Cc:" doesn't contain your long email address ("Your.Name@triumf.ca"). If you wish, create a third criterion with "sender" doesn't contain "triumf.ca".
Set the action "Move to folder" "junk". Click OK.
This filter should appear before the default filter (if used); select the filter then use the up/down arrows to re-order it.

Note: Some legitimate mail, such as mail from a mailing list, will also be filtered using these rules. You may need to create prior rules which handle this. The "triumf.ca" criterion attempts to preserve email from TRIUMF mailing lists or mail to multiple recipients in the inbox; however, some junkmail may appear to be from a bogus TRIUMF account and you may wish to delete this rule.

Note: Most legitimate mailing list mail includes a "Precedence: list" or "Precedence: bulk" header. You can add extra criteria "Precedence" "doesn't contain" "list" and "Precedence" "doesn't contain" "bulk" to the junk filter.

All this happens in Netscape, when you connect to the mail server. Mail is automatically marked deleted and copied to another folder, and cleaned up in the normal manner. Until you log in, it remains in your inbox on trmail.

It is also possible for mail to be filtered on the server, before you see it. If you do not subscribe to any mailing lists, so that your filtering requirements are simple, and are interested in this solution, please contact Andrew Daviel.

Typical Spam

This is a typical piece of spam email, tagged by mail-abuse.org. All headers are shown.
delivered to the local mail folder
Received: via dmail-4.1(9) for +mail/spam; Sat,
     7 Apr 2001 23:59:40 -0700 (PDT)
added by the mail software as best-guess sender
Return-Path: <achilles108@hotmail.com>
the mail was sent to trmail from golf210.co.jp
the numeric address (210.169.229.82) as seen by trmail is the ONLY thing that can be trusted in the entire message
The mail server said it was golf210.co.jp (SMTP HELO)
The mail server said this connection belonged to user root on machine nitto.golf210.co.jp
Received: from golf210.co.jp (root@nitto.golf210.co.jp [210.169.229.82])
        by trmail.triumf.ca (8.10.2/8.10.2) with ESMTP id f386xa322745;
        Sat, 7 Apr 2001 23:59:37 -0700
the spam tag added by trmail because the address 210.169.229.82 is in the RSS database
X-Spam-List: relays.mail-abuse.org
the mail probably originated at 209-239-204-142.oak.jps.net
This may be a dialup account. If the mail is so offensive that you feel you must complain, 
this is probably the domain to complain to (the owner of the address in square brackets,
[209.239.204.142]). 
This line was added by golf210.co.jp
Received: from 209.239.204.142 (209-239-204-142.oak.jps.net
    [209.239.204.142]) by golf210.co.jp (8.8.8+3.0Wbeta13/3.4W3) with SMTP id
    PAA26535; Sun, 8 Apr 2001 15:56:09 +0900
The "from" address. This may be completely bogus. Some mail servers require that
the From domain must exist (hotmail here) but there is no way to verify the username
From: achilles108@hotmail.com
in theory the message id is preserved if you reply. Sometimes it includes
the sending address or the mail tool
Message-ID: 00000aff71c3$00002fb6$0000114e@>
The "to" address. This may be completely bogus. If the To address is empty,
trmail will add "Undisclosed.Recipients" itself. It may be the name of a mailing list used by the sender,
or may be the first name on the list. This line is not actually used by trmail
to deliver the mail but is normally added for the benefit of humans
To: <Undisclosed.Recipients@golf210.co.jp>
Subject: Are you paying to much for inkjets?
The date that the message was originally sent. Usually includes the timezone
which in this case is GMT-8, i.e. PST
Date: Sat, 07 Apr 2001 23:09:01 -0800
priority levels added by Microsoft mail clients. Spammers sometimes like to 
set this to "high". Generally ignored by trmail, but may be displayed in Netscape
X-Priority: 3
X-MSMail-Priority: Normal
A reply-to header added by the sender. May be completely bogus, or it may be real.
A common technique I believe is to obtain a free address from Hotmail, Yahoo etc.,
then use it as a reply address for spam until it is deleted or overflows.
Reply-To: achilles108@hotmail.com

We currently have the follow specials on inkjet cartridges ...
A typical piece of spam caught by the "nobcc" filter:
Received: via dmail-4.1(9) for +mail/junk.in; Sun,
     8 Apr 2001 11:41:05 -0700 (PDT)
Return-Path: <drheim@jahoopa.com>
this was delivered to "postmaster" from gshnet.com.br (Brazil)
Received: from pagina.gshnet.com.br (IDENT:root@[200.216.236.225])
        by trmail.triumf.ca (8.10.2/8.10.2) with ESMTP id f38Ies323416
        for <postmaster@triumf.ca>; Sun, 8 Apr 2001 11:41:03 -0700
The sender probably has his email address set to "xxx@netscape.com". The mail
was sent from dialinx.net, not Netscape
Received: from netscape.com (PPPa61-ResaleCanogaPark1-1R7055.dialinx.net
    [4.4.110.90])
        by pagina.gshnet.com.br (8.9.3/8.8.7) with SMTP id MAA16718;
        Sun, 8 Apr 2001 12:42:09 -0300
This is going to be an HTML mail in the default western encoding
Content-Type: text/html;
         charset="iso-8859-1"
Content-Transfer-Encoding: 8BIT
Subject: Get the inside scoop on Anyone or Anything!
The sender is using Netscape on a Windows 98 machine
X-Mailer: Mozilla 4.07 [en] (Win98; I)
Message-Id: <61jo5v6pacfn8i1.mn6o6k67fb@netscape.com>
Date: Sun, 08 Apr 2001 11:38:45 -0800
the To address may be bogus
To: largeshark@honduras.com
The from address is almost certainly bogus, since this mail wants the recipient
to visit a website.
From: duntdunt@stribmail.com

This piece of spam was reported as coming from us. However, most of the headers are forged. (the domain and recipient have been altered to protect their privacy)

Return-Path: <lastChance48Ye3gyW@yahoo.com>
Received: from mx13.boston.jyno.com (mx13.boston.jyno.com [64.136.24.135])
        by m6.boston.jyno.com with SMTP id AAA8Q2UJQA2PVEQJ
        for <deye1@jyno.com> (sender <lastChance48Ye3gyW@yahoo.com)>;
        Wed, 22 May 2002 21:42:38 -0400 (EST)
The mail was sent to deye1@jyno.com and these headers were added by the mail server at jyno.com (and are real)
Received: from yahoo.com ([61.114.157.97])
        by mx13.boston.jyno.com with SMTP id AAA8Q2UJPASSX2FJ
        for <deye1@jyno.com >(sender <lastChance48Ye3gyW@yahoo.com)>;
        Wed, 22 May 2002 21:42:37 -0400 (EST)
The mail was received from 61.114.157.97, which pretended to be yahoo.com.
It's actually a school in Korea running an open proxy
Received: from m10.grp.snv.yahoo.com ([142.90.127.53])
        by f64.law4.hotmail.com with QMQP; Tue, 21 May 2002 10:16:08 -0400
This header is totally bogus. m10.grp.snv.yahoo.com has address 66.218.67.192, not 142.90.127.53,
which is an unused TRIUMF address. The proxy does not add header information, so the real sender is unknown (unless the
computer in Korea is examined). f64.law4.hotmail.com is irrelevant; normally you would expect to see
"received ... by yahoo.com". You would also expect to see sequential timestamps, when converted to the same timezone
Reply-To: <lastChance48Ye3gyW@yahoo.com>
Message-ID: <FAA78C9D-6DED-11D6-8DFF-00105A6A4089@JkZqiHVE>
From: <lastChance48Ye3gyW@yahoo.com>
To: <deye1@jyno.com>
Subject:               
Date: Wed, 22 May 2002 21:18:32 -0460

This mail was filtered by the "list" rule. It is legitimate mail from a Perl mailing list.
Received: via dmail-4.1(9) for +mail/list; Thu, 1 Mar 2001 12:49:27 -0800 (PST)
added by the list software I think to track delivery errors
Return-Path: <libwww-return-2184-advax=triumf.ca@perl.org>
received from the qmail program at perl.org for 
Received: from tmtowtdi.perl.org (IDENT:qmailr@tmtowtdi.perl.org [209.85.3.25])
        by trmail.triumf.ca (8.10.2/8.10.2) with SMTP id f21KnO316789
        for <>; Thu, 1 Mar 2001 12:49:25 -0800
Received: (qmail 26520 invoked by uid 508); 1 Mar 2001 20:49:18 -0000
added by mailing list software
Mailing-List: contact libwww-help@perl.org; run by ezmlm
The precedence header
Precedence: bulk
headers added by mailing list software and understood by Pine
list-help: <mailto:libwww-help@perl.org>
list-unsubscribe: <mailto:libwww-unsubscribe@perl.org>
list-post: <mailto:libwww@perl.org>
Delivered-To: mailing list libwww@perl.org
Received: (qmail 26508 invoked from network); 1 Mar 2001 20:49:16 -0000
Where the message to the list came from
Received: from mardy.hank.org (root@63.205.225.170)
  by tmtowtdi.perl.org with SMTP; 1 Mar 2001 20:49:16 -0000
Received: from whmoak (cory [192.168.0.98])
        by mardy.hank.org (8.10.1/8.10.1/HaNk 2000/05/23) with SMTP id
    f21Kn3d21474
        for <libwww@perl.org>; Thu, 1 Mar 2001 12:49:03 -0800
Message-Id: <3.0.3.32.20010301124901.02163464@pop3.hank.org>
humorous header added by the originator. "X-" headers are "experimental"
X-Windy: Is it blowing?
X-Nil: 
Date: Thu, 01 Mar 2001 12:49:01 -0800
Where the original message was sent
To: libwww@perl.org
the original sender
From: Bill Moseley <moseley@hank.org>
Subject: ctype.h: No such file
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

False Positives

Why is my legitimate mail being labelled spam ?

Either:

In order to determine which, you need to examine the mail headers. In Netscape, select View -> Headers -> All. You should see some lines starting X-Spam-List. If the line says razor, then the message was reported as spam (last two options above). If the line says something else, like list.dsbl.org, then the first option applies. In some cases, you can find examples of spam sent through the relay. Take the ip address in [] brackets from the Received: line above, and look it up in spamalyzer (fast) or RBcheck (more detailed), then query the pertinent database.

e.g. the ip address in this message is 64.136.24.135

Return-Path: <lastChance48Ye3gyW@yahoo.com>
Received: from mx13.boston.jyno.com (mx13.boston.jyno.com [64.136.24.135])
        by m6.boston.jyno.com with SMTP id AAA8Q2UJQA2PVEQJ
        for <deye1@jyno.com> (sender <lastChance48Ye3gyW@yahoo.com)>;
        Wed, 22 May 2002 21:42:38 -0400 (EST)

Preventing spam

It is difficult for someone who uses the Internet regularly to avoid receiving spam. Things to avoid: Unfortunately, most of this advice is good only for children, and having multiple email addresses can drive you crazy, so it may be easier just to delete spam (and maybe report it).
We try though to avoid publishing TRIUMF addresses in machine-readable form online. See the "email cloaking device" below for a way to hide email addresses on the Web.

Web pages with email addresses may be "poisoned" by including spamtrap addresses - such as R.T.Fishall on this page. This may eventually reduce the effectiveness of mail harvesting.

See also:

New stuff (2004):

Andrew Daviel <>

R.T.Fishall has nothing to do with this page. Do not send him mail! This address is a spamtrap.
Same goes for Nemo, Odysseus, Ulysses, Aaron, 123@triumf.ca, Aaron Aardvaark <aaardvaark@triumf.ca>, Aaron Anthony Aardvaark, 911@triumf.ca, Abracadabra, Abigail Too, Xerxes 123, 411@triumf.ca, Oolon Colluphid.