Last Updated Fri Jun 25 18:35:02 2010

[problem] [history]

Problems with Hotmail losing email

The Problem

Hotmail loses some incoming mail. Deliberately. Their servers accept it (SMTP 200 "success" response) but then fail to deliver it to user accounts.

History

In November 2009, TRIUMF was hit by a phishing attack from "University Web Centre" which persuaded several users (less than 1%) to send passwords to a site in Israel. Phishers logged in to accounts on TRIUMF webmail (SquirrelMail) and sent a number of spam/phish messages. The attackers modified the user's mail signatures to contain the message body, achieving a significant bandwidth amplification betwen incoming and outgoing mail.
Accounts were cleaned up, passwords changed and users told to be more careful. Again (4th time that year). A SquirrelMail plugin was installed to properly log webmail access and to log mass mailing, in order to benchmark legitimate mailouts.

In February 2010, attackers using an iterative method found a pair of accounts where a user had set the password=userid in defiance of policy. They used the same amplification technique to send a substantial amount of mail, starting in the late evening. Many of these messages used a "live.com" return address, which would cause rejection messages to go to Hotmail.

In March 2010, a couple of people again sent passwords to a phisher.
Another SquirrelMail plugin was installed to block outbound mass mailing, using statistics gathered from benchmarking to set a realistic limit. On March 19, this plugin successfully blocked an attempt to use another compromised account. The last successful phishing mailout was on March 12.

Hotmail response

During the mail attack in February, some messages were rejected (SMTP status 5xx permanent failure) by Hotmail servers, e.g.
550 SC-001 Mail rejected by Windows Live Hotmail for policy reasons. Reasons for rejection may be related to content with spam-like
characteristics or IP/domain reputation problems. If you are not an email/network admin please contact your E-mail/Internet Service Provider for help. Email/network
admins, please visit http://postmaster.live.com for email delivery information and support
Checking the above URL, after creating an account, showed some statistics for rejected mail. Later, mail appeared to be working normally (being accepted by hotmail servers). The postmaster.live.com tracker showed no rejected mail.

On March 24, a TRIUMF user reported that mail sent to a Hotmail account was never received. Our logs showed that it had been accepted by the Hotmail server. Creation of a test account at Hotmail confirmed that mail from our webmail server was being deleted (not delivered to a junk folder), with the possible exception of mail sent in direct reply to mail from Hotmail account. Mail from other TRIUMF servers was delivered normally.

We contacted Hotmail support on March 24. As a pre-requisite to helping us, they required us to join the "Junk Mail Reporting Partner Program" - essentially a scheme whereby they actually send complaints to our abuse account (per RFC 2142, listed in whois.arin.net and whois.abuse.net) instead of discarding them. They also made suggestions that would allow a large bulk-mailing company to successfuly target Hotmail, such as maintaining separate outbound servers for different purposes and creating good SPF and Sender-ID records.

Following the creation of a JMRPP account in April, complaints started to trickle in - from November 2009 and February 2010. Our mail to Hotmail was still being silently discarded, so we blocked it on our server - our users would at least receive a delivery failure notice.

On April 14, Hotmail indicated that they would remove an entry from their reputation list "in 24-48 hours". On April 22, mail was still being discarded.

On April 26, Hotmail indicated that they had "taken steps to implement a temporary mitigation to our mail delivery problem". At this point, test messages were being successfully delivered to my test account at Hotmail. I removed the outgoing mail blocks, and since then have had no complaints.

Since successfully signing up for the JMRPP program, I had as stated stale complaints. It appears that we get them whenever someone clicks "this is junk", regardless of when it was sent. We had one complaint about a mailman message (opt-in list, with unsubsciption instructions at the bottom), but otherwise it has been quiet.


[history] [response]
A.Daviel