SPF- Motivations, Risks, and Alternatives
By Douglas Otis, Trend Micro Inc. 10/02/06
The purpose of this paper is primarily to evaluate security risks associated with the use of SPF authorization records. To weigh these risks against the intended benefits of SPF/Sender-ID, a brief analysis of who is publishing these records allows some deductions as to what are the motivating benefits. From the distribution of SPF publishers, the possible purposes appear aimed at white-listing bulk-senders and thwarting phishing attempts.
A pitch often made for implementing SPF/Sender-ID is to ensure delivery at various ISPs. Frequently MSN, Hotmail, and AOL are mentioned. Unfortunately, neither SPF or Sender-ID represents a comprehensive solution that avoids delivery problems, especially when recipients are forwarding their email. For example, Jon Doe might use an account at forwarding to accounts at a major ISP. Jon does this to avoid checking his accounts separately for arriving email.
A clear warning is included by the IESG within the experimental RFC4406 for Sender-ID:
Participants in the Sender-ID experiment need to be aware that the way Resent-* header fields are used will result in failure to receive legitimate email when interacting with standards-compliant systems (specifically automatic forwarders which comply with the standards by not adding Resent-* headers, and systems which comply with RFC 822 but have not yet implemented RFC 2822 Resent-* semantics). It would be inappropriate to advance Sender-ID on the standards track without resolving this interoperability problem.
This warning relates to these instructions included in RFC4406:
7.2. E-Mail Forwarders
In order to pass the PRA variant of the test, a program that forwards received mail to other addresses MUST add an appropriate header that contains an e-mail address that it is authorized to use. Such programs SHOULD use the Resent-From header for this purpose.
Not all forwarding services violate RFC2822. A method to mitigate SPF/Sender-ID induced delivery problems is to assert a neutral status in SPF records when the SMTP Client is not authorized due to these interoperability issues. In effect, these records ask that messages be treated “as-if” no records were published.
To overcome the purported coercion caused when major ISPs reject messages lacking SMTP Client authorization, these records must then authorize all SMTP Clients! The next concern might be whether authorization is considered by these domains as being sufficient for accruing reputations based upon the referencing domain-name.
One may wonder why large providers publish negative SPF records for email-accounts used by residential users. With respect to Microsoft and Sender-ID, for resolving the resulting forwarding and associated support issues, their solution requires the acquisition of their proprietary and licensed header selection algorithm. Most open source vendors still consider provisions in this license to be unacceptable. See:
http://www.apache.org/foundation/docs/sender-id-position.html
The use of negative SPF records by large providers remains problematic, and might be seen as another means to coerce adoption of Sender-ID and to thwart the distribution of open source software, while simultaneously putting the DNS infrastructure at risk. Often the pretext for the negative authorization records is to protect recipients from being phished. Does Sender-ID actually offer this protection? No. And SPF/Sender-ID increases the risks of far more serious threats that extend well beyond email.
Headers tested by SPF/Sender-ID may remain unseen. For there to be any protection, message annotations must be limited to trusted domains. Various annotation techniques may involve a browser or email client plug-in. It is common for recipients to collect messages from many different sources. To confirm the SPF records for a message being viewed by a recipient, reliance must be placed upon the the Received header (time-stamp). Unfortunately, noting the IP address of the SMTP client is optional in the Received header. While some are suggesting an authentication header may be added by the MDA, without specific knowledge of the MDA domain, there is no way for a generic client to know whether such headers can be used for this purpose.
SPF/Sender-ID results might even be based upon email-addresses not seen by the recipient. As a result, SPF/Sender-ID is not very practical for even annotating known trustworthy domains. In addition, SPF/Sender-ID will not reliably define the authorized SMTP Clients when either referenced from the EHLO, MailFrom, or Purported Responsible Addresses for various reasons. In addition, few SPF records define the intended reference or the email-address selection protocol.
When there is an SPF record published, it is typically just the version 1 record. This version lacks a means to define the intended reference, largely due to the animosity created when Sender-ID utilized SPF version 1 records. Sender-ID only recommends publishing version 2 records, together with version 1, when defining the Purported Responsible Address differently from that of the EHLO and MailFrom.
Will all the overhead involved with SPF records offer reliable benefits? See Appendix C for the example of the paypal.com SPF record which requires 20 DNS transactions to obtain both SPF and Sender-ID mechanisms, and then another 3 transactions to resolve MX addresses. When the goal is to utilize SPF status as a means to prevent spoofing, the PayPal “softfail” is problematic and may cause phished messages to appear more credible when they covertly assert an authorization for all SMTP Clients, as possible when using the SPF “exists” macro, for example.
In addition to not offering a reliable means to annotate messages, SPF/Sender-ID does not resolve look-alike and “cousin” domain risks which remain a serious problem. A list of such domains have been compiled by John Levine:
MYPAYPAL.COM
PAYPALSHOPS.COM
PAYPALSTORES.COM
PAYPALCARDSERVICES.COM
PAYPALMC.COM
PAYPALVISA.COM
PAYPALVERIFY.COM
SECURITY-PAYPAL.COM
PAYPALSTORE.COM
PAYPALSYS.COM
PAYPALMAIL.COM
SECURITIES-PAYPAL.COM
PAYPAL-EBAY.COM
PAYPALESCROW.COM
SECUREDPAYPAL.COM
PAYPALSERVICE.COM
WWW--PAYPAL.COM
PAYPALESHOP.COM
PAYPAL-SIGN-IN.COM
PAYPAL-VERIFYD.COM
PAYPALEMAIL.COM
PAYPAL-DATABASE.COM
PAYPAL-REDIRECT.COM
PAYPAL-BILLING.COM
PAYPALCOM.COM
PAYPALINC.COM
PAYPAL-MEMBERS.COM
PAYPAL-MEMBER.COM
WWWWPAYPAL.COM
PAYPAL-VERIFIC.COM
PAYPALMONITOR.COM
SECURITY-MEASURES-PAYPAL.COM
PAYPAL-SECURE-UPDATES.COM
PAYPALPURCHASE.COM
RELOGIN-PAYPAL.COM
WWWPAYPALL.COM
PAYPAL-VERIFICATION-SYSTEM.COM
PAYPALCODE.COM
PAYPAL-SECURED-UPDATE.COM
PAYPAL-SERVICE.COM
PAYPAL-SECURE-INFO.COM
PAYPAL-VERIFICATION-MEMBERS.COM
CUSTOMER-PAYPAL.COM
USERS-PAYPAL.COM
CGI5-PAYPAL-VERIFYUSER.COM
PAYPAL-DBS.COM
ACCOUNTS-PAYPAL.COM
VERIFY-PAYPAL-ACCOUNT.COM
BILLINGUPDATEPAYPAL.COM
SSLSERVER-PAYPAL.COM
PAYPAL-HOME.COM
INFOUPDATE-PAYPAL.COM
PAYPAL-COM.COM
PAYPALSECURITY.COM
UPDATED-PAYPAL.COM
CLIENTS-PAYPAL.COM
REACTIVATE-PAYPAL.COM
SECURIZED-CGI-PAYPAL.COM
PAYPALSUPPORT.COM
EBAYPAYPAL.COM
PAYPALUPD.COM
PAYPAL-UPDATE-VERIFY.COM
WWW-PAYPAL.COM
EXPIRE-PAYPAL.COM
RE-PAYPAL.COM
WWW-PAYPAL-COM-LOGIN-CGI-WEBSCR-CMD-LOGIN-RUN.COM
PAYPAL-CGI.COM
SECURE-PAYPAL-LOGIN.COM
UPDATE-YOUR-PAYPAL-ACCOUNT.COM
PAYPAL-INFORMATION-UPDATE.COM
SUSPENDED-PAYPAL.COM
PAYPAL-UPDATE-INFO.COM
UPDATE-PAYPAL-INFORMATION.COM
PAYPAL-UPDATE-INFORMATION.COM
UPDATE-YOUR-PAYPAL-ACCOUNT.COM
UPDATE-YOUR-PAYPAL.COM
PAYPAL-ACCOUNT-UPDATE.COM
PAYPAL-CONTROL.COM
PAYPAL-UPDATE-YOUR-INFO.COM
PAYPALINSURANCE.COM
USERS-UPDATES-PAYPAL.COM
SERCURE-PAYPAL.COM
PAYPAL-INFOUP.COM
SAFE-PAYPAL.COM
PAYPALREPORT.COM
VERIFYBILLING-PAYPAL.COM
PAYPALACCOUNTSERVICE.COM
PAYPAL-CREDITCARD.COM
VERIFYBILLING-PAYPAL.COM
PAYPALACCOUNTSERVICE.COM
SUPPORTS-PAYPAL.COM
PAYPALBILLING.COM
PAYPALACCOUNTSERVICES.COM
PAYPALCUSTOMERCARE.COM
US-PAYPAL.COM
PAYPAL-UPGRADE.COM
PAYPAL-SSL.COM
PAYPAL-EUROPE.COM
ALERTS-PAYPAL.COM
PAYPAL-ONLINEINFO.COM
CGI-PAYPAL-UPDATE.COM
SSL-PAYPAL.COM
PAYPAL-CONFIRMINFO.COM
PAYPALNEWUPDATE.COM
PAYPAL-CGI-UPDATE.COM
PAYPAL-CGI-BIN.COM
RESTORE-PAYPAL-ACCOUNTS.COM
CUSTOMER-UPDATE-PAYPAL.COM
PAYPAL1.COM
PAYPAL-CUSTOMER-CENTER.COM
PAYPAL-ACCOUNT-UPDATES.COM
PAYPAL-SM.COM
SERVICE-ACCOUNT-PAYPAL.COM
PAYPAL-ONLINECONFIRM.COM
PAYPAL-INFOCONFIRM.COM
PAYPAL-VALIDATEINFO.COM
PAYPAL-INFO-UPDATE.COM
INFO-PAYPAL-UPDATE.COM
CGI-UPDATE-PAYPAL-ACCOUNT.COM
PAYPAL-SECUREINFOS.COM
PAYPAL-CGI-BIN-SECURE.COM
SECURE-PAYPAL-CGI-URL.COM
PAYPALAUSTRALIA.COM
PAYPAL-INTL-SERVICE.COM
ACCOUNTUPDATE-PAYPAL.COM
PAYPALNEWUPDATES.COM
EMAIL-PAYPAL.COM
PAYPAL-ORGANIZATION.COM
PAYPAL-CUSTOMER-CENTRE.COM
EBAY-AUCTION-PAYPAL.COM
EMAILPENDING-PAYPAL.COM
PAYPALSERVIICE.COM
SSLCONNECTION-PAYPAL.COM
E-PAYPAL.COM
PAYPAL-ACOUNT-CENTER.COM
PAYPAL-UPDATE-CENTER.COM
PAYPALCUSTOMERSERVICES.COM
PAYPALSECUREDPAYMENT.COM
CGI-INFO-PAYPAL.COM
PAYPAL-CLIENT-SUPPORT.COM
CONFIRM-PAYPAL-ACCOUNTS.COM
PAYPAL-INFOVALIDATE.COM
RESTORE-PAYPAL-INFO.COM
CS-PAYPAL.COM
IMG-PAYPAL.COM
PAYPAL-COM-CGI-BIN-CONFIRMATION-PP545454.COM
SIGNIN-ACCOUNT-PAYPAL.COM
PAYPAL-MEMBERS-VALIDATION.COM
PAYPAL-SECURE-UPDATE.COM
PAYPAL-ACCOUNT-SERVICE.COM
UNLOCK-PAYPAL.COM
EMAILS-PAYPAL.COM
PAYPAL-SUBMIT.COM
PAYPAL-SECURITY-SYSTEM.COM
UPDATE-PAYPAL-SYSTEM.COM
CONFIRM-PAYPAL.COM
PAYPAL-SETUP.COM
PAYPAL-SECURITY-UPDATES.COM
ONLINE-PAYPAL-UPDATE.COM
US-PAYPAL-SUPPORT.COM
PAYPAL-SUPPORT-SSL.COM
PAYPAL-PAYMENTS.COM
PAYPAL-SERVERS.COM
PAYPAL3.COM
PAYPAL-LOGIN.COM
VERIFY-PAYPAL-US.COM
VERY-PAYPAL.COM
RESUBMIT-PAYPAL.COM
UPDATEUSER-PAYPAL.COM
One of these domains is valid. Can you spot the real one from the fakes? Since many recipients only view the “display-name”, blocking exact spoofs (which the paypal SPF record does not do) will still provide little protection. Adding annotation that indicates a “pass” SPF result will seriously risk misleading recipients and actually increase their risks. For example, in the case of Sender-ID, the From header field may contain an exact spoof of a phished domain and still obtain a “pass” result.
SPF Adoption
So where does SPF adoption stand, and is there reason to think the risks posed by SPF are increasing? SPF records are published in about 3% of the domain that have an MX record according to a study published by Andrew Newton, one of the chairs of the now closed IETF MARID working group (http://hxr.us/blojsom/blog/grumpops/). In comparison, Trend Micro evaluated a small sample of messages from known abusive sources. The results of the referenced SPF records are as follows:
none/invalid/error 130557 # 77%
pass 348 + 0.2%
neutral 24207 ? 14%
softfail 9643 ~ 6%
fail 4417 - 2.6%
TOTAL 169172
For a better understanding of who is publishing SPF records and perhaps why, a sampling from a group of Fortune 100 and top 20 volume domains were reviewed. While this group has adopted the use of SPF to a greater amount than the typical domain, the distribution of domains tends to suggest some possible motivations.
Percentage distribution of Fortune 100 SPF results:
none/invalid 58%
neutral 9%
softfail 22%
fail 11%.
Of those Fortune 100 domains with above average traffic, the results are:
none/invalid 73%
neutral 6%
softfail 13%
fail 9%,
which is also fairly consistent with the top 20 high volume domains:
none/invalid 70%
neutral 10%
softfail 10%
fail 10%.
There seems to be a disproportionate number of neutral results for known abusive sources compared to the percentages found within the group of Fortune 100 or the top 20 high volume domains. One explanation for a higher number of neutral results of abusive sources could be due to many large providers publishing SPF records that request the neutral handling of undefined sources. In other words, these records request that messages from a source not defined in the SPF record be handled in the same fashion as when no SPF record is published.
Providers, such as AOL, Bell South, Gmail, and Verizon, utilize neutral SPF records. Some providers even request the positive handling of messages when the SMTP client is not authorized, such as Bell Canada. A request for positive handling might indicate that neutral records are not always treated “as-if” no SPF record exists when the source is undefined, or that lacking an SPF record is not always acceptable. Fortunately, providers such as AT&T et al, Cablevision, Comcast, Cox, Earthlink, PeoplePC, Mindspring, and Yahoo!, do not offer any SPF records which should allay some concerns about not publishing.
To summarize, about 10% of the Fortune 100 and top 20 volume domains publish SPF records intended to block messages coming from undefined sources. One of the notable high volume domains is charter.com which uses a strict SPF record that asserts “fail” rather than “softfail” for residential emails. This company uses Hotmail for outbound MTAs and is owned by Paul Gardner Allen, a co-founder of Microsoft.
Charter.com's use of negative SPF records may be seen as a means for inducing the use of Sender-ID in the face of the resulting (and often expensive) support calls. However, Charter Communications (charter.com) may be having difficulties of their own. Either charter.com customers are willing to have their outbound messages blocked when a recipient is using a forwarding account, make irresolvable support calls complaining about the problem, or they find their email-services elsewhere. The other high volume domain asserting a “fail”, unlike charter.com, does not appear to have these SPF records referenced from email-addresses. There is only a single MTA for this high-volume domain which likely ensures this domain is not being used to receive the high volume of email traffic.
Still, when “soft-fail” and “fail” categories are combined, SPF records only differentiate about 20% of domains when a message is from an undefined source. However, combining these categories is not how these results were intended to be used. The expectation was that a “softfail” only lowered the ratings of a message. Of the known abusive sources, combining these two categories still only blocks about 9% of the messages. When rejection is done properly, i.e. for only those sources that result in a “fail”, less than 3% of the messages are being blocked.
SPF does not appear to be at any tipping point. There also appears to be valid reasons why providers will continue to not publish these records. From our experience, often the CIDRs in these records encompass too many addresses. The lack of granularity makes these records less practical for dealing with abuse related issues. SPF records also create problems when a receiving domain applies more stringent handling than what the SPF record requested. Even requests for neutral handling may still result in strict handling being applied. As a result, message are blocked and support desks are being called.
It might be safe to just let SPF fade away, provided that SPF actually does fall out of favor. Many of the problems which SPF have been unable to resolve are successfully handled by DKIM. DKIM overcomes forwarding issues and does not depend upon any headers added by the MDA for authentication. When the goal is to apply annotations for trustworthy domains, DKIM far better fulfills the role of authenticating the source of the message.
DKIM does not need to stop at just authenticating the source. Names can be associated using a simple MailFrom policy record. For example:
<base32(sha-1(signing-domain))>._DKIM-M.<mailfrom domain> TXT “d=signing-domain”
See: http://www.ietf.org/internet-drafts/draft-otis-dkim-dosp-01.txt
This DKIM Originator's Signing Policy (DOSP) record can indicate which signing-domains validate the MailFrom email-address prior to a bounce being issued, for example. Such a scheme would allow any number of signing-domains to be associated with the MailFrom header and still need only a single DNS transaction. This method would not suffer from the extreme amplification and forwarding concerns created by SPF address path registration schemes. Using this strategy only requires a single DNS transaction prior to safely bouncing a message.
If Microsoft wishes to continue the use of the PRA algorithm, the DOSP draft defines how all the Purported Responsible Address originating domains can be defined by using this scheme. DOSP in conjunction with DKIM can even validate the use of the EHLO. DKIM and DOSP provides a safe means for validating various email message headers while still avoiding much of the risks caused by a need to compile a comprehensive list of IP addresses of all possible SMTP Clients as required by SPF.
The Future of SPF
Should MAAWG promote or demote the use of SPF? This document describes in the text below an email induced Denial of Service threat caused by SPF scripts. These scripts are used to evaluate the association of a source domain-name with the sending-system. SPF scripts attempt to establish a domain-name association through the construction of an extensive IP address list of all the sending-systems. Expectations of an association have become problematic. Message handling might be negatively affected without an apparent domain-name relationship discovered between the sending-system and either the message envelope or the message itself.