Rationale Document Tue Jun 1 10:22:27 EDT 2004 ---------------------------------------------------------------------- OVERALL FRAMEWORK ---------------------------------------------------------------------- SPF fits into a larger framework. That framework has these parts: 1) authentication 2) accreditation 3) reputation Authentication tries to answer the question: "How confident can a receiver be that an email message really is from who it claims to be from?" It counters forgery and fraud. Accreditation lets third parties vouch for senders with whom they have a prior relationship. Reputation systems help receivers decide the disposition of incoming messages. Reputation systems rate senders and their accreditors. Receivers may develop local reputation systems, employ third party reputation services, trust certain accreditors directly, or all of the above. The absence of prior trust between arbitrary senders and receivers implies a correspondingly skeptical relationship between accreditation and reputation services. This framework resonates with real-world models of human interaction. For example, reputation services exist today in the form of credit bureaus and movie reviews. Accreditation services exist today in the form of passports and university diplomas. SPF directly enables authentication and accreditation, and indirectly supports reputation technologies. ---------------------------------------------------------------------- TWO CULTURES OF AUTHENTICATION ---------------------------------------------------------------------- SPF provides syntax and semantics that help answer the authentication question: "How confident can a receiver be that an email message really is from who it claims to be from?" But there are at least two ways to ask and answer the question. In a designated sender scheme, the question becomes: "is the SMTP client of an incoming message authorized by the purported sender of that message to send the message?" The primary identity is the sender of the message --- the (administratively independent) entity responsible for the most recent injection of the message into the mailstream. Designated sender schemes consider the channels through which a message is transported. They do not consider the content of the message. In a store-and-forward model of email, designated sender schemes apply to hops between administratively independent domains. In a cryptographic scheme, the question becomes: "do the credentials embedded in the message validate its purported authorship, according to a public-key cryptosystem?" The primary identity is the author of the message --- the entity which created the content. Cryptographic schemes consider the content of a message. They do not consider the means by which it was delivered. Cryptographic schemes are congruent with an end-to-end model of email. Note that designated sender schemes identify senders and cryptographic schemes identify authors. SPF defines mechanisms that implement a designated sender scheme. SPF can also be extended to describe cryptographic schemes. ---------------------------------------------------------------------- ARCHITECTURAL CONSIDERATIONS ---------------------------------------------------------------------- The greatest virtue of Internet email is openness. Open systems suffer the problem of abuse. SPF tries to curtail abuse as much as possible while encroaching upon openness as little as possible. It does this by aiming for mechanism, not policy. A number of competing requirements inform the design. On the one hand, position A is valid; on the other, so is position B. - Zombies versus hobbyists. - Spamfolders versus SMTP rejects. - Rejecting before DATA vs rejecting after DATA. - 100% backward compatibility versus an unacceptable status quo. - Nipping the innocent in the bud. - Inconveniencing nonparticipants. - The perfect is the enemy of the good. - There is no such thing as an interim standard. - This is a job for sysadmins. - DNS: Caching versus parsing. - DNS: speed vs expressiveness. - DNS: ease of writing vs ease of reading. - DNS: use of XML vs development of a little language. Zombies versus hobbyists: A - A majority of spam today originates from end-user machines infected by viruses that send spam directly to receivers' MX servers. Such zombies should no longer be able to send direct-to-mx spam. B - Linux hobbyists should still be able to send mail from their consumer-grade broadband connections. They may have no control over the PTR records of their assigned IP addresses, but the concept of first-class and second-class citizenship rankles many idealists who believe in an unfettered end-to-end model of pure IP connectivity. Spamfolders versus SMTP rejects: A - Rejecting messages at SMTP time is too abrupt. Filing to a spamfolder is better; that way, recipients have a chance to pull false positives back from the brink without embarrassment. B - Filing to a spamfolder often means nobody sees a message, and false positives silently disappear. If a sender receives no notification that a message was lost, the integrity of the email system suffers. Rejecting before DATA vs rejecting after DATA. A - If the protocol makes spoofs rejectable before DATA, we save bandwidth. We can even honour per-user policies. B - If the protocol rejects spoofs only after the message DATA has been received, we have the opportunity to review the message content in greater detail. 100% backward compatibility versus an unacceptable status quo: A - the Old Email is broken in many ways, and spam will continue to be a problem until it is fixed. B - the New Email should be 100% backward compatible with the Old Email, and nobody should have to change anything for the New Email to work. Nipping the innocent in the bud: A - Spam should be stopped before it starts; to limit the cost of a spam attack, receivers should by default adopt a skeptical stance to any unknown input. In big cities, people don't talk to strangers. B - In small towns, people smile at strangers. Senders should be assumed innocent until proven guilty. Any person should be able to get onto the Internet, register a domain, and immediately email a hundred of his closest friends. Inconveniencing nonparticipants: A - The designated sender model works correctly for the vast majority of all mail, which is sent directly from sender to recipient. In its strong form, SPF permits rejection of unwanted mail before the DATA command; this represents a significant bandwidth savings. Under the designated sender model, the strategy that minimizes the global amount of work requires nonconformant sites (primarily, forwarders and web-generated emailers) to work toward conformance (ie. prepending headers and adding SUBMITTER support). When Spock says "the needs of the many outweigh the needs of the few", he refers to the principle of eminent domain. B - Forwarders and web-generated emailers are merely operating according to time-honoured traditions. SPF changes the terms of their unwritten contract; it is unfair to demand that they change. Placing the needs of the many above the needs of the few can lead to a tyranny of the majority. Spock would have been considerably less heroic if he had used that argument to volunteer somebody else. Cryptographic systems, where credentials are embedded in the message data, do not inconvenience forwarders, but they are unrejectable before the DATA command. The perfect is the enemy of the good: A - people who are losing money every day due to spam want the New Email to be done quickly. B - people who worry about the overall architectural integrity of the Internet want the New Email to be done right, even if that takes longer. There is no such thing as an interim standard: A - it is important to experiment with new protocols on a large scale before officially blessing anything. B - there is no such thing as an "interim" standard: anything that rolls out at all will be there forever. Perhaps this conflict can be solved by moving to a "controlled burn" model of change management on the Internet. This is a job for sysadmins: A - the New Email should be fully transparent to the end-user and require no reconfiguration. B - no disagreement! DNS: Caching versus parsing. A - To reduce the burden on clients, records should be easily cached. Records that describe the full set of designated senders up front are easily cached and require no subsequent lookups. However, such records require parsing. B - To reduce the burden on clients, records should require minimal parsing. Records that answer "yes/no" for a given query aganst a domain/IP tuple can be easily parsed using existing DNSBL logic. However, such records are not easily cached, particularly when a wide range of IP addresses produce multiple negative results. DNS: speed vs expressiveness. A - The protocol should minimize the number of DNS queries needed, particularly serial queries. This reduces start-to-finish time. B - The protocol should be rich and expressive, even if that means multiple lookups are needed to resolve an answer. This increases start-to-finish time. DNS: ease of writing vs ease of reading A - it should be easy for senders to publish records. Maybe senders should describe their designated sender policies in a high-level form and leave correct interpretation up to the receivers. If they change an MX server's IP address, SPF should be smart enough to do the right thing automatically. The whole point of the DNS is to map symbolic names to lower-level data. Unnecessary redundancy is undesirable. B - It should be easy for receivers to query records. Maybe senders should be expected to precompile designated sender information into a canonical form. Every time an MX server changes its IP address, it's the sender's job to remember to republish the designated sender data. DNS: use of XML vs development of a little language A - The syntax should be easy for receivers to implement. The adoption of existing data formats such as XML means that existing libraries can be used to parse the data. XML is popular among the Microsoft community. B - The syntax should be easy for receivers to implement. The use of a custom-made little language means that heavyweight XML libraries are not required. XML is generally unpopular among the open community. ---------------------------------------------------------------------- Design Goals ---------------------------------------------------------------------- This section discusses ways to measure the success of the project. It also lays out the design philosophy which guides decision-making. * Market-based metrics. SPF is not the only specification to take aim at the problem of authentication. Other designs reflect different goals. Some designs are within the LMAP family (eg. DMP, RMX). Other designs are not (Call-Back Verification, Challenge/Response). With a variety of designs on the market, the success of this effort can be measured by relative market share; how many domains publish SPF records, and how many receivers check them? * The Fax Effect The value to a domain of adopting the standard depends, in part, on the number of other domains who have already adopted the standard. This is commonly called the fax effect: the first two people to buy a fax machine could only use it to communicate with each other, but every subseqent adopter could use it to communicate with everybody else who already had one. Today, faxes are everywhere. SPF represents a major shift in the paradigm of email. For the fax effect to fully manifest, a significant majority of legitimate email on the Internet should be covered by SPF. The design goals follow logically. * Goal 1: Fast widespread adoption. To get the fax effect working for us, we want to make it possible for people to quickly and easily adopt the standard. On the publishing end, this means that the syntax and semantics must be easily learned by busy people whose level of DNS technological savvy we could not take for granted. Alternatively, a wizard can help them set up records which they can treat as opaque. On the receiving end, this means that the standard should be reasonably straightforward to implement for the most popular MTAs. There are many more sender domains than receiving MTA implementors. Therefore, the design should optimize for ease of publishing over ease of implementation. * Goal 2: Minimizing friction. The smallest amount of friction will impede adoption. The design should take advantage of existing capabilities wherever possible. The design should optimize for the path of least resistance. The less overhead needed to publish a record, the better. The design guidelines that result from the above goals are: * Guideline 1: optimize for ease of publishing over ease of implementation. * Guideline 2: follow the path of least resistance. * Overall criteria for success. The ultimate measure of success is whether the standard works as intended: - how well does authentication work to suppress nonconformant mail and pass legitimate mail? - how well do accreditation and reputation systems work together to determine legitimacy? - has the standard contributed to a reduction in spam for participating receivers? - has the standard reduced forgery and fraud for participating publishers? - has the standard reduced costs and brought benefits elsewhere in the ecosystem? ---------------------------------------------------------------------- Life Cycle Design ---------------------------------------------------------------------- The designed protocol is a thing that does not stand in isolation, but moves through a life cycle; to be successful, it must be pass both economic and marketing muster at each stage in its life. Cautious people will want to transition gradually toward full acceptance of SPF. Therefore SPF cannot be designed with only on/off states; it should have a transitional adoption strategy. Overspecifying the protocol leads to untimely ossification. Underspecifying the protocol leads to a lack of expressiveness. The solution to both overspecification and underspecification is to ensure extensibility. Extensibility should be well defined on both syntactic and semantic levels. A well-designed protocol often finds itself being legitimately used in ways the original designers did not anticipate. Unexpected uses validate the expression model. To date, a number of unexpected uses have arisen: for example, use of the "exists" mechanism to perform crude logging. ---------------------------------------------------------------------- Choice: Identity ---------------------------------------------------------------------- Authentication schemes can focus on a number of identities. The simplest schemes ask if the IP address of the SMTP client should be permitted to send mail or not. More complex designated sender schemes add a domain-name dimension. They ask if the IP address is permitted to send mail *for that domain*. * Am I MTA or Not? The simplest scheme asks a network owner if a given IP address is allowed to send mail to the public Internet. Instances of this scheme include: - .mxout. - MTAMark - Selective Sender Such schemes generally store the permission information in the PTR tree, alongside it, or in a parallel domain tree. These schemes make the "zombies versus hobbyists" tradeoff in favour of blocking zombies. If we assume that a hobbyist has no control over their PTR record, and will have no control over their MTAMark/SS/.mxout. record either, these schemes disenfranchise hobbyists. In accordance with its "mechanism, not policy" design philosophy, SPF makes the "zombies versus hobbyists" tradeoff in favour of hobbyists who are assumed to control the forward DNS of a domain they own. The important thing is to establish accountability: if a spammer wishes to designate a zombie as a permitted sender, the message is SPF conformant; it is the responsibility of the reputation system to reject the message. * Am I an MTA for the HELO domain? The next simplest scheme asks the domain of the HELO command if the IP address is a permitted sender for that domain. - DRIP - CSV If we weren't going to show the result of an SPF evaluation to the end-user, crossing the IP with the HELO would be perfectly sensible. Reputation services would begin to evaluate the relays as identified in the HELO domain. * Am I an MTA for the MAIL FROM domain? Schemes that ask this question include: - DMP - RMX - FSV - DVP - SPF DMP allows the HELO domain to override the MAIL FROM. SPF uses the HELO domain only when the MAIL FROM is null. The advantage of working with MAIL FROM is that it tends to be better defined than HELO. In the absence of a SUBMITTER parameter, it also deters joe-jobs. Using the MAIL FROM address is considered useful for fighting joe-job bounce blowback, but it has the disadvantage of requiring forwarders to use some sort of return-path rewriting scheme and thereby become transport remailers. Paul Vixie alluded to this in his 2002 paper. * Am I an MTA for the responsible sender according to the 2822 headers? Microsoft wants to use Designated Sender techniques to validate sender information seen in the MUA. If conforming mailers prepend an appropriate header to indicate a relay hop, MUAs can extract a Purported Responsible Address (PRA) from the headers and display that information. The SUBMITTER parameter to MAIL FROM is a preview of the Purported Responsible Address from the 2822 headers. If present, it overrides MAIL FROM. Using 2822 sender information is considered useful for fighting phishing. However, it requires that MUAs display both author and sender information. Outlook already does this, with "from X on behalf of Y". Other MUAs who wanted to use the benefits of 2822 SPF authentication would have to also display "from X via Y" information. If a message was forwarded through multiple hops, it may be necessary to display "from X via Y via Z". In the simple legitimate case, the Purported Responsible Address would be the "From" header, and there would be no "Sender" or "Resent-From" type headers. MUAs might be able to mark mail from certain domains with a "trusted" hint. In a phishing attempt, spoofs would lack the "trusted" hint. The "trusted" hint would also be missing in a message sent through a forwarding system, but presumably the end-user would know what messages were forwarded, and adjust their expectations accordingly. * Why use SUBMITTER instead of the HELO name? In both cases the relay is the subject of reputation. However, the HELO is not displayed to the end-user; in Outlook, the purported responsible address may be, in the form "from X via Y". The "X", indicating authorship in the "From:" header, would remain unverified by designated sender means; only the sender, the "Y", would be verified by SPF. To verify authorship, cryptographic techniques are more appropriate. Promoting the Purported Responsible Address into the MAIL command provides two benefits: it makes it possible to reject spoofs before DATA, and it potentially makes adjustment easier for forwarders. However, it takes away the protection from joe-jobs, as MAIL FROM becomes spoofable again when SUBMITTER is present. ---------------------------------------------------------------------- Choice: Per User Lookups ---------------------------------------------------------------------- Doing per-user lookups reduces cacheability but increases granularity. Per-user lookups can be supported all of the time, some of the time, or none of the time. SPF chooses to support per-user lookups some of the time using the "exists" mechanism, with the assumption that only low volume senders will use this feature; therefore the burden on DNS is believed to be acceptable. ---------------------------------------------------------------------- Choice: Response Codes ---------------------------------------------------------------------- The simplest scheme has only two response codes: good or bad. More complex schemes expand the set of responses to good, bad, and unknown. In response to criticism from Steve Bellovin, SPF provides a range of seven response codes. "Error" and "Unknown" correspond to temporary and permanent failures. "Neutral" indicates the queried omain explicitly wishes to pretend it does not publish SPF records. "None" indicates the queried domain does not, in fact, publish SPF records. "Pass" means the sender accepts responsibility for the message. "Fail" means the sender explicitly requests the receiver to reject the message transmission. "Softfail" indicates a transitional state: while not as strong as "fail", it means a receiver should treat it with skepticism. This has significance for spam scoring systems. The adoption path is greatly smoothed by the presence of "softfail" as a standard result code. It was removed for a few months during the evolution of SPF, but was later put back by popular demand. ---------------------------------------------------------------------- Choice: Block or Factored ---------------------------------------------------------------------- In a purely block-style protocol, published records completely describe the sets of designated senders. Receivers are expected to figure everything out based on those records. Receivers only need to fetch information from senders the first time a query occurs; subsequent fetches can be retrieved from the DNS resolver cache or from a nearer application cache. In a purely factored-style protocol, receivers include pertinent information in the DNS query; sender servers are expected to return a response based on those inputs. While the parsing burden tends to be lower with factored-style protocols, receivers need to perform a DNS lookup for each new IP client for that domain. Large receiving sites tend to prefer the block style because it avoids repeated lookups; for efficiency they can also transform the block records into an internal representation that is locally cached and distributed. Specifications which use purely factored-style records are not transformable. They include MTAMark, SS, DVP, and DMP. Specifications which use purely block-style records are fully transformable. They include Caller-ID and RMX. FSV offers both block and factored record styles. SPF is largely a block-style protocol but provides for factored lookups using the "exists" mechanism. SPF is therefore partly transformable. Publishing domains which refrain from using "ptr", "exists", and per-user macros can be cached. Most high-volume senders are expected to publish data in IP-only notation as a courtesy to receivers. Noncacheable mechanisms are therefore expected to be used only by smaller sites. This is considered an acceptable tradeoff. ---------------------------------------------------------------------- Choice: Set-theoretic or Procedural ---------------------------------------------------------------------- Block-style declarations can be further expressed in two ways: set-theoretic or procedural. In a set-theoretic scheme, publishers explicitly partition the input space into result codes. Receivers locate the input tuple in that space and return the result. In a procedural scheme, publishers provide an ordered list of mechanisms which are evaluated sequentially. The first mechanism to match determines the result. With the exception of the "exists" mechanism, the procedural scheme can be mapped to a set-theoretic representation. The two schemes are therefore isomorphic except when "exists" is used. ---------------------------------------------------------------------- Choice: Record Type ---------------------------------------------------------------------- Several alternatives are possible: * new, custom RR type. * TXT record. * record with underscore subdomain. * a note on wildcard support. The record type is subject to discussion in the MARID working group. ---------------------------------------------------------------------- Choice: Record Syntax ---------------------------------------------------------------------- Two alternatives include: * Ad-hoc SPF notation * XML Receiver systems should not have trouble supporting both using a "two parsers, one interpreter" model. The ad-hoc SPF notation has been specified and field-tested, and is not expected to change. The XML representation is presented in the marid-core draft and is subject to discussion in the MARID working group. ---------------------------------------------------------------------- BUSINESS ANALYSES ---------------------------------------------------------------------- * Economic Analysis: Market failure and collective action To successfully deploy, the New Email requires significant resource allocation: at the minimum, it needs implementation labour, interoperability testing, and a big, expensive, PR campaign. Given the amount of work that needs to be done, a single organization is unlikely to create a pure public good at its own expense. On the multinational Internet, this is as true when the organization is a corporation as when the organization is a state. The technical name for this problem is "market failure", though it encompasses the absence of government provision as well. Intellectual property rights over the developed standard are one way out of the market failure problem. However, the public's distaste for kingmaking reduces the chances of successful adoption of any licensed technology. Market failure in this case is solved by collective action by major ISPs. A mechanism that reduces the existing cost of imperfect spam filtering will be adopted and evangelized by ISPs who bear that cost. * Marketing Analysis: Chasm Theory The designed product, and the augmentations that surround it, must survive each phase of Geoffrey Moore's Chasm model. The core specification should be attractive enough to the visionaries to seed the fax effect by publishing records and writing libraries. After an initial round of discovery and experimentation by the early adopters, the mainstream must be encouraged to start publishing records. This is the first chasm, on the publishing side. They need the assurance that publishing records will, first, do no harm. The beachhead for crossing the publishing-side chasm turned out to be industry leaders. When enough well known domains published records, they seeded the second round of the fax effect. The second round turns potential receiver-side implementors into actual implementors: the early adopters here are the makers of antispam email appliances and edge MTAs such as CipherTrust, IronPort, Postini, and Brightmail. They are ideally positioned to implement SPF checking on inbound mail and have a business reason to do so. The second chasm, on the receiving side, requires major ISPs to start implementing inbound checking. Their initial motivation is to reduce the costs of whitelisting. Major ISPs negotiate email peering relationships with major senders and maintain, often by hand, whitelists of IP addresses. Conversion to SPF-based whitelisting is an obvious step. Note that while SPF is often viewed as an anti-forgery mechanism, using it as a whitelisting mechanism is simply the other side of the coin. After these chasms are crossed, there remains the challenge of getting publishers to transition through increasing levels of severity: from "?" to "~" to "-". This effort requires cooperation from receivers. * The need for coordinated deployment. The chicken-and-egg problem is this: until forwarders are SPF compliant, publishing domains will be reluctant to advance their defaults from "neutral" to "softfail" to "fail". Until publishing domains have a default of "fail", forwarders may see no reason to comply. This problem can be overcome if industry can collectively agree on a deployment schedule that gives all parties enough notice to make the necessary changes.