Receivers take advantage of "Store and Forward", and so should senders

Recently I shed some light on what happens behind the scenes when an email message is sent on its way. The "store and forward" model is not a perfect guarantee of delivery, but is very robust in the face of temporary breakdowns. You may have noticed, however, that if a message has to be passed along through several "hops", only the final "hop" can determine exactly when or even whether the message has reached its intended recipient. To deal with this, the Simple Mail Transfer Protocol (SMTP), the standard way to transmit email over the Internet, defines all the elements necessary to complete a successful transfer of email and to report either a temporary service interruption or a complete failure.

This is where notification messages come in. Often appearing to come from some cryptic sender such as a "Mailer Daemon" (which is just another name for mail transport software), notifications tell us that a recipient does not exist or that a mailbox is full, or about other errors. Each of the "hops" through which a message passes, including the last, may send back one of these "bounces" to let you know that something unexpected is happening to your email.

It's also possible for a sender to request that a notification be sent at final delivery when everything has gone right. Keith Moore's lament, as the NYT reported, is that end-user email software rarely makes it simple enough to answer such a request for it to be worth the sender's effort to ask.

Part of the reason for using the store and forward email model and this system of notifications is its ability to allow receivers of email to make assertions about the availability of their resources. Returning to the analogy of the phone call, when a particular mail exchanger is becoming overloaded, it can "answer the phone" but then immediately request that the sender "call back later," confident that the sender has stored a copy of the message for later forwarding. When you hear the term "soft bounce," it refers to such a delay.

Increasingly, however, receivers are using the "call back later" mechanism to control on the rate at which they accept email from any particular source as a matter only of policy, not of necessity. Recently, the operator of one of the most venerable newsletters on the Internet reported that his deliveries were being blocked by a major ISP. In fact, that ISP was engaging in a delaying tactic, repeatedly asking that the newsletter sender wait and try again.

Intentional delays are, of course, not the only kind. It is rare, for example, that the computer that acts as a mail exchanger is the same one that eventually places the email in a recipient's mailbox, and even more rare for that to be the same computer that eventually presents the email to the recipient. The store and forward model is as prevalent in the systems employed inside domains as it is outside, so it is always possible that a delay is introduced at some point.

So is there anything that you, as an email sender, can do about this? Yes. Remember that the nature of store and forward is to give email the best chance of eventually arriving. It may not get there in an instant, but the chances are that it will get there. This is critical to keep in mind when planning email marketing campaigns. Do you have a last-minute offer? Consider sending it to a smaller, more actively responsive list of recipients, or include a consolation offer so the email remains relevant to those who miss the deadline through no fault of their own. Announcing a one-time event? Plan ahead so there is time for the mail to arrive. You may want to consider a second reminder email, sent closer to the date but only to those who have already reacted to the initial announcement.

Email has persistence by design, as well as (usually) immediacy. It's wise to exploit both.

Is your email being "miscarried"?

Networking consultant Keith Moore doesn't ask for return receipts to confirm that his email has been delivered. This is somewhat ironic, as he was co-author of the 1996 specification that generates a confirmation message when an e-mail safely reaches its destination server. Moore, an advocate of measures to confirm email delivery, is "frustrated because end users' e-mail software still lacks the design capability to use the server-to-server messages about completed email delivery" without user intervention, reports the New York Times.

We're all familiar with email. Messages to and from colleagues arrive in seconds. Such efficiency often leads to the expectation that every email, no matter where it originates or is destined to go, must behave in the very same way. Yet, "in the wild" of the Internet the actual dynamics of email transmission may be different each and every time an email is sent -- even when the sender and the recipient are the same.

Why? It goes back to the creation of the Internet and its architecture. Initially the network was a cooperating group of educational institutions and a few technology corporations. Because data transmission lines were expensive, few of these institutions were directly connected to one another. As a result, senders had to completely describe each of the "hops" a message would need to make from one computer to the next to which it was connected until it reached its destination. Sometimes those transmission lines or the computers connected to them would temporarily stop working, so each computer stored the message locally before passing it along, then deleted the copy after it was passed.

This ensured a robust system because at each step of the way a copy of the file existed on both sides of the transfer. If a line or computer failed during transmission it was possible to wait for the problem to be corrected and then pick up the transmission at a prior step. Called "store and forward" transmission, this technique is still used today. The difference is that transmission details are now mostly hidden from the sender and the recipient. Also the number of hops required to send an email is now far less because the modern Internet passes data through a connected series of transmission lines at a much smaller granularity than that of entire files, providing the illusion that any two computers can connect directly to one another.

The advent of such "direct" connections has led to its own set of problems. Two computers communicating over the network is similar to a phone call, and although it is possible for a computer to answer a lot of calls, it eventually reaches a limit. To keep email flowing, a mechanism is used to identify more than one "phone number" that can be used to answer the "calls" going to the same destination. Called "mail exchangers," these numbers are announced by an Internet domain by publishing "MX records" that also include instructions that senders are expected to obey for how to choose an exchanger for each new email transmission. Generally, the more end recipients an Internet domain represents, the more mail exchangers it will advertise.

AOL, Yahoo, MSN and other large ISPs publish many MX records, as do large corporations. Smaller businesses publish fewer records or share MXs with other small businesses through their ISP. Each record represents a finite resource under the control of its provider. Line and computer failures are far less common than they were back when "store and forward" transmission was invented, but they still happen, and in most cases the system works and gets your email through. Keep in mind, however, that for email the Internet is still largely a cooperative that depends more on the resources of the receivers than of the senders.

Successful email campaigns rely on emails reaching their destination. They rely on the ecosystem working. In a future post, I'll discuss some of the ways that system can work against you, but also some ways you can take better advantage of it.