E-mail is three components and one old protocol: user agents, mail servers, and SMTP, which pushes messages directly from one server to another. That push is exactly why a second protocol is needed to get a message back out again.
Words you will meet
- User agent — the mail program you actually use to read and write messages.
- Mail server — an always-on machine that holds mailboxes and passes mail on.
- Mailbox — the store on a mail server holding one person’s messages.
- Message queue — the waiting line of outgoing mail not yet delivered.
- SMTP — Simple Mail Transfer Protocol: how mail servers hand messages to each other.
- Mail access protocol — how your mail program pulls messages out of your mailbox.
- Push protocol — one where the side holding the data starts the transfer.
Why this matters
E-mail was the Internet’s first killer application, and it is still one of the most used. It is also more complex than the Web in one specific way: it makes use of not one but several application-layer protocols.
That complexity is not an accident, and it is the lesson of this section. SMTP pushes. Reading your mail is a pull. One protocol cannot do both, so the system has a shape that a single protocol would never have produced.
The three components
E-mail is an asynchronous communication medium: people send and read messages when it is convenient for them, without having to coordinate with anyone else’s schedule. Unlike postal mail it is fast, easy to distribute, and inexpensive.
The Internet mail system has three major components.
Drag any device to rearrange the picture. Hover a link to see its rate, delay and length.
Three mail servers, each with its own users. Play a journey: mail always travels server to server, never through a third one.
Read this diagram as text
- User agent — wired link — Mail server A
- User agent — wired link — Mail server B
- User agent — wired link — Mail server B
- User agent — wired link — Mail server C
- User agent — wired link — Mail server C
- User agent — wired link — Mail server C
- Mail server A — wired link — Mail server B
- Mail server A — wired link — Mail server C
- Mail server B — wired link — Mail server C
User agents
User agents user agent Simple The mail program you actually use to read and write messages. Precise The component of the e-mail system that allows users to read, reply to, forward, save and compose messages. Examples include Microsoft Outlook, Apple Mail, Web-based Gmail and the Gmail app on a smartphone. introduced in ch. 2 — open in glossary let users read, reply to, forward, save and compose messages. Microsoft Outlook, Apple Mail, web-based Gmail and the Gmail app on a phone are all user agents.
When Alice finishes composing, her user agent sends the message to her mail server, where it is placed in the server’s outgoing message queue message queue Simple The waiting line of outgoing mail that a server has not yet managed to deliver. Precise The outgoing queue in which a mail server places a message it must send. If the destination mail server cannot be reached, the message stays in this queue and delivery is reattempted, often every 30 minutes. After several days without success the server removes the message and notifies the sender by e-mail. introduced in ch. 2 — open in glossary . When Bob wants to read a message, his user agent retrieves it from his mailbox on his mail server.
Mail servers
Mail servers mail server Simple An always-on machine that holds mailboxes and passes mail on. Precise The core of the e-mail infrastructure. Each recipient has a mailbox on one of the mail servers. Every mail server runs both the client and the server side of SMTP: it is an SMTP client when sending to another server and an SMTP server when receiving. introduced in ch. 2 — open in glossary form the core of the infrastructure. Each recipient has a mailbox mailbox Simple The store on a mail server that holds one person’s messages. Precise The place on a mail server that manages and maintains the messages sent to a particular recipient. The server authenticates the recipient with a username and password before granting access. introduced in ch. 2 — open in glossary located in one of them. When Bob wants to access the messages in his mailbox, the server holding it authenticates him with his username and password.
What happens when the far side is down
Alice’s mail server must also deal with failures in Bob’s.
If it cannot deliver, it holds the message in a message queue and attempts the transfer later. Reattempts are often made every 30 minutes or so. If there is no success after several days, the server removes the message and notifies Alice — by e-mail.
Remember this when you meet reliable data transfer in chapter 3. This is the same idea at a completely different time scale: TCP (Transmission Control Protocol) Transmission Control Protocol The Internet transport protocol that delivers data reliably and in order, with flow control and congestion control. introduced in ch. 1 retries after milliseconds, a mail server after half an hour.
SMTP
SMTP (Simple Mail Transfer Protocol) Simple Mail Transfer Protocol The application-layer protocol that transfers e-mail messages. introduced in ch. 1 simple mail transfer protocol Simple The rules mail servers use to hand a message from one server to the next. Precise SMTP, defined in RFC 5321, the principal application-layer protocol for Internet electronic mail. It uses TCP on port 25 to transfer mail directly from the sender’s mail server to the recipient’s, never through an intermediate mail server. It uses persistent connections, and it restricts the body as well as the headers of all messages to 7-bit ASCII, so binary attachments must be encoded first. introduced in ch. 2 — open in glossary is the principal application-layer protocol for Internet electronic mail. It uses the reliable data transfer service of TCP (Transmission Control Protocol) Transmission Control Protocol The Internet transport protocol that delivers data reliably and in order, with flow control and congestion control. introduced in ch. 1 to move mail from the sender’s mail server to the recipient’s.
Like most application-layer protocols it has two sides: a client side, which runs on the sender’s mail server, and a server side, which runs on the recipient’s.
In plain words
Both sides of SMTP run on every mail server. When a server sends mail to another server it acts as an SMTP client. When it receives mail it acts as an SMTP server.
So “client” and “server” here are roles a single machine plays at different moments — exactly the definition section 2.1.2 gave, where whoever speaks first is the client.
SMTP, step by step
SMTP is defined in RFC (Request For Comments) Request For Comments The name of an IETF standards document. There are currently nearly 9000 of them. introduced in ch. 1 5321, and it is much older than HTTP (HyperText Transfer Protocol) HyperText Transfer Protocol The application-layer protocol that requests and transfers Web documents. introduced in ch. 1 : the original SMTP RFC dates from 1982, and SMTP was around long before that.
The book is candid about the consequences. SMTP has wonderful qualities, as its ubiquity shows, but it is a legacy technology with certain archaic characteristics. The clearest one is that it restricts the body, not just the headers, of all mail messages to simple 7-bit ASCII (American Standard Code for Information Interchange) American Standard Code for Information Interchange The 7-bit text encoding to which SMTP restricts mail bodies. introduced in ch. 2 .
Why 7-bit ASCII is a problem now
The restriction made sense in the early 1980s, when transmission capacity was scarce and nobody was mailing large attachments or image, audio and video files.
Today it is, in the book’s words, a bit of a pain. Binary multimedia data must be encoded to ASCII before being sent over SMTP, and the ASCII decoded back to binary afterwards. Section 2.2 noted that HTTP has no such requirement.
The journey of one message
Alice invokes her user agent for e-mail, provides Bob’s e-mail address (bob@someschool.edu), composes a message, and instructs the user agent to send it.
Step forward. Notice how much of the journey happens without Alice or Bob being present at all.
Read all steps as text
- 1 — Alice writes the message — Alice invokes her user agent for e-mail, provides Bob’s e-mail address (bob@someschool.edu), composes a message, and instructs the user agent to send it.
- 2 — The agent hands it to her mail server — Alice’s user agent sends the message to her own mail server, where it is placed in a message queue. Her agent uses SMTP or HTTP for this short hop.
- 3 — The client side of SMTP notices it — The client side of SMTP, running on Alice’s mail server, sees the message in the queue. It opens a TCP connection to port 25 of the SMTP server running on Bob’s mail server. If that server is down, it tries again later.
- 4 — SMTP carries it across the world — After some initial SMTP handshaking, the client sends Alice’s message into the TCP connection. This is a direct connection between the two mail servers, however far apart they are. No intermediate mail server is involved.
- 5 — It lands in Bob’s mailbox — At Bob’s mail server the server side of SMTP receives the message and places it in Bob’s mailbox. Bob need not be online, awake, or even alive to this point — none of it required him.
- 6 — Bob reads it when he feels like it — Bob invokes his user agent to read the message at his convenience. This is what makes e-mail asynchronous: people send and read messages when it suits them, without coordinating with anyone else’s schedule.
No intermediate mail servers — ever
It is important to observe that SMTP does not normally use intermediate mail servers, even when the two servers are at opposite ends of the world.
If Alice’s server is in Hong Kong and Bob’s is in St. Louis, the TCP connection is a direct connection between the Hong Kong and St. Louis servers. If Bob’s mail server is down, the message remains in Alice’s mail server and waits for a new attempt. It does not get placed in some intermediate mail server along the way.
This is the opposite of how the network layer works, where a datagram is handed from router to router. Do not let the two pictures blur together.
The dialogue
The client SMTP has TCP establish a connection to port 25 at the server SMTP. If the server is down, the client tries again later. Once the connection exists, the two perform some application-layer handshaking — just as humans introduce themselves before transferring information from one to another.
During that handshaking the client states the e-mail address of the sender and the e-mail address of the recipient. Then it sends the message. SMTP can count on TCP’s reliable data transfer to get the message there without errors.
Click any arrow to see what that message says and why it is sent.
crepes.fr is the client, hamburger.edu the server. Click any arrow for the exact ASCII line and what it does. Every line ends with a carriage return and a line feed.
Read this diagram as text
- hamburger.edu sends 220 hamburger.edu to crepes.fr. The server speaks first. This is worth noticing: an HTTP server never says anything until it is asked. Here the greeting arrives as soon as the TCP connection is established.
- crepes.fr sends HELO crepes.fr to hamburger.edu. HELO is an abbreviation for HELLO. The client introduces itself by hostname — just as humans introduce themselves before transferring information from one to another.
- hamburger.edu sends 250 Hello crepes.fr, pleased to meet you to crepes.fr. Every server reply has a numeric reply code and some optional English explanation. Software reads the number; the words are for people reading a transcript like this one.
- crepes.fr sends MAIL FROM: <alice@crepes.fr> to hamburger.edu. The handshaking phase: the client states the e-mail address of the sender. Careful — this is an SMTP command, not the From: header line inside the message. They look alike and are different things.
- hamburger.edu sends 250 … Sender ok to crepes.fr.
- crepes.fr sends RCPT TO: <bob@hamburger.edu> to hamburger.edu. And the recipient. The server checks it holds a mailbox for that address.
- hamburger.edu sends 250 … Recipient ok to crepes.fr.
- crepes.fr sends DATA to hamburger.edu. The introductions are over. The client announces that the message itself follows.
- hamburger.edu sends 354 Enter mail, end with "." on a line by itself to crepes.fr. The server explains how it will know the message has finished. Note the reply code family: 354 is neither success nor failure, but "go ahead".
- crepes.fr sends the message body to hamburger.edu. The message itself, in 7-bit ASCII. A binary attachment would have to be encoded into ASCII before this point and decoded after it.
- crepes.fr sends . to hamburger.edu. A line holding a single period ends the message. In ASCII terms each message ends with CRLF.CRLF — carriage return, line feed, a dot, carriage return, line feed.
- hamburger.edu sends 250 Message accepted for delivery to crepes.fr.
- crepes.fr sends QUIT to hamburger.edu. SMTP uses persistent connections. If the client had more messages for this same server it would begin again with MAIL FROM: and only issue QUIT once all of them had been sent.
- hamburger.edu sends 221 hamburger.edu closing connection to crepes.fr.
Lifelines, left to right: crepes.fr (server), hamburger.edu (server).
The client issued five commands: HELO (an abbreviation for HELLO),
MAIL FROM, RCPT TO, DATA and QUIT. It also sent a line consisting
of a single period, which tells the server the message has ended. In ASCII (American Standard Code for Information Interchange) American Standard Code for Information Interchange The 7-bit text encoding to which SMTP restricts mail bodies.
introduced in ch. 2 terms,
each message ends with CRLF (Carriage Return, Line Feed) Carriage Return, Line Feed The two characters that end every line in HTTP and SMTP. A lone dot between two CRLFs ends an SMTP message body.
introduced in ch. 2 . CRLF Carriage Return, Line Feed The two characters that end every line in HTTP and SMTP. A lone dot between two CRLFs ends an SMTP message body.
introduced in ch. 2 , where
CR and LF stand for carriage return and line feed.
The server issues a reply to each command, each reply having a reply code and some optional English-language explanation.
In plain words — SMTP uses persistent connections
If the sending mail server has several messages for the same receiving server, it sends all of them over the same TCP connection.
For each message the client begins again with a new MAIL FROM:, marks the end
with an isolated period, and issues QUIT only after all the messages have
been sent.
Try it yourself
SMTP is text, so you can be the client:
telnet serverName 25where serverName is a local mail server. You are simply establishing a TCP
connection between your host and that server. You should immediately receive the
220 reply. Then issue HELO, MAIL FROM, RCPT TO, DATA, the message, the
lone period, and QUIT.
Where this breaks today: almost every mail server now refuses to relay mail from a machine that has not authenticated. Many providers also block outbound port 25 entirely, to slow spam. You will usually get the 220 greeting and then a refusal. That refusal is itself the lesson. The protocol in the book is still the protocol on the wire, but a modern server wraps it in checks the 1982 RFC never imagined.
The message format
When Alice writes a paper letter she puts peripheral information at the top: the address, her return address, the date. An e-mail message does the same. The peripheral information sits in header lines, defined in RFC 5322, and the headers are separated from the body by a blank line.
As with HTTP, each header line contains readable text: a keyword, a colon, and a
value. Some keywords are required and others optional. Every header must have
a From: line and a To: line, and may include Subject: and others.
Click any box to read what that part of the message is for.
A typical message header, then a blank line, then the body
From: alice@crepes.fr To: bob@hamburger.edu Subject: Searching for the meaning of life. Do you like ketchup? How about pickles?
Every field, as text
- From: — Required. Who the message is from. (example: From: alice@crepes.fr)
- sp
- value — The address. (example: alice@crepes.fr)
- cr
- lf
- To: — Required. Who it is for. (example: To: bob@hamburger.edu)
- sp
- value — The address. (example: bob@hamburger.edu)
- cr
- lf
- Subject: — Optional, along with other optional header lines. (example: Subject: Searching for the meaning of life.)
- sp
- value — Free text. (example: Searching for the meaning of life.)
- cr
- lf
- cr — Separates the header lines from the body.
- lf
- The message body — 7-bit ASCII only — What Alice actually wrote.
The same shape as an HTTP message, and for the same reason: readable header lines, a blank line, then the body.
These are not the SMTP commands
The header lines here are different from the SMTP commands of §2.3.1, even though they contain some of the same words.
MAIL FROM: <alice@crepes.fr>is part of the SMTP handshaking protocol between two mail servers. It tells the receiving server where the message came from, so it can be delivered or rejected.From: alice@crepes.fris part of the message itself. It is what Bob’s user agent displays.
One is the envelope, the other is the letterhead. They usually agree, and nothing in the protocol requires them to.
Getting mail back out
Once SMTP has delivered the message into Bob’s mail server, it sits in his mailbox. Bob runs a user agent on his own host — a phone or a laptop. So why not put a mail server there too, and let Alice’s server talk straight to it?
In plain words — why your phone is not a mail server
A mail server manages mailboxes and runs both sides of SMTP. If Bob’s mail server lived on his own host, that host would have to remain always on and connected to receive new mail, which can arrive at any time.
That is impractical for most people. So a typical user runs a user agent locally but keeps the mailbox on an always-on shared mail server.
The two-step path, and why
Alice’s user agent could send the message directly to Bob’s mail server. Typically it does not.
Alice’s user agent uses SMTP or HTTP to deliver the message into her mail server. Her mail server then acts as an SMTP client to relay it to Bob’s.
Why the two-step procedure? Primarily because without relaying through her own mail server, Alice’s user agent has no recourse to an unreachable destination. By depositing the message in her own server first, that server can repeatedly try to send it — say every 30 minutes — until Bob’s server is working again. And if Alice’s own mail server is down, she has the recourse of complaining to her system administrator.
The missing piece
How does Bob, running a user agent on his own host, obtain messages sitting in a mail server?
Not with SMTP. Obtaining messages is a pull operation, and SMTP is a push protocol push protocol Simple A protocol in which the side holding the data starts the transfer. Precise A protocol in which the machine holding the data initiates its transfer. SMTP is a push protocol: it pushes a message from the sender’s mail server to the recipient’s. Retrieving mail from a mailbox is the opposite, a pull operation, which is why a separate mail access protocol is needed. introduced in ch. 2 — open in glossary . That is the whole reason a separate mail access protocol mail access protocol Simple The rules your mail program uses to pull messages out of your mailbox. Precise A protocol a user agent uses to retrieve messages from a mail server. It is needed because obtaining messages is a pull operation while SMTP is a push protocol. Today the two common choices are HTTP (Web-based e-mail and smartphone apps) and IMAP, defined in RFC 3501. Both let the user manage folders held on the server. introduced in ch. 2 — open in glossary exists.
Today there are two common ways:
| Bob uses | Protocol | What his mail server must provide |
|---|---|---|
| Web-based mail or a phone app, such as Gmail | HTTP (HyperText Transfer Protocol) HyperText Transfer Protocol The application-layer protocol that requests and transfers Web documents. introduced in ch. 1 | an HTTP interface as well as its SMTP interface |
| A mail client such as Microsoft Outlook | IMAP (Internet Mail Access Protocol) Internet Mail Access Protocol Lets a user agent read and manage a mailbox held on a mail server (RFC 3501). introduced in ch. 2 (RFC 3501) | an IMAP interface as well as its SMTP interface |
Both approaches let Bob manage folders maintained on his mail server. He can move messages into folders he creates, delete messages, mark messages as important, and so on.
| SMTPRFC 5321 | HTTPRFC 7230 | |
|---|---|---|
| Which way does data move? | ||
| Who speaks first? | ||
| Restricted to 7-bit ASCII? | ||
| Port | ||
| Persistent connections? | ||
| Between which machines? | ||
| Age |
Cells marked ⓘ have a reason behind them — click to read it.
Drawn only from what this section and §2.2 actually state. Click any ⓘ.
Check yourself
Check yourself
0 of 7 answered1.Alice's mail server in Hong Kong sends a message to Bob's mail server in St. Louis. How many mail servers does the message pass through?
2.Why does Alice's user agent deposit the message in her own mail server rather than sending it straight to Bob's?
Think about what happens when Bob's server is down.
3.predictIn the SMTP transcript, who sends the first line, and how does that compare with HTTP?
4.You want to e-mail a photograph. What does SMTP's 7-bit ASCII restriction mean for it?
5.`MAIL FROM: <alice@crepes.fr>` and `From: alice@crepes.fr` look almost identical. What is the difference?
6.Why can Bob not use SMTP to fetch his mail from his mail server?
7.Bob reads his mail in a browser at gmail.com. Which protocols are involved on his side, and what must his mail server provide?
What to remember
- E-mail has three components: user agents, mail servers, and SMTP.
- No intermediate mail servers, ever. Hong Kong connects directly to St. Louis. If the far server is down the message waits at the sender’s server and is retried about every 30 minutes.
- Fetching mail is a pull, SMTP is a push, so a mail access protocol is required: HTTP for web mail and phone apps, IMAP for clients like Outlook.