Classic TCP (Transmission Control Protocol) Transmission Control Protocol The Internet transport protocol that delivers data reliably and in order, with flow control and congestion control. introduced in ch. 1 only learns about congestion once it has caused some. Two mechanisms learn it sooner — one asking the network, one watching the delay — and the share each connection ends up with is mostly not a fair one.
Words you will meet
- ECN — Explicit Congestion Notification: routers marking datagrams instead of dropping them.
- ECE / CWR — the two TCP flag bits that carry the news back and confirm it was acted on.
- RTT_min — the smallest round-trip time a sender has seen: the path with nothing queued.
- Fair — each of K connections through a bottleneck of rate R gets about R/K.
Why this matters
Everything in section 3.7.1 shares one weakness. A loss-based sender must cause the problem before it can detect it — it discovers the limit by exceeding it, and a dropped segment is the notification.
Both halves of this section attack that. ECN lets a router say so before its buffer overflows; delay-based delay-based congestion control Simple Reading rising round-trip times, rather than losses, as the sign of a filling queue. Precise An approach that detects the onset of congestion before packets are dropped. TCP Vegas [Brakmo 1995] records RTT_min, the smallest round-trip time seen, computes the throughput an uncongested path would give as cwnd/RTT_min, and slows down when the measured throughput falls well below it. The guiding idea is to keep the pipe just full, but no fuller [Kleinrock 2018]. introduced in ch. 3 — open in glossary control watches the queue build up. And then section 3.7.3 asks the question that has been implicit since section 3.6: if everyone runs AIMD (additive-increase, multiplicative-decrease) additive-increase, multiplicative-decrease TCP’s congestion-control shape: grow the window by one MSS each round trip, halve it on a loss event. It draws the sawtooth (§3.7.1). introduced in ch. 3 , does everyone get an equal share?
Explicit Congestion Notification
ECN explicit congestion notification Simple Routers marking a bit in passing datagrams so senders slow down before anything is lost. Precise ECN [RFC 3168], the Internet’s network-assisted congestion control. Two bits in the IPv4 Type of Service field carry it: one setting lets a sender announce that both ends understand ECN, another lets a congested router mark a datagram. The receiving TCP sets the ECE bit in an acknowledgement; the sender halves its congestion window as if a segment had been lost, and sets CWR in its next segment. What counts as congested is left to the operator. introduced in ch. 3 — open in glossary [RFC 3168] is the form of network-assisted congestion control performed within the Internet, and both TCP (Transmission Control Protocol) Transmission Control Protocol The Internet transport protocol that delivers data reliably and in order, with flow control and congestion control. introduced in ch. 1 and IP (Internet Protocol) Internet Protocol The network-layer protocol that defines the datagram format and addressing every Internet device must use. introduced in ch. 1 are involved.
At the network layer, two bits — four possible values overall — in the Type of Service field of the IPv4 (Internet Protocol version 4) Internet Protocol version 4 The Internet Protocol with 32-bit addresses and a variable-length header. introduced in ch. 4 header are used for ECN. Section 4.3 covers that header properly.
Click any arrow to see what that message says and why it is sent.
Four hops for one bit of news. Click each arrow — note which layer sets the mark and which layer reports it.
Read this diagram as text
- Host A sends datagram, ECN bits = "I understand ECN" to router. One of the four settings of the two ECN bits is used by the sending host to tell routers that both ends are ECN-capable, and can therefore act on a congestion mark. Without it a router would have no reason to mark rather than drop.
- router sends same datagram, ECN bits = "I am congested" to Host B. The router sets the congestion indication. RFC 3168 deliberately does NOT define when a router counts as congested — that is a configuration choice offered by the vendor and decided by the operator. The intuition is to mark at the ONSET of congestion, before buffers actually overflow.
- Host B sends ACK with ECE = 1 to Host A. The receiving TCP saw the mark on an arriving datagram. It cannot do anything about it itself, so it tells the sender by setting the ECE bit — ECN Echo — in a TCP acknowledgement. This is the moment the news crosses from the network layer back into the transport layer.
- Host A sends next segment, CWR = 1 to router. The sender halves its congestion window — exactly as it would for a loss detected by fast retransmit — and sets CWR, Congestion Window Reduced, on its next segment. That tells the receiver the message got through, so it can stop echoing.
Lifelines, left to right: Host A (host), router (router), Host B (server).
Two bits, two jobs, two layers
One setting is used by the sending host to tell routers that the sender and receiver are ECN-capable, and can therefore act on a congestion indication. Without it, a router has no reason to mark rather than drop.
Another setting is used by a router to indicate that it is experiencing congestion. That indication rides in the marked datagram to the destination host.
Then the news has to get back, and it changes layers to do so. The receiving TCP sets the ECE bit — ECN Echo — in a receiver-to-sender acknowledgement. The sender then halves its congestion window, exactly as it would for a loss detected by fast retransmit. It also sets the CWR bit — Congestion Window Reduced — on its next segment.
Both bits are in the TCP flag field of section 3.5.2. They are two of the eight the book’s text miscounts as six.
RFC 3168 does not say when a router is congested
That is deliberate, and worth noticing. The specification defines the signal and leaves the trigger to “a configuration choice made possible by the router vendor, and decided by the network operator”.
The intuition, though, is the whole point: the bit can be set to signal the onset of congestion before loss actually occurs. A protocol that only ever learned from loss could never do that.
Other transport protocols use it too. DCCP (Datagram Congestion Control Protocol) Datagram Congestion Control Protocol A low-overhead, congestion-controlled, UDP-like unreliable transport protocol that uses ECN [RFC 4340] (§3.7.2). introduced in ch. 3 [RFC 4340] provides a low-overhead, congestion-controlled, UDP (User Datagram Protocol) User Datagram Protocol A simple transport protocol with no reliability, no flow control and no congestion control. introduced in ch. 1 -like unreliable service that uses ECN. DCTCP (Data Center TCP) Data Center TCP A TCP variant designed for data-centre networks, which also uses ECN [Alizadeh 2010, RFC 8257] (§3.7.2). introduced in ch. 3 [Alizadeh 2010, RFC 8257] and DCQCN [Zhu 2015], both designed for data-centre networks, use it as well. Measurements show increasing deployment in popular servers and in the routers along paths to them [Kühlewind 2013].
Delay-based congestion control
The second approach needs nothing from the network at all. It notices congestion by watching the delay.
In TCP Vegas tcp vegas Simple The first widely known TCP to slow down on rising delay instead of on loss. Precise A delay-based congestion-control variant [Brakmo 1995] in which the sender compares its measured throughput against cwnd/RTT_min and reduces its rate when the two diverge, so that queues never grow enough to overflow. BBR [Cardwell 2017] builds on these ideas while adding mechanisms to compete fairly with loss-based senders. introduced in ch. 3 — open in glossary [Brakmo 1995], the sender measures the round-trip time for every acknowledged packet. Let RTT_min be the smallest of those measurements — which occurs when the path is uncongested and packets meet minimal queuing delay.
If the congestion window is cwnd, then the uncongested throughput would be
cwnd/RTT_min. So:
- if the actual measured throughput is close to that value, the path is not yet congested, and the rate can be increased;
- if it is significantly less, queues are building, and the Vegas sender decreases its rate.
In plain words
The book quotes the guiding intuition, and it is worth memorising:
“Keep the pipe just full, but no fuller.” [Kleinrock 2018]
Keeping the pipe full means the links — especially the bottleneck that limits the connection — stay busy doing useful work. But no fuller means there is nothing to gain, except delay, from letting large queues build up once the pipe is already full.
A loss-based sender cannot tell the difference between those two states. It keeps pushing until the queue overflows, because overflowing is the only signal it has.
BBR (Bottleneck Bandwidth and Round-trip propagation time) Bottleneck Bandwidth and Round-trip propagation time A delay-based congestion-control protocol [Cardwell 2017] that Google uses on its B4 network and public servers (§3.7.2). introduced in ch. 3 , and the fairness problem it had to solve
The BBR bbr Simple A newer delay-based congestion control, used by Google for much of its traffic. Precise A congestion-control protocol [Cardwell 2017] that builds on TCP Vegas’ delay-based ideas and adds mechanisms allowing it to compete fairly against non-BBR senders. Google began using it for all TCP traffic on its internal B4 network in 2016, replacing CUBIC, and has deployed it on Google and YouTube Web servers. introduced in ch. 3 — open in glossary congestion-control protocol [Cardwell 2017] builds on TCP Vegas’ ideas — and adds mechanisms that let it compete fairly with non-BBR senders.
That addition is not optional politeness. A purely delay-based sender backs off when queues start to grow, while a loss-based sender alongside it keeps pushing until they overflow. The polite one would simply lose.
Google began using BBR for all TCP traffic on its private B4 network [Jain 2013] in 2016, replacing CUBIC, and has deployed it on Google and YouTube web servers. Other delay-based protocols include TIMELY for data centres [Mittal 2015], and Compound TCP [Tan 2006] and FAST [Wei 2006] for high-speed, long-distance paths.
A misprint in the name
This section writes “Compound TCP (CTPC)”. Section 3.8 writes CTCP, which is the correct abbreviation — Compound TCP.
| Loss-basedReno, CUBIC | ECNnetwork-assisted | Delay-basedVegas, BBR | |
|---|---|---|---|
| The signal | |||
| Arrives | |||
| Needs from the network | |||
| The sender then | |||
| Deployed |
Cells marked ⓘ have a reason behind them — click to read it.
All three end with the sender slowing down. They differ entirely in what tells it to.
Fairness
Take K TCP connections, each with a different end-to-end path, but all passing through one bottleneck link of rate R. By bottleneck we mean that every other link on each connection’s path is uncongested and has abundant capacity by comparison.
A congestion-control mechanism is fair fairness Simple Each connection through a congested link getting roughly the same share of it. Precise A congestion-control mechanism is fair if K connections sharing a bottleneck link of rate R each achieve about R/K. TCP’s AIMD converges to this for connections with equal round-trip times and equal segment sizes. In practice it does not hold: connections with shorter round-trip times open their windows faster and get more [Lakshman 1997], UDP flows never back off at all, and an application can simply open several connections at once. introduced in ch. 3 — open in glossary if the average transmission rate of each connection is approximately R/K.
Is AIMD (additive-increase, multiplicative-decrease) additive-increase, multiplicative-decrease TCP’s congestion-control shape: grow the window by one MSS each round trip, halve it on a loss event. It draws the sawtooth (§3.7.1). introduced in ch. 3 fair, given that connections start at different times and so have different window sizes? [Chiu 1989] gives an elegant answer.
Point A. Connection 2 is getting far more than connection 1 — an unfair split, and one the two senders know nothing about. Neither can see the other.
Two connections sharing one bottleneck of rate R, same MSS, same round-trip time. Step through it and watch the gap between them.
Read all steps as text
- 1 — Start anywhere at all — Point A. Connection 2 is getting far more than connection 1 — an unfair split, and one the two senders know nothing about. Neither can see the other.
- 2 — Below capacity, so both grow — The two together are using less than R, so nothing is lost and both add 1 MSS per round trip. Equal increase for both means the pair moves at 45° — parallel to the equal-share line, and therefore no closer to it.
- 3 — Loss, at point B — Eventually the joint use exceeds R and packets are dropped. Both connections see a loss. Point B sits on the full-utilization line.
- 4 — Both halve — and this is the step that matters — Each cuts its window by a factor of two, so the pair moves to C: halfway along a straight line from B back to the origin. Because BOTH are scaled by the same factor, the difference between them is also halved. This is the only move that closes the gap.
- 5 — And again, from C — Below capacity once more, so both grow at 45° to a new loss at D, halve again, and so on. Each cycle leaves the gap unchanged during the climb and halves it at the drop.
- 6 — It converges on the equal-share line — The gap halves on every loss and never grows, so it goes to zero. The pair ends up oscillating along the equal-share line, near where it crosses full utilization — and it gets there from ANY starting point, with no communication between the senders at all.
The one-line reason, which the book leaves to you
The book says “you should convince yourself” that the two connections converge on the equal-share line. Here is the argument in one sentence.
Additive increase adds the same amount to both, so it leaves the difference between them completely unchanged — the pair moves parallel to the equal-share line, never towards it.
Multiplicative decrease scales both by the same factor, so it halves the difference.
The gap therefore never grows and halves at every loss, so it goes to zero — from any starting point, and with no communication between the two senders whatsoever. Only the multiplicative half of AIMD does any of the fairness work.
That is also why problem P41 asks what happens with additive decrease instead: subtracting a constant from both would leave the gap unchanged too, and nothing would ever converge.
Two figure references are off by one here
The text says the two connections share a link “as shown in Figure 3.55” — but Figure 3.55 is the ECN diagram; the two-connection picture is Figure 3.56. The next paragraph says “Figure 3.56 plots the throughput realized by the two TCP connections”, which is Figure 3.57.
Problem P41 inherits the same slip, telling you to refer to Figure 3.55 for the convergence of AIMD when it means Figure 3.57.
Everything else in the argument is exactly as written.
Where the equal share goes wrong
The convergence above rests on three idealised assumptions: only TCP connections traverse the bottleneck, all connections have the same round-trip time, and each host-destination pair has a single connection.
In practice these are typically not met, and client-server applications can end up with very unequal portions.
| What actually happens | |
|---|---|
Cells marked ⓘ have an explanation — click to read it. Sortable columns have a ↕ in the heading.
The convergence argument needs three assumptions. Real networks break all of them.
In plain words
TCP is fair between connections, and nobody ever promised it would be fair between applications, users or organisations.
That gap is not a bug that was overlooked. It is a consequence of where TCP sits. It runs in end systems and sees one connection at a time. It cannot tell that eleven connections through a link belong to one browser. It cannot tell that the flow beside it is UDP and will never yield an inch.
Section 3.8 closes the chapter with a protocol that takes all of this as given, and asks what you would build now.
Check yourself
Check yourself
0 of 6 answered1.Where do ECN's two bits live, and who sets each of the settings?
2.predictStep the ECN ladder to the end. What does the sender do on receiving an acknowledgement with ECE set?
3.What does TCP Vegas measure, and what does it compare it against?
4.predictIn the convergence walkthrough, why does the gap between the two connections shrink?
Ask what each of the two moves does to the difference between them.
5.Nine applications share a link, each with one TCP connection. A tenth arrives and opens 11 parallel connections. What does it get?
6.Two TCP connections share a bottleneck but one has a much shorter round-trip time. What happens?
What to remember
- ECN uses two bits in the IP header: one says both ends understand ECN, the other says a router is congested. The news returns in the TCP header — the receiver sets ECE, the sender halves
cwndand sets CWR. Two layers, one signal. - AIMD converges because additive increase leaves the gap between two connections unchanged and multiplicative decrease halves it.
- The equal share fails in practice for three reasons. Shorter round trips win, UDP never backs off, and an application can open more connections — eleven against nine takes 0.55R.