BGP (Border Gateway Protocol) Border Gateway Protocol The protocol ISPs use to tell each other which destinations they can reach; its rules follow the customer-provider and peering relationships. introduced in ch. 5 does three more jobs. It can send every user to the nearest copy of a service. It lets each organisation enforce its own commercial rules by choosing what to advertise. And it is the last step that puts a new company on the Internet.
Words you will meet
-
Anycast ip-anycast Simple Send it to the nearest of several identical servers. Precise Several servers share one IP address, and BGP’s own route selection delivers each client to whichever is closest. introduced in ch. 5 — open in glossary — many servers share one IP (Internet Protocol) Internet Protocol The network-layer protocol that defines the datagram format and addressing every Internet device must use. introduced in ch. 1 address, and BGP delivers each user to the nearest. Written IP-anycast.
-
Routing policy routing policy Simple An organisation's own rules about which routes it will use and advertise. Precise The rules an autonomous system applies when selecting and advertising routes. Policy can override every other consideration, including shortest AS-PATH and hot potato routing, because BGP's route-selection algorithm consults the local-preference attribute first, and that attribute's value is fixed by the local AS. Policy is enforced by controlling which routes an AS advertises, not only which it uses. introduced in ch. 5 — open in glossary — an organisation’s own rules about which routes it will use and advertise.
-
Multi-homed access ISP (Internet Service Provider) Internet Service Provider Connects hosts, and other ISPs, to the rest of the Internet. introduced in ch. 1 multi-homed access isp Simple A customer network attached to two providers at once, and still only a customer. Precise An access ISP connected to the rest of the Internet through two or more different provider networks. Being attached twice does not make it a transit network: like any access ISP, all traffic leaving it must have originated in it, and all traffic entering it must be destined for it. It enforces this by advertising to its providers that it has no path to any destination except itself. introduced in ch. 5 — open in glossary — a customer network attached to two providers, and still only a customer.
-
Selective route advertisement selective route advertisement Simple Not telling a neighbour about a path you know, so it never sends you that traffic. Precise The mechanism by which BGP routing policy is enforced. An AS that knows a path may simply decline to advertise it. A neighbour unaware of the path will never forward traffic over it, so withholding an advertisement is how customer and provider relationships are implemented. introduced in ch. 5 — open in glossary — not telling a neighbour about a path you know.
-
Transit traffic transit traffic Simple Traffic passing through a network that is neither its source nor its destination. Precise Traffic that an ISP carries across its own backbone even though neither the sender nor the receiver is one of its customers. The rule of thumb commercial ISPs follow is that any traffic crossing an ISP's backbone must have a source or a destination, or both, in one of that ISP's customer networks; otherwise the traffic is getting a free ride. introduced in ch. 5 — open in glossary — traffic crossing a network that is neither its source nor its destination.
Why this matters
Section 5.4.3 ended with a route-selection algorithm whose very first rule was local preference — a number an operator sets by hand. That rule was left unexplained.
This section explains it. Routing between organisations is not really about distance. It is about who pays whom, and an autonomous system enforces its commercial position by choosing what to say, not only by choosing what to use.
The section also collects everything from chapters 2, 4 and 5 into one worked example: what it actually takes to put a new company on the Internet.
IP-anycast
Besides being the Internet’s inter- AS (Autonomous System) Autonomous System A group of routers under the same administrative control, identified by a globally unique number. The unit the Internet is divided into for routing. introduced in ch. 5 routing protocol, BGP (Border Gateway Protocol) Border Gateway Protocol The protocol ISPs use to tell each other which destinations they can reach; its rules follow the customer-provider and peering relationships. introduced in ch. 5 is often used to implement the IP-anycast service, and that service is used heavily by DNS (Domain Name System) Domain Name System Translates a human-friendly name such as www.ietf.org into a network address. introduced in ch. 1 .
The motivation is a familiar pair of wishes. First, put the same content on many servers in widely separated places. Second, have each user reach the copy that is closest. A CDN (Content Distribution Network) Content Distribution Network Geographically distributed servers holding copies of content close to users. introduced in ch. 2 may replicate videos on servers in different countries, and the DNS system replicates records on servers throughout the world.
BGP’s route-selection algorithm gives an easy and natural way to do this.
The trick
During the configuration stage the CDN company assigns the same IP address to each of its servers. This is the only unusual thing in the whole design, and it happens once.
Nothing here is a new protocol. Every phase uses BGP exactly as sections 5.4.2 and 5.4.3 described it.
Read all steps as text
- 1 · give every server the same address — During the configuration stage the CDN company assigns the same IP address to each of its servers. This is the only unusual thing in the whole design, and it happens once.
- 2 · advertise it from every server, with ordinary BGP — Each location advertises that one address using standard BGP. No new message type, no new attribute. To the rest of the Internet these look like ordinary advertisements.
- 3 · every router picks one, believing there is one place — A BGP router receiving several advertisements for this address treats them as different paths to the same physical location. They are in fact paths to different physical locations, and the router cannot tell. It runs its normal route-selection algorithm and keeps the best — here, the route one AS hop away.
- 4 · hand every client the same address — When a client requests the video, the CDN returns that one common address, no matter where the client is. The client sends its request there, and the routers already decided which machine that means. The client never chose a server and never knew there was a choice.
The whole design rests on one deception, and it is worth stating plainly.
When a BGP router receives several route advertisements for this address, it treats them as different paths to the same physical location. In fact they are paths to different physical locations. The router cannot tell, and does not need to. It picks the best route by the ordinary algorithm — for example the one fewest AS hops away — and packets follow.
Drag any device to rearrange the picture. Hover a link to see its rate, delay and length.
Both CDN servers answer to 212.21.21.21. Play the first journey: from AS2 the route through AS1 is one AS hop, and every other route is two or more, so AS2 forwards to Server B. Play the second to see the route it rejected.
Read this diagram as text
- 4a — wired link — 4b
- 4b — wired link — 4c
- 3c — wired link — 3a
- 3c — wired link — 3b
- 3a — wired link — 3b
- 1a — wired link — 1c
- 1c — wired link — 1b
- 1a — wired link — 1b
- 1a — wired link — 1d
- 1d — wired link — 1b
- 2a — wired link — 2c
- 2c — wired link — 2b
- 2a — wired link — 2b
- 3a — wired link — 4a
- 4c — wired link — 2a
- 3a — wired link — 1a
- 1b — wired link — 2a
- 3b — wired link — CDN Server A
- 1d — wired link — CDN Server B
Suppose one BGP route is only one AS hop away from a router, and every other route is two or more AS hops away. The router then chooses the location that is one hop away. In Figure 5.12 that is Server B, seen from AS2.
Why this page draws a topology and not a world map
IP-anycast is usually described in geographic language: the nearest server, the closest country. That language is misleading, and a map would make it worse.
“Closest” here means fewest AS hops. It is measured by BGP’s route-selection algorithm, on the graph of autonomous systems, and that graph does not follow geography. A server two streets away can be several AS hops distant, and a server on another continent can be one.
The book’s own Figure 5.12 is a topology for this reason, and this page keeps it one.
CDNs mostly do not use IP-anycast
The example above illustrates the idea well, and then the book says that in practice CDNs generally choose not to use it.
The reason is that BGP routing changes can send different packets of the same TCP (Transmission Control Protocol) Transmission Control Protocol The Internet transport protocol that delivers data reliably and in order, with flow control and congestion control. introduced in ch. 1 connection to different instances of the Web server. Each instance sees half a conversation it never started. Section 3.5 explains why that breaks a connection: the sequence numbers and the connection state live in one machine, not in the address.
DNS does use it, extensively. There are currently 13 IP addresses for root DNS servers, as section 2.4.2 described. Behind those 13 addresses sit many more machines — some single addresses are served by over 100 root servers scattered around the world. IP-anycast routes each query to the nearest one.
A DNS query is a single message with a single reply, so a routing change between one query and the next costs nothing.
Routing policy
When a router selects a route, the AS routing policy can trump every other consideration — shortest AS-PATH, hot potato routing, all of it. That is already visible in the route-selection algorithm: routes are selected first by the local-preference attribute, and that attribute’s value is fixed by the policy of the local AS.
W, X and Y are access ISPs. A, B and C are backbone provider networks that send traffic directly to each other and give full BGP information to their customers. The rule for an access ISP: all traffic entering it must be destined for it, and all traffic leaving it must have originated in it.
Six autonomous systems. W, X and Y are access ISPs; A, B and C are backbone providers. The link colours are the frozen ones from section 1.3.3: amber for customer-pays-provider, teal for backbones exchanging traffic directly.
Read all steps as text
- Who is who — W, X and Y are access ISPs. A, B and C are backbone provider networks that send traffic directly to each other and give full BGP information to their customers. The rule for an access ISP: all traffic entering it must be destined for it, and all traffic leaving it must have originated in it.
- X is multi-homed — X reaches the rest of the network through two different providers, B and C. That is increasingly common. It does not change what X is: like W and Y, X must be the source or the destination of every packet that leaves or enters it.
- So how is X stopped from carrying traffic between B and C? — By controlling which routes X advertises. X advertises to B and to C that it has no path to any destination except itself. Even if X knows a path XCY that reaches Y, it does not advertise that path to B. B never learns it, so B never sends Y-bound traffic through X.
- Now a provider: B has learned AW from A — B installs the route AW. B certainly wants to advertise BAW to its customer X, so that X can reach W through B. A customer being able to reach more of the Internet is what the customer is paying for.
- Should B advertise BAW to C as well? — If B did, C could route traffic to W by way of BAW. A, B and C are all backbone providers, so B would then be carrying traffic between A and C, and paying for the privilege. B may rightly feel that connecting A and C is A and C’s job, and A and C’s cost.
- The rule of thumb — There are no official standards for how backbone ISPs route among themselves. The rule commercial ISPs follow is that any traffic crossing an ISP’s backbone must have a source or a destination, or both, in a customer network of that ISP. Otherwise the traffic is getting a free ride. Individual peering agreements are negotiated between pairs of ISPs and are often confidential.
These are autonomous systems, not routers
The book states this explicitly, and it is easy to slip. A, B, C, W, X and Y in Figure 5.13 are whole autonomous systems. Each box is an entire network with its own routers and its own intra-AS protocol inside it.
Every earlier figure in this chapter drew routers. This one does not.
In plain words
Policy is enforced by silence, not by refusal.
X does not block traffic from B to C at some checkpoint. X simply never tells B that it can reach C. B cannot forward traffic along a path it has never heard of, so no filtering is needed and no packet is ever dropped.
The same trick works upward. B declines to tell C about the path BAW, so C never sends W-bound traffic through B, so B never carries traffic between two networks that pay it nothing.
This is why the route-selection algorithm’s first rule is a policy number. Choosing what to advertise and choosing what to use are the same lever.
Everyday picture
You know a shortcut through your garden between two streets. Your neighbours would use it constantly, wearing a path across your lawn, and you would gain nothing.
You do not build a fence. You simply never mention that the shortcut exists. People route around your house because, as far as they know, there is no way through.
Where the picture stops. A neighbour could still discover the garden by looking. An autonomous system genuinely cannot see a path that nobody advertises — there is no other source of the information.
Why two kinds of protocol at all
The book closes section 5.4 with the question it says you should have been wondering all along. Having seen OSPF (Open Shortest Path First) Open Shortest Path First The Internet’s intra-AS link-state routing protocol. Each router builds a complete map of its autonomous system and runs Dijkstra locally. introduced in ch. 5 and BGP in detail: why are they different protocols?
| Inside one ASOSPF, IS-IS | Between autonomous systemsBGP | |
|---|---|---|
| Policy | ||
| Scale | ||
| Performance |
Cells marked ⓘ have a reason behind them — click to read it.
The book’s "Principles in Practice" answer, on book page 409. Click any cell for the reason.
The three answers are not independent. Because inter-AS routing is so policy oriented, the quality of the routes chosen is often of secondary concern. A longer or costlier route that satisfies a policy may well be taken over a shorter one that does not.
Putting the pieces together: obtaining Internet presence
This last part is not about BGP as such. It collects IP addressing from chapter 4, DNS from chapter 2, and BGP from this chapter into one worked example.
Suppose you have created a small company with a public Web server, a mail server and a DNS server. You want the whole world to be able to visit your Web site, and your employees to exchange e-mail with customers everywhere.
Contract with a local ISP and connect to it. Your company has a gateway router, and it connects to a router in your local ISP. The link might be DSL over the existing telephone infrastructure, a leased line, or any other access technology from chapter 1.
The book’s own worked example, in its own order. Every step uses something from an earlier chapter, and the last one is BGP.
Read all steps as text
- 1 · buy connectivity — Contract with a local ISP and connect to it. Your company has a gateway router, and it connects to a router in your local ISP. The link might be DSL over the existing telephone infrastructure, a leased line, or any other access technology from chapter 1.
- 2 · get an address range — Your local ISP also provides an IP address range — for example a /24, which is 256 addresses. This is the prefix that will later be advertised on your behalf.
- 3 · assign the addresses — One address to the Web server, one to the mail server, one to the DNS server, one to the gateway router, and the rest to other servers and networking devices inside the company.
- 4 · get a domain name — Contract with an Internet registrar to obtain a domain name — xanadu.com, if the company is Xanadu Inc. This is chapter 2 territory, not BGP.
- 5 · get into the DNS system — Outsiders need to contact your DNS server, so give the registrar its IP address. The registrar puts an entry for it — domain name and address — into the .com top-level-domain servers. After that, anyone who knows your domain name can find your DNS server.
- 6 · fill in your own DNS server — So that people can discover the address of your Web server, add entries in your own DNS server mapping www.xanadu.com to its IP address, and similar entries for the mail server and any other public server.
- 7 · and one step is still missing — Alice knows the address and sends a TCP SYN segment to it. That datagram is routed through many autonomous systems, and every router along the way must find an entry for your /24 prefix. It gets one from BGP: your local ISP advertises your prefix to the ISPs it connects to, they propagate it, and eventually every Internet router knows your prefix or an aggregate containing it.
In plain words
Steps 1 to 6 give you a working network with a name. They do not give you a reachable network.
Alice’s first datagram has to be forwarded by routers in autonomous systems that have never heard of you. Each of them looks in its forwarding table for an entry matching your prefix, and there will not be one unless somebody told them.
BGP is what tells them. Your ISP advertises your prefix, its neighbours propagate the advertisement, and eventually every router in the Internet holds your prefix or an aggregate that contains it.
That is the sense in which the book calls BGP the protocol that glues the Internet together. Without step 7 you are the isolated island from the start of section 5.4.
Check yourself
Check yourself
0 of 7 answered1.What does a BGP router believe when it receives several advertisements for one IP-anycast address?
2.In Figure 5.12, why do AS2's routers forward traffic for 212.21.21.21 toward Server B rather than Server A?
3.Why do CDNs generally avoid IP-anycast, even though DNS uses it heavily?
4.predictX is a multi-homed access ISP attached to providers B and C, and X knows a path XCY that reaches network Y. What does X advertise to B?
5.B has learned the path AW from A. It advertises BAW to its customer X. Why might B refuse to advertise BAW to C?
6.The book asks why the Internet uses different protocols inside an AS and between ASs. Which statement matches its answer?
7.You have connectivity, a /24 address range, a domain name, and DNS records mapping www.xanadu.com to your Web server. Alice sends a TCP SYN to that address and it never arrives. What is missing?
What to remember
- IP-anycast gives many servers one address, and every router picks a route to it with the ordinary BGP algorithm. So “closest” means fewest AS hops, not fewest kilometres — the graph BGP measures on is the graph of autonomous systems.
- Policy is enforced by not advertising. X tells its providers it has a path to nothing but itself, so no provider ever sends transit traffic through it.
- Getting online ends with BGP. Addresses, a domain name and DNS records are not enough; every router in the Internet must learn your prefix.