A data centre is tens to hundreds of thousands of hosts in one building. The hard part is not connecting them, but giving any two of them a fast path at the same time as every other pair.
Words you will meet
-
Data centre network data center network Simple The network inside one company's building full of servers. Precise The internal network of a data centre, connecting tens to hundreds of thousands of hosts — called blades — mounted in racks. Each rack has a top-of-rack switch, and those switches interconnect through a hierarchy of tier-2 and tier-1 switches with border routers at the top. introduced in ch. 6 — open in glossary — the network inside the building, joining racks to each other and to the border routers.
-
Blade — a host, resembling a pizza box; racks typically hold 20 to 40 of them.
-
TOR (Top of Rack) Top of Rack The switch at the top of a rack of data-centre hosts, connecting every host in that rack and linking upward into the switching hierarchy. introduced in ch. 6 — the third-tier switch at the top of each rack.
-
Load balancer load balancer Simple Spreads incoming requests across the hosts that can serve them. Precise The device external requests reach first inside a data centre. It forwards each request to one of the hosts handling that application, choosing by their current load, and also performs a NAT-like translation between the application's public IP address and the chosen host's internal one. Sometimes called a layer-4 switch, because it decides using the destination port number as well as the destination IP address. introduced in ch. 6 — open in glossary — spreads external requests across the hosts that serve an application.
-
Clos network clos network Simple A multi-stage interconnect built from many small switches. Precise A multi-switch layered interconnection network, named after Charles Clos, who studied them in 1953 for telephone switching. Modern data centre fabrics are Clos networks: they serve the same purpose as one very large crossbar switch, but offer multiple switch-disjoint and link-disjoint paths between any two hosts, giving both extra capacity and reliability that a single switch could not. introduced in ch. 6 — open in glossary — the multi-stage interconnect of Figure 6.31, named after Charles Clos.
Why this matters
Every earlier section of this chapter assumed a network built to reach somewhere else. A data centre network is built to reach itself, and that changes which problems are hard.
There is no addressing shortage, no autonomous systems, no policy about whose traffic to carry. There is one owner, one building, and one question: can any two of a hundred thousand hosts talk quickly, all at once?
The section is also this book’s clearest case of an economic argument driving a network design. Networking is only 15 per cent of a data centre’s cost — and the book’s point is that it is where the leverage is.
What is in the building
Hosts are called blades and resemble pizza boxes: commodity machines with a processor, memory and disk. They are stacked in racks, typically 20 to 40 to a rack, and every rack has a top-of-rack switch.
A border router connects the whole thing to the public Internet, because the network carries two kinds of traffic: between external clients and internal hosts, and between internal hosts.
| Cost | Share |
|---|---|
Cells marked ⓘ have an explanation — click to read it.
The book’s breakdown for a 100,000-host data centre, which cost over $12 million a month in 2009. Click a row.
The load balancer
Each application has one publicly visible IP (Internet Protocol) Internet Protocol The network-layer protocol that defines the datagram format and addressing every Internet device must use. introduced in ch. 1 address. Requests arriving for it go first to a load balancer, which forwards each one to a host that handles that application, choosing according to the hosts’ current load.
It is sometimes called a layer-4 switch, because it decides using the destination port number as well as the destination IP address.
In plain words
It does a second job too, and it is the one worth remembering. The load balancer translates the public external address to the internal address of the host it chose, and translates back on the way out. That is exactly the NAT (Network Address Translation) Network Address Translation Letting a whole private network share one public IP address by rewriting addresses and ports. introduced in ch. 4 trick from section 4.3.3.
That means clients can never contact a host directly. The internal structure of the data centre is hidden, and nobody outside can aim at a particular machine.
The hierarchical topology
Drag any device to rearrange the picture. Hover a link to see its rate, delay and length.
Border router, access routers, then three tiers of switches down to the top-of-rack switches. Every rack pair here stands for two of the book’s eight. Switch between the journeys: one crosses the A–B and B–C links, the other never leaves its rack.
Read this diagram as text
- Internet — wired link — border router
- border router — wired link — access router
- border router — wired link — access router
- access router — wired link — tier-1
- tier-1 — wired link — load balancer
- access router — wired link — tier-1
- tier-1 — wired link — tier-2 · A
- tier-1 — wired link — tier-2 · C
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-2 · A — wired link — racks 1–2
- tier-2 · C — wired link — racks 3–4
- tier-2 — wired link — racks 5–6
- tier-2 — wired link — racks 7–8
Below the border router are access routers; below each access router, three tiers of switches. Every link is Ethernet, over a mix of copper and fibre. Hundreds of thousands of hosts fit.
The hosts below one access router form a single subnet. Each such subnet is then cut into smaller VLAN (Virtual Local Area Network) Virtual Local Area Network Several logical LANs configured over one physical switched infrastructure, so that broadcast traffic and group membership are decided by configuration rather than by cabling. introduced in ch. 6 subnets of a few hundred hosts, using section 6.4.4’s mechanism. The reason is to localise ARP (Address Resolution Protocol) Address Resolution Protocol Translates an IP address into the MAC address of an interface on the same subnet. It works only within one subnet, which is the whole point. introduced in ch. 6 broadcast traffic.
The problem with the hierarchy: host-to-host capacity
Two hosts in the same rack can always talk at the full interface rate. Two hosts in different racks often cannot.
The book’s example. Each host has a 10 Gbps link to its top-of-rack switch, and the switch-to-switch links are 100 Gbps. Now suppose ten hosts in rack 1 each send to a host in rack 5. Add ten more between racks 2 and 6, ten between 3 and 7, and ten between 4 and 8.
That is 40 simultaneous flows across the same 100 Gbps A-to-B link. Sharing it evenly gives each flow 2.5 Gbps — a quarter of what its own interface could carry. And it gets worse the higher up the hierarchy a flow has to climb.
What each symbol means
- interface rate — the host’s own link to its top-of-rack switch (bits per second)
- link rate — the switch-to-switch link the flows must share (bits per second)
- flows — how many simultaneous flows cross that link (flows)
Read aloud: A flow gets its fair share of the busiest link on its path, and can never beat its own interface.
Rate per flow2.50 Gbps
each flow's share of the bottleneck = 100 Gbps / 40 = 2.50 Gbps a host cannot exceed its own interface either, so the rate is min(10.0 Gbps, 2.50 Gbps) = 2.50 Gbps the topology is the limit, not the host — this flow gets 25% of what its interface could carry
Fraction of the interface rate delivered—
Change any number above and the arithmetic re-runs, carrying the units through.
The defaults are the book’s own example: ten hosts in each of racks 1–4 sending to a matching host in racks 5–8, so 40 flows cross the A-to-B link. Set the flows to 1 and the host’s own interface becomes the limit again — which is the case for two hosts in the same rack.
Set the flow count to 1 and the host’s own interface becomes the limit again. That is the same-rack case, and it is the only case the hierarchy handles well.
Three ways out, and only one that works
Buy faster switches. It works, and it is expensive. Switches and routers with high port speeds cost a great deal, and the whole network is only 15 per cent of the budget.
Put related things near each other. Co-locate services and data in the same rack or a nearby one, so less traffic climbs the hierarchy. It helps, and it can only go so far.
A search engine may run across thousands of hosts on many racks, with heavy bandwidth needs between all pairs. And a cloud provider wants to place a customer’s virtual machines on whichever hosts have capacity, wherever those hosts happen to be.
Change the topology. This is the one that worked.
The highly interconnected topology
Drag any device to rearrange the picture. Hover a link to see its rate, delay and length.
Every tier-2 switch reaches every tier-1 switch. Switch between the journeys: the two paths share no switch in the middle, which is where both the extra capacity and the reliability come from.
Read this diagram as text
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-1 — wired link — tier-2
- tier-2 — wired link — racks 1–4
- tier-2 — wired link — racks 5–8
- tier-2 — wired link — racks 9–12
- tier-2 — wired link — racks 13–16
Every tier-2 switch connects to every tier-1 switch. The consequence is that multi-path routing becomes the default — flows are multipath flows, not special cases.
The simplest scheme for it is ECMP (Equal Cost Multi Path) Equal Cost Multi Path The simplest multipath scheme: pick the next hop at random among the switches between source and destination. introduced in ch. 6 , which picks the next hop at random among the switches along the way. Finer-grained load balancing has been proposed, and so have designs that route individual packets within one flow along different paths.
| HierarchicalFigure 6.30 | Highly interconnectedFigure 6.31 | |
|---|---|---|
| Does it scale to 100,000 hosts? | ||
| Host-to-host capacity between racks | ||
| Paths between two racks | ||
| What one switch failure costs | ||
| What the switches are like |
Cells marked ⓘ have a reason behind them — click to read it.
Two answers to the same problem. Click any cell.
In plain words
The book compares this tiered network to a single, very very large crossbar switch of the kind section 4.2.2 described — any host able to reach any other.
But it says the tiered network is better than that imaginary crossbar, for two reasons that come from the same property. There are multiple paths from source to destination, which gives capacity. And those paths are switch-disjoint and link-disjoint, which gives reliability.
One big switch could offer neither.
This kind of multi-stage interconnect is a Clos network, named after Charles Clos, who studied them in 1953 for telephone switching. The theory built since then now serves data centres and multiprocessor interconnects.
The switches themselves are small and numerous. Take Google’s Jupiter fabric. In one configuration a TOR switch has 48 links down to its servers, and links up to 8 tier-2 switches. A tier-2 switch has links to 256 TOR switches, and up to 16 tier-1 switches. Facebook’s design gives each TOR switch links to four tier-2 switches, each in a different “spline plane”, with four planes in all. The largest operators build these switches in-house from commodity, off-the-shelf silicon rather than buying them.
Trends
Centralised control. A data centre has a single owner, so SDN (Software-Defined Networking) Software-Defined Networking Separating the control plane from the data plane by computing forwarding tables in a remote controller. introduced in ch. 4 -like logically centralised control is a natural fit. The architectures separate a data plane of simple commodity switches from a software control plane, exactly as section 5.5 described. The scale also makes the automated configuration management of section 5.7 essential.
Virtualisation. Virtual machines decouple software from hardware, and can migrate between physical servers on different racks. Standard Ethernet and IP make it hard to move a virtual machine while keeping its connections alive.
One elegant answer, available because a single authority runs everything, is to treat the whole data centre as one flat layer-2 network.
Check yourself
Check yourself
0 of 7 answered1.Besides spreading requests across hosts, what else does a data centre load balancer do?
2.predictEach host has a 10 Gbps link to its TOR switch, switch-to-switch links are 100 Gbps, and 40 simultaneous flows cross the same link. What rate does each flow get?
3.Two hosts in the same rack want to exchange data. What limits them?
4.Why is co-locating related services in the same rack only a partial answer to the capacity problem?
5.What does the highly interconnected topology of Figure 6.31 buy over the hierarchy?
6.Networking is only 15 per cent of a data centre's cost. What does the book conclude from that?
7.Why do data centres want to treat the whole network as a single flat layer-2 network?
What to remember
- Hosts are blades, 20 to 40 to a rack, each rack with a top-of-rack switch. A load balancer spreads requests and does NAT, so clients never reach a host directly.
- The hierarchical topology scales to hundreds of thousands of hosts, but limits host-to-host capacity. The book’s 40 flows on a 100 Gbps link get 2.5 Gbps each, against 10 Gbps interfaces. Same-rack traffic is always full speed; crossing racks is what costs.
- The highly interconnected topology gives many paths, so multipath routing is the default. Those paths buy capacity and, being switch- and link-disjoint, reliability — which one giant crossbar would give neither of.