8.8Case Studies: SGI Origin 2000, Coherent HyperTransport, HT Assist, Intel QPI

Four commercial directory protocols, spanning two eras and two motivations: the Origin wanted scale (1024 cores, mid-1990s); the three modern ones want point-to-point links — even at small scale.

8.8.1 SGI Origin 2000

Up to 512 nodes, each with two MIPS R10000 processors on a bus that — unusually — does no coherent snooping: it merely connects the processors to the node’s Hub, a specialized ASIC that runs the coherence protocol, interfaces to the network, and fronts the node’s slice of the distributed memory and directory. The protocol descends directly from the Stanford DASH machine (overlapping architecture teams). The network guarantees no ordering whatsoever — not even point-to-point.

Six features worth remembering:

1. Adaptive entry format. Each directory entry is too small to name every possible sharer, so the directory chooses per entry between a coarse bit vector and a limited pointer format.
2. Unordered network ⇒ real races. The §8.7.3 races are not hypothetical here; the protocol handles all of them.
3. Non-ownership E. E (and S) blocks are silently evicted silent eviction Evicting a block without sending any message; permitted for non-owned states (S, sometimes E) in some protocols. defined in Chapter 6 — open in glossary .
4. The Upgrade upgrade A coherence request that asks for S-to-M permission without data transfer (the requestor already holds a valid copy). Vulnerable to a race: if a rival GetM is serialized first, the upgrader's copy is invalidated and it needs data after all (Origin: NACK + retry as GetM; ch08 baseline MOSI: OM^AC demotion to IM^AD). defined in Chapter 8 — open in glossary request + NACK. S→M without pointlessly moving data — but with a window of vulnerability: if a rival GetM/Upgrade is serialized first, P1 is already I when its Upgrade lands. The directory answers with a NACK nack Negative acknowledgment forcing a requestor to re-issue its request; risks livelock (section 9.3.2), so this book's protocols avoid it (the SGI Origin's Upgrade race uses one). defined in Chapter 8 — open in glossary , and P1 falls back to a full GetM.
5. Dual responses for E blocks. P1 in E might have silently evicted, might hold a clean copy, or might have upgraded to M. On P2’s GetS the directory therefore sends memory’s data and forwards to P1, who sends either fresh data (if M) or an ack. P2 waits for both before it knows which data to trust.
6. Two networks, not three. Request + response only. When forwarding could deadlock, the directory sends a backoff backoff SGI Origin mechanism for running a directory protocol on two networks instead of three: when forwarding could deadlock, the directory sends the requestor a backoff message (on the response network) listing the nodes to contact, and the requestor re-sends to them on the request network. defined in Chapter 8 — open in glossary message to the requestor on the response network, listing the nodes to contact — the requestor re-sends to them itself on the request network.

8.8.2 Coherent HyperTransport

Directories were born for scalability — but AMD adopted one for systems of at most eight chips. The prize is different: point-to-point links with no totally ordered broadcast network. Eight chips, three links each, at most three hops apart; with 6 cores per chip, a respectable 48-core system, glueless.

The protocol is a Dir₀B / null directory cache null directory cache No directory state at all (Dir0B): every forwarded request is a broadcast, but the directory controller survives as the LLC controller and the protocol's ordering point. defined in Chapter 8 — open in glossary (§8.5.2, §8.6.3): no stable directory state at all. Step through one transaction:

Judge it either way: optimistically, point-to-point links, zero directory storage or complexity, scalable enough for its goal. Pessimistically, directory-style 3-hop latency (4 counting the off-critical-path completion) plus snooping-style broadcast traffic — in fact more bandwidth than snooping, because every broadcast forward generates a response.

8.8.3 HyperTransport Assist

For the 12-core “Magny Cours” Opteron systems, AMD stopped broadcasting: HT Assist adds an inclusive directory cache at each home chip, covering every block that (a) has this chip as home and (b) is cached anywhere. There is still no DRAM directory — a miss means “not cached anywhere,” and full sets are handled by Recalls recall A directory-initiated eviction of a block from all caches, used when a limited-associativity directory-cache set fills; kept rare by sizing (at least 2x aggregate cache capacity) and non-silent PutS. defined in Chapter 8 — open in glossary . Three distinguishing choices:

8.8.4 Intel QPI

Intel’s QuickPath Interconnect (2008, first in the Core i7-9xx) replaced the electrically bottlenecked shared-wire Front-Side Bus with point-to-point links, specifying everything from the physical layer up. The protocol layer brings two novelties.

MESIF. The usual MESI states plus F(orward) forward (f) A non-MOESI stable state like Owned except clean (the LLC/memory copy is up-to-date). defined in Chapter 6 — open in glossary : a clean, read-only state whose single holder may respond with data to read requests. Compare the neighboring states:

The point of F: read-only data gets sourced from a cache rather than from memory — usually faster, since memory typically answers when a block is read-only.

Two protocol modes. QPI scales its protocol to the system:

^† Intel uses “snoop” for what a core does when it receives a coherence request from another node.

Source Snoop is Coherent HT’s close cousin with one key inversion: in Coherent HT the home broadcasts (so the home still orders every request); in Source Snoop the requestor broadcasts — there is no single arrival point, which is precisely what makes its race handling more complex.