7.7Case Studies: Sun Starfire E10000 and IBM Power5

7.7.1 Sun Starfire E10000

The E10000’s coherence protocol is an unremarkable MOESI with write-back caches. What’s remarkable is the machine around it — built to scale snooping to 64 processors on three observations:

1. Shared-wire buses don’t scale electrically → use only point-to-point links, with a tree as the logical bus logical bus Any interconnect providing totally ordered broadcast without a physical shared-wire bus: a tree serialized at its root (Starfire), or logical-time ordering on arbitrary topologies (Timestamp Snooping, INSO). defined in Chapter 7 — open in glossary : requests unicast up, the root serializes, broadcasts flow down.
2. One logical bus limits request bandwidth → use four address-interleaved logical buses, with processors snooping them in a fixed, pre-determined order to preserve the total order.
3. Data responses need neither broadcast nor ordering → carry them on a crossbar whose bandwidth no bus could match.

The price of this optimized design: the architects had to reason about non-atomic requests and non-atomic transactions — §7.5’s entire menagerie, at commercial scale.

7.7.2 IBM Power5

The Power5 is a 2-core chip (shared L2) whose fabric bus controller links up to eight 4-chip multi-chip modules. Viewed abstractly it’s a MESI protocol on a split-transaction bus — but two features break the mold.

Snooping on rings. Nodes connect by three unidirectional rings (requests; snoop responses/decisions; data), and unidirectional rings provide no total order. So nobody may act immediately. Instead:

1. The requestor sends its request around the request ring (absorbing it when it returns).
2. Each node observes it and contributes one aggregated snoop response snoop response (aggregated) Power5 ring mechanism: each node appends its aggregated intended action as the request circles the ring; the requestor then broadcasts a decision message naming the responder. defined in Chapter 7 — open in glossary — a description of what its processors would do, accumulated node by node around the response ring.
3. When the fully aggregated response reaches the requestor, it determines the outcome and broadcasts a decision message around the ring.
4. The node named responsible sends the data on the data ring.

Far more complicated than bus snooping — the total order exists only logically — but every link is point-to-point and ring switching is delightfully simple and fast.

Extra state flavors. The Power5’s L2 states (Table 7.22):