§50.4–50.5AFS Version 2: Callbacks, File Identifiers, and Cache Consistency

The two scale problems demanded a new protocol. AFSv2’s answer is two ideas: a promise mechanism (callbacks) and a compact file name (the FID).

50.4 AFS Version 2

The big idea is the callback callback In AFS, a promise the server makes to a client that it will notify (recall) the client whenever a file the client has cached is modified by someone else. Callbacks let a client assume its cached copy is valid until told otherwise, replacing NFS-style polling (repeated GETATTR/TestAuth checks) with an interrupt-like model — the key to AFS's scalability. When a writer closes a modified file, the server 'breaks' the callbacks of all other clients caching it, forcing them to re-fetch. defined in ch. 50 — open in glossary : a promise from the server to tell the client whenever a file it is caching gets modified. The client no longer polls (“has this changed?”); it just assumes its cache is valid until the server says otherwise — exactly the shift from polling to interrupts. That alone kills the TestAuth flood.

The second idea is the file identifier file identifier AFS's FID, the analog of an NFS file handle: {volume identifier, file identifier, uniquifier}. The uniquifier allows the volume and file IDs to be safely reused after a file is deleted. Using FIDs, the client walks a pathname one component at a time (caching each directory) rather than shipping whole pathnames for the server to traverse. defined in ch. 50 — open in glossary (FID), AFS’s answer to the NFS file handle, so the client (not the server) walks the pathname one component at a time:

Putting both together, watch a file being read — and then re-read almost for free:

The first open() of /home/remzi/notes.txt costs several Fetches, but it establishes callbacks on every directory and the file — so later accesses are entirely local, behaving just like a local disk file system. That is exactly the scalability AFS was after.

50.5 Cache Consistency

Recall NFS’s two worries: update visibility update visibility One half of the cache consistency problem: the question of when updates made by one client become visible to other clients. If a client buffers its writes in its local cache before propagating them to the server, other clients that read the file in the meantime will see an old version — until the buffered writes reach the server. defined in ch. 49 — open in glossary (when does the server get the new version?) and the stale cache stale cache The other half of the cache consistency problem: after one client (C2) has flushed its update F[v2] to the server, another client (C1) may still hold the old version F[v1] in its cache. Unless C1 re-validates before reading, it returns outdated data instead of the latest copy. defined in ch. 49 — open in glossary (how long until others see it?). Thanks to callbacks and whole-file caching, AFS’s answer is refreshingly simple. Between different machines, both happen at the same instant — when the writer calls close():

There’s one deliberate exception: between processes on the same machine, writes are immediately visible (normal UNIX semantics) — you don’t wait for a close(). Only when you move to a different machine do you see the close-based model. And when two machines race to modify one file, AFS’s whole-file Store gives last writer wins last writer wins AFS's behavior when processes on different machines modify the same file concurrently — more precisely 'last closer wins': whichever client calls close() last flushes its whole file to the server last and becomes the winning version. The result is a file produced entirely by one client or the other, never a block-level intermixing of both clients' writes (which a block-based protocol like NFS can produce, e.g. a corrupt half-and-half JPEG). defined in ch. 50 — open in glossary (really last closer wins): the surviving file is entirely one client’s or the other’s — a big contrast with NFS, whose block-by-block flushes can leave a nonsensical mixture of two writers’ data.