§49.1–49.4A Basic Distributed File System and the Key to Recovery: Statelessness

Distributed computing’s first great use was the distributed file system distributed file system A file system in which client machines access files and directories stored on a server (or a few servers) over a network, rather than on their own local disks. The arrangement enables data sharing across clients (the same view of the files from any machine), centralized administration and backup, and better security. Ideally it offers transparent access — behaving just like a local file system. defined in ch. 49 — open in glossary : many client machines, one server (or a few) that holds the disks, and files accessed over a network.

Why centralize like this instead of using local disks? Chiefly sharing — you get the same view of your files from any client — plus centralized administration/backup and security (lock the servers in a room). The catch is a new challenge that runs through the whole chapter: failure.

49.1 A Basic Distributed File System

A DFS has two software halves. The client-side file system client-side file system The client half of a distributed file system. It receives the application's file-system system calls, holds what little client state exists (the mapping from an integer file descriptor to a server file handle, and the current file offset), and translates each call into the appropriate protocol messages sent to the server. defined in ch. 49 — open in glossary fields the application’s ordinary syscalls (open, read, write, close, mkdir, …) and turns them into protocol messages; the file server file server The server half of a distributed file system: it stores the data on its disks and responds to protocol messages from clients. In a stateless design each message it receives is self-contained, carrying all the information needed to complete the request, so the server need remember nothing between requests. defined in ch. 49 — open in glossary stores the data and answers those messages (reading from its disk or in-memory cache). Together they define the system’s behavior:

A crucial goal is transparent access transparent access The goal that a distributed file system look and feel just like a local one: the client application uses the same standard API (open, read, write, close, mkdir, …) and gets the same view of files, differing only — perhaps — in performance. No one would want a file system that required a different, painful set of APIs. defined in ch. 49 — open in glossary : the DFS presents the same POSIX API as a local file system, so to applications it looks no different (except, perhaps, in performance). A cached re-read might even generate no network traffic at all.

49.2 On to NFS

Sun’s Network File System network file system NFS — Sun Microsystems' early and hugely successful distributed file system. Rather than a proprietary product, Sun defined it as an open protocol (publishing the exact client/server message formats), so many vendors could build interoperable NFS servers. The classic version studied is NFSv2, whose overriding goal is simple and fast server crash recovery. defined in ch. 49 — open in glossary (NFS) was one of the earliest and most successful. Sun’s clever move: instead of a proprietary product, they defined an open protocol — just the exact message formats clients and servers exchange. Anyone could build an NFS server, so a competitive market grew (Oracle/Sun, NetApp, EMC, IBM, …), and interoperability held. We study the classic NFSv2.

49.3 Focus: Simple and Fast Server Crash Recovery

NFSv2’s overriding design goal is simple, fast server crash recovery. In a many-client, single-server world, every minute the server is down idles every client — “as the server goes, so goes the entire system.” So recovery had to be trivial.

49.4 Key to Fast Crash Recovery: Statelessness

NFSv2’s answer is a stateless protocol stateless protocol A protocol in which the server keeps no state about individual clients — it does not track which files a client has open, current file offsets, or which blocks a client caches. Every request carries all the information needed to complete it. This is NFSv2's central design choice: it makes crash recovery trivial, because after a crash the server simply restarts and clients, at worst, retry their requests. defined in ch. 49 — open in glossary : the server tracks nothing about clients — not which files are open, not file offsets, not which blocks a client caches. Every request carries all the information needed to complete it.

Contrast a stateful protocol stateful protocol A protocol in which the server maintains state about each client — for example, an open file descriptor mapping to a particular file. Such shared state complicates crash recovery: a server crash loses the state (so outstanding operations can no longer be completed), and a client crash leaks server-side resources (like the open file) that the server must somehow detect and reclaim. defined in ch. 49 — open in glossary . Consider ordinary open(), which returns a file descriptor used by later reads:

char buffer[MAX];
int fd = open("foo", O_RDONLY);  // get descriptor "fd"
read(fd, buffer, MAX);           // read MAX from foo via "fd"
read(fd, buffer, MAX);           // read MAX again
...
close(fd);                       // close file

If the server tracked that fd, it would be distributed state distributed state Ousterhout's term for state that is shared between a client and a server — such as the meaning of an open file descriptor that the server tracks on the client's behalf. Distributed state is exactly what a stateless protocol avoids, precisely because keeping it consistent across crashes of either side is hard. defined in ch. 49 — open in glossary (Ousterhout’s term) — shared between client and server — and that is exactly what makes recovery hard:

It’s worse still with client crashes: a client that open()s a file and then dies never sends close(), so the server’s open descriptor leaks until it somehow notices. For all these reasons NFS went stateless — each request self-contained, no recovery protocol, the server just restarts and clients (at worst) retry.