λFS: A Scalable and Elastic Distributed File System Metadata Service using Serverless Functions

Every file open, list or rename in a big distributed file system first goes through a metadata service, and today that service is a fixed cluster of servers that is either too small for a burst or wastefully large the rest of the time — λFS runs the metadata servers as serverless functions that keep a cache in their memory, adds a fast direct TCP path so the functions are not slowed by HTTP, and a cache-invalidation protocol so many function copies stay consistent, reaching up to 4.13 times the throughput, 90 percent lower latency and 86 percent lower cost than HopsFS on a Spotify-derived workload.

Why this problem is worth solving

A

Two facts make this hard. First, metadata workloads are bursty; the paper cites measurements from production clusters and uses a workload derived from Spotify’s 1,600-node HDFS cluster in which spikes reach 7 times the base rate. Second, it is very hard to pick the right size for a metadata cluster ahead of time. If you provision for the spike, you pay for idle machines almost all the time. If you provision for the average, the system falls behind whenever a spike arrives. That is the choice today’s scaled-out designs force on an administrator.

What people did before

IndexFS and InfiniFS use a fixed cluster of metadata servers with heavy client-side caching. InfiniCache, an earlier system from the same lab, used the memory of AWS Lambda functions as a cache for large read-only objects, but invoked a function over HTTP for every operation and never scaled, which the evaluation shows is far too slow for a metadata service.

All of these share one gap: none of them is elastic at the metadata layer. Someone still has to choose a cluster size.

The idea

The core move is to make each

That simple statement runs into three problems with serverless platforms, and most of the paper is the answer to each.

Problem 1: HTTP is slow. The normal way to call a function is an HTTP request through the platform’s API gateway, which the paper measures at 8 to 20 milliseconds end to end for a read. That is far worse than HopsFS. So λFS uses a hybrid path. The first request to a deployment goes over HTTP. The function that serves it then opens a TCP connection directly back to the client’s machine (the client puts its address in the HTTP payload). From then on the client sends requests over that connection, at 1 to 2 milliseconds. Clients on the same virtual machine share one TCP server, and a client with no connection to some deployment can borrow another client’s connection on the same machine.

Problem 2: TCP hides load from the platform. The platform only sees HTTP requests, so if every request went over TCP it would never start new functions. λFS’s auto-scaling policy is deliberately simple: each TCP request is replaced by an HTTP request with a small configurable probability, empirically at most 1 percent. As load grows, the absolute number of HTTP requests grows too, and the platform scales out the deployment. A second, coarser knob is how many concurrent HTTP requests one function instance may serve (the authors extended OpenWhisk to expose this); setting it near 1 forces the platform to start more instances. The paper notes it avoided feedback-loop autoscalers such as Kubernetes’ Horizontal Pod Autoscaler because their delay is too long for sudden bursts.

Problem 3: many copies of the same cached entry. Functions cannot be addressed by name, so the same directory’s metadata may be cached in several instances of the same deployment. λFS partitions the

The hashing of the parent directory, not the full path, means listing a directory or resolving a path touches one deployment. λFS keeps HopsFS INode Hint Cache on the client side so path resolution is one batched query instead of one round trip per path component.

Why it works

The design turns each weakness of serverless platforms into something manageable rather than fighting it.

• Small functions, many of them. A serverless function has little CPU and memory compared to a server. λFS runs each NameNode with about 5 to 6 vCPUs and, for the industrial workload, as little as 6 gigabytes of RAM, but runs dozens of them. Because each instance is small and busy, utilization is high; HopsFS NameNodes, by contrast, sat at about 70 percent CPU because they merely forward requests to the database.
• A cache in front of the database. HopsFS’s throughput is capped by NDB because every request reaches it. In λFS, 95 percent of operations in the Spotify workload are reads and most of those hit the function cache, so the database sees mostly writes. The paper confirms this with a run where the cache was cut to below half the working set: throughput dropped to 70,000 to 80,000 operations per second in the largest burst, still above HopsFS.
• Elasticity without a controller. The random HTTP replacement makes scale-out a side effect of load. In the Spotify run, the number of active NameNodes visibly rises at the start and again at the 200-second spike.
• Pay only while serving. Under AWS Lambda pricing, a function that served no request in a millisecond costs nothing in that millisecond. Combined with fewer total vCPUs (at most 165 to 180 of the 512 available), the same workload cost $0.35 on λFS versus$2.50 on HopsFS.

What it costs

• Writes are slower. Every write runs the invalidation protocol before it can commit. In the Spotify workload HopsFS finished writes 1.5 to 5.55 times faster than λFS. Since the workload is 95 percent reads, average latency still fell sharply, but a write-heavy workload would look different.
• Write throughput is capped by the database. For create file λFS was 1.49 times faster than HopsFS; for mkdir the two were about equal, because NDB, not the NameNodes, is the bottleneck.
• Fixed partitioning. The number of deployments n is fixed; only the number of instances inside each deployment scales. That makes the hashing deterministic and simple but means a hot directory is served by one deployment (scaled out to many instances).
• Engineering effort. λFS is a fork of HopsFS 3.2.0.3 with about 36,700 lines of new code, more than two person-years of work, plus a modified OpenWhisk and a set of failure-handling rules (resubmission with exponential backoff and jitter, cached results for retried requests, straggler cancellation at 10 times the average latency).
• Reclamation. Cloud providers can reclaim warm functions at any time; the paper leaves minimizing that impact on commercial platforms as future work.

Evaluation

All experiments ran on AWS r5.4xlarge machines (16 vCPU, 128 GB RAM) with OpenWhisk on Elastic Kubernetes Service and a 5-node MySQL Cluster NDB, and results were cross-checked on Google Cloud. Both λFS and HopsFS were capped at 512 vCPUs, which the authors note limits λFS relative to a real, nearly unbounded FaaS platform. The main workload replays the operation mix from Spotify’s 1,600-node HDFS cluster (69 percent reads, 17 percent stat, 9 percent ls, 2.7 percent create, 1.3 percent mv) for 5 minutes at a random rate drawn every 15 seconds from a Pareto distribution, with spikes up to 7 times the base rate. HopsFS+Cache is HopsFS with an in-memory cache added to its NameNodes, a serverful stand-in for λFS’s caching layer.

Other results: a cost-normalized HopsFS+Cache given the same dollar budget as λFS (72 or 144 vCPUs) could not sustain the bursts. λFS’s performance-per-cost was 3.33 times that of HopsFS+Cache. Killing an active NameNode every 30 seconds during the Spotify workload caused only brief dips; the workload completed, including the 164,000 ops/s spike. Large mv of directories with 262,000 to 1,000,000 files took 6.5 to 25 seconds on λFS, 13 to 16 percent less than HopsFS for the smaller sizes and equal for the largest, because the database dominates. Porting λFS to IndexFS on a BeeGFS cluster (4,472 lines) gave consistently higher read throughput and much higher write throughput than IndexFS on a 64-vCPU OpenWhisk cluster. Adding a second FaaS platform, Nuclio, took 108 lines.