Design and Operation of Shared Machine Learning Clusters on Campus

Kaiqiang Xu, Decang Sun, Hao Wang, Zhenghang Ren, Xinchen Wan, Xudong Liao and 3 others

A university buys a pile of GPUs for its researchers to share, and then discovers the hard part is not the hardware but running it — the three obvious options each fail a small team, since raw shell access is unstable, traditional HPC tools do not understand machine-learning jobs, and Kubernetes is a full-time job to operate; this paper from HKUST describes SING, a deliberately minimal four-layer system that a tiny staff can keep running, and reports two and a half years of real operating data from a cluster serving hundreds of researchers.

The argument in one glance

Problem

A campus buys GPUs, then cannot afford to run them well

Universities are pouring money into GPU clusters for machine-learning research, but many lack the expertise and, critically, the manpower to operate a shared cluster properly. The binding constraint is not compute — it is the handful of graduate students or IT staff who have to keep the thing running while also doing their own work. A cluster that demands constant expert attention is one that ends up underused, misconfigured, or down, which quietly puts a barrier in front of the researchers it was meant to serve.
What people did

Reach for one of the three familiar options

The usual choices are direct shell access over SSH, a traditional high-performance-computing scheduler like Slurm, or a container platform like Kubernetes. Each is a reasonable, well-trodden path, and each is what some campus somewhere is running today.
Why it fell short

Each option fails a small team in its own way

Raw SSH access has no isolation, so users collide over memory and dependencies, the system grows unstable, and fairness is impossible. Slurm queues and schedules well but has no idea what an ML dependency environment is, forcing users to hand-build one in every job script, and it funnels everything through a login node that a single careless user can crash. Kubernetes is powerful but built for internet services — load balancing, rollbacks, service discovery the cluster does not need — and lacks native user access control or job queuing, so operating it becomes a full-time job the team does not have.
This paper's idea

Build the least system that does the job, and prove it by running it

SING is a four-layer architecture chosen for minimal viability: the simplest, most stable software stack that still lets researchers submit, schedule, and monitor ML jobs, so a tiny staff can keep it running. It augments Slurm with the ML-specific pieces it lacks, schedules fairly with plain first-come-first-served plus backfill rather than an intrusive optimizer, and — the real contribution — is validated by two and a half years of operational data from a live campus cluster, released as open traces and code.

Why this problem is worth solving

The thing being shared is expensive and scarce, and how it is shared decides who actually gets to do research.

A shared GPU cluster is a pool of GPU-equipped machines that many users submit jobs to, rather than each researcher owning their own hardware. Sharing is what makes the investment worthwhile: GPUs are costly and often idle in any one person’s hands, so pooling them raises utilization and widens access. But sharing is also what creates every hard problem — multiple users now compete for the same memory and compute, need isolation from each other’s runtime environments, and require some fair way to take turns, none of which a single-owner machine ever has to solve. The whole discipline of cluster management exists because the moment a resource is shared, questions of allocation, isolation, and fairness appear that did not exist before, and answering them badly means the cluster serves a few power users well and everyone else poorly.

What makes the campus version distinct from an industrial one is who is available to keep it alive.

Operational efficiency, as this paper uses it, means keeping both the maintenance cost and the resource waste low at the same time — a cluster that a small team can run without heroics and that still keeps its GPUs busy. It is a different target from raw performance. An industrial platform can throw a dedicated operations team at a complex system; a research group has a few people whose real job is research, so every configuration knob, every fragile component, every incident that needs a human at 2am is a direct tax on the science the cluster is supposed to enable. This is why the paper’s design principle is minimal viability rather than maximal capability: the right system is the simplest one that still meets the essential needs, because complexity the team cannot sustain is worse than a missing feature, and the measure of success is a cluster that mostly runs itself.

What people did before, and where it broke

Give everyone direct shell access over SSH. The simplest possible sharing: hand out logins to the GPU machines and let people work directly. Where it broke. There is no resource isolation or coordination, so concurrent users collide over GPU memory, disrupt each other’s dependencies, and destabilize the system, and fair allocation is impossible without constant manual policing.
Adopt a traditional HPC scheduler like Slurm. Universities already know Slurm from scientific computing, and it does job queuing and scheduling well. Where it broke. Slurm has no notion of the complex dependency environments ML jobs need, so users must hand-build one in every job script; and it requires a login node for job preparation that becomes a single point of failure — a user who accidentally runs a heavy job there can render it unresponsive and take the whole cluster offline.
Deploy a container platform like Kubernetes. Kubernetes is the industry standard for orchestrating containerized workloads at scale. Where it broke. It was designed for internet backend services, so it carries machinery — load balancing, rollbacks, service discovery — that an ML cluster does not need, while lacking native user access control and job queuing that it does need, which have to be bolted on; the result is a steep learning curve and high maintenance for both operators and users, who must also write error-prone container files.
Adopt a research-grade optimizing scheduler. State-of-the-art schedulers squeeze more out of a cluster by preempting, resizing, and reordering jobs to optimize completion time or utilization. Where it broke. They optimize cluster-wide metrics that can be unfair to individual users, and many require modifying user code — batch sizes, hyperparameters — which is unacceptable in research where the code is the experiment.
The gap. Every option was either too unstructured to be stable, too generic to understand ML jobs, too heavy to operate with a small team, or too intrusive to be acceptable to researchers. What was missing was a system deliberately scoped to a research group’s actual constraint — limited manpower — and a body of real operating evidence that such a scoped system works.

How SING is put together

SING's four layers, from a user's job profile to execution

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Choose the least complex system that meets the need, because the scarce resource is people, not GPUs. The instinct in systems design is to reach for the most capable tool — Kubernetes for orchestration, a research scheduler for utilization — but capability carries a maintenance cost, and for a research group that cost is paid in the time of a few overworked people. SING inverts the priority: it treats operational simplicity as the primary objective and asks, for every component, whether a simpler choice would still meet the essential requirement. It keeps Slurm, which the team already understands, and adds only the ML-specific pieces Slurm lacks, rather than adopting a heavier platform and fighting its complexity. The payoff is concrete and measured — building it fresh takes about seventeen person-days, adopting the open-source version about eight, and over a year the cluster generated only one service request per roughly nine hundred and forty jobs submitted — because a minimal architecture has fewer parts to fail and fewer knobs to misconfigure. Deliberately refusing the powerful-but-intrusive option in favor of the plain one that does not fight the people who use it is the same judgment the non-intrusive cluster-scheduler work covered elsewhere on this site makes, choosing not to preempt and rewrite users’ jobs precisely because the clever thing is the thing production operators cannot live with.

Make fairness the default and buy efficiency at the edges, without ever touching user code. A research cluster is a shared public facility, so SING schedules the way a university lends its facilities: first come, first served, so everyone gets equal access and no optimizer decides that one person’s job matters less than another’s. But pure ordering wastes resources when a large job at the front blocks smaller ones that would fit, so SING adds a backfill mechanism that runs those smaller jobs in the gaps — capped by a threshold, equal to the front job’s size, so the front job cannot be starved indefinitely. It further tidies allocation into powers of two and reserves a slice of hardware, carved out with virtualized network cards, exclusively for small jobs, which both cuts fragmentation and gently pushes users to test small before scaling up. Crucially, none of this asks the researcher to change a batch size or a line of code. In the replayed trace, this kept GPUs assigned about ninety-six percent of the time, far above plain first-come-first-served’s seventy-five, while avoiding the long-job starvation that a shortest-job-first policy would cause. Raising utilization for the many while protecting fairness for each individual, rather than maximizing a single global number, is the same balance the fair GPU-sharing work covered elsewhere on this site strikes when it serves multiple tenants efficiently without letting throughput trample anyone’s guaranteed share.

The contribution is not a clever algorithm but a system proven by operating it, and given away as evidence. SING’s scheduler is textbook first-come-first-served with backfill; its file system, its container runtime, its scheduler backend are all existing, boring components. What is novel is not any one piece but the demonstrated judgment of how to assemble them for a specific, under-served setting, backed by two and a half years of running a live cluster for hundreds of researchers — and the decision to release that experience as data: usage patterns, incident write-ups, and a job trace more detailed than the well-known Microsoft and Alibaba traces, recording software dependencies and per-node traffic that earlier ones omit. The value is the operational record itself — that GPU utilization falls as jobs span more nodes, that eight-node jobs average only a quarter GPU utilization because communication-efficient training is rare, that a network-card upgrade once broke every container for three days — the kind of ground truth that only comes from years of production. Treating the lessons of operating a real system at scale as a first-class research artifact, mined for patterns and published so others can build on them, is the same move the warehouse-scale allocator-characterization work covered elsewhere on this site makes, where the finding worth having is what a real fleet actually does, not what a model predicts it should.

Following one job through SING

A researcher submits, debugs, and scales a training job

An ordinary ML research workflow on SING: test small, then scale. The steps mirror the four layers and the features that make each cheap to operate.

Step 1 of 5

Write a job profile and submit from the laptop

The user writes one self-contained profile naming the resource group, the number of nodes and GPUs, a script or container entry point, and the Python and CUDA dependencies, then submits it with the local SING command-line tool. No logging into a cluster login node — that fragile single point of failure is gone.

The same profile runs locally and remotely with identical steps, and can be saved, reused, and shared with collaborators to reproduce an experiment.

Why it works

Minimalism lowers the maintenance the team cannot afford. A research group’s binding constraint is a few people’s time → so a stack with fewer parts and fewer knobs fails and misconfigures less often → which is why SING needs only about eight person-days to adopt and generates one service request per roughly nine hundred and forty jobs.
Removing the login node removes a whole class of outage. In HPC setups users prepare jobs on a shared login node that one careless heavy process can crash → but a self-contained job profile submitted from the user’s own laptop means there is no such node to bring down → so an entire common cause of cluster-wide outages simply does not exist.
Backfill recovers the utilization that pure fairness leaves on the table. First-come-first-served is fair but idles resources when a big job blocks smaller ones behind it → so backfilling those smaller jobs into the gaps, capped to protect the front job, fills the idle GPUs → lifting GPU assignment to about ninety-six percent against seventy-five for plain ordering, without starving anyone.
Restricting interactivity to small jobs steers behavior toward efficiency. Interactive debugging on many GPUs holds expensive hardware nearly idle → so allowing port forwarding and remote shells only on jobs of two or fewer GPUs pushes users to test small first → which concentrates the wasteful exploratory phase on cheap resources and frees the big ones for real work.
Operating for years turns the system into evidence others can trust. Any design can look good on paper → but running a live cluster for hundreds of users over two and a half years surfaces what actually happens — bimodal job lengths, low utilization on multi-node jobs, a three-day outage from a network-card swap → so the released traces and incident reports carry ground truth that a benchmark cannot.

What it costs (trade-offs and scope)

This is an experience paper, not a new algorithm. The scheduler is standard first-come-first-served with backfill, and the file system, container runtime, and Slurm backend are all existing components; the contribution is engineering judgment plus operational data, so readers looking for a novel scheduling theory will not find one, by design.
The headline scheduling numbers come from one week, replayed by the authors. The ninety-six-percent assignment rate and the comparisons against first-come-first-served and shortest-job-first are from a single busy-week trace replayed in the authors’ own simulator, with shortest-job-first given oracle job runtimes — a reasonable illustration, not a broad benchmark, and from one institution.
On average waiting time, it is not the winner. Shortest-job-first actually beats SING on mean queueing time (about 0.64 hours versus 0.81); SING’s case is that it avoids that policy’s long-job starvation (its tail queueing is far lower) and stays fair — a deliberate trade, not a clean dominance.
The reported scale shifts between sections. The introduction cites over 160 GPUs and 480 users, while the operational analysis studies a 128-GPU, 16-node configuration, and trace windows range from 30 to 36 months — natural for an evolving production system described at different snapshots, but it means no single consistent system is measured throughout.
Security is acknowledged as under-hardened. The command-line tool reaches the cluster by executing remote commands over SSH, and interactive services are exposed by opening ports on a public gateway; the authors note both could allow unauthorized actions and point to safer designs (a real API, SSH tunneling) they have not yet adopted.
Isolation and failure recovery lean on the substrate. Job isolation relies entirely on Slurm’s cgroups, whose security and performance separation the paper says needs further study, and SING does not auto-recover from node failures — a failed node disconnects its jobs, which lose progress to their last checkpoint and must be resubmitted by hand.
Some findings are about users, not the system. That eight-node jobs average only twenty-five percent GPU utilization reflects researchers not adopting communication-efficient training; SING surfaces this waste but does not fix it, so a large amount of inefficient work rides on the cluster regardless of scheduling.
The scope is a trusting academic setting. The whole minimal-viability bet assumes cooperative users, limited manpower, and no strict service-level guarantees; a commercial or adversarial multi-tenant environment might genuinely need the heavier access control and isolation that SING deliberately omits.

The evaluation

Four ways to run a shared campus ML cluster

Property	Direct SSH access	Slurm as-is / Kubernetes
Operator maintenance burden
Understands ML dependency environments
Touches user code to gain efficiency	No	No
Fairness across individual users
Evidence it works	Anecdotal	Widely used, mixed for ML

A qualitative comparison of shared-cluster approaches for a research setting, drawn from the paper's own analysis and its two-and-a-half-year deployment. The scheduling percentages come from a single-week trace replayed in the authors' simulator.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SING’s contribution is operational judgment and evidence, not a new algorithm: its scheduler is ordinary first-come-first-served with backfill, and its components are all existing software, chosen for minimal viability so a small team can run the cluster. Read the numbers with that framing. The headline ninety-six-percent GPU assignment and the comparisons against first-come-first-served and shortest-job-first come from a single busy-week trace replayed in the authors’ own simulator, at one institution — and on average waiting time shortest-job-first actually beats SING; SING’s real claim is fairness plus avoiding that policy’s long-job starvation. Reported scale shifts between sections (160-plus GPUs and 480 users in the framing, a 128-GPU analysis, 30-to-36-month traces) because it is a living system snapshotted at different times. Security is explicitly under-hardened (remote command execution over SSH, public gateway ports), isolation rests on Slurm’s cgroups without a separation guarantee, and there is no automatic recovery from node failure. Some of the most interesting findings — eight-node jobs averaging a quarter GPU utilization — are about how researchers write jobs, not about SING, and the whole minimal-viability bet assumes a cooperative academic setting rather than an adversarial commercial one. What is genuinely valuable is the honesty and detail of the operating record, released as open traces and incident reports.

Background

Why the plumbing of shared research clusters matters

It is worth stepping back to see why a paper about the operations of a campus cluster belongs at a top systems venue alongside new chips and compilers. The headline story of machine learning is ever-larger models trained on ever-larger fleets, but underneath that story is a quieter question of access: who actually gets to run these workloads. For most researchers the answer is a shared institutional cluster, and whether that cluster is usable — whether a graduate student can submit a job without becoming a systems administrator, whether the GPUs they were promised are actually reachable — is decided by exactly the unglamorous plumbing this paper documents. The democratization of machine learning is often framed as a matter of open models and open data, but it is just as much a matter of shared infrastructure that ordinary labs can afford to operate, and that infrastructure only works when someone does the patient, undramatic work of making a messy real fleet serve many people fairly. That is the same territory the heterogeneous-cluster elastic-training work covered elsewhere on this site inhabits, wringing usable throughput out of a jumble of real, imperfect, unevenly-fast hardware rather than an idealized machine. The broader point is that the reach of machine learning is limited not only by the frontier of what the largest labs can train, but by how well the thousands of smaller shared clusters — the ones most researchers actually touch — can be run by the few people who keep them alive.

If you remember only three things

1. Running a shared GPU cluster for ML research is dominated by operational burden under limited staffing, and the three familiar options each fail that constraint: direct SSH access has no isolation and collapses into conflicts, traditional HPC schedulers like Slurm do not understand ML dependency environments and hang everything on a crash-prone login node, and Kubernetes carries so much internet-service machinery that operating it becomes a second full-time job.
2. SING answers with deliberate minimalism — a four-layer architecture (self-contained job profiles, a thin adapter CLI plus cluster service, a fair first-come-first-served-with-backfill scheduler, and swappable Slurm-based execution) built from existing components and scoped to the simplest stack a tiny team can sustain, augmenting Slurm with the ML-specific pieces it lacks, scheduling fairly without ever touching user code, and eliminating the fragile login node.
3. The real contribution is evidence, not an algorithm: two and a half years operating a live cluster for hundreds of researchers, released as open code and an unusually detailed job trace, showing about ninety-six-percent GPU assignment and roughly one service request per nine hundred jobs — though the scheduling numbers come from a single replayed week, shortest-job-first beats it on average wait, security and isolation are acknowledged as under-hardened, and the whole bet assumes a cooperative academic setting.

Check yourself

Check your understanding

1. What is the binding constraint that shapes SING's entire design?
2. Why does SING use plain first-come-first-served with backfill instead of a research-grade optimizing scheduler?
3. How does removing the login node improve stability?
4. Which limitation most affects how SING's results should be read?

Sources

The paper is “Design and Operation of Shared Machine Learning Clusters on Campus” by Kaiqiang Xu, Decang Sun, Hao Wang, Zhenghang Ren, Xinchen Wan, Xudong Liao, Zilong Wang, Junxue Zhang, and Kai Chen, published at ASPLOS 2025, pages 295-310, DOI 10.1145/3669940.3707266. The authors are at the Hong Kong University of Science and Technology.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction and background motivating operational efficiency under limited manpower and analyzing the three existing approaches (direct shell access, traditional HPC/Slurm, and Kubernetes) summarized in the paper’s comparison table; the design section detailing the four layers — job profile, adapter, scheduler, and executor — and the seven features, including multi-format job profiles, the SING-cli and SING-service adapter with Conda caching and interactive port-forwarding, the optimized first-come-first-served-with-backfill scheduler with its starvation threshold, tidy power-of-two granularity, and reserved small-job pool, early-initialization caching, distributed bootstrapping over a GlusterFS RDMA file system, and heterogeneous swappable executor backends; the usability section with the person-day setup estimates and the one-request-per-940-jobs figure; and the operation and analysis section reporting the two-and-a-half-year usage patterns, the roughly ninety-six-percent GPU assignment rate, the simulator comparison of SING against first-come-first-served and shortest-job-first, the job-level runtime, queueing, size, and utilization distributions, the incident write-ups (network file system delay, the three-day network-card-upgrade outage), the open-source job trace schema, and the stated limitations on isolation, connection security, port forwarding, and cluster failure recovery.