Everywhere All at Once: Co-Location Attacks on Public Cloud FaaS

A whole family of attacks works by watching how a victim program disturbs shared processor hardware, and every one of them needs the same first step — the attacker's code must end up on the SAME physical machine as the victim's, which used to be easy and has become hard, since cloud networks now isolate tenants from each other, datacenters hold thousands of machines, and serverless platforms hide placement entirely by deciding for you where your code runs; this paper (Illinois and Tel Aviv) shows that step is far easier than assumed, because the sandbox isolating tenants carefully conceals every piece of host identity it thought of — processor model, cache sizes, network address, uptime — while leaving readable a counter that simply ticks once per processor cycle since the machine last booted, from which the boot moment can be computed and used as a machine fingerprint; combining that with a measured quirk of how the platform spreads instances under sudden load, the authors reach every victim instance in one datacenter for about 23 dollars, and the paper closes with mitigations, one of which already exists in hardware and costs nothing.

The argument in one glance

Problem

Every hardware side-channel attack needs a first step nobody had studied

A large family of attacks extracts secrets — cryptographic keys among them — by watching how a victim program disturbs hardware that both parties share, such as caches or the units that perform arithmetic. Enormous research effort has gone into that watching. But all of it presumes something first: the attacker's code must be running on the SAME PHYSICAL MACHINE as the victim's, since there is nothing to share otherwise. That step has quietly become hard. Cloud networks now isolate tenants from one another, which defeated the older approach of finding neighbours by probing the network. Datacenters have grown so large that landing on any specific machine by chance is remote. And serverless platforms remove the last handle, because the provider decides entirely where code runs, instances appear and vanish with demand, and the customer cannot pick a zone or a processor as they could when renting whole machines.
What people did

Find your neighbours by network probing, or test pairs with a covert channel

Two approaches exist. The original work located neighbours by probing the network, since machines sharing a host once shared observable network characteristics. When tenant isolation closed that route, researchers turned to direct testing — take two of your own instances, have both hammer some shared hardware resource at once, and see whether they interfere with each other. Interference means they are on the same machine, which is ground truth rather than inference.
Why it fell short

Network probing no longer works, and pairwise testing cannot scale

Network probing was obsoleted by the widespread adoption of tenant network isolation, which logically separates each account's networking so that neighbours are no longer visible. Pairwise interference testing still works but has a fatal cost — it tests one PAIR at a time. In a serverless datacenter an attacker may need to launch thousands of instances to reach a victim, and checking every pair among thousands is quadratic and prohibitively expensive. There is also a subtler gap. Neither technique says anything about WHY instances land where they do, and on a serverless platform placement is the provider's decision, made by an algorithm nobody outside has described. Without understanding that algorithm you cannot know whether co-location is a matter of luck or something an attacker can arrange.
This paper's idea

Fingerprint the machine from a counter the sandbox forgot to hide

This paper (Illinois and Tel Aviv — Zhao, Morrison, Fletcher, Torrellas) studies that neglected first step. Its central observation is about what a sandbox conceals. The isolation layers used by serverless platforms carefully hide every piece of host identity anyone thought to name — the processor model, the cache sizes, the network address, how long the machine has been up — but they leave readable a counter that simply ticks once per processor cycle since the machine last booted. Read that counter, note the wall-clock time at the same instant, and dividing gives the moment the machine booted, which differs between machines and therefore identifies one. That makes it cheap to tell whether two instances share a host, which in turn makes it possible to study the placement algorithm at scale. Doing so reveals that two ordinary scheduling optimizations — spreading instances widely under sudden load, and clustering an account's work together — can be used to reach a large fraction of the machines in a datacenter. The authors disclosed to the provider and propose mitigations, including one already present in hardware that costs nothing.

Why this problem is worth solving

The attacks this enables are well known; the step they all depend on is not, so it is worth being precise about the division.

A MICROARCHITECTURAL SIDE CHANNEL extracts information from a program that never intentionally reveals it, by observing how that program disturbs hardware which the observer also uses. Caches are the classic example — if a victim looks up a table entry chosen according to a secret, it displaces something from a shared cache, and an attacker who notices which of their own data became slower to access learns something about the secret. The same works through other shared structures. These attacks are treated as a serious threat in clouds precisely because hardware is shared among parties who do not trust each other. What is often left implicit is that every one of them has TWO steps. The second, EXTRACTION, is the part that gets studied — the clever measurement that turns timing differences into a key. The first, CO-LOCATION, is getting your code onto the same physical machine as the victim in the first place, without which there is no shared hardware to observe and no attack at all. Almost all research addresses the second step while assuming the first, which is why understanding how hard the first actually is matters for judging real risk

The environment where this now has to happen is one that removed the attacker’s control on purpose.

In the SERVERLESS model a customer supplies a small self-contained unit of code packaged in a container, and the provider runs copies of it as demand requires, creating instances when requests arrive and destroying them after a period of idleness. The customer specifies almost nothing about placement — unlike renting a whole virtual machine, where a zone and a processor model can be chosen, here an ORCHESTRATOR decides which physical host each instance lands on, scoring hosts by criteria such as current utilisation and load balancing. Because customers run arbitrary code, the platform isolates them, either with a layer that intercepts and emulates the operating system calls a container makes, or by running each instance inside a lightweight virtual machine. Both are careful to conceal facts that would identify the underlying host — the processor model name and cache sizes are faked, the network address and machine uptime are virtualized. The design intent is clear, which is that a tenant should not be able to tell what hardware it is on or who else is there

. The question this paper asks is whether that intent is actually achieved.

What people did before, and where it broke

Locating neighbours by probing the network. The original technique, from work on early cloud platforms, inferred co-location from network characteristics that machines sharing a host had in common. Effective at the time and the reason the problem was taken seriously.
Tenant network isolation closed that route. Modern clouds logically separate each account’s networking, so an account cannot observe or probe another’s instances, which made the network-based techniques obsolete rather than merely harder.
Pairwise testing with a covert channel. The replacement — take two of your own instances, have both stress a shared hardware resource simultaneously, and detect interference. This gives ground truth rather than inference and works despite network isolation.
But pairwise testing is quadratic. It examines one pair at a time, and a serverless attacker may need thousands of instances to reach a victim, so the number of pairs becomes prohibitive. The technique that works cannot scale to the setting where it is now needed.
Fooling the placement algorithm. More recent work tries to influence scheduling, either by generating fake resource-usage patterns to mislead a learning-based scheduler, or by requesting the same configuration the victim requested. The first was evaluated only on a small private cluster, and this paper measures the second and finds it produces no significant benefit on the platform studied.
Fingerprinting machines by clock skew. A separate line of work identifies remote devices from the gradual drift of their clocks, observed through network timestamps. That no longer works where clocks are synchronised over the network and the available timestamps are far too coarse.
The gap. Nobody had a way to check co-location cheaply enough to do it at the scale a modern datacenter requires, and consequently nobody had studied how a serverless platform actually places instances — so whether co-location was a matter of luck or something arrangeable was simply unknown. That is the gap this paper fills.

How the placement policy was measured

Find a host identifier, use it to verify co-location cheaply, then study the scheduler

Co-location cannot be checked at scale

Network probing is defeated by tenant isolation, and testing pairs of instances for interference is quadratic, so verifying which of thousands of instances share hosts is prohibitively expensive. Without that, the platform placement algorithm cannot be studied at all.
↓ →
Notice what the sandbox does not conceal new

The isolation layers carefully fake every named piece of host identity — processor model, cache sizes, network address, uptime. But a counter that ticks once per processor cycle since the machine booted is readable by a single ordinary instruction, and nobody had treated a cycle counter as an identifier.
↓ →
Turn the counter into a machine fingerprint new

Reading that counter while noting the wall-clock time, and dividing by the counter rate, gives the moment the host last booted. Machines boot at different moments, so this value identifies a host, and rounding it absorbs measurement noise so repeated readings agree.
↓ →
Account for the fingerprint going stale new

The counter rate the platform reports is slightly wrong, so the derived boot moment drifts steadily further from the truth, giving each fingerprint an effective expiry. Measuring the rate directly avoids the drift but proved unreliable on about a tenth of hosts, so the reported rate is used and the expiry characterised.
↓ →
Verify co-location cheaply and at scale new

With a fingerprint per instance, instances can be grouped by apparent host and only groups need confirming, replacing pairwise testing with something affordable across thousands of instances — which is what finally makes studying the scheduler possible.
↓ →
Measure what the orchestrator actually does new

Systematic experiments reveal two ordinary optimisations with security consequences. Under a sudden surge the platform spreads instances of one function across many hosts, and instances belonging to one account tend to land on a set of hosts characteristic of that account.

Measured on a major commercial serverless platform across three United States datacenters, observing at least 1702 distinct hosts in the largest of them. The findings were disclosed to the provider in August 2023 and the implementation was released openly.

The idea

A sandbox hides the identifiers someone thought of, which is not the same as hiding identity. The isolation layers here are carefully built and do exactly what they set out to do. They fake the processor model name, conceal cache sizes, virtualize the network address and the machine’s uptime — every value that obviously announces which host you are on. What they leave alone is a counter that increments once per processor cycle and resets when the machine boots, because it looks like a clock rather than an identity. But a value that started at zero at a moment unique to each machine is an identifier, if you know to combine it with the current time. The lesson generalises well beyond clouds: enumerating the channels you know about leaves you covered exactly as far as the enumeration went, which is the same completeness-by-enumeration-is-not-completeness lesson covered elsewhere on this site, where per-bug-class runtime checks leave whole categories uncovered because nobody knew how to write a check for them.

Cheap verification is what turns an anecdote into a study. The interesting contribution is arguably not the fingerprint but what it enables. Establishing whether two instances share a host used to require making them interfere with each other, one pair at a time, which is fine for demonstrating that co-location is possible and useless for characterising a datacenter containing thousands of machines. Once a host can be identified by a value each instance computes locally, instances can simply be grouped, and the cost of surveying the whole system collapses. That is what makes it possible to ask the question nobody had answered — not whether co-location can happen, but how the provider’s placement algorithm behaves and whether its behaviour is exploitable. Reducing the cost of a measurement until a previously unaskable question becomes routine is the same build-the-instrument move covered elsewhere on this site, where supplying the missing measurement capability is the contribution and the findings follow from it.

The exploitable behaviours are good engineering, not mistakes. What the study finds are two entirely sensible optimisations. Spreading instances of one function across many hosts when demand surges is load balancing and fault tolerance working correctly — you would not want every copy of a service on one machine. Clustering an account’s instances onto a characteristic set of hosts reduces communication cost between services that talk to each other, which is exactly what an account’s services tend to do. Neither is a bug. But the first means an attacker who generates a burst gets spread across a large fraction of the datacenter, and the second means the search can be narrowed. That a feature built for reliability or efficiency becomes a capability for someone else is a pattern covered elsewhere on this site wherever behaviour that is entirely permitted, and even desirable, turns out to have consequences its designers were not reasoning about.

From a cycle counter to a datacenter map

How the study measured co-location cheaply enough to survey a whole datacenter. Numbers from the paper.

Step 1 of 5

What the sandbox is protecting

A serverless instance runs inside an isolation layer designed so that a tenant cannot tell what hardware it is on. The processor model name is faked, cache sizes are concealed, the network address and the machine uptime are virtualized. Every value that plainly announces a host has been considered and handled.

two isolation designs are used, one intercepting and emulating operating system calls and one running each instance in a lightweight virtual machine
both conceal host details, and the technique works against both

Why it works

Every shared-hardware attack needs the attacker and victim on one machine → so the practical risk depends on how hard that is, not only on how good the measurement is → and nobody had studied it in the environment where placement is chosen by the provider.
The isolation layers conceal each named host attribute → but a counter reset at boot is an identifier rather than a clock → so combining it with wall-clock time yields a per-machine value that passes straight through both sandbox designs.
Confirming co-location used to require making two instances interfere → which is quadratic and unaffordable across thousands → whereas a locally computed fingerprint lets instances be grouped, making a datacenter-scale survey cheap.
Spreading a function’s instances across hosts under load is correct load balancing → but it also distributes an attacker’s instances widely on demand → turning a reliability feature into reach.
Clustering an account’s instances reduces communication cost between related services → but it also makes placement predictable per account → which is why measured coverage differs depending on whose instances are being targeted.

What it costs (trade-offs and scope)

The findings are specific to one platform at one time. All measurements are of a single commercial serverless platform in three datacenters, and placement algorithms are internal, undocumented and subject to change, so the specific behaviours may not persist or generalise.
The fingerprint is imperfect and the paper says so. The derived boot moment drifts because the reported tick rate is slightly wrong, giving each fingerprint a limited lifetime, and the alternative of measuring the rate directly was unstable on about a tenth of hosts tested.
Co-location is necessary but not sufficient. Sharing a machine is the precondition for a shared-hardware attack, not the attack itself, and the extraction step brings its own difficulties that this work does not address.
A previously proposed technique did not reproduce. Requesting the same configuration as the victim, suggested by earlier work as a way to improve co-location odds, produced no significant benefit on the platform measured — reported as a negative result.
The defences carry real costs, unevenly. Trapping the counter instruction works but forces applications into the kernel for high-precision timing, which matters for real-time processing, databases using fine-grained timestamps, distributed synchronisation and heavy logging — one database is reported to have gained 43 percent on write latency by moving to this counter, which such a defence would give back.

The evaluation

The study was carried out on a major commercial serverless platform across three United States datacenters, using both of its isolation environments. Fingerprint accuracy was validated against interference-based ground truth rather than assumed, over 586 hosts. Placement behaviour was then characterised through systematic experiments varying container size, launch timing, repeated launches, and account identity, with instance lifetimes and idle termination measured directly. Reach was reported as the fraction of a target account’s instances that ended up sharing a host with the authors’ own, and the monetary cost of the experiments was recorded.

Establishing co-location in a public cloud

Property	Network probing	Pairwise interference testing	Counter-derived fingerprints
Works under tenant isolation
Cost of checking many instances
Enables studying placement policy
Defeated by the sandbox
Result reported

With cheap verification available, the placement policy could be characterised for the first time: a demand surge spreads a function's instances across many hosts, and an account's instances cluster on a characteristic set. Reach varied by target account and datacenter, and requesting the victim's configuration — proposed by earlier work — gave no significant benefit here.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Read this as a measurement of one platform at one moment. Placement algorithms are internal, undocumented and change over time, so the specific behaviours observed need not persist or generalise to other providers. The fingerprinting technique is imperfect and the authors characterise its limits rather than glossing them — the derived boot moment drifts because the reported counter rate is slightly inaccurate, giving each fingerprint a limited useful lifetime, and the alternative of measuring the rate directly proved unstable on about a tenth of the hosts tested. Co-location is also only the precondition for a shared-hardware attack rather than the attack itself, and the extraction step has its own substantial difficulties this work does not address. One earlier proposal, requesting the same configuration as the victim, produced no significant benefit here, which is reported as a negative result. Most importantly, the paper follows responsible disclosure — the findings went to the provider in August 2023, who assigned them to their trust and safety team — and devotes real space to mitigations, noting honestly that the straightforward defence of trapping the counter instruction imposes a genuine cost on applications needing high-precision timing. Within that scope the lessons travel well: a sandbox that conceals every identifier someone enumerated is not the same as one that conceals identity; reducing the cost of a measurement can convert an unanswerable question into a routine one; and an optimisation built for reliability or efficiency can become a capability its designers were not reasoning about.

Background

Enumerated defences leave gaps, and cheap measurement changes what can be asked

This paper sits in the site’s security and cloud cluster and rhymes with several ideas covered elsewhere on this site. Its central observation, that isolation layers conceal each host attribute someone thought to name while leaving readable a counter that is an identifier in disguise, is the enumeration-is-not-coverage lesson the compiler-testing page teaches when it shows that per-class runtime checks stop exactly where the design effort stopped and leave whole categories uncovered because nobody knew how to write a check for them. Its most consequential contribution is arguably methodological rather than adversarial — making co-location verification cheap enough to survey a datacenter is what allowed placement policy to be characterised at all, which is the same supply-the-missing-instrument move the open-hardware graphics page makes when a question cannot be asked until the measurement exists. Its finding that the exploitable behaviours are sensible optimisations, spreading instances for fault tolerance and clustering an account’s work for communication efficiency, is the same permitted-behaviour-becomes-a-problem pattern the persistency-race and compiler-testing pages examine, where nothing is a bug and the consequence is still real. And its most encouraging note belongs to a pattern the site sees often on the constructive side — the mitigation that costs nothing already exists in hardware, in counter virtualization features built for moving running virtual machines between hosts, which is the same use-what-is-already-there observation the storage and cryptography pages make when relief for a bottleneck turns out to be sitting idle nearby.

If you remember only three things

1. Attacks that read secrets by watching a victim disturb shared processor hardware all require a first step that receives far less study than the attacks themselves — the attacker's code must run on the same physical machine as the victim's. That step became hard for three reasons: cloud networks now isolate tenants, defeating the older technique of finding neighbours by probing; datacenters grew large enough that landing on a specific machine by chance is remote; and serverless platforms removed the remaining control, since the provider decides placement, instances appear and vanish with demand, and customers cannot choose a zone or processor. Whether co-location remained practical was simply unknown.
2. The paper finds that the isolation layers conceal every host attribute anyone thought to name — processor model, cache sizes, network address, uptime — while leaving readable a counter that ticks once per processor cycle and resets when the machine boots. Read together with wall-clock time, that yields the moment the host last started, which differs between machines and therefore identifies one. The fingerprint is imperfect, drifting because the reported tick rate is slightly wrong, but it is cheap — and cheapness is the real contribution, because confirming co-location previously meant making two instances interfere one pair at a time, which cannot survey thousands. Cheap verification is what finally made the provider's placement algorithm studiable.
3. That study found two ordinary optimisations with security consequences: a demand surge spreads a function's instances across many hosts, which is load balancing working correctly, and an account's instances cluster on a characteristic set of hosts, which is affinity scheduling reducing communication cost. Neither is a bug. In the largest datacenter at least 1702 hosts were observed and 904 reached at once for about 23 dollars. The authors disclosed to the provider and propose mitigations, including hardware counter-virtualization features that already exist for live machine migration and cost nothing. The transferable lessons: concealing every enumerated identifier is not concealing identity; cheaper measurement can make an unanswerable question routine; and an optimisation built for reliability can become a capability.

Check yourself

Check your understanding

1. The sandbox conceals the processor model, cache sizes, network address and machine uptime, yet a host can still be identified. Predict what was overlooked.
2. The paper argues that cheap co-location verification matters more than the fingerprint itself. Predict why.
3. The two exploitable placement behaviours are described as sensible optimisations rather than bugs. Predict what they are and why that framing matters.
4. The paper proposes mitigations, one of which costs nothing. Predict what makes that one free while the obvious defence is expensive.

Sources

Everywhere All at Once: Co-Location Attacks on Public Cloud FaaS (ACM DL, DOI 10.1145/3617232.3624867) — ASPLOS 2024.
Author PDF — the full 17-page paper, read in full for this explainer.
Released implementation (GitHub) — the authors’ open-source release accompanying the paper.