Neuralink: Fast on-Device LLM Inference with Neuron Co-Activation Linking

Running a full-accuracy large language model on a phone means keeping the whole model in flash and, for each token, loading into memory only the small set of neurons that actually fire — but those active neurons are scattered all over flash, so fetching them takes many tiny random reads, and a phone's flash is limited not by bandwidth but by how many read operations per second it can do (IOPS), so the reads become the bottleneck; this paper (Tsinghua and Microsoft Research — Tuowei Wang, Ruwen Fan, Minxing Huang, Zixu Hao, Kun Li, Ting Cao, Youyou Lu, Yaoxue Zhang, Ju Ren) presents Neuralink, which rearranges where neurons live in flash so that neurons which tend to fire together sit next to each other — linked into contiguous runs read in one continuous access — using an offline clustering pass that frames placement as finding a path through a co-activation graph and an online pass that merges nearby reads and caches whole linked segments, cutting end-to-end latency by 1.49 times on average over the state of the art.

The argument in one glance

Problem

A phone cannot hold a full LLM in memory, so it streams the needed neurons from flash

Large language models are far too big for a smartphone's memory, and shrinking them into lightweight variants that do fit costs accuracy. A clever way to keep full accuracy is to exploit activation sparsity: in each layer of an LLM, only a small fraction of the neurons actually fire for any given token, so you can store the entire model in the phone's roomy flash storage and, at each step, load into memory only the neurons a predictor says will activate. Memory stays small, accuracy stays full. The catch is that fetching those neurons from flash on demand is now on the critical path of every token, and how fast you can read them decides how fast the model runs.
What people did

Store the full model in flash and load only the activated neurons each step

The sparsity-based approach, exemplified by systems like LLM-in-a-flash, does exactly this: keep all parameters in flash, use a predictor to decide which neurons will activate, read just those neurons into memory, and compute. It adds engineering such as bundling related rows and columns of the weight matrices together so a related group can be read at once, and it works well enough to make on-device inference feasible at all.
Why it fell short

Scattered neurons plus a smartphone's IOPS limit make I/O the bottleneck

The paper pins down what actually throttles this scheme, and it is not bandwidth. Smartphones use Universal Flash Storage, and its performance for this workload is limited by IOPS — the number of separate read operations it can do per second — not by how many bytes per second it can stream; the paper measures that bandwidth grows almost linearly with the size of each continuous read, meaning the real cost is per-operation. The activated neurons, however, are placed arbitrarily across flash, scattered with no regard to which fire together, so loading the set that a token needs takes many tiny, separate random reads. That flood of small operations slams straight into the phone's IOPS ceiling, and I/O comes to dominate inference latency. The misalignment between scattered activation patterns and the hardware's preference for few large reads is the whole problem.
This paper's idea

Place co-activated neurons next to each other so one continuous read fetches them all

Neuralink attacks the IOPS bottleneck by optimizing where neurons live in flash, a dimension prior work left untouched. Its central concept is Neuron Co-Activation: neurons that frequently fire together are linked and placed contiguously, so the set a token needs can be pulled in with one continuous read instead of many scattered ones. This has two stages. Offline, a correlation-aware clustering pass profiles which neurons co-activate, casts the placement problem as a graph — neurons are nodes, co-activation frequency weights the edges — and searches, via a Hamiltonian-path formulation solved greedily, for an ordering that puts frequently co-firing neurons adjacent in flash. Online, a continuity-centric pass refines this at run time: an IOPS-friendly access collapse inserts a few extra neurons between two nearby links so their reads merge into one, and a linking-aligned cache policy caches whole linked segments rather than individually hot neurons, keeping accesses continuous. Across smartphones and LLMs, Neuralink cuts end-to-end latency by 1.49 times on average over the state of the art.

Why this problem is worth solving

Getting a capable LLM to run locally on a phone, without giving up accuracy, is a goal with real stakes for privacy, cost, and offline use.

A large language model has billions of parameters, organized mostly into the weight matrices of each transformer layer, and holding all of them in a smartphone’s small memory is impossible. One response is to build a smaller model, but fewer parameters generally means lower accuracy, so you are trading away the very quality that made the model worth running. Activation sparsity offers a way out that keeps the full model. The observation is that in the feed-forward blocks of an LLM, for any single token only a small fraction of the neurons produce a nonzero output — the rest contribute nothing to that token’s result. So you do not need all the weights in memory at once; you need only the ones that will activate for the token you are processing. That makes a different arrangement possible: store the complete, full-accuracy model in the phone’s flash storage, which is large, and at each step load into scarce memory just the handful of neurons that fire. A predictor decides in advance which neurons those are. This preserves accuracy while shrinking the memory footprint dramatically, which is why it is the leading route to on-device inference — but it converts a memory problem into a storage-access problem, because now the model’s speed depends on how quickly the right neurons can be fetched from flash on every token.

The reason this storage-access problem is subtle is that smartphone flash behaves very differently from the disks most systems reasoning is tuned for.

When people think about storage speed they usually think about bandwidth — how many megabytes per second can flow. But for the pattern of reads an on-device LLM generates, the smartphone’s Universal Flash Storage is not limited by bandwidth at all; it is limited by IOPS, the number of independent input-output operations it can perform per second. The paper demonstrates this directly: as you make each continuous read larger, the effective bandwidth rises almost linearly, which tells you the storage is not saturating its data pipe — it is spending its time on the fixed per-operation overhead of issuing and servicing each separate read. In other words, ten small reads of a kilobyte each are far more expensive than one read of ten kilobytes, even though the bytes moved are the same. This flips the usual optimization target. The goal is no longer to move fewer bytes; it is to issue fewer, larger, continuous operations. And that is exactly what the naive sparsity approach fails to do, because the neurons a token needs are scattered across flash, forcing one small operation per scattered neuron. Recognizing that the enemy is operation count, not data volume, is what reframes the entire design toward controlling where data sits so that reads can be made continuous.

What people did before, and where it broke

Deploy a lightweight LLM that fits in memory. Use a smaller model designed for mobile so everything sits in DRAM. Where it broke. Fewer parameters means degraded accuracy, giving up the quality that motivated running an LLM in the first place.
Quantize and compress the model. Shrink the weights with low-bit quantization so the model fits or nearly fits. Where it broke. Compression helps the footprint but does not, by itself, solve serving a model still too large for memory, and aggressive compression again erodes accuracy; the storage-access pattern is left untouched.
Sparsity-based flash offloading. Keep the full model in flash and load only the activated neurons per token, as LLM-in-a-flash does, with row-column bundling to group related reads. Where it broke. The activated neurons are scattered across flash, so each token still triggers many small random reads, and on a smartphone limited by IOPS this flood of operations dominates latency.
Push flash bandwidth or add more caching of hot neurons. Try to read faster, or cache the most frequently used neurons individually. Where it broke. The bottleneck is operation count, not bandwidth, so faster streaming does not help; and caching the individually hottest neurons ignores whether they are contiguous, so the reads that remain are still fragmented.
The gap. No one optimized where neurons are placed in flash. Under sparsity, the arrangement of neurons in storage was arbitrary, so nothing made the neurons that fire together also sit together — which is the one change that turns many scattered reads into a few continuous ones and relieves the IOPS bottleneck.

How Neuralink works

Rearranging neurons in flash so co-firing ones are read together

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Change where data lives, not how fast you read it. The pivotal move in Neuralink is to treat neuron placement in flash as the thing to optimize, which nobody under the sparsity paradigm had done. Prior systems accepted the neurons’ arbitrary layout and tried to cope with the scattered reads it produced; Neuralink instead rearranges the neurons so that the ones a token needs are physically adjacent, converting many small random reads into a few large continuous ones. Because the smartphone’s flash is bottlenecked by the number of operations, not the bytes, making reads continuous is worth far more than making them faster or smaller. Placing data that is accessed together into the same physical unit of storage, so a single read returns many useful items instead of one, is the same principle the work on packing co-accessed embeddings onto shared storage pages covered elsewhere on this site is built on, arranging storage layout around access patterns rather than accepting whatever layout the data happened to have. Neuralink applies that principle to LLM neurons and to the operation-count limit of phone flash.

Let the co-activation structure decide the layout, via a graph. To place co-firing neurons together you first have to know which neurons co-fire, and then find an ordering that keeps as many co-firing pairs adjacent as possible. Neuralink profiles co-activation offline and encodes it as a weighted graph, then recognizes that the ideal flash ordering is a path through that graph visiting every neuron while favoring heavy edges — a Hamiltonian path — and solves it greedily by repeatedly merging the most strongly linked pieces. This is a clean separation: the expensive analysis and reorganization happen once, offline, and the phone at run time simply reads from the improved layout. Streaming just the needed pieces of a large model from storage onto a constrained device, and arranging in advance how those pieces are staged so the device is not starved, is the same discipline the work on turbocharging edge NLP inference by elastic pipelining from storage covered elsewhere on this site pursues, planning the movement of model parameters around the device’s limits. Neuralink’s twist is that the plan is a physical placement chosen to maximize continuous reads.

Make caching respect the layout, not fight it. Placement alone is not enough, because a poor caching policy can re-fragment access: if you cache the individually hottest neurons, the reads that remain skip over the cached ones and become discontinuous again. Neuralink’s linking-aligned cache instead keeps whole linked segments resident, so caching and reading cooperate to preserve continuity, and its access-collapse trick spends a few extra bytes to weld two nearby reads into one. The common thread is a cache whose organization mirrors the structure of the data it serves rather than a generic hottest-first rule. Designing a cache around the shape of the access pattern — keeping things resident in the units they are actually traversed in — is the same insight the work on caching multi-level index structures by their key ranges covered elsewhere on this site is built on, tailoring what is cached to how the data is walked. Neuralink tailors its cache to the linked runs of neurons its placement creates.

Following one token through Neuralink

Fetching the neurons for one token from a reorganized flash layout

A trace, drawn from the paper's description, of how Neuralink serves one decoding step. The offline reorganization has already run; here the phone is generating a token.

Step 1 of 5

The predictor names the neurons this token needs

As the model processes a token, an activation predictor determines which neurons in the current feed-forward block will fire. Only these need to be brought from flash into memory; the rest of the layer stays on storage and is never read for this token.

This is the activation-sparsity foundation: full model in flash, only the active neurons loaded.

Why it works

Continuous reads beat the IOPS ceiling. A smartphone’s flash is limited by operations per second, not bytes per second → but placing co-activated neurons adjacently turns many small reads into a few large continuous ones → so the same neurons arrive with far fewer operations and the IOPS bottleneck loosens.
Co-activation is stable enough to plan around. If which neurons fire together changed every token, no fixed layout could help → but co-activation patterns are consistent enough to profile offline → so a single reorganized placement serves many future tokens well.
The graph formulation captures the real objective. Placing neurons well means keeping the most-co-firing pairs adjacent, a global ordering problem → but modeling it as a weighted graph and a Hamiltonian path expresses exactly that → so a greedy search on it produces layouts aligned with actual access.
Access collapse trades cheap bytes for scarce operations. Two nearby links would normally cost two separate reads → but reading the few neurons between them merges the pair into one operation → so a tiny amount of extra data eliminates a costly I/O operation on the critical path.
A layout-aware cache preserves the gains. Caching the individually hottest neurons would leave gaps that re-fragment the reads → but caching whole linked segments keeps resident and fetched data contiguous → so the online policy reinforces the offline placement instead of undermining it.

What it costs (trade-offs and scope)

It needs an offline profiling and reorganization pass. Neuralink must observe co-activation and compute a new placement before deployment, which is extra work and depends on the profiling being representative of real usage; a workload whose activation patterns drift far from the profile would see reduced benefit until re-profiled.
The gains assume co-activation is consistent. The whole approach rests on the same neurons tending to fire together across tokens. Inputs or models with highly variable, low-structure activation would offer fewer stable links to exploit, weakening the continuity a fixed placement can achieve.
Access collapse spends extra bandwidth to save operations. Reading in-between neurons to merge two reads moves data that is not needed for the token. This pays off only because the flash is operation-limited rather than bandwidth-limited; on hardware where bandwidth were the true constraint, the trade would be less favorable.
It targets the sparsity-plus-flash regime specifically. Neuralink is built for on-device inference that keeps the full model in flash and loads activated neurons; models small enough to fit fully in memory, or serving setups not bottlenecked by flash IOPS, are outside its intended scope.
The reported figure is an average over the tested devices and models. The 1.49 times end-to-end latency improvement is an average across the smartphones and LLMs evaluated against the state-of-the-art baseline; the benefit varies with the device’s IOPS characteristics, the model’s sparsity, and the caching budget, and specific settings would differ.

The evaluation

Serving a sparse on-device LLM from smartphone flash

Property	Lightweight in-memory LLM	Sparsity offloading (prior)	Neuralink
Model accuracy
Neuron placement in flash
Reads per token
Effect of IOPS limit
Reported outcome

A qualitative comparison of ways to run an LLM on a smartphone. Neuralink keeps the full accuracy of the sparsity approach while cutting end-to-end latency by 1.49 times on average over the state-of-the-art baseline, by making flash reads few and continuous rather than many and scattered.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and its arXiv version (arXiv:2410.19274), which describe the mechanism in full: activation sparsity keeps the full model in flash and loads only activated neurons; the bottleneck on smartphones is IOPS, not bandwidth, because effective bandwidth rises near-linearly with continuous read size; Neuralink optimizes neuron placement in flash via the concept of Neuron Co-Activation; an offline correlation-aware clustering stage extracts co-activation patterns, abstracts placement as a complete graph and a Hamiltonian-path search, and solves it with a greedy algorithm; an online continuity-centric stage adds IOPS-friendly access collapse (inserting a few neurons between links to merge reads) and a linking-aligned cache policy (caching linked segments rather than lone hot neurons). The headline figure is the paper’s own: 1.49 times lower end-to-end latency on average versus the state-of-the-art baseline (LLM-in-a-flash, which the authors port into llama.cpp for comparison). Details such as the exact predictor, the specific smartphones and models, the profiling procedure, and the ablation results should be read from the paper; the explanations here of why IOPS dominates and why continuity helps follow the paper’s own analysis.

Background

Why on-device LLM inference keeps pushing on storage

It helps to see Neuralink as part of a broader push to run capable models on hardware that was never meant for them. Phones, wearables, and other edge devices have small memory and modest compute, yet the demand to run large models locally — for privacy, latency, offline use, and cost — keeps growing, so a whole line of work asks how to squeeze a big model onto a small device without giving up too much. Some of that work compresses the model itself; some redesigns the compute; and some, like Neuralink, treats the device’s storage hierarchy as the lever, because once a model is too big for memory the question becomes how cheaply you can move the right pieces in from flash at the right moment. What makes the storage angle sharp on phones is that their flash has its own peculiar performance profile, dominated by operation count, so the general problem of fitting a model to a device turns into the specific problem of shaping data movement to the storage hardware’s quirks. That same ambition of running heavy transformer inference within a tight edge hardware budget is what the work on 8-bit transformer inference and fine-tuning for edge accelerators covered elsewhere on this site pursues, adapting the model and its execution to what small hardware can afford. Neuralink pursues it from the storage side, reorganizing where parameters sit so the phone’s flash can feed inference fast.

If you remember only three things

1. Running a full-accuracy LLM on a smartphone is possible via activation sparsity: keep the whole model in flash and, per token, load only the small set of neurons that actually fire, so memory stays small and accuracy stays full. But this makes flash reads the critical path, and the activated neurons are placed arbitrarily in flash, so each token triggers many small scattered random reads.
2. The real limit on smartphone flash is IOPS — operations per second — not bandwidth, so the flood of tiny scattered reads, not the bytes moved, is what dominates latency. Neuralink optimizes neuron placement in flash using Neuron Co-Activation: it profiles which neurons fire together, casts placement as a co-activation graph and a Hamiltonian-path search solved greedily, and reorganizes flash so co-firing neurons are contiguous, so one continuous read fetches many needed neurons. An online stage merges nearby reads (access collapse) and caches whole linked segments (linking-aligned cache) to keep accesses continuous.
3. Across a range of smartphones and LLMs, Neuralink cuts end-to-end latency by 1.49 times on average over the state-of-the-art baseline, as the first solution to optimize storage placement under sparsity. It requires an offline profiling and reorganization pass, assumes co-activation patterns are consistent, trades a little extra bandwidth for fewer operations, and targets the sparsity-plus-flash on-device regime specifically.

Check yourself

Check your understanding

1. Under the sparsity-based approach, what actually bottlenecks LLM inference on a smartphone?
2. What is Neuralink's core idea?
3. How does Neuralink's online stage keep flash accesses continuous at run time?
4. What did Neuralink achieve, and what are its main assumptions?

Sources

The paper is “Neuralink: Fast on-Device LLM Inference with Neuron Co-Activation Linking” (arXiv version titled “Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linking”) by Tuowei Wang, Ruwen Fan, Minxing Huang, Zixu Hao, Kun Li, Ting Cao, Youyou Lu, Yaoxue Zhang, and Ju Ren (Tsinghua University and Microsoft Research), published at ASPLOS 2025, DOI 10.1145/3676642.3736114. It is licensed CC BY-NC-SA.

This explainer is written from the paper’s abstract and its arXiv preprint (arXiv:2410.19274), which describe the mechanism in full: large language models exceed smartphone memory and compute; lightweight models fit but lose accuracy; sparsity-based techniques keep the full model in flash and transfer only activated neurons to memory, preserving accuracy but incurring many I/O operations that overwhelm smartphones with severe IOPS constraints. The paper identifies IOPS, not bandwidth, as the bottleneck, attributing it to the misalignment between scattered activation patterns and flash hardware. Neuralink optimizes neuron placement in flash via Neuron Co-Activation: an offline correlation-aware clustering stage that extracts co-activation patterns (including the bundling of gate, up, and down projections in the feed-forward blocks), abstracts placement as a complete graph, reformulates it as a Hamiltonian-path search, and solves it with a greedy priority-queue algorithm; and an online continuity-centric stage that adds IOPS-friendly access collapse (inserting neurons between links to merge reads) and a linking-aligned in-memory cache policy (caching linked segments rather than lone hot neurons). The reported result is an average 1.49 times end-to-end latency improvement over the state of the art across a range of smartphones and LLMs. No numbers beyond the paper’s have been introduced; exact experimental setups, models, devices, and ablations should be confirmed in the paper.