Taurus: a data plane architecture for per-packet ML

Network switches forward a packet in nanoseconds but ask a millisecond-away server for every intelligent decision — Taurus builds a small SIMD machine-learning engine into the switch itself, so every single packet can pass through a neural network at line rate.

Why this problem is worth solving

A datacenter network has two brains. The data plane is the forwarding hardware in switches and network cards: it touches every packet and must keep up with line rate — the full speed of the wires, around a billion packets per second for a top-end switch chip. To go that fast it runs only simple, fixed logic. The control plane is software on servers: it sees sampled telemetry, runs rich data-driven policies, and configures the data plane — at the pace of software, milliseconds.

The paper’s motivating table lists what falls between these two stools: active queue management must mark or drop each packet; denial-of-service and intrusion detection must react before a flood of malicious packets gets through; congestion control and load balancing want decisions per packet or per flowlet. All of these are exactly the tasks where machine learning beats hand-written heuristics — learned congestion control and anomaly detectors have smoother, more accurate decision boundaries than rules people write.

The arithmetic of the gap is merciless. A round trip to a control server takes at least 10 microseconds; even the switch’s own CPU sits ~900 nanoseconds away across PCIe, and in that time a 12.8-terabit-per-second chip forwards about twelve thousand 128-byte packets. Unbatched neural- network inference takes 0.67 milliseconds on a server CPU (GPUs and TPUs are slower for single inputs: 1.15 and 3.51 milliseconds, dominated by setup). Every packet that passes unjudged during a SYN-flood attack opens another connection on some victim server. Speed without intelligence, or intelligence without speed — the paper’s phrase for the dilemma is that networks “must choose between per-packet, line-rate execution or computational complexity.”

What people did before, and where it broke

Cache the verdicts. Let servers infer and install results as flow rules in the switch’s match-action tables (MATs) — hardware lookup tables that match packet fields and apply actions. This works when network behavior is stable. It collapses when model inputs vary per packet (every packet is a cache miss), and each miss costs a control-plane round trip plus ~3 milliseconds of rule installation. It is also memory-hungry: matching the paper’s benchmark neural network’s behavior with rules would take 12 megabytes of table entries, while the model itself is 5.6 kilobytes of weights — a 2,135× difference.

Run ML on the MATs themselves. Programmable switches (the PISA architecture) execute a VLIW instruction set designed for header edits. It has no loops and no multiplication, and its all-to-all communication crossbars are expensive precisely where ML needs none of that flexibility. Binary neural networks have been forced through anyway — at tens of tables for imprecise models (one implementation needs ~12 MATs per layer; the paper’s anomaly detector would consume 48, versus an area equivalent of 3 in Taurus).

Bolt an accelerator beside the switch. GPUs and TPUs get their throughput from batching, and a switch cannot wait for a batch to fill — deterministic per-packet latency requires unbatched execution, where these accelerators are at their worst. A separate box also burns switch ports and duplicates parsing and match logic.

The diagnosis: ML’s computation is regular — many multiply-adds over few weights — and what is missing is not raw compute but the right abstraction sitting inside the pipeline, sharing its parsers and tables.

The idea

The right abstraction: MapReduce

Look at one neuron of a neural network: multiply each of its inputs by a weight (an element-wise map), sum the products into one number (an associative reduce), then apply an activation function such as ReLU (another map). Support-vector machines, K-means clustering, and the layers of deep networks are all built from this same pair of patterns. That pair — MapReduce — is Taurus’s programming abstraction: expressive enough for a family of models, restrictive enough to build ruthlessly efficient hardware for. The paper extends the P4 switch-programming language with a MapReduce control block, compiled through the Spatial language into a static dataflow configuration.

The abstraction pays a second dividend: the same block can run non-ML data-parallel tasks — sketches for flow-size estimation, consistent hashing for packet scheduling, even in-network gradient aggregation for distributed training.

Hardware shaped exactly to the job

The MapReduce block is a coarse-grained reconfigurable array (adapted from the Plasticine accelerator): a checkerboard grid of compute units and memory units joined by a static interconnect, clocked at 1 GHz to match the switch. Every generality that ML inference does not need was removed, and each removal is a design lesson:

• 8-bit fixed-point arithmetic, not floating point: quantizing the benchmark networks costs at most 0.07 percentage points of accuracy, while 16-bit hardware would double area and 32-bit would quadruple it.
• No DRAM at all: weights live in banked on-chip SRAM with single-cycle access, because a 100-cycle DRAM access cannot hide inside a nanosecond budget. This caps model size — a price, accepted knowingly.
• A fixed shape fitted to measured workloads: each compute unit has 16 lanes (the largest benchmark layer has 12 neurons, so a dot product unrolls into one unit) and 4 pipelined stages (linear algebra needs two — map then reduce; sigmoid and tanh approximations use more). The final grid is 12×10 units at a 3:1 compute-to-memory ratio, 4.8 mm².
• Static, compile-time parallelism: the compiler unrolls loops in space; throughput is deterministic — line rate, or a known fraction if the model is too big, never a surprise stall.

Feature engineering stays in the tables

The MATs around the block are not bystanders. Preprocessing tables accumulate per-flow state (count the urgent flags across a connection), convert categorical fields into numeric likelihoods via lookup tables, and even re-shape distributions (a logarithm turns an exponential feature into a uniform one) — classic feature engineering, done in existing hardware, which lets the neural network itself stay small and shallow. In-band network telemetry brings global state into the features, so a single switch can act on network-wide signals. Postprocessing tables convert the model’s score into an action — and, because operators do not fully trust learned models, hard safety and liveness bounds are compiled into those same tables, constraining what any model output can do.

headers -> packet header vector (PHV)
bypass? no — this flow is under inspection

Why it works

Because ML inference is regular linear algebra — many multiply-adds, few weights, no data-dependent control flow — it fits a SIMD pipeline with static routing, which costs a fraction of the area of the flexible VLIW crossbars in match-action stages. → Because ranking-quality decisions survive 8-bit arithmetic and the models are small, weights fit in on-chip SRAM, and every memory access is single-cycle — nanosecond budgets stay intact. → Because the block is deeply pipelined at 1 GHz, one packet enters and one verdict exits every cycle; per-packet latency (61–221 ns) and line-rate throughput coexist. → Because the surrounding MATs do feature engineering and decision interpretation, the neural network only spans the gap features cannot — keeping it shallow enough for the latency budget. → Therefore intelligence moves from milliseconds away to inside the forwarding path: versus a control plane that judges sampled packets 0.67+ milliseconds late and installs rules 3 milliseconds later, a Taurus switch judges every packet as it passes — for 3.8% more silicon.

What it costs

• Model size is capped by SRAM. No DRAM means no large models; the authors point to compression, pruning and distillation as the path, and note control models are shrinking (structured control nets do with four neurons per layer what once took 512).
• Not everything reaches line rate. The Indigo congestion-control LSTM takes 805 ns — a decision every ~800 packets, not every packet (still ~12,000× fresher than its 10-millisecond software deployment). A 1-D convolution maps poorly to the vector shape and needs 8× unrolled area to hit line rate.
• Trust must be engineered around the model. ML outputs are probabilistic; Taurus confines them to heuristic-tolerant tasks and bounds them with compiled safety rules rather than verifying the models.
• Training stays slow and central. The data plane reacts instantly only to learned event types; genuinely new behavior needs control-plane retraining (tens to hundreds of milliseconds in the testbed) before the switch can recognize it.
• The ASIC is projected, not fabricated. Area, power and latency come from synthesis with a predictive 15-nanometer library; the end-to-end system runs the MapReduce block on an FPGA beside a real Tofino switch.

The evaluation

Two levels: ASIC synthesis of the MapReduce block against real networking models, and an end-to-end testbed — a Barefoot Tofino switch implementing the MAT stages, connected by 100-gigabit Ethernet to a Xilinx Alveo U250 FPGA emulating the MapReduce block, with an 80-core server running the state-of-the-art control-plane stack (XDP capture, InfluxDB, TensorFlow, ONOS rule installation) as the baseline.