TNIC: A Trusted NIC Architecture: A hardware-network substrate for building high-performance trustworthy distributed systems

Dimitra Giantsidi, Julian Pritzi, Felix Gust, Antonios Katsarakis, Atsushi Koshiba and Pramod Bhatotia

To run distributed systems safely in a cloud where any machine might be buggy or malicious, the usual fix is to lock secrets inside the CPU's trusted enclave — but those enclaves differ from chip to chip, are huge and impossible to fully verify, and are slow on the network; this work from TU Munich moves the root-of-trust off the CPU and onto the network card, building a tiny, formally-verified trusted circuit on a SmartNIC that stamps and checks every message, letting cheap crash-tolerant protocols survive malicious faults at up to six times the throughput of enclave-based systems.

The argument in one glance

Problem

Making distributed systems survive malicious faults, cheaply

In an untrusted cloud, machines can fail arbitrarily or maliciously — Byzantine faults. Tolerating them lets systems stay correct, but classic Byzantine protocols are complex, slow, and need many extra replicas, so most systems settle for crash-fault tolerance and are unsafe if a node lies or is compromised.
What people did

Add trust with CPU hardware enclaves (TEEs)

Use a CPU trusted execution environment, a hardware enclave such as Intel SGX, to provide the two security properties that let a crash-tolerant protocol be upgraded to a Byzantine-tolerant one without adding nodes.
Why it fell short

CPU enclaves are heterogeneous, huge, and slow

Enclaves differ from machine to machine, complicating correctness across a heterogeneous cloud. They force you to trust an enormous amount of code — millions of lines, tens of megabytes — which cannot be formally verified and widens the attack surface. And their network input and output is several times slower than untrusted networking.
This paper's idea

Move the root-of-trust to the network card (TNIC)

TNIC builds a minimal, formally verified silicon root-of-trust at the network interface, not the CPU. A tiny attestation kernel on an FPGA SmartNIC provides exactly two properties — transferable authentication and non-equivocation — by stamping and checking every message. Its trusted base is about 2,114 lines of hardware, a fraction of a percent of a TEE, so it can be formally verified. It is uniform across heterogeneous machines and up to 6 times faster than CPU-enclave systems.

Why this problem is worth solving

Running correctly in an untrusted cloud means surviving not just crashes but lies, and doing so is expensive.

A distributed system spreads work across many machines that coordinate over a network, and it must keep working when some of them fail. There are two models of failure. In the crash-fault model, a machine can only fail by stopping — it never sends a wrong or malicious message. Most cloud systems assume this, because crash-fault-tolerant protocols are fast and need relatively few replicas (2f+1 machines to tolerate f failures). But the assumption is optimistic: it treats the whole cloud as trusted. In reality, machines and networks can misbehave arbitrarily — corrupted by bugs, hardware faults, or an actual adversary — sending conflicting or forged messages. These are called Byzantine faults. Systems that tolerate them, Byzantine Fault Tolerant systems, remain correct even under malicious behavior, but classically they need many more replicas (3f+1 machines to tolerate f faults) and involve complex, slow protocols. As a result, despite the cloud genuinely being untrusted, Byzantine fault tolerance is rarely adopted in practice — it is seen as too costly and too complicated — and systems run in the fast-but-unsafe crash model instead.

There is a known shortcut to bridge the two, but the usual way to take it is a bad fit.

A foundational result says you do not need a full heavyweight Byzantine protocol: if the system has just two security properties, any crash-tolerant protocol can be transformed into a Byzantine-tolerant one without adding any nodes. The first property is transferable authentication — a node can verify who really originated a message, using an unforgeable token, even when the message was forwarded by someone else. The second is non-equivocation — a node cannot make conflicting statements to different peers, or replay stale messages, because each message is bound to a unique, monotonically increasing counter. Given these two, a crash-tolerant system can tolerate Byzantine faults with the same 2f+1 nodes it already uses. The established way to obtain the two properties is a trusted execution environment (TEE): a hardware enclave in the CPU, such as Intel SGX, that protects and attests code. But TEEs are a poor fit for networked systems. They are heterogeneous — different machines have different enclaves with different programming models and guarantees, so stitching them together is hard and error-prone. They have an enormous trusted computing base: the trusted code you must rely on for security includes an entire operating-system kernel, so a state-of-the-art TEE trusts tens of megabytes of code, which is impossible to formally verify and widens the attack surface. And their network input and output is slow, adding several times the latency of untrusted networking.

What people did before, and where it broke

Assume the cloud is trusted and use crash-fault tolerance. Run the fast, common crash-tolerant protocols with 2f+1 replicas. Where it broke. These are unsafe against arbitrary or malicious faults, so they cannot be trusted in a genuinely untrusted cloud.
Use full Byzantine Fault Tolerant protocols. Run classic Byzantine protocols that tolerate malicious faults directly. Where it broke. They need many more replicas (3f+1) and are complex and slow, so they are rarely adopted in practice.
Get the two security properties from CPU enclaves (TEEs). Use hardware enclaves like Intel SGX to provide transferable authentication and non-equivocation. Where it broke. Enclaves are heterogeneous across machines, carry an enormous unverifiable trusted computing base, and make network operations several times slower.
The gap. No approach provided the two security properties from a minimal, uniform, formally verifiable, and fast foundation. Building that foundation at the network interface is what TNIC does.

How TNIC works

From CPU enclaves to a tiny trusted circuit on the network card

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Move the root-of-trust off the CPU and onto the network card. TNIC’s central move is to stop relying on a CPU enclave and instead put a small trusted circuit on an FPGA-based SmartNIC, where it stamps and checks messages as they pass, and run the network stack in user space bypassing the operating-system kernel. This makes the trust independent of whatever CPU each machine has, and removes the costly enclave syscalls from the network path. Offloading the critical work onto dedicated network-card hardware and bypassing the kernel so the CPU is no longer in the fast path is the same move the direct-storage-access work covered elsewhere on this site makes, pushing the data path down into hardware and out of the kernel so operations run at the device’s speed rather than the operating system’s.

Shrink the trusted base until it can be proven correct. Rather than trusting an entire operating-system kernel inside an enclave — tens of megabytes that cannot be checked — TNIC provides only the two security primitives it actually needs, in an attestation kernel of about 2,114 lines of hardware code, a fraction of a percent of a TEE’s trusted base, and then formally verifies its safety and security with a prover. Keeping the thing you must trust small and precise enough to be machine-checked, rather than large and merely hoped-correct, is the same discipline the GPU memory-model work covered elsewhere on this site follows, writing the exact rules compactly enough that an automated tool can check real behavior against them rather than leaving correctness to informal argument.

Add a minimal primitive that upgrades a whole class of systems. With just transferable authentication and non-equivocation, TNIC transforms any cheap crash-fault-tolerant protocol into a Byzantine-tolerant one using the same number of nodes, so existing 2f+1 systems gain resilience to malicious faults without a heavyweight redesign. Introducing one small, well-defined mechanism that lets a system meet a much stronger guarantee than it was built for, without paying the full cost of that guarantee everywhere, is the same strategy the leaky-instruction hardening work covered elsewhere on this site takes, adding a targeted change that raises the whole program to a security property it did not previously have while leaving the rest of it untouched.

Following a message through TNIC

Sending and verifying an attested message in an untrusted cloud

A trace of TNIC's attestation kernel handling one message, following the paper's Attest and Verify functions.

Step 1 of 5

A node sends a message into a hostile cloud

A node in a distributed system wants to send a message to its peers, in a cloud where any machine might lie, corrupt data, or replay old messages.

The threat model is Byzantine: arbitrary, possibly malicious, behavior.

Why it works

The two properties, not a whole enclave, are what Byzantine tolerance needs. Because theory shows transferable authentication and non-equivocation suffice to turn any crash-fault protocol into a Byzantine one → a system needs only those two properties, not a general trusted computer → so a minimal device providing exactly them is enough.
A small trusted base can be verified. Because TNIC’s attestation kernel is only about 2,114 lines of hardware, a fraction of a percent of a TEE’s millions → its whole trusted surface is small enough to reason about → so its safety and security can be formally proven, which a tens-of-megabytes enclave cannot.
The network card is the right place for the root-of-trust. Because putting the trust in the NIC hardware makes it independent of whatever CPU or enclave each machine has → the security becomes uniform across a heterogeneous cloud → so systems no longer require every machine to carry the same trusted processor.
Offloading to the card removes the CPU bottleneck. Because attestation and networking run on the SmartNIC with kernel-bypass → the expensive enclave syscalls and trusted-to-untrusted crossings disappear → so trusted networking runs at microsecond latency instead of the several-times slowdown enclaves impose.
Fewer nodes make the protocol cheaper. Because the two properties let a Byzantine system use 2f+1 nodes instead of the usual 3f+1 → the protocol needs fewer replicas and less coordination → so it scales better while still tolerating malicious faults.

What it costs (trade-offs and scope)

It needs a special trusted network card. TNIC is FPGA SmartNIC hardware with a burned-in secret key and a verified attestation kernel; deploying it requires this hardware in each machine, not just a software update, along with a manufacturer-rooted key setup.
Attestation adds latency to each message. The keyed authentication code computed on the card cannot be parallelized, so TNIC adds 3 to 20 times the latency of untrusted hardware networking, growing with message size — though it is still far faster than the CPU-enclave baselines it replaces.
The verification covers the protocols, not the whole silicon. The formal proof (with the Tamarin prover) establishes the safety and security of TNIC’s attestation and bootstrapping protocols; it is a strong guarantee about the design, but it rests on assumptions such as the secrecy of the burned-in keys and the correctness of the underlying hardware.
Trust rests on shared secret keys. The properties depend on per-session secret keys that are shared among the trusted cards and unknown to untrusted parties; key distribution, rotation, and the consequences of key compromise are part of the trust model that must be managed.
It targets the two-property transformation, not all of Byzantine tolerance. TNIC provides the lower-bound primitives that upgrade crash-tolerant protocols; systems must still be expressed through its transformation and APIs, and it inherits the classical fault and threat model assumptions of that approach.
The four demonstrated systems are the evidence. Generality is shown by building four trusted systems (attested memory, Byzantine fault tolerance, chain replication, and accountability); these are strong case studies rather than a proof that every distributed system maps on equally well.

The evaluation

Three ways to build a distributed system for an untrusted cloud

Property	Plain crash-fault system	TNIC
Tolerates Byzantine (malicious) faults
Where the root-of-trust lives	Nowhere
Trusted computing base	Not applicable
Formally verified	Not applicable
Works across heterogeneous machines	Yes
Performance	Fast but unsafe	Up to 6 times throughput, 3 to 5 times lower latency

Three ways to build a distributed system for an untrusted cloud, as compared in the paper. A plain crash-fault-tolerant system is fast but unsafe against malicious faults. A CPU-enclave (TEE) trusted system tolerates Byzantine faults, but its root-of-trust is tied to a specific CPU, its trusted computing base is enormous (an entire kernel, tens of megabytes) and unverifiable, and its network input and output is slow. TNIC places the root-of-trust on an FPGA SmartNIC: a tiny attestation kernel (about 2,114 lines of hardware code, only 0.09 percent of the trusted base of TEE-hosted systems) stamps every message with a keyed authentication code and a monotonic counter, giving transferable authentication and non-equivocation. Because it is minimal, it is formally verified with the Tamarin prover; because it lives on the NIC, it is uniform across heterogeneous machines; and because it offloads to hardware with kernel-bypass, it delivers up to 6 times the throughput and 3 to 5 times lower latency than CPU-enclave systems, at microsecond attestation latency. The paper demonstrates generality by building four trusted distributed systems on TNIC.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

TNIC is a clean rethinking of where trust should live in a distributed system. It starts from a precise theoretical fact — that two security properties, transferable authentication and non-equivocation, are enough to upgrade any crash-tolerant protocol to a Byzantine-tolerant one without extra nodes — and asks what the smallest, best-placed hardware to provide them would be. The answer, a tiny attestation kernel on the network card rather than a general-purpose enclave in the CPU, addresses all three weaknesses of CPU enclaves at once: it is uniform across heterogeneous machines because it does not depend on the CPU; its trusted base is about 2,114 lines of hardware, a fraction of a percent of a TEE’s, small enough to be formally verified with the Tamarin prover; and it is fast because attestation and networking are offloaded to the SmartNIC with kernel-bypass, giving up to 6 times the throughput and 3 to 5 times lower latency of enclave-based systems. The demonstration is substantial — four different trusted distributed systems built on the same substrate. Read the scope carefully. TNIC is real hardware that must be deployed, with a manufacturer-rooted key setup; its attestation adds latency that grows with message size, though it stays well ahead of the enclave baselines; and the formal verification covers its protocols under assumptions like key secrecy and correct underlying hardware, not the entire silicon stack. The durable idea is that trust is best built on a minimal, well-defined, verifiable foundation placed at exactly the layer that needs it — here, the network — rather than on a large, complex, general-purpose trusted component that is fast to reach for but hard to reason about.

Background

Why a small, well-placed trusted core keeps winning

It helps to see TNIC within a recurring principle in secure systems: the less you have to trust, and the more precisely you can describe it, the stronger and more analyzable your security becomes. A CPU enclave asks you to trust an entire operating-system kernel; TNIC asks you to trust about two thousand lines of hardware that do exactly two things, and then proves them correct. This is the same instinct behind the secure-undervolting work covered elsewhere on this site, which argues its security is equivalent to today’s systems by reducing the question to a small, well-understood core and reasoning about that, rather than making a sweeping claim about the whole chip. The common thread is that security scales with simplicity: a minimal trusted component can be verified, audited, and reasoned about in ways a sprawling one never can, and shrinking the trusted base is often worth more than any single defensive feature. TNIC adds a second lesson about placement — that the right home for a mechanism is the layer where the work actually happens, so a trust primitive for networked systems belongs on the network card, close to the messages, and off the general-purpose CPU. As clouds grow more heterogeneous and more adversarial, both instincts — trust less and verify it, and put each mechanism where it belongs — are likely to shape how the next generation of trustworthy systems is built.

If you remember only three things

1. Cloud infrastructure is untrusted: machines and networks can fail arbitrarily or maliciously (Byzantine faults). Byzantine-tolerant systems stay correct anyway but classically need many extra replicas (3f+1 to tolerate f faults) and are complex and slow, so most systems use the weaker crash-fault model (2f+1) and are unsafe if a node lies or is compromised. Theory shows that two security properties — transferable authentication (verify who really sent a message, even if forwarded) and non-equivocation (a node cannot tell different peers conflicting things) — suffice to upgrade any crash-tolerant protocol to Byzantine-tolerant without adding nodes. The usual way to get them, CPU trusted execution environments (enclaves like Intel SGX), is heterogeneous across machines, carries an enormous unverifiable trusted computing base (an entire kernel, tens of megabytes), and is slow on the network.
2. TNIC builds a minimal, formally verified silicon root-of-trust at the network interface instead of the CPU. A small attestation kernel on an FPGA SmartNIC provides exactly the two properties by stamping each outgoing message with the node identifier, a monotonically increasing per-session counter, and a keyed authentication code (HMAC) using a shared secret key, and by checking all of that on receipt — rejecting forged, replayed, reordered, or equivocating messages. Its trusted base is about 2,114 lines of hardware code, only 0.09 percent of TEE-hosted systems, small enough to be formally verified with the Tamarin prover. It runs over a hardware-accelerated, kernel-bypass network stack and exposes APIs plus a recipe to transform crash-tolerant systems into Byzantine-tolerant ones with the same node count.
3. Because the root-of-trust lives on the NIC, it is uniform across heterogeneous machines; because it is tiny, it is verifiable; and because it offloads to hardware with kernel-bypass, it is fast — up to 6 times the throughput and 3 to 5 times lower latency of CPU-enclave systems, at microsecond attestation latency. The authors demonstrate generality by building four trusted distributed systems on it (attested memory, Byzantine fault tolerance, chain replication, and accountability). Scope: it requires deploying special trusted network cards with manufacturer-rooted keys, attestation adds latency that grows with message size (still ahead of enclaves), and the formal verification covers the protocols under assumptions like key secrecy. The durable lesson is that trust is best built on a minimal, well-defined, verifiable foundation placed at the layer that needs it, rather than a large general-purpose trusted component.

Check yourself

Check your understanding

1. What is the difference between crash faults and Byzantine faults, and why does it matter for the cloud?
2. What two security properties let a crash-tolerant protocol become Byzantine-tolerant, and how does TNIC provide them?
3. Why does TNIC put the root-of-trust on the network card instead of in a CPU enclave?
4. Why is the small size of TNIC's trusted computing base so important?

Sources

The paper is “TNIC: A Trusted NIC Architecture” by Dimitra Giantsidi, Julian Pritzi, Felix Gust, Antonios Katsarakis, Atsushi Koshiba, and Pramod Bhatotia, published at ASPLOS 2025, DOI 10.1145/3676641.3716277. The authors are at the Technical University of Munich, with related work at the University of Edinburgh; the paper is open access under CC-BY.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2502.05338), which matches the open-access version of record. The problem framing (the untrusted Byzantine cloud, the crash-fault and Byzantine-fault models and their 2f+1 and 3f+1 replica requirements, and the theoretical result that transferable authentication and non-equivocation suffice to transform crash-tolerant protocols to Byzantine-tolerant ones without adding nodes), the critique of CPU trusted execution environments (their heterogeneity, their large trusted computing base — the calculation that a state-of-the-art TEE trusts an entire kernel and tens of megabytes of code — and their network performance overhead), the TNIC design (the network-interface-level root-of-trust on an FPGA SmartNIC; the attestation kernel with its HMAC, keystore, and monotonic counters; the Attest and Verify functions; the RoCE protocol kernel and kernel-bypass network stack; the network library and the recipe to transform crash-tolerant systems; and the formal verification with the Tamarin prover), the measured results (the trusted computing base of about 2,114 lines of hardware code, only 0.09 percent of TEE-hosted systems; the microsecond attestation latency; the 3 to 5 times lower latency and up to 6 times higher throughput versus CPU-enclave systems; and the added latency of trusted over untrusted hardware networking), and the four demonstrated trusted distributed systems (attested append-only memory, Byzantine fault tolerance, chain replication, and accountability with peer review) are all drawn directly from the paper. Background explanations of distributed systems, fault models, message authentication codes, trusted execution environments, SmartNICs, and formal verification are standard and provided to make the paper’s contribution understandable.