Harmonia: A Unified Framework for Heterogeneous FPGA Acceleration in the Cloud

Cloud providers rent out reconfigurable FPGA chips by splitting each one into a fixed provider-managed shell and a user's custom accelerator role, but this tidy arrangement assumed every FPGA in the fleet was identical — and now they come from different vendors with different interfaces, forcing providers to hand-build a separate shell for each and users to rewrite their software for each; this work from the Chinese Academy of Sciences and ByteDance builds Harmonia, which hides the hardware differences behind an abstraction layer and assembles each shell from reusable, trimmable building blocks with one common interface, cutting shell-building effort by up to 93 percent and running in production at Douyin for years.

The argument in one glance

Problem

The tidy shell-role split assumed all cloud FPGAs were the same

Cloud providers split each rented FPGA into a provider-managed shell (the static infrastructure) and a user role (the custom accelerator that plugs in). This works when every FPGA is identical, but fleets now mix chips from different vendors and boards with different interfaces, breaking the assumption.
What people did

Build a separate shell per platform and port roles by hand

Have providers develop a distinct shell for each FPGA platform, and have users port their roles and adjust their host software for each platform they want to run on.
Why it fell short

Per-platform shells and software changes do not scale

Developing a shell for every platform is laborious and repetitive for providers, and porting roles plus making ad-hoc software modifications for each platform burdens users. Prior work either handled only homogeneous FPGAs or only partially fixed role portability, still requiring heavy shell development and ad-hoc software changes.
This paper's idea

One framework that hides the differences (Harmonia)

Harmonia is a unified framework for heterogeneous cloud FPGA acceleration with two layers: a platform-specific layer that abstracts hardware differences using automated adapters and lightweight interface wrappers, and a platform-independent layer that provides one unified shell for diverse roles and host software. The shell is built from Reusable Building Blocks with hierarchical tailoring, and a command-based interface minimizes per-platform software changes. Deployed at Douyin for years, it cuts shell-development effort by 69 to 93 percent at under 0.63 percent overhead.

Why this problem is worth solving

Cloud FPGAs are useful precisely because they can be reshaped for any accelerator, and the shell-role split is what makes that practical to rent out.

An FPGA, a field-programmable gate array, is a chip whose digital logic can be reconfigured after it is manufactured, so it can be turned into a custom hardware accelerator for whatever a workload needs — packet processing, machine learning, compression, and more. Cloud providers now rent FPGAs as accelerators, but handing a user raw, full control of a chip that also connects to the provider’s servers and network would be unmanageable and unsafe. So providers adopt the shell-role architecture, which divides the FPGA into two parts. The shell is the static, provider-managed infrastructure: it handles the plumbing every accelerator needs — the connection to the host server, memory controllers, networking, and management functions — and presents stable interfaces. The role is the user’s custom accelerator logic, loaded into the reconfigurable region, which plugs into the shell’s interfaces. This separation is what makes cloud FPGAs accessible: users write only their accelerator and rely on the shell for everything else, while the provider maintains one shell and manages the fleet. It works beautifully as long as every FPGA in the fleet is identical.

The assumption of a uniform fleet is exactly what modern clouds have outgrown.

The shell-role architecture was designed for homogeneous FPGA servers, where every board is the same. But cloud FPGA fleets have become heterogeneous: they mix chips from different vendors — for example Xilinx (now AMD) and Intel (formerly Altera) — across different device families and board designs, each exposing different low-level interfaces. This heterogeneity breaks the clean split. Because the shell is tied to the specific hardware it manages, a provider must develop a separate shell for every platform, which is laborious and endlessly repeated as new hardware arrives. And because a role and its host software are written against a specific shell’s interfaces, users must port their roles to each platform and make ad-hoc software modifications to match — a burden that grows with every kind of FPGA in the fleet. Prior efforts have not fully solved this: some target only homogeneous FPGAs, and others address only part of the role-portability problem while still requiring laborious shell development from providers and ad-hoc software changes from users. The result is that the very heterogeneity that lets clouds adopt the best FPGA for each job also multiplies the engineering cost of supporting them.

What people did before, and where it broke

Build one shell for a homogeneous fleet. Develop a single shell and have users write roles against it. Where it broke. It assumes every FPGA is identical, which no longer holds as fleets mix vendors, families, and boards.
Develop a separate shell for each platform. Hand-build a shell per FPGA type as new hardware is added. Where it broke. This is laborious and repetitive for providers, redone from scratch for every platform.
Have users port roles and modify software per platform. Let users adapt their roles and host software to each platform’s interfaces. Where it broke. Porting roles and making ad-hoc software changes for every platform is a heavy, error-prone burden that grows with fleet diversity; prior frameworks only partially relieved it.
The gap. No framework unified heterogeneous cloud FPGAs so that one shell and unmodified software could span diverse platforms without laborious per-platform work. Providing that is what Harmonia does.

How Harmonia works

From per-platform shells to one framework spanning heterogeneous FPGAs

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Hide each FPGA’s hardware differences behind an abstraction layer. Harmonia’s foundation is a platform-specific layer that absorbs the differences between vendors, families, and boards using automated platform adapters and lightweight interface wrappers, so that everything above it sees a common interface instead of vendor-specific details. Presenting heterogeneous hardware through one uniform, hardware-agnostic layer, so the components built on top do not have to know or care which underlying device they are running on, is the same move the trusted-network-card work covered elsewhere on this site makes, offering a single unified abstraction that is independent of the specific hardware beneath it so diverse machines can be treated the same way.

Assemble the shell from reusable, trimmable building blocks. Rather than hand-writing a monolithic shell for each platform, Harmonia composes the shell from modular Reusable Building Blocks and applies hierarchical tailoring to include only the modules a given role actually needs, making the shell both reusable across platforms and lean in its use of scarce FPGA resources. Building a system by assembling modular, individually-developed components and tailoring the assembly to each case, instead of constructing a rigid whole from scratch every time, is the same design the deep-learning-training compiler work covered elsewhere on this site follows, composing its kernels from individually-built pieces the compiler assembles and trims to fit each specific pattern.

Give host software one interface that spans all platforms. So that software does not have to be rewritten for each FPGA, Harmonia exposes a single command-based interface that is identical across platforms, letting the same role and the same host software run on any device in a heterogeneous fleet. Presenting heterogeneous underlying hardware as one uniform system so that software can run across it without per-device changes is the same goal the fused-kernel operating-system work covered elsewhere on this site pursues, unifying a set of different machines under a single coherent interface so that programs need not be specialized to each one.

Following a role across two different FPGAs through Harmonia

Deploying one accelerator across a mixed-vendor FPGA fleet

A high-level trace of the approach as described in the abstract; the exact adapter, building-block, and tailoring mechanisms are beyond the abstract this page is based on.

Step 1 of 5

A user has a role and a mixed fleet

A user has an accelerator (a role) and wants to run it in the cloud, where the provider's fleet mixes FPGAs from different vendors with different interfaces.

Different vendors mean different low-level plumbing.

Why it works

Heterogeneity multiplies per-platform work. Because each FPGA vendor and board differs while the shell-role model assumed uniformity → providers must rebuild the shell and users must rewrite software for every platform → so a unified framework that hides the differences removes that repeated effort.
An abstraction layer hides hardware differences. Because automated adapters and interface wrappers translate each platform into a common form → the layers above never see the vendor-specific details → so one shell and one role can span diverse FPGAs.
Reusable blocks avoid rebuilding shells. Because a shell is assembled from modular Reusable Building Blocks rather than written from scratch → each new platform or role reuses existing modules → so shell development drops by most of its former cost.
Tailoring keeps resources lean. Because hierarchical tailoring includes only the blocks a given role needs → the shell does not carry unused infrastructure → so scarce FPGA resources are saved rather than spent on a one-size shell.
A uniform interface frees the software. Because host software talks to a single command-based interface identical across platforms → it needs no ad-hoc per-platform modification → so the same software configuration runs everywhere, with negligible overhead.

What it costs (trade-offs and scope)

The abstraction must cover real hardware differences. Hiding vendor, family, and board differences behind adapters and wrappers only works to the extent those adapters faithfully cover each platform’s behavior; a genuinely new or unusual interface still requires building the platform-specific adapter for it.
Abstraction can add overhead. Interposing a unifying layer between roles, software, and hardware risks a performance cost. The abstract reports the overhead as negligible, under 0.63 percent, but any abstraction layer is a trade-off that must be kept thin.
It targets the shell-role cloud model. Harmonia is built around the shell-role architecture that cloud providers use. Its benefits are specific to that deployment model and to providers managing heterogeneous FPGA fleets, not to every FPGA use.
Tailoring depends on knowing role needs. Hierarchical tailoring saves resources by including only what a role requires, so its efficiency depends on correctly determining each role’s needs; a role with broad or changing requirements benefits less from trimming.
The reported gains are against specific baselines. The 69 to 93 percent shell-development reduction, 3.5 to 14.9 percent resource savings, and 15 to 23 times software-configuration simplification are measured against particular prior frameworks and workloads; other comparisons would shift the figures.
This page is written from the abstract. The full text was not available when this explainer was prepared, so the exact designs of the platform adapters, the Reusable Building Blocks, the hierarchical tailoring, and the command-based interface, along with the full evaluation, are summarized only at the level the abstract states, without invented specifics.

The evaluation

Three ways to run accelerators on a heterogeneous cloud FPGA fleet

Property	Per-platform shells	Partial-portability frameworks
Spans cross-vendor FPGAs with one framework
Shell-development effort	High, redone per platform	Still laborious
Host software changes per platform	Ad-hoc, heavy	Some
Resource efficiency	Not applicable	Not applicable
Overhead and maturity	Not applicable	Research

Three ways to run accelerators on a heterogeneous cloud FPGA fleet, as described in the paper's abstract. Building a separate shell for each platform, as the homogeneous model implies, is laborious and repeated for providers and forces heavy per-platform software changes on users. Partial-portability frameworks address only part of the role-portability problem while still requiring laborious shell development and ad-hoc software changes. Harmonia unifies heterogeneous FPGAs with a platform-specific layer (automated adapters and interface wrappers that hide hardware differences) and a platform-independent layer (a modular shell built from Reusable Building Blocks with hierarchical tailoring, plus a command-based interface that minimizes software changes). The abstract reports that Harmonia reduces shell-development workload by 69 to 93 percent with under 0.63 percent overhead, supports cross-vendor FPGAs, reduces resource consumption by 3.5 to 14.9 percent, and simplifies software configuration by 15 to 23 times while maintaining comparable performance, and that it has been deployed in production at the large service provider Douyin for several years. These figures are from the abstract; the full paper was not available for this explainer.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Harmonia addresses a concrete and growing operational pain in cloud infrastructure: the shell-role architecture that made FPGAs practical to rent was built for uniform fleets, and as clouds adopt FPGAs from multiple vendors and boards to get the best hardware for each job, that architecture forces providers to hand-build a shell per platform and users to rewrite their software per platform. Harmonia’s design is a clean layering. A platform-specific layer uses automated adapters and lightweight wrappers to hide each FPGA’s hardware differences, so a platform-independent layer above it can present one unified shell to diverse roles and host software. That shell is assembled from Reusable Building Blocks and hierarchically tailored so each role gets a lean, resource-efficient shell rather than a monolithic one, and a single command-based interface lets host software run across platforms without ad-hoc changes. What lends the work weight is its production pedigree: it reports being deployed at Douyin, a large service provider, for several years, which is strong evidence it works at scale. The reported gains are substantial — a 69 to 93 percent reduction in shell-development workload, cross-vendor support, 3.5 to 14.9 percent lower resource consumption, and 15 to 23 times simpler software configuration, all at under 0.63 percent overhead. Read the specifics with care. This explainer is written from the abstract, so the exact adapters, building blocks, tailoring, and interface, and the full evaluation, are not reproduced here. As trade-offs, the abstraction only covers the platforms its adapters are built for, any unifying layer adds some overhead, the approach is specific to the shell-role cloud model, and the numbers are relative to particular baselines. The durable idea is that when a hardware fleet becomes heterogeneous, the right response is a unifying abstraction plus reusable, tailorable infrastructure, so that the per-platform engineering cost stops scaling with the number of platforms.

Background

Why unifying heterogeneous hardware behind one abstraction keeps recurring

It helps to see Harmonia within a pattern that recurs whenever hardware fleets diversify: the number of distinct devices grows, the cost of supporting each one separately grows with it, and the winning response is almost always a unifying abstraction that hides the differences plus reusable infrastructure that is assembled rather than rebuilt per device. Here the devices are cloud FPGAs from different vendors, and the abstraction is a two-layer framework that lets one shell and unmodified software span them. The disaggregated-memory work covered elsewhere on this site tackles the same underlying force in the memory domain, defining a single precise model over hardware that has become diverse and coherent in new ways, so that software can be built once against the unified abstraction rather than against each variant. The common thread is that heterogeneity is valuable — it lets a system use the best hardware for each purpose — but only if the software and management layers above are not forced to multiply their effort by the number of hardware kinds. A good abstraction converts a fleet of many different devices into something that presents as one, and reusable, tailorable building blocks convert per-device construction into per-device configuration. As accelerators of every kind proliferate in datacenters, and as no single vendor or device dominates, frameworks that absorb this diversity behind clean interfaces — and prove themselves in production rather than only in the lab — are likely to be what keeps heterogeneous infrastructure economically manageable.

If you remember only three things

1. Cloud providers rent FPGAs — reconfigurable chips users program into custom accelerators — using the shell-role architecture: the shell is the static, provider-managed infrastructure (host connection, memory, networking, management) and the role is the user's custom accelerator that plugs into it. This makes FPGAs accessible and manageable, but it assumes a homogeneous fleet where every FPGA is identical. Modern cloud fleets are heterogeneous, mixing chips from different vendors (such as Xilinx/AMD and Intel/Altera), families, and boards with different interfaces, which breaks the model: providers must develop a separate shell for every platform, and users must port roles and make ad-hoc software modifications per platform. Prior work handled only homogeneous FPGAs or partially fixed role portability while still requiring heavy shell development and ad-hoc software changes.
2. Harmonia is a unified framework for heterogeneous cloud FPGA acceleration with two layers. A platform-specific layer abstracts hardware differences using automated platform adapters and lightweight interface wrappers, so the differences between vendors and boards are hidden behind a common interface. A platform-independent layer provides one unified shell for diverse roles and host software: it is built from modular Reusable Building Blocks and uses hierarchical tailoring to include only the modules each role needs (keeping the shell resource-efficient and easy to use), and it exposes a command-based interface that minimizes software modifications across platforms so the same host software runs everywhere.
3. Harmonia has been deployed in production at Douyin, a large service provider, for several years. The abstract reports it reduces shell-development workload by 69 to 93 percent and simplifies role and software configuration with negligible overhead (under 0.63 percent); and that, compared with other frameworks, it supports cross-vendor FPGAs, reduces resource consumption by 3.5 to 14.9 percent, and simplifies software configuration by 15 to 23 times while maintaining comparable performance. Scope: the abstraction covers only the platforms its adapters are built for, any unifying layer adds some overhead, it is specific to the shell-role cloud model, and the numbers are relative to particular baselines. Note: this explainer is written from the abstract, as the full text was not available, so the exact mechanisms and full evaluation are summarized only at the level the abstract states. The durable lesson is that heterogeneity is best absorbed by a unifying abstraction plus reusable, tailorable infrastructure, so per-platform engineering stops scaling with the number of platforms.

Check yourself

Check your understanding

1. What is the shell-role architecture, and why do cloud providers use it?
2. Why does the increasing heterogeneity of cloud FPGAs break the shell-role architecture?
3. How does Harmonia unify heterogeneous FPGAs?
4. What does Harmonia's deployment and reported results suggest?

Sources

The paper is “Harmonia: A Unified Framework for Heterogeneous FPGA Acceleration in the Cloud” by Luyang Li, Heng Pan, Xinchen Wan, Kai Lv, Zilong Wang, Qian Zhao, Feng Ning, Qingsong Ning, Shideng Zhang, Zhenyu Li, Layong Luo, and Gaogang Xie, published at ASPLOS 2025, DOI 10.1145/3676641.3716259. The authors are affiliated with the Chinese Academy of Sciences and ByteDance.

This explainer is written from the paper’s abstract, because the full text was not available at the time of writing: the publisher’s copy was behind automated-access protection, no preprint was found, and no separately hosted open-access copy could be retrieved. The problem framing (FPGAs as cloud accelerators, the shell-role architecture on homogeneous FPGA servers, and the challenges heterogeneity poses — laborious per-platform shell development for providers and ad-hoc software modifications for users, which prior work only partially addressed), the description of the approach (Harmonia’s two layers, the platform-specific layer with automated platform adapters and lightweight interface wrappers, the platform-independent layer providing a unified shell built from Reusable Building Blocks with hierarchical tailoring, and the command-based interface to minimize software modifications), and the results (deployment at Douyin for several years; a 69 to 93 percent reduction in shell-development workload; under 0.63 percent overhead; support for cross-vendor FPGAs; a 3.5 to 14.9 percent reduction in resource consumption; and a 15 to 23 times simplification of software configuration while maintaining comparable performance) come directly from the abstract. The surrounding explanation of FPGAs, reconfigurable logic, the shell-role model, hardware abstraction layers, and FPGA vendors is standard, widely-known background provided to make the abstract’s claims understandable, and is not specific to this paper. The exact designs of the platform adapters, the Reusable Building Blocks, the hierarchical tailoring, and the command-based interface, along with the detailed evaluation, are not reproduced here because they were not available.