A Framework for Developing and Optimizing Fully Homomorphic Encryption Programs on GPUs

Jianyu Zhao, Xueyu Wu, Guang Fan, Mingzhe Zhang, Shoumeng Yan, Lei Ju and 1 other

Fully homomorphic encryption (FHE) lets a computer compute directly on encrypted data, so a hospital or bank could run machine learning without ever seeing the raw records — but writing efficient FHE programs on GPUs is brutally hard, because developers must juggle the intricate FHE math and low-level GPU details at once, and even well-tuned individual GPU kernels leave the whole program slow due to FHE's huge memory needs, fine-grained memory operations, and redundant computation; this paper (Jianyu Zhao, Zhuoran Ji, and colleagues) presents EasyFHE, a PyTorch-like framework that lets developers write high-level FHE logic while it automatically picks implementations, manages memory, and applies tailored optimization passes when lowering the program to GPU kernels — even running FHE programs whose memory exceeds the GPU's capacity, for an average 2.88 times speedup (peak 4.39 times) over state-of-the-art open-source GPU FHE libraries.

The argument in one glance

Problem

Computing on encrypted data is powerful, but writing it efficiently on GPUs is brutal

Fully homomorphic encryption, or FHE, is a way to compute directly on encrypted data: a server can add, multiply, and run whole machine-learning models on ciphertext and return an encrypted result, without ever seeing the plaintext. This is transformative for sensitive domains like healthcare and finance, because a hospital could have a cloud service run a model on encrypted patient records, protecting both the patient data and the model. FHE involves a lot of independent arithmetic, so it maps naturally onto a graphics processing unit (GPU), which has thousands of parallel lanes. But turning that potential into a fast, complete application is very hard. FHE schemes are mathematically intricate, and GPUs have their own low-level details, so a developer trying to build an FHE application must fight both at once instead of focusing on the high-level logic they care about. And FHE is memory-hungry: it needs a lot of memory, does many fine-grained memory operations, and repeats redundant computations. As a result, even when each individual GPU kernel is carefully optimized, the end-to-end program is still inefficient.
What people did

Hand-write and optimize individual GPU kernels in an FHE library

The established approach is to build GPU FHE libraries that provide highly optimized kernels for the core operations, and have developers assemble their application by calling those kernels. Effort goes into making each kernel — each encrypted multiply, each transform — as fast as possible on the GPU.
Why it fell short

Fast kernels are not a fast program, and big FHE programs exceed GPU memory

Optimizing kernels one at a time is not enough. Many of FHE's costs live between the kernels, not inside them: the high memory requirements, the fine-grained memory operations, and redundant computations that span multiple operations. A library of individually fast kernels still leaves those cross-kernel inefficiencies in place, so the whole application runs slower than it should. On top of that, developers are still exposed to the full complexity of the FHE scheme and the GPU, which slows development and invites mistakes. And a hard practical wall appears when an FHE program's memory requirements exceed the GPU's capacity: existing GPU FHE libraries simply cannot run such programs. What is missing is a framework that lets developers express high-level FHE logic and then optimizes across the whole program as it turns that logic into GPU kernels, including programs too big to fit in GPU memory.
This paper's idea

A PyTorch-like framework that optimizes the whole FHE program down to GPU kernels

EasyFHE is a framework that simplifies both the development and the optimization of GPU-accelerated FHE applications. On the development side, it works much like PyTorch: the developer writes high-level interfaces that describe the computational logic, and EasyFHE automatically handles the low-level tasks — selecting which implementation of each operation to use, and managing memory — so the developer can stay focused on the program logic rather than FHE and GPU minutiae. On the optimization side, EasyFHE adds an optimization framework that systematically attacks performance bottlenecks by applying tailored optimization passes during the lowering from the high-level FHE program to the actual GPU kernels. Because it sees and transforms the whole program, not just isolated kernels, it can address the cross-kernel costs — the high memory use, the fine-grained memory operations, and the redundant computations — that per-kernel optimization misses. Crucially, EasyFHE uniquely supports FHE programs whose memory requirements exceed typical GPU capacities, which existing GPU FHE libraries cannot run at all. Compared with state-of-the-art open-source GPU FHE libraries, it achieves an average speedup of 2.88 times, with a peak of 4.39 times.

Why this problem is worth solving

Fully homomorphic encryption is one of the few technologies that could let sensitive data be processed by someone you do not fully trust, and making it fast enough to use is the whole game.

Ordinarily, to compute on data you must be able to read it, so sending data to a cloud service means trusting that service with the plaintext. Fully homomorphic encryption breaks that rule: it lets a party compute on data while it stays encrypted, producing an encrypted result that only the data owner can decrypt. For healthcare and finance, this is exactly the missing piece — a hospital or bank could outsource machine learning on encrypted records, so neither the raw data nor, potentially, the model is ever exposed to the server. The obstacle has always been speed: FHE turns each simple operation into a large amount of arithmetic on big encrypted objects, making it far slower than computing on plaintext. The saving grace is that this arithmetic is highly parallel, which is precisely what a graphics processing unit is built for, with its thousands of parallel lanes. So GPUs are the natural platform to make FHE practical. But hardware potential is not the same as delivered performance, and closing that gap for real applications is what determines whether FHE can move from a promising idea into everyday use on sensitive data.

The difficulty is that FHE’s costs are spread across a whole program, while developers are forced to work one low-level kernel at a time.

Building an FHE application on a GPU today means composing calls to a library of optimized GPU kernels, each performing one core FHE operation. Two things go wrong. First, the developer is exposed to the full complexity of both the FHE scheme (which is mathematically intricate, with many moving parts and parameters) and the GPU (with its own low-level performance details), so instead of expressing the high-level logic of their application they spend their effort managing minutiae, which is slow and error-prone. Second, and more subtly, optimizing each kernel in isolation cannot fix the costs that live between kernels. FHE has very high memory requirements, performs many fine-grained memory operations, and repeats computations that a whole-program view could reuse or eliminate. A collection of individually fast kernels still pays all of those cross-kernel costs, so the end-to-end program is inefficient even though every piece looks optimized. And when a program’s data simply does not fit in the GPU’s memory, a per-kernel library has no answer at all and cannot run it. Solving this needs a system that understands the entire program, optimizes across it, and manages memory on the developer’s behalf — which is the role EasyFHE sets out to fill.

What people did before, and where it broke

Optimized GPU FHE kernel libraries. Provide a library of individually fast GPU kernels for core FHE operations and let developers call them. Where it broke. Fast kernels do not make a fast program: FHE’s high memory use, fine-grained memory operations, and redundant computation live between kernels, so per-kernel optimization leaves the end-to-end application slow.
Hand-tuning at the GPU level. Have developers manage FHE-scheme details and GPU specifics directly to squeeze out performance. Where it broke. It forces developers away from their high-level logic into intricate low-level detail, which is slow to write, hard to get right, and does not scale across a large application.
Keep the whole program in GPU memory. Rely on the GPU having enough memory to hold the FHE program’s data. Where it broke. FHE programs can require more memory than a typical GPU has, and existing GPU FHE libraries cannot run programs whose memory requirements exceed GPU capacity.
The gap. No framework let developers write high-level FHE logic while a system optimized across the entire program during lowering to GPU kernels and managed memory automatically, including for programs too large to fit in GPU memory.

How EasyFHE works

From high-level FHE logic to optimized GPU kernels

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Let developers write the logic; hide the FHE and GPU minutiae. The first pillar of EasyFHE is a PyTorch-like programming model: the developer expresses the high-level computational logic, and the framework automatically handles low-level tasks such as implementation selection and memory management. This frees them from juggling the intricate FHE scheme and GPU details, which is what made FHE development so slow and error-prone. Providing a high-level interface that lets a developer state what to compute while the system takes care of turning it into efficient low-level execution is the same design philosophy the work on faster machine learning through dynamic Python bytecode transformation covered elsewhere on this site is built on, capturing high-level programs and compiling them down for speed. EasyFHE brings that philosophy to encrypted computation, where the low-level details being hidden are those of both the FHE scheme and the GPU.

Optimize the whole program during lowering, not one kernel at a time. The second pillar is an optimization framework that applies tailored passes as the high-level FHE program is lowered to GPU kernels. Because it sees the entire program, it can eliminate the cross-kernel inefficiencies — redundant computation, excessive and fine-grained memory operations — that survive when each kernel is optimized alone. Accelerating computation on encrypted data by optimizing across the whole pipeline rather than tuning isolated pieces is the same approach the work on client-optimized algorithms and acceleration for encrypted compute offloading covered elsewhere on this site takes, treating end-to-end efficiency, not a single operation, as the target. EasyFHE realizes this as compiler-style passes during lowering, so the optimizations are systematic rather than manual.

Manage memory so programs can exceed the GPU’s capacity. The third, distinctive contribution is that EasyFHE’s automatic memory management lets it run FHE programs whose memory requirements exceed typical GPU capacities — something existing GPU FHE libraries cannot do at all. Enabling a computation to proceed even when its data does not fit in GPU memory, by moving data intelligently rather than demanding it all fit at once, is the same capability the work on tensor migration and prefetching in unified memory covered elsewhere on this site provides for large models, keeping the GPU fed from a larger memory pool. EasyFHE applies that idea to the especially memory-hungry world of FHE, turning an out-of-memory failure into a program that simply runs.

Following a program through EasyFHE

Running an encrypted machine-learning model too big for GPU memory

An illustrative walk-through, based on the paper's abstract, of how EasyFHE takes a high-level FHE program to efficient GPU execution. Exact internal steps should be read from the full paper.

Step 1 of 5

The developer writes high-level FHE logic

Using EasyFHE's PyTorch-like interfaces, the developer describes an encrypted computation — say, running an inference model on encrypted patient data — as high-level logic, without hand-coding FHE-scheme details or GPU kernels.

The developer focuses on what to compute, not on low-level FHE or GPU specifics.

Why it works

A high-level interface removes the wrong kind of work. Developers were forced to manage FHE-scheme and GPU details by hand → but a PyTorch-like interface lets them state the logic while EasyFHE automates implementation selection and memory → so effort goes to the program, not the minutiae.
Whole-program passes catch cross-kernel costs. FHE’s memory operations and redundant computation span multiple kernels → but optimization passes applied during lowering see the whole program → so they eliminate inefficiencies that per-kernel optimization structurally cannot reach.
Automatic memory management breaks the GPU-capacity wall. FHE programs can need more memory than the GPU has → but EasyFHE manages memory rather than assuming everything fits → so programs that existing libraries cannot run at all become runnable.
Systematic passes beat manual tuning. Hand-optimizing each kernel is slow and leaves gaps → but a framework that applies tailored passes attacks bottlenecks systematically → so optimization is both broader and more consistent across the program.
Matching FHE’s parallelism to the GPU pays off end to end. FHE’s arithmetic is highly parallel and the GPU has many lanes → but only whole-program optimization and memory handling turn that fit into real speed → so EasyFHE delivers an average 2.88 times and peak 4.39 times speedup over prior GPU FHE libraries.

What it costs (trade-offs and scope)

This explainer is from the abstract only. The full paper was not available when this page was written, so the description of EasyFHE’s mechanisms is at the level the abstract states; the specific optimization passes, the memory-management scheme, and the benchmark details are not reproduced here and should be read from the paper.
A framework adds a layer between developer and hardware. Hiding implementation selection and memory management behind high-level interfaces is convenient, but it means the framework’s choices, not the developer’s, determine performance; a case the framework handles poorly is harder to hand-tune than in a raw kernel library.
Out-of-memory support is not free performance. Running programs whose memory exceeds GPU capacity is a capability existing libraries lack, but computations that spill beyond GPU memory generally pay a data-movement cost; the abstract reports strong speedups, yet such programs are enabled rather than made as cheap as if they fit.
The speedups are relative to open-source GPU FHE libraries. The 2.88 times average and 4.39 times peak are measured against state-of-the-art open-source GPU FHE libraries on the paper’s workloads; comparisons against other baselines, or on different applications, could differ.
It targets GPU FHE, within FHE’s inherent overhead. EasyFHE improves how FHE runs on GPUs; it does not remove the fundamental fact that FHE is far more expensive than computing on plaintext, so it makes encrypted computation more practical rather than free.

The evaluation

Developing and running FHE applications on GPUs

Property	Hand-assembled optimized kernels	GPU FHE library	EasyFHE
Developer works at high level
Optimizes across the whole program
Handles cross-kernel memory and redundancy
Runs programs bigger than GPU memory
End-to-end performance

A comparison, drawn from the paper's abstract, of ways to build GPU FHE applications. EasyFHE provides a PyTorch-like high-level interface, optimizes across the whole program during lowering to GPU kernels, manages memory automatically (uniquely supporting programs whose memory exceeds GPU capacity), and reports an average 2.88 times speedup, peak 4.39 times, over state-of-the-art open-source GPU FHE libraries.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based only on the paper’s abstract; the full text was not available when it was written. The claims stated here are the abstract’s: EasyFHE is a framework that simplifies the development and optimization of GPU-accelerated fully homomorphic encryption (FHE) applications; it provides PyTorch-like high-level interfaces for defining computational logic while automatically handling low-level tasks such as implementation selection and memory management; it incorporates an optimization framework that applies tailored optimization passes during the lowering from high-level FHE programs to GPU kernels to address performance bottlenecks; it uniquely supports FHE programs with memory requirements exceeding typical GPU capacities; and it reports an average speedup of 2.88 times, with a peak of 4.39 times, over state-of-the-art open-source GPU FHE libraries. The specific optimization passes, the memory-management design, the exact FHE scheme and parameters, the workloads, and the hardware are not described in the abstract and have not been invented here; they should be confirmed in the full paper. The illustrative walk-through above is a plausible reconstruction, not a trace from the paper.

Background

Why making encrypted computation usable is a systems problem

It helps to place EasyFHE in the broader effort to make privacy-preserving computation practical. Fully homomorphic encryption has existed in theory for over a decade, and its guarantee — compute on data you cannot read — is exactly what sensitive applications in healthcare and finance need. The obstacle has never been whether it works, but whether it is fast and usable enough, because FHE inflates every operation into heavy arithmetic on large encrypted objects and demands large amounts of memory. Progress has therefore come as much from systems and hardware as from cryptography: custom accelerators, GPU kernels, compilers, and now frameworks that raise the level at which developers work. A recurring theme is that isolated speedups are not enough; the whole pipeline, from how a developer expresses a computation to how memory and kernels are managed, has to be engineered for the technology to be adopted. The wider push to characterize and optimize entire private-inference systems end to end, rather than a single primitive, is the same undertaking the work on characterizing and optimizing end-to-end systems for private inference covered elsewhere on this site pursues, treating the full system as the unit of optimization. EasyFHE contributes on the developer-productivity and whole-program-optimization side for GPUs, arguing that a PyTorch-like framework with cross-program optimization and memory management is what moves GPU FHE from expert-only to broadly usable.

If you remember only three things

1. Fully homomorphic encryption (FHE) lets a computer compute directly on encrypted data, which is transformative for sensitive machine learning in healthcare and finance, and its parallelism suits GPUs. But building efficient end-to-end FHE applications on GPUs is very hard: developers must juggle the intricate FHE scheme and low-level GPU details instead of their high-level logic, and FHE's high memory needs, fine-grained memory operations, and redundant computation leave whole programs slow even when each GPU kernel is individually optimized.
2. EasyFHE is a PyTorch-like framework that lets developers write high-level FHE logic while it automatically handles implementation selection and memory management, and it adds an optimization framework that applies tailored optimization passes during the lowering from the high-level program to GPU kernels. Because it optimizes across the whole program rather than one kernel at a time, it attacks the cross-kernel costs — memory operations and redundant computation — that per-kernel libraries leave in place.
3. Uniquely, EasyFHE's automatic memory management lets it run FHE programs whose memory requirements exceed typical GPU capacities, which existing GPU FHE libraries cannot run at all, and it reports an average speedup of 2.88 times (peak 4.39 times) over state-of-the-art open-source GPU FHE libraries. (This page is written from the abstract only; the specific passes, memory scheme, and evaluation details should be confirmed in the full paper.)

Check yourself

Check your understanding

1. Why is optimizing individual GPU kernels not enough to make an FHE application fast?
2. How does EasyFHE change the way developers build GPU FHE applications?
3. What can EasyFHE do that existing GPU FHE libraries cannot?
4. Predict: a developer has an encrypted model whose data is slightly larger than their GPU's memory. What happens under a traditional GPU FHE library versus EasyFHE, and why?

Sources

The paper is “A Framework for Developing and Optimizing Fully Homomorphic Encryption Programs on GPUs” by Jianyu Zhao, Xueyu Wu, Guang Fan, Mingzhe Zhang, Shoumeng Yan, Lei Ju, and Zhuoran Ji, published at ASPLOS 2026, DOI 10.1145/3779212.3790120. It is licensed CC BY.

This explainer is written from the paper’s abstract only; the full text was not available at the time of writing, so this page is marked as a draft. According to the abstract, machine learning in sensitive domains such as healthcare and finance increasingly employs fully homomorphic encryption (FHE) to secure both user data and models, and although FHE’s intrinsic parallelism aligns with GPU architectures, optimizing GPU kernels alone is insufficient for efficient end-to-end FHE application development. The complexity of FHE schemes and intricate GPU-specific details impede developers from focusing on high-level program logic, and FHE’s high memory requirements, fine-grained memory operations, and redundant computations introduce further optimization challenges, resulting in inefficiencies even when GPU kernels are individually optimized. The paper introduces EasyFHE, a framework designed to simplify the development and optimization of GPU-accelerated FHE applications; similar to PyTorch, it provides high-level interfaces for defining computational logic while automatically handling low-level tasks such as implementation selection and memory management, and it incorporates an optimization framework that systematically addresses performance bottlenecks by applying tailored optimization passes during the lowering from high-level FHE programs to GPU kernels. Compared to state-of-the-art open-source GPU FHE libraries, EasyFHE uniquely supports FHE programs with memory requirements exceeding typical GPU capacities, achieving an average speedup of 2.88 times with a peak of 4.39 times. No numbers or mechanisms beyond the abstract have been introduced here; the specific optimization passes, memory-management design, FHE scheme and parameters, workloads, and hardware should be confirmed in the full paper.