DefT: Boosting Scalability of Deformable Convolution Operations on GPUs

Deformable convolutions let a CNN bend its sampling grid to follow the shape of what it sees, which sharply boosts accuracy for only 1 percent more compute, yet a realistically sized deformable layer runs about 4 times slower on a GPU because the bilinear-interpolation step that gathers the shifted pixels has wildly irregular, input-dependent memory access and almost no arithmetic per byte; this paper (Duke — Edward Hanson, Mark Horton, Hai Li, Yiran Chen) presents DefT, which fuses that memory-bound step with the offset convolution to raise its arithmetic intensity and adapts the GPU tiling to the statistically-predictable offset distribution so sampled pixels stay on-chip, halving the slowdown for up to 134 percent layerwise speedup and 46 percent less training time.

The argument in one glance

Problem

A smarter convolution that bends to the shape of things, but it will not scale

An ordinary convolution samples the input on a rigid square grid, which makes it awkward at handling objects that rotate, stretch, or shear. A deformable convolution fixes this by learning, for each position, small offsets that shift where the filter samples — so the grid bends to follow the actual shape of what is in the image. This adaptive sampling boosts accuracy a lot for almost nothing: dropping deformable layers into an object detector raised its detection score by more than ten points while adding only about 1 percent to the parameters and compute. The natural next step is to use more of them, earlier in the network, the way vision transformers scaled up attention. But that scaling has not happened, because deformable layers are slow on GPUs: a realistically sized one runs about four times slower than a plain convolution, which discourages using them widely.
What people did

Just run the deformable layer as three standard operators

The standard implementation, as in PyTorch, splits the deformable layer into three operators run back to back: an offset convolution that computes where to sample, a bilinear interpolation that fetches and blends the shifted input pixels into a deformed feature map, and a main convolution over that deformed map. Each is a familiar tensor operation, so each is handed to an existing GPU library and executed as its own kernel.
Why it fell short

The interpolation step is memory-bound and drags the whole layer down

The middle operator is the problem. The learned offsets are fractional, so each one lands between pixels and the interpolation must gather the four neighboring input pixels around it and average them by weight. Because the offsets are input-dependent, these gathers are irregular and unpredictable, and the operator does very little arithmetic per byte it reads — its arithmetic intensity is disproportionately low. On a GPU that makes it memory-bound: it stalls waiting on scattered global-memory accesses while the compute units sit idle, and this one low-intensity step is what inflates the whole deformable layer to a four-times slowdown.
This paper's idea

Fuse the memory-bound step to raise its intensity, and tile to the offset statistics

DefT is a set of workload-aware GPU optimizations for deformable layers. Its first move is to fuse the offset convolution with the bilinear interpolation into one kernel: the two share the same input feature map, so fusing lets the interpolation consume the offsets on-chip immediately instead of round-tripping through global memory, which raises the fused kernel arithmetic intensity by 7.6 times and lifts it off the memory bound. Its second move is to notice that the offsets are not arbitrary — across images they follow a predictable, tight distribution — and to adapt the GPU tiling and a padding scheme to that distribution so the pixels each tile needs stay inside the on-chip tile, minimizing costly out-of-bounds global accesses. The main convolution is left untouched so it can keep using the highly-tuned library GEMM. DefT halves the slowdown, giving up to 134 percent layerwise speedup and 46 percent less training time.

Why this problem is worth solving

The value of deformable convolution is real, which is exactly why its poor scaling is worth fixing.

A standard convolution reads its input at fixed grid locations relative to each output position; a deformable convolution learns an extra set of offsets — computed by a small offset convolution over the same input — that displace those sampling locations. This lets the receptive field deform to match geometric structure in the image, so the network becomes robust to rotation, scaling, and shearing without needing to be trained on masses of augmented data. The effect is likened to attention: it is a flexible, input-dependent inductive bias, and it pays off dramatically. Adding deformable layers to a strong detector improved its box detection score by over ten points while increasing parameters and floating-point operations by only about 1 percent. Because the accuracy gain is so cheap in raw operation count, the field wanted to do what worked for transformers — use the mechanism more heavily, more layers of it, placed earlier — expecting further gains. That scaling stalled not because the accuracy stopped improving but because the wall-clock cost exploded, which means the barrier is an implementation-efficiency problem, not a modeling one. Solving it unlocks a modeling direction that is otherwise strangled by its GPU runtime.

The reason it is slow is a classic hardware mismatch hiding inside one operator.

Arithmetic intensity is the ratio of compute operations to bytes moved from memory; a GPU delivers its peak throughput only when a kernel does enough arithmetic per byte to keep the compute units fed while data streams in. A kernel with low arithmetic intensity is memory-bound: it spends its time waiting on memory and leaves the arithmetic units idle. The bilinear interpolation in a deformable layer is exactly such a kernel. For every sampled position it must fetch four neighboring input pixels around a fractional coordinate and combine them with a handful of multiply-adds — very little arithmetic for the bytes it touches — and because the sampling coordinates come from learned, input-dependent offsets, the fetches are irregular and scattered rather than the neat contiguous streams GPUs favor. So the operator sits far below the machine’s compute roofline, bottlenecked on scattered global-memory traffic. The rest of the deformable layer, the two ordinary convolutions, are compute-bound and efficient, but a pipeline runs at the speed of its slowest stage, so this one disproportionately low-intensity operator sets the pace and produces the four-times slowdown. Fixing the layer means fixing the intensity and the access pattern of that single operator without disturbing the parts that already run well.

What people did before, and where it broke

Run the three operators as separate library kernels (PyTorch). Execute the offset convolution, the bilinear interpolation, and the main convolution each as its own GPU kernel using existing libraries. Where it broke. The interpolation kernel is memory-bound and its output, an unrolled deformed feature map, is materialized in global memory; this both wastes bandwidth and, in the standard implementation, prevents the main convolution from using the fastest library path, so the layer stays far below peak.
Add more deformable layers to chase accuracy. Following evidence that more deformations earlier in the network help, simply scale up their use. Where it broke. Each added deformable layer multiplies the slow interpolation cost; an all-deformable ResNet incurs a 46 percent slowdown, so the accuracy trend could not be pursued because the runtime became prohibitive.
Specialized hardware for the interpolation. Prior work proposed custom accelerators to raise the operator’s effective arithmetic intensity or constrained the deformations to regularize its access. Where it broke. These require new hardware or restrict the model, rather than making the operator fast on the commodity GPUs practitioners already use to train and deploy.
Generic operator fusion of all three operators. Fuse the whole deformable layer into a single kernel so intermediate feature maps never touch global memory. Where it broke. Fusing in the large main convolution forgoes the benefit of the highly-optimized library GEMM that already runs it efficiently, so folding everything together loses more than it saves.
The gap. No approach targeted precisely the low-intensity, irregular interpolation operator — raising its arithmetic intensity by fusing it with the offset convolution while leaving the efficient main convolution to its library, and shaping the GPU tiling to the statistical structure of the offsets so the scattered accesses stay on-chip.

How DefT works

Turning the memory-bound interpolation into an on-chip, compute-friendly kernel

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Attack the arithmetic intensity of the one operator that is memory-bound. The whole diagnosis is that a deformable layer is fast everywhere except the bilinear interpolation, which does too little math per byte and gathers pixels from scattered, input-dependent locations. So DefT does not try to speed up the layer in general; it targets that operator’s intensity directly. By fusing the interpolation with the offset convolution that feeds it, the offsets are produced and consumed on-chip in one kernel over the same shared input feature map, so the bytes that used to stream through global memory as an intermediate deformed feature map never leave the chip, and the ratio of useful arithmetic to memory traffic jumps — a 7.6-times increase in arithmetic intensity for the fused kernel. That is enough to lift the operator off the memory bound and toward the compute roofline where the rest of the layer already lives. Recognizing that a pipeline is throttled by a single low-intensity, memory-bound stage and fusing precisely to raise that stage’s compute-to-memory ratio is the same move the optimized-attention-dataflow work covered elsewhere on this site is built on, restructuring a memory-bound operator so its intermediate data stays on-chip instead of being paid for in bandwidth.

Let the workload’s statistics shape the tiling, because the offsets are not random. Fusing gets the data on-chip only if a tile’s worth of it actually fits, and the danger with deformable sampling is that an offset could reach anywhere, forcing an expensive out-of-bounds trip to global memory. DefT’s insight is that although the offsets are input-dependent, they are not arbitrary: measured across a dataset they form a tight distribution concentrated near zero — most offsets land within a pixel or so of the base position, and they shrink further in the deeper, spatially smaller layers. DefT turns this statistical regularity into a concrete tiling and padding decision, padding each tile by just enough margin that the overwhelming majority of sampled pixels stay inside the on-chip tile, so out-of-bounds accesses become rare rather than routine. Adapting the tile shape to the measured access behavior of the workload, rather than choosing a fixed tiling blind to it, is the same discipline the dynamic reflexive tiling work covered elsewhere on this site applies to irregular data, letting the observed structure of the accesses dictate how the computation is blocked so the common case stays local.

Optimize surgically, and leave the already-fast parts to the experts. A tempting but wrong generalization would be to fuse all three operators into one megakernel so no intermediate ever touches memory. DefT explicitly refuses this, because the main convolution is large and is already executed near peak by a decades-tuned library GEMM; folding it into a custom fused kernel would forfeit that efficiency and lose more than the fusion saves. So DefT fuses only the two operators that need help and leaves the main convolution as its own kernel on the library path, preserving what is already good while fixing what is broken. This is a deliberate scoping decision: change the memory-bound operator and nothing else. Improving a tensor computation on a GPU by respecting the existing highly-optimized kernels and intervening only where the hand-tuned libraries fall short is the same pragmatic stance the tensor-computation IR work covered elsewhere on this site takes toward GPU code, generating specialized kernels for the cases that need them while deferring to established library routines where they already win.

Following a deformable layer through DefT

Running one deformable convolution without stalling on interpolation

A trace, drawn from the paper's design, of how DefT executes a deformable layer that the standard implementation would run memory-bound.

Step 1 of 5

The offsets are computed for each sampling position

The offset convolution reads the input feature map and produces, for every filter position at every output location, a small fractional (x, y) offset that shifts where the main filter will sample. These offsets are the learned deformation, and they are input-dependent, so their pattern differs from image to image.

The offsets are added to each base sampling coordinate; because they are fractional, they rarely land on a single pixel and instead fall among four neighbors.

Why it works

Fusion converts wasted bandwidth into on-chip reuse. The interpolation and offset convolution read the same input and pass an intermediate between them → but fusing them keeps that intermediate on-chip and reuses staged input → so the operator’s arithmetic intensity rises 7.6 times and it stops being memory-bound.
Tiling puts the scattered gathers into fast memory. Bilinear interpolation gathers four irregular neighbors per sample from global memory → but tiling by the output’s spatial dimensions fits each block’s inputs into registers and shared memory → so the gathers hit on-chip storage instead of stalling on scattered global accesses.
The offset distribution makes padding cheap and effective. An input-dependent offset could in principle reach anywhere, forcing out-of-bounds fetches → but measured offsets cluster tightly near zero, so a modest tile padding captures almost all of them → so out-of-bounds accesses become rare and the common case stays local.
Leaving the main convolution alone preserves what is already fast. Fusing all three operators would pull the large convolution off its tuned library path → but DefT fuses only the two low-intensity operators and keeps the main convolution on the library GEMM → so the layer gains on interpolation without sacrificing the efficient convolution.
Fixing the slowest stage lifts the whole layer. The layer ran at the speed of its memory-bound interpolation → but raising that operator’s intensity and locality removes the bottleneck → so the reported slowdown roughly halves, yielding up to 134 percent layerwise speedup and 46 percent less training time on a fully deformable ResNet.

What it costs (trade-offs and scope)

The tiling benefit rides on the offset distribution holding. DefT’s padding is sized to the statistically observed, near-zero-concentrated offsets; a workload whose learned offsets are unusually large or diffuse would push more samples out of bounds, eroding the on-chip advantage. The gains are strongest where the deformation is moderate, which is the common regime but not guaranteed.
It optimizes the operator, not the model’s accuracy. DefT changes runtime without altering functional correctness, so it makes deformable layers cheaper to run but does not itself improve what they learn; its value is in removing the efficiency barrier that discouraged using more of them, not in a new modeling capability.
Fusion and tiling add implementation complexity. Writing a fused, tiled, swizzled custom kernel with workload-aware padding is more involved than calling three library operators, and the tile sizing depends on both the GPU platform and the layer dimensions, so the optimization must be tuned per platform and per layer shape rather than applied blindly.
The main-convolution decision is a deliberate scope limit. By leaving the main convolution to the library, DefT does not attempt whole-layer fusion; on layers or hardware where the main convolution is not the dominant cost, or where a different fusion boundary would help, the chosen split may not be optimal.
Evidence is one GPU and specific models. Results come from an NVIDIA RTX 3090 on deformable Mask RCNN (COCO) and an all-deformable ResNet-18 (ImageNet); the up-to-134-percent and 46-percent figures are for these settings, and behavior on other GPUs, batch sizes, or model configurations is inferred rather than exhaustively shown.

The evaluation

Executing a deformable convolution layer on a GPU

Property	PyTorch (three kernels)	Fuse all three operators	DefT
Bilinear interpolation arithmetic intensity
Main convolution path
Handling irregular sampling
Cache reuse across tiles
Reported outcome

A comparison drawn from the paper. DefT is evaluated on an NVIDIA RTX 3090 against the current-art PyTorch implementation, on deformable Mask RCNN (COCO) and a fully deformable ResNet-18 (ImageNet); the 7.6-times, 134-percent, and 46-percent figures are the paper's reported results.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The full paper text was available (an open-access copy from the NSF public-access repository, retrieved via the Internet Archive), so this account of the mechanism — the three deformable-layer operators, the memory-bound low-arithmetic-intensity bilinear interpolation, the fusion of the offset convolution with the interpolation, the output-spatial tiling, the offset-distribution-driven tile padding, the block-ID swizzle, and the decision to leave the main convolution on the library GEMM — follows the paper’s own description. The headline numbers are the paper’s reported results: a 7.6-times arithmetic-intensity increase for the fused kernel, up to 134 percent layerwise speedup over the current-art PyTorch implementation, and a 46 percent reduction in normalized training time on a fully deformable ResNet, measured on an NVIDIA RTX 3090. Those figures are for specific models (deformable Mask RCNN on COCO, ResNet-18 on ImageNet) and one GPU; per-layer results vary with layer shape and offset distribution. Exact per-configuration numbers, the offset-distribution parameters, and the tiling and swizzle details should be read from the paper itself.

Background

Why a dynamic, input-dependent network is hard to run fast

It helps to see what makes deformable convolution awkward for hardware. A plain convolution is a static computation: the same fixed pattern of memory accesses runs no matter what the input is, which is exactly what lets GPUs and their libraries schedule it near peak. A deformable convolution is a dynamic neural network — its sampling locations are decided at runtime from the input itself, so the memory-access pattern changes with every image and cannot be laid out neatly in advance. This input-dependence is the source of both its power (it can adapt to the content) and its performance pain (the hardware cannot rely on a predictable access pattern). The same tension appears wherever a network’s shape or work is decided at runtime rather than fixed at compile time, and it is why such models resist the standard, statically-tuned acceleration path. That challenge of running networks whose computation is not fixed ahead of time, but varies with the input or the moment, is the same problem the dynamic-workload scheduling work covered elsewhere on this site is built around, where the system must cope with computation whose demands are only known at runtime. DefT’s contribution is to show that even a dynamic operator can be made GPU-friendly by exploiting the statistical regularity hiding inside its apparent irregularity.

If you remember only three things

1. Deformable convolutions bend a CNN's sampling grid to follow the geometry of the input, boosting accuracy substantially (over ten detection points) for only about 1 percent more compute, but they do not scale on GPUs: a realistically sized deformable layer runs about 4 times slower, and an all-deformable ResNet 46 percent slower, because the bilinear-interpolation operator that gathers and blends the four neighbors around each fractional, input-dependent sampling coordinate has irregular access and disproportionately low arithmetic intensity, making it memory-bound and setting the whole layer's pace.
2. DefT is a set of workload-aware GPU optimizations that target exactly that operator: it fuses the offset convolution with the bilinear interpolation so the intermediate stays on-chip and the fused kernel's arithmetic intensity rises 7.6 times; it tiles the fused work by the output's spatial dimensions and, exploiting the statistically tight, near-zero offset distribution, pads each tile so sampled pixels stay in-boundary and out-of-bounds global accesses become rare; a block-ID swizzle improves L2 reuse; and it deliberately leaves the large main convolution on the highly-optimized library GEMM rather than fusing all three operators.
3. On an NVIDIA RTX 3090 against the current-art PyTorch implementation, DefT roughly halves the deformable-layer slowdown, delivering up to 134 percent layerwise speedup and a 46 percent reduction in normalized training time on a fully deformable ResNet, without changing functional correctness. Gains depend on the offset distribution holding, the tiling being tuned per platform and layer shape, and the results are demonstrated on specific models and one GPU.

Check yourself

Check your understanding

1. Why is a deformable convolution layer so much slower on a GPU than a standard convolution, despite adding little compute?
2. What is DefT's first main optimization, and why does it help?
3. How does DefT keep the irregular, input-dependent pixel gathers from causing costly global-memory accesses?
4. What are DefT's main results, and what should a reader keep in mind about them?

Sources

The paper is “DefT: Boosting Scalability of Deformable Convolution Operations on GPUs” by Edward Hanson, Mark Horton, Hai (Helen) Li, and Yiran Chen (Duke University), published at ASPLOS 2023 (Volume 3), DOI 10.1145/3582016.3582017.

This explainer is based on the paper’s full text, obtained from an open-access copy in the NSF public-access repository (PAR ID 10435119), retrieved via an Internet Archive snapshot when the direct host was unreachable. The account of the mechanism — the three deformable-layer operators (offset convolution, bilinear interpolation, main convolution), the low-arithmetic-intensity and irregular-access nature of the bilinear interpolation that makes it memory-bound, the fusion of the offset convolution with the interpolation to raise arithmetic intensity, the output-spatial tiling to fit data on-chip, the offset-distribution analysis (a tight distribution concentrated near zero) that drives a tile-padding scheme to keep samples in-boundary, the block-ID swizzle for L2 reuse, and the decision to leave the large main convolution on the optimized library GEMM rather than fusing all three operators — follows the paper’s own description. The reported results cited here are the paper’s own: a 7.6-times arithmetic-intensity increase for the fused kernel, up to 134 percent layerwise speedup over the current-art PyTorch implementation, and a 46 percent reduction in normalized training time on a fully deformable ResNet, measured on an NVIDIA RTX 3090 with deformable Mask RCNN (COCO) and ResNet-18 (ImageNet). Exact per-configuration numbers, the offset-distribution parameters, and the tiling and swizzle details should be read from the paper itself.