Scientific simulations now produce more data than can be stored or moved, and compressing it only helps if the compressor is not itself the new bottleneck — a modern accelerator link carries 900 gigabytes per second, and to keep it busy a compressor must run faster still, by exactly the factor it compresses; meanwhile the data must often be compressed on a graphics processor and decompressed on an ordinary one, which almost no existing compressor supports; this paper (Texas State) builds four algorithms under a single self-imposed constraint — every step must parallelise well on both kinds of processor — which rules out the standard techniques and forces genuinely different ones, reaching over 500 gigabytes per second on a consumer graphics card while still compressing among the best.
The argument in one glance
-
Problem
The data outgrew the pipes, and the compressor must not become the pipe
Scientific simulations produce far more data than can be comfortably stored or moved. Processing got faster; storage and transfer did not keep up. Compression is the obvious answer, but it only helps if the compressor itself is not the new bottleneck — and the bar is higher than it looks. A current accelerator link carries 900 gigabytes per second and the current expansion bus 242. To keep such a link saturated, a compressor must consume uncompressed data at the link speed MULTIPLIED BY its own compression ratio. Very few compressors are anywhere near that.
-
What people did
Pick a point on the well-understood trade-off
A large literature offers real choices. Special-purpose floating-point compressors predict each value from earlier ones and store only the residual. General-purpose compressors apply Lempel-Ziv matching, Huffman or arithmetic coding, and the Burrows-Wheeler transform. These work, and the trade-off between them is well understood: the ones that compress well are slow, and the fast ones compress poorly. Users choose a point on that curve according to whether storage or time is scarcer.
-
Why it fell short
Nobody was serving both ends at once, or both kinds of processor
The problem is that scientific workflows need both ends of the curve, and a third thing besides. Data is routinely generated and compressed on a graphics processor and then decompressed and analysed on an ordinary one, or the reverse — yet of the eighteen compressors surveyed here, exactly ONE besides this work runs compatibly on both. The strongest general-purpose compressors rely on Lempel-Ziv matching, which is hard to parallelise at all and especially hard on a graphics processor. And the best floating-point predictor keeps two hash tables per thread, which is untenable when there are tens of thousands of threads.
-
This paper's idea
Impose portability as a constraint, and let it select the techniques
This paper (Texas State — Azami, Fallin and Burtscher) inverts the usual order. Rather than designing an algorithm and then porting it, they admit only transformations that parallelise well on BOTH kinds of processor, and build within that constraint. It rules out the standard tools and forces substitutes: a hash-table predictor becomes a SORT, variable-length codes become fixed-width stripping so every value decodes independently, and a bitmap of which bytes are zero is itself compressed recursively, 16384 bits down to 32. They searched over 100,000 machine-generated combinations, studied what the winners had in common, and invented new transformations from that. On a consumer graphics card the fastest of the four compresses and decompresses at over 500 gigabytes per second while still compressing among the best.
Why this problem is worth solving
Floating-point data is famously hostile to compression, and the reason points
directly at the fix.
A
floating-point number stores a SIGN, an EXPONENT giving its rough magnitude, and
a MANTISSA giving its precise value. Compression works by finding repetition, and
mantissas do not repeat — worse, as values pass through arithmetic in a
simulation, their low-order mantissa bits become progressively more random, and
random bits cannot be compressed at all. But scientific data is usually SMOOTH:
neighbouring values are close together, which means their exponents and
high-order mantissa bits are nearly identical. The way to exploit that is to
reinterpret each value as an integer — not converting or rounding it, but reading
the same bits as a whole number — and then SUBTRACT each from the one before.
Nearly-equal values yield differences near zero, which as integers means long runs
of leading zero bits. Everything after that is about removing those zeros cheaply.
This is why the algorithms here compress smooth data well and, as the authors
state plainly, are not expected to do well on data that is not smooth
Judging such a compressor needs two numbers at once, and the paper is careful
about how it presents them.
Compression has two
outputs that matter — how small the result is, and how fast you got there — and
almost every technique trades one against the other, so quoting either alone is
meaningless. The honest presentation is a PARETO FRONT: plot every compressor as
a point with ratio on one axis and throughput on the other, and mark those for
which no other compressor is BOTH faster and better-compressing. Sitting on that
front is a defensible claim; being fastest overall usually is not. The scale
matters too. Lossless ratios on floating-point data are modest — around 1.4 to
1.8 here, against roughly 2.1 for the best-compressing competitor — because the
random mantissa tail simply cannot shrink. What distinguishes the results is that
these ratios come at hundreds of gigabytes per second rather than tens of
megabytes
What people did before, and where it broke
- Predict each value and store the difference. Floating-point compressors such as FPzip and FPC predict the next value from earlier ones and encode only the residual. FPC uses two hash tables, picks whichever prediction was closer, and strips leading zero bytes. These compress well and remain competitive.
- But the predictor state does not survive parallelism. Two hash tables per thread is workable with a handful of threads and impossible with tens of thousands, which is why the best predictor-based method has no graphics-processor implementation.
- Apply general-purpose compression. Gzip, Bzip2 and Zstandard combine Lempel-Ziv matching with Huffman or arithmetic coding, and Zstandard at its highest setting still achieves the best compression ratio measured in this paper.
- But Lempel-Ziv resists parallelisation. Finding repeated substrings is inherently sequential in the way that matters, which the authors give as their explicit reason for excluding it, and the cost shows: the best ratio comes at throughputs measured in tens of megabytes per second.
- Build for the graphics processor specifically. GFC, MPC and NVIDIA’s own Bitcomp are designed for parallel hardware and are genuinely fast. One of them reaches the Pareto front on compression throughput at a ratio of 1.04 — which, as the authors note, is not a useful amount of compression.
- But none of them runs on an ordinary processor. Since data is compressed on one machine and read on another, a compressor that exists on only one side is awkward to deploy. Of eighteen compressors surveyed, one besides this work is compatible across both.
- The gap. Everyone treated portability as an implementation concern to be addressed after the algorithm was chosen. Nobody had tried making it the CONSTRAINT that selects the algorithm in the first place — which turns out to exclude most of the standard toolkit and force genuinely different techniques.
How portability picks the techniques
Manufacture leading zeros, then remove them in ways that parallelise on both processors
-
Predict values and encode the residuals
The classical approach keeps predictor state, guesses each value from earlier ones, and stores only the difference. It compresses well and is the basis of the strongest floating-point compressors, but the per-thread state does not survive massive parallelism.
-
Read the float as an integer and subtract its predecessor new
Reinterpret each value bit-for-bit as a 32 or 64-bit integer, without converting or rounding, then subtract the previous one. Because smooth data has nearly-equal neighbours with nearly-identical exponents, the differences cluster near zero. The first value in each block is kept as-is.
-
Turn leading ones into leading zeros new
Differences can be negative, and a negative integer in the usual representation begins with a long run of ONES rather than zeros. A single reversible operation — shift left by one and combine with the sign-extended shift right — moves the sign to the bottom and leaves every value with leading zeros regardless of direction.
-
Strip a fixed number of leading zeros per block new
Find the largest value in a block, count its leading zeros, and remove exactly that many bits from every value. Fixing the count wastes some opportunity, but it means each value decodes independently rather than requiring the previous one — which is what makes parallel decompression possible at all.
-
Or transpose the bits so the zeros line up, then delete them new
For the higher-compression variant, gather the first bit of every value together, then all the second bits, and so on. The leading zeros of many values become long runs of zero bytes. Record which bytes are zero in a bitmap, drop them, and then compress the bitmap by the same idea recursively.
-
For doubles, find the boundary between signal and noise new
Arithmetic randomises low mantissa bits and doubles have more of them, so the bottom of each word is genuinely incompressible. Choose per block how many top bits are worth processing, by building a histogram of leading-zero counts and picking the split that minimises output, then compress only that part.
The idea
Let the constraint choose the algorithm. The usual order is to design a compressor and then work out how to parallelise it, which is how the field ended up with excellent compressors that exist on only one kind of processor. This paper reverses that: a transformation is admissible only if it parallelises well on both, and everything else follows. The exclusions are specific and consequential. Lempel-Ziv matching, the engine of nearly every general-purpose compressor, is out because it does not parallelise. The strongest floating-point predictor is out because it keeps hash tables per thread. Even variable-length encoding of each value is out, because a value whose length depends on its content cannot be decoded without decoding its predecessor. What remains is a smaller toolkit than the field normally works with — differences, bit permutations, zero elimination, prefix sums — and the paper’s contribution is showing that this restricted set, combined carefully, reaches the frontier anyway. Treating a deployment requirement as a design constraint rather than a later porting exercise is the same shape covered elsewhere on this site where a system is built from primitives that survive the environment it must run in, rather than from the technique that would be best if the environment were free.
Every step manufactures the structure the next one exploits. The pipelines read as a chain in which each transformation exists to create the raw material for the following one. Reinterpreting a float as an integer makes arithmetic on its bit pattern meaningful. Subtracting consecutive values turns nearly-identical exponents into near-zero differences. Converting to magnitude-sign form turns the leading ONES of negative differences into leading zeros, so both directions look the same to what follows. Bit transposition gathers those leading zeros from many values into long runs of zero bytes. Zero elimination then removes the runs and records their positions in a bitmap — and since that bitmap is itself mostly zeros at the front and mostly ones at the back, the same idea is applied to it recursively, shrinking 16384 bits to 2048, then 256, then 32. None of these steps compresses much alone; the compression is in the composition. Building a pipeline where each stage’s job is to make the next stage’s job possible is the same compose-cheap-transformations discipline covered elsewhere on this site where a sequence of individually modest rewrites achieves what no single clever step could.
Replace stateful lookup with a sort. The most striking substitution answers a specific problem. The best predictor for double-precision data works by hashing recent history and looking up whether that context appeared before — excellent on a processor with a few threads, hopeless with tens of thousands each needing its own tables. The replacement computes, for every position at once, a pair of the hash of the preceding three values and the position’s own index, then SORTS those pairs. Sorting brings equal hashes together, and because each pair carries its index, an earlier occurrence is immediately adjacent. A lookup structure has become a data-parallel primitive. The step doubles the data, which sounds like a loss and is not, because half the result is zeros and the rest compresses far better downstream. Decompression is a chain-following problem, which is normally sequential, and they make it parallel by observing that it is exactly the FIND operation of union-find — threads shorten each other’s chains as they go. Reformulating a stateful algorithm as a sort so it becomes data-parallel is the same change-the-primitive move covered elsewhere on this site where an operation is not made faster but replaced by a different one with the parallel structure the hardware rewards.
Three numbers becoming fewer bits
The paper's own worked example, following three single-precision values
The higher-compression single-precision pipeline: difference, magnitude-sign, bit transposition, zero elimination.
Step 1 of 5
Read the floats as integers and subtract
Three nearby values are reinterpreted bit-for-bit as 32-bit integers — no rounding, no conversion, the same bits read a different way. The first is kept as-is. Each of the others becomes its difference from the value before it. Because the three values are close together, their exponent bits are identical and cancel.
the exponent lives in the high-order bits, so identical exponents mean the difference has zeros exactly where compression needs them the subtraction wraps around rather than overflowing, which is what makes it exactly reversible
Why it works
- Smooth scientific data has nearly-identical neighbouring exponents → integer differencing collapses the high-order bits to zero → giving every downstream stage long runs of leading zeros to remove.
- Magnitude-sign conversion maps both leading ones and leading zeros to leading zeros → one stripping stage handles positive and negative differences alike → removing the need for any branch or second code path.
- A fixed number of bits is stripped per block rather than per value → every value occupies a known position → so decompression of each value is independent, which is the property that makes it run at hundreds of gigabytes per second.
- The zero-position bitmap is itself highly structured → applying the same elimination recursively shrinks it from 16384 bits to 32 → turning what would be a fixed overhead into a negligible one.
- Sorting replaces per-thread hash tables, and chain-following decompression becomes union-find → both halves of the predictor become data-parallel → so the highest-compression double-precision algorithm runs on a graphics processor at all, and decompresses far faster than it compresses since only compression sorts.
What it costs (trade-offs and scope)
- It is built for smooth data and says so. Every stage assumes neighbouring values are close. The authors state directly that they do not expect good compression on data that is not smooth, and their evaluation set was chosen because scientific data generally is.
- The best ratio still belongs to someone else. On double-precision data on an ordinary processor, Zstandard at its highest setting compresses more than any of these four — at a throughput lower by orders of magnitude, which is the trade being made, but the claim is Pareto-optimality rather than dominance.
- The high-ratio double-precision algorithm is slow to compress. Its sorting stage makes compression over ten times slower than its own decompression. That is a reasonable shape for write-once-read-many data and a bad one otherwise.
- Stripping a fixed width leaves compression on the table. Every value in a block loses the same number of leading bits even when it could lose more, which is the price of independent decoding. Splitting blocks into 32 sub-blocks recovers some of it, not all.
- The design came partly from automated search. Over 100,000 candidate algorithms were generated and the best analysed, which is an honest and effective method but means the choices are justified empirically rather than derived, and it is not obvious how they would transfer to a different data profile.
The evaluation
The comparison is unusually broad: 19 compressors in total, including ten built specifically for floating-point data and eight general-purpose ones, run over 90 single-precision files from seven scientific domains and 20 double-precision files from five. Hardware covers two processors and two graphics processors of different generations. Results are reported as geometric means computed per dataset and then across datasets, deliberately, so that domains contributing more files do not dominate. Competitors requiring extra information — array dimensions or tuple sizes — were given it, though the algorithms here need neither.
Losslessly compressing scientific floating-point data
| Property | General-purpose compressors | Graphics-processor compressors | Portable transformation pipelines |
|---|---|---|---|
| Runs on both kinds of processor | |||
| Core technique | |||
| Best compression ratio reached | |||
| Throughput at that ratio | |||
| Position on the Pareto front |
Check yourself
Check your understanding
1. After subtracting consecutive values, the algorithm converts each difference from the usual signed representation into magnitude-sign form. Predict what that buys.
2. The stripping stage finds the largest value in a block and removes that many leading zeros from EVERY value, even though most values have more zeros that could be removed. Predict why it gives up that compression.
3. The strongest double-precision predictor uses hash tables to find whether a context appeared before. This design replaces that with a SORT. Predict why sorting is the right substitute.
4. The single-precision and double-precision high-ratio algorithms are genuinely different rather than the same code at two widths. Predict the physical reason.
Sources
- Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs (author PDF, Texas State University) — ASPLOS 2025; the full 15-page paper, read in full for this explainer.
- ACM Digital Library (DOI 10.1145/3669940.3707280) — published version.