Exploiting Human Color Discrimination for Memory- and Energy-Efficient Image Encoding in Virtual Reality

A headset spends a large share of its energy just moving pixels to and from memory, and the compression that reduces that traffic is always exactly lossless — which is a strange requirement, since the only thing that ever looks at those pixels is a human eye that cannot tell nearby colours apart at all, and can tell even less in the corner of vision where over 90 percent of a headset's pixels live; this work (Rochester and NYU) nudges each pixel within the range nobody can see, purely to make the existing compressor do better.

Why this problem is worth solving

The mechanism being improved is deployed everywhere and constrained in a specific way.

The unspent budget comes from a century of vision science.

What people did before, and where it broke

• Store one reference pixel per tile and encode the rest as differences. The deployed standard, simple enough to run in a display controller and used across graphics, video and display blocks. Where it broke. Nothing about it is wrong — but it is numerically lossless, so it preserves distinctions no human will ever perceive, and its bit cost is dictated entirely by the spread of values in a tile.
• Use a proper image codec instead. Formats that compress far better exist. Where it broke. Too slow. A commercial hardware implementation of one lossless format manages a small image at 20 frames per second, which is nowhere near a display refresh rate.
• Use a video codec. Even better compression, by exploiting similarity between frames. Where it broke. It requires buffering a sequence of frames before compressing, and the latency that introduces is unacceptable in a headset.
• Reduce the rendering work instead, using the same perceptual facts. Foveated rendering does exactly this, drawing the periphery at lower detail, and is well established. Where it broke. Nothing — but it reduces the work of producing pixels, not the traffic of moving them, so it is complementary rather than competing.
• Shrink the palette to only distinguishable colours. The other obvious way to exploit colour discrimination — find the smallest set of colours covering all others, and encode into it. The paper implements this as a baseline. Where it broke. It maps 16.8 million colours down to 32,274, which sounds dramatic but is a global substitution: it does nothing about the spread within a tile, which is what actually sets the bit cost, and it loses to the proposed approach by 50 percent.
• The gap. Everyone treated perceptual slack as something to spend on rendering, and treated memory traffic as a problem for exact compression. Nobody had used the first to help the second, even though the quantity the compressor cares about is precisely the one perceptual freedom lets you shrink.

How the pixels get nudged

The idea

Numerically lossless is the wrong guarantee when the only consumer is an eye. This is the reframing, and its force comes from how large the unspent budget turns out to be. Framebuffer compression is exactly lossless everywhere it ships, which is the correct default for data whose consumer is unknown — but these particular bits are read once, by a display, and looked at by a person. Vision science has spent decades measuring what that person can actually distinguish, and the answer is that around any colour there is a whole region of shades that look identical. Better still, the region grows toward the edge of vision, and a headset with a hundred-degree field of view puts over 90 percent of its pixels outside the central twenty degrees. So there is a large, physiologically measured allowance for changing pixels that nobody will notice, sitting entirely unused while the compressor works hard to preserve differences below the threshold of perception. Discarding information because the actual consumer demonstrably does not need it is the same argument as the satellite imagery work covered elsewhere on this site, made against human vision rather than against downstream analytics.

Do not build a better codec — move the inputs so the existing codec does better. The tactical move is what makes this deployable rather than merely interesting. The deployed compressor’s bit cost is determined by one number: the spread between the largest and smallest value in a tile. That is precisely the quantity perceptual freedom lets you shrink, because each pixel can be moved anywhere inside its own ellipsoid. So the design does not replace anything — it inserts a stage before the encoder, leaves the encoder alone, and leaves the decoder completely unchanged. Writing that down gives a constrained optimisation that is unfortunately non-convex, thanks to a minimum, a maximum, a floor, and the non-linear transform between colour representations, and the authors report that standard solvers get stuck in local optima. Two relaxations rescue it. Optimise along only one colour axis, choosing red or blue rather than green — justified because the ellipsoids are elongated along green precisely because human vision is most sensitive to green and therefore has least freedom there. And work before the non-linear transform rather than after. The constraints are untouched, so perceptual quality is preserved exactly, and what remains has an analytical solution. Spending measured perceptual slack to buy efficiency while holding a real-time budget is the same discipline as the neural rendering work covered elsewhere on this site.

The strongest evidence in the paper is a bad number, reported deliberately. Peak signal-to-noise ratio is the standard objective measure of image compression quality, and the compressed frames here average 46 decibels with a standard deviation of 19.5, with all but two scenes below 37 — a range that ordinarily signals visible artefacts, and which the authors confirm participants do see when the images are shown side by side on an ordinary monitor. Reporting that is the point: it demonstrates that the numerical measure and the subjective experience have genuinely come apart, which is what the whole thesis requires. The subjective result is honest in the same way. Of eleven participants exploring six scenes, an average of 2.8 noticed artefacts, which the authors describe as on par with prior colour perception studies rather than as a clean sweep, and they interview the participants to find out why. One was a visual artist with unusually sensitive colour vision. Some saw artefacts only during rapid head movement, which they attribute to rendering lag rather than the algorithm. And the two dark scenes were worst, which they turn into a request to the vision science community for better discrimination models at low luminance. Introducing error on purpose because the consumer tolerates it, and then measuring how much it actually tolerates, is the same posture as the approximate quantum circuit work covered elsewhere on this site.

Adjusting one tile

This network is not a contribution of this paper — it comes from earlier work and already runs at 72 frames per second on a real headset for under a milliwatt. The central ten degrees are left alone entirely, so nothing changes where vision is sharpest.

Why it works

• The compressor charges for spread, and perception permits shrinking spread. The bits per tile follow from the range between its largest and smallest value → and each pixel may move anywhere inside a region of colours nobody can distinguish from it → so the exact quantity being billed is the one that perceptual freedom is free to reduce.
• Peripheral vision is where both the pixels and the freedom are. Discrimination weakens substantially away from the centre of gaze → and a hundred-degree field of view puts over 90 percent of pixels outside the central twenty degrees → so the largest ellipsoids apply to almost the entire frame.
• Giving up the green axis costs almost nothing. The ellipsoids are elongated along green precisely because vision is most sensitive to green light → so green is where there is least room to move anyway → and abandoning it turns an intractable non-convex problem into one with a closed-form answer while leaving the constraints intact.
• Most tiles can collapse to a single value. If every pixel’s permitted range on the chosen axis shares a common point, all sixteen can be moved onto it → so the differences become exactly zero and consume no bits → which happens for 78.92 percent of tiles and is where the bulk of the saving originates.
• Nothing downstream has to change. The adjustment only rewrites pixel values before they reach the encoder → so the encoder and, critically, the decoder are untouched → which also means the whole stage can be switched off for a user whose colour vision the model does not describe.

What it costs (trade-offs and scope)

• A quarter of participants notice. On average 2.8 of 11 saw artefacts, which the authors report as comparable to prior colour perception studies rather than as zero. Their interviews identify one participant with professionally trained colour vision, artefacts appearing during rapid head movement, and dark scenes being worst.
• The models describe an average person. Published discrimination models target population averages, so individual variation is unaccounted for. A per-user calibration is proposed and described as readily doable, drawing on prior work, but is not performed here.
• Colour vision deficiency is out of scope, explicitly. The underlying model does not describe such users, and the authors say the compression should simply be turned off for them until a suitable model exists — which the plug-and-play structure makes easy, and which is still a real gap.
• Dark scenes are the weak case. The two darkest scenes produced the most noticeable artefacts, and the authors turn this into a request to vision science for better discrimination models at low luminance rather than a fix.
• The user study could not run on the target hardware. Because the algorithm reaches only two frames per second as a shader on a mobile headset — the very reason the hardware exists — the study used a tethered headset driven by a desktop graphics card.
• The tile size is not a free parameter. Compression degrades beyond four-by-four tiles and becomes worse than the unmodified baseline beyond eight-by-eight, because a larger tile must accommodate the worst-case pair of pixels within it.
• The central field of view is excluded. Nothing is adjusted within ten degrees of gaze, so the technique depends on eye tracking being available and correct, and on peripheral pixels dominating — which they do, but by construction.
• The hardware is synthesised, not fabricated. Area, latency and power come from an industrial flow at a 7 nanometre node, with memory energy from a vendor calculator, and the compression is measured against a published implementation of the deployed baseline rather than a shipping product.
• Eleven participants, six scenes. Small even by the standards of perceptual studies, and the scenes are reused from a prior colour perception paper.

The evaluation

Check yourself

Sources

The paper is “Exploiting Human Color Discrimination for Memory- and Energy-Efficient Image Encoding in Virtual Reality” by Nisarg Ujjainkar, Ethan Shahan, Kenneth Chen, Budmonde Duinkharjav, Qi Sun and Yuhao Zhu, at the University of Rochester and New York University, published at ASPLOS 2024 Volume 1, pages 166-180, DOI 10.1145/3617232.3624860. It is open access under a Creative Commons Attribution licence.

This explainer is written from the full text, read from the authors’ copy on arXiv (2310.00441). Everything above comes from the paper’s own sections 2 through 7: the background on colour spaces, discrimination ellipsoids and their dependence on eccentricity, together with the observation that a hundred-degree field of view places over 90 percent of pixels outside the central twenty degrees and that the ellipsoid-generating network already runs at display refresh rate for under a milliwatt; the description of base-and-difference framebuffer compression, its bit cost formula, why video and offline image codecs are unsuitable, and the roughly 800-fold energy ratio between moving a byte through memory and computing on one; the section 3 formulation of pixel adjustment as constrained optimisation, the statement that it is non-convex and that popular solvers reach local optima, the two relaxations restricting the optimisation to one axis and working before the non-linear transform, the justification that ellipsoids are elongated along green because vision is most sensitive there, and the two cases with the measurement that 78.92 percent of tiles admit a common plane; the section 4 hardware design with its pipelined processing elements sized to match the graphics processor throughput, its area and power figures, and the observation that the algorithm reaches only two frames per second as a shader on a real headset; the section 5 methodology covering synthesis at 7 nanometres, memory energy estimation, the exclusion of the central ten degrees, the four baselines including the palette-reduction alternative, and the tethered study apparatus with its stated reason; and the section 6 results covering traffic reduction against each baseline, power savings at different resolutions, the objective quality discussion, the participant study with its interviews and three explanations, and the tile size sensitivity study.