SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile

On a phone GPU, a modern vision Transformer spends 43% to 70% of its time not computing but rearranging tensors — reshaping and transposing them so the next operator will accept them — because the efficient local-attention designs buy their lower arithmetic cost by chopping the input into windows; SmartMem (William and Mary, NC State, Georgia and UT Arlington) argues that almost all of that rearranging is avoidable, classifying every operator by whether its speed depends on the incoming layout and whether its outgoing layout can be dictated, using that two-by-two grid to decide for each producer-consumer pair whether to fuse, delete one side or delete both, and then choosing the surviving layouts so the producer writes in whatever shape the consumer wants to read.

Why this problem is worth solving

Running a model on the phone instead of in the cloud means the data never leaves the device, which is both a privacy property and the only way to work offline.

The reason that hurts so much here rather than on a server is the memory.

What people did before, and where it broke

• Operator fusion, in every mobile framework. Merge adjacent operators into one kernel so the intermediate tensor never reaches memory. This is well-understood, widely deployed and genuinely effective on convolutional networks. Where it broke. Fusing a reshape does not remove the data movement, it relocates it — the fused kernel still reads elements in a scattered order, and now carries the index arithmetic to compute where each one moved. The paper notes that after fusing several layout transformations together, the memory access pattern is fragmented and hard to optimise further.
• Let the framework insert the conversions it needs. Frameworks define a preferred input layout per operator and silently insert reshapes and transposes to satisfy them. This is what makes it possible to compose operators from different families at all. Where it broke. The conversions are invisible to the model author and are not free. The paper separates them out and measures them as implicit transformation cost, which on the Transformer models runs between 11% and 19% of total runtime on top of the explicit ones.
• Choose data layouts jointly with loop transformations. There is a long compiler literature on exactly this, and the paper names it as the closest prior work. Where it broke. All of it was aimed at a single nested loop, motivated by scientific codes. Here the problem is choosing layouts across a whole directed graph of operators, where a producer may have several consumers wanting different things and every choice propagates.
• Search for the globally best set of layouts. This is the principled formulation, and the paper notes the problem maps onto a known NP-hard graph problem. Where it broke. The search space over a large model is far too big to be practical inside an inference framework that has to produce a plan quickly.
• The gap. Nobody had asked which operators actually need a particular layout and which are simply being served one out of habit. Without that distinction, a framework has no basis for deciding whether a transformation between two operators is necessary, and so inserts it.

How SmartMem removes a layout transformation

The idea

The expensive part of a modern mobile model is the space between the operators, not the operators. The paper opens with a measurement table rather than a technique, and it is the table that carries the argument. Three older convolutional networks spend at most a couple of percent of their time on layout transformation. Six Transformer and hybrid models spend between 43% and 70%, split between explicit reshapes and transposes the author wrote and implicit ones the framework inserted. Their achieved throughput is roughly an order of magnitude lower per unit of arithmetic as a result. The efficient local-attention designs made the arithmetic cheap and paid for it in shuffling, and on a device with narrow memory bandwidth that is a bad trade nobody had priced. Discovering that the real bottleneck of an attention-based model is moving data rather than multiplying it is the same finding as the attention dataflow work covered elsewhere on this site, which reached it from the accelerator side rather than the mobile one.

Ask what each operator actually demands, and what it can merely be talked into. The classification is the contribution, and it is deliberately coarse — two yes-or-no questions. Does performance depend on the layout coming in? Matrix multiplication reuses data across iterations, so yes; elementwise addition touches everything once, so no. Can the layout going out be chosen freely? A convolution can be told to write in whatever order suits the next operator, so yes; a transpose has an output layout that is its definition, so no. Those four boxes are enough to decide every edge in the graph mechanically. Anything whose output is fixed is a pure rearrangement and can be deleted outright, turning into index arithmetic in whatever survives. And crucially, the two properties are not the same question asked twice — an operator can be picky about what it reads and completely indifferent about what it writes, and it is that asymmetry the whole method exploits. Treating what a computation demands and how its data is arranged as one joint decision rather than two sequential ones is the same reframing as the format-and-schedule co-optimisation work covered elsewhere on this site.

When two operators disagree, make the writer give in rather than the reader. After the deletions, everything left is sensitive to its input and flexible about its output, so every remaining edge is a negotiation, and SmartMem resolves all of them the same way: the producer emits whatever the consumer wants. What the consumer wants is summarised by a single property, the dimension along which it aggregates — the shared inner dimension of a matrix multiply, the reduced axis of a normalisation — because keeping that dimension contiguous is what lets the consumer stream and vectorise. The justification is a measurement rather than an argument: microbenchmarks show that writing sub-optimally costs less than reading sub-optimally, by 1.7 times on convolution, 1.4 on matrix multiply and 1.1 on activation. Then the chosen layout is mapped onto the two addressable directions of texture memory, with one dimension partitioned into groups of four to match the vector width of a cache line. Choosing where a tensor physically sits by reasoning about the shape of the memory rather than the shape of the tensor is the same discipline as the tensor-computation IR work for GPUs covered elsewhere on this site.

Walking one producer-consumer pair through SmartMem

The same operator can land in different boxes depending on circumstances. The paper notes that an operator may be classified differently according to whether its several operands share a layout or not — addition with two identically laid out inputs is insensitive, but that stops being true if they differ.

Why it works

• A transformation only exists to settle a disagreement. Reshapes and transposes carry no information of their own, they only reconcile two operators with different expectations → if the producer can be told to write differently, the disagreement never arises → the transformation can be deleted rather than optimised, and 21% to 65% of the operators in the graph disappear.
• The two questions being asked are genuinely independent. Whether an operator cares what it reads is a different fact from whether it can be told what to write → an operator can be picky on one side and indifferent on the other → the classification finds flexibility that a framework treating each operator as a fixed contract cannot see.
• The index arithmetic is collapsible because the shapes are static. Every dimension size is known at compile time → chained divisions and remainders can be reduced algebraically instead of executed → the cost of deleting a transformation does not simply reappear as address computation, which is what would otherwise sink the whole approach on a GPU.
• Writing badly is cheaper than reading badly. A scattered write issues its stores once and moves on, while a scattered read destroys locality for every reuse the consumer performs → measured at 1.7 times on convolution and 1.4 on matrix multiplication → so resolving every disagreement in the consumer favour is the right default rather than an arbitrary one.
• The layout is chosen for the memory that exists, not an idealised one. Mobile GPU texture memory is addressed in two directions with four-wide cache elements → mapping the reduction dimensions onto those directions, with one partitioned into groups of four, lets consumers stream contiguously → the result is 1.8 times fewer memory accesses and 2.0 times fewer cache misses on average than the other frameworks.

What it costs (trade-offs and scope)

• It barely helps the models that were already fine. The paper says so in its own limitations section: on traditional convolutional networks such as Yolo-V8 and RegNet, or any model with few layout transformations, the improvement over the baseline framework is modest — 1.2 and 1.3 times. The large gains are concentrated where the problem was, which is honest but narrows the claim considerably.
• The headline multipliers come from frameworks that mostly could not run the models. The 7.9 and 6.9 times geometric means are against MNN and TVM, and the comparison table is full of blanks because two of the five baselines do not support Transformer or hybrid models on a mobile GPU at all. The meaningful comparison is the 2.8 times against DNNFusion — which is the authors own prior work and the framework SmartMem is built on top of.
• Layout selection is a heuristic, and the paper says the real problem is NP-hard. Choosing layouts across the whole graph maps onto a known hard formulation, so SmartMem substitutes a local rule based on the consumer reduction dimension. There is no claim of optimality and no bound on how far from it the result can be.
• Duplicate tensors are kept when consumers cannot agree. If a producer has more consumers wanting distinct reduction dimensions than the memory can index directly, extra copies are materialised. The measured worst case is small, at 3.0 MB and 2.3 MB live for the two models reported, but only two models are reported and the argument that it does not matter rests on a memory pool and on operator deletion offsetting it.
• The classification does not cover everything. An operator can fall into different boxes depending on whether its operands share a layout, and the paper works through a curated set of common operators. Nothing establishes that the four boxes are exhaustive for operators outside that set.
• The desktop result is preliminary and small. Against PyTorch Inductor on a Tesla V100, the improvement is 1.23 and 1.11 times on two models. The authors describe this as a quick and separate implementation and expect more later, which is a fair way to present it and also means it is not evidence of much yet.
• Even after all of this, the hardware is far from saturated. The roofline analysis puts SmartMem at 24% to 35% of theoretical peak, and that is under what the paper itself calls the unrealistic assumption that all data comes from texture memory. Models made of low-intensity operators do worse.
• Accuracy is assumed rather than measured. Since the transformations are semantics-preserving the output should be identical, and the paper states model accuracy is the same across all frameworks and therefore reports only latency. That is reasonable, but it means no numerical validation is presented.

The evaluation

Check yourself

Sources

The paper is “SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile” by Wei Niu, Md Musfiqur Rahman Sanim, Zhihao Shu, Jiexiong Guan, Xipeng Shen, Miao Yin, Gagan Agrawal and Bin Ren, published at ASPLOS 2024 Volume 3, pages 916-931, DOI 10.1145/3620666.3651384. The authors are at William and Mary, North Carolina State University, the University of Texas at Arlington and the University of Georgia.

This explainer is written from the full text, read from the authors’ preprint on arXiv (2404.13528). Everything above comes from the paper’s own sections 2 through 4: the breakdown table showing convolutional networks at a few percent layout cost against Transformers at 43% to 70%, split into explicit and implicit components; the comparison of one-dimensional buffer against 2.5D texture memory and the note that texture memory has been measured at 3.5 times lower convolution latency; the four-quadrant operator classification with its formal definitions and its sample operators; the sixteen-cell action table and the corresponding search-policy table; the index dependency classification into identity, split and merge and the strength reduction rule for nested remainders; the reduction-dimension heuristic with its microbenchmark justification at 1.7, 1.4 and 1.1 times; the handling of producers with multiple consumers and the sample texture-memory layouts partitioned into groups of four; the operator counts showing a 21% to 65% reduction and 1.1 to 1.7 times more fusion opportunities than the baseline; the end-to-end latency table with its geometric means of 7.9, 6.9 and 2.8 times and its blanks where frameworks lack support; the ablation with deletion, selection and other optimisations broken out separately for Transformer and convolutional families; the memory counters showing 1.8 times fewer accesses and 2.0 times fewer cache misses; the batch-size and portability results; the roofline analysis placing achieved performance at 24% to 35% of theoretical peak; and section 4.6, where the authors state the limitation on convolutional networks and quantify the live redundant copies at 3.0 MB and 2.3 MB.