Two lotteries, and the space between them
You can prune more than half of the parameters inside a trained neural network and it still works. So why does it not get proportionally faster? The answer turns out to be a story about hardware, not about algorithms.
You can prune more than half of the parameters inside a trained neural network and it will still recognise a face, still answer a question, still steer a car. On many models you can go well past half. Some version of this has been known since 1989.
That ought to be the end of the story. Fewer numbers means less arithmetic, less arithmetic means less time, and the model should come out proportionally faster. It does not. Very often it does not come out faster at all.
The distance between those two paragraphs is where I have spent my PhD. I design the hardware that is supposed to collect the saving, and the thing nobody tells you going in is that creating the saving and collecting it are separate problems, solved by different people, measured in different units. This summer I put everything the field knows about that distance into one catalogue, and what follows is the argument that shaped it.
The catalogue this piece came out of. Sparsity in deep learning from pruning theory to silicon, arranged by mechanism so you can read down a column and see a design space.
Why skipping the work does not make it faster
The field uses two verbs for this, and the distinction between them is the whole problem. Algorithm researchers prune: they decide which numbers do not matter and set them to zero. Hardware researchers skip: they try to make the chip step over those zeros instead of multiplying by them. Pruning is easy and it is done. Skipping is the hard part, and nothing about pruning guarantees it.
So when an algorithm paper reports how much of a model it pruned away and an architecture paper reports that the same model got twenty percent faster, neither is exaggerating. They are reporting different quantities. Three things stand between the two, and they compound.
You have to record where the zeros are. A dense matrix needs no explanation: the hardware knows the third number in the fourth row sits at a fixed offset. The moment you prune numbers out of it, that arithmetic breaks, so you have to store a map of which survivors sit where. That map is data too, and it has to be stored and read alongside the values it describes. At moderate sparsity it costs more bytes than the pruning saved, so the compressed form is larger than what it replaced.
The survivors are scattered. Processors are fast because they guess what you will need next, and dense data makes them right almost every time: the next number is the one sitting beside this one. Scatter the survivors and every guess misses. The chip spends its time waiting on memory rather than computing, and the arithmetic units you paid for sit idle.
The multiplier does not care. This is the one that surprises people. The array of multipliers inside a GPU runs at a fixed rate. Give it a zero and it multiplies by zero, on schedule, drawing the same power as any other number. It has no mechanism for noticing that the work was pointless, so it does not finish early. Skipping is not something a processor does for free when the data happens to be zero. It is a capability that has to be designed in, and for most of the last decade it was not.
Put those together and the shape of the problem is clear. How sparse a model is, is a property of the model. How long it takes, is a property of the machine. The function connecting the two is the thing worth studying, and it is built from formats, dataflows, intersection units, load balancing and sparsity predictors. That work is scattered across ISCA, MICRO, HPCA, ASPLOS, NeurIPS, ICLR, MLSys and arXiv with no organising principle beyond the date of publication, which is the hole the catalogue was meant to fill.
Not all zeros are the same
The field's word for all of this is sparsity. A tensor is sparse when most of its values are zero, wherever those zeros happen to come from, and that last clause is doing more work than it looks. Sparsity is one word for about a dozen different situations, and most arguments in this area turn out to be arguments about which one is meant. They behave nothing alike.
The oldest kind is weight sparsity: you prune the trained parameters and most of them become zero. It is the most studied and, ironically, the least used in production. The kinds that actually carry traffic today are newer. In attention, the mechanism that lets a language model relate every word to every other word, the vast majority of those relationships turn out to matter not at all, and at long context almost the whole table is zero. The KV cache, the model's running memory of the conversation so far, holds far more of the past than any single new word actually consults. A mixture-of-experts model contains many specialist sub-networks and wakes only a couple per word, so most of the parameters sit idle on any given step. Graph networks inherit their sparsity from the data: a social network is almost entirely non-friendships. And a recommender system holds an enormous table of embeddings and touches a handful of rows per query.
What separates these is not how many zeros there are. It is two other questions. Where are the zeros, and when do you find out? Pruned weights are known before the program runs, so the bookkeeping can be done once, offline, and the hardware can be told in advance. Attention and expert sparsity are different: the zeros depend on the input, so nobody knows where they are until the model is halfway through the calculation. Something has to predict them, and the prediction costs time. Every dynamic-sparsity accelerator I know of lives or dies on that trade, because whatever the predictor costs comes straight out of the saving it exists to capture.
Two lotteries
That is the mechanics of it. The harder question is why the situation lasted as long as it did, and the best answer I know comes from two arguments about lotteries, made seven years apart and never put in the same room. The first says the sparse network was there all along. The second says the hardware of the day decides whether that does you any good.
The first is Frankle and Carbin's lottery ticket hypothesis. They found that hidden inside a big randomly initialised network there is a much smaller one that, trained by itself from the same starting point, does just as well.
"dense, randomly-initialized, feed-forward networks contain subnetworks ('winning tickets') that — when trained in isolation — reach test accuracy comparable to the original network in a similar number of iterations."
Frankle & Carbin, The Lottery Ticket Hypothesis, ICLR 2019
Read narrowly it is a claim about lucky starting values. Read broadly it is the sharpest statement anyone has made of the thing this whole field rests on: most of a trained network is surplus, and the small network was sitting inside the large one the entire time. On that reading sparsity is not a compromise you accept in exchange for accuracy. It is a description of what the network always was.
Which makes the second argument, Sara Hooker's hardware lottery, land harder, because it explains why none of that helped. Hooker's point is that ideas do not win on merit alone. They win when the machines of the day happen to suit them.
"…when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions." Such lotteries, she writes, "can delay research progress by casting successful ideas as failures."
Sara Hooker, The Hardware Lottery, CACM 2021
Set the two side by side and the lost decade explains itself. Fine-grained sparsity won the first lottery and lost the second. The winning ticket was real and it was worthless, because no chip you could actually buy knew how to cash it. GPUs had spent years getting very good at dense matrix multiplication, dense matrix multiplication had become very cheap, and a sparse layer ran at the speed of a dense one because that is what a machine does with zeros it cannot see. That was never a failure of the pruning research. It was Hooker's mechanism working exactly as she described it.
Rigging the draw
What changed is that people stopped waiting to win the hardware lottery and started rigging the draw.
The clearest case is a pattern called 2:4. The rule is blunt: in every group of four adjacent weights, exactly two must be zero. Nobody discovered that rule in the data. No trained network naturally organises itself that way. It exists because a small fixed circuit sitting in front of a multiplier can decode "two of these four" instantly, every cycle, without stalling anything, which makes it the rare sparsity pattern a dense-throughput machine can genuinely exploit. NVIDIA built the constraint into the Sparse Tensor Core, and the research community was then asked to train models that satisfy it. The papers on hitting 2:4 without losing accuracy came after the silicon, not before it.
That reverses the usual order of things, and it worked, which is why the same reversal now runs through the field. DeepSeek's native sparse attention is the same move at a larger scale: the sparsity pattern was chosen for what the hardware can stream efficiently, described in their own paper as an "arithmetic intensity-balanced algorithm design", and the model was trained to want it from the start rather than having it imposed after the fact. Both are hardware decisions wearing algorithm clothing, and together they have quietly changed the research question from "what is the best set of weights to delete?" to "which patterns can a model be taught to accept?" Those are not the same question, and the second one is set by whoever designs the chip.
Why sparsity came back
The timing is not an accident. Sparsity returned because scaling made it load-bearing.
For years the bottleneck in running a model was arithmetic, and sparsity's pitch was that it removed arithmetic, which as we have seen it does not reliably do. Then the bottleneck moved. Generating text one word at a time is limited not by multiplication but by how many bytes you must haul from memory for each word produced, and moving fewer bytes is the one saving sparsity was always genuinely good at. Meanwhile context windows grew until attention's cost, which rises with the square of the length, dominated everything else. KV caches grew larger than the models they serve. Mixture-of-experts stopped being a clever trick and became the default shape of a frontier model.
Every one of those is a sparsity problem, and the hardware lottery now rewards them instead of punishing them. That is the real change. The algorithms did not suddenly get better. What was scarce moved, and sparsity happened to hold the right ticket for the new scarcity.
Where I think this goes
I will state my position plainly, with the caveat that I am a PhD student and this field has embarrassed more confident people than me.
I think the co-design inversion is permanent, and I think it is going to get more extreme. The next decade of sparsity will be written by whoever controls the datapath. 2:4 was not a discovery about neural networks; it was a decision about silicon that the algorithm side was then obliged to satisfy. Native sparse attention is the same move. I expect that to become the default posture rather than the exception: an accelerator states the pattern it can decode at line rate, and models are trained into that constraint from initialization.
If that is right, a few things follow. Sparsity patterns start arriving from hardware vendors rather than from ML papers, on hardware release cycles rather than conference cycles. The valuable algorithmic work becomes showing that a vendor-specified pattern is learnable without loss, which is a narrower and less romantic question than the one the pruning literature started with. And the accelerators that win will be the ones flexible enough to support a family of patterns, because committing a fabbed datapath to exactly one sparsity structure is a bet on a research direction that has to hold for the life of the chip.
There is a real risk buried in this, and it is Hooker's own. If the hardware dictates the pattern, we will get very good at the patterns silicon happens to like, and we will never find out what we gave up. The fix for one hardware lottery can be the setup for the next one. The honest response is to make the negotiation genuinely two-way: build datapaths that can be told what to skip at runtime, and stop pretending that the pattern which is cheapest to decode is the one the model wanted.
That is the part I want to spend my time on.
Corrections and papers I missed are the most useful thing you can leave here. Sign in with GitHub to reply; threads are stored as Discussions on this repo.