Day 3 introduced fusion in one sentence: merge several consecutive small ops into a single kernel. Day 4's ruler has also measured it: it cuts data movement, so it helps memory-bound work, and memory-bound is the norm on modern GPUs. That makes fusion probably the highest-ROI move in all of compiler optimization, worth a whole post of digging.
This post answers four questions: what shapes does fusion come in? How does a compiler decide who fuses with whom? Where is the boundary (why are reductions so hard)? And one counterintuitive fact: sometimes fusion makes you slower.
First, the Cost Model
The time cost of one kernel splits roughly into four parts: launch overhead, reading inputs from HBM, computing, and writing outputs back to HBM. The entire magic of fusion is keeping intermediate results off the ground: they stay in registers (or shared memory) and pass straight to the next step, so all the "write it back, read it again" round-trips and the extra launches disappear.
Hold on to that phrase, "off the ground". When we get to the boundary, every difficulty grows out of it.
Vertical Fusion: Merge Along the Dataflow
Vertical fusion merges producers with consumers: when one op's output is exactly the next op's input, chain them into one kernel.
Day 1's a+b → *c → relu is vertical fusion: three kernels become one, and each element, once loaded into a register, gets added, multiplied, and relu'd all the way through before being written back. Six HBM round-trips (one read and one write per kernel) become one read and one write.
The condition for this is actually strict; element-wise ops just happen to satisfy it: each output element depends only on the input elements at the same position. Element i's whole computation chain fits inside a single thread from start to finish, with no data exchanged with any other thread. This is exactly the property that reductions will break.
Horizontal Fusion: Merge the Unrelated
Horizontal fusion merges independent parallel branches: ops with no dataflow between them, but similar shapes, runnable at the same time, packed into one kernel (or merged into one bigger op).
What it saves differs from vertical fusion: with no intermediates flowing between the branches, there is nothing to keep "off the ground". The savings are mainly launch overhead, plus one bigger kernel filling the GPU better (several small ops that individually cannot saturate the hardware occupy it better when merged).
LLMs have a canonical example: attention's Q, K, and V projections multiply the same input by three different weight matrices. Three matmuls, mutually independent, same shape, so nearly every inference framework concatenates the three weight matrices and computes QKV in one matmul. The input is read once instead of three times, one launch instead of three, and the matrix gets fatter; recall Day 4: fatter matrices have higher arithmetic intensity.
Epilogue Fusion: Riding the Heavy Kernel
Day 3 mentioned epilogue fusion: folding the bias and activation after a matmul or conv into the tail of the heavy kernel. Now we can say precisely why it is such a bargain: the moment the matmul finishes, its result is already sitting in registers, about to be written to HBM. Adding a bias and applying a relu right there is nearly free: the extra arithmetic is a rounding error next to the matmul, and not a single extra byte gets moved.
Without fusing, you would write the result to HBM, launch a new kernel, and read the whole thing back, just to do one addition. Against Day 4's ruler: an op like bias-add, whose arithmetic intensity approaches zero, is pure movement waste when it stands alone.
This is why real-world matmul kernels (cuBLAS epilogues, CUTLASS, Triton-generated) almost all take the shape "heavy computation + a small customizable tail".
The Boundary: Why Reductions Resist Fusion
Vertical fusion rests on "each output depends only on the same-position input". Reductions are the exact opposite: one output depends on an entire row of inputs. Sums, means, softmax denominators, layernorm statistics, all of them.
Where is the difficulty? In the element-wise world, threads never speak to each other; each computes its own. The moment a reduction appears, threads must exchange data and synchronize: collecting the partial sums held by thousands of threads into one number goes through shared memory at minimum (across blocks it may even touch HBM or split into two kernels). The "off the ground" assumption is broken.
Worse still are the ops after the reduction. Softmax is the classic case: every element divides by the sum of the whole row, and until that sum is finished, nothing downstream can move. The dataflow narrows to a single point, then fans back out. Forcing the "before reduction" and "after reduction" parts into one kernel demands global synchronization inside the kernel, something the GPU programming model is deeply unfriendly toward.
So most compilers' fusion strategy is: fuse element-wise chains freely, and cut at every reduction. The reduction gets its own kernel (or fuses with the element-wise ops before it), and fusion restarts after it. The kernel boundaries you see in torch.compile or TVM output mostly fall exactly there.
This boundary is not a law of nature; algorithms can push it. The most famous example is FlashAttention: attention has a softmax (a reduction) stuck in the middle, so by rights QKᵀ, softmax, and the multiply by V should split into several kernels, with the huge attention matrix touching down in between. FlashAttention rewrites "compute the full sum, then divide" into "compute and correct as you go" (online softmax), and forces the whole span into one kernel; the attention matrix never touches the ground. That is the entire secret of its memory savings.
When Fusion Makes You Slower
Fusion is not "the more the better". There are at least three ways to crash:
Recomputation. Vertical fusion assumes "the producer's output feeds one consumer". If an intermediate feeds two downstream ops, fusion faces a choice: fuse with A, fuse with B, or compute the producer twice (once inside A, again inside B). When the computation is cheap, recomputing is a good trade (recall Day 4: in a memory-bound world the ALUs are often idle anyway), but if the recomputed piece is expensive, the deal turns bad.
Register pressure. The more ops you fuse, the more intermediate values are alive at once, and the more registers each thread occupies. An SM's register file is fixed; when a single thread eats too much, fewer threads can be resident (occupancy drops), and the mechanism of hiding memory latency behind masses of rotating threads stops working. Worse, when registers run out, values spill to local memory, which is physically HBM. The fusion that was meant to save movement has manufactured new movement.
Fusing away a faster kernel. Folding a computation into a big auto-generated kernel also means giving up the highly tuned library version of that computation (cuBLAS, cuDNN). An auto-generated fused kernel does not necessarily beat "two separate kernels, one of which is hand-crafted excellence".
This is why mature compilers make fusion decisions with a cost model: estimating data movement, launch counts, and register usage before and after, and some (like TVM's auto-tuning) simply run the candidates and pick the winner. Fusion is a search problem, not blind greed.
Conclusion
Closing with Day 4's ruler: fusion's payoff equals the HBM round-trips it removes, so it only registers on the roofline's slope. The three shapes (vertical, horizontal, epilogue) save different things: intermediates kept off the ground, launches and occupancy, a free ride on the heavy kernel. The boundary sits at reductions, because staying off the ground requires threads to stay independent, and reductions force them to talk. And fusion has costs of its own: recomputation, register pressure, giving up tuned library kernels. It is a decision with a ledger, not a free win.
Next Up
Fusion decides which computations live in the same kernel. The natural next question: what should the loops inside that kernel look like? Day 6 covers loop scheduling: tiling, reordering, vectorizing, binding to threads, and one big idea that shaped the whole field: separating compute from schedule.