The first seven posts shared an unstated premise: at compile time, tensor shapes are known numbers. Day 6 picked tiles with them; Day 7 banked tuning results by them.
The real world does not cooperate. In LLM serving, every request's prompt has a different length; batch size fluctuates with load; multimodal models take images of every size. Shapes are revealed only at runtime, and can differ at every step. This post is about how compilers cope once the static assumption collapses.
First, Count What Fixed Shapes Buy
To understand the price, first see clearly what the compiler earns when shapes are known:
- Loop bounds are constants. Knowing N=4096 is what lets you confidently unroll by eight, vectorize by four, and verify alignment. With variable bounds, all of it turns conservative.
- Memory planning is byte-exact. Day 3's memory planning assigns parking spots based on how big each intermediate tensor is; unknown shapes mean planning against upper bounds, or deferring allocation to runtime.
- Tuning specializes per shape. Day 7's lookup table is keyed by (op, shape, hardware); no shape, no lookup.
In one sentence: shapes are the fuel of static optimization. The three moves below are all choices along one axis: how to get shape information back, versus how much specialization to give up.
Move One: One Compile per Shape, with Guards
The most direct idea: compile when the shape arrives. First time you see (8, 512), compile a version for it and cache it; next (8, 512) hits the cache.
The catch is knowing "this time is the same as last time". torch.compile's answer is guards: at compile time, record the assumptions the optimization leaned on ("batch is 8", "length is 512"), and check them before every run. All guards pass: use the cached version. Any guard breaks: recompile, and store one more entry.
The upside: every entry is a fully specialized, first-class kernel. Two downsides: compile latency hides in the runtime (every new shape stalls once), and cache explosion: an LLM sees thousands of possible sequence lengths, and one entry each is unbearable. So in practice you declare certain dimensions dynamic (mark_dynamic), telling the compiler to handle that dimension with the next move or the third, instead of naively compiling per length.
Move Two: Padding and Bucketing
The second move flips the problem: stop the shapes from moving. A sequence of length 300 gets zero-padded to 512 before entering; the kernel only ever sees 512.
Pure padding wastes plenty: pad everything to the maximum 4096 and a length-300 sequence drags along 13x of dead computation. Bucketing is the compromise: prepare a few steps (256, 512, 1024, 2048...), and pad each sequence up to the nearest step. The kernel only needs one compiled version per bucket, and the waste is capped by the step spacing.
Old-fashioned but extremely practical; serving systems are full of its variants. The virtue is that it turns the dynamic problem fully static again: every weapon from Day 6 and Day 7 applies unchanged. The price is the padded computation, and the need to tune the steps against your traffic distribution.
Move Three: Symbolic Shapes
The third move meets the problem head-on: let shapes live in the IR as symbols. The dimension is not 4096 but a variable n; the kernel's loop bounds take values passed in at runtime. One kernel serves every n.
TVM's Relax builds symbolic shapes into the IR's type system: a tensor's type can be (n, 4096), with n flowing through the whole graph and participating in inference (adding two (n, 4096) tensors yields (n, 4096), checkable at compile time). torch.compile's counterpart is SymInt: dynamic dimensions are tracked symbolically, and guards relax from "n equals 512" to coarser conditions like "n greater than 1", cutting recompiles by a lot.
The price is Day 6's list read backwards: variable bounds force conservative unrolling and alignment; memory planning can only reserve upper bounds. So real systems mix: a general symbolic kernel as the floor, plus a few specialized fast paths for hot shapes, with guards routing between them. The three moves are not exclusive; they are a spectrum.
PyTorch automatic dynamic
Move one mentioned that torch.compile lets you declare dynamic dimensions, but the default behavior is smarter: dynamism is not declared, it is learned. The first compile is always fully static, every shape burned in, collecting all the specialization there is; when a dimension later moves and breaks a guard, the recompile marks that dimension as symbolic. Dimensions that vary reveal themselves; stable ones keep their static specialization. mark_dynamic just says it upfront, saving one recompile.
Two exceptions are worth remembering. One: sizes 0 and 1 always stay specialized, because they change the semantics of broadcasting, so a graph compiled at batch=1 recompiles when batch=2 arrives. Two: recompiling has a fuse: the same code recompiles at most 8 times (recompile_limit) before compilation gives up and falls back to eager. Move one's cache explosion, in engineering practice, looks exactly like recompile counts creeping toward that limit in your logs.
TVM Relax
What is worth seeing in Relax is how symbolic shapes connect to Day 3's memory planning. Move three said memory planning can only reserve upper bounds; where does the bound come from? Relax admits it into the IR as a function attribute: tir_var_upper_bound records "n is at most 1024", and StaticPlanBlockMemory plans the (n, 4096) intermediate as if it were (1024, 4096): allocated once, shared by every n. The bound pulls "defer allocation to runtime" back into compile time.
Better still, the bound can come from PyTorch. torch.export carries range_constraints out with the model, the max in your Dim("seq", max=1024) declaration; TVM's exported-program frontend translates them straight into tir_var_lower_bound / tir_var_upper_bound function attributes. The two symbolic shape machineries shake hands right there: PyTorch tracks and exports the constraints, Relax spends them on static memory planning. At runtime, MatchCast binds the actual arriving shape to the symbol, doubling as the check.
Conclusion
To close: shapes are the fuel of static optimization, and dynamic shapes burn exactly that. The three moves each stake a position: recompile-with-guards buys maximum specialization at the cost of compile latency and cache bloat; padding and bucketing turn the problem static again at the cost of dead computation; symbolic shapes serve every size from one kernel at the cost of specialization. The two examples are living proof of the mixing: torch.compile defaults to fully static first, learning which dimensions should move from recompiles; Relax writes upper bounds into IR attributes so symbolic shapes can still plan memory at compile time. Real systems mix all three: declare the dynamic dimensions, specialize the hot paths, let the long tail ride symbolic kernels or buckets.
Next Up
That settles shapes. Now look back at Day 4's ledger: bytes moved = element count × size of each element. Every move so far attacked the first factor. There is a whole road untraveled: shrink the numbers themselves. Day 9 covers quantization: how the INT8 and INT4 arithmetic works out, where the scales live, and why dequantization must fuse into the kernel or the whole exercise is wasted.