Day 20 - Symbolic Shapes: One Graph for Every Batch Size

The first compile specializes on (4, 4); (8, 4) arrives, the guard fails and triggers a recompile, frame_state notices dim 0 keeps changing, the second compile switches to symbol s0, and one graph swallows every batch size from then on

Day 14 said baking an int into a constant is a bet, Day 15 said losing the bet costs a recompile, Day 19 finished breaks. Now the last piece: the stop-loss after a lost bet. After three posts of foreshadowing, automatic dynamic takes the stage.

Static by default, dynamic under duress

Dynamo defaults to assume_static_by_default: on the first compile, every shape specializes to its concrete value, (4, 4) means exactly (4, 4). Specialized graphs optimize well; that is Day 14's dividend.

The change comes on call two. Batch size goes from 4 to 8, TENSOR_MATCH's size check fails, recompile. But this recompile is not a plain do-over: frame_state remembers the last value seen for every dimension of every input, and the comparison says "dim 0 was 4, now 8, it varies", so the second compile replaces that dimension with a symbol. From then on batch 16, 32, 100 all run the same graph, no more recompiles.

Two compiles are this mechanism's fixed cost. If you already know a dimension will vary, declare it with torch._dynamo.mark_dynamic(x, 0) and the first compile uses a symbol, skipping the recompile; mark_static locks in specialization the other way. PGO goes further: it persists the observation of which dimensions ended up dynamic, so the next process applies it at cold start and never places the first bet at all.

mark_dynamic has one easily missed second job: a debugging tool. Consider the world without it first. Suppose the code hides an if x.shape[0] == 4. Automatic dynamic switches dim 0 to s0 on the second compile, but translation reaching that if must pick a side, so it mints the guard s0 == 4, and the symbol is pinned right back to 4. Nothing actually changed: batch 8 arrives, the guard fails, a new graph gets compiled carrying s0 == 8, every unseen batch size costs another recompile, and eventually the recompile limit (Day 21) trips and the whole frame falls back to eager. No error message anywhere, just production silently getting slower. mark_dynamic turns that silent failure into a loud compile-time one: you declared "this dimension is dynamic", translation produced a constraint that wants to pin it, the two contradict, and a ConstraintViolation fires pointing at the culprit line. That is a feature, not a bug: it forces you to face the fact that this code simply does not support dynamism, instead of silently specializing and recompiling to death.

SymInt and ShapeEnv

After the switch, x.shape[0] no longer returns an int but a SymInt: a symbolic object standing for s0. It does not collapse into a number under arithmetic; it grows expressions: x.shape[0] * 2 is 2*s0, and the output shapes of reshape and cat are all computed from these expressions.

ShapeEnv as the central office: s0 is born at an input, propagates through ops into expressions, comparisons produce guards, and harvest emits the symbolic guard list

Every symbol lives in the ShapeEnv, which does three jobs:

Issue symbols: every dynamic dimension gets an s0, s1. The same symbol can appear on multiple inputs, naturally expressing "these two inputs share their batch dimension".
Propagate expressions: output shapes are computed from input symbols; matmul's (s0, 4) @ (4, 8) yields (s0, 8), and mismatched shapes fail to line up at translation time.
Collect constraints: every time the code asks a shape question, say if x.shape[0] > 10, the answer steers translation, so the comparison becomes a guard: s0 > 10. A graph traced down the true branch is only valid while s0 > 10.

What the guards become

TORCH_LOGS=guards shows it best side by side. The static era:

TENSOR_MATCH: check_tensor(L['x'], size=[4, 4], stride=[4, 1])

After going dynamic:

TENSOR_MATCH: check_tensor(L['x'], size=[None, 4], stride=[4, 1])
L['x'].size()[0] == L['y'].size()[0]
2 <= L['x'].size()[0]

The first size slot is no longer pinned, traded for two symbolic guards: the two inputs' batch dimensions must match (same s0), and 2 <= s0. That last one is 0/1 specialization: sizes 0 and 1 are too special (empty tensors and broadcasting both behave differently), so symbols are assumed to be at least 2, and an actual 0 or 1 gets its own specialized graph. That is the compromise line between "general" and "still optimizable".

When if meets a symbol

Two kinds of if meet entirely different fates, and telling them apart saves a lot of wasted trips:

Asking about shape: if x.shape[0] > 10. ShapeEnv can answer at translation time, first by reasoning over collected constraints; when reasoning falls short, it bets using the hint (the concrete value the symbol carried when first seen) and takes one side, minting a guard s0 > 10 the moment the bet is placed. This is the symbolic edition of "the more specialized the graph, the more guards": every bet adds a premise. When an input from the other side actually arrives, the guard fails and a second specialized graph joins the first (Day 21's cache).
Asking about tensor values: if x.sum() > 0. The answer lives in GPU memory and simply does not exist at translation time; no amount of ShapeEnv cleverness can bet on it. The default is a graph break (Day 19); if you truly need a data-dependent branch inside the graph, rewrite it as torch.cond, letting each branch become its own subgraph (exactly what Day 17's SubgraphTracer exists for).

One deeper pit: operations whose output depends on data, like x.item() and nonzero(), produce symbols with no hint at all (unbacked SymInts). No branch depending on them can be bet on, so the default is an immediate break. A manual assertion like torch._check(u0 >= 0) feeds ShapeEnv facts you know but Dynamo does not, letting translation proceed.

The price

Symbols are not free. A dynamic graph lacks concrete numbers, so the backend cannot pick kernels by shape or fully unroll loops, and it may run slower than the specialized graph; ShapeEnv's symbolic reasoning also slows compilation itself. So the whole package, static by default, dynamic under duress, 0 and 1 always specialized, is a deliberate compromise: pay the symbolic cost only for dimensions proven to vary, and let every other dimension keep the specialization dividend.

Key takeaways

The difference between shape and tensor value is, at its core, "does it exist at translation time". Shape is tensor metadata, laid out on the table during translation, and its symbols carry a hint (the concrete value seen when the symbol was first minted), so when ShapeEnv cannot reason its way to an answer it substitutes the hint and picks a side; the moment the bet is placed, a guard is minted and the graph's validity range narrows. Losing the bet means a guard failure and a second specialized graph, both coexisting in the cache. A tensor's value lives in GPU memory and simply does not exist at translation time; there is no basis for a bet at all, which is why if x.sum() > 0 defaults to a break. The two tools govern different segments, so keep them apart: to keep a data-dependent branch inside the graph, rewrite it as torch.cond; torch._check feeds ShapeEnv facts so translation depending on an unbacked SymInt can proceed, but it never moves a branch into the graph. As for 0 and 1 always specializing, the reason is reasoning, not optimization: size 1 gets special treatment under broadcasting, so whether s0 is 1 changes the operation's semantics themselves, and size 0 changes how loops and reductions behave; letting a symbol cover 0 and 1 would split every shape question three ways. Assuming ≥ 2 keeps every answer unique.

Dynamo's machinery is now fully covered, with one thing left: where all these graphs live, who evicts whom, and how many recompiles is too many. The next post (Day 21) covers the cache's C++ implementation, the recompile limit and how to read TORCH_LOGS=recompiles, how trace_rules decides whether a function gets traced or skipped, and closes this part with a source map of all of torch/_dynamo.