Day 15 said one function can hold many coexisting graphs; Day 20 made one graph swallow more inputs. Three closing questions: where do these graphs live, how many recompiles is too many, and which functions should never enter at all. Answers, then the source map, and the Dynamo part is complete.
Where graphs live: the code object's extra slot
CPython reserves an extra slot on every code object for custom data. Dynamo hangs an ExtraState there, holding a chain of CacheEntrys, each one a pair of (guard tree, compiled code).
An incoming call walks the entire path in C++: the eval hook takes the frame, finds the ExtraState, runs each entry's guard tree (Day 15's tree) down the chain, and the first entry that fully passes executes its compiled code directly. Not a line of Python runs, which is the key to cache hits being as fast as eager. Freshly compiled entries go to the head of the chain: the newest graph is the likeliest next hit.
How many recompiles is too many: the recompile limit
Every full miss means a recompile and a new entry. The longer the chain, the slower each ticket check, and a chain that keeps growing means guards keep failing; this function is simply not fit for specialization. recompile_limit (default 8) is the stop-loss: past it, Dynamo gives up on this frame entirely, falls back to eager, and warns.
Catch the culprit with TORCH_LOGS=recompiles, which prints, per recompile, which guard failure caused it:
Recompiling function f in demo.py:3
triggered by the following guard failure(s):
- L['n'] == 3
The same guard failing over and over is Day 14's old problem: something that varies got baked into a constant. The fix is usually one line: pass a tensor instead, or mark_dynamic.
The moment the line gets hit looks like this:
torch._dynamo hit config.recompile_limit (8)
function: 'f' (demo.py:3)
last reason: L['n'] == 9
There are actually two limits: recompile_limit governs a single frame (default 8), and accumulated_recompile_limit governs the sum across every frame in the process (default higher), preventing a crowd of small functions from each recompiling a few times and eating the whole startup budget together. Between tests and benchmark rounds, torch._dynamo.reset() clears every cache entry so the next round starts clean.
In practice: it got slow, now what
String this post and the previous ones into a single debugging routine:
- Count breaks first:
explain()orTORCH_LOGS=graph_breaks(Day 19). If the graph is chopped up, nothing else matters yet. - Then check recompiles:
TORCH_LOGS=recompiles, watch which guard keeps failing, and triage:- Scalar equality like
L['n'] == 3: something that varies got baked into a constant; pass a tensor ormark_dynamic(Days 14, 20). - Size-related: automatic dynamic should have taken over on the second compile; if it still blows up, suspect 0/1 specialization (batch is sometimes 1) or a
mark_staticlock. ___check_global_state(): global switches like grad mode and autocast keep flipping; pin them.
- Scalar equality like
- Neither and still slow: the problem is not in Dynamo; head for the backend (the Inductor posts from Day 25 on).
Who gets in: trace_rules, config, polyfills
Dynamo holds one of three attitudes toward every function it meets, and the lists in trace_rules.py decide which:
- inline: dive in and translate (your own functions, Day 14).
- skip: do not translate; the whole call is handled as a graph break (most of stdlib, numpy internals).
- into the graph as a node: neither dive nor break; the whole call becomes one op in the graph (torch.* operations).
Three switches let you overrule: @torch.compiler.allow_in_graph puts a function into the graph as an opaque node (Dynamo skips inspecting it; you vouch for its good behavior); @torch.compiler.disable keeps a function on eager forever; torch._dynamo.config.patch(...) temporarily changes any setting, most common in tests.
The trade-offs of the three switches, in one table:
| switch | effect | when to use |
|---|---|---|
@torch.compiler.allow_in_graph | the function becomes one opaque node in the graph, internals uninspected | it does tricks Dynamo cannot swallow, but you vouch that it is pure and reorderable |
@torch.compiler.disable | the function always runs eager, no attempt made | regions you know are not worth compiling: logging, IO, debug code |
config.patch(...) | temporarily change any setting, restored on context exit | tests, isolating problems, controlled experiments |
polyfills/ is the fourth road: for stdlib functions that could otherwise only be skipped, Dynamo keeps pure-Python stand-ins and swaps them in during translation. itertools.chain, for example, is written in C and untraceable, but polyfills carries a behaviorally equivalent Python version that Dynamo swaps in and keeps inlining, which is how parts of itertools and functools stay in the graph instead of breaking it.
The source map: eleven posts, eleven files
Looking back, this series followed the file structure of torch/_dynamo/ all along:
| file | topic | day |
|---|---|---|
eval_frame.py | the PEP 523 eval hook | Day 12 |
convert_frame.py, symbolic_convert.py | the translation main loop | Day 13 |
variables/, source.py | VariableTracker and Source | Day 14 |
guards.py (+ C++ guards.cpp) | guard generation and checking | Day 15 |
side_effects.py | the mutation ledger | Day 16 |
output_graph.py | graph harvest | Day 17 |
codegen.py, bytecode_transformation.py | bytecode synthesis | Day 18 |
resume_execution.py, exc.py | graph breaks and resumption | Day 19 |
torch/fx/experimental/symbolic_shapes.py | SymInt and ShapeEnv | Day 20 |
cache_size.py, trace_rules.py, config.py, polyfills/ | cache and behavioral boundaries | Day 21 |
Next time you read Dynamo source or a stack trace, find your station on this table first, then revisit the matching post.
Next post
What Dynamo delivers is a forward graph with Python semantics captured, but training needs a backward too. The next post (Day 22) enters the middle layer, AOTAutograd: how the backward gets traced at compile time as well, how forward and backward merge into one joint graph, and why this step cannot be skipped.