Day 19 - The Full Graph Break Machinery: Where It Splits, How It Reconnects

Day 13 said a graph break is not a lookup miss; it is a handler raising its hand halfway through. The last five posts walked the "translates cleanly" route to the end. This post follows everything that happens after the hand goes up.

A break does three things

Translation reaches some instruction and the handler finds something it cannot model (a C extension object's method, a call to input()), so it throws Unsupported. Dynamo does not abandon the function; it does three things:

1. Harvest the first half: nodes up to the break point get collected into a graph via Day 17's compile_subgraph, and the ledger settles (Day 16).
2. Let that instruction run eager: in the generated bytecode, the breaking instruction is kept as-is for CPython to execute itself.
3. Wrap the rest into a resume function: the bytecode after the break becomes a new function with a name like __resume_at_30_f, generated directly with Day 18's toolbox, achieving "start executing from the middle of a function", something legal Python cannot write.

The key is what follows step three: a resume function is a function, so the moment it is called, Day 12's eval hook intercepts it too, and the code after the break compiles into a second graph. One break therefore yields two graphs with a small stretch of eager in between, control passing across all three.

How the resume function takes over seamlessly

A resume function is no ordinary function: its parameter list is all the state alive at the break point, every value stacked on the symbolic stack and every local that will still be read, all passed in as arguments. Its generated prologue puts those arguments back where they belong, on the stack and in the locals, then jumps straight to the instruction after the break and starts executing. The original function's variables survive across the break as if nothing happened.

Legal Python cannot express "start running from bytecode instruction 30 of a function", but the bytecode layer has no such restriction, and resume_execution.py assembles it directly with Day 18's toolbox. A resume function is generated once per break point; ContinueExecutionCache keeps it, and the next trip through the same break reuses it.

Breaks are contagious: a break inside an inline

One more behavior that is easy to trip on. What if Unsupported fires inside an inlined function (Day 14)? Dynamo cannot break inside the callee: an inlined function has no frame of its own, so there is nowhere to resume. The rule is therefore: a break mid-inline turns the whole call into a break point on the caller's side; SpeculationLog records the caller's CALL instruction, and this propagates upward level by level until a real frame boundary.

The practical consequence: one print deep inside a utility function chops the outermost function's graph in half. When hunting breaks, the culprit is often not the line explain() reports but something deep inside the function that line calls.

See one for real

import torch
 
def f(x):
    x = x * 2
    print("mid")
    return x + 1
 
torch.compile(f)(torch.randn(4))

TORCH_LOGS="graph_breaks" python demo.py

Output (excerpted and tidied):

Graph break in user code at demo.py:5
Graph Break Reason: Attempted to call function marked as skipped
  Explanation: the builtin `print` was called ...

The result is exactly the structure in the opening diagram: x * 2 compiles into graph one, print stays eager, and return x + 1 enters __resume_at_XX_f and becomes graph two. The reason in the message carries a gb_type category and fix hints; this is ground zero for break hunting.

Mid-flight explosions: SpeculationLog and the two-pass analysis

One class of break is especially messy. Handlers like CALL dive into functions (Day 14's inlining) and may only discover something untranslatable halfway down. By then translation is deep in: the symbolic stack is half-stacked, the ledger half-written. Harvesting a graph right there means harvesting dirty state.

Dynamo's answer is to tear it down and redo: throw RestartAnalysis and translate the whole frame again from the top. Before the first pass exploded, SpeculationLog had recorded "instruction N will fail"; the second pass stops diving at N, harvests the graph cleanly first, and leaves N to the eager stretch. The two passes cost compile time and buy an invariant: breaks always land on clean instruction boundaries.

The exception hierarchy: exc.py

Translation-time communication runs entirely on exceptions, each with its own meaning:

ObservedException deserves a note: raise and try/except in user code are semantics too, so Dynamo must figure out at translation time which except will catch a given exception; if one catches it, translation continues, and only uncaught ones propagate out.

The break-hunting toolbox

Breaks do not error; they silently slow you down, so hunt them proactively:

• torch.compile(f, fullgraph=True): upgrades every break into an error, with a gb_type category and fix hints in the message. Cheapest pre-launch sweep there is.
• torch._dynamo.explain(f)(*args): changes nothing, lists every break's location and reason.
• TORCH_LOGS=graph_breaks: prints them one by one at runtime.

Why breaks are expensive

Settling the bill, one break costs at least four ways: the graph gets chopped smaller, so every fusion opportunity across the break is gone; the eager stretch in the middle is itself slow; the ledger is forced into early settlement (Day 16); and the resume function needs one more compilation. Which is why performance tuning lesson one is always: count your breaks first, then talk about anything else.

Next post

Remember Day 15's EQUALS_MATCH: L['n'] == 3? One changed value, one recompile; far too narrow. After three posts of foreshadowing, automatic dynamic finally takes the stage: the next post (Day 20) covers symbolic shapes, how SymInt swaps the concrete 4 for a symbol s0, and how ShapeEnv manages the constraints between symbols so one graph swallows every batch size.