Day 12 - What TorchDynamo Intercepts: CPython's Frame Eval Hook

The two paths CPython can take to run a frame: by default through _PyEval_EvalFrameDefault, with Dynamo installed through Dynamo's own eval

The previous post drew the four stages of torch.compile; the first is Dynamo, "capture". We said it intercepts your bytecode "as Python runs". That sounds light, but it hides a concrete question: how does a third-party package get to insert itself into CPython's execution and take over before your function runs?

The answer is not a monkey patch, and not source rewriting. It is a hole CPython left on purpose: PEP 523, the frame evaluation API. This post makes that hole clear.

First, how CPython normally runs a function

Every function call in Python makes CPython build a frame. The frame holds all the state of that call: local variables, the value stack, which bytecode instruction is next. The logic of a function does not run as source; it is first compiled into bytecode, stored on a code object.

dis lays the bytecode out:

import dis
 
def f(x):
    return torch.sin(x) + 1
 
dis.dis(f)

You will see something like:

LOAD_GLOBAL              torch
LOAD_METHOD              sin
LOAD_FAST                x
CALL_METHOD              1
LOAD_CONST               1
BINARY_ADD
RETURN_VALUE

Normally, once CPython has a frame, it hands it to a C function, _PyEval_EvalFrameDefault. That is the big switch loop: read one bytecode at a time, execute it, update the stack. It is the heart of Python.

PEP 523: making "how to run a frame" replaceable

PEP 523 (introduced in Python 3.6) does exactly one thing: it adds a function pointer, eval_frame, to the interpreter state. When CPython is about to run a frame, it no longer hardcodes a call to _PyEval_EvalFrameDefault; it calls whatever this pointer points at. By default it points at the default, so behavior is unchanged.

But it means a C extension can swap that pointer for its own function. From that moment, before every frame runs, CPython first asks your custom evaluator: "this frame, how do you want to run it?"

That is Dynamo's foothold. It does not touch your source and does not wrap your function object; it sits one layer below the function, catching CPython's request to execute each frame.

What Dynamo does once it takes over

The first time you call f after torch.compile(f), f's frame reaches Dynamo's custom evaluator, and it roughly does this:

Grab the frame's code object and bytecode.
Symbolically execute the bytecode: instead of computing real numbers, it walks each instruction with symbolic values like FakeTensor, recording "a sin here, an add there" into an FX graph.
Record the premises under which this holds, the guards (the subject of the next post), for example that x has a certain dtype and shape.
Hand the graph to the backend to compile, then rewrite this frame's bytecode: replace that stretch of computation with "call the compiled artifact".
Cache the result on the code object, keyed by the guards. Next time the same function comes through, if the guards pass it uses the cache directly, no need to rerun Dynamo.

So Dynamo's "just in time" is not that it runs alongside watching; it stands directly on the path CPython must take to execute every frame.

Why bytecode, not source or AST

A natural question: to analyze a program, why not read the source, or parse the AST, instead of picking the lowest, hardest-to-read bytecode?

Because bytecode is the only form guaranteed to exist. Source may not be available at all (C extensions, code built with exec, lambdas, things behind decorators), and neither may the AST. But as long as a function can be executed by CPython, it has a code object and it has bytecode. Working at the frame layer, Dynamo can swallow Python from almost any origin, no matter who wrote it or how it was produced.

The cost is that it must implement its own symbolic interpreter for bytecode, re-specifying the semantics of hundreds of CPython instructions. This is the heaviest part of Dynamo.

Why "almost any" still keeps the "almost"

The frame eval hook lets Dynamo see every bytecode, but seeing it is not the same as being able to model it. Some instructions it cannot walk with symbolic values, the most typical being control flow that depends on an actual value:

def f(x):
    if x.sum() > 0:      # you must really compute x.sum() to know which branch
        return x * 2
    return x + 1

x.sum() > 0 is a value that has to become a concrete Python bool, but Dynamo holds a symbolic tensor and does not know which branch to take. It does not guess; it breaks the graph: it collects what it captured before the if into one graph, hands control back to CPython to run the if, and once the branch is settled, resumes capturing the next graph after it. That is the graph break from the previous post; Day 14 is devoted to it.

The same forced break also comes from: calls into C functions it has no symbolic model for, .item()-style operations that pull a tensor value out into a Python scalar, and side-effecting things like print. The frame eval hook gives Dynamo the ability to "see everything", but "everything can be symbolized" is a different matter, and that boundary is the spine of the rest of Part 1.

See it rewrite the bytecode yourself

Dynamo's rewritten bytecode is not a black box; TORCH_LOGS dumps it:

TORCH_LOGS="bytecode" python your_script.py

You will see two chunks of bytecode: the original, and Dynamo's rewritten version. The rewritten one contains a call into the compiled artifact, replacing that run of LOAD_METHOD / CALL_METHOD. This is the most direct way to confirm "Dynamo really acts at the bytecode layer", and it echoes the method from the previous post: read each stage with your own eyes before moving on.

Key takeaways

Dynamo's interception point is the frame, not the function. PEP 523 leaves an eval_frame function pointer on CPython's interpreter state, and Dynamo swaps it for its own. From then on, before every frame runs, CPython calls Dynamo first. That is why it needs neither to change your source nor to wrap your function object, yet can take over "as Python runs": it stands one layer below the function.

Choosing bytecode is for generality, not for show. Source and AST may be unavailable, but as long as a function can be executed by CPython it has bytecode. Working at that layer is what lets Dynamo swallow almost any Python regardless of origin. The cost is writing its own symbolic interpreter for bytecode, and it breaks the graph precisely because some instructions (value-dependent control flow, .item(), C calls with no symbolic model) cannot be walked with symbolic values alone.

"Intercepting at the frame" and "reading bytecode" are two different things; only together do they explain Dynamo's reach. Bytecode always existing is about being able to read it; the interception point being the frame is about being able to catch it. The frame sits one layer below the function, and CPython must consult the eval_frame pointer to run every frame, so Dynamo needs neither to change your source nor to wrap your function object, and does not care how the function came to be: lambdas, code built with exec, things behind decorators, all caught the same way. Drop either half and the generality collapses.

Swap the pointer and every frame detours through Dynamo first, including the ones it skips. eval_frame is swapped on the interpreter state, not hung on one function. So once Dynamo is installed, every frame on the execution path first enters Dynamo's evaluator for a decision, and even when it decides a stretch should not be compiled and hands it straight back to _PyEval_EvalFrameDefault, it has already taken that detour. That is why a program full of many tiny functions, or one that keeps breaking the graph, can actually get slower under torch.compile: neither the graph breaks nor this per-frame detour is free.

This post settled "how Dynamo takes over". But "symbolically execute the bytecode" went by too fast: Dynamo holds no real values, so how does it walk LOAD_FAST, CALL_METHOD, BINARY_ADD one by one while maintaining a stack identical to CPython's? The next post (Day 13) opens the body of that symbolic interpreter: InstructionTranslator. We will read along with torch/_dynamo/symbolic_convert.py and see how it rewrites CPython's stack machine at the Python level, one handler per instruction, the graph growing as it walks.