Day 15 - Guards: When a Compiled Result Can Be Reused

Every call first walks the C++ GuardManager tree: a full pass reuses that cache entry's compiled code, any failure moves on to the next entry, and only when all fail does recompilation happen

The last two posts kept running up a tab: Day 13 said translation "records premises along the way", Day 14 said the price of baking constants "is bookkept by guards". Time to pay. Guards are the soul of Dynamo's caching machinery, and the answer behind many a "why did it get slow again" with torch.compile.

Why premises are unavoidable

What Dynamo compiles is not your function; it is your function as it looked under one concrete set of inputs. During translation it placed a pile of bets: x is float32, on CUDA, shape (4, 4); n is 3, baked straight into the graph; the if went left because the condition was true at the time. Every bet makes the graph faster, and also narrower.

On the next call, do those bets still hold? Reuse without checking and you will not even know when the answer is wrong. Checking them one by one is what guards are. So guards are not defensive decoration; in the "specialization for speed" trade they are the side that keeps the books: as specialized as the graph is, that many guards there will be.

From Source to guard

Day 14 showed every outside value carries a Source chain; guards grow on that chain:

install_guard(source.make_guard(GuardBuilder.TYPE_MATCH))

Each method on GuardBuilder is one kind of check. The most common:

guard	what it checks	typical origin
`TYPE_MATCH`	same type (type pointer compare)	most objects
`ID_MATCH`	still the same object (id compare)	functions, modules, things that must not be swapped
`EQUALS_MATCH`	value equality	ints and strs baked into constants
`TENSOR_MATCH`	dtype, device, shape, stride, requires_grad	every tensor input
`SEQUENCE_LENGTH`	list or tuple length	unrolled containers

TENSOR_MATCH deserves attention: it does not just check "is a tensor" but the whole set of dtype, device, shape, stride, requires_grad. Change any one and the graph's assumptions collapse; a kernel Inductor generated for float32 is simply wrong on float64.

Read one for real

import torch
 
cfg_scale = 2
 
def f(x, n):
    return x * n * cfg_scale
 
compiled = torch.compile(f)
compiled(torch.randn(4, 4), 3)

TORCH_LOGS="guards" python demo.py

The output (excerpted and tidied) looks like:

GUARDS:
  TENSOR_MATCH: check_tensor(L['x'], dtype=torch.float32,
                device=None, requires_grad=False, size=[4, 4], stride=[4, 1])
  EQUALS_MATCH: L['n'] == 3
  EQUALS_MATCH: G['cfg_scale'] == 2
  GLOBAL_STATE: ___check_global_state()

Line by line:

L['x'] is a LocalSource in print form; Day 14's Source chains surface here. A tensor input takes one TENSOR_MATCH, with dtype, size, stride all pinned.
n is a Python int, baked into a constant, hence EQUALS_MATCH: L['n'] == 3. Pass 4 and this line fails, and all of f recompiles. This is the most common novice source of recompilation: passing something that varies as a scalar argument.
cfg_scale is a global: G['cfg_scale'] == 2; changing it also triggers recompilation.
GLOBAL_STATE is the invisible premise: global switches like grad mode and autocast are inside the bet too. A graph compiled under torch.no_grad() cannot be reused with grad on.

One planted seed: after n=4 triggers a recompile, Dynamo notices "the value at this position varies" and the second compilation uses a symbolic integer instead of baking another constant. That is automatic dynamic, Day 20's topic.

Why compile it into a C++ tree

Guard checking sits on the hot path of every call. A Python loop over the list is too slow; it would hand back the time compilation won. So at the end of translation, CheckFunctionManager compiles the whole guard set into a C++ tree of GuardManagers (guards.cpp; its seven-thousand-plus lines exist for this).

The tree's shape mirrors the data access paths: the root receives the frame's locals; the branch for L['x'] hangs a TENSOR_MATCH, the branch for G['cfg_scale'] hangs an EQUALS_MATCH. The design ships with some ruthless optimizations:

No dict construction to fetch values: FrameLocalsMapping reads locals by index, skipping the cost of materializing f_locals.
Fail fast: guards that historically fail most often get reordered to the front; the earlier the failure, the less wasted work.
Dict version tags skip subtrees: dicts carry version numbers; unchanged version, whole subtree skipped.

One function, many graphs

A guard failure does not immediately mean recompilation. One code object can hold multiple cache entries, each a pair of (guards, compiled code). A call comes in, entries get their tickets inspected one by one, and the first full pass wins; only when all fail does recompilation run, adding one more entry afterwards. So f(x, 3) and f(x, 4) each keep a graph, coexisting, and whichever guards pass, that graph runs. Entries have a count limit; how they are stored and evicted is Day 21's topic.

Key takeaways

The number of guards is not set by how many inputs you have but by how many bets tracing placed: every constant baked, every shape specialized, every branch taken adds one more premise to verify. "The more specialized the graph, the more guards" is a conservation law; fewer checks can only be bought with fewer bets. Also, passing a changing scalar as an argument means the first call bakes n=3 into a constant with an EQUALS_MATCH guard, and n=4 triggers a recompile, but this does not settle into "one graph per value, fast once they are all compiled": cache entries have a count limit, and an ever-increasing value always brings a new one, so stacking entries never catches up. The real stop-loss is automatic dynamic: on the second compile Dynamo notices the value at that position keeps changing and switches its bet to a symbolic integer, letting one graph absorb every value.

So far the traced programs have been well behaved: compute, then return. But real Python mutates: self.counter += 1, appending to a list, writing a global. These side effects cannot enter the graph (the graph is purely functional) and cannot be dropped (semantics would break). The next post (Day 16) covers Dynamo's third ledger: SideEffects, how every mutation is recorded during translation, and after the graph runs, replayed back into the real world one entry at a time by the generated bytecode.