Day 10 - The Map Unfolded: torch.compile, TVM, XLA, and TensorRT

Four real systems over one shared skeleton: capture, graph optimization, kernel acquisition, runtime

Nine days of concepts; the last post is for meeting the cast.

Day 3 ended with a set of coordinates: for any tool, ask three questions. Where does it capture? Which moves does it apply? Whom does it generate code for? This post runs four of the most representative systems through those questions. You will find them strikingly isomorphic: all built on the same skeleton of "capture → graph optimization → get kernels → runtime", differing in what fills each segment.

torch.compile: The Compiler That Coexists with Eager

PyTorch's official compilation path, designed around one philosophy: do not disturb. One line, torch.compile(model), everything else unchanged.

Capture: TorchDynamo. It does not ask you to rewrite the model; it intercepts Python bytecode and traces it dynamically, collecting what it can compile into a graph and cutting around what it cannot (arbitrary Python logic), which falls back to eager, the two sides handing off seamlessly. The assumptions it leans on are recorded as guards, and a broken guard means recompile, exactly Day 8's machinery.
Graph level: FX graphs carry the program; for training, AOTAutograd captures the backward pass into the graph too.
Kernel acquisition: Inductor. It makes the fusion decisions (Day 5) and the loop schedules (Day 6), and on GPU it mainly generates Triton code; mode="max-autotune" switches on Day 7's search.
For whom: NVIDIA GPUs first, CPU second, other backends via the community.

One-line identity: the master of compromise. It does not try to swallow the whole graph; it compiles what it can inside a dynamic Python world, with eager always there as the safety net.

TVM: The Open Stack with Every Layer Exposed

Apache TVM is a complete compilation stack with academic roots, and nearly every concept in this series has a named component inside it.

Capture: import from ONNX, PyTorch, and friends into Relax, the graph-level IR. Relax's signature is symbolic shapes as first-class citizens: types like (n, 4096) flow through the IR directly (Day 8's third move).
Graph level: fusion, layout, constant folding, the whole Day 3 kit lives here.
Kernel acquisition: TensorIR handles loop scheduling, the direct heir of compute/schedule separation (Day 6); MetaSchedule does the auto-tuning (Day 7): template-free candidate generation, cost-model filtering, measured results banked. A textbook implementation.
For whom: TVM's biggest differentiator. Beyond NVIDIA: AMD, ARM, mobile GPUs, WebGPU, all manner of embedded backends.

One-line identity: hardware breadth first. When the target is not an A100 but some phone SoC or a startup's accelerator, an open stack like TVM is often the only road.

XLA: The Fusion Machine of the Static World

XLA is Google's compiler; every JAX jit runs on it, and the TPU is its home turf.

Capture: JAX or TensorFlow traces programs into StableHLO, a graph IR with a deliberately small operator set: few ops, each with clean semantics, exactly Day 2's "an IR that is good to optimize".
Graph and kernels: XLA's temperament is aggressive fusion (Day 5's logic pushed to the limit), plus strength in algebraic simplification and layout; kernels are mostly generated.
Shape philosophy: XLA traditionally prefers static shapes: a changed shape means re-trace and re-compile, which is why padding and bucketing (Day 8's second move) are daily practice in the JAX ecosystem. In exchange, the compiler holds complete shape information and dares the most thorough optimizations.
For whom: the TPU is its own child; GPU and CPU are supported too.

One-line identity: constraints traded for performance. It demands the most from its users (pure functions, stable shapes) and is exactly why it can optimize the hardest.

TensorRT: The Closed but Ruthless Vendor Road

The other three generate kernels; TensorRT walks a different road: it selects them.

Capture: import ONNX or framework formats, build an inference engine offline.
Kernel acquisition: a foundation of NVIDIA-engineer-hand-tuned kernels; at build time, each op measures multiple candidate implementations and picks the winner (tactic selection). That is Day 7's search, except the space being searched is a set of precompiled kernels instead of a schedule space.
LLM specialization: TensorRT-LLM stacks on large-model weapons: paged KV cache, in-flight batching, FP8 and INT4 quantization kernels (Day 9).
For whom: NVIDIA only. The limitation is also the reason it is fast: with zero portability concerns, every kernel can be written flush against the hardware.

One-line identity: squeezing one vendor's hardware to the limit. In production that runs NVIDIA only and wants the last ten percent, it is often the answer.

One Skeleton, Four Temperaments

Side by side, the differences collapse onto four axes:

Open vs closed: TVM, torch.compile, and XLA are open source; TensorRT is a black box.
Generate vs select: the first three generate kernels automatically; TensorRT picks from a hand-tuned library.
Dynamic shapes: torch.compile catches them with guards plus SymInt; TVM's Relax is natively symbolic; XLA asks you to pad; TensorRT uses pre-declared shape profiles.
Hardware breadth: TVM the widest, XLA deep on TPU, torch.compile and TensorRT both centered on NVIDIA.

Nobody wins everywhere; there are only pairings of tool to scenario: research iteration wants flexibility, torch.compile; non-mainstream hardware, TVM; JAX on TPUs, XLA; the last ten percent on NVIDIA, TensorRT.

Closing the Series

Three questions, the whole series distilled

Ten days, and the map is fully unfolded. Looking back: Day 1 posed the problem (movement and wasted work), Day 2 drew the layers, Day 3 named the four moves, Day 4 handed over the ruler, Days 5 through 9 dug into the moves one by one (fusion, scheduling, search, dynamic shapes, quantization), and Day 10 pinned the concepts onto real systems.

From here on, facing any new tool, new paper, or new hardware launch, you carry the same set of questions: where does it capture? Which moves does it apply? Whom does it generate code for? Is the bottleneck movement or compute? The answers may not come immediately, but asking the right questions already puts you on the map.

Key takeaways

What fills the "kernel acquisition" slot already answers "whom does it generate code for". torch.compile's Inductor generates Triton; TVM's TensorIR plus MetaSchedule also generate, but the targets can be AMD, ARM, mobile GPUs, WebGPU; TensorRT generates nothing at all, it measures candidates from a library of NVIDIA-hand-tuned kernels and picks the fastest. Generation is what makes changing hardware possible; selection means being locked to one vendor. Two sides of the same fact.

Pull that line into practice: an inference service whose batch size changes every request, running on a non-NVIDIA embedded chip, leaves essentially one answer. torch.compile and TensorRT are both centered on NVIDIA, so they fail on hardware alone; TensorRT also wants shape profiles declared up front, which a shifting batch makes awkward. XLA prefers static shapes, recompiling whenever they change, and its home is the TPU. That leaves TVM: the widest hardware reach, and a Relax IR whose symbolic shapes swallow a varying batch natively.

That closes this series. Next up is a new one that digs into torch.compile itself, from Dynamo through AOTAutograd down to Inductor, tracing what each stage actually does.