Day 9 - Quantization: Shrink the Bytes Themselves

The same weight matrix: two bytes per number in FP16, one in INT8, half in INT4; the traffic halves and halves again

Day 4's ledger reads: movement time = bytes moved ÷ bandwidth. And bytes moved = element count × size of each element.

Every move so far, fusion cutting trips, layout smoothing each trip, scheduling keeping reuse in cache, attacks the first factor. This post attacks the second: make each number smaller. FP16 spends two bytes per number, INT8 one, INT4 half. That road is quantization.

The Arithmetic First

Why is the LLM world so fanatical about quantization? Recall Day 4's decode arithmetic: generating one token means reading the entire set of weights from HBM once, and decode is memory-bound through and through; the time is almost all spent reading weights.

Now swap the weights from FP16 to INT8: the bytes halve, and decode gets almost twice as fast. INT4 halves it again. A 7B model needs 14GB in FP16 but 3.5GB in INT4, which incidentally fits on consumer GPUs. In roofline terms: same flops, half the bytes, double the arithmetic intensity; the whole workload slides one notch to the right on the chart.

There is a compute dividend too: tensor cores run low precision faster, INT8 peaking at roughly twice FP16, with FP8 added on newer hardware. But keep the priorities straight: for a memory-bound scenario like decode, quantization's main payoff is bandwidth, not compute (Day 4's ruler, once again).

What Quantization Is, Mathematically

Map a continuous range of floats onto a small set of integer cells:

q = clamp(round(x / scale) + zero_point)
x ≈ (q - zero_point) * scale

The first line quantizes, the second decodes back. Take the parts one at a time.

scale is the width of one cell. If your ruler has only sixteen ticks (INT4) and the numbers span -8 to 8, each cell has to be 16 / 16 = 1 wide. The formula is scale = range width / cell count. Wider range or fewer cells means coarser cells.

round is where precision dies. x / scale says roughly which cell the number falls in, usually not a whole number, say cell 2.7. Rounding to 3 loses that 0.3 forever; that is quantization error. Note the second line uses ≈, not =: a decoded value always lands at the center of a cell, never back on the original number.

clamp is the seatbelt. If some value is large enough to land at cell 20 when INT4 only has 16, it gets pinned to the last cell. Out-of-range values are truncated, another form of error and a far more violent one than rounding.

zero_point aligns the ruler with the data. If all the numbers are positive, sitting between 0 and 10, a ruler running from -8 to 8 wastes half its cells on the negative side. The zero point shifts the whole ruler over the region where the data actually lives. Weights tend to spread roughly symmetrically around zero, so symmetric quantization with zero_point = 0 is common; the formula collapses to q = round(x / scale), one subtraction fewer and a faster kernel. Activations after a ReLU are all positive, so they need the zero point to push the ruler across.

The cell count caps everything. INT8 has 256 cells, INT4 only 16. Over the same range, an INT4 cell is sixteen times wider than an INT8 one, and the error is roughly sixteen times larger. That is why dropping from FP16 to INT8 usually goes unnoticed while dropping to INT4 takes care.

The main enemy is the outlier: one huge value in a batch stretches the scale, and every other number gets smeared into a few cells.

How Many Numbers Should One Scale Cover

Make it concrete with INT4's sixteen cells. Say a group of numbers reads 0.1, 0.2, 0.15, ..., 20.0, where that 20 is the outlier. The scale has to cover the largest value, so 20 / 8 = 2.5, and one cell is now 2.5 wide. Every one of those 0.1, 0.2, 0.15 lands in the same cell and decodes back to the same number. The differences between them are erased.

The problem is not the outlier itself but who shares a ruler with it. So the remedy is shrinking that shared region, letting the outlier ruin only its own neighborhood:

per-tensor: one scale for the entire weight matrix. Cheapest, and one outlier smears the whole thing.
per-channel: one scale per output channel. An outlier only affects its own row or column.
per-group: one scale per 128 numbers. The norm in the INT4 world; that 20 now drags down only its own 127 neighbors and nothing else.

Finer is more accurate, so why not one scale per number? Because scales are themselves FP16 values that must be stored and moved. One 2-byte scale per 128 INT4 values (64 bytes) is about 3% overhead, which is fine; one scale per 8 values pushes the overhead to fifty percent, handing back the very bandwidth quantization just saved. That is quantization's standard slider between precision and cost: fine enough to contain the outliers, not so fine that it spends the savings.

Weight-only: The LLM Standard

You can quantize weights, activations, or both. Mainstream LLM inference quantizes weights only: weights compressed to INT4 or INT8, activations kept in FP16, arithmetic done in FP16. GPTQ and AWQ live in this family.

Why cut there? Back to the ledger: in decode's HBM traffic, weights are the overwhelming majority (gigabytes of weights versus loose change of activations). Quantizing only the weights already collects ninety percent of the bandwidth payoff; activations meanwhile are far harder to quantize (fiercer outliers, distributions that shift with the input), so forcing them carries a big accuracy risk for a small gain. The knife lands only where cutting pays.

Dequantization Must Fuse into the Kernel

This section is the heart of the post, and where the compiler actually walks on stage.

Why Decoding Is Required at All

First, a question that is easy to skip: the weights are already integers, so why not just multiply integers?

First reason, the two sides disagree on type. In weight-only, activations are still FP16, and a hardware multiply-accumulate unit takes two operands of the same type. There is no "INT4 times FP16" instruction. Something has to move to the other side's type, and since activations were not quantized, the weights are what moves.

Second reason, the integer product is not the answer. A quantized q is only a cell index; the real value is (q - zero_point) * scale. Even when both sides are INT8 and real INT8 tensor cores do the multiply, the accumulated integer still has to be multiplied by the scales and corrected for the zero-point cross terms before it is a float. That step, called requantization in fully quantized INT8, is decoding too, just moved after the matmul.

Third reason, per-group scales make "move it after" impossible. In the INT4 world one scale covers 128 numbers, so a single dot product crosses several groups along K, each needing a different coefficient. There is no single factor to apply once the accumulation finishes. Scales must be applied group by group while the accumulation runs, which is to say inside the kernel.

So decoding is not a detour; the instruction set and the grouping of scales force it. The only real choice is where the decoding happens.

Decode into HBM, or Decode into Registers

INT4 weights cannot be multiplied directly; they must be decoded back to FP16 first (dequantized). The naive approach: run a dequantize kernel that expands INT4 weights to FP16 and writes them back to HBM, then run an ordinary FP16 matmul.

See the problem? The matmul still reads FP16 weights from HBM: not one byte of the traffic went away. The bandwidth payoff of quantization drops to zero, and you pay an extra kernel's round-trip on top. The whole exercise, wasted.

Dequantizing in a separate kernel erases the payoff; fusing it into the matmul is what actually saves bandwidth

The correct approach: fuse the dequantization inside the matmul kernel. The kernel loads INT4 weights straight from HBM (so the traffic is INT4-sized), decodes them to FP16 in registers, and multiplies immediately. The FP16 version of the weights exists only in registers, never touching the ground.

This is Day 5's fusion logic replayed: the payoff exists only if the intermediate (the decompressed weights) stays off the ground. Every practical weight-only inference kernel is built this way, and it explains why quantization is not just an algorithms problem but a kernel-and-compiler problem: quantization without fusion is quantization on paper.

How TVM and PyTorch Each Do It

Same conclusion, two quite different landings, which map neatly onto Day 2's layers.

TVM generates it. Dequantization is not a hand-written kernel but a TE / TIR compute expression: pull the right 4 bits out of the packed integer (shift and mask), subtract the zero point, multiply by that group's scale. The expression is the matmul's producer, and a scheduling primitive like compute_inline folds it into the matmul's loops, so the decoded FP16 lives only in registers. It is Day 5's fusion mechanism, applied to decoding. MLC-LLM's group quantization is built this way: the format (how many bits, how large a group, how it is packed) is a parameter, and the compiler generates the kernel from it, so a new format does not mean a new kernel.

PyTorch runs two tracks. One is prewritten packed kernels: APIs like torchao's int4_weight_only swap the weights for a packed integer tensor and route the call to a dedicated op such as _weight_int4pack_mm, where dequantization happens inside the kernel. The hottest path is handed to a hand-written implementation. The other track writes dequantization in ordinary PyTorch operators (shift, mask, multiply by scale, then matmul) and hands it to torch.compile, where Inductor fuses that elementwise chain into its consumer at the Triton level. Projects like gpt-fast demonstrate this one: the code reads as three naive lines of dequantization, and what lands is a fused kernel.

Put differently, TVM treats the quantization format as a compiler parameter, while PyTorch splits the work between tuned kernels for common formats and on-the-fly Inductor fusion for the rest. Both care about the same thing: the decoded FP16 must never touch the ground.

What the Compiler Does Here

Collecting the compiler's role, which lines up neatly with Day 2's layers:

Graph level: treat quantize and dequantize as graph nodes, then move and eliminate them. Adjacent q-dq pairs cancel; dequantize sinks toward its consumer so it can fuse. Essentially Day 3's constant folding and fusion, applied to a quantized graph.
Tensor level: generate the fused "load INT4, dequantize in registers, FP16 multiply-add" kernel, or, in fully quantized INT8 scenarios, select INT8 tensor-core kernels outright.
Where ranges come from: quantizing activations requires knowing their numeric range, which comes from calibration (run representative data, record distributions) or quantization-aware training. That side belongs to the algorithms people; the compiler's job is landing the result as graphs and kernels.

How quantization affects model quality, and which layers refuse to be quantized, belongs to another field; this series stays in the systems seat.

Conclusion

To close: quantization attacks the "size of each element" factor, and on memory-bound LLM decode the payoff is written directly in the bandwidth ledger: INT8 halves it, INT4 quarters it. The precision guardrail is the scale's granularity, a slider from per-tensor to per-group trading overhead for fidelity. The LLM mainstream is weight-only, because weights dominate the traffic and activations resist. And the whole thing stands only if dequantization fuses into the kernel so the decoded weights never touch the ground; otherwise it is all for nothing.

Key takeaways

Two points that are easy to trip over, collected here.

First, why "decode INT4 to FP16 once, store it back to HBM, and reuse it every token" does not pay off. It comes down to what decode is: generating each token rereads the entire set of weights from HBM. What you stored back is FP16, so every token now reads FP16 traffic, and none of the bandwidth quantization saved is ever collected. The dequant runs only once, true, but it cannot rescue the thousands of rereads that follow. This is exactly why dequantization must fuse into the kernel so that only INT4 enters HBM.

Second, where decoding sits differs between INT4 weight-only and INT8 full quantization. In INT4 weight-only, the hardware has no INT4-times-FP16 instruction, so the weights must be decoded back to FP16 before the multiply; decoding is before the matmul. In INT8 full quantization, both sides share a type and INT8 tensor cores multiply directly, then the accumulated INT32 is multiplied by the scales to recover a float; decoding moves after the matmul (this step is called requantization). The single test is whether the hardware can multiply the two types directly. And moving it after has a precondition: the scale must be constant across all of K, which holds for per-tensor and per-channel. Once scales go per-group, each group's coefficient differs, so they must be applied group by group as the accumulation runs.

Next Up

Ten days in, the concepts are all on the table: why compilers exist, the layers, the four moves, the roofline ruler, fusion, scheduling, search, dynamic shapes, quantization. The final post unfolds the map: torch.compile, TVM, XLA, TensorRT, where each captures, which moves each applies, and whom each generates code for. Concepts learned; time to meet the cast.