saw-int4: The Hadamard Trick That Makes 4-Bit KV Caches Work in Real Serving Systems

A system-aware quantization stack that rotates away KV outliers, keeps accuracy close to BF16, and treats inference speed as a kernel problem, not a paper abstraction.

8 to 10 min read • View on GitHub • More from togethercomputer

A wide black-ink illustration of a KV cache lattice being passed through a compact rotating mechanism. Dark outlier spikes on the left are spread into smaller marks by a block-diagonal transform in the center, then packed into a narrow INT4 storage grid on the right while attention continues flowing. It explains that the project wins by reshaping the data before compression, not by simply hoping 4-bit quantization behaves.
SAW-INT4 does not remove outliers. It rotates them into something the cache can hold.
Key Takeaways

The 4-bit trick is not compression. It is outlier management.

KV cache pressure is the part of long-context serving that quietly eats your budget. Every new token adds memory, bandwidth, and latency, and naive 4-bit quantization tends to fail where the distribution is ugly: outliers in K, and sometimes V, get crushed into error.

`saw-int4` does something smarter. It does not pretend those outliers are harmless. It rotates them first, so the large values get smeared across dimensions before the cache is quantized to INT4. The result is less distortion where it matters, and a serving stack that can trade a little math for a lot of memory relief.

We present System-Aware KV Cache Quantization (SAW), a new algorithm for 4-bit KV cache quantization. Designed to work well with existing LLM inference systems such as vLLM, SAW is both outlier-aware and system-aware to balance accuracy and efficiency.

Why the runtime matters more than the math

This repo is not a notebook with a clever trick taped to it. It is a serving integration. The code is built around SGLang forks, CUDA and Triton kernels, environment-variable switches, and benchmark orchestration that asks the same question production asks: what happens when the model actually serves traffic?

That matters because the right answer is rarely just lower error. It is lower error, in a layout the cache system can handle, with kernels the runtime can fuse, and with enough reproducibility to trust the throughput numbers.

The system-aware part is the point: rotation, quantization, cache storage, and decode-time restoration are all part of one runtime path.

Excited to share our work on 4-bit KV cache quantization that actually speeds up LLM serving! SAW is system-aware and outlier-aware, achieving up to 2.6x throughput with minimal accuracy loss. Code is open source.

Tri Dao, Author, Chief Scientist at Together AI · X Post by Tri Dao

BDR: rotate first, quantize second

Block Diagonal Rotation, or BDR, is the core trick. The Hadamard transform is appealing because it is cheap, structured, and orthogonal. Cheap means it does not need a heavy multiply pipeline. Orthogonal means the dot product survives the change of basis.

The block-diagonal part is the compromise that makes the idea practical on GPU hardware. A full rotation can be elegant and expensive. A block-wise rotation keeps the structure local enough to fit the runtime and cache hierarchy while still spreading the ugly values around enough to make INT4 behave.

A close-up black-ink illustration of two lanes in a serving pipeline. One lane is jagged and crowded with outlier spikes, while the other passes through a block-diagonal rotation gate that smooths the values before they enter a compact INT4 cache. The lanes reconnect at a shared attention output, showing that the transform changes storage format without changing the final computation.
The mechanism is simple in principle: rotate, store, then recover the same attention math later.

The important identity is the boring-looking one that saves the whole design: if the rotation is orthogonal, the attention dot product does not change. That is why the system can rotate K before storage and rotate Q at decode time without changing the meaning of the computation.

How the serving path actually changes

The runtime is controlled with switches like `HADAMARD`, `HADAMARD_ORDER`, and `ROTATE_V`. That is not cosmetic. It means the same deployment can flip between behaviors without a rebuild, which is exactly what you want when you are testing whether a quantization method survives contact with a real server.

HADAMARD=1 \
HADAMARD_ORDER=128 \
ROTATE_V=1 \
python -m sglang.launch_server \
  --model-path ... \
  --enable-paged-kv-cache

During prefill, K is rotated before it enters the paged KV cache. During decode, Q goes through the same transform so the dot product still lines up. If `ROTATE_V` is enabled, V also participates, which makes the system more flexible but adds another trade-off to judge carefully.

Why the benchmark script is part of the story

`run_throughput_sweep.sh` is not support code. It is evidence. The script checks free GPU memory, waits for readiness, and then runs a controlled sweep across BF16, INT4, and BDR modes. That is how a paper claim becomes a measurement instead of a promise.

# simplified shape of the workflow
if [ "$GPU_FREE_MEM_MB" -gt "$MIN_MEM_MB" ]; then
  wait_for_health_endpoint
  run_genai_bench
fi

This is the difference between a quantization idea and a serving system. The first asks whether the math is elegant. The second asks whether the runtime behaves, whether the backend is ready, and whether the throughput numbers survive a real benchmark harness.

ModeMemory footprintThroughputAccuracy riskKernel complexity
BF16 KV cacheHighestBaselineLowestLowest
Plain INT4 KV cacheLowestOften good on paper, but unstable in practiceHigh around outliersLow to moderate
SAW-INT4 / BDRLowDesigned to improve serving throughputLow relative to plain INT4Higher, but integrated into the runtime

The comparison that matters: BF16, plain INT4, and SAW-INT4

The right comparison is not academic completeness. It is deployment friction. BF16 is easy and expensive. Plain INT4 saves memory but tends to pay for it in accuracy. SAW-INT4 sits in the narrow space where the cache gets smaller and the model stays usable.

That balance is why the project is interesting. It is not trying to win every benchmark in every setting. It is trying to make 4-bit KV caches operationally believable for serving workloads that need longer context and higher throughput on the same hardware.

Where it still stops short

The scope is real, but it is not universal. The current implementation is tied to specific backends and to MHA-style attention, which means it does not automatically cover every modern architecture. Rotating V also introduces extra complexity that some deployments will not want.

That does not weaken the result. It clarifies it. `saw-int4` is a serious serving technique with a narrow lane, and that lane is valuable: make 4-bit KV cache storage practical without giving up the runtime behavior that serving systems depend on.