saw-int4: The Hadamard Trick That Makes 4-Bit KV Caches Work in Real Serving Systems
A system-aware quantization stack that rotates away KV outliers, keeps accuracy close to BF16, and treats inference speed as a kernel problem, not a paper abstraction.
- SAW-INT4 matters because it turns KV-cache quantization from a compression problem into an outlier-shaping problem.
- The Hadamard rotation is only useful here because it is wired into the serving runtime, not left as a paper-only transform.
- The project’s real edge is operational fit: runtime toggles, fused kernels, and benchmark scripts make the math measurable in production-like conditions.
- Its strongest claim is practical, not universal, because it targets a narrow but valuable lane between BF16 accuracy and INT4 efficiency.
The 4-bit trick is not compression. It is outlier management.
KV cache pressure is the part of long-context serving that quietly eats your budget. Every new token adds memory, bandwidth, and latency, and naive 4-bit quantization tends to fail where the distribution is ugly: outliers in K, and sometimes V, get crushed into error.
`saw-int4` does something smarter. It does not pretend those outliers are harmless. It rotates them first, so the large values get smeared across dimensions before the cache is quantized to INT4. The result is less distortion where it matters, and a serving stack that can trade a little math for a lot of memory relief.
We present System-Aware KV Cache Quantization (SAW), a new algorithm for 4-bit KV cache quantization. Designed to work well with existing LLM inference systems such as vLLM, SAW is both outlier-aware and system-aware to balance accuracy and efficiency.
Why the runtime matters more than the math
This repo is not a notebook with a clever trick taped to it. It is a serving integration. The code is built around SGLang forks, CUDA and Triton kernels, environment-variable switches, and benchmark orchestration that asks the same question production asks: what happens when the model actually serves traffic?
That matters because the right answer is rarely just lower error. It is lower error, in a layout the cache system can handle, with kernels the runtime can fuse, and with enough reproducibility to trust the throughput numbers.
Excited to share our work on 4-bit KV cache quantization that actually speeds up LLM serving! SAW is system-aware and outlier-aware, achieving up to 2.6x throughput with minimal accuracy loss. Code is open source.
BDR: rotate first, quantize second
Block Diagonal Rotation, or BDR, is the core trick. The Hadamard transform is appealing because it is cheap, structured, and orthogonal. Cheap means it does not need a heavy multiply pipeline. Orthogonal means the dot product survives the change of basis.
The block-diagonal part is the compromise that makes the idea practical on GPU hardware. A full rotation can be elegant and expensive. A block-wise rotation keeps the structure local enough to fit the runtime and cache hierarchy while still spreading the ugly values around enough to make INT4 behave.
The important identity is the boring-looking one that saves the whole design: if the rotation is orthogonal, the attention dot product does not change. That is why the system can rotate K before storage and rotate Q at decode time without changing the meaning of the computation.
How the serving path actually changes
The runtime is controlled with switches like `HADAMARD`, `HADAMARD_ORDER`, and `ROTATE_V`. That is not cosmetic. It means the same deployment can flip between behaviors without a rebuild, which is exactly what you want when you are testing whether a quantization method survives contact with a real server.
HADAMARD=1 \
HADAMARD_ORDER=128 \
ROTATE_V=1 \
python -m sglang.launch_server \
--model-path ... \
--enable-paged-kv-cache
During prefill, K is rotated before it enters the paged KV cache. During decode, Q goes through the same transform so the dot product still lines up. If `ROTATE_V` is enabled, V also participates, which makes the system more flexible but adds another trade-off to judge carefully.
Why the benchmark script is part of the story
`run_throughput_sweep.sh` is not support code. It is evidence. The script checks free GPU memory, waits for readiness, and then runs a controlled sweep across BF16, INT4, and BDR modes. That is how a paper claim becomes a measurement instead of a promise.
# simplified shape of the workflow
if [ "$GPU_FREE_MEM_MB" -gt "$MIN_MEM_MB" ]; then
wait_for_health_endpoint
run_genai_bench
fi
This is the difference between a quantization idea and a serving system. The first asks whether the math is elegant. The second asks whether the runtime behaves, whether the backend is ready, and whether the throughput numbers survive a real benchmark harness.
| Mode | Memory footprint | Throughput | Accuracy risk | Kernel complexity |
|---|---|---|---|---|
| BF16 KV cache | Highest | Baseline | Lowest | Lowest |
| Plain INT4 KV cache | Lowest | Often good on paper, but unstable in practice | High around outliers | Low to moderate |
| SAW-INT4 / BDR | Low | Designed to improve serving throughput | Low relative to plain INT4 | Higher, but integrated into the runtime |
The comparison that matters: BF16, plain INT4, and SAW-INT4
The right comparison is not academic completeness. It is deployment friction. BF16 is easy and expensive. Plain INT4 saves memory but tends to pay for it in accuracy. SAW-INT4 sits in the narrow space where the cache gets smaller and the model stays usable.
That balance is why the project is interesting. It is not trying to win every benchmark in every setting. It is trying to make 4-bit KV caches operationally believable for serving workloads that need longer context and higher throughput on the same hardware.
Where it still stops short
The scope is real, but it is not universal. The current implementation is tied to specific backends and to MHA-style attention, which means it does not automatically cover every modern architecture. Rotating V also introduces extra complexity that some deployments will not want.
That does not weaken the result. It clarifies it. `saw-int4` is a serious serving technique with a narrow lane, and that lane is valuable: make 4-bit KV cache storage practical without giving up the runtime behavior that serving systems depend on.





