Dev Tools · 1h ago
Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill
INT4 weight-only quantization reduces memory traffic but not FLOPs, so it speeds up decode but not prefill. Prefill is compute-bound, while decode is memory-bound. On H100, the crossover is around 74 concurrent tokens for INT4 vs 295 for BF16.
Meridian48 take
This clarifies a common misconception, showing that quantization benefits depend on workload characteristics, not just model size.
quantizationllm-inference