Dev Tools · 18h ago
KV Cache Quantization Stretches Qwen 35B Context 8x on 12GB GPU
A developer used llama.cpp's KV cache quantization flags (-ctk q8_0 -ctv q8_0) to shrink cache memory by 50%, enabling a context window jump from 4096 to 32768 tokens on an RTX 4070 with 12GB VRAM. Decode speed dropped negligibly from ~34.6 to ~34.1 tokens/sec, with perplexity impact under 0.1%. The technique frees VRAM for agent workloads without model weight changes.
Meridian48 take
The trick is simple and reproducible, but the 8x context gain is specific to models where MoE layers are offloaded to CPU — a niche setup that limits broader applicability.
Read the full reporting
KV Cache Quantization: I Stretched Qwen 35B's Context 8 on 12GB VRAM →
DEV Community
kv-cache-quantizationllama-cpp