Speculative Decoding: Speed Gains vs. Compute Costs in LLM Inference

Speculative decoding uses a small draft model to accelerate large language model inference, claiming 60-85% speedup. The technique is mathematically lossless, but worst-case scenarios can be slower than standard autoregressive generation. Engineers must weigh the extra compute of the draft model against potential throughput gains.