AI · 2h ago
PIVOT Speeds Up Sparse Attention 4x Without Retraining
PIVOT groups adjacent queries that share ~90% of top-k token selections, running one proxy scan per group instead of per query. This reduces indexer complexity from O(L²) to O(L²/g), achieving a 4x speedup on DeepSeek-V3.2 and GLM-5.1. The method is training-free and offers two modes: PIVOT-Reuse for max speed and PIVOT-Refine for matching dense indexer accuracy.
Meridian48 take
A clever algorithmic trick that fixes a practical bottleneck in long-context models, but the 1.6x end-to-end gain may vary with hardware and batch size.
sparse-attentioninference-optimization