Kimi Delta Attention: A New Efficiency Trick for LLMs

A blog post explains Kimi Delta Attention, a method to reduce memory and compute in transformer models by caching attention differences. It claims up to 5x speedup on long sequences without accuracy loss. The technique is open-source and could lower costs for large-scale inference.