Dev Tools · 8h ago
PyTorch 2.13 FlexAttention on M1 Max: 7.83x Speedup for Sparse Attention
PyTorch 2.13 adds Metal GPU support for FlexAttention on Apple Silicon, enabling custom sparse attention rules. Benchmarks on an M1 Max show a 7.83x speedup over standard SDPA for 32,768 tokens with a 256-token local window. However, for dense causal attention, SDPA remains 19x faster.
Meridian48 take
The speedup is impressive for long, sparse sequences, but the narrow use case and unstable API mean most developers won't benefit yet.
Read the full reporting
Testing PyTorch 2.13 MPS FlexAttention on M1 Max: Up to 7.83x Faster for Sparse Attention →
DEV Community
pytorchapple-silicon