FlashAttention

FlashAttention is an algorithm that computes exact transformer attention dramatically faster by being smart about GPU memory movement. Standard attention materializes the full N×N score matrix in the GPU's slow high-bandwidth memory, and for long sequences the bottleneck is not arithmetic but shuttling that matrix back and forth. FlashAttention (Tri Dao et al., 2022) tiles the computation so blocks of queries and keys stay in fast on-chip SRAM, fusing softmax and matrix multiplies into one pass and recomputing small pieces during backprop instead of storing them. The result is the same mathematical output with several-fold speedups and memory that scales linearly rather than quadratically with sequence length — a key enabler of today's 100K+ context windows. Successors (FlashAttention-2 and 3) squeeze newer hardware harder. For builders it is infrastructure, not an API knob: the serving stacks and training frameworks you already use ship it by default. The word doing the most work in that description is exact. FlashAttention is not an approximation that buys speed with accuracy — it is an IO-aware reorganization of the same computation, producing mathematically identical attention outputs to a naive implementation. What changes is where the intermediate values live: by tiling queries, keys, and values into blocks that fit in on-chip SRAM and recomputing a few cheap quantities during the backward pass instead of storing them, it avoids repeatedly writing and reading the large intermediate score matrix to and from high-bandwidth memory, which on long sequences is the real bottleneck rather than arithmetic. Because it cuts memory traffic for both training and inference, the saving shows up as fewer GPU-hours on both sides of the model lifecycle — a cost reduction that gets passed down the stack to per-token API prices rather than appearing as a feature you buy. It is also worth being clear that FlashAttention is not a model architecture. It is an implementation-level kernel optimization applicable to any standard transformer, which is why it slots into existing models without retraining and why it has become a default component of production inference engines and mainstream training frameworks rather than a differentiator any single vendor owns. For builders the practical consequences are indirect but real: long-context features are affordable partly because of it, and if you self-host, confirming that your serving stack is actually using a FlashAttention kernel — rather than silently falling back to a naive path on an unsupported GPU generation or dtype — is one of the cheapest performance checks available, since the fallback is functionally correct and therefore easy to miss.