FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
Research paper
Together AI presents FlashAttention-4, an algorithm and kernel co-design for Blackwell GPUs that maximizes overlap between matmul and resource bottlenecks like exponential units and shared memory traffic, achieving up to 1605 TFLOPs/s on B200 with BF16.
