BF16 rounding may destabilize FlashAttention gradients
A September 29, 2026 report says BF16 rounding in FlashAttention may break a softmax-gradient symmetry and cause gradients to explode near the end of training. The authors propose GProj as a correction.
On September 29, 2026, the authors reported that BF16 rounding in FlashAttention may break a softmax-gradient symmetry and cause gradients to explode near the end of training. The issue matters to engineers training models with fused attention kernels in BF16.
The proposed method, GProj, is described as a correction. To verify the details, consult the original material and check how it characterizes the instability and the correction; the available summary gives no quantitative results or specific evaluation conditions. If you use AI to study or apply the method, do not submit internal code or data without authorization.