Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention

T1Covered by 1 source · first reported by arXiv at 24 Sept, 04:00 UTC

arXiv:2605.21292v2 Announce Type: replace-cross Abstract: Unbiased stochastic gradients can match the full gradient in expectation while changing finite-step stability. We study this effect in one-layer linear self-attention for in-context linear regression, where shared-mode mini-batch training…

science

Sources

• T1Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-AttentionarXiv24 Sept, 04:00