A paper dated June 30, 2026 evaluates learning-rate scaling in GPT-2-style models and reports an upward trend for the optimal rate.
Published on June 30, 2026, the paper evaluates how learning rate scales when training GPT-2-style models ranging from 22 million to 707 million parameters, using 5 billion to 100 billion tokens. Its summary reports an upward trend in the optimal learning rate.
The text also cautions that extrapolating rates from smaller runs may require accounting for nonlinearity and effective learning rate. To verify the finding, consult the original paper by title and check its methods, settings, and results; the available summary does not state the trend’s magnitude.