SAM in mid-training and retention after fine-tuning
A post published on May 8, 2026 proposes using SAM during the final roughly 10% of mid-training to reduce forgetting after fine-tuning or quantization.
A post dated May 8, 2026 claims that controlling sharpness with SAM during the final roughly 10% of mid-training can reduce forgetting by more than 35% after fine-tuning or quantization. The text says this may happen even when base-model quality declines; it also suggests testing learning rates up to about 10 times higher.
The proposal aims to balance initial model quality with later retention, but the figures are claims from the post, not independently validated findings provided in the material. To assess them, consult the original publication and check its method, metrics, and experimental conditions before applying the results. When using AI to study the topic, do not submit personal data or internal content without checking your organization’s policy.