An article published on December 8, 2025 reports that models of up to 7 billion parameters retained their original pretraining loss with about 0.4% of attention connections.
Published on December 8, 2025, the article presents a post-training technique that uses sparsity regularization with a constrained loss to make transformer attention sparse. The text reports that, in models of up to 7 billion parameters, the approach retained the original pretraining loss while using about 0.4% of attention connections.
The result may interest engineers studying attention circuits: sparsity is proposed as a structural prior for interpretability. The available summary does not detail the protocol, additional metrics, or evaluation limits. To verify the finding, consult the original article by its title and publication date and review its full methods and results. If you use AI to study it, avoid submitting personal data or internal information unless your organization's policy permits it.