A study investigates scalable pretraining from random initialization using a program generator and a learner, without training on real data.
The work explores whether self-play could enable scalable pretraining without training on real data. Starting from random initialization, a generator proposes programs for a universal Turing machine, and a learner trains on the results.
The study reports predictable reductions in zero-shot validation loss across images, text, audio, and melodies, as well as in-context learning results. If you use AI to study or apply this approach, avoid submitting personal data or internal documents without authorization, and follow your organization’s data-protection rules.