A September 21, 2026 publication describes Trust Region DPO (TR-DPO), which incorporates a KL divergence penalty into the DPO loss to limit deviation from the reference model.
On September 21, 2026, a publication presented Trust Region DPO (TR-DPO). The approach adds a KL divergence penalty directly to the DPO loss, aiming to limit deviation from the reference model. According to the author, this stabilizes offline RL training.
The method may interest engineers assessing stability and distribution shift in DPO training. Consult the original publication to check its formulation and reported results; the available summary does not detail experiments or metrics.