Overview
Post-training approaches rely heavily on on-policy learning, where models train on their own generated outputs to curb distribution shift and safeguard previously acquired capabilities. However, on difficult reasoning tasks where a model fails every attempt, standard on-policy reinforcement learning yields no reward signal. While on-policy distillation provides token-level teacher supervision, early mistakes in a generated rollout cause subsequent teacher tokens to condition on already corrupted prefixes. Conversely, supervising models with off-policy teacher or gold references introduces large distribution shifts that degrade general retention.
To address this dilemma, researchers introduced projected distillation, a framework that conceptualizes on-policyness as a continuum rather than a binary distinction. Instead of replacing failed generations entirely with off-policy solutions, projected distillation minimally edits a student model's unsuccessful attempts into valid, correct targets. The system pairs these minimal corrections with contrastive optimization updates and periodically updates targets as the student model improves over training.
Research Context
In post-training workflows, training on a model's self-generated rollouts is standard practice to prevent catastrophic forgetting and maintain existing capabilities. On hard benchmark tasks, standard techniques face distinct failure modes:
- On-policy reinforcement learning: Provides zero reward-driven learning signal whenever a model fails across all sampled attempts.
- On-policy distillation: Employs next-token supervision from a stronger teacher model; however, if the student model makes an error early in generation, the teacher's subsequent supervision is conditioned on prefixes that already contain flawed steps.
- Off-policy reference supervision: Supplies gold answers or independent teacher solutions, but these distributions diverge markedly from the student's native generation patterns, accelerating capability erosion.
Approach
Projected distillation operates by treating the gap between on-policy and off-policy supervision as an adjustable continuum. The method takes failed student rollouts and applies minimal edits to transform them into correct solution targets. This ensures learning targets remain as close to the student's native distribution as possible while resolving the initial reasoning errors.
The training procedure relies on two core mechanisms:
- Contrastive updates: The optimization loss explicitly favors the minimally corrected targets over the original failed attempts.
- Online target refreshing: Corrections are continuously re-generated and updated as the student model's policy evolves, maintaining near-on-policy status even on problems where the student initially achieves zero task success.
The researchers also provide a theoretical analysis of the student's learning regret relative to the magnitude of the edits. They proved that regret scales linearly with the task horizon under minimal corrections—mirroring the theoretical guarantees of DAgger—whereas full-answer replacement causes regret to scale quadratically with the horizon, consistent with standard behavior cloning.
Findings
The framework was evaluated on two student models—Qwen3-8B and OLMo-2-7B—across five task domains spanning mathematics, code generation, science, and tool use. Evaluation assessed the trade-off between domain-specific performance gains and the retention of existing skills, measured as the area under the specialization-retention frontier.
Key experimental outcomes include:
- Projected distillation achieved the highest area under the specialization-retention frontier, outperforming Group Relative Policy Optimization (GRPO) and On-Policy Distillation (OPD) on 8 of 9 evaluation cells.
- Against SPOT and SODA baselines, the method recorded superior frontier area on 7 of 9 evaluation cells.
- Ablation experiments demonstrated that moving targets further off-policy steadily degraded the specialization-retention trade-off. Specifically, replacing minimal corrections with student-agnostic gold references caused retention scores to drop from 0.916 down to 0.744.
- Ablations also confirmed that online target refreshing and contrastive loss formulation were critical to preserving near-on-policy stability.
Why This Matters
The findings show that learning hard problem domains does not require defaulting to unconstrained off-policy demonstrations that erode model stability. By enforcing minimal edits and refreshing targets during training, models can resolve complete task failures while preserving base capabilities.