Overview
REGEN (Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning) is a proposed method designed to advance the scalability and computational efficiency of training large language models (LLMs) via reinforcement learning (RL). Large-scale online RL is frequently employed for developing advanced LLM abilities, including long-term reasoning and agentic tool use. However, scaling online RL across extensive task domains poses challenges related to computational infrastructure and cost.
Research Context
Multi-teacher on-policy distillation (MOPD) is a known technique for distilling knowledge across various domains and training stages, decoupling the RL stage and potentially saving costs while maintaining generality. Despite its benefits, MOPD, similar to online RL, necessitates coupled inference and backward passes. This coupling continues to limit MOPD's scalability and computational efficiency. The research addresses these limitations, seeking alternative strategies for knowledge distillation in the context of LLMs.
Approach
REGEN diverges from the MOPD approach. Instead of relying on distillation from multiple teacher models, REGEN trains a generalist model by recycling the replay memory generated during the specialized RL training of teacher models. This replay memory is described as a 'free by-product' of the teacher models' training. The method employs offline RL algorithms on this recycled data. A key characteristic of REGEN is its complete decoupling of the rollout sampling from the backward training process. This architectural separation is intended to reduce training costs.
Findings
- REGEN achieves accuracy levels comparable to MOPD. This equivalence was observed across tested domains: mathematical reasoning, code generation, and instruction following.
- The proposed method operates at substantially lower training costs compared to MOPD. This cost reduction is attributed to the decoupling of rollout sampling and backward training.
- The approach suggests a potential recharacterization of online RL as a data synthesis process rather than a standalone learning stage.
- REGEN potentially enables large-scale post-training without requiring heavy computational loads.