Overview
Research introduces SOLID (Solver-Informed On-Policy LearnIng through Self-Distillation), a framework designed to enhance operations research (OR) language models. This method enables self-improvement by leveraging solver-artifact feedback generated from a model's own rollouts, addressing limitations in current training methodologies. SOLID operates without requiring verified answers or external evaluators, utilizing self-distillation to provide dense supervision.
Research Context
Modern large language models (LLMs) can convert natural-language descriptions into operations research (OR) formulations. Post-training techniques, including reinforcement learning and on-policy self-distillation, have previously refined this capability. However, existing training paradigms for LLMs in OR formulations face specific limitations:
- Training often relies on synthetic formulations validated by human experts or more advanced models, which restricts scalable supervision.
- Credit assignment mechanisms are either coarse or costly. Outcome rewards score an entire trajectory without pinpointing the specific responsible modeling decision, while process-level supervision necessitates an additional evaluator.
- Privileged self-distillation may introduce a style mismatch due to its use of solver context that is unavailable during deployment.
This work identifies that a model can achieve improvement through solver-artifact feedback derived from its own rollouts, positioning self-distillation as a practical, evaluator-free source for dense supervision.
Approach
The SOLID framework is built upon the principle of self-improvement for OR language models without relying on verified answers or external evaluators. The core process involves:
- Executing candidate programs generated from multiple rollouts of the model.
- Clustering the objectives derived from these executed programs.
- Selecting a majority-group artifact to serve as a pseudo-reference.
- Performing model updates. These updates incorporate group-relative advantages and dense self-supervision signals, which are derived from the solver artifacts.
Findings
The application of SOLID demonstrated improvements in solution accuracy across several OR benchmarks. This enhancement was observed in both general-purpose models and models specifically tuned for OR tasks. These results indicate that solver artifacts can facilitate scalable self-improvement for language models without requiring trusted answers or external ground truth for validation.
Why This Matters
The development of SOLID provides a mechanism for OR language models to self-improve their formulation capabilities without dependency on verified answers or external human/stronger model evaluators. This approach can address scalability constraints in supervision and the cost or coarseness of credit assignment in current LLM training for OR tasks.