Overview
EvalStop is a proposed composable scheduling primitive designed to detect and correct reward overoptimization within multi-tenant Reinforcement Learning from Human Feedback (RLHF) platforms. These platforms typically use learned reward models as proxies for human quality during fine-tuning of Large Language Models (LLMs). The system identifies reward overoptimization by monitoring downstream evaluation metrics and intervenes by terminating jobs exhibiting consistent declines in these metrics. This approach aims to improve resource utilization and job completion times by releasing shared GPU resources and preserving the best-performing checkpoints.
Research Context
Cloud LLM fine-tuning platforms are increasingly integrating RLHF workloads. A central challenge in these operations involves optimizing a learned reward model, which functions as a proxy for human quality. Prior research, specifically by Gao et al. (2023), has demonstrated that under sustained optimization pressure, this proxy can diverge from actual 'world feedback'—defined as downstream evaluation metrics. This phenomenon is termed 'reward overoptimization'.
Existing platform schedulers often do not account for this divergence. Specifically:
- Non-clairvoyant schedulers primarily optimize Job Completion Time (JCT) without incorporating any quality signal.
- SLAQ-style quality-aware schedulers utilize training loss, which is identified as a weaker proxy and can monotonically decrease even when undesirable 'hacking' behavior occurs.
- Classical per-job early stopping mechanisms typically require manual human monitoring and do not automatically release shared GPU resources.
The research frames scheduler-level early stopping as a detection problem, aiming to automatically identify instances of reward overoptimization.
Approach
EvalStop operates by terminating jobs when they exhibit $k$ consecutive declines in evaluation scores. Upon termination, it releases the associated GPUs and preserves the best-performing checkpoint from that job. The system is designed to be a composable scheduling primitive, compatible with any base scheduler.
The evaluation of EvalStop was conducted within a discrete-event simulator. This simulator modeled RLHF workloads comprising a mix of 'reward-hacking' runs and 'structurally healthy' runs. Critically, the ground-truth labels differentiating these run types were not accessible to the schedulers during simulation. This setup allowed for an objective assessment of EvalStop's detection capabilities.
Findings
In RLHF-heavy workloads, characterized by 80% RLHF jobs and utilizing 64 GPUs, EvalStop demonstrated specific performance metrics:
- Detection Performance: It achieved a precision of 98%, a recall of 99%, and a False Positive Rate (FPR) of 1.5%.
- Job Completion Time (JCT): EvalStop improved JCT by 9% compared to SRTF-Est ($p < 0.05$).
- Compute Efficiency: It reduced wasted compute by 22% when compared to SRTF-Est.
The study also compared EvalStop against competitor approaches:
- Trivial fixed-progress competitors incurred a 65% FPR on healthy RLHF runs.
- Loss-plateau competitors also exhibited a 65% FPR on healthy RLHF runs.
- Both trivial fixed-progress and loss-plateau competitors missed over half of the actual reward-hacking cases.
EvalStop's gains were observed to compose across every tested base scheduler, resulting in JCT improvements ranging from 9-25%.
The detection quality of EvalStop remained stable under varying conditions:
- Eval Noise: Precision was at least 91% even with an evaluation noise standard deviation of 0.05.
- Hacking Base Rate: Precision was at least 89% across hacking fractions ranging from 20% to 80%.
Why This Matters
The findings indicate that EvalStop effectively detects and mitigates reward overoptimization in RLHF platforms. Its ability to improve Job Completion Time and reduce wasted compute suggests a potential for increased efficiency and cost-effectiveness in cloud LLM fine-tuning operations. The robustness of its detection quality under varying noise levels and hacking rates suggests it could be a reliable mechanism for managing complex, real-world RLHF workloads.