Overview
Research introduces a novel two-stage reward scheduling strategy, termed $\textit{Run-then-Walk}$, for Vision-Language Model (VLM)-based autonomous driving planners. This strategy addresses challenges in existing GRPO-style reinforcement learning (RL) optimization approaches, which typically exhibit trade-offs between driving efficiency and safety, or require extensive training durations. The $\textit{Run-then-Walk}$ method aims to enhance both driving performance and convergence speed by decoupling the exploration of high-progress behaviors from the subsequent repair of unsafe tendencies.
Research Context
VLM-based autonomous driving planners frequently employ GRPO-style reinforcement learning for optimizing driving performance. Current GRPO recipes for these systems present a dilemma: they either prioritize driving efficiency, potentially leading to behaviors that are progress-seeking but lack safety, or they impose early safety constraints, which can result in overly conservative driving. Both of these existing optimization paradigms necessitate lengthy training periods for their respective objectives.
Approach
The proposed $\textit{Run-then-Walk}$ strategy is predicated on the identification of two distinct RL regimes: a 'progress regime' (Run-GRPO) designed for aggressive exploration of high progress, and a 'safety regime' (Walk-GRPO) intended to restore safety once stable progress has been achieved. The strategy itself is a two-stage reward scheduling mechanism that explicitly separates the discovery of progress from the repair of safety issues.
- Run Phase: This initial phase focuses on discovering progress. Its objective is to allow the policy to overcome conservative biases and identify high-progress operational modes.
- Walk Phase: Following the Run phase, the Walk phase is introduced. During this stage, endpoint and safety strategies are integrated to address and repair unsafe behaviors that may have emerged during the preceding Run phase.
This sequential approach is presented as an alternative to one-stage RL methods, which might attempt to optimize for progress, safety, or a combination thereof within a single training phase. By reversing the schedule compared to 'Walk-first' methods (which prioritize safety initially) and avoiding the joint optimization of 'unsafe progress-seeking' approaches, $\textit{Run-then-Walk}$ aims to overcome their respective limitations.
Findings
The $\textit{Run-then-Walk}$ strategy was validated using various VLM-based planners across multiple benchmarks. These benchmarks included NAVSIMv1, NAVSIMv2, Navhard, and nuScenes. Extensive experiments indicated the following outcomes:
- Improved driving performance across the evaluated benchmarks.
- A reduction in the required RL training epochs, specifically 40% to 50% fewer epochs compared to baseline methodologies.
Why This Matters
The $\textit{Run-then-Walk}$ strategy offers a method to enhance the training efficiency and performance of VLM-based autonomous driving planners. By reducing the training epochs by 40-50% while improving driving performance, this approach could address significant bottlenecks in the development and deployment of autonomous driving systems. Its ability to simultaneously improve safety and progress without lengthy training iterations is critical for practical applications.