Overview
Pretraining navigation diffusion policies typically relies on large-scale expert demonstrations, often generated by an oracle planner tailored to a single nominal robot. This approach can limit the policy's ability to generalize to diverse robot embodiments and complex scenarios that require varied local reactive behaviors using only onboard local observations. Examples of such challenging scenarios include escaping dead ends or detouring long obstacles. While post-training with reinforcement learning (RL) offers a potential solution, previous RL methods for diffusion policies have yielded only marginal improvements. This is partly due to the intractable likelihood of diffusion policies, which can lead to unstable policy gradients and inefficient policy exploration.
To address these limitations, a new framework called Group Q-score Reweighted Matching (GQRM) has been proposed. This framework is designed for data-efficient diffusion RL post-training. The resulting fine-tuned policy, named X-NavDP, has been developed to enhance cross-embodiment visual navigation performance.
Approach
The GQRM framework incorporates two primary complementary designs:
- Self-bootstrapped exploration strategy with behavior perturbation: This strategy aims to preserve the pretrained policy prior while facilitating exploration.
- Group Q-score normalization mechanism: This mechanism computes per-trajectory values on each state. Its purpose is to enable efficient reweighted score matching.
The development of X-NavDP involved distributed online RL training conducted across heterogeneous embodiments. This distributed training approach allowed the policy to learn and adapt to different robot configurations and operational characteristics.
Findings
The X-NavDP policy demonstrated improvements in cross-embodiment visual navigation performance:
- In simulation, the overall success rate increased from 61.20% to 84.28%.
- In real-world hard cases, the success rate improved from 10% to 65%.
These results indicate that the GQRM framework's design, including its exploration strategy and Q-score normalization, contributes to more stable policy gradients and potentially more efficient policy exploration, overcoming some of the challenges observed in prior diffusion RL approaches.
Why This Matters
The generalization of navigation policies to diverse robot embodiments and challenging environments is critical for expanding the applicability of autonomous systems. Improving success rates in complex navigation scenarios, both in simulation and real-world conditions, can contribute to more robust and adaptable robotic systems.