Overview
DIPOLE (DIffusion POlicy with compLementarity Encoders) is a visuomotor policy designed to enhance generalization in imitation learning. It addresses limitations of existing methods that struggle when test-time conditions diverge from demonstration conditions, such as alterations in lighting, texture, viewpoint, object placement, or object identity.
The policy operates by fusing complementary modalities through a training-time mechanism, distinct from relying on a specialized fusion architecture. This approach incorporates a modality-wise dropout during training to ensure each modality maintains individual informativeness. A lightweight cross-attention layer facilitates the exchange of complementary cues between the modalities.
Research Context
Imitation learning is an approach for acquiring visuomotor skills based on demonstrations. A critical aspect of this process involves the design of effective observation encoders, which are essential for achieving policy generalization. However, current methodologies encounter difficulties when conditions during testing vary from those present during training demonstrations. These variations can encompass diverse visual changes and spatial reconfigurations.
Approach
DIPOLE employs a mechanism for fusing complementary modalities during the training phase. This mechanism is not a dedicated fusion architecture but rather an integrated training routine. A key component is modality-wise dropout, which selectively masks one modality branch at each training step. This encourages each modality to retain its individual informational value. Subsequently, a lightweight cross-attention layer is utilized to facilitate the exchange of complementary cues between the two modalities being processed.
Findings
DIPOLE exhibited five core strengths:
- Stable high performance across diverse tasks.
- Robustness to visual changes.
- Spatial generalization at sub-centimeter precision.
- Emergent capabilities extending beyond what either individual modality could achieve.
- Zero-shot transferability to previously unseen objects.
Across evaluation involving 18 simulated and 4 real-world tasks, DIPOLE demonstrated an average performance gain of 39.1% compared to six baseline methods. Specifically, it showed gains of 41.5% when confronted with unseen visual distractors and 15.2% under conditions of randomized object placement.
The system's design contributes to its ability to perform robustly under conditions where test-time visual and spatial configurations differ from those encountered during demonstrations. The training-time fusion and modality-wise dropout encourage complementary learning between the modalities.
Why This Matters
The robust generalization capabilities demonstrated by DIPOLE in visuomotor tasks, particularly under varying visual and spatial conditions, could contribute to developing more adaptable robotic systems. Its ability to perform zero-shot transfer to unseen objects and maintain performance despite distractions suggests potential for broader applicability in dynamic environments.