Overview
P$^{3}$-WAM is introduced as a unified world action model specifically designed for egocentric dexterous manipulation. This model learns three distinct, yet complementary, capabilities: 3D perception, action-conditioned prediction, and goal-directed planning.
Approach
P$^{3}$-WAM operates through three primary modes:
- Perceive: This mode reconstructs articulated hand motion and camera trajectories, deriving this information from video input.
- Predict: In this mode, the model generates visual futures. These futures are conditioned on prescribed actions, indicating the anticipated visual outcomes of specific actions.
- Plan: The Plan mode is responsible for jointly generating action trajectories and their anticipated visual consequences. This generation is conditioned on an initial image, a language instruction, and an optional visual goal.
The model's architecture integrates a compact, structured 3D action representation within a pretrained video diffusion transformer. This integration is achieved through a mixture-of-transformers architecture. Key architectural features include shared base weights, modality-specific adaptation, and bidirectional visual-action attention. The training methodology employs randomized modality visibility, allowing both video and 3D actions to function interchangeably as either conditioning context or prediction targets across the three operational modes.
Findings
Experimental evaluations demonstrate that unified training within P$^{3}$-WAM enhances all three integrated capabilities when compared to training each capability in a single-mode context. Specifically, gains in planning quality were observed, which were assessed under a vision-language model evaluation. Through this unified training approach, the model exhibited superior performance against specialized baselines. This included improved results in 3D hand reconstruction and action-conditioned video prediction across four distinct datasets. Furthermore, the model produced goal-directed manipulation videos that showed stronger adherence to instructions and more robust action progression when compared to other video-generation baselines.
Why This Matters
P$^{3}$-WAM's unified training approach leads to improved performance in 3D hand reconstruction, action-conditioned video prediction, and goal-directed manipulation, suggesting advancements in complex robotic tasks. The integration of perception, prediction, and planning within a single framework could simplify the development of dexterous manipulation systems.