Overview
ST-WAM (Semantic-Temporal World Action Model) is introduced as a method to enhance the robustness of robotic manipulation under visual distribution shifts. This model addresses a limitation identified in existing World Action Models (WAMs), which often struggle when visual content changes due to their reliance on pixel-generative future supervision. The core concept behind ST-WAM is to disentangle action-relevant state transitions from task-irrelevant visual content by incorporating semantic representations.
Research Context
World Action Models (WAMs) represent an approach that jointly models robot actions and future visual dynamics. A recurring issue identified in these models is "Training-Distribution Hallucination." This phenomenon occurs when futures conditioned on visually shifted observations tend to hallucinate content from the training domain rather than accurately reflecting the current scene. This limitation suggests that the reliance on pixel-generative future supervision can entangle action-relevant state transitions with visual content that is not pertinent to the task, thereby limiting robustness when the visual environment changes.
A controlled frame-triplet diagnosis indicated that DINOv3 features demonstrated greater stability across visual shifts while better preserving task-state distinctions compared to Wan-VAE latents. This observation prompted the development of a model that leverages these more stable semantic features.
Approach
ST-WAM proposes an alternative to correcting predicted futures by using DINOv3 as a shared semantic representation. This semantic representation is employed for both future prediction and history retrieval, while retaining fine-grained VAE dynamics. The model does not explicitly generate futures during inference.
Key components of ST-WAM include:
- Dual-Space Future Experts (DSFE): This component is designed to jointly predict future VAE latents and DINO features. By operating in these dual spaces, DSFE aims to capture both fine-grained dynamics and semantic information.
- Current-Anchored Intent Retrieval (CAIR): CAIR retrieves task-relevant evidence from recent DINO history. This retrieval process is anchored by the current visual-language context, suggesting a mechanism for grounding past semantic information to the present task.
ST-WAM is trained end-to-end. The training process does not necessitate additional embodied pretraining or task-specific annotations.
Findings
ST-WAM achieved specific performance metrics on standard benchmarks and demonstrated improvements under visual distribution shifts:
- On the LIBERO benchmark, ST-WAM achieved a success rate of 98.7%.
- On the RoboTwin 2.0 benchmark, it achieved a success rate of 92.8%.
- Compared to Fast-WAM, ST-WAM improved zero-shot LIBERO-Plus performance by 21.3 percentage points.
- In real-world scenarios, ST-WAM more than doubled the success rate under visual shifts, increasing it from 25.8% to 61.5%.
These results suggest that semantic-temporal modeling can complement pixel-generative dynamics effectively to achieve robust manipulation.
Why This Matters
The observed improvements in performance, particularly under visual distribution shifts and in real-world settings, indicate that ST-WAM's approach to semantic-temporal modeling can lead to more reliable robotic manipulation systems. Addressing "Training-Distribution Hallucination" and enhancing robustness in varied visual environments are critical for deploying robotic systems in dynamic and unstructured settings.