Overview
OmniDreams is introduced as a foundation generative world model designed for real-time, closed-loop autonomous vehicle simulation. This model is mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos. OmniDreams functions within a closed-loop system where a driving policy model interacts with the environment, influencing the simulator state and subsequent generated sensor observations. The model's architecture enables the synthesis of phenomena such as extreme weather and dynamic agent behaviors. It acts as an environment in a system that includes the Alpamayo 1 policy model and the AlpaSim orchestrator.
Research Context
The advancement of autonomous vehicle capabilities necessitates robust evaluation of driving policies, particularly in long-tail scenarios. A critical bottleneck identified is the safe evaluation of these policies. Existing reconstruction-based neural simulators offer photorealism but are limited by their initial captured data, struggling to generalize to highly dynamic or novel scenes. The need to overcome these constraints motivates the development of generative world models capable of synthesizing complex, unobserved events. Closed-loop simulation specifically requires a dynamic environment where a driving policy's actions directly influence the simulated world and its sensor feedback.
Approach
OmniDreams was developed by mid- and post-training from the Cosmos diffusion model, leveraging its visual priors. The training process utilized 21,000 hours of driving scenarios. The model is designed to autoregressively condition its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. This allows it to generate action-conditioned videos in real time. For deployment and evaluation, OmniDreams is integrated into a closed-loop system alongside the Alpamayo 1 policy model and the AlpaSim orchestrator. This integration designates OmniDreams as a responsive and reactive environment within the simulation framework. Additionally, a world-action model (WAM) was post-trained from OmniDreams.
Findings
- OmniDreams autoregressively generates action-conditioned videos in real time, leveraging visual priors from the Cosmos diffusion model.
- The model effectively synthesizes complex, unobserved phenomena, including extreme weather and unpredictable dynamic agent behaviors, which are challenging for traditional simulators to capture.
- Deployed in a closed-loop system, OmniDreams performs as a responsive, reactive environment for training and evaluating autonomous driving policies.
- Preliminary results indicate that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset.
- This WAM surpassed the VLA-based Alpamayo 1.5 research policy model on the NuRec dataset, using 1/5 the total parameters.
Why This Matters
The ability of OmniDreams to generate complex, unobserved scenarios like extreme weather and unpredictable agent behaviors addresses a limitation in current reconstruction-based neural simulators, which are constrained by initially captured data and struggle to generalize. Its real-time, autoregressive nature in a closed-loop system provides a scalable solution for training and evaluating next-generation autonomous driving policies. The demonstrated performance of a derived world-action model, surpassing existing policy models with fewer parameters, indicates potential for world models like OmniDreams to also serve as a foundation for policy architectures.