Overview
Orca is introduced as an initial instantiation of a general world foundation model. This model learns a unified world latent space by processing multimodal world signals. Its approach centers on Next-State-Prediction modeling, aiming to provide a unified state-transition modeling route for understanding, predicting, and acting upon the world. Orca's learned latent space is exposed through multimodal readout interfaces.
Research Context
The development of Orca diverges from optimization strategies focused on isolated next-token, next-frame, or next-action prediction. Instead, it emphasizes a unified state-transition modeling paradigm. This paradigm is designed to facilitate understanding, prediction, and action within the world through a consolidated framework.
Approach
Orca's learning mechanism integrates two complementary paradigms:
- Unconscious Learning: This paradigm captures dense natural state transitions. It operates on continuous video data.
- Conscious Learning: This paradigm models sparse, meaningful state transitions. It utilizes supervision from language-described events and Visual Question Answering (VQA).
For pre-training, a large-scale world-learning inventory data set was constructed. This dataset includes approximately 125,000 hours of video data and 160 million event annotations. Following pre-training, Orca establishes a unified world latent space. The backbone of Orca remains frozen, with only lightweight, modality-specific decoders being trainable.
Findings
To assess the learned latent space's efficacy for downstream applications, Orca was evaluated using three representative downstream readouts:
- Text generation.
- Image prediction.
- Embodied action generation.
Experiments indicated the scalability of the proposed paradigm. The results also verified that a stronger world latent space enables stronger downstream readouts. In comparisons, Orca demonstrated superior performance against similar-sized specialized baselines in these evaluations. These outcomes suggest that Orca, as a general world foundation model, presents a promising approach for understanding, predicting, and acting upon the world.
Key Limitations Mentioned by Researchers
The research discusses current limitations of Orca. The aim of highlighting these limitations is to provide insights and inspiration for the community.