Overview
ARDY is a streaming generation framework designed for the interactive generation of 3D human motions. The system aims to bridge the gap between offline approaches, which offer precise control but lack real-time inference speed, and online methods, which achieve real-time synthesis but may compromise controllability or struggle with complex text semantics and long-horizon goals. ARDY integrates high-fidelity motion generation with controllability through online text prompts and flexible kinematic constraints.
Research Context
Generating realistic 3D human motions in real-time is identified as a key requirement for various applications, including animation, simulation, and humanoid robotics. The existing landscape of motion generation techniques presents a dichotomy: offline methods typically provide precise control via text and kinematic constraints but are limited by their inference speed for interactive scenarios. Conversely, online methods facilitate real-time synthesis but often exhibit limitations in controllability, handling complex text semantics, or managing long-horizon goals due to restricted context windows. The development of ARDY addresses this challenge by seeking to combine the benefits of both approaches.
Approach
ARDY’s methodology incorporates a hybrid representation and a two-stage autoregressive transformer denoiser. The hybrid representation combines explicit root features with a latent body embedding. This design aims to balance precise trajectory control with efficient generative learning. The two-stage autoregressive transformer denoiser features variable history context and is engineered to support conditioning on flexible, long-horizon kinematic constraints.
The training of ARDY was conducted on a large-scale motion capture dataset. The model was directly conditioned on text labels and kinematic constraints, which were sampled from ground truth poses. This training regimen was intended to enable ARDY to natively learn controllable generation capabilities, specifically supporting online prompting and flexible long-horizon goals.
Findings
- ARDY demonstrated high motion quality and constraint adherence during extensive evaluations.
- Evaluations were conducted on the HumanML3D benchmark and the Bones Rigplay dataset, which is described as large-scale and high-fidelity.
- The evaluation results validated the efficacy of ARDY's architectural decisions, including its hybrid representation and two-stage autoregressive transformer denoiser.
Potential Applications
The practical versatility of ARDY was demonstrated through an interactive demo. This demonstration highlighted several capabilities: dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard.