Overview
SalsaAgent represents a multimodal embodied language model developed for interactive dance generation. This system focuses on enabling embodied agents to perform reactive, full-body motion, specifically in the context of salsa dancing. The model generates expressive follower motions for salsa, responding to both a human leader's movements and accompanying music.
Research Context
Embodied interaction involving humanoids necessitates specific capabilities, including bidirectional nonverbal reactivity, coordination, and synchrony. These elements are crucial for conveying cues and facilitating synchronized movement with a partner. For socially interactive embodied agents, the generation of reactive motion requires full-body expressiveness. This motion must also maintain contextual appropriateness while simultaneously achieving spatial and temporal synchrony with an interaction partner.
Approach
The development of SalsaAgent formulates partner interaction as a process of nonverbal token passing. This approach extends the vocabulary of a large language model (LLM) to incorporate and process several distinct token types: discrete motion tokens, pairwise relation tokens, and audio tokens. The methodology integrates full-body and pairwise-relation tokenizers. These tokenizers align language tokens and motion tokens with text descriptions of skeleton dynamics, which are automatically derived. The generation process then employs a two-stage token-to-diffusion pipeline.
Findings
Subjective and objective evaluations of SalsaAgent indicated improvements across several key metrics. The model demonstrated enhanced motion quality when compared to prior baselines. Furthermore, the evaluations showed improved two-person spatial coordination. The system also exhibited better coordination with both the music and the human partner relative to existing baselines.
Potential Applications
The source abstract focuses on the technical development and evaluation of SalsaAgent for interactive dance generation, but does not explicitly discuss broader potential applications beyond its immediate function in generating reactive salsa follower motions for embodied agents.