Overview
This research introduces PoseRefer, a system employing pathway-local parameters for semantically grounded reference resolution. The work focuses on how a robot integrates gesture, language, and scene geometry to interpret commands like "put the cup on that one." It addresses limitations in existing 3D grounding benchmarks, which often feature post-hoc descriptions, templated gestures, or staged pointing.
Research Context
Current 3D grounding benchmarks present several challenges for natural human-robot interaction. Descriptions in these benchmarks are frequently generated after the fact, gestures are often templated rather than naturally occurring, and pointing actions are typically staged for recording. This contrasts with real-world scenarios where robots need to process natural co-speech gestures alongside linguistic input and 3D scene information.
Approach
The study utilized the MM-Conv dataset, which captures natural co-speech gestures from dyadic virtual reality (VR) interactions, coupled with full-body motion capture and 3D scene graphs. This dataset provided the multimodal input necessary for evaluating pose-language fusion. The researchers implemented a decoupled late-fusion architecture. In this design, the pose and text pathways do not share learned parameters across their respective processing. This architectural choice, combined with the late-fusion strategy, facilitated the isolation of contributions from category, pose, and text through controlled ablations.
Findings
- Fusion with frozen MiniLM category embeddings consistently surpassed the performance of both pose-alone and text-only pathways.
- The top-1 accuracy for this fusion method reached 31.9% across every reference type evaluated.
- A learned scalar gate within the system dynamically adjusted its operating policies. This adjustment was dependent on whether the text pathway had access to category information.
- The architectural decoupling of pathways was identified as a diagnostic tool. It suggests that claims regarding fusion accuracy in semantic grounding systems could be indistinguishable from artifacts related to category representation if pathways are not independently structured.
Why This Matters
The study's findings indicate that a decoupled architecture is crucial for accurately assessing the individual contributions of different modalities (pose, language, category information) in semantic grounding systems. Without such decoupling, performance gains attributed to fusion might be misleading, potentially stemming from inherent category representations within the pathways rather than true multimodal integration. This methodological clarity can lead to more robust and explainable multimodal AI development.