Overview
Scene-specific Ambiguity-aware 3D Language Fields (SaaF) is a Gaussian Splatting-based 3D language field designed for interactive object retrieval within real-world environments. This system aims to address challenges in object retrieval where natural language is used as an input for service robots operating in complex settings.
Research Context
Interactive object retrieval using natural language is identified as a critical capability for service robots in intricate real-world scenarios. Prior 3D language field methods for object retrieval typically establish associations between rendered pixels and autoencoder-compressed CLIP features. However, these methods exhibit two primary limitations:
- Reduced discriminability among objects that are visually similar, attributed to feature compression.
- Suboptimal handling of query ambiguity, which frequently leads to unstable or incorrect retrieval outcomes.
Approach
SaaF introduces a metric learning strategy with the objective of constructing a unified feature space. This feature space is engineered to be both instance-discriminative and ambiguity-aware. The approach integrates two main components:
-
Instance-level Visual Discrimination Enhancement: To improve distinguishing between individual instances, SaaF employs a metric learning technique. This technique functions by drawing image features, captured from multiple viewpoints of the same object, closer together within the feature space.
-
Ambiguity Awareness Establishment: The model achieves ambiguity awareness by jointly training on multiple text labels. These labels are generated by a proposed method from each tracked object image sequence. This includes ambiguous descriptions, enabling the model to learn the semantic relationships between ambiguous and specific features pertinent to a target scene. This integrated feature space allows for fine-grained visual understanding and supports the system's capacity to estimate query ambiguity, facilitating interactive requests for clarification when a need for it arises.
Findings
Experimental results demonstrated the performance of SaaF. The key findings include:
- SaaF improved retrieval accuracy when compared to previous methods.
- The system robustly detects ambiguity in user text queries under open-vocabulary settings.
- SaaF effectively handles ambiguity in user text queries under open-vocabulary settings.
Why This Matters
The development of SaaF is relevant for service robots operating in complex real-world environments, where interactive object retrieval based on natural language is a crucial operational capability. The ability to enhance instance-level visual discrimination and manage ambiguous queries contributes to more stable and accurate retrieval processes for these robotic systems.