Overview
Research addresses a failure mode in Grounded Video Question Answering (VideoQA) identified as question-invariant grounding. This phenomenon manifests when models produce nearly identical temporal segments as evidence for different questions asked about the same video content. To mitigate this, a new approach named GroundFormer has been developed.
Research Context
Grounded VideoQA systems aim to answer natural language questions about video content by identifying and presenting specific temporal segments (evidence) from the video that support the answer. The observed issue of question-invariant grounding suggests a limitation in how current models link question semantics to the precise temporal evidence needed for a given query. Prior common designs are characterized by two structural limitations contributing to this behavior:
- Modality Isolation: Video representations are fixed before the integration of question semantics. This design choice restricts the dynamic interplay between visual information and textual query intent.
- Weak Question Injection: The grounding module, responsible for localizing temporal evidence, incorporates question information in a suboptimal manner, leading to insufficient influence of the question on the grounding process.
Approach
GroundFormer is proposed to address the identified limitations. Its design principles are centered on integrating question intent more deeply into the video feature processing and temporal localization stages.
Question-Conditioned Feature Generation
GroundFormer conditions video features on question intent before the localization process. This is achieved through the use of learnable communication tokens. These tokens facilitate a directed visuo-lingual interaction, enabling question semantics to influence the construction of video representations directly.
Answer Selection and Temporal Evidence Coupling
Building upon the question-conditioned features, the system employs a factorized Multiple Instance Learning (MIL) cross-attention mechanism. This mechanism couples answer selection with the identification of temporal evidence. The process operates under candidate-level supervision, ensuring that the model learns to associate specific answer candidates with corresponding video segments.
Temporal Segment Generation
To convert the often-peaked attention scores into coherent temporal segments, Gaussian smoothing is applied. This technique transforms attention distributions into continuous and temporally meaningful video segments.
Multi-modal Alignment
A hierarchical multi-modal contrastive loss is introduced. This loss function is designed to align video, question, and answer embeddings. This alignment is conducted across a two-pass training pipeline, which likely refers to an iterative or multi-stage training process, though specifics of the two passes are not detailed in the source.
Findings
GroundFormer demonstrates enhanced performance in Grounded VideoQA:
- It achieves state-of-the-art grounded VideoQA performance.
- It substantially improves question-discriminative temporal grounding.
Performance was evaluated on two specific datasets: NExT-GQA and STAR.
Why This Matters
The research addresses a critical yet previously overlooked failure mode in Grounded Video Question Answering, which directly impacts the reliability and precision of temporal evidence provided by these models. By improving question-discriminative temporal grounding, the system's ability to accurately pinpoint video segments relevant to specific questions is enhanced.