Overview
Research investigated the extent to which benchmark accuracy in vision-language models (VLMs) reflects grounded visual understanding. The study was motivated by an observation that removing a significant portion of image tokens resulted in only a marginal performance degradation on a widely used VLM hallucination benchmark. This discrepancy led to a systematic analysis of several open-source VLMs to understand the fidelity of their visual grounding.
Research Context
Benchmark accuracy is frequently presumed to indicate a VLM's reliance on visual evidence. However, the degree to which these scores genuinely reflect such reliance has remained unclear. The initial observation regarding minimal impact from image token removal on a hallucination benchmark prompted a deeper inquiry into this assumption.
Approach
The investigation utilized a multi-level approach to analyze the behavior of open-source VLMs. This spanned:
- Global visual degradation.
- Localized occlusion.
- Question reformulation.
- Answer-space expansion.
- Decision-level analyses, extending beyond standard accuracy metrics.
In addition to behavioral analyses, the study incorporated a layer-wise investigation of vision-token geometry. This was complemented by a representation-level analysis.
Findings
The experiments revealed that VLMs do integrate visual input, yet their predictions exhibit less sensitivity to the loss of fine-grained visual evidence than what standard accuracy metrics would suggest. Even when the final prediction from a VLM remained unaltered despite visual degradation, the model's internal support for the correct answer was observed to be weakened. The representation-level analysis further indicated an increasing similarity among visual tokens in deeper layers of the models. These findings collectively suggest that contemporary benchmarks may not be sufficient for reliably evaluating fine-grained visual grounding within VLMs.
Specifically, a surprising observation was that removing a substantial fraction of image tokens led to only a slight degradation in model performance on a widely used hallucination benchmark. This indicates a potential disconnect between measured accuracy and true visual reliance.
Why This Matters
The findings suggest that current benchmarks may not adequately assess the fine-grained visual grounding capabilities of Vision-Language Models. This implies that high benchmark scores might not correspond to a robust understanding of visual details, potentially impacting the development and evaluation of VLMs where precise visual interpretation is critical.