Overview
Research introduces OccludeBench, a manually curated benchmark designed to evaluate how well vision-language models (VLMs) reason about missing objects within images. The benchmark focuses on the recovery of fully occluded objects based solely on the remaining visual evidence. OccludeBench differentiates between two primary types of cases: direct cases, which require recovering a hidden object from specific physical cues such as shadows or reflections, and inferential cases, where the object's presence can be deduced from broader scene context.
The study utilized OccludeTrace, a four-stage diagnostic tool, to identify specific points where the visual reasoning process might fail in VLMs. A key intervention, termed inpaint-then-confront, was explored to address identified limitations in model performance.
Approach
OccludeBench was created as a manually curated benchmark to specifically test the capacity of VLMs to infer occluded objects. The benchmark categorizes scenarios into two distinct types:
- Direct cases: In these scenarios, the identification of the hidden object relies on specific physical evidence, such as a shadow or a reflection.
- Inferential cases: Here, the hidden object is inferred from the broader context of the scene rather than a singular physical cue.
Five distinct VLMs were evaluated using this benchmark. For diagnostic purposes, OccludeTrace was introduced. This diagnostic consists of four stages and aims to pinpoint the exact step in the visual reasoning process where models encounter difficulties. To potentially mitigate identified weaknesses, the study also investigated the effects of extended reasoning, tool use, and few-shot prompting. An additional intervention, described as inpaint-then-confront, was applied, prompting models first to generate a candidate hidden object and subsequently to evaluate this hypothesis against the visible physical cue.
Findings
Across the five evaluated VLMs, the average accuracy on OccludeBench was 42.7% for direct cases and 62.7% for inferential cases. This represents a 20.0-point accuracy gap between the two case types in VLM performance. In contrast, human participants achieved higher accuracies: 63.4% for direct images and 68.5% for inferential images, exhibiting no significant gap between these categories.
The OccludeTrace diagnostic identified a specific bottleneck in the VLM reasoning process. Models struggled most when attempting to recover the relevant object from its projection within the full image. They succeeded in this step for only 50% of examples where they ultimately failed to answer correctly, compared to 91% success on examples where they answered correctly.
Attempts to improve VLM performance through extended reasoning, tool use, and few-shot prompting yielded inconsistent improvements and did not reliably eliminate the observed gap. However, the inpaint-then-confront intervention demonstrated a more consistent impact. This intervention, which involved prompting models to first generate a candidate hidden object and then evaluate it against visible physical cues, successfully reduced the accuracy gap to a statistically non-significant difference for models possessing image-generation capabilities.
Why This Matters
The study establishes OccludeBench and OccludeTrace as a tractable, controlled diagnostic framework. This framework is designed to isolate and analyze how VLMs reason about visual content that is not explicitly present but must be inferred from an image. The findings highlight specific differences in VLM and human capabilities when processing occluded visual information.