Overview
This research introduces a novel benchmark designed to evaluate visual evidence identification capabilities within multi-view Multimodal Large Language Models (MLLMs), particularly in the context of autonomous driving. The primary objective is to determine if MLLMs rely on the correct visual evidence, specifically the appropriate camera view, when generating answers to visual questions in complex multi-view scenes. The benchmark explicitly addresses a potential gap where models might produce seemingly accurate answers without necessarily grounding them in the correct visual source, a critical concern for safety-sensitive applications like autonomous driving.
Research Context
Multimodal Large Language Models have demonstrated proficiency in various visual reasoning benchmarks. However, the accuracy of their answers alone does not confirm that the underlying visual evidence used for reasoning was correct. This issue is particularly salient in autonomous driving scenarios, which frequently utilize multi-view camera setups. In such environments, a model could generate a plausible answer to a driving-related query while incorrectly attributing that answer to an irrelevant or incorrect camera view. The current evaluation methodologies often overlook this discrepancy between answer correctness and proper visual grounding.
Approach
The core of the research involves the creation of a multi-view visual question answering benchmark. This benchmark requires a model to identify the supporting camera view from a set of six synchronized NuScenes views, in addition to answering a given question. The dataset comprises 122 conflict-centric question-answer pairs derived from 73 distinct scenes. These questions are designed to probe causality, counterfactual reasoning, and intent prediction within complex driving situations.
The development of the benchmark involved an automatic conflict-mining pipeline to propose initial view labels, which were subsequently subjected to manual verification by human annotators. This dual approach aimed to ensure the relevance and accuracy of the designated supporting views.
Three distinct evaluation settings were employed:
- Camera-View Selection: Models were tasked solely with identifying the correct camera view.
- Oracle QA with Golden View: Models were provided with the 'golden' (correct) view and then asked to answer the question, simulating perfect visual grounding.
- Joint Prediction: Models were required to simultaneously select a view and provide an answer in a single step.
Answer evaluations were conducted using both multiple-choice and free-form formats. For structured predictions (e.g., multiple-choice), an exact match criterion was applied. For free-form responses, an LLM judge was utilized for assessment.
Findings
The benchmark's methodology, by explicitly separating visual-source identification from mere answer correctness, successfully exposed grounding failures within MLLMs that would not be detectable through answer-only evaluations. The use of conflict-centric scenarios, causality, counterfactual reasoning, and intent prediction questions in 122 question-answer pairs from 73 scenes highlighted instances where models produced plausible answers but failed to ground them in the correct visual evidence from the six synchronized NuScenes views. The evaluation across camera-view selection, oracle QA, and joint prediction settings provided different perspectives on these grounding capabilities and failures, with different methods for answer evaluation (exact match for structured and LLM judge for free-form responses).
Why This Matters
For applications such as autonomous driving, where safety and reliability are paramount, understanding not just what an MLLM answers, but also how it arrives at that answer and what visual data it relies upon, is critical. This benchmark provides a necessary tool to evaluate and improve the trustworthiness of MLLMs in multi-view scenarios. By identifying and quantifying visual grounding failures, it facilitates the development of MLLMs that are more transparent and reliable in their decision-making processes in complex real-world environments.