SAVOR: Enhancing Visual Grounding in MLLMs via Confidence-Calibrated Reinforcement Learning for Hallucination Mitigation

Overview

Multimodal large language models (MLLMs) have advanced significantly in tasks such as visual question answering and image captioning. However, a persistent challenge involves their tendency to generate fluent claims concerning objects, attributes, or relations that lack grounding in the provided image – a phenomenon referred to as hallucination. Existing mitigation strategies often involve modifications to decoding processes during test time, which can introduce latency, or fine-tuning approaches using preference learning techniques like DPO variants. While these methods teach which answers are preferred, they do not inherently address situations where the model's own answer is unreliable.

The SAVOR framework posits that calibrated self-assessment provides a crucial missing signal for addressing this issue. SAVOR is a training framework designed to enhance visual grounding and mitigate hallucinations by enabling MLLMs to assess their own confidence in generated outputs.

Research Context

The problem of hallucination in MLLMs manifests as the generation of statements that are semantically coherent but factually inconsistent with the visual input. Traditional approaches to mitigate these hallucinations, as noted, include test-time decoding modifications and preference-based fine-tuning. Test-time decoding modifications can increase latency. Preference-based fine-tuning, such as DPO variants, trains models on preferred answers but does not intrinsically equip them with the ability to discern when their own generated answers are unreliable. This indicates a gap in current methodologies for enabling MLLMs to perform internal self-assessment of their output reliability.

Approach

The SAVOR framework integrates three primary components to address MLLM hallucination:

1. Output Schema Augmentation: SAVOR augments the MLLM's output schema to include explicit token and answer confidence scores. This allows the model to not only generate an answer but also an associated measure of its certainty.
2. Policy Optimization with GRPO: The framework optimizes the MLLM's policy using a novel objective function called Group Relative Policy Optimisation (GRPO). This objective specifically penalizes two types of errors: calibration error, where the model's stated confidence does not align with its accuracy, and poor abstention decisions, where the model fails to abstain when it should, or abstains when it shouldn't.
3. Inference-time Confidence-based Revisit: During inference, SAVOR leverages the learned confidence scores. The model is instructed to revisit visual evidence only when it expresses uncertainty regarding its initial output. This conditional re-evaluation mechanism aims to improve grounding by re-engaging with the visual input when the model's self-assessment indicates a potential for error.

The approach was evaluated across two recent MLLM backbones: InternVL3-8B and Qwen3-VL-8B. The experimental setup involved testing the framework on several benchmarks known for assessing hallucination and general MLLM capabilities.

Findings

Experiments conducted on various benchmarks yielded several key findings:

• SAVOR demonstrated a reduction in hallucination across the evaluated MLLMs. This was observed on specific benchmarks designed to measure hallucination, including POPE, HallusionBench, AMBER, and MMHal-Bench.
• The framework preserved general capabilities of the MLLMs, as evidenced by its performance on general MLLM capability benchmarks, specifically MME and MMBench. This indicates that the hallucination mitigation did not come at the cost of overall model performance on standard tasks.
• SAVOR achieved a lower Expected Calibration Error (ECE) compared to DPO and other decoding baselines. This suggests that the confidence scores generated by SAVOR are better aligned with the model's actual accuracy, indicating improved self-assessment capabilities.

Why This Matters

The ability of MLLMs to provide calibrated self-assessment, as enabled by SAVOR, is significant for improving the trustworthiness and reliability of AI systems. By reducing hallucinations and enabling models to identify when their answers might be unreliable, it addresses a core challenge in the deployment of MLLMs in sensitive applications. The reduced Expected Calibration Error further indicates a move towards more interpretable and predictable model behavior.