See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
Authors: Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia · Macquarie University, Sydney, Australia · University of Surrey, Guildford, United Kingdom · University of New South Wales, Sydney, Australia
Multimodal large language models (MLLMs) frequently hallucinate objects that are absent from the visual input, often because attention during decoding is disproportionately drawn to visually dominant or frequently occurring content. We observe that this inequity in attention allocation is a root cause of object hallucination: when rare, small, or contextually peripheral objects receive insufficient attention, the model fails to ground its generation in the full visual scene. We argue that every object in an image, regardless of its size, frequency or visual salience, deserves equal representational opportunity during decoding. To this end, we propose DOP-OBC, a training-free and architecture-agnostic decoding strategy built on the principle of equitable attention. Two complementary object-aware signals work in tandem: a Dominant Object Penalty (DOP) that softly suppresses attention over-concentration on visually dominant regions, and an Outlier Boost Coefficient (OBC) that amplifies attention toward rare yet confidently detected objects. These signals are injected as per-row logit modulations within the causal attention mask, requiring no weight updates and preserving autoregressive decoding properties. Extensive experiments across image and video MLLMs demonstrate consistent reductions in object hallucination on CHAIR and POPE benchmarks, alongside improvements in GPT-4o assessed captioning quality across correctness, consistency, detail, context and temporal dimensions. DOP-OBC establishes that fairness in attention allocation is not merely a design principle but a practical and effective path toward more faithful multimodal generation.
Figures & tables
Figure 1: DOP–OBC Method Overview. (I) Standard causal masking in transformer attention. (II) Equity-aware modulation adjusts row-wise logit amplitudes: the Dominant Object Penalty (DOP) softly suppresses over-attended, visually dominant objects to free representational capacity, while the Outlier Boost Coefficient (OBC) amplifies rare yet confidently detected objects. (III) These object-aware signals are integrated with an upper-triangular positional decay register to form an equitable attention matrix at inference time, requiring no weight updates.
Figure 2: Example showing that DOP–OBC produces a more complete, grounded description than the LLaVA-1.5 baseline: the method adds correct background entities (highlighted in green in the figure text), and the accompanying attention-map comparison illustrates the reallocation of attention toward those previously neglected objects.
Figure 3: Cross-benchmark percentage improvement of DOP–OBC over base models across general multimodal benchmarks and hallucination metrics, showing consistent gains in task performance and reductions in CHAIR/POPE errors.
Figure 4: CHAIR and POPE-P comparisons on LLaVA-1.5 and Video-LLaVA. We report CHAIR S (lower is better) and POPE-P (higher is better) under RoPE, FixVPE, EDVT, FarSight, and DOP-OBC(Ours) . The rightmost bars ( DOP-OBC(Ours) ) achieve the largest improvements across both backbones.
Base Model
Method
CHAIR S ↓
CHAIR I ↓
POPE-R ↑
POPE-P ↑
LLaVA-1.5 [ 17 ]
LLaVA-1.5 (RoPE)
48.0
13.9
87.0
82.8
FixVPE
47.3
13.4
87.5
84.7
EDVT
46.8
14.5
87.8
85.4
FarSight
41.6
13.2
90.5
86.1
DOP–OBC (Ours)
39.7 (Δ-8.3)
11.2 (Δ-2.7)
92.0 (Δ+5.0)
89.3 (Δ+6.5)
Video-LLaVA [ 13 ]
Video-LLaVA (RoPE)
50.2
15.6
81.6
85.3
Table 1: Comparison of decoding methods on CHAIR and POPE benchmarks. ↓ indicates lower is better; ↑ indicates higher is better. Δ denotes improvement of DOP–OBC over the respective LLAVA-1.5 baseline.
Base Model
Method
MSVD-QA
ActivityNet-QA
Video-Based Text Generation
Accuracy ↑
Score ↑
Accuracy ↑
Score ↑
Cr. ↑
Cs. ↑
De. ↑
Ct. ↑
Te. ↑
Char-UniVi [ 9 ]
Base
64.6
3.6
43.1
3.2
2.84
2.93
2.55
3.16
2.43
DOP-OBC
69.2 (+4.6)
4.0 (+0.4)
43.0 (-0.1)
3.6 (+0.4)
2.90 (+0.06)
3.00 (+0.07)
2.62 (+0.07)
3.30 (+0.14)
2.56 (+0.13)
Video-LLaVA [ 13 ]
Base
64.8
3.7
41.5
3.3
2.32
2.34
2.65
2.75
2.09
DOP-OBC
72.2 (+7.4)
4.1 (+0.4)
44.2 (+2.7)
3.8 (+0.5)
2.97 (+0.65)
2.65 (+0.31)
3.12 (+0.47)
3.21 (+0.46)
2.47 (+0.38)
VILA [ 15 ]
Base
72.6
4.0
50.2
3.3
3.14
3.40
2.71
3.43
2.58
Table 2: Video benchmarks. For Video-Based Text Generation we report GPT-4o–assessed metrics: Correctness (Cr.), Consistency (Cs.), Detail (De.), Context (Ct.), Temporal (Te.). Higher is better.
Base Model
Method
MMBench ↑
LLaVAW ↑
MM - Vet ↑
VizWiz ↑
SQA ↑
CHAIRS ↓
CHAIRI ↓
POPE - R ↑
POPE - P ↑
POPE - A ↑
LLaVA-1.5 [ 17 ]
Base
64.3
72.5
30.5
48.5
64.5
48.0
13.9
87.0
82.8
76.6
ICD
63.1
69.7
30.4
46.9
62.8
47.7
13.6
87.9
84.0
80.2
VCD
63.9
70.9
29.5
43.4
63.3
46.8
13.2
87.0
83.5
78.1
OPERA
64.4
72.0
31.4
50.0
64.9
45.2
12.7
88.8
82.8
79.2
FarSight
66.0
74.7
32.5
50.8
67.4
41.6
13.2
90.5
86.1
80.4
DOP-OBC
69.6 (+5.3)
77.3 (+4.8)
35.1 (+4.6)
52.0 (+3.5)
68.7 (+4.2)
39.7 (+8.3)
11.2 (+2.7)
92.0 (+5.0)
89.3 (+6.5)
83.2 (+6.6)
Table 3: Image benchmarks. Lower is better for CHAIRS / CHAIRI ; higher is better for all others.
Figure 5: Comprehensive analysis of DOP–OBC’s impact on multimodal performance. (a) Accuracy ( ↑ ) vs. CHAIRS ( ↓ ): DOP–OBC defines the empirical frontier, achieving higher accuracy with lower hallucination. (b) Relative Difference Heatmap (DOP–OBC ∣ Base): consistent gains across ten benchmarks, improving general metrics and POPE while reducing CHAIRS/CHAIRI. (c) Equity-Aware Quality Growth: outward shifts across Correctness, Consistency, Detail, Context, and Temporal dimensions indicate uniform improvements in video caption quality.
Figure 6: Qualitative comparison and attention dynamics during decoding. LLaVA-1.5 (base model) exhibits early attention collapse, leading to omission of peripheral objects (e.g., traffic lights, street signs, red vehicle). FarSight partially mitigates this effect but remains unstable. DOP–OBC maintains stable image attention throughout generation, correctly grounding all GT-mentioned objects without architectural modification or retraining.
Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Vishal Narnaware, Animesh Gupta, Kevin Zhai +2
Institute of Artificial Intelligence, University of Central Florida
Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence, causing them to fall back on linguistic priors, rather than faithful grounding. In this work, we start from an empirical observation: when query-relevant visual evidence is explicitly strengthened using the model's own attention, generation becomes more accurate, suggesting that many failures do not arise solely from missing perception, but from an insufficient tendency to trust the evidence the model has already attended to. Motivated by this finding, we propose Oriented Pickup Preference Optimization (OPPO), an evidence-aware alignment objective that learns preferences over the strength of visual evidence, rather than only response quality. Concretely, OPPO contrasts the same faithful response under stronger, anchored, weaker-evidence views, turning naive visual preference into ordered visual-evidence alignment. We further combine this objective with fine-grained span-level and token-level regularization to stabilize the training. Besides, we provide a theoretical analysis showing that ordered evidence margins induce a positive lower bound on local visual sensitivity. Extensive evaluations across hallucination and general-purpose benchmarks demonstrate that OPPO consistently outperforms baseline methods.