Causal Decoding for Hallucination-Resistant Multimodal Large Language Models

Shiwei Tan; Hengyi Wang; Weiyi Qin; Qi Xu; Zhigang Hua; Hao Wang

cs.LGFeb 24, 2026

Causal Decoding for Hallucination-Resistant Multimodal Large Language Models

Authors: Shiwei Tan, Hengyi Wang, Weiyi Qin, Qi Xu, Zhigang Hua, Hao Wang

Organizations: Rutgers University · Meta Ranking AI

Abstract

Multimodal Large Language Models (MLLMs) deliver detailed responses on vision-language tasks, yet remain susceptible to object hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, our framework substantially lowers object-hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.

Figures & tables

Figure 1: Simplified causal graphs for typical MLLMs and our COAD. Left: Typical MLLMs implicitly hallucinate objects (e.g., “fork”) in the hidden states ${\bf z}$ due to previously generated text ${\bf x}$ (e.g., “knife”). Right: Our COAD performs causal inference to remove links between the hidden states ${\bf z}$ and generated text ${\bf x}$ , thereby avoiding hallucination.

Panel of Figure 2 — Figure 2: Illustration of confounding. (a) $z$ induces a spurious association between $x$ and $y$ even without a causal effect. (b) Adding $x\rightarrow y$ introduces a genuine causal effect, but $P(y|x)$ remains confounded by $z$ .

Figure 3: Overview of our COAD. We employ an object detector to identify the objects present in an image. The MLLM is then finetuned to condition its token predictions on both these detected objects and the input image. COAD subsequently use causal inference to combine the output distributions of both the pretrained and finetuned MLLMs to generate the final prediction.

Panel of Figure 4 — Figure 4: Rolled-out causal structures of the MLLM decoding process. (a) Full temporal (rolled-out) causal graph over decoding timesteps: the image ${\bf S}$ and initial prompt ${\bf x}^{(0)}$ are fixed, ${\bf z}$ denotes the image-derived object-belief variable (constant over time), and ${\bf x}^{(t)}$ , $y^{(t)}$ evolve according to ${\bf x}^{(t)},{\bf S},{\bf z}\rightarrow y^{(t)}$ and ${\bf x}^{(t)},y^{(t)}\rightarrow{\bf x}^{(t+1)}$ . (b) Collapsed version of (a) obtained by treating $y^{(t)}$ as an implicit step in the autoregressive update, so that each ${\bf x}^{(t)}$ ( $t>0$ ) is jointly determined by ${\bf S}$ , ${\bf z}$ , and ${\bf x}^{(t-1)}$ . (c) Time-compressed representation in which the accumulated influence of all previous timesteps on ${\bf x}^{(t)}$ is summarized by dashed edges from $({\bf S},{\bf z})$ to ${\bf x}^{(t)}$ .

Panel of Figure 5 — Figure 5: Illustration of our COAD’s causal model before and after intervention.

Figure 6: Case study on caption generation. MSCOCO objects mentioned in the text are highlighted in red (hallucinated) or green (correct). We compare the baseline LLaVA with our COAD-enhanced model. While LLaVA hallucinates nonexistent objects (e.g., knife and fork ), the ${\bf z}$ -vector produced by the object detector suggests that these objects are absent. By leveraging this signal, COAD produces a faithful caption grounded in the actual image content, consistent with the improvements shown in CHAIR metrics.

Method	Base	PAI	DoLa	VCD	CAD	OPERA	EOS	HALC	COAD
$\text{CHAIR}_{I}\downarrow$	9.9	5.8	13.0	11.4	9.9	4.5	5.8	5.2	3.4
$\text{CHAIR}_{S}\downarrow$	29.6	11.3	37.0	32.5	28.0	7.4	10.6	11.1	5.3

Table 1: Comparison of different methods in terms of CHAIR metrics. Boldface and underlining denote the best and the second-best performance, respectively.

Method	Avg. Score	Hall. Rate	attr	adv	cmp	cnt	rel	env	hol	oth
Base	1.88	0.68	2.33	1.25	2.67	0.83	1.75	3.17	1.42	1.58
PAI	2.10	0.65	1.92	1.33	2.25	2.17	2.17	3.67	1.75	1.58
Dola	2.01	0.62	2.08	1.42	2.75	1.67	1.17	4.00	1.75	1.25
VCD	1.98	0.67	2.17	1.83	1.83	1.33	2.42	3.33	1.33	1.58
CAD	2.00	0.64	2.50	1.25	2.42	0.75	1.33	3.83	1.83	2.08
OPERA	2.09	0.65	2.58	1.67	2.67	2.50	1.58	3.08	1.17	1.50

Table 2: Evaluation on MMHal-Bench across 8 hallucination dimensions: attributes (attr), adversarial objects (adv), comparison (cmp), counting (cnt), spatial relations (rel), environment (env), holistic/overall description (hol), and others (oth). Boldface and underlining denote the best and the second-best performance, respectively.

Method	Random					Popular					Adversarial
Method	Acc	P	R	F1	Yes	Acc	P	R	F1	Yes	Acc	P	R	F1	Yes
Base	89.0	89.3	88.6	89.0	49.6	85.0	82.6	88.7	85.5	53.7	78.8	74.0	88.8	80.8	60.0
PAI	89.3	89.6	88.9	89.2	49.6	86.1	84.2	89.0	86.5	52.9	78.9	74.4	88.3	80.7	59.4
Dola	86.3	85.3	87.7	86.5	51.4	83.0	80.5	87.1	83.6	54.1	78.2	73.9	87.4	80.1	59.2
VCD	88.8	88.8	88.7	88.8	50.0	85.4	83.6	88.1	85.8	52.7	79.2	74.5	88.6	81.0	59.4
CAD	88.6	88.7	88.5	88.6	49.9	84.8	82.5	88.3	85.3	53.5	78.5	74.0	87.9	80.3	59.4

Table 3: POPE evaluation results on the MSCOCO dataset. Boldface and underlining denote the best and the second-best performance, respectively.

Method	CHAIR ${}_{I}\downarrow$	CHAIR ${}_{S}\downarrow$
COAD (Full)	3.4	5.3
COAD ( $M_{f}$ only)	5.4	10.8
COAD (w/o $\mathbf{z}$ )	6.9	18.1

Table 4: Results of COAD and ablations on CHAIR. “

M_{f}

only” means only using the finetuned model

M_{f}

for generation; “w/o

{\bf z}

” means replacing

M_{f}

by a normally finetuned MLLM and applying COAD, without any

{\bf z}

vectors involved in the whole process.

Method	Base	COAD (Ours)	OPERA	PAI	DoLa	VCD	CAD	EOS	HALC
#tokens/s $\uparrow$	24.37	10.49	4.52	43.62	29.38	7.98	7.92	9.91	7.32

Table 5: Decoding throughput of different methods on 100 MSCOCO images used in the CHAIR evaluation. We report the number of tokens per second during generation (higher is better).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: The role of hidden states in the decoding causal graph of an MLLM. This figure illustrates a causal graph that explicitly includes token-level hidden states ${\bf h}^{(t)}$ during generation (a) , and shows that collapsing these hidden states into the generation of $y^{(t)}$ yields an equivalent causal graph consistent with the one used by COAD (b) .

Panel of Figure 8 — Figure 8: Visualization of LLaVA’s hallucinatory object beliefs. This figure provides empirical evidence that LLaVA can form incorrect internal object-existence beliefs (corresponding to an incorrect ${\bf z}$ ) and shows how COAD corrects the resulting hallucination. (a) Linear-probe output of LLaVA’s object probabilities for all tokens over the full dialog, showing that the probed probability for the object “bench” increases both around certain image tokens and around the tokens where LLaVA actually generates the word “benches” (in red boxes). (b) The baseline LLaVA’s probed probability of the object “bench” over the image tokens (patches), showing high “bench” probabilities in some image tokens (highlighted in yellow); this indicates that the hallucinated belief may originate from the image-perception stage. (c) LLaVA hallucinates the object “bench” in the response. (d) COAD’s response, which eliminates the bench hallucination.

Figure 9: Simplified LLaVA architecture ( Liu et al., 2024c ) . The image is encoded by a vision encoder and then projected into the language model’s embedding space. The resulting visual embeddings $H_{v}$ are inserted as image tokens. Because $H_{v}$ is computed independently of textual tokens, the object-belief variable ${\bf z}$ corresponding to $H_{v}$ remains fixed during decoding. (This figure is adapted from ( Liu et al., 2024c ) .)

Panel of Figure 10 — Figure 10: Case studies on hallucination reduction.

Panel of Figure 11 — Figure 11: Case studies on hallucination reduction.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
Aug 31, 2026Peiyang Xu, Xiaopei Zhu, Jun Zhu +1Object Hallucination in VLMs Multimodal Contrastive Learning
Tsinghua University Beijing, China
Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
May 27, 2026Jingwen Wu, Xijun Zhang, Ge SongLLM Hallucination Mitigation Hallucination in Language Models
School of Computer and Electronic Information, Nanjing Normal University, China
HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
Jul 23, 2026Weilin Jin, Mingyu Wang, Wenbo Li +5Hallucination Detection in VLMs Multimodal Hallucination
Peking University Beijing, China · Joy Future Academy Beijing, China

Aug 31, 2026J/K move · Enter open · S save

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Peiyang Xu, Xiaopei Zhu, Jun Zhu, Xiaolin Hu

Tsinghua University Beijing, China

Object Hallucination in VLMs Multimodal Contrastive Learning VLM Hallucination Mitigation

Open paper AskPDF arXiv

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Abstract

Object Hallucination in VLMs

Explore similar work

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models