Causal Decoding for Hallucination-Resistant Multimodal Large Language Models

Shiwei Tan; Hengyi Wang; Weiyi Qin; Qi Xu; Zhigang Hua; Hao Wang

cs.LGFeb 24, 2026

Causal Decoding for Hallucination-Resistant Multimodal Large Language Models

Authors: Shiwei Tan, Hengyi Wang, Weiyi Qin, Qi Xu, Zhigang Hua, Hao Wang

Organizations: Rutgers University · Meta Ranking AI

Abstract

Multimodal Large Language Models (MLLMs) deliver detailed responses on vision-language tasks, yet remain susceptible to object hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, our framework substantially lowers object-hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.

Figures & tables

Figure 1: Simplified causal graphs for typical MLLMs and our COAD. Left: Typical MLLMs implicitly hallucinate objects (e.g., “fork”) in the hidden states ${\bf z}$ due to previously generated text ${\bf x}$ (e.g., “knife”). Right: Our COAD performs causal inference to remove links between the hidden states ${\bf z}$ and generated text ${\bf x}$ , thereby avoiding hallucination.

Panel of Figure 2 — Figure 2: Illustration of confounding. (a) $z$ induces a spurious association between $x$ and $y$ even without a causal effect. (b) Adding $x\rightarrow y$ introduces a genuine causal effect, but $P(y|x)$ remains confounded by $z$ .

Figure 3: Overview of our COAD. We employ an object detector to identify the objects present in an image. The MLLM is then finetuned to condition its token predictions on both these detected objects and the input image. COAD subsequently use causal inference to combine the output distributions of both the pretrained and finetuned MLLMs to generate the final prediction.

Panel of Figure 4 — Figure 4: Rolled-out causal structures of the MLLM decoding process. (a) Full temporal (rolled-out) causal graph over decoding timesteps: the image ${\bf S}$ and initial prompt ${\bf x}^{(0)}$ are fixed, ${\bf z}$ denotes the image-derived object-belief variable (constant over time), and ${\bf x}^{(t)}$ , $y^{(t)}$ evolve according to ${\bf x}^{(t)},{\bf S},{\bf z}\rightarrow y^{(t)}$ and ${\bf x}^{(t)},y^{(t)}\rightarrow{\bf x}^{(t+1)}$ . (b) Collapsed version of (a) obtained by treating $y^{(t)}$ as an implicit step in the autoregressive update, so that each ${\bf x}^{(t)}$ ( $t>0$ ) is jointly determined by ${\bf S}$ , ${\bf z}$ , and ${\bf x}^{(t-1)}$ . (c) Time-compressed representation in which the accumulated influence of all previous timesteps on ${\bf x}^{(t)}$ is summarized by dashed edges from $({\bf S},{\bf z})$ to ${\bf x}^{(t)}$ .

Panel of Figure 5 — Figure 5: Illustration of our COAD’s causal model before and after intervention.

Figure 6: Case study on caption generation. MSCOCO objects mentioned in the text are highlighted in red (hallucinated) or green (correct). We compare the baseline LLaVA with our COAD-enhanced model. While LLaVA hallucinates nonexistent objects (e.g., knife and fork ), the ${\bf z}$ -vector produced by the object detector suggests that these objects are absent. By leveraging this signal, COAD produces a faithful caption grounded in the actual image content, consistent with the improvements shown in CHAIR metrics.

Method	Base	PAI	DoLa	VCD	CAD	OPERA	EOS	HALC	COAD
$\text{CHAIR}_{I}\downarrow$	9.9	5.8	13.0	11.4	9.9	4.5	5.8	5.2	3.4
$\text{CHAIR}_{S}\downarrow$	29.6	11.3	37.0	32.5	28.0	7.4	10.6	11.1	5.3

Table 1: Comparison of different methods in terms of CHAIR metrics. Boldface and underlining denote the best and the second-best performance, respectively.

Method	Avg. Score	Hall. Rate	attr	adv	cmp	cnt	rel	env	hol	oth
Base	1.88	0.68	2.33	1.25	2.67	0.83	1.75	3.17	1.42	1.58
PAI	2.10	0.65	1.92	1.33	2.25	2.17	2.17	3.67	1.75	1.58
Dola	2.01	0.62	2.08	1.42	2.75	1.67	1.17	4.00	1.75	1.25
VCD	1.98	0.67	2.17	1.83	1.83	1.33	2.42	3.33	1.33	1.58
CAD	2.00	0.64	2.50	1.25	2.42	0.75	1.33	3.83	1.83	2.08
OPERA	2.09	0.65	2.58	1.67	2.67	2.50	1.58	3.08	1.17	1.50

Table 2: Evaluation on MMHal-Bench across 8 hallucination dimensions: attributes (attr), adversarial objects (adv), comparison (cmp), counting (cnt), spatial relations (rel), environment (env), holistic/overall description (hol), and others (oth). Boldface and underlining denote the best and the second-best performance, respectively.

Method	Random					Popular					Adversarial
Method	Acc	P	R	F1	Yes	Acc	P	R	F1	Yes	Acc	P	R	F1	Yes
Base	89.0	89.3	88.6	89.0	49.6	85.0	82.6	88.7	85.5	53.7	78.8	74.0	88.8	80.8	60.0
PAI	89.3	89.6	88.9	89.2	49.6	86.1	84.2	89.0	86.5	52.9	78.9	74.4	88.3	80.7	59.4
Dola	86.3	85.3	87.7	86.5	51.4	83.0	80.5	87.1	83.6	54.1	78.2	73.9	87.4	80.1	59.2
VCD	88.8	88.8	88.7	88.8	50.0	85.4	83.6	88.1	85.8	52.7	79.2	74.5	88.6	81.0	59.4
CAD	88.6	88.7	88.5	88.6	49.9	84.8	82.5	88.3	85.3	53.5	78.5	74.0	87.9	80.3	59.4

Table 3: POPE evaluation results on the MSCOCO dataset. Boldface and underlining denote the best and the second-best performance, respectively.

Method	CHAIR ${}_{I}\downarrow$	CHAIR ${}_{S}\downarrow$
COAD (Full)	3.4	5.3
COAD ( $M_{f}$ only)	5.4	10.8
COAD (w/o $\mathbf{z}$ )	6.9	18.1

Table 4: Results of COAD and ablations on CHAIR. “

M_{f}

only” means only using the finetuned model

M_{f}

for generation; “w/o

{\bf z}

” means replacing

M_{f}

by a normally finetuned MLLM and applying COAD, without any

{\bf z}

vectors involved in the whole process.

Method	Base	COAD (Ours)	OPERA	PAI	DoLa	VCD	CAD	EOS	HALC
#tokens/s $\uparrow$	24.37	10.49	4.52	43.62	29.38	7.98	7.92	9.91	7.32

Table 5: Decoding throughput of different methods on 100 MSCOCO images used in the CHAIR evaluation. We report the number of tokens per second during generation (higher is better).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: The role of hidden states in the decoding causal graph of an MLLM. This figure illustrates a causal graph that explicitly includes token-level hidden states ${\bf h}^{(t)}$ during generation (a) , and shows that collapsing these hidden states into the generation of $y^{(t)}$ yields an equivalent causal graph consistent with the one used by COAD (b) .

Panel of Figure 8 — Figure 8: Visualization of LLaVA’s hallucinatory object beliefs. This figure provides empirical evidence that LLaVA can form incorrect internal object-existence beliefs (corresponding to an incorrect ${\bf z}$ ) and shows how COAD corrects the resulting hallucination. (a) Linear-probe output of LLaVA’s object probabilities for all tokens over the full dialog, showing that the probed probability for the object “bench” increases both around certain image tokens and around the tokens where LLaVA actually generates the word “benches” (in red boxes). (b) The baseline LLaVA’s probed probability of the object “bench” over the image tokens (patches), showing high “bench” probabilities in some image tokens (highlighted in yellow); this indicates that the hallucinated belief may originate from the image-perception stage. (c) LLaVA hallucinates the object “bench” in the response. (d) COAD’s response, which eliminates the bench hallucination.

Figure 9: Simplified LLaVA architecture ( Liu et al., 2024c ) . The image is encoded by a vision encoder and then projected into the language model’s embedding space. The resulting visual embeddings $H_{v}$ are inserted as image tokens. Because $H_{v}$ is computed independently of textual tokens, the object-belief variable ${\bf z}$ corresponding to $H_{v}$ remains fixed during decoding. (This figure is adapted from ( Liu et al., 2024c ) .)

Panel of Figure 10 — Figure 10: Case studies on hallucination reduction.

Panel of Figure 11 — Figure 11: Case studies on hallucination reduction.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Aug 31, 2026

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Peiyang Xu, Xiaopei Zhu, Jun Zhu +1

Tsinghua University Beijing, China

Object Hallucination in VLMs Multimodal Contrastive Learning VLM Hallucination Mitigation

Ask PDF

May 27, 2026

Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.

Jingwen Wu, Xijun Zhang, Ge Song

School of Computer and Electronic Information, Nanjing Normal University, China

LLM Hallucination Mitigation Hallucination in Language Models Object Hallucination in VLMs Multimodal Large Language Models Activation Steering VLM Hallucination Mitigation

Ask PDF

Jul 23, 2026

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models

Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual content, textual context, or commonsense knowledge. Existing studies primarily address this problem through coarse-grained detection. However, these approaches often provide insufficient diagnostic information for understanding hallucination types and supporting downstream hallucination mitigation. To bridge this gap, we propose fine-grained hallucination diagnosis for MLLMs, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation. We develop an automated data generation pipeline and construct HalluScope-30K, a large-scale diagnostic dataset covering eight sources and five task categories. Based on this dataset, we design a multi-granular joint reward function and train two diagnosis models, HalluScope-4B and HalluScope-8B, which achieve state-of-the-art performance on both the MHALO benchmark and our fine-grained hallucination classification benchmark. Notably, detection and classification are mutually beneficial under joint optimization. Furthermore, diagnosis-driven feedback experiments show that the fine-grained diagnostic explanations produced by our model effectively guide target models to correct their hallucinations, with full diagnosis substantially outperforming all baselines on both Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Our code, data, and models are available at https://github.com/wkinglin/HalluScope.

Weilin Jin, Mingyu Wang, Wenbo Li +5

Peking University Beijing, China · Joy Future Academy Beijing, China

Hallucination Detection in VLMs Multimodal Hallucination LLM Hallucination Detection Multimodal Model Interpretability VLM Hallucination Mitigation

Ask PDF

Abstract

Object Hallucination in VLMs

Explore similar work

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models