Support-Guided Triadic Decoding for Hallucination Mitigation in Multimodal Dialogue with LVLMs
Authors: Yiheng Li, Wenhao Wang, Xingyi Li, Zhilin Tang, Zichang Tan, Zhen Lei, Yang Yang
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Vast Intelligence Lab · Institute of Information Engineering, Chinese Academy of Sciences · Sangfor Technologies Inc. · CAIR, HKISI, Chinese Academy of Sciences · SCSE, FIE, M.U.S.T
In multimodal multi-turn dialogue, dialogue history plays a dual role: reliable responses provide useful context for subsequent turns, whereas hallucinated responses can propagate errors across turns. Existing training-free methods globally adjust the effect of dialogue history on generation, making it difficult to suppress misleading information without weakening useful context. To address this trade-off, we introduce Support-Guided Triadic Decoding (SGTD), a training-free framework that selectively controls the influence of unreliable history and fuses complementary predictions during decoding. Specifically, Historical Answer Support Estimation measures the visual support of historical responses against type-matched alternatives and identifies low-support content that may mislead later generation. Support-Targeted Attention Intervention then weakens the influence of this content while preserving the surrounding dialogue, producing a support-refined prediction. Triadic Decoding combines this prediction with the full-history prediction and answer-free visual evidence, while the Support-Adaptive Plausibility Constraint filters implausible tokens using visual agreement and historical support. Experiments on MMHalSnowball and VisDiaHalBench across representative LVLMs show that SGTD consistently achieves the highest accuracy under hallucinated histories while preserving the benefits of factual context.
Figures & tables
Figure 1: Comparison of decoding strategies under unreliable dialogue history. (a) Standard decoding can follow an unsupported prior answer. (b) RVD applies a distribution-level correction without localizing unreliable history, potentially weakening reliable context. (c) SGTD uses answer support to refine low-support history, then jointly decodes from the Full-History, Support-Refined History, and Answer-Free Visual Branches.
Figure 2: Overview of Support-Guided Triadic Decoding (SGTD). Frozen BLIP compares checkable historical answers with type-matched alternatives to identify low-support turns. The Full-History, Support-Refined History, and Answer-Free Visual Branches preserve context, reduce attention to selected answer spans, and provide visual evidence independent of previous answers. Triadic Decoding combines their logits under the Support-Adaptive Plausibility Constraint.
Figure 3: Support-Targeted Attention Intervention. Declarative claims from low-support turns locate the corresponding answer tokens. SGTD subtracts a penalty from the pre-softmax attention scores directed to these tokens by subsequent queries.
Method
LLaVA-1.5-7B
mPLUG-Owl2-7B
ShareGPT4V-7B
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Regular
71.00
88.74
14.72
58.15
56.28
60.67
87.69
7.76
52.04
52.91
73.56
90.63
15.60
59.93
57.95
VCD [ 17 ]
70.02
88.84
17.60
58.82
52.42
61.51
88.62
8.57
52.90
52.95
73.56
90.63
19.18
61.12
54.37
M3ID [ 9 ]
72.37
90.27
15.02
59.22
57.35
62.54
88.88
7.22
52.88
55.32
74.40
91.76
15.79
60.65
58.62
PAI [ 25 ]
72.83
89.66
15.95
59.48
56.89
62.50
88.76
7.76
53.01
54.74
74.72
91.59
16.91
61.08
57.81
SID [ 11 ]
72.91
89.42
20.61
60.98
52.30
60.18
87.89
8.91
52.33
51.28
75.25
90.35
21.34
62.31
53.91
Table 1: Results on MMHalSnowball (%). Avg. is the mean over Clean, Fact., and Hallu., while H.D. is Clean minus Hallu. Under the Clean setting, RVD reduces to Regular decoding. Best and second-best results are shown in bold and underlined.
Method
LLaVA-1.5-7B
mPLUG-Owl2-7B
ShareGPT4V-7B
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Regular
77.16
86.38
30.72
64.75
46.44
70.90
86.10
31.14
62.71
39.76
79.04
84.82
23.18
62.35
55.86
VCD [ 17 ]
77.60
86.80
32.04
65.48
45.56
69.14
86.64
34.08
63.29
35.06
79.86
86.02
24.50
63.46
55.36
M3ID [ 9 ]
79.10
87.32
30.74
65.72
48.36
72.42
87.18
31.20
63.60
41.22
81.10
86.12
23.48
63.57
57.62
PAI [ 25 ]
79.92
87.24
31.34
66.17
48.58
72.44
87.08
31.90
63.81
40.54
80.68
85.78
24.02
63.49
56.66
SID [ 11 ]
78.72
86.86
33.84
66.47
44.88
73.90
86.40
30.82
63.71
43.08
80.16
85.74
25.68
63.86
54.48
Table 2: Results on VisDiaHalBench (%). Avg. is the mean over Clean, Fact., and Hallu., while H.D. is Clean minus Hallu.
Setting
MMHalSnowball
VisDiaHalBench
RVS
LSS
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
77.30
91.88
24.33
64.50
83.06
90.92
61.36
78.45
✓
77.30
79.25
51.23
69.26
83.06
89.66
82.77
85.16
✓
✓
77.30
91.17
48.34
72.27
83.06
91.28
83.04
85.79
Table 3: Ablation of Historical Answer Support Estimation. Results are reported for LLaVA-1.5-7B (%). RVS and LSS denote Relative Visual Support and Low-Support Selection.
Branch
MMHalSnowball
VisDiaHalBench
Full
Ref.
Vis.
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
✓
✓
77.30
91.19
45.59
71.36
83.06
91.68
81.60
85.45
✓
✓
77.30
91.74
24.39
64.47
83.06
90.86
61.38
78.43
✓
✓
71.00
88.08
39.05
66.04
77.16
85.60
72.66
78.47
✓
✓
✓
77.30
91.17
48.34
72.27
83.06
91.28
83.04
85.79
Table 4: Ablation of Decoding Branches. Results are reported for LLaVA-1.5-7B (%). Full, Ref., and Vis. denote the Full-History, Support-Refined History, and Answer-Free Visual Branches.
Figure 4: Sensitivity to intervention strength. Results are reported for (a) MMHalSnowball and (b) VisDiaHalBench with LLaVA-1.5-7B. Curves report Fact., Hallu., and (Fact.+Hallu.)/2 . The shaded marker denotes the selected λ=4 across both benchmarks. Setting λ=0 disables the attention penalty.
Constraint
MMHalSnowball
VisDiaHalBench
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
None
75.21
88.78
49.24
71.08
81.96
90.80
82.36
85.04
Fixed 0.1
75.69
89.93
47.66
71.09
82.24
91.34
81.54
85.04
Fixed 0.2
75.93
91.25
44.46
70.55
81.96
91.38
79.26
84.20
Fixed 0.3
76.59
91.55
42.53
70.23
82.94
91.34
78.04
84.11
Fixed 0.4
77.26
92.56
40.50
70.11
83.30
90.96
76.82
83.69
Table 5: Ablation of the Support-Adaptive Plausibility Constraint. Results are reported for LLaVA-1.5-7B (%).
Figure 5: Answer deletion versus attention intervention. Accuracy on MMHalSnowball and VisDiaHalBench with LLaVA-1.5-7B (%). Delete Answer removes the complete answer from each low-support turn, while SGTD keeps it in the history and applies Support-Targeted Attention Intervention to the selected span.
Backbone
Method
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
InternVL3.5-8B [ 35 ]
Regular
75.58
87.66
36.38
66.54
39.20
RVD
75.58
80.50
72.44
76.17
3.14
SGTD (Ours)
78.84
90.56
82.78
84.06
−3.94
Qwen2.5-VL-7B [ 3 ]
Regular
73.52
79.20
65.20
72.64
8.32
RVD
73.52
75.28
71.18
73.33
2.34
SGTD (Ours)
75.20
79.18
78.00
77.46
−2.80
Table 6: Generalization to Modern LVLM Backbones. Accuracy on VisDiaHalBench (%).
Figure 6: Qualitative comparison under hallucinated histories. The top and bottom rows show examples from MMHalSnowball and VisDiaHalBench, respectively, using LLaVA-1.5-7B. Regular decoding and RVD follow unsupported history, whereas SGTD recovers the image-supported answers. Red and green denote incorrect and correct predictions.
Multimodal Large Language Models frequently suffer from inference hallucinations, partially stemming from language priors dominating visual evidence. Existing training-free mitigation methods either perturb the visual representation and deviate from the natural image distribution, or enforce intrusive manipulations that compromise the model's inherent generative fluency. We introduce a novel perspective that multimodal hallucination manifests as the hypersensitivity of visual grounding to textual phrasing during the decoding phase. Building on this insight, we propose Decoding by Perturbation (DeP), a training-free framework mitigating prior-induced hallucinations via controlled textual interventions. DeP employs a dynamic probe applying multi-level textual perturbations to elicit latent language priors. Leveraging attention variance, it enhances stable evidence regions while suppressing suspicious noise in the feature space. Furthermore, it constructs an interpretable prior drift direction using logits statistics to counteract probability biases from textual co-occurrences. Extensive experiments confirm DeP effectively reduces hallucinations and achieves superior performance across multiple benchmarks.
Sihang Jia, Shuliang Liu, Songbo Yang +3
The Hong Kong University of Science and Technology (Guangzhou)
Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.
Large vision-language models (LVLMs) often hallucinate when language priors dominate weak or ambiguous visual evidence. Existing contrastive decoding methods mitigate this problem by comparing predictions from the original image with those from externally perturbed visual inputs, but such references can introduce off-manifold artifacts and require costly extra forward passes. We propose SIRA, a training-free internal contrastive decoding framework that constructs a counterfactual reference inside the same LVLM by exploiting the staged information flow of multimodal transformers. Instead of removing visual information from the input, SIRA first lets image and text tokens interact through a shared prefix, forming an aligned multimodal state that preserves prompt interpretation, decoding history, positional structure, and early visual grounding. It then forks a counterfactual branch in later transformer layers, where attention to image-token positions is masked. This branch retains the shared multimodal context but lacks continued access to fine-grained visual evidence, yielding a language-prior-dominated internal reference for token-level contrast. During decoding, SIRA suppresses tokens that remain strong without late visual access and favors predictions whose advantage depends on the full visual pathway. Experiments on POPE, CHAIR, and AMBER with Qwen2.5-VL and LLaVA-v1.5 show that SIRA consistently reduces hallucinations while preserving descriptive coverage and incurring lower overhead than two-pass contrastive decoding. SIRA requires no training, external verifier, or perturbed input, and applies to open-weight LVLMs with white-box inference access.
Tian Qin, Junzhe Chen, Yuqing Shi +3
Tsinghua University · The University of Sydney · Stanford University +1