Support-Guided Triadic Decoding for Hallucination Mitigation in Multimodal Dialogue with LVLMs
Authors: Yiheng Li, Wenhao Wang, Xingyi Li, Zhilin Tang, Zichang Tan, Zhen Lei, Yang Yang
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Vast Intelligence Lab · Institute of Information Engineering, Chinese Academy of Sciences · Sangfor Technologies Inc. · CAIR, HKISI, Chinese Academy of Sciences · SCSE, FIE, M.U.S.T
In multimodal multi-turn dialogue, dialogue history plays a dual role: reliable responses provide useful context for subsequent turns, whereas hallucinated responses can propagate errors across turns. Existing training-free methods globally adjust the effect of dialogue history on generation, making it difficult to suppress misleading information without weakening useful context. To address this trade-off, we introduce Support-Guided Triadic Decoding (SGTD), a training-free framework that selectively controls the influence of unreliable history and fuses complementary predictions during decoding. Specifically, Historical Answer Support Estimation measures the visual support of historical responses against type-matched alternatives and identifies low-support content that may mislead later generation. Support-Targeted Attention Intervention then weakens the influence of this content while preserving the surrounding dialogue, producing a support-refined prediction. Triadic Decoding combines this prediction with the full-history prediction and answer-free visual evidence, while the Support-Adaptive Plausibility Constraint filters implausible tokens using visual agreement and historical support. Experiments on MMHalSnowball and VisDiaHalBench across representative LVLMs show that SGTD consistently achieves the highest accuracy under hallucinated histories while preserving the benefits of factual context.
Figures & tables
Figure 1: Comparison of decoding strategies under unreliable dialogue history. (a) Standard decoding can follow an unsupported prior answer. (b) RVD applies a distribution-level correction without localizing unreliable history, potentially weakening reliable context. (c) SGTD uses answer support to refine low-support history, then jointly decodes from the Full-History, Support-Refined History, and Answer-Free Visual Branches.
Figure 2: Overview of Support-Guided Triadic Decoding (SGTD). Frozen BLIP compares checkable historical answers with type-matched alternatives to identify low-support turns. The Full-History, Support-Refined History, and Answer-Free Visual Branches preserve context, reduce attention to selected answer spans, and provide visual evidence independent of previous answers. Triadic Decoding combines their logits under the Support-Adaptive Plausibility Constraint.
Figure 3: Support-Targeted Attention Intervention. Declarative claims from low-support turns locate the corresponding answer tokens. SGTD subtracts a penalty from the pre-softmax attention scores directed to these tokens by subsequent queries.
Method
LLaVA-1.5-7B
mPLUG-Owl2-7B
ShareGPT4V-7B
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Regular
71.00
88.74
14.72
58.15
56.28
60.67
87.69
7.76
52.04
52.91
73.56
90.63
15.60
59.93
57.95
VCD [ 17 ]
70.02
88.84
17.60
58.82
52.42
61.51
88.62
8.57
52.90
52.95
73.56
90.63
19.18
61.12
54.37
M3ID [ 9 ]
72.37
90.27
15.02
59.22
57.35
62.54
88.88
7.22
52.88
55.32
74.40
91.76
15.79
60.65
58.62
PAI [ 25 ]
72.83
89.66
15.95
59.48
56.89
62.50
88.76
7.76
53.01
54.74
74.72
91.59
16.91
61.08
57.81
SID [ 11 ]
72.91
89.42
20.61
60.98
52.30
60.18
87.89
8.91
52.33
51.28
75.25
90.35
21.34
62.31
53.91
Table 1: Results on MMHalSnowball (%). Avg. is the mean over Clean, Fact., and Hallu., while H.D. is Clean minus Hallu. Under the Clean setting, RVD reduces to Regular decoding. Best and second-best results are shown in bold and underlined.
Method
LLaVA-1.5-7B
mPLUG-Owl2-7B
ShareGPT4V-7B
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
Regular
77.16
86.38
30.72
64.75
46.44
70.90
86.10
31.14
62.71
39.76
79.04
84.82
23.18
62.35
55.86
VCD [ 17 ]
77.60
86.80
32.04
65.48
45.56
69.14
86.64
34.08
63.29
35.06
79.86
86.02
24.50
63.46
55.36
M3ID [ 9 ]
79.10
87.32
30.74
65.72
48.36
72.42
87.18
31.20
63.60
41.22
81.10
86.12
23.48
63.57
57.62
PAI [ 25 ]
79.92
87.24
31.34
66.17
48.58
72.44
87.08
31.90
63.81
40.54
80.68
85.78
24.02
63.49
56.66
SID [ 11 ]
78.72
86.86
33.84
66.47
44.88
73.90
86.40
30.82
63.71
43.08
80.16
85.74
25.68
63.86
54.48
Table 2: Results on VisDiaHalBench (%). Avg. is the mean over Clean, Fact., and Hallu., while H.D. is Clean minus Hallu.
Setting
MMHalSnowball
VisDiaHalBench
RVS
LSS
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
77.30
91.88
24.33
64.50
83.06
90.92
61.36
78.45
✓
77.30
79.25
51.23
69.26
83.06
89.66
82.77
85.16
✓
✓
77.30
91.17
48.34
72.27
83.06
91.28
83.04
85.79
Table 3: Ablation of Historical Answer Support Estimation. Results are reported for LLaVA-1.5-7B (%). RVS and LSS denote Relative Visual Support and Low-Support Selection.
Branch
MMHalSnowball
VisDiaHalBench
Full
Ref.
Vis.
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
✓
✓
77.30
91.19
45.59
71.36
83.06
91.68
81.60
85.45
✓
✓
77.30
91.74
24.39
64.47
83.06
90.86
61.38
78.43
✓
✓
71.00
88.08
39.05
66.04
77.16
85.60
72.66
78.47
✓
✓
✓
77.30
91.17
48.34
72.27
83.06
91.28
83.04
85.79
Table 4: Ablation of Decoding Branches. Results are reported for LLaVA-1.5-7B (%). Full, Ref., and Vis. denote the Full-History, Support-Refined History, and Answer-Free Visual Branches.
Figure 4: Sensitivity to intervention strength. Results are reported for (a) MMHalSnowball and (b) VisDiaHalBench with LLaVA-1.5-7B. Curves report Fact., Hallu., and (Fact.+Hallu.)/2 . The shaded marker denotes the selected λ=4 across both benchmarks. Setting λ=0 disables the attention penalty.
Constraint
MMHalSnowball
VisDiaHalBench
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
None
75.21
88.78
49.24
71.08
81.96
90.80
82.36
85.04
Fixed 0.1
75.69
89.93
47.66
71.09
82.24
91.34
81.54
85.04
Fixed 0.2
75.93
91.25
44.46
70.55
81.96
91.38
79.26
84.20
Fixed 0.3
76.59
91.55
42.53
70.23
82.94
91.34
78.04
84.11
Fixed 0.4
77.26
92.56
40.50
70.11
83.30
90.96
76.82
83.69
Table 5: Ablation of the Support-Adaptive Plausibility Constraint. Results are reported for LLaVA-1.5-7B (%).
Figure 5: Answer deletion versus attention intervention. Accuracy on MMHalSnowball and VisDiaHalBench with LLaVA-1.5-7B (%). Delete Answer removes the complete answer from each low-support turn, while SGTD keeps it in the history and applies Support-Targeted Attention Intervention to the selected span.
Backbone
Method
Clean ↑
Fact. ↑
Hallu. ↑
Avg. ↑
H.D. ↓
InternVL3.5-8B [ 35 ]
Regular
75.58
87.66
36.38
66.54
39.20
RVD
75.58
80.50
72.44
76.17
3.14
SGTD (Ours)
78.84
90.56
82.78
84.06
−3.94
Qwen2.5-VL-7B [ 3 ]
Regular
73.52
79.20
65.20
72.64
8.32
RVD
73.52
75.28
71.18
73.33
2.34
SGTD (Ours)
75.20
79.18
78.00
77.46
−2.80
Table 6: Generalization to Modern LVLM Backbones. Accuracy on VisDiaHalBench (%).
Figure 6: Qualitative comparison under hallucinated histories. The top and bottom rows show examples from MMHalSnowball and VisDiaHalBench, respectively, using LLaVA-1.5-7B. Regular decoding and RVD follow unsupported history, whereas SGTD recovers the image-supported answers. Red and green denote incorrect and correct predictions.