Entropogram-guided Visual Realignment Decoding: Capturing the Rhythm of Multimodal Reasoning

Jiaqi Deng; Zhe Fu; Yakun Chen; Dichang Zhang; Zonghan Wu; Xiaoshui Huang; Huan Huo; Guandong Xu

cs.CVNov 4, 2026

Entropogram-guided Visual Realignment Decoding: Capturing the Rhythm of Multimodal Reasoning

Authors: Jiaqi Deng, Zhe Fu, Yakun Chen, Dichang Zhang, Zonghan Wu, Xiaoshui Huang, Huan Huo, Guandong Xu

Organizations: University of Technology Sydney · Alibaba Cloud Computing · Stony Brook University · East China Normal University ·

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable progress on cross-modal reasoning tasks, yet they continue to suffer from hallucinations in long-chain reasoning processes. In this work, we revisit hallucination in MLLMs from the perspective of reasoning generation dynamics. We argue that entropy is not merely a static measure of uncertainty but a structured signal that reflects how modality information is integrated throughout decoding. Through systematic analysis, we identify two characteristic entropy patterns associated with hallucination behaviors: (1) a generally high-confidence regime corresponds to premature convergence to inappropriate multimodal interactions; and (2) local abrupt entropy transitions are associated with reasoning instability and hallucination onset. Building on these observations, we propose an entropy-aware self-introspective framework for MLLM reasoning. Our method continuously tracks entropy dynamics during decoding and performs targeted visual re-alignment when suspicious hallucination states are detected, encouraging the model to re-examine salient visual evidence at critical reasoning stages. Extensive experiments across six multimodal reasoning benchmarks spanning nine distinct evaluation settings demonstrate that our approach delivers robust improvements in reasoning accuracy and hallucination robustness, while providing new insights into the generation dynamics underlying MLLM failures. The code is available at: https://github.com/jden4524/SelfRetrospectiveDecoding.

Figures & tables

Figure 1: Comparison of generation dynamics between correct and incorrect multimodal reasoning trajectories. (a) Average reasoning entropy across the generation process over samples, where correct predictions consistently maintain higher entropy than incorrect ones. (b) Evolution of visual attention during CoT generation, showing that correct reasoning trajectories preserve stronger visual grounding at the early stage of reasoning, while incorrect trajectories exhibit progressively weakened but still higher attention to visual evidence. Shaded regions denote standard deviations.

Figure 2: A case study illustrating hallucination emergence during MLLM reasoning. The left panel shows entropy over generated tokens, while the right panel visualizes the corresponding visual attention patterns at selected reasoning steps (P5–P8). The gray region denotes the span of incorrect generation. Hallucinated reasoning coincides with sharp entropy spikes (P8) and progressively strong but diffused visual attention. More examples can be found in Appendix A .

	$\Delta H$ @onset	Low-entropy duration	Spike frequency	Max $\Delta H$
Correct ( $n{=}465$ )	0.012	17.6	0.085	1.762
Hallucinated ( $n{=}763$ )	0.245	21.4	0.101	1.974
$p$ -value	$4\times 10^{-4}$	$<10^{-35}$	$<10^{-9}$	$<10^{-16}$

Table 1: Population-level entropy statistics on 1,228 VQA trajectories (Qwen3-VL-8B).

\Delta H

@onset is the mean entropy change at the aligned hallucination onset (

p{=}0

); the remaining columns are per-trajectory statistics.

p

-values are from two-sided Mann–Whitney U tests.

Figure 3: Framework overview of the proposed SEED . The figure presents three parts: (a) A standard MLLM architecture processes system instructions and visual tokens through stacked decoder layers to perform multimodal reasoning. (b) A self-introspective mechanism monitors token-level entropy dynamics throughout the decoding process, capturing locally important spikes that signal shifts in reasoning dynamics. (c) When both detection conditions are met, our method performs an attention intervention during decoding to rectify the multimodal interaction. We refer to this process as Visual Attention Re-alignment Decoding.

Method	Multimodal Reasoning & Understanding					Summary $\Delta$ (%) / improved
Method	VStar $\uparrow$	MMVP $\uparrow$	VisOnly s $\uparrow$	VisOnly r $\uparrow$	M 3 CoT $\uparrow$	Summary $\Delta$ (%) / improved
Qwen3-VL-8B	73.82	76.23	42.20	54.60	78.21	–
+ VCD	54.45( -19.4 )	80.00( +3.8 )	42.57( +0.4 )	55.71( +1.1 )	77.68( -0.5 )	$-2.93$ / 3/5
+ MemVR	74.87( +1.0 )	80.67( +4.4 )	41.86( -0.3 )	54.44( -0.2 )	77.68( -0.5 )	$+0.89$ / 2/5
+ SEED (Ours)	83.25( +9.4 )	76.33( +0.1 )	47.00( +4.8 )	54.37( -0.2 )	79.97( +1.8 )	+3.17 / 4/5
Qwen3-VL-4B	68.06	76.33	42.14	52.22	76.79	–

Table 2: Comparisons across general reasoning benchmarks. The summary column reports the macro-average change in percentage points and the number of strictly positive changes over available settings. Base MLLMs use the default CoT setting.

Method	Multimodal Hallucination				Summary $\Delta$ (%) / improved
Method	HalluBench $\uparrow$	POPE-R $\uparrow$	POPE-P $\uparrow$	POPE-A $\uparrow$	Summary $\Delta$ (%) / improved
Qwen3-VL-8B	58.50	90.25	87.53	86.42	–
+ VCD	56.24( -2.3 )	91.67( +1.4 )	87.03( -0.5 )	86.30( -0.1 )	$-0.37$ / 1/4
+ MemVR	55.87( -2.6 )	90.63( +0.4 )	88.12( +0.6 )	86.85( +0.4 )	$-0.31$ / 3/4
+ SEED (Ours)	59.28( +0.8 )	97.39( +7.1 )	90.94( +3.4 )	85.07( -1.3 )	+2.49 / 3/4
Qwen3-VL-4B	54.80	90.92	88.64	87.96	–

Table 3: Comparisons across hallucination benchmarks. The summary column reports the macro-average change in percentage points and the number of positive changes over available settings. Base MLLMs use the default CoT setting.

Attention Intervention Strategy	Qwen3-VL-8B		Qwen3-VL-4B
	Accuracy $\uparrow$	Efficiency $\uparrow$	Accuracy $\uparrow$	Efficiency $\uparrow$
None (baseline)	78.21	0.20	76.79	0.22
+ Equal attention	80.24(+2.0)	0.05( $\times$ 0.25)	78.02(+1.2)	0.07( $\times$ 0.32)
+ Truncated SVD Project	54.28(-23.9)	0.17( $\times$ 0.85)	50.23(-26.6)	0.18( $\times$ 0.82)
+ ViT attention intervene	60.32(-17.9)	0.15( $\times$ 0.75)	55.36(-21.4)	0.16( $\times$ 0.73)
+ SEED (Ours)	79.97 (+1.8)	0.19( $\times$ 0.95)	78.03 (+1.2)	0.21( $\times$ 0.95)

Table 4: Comparison of attention intervention strategies across backbones on M 3 CoT ( Chen et al., 2024 ) . Each backbone is evaluated in terms of accuracy and efficiency (instance/s). All hyperparameters are kept consistent. The best and second-best results are highlighted in green and blue, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: More examples illustrating hallucination emergence during MLLM reasoning. The left panel shows entropy over generated tokens, while the right panel visualizes the corresponding visual attention patterns at selected reasoning steps. The gray region denotes the span of incorrect generation. Hallucinated reasoning coincides with regions containing sharp entropy spikes and increasingly strong yet diffuse visual attention.

Figure 5: Statistical analysis of wrong reasoning span sizes. The reasoning processes of 270 VQA samples are recorded. Results show that wrong reasoning claims concentrate in the first 20% of the generation process.

$p$	$-20{\sim}-11$	$-5$	$-4$	$-3$	$-2$	$-1$	$\mathbf{0}$	$+1$	$+2{\sim}+5$	$+6{\sim}+20$
Hallucinated	0.0020	0.0560	0.0011	0.0188	$-$ 0.0423	0.0065	0.2447	$-$ 0.1728	$-$ 0.0227 $\sim$ 0.0081	$-$ 0.0025
Correct	0.0007	0.0119	0.0140	$-$ 0.0010	$-$ 0.0109	0.0047	0.0117	$-$ 0.0125	$-$ 0.0404 $\sim$ 0.0265	0.0007

Appendix

Table 5: Mean token-to-token entropy change

\Delta H

aligned to the hallucination onset (

p{=}0

). For ranges, the mean over the range (or its min–max) is reported.

Method	Latency $\downarrow$ (ms/token)	Throughput $\uparrow$ (instance/s)	Time Cost $\downarrow$ (min)
Greedy	3.30 ( $\times$ 1.00)	0.941 ( $\times$ 1.00)	20.0 ( $\times$ 1.00)
OPERA	14.28 ( $\times$ 4.33)	0.224 ( $\times$ 0.24)	85.0 ( $\times$ 4.25)
VCD	5.08 ( $\times$ 1.54)	0.598 ( $\times$ 0.63)	31.5 ( $\times$ 1.58)
MemVR	8.47 ( $\times$ 2.57)	0.359 ( $\times$ 0.38)	52.5 ( $\times$ 2.63)
SEED	3.45 ( $\times$ 1.05)	0.897 ( $\times$ 0.95)	21.0 ( $\times$ 1.05)

Appendix

Table 6: Performance comparison of SOTA methods and SEED in latency, throughput, and time cost. All methods are built on the Qwen3-VL-8B-Instruct backbone. The experiment is conducted on

\text{M}^{3}\text{CoT}

and all models are prompted to generate CoT. Note that all results are evaluated under an inference acceleration framework.

Transition	Interpretation	Count	Rate	Trigger ratio	Avg. triggers	Avg. span
Correct $\rightarrow$ Wrong	Harmful intervention	454	15.13%	46.26%	2.04	7.95
Wrong $\rightarrow$ Correct	Successful correction	408	13.60%	43.38%	2.20	8.69
Wrong $\rightarrow$ Wrong	Unresolved error	729	24.30%	60.22%	2.93	11.84

Appendix

Table 7: Paired transition analysis between the base model and SEED on POPE-Adversarial with Llama-3.2-11B-Vision.

Figure 6: Hyperparameter analysis of injection ratios (left) and entropy threshold (right).

$\alpha$	0.40	0.45	0.50	0.70
	78.30 (+0.09)	78.79 (+0.58)	79.89 (+1.68)	78.21 (+0.00)
$\tau_{\text{rise}}$	0.225	0.250	0.275	0.300
	78.42 (+0.21)	79.89 (+1.68)	79.20 (+0.99)	78.53 (+0.32)
$t_{1}$	40	50	60	70
	79.12 (+0.91)	79.48 (+1.27)	79.89 (+1.68)	78.91 (+0.70)

Appendix

Table 8: Hyperparameter sensitivity on M 3 CoT with Qwen3-VL-8B (base model: 78.21). Gains over the base model are shown in parentheses.

Figure 7: Additional hyperparameter analysis of the entropy threshold, injection ratio, and the start and end steps of the monitoring window.

Figure 8: A case study illustrating that SEED not only improves final answer accuracy but also produces more reliable intermediate reasoning steps. Hallucinated content is highlighted in purple, while correct reasoning is highlighted in yellow.

Figure 9: Full analysis on one example to demonstrate the effectiveness of SEED. The hallucination related text is highlighted in yellow and the statement related to correct reasoning is highlighted in blue.

Entropy-Based Decoding

See all 8 papers

Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding
Mar 9, 2026Zhongxing Xu, Zhonghua Wang, Zhe Qian +10Entropy-Based Decoding VLM Hallucination Mitigation
Monash University · Georgia Tech · Northeastern University +3
Seeing Through the Chain: Understanding and Mitigating Hallucinations in Multimodal Large Reasoning Models
Feb 3, 2026Hao Fang, Jinyu Li, Jiawei Kong +4Hallucination Mitigation Multimodal Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1

Mar 9, 2026J/K move · Enter open · S save

Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding

Zhongxing Xu, Zhonghua Wang, Zhe Qian, Dachuan Shi+9

Monash University · Georgia Tech · Northeastern University+3

Entropy-Based Decoding VLM Hallucination Mitigation Multimodal Large Language Models Continuous Latent Reasoning

Open paper AskPDF arXiv

Recent advancements in multimodal large reasoning models (MLRMs) have significantly improved performance in visual question answering. However, we observe that transition words (e.g., because, however, and wait) are closely associated with hallucinations and tend to exhibit high-entropy states. We argue that adequate contextual reasoning information can be directly extracted from the token probability distribution. Inspired by superposed representation theory, we propose leveraging latent superposed reasoning to integrate multiple candidate semantics and maintain latent reasoning trajectories. The hypothesis is that reliance on discrete textual inputs may drive the model toward sequential explicit reasoning, underutilizing dense contextual cues during high-entropy reasoning stages. Therefore, we propose constructing rich semantic representations from the token probability distributions to enhance in-context reasoning. With this goal, we present Latent Entropy-Aware Decoding (LEAD), an efficient plug-and-play decoding strategy that leverages semantic context to achieve reliable reasoning. The heart of our method lies in entropy-aware reasoning mode switching. The model employs probability-weighted continuous embeddings under high-entropy states and transitions back to discrete token embeddings as entropy decreases. Moreover, we propose a prior-guided visual anchor injection strategy that encourages the model to focus on visual information. Extensive experiments show that LEAD effectively mitigates hallucinations across various MLRMs on multiple benchmarks.

Abstract

Entropy-Based Decoding

Explore similar work

Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding

Seeing Through the Chain: Understanding and Mitigating Hallucinations in Multimodal Large Reasoning Models

Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification