One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination

Zhan Fa; Yue Duan; Jian Zhang; Lei Qi; Yinghuan Shi

cs.CVMar 11, 2026

One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination

Authors: Zhan Fa, Yue Duan, Jian Zhang, Lei Qi, Yinghuan Shi

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Computer Science and Engineering, Southeast University, China

Abstract

Current training-free methods tackle MLLM hallucination with separate strategies: either enhancing visual signals or suppressing text inertia. However, these separate methods are insufficient due to critical trade-offs: simply enhancing vision often fails against strong language prior, while suppressing language can introduce extra image-irrelevant noise. Moreover, we find their naive combination is also ineffective, necessitating a unified framework. We propose such a framework by focusing on the core asset: the vision token. Our design leverages two key insights: (1) augmented images offer complementary visual semantics, and (2) removing vision tokens (information-gap) isolates hallucination tendencies more precisely than distorting images (modality-gap). Based on these, our framework uses vision tokens in two distinct ways, both operating on latent representations: our Synergistic Visual Calibration (SVC) module incorporates augmented tokens to strengthen visual representations, while our Causal Representation Calibration (CRC) module uses pruned tokens to create latent-space negative samples for correcting internal model biases. By harmonizing these two roles, our framework effectively restores the vision-language balance, significantly reducing object hallucinations, improving POPE accuracy by an average of 2% absolute on LLaVA-1.5 across multiple benchmarks with only a 1.06x inference latency overhead.

Figures & tables

Figure 1: Disjoint Paradigms vs. Our Unified Latent Calibration. (a) Prior work includes Visual Attention Enhancement [ 27 ] and (b) Textual Decoding Refinement at final logits [ 20 ] . Naively combining these disjoint paradigms (a)+(b) degrades performance, highlighting conflicting signals. (c) Our Unified Latent Calibration is a unified system operating entirely at the representation level, using the vision token as a single source for both SVC and CRC, achieving superior and synergistic results.

Figure 2: Our Three Core Findings. (F1) Diagnosing the Imbalance: Inverse Correlation. Visual attention decays sharply as generation proceeds, while hallucination frequency surges where visual grounding is weakest. (F2) Enabling Enhancement: Semantic Complementarity. Original and augmented image attentions show complementary focus (e.g., on ’Camera’); their synergy enables enhanced visual grounding. (F3) Enabling Calibration: Superiority of Information-Gap. Latent-space token removal ( information-gap ) generates stable, grounded hallucinations, proving more suitable for bias probing than unstable, noisy pixel-level masking ( modality-gap ).

Figure 3: Overview of our unified framework. The model processes an original input stream (orange path) and a parallel hallucination-probe stream (purple path) derived from pruned vision tokens. Our Synergistic Visual Calibration (SVC) module injects complementary visual context from augmented images into a critical middle layer ( $L_{c}$ ) to counteract visual fading. Simultaneously, the Causal Representation Calibration (CRC) module uses the differential representations between the two streams to purify hidden states in shallow layers ( $1\dots L_{c}$ ), suppressing linguistic priors.

Figure 4: Illustration of the Causal Representation Calibration (CRC) mechanism. By subtracting the hallucinated representation ( $\mathbf{H}_{\text{neg}}$ ) from the original ( $\mathbf{H}_{\text{org}}$ ), we obtain a differential vector ( $\Delta\mathbf{H}$ ). Averaging these vectors across multiple negative samples yields a stable hallucination direction ( $\mathbf{v}_{\text{crc}}$ ). The final calibration modifies the original representation away from this direction to produce a purified output ( $\mathbf{H}_{\text{pos}}$ ).

Figure 5: The simplified Structural Causal Model (SCM) for MLLM hallucination. We posit that hallucination arises from a spurious causal path from the model’s intrinsic bias ( $B$ ) to its latent representation ( $H^{(l)}$ ), which confounds the true visual path from the image ( $V$ ).

Benchmark	Method	LLaVA-1.5		MiniGPT-4		Shikra		InstructBLIP
Benchmark	Method	Avg. Accuracy $\uparrow$	Avg. F1 $\uparrow$	Avg. Accuracy $\uparrow$	Avg. F1 $\uparrow$	Avg. Accuracy $\uparrow$	Avg. F1 $\uparrow$	Avg. Accuracy $\uparrow$	Avg. F1 $\uparrow$
MSCOCO	Vanilla	84.79	85.61	76.76	76.82	81.32	82.01	84.36	84.64
	VCD [CVPR’24]	84.80	85.65	76.01	76.28	81.34	82.25	84.81	85.28
	PAI [ECCV’24]	85.85	86.08	75.64	77.57	81.30	80.81	83.98	84.02
	VISTA [ICML’25]	86.15	86.29	76.06	76.80	82.44	82.47	84.87	84.95
	ONLY [ICCV’25]	86.03	86.22	76.98	77.62	82.75	82.85	84.90	85.01

Table 1: Evaluation results on POPE benchmark across four MLLMs. Results show averaged accuracy and F1 scores in % computed across random , popular , and adversarial object splits. Best and second best results are bolded and underlined, respectively.

Max Tokens	Method	LLaVA-1.5		MiniGPT-4		Shikra		InstructBLIP
Max Tokens	Method	$\text{CHAIR}_{\text{Sen}}\downarrow$	$\text{CHAIR}_{\text{Ins}}\downarrow$	$\text{CHAIR}_{\text{Sen}}\downarrow$	$\text{CHAIR}_{\text{Ins}}\downarrow$	$\text{CHAIR}_{\text{Sen}}\downarrow$	$\text{CHAIR}_{\text{Ins}}\downarrow$	$\text{CHAIR}_{\text{Sen}}\downarrow$	$\text{CHAIR}_{\text{Ins}}\downarrow$
64 tokens	Vanilla	25.4	8.9	25.0	9.4	25.4	8.8	26.8	9.2
	VCD [CVPR’24]	24.1	7.9	25.4	9.4	23.2	8.1	26.8	9.3
	VISTA [ICML’25]	21.4	8.1	22.2	8.2	19.5	6.7	24.4	8.8
	ONLY [ICCV’25]	19.2	7.5	24.5	9.3	21.4	7.5	25.7	9.0
	Ours	18.1	8.4	23.8	8.3	16.7	5.9	24.2	7.9

Table 2: CHAIR hallucination evaluation results. We compare our method to state-of-the-art training-free methods. Maximum new token is set to 64 and 128. Best and second best results are bolded and underlined, respectively.

Figure 6: MMHal-Bench Evaluation across four MLLMs. Our method (Ours, orange line) consistently achieves superior performance across all eight categories, demonstrated by the larger area covered in the radar plots compared to Vanilla, PAI, and VISTA.

Method	LLaVA-1.5		InstructBlip
Method	$\text{Perception}_{\text{sum}}$	$\text{Cognition}_{\text{sum}}$	$\text{Perception}_{\text{sum}}$	$\text{Cognition}_{\text{sum}}$
Vanilla	1444.23	323.14	1181.15	233.71
VCD	1445.13	324.25	1182.34	234.95
VISTA	1450.54	329.08	1189.23	236.25
Ours	1456.28	332.86	1192.76	240.83

Table 3: Evaluation on MME benchmark (Perception and Cognition scores). Our method (Ours) consistently achieves state-of-the-art performance across all tested models, demonstrating robust general capabilities.

Method	Latency (ms/token)	Throughput (token/ms)	Memory Cost (MB)
Greedy	30.3 (×1.00)	0.033 (×1.00)	14257
ICD	33.03 (×1.09)	0.030 (×0.91)	14263
VCD	72.72 (×2.4)	0.014 (×0.42)	14984
VISTA	33.35 (×1.1)	0.030 (×0.91)	15024
Ours	32.1 (×1.06)	0.031 (×0.94)	14924

Table 4: Comparison of overhead among different methods (with best results bolded). Latency is measured in milliseconds per token. Throughput is calculated as tokens per millisecond. Memory cost is the peak GPU memory usage in megabytes.

Method Configuration	COCO		AOKVQA
Method Configuration	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Vanilla LLaVA-1.5	84.79	85.61	77.23	80.62
+ SVC ( $V_{\text{ori}}$ only)	85.04	85.68	79.03	80.96
+ SVC ( $V_{\text{syn}}$ , Ours)	85.55	86.04	79.43	81.73
+ CRC (Masked Image neg.)	84.77	85.72	79.15	81.34
+ CRC (Pruned Token neg., Ours)	86.11	86.39	81.65	81.98

Table 5: Ablation study on LLaVA-1.5 (POPE benchmark in %) evaluating our SVC variants (visual context) and CRC variants (negative sampling strategy). Best results are bolded .

Figure 7: Visualizing SVC’s effect with TAM [ 21 ] . For the token ’bulldog’, Vanilla LLaVA shows diffuse attention. $V_{\text{ori}}$ and $V_{\text{aug}}$ alone show different, complementary focus. ( $+SVC(\text{Ours})$ ) integrates these cues, creating a accurate focus on the bulldog.

Figure 8: t-SNE visualization of hidden states. Our pruned-token negative samples (Our_Neg1/2/3) consistently cluster near the original (Ori_IMG) representations, suggesting an in-distribution probe. In contrast, Masked_IMG representations are divergent, indicating a noisy, out-of-distribution perturbation.

Figure 9: Hyperparameter Analysis. (a) POPE Accuracy vs. Latency for different $K$ in CRC. We choose $K=3$ (red star) as the best balance of performance and speed. (b) Heatmap for SVC ( $\lambda_{s}$ ) and CRC ( $\lambda_{c}$ ) strength. Performance is robust, with an optimal region around $\lambda_{s}=0.06$ and $\lambda_{c}=0.10$ .

Figure 10: Effect of the number of retained vision tokens ( $N_{h}$ ) on POPE Accuracy (LLaVA-1.5, COCO). The red star indicates our chosen value $N_{h}=5$ . Accuracy drops sharply only when fewer than 20 tokens are retained.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
Jul 5, 2026Kai Tang, Jinhao You, Bohua Zhang +6Large Vision-Language Models Object Hallucination in VLMs
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1

Jul 5, 2026J/K move · Enter open · S save

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Kai Tang, Jinhao You, Bohua Zhang, Yichen Guo+5

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania+4

Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Abstract

VLM Hallucination Mitigation

Explore similar work

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration