Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.
Figures & tables
Figure 1: ClipTTT improves robustness and reduces hallucinations under corruptions. Left: LVLMs suffer from hallucinations. Such an issue becomes worse with corruption, indicating that degradation leads to unreliable generation. Right: We benchmark the hallucinations on CHAIR [ 42 ] across 15 corruptions adopted in [ 22 ] . Naïvely applying CLIP for test-time response selection may even degrade the performance. In contrast, by performing ClipTTT at test time, we consistently reduce the hallucinations of the vanilla models ( CHAIRS↓ and CHAIRI↓ ).
Figure 2: Left: Hallucinations increase with corruption severities, indicating that stronger degradations reduce generation reliability (example corruption: Zoom Blur). Right: Captions with higher image-text alignment (CLIP Score) tend to be more factually correct. Results are averaged over 15 corruption types. Both experiments use LLaVA-v1.5-7B [ 34 ] .
Figure 3: Overview of our ClipTTT framework. For each single corrupted test input, we employ a student-teacher framework for on-the-fly adaptation. (1) The Teacher model generates n diverse caption candidates via sampling. (2) An external CLIP model scores each candidate, and the one with the highest visual-semantic alignment is selected as the pseudo-label. (3) The Student model is trained for one step on this pseudo-label, with gradients updating only its parameter-efficient LoRA weights. (4) The Teacher’s LoRA weights are then updated via an exponential moving average (EMA) of the Student’s, ensuring a stable yet progressively improving training target.
Method
brigh
cont
defoc
elast
fog
frost
gauss
glass
impul
jpeg
motn
pixel
shot
snow
zoom
AVG
Greedy Decoding
56.8
51.8
57.8
54.2
53.8
52.6
55.4
55.0
55.4
55.8
58.0
58.2
55.4
57.0
56.4
55.6
VCD [ 30 ]
55.6
50.0
56.6
51.4
54.2
53.6
54.0
54.8
54.6
57.6
58.8
55.8
54.0
55.2
53.2
54.6
PAI [ 35 ]
58.6
48.6
56.6
54.6
51.6
54.0
56.4
55.6
56.8
57.2
62.2
58.2
54.0
55.2
57.0
55.8
VAP [ 57 ]
54.0
46.8
56.7
53.9
50.7
53.1
52.0
50.9
52.3
55.2
55.9
57.5
52.6
49.7
50.2
52.8
CGD [ 15 ]
52.4
51.4
59.1
56.0
53.8
52.4
54.0
56.0
55.4
57.2
56.1
59.8
52.2
55.6
51.6
54.9
Restormer [ 55 ]
56.2
47.4
57.6
53.6
53.8
52.8
57.2
54.0
59.4
56.8
58.6
57.0
57.6
54.2
55.2
55.4
Table 1: Comparison of sentence-level hallucination ( CHAIRS↓ ) . Lower scores indicate better performance. The results are evaluated across 15 corruption types from the CHAIR benchmark. The best result in each column is in bold . Column headers are abbreviated.
Method
brigh
cont
defoc
elast
fog
frost
gauss
glass
impul
jpeg
motn
pixel
shot
snow
zoom
AVG
Greedy Decoding
13.7
14.8
16.4
14.6
13.5
13.9
16.4
16.5
15.8
13.6
15.2
14.5
14.8
16.2
21.2
15.4
VCD [ 30 ]
13.9
15.2
17.0
14.4
13.6
15.5
16.2
15.3
15.5
14.5
15.3
14.2
14.8
16.2
20.3
15.5
PAI [ 35 ]
14.3
13.8
15.5
14.8
12.9
15.1
16.1
16.7
15.7
14.1
16.1
14.6
14.5
15.6
21.0
15.4
VAP [ 57 ]
13.7
14.3
14.4
15.3
14.0
15.5
15.4
15.5
16.3
14.7
15.7
15.2
15.4
14.7
19.7
15.3
CGD [ 15 ]
14.1
14.6
15.5
15.2
13.9
15.1
15.9
14.3
15.5
13.7
15.6
15.4
16.0
15.6
19.6
15.3
Restormer [ 55 ]
14.7
16.7
17.2
15.5
14.1
15.1
24.4
16.9
26.6
14.3
15.8
14.4
22.0
15.0
22.0
17.7
Table 2: Comparison of instance-level hallucination ( CHAIRI↓ ) . Lower scores indicate better performance. The best result in each column is in bold .
Table 3: Effect of model scales and architectures. Left: Scalability analysis across model sizes (1.1B to 13B). Right: Evaluation on diverse architectures. ClipTTT consistently reduces hallucinations across all setups.
Table 4: Left: NoCaps (Near-Domain) results. CHAIR score for NoCaps data for objects present in COCO. Right: Nocaps (Out-of-Domain) results. CHAIR score for NoCaps data for objects beyond in COCO. ClipTTT demonstrates strong generalization by lowering hallucination rates on unseen categories.
Method CHAIRS↓CHAIRI↓ Final Iteration 46.2 18.4 Best Loss Iteration 48.2 19.2 Ours (Max CLIP Score) 45.6 17.9
Table 5: Left: Ablation on checkpoint strategy. CHAIR score based on which checkpoint is used for inference. Right: Ablation on teacher update strategy. CHAIR score based on how the teacher model is updated.
Table 6: Left: Ablation on LoRA module placement. CHAIR score for different placements of LoRA training. Right: Ablation on pseudo-label quality. CHAIR score compared between the baseline, the CLIP ranked pseudo-label before TTT starts, and ClipTTT. Our CLIP-based selection provides a better starting point than the baseline, which is further improved by TTT.
Figure 4: Left: CLIP Score distribution comparison. Kernel density estimates of CLIP Scores for captions of Clean images (upper bound), corrupted images with baseline Greedy decoding, and corrupted images after applying Our Method (ClipTTT). Our method shifts the degraded distribution significantly towards the clean distribution. Right: Improvement trend by baseline score. The average improvement provided by ClipTTT is higher when the initial baseline caption has a low-to-medium quality score. The method adaptively reduces its intervention as the baseline quality improves.
Figure 5: Left: Ablation on corruption severity. Sev. 1 and Sev. 3 CHAIR scores for Greedy Decoding, training-free test-time methods (VAP [ 57 ] , VCD [ 30 ] ), and ClipTTT. Right: Inference time vs. performance tradeoff. Tradeoff curve of ClipTTT at different iterations and against different test-time approaches.
Table 7: Left: Average CLIP Score improvement. CLIP Score comparison between Greedy, pseudo-label, responses before TTT, and ClipTTT. Right: GPT-4o assisted evaluation. Accuracy reflects the absence of hallucinations, while Detailedness reflects the richness of the description. ClipTTT improves accuracy and maintains detailedness across methods.
Figure 6: Qualitative Comparison. Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Corruptions (severity 5): frost, impulse noise, gaussian noise and defocus blur. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .
Method
TinyLLaVA-1.1B [ 59 ]
InstructBLIP-7B [ 13 ]
Qwen2-VL-2B [ 4 ]
CHAIRS↓
CHAIRI↓
CHAIRS↓
CHAIRI↓
CHAIRS↓
CHAIRI↓
Greedy Decoding
63.0
21.1
55.7
15.9
47.6
15.0
VCD [ 30 ]
62.9
20.7
55.3
15.5
47.7
15.3
VAP [ 57 ]
62.9
21.6
55.0
15.7
47.8
15.2
ClipTTT (Ours)
62.4
18.7
54.6
15.3
47.2
14.2
Table 8: Generalization Across Models. We report average hallucination across 15 corruptions showing the sentence-level ( CHAIRS↓ ) and instance-level ( CHAIRI↓ ) hallucination for three LVLMs. Lower is better.
Method
F-Score ↑
F-Score S ↑
Greedy (Base)
76.3
60.9
VCD [ 30 ]
80.0
63.2
VAP [ 57 ]
81.7
64.1
ClipTTT (Ours)
83.1
64.3
Table 9: Comparison with FAITHSCORE [ 26 ] . We evaluate ClipTTT against greedy decoding and strong training-free test-time baselines. FAITHSCORE is a reference-free measure of visual faithfulness computed via GPT-4o-based verification [ 24 ] . F-Score (fact-level) measures overall atomic-fact faithfulness of the response. F-Score S (sentence-level) is the fraction of sentences that contain no hallucinated facts. Experiments on Zoom Blur.
LLaVA-1.5-7B [ 34 ]
LLaVA-1.5-13B [ 34 ]
InstructBLIP-7B [ 13 ]
Base
59.7
61.5
59.0
+ ClipTTT
58.1
67.4
59.1
Qwen2.5-VL-7B [ 3 ]
Qwen2-VL-2B [ 49 ]
Intern2-VL-2B [ 12 ]
Base
53.0
44.7
51.2
+ ClipTTT
52.8
44.2
52.4
Table 10: Left: F1 comparison across LVLMs on COCO ( CHAIR subset). Right: NoCaps (Out-of-Domain) [ 1 ] F1 results. Higher is better.
Method
Severity 1
Severity 3
CHAIRS↓
CHAIRI↓
CHAIRS↓
CHAIRI↓
Greedy Decoding
49.6
13.5
49.9
14.0
VCD [ 30 ]
50.7
13.9
51.4
14.3
VAP [ 57 ]
49.2
13.4
50.3
13.8
ClipTTT (Ours)
48.4
12.3
48.8
12.5
Table 11: Comparison Across Severities. We report average hallucination across 15 corruptions showing sentence-level ( CHAIRS , ↓ ) and instance-level ( CHAIRI , ↓ ) hallucination for severity levels 1 and 3. Lower is better.
Method
CHAIRS↓
CHAIRI↓
Greedy Decoding
49.2
13.1
VCD [ 30 ]
49.2
14.1
VAP [ 57 ]
47.6
13.9
ClipTTT (Ours)
49.1
12.2
Table 12: Left: Clean images evaluation showing ClipTTT is not restricted to corrupted images. Right: Deterministic teacher ablation (stochastic teacher performs better) on Zoom Blur. Experiments on LLaVA-v1.5-7B [ 34 ] .
Figure 7: Left: CLIP Scores before and after test-time training across all corruptions. The largest gains are seen in challenging weather and noise-based corruptions like ‘Snow’,‘Gaussian Noise’, and ‘Shot Noise’. Right: Meta-ClipTTT provides a stronger LoRA initialization, leading to faster convergence and improved performance. Experiment on Zoom Blur.
Method
CHAIRS↓
CHAIRI↓
With Prompt
49.1
18.5
With Null Prompt (Ours)
45.6
17.9
Table 13: Left: Null-prompt ablation. ClipTTT performs test-time training without textual instructions. We compare against a prompted variant using “ Describe the image. ” Right: Sentence-level similarity ablation. ClipTTT computes CLIP similarity by averaging over sentences rather than scoring the full caption. We compare against a full-caption similarity variant. Experiment on Zoom Blur.
Figure 8: Left: Stochastic candidate generation frequency. Ablation of the performance-efficiency trade-off when generating candidates at different re-evaluation intervals. Interval 0 corresponds to the Fixed Teacher setting in Tab. 5 , where captions are generated once at the first iteration and then kept fixed throughout test-time training. Experiment on Zoom Blur. Right: CLIP Score distributions averaged over 15 corruptions.
Figure 9: Visualization of attention weight shifts in LLaVA-1.5-7B. We visualize the average change in attention weights ( ΔAttention=AttentionClipTTT−AttentionBase ). The x-axis represents the Transformer layers (0-31), and the y-axis represents attention heads (0-31). Left: Attention from Instruction tokens to Vision tokens. Middle: Attention from Response tokens to Vision tokens. Right: Attention from Response tokens to Instruction tokens. Red regions indicate where ClipTTT amplifies attention, effectively countering the visual disconnect caused by corruptions. Experiments on Zoom Blur.
Figure 10: CLIP Similarity for Factual Grounding. We compute the CLIP Score (Sec. 3.1 ) between the ground-truth caption and corrupted images across severity levels. The score remains relatively stable even under severe corruptions, motivating its use as a grounding proxy. Results are averaged over 500 COCO validation images.
GPT-4o Prompt
You are required to score the performance of four AI assistants in describing a given image. You should pay extra attention to the hallucination, which refers to the part of descriptions that are inconsistent with the image content, such as claiming the existence of something not present in the image or describing incorrectly in terms of the counts, positions, or colors of objects in the image. Please rate the responses of the assistants on a scale of 1 to 10, where a higher score indicates better performance, according to the following criteria:
1: Accuracy: whether the response is accurate with respect to the image content. Responses with fewer hallucinations should be given higher scores.
2: Detailedness: whether the response is rich in necessary details. Note that hallucinated descriptions should not count as necessary details.
Please output the scores for each criterion, containing only four values indicating the scores for Assistant 1, 2, 3 and 4, respectively. The four scores are separated by a space. Following the scores, please provide an explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
[Assistant 1]
Table 14: The prompt used for GPT-4o assisted evaluation.
Figure 11: Qualtiative Comparison on Real-World Street Scenes : Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .
Figure 12: Additional Qualitative Comparisons: Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .
Object Hallucination in large vision-language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the token level for object tokens selected from a candidate pool. An entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH
Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian +3
University of Technology Sydney · York University, Canada · Australian National University +2
Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.
Sunoh Kim, Daeho Um
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea
Vision-Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient attention rollout mechanism to identify semantically meaningful regions surviving under adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT .
Jia-Wei Hai, Yijun Wang, Xiu-Shen Wei
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.