ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

Mriganka Nath; Anurag Das; Jiahao Xie; Bernt Schiele

Figure 1: ClipTTT improves robustness and reduces hallucinations under corruptions. Left: LVLMs suffer from hallucinations. Such an issue becomes worse with corruption, indicating that degradation leads to unreliable generation. Right: We benchmark the hallucinations on CHAIR [ 42 ] across 15 corruptions adopted in [ 22 ] . Naïvely applying CLIP for test-time response selection may even degrade the performance. In contrast, by performing ClipTTT at test time, we consistently reduce the hallucinations of the vanilla models (

\text{CHAIR}_{\text{S}}

\downarrow

and

\text{CHAIR}_{\text{I}}

\downarrow

).

Figure 2: Left: Hallucinations increase with corruption severities, indicating that stronger degradations reduce generation reliability (example corruption: Zoom Blur). Right: Captions with higher image-text alignment (CLIP Score) tend to be more factually correct. Results are averaged over 15 corruption types. Both experiments use LLaVA-v1.5-7B [ 34 ] .

Figure 3: Overview of our ClipTTT framework. For each single corrupted test input, we employ a student-teacher framework for on-the-fly adaptation. (1) The Teacher model generates

n

diverse caption candidates via sampling. (2) An external CLIP model scores each candidate, and the one with the highest visual-semantic alignment is selected as the pseudo-label. (3) The Student model is trained for one step on this pseudo-label, with gradients updating only its parameter-efficient LoRA weights. (4) The Teacher’s LoRA weights are then updated via an exponential moving average (EMA) of the Student’s, ensuring a stable yet progressively improving training target.

Method	brigh	cont	defoc	elast	fog	frost	gauss	glass	impul	jpeg	motn	pixel	shot	snow	zoom	AVG
Greedy Decoding	56.8	51.8	57.8	54.2	53.8	52.6	55.4	55.0	55.4	55.8	58.0	58.2	55.4	57.0	56.4	55.6
VCD [ 30 ]	55.6	50.0	56.6	51.4	54.2	53.6	54.0	54.8	54.6	57.6	58.8	55.8	54.0	55.2	53.2	54.6
PAI [ 35 ]	58.6	48.6	56.6	54.6	51.6	54.0	56.4	55.6	56.8	57.2	62.2	58.2	54.0	55.2	57.0	55.8
VAP [ 57 ]	54.0	46.8	56.7	53.9	50.7	53.1	52.0	50.9	52.3	55.2	55.9	57.5	52.6	49.7	50.2	52.8
CGD [ 15 ]	52.4	51.4	59.1	56.0	53.8	52.4	54.0	56.0	55.4	57.2	56.1	59.8	52.2	55.6	51.6	54.9
Restormer [ 55 ]	56.2	47.4	57.6	53.6	53.8	52.8	57.2	54.0	59.4	56.8	58.6	57.0	57.6	54.2	55.2	55.4

Table 1: Comparison of sentence-level hallucination (

\text{CHAIR}_{\text{S}}

\downarrow

) . Lower scores indicate better performance. The results are evaluated across 15 corruption types from the CHAIR benchmark. The best result in each column is in bold . Column headers are abbreviated.

Method	brigh	cont	defoc	elast	fog	frost	gauss	glass	impul	jpeg	motn	pixel	shot	snow	zoom	AVG
Greedy Decoding	13.7	14.8	16.4	14.6	13.5	13.9	16.4	16.5	15.8	13.6	15.2	14.5	14.8	16.2	21.2	15.4
VCD [ 30 ]	13.9	15.2	17.0	14.4	13.6	15.5	16.2	15.3	15.5	14.5	15.3	14.2	14.8	16.2	20.3	15.5
PAI [ 35 ]	14.3	13.8	15.5	14.8	12.9	15.1	16.1	16.7	15.7	14.1	16.1	14.6	14.5	15.6	21.0	15.4
VAP [ 57 ]	13.7	14.3	14.4	15.3	14.0	15.5	15.4	15.5	16.3	14.7	15.7	15.2	15.4	14.7	19.7	15.3
CGD [ 15 ]	14.1	14.6	15.5	15.2	13.9	15.1	15.9	14.3	15.5	13.7	15.6	15.4	16.0	15.6	19.6	15.3
Restormer [ 55 ]	14.7	16.7	17.2	15.5	14.1	15.1	24.4	16.9	26.6	14.3	15.8	14.4	22.0	15.0	22.0	17.7

Table 2: Comparison of instance-level hallucination (

\text{CHAIR}_{\text{I}}

\downarrow

) . Lower scores indicate better performance. The best result in each column is in bold .

Model

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

TinyLLaVA-1.1B [ 59 ] 59.8 20.2 + ClipTTT 59.4 19.9 LLaVA-1.5-7B [ 34 ] 56.4 21.2 + ClipTTT 45.6 18.0 LLaVA-1.5-13B [ 34 ] 55.8 20.2 + ClipTTT 52.2 15.1

Model

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

InstructBLIP-7B [ 13 ] 45.8 17.4 + ClipTTT 45.3 16.9 InternVL2-2B [ 12 ] 49.3 16.6 + ClipTTT 49.1 16.1 Qwen2-VL-2B [ 49 ] 43.5 19.1 + ClipTTT 40.2 18.7 Qwen2.5-VL-7B [ 3 ] 42.4 18.0 + ClipTTT 35.4 17.9

Table 3: Effect of model scales and architectures. Left: Scalability analysis across model sizes (1.1B to 13B). Right: Evaluation on diverse architectures. ClipTTT consistently reduces hallucinations across all setups.

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Greedy Decoding 63.3 24.0 VAP [ 57 ] 51.5 21.0 VCD [ 30 ] 51.8 23.9 ClipTTT (Ours) 50.0 19.8

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Greedy Decoding 64.2 34.0 VAP [ 57 ] 56.6 31.2 VCD [ 30 ] 56.0 34.4 ClipTTT (Ours) 49.5 30.2

Table 4: Left: NoCaps (Near-Domain) results. CHAIR score for NoCaps data for objects present in COCO. Right: Nocaps (Out-of-Domain) results. CHAIR score for NoCaps data for objects beyond in COCO. ClipTTT demonstrates strong generalization by lowering hallucination rates on unseen categories.

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Final Iteration 46.2 18.4 Best Loss Iteration 48.2 19.2 Ours (Max CLIP Score) 45.6 17.9

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Fixed Teacher 53.9 18.1 Dynamic Teacher 52.0 18.7 Ours (EMA Teacher) 45.6 17.9

Table 5: Left: Ablation on checkpoint strategy. CHAIR score based on which checkpoint is used for inference. Right: Ablation on teacher update strategy. CHAIR score based on how the teacher model is updated.

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Vision Encoder LoRA 48.0 18.1 Vision + LLM LoRA 45.8 18.0 Ours (LLM LoRA only) 45.6 17.9

Method

\text{CHAIR}_{\text{S}}

\downarrow

\text{CHAIR}_{\text{I}}

\downarrow

Greedy Decoding 56.4 21.2 CLIP w/o TTT 55.2 20.1 Ours (ClipTTT) 45.6 17.9

Table 6: Left: Ablation on LoRA module placement. CHAIR score for different placements of LoRA training. Right: Ablation on pseudo-label quality. CHAIR score compared between the baseline, the CLIP ranked pseudo-label before TTT starts, and ClipTTT. Our CLIP-based selection provides a better starting point than the baseline, which is further improved by TTT.

Figure 4: Left: CLIP Score distribution comparison. Kernel density estimates of CLIP Scores for captions of Clean images (upper bound), corrupted images with baseline Greedy decoding, and corrupted images after applying Our Method (ClipTTT). Our method shifts the degraded distribution significantly towards the clean distribution. Right: Improvement trend by baseline score. The average improvement provided by ClipTTT is higher when the initial baseline caption has a low-to-medium quality score. The method adaptively reduces its intervention as the baseline quality improves.

Figure 5: Left: Ablation on corruption severity. Sev. 1 and Sev. 3 CHAIR scores for Greedy Decoding, training-free test-time methods (VAP [ 57 ] , VCD [ 30 ] ), and ClipTTT. Right: Inference time vs. performance tradeoff. Tradeoff curve of ClipTTT at different iterations and against different test-time approaches.

Method Avg. CLIP Score

\uparrow

Clean Images (Upper Bound) 23.7 Greedy Decoding 22.5 CLIP w/o TTT 22.9 ClipTTT (Ours) 23.6

Method Accuracy

\uparrow

Detailedness

\uparrow

Greedy (Base) 5.2 5.2 VCD [ 30 ] 4.9 5.1 VAP [ 57 ] 5.3 5.2 ClipTTT (Ours) 5.7 5.3

Table 7: Left: Average CLIP Score improvement. CLIP Score comparison between Greedy, pseudo-label, responses before TTT, and ClipTTT. Right: GPT-4o assisted evaluation. Accuracy reflects the absence of hallucinations, while Detailedness reflects the richness of the description. ClipTTT improves accuracy and maintains detailedness across methods.

Figure 6: Qualitative Comparison. Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Corruptions (severity 5): frost, impulse noise, gaussian noise and defocus blur. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .

Method	TinyLLaVA-1.1B [ 59 ]		InstructBLIP-7B [ 13 ]		Qwen2-VL-2B [ 4 ]
Method	$\textbf{CHAIR}_{\textbf{S}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{I}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{S}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{I}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{S}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{I}}$ $\downarrow$
Greedy Decoding	63.0	21.1	55.7	15.9	47.6	15.0
VCD [ 30 ]	62.9	20.7	55.3	15.5	47.7	15.3
VAP [ 57 ]	62.9	21.6	55.0	15.7	47.8	15.2
ClipTTT (Ours)	62.4	18.7	54.6	15.3	47.2	14.2

Table 8: Generalization Across Models. We report average hallucination across 15 corruptions showing the sentence-level (

\text{CHAIR}_{\text{S}}

\downarrow

) and instance-level (

\text{CHAIR}_{\text{I}}

\downarrow

) hallucination for three LVLMs. Lower is better.

Method	F-Score $\uparrow$	F-Score S $\uparrow$
Greedy (Base)	76.3	60.9
VCD [ 30 ]	80.0	63.2
VAP [ 57 ]	81.7	64.1
ClipTTT (Ours)	83.1	64.3

Table 9: Comparison with FAITHSCORE [ 26 ] . We evaluate ClipTTT against greedy decoding and strong training-free test-time baselines. FAITHSCORE is a reference-free measure of visual faithfulness computed via GPT-4o-based verification [ 24 ] . F-Score (fact-level) measures overall atomic-fact faithfulness of the response. F-Score S (sentence-level) is the fraction of sentences that contain no hallucinated facts. Experiments on Zoom Blur.

	LLaVA-1.5-7B [ 34 ]	LLaVA-1.5-13B [ 34 ]	InstructBLIP-7B [ 13 ]
Base	59.7	61.5	59.0
+ ClipTTT	58.1	67.4	59.1
	Qwen2.5-VL-7B [ 3 ]	Qwen2-VL-2B [ 49 ]	Intern2-VL-2B [ 12 ]
Base	53.0	44.7	51.2
+ ClipTTT	52.8	44.2	52.4

Table 10: Left: F1 comparison across LVLMs on COCO ( CHAIR subset). Right: NoCaps (Out-of-Domain) [ 1 ] F1 results. Higher is better.

Method	Severity 1		Severity 3
Method	$\textbf{CHAIR}_{\textbf{S}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{I}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{S}}$ $\downarrow$	$\textbf{CHAIR}_{\textbf{I}}$ $\downarrow$
Greedy Decoding	49.6	13.5	49.9	14.0
VCD [ 30 ]	50.7	13.9	51.4	14.3
VAP [ 57 ]	49.2	13.4	50.3	13.8
ClipTTT (Ours)	48.4	12.3	48.8	12.5

Table 11: Comparison Across Severities. We report average hallucination across 15 corruptions showing sentence-level (

\text{CHAIR}_{\text{S}}

,

\downarrow

) and instance-level (

\text{CHAIR}_{\text{I}}

,

\downarrow

) hallucination for severity levels 1 and 3. Lower is better.

Method	$\text{CHAIR}_{\text{S}}$ $\downarrow$	$\text{CHAIR}_{\text{I}}$ $\downarrow$
Greedy Decoding	49.2	13.1
VCD [ 30 ]	49.2	14.1
VAP [ 57 ]	47.6	13.9
ClipTTT (Ours)	49.1	12.2

Table 12: Left: Clean images evaluation showing ClipTTT is not restricted to corrupted images. Right: Deterministic teacher ablation (stochastic teacher performs better) on Zoom Blur. Experiments on LLaVA-v1.5-7B [ 34 ] .

Figure 7: Left: CLIP Scores before and after test-time training across all corruptions. The largest gains are seen in challenging weather and noise-based corruptions like ‘Snow’,‘Gaussian Noise’, and ‘Shot Noise’. Right: Meta-ClipTTT provides a stronger LoRA initialization, leading to faster convergence and improved performance. Experiment on Zoom Blur.

Method	$\text{CHAIR}_{\text{S}}$ $\downarrow$	$\text{CHAIR}_{\text{I}}$ $\downarrow$
With Prompt	49.1	18.5
With Null Prompt (Ours)	45.6	17.9

Table 13: Left: Null-prompt ablation. ClipTTT performs test-time training without textual instructions. We compare against a prompted variant using “ Describe the image. ” Right: Sentence-level similarity ablation. ClipTTT computes CLIP similarity by averaging over sentences rather than scoring the full caption. We compare against a full-caption similarity variant. Experiment on Zoom Blur.

Figure 8: Left: Stochastic candidate generation frequency. Ablation of the performance-efficiency trade-off when generating candidates at different re-evaluation intervals. Interval 0 corresponds to the Fixed Teacher setting in Tab. 5 , where captions are generated once at the first iteration and then kept fixed throughout test-time training. Experiment on Zoom Blur. Right: CLIP Score distributions averaged over 15 corruptions.

Figure 9: Visualization of attention weight shifts in LLaVA-1.5-7B. We visualize the average change in attention weights (

\Delta\text{Attention}=\text{Attention}_{\text{ClipTTT}}-\text{Attention}_{\text{Base}}

). The x-axis represents the Transformer layers (0-31), and the y-axis represents attention heads (0-31). Left: Attention from Instruction tokens to Vision tokens. Middle: Attention from Response tokens to Vision tokens. Right: Attention from Response tokens to Instruction tokens. Red regions indicate where ClipTTT amplifies attention, effectively countering the visual disconnect caused by corruptions. Experiments on Zoom Blur.

Figure 10: CLIP Similarity for Factual Grounding. We compute the CLIP Score (Sec. 3.1 ) between the ground-truth caption and corrupted images across severity levels. The score remains relatively stable even under severe corruptions, motivating its use as a grounding proxy. Results are averaged over 500 COCO validation images.

GPT-4o Prompt
You are required to score the performance of four AI assistants in describing a given image. You should pay extra attention to the hallucination, which refers to the part of descriptions that are inconsistent with the image content, such as claiming the existence of something not present in the image or describing incorrectly in terms of the counts, positions, or colors of objects in the image. Please rate the responses of the assistants on a scale of 1 to 10, where a higher score indicates better performance, according to the following criteria:
1: Accuracy: whether the response is accurate with respect to the image content. Responses with fewer hallucinations should be given higher scores.
2: Detailedness: whether the response is rich in necessary details. Note that hallucinated descriptions should not count as necessary details.
Please output the scores for each criterion, containing only four values indicating the scores for Assistant 1, 2, 3 and 4, respectively. The four scores are separated by a space. Following the scores, please provide an explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.

[Assistant 1]

Table 14: The prompt used for GPT-4o assisted evaluation.

Figure 11: Qualtiative Comparison on Real-World Street Scenes : Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .

Figure 12: Additional Qualitative Comparisons: Comparison on corrupted images with identical prompt - “ Please describe the image in detail. ”; clean images shown for reference. Red text denotes hallucinations in the baseline’s greedy decoding, while green text highlights accurate corrections by ClipTTT. Evaluation on LLaVA-v1.5-7B [ 34 ] .

ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

Abstract

Figures & tables

VLM Hallucination Mitigation

Test-Time Hallucination Control in Large Vision-Language Models

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models

Abstract

VLM Hallucination Mitigation

Explore similar work

Test-Time Hallucination Control in Large Vision-Language Models

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models