On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs

Mikołaj Piórczyński; Wojciech Masarczyk; Damian Sójka; Sebastian Cygert; Bartłomiej Twardowski

cs.LGOct 14, 2026

On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs

Authors: Mikołaj Piórczyński, Wojciech Masarczyk, Damian Sójka, Sebastian Cygert, Bartłomiej Twardowski

Organizations: IDEAS Research Institute · Warsaw University of Technology · Pozna´n University of Technology · Akces NCBR ·

Abstract

Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space.These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.

Figures & tables

Figure 1: Matching corrupted activations to clean statistics repairs a linear probe, but not the MLLM’s answers. For the probe, the statistics are matched on one pooled vector per image. For the MLLM, they are matched on individual visual tokens.

	InternVL3-2B			Qwen2.5-VL-3B			Ovis2-4B
	no-TTA	Gaussian- AcT	+ dynamic weighting	no-TTA	Gaussian- AcT	+ dynamic weighting	no-TTA	Gaussian- AcT	+ dynamic weighting
Clean	62.5	–	–	65.6	–	–	69.4	–	–
Noise
Gaussian	$54.1_{\text{\tiny$ \pm $0.4}}$	${-0.5}_{\text{\tiny$ \pm $1.9}}$	$\mathbf{+1.4}_{\text{\tiny$ \pm $0.8}}$	$50.8_{\text{\tiny$ \pm $0.5}}$	${-4.1}_{\text{\tiny$ \pm $0.5}}$	$\mathbf{+0.6}_{\text{\tiny$ \pm $1.0}}$	$58.9_{\text{\tiny$ \pm $0.3}}$	${-1.7}_{\text{\tiny$ \pm $1.1}}$	$\mathbf{-0.1}_{\text{\tiny$ \pm $0.5}}$
Shot	$54.2_{\text{\tiny$ \pm $0.5}}$	${+0.4}_{\text{\tiny$ \pm $1.8}}$	$\mathbf{+1.7}_{\text{\tiny$ \pm $1.1}}$	$52.3_{\text{\tiny$ \pm $0.1}}$	${-3.7}_{\text{\tiny$ \pm $1.4}}$	$\mathbf{-0.0}_{\text{\tiny$ \pm $0.9}}$	$59.6_{\text{\tiny$ \pm $0.6}}$	$\mathbf{+1.0}_{\text{\tiny$ \pm $0.5}}$	${+0.6}_{\text{\tiny$ \pm $0.5}}$
Impulse	$55.0_{\text{\tiny$ \pm $1.2}}$	${+0.1}_{\text{\tiny$ \pm $1.3}}$	$\mathbf{+0.7}_{\text{\tiny$ \pm $1.0}}$	$52.2_{\text{\tiny$ \pm $0.6}}$	${-5.4}_{\text{\tiny$ \pm $1.0}}$	$\mathbf{+0.8}_{\text{\tiny$ \pm $0.9}}$	$59.5_{\text{\tiny$ \pm $1.9}}$	$\mathbf{+0.4}_{\text{\tiny$ \pm $2.0}}$	${+0.4}_{\text{\tiny$ \pm $1.7}}$

Table 1: Online TTA with Gaussian-AcT barely moves accuracy on any of the three MLLMs. RealWorldQA accuracy (%) under the 15 common corruptions at severity 5. no-TTA is the accuracy without adaptation. The Gaussian-AcT columns give the change

\Delta

over it in pp when applied at all intervention layers, without and with dynamic weighting. Cells with

|\Delta|\geq 2

pp are shaded green for a gain and red for a loss. The better of the two corrections per corruption is in bold .

Figure 2: A more expressive map does not recover more. Change in accuracy over the no-TTA baseline, averaged over the 15 corruptions, when a single layer is corrected. Gaussian-AcT and WCT are fitted from distribution statistics. Ridge and MLP on paired clean–corrupted tokens.

Figure 3: Decomposition of the total squared shift ( Equation 1 ) across layers , averaged over the 15 corruptions. Solid bars: token-level shifts; hatched bars: sample-pooled shifts. The residual dominates the token-level shift at every layer, whereas the global shift dominates the pooled shift almost everywhere.

Figure 4: Token-level shifts (solid) and sample-pooled shifts (dashed) versus encoder depth. (a) Mean cosine similarity between the shifts of different images and (b) normalized effective rank [ 31 ] of the shift matrix, averaged over the 15 corruptions.

Figure 5: No small subset of the representation carries the degradation. The fraction of the clean–corrupted gap recovered on both axes (dotted: proportional), for Qwen2.5-VL-3B under Gaussian noise at severity 5.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Adaptation is slow and the gain stays small. Cumulative change in RealWorldQA accuracy over the no-TTA baseline along the stream (mean over corruptions and three orders; band: standard error). The unweighted correction drops sharply early, while the online estimate is still poor, and recovers to about break-even; the weighted variant stays near zero and ends with a small gain.

Figure 7: The weight concentrates on the early layers. Mean per-layer weight over the stream against relative encoder depth.

Figure 8: Per-token moments converge more slowly but higher. Cumulative accuracy gain over the no-TTA baseline along the stream for single-layer Gaussian-AcT fitted online with global and per-token target moments, averaged over the 15 corruptions (band:

\pm 1

standard error over corruptions; each corruption averaged over three stream orders). The dashed line marks the same per-token Gaussian-AcT fitted offline and the inset text the gain of clean images; the first 16 samples are not adapted.

Figure 9: Online estimation matches offline fitting. Whole-episode accuracy gain over the no-TTA baseline of per-token Gaussian-AcT with target moments fitted offline (

x

) or estimated online (

y

); one point per model and corruption (mean over three stream orders; bars: standard deviation); dashed line: identity.

Figure 10: Per-corruption encoder repair map. Share of the clean–corrupted margin gap recovered by patching a single attention (left) or MLP (right) sub-module to its clean activation, for all 15 corruptions at severity 5 (rows, colored by family) and every encoder block (columns, relative depth); 100 images per cell, color saturates at $\pm 0.5$ .

Figure 11: A perfect single-layer correction recovers clean accuracy. Accuracy of Qwen2.5-VL-3B under Gaussian noise at severity 5 when the activation of one site is interpolated between its corrupted (

\alpha=0

) and clean (

\alpha=1

) value; one curve per layer, dashed line: clean accuracy.

InternVL3-2B

Qwen2.5-VL-3B

Ovis2-4B

Class.

MCQ

Class.

MCQ

Class.

MCQ

Corruption

\Delta

(pp)

\rho

(of gap)

\Delta

(pp)

\rho

(of gap)

\Delta

(pp)

\rho

(of gap)

\Delta

(pp)

\rho

(of gap)

\Delta

(pp)

\rho

(of gap)

\Delta

(pp)

\rho

(of gap)

Noise

Gaussian

+41.3

+0.89

+0.4

+0.05

+35.9

+0.75

-0.7

-0.04

+33.0

+0.81

-0.3

-0.02

Shot

+39.7

+0.84

-0.5

-0.06

+36.5

+0.75

-0.7

-0.05

+35.8

+0.84

-0.1

-0.01

Impulse

+40.3

+0.89

-2.0

-0.29

+36.3

+0.79

+1.4

+0.10

+31.7

+0.81

-0.8

-0.07

Appendix

Table 2: Per-corruption effect of the projector transport, by readout.

\Delta

is the change in accuracy over no TTA (percentage points) and

\rho=\Delta/(\text{clean}-\text{no TTA})

the fraction of the corruption gap recovered. Classification fits and applies the transport on one pooled vector per image and reads it out with a linear probe on scene pseudo-classes (mean over five stratified splits); MCQ fits and applies it on the unpooled token sequence inside the model, shared across positions, and reads out RealWorldQA accuracy. Per-corruption

\Delta

is a point estimate; the means carry the SEM of the mean.

Figure 12: Per-corruption decomposition of the total squared shift ( Equation 1 ) at a shallow block (blocks 5/7/5 for InternVL3-2B/Qwen2.5-VL-3B/Ovis2-4B), the most favorable case for sample-blind adaptation: share of the total squared shift captured by the global shift

\mu

, the position-specific shift

\mu_{t}-\mu

, the sample-specific shift

m_{i}-\mu

, and the residual. Only the noise family and contrast produce a sizable global component (up to

0.30

); geometric and compression corruptions (elastic transform, JPEG, motion blur) are almost entirely residual.

Figure 13: Importance-based selection does not beat random substitution. Fraction of the clean–corrupted gap recovered against the fraction of patches (left) or channels (right) whose clean activation is substituted, choosing the subset at random or by ranking units on the magnitude or on the across-dataset variance of their clean activations, highest or lowest first; Qwen2.5-VL-3B, Gaussian noise at severity 5, dotted line

y=x

Figure 16: Prompting does not recover corrupted accuracy. One point per model and condition; hollow markers are the clean cells. Error bars are the paired (McNemar) standard error of the difference, so a bar crossing the identity line is consistent with no effect.

Figure 17: Confidence is a weak predictor of correctness. ROC curves of the model’s confidence in its answer as a predictor of correctness, for Qwen2.5-VL-3B on RealWorldQA, clean and under Gaussian noise at severity 5; shaded bands are 95% confidence intervals.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 318 papers

Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
Mar 27, 2026Mriganka Nath, Anurag Das, Jiahao Xie +1VLM Hallucination Mitigation Test-Time Adaptation of VLMs
Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Aug 3, 2026Yan Huang, Guowei Wang, Xu Wang +2VLM Robustness Test-Time Adaptation
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1

Jun 2, 2026J/K move · Enter open · S save

Investigating Adversarial Robustness of Multi-modal Large Language Models

Hashmat Shadab Malik, Muzammal Naseer, Salman Khan

Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia

Adversarial Training VLM Robustness Adversarial Attacks on VLMs Adversarial Robustness Multimodal Large Language Models Multimodal Foundation Models+1

Open paper AskPDF arXiv

Multi-modal Large Language Models (MLLMs) achieve strong performance on vision-language tasks, but incorporating visual inputs through a vision encoder (e.g., CLIP) substantially expands the attack surface, making these models vulnerable to visual adversarial perturbations. Prior defenses typically preserve compatibility with pretrained MLLMs by enforcing strict alignment to CLIP's original embedding space during adversarial fine-tuning; while practical, this constraint fundamentally limits achievable robustness. We present a systematic investigation of adversarial robustness in MLLMs. We first introduce a diagnostic CLIP-alignment protocol that predicts, prior to full MLLM training, which robust vision encoders will transfer effectively to the multimodal setting, revealing that large-scale multimodal adversarial pretraining, rather than unimodal scale alone, is the critical factor for strong robustness transfer. Integrating such encoders into MLLMs via end-to-end multimodal training yields average gains of 28 CIDEr points on captioning and 11.7% VQA accuracy under strong adversarial attacks compared to constrained plug-and-play baselines. We further show that adversarial training applied directly to a standard non-robust MLLM degrades both clean and adversarial performance, establishing robust visual representations as a strict prerequisite, while end-to-end adversarial training from a robust backbone delivers additional gains of 1.9 CIDEr points and 4.3% VQA accuracy. Beyond training-time defenses, lightweight test-time visual stochastic transformations serve as an effective black-box defense for non-robust MLLMs, elevating adversarial performance from near-zero to levels comparable with robust models. Finally, we show that our robust models substantially reduce toxic generation under white-box visual jailbreak attacks. Code and pretrained weights will be released publicly.

Abstract

Test-Time Adaptation

Explore similar work

Investigating Adversarial Robustness of Multi-modal Large Language Models

ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models