Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models

Enguang Wang; Qiang Wang; Yuanchen Wu; Ke Yan; Xinbin Yuan; Shouhong Ding; Xialei Liu; Ming-Ming Cheng

cs.CVMar 21, 2026

Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models

Authors: Enguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan, Xinbin Yuan, Shouhong Ding, Xialei Liu, Ming-Ming Cheng

Organizations: NKIARI, Shenzhen Futian · VCIP, CS, Nankai University · Tencent Youtu Lab · AAIS, Nankai University

Abstract

While Multimodal Large Language Models (MLLMs) excel at vision-language tasks, the cost of their language-driven training on internal visual foundational competence remains unclear. In this paper, we conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs. Specifically, we find that compared to the initial visual features, the visual representation in the middle layers of LLM exhibits both a degradation in global function and patch structure. We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation. We argue that a robust MLLM requires both strong cross-modal reasoning and core visual competence, and propose Predictive Regularization (PRe) to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations. Extensive experiments confirm that mitigating this visual degradation effectively boosts vision-language performance, underscoring the critical importance of fostering robust internal visual representations within MLLMs for comprehensive multimodal understanding.

Figures & tables

Figure 1: The Proposed PRe Framework. PRe enforces the degraded representations to predict their initial, clean anchor representations via a lightweight prediction head.

Figure 2: Linear probe results on global visual features. Relative to the initial representation (Layer 0), a consistent performance drop is observed in the intermediate layers.

Figure 3: Evolution of patch-level semantics. The intra- and inter-object similarities rise through most layers, leading to reduce the semantic contrast ratio, revealing patch structure degradation.

Figure 4: The cosine similarity between the patch noted in green and all other patches. Compared with the initial representation, the middle layer blurs the visual semantic boundaries. (Images are from opensource COCO-stuff [ 8 ] dataset)

Figure 5: The statistical properties of the global representations. The increased PCA effective dimension and reduced mean off-diagonal correlation indicate that the representations become both more geometrically complex and statistically independent.

Figure 6: Visual and language capabilities during pre-training. Results show that the language-driven optimization degrades intrinsic visual fidelity.

Figure 7: The effectiveness of PRe in enhancing intrinsic visual capabilities of LLMs. Baseline architecture using CLIP-ViT-L/14@336 (frozen) and Vicuna-7B-v1.5.

Figure 8: The cosine similarity between the patch noted in green and all other patches in the middle layer. Images are from opensource COCO-stuff [ 8 ] dataset.

Visual Encoder	$\mathcal{L}_{\text{PRe}}$	General & Knowledge					OCR-related		Vision-centric
Visual Encoder	$\mathcal{L}_{\text{PRe}}$	GQA	MMMU	AI2D	MMStar	SQA ${}^{\text{I}}$	VQA ${}^{\text{T}}$	OCRbench	RWQA	MMVP
LLM: Vicuna-7B-v1.5
CLIP-ViT-L/14@336 *	–	62.0	35.7	55.4	30.3	66.8	45.5	318	54.8	20.0
CLIP-ViT-L/14@336 *	✓	$\text{62.7}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.7}}}}}$	$\text{36.1}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.4}}}}}$	$\text{57.1}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.7}}}}}$	$\text{34.6}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+4.3}}}}}$	$\text{67.5}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.7}}}}}$	$\text{46.6}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.1}}}}}$	$\text{329}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+11}}}}}$	$\text{55.4}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.6}}}}}$	$\text{22.0}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+2.0}}}}}$
SigLIP2-ViT-SO400M/14@384 *	–	62.5	36.5	58.8	36.8	68.4	57.9	396	56.9	35.3
SigLIP2-ViT-SO400M/14@384 *	✓	$\text{64.3}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.8}}}}}$	$\text{36.4}_{\text{\footnotesize{\color[rgb]{0.9375,0.5273,0.2109}{-0.1}}}}$	$\text{59.3}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.5}}}}}$	$\text{36.7}_{\text{\footnotesize{\color[rgb]{0.9375,0.5273,0.2109}{-0.1}}}}$	$\text{70.0}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.6}}}}}$	$\text{59.2}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.3}}}}}$	$\text{401}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+5}}}}}$	$\text{57.1}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.2}}}}}$	$\text{36.7}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+1.4}}}}}$

Table 1: The effectiveness of the predictive regularization among various LLMs and visual encoders . * indicates the visual encoder is frozen during training, without * indicates the visual encoder is trainable.

Select Layer	GQA	MMMU	VQA ${}^{\text{T}}$	RWQA	MMVP
CLIP (frozen) + Vicuna-7B-v1.5
baseline	62.0	35.7	45.5	54.8	20.0
mid-layer	62.7	36.1	46.6	55.4	22.0
last-layer	62.4	35.6	45.7	54.5	25.3
SigLIP2 (trainable)+ Qwen2.5-7B-Instruct
baseline	63.5	45.8	59.2	60.3	46.0

Table 2: Results of different regularized layers.

Figure 9: Visual token decoding distributions across layers.

Anchor Source	GQA	MMMU	VQA ${}^{\text{T}}$	RWQA	MMVP
CLIP (frozen) + Vicuna-7B-v1.5
Baseline	62.0	35.7	45.5	54.8	20.0
Pre-LLM	62.7	36.1	46.6	55.4	22.0
Pre-Proj	62.7	35.1	46.4	54.4	32.7
DINOv2	62.8	35.9	46.5	54.6	28.7
SigLIP2 (trainable) + Qwen2.5-7B-Instruct

Table 3: Results using different anchor features.

Figure 10: Comparison of PRe applied to global features vs. patch features. The baseline architecture employed is CLIP- ViT-L/14@336 (frozen) and Vicuna-7B-v1.5.

Figure 11: The impact of

\lambda

. The baseline architecture employed is CLIP-ViT-L/14@336 (frozen) and Vicuna-7B-v1.5.

LLM: Qwen2.5-3B-Instruct
Visual Encoder	$\mathcal{L}_{\text{PRe}}$	General & Knowledge					OCR-related		Vision-centric
Visual Encoder	$\mathcal{L}_{\text{PRe}}$	GQA	MMMU	AI2D	MMStar	SQA ${}^{\text{I}}$	VQA ${}^{\text{T}}$	OCRbench	RWQA	MMVP
CLIP-ViT-L/14@336 *	–	61.3	40.7	62.2	41.3	71.9	43.3	318	52.5	31.3
CLIP-ViT-L/14@336 *	✓	$\text{61.3}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.0}}}}}$	$\text{40.9}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.2}}}}}$	$\text{62.5}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.3}}}}}$	$\text{41.1}_{\text{\footnotesize{\color[rgb]{0.9375,0.5273,0.2109}{-0.2}}}}$	$\text{72.5}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.6}}}}}$	$\text{43.4}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.1}}}}}$	$\text{320}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+2}}}}}$	$\text{55.3}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+2.8}}}}}$	$\text{30.7}_{\text{\footnotesize{\color[rgb]{0.9375,0.5273,0.2109}{-0.6}}}}$
CLIP-ViT-L/14@336	–	61.0	39.6	62.2	42.6	72.7	45.5	359	55.6	38.0
CLIP-ViT-L/14@336	✓	$\text{60.9}_{\text{\footnotesize{\color[rgb]{0.9375,0.5273,0.2109}{-0.1}}}}$	$\text{40.0}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.4}}}}}$	$\text{62.4}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.2}}}}}$	$\text{43.2}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.6}}}}}$	$\text{72.9}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.2}}}}}$	$\text{45.9}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.4}}}}}$	$\text{362}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+3}}}}}$	$\text{55.8}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.2}}}}}$	$\text{38.7}_{\textbf{\text{\footnotesize{\color[rgb]{0.3203,0.6211,0.2461}{+0.7}}}}}$

Table 4: The effectiveness of the predictive regularization on a 3B-LLM . * indicates the visual encoder is frozen during training, without * indicates the visual encoder is trainable.

Figure 12: The impact of

\lambda

. The baseline architecture employed is CLIP-ViT-L/14@336 (frozen) and Vicuna-7B-v1.5.

Figure 13: Case studies comparing LLaVA-1.5-7B with and without our PRe method.

CLIP (frozen) + Vicuna-7B-v1.5
Anchor Source	GQA	MMMU	VQA ${}^{\text{T}}$	RWQA	MMVP
Baseline	62.0	35.7	45.5	54.8	20.0
Pre-LLM	62.7	36.1	46.6	55.4	22.0
Pre-Proj	62.7	35.1	46.4	54.4	32.7
DINOv2-vitb14-reg4	62.8	35.9	46.5	54.6	28.7
DINOv3-vitb16	62.6	35.3	46.1	54.9	28.7

Table 5: Results using different anchor features. When the CLIP is frozen, the ‘Pre-Proj’ setting corresponds to using the penultimate layer features from a pre-trained CLIP model, together with DINOv2, DINOv3, and SAM, which form an ablation using vision foundation models

Metric	clip+qwen2.5-3b	+ PRe	Overhead ( $\Delta$ )
Total Training PFLOPs	$2.214$	$2.215$	$+0.045\%$
Throughput (samples/s)	18.74	18.58	$-0.85\%$
Total Training Time (s)	$35.5\times 10^{3}$	$35.8\times 10^{3}$	$+0.85\%$

Table 6: Computational overhead.

	$\text{mme}^{\text{p}}$	$\text{mmb}^{\text{en}}$	$\text{mmb}^{\text{cn}}$	$\text{seed}^{\text{i}}$	pope(acc/f1)
clip+qwen2.5-3b	1408.3	70.1	66.1	69.9	87.5/86.3
+Pre	1434.8	68.3	68.1	70.8	88.7/87.8
clip*+vicuna-7b	1510.7	64.3	46.4	58.6	84.7/85.8
+Pre	1528.3	65.7	56.2	66.3	86.9/85.8

Table 7: Results on more datasets.

	GQA	AI2D	MMStar	$\text{VQA}^{\text{T}}$	RWQA
CLIP (Res: $\text{336}^{\text{2}}$ )	61.3	62.2	41.3	43.3	52.5
+Pre	61.3	62.5	41.1	43.4	55.3
CLIP (Res: $\text{672}^{\text{2}}$ )	60.6	62.1	42.6	45.1	52.7
+Pre	61.0	62.6	44.0	45.9	56.7
Qwen2.5-VL(NaViT)	59.4	60.7	42.5	50.0	50.5
+Pre	60.1	62.2	43.0	53.2	51.9

Table 8: Results on stronger encoders and higher resolutions. LLM: Qwen2.5-3b-Instruct.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Reinforcing Multimodal Reasoning Against Visual Degradation
May 10, 2026Rui Liu, Dian Yu, Haolin Liu +6VLM Robustness Image Corruption Robustness
Tencent Hunyuan · University of Maryland, College Park · University of Virginia +1
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
Aug 3, 2026Qianlong Yang, Bowen Ye, Xianda Guo +4VLM Robustness Multimodal Large Language Models
China University of Petroleum (East China) Qingdao, Shandong, China · Great Wall Motor · Shanghai Jiao Tong University Shanghai, China +3
Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models
Mar 15, 2026Ruiying Peng, Xueyu Wu, Jing Lei +3Multimodal Large Language Models VLM Reasoning
Tsinghua Shenzhen International Graduate School · Huawei Technologies

May 10, 2026J/K move · Enter open · S save

Reinforcing Multimodal Reasoning Against Visual Degradation

Rui Liu, Dian Yu, Haolin Liu, Yucheng Shi+5

Tencent Hunyuan · University of Maryland, College Park · University of Virginia+1

VLM Robustness Image Corruption Robustness Multimodal Large Language Models VLM Reasoning Multimodal Foundation Models RL for VLMs

Open paper AskPDF arXiv

Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.

Abstract

Multimodal Large Language Models

Explore similar work

Reinforcing Multimodal Reasoning Against Visual Degradation

Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning

Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models