Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR

Ziye Yuan; Ruchang Yao; Chengxin Zheng; Yusheng Zhao; Daxiang Dong; Ming Zhang

cs.CVMar 20, 2026

Detached Skip-Links and $R$ -Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR

Authors: Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang

Organizations: State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China · Tsinghua University, Beijing, China · Baidu Inc, Beijing, China

Abstract

Multimodal large language models (MLLMs) excel at high-level reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce $R$ -Probe, which measures pixel-level reconstructability of projected visual tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric benchmarks and delivers clear gains on general multimodal tasks.

Figures & tables

Figure 1: Overview of Detached Skip Links. Intermediate features are concatenated with the final output along the channel dimension, here $S$ denotes stride. We apply a stop-gradient operation to shallow skip features before fusion. This design effectively delivers fine-grained details to the LLM while shielding early layers from optimization conflicts.

Figure 2: Impact of gradient backpropagation on shallow-layer representations. We visualize [CLS] attention maps of the 4th ViT block. Left : Original frozen attention patterns. Middle ( Full Gradients ): Backpropagation from the LLM leads to diffused and structurally inconsistent attention, as semantic-heavy gradients disrupt pre-trained spatial priors. Right ( Detached ): Detaching gradients preserves fine-grained structural consistency.

Panel of Figure 3 — Figure 3: Gradient Analysis on the First Skip-Link Block. We visualize the training dynamics during the early phase of the joint training stage. (a) Gradient norms of the skip and main branches, measured by $\mathbb{E}[\|\mathbf{g}\|^{2}]$ , together with their MA(Moving Average) counterparts, where the MA serves as an estimation to $\|\mathbb{E}[\mathbf{g}]\|^{2}$ . (b) Trace statistics computed as $\mathbb{E}[\|\mathbf{g}\|^{2}]-\|\mathbb{E}[\mathbf{g}]\|^{2}$ , characterizing the variance of the gradients over training. (c) The cosine similarity between $g^{\mathrm{skip}}$ and $g^{\mathrm{main}}$ remains close to zero, indicating approximate orthogonality. (d) Scatter plot of $\delta$ defined in Assumption 4.2 . Additional analysis is provided in Appendix C .

Figure 4: Overview of the $R$ -Probe and its use as an auxiliary training signal. The $R$ -Probe corresponds to the right reconstruction branch, which evaluates information retention and LLM-aligned consumability of visual tokens under frozen encoders. When attached to the full model (left), the same reconstruction head supplies an auxiliary loss during training, encouraging visually faithful representations without modifying the primary LLM decoding objective.

Figure 5: Sequence Construction Strategy. For reconstruction used in

R

-probe, we prioritize context (background image and text prompts) before the target image tokens. This forces the probe to verify if the adapter successfully projects visual features into a space that the LLM can utilize for context-dependent reconstruction.

Figure 6: Ablation study on feature sampling stride ( $S$ ) and the number of detached layers ( $D$ ) . The chart visualizes the OCR performance (top) and the Average score across all benchmarks (bottom).Green nodes indicate improvement, while red nodes indicate degradation.

Setting	STEM	General	Align.	OCR	Overall
PE-baseline	63.0	53.2	72.6	65.2	61.1
Ours (PE-best)	64.1	54.6	73.6	68.3	63.0
DC	63.2	54.0	72.5	66.7	62.0
DC-detached	64.2	54.4	72.8	67.6	62.6
ML	63.5	54.1	72.6	66.9	62.1
ML-detached	63.1	54.0	73.2	68.1	62.5

Table 1: Performance Comparison (Categorized Averages). Full results in Table 9

Setting	STEM	General	Align.	OCR	Overall
PE-baseline	63.0	53.2	72.6	65.2	61.1
PE-ours	64.1	54.6	73.6	68.3	63.0
$\Delta$	+1.1	+1.4	+1.0	+3.1	+1.9
InternViT-baseline	58.7	49.0	64.0	60.3	56.2
InternViT-ours	61.1	50.8	71.4	62.2	58.7
$\Delta$	+2.4	+1.8	+7.4	+1.9	+2.5

Table 2: Performance Across Different ViT Backbones (Categorized Averages). Full results in Table 10

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

LR	$t_{\mathrm{trans}}$	median $(\cos)$	median $(\delta)$
8e-5	400	-0.0011	0.0577
2e-5	900	-0.0005	0.0624
5e-6	/	0.0008	0.0315

Appendix

Table 3: Learning-rate robustness summary of early-phase gradient statistics

Method	OCRBench	DocVQA	TextVQA	ChartQA	OCRVQA	AI2D	CharXiv_DQ	CharXiv_RQ
Baseline	714.0	71.4	68.7	70.0	55.0	72.2	69.7	43.9
Baseline + R-probe	721.0	72.1	68.9	70.1	55.4	69.2	68.3	42.8

Appendix

Table 4: Comparison of OCR capabilities with and without R-probe. “Baseline+R-probe” indicates the model jointly trained with the R-probe reconstruction objective. Note the improvement in pure OCR tasks (OCRBench, DocVQA) versus the trade-off in reasoning-heavy tasks (CharXiv_RQ).

ID	Setting	Final Loss (MSE)
1	Masked Image, w/o Language	1.980
2	Masked Image, w/ Language	1.103
3	Full Image, w/o Language	0.757
4	Full Image, w/ Language	0.724

Appendix

Table 5: Ablation study on input modalities for the R-probe. Lower Final Loss indicates better reconstruction quality.

ID	Configuration	Steps ( $<$ 0.75)	Final Loss
1	Baseline	2158	0.724
2	Ours Stride=12 (w/o Detach)	1935	0.719
3	Ours Stride=12 (Detached)	1956	0.716
4	Ours Stride=6 (w/o Detach)	1834	0.709
5	Ours Stride=6 (Detached)	1689	0.698

Appendix

Table 6: Impact of layer selection and gradient detachment on R-probe convergence and performance. ”Steps” denotes steps to reach MSE

<

0.75.

Backbone	Setting	Steps ( $<$ 0.75)	Final Loss
PE	Original	2158	0.724
PE	Ours	1689	0.698
AimV2	Original	2301	0.741
AimV2	Ours	1889	0.706
InternViT	Original	2189	0.735
InternViT	Ours	1835	0.703

Appendix

Table 7: Performance of R-probe across different MLLM backbones. ”Ours” refers to the best detached multi-layer configuration (Setting ID=5 from Table 6 ).

Backbone	Setting	Recon. Loss $\downarrow$	OCR Score $\uparrow$	General Score $\uparrow$
PE	Original	0.724	65.2	61.1
	Ours Stride=12 (w/o Detach)	0.719	66.1	61.1
	Ours Stride=12 (Detached)	0.716	67.0	61.7
	Ours Stride=6 (w/o Detach)	0.709	66.4	61.8
	Ours Stride=6 (Detached)	0.698	68.3	63.0
AimV2	Original	0.741	63.0	58.8

Appendix

Table 8: Relationship between R-Probe reconstruction loss and downstream benchmark performance within each backbone.

Panel of Figure 7 — Figure 7: Data distribution of the pre-training and SFT stages.

STEM Puzzle
Benchmark	PE-baseline	Ours	DC	DC-detached	ML	ML-detached	DeepStack
STEM Puzzle Avg	63.0	64.1	63.2	64.2	63.5	63.1	63.8
MMMU ${}_{\text{val}}$	41.0	42.8	42.3	43.3	42.6	42.0	43.0
MathVista ${}_{\text{mini}}$	43.3	45.2	44.3	45.1	44.9	44.4	44.8
ScienceQA ${}_{\text{TEST}}$	83.9	83.9	82.9	84.2	83.0	82.8	83.6
ScienceQA ${}_{\text{VAL}}$	83.8	84.7	83.5	83.9	83.4	83.2	84.0

Appendix

Table 9: Detailed performance comparison across all benchmarks. We report results for PE-baseline , our proposed Ours (PE-best), and other fusion strategies including DenseConnector (DC), Multi-Layer Fusion (ML), and DeepStack. For DC and ML , we also show results with our detached gradient strategy applied (-detached). All scores are rounded to one decimal place.

STEM Puzzle
Benchmark	PE		InternViT		AimV2		SigLip
Benchmark	Baseline	Ours	Baseline	Ours	Baseline	Ours	Baseline	Ours
STEM Puzzle Avg	63.0	64.1	58.7	61.1	60.5	62.2	57.1	59.7
MMMU ${}_{\text{val}}$	41.0	42.8	39.0	40.7	40.2	41.5	37.5	40.8
MathVista ${}_{\text{mini}}$	43.3	45.2	38.6	40.3	40.5	42.1	37.8	39.2
ScienceQA ${}_{\text{TEST}}$	83.9	83.9	78.0	82.1	80.5	82.5	76.3	79.4

Appendix

Table 10: Detailed ablation results across different ViT backbones: PE , InternViT , AimV2 , and SigLip2 . For each backbone, we compare the Baseline with our proposed Detached Skip-Links method ( Ours ). All scores are rounded to one decimal place.

Explore similar work

Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs
Jun 7, 2026Jie Ma, Zhike Qiu, Jiayi Ji +2Efficient Multimodal Inference Multimodal Large Language Models
Xiamen University
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
May 13, 2026Qinwu Xu, Yifan Jiang, Haoyu RenMultimodal Grounding CoT Reasoning
Meta AI · UT Austin · Current independent researcher
Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs
Oct 13, 2026Peiran Yin, Hao Wu, Zihao Yang +6Visual Token Pruning Efficient VLM Inference
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University +1

Jun 7, 2026J/K move · Enter open · S save

Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs

Jie Ma, Zhike Qiu, Jiayi Ji, Xiaoshuai Sun+1

Xiamen University

Efficient Multimodal Inference Multimodal Large Language Models Efficient Attention Sparse Attention

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) face a significant inference bottleneck due to the quadratic computational cost of self-attention over long visual token sequences. However, we identify a critical inefficiency in current architectures: Visual Attention Saturation. Our analysis reveals that visual tokens rapidly establish their spatial structure and intra-modal relationships in early layers, rendering visual-to-visual self-attention in deeper layers computationally redundant. Conversely, Feed-Forward Networks (FFNs) in these layers remain essential for projecting visual features into the evolving textual semantic space. Leveraging this insight, we present Visual-Skip (V-Skip), a training-free inference paradigm that decouples spatial interaction from semantic evolution. Rather than discarding tokens, V-Skip imposes block-wise structured sparsity by selectively bypassing saturated visual self-attention modules. Furthermore, recognizing that varying downstream tasks demand distinct reasoning depths, V-Skip employs a lightweight, few-shot calibration to dynamically route the task-optimal sparsity path. Extensive experiments demonstrate that V-Skip effectively bypasses redundant vision attention to achieve block-wise sparsity, maintaining a 94.16% to 100.31% performance retention across diverse MLLMs. Ultimately, we prove that to reason more effectively, models do not need to discard what they see -- they simply need to "look less" at the right depth.

Abstract

Explore similar work

Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs

Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models

Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs