Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation

Yiming Ren; Yujiu Yang; Junjie Wang

cs.CVMar 27, 2026

Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation

Authors: Yiming Ren, Yujiu Yang, Junjie Wang

Organizations: Tsinghua University

Abstract

Supervised fine-tuning (SFT) on visual instruction data often improves perceptual capabilities in vision-language models (VLMs) while degrading reasoning performance, creating a persistent reasoning tax during post-training. We investigate whether this degradation is related to disrupted access to depth-wise representations, and find that even fixed cross-depth aggregation substantially restores reasoning, suggesting that preserved cross-depth access is an important missing factor in VLM fine-tuning. Building on this observation, we propose Input-Adaptive Depth Aggregation (IADA), a lightweight mechanism that makes cross-depth retrieval input-adaptive, modality-aware, and efficiently parameterized through a low-rank bottleneck. On Qwen3-VL-2B, IADA improves the average reasoning score by 9.5 points and the average perception score by $3.3$ points over LoRA-only fine-tuning with only 0.14M additional parameters, with the strongest gains appearing in parameter-efficient low-rank settings.

Figures & tables

Figure 1: Overview of IADA. Compared with LoRA fine-tuning in (a), IADA in (b) inserts an IADA block at block boundaries to enable explicit cross-depth routing. IADA generates modality-specific queries from the current hidden states, retrieves information from cached states at previous $n_{d}$ block boundaries, and injects the resulting aggregated residuals back through gated residual addition.

LoRA Rank	LoRA Params	Method	R-Avg	P-Avg	All-Avg
16	3.8M	B: LoRA-only	52.2	72.0	63.0
		D: IADA +LoRA	61.7	75.3	69.1
		$\Delta$ (D $-$ B)	+9.5	+3.3	+6.1
32	7.6M	B: LoRA-only	54.8	72.8	64.6
		D: IADA +LoRA	55.3	75.1	66.1
		$\Delta$ (D $-$ B)	+0.5	+2.3	+1.5

Table 3: Interaction between IADA and LoRA rank. LoRA rank and its corresponding trainable parameter count are shown explicitly, where LoRA Params denotes the trainable parameters introduced by LoRA alone. The gains of IADA are substantially larger at rank

16

than at rank

32

, indicating that explicit depth routing is especially beneficial when PEFT capacity is constrained.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
May 19, 2026Juncheng Wu, Hardy Chen, Haoqin Tu +6Vision-Language Models VLM Reasoning
Amazon · UC Santa Cruz · University of Waterloo +1
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
Jun 10, 2026Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8Visual Token Pruning Test-Time Adaptation of VLMs
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1
On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training
May 28, 2026Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang +1Supervised Fine-Tuning Vision-Language Models
University of California, Los Angeles

May 19, 2026J/K move · Enter open · S save

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang+5

Amazon · UC Santa Cruz · University of Waterloo+1

Vision-Language Models VLM Reasoning Visually Grounded Reasoning Curriculum Learning VLM Post-Training Reasoning for VLMs

Open paper AskPDF arXiv

Recent advances in vision-language models (VLMs) emphasize long chain-of-thought reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to reasoning itself. In this work, we systematically study the interplay between perception and reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual reasoning, and textual reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces, suggesting that superior perception reduces the need for excessive reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.

Abstract

VLM Adaptation

Explore similar work

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

AVIS: Adaptive Test-Time Scaling for Vision-Language Models

On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training