What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

Ziran Li; Qiang Wang; Zhengyu Chen; Shanglin Lei; Borun Chen; Jingang Wang; Xunliang Cai

cs.CLJun 24, 2026

What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

Authors: Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai

Organizations: Meituan · Tsinghua University

Abstract

Choosing the right large language model (LLM) backbone is the most consequential decision when building a vision-language model (VLM), yet it remains fundamentally unprincipled: compute-based scaling laws fail to generalize across model families, and no framework exists for directly predicting VLM performance before training begins. We propose the Capability-Driven Multimodal Scaling Law, the first cross-family framework that predicts VLM benchmark accuracy from directly observable textual capability. Given a low-dimensional capability score $S$ extracted from LLM textual benchmarks via PCA, we model VLM performance as a function of $S$ , with a per-backbone transfer rate and an absorption rate that quantifies data-scaling efficiency. To fit and validate the framework, we train over 150 VLMs on 34 LLMs spanning 7 model families under a strictly controlled recipe. Evaluations on more than 200 textual and 50 multimodal benchmarks show that the law accurately extrapolates transfer rate from models up to 8B parameters to 72B-scale backbones, predicts full VLM training trajectories with high fidelity, and generalizes to entirely held-out model families. Beyond the scaling law, our analysis surfaces actionable insights: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark-gaming behavior; base LLMs outperform instruction-tuned counterparts as VLM backbones due to higher absorption rates and lower data-scaling decay; and different model families occupy distinct positions in the transfer--absorption space. The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision. Code and data are available at https://github.com/wangq-dev/CDMScaling.

Figures & tables

Figure 1: Comparison of two scaling law formulations fitted to the Falcon3 and Llama-3.2 base model families. Left: Compute-based scaling law, where model scale is parameterized directly by the number of parameters

N

. Right: Capability-driven scaling law, where the effective model scale is parameterized by principal components derived from downstream benchmark performance. Solid lines denote observed training loss trajectories and dashed lines denote fitted predictions.

Figure 2: Multimodal accuracy scaling law fitting results across all 32 LLM backbones. For clarity, only the first 4 training checkpoints are shown for each backbone. The dashed line shows the fitted scaling law (Eq. 7 ).

Panel of Figure 3 — Figure 3: Comparison of predicted vs. observed multimodal accuracy under different evaluation settings, with fitted scaling laws and prediction error bands ( $\pm 1.96\sigma$ ), where $\sigma$ is the standard deviation of the training residuals.

Variant	$\leq$ 8B MAE $\downarrow$	72B-Base MAE $\downarrow$	72B-Inst MAE $\downarrow$
	$S\in[-0.369,0.339]$	$S>0.6$
$B_{0}\cdot\ln D_{\text{mm}}$	1.266%	2.374%	0.476%
$(B_{0}-B_{m}\cdot S)\cdot\ln D_{\text{mm}}$	1.259%	2.055%	0.394%

Table 1: Comparison of in-distribution (

\leq

8B) and out-of-distribution (72B) prediction errors (MAE %) with and without capability-modulated absorption (

B_{m}S

Figure 4: Cross-family generalization using the capability-driven formula (Eq. 7 ).

Type	$A$	$B_{0}$ ( $10^{-2}$ )	$B_{m}$ ( $10^{-2}$ )	$\bar{B}$ ( $10^{-2}$ )	MAE (%)
Base	0.212	1.60	0.78	1.61	1.33
Chat	0.254	1.49	1.04	1.47	1.11

Table 2: Comparison of scaling and transfer parameters between Base and Instruct backbones fitted via Eq. 7 .

Family	$A$	$B_{0}$ ( $10^{-2}$ )	$B_{m}$ ( $10^{-2}$ )	$\bar{B}$ ( $10^{-2}$ )	MAE (%)
Llama	0.361	0.28	4.55	1.22	0.90
Gemma	0.324	2.46	4.51	2.13	1.02
Falcon3	0.288	1.12	1.33	1.17	0.60
Qwen3	0.246	1.40	2.19	1.26	1.11

Table 3: Fitted scaling and transfer parameters of Eq. 7 across representative model families. For consistency,

B_{0},B_{m}

, and

\bar{B}

are scaled by

10^{2}

Figure 5: Relationship between the optimal hyperparameter batch size and the proposed composite scaling metric in multimodal finetuning. The predicted optimal batch sizes for Qwen2.5-72B and Qwen2.5-72B-Instruct at 50B tokens are annotated in the figure.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Correlation between LLM text scores and VLM multimodal scores across 17 models. Each point represents an individual model. The dashed line indicates the linear regression fit. Spearman’s rank correlation coefficient (

\rho=0.791

p<0.001

) is shown in the upper left, and the fitted equation (

y=0.23x+59.20

) is located in the lower right. All evaluation scores are sourced from the OpenCompass leaderboard [ OpenCompass Contributors, 2023 ] , available at https://opencompass.org.cn .

Panel of Figure 7 — Figure 7: Multimodal training loss scaling law fitting and cross-family comparison, using the methodology of Hoffmann et al. [2022] . (a) Falcon3 family fitted with a compute-based scaling law. (b) Despite similar parameter counts, Llama-3.2-3B and Falcon3-3B-Base show divergent loss trajectories that cannot be captured by a single compute-based scaling curve.

Family	Model	Type	Size
Qwen3	Qwen3-0.6B-Base	Base	0.6B
	Qwen3-0.6B	Instruct	0.6B
	Qwen3-1.7B-Base	Base	1.7B
	Qwen3-1.7B	Instruct	1.7B
	Qwen3-4B-Base	Base	4B
	Qwen3-4B	Instruct	4B

Appendix

Table 4: Summary of all 34 LLM backbones used in this study. Model names correspond to their Hugging Face identifiers.

Category	Benchmark	Source
Information Extraction	CrossNER	[ Liu et al., 2021 ]
Knowledge	hindu_knowledge	BB
	mmlu_stem	MMLU
	mmlu_humanities	MMLU
	mmlu_other	MMLU
	dark_humor_detection	BB

Appendix

Table 5: Textual benchmarks used in this study, grouped by capability dimension. BB: BIG-Bench [ Srivastava et al., 2022 ] ; BBH: BIG-Bench Hard [ Suzgun et al., 2022 ] ; MMLU: Massive Multitask Language Understanding [ Hendrycks et al., 2021a ] .

Category	Benchmark
General VQA	MMBench_DEV_EN_V11 [ Liu et al., 2024c ]
	MMStar [ Chen et al., 2024 ]
	SEEDBench_IMG [ Li et al., 2023b ]
	SEEDBench2 [ Li et al., 2023a ]
	SEEDBench2_Plus [ Li et al., 2024d ]
	MME [ Fu et al., 2026 ]

Appendix

Table 6: Multimodal benchmarks used in this study, grouped by capability dimension.

Rank	Benchmark	$\lambda_{j}$	Rank	Benchmark	$\lambda_{j}$
Positive Transfer ( $\lambda_{j}>0$ )			Positive Transfer (Continued)
1	dyck_languages_hard	+0.119	21	checkmate_in_one	+0.019
2	hindu_knowledge	+0.108	22	undo_permutation	+0.019
3	play_dialog_same_or_different	+0.082	23	mmlu_other	+0.016
4	word_unscrambling	+0.063	24	causal_judgement_hard	+0.015
5	mmlu_stem	+0.057	25	winowhy	+0.015

Appendix

Table 7: Complete list of transfer coefficients

\lambda_{j}

for all textual benchmarks (subsets are merged), ranked by absolute value and categorized into Positive Transfer and Transfer Tax regimes.

		Stage-1	Stage-2
Vision	Resolution	384	384 $\times$ {(1 $\times$ 1),…,(6 $\times$ 6)}
Vision	#tokens	729	Max 10 $\times$ 729
Data	Samples	5M	12M
Model	Trainable	Projector	Full Model
Training	Batch Size	512	512
	LR	$1\times 10^{-3}$	$1\times 10^{-5}$

Appendix

Table 8: Configuration for training across various stages.

Panel of Figure 8 — Figure 8: Comparison of two alternative predictors for VLM performance. Both average textual benchmark score and model parameter count provide coarse fitting trends but exhibit substantially weaker extrapolation behavior compared with the proposed capability-driven formulation shown in Figure 2 .

Figure 9: Robustness analysis after removing highly correlated textual benchmarks. All textual benchmarks whose maximum Spearman correlation with any multimodal benchmark exceeds 0.9 are excluded before refitting the capability-driven scaling law. The resulting model maintains nearly identical fitting error and only moderately increased holdout prediction error, indicating that the proposed framework does not rely on a small set of highly correlated benchmarks.

Category	$\hat{A}$	$B_{0}$ ( $10^{-2}$ )	$B_{m}$ ( $10^{-2}$ )	$\bar{B}$ ( $10^{-2}$ )	MAE (%)
General VQA	0.322	1.502	1.056	1.501	1.552
Document Understanding	0.228	2.601	0.917	2.601	1.974
STEM Puzzle	0.212	1.665	0.098	1.665	1.583
Alignment	0.245	0.728	2.309	0.728	0.996

Appendix

Table 9: Category-specific fitting of the Capability-Driven Multimodal Scaling Law. Multimodal benchmarks are grouped according to the taxonomy in Table 6 , and Eq. 7 is fitted independently for each category. For consistency,

B_{0}

B_{m}

, and

\bar{B}

are scaled by

10^{2}

Stable Positive Transfer
Benchmark	Mean $\lambda$	Std. $\lambda$	Pos.	Neg.	Benchmark	Mean $\lambda$	Std. $\lambda$	Pos.	Neg.
dyck_languages_hard	0.374	0.156	6	0	symbol_interpretation	0.118	0.038	6	0
matrixshapes	0.341	0.106	5	0	disfl_qa	0.116	0.048	6	0
hindu_knowledge	0.282	0.091	6	0	wic	0.116	0.109	5	1
word_unscrambling	0.261	0.052	6	0	mmlu_humanities	0.106	0.062	5	1
winogrande	0.217	0.097	5	0	winowhy	0.105	0.144	5	1

Appendix

Table 10: Robustness of benchmark-level transfer coefficients across six complementary fitting settings. Benchmarks are classified as Stable Positive Transfer if

\lambda_{j}>0

in at least five settings, as Stable Transfer Tax if

\lambda_{j}<0

in at least five settings, and as Unstable otherwise. For benchmarks excluded by the contamination-resistant filtering procedure, statistics are computed over the remaining available fits. Benchmarks within each group are ordered by the absolute value of their mean transfer coefficient.

Family	Backbone	$\Delta\beta$	$D^{*}$ (B)	Interpretation
LLaMA-3.2	Llama-3.2-1B	+0.828	85.2	Crossover
Gemma-2	Gemma-2-9B	+0.903	234.5	Crossover
Qwen3	Qwen3-30B-A3B	+1.117	587.8	Large-scale crossover
Qwen3	Qwen3-4B	+0.084	–	Base-favorable
Falcon3	Falcon3-3B	+0.024	$7.6\times 10^{13}$	Weak / unstable
Falcon3	Falcon3-10B	-0.098	–	Chat-favorable

Appendix

Table 11: Representative pair-wise crossover analysis between Base and Instruct variants.

\Delta\beta=\beta_{\rm base}-\beta_{\rm chat}

measures the relative multimodal scaling slope of Base variants, and

D^{*}

denotes the estimated crossover point in billion multimodal training tokens.

Explore similar work

Oct 1, 2026cs.CV

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.

Xinye Zhao, Yunkai Dang, Yunchen Wu +1

School of Intelligence Science and Technology, Nanjing University

Vision-Language Models Efficient VLM Inference Language Model Scaling Laws

Ask PDF

Jun 27, 2026cs.CV

On Test-Time Scaling for Vision-Language Models

Test-time scaling is a paradigm where large models use additional compute at inference to achieve better performance, without changing model weights. While it has been widely studied for Large Language Models (LLMs), its applicability to Large Vision-Language Models (LVLMs) remains less explored and analyzed, with limited analysis of whether, when, and to what extent these approaches transfer to LVLMs. In this work, we ask a simple but fundamental question: can conventional test-time scaling methods developed for LLMs be directly applied to LVLMs? We present the first comprehensive study of test-time scaling for LVLMs, spanning multiple models and model sizes, nine test-time scaling methods, and six diverse benchmarks. Our main findings is that 1) different from previous findings, small, well-performing models benefit the most from test-time scaling, enabling performance improvements of up to around 30%, reaching large models performance, and often outperforming them, 2) LVLMs lose focus when given more compute than necessary, and 3) Visual information is encoded early in the reasoning chain, after which the chain is dominated by text-only reasoning and the contribution of image tokens drops significantly. Finally, we also provide a global and fine-grained analysis on the quality and information sufficiency of the reasoning chains produced. Overall, our findings and analysis provide practical guidance and insights into LVLMs and their deployment in research and industry.

Fawaz Sammani, Tzoulio Chamiti, Nikos Deligiannis

ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium

VLM Evaluation Test-Time Scaling for VLMs Test-Time Scaling Large Vision-Language Models VLM Reasoning

Ask PDF

Aug 4, 2026cs.CV

ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.

Yang Yang, Qinyu Zhao, Mouxiang Chen +5

1The Australian National University · 2Shanghai AI Laboratory · 3Zhejiang University +2

Vision-Language Models Multimodal Large Language Models

Ask PDF