Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Xinye Zhao; Yunkai Dang; Yunchen Wu; Wenbin Li

cs.CVOct 1, 2026

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Authors: Xinye Zhao, Yunkai Dang, Yunchen Wu, Wenbin Li

Organizations: School of Intelligence Science and Technology, Nanjing University

Abstract

Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.

Figures & tables

Figure 1: Fixed-budget VLM inference.

Figure 2: Motivation. Left: Language backbone size ( $N$ ) and visual tokens ( $R$ ) both consume inference compute, which raises the question of how to split a fixed budget between them. Right: Scaling helps many questions but leaves others wrong at every scale we test.

Figure 3: Fit quality and form comparison for the Separable Law. Left: predicted vs. observed error, one dot per

(N,R)

cell, with QwenVL in blue, InternVL in orange, and the line

y{=}x

. Middle: residual vs. observed error, with mean

\pm

standard error of the mean (SEM) in each of five error bins. Right: candidate forms in

\Delta

-space relative to the separable form, marked by a charcoal star at the origin. The axes show

\Delta

SSE % and

\Delta

BIC, and

k

is the nominal number of free parameters.

Task type	Mean $e_{\infty}$	Median $e_{\infty}$	Floor	Saturation
Perception	0.073	0.001	72%	20%
Reasoning	0.155	0.001	53%	33%

Table 1: Results of per-task

e_{\infty}

across 40 fits.

Figure 4: Question-level scaling heterogeneity and responses by domain and skill. Left: Mean probability of the correct option versus its variability across $(N,R)$ configurations. Middle: Probability gain $\bar{p}_{Q_{4}}-\bar{p}_{Q_{1}}$ versus its low-compute value $\bar{p}_{Q_{1}}$ . Right: Accuracy across compute quartiles, with one panel per visual domain and one curve per skill.

Domain $D$		Skill $S$
D1	Documents, Charts, Slides	S1	Attribute Recognition
D2	Aerial, Satellite	S2	Text Reading
D3	Vehicles, Driving	S3	Counting
D4	Indoor	S4	Spatial Relation
D5	Outdoor	S5	Object Identification
D6	People, Surveillance	S6	Scene Reasoning

Table 2: Unified

6\times 6

domain and skill taxonomy.

Figure 5: Visual frontends expose different token ranges and scaling exponents. (a) InternVL uses discrete tiling, whereas QwenVL uses continuous resizing. (b) Median exponents reverse, with

\beta>\alpha

for InternVL and

\alpha>\beta

for QwenVL. (c) Task-level fits in the

(\alpha,\beta)

plane, where filled markers denote interior estimates and hollow markers denote boundary-clipped ones.

Figure 6: Allocation and benchmark diagnostics. (a) VPE trajectories

(N^{*},R^{*})

for one series per family, arrows marking increasing budget. (b) Mean (solid) and p90 (hatched) regret under joint holdout. (c) Regret CDFs for projection and direct selection. (d) Per benchmark and family, the Scaling Bound share against the share of those questions responding more to

N

than to

R

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Version	Input strategy	Visual token mechanism	Effect on visual token count
InternVL 1.0	Fixed input size	ViT features projected to the language backbone	Fixed count
InternVL 1.5	Dynamic tiling	Up to 40 tiles of $448{\times}448$	Changes in whole tiles
InternVL2.5	Dynamic tiling	Pixel unshuffle to 256 tokens per tile	Changes in whole tiles
InternVL3	Dynamic tiling	256 tokens per tile, V2PE positions	Changes in whole tiles
InternVL3.5	Dynamic tiling	256 tokens per tile, ViR only in Flash variants	Changes in whole tiles

Appendix

Table 3: Evolution of the InternVL series, focusing on how each version turns an image into tokens.

Version	Input strategy	Visual token mechanism	Effect on visual token count
Qwen-VL	Fixed $448{\times}448$ input	Cross-attention adapter to a fixed number of tokens	Fixed count
Qwen2-VL	Dynamic resolution	2D-RoPE and $2{\times}2$ MLP patch merger	Changes with input size in small steps
Qwen2.5-VL	Dynamic resolution	Window attention, same patch merger	Changes with input size in small steps
Qwen3-VL	Dynamic resolution	SigLIP-2, DeepStack, same patch merger	Changes with input size in small steps

Appendix

Table 4: Evolution of the QwenVL series, focusing on how each version turns an image into tokens.

Family	Series	Evaluated sizes	Visual frontend
InternVL	InternVL2.5	1B, 2B, 4B, 8B, 26B, 38B	Dynamic tiling
InternVL	InternVL3	1B, 2B, 8B, 9B, 14B, 38B	Dynamic tiling
InternVL	InternVL3.5	1B, 2B, 4B, 8B, 14B, 38B	Dynamic tiling
QwenVL	Qwen2.5-VL	3B, 7B, 32B, 72B	Dynamic resolution
QwenVL	Qwen3-VL	2B, 4B, 8B, 32B	Dynamic resolution

Appendix

Table 5: Evaluated model series and their visual frontends, 26 models in total. Sizes follow the model names, and

N

is computed as described above.

Benchmark	Native size ( $W\times H$ )	Input sizes (long edge, px)
HR-Bench-4K	$\approx 4023\times 3503$	$224,336,512,768,1024,2048$
HR-Bench-8K	$\approx 5727\times 4430$	$4096$
MME-RealWorld	$\approx 2000\times 1500$	$224,336,512,768,1024,2048$
TreeBench	$\approx 2152\times 1615$	$224,336,512,768,1024,2048$
V * Bench	$\approx 2246\times 1582$	$224,336,512,768,1024,2048$

Appendix

Table 6: Native image sizes and evaluated input sizes.

Group		Separable-law parameters					Fit diagnostics
Series	Benchmark	$e_{\infty}$	$A_{n}$	$\alpha$	$B_{r}$	$\beta$	SSE	$n$	$R^{2}$
InternVL2.5	HR-Bench	0.001	0.308	0.149	1.24	4.66	0.031	42	0.92
InternVL2.5	MME-RealWorld	0.001	0.527	0.058	11.41	$10.0^{\dagger}$	0.043	36	0.77
InternVL2.5	TreeBench	0.001	0.582	0.057	9.65	$10.0^{\dagger}$	0.030	36	0.73
InternVL2.5	V * Bench	0.001	0.089	0.381	56.31	$10.0^{\dagger}$	0.203	36	0.59
InternVL3	HR-Bench	0.001	0.262	0.183	1.05	3.87	0.025	42	0.94

Appendix

Table 7: Per-group Separable-law point estimates and fit diagnostics. Amplitudes

A_{n}

and

B_{r}

are evaluated at the common reference point

\widetilde{N}=6.60

B and

\widetilde{R}=1{,}135

. † and ‡ mark

\beta

and

\alpha

, respectively, at the upper bound of

10

;

n

is the cell count.

Figure 7: Predicted versus observed error for each group. Rows correspond to the five model series and columns to the four benchmarks. Each point is one

(N,R)

cell, all panels share

[0,0.82]^{2}

axes, and the dashed line marks perfect agreement,

y=x

. Orange denotes InternVL and blue denotes QwenVL; each panel reports its

R^{2}

and cell count

n

. † marks groups whose fitted

\beta

reaches the search upper bound of

10

. The three InternVL groups on V * Bench have the lowest

R^{2}

values, consistent with their narrow visual-token ranges.

Figure 8: Separable-law fits across model series and benchmarks. Rows show model series and columns show benchmarks; each panel plots observed error against

N

on a logarithmic scale. Points are

(N,R)

cells, and curves are predictions at the median

R

of each visual-token quartile; color denotes family and shade or marker denotes quartile. † marks

\beta=10

; Table 7 reports the parameter estimates.

Figure 9: Residual diagnostics for the per-group Separable-law fits. Left: Distribution of all

650

residuals (observed minus predicted), with zero marked by the dashed line and a matched Gaussian shown for reference. Right: Residual standard deviation by model series; the dashed line gives the pooled value. Residuals are centered near zero, but InternVL shows greater dispersion than QwenVL.

Form	$k$	SSE	BIC	$\Delta$ SSE	$\Delta$ BIC
Separable	62	1.221	$-1834.3$	—	—
Multiplicative	42	1.870	$-1686.4$	$+53.2\%$	$+147.8$
Coupled	83	1.182	$-1719.0$	$-3.1\%$	$+115.2$
CES	42	2.123	$-1604.0$	$+73.9\%$	$+230.3$
$p$ -norm	63	1.913	$-1535.6$	$+56.8\%$	$+298.7$

Appendix

Table 8: Comparison of functional forms on the same 650 cells, with exponents and shape parameters shared across groups.

k

counts the parameters of each fitted law.

Form	Pooled SSE	Median group $R^{2}$	Groups with lower SSE
Separable	1.088	0.892	19/20
Multiplicative	1.254	0.840	1/20

Appendix

Table 9: Separable and multiplicative laws with exponents specific to each group.

Objective	$\Delta Q$	Relative $\Delta Q$ (%)	$\Delta\mathrm{BIC}$
Unweighted	$-0.0384$ $(-0.0823,-0.0277)$	$-3.15$ $(-6.27,-1.97)$	$\mathbf{115.23}$ $(93.95,123.10)$
Weighted by precision	$-0.0435$ $(-0.0758,-0.0314)$	$-4.38$ $(-6.88,-2.90)$	$\mathbf{106.91}$ $(89.65,116.90)$

Appendix

Table 10: Paired image-bootstrap comparison with shared exponents (

500

replicates). Cells give point estimates and

95\%

percentile intervals; differences are Coupled minus Separable.

Q

denotes unweighted or precision-weighted SSE, and positive

\Delta\mathrm{BIC}

favors Separable.

Exponents	Held-out axis	Separable	Coupled	$\Delta$ RMSE (pp)
All cells
Shared	Backbone size	$0.0655$	$0.0579$	$-0.755$ $(-1.668,2.940)$
Shared	Input size	$0.0763$	$0.0891$	$1.283$ $(-0.412,2.388)$
Per-group	Backbone size	$0.0577$	$0.0781$	$\mathbf{2.040}$ $(0.488,5.111)$
Per-group	Input size	$0.0601$	$0.0599$	$\mathbf{-0.028}$ $(-0.182,0.044)$
Interior levels

Appendix

Table 11: Held-out RMSE over

200

paired image bootstraps. Differences are Coupled minus Separable, with

95\%

percentile intervals below; positive values favor Separable. All-cell predictions are clipped to

[0,1]

, whereas interior predictions are not.

Figure 10: Bootstrap comparison of Separable and Coupled laws. Left and middle show bootstrap objective and BIC differences for unweighted (blue) and precision-weighted (orange) fits; right shows held-out RMSE differences with

95\%

intervals. All differences are Coupled minus Separable: negative objective or RMSE differences favor Coupled, whereas positive BIC differences favor Separable.

Held-out design	Eligible decisions	Mean regret (pp)		Same choice (%)
		Separable	Bounded
Backbone holdout	297	0.48	0.48	100.0
Input-size holdout	411	0.66	0.67	99.3
Joint holdout	2,379	0.76	0.75	98.3
Joint holdout, two-axis tradeoff	1,284	0.88	0.87	97.1

Appendix

Table 12: Configuration selection is stable under bounded outputs. Mean regret is reported in percentage points (pp), and agreement is the share of decisions for which the two laws select the same configuration. Each decision includes at least two feasible candidates.

Family	Law	Median $\alpha$	Median $\beta$	$\alpha=0.05$	$\beta=0.05$	$\beta=10$
InternVL	Separable	0.196	10.000	0	0	9
InternVL	Bounded	0.113	10.000	5	0	9
QwenVL	Separable	0.779	0.174	1	0	0
QwenVL	Bounded	0.664	0.074	2	3	0

Appendix

Table 13: Median exponents and numbers of groups at the search bounds.

		Held-out RMSE (pp)		Predictions outside $[0,1]$
Design	Held-out level	Separable	Bounded	Separable	Bounded
Backbone size	Interior	4.47	4.43	0	0
Backbone size	Endpoint	7.38	7.18	6	0
Input size	Interior	4.56	4.45	0	0
Input size	Endpoint	8.29	11.79	0	0
Joint holdout	Interior	4.59	4.54	0	0

Appendix

Table 14: Predictive stability under bounded outputs. Held-out RMSE is reported in percentage points (pp), together with the number of predictions outside

[0,1]

. Separable predictions are clipped only when computing RMSE; neither law clips predictions during configuration selection. Highlighted cells mark the lower RMSE or the bounded law’s removal of invalid predictions.

Series	Benchmark	$95\%$ bootstrap interval			$\beta$ at search bound (%)
		$\alpha$	$\beta$	$e_{\infty}$
InternVL2.5	HR-Bench	$[0.106,0.332]$	$[1.654,10.000]$	$[0.001,0.140]$	3.2
	MME-RealWorld	$[0.050,0.068]$	$[10.000,10.000]$	$[0.001,0.020]$	100.0
	TreeBench	$[0.050,0.077]$	$[6.695,10.000]$	$[0.001,0.140]$	95.4
	V * Bench	$[0.201,0.651]$	$[10.000,10.000]$	$[0.001,0.030]$	99.0
InternVL3	HR-Bench	$[0.135,0.601]$	$[1.310,9.660]$	$[0.001,0.200]$	1.8

Appendix

Table 15: Bootstrap uncertainty of the Separable-law parameters. Entries report

95\%

percentile intervals from

500

paired image bootstraps and the share of refits in which

\beta

reaches either search bound. Intervals are conditional on the original exponent box

[0.05,10]

; rates of at least

90\%

are highlighted.

Series	Benchmark	$\alpha$	$\beta$	$e_{\infty}$	$R^{2}$
InternVL2.5	HR-Bench	0.17	$3.00^{\dagger}$	0.001	0.91
InternVL2.5	MME-RealWorld	0.07	$3.00^{\dagger}$	0.001	0.69
InternVL2.5	TreeBench	0.06	$3.00^{\dagger}$	0.001	0.72
InternVL2.5	V * Bench	0.49	$3.00^{\dagger}$	0.001	0.44
InternVL3	HR-Bench	0.21	$3.00^{\dagger}$	0.001	0.94
InternVL3	MME-RealWorld	0.22	$3.00^{\dagger}$	0.300	0.65

Appendix

Table 16: Fits under the narrower exponent box

[0.05,3]

. † and ‡ mark upper-bound

\beta

and

\alpha

; pooled SSE is

1.400

Parameterization	Pooled RMSE (pp)	$\beta$ at upper bound	$\alpha>\beta$
Cell mean	$4.092$	$9$	$7$
Question moment	$3.077$	$0$	$7$

Appendix

Table 17: Effect of visual-token aggregation on fit quality and exponent ordering.

Series	Perception	Reasoning	Gap
InternVL2.5	0.001	0.067	$+0.066$
InternVL3	0.097	0.187	$+0.090$
InternVL3.5	0.180	0.194	$+0.013$
Qwen2.5-VL	0.001	0.187	$+0.186$
Qwen3-VL	0.085	0.141	$+0.056$
Pooled	0.073	0.155	$+0.082$

Appendix

Table 18: Mean fitted

e_{\infty}

by model series and by the perception and reasoning labels, pooling the fits of § 3.1 . The gap between reasoning and perception is positive in all five series. Gaps are computed from the unrounded means and rounded on their own.

Series	Benchmark	Task	Exponent
			$\alpha$	$\beta$
InternVL2.5	HR-Bench	Cross	0.099	5.18
	HR-Bench	Single	0.221	9.50
	MME-RealWorld	Perception	0.068	$10.0^{\dagger}$
	MME-RealWorld	Reasoning	0.065	7.05
	TreeBench	Perception	0.115	$10.0^{\dagger}$

Appendix

Table 19: Task-level Separable-law exponents. The table covers all

40

series–benchmark–task fits; † and ‡ mark the upper and lower exponent bounds,

10

and

0.05

. Interior values are point estimates and need not be statistically identified.

	$\tau_{\text{easy}}=0.85$	$\tau_{\text{easy}}=0.90$	$\tau_{\text{easy}}=0.95$
$\tau_{\text{cog}}=0.15$	22.3 / 52.5 / 25.2	20.0 / 54.8 / 25.2	17.4 / 57.4 / 25.2
$\tau_{\text{cog}}=0.20$	22.3 / 47.0 / 30.7	20.0 / 49.4 / 30.7	17.4 / 51.9 / 30.7
$\tau_{\text{cog}}=0.25$	22.3 / 44.4 / 33.2	20.0 / 46.8 / 33.2	17.4 / 49.3 / 33.2

Appendix

Table 20: Share of trajectories in each regime (%) under the cutoff sweep, written as Easy / Scaling Bound / Ceiling Bound . The default cutoffs,

\tau_{\text{easy}}=0.90

and

\tau_{\text{cog}}=0.20

, are in bold.

	Ceiling Bound		Scaling Bound		Easy
Series	Accuracy	Slope	Accuracy	Slope	Accuracy	Slope
InternVL2.5	$7.08$	$0.30$	$53.40$	$4.55$	$96.56$	$1.01$
InternVL3	$7.62$	$0.13$	$54.79$	$4.36$	$96.48$	$0.80$
InternVL3.5	$8.21$	$0.60$	$52.82$	$3.82$	$95.35$	$1.47$
Qwen2.5-VL	$7.50$	$0.58$	$50.93$	$3.13$	$93.28$	$1.22$
Qwen3-VL	$8.85$	$1.24$	$50.53$	$4.49$	$94.74$	$0.62$

Appendix

Table 21: Held-out performance after defining the regimes on the other half of the configurations. Accuracy is in percent; slopes are percentage points per doubling of

NR

on the held-out half. Light-blue cells mark the held-out Ceiling Bound results.

Criterion	$a$	$p$	$a^{*}$	$p^{*}$
Low $Q_{4}$ performance $H_{z}$	37.5	30.6	39.1	40.0
Small endpoint change $E_{z}$	20.8	13.1	20.8	13.0
Small late change $P_{z}$	23.5	18.3	23.4	18.1
Small fitted slope $B_{z}$	19.0	14.1	18.3	13.9
All three together $J_{z}$	16.4	9.5	16.3	8.2

Appendix

Table 22: Share of the same

15{,}830

trajectories selected by each criterion (%). Every row requires low

Q_{4}

performance. The raw columns use

\tau=0.20

and the normalized columns

\tau=0.10

, with

\delta=0.05

and

\eta=0.01

per doubling. The Ceiling Bound rule selects

34.1\%

of this cohort, and the

30.7\%

of § 3.2 uses the full cohort instead.

Group	Trajectories	Ceiling Bound	$J_{p}$	$J_{p^{*}}$
HR-Bench	4,000	23.0	4.5	3.9
MME-RealWorld	8,900	39.3	12.4	10.8
TreeBench	1,975	40.8	9.7	8.3
V * Bench	955	17.6	2.4	1.6
Perception	10,525	30.0	7.9	6.7
Reasoning	5,305	42.1	12.7	11.1

Appendix

Table 23: Benchmarks and tasks compared on the same observations (%). The benchmark rows and the task rows are two separate partitions of the same

15{,}830

trajectories. HR-Bench covers only the six shared input sizes, since the trajectories that exist only at

4096

px are excluded. The Ceiling Bound column uses mean accuracy below

0.20

, and

J_{p}

and

J_{p^{*}}

follow Eq.( 18 ).

Skill marginals			Domain marginals
Skill	Mean $e_{\infty}$	Pairings	Domain	Mean $e_{\infty}$	Pairings
S1 Attribute Recognition	0.432	5	D1 Documents/Charts/Slides	0.041	4
S2 Text Reading	0.001	6	D2 Aerial/Satellite	0.227	6
S3 Counting	0.250	6	D3 Vehicles/Driving	0.204	6
S4 Spatial Relation	0.213	5	D4 Indoor	0.124	6
S5 Object Identification	0.044	6	D5 Outdoor	0.214	6

Appendix

Table 24: Marginal fitted floors by skill and domain. Each mean equally weights the supported domain–skill pairings; Pairings gives their number.

Pairing	Domain $\times$ Skill	$e_{\infty}$
D2 $\times$ S6	Aerial/Satellite $\times$ Scene Reasoning	0.780
D6 $\times$ S4	People/Surveillance $\times$ Spatial Relation	0.580
D3 $\times$ S3	Vehicles/Driving $\times$ Counting	0.540
D5 $\times$ S1	Outdoor $\times$ Attribute Recognition	0.540
D2 $\times$ S1	Aerial/Satellite $\times$ Attribute Recognition	0.520
D5 $\times$ S4	Outdoor $\times$ Spatial Relation	0.480

Appendix

Table 25: The ten domain–skill pairings with a fitted floor of at least

0.30

, under the taxonomy of Table 2 and the protocol above.

Resampling	Complete	$95\%$ percentile interval			$V_{S}>V_{D}$ (% replicates)
Resampling	Complete	$V_{S}$ (%)	$V_{D}$ (%)	$V_{S}/V_{D}$	$V_{S}>V_{D}$ (% replicates)
Ordinary cluster	$397/500$	$10.60$ – $40.89$	$6.25$ – $29.07$	$0.60$ – $4.27$	$82.6$
Positive-weight cluster	$500/500$	$11.94$ – $39.13$	$6.28$ – $28.78$	$0.57$ – $4.65$	$81.6$

Appendix

Table 26: Image-cluster bootstrap stability of the domain–skill variance decomposition. Intervals are the

2.5

th–

97.5

th percentiles; ordinary-bootstrap intervals use the 397 replicates retaining all 33 pairings.

Series	Backbone capacity exponent $\alpha$			Interior $\beta$
Series	Median	Range	At bound	Interior $\beta$
InternVL
InternVL2.5	0.10	$0.057$ – $0.381$	0	4.66
InternVL3	0.19	$0.071$ – $1.262$	0	3.87
InternVL3.5	0.44	$0.145$ – $1.203$	0	4.93
QwenVL

Appendix

Table 27: Group-level exponents by series, with four benchmark fits each. Interior means strictly inside

[0.05,10]

. For

\beta

, InternVL reports its sole interior HR-Bench estimate and QwenVL reports the median of four interior estimates.

Figure 11: Effective pixels as an alternative visual scaling measure on V * Bench. In-sample

R^{2}

for fits using

(N,R)

(N,P_{\mathrm{eff}})

, and

(N,R,P_{\mathrm{eff}})

under the same constraints. Effective pixels substantially improve the InternVL fits but not the Qwen3-VL fit. For InternVL, adding

R

after

P_{\mathrm{eff}}

yields almost no further increase in

R^{2}

; together with boundary estimates or zero amplitudes, this means that the three-variable fits do not statistically separate their contributions.

Series	Benchmark	Questions	Gain	$95\%$ interval	Holm $p$
InternVL3.5-8B	HR-Bench	$100$	$14.00$	$[4.00,24.00]$	$0.0692$
InternVL3.5-8B	MME-RealWorld	$100$	$7.00$	$[-1.00,15.00]$	$0.225$
InternVL3.5-8B	TreeBench	$100$	$8.00$	$[1.00,16.00]$	$0.225$
InternVL3.5-8B	V * Bench	$191$	$30.37$	$[23.56,37.17]$	$2.62{\times}10^{-14}$
Qwen3-VL-8B	HR-Bench	$100$	$30.00$	$[19.00,41.00]$	$3.21{\times}10^{-6}$
Qwen3-VL-8B	MME-RealWorld	$100$	$17.00$	$[8.00,26.00]$	$0.00554$

Appendix

Table 28: Endpoint gains in the native-source visual-token sweeps. Gains and intervals are in percentage points. Light-blue rows mark gains with Holm-adjusted

p<0.05

Figure 12: Visual-token sweeps on native source images. Accuracy against the mean actual visual-token count for InternVL3.5-8B and Qwen3-VL-8B on the four benchmarks. The two frontends expose different token ranges, so differences between their endpoint gains do not by themselves isolate an architectural effect.

Figure 13: Source-image information quality modulates the return to visual tokens. InternVL3.5-8B accuracy under native,

1024

-pixel, and

224

-pixel source-quality conditions crossed with seven tile limits. Actual visual-token counts and input-sequence hashes match across quality conditions for every combination of question and cap, so the curves vary pixel information while holding the input sequence structure fixed.

		FLOPs of the language backbone				Latency
Scope	$n$	$\kappa$	$p$	$q$	$R^{2}$	$p$	$q$	$R^{2}$
Pooled	650	$1.24{\times}10^{10}$	1.00	0.77	0.995	0.07	0.54	0.366
InternVL2.5	150	$2.95{\times}10^{9}$	1.00	0.96	1.000	-0.04	-0.15	0.019
InternVL3	150	$2.96{\times}10^{9}$	1.00	0.96	1.000	0.14	-0.53	0.029
InternVL3.5	150	$2.93{\times}10^{9}$	1.00	0.96	1.000	0.24	1.04	0.150
Qwen2.5-VL	100	$1.38{\times}10^{10}$	1.00	0.75	0.986	0.03	0.43	0.764

Appendix

Table 29: Cost-law fits

C=\kappa N^{p}R^{q}

, with

N

in billions of language backbone parameters and

R

in visual tokens. The law is fitted to the FLOPs measure and, separately, to wall-clock latency, and

R^{2}

is computed in log space. Only the pooled fit of the FLOPs measure sets budgets, and the fits for each series and for latency are diagnostic.

Rule	Mean	90th pct.	Worst
Direct selection	0.76	2.74	19.37
Projection	2.33	7.71	34.55
Largest token count	4.09	12.23	27.00
Largest backbone	7.21	20.14	42.93
Log-center	7.56	16.22	38.74

Appendix

Table 30: Allocation regret in percentage points over the

2{,}379

joint-holdout decisions. All five rules see the same candidates under the same budgets, the

90

th percentile uses linear interpolation, and Worst is the largest regret of a single decision. The two rules above the horizontal line read the fitted Separable law, and the three below it do not.

Rule	Mean	Median	90th pct.	Worst	Exact hit
Projection	1.98	0.00	6.34	11.52	51.25%
Largest token count	6.51	6.10	13.45	19.37	27.50%
Log-center	8.05	7.70	16.24	20.00	1.25%
Largest backbone	15.40	14.23	28.32	41.36	0.00%

Appendix

Table 31: In-sample allocation regret in percentage points over the same 80 decisions, for projection and the three static rules. Exact hit counts the decisions that select the designated best cell, with ties broken by dataset order.

Benchmark	Projection	Largest token count	Log-center	Largest backbone
HR-Bench	0.00	5.63	11.06	14.23
MME-RealWorld	0.07	3.54	6.52	17.77
TreeBench	2.49	6.09	4.10	4.85
V * Bench	0.00	7.85	9.16	27.23

Appendix

Table 32: Median allocation regret in percentage points by benchmark, 20 decisions each, from the same evaluation. Each entry pools four budgets across five model series.

Threshold	Pooled	InternVL	QwenVL
$\leq 0$ pp	52.50%	52.08%	53.13%
$\leq 1$ pp	58.75%	58.33%	59.38%
$\leq 2$ pp	65.00%	64.58%	65.63%
$\leq 5$ pp	85.00%	81.25%	90.63%
$\leq 8$ pp	96.25%	93.75%	100.00%
$\leq 10$ pp	98.75%	97.92%	100.00%

Appendix

Table 33: Share of projection decisions with regret at or below each threshold. The denominators are 80 decisions pooled, 48 for InternVL, and 32 for QwenVL. The zero-regret row counts every tied minimum, whichever cell was chosen.

Series	Benchmark	Mean	Median	90th pct.	Worst	Hits
InternVL2.5	HR-Bench	2.16	0.00	6.04	8.63	3/4
InternVL2.5	MME-RealWorld	3.44	2.02	8.03	9.74	2/4
InternVL2.5	TreeBench	3.11	2.74	6.07	6.97	1/4
InternVL2.5	V * Bench	4.19	2.62	9.63	11.52	2/4
InternVL3	HR-Bench	0.44	0.00	1.23	1.76	3/4
InternVL3	MME-RealWorld	2.16	0.68	5.51	7.30	2/4

Appendix

Table 34: Projection regret in percentage points by series and benchmark, from the same 80 decisions as Table 31 . Each row summarizes four decisions, computed from unrounded regrets, and the

90

th percentile uses linear interpolation. Hits count the selections of the designated best cell, with ties broken by dataset order. For InternVL3 on V * Bench, all four decisions reach the lowest feasible error and one of them selects a different cell tied with the designated best.

Benchmark	Easy	Scaling Bound	Ceiling Bound
HR-Bench	33.5%	46.5%	20.0%
MME-RealWorld	10.3%	50.3%	39.4%
TreeBench	10.4%	48.6%	41.0%
V * Bench	16.2%	66.2%	17.6%

Appendix

Table 35: Regime shares for each benchmark over the

19{,}890

trajectories, under the accuracy cutoffs of § 3.2 . A separate response-based analysis uses a smaller matched cohort and stricter criteria.

		Regime share			Dominant axis within Scaling Bound
Family	Benchmark	Easy	Scaling	Ceiling	$N$ -dominated	$R$ -dominated	Tie
InternVL	HR-Bench	29.0%	49.6%	21.4%	89.7%	6.7%	3.6%
InternVL	MME-RealWorld	10.1%	51.8%	38.1%	70.5%	20.7%	8.8%
InternVL	TreeBench	8.6%	49.4%	42.0%	88.5%	7.0%	4.5%
InternVL	V * Bench	16.8%	63.9%	19.4%	47.8%	38.5%	13.7%
QwenVL	HR-Bench	40.3%	41.8%	17.9%	63.0%	31.6%	5.5%

Appendix

Table 36: Regime shares and dominant axis for each benchmark and family, over the same

19{,}890

trajectories. Regime shares are fractions of all trajectories of that family and benchmark, and the dominant axis is computed over the Scaling Bound trajectories only. The HR-Bench rows include the trajectories that exist only at

4096

px, which the text discusses.

Panel of Figure 14 — Figure 14: Sensitivity to input size across model sizes. Selected outputs of Qwen3-VL at 2B, 4B, 8B, and 32B and at input sizes from $224$ to $2048$ px. Green marks the correct option and red marks an incorrect answer. This page shows the 2B and 4B models. The 2B example moves in both directions; one 4B example becomes correct at $512$ px and the remote-sensing example at $768$ px.

Panel of Figure 15 — Figure 15: Sensitivity to input size for the 8B and 32B models. The 8B answer becomes correct at $1024$ px, and the two 32B answers become correct at $768$ and $512$ px.

Panel of Figure 16 — Figure 16: Sensitivity to model size across input sizes. Each panel compares the Qwen3-VL models on one question at a fixed input size. These three examples use $224$ , $336$ , and $512$ px, and the question differs from panel to panel.

Panel of Figure 17 — Figure 17: Model comparisons at $768$ , $1024$ , and $2048$ px. In these three examples the 8B and 32B models answer correctly while the 2B and 4B models do not.

Explore similar work

On Test-Time Scaling for Vision-Language Models
Jun 27, 2026Fawaz Sammani, Tzoulio Chamiti, Nikos DeligiannisTest-Time Scaling Recent Vision-Language Models
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
Jun 10, 2026Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8Fast Inference Efficient Inference
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1
The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
Jun 5, 2026Lujun Li, Lama Sleem, Niccolo Gentile +4Recent Vision-Language Models Fine-Grained Perception
University of Luxembourg · Foyer S.A. · Université Paris-Saclay

Jun 27, 2026, cs.CVJ/K move · Enter open · S save

On Test-Time Scaling for Vision-Language Models

Fawaz Sammani, Tzoulio Chamiti, Nikos Deligiannis

ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium

Test-Time Scaling Recent Vision-Language Models Scaling

Open paper AskPDF arXiv

Test-time scaling is a paradigm where large models use additional compute at inference to achieve better performance, without changing model weights. While it has been widely studied for Large Language Models (LLMs), its applicability to Large Vision-Language Models (LVLMs) remains less explored and analyzed, with limited analysis of whether, when, and to what extent these approaches transfer to LVLMs. In this work, we ask a simple but fundamental question: can conventional test-time scaling methods developed for LLMs be directly applied to LVLMs? We present the first comprehensive study of test-time scaling for LVLMs, spanning multiple models and model sizes, nine test-time scaling methods, and six diverse benchmarks. Our main findings is that 1) different from previous findings, small, well-performing models benefit the most from test-time scaling, enabling performance improvements of up to around 30%, reaching large models performance, and often outperforming them, 2) LVLMs lose focus when given more compute than necessary, and 3) Visual information is encoded early in the reasoning chain, after which the chain is dominated by text-only reasoning and the contribution of image tokens drops significantly. Finally, we also provide a global and fine-grained analysis on the quality and information sufficiency of the reasoning chains produced. Overall, our findings and analysis provide practical guidance and insights into LVLMs and their deployment in research and industry.

Abstract

Explore similar work

On Test-Time Scaling for Vision-Language Models

AVIS: Adaptive Test-Time Scaling for Vision-Language Models

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models