On the Limits of Scale: Inverse Scaling in Vision-Language Models

Rohit Saxena; Alessandro Suglia; Marek Rei; Pasquale Minervini

cs.CVOct 19, 2026

On the Limits of Scale: Inverse Scaling in Vision-Language Models

Authors: Rohit Saxena, Alessandro Suglia, Marek Rei, Pasquale Minervini

Organizations: School of Informatics, University of Edinburgh, United Kingdom · Imperial College London, United Kingdom · Miniml.AI, United Kingdom

Abstract

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.

Figures & tables

Figure 1: Scaling curves for four cross-family subtasks. Filled circles: dense models; open diamonds: MoE variants. All show non-monotonic behaviour across families.

Family	Models
Qwen3-VL	2B, 4B, 8B, 30B-A3B ∗ (3B),
	32B, 235B-A22B ∗ (22B)
InternVL3.5	1B, 2B, 4B, 8B, 14B,
	30B-A3B ∗ (3B), 38B,
	241B-A28B ∗ (28B)
Gemma 3	4B, 12B, 27B

Table 1: Left: Model families evaluated; ∗ MoE models with active parameters in parentheses. Right: Evaluation benchmarks; subtask counts reflect the category granularity used in our analysis. † ScienceQA is filtered to the 2,017 questions that include an image, excluding text-only questions.

Dataset	Qwen3-VL	InternVL	Gemma 3	Molmo 2	Total
MMT-Bench	31/480	49/896	6/96	1/32	87/1504
MMMU (dev_val+test)	34/1620	67/3024	8/324	6/108	115/5076
ScienceQA	7/450	0/840	10/90	0/30	17/1410
MME	10/210	23/392	0/42	2/14	35/658
AI2D	0/195	8/364	0/39	1/13	9/611
MathVista	1/75	2/140	0/15	0/5	3/235

Table 2: Pairwise inverse scaling cases by family and dataset. Each cell reports the number of detected cases over all eligible model-pair–subtask comparisons (

n\geq 20

shared questions); a case is a unique (subtask, smaller model, larger model) tuple meeting the thresholds in Section 4 .

#	Family	Dataset	Subtask	Small	Large	$n$	FI	FN	Net	Acc $\Delta$
1	Qwen3-VL	BLINK	Jigsaw	4B	30B-A3B	150	45	9	36	24.0
2	Qwen3-VL	BLINK	Jigsaw	4B	32B	150	44	11	33	22.0
3	Molmo 2	BLINK	Jigsaw	4B	8B	150	43	12	31	20.7
4	InternVL	BLINK	Forensic Detection	2B	30B-A3B	132	30	0	30	22.7
5	Qwen3-VL	BLINK	Multi-view Reasoning	8B	32B	133	38	8	30	22.6
6	InternVL	BLINK	Forensic Detection	4B	30B-A3B	132	29	0	29	22.0

Table 3: Top 10 pairwise cases ranked by Net (FlipInv

-

FlipNorm); Acc

\Delta

in pp.

Figure 2: Cross-family inverse scaling across subtasks. Each bar shows the number of pairwise inverse-scaling cases for a given model family on that subtask; the annotation shows the count and the strongest case’s Net flip. Right column: total cases and number of families. Only the 12 subtasks with the most cases are shown; all 13 appear in Table 36 .

Dataset	Qwen3-VL	InternVL	Gemma 3	Molmo 2	Total
MMT-Bench	10	20	6	1	37
MMMU (dev_val+test)	13	40	8	6	67
ScienceQA	4	0	10	0	14
MME	3	14	0	2	19
AI2D	0	1	0	1	2
MathVista	0	1	0	0	1

Table 4: Dense-only pairwise inverse scaling cases by family and dataset (MoE models excluded). Gemma 3 and Molmo 2 contain no MoE models, so their counts equal those of the full analysis.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Size	HuggingFace Identifier
Qwen3-VL	2B	Qwen/Qwen3-VL-2B-Instruct
	4B	Qwen/Qwen3-VL-4B-Instruct
	8B	Qwen/Qwen3-VL-8B-Instruct
	30B-A3B	Qwen/Qwen3-VL-30B-A3B-Instruct
	32B	Qwen/Qwen3-VL-32B-Instruct
	235B-A22B	Qwen/Qwen3-VL-235B-A22B-Instruct

Appendix

Table 5: HuggingFace model identifiers for all 19 evaluated models.

Dataset	2B	4B	8B	30B-A3B	32B	235B-A22B
MMT-Bench	60.2	67.3	68.5	69.7	71.7	73.3
MMMU dev_val	44.0	51.0	53.1	59.6	61.7	67.3
MMMU test	41.6	48.7	51.0	56.0	57.0	64.2
ScienceQA	87.3	92.6	94.4	95.7	96.2	98.0
MME	82.5	87.2	89.6	90.4	90.9	91.6
AI2D	76.6	83.4	85.1	87.3	87.9	89.5

Appendix

Table 6: Overall accuracy (%) for Qwen3-VL. Bold = best in family.

Dataset	1B	2B	4B	8B	14B	30B-A3B	38B	241B-A28B
MMT-Bench	53.1	58.9	65.2	67.6	68.4	67.3	71.9	72.6
MMMU dev_val	40.4	47.4	57.1	57.0	59.3	61.6	63.4	65.2
MMMU test	37.6	44.0	50.3	52.7	54.0	55.4	57.6	63.2
ScienceQA	84.1	91.7	97.3	97.5	98.3	99.2	98.8	99.5
MME	77.7	79.2	84.0	88.3	85.9	87.6	88.7	90.0
AI2D	70.5	78.0	82.4	83.3	84.5	86.4	87.1	87.1

Appendix

Table 7: Overall accuracy (%) for InternVL3.5. Bold = best in family.

Dataset	Gemma 3			Molmo 2
	4B	12B	27B	4B	8B
MMT-Bench	54.3	60.1	61.5	64.0	66.5
MMMU dev_val	40.7	47.9	54.8	50.5	52.9
MMMU test	39.3	47.4	50.6	46.3	48.0
ScienceQA	78.5	82.2	82.7	97.8	98.1
MME	79.1	83.4	85.3	76.8	79.3

Appendix

Table 8: Overall accuracy (%) for Gemma 3 and Molmo 2. Bold = best in family. Underline = largest model is not best.

Dataset	Subtask	Families	Max Net
BLINK	Multi-view Reasoning	4	30
BLINK	Jigsaw	3	33
MMMU test	Organic Chemistry	3	15
MME	Scene	3	14
MMMU test	Investment	3	13
BLINK	Object Localization	3	11

Appendix

Table 9: Dense-only cross-family replication: subtask–dataset pairs with inverse scaling in at least three of the four families using only dense model pairs.

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
autonomous_driving	30B-A3B	235B-A22B	100	23.0	23	0	23
autonomous_driving	2B	235B-A22B	100	19.0	23	4	19
autonomous_driving	8B	235B-A22B	100	17.0	18	1	17
autonomous_driving	32B	235B-A22B	100	16.0	17	1	16
autonomous_driving	4B	235B-A22B	100	14.0	16	2	14
cross_image_matching	8B	30B-A3B	60	23.3	21	7	14

Appendix

Table 12

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Surveying and Mapping	8B	30B-A3B	412	4.1	63	46	17
Electrical Circuit	30B-A3B	32B	128	13.3	24	7	17
Cost Management	30B-A3B	32B	58	24.1	17	3	14
Data Structure and Algorithm	4B	30B-A3B	92	12.0	18	7	11
Cost Management	30B-A3B	235B-A22B	58	19.0	15	4	11
Data Structure and Algorithm	8B	30B-A3B	92	9.8	17	8	9

Appendix

Table 13

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Velocity, acceleration, and forces	30B-A3B	32B	62	14.5	12	3	9
The Americas: geography	2B	4B	45	17.8	14	6	8
Velocity, acceleration, and forces	8B	32B	62	12.9	11	3	8
Velocity, acceleration, and forces	4B	32B	62	9.7	8	2	6
The Americas: geography	30B-A3B	32B	45	13.3	7	1	6
Ecosystems	4B	8B	73	6.8	6	1	5

Appendix

Table 14

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
scene	2B	235B-A22B	400	2.8	20	9	11
scene	32B	235B-A22B	400	2.8	18	7	11
scene	30B-A3B	235B-A22B	400	2.5	14	4	10
scene	4B	235B-A22B	400	1.5	14	8	6
celebrity	8B	32B	340	1.8	20	14	6
scene	8B	235B-A22B	400	1.5	13	7	6

Appendix

Table 15

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
math word problem	8B	30B-A3B	186	3.8	15	8	7

Appendix

Table 16

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Jigsaw	4B	30B-A3B	150	24.0	45	9	36
Jigsaw	4B	32B	150	22.0	44	11	33
Multi-view Reasoning	8B	32B	133	22.6	38	8	30
Multi-view Reasoning	2B	32B	133	21.8	37	8	29
Object Localization	4B	30B-A3B	122	18.0	35	13	22
Multi-view Reasoning	4B	32B	133	16.5	37	15	22

Appendix

Table 17

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
coarse_perception	30B-A3B	32B	1109	1.1	38	26	12
attribute_reasoning	32B	235B-A22B	699	0.9	19	13	6

Appendix

Table 18

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
cross_image_matching	14B	30B-A3B	60	31.7	21	2	19
localization	8B	30B-A3B	169	9.5	29	13	16
localization	14B	30B-A3B	169	9.5	31	15	16
image_retrieval	14B	30B-A3B	140	10.0	22	8	14
image_retrieval	38B	241B-A28B	140	10.0	19	5	14
temporal_understanding	38B	241B-A28B	99	13.1	20	7	13

Appendix

Table 19

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Mechanics of Materials	38B	241B-A28B	29	34.5	10	0	10
Mechanics of Materials	30B-A3B	241B-A28B	29	31.0	9	0	9
Music	8B	14B	35	22.9	11	3	8
Market Research	4B	14B	35	20.0	9	2	7
Mechanics of Materials	14B	241B-A28B	29	24.1	8	1	7
Mechanics of Materials	2B	241B-A28B	29	20.7	10	4	6

Appendix

Table 20

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Organic Chemistry	8B	14B	322	4.7	52	37	15
Engineering Graphics	4B	14B	89	15.7	22	8	14
Music	38B	241B-A28B	334	3.9	63	50	13
Organic Chemistry	4B	14B	322	3.7	51	39	12
Engineering Graphics	4B	38B	89	13.5	28	16	12
Genetics	14B	30B-A3B	57	21.1	15	3	12

Appendix

Table 21

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
landmark	1B	2B	400	7.2	55	26	29
celebrity	8B	14B	340	7.4	31	6	25
landmark	8B	14B	400	5.0	34	14	20
scene	8B	241B-A28B	400	4.0	20	4	16
landmark	8B	30B-A3B	400	3.8	35	20	15
celebrity	8B	30B-A3B	340	4.1	31	17	14

Appendix

Table 22

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
moonPhaseEquinox	30B-A3B	38B	277	5.8	29	13	16
moonPhaseEquinox	30B-A3B	241B-A28B	277	3.2	20	11	9
photosynthesisRespiration	14B	38B	79	7.6	8	2	6
rockCycle	38B	241B-A28B	67	9.0	9	3	6
partsOfA	14B	30B-A3B	487	1.0	33	28	5
photosynthesisRespiration	14B	30B-A3B	79	6.3	5	0	5

Appendix

Table 23

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
geometry problem solving	8B	14B	208	4.3	33	24	9
visual question answering	38B	241B-A28B	179	3.4	20	14	6

Appendix

Table 24

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Forensic Detection	2B	30B-A3B	132	22.7	30	0	30
Forensic Detection	4B	30B-A3B	132	22.0	29	0	29
Forensic Detection	14B	30B-A3B	132	22.0	29	0	29
Forensic Detection	8B	30B-A3B	132	21.2	28	0	28
Visual Correspondence	14B	30B-A3B	172	9.9	43	26	17
Multi-view Reasoning	1B	4B	133	12.0	52	36	16

Appendix

Table 25

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
coarse_perception	2B	8B	1109	1.9	49	28	21
attribute_reasoning	8B	14B	699	2.6	35	17	18
coarse_perception	2B	4B	1109	1.5	42	25	17
attribute_reasoning	4B	14B	699	2.3	29	13	16
attribute_reasoning	2B	14B	699	2.1	29	14	15
coarse_perception	30B-A3B	241B-A28B	1109	1.1	27	15	12

Appendix

Table 26

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
visual_recognition	12b	27b	658	2.3	37	22	15
autonomous_driving	12b	27b	100	8.0	14	6	8
medical_understanding	4b	27b	100	7.0	20	13	7
action_recognition	4b	12b	100	6.0	12	6	6
relation_reasoning	4b	12b	80	6.2	13	8	5
threed	12b	27b	40	12.5	8	3	5

Appendix

Table 27

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Investment	4b	12b	119	10.9	31	18	13
Fluid Mechanics	12b	27b	225	4.4	49	39	10
Corporate Finance	12b	27b	131	6.9	25	16	9
Pathology	12b	27b	33	21.2	7	0	7
Physiology	4b	12b	73	8.2	16	10	6
Management Models	12b	27b	72	8.3	12	6	6

Appendix

Table 28

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Basic economic principles	4b	27b	71	39.4	31	3	28
Particle motion and energy	12b	27b	41	43.9	18	0	18
Basic economic principles	12b	27b	71	23.9	21	4	17
Particle motion and energy	4b	27b	41	34.1	19	5	14
Magnets	12b	27b	110	11.8	31	18	13
Basic economic principles	4b	12b	71	15.5	14	3	11

Appendix

Table 29

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Multi-view Reasoning	4b	27b	133	5.3	9	2	7
Multi-view Reasoning	12b	27b	133	5.3	14	7	7

Appendix

Table 30

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
emotion	4B	8B	120	5.0	11	5	6

Appendix

Table 31

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Materials Science	4B	8B	139	11.5	24	8	16
Music	4B	8B	334	4.2	54	40	14
Organic Chemistry	4B	8B	322	3.7	42	30	12
Thermodynamics	4B	8B	163	5.5	15	6	9
Mechanics of Materials	4B	8B	364	1.9	32	25	7
Microbiology	4B	8B	60	8.3	8	3	5

Appendix

Table 32

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
artwork	4B	8B	400	4.3	45	28	17
scene	4B	8B	400	3.2	21	8	13

Appendix

Table 33

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
foodChainsWebs	4B	8B	1162	0.6	36	29	7

Appendix

Table 34

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
Jigsaw	4B	8B	150	20.7	43	12	31
Visual Correspondence	4B	8B	172	9.3	29	13	16
Multi-view Reasoning	4B	8B	133	11.3	73	58	15
Art Style	4B	8B	117	10.3	21	9	12
Relative Reflectance	4B	8B	134	6.7	21	12	9
Semantic Correspondence	4B	8B	139	6.5	24	15	9

Appendix

Table 35

Subtask	Small	Large	$n$	Acc $\Delta$	FI	FN	Net
coarse_perception	4B	8B	1109	1.6	33	15	18
logic_reasoning	4B	8B	412	1.2	28	23	5

Appendix

Table 36

Net Range	Cases	Significant ( $p<0.05$ )	%	BH (cases)	BH (all)
5–7	183	19	10.4	0	0
8–10	76	19	25.0	3	0
11–15	68	49	72.1	16	0
16–20	32	30	93.8	20	4
$\geq$ 21	16	16	100.0	15	12
Total	375	133	35.5	54	16

Appendix

Table 35: McNemar’s exact test significance by Net range, uncorrected and after Benjamini–Hochberg (BH) correction at FDR

0.05

. “BH (cases)” corrects over the 375 reported cases; “BH (all)” corrects over all 10,434 eligible comparisons.

Dataset	Subtask	Families	Max Net	Cases
BLINK	Multi-view Reasoning	4	30	22
	Jigsaw	3	36	16
	Forensic Detection	3	30	13
	Object Localization	3	22	10
	Relative Reflectance	3	17	11
MMBench	Coarse Perception	3	21	8

Appendix

Table 36: Subtask–dataset pairs with pairwise inverse scaling in at least three of the four families. “Max Net” is the highest Net value across all model pairs and families for that subtask. “Cases” is the total number of pairwise cases.

Dataset	Subtask	Families	Max Net
MME	celebrity	InternVL, Qwen	25
MMT-Bench	cross_image_matching	InternVL, Qwen	19
MMBench	attribute_reasoning	InternVL, Qwen	18
MMMU test	Electrical Circuit	InternVL, Qwen	17
BLINK	IQ Test	InternVL, Qwen	17
MME	artwork	InternVL, Molmo 2	17

Appendix

Table 37: Subtask–dataset pairs with pairwise inverse scaling in exactly two families. Max Net is the strongest case across all model pairs in those families.

Net threshold	$\geq 5$	$\geq 10$	$\geq 15$	$\geq 20$	$\geq 25$	$\geq 30$
Pairwise cases	375	132	56	21	12	5
Unique subtask–dataset pairs	115	44	25	9	6	3
Cross-family ( $\geq$ 3 families)	13	4	2	0	0	0

Appendix

Table 38: Robustness to threshold choice: pairwise inverse scaling cases (

n\geq 20

) at different Net thresholds.

	Genuine Error	Bad Annot.	Format Mis.
Annotator 1	105 (82.7%)	22 (17.3%)	0
Annotator 2	110 (86.6%)	17 (13.4%)	0
Annotator 3	101 (79.5%)	26 (20.5%)	0
Majority vote	107 (84.3%)	20 (15.7%)	0

Appendix

Table 39: Human annotation results on 127 sampled taxonomy questions.

Dataset	Qwen3-VL	InternVL	Gemma 3	Molmo 2	Total
MMT-Bench	29/448	46/832	6/96	1/32	82/1408
MMMU (dev_val+test)	24/1512	50/2808	8/324	6/108	88/4752
ScienceQA	4/420	0/780	10/90	0/30	14/1320
MME	10/196	18/364	0/42	2/14	30/616
AI2D	0/182	7/338	0/39	1/13	8/572
MathVista	1/70	1/130	0/15	0/5	2/220

Appendix

Table 40: Pairwise inverse scaling cases over eligible comparisons, restricted to pairs in which the larger model has at least twice the total parameters of the smaller. Gemma 3 and Molmo 2 have no pair below this ratio, so their counts are unaffected.

Dataset	Qwen3-VL	InternVL	Gemma 3	Molmo 2	Total
MMT-Bench	20/352	24/704	6/96	1/32	51/1184
MMMU (dev_val+test)	26/1188	33/2376	8/324	6/108	73/3996
ScienceQA	7/330	0/660	10/90	0/30	17/1110
MME	8/154	16/308	0/42	2/14	26/518
AI2D	0/143	3/286	0/39	1/13	4/481
MathVista	0/55	0/110	0/15	0/5	0/185

Appendix

Table 41: As Table 40 , restricted to pairs in which the larger model has at least twice the active parameters of the smaller. Gemma 3 and Molmo 2 are dense and have no pair below this ratio, so their counts are unaffected.

Pairs kept	Comparisons	Cases	Rate	Larger ahead	Replicated
All pairs	10434	375	3.6%	5000 (47.9%)	13
Total parameters $\geq 2\times$	9768	326	3.3%	4856 (49.7%)	11
Active parameters $\geq 2\times$	8214	239	2.9%	4190 (51.0%)	11

Appendix

Table 42: Summary of the two restrictions, with the unrestricted analysis for reference. “Larger ahead” counts comparisons with Net

\leq-5

; “Replicated” is the number of subtask–dataset pairs with a case in at least three families.

#	Family	Subtask	Pair	Ordering	Acc S	Acc L	FI	FN	Net
1	Qwen3-VL	Jigsaw	4B vs. 30B-A3B	identity	80.7	64.0	38	13	$+25$
1	Qwen3-VL	Jigsaw	4B vs. 30B-A3B	reversal	76.0	59.3	38	13	$+25$
2	Qwen3-VL	Jigsaw	4B vs. 32B	identity	80.7	60.7	40	10	$+30$
2	Qwen3-VL	Jigsaw	4B vs. 32B	reversal	76.0	70.0	29	20	$+9$
3	Molmo 2	Jigsaw	4B vs. 8B	identity	80.0	58.7	44	12	$+32$
3	Molmo 2	Jigsaw	4B vs. 8B	reversal	80.7	54.0	49	9	$+40$

Appendix

Table 43: Option-order sensitivity for the six BLINK cases of Table 3 . Each row is one deterministic reordering of the answer options; “identity” replays the original order. Acc is accuracy (%) of the smaller (S) and larger (L) model; FI, FN and Net as in the main text.

Family	Models	Encoder	Params	Same weights
Qwen3-VL	2B, 4B	SigLIP2-Large	0.3B	no (jointly trained)
Qwen3-VL	8B, 30B-A3B, 32B, 235B-A22B	SigLIP2-SO-400M	0.4B	no (jointly trained)
InternVL3.5	1B, 2B, 4B, 8B, 14B, 30B-A3B	InternViT-300M	0.3B	no (jointly trained)
InternVL3.5	38B, 241B-A28B	InternViT-6B	5.5B	no (jointly trained)
Gemma 3	4B, 12B, 27B	SigLIP	0.4B	yes (frozen)
Molmo 2	4B, 8B	SigLIP 2 SO-400M	0.4B	same configuration

Appendix

Table 44: Vision encoders by model. “Same weights” indicates whether the encoder weights are shared across the sizes in the row according to the family’s report.

Family	Cases	Median Net		Persist	Dissolve	Reverse	Recovery
		direct	CoT
Qwen3-VL	122	$+8$	$0$	35 (28.7%)	59	28	40.1%
InternVL3.5	207	$+7$	$0$	54 (26.1%)	99	54	45.4%
Gemma 3	26	$+7$	$-2$	2 (7.7%)	14	10	46.9%
Molmo 2	20	$+10$	$-4$	5 (25.0%)	6	9	39.7%
All	375	$\mathbf{+8}$	$\mathbf{0}$	96 (25.6%)	178	101	43.5%

Appendix

Table 45: Both-model chain-of-thought control, by family. Persist counts cases still meeting Net

\geq 5

under CoT; reverse counts cases where the larger model gains a net advantage of 5 or more. Recovery is the share of the questions the larger model lost in direct mode that it answers correctly under CoT.

Dataset	Cases	Median Net		Persist	Reverse	Recovery
		direct	CoT
MME	35	$+7$	$+4$	17 (48.6%)	7	46.2%
BLINK	95	$+10$	$+3$	40 (42.1%)	17	46.4%
AI2D ∗	9	$+6$	$+1$	3 (33.3%)	1	14.8%
MMT-Bench	87	$+7$	$0$	22 (25.3%)	16	37.3%
ScienceQA	17	$+8$	$-2$	2 (11.8%)	6	59.0%

Appendix

Table 46: Both-model chain-of-thought control, by dataset. Inverse scaling survives CoT on perception-centric benchmarks and collapses on knowledge-centric ones.

Reasoning for VLMs

See all 58 papers

Oct 1, 2026

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.

Xinye Zhao, Yunkai Dang, Yunchen Wu +1

School of Intelligence Science and Technology, Nanjing University

Vision-Language Models Efficient VLM Inference Language Model Scaling Laws

Ask PDF

Jun 27, 2026

On Test-Time Scaling for Vision-Language Models

Test-time scaling is a paradigm where large models use additional compute at inference to achieve better performance, without changing model weights. While it has been widely studied for Large Language Models (LLMs), its applicability to Large Vision-Language Models (LVLMs) remains less explored and analyzed, with limited analysis of whether, when, and to what extent these approaches transfer to LVLMs. In this work, we ask a simple but fundamental question: can conventional test-time scaling methods developed for LLMs be directly applied to LVLMs? We present the first comprehensive study of test-time scaling for LVLMs, spanning multiple models and model sizes, nine test-time scaling methods, and six diverse benchmarks. Our main findings is that 1) different from previous findings, small, well-performing models benefit the most from test-time scaling, enabling performance improvements of up to around 30%, reaching large models performance, and often outperforming them, 2) LVLMs lose focus when given more compute than necessary, and 3) Visual information is encoded early in the reasoning chain, after which the chain is dominated by text-only reasoning and the contribution of image tokens drops significantly. Finally, we also provide a global and fine-grained analysis on the quality and information sufficiency of the reasoning chains produced. Overall, our findings and analysis provide practical guidance and insights into LVLMs and their deployment in research and industry.

Fawaz Sammani, Tzoulio Chamiti, Nikos Deligiannis

ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium

VLM Evaluation Test-Time Scaling for VLMs Test-Time Scaling Large Vision-Language Models VLM Reasoning

Ask PDF

Jul 30, 2026

Scaling Vision-Language Models Is Not Enough to Mitigate Bias

Vision-Language Models (VLMs) such as CLIP are now foundational to multimodal systems, yet their robustness to spurious correlations remains poorly understood at scale. We present the first large-scale empirical study of 194 publicly available VLMs, including 16 model families, covering a wide range of model sizes, 24 training datasets, and three evaluation benchmarks, namely ImageNet (overall performance), CelebA (typical single-attribute bias), and UrbanCars (complex multi-attribute biases). Across these settings, the Spearman correlation between model scale and performance weakens as evaluation shifts from ImageNet (

ρ{=}0.68

) to single-attribute (

ρ{=}0.48

) and further to multi-attribute (

ρ{=}0.05

) bias benchmarks. In contrast, properties of the training data (size and quality) show more consistent relationships with worst-group accuracy across both bias benchmarks. Notably, curated datasets yield improvements of up to 25% over uncurated alternatives at a comparable scale. Finally, the effect of architectural choices (e.g., patch size, image resolution) is highly context-dependent, varying with the nature of the benchmark, including the type of bias and its spatial distribution within images.

Ioannis Sarridis, Ioannis Kompatsiaris, Symeon Papadopoulos

Information Technologies Institute, CERTH Thessaloniki, Greece

VLM Evaluation VLM Robustness Vision-Language Models

Ask PDF

Abstract

Reasoning for VLMs

Explore similar work

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

On Test-Time Scaling for Vision-Language Models

Scaling Vision-Language Models Is Not Enough to Mitigate Bias