VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Models

Khurram Azeem Hashmi; Mohammadreza Zolfaghari; Changdae Park; Rishabh Jain; Nicholas Moratelli; Pengfei Wei; Louis Lu; Tianchi Liu

cs.CVSep 23, 2026

VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Models

Authors: Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Changdae Park, Rishabh Jain, Nicholas Moratelli, Pengfei Wei, Louis Lu, Tianchi Liu,

Organizations: Tether AI Research

Abstract

Sub-billion-parameter Vision-Language Models are increasingly viable for on-device deployment, yet compact model size alone does not guarantee usability. On a phone, such a model can still require more than two minutes to produce its first token. On-device usability depends on three axes: accuracy, efficiency, and behavioral reliability; standard benchmarks miss the third, with answers too short to expose doom loops and prompts too benign to probe adversarial safety. We introduce a diagnosis-driven post-training recipe in which a teacher VLM stress-tests the student, uncovers failure modes beyond human priors, and converts them into targeted supervision and preference alignment, supplementing generic data scaling with failure-driven optimization. Coupled with two visual-token policies, the recipe yields two accuracy-efficiency variants with improved behavioral reliability. \NanoFull attains a 62.3 normalized average over 17 benchmarks, the highest among openly released $\sim$ 0.5B models, +7.4 over its base at identical architecture and token budget, with doom-loop rates at or below the strongest baseline's. \FlashFull retains 61.4 while cutting warm time-to-first-token on a Pixel 9 from 138,s to 6.1,s (23 $\times$ ). By jointly addressing all three axes, we move compact VLMs toward practical on-device usability.

Figures & tables

Figure 1: Accuracy–latency comparison at the 0.5B scale. Normalized average over 17 benchmarks against warm TTFT at $512^{2}$ on a Pixel 9. VPsy gains 7.4 points over its nanoVLM-460M-8k starting checkpoint at identical latency and token budget; VPsy-Flash retains 61.4 of 62.3 at 6.1 s TTFT. Scores are measured at full precision, TTFT with Q4_0 weights.

Figure 2: The five-stage post-training recipe. From the public nanoVLM-460M-8k checkpoint $\theta_{0}$ , Stage 1 installs broad multimodal coverage ( $\ell_{\mathrm{CE}}$ on $\mathcal{D}_{1}$ ) and Stage 2 targets deployment capabilities on $\mathcal{D}_{1}\cup\mathcal{D}_{2}$ with full rehearsal, producing $\theta_{2}$ , the reference for the failure analysis of Sec. 3.1 and the merge base of Stage 4 (dashed arrow). Stage 3 fine-tunes $\theta_{2}$ into $K$ weakness-targeted specialists and Stage 4 TIES-merges [ 54 ] them into one checkpoint $\theta_{4}$ at unchanged architecture and serving cost ( Fig. 4 ). Stage 5 aligns doom-loop termination and safety with capability replay, yielding the released checkpoint $\theta_{5}$ .

Figure 3: One round of the active failure-discovery loop: a teacher VLM probes the student on images sampled toward its weak spots, grades the answers with the image in view, and reflects over the failures to rewrite the weakness catalogue seeding the next round.

Figure 4: Stages 3–4. Stage 3 turns the base $\theta_{2}$ into $K$ weakness-targeted specialists $\{\theta_{3}^{(k)}\}$ ; Stage 4 merges their task vectors $\tau^{(k)}=\theta_{3}^{(k)}-\theta_{2}$ with TIES into a single checkpoint.

	Document Understanding & OCR					Visual Perception				Reasoning & Knowledge						IF & Reliab.
Model	OCRB †	DocVQA †	ChartQA †	InfoVQA †	TextVQA †	MME	SEED	MMB	RWQA	SQA	AI2D	MMStar	MMMU	MathV	MMVet	MM-IF	POPE	Avg ${}_{\text{norm}}$
SmolVLM2-500M [ 31 ]	619	74.7	64.9	39.0	71.3	1455	62.1	51.4	50.1	76.3	57.3	38.3	31.6	38.5	28.6	11.1	82.7	52.5
nanoVLM-460M-8k [ 49 ]	745	82.3	70.4	46.0	74.5	1442	64.3	53.1	51.8	76.8	55.3	37.0	31.3	33.3	28.4	20.2	82.7	54.9
LFM2.5-VL-450M [ 25 ]	710	82.7	76.6	48.9	78.8	1453	67.5	56.8	59.0	77.7	62.2	42.7	30.7	42.2	36.0	42.0	86.5	59.6
VisionPsy-Nano-460M	757	85.7	78.7	49.8	79.3	1541	69.2	61.9	60.0	86.5	66.5	47.6	31.4	48.9	32.3	42.3	87.9	62.3
VisionPsy-Nano-460M-Flash	738	85.1	77.4	51.1	75.6	1619	67.6	60.5	58.4	84.0	66.4	45.5	30.7	47.8	31.1	43.7	87.5	61.4

Table 1: Per-benchmark comparison across the

\sim

0.5B cohort on all 17 public benchmarks. † Scores judged with Qwen3.6-27B [ 37 ] .

Stage	OCRB †	DocVQA †	ChartQA †	InfoVQA †	TextVQA †	MME	SEED	MMB	RWQA	SQA	AI2D	MMStar	MMMU	MathV	MMVet	MM-IF	POPE	Avg ${}_{\text{norm}}$
	Document Understanding & OCR					Visual Perception				Reasoning & Knowledge						IF & Reliab.
$\theta_{0}$ : nanoVLM-460M-8k [ 49 ]	745	82.3	70.4	46.0	74.5	1442	64.3	53.1	51.8	76.8	55.3	37.0	31.3	33.3	28.4	20.2	82.7	54.9
$\theta_{1}$ : + S1 Broad SFT	714	85.1	74.7	47.3	78.1	1555	69.1	62.8	56.1	85.9	67.8	44.6	34.1	50.2	33.7	17.0	86.0	60.0
$\theta_{2}$ : + S2 Capability-targeted SFT	727	83.7	75.5	44.5	78.0	1545	69.4	61.0	61.2	86.2	66.0	47.8	32.9	45.3	32.5	42.9	87.6	61.3
$\theta_{3}\!\rightarrow\!\theta_{4}$ : + $\{\text{S3}\}\!\rightarrow\!\text{S4}$ Specialist merging	753	85.5	77.8	49.2	79.4	1542	69.8	61.8	59.2	86.6	67.3	48.3	31.8	48.2	32.6	43.4	87.5	62.3
$\theta_{5}$ : + S5 Preference alignment (MPO)	757	85.7	78.7	49.8	79.3	1541	69.2	61.9	60.0	86.5	66.5	47.6	31.4	48.9	32.3	42.3	87.9	62.3

Table 2: Stage-wise ablation. Stage 5 targets behavioral reliability ( Tab. 5 , Fig. 5 ) at a flat aggregate. Best scores are highlighted .

Table 3: Accuracy-axis ablations of the training stages. 3(a) The best failure-guided specialist improves

\theta_{2}

where the best size-matched random-pool specialist does not. 3(b) TIES [ 54 ] leads naive averaging and the best single specialist on three of four axes at unchanged inference cost. 3(c) Stage-2 retention and staging: training on

\mathcal{D}_{2}

alone degrades every untargeted axis; full Stage 2 recovers them; single-stage

\mathcal{D}_{1}{\cup}\mathcal{D}_{2}

trails staged

\theta_{2}

Model	Pixel 9	S23	S25 Ultra	iPhone 15
nanoVLM (VPsy)	138.3	116.7	58.8	10.9
LFM2.5-VL	14.9	13.6	5.7	0.4
SmolVLM2	128.2	113.7	48.7	10.7
VPsy-Flash	6.1	5.9	2.6	0.3

Table 4: Warm TTFT (s) at

512^{2}

on smartphones. VPsy-Flash is the fastest on every device (23

\times

over the base on Pixel 9). Since VPsy shares nanoVLM’s architecture and token budget, the base row gives its TTFT. Best lane per model-device pair.

Model	Doom loops	Rate $\downarrow$
nanoVLM-460M-8k ( $\theta_{0}$ )	4 / 395	1.01%
SmolVLM2-500M	6 / 395	1.52%
LFM2.5-VL-450M	2 / 395	0.51%
VPsy ( $\theta_{4}$ )	17 / 395	4.30%
VPsy ( $\theta_{5}$ )	1 / 395	0.25%
VPsy-Flash ( $\theta_{4}$ )	6 / 395	1.52%

Table 5: Doom-loop rate on the 395-prompt long-form stress test. Post-training amplifies the failure in both runs (

\theta_{0}\!\to\!\theta_{4}

). Stage-5 alignment repairs both to the level of the strongest baseline or below. Lower is better.

Figure 5: Safety pass rates on eight vulnerability categories under the 288-case red-teaming protocol; higher is better.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Seed weakness (category)	Seeded hypothesis
Doom-loop / degenerate repetition (repetition)	On long generations the model repeats a word, phrase, or line until the token limit instead of stopping cleanly.
Ignores output-format instructions (instruction following)	Given a required output shape (JSON with fixed keys, one word, a bare number) it adds commentary or picks its own format.
Fails exhaustive extraction (instruction following)	Told to list all text or items, it omits entries, stops early, or pads with commentary.
Drops multi-part / conditional instructions (instruction following)	With multiple steps or a condition (“if $X$ then $\dots$ , else NONE”) it handles only the first part.
Brittle to natural-language chart phrasing (chart reading)	Reads chart values under templated phrasing but fails on paraphrased, conversational versions of the same lookup.
Numeric reasoning over read values (math / numeric)	Reads individual numbers correctly but cannot sum, difference, average, or count over them.

Appendix

Table 6: The

13

seeded weaknesses that initialise the catalogue, derived from an error analysis of

\theta_{2}

, inspection of its failing generations, and failure modes generic to sub-billion VLMs. Seeds are hypotheses rather than ground truth: the loop re-estimates their severity, rewrites their descriptions, and replaces their probe templates from observed evidence.

Seed: Digit / dense-numeral OCR severity $4\rightarrow 5$
Seeded hypothesis	Misreads prices, times, codes, and small or dense numerals in scene text and documents (numeric scene text much weaker than word reading).
Refined hypothesis	Misreads prices, times, codes, and dense numerals; high error rate on alphanumeric IDs; confuses similar digits; misses small text in footers and badges. New evidence: misreads numeric labels in geometry diagrams, omits the “%” symbol, hallucinates license-plate digits, fails on axis tick marks, reads “S$6” as “56” and “35%” as “39”, misreads scale-bar numbers and zip codes in footers.
Seeded probes	“Read the exact number/price/time shown here.”; “Transcribe the full code/serial/plate exactly.”
Refined probes	“Read the price on this tag.”; “Transcribe the license plate.”; “What time is on the clock?”

Appendix

Table 7: Representative refinement of a seeded weakness after

300

rounds. The teacher replaces a broad operator prior with an enumeration of observed mechanisms and distils the probe templates into short elicitations that reliably trigger the failure.

Rd.	Sev.	Discovered failure mode (category)
1	5	Hallucination of visual content and text (hallucination) †
1	5	Refusal or empty output on complex tasks (refusal) †
2	5	Attribute recognition errors (attributes) †
3	5	Chinese / complex-script OCR failure (OCR)
3	4	Hallucinated errors when asked to spot textual mistakes (hallucination) †
5	5	Counting failures in cluttered scenes (counting) †

Appendix

Table 8: All

24

weaknesses authored by the teacher over

300

rounds, with the round each first entered the catalogue and its final severity;

20

of the

24

carry the maximum severity, and together with the

13

seeds they form a final catalogue of

37

clusters. † marks the seven categories no seed covered. Four of these seven (hallucination, refusal, world knowledge, generation idiosyncrasy) are absent from the seed set in any form; the other three (counting, attributes, grounding) name behaviors mentioned inside a seed hypothesis but never promoted to a category of their own. Entries without a dagger are novel mechanisms discovered within a seeded category, such as the resolution of a single “digit OCR” prior into axis-label extraction, row/column misalignment, and multi-column reading order.

LM format	Imatrix	VLM size (MB)	VPsy $\Delta$ Norm.	VPsy-Flash $\Delta$ Norm.
FP32	N/A	1,666	$0.00$	$0.00$
BF16	N/A	886	$-0.09$	$-0.02$
Q8_0	No	520	$0.00$	$-0.06$
Q5_K_M	No	414	$-0.12$	$-0.52$
Q5_K_M	Yes	414	$-0.59$	$-0.07$
Q4_K_M	Yes	393	$-0.24$	$-0.53$

Appendix

Table 9: Accuracy retention across GGUF quantization formats.

Device	SoC	GPU backend	RAM
Google Pixel 9	Tensor G4	Vulkan	12 GB
Samsung Galaxy S23	Snapdragon 8 Gen 2	OpenCL	8 GB
Samsung Galaxy S25 Ultra	Snapdragon 8 Elite	OpenCL	12 GB
Apple iPhone 15	A16 Bionic	Metal	6 GB

Appendix

Table 10: Hardware specifications and compute backends for on-device evaluation.

Model	$512^{2}$	$512{\times}1024$	$2048{\times}1024$
nanoVLM (VPsy)	1,088	576	576
SmolVLM2	1,088	576	576
LFM2.5-VL	256	242	2,290
Qwen3.5-0.8B	256	512	2,048
VPsy-Flash	64	192	576

Appendix

Table 11: Projected visual-token count across input resolutions.

Resolution	Model	Pixel 9		Galaxy S23		S25 Ultra		iPhone 15
		CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU
$512^{2}$	nanoVLM (VPsy)	153.0	138.3	119.0	116.7	67.0	58.8	70.3	10.9
	SmolVLM2	128.2	136.1	135.5	113.7	48.7	64.3	72.1	10.7
	LFM2.5-VL	14.9	21.9	13.6	14.2	7.7	5.7	3.4	0.4
	Qwen3.5-0.8B	21.4	25.7	22.0	19.9	8.4	10.4	5.0	0.7
	VPsy-Flash	6.1	16.4	6.1	5.9	2.7	2.6	5.9	0.3

Appendix

Table 12: Warm TTFT (s) for CPU and GPU execution at

512^{2}

and

2048{\times}1024

. Lower is better.

Resolution	Model	Pixel 9		Galaxy S23		S25 Ultra		iPhone 15
		CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU
$512^{2}$	nanoVLM (VPsy)	1,236	2,224	1,232	1,533	1,251	1,552	827	482
	SmolVLM2	1,236	2,224	1,232	1,534	1,252	1,552	780	482
	LFM2.5-VL	501	1,123	497	588	510	602	238	181
	Qwen3.5-0.8B	866	2,243	853	942	878	972	903	836
	VPsy-Flash	750	1,692	746	1,047	763	1,066	292	285

Appendix

Table 13: Peak resident memory (MiB) during on-device inference at

512^{2}

and

2048{\times}1024

. Measurements exclude model loading and follow the protocol in Sec. B.2 .

Resolution	Model	Pixel 9		Galaxy S23		S25 Ultra		iPhone 15
		CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU
$512^{2}$	nanoVLM (VPsy)	15.6	27.9	23.2	17.8	47.0	51.9	8.3	52.6
	SmolVLM2	18.2	28.1	18.5	18.3	64.8	49.6	11.4	53.6
	LFM2.5-VL	42.6	47.8	52.3	42.6	69.8	122.4	28.2	116.0
	Qwen3.5-0.8B	17.9	22.9	12.8	21.1	45.2	24.6	6.9	40.5
	VPsy-Flash	28.1	30.7	32.7	33.8	93.5	93.7	7.6	78.9

Appendix

Table 14: Autoregressive decode throughput (tokens/s) at

512^{2}

and

2048{\times}1024

Figure 6: End-to-end time to last token for free-form generation. The optimal CPU or GPU execution lane is reported for each device.

	Qwen3.6-27B		GPT-4o-mini		GPT-5-chat
Benchmark	Ours	LFM	Ours	LFM	Ours	LFM
OCRBench	757	710	775	725	766	711
DocVQA	85.7	82.7	85.1	82.4	85.7	82.7
ChartQA	78.7	76.6	78.2	76.7	78.8	77.0
InfoVQA	49.8	48.9	50.0	49.0	50.0	49.0
TextVQA	79.3	78.8	78.2	77.1	80.2	79.5

Appendix

Table 15: Robustness to the choice of LLM judge. “Ours” is VPsy and “LFM” is LFM2.5-VL-450M, the previous best competitor in the

\sim

0.5B cohort. Both are rescored under three judges on the eight benchmarks whose scoring depends on the judge; the remaining nine are multiple-choice or rule-scored and return identical values under all three, so they are omitted here and included unchanged in Avg

{}_{\text{norm}}

. Absolute scores shift with the judge, most visibly on MM-IFEval, but the comparison does not: Ours leads by 2.7, 2.9 and 2.8 normalized points under Qwen3.6-27B, GPT-4o-mini and GPT-5-chat, and takes 16 of 17 benchmarks under every judge. Higher score per judge in bold .

	Baselines			Ours
Benchmark	nanoVLM	SmolVLM2	LFM	VPsy	VPsy-Flash
ChartQA	+4.5	+4.0	+1.6	+1.8	+2.3
DocVQA	+5.0	+6.1	+4.4	+5.1	+4.4
InfoVQA	+11.0	+11.4	+5.6	+6.5	+6.3
TextVQA	+9.6	+10.7	+8.9	+8.4	+8.1
OCRBench	+35	+8	+31	+31	+39

Appendix

Table 16: Score delta (judge

-

strict) each model receives on the five benchmarks where our evaluation protocol replaces the conventional strict metric with a judge; OCRBench is on its native /1000 scale and divided by 10 in the Avg row. The judge rescues every model on every benchmark, and the largest corrections go to the baselines rather than to our models.

Document Understanding & OCR
Benchmark	Zero-shot	Full Recipe	$\Delta$
OCRBench	709	738	+29
DocVQA	85.4	85.1	-0.3
ChartQA	69.4	77.4	+8.0
InfoVQA	48.8	51.1	+2.3
TextVQA	73.3	75.6	+2.3

Appendix

Table 17: Impact of our complete post-training recipe under the bounded native-resolution policy. The Zero-shot baseline evaluates the full-resolution checkpoint using the VPsy-Flash preprocessing logic strictly at inference time, whereas Full Recipe denotes our final VPsy-Flash model.

Benchmark	Overlapping images
SEEDBench	4,463
ChartQA	1,611
ScienceQA	1,610
POPE	1,142
AI2D	631
DocVQA	560

Appendix

Table 18: Per-benchmark count of held-out evaluation images that appear in (and are removed from) the raw training corpus. Counts are non-exclusive overlap events: one training image may overlap multiple benchmarks. In total 5,741 unique training images are removed; since one image is typically reused across several question–answer pairs, this corresponds to 35,649 training samples.

Stage	Corpus	Description	#Examples
1	Filtered Nemotron-V3 [ 35 ]	Decontam. + Single-image filtering; think tokens stripped	5,950,891
2	Upperbound Mixed (full)	Filtered Nemotron-V3 [ 35 ] + Sampled FineVision [ 50 ] + MM-IFEngine [ 9 ]	7,527,286
2	UPMB2M	Capability-balanced 2M subset of upperbound Mixed	2,000,000
3	3Bsynth	Synthetic / full-real preference-style mix	656,160
	MathEnrich (total)	Decontaminated math/chart/geometry SFT	448,088
	MathV360K (filtered) [ 41 ]	Kept math/chart/science sources only	176,879

Appendix

Table 19: Training-corpus statistics for the datasets used in different stages. Counts are example rows after decontamination / filtering where applicable.

Panel of Figure 7 — Figure 7: Qualitative comparison against sub-billion-parameter baselines. Responses from VPsy and VPsy-Flash compared to nanoVLM-460M-8k and LFM2.5-VL-450M. The baselines frequently struggle with spatial reasoning, hallucinate scene details, or misidentify object counts and colors.

Panel of Figure 8 — Figure 8: Failure cases at the ${\sim}$ 0.5B scale. Common failure modes across the entire sub-billion-parameter class. All evaluated models struggle with complex multi-step geometry (top), fine-grained symbol recognition (middle), and conditional data extraction from dense charts (bottom). Notably, while baseline models often generate verbose and hallucinated reasoning chains (e.g., LFM2.5-VL-450M calculating nonsensical $\arcsin$ values), our models tend to fail more concisely.

Figure 9: Live on-device deployment (Apple A16 Bionic). Inference on an iPhone 15 using llama.cpp (4-bit). VPsy-Flash achieves an interactive 0.9s time-to-first-token, completely bypassing the massive delays of default-policy models like nanoVLM-460M-8k (13.7s).

Figure 10: Live on-device deployment (Qualcomm Snapdragon 8 Elite). Inference on a Samsung Galaxy S25 Ultra. The visual prefix bottleneck affects even the newest mobile chips, with baselines taking up to 54.3s. VPsy-Flash reduces this delay to just 3.2s.

Figure 11: Live on-device deployment (Google Tensor G4). Inference on a Google Pixel 9. This represents the most constrained environment, where base models freeze for nearly two minutes (115.8s). VPsy-Flash successfully cuts this down to 9.5s, enabling usability.

VLM Post-Training

See all 11 papers

Mar 17, 2026

Empirical Recipes for Efficient and Compact Vision-Language Models

Deploying vision-language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.

Jiabo Huang, Zhizhong Li, Sina Sajadmanesh +2

Sony AI.

Vision-Language Models Efficient VLM Inference

Ask PDF

Mar 24, 2026

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

Mehmet Kaan Erol

Marmara University, Institute of Pure and Applied Sciences

Compositional Reasoning in VLMs VLM Evaluation VLM Robustness VLM Reasoning VLM Quantization VLM Calibration

Ask PDF

Oct 14, 2026

Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models

Improving the reasoning capabilities of vision-language models (VLMs) often relies on post-training with supervision from stronger models. Two common strategies are supervised fine-tuning (SFT), which trains on teacher-generated demonstrations, and on-policy distillation (OPD), which provides teacher feedback on trajectories generated by the student itself. While OPD can be more data-efficient by supervising student-generated trajectories, it is unclear how the teacher's supervision at each state affects the student's learned behavior. To study this question, we conduct a controlled comparison of OPD and filtered SFT across data budgets and tasks where the teacher exhibits different levels of reliability, and analyze how these differences shape student accuracy and completion behavior. Across the two tasks, we find that OPD is effective only when the teacher remains reliable on student-generated trajectories. On ChartQA, where the teacher completes 99% of its solutions, OPD trained on only 10% of the data comes within 1.4 accuracy points of full-data SFT. In contrast, on Geometry3K, where the teacher fails to complete 22% of its solutions, OPD underperforms SFT by 7.9 points, with most of the gap explained by incomplete responses. More strikingly, starting OPD from an SFT student reduces its completion rate from 76% to 62%, effectively erasing the completion behavior learned through SFT. Filtering teacher demonstrations for completeness alone recovers much of SFT's advantage. These results show that on-policy supervision can transfer not only teacher knowledge but also persistent failure modes on student-visited states. Code is available at https://github.com/ffy208/opd-inherits-teacher-vlm.

Frank F. Yang, Kefei Yan, Yiren Wang +1

Georgia Institute of Technology, Atlanta, GA, USA

VLM Distillation On-Policy Distillation VLM Post-Training VLM Reasoning

Ask PDF

Abstract

VLM Post-Training

Explore similar work

Empirical Recipes for Efficient and Compact Vision-Language Models

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models