Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Minhao Fan; Yinyi Liu; Jiayu Zhao; Zihan Teng; Song Chen; Weichen Liu

cs.AIOct 8, 2026

Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Authors: Minhao Fan, Yinyi Liu, Jiayu Zhao, Zihan Teng, Song Chen, Weichen Liu

Organizations: College of Computing and Data Science, Nanyang Technological University · School of Microelectronics, University of Science and Technology of China

Abstract

Small vision-language models may be able to read external evidence yet struggle to obtain it. We introduce Harness Compilation (HC), an offline procedure that adapts the division of work between a frozen small VLM and its external harness. A large teacher uses student execution traces to revise reusable content and control, while a separate validation set selects the deployed harness. Deployment requires neither weight updates nor teacher calls. Across seven visual question-answering settings with students of at most 9B parameters, HC improves scores over bare students by 9.9-23.9 points, averaged over three independent builds per setting. Interventions on five runtime decision types (invocation, selection, argument generation, evidence integration and abstention) show why this allocation matters: requesting evidence and generating open queries can be costly, whereas bounded choices and reading supplied text can remain useful student work. Fact cards benefit all ten evaluated students, but decision policies transfer unevenly. Recompilation for a new student model helps when the transferred interface no longer fits the student. With 100 practice items, HC exceeds answer-only LoRA on three tasks. Larger training budgets can match or surpass a fixed harness, while combining the two improves SlideVQA beyond either alone. These findings support allocating work from measured student behavior rather than uniformly removing decisions.

Figures & tables

Figure 1: Compile the division of work for reuse at deployment. (A) Within a fixed runtime, the teacher revises the latest proposal from probe traces, while disjoint validation selects $H^{*}$ . (B) Local control invokes the frozen student or emits a permitted code verdict. In the archived InfoSeek test example, the student selects Pizza, then reads its card to answer Italy. Both calls receive the image, with fresh context for answering. Card text is excerpted (Appendix A.4 ).

Setting	Compiled work	Student retains
InfoSeek	Route retrieval and prepare a card	Choose among names on ambiguous inputs, then answer
DocVQA	Supply OCR and an answer format	Read the page and supplied text
SlideVQA	Rank and select evidence slides	Read selected slides and answer
LiveVQA	Supply three article cards	Select and combine evidence while answering
BLINK	Compute a verdict from measurements	Answer when the rule defers

Table 1: Examples of the division of work. Each row describes a fixed harness used in the diagnostic studies. Policies can differ across builds, so these allocations are not universal policies for each task.

Setting	Bare	Student-routed	Human recipe	HC (3 builds)
InfoSeek	18.0	29.1	33.8	37.7
DocVQA	72.7	80.4	82.4	85.0
SlideVQA	44.7	44.4	48.7	54.6
LiveVQA	14.7	36.9	25.5	38.6
BLINK reflectance	50.8	50.0	59.0	62.9
BLINK correspondence	54.9	64.9	64.2	65.7

Table 2: Overall test scores. HC averages three independent builds per setting. Other columns are the specified comparators. Higher is better. The macro-average weights all seven task-score scales equally. Human recipes are alternative policies, not an upper bound.

Setting	Initial $H_{0}$	Selected $H_{\mathrm{val}}$
InfoSeek	33.5	37.7
DocVQA	78.4	85.0
SlideVQA	54.4	54.6
LiveVQA	34.1	38.6
BLINK reflectance	61.7	62.9
BLINK correspondence	65.2	65.7

Table 3: Revision and validation selection after the initial teacher-built harness. Means over the same three builds per setting, evaluated on identical test items. Validation alone selects the retained round.

Figure 2: Changing the interface changes the work left to the student. Three interventions from Table 4 . Differences are changes from the fixed diagnostic reference, not additive capability effects. Naming changes the lookup interface, while raw BLINK signals also remove compiled verdict control.

Setting	Decide	Select	Generate	Integrate	Persist
InfoSeek	$-9.0$	$-0.5^{\mathrm{ns}}$	$-13.3$	$-3.1$	$-4.3$
DocVQA	$-2.7$	$+0.5^{\mathrm{ns}}$	n/a	$+0.1^{\mathrm{ns}}$	$+0.6^{\mathrm{ns}}$
SlideVQA	$-0.5^{\mathrm{ns}}$	$-12.1$	n/a	$+0.6^{\mathrm{ns}}$	$-2.8$
LiveVQA	n/a	n/a	n/a	$-0.1^{\mathrm{ns}}$	$+0.2^{\mathrm{ns}}$
BLINK reflectance	$+0.0^{\mathrm{ns}}$	n/a	n/a	$-14.2$	$-4.5^{\mathrm{ns}}$
BLINK correspondence	$-2.5$	n/a	n/a	$-2.3^{\mathrm{ns}}$	$-1.4^{\mathrm{ns}}$

Table 4: Effect of changing one decision interface. Paired score changes from one fixed reference harness per setting, with negative values indicating harm. These references differ from the three-build means in Table 2 . ns :

p>0.05

. n/a: untested. Each column changes how a decision is executed, rather than measuring an isolated capability. Appendix C.1 defines the interventions and comparison conditions.

Change to the selected interface	Evaluated harnesses	Score-change range
Restore stored cards’ original lines	A and B (6)	$+0.27$ to $+2.87$
Remove question-text hints	A (3)	$-2.23$ to $-0.13$
Default extractor with contract	A (3) / B (3)	$-0.02$ to $0.00$ / $-0.07$ to $0.00$
Artifact extractor without contract	A (3) / B (3)	$-6.07$ to $-1.78$ / $-2.57$ to $-1.72$

Table 5: Controls within formal InfoSeek harnesses. Each change is paired with its own fixed harness on 6,000 test items. Ranges span harnesses, not confidence intervals. All use interface v2, with A editing the last proposal and B the incumbent. Extractor and contract rows report A and B separately and do not decompose the diagnostic

-4.3

effect. Full controls: Appendix A.5 .

Figure 3: Recompilation can repair a mismatched interface. (A) Fresh InfoSeek builds versus unchanged transfer of the Qwen3-VL-4B diagnostic harness (37.8 on its source student): nine recipients, one fresh build each, interface v2. Positive values favor recompilation (reported mean

-0.4

). (B) DocVQA changes are relative to each bare student. The transplant is the formal replicate-1 Gemma harness, and fresh builds use the current gate with its bare floor. The panels use different baselines. Source harnesses and paired tests are in Appendix C.3 .

Setting	HC	Answer LoRA	Trajectory SFT
InfoSeek	33.6	23.3	35.3
DocVQA	84.7	81.1	n/a
SlideVQA	54.7	47.3	n/a

Table 6: Adaptation with 100 practice items. Mean test scores over three runs. HC uses 50 probe and 50 validation items. Training columns use the stronger reported epoch variant. Schedules and build-level results are in Appendix F.1 .

Setting	LoRA labels	Fixed HC	LoRA	LoRA + HC
SlideVQA	1,300	58.1	53.2	63.7
DocVQA	999	85.4	84.7	86.4

Table 7: Fixed-harness component comparisons. The same serialized harness, verified by identical hashes, is used before and after tuning. LoRA labels count training items. Harness practice budgets are 450 (SlideVQA) and 999 (DocVQA). These are separate component studies, not the formal three-build means. Training and label-scaling protocols: Appendix F .

Setting	Test score $\uparrow$		Tokens per item $\downarrow$
Setting	CoT + vote	HC	CoT + vote	HC
InfoSeek	24.6	37.3	1,120	5.9
DocVQA	80.4	85.7	580	8.5
SlideVQA	51.1	58.1	970	5.8
BLINK reflectance	64.9	64.2	2,000	0.1
BLINK correspondence	68.5	67.4	2,510	0.9

Table 8: Task performance and generated student tokens. Each HC row pairs the score and token count from one execution of the formal replicate-1 harness (fold artifacts on BLINK). CoT+vote records the comparator runs on the same items. Counts include all sampled rollouts and student passes, excluding input prefill, tools and offline compilation. Appendix G.3 gives the accounting.

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Student	Metric	Test items	Practice
InfoSeek	Qwen3-VL-4B	Relaxed exact match	6,000	150 / 300
DocVQA	Gemma-4-E4B	ANLS	3,568	150 / 300
SlideVQA	Qwen3-VL-4B	Exact match	2,215	150 / 300
LiveVQA	Qwen3-VL-4B	Judged accuracy	2,000	150 / 300
BLINK reflectance	Qwen3-VL-4B	Accuracy	134	44 / 45 per fold
BLINK correspondence	Qwen3-VL-4B	Accuracy	441	147 / 147 per fold

Appendix

Table 9: Main evaluation settings. Practice counts are probe / validation items per build. The additional outer selection budget is stated below.

Setting	Human recipe	Score
InfoSeek	Select a candidate name, then read retrieved Wikipedia passages	33.8
DocVQA	Supply page OCR in typed fields	82.4
SlideVQA	Read the three slides selected by the text ranker	48.7
LiveVQA	Select one candidate headline and read its article card	25.5
BLINK reflectance	Fixed Retinex verdict rule	59.0
BLINK correspondence	Fixed LightGlue and DINO rule	64.2

Appendix

Table 10: Human recipes used in Table 2 . Full prompts are in Appendix E.1 .

Comparison	InfoSeek	DocVQA	Reference and practice budget
Formal results (Tables 2 – 3 )	37.7	85.0	Interface v2, reflection A; three replicates, 450 items each
Decision interventions (Table 4 )	37.8	84.1	Current runtime; one harness, 450 items (InfoSeek, DocVQA, LiveVQA); SlideVQA and BLINK use the formal replicate-1 artifacts. The BLINK integrate entries are the student-routed arm of Table 2
Generated-token comparison (Table 8 ) and compute baselines (Table 41 )	37.3	85.7	Formal replicate-1 harnesses re-executed with token logging; score and tokens from the same execution
One-shot construction (Table 22 )	39.7	84.9	Legacy runs; 1,200 / 999 items, respectively
Fixed-harness comparison with tuning (Table 7 )	–	85.4	Identical artifact hashes on both students: SlideVQA = the formal replicate-1 harness; DocVQA = a current-protocol build on the 999 adapter-training items
Current transfer source (Table 28 )	37.8	–	Current runtime; 450 items

Appendix

Table 11: Which HC result belongs to which comparison? Scores identify the reference harness, not a new comparison across rows. Formal means use three replicates. All other numerical entries identify single harnesses. Practice counts include probe and validation items.

Setting	GPT-5.5	Evaluation population and input
InfoSeek	50.0	300-item practice subset
DocVQA	93.3	Full test set, 2,048-pixel images
SlideVQA	64.7	150-question practice pool with the linker’s top-three slides
LiveVQA	58.3	300-question practice pool with the same candidate passages
BLINK reflectance	79.9	Same items as the student and harness
BLINK correspondence	78.7	Same items as the student and harness

Appendix

Table 12: Teacher reference scores. Different populations prevent treating this column as a common upper bound.

Setting	Items	What the comparison tests
Relative reflectance	134	Whether local photometric measurements can support a useful verdict despite incomplete illumination invariance.
Correspondence	441	Visual (172), semantic (139), and functional (130) correspondence share a matching interface but differ in what geometric signals can resolve.
Relative depth	124	Whether a depth estimator’s measurements are better used through code than interpreted by the student.

Appendix

Table 13: BLINK task selection: the five selected tasks form three evaluation settings.

Requesting resources and following rules
Measurement	Qwen3-VL 2B	Qwen3-VL 4B	Qwen3-VL 8B	Gemma-4 E4B	InternVL3.5 4B
Memory activation rate	0.0	82.3	89.3	97.3	58.7
Generated entity-name precision	0.0	30.4	36.2	15.1	0.6
Rule compliance, turn 1	36	100	98	100	22
Rule compliance, turn 6	32	100	98	90	18
Correct choice, 1-tool menu	19	66	75	46	45

Appendix

Table 14: Student profiles, one measurement per row. Rates and precision are percentages, and

\Delta

denotes a task-score change in points. Greedy decoding on 300 practice items unless the protocol states otherwise. These contract-specific measurements are not a common capability scale.

Stage	Supplied information or required output
Runtime manual	Allowed observations, JSON fields and defaults, router modes, code-hook signatures, and output constraints.
Initial compilation	Task description, bare-student probe score and resource examples. No separate student profile on InfoSeek, LiveVQA and SlideVQA. The DocVQA builds receive the Gemma profile block (a deviation from the stated protocol, recorded in Appendix A.1 ).
Reflection A	Latest proposal JSON and its traced probe report, plus round history with scores and the retained incumbent. The incumbent’s JSON is not separately supplied as the edit target.
Probe evidence	Stage-level errors, 30 stratified InfoSeek failure traces, pick precision and abstention, with one additional trace query per round. Gold labels serve offline diagnosis only.
Teacher response	One complete harness JSON containing prompt, resource and decoding fields, plus executable router and parser source.
Selection	Strict improvement of the unrounded validation mean, with the bare-model floor. The selected artifact is frozen before test evaluation.

Appendix

Table 15: Compiler inputs and outputs under the current interface. The subsequent listings expand this documented contract with archived student requests and generated harness components.

Setting	Router inputs	Keys beyond prompt , decode	External control	Student work
InfoSeek	Question text; CLIP top-3 entity names, similarities and cards from the fixed inventory	memory , discriminate_prompt , name_prompt , parse_py	when to show a card, when to invoke the bounded pick, which card lines to show	bounded choice among $\leq 3$ named candidates when routed to it (with the escape), then reading the card and answering
LiveVQA	Question text; CLIP top-3 news headlines, similarities and article cards	same keys; no store rebuild	retrieve and present the article evidence; card rewriting	read the passages and answer, with an explicit headline pick on paths that request one
DocVQA	Question text and noisy page OCR	tool ( ocr , max_chars ), parse_py	OCR provision and the answer contract	read the page and the supplied OCR, then answer
SlideVQA	MiniLM and CLIP slide rankings, per-slide OCR, deck size	pages ( $k$ , ranker, whole deck), tool , parse_py ; router may return a verdict	evidence-page ranking and selection	read and combine the selected slides, then answer
BLINK (3)	the bed’s pixel, matcher or depth measurements at the marked points (§ A.2 )	tool.signals , answer_mode , parse_py	compute a verdict from the measurements and decide when to defer	answer when the rule defers; in the suggestion variants, read the verdict as advice
LIBERO-plus	instruction, nearest training sentence with similarity, detector scores of the instruction’s noun phrases, end-effector state, steps since last motion	tools (detector, check_every , restore), persist ( max_steps , stall)	instruction override and the exposed termination controls (abort, stop)	generate actions from the observations and the current instruction

Appendix

Table 16: Division of work under the current runtime interfaces. External control lists operations available to the selected programs. Student work occurs on paths that invoke the frozen model. Individual formal artifacts can use different policies (Appendix A.5 ). The compiler fills these fields subject to the input and output constraints above.

Replicate	Protocol	Selected round	Validation	Test	Teacher $
1	A	1	41.0	37.3	1.19
1	B	3	40.7	36.1	0.54
2	A	1	43.3	39.1	0.83
2	B	0	40.0	36.9	0.57
3	A	1	40.0	36.6	0.91
3	B	2	37.0	34.6	0.71

Appendix

Table 17: Formal InfoSeek replicates. Each pair starts from the same initial harness: A uses the reported last-proposal reflection, B the incumbent-anchored variant. All use Qwen3-VL-4B with greedy decoding, sampled GPT-5.5 proposals, the same store and practice split, the current runtime, and the frozen strict gate. Teacher cost is per arm. Round 0 denotes the initial harness. Full test records across settings are in Table 24 .

Replicate	Protocol	Genus	Hints	Card budget	Routing
1	A	Yes	Yes	850	Similarity thresholds + category abort
1	B	No	No	850	Similarity thresholds only
2	A	No	Yes	850	Similarity thresholds only
2	B	No	No	850	Similarity thresholds only
3	A	Yes	Yes	560	Always show the top three cards
3	B	Yes	No	560	Always show the top three cards

Appendix

Table 18: Mechanisms present in the formal InfoSeek routers. All six use keyword-scored card-line selection. The budget is characters per card. A = reported reflection. B = incumbent-anchored variant. “Genus” means a line derived from the scientific name, and “hints” means question-type notes.

Replicate	Protocol	archived / R0	R1 derived line	R2 hints	R3 both	R4 original cards	R5 similarity routing
1	A	37.30 / 37.30	$+0.00$ (0/0)	$-2.23$ (37/171 ∗ )	$-2.38$ (37/180 ∗ )	$+0.72$ (65/22 ∗ )	$+0.68$ (66/25 ∗ )
1	B	36.12 / 36.13	n/a	n/a	n/a	$+0.75$ (77/32 ∗ )	n/a
2	A	39.13 / 39.12	n/a	$-0.13$ (6/14)	n/a	$+0.27$ (21/5 ∗ )	n/a
2	B	36.93 / 36.93	n/a	n/a	n/a	$+0.37$ (28/6 ∗ )	n/a
3	A	36.62 / 36.60	$-0.10$ (11/17)	$-1.18$ (12/83 ∗ )	$-1.55$ (11/104 ∗ )	$+2.87$ (315/143 ∗ )	n/a
3	B	34.63 / 34.72	$-0.27$ (2/18 ∗ )	n/a	n/a	$+1.67$ (299/199 ∗ )	n/a

Appendix

Table 19: Frozen-artifact controls on the formal InfoSeek artifacts (6,000 test items, paired against the R0 replay). R0 aggregate scores are within 0.1 point of the archived scores. The small replay differences are shown explicitly.

Replicate	Protocol	O11 selected / hook	O10 selected / default	O01 removed / hook	O00 removed / default
1	A	37.30	37.30 ( $+0.00$ )	35.52 ( $-1.78$ )	35.52 ( $-1.78$ )
2	A	39.13	39.12 ( $-0.02$ )	34.38 ( $-4.75$ )	35.15 ( $-3.98$ )
3	A	36.62	36.62 ( $+0.00$ )	30.55 ( $-6.07$ )	32.03 ( $-4.58$ )
1	B	36.12	36.12 ( $+0.00$ )	34.05 ( $-2.07$ )	32.88 ( $-3.23$ )
2	B	36.93	36.87 ( $-0.07$ )	34.37 ( $-2.57$ )	33.90 ( $-3.03$ )
3	B	34.63	34.63 ( $+0.00$ )	32.92 ( $-1.72$ )	33.38 ( $-1.25$ )

Appendix

Table 20: Contract-versus-extractor decomposition on the six formal InfoSeek artifacts (test accuracy, with paired differences against O11 in parentheses). Every paired difference of 1.7 points or more is significant at

p<10^{-5}

Configuration	InfoSeek	DocVQA
DSL default: name card / plain OCR	23.6	80.1
Store-only / OCR-only control	33.5	80.1
Sensitivity profile (3 / 2 builds)	33.7 to 34.0	84.9 / 85.7
Profile withheld (3 / 1 builds)	35.6 / 39.7 / 40.3	86.0
Measurements-only profile	34.0	n/a
Decision-menu profile	38.8	n/a

Appendix

Table 21: Compiler-input ablations (historical: earlier runtime and manual, budgets in Appendix F.1 ). The matched score-only comparison under the current protocol is in the text.

Setting	One-shot	HC	One-shot minus HC
BLINK reflectance	65.7	64.9	$+0.7$ [ $-7.5$ , $+9.0$ ], n.s.
BLINK correspondence	62.8	67.6	$-4.8$ [ $-9.1$ , $-0.7$ ], $p=.035$
BLINK depth	95.2	95.2	Identical verdicts
DocVQA	85.5	84.9	$+0.6$ [ $-0.3$ , $+1.4$ ], n.s.
InfoSeek	25.0	39.7	$-14.7$ [ $-15.8$ , $-13.5$ ]
LiveVQA	30.6	37.1	$-6.5$ , $p=4\times 10^{-14}$

Appendix

Table 22: Independent one-shot construction versus iterative HC (historical: one teacher call under the earlier manual with guidance removed, paired against the earlier single HC builds listed in Table 11 , without a rerun under the current runtime). Average weights the seven settings equally.

Figure 4: Probe improvements can fail to generalize. Seed-0 BLINK trajectories, with tasks in columns and folds in rows. Dashed lines show teacher-visible probe scores, solid lines candidate validation scores, and rings gate-selected rounds. Held-out test scores are printed below each panel as initial

\to

selected, not used for selection. Reflectance fold 2 loses test accuracy despite its gate improvement. These are the archived per-fold trajectories replotted at readable scale.

Setting	Initial $H_{0}$	Selected $H_{\mathrm{val}}$	Gain per build	Selected, protocol B
InfoSeek	33.5	37.7	$+1.5$ , $+2.2$ , $+8.8$	35.9
DocVQA	78.4	85.0	$+11.5$ , $+1.5$ , $+7.0$	79.8
SlideVQA	54.4	54.6	$+1.5$ , $+1.1$ , $-1.8$	55.7
LiveVQA	34.1	38.6	$+5.6$ , $+3.9$ , $+4.1$	37.3
BLINK reflectance	61.7	62.9	$+3.0$ , $+0.7$ , $+0.0$	61.2
BLINK correspondence	65.2	65.7	$+0.5$ , $+0.7$ , $+0.2$	65.4

Appendix

Table 23: Build-level gains and the reflection comparison. Each initial-to-selected comparison uses the same build and validation-only selection. Gains are ordered by replicates 1–3. Protocol B starts from the same initial harnesses and uses the same gate. Average weights the seven settings equally. Individual gains use unrounded scores.

Setting	Replicate	$H_{0}$	selected (A)	selected (B)	round, A	round, B (no-ops)	$ A	$ B
InfoSeek	1	35.8	37.3	36.1	1	3 (0)	1.19	0.54
InfoSeek	2	36.9	39.1	36.9	1	0 (0)	0.83	0.57
InfoSeek	3	27.8	36.6	34.6	1	2 (0)	0.91	0.71
DocVQA	1	74.3	85.8	79.1	1	3 (0)	0.60	0.49
DocVQA	2	83.1	84.6	83.9	1	1 (0)	0.66	0.54
DocVQA	3	77.7	84.6	76.4	3	3 (0)	0.62	0.41

Appendix

Table 24: Formal builds: test scores of the shared

H_{0}

, the selected harness under the reported protocol (A) and under the incumbent-anchored variant (B), selected rounds (BLINK: per fold), and teacher cost in USD per arm. The

H_{0}

column is the arm-A re-evaluation. The arm-B re-evaluation of the same

H_{0}

agreed with it on 98.1–100% of items (DocVQA lowest), an observed replay agreement under batched inference rather than a bound on future variability.

Component	A: last proposal	B: accepted incumbent
Edit target	Latest proposal, accepted or rejected	Current incumbent, supplied as full JSON with its hash
Probe report	Latest proposal’s report; paired traces against the preceding proposal	Incumbent’s report; latest rejection shown separately with a compact paired report against the incumbent
History	Scores and accepted incumbent identified in the round history	Parent/proposal/incumbent hashes, changed components, scores and gate decision
Edit instruction	Return a revised complete harness	One coherent mechanism change, evidence and denominator, and a possible regression; unchanged incumbent is allowed
Stopping	Up to three reflection rounds	Up to three rounds; stop after two consecutive no-ops
Selection	Strict validation improvement, with bare floor	Same gate

Appendix

Table 25: Reflection procedures compared on the same runtime. B is also called reflection protocol v2 in the experiment records. This name is independent of runtime interface v2.

Setting	Score change	95% interval	Paired $p$ -value
InfoSeek	$-8.7$	–	$<0.001$
DocVQA	$-3.6$	$[-4.8,-2.5]$	–
SlideVQA	$-13.6$	$[-15.6,-11.6]$	$<0.001$
LiveVQA	$-1.2$	–	$0.06$
BLINK reflectance	$-14.2$	–	$0.013$
BLINK correspondence	$-2.3$	–	$0.27$

Appendix

Table 26: Cost of removing compiled control. Differences are student-routed minus HC on matched test items, with negative values favoring HC. The single diagnostic harnesses differ from the formal means (Appendix A.1 ). A dash means that the interval or exact

p

-value was not reported. The average weights settings equally.

Decision	Intervention	Interpretation
Decide	Supply the resource only after a yes/no request	Tests activation under that request contract
Select	Show all candidates without the explicit pick or ranker	Changes the selection interface, not only the decision owner
Generate	Use a student-generated entity name for lookup	Tests open argument generation on InfoSeek
Integrate	Use raw content instead of rewritten content. On BLINK, use raw signals instead of the processed rule output	The intervention differs between content and perception tasks
Persist	Remove the answer contract and parser together (a bundled output interface). On BLINK, show the verdict as a suggestion	Tests the output contract and abstention control, or verdict emission; separate formal contract/extractor controls are in App. A.5

Appendix

Table 27: Definitions of the decision-interface interventions.

Figure 5: Decision ownership in historical diagnostic programs. Rates are program-specific, not formal means or capability scores. Interventions: Table 4 .

Student	Bare	Transplant	Own build	Difference	Paired $p$ -value
Qwen3-VL-2B	13.2	30.1	32.1	$+2.0$	$<0.001$
Qwen3-VL-8B	20.3	39.7	38.0	$-1.8$	$<0.001$
Gemma-4-E4B	6.7	30.8	33.0	$+2.2$	$<0.001$
InternVL3.5-4B	9.6	30.9	26.2	$-4.7$	$<0.001$
Qwen3.5-0.8B	6.8	16.1	27.3	$+11.3$	$<0.001$
Qwen3.5-2B	9.0	30.7	26.0	$-4.7$	$<0.001$

Appendix

Table 28: InfoSeek transfer and recompilation at 450 practice items under interface v2. The transplant is the earlier single interface-v2 Qwen3-VL-4B harness (37.8 on its source student) evaluated unchanged. Own builds use the same protocol (restart rule, bare floor, no profile). Two-sided exact McNemar tests compare the own build with the transplant on 6,000 matched test items. Values below 0.001 are shown as

<0.001

. These per-comparison tests do not measure variation across teacher draws. Average weights the nine receiving students equally. Each own build is a single teacher draw.

Student	Pick / net	“0”	Routed gain	$H-S$ ( $p$ )	$S-$ bare
Qwen3-VL-2B	57.0 / 55.3	1%	$+3.8$	$+2.2$ ( $10^{-5}$ )	$+17.1$
Qwen3-VL-4B	66.0 / 58.0	17%	$+7.5$	$+6.3$ ( $5\!\times\!10^{-36}$ )	$+15.5$
Qwen3-VL-8B	61.2 / 55.7	12%	$+6.6$	$+6.1$ ( $5\!\times\!10^{-31}$ )	$+13.8$
Gemma-4-E4B	49.3 / 47.0	8%	$+1.9$	$+3.1$ ( $3\!\times\!10^{-10}$ )	$+22.3$
InternVL3.5-4B	50.8 / 48.3	21%	$+0.5$	$+2.6$ ( $3\!\times\!10^{-6}$ )	$+19.9$
Qwen3.5-0.8B	39.8 / 39.7	0%	$+0.7$	$+3.2$ ( $3.5\!\times\!10^{-10}$ )	$+11.8$

Appendix

Table 29: InfoSeek decision diagnostics for the 1,200-item source harness. Pick/net precision is measured on a 300-item probe (CLIP top-1 approximately 52%). “0” is abstention on routed test items.

S

uses the reference store, and routed gain is measured against

S^{\prime}

using the harness’s own store.

student	“0”	linked-correct on fired $S\to H_{B}$	acc. on fired $S\to H_{B}$	acc. not fired $S\to H_{B}$	$H_{B}-S$
Qwen3-VL-2B	1%	32.5 $\to$ 39.6	20.8 $\to$ 25.1 ( $+4.3$ )	44.7 $\to$ 43.8 ( $-0.9$ )	$+2.2$
Qwen3-VL-4B	17%	32.5 $\to$ 42.8	23.4 $\to$ 31.9 ( $+8.5$ )	48.7 $\to$ 51.7 ( $+3.0$ )	$+6.3$
Qwen3-VL-8B	12%	32.5 $\to$ 43.7	24.2 $\to$ 31.8 ( $+7.6$ )	49.1 $\to$ 52.9 ( $+3.8$ )	$+6.1$
Gemma-4-E4B	8%	32.5 $\to$ 33.0	20.0 $\to$ 22.4 ( $+2.4$ )	42.8 $\to$ 47.0 ( $+4.2$ )	$+3.1$
InternVL3.5-4B	21%	32.5 $\to$ 31.4	20.8 $\to$ 22.1 ( $+1.3$ )	42.7 $\to$ 47.2 ( $+4.5$ )	$+2.6$
Qwen3.5-0.8B	0%	32.5 $\to$ 25.1	12.9 $\to$ 14.3 ( $+1.4$ )	27.1 $\to$ 33.0 ( $+5.9$ )	$+3.2$

Appendix

Table 30: Store-only (

S

) vs discriminate harness (

H_{B}

) on the same test items, split by whether the router fires (60%, CLIP margin small) or not (40%). “0” = share of fired items on which the student takes the escape option.

Student	DocVQA bare	OCR $\Delta$	Reflectance	Correspondence	Depth
Qwen3-VL-2B	91.4	$-3.7$	29.9 $\to$ 61.9	51.3 $\to$ 58.1	74.2 $\to$ 93.5
Qwen3-VL-4B	94.9	$-0.7$	50.8 $\to$ 64.9	54.9 $\to$ 67.6	82.3 $\to$ 95.2
Qwen3-VL-8B	95.3	$-0.7$	–	–	–
Gemma-4-E4B	72.7	$+12.2$	46.3 $\to$ 64.2	39.5 $\to$ 62.6	64.5 $\to$ 92.7
InternVL3.5-4B	–	–	–	–	–
Qwen3.5-0.8B	84.9	$-10.3$	28.4 $\to$ 62.7	0.0 (17.9) $\to$ 57.4	67.7 $\to$ 93.5

Appendix

Table 31: Cross-task transfers on held-out test items. DocVQA uses the Gemma-compiled OCR harness, and BLINK uses the 4B’s fold-specific harnesses. Correspondence bare scores in parentheses use 96 rather than 16 decode tokens. Dashes denote unreported comparisons.

Student	Bare	Transplant	Initial	Selected	$p$ vs bare	Teacher $
Qwen3.5-2B	91.5	82.7	91.5	91.6	–	0.81
Qwen3.5-0.8B	84.9	56.1	85.7	85.8	–	0.46
MiniCPM-V-4.6	86.1	71.0	13.7	86.3 (bare retained)	–	1.04
Average	87.5	69.9	63.6	87.9	–	0.77

Appendix

Table 32: Fresh DocVQA builds under the current protocol (interface v2, reflection A, strict gate with the bare floor and up to two round-0 restarts). The transplant is the formal replicate-1 Gemma harness evaluated unchanged. The two Qwen3.5 builds turn OCR off. For MiniCPM-V-4.6, every proposal was rejected and the gate retained the bare model (after two restarts, with the third initial harness breaking the parser). Selected scores are not significantly different from bare. Average weights the three students equally.

Evidence / setting	Observation	Supported interpretation
BLINK diagnostic builds	HC trails the teacher by 14.9 points on reflectance, 20.1 on semantic and 12.3 on functional correspondence; depth is 5.6 higher (n.s.).	Available measurements support some decisions but do not close every task gap (Fig. 8 ).
Current compute control	CoT with voting reaches HC on reflectance and correspondence, with much higher generated-token counts (Table 8 ).	Some gains can also be obtained by increasing the student’s reasoning compute.
Exploratory MathVista / IQ Test	MathVista: best tested tool adds 4.2 against a 19.2 teacher gap. IQ Test: teacher 51.3 versus 4B 24.0 on 150 items; no suitable carrier identified.	A teacher advantage alone does not establish a usable external channel. Unrecovered gap is not an isolated measure of reasoning.
Earlier InfoSeek lookup	Student-routed lookup: $+1.6$ versus $+5.9$ for mechanical linking; bounded candidate selection: $+6.9$ .	Open argument generation and bounded selection impose different demands.
Earlier code-tool diagnostic	MathVerse: self-routed interpreter adds 10.2 for the 4B but loses 3.6 for InternVL; execution depends on the generated code.	A computation tool still requires valid, grounded arguments from its executor.
Earlier advice-block diagnostic	On WeMath, a failure-derived standing block costs the 4B 12.8 points; its benefit over generic advice is not significant.	Added guidance can interfere with reasoning under a particular model and prompt contract.

Appendix

Table 33: Evidence summary for decision constraints and reasoning limits. Current results and exploratory diagnostics are separated from earlier measurements, which used different splits or runtimes and are not pooled with the main HC comparisons.

Figure 7: Content, placement and compiler version are separate comparisons. InfoSeek

\times

Qwen3-VL-4B, test

n=6{,}000

. Names-only uses the cleaned store, while teacher cards and tuned human placement use the reference store. The two 150/300 builds use the same seed, splits and store, with different loop procedures. The restart-rule draw started sound without triggering a restart. Points are named arms, not an additive decomposition. The weight-trained and gold-linked references are separate regimes.

Harness	Store	Routed	Routed gain (linking)	Rest gain	Total
Store-only $S$ / $S^{\prime}$	Ref. / rewritten	–	–	–	33.5 / 34.3
Tuned human pick	Ref.	100%	–	–	37.4
HC build 1 (1,200 items)	Rewritten	60%	$+7.5$ (32.5 $\to$ 42.8)	$+2.3$	39.7
HC build 2	Ref.	68%	$+8.0$ (36.7 $\to$ 46.2)	$+4.4$	40.3
HC build 3	Ref.	61%	$+4.2$ (34.0 $\to$ 44.6)	$-1.1$	35.6
HC, decision-menu profile	Ref.	76%	$+6.5$ (38.8 $\to$ 48.4)	$+1.8$	38.8

Appendix

Table 34: InfoSeek

\times

Qwen3-VL-4B: the bounded pick decomposed.

S

uses the reference store, and

S^{\prime}

uses build 1’s rewritten store. Gains are relative to

S^{\prime}

for build 1 and

S

otherwise. Parentheses report linked-correct rates before

\to

after the pick. “Rest” means items answered without the pick.

Configuration	Practice items	ANLS
Bare	None	72.7
Human: plain OCR, 4,000 characters	Tuned on 500	80.1
Human: typed OCR field	Tuned on 500	82.4
Unoptimized DSPy typed program	None	86.2
One-shot teacher harness	None	85.5
HC, 50 probe / 50 validation	100	84.4

Appendix

Table 35: DocVQA baselines with Gemma-4-E4B. Scores are ANLS on 3,568 test items.

Figure 8: Perception gains and remaining teacher gaps differ by task. (A) Bare, human rule, HC , matched-label LoRA and GPT-5.5 at medium effort on the same items. HC and LoRA pool held-out folds. LoRA uses 89 / 291 / 82 labels per fold. (B) The correspondence subsets, not additional independent benchmarks. Semantic and functional gaps remain large, and no HC build significantly exceeds its fixed human rule.

Setting	Harness	Verdict share	Verdict / bare	Fallback / bare	Total
Reflectance	HC seed 0	92%	65.9 / 50.4	54.5 / 54.5	64.9
	HC seed 1	100%	61.9 / 50.7	–	61.9
	Human	84%	60.2 / 50.4	52.4 / 52.4	59.0
Correspondence	HC seed 0	55%	83.1 / 65.0	48.5 / 42.4	67.6
	HC seed 1	65%	76.0 / 59.9	47.4 / 45.5	66.0
	Human	93%	65.0 / 55.0	53.3 / 53.3	64.2

Appendix

Table 36: BLINK verdict and fallback subsets (three held-out folds pooled). Paired scores compare the harness with bare on the same subset. A dash denotes no fallback items.

Student	Bare	Human	$H_{0}$	$H_{\mathrm{val}}$	Top-1	Gold
Qwen3-VL-4B (interface v2)	14.7	25.5	37.5	38.1	27.4	51.6
Qwen3.5-4B	14.1	16.6	35.2	35.2	24.3	44.5
Qwen3.5-9B	16.9	22.5	37.9	36.1	29.9	55.1
MiniCPM-V-4.6	7.1	13.8	22.5	30.1	5.0	8.3
Average	13.2	19.6	33.3	34.9	21.7	39.9

Appendix

Table 37: LiveVQA test accuracy (

n=2,000

), judged by Claude Sonnet 5. Human selects one headline, top-1 injects the mechanically linked article, and gold supplies the gold article. HC beats human (

p<10^{-35}

) and top-1 (

p<10^{-5}

) for each student. Average weights the four students equally. All four HC rows are single builds under interface v2 with reflection A (the Qwen3-VL-4B row is the diagnostic reference, not the three-build main-table mean, and the other three were recompiled under the current protocol for this version, replacing earlier-interface builds). The 9B’s validation gate, which optimises containment, selected a round-3 harness that the judge scores 1.8 below its initial harness.

Student	Bare	Human	Initial HC	Selected HC	Transplant
Qwen3-VL-4B	44.7	48.7	41.9	56.7	n/a
Qwen3.5-4B	52.6	38.0	48.2	61.4	50.7
Qwen3.5-9B	51.4	47.4	56.4	62.1	63.3
MiniCPM-V-4.6	34.8	33.0	28.4	38.8	29.4
Gemma-4-E4B	43.2	35.8	38.5	48.4	43.0
Average (4 recipients)	45.5	38.6	42.9	52.7	46.6

Appendix

Table 38: SlideVQA adaptation across students. Exact match on 2,215 test items. Average uses the same four receiving students in every column, excluding the source Qwen3-VL-4B.

Student	Bare, 224 pixels	Top-three images	Gold slides
Qwen3-VL-4B	32.6	48.7	58.6
Qwen3.5-4B	36.7	33.6	37.0 a
Qwen3.5-9B	n/a	31.2	55.1 a
MiniCPM-V-4.6	n/a	20.1	26.6 a
Gemma-4-E4B	34.6	39.0	47.6

Appendix

Table 39: SlideVQA input controls. Gold-slide controls marked a use the earlier human contract.

bed	setting	prompt text (verbatim)	test
InfoSeek, recognise then read	stage 1: the three CLIP candidate names as options, store of name cards; stage 2: the two Wikipedia passages of the picked entity most similar to the question (MiniLM), fallback to the CLIP top-1 entity on abstention; 64 decode tokens	stage 1: “Which of these is the entity shown in the image?\n{options}\n0. None of them\nAnswer with the number only.”; stage 2: “Retrieved from Wikipedia (about the entity the retriever believes is shown; may be the wrong entity):” + passages + “Answer the question with a single word, number, or short phrase.”	33.8
DocVQA, typed OCR field	page image + question + 4,000 characters of Tesseract OCR in the DSPy typed-fields serialisation; generic field extractor; 96 decode tokens	context frame “Context (OCR text of the page; may contain errors):\n{tool_text}”; objective “Answer the question about the document image with a single word, number, or short phrase, copied from the page.”	82.4
BLINK reflectance	Retinex log-reflectance ( $\sigma{=}15$ ) at A and B; $\|\Delta\|<0.08\to$ “same”, else the lower one is darker; verdict answers the item, else the student	tool text shown to the student on deferred items: “Local reflectance estimate (brightness relative to surroundings): A = %.2f, B = %.2f”	59.0
BLINK correspondence	LightGlue projects REF into image 2; if the nearest candidate is within 8% of the diagonal, pick it; else the candidate with the highest DINOv2 $3{\times}3$ cosine to REF; defer if no signal	“Keypoint matcher projects REF next to point %s” / “Feature similarity to REF: A 0.xx, B 0.xx, …”	64.2
BLINK depth	Depth-Anything-V2-Small at radius 4; larger = closer; defer if a marker is missing	“Depth estimator: relative closeness at A = %.2f, at B = %.2f (larger = closer)”	94.4
LiveVQA, pick a headline, read it	the three CLIP-linked article titles as options, the picked article’s card ( $k{=}1$ )	the InfoSeek stage-1 prompt over article titles; card frame “Retrieved from your memory store (may or may not describe the right entity):\n{cards}”	25.5

Appendix

Table 40: Human heuristics: settings, routing and verbatim prompt text. LiveVQA scores use the Sonnet judge.

bed $\times$ student	vote	CoT+vote	TTAug ( $\Delta$ vs own ctrl)	$H_{\mathrm{val}}$
DocVQA $\times$ Gemma-4-E4B	72.5	80.4	74.8 ( $+1.6$ )	85.7
SlideVQA $\times$ 4B	44.6	51.1	46.8 ( $+2.1$ , $N{=}8$ )	58.1
InfoSeek $\times$ 4B	17.6	24.6	–	37.3
BLINK reflectance $\times$ 4B	50.8	64.9	51.5 ( $-1.5$ )	64.2
BLINK correspondence $\times$ 4B	54.0	68.5	56.9 ( $+2.7$ )	67.4
BLINK depth $\times$ 4B	82.3	85.5	86.3 ( $+4.0$ , n.s.)	96.0

Appendix

Table 41: Compute-only baselines (test, same items). The

H_{\mathrm{val}}

column is the formal replicate-1 harness of each setting re-executed with token logging (fold artifacts pooled on BLINK). Bare scores are in Table 2 . Vote = majority over 5 sampled direct answers (9 on BLINK). CoT+vote uses reasoning-first rollouts. TTAug = 16 augmented views with token-level averaging ( Kaya et al., 2026 ) , paired against its own single-view control on the same backend (

\Delta

shown). Generated-token costs for these same comparison runs are in Table 8 . Average weights all six settings equally for vote, CoT+vote and HC. TTAug is omitted because InfoSeek is missing.

arm	InfoSeek $\times$ 4B	DocVQA $\times$ Gemma
RAG-wiki	30.9	–
RAG-wiki (gold entity)	48.8	–
GEPA-prompt	17.9 $\to$ 19.3	82.6 $\to$ 82.9
GEPA-content (fixed store / OCR)	32.9 $\to$ 33.3	86.2 $\to$ 86.2
GEPA-structured (two modules)	34.9 $\to$ 36.0	–

Appendix

Table 42: External baselines (test). GEPA uses DSPy 3.3.1, a multimodal proposer, 3,000 metric calls per run and GPT-5.5 reflection. Arrows show the unoptimised seed

\to

the selected program.

search	seed 0	seed 1	seed 2	mean (SD)
HC loop	34.1	40.3	35.6	36.6 (3.3)
GEPA	33.5	37.8	36.6	35.9 (2.2)
HC $-$ GEPA, paired	$+0.6$ (.24)	$+2.5$	$-1.0$ (.037)	$-0.2$ (matched)

Appendix

Table 43: Search-step ablation on InfoSeek

\times

Qwen3-VL-4B, fixed store, no profile (test

n=6{,}000

). Parentheses in the paired row give McNemar

p

-values. Seed 1’s HC build used a 600-item gate, so the matched difference averages seeds 0 and 2 only. Both arms ran under the earlier runtime interface (v1). The comparison is historical and was not repeated under the current admissibility checks, which GEPA’s seed-2 program (hard-coded entity names) would fail.

Setting	Input	Plain	Typed
DocVQA, Gemma-4-E4B	Bare	72.7	81.7
	OCR recipe	82.4	85.7
DocVQA, Qwen3-VL-4B	Bare	94.9	93.6
InfoSeek, Qwen3-VL-4B	Bare	18.0	18.0
	Name card	23.6	22.9
	Store only	33.5	33.4

Appendix

Table 44: Typed fields with the same items and content. Only serialization changes.

Setting and student	ViperGPT-S		DFSDT-S	Tool use	HC
	Prose	Example
BLINK reflectance, 4B	44.0	31.3	44.8	0%	64.9
BLINK depth, 4B	83.9 a	78.2	81.5	0%	95.2
BLINK correspondence, 4B	42.6	51.5	58.1	0%	67.6
DocVQA, Gemma-4-E4B	75.0	n/a	75.2	81%	84.9
DocVQA, Qwen3-VL-4B	44.5	n/a	95.0	97%	n/a

Appendix

Table 45: Student-driven agent controls with the same tools. Scores use the test splits. Average gives equal weight to the five settings with HC results. Only the three performance columns available on all five are averaged.

Setting	Reasoning setting	Items	Score
DocVQA	Medium	32	81.2
DocVQA	Low	58	65.5
InfoSeek	Reported pilot	44	29.5

Appendix

Table 46: Teacher-written programs on covered practice items. These are not full-test comparisons.

Figure 9: Budget, build variation and loop version should not be conflated. HC dots are individual builds, with hollow diamonds marking the restart-rule procedure. Training points use the same unique practice items where matched, with 100-item training means where reported. Hollow training markers are 10-epoch variants. HC counts include probe and gate. The 100-item HC points use the current protocol, while the larger-budget points are the earlier diagnostic builds. The formal means are in Table 2 . Dots are slightly offset horizontally to reveal repeated seeds. Different allocations and procedures are not connected into a scaling law. Full splits and accepted trajectory counts are in Tab. 47 .

Setting	HC probe / gate	HC	Answer-LoRA	Trajectory SFT
InfoSeek	50 / 50	33.6 (38.0, 25.9, 36.9)	20.0 (3 ep) / 23.3 (10 ep)	30.2 (3 ep) / 35.3 (10 ep)
	150 / 300, original	34.1 / 35.6	23.7	36.9
	150 / 600	40.3	–	–
	600 / 600	39.7	–	40.7 (1,199 accepted)
DocVQA	50 / 50	84.7 (82.8, 84.7, 86.6)	81.1	–
	150 / 300	85.6	–	–

Appendix

Table 47: Practice budgets and weight adaptation. HC counts probe / validation items. At 100 items, entries are three-seed means under the current protocol with the individual builds in parentheses (Fig. 9 ). Epoch counts are shown when varied. Training counts are stated where they differ from the row budget. The DocVQA 500/500 setting contains 999 unique items.

Figure 10: Large external training sets can overtake a frozen harness. (A) LoRA trained on contamination-audited IIW pairs. Dots show all available seeds, with a line through budget means only as a visual guide. HC uses 44 probe plus 45 gate BLINK items per fold. The teacher reference is 79.9 on the full split at medium effort. (B) LoRA trained on the same 89 BLINK practice items as HC , a distinct data regime.

Figure 11: Weight adaptation can reverse a harness component’s value. The old reflectance harness suppresses the 20k-label LoRA. A fresh cross-build removes the tool in two folds and uses a conservative router in the third. Recompilation consumes additional BLINK practice labels. Its

+1.5

over the tuned bare student is not significant.

Policy	Bare	Canonical only	Base $H_{0}$	Own $H_{\mathrm{val}}$
SmolVLA-0.45B	3.8	33.2	37.0	34.5
SmolVLA, tuned on perturbations	11.8	59.0	64.7	63.2
X-VLA-0.9B	59.7	60.7	63.3	–

Appendix

Table 48: Exploratory policy evaluation (600 episodes). “Base

H_{0}

” is the first SmolVLA harness, reused unchanged on the other checkpoints. It is a diagnostic artifact, distinct from the validation-selected own build.

Checkpoint	arm	valid completion (%)	false aborts	correct refusals / 40	mixed
SmolVLA base	bare	4.1	0	0	3.8
SmolVLA base	canonicalization	35.5	0	0	33.2
SmolVLA base	human recipe v2	30.7	106	9	30.2
SmolVLA base	HC $H_{0}$	39.6	0	0	37.0
SmolVLA base	HC $H_{\mathrm{val}}$ (round 2)	35.9	62	6	34.5
SmolVLA plus-tuned	bare	12.7	0	0	11.8

Appendix

Table 49: LIBERO-plus outcomes decomposed from the archived records: completion on the 560 valid episodes, false aborts on valid episodes (all within the first quarter of the budget), correct refusals among the 40 mismatched instructions (abort within the first quarter), and the mixed score of Table 48 . Human recipe v2 refuses by detection and pays for it with 106 false aborts.

Replicate	arm	round	mixed	completion (%)	false aborts	refusals /40	vs own $H_{0}$ rollout
1	$H_{0}$ (rollout in A)	–	28.8	28.2	200	15	–
1	A selected	0	29.3	28.8	200	15	$+0.5$ (39/36, $p=0.818$ )
1	$H_{0}$ (rollout in B)	–	29.7	29.1	200	15	–
1	B selected	2	34.3	35.9	91	5	$+4.7$ (86/58, $p=0.0241$ )
2	$H_{0}$ (rollout in A)	–	23.7	22.3	279	17	–
2	A selected	1	36.3	37.3	71	9	$+12.7$ (111/35, $p=2.09e-10$ )

Appendix

Table 50: LIBERO-plus matched protocol comparison on the base SmolVLA checkpoint: test outcomes of the shared

H_{0}

(re-rolled in each arm) and of the harness each protocol selected, with completion on the 560 valid episodes, false aborts, correct refusals among the 40 mismatched instructions, and the paired difference against the arm’s own

H_{0}

rollout (exact McNemar).

Setting	HC calls/item	HC output tokens	CoT output tokens	Teacher $ / build
InfoSeek	1.6	4.9	262	6.9 (with store)
DocVQA	1.0	11.2	97	0.9
BLINK reflectance	0.08	0.1	237	2.5
BLINK correspondence	0.45	0.7	278	3.0
BLINK depth	0.07	0.1	255	1.8

Appendix

Table 51: Representative VQA runtime costs, measured on historical reference records and not re-measured on the formal interface-v2 artifacts. HC output tokens include all student passes. Calls below one reflect direct code verdicts. CoT is a single reasoning rollout. The multi-sample CoT+vote comparison is in Table 8 . The InfoSeek and DocVQA cost records correspond to the earlier 39.7- and 85.7-score builds, respectively.

Explore similar work

May 12, 2026cs.SE

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - are applied to three models (Gemma4 E2B, Qwen3.5:2B, LLaMA 3.2 3B) across 24 tasks, comparing Task Success Rate (TSR) and Valid TSR (VTSR). The pipeline harness achieves TSR=0.952 and VTSR=1.000 on Gemma4 E2B (T1-T5, 21 tasks). A non-monotonic phenomenon - minimal-shell TSR < model-only TSR - is observed in two models. In LLaMA 3.2 3B model-only, seven format violations yield TSR=0.429, revealing scaffold collapse: the model abandons JSON structure under complex format requirements without harness support. Ablation shows planning and recovery each contribute approximately 24.7% of total gain. VCR (Verification Catch Rate)=0.625 across all pipeline runs.

Yong-eun Cho

KailosLab, Seoul, Republic of Korea

Small Language Models

Ask PDF

Jul 10, 2026cs.CL

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear. We examine this on EXAMS-V, a multilingual visual multiple-choice benchmark, comparing self-consistency, describe-then-reason with PRM-guided beam search, and two post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B. What matters is the conditions under which TTS runs, not the search or verification machinery. The largest factor is parseability: an early prompt format left many chains reasoning correctly yet never committing to an answer letter, which a standard answer cue and a guided repair step largely remove. A larger decoding budget removes the rest: raising the per-chain token limit from 1k to 2k recovers 3.7 pp, whereas sampling more chains (8 to 16) adds only 0.15 pp. Once chains have room to finish, elaborate methods contribute little: PRM-guided beam search trails plain self-consistency by 0.39 pp at over eight times the cost, and neither a training-free generative critic nor a trained multimodal PRM beats majority vote across both policies. The largest gain comes instead from the policy model itself (+11.4 pp). Our best configuration reaches 84.1% on the held-out ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard.

Spiros Baxevanakis, Peng-Jian Yang

University of Amsterdam, Science Park 904, 1098 XH Amsterdam, The Netherlands

Visual Question Answering Test-Time Scaling for VLMs Test-Time Scaling Multilingual VLM Evaluation VLM Reasoning Multimodal Foundation Models

Ask PDF

Sep 28, 2026cs.CV

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM Inference VLM Interpretability Latent Visual Reasoning VLM Reasoning

Ask PDF