VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It

Binxiao Xu; Ruichuan An; Bocheng Zou; Weihua Du; Xinyu Wei; Hangyu Guo; YiFan Zhang; Renye Yan

cs.CVOct 11, 2026

VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It

Authors: Binxiao Xu, Ruichuan An, Bocheng Zou, Weihua Du, Xinyu Wei, Hangyu Guo, YiFan Zhang, Renye Yan, +2 more

Organizations: PKU · UW–Madison · CMU · PolyU · CASIA · HKU · MIT-IBM Computing Research Lab

Abstract

Reusable skills improve multimodal agents without updating model weights, but their effectiveness depends on both the evidence presented and the procedure used to read it. We introduce ViewCraft, which jointly evolves an executable evidence interface and its reading procedure. The interface selects, transforms, and organizes observations; the reader interprets the resulting views. Failure traces guide revisions, replay and paired evaluation retain useful changes, and interface projection recovers visual edits from eligible rejected joint proposals. Across eight benchmarks, ViewCraft improves over direct execution by an average of $6.1$ points in task score, with gains on every benchmark. The advantage persists at a matched input-token budget. On all five multi-source tasks, the learned interface uses 8%-15% fewer input tokens than Initial skill. The learned skills also transfer across executors and outperform direct execution in every evaluated transfer, supporting evidence construction and reading as a coupled, reusable skill.

Figures & tables

Figure 1: Joint evolution of evidence interfaces and reading procedures. Left: early frames recover pre-shuffle letter order, endpoint rows expose motion, and a background grid supports counting. Center/right: gains over Initial skill use a shared $0$ – $15$ point radial scale (mean $8.2$ ; range $4.3$ – $12.4$ across eight tasks). On all five multi-source tasks, input tokens fall $8$ – $15\%$ from Initial skill.

Figure 2: Four probes of the evidence interface. (a) Accuracy for three selectors at four frame budgets on 96 questions from 26 non-test films. (b) Mean accuracy across these 12 settings with plain and enhanced prompts. (c) With their frames replaced by another film, 12 of 64 previously correct questions are answered correctly (Wilson 95% CI

[11\%,30\%]

). (d) Mean field-value ANLS on 240 training-form crops at four delivered resolutions. All probes are independent of Table 2 ’s test sets.

Method family	Visual support	Reader–visual co-edit	Executable visual edits	Replay + paired gate	Interface retry
Textual skills	–	–	–	–	–
Grounded multimodal skills		–	–	–	–
MPO [ 1 ]			–	–	–
Dynamo [ 2 ]				–	–
ViewCraft

Table 1: What becomes part of an editable skill? Columns track reuse, visual support, coupled editing, executable visual transformations, replay-screened paired evaluation, and interface-only retries. Checks denote explicit capabilities; a dash denotes a capability outside the described design. ViewCraft combines all six capabilities in its evolution loop.

Figure 3: Joint view construction and evolution. (a) $\phi$ selects/transforms evidence; $\psi$ arranges views and surface contracts. (b) Frozen $M$ follows $\theta$ . (c) Traces guide proposals. (d) Replay and guards precede the paired gate. Projection retries the interface with incumbent $\theta$ when support is non-degraded and budget remains. Every verdict updates strategy.

Method	MMComp	ReasonMap	FunSD	STI-Bench	VideoHolmes	SlideVQA	OpenEQA	MMIU	Mean
Direct	0.834	0.858	0.306	0.410	0.408	0.802	0.512	0.921	0.631
Initial skill	0.786	0.841	0.288	0.385	0.381	0.812	0.478	0.908	0.610
Text-only	0.848	0.876	0.372	0.430	0.424	0.837	0.528	0.926	0.655
Interface-only	0.779	0.856	0.341	0.466	0.452	0.834	0.545	0.935	0.651
$\theta$ -only	0.852	0.867	0.352	0.416	0.408	0.833	0.508	0.921	0.645
$(\phi,\psi)$ -only	0.784	0.850	0.325	0.448	0.430	0.828	0.526	0.926	0.640

Table 2: Held-out task scores and component contributions. Scores pool three runs under Table 6 ’s metrics. The Mean column averages tasks equally; the final row subtracts Direct. Bold marks column bests. Tints distinguish baselines, restored-coordinate states, and joint evolution; lavender and teal headers group single-source and multi-source tasks.

Search strategy	MMComp	RMap	FunSD	STI	VHolm	Slide	OEQA	MMIU	Mean
Direct (no search)	0.834	0.858	0.306	0.410	0.408	0.802	0.512	0.921	0.631
MPO-adapted (prompt space)	0.852	0.869	0.368	0.437	0.428	0.834	0.524	0.922	0.654
Dynamo-style (fixed tools)	0.841	0.865	0.380	0.440	0.426	0.840	0.534	0.925	0.656
Independent, then combine	0.857	0.878	0.378	0.468	0.452	0.842	0.551	0.933	0.670
Staged (visual, then $\theta$ )	0.859	0.880	0.385	0.472	0.455	0.846	0.556	0.938	0.674
Alternating ( $\theta\leftrightarrow\phi,\psi$ )	0.861	0.885	0.391	0.478	0.461	0.849	0.562	0.940	0.678

Table 3: Search strategies with matched submissions: 250 per arm across 24 task–seed runs; each run has four rounds and 96 training rollouts. Conditions share executor and interface. Scores pool three runs; filtering determines candidate evaluations.

Figure 4: Where the gain sits. (a) Per-benchmark held-out gains over a bare Direct call (all eight positive; median

+5.9

points). (b) Mean task score of the four single-coordinate controls against joint search (margins

+3.7

+5.2

points).

	MMComp	RMap	FunSD	STI	VHolm	Slide	OEQA	MMIU	Mean
MPO-adapted (prompt space)	0.852	0.869	0.368	0.437	0.428	0.834	0.524	0.922	0.654
Random proposals (same space, gate)	0.852	0.869	0.378	0.445	0.434	0.842	0.539	0.928	0.661
Dynamo-gen (tool generation)	0.857	0.876	0.381	0.459	0.444	0.835	0.545	0.903	0.662
MPO-fullspace (UCB selection)	0.861	0.878	0.385	0.463	0.450	0.838	0.549	0.907	0.666
Gate-only joint	0.861	0.887	0.393	0.478	0.457	0.844	0.560	0.932	0.676
ViewCraft	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951	0.692

Table 4: Baselines and value decomposition at matched budget (per arm: 250 submitted proposals; per task–seed run: four rounds, 96 rollouts). Random proposals samples uniformly from the same expression grammar under the same paired gate; MPO-fullspace and Dynamo-gen optimize ViewCraft’s executable interface; row 1 adapts prompt-space search; row 5 uses the full joint space and paired gate with replay, interface projection, and the evidence tiebreak disabled.

Figure 5: Matched search and co-adaptation. (a) Accepted co-edits across the three-arm log: circle area encodes pair counts and diagonal squares solo accepts; total accepts 204. (b) Full-model score minus each ablation, in points (Table 24 ). (c) Terminal interaction $I$ under the clause-content partition; bars show 95% seed-stratified media-cluster bootstrap intervals, including zero for SlideVQA.

Executor	Skill source	MMComp	FunSD	STI-Bench	VideoHolmes
	Direct	0.834	0.306	0.410	0.408
	gemini-3.8-flash	0.874	0.412	0.496	0.472
	gpt-5.5	0.868	0.385	0.508	0.465
gemini-3.8-flash	claude-sonnet-4-6	0.852	0.379	0.484	0.458
	Direct	0.850	0.271	0.567	0.550
	gemini-3.8-flash	0.871	0.388	0.623	0.598

Table 5: Skill transfer across executors. Bold marks the highest held-out score within each executor and task; tints group target executors. Self-transfer pools three runs; the Gemini diagonal reproduces Table 2 . Cross-transfer uses the source’s median-gate run. Direct uses the same pooling.

Figure 6: Ten repairs linking views to readings. Panels link questions, view edits, reading summaries, and answer changes.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Medium	Train	Gate	Test	Metric
MMComp	image	48 / 48	96 / 96	151 / 151	option-letter accuracy
ReasonMap	transit diagram	24 / 2	48 / 2	153 / 4	route accuracy
FunSD	form	64 / 58	96 / 72	248 / 47	field-value ANLS
STI-Bench	video	36 / 12	72 / 18	221 / 36	option-letter accuracy
VideoHolmes	video	24 / 10	72 / 16	183 / 28	option-letter accuracy
SlideVQA	slide deck	40 / 10	60 / 14	155 / 30	ANLS

Appendix

Table 6: Evaluation units, source-disjoint splits, and scoring. Cells report questions / unique source media. Train, gate, and test draws have pairwise-disjoint source media; no form, film, diagram, or deck appears in two splits. Gate questions are never shown to the proposal stage, which uses training traces only.

Gate scale (items/task)	8	25%	50%	100% (default)	200%
Mean task score	0.679	0.686	0.690	0.692	0.693

Appendix

Table 7: Gate-size sensitivity: mean task score when every benchmark’s gate set is scaled as shown, holding proposals, rollouts, and test sets fixed.

Method	Proposals	Gate evals	Exec. calls	Verif. calls	Optim. calls	Tokens
Shared initialization	0	8	0.7k	0	0	0.5M
Text-only / Interface-only	250	200	45.2k	38.8k	384	42.2M
ViewCraft	250	208	47.0k	40.4k	384	43.8M
Alternating / Staged / Indep.	250	200	45.2k	38.8k	384	42.2M
MPO-adapted / MPO-fullspace	250	200	45.2k	38.8k	384	42.2M
Dynamo-style / Dynamo-gen	250	200	45.4k	38.8k	384	42.4M

Appendix

Table 8: Per-arm development cost over 24 task–seed runs. Grouped methods each incur the listed cost. Tokens sum logged input tokens at measured per-call rates: executor 774, verifier 151, optimizer 3.4k. Shared initialization is executed once and reused by every arm.

Submitted (incl. 21 projections)	750
Pre-gate rejects (64 inert / 78 guards)	142
Paired calls (204 accepts / 404 rejects)	608
Candidate evaluations	608

Appendix

Table 9: Development-evaluation accounting.

	Submitted		Pre-filtered		Evaluated
	Ordinary	Projection	Inert	Guards	Ordinary	Projection
Joint	229	21	30	12	187	21
Text-only	250	0	0	50	200	0
Interface-only	250	0	34	16	200	0
Total	729	21	64	78	587	21

Appendix

Table 10: Submission and filtering ledger for the three main arms. Ordinary and projection columns sum to each arm’s submissions or evaluations. This aggregate excludes strategy and ablation variants.

Benchmark	Gap	95% CI	$p$	Benchmark	Gap	95% CI	$p$
MMComp	1.3	[0.1, 2.6]	0.06	VideoHolmes	1.1	[0.0, 2.4]	0.07
ReasonMap	1.3	[0.1, 2.6]	0.19	SlideVQA	0.9	[ $-$ 0.2, 2.1]	0.12
FunSD	2.1	[0.8, 3.5]	0.01	OpenEQA	1.2	[0.2, 2.5]	0.09
STI-Bench	1.8	[0.5, 3.3]	0.04	MMIU	1.1	[0.0, 2.3]	0.08
Mean gap: 1.3 points 95% CI: [0.6, 2.1]

Appendix

Table 11: Joint versus alternating search. Gaps and 95% seed-stratified media-cluster bootstrap CIs are in points. Permutation tests exchange whole source media (10,000 resamples), using per-item means over three runs.

Ladder step	Gain	ViewCraft comparison	Margin
Prompt space to random same-space	+0.7	MPO-fullspace	+2.6
Random to UCB selection	+0.5	Dynamo-gen	+3.0
UCB to failure-guided joint	+1.0	Random proposals	+3.1
Gate-only to full retention	+1.6	Random to failure-guided joint	+1.5

Appendix

Table 12: Mean task-score increments along the matched-budget ladder, and final margins. All differences are in points.

Benchmark	ViewCraft (std)	Gain (pts)	Cluster CI (pts)	vs. Direct $p$
MMComp	0.874 $\pm$ 0.007	4.0	[0.3, 7.8]	0.03
ReasonMap	0.898 $\pm$ 0.008	4.0	[0.2, 7.9]	0.05
FunSD	0.412 $\pm$ 0.015	10.6	[5.8, 15.2]	$<$ 0.001
STI-Bench	0.496 $\pm$ 0.011	8.6	[2.9, 14.2]	0.006
VideoHolmes	0.472 $\pm$ 0.010	6.4	[1.2, 11.6]	0.01
SlideVQA	0.858 $\pm$ 0.007	5.6	[0.1, 11.1]	0.05

Appendix

Table 13: Uncertainty versus Direct. Std is over three search runs. Gain CIs are 95% seed-stratified media-cluster bootstraps; permutation tests use each item’s mean score over the three runs. Gains and intervals are in points.

Summary	Value	Holm-adjusted comparison	$p$
Mean gain (points)	+6.1	FunSD	0.004
Median gain (points)	+5.9	STI-Bench	0.042
Relative gain range (%)	3.3–34.6	Other tasks	$>0.05$
Median relative gain (%)	9.5

Appendix

Table 14: Across-task gain summaries and multiplicity correction versus Direct.

Variant	MM	RMap	FunSD	STI	VH	Slide	OEQA	MMIU	Mean	Gap [CI]	$p$
No retry (drop on joint rejection)	0.870	0.893	0.407	0.483	0.461	0.849	0.565	0.944	0.684	0.8 [0.3, 1.3]	0.004
Plain retry (fresh proposals)	0.872	0.893	0.408	0.484	0.463	0.850	0.566	0.944	0.685	0.7 [0.2, 1.2]	0.008
Symmetric projection (keep $\theta^{\prime}$ )	0.872	0.895	0.409	0.486	0.463	0.851	0.567	0.946	0.686	0.6 [0.1, 1.1]	0.020
No $\Delta C$ trigger (retry all eligible)	0.872	0.895	0.408	0.487	0.464	0.852	0.568	0.947	0.687	0.5 [0.1, 1.0]	0.030
Asymmetric projection (ours)	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951	0.692	—	—

Appendix

Table 15: Projection design at matched budget (per arm: 250 submitted proposals). Gap is full ViewCraft minus the variant in points; CIs are 95% seed-stratified media-cluster bootstraps of the paired per-item difference;

p

is a paired permutation test on per-item means (10,000 resamples).

Condition	MMComp	RMap	FunSD	STI	VHolm	Slide	OEQA	MMIU
ViewCraft	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951
format-normalized	0.876	0.900	0.421	0.498	0.474	0.861	0.581	0.953
Evidence-only (contracts frozen)	0.872	0.895	0.401	0.492	0.468	0.854	0.566	0.947
format-normalized	0.872	0.895	0.409	0.495	0.470	0.857	0.574	0.947

Appendix

Table 16: Isolating evidence from contract effects. “Evidence-only” freezes the answer and protocol contracts at their initial values and evolves only evidence selection, transformation, layout, and the paired reading rules. “Format-normalized” rescores all responses case-/unit-insensitively with letter extraction.

Replayed state	MMComp	RMap	FunSD	STI	VHolm	Slide	OEQA	MMIU	Mean
Initial skill $(\theta_{0},v_{0})$	0.786	0.841	0.288	0.385	0.381	0.812	0.478	0.908	0.610
Evidence transplant $(\theta_{0},v_{1})$	0.775	0.845	0.318	0.440	0.424	0.823	0.519	0.922	0.633
Reading transplant $(\theta_{1},v_{0})$	0.848	0.861	0.345	0.410	0.403	0.829	0.503	0.918	0.640
ViewCraft $(\theta_{1},v_{1})$	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951	0.692

Appendix

Table 17: Fixed-trajectory decomposition. Each row replays the terminal skill’s recorded observation sequence and substitutes one coordinate. The evidence transplant delivers the evolved views to the initial reader; the reading transplant gives the evolved reader the initial views.

Evidence transplant	+2.3
Reading transplant	+3.0
Joint pairing	+8.2
Joint-minus-transplants contrast	+2.9

Appendix

Table 18: Fixed-trajectory contrasts relative to Initial skill, in points. The final row is the joint gain minus the two separate transplant gains.

Partition	MM	RMap	FunSD	STI	VH	Slide	OEQA	MMIU	Mean
Storage fields	1.4	1.0	1.6	0.8	0.6	$-$ 0.3	0.5	0.8	+0.8
Strict (render clauses out of $\theta$ )	1.8	1.5	1.8	1.2	1.0	0.5	1.2	0.9	+1.2
Clause-content (used throughout)	2.4	2.2	2.3	1.7	1.5	0.9	1.8	1.2	+1.8

Appendix

Table 19: Terminal interaction

I

(points) under three coordinate partitions. Storage keeps exported files intact; clause-content assigns inference and decision rules to

\theta

; strict additionally moves render-referencing clauses (note placement, rendered-color references) to the interface.

Strategy	Split A	Split B	Difference (points)
Alternating	0.678	0.674	$-$ 0.4
Gate-only joint	0.676	0.672	$-$ 0.4
ViewCraft	0.692	0.688	$-$ 0.4
Joint $-$ Alternating (points)	+1.4	+1.4	0.0

Appendix

Table 20: Gate-split sensitivity. Differences use the displayed mean task scores; the final row reports joint minus alternating in points.

Figure 7: Search selectivity. Rates pool the three evolving arms and use submitted proposals, including projection retries and pre-gate rejects. Rejected replays and guard failures consume no development calls.

Figure 8: Two answer-contract regressions. In (k), the response retains the numbers but drops the labels required by the compound-field query. In (l), both responses denote 6 cm, yet only option letter C passes the recorded scorer.

Figure 9: Task scores and input-token cost. Left: score gains and token changes relative to Initial skill; negative horizontal values mean savings. Circles denote multi-source tasks, diamonds single-source tasks. Right: ViewCraft input tokens divided by mean score (a score-normalized cost for continuous metrics). Counts include all requests; Appendix Table 21 lists absolute counts.

Benchmark	Direct	Initial skill	ViewCraft	VC $-$ Direct
STI-Bench	2,380	3,210	2,730	+350
VideoHolmes	2,410	3,260	2,850	+440
SlideVQA	2,240	3,020	2,660	+420
OpenEQA	2,360	3,190	2,720	+360
MMIU	2,010	2,710	2,480	+470
MMComp	1,620	2,291	3,375	+1,755

Appendix

Table 21: Mean input tokens per item across all requests. Original counts determine Figure 9 and the appended ViewCraft-minus-Direct column. Relative to Direct, increases are

15

–

23\%

for multi-source tasks and

67

–

108\%

for single-source tasks; multi-source counts fall approximately

8

–

15\%

from Initial skill.

	Input tok/item	Output tok/item	Calls/item	Latency (s/item)
Direct	2.08k	38	1.0	4.2
Initial skill	2.84k	61	2.6	9.4
ViewCraft	2.86k	69	3.4	11.8

Appendix

Table 22: Deployment cost averaged equally over the eight tasks. Input means are the arithmetic means of Table 21 ; search cost is amortized over reuse.

	MMComp	RMap	FunSD	STI	VHolm	Slide	OEQA	MMIU	Mean	Tok/item
Direct	0.834	0.858	0.306	0.410	0.408	0.802	0.512	0.921	0.631	2.08k
Direct (token-matched)	0.841	0.863	0.318	0.421	0.417	0.809	0.521	0.926	0.640	2.86k
Direct $\times$ 3 (self-consistency)	0.848	0.867	0.311	0.418	0.413	0.808	0.519	0.929	0.639	6.24k
Initial skill	0.786	0.841	0.288	0.385	0.381	0.812	0.478	0.908	0.610	2.84k
ViewCraft	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951	0.692	2.86k

Appendix

Table 23: Task scores at matched and expanded input budgets. “Token-matched” gives Direct ViewCraft’s measured per-item input budget through content-independent means (more frames or pages on multi-source tasks; upsampling plus a regular crop grid on single-source tasks). “

\times

3” is self-consistency over three samples. Tokens/item is the task-equal-weight macro-average.

Variant	MM	RMap	FunSD	STI	VH	Slide	OEQA	MMIU	Mean	Gap [CI]	$p$
w/o replay filter	0.870	0.891	0.404	0.487	0.463	0.848	0.562	0.942	0.683	0.9 [0.4, 1.5]	0.002
w/o interface projection	0.870	0.893	0.407	0.483	0.461	0.849	0.565	0.944	0.684	0.8 [0.3, 1.3]	0.004
w/o evidence tiebreak	0.870	0.891	0.403	0.489	0.464	0.852	0.566	0.947	0.685	0.7 [0.2, 1.2]	0.008
Gate-only joint (all three off)	0.861	0.887	0.393	0.478	0.457	0.844	0.560	0.932	0.676	1.6 [0.8, 2.4]	$<$ 0.001
ViewCraft (full)	0.874	0.898	0.412	0.496	0.472	0.858	0.574	0.951	0.692	—	—

Appendix

Table 24: Mechanism ablations across eight tasks and three seeds at 250 submissions per arm. Paired intervals and tests follow Table 15 . Gate evaluations: full 208 (21 projections; 30 inert and 12 guard failures filtered), no replay 238, no projection 204, no tiebreak 208, and gate-only 200.

Explore similar work

Agent Skills Should Go Beyond Text: The Case for Visual Skills
May 31, 2026Binxiao Xu, Ruichuan An, Bocheng Zou +1Multimodal Grounding Agent Skill Learning
Peking University · University of Wisconsin · MIT-IBM Watson AI Lab
MMSkills: Towards Multimodal Skills for General Visual Agents
May 13, 2026Kangning Zhang, Shuai Shao, Qingyao Li +8GUI Agents LLM Agent Skill Learning
1Shanghai Jiao Tong University · 2Xiaohongshu Inc. · 3Southeast University
Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck
May 8, 2026Zihan Huang, Junda Wu, Tong Yu +6LLM Agent Skill Learning Multimodal Agents
UC San Diego · Adobe Research · UC Santa Barbara +1

May 31, 2026J/K move · Enter open · S save

Agent Skills Should Go Beyond Text: The Case for Visual Skills

Binxiao Xu, Ruichuan An, Bocheng Zou, Hang Hua

Peking University · University of Wisconsin · MIT-IBM Watson AI Lab

Multimodal Grounding Agent Skill Learning

Open paper AskPDF arXiv

Reusable skills are a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck for visual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose \NAME, a multimodal skill paradigm that combines declarative textual logic with explicit visual support. We distinguish three reusable forms: static priors for stable spatial conventions, dynamic priors for in-situ visual working memory, and interleaved visual skills that bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do, visual skills also encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce \SYSTEM, an automatic system that converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns from task trajectories. Experiments on GUI and other visual-centric tasks show that visual skills consistently outperform text-only skills, particularly when success requires spatial correspondence, visual evidence, and state-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.

Abstract

Explore similar work

Agent Skills Should Go Beyond Text: The Case for Visual Skills

MMSkills: Towards Multimodal Skills for General Visual Agents

Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck