VisCurate: Evaluating Behavioral Equivalence of Visual Skills

Md Farhan Ishraq; Md Farhan Ishmam; Reaz Hassan Joarder; Md Hasanul Kabir

cs.CVOct 21, 2026

VisCurate: Evaluating Behavioral Equivalence of Visual Skills

Authors: Md Farhan Ishraq, Md Farhan Ishmam, Reaz Hassan Joarder, Md Hasanul Kabir

Organizations: Islamic University of Technology · University of Utah

Abstract

Modern vision-based agents have shifted toward skill-based problem-solving, where a skill is an executable image-to-image operation, e.g., cropping, thresholding, or edge detection. As these skill libraries grow, they accumulate duplicated, mislabeled, and defective skills. The signals normally used to curate them (a skill's name, description, or source code) say little about how the skill visually changes the image. We present VisCurate, a framework and benchmark that curates visual skill libraries from observed execution rather than from text. VisCurate runs each candidate skill pair on a shared probe battery and assigns one of six behavioral relations, each tied to a library edit that is safe to make. The benchmark injects seven defect types into a clean library at controlled rates, producing 300 reproducible instances with ground-truth relations and repairs. While text baselines accept up to five of six false merges on hard negatives, our verifier accepts none and recovers 51.2% of the pairs that are truly mergeable. On two real, uncurated libraries, VisCurate finds that 92% of the skills a package shares with its downstream copy are redundant, while execution confirms only 10 of 17 published equivalence claims. The curation problem also remains open, as the strongest of the nine automated curators recovers only about half of the repairs. We emphasize that visual skill libraries should be curated by behavior, not by description.

Figures & tables

Figure 1: Overview of the VisCurate pipeline: (a) A noisy seed skill library is evaluated through an output-grounded equivalence verifier, which executes candidate skills on shared probes and certifies their behavioral relations from the resulting outputs. (b) These verified relations guide the structural edits proposed by the curation agent, which are validated before updating the library. The process repeats until a compact and organized skill library is produced.

Action	Meaning
add	Add a new skill
remove	Remove a skill
modify	Modify metadata/schema or function behavior
retrieve	Retrieve/query a skill
merge	Merge redundant skills
split	Split one skill into multiple skills

Table 1: Description of the action space of the curation agent.

Method	Reads	FPR dist (%) $\downarrow$	FM hard $\downarrow$
Name match	name	5.40	5
TF–IDF cosine	description	0.21	0
Sentence embedding	description	5.61	5
Code embedding	source	1.80	4
AST (structural)	source	0.00	0
AST (semantic)	source	0.00	0

Table 2: The baseline ladder on the

944

designed pairs of

G_{0}

at matched operating points. FPR dist : false-merge rate on truly- DISTINCT pairs, in percent. FM hard : false merges accepted on the six engineered hard negatives. VisCurate abstains on

1.5\%

of decisions; no baseline abstains. The source-reading LLM judge is scored on a 120-pair subsample (mean of three repeats), so its FPR is not directly comparable and is omitted. Exact 95% CIs for every count are in the appendix § C.1 .

Relation	Support	Recall	Precision
Exact	1,640	0.968	1.000
Perceptual	1,670	0.046	0.704
Subsumption	4,672	0.758	0.996
Semantic-Preserving	3,424	0.319	0.278
Complementary	310	0.000	0.000
Distinct	1,942	0.384	0.188

Table 3: Verifier self-validation against ground-truth graphs

G_{\rho}

(300 instances,

13{,}658

pairs). The binary merge decision scores recall

0.512

(exact 95% CI

[0.495,0.530]

) at precision

1.000

(tp/fp/fn

=1{,}696/0/1{,}614

Model	$\mu$ P $\uparrow$	$\mu$ R $\uparrow$	$\mu$ F1 $\uparrow$ [95% CI]	mF1 $\uparrow$	IS $\uparrow$	AC $\downarrow$	CR	Comp.	Pred.
Llama 3.2 1B [ 20 ]	0.000	0.000	0.000 [0.000, 0.000]	0.000	0.000	152.11	$-0.001$	$-0.11$	0.01
Llama 3.2 3B [ 20 ]	0.443	0.013	0.024 [0.019, 0.028]	0.018	0.012	200.00	0.008	1.05	1.05
Llama 3.1 8B [ 17 ]	0.615	0.061	0.095 [0.085, 0.106]	0.070	0.049	200.00	0.030	3.67	4.08
Qwen 3.5 2B [ 23 ]	0.134	0.008	0.013 [0.009, 0.018]	0.006	0.007	128.35	0.005	0.57	0.59
Qwen 3.5 4B [ 23 ]	0.290	0.098	0.109 [0.097, 0.122]	0.089	0.063	185.37	0.098	11.48	11.48
Qwen 3.5 9B [ 23 ]	0.357	0.074	0.101 [0.085, 0.117]	0.065	0.060	171.52	0.034	3.92	3.92

Table 4: Performance of nine curation agents on the corrupted-library benchmark. Every cell aggregates the same 300 instances, with instance-clustered bootstrap 95% CIs (2,000 resamples).

\mu

\mu

\mu

F1: mean per-episode precision, recall, and F1. mF1: micro F1. IS: Intrinsic Score. AC: action cost. CR: compression ratio. Comp.: mean compression. Pred.: mean predicted actions. Higher CR, Comp., and Pred. are not better on their own and carry no arrow, since they are only meaningful alongside precision or F1. Metric definitions are in § B.2 . Per-

\rho

breakdowns and action outcomes are in § D .

Panel of Figure 2 — Figure 2: Mean F1 by corruption rate $\rho$ for the four strongest curators ( $n{=}30$ episodes per cell). Difficulty is not monotone in $\rho$ as GPT-5.5 improves as defects multiply, Qwen 3.5 27B stays flat, and the smaller models collapse. Full results in Tab. A.4 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Threshold	Value	Role
$\epsilon$ (exact $L_{\infty}$ )	$1/255$	numerical-equality tolerance
$\tau_{\mathrm{pc}}$ (LPIPS)	$0.05$	worst-case perceptual bound
SSIM floor ( $1-$ SSIM)	$0.10$	LPIPS blind-spot guard
$\tau_{\mathrm{sm}}$ (DINO, p90)	$0.15$	semantic-preserving bound
complementary (LPIPS)	$0.05$	composition-effect tolerance
abstention $\delta$	$0.05$	UNCERTAIN -band width

Appendix

Table A.1 : Operating thresholds for the output-grounded verifier, calibrated before evaluation and held fixed across all experiments.

Figure A.1 : Per-relation precision, recall, and F1 of the output-grounded verifier on

G_{0}

(one-vs-rest). EXACT and PERCEPTUAL have no support in a clean reference graph and are omitted.

$\rho$	pairs	mergeable	merge recall
0.1	838	60	0.433
0.2	950	120	0.450
0.3	1,070	180	0.544
0.4	1,196	240	0.550
0.5	1,300	300	0.527
0.6	1,440	370	0.535

Appendix

Table A.2 : Merge safety by corruption rate

\rho

on the full

G_{\rho}

replay (

n{=}300

instances). False merges are zero at every rate.

Model	Applied	Rejected	Blocked	Invalid
Llama 3.2 1B	0.12	7.32	8.83	135.84
Llama 3.2 3B	1.05	4.14	0.00	194.81
Llama 3.1 8B	4.08	7.34	0.00	188.21
Qwen 3.5 2B	0.60	3.41	0.05	124.25
Qwen 3.5 4B	11.48	46.66	0.00	127.15
Qwen 3.5 9B	3.92	134.15	0.00	33.44

Appendix

Table A.3 : Mean per-episode action outcomes. Applied actions changed the library, while rejected, blocked, and invalid actions were discarded by VisCurate’s validation checks. GPT-5.5 as in Tab. 4 .

Model	$\rho{=}0.1$	$0.2$	$0.3$	$0.4$	$0.5$	$0.6$	$0.7$	$0.8$	$0.9$	$1.0$
GPT-5.5	0.420	0.521	0.579	0.596	0.621	0.641	0.624	0.628	0.625	0.612
Qwen 3.5 27B	0.457	0.476	0.522	0.559	0.523	0.512	0.524	0.519	0.484	0.474
Gemma 4 12B	0.378	0.384	0.356	0.353	0.321	0.252	0.231	0.209	0.210	0.206
Qwen 3.5 4B	0.216	0.226	0.185	0.160	0.120	0.068	0.048	0.033	0.017	0.018
Qwen 3.5 9B	0.332	0.229	0.176	0.098	0.055	0.043	0.025	0.007	0.027	0.018
Llama 3.1 8B	0.205	0.186	0.124	0.118	0.079	0.067	0.049	0.053	0.038	0.028

Appendix

Table A.4 : Mean F1 by corruption rate

\rho

(

n{=}30

episodes per cell). Difficulty is not monotone in

\rho

, and the direction is model-dependent.

Panel of Figure 12 — Figure A.2 : Percentage of runs that the models have ended using the END action.

Figure A.3 : Screening trade-off across neighbor count

k

at radius

0.05

, over ten corrupted instances (80 mergeable positives,

76{,}031

pairs). Merge recall saturates at

k{=}10

while

95.1\%

of pairs are still discarded, and the curves are insensitive to radius from

0.05

2.0

Figure A.4 : Candidate-set growth over nested random subsamples of

L_{0}

. The log-log fit

|C|=2.38\,n^{1.13}

(

R^{2}=0.998

) is close to linear, and the share of all pairs that screening proposes falls from

66.7\%

n{=}10

8.2\%

n{=}100

Deciding stage	$n$	share
residual (distinct by elimination)	6,020	47.2%
semantic	2,672	21.0%
subsumption	2,032	15.9%
complementary	938	7.4%
exact	918	7.2%
perceptual	110	0.9%

Appendix

Table A.5 : Which cascade stage decides each pair (

12{,}745

decisions). A majority is settled before any learned backend runs.

Figure A.5 : Cache footprint by library size for (a) raw output sets and (b) backend features only. The measured unit is

214.8

MB per skill for raw outputs over the 177-probe battery (median of 8 builtin skills), against an analytic features-only cache, a

395\times

saving at

10^{4}

skills.

Model	actions	applied	correct	tokens (in/out)	tok./repair
GPT-5.5	4,233	2,537	2,200	18.3M / 1.96M	9,202
GPT-5.6-luna	6,122	2,994	2,338	22.5M / 2.39M	10,667

Appendix

Table A.6 : Token cost per correct repair on the 41 telemetry-instrumented episodes shared by both hosted models. The denominator is correct repairs, not actions applied.

Comparison	Domain	Verdict	min $p$
DAv2-Large vs DAv2-Small	full	distinguishable	0.0005
SAM2-Large vs SAM2-Small	full	distinguishable	0.0005
SAM2-Large vs SAM2-Small	applicable	not disting.	0.0355
depth vs disparity (hard neg.)	full	distinguishable	0.0005
instance vs semantic (hard neg.)	applicable	distinguishable	0.0010

Appendix

Table A.7 : Distributional comparison of neural skills (permutation-null MMD, per probe). “Applicable” restricts to object-bearing images, the domain where segmentation is defined.

Strategy	$\|B\|{=}8$	16	32	64
stratified	0	0	0	0
first (single-class)	1	1	1	1
random	0	0	0	0

Appendix

Table A.8 : Hard-negative false merges by battery size and probe-selection strategy. Single-class selection produces a false merge that stratified and random selection never do.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 398 papers

VISUALSKILL: Multimodal Skills for Computer-Use Agents
Jun 16, 2026Ziyan Jiang, Li An, Yujian Liu +5Computer-Use Agents LLM Agent Skill Learning
1UC Santa Barbara · 2MIT CSAIL · 3MIT-IBM Watson AI Lab
Agent Skills Should Go Beyond Text: The Case for Visual Skills
May 31, 2026Binxiao Xu, Ruichuan An, Bocheng Zou +1Multimodal Grounding Agent Skill Learning
Peking University · University of Wisconsin · MIT-IBM Watson AI Lab
Skill-V: Verifiable Self-Evolving Skill Library for Interactive Agents
Oct 8, 2026Jie Ma, Zhipeng Qian, Yufei Ma +6LLM Agent Skill Learning LLM Agent Skill Retrieval
Xiamen University · Kuaishou Technology · Nankai University

Jun 16, 2026J/K move · Enter open · S save

VISUALSKILL: Multimodal Skills for Computer-Use Agents

Ziyan Jiang, Li An, Yujian Liu, Jiabao Ji+4

1UC Santa Barbara · 2MIT CSAIL · 3MIT-IBM Watson AI Lab

Computer-Use Agents LLM Agent Skill Learning

Open paper AskPDF arXiv

Computer-use agents (CUAs) approach human-level performance on standardised benchmarks but still struggle on long-horizon tasks and unseen software. Existing skill libraries address this with reusable skills, but represent the skill artifact as text only, despite the visual nature of GUI interaction. We propose VISUALSKILL: a hierarchical multimodal skill, tailored to each target application and organised as a central index over per-topic files, which the agent consumes through a load_topic MCP tool that fetches the relevant topic's text and figures on demand. We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration. On two CUA benchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL, a +15.3 point absolute lift over the no-skill baseline (0.303). Against a matched text-only skill that is generated from the same source content and differs from VISUALSKILL only in modality, VISUALSKILL yields a further +8.3 point absolute gain over the matched text-only skill (0.373 vs. 0.456), providing direct evidence that retaining visual figures in the skill artifact, rather than verbalizing them away, helps the agent both identify UI elements and verify workflow state after each action. Our code is available at https://github.com/XMHZZ2018/VisualSkills.

Abstract

Automated Evaluation

Explore similar work

VISUALSKILL: Multimodal Skills for Computer-Use Agents

Agent Skills Should Go Beyond Text: The Case for Visual Skills

Skill-V: Verifiable Self-Evolving Skill Library for Interactive Agents