SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agents

Bingqing Jiang; Guoxi Zhang; Jasper Wang; Auric Wang; Bingning Wang; Tianyi Lin; Zichao Yu; Yujin Han

cs.AISep 28, 2026

SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agents

Authors: Bingqing Jiang, Guoxi Zhang, Jasper Wang, Auric Wang, Bingning Wang, Tianyi Lin, Zichao Yu, Yujin Han, +2 more

Organizations: The University of Hong Kong · WeChat, Tencent · Peking University · The Hong Kong Polytechnic University · City University of Hong Kong

Abstract

Recent work incorporates reusable skills distilled from past interactions into multimodal agent training, providing procedural guidance for long-horizon planning and tool use. However, policy optimization in these methods remains driven primarily by sparse outcome rewards, providing little supervision for intermediate decisions. Rubric-based rewards address this limitation through explicit intermediate criteria, but reliable rubrics are difficult to construct at scale and often disconnected from the procedure followed by the actor. We observe that a well-structured skill naturally specifies both how to act and what successful execution should achieve. Based on this insight, we introduce SkillRubric, which represents each skill through aligned actor-facing guidance and an evaluator-facing rubric. A multimodal verifier evaluates skill-defined goals using screenshots and tool outputs, assigning completion and progress rewards to the responsible turns. We further introduce an alternating co-evolution scheme that validates guidance revisions through paired rollouts under a frozen policy and rubric revisions offline under fixed guidance. Experiments across diverse multimodal agent benchmarks demonstrate consistent performance gains, while controlled paired rollouts further show that evolved skills provide more effective guidance for planning and tool use than their preceding versions.

Figures & tables

Figure 1 : Aligned actor guidance and evaluator rubrics through a shared skill representation. SkillRubric derives both views from a shared procedural structure, aligning guidance stages with observable evaluation criteria.

Figure 2 : Overview of SkillRubric. Stage 1: Success-filtered multi-teacher trajectories initialize the policy, while all collected trajectories are distilled into a dual-view SkillBank. Stage 2: Actor-facing guidance conditions grouped rollouts, and evaluator-facing rubrics provide process credit alongside a fixed outcome signal. Stage 3: Guidance and rubrics are alternately evolved through outcome-grounded validation, with rubric validation delayed by $H$ policy updates.

Method	IID Benchmarks				OOD Benchmarks				Avg.
Method	AgentVista	BrowseComp-VL	VDR-Bench	VisualToolBench	VisBrowse	MMSearch+	MMSearchExam	TIR-Bench	Avg.
Closed-source model references
Gemini 3.1 Flash-Lite	26.61	43.81	20.00	31.01	33.14	33.76	26.15	27.00	30.19
GPT-5 nano	12.84	35.79	5.67	14.95	24.85	5.14	24.73	17.50	17.68
GPT-5 mini	16.51	45.15	11.67	27.54	36.09	14.79	34.98	24.50	26.40
Open-source model references

Table 1 : Main results across eight multimodal agent benchmarks (%). Each result is based on a single rollout per task. All trained methods are initialized from the corresponding backbone-specific SFT checkpoint and evaluated using the same tool suite and evaluation protocol. Best and second-best results among trained methods under each backbone are shown in bold and underline , respectively. Closed- and open-source reference models are excluded from ranking. For our method, “w/ skill” and “w/o skill” denote inference with and without skill guidance, respectively.

Backbone	Variant	Components			Average Success Rate (%)
Backbone	Variant	Outcome RL	Rubric Reward	Co-Evolution	IID	OOD	Overall
Qwen3-VL-8B	SFT-only	–	–	–	11.81	7.59	9.70
	GRPO-only	$\checkmark$	–	–	13.38	9.70	11.54
	GRPO-Rubric	$\checkmark$	$\checkmark$	–	15.46	11.70	13.58
	Ours	$\checkmark$	$\checkmark$	$\checkmark$	19.91	15.86	17.88
Qwen3.5-9B	SFT-only	–	–	–	21.77	21.60	21.68

Table 2 : Ablation study of the proposed components. All variants share the same backbone-specific SFT initialization and initial SkillBank. IID, OOD, and Overall are unweighted averages over the corresponding four, four, and all eight benchmarks, respectively.

Backbone	Variant	$\bm{H}$	IID	OOD	Overall
Qwen3-VL-8B	Ours	5	16.34	12.79	14.57
	Ours	10	19.91	15.86	17.88
	Ours	20	18.35	13.98	16.17
	GRPO-Rubric	$\infty$	15.46	11.70	13.58
	Guidance evolution only	10	18.68	15.04	16.86
	Rubric evolution only	10	17.81	14.84	16.33

Table 3 : Ablation on the SkillBank evolution interval and components.

H

denotes the number of policy-optimization steps between successive alternating updates in the full method.

H=\infty

keeps the SkillBank fixed and corresponds to GRPO-Rubric. Guidance evolution only and rubric evolution only update the indicated component while keeping the other fixed, with

H=10

Model	IID Benchmarks				OOD Benchmarks				Avg.
Model	AgentVista	BrowseComp-VL	VDR-Bench	VisualToolBench	VisBrowse	MMSearch+	MMSearchExam	TIR-Bench	Avg.
Qwen3-VL-8B-Instruct	6.7	4.6	3.9	8.4	5.8	5.0	4.6	7.9	5.9
Ours (Qwen3-VL-8B, w/ skill)	5.9	3.3	3.1	6.1	5.2	3.2	3.1	6.0	4.5
Qwen3.5-9B	10.7	6.8	9.6	9.5	11.3	9.7	9.6	8.9	9.5
Ours (Qwen3.5-9B, w/ skill)	4.8	4.7	5.7	3.1	6.4	4.6	5.5	1.6	4.6

Table 4 : Average interaction turns across eight multimodal agent benchmarks. Our models are conditioned on guidance retrieved from the final SkillBank at inference, while the backbone models are evaluated without skill guidance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Setting	Available	Train	Eval
AgentVista	Comprehensive Multimodal Tool Use	IID	209	100	109
BrowseComp-VL	Multimodal Browsing	IID	399	100	299
VDR-Bench	Multimodal Deep Research	IID	2,000	100	300
VisualToolBench	Tool-Assisted Visual Reasoning	IID	1,204	100	200
VisBrowse-Bench	Visual-Native Browsing	OOD	169	–	169
MMSearch-Plus	Multimodal Search	OOD	311	–	311

Appendix

Table 5: Dataset statistics and train/evaluation splits. Available denotes the size of the complete benchmark, while Train and Eval report the subsets used in our experiments. Unselected samples are not used elsewhere, and all OOD evaluation tasks are excluded from training.

Tool	Function	Backend	Call limit
Zoom	Crops or enlarges selected image regions for fine-grained visual inspection.	Local image processor	No separate limit
Code interpreter	Executes Python code for image processing, numerical computation, and data analysis.	Isolated code sandbox	No separate limit
Web search	Retrieves relevant webpages and textual search results for a text query.	SerpAPI	7 per trajectory
Image search	Retrieves visually related images and their associated webpages and sources.	SerpAPI; ImgBB for temporary hosting	5 per trajectory
Visit	Opens a specified webpage and extracts its textual and visual content.	Jina and Firecrawl	No separate limit

Appendix

Table 6: Tool definitions and execution settings. All compared methods use the same tools and per-trajectory call limits.

Group	Setting	Value
Inference	Serving backend	SGLang (OpenAI-compatible)
	Sampling temperature	0.2
	Top- $p$	1.0
	Rollouts per task	1
	Maximum interaction turns	20
	Maximum context length	131,072 tokens

Appendix

Table 7: Final evaluation configuration. The inference and judging settings are shared by all compared models; skill settings apply only to skill-conditioned models.

Explore similar work

SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
Jul 2, 2026Jiayin Zhu, Kelong Mao, Yudong Guo +4Skills Rubrics
1HKUST(GZ), 2JD.COM · ∗Yutao Yue is also affiliated with Institute of Deep Perception Technology, JITRI, Wuxi, China.
SkillLift: Learning Dense Rubrics from Sparse Oracles for Efficient Skill Evolution
Sep 14, 2026Haoxiang Kang, Ming WenSkill Evolution Skills
1Independent Researcher · 2Fudan University
SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing
Jun 12, 2026Haowen Gao, Haoran Chen, Can Wang +5Skill Evolution Skills
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · Tongyi Lab, Alibaba Group, Beijing, China

Jul 2, 2026, cs.AIJ/K move · Enter open · S save

SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use

Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He+3

1HKUST(GZ), 2JD.COM · ∗Yutao Yue is also affiliated with Institute of Deep Perception Technology, JITRI, Wuxi, China.

Skills Rubrics

Open paper AskPDF arXiv

Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.

Abstract

Explore similar work

SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use

SkillLift: Learning Dense Rubrics from Sparse Oracles for Efficient Skill Evolution

SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing