AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents

Xiangrui Liu; Shuo Tang; Siheng Chen

cs.AIOct 13, 2026

AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents

Authors: Xiangrui Liu, Shuo Tang, Siheng Chen

Organizations: Shanghai Jiao Tong University (sjtu.edu.cn)

Abstract

Training business automation agents requires executable tasks that connect a natural-language request to an application state and a verifiable end state. Traditional Rubrics requires Authors to recreate application interfaces, initial records, policy evidence, and outcome checks for each task, entangling business logic with repeated infrastructure work. We introduce AssetForge, an asset-augmented task-construction pipeline that pairs a reusable construction-asset library with task-specific Rubrics. The assets provide application interfaces, world-state components, policy/evidence carriers, and native assertion definitions; the Rubric directs their instantiation for a particular business task. Executable checks and an additional model-based review validate each candidate, return targeted feedback, and support bounded repair before acceptance. On AutomationBench v1.0.6 public-600, fine-tuning Qwen3.6-35B-A3B on 6,000 AssetForge examples yields 41.17% on the official strict pass@1 metric, 4.4 times the 9.33% base score (+31.83 points). With the same backbone and GLM-5.3 supervision, AssetForge reaches 41.17% versus 30.33% for traditional Rubrics, a 10.84-point gain (35.7% relative) at the same accepted-example scale (6,000 versus 6,000 examples). Both DeepSeek- and GLM-supervised AssetForge models improve AppWorld completion over the base model on Test-Normal and Test-Challenge. These results establish the advantage of AssetForge's complete construction pipeline on AutomationBench public: the coupled asset-Rubric interface directs Authors toward business logic, while executable checks and model-based review validate their tasks.

Figures & tables

Figure 1: AutomationBench results overview. Official

\mathrm{pass@1}

(%) on v1.0.6 public-600; local scores use the first complete trial and

\dagger

marks published aggregates. Models are ordered by score within each group. Blue bars denote AssetForge; the 35B group includes the 27B baseline.

Figure 2: Overview of AssetForge . Rubrics specify the business scenario, task goals, and construction requirements; assets supply application interfaces, world-state components, policy/evidence carriers, and assertion definitions. The Author combines these inputs to construct a candidate request, initial state, application context, and success assertions. The code Gate checks format and execution, and an additional model-based review checks rubric requirements, accessible evidence, task consistency, and assertion coverage. Failed candidates return to the Author with feedback and repeat both checks. A candidate version is accepted only after passing both stages, yielding an executable QA task.

Model	AutomationBench domains						Overall
	Fin.	HR	Mkt.	Ops.	Sales	Supp.	$\mathrm{pass@1}$	$\mathrm{avg3}$
Closed-Source Models
Claude Opus 5	—	—	—	—	—	—	50.30 $\dagger$	—
Claude Fable 5	—	—	—	—	—	—	46.17 $\dagger$	—
GPT-5.6 Sol	—	—	—	—	—	—	45.83 $\dagger$	—
Gemini 3.6 Flash	—	—	—	—	—	—	45.00 $\dagger$	—

Table 1: AutomationBench v1.0.6 public-600 results (%). Domain scores and official

\mathrm{pass@1}

use the first complete trial;

\mathrm{avg3}

averages three repeats.

\dagger

: published aggregate scores; dashes: unavailable values. Bold marks the two AssetForge supervision branches in the 35B group; base and Rubrics are local baselines. The 35B group also includes the 27B baseline. Figure 5 gives the matched GLM-5.3 pipeline comparison at the same accepted-example scale.

Method	LLM	Test-Normal		Test-Challenge
		TGC	SGC	TGC	SGC
Official leaderboard systems
LARA MAS †	claude-opus-4-7	88.7	82.1	85.6	77.0
kecaipan capybara †	gpt-5.6-luna	85.1	73.2	73.4	52.5
IBM CUGA †	GPT-4.1	73.2	62.5	57.6	48.2
ReAct + 2 SetBSR Demos †	GPT-4o	68.5	57.1	38.9	23.0

Table 2: AppWorld TGC and SGC (%). Test-Normal has 168 tasks; Test-Challenge has 417.

\dagger

: official leaderboard references with their method–LLM pairing ( AppWorld Contributors, 2026 ) , not matched local reruns. All three local rows use ReAct. AssetForge-GLM and AssetForge-DS are the two supervision branches of the same AssetForge construction pipeline; bold marks the AssetForge rows.

Figure 3: Comparison of supervision branches. AssetForge-GLM and AssetForge-DS share the 35B backbone and 6,000-example scale. Domain scores and

\mathrm{pass@1}

use the first complete trial;

\mathrm{avg3}

averages three trials. The separator distinguishes domain and aggregate metrics.

Figure 4: Task acceptance at first and final submission. Both stages use all initial candidates as the denominator: 6,665 for AssetForge and 8,469 for Rubrics. Final submission records each candidate’s outcome after acceptance or exhaustion of the two-round repair budget, yielding 6,000 accepted tasks for each pipeline.

Figure 5: Data quantity and construction pipeline. All points use GLM-5.3 supervision and official first-trial

\mathrm{pass@1}

. The x-axis groups accepted-example counts, not token budgets; Rubrics (6,000) and AssetForge (6,000, star) share the 6k position and differ by 10.84 points. Tick labels are approximate categories; the y-axis breaks between 0% and 20%.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Benchmark	AutomationBench v1.0.6 public-600.
Domains	Finance, HR, Marketing, Operations, Sales, and Support; 100 tasks each.
Tools	api_search , api_fetch , base64_encode .
Interaction budget	At most 50 model responses per task.
Scorer	Deterministic native final-state assertions; no LLM judge.
$\mathrm{pass@1}$	Official AutomationBench metric: strict completion in the first complete trial, divided by 600.

Appendix

Table 3: AutomationBench evaluation setting and metric definitions.

Corpus	Examples	Total tokens (M)	Mean tokens (k)	Mean apps	Mean assertions
Rubrics	6,000	128	21	2.42	42.4
AssetForge	6,000	176	29	3.86	22.2

Appendix

Table 4: Training-corpus accounting for the Rubrics 6,000-example baseline and the 6,000-example AssetForge corpus.

Test	AssetForge–GLM	AssetForge–DS	Decision rule
D1 exact user-message SHA-256 intersection	0	0	no exact prompt reuse
D2 structural fingerprint intersection canon(initial_state, assertions)	0	0	no identical state/assertion structure
D3 opaque record-identity intersection	0	0	no shared runtime record ID
D4 flagged 5-gram/token-similarity pairs	0	0	Jaccard $\geq$ 0.12 or token ratio $\geq$ 0.55
D4 maximum Jaccard / token ratio	0.0329 / 0.1993	0.0329 / 0.1993	both below thresholds
D5 flagged semantic-cosine pairs	0	0	cosine $\geq$ 0.9270

Appendix

Table 5: Five-detector contamination audit against all 600 AutomationBench public-600 tasks. Zero means an empty intersection or no flagged record; the two columns are the complete 6,000-record training files.

Artifact	AssetForge–GLM	AssetForge–DS
Training file SHA-256	7aaa2ec3774410ccacc1f6dcbf49a85d 99215d83639b2e070da88c2a7d4f7616	b3d727d5e10be9f2547d4bd5ed1e238b 71130f7fa8c319cd3ceacea3206695d8
Index SHA-256	9938d59173ba4051ce58c45c894d365 865c91470d3403e71d69c20528c86eccc	58b41726203ba0d54c9ab5388068392 74f53da5cd6394eaa047045fd13afd435
D4 audit report SHA-256	7caa20d7813a21636ccb21cb03e85cf1 ba4bd6fcb3ee32017b78265a9e7ac9db	21b98b3f2678f7f7b5885c01ff18544 7acbd994a1637a9161f921a33eba93d
D5 semantic-audit SHA-256	6d4c46fb49c41b922b9da4176fd79d52 4ed26fd353eb86f8b5fe98020390e0c7	eb87bf398606cdf6b022b319ef387ad4 7c207845b6300643def7232f4dbcdd54
Consolidated audit SHA-256	5af5abaa3b363080a23fd17ca160d0f1e9103d2cff3eef8a8b4a3855fb08ff4f

Appendix

Table 6: Training-file and report identifiers recorded in our contamination audit.

Pipeline	Candidates	First pass	Final accepted	Initial rejects	Repaired accepts	Repairs	Gate calls	Review calls
Rubrics	8,469	5,880	6,000	2,589	120	5,058–5,178	13,527–13,647	6,000–13,647
AssetForge	6,665	5,913	6,000	752	87	1,417–1,504	8,082–8,169	6,000–8,169

Appendix

Table 7: Construction accounting underlying Figure 4 .

Resource	Role in construction
Application interfaces	Describe supported operations and the records on which they act.
World-state components	Specify record structures and support preparing the initial application state.
Assertion types and tools	Describe the available success checks and their required parameters.
Policy and evidence records	Provide readable locations for task-specific rules and evidence supplied by the Author.

Appendix

Table 8: Construction resources supplied to the Author by the AssetForge pipeline.

Application	Example role
Gmail	Messages, threads, labels, and business replies.
Slack	Readable policies, coordination messages, and notifications.
Google Sheets	Structured records and row-based task state.
Jira	Ticket and operational-workflow records.
DocuSign	Document-signature workflows.
LinkedIn	Candidate or outreach context.

Appendix

Table 9: Representative applications named in the AssetForge construction inventory. Each row shows the application context used during task construction.

Record	Required state	Evidence and exclusions
Zendesk 87421	Open, urgent, Engineering Escalation; tag primary_nd7742	Internal note contains ND-7742 , Jasper Quill, and Tina Vu; no duplicate tag.
Zendesk 87438	Solved, urgent, Customer Care; tag duplicate_nd7742	Internal note contains rb-nd-7742-primary and ND-7742 .
Zendesk 87444	Solved, high, Tier 2 Support; tag duplicate_nd7742	Internal note contains rb-nd-7742-primary and ND-7742 .
Zendesk 87380	Closed	No primary or duplicate tag.
Zendesk 87445	Open	Lookalike account; no primary or duplicate tag.
Zendesk 87452	Hold	Vendor or non-counting record; no primary or duplicate tag.

Appendix

Table 10: Case A final-state contract. Protected records must remain untouched.

Output group	Required result	Native checks
Five Gmail labels	MAR1-EX , MAR1-OK , MAR1-HOLD , MAR1-EX , and MAR1-RATE on SES-3412, 3418, 3421, 3427, and 3433	Five message-label assertions.
Five producer replies	EUR 3,060.00; 1,235.00; 0.00; 720.00; and 580.00, each with the policy rationale	Five thread-reply assertions, including recipients and decisive phrases.
Jonas summary	Five sessions billed; total EUR 5,595.00; held 1; not billed 1	Two summary-message assertions.
Protection checks	No reply or label for SES-3421b and SES-3405; no SES-3405 message; no trash; the MAR1-RATE label exists	Seven negative, count, and label-existence assertions.

Appendix

Table 11: Case B scored outputs and protected outcomes.

Step	Mode	Target	Decisive argument
0	GET	Gmail labels	maxResults=100
1	GET	Gmail inbox messages	labelIds=INBOX; maxResults=100
2	GET	msg_run_procedure	format=full
3	GET	msg_ses_3412	format=full
4	GET	msg_ses_3418	format=full
5	GET	msg_ses_3421	format=full

Appendix

Table 12: Case B validated execution trace, grouped by phase.

Lane	Turns	Tool calls	Strict	Input tokens
GLM-5.3	22	48	1.0	511,519
DeepSeek V4.1 Flash	13	43	1.0	257,401
Qwen3.8-Flash	13	39	1.0	183,103

Appendix

Table 13: Case B solver-lane replication. Every lane received the same task and reached strict score 1.0.

Panel of Figure 6 — Figure 6: Selected excerpts from the two Author rubrics. The traditional rubric specifies requirements for the request, executable environment, and scorer. Both pipelines require specification-first construction and self-checks; the AssetForge excerpt makes the supplied construction resources explicit within that procedure. Wording is retained from the supplied rubrics except that internal validation terminology is normalized; [...] marks omissions, and only typesetting is normalized.

Model	Trial 1	Trial 2	Trial 3
Qwen3.8-Max-0902	308	302	299
GLM-5.3-max	292	290	288
Qwen3.8-27B	215	223	213
GLM-5.2-max	177	168	165
Qwen3.6-35B-A3B (base)	56	60	53
Rubrics	182	190	175

Appendix

Table 14: Strict-success counts over three AutomationBench trials. Each trial contains 600 tasks.

Domain	AssetForge (DS)				AssetForge (GLM)				Rubrics (GLM)
	T1	T2	T3	Avg3	T1	T2	T3	Avg3	T1	T2	T3	Avg3
Finance	45	42	42	43.00	39	36	35	36.67	33	39	37	36.33
HR	53	49	54	52.00	51	53	52	52.00	27	31	19	25.67
Marketing	39	41	39	39.67	38	43	40	40.33	29	33	36	32.67
Operations	46	47	51	48.00	45	43	42	43.33	43	41	40	41.33
Sales	39	47	40	42.00	44	39	38	40.33	34	28	26	29.33

Appendix

Table 15: Per-trial domain counts for AssetForge and Rubrics. Both AssetForge supervision branches and Rubrics report three trials. T1–T3 are success counts out of 100 in each domain; the final row has denominator 600 per trial. Avg3 is a percentage.

Construction pipeline	Training examples	$\mathrm{pass@1}$
Rubrics	1,500	24.50
Rubrics	2,500	27.17
Rubrics	4,000	28.67
Rubrics	6,000	30.33
AssetForge-GLM	6,000	41.17

Appendix

Table 16: First-trial AutomationBench completion for the GLM-5.3-supervised comparison (%). Each evaluation contains 600 tasks. The four Rubrics configurations share the same task-construction rubric; AssetForge changes the complete pipeline at the same 6,000-example scale as the largest Rubrics configuration.

Business Process Automation

Business Process Automation uses software, process models, and AI agents to execute, monitor, and improve organizational workflows.

See all 39 papers

Sep 30, 2026

GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

Working agents need to read diverse files, coordinate tools, and produce deliverables. Training such agents requires tasks built on many real files with verifiable results, but few pipelines exist to synthesize this kind of data. Existing pipelines either generate files with models, which lack realism and diversity, or build tasks on real files without task-specific verifiers, leaving result quality unchecked. We introduce GraphForge, an evidence-graph based framework that grounds both the task and its verification in real files. Starting from occupation-grounded seeds for controlled diversity, GraphForge assembles a workspace of real files for each seed and builds an evidence graph over their relations. Since the task statement and rubrics are both derived from this graph, task requirements are backed by the workspace files and each criterion is anchored to the files needed to verify it. An initial rollout further tests executability, and a revision agent repairs the task and rubrics against the original files before trajectories are collected. Fine-tuning Qwen3.6-27B on 2,169 GraphForge trajectories brings GDPVal to 1445.7 (+65.7) under OpenHands, and Workspace-Bench-Lite and SpreadsheetBench II to 63.7 (+7.7) and 24.0 (+13.7) under Claude Code. Rejection fine-tuning on the SFT model's own rollouts, with candidates selected by the evidence-anchored rubrics, yields further improvements on all three benchmarks, suggesting that the rubrics provide a useful selection signal.

Qisheng Su, Hanchen Wang, Guanru Zhu +11

University of Science and Technology of China · Shanghai Innovation Institute · Fudan University

AI Agent Benchmarks LLM Agent Training LLM Agent Planning

Ask PDF

Apr 21, 2026

AutomationBench

Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform - requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. To address this gap, we introduce AutomationBench, a benchmark for evaluating AI agents on cross-application workflow orchestration via REST APIs. Drawing on real workflow patterns from Zapier's platform, tasks span Sales, Marketing, Operations, Support, Finance, and HR domains. Agents must discover relevant endpoints themselves, follow layered business rules, and navigate environments with irrelevant and sometimes misleading records. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. Even the best frontier models currently score below 10%. AutomationBench provides a challenging, realistic measure of where current models stand relative to the agentic capabilities businesses actually need.

Daniel Shepard, Robin Salimans

LLM Agent Evaluation Business Process Automation AI Agent Benchmarks Workflow Automation Tool Use Automated Agent Evaluation

Ask PDF

May 27, 2026

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

As agent capabilities advance, existing benchmarks, such as

τ^2

-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive

n

-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct

τ^c

-Bench, a challenging extension of the three domains of

τ^2

-Bench. We evaluate

11

agent/user LLM pairs and find that models nearly saturating

τ^2

-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from

0.82\!-\!0.94

0.28\!-\!0.61

). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.

Tomer Keren, Nitay Calderon, Asaf Yehudai +3

Faculty of Data and Decision Science Technion · IBM Research

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents Benchmark Construction LLM Agent Evaluation Synthetic Benchmark Generation+1

Ask PDF

Abstract

Business Process Automation

Explore similar work

GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

AutomationBench

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks