VABAR: A Variational Automation Benchmark for Agentic Robotics

Shuangyu Xie; Kaiyuan Chen; Uksang Yoo; Xinyue Zhu; Meadow Shen; Jaimyn Drake; Eugen Solowjow; Lionel Gulich

cs.RONov 4, 2026

VABAR: A Variational Automation Benchmark for Agentic Robotics

Authors: Shuangyu Xie, Kaiyuan Chen, Uksang Yoo, Xinyue Zhu, Meadow Shen, Jaimyn Drake, Eugen Solowjow, Lionel Gulich,

Organizations: University of California, Berkeley · Siemens · NVIDIA

Abstract

For robots to operate reliably in commercial and industrial applications, they must persistently perform specific tasks with known variations in object shape and pose. We call this Variational Automation (VA) to differentiate these settings from fixed automation. Inspired by recent papers on Agentic Robotics, where pretrained LLM and VLM coding agents combine interpretable robot programming with the open-world adaptability of model free policies, we propose a general set of agentic robotics benchmark design criteria that includes accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency. We instantiate these principles in six manipulation task classes spanning industrial, medical, and infrastructure applications: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. We introduce the Variational Automation Benchmark for Agentic Robotics (VABAR), an open, easily replicable, simulation-based manipulation benchmark to systematically evaluating agentic robotics systems. Built fully on open-source tools, VABAR features detailed implementations using the Newton physics engine, standardized task variations, difficulty tiers, success metrics, and a library of 33 generalizable skills with human-authored programmatic reference solutions. We provide initial baseline evaluations using an open agentic manipulation harness, Graph-as-Policy (GaP), and Isaac Lab Arena.

Figures & tables

Fig. 2 : Overview of the VABAR evaluation framework. Left: Tasks are specified by initial and goal configurations, an asset library, and robot definitions. Top right: Admissible task variations span objects, arrangements, and positions, with examples at two levels. Bottom right: Robot policies are evaluated across diverse manipulation tasks, including object sorting, tool hanging, gear assembly, insertion, and cable routing.

Panel of Figure 2 — Fig. 3 : Diagnostic execution flow. VABAR records observations and estimated state, system actions, resulting physics states and contacts, and verification outcomes. Dashed boxes denote privileged simulator state and dotted box denote the physics object contact information used for diagnosis.

Fig. 4 : VABAR skill library. The 33 skills span perception, grasping, motion planning and execution, and verification. Colored cards marked “+” identify 25 added skill bundles relative to the May Graph-as-Policy library; white cards denote eight retained counterparts. Additions support functional-feature manipulation, constrained placement, cable handling, and execution checks.

Task	Level 1	Level 2
1 TS	Singulated tools; no distractors	Cluttered tools
2 TH	One tool; known fixture pose	Multiple tools and fixture types; randomized poses
3 GM	Assemble two gears	Assemble three gears
4 SP	Singulated targets	Cluttered scenes
5 CI	Fixed, visible port; nearby connector with small misalignment	Randomized port pose; moderate connector misalignment
6 CR	2–3 widely spaced pegs; short route; low turning angle	4–6 moderately spaced pegs; multiple turns; moderate initial cable variation

TABLE I : Task difficulty levels. 1. TS: Tool Sorting; 2. TH: Tool Hanging; 3. GM: Gear Meshing; 4. SP: Syringe Packing; 5. CI: Cable Insertion; 6. CR: Cable Routing.

		a-i. GaP [-1pt] (Human w/ Newton)		a-ii. GaP [-1pt] (Human w/ ILA)		b. Claude [-1pt] w/ Newton		c. GaP [-1pt] w/ Newton
Task	Diff.	Succ. (%)	Cyc. (s)	Succ. (%)	Cyc. (s)	Succ. (%)	Cyc. (s)	Succ. (%)	Cyc. (s)
Tool Sorting	Level 1	47.1	421.6	20.0	555.0	0.0	N/A	26.7	134.2
Tool Sorting	Level 2	33.7	913.8	–	–	0.0	N/A	13.3	232.3
Tool Hanging	Level 1	86.7	86.1	67.5	98.3	16.7	96.2	33.3	81.5
Tool Hanging	Level 2	81.7	158.4	50.0	138.3	10.0	190.8	10.0	270.9
Gear Meshing	Level 1	100.0	165.9	87.5	198.9	0.0	N/A	70.0	156.6

TABLE II : Task performance comparison across VABAR tasks and difficulty levels.

Task	Difficulty	Perception	Grasp	Motion
		(%)	(%)	(%)
Tool Sorting	Level 1	37.8	51.4	10.8
Tool Sorting	Level 2	56.4	28.7	14.9
Tool Hanging	Level 1	0.0	0.0	100.0
Tool Hanging	Level 2	21.4	28.6	50.0
Gear Meshing	Level 1	N/A	N/A	N/A

TABLE III : Failure modes across VABAR difficulty levels.

Fig. 5 : Example: Agent inspects Newton physics model to diagnose and fix failure due to syringes jamming inside disposal box . Left: simulated syringe-disposal scene with a top-down inset. Right: Netwon ground-truth front and side internal views of jammed episodes t01 and t29 . Red segments highlight syringes straddling the lid plane; blue segments show other syringes.

Robot Manipulation Benchmarks

Robot manipulation benchmarks measure how reliably robots perceive, plan, and execute physical tasks across objects, settings, and embodiments.

See all 346 papers

RobotAgentBench: Towards Open-world Robot Autonomy
Oct 26, 2026Lingjun Mao, Jinglin Cao, Lukun He +22Embodied Agent Benchmarks Robot Manipulation Benchmarks
Aether AI · University of California San Diego · University of California Davis
RoboARCade: Benchmarking Agentic Robot Control
Nov 5, 2026Satvik Sharma, Hengyuan Hu, Mihika Kamat +2Embodied Agent Benchmarks Robot Manipulation Benchmarks
Stanford University
GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks
Jul 6, 2026Kaiyuan Chen, Shuangyu Xie, Letian Fu +21Robot Policy Learning Graph-Based Planning
University of California, Berkeley · Carnegie Mellon University · Bosch +1

Oct 26, 2026J/K move · Enter open · S save

RobotAgentBench: Towards Open-world Robot Autonomy

Lingjun Mao, Jinglin Cao, Lukun He, Wenpeng Xu+21

Aether AI · University of California San Diego · University of California Davis

Embodied Agent Benchmarks Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Recent advances in vision-language-action (VLA) and world-action models (WAMs) have significantly improved robotic manipulation, yet deploying robots in the open world demands a higher level of autonomy beyond executing predefined tasks. Robot agents offer a promising paradigm for this transition, integrating task discovery, contextual reasoning, physical execution, and adaptive recovery into a closed interaction loop. However, existing benchmarks largely evaluate isolated capabilities, failing to capture the complexity of open-world robot autonomy. We introduce RobotAgentBench (RAB), a comprehensive benchmark for evaluating robot agents in realistic, open-world household environments. Through an agent-assisted, human-in-the-loop real-to-sim pipeline, RAB constructs 50 diverse household layouts with interactive objects and physically grounded tasks, encompassing 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 open-ended scenarios. RAB provides a multi-level evaluation of robot agent capabilities: (1) task-level assessments systematically examine five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness; (2) scenario-level evaluation assesses how agents integrate these capabilities to autonomously fulfill high-level user needs. Furthermore, RAB enables systematic comparisons of standalone VLA/WAM models and diverse robot agent architectures under a unified planning-execution-verification framework. Extensive experiments reveal substantial gaps between manipulation and autonomy, demonstrate the advantages of agentic architectures and recursive self-improvement, and highlight the remaining challenges toward open-world robot autonomy.

Abstract

Robot Manipulation Benchmarks

Explore similar work

RobotAgentBench: Towards Open-world Robot Autonomy

RoboARCade: Benchmarking Agentic Robot Control

GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks