ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making

Liu Dai; Haina Wang; Weikang Wan; Hao Su

cs.ROMay 27, 2025

ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making

Authors: Liu Dai, Haina Wang, Weikang Wan, Hao Su

Organizations: University of California, San Diego

Abstract

Building embodied agents capable of accomplishing arbitrary tasks is a core objective towards achieving embodied artificial general intelligence (E-AGI). While recent work has advanced such general robot policies, their training and evaluation are often limited to tasks within specific scenes, involving restricted instructions and scenarios. Existing benchmarks also typically rely on manual annotation of limited tasks in a few scenes. We argue that exploring the full spectrum of feasible tasks within any given scene is crucial, as they provide both extensive benchmarks for evaluation and valuable resources for agent improvement. Towards this end, we introduce ManiTaskGen, a novel system that automatically generates comprehensive, diverse, feasible mobile manipulation tasks for any given scene. The generated tasks encompass both process-based, specific instructions (e.g., "move object from X to Y") and outcome-based, abstract instructions (e.g., "clear the table"). We apply ManiTaskGen to both simulated and real-world scenes, demonstrating the validity and diversity of the generated tasks. We then leverage these tasks to automatically construct benchmarks, evaluating high-level embodied decision-making through a grounded skill interface for agents built upon existing vision-language models (VLMs). Furthermore, we propose a simple yet effective method that utilizes ManiTaskGen tasks to enhance embodied decision-making. Overall, this work presents a cross-scene task generation framework facilitating both benchmarking and improvement of embodied decision-making agents. Our code is available at https://manitaskgen.github.io.

Figures & tables

Figure 1: Framework Overview. ManiTaskGen systematically generates process-based and outcome-based navigate–pick–place tasks from scene information. The generated tasks support benchmark construction and feedback-based improvement of embodied decision-making agents.

Figure 2: Agent-Env Interaction Cycle. This cycle provides a fundamental depiction of how an embodied agent executes a task in a scene.

Figure 3: Visualization of Segmented Receptacle Regions on a Surface.

Figure 4: The "lightmap"s Which Show the Diversity of Generated Tasks. Each time an object or location is mentioned in a task, we add a highlight at the corresponding position. The brightness intensity and distribution reveal that our method generates tasks covering more objects and locations.

Datasets	Num. of Scenes	Num. of Tasks
GenSim [ 33 ]	-	100
$\lambda$ [ 51 ]	20	521
M3Bench [ 16 ]	119	31,050
ALFRED [ 34 ]	120	25,743
Language Rearrangement [ 7 ]	1	1,000
Embodied Agent Interface [ 17 ]	2	438

Table 1: Comparison among Scene-Task Dataset Statistics.

	Process-based Tasks	Outcome-based Tasks
GPTTaskGen-RAS	29.4%	21.1 %
ManiTaskGen-RAS-40K (Ours)	94.0%	86.5 %

Table 2: Human-Verified Task Validity Rate.

Figure 5: Visualization of An Example Testing Episode. The agent is equipped with abstracted navigation ( go_to, change_view ), grasping ( pick ) and placing ( show_receptacle, place ) skills. Blue marks indicate the walkable locations around ground objects involved in this episode.

	Level 1		Level 2		Level 3		Level 4	Average
	IP	SR (%)	IP	SR (%)	IP	SR (%)	SR(%)	IP	SR (%)
Human	96.0	82.5	95.0	80.5	95.0	80.0	90.0	95.3	83.3
Random	0.7	0.0	0.7	0.0	0.7	0.0	0.0	0.7	0.0
GPT-5.2 [ 52 ]	88.1	63.9	67.9	34.7	72.8	20.0	30.0	76.3	37.2
GPT-4o [ 52 ]	71.6	40.1	42.9	16.3	57.0	8.2	26.0	57.2	22.7
GPT-4.1 [ 52 ]	67.5	36.4	46.4	20.4	54.4	4.2	28.0	56.1	22.3

Table 3: Evaluation Results on Existing VLMs. We cover both proprietary (upper part) and open-source models (lower part). Here, IP refers to Intermediate Points, and SR refers to Success Rate.

Figure 6: Success and Failure Modes from Gemini-2.5-pro in each stage of Level 3 tasks.

Figure 7: Set-up of Real Robot Experiments. The camera view was manually annotated with numerical tags included in the agent’s visual input. All irrelevant objects are masked out.

Model Name	IP (Level1&2)	SR (Level1&2)	IP (Level3)	SR (Level3)
Gemini-2.5-pro [ 53 ]	97.5	90%	100	100%

Table 4: Real Robot Experiment Results.

Model Name	IP (Before)	IP (After)	SR (Before)	SR (After)
GPT-4.1 [ 52 ]	75.5	79.4	38%	49%
Gemini-2.5-flash [ 53 ]	79.4	81.7	36%	58%
Claude-3.5-haiku [ 54 ]	68.9	79.4	36%	51%

Table 5: Improvement of Agent Performance.

Figure 8: Improvement of Gemini-2.5-flash Agent. We find that more trial episodes continuously contribute to the agent performance.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Per-axis noise bound	L1 (%)	L2 (%)	L3, two-step (%)	L4 (%)
No noise	94	94	90	86
5 cm	89	87	79	85
25 cm	71	72	53	85

Appendix

Table 6: Task validity under input localization noise. Each cell summarizes 100 manually verified tasks generated after rebuilding the scene graph for that noise setting. This separate study does not replace the original validity comparison or measure robot success rates.

Figure 9: Visualization of Rules Involved in Scene Graph Construction.

Primary intent	Definition / decision rule	Templates	Instances
		$n$ (%)	$n$ (%)
Ordering / alignment	Explicit order, gradient, row, grid, line, symmetry, or regular alignment.	36 (18.0)	57 (8.8)
Display / presentation / curation	Informative, interactive, aesthetic, product, gallery, or focal-point display.	31 (15.5)	104 (16.0)
Functional workspace / workflow setup	Workspace, studio, lab, meeting/collaboration area, or multi-step workflow.	24 (12.0)	51 (7.8)
Artistic / geometric arrangement	Non-representational artistic composition, geometric pattern, sculpture, or installation.	24 (12.0)	57 (8.8)
Thematic scene / structural construction	Recognizable scene, model, navigable layout, or physical structure, e.g., city, maze, garden, bridge.	24 (12.0)	40 (6.2)

Appendix

Table 7: Primary-intent distributions of outcome-based templates and task instances. Percentages use 200 templates and 650 instances, respectively; instance percentages are rounded independently. Counts are taken from the archived template classification and instance-intent statistics.

Semantic / reasoning tag	Scope / examples	Templates	Instances
		$n$ (%)	(%)
Thematic / aesthetic composition	Visual harmony, display, story/theme, color scheme, atmosphere, centerpiece.	99 (49.5)	49.5
Semantic categorization	Category, type, material, function, power source, project, similarity, thematic grouping.	61 (30.5)	16.0
Symmetry / geometric reasoning	Symmetry, mirror, circle, spiral, grid, triangle, geometric shapes, maze/path geometry.	60 (30.0)	19.5
Attribute comparison & ordering	Size, height, length, color/brightness, frequency, relevance, chronology, alphabetical order.	54 (27.0)	28.0
Accessibility / ergonomics / workflow	Access, reachability, retrieval, interaction, efficiency, collaboration, handover, navigation.	53 (26.5)	25.8

Appendix

Table 8: Multi-label semantic and reasoning coverage of outcome-based templates and task instances. Percentages use 200 templates and 650 instances; they need not sum to 100%. Instance-level tag frequencies are reported as rounded percentages, with approximately 2.18 tags per instance.

Figure 10: Visualization of the 3 Scenes Used to Generate ManiTaskGen-RAS-40K.

Dataset	Scenes	Represented task-space count
		Level 1+2	Level 3	Level 4
SUNRGBD	5	$486$	$52,956$	$200$
ReplicaCAD	5	$119,682$	$2.86\times 10^{9}$	$75,000$
AI2THOR	90	$213,446$	$1.10\times 10^{9}$	$250,000$
Total	100	$\mathbf{333,614}$	$\mathbf{3.96\times 10^{9}}$	$\mathbf{325,200}$

Appendix

Table 9: Task-space statistics across 100 source scenes. Levels 1–2 count represented single-step candidates; Level 3 counts two-step combinations under the generation rules; Level 4 is an estimate rather than an exhaustive VLM-filtered enumeration. None of these columns measures semantically deduplicated behaviors. Actual RAS-40K instances are listed in Table 10 .

Dataset	Level 1	Level 2	Level 3	Level 4	Total
ReplicaCAD	2195	24153	5000	500	31848
AI2THOR	2218	0	5000	100	7318
SUN-RGBD	35	120	500	50	705
Total	4448	24273	10500	650	39871

Appendix

Table 10: Statistics of ManiTaskGen-RAS-40K across 3 Source Scenes.

Panel of Figure 11 — Figure 11: Ambiguous Item Naming in ReplicaCAD.

Name (before)	Name (after)
frl_apartment_kitchen_utensil_01	kitchenware_ceramic_lidded_container
frl_apartment_kitchen_utensil_02	bathroom_light_green_pump_bottle
frl_apartment_kitchen_utensil_03	kitchenware_spice_container
frl_apartment_kitchen_utensil_04	kitchenware_white_paper_towel_holder
frl_apartment_kitchen_utensil_05	kitchenware_navy_blue_lidded_dish
frl_apartment_kitchen_utensil_06	kitchenware_short_glass

Appendix

Table 11: Renaming Results of ReplicaCAD.

Atomic Action Pattern	Generation Logic
Move to a named surface	If the object can fit on the surface.
Move to a location around a specific object	If there exists a place to fit this object, which only involves receptacle regions of one specific object.
Move to a location with specific direction relative to a specific object	If there exists a place to fit this object, which only involves receptacle regions of specific direction relative to the specific object.
Move to a location between two objects	If there exists a place to fit this object, and the covered receptacle regions correspond to complementary directional pairs from two different objects (e.g., left side of object A and right side of object B; front-left of object A and back-right of object B).

Appendix

Table 12: Patterns and Logics for Generating Atomic Actions.

Method	Validity	Normalized entity coverage (%)
	(%)	Source plat.	Target plat.	Moved obj.	Anchor obj.
GPTTaskGen	29	47.2	48.8	45.6	42.5
GPTTaskGen-SG	68	42.2	41.7	73.2	71.3
GPTTaskGen-SG-validated	100	34.5	36.1	61.2	60.5
ManiTaskGen	94	85.4	87.3	72.9	70.9

Appendix

Table 13: Validity and entity coverage of single-step tasks generated by four methods.

Stage	Evaluator	Agreement	FP	FN	Human $\kappa$	MPA
Filtering	Three-VLM vote, $\geq 2$ accepts	92%	7	1	0.902	0.953
Execution	GPT-5.6-Luna	79%	13	8	—	—
Execution	Claude-Haiku-4.5	75%	11	14	—	—
Execution	Gemini-3.5-Flash	84%	6	10	—	—
Execution	Three-VLM majority	85%	5	10	0.851	0.940

Appendix

Table 14: Agreement between VLM judgments and human consensus for feasibility filtering and execution evaluation. Each study contains 100 instances; all execution rows use the same episode set and human reference. Human reliability is repeated only on the ensemble row.

Feasible votes	Decision	Agreement with humans	FP	FN
0	Reject	15/15	0	0
1	Reject	17/18	0	1
2	Accept	29/34	5	0
3	Accept	31/33	2	0
Total	$\geq 2$ accepts	92/100	7	1

Appendix

Table 15: Feasibility-filtering agreement by the number of feasible votes.

Model	Human-based SR	VLM-based SR	Discrepancy
GPT-5.2	30%	34%	4
Gemini-3.1-pro	33%	38%	5
Claude-4.5-sonnet	35%	40%	5

Appendix

Table 16: Comparison between Human and VLM-based evaluation on Level-4 tasks.

Panel of Figure 12 — Figure 12: An example of VLM evaluation error. Left: Initial state with objects at the center of table. Right: Final state where objects mistakenly placed on the right instead of left.

Figure 13: Images Given at the Step 0 of the Successful Episode.

Figure 14: Images Given after Step 1 of the Successful Episode.

Figure 15: Images Given after Step 2 of the Successful Episode.

Figure 16: Images Given after Step 3 of the Successful Episode.

Figure 17: Images Given after Step 4 of the Successful Episode.

Figure 18: Images Given after Step 5 of the Successful Episode.

Figure 19: Images Given after Step 6 of the Successful Episode.

Figure 20: Images Given after Step 7 of the Successful Episode.

Figure 21: Images Given after Step 8 of the Successful Episode.

Figure 22: Images Given after Step 9 of the Successful Episode.

Figure 23: Images Given after Step 10 of the Successful Episode.

Figure 24: Images Given after Step 11 of the Successful Episode.

Figure 25: Images Given after Step 12 of the Successful Episode.

Figure 26: Images Given after Step 13 of the Successful Episode.

Figure 27: Images Given after Step 1 of the Failed Episode.

Figure 28: Images Given after Step 2 of the Failed Episode.

Figure 29: Images Given after Step 3 of the Failed Episode.

Figure 30: Images Given after Step 4 of the Failed Episode.

Figure 31: Images Given After Step 5 of the Failed Episode.

Embodied AI

Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.

See all 267 papers

Feb 12, 2026

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

We present Scene2Demo, an automated framework for offline embodied data generation. Given a single RGB scene image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, primitive action flows, multi-view execution videos, and offline robot-learning datasets. Its five role-specialized generation modules perform task understanding, symbolic decomposition, BDDL configuration generation, object-size adjustment, and action-flow planning. The resulting object-centric task specifications and action transitions are organized as path search over a scene-conditioned object-action graph. Failed or incomplete executions are refined by two feedback agents: a Robotic Safety Inspector that produces step-wise visual critiques and a Robotic Task Supervisor that revises action sequences or parameters using the rollout and previous evolution history. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6% execution success rate; on four representative long-horizon tasks, feedback-driven refinement improves both task success and subtask-level execution quality over primitive-only execution. Comparisons with RoboGen and GenSim2 show stronger execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0% and 92.0% success on two representative tasks, showing that the generated data supports downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/.

Xiang Liu, Sen Cui, Guocai Yao +4

Robot Demonstration Generation Synthetic Data Generation for Robotics Robot Manipulation Datasets Robot Task Planning

Ask PDF

Feb 5, 2026

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.

Wenbo Wang, Fangyun Wei, QiXiu Li +5

University of Sydney · 1Microsoft Research Asia · 3Tsinghua University

Mobile Manipulation Robotics Simulation Synthetic Data Generation for Robotics Vision-Language-Action Models Robot Manipulation Benchmarks Robot Manipulation Datasets

Ask PDF

Feb 24, 2026

How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective

Recent advances in vision-language models (VLMs) have shown promise for human-level embodied intelligence. However, existing benchmarks for VLM-driven embodied agents often rely on high-level commands or discretized action spaces, which are non-native settings that differ markedly from real-world control. In addition, current benchmarks focus primarily on high-level tasks and lack joint evaluation and analysis at both low and high levels. To address these limitations, we present NativeEmbodied, a challenging benchmark for VLM-driven embodied agents that uses a unified, native low-level action space. Built on diverse simulated scenes, NativeEmbodied includes three representative high-level tasks in complex scenarios to evaluate overall performance. For more detailed analysis, we further decouple the skills required by complex tasks and construct four types of low-level tasks, each targeting a fundamental embodied skill. This joint evaluation across task and skill granularities enables fine-grained assessment of embodied agents. Experiments with state-of-the-art VLMs reveal clear deficiencies in several fundamental embodied skills, and further analysis shows that these bottlenecks significantly limit performance on high-level tasks. NativeEmbodied highlights key challenges for current VLM-driven embodied agents and provides insights to guide future research.

Bo Peng, Pi Bu, Keyu Pan +7

University of Science and Technology of China · Alibaba Group · Independent Researcher +1

Embodied Agent Benchmarks Embodied AI Vision-Language Agents VLM Evaluation Robotic Control

Ask PDF

Abstract

Embodied AI

Explore similar work

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective