cs.ROMay 27, 2025
SaveManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
Organizations: University of California, San Diego
Abstract
Building embodied agents capable of accomplishing arbitrary tasks is a core objective towards achieving embodied artificial general intelligence (E-AGI). While recent work has advanced such general robot policies, their training and evaluation are often limited to tasks within specific scenes, involving restricted instructions and scenarios. Existing benchmarks also typically rely on manual annotation of limited tasks in a few scenes. We argue that exploring the full spectrum of feasible tasks within any given scene is crucial, as they provide both extensive benchmarks for evaluation and valuable resources for agent improvement. Towards this end, we introduce ManiTaskGen, a novel system that automatically generates comprehensive, diverse, feasible mobile manipulation tasks for any given scene. The generated tasks encompass both process-based, specific instructions (e.g., "move object from X to Y") and outcome-based, abstract instructions (e.g., "clear the table"). We apply ManiTaskGen to both simulated and real-world scenes, demonstrating the validity and diversity of the generated tasks. We then leverage these tasks to automatically construct benchmarks, evaluating high-level embodied decision-making through a grounded skill interface for agents built upon existing vision-language models (VLMs). Furthermore, we propose a simple yet effective method that utilizes ManiTaskGen tasks to enhance embodied decision-making. Overall, this work presents a cross-scene task generation framework facilitating both benchmarking and improvement of embodied decision-making agents. Our code is available at https://manitaskgen.github.io.
Figures & tables
Figure 1: Framework Overview. ManiTaskGen systematically generates process-based and outcome-based navigate–pick–place tasks from scene information. The generated tasks support benchmark construction and feedback-based improvement of embodied decision-making agents.
Figure 2: Agent-Env Interaction Cycle. This cycle provides a fundamental depiction of how an embodied agent executes a task in a scene.
Figure 3: Visualization of Segmented Receptacle Regions on a Surface.
Figure 4: The "lightmap"s Which Show the Diversity of Generated Tasks. Each time an object or location is mentioned in a task, we add a highlight at the corresponding position. The brightness intensity and distribution reveal that our method generates tasks covering more objects and locations.
| Datasets | Num. of Scenes | Num. of Tasks |
|---|---|---|
| GenSim [ 33 ] | - | 100 |
| [ 51 ] | 20 | 521 |
| M3Bench [ 16 ] | 119 | 31,050 |
| ALFRED [ 34 ] | 120 | 25,743 |
| Language Rearrangement [ 7 ] | 1 | 1,000 |
| Embodied Agent Interface [ 17 ] | 2 | 438 |
Table 1: Comparison among Scene-Task Dataset Statistics.
| Process-based Tasks | Outcome-based Tasks | |
|---|---|---|
| GPTTaskGen-RAS | 29.4% | 21.1 % |
| ManiTaskGen-RAS-40K (Ours) | 94.0% | 86.5 % |
Table 2: Human-Verified Task Validity Rate.
Figure 5: Visualization of An Example Testing Episode. The agent is equipped with abstracted navigation ( go_to, change_view ), grasping ( pick ) and placing ( show_receptacle, place ) skills. Blue marks indicate the walkable locations around ground objects involved in this episode.
| Level 1 | Level 2 | Level 3 | Level 4 | Average | |||||
| IP | SR (%) | IP | SR (%) | IP | SR (%) | SR(%) | IP | SR (%) | |
| Human | 96.0 | 82.5 | 95.0 | 80.5 | 95.0 | 80.0 | 90.0 | 95.3 | 83.3 |
| Random | 0.7 | 0.0 | 0.7 | 0.0 | 0.7 | 0.0 | 0.0 | 0.7 | 0.0 |
| GPT-5.2 [ 52 ] | 88.1 | 63.9 | 67.9 | 34.7 | 72.8 | 20.0 | 30.0 | 76.3 | 37.2 |
| GPT-4o [ 52 ] | 71.6 | 40.1 | 42.9 | 16.3 | 57.0 | 8.2 | 26.0 | 57.2 | 22.7 |
| GPT-4.1 [ 52 ] | 67.5 | 36.4 | 46.4 | 20.4 | 54.4 | 4.2 | 28.0 | 56.1 | 22.3 |
Table 3: Evaluation Results on Existing VLMs. We cover both proprietary (upper part) and open-source models (lower part). Here, IP refers to Intermediate Points, and SR refers to Success Rate.
Figure 6: Success and Failure Modes from Gemini-2.5-pro in each stage of Level 3 tasks.
Figure 7: Set-up of Real Robot Experiments. The camera view was manually annotated with numerical tags included in the agent’s visual input. All irrelevant objects are masked out.
| Model Name | IP (Level1&2) | SR (Level1&2) | IP (Level3) | SR (Level3) |
|---|---|---|---|---|
| Gemini-2.5-pro [ 53 ] | 97.5 | 90% | 100 | 100% |
Table 4: Real Robot Experiment Results.
| Model Name | IP (Before) | IP (After) | SR (Before) | SR (After) |
|---|---|---|---|---|
| GPT-4.1 [ 52 ] | 75.5 | 79.4 | 38% | 49% |
| Gemini-2.5-flash [ 53 ] | 79.4 | 81.7 | 36% | 58% |
| Claude-3.5-haiku [ 54 ] | 68.9 | 79.4 | 36% | 51% |
Table 5: Improvement of Agent Performance.
Figure 8: Improvement of Gemini-2.5-flash Agent. We find that more trial episodes continuously contribute to the agent performance.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Per-axis noise bound | L1 (%) | L2 (%) | L3, two-step (%) | L4 (%) |
|---|---|---|---|---|
| No noise | 94 | 94 | 90 | 86 |
| 5 cm | 89 | 87 | 79 | 85 |
| 25 cm | 71 | 72 | 53 | 85 |
Table 6: Task validity under input localization noise. Each cell summarizes 100 manually verified tasks generated after rebuilding the scene graph for that noise setting. This separate study does not replace the original validity comparison or measure robot success rates.
Figure 9: Visualization of Rules Involved in Scene Graph Construction.
| Primary intent | Definition / decision rule | Templates | Instances |
|---|---|---|---|
| (%) | (%) | ||
| Ordering / alignment | Explicit order, gradient, row, grid, line, symmetry, or regular alignment. | 36 (18.0) | 57 (8.8) |
| Display / presentation / curation | Informative, interactive, aesthetic, product, gallery, or focal-point display. | 31 (15.5) | 104 (16.0) |
| Functional workspace / workflow setup | Workspace, studio, lab, meeting/collaboration area, or multi-step workflow. | 24 (12.0) | 51 (7.8) |
| Artistic / geometric arrangement | Non-representational artistic composition, geometric pattern, sculpture, or installation. | 24 (12.0) | 57 (8.8) |
| Thematic scene / structural construction | Recognizable scene, model, navigable layout, or physical structure, e.g., city, maze, garden, bridge. | 24 (12.0) | 40 (6.2) |
Table 7: Primary-intent distributions of outcome-based templates and task instances. Percentages use 200 templates and 650 instances, respectively; instance percentages are rounded independently. Counts are taken from the archived template classification and instance-intent statistics.
| Semantic / reasoning tag | Scope / examples | Templates | Instances |
| (%) | (%) | ||
| Thematic / aesthetic composition | Visual harmony, display, story/theme, color scheme, atmosphere, centerpiece. | 99 (49.5) | 49.5 |
| Semantic categorization | Category, type, material, function, power source, project, similarity, thematic grouping. | 61 (30.5) | 16.0 |
| Symmetry / geometric reasoning | Symmetry, mirror, circle, spiral, grid, triangle, geometric shapes, maze/path geometry. | 60 (30.0) | 19.5 |
| Attribute comparison & ordering | Size, height, length, color/brightness, frequency, relevance, chronology, alphabetical order. | 54 (27.0) | 28.0 |
| Accessibility / ergonomics / workflow | Access, reachability, retrieval, interaction, efficiency, collaboration, handover, navigation. | 53 (26.5) | 25.8 |
Table 8: Multi-label semantic and reasoning coverage of outcome-based templates and task instances. Percentages use 200 templates and 650 instances; they need not sum to 100%. Instance-level tag frequencies are reported as rounded percentages, with approximately 2.18 tags per instance.
Figure 10: Visualization of the 3 Scenes Used to Generate ManiTaskGen-RAS-40K.
| Dataset | Scenes | Represented task-space count | ||
|---|---|---|---|---|
| Level 1+2 | Level 3 | Level 4 | ||
| SUNRGBD | 5 | |||
| ReplicaCAD | 5 | |||
| AI2THOR | 90 | |||
| Total | 100 | |||
Table 9: Task-space statistics across 100 source scenes. Levels 1–2 count represented single-step candidates; Level 3 counts two-step combinations under the generation rules; Level 4 is an estimate rather than an exhaustive VLM-filtered enumeration. None of these columns measures semantically deduplicated behaviors. Actual RAS-40K instances are listed in Table 10 .
| Dataset | Level 1 | Level 2 | Level 3 | Level 4 | Total |
|---|---|---|---|---|---|
| ReplicaCAD | 2195 | 24153 | 5000 | 500 | 31848 |
| AI2THOR | 2218 | 0 | 5000 | 100 | 7318 |
| SUN-RGBD | 35 | 120 | 500 | 50 | 705 |
| Total | 4448 | 24273 | 10500 | 650 | 39871 |
Table 10: Statistics of ManiTaskGen-RAS-40K across 3 Source Scenes.
Figure 11: Ambiguous Item Naming in ReplicaCAD.
| Name (before) | Name (after) |
|---|---|
| frl_apartment_kitchen_utensil_01 | kitchenware_ceramic_lidded_container |
| frl_apartment_kitchen_utensil_02 | bathroom_light_green_pump_bottle |
| frl_apartment_kitchen_utensil_03 | kitchenware_spice_container |
| frl_apartment_kitchen_utensil_04 | kitchenware_white_paper_towel_holder |
| frl_apartment_kitchen_utensil_05 | kitchenware_navy_blue_lidded_dish |
| frl_apartment_kitchen_utensil_06 | kitchenware_short_glass |
Table 11: Renaming Results of ReplicaCAD.
| Atomic Action Pattern | Generation Logic |
|---|---|
| Move to a named surface | If the object can fit on the surface. |
| Move to a location around a specific object | If there exists a place to fit this object, which only involves receptacle regions of one specific object. |
| Move to a location with specific direction relative to a specific object | If there exists a place to fit this object, which only involves receptacle regions of specific direction relative to the specific object. |
| Move to a location between two objects | If there exists a place to fit this object, and the covered receptacle regions correspond to complementary directional pairs from two different objects (e.g., left side of object A and right side of object B; front-left of object A and back-right of object B). |
Table 12: Patterns and Logics for Generating Atomic Actions.
| Method | Validity | Normalized entity coverage (%) | |||
|---|---|---|---|---|---|
| (%) | Source plat. | Target plat. | Moved obj. | Anchor obj. | |
| GPTTaskGen | 29 | 47.2 | 48.8 | 45.6 | 42.5 |
| GPTTaskGen-SG | 68 | 42.2 | 41.7 | 73.2 | 71.3 |
| GPTTaskGen-SG-validated | 100 | 34.5 | 36.1 | 61.2 | 60.5 |
| ManiTaskGen | 94 | 85.4 | 87.3 | 72.9 | 70.9 |
Table 13: Validity and entity coverage of single-step tasks generated by four methods.
| Stage | Evaluator | Agreement | FP | FN | Human | MPA |
|---|---|---|---|---|---|---|
| Filtering | Three-VLM vote, accepts | 92% | 7 | 1 | 0.902 | 0.953 |
| Execution | GPT-5.6-Luna | 79% | 13 | 8 | — | — |
| Execution | Claude-Haiku-4.5 | 75% | 11 | 14 | — | — |
| Execution | Gemini-3.5-Flash | 84% | 6 | 10 | — | — |
| Execution | Three-VLM majority | 85% | 5 | 10 | 0.851 | 0.940 |
Table 14: Agreement between VLM judgments and human consensus for feasibility filtering and execution evaluation. Each study contains 100 instances; all execution rows use the same episode set and human reference. Human reliability is repeated only on the ensemble row.
| Feasible votes | Decision | Agreement with humans | FP | FN |
|---|---|---|---|---|
| 0 | Reject | 15/15 | 0 | 0 |
| 1 | Reject | 17/18 | 0 | 1 |
| 2 | Accept | 29/34 | 5 | 0 |
| 3 | Accept | 31/33 | 2 | 0 |
| Total | accepts | 92/100 | 7 | 1 |
Table 15: Feasibility-filtering agreement by the number of feasible votes.
| Model | Human-based SR | VLM-based SR | Discrepancy |
|---|---|---|---|
| GPT-5.2 | 30% | 34% | 4 |
| Gemini-3.1-pro | 33% | 38% | 5 |
| Claude-4.5-sonnet | 35% | 40% | 5 |
Table 16: Comparison between Human and VLM-based evaluation on Level-4 tasks.
Figure 12: An example of VLM evaluation error. Left: Initial state with objects at the center of table. Right: Final state where objects mistakenly placed on the right instead of left.
Figure 13: Images Given at the Step 0 of the Successful Episode.
Figure 14: Images Given after Step 1 of the Successful Episode.
Figure 15: Images Given after Step 2 of the Successful Episode.
Figure 16: Images Given after Step 3 of the Successful Episode.
Figure 17: Images Given after Step 4 of the Successful Episode.
Figure 18: Images Given after Step 5 of the Successful Episode.
Figure 19: Images Given after Step 6 of the Successful Episode.
Figure 20: Images Given after Step 7 of the Successful Episode.
Figure 21: Images Given after Step 8 of the Successful Episode.
Figure 22: Images Given after Step 9 of the Successful Episode.
Figure 23: Images Given after Step 10 of the Successful Episode.
Figure 24: Images Given after Step 11 of the Successful Episode.
Figure 25: Images Given after Step 12 of the Successful Episode.
Figure 26: Images Given after Step 13 of the Successful Episode.
Figure 27: Images Given after Step 1 of the Failed Episode.
Figure 28: Images Given after Step 2 of the Failed Episode.
Figure 29: Images Given after Step 3 of the Failed Episode.
Figure 30: Images Given after Step 4 of the Failed Episode.
Figure 31: Images Given After Step 5 of the Failed Episode.
More in this topic
Embodied AI
Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.
See all 267 papers