Many everyday robot manipulation skills are affordance-dependent, with success determined by whether the robot contacts the functional object region required by the subsequent action. Current simulation data generators obtain contacts from generic grasp estimators or per-object manual contact annotations, but generic estimators rank stable grasps without task semantics and often select contacts that are misaligned with the downstream action, while manual contact annotations must be rewritten for each new object and task. To solve these challenges, we introduce AffordSim, a scalable data generator and benchmark that integrates open-vocabulary 3D affordance prediction into simulation-based trajectory generation. Given a natural-language task description, AffordSim synthesizes a task-relevant scene, emits affordance queries, grounds them on object surfaces, samples region-conditioned grasps, and selects executable candidates with motion planning. It further randomizes object pose, texture, lighting, image noise, and cross-viewpoint backgrounds for sim-to-real transfer. We instantiate AffordSim as a 50-task benchmark across diverse manipulation skills, five robot embodiments, and 500+ rigid and articulated objects. AffordSim achieves 93% of the trajectory collection success rate of manual contact annotations on affordance-critical tasks and 89% on hard composite tasks. Vision-language-action policies trained on AffordSim data transfer zero-shot to a real Franka FR3, reaching 24% average success.
Figures & tables
Figure 1: Overview of AffordSim. A scalable affordance-aware manipulation benchmark of 50 tasks, 500+ rigid and articulated objects, and 5 robot embodiments, paired with cross-viewpoint background replacement and affordance-guided trajectory generation.
Figure 2: Affordance-guided data generation pipeline of AffordSim. The full module breakdown and data flow are described in Section 3 . In the Stage 2 bar chart, the x-axis lists the candidate grasps gk generated in Stage 1 and the y-axis gives their motion-planner score Sk , with the red cross marking the colliding candidate ( Sk=0 ) and the green dashed line indicating the maximum-score grasp ultimately selected.
Figure 3: Domain randomization in AffordSim. Each column shows the cumulative effect of adding one randomization axis: object pose, surface texture, lighting, and background. The rightmost column shows example background textures used for randomization. Two representative tasks are shown across rows.
Figure 4: Task gallery. Representative frames from all 50 tasks of the AffordSim benchmark, grouped into seven manipulation categories: pick & place (20), open/close (2), pull/push (5), hang (2), pour (7), stack (4), and long-horizon composite (10).
Tier
Task
Manual
AnyGrasp
AnyGrasp+Filter
AffordSim
Human Aff.
Easy
pick banana place bowl
100
80
100
100
100
pick soupcan place table
100
55
75
95
100
⋯ (8 more tasks)
⋯
⋯
⋯
⋯
⋯
Average (10 tasks)
100
67
87
98
100
Medium
pick mugcup place coffeemachine
90
25
55
88
95
hang cup
85
0
0
25
90
Table 1: Affordance integration on the AffordSim benchmark. Trajectory collection success rate (%) per task across the five grasp methods. AnyGrasp+Filter denotes AnyGrasp candidates filtered against a 2D SAM2 task-relevant mask, mirroring the GenManip-style affordance-filtering pipeline. Two representative tasks per tier are listed; full breakdown in Appendix D . Tier averages bolded.
Tier
Task
BC
DP
ACT
VLA-adapter
Pi 0.5
Easy
pick banana place plate
40
82
47
87
93
pick leek place cuttingboard
33
70
68
84
86
pick chocolatebox place sugarbox
43
76
79
88
89
Tier average (3 tasks)
39
76
65
86
89
Medium
pick cup place shelf
7
28
26
55
61
pick mug place coffee machine
10
33
27
51
53
Table 2: Benchmark evaluation results (Success Rate %) on 20 representative tasks organized by the three difficulty tiers of Section 3.4 . Each task is evaluated over 100 rollouts. Best result per task in bold ; tier averages bolded.
Figure 7
Figure 6: Zero-shot sim-to-real deployment. The four real robot tasks in Table 3 , executed by Pi 0.5 on a Franka FR3 with no real world fine tuning.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Why affordance is essential for trajectory generation. Comparison of AnyGrasp and our affordance-guided grasping on two affordance-dependent tasks, pick & place and hang.
Figure 8: Cross-embodiment support. Start and end frames of three representative affordance-dependent tasks ( hang mug on rack , pour kettle into bowl , pick scallions on board ) generated by AffordSim for each of the five supported embodiments.
Tier
Task
Manual
AnyGrasp
AnyGrasp+Filter
AffordSim
Human Aff.
Easy
pick banana place bowl
100
80
100
100
100
pick bottle place table
100
70
90
100
100
pick bowl place plate
100
67
87
100
100
pick carrot place bowl
100
73
90
100
100
pick carrot place plate
100
67
87
100
100
pick chocolatebox place sugarbox
100
70
87
93
100
Appendix
Table 4: Full per-task AffordSim trajectory collection success rate. Per-task success rate (%) for every task of the AffordSim benchmark across the five grasp methods, grouped by difficulty tier as in Table 1 . Each task is evaluated over 30 rollouts. Per-tier averages are bolded.
Affordance backbone
Easy (4)
Medium (3)
Hard (3)
Avg
OpenAD [ 38 ]
65
22
5
28
IAGNet [ 64 ]
72
32
10
35
LASO [ 65 ]
82
50
20
50
3D-AffordanceLLM [ 40 ]
90
65
33
63
VoxAfford [ 12 ]
96
80
52
76
Appendix
Table 5: Affordance model ablation. Trajectory collection success rate (%) when only the affordance backbone is swapped, with all other modules of AffordSim held fixed. The ablation set has 10 tasks distributed as 4 Easy, 3 Medium, and 3 Hard, each evaluated over 100 rollouts. Best per tier in bold .
Task
Stage
BC
DP
ACT
VLA-adapter
Pi 0.5
Pick & Place
pick banana place plate
pick
—
90
63
93
93
place
—
87
47
87
93
pick cup place shelf
pick
—
67
87
100
100
place
—
28
26
55
61
pick mug place coffee machine
pick
—
63
57
93
93
Appendix
Table 6: Sub-task success rate breakdown (%) for the 10 multi-stage tasks in the 20-task benchmark. The final stage of each task equals the overall task SR reported in Table 2 . Best per stage in bold .
Figure 9: Depth Anything 3 input photographs. Five representative phone photographs of each deployment scene, drawn from a set of 14–20 captures per scene. Each row corresponds to one of the three scenes reconstructed in Figure 10 . The same photograph set is fed to Depth Anything 3 for joint depth and pose recovery, after which the 3D Gaussian Splatting field is fit and rendered into the simulator.
Figure 10: Depth Anything 3 background reconstruction across three deployment scenes. Each row is one real workspace reconstructed from 10–20 photographs by the pipeline described in Appendix N . The first column shows the canonical viewpoint render and the next four columns show novel viewpoints generated by orbiting a virtual camera around the workspace, at four representative angles within the supported [−120∘,+45∘] orbit range.
Despite the recent success of modern imitation learning methods in robot manipulation, their performance is often constrained by geometric variations due to limited data diversity. Leveraging powerful 3D generative models and vision foundation models (VFMs), the proposed AffordGen framework overcomes this limitation by utilizing the semantic correspondence of meaningful keypoints across large-scale 3D meshes to generate new robot manipulation trajectories. This large-scale, affordance-aware dataset is then used to train a robust, closed-loop visuomotor policy, combining the semantic generalizability of affordances with the reactive robustness of end-to-end learning. Experiments in simulation and the real world show that policies trained with AffordGen achieve high success rates and enable zero-shot generalization to truly unseen objects, significantly improving data efficiency in robot learning. Project Page: https://jiaweiz9.github.io/AffordGen-release/
Jiawei Zhang, Kaizhe Hu, Yingqian Huang +3
Shanghai Qi Zhi Institute · Tsinghua University · Fudan University +1
Robot learning in simulation depends on the objects the simulator offers. Many tasks need objects with separate parts, joints that allow the required motion, and physical properties that remain valid under contact. Existing methods recover this structure anew for every image: generative models predict parts and joints that mostly fail to settle or move in simulation, and general-purpose agents need a long session of model calls for each photograph. AffordCraft builds such an asset from a single RGB image and a task instruction by retrieval instead of generation: it locates the object and the part to operate, selects a matching entry from a library of articulated assets, and fits it to the image while keeping its parts and joints intact. Without any box or mask marking the object, AffordCraft produces a physically valid asset for 1,703 of 2,000 photographs from 31 categories. Five generative methods pass on at most 45% of the same photographs and, at the median, need 10 to 78 times our GPU time per valid asset. On 50 cluttered images, 162 of 237 annotated objects pass the same physical test after automatic detection. Growing the library from 141 to 11,372 entries needs no change to the method and raises category coverage from 46% to 100% and the share of selections with the requested label from 18% to 51%. We also build manipulation tasks from the constructed assets, both with single objects and in composed scenes; policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training.
Haoyun Yang, Xueyang Zhou, Ziyi Xie +1
College of AI, Tsinghua University · School of Computer Science, Chongqing University · School of Cyber Science and Engineering, Huazhong University of Science and Technology +1
Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.
Bowei Li, Yuner Zhang, Changliu Liu
Carnegie Mellon University, Pittsburgh, PA 15213, USA