cs.ROOct 29, 2026
SaveGym-Anything for Robotics: Scaling physical simulations with digital agents
Organizations: Carnegie Mellon University
Abstract
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as . Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Figures & tables
Figure 1: Built with Gym-Anything for Robotics, RoboWorld covers physical work across major sectors of the U.S. economy , spanning 200 RL environments across 25 simulations.
Figure 2: Overview of Gym-Anything for Robotics. Phase 1: We select 120 simulations grounded in U.S. GDP data, covering the most valuable physical work across occupations and everyday life, and build 25 of them. Phase 2: A creation agent builds each simulation in code and produces evidence of its quality, which four audit agents check against our criteria. Phase 3: A task creation agent proposes tasks of targeted difficulty, which six audit agents check, and each task runs with many robots and variations. Phase 4: Models learn from privileged simulation state through in-context learning, coaching, and code policies that train VLAs.
Figure 3: The problem setup, illustrated with the wine-pouring simulation. A simulation provides the scene, physics, robots, and variations, and each of its tasks pairs an initial state with an instruction and a verifier. A robot policy observes camera images and robot feedback, and in simulation can also read internal state, such as the wine in each glass. It acts only through robot commands.
Figure 4: The creation-audit loop, shown with round 1 of the wine-pouring simulation. The creation agent builds the simulation and a report of evidence. Audit agents read the code, re-run the experiments, and report issues, quoted here from their reports. The creation agent fixes each issue and measures the result, and the loop repeats for three rounds.
| Simulation | Physics | Objects | Robot | Success check | Physics checks |
|---|---|---|---|---|---|
| Liquid: wine pouring | |||||
| SoftGym ( Lin et al., 2020 ) | position-based fluid | 1 liquid box cups | none, cup moved directly | fraction poured | – |
| ManiSkill2 ( Gu et al., 2023 ) | MLS-MPM | 1 liquid bottle beaker | 1 Panda end-effector deltas | level within 4 mm under 100 particles spilt | – |
| Ours | DFSPH, 2 mm particles, 93,750 per bottle cohesion Young’s-law wetting lip separation contact-line pinning headspace air and glugging added-mass coupling absorbing optics | 4 liquids: red, white, and rosé wine, water datasheet 75 cl bottle, Bordeaux, ISO tasting and lined glasses, carafes | 2 FR3 arms 4 robot setups Franka velocity, acceleration, and jerk limits friction grasp up to 70 N | volume in mL, e.g., 135–165 mL spill and drips in mL glass upright bottle returned | 12 physics checks, e.g., hydrostatic load, sloshing frequency, tipping force, pour onset 12 adversarial evaluator cases |
| Chemistry: titration | |||||
| Chemistry3D ( Li et al., 2025b ) | particle fluid reaction database whole-solution color | 3D-scanned glassware | Franka scripted pick, pour, stir | – | – |
Table 1: RoboWorld simulations compared with the closest prior simulations. Prior entries list what each paper reports, and – marks nothing reported. Blue marks physics, objects, and robot limits taken from or calibrated to real data, such as measured liquid properties, manufacturer datasheets, and reference tables.
Figure 5: The whiteboard simulation with each of its six robot setups. All six share the same classroom, markers, eraser, and physics.
| Benchmark | # Tasks | Liquids | Cloth | Built by AI | From real-world data | From job data | Physics checked against real data |
|---|---|---|---|---|---|---|---|
| Rigid objects | |||||||
| Meta-World ( Yu et al., 2020 ) | 50 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| RLBench ( James et al., 2020 ) | 100 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| RoboTwin 2.0 ( Chen et al., 2026 ) | 50 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| LIBERO ( Liu et al., 2023 ) | 130 | ✗ | ✗ | ✗ | ✓ † | ✗ | ✗ |
| RoboCasa365 ( Nasiriany et al., 2026 ) | 365 | ✗ | ✗ | ✓ ∗ | ✗ | ✗ | ✗ |
Table 2: Comparison of RoboWorld with robotics simulation benchmarks. ✓ yes, ✗ no, ✓ partial, following what each paper reports. Liquids and Cloth: at least one task simulates liquid or cloth, respectively. Built by AI: an AI model writes the scenes and task code. From real-world data: the tasks or simulations are chosen using surveys, activity datasets, or job data. From job data: they are chosen using occupation data, such as O*NET tasks and BLS wage data. Physics checked against real data: the paper compares simulated physical quantities with real measurements. ∗ An LLM proposes task outlines, and the authors write the task code. † Ego4D provides action templates, and the authors compose the tasks. ‡ Task types, since the paper defines no fixed set of tasks. § Tasks evaluated in the paper, which gives no total.
Figure 6: Frontier models fail every RoboWorld test task, but succeed on some easier tasks. Success rate (left) and average reward (right) on the 25 test tasks and on an easier subset of 25 tasks from the training split. All models use extra-high reasoning effort.
| Success (%) | Avg. reward | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Robot level | Robots | DoF | Tasks | Astra | Terra | Opus | Astra | Terra | Opus |
| Fixed-base arms | FR3, Panda, UR10e | 8 to 16 | 17 | 29 | 18 | 35 | 0.47 | 0.19 | 0.55 |
| Mobile manipulator | Mobile FR3 Duo | 20 | 5 | 20 | 0 | 20 | 0.47 | 0.21 | 0.34 |
| Humanoid | G1, H1-2 | 39 to 43 | 3 | 0 | 0 | 33 | 0.00 | 0.00 | 0.33 |
Table 3: Success drops as the robot’s degrees of freedom increase. Success rate and average reward of GPT-6 Astra, GPT-6.1 Terra, and Claude Opus 5.5 on the 25 easier tasks, grouped by robot.
Figure 7: Frontier models learn in simulation, and their code policies train a small VLA. Left: average reward of Claude Opus 5.5 over 11 RoboWorld simulations after each of three attempts. Only the teacher has privileged access during its attempts, and self-transfer resumes the teacher with restricted access. Right: success rate on two gift box tasks. A frontier model writes code policies with privileged access, and their demonstrations train , which observes only camera images and robot proprioception, as on a real robot. Opus 5.5 controls the robot step by step.
Figure 8: Frontier models often build their own tools to solve tasks, but also overstate their success. (a) In football, Claude Opus 5.5 kicks the ball while the robot falls. (b) In percussion, Claude Opus 5.5 detects each strike and its note from the simulated microphone (bottom right). (c) In the whiteboard task, GPT-6 Astra estimates the tilt of its fallen robot from the ceiling grid in this head-camera image. (d) When GPT-6 Astra reported that the brake had stopped the forklift, the truck had in fact reversed into the dock wall. (e) In filleting, Claude Opus 5.5 places the whole uncut fish in the pan meant for the fish frame. Each image is either the camera image that the model received or a render of a verified replay of the evaluated episode.
Figure 9: Blind comparison with the strongest prior simulator of each interaction , over 25 pairs judged twice with swapped order. Visual ties are 6%.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Command | Parameters |
| Task space | Cartesian pose | end-effector position (m) and orientation, duration |
| Cartesian delta | end-effector displacement (m), duration | |
| Bimanual Cartesian | one target per arm, executed together | |
| Cartesian wrench | force (N) and moment (N m) at the tool | |
| Joint space | Joint position | joint angles (rad), duration |
| Named joint target | targets by joint name (rad or m), physics ticks |
Table 4: Robot commands. Each simulation accepts the subset that its robot supports.
| Keys | Effect |
|---|---|
| A / D, W / S, R / F | move the end effector along , , and |
| I / K, J / L, U / O | rotate the end effector about , , and |
| Q / E | close or open the gripper |
| H, G + H, B + H | squeeze with default, gentle, or strong force |
| ALT, SHIFT | slow and precise, or fast arm motion |
| P, T | slow or fast gripper motion |
Table 5: Keyboard map. Keys in one chord are held together, for instance W and R to move along and at once.
| Simulation | Physics | Objects | Robot | Success check | Physics checks |
|---|---|---|---|---|---|
| Acoustic contact: percussion playing | |||||
| RoboPianist [ Zakka et al., 2023 ] | MuJoCo 88 keys as joints with linear springs synthesized note when key within 0.5 ∘ of full press | 88-key keyboard, Kawai manual dimensions 150 annotated pieces | 2 Shadow Dexterous Hands, 44 actuators joint targets at 20 Hz, PD at 500 Hz | precision, recall and F1 of key states vs piano roll | – |
| DexDrummer [ Fang et al., 2026 ] | ManiSkill simulation stick held by finger contact | drum set: snare, tom, ride, hi-hat, crash MIDI songs, six genres | 2 Franka Panda arms 2 Tesollo DG-5F hands, 20 DoF each joint targets at 20 Hz, PID at 100 Hz | F1 of drum hits vs MIDI stick-hold ratio stick trajectory error and energy | – |
| Ours | 48 kHz implicit Hertz/Hunt–Crossley vibro-contact 13 undercut Timoshenko bars fitted to measured partials, modes to 20 kHz measured rosewood loss-factor floor air-loaded drum membrane, 474–810 modes per tension coupled bar–Helmholtz resonator bank, 289 complex modes drumstick bending modes as contact compliance Coulomb friction with Mindlin static-friction spring modal sound radiation: bar dipoles, membrane, resonator monopoles | Orff alto xylophone: 13 rosewood bars C4–A5, 6 box resonators 2 resonator bars, C4 and G4, datasheet boxes 14 5 in drum, coated Mylar head, 3 tensions fitted to measured modes 5A hickory drumsticks, rubber mallets, felt beater 6 5 2.75 m therapy room, wall and furniture colliders | 2 FR3 arms with Franka Hands 4 robot setups: 2 FR3, 1 FR3, 2 Panda, G1 Unitree G1 with 2 Dex3-1 hands, floating base Franka velocity, 10 rad/s 2 , 5000 rad/s 3 limits libfranka collision and velocity reflexes | 8 tasks: finger tap, take sticks, strike and stop, accent ostinato, echo, pentatonic setup, steady beat, melody clean stroke: one contact 15 ms on drum, 3 ms on bar tempo within 1–3 %, CV 3–4 % level within 4 dB of a replayed reference stroke accents 6 dB above soft strokes | 18 physics checks, e.g., bar partials and T60 vs Rucz and Hill, drumhead modes vs Wagner, stick contact time vs Wagner and Dahl |
| Adhesion: label application | |||||
| TapeSim [ Luo et al., 2026 ] | triangular thin shell, StVK membrane, discrete-shell bending IPC barrier contact and friction cohesive adhesion or releasable bonds | 19 mm tape roll, 2–32 turns tabletop, stand and core box for a sealing demo | Marvin arms with Wuji hands grasp constraints | Peel: complete tabletop release Stick: tape lifted and attached to the stand Wrap: at least 1 or 2 turns | peel force fit to measurement |
Table 6: All 25 RoboWorld simulations compared with the two closest prior simulations each. Blue marks physics, objects, and robot limits taken from or calibrated to real data, such as measured material properties, manufacturer datasheets, standards, and reference tables.
More in this topic
Robotics Simulation
Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.
See all 322 papers