cs.ROOct 29, 2026

Gym-Anything for Robotics: Scaling physical simulations with digital agents

Authors: Pranjal Aggarwal, Sean Welleck

Organizations: Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as π0.5π_{0.5}. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics

Figures & tables

Figure 1: Built with Gym-Anything for Robotics, RoboWorld covers physical work across major sectors of the U.S. economy , spanning 200 RL environments across 25 simulations.
Figure 2: Overview of Gym-Anything for Robotics. Phase 1: We select 120 simulations grounded in U.S. GDP data, covering the most valuable physical work across occupations and everyday life, and build 25 of them. Phase 2: A creation agent builds each simulation in code and produces evidence of its quality, which four audit agents check against our criteria. Phase 3: A task creation agent proposes tasks of targeted difficulty, which six audit agents check, and each task runs with many robots and variations. Phase 4: Models learn from privileged simulation state through in-context learning, coaching, and code policies that train VLAs.
Figure 3: The problem setup, illustrated with the wine-pouring simulation. A simulation provides the scene, physics, robots, and variations, and each of its tasks pairs an initial state with an instruction and a verifier. A robot policy observes camera images and robot feedback, and in simulation can also read internal state, such as the wine in each glass. It acts only through robot commands.
Figure 4: The creation-audit loop, shown with round 1 of the wine-pouring simulation. The creation agent builds the simulation and a report of evidence. Audit agents read the code, re-run the experiments, and report issues, quoted here from their reports. The creation agent fixes each issue and measures the result, and the loop repeats for three rounds.
SimulationPhysicsObjectsRobotSuccess checkPhysics checks
Liquid: wine pouring
SoftGym ( Lin et al., 2020 )position-based fluid1 liquid ⋅\cdot box cupsnone, cup moved directlyfraction poured–
ManiSkill2 ( Gu et al., 2023 )MLS-MPM1 liquid ⋅\cdot bottle ⋅\cdot beaker1 Panda ⋅\cdot end-effector deltaslevel within 4 mm ⋅\cdot under 100 particles spilt–
OursDFSPH, 2 mm particles, 93,750 per bottle ⋅\cdot cohesion ⋅\cdot Young’s-law wetting ⋅\cdot lip separation ⋅\cdot contact-line pinning ⋅\cdot headspace air and glugging ⋅\cdot added-mass coupling ⋅\cdot absorbing optics4 liquids: red, white, and rosé wine, water ⋅\cdot datasheet 75 cl bottle, Bordeaux, ISO tasting and lined glasses, carafes2 FR3 arms ⋅\cdot 4 robot setups ⋅\cdot Franka velocity, acceleration, and jerk limits ⋅\cdot friction grasp up to 70 Nvolume in mL, e.g., 135–165 mL ⋅\cdot spill and drips in mL ⋅\cdot glass upright ⋅\cdot bottle returned12 physics checks, e.g., hydrostatic load, sloshing frequency, tipping force, pour onset ⋅\cdot 12 adversarial evaluator cases
Chemistry: titration
Chemistry3D ( Li et al., 2025b )particle fluid ⋅\cdot reaction database ⋅\cdot whole-solution color3D-scanned glasswareFranka ⋅\cdot scripted pick, pour, stir––
Table 1: RoboWorld simulations compared with the closest prior simulations. Prior entries list what each paper reports, and – marks nothing reported. Blue marks physics, objects, and robot limits taken from or calibrated to real data, such as measured liquid properties, manufacturer datasheets, and reference tables.
Figure 5: The whiteboard simulation with each of its six robot setups. All six share the same classroom, markers, eraser, and physics.
Benchmark# TasksLiquidsClothBuilt by AIFrom real-world dataFrom job dataPhysics checked against real data
Rigid objects
Meta-World ( Yu et al., 2020 )50✗✗✗✗✗✗
RLBench ( James et al., 2020 )100✗✗✗✗✗✗
RoboTwin 2.0 ( Chen et al., 2026 )50✗✗✗✗✗✗
LIBERO ( Liu et al., 2023 )130✗✗✗✓ †✗✗
RoboCasa365 ( Nasiriany et al., 2026 )365✗✗✓ ∗✗✗✗
Table 2: Comparison of RoboWorld with robotics simulation benchmarks. ✓ yes, ✗ no, ✓ partial, following what each paper reports. Liquids and Cloth: at least one task simulates liquid or cloth, respectively. Built by AI: an AI model writes the scenes and task code. From real-world data: the tasks or simulations are chosen using surveys, activity datasets, or job data. From job data: they are chosen using occupation data, such as O*NET tasks and BLS wage data. Physics checked against real data: the paper compares simulated physical quantities with real measurements. ∗ An LLM proposes task outlines, and the authors write the task code. † Ego4D provides action templates, and the authors compose the tasks. ‡ Task types, since the paper defines no fixed set of tasks. § Tasks evaluated in the paper, which gives no total.
Figure 6: Frontier models fail every RoboWorld test task, but succeed on some easier tasks. Success rate (left) and average reward (right) on the 25 test tasks and on an easier subset of 25 tasks from the training split. All models use extra-high reasoning effort.
    Success (%)Avg. reward
Robot levelRobotsDoFTasksAstraTerraOpusAstraTerraOpus
Fixed-base armsFR3, Panda, UR10e8 to 16172918350.470.190.55
Mobile manipulatorMobile FR3 Duo205200200.470.210.34
HumanoidG1, H1-239 to 43300330.000.000.33
Table 3: Success drops as the robot’s degrees of freedom increase. Success rate and average reward of GPT-6 Astra, GPT-6.1 Terra, and Claude Opus 5.5 on the 25 easier tasks, grouped by robot.
Figure 7: Frontier models learn in simulation, and their code policies train a small VLA. Left: average reward of Claude Opus 5.5 over 11 RoboWorld simulations after each of three attempts. Only the teacher has privileged access during its attempts, and self-transfer resumes the teacher with restricted access. Right: success rate on two gift box tasks. A frontier model writes code policies with privileged access, and their demonstrations train π0.5\pi_{0.5} , which observes only camera images and robot proprioception, as on a real robot. Opus 5.5 controls the robot step by step.
Figure 8: Frontier models often build their own tools to solve tasks, but also overstate their success. (a) In football, Claude Opus 5.5 kicks the ball while the robot falls. (b) In percussion, Claude Opus 5.5 detects each strike and its note from the simulated microphone (bottom right). (c) In the whiteboard task, GPT-6 Astra estimates the tilt of its fallen robot from the ceiling grid in this head-camera image. (d) When GPT-6 Astra reported that the brake had stopped the forklift, the truck had in fact reversed into the dock wall. (e) In filleting, Claude Opus 5.5 places the whole uncut fish in the pan meant for the fish frame. Each image is either the camera image that the model received or a render of a verified replay of the evaluated episode.
Figure 9: Blind comparison with the strongest prior simulator of each interaction , over 25 pairs judged twice with swapped order. Visual ties are 6%.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

LevelCommandParameters
Task spaceCartesian poseend-effector position (m) and orientation, duration
 Cartesian deltaend-effector displacement (m), duration
 Bimanual Cartesianone target per arm, executed together
 Cartesian wrenchforce (N) and moment (N m) at the tool
Joint spaceJoint positionjoint angles (rad), duration
 Named joint targettargets by joint name (rad or m), physics ticks
Appendix
Table 4: Robot commands. Each simulation accepts the subset that its robot supports.
KeysEffect
A / D, W / S, R / Fmove the end effector along xx , yy , and zz
I / K, J / L, U / Orotate the end effector about xx , yy , and zz
Q / Eclose or open the gripper
H, G + H, B + Hsqueeze with default, gentle, or strong force
ALT, SHIFTslow and precise, or fast arm motion
P, Tslow or fast gripper motion
Appendix
Table 5: Keyboard map. Keys in one chord are held together, for instance W and R to move along yy and zz at once.
SimulationPhysicsObjectsRobotSuccess checkPhysics checks
Acoustic contact: percussion playing
RoboPianist [ Zakka et al., 2023 ]MuJoCo ⋅\cdot 88 keys as joints with linear springs ⋅\cdot synthesized note when key within 0.5 ∘ of full press88-key keyboard, Kawai manual dimensions ⋅\cdot 150 annotated pieces2 Shadow Dexterous Hands, 44 actuators ⋅\cdot joint targets at 20 Hz, PD at 500 Hzprecision, recall and F1 of key states vs piano roll–
DexDrummer [ Fang et al., 2026 ]ManiSkill simulation ⋅\cdot stick held by finger contactdrum set: snare, tom, ride, hi-hat, crash ⋅\cdot MIDI songs, six genres2 Franka Panda arms ⋅\cdot 2 Tesollo DG-5F hands, 20 DoF each ⋅\cdot joint targets at 20 Hz, PID at 100 HzF1 of drum hits vs MIDI ⋅\cdot stick-hold ratio ⋅\cdot stick trajectory error and energy–
Ours48 kHz implicit Hertz/Hunt–Crossley vibro-contact ⋅\cdot 13 undercut Timoshenko bars fitted to measured partials, modes to 20 kHz ⋅\cdot measured rosewood loss-factor floor ⋅\cdot air-loaded drum membrane, 474–810 modes per tension ⋅\cdot coupled bar–Helmholtz resonator bank, 289 complex modes ⋅\cdot drumstick bending modes as contact compliance ⋅\cdot Coulomb friction with Mindlin static-friction spring ⋅\cdot modal sound radiation: bar dipoles, membrane, resonator monopolesOrff alto xylophone: 13 rosewood bars C4–A5, 6 box resonators ⋅\cdot 2 resonator bars, C4 and G4, datasheet boxes ⋅\cdot 14 ×\times 5 in drum, coated Mylar head, 3 tensions fitted to measured modes ⋅\cdot 5A hickory drumsticks, rubber mallets, felt beater ⋅\cdot 6 ×\times 5 ×\times 2.75 m therapy room, wall and furniture colliders2 FR3 arms with Franka Hands ⋅\cdot 4 robot setups: 2 FR3, 1 FR3, 2 Panda, G1 ⋅\cdot Unitree G1 with 2 Dex3-1 hands, floating base ⋅\cdot Franka velocity, 10 rad/s 2 , 5000 rad/s 3 limits ⋅\cdot libfranka collision and velocity reflexes8 tasks: finger tap, take sticks, strike and stop, accent ostinato, echo, pentatonic setup, steady beat, melody ⋅\cdot clean stroke: one contact ≤\leq 15 ms on drum, ≤\leq 3 ms on bar ⋅\cdot tempo within 1–3 %, CV ≤\leq 3–4 % ⋅\cdot level within 4 dB of a replayed reference stroke ⋅\cdot accents ≥\geq 6 dB above soft strokes18 physics checks, e.g., bar partials and T60 vs Rucz and Hill, drumhead modes vs Wagner, stick contact time vs Wagner and Dahl
Adhesion: label application
TapeSim [ Luo et al., 2026 ]triangular thin shell, StVK membrane, discrete-shell bending ⋅\cdot IPC barrier contact and friction ⋅\cdot cohesive adhesion or releasable bonds19 mm tape roll, 2–32 turns ⋅\cdot tabletop, stand and core ⋅\cdot box for a sealing demoMarvin arms with Wuji hands ⋅\cdot grasp constraintsPeel: complete tabletop release ⋅\cdot Stick: tape lifted and attached to the stand ⋅\cdot Wrap: at least 1 or 2 turnspeel force fit to measurement
Appendix
Table 6: All 25 RoboWorld simulations compared with the two closest prior simulations each. Blue marks physics, objects, and robot limits taken from or calibrated to real data, such as measured material properties, manufacturer datasheets, standards, and reference tables.
More in this topic

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 322 papers

Explore similar work

CardsList
Sep 30, 2026

SimEX: Simulation-Integrated Robotics AutoResearch

Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the LLMs' insufficient understanding of robots and physical environments. On the other hand, iterative trial-and-error tuning in the physical world (e.g., physical autoresearch) induces significant experimental cost and safety concerns. We introduce SimEX: Simulation-Integrated Robotics AutoResearch, an autoresearch framework that tightly integrates simulated experimentation, enabling coding agents to efficiently acquire physical capabilities for controlling real robots. SimEX operates in two stages. First, the agent conducts open-ended probe-and-optimize iterations in simulation, developing a robot toolbox with robust and generalizable capabilities. Second, the agent adapts the toolbox and the simulator together through only a few physical trials: each trial corrects the simulator, and the corrected simulator is used to diagnose failures and screen candidate repairs. We evaluate SimEX extensively in sim-to-sim settings and on physical robots. On challenging real-world manipulation tasks including towel folding, barcode scanning, and plate manipulation, SimEX enables coding agents to efficiently acquire robot skills without any demonstration and with only 10 minutes of real-robot interaction. These results suggest that simulation can be a critical component in achieving physical intelligence, not only as a source of training data that must closely replicate the real world, but also as a roughly correct laboratory where a coding agent develops the knowledge and procedures needed to act on the robot. More details and robot videos at https://robo-simex.github.io/

Jiaheng Hu, Roberto Martin-Martin, Peter Stone +3

Amazon FAR (Frontier AI & Robotics) · The University of Texas at Austin · Carnegie Mellon University

Robot Skill LearningRobotics SimulationSim-to-Real TransferRobotic ManipulationAutonomous ExperimentationCode Generation for Robotics
AskPDF
May 14, 2026

Coding Agent Is Good As World Simulator

World models have emerged as a powerful paradigm for building interactive simulation environments, with recent video-based approaches demonstrating impressive progress in generating visually plausible dynamics. However, because these models typically infer dynamics from video and represent them in latent states, they do not explicitly enforce physical constraints. As a result, the generated video rollouts are not physically plausible, exhibiting unstable contacts, distorted shapes, or inconsistent motion. In this paper, we present an agentic framework constructing physics-based world models through executable simulation code. The framework coordinates planning, code generation, visual review, and physics analysis agents. The planning agent converts the natural language prompt into a structured scene plan, the code agent implements it as executable simulation code, and the visual review agent provide visual feedback while the physics analysis agent checks physical consistency. The code is iteratively revised based on the feedback until the simulation matches the prompt reqirements and physical constraints. Experimental results show that our framework outperforms advanced video-based models in physical accuracy, instruction fidelity and visual quality, which could be applied to various scenarios including driving simulation and embodied robot tasks.

Hongyu Wang, Jingquan Wang, Bocheng Zou +2

Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison

Robotics SimulationWorld ModelsAgentic Code GenerationPhysics-Based SimulationEmbodied World Models
AskPDF
Sep 29, 2026

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

Kerui Ren, Yingxiang Xu, Kaiwen Song +4

Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University +3

Robot Skill LearningRobotics SimulationSim-to-Real TransferRobotic ManipulationRobot Learning from VideoRobot Learning
AskPDF