RoboSimStudio: An Interactive Toolkit for Human and Agent Rapid Real2Sim of Robot Manipulation Tasks

Kaiyuan Chen; Shuangyu Xie; Uksang Yoo; Xinyue Zhu; Cael Magner; Alvin Zhu; Daehwa Kim; Eric Heiden

cs.ROOct 13, 2026

RoboSimStudio: An Interactive Toolkit for Human and Agent Rapid Real2Sim of Robot Manipulation Tasks

Authors: Kaiyuan Chen, Shuangyu Xie, Uksang Yoo, Xinyue Zhu, Cael Magner, Alvin Zhu, Daehwa Kim, Eric Heiden,

Organizations: University of California, Berkeley · NVIDIA

Abstract

It has been difficult to accurately model the physics of the interaction between a robot and the objects it manipulates, including contact mechanics, coupled contact dynamics, deformations, stiffness and friction. We present RoboSimStudio (RSS), an interactive toolkit that makes choices of solver, contact model, and physical parameters explicit and editable, allowing both humans and coding agents to quickly create and iteratively refine simulations based on observed physical behavior. We implement RSS with the open Newton physics simulation system and describe experiments with human-guided real2sim using (1) visual evaluation, (2) force-feedback and (3) agentic Real2Sim2Real. Code, video and additional information can be found on the project website: https://robosimstudio.github.io

Figures & tables

Physics Simulation
Feature	MuJoCo	IsaacLab	ManiSkill3	Genesis	Newton	RoboSimStudio
Rigid/articulated bodies	✓	✓	✓	✓	✓	✓
Deformable objects	✓	✓	$\sim$	✓	✓	✓
Coupled multi-physics solvers	–	$\sim$	–	✓	$\sim$	✓
Interactive Interfaces
Interactive scene inspection	✓	✓	✓	✓	✓	✓

Table 1: Comparison of robotics simulation frameworks. Comparison of physics capabilities, interactive interfaces, and Real2Sim workflows. ✓ indicates built-in support,

\sim

indicates partial support, and “Custom” indicates that additional implementation is required.

Figure 2: RoboSimStudio Architecture. RoboSimStudio integrates asset creation, robot interfaces, and multi-physics simulation through Newton and external physics solvers. It supports interactive task visualization, human-guided Real2Sim, and agentic Real2Sim through a shared simulation framework.

Panel of Figure 3 — Figure 3: Multi-physics coupling in RoboSimStudio. Five manipulation tasks using different physics solvers and coupling methods. Top: rendered simulation scenes. Bottom: visualization of the physics solvers used for each object, including MuJoCo-Warp (violet), finite-element simulation (FEM, teal), and Vertex Block Descent (VBD, amber). All tasks use the same robot interface and scene description framework in RoboSimStudio.

Figure 4: RoboSimStudio Interface. Left: a towel and its inspector with material controls, for example, tri_ke controls the stretch stiffness and edge_ke controls the bending stiffness. Right: one towel, four bending and stretch stiffness: the same dragged fold at the same instant.

Panel of Figure 5 — Figure 5: 18 Challenging contact-rich deformable manipulation task simulations created by RoboSimStudio

Figure 6: Tool-hanging Teleoperation with RoboSimStudio

Panel of Figure 7 — Figure 7: Agentic Real2Sim2Real for sponge wiping. (a) A real-world bimanual manipulation task, where one arm holds a bar while the other wipes a marker stripe using a sponge. (b) Simulation challenges involving robot motion, object-object contact, and robot-object contact. (c) A coding agent constructs and iteratively refines the simulation through the interactive studio, completing a verified simulated wipe in 56 minutes. (d) A multi-physics simulation in RoboSimStudio. (e) Physical variation of sponge stiffness and spatial variation of the bar and sponge across 30 generated layouts (bottom). (f) GaP [ 5 ] improves the simulated wiped area from 0% to 100% over 14 iterations using spatial-temporal simulation feedback from rss. (g) The improved policy is deployed on the real robot to perform sponge wiping. (h) A real-world failure caused by displacement of a bar support that was fixed in the original simulation. (i) Given the observed failure, the coding agent revises the physical simulation to allow the bar and its supports to move under contact. Under an off-center wiping motion, the revised simulation captures bar displacement and tipping.

Real-to-Sim Transfer

See all 14 papers

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
Jul 21, 2026Guanxiong Chen, Qianjun Xia, Jiawei Peng +24World Models for Robotics Robotics Simulation
University of British Columbia. · National University of Singapore. · Johns Hopkins University. +4
CRISP: Contact-Rich Robotic Simulation Platform with Extensive Geometries and Contact Solvers
Sep 18, 2026Somang Lee, Sunkyung Park, Jinhee Yun +2Robotics Simulation Physics-Based Simulation
Department of Mechanical Engineering, Seoul National University
Beyond Heuristics: A Standardized Real2Sim Pipeline for Physical Human Robot Interaction in Human-in-the-Loop Simulation
Jul 3, 2026Chengyuan Yang, Yifan Wang, Chun Kwang Tan +5Robot System Identification System Identification
School of Mechanical and Aerospace Engineering, Nanyang Technological University, 639798 Singapore · Guangdong Zhongxin Intelligent Rehabilitation Research Institute, Foshan, China 528200

Jul 21, 2026J/K move · Enter open · S save

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang+23

University of British Columbia. · National University of Singapore. · Johns Hopkins University.+4

World Models for Robotics Robotics Simulation Real-to-Sim Transfer VLMs for Robotics

Open paper AskPDF arXiv

Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on brittle workflow glue across visual perception tools and simulators: manual tuning of visual foundation models, mesh cleanup, coordinate frame alignments, etc. We introduce Agentic Real2Sim, a framework for generalized physical world modeling with vision-language agents that converts a real-world recording of object-robot interaction into a simulatable episodic twin, and connects the resulting twin to downstream policy fine-tuning and evaluation. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining a comparable conversion success rate. The framework further supports custom scene conversion, fine-tuning of a pretrained policy with data generated from converted episodes, and works effectively as a surrogate for real-world policy evaluation. The project site, including code is available at https://agentic-real2sim.github.io.

Abstract

Real-to-Sim Transfer

Explore similar work

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

CRISP: Contact-Rich Robotic Simulation Platform with Extensive Geometries and Contact Solvers

Beyond Heuristics: A Standardized Real2Sim Pipeline for Physical Human Robot Interaction in Human-in-the-Loop Simulation