cs.GRSep 29, 2026
SaveText2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise
Organizations: Tsinghua University · Carnegie Mellon University · Genesis AI · Shanghai Qi Zhi Institute
Abstract
Creating diverse physical simulations remains labor-intensive because assets, layout, physical parameters, motion, control, and rendering must be designed and debugged jointly. We present Text2Sim, a simulation-specialized agentic pipeline that converts a text-only request into an executable, editable dynamic case. Built on Genesis, Text2Sim uses a hierarchical agentic structure that combines a Planner with specialized Writers, asset-generation tools, and an independent Critic. Compact skills (Debug Cards) distilled from graphics demonstrations provide role-specific physical guidance for execution-based repair. We evaluate physical quality, visual quality, and human preference on 42 held-out prompts spanning rigid, articulated, deformable, and cloth phenomena, with a paper-level split between experience construction and evaluation. We design automatic physical and visual scorers to evaluate the quality of the results, and Text2Sim achieves higher scores than all four state-of-the-art baselines on both metrics. In blinded user studies with these baselines, significantly more participants prefer Text2Sim than prefer the baselines, which is consistent with the results from our automatic scorers. The pipeline also supports a broad range of downstream applications; we select dataset construction and extension to multimodal input as two representative examples. We will release the code, the Debug Card library, and a dataset of generated cases, each pairing the text prompt and rendered video with the executable program, assets, physical parameters, controls, and recorded states.
Figures & tables
Figure 1. Text2Sim generates physical simulations from text. (a) A chain-suspended sphere scatters rigid blocks. (b) A piston compresses and twists soft blocks inside a cylindrical cup. (c) Colorful cloth pieces tumble inside a rotating drum. The embedded sentences summarize the input prompts. The large views show interaction snapshots from simulations lasting 10, 10, and 12 s, respectively; upper-right insets show the initial frames. Three edge-to-edge panels show rigid impact, soft-body compression, and cloth tumbling, with a short prompt centered near the bottom of each image and labels a, b, and c at the lower left. Panel a frames the complete chain-suspended sphere and scattered colored blocks at 1.44 seconds. Panel b shows orange blocks compressed and twisted beneath a piston in a transparent cup at 8.64 seconds. Panel c looks into a rotating dryer drum at 5.40 seconds, emphasizing the lifted, falling, and folded cloth. Equal-sized, softly bordered upper-right insets show the first recorded states without covering the main interaction: 0 seconds for panels a and b, and 2.04 seconds for panel c.
Figure 2. Text2Sim pipeline. The Planner receives a text prompt, accesses simulator documentation and Debug Cards (reusable simulation skills), and invokes the tools in the dashed box to produce simulation code and video. Four Writers author the Scene, Body, Action, and Render modules; the Critic evaluates executed cases; Mesh and Articulation agents provide assets; and harness tools execute the code. Solid arrows indicate control or artifact flow, and purple dashed arrows indicate on-demand retrieval. The antenna of the Planner indicates that it is responsible for sending commands, while the headphones of the subagents indicate that they are responsible for receiving commands. The two modules are colored green to indicate that, unlike the blue modules, they are not built on general-purpose LLMs. The text prompt enters a Planner connected to simulator documentation and Debug Cards. The Planner invokes four Writers, a Critic, mesh and articulation asset tools, and an execution tool to produce code and video.
Figure 3. Writer responsibilities in a projectile–brick wall example, shown before and after impact. Green links the Scene Writer to the ground and simulation settings; blue links the Body Writer to object geometry and physical properties; orange denotes the Action Writer’s launch, simulation, and logging; purple denotes the Render Writer’s camera and lighting. The two path-traced frames share a crop, with simulation times labeled in seconds. Two frames show an intact brick wall and a projectile at 0 seconds, followed by scattered bricks at 1.04 seconds. Blue connects the Body Writer to the wall, green connects the Scene Writer to the ground, orange links the Action Writer to the progression between states, and purple links the Render Writer to the image frame and a camera symbol.
| Configuration | Scene | Body | Action | Render | Overall |
|---|---|---|---|---|---|
| Text2Sim | |||||
| GPT-5.6-Sol | |||||
| Qwen-3.8-Max | |||||
| Claude-Opus-5 | |||||
| Code2Worlds |
Table 1. Physical scores for our method and four baselines over all 42 cases (Total). Entries report the original means with their associated errors; bold marks the largest mean (best result) in each metric.
| Configuration | Content | Clarity | Polish | Appeal | Overall |
|---|---|---|---|---|---|
| Text2Sim | |||||
| GPT-5.6-Sol | |||||
| Qwen-3.8-Max | |||||
| Claude-Opus-5 | |||||
| Code2Worlds |
Table 2. Visual scores for our method and four baselines over all 42 cases (Total). Entries report the original means with their associated errors; bold marks the largest mean (best result) in each metric. Content, Clarity, Polish, and Appeal denote content fulfillment, demonstration clarity, visual polish, and demo appeal, respectively.
| Method | Task faithfulness | Physical plausibility | Visual quality | Overall preference |
|---|---|---|---|---|
| Text2Sim | ||||
| GPT-5.6-Sol | 50 (anchor) | 50 (anchor) | 50 (anchor) | 50 (anchor) |
| Qwen-3.8-Max | ||||
| Claude-Opus-5 | ||||
| Code2Worlds |
Table 3. User-study scores for the five main methods over all 42 cases (Total). Entries report the original means with their associated errors; GPT-5.6-Sol is a fixed reference at 50. Bold marks the largest mean (best result) for each dimension. Cell backgrounds encode the mean relative to the reference: white at 50, increasingly green above 50, and increasingly red below 50.
Figure 4. Jointed-net Debug Cards comparison. Full prompt: “Create a jointed net catching rigid body stacks demo: one 10*10-sized articulated rigid lattice made up of small capsule links and small sphere joints is hung above the ground with the four edges fixed. Gravity is enabled. Drop a compact stack of simple humanoids onto the center of the net so the lattice sags deeply, wrinkles, and settles.” Rows show Text2Sim with and without Debug Cards, from top to bottom. Columns show shared video timestamps of 0, 0.48, 0.64, 0.8, and 1.2 seconds, from left to right. Two rows of five early snapshots compare Text2Sim with and without Debug Cards. In the top row, a separated upright stack descends, its lower layer strikes the regular articulated net, and successive bodies rotate and pile as the net sags. The bottom row shows the flatter body arrangement contacting a net with folded-back strands. The first column shows the original initial rendered frames. Equal-height panels preserve image proportions, original camera views, and shared video timestamps.
| Setting | Value |
|---|---|
| Simulator | Genesis 0.4.5 |
| Renderer | LuisaRender, commit d176fd90 |
| Runtime | Python 3.12.13; PyTorch 2.11.0; CUDA 12.8 |
| GPUs | NVIDIA GeForce RTX 4090 D, 24 GB each |
| CPUs | AMD EPYC 7H12 (128 cores total) |
| RAM | 512 GiB |
Table 4. Current environment and implementation settings. GPU count describes the host.
Figure 5. Wheel-crawler task. Full prompt: “Create a Cosserat-rod scene: a roller contains two large rigid wheel-like circular frames on either side and a central axle connecting them, allowing it to rotate along a fixed horizontal axis. A long chain containing several separate interlocked colorful rigid rings is connected to the roller, with the top link attached to the roller and the bottom link attached to a rigid anchor. Gravity is enabled. Drive the roller actuators consistently so the chain starts winding around the roller, lifting the anchor up. After fully rolled, the roller continues to roll, throwing the chain and the anchor out until the chain gets stretched again.” Rows show Text2Sim, GPT-5.6-Sol, Qwen-3.8-Max, Claude-Opus-5, and Code2Worlds, from top to bottom. Columns show shared video timestamps of 0, 2, 6, 8, and 10 seconds, from left to right. Red boxes highlight regions discussed in Sec. 5.3 . See the accompanying video for the complete motion sequences. Five methods in a gapless grid, with columns synchronized at 0, 2, 6, 8, and 10 seconds. Text2Sim shows floor contact, lift, compact winding, and subsequent extension. GPT's links disperse. Qwen and Claude-Opus-5 show progressive anchor lifting. Code2Worlds shows winding near the drum followed by a chain extended downward. Red boxes mark dispersed GPT rings, poorly distinguished Qwen links, Claude's hanging anchor, and the Code2Worlds ring–drum contact region at 8 seconds.
Figure 6. Deformable dolphin task. Full prompt: “Create a scene inspired by the dolphin-through-tube demo: a blue soft dolphin starts in front of a rigid funnel that narrows into a long thin horizontal tube. The diameter of the tube should be much smaller than the size of the dolphin. A strong force drags the front part of the dolphin through the funnel and tube, so the body elongates and squeezes through the narrow tube.” Rows show Text2Sim, GPT-5.6-Sol, Qwen-3.8-Max, Claude-Opus-5, and Code2Worlds, from top to bottom. Columns show shared video timestamps of 0, 2, 3, 4, and 9 seconds, from left to right. Red boxes highlight regions discussed in Sec. 5.3 . See the accompanying video for the complete motion sequences. Five methods at five shared video times. Text2Sim is shown from the side with its original blue dolphin, transparent blue funnel and tube, and blue-gray background. The sequence shows the dolphin's elongation and passage, followed by a final view tracking its recovered compact shape after exit. The other rows retain their full original frames, showing different body shapes, funnel views, and degrees of passage at the same times. Red boxes mark GPT's body remaining in the tube, Qwen's body at the inlet, Claude's pale tube boundary, and Code2Worlds' elongated tail-to-body transition at 9 seconds.
Figure 7. Cloth slow-pull task. Full prompt: “Create a scene inspired by a table-cloth slow-pull trick demo: one square tablecloth starts spread over a fixed rigid black table with one of the exposed edges attached to a moving clamp. Several simple rigid objects, including a white circular plate with a piece of pork on it, a knife and fork beside the plate separately, a plate with a piece of butter on it, a tall green wine bottle, and a gold wine cup, are placed on the tablecloth. Gravity is enabled. Move the clamp to pull the tablecloth so the cloth slides across the table and eventually falls away.” Rows show Text2Sim, Qwen-3.8-Max, Claude-Opus-5, and Code2Worlds, from top to bottom. Columns show shared video timestamps of 0, 5, 7, 8, and 9 seconds, from left to right. GPT-5.6-Sol failed to execute this case and produced no completed video, so its row is omitted. Red boxes highlight regions discussed in Sec. 5.3 . See the accompanying video for the complete motion sequences. Four method rows for the tablecloth task: Text2Sim, Qwen-3.8-Max, Claude-Opus-5, and Code2Worlds. GPT-5.6-Sol failed on this case and is omitted. The available sequences show the cloth moving across the table, hanging over its edge, and reaching later configurations with different tabletop-object positions. A red box marks Code2Worlds' bottle–cloth contact region at 8 seconds.
Figure 8. Debug Card results and changes. Paired points show Overall scores without cards (open) and with cards (filled), for (a) physical and (b) visual quality on 42 cases. Endpoint labels report the means; whiskers show within-case RMS scoring SD. Connecting segments and signed labels emphasize With minus Without. The panels use different score ranges. Paired absolute scores and signed changes for four pipelines. Text2Sim improves from 91.34 to 95.32 physically and from 63.50 to 70.23 visually. Qwen's visual score changes from 56.31 to 56.11; the other displayed changes are positive.
| Method | Tokens (M) | Time (min) | Time excl. GPU simulation (min) |
|---|---|---|---|
| Text2Sim | 29.1 | 101.4 | 58.0 |
| GPT-5.6-Sol | 24.8 | 60.9 | 41.9 |
| Qwen-3.8-Max | 23.9 | 95.1 | 55.8 |
| Claude-Opus-5 | 28.7 | 58.7 | 40.6 |
| Code2Worlds | 27.26 | 113.0 | 59.8 |
Table 5. Resource consumption statistics. Token counts are in millions; times are in minutes.
Figure 9. Comparison with GPT-6-Astra. Mean Overall scores with error bars over 42 cases for Text2Sim (blue), GPT-5.6-Sol (gray), and GPT-6-Astra (orange). Two side-by-side panels each show three bars. The physical panel uses a truncated vertical axis from 80 to 100; the visual panel uses 55 to 80. Diagonal marks indicate truncation. Text2Sim scores 95.32 and 70.23, GPT-5.6-Sol scores 86.59 and 59.89, and GPT-6-Astra scores 89.46 and 60.64, respectively.
Figure 10. Rigid tumbler comparison with GPT-6-Astra. Full prompt: “Create a scene inspired by a rigid-body tumbler demo: hundreds of small gray rigid cubes start inside a hollow cylindrical drum with rigid circular side walls and a glass front wall. Gravity is enabled, and the drum is actuated with a slow continuous rotation. Friction lifts the bodies up, and then they avalanche down.” Rows show Text2Sim and GPT-6-Astra, from top to bottom. Columns show shared video timestamps of 0, 2.4, 2.8, 10, and 10.32 seconds, from left to right. The paired early and late views show successive airborne and returning grains. Two rows of five frames compare Text2Sim and GPT-6-Astra at shared video times. Text2Sim shows gray rigid blocks lifted along the drum wall and descending toward the bulk pile. GPT-6-Astra shows groups of blocks separated above the pile at 2.4 and 10 seconds and returning toward it at 2.8 and 10.32 seconds. The baseline retains its full original view; Text2Sim uses one fixed crop preserving the complete drum and its supports.
Figure 11. Sketch-conditioned soft letters. The handwritten “SIGGRAPH” input sketch is shown above four rendered frames of the generated deformable letters. Frames are arranged left to right, then top to bottom, at video times of 0, 0.48, 0.80, and 1.20 seconds. Full prompt: “Create a simulation of colorful soft letters (5 rows) falling, scattering, and settling into a loose pile.” A handwritten SIGGRAPH sketch above a two-by-two grid of full video frames at 0, 0.48, 0.8, and 1.2 seconds, read left to right and top to bottom. Initially, five colored rows of letters stand above a floor; later frames show lower rows bending against the floor, upper rows descending, and letters scattering.
Figure 12. Asset-conditioned curtains. From left to right: the supplied cloth panel with its baked ACM SIGGRAPH texture, followed by four full frames of the generated scene at 0, 3, 6, and 15 s. Full prompt: “Create a window curtain opening demo: two rectangular curtains bearing the supplied ACM SIGGRAPH graphic start hanging on two fixed horizontal rigid rods in front of a window, with their top edges attached to many small hooks along the rod. Gravity is enabled. Drive the slider hooks so the curtains first gather into dense folds at the sides and reveal the window, then slide back toward the center to stretch them again.” Five panels in one horizontal row. The leftmost panel shows a single white rectangular cloth asset with the ACM SIGGRAPH logo. The following four panels show two curtains initially closed at 0 seconds, opening at 3 seconds, gathered at the sides at 6 seconds, and closed again at 15 seconds. Both curtains carry the supplied logo, and all output panels retain the full camera view.