cs.AIOct 12, 2026
SavePhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction
Organizations: Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute · Southeast University ·
Abstract
Building physical simulations from video requires coding agents to translate observations into executable models and refine them through tool use. We introduce PhysCodeGym, a benchmark for physically verifiable 4D reconstruction with tasks covering rigid-body, liquid, deformable, and wind-driven motion. Agents autonomously inspect videos, write and run Blender programs, and revise their scenes using simulation feedback. Six metrics combine rule-based measurements with rubric-based judgments to assess visual and physical reconstruction quality. Program inspection, scene measurements, and counterfactual interventions test physical consistency under original and modified conditions. Experiments with frontier coding agents reveal a tradeoff between artifact quality and completion across reasoning-effort settings, while harness comparisons and agent traces link verification behavior with reconstruction performance. Case studies show that matching visible objects and events can coexist with physical defects. Counterfactual tests further expose latent contact and support failures in reconstructions that pass ordinary physical checks, highlighting the gap between reproducing observations and building reliable physical models. Code is available at https://github.com/ShaoShuai0605/PhysCodeGym.
Figures & tables
| Benchmark | Task | Input | Visual Eval. | Behavioral Fidelity | Physical Implementation | Harness | ||
| Event Eval. | Motion Alignment | Internal Physics Checks | Mechanism Dependence | |||||
| Code4D [ 66 ] | Gen. | Text | ✓ | ✗ | ✗ | ✗ | ✗ | Multi-agent code loop |
| 4DBuildBench [ 32 ] | Gen. | Text | ✓ | ✓ | ✗ | ✓ | ✗ | Multi-agent MCP |
| PhysCodeBench [ 56 ] | Gen. | Text | ✓ | ✓ | ✗ | ✓ | ✗ | Direct codegen. / SMRF |
| SceneActBench [ 68 ] | Recon. | Syn. frames | ✓ | ✗ | ✓ | ✗ | ✗ | Shared MCP |
| VisPhyBench [ 31 ] | Recon. | Syn. frames | ✓ | ✗ | ✓ | ✓ | ✗ | Codegen. + repair |
Table 1: Comparison of programmatic scene generation and reconstruction benchmarks. Task distinguishes generation (Gen.) from reconstruction (Recon.); Input specifies the conditioning modality and source (Syn.: synthetic). Visual Eval. assesses rendered appearance or spatial layout. Under Behavioral Fidelity , Event Eval. checks target actions or outcomes, and Motion Alignment compares motion against a reference. Under Physical Implementation , Internal Physics Checks inspect scene structures, physical parameters, or simulation states, and Mechanism Dependence uses matched physical interventions to test whether the modeled mechanisms drive the observed behavior. Harness lists reported execution workflows or interfaces. ✓ included, ✓ limited scope, ✗ not evaluated under the column definition.
Figure 2: Reconstruction and verification pipeline of PhysCodeGym. Stage 1: A reference video defines the reconstruction task and supplies evidence for evaluation. Stage 2: A coding agent iterates between observing the video, writing a Blender scene program, running it, and comparing rendered previews, then submits the scene program. Stage 3: Independent evaluation combines rubric-based object and event judgments with image and motion metrics, while native-scene inspection and controlled re-simulation assess physical validity. In the illustrated contact test, disabling ball contacts preserves incoming motion but removes the target response; disabling the launch field removes the incoming motion as well. Visual and physical scores are reported separately.
| Harness | Model | Visual | Physical | |||||||
| Codex | GPT-6-Astra | 0.610 | 0.561 | 0.909 | 0.693 | 0.805 | 0.324 | 0.768 | 0.633 | 0.663 |
| GPT-5.6-Sol | 0.550 | 0.448 | 0.835 | 0.611 | 0.687 | 0.210 | 0.690 | 0.529 | 0.570 | |
| GPT-6-Luna | 0.518 | 0.363 | 0.633 | 0.505 | 0.393 | 0.128 | 0.555 | 0.359 | 0.432 | |
| GPT-5.6-Luna | 0.506 | 0.355 | 0.639 | 0.500 | 0.448 | 0.157 | 0.564 | 0.391 | 0.446 | |
| Claude Code | DeepSeek-V4.1-Flash | 0.514 | 0.398 | 0.615 | 0.509 | 0.446 | 0.200 | 0.522 | 0.391 | 0.450 |
Table 2: Visual and physical reconstruction scores on PhysCodeGym (higher is better). Metric definitions and aggregation follow Sec. 4.3 . Best and second-best scores are bold and underlined .
Figure 3: Overall score versus (a) estimated API cost and (b) recorded token use per task. Greater cost or token use does not consistently yield better reconstruction across models.
Figure 4: Comparison of reasoning effort levels. We evaluate DeepSeek-V4.1-Flash on (a) visual metrics, (b) physical metrics, and (c) aggregate scores. Under the same budget of hours and Blender launches, low effort achieves the highest overall score, largely because it completes more tasks than max effort.
Figure 5: Claude Code and Codex harness comparison. Visual, physical, and aggregate scores of DeepSeek-V4.1-Flash at high reasoning effort. In this comparison, Codex achieves a higher overall score, with the largest gain in event reconstruction despite slightly lower perceptual similarity and motion overlap.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Subcategory | Tasks | Event items |
| Rigid body | Falling and bouncing | 16 | 71 |
| Rolling and tracks | 18 | 68 | |
| Collision and chain reactions | 14 | 56 | |
| Swinging and suspension | 16 | 51 | |
| Rotation | 20 | 63 | |
| Granular and multi-object piling | 7 | 22 |
Table 3: Composition of PhysCodeGym: number of tasks and event items per main category and subcategory.
| Source | Rigid body | Liquid | Deformable | Wind-driven | Total |
| Pexels | 69 | 35 | 9 | 9 | 122 |
| Physics-IQ | 19 | 5 | 3 | 0 | 27 |
| BAAI-DataCube Physics-aware Videos | 5 | 0 | 0 | 0 | 5 |
| IRIS | 2 | 0 | 0 | 0 | 2 |
| Total | 95 | 40 | 12 | 9 | 156 |
Table 4: Data sources of PhysCodeGym. Each retained reference clip contributes one task.
Figure 6: Rigid-body reference examples (1/2): falling and bouncing, rolling and tracks, collision and chain reactions, and granular or multi-object piling. Each row contains four uniformly sampled frames from one reference video.
Figure 7: Rigid-body reference examples (2/2): swinging and suspension, rotation, and stacking or static equilibrium. Each row contains four uniformly sampled frames from one reference video.
Figure 8: Liquid reference examples: pouring and container filling, jets and surface flow, and liquid–object interaction. Each row contains four uniformly sampled frames from one reference video.
Figure 9: Deformable-category and wind-driven reference examples. Rows (a)–(d) show draping, membrane inflation, pillow deformation, and hammock motion; rows (e)–(h) show flag and plant motion in wind. Each row contains four uniformly sampled frames from one reference video.
| Model | Harness | ||||||
| GPT-6-Astra | Codex | 0.694 | 0.806 | 0.325 | 0.768 | 0.633 | 0.664 |
| GPT-5.6-Sol | Codex | 0.611 | 0.687 | 0.210 | 0.690 | 0.529 | 0.570 |
| GPT-6-Luna | Codex | 0.507 | 0.392 | 0.129 | 0.555 | 0.359 | 0.433 |
| GPT-5.6-Luna | Codex | 0.503 | 0.450 | 0.157 | 0.564 | 0.391 | 0.447 |
| DeepSeek-V4.1-Flash | Claude Code | 0.511 | 0.452 | 0.200 | 0.522 | 0.391 | 0.451 |
| Claude Sonnet 5 | Claude Code | 0.436 | 0.370 | 0.111 | 0.559 | 0.347 | 0.392 |
Table 5: Scores recomputed on each configuration’s physical-evaluation task subset.
| Harness | Model | Visual | Physical | |||||||
| Codex | GPT-6-Astra | 0.610 | 0.561 | 0.909 | 0.693 | 0.805 | 0.324 | 0.768 | 0.633 | 0.663 |
| GPT-5.6-Sol | 0.553 | 0.451 | 0.840 | 0.615 | 0.692 | 0.211 | 0.694 | 0.532 | 0.574 | |
| GPT-6-Luna | 0.554 | 0.387 | 0.677 | 0.539 | 0.420 | 0.136 | 0.593 | 0.384 | 0.462 | |
| GPT-5.6-Luna | 0.529 | 0.371 | 0.669 | 0.523 | 0.469 | 0.165 | 0.591 | 0.409 | 0.466 | |
| Claude Code | DeepSeek-V4.1-Flash | 0.599 | 0.463 | 0.716 | 0.593 | 0.520 | 0.233 | 0.609 | 0.456 | 0.525 |
Table 6: Mean scores on tasks with rendered outputs for the configurations in Tab. 2 . Higher is better. Best and second-best scores are bold and underlined .
| Averaging set | Low | High | Max |
| Benchmark aggregate | 0.476 | 0.450 | 0.460 |
| Each setting’s produced artifacts | 0.527 | 0.525 | 0.547 |
| Shared tasks with outputs and physical judgments | 0.533 | 0.535 | 0.564 |
Table 7: Overall scores under different averaging sets. Best and second-best scores are bold and underlined .
Figure 10: DeepSeek-V4.1-Flash scores on tasks with rendered outputs (axis ranges differ).
| Question | Cases | Main observation |
| What differs when one metric is high? | Two pendulums; rolling quail egg | High motion overlap can coexist with incomplete events or physical violations. |
| What differs when one metric is equal? | Wine pouring; avocado halves | Equal event or validity scores coexist with differences in appearance and event progression. |
| What does scene inspection reveal? | Thai flag | An event score of coexists with cloth self-intersections. |
Table 8: Overview of the five case studies and their diagnostic questions.
Figure 11: Motion overlap and event completion. GPT-5.6-Luna has the highest motion IoU but an event score of ; GPT-6-Astra ranks fourth on motion IoU and scores on events. Luna’s validity report identifies connector-solid overlap outside the shared pivot throughout frames 55–240, yielding validity versus Astra’s . Displayed frames cover early motion; scores cover all seconds.
Figure 12: Motion overlap and scene constraints. DeepSeek-V4.1-Flash exceeds GPT-6-Astra on motion IoU ( versus ), but its validity is capped at , versus Astra’s . The validity report identifies persistent interpenetration between two stationary eggs across all frames; this violation concerns the resting cluster rather than the moving egg. Astra retains the cluster and speckled appearance, with a timing mismatch.
Figure 13: Appearance differences under equal event scores. All six candidates receive an event score of , despite differences in stream color, glass visibility, and accumulated liquid. Claude Sonnet 5 has the highest SSIM ( ) despite its pale rendering, but lower perceptual similarity than Astra ( versus ). Equal event scores thus coexist with appearance differences reflected in the frames and visual metrics.
Figure 14: Equal validity with different reconstruction fidelity. GPT-6-Astra, GPT-5.6-Sol and GPT-6-Luna receive validity , but event scores of , , and , respectively. Luna’s validity report accepts a single composite rigid body in a homogeneous analytical medium: measured motion responds to effective gravity and linear drag. Passing these checks under the permitted simplification does not establish reproduction of the reference events.
Figure 15: Event completion with cloth self-intersections. Qwen3.8-Flash receives an event score of , yet physical-validity checks detect self-intersections between nonadjacent cloth regions in frames, capping validity at . This case shows that reproducing the visible event can coexist with invalid cloth geometry. Displayed frames also show differences in flag extension and cloth phase.
Figure 16: GPT-6-Astra: energy addition in a Newton cradle. After the gripper is clear at frame 85, four fits to the free pre-impact swing estimate initial total energy at approximately mJ; the dashed line is the largest fitted estimate. At frame 236, gravitational potential alone reaches mJ. With fixed supports and no admitted continuing work source, the native trajectory violates the passive energy budget (N3). This witness does not require differentiating impact-frame velocities.
Figure 17: GPT-5.6-Sol: resting eggs embedded in the tabletop. All nine resting eggs intersect the actual evaluated table surface throughout frames (E1). Their deepest below-surface vertices are – scene units beneath the local surface, with persistent material-boundary intersections. The bars describe the resting eggs only, excluding the mover’s isolated initial overlap.
Figure 18: DeepSeek-V4.1-Flash: a gripper passes through a released block. The right jaw’s centre lies inside the blue block on frames 95–100, reaching mm from the nearest block face at frame 99 (K1). The inset is a block-local cross-section through that centre, derived from measured coordinates. The plotted quantity is centre interior depth, not a universal surface-penetration depth; the original-condition score also includes an independent guide-attachment failure.
Figure 19: GLM-5.3-Flash: the crate floor occupies the wooden tabletop. The measured solid intervals overlap by scene units, embedding of the crate-floor volume in the table (B1). The right panel is a section through the actual axis-aligned box materials, with their common material shaded. Both bodies remain unchanged throughout all frames; this is a persistent original-state overlap, not a failure inferred by modifying the scene.
Figure 20: Claude Sonnet 5: rotation supported by an oversized collision proxy. Joined top decoration enlarges the spherical collider beyond the ball material, producing persistent remote support/traction (S2). The main curve is the minimum over sampled vertices, edge midpoints and face centroids, not an exact global surface-distance minimum. The unchanged scene continues to support and rotate the ball while sampled material distances remain positive; the dashed line denotes zero material separation. Only original-state measurements are displayed.
Figure 21: GPT-6-Luna: nonadjacent flag material crosses itself. Exhaustive checks of unchanged Blender-scene frames 54 and 60 find and intersecting nonlocal triangle pairs (F2). Pairs sharing a material vertex or lying within a two-quad material neighbourhood are excluded; a frame-54 witness joins patches separated by material rows. These are geometric pair counts at the two inspected frames, not independent events or a claim about the first onset of crossing. The raw validity score is the resolved value .
Figure 22: GPT-6-Astra: the wooden piece passes through the dish. The views show the shifted release before contact and during rim crossing. The original release avoids the dish. A shifted release starts mm clear of its surface but puts piece vertices inside the actual closed dish material for frames, reaching mm depth. The grey disc in (c) is only the dish footprint; the failure witness in (d) measures intrusion into its three-dimensional material. Removing the dish leaves the candidate’s centre-of-mass trajectory unchanged. An independent contact control using the source mesh, mass, inertia and integration scale arrests the approach, with only mm maximum local contact intrusion. Thus the issue is absent obstacle response, not merely overlap in the image or the top-down projection.
Figure 23: GPT-6-Astra: the cue ball remains suspended beyond the table edge. The views compare the outward-release candidate and finite-table control at frame 120. An outward release moves the entire cue-ball envelope beyond the actual support boundary at m from frame 117. Its centre remains at approximately mm height through frame 240 ( samples), despite the absence of table material below it. An explicit infinite-support control reproduces the candidate trajectory exactly. Restricting support to the finite tabletop reproduces the prefix through frame 116 and then gives m/s 2 . Only frames 115–127 are shown for that control, before it reaches the lower floor; its later floor-contact behavior is outside this witness. The plotted table patch marks the top support boundary, not a scaled cross-section of the entire table.
Figure 24: DeepSeek-V4.1-Flash (max effort): releasing the rotor exposes missing table contact. The views show the assembly just before release and intersecting the table at frame 125. The original bearing supports the moving assembly. The intervention retains the exact trajectory through frame 120, then removes the sole base-to-beam hinge. The body intersects the unchanged table on frames 123–126 and lies entirely below the table bottom at frame 127. The shaded red and blue bands show each branch’s measured lowest-to-highest material extent, not a filled-body cross-section. Giving the same table geometry an operative collision path arrests the body at its surface; table-removal and disabled-contact controls reproduce the candidate’s free fall. Falling after a bearing is removed is expected. Passing through the table is the counterfactual failure.
Figure 25: GPT-5.6-Sol: the rolling piece passes through the knight. The views follow the left-release branch before and during the crossing. This reconstruction of the same rolling task contains a nearby closed knight assembly whose five material meshes have no collision owner. Two initially separated, table-supported releases put the moving piece inside knight material for and frames, with maximum interior vertex depths of and scene units. Depth is the largest measured vertex intrusion in either direction between the two meshes. In (c), the left-release positions and velocities are exactly unchanged when the knight is removed, across all recorded samples; crossing the centre plane is only a path illustration, while (d) measures actual material intrusion. Adding passive contact to the unchanged target geometry produces a response and avoids material intrusion for both releases. Repeated runs and doubled solver substeps retain the failure. Length values use native scene units.
Figure 26: GPT-5.6-Luna: the rolling piece traverses a solid wooden token. The views track the left-release branch as the body enters the selected token. The selected target is a filled cylinder with no hole and no physical contact owner. Two target-reaching releases produce mesh intersections on frames 114–145 and 118–149, respectively: consecutive frames each. The minimum cylinder signed distance over sampled moving-body surface points is approximately scene units in both branches. For the left release, removing the token changes none of the sampled positions. A native contact control matches the prefix through frame 111 and then stops the approach without token penetration. Its collider responds before the actual material surface, with a dense measured separation of scene units; it is a sensitivity control, not an exact surface-contact trajectory. The grey band in (c) is only the target’s projected extent; (d) gives the material-distance witness.
Figure 27: DeepSeek-V4.1-Flash: wooden-piece rolling. Codex tests and revises the contact setup before completing the final render and checking its motion.
Figure 28: DeepSeek-V4.1-Flash: wine pouring. Codex’s checks lead to repeated inlet corrections and a new full render after each correction.
| Evidence | Claude Code | Codex |
| Benchmark task | Explicitly requires verification and finalization. | Same task body; an additional leading reminder repeats the final-validation budget requirement. |
| Native prompt | Historical full prompt unavailable. Version-matched local captures emphasize faithful reporting of tests and skipped steps. | Historical session metadata records an explicit proactive-testing instruction under never . |
| Interpretation | Earlier checks are present, but the final revisions in these two cases remain unexecuted. | The final revisions in these two cases are run and inspected before the attempt ends. |
Table 9: Prompt evidence for the selected DeepSeek-V4.1-Flash configurations.
| Task | Claude Code | Codex |
| Wooden piece | Revises the final program after the cap is reached; static checks pass, but execution is blocked and output checks fail (166–205). | Detects stalling, revises contact settings, executes the final program at launch 20, then checks frames and finalizes (132–169). |
| Wine pouring | After a bake abort, changes numerical settings without a successful subsequent execution; finalizer reports missing frames (224–248). | Detects excessive accumulation, adjusts the inlet twice, reruns after each edit, inspects final frames, and checks a saved-scene preview (80–94). |
Table 10: Final-revision verification in the two selected tasks. Step numbers refer to each case’s own agent trace.