TelePhysica: Agentic Reconstruction of Physically Interactive Scenes
Organizations: Institute of Artificial Intelligence, China Telecom (TeleAI)
Abstract
Direct image-to-video diffusion models entangle pixel synthesis with implicit latent motion, precluding persistent 3D mechanical equilibrium, mid-rollout intervention, and counterfactual force control. Conversely, single-pass programmatic 3D reconstruction from an uncalibrated photograph leaves monocular depth and support clearance underconstrained, causing reconstructed scenes to interpenetrate or collapse under gravity. We present TelePhysica, a closed-loop framework that reconstructs an executable, multi-regime physical world from a single photograph to drive stateful 3D interaction and trajectory-grounded video synthesis. A vision-language coding agent synthesizes separable visual meshes and declarative physical models within a two-stage verification loop that pairs pixel-aligned silhouette critique with a zero-force gravitational equilibrium preflight. The verified world compiles into a unified solver coupling rigid bodies, articulated chains, co-dimensional cloth shells, and 3D tetrahedral elastodynamics, whose simulated rollouts condition pretrained video diffusion models through pixel-aligned structural streams. On a 100-scene single-image benchmark, closed-loop execution raises equilibrium-verified world recovery from in single-pass synthesis to . Controlled constitutive ablations show over an order-of-magnitude larger localized corner displacement and out-of-plane pleating in finite-element cloth than in matched rigid-plate proxies. Conditioning video diffusion on simulated trajectories increases dynamic motion-mask overlap by and preserves motion direction under counterfactual force reversals where direct image-to-video models fail.
Figures & tables
| Benchmark & Verification Metric | Open-Loop (Pass 1) | Closed-Loop | Dominant Failure Modes Diagnosed & Repaired |
| 100-Scene Single-Image Tabletop Benchmark ( ) | |||
| Stage-1 visual alignment pass | ( ) | ( ) | Pass-1 rejections: mesh topology or instance-binding errors, script/render exceptions, silhouette mismatches. |
| Stage-1 + Stage-2 equilibrium verified | ( ) | ( ) | First physics preflight ( visually passing candidates): collider overlap or role errors, pose/inertia metadata errors, zero-force drift ( pass on first check). |
| Full surface-force rollout verified | ( ) | ( ) | Verified packages converge in median scene submissions (mean ) and median tool calls (mean ). |
| Single-Image Deformable & Hybrid Finite-Element Reconstructions ( : Bath Towel, Seat Cushion & Chair, Kitchen Sponges & Bottle) | |||
| Stage-1 visual + contact-clearance pass | ( ) | ( ) | Iterative camera/pose alignment reaches silhouette IoU with zero initial mesh interpenetration across all three scenes. |
| Scene & Regime | Discretization | Zero CoM Drift | Perturbation Protocol, Counterfactual / Ablation Contrast & Sensitivity |
|---|---|---|---|
| Ten-pin bowling ( Rigid cascade ) | rigid bodies ( ) | horizontal launch topples pins; live mid-rollout continuation alters pin cascade to pins ( divergence; pre-impact diff). | |
| Dual-pen holder ( Container clearance ) | rigid bodies ( gap) | Axial pulses ( , ) lift target pen out of cup while unactuated neighbor shifts ( airborne diff). | |
| Wired earphones ( Articulated chain ) | capsule links + terminal bodies | Prescribed left-bud lift raises splitter and controller while frictional slack holds right bud to ( RMS diff). | |
| Bath towel ( Rigid-plate ablation ) | rigid plate ( , ) | Identical corner tug ( , ) yields X, Y flat slab shift ( norm, fold; RMS). | |
| Bath towel ( Thin-shell cloth ) | nodes, tris ( , ) | Same tug pulls corner X, Y ( peak 3D) with pleats; far corner moves ( RMS). | |
| Knitted wool gloves ( Layered shells ) | nodes, tris ( each) | Oblique cuff traction ( , ) peels upper glove peak while lower moves ; shifts ( RMS). |
| Subset | Generator & Control | Mask IoU | Traj (%) | PSNR | SSIM | DINOv2 (Full/ROI) | VRAM | Time (s) | |
| Rigid ( ) | Direct I2V 1.3B (No Phys) | 0.1906 | 12.58 | 25.71 | 0.9351 | 0.8021 / 0.7862 | 19.4 GB | 74.5 | |
| Direct I2V A14B (No Phys) | 0.2301 | 10.40 | 22.49 | 0.9520 | 0.7643 / 0.7210 | 41.2 GB | 354.4 | ||
| Wan2.1-VACE 1.3B (Canny) | 0.4556 | 3.84 | 10.21 | 0.6918 | 0.5912 / 0.6174 | 19.4 GB | 55.9 | ||
| Wan2.1-VACE 1.3B (RGB) | 0.8424 | 1.00 | 16.87 | 0.8676 | 0.6502 / 0.5696 | 19.4 GB | 55.1 | ||
| Wan2.2-VACE A14B (Canny) | 0.6936 | 2.31 | 22.29 | 0.9286 | 0.6416 / 0.6526 | 41.0 GB | 291.4 | ||
| Wan2.2-VACE A14B (RGB) | 0.6352 | 3.42 | 23.46 | 0.9306 | 0.6898 / 0.6540 | 41.0 GB | 257.7 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Verification Stage | Evaluated Quantity | Quantitative Acceptance Criterion |
|---|---|---|
| Role Inventory | Mobility role | Every scene object declares an explicit role; movable foreground bodies and deformables are assigned dynamic . |
| Mesh Topology | Manifoldness & area | Zero non-manifold edges ( faces per edge) and zero degenerate triangles below scale-normalized area . |
| Stage-1 Visual Gate | Silhouette & edge IoU | Combined occupancy, 72-bin radial contour profile, and Sobel interior edge score with foreground mask IoU and silhouette structure . |
| Stage-2 Equilibrium | Initial contact & settle | Pairwise initial overlap ( : smallest declared primitive collider thickness of dynamic object in pair), drift , terminal speed ( rigid; deformable), rigid rotation , angular speed , and solver warnings. |
| Perturbation Rollout | Multi-trial execution | Schema-validated interventions execute across forward, negated counterfactual ( ), replay, and trials with finite integration state, solver warnings, and logged force-time integrals and RMS sensitivity. |
| State Determinism | Checkpoint & continuation | Full-state checkpoint replay, midpoint pause/restore continuation, and chunked stepping reproduce complete integration state vectors within maximum absolute error. |
| Scene (Regime) | Model & Conditioning | Mask IoU | Traj Err (%) | PSNR (dB) | SSIM | DINOv2 (Full / ROI) | Time (s) | |
| Yellow Ceramic Mug ( Rigid tumble ) | Direct I2V 1.3B | 0.1004 | 13.57 | 30.49 | 0.9660 | 0.9525 / 0.9292 | 69.0 | |
| Direct I2V A14B | 0.2070 | 8.99 | 24.26 | 0.9613 | 0.8562 / 0.8319 | 376.1 | ||
| VACE 1.3B (Canny) | 0.4968 | 3.63 | 11.14 | 0.7200 | 0.8253 / 0.7989 | 55.8 | ||
| VACE 1.3B (RGB) | 0.7125 | 2.07 | 12.04 | 0.8397 | 0.7480 / 0.6426 | 56.0 | ||
| VACE A14B (Canny) | 0.6194 | 3.64 | 18.46 | 0.8747 | 0.8738 / 0.7668 | 253.6 | ||
| VACE A14B (RGB) | 0.4334 | 7.64 | 18.90 | 0.8746 | 0.8367 / 0.7964 | 245.4 |
Physical Consistency in Video Generation
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
See all 202 papers