From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video
Organizations: Independent Researcher.
Abstract
Generative world models and 3D Gaussian Splatting promise to turn ordinary video into explorable 3D scenes, but little is reported on how they behave on uncurated, in-the-wild footage. We study what these models recover when the only input is a publicly available news walkthrough of a food bank warehouse, with presenters walking through every shot and no drawings, scans or site visit, and we use the result as the geometry of a digital twin. A commercial world model given 20 frames recovered camera poses and metric depth for three areas, which we fused into about 747,000 Gaussian splats for 200 credits (20 US cents). Splat models optimized locally on a CPU, at no credit cost, were photorealistic near the filmed path but degraded quickly away from it. On held-out frames, scored in pixels not covered by people, they reached 21.2 and 23.1 dB PSNR from 64 and 43 training frames, and lost 3.8 and 5.4 dB when given only the 8 frames a world model accepts. Removing the trainer's background fill and person-hull carving raised held-out PSNR, showing that cleaning the space behind moving people costs fidelity in observed pixels while its gain lies where no ground truth exists. Asking the world model to generate views of areas the video never showed failed on the provider's side twice, so extrapolation could not be evaluated and unseen space was filled by a scripted model instead. We also describe the corrections needed to turn world-model output into a valid, metric Universal Scene Description (OpenUSD) asset with its native splat schema, which serves as the only interface to a data layer, a browser viewer, Blender and NVIDIA Omniverse. The findings suggest that today's models recover the observed parts of a real scene cheaply and at metric scale, while few-view fidelity and extrapolating unobserved space remain open. The work is an exploration of these methods, not a deployed product.
Figures & tables
| Boundary | Artifact | Rules |
|---|---|---|
| 1 to 3 | factory.usdz , factory.spz , factory.meta.json | Metric ( metersPerUnit = 1), Y-up, ground at ; alignment and provenance in the metadata file. |
| 2 to 3 | GET /telemetry (JSON) | zones[] (temperature, setpoint, humidity, door, status, position) and forklifts[] (position, heading, speed, battery); positions in meters in the twin frame; simulated flag. |
| Part of the twin | Kind | Source |
|---|---|---|
| Racking aisle, sorting room, client market (geometry) | Captured, partial | 20 frames of a public walkthrough of a Central Texas Food Bank building; world-model depth fused locally. The same three are also trained locally from 49 to 72 frames each (Table 4 ). |
| Community kitchen interior (geometry) | Captured, partial | Trained locally from 65 frames of the same video; no world-model capture. |
| Absolute scale | Estimated | The world model’s metric depth; not checked against a survey. |
| Building shell, receiving, freezer, cooler, order staging, kitchen annex footprint | Designed | Footprints chosen for this paper. |
| Modeled facility (racking, cold rooms, sorting lines, market, kitchen equipment, forklifts) | Designed | Built in Blender by one script from the designed footprints, with fit-out inspired by the video; a compact 26 20 m building plus an 11.5 11 m annex, much smaller than the facility whose volumes drive the operations model. |
| Annual volume, agencies, floor area, pallet positions, cold storage, docks, kitchen equipment, meals per day | Published | Figures for the Central Texas Food Bank’s Austin facility (Table 7 ). |
| Time (min:s) | Zone | Assessment |
|---|---|---|
| 0:45–1:35 | Volunteer sorting room | Captured from 0:48–1:06, where the camera moves; 6 frames. |
| 2:10–2:50 | Freezer and cooler doors | Mostly blank walls; fogged at 2:40. |
| 5:05–6:00 | Loading dock | People fill the frame. |
| 6:10–7:45 | Pallet-racking aisle | Captured. Racking, wrapped pallets and a pallet jack; the camera circles a bay and looks down the aisle; one view has no people; 8 frames. |
| 9:20–10:00 | Client grocery market | Captured from 9:16–9:50; people block the center and were masked; 6 frames. |
| 13:50–16:40 | Commercial kitchen | Reflective stainless steel, hard for splats. |
| Zone | Video segment | Frames | Splats | SPZ (MB) | USDZ (MB) |
|---|---|---|---|---|---|
| Racking aisle | 6:08, 100 s | 72 | 128,884 | 2.1 | 8.4 |
| Volunteer sorting room | 0:52, 30 s | 68 | 111,400 | 1.8 | 7.3 |
| Client market | 9:13, 45 s | 49 | 105,899 | 1.6 | 6.9 |
| Community kitchen | 14:23, 34 s | 65 | 161,425 | 2.5 | 10.6 |
| Zone | Variant | Train | Test | PSNR (dB) | SSIM | LPIPS | Splats |
|---|---|---|---|---|---|---|---|
| Racking aisle | full trainer | 64 | 9 | 21.16 | 0.799 | 0.094 | 149,063 |
| 8 training frames | 8 | 9 | 17.34 | 0.632 | 0.144 | 142,124 | |
| no fill, no carving | 64 | 9 | 23.64 | 0.893 | 0.041 | 189,057 | |
| Client market | full trainer | 43 | 6 | 23.10 | 0.815 | 0.127 | 121,818 |
| 8 training frames | 8 | 6 | 17.69 | 0.660 | 0.161 | 123,703 | |
| no fill, no carving | 43 | 6 | 23.64 | 0.837 | 0.101 | 163,666 |
| Quantity (unit) | Value | How measured |
|---|---|---|
| Input frames used | 20 (8 + 6 + 6) | hand-picked from 40 per zone |
| Frame extraction, racking window (s) | 88 | wall clock, 4 CPU cores |
| World-model cost, posing (credits; USD) | 200; 0.20 | 10 credits per frame, rate card |
| World-model cost, failed generation (credits) | 0 to 800 | billing of failures unknown |
| Splats, racking / sorting / market | 270k / 282k / 195k | fused files |
| SPZ size, racking / sorting / market (MB) | 2.1 / 2.2 / 1.6 | committed files |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Figure | Source |
|---|---|
| 74 million lb distributed per year, 26.6 million lb of it produce (fiscal year 2024) | [ 10 ] |
| 251 partner agencies in a 21-county service area | [ 10 ] |
| 135,000 sq ft facility | [ 11 ] |
| More than 7,000 pallet positions, more than 35,000 sq ft of cooler and freezer, 15 docks | [ 14 ] |
| 4,200 sq ft production kitchen with kettles, combi ovens, blast chillers and tilt skillets | [ 21 ] |
| About 950 kitchen meals per day for after-school meals (2018); about 2,000 a day by 2025, including summer meal sites | [ 12 , 13 ] |
| Item | One-time | Monthly |
|---|---|---|
| 24 temperature sensors, 21 door sensors, 2 LoRaWAN gateways | $1,850 | |
| 8 forklift beacons, 10 BLE gateways (area-level location) | $840 | |
| AWS: IoT Core, Lambda, API Gateway, DynamoDB, logs (20 viewers) | $12 | |
| Optional: Omniverse on a g6e.xlarge, 10 h a week | $124 | |
| Measured: Omniverse trial, 7 runs, 94 instance-minutes a | $1.34 | |
| Scaled: on-demand rendering on g6.xlarge, 10 h a week | $35 |
| Role | AWS | Azure | Google Cloud |
|---|---|---|---|
| LoRaWAN network server | IoT Core for LoRaWAN | Partner network server | Partner network server |
| MQTT ingest | IoT Core | IoT Hub | Partner or self-hosted broker |
| Routing | IoT Core rules | Event Grid | Pub/Sub |
| Functions | Lambda | Functions | Cloud Run functions |
| Readings store | DynamoDB | Cosmos DB (serverless) | Firestore |
| Object storage | S3 | Blob Storage | Cloud Storage |
3D Reconstruction
3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.
See all 1,009 papers