RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins

Dicong Qiu; Zhiyuan Xu; Yaosheng Liu; Feng Han; Bo Ye

cs.CVOct 6, 2026

RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins

Authors: Dicong Qiu, Zhiyuan Xu, Yaosheng Liu, Feng Han, Bo Ye

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Southeast University · Independent Researcher

Abstract

Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion. Evaluating this render-to-real capability requires a real target video depicting the same scene evolution, paired with an editable, geometrically registered 3D replica. Such data has traditionally required substantial manual modeling, calibration, and animation effort. We introduce RenderBench, a benchmark of 12 reconstructed real-world scenes spanning large-scale indoor environments and egocentric viewpoints, with both static and dynamic settings. Our construction pipeline combines visual geometry, neural reconstruction, and assisted 3D authoring. Each scene is decomposed into static objects and dynamic actors, registered to the capture cameras, and accepted only after multi-view geometric and temporal validation. Each evaluation unit contains appearance reference images, a held-out real target video, an editable digital twin, a matched proxy render, and renderer-native scene annotations. We evaluate transfer models against paired real target videos, retain PAI-Bench-C-compatible structural projections, and use scene annotations to localize failures by object, visibility, articulation, and motion. The first release retains 12 of 14 registered samples (85.7%), comprising 1,496 paired real-proxy frames. All released scenes pass file-integrity and environment-edit audits, while proxy diagnostics yield a depth si-RMSE of 0.2170 and instance mIoU of 0.3673. RenderBench provides paired real observations and editable scene state for assessing both appearance fidelity and preservation of geometry and dynamics.

Figures & tables

Figure 2 : Construction and evaluation pipeline of RenderBench. Multi-view observations are converted into a validated digital twin with editable objects, articulated actors, and aligned cameras. Transfer models receive appearance references and an aligned proxy video, while the matched real target remains evaluator-only. Generated outputs are compared directly with the real target using paired-reference metrics, while PAI-Bench-C-compatible structural projections and paired scene state support diagnostic analysis.

Category	Audit item	Result
Dataset scale	Registered samples / released scenes	14 / 12
	Sample retention rate	85.7%
	Paired real–proxy frames	1,496
	Video frame rate	30 fps
File verification	Real and proxy frame counts match	12 / 12 scenes
	File SHA-256 matches the release manifest	48 / 48 files

Table 1 : Dataset release audit and proxy-quality diagnostics. Bracketed values denote the reported scene-level intervals.

Figure 3 : Dataset diversity and paired supervision in RenderBench. The 12reconstructed scenes span large-scale indoor environments and egocentric viewpoints, with both dynamic and static settings. Each real capture is paired with an editable digital twin, aligned proxy renders, and renderer-native scene annotations.

Model	LPIPS-GT $\downarrow$	MS-SSIM-GT $\uparrow$	Lab $\Delta E_{2000}$ $\downarrow$	Edge F1 $\uparrow$	Depth si-RMSE $\downarrow$	Inst. mIoU $\uparrow$	Track err. $\downarrow$
Proxy render	0.2683	0.7607	10.8193	0.1987	0.2170	0.3673	0.0089
Cosmos-Transfer2.5	0.2728	0.7868	9.4432	0.1529	0.2532	0.3674	0.0077
Wan2.2-VACE-Fun-A14B	0.1897	0.8612	5.5227	0.2482	0.1566	0.4477	0.0090
MiniMax H3	0.1229	0.8996	4.6114	0.4556	0.1142	0.6202	0.0068
Seedance 2.5	0.1188	0.9049	2.7857	0.4810	0.1334	0.6142	0.0073

Table 2 : Paired-reference results against the aligned real target videos. Seedance 2.5 is evaluated using its

(0,0)

-aligned output.

Figure 4 : Qualitative comparison on representative paired scenes. The proxy input specifies geometry and motion, while the held-out real target provides the ground-truth video for evaluation. Zoomed regions highlight structural, articulation, and appearance failures that may be obscured by aggregate scores.

Explore similar work

Jun 23, 2026cs.CV

GeoT2V-Bench: Benchmarking 3D Consistency in Text-to-Video Models via 3D Reconstruction

Camera-prompted text-to-video (T2V) models are increasingly used to synthesize virtual camera captures, such as orbiting objects or moving through static scenes. For these outputs, visual plausibility is insufficient: the generated frames should also provide coherent multi-view evidence for a single static 3D scene. We introduce GeoT2V-Bench, a reconstruction-based diagnostic benchmark for evaluating whether camera-prompted T2V clips can support explicit rigid 3D reconstruction. Our pipeline estimates per-frame camera intrinsics and poses with VGGT-style geometry estimation, fits DeformableGS, derives a static MedianGS proxy by temporal-median aggregation, and renders this proxy along the estimated camera path. Instead of producing a pass/fail label or a single scalar score, GeoT2V-Bench reports a continuous reconstruction profile covering apparent image motion, estimated trajectory behavior, MedianGS static rendering error, static-render flow agreement, and the gap between flexible and static fits. On a fair-format four-seed evaluation with 3,840 completed reconstructions from 12 open-weight model configurations and 80 GeCo-Eval static-scene prompts, we find that visible motion, static rendering error, flow agreement, and flexible-vs-static behavior often disagree. GeoT2V-Bench therefore captures complementary failure modes that emerge when generated videos are tested as global static-scene acquisitions.

Chenrui Fan, Paolo Favaro

Computer Vision Group, Institute of Computer Science, University of Bern, Switzerland

Text-To-Video Generation Model Robust Geometric Model Estimation 3D Reconstruction Geogs-Slam Extrinsic Calibration

Ask PDF

Oct 1, 2026cs.CV

PROWBench: Do Video Models Render What the Program Specifies?

Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However, their visual adherence to explicit rules and interactions remains insufficiently evaluated. Existing benchmarks assess visual quality, controllability, and instruction or physical adherence, but rarely test fidelity to fine-grained, program-specified world events. We introduce PROWBench, comprising 170 programmatically constructed episodes and 600 proxy videos covering diverse scenes and interactions. PROWBench logs entity states and timestamped events, including those outside the camera's field of view, as replayable world records, from which it renders synchronized views and proxy representations. This enables generated videos to be checked against the observable consequences of program execution. An extensible framework constructs scenes, controls behaviors, and can render each camera view in different representations, such as coarse 3D, and bounding boxes. The benchmark covers first- and third-person perspectives, with synchronized multi-view observations available for a subset of episodes. Grounded in these records, PROWBench evaluates entity control, long-horizon memory, and, with two VLM-based metrics, Logic-Render Alignment and Interaction Success Rate, adherence to the prescribed timeline and the visual realization of timestamped engine-recorded events.

Zheng-Hui Huang, Guixu Lin, Yu-Ju Tsai +6

Alaya Lab

Video World Models Long-Video Benchmarks World Models

Ask PDF

May 14, 2026cs.CV

Quantitative Video World Model Evaluation for Geometric-Consistency

Generative video models are increasingly studied as implicit world models, yet evaluating whether they produce physically plausible 3D structure and motion remains challenging. Most existing video evaluation pipelines rely heavily on human judgment or learned graders, which can be subjective and weakly diagnostic for geometric failures. We introduce PDI-Bench (Perspective Distortion Index), a quantitative framework for auditing geometric coherence in generated videos. Given a generated clip, we obtain object-centric observations via segmentation and point tracking (e.g., SAM 2, MegaSaM, and CoTracker3), lift them to 3D world-space coordinates via monocular reconstruction, and compute a set of projective-geometry residuals capturing three failure dimensions: scale-depth alignment, 3D motion consistency, and 3D structural rigidity. To support systematic evaluation, we build PDI-Dataset, covering diverse scenarios designed to stress these geometric constraints. Across state-of-the-art video generators, PDI reveals consistent geometry-specific failure modes that are not captured by common perceptual metrics, and provides a diagnostic signal for progress toward physically grounded video generation and physical world model. Our code and dataset can be found at https://pdi-bench.github.io/.

Jiaxin Wu, Yihao Pi, Yinling Zhang +2

1Tsinghua University - IEI Lab · 2UW-Madison · 3Adobe Research

Generative Video Models Video World Models 3D Motion Distortion

Ask PDF