cs.CVOct 20, 2026
SavePair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision
Organizations: Universidad Rey Juan Carlos · New York University · Roblox · Adobe
Abstract
Coarse-to-real (C2R) video generation converts coarse animation, simulation, or game-rendered videos into photorealistic videos while preserving scene layout, camera motion, and dynamics. Training such models requires aligned (coarse, realistic) video pairs, which are expensive to create and confine prior systems to narrow synthetic domains. We exploit a simple asymmetry: paired videos are hard to obtain, but paired real-to-coarse images are easy to construct with existing editing and 3D reconstruction tools. Using these image pairs, we train the DINO-R2C model to transfer realistic images to coarse images in DINO feature space, and apply it frame-wise to ordinary real videos to synthesize pseudo-coarse videos. The C2R# model then learns to generate each real video from its pseudo-coarse counterpart, free from needing (coarse, realistic) video pairs. At inference, the DINO-R2C model is discarded and actual coarse videos drive the C2R# model directly. Thus, our pair-free training scales with any real-video corpora rather than with paired synthetic videos. We will release the model and project webpage at https://gonzalognogales.github.io/coarse2real-sharp/.
Figures & tables
Figure 1: The C2R# model transforms game renders (top), geometry blockouts (middle), and stylized animations (bottom) into photorealistic videos.
Figure 2: Overview of the DINO-R2C and C2R# models. Top: The DINO-R2C model is trained from paired realistic and coarse images using feature reconstruction and pixel-space reconstruction losses. Once trained, it is applied to real videos frame-wise to synthesize pseudo-coarse videos. Bottom left: The C2R# model is trained on these pseudo-paired videos. The pseudo-coarse video is encoded by the frozen DINO encoder and injected into the frozen Wan2.2 backbone through trainable control and fusion adapters, while the original real video provides the training target. Bottom right: at inference, an actual coarse video is encoded directly by and used to condition the same C2R# model; the DINO-R2C model is no longer required. Orange modules are trainable, blue modules are frozen, gray boxes are tensors, and red boxes denote training losses.
Figure 3: Example (coarse, realistic) image pairs. A realistic image (left) is paired with coarse images in four modalities (right): Coarser geometry, Coarse geometry, Coarser Anime, and Coarse Anime.
| DINO-R2C Recipe | Perceptual flicker (relative) | Smoothed-frame flicker (relative) | Motion error (EPE) | Motion magnitude ratio | Boundary stability (Dice) | Content consistency (DINO) |
|---|---|---|---|---|---|---|
| None, no LPIPS | 2.2028 | 4.1565 | 2.2685 | 2.1909 | 0.7195 | 0.9816 |
| None + LPIPS | 2.1793 | 3.9131 | 1.8678 | 1.8304 | 0.7271 | 0.9877 |
| + LPIPS | 0.9314 | 1.7901 | 0.8819 | 1.0844 | 0.8345 | 0.9946 |
| , no LPIPS | 0.7965 | 1.6825 | 0.9066 | 1.1504 | 0.8327 | 0.9914 |
| + LPIPS | 0.9023 | 1.7108 | 0.8695 | 1.0875 | 0.8354 | 0.9946 |
| + LPIPS (ours) | 0.8649 | 1.6294 | 0.8429 | 1.0797 | 0.8376 | 0.9948 |
Table 1: DINO-R2C design ablations, video stability. Evaluated on 1,000 real videos in the Coarse geometry modality at . Columns report means across videos; both flicker measures are per-video medians relative to source-video error. Bold/underlined values are best/second-best. Metrics are explained in Section 4.2 and Appendix B.3 .
Figure 4: Effect of the RGB bottleneck during training. Matched frames show the control input (left), C2R# (center), and a variant conditioned directly on predicted DINO-R2C features, bypassing decoding and re-encoding (right). The latter produces visible artifacts.
Figure 5: Qualitative comparison with baseline methods. Three groups of two rows show game renders (top), geometry blockouts (middle), and stylized animations (bottom). “Rep.” denotes our Editto Sim2Real and VACE 2.2 reproductions, fine-tuned on the same pseudo-paired videos as C2R# (ours).
Figure 6: Domain transfer in CSD-L style-feature space. Left: DINO-R2C maps 1,000 real videos to four distinct pseudo-coarse style clusters. Right: C2R# maps actual coarse videos toward the real-reference distribution. Points denote videos; arrows connect input/output centroids. PCA is fitted per panel (Appendix B.4 ).
Figure 7: Limitations under abrupt control changes. Input (left) and C2R# output (right): abrupt control changes can lead to unwanted content and reduced alignment.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| DINO-R2C Recipe | Latent MSE | RGB MSE | RGB L1 | LPIPS | DINO patch cosine | Depth RMSE | SSIM | Boundary F1 |
|---|---|---|---|---|---|---|---|---|
| No Aug, no LPIPS | 0.4443 | 0.0217 | 0.1138 | 0.3980 | 0.9055 | 0.4537 | 0.6225 | 0.3995 |
| No Aug, w/ LPIPS | 0.4406 | 0.0218 | 0.1137 | 0.3707 | 0.9121 | 0.4447 | 0.6122 | 0.5150 |
| Aug, w/ LPIPS | 0.4296 | 0.0214 | 0.1129 | 0.3700 | 0.9128 | 0.4380 | 0.6236 | 0.4654 |
| Aug, no LPIPS | 0.4276 | 0.0211 | 0.1123 | 0.4127 | 0.9020 | 0.4483 | 0.6359 | 0.2889 |
| Aug, w/ LPIPS | 0.4280 | 0.0212 | 0.1126 | 0.3682 | 0.9127 | 0.4357 | 0.6247 | 0.4644 |
| Aug, w/ LPIPS (ours) | 0.4237 | 0.0212 | 0.1122 | 0.3659 | 0.9136 | 0.4352 | 0.6282 | 0.4573 |
Table 3: DINO-R2C design ablations, image quality. Evaluated on 1,000 held-out images in the Coarse geometry modality, comparing predictions with paired coarse targets. Latent MSE measures feature reconstruction; RGB MSE and L1 measure pixel errors; LPIPS and DINO patch cosine measure perceptual and local content agreement; depth RMSE, SSIM, and boundary F1 assess geometry, local image structure, and outlines. Latent MSE uses the native feature grid; other metrics use images. Values are means across images. Bold and underlining mark the best and second-best values. Definitions are given in Appendix B.3 .
Figure 8: Qualitative DINO-R2C results. Each example shows a realistic image and decoded coarse predictions for Coarser geometry, Coarse geometry, Coarser Anime, and Coarse Anime. The predictions illustrate modality-dependent appearance transfer while retaining scene structure.
| Method / training stage | Total params | Trainable params | Trainable (%) |
|---|---|---|---|
| C2R orig. : Stage I | 14.288B | 14.288B | 100.00 |
| C2R orig. : Stage II | 14.333B | 44.33M | 0.31 |
| Editto Global | 17.460B | 122.68M | 0.70 |
| Editto Sim2Real | 17.460B | 122.68M | 0.70 |
| VACE 2.2 | 34.675B | 6.098B | 17.59 |
| C2R# (ours) | 28.758B | 180.54M | 0.63 |
Table 4: Training parameters. Totals include the diffusion backbone and control/fusion modules, including unmerged LoRA where applicable, but exclude frozen external text and feature encoders, the VAE, and separately trained DINO-R2C models. Counts include both denoising experts for VACE 2.2 and the C2R# . B and M denote billions and millions of parameters.
| Quantity | Value |
|---|---|
| Model params | |
| Shared backbone + four heads | M trainable |
| Training efficiency | |
| Hardware | B200 GPUs |
| Training iterations | 20,000 |
| Global batch size | 192 |
Table 5: Model size and efficiency for a shared-backbone, four-head DINO-R2C architecture. Inference uses BF16 at and includes the projector and frozen RAE decoder, excluding model loading, transfers, video codecs, and storage I/O.
Figure 9: The C2R# model with a reference image. Each example includes a control video, the reference image used at inference, a text prompt, and the generated result. Examples cover game renders, geometry blockouts, and stylized animations. This extension uses separately trained weights and an additional image input, and is evaluated separately from the standard control-video-and-text setting.
Figure 10: First-attempt comparison with Seedance 2.5. Columns show the input control, C2R# output, and Seedance 2.5 output. Rows show examples of game renders, geometry blockouts, and stylized animations, respectively. Both methods receive the same control video and text prompt, and the displayed results are from the first generation attempt. In these examples, the C2R# model produces a more photorealistic appearance, while Seedance 2.5 retains more of the coarse input style or changes the scene layout.