FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Hugo Caselles-Dupré; Mathis Koroglu; Guillaume Jeanneret; Arnaud Dapogny; Matthieu Cord

cs.CVMar 18, 2026

FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Authors: Hugo Caselles-Dupré, Mathis Koroglu, Guillaume Jeanneret, Arnaud Dapogny, Matthieu Cord

Organizations: Obvious Research, Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Valeo.ai, France

Abstract

Diffusion-based image-to-video (I2V) models are increasingly effective, yet they struggle to scale to ultra-high-resolution inputs (e.g., 4K). Generating videos at the model's native resolution often loses fine-grained structure, whereas high-resolution tiled denoising preserves local detail but breaks global layout consistency. This failure mode is particularly severe in the fresco animation setting: monumental artworks containing many distinct characters, objects, and semantically different sub-scenes that must remain spatially coherent over time. We introduce FrescoDiffusion, a training-free method for coherent large-format I2V generation from a single complex image. The key idea is to augment tiled denoising with a precomputed latent prior: we first generate a low-resolution video at the underlying model resolution and upsample its latent trajectory to obtain a global reference that captures long-range temporal and spatial structure. For 4K generation, we compute per-tile noise predictions and fuse them with this reference at every diffusion timestep by minimizing a single weighted least-squares objective in model-output space. The objective combines a standard tile-merging criterion with our regularization term, yielding a closed-form fusion update that strengthens global coherence while retaining fine detail. We additionally provide a spatial regularization variable that enables region-level control over where motion is allowed. Experiments on the VBench-I2V dataset and our proposed fresco I2V dataset show improved global consistency and fidelity over tiled baselines, while being computationally efficient. Our regularization enables explicit controllability of the trade-off between creativity and consistency.

Figures & tables

Figure 1: From a single ultra-high-definition image ( $3500\times 3500$ ), FrescoDiffusion animates it at the same resolution. We show three frames from the generated video. The red box marks a fixed spatial region tracked across time, illustrating motion temporal consistency and fine-detail preservation.

Figure 2: Overview of FrescoDiffusion. Starting from a 4K fresco image, we first build a global latent prior, by resizing the image to the native input size of the image-to-video backbone. Next, we upsample the prior latents $x_{\text{prior}}$ to fit the 4K image size. We then apply tiled denoising to the large latent canvas, $x_{t}^{\text{4K}}$ , obtaining per-tile flow predictions, $\{y_{i}\}$ . We then use $\{y_{i}\}$ and $x_{\text{prior}}$ to compute the optimal output velocity field ( Eq. 6 ) according to our loss $\ell_{\text{FD}}$ ( Eq. 5 ). This updated field is then used to update the large latent canvas, $x_{t}^{\text{4K}}$ , with the flow-matching scheduler.

Figure 3: Spatial prior schedule

\lambda_{R}

( Eq. 8 ). (Top) MSE between the foreground / background regions and the prior: the background stays bound to the prior longer than the foreground. (Bottom) Schedule for both regions.

Figure 4: Overlay of the spatial activity map onto the input fresco.

Figure 5: A qualitative comparison of fresco-scale inputs. FrescoDiffusion generates coherent global scenes and animates details at a local level. By contrast, DemoFusion, DynamicScaler and MultiDiffusion only manage to produce either coherent scenes or high-quality details, but not both.

	FrescoArchive						VBench-I2V
	FrescoDiffusion			R-FrescoDiffusion			FrescoDiffusion			R-FrescoDiffusion
	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.
vs. DemoFusion	63%	60%	62%	54%	62%	58%	55%	46%	51%	66%	57%	62%
vs. DynamicScaler	78%	80%	79%	88%	80%	84%	57%	64%	61%	68%	64%	66%
vs. MultiDiffusion	78%	74%	76%	78%	74%	76%	61%	66%	64%	73%	64%	69%

Table 1: User study results: human preference rates (% of annotators preferring our method over each baseline) on both FrescoArchive (FA; 371 source images, 1344 ratings) and VBench-I2V (361 source images, 1465 ratings). Each pair is drawn from the same source image, shown at identical resolution and duration in randomized order. Green cells mark a statistically significant preference (

>50\%

at the 95% level); all rates use 95% confidence intervals of at most

\pm 6\%

Method	FrescoArchive	VBench-I2V	Time (min)	Rel. Time
DynamicScaler (original)	0.857	0.862	18.45	2.27 $\times$
DynamicScaler ∗ (CVPR’25)	0.871	0.865	10.25	1.26 $\times$
DemoFusion ∗ (CVPR’24)	0.903	0.879	13.5	1.66 $\times$
MultiDiffusion ∗ (ICML’23)	0.876	0.860	8.15	1.00 $\times$
FrescoDiffusion	0.904	0.875	8.58	1.05 $\times$
R-FrescoDiffusion	0.907	0.878	9.08	1.11 $\times$

Table 2: Standard low-resolution I2V metrics. The table shows the average performance (higher is better) using the VBench evaluation suite on both FrescoArchive and VBench-I2V image sets. We also display the average generation time in minutes.

Method	$\lambda$ function	SC	MS	A	I	ISC	IBC	Avg
MultiDiffusion	$0$	0.876	0.974	0.686	0.754	0.981	0.989	0.876
FrescoDiffusion	$\lambda_{\text{base}}$	0.942	0.991	0.645	0.598	0.979	0.983	0.856
FrescoDiffusion	$\lambda_{\text{base}}\,\cos(t\frac{\pi}{2})$	0.946	0.991	0.724	0.730	0.987	0.992	0.895
FrescoDiffusion	$\lambda_{G}$ ( Eq. 7 )	0.958	0.990	0.738	0.753	0.991	0.995	0.904
R-FrescoDiffusion	$\lambda_{R}$ ( Eq. 8 )	0.977	0.991	0.736	0.753	0.991	0.994	0.907

Table 3: Prior strength schedule ablation study on FrescoArchive. Best in bold . The results suggest that including the schedule, the gating, and the spatial regularization enhances the quantitative performance.

Figure 6: Regional constraint. Prior shows an overlay of the activity map. The red / blue boxes represent background/foreground regions. Our regional loss pulls the background toward the prior while allowing new details in foreground regions.

Panel of Figure 7 — Figure 7: Quantitative evaluation of the trade-off between creativity and prior similarity controlled by $\lambda$ . (a) Temporal consistency versus sharpness, illustrating the Pareto frontier between preserving temporal coherence and maintaining high image sharpness. (b) Evolution of prior similarity over time, showing how increasing prior strength and temporal gating progressively aligns the generated outputs with the prior.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

y\leftarrow\frac{\textit{num}+\lambda\,\sigma_{t}\,(x_{t}^{\text{4K}}-x_{\text{prior}})}{\textit{den}+\lambda\,\sigma_{t}^{2}}

Appendix

Algorithm 1 FrescoDiffusion: one sampler step at time

t

Dataset	Samples	Words	Image Width	Image Height
FrescoArchive	371	355.79 $\pm$ 88.0	2265 $\pm$ 1257	1552 $\pm$ 864
VBench	361	14.70 $\pm$ 2.24	4592 $\pm$ 1305	3748 $\pm$ 1214

Appendix

Table 4: Dataset statistics. We compare FrescoArchive and VBench datasets’ low-level statistics.

Figure 8: Analysis of FrescoArchive dataset. Left: distribution of word in caption. Right: Resolution of images distribution.

Figure 9: PCA between FrescoArchive and VBench I2V dataset. We visualize the DINOv2 [ 27 ] feature PCA between our proposed dataset and VBench’s images (red and green points, respectively). Also, in light green and red, random crops of the images. This plot shows that our dataset contains images with different statistics than standard ones, found in the VBench I2V set.

FrescoArchive
Method	Quality Metrics				I2V Metrics		Avg	Time
Method	SC	MS	A	I	ISC	IBC	Avg	Time
DynamicScaler (original)	0.945	0.975	0.693	0.706	0.893	0.930	0.857	18.45
DynamicScaler ∗ (CVPR’25)	0.852	0.971	0.681	0.754	0.980	0.989	0.871	10.25
DemoFusion ∗ (CVPR’24)	0.960	0.987	0.734	0.752	0.990	0.994	0.903	13.5
MultiDiffusion ∗ (ICML’23)	0.876	0.974	0.686	0.754	0.981	0.989	0.876	8.15

Appendix

Table 5: Quantitative evaluation on FrescoArchive and VBench-I2V. Higher is better for all metrics except time. Best in bold and second best underlined . ∗ denotes methods adapted to the video prior setting. SC, MS, A, I, ISC, and IBC stand for Subject Consistency, Motion Smoothness, Aesthetic, Imaging, I2V Subject Consistency, and I2V Background Consistency, respectively.

Method	FA	VBench-I2V
Wan (native resolution)	0.288	5.837
FlashVSR	0.247	6.421
DemoFusion	1.251	4.638
DynamicScaler	24.085	17.358
MultiDiffusion	26.217	18.469
FrescoDiffusion	1.318	4.424

Appendix

Table 6: Optical flow magnitude (RAFT-Large, median over pixels,

Q_{3}+1.5\,\text{IQR}

outliers removed) at

2560\times 1440

on FrescoArchive (FA) and VBench-I2V.

High-Resolution Video Generation

See all 13 papers

May 17, 2026

Image-to-Video Diffusion: From Foundations to Open Frontiers

Diffusion-based image-to-video (I2V) generation has become a central direction in generative models by turning a reference image, with optional conditions, into a temporally coherent video. Compared with broader video generation settings, this task places stricter demands on content consistency, identity preservation, and motion coherence. Although the literature grows rapidly, existing works mostly discuss I2V generation within broader topics and still lack a dedicated taxonomy together with a systematic analysis centered on this field. This work addresses that gap by treating diffusion I2V generation as a standalone subject. It first reviews the task formulation, model architectures, datasets, and evaluation metrics, and then organizes existing methods through a taxonomy based on architecture and training paradigm. It further distills four core designs, namely condition encoding, temporal modeling, noise prior design, and spatial-temporal upsampling, and discusses representative application scenarios together with major open challenges.

Xianlong Wang, Wenbo Pan, Shijia Zhou +6

Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China +3

Image-to-Video Generation Video Diffusion Models Temporal Consistency in Video Generation

Ask PDF

Aug 21, 2025

CineScale: Tuning-Free High-Resolution Video Generation

Video diffusion models have achieved remarkable progress in recent years, yet generating high-resolution videos remain a fundamental challenge. Most video generators are trained at limited spatial resolutions due to the scarcity of high-resolution 4K video data and the prohibitive computational cost of large-scale training on such data. Most video diffusion models are trained on 720p videos and are therefore effectively limited to generating videos at similar resolutions during inference. To address this gap, we propose CineScale. CineScale, to the best of our knowledge, is the first tuning-free inference framework enabling pretrained video diffusion models to generate high-quality videos at resolutions far beyond those seen during training. Our key observation is that generation quality degrades at higher resolutions because positional encodings shift beyond their training distribution, producing blurred details and structurally incoherent videos. To address this gap, we introduce Adaptively Rectified RoPE. Our extensive experiments show that CineScale enables pretrained diffusion models, despite never being trained on high-resolution data, to generate high-fidelity 4K video without any fine-tuning, improving local detail and sharpness while preserving temporal coherence. This demonstrates that high-resolution generation capabilities can be unlocked purely at inference time.

Gordon Chen, Haonan Qiu, Ning Yu +3

Nanyang Technological University · Netflix Eyeline Studios

Rotary Positional Embeddings Video Diffusion Models High-Resolution Video Generation Video Generation Positional Encoding

Ask PDF

Jun 8, 2026

MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation

Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information, such as scene layout and semantics, while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.

Ishaan Preetam Chandratreya, David Charatan, Basile Van Hoorick +4

Massachusetts Institute of Technology · Toyota Research Institute

Video Diffusion Models Long Video Generation Video Generation Temporal Consistency in Video Generation Geometric Consistency in Video Generation

Ask PDF

Abstract

High-Resolution Video Generation

Explore similar work

Image-to-Video Diffusion: From Foundations to Open Frontiers

CineScale: Tuning-Free High-Resolution Video Generation

MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation