FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Hugo Caselles-Dupré; Mathis Koroglu; Guillaume Jeanneret; Arnaud Dapogny; Matthieu Cord

cs.CVMar 18, 2026

FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Authors: Hugo Caselles-Dupré, Mathis Koroglu, Guillaume Jeanneret, Arnaud Dapogny, Matthieu Cord

Organizations: Obvious Research, Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Valeo.ai, France

Abstract

Diffusion-based image-to-video (I2V) models are increasingly effective, yet they struggle to scale to ultra-high-resolution inputs (e.g., 4K). Generating videos at the model's native resolution often loses fine-grained structure, whereas high-resolution tiled denoising preserves local detail but breaks global layout consistency. This failure mode is particularly severe in the fresco animation setting: monumental artworks containing many distinct characters, objects, and semantically different sub-scenes that must remain spatially coherent over time. We introduce FrescoDiffusion, a training-free method for coherent large-format I2V generation from a single complex image. The key idea is to augment tiled denoising with a precomputed latent prior: we first generate a low-resolution video at the underlying model resolution and upsample its latent trajectory to obtain a global reference that captures long-range temporal and spatial structure. For 4K generation, we compute per-tile noise predictions and fuse them with this reference at every diffusion timestep by minimizing a single weighted least-squares objective in model-output space. The objective combines a standard tile-merging criterion with our regularization term, yielding a closed-form fusion update that strengthens global coherence while retaining fine detail. We additionally provide a spatial regularization variable that enables region-level control over where motion is allowed. Experiments on the VBench-I2V dataset and our proposed fresco I2V dataset show improved global consistency and fidelity over tiled baselines, while being computationally efficient. Our regularization enables explicit controllability of the trade-off between creativity and consistency.

Figures & tables

Figure 1: From a single ultra-high-definition image ( $3500\times 3500$ ), FrescoDiffusion animates it at the same resolution. We show three frames from the generated video. The red box marks a fixed spatial region tracked across time, illustrating motion temporal consistency and fine-detail preservation.

Figure 2: Overview of FrescoDiffusion. Starting from a 4K fresco image, we first build a global latent prior, by resizing the image to the native input size of the image-to-video backbone. Next, we upsample the prior latents $x_{\text{prior}}$ to fit the 4K image size. We then apply tiled denoising to the large latent canvas, $x_{t}^{\text{4K}}$ , obtaining per-tile flow predictions, $\{y_{i}\}$ . We then use $\{y_{i}\}$ and $x_{\text{prior}}$ to compute the optimal output velocity field ( Eq. 6 ) according to our loss $\ell_{\text{FD}}$ ( Eq. 5 ). This updated field is then used to update the large latent canvas, $x_{t}^{\text{4K}}$ , with the flow-matching scheduler.

Figure 3: Spatial prior schedule

\lambda_{R}

( Eq. 8 ). (Top) MSE between the foreground / background regions and the prior: the background stays bound to the prior longer than the foreground. (Bottom) Schedule for both regions.

Figure 4: Overlay of the spatial activity map onto the input fresco.

Figure 5: A qualitative comparison of fresco-scale inputs. FrescoDiffusion generates coherent global scenes and animates details at a local level. By contrast, DemoFusion, DynamicScaler and MultiDiffusion only manage to produce either coherent scenes or high-quality details, but not both.

	FrescoArchive						VBench-I2V
	FrescoDiffusion			R-FrescoDiffusion			FrescoDiffusion			R-FrescoDiffusion
	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.	Motion	Fidelity	Avg.
vs. DemoFusion	63%	60%	62%	54%	62%	58%	55%	46%	51%	66%	57%	62%
vs. DynamicScaler	78%	80%	79%	88%	80%	84%	57%	64%	61%	68%	64%	66%
vs. MultiDiffusion	78%	74%	76%	78%	74%	76%	61%	66%	64%	73%	64%	69%

Table 1: User study results: human preference rates (% of annotators preferring our method over each baseline) on both FrescoArchive (FA; 371 source images, 1344 ratings) and VBench-I2V (361 source images, 1465 ratings). Each pair is drawn from the same source image, shown at identical resolution and duration in randomized order. Green cells mark a statistically significant preference (

>50\%

at the 95% level); all rates use 95% confidence intervals of at most

\pm 6\%

Method	FrescoArchive	VBench-I2V	Time (min)	Rel. Time
DynamicScaler (original)	0.857	0.862	18.45	2.27 $\times$
DynamicScaler ∗ (CVPR’25)	0.871	0.865	10.25	1.26 $\times$
DemoFusion ∗ (CVPR’24)	0.903	0.879	13.5	1.66 $\times$
MultiDiffusion ∗ (ICML’23)	0.876	0.860	8.15	1.00 $\times$
FrescoDiffusion	0.904	0.875	8.58	1.05 $\times$
R-FrescoDiffusion	0.907	0.878	9.08	1.11 $\times$

Table 2: Standard low-resolution I2V metrics. The table shows the average performance (higher is better) using the VBench evaluation suite on both FrescoArchive and VBench-I2V image sets. We also display the average generation time in minutes.

Method	$\lambda$ function	SC	MS	A	I	ISC	IBC	Avg
MultiDiffusion	$0$	0.876	0.974	0.686	0.754	0.981	0.989	0.876
FrescoDiffusion	$\lambda_{\text{base}}$	0.942	0.991	0.645	0.598	0.979	0.983	0.856
FrescoDiffusion	$\lambda_{\text{base}}\,\cos(t\frac{\pi}{2})$	0.946	0.991	0.724	0.730	0.987	0.992	0.895
FrescoDiffusion	$\lambda_{G}$ ( Eq. 7 )	0.958	0.990	0.738	0.753	0.991	0.995	0.904
R-FrescoDiffusion	$\lambda_{R}$ ( Eq. 8 )	0.977	0.991	0.736	0.753	0.991	0.994	0.907

Table 3: Prior strength schedule ablation study on FrescoArchive. Best in bold . The results suggest that including the schedule, the gating, and the spatial regularization enhances the quantitative performance.

Figure 6: Regional constraint. Prior shows an overlay of the activity map. The red / blue boxes represent background/foreground regions. Our regional loss pulls the background toward the prior while allowing new details in foreground regions.

Panel of Figure 7 — Figure 7: Quantitative evaluation of the trade-off between creativity and prior similarity controlled by $\lambda$ . (a) Temporal consistency versus sharpness, illustrating the Pareto frontier between preserving temporal coherence and maintaining high image sharpness. (b) Evolution of prior similarity over time, showing how increasing prior strength and temporal gating progressively aligns the generated outputs with the prior.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

y\leftarrow\frac{\textit{num}+\lambda\,\sigma_{t}\,(x_{t}^{\text{4K}}-x_{\text{prior}})}{\textit{den}+\lambda\,\sigma_{t}^{2}}

Appendix

Algorithm 1 FrescoDiffusion: one sampler step at time

t

Dataset	Samples	Words	Image Width	Image Height
FrescoArchive	371	355.79 $\pm$ 88.0	2265 $\pm$ 1257	1552 $\pm$ 864
VBench	361	14.70 $\pm$ 2.24	4592 $\pm$ 1305	3748 $\pm$ 1214

Appendix

Table 4: Dataset statistics. We compare FrescoArchive and VBench datasets’ low-level statistics.

Figure 8: Analysis of FrescoArchive dataset. Left: distribution of word in caption. Right: Resolution of images distribution.

Figure 9: PCA between FrescoArchive and VBench I2V dataset. We visualize the DINOv2 [ 27 ] feature PCA between our proposed dataset and VBench’s images (red and green points, respectively). Also, in light green and red, random crops of the images. This plot shows that our dataset contains images with different statistics than standard ones, found in the VBench I2V set.

FrescoArchive
Method	Quality Metrics				I2V Metrics		Avg	Time
Method	SC	MS	A	I	ISC	IBC	Avg	Time
DynamicScaler (original)	0.945	0.975	0.693	0.706	0.893	0.930	0.857	18.45
DynamicScaler ∗ (CVPR’25)	0.852	0.971	0.681	0.754	0.980	0.989	0.871	10.25
DemoFusion ∗ (CVPR’24)	0.960	0.987	0.734	0.752	0.990	0.994	0.903	13.5
MultiDiffusion ∗ (ICML’23)	0.876	0.974	0.686	0.754	0.981	0.989	0.876	8.15

Appendix

Table 5: Quantitative evaluation on FrescoArchive and VBench-I2V. Higher is better for all metrics except time. Best in bold and second best underlined . ∗ denotes methods adapted to the video prior setting. SC, MS, A, I, ISC, and IBC stand for Subject Consistency, Motion Smoothness, Aesthetic, Imaging, I2V Subject Consistency, and I2V Background Consistency, respectively.

Method	FA	VBench-I2V
Wan (native resolution)	0.288	5.837
FlashVSR	0.247	6.421
DemoFusion	1.251	4.638
DynamicScaler	24.085	17.358
MultiDiffusion	26.217	18.469
FrescoDiffusion	1.318	4.424

Appendix

Table 6: Optical flow magnitude (RAFT-Large, median over pixels,

Q_{3}+1.5\,\text{IQR}

outliers removed) at

2560\times 1440

on FrescoArchive (FA) and VBench-I2V.

High-Resolution Video Generation

See all 13 papers

Image-to-Video Diffusion: From Foundations to Open Frontiers
May 17, 2026Xianlong Wang, Wenbo Pan, Shijia Zhou +6Image-to-Video Generation Video Diffusion Models
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China +3
CineScale: Tuning-Free High-Resolution Video Generation
Aug 21, 2025Gordon Chen, Haonan Qiu, Ning Yu +3Rotary Positional Embeddings Video Diffusion Models
Nanyang Technological University · Netflix Eyeline Studios
MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation
Jun 8, 2026Ishaan Preetam Chandratreya, David Charatan, Basile Van Hoorick +4Video Diffusion Models Long Video Generation
Massachusetts Institute of Technology · Toyota Research Institute

May 17, 2026J/K move · Enter open · S save

Image-to-Video Diffusion: From Foundations to Open Frontiers

Xianlong Wang, Wenbo Pan, Shijia Zhou, Ke Li+5

Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China+3

Image-to-Video Generation Video Diffusion Models Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Diffusion-based image-to-video (I2V) generation has become a central direction in generative models by turning a reference image, with optional conditions, into a temporally coherent video. Compared with broader video generation settings, this task places stricter demands on content consistency, identity preservation, and motion coherence. Although the literature grows rapidly, existing works mostly discuss I2V generation within broader topics and still lack a dedicated taxonomy together with a systematic analysis centered on this field. This work addresses that gap by treating diffusion I2V generation as a standalone subject. It first reviews the task formulation, model architectures, datasets, and evaluation metrics, and then organizes existing methods through a taxonomy based on architecture and training paradigm. It further distills four core designs, namely condition encoding, temporal modeling, noise prior design, and spatial-temporal upsampling, and discusses representative application scenarios together with major open challenges.

Abstract

High-Resolution Video Generation

Explore similar work

Image-to-Video Diffusion: From Foundations to Open Frontiers

CineScale: Tuning-Free High-Resolution Video Generation

MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation