cs.CVMar 24, 2026

ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images

Authors: Yunfeng Wu, Hongying Cheng, Zihao He, Songhua Liu

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University · Xi’an Jiaotong-Liverpool University · Jilin University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transformer-based video diffusion models rely on 3D attention over spatial and temporal tokens, which incurs quadratic time and memory complexity and makes end-to-end training for ultra-high-resolution videos prohibitively expensive. To overcome this bottleneck, we propose a pure image adaptation framework that upgrades a video Diffusion Transformer pre-trained at its native scale to synthesize higher-resolution videos. Unfortunately, naively fine-tuning with high-resolution images alone often introduces noticeable noise due to the image-video modality gap. To address this, we decouple the learning objective to separately handle modality alignment and spatial extrapolation. At the core of our approach is Relay LoRA, a two-stage adaptation strategy. In the first stage, the video diffusion model is adapted to the image domain using low-resolution images to bridge the modality gap. In the second stage, the model is further adapted with high-resolution images to acquire spatial extrapolation capability. During inference, only the high-resolution adaptation is retained to preserve the video generation modality while enabling high-resolution video synthesis. To enhance fine-grained detail synthesis, we further propose a High-Frequency-Awareness-Training-Objective, which explicitly encourages the model to recover high-frequency components from degraded latent representations via a dedicated reconstruction loss. Extensive experiments demonstrate that our method produces ultra-high-resolution videos with rich visual details without requiring any video training data, even outperforming previous state-of-the-art models trained on high-resolution videos by 0.8 on the VBench benchmark. Code will be available at https://github.com/WillWu111/ViBe.

Figures & tables

Figure 2: The two plots show how training VRAM consumption and iteration time scale with the number of frames under different resolutions.
Figure 3: The plots on the left shows the general pipeline of our Relay LoRA. The two panels on the right provide a qualitative analysis on Wan2.2 [ 39 ] : naive high-resolution image fine-tuning (Naive LoRA) introduces noticeable artifacts, whereas ours methods (Relay LoRA) effectively removes them.
Figure 4: Upper Left : Stage-1 fine-tuning trains LoRA1\mathrm{LoRA}_{1} to adapt the DiT backbone to single low-resolution frame generation. Upper Middle : Stage-2 first merges LoRA1\mathrm{LoRA}_{1} into the base model and freezes the merged weights, then trains a Relay LoRA2\mathrm{LoRA}_{2} to adapt the DiT backbone to single high-resolution frame generation. Upper Right : During inference, only LoRA2\mathrm{LoRA}_{2} is loaded on the base model for video generation. Bottom Left : Global-Coarse-Local-Fine-Attention combines an inward sliding-window local attention with pooled coarse attention. Bottom Right : The training objective first degrades the latents via a downsample–upsample operation, then add noise on the degraded sample. The model applies the predicted flow and computes the loss against the clean latents.
Figure 5: Qualitative comparison. ViBe yields high-resolution videos characterized by high-fidelity details and coherent structure. The red boxes highlight regions with incorrect semantics or layout. The blue boxes provide zoomed-in views.
Resolution (W ×\times H)ModelSubject ConsistencyBackground ConsistencyMotion SmoothnessAesthetic QualityImaging QualityOverall ConsistencyOverall Score
1920 ×\times 1088Wan2.2 [ 39 ]94.9%95.4%98.5%59.1%64.0%23.3%72.5%
ESRGAN [ 40 ]95.3%93.8%96.9%60.4%63.6%25.7%72.6%
UAV [ 45 ]95.2%93.2%96.8%58.7%62.5%25.2%71.9%
I-Max [ 8 ]95.7%95.3%97.7%60.2%62.0%24.0%72.5%
HiFlow [ 3 ]95.5%95.6%97.4%60.1%59.6%25.4%72.3%
CineScale [ 32 ]93.9%96.8%97.4%59.1%60.7%23.5%71.9%
Table 1: Video comparison with both training-based and training-free models at 1080P and 4K resolutions. The highest value is bold , and the second-highest is underlined . Overall Score is computed as the mean of the six metrics in each row.
Figure 6: Results of user study for high-resolution video generation. Participants were expected to select the best method based on Text Alignment, Aesthetic Quality, and Video Quality.
Figure 7: Qualitative ablations. We perform controlled comparisons of our method with alternative variants under controlled settings. Best viewed zoomed in. The red boxes highlight regions with more noticeable noise, and the blue boxes indicate the corresponding zoomed-in views.
MethodsSubject ConsistencyBackground ConsistencyMotion SmoothnessAesthetic QualityImaging QualityOverall ConsistencyOverall Score
w/o Relay LoRA94.6%97.9%98.3%46.7%37.4%21.1%66.0%
w/o GCLFA94.9%95.4%99.0%52.2%49.0%24.3%69.1%
w/o HFATO95.0%94.2%99.1%53.7%59.0%24.0%70.8%
w/o X0X_{0} Loss96.3%95.3%99.2%57.3%59.2%26.7%72.3%
Ours (DR = 1/2)95.2%97.1%99.2%61.4%66.1%27.1%74.4%
Ours (DR = 1/4)95.0%97.2%99.3%61.3%65.6%26.5%74.2%
Table 2: Quantitative ablations. Ours Methods achieves the optimal overall video quality. The highest value is bold , and the second-highest is underlined .
Figure 8: Left : Illustration 4 steps generation results integrated with our framework. Right : Illustration Image-to-Video generation results integrated with our framework.
Figure 9: More Qualitative ablations. We perform controlled comparisons of our method with alternative variants under controlled settings. Best viewed zoomed in. The red boxes highlight regions with more noticeable noise, and the blue boxes indicate the corresponding zoomed-in views.
Figure 10: A fair comparison of Local Attention and GCLFA. We perform controlled comparisons of our method with attnetion variants under controlled settings. Best viewed zoomed in. The red boxes highlight regions with incorrect semantics or layout.
Figure 11: Left : Illustration 4 steps generation results integrated with our framework. Right : Illustration Image-to-Video generation results integrated with our framework.
Figure 12: Two examples of style transfer with ViBe. In each case, the top row shows the original video, while the bottom row presents the stylized result.
Figure 13: Qualitative results of 1080P and 2K generated by our method ViBe integrated with Wan2.2 [ 39 ] .
Figure 14: Qualitative results of 3K and 4K generated by our method ViBe integrated with Wan2.2 [ 39 ] .
More in this topic

High-Resolution Video Generation

See all 13 papers

Explore similar work

CardsList

  1. AtlasVid: Efficient Ultra-High-Resolution Long Video Generation via Decoupled Global-Local Modeling

    May 15, 2026Ziyang Mai, Yuyao Zhang, Yu-Wing TaiVideo Diffusion ModelsHigh-Resolution Video Generation

    Dartmouth College

  2. LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution

    Jun 8, 2026Yu Cao, Ziquan Liu, Zhensong Zhang +3Video Diffusion ModelsDiffusion Transformer

    Queen Mary University of London · Huawei Darwin Research Center · Imperial College London

  3. High-Throughput Generative Video Super-Resolution on a Single Consumer GPU

    Oct 9, 2026Georgios Panaretou, Sotirios ChatzisVideo Super-ResolutionTemporal Consistency in Video Generation

    Ethical AI Novelties Limassol, Cyprus · Cyprus University of Technology Limassol, Cyprus