Recent video diffusion models achieve high-quality generation through recurrent frame processing where each frame generation depends on previous frames. However, this recurrent mechanism means that training such models in the pixel domain incurs prohibitive memory costs, as activations accumulate across the entire video sequence. This fundamental limitation also makes fine-tuning these models with pixel-wise losses computationally intractable for long or high-resolution videos. This paper introduces ChopGrad, a truncated backpropagation scheme for video decoding, limiting gradient computation to local frame windows while maintaining global consistency. We provide a theoretical analysis of this approximation and show that it enables efficient fine-tuning with frame-wise losses. ChopGrad reduces training memory from scaling linearly with the number of video frames (full backpropagation) to constant memory, and compares favorably to existing state-of-the-art video diffusion models across a suite of conditional video generation tasks with pixel-wise losses, including video super-resolution, video inpainting, video enhancement of neural-rendered scenes, and controlled driving video generation.
Figures & tables
Figure 1: ChopGrad Method . ChopGrad unlocks pixel-wise losses for high resolution, long-duration video diffusion models. It leverages truncated backpropagation to eliminate recursive activation accumulation in video autoencoders with causal caching. Solid arrows indicate the flow of information in the decoder forward pass, dashed ones indicate the backward flow of gradients with ChopGrad . Adding ChopGrad to training procedures is easy and produces state of the art performance in a variety of applications that benefit from pixel-wise losses, such as video super-resolution, video inpainting, video enhancement of neural rendered scenes, and controlled driving video generation.
Figure 2: ChopGrad Model Architecture . Given the processed video frame latents, the video decoder iteratively applies causal caching at each layer, producing pixel outputs. Caching is performed by taking a subset of the layer outputs and appending these to the beginning of the layer inputs for the next frame group. While substantially reducing memory use at inference time compared to full 3D convolution over all frame groups, during training this process introduces recursive activation accumulation in the decoder, making backpropagation prohibitively expensive for high-resolution or long videos when using pixel-wise losses. Using truncated backpropagation, we only allow gradients to accumulate through a fixed number ( Dtrunc ) of previous frame groups.
Figure 3: Temporal Locality . Influence measure samples ( 2 ) as a function of temporal distance between decoder inputs (i.e. latent embeddings) and outputs (i.e. pixels) alongside the mean and line of best fit. As temporal distance increases, the influence between embeddings decreases exponentially, resulting in minimal gradient contributions ( 5 ).
Figure 4: Impact of Truncation Distance on Backbone Model Parameter Gradients . Normalized MAE and cosine distance (computed by flattening all model parameters) are shown. Though error is significant at small truncation distances, the cosine similarity remains high across all distances, implying that the errors are primarily of magnitude, not direction.
Figure 5: Truncation Induced Gradient Error. Mean gradient error ( 6 ) between the truncated and full backpropagation algorithms as a function of truncation distance.
Figure 6: Resource Utilization. Computational time and memory requirements as a function of truncation distance.
Figure 7: Spatial Locality in 3D VAEs . The video frame on the left is decoded from the original latents, while on the right a section of latents is zeroed. The red line indicates the boundary between original and zeroed latents. The upper portion of the frame is entirely unaffected by the corruption of the bottom.
Dataset
Metrics
RealESRGAN
ResShift
RealBasicVSR
Upscale-A-Video
MGLD-VSR
VEnhancer
STAR
DOVE
ChopGrad
[ 56 ]
[ 77 ]
[ 7 ]
[ 82 ]
[ 67 ]
[ 20 ]
[ 65 ]
[ 12 ]
(Ours)
UDM10
PSNR ( ↑ )
24.04
23.65
24.13
21.72
24.23
21.32
23.47
26.48
26.70
SSIM ( ↑ )
0.7107
0.6016
0.6801
0.5913
0.6957
0.6811
0.6804
0.7827
0.7753
LPIPS ( ↓ )
0.3877
0.5537
0.3908
0.4116
0.3272
0.4344
0.4242
0.2696
0.2346
DISTS ( ↓ )
0.2184
0.2898
0.2067
0.2230
0.1677
0.2310
0.2156
0.1492
0.1143
SPMCS
PSNR ( ↑ )
21.22
21.68
22.17
18.81
22.39
18.58
21.24
23.11
23.67
Table 1: Quantitative Comparison for Video Super-Resolution . The first, second, and third best results are highlighted with dark green, light green, and yellow, respectively. ChopGrad outperforms all baselines in the majority of metrics and datasets, and achieves competitive performance otherwise.
Figure 8: Video Super-Resolution Comparison . Shown from left to right: high-resolution, low-resolution input, DOVE [ 12 ] , and the proposed approach, ChopGrad . ChopGrad synthesizes fine textures better and reduces motion blur, especially in regions with high-frequency details like fur, hair, cloth, and clouds. LPIPS scores for each frame are shown in the bottom right-hand corner, where a lower score indicates better perceptual quality. The associated videos can be found in the Appendix.
Method
FID ( ↓ )
PSNR ( ↑ )
SSIM ( ↑ )
LPIPS ( ↓ )
Dists ( ↓ )
VBench Overall
VBench Temporal
Inference Time
Train Time
Quality ( ↑ )
Flickering ( ↑ )
[s/frame]
[H]
Difix [ 61 ]
16.637
17.213
0.561
0.407
0.122
0.766
0.898
0.37
2.0
MVSplat-360 [ 11 ]
38.203
15.502
0.492
0.532
0.231
0.743
0.926
2.89
-
ChopGrad
11.209
19.237
0.610
0.342
0.113
0.783
0.921
1.11
4.0
ChopGrad *
48.525
19.501
0.588
0.440
0.244
0.753
0.933
1.11
2.3
ChopGrad †
48.173
19.401
0.586
0.439
0.238
0.751
0.932
1.11
4.5
Table 2: Neural Novel View Synthesis Results . Top section: ChopGrad out-performs all baselines across all metrics except temporal flickering, where it achieves competitive performance with MVSplat-360 [ 11 ] . Interestingly, while increasing the truncation distance noticeably increases training time memory, the metric differences are minimal. Bottom section: Ablation Results for ChopGrad . ChopGrad * uses the same 1-step diffusion network, but is only trained using latent mean-squared error. ChopGrad † likewise uses latent mean-squared error for training but is trained twice as long. As such, both ablations do not propagate gradients through the video decoder. The performance of ChopGrad using various truncation distances is also presented.
Figure 9: ChopGrad vs Baselines for Neural Novel View Synthesis. Ground truth video frames and 3D Gaussian Splat renders are shown on the left. Results for MVSplat-360 [ 11 ] and Difix [ 61 ] are presented alongside ChopGrad .
Figure 10: Ablation Experiments for Neural Novel View Synthesis. ChopGrad * and ChopGrad † are trained using only the MSE loss in the latent space. The Dtrunc cases show ChopGrad results at various truncation distances.
Dataset
Method
FID ↓
FVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
VBench Overall ↑
DL3DV
VACE
45.060
574.441
20.678
0.757
0.236
0.083
0.792
ChopGrad
40.948
583.581
21.699
0.765
0.221
0.077
0.792
Waymo
VACE
34.856
440.651
23.229
0.804
0.212
0.079
0.836
ChopGrad
27.057
470.545
25.048
0.823
0.192
0.071
0.835
ROVI
VACE
30.491
201.610
22.618
0.834
0.223
0.112
0.752
ChopGrad
27.547
188.546
25.200
0.859
0.199
0.095
0.747
Table 3: Video Inpainting Evaluation . ChopGrad results are output in a single step, a 50× compute time improvement over VACE. VBench components are provided in the Appendix. Dark green is best, light green is second best.
Figure 11: Video Inpainting . We find that the recent VACE [ 28 ] tends to hallucinate (e.g., top section, top panel), while ChopGrad stays closer to the input but can also produce implausible results. ChopGrad results are output in a single step, a 50× compute time improvement over VACE. Top:DL3DV, Middel: Waymo, Bottom: ROVI.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
FID ↓
FVD ↓
Mirage (HR)
27.30
0.8912
0.2067
0.0740
10.28
204.66
ChopGrad
29.49
0.9031
0.1719
0.0561
5.86
154.49
Table 4: Controlled Driving Video Generation Results . ChopGrad was produced by initializing with Mirage followed by further finetuning Mirage’s Harmonization stage for 1000 steps at high resolution (HR) / duration using ChopGrad . Dark green is best, light green is second best.
Figure 12: Controlled Driving Video Generation . Training with ChopGrad improves lighting, removes more artifacts, and produces better shadows.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Computational time and memory requirements as a function of truncation distance for the modified ChopGrad algorithm. The full video length is 24 frame groups. The dashed horizontal lines indicate time and memory requirements of the original backpropagation scheme.
Figure 14: Additional Video Super-Resolution Comparison. Shown from left to right: high-resolution, low-resolution input, DOVE [ 12 ] , and the proposed approach, ChopGrad . ChopGrad synthesizes fine textures better and reduces motion blur, especially in regions with high-frequency details like fur, hair, cloth, and clouds.
Figure 15: Additional Qualitative Results for Artifact Removal in Novel View Synthesis on the DL3DV-Benchmark Dataset [ 33 ] . Ground truth video frames and 3DGS model renders are shown on the left. Results for MVSplat-360 [ 11 ] and Difix [ 61 ] are presented alongside the ChopGrad with a truncation distance of 1 and 2 . ChopGrad corrects significantly more artifacts than other methods (e.g., fourth row from the top) with fewer hallucinations (e.g., 5th row from the bottom), and maintains temporal consistency over the entire video sequence.
Figure 16: Additional Video Inpainting Comparison on D3LDV Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite 50× lower inference budget.
Figure 17: Additional Video Inpainting Comparison on Waymo Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite 50× lower inference budget.
Figure 18: Additional Video Inpainting Comparison on Waymo-Bbox Task. In this task, 50% of the vehicles are randomly selected for masking. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite 50× lower inference budget.
Figure 19: Additional Video Inpainting Comparison on ROVI Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite 50× lower inference budget.
Figure 20: Additional Controlled Driving Video Generation Comparison. Shown from left to right: Naive Insertion, Mirage [ 55 ] , ChopGrad , and Ground Truth. The top six rows demonstrate that training with ChopGrad increases realism by improving lighting and shadows, and removing more Gaussian Splat artifacts. The bottom two rows, which have been cropped, demonstrate that training with ChopGrad enables the model to make stronger collections in the presence of very poor vehicle model quality – note that these very poor vehicle models are relatively rare in the dataset.
Video diffusion models have recently enabled high-quality video generation with ViT-based architectures, but remain computationally intensive because generation requires attention computation over long spatiotemporal sequences. Token pruning has proven effective for ViTs and VLMs. However, most prior pruning methods are attention-based and operate per frame, failing to ensure the vital temporal coherence across frames in video generation tasks. In practice, naively adopting attention-only pruning causes noticeable degradation due to worsened background consistency, flickering, and reduced image quality. To address this, we propose TAPE, a training-free Temporal Aware Pruning for Efficient diffusion-based video generation. TAPE (i) applies temporal smoothing to align token-importance across adjacent frames and suppress selection jitter; and (ii) performs token reselection in selected layers to align token pruning with layers' diverse semantic focus and avoid error accumulation in specific areas; it also (iii) adopt a timestep-level budget scheduling that prunes aggressively at early noisy steps and relaxes pruning during fidelity-critical refinement. The experimental results show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.
Sheng Li, Yang Sui, Junhao Ran +3
University of Pittsburgh · Rice University · Rutgers University +1
Recently, diffusion models have made great progress in video generation. However, most existing video diffusion models are trained with short videos, and degrade when extrapolated to long videos, struggling to maintain long-range temporal coherence while retaining diverse motions. To generate consistent, high-quality and dynamic long videos, we propose Diff-VF, a training-free, plug-and-play and model-agnostic framework that converts existing short-video diffusion backbones into long-video generators without modifying or fine-tuning the base model. Diff-VF couples three complementary strategies: Hybrid Noise Initialization (HNI) to constrain global semantics, Weighted Window Sampling (WWS) to remove inter-window discontinuities, and Temporal Extended Sampling (TES) to establish long-range dependencies with a timestep-varying fusion. We further extend Diff-VF to long-video enhancement via Skip Residual Guidance that balances fidelity and realism through timestep-dependent guidance. VBench-Long evaluation results show that Diff-VF achieves a more favorable balance between temporal coherence and motion diversity than base models and recent training-free long video generation baselines, including FreeNoise, FreeLong, and RIFLEx, while maintaining competitive frame-wise quality. Experiments on two base models demonstrate the applicability to video diffusion models with different spatial-temporal modeling strategies. Extensive ablations validate the contribution of each component and hyperparameters.
Haoning Yang, Xinyuan Chen, Yaohui Wang +1
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.
Xiaoxuan He, Siming Fu, Zeyue Xue +9
Zhejiang University · Joy Future Academy · Independent Researcher +1