ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation

Dmitriy Rivkin; Parker Ewen; Lili Gao; Julian Ost; Stefanie Walz; Rasika Kangutkar; Mario Bijelic; Felix Heide

cs.CVMar 18, 2026

ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation

Authors: Dmitriy Rivkin, Parker Ewen, Lili Gao, Julian Ost, Stefanie Walz, Rasika Kangutkar, Mario Bijelic, Felix Heide

Organizations: Torc Robotics · Princeton University

Abstract

Recent video diffusion models achieve high-quality generation through recurrent frame processing where each frame generation depends on previous frames. However, this recurrent mechanism means that training such models in the pixel domain incurs prohibitive memory costs, as activations accumulate across the entire video sequence. This fundamental limitation also makes fine-tuning these models with pixel-wise losses computationally intractable for long or high-resolution videos. This paper introduces ChopGrad, a truncated backpropagation scheme for video decoding, limiting gradient computation to local frame windows while maintaining global consistency. We provide a theoretical analysis of this approximation and show that it enables efficient fine-tuning with frame-wise losses. ChopGrad reduces training memory from scaling linearly with the number of video frames (full backpropagation) to constant memory, and compares favorably to existing state-of-the-art video diffusion models across a suite of conditional video generation tasks with pixel-wise losses, including video super-resolution, video inpainting, video enhancement of neural-rendered scenes, and controlled driving video generation.

Figures & tables

Figure 1: ChopGrad Method . ChopGrad unlocks pixel-wise losses for high resolution, long-duration video diffusion models. It leverages truncated backpropagation to eliminate recursive activation accumulation in video autoencoders with causal caching. Solid arrows indicate the flow of information in the decoder forward pass, dashed ones indicate the backward flow of gradients with ChopGrad . Adding ChopGrad to training procedures is easy and produces state of the art performance in a variety of applications that benefit from pixel-wise losses, such as video super-resolution, video inpainting, video enhancement of neural rendered scenes, and controlled driving video generation.

Figure 2: ChopGrad Model Architecture . Given the processed video frame latents, the video decoder iteratively applies causal caching at each layer, producing pixel outputs. Caching is performed by taking a subset of the layer outputs and appending these to the beginning of the layer inputs for the next frame group. While substantially reducing memory use at inference time compared to full 3D convolution over all frame groups, during training this process introduces recursive activation accumulation in the decoder, making backpropagation prohibitively expensive for high-resolution or long videos when using pixel-wise losses. Using truncated backpropagation, we only allow gradients to accumulate through a fixed number ( $D_{trunc}$ ) of previous frame groups.

Figure 3: Temporal Locality . Influence measure samples ( 2 ) as a function of temporal distance between decoder inputs (i.e. latent embeddings) and outputs (i.e. pixels) alongside the mean and line of best fit. As temporal distance increases, the influence between embeddings decreases exponentially, resulting in minimal gradient contributions ( 5 ).

Figure 4: Impact of Truncation Distance on Backbone Model Parameter Gradients . Normalized MAE and cosine distance (computed by flattening all model parameters) are shown. Though error is significant at small truncation distances, the cosine similarity remains high across all distances, implying that the errors are primarily of magnitude, not direction.

Figure 5: Truncation Induced Gradient Error. Mean gradient error ( 6 ) between the truncated and full backpropagation algorithms as a function of truncation distance.

Figure 6: Resource Utilization. Computational time and memory requirements as a function of truncation distance.

Figure 7: Spatial Locality in 3D VAEs . The video frame on the left is decoded from the original latents, while on the right a section of latents is zeroed. The red line indicates the boundary between original and zeroed latents. The upper portion of the frame is entirely unaffected by the corruption of the bottom.

Dataset	Metrics	RealESRGAN	ResShift	RealBasicVSR	Upscale-A-Video	MGLD-VSR	VEnhancer	STAR	DOVE	ChopGrad
Dataset	Metrics	[ 56 ]	[ 77 ]	[ 7 ]	[ 82 ]	[ 67 ]	[ 20 ]	[ 65 ]	[ 12 ]	(Ours)
UDM10	PSNR ( $\uparrow$ )	24.04	23.65	24.13	21.72	24.23	21.32	23.47	26.48	26.70
	SSIM ( $\uparrow$ )	0.7107	0.6016	0.6801	0.5913	0.6957	0.6811	0.6804	0.7827	0.7753
	LPIPS ( $\downarrow$ )	0.3877	0.5537	0.3908	0.4116	0.3272	0.4344	0.4242	0.2696	0.2346
	DISTS ( $\downarrow$ )	0.2184	0.2898	0.2067	0.2230	0.1677	0.2310	0.2156	0.1492	0.1143
SPMCS	PSNR ( $\uparrow$ )	21.22	21.68	22.17	18.81	22.39	18.58	21.24	23.11	23.67

Table 1: Quantitative Comparison for Video Super-Resolution . The first, second, and third best results are highlighted with dark green, light green, and yellow, respectively. ChopGrad outperforms all baselines in the majority of metrics and datasets, and achieves competitive performance otherwise.

Figure 8: Video Super-Resolution Comparison . Shown from left to right: high-resolution, low-resolution input, DOVE [ 12 ] , and the proposed approach, ChopGrad . ChopGrad synthesizes fine textures better and reduces motion blur, especially in regions with high-frequency details like fur, hair, cloth, and clouds. LPIPS scores for each frame are shown in the bottom right-hand corner, where a lower score indicates better perceptual quality. The associated videos can be found in the Appendix.

Method	FID ( $\downarrow$ )	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	Dists ( $\downarrow$ )	VBench Overall	VBench Temporal	Inference Time	Train Time
Method	FID ( $\downarrow$ )	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	Dists ( $\downarrow$ )	Quality ( $\uparrow$ )	Flickering ( $\uparrow$ )	[s/frame]	[H]
Difix [ 61 ]	16.637	17.213	0.561	0.407	0.122	0.766	0.898	0.37	2.0
MVSplat-360 [ 11 ]	38.203	15.502	0.492	0.532	0.231	0.743	0.926	2.89	-
ChopGrad	11.209	19.237	0.610	0.342	0.113	0.783	0.921	1.11	4.0
ChopGrad *	48.525	19.501	0.588	0.440	0.244	0.753	0.933	1.11	2.3
ChopGrad †	48.173	19.401	0.586	0.439	0.238	0.751	0.932	1.11	4.5

Table 2: Neural Novel View Synthesis Results . Top section: ChopGrad out-performs all baselines across all metrics except temporal flickering, where it achieves competitive performance with MVSplat-360 [ 11 ] . Interestingly, while increasing the truncation distance noticeably increases training time memory, the metric differences are minimal. Bottom section: Ablation Results for ChopGrad . ChopGrad * uses the same 1-step diffusion network, but is only trained using latent mean-squared error. ChopGrad † likewise uses latent mean-squared error for training but is trained twice as long. As such, both ablations do not propagate gradients through the video decoder. The performance of ChopGrad using various truncation distances is also presented.

Figure 9: ChopGrad vs Baselines for Neural Novel View Synthesis. Ground truth video frames and 3D Gaussian Splat renders are shown on the left. Results for MVSplat-360 [ 11 ] and Difix [ 61 ] are presented alongside ChopGrad .

Figure 10: Ablation Experiments for Neural Novel View Synthesis. ChopGrad * and ChopGrad † are trained using only the MSE loss in the latent space. The $D_{trunc}$ cases show ChopGrad results at various truncation distances.

Dataset	Method	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	VBench Overall $\uparrow$
DL3DV	VACE	45.060	574.441	20.678	0.757	0.236	0.083	0.792
DL3DV	ChopGrad	40.948	583.581	21.699	0.765	0.221	0.077	0.792
Waymo	VACE	34.856	440.651	23.229	0.804	0.212	0.079	0.836
Waymo	ChopGrad	27.057	470.545	25.048	0.823	0.192	0.071	0.835
ROVI	VACE	30.491	201.610	22.618	0.834	0.223	0.112	0.752
ROVI	ChopGrad	27.547	188.546	25.200	0.859	0.199	0.095	0.747

Table 3: Video Inpainting Evaluation . ChopGrad results are output in a single step, a

50\times

compute time improvement over VACE. VBench components are provided in the Appendix. Dark green is best, light green is second best.

Figure 11: Video Inpainting . We find that the recent VACE [ 28 ] tends to hallucinate (e.g., top section, top panel), while ChopGrad stays closer to the input but can also produce implausible results. ChopGrad results are output in a single step, a $50\times$ compute time improvement over VACE. Top:DL3DV, Middel: Waymo, Bottom: ROVI.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Mirage (HR)	27.30	0.8912	0.2067	0.0740	10.28	204.66
ChopGrad	29.49	0.9031	0.1719	0.0561	5.86	154.49

Table 4: Controlled Driving Video Generation Results . ChopGrad was produced by initializing with Mirage followed by further finetuning Mirage’s Harmonization stage for 1000 steps at high resolution (HR) / duration using ChopGrad . Dark green is best, light green is second best.

Figure 12: Controlled Driving Video Generation . Training with ChopGrad improves lighting, removes more artifacts, and produces better shadows.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Computational time and memory requirements as a function of truncation distance for the modified ChopGrad algorithm. The full video length is 24 frame groups. The dashed horizontal lines indicate time and memory requirements of the original backpropagation scheme.

Figure 14: Additional Video Super-Resolution Comparison. Shown from left to right: high-resolution, low-resolution input, DOVE [ 12 ] , and the proposed approach, ChopGrad . ChopGrad synthesizes fine textures better and reduces motion blur, especially in regions with high-frequency details like fur, hair, cloth, and clouds.

Figure 15: Additional Qualitative Results for Artifact Removal in Novel View Synthesis on the DL3DV-Benchmark Dataset [ 33 ] . Ground truth video frames and 3DGS model renders are shown on the left. Results for MVSplat-360 [ 11 ] and Difix [ 61 ] are presented alongside the ChopGrad with a truncation distance of $1$ and $2$ . ChopGrad corrects significantly more artifacts than other methods (e.g., fourth row from the top) with fewer hallucinations (e.g., 5th row from the bottom), and maintains temporal consistency over the entire video sequence.

Figure 16: Additional Video Inpainting Comparison on D3LDV Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite $50\times$ lower inference budget.

Figure 17: Additional Video Inpainting Comparison on Waymo Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite $50\times$ lower inference budget.

Figure 18: Additional Video Inpainting Comparison on Waymo-Bbox Task. In this task, 50% of the vehicles are randomly selected for masking. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite $50\times$ lower inference budget.

Figure 19: Additional Video Inpainting Comparison on ROVI Dataset. Shown from left to right: VACE, ChopGrad , Ground Truth. Training with ChopGrad reduces hallucinations despite $50\times$ lower inference budget.

Figure 20: Additional Controlled Driving Video Generation Comparison. Shown from left to right: Naive Insertion, Mirage [ 55 ] , ChopGrad , and Ground Truth. The top six rows demonstrate that training with ChopGrad increases realism by improving lighting and shadows, and removing more Gaussian Splat artifacts. The bottom two rows, which have been cropped, demonstrate that training with ChopGrad enables the model to make stronger collections in the presence of very poor vehicle model quality – note that these very poor vehicle models are relatively rare in the dataset.

Truncated Backpropagation Through Time

See all 8 papers

Temporal Aware Pruning for Efficient Diffusion-based Video Generation
May 18, 2026Sheng Li, Yang Sui, Junhao Ran +3Video Diffusion Models Token Pruning
University of Pittsburgh · Rice University · Rutgers University +1
Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model
Aug 6, 2026Haoning Yang, Xinyuan Chen, Yaohui Wang +1Diffusion Model Guidance Video Diffusion Models
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
May 15, 2026Xiaoxuan He, Siming Fu, Zeyue Xue +9Diffusion Model Alignment Video Diffusion Models
Zhejiang University · Joy Future Academy · Independent Researcher +1

May 18, 2026J/K move · Enter open · S save

Temporal Aware Pruning for Efficient Diffusion-based Video Generation

Sheng Li, Yang Sui, Junhao Ran, Bo Yuan+2

University of Pittsburgh · Rice University · Rutgers University+1

Video Diffusion Models Token Pruning Visual Token Pruning Diffusion Model Inference Acceleration Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Video diffusion models have recently enabled high-quality video generation with ViT-based architectures, but remain computationally intensive because generation requires attention computation over long spatiotemporal sequences. Token pruning has proven effective for ViTs and VLMs. However, most prior pruning methods are attention-based and operate per frame, failing to ensure the vital temporal coherence across frames in video generation tasks. In practice, naively adopting attention-only pruning causes noticeable degradation due to worsened background consistency, flickering, and reduced image quality. To address this, we propose TAPE, a training-free Temporal Aware Pruning for Efficient diffusion-based video generation. TAPE (i) applies temporal smoothing to align token-importance across adjacent frames and suppress selection jitter; and (ii) performs token reselection in selected layers to align token pruning with layers' diverse semantic focus and avoid error accumulation in specific areas; it also (iii) adopt a timestep-level budget scheduling that prunes aggressively at early noisy steps and relaxes pruning during fidelity-critical refinement. The experimental results show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.

Abstract

Truncated Backpropagation Through Time

Explore similar work

Temporal Aware Pruning for Efficient Diffusion-based Video Generation

Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model

Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization