Learning Motion-Appearance Coupling Priors for Solving Video Inverse Problems
Organizations: School of Computation, Information and Technology Technical University of Munich Munich Center for Machine Learning
Abstract
Reconstructing video from degraded measurements requires priors that capture both frame appearance and temporal dynamics. Many recent approaches combine pretrained image diffusion priors with motion guidance, sidestepping the costs of training large video models. Temporal consistency is often enforced via photometric losses or warping-based constraints, which implicitly assume that motion explains frame-to-frame or latent-to-latent changes up to small deviations, an assumption that fails under occlusions, lighting changes, and non-rigid deformation. In this paper, we propose to learn the coupling between motion and frame appearance as a generative prior. We learn the distribution of motion-induced appearance residuals with a conditional diffusion model and use it to regularize reconstruction in video inverse problems. Our motion-appearance prior acts as a plug-in regularizer for diffusion-based video reconstruction, preserving the benefits of image diffusion models while remaining compatible with existing appearance regularization techniques. Experiments on dynamic video inverse problems show that reconstruction with a learnt motion-appearance coupling prior outperforms photometric-consistency and noise-warping baselines, and that the approach also transfers to cardiac MRI reconstruction.
Figures & tables
| Methods | Dynamic Compressive Sens. | Video Super-Resolution | ||||
|---|---|---|---|---|---|---|
| PSNR | LPIPS | Warp. | PSNR | LPIPS | Warp. | |
| ADM (DDS-Sampling) | 27.77 | 0.0271 | 0.00590 | 32.79 | 0.0711 | 0.00084 |
| + Batch Noise | 26.23 | 0.0344 | 0.00762 | 32.74 | 0.0702 | 0.00080 |
| + Integral Noise | 26.86 | 0.0282 | 0.00720 | 32.81 | 0.0728 | 0.00081 |
| + ConfPM | 37.40 | 0.0144 | 0.00079 | 32.96 | 0.0731 | 0.00076 |
| TV (Variational) | 17.95 | 0.1420 | 0.04723 | 32.42 | 0.0547 | 0.00068 |
| Method | PSNR | SSIM |
|---|---|---|
| Classical baseline (L+S; [ OCS15 ] ) | 31.28 | 0.77 |
| Video diffusion model (PDDR; [ FH26 ] ) | 32.36 | 0.87 |
| Classical coupling (ConfPM) | 29.48 | 0.78 |
| Learnt coupling (Ours) | 34.47 | 0.89 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| PSNR | (dB) | ||||
|---|---|---|---|---|---|
| Method | No shift | Dataset-shift | Task-shift | Dataset-shift | Task-shift |
| ADM + Integral noise | 24.80 | 24.64 | 24.26 | ||
| TV + Learnt Coupling (Ours) | 25.44 | 25.28 | 25.21 | ||
| REDS | Vimeo | |||||
| Method | PSNR | LPIPS | Warp. | PSNR | LPIPS | Warp. |
| ADM (DDS-Sampling) | ||||||
| + Integral noise | 24.80 | 0.317 | 0.00119 | 29.49 | 0.136 | 0.00116 |
| + ConfPM | 24.90 | 0.381 | 0.00104 | 30.05 | 0.187 | 0.00078 |
| Variational (Ours) | ||||||
| TV + Learnt coupling | 25.44 | 0.292 | 0.00093 | 30.23 | 0.120 | 0.00076 |
| Dynamic Compressive Sens. | Super-Resolution | Deblurring | |||||
|---|---|---|---|---|---|---|---|
| Method | PSNR | LPIPS | Warp. | PSNR | LPIPS | PSNR | LPIPS |
| Classical coupling | 33.98 | 0.0591 | 0.001031 | 33.35 | 0.0678 | 25.20 | 0.2789 |
| Learnt coupling | 17.82 | 0.1018 | 0.040157 | 36.34 | 0.0331 | 25.84 | 0.2355 |
| Fixed combination | 38.54 | 0.0161 | 0.000752 | 36.92 | 0.0257 | 25.78 | 0.2726 |
| Cosine schedule | 40.20 | 0.0142 | 0.000733 | 36.63 | 0.0408 | 25.80 | 0.2482 |
| Model distribution | PSNR | LPIPS |
|---|---|---|
| Frame distance | PSNR | LPIPS |
|---|---|---|
| 1 | 40.20 | 0.0142 |
| 2 | 40.29 | 0.0146 |
| 4 | 40.46 | 0.0147 |
| 6 | 40.39 | 0.0144 |
| PSNR (clean motion in parentheses) | (dB) | |||||
|---|---|---|---|---|---|---|
| Method | ||||||
| ConfPM | 30.7 (38.1) | 26.7 (35.0) | 21.0 (32.6) | |||
| Learnt Coupling (Ours) | 40.1 (44.1) | 32.4 (37.9) | 23.6 (34.8) | |||
| Measurement used for | Motion estimation quality | Reconstruction quality | ||||
|---|---|---|---|---|---|---|
| motion estimation | EPE (px) | % px px | Ang. err. ( ∘ ) | PSNR | LPIPS | Warp. |
| 0.04 | 0.1 | 0.6 | 44.23 | 0.0076 | 0.00059 | |
| 0.47 | 2.2 | 6.5 | 44.08 | 0.0066 | 0.00056 | |
| 1.86 | 9.3 | 20.5 | 42.87 | 0.0096 | 0.00058 | |
| (matched) | 3.49 | 15.6 | 27.9 | 40.27 | 0.0132 | 0.00068 |
| 7.15 | 26.1 | 36.5 | 36.24 | 0.0277 | 0.00109 | |
| Inference-time motion estimator | PSNR | LPIPS | Warp. |
|---|---|---|---|
| RAFT (synthetic + real, same as training) | 25.80 | 0.2482 | 0.0011 |
| RAFT (synthetic only) | 25.78 | 0.2506 | 0.0011 |
| Classical (Farneback) | 25.64 | 0.2786 | 0.0011 |
| Prediction | PSNR | LPIPS |
|---|---|---|
| Warping (RAFT) | 26.71 | 0.0715 |
| Learnt coupling | 27.82 | 0.0617 |
| Learnt coupling (avg. of 10 samples) | 28.44 | 0.0603 |
| MSE | ||||
|---|---|---|---|---|
| Region | % of image | RAFT | Learnt coupling | MSE reduction |
| Reliable | 87.5% | 0.00236 | 0.00225 | 4.4% |
| Unreliable/occluded | 12.5% | 0.0294 | 0.0204 | 30.7% |
| Method | PSNR | LPIPS | Warp. |
|---|---|---|---|
| SD1.5 (Sampling) | |||
| + Random noise (latent) | 24.76 | 0.291 | 0.00456 |
| + Integral noise (latent) | 25.07 | 0.266 | 0.00427 |
| No app. (Variational) | |||
| + Classical Coupling (latent) | 28.88 | 0.185 | 0.00242 |
| + Learnt Coupling (latent) | 29.71 | 0.155 | 0.00206 |
| Steps | Learnt Coupling (Ours) | ADM + ConfPM (DDS) |
|---|---|---|
| 50 | 8.8s | 9.0s |
| 100 | 15.9s | 16.4s |
| 250 | 37.2s | 38.6s |
| 500 | 73.1s | 75.9s |
| 1000 | 144.2s | 150.5s |
| Comparison: TV + Learnt Coupling (Ours) | Metric (Ours - Baseline) | Wilcoxon |
|---|---|---|
| PSNR | ||
| vs. ADM (DDS) + Integral Noise | 3.60dB | |
| vs. ADM (DDS) + ConfPM | 3.45dB | |
| vs. ADM (Var.) + ConfPM | 2.01dB | |
| LPIPS | ||
| vs. ADM (DDS) + Integral Noise | -0.0388 |
Diffusion-Based Inverse Problems
Diffusion-based inverse problems use generative priors to recover images and physical fields from incomplete, noisy, or indirect measurements.
See all 156 papers