Compression as Adaptation: Implicit Visual Representation with Diffusion Foundation Models

Zongyu Guo; Jiajun He; Zhaoyang Jia; Xiaoyi Zhang; Jiahao Li; Xiao Li; Bin Li; José Miguel Hernández-Lobato

cs.LGMar 8, 2026

Compression as Adaptation: Implicit Visual Representation with Diffusion Foundation Models

Authors: Zongyu Guo, Jiajun He, Zhaoyang Jia, Xiaoyi Zhang, Jiahao Li, Xiao Li, Bin Li, José Miguel Hernández-Lobato,

Organizations: Microsoft Research Asia · University of Cambridge

Abstract

Modern visual generative models acquire rich visual knowledge through large-scale training, yet existing visual representations (such as pixels, latents, or tokens) remain external to the model and cannot directly exploit this knowledge for compact storage or reuse. In this work, we introduce a new visual representation framework that encodes a signal as a function, which is parametrized by low-rank adaptations attached to a frozen visual generative model. Such implicit representations of visual signals, e.g., an 81-frame video, can further be hashed into a single compact vector, achieving strong perceptual video compression at extremely low bitrates. Beyond basic compression, the functional nature of this representation enables inference-time scaling and control, allowing additional refinement on the compression performance. More broadly, as the implicit representations directly act as a function of the generation process, this suggests a unified framework bridging visual compression and generation.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Explicit representations by encoding signals into symbolic latent variables. (b) Implicit representations that encode signal information implicitly in functions. (c) One-vector adaptation of large-scale generative models can serve as implicit visual representations.

Figure 2: Framework for visual compression with a single vector that adapts the diffusion foundation model.

Panel of Figure 3 — Figure 3: Reconstruction quality v.s. training step. (a) Common LoRA representations with different ranks for image Kodim03 from Kodak dataset ( Kodak, 1993 ) . (b) One-vector representations for image Kodim03 , varying LoRA rank and vector size after hashing. (c) One-vector representations for video Beauty from UVG dataset ( Mercat et al., 2020 ) .

Figure 4: Comparisons with existing video codecs on UVG. For DISTS and FVD, lower is better. For PSNR, higher is better.

Panel of Figure 5 — Figure 5: Visual comparisons with different methods. More results are in Appendix F.3 and supplementary materials.

Figure 6: Reconstruction quality by inference-time scaling with different number of steps and different sample sizes per step.

Figure 7: Use LoRA-based representations for personalized generation, such as image editing or merging. Base model is the text-to-image diffusion foundation model ( Wu et al., 2025 ) . Note that the original base model does not accept visual input.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

w^{(m)}=\frac{p^{*}(x_{t_{n-1}}^{(m)}\mid x_{t_{n}})}{p_{\theta}(x_{t_{n-1}}^{(m)}\mid x_{t_{n}})}.

Appendix

Algorithm 1 Encoding-Time Scaling via Importance-Sampling

Figure 8: Distortion-Perception Trade-off via early stopping on UVG.

Figure 9: Distortion-Perception Trade-off via early stopping coupled with inference-time scaling on UVG.

Panel of Figure 10 — Figure 10: Rate-distortion comparisons with the previous INR-based video codec NVRC on ( Kwan et al., 2024 ) on UVG.

Figure 11: Compression performance of direct scaling on the pretrained diffusion model at different bitrates and scaling budgets. Here,

\lambda=\infty

denotes the setting without one-vector adaptation.

Figure 12: Image editing results with one-vector representation. In the last example, we also show that image generation ability is largely maintained.

Figure 13: Video editing results on Wan-2.1-14B, with one vector of length 524288. In the first example, changing the dog to a panda results in a coherent edit. However, in the second example, due to biases in the base model, black hair is often associated with Asian individuals, which leads the model to modify the face accordingly.

Figure 14: Overfitting in the representation can also influence the generation under irrelevant prompts. When the prompt is highly relevant to the scene or contains key objectives, the model’s output is strongly influenced and may even directly reproduce the overfitted video. For prompts that are less relevant, the influence is weaker but still observable, affecting attributes such as color, motion, or background.

Panel of Figure 15 — Figure 15: Above: Reconstruction with original image captions “A row of brightly colored ’Bahamas’ baseball caps hangs on a sunlit wooden wall. Neon yellow, orange, green, pink, and blue hats cast sharp shadows. Above, a deep blue sky with wispy white clouds enhances the vivid tropical atmosphere.”. Below: Editing with image captions “A row of brightly colored ’Bahamas’ baseball caps hangs on a sunlit wooden wall. Blue, orange, green, pink, and blue hats cast sharp shadows. Above, a deep blue sky with wispy white clouds enhances the vivid tropical atmosphere.”

Figure 16: Overfitting prompt: “A lone rockhopper penguin stands on rugged, dark gray coastal rocks, wings slightly outstretched. Its black-and-white body contrasts with bright yellow head crests and pink beak and feet, creating a crisp, windswept, subantarctic shoreline atmosphere.” Generation prompt: “A lone penguin.”

Panel of Figure 17 — Figure 17: Left: Original images that are used for learning a single group of implicit visual representations. Right: Reconstructed images from the learned single group of implicit visual representations. The content in two rows of images are relevant.

Panel of Figure 18 — Figure 18: Left: Original images that are used for learning a single group of implicit visual representations. Right: Reconstructed images from the learned single group of implicit visual representations. The content in all ten images are irrelevant.

Table 20

Training
Resolution	Cropped and resized to 832x480x81
Generative model	Wan-2.1-1.3B ( Wan et al., 2025 )
Transformer number	30
LoRA rank $r$	1
LoRA position	QKVO
One-vector size $k$	$2^{17}=131072$

Appendix

Table 3: Hyperparameters for compression and scaling experiments.

Method	Encoding per frame	Decoding per frame
VTM/H.266 RA mode	30.51 s	19.0 ms
DCVC-RT	5.84 ms	5.90 ms
Our VOV	$\sim$ 18 min (overfitting)	2.48 s
+ 1000-step scaling	+27 s (importance sampling)	25 s

Appendix

Table 4: Encoding and Decoding time.

Figure 19: Comparisons with existing video codecs on HEVC-B. For DISTS, FVD and LPIPS, lower is better. For PSNR, higher is better.

Figure 20: Comparisons with existing video codecs on HEVC-C. For DISTS, FVD and LPIPS, lower is better. For PSNR, higher is better.

Figure 21: Comparisons with existing video codecs on HEVC-E. For DISTS, FVD and LPIPS, lower is better. For PSNR, higher is better.

Figure 22: Comparisons with existing video codecs on UVG. For DISTS, FVD and LPIPS, lower is better. For PSNR, higher is better.

Panel of Figure 25 — Figure 25: Additional visual comparisons with different video codecs.

Generative Video Compression

Generative video compression uses learned generative models to reconstruct perceptually convincing video from compact transmitted signals.

See all 22 papers

Generative Video Compression with Adaptive Score Distillation
Jul 24, 2026Naifu Xue, Zhaoyang Jia, Haosen Li +7Neural Video Compression Diffusion Model Distillation
Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China +1
ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors
Jun 21, 2026Yixin Gao, Xiaohan Pan, Lin Liu +3Video Diffusion Models Neural Video Compression
University of Science and Technology of China · 2Huawei Inc.
Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion
May 4, 2026Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara JavidiLearned Image Compression Video Diffusion Models
Department of Electrical and Computer Engineering University of California San Diego · Department of Electrical and Systems Engineering University of Pennsylvania · Department of Electrical and Computer Engineering2026 University of California San Diego

Jul 24, 2026J/K move · Enter open · S save

Generative Video Compression with Adaptive Score Distillation

Naifu Xue, Zhaoyang Jia, Haosen Li, Zihan Zheng+6

Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China+1

Neural Video Compression Diffusion Model Distillation Pixel-Space Diffusion Models Video Diffusion Models Generative Video Compression Score Distillation Sampling+1

Open paper AskPDF arXiv

Diffusion models provide strong generative capabilities for video compression at ultra-low bitrates. Existing diffusion-based video codecs adapt base models originally developed for text-conditioned generation, whereas diffusion models designed and trained specifically for compression remain unexplored. To fill this gap, we introduce our Generative Video Codec (GenVC), built on a video diffusion model trained from scratch for compression. To our knowledge, this is the first compression-oriented video diffusion model. We realize this model directly in pixel space with a global-to-local hierarchy that recovers fine spatio-temporal details, enabling high-quality generative reconstruction from compressed representations. To accelerate inference, we distill the multi-step model into one step using distribution matching distillation (DMD). Applying DMD directly, however, drives the student toward motion-stalled reconstructions. We trace this to a teacher-side guidance failure: once student-induced perturbations leave the frozen teacher's training region, its guidance can become misleading, causing DMD updates to reinforce rather than correct the student drift. To break the resulting feedback loop, we propose Adaptive Score Distillation, which gates DMD updates according to their alignment with the ground-truth direction, enabling high-quality reconstruction with coherent motion. Experimental results show that GenVC achieves state-of-the-art perceptual quality at ultra-low bitrates, with average bitrate savings of 62.5% at matched LPIPS and 71.3% at matched FID over GLVC. Unlike prior codecs that inherit billion-scale pretrained backbones, our diffusion model has only 478.0M parameters and decodes 1080p video in a single step at 15.1 fps on an A100 GPU.

Abstract

Generative Video Compression

Explore similar work

Generative Video Compression with Adaptive Score Distillation

ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion