Diffusion Model Fine-Tuning

Ask about this topic

Overview

Diffusion model fine-tuning studies how to adapt pretrained image, video, audio, molecular, and language generators without rebuilding them from scratch. Parameter-efficient methods such as LoRA, prompt tuning, and adapters support personalization and domain adaptation, while supervised training and self-generated trajectories target tasks such as image editing, restoration, and masked-language-model refinement. For preference alignment and constrained generation, reinforcement learning methods such as GRPO, reward matching, and distillation optimize outputs using reward signals, often on benchmarks for text-image alignment, generation quality, or downstream utility. A central tension is improving the requested behavior while preserving the model’s general capabilities and diversity, with training cost, limited data, and reward hacking complicating that balance.

Latest papers

CardsList

181 papers32 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Nov 16, 2024

C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation

Synthetic Aperture Radar (SAR) imagery provides robust environmental and temporal coverage (e.g., during clouds, seasons, day-night cycles), yet its noise and unique structural patterns pose interpretation challenges, especially for non-experts. SAR-to-EO (Electro-Optical) image translation (SET) has emerged to make SAR images more perceptually interpretable. However, traditional approaches trained from scratch on limited SAR-EO datasets are prone to overfitting. To address these challenges, we introduce Confidence Diffusion for SAR-to-EO Translation, called C-DiffSET, a framework leveraging pretrained Latent Diffusion Model (LDM) extensively trained on natural images, thus enabling effective adaptation to the EO domain. Remarkably, we find that the pretrained VAE encoder aligns SAR and EO images in the same latent space, even with varying noise levels in SAR inputs. To further improve pixel-wise fidelity for SET, we propose a confidence-guided diffusion (C-Diff) loss that mitigates artifacts from temporal discrepancies, such as appearing or disappearing objects, thereby enhancing structural accuracy. C-DiffSET achieves state-of-the-art (SOTA) results on multiple datasets, significantly outperforming the very recent image-to-image translation methods and SET methods with large margins.

Jeonghyeok Do, Jaehyup Lee, Munchurl Kim

KAIST · KNU

Image-to-Image TranslationLatent Diffusion ModelsDiffusion Model Fine-TuningSynthetic Aperture Radar
AskPDF