Diffusion Model Fine-Tuning

Ask about this topic

Overview

Diffusion model fine-tuning studies how to adapt pretrained image, video, audio, molecular, and language generators without rebuilding them from scratch. Parameter-efficient methods such as LoRA, prompt tuning, and adapters support personalization and domain adaptation, while supervised training and self-generated trajectories target tasks such as image editing, restoration, and masked-language-model refinement. For preference alignment and constrained generation, reinforcement learning methods such as GRPO, reward matching, and distillation optimize outputs using reward signals, often on benchmarks for text-image alignment, generation quality, or downstream utility. A central tension is improving the requested behavior while preserving the model’s general capabilities and diversity, with training cost, limited data, and reward hacking complicating that balance.

Latest papers

CardsList

181 papers32 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

  1. GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution

    Apr 28, 2026Fabio D'Oronzio, Federico Putamorsi, Leonardo Zini +2Diffusion Models for Image RestorationDiffusion Model Fine-Tuning

    University of Modena and Reggio Emilia, Italy

  2. Learning Illumination Control in Diffusion Models

    Apr 27, 2026Nishit Anand, Manan Suri, Christopher Metzler +2Diffusion Model Fine-TuningImage Relighting

    University of Maryland College Park

  3. Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing

    Apr 26, 2026Honghao Cai, Xiangyuan Wang, Yunhao Bai +8Diffusion TransformerText-Guided Image Editing

    The Chinese University of Hong Kong, Shenzhen · Xiaohongshu Inc. · Peking University +2

  4. DiffNR: Diffusion-Enhanced Neural Representation Optimization for Sparse-View 3D Tomographic Reconstruction

    Apr 23, 2026Shiyan Su, Ruyi Zha, Danli Shi +2Diffusion Models for Image RestorationSparse-View CT Reconstruction

    1Monash University · 2The Australian National University · 3Hong Kong Polytechnic University

  5. Optimizing Diffusion Priors in Image Reconstruction from a Single Observation

    Apr 22, 2026Frederic Wang, Katherine L. BoumanBayesian Inverse ProblemsImage Reconstruction

    Caltech, Pasadena CA 91125, USA

  6. Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks

    Apr 22, 2026Aljalila Aladawi, Mohammed Talha Alam, Fakhri KarrayDiffusion Model Fine-TuningDiffusion Model Unlearning

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE. · University of Waterloo, Ontario, Canada.

  7. Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation

    Apr 21, 2026Rui Li, Ke Hao, Yuanzhi Liang +4Multi-Objective Reinforcement LearningVideo Diffusion Models

    University of Science and Technology of China HeFei, China · Shanghai Jiao Tong University ShangHai, China · TeleAI ShangHai, China

  8. R2R^2-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction

    Apr 21, 2026Zhenbang Du, Kejing Xia, Xinrui Zhong +6Diffusion Language Model InferenceDiffusion Model Fine-Tuning

    Georgia Institute of Technology · NVIDIA

  9. Class-specific diffusion models improve military object detection in a low-data domain

    Apr 20, 2026Ella P. Fokkinga, Jan Erik van Woerden, Thijs A. Eker +4Synthetic Data AugmentationDiffusion Model Guidance

    TNO - Intelligent Imaging, Oude Waalsdorperweg 63, the Hague, the Netherlands

  10. Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

    Apr 19, 2026Jeongjae Lee, Jinho Chang, Jeongsol Kim +1Diffusion Model AlignmentDiffusion Model Fine-Tuning

    Graduate School of AI KAIST, Korea

  11. TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation

    Apr 18, 2026Xinran Liu, Diptesh Kanojia, Wenwu Wang +1Text-to-Motion GenerationCross-Modal Alignment

    University of Surrey, Guildford, Surrey, UK · 2Jiangnan University, Wuxi, Jiangsu, China

  12. LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories

    Apr 16, 2026Zhanhao Liang, Tao Yang, Jie Wu +2Diffusion Model AlignmentFlow Matching

    1The Australian National University · 2ByteDance Seed

  13. SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Models

    Apr 14, 2026You Qin, Linqing Wang, Hao Fei +4Diffusion Model AlignmentDiffusion Model Fine-Tuning

  14. OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models

    Apr 6, 2026Ali Aliev, Kamil Garifullin, Nikolay Yudin +5Model MergingDiffusion Model Adaptation

    HSE University · FusionBrain Lab · AXXX

  15. Unlocking Prompt Infilling Capability for Diffusion Language Models

    Apr 4, 2026Yoshinari Fujinuma, Keisuke SakaguchiDiffusion Language ModelsMasked Diffusion Models

    Patronus AI · Tohoku University, RIKEN

  16. IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

    Apr 1, 2026Linyan Dai, Xinwei Zhang, Haoyang Li +2T2I Diffusion ModelsImage Generation

    The Hong Kong Polytechnic University, Hong Kong, China

  17. Let Triggers Control: Frequency-Aware Dropout for Effective Token Control

    Mar 28, 2026Junyoung Koh, Hoyeon Moon, Dongha Kim +3Controllable Image GenerationDiffusion Model Fine-Tuning

    Yonsei University · Onoma AI

  18. MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generation

    Mar 28, 2026Xiaofeng Tan, Wanjiang Weng, Hongsong Wang +3Text-to-Motion GenerationHuman Motion Generation

    School of Computer Science and Engineering, Southeast University, Nanjing 211189, China, and also with Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · State Key Laboratory for Novel Software Technology, School of Intelligence Science and Technology, Nanjing University, Nanjing 210023, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), and also with School of Artificial Intelligence, University of Chinese Academy of Sciences

  19. ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images

    Mar 24, 2026Yunfeng Wu, Hongying Cheng, Zihao He +1High-Resolution Video GenerationVideo Generation

    School of Artificial Intelligence, Shanghai Jiao Tong University · Xi’an Jiaotong-Liverpool University · Jilin University

  20. Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation

    Mar 23, 2026Donald Shenaj, Federico Errica, Antonio CartaLow-Rank AdaptationAdaptive Rank Allocation

    University of Pisa · NEC Laboratories Europe

  21. MCLR: Improving Conditional Modeling via Inter-Class Likelihood-Ratio Maximization and Unifying Classifier-Free Guidance with Alignment Objectives

    Mar 23, 2026Xiang Li, Yixuan Jia, Xiao Li +3Conditional Diffusion ModelsDiffusion Model Alignment

    University of Michigan · Michigan State University

  22. Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows

    Mar 22, 2026Janne Perini, Rafael Bischof, Moab Arar +4Video Diffusion ModelsPDE Surrogate Modeling

    1Computational Design Lab, ETH Zurich, Switzerland · 2Tel Aviv University, Israel · 3Architecture and Building Systems, ETH Zurich, Switzerland +2

  23. Reward Sharpness-Aware Fine-Tuning for Diffusion Models

    Mar 22, 2026Kwanyoung Kim, Byeongsu SimDiffusion Model AlignmentRL for Diffusion Models

    Department of AI Convergence, GIST · Samsung Research

  24. Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skipping

    Mar 21, 2026Sunghyun Park, Jeongho Kim, Hyoungwoo Park +6Memory-Efficient Fine-TuningDiffusion Model Fine-Tuning

    Qualcomm AI Research · KAIST

  25. CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think

    Mar 19, 2026Zening Sun, Zhengpeng Xie, Lichen Bai +3Diffusion Model AlignmentDiffusion Model Fine-Tuning

    HKUST (GZ) · HIT (SZ)

  26. RadioDiff-FS: Physics-Informed Manifold Alignment in Few-Shot Diffusion Models for High-Fidelity Radio Map Construction

    Mar 19, 2026Xiucheng Wang, Zixuan Guo, Nan ChengRadio Propagation ModelingPhysics-Informed Diffusion Models

    State Key Laboratory of ISN and School of Telecommunications Engineering, Xidian University, Xi’an 710071, China · School of Physics, Xidian University, Xi’an 710071, China

  27. ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation

    Mar 18, 2026Dmitriy Rivkin, Parker Ewen, Lili Gao +5Truncated Backpropagation Through TimeVideo Diffusion Models

    Torc Robotics · Princeton University

  28. SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning

    Mar 18, 2026Xi Ye, Wenjia Yang, Yangyang Xu +4Diffusion Model AlignmentVideo Diffusion Models

    Tsinghua University · Ant Group · University of Chinese Academy of Sciences

  29. Diffusion Reinforcement Learning via Centered Reward Distillation

    Mar 14, 2026Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière +1RL for Diffusion ModelsDiffusion Model Alignment

    LIX, École Polytechnique, CNRS, IPP · Inria at Univ. Grenoble Alpes, CNRS, LJK

  30. IMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation

    Mar 14, 2026Chenru Wang, Yunyi Chen, Zijun Yang +2Diffusion-Based Dataset DistillationDataset Distillation

    AGI Lab, Westlake University · Eindhoven University of Technology · Jinan University +2