cs.CVOct 17, 2026

Knowledge Diversion: A Novel Paradigm for Pre-training Decomposable Image Generation Models

Authors: Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, Xin Geng, Yong Rui

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large-scale pre-trained models have become foundational to modern machine learning, yet conventional pre-training objectives primarily optimize source-domain performance without explicitly considering downstream requirements. Consequently, transferable and task-specific knowledge remains entangled within a shared parameter space, limiting selective knowledge reuse and efficient downstream adaptation. This limitation is particularly consequential for image-generation models, which must accommodate diverse conditions and domains. To address this challenge, we introduce Knowledge Diversion (KDiv), a condition-aware pre-training paradigm for learning inherently decomposable generative models. Before pre-training, KDiv factorizes randomly initialized weights into rank-one Knowledge Units, which are organized into learngenes and tailors. During pre-training, continuous condition-aware routing consolidates knowledge shared across conditions in learngenes while directing condition-dependent variations to tailors. This structured organization enables learngenes to capture transferable generative priors and tailors to develop condition-specific expertise, allowing the resulting model to be selectively recombined for diverse downstream conditions. Extensive experiments across class-, text-, and image-conditioned generation demonstrate that KDiv consistently matches or surpasses dense baselines at comparable pre-training cost and achieves the strongest average downstream performance with a trainable parameter budget comparable to low-rank adaptation methods.

Figures & tables

Fig. 1: (a) Conventional pre-training entangles transferable and source-specific knowledge within a unified backbone, forcing limited downstream updates to both mitigate source-domain bias and capture target-specific variation. (b) KDiv separates condition-agnostic knowledge into shared learngenes and condition-specific knowledge into tailors during pre-training, enabling target models to recombine the shared prior with relevant specialized components. This structured recombination concentrates adaptation on target-specific knowledge and improves alignment with the target distribution.
Fig. 2: Overview of the KDiv framework. KDiv factorizes the principal weight matrices of each DiT block into rank-one Knowledge Units and partitions them into condition-agnostic learngenes and condition-specific tailors. Modality-specific encoders and continuous gating predict tailor modulation values, which combine the dynamically weighted tailors with the always-active learngenes to reconstruct condition-specific weights. This unified formulation supports class-, text-, and image-conditioned generation under the diffusion training objective.
Fig. 3: Qualitative comparison on novel downstream conditions. Representative samples generated by KDiv and competing PEFT methods for Class-Conditional Generation (C2I) , Text-Conditional Generation (T2I) , and Image-Conditional Generation (I2I) tasks. Additional qualitative comparisons are provided in Appendix Figs. 12 and 13 .
Class-Conditional Generation (C2I)
 CUB-200DF20MFlowers-102Food-101Cost
 FDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowGPU TimeParam.
SVDiff [ 22 ]19873.7751.59718786.3322.37820833.5101.58715943.8322.6900.92 h0.02M
OFT [ 54 ]11301.4520.6359773.2131.36616511.9921.17512902.9792.2171.26 h15.03M
LoRA [ 4 ]10951.3130.53510533.5441.38715751.6740.88912452.8352.0071.13 h11.80M
PiSSA [ 55 ]10511.2300.5319303.0891.26315961.7670.97312082.6971.9641.14 h11.80M
TABLE II: Performance of knowledge diversion in adapting to novel conditions. We compare KDiv with parameter-efficient adaptation methods on previously unseen downstream conditions. Performance is evaluated using FDD ( ↓\downarrow ), KDD ( ↓\downarrow ), and CMMD ( ↓\downarrow ) for C2I, and LPIPS ( ↓\downarrow ), SSIM ( ↑\uparrow ), and CLIP-T/CLIP-I ( ↑\uparrow ) for T2I/I2I, respectively. “GPU Time” denotes the adaptation cost, and “Param.” denotes the average number of trainable parameters.
Fig. 4: Downstream learning efficiency of KDiv. (a) Training loss curves across eight Text-Conditional Generation (T2I) datasets. (b) Qualitative evolution of Class-Conditional Generation (C2I) samples at 5K, 10K, 30K, and 50K training steps. Compared with representative PEFT baselines, KDiv exhibits faster convergence and produces high-quality samples at earlier adaptation stages. Additional results are provided in Appendix Fig. 14 .
Class-Conditional Generation (C2I)
   RoutingDownstream Adaptation
MethodLGTailorExclusiveOverlappingFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrow
Baseline    11192.6561.514
Only LG✓   10302.3821.392
KIND✓✓✓ 9832.2871.299
KDiv✓✓ ✓9662.2391.190
TABLE III: Ablation study of knowledge diversion for downstream adaptation. “LG” is the abbreviation for learngene, while “Exclusive” and “Overlapping” refer to the tailor-routing strategies adopted by KIND and KDiv, respectively. “N/A” denotes failure under the corresponding task setting.
Fig. 5: Emergent orthogonality among tailors. (a) Distributions of pairwise angular deviations from 90∘90^{\circ} across C2I, I2I, and T2I. (b) Fractions of tailor pairs satisfying different orthogonality thresholds. (c) Learned UU and VV matrices from representative FCin\mathrm{FC}_{\mathrm{in}} and FCout\mathrm{FC}_{\mathrm{out}} layers. Together, these results reveal consistently strong tailor orthogonality across conditioning modalities.
Fig. 6: Effect of tailor rank on downstream adaptation. Comparison of KDiv and LoRA at different ranks on T2I using LPIPS ↓\downarrow , SSIM ↑\uparrow , and CLIP-T ↑\uparrow . Curves report generation performance, while bars indicate active parameters. KDiv consistently achieves superior performance with fewer active parameters.
Fig. 7: Effect of the learngene allocation on generation performance. Relative improvements in FID and IS over dense training as the proportion of learngenes varies in C2I. A 25%25\% learngene ratio achieves the largest gains, whereas allocating all Knowledge Units to learngenes removes condition-specific routing and recovers the dense baseline.
Fig. 8: Condition relationships captured by the gating network. Pairwise cosine similarities between the tailor modulation vectors σTc\boldsymbol{\sigma}_{\mathcal{T}}^{c} predicted by Eq. ( 9 ) for conditions in (a) C2I, (b) T2I, and (c) I2I. The resulting similarity structures demonstrate that the gating network captures relationships among conditions, enabling the adaptive activation and reuse of relevant tailors.
Fig. 9: Visualization of KDiv with and without tailors (i.e., learngenes only) across 15 classes and three random seeds. Learngene-only generation exhibits semantic invariance, reflecting a condition-agnostic prior, whereas tailors inject condition-specific knowledge for semantically aligned generation. Additional qualitative results are provided in Appendix Fig. 10 .
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetUsageDomainNumber of Samples
Class-Conditional Generation (C2I) 
ImageNet-1K [ 49 ]Source Pre-trainingover 1,000 object categories1,281,167 train / 50,000 val
CUB-200-2011 [ 20 ]Downstream Adaptation200 fine-grained bird categories11,788 images
DF20M [ 21 ]Downstream Adaptation182 fungal species from six genera36,393 images
Flowers-102 [ 22 ]Downstream Adaptation102 flower categories8,189 images
Food-101 [ 23 ]Downstream Adaptation101 food categories101,000 images
Appendix
TABLE IV: Datasets used for pre-training and downstream adaptation.
Training SettingsClass-Conditional Gen.Text-Conditional Gen.Image-Conditional Gen.
OptimizerAdamWAdamWAdamW
Learning rate1×10−41{\times}10^{-4}1×10−41{\times}10^{-4}1×10−51{\times}10^{-5}
Weight decay00.030.03
Warmup steps01000
LR schedulerConstantConstant (400K) + cosine decay (100K)Constant
Batch size25625616
Appendix
TABLE V: Hyperparameter configurations for pre-training decomposable models with KDiv.
LevelHyperparameterClass-Conditional Gen.Text-Conditional Gen.Image-Conditional Gen.
Task LevelBatch size256324
Training steps50,0003,0003,000
OptimizerAdamWAdamWAdamW
Weight decay00.030.03
LR schedulerConstantConstantConstant
Warmup steps000
Appendix
TABLE VI: Hyperparameter configurations for downstream adaptation. Task-level training settings and method-specific configurations used to evaluate KDiv and competing PEFT methods across class-, text-, and image-conditioned generation.
Fig. 10: Additional visualization of condition-agnostic knowledge in learngenes. Learngene-only outputs exhibit no condition-specific semantics, reflecting the condition-agnostic knowledge encoded by learngenes, whereas incorporating tailors introduces the semantics required for condition-aligned generation.
Fig. 11: Zero-shot generalization to unseen image conditions. Qualitative comparison of the dense baseline and KDiv under unseen Inpainting , Lineart , and Illusion conditions without downstream adaptation. KDiv demonstrates stronger condition adherence and better preserves the semantic content and spatial structure of the reference images.
Fig. 12: Additional qualitative comparisons on downstream Class- and Text-Conditional Generation tasks.
Fig. 13: Additional qualitative comparisons on downstream Image-Conditional Generation tasks.
Fig. 14: Additional qualitative evolution of Class-Conditional Generation (C2I) samples at 10K-step training intervals.
More in this topic

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

Explore similar work

CardsList
May 22, 2026

Diffusion Domain Expansion: Learning to Coordinate Pre-trained Diffusion Models

In this paper, we propose Diffusion Domain Expansion (DDE), a method that efficiently extends pre-trained diffusion models to generate larger objects and handle more complex conditioning beyond their original capabilities. Our method employs a compact trainable network designed to coordinate the denoised outputs of pre-trained diffusion models. We demonstrate that the coordinator can be universally simple while being capable of generalizing to domains larger than those observed during its training time. We evaluate DDE on long audio track generation and conditional image generation, demonstrating its applicability across domains. DDE outperforms other approaches to coordinated generation with diffusion models in qualitative and quantitative evaluations.

Egor Lifar, Semyon Savkin, Timur Garipov +2

MIT CSAIL, Cambridge, USA

Diffusion-Based Image GenerationConditional Diffusion ModelsDiffusion Models
AskPDF
Jun 18, 2026

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion

We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature alignment through learnable flow fields and attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and memory consumption. Extensive experiments across multiple datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior memory efficiency. Code: https://github.com/arya-domain/UNITY

Aryan Das, Koushik Biswas, Moloud Abdar +1

VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1

Compositional Image GenerationConditional Diffusion ModelsDiffusion Model Inference AccelerationDiffusion Model Adaptation
AskPDF
Jan 18, 2026

Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation

Reinforcement learning (RL) has emerged as a paradigm for fine-tuning large-scale generative models, such as diffusion and flow models, to align with complex human preferences and user-specified tasks. A fundamental limitation remains the curse of diversity collapse, where the objective formulation and optimization landscape inherently collapse the policy to a Dirac delta distribution. To address this challenge, we propose DRIFT (DiveRsity-Incentivized Reinforcement Fine-Tuning for Versatile Image Generation), an innovative framework that systematically incentivizes output diversity throughout the on-policy fine-tuning process, reconciling strong task alignment with high generation diversity to enhance versatility essential for applications that demand diverse candidate generations. We approach the problem across three representative perspectives: i) sampling a reward-concentrated subset that filters out reward outliers to prevent premature collapse; ii) prompting with stochastic variations to expand the conditioning space, and iii) optimization of the intra-group diversity with a potential-based reward shaping mechanism. Experimental results show that DRIFT exhibits clear Pareto dominance in task alignment and generation diversity, achieving 3.88%∼\sim81.42% higher diversity at matched alignment and 14.38%∼\sim56.45% higher alignment at matched diversity. Our code is available at \textcolor{magenta}{https://github.com/NJU-RL/DRIFT}.

Jinmei Liu, Haoru Li, Zhenhong Sun +6

Nanjing University · Australian National University · Wuhan University +2

RL for Image GenerationBehavioral Diversity in RLDiffusion Model Fine-TuningDiversity-Aware Sampling
AskPDF