cs.CVOct 17, 2026

Knowledge Diversion: A Novel Paradigm for Pre-training Decomposable Image Generation Models

Authors: Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, Xin Geng, Yong Rui

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large-scale pre-trained models have become foundational to modern machine learning, yet conventional pre-training objectives primarily optimize source-domain performance without explicitly considering downstream requirements. Consequently, transferable and task-specific knowledge remains entangled within a shared parameter space, limiting selective knowledge reuse and efficient downstream adaptation. This limitation is particularly consequential for image-generation models, which must accommodate diverse conditions and domains. To address this challenge, we introduce Knowledge Diversion (KDiv), a condition-aware pre-training paradigm for learning inherently decomposable generative models. Before pre-training, KDiv factorizes randomly initialized weights into rank-one Knowledge Units, which are organized into learngenes and tailors. During pre-training, continuous condition-aware routing consolidates knowledge shared across conditions in learngenes while directing condition-dependent variations to tailors. This structured organization enables learngenes to capture transferable generative priors and tailors to develop condition-specific expertise, allowing the resulting model to be selectively recombined for diverse downstream conditions. Extensive experiments across class-, text-, and image-conditioned generation demonstrate that KDiv consistently matches or surpasses dense baselines at comparable pre-training cost and achieves the strongest average downstream performance with a trainable parameter budget comparable to low-rank adaptation methods.

Figures & tables

Fig. 1: (a) Conventional pre-training entangles transferable and source-specific knowledge within a unified backbone, forcing limited downstream updates to both mitigate source-domain bias and capture target-specific variation. (b) KDiv separates condition-agnostic knowledge into shared learngenes and condition-specific knowledge into tailors during pre-training, enabling target models to recombine the shared prior with relevant specialized components. This structured recombination concentrates adaptation on target-specific knowledge and improves alignment with the target distribution.
Fig. 2: Overview of the KDiv framework. KDiv factorizes the principal weight matrices of each DiT block into rank-one Knowledge Units and partitions them into condition-agnostic learngenes and condition-specific tailors. Modality-specific encoders and continuous gating predict tailor modulation values, which combine the dynamically weighted tailors with the always-active learngenes to reconstruct condition-specific weights. This unified formulation supports class-, text-, and image-conditioned generation under the diffusion training objective.
Fig. 3: Qualitative comparison on novel downstream conditions. Representative samples generated by KDiv and competing PEFT methods for Class-Conditional Generation (C2I) , Text-Conditional Generation (T2I) , and Image-Conditional Generation (I2I) tasks. Additional qualitative comparisons are provided in Appendix Figs. 12 and 13 .
Class-Conditional Generation (C2I)
 CUB-200DF20MFlowers-102Food-101Cost
 FDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrowGPU TimeParam.
SVDiff [ 22 ]19873.7751.59718786.3322.37820833.5101.58715943.8322.6900.92 h0.02M
OFT [ 54 ]11301.4520.6359773.2131.36616511.9921.17512902.9792.2171.26 h15.03M
LoRA [ 4 ]10951.3130.53510533.5441.38715751.6740.88912452.8352.0071.13 h11.80M
PiSSA [ 55 ]10511.2300.5319303.0891.26315961.7670.97312082.6971.9641.14 h11.80M
TABLE II: Performance of knowledge diversion in adapting to novel conditions. We compare KDiv with parameter-efficient adaptation methods on previously unseen downstream conditions. Performance is evaluated using FDD ( ↓\downarrow ), KDD ( ↓\downarrow ), and CMMD ( ↓\downarrow ) for C2I, and LPIPS ( ↓\downarrow ), SSIM ( ↑\uparrow ), and CLIP-T/CLIP-I ( ↑\uparrow ) for T2I/I2I, respectively. “GPU Time” denotes the adaptation cost, and “Param.” denotes the average number of trainable parameters.
Fig. 4: Downstream learning efficiency of KDiv. (a) Training loss curves across eight Text-Conditional Generation (T2I) datasets. (b) Qualitative evolution of Class-Conditional Generation (C2I) samples at 5K, 10K, 30K, and 50K training steps. Compared with representative PEFT baselines, KDiv exhibits faster convergence and produces high-quality samples at earlier adaptation stages. Additional results are provided in Appendix Fig. 14 .
Class-Conditional Generation (C2I)
   RoutingDownstream Adaptation
MethodLGTailorExclusiveOverlappingFDD ↓\downarrowKDD ↓\downarrowCMMD ↓\downarrow
Baseline    11192.6561.514
Only LG✓   10302.3821.392
KIND✓✓✓ 9832.2871.299
KDiv✓✓ ✓9662.2391.190
TABLE III: Ablation study of knowledge diversion for downstream adaptation. “LG” is the abbreviation for learngene, while “Exclusive” and “Overlapping” refer to the tailor-routing strategies adopted by KIND and KDiv, respectively. “N/A” denotes failure under the corresponding task setting.
Fig. 5: Emergent orthogonality among tailors. (a) Distributions of pairwise angular deviations from 90∘90^{\circ} across C2I, I2I, and T2I. (b) Fractions of tailor pairs satisfying different orthogonality thresholds. (c) Learned UU and VV matrices from representative FCin\mathrm{FC}_{\mathrm{in}} and FCout\mathrm{FC}_{\mathrm{out}} layers. Together, these results reveal consistently strong tailor orthogonality across conditioning modalities.
Fig. 6: Effect of tailor rank on downstream adaptation. Comparison of KDiv and LoRA at different ranks on T2I using LPIPS ↓\downarrow , SSIM ↑\uparrow , and CLIP-T ↑\uparrow . Curves report generation performance, while bars indicate active parameters. KDiv consistently achieves superior performance with fewer active parameters.
Fig. 7: Effect of the learngene allocation on generation performance. Relative improvements in FID and IS over dense training as the proportion of learngenes varies in C2I. A 25%25\% learngene ratio achieves the largest gains, whereas allocating all Knowledge Units to learngenes removes condition-specific routing and recovers the dense baseline.
Fig. 8: Condition relationships captured by the gating network. Pairwise cosine similarities between the tailor modulation vectors σTc\boldsymbol{\sigma}_{\mathcal{T}}^{c} predicted by Eq. ( 9 ) for conditions in (a) C2I, (b) T2I, and (c) I2I. The resulting similarity structures demonstrate that the gating network captures relationships among conditions, enabling the adaptive activation and reuse of relevant tailors.
Fig. 9: Visualization of KDiv with and without tailors (i.e., learngenes only) across 15 classes and three random seeds. Learngene-only generation exhibits semantic invariance, reflecting a condition-agnostic prior, whereas tailors inject condition-specific knowledge for semantically aligned generation. Additional qualitative results are provided in Appendix Fig. 10 .
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetUsageDomainNumber of Samples
Class-Conditional Generation (C2I) 
ImageNet-1K [ 49 ]Source Pre-trainingover 1,000 object categories1,281,167 train / 50,000 val
CUB-200-2011 [ 20 ]Downstream Adaptation200 fine-grained bird categories11,788 images
DF20M [ 21 ]Downstream Adaptation182 fungal species from six genera36,393 images
Flowers-102 [ 22 ]Downstream Adaptation102 flower categories8,189 images
Food-101 [ 23 ]Downstream Adaptation101 food categories101,000 images
Appendix
TABLE IV: Datasets used for pre-training and downstream adaptation.
Training SettingsClass-Conditional Gen.Text-Conditional Gen.Image-Conditional Gen.
OptimizerAdamWAdamWAdamW
Learning rate1×10−41{\times}10^{-4}1×10−41{\times}10^{-4}1×10−51{\times}10^{-5}
Weight decay00.030.03
Warmup steps01000
LR schedulerConstantConstant (400K) + cosine decay (100K)Constant
Batch size25625616
Appendix
TABLE V: Hyperparameter configurations for pre-training decomposable models with KDiv.
LevelHyperparameterClass-Conditional Gen.Text-Conditional Gen.Image-Conditional Gen.
Task LevelBatch size256324
Training steps50,0003,0003,000
OptimizerAdamWAdamWAdamW
Weight decay00.030.03
LR schedulerConstantConstantConstant
Warmup steps000
Appendix
TABLE VI: Hyperparameter configurations for downstream adaptation. Task-level training settings and method-specific configurations used to evaluate KDiv and competing PEFT methods across class-, text-, and image-conditioned generation.
Fig. 10: Additional visualization of condition-agnostic knowledge in learngenes. Learngene-only outputs exhibit no condition-specific semantics, reflecting the condition-agnostic knowledge encoded by learngenes, whereas incorporating tailors introduces the semantics required for condition-aligned generation.
Fig. 11: Zero-shot generalization to unseen image conditions. Qualitative comparison of the dense baseline and KDiv under unseen Inpainting , Lineart , and Illusion conditions without downstream adaptation. KDiv demonstrates stronger condition adherence and better preserves the semantic content and spatial structure of the reference images.
Fig. 12: Additional qualitative comparisons on downstream Class- and Text-Conditional Generation tasks.
Fig. 13: Additional qualitative comparisons on downstream Image-Conditional Generation tasks.
Fig. 14: Additional qualitative evolution of Class-Conditional Generation (C2I) samples at 10K-step training intervals.
More in this topic

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

Explore similar work

CardsList

  1. Diffusion Domain Expansion: Learning to Coordinate Pre-trained Diffusion Models

    May 22, 2026Egor Lifar, Semyon Savkin, Timur Garipov +2Diffusion-Based Image GenerationConditional Diffusion Models

    MIT CSAIL, Cambridge, USA

  2. UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion

    Jun 18, 2026Aryan Das, Koushik Biswas, Moloud Abdar +1Compositional Image GenerationConditional Diffusion Models

    VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1

  3. Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation

    Jan 18, 2026Jinmei Liu, Haoru Li, Zhenhong Sun +6RL for Image GenerationBehavioral Diversity in RL

    Nanjing University · Australian National University · Wuhan University +2