Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Generative model pre-training typically optimizes a model as a whole without explicitly organizing its parameters for selective reuse. Decomposable pre-training instead learns reusable components for downstream adaptation while retaining strong generative performance. However, existing knowledge diversion methods incur substantial parameter overhead through weight factorization and rely on fixed component assignments to induce specialization, potentially compromising in-distribution generation quality. We therefore propose Specificity-Guided Knowledge Diversion (SGKDiv), which learns how generative knowledge is shared and specialized for effective generation and selective downstream reuse. SGKDiv constructs Knowledge Units from paired feed-forward channels, preserving native computational structure without SVD-based reparameterization. Calibrated condition-wise routing combines a shared usage prior with conditional adjustments, while complementary regularization shapes their allocation, diversity, and magnitude. Together, these designs replace predefined component assignments with adaptively learned and explicitly regulated knowledge specialization. Experiments across diverse model and training settings demonstrate better in-distribution generation than dense and MoE baselines, alongside stronger downstream adaptation than representative PEFT methods across eight datasets.
Figures & tables
Figure 1: (a) Traditional knowledge diversion constructs SVD-based units with predefined shared–specific assignments, improving downstream adaptation (FDD) at the cost of degraded in-distribution generation (FID) and substantial parameter overhead. (b) SGKDiv constructs units from paired native FFN channels and learns their contributions through calibrated routing, improving both generation and adaptation with modest parameter overhead.
Figure 2: Overview of Specificity-Guided Knowledge Diversion (SGKDiv). (a) Paired FFN input columns and output rows form native Knowledge Units without matrix factorization. (b) Calibrated condition-specific routing combines a shared prior p with conditional logit adjustments qc to determine unit contributions. (c) Specificity-regularized optimization promotes polarized shared gates, diverse conditional deviations, and appropriate deviation magnitudes, enabling shared and specialized contributions to emerge during pre-training.
Model
Method
# Activated Param.
# Total Param.
cfg=1.0
cfg=1.5
FID ↓
IS ↑
Precision ↑
Recall ↑
FID ↓
IS ↑
Precision ↑
Recall ↑
DiT-S
Baseline [ 37 ]
32.96 M
32.96 M
57.845
24.731
0.408
0.605
33.645
47.460
0.538
0.567
KIND [ 61 ]
32.49 M
60.62 M
57.340
25.425
0.412
0.599
30.981
50.767
0.565
0.525
KDiv [ 60 ]
31.17 M
34.27 M
55.912
26.020
0.424
0.595
30.661
50.964
0.584
0.538
SGKDiv
30.13 M
34.59 M
54.092
28.231
0.425
0.606
27.439
59.718
0.593
0.526
DiT-B
Baseline [ 37 ]
130.51 M
130.51 M
33.584
44.693
0.537
0.639
12.960
101.200
0.714
0.544
Table 1: Comparison with dense and knowledge diversion methods. We evaluate class-conditional generation on ImageNet-1K at 256×256 resolution across DiT, DiT-Flow, REPA, and JiT configurations. We report activated and total parameters (M), alongside FID ↓ , IS ↑ , Precision ↑ , and Recall ↑ at CFG 1.0 and 1.5 . All models are trained for 800K updates.
Model
Method
# Activated Param.
# Total Param.
cfg=1.0
cfg=1.5
FID ↓
IS ↑
Precision ↑
Recall ↑
FID ↓
IS ↑
Precision ↑
Recall ↑
DiT-S
DiTMoE [ 10 ]
32.98 M
82.61 M
61.339
22.427
0.398
0.569
38.652
39.136
0.512
0.547
EC-DiT [ 51 ]
32.98 M
82.61 M
62.967
22.212
0.390
0.568
39.541
38.993
0.505
0.552
ProMoE-EC [ 56 ]
32.99 M
75.54 M
64.320
21.885
0.376
0.571
40.184
39.155
0.493
0.560
ProMoE-TC [ 56 ]
32.99 M
75.54 M
62.619
22.832
0.387
0.581
38.147
41.707
0.506
0.580
SGKDiv
31.79 M
34.59 M
59.783
24.598
0.395
0.584
33.070
48.850
0.546
0.536
Table 2: Comparison with mixture-of-experts methods. We compare SGKDiv with DiTMoE, EC-DiT, and ProMoE variants using expert-choice (EC) and token-choice (TC) routing on ImageNet-1K at 256×256 resolution. We report activated and total parameters (M), alongside FID ↓ , IS ↑ , Precision ↑ , and Recall ↑ at CFG 1.0 and 1.5 . All models are trained for 500K updates following the ProMoE protocol.
Figure 3: Qualitative comparison of downstream adaptation. Samples from novel target classes generated by SGKDiv and competing methods at 256×256 resolution with CFG 1.5 .
CUB-200
DF20M
Flowers-102
Food-101
Cost
FDD ↓
KDD ↓
CMMD ↓
FDD ↓
KDD ↓
CMMD ↓
FDD ↓
KDD ↓
CMMD ↓
FDD ↓
KDD ↓
CMMD ↓
GPU Time
Param.
OFT [ 40 ]
1130
1.452
0.635
977
3.213
1.366
1651
1.992
1.175
1290
2.979
2.217
1.26 h
15.03 M
LoRA [ 22 ]
1095
1.313
0.535
1053
3.544
1.387
1575
1.674
0.889
1245
2.835
2.007
1.13 h
11.80 M
PiSSA [ 33 ]
1051
1.230
0.531
930
3.089
1.263
1596
1.767
0.973
1208
2.697
1.964
1.14 h
11.80 M
LoHa [ 67 ]
1313
1.819
0.704
1262
4.274
1.663
1649
1.920
0.877
1349
3.107
2.216
1.85 h
11.80 M
DoRA [ 31 ]
1077
1.286
0.557
1037
3.516
1.377
1553
1.652
0.959
1245
2.865
1.992
1.38 h
11.82 M
Table 3: Performance of SGKDiv in downstream adaptation. We compare SGKDiv with PEFT and knowledge diversion methods on datasets using FDD ↓ , KDD ↓ , and CMMD ↓ . “GPU Time” and “Param.” denote adaptation time and trainable parameter count, respectively.
Figure 4: Compatibility of SGKDiv with model pruning. (a) FID of OBS-Diff with and without SGKDiv and (b) corresponding relative improvements across pruning ratios, evaluated before and after 50K fine-tuning steps.
Figure 5: Generation performance under varying classifier-free guidance (CFG). (a) FID and IS across CFG scales. (b) Precision–Recall trade-offs across the same CFG scales. SGKDiv improves quality over a broad guidance range while balancing fidelity and coverage.
Figure 6: Semantic structure of condition-specific knowledge allocation. (a) Loss increase under router-embedding replacements at different CLIP-based semantic ranks and random replacement. Semantically closer replacements incur smaller increases, indicating structured cross-condition reuse. (b) Recall of CLIP top-10 neighbors among the K nearest classes by routing-coefficient similarity.
Method
FID ↓
IS ↑
Precision ↑
Recall ↑
SGKDiv
54.092
28.231
0.425
0.606
w/o Annealing α
56.696
26.583
0.409
0.607
w/o Lpol
57.618
24.862
0.413
0.604
w/o Ldiv
58.514
24.314
0.410
0.604
w/o Lmag
55.516
27.138
0.419
0.605
w/o Learnable Gagn
58.492
24.471
0.408
0.596
Table 4: Ablation of routing and specialization regularization. We evaluate the learnable condition-agnostic gate, regularization losses, and polarization-guided annealing. “w/o Annealing” fixes α=1 , disabling annealing of the diversity and magnitude losses.
Large-scale pre-trained models have become foundational to modern machine learning, yet conventional pre-training objectives primarily optimize source-domain performance without explicitly considering downstream requirements. Consequently, transferable and task-specific knowledge remains entangled within a shared parameter space, limiting selective knowledge reuse and efficient downstream adaptation. This limitation is particularly consequential for image-generation models, which must accommodate diverse conditions and domains. To address this challenge, we introduce Knowledge Diversion (KDiv), a condition-aware pre-training paradigm for learning inherently decomposable generative models. Before pre-training, KDiv factorizes randomly initialized weights into rank-one Knowledge Units, which are organized into learngenes and tailors. During pre-training, continuous condition-aware routing consolidates knowledge shared across conditions in learngenes while directing condition-dependent variations to tailors. This structured organization enables learngenes to capture transferable generative priors and tailors to develop condition-specific expertise, allowing the resulting model to be selectively recombined for diverse downstream conditions. Extensive experiments across class-, text-, and image-conditioned generation demonstrate that KDiv consistently matches or surpasses dense baselines at comparable pre-training cost and achieves the strongest average downstream performance with a trainable parameter budget comparable to low-rank adaptation methods.
Yucheng Xie, Fu Feng, Ruixiao Shi +3
School of Computer Science and Engineering, Southeast University, Nanjing, China and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Existing generative models, such as diffusion and auto-regressive networks, are inherently static, relying on a fixed set of pretrained parameters to handle all inputs. In contrast, humans flexibly adapt their internal generative representations to each perceptual or imaginative context. Inspired by this capability, we introduce Composer, a new paradigm for adaptive generative modeling based on test-time instance-specific parameter composition. Composer generates input-conditioned parameter adaptations at inference time, which are injected into the pretrained model's weights, enabling per-input specialization without fine-tuning or retraining. Adaptation occurs once prior to multi-step generation, yielding higher-quality, context-aware outputs with minimal computational and memory overhead. Experiments show that Composer substantially improves performance across diverse generative models and use cases, including lightweight/quantized models and test-time scaling. By leveraging input-aware parameter composition, Composer establishes a new paradigm for designing generative models that dynamically adapt to each input, moving beyond static parameterization.
Minh-Tuan Tran, Xuan-May Le, Quan Hung Tran +3
Monash University · The University of Melbourne · Meta Inc
To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. We present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts similar to those that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets. It is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy than prior approaches, enabling ease of deployment and more effective transfer of the specialized knowledge.
Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3
Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada