cs.LGNov 16, 2026

Specificity-Guided Knowledge Diversion for Decomposable Generative Models

Authors: Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, Yong Rui

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Abstract

Generative model pre-training typically optimizes a model as a whole without explicitly organizing its parameters for selective reuse. Decomposable pre-training instead learns reusable components for downstream adaptation while retaining strong generative performance. However, existing knowledge diversion methods incur substantial parameter overhead through weight factorization and rely on fixed component assignments to induce specialization, potentially compromising in-distribution generation quality. We therefore propose Specificity-Guided Knowledge Diversion (SGKDiv), which learns how generative knowledge is shared and specialized for effective generation and selective downstream reuse. SGKDiv constructs Knowledge Units from paired feed-forward channels, preserving native computational structure without SVD-based reparameterization. Calibrated condition-wise routing combines a shared usage prior with conditional adjustments, while complementary regularization shapes their allocation, diversity, and magnitude. Together, these designs replace predefined component assignments with adaptively learned and explicitly regulated knowledge specialization. Experiments across diverse model and training settings demonstrate better in-distribution generation than dense and MoE baselines, alongside stronger downstream adaptation than representative PEFT methods across eight datasets.

Figures & tables

Explore similar work

CardsList
  1. Knowledge Diversion: A Novel Paradigm for Pre-training Decomposable Image Generation Models

    Oct 17, 2026Yucheng Xie, Fu Feng, Ruixiao Shi +3Image GenerationConditional Generative Modeling

  2. Test-Time Instance-Specific Parameter Composition: A New Paradigm for Adaptive Generative Modeling

    Mar 29, 2026Minh-Tuan Tran, Xuan-May Le, Quan Hung Tran +3Implicit Generative ModelingGenerative Modeling

  3. TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

    Mar 25, 2026Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3Language Model DistillationCross-Architecture Knowledge Distillation