A Survey of Weight Space Learning: Understanding, Representation, and Generation
Authors: Xiaolong Han, Zehong Wang, Bo Zhao, Binchi Zhang, Jundong Li, Damian Borth, Rose Yu, Haggai Maron, +3 more
Organizations: University of Surrey · University of Notre Dame · University of California San Diego · University of Virginia · University of St.Gallen · Technion · Nvidia
Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.
Figures & tables
Figure 1: Overview of the survey. We organize weight space learning into three dimensions: understanding weight-space properties, learning compact representations, generating weights via auxiliary models. In addition, it consolidates applications to related domains and benchmarks, paving the way for future research.
Figure 2: Conceptual overview of weight space learning. (a) Weight space understanding analyzes symmetries in weight space and introduces quotient spaces to reduce redundancy and clarify functional equivalence. (b) Weight space representation learns compact embeddings of neural weights, discriminating symmetry in weight space well. (c) Weight space generation synthesizes network weights, supporting efficient model creation and adaptation.
Figure 4: Weight space symmetry and its induced functional manifestations. (a) illustrates the invariance under group action: applying a transformation ρin to weights θ in weight space does not alter the induced function in quotient space 2 2 2 The quotient space is defined as the set of equivalence classes of weights that parameterize the same function. Mathematically, let G be the symmetry group acting on the weight space W . The quotient space is defined as the orbit space W/G:={[θ]∣θ∈W} , where the equivalence class [θ]={g⋅θ∣g∈G} contains all weight configurations reachable via symmetry transformations. . (b) illustrates the equivariance under group action: transforming weights θ by ρin in weight space induces a corresponding transformation ρout in quotient space.
Functional Relation
Transformation Example
Functional Effect
Neuron permutation
Functionally identical network under reordered hidden units
Invariance
Positive scaling
Scaling is neutralized in batch normalization
Bias translation in Softmax
Constant shift of logits leaves probabilities unchanged
Orthogonal rotation
Rotating neuron basis with output changing accordingly
Equivariance
Sign flipping
Output changes predictably under the corresponding sign transformation
Table 1: Illustrative examples of functional invariance and equivariance induced by weight space symmetry. Under weight space symmetry transformations, invariance means that the realized function remains unchanged, while equivariance means that the realized function changes in a predictable, structured manner. These relations explain that many different parameter configurations correspond to the same or systematically related functions. See Zhao et al. [2025b] for a theoretical discussion.
Figure 5: Weight Space Representation Methods: Overview and Comparison. (a) Model-based approaches. All take raw weight space θ as input but differ in how they handle symmetry. Symmetry-agnostic methods learn representations directly from weight distributions without explicitly encoding neuron permutation symmetry. Symmetry-aware methods incorporate dedicated equivariant modules to respect permutation invariance. Graph-based symmetry approaches leverage GNNs to automatically capture structural symmetries among neurons. (b) Model-free approaches. Probe-based behavioral learning infers representations from functional responses to designed probes, enabling embeddings that encode behavioral properties and functional symmetries of the underlying network.
Figure 6: Weight Space Generation Methods: Overview and Comparison. (a) Hypernetworks. A conditioning prompt is processed by a hypernetwork H(⋅) to generate the parameters θ of a target model f(⋅) . The hypernetwork is trained end-to-end using the downstream task loss Ltask(fθ) , enabling fast and conditional weight generation. (b) Generative models. Taking a diffusion-based approach as an example, the process begins by sampling from a noise distribution in weight space. Through iterative denoising steps, the model progressively refines the noisy initialization toward a structured representation of network parameters. This forward–reverse diffusion mechanism learns the underlying distribution of weights and enables conditional generation, producing diverse and task-aligned neural network configurations without explicit gradient-based optimization.
Method
Conditioning
Diversity
Data Source
Suitable Scenarios
Hypernetworks
✓
×
real-world data
Few-shot and fast adaptation
Generative models
P
✓
model zoo
Large-scale and high-fidelity generation
Table 2: Comparison of Hypernetworks and generative models. ✓ indicates full satisfaction, × indicates the condition is not satisfied, and P denotes partial satisfaction.
Domain
Representative References
Why WSL Fits
Implicit Neural Representations (INR)
Shamsian et al. [2023] ; Shamsian et al. [2024] ; Huang et al. [2025] ; Zhou et al. [2023b] ; inr2vec [ Luigi et al., 2023 ] ; Functa [ Dupont et al., 2022a ] ; Spatial Functa [ Bauer et al., 2023 ] ; NeoMLP [ Kofinas et al., 2024b ] ; GRAF Schwarz et al. [2020] ; pi-GAN Chan et al. [2021] ; HyperDiffusion Erkoç et al. [2023] ; Yang and Wang [2024] ; Gielisse and van Gemert [2025] .
Data can be naturally represented by weights.
Model Unification
GEM [ Du et al., 2021 ] ; Functa [ Dupont et al., 2022a ] ; Spatial Functa [ Bauer et al., 2023 ] ; uGNN [ Pala and Rekik, 2025 ] .
Architecture-agnostic model space
Continual Learning
von Oswald et al. [2020] ; W-Diff [ Xie et al., 2024 ] ; Li et al. [2025] .
Forgetting mitigation via weight regeneration
Meta Learning
MetaDiff [ Zhang et al., 2024a ] ; Mc-Di [ Guan et al., 2025 ] ; MH [ Zhao et al., 2020 ] .
Fast gradient-free adaptation
Federated Learning
FedDiff [ Li et al., 2024a ] ; pFedHN [ Shamsian et al., 2021 ] ; pFedGPA [ Lai et al., 2025 ] .
Personalized weights, low communication
Neural Architecture Search
GHN [ Zhang et al., 2019 ] ; GHN-1/2 [ Knyazev et al., 2021 ] .
Efficient weights prediction
Table 3: Applications of Weight space learning to Related Domains.
Weight space learning aims to learn representations of neural network (NN) weights, enabling different downstream tasks. Existing approaches show promising performance, but lacking a way to shape these weight-space representations using information about the datasets the models were trained on, thus limiting downstream applications. We propose WeightCLIP, a method for learning a dataset-aligned latent space for neural networks, where datasets information is induced during training. The NNs are encoded as latent representations using an autoencoder, while dataset samples are encoded using a dataset encoder. The two representations are aligned using a contrastive objective, effectively reshaping the weight-space representations according to the datasets. We demonstrate that such representations can be used for different downstream tasks, including mapping dataset information to a weight-space representation that decode to strong models. In addition, we introduce a latent refinement process for generating models that outperforms standard fine-tuning. Overall, our results demonstrate that explicitly incorporating dataset information improves what can be achieved with weight-space representations across retrieval, generation, and refinement. Code will be available at https://github.com/HSG-AIML/WeightCLIP.
We investigate the potential of weights to serve as effective representations, focusing on neural fields. Our key insight is that constraining the optimization space through a pre-trained base model and low-rank adaptation (LoRA) can induce structure in weight space. Across reconstruction, generation, and analysis tasks on 2D and 3D data, we find that multiplicative LoRA weights achieve high representation quality while exhibiting distinctiveness and semantic structure. When used with latent diffusion models, multiplicative LoRA weights enable higher-quality generation than existing weight-space methods.
Neural network checkpoints have quietly become a large-scale data resource: millions of trained weight vectors now exist, each encoding task-, domain-, and architecture-specific knowledge. This position paper argues that model checkpoints should be treated as a first-class data modality, and that generative modeling in weight space should be standardized as a core machine learning primitive. Recent advances demonstrate that neural weights can be synthesized on demand, often matching fine-tuning performance while reducing adaptation cost by orders of magnitude. We contend that these results reflect an underlying structural fact: high-performing models occupy low-dimensional, highly structured regions of weight space shaped by symmetry, flatness, modularity, and shared subspaces. Building on this view, we organize existing methods into a five-stage pipeline, survey applications where the approach is already practical, and clarify current limits: adapter-scale and conditional generation are advancing rapidly, while unrestricted frontier-scale checkpoint synthesis remains open. Our goal is to shift the community's default mindset from optimizing models per task to sampling models from learned weight distributions, accelerating toward an era in which AI systems routinely improve or create other AI systems.