The Universal Weight Subspace Hypothesis

Prakhar Kaushik; Shravan Chaudhari; Ankit Vaidya; Rama Chellappa; Alan Yuille

cs.LGDec 4, 2025

The Universal Weight Subspace Hypothesis

Authors: Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, Alan Yuille

Organizations: Department of Computer Science Johns Hopkins University Baltimore, MD, USA

Abstract

We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1200 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.

Figures & tables

Figure 1 : Deep Networks Exhibit Shared, Low-Rank (Universal) Subspaces. Across distinct architectures and modalities, neural networks systematically learn to operate within remarkably similar low-dimensional parameter subspaces. Left: PCA of $\sim$ 200 GPT2, 500 ViTs, 50 LLaMA-8B, & 8 Flan-T5 models reveals consistent sharp spectral decay - strong evidence that a few weight directions capture dominant variance despite vast differences in training data, objectives, and initialization. The black baseline (independent subspaces reference) represents the naive expectation that models would learn distinct directions; our empirical findings contradict this. Right: Strikingly, 500 distinct ViT models exhibit a shared low-rank subspace across diverse datasets, training configurations, and domains. This emergent structure unlocks powerful applications: parameter-efficient adaptation, efficient model merging, compressed storage, and accelerated training and inference. Further discussion in Appendix A .

Figure 2 : Universal subspaces in CNNs. Decomposing 5 ResNet50 models trained on different tasks reveals a low-rank shared subspace, with most information concentrated in top fewer directions across layers.

Figure 3 : Lots of LoRA model size vs. performance.

Panel of Figure 4 — Figure 4 : Proving existence of universal subspaces in deep networks. Decomposing 500 LoRAs trained on different tasks using Mistral-7B reveals a low-rank universal subspace, with most information concentrated in 16 or fewer directions across all layers. Additional layer-wise plots are provided in Section B.5 .

Method	Style 1	Style 2	Style 3	Style 4	Style 5	Style 6	Style 7	Style 8	Style 9	Style 10	Avg
LoRA	21.95	15.59	22.18	18.84	16.65	17.99	24.66	17.47	22.07	19.93	19.73
Universal SDXL LoRA	21.96	16.07	22.07	18.79	16.68	17.99	24.66	17.56	22.46	20.09	19.83

Table 1 : CLIP scores (higher is better) of images generated using SDXL.

Figure 5 : Text-to-Image Generation Results for Individual models vs. Universal Subspace model. We observe no significant reduction in visual quality despite huge reduction in total model size.

Figure 6 : Universal Subspaces in Classical Weights. Spectral decomposition of weights from (1) $\sim$ 500 Vision Transformers, (2) 50 LLaMA-8B models, (3) 177 GPT-2 models, and (4) GLUE Flan-T5 models reveals a consistent low-rank structure, with most variance captured by the top few spectral bases. Despite diverse tasks, datasets, and training configurations, learned weights consistently align with a shared low-dimensional subspace. Only a subset of bases is shown; extended plots are provided in Section B.6 .

Method	IID	OOD
Full Training	94.4 $\pm$ 1.7	91.3 $\pm$ 2.1
Universal ViT	94.1 $\pm$ 2.0	87.8 $\pm$ 1.5

Table 2 : Image Classification

Method	Speedup	CoLA	MRPC	RTE	QNLI	SST-2	STS-B	Avg.
LoRA	$1\times$	59.56	86.76	77.61	92.53	94.72	90.81	83.67
Universal order-2	$2\times$	61.82	87.25	77.62	92.71	94.15	90.48	84.01
Universal order-3	$1.8\times$	62.06	86.52	75.81	92.98	94.26	90.39	83.67

Table 3 : Performance on the GLUE Benchmark.

	Params	CIFAR100	Food101	Flowers	CIFAR10	Pets
Full Training	86M	92.8	90.7	98.82	99.0	91.2
Universal ViT	10K	90.1	89.1	90.1	96.7	89.4

Table 4 : Image Classification with Vision Transformer.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7 : Empirical Evidence for (Universal) Joint Weight Subspaces. This figure illustrates the existence of joint low-dimensional subspaces across models trained on diverse tasks. We plot the average explained variance of the top few principal components of weight matrices from 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models. Despite differences in modality, data, and training objective, all models exhibit rapid spectral decay - indicating that a small number of directions dominate across layers and settings. This consistent structure provides strong evidence for the presence of joint/universal subspaces, supporting our hypothesis that deep networks systematically reuse a common representational basis. Often, this shared subspace can be seen distinctly. The presence of the subspace has significant implications for deep learning. Not only can large number of models be compressed into a single, lighter Universal model with difference represented as lightweight coefficients, training on future tasks simply becomes tuning those coefficients. Since the basis are fixed, training becomes simpler and quicker. However, this convergence to similar subspace raises few important questions - is it possible to recover the "true" Universal Subspace without learning with huge amounts of data? Is this lack of diversity a bottleneck from current family of deep models?

Notation	Description
$\mathcal{H}$	Separable Hilbert space with inner product $\langle\cdot,\cdot\rangle$ , norm $\\|\cdot\\|$ .
$a\otimes b$	Rank-one operator $g\mapsto\langle b,g\rangle a$ , $\\|a\otimes b\\|_{\mathrm{op}}=\\|a\\|\,\\|b\\|$ .
$T$	Number of tasks.
$\mathcal{T}$	Distribution over tasks.
$D_{t}$	Data distribution for task $t$ .
$S_{t}=\{(x_{t,i},y_{t,i})\}_{i=1}^{n_{t}}$	Dataset of size $n_{t}$ for task $t$ .

Appendix

Table 5 : Notation reference.

Figure 8 : Layer-wise suitability diagnostics for the randomly initialized and trained from scratch ResNet-50 models. The strict metric reports

q_{\mathrm{all}}=0.0183

reflecting the relatively poor quality of the universal space extractors due to the small number and diversity of models available. However, as our experiments have shown, our hypothesis still stands and and for universal space approximation still performs adequately.

Protocol	Variance Threshold	Original Acc.	Reconstructed Acc.
IID	60%	81.00	62.71
IID	70%	81.00	71.56
IID	80%	81.00	78.26
IID	85%	81.00	79.86
IID	90%	81.00	80.45
IID	95%	81.00	80.85

Appendix

Table 6 : Direct reconstruction of scratch-trained ResNet-50 models. Mean accuracy across five evaluated targets: ImageNet-1K, Places365, CIFAR-10, CIFAR-100, and SVHN. Models are reconstructed directly from the Universal Subspace without any task-specific fine-tuning. Variance Threshold denotes the fraction of spectral variance retained when constructing the subspace.

Protocol	Variance Threshold	Original Acc.	Reconstructed Acc.	Adapted Acc.	Coeff. Params
IID	60%	90.63	70.52	86.25	1.28M
IID	70%	90.63	80.98	87.63	1.64M
IID	80%	90.63	88.07	88.76	2.10M
OOD	60%	90.59	69.82	86.74	1.28M
OOD	70%	90.59	79.82	87.86	1.64M
OOD	80%	90.59	87.06	88.87	2.10M

Appendix

Table 7 : Lightweight coefficient adaptation within the Universal Subspace. Mean accuracy across CIFAR-10, CIFAR-100, and SVHN. IID optimizes only subspace coefficients; OOD additionally adapts the classifier head. Coefficient parameters exclude the classifier head.

Protocol	Original IS	Reconstructed IS	$\Delta$ IS	PSNR (dB)
IID	$3.208\pm 1.273$	$3.204\pm 1.167$	$-0.0041\pm 0.2620$	$19.98\pm 2.25$
OOD	$3.208\pm 1.273$	$3.204\pm 1.166$	$-0.0040\pm 0.2626$	$19.92\pm 2.28$

Appendix

Table 8 : StyleGAN2 reconstruction from the Universal Subspace. Mean results across 18 evaluated generators. IID includes the target model during basis construction, whereas OOD excludes it.

Figure 9 : StyleGAN2 samples from the original models (top) and their Universal Subspace reconstructions (bottom), using matched latent seeds. Each target model is excluded when estimating its reconstruction basis (OOD).

Model	GT CD	Pair CD
Original	$0.0506\pm 0.0070$	–
Universal reconstruction	$0.0867\pm 0.0802$	$0.0250\pm 0.0724$

Appendix

Table 9 : Geometry Distributions reconstruction from the Universal Subspace. Reconstruction uses a 95% explained-variance threshold with no fine-tuning. Chamfer distance (CD) is lower-is-better.

Figure 10 : Geometry Distributions samples from the original models and their Universal Subspace reconstructions (IID), shown in pairs for Loong and Archimedes. Colors encode coordinates, not predicted texture.

Figure 11 : Spectral analysis of the Mistral-7B-Instruct-v0.2 model: Aggregated eigenvalue (scree) plot across 500 LoRA models and all layers. The plot demonstrates that the majority of the variance is consistently captured by the top few principal directions, indicating the presence of a shared low-dimensional universal subspace.

Panel of Figure 12 — Figure 12 : Layerwise Eigenvalue Plots of 500 Mistral-7B-Instruct-v0.2 models. Each layer has 3 sets of parameters - $k\_proj,q\_proj,v\_proj$

Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task391	Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task290
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task442	Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task1598
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task039
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task076	Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task627
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task664	Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task819
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task1631
Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task190	Lots-of-LoRAs/Mistral-7B-Instruct-v0.2-4b-r16-task1391

Appendix

Table 10 : Models from HuggingFace for the Universal Mistral LoRA. Models in blue indicate the OOD models and the ones in red are the IID models used for evaluation.

alphonse-mucha-style	directors-coen-brothers-style	larry-carlson-style	rene-magritte-style
beeple-mike-winkelmann-style	director-sergei-eisenstein-style	lascaux	richard-corben-style
character-design	director-sofia-coppola-style	laurel-burch-style	richard-dadd-style
director-christopher-nolan-style	director-terrence-malick-style	lawrence-alma-tadema-style	richard-hescox-style
director-lars-von-trier-style	director-tim-burton-style	leonid-afremov-style	richard-scarry-style
director-ridley-scott-style	director-wes-anderson-style	leonora-carrington-style	robert-adams-style
director-stanley-kubrick-style	director-wong-kar-wai-style	levitating-cube	robert-crumb-style

Appendix

Table 11 : Models from HuggingFace used for the Universal Stable Diffusion-XL subspace extraction

Figure 13 : Spectral analysis of the Vision Transformer (ViT-base-patch16-224) model: Aggregated eigenvalue (scree) plot across $~500$ ViT models and all layers. The plot demonstrates that the majority of the variance is consistently captured by the top 16 principal directions, indicating the presence of a shared low-dimensional universal subspace.

Family	Models	Mean $\alpha$	$q_{\mathrm{all}}$	$q_{\mathrm{arch}}$	Retained Layers
ViT	464	4.7066	0.0518	0.3545	18 / 40
GPT-2	177	4.0951	0.0725	0.3059	23 / 49
LLaMA-3-8B	50	5.2391	0.0940	0.3290	116 / 224
Flan-T5 GLUE	196	3.9136	0.0224	0.4850	179 / 216

Appendix

Table 12 : Suitability diagnostics for classical-transformer model families. Retained layers satisfy the statistical and

q_{l}

criteria in Section B.1 .

Panel of Figure 14 — Figure 14 : Per-layer alpha diagnostics for classical-transformer families. Each violin aggregates valid raw-weight alpha estimates over all available models for a fixed layer; retained layers are summarized numerically in Table 12 .

Figure 15 : Layerwise Eigenvalue Plots of 500 ViT models.

0.50-200Train-100Test-vit-base	2025-01-21-16-13-04-vit-base-patch16-224
2025-02-05-14-22-36-vit-base-patch16-224	21BAI1229
Accomodation_room_classification	adam_VitB-p16-224-1e-4-batch_16_epoch_4_classes_24
age_face_detection_base	AIvisionGuard-v2
alea	amns
AnimeCharacterClassifierMark1	autotrain-48ci8-roib9
autotrain-8oqr6-image0807-20	autotrain-ap-pass-fail-v1

Appendix

Table 13 : Finetuned Models from HuggingFace used for the Universal Vision Transformer subspace extraction (vit-base-patch16-224)

Figure 16 : Spectral analysis of 50 LLaMA-3-8B model: Aggregated eigenvalue (scree) plot across $50$ LlaMa-8B models and all layers. The plot demonstrates that the majority of the variance is consistently captured by few top principal directions, indicating the presence of a shared low-dimensional universal subspace.

Figure 17 : Layerwise Scree Plots for 50 LLaMA-3-8B Models. For enhanced clarity, each subplot presents a truncated view of the total possible principal directions. These plots consistently demonstrate that the dominant information, as represented by explained variance, resides within a small number of leading principal directions for all models. Components beyond this initial set are characterized by eigenvalues approaching zero, signifying their redundancy for the universal subspace.

Meta-Llama-3-8B-Instruct-Jailbroken	Llama-3-13B-Instruct	large_crafting_sft_success	suzume-llama-3-8B-multilingual
summary-llama3-8b-f16-full	Llama-3-13B-Instruct-v0.1	Llama-3-8B-ProLong-64k-Base	LLaMAntino-3-ANITA-8B-Inst-DPO-ITA
ai-medical-model-32bit	filtered_crafting_train_data_shorter_length	Llama-3-portuguese-Tom-cat-8b-instruct	Llama-3-MAAL-8B-Instruct-v0.1
Human-Like-LLama3-8B-Instruct	LLaMA-3-8B-Instruct-TR-DPO	CabraLlama3-8b	chartgpt-llama3
KoLlama-3-8B-Instruct	honeypot-llama3-8B	Llama-SEA-LION-v2-8B	TR
Llama3-8B-Instruct-Turkish-Finetuned	Llama-3-15B-Instruct-zeroed	Llama-3-8B-Instruct-TAR-Bio-v2	Bio-Medical-Llama-3-8B
filtered_construction_train_data	shisa-v1-llama3-8b	REFUEL-Llama-3-Armo-iter_1	llama3-instrucTrans-enko-8b

Appendix

Table 14 : Models from HuggingFace used for the Universal LlaMa3-8B subspace extraction

openai-community/gpt2	pierreguillou/gpt2-small-portuguese
ai-forever/rugpt3small_based_on_gpt2	uer/gpt2-chinese-cluecorpussmall
axtonyao/gpt2-fp16-tflite	Xenova/gpt2
tensorblock/rakeshkiriyath_gpt2Medium_text_to_sql-GGUF	13on/gpt2-wishes
13onn/gpt2-wishes-2	13onnn/gpt2-wish
3koozy/gpt2-HxH	850886470/xxy_gpt2_chinese
AAli/gpt2-wikitext2	ARCYVILK/gpt2-bot

Appendix

Table 15 : Models from HuggingFace used for the GPT-2 subspace extraction

tanganke/flan-t5-base_glue-cola	tanganke/flan-t5-base_glue-mnli
tanganke/flan-t5-base_glue-mrpc	tanganke/flan-t5-base_glue-qnli
tanganke/flan-t5-base_glue-rte	tanganke/flan-t5-base_glue-qqp
tanganke/flan-t5-base_glue-sst2	tanganke/flan-t5-base_glue-stsb

Appendix

Table 16 : Finetuned Flan-T5 Models from HuggingFace used for the Universal Flan-T5 subspace extraction

Method	Model Number	Rouge-L Score
Normal Model	-	73.7
Universal model	50	55.8
Universal model	150	66.1
Universal model	250	71.9
Universal model	450	72.3

Appendix

Table 17 : Lots of LoRAs (Mistral-7B) OOD evaluation per increasing number of models used to extract Universal Subspace

Per-Task Absolute Accuracies (%)
Method	Datasets								Avg.
	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN
Finetuned	74.0	58.3	99.0	92.7	99.3	88.4	64.5	96.2	84.1
Combined-Model Accuracy Normalized by Finetuned Accuracy (%)
RegMean	80.2	71.3	37.9	47.3	43.1	70.5	93.9	43.0	60.9
TA	82.0	73.6	48.8	42.1	53.1	71.5	97.5	41.2	63.7

Appendix

Table 18 : Per-task results for eight ViT-B/32 models, each fine-tuned with LoRA on a different image-classification dataset. “Finetuned” reports the absolute accuracy of each task-specific model. For merging methods, we report per-task accuracy normalized by the corresponding fine-tuned model, together with the average across tasks.

Explore similar work

Sep 30, 2026cs.LG

Learning Functional Subspaces for Neural Network Compression

Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standard hardware. Existing methods, however, choose the subspace to remove from each weight matrix with local closed-form criteria: activation energy, layer-wise reconstruction error, or a quadratic approximation of the loss. These criteria ignore how errors propagate through the network, so at high compression the errors compound with depth and performance collapses. We introduce Learnable Subspace Projections (LSP), which instead learns the subspaces to discard end-to-end. Each linear layer, or tied group of layers that read the same activations, is assigned an orthogonal projector. All projectors are optimized jointly against a global objective--the KL divergence to the dense model's output distribution or the model's original training loss--while the pretrained weights remain frozen. Projectors are initialized from a whitened SVD truncation, and ranks are allocated by the output KL each projector induces per parameter saved. After training, the projectors merge into standard low-rank factors, with each tied group sharing one factor. In attention, this also lets the model cache one narrow latent in place of full keys and values. Across LLMs (OPT-125M/1.3B, Qwen3-4B, Llama-2-7B) and ViT-B/16, LSP outperforms baselines, and its advantage widens as compression increases. At -70% compression, LSP brings Llama-2-7B to 10.9 WikiText-2 perplexity and 42.2% mean zero-shot accuracy, versus 13.3 and 36.0% for the strongest baseline. The factorized model decodes up to 1.6x faster than the dense model at small batch sizes, and aching the shared latent shrinks the combined memory of weights and KV cache by 13.5x at a 128k-token context, versus at most 6.5x for untied baseline factorizations.

Massimo Bini, Anders Christensen, Stephan Alaniz +5

Helmholtz Munich · Technical University of Munich · MCML +6

Large Language Model Compression Subspace Low-Rank Structure Data Compression Methods Peak Memory

Ask PDF

May 11, 2026cs.LG

Flag Varieties: A Geometric Framework for Deep Network Alignment

Alignment, the tendency of adjacent weight matrices in deep networks to develop compatible subspace orientations, underlies gradient flow, Neural Collapse, and representation similarity across architectures. Despite extensive empirical documentation, these phenomena have resisted unified theoretical treatment: existing explanations are post-hoc, each fitted to a specific observation with whatever mathematics is at hand. We reverse this direction by deriving the mathematical structure that layerwise alignment inherently demands. Using geometric invariant theory, we prove that alignment geometry has a canonical closed, polystable stratum given by a flag variety, and that subspace intersection dimension is its unique reparameterization-invariant observable, establishing that subspace metrics are not empirical conventions but mathematical necessities. This unified framework yields two dynamical consequences: ridge regularization drives subspace alignment at an exponential rate set by weight decay, whereas nonlinear activations induce a commutator obstruction to exact basis alignment, generically present in nonlinear networks and absent in linear ones. Together these give a geometric explanation of the Level-2/3 hierarchy in Neural Collapse from first principles rather than post-hoc analysis. The commutator magnitude and head subspace overlap further serve as weight-space windows into internal alignment structure, requiring no forward passes. Experiments on multilayer perceptrons, residual networks, and pretrained language models support the proposed diagnostics and delineate their scope.

Jingchuan Xiao, Xinyi Sui, Cihan Ruan

Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA

Neural Collapse Subspace Gradient Similarity and Metrics

Ask PDF

Jun 22, 2026stat.ML

Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions

Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, a natural question is: can DNNs attain feature-learning and prediction consistency comparable to that of classical models? While a full characterization is open, we provide positive results for a broad subclass. We establish feature-learning consistency guarantees for sublinearly structured DNNs-architectures whose input/output dimensions and number of hidden neurons grow sublinearly with the sample size-when learning hierarchically compositional target functions. Importantly, this consistency still holds even in the conventional "over-parameterized" regime where the total number of parameters exceeds the number of training samples. Empirically, sublinearly structured DNNs match or surpass wide DNNs in prediction. A structural audit further indicates that widely used convolutional neural networks (CNNs), including AlexNet, VGGNet, ResNet, GoogLeNet, are sublinearly structured on their image classification benchmarks. We further prove that the sublinearly structured DNNs achieve universal approximation for hierarchically compositional functions in the large-sample limit. Moreover, images exhibit an inherent hierarchical, compositional structure. Taken together, these results explain, through a statistical lens, why many large-scale deep learning models succeed after adequate training on massive image datasets.

Sehwan Kim, Yan Sun, Faming Liang

Ewha Womans University, Seoul 03760, Republic of Korea · New Jersey Institute of Technology, Newark, NJ 07102, USA · Purdue University, West Lafayette, IN 47907, USA

Modern Deep Networks Neural Network Convolutional Neural Networks

Ask PDF