Universal Hypernetworks for Arbitrary Models

Xuanfeng Zhou

cs.LGApr 2, 2026

Universal Hypernetworks for Arbitrary Models

Authors: Xuanfeng Zhou

Organizations: Independent Researcher.

Abstract

Conventional hypernetworks are typically engineered around a specific base-model parameterization, so changing the target architecture often entails redesigning the hypernetwork and retraining it from scratch. We introduce the Universal Hypernetwork (UHN), a fixed-architecture generator that predicts weights from deterministic parameter, architecture, and task descriptors. This descriptor-based formulation decouples the generator architecture from target-network parameterization, so one generator can instantiate heterogeneous models across the tested architecture and task families. Our empirical claims are threefold: (1) one fixed UHN remains competitive with direct training across vision, graph, text, and formula-regression benchmarks; (2) the same UHN supports both multi-model generalization within a family and multi-task learning across heterogeneous models; and (3) UHN enables stable recursive generation with up to three intermediate generated UHNs before the final base model. Our code is available at https://github.com/Xuanfeng-Zhou/UHN.

Figures & tables

Figure 1: UHN settings: (a) single-model, UHN $H$ generates a single base model $f$ ; (b) multi-model, UHN $H$ generates a set of base models $\{f_{i}\}$ within one model family; (c) multi-task, UHN $H$ generates base models $f,g,q,\ldots$ across potentially heterogeneous architectures for different tasks; and (d) recursive generation, in which the root UHN $H_{0}$ generates an intermediate UHN $H_{1}$ , which then generates the base model $f$ .

Figure 2: Method overview. We sample the task and architecture of the base model $f_{\mathbf{w}}$ and collect their task-structure descriptors. For each parameter in $f_{\mathbf{w}}$ , we collect its index descriptor $\mathbf{v}_{i}$ and feed it into UHN $H_{\boldsymbol{\theta}}$ along with the task-structure descriptors to obtain each individual weight $w_{i}$ of $f$ . These weights are then stacked into the weight vector $\mathbf{w}$ , which parameterizes $f_{\mathbf{w}}$ .

Task type	Dataset	Model	#Params (Direct)	#Params (UHN)	Acc. (Direct)	Acc. (UHN)
Image	MNIST	MLP	118,282	158,613	$0.9837\pm 0.0007$	$0.9841\pm 0.0006$
Image	MNIST	CNN-8	74,762	158,613	$0.9938\pm 0.0004$	$0.9944\pm 0.0010$
Image	CIFAR-10	CNN-20	269,034	612,117	$0.8999\pm 0.0027$	$0.8993\pm 0.0016$
Graph	Cora	GCN	92,231	612,117	$0.8172\pm 0.0039$	$0.7950\pm 0.0069$
Graph	CiteSeer	GCN	237,446	612,117	$0.6996\pm 0.0051$	$0.6529\pm 0.0190$
Graph	PubMed	GCN	32,259	612,117	$0.7691\pm 0.0028$	$0.7815\pm 0.0075$

Table 1: Single-model universality results for classification tasks (image/graph/text). #Params denotes the number of trainable parameters; Acc. denotes test accuracy. Due to space constraints, formula-regression results are reported in Table 11 .

Model	Direct	HA	Chunked	UHN	Direct	HA	Chunked	UHN
	#Params				Acc.
CNN-20	269,034	619,306	625,480	612,117	$0.8999\pm 0.0027$	$0.9070\pm 0.0039$	$0.9010\pm 0.0008$	$0.8993\pm 0.0016$
CNN-32	463,018	635,546	627,592	612,117	$0.9069\pm 0.0028$	$0.9124\pm 0.0021$	$0.9010\pm 0.0002$	$0.9035\pm 0.0026$
CNN-44	657,002	651,786	629,640	612,117	$0.9076\pm 0.0010$	$0.9161\pm 0.0017$	$0.9063\pm 0.0030$	$0.9043\pm 0.0031$
CNN-56	850,986	668,026	631,688	612,117	$0.9043\pm 0.0036$	$0.9159\pm 0.0004$	$0.9027\pm 0.0021$	$0.9069\pm 0.0015$

Table 2: Single-model scalability results on CIFAR-10. #Params denotes the number of trainable parameters; Acc. denotes test accuracy.

Model family	#Models	Avg. #Params	Max #Params	UHN #Params	Seen	Unseen
	Model statistics				Acc.
CNN Mixed Depth	100	366,995	463,018	663,151	$0.8842\pm 0.0031$	$0.8430\pm 0.0023$
CNN Mixed Width	500	624,987	1,072,586	663,151	$0.9145\pm 0.0014$	$0.9145\pm 0.0013$
CNN Mixed Depth $\times$ Width	1000	734,206	1,372,110	663,151	$0.9038\pm 0.0012$	$0.9040\pm 0.0016$
Transformer Mixed	1000	523,650	1,086,212	663,151	$0.9063\pm 0.0004$	$0.9066\pm 0.0002$

Table 3: Multi-model results. #Models is the size of the model set

M

. Avg./Max #Params are the average/maximum number of trainable parameters over base models in

M

. UHN #Params is the number of trainable parameters of UHN. “Seen” averages test accuracy over

M^{\prime}_{\mathrm{train}}

and “Unseen” averages test accuracy over

M_{\mathrm{test}}

Task type	Dataset	Model	Perf. (Direct)	Perf. (UHN Single-task)	Perf. (UHN Multi-task)
Image	MNIST	MLP	$0.9837\pm 0.0007$	$0.9841\pm 0.0006$	$0.9786\pm 0.0012$
Image	CIFAR-10	CNN-44	$0.9076\pm 0.0010$	$0.9043\pm 0.0031$	$0.8927\pm 0.0011$
Graph	Cora	GCN	$0.8172\pm 0.0039$	$0.7950\pm 0.0069$	$0.7930\pm 0.0053$
Graph	PubMed	GAT	$0.7700\pm 0.0033$	$0.7801\pm 0.0037$	$0.7697\pm 0.0031$
Text	AG News	Transformer-2L	$0.9186\pm 0.0008$	$0.9099\pm 0.0005$	$0.9062\pm 0.0005$
Formula (RMSE)	kv	KAN-g5	$0.0211\pm 0.0126$	$0.0104\pm 0.0048$	$0.0172\pm 0.0112$

Table 4: Multi-task results. Perf. denotes test accuracy for classification (higher is better) and test RMSE for formula regression (lower is better). “UHN Single-task” denotes a separate UHN trained per task; “UHN Multi-task” denotes one shared UHN trained jointly on all tasks. Direct and UHN Single-task baselines are taken from Section 4.2.1 .

Setting	Acc.
Direct training	$0.9837\pm 0.0007$
UHN (no recursion)	$0.9841\pm 0.0006$
UHN rec. ( $K=1$ )	$0.9825\pm 0.0007$
UHN rec. ( $K=2$ )	$0.9795\pm 0.0011$
UHN rec. ( $K=3$ )	$0.9741\pm 0.0021$

Table 5: Recursive-task results on MNIST. “Direct training” and “UHN (no recursion)” are taken from the single-model MNIST MLP experiment ( Section 4.2.1 ). “UHN rec.” reports the accuracy of the leaf MLP generated via a recursion chain of depth

K

. Acc. denotes test accuracy.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Idx	Field	Description
0	layer_idx	Layer index.
1	layer_type	Layer type (categorical).
2	param_type	Parameter type within the layer (categorical).
3	out_idx	Output index / output channel / output embedding channel.
4	in_idx	Input index / input channel / input embedding channel.
5	kernel_h_idx	Convolution kernel height index ( $h$ coordinate).

Appendix

Table 6: Index descriptor layout

\mathbf{v}_{i}\in\mathbb{R}^{10}

Idx	Field	Description
0	model_type	Model family type (categorical).
1	num_layers	Number of layers.
2	cnn_stage_num	Number of stages (CNN).
3	num_encoders	Number of encoder blocks (Transformer ( Vaswani et al., 2017 ) ).
4	num_structure_freqs	Number of frequencies for structure/task Fourier features ( Tancik et al., 2020 ) (Recursive UHN).
5	num_index_freqs	Number of frequencies for index Fourier features (Recursive UHN).

Appendix

Table 7: Global structure descriptor layout

\mathbf{s}_{g}\in\mathbb{R}^{6}

Idx	Field	Description
0	layer_idx	Layer index.
1	layer_type	Layer type (categorical).
2	bias_type	Bias option (categorical).
3	norm_type	Normalization option (categorical).
4	shortcut_type	Shortcut ( He et al., 2016 ) option (categorical).
5	output_size	Output dimension / channels / embedding dim.

Appendix

Table 8: Local structure descriptor layout

\mathbf{s}_{\ell,j}\in\mathbb{R}^{21}

Idx	Field	Description
0	task_type	Task type (categorical).
1	dataset_type	Dataset type (categorical).

Appendix

Table 9: Task descriptor layout

\mathbf{t}\in\mathbb{R}^{2}

	Cora	CiteSeer	PubMed
Nodes	2708	3327	19717
Edges	5429	4732	44338
Features	1433	3703	500
Classes	7	6	3

Appendix

Table 10: Citation-network dataset statistics, as reported in ( Yang et al., 2016 ) , used in our experiments.

Function	Model (#Params)	RMSE (Direct)	RMSE (UHN)
Jacobian elliptic functions (ellipj)	KAN-g5 (407)	$6.78\times 10^{-4}\pm 1.32\times 10^{-4}$	$6.44\times 10^{-4}\pm 5.20\times 10^{-5}$
Incomplete elliptic integral of the first kind (ellipkinc)	KAN-g5 (254)	$5.37\times 10^{-4}\pm 1.40\times 10^{-4}$	$5.99\times 10^{-4}\pm 9.32\times 10^{-5}$
Incomplete elliptic integral of the second kind (ellipeinc)	KAN-g5 (254)	$6.02\times 10^{-4}\pm 2.58\times 10^{-4}$	$5.23\times 10^{-4}\pm 4.51\times 10^{-5}$
Bessel function of the first kind (jv)	KAN-g10 (364)	$1.80\times 10^{-3}\pm 8.04\times 10^{-4}$	$2.32\times 10^{-3}\pm 1.12\times 10^{-3}$
Bessel function of the second kind (yv)	KAN-g10 (364)	$1.09\times 10^{-2}\pm 4.05\times 10^{-3}$	$8.19\times 10^{-3}\pm 3.67\times 10^{-3}$
Modified Bessel function of the second kind (kv)	KAN-g5 (254)	$2.11\times 10^{-2}\pm 1.26\times 10^{-2}$	$1.04\times 10^{-2}\pm 4.77\times 10^{-3}$

Appendix

Table 11: Single-model universality results for formula regression on the 15 special functions following the KAN benchmark ( Liu et al., 2024 ) . Model (#Params) reports the base model architecture and its trainable parameter count; the trainable parameter count of UHN is fixed at 612,117 for all functions. RMSE denotes test root-mean-squared error (lower is better).

Task	Dataset	Model	$E_{0}$	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
Image	MNIST	MLP	100	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$
Image	MNIST	CNN-8	50	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{30,50,100\}$
Image	CIFAR-10	CNN-20	200	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{200,400,800\}$
Text	AG News	Transformer-2L	50	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{30,50,100\}$
Text	IMDB	Transformer-1L	100	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$
Graph	All	All	200	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{50,100,200\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{100,200,400\}$

Appendix

Table 12: UHN hyperparameter sweep grids (universality experiment). Table entries list the sweep-grid values for

\eta_{\mathrm{init}}

S_{\mathrm{init}}

\eta_{\mathrm{train}}

, and

E_{\mathrm{train}}

; we use “All” when the same grid applies to all datasets/models within that task.

Task	Dataset	Model	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
Image	MNIST	MLP	$2\mathrm{e}{-4}$	100	$2\mathrm{e}{-4}$	100
Image	MNIST	CNN-8	$2\mathrm{e}{-4}$	200	$1\mathrm{e}{-4}$	100
Image	CIFAR-10	CNN-20	0	0	$2\mathrm{e}{-4}$	800
Graph	Cora	GCN	$1\mathrm{e}{-4}$	200	$2\mathrm{e}{-5}$	400
Graph	CiteSeer	GCN	$1\mathrm{e}{-4}$	50	$1\mathrm{e}{-4}$	200
Graph	PubMed	GCN	$1\mathrm{e}{-4}$	200	$5\mathrm{e}{-5}$	200

Appendix

Table 13: UHN best hyperparameters selected by grid search (universality experiment).

Task	Dataset	Model	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
Image	MNIST	MLP	$\{0.002,0.005,0.01\}$	$\{50,100,200\}$
Image	MNIST	CNN-8	$\{0.005,0.01,0.02\}$	$\{30,50,100\}$
Image	CIFAR-10	CNN-20	$\{0.002,0.005,0.01\}$	$\{100,200,400\}$
Graph	Cora	GCN	$\{0.002,0.005,0.01\}$	$\{100,200,400\}$
Graph	CiteSeer	GCN/GAT	$\{0.001,0.002,0.005\}$	$\{100,200,400\}$
Graph	PubMed	GCN	$\{0.002,0.005,0.01\}$	$\{100,200,400\}$

Appendix

Table 14: Direct-training hyperparameter sweep grids (universality experiment). Table entries list the sweep-grid values for

\eta_{\mathrm{train}}

and

E_{\mathrm{train}}

Task	Dataset	Model	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
Image	MNIST	MLP	0.01	200
Image	MNIST	CNN-8	0.005	50
Image	CIFAR-10	CNN-20	0.005	400
Graph	Cora	GCN	0.01	100
Graph	CiteSeer	GCN	0.005	100
Graph	PubMed	GCN	0.005	200

Appendix

Table 15: Direct-training best hyperparameters selected by grid search (universality experiment).

Hypernetwork	$\eta_{\mathrm{train}}$ grid	$E_{\mathrm{train}}$ grid	Best $\eta_{\mathrm{train}}$	Best $E_{\mathrm{train}}$
HA	$\{0.002,0.005,0.01\}$	$\{200,400,800\}$	0.002	800
Chunked	$\{0.001,0.002,0.005\}$	$\{200,400,800\}$	0.002	800

Appendix

Table 16: Sweep grids and best hyperparameters for the hypernetwork baselines (scalability experiment).

Model family	$\|M\|$	$\|M_{\mathrm{train}}\|$	$\|M_{\mathrm{test}}\|$	$\|M_{\mathrm{val}}\|$	$\|M^{\prime}_{\mathrm{train}}\|$
CNN Mixed Depth	100	80	20	20	20
CNN Mixed Width	500	450	50	50	50
CNN Mixed Depth $\times$ Width	1000	950	50	50	50
Transformer Mixed	1000	950	50	50	50

Appendix

Table 17: Model-set splits sizes for the multi-model experiments.

Model family	$E_{0}$	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
CNN Mixed Depth	200	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{3200,6400,12800\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{800,1600,3200\}$
CNN Mixed Width	200	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{3200,6400,12800\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{800,1600,3200\}$
CNN Mixed Depth $\times$ Width	200	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{3200,6400,12800\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{800,1600,3200\}$
Transformer Mixed	50	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{2000,4000,8000\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{50,100,200\}$

Appendix

Table 18: Hyperparameter sweep grids for the multi-model experiments. Table entries list the sweep-grid values for

\eta_{\mathrm{init}}

S_{\mathrm{init}}

\eta_{\mathrm{train}}

, and

E_{\mathrm{train}}

Model family	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$E_{\mathrm{train}}$
CNN Mixed Depth	$2\mathrm{e}{-4}$	12800	$5\mathrm{e}{-5}$	3200
CNN Mixed Width	$1\mathrm{e}{-4}$	12800	$1\mathrm{e}{-4}$	3200
CNN Mixed Depth $\times$ Width	$2\mathrm{e}{-4}$	3200	$1\mathrm{e}{-4}$	3200
Transformer Mixed	$2\mathrm{e}{-4}$	8000	$5\mathrm{e}{-5}$	100

Appendix

Table 19: Best hyperparameters selected by grid search for the multi-model experiments.

Panel of Figure 3 — Figure 3: Model-family performance scatter plots.

Task	Dataset	Model	Guardrail (val)
Image	MNIST	MLP	0.95
Image	CIFAR-10	CNN-44	0.85
Graph	Cora	GCN	0.75
Graph	PubMed	GAT	0.75
Text	AG News	Transformer-2L	0.85
Formula	kv	KAN-g5	$5\mathrm{e}{-2}$

Appendix

Table 20: Task-specific validation guardrails used during hyperparameter selection in the multi-task setting.

	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$S_{\mathrm{train}}$
Grid ( $S_{0}=40000$ )	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{500,1000,2000\}$	$\{1\mathrm{e}{-5},2\mathrm{e}{-5},5\mathrm{e}{-5}\}$	$\{100000,200000,300000\}$
Selected	$1\mathrm{e}{-4}$	500	$2\mathrm{e}{-5}$	200000

Appendix

Table 21: Hyperparameter sweep grid and selected hyperparameters for multi-task training (subject to the guardrails in Table 20 ).

	$\eta_{\mathrm{init}}$	$S_{\mathrm{init}}$	$\eta_{\mathrm{train}}$	$S_{\mathrm{train}}$
Grid ( $S_{0}=30000$ )	$\{5\mathrm{e}{-5},1\mathrm{e}{-4},2\mathrm{e}{-4}\}$	$\{1000,2000,4000\}$	$\{2\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\}$	$\{15000,30000,60000\}$
Selected	$1\mathrm{e}{-4}$	4000	$2\mathrm{e}{-5}$	30000

Appendix

Table 22: Hyperparameter sweep grid and selected hyperparameters for the recursive setting with recursion depth

K=1

Encoding type	#Params	Acc.
Raw	68,629	$0.6642\pm 0.0199$
Positional	149,301	$0.8677\pm 0.0034$
GFF	612,117	$0.8993\pm 0.0016$

Appendix

Table 23: Index-encoding ablation results. #Params denotes the number of trainable parameters of UHN; Acc. denotes test accuracy.

Panel of Figure 4 — Figure 4: Visualization of the third convolution-layer kernels of the generated base model.

$F_{\mathbf{v}}$			$d$			$N_{\mathrm{blk}}$
Value	#Params	Acc.	Value	#Params	Acc.	Value	#Params	Acc.
256	135,445	$0.8894\pm 0.0020$	32	156,117	$0.8900\pm 0.0037$	0	545,045	$0.8901\pm 0.0028$
512	203,541	$0.8955\pm 0.0005$	64	299,925	$0.8960\pm 0.0015$	1	578,581	$0.8995\pm 0.0008$
1024	339,733	$0.8964\pm 0.0012$	128	612,117	$0.8993\pm 0.0016$	2	612,117	$0.8993\pm 0.0016$
2048	612,117	$0.8993\pm 0.0016$	256	1,334,805	$0.9019\pm 0.0041$	3	645,653	$0.9009\pm 0.0010$
4096	1,156,885	$0.9018\pm 0.0012$	–	–	–	–	–	–

Appendix

Table 24: Capacity ablation results. We vary one hyperparameter at a time: the number of Fourier-feature frequencies

F_{\mathbf{v}}

for index encoding (left), the UHN hidden dimension

d

(middle), and the number of UHN residual blocks

N_{\mathrm{blk}}

(right). Underlined entries indicate the single-model experiment setting ( Section 4.2 ). #Params denotes the number of trainable parameters of UHN; Acc. denotes test accuracy.

Model family	TSE	#Params	Seen Acc.	Unseen Acc.
CNN Mixed Width	w/	663,151	$0.9145\pm 0.0014$	$0.9145\pm 0.0013$
CNN Mixed Width	w/o	612,117	$0.9143\pm 0.0012$	$0.9141\pm 0.0013$
Transformer Mixed	w/	663,151	$0.9063\pm 0.0004$	$0.9066\pm 0.0002$
Transformer Mixed	w/o	612,117	$0.9073\pm 0.0021$	$0.9074\pm 0.0023$

Appendix

Table 25: Task-structure-encoder (TSE) ablation in the multi-model setting. “w/” and “w/o” indicate whether the task-structure encoder is enabled or disabled. #Params is the number of trainable parameters of UHN. “Seen Acc.” averages test accuracy over

M^{\prime}_{\mathrm{train}}

(the subset of training architectures used for hold-in evaluation) and “Unseen Acc.” averages test accuracy over

M_{\mathrm{test}}

(held-out test architectures).

Task	Dataset	Model	UHN perf. (multi-task, w/ TSE)	UHN perf. (multi-task, w/o TSE)
Image	MNIST	MLP	$0.9786\pm 0.0012$	$0.9783\pm 0.0011$
Image	CIFAR-10	CNN-44	$0.8927\pm 0.0011$	$0.8895\pm 0.0015$
Graph	Cora	GCN	$0.7930\pm 0.0053$	$0.7927\pm 0.0021$
Graph	PubMed	GAT	$0.7697\pm 0.0031$	$0.7703\pm 0.0042$
Text	AG News	Transformer-2L	$0.9062\pm 0.0005$	$0.9057\pm 0.0002$
Formula (RMSE)	kv	KAN-g5	$0.0172\pm 0.0112$	$0.0082\pm 0.0018$

Appendix

Table 26: Task-structure-encoder (TSE) ablation in the multi-task setting. “w/ TSE” and “w/o TSE” indicate whether the task-structure encoder is enabled or disabled. “Perf.” denotes test accuracy for classification (higher is better) and test RMSE for formula regression (lower is better).

Model family	init	Seen Acc.	Unseen Acc.
CNN Mixed Width	w/	$0.9145\pm 0.0014$	$0.9145\pm 0.0013$
CNN Mixed Width	w/o	$0.9130\pm 0.0011$	$0.9130\pm 0.0011$
Transformer Mixed	w/	$0.9063\pm 0.0004$	$0.9066\pm 0.0002$
Transformer Mixed	w/o	$0.8966\pm 0.0095$	$0.8963\pm 0.0096$

Appendix

Table 27: Initialization (init) ablation in the multi-model setting. “w/” and “w/o” indicate whether the initialization is enabled or disabled. “Seen Acc.” averages test accuracy over

M^{\prime}_{\mathrm{train}}

(the subset of training architectures used for hold-in evaluation) and “Unseen Acc.” averages test accuracy over

M_{\mathrm{test}}

(held-out test architectures).

Task	Dataset	Model	UHN perf. (multi-task, w/ init)	UHN perf. (multi-task, w/o init)
Image	MNIST	MLP	$0.9786\pm 0.0012$	$0.9780\pm 0.0016$
Image	CIFAR-10	CNN-44	$0.8927\pm 0.0011$	$0.8875\pm 0.0023$
Graph	Cora	GCN	$0.7930\pm 0.0053$	$0.7750\pm 0.0040$
Graph	PubMed	GAT	$0.7697\pm 0.0031$	$0.7690\pm 0.0017$
Text	AG News	Transformer-2L	$0.9062\pm 0.0005$	$0.9051\pm 0.0025$
Formula (RMSE)	kv	KAN-g5	$0.0172\pm 0.0112$	$0.0196\pm 0.0085$

Appendix

Table 28: Initialization (init) ablation in the multi-task setting. “w/ init” and “w/o init” indicate whether the initialization is enabled or disabled. “Perf.” denotes test accuracy for classification (higher is better) and test RMSE for formula regression (lower is better).

Figure 5: Multi-model Transformer Mixed epoch-averaged training loss with and without the initialization stage (“w/ init” and “w/o init”; log-scale

y

-axis). Loss is averaged over all training samples within each epoch.

Hypernetworks

Hypernetworks are neural networks that generate or modulate another model’s weights from task, input, or context information.

See all 62 papers

On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D
Sep 30, 2026Fabio J. Fehr, Philip TorrFew-Shot Learning Hypernetworks
University of Oxford
NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces
Feb 26, 2026Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu +3Diffusion Transformer Neural Network Generalization
Department of Electrical and Computer Engineering, Duke University, NC, USA · Department of Computer Science, Duke University, NC, USA · Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, USA
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
May 11, 2026Qi Cai, Jingwen Chen, Chengmin Gao +22Diffusion Transformer Text-Guided Image Editing

Sep 30, 2026J/K move · Enter open · S save

On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D

Fabio J. Fehr, Philip Torr

University of Oxford

Few-Shot Learning Hypernetworks

Open paper AskPDF arXiv

General-purpose models can adapt to many tasks from context, while specialised models can execute individual functions with less capacity. Yet obtaining such specialists requires task-specific training or adaptation. We ask whether they can instead be generated directly from a few demonstrations. Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context. The generated parameters form a structured weight space, while the resulting specialists show partial compositional generalisation and generalisation to transformations not seen during training. In both settings, removing explicit task identifiers improves generalisation beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters, and that the resulting weight space can support reuse and generalisation beyond known functions.

Abstract

Hypernetworks

Explore similar work

On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D

NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces

HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer