Context-dependent manifold learning: A neuromodulated constrained autoencoder approach

Jérôme Adriaens; Gustave Bainier; Guillaume Drion; Pierre Sacré

cs.LGMar 12, 2026

Context-dependent manifold learning: A neuromodulated constrained autoencoder approach

Authors: Jérôme Adriaens, Gustave Bainier, Guillaume Drion, Pierre Sacré

Organizations: Neuroengineering Lab, Department of Electrical Engineering and Computer Science, University of Liège · University of Liège, Liège, Belgium

Abstract

Many physical systems exhibit a low-dimensional structure that varies with external parameters: link lengths in a robot, forcing constants in a fluid, or Reynolds numbers in a flow shift the underlying manifold while preserving its intrinsic dimension. Constrained AutoEncoders (cAEs) learn such manifolds through an idempotent encoder-decoder projection, a property that unconstrained autoencoders cannot match and that is essential whenever the model is applied iteratively. However, the standard strategies for making a cAE context-dependent, namely concatenating the context to the input or affinely modulating hidden activations, break the encoder-decoder idempotency, sacrificing the projection guarantee precisely in the setting where it would be most valuable. To restore this guarantee under context variation, we developed the Neuromodulated Constrained Autoencoder (NcAE), which modulates the activation slope and bias of a cAE through a context-driven hyper-network. This paper presents the NcAE, its theoretical foundation, and its empirical validation. We prove that for every context, including contexts unseen at training time, the reconstruction map remains an idempotent projection, the topology of the learned manifold is invariant, and context perturbations induce smooth changes in the manifold. We evaluated our approach on a 16-DoF pendulum with context-dependent coupling and the Lorenz96 system across a bifurcation. The NcAE matched or exceeded the best of six baselines on reconstruction, idempotency, and latent-geometry metrics, while being the only architecture that preserves geometric consistency by construction. The NcAE thereby provides a stable, geometry-preserving coordinate system across families of physical regimes.

Figures & tables

Panel of Figure 1 — Figure 1: Neuromodulation dynamically reconfigures the autoencoder manifold by parameterizing activation functions and biases based on external context. This schematic illustrates the integration of the context vector $\boldsymbol{c}$ , which a fully connected MLP processes to generate the modulation signal $\boldsymbol{s}$ . This signal is then multiplied by a layer-specific weight matrix $\boldsymbol{W}_{l,\alpha}$ and passed through the saturating function $g$ to compute the activation parameters $\boldsymbol{\alpha}^{(l)}$ . Simultaneously, $\boldsymbol{s}$ passes through a linear layer defined by parameters $\boldsymbol{W}_{l,\beta}$ and $\boldsymbol{b}_{l}$ to determine the layer-wise bias $\boldsymbol{\beta}^{(l)}$ .

	Pendulum		Lorenz96
Method	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$
cAE	$0.049{\pm 0.001}$	$0.220{\pm 0.002}$	$0.468{\pm 0.25}$	$0.653{\pm 0.1}$
Context-cAE	$0.054{\pm 0.001}$	$0.226{\pm 0.001}$	$0.319{\pm 0.254}$	$0.644{\pm 0.204}$
AE	$0.050{\pm 0.001}$	$2.252{\pm 3.506}$	$0.177{\pm 0.090}$	$18.634{\pm 8.044}$
Context-AE	$0.051{\pm 0.002}$	$0.883{\pm 0.803}$	$0.237{\pm 0.046}$	$11.137{\pm 2.797}$
FiLM+AE	$\mathbf{0.008{\pm 0.01}}$	$0.348{\pm 0.754}$	$0.323{\pm 0.073}$	$2.825{\pm 2.676}$

Table 1: NcAE achieves the best overall reconstruction across both systems and both metrics. FiLM+AE matches NcAE on pendulum position but with five times the variance; unconstrained autoencoders collapse on velocity in the bifurcating regime. RMSE reported as mean

\pm

std over 10 seeds.

Panel of Figure 2 — Figure 2: NcAE and cAE preserve idempotency by construction (error $\approx 10^{-5}$ at every iteration); context-aware baselines drift four to six orders of magnitude higher, and FiLM+AE diverges by step 10 on Lorenz96. Idempotency error $\|P^{k}(\boldsymbol{x})-P^{k-1}(\boldsymbol{x})\|$ on log scale, mean $\pm$ std over 10 seeds; numerical values in Tables 23 and 24 . † FiLM+AE diverged (numerical blow-up) at $k{=}10$ and $k{=}15$ ; points omitted.

	Pendulum		Lorenz96
Method	CV $\downarrow$	CN $\downarrow$	CV $\downarrow$	CN $\downarrow$
cAE	$0.50_{\pm 0.12}$	$248.15_{\pm 172.52}$	$0.72_{\pm 0.09}$	$1.87_{\pm 0.51}$
Context-cAE	$0.46_{\pm 0.07}$	$157.68_{\pm 122.44}$	$0.63_{\pm 0.13}$	$1.93_{\pm 0.87}$
AE	$2.51_{\pm 4.02}$	$49.01_{\pm 36.77}$	$1.35_{\pm 0.54}$	$2.97_{\pm 2.57}$
Context-AE	$0.70_{\pm 0.31}$	$148.14_{\pm 143.58}$	$0.72_{\pm 0.07}$	$2.41_{\pm 2.62}$
FiLM+AE	$0.51_{\pm 0.30}$	$39.85_{\pm 50.07}$	$0.85_{\pm 0.35}$	$3.78_{\pm 2.14}$

Table 2: NcAE produces the most isotropic latent axes on the pendulum (

\text{CN}\approx 17

, against

\geq 40

for every baseline) and the most homogeneous projected dynamics on Lorenz96 (

\text{CV}\approx 0.22

, against

\geq 0.41

for every baseline). CV: coefficient of variation of latent velocity norm (lower = more uniform dynamics). CN: condition number of latent covariance (lower = more isotropic axes).

	Pendulum		Lorenz96
Variant	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$
Bias-only NcAE	$0.015_{\pm 0.002}$	$0.067_{\pm 0.005}$	$0.076_{\pm 0.083}$	$0.277_{\pm 0.221}$
$\alpha$ -only NcAE*	$0.018_{\pm 0.002}$	$0.071_{\pm 0.004}$	$0.089_{\pm 0.106}$	$0.322_{\pm 0.374}$
NcAE (both)	$\mathbf{0.012_{\pm 0.002}}$	$\mathbf{0.059_{\pm 0.003}}$	$\mathbf{0.046_{\pm 0.032}}$	$\mathbf{0.209_{\pm 0.148}}$

Table 3: Each modulation channel alone enables context adaptation in mean, but slope-only training is unstable (3/10 pendulum and 2/10 Lorenz96 runs diverged and were excluded). Bias modulation stabilizes training, slope modulation captures the nonlinear deformation, and the combination yields the lowest error. RMSE reported as mean

\pm

std over 10 seeds. * denotes that 2 runs of 10 diverged for Lorenz and 3 out of 10 for the pendulum, and were removed from the statistics computation.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Activation functions $\sigma_{\pm}$ for different values of $\alpha$ .

Panel of Figure 4 — Figure 4: Activation functions $\sigma_{\pm}$ for $\alpha$ near $\frac{\pi}{4}$ .

Table 4: Coupling functions.

Hyperparameter	cAE	Context-cAE	NcAE
Main architecture
Layer sizes ( $n_{l}$ )	$[8,16,16,16]$	$[8,12,14,16]$
Latent dimension ( $d$ )	4
Activation parameter $\alpha$	$\pi/8$	$\pi/10$	N/A
Contextual mechanism
Mechanism	None	Concatenation	Neuromodulation

Appendix

Table 5: Hyperparameters for the 16-DoF pendulum — constrained architectures (cAE, Context-cAE, NcAE).

Hyperparameter	AE	Context-AE	FiLM+AE	SoftNcAE
Main architecture
Encoder layers	$[16,16,16,8,4]$	$[20,16,16,8,4]^{\dagger}$	$[16,16,16,8,4]$	$[16,14,12,8,4]$
Decoder layers	$[4,8,16,16,16]$	$[4,8,16,16,16]$	$[4,8,16,16,16]$	$[4,8,12,14,16]$
Latent dimension ( $d$ )	4
Activation function	ReLU	ReLU	SiLU	$\sigma_{\pm}$
Contextual mechanism

Appendix

Table 6: Hyperparameters for the 16-DoF pendulum — unconstrained baselines (AE, Context-AE, FiLM+AE, SoftNcAE).

\dagger

Context-AE input dimension is

20=16+4

due to context concatenation.

Panel of Figure 5 — Figure 5: Hovmöller diagrams of the Lorenz96 system for values of $F$ before and after the bifurcation.

Hyperparameter	cAE	Context-cAE	NcAE
Main architecture
Layer sizes ( $n_{l}$ )	$[21,36]$	$[20,36]$	$[18,36]$
Latent dimension ( $d$ )	2
Activation parameter $\alpha$	$\pi/8$	$\pi/10$	N/A
Contextual mechanism
Mechanism	None	Concatenation	Neuromodulation

Appendix

Table 7: Hyperparameters for the Lorenz96 experiment — constrained architectures (cAE, Context-cAE, NcAE).

Hyperparameter	AE	Context-AE	FiLM+AE	SoftNcAE
Main architecture
Encoder layers	$[36,24,12,2]$	$[37,20,2]^{\dagger}$	$[36,24,12,2]$	$[36,18,2]$
Decoder layers	$[2,12,24,36]$	$[2,12,24,36]$	$[2,12,24,36]$	$[2,18,36]$
Latent dimension ( $d$ )	2
Activation function	ReLU	ReLU	SiLU	$\sigma_{\pm}$
Contextual mechanism

Appendix

Table 8: Hyperparameters for the Lorenz96 experiment — unconstrained baselines (AE, Context-AE, FiLM+AE, SoftNcAE).

\dagger

Context-AE input dimension is

37=36+1

due to context concatenation.

	OOD +10%		OOD +20%		OOD +50%
Model	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$
cAE	$0.467_{\pm 0.250}$	$0.649_{\pm 0.099}$	$0.468_{\pm 0.250}$	$0.650_{\pm 0.097}$	$0.472_{\pm 0.250}$	$0.659_{\pm 0.096}$
Context-cAE	$0.315_{\pm 0.251}$	$0.646_{\pm 0.197}$	$0.318_{\pm 0.252}$	$0.654_{\pm 0.197}$	$0.326_{\pm 0.253}$	$0.689_{\pm 0.195}$
AE	$0.176_{\pm 0.091}$	$16.65_{\pm 6.327}$	$0.157_{\pm 0.087}$	$12.38_{\pm 5.653}$	$\mathbf{0.177_{\pm 0.092}}$	$18.09_{\pm 7.681}$
Context-AE	$0.237_{\pm 0.051}$	$10.81_{\pm 3.046}$	$0.234_{\pm 0.049}$	$10.32_{\pm 2.658}$	$0.251_{\pm 0.048}$	$10.72_{\pm 2.887}$
FiLM+AE	$0.928_{\pm 0.420}$	$5.989_{\pm 2.940}$	$2.594_{\pm 2.212}$	$14.87_{\pm 12.06}$	$43.66_{\pm 52.39}$	$273.9_{\pm 473.7}$

Appendix

Table 9: Out-of-distribution context performance on Lorenz96 (mean

\pm

std RMSE). OOD distance is expressed as a percentage beyond the training boundary. The projection remains geometrically valid by construction ( Proposition 3.3 ); Only reconstruction fidelity degrades with distance.

	OOD +10%		OOD +20%		OOD +50%
Model	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$	Pos. RMSE $\downarrow$	Vel. RMSE $\downarrow$
cAE	$0.079_{\pm 0.002}$	$0.499_{\pm 0.021}$	$0.099_{\pm 0.003}$	$0.634_{\pm 0.041}$	$0.117_{\pm 0.003}$	$0.794_{\pm 0.055}$
Context-cAE	$0.087_{\pm 0.001}$	$0.495_{\pm 0.006}$	$0.108_{\pm 0.002}$	$0.625_{\pm 0.013}$	$0.131_{\pm 0.005}$	$0.803_{\pm 0.029}$
AE	$0.082_{\pm 0.004}$	$2.919_{\pm 3.836}$	$0.103_{\pm 0.004}$	$3.708_{\pm 4.885}$	$0.121_{\pm 0.006}$	$4.252_{\pm 5.724}$
Context-AE	$0.084_{\pm 0.004}$	$1.226_{\pm 0.811}$	$0.105_{\pm 0.005}$	$1.458_{\pm 0.947}$	$0.124_{\pm 0.007}$	$1.587_{\pm 0.843}$
FiLM+AE	$\mathbf{0.021_{\pm 0.029}}$	$0.518_{\pm 0.935}$	$\mathbf{0.033_{\pm 0.053}}$	$0.656_{\pm 1.136}$	$0.079_{\pm 0.156}$	$0.777_{\pm 0.958}$

Appendix

Table 10: Out-of-distribution context performance on the pendulum (mean

\pm

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
cAE	$0.336_{\pm 0.154}$	$0.613_{\pm 0.075}$	$0.534_{\pm 0.251}$	$0.908_{\pm 0.228}$	$0.526_{\pm 0.288}$	$1.179_{\pm 0.288}$
Context-cAE	$0.453_{\pm 0.252}$	$0.745_{\pm 0.128}$	$0.499_{\pm 0.267}$	$0.951_{\pm 0.253}$	$0.409_{\pm 0.238}$	$1.082_{\pm 0.191}$
AE	$0.165_{\pm 0.061}$	$21.077_{\pm 7.938}$	$0.179_{\pm 0.045}$	$17.901_{\pm 8.541}$	$0.260_{\pm 0.048}$	$16.748_{\pm 4.014}$
Context-AE	$0.260_{\pm 0.046}$	$10.125_{\pm 2.014}$	$0.300_{\pm 0.053}$	$9.014_{\pm 1.829}$	$0.346_{\pm 0.054}$	$8.221_{\pm 1.912}$
FiLM+AE	$0.327_{\pm 0.072}$	$2.792_{\pm 2.610}$	$0.335_{\pm 0.060}$	$2.633_{\pm 2.187}$	$0.379_{\pm 0.054}$	$2.674_{\pm 1.850}$

Appendix

Table 11: Lorenz96 — state noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

is added to states at training time.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.136_{\pm 0.063}$	$0.463_{\pm 0.064}$	$\mathbf{0.166_{\pm 0.126}}$	$0.498_{\pm 0.115}$	$0.230_{\pm 0.200}$	$0.603_{\pm 0.267}$
Context-AE	$0.240_{\pm 0.039}$	$11.363_{\pm 2.372}$	$0.239_{\pm 0.038}$	$11.714_{\pm 2.510}$	$0.240_{\pm 0.039}$	$11.663_{\pm 2.381}$
FiLM+AE	$0.329_{\pm 0.037}$	$2.897_{\pm 2.065}$	$0.338_{\pm 0.032}$	$3.207_{\pm 2.004}$	$0.395_{\pm 0.033}$	$3.893_{\pm 2.980}$
SoftNcAE	$0.844_{\pm 0.111}$	$2.265_{\pm 0.299}$	$0.986_{\pm 0.113}$	$2.575_{\pm 0.379}$	$1.003_{\pm 0.121}$	$2.609_{\pm 0.448}$
NcAE (ours)	$\mathbf{0.083_{\pm 0.013}}$	$\mathbf{0.319_{\pm 0.031}}$	$0.170_{\pm 0.035}$	$\mathbf{0.400_{\pm 0.041}}$	$\mathbf{0.194_{\pm 0.093}}$	$\mathbf{0.465_{\pm 0.094}}$

Appendix

Table 12: Lorenz96 — context noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to context at training time. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.222_{\pm 0.087}$	$0.516_{\pm 0.074}$	$0.403_{\pm 0.228}$	$0.842_{\pm 0.293}$	$0.449_{\pm 0.235}$	$0.982_{\pm 0.180}$
Context-AE	$0.263_{\pm 0.048}$	$9.966_{\pm 1.854}$	$0.298_{\pm 0.056}$	$8.990_{\pm 1.793}$	$0.346_{\pm 0.057}$	$8.289_{\pm 2.000}$
FiLM+AE	$0.334_{\pm 0.036}$	$2.363_{\pm 0.853}$	$0.368_{\pm 0.034}$	$3.417_{\pm 2.000}$	$0.436_{\pm 0.038}$	$4.144_{\pm 3.268}$
SoftNcAE	$0.876_{\pm 0.143}$	$2.342_{\pm 0.414}$	$0.972_{\pm 0.104}$	$2.605_{\pm 0.307}$	$1.000_{\pm 0.078}$	$2.684_{\pm 0.296}$
NcAE (ours)	$\mathbf{0.088_{\pm 0.021}}$	$\mathbf{0.362_{\pm 0.043}}$	$\mathbf{0.193_{\pm 0.047}}$	$\mathbf{0.606_{\pm 0.064}}$	$\mathbf{0.209_{\pm 0.051}}$	$\mathbf{0.766_{\pm 0.051}}$

Appendix

Table 13: Lorenz96 — state and context noise (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to both states and context at training time. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
cAE	$0.047_{\pm 0.001}$	$0.210_{\pm 0.002}$	$0.045_{\pm 0.001}$	$0.203_{\pm 0.004}$	$0.045_{\pm 0.001}$	$0.201_{\pm 0.003}$
Context-cAE	$0.054_{\pm 0.001}$	$0.224_{\pm 0.002}$	$0.054_{\pm 0.001}$	$0.224_{\pm 0.002}$	$0.055_{\pm 0.002}$	$0.226_{\pm 0.004}$
AE	$0.050_{\pm 0.001}$	$0.635_{\pm 0.223}$	$0.050_{\pm 0.001}$	$2.201_{\pm 4.181}$	$0.051_{\pm 0.002}$	$5.692_{\pm 5.969}$
Context-AE	$0.051_{\pm 0.001}$	$0.586_{\pm 0.303}$	$0.050_{\pm 0.001}$	$0.432_{\pm 0.045}$	$0.053_{\pm 0.001}$	$1.060_{\pm 1.018}$
FiLM+AE	$\mathbf{0.010_{\pm 0.012}}$	$0.220_{\pm 0.366}$	$\mathbf{0.008_{\pm 0.007}}$	$0.083_{\pm 0.032}$	$\mathbf{0.013_{\pm 0.008}}$	$0.230_{\pm 0.381}$

Appendix

Table 14: Pendulum — state noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to states at training time.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.054_{\pm 0.001}$	$0.225_{\pm 0.002}$	$0.054_{\pm 0.001}$	$0.225_{\pm 0.002}$	$0.052_{\pm 0.001}$	$0.222_{\pm 0.002}$
Context-AE	$0.051_{\pm 0.001}$	$0.492_{\pm 0.188}$	$0.050_{\pm 0.001}$	$0.859_{\pm 0.940}$	$0.050_{\pm 0.001}$	$0.582_{\pm 0.403}$
FiLM+AE	$\mathbf{0.007_{\pm 0.008}}$	$0.288_{\pm 0.706}$	$\mathbf{0.006_{\pm 0.001}}$	$\mathbf{0.055_{\pm 0.005}}$	$\mathbf{0.013_{\pm 0.001}}$	$0.167_{\pm 0.234}$
SoftNcAE	$0.037_{\pm 0.016}$	$0.146_{\pm 0.052}$	$0.032_{\pm 0.014}$	$0.131_{\pm 0.046}$	$0.047_{\pm 0.046}$	$0.174_{\pm 0.103}$
NcAE (ours)	$0.013_{\pm 0.002}$	$\mathbf{0.058_{\pm 0.009}}$	$0.015_{\pm 0.003}$	$0.069_{\pm 0.006}$	$0.017_{\pm 0.001}$	$\mathbf{0.088_{\pm 0.004}}$

Appendix

Table 15: Pendulum — context noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to context at training time. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.054_{\pm 0.001}$	$0.223_{\pm 0.002}$	$0.053_{\pm 0.001}$	$0.221_{\pm 0.002}$	$0.051_{\pm 0.001}$	$0.220_{\pm 0.002}$
Context-AE	$0.050_{\pm 0.001}$	$0.627_{\pm 0.381}$	$0.051_{\pm 0.001}$	$0.522_{\pm 0.223}$	$0.052_{\pm 0.001}$	$0.587_{\pm 0.166}$
FiLM+AE	$\mathbf{0.007_{\pm 0.007}}$	$0.064_{\pm 0.022}$	$\mathbf{0.008_{\pm 0.0004}}$	$0.077_{\pm 0.004}$	$\mathbf{0.017_{\pm 0.002}}$	$0.365_{\pm 0.659}$
SoftNcAE	$0.045_{\pm 0.043}$	$0.159_{\pm 0.104}$	$0.042_{\pm 0.010}$	$0.160_{\pm 0.038}$	$0.045_{\pm 0.015}$	$0.175_{\pm 0.033}$
NcAE (ours)	$0.013_{\pm 0.002}$	$\mathbf{0.064_{\pm 0.006}}$	$0.014_{\pm 0.001}$	$\mathbf{0.073_{\pm 0.005}}$	$0.021_{\pm 0.002}$	$\mathbf{0.107_{\pm 0.005}}$

Appendix

Table 16: Pendulum — state and context noise (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to both states and context at training time. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.680_{\pm 0.297}$	$1.928_{\pm 0.592}$	$1.413_{\pm 0.579}$	$4.015_{\pm 1.261}$	$2.776_{\pm 1.232}$	$7.099_{\pm 2.254}$
Context-AE	$0.378_{\pm 0.093}$	$11.611_{\pm 2.839}$	$0.725_{\pm 0.202}$	$11.963_{\pm 2.820}$	$1.272_{\pm 0.357}$	$12.988_{\pm 2.848}$
FiLM+AE	$0.462_{\pm 0.037}$	$3.406_{\pm 2.773}$	$1.348_{\pm 0.733}$	$7.264_{\pm 4.455}$	$8.677_{\pm 10.330}$	$34.120_{\pm 40.071}$
SoftNcAE	$0.820_{\pm 0.192}$	$2.319_{\pm 0.507}$	$0.962_{\pm 0.149}$	$2.589_{\pm 0.410}$	$1.210_{\pm 0.152}$	$3.063_{\pm 0.351}$
NcAE (ours)	$\mathbf{0.236_{\pm 0.024}}$	$\mathbf{0.657_{\pm 0.089}}$	$\mathbf{0.536_{\pm 0.047}}$	$\mathbf{1.373_{\pm 0.108}}$	$\mathbf{1.001_{\pm 0.099}}$	$\mathbf{2.469_{\pm 0.235}}$

Appendix

Table 17: Lorenz96 — state and context noise (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to both states and context at test time; models trained on clean data. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.319_{\pm 0.254}$	$0.644_{\pm 0.204}$	$0.319_{\pm 0.253}$	$\mathbf{0.645_{\pm 0.204}}$	$0.320_{\pm 0.253}$	$\mathbf{0.650_{\pm 0.203}}$
Context-AE	$0.237_{\pm 0.046}$	$11.142_{\pm 2.794}$	$\mathbf{0.237_{\pm 0.046}}$	$11.108_{\pm 2.817}$	$\mathbf{0.239_{\pm 0.046}}$	$11.084_{\pm 2.761}$
FiLM+AE	$0.370_{\pm 0.066}$	$3.004_{\pm 2.735}$	$0.848_{\pm 0.443}$	$4.747_{\pm 2.597}$	$7.147_{\pm 9.760}$	$31.344_{\pm 48.653}$
SoftNcAE	$0.810_{\pm 0.195}$	$2.306_{\pm 0.512}$	$0.920_{\pm 0.157}$	$2.548_{\pm 0.422}$	$1.085_{\pm 0.137}$	$2.909_{\pm 0.353}$
NcAE (ours)	$\mathbf{0.148_{\pm 0.039}}$	$\mathbf{0.503_{\pm 0.158}}$	$0.343_{\pm 0.038}$	$0.993_{\pm 0.145}$	$0.637_{\pm 0.066}$	$1.712_{\pm 0.178}$

Appendix

Table 18: Lorenz96 — context noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to context at test time; models trained on clean data. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
cAE	$0.725_{\pm 0.277}$	$1.822_{\pm 0.366}$	$1.406_{\pm 0.456}$	$3.914_{\pm 0.796}$	$2.690_{\pm 0.882}$	$6.983_{\pm 1.416}$
Context-cAE	$0.680_{\pm 0.297}$	$1.928_{\pm 0.592}$	$1.413_{\pm 0.579}$	$4.014_{\pm 1.261}$	$2.776_{\pm 1.232}$	$7.098_{\pm 2.254}$
AE	$0.255_{\pm 0.066}$	$19.152_{\pm 6.605}$	$0.468_{\pm 0.098}$	$18.846_{\pm 6.660}$	$0.836_{\pm 0.151}$	$18.608_{\pm 5.445}$
Context-AE	$0.378_{\pm 0.093}$	$11.595_{\pm 2.828}$	$0.724_{\pm 0.202}$	$11.984_{\pm 2.810}$	$1.271_{\pm 0.356}$	$13.071_{\pm 2.810}$
FiLM+AE	$0.416_{\pm 0.043}$	$3.281_{\pm 2.815}$	$0.801_{\pm 0.292}$	$4.857_{\pm 2.840}$	$2.015_{\pm 1.711}$	$9.491_{\pm 7.808}$

Appendix

Table 19: Lorenz96 — state noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to states at test time; models trained on clean data.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.063_{\pm 0.002}$	$0.237_{\pm 0.002}$	$0.098_{\pm 0.006}$	$0.291_{\pm 0.010}$	$0.192_{\pm 0.024}$	$0.496_{\pm 0.086}$
Context-AE	$0.055_{\pm 0.002}$	$0.908_{\pm 0.809}$	$0.070_{\pm 0.002}$	$0.975_{\pm 0.790}$	$\mathbf{0.108_{\pm 0.007}}$	$1.137_{\pm 0.853}$
FiLM+AE	$0.027_{\pm 0.010}$	$0.376_{\pm 0.755}$	$0.063_{\pm 0.019}$	$0.461_{\pm 0.775}$	$0.126_{\pm 0.040}$	$0.639_{\pm 0.883}$
SoftNcAE	$0.035_{\pm 0.004}$	$0.129_{\pm 0.016}$	$\mathbf{0.063_{\pm 0.004}}$	$\mathbf{0.190_{\pm 0.011}}$	$0.128_{\pm 0.010}$	$\mathbf{0.357_{\pm 0.032}}$
NcAE (ours)	$\mathbf{0.027_{\pm 0.002}}$	$\mathbf{0.091_{\pm 0.007}}$	$0.068_{\pm 0.008}$	$0.203_{\pm 0.022}$	$0.180_{\pm 0.036}$	$0.520_{\pm 0.137}$

Appendix

Table 20: Pendulum — state and context noise (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to both states and context at test time; models trained on clean data. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
Context-cAE	$0.054_{\pm 0.001}$	$0.226_{\pm 0.001}$	$0.055_{\pm 0.001}$	$0.227_{\pm 0.001}$	$0.055_{\pm 0.001}$	$0.229_{\pm 0.001}$
Context-AE	$0.051_{\pm 0.002}$	$0.881_{\pm 0.802}$	$0.051_{\pm 0.002}$	$0.869_{\pm 0.774}$	$0.051_{\pm 0.002}$	$0.882_{\pm 0.811}$
FiLM+AE	$\mathbf{0.010_{\pm 0.011}}$	$0.351_{\pm 0.750}$	$0.018_{\pm 0.020}$	$0.365_{\pm 0.745}$	$0.034_{\pm 0.041}$	$0.399_{\pm 0.740}$
SoftNcAE	$0.026_{\pm 0.005}$	$0.115_{\pm 0.019}$	$0.028_{\pm 0.005}$	$0.120_{\pm 0.018}$	$0.035_{\pm 0.005}$	$0.138_{\pm 0.016}$
NcAE (ours)	$0.014_{\pm 0.003}$	$\mathbf{0.061_{\pm 0.007}}$	$\mathbf{0.017_{\pm 0.003}}$	$\mathbf{0.071_{\pm 0.006}}$	$\mathbf{0.025_{\pm 0.003}}$	$\mathbf{0.100_{\pm 0.011}}$

Appendix

Table 21: Pendulum — context noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to context at test time; models trained on clean data. Context-free architectures (cAE, AE) are omitted as they are insensitive to context noise.

	$\sigma=0.10$		$\sigma=0.25$		$\sigma=0.50$
Model	Pos.	Vel.	Pos.	Vel.	Pos.	Vel.
cAE	$0.063_{\pm 0.004}$	$0.245_{\pm 0.011}$	$0.139_{\pm 0.041}$	$0.423_{\pm 0.129}$	$0.435_{\pm 0.248}$	$1.150_{\pm 0.660}$
Context-cAE	$0.063_{\pm 0.002}$	$0.237_{\pm 0.002}$	$0.097_{\pm 0.006}$	$0.290_{\pm 0.009}$	$0.190_{\pm 0.023}$	$0.491_{\pm 0.082}$
AE	$0.053_{\pm 0.001}$	$2.040_{\pm 2.974}$	$0.066_{\pm 0.002}$	$2.220_{\pm 3.065}$	$\mathbf{0.097_{\pm 0.003}}$	$2.464_{\pm 3.355}$
Context-AE	$0.055_{\pm 0.002}$	$0.905_{\pm 0.805}$	$0.070_{\pm 0.002}$	$0.988_{\pm 0.824}$	$0.108_{\pm 0.007}$	$1.136_{\pm 0.845}$
FiLM+AE	$\mathbf{0.026_{\pm 0.009}}$	$0.374_{\pm 0.755}$	$\mathbf{0.060_{\pm 0.011}}$	$0.455_{\pm 0.786}$	$0.116_{\pm 0.016}$	$0.619_{\pm 0.886}$

Appendix

Table 22: Pendulum — state noise only (mean

{}_{\pm\text{std}}

RMSE). Gaussian noise

\mathcal{N}(0,\sigma^{2})

added to states at test time; models trained on clean data.

Method	Step 1	Step 5	Step 10	Step 15
cAE	$1.46\times 10^{-5}_{\pm 4.99\times 10^{-6}}$	$1.38\times 10^{-5}_{\pm 4.87\times 10^{-6}}$	$1.33\times 10^{-5}_{\pm 4.85\times 10^{-6}}$	$1.28\times 10^{-5}_{\pm 4.99\times 10^{-6}}$
Context-cAE	$1.68\times 10^{0}_{\pm 1.36\times 10^{0}}$	$8.45\times 10^{0}_{\pm 2.19\times 10^{1}}$	$1.71\times 10^{2}_{\pm 5.84\times 10^{2}}$	$4.22\times 10^{3}_{\pm 1.58\times 10^{4}}$
AE	$4.99\times 10^{-1}_{\pm 2.82\times 10^{-1}}$	$5.26\times 10^{-1}_{\pm 3.02\times 10^{-1}}$	$2.58\times 10^{0}_{\pm 4.22\times 10^{0}}$	$6.73\times 10^{1}_{\pm 1.83\times 10^{2}}$
Context-AE	$8.10\times 10^{-1}_{\pm 3.79\times 10^{-1}}$	$1.24\times 10^{0}_{\pm 9.71\times 10^{-1}}$	$2.25\times 10^{0}_{\pm 1.66\times 10^{0}}$	$8.65\times 10^{0}_{\pm 1.45\times 10^{1}}$
FiLM+AE	$1.13\times 10^{0}_{\pm 2.91\times 10^{-1}}$	$1.29\times 10^{6}_{\pm 3.86\times 10^{6}}$	$\infty$	$\infty$
SoftNcAE	$9.98\times 10^{-1}_{\pm 6.87\times 10^{-1}}$	$2.27\times 10^{0}_{\pm 2.72\times 10^{0}}$	$1.01\times 10^{1}_{\pm 2.62\times 10^{1}}$	$8.92\times 10^{1}_{\pm 3.82\times 10^{2}}$

Appendix

Table 23: Idempotency error

\|P^{k}(\boldsymbol{x})-P^{k-1}(\boldsymbol{x})\|

(mean

\pm

std) over successive projection steps

k

, reported on the Lorenz96 system. Constrained architectures (cAE, NcAE) maintain error

\approx 0

by construction;

\infty

indicates numerical blow-up.

Method	Step 1	Step 5	Step 10	Step 15
cAE	$1.26\times 10^{-5}_{\pm 2.53\times 10^{-6}}$	$1.25\times 10^{-5}_{\pm 2.52\times 10^{-6}}$	$1.24\times 10^{-5}_{\pm 2.52\times 10^{-6}}$	$1.24\times 10^{-5}_{\pm 2.51\times 10^{-6}}$
Context-cAE	$2.72\times 10^{-1}_{\pm 3.33\times 10^{-2}}$	$3.09\times 10^{-1}_{\pm 5.96\times 10^{-2}}$	$3.38\times 10^{1}_{\pm 1.00\times 10^{2}}$	$1.20\times 10^{5}_{\pm 3.60\times 10^{5}}$
AE	$9.77\times 10^{-2}_{\pm 1.65\times 10^{-2}}$	$8.80\times 10^{-2}_{\pm 1.52\times 10^{-2}}$	$8.18\times 10^{-2}_{\pm 1.57\times 10^{-2}}$	$7.85\times 10^{-2}_{\pm 1.64\times 10^{-2}}$
Context-AE	$1.59\times 10^{-1}_{\pm 1.04\times 10^{-2}}$	$1.46\times 10^{-1}_{\pm 1.27\times 10^{-2}}$	$1.37\times 10^{-1}_{\pm 1.54\times 10^{-2}}$	$1.33\times 10^{-1}_{\pm 1.92\times 10^{-2}}$
FiLM+AE	$7.24\times 10^{-2}_{\pm 1.02\times 10^{-1}}$	$8.06\times 10^{-2}_{\pm 1.28\times 10^{-1}}$	$1.01\times 10^{-1}_{\pm 1.88\times 10^{-1}}$	$1.31\times 10^{-1}_{\pm 2.73\times 10^{-1}}$
SoftNcAE	$8.95\times 10^{-2}_{\pm 5.34\times 10^{-2}}$	$9.71\times 10^{-2}_{\pm 7.29\times 10^{-2}}$	$9.96\times 10^{-2}_{\pm 7.29\times 10^{-2}}$	$1.02\times 10^{-1}_{\pm 6.50\times 10^{-2}}$

Appendix

Table 24: Idempotency error

\|P^{k}(\boldsymbol{x})-P^{k-1}(\boldsymbol{x})\|

(mean

\pm

std) over successive projection steps

k

, reported on the pendulum system. Constrained architectures (cAE, NcAE) maintain error

\approx 0

by construction.

Autoencoders

Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.

See all 152 papers

Aug 30, 2026

Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction

Autoencoders are widely used for nonlinear dimensionality reduction and manifold learning. While most common implementations rely on both nonlinear encoders and decoders, we investigate the specific role of the encoder and the extent to which it can be constrained to be linear without reducing accuracy. We conduct a comparative study on four autoencoder architectures: standard fully nonlinear autoencoders (AE), linear-encoder autoencoders (Lenc-AE), linear-decoder autoencoders (Ldec-AE), and fully linear autoencoders (LAE), evaluated on synthetic manifolds, computational mechanics data sets, and real-world image data sets including MNIST. We demonstrate that imposing a linear encoder preserves most of the representational capacity of the autoencoder, provided the decoder remains nonlinear. In particular, Lenc-AE consistently outperforms both Ldec-AE and LAE, and achieves reconstruction quality comparable to fully nonlinear AE, while offering advantages in terms of parsimony and interpretability of the latent representation. These results suggest that the nonlinear decoder is the critical component for manifold learning, rather than the encoder. A geometric interpretation of this finding is developed, which identifies the precise conditions under which a linear encoder is sufficient, and the specific manifold configurations that expose its limitations.

Louen Pottier, Louis Lesueur, Anders Thorin

CEA List, Université Paris-Saclay · Palaiseau, F-91120, France · LMS, École Polytechnique, Institut Polytechnique de Paris +1

Autoencoders Manifold Learning Dimensionality Reduction

Ask PDF

Apr 30, 2026

Do Sparse Autoencoders Capture Concept Manifolds?

Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.

Usha Bhalla, Thomas Fel, Can Rager +9

aHarvard University · bNortheastern University · cTechnion IIT +1

Neural Representation Geometry Sparse Autoencoders Neural Network Interpretability Manifold Learning Sparse Coding

Ask PDF

Mar 17, 2026

Manifold-Matching Autoencoders

We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).

Laurent Cheret, Vincent Létourneau, Isar Nejadgholi +3

Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada

Multidimensional Scaling Autoencoders Latent-Space Regularization Dimensionality Reduction Latent Space Geometry Latent Space Alignment+2

Ask PDF

Abstract

Autoencoders

Explore similar work

Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction

Do Sparse Autoencoders Capture Concept Manifolds?

Manifold-Matching Autoencoders