The Geometry of Meaning in Neural Manifolds

Valentino Maiorca

Figure 1: High-dimensional evidence of coordinate mismatch. Five independent trainings of a high-dimensional \gls mnist autoencoder. Top : \gls pca fitted independently on each space produces highly similar projections—intrinsic geometry is shared. Bottom : \gls pca fitted on the leftmost space and applied to all five reveals severe misalignment—extrinsic coordinates differ. The incompatibility is purely optimization-induced. Colors: \gls mnist classes.

Figure 2: Realizing abstract structure through parameterized optimization. Two models optimized with the same data

\mathcal{D}

and task loss

\mathcal{L}_{\mathrm{task}}

but different extrinsic factors (architecture and regularization

\mathcal{R}^{1}

vs.

\mathcal{R}^{2}

) produce learned functions that encode similar intrinsic semantic relationships yet through incompatible coordinate systems (

\mathbb{R}^{d_{\mathcal{X}}}

vs.

\mathbb{R}^{d_{\mathcal{Y}}}

, with

d_{\mathcal{X}}

,

d_{\mathcal{Y}}

the latent dimensions). The resulting manifolds

\mathcal{M}(E_{\theta^{*}})

exhibit similar geometry embedded in extrinsic coordinates that differ by arbitrary transformations;

f_{\theta}(\mathcal{D},\mathcal{R}^{1})

and

g_{\theta}(\mathcal{D},\mathcal{R}^{2})

in the figure denote the encoders of the two models (trained with regularizers

\mathcal{R}^{1}

,

\mathcal{R}^{2}

) applied to

\mathcal{D}

,

\mathcal{M}_{\mathcal{D}}

denotes each resulting manifold, and

\mathbb{R}^{d}

,

\mathbb{R}^{e}

denote

\mathbb{R}^{d_{\mathcal{X}}}

,

\mathbb{R}^{d_{\mathcal{Y}}}

.

Property	Parallel ( $\Pi_{\Gamma}^{\mathcal{X}}$ )	Coupled ( $T_{\Gamma}$ )
How $\Gamma$ is used	Independently per space	Both sides together
	( $\Gamma_{\mathcal{X}}$ and $\Gamma_{\mathcal{Y}}$ separate)	(paired)
Output space	Auxiliary coordinates	Target extrinsic coordinates
	(defined by $\Gamma$ structure)	(native to $\mathcal{Y}$ )
Decoder compatibility	Requires auxiliary decoder	Direct reuse of target decoder
Modularity	High (one-to-many)	Lower (pairwise)

Table 1: Comparison of Operator Families. The families differ in how they exploit correspondence structure, where outputs live, and the resulting trade-offs between modularity and decoder compatibility.

Figure 3: Latent spaces learned by distinct trainings of the same AE on the \gls mnist dataset. The bottleneck has size

2

, thus there is no dimensionality reduction involved in the visualization of the latent space. The stochasticity in the training phase induces intrinsically similar representations.

Figure 4: Left : Three anchors (colored stars) are selected on the data manifold; given a point on the manifold (blue dot), its similarity w.r.t. the three anchors is computed, yielding a vector of dimensionality 3 (middle). Right : Each dimension is treated as coefficients in a coordinate system defined by the anchors. Anchors are orthogonal in this example only for visualization purposes.

Table 2: Qualitative (left) and quantitative (right) comparisons of English word embeddings using absolute and relative representations. PCA is applied only for visualization. All metrics are calculated with

k=10

averaged over 20K words and across 10 different random seeds. For visualizations with other dimensionality reductions and similar experiments on \gls cifart, see the “Anchors analysis” and “Additional results” sections of the appendix of Moschella

ˆ

et al. (2023a) .

Figure 5: Self-similarity correlations between absolute and relative spaces for FastText and Word2Vec . Blue nodes represent self-similarities in the absolute embeddings

{\bm{X}}

,

{\bm{Y}}

; green nodes represent relative spaces

{\bm{X}}_{\mathrm{rel}}

,

{\bm{Y}}_{\mathrm{rel}}

(in the figure,

\mathbb{A}

,

\mathbb{B}

are

{\bm{X}}

,

{\bm{Y}}

and

\mathbb{C}

,

\mathbb{D}

are

{\bm{X}}_{\mathrm{rel}}

,

{\bm{Y}}_{\mathrm{rel}}

). Vertical arrows indicate how much the underlying metric in the absolute space is preserved by the relative coordinate transformation, measured via Pearson correlation coefficient. Horizontal and diagonal arrows give the same correlation between the two models’ (absolute or relative) self-similarities.

Figure 6: Graph node classification task on \gls cora. Left: Correlation between the performance of

\approx 2000

models and the similarity of their latent spaces with respect to a well-performing reference model. Right: The same correlation plotted over time. The mean Pearson correlation over all models is

0.955

, after filtering out the models having best validation accuracy below

0.5

.

	Image Classification				Graph Node Classification
	\gls mnist	\glsxtrshort fmnist	\gls cifart	\gls cifarh	\gls cora	CiteSeer	PubMed
Relative	$97.91\pm 0.07$	$90.19\pm 0.27$	$87.70\pm 0.09$	$66.72\pm 0.35$	$0.89\pm 0.02$	$0.77\pm 0.03$	$0.91\pm 0.01$
Absolute	$97.95\pm 0.10$	$90.32\pm 0.21$	$87.85\pm 0.06$	$68.88\pm 0.14$	$0.90\pm 0.01$	$0.78\pm 0.03$	$0.91\pm 0.01$

Table 3: Performance comparison between relative and absolute representations on several image and graph datasets. The metric is the classification weighted F1 score (

\pm

std), over 6 seeds (image datasets in %, graph datasets as fractions).

Figure 7: Reconstruction examples. Each column is a different image. The first row (S) shows the source images. Each block of rows is an architecture (AE, VAE); within it, each pair of rows is a representation (Abs., Rel.), non-stitched first, then stitched.

			\gls mnist	\glsxtrshort fmnist	\gls cifart	\gls cifarh	MSE ↓
AE	Abs.	Non-Stitch.	$0.66\pm 0.02$	$1.57\pm 0.03$	$1.94\pm 0.08$	$2.13\pm 0.08$	$1.58\pm 0.05$
	Abs.	Stitch.	$97.79\pm 2.48$	$120.54\pm 6.81$	$86.74\pm 4.37$	$97.17\pm 3.50$	$100.56\pm 4.29$
	Rel.	Non-Stitch.	$1.18\pm 0.02$	$3.59\pm 0.04$	$2.83\pm 0.13$	$3.50\pm 0.08$	$2.78\pm 0.07$
	Rel.	Stitch.	$2.83\pm 0.20$	$6.37\pm 0.29$	$5.39\pm 1.18$	$18.03\pm 12.46$	$8.16\pm 3.53$
VAE	Abs.	Non-Stitch.	$1.31\pm 0.04$	$4.38\pm 0.03$	$2.68\pm 0.06$	$3.00\pm 0.03$	$2.84\pm 0.04$
	Abs.	Stitch.	$98.51\pm 1.49$	$118.96\pm 2.96$	$69.02\pm 1.54$	$78.57\pm 1.88$	$91.27\pm 1.97$

Table 4: Stitching performance. The MSE (

\times 10^{-3}

,

\pm

std) between the input images and their reconstructions is computed over 5 different seeds; the last column averages the four datasets. Stitching with relative representations yields an error between 5 and 35 times lower than the absolute counterpart.

		Absolute		Relative
				Translated		Wikipedia
Decoder	Encoder	FScore	MAE	FScore	MAE	FScore	MAE
en	en	$91.54\pm 0.58$	$0.08\pm 0.01$	$90.06\pm 0.60$	$0.10\pm 0.01$	$90.45\pm 0.52$	$0.10\pm 0.01$
	es	$43.67\pm 1.09$	$0.56\pm 0.01$	$82.78\pm 0.81$	$0.17\pm 0.01$	$78.53\pm 0.30$	$0.21\pm 0.00$
	fr	$54.41\pm 1.61$	$0.45\pm 0.02$	$78.49\pm 0.66$	$0.21\pm 0.01$	$70.41\pm 0.57$	$0.29\pm 0.01$
	ja	$48.72\pm 0.90$	$0.51\pm 0.01$	$65.72\pm 0.55$	$0.34\pm 0.01$	$66.31\pm 0.80$	$0.34\pm 0.01$

Table 5: Cross-lingual stitching performance comparison. The table reports the mean weighted F1 (

\pm

std) and MAE on Amazon Reviews coarse-grained, across 5 seeds.

		\gls trec	\gls dbpedia	\gls amazon
				Coarse	Fine
Abs.	Non-Stitch	$91.70\pm 1.39$	$98.62\pm 0.58$	$87.81\pm 1.58$	$55.35\pm 3.19$
Abs.	Stitch	$21.49\pm 3.64$	$6.96\pm 1.46$	$49.58\pm 2.95$	$19.01\pm 2.04$
Rel.	Non-Stitch	$88.08\pm 1.37$	$97.42\pm 2.05$	$85.08\pm 1.93$	$48.92\pm 3.57$
Rel.	Stitch	$75.89\pm 5.38$	$80.47\pm 21.14$	$72.37\pm 7.32$	$33.24\pm 7.21$

Table 6: Cross-architecture stitching performance comparison. The table reports the mean weighted F1 (

\pm

std) for each dataset, across 5 different seeds.

		\gls cifarh		\gls imagenet1k
Decoder	Encoder	Absolute	Relative	Absolute	Relative
rexnet-100	rexnet-100	$82.06\pm 0.15$	$80.22\pm 0.28$	$73.78\pm 0.29$	$72.61\pm 0.16$
	vit-base-patch16-224	-	$54.98\pm 0.44$	-	$37.39\pm 0.36$
	vit-base-resnet50-384	-	$53.33\pm 0.37$	-	$42.36\pm 0.36$
	vit-small-patch16-224	-	$59.82\pm 0.32$	-	$43.75\pm 0.27$
vit-base-patch16-224	rexnet-100	-	$76.81\pm 0.49$	-	$30.78\pm 0.81$

Table 7: Stitching performance comparison with different encoding techniques. The table reports the mean weighted F1 (

\pm

std) on \gls cifarh coarse-grained and \gls imagenet1k, across 5 seeds.

Figure 8: Motivation for product of invariances . Different transformations can affect latent spaces in complex ways: the CKA similarity between pretrained models on \gls fmnist, measured after infusing invariance to one class of transformations at a time (conformal, Euclidean,

L_{1}

isometries; legend: Conformal, E(n), ISO

{}_{L_{1}}

(n), with

n

the latent dimension), changes with the model, and no single transformation connects all latent spaces. Combining multiple distance functions, each invariant to specific transformations, captures these complex relationships between latent spaces.

Figure 9: Product space framework . The intrinsic manifold

\mathcal{M}

shared by two latent spaces

\mathcal{X},\mathcal{Y}

related by an unknown transformation is approximated through the product space

\tilde{\mathcal{M}}

. Each component is a \gls relrep space computed using a similarity function

\simf_{i}

invariant to a specific transformation class. Combining these spaces yields a representation

\rho

that approximates projection onto the unknown intrinsic manifold

\mathcal{M}

.

Similarity	Isotropic	Orthogonal	Translation	Permutation	Affine	Linear	Manifold
Function	Scaling	Transf.			Transf.	Transf.	Isometry
Absolute	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
Cosine	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$	$\times$
Euclidean ( $L_{2}$ )	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
Manhattan ( \gls l1)	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
Chebyshev ( \gls linf)	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$

Table 8: Distance-induced invariances. No single metric provides universal coverage.

Figure 10: Latent Spaces Cross-Seed Similarity. Cross-seed Pearson correlation of latent spaces for \glspl ae and \glspl vae, with and without linear bottleneck, trained on \gls mnist ( left ) and \gls cifarh ( right ) until convergence. No single projection consistently outperforms others across all settings. The \gls linf projection is not displayed to improve visualization.

Figure 11: Latent Spaces Cross-Architecture Similarity. Linear \gls cka similarity of latent spaces across several pretrained models on \gls cifart ( left ) and \gls trec ( right ). In each bar, we report the space similarities distribution to the other models while infusing a specific invariance. There is no singular projection that consistently outperforms others across all configurations.

Projection	\gls dbpedia↑	\gls trec↑	\gls cora ↑	Stitching Index ↑
	Text		Graph
	ALBERT		GCN
\glsentrylong cos	$0.50\pm 0.02$	$0.54\pm 0.03$	$\textbf{0.53}\pm 0.06$	0.71
\glsentrylong eu	$0.50\pm 0.00$	$0.60\pm 0.03$	$0.27\pm 0.06$	0.58
\gls l1	$\textbf{0.52}\pm 0.01$	$\textbf{0.65}\pm 0.02$	$0.26\pm 0.06$	0.58
\gls linf	$0.18\pm 0.02$	$0.29\pm 0.06$	$0.12\pm 0.03$	1.00

Table 9: Graph and Text Classification Stitching Performance. Zero-shot accuracy scores (mean

\pm

standard deviation over seeds) across various decoders, seeds, and datasets. For the text domain, we use pretrained models, while for the graph domain, we train GCN models from scratch and evaluate the stitching across seeds. Additionally, we calculate the stitching index for the graph modality, showing that composing different projections using the Aggregation by sum enables zero-shot stitching without any performance drop in this setting, ensuring competitive end-to-end performance.

		Accuracy ↑
Encoder	Projection	\gls cifarh	\gls cifart	\gls mnist	\gls fmnist
CViT-B/32	\glsentrylong cos	$0.52\pm 0.03$	$\textbf{0.87}\pm 0.02$	$0.61\pm 0.06$	$0.68\pm 0.02$
	\glsentrylong eu	$0.53\pm 0.02$	$\textbf{0.87}\pm 0.02$	$\textbf{0.66}\pm 0.05$	$\textbf{0.70}\pm 0.03$
	\gls l1	$\textbf{0.53}\pm 0.04$	$\textbf{0.87}\pm 0.02$	$\textbf{0.66}\pm 0.05$	$\textbf{0.70}\pm 0.03$
	\gls linf	$0.27\pm 0.04$	$0.52\pm 0.04$	$0.57\pm 0.03$	$0.55\pm 0.01$
	\glsentrylong cos, \glsentrylong eu, \gls l1, \gls linf	$\textbf{0.58}\pm 0.03$	$\textbf{0.88}\pm 0.02$	$\textbf{0.68}\pm 0.05$	$\textbf{0.70}\pm 0.01$

Table 10: Image Classification Stitching Performance Cross-Architecture and Cross-Seed. Zero-shot accuracy score (mean

\pm

standard deviation over seeds) across different pretrained models, seeds, and datasets. The proposed method using the Aggregation by sum consistently achieves the highest accuracy score or comparable results, without prior knowledge of the optimal projection to employ.

Figure 12: Reconstruction Qualitative Results on \gls cifarh using \glspl ae. The last two sets of columns (

D_{\mathcal{X}}\circ E_{\mathcal{X}}

and

D_{\mathcal{Y}}\circ E_{\mathcal{Y}}

) are the end-to-end \glspl ae with unique initialization seeds, while the first two (

D_{\mathcal{X}}\circ E_{\mathcal{Y}}

and

D_{\mathcal{Y}}\circ E_{\mathcal{X}}

) illustrate the outputs of the zero-shot stitching of these independently trained models (in every column

D\circ E

abbreviates

D\circ\rho\circ E

, with

\rho

the product projection built on each encoder’s own anchors; in the Absolute row no projection is applied). The first row displays the source images, the two subsequent rows show the results from distinct combinations of projection aggregated through the Aggregation by sum , while the last one shows the outputs from a baseline model that does not incorporate this methodology. When using the \gls linf projection ( second to last row ), the reconstructed images are blurred, but when removing the \gls linf space from the aggregation ( third row ) the reconstruction drops in performance, meaning that it captures information not captured by the others.

	Vision	Text	Graph
Aggregation	RViT-B/16	BERT-C	GCN
Concat*	$0.81\pm 0.00$	$0.54\pm 0.03$	$0.75\pm 0.02$
MLP+SelfAttention	$0.84\pm 0.01$	$0.51\pm 0.03$	$0.63\pm 0.13$
MLP+Sum	$\textbf{0.85}\pm 0.00$	$\textbf{0.55}\pm 0.04$	$\textbf{0.77}\pm 0.01$
SelfAttention	$0.76\pm 0.03$	$0.36\pm 0.22$	$0.76\pm 0.02$

Table 11: Ablation Study on the Aggregation Functions. Zero-shot classification accuracy score (mean

\pm

standard deviation over seeds) across different architectures, seeds, and datasets. The Aggregation by sum ( third row ) obtains consistently the best accuracy score.

Projection	Aggregation	Accuracy ↑
\glsentrylong cos	-	0.50
\glsentrylong eu	-	$0.38$
\gls l1	-	$0.24$
\gls linf	-	$0.21$
\glsentrylong cos, \glsentrylong eu, \gls l1, \gls linf	SelfAttention	$0.17$
\glsentrylong cos, \glsentrylong eu, \gls l1, \gls linf	MLP+Sum	0.45

Table 12: Optimal Space Selection . Classification accuracy for the stitched model with RexNet as encoder and ViT-B/16 as decoder on \gls cifarh, using different projection functions and aggregation strategies. Fine-tuning the space selection and blending module ( QKV opt ) has a larger effect on performance improvement than fine-tuning the \gls mlp head ( MLP opt ). Bold: best per block.

Figure 13: Attention Weights Visualization. Comparison of attention weights for the stitched model with RexNet as encoder and ViT-B/16 as decoder on \gls cifarh, before and after fine-tuning. (left) the attention weights of the initial zero-shot stitched model, which remain unchanged when fine-tuning the classifier ( MLP opt ) (right) . Conversely, fine-tuning the \gls qkv projections ( QKV opt ) (center) leads to a notable shift in attention weights, assigning lower importance to the space that performs worst individually. L1 and Linf in the axes are

L_{1}

and

L_{\infty}

.

Table 13: Qualitative ( left ) and quantitative ( right ) evaluation of the \gls ao method in the retrieval task between FastText (FT) and Word2Vec (W2V) embeddings of about 20k English words ( Chapter 4 ); mean and standard deviation across five random seeds.

	Src	Tgt	Jaccard $\uparrow$	MRR $\uparrow$	Cosine $\uparrow$
GT	ViT-base	ViT-small	$0.11\pm 0.00$	$0.27\pm 0.01$	$0.97\pm 0.00$
GT	ViT-small	ViT-base	$0.10\pm 0.00$	$0.28\pm 0.01$	$0.97\pm 0.00$
Seed	ViT-base	ViT-small	$0.03\pm 0.00$	$0.03\pm 0.01$	$0.97\pm 0.00$
Seed	ViT-small	ViT-base	$0.03\pm 0.00$	$0.04\pm 0.01$	$0.96\pm 0.00$
AO	ViT-base	ViT-small	$0.10\pm 0.01$	$0.23\pm 0.03$	$0.97\pm 0.00$
AO	ViT-small	ViT-base	$0.10\pm 0.00$	$0.28\pm 0.01$	$0.97\pm 0.00$

Table 14: Retrieval on about 20k \gls cifart images encoded by ViT-base (768 dimensions) and ViT-small (384 dimensions), comparing the \gls ao anchors with the GT and Seed baselines and extending the word-embedding results of Table 13 to images. All three anchor sets are used to build relative representations; mean and standard deviation across five random seeds.

Hyperparameter	Retrieval	Zero-shot stitching
Random seed	0 , 1 , 2 , 3 , 4	0 , 1 , 2 , 3 , 4
Number of anchors to approximate	300	300
Number of seed anchors	15	15
Number of optimization steps	250	125
Learning Rate	0.02	0.05
Optimizer	Adam	Adam

Table 15: Hyperparameters for the \gls ao method in retrieval and zero-shot stitching tasks.

Dec.	Enc.	Fscore	MAE	Fscore	MAE	Fscore	MAE
		GT		Seed		AO
en	es	0.51 $\pm$ 0.01	0.67 $\pm$ 0.02	0.44 $\pm$ 0.01	0.80 $\pm$ 0.01	0.48 $\pm$ 0.01	0.70 $\pm$ 0.02
es	en	0.50 $\pm$ 0.02	0.72 $\pm$ 0.04	0.41 $\pm$ 0.01	0.92 $\pm$ 0.02	0.46 $\pm$ 0.01	0.76 $\pm$ 0.02

Table 16: Cross-lingual zero-shot stitching performance evaluation on Amazon Reviews, mean

\pm

std across five seeds; Fscore

\uparrow

, MAE

\downarrow

.

Figure 14: ASIF is a simple recipe to align the representations of two frozen pretrained encoders.

Figure 15: Captions of similar images are themselves similar. Distribution of similarities of 100k embedded pairs in the training set versus the above image and caption embeddings. Highlighted in orange are the 1000 pairs with the highest image similarity. Dashed lines: mean similarity over all pairs (blue) and over the highlighted 1000 (orange); y-axis: count, log scale.

Method	Dataset size	ImageNet	CIFAR-100	Pets	ImageNet v2
CLIP ( Radford et al., 2021 )	400M (private)	68.6	68.7	88.9	-
CLIP ( Radford et al., 2021 )	15M (public)	31.3	-	-	-
LiT ( Zhai et al., 2022 )	10M (public)	66.9	-	-	-
CLIP †	901M (private)	50.6	47.9	70.3	43.3
LiT ( Zhai et al., 2022 )	901M (private)	70.1	70.9	88.1	61.7
ASIF (sup vis. encoder)	1.6M (public)	60.9 ∗	50.2	81.5	52.2

Table 17: Zero-shot classification accuracy of different multimodal designs. CLIP and LiT implementations vary by dataset and the visual transformer used as image encoder. The first CLIP and LiT entries use a VITb16 as ASIF, the last CLIP and LiT entries use a VITb32 (larger patch size). The public dataset of CLIP is a curated subset of YFCC100m ( Thomee et al., 2016 ) , while LiT and ASIF use CC12M. † CLIP as reported by Zhai et al. (2022) , setting “uu” (both towers trained from scratch). ∗ A subset of the ImageNet validation set was used to tune the two hyperparameters of ASIF which were then applied to the other datasets. The reported ImageNet accuracy is measured on a separate test set. When tuning on different datasets, accuracies remain consistent (see Norelli et al. (2023a) for additional details).

Figure 16: ASIF is a learning algorithm: ImageNet validation-set accuracy improves smoothly as the multimodal dataset size increases. Colors show the impact of

k

and

p

(non-zeros, val exp).

Figure 17: ASIF representations are interpretable and amendable. Thorough analysis of the relative representations—the four vectors in green—produced by ASIF to assign the best caption to a given image

o^{*}

(

x^{*}

and

y^{*}

in the figure are

o^{*}

and the ideal caption

o^{\prime*}

). The values shown are the raw similarities of the retained entries, before exponentiation and normalization. Every dimension corresponds to the similarity of the input image (text) to a unique image (text) entry in the multimodal dataset. The visualization shows training samples leading to correct or incorrect predictions, which is useful to curate better data. For example, the second training sample is broken; its removal enables production of an updated ASIF model in seconds.

Figure 18: Entropic Gromov-Wasserstein solver (left) solves a sequence of regularized optimal transport (

\epsilon

-OT) problems using the Sinkhorn algorithm. In contrast, the proposed approach learns, via a pair of embeddings,

f_{\theta}

and

g_{\phi}

, the transport cost that directly produces the sought alignment

{\bm{P}}^{*}

by solving a single

\epsilon

-OT problem. While the learning of the embeddings still requires multiple calls to the

\epsilon

-OT solver, their cost is amortized at inference time. In the figure,

\Pi

denotes the coupling

{\bm{P}}

,

k

the index of the solver iteration

t

, and

\varepsilon

the regularization strength

\epsilon

.

Figure 19: Simulated annealing of

\epsilon

and spectral geometric regularization are effective in stabilizing the solver and improving the accuracy of the assignment. Left: simulated annealing schedule used. Middle: distribution of the alignment error (measured as FOSCTTM) over

20

runs with and without

\epsilon

-annealing. Right: distribution of the top-1 assignment accuracy with and without the spectral geometric regularization of the transport cost.

Figure 20: The proposed solver generalizes to unseen samples and scales to large-sample sizes post-training. In both top and bottom experiments,

\mathcal{X}

and

\mathcal{Y}

are ViT embeddings. The entropic GW solver can only operate in the transductive regime and runs out of memory for

n>25000

(

N

on the horizontal axis is

n

; E-GW is the entropic GW solver; OOM marks where it runs out of memory).

Figure 21: Qualitative evaluation of the proposed GW solver in inductive setting. The plot depicts the assignment produced by the distance-based GW solver ( Eq. 8 ) on a new set of samples.

Figure 22: Qualitative and quantitative results on the scSNARE-seq dataset. Left and middle: Aligned samples from ATAC and RNA, colored by the domains (ATAC: black, RNA: red) and cell types, respectively. Right: the sorted FOSCTTM plot, a quantitative metric measuring the quality of the assignment (GW in the legend is the entropic GW baseline of Demetci et al. (2022) ).

Figure 23: Qualitative and quantitative results on the human bone marrow single-cell dataset. Top plots depict the UMAP of the translated cells colored by domain (left) and by the cell type (right). Bottom plots report the FOSCTTM metrics for

\mathcal{Y}

projected onto

\mathcal{X}

(left) and

\mathcal{X}

projected onto

\mathcal{Y}

(right).

Figure 24: Cross-subject fMRI alignment via task-informed encoding. (Left) Confusion matrix of \glslink mrrMRR similarity between 8 subjects’ relative representation spaces; the diagonal is left empty. (Right) Comparison with baselines: Relative coordinates (blue) achieve substantially higher similarity than \gls pca dimensionality reduction (orange) or absolute coordinates (green). Boxes aggregate the scores of all subject-hemisphere pairs over three anchor sampling seeds, grouped by subject.

Figure 25: Alignment quantifies reparameterization removal in ensemble latent spaces. (Top) Curve fitting results for VGG-16 (red) and PreResNet-110 (blue), pairwise connecting three modes. Individual model error rate (left) and cumulative ensemble error rate (right). (Bottom) Alignment of embedding spaces along curves for absolute (green) and relative (purple) coordinates. Dashed lines show error rates (top) and alignment (bottom) across eleven independently trained networks (baseline). In most panels, relative spaces achieve higher alignment, indicating reduced reparameterization noise, and the steepest slope occurs at curve midpoint (

t=0.5

).

Figure 26: Conceptual visualization and empirical measurement of alignment metric. (Left) Schematic: given observations

o_{i=1}

and

o_{i=2}

with

J

ensemble realizations each (the drawing writes their ensemble means

\bar{{\bm{x}}}_{i}

as

\bar{{\bm{z}}}_{i}

and the global mean as

\bar{{\bm{z}}}

, and labels the two variances

\sigma_{W}^{2}

,

\sigma_{B}^{2}

as

{\bm{S}}_{W}

,

{\bm{S}}_{B}

), high alignment requires compact within-observation clustering (small

\sigma_{W}^{2}

) and large between-observation separation (large

\sigma_{B}^{2}

). (Right) t-SNE projection of VGG-16 embeddings for 10 \gls cifarh classes (same t-SNE layout; relative space via cosine similarity). Dot size represents uncertainty. Absolute space (left projection) shows larger uncertainty; relative space (right projection) shows smaller uncertainty.

Figure 27: Data collection for parallel anchors. Correspondence across visual variations established via deterministic replay (same action sequence with fixed seed) or pixel-space transformations (

u

,

v

and

{\bm{D}}_{u}

,

{\bm{D}}_{v}

in the figure are the variations

v

,

w

and their observation sets). Both methods assume deterministic environments or well-defined transformations to establish correspondence.

Figure 28: Correspondence-parallel projection reveals semantic structure. (a) Cosine similarity matrices for absolute (left) vs. relative (right) representations from aligned frames (green/red grass). Absolute space shows uniformly low similarity, even on the diagonal; relative space reveals semantically meaningful off-diagonal matches—functionally similar track positions (e.g., a turn or straight segment of similar shape) despite temporal misalignment. (b) Example frame pairs from high-similarity regions of (a); three of the four are off-diagonal.

			Controller
			Visual Variations (standard task)			Task Variations (green)
			green	red	blue	slow	scrambled	no idle
Encoder	green	S. Abs	175 $\pm$ 304	167 $\pm$ 226	$-4$ $\pm$ 79	148 $\pm$ 328	106 $\pm$ 217	213 $\pm$ 201
	green	S. R3L	781 $\pm$ 108	787 $\pm$ 62	794 $\pm$ 61	268 $\pm$ 14	781 $\pm$ 126	824 $\pm$ 82
	red	S. Abs	157 $\pm$ 248	43 $\pm$ 205	22 $\pm$ 112	83 $\pm$ 191	138 $\pm$ 244	252 $\pm$ 228
	red	S. R3L	810 $\pm$ 52	776 $\pm$ 92	803 $\pm$ 58	476 $\pm$ 430	790 $\pm$ 72	817 $\pm$ 69

Table 18: Comparing R3L and absolute in zero-shot stitching performance. The original domains for the encoders and the controllers are listed in the rows and columns, respectively. R3L outperforms the absolute model in all the tests.

	V1	V2	V3	V4
T1	3h	3h	3h	3h
T2	4h	4h	4h	4h
T3	3h	3h	3h	3h
T4	3h	3h	3h	3h

Table 19: Table of training times. We only need to train combinations for the cells highlighted in blue and then perform zero-shot stitching to assemble all the other agents ( totaling 13 hrs ). Normally, it would be required to train agents for each domain-task combination ( totaling 52 hrs ). Visual variations V1 : green, V2 : red, V3 : blue, V4 : far (green) and task variations T1 : standard, T2 : slow, T3 : no idle, T4 : scrambled.

Figure 29: Scale invariance of RoBERTa according to the performance of a downstream classifier trained on the encodings of the last attention layer. At each layer (with 0 being the embedding layer and 12 the output one), for each run, the encodings are normalized to unit norm and multiplied by

\alpha

, and the effect on final accuracy is measured. The performance without any rescaling is

0.92

.

Figure 30: On the top row, reconstruction similarity sensitivity at different number of subspaces and pruning threshold (

\delta

) of intra-space inversion ( left ) and inter-space inversion between different encoders ( right ) on the coarse-grained \gls cifarh. On the bottom row, the corresponding condition number average over the subspaces. Higher pruning thresholds lower the condition number, stabilizing the matrix inverse; in the inter-space setting this increases the reconstruction similarity up to

\delta\approx 0.8

.

Enc.	Dec.	Absolute	XLM-R	IRP
en	en	1.00	1.00	0.97
es	en	0.00	0.99	0.97
fr	en	-0.02	0.99	0.97
ja	en	0.03	0.99	0.97

Table 20: Reconstruction similarities between parallel encodings across languages. Absolute refers to the original encodings, XLM-R to the encodings of a multilingual LLM, IRP to Inverse Relative Projection, comparing the translated source embeddings with the target ones.

Figure 31: Performance comparison of three Multilayer Perceptrons (MLPs) with different activation functions, namely cosine (blue), ReLU (orange), and tanh (green) at different rescaling factors

\alpha

. The ReLU and tanh MLPs exhibit scale invariance for

\alpha>0

, while the cosine activation function is only invariant on the mean data scale and its periodic cycles.

Figure 32: Distribution of the embedding scales in different pretrained encoders on the \gls cifarh dataset. Well-behaved roughly Gaussian distributions with a single mode and well-defined mean are crucial for the method, as they support the ability to rescale the embeddings to a mean scale.

				Accuracy
		$\omega$	$\delta$	zero-shot	absolute	non-stitch	available
Vision	\gls mnist	8	0.65	$0.88\pm 0.03$	$0.10\pm 0.03$	$0.97\pm 0.01$	20
	\gls fmnist	8	0.65	$0.82\pm 0.03$	$0.10\pm 0.02$	$0.91\pm 0.01$	20
	\gls cifart	8	0.75	$0.92\pm 0.05$	$0.11\pm 0.04$	$0.95\pm 0.03$	20
	\gls cifarh (coarse)	8	0.70	$0.72\pm 0.10$	$0.05\pm 0.01$	$0.88\pm 0.05$	20
	\gls cifarh (fine)	8	0.75	$0.62\pm 0.09$	$0.01\pm 0.00$	$0.82\pm 0.07$	20

Table 21: Zero-shot stitching for classification . Comparison between stitching and non-stitching accuracies. Stitching includes the translation method ( zero-shot ) with its hyperparameters and the stitching of absolute spaces. The achievable rows show the number of possible combinations the stitching methods can tackle among the available ones. The method works between encoders of arbitrary embedding dimensionality, enabling stitching of all available pairs. The results are averaged over 5 different random seeds. The non-stitch column reports, for reference, each classification head applied to the encoder it was trained on.

Figure 33: On the top row, the reconstruction similarity ( left ) and accuracy ( right ) sensitivity analysis on the number of subspaces (

\omega

) and pruning threshold (

\delta

) on the \gls trec dataset. On the bottom row, the corresponding condition number ( left ) and the number of anchors after pruning ( right ) averaged across subspaces. The metrics are averaged over stitching between all possible pairs of language encoders across 3 seeds.

Figure 34: Performance comparison between different encoders and data modalities on the N24News multimodal dataset. On the right, each classification head applied to the encoder it was trained on. On the left the stitching performance between pairs of encoders and decoders. The stitching achieves satisfactory results even when translating across modalities. Results obtained with

1000

anchors,

\omega=16

and

\delta=0.8

.

Figure 35: Zero-shot stitching of

\mathcal{X}

and

\mathcal{Y}

\glslink extrinsic-factorextrinsic spaces (labelled X and Y in the figure, where Z is the relative space and A X , A Y are the anchor matrices

{\bm{A}}_{\mathcal{X}}

,

{\bm{A}}_{\mathcal{Y}}

; in the edge labels, X and Y are the stacked representations

{\bm{X}}

,

{\bm{Y}}

, and the superscript T is the transpose ⊤ ) utilizing relative representations and semantic alignment (the estimation of

T_{\Gamma}

,

\mathcal{T}

in the figure). This approach does not require a decoder specifically trained on relative representations (

D

on the relative space

\mathbb{R}^{K}

, the central decoder in the figure). Instead, it directly translates representation spaces, enabling the use of arbitrarily pretrained decoders (dec X , dec Y in the figure, i.e. the decoders

D_{\mathcal{X}}

,

D_{\mathcal{Y}}

) originally trained on \glslink extrinsic-factorextrinsic spaces.

Figure 36: Method illustration on a synthetic example. Given a source space

\mathcal{X}

, the steps to translate it to a target

\mathcal{Y}

are sequentially applied as described in Section 38.2 . Note that the translation is not perfect due to an arbitrary distortion of the data.

Figure 37: Performance comparison of affine , linear , l-ortho , and ortho at varying number of anchors on classification accuracy. Results on \gls cifarh fine-grained.

	Dataset	no-stitch	absolute	relative	affine	linear	l-ortho	ortho
Vision	\gls cifart	$0.95\pm 0.03$	$0.16\pm 0.22$	$0.80\pm 0.22$	$0.92\pm 0.05$	$0.88\pm 0.11$	$0.90\pm 0.09$	$0.93\pm 0.04$
	\gls cifarh- C	$0.85\pm 0.07$	$0.11\pm 0.21$	$0.54\pm 0.25$	$0.78\pm 0.09$	$0.73\pm 0.16$	$0.77\pm 0.11$	$0.81\pm 0.07$
	\gls cifarh- F	$0.76\pm 0.09$	$0.07\pm 0.21$	$0.30\pm 0.24$	$0.68\pm 0.11$	$0.62\pm 0.19$	$0.64\pm 0.16$	$0.71\pm 0.09$
	\gls fmnist	$0.88\pm 0.01$	$0.15\pm 0.20$	$0.63\pm 0.23$	$0.86\pm 0.01$	$0.83\pm 0.06$	$0.82\pm 0.05$	$0.85\pm 0.02$
	\gls mnist	$0.96\pm 0.01$	$0.15\pm 0.21$	$0.50\pm 0.22$	$0.94\pm 0.01$	$0.89\pm 0.08$	$0.81\pm 0.11$	$0.91\pm 0.02$
Text	\gls trec	$0.87\pm 0.12$	$0.20\pm 0.06$	$0.36\pm 0.13$	$0.82\pm 0.12$	$0.74\pm 0.25$	$0.57\pm 0.25$	$0.79\pm 0.11$

Table 22: Cross-architecture stitching with various methods for estimating

T_{\Gamma}

and applying standard scaling (mean

\pm

standard deviation over seeds). The stitched decoders are SVMs with a linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to \gls cifarh indicate, respectively, coarse-grained and fine-grained.

	Dataset	no-stitch	absolute	relative	affine	linear	l-ortho	ortho
Vision	\gls cifart	$0.95\pm 0.03$	$0.16\pm 0.22$	$0.80\pm 0.22$	$0.93\pm 0.04$	$0.78\pm 0.27$	$0.88\pm 0.12$	$0.91\pm 0.09$
	\gls cifarh- C	$0.85\pm 0.07$	$0.11\pm 0.21$	$0.54\pm 0.25$	$0.79\pm 0.07$	$0.65\pm 0.25$	$0.73\pm 0.17$	$0.79\pm 0.10$
	\gls cifarh- F	$0.76\pm 0.09$	$0.07\pm 0.21$	$0.30\pm 0.24$	$0.69\pm 0.10$	$0.52\pm 0.25$	$0.62\pm 0.19$	$0.68\pm 0.13$
	\gls fmnist	$0.88\pm 0.01$	$0.15\pm 0.20$	$0.63\pm 0.23$	$0.86\pm 0.01$	$0.65\pm 0.23$	$0.83\pm 0.06$	$0.84\pm 0.05$
	\gls mnist	$0.96\pm 0.01$	$0.15\pm 0.21$	$0.50\pm 0.22$	$0.94\pm 0.01$	$0.61\pm 0.23$	$0.90\pm 0.08$	$0.90\pm 0.04$
Text	\gls trec	$0.87\pm 0.12$	$0.20\pm 0.06$	$0.36\pm 0.13$	$0.82\pm 0.12$	$0.44\pm 0.20$	$0.74\pm 0.23$	$0.77\pm 0.12$

Table 23: Cross-architecture stitching with various methods for estimating

T_{\Gamma}

and applying

L_{2}

normalization (mean

\pm

standard deviation over seeds). The stitched decoders are SVMs with linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to \gls cifarh indicate, respectively, coarse-grained and fine-grained.

Figure 38: Performance comparison between different encoders and data modalities on the \gls n24news multimodal dataset. On the right the accuracy of models trained end-to-end on a single data modality (Score) and their average norm (Scale). On the left the stitching performance between pairs of encoders and decoder. This shows the importance of translating from good encoders, that can even improve unimodal decoder performances. Results obtained with

2000

anchors and ortho , with an SVM as classification head.

Figure 39: Reconstruction examples grouped by dataset (from left to right: \gls mnist, \gls fmnist, \gls cifart, \gls cifarh). Each column is a different image, from top to bottom: original image, absolute stitching, affine stitching, linear stitching, l-ortho stitching, and ortho stitching. No additional normalization applied on the decoder part.

	\gls mnist			\gls fmnist			\gls cifart			\gls cifarh
	lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse
absolute	0.09	0.27	0.14	0.17	0.23	0.23	0.30	0.29	0.34	0.34	0.53	0.40
affine	0.94	0.08	0.02	0.94	0.06	0.03	0.96	0.03	0.05	0.96	0.04	0.05
linear	0.92	0.09	0.02	0.93	0.07	0.04	0.94	0.03	0.05	0.94	0.04	0.06
l-ortho	0.79	0.14	0.02	0.78	0.12	0.05	0.85	0.05	0.06	0.84	0.07	0.07
ortho	0.90	0.10	0.02	0.90	0.08	0.04	0.94	0.03	0.06	0.93	0.04	0.06

Table 24: Zero-shot stitching for generation with various methods for estimating

T_{\Gamma}

. Standard scaling used as normalization and the stitched decoders do not have any additional normalization. Reported metrics: latent cosine similarity ( lcos ) and MSE ( lmse ) between the target encoding and the translated one, and reconstruction MSE ( rmse ) between the input and the output. 1000 anchors used on 500-dimensional spaces.

Figure 40: Framework overview: Given two spaces

\mathcal{X}

,

\mathcal{Y}

, their samples lie on manifolds

\mathcal{M}

,

\mathcal{N}

, which are approximated with k-NN graphs

\mathcal{G}_{\mathcal{X}}

,

\mathcal{G}_{\mathcal{Y}}

. A latent functional map

{\bm{C}}

is optimized between the eigenbases of operators defined on the graphs. This map is a mapping between functions defined on the two manifolds and can be used for (i) comparing representational spaces, (ii) solving correspondence problems, and (iii) transferring information between the spaces.

Figure 41: Similarity across layers. Similarity matrices between internal layer representations of \gls cifart comparing the LFM-based similarity with the CCA and CKA baselines, averaged across 10 models. For each method, the accuracy scores for matching the corresponding layer by maximal similarity are reported.

Figure 42: Robustness of LFM similarity. Left: Similarity scores as a function of perturbation strength (“Distance” on the horizontal axis is the norm of the perturbation vector): while the CKA baseline degrades, LFM similarity scores are robust to perturbations that preserve linear separability of the space. Right: Visualization of area distortion of the map by projecting the first singular component of the LFM in the perturbed space: the distortion localizes on the samples of the perturbed class, making LFM similarity interpretable.

Figure 43: Stitching results. Accuracy performance in stitching between pretrained encoders on five datasets ( \gls cifarh with two labelings), comparing orthogonal transformation (Ortho) and LFM+Ortho at varying anchor counts. Also shown is LFM+Ortho (Labels), which uses the dataset labels instead of anchors. Results are presented with mean accuracy values reported on each violin.

Figure 44: Qualitative results on stitching. We combine the encoder decoder modules of two convolutional autoencoders ( E1 D1 , E2 D2 ; E1, D1 and E2, D2 are

E_{\mathcal{X}},D_{\mathcal{X}}

and

E_{\mathcal{Y}},D_{\mathcal{Y}}

) using three different approaches: no transformation ( Absol. ), the orthogonal transformation from Maiorca

ˆ

et al. (2023a) ( Ortho ), the orthogonal transformation augmented with LFM ( LFM ).

Figure 45: Retrieval of word embeddings. We compare the retrieval performance of the functional map framework with baseline methods as the number of anchors increases. The left panel shows the Mean Reciprocal Rank (MRR) across different numbers of anchors. The right panels depict the first two components of PCA for a subsample of the latent space (b) and the functional space (c), both before and after alignment using the functional map.

Figure 46: Direct contributions from individual heads can be analyzed from the transformer’s residual stream. In a multimodal, zero-shot classification setting (e.g., in CLIP), task boundaries are defined by text prompts that may vary in their conceptual granularity. When certain heads are specialized in particular features (e.g., shape , pattern , color ), they may more accurately apply these boundaries than the model’s original output. In this example, only the fine-grained task ( brown ) effectively separates the samples at the output level.

Figure 47: Heads in early layers show low-dimensional, linear structures, as suggested by similar intrinsic dimension estimates from PCA and TwoNN (column labels L and N). Moving toward the output layer, the nonlinear dimensionality peaks and then decreases, while PCA’s linear estimate continues to rise, indicating increasing nonlinearity in head manifolds (

\text{Ratio}=\frac{L}{N}

). The first principal component (EVR 1 ) explains around 50% of the variance in early layers, dropping to around 10% in later layers.

Figure 48: Comparison between TextSpan and Orthogonal Matching Pursuit on the first principal component (OMP 1 ), applied to the heads of OpenCLIP-L. Left : agreement score between the descriptions returned by the two methods. Right : qualitative comparison of selected descriptions for 4 heads, one per layer, at different agreement levels.

Figure 49: (a) Attention head similarity across layers of OpenCLIP-L, computed between \gls imagenet1k head representations and those obtained on other datasets. (b) Descriptions picked by SOMP for three specialized heads found in the analysis of panel (a) .

	BLIP-L							OpenCLIP-L
Dataset	U	U\|T	S	R	H	B	O	U	U\|T	S	R	H	B	O
CIFAR-10	0.94	0.94	0.93	0.56	0.93	0.94	0.96	0.96	0.96	0.96	0.59	0.94	0.97	0.98
CIFAR-100	0.70	0.71	0.72	0.33	0.69	0.71	0.75	0.80	0.78	0.79	0.38	0.76	0.82	0.84
Cars	0.67	0.68	0.71	0.27	0.66	0.72	0.77	0.92	0.92	0.93	0.39	0.89	0.93	0.93
DTD	0.52	0.54	0.54	0.27	0.50	0.55	0.61	0.59	0.59	0.59	0.29	0.54	0.63	0.69
EuroSAT	0.49	0.53	0.53	0.24	0.37	0.50	0.92	0.64	0.64	0.67	0.34	0.55	0.64	0.95

Table 25: Accuracy when doing zero ablation of all units except top 5% of attention heads. Heads are assigned a binary weight using an Unsupervised (U), Task-conditioned Unsupervised (U|T), Supervised (S), and Random (R) strategy (a mean over 10 different seeds). H corresponds to using all the attention heads available, B is the original model performance, and O is the optimized continuous weighting case. Bold: best average among U, U|T, S.

Figure 50: Performance comparison between text-image alignment methods on zero-shot classification tasks.

	BLIP-L				CLIP-L				OpenCLIP-L
Dataset	Lin	RD	RD ∗	RD Y	Lin	RD	RD ∗	RD Y	Lin	RD	RD ∗	RD Y
CIFAR-10	0.97	0.97	0.97	0.96	0.97	0.98	0.98	0.97	0.98	0.98	0.98	0.98
CIFAR-100	0.82	0.83	0.81	0.74	0.85	0.86	0.85	0.80	0.88	0.88	0.87	0.85
DTD	0.79	0.77	0.76	0.58	0.81	0.80	0.79	0.63	0.84	0.84	0.83	0.69
EuroSAT	0.95	0.98	0.98	0.83	0.97	0.99	0.98	0.95	0.97	0.98	0.98	0.94
GTSRB	0.87	0.87	0.84	0.59	0.92	0.92	0.92	0.77	0.94	0.92	0.92	0.78

Table 26: Accuracy produced by different configurations of ResiDual, compared with a linear aligner. (Lin): linear aligner at the output level; (RD): ResiDual in the original formulation (all principal components of all residual units); (RD ∗ ): ResiDual limited to head units alone (no MLPs), and PCs truncated to 90% of explained variance; (RD Y ): ResiDual applied to the output encoding. Bold: best average per model.

Figure 51: Overview of the method. Given a language or vision-language model and a target property defined by text (e.g., colors , sentiments , numbers ), HeadPursuit scores all attention heads according to how well they align with interpretable directions from a fixed dictionary, using a method based on Matching Pursuit ( Mallat & Zhang, 1993 ) (1). The method then selects the top-

k

heads (2) and intervenes by rescaling their contribution to the residual stream (3), either enhancing or suppressing the attribute in the model’s output.

L18.H27 (“Politics”)	L24.H20 (“Nationality”)	L25.H14 (“Months”)	L30.H28 (“Numbers”)
COVID	British	December	9
Soviet	American	July	1
Obama	European	April	3
Biden	German	October	7
Clinton	English	February	five

Table 27: Top-5 tokens identified by SOMP on selected attention heads of Mistral-7B, evaluated on TriviaQA prompts. L

\ell

.H

h

denotes head

h

of layer

\ell

. Layers and heads are 0-indexed.

L18.H27 (“Politics”)	L24.H20 (“Nationality”)	L25.H14 (“Months”)	L30.H28 (“Numbers”)
vaccine	American	Sunday	8
Covid	Americans	breakfast	u
pandemic	California	Oct	u
COVID	America	October	n
Soviet	American	February	9

Table 28: Top-5 tokens identified by aggregated logit lens on selected attention heads of Mistral-7B, evaluated on TriviaQA data.

Figure 52: Question answering performance of Mistral-7B on TriviaQA. F1 scores are shown for samples associated (country) or not (other) with the target attribute, normalized to the F1 score of the base model without intervention. Random baselines are displayed as medians and interquartile ranges.

Dataset	SOMP	LL	Rand.	SOMP	LL	Rand.	SOMP	LL	Rand.
	8 heads			16 heads			32 heads
RTP	0.83	$0.91$	$1.02$	0.67	$0.79$	$1.00$	0.66	$0.71$	$1.13$
TET	$0.83$	0.81	$0.97$	0.68	$0.73$	$0.95$	0.49	$0.68$	$0.95$

Table 29: Normalized count of toxic generations after intervention, as judged by the toxicity classifier. Lower values indicate better mitigation. Targeted heads reduce toxicity, while heads selected using LL have a weaker impact and random heads maintain or increase toxicity. For random baselines, only the median is shown; the interquartile ranges are reported in the appendix of Basile et al. (2025b) .

Figure 53: Classification results under different head selection strategies: (light blue) 16 heads with highest variance ratio explained by SOMP; (dark blue) 32 heads with highest explained variance ratio; (yellow) 16 random heads, with the same layer-wise counts of top 16; (orange) 32 random heads, with the same layer-wise counts of top 32.

Figure 54: (left) Jaccard similarity between sets of top-

16

LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on a target dataset, denoted by row, when the top-

16

heads are selected with SOMP on a source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy on target dataset.

Figure 55: Task-agnostic head selection on image classification benchmarks. Heads were selected using dataset-specific lists of keywords obtained from an external VLM, with no task knowledge. Classification results under different head selection strategies: (light blue) 16 heads with highest variance ratio explained by SOMP; (dark blue) 32 heads with highest explained variance ratio; (yellow) 16 random heads, with the same layer-wise counts of top 16; (orange) 32 random heads, with the same layer-wise counts of top 32.

Figure 56: (left) Jaccard similarity between sets of top-

8

LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on a target dataset, denoted by row, when the top-

8

heads are selected with SOMP on a source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy on target dataset.

Figure 57: (left) Jaccard similarity between sets of top-

32

LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on a target dataset, denoted by row, when the top-

32

heads are selected with SOMP on a source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy on target dataset.

Figure 58: Effect of inhibitory interventions on LLaVA attention heads for Flickr30k captioning. Bars show the normalized presence of target attributes ( colors , sentiments , quantity ) under different head selection strategies (SOMP and random). Random baselines are reported in terms of medians and interquartile ranges.

Figure 59: Effect of enhancing interventions on LLaVA attention heads for Flickr30k captioning. Bars show the normalized presence of target attributes ( colors , sentiments , quantity ) under different head selection strategies (SOMP and random). Random baselines are reported in terms of medians and interquartile ranges.

Figure 60: Effect of enhancing interventions on 32 color-specialized heads in image captioning. Increasing the rescaling coefficient

\alpha

leads to a stronger presence of color-related words in captions, accompanied by a gradual drop in overall caption quality as measured by CIDEr.

Table 30: Examples of captions produced by LLaVA-NeXT on Flickr30k images, before and after inhibiting (top) or enhancing (bottom) 16 heads specialized on colors (left) and sentiments (right). Analogous examples for quantity are reported in the appendix of Basile et al. (2025b) .

Model	Method	#Params	Tox ${}_{\mathrm{RTC}}^{\mathrm{RTP}}$ $(\downarrow)$	Tox ${}_{\mathrm{RTC}}^{\mathrm{TET}}$ $(\downarrow)$	PPL WIK $(\downarrow)$	MMLU $(\uparrow)$
Qwen2.5-1.5B	None	-	$3.00_{\color[rgb]{0.5,0.5,0.5}{\,0.54}}$	$23.09_{\color[rgb]{0.5,0.5,0.5}{\,0.67}}$	$13.67_{\color[rgb]{0.5,0.5,0.5}{\,0.00}}$	$60.95_{\color[rgb]{0.5,0.5,0.5}{\,0.00}}$
	LoFIT-RL	0.86M	$0.37_{\color[rgb]{0.5,0.5,0.5}{\,0.06}}$	$4.36_{\color[rgb]{0.5,0.5,0.5}{\,0.00}}$	$14.12_{\color[rgb]{0.5,0.5,0.5}{\,0.07}}$	$59.74_{\color[rgb]{0.5,0.5,0.5}{\,0.14}}$
	Prompt	-	$4.07_{\color[rgb]{0.5,0.5,0.5}{\,0.38}}$	$21.02_{\color[rgb]{0.5,0.5,0.5}{\,1.44}}$	$13.65_{\color[rgb]{0.5,0.5,0.5}{\,0.00}}$	$60.96_{\color[rgb]{0.5,0.5,0.5}{\,0.00}}$
	Chinese Astron. Astrophys. ⋆	0.043M	$1.15_{\color[rgb]{0.5,0.5,0.5}{\,0.37}}$	$5.77_{\color[rgb]{0.5,0.5,0.5}{\,2.14}}$	${\color[rgb]{1,0,0}19.30}_{\color[rgb]{0.5,0.5,0.5}{\,2.76}}$	${\color[rgb]{1,0,0}37.67}_{\color[rgb]{0.5,0.5,0.5}{\,6.95}}$
	ReFT ⋆	0.39M	$2.57_{\color[rgb]{0.5,0.5,0.5}{\,0.60}}$	$18.17_{\color[rgb]{0.5,0.5,0.5}{\,3.04}}$	$15.58_{\color[rgb]{0.5,0.5,0.5}{\,0.52}}$	$58.84_{\color[rgb]{0.5,0.5,0.5}{\,0.23}}$
	ITI	0.043M	$1.87_{\color[rgb]{0.5,0.5,0.5}{\,0.21}}$	$18.16_{\color[rgb]{0.5,0.5,0.5}{\,0.62}}$	$12.39_{\color[rgb]{0.5,0.5,0.5}{\,0.09}}$	$60.88_{\color[rgb]{0.5,0.5,0.5}{\,0.08}}$

Table 31: Toxicity mitigation on the RTP and TET datasets using three different models, Qwen2.5-1.5B, Gemma2-2B, and Qwen2.5-7B. Strongly degraded utility is marked in red . Results shown are at low data regime (

N=32

sentences to estimate the interventions). Results for LinEAS improve on ITI and Lin- AcT with similar impact on utility metrics. The quality of these interventions is often on par with the oracle baseline LoFIT-RL in terms of utility, although the strong oracle supervision yields better mitigation. ⋆ Chinese Astron. Astrophys. and ReFT are designed to use paired data, which is not available in the toxicity setting.

Figure 61: LinEAS is effective at low data regime. Toxicity mitigation (two left-most plots) and utility (two right-most plots) as a function of the amount of data available to learn interventions. LinEAS shows better performance (low toxicity and utility close to original dashed lines) for low data, and stable performance for

N\geq 32

. ITI-c in the legends denotes ITI.

Figure 62: Sparsity improves utility while mitigating toxicity . Toxicity results on Qwen2.5-7B using only 32 sentences, at different levels of sparsity

\gamma

that result in different support sizes (x axis). At 1K optimization steps, with a support of about 1%, similar toxicity is maintained (left, center-left) while PPL WIK decreases (center-right) and MMLU increases (right). Note that too long optimizations (10k steps) might harm utility due to overfitting. Similarly, short optimizations ( e.g. 100 steps) and strong sparsity lead to low conditioning (mild toxicity mitigation).

Figure 63: Generations using DMD2 ( Yin et al., 2024a ) . (Left) Concept deletion using ITI, Lin- AcT and LinEAS for two different concepts, starting from prompts that contain the concept. LinEAS shows a more gradual deletion (

\lambda=0.4,0.7,1

), and better preservation of the original image (

\lambda=0

). (Right) Qualitative examples of LinEAS on 6 more concepts. Inverting the steering maps results in concept induction, probably due to strong structure in activation space. LinEAS also outperforms the other methods under this setting. ITI-c in the row labels denotes ITI.

Figure 95

Intervened Layers	# Modules	IMGScore $\uparrow$	CLIPScore $\downarrow$
unet - all layer norms	256	$0.714\pm 0.054$	$0.131\pm 0.033$
unet.transformer - all FeedForward	70	$0.780\pm 0.053$	$0.137\pm 0.031$
unet attentions - K and Q projections	280	$0.935\pm 0.016$	$0.157\pm 0.028$
unet attentions - V projections	140	$0.818\pm 0.051$	$0.140\pm 0.031$
unet attentions - in projections	11	$0.923\pm 0.017$	$0.155\pm 0.028$
unet.resnet - all layer norms	34	$0.896\pm 0.023$	$0.156\pm 0.027$

Table 33: Study on the choice of intervened layers for LinEAS in the DMD2 UNet. Image consistency (IMGScore

\uparrow

) and concept removal (CLIPScore

\downarrow

) are reported, averaged over 15 concepts. The method shows robustness, with the default setting (all layer norms) offering the best trade-off.

The Geometry of Meaning in Neural Manifolds

Abstract

Figures & tables

Neural Representation Geometry

Multi-Way Representation Alignment

Geometric Stability: The Missing Axis of Representations

Language Models Represent and Transform Concepts with Shared Geometry

Abstract

Neural Representation Geometry

Explore similar work

Multi-Way Representation Alignment

Geometric Stability: The Missing Axis of Representations

Language Models Represent and Transform Concepts with Shared Geometry