On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

Krisanu Sarkar

cs.LGMar 28, 2026

On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

Authors: Krisanu Sarkar

Organizations: Indian Institute of Technology Bombay, Mumbai, India

Abstract

We study cross-modal alignment between independently pretrained vision (DINOv2) and language (all-MiniLM-L6-v2) encoders using the functional map framework from computational geometry, which represents correspondence between representation manifolds as a compact linear operator between graph Laplacian eigenbases. While the framework underperforms Procrustes alignment and relative representations for cross-modal retrieval across all supervision budgets, it reveals a structural property of multimodal representations. We find that the Laplacian eigenvalue spectra of the two encoders are quantitatively similar (normalized spectral distance 0.043), indicating that independently trained models develop manifolds of comparable intrinsic complexity. However, the functional map exhibits near-zero diagonal dominance (mean below 0.05) and large orthogonality error (70.15), showing that the eigenvector bases are effectively unaligned. We term this decoupling the spectral complexity--orientation gap: models converge in how much structure they capture but not in how they organize it. This gap defines a boundary condition for spectral alignment methods and motivates three diagnostic quantities : diagonal dominance, orthogonality deviation, and Laplacian commutativity error for characterizing cross-modal representation compatibility.

Figures & tables

		Image $\to$ Text			Text $\to$ Image
Method	$\|S\|$	R@1	R@5	R@10	R@1	R@5	R@10
Raw Cosine	0	0.1	0.1	0.1	0.1	0.5	1.0
FMap Unsupervised (HKS)	0	0.7	0.7	1.1	0.4	1.7	3.2
FMap (ours)	20	0.9	0.9	2.1	0.8	4.5	8.1
Procrustes	20	2.1	2.1	3.2	2.0	5.7	10.4
Relative Reps	20	3.4	3.4	5.0	3.5	9.3	14.1

Table 1. Image–text retrieval on Flickr30k (1,000 images, 5,000 captions). R@

K

(%) for image-to-text (i2t) and text-to-image (t2i). All training-free methods use the same DINOv2 + MiniLM encoder pair with

k_{s}{=}50

, ZoomOut refinement to

k_{\mathrm{max}}{=}100

, and a single random seed. CCA is omitted at

|S|{<}20

(insufficient anchors). Bold indicates best training-free result per column; CLIP is shown as a strong supervised reference.

Figure 1. Image-to-text R@1 and R@5 as a function of anchor budget $|S|$ (log scale). The functional map (blue) improves with more anchors but grows substantially slower than Procrustes (orange) and relative representations (green). The CLIP reference line (red, dashed) indicates the performance achievable with full joint training on 400M pairs. CCA (purple) fails across all budgets. Line plot of image-to-text recall versus anchor budget, comparing functional maps, Procrustes, relative representations, CCA, and a CLIP reference line.

	i2t			t2i
$k_{s}$	R@1	R@5	R@10	R@1	R@5	R@10
10	0.7	0.7	1.3	0.6	2.9	4.5
20	0.5	0.5	0.9	0.1	0.8	1.2
30	0.7	0.7	1.1	0.1	1.0	1.7
50	2.2	2.2	3.3	0.1	0.4	1.0
70	2.5	2.5	4.3	0.0	0.5	0.9

Table 2. Effect of spectral dimension

k_{s}

on functional map retrieval (R@

K

, %,

|S|{=}50

anchors) without ZoomOut refinement. Higher

k_{s}

improves i2t retrieval monotonically but does not improve t2i, suggesting a floor imposed by eigenvector misalignment.

Figure 2. Image-to-text recall as a function of spectral dimension $k_{s}$ . Performance increases with $k_{s}$ but remains an order of magnitude below ambient-space baselines at all truncations. Line plot showing image-to-text recall as spectral dimension increases, with gradual gains that remain below ambient-space baselines.

Figure 3. Spectral diagnostics for the DINOv2–MiniLM encoder pair. Top left: Laplacian eigenvalue spectra are quantitatively similar. Top right: eigenvalue ratio $\lambda_{i}^{v}/\lambda_{i}^{t}$ deviates from 1.0 primarily at low frequencies (indices 0–10), then stabilizes. Bottom left: absolute values of the functional map matrix $|\mathbf{C}|$ ; energy is scattered across off-diagonal bands (rows $\sim$ 15 and $\sim$ 30) rather than concentrated on the diagonal. Bottom right: diagonal energy fraction $\rho_{i}$ per spectral index; all values are below 0.2, most below 0.05. Four-panel diagnostic figure showing similar Laplacian spectra, frequency-wise eigenvalue ratios, an off-diagonal functional map matrix magnitude heatmap, and low diagonal energy fractions across spectral indices.

Diagnostic	Observed	Shape matching
Spectral distance $d_{\mathrm{spec}}$	0.043	$<0.01$
Mean diagonal dominance $\bar{\rho}$	$<0.05$	$>0.7$
Orthogonality error $\epsilon_{\mathrm{orth}}$	70.15	$<0.1$
Eigenvalue range (vision)	$[0.032,0.662]$	—
Eigenvalue range (text)	$[0.030,0.655]$	—

Table 3. Spectral diagnostic quantities for the DINOv2 (vision) and MiniLM (text) encoder pair, computed on

N{=}1{,}000

Flickr30k samples with

k_{s}{=}50

and

|S|{=}50

anchors. For reference, values typical of near-isometric shape correspondence are shown in the right column ( Ovsjanikov et al., 2012 ; Melzi et al., 2019 ) .

	i2t R@ $K$ (%)
Method	R@1	R@5	R@10
Composed ( $v{\to}t_{1}{\to}t_{2}$ )	0.3	0.3	0.7
Direct ( $v{\to}t_{2}$ , 20 anchors)	1.3	1.3	2.1
Random baseline	0.1	0.5	1.0

Table 4. Composability evaluation. The composed map (DINOv2

\to

MiniLM

\to

mpnet) uses no direct DINOv2–mpnet anchor pairs, while the direct map uses 20.

Vision	Text	i2t R@ $K$ (%)
		R@1	R@5	R@10
DINOv2-B	MiniLM	1.7	1.7	3.3
DINOv2-B	mpnet	1.0	1.0	2.3

Table 5. Functional map retrieval across encoder pairings (

|S|{=}50

anchors,

k_{s}{=}50

, ZoomOut refinement to

k_{\mathrm{max}}{=}100

Functional Maps

See all 12 papers

When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Cross-Modal Representation Learning Cross-Modal Alignment
Technion · Genentech · Brown University
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion
Apr 23, 2026Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko +1Cross-Modal Alignment Vision-Language Alignment
Department of Brain and Cognitive Sciences, MIT · CSAIL, MIT · Center for Neural Science, NYU +1

Jun 9, 2026J/K move · Enter open · S save

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets+1

Technion · Genentech · Brown University

Cross-Modal Representation Learning Cross-Modal Alignment

Open paper AskPDF arXiv

Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds and when each fails, a gap that leaves practitioners, especially in scientific domains with heterogeneous instruments and multiple levels of measurement, unable to diagnose why standard methods underperform the best single modality. We study both objectives under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, the ingredient that breaks the classical recovery guarantees, and derive separation ratios that expose complementary failure modes: alignment whitens each modality and fails when nuisance is strongly correlated across views; prediction encodes whatever is cross-predictable through a one-sided whitening, with recovery governed by source-modality quality. The resulting phase diagram partitions multimodal problems into four regimes --- Both, CA only, CP only, and Neither --- refined by a recovery count that separates partial recovery from complete failure. We present a data-driven procedure to locate real-world datasets in this diagram using a small labeled subsample, identifying the preferred objective and prediction direction before any cross-modal training, and identifying when no objective in the CA/CP family can improve on the stronger modality alone. Experiments on synthetic data, stereo-vision benchmarks, image--caption pairs, and two real scientific domains, astronomy and single-cell multi-omics, validate the predictions in the nonlinear regime, including both faces of the Neither regime. Code to reproduce the results is available at https://github.com/IlayMalinyak/mm_align_vs_pred.

Abstract

Functional Maps

Explore similar work

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion