On the Geometry of Multimodal Saturation: Riemannian VICReg

Nessim Ben Abbes; Duc Han Le; Sabri Mtibaa; Van-Tam Nguyen

cs.LGOct 5, 2026

On the Geometry of Multimodal Saturation: Riemannian VICReg

Authors: Nessim Ben Abbes, Duc Han Le, Sabri Mtibaa, Van-Tam Nguyen

Organizations: Protectline · Télécom Paris

Abstract

In self-supervised learning, a third modality should improve, or at least preserve, performance. Across nine image-text-tabular datasets, we show that it instead harms performance: the trimodal model underperforms its own best bimodal subset in 55.6% of paired runs under VICReg. The same failure occurs in 51.1% of paired runs under SimSiam. We call this failure multimodal saturation. We propose that the failure lies in the alignment geometry. Riemannian VICReg (R-VICReg) generalizes classical VICReg: it aligns views by squared geodesic distance on learnable negative-curvature product factors and recovers VICReg exactly as curvature vanishes. Over the same 45 paired runs, R-VICReg raises the probability that the third modality helps from 44.4% to 64.4%, with gains concentrated where VICReg saturates.

Figures & tables

Figure 1: The alignment geometry decides whether aligned states stay readable. Masked-view alignment contracts each semantic state (one marker shape per state) inside the same radius budget, and a downstream head resolves states only at its margin scale

\varepsilon=\frac{1}{L}

. (a) Under Euclidean alignment, several states land in one

\varepsilon

-cell and merge: induced redundancy. (b) Under geodesic alignment at

\kappa<0

, the same budget offers exponentially more

\varepsilon

-cells, so each state keeps its own. Lemmas 8 – 10 prove the two growth laws.

Figure 2: Three views render the learned metric of one trimodal CSGO run (seed 456). (a) In the flat VICReg projector, equal-distance rings stay evenly spaced. (b) In the most-utilized R-VICReg factor (

\kappa_{32}=-0.988

), the rings at the same geodesic distances crowd toward the boundary. (c) The same points sit on the hyperboloid above their Poincaré-ball shadow (construction in Appendix D ).

Figure 3: Panel (a) reports the per-dataset positive-gain rate

p^{+}(0)=\Pr(\Delta_{\mathrm{tri}}>0)

over five seeds, so rates are multiples of

0.2

. VICReg scores

0

5

on CSGO and Pokemon Secondary (marked

0

). Panel (b) reports the paired difference in trimodal test accuracy between R-VICReg and VICReg: bars give the five-seed mean and the orange line the VICReg baseline.

Figure 4: Reliability

p^{+}(\tau)=\Pr(\Delta_{\mathrm{tri}}>\tau)

over

N=45

paired runs per method, swept over

\tau\in\{0,0.005,0.01\}

: R-VICReg stays above both baselines at every threshold, so the ranking is no artifact of

\tau=0

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	Closed form	Limit $\kappa\to 0$
Exponential map at $0$	$\exp_{0}^{\kappa}(u)=\tan_{\kappa}(\left\lVert u\right\rVert_{2})\,\dfrac{u}{\left\lVert u\right\rVert_{2}}$	$u$
Logarithmic map at $0$	$\log_{0}^{\kappa}(x)=\arctan_{\kappa}(\left\lVert x\right\rVert_{2})\,\dfrac{x}{\left\lVert x\right\rVert_{2}}$	$x$
Möbius addition	$x\oplus_{\kappa}y=\dfrac{\big(1-2\kappa\langle x,y\rangle-\kappa\left\lVert y\right\rVert_{2}^{2}\big)x+\big(1+\kappa\left\lVert x\right\rVert_{2}^{2}\big)y}{1-2\kappa\langle x,y\rangle+\kappa^{2}\left\lVert x\right\rVert_{2}^{2}\left\lVert y\right\rVert_{2}^{2}}$	$x+y$
Geodesic distance	$d_{\kappa}(x,y)=2\arctan_{\kappa}\!\big(\left\lVert(-x)\oplus_{\kappa}y\right\rVert_{2}\big)$	$2\left\lVert x-y\right\rVert_{2}$
Conformal factor	$\lambda_{\mu}^{\kappa}=2/(1+\kappa\left\lVert\mu\right\rVert_{2}^{2})$	$2$
Logarithmic map at $\mu$	$\log_{\mu}^{\kappa}(x)=\dfrac{2}{\lambda_{\mu}^{\kappa}}\,\log_{0}^{\kappa}\big((-\mu)\oplus_{\kappa}x\big)$	$x-\mu$

Appendix

Table 1:

\kappa

-stereographic operations used by R-VICReg and their flat limits. Every limit is uniform on compacta (Lemma 12 ); the distance limit squares to the invariance-term limit

d_{\kappa}(x,y)^{2}\to 4\left\lVert x-y\right\rVert_{2}^{2}

of Proposition 4 .

Figure 5: A rescue is a paired run that VICReg saturates (

\Delta_{\mathrm{tri}}\leq\tau

) and R-VICReg does not; a harm is the reverse. (a) The quadrants at

\tau=0

hold

13

rescues,

4

harms, and

28

concordant runs. (b) The paired improvement

\delta

is positive where VICReg saturates and negative where it does not; bars show

95\%

Student-

t

intervals.

Figure 6: The learned R-VICReg geometry stays hyperbolic on every dataset: curvature remains negative and boundary utilization stays high. The left axis reports

u_{\max}

, with bars spanning the five-seed minimum–maximum range; the right axis reports the mean learned curvature

\kappa

Hyperparameter	Value
Pretrain / finetune epochs	$50$ / $50$
Batch size	$64$
Optimizer / learning rate / weight decay	AdamW / $10^{-4}$ / $0.02$
Warmup epochs / gradient clip	$5$ / $1.0$
Text / image / tabular reconstruction weights	$(1.0,\,1.0,\,0.01)$
Alignment weight $\lambda_{\mathrm{MMM}}$	$1.0$

Appendix

Table 2: Hyperparameters shared by all experiments.

Explore similar work

OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Multimodal Alignment Modality Alignment
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
Binding Multiple Modalities via Multimodal Wasserstein Barycenter
Sep 27, 2026Xiaole Tang, Jiayi Xu, Xiang Gu +2Multimodal Alignment Complementary Modalities
Xi’an Jiaotong University
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Alignment
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

May 28, 2026, cs.LGJ/K move · Enter open · S save

OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

Tianchao Li, Shujian Yu, Xinrui Zu, Zhaolong Wei+3

Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway+3

Multimodal Alignment Modality Alignment Contrastive Learning Information Bottleneck

Open paper AskPDF arXiv

Contrastive learning is effective for aligning paired views or modalities, but alignment beyond two modalities remains non-trivial and comparatively underexplored. Pairwise CLIP-style losses decompose multi-modal alignment into independent two-way comparisons and therefore do not explicitly model higher-order dependencies among multiple modalities. Recent beyond-pairwise objectives approach this problem from statistical or geometric perspectives, but arbitrary-modality alignment still lacks a principled criterion for defining what each modality should preserve and compress relative to the others. We revisit arbitrary-modality alignment through the Information Bottleneck principle. In multi-modal learning, sufficiency should preserve information predictable from the remaining modalities, while minimality should compress modality-specific information not supported by them. This naturally leads to a One-vs-All view, where each modality is characterized with respect to the remaining modalities. We propose OVA-IB, an Information Bottleneck framework for arbitrary-modality alignment. OVA-IB optimizes a tractable One-vs-All contrastive lower bound for sufficiency connected to a Dual Total Correlation-style objective, uses a parameter-free geometry-aware projection score, and derives a tractable upper-bound regularizer for minimality by bounding each representation's dependence on its own input with representation distributions induced by the remaining modalities. Experiments on classification, regression, modality-agnostic evaluation, and cross-modal retrieval benchmarks demonstrate strong and robust performance.

Abstract

Explore similar work

OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

Binding Multiple Modalities via Multimodal Wasserstein Barycenter

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning