Learned representations (or embeddings) are the foundation of modern machine learning systems, yet they are typically trained as fixed-size embeddings, without accounting for varying downstream resource or task constraints. Matryoshka Representation Learning (MRL) addresses this limitation by learning nested representations, but requires costly end-to-end retraining and does not support post hoc expansion, while Contrastive Sparse Representations (CSR) reduce training overhead via lightweight adaptors but operate at a 4x larger dimensionality. Additionally, all prior works restrict their framework towards compression only. In this work, we introduce Hyperbolic Elastic Representation Learning (HERL), a framework for learning dimension-adaptive embeddings (compression and expansion) through a lightweight adaptor without retraining the base encoder. Our key idea is to exploit the exponential capacity of hyperbolic geometry by learning a non-linear mapping from the encoder space to hyperbolic space. We train a simple MLP to downsample the representations before projecting them onto a Lorentzian and learn adaptive representations that respect the Lorentzian geometry. Our approach is model-agnostic and effective in both supervised and unsupervised settings. Through extensive experiments across image and text modalities, we show that HERL preserves downstream performance across a wide range of embedding sizes, exhibits smooth interpolation between dimensions, and achieves these benefits at a fraction of the computational cost. Code: https://github.com/arjunrkaushik/herl.
Figures & tables
Figure 1: Overview. (a) We propose Hyperbolic Elastic Representation Learning (HERL), a post-hoc adaptor (without encoder retraining) that utilizes the inherent exponential volume capacity and faithful structural preservation properties of hyperbolic geometry. (b) HERL enables elastic representations at a fraction of the training cost. Approximately two hours to train representations of size 8 to 8192, on a single H100 GPU, for the ImageNet-1k dataset. On the other hand, Matryoshka Representation Learning (MRL) needs retraining of the encoder taking about sixty hours on four H100 GPUs.
Dim
Text Classification Top-1 Acc (%) ↑
Text Clustering Top-1 Acc (%) ↑
Text Retrieval NDCG@10 (%) ↑
Category
Model
MTOPIntent
Banking77
TweetSentiment
BiorxivP2P
BiorxivS2S
TwentyNews
FiQA2018
NFCorpus
SciFACT
Full Rep
NV-Embed-V2
4096
93.58
92.20
79.73
53.61
49.60
64.82
62.65
43.97
77.93
MRL
Stella-1.5B-v5
256
90.45
86.14
76.75
50.81
46.42
60.07
55.59
36.97
77.48
Jina-v3
256
78.81
84.08
73.81
38.14
34.39
51.96
55.73
36.63
66.63
Nomic-Embed-V1.5
256
72.47
83.69
59.20
38.19
31.83
48.56
35.00
32.54
68.24
Gecko-Embed-004(Google)
256
77.82
86.01
72.97
36.28
33.09
50.60
55.54
37.81
70.86
Table 1: Results on textual representations. We compare the downstream utility of representations across three tasks: classification, clustering and retrieval. HERL (Ours) consistently outperforms existing MRL-based representations. Importantly, in some cases, HERL enables a 32 dimensional representation to surpasses full representation (4096 dimension) performance, achieving 128x efficiency in terms of dimension size.
Figure 2: Performance of HERL on ImageNet-1k and comparison with Euclidean geometry. (a) Top-1 classification accuracy and ( b) 1-NN retrieval accuracy over embedding dimensions, comparing our supervised and unsupervised HERL adaptors against baselines. The dashed green line ("Full Rep.") denotes the frozen encoder’s native-dimension performance. Both HERL variants show improved performance over baselines. The "No solution exists" annotation marks the dimensionality support region that does not exist, whereas HERL extends naturally into this regime. The "Deeper Projector / LoC (Line of Control)" annotation in (b) indicates the dimension below which the adaptor requires a deeper projection head. (c) Hyperbolic vs. Euclidean optimization of the adaptor at 32D and 8192D on three tasks: classification (TweetSentiment), clustering (BiorxivS2S) and retrieval (NFCorpus). Hyperbolic geometry dominates classification and clustering tasks, but struggles with retrieval. Solely due to the fact that the evaluation pipeline normalizes embeddings onto a hypersphere, killing the primal component of distinguishability in hyperbolic geometry - the radius.
Figure 3: Ablation studies across geometry, curvature, and neighborhood size. (a) Lorentz embeddings outperform Poincaré across dimensions, especially in low-dimensions. (b) Increasing curvature significantly improves compact representations (8D), with limited impact in high dimensions (3072D). (c) Low-dimensional embeddings are highly sensitive to neighborhood size k that is used in the contrastive loss. All experiments have been performed on ImageNet-1k.
Figure 4: More ablation studies. (a) Retrieval with Lorentz distance: Cosine scoring projects embeddings onto the unit hypersphere and discards the hyperbolic radius. Evaluation with the native hyperbolic (Lorentz) distance improves results without retraining. (b) Fully hyperbolic adaptor: As expected a fully hyperbolic adaptor architecture outperforms a Euclidean adaptor, both trained with a hyperbolic objective. However, the negligible gains come at a high computational cost. (c) Computational cost: The hyperbolic adaptor is 4.3× slower to train and 1.8× slower at inference.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Failure mode
Active when
Root cause
Fix (low d )
Rank bottleneck
d<s
Linear P cannot rotate S to axis-aligned
Deep nonlinear P
Volume collapse
d<dvol
K classes not ε -separable in Hcd
Class-aware P
Norm collapse
d<dnorm
∥zˉ∥2<r∗ ; geometry degenerates to Euclidean
Scale-amplifying P
Appendix
Table 2: Summary of the three failure modes, their dependence on target dimension d , and the corresponding projector remedy.
Dim
Full Rep.
Rand. FS
SVD
MRL
MRL-E
Unsup. HERL
Sup. HERL
CSR
8
-
2.36
19.14
62.19
57.45
63.90
64.60
67.78
16
-
12.06
46.02
67.91
67.05
67.89
69.90
69.17
32
-
32.91
60.78
69.46
68.60
70.20
71.23
70.15
64
-
49.91
67.04
70.17
69.61
70.71
71.14
70.94
128
-
60.91
69.63
70.52
70.12
71.00
71.72
70.99
256
-
65.75
70.67
70.62
70.36
71.20
71.89
71.31
Appendix
Table 3: 1-NN accuracy. Comparison of different methods on ImageNet-1k for 1-NN retrieval accuracy. Note that, CSR [ 52 ] uses an ambient dimension of 8192 (4x), activating only certain dimensions. Hence, has been omitted from comparison. The highest score has been bolded .
Backbone
Input dim
LR
Epochs
Batch size
Optimizer
Curvature
ResNet50
2048
1e-3
10
2048
RiemannianAdam
0.9
Appendix
Table 4: Implementation details for experiments on image representations with ImageNet-1k
Dim
Full Rep.
Rand. LP
SVD
MRL
MRL-E
Unsup. HERL
Sup. HERL
CSR
8
-
4.56
2.34
66.63
56.66
73.52
74.16
73.62
16
-
11.29
7.17
73.53
71.94
74.75
75.04
74.75
32
-
27.21
20.46
75.03
74.48
75.48
76.02
75.44
64
-
49.47
48.10
75.82
75.35
76.01
76.53
75.88
128
-
65.70
67.24
76.30
75.80
76.29
76.88
76.24
256
-
72.43
74.59
76.47
76.22
76.55
77.01
76.25
Appendix
Table 5: Top-1 accuracy. Comparison of different methods on ImageNet-1k using a linear probe for classification. Note that, CSR [ 52 ] uses an embedding dimension of 8192 (4x) and activate only certain dimensions. Hence, has been omitted from comparison. The highest score has been bolded .
Dataset
Backbone
Input dim
LR
Epochs
Batch size
Optimizer
Curvature
MTOPIntent
NV-Embed-V2
4096
4e-5
10
128
RiemannianAdam
0.9
Banking77
TweetSentiment
BiorxivP2P
NV-Embed-V2
4096
1e-3
10
128
RiemannianAdam
0.9
BiorxivS2S
1e-3
TwentyNews
1e-3
Appendix
Table 6: Implementation details for experiments on text representations across nine datasets.
Dim
Text Classification Top-1 Acc (%) ↑
Text Clustering Top-1 Acc (%) ↑
Text Retrieval NDCG@10 (%) ↑
Category
Model
MTOPIntent
Banking77
TweetSentiment
BiorxivP2P
BiorxivS2S
TwentyNews
FiQA2018
NFCorpus
SciFACT
Full Rep
NV-Embed-V2
4096
93.58
92.20
79.73
53.61
49.60
64.82
62.65
43.97
77.93
MRL
Stella-1.5B-v5
256
90.45
86.14
76.75
50.81
46.42
60.07
55.59
36.97
77.48
Jina-v3
256
78.81
84.08
73.81
38.14
34.39
51.96
55.73
36.63
66.63
Nomic-Embed-V1.5
256
72.47
83.69
59.20
38.19
31.83
48.56
35.00
32.54
68.24
Gecko-Embed-004(Google)
256
77.82
86.01
72.97
36.28
33.09
50.60
55.54
37.81
70.86
Appendix
Table 7: Results on textual representations. We compare the downstream utility of representations across three tasks: classification, clustering and retrieval. Note that, CSR [ 52 ] uses a 4x ambient dimension, activating only certain dimensions. Hence, has been omitted from comparison. The highest score has been bolded .
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Beijing Key Lab of Intelligent Telecommunications Software and Multimedia, Beijing University of Posts and Telecommunications, Beijing 100876, China