Elastic Representations via Hyperbolic Geometry
Organizations: University at Buffalo, SUNY · Lawrence Livermore National Lab
Abstract
Learned representations (or embeddings) are the foundation of modern machine learning systems, yet they are typically trained as fixed-size embeddings, without accounting for varying downstream resource or task constraints. Matryoshka Representation Learning (MRL) addresses this limitation by learning nested representations, but requires costly end-to-end retraining and does not support post hoc expansion, while Contrastive Sparse Representations (CSR) reduce training overhead via lightweight adaptors but operate at a 4x larger dimensionality. Additionally, all prior works restrict their framework towards compression only. In this work, we introduce Hyperbolic Elastic Representation Learning (HERL), a framework for learning dimension-adaptive embeddings (compression and expansion) through a lightweight adaptor without retraining the base encoder. Our key idea is to exploit the exponential capacity of hyperbolic geometry by learning a non-linear mapping from the encoder space to hyperbolic space. We train a simple MLP to downsample the representations before projecting them onto a Lorentzian and learn adaptive representations that respect the Lorentzian geometry. Our approach is model-agnostic and effective in both supervised and unsupervised settings. Through extensive experiments across image and text modalities, we show that HERL preserves downstream performance across a wide range of embedding sizes, exhibits smooth interpolation between dimensions, and achieves these benefits at a fraction of the computational cost. Code: https://github.com/arjunrkaushik/herl.
Figures & tables
| Dim | Text Classification Top-1 Acc (%) | Text Clustering Top-1 Acc (%) | Text Retrieval NDCG@10 (%) | ||||||||
| Category | Model | MTOPIntent | Banking77 | TweetSentiment | BiorxivP2P | BiorxivS2S | TwentyNews | FiQA2018 | NFCorpus | SciFACT | |
| Full Rep | NV-Embed-V2 | 4096 | 93.58 | 92.20 | 79.73 | 53.61 | 49.60 | 64.82 | 62.65 | 43.97 | 77.93 |
| MRL | Stella-1.5B-v5 | 256 | 90.45 | 86.14 | 76.75 | 50.81 | 46.42 | 60.07 | 55.59 | 36.97 | 77.48 |
| Jina-v3 | 256 | 78.81 | 84.08 | 73.81 | 38.14 | 34.39 | 51.96 | 55.73 | 36.63 | 66.63 | |
| Nomic-Embed-V1.5 | 256 | 72.47 | 83.69 | 59.20 | 38.19 | 31.83 | 48.56 | 35.00 | 32.54 | 68.24 | |
| Gecko-Embed-004(Google) | 256 | 77.82 | 86.01 | 72.97 | 36.28 | 33.09 | 50.60 | 55.54 | 37.81 | 70.86 | |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Failure mode | Active when | Root cause | Fix (low ) |
| Rank bottleneck | Linear cannot rotate to axis-aligned | Deep nonlinear | |
| Volume collapse | classes not -separable in | Class-aware | |
| Norm collapse | ; geometry degenerates to Euclidean | Scale-amplifying |
| Dim | Full Rep. | Rand. FS | SVD | MRL | MRL-E | Unsup. HERL | Sup. HERL | CSR |
| 8 | - | 2.36 | 19.14 | 62.19 | 57.45 | 63.90 | 64.60 | 67.78 |
| 16 | - | 12.06 | 46.02 | 67.91 | 67.05 | 67.89 | 69.90 | 69.17 |
| 32 | - | 32.91 | 60.78 | 69.46 | 68.60 | 70.20 | 71.23 | 70.15 |
| 64 | - | 49.91 | 67.04 | 70.17 | 69.61 | 70.71 | 71.14 | 70.94 |
| 128 | - | 60.91 | 69.63 | 70.52 | 70.12 | 71.00 | 71.72 | 70.99 |
| 256 | - | 65.75 | 70.67 | 70.62 | 70.36 | 71.20 | 71.89 | 71.31 |
| Backbone | Input dim | LR | Epochs | Batch size | Optimizer | Curvature |
| ResNet50 | 2048 | 1e-3 | 10 | 2048 | RiemannianAdam | 0.9 |
| Dim | Full Rep. | Rand. LP | SVD | MRL | MRL-E | Unsup. HERL | Sup. HERL | CSR |
| 8 | - | 4.56 | 2.34 | 66.63 | 56.66 | 73.52 | 74.16 | 73.62 |
| 16 | - | 11.29 | 7.17 | 73.53 | 71.94 | 74.75 | 75.04 | 74.75 |
| 32 | - | 27.21 | 20.46 | 75.03 | 74.48 | 75.48 | 76.02 | 75.44 |
| 64 | - | 49.47 | 48.10 | 75.82 | 75.35 | 76.01 | 76.53 | 75.88 |
| 128 | - | 65.70 | 67.24 | 76.30 | 75.80 | 76.29 | 76.88 | 76.24 |
| 256 | - | 72.43 | 74.59 | 76.47 | 76.22 | 76.55 | 77.01 | 76.25 |
| Dataset | Backbone | Input dim | LR | Epochs | Batch size | Optimizer | Curvature |
| MTOPIntent | NV-Embed-V2 | 4096 | 4e-5 | 10 | 128 | RiemannianAdam | 0.9 |
| Banking77 | |||||||
| TweetSentiment | |||||||
| BiorxivP2P | NV-Embed-V2 | 4096 | 1e-3 | 10 | 128 | RiemannianAdam | 0.9 |
| BiorxivS2S | 1e-3 | ||||||
| TwentyNews | 1e-3 |
| Dim | Text Classification Top-1 Acc (%) | Text Clustering Top-1 Acc (%) | Text Retrieval NDCG@10 (%) | ||||||||
| Category | Model | MTOPIntent | Banking77 | TweetSentiment | BiorxivP2P | BiorxivS2S | TwentyNews | FiQA2018 | NFCorpus | SciFACT | |
| Full Rep | NV-Embed-V2 | 4096 | 93.58 | 92.20 | 79.73 | 53.61 | 49.60 | 64.82 | 62.65 | 43.97 | 77.93 |
| MRL | Stella-1.5B-v5 | 256 | 90.45 | 86.14 | 76.75 | 50.81 | 46.42 | 60.07 | 55.59 | 36.97 | 77.48 |
| Jina-v3 | 256 | 78.81 | 84.08 | 73.81 | 38.14 | 34.39 | 51.96 | 55.73 | 36.63 | 66.63 | |
| Nomic-Embed-V1.5 | 256 | 72.47 | 83.69 | 59.20 | 38.19 | 31.83 | 48.56 | 35.00 | 32.54 | 68.24 | |
| Gecko-Embed-004(Google) | 256 | 77.82 | 86.01 | 72.97 | 36.28 | 33.09 | 50.60 | 55.54 | 37.81 | 70.86 | |
Representation Learning
Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.
See all 855 papers