Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.
Figures & tables
Figure 1: Schematic comparison of standard MRL and SpecMRL. Text embeddings are shown as points in the middle, with colors and marker shapes denoting distinct semantic groups. MRL retains coordinate prefixes in the original basis {ei} as compact representations, whereas SpecMRL performs the readout in an ordered spectral basis {ui} derived from the parameter structure of WProj . The 2D views and enlarged neighborhoods of q (red star) illustrate how this change better organizes the semantic space and supports more effective low-dimensional representations.
Figure 2: Illustration and pseudocode of SpecMRL. (a) Embeddings are rotated into a learned spectral basis before prefix readouts. (b) The basis is updated by one warm-started orthogonal iteration per training step and cached for the next step. mrl_loss denotes the original Matryoshka objective.
Qwen3-0.6B
Qwen2-0.5B
Llama3.2-1B
τ
MRL
SpecMRL
MRL
SpecMRL
MRL
SpecMRL
1
60.97
60.94 ( − 0.03)
58.96
58.74 ( − 0.22)
61.92
61.87 ( − 0.05)
2
60.54
61.61 (+1.07)
58.67
59.44 (+0.77)
61.74
62.42 (+0.68)
4
59.40
61.28 (+1.88)
58.32
59.20 (+0.88)
61.36
62.23 (+0.87)
8
57.88
60.46 (+2.58)
57.26
58.44 (+1.18)
60.62
61.80 (+1.18)
16
54.73
58.53 (+3.80)
55.23
56.61 (+1.38)
59.42
60.91 (+1.49)
Table 1: Performance of SpecMRL averaged across MTEB tasks, with task-level breakdown provided in Table 15 . τ controls the reduction ratio, reducing the output dimension to 1/τ of its original size. Colored entries highlight improvements over vanilla MRL, while bold text marks the best result for each backbone. Parenthetical values denote the gains of SpecMRL over MRL.
τ
SpecMRL
FeatMRL (β=0)
FeatMRL (β=0.1)
FeatMRL (β=0.5)
FeatMRL (β=0.9)
1
60.94
60.72 ( − 0.22)
60.68 ( − 0.26)
60.69 ( − 0.25)
60.38 ( − 0.56)
2
61.61
60.45 ( − 1.16)
60.36 ( − 1.25)
60.42 ( − 1.19)
60.01 ( − 1.60)
4
61.28
59.99 ( − 1.29)
59.84 ( − 1.44)
59.91 ( − 1.37)
59.35 ( − 1.93)
8
60.46
58.82 ( − 1.64)
58.68 ( − 1.78)
58.75 ( − 1.71)
58.14 ( − 2.32)
16
58.53
55.83 ( − 2.70)
55.62 ( − 2.91)
55.90 ( − 2.63)
55.15 ( − 3.38)
Table 2: Comparison of SpecMRL and FeatMRL on MTEB. Bold values indicate the best performance, while parenthetical values denote performance drops relative to SpecMRL.
τ
D
4D
16D
64D
128D
256D
1
60.77
60.73
60.92
60.72
60.94
60.90
2
61.02
61.27
61.34
61.45
61.61
61.55
4
60.65
60.84
61.04
61.14
61.28
61.26
8
59.64
60.00
60.27
60.43
60.46
60.44
16
57.80
58.12
58.30
58.38
58.53
58.41
AVG.
59.98
60.19
60.37
60.42
60.56
60.51
Table 3: Hyperparameter search for the over-parameterization scale of WProj . Darker shades indicate better relative performance; the top-two results are shown in bold and underline .
Solver
Qwen3
Qwen2
Llama3
SVD
−4.34%
−3.36%
−8.35%
Eigh
−4.59%
−3.52%
−4.57%
Power
−0.62%
−0.53%
−0.74%
Table 4: Training throughput degradation of SpecMRL with different eigen solvers.
τ
MRL
SpecMRL
(a) MRL w. Gram
(b) MRL w. Orth
1
60.97
60.94
61.35 ( + 0.41)
60.59 ( − 0.35)
2
60.54
61.61
60.80 ( − 0.81)
60.91 ( − 0.70)
4
59.40
61.28
59.64 ( − 1.64)
60.24 ( − 1.04)
8
57.88
60.46
57.70 ( − 2.76)
59.28 ( − 1.18)
16
54.73
58.53
54.11 ( − 4.42)
57.19 ( − 1.34)
Table 5: Results of ablation studies. Parenthetical values denote drops relative to SpecMRL.
Figure 8
Qwen3-0.6B
Qwen2-0.5B
Llama3.2-1B
Bands
MRL
SpecMRL
MRL
SpecMRL
MRL
SpecMRL
0:20
60.97
60.94 ( − 0.03)
58.96
58.74 ( − 0.22)
61.92
61.87 ( − 0.05)
0:19
60.94
61.35 ( + 0.41)
59.00
59.20 ( + 0.20)
61.93
62.37 ( + 0.44)
0:10
60.54
61.61 (+1.07)
58.67
59.44 (+0.77)
61.74
62.42 (+0.68)
Table 7: Comparison of SpecMRL performance with and without the final spectral bands.
Figure 4: MTEB subset results evaluated every 100 steps. The advantage of SpecMRL is consistent throughout the training process.
τ
MRL
ESE (L2C)
MIPIC (SIA)
SpecMRL
1
60.97
57.97 ( − 3.00)
60.42 ( − 0.55)
60.94 ( − 0.03)
2
60.54
57.26 ( − 3.28)
59.93 ( − 0.61)
61.61 ( + 1.07)
4
59.40
56.38 ( − 3.02)
59.08 ( − 0.32)
61.28 ( + 1.88)
8
57.88
55.18 ( − 2.70)
57.74 ( − 0.14)
60.46 ( + 2.58)
16
54.73
50.26 ( − 4.47)
55.09 ( + 0.36)
58.53 ( + 3.80)
Table 8: Comparison of SpecMRL with MRL variants on Qwen3-0.6B.
Default
Uniform Weights
Sparse Supervision
τ
MRL
SpecMRL
MRL
SpecMRL
MRL
SpecMRL
1
60.97
60.94 ( − 0.03)
60.35
60.37 (+0.02)
60.73
60.76 ( + 0.03)
2
60.54
61.61 (+1.07)
60.09
61.79 (+1.70)
60.11
61.53 (+1.42)
4
59.40
61.28 (+1.88)
59.31
61.60 (+2.29)
58.76
61.20 (+2.44)
8
57.88
60.46 (+2.58)
58.38
61.04 (+2.66)
55.88
60.15 (+4.27)
16
54.73
58.53 (+3.80)
56.48
59.30 (+2.82)
49.94
57.23 (+7.29)
Table 9: Robustness to different MRL training configurations.
τ
B=1
B=64
B=256
B=1024
B=2048
1
140 ( 1.00× )
5,207 ( 1.00× )
7,103 ( 1.00× )
7,859 ( 1.00× )
7,961 ( 1.00× )
2
176 ( 1.26× )
7,713 ( 1.48× )
12,558 ( 1.77× )
14,605 ( 1.86× )
14,791 ( 1.86× )
4
205 ( 1.46× )
9,891 ( 1.90× )
20,174 ( 2.84× )
25,508 ( 3.25× )
25,906 ( 3.25× )
8
231 ( 1.65× )
11,414 ( 2.19× )
28,986 ( 4.08× )
40,594 ( 5.17× )
41,282 ( 5.19× )
16
352 ( 2.52× )
12,319 ( 2.37× )
36,712 ( 5.17× )
56,884 ( 7.24× )
57,814 ( 7.26× )
Table 10: Retrieval throughput (QPS) across compression ratios τ and query batch sizes B , with output dimension 1024/τ . Parentheses show speedups over τ=1 at the same B . Bold highlights the retrieval speedup of the representation with the highest average MTEB score in Table 1 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
τ
SpecMRL
MRL w. Linear
1
60.94
60.92 ( − 0.02)
2
61.61
60.72 ( − 0.89)
4
61.28
60.14 ( − 1.14)
8
60.46
59.13 ( − 1.33)
16
58.53
57.03 ( − 1.50)
Appendix
Table 11: Comparison with learnable readout.
k
Rand
MRL
SpecMRL
64
0.0628
0.2522
0.3025 (+ 5.03%)
128
0.1237
0.2783
0.4295 (+ 15.12%)
256
0.2513
0.3171
0.4715 (+ 15.44%)
512
0.4999
0.5197
0.5518 (+ 3.21%)
Appendix
Table 12: Readout subspace alignment.
Figure 5: Average projection energy fractions.
Table 13: MRL training configurations for different backbone models.
Category
Metric
Dataset
STS. (3)
Spearman
STS17 ( Cer et al., 2017b )
SICK-R ( Marelli et al., 2014 )
STSBenchmark ( Cer et al., 2017a )
Class. (3)
Accuracy
Banking77Classification ( Casanueva et al., 2020 )
EmotionClassification ( Saravia et al., 2018 )
MassiveIntentClassification ( FitzGerald et al., 2022 )
Appendix
Table 14: MTEB evaluation metrics and the 15-dataset subset used for scaling experiments.
STS.
Class.
Cluster.
PairClass.
Rerank.
Retr.
Sum.
Avg
# Datasets
10
12
11
3
4
15
1
56
Qwen3-0.6B
1024 ( τ=1 )
80.77
74.25
48.72
72.74
48.93
49.02
30.38
60.97
80.24
74.45
49.06
72.44
47.67
49.30
29.66
60.94 ( − 0.03)
512 ( τ=2 )
80.87
73.99
48.43
72.09
48.25
48.10
29.92
60.54
80.83
75.11
49.43
74.49
50.98
49.24
30.46
61.61 (+1.07)
Appendix
Table 15: Task-level breakdown of MTEB results in Table 1 . Shaded rows denote SpecMRL, bold values indicate better overall results, parenthetical values report gains over MRL under each setup.
STS.
Class.
Cluster.
PairClass.
Rerank.
Retr.
Sum.
Avg
# Datasets
10
12
11
3
4
15
1
56
τ=1
80.24
74.45
49.06
72.44
47.67
49.30
29.66
60.94
β=0
80.70
73.08
48.86
72.31
49.35
48.99
29.80
60.72 ( − 0.22)
β=0.1
80.73
72.54
48.77
72.27
49.82
49.10
31.06
60.68 ( − 0.26)
β=0.5
80.65
72.30
48.79
72.87
49.54
49.29
31.65
60.69 ( − 0.25)
β=0.9
80.78
71.97
48.85
72.37
48.71
48.72
29.69
60.38 ( − 0.56)
Appendix
Table 16: Task-level breakdown of FeatMRL variants. Bold and underlined average scores indicate the best and second-best FeatMRL variants within each group.
STS.
Class.
Cluster.
PairClass.
Rerank.
Retr.
Sum.
Avg
# Datasets
10
12
11
3
4
15
1
56
1024 ( τ=1 )
80.52
81.40
52.79
70.19
48.99
50.69
28.91
63.54
80.74
81.15
52.36
68.12
47.86
50.59
30.38
63.25 (-0.29)
512 ( τ=2 )
80.53
81.34
52.74
69.44
48.43
49.80
28.28
63.19
81.35
81.50
52.64
71.39
50.95
50.59
29.75
63.88 (+0.69)
256 ( τ=4 )
80.20
81.12
52.49
68.01
46.83
47.52
27.82
62.23
Appendix
Table 17: Task-level breakdown of MTEB in an over-training regime on Qwen3-0.6B, with training steps extended from 3K to 15K. The improvement of SpecMRL remains consistent with Table 1 , suggesting that the gains of SpecMRL persist beyond faster convergence.
Method
STS.
Class.
Cluster.
PairClass.
Rerank.
Retr.
Sum.
Avgw/oCls
1024 (τ=1)
MRL
80.77
74.25
48.72
72.74
48.93
49.02
30.38
57.35
ESE
81.12
59.88
48.61
70.96
49.12
49.49
29.93
57.44 ( + 0.10)
MIPIC
79.88
73.05
48.75
69.69
48.51
49.21
30.77
56.98 ( − 0.37)
SpecMRL
80.24
74.45
49.06
72.44
47.67
49.30
29.66
57.26 ( − 0.09)
512 (τ=2)
MRL
80.87
73.99
48.43
72.09
48.25
48.10
29.92
56.87
ESE
80.92
59.12
48.47
69.47
48.49
48.23
28.85
56.75 ( − 0.12)
Appendix
Table 18: Task-level breakdown of MTEB results for SpecMRL and existing MRL variants. Column Avgw/oCls denotes the average MTEB results over non-classification MTEB datasets.
Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer-induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.
Arghamitra Talukder, Philippe Chlenski, Itsik Pe'er
Although multi-scales representation learning enables elastic-dimension embeddings, nested subspaces often suffer from dimensional redundancy and spectral collapse. To address this, we introduce MIC, a framework that optimizes the geometric landscape of multi-granular embeddings through isotropic subspace alignment. MIC employs Soft Collapse Regularization (SCR) to mitigate redundancy between prefix and residual subspaces via cross-correlation penalties, alongside Spectral Isotropy Regularization (SIR) to ensure hyper-spherical uniformity in low-dimensional prefixes. By unifying these strategies through a self-distillation objective, MIC generates semantically dense representations that maintain high discriminative power. Our experiments demonstrate that MIC significantly outperforms standard baselines, particularly in high-compression scenarios where maintaining informational capacity is most critical.
Hanoi University of Science and Technology, Hanoi, Vietnam · VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam +1
Matryoshka Representation Learning (MRL) is a widely adopted approach for training text encoders so they provide useful text representations at various sizes, available by simply truncating the resulting vectors at sizes pre-determined at training time. Recent works have shown that randomly truncating text embeddings has minimal impact in downstream performance unless vectors are reduced in size by at least 70%, suggesting that embeddings are already robust to truncation without the use of MRL. However, no prior work has compared random truncation to MRL, so it is unclear how the two methods compare as effective embedding reduction methods. In this paper, we study this by applying the same truncation used by MRL to models trained with and without MRL. Our results across several models and downstream tasks show that, unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL. This suggests that truncation robustness may not necessarily come from MRL, and that the choice of spending the additional training cost of MRL depends on whether heavy truncation is desired. We make our code available for reproduction.
Sotaro Takeshita, Yurina Takeshita, Simone Paolo Ponzetto +1
Data and Web Science Group, University of Mannheim, Germany · Independent Researcher · NEC Laboratories Europe, Heidelberg, Germany