SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Zhongxin Chen; Songhao Wu; Heng Cui; Cong Li; Xiao Zhou; Rui Yan

cs.CLOct 19, 2026

SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Authors: Zhongxin Chen, Songhao Wu, Heng Cui, Cong Li, Xiao Zhou, Rui Yan

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · School of Computer Science, Wuhan University · Lenovo Group

Abstract

Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.

Figures & tables

Figure 1: Schematic comparison of standard MRL and SpecMRL. Text embeddings are shown as points in the middle, with colors and marker shapes denoting distinct semantic groups. MRL retains coordinate prefixes in the original basis

\{e_{i}\}

as compact representations, whereas SpecMRL performs the readout in an ordered spectral basis

\{u_{i}\}

derived from the parameter structure of

{\bm{W}}_{\mathrm{Proj}}

. The 2D views and enlarged neighborhoods of

q

(red star) illustrate how this change better organizes the semantic space and supports more effective low-dimensional representations.

Figure 2: Illustration and pseudocode of SpecMRL. (a) Embeddings are rotated into a learned spectral basis before prefix readouts. (b) The basis is updated by one warm-started orthogonal iteration per training step and cached for the next step. mrl_loss denotes the original Matryoshka objective.

	Qwen3-0.6B		Qwen2-0.5B		Llama3.2-1B
$\mathbf{\tau}$	MRL	SpecMRL	MRL	SpecMRL	MRL	SpecMRL
1	60.97	60.94 ( $-$ 0.03)	58.96	58.74 ( $-$ 0.22)	61.92	61.87 ( $-$ 0.05)
2	60.54	61.61 (+1.07)	58.67	59.44 (+0.77)	61.74	62.42 (+0.68)
4	59.40	61.28 (+1.88)	58.32	59.20 (+0.88)	61.36	62.23 (+0.87)
8	57.88	60.46 (+2.58)	57.26	58.44 (+1.18)	60.62	61.80 (+1.18)
16	54.73	58.53 (+3.80)	55.23	56.61 (+1.38)	59.42	60.91 (+1.49)

Table 1: Performance of SpecMRL averaged across MTEB tasks, with task-level breakdown provided in Table 15 .

\tau

controls the reduction ratio, reducing the output dimension to

1/\tau

of its original size. Colored entries highlight improvements over vanilla MRL, while bold text marks the best result for each backbone. Parenthetical values denote the gains of SpecMRL over MRL.

$\mathbf{\tau}$	SpecMRL	FeatMRL $(\beta=0)$	FeatMRL $(\beta=0.1)$	FeatMRL $(\beta=0.5)$	FeatMRL $(\beta=0.9)$
1	60.94	60.72 ( $-$ 0.22)	60.68 ( $-$ 0.26)	60.69 ( $-$ 0.25)	60.38 ( $-$ 0.56)
2	61.61	60.45 ( $-$ 1.16)	60.36 ( $-$ 1.25)	60.42 ( $-$ 1.19)	60.01 ( $-$ 1.60)
4	61.28	59.99 ( $-$ 1.29)	59.84 ( $-$ 1.44)	59.91 ( $-$ 1.37)	59.35 ( $-$ 1.93)
8	60.46	58.82 ( $-$ 1.64)	58.68 ( $-$ 1.78)	58.75 ( $-$ 1.71)	58.14 ( $-$ 2.32)
16	58.53	55.83 ( $-$ 2.70)	55.62 ( $-$ 2.91)	55.90 ( $-$ 2.63)	55.15 ( $-$ 3.38)

Table 2: Comparison of SpecMRL and FeatMRL on MTEB. Bold values indicate the best performance, while parenthetical values denote performance drops relative to SpecMRL.

$\tau$	$\mathrm{D}$	$\mathrm{4D}$	$\mathrm{16D}$	$\mathrm{64D}$	$\mathrm{128D}$	$\mathrm{256D}$
1	60.77	60.73	60.92	60.72	60.94	60.90
2	61.02	61.27	61.34	61.45	61.61	61.55
4	60.65	60.84	61.04	61.14	61.28	61.26
8	59.64	60.00	60.27	60.43	60.46	60.44
16	57.80	58.12	58.30	58.38	58.53	58.41
AVG.	59.98	60.19	60.37	60.42	60.56	60.51

Table 3: Hyperparameter search for the over-parameterization scale of

{\bm{W}}_{\mathrm{Proj}}

. Darker shades indicate better relative performance; the top-two results are shown in bold and underline .

Solver	Qwen3	Qwen2	Llama3
SVD	$-4.34\%$	$-3.36\%$	$-8.35\%$
Eigh	$-4.59\%$	$-3.52\%$	$-4.57\%$
Power	$-0.62\%$	$-0.53\%$	$-0.74\%$

Table 4: Training throughput degradation of SpecMRL with different eigen solvers.

$\mathbf{\tau}$	MRL	SpecMRL	(a) MRL w. Gram	(b) MRL w. Orth
1	60.97	60.94	61.35 ( $+$ 0.41)	60.59 ( $-$ 0.35)
2	60.54	61.61	60.80 ( $-$ 0.81)	60.91 ( $-$ 0.70)
4	59.40	61.28	59.64 ( $-$ 1.64)	60.24 ( $-$ 1.04)
8	57.88	60.46	57.70 ( $-$ 2.76)	59.28 ( $-$ 1.18)
16	54.73	58.53	54.11 ( $-$ 4.42)	57.19 ( $-$ 1.34)

Table 5: Results of ablation studies. Parenthetical values denote drops relative to SpecMRL.

	Qwen3-0.6B		Qwen2-0.5B		Llama3.2-1B
$\mathrm{Bands}$	MRL	SpecMRL	MRL	SpecMRL	MRL	SpecMRL
$0:20$	60.97	60.94 ( $-$ 0.03)	58.96	58.74 ( $-$ 0.22)	61.92	61.87 ( $-$ 0.05)
$0:19$	60.94	61.35 ( $+$ 0.41)	59.00	59.20 ( $+$ 0.20)	61.93	62.37 ( $+$ 0.44)
$0:10$	60.54	61.61 (+1.07)	58.67	59.44 (+0.77)	61.74	62.42 (+0.68)

Table 7: Comparison of SpecMRL performance with and without the final spectral bands.

Figure 4: MTEB subset results evaluated every 100 steps. The advantage of SpecMRL is consistent throughout the training process.

$\bm{\tau}$	MRL	ESE (L2C)	MIPIC (SIA)	SpecMRL
1	60.97	57.97 ( $-$ 3.00)	60.42 ( $-$ 0.55)	60.94 ( $-$ 0.03)
2	60.54	57.26 ( $-$ 3.28)	59.93 ( $-$ 0.61)	61.61 ( $+$ 1.07)
4	59.40	56.38 ( $-$ 3.02)	59.08 ( $-$ 0.32)	61.28 ( $+$ 1.88)
8	57.88	55.18 ( $-$ 2.70)	57.74 ( $-$ 0.14)	60.46 ( $+$ 2.58)
16	54.73	50.26 ( $-$ 4.47)	55.09 ( $+$ 0.36)	58.53 ( $+$ 3.80)

Table 8: Comparison of SpecMRL with MRL variants on Qwen3-0.6B.

	Default		Uniform Weights		Sparse Supervision
$\bm{\tau}$	MRL	SpecMRL	MRL	SpecMRL	MRL	SpecMRL
1	60.97	60.94 ( $-$ 0.03)	60.35	60.37 (+0.02)	60.73	60.76 ( $+$ 0.03)
2	60.54	61.61 (+1.07)	60.09	61.79 (+1.70)	60.11	61.53 (+1.42)
4	59.40	61.28 (+1.88)	59.31	61.60 (+2.29)	58.76	61.20 (+2.44)
8	57.88	60.46 (+2.58)	58.38	61.04 (+2.66)	55.88	60.15 (+4.27)
16	54.73	58.53 (+3.80)	56.48	59.30 (+2.82)	49.94	57.23 (+7.29)

Table 9: Robustness to different MRL training configurations.

$\tau$	$B=1$	$B=64$	$B=256$	$B=1024$	$B=2048$
1	140 ( $1.00\times$ )	5,207 ( $1.00\times$ )	7,103 ( $1.00\times$ )	7,859 ( $1.00\times$ )	7,961 ( $1.00\times$ )
2	176 ( $1.26\times$ )	7,713 ( $1.48\times$ )	12,558 ( $1.77\times$ )	14,605 ( $1.86\times$ )	14,791 ( $\bm{1.86\times}$ )
4	205 ( $1.46\times$ )	9,891 ( $1.90\times$ )	20,174 ( $2.84\times$ )	25,508 ( $3.25\times$ )	25,906 ( ${3.25\times}$ )
8	231 ( $1.65\times$ )	11,414 ( $2.19\times$ )	28,986 ( $4.08\times$ )	40,594 ( $5.17\times$ )	41,282 ( $5.19\times$ )
16	352 ( $2.52\times$ )	12,319 ( $2.37\times$ )	36,712 ( $5.17\times$ )	56,884 ( $7.24\times$ )	57,814 ( $7.26\times$ )

Table 10: Retrieval throughput (QPS) across compression ratios

\tau

and query batch sizes

B

, with output dimension

1024/\tau

. Parentheses show speedups over

\tau=1

at the same

B

. Bold highlights the retrieval speedup of the representation with the highest average MTEB score in Table 1 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$\mathbf{\tau}$	SpecMRL	MRL w. Linear
1	60.94	60.92 ( $-$ 0.02)
2	61.61	60.72 ( $-$ 0.89)
4	61.28	60.14 ( $-$ 1.14)
8	60.46	59.13 ( $-$ 1.33)
16	58.53	57.03 ( $-$ 1.50)

Appendix

Table 11: Comparison with learnable readout.

$k$	Rand	MRL	SpecMRL
64	0.0628	0.2522	0.3025 (+ 5.03%)
128	0.1237	0.2783	0.4295 (+ 15.12%)
256	0.2513	0.3171	0.4715 (+ 15.44%)
512	0.4999	0.5197	0.5518 (+ 3.21%)

Appendix

Table 12: Readout subspace alignment.

Figure 5: Average projection energy fractions.

Category	Metric	Dataset
STS. (3)	Spearman	STS17 ( Cer et al., 2017b )
		SICK-R ( Marelli et al., 2014 )
		STSBenchmark ( Cer et al., 2017a )
Class. (3)	Accuracy	Banking77Classification ( Casanueva et al., 2020 )
		EmotionClassification ( Saravia et al., 2018 )
		MassiveIntentClassification ( FitzGerald et al., 2022 )

Appendix

Table 14: MTEB evaluation metrics and the 15-dataset subset used for scaling experiments.

	STS.	Class.	Cluster.	PairClass.	Rerank.	Retr.	Sum.	Avg
# Datasets	10	12	11	3	4	15	1	56
	Qwen3-0.6B
1024 ( $\tau=1$ )	80.77	74.25	48.72	72.74	48.93	49.02	30.38	60.97
1024 ( $\tau=1$ )	80.24	74.45	49.06	72.44	47.67	49.30	29.66	60.94 ( $-$ 0.03)
512 ( $\tau=2$ )	80.87	73.99	48.43	72.09	48.25	48.10	29.92	60.54
512 ( $\tau=2$ )	80.83	75.11	49.43	74.49	50.98	49.24	30.46	61.61 (+1.07)

Appendix

Table 15: Task-level breakdown of MTEB results in Table 1 . Shaded rows denote SpecMRL, bold values indicate better overall results, parenthetical values report gains over MRL under each setup.

	STS.	Class.	Cluster.	PairClass.	Rerank.	Retr.	Sum.	Avg
# Datasets	10	12	11	3	4	15	1	56
$\mathbf{\tau=1}$	80.24	74.45	49.06	72.44	47.67	49.30	29.66	60.94
$\,\beta=0$	80.70	73.08	48.86	72.31	49.35	48.99	29.80	60.72 ( $-$ 0.22)
$\,\beta=0.1$	80.73	72.54	48.77	72.27	49.82	49.10	31.06	60.68 ( $-$ 0.26)
$\,\beta=0.5$	80.65	72.30	48.79	72.87	49.54	49.29	31.65	60.69 ( $-$ 0.25)
$\,\beta=0.9$	80.78	71.97	48.85	72.37	48.71	48.72	29.69	60.38 ( $-$ 0.56)

Appendix

Table 16: Task-level breakdown of FeatMRL variants. Bold and underlined average scores indicate the best and second-best FeatMRL variants within each group.

	STS.	Class.	Cluster.	PairClass.	Rerank.	Retr.	Sum.	Avg
# Datasets	10	12	11	3	4	15	1	56
1024 ( $\tau=1$ )	80.52	81.40	52.79	70.19	48.99	50.69	28.91	63.54
1024 ( $\tau=1$ )	80.74	81.15	52.36	68.12	47.86	50.59	30.38	63.25 (-0.29)
512 ( $\tau=2$ )	80.53	81.34	52.74	69.44	48.43	49.80	28.28	63.19
512 ( $\tau=2$ )	81.35	81.50	52.64	71.39	50.95	50.59	29.75	63.88 (+0.69)
256 ( $\tau=4$ )	80.20	81.12	52.49	68.01	46.83	47.52	27.82	62.23

Appendix

Table 17: Task-level breakdown of MTEB in an over-training regime on Qwen3-0.6B, with training steps extended from 3K to 15K. The improvement of SpecMRL remains consistent with Table 1 , suggesting that the gains of SpecMRL persist beyond faster convergence.

	Method	STS.	Class.	Cluster.	PairClass.	Rerank.	Retr.	Sum.	$\mathrm{Avg}_{\mathrm{w/o\,Cls}}$
1024 $\mathbf{(\tau=1)}$	MRL	80.77	74.25	48.72	72.74	48.93	49.02	30.38	57.35
	ESE	81.12	59.88	48.61	70.96	49.12	49.49	29.93	57.44 ( $+$ 0.10)
	MIPIC	79.88	73.05	48.75	69.69	48.51	49.21	30.77	56.98 ( $-$ 0.37)
	SpecMRL	80.24	74.45	49.06	72.44	47.67	49.30	29.66	57.26 ( $-$ 0.09)
512 $\mathbf{(\tau=2)}$	MRL	80.87	73.99	48.43	72.09	48.25	48.10	29.92	56.87
	ESE	80.92	59.12	48.47	69.47	48.49	48.23	28.85	56.75 ( $-$ 0.12)

Appendix

Table 18: Task-level breakdown of MTEB results for SpecMRL and existing MRL variants. Column

\mathrm{Avg}_{\mathrm{w/o\ Cls}}

denotes the average MTEB results over non-classification MTEB datasets.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning
May 9, 2026Arghamitra Talukder, Philippe Chlenski, Itsik Pe'erRepresentation Learning Representation Identifiability
Computer Science, Columbia University
MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment
May 28, 2026Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu PhamRepresentation Learning Hierarchical Representation Learning
Hanoi University of Science and Technology, Hanoi, Vietnam · VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam +1
To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios
May 15, 2026Sotaro Takeshita, Yurina Takeshita, Simone Paolo Ponzetto +1Text Embeddings Representation Learning
Data and Web Science Group, University of Mannheim, Germany · Independent Researcher · NEC Laboratories Europe, Heidelberg, Germany

May 9, 2026J/K move · Enter open · S save

Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning

Arghamitra Talukder, Philippe Chlenski, Itsik Pe'er

Computer Science, Columbia University

Representation Learning Representation Identifiability Hierarchical Representation Learning Matryoshka Representation Learning Invariant Representation Learning

Open paper AskPDF arXiv

Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer-induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.

Abstract

Representation Learning

Explore similar work

Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning

MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment

To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios