Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

Ailiang Lin; Zhuoyun Li; Yusong Wang; Keyu Mao; Kotaro Funakoshi; Manabu Okumura

cs.CLOct 16, 2026

Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

Authors: Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, Manabu Okumura

Organizations: Institute of Science Tokyo · Tencent

Abstract

While deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs. Our analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. We hypothesize that language-specific variation can be disentangled from semantic content to improve embedding quality. To this end, we propose Paired-Residual Subspace Removal (PRSR), a training-free inference-time intervention that steers LLM representations toward a more language-invariant semantic space. PRSR first constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. It then estimates layer-wise low-rank subspaces from these residuals and suppresses the corresponding components in the last-token hidden states. We conduct more than 200 monolingual evaluations across 18 languages using MTEB and MMTEB, along with cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.

Figures & tables

Figure 1: Illustration of PRSR . (a) Training-free text embeddings remain separated by languages despite shared semantics. (b) PRSR estimates language-sensitive subspaces from paired residuals of parallel sentences and removes the corresponding components within the frozen LLM to improve semantic alignment.

Figure 2: Language-conditioned geometry of training-free text embeddings on FLORES+. (a) t-SNE visualization showing language-specific clustering. (b) Cosine-similarity margins between anchor–translation and anchor–negative pairs; in 36.2% of comparisons, negatives are more similar to anchors than translations.

Figure 3: Overview of PRSR . (a) Paired-residual subspace construction. At each selected layer, we first construct paired residuals by subtracting the group mean from the last-token representations of parallel sentences. We then stack the residuals across groups and languages, and apply SVD to estimate a low-rank language-sensitive subspace. (b) Inference-time intervention. We remove the projections of last-token hidden states onto these subspaces at the corresponding layers of the frozen LLM, steering representations toward a more language-invariant semantic space.

Method	en	zh	fr	ru	de	ja †	it †	es	pt †	ar	id †	sw	vi	ko †	bn †	th	te †	ms †	AVG
PromptEOL	46.8	38.7	43.4	45.0	39.4	46.6	55.1	47.9	47.9	41.2	49.6	42.8	44.7	53.0	32.0	40.8	18.4	25.2	43.8
w/ PRSR	54.6	42.1	53.8	49.1	45.0	51.1	66.0	57.7	62.0	47.1	59.3	45.7	54.5	60.2	36.4	49.7	28.8	51.5	51.1 ↑7.3
Pretended CoT	49.0	40.0	44.6	46.7	40.3	47.2	58.6	49.5	51.9	41.9	53.0	40.9	46.0	49.3	32.2	42.1	19.6	34.0	45.3
w/ PRSR	54.6	42.5	53.3	50.4	45.8	51.2	67.5	59.1	63.4	47.9	59.2	45.7	54.7	56.6	39.2	50.0	28.1	52.7	51.5 ↑6.2
ECHO	48.4	38.4	44.3	44.2	38.4	47.5	56.5	48.3	47.1	42.4	49.0	41.9	40.5	47.5	25.8	39.9	20.7	21.5	43.7
w/ PRSR	54.1	40.7	52.3	47.4	42.6	50.7	66.9	57.6	60.6	48.5	55.3	43.9	51.1	55.2	36.0	46.9	23.7	41.2	49.7 ↑6.0

Table 1: Monolingual performance across 18 languages on MTEB and MMTEB. Each language column reports the average over its monolingual tasks, and AVG denotes the overall mean across all 208 evaluations. Languages marked with

\dagger

were not used for subspace estimation. The best results are highlighted in bold . PRSR significantly outperforms each corresponding baseline (paired

t

-test,

p<0.01

Method	FLORES			Tatoeba	BUCC.v2
Method	En $\rightarrow$ X	X $\rightarrow$ En	Mean	Tatoeba	BUCC.v2
PromptEOL	27.42	23.43	25.43	12.95	44.25
w/ PRSR	48.61	50.16	49.39	47.85	97.13
Pretended CoT	30.74	24.97	27.85	16.10	49.31
w/ PRSR	48.88	50.07	49.47	48.32	97.43
ECHO	34.42	31.13	32.78	24.60	66.20

Table 2: Cross-lingual bitext mining performance of training-free embedding methods on FLORES, Tatoeba, and BUCC.v2. For FLORES, we report English-to-X and X-to-English results together with their mean. The best results are highlighted in bold .

Method	MTEB	Tatoeba	FLORES	BUCC.v2
Baseline	46.82	12.95	25.43	44.25
Random	46.80	12.20	24.92	43.13
Language-Mean	50.00	42.95	40.90	90.91
Shuffled-Pair	52.34	43.90	44.33	95.73
PRSR	54.62	47.85	49.39	97.13

Table 3: Performance comparison of different subspace construction strategies using LLaMA-2-7B with PromptEOL. MTEB denotes MTEB (eng, v2). The best results are highlighted in bold .

Intervention	MTEB	Tatoeba	FLORES	BUCC.v2
$h_{\ell}$ (Baseline)	46.82	12.95	25.43	44.25
$h_{\ell}+P_{\ell}$	24.16	0.06	0.04	0.06
$h_{\ell}+2P_{\ell}$	23.77	0.05	0.03	0.04
$h_{\ell}+3P_{\ell}$	23.64	0.05	0.03	0.03
$h_{\ell}-P_{\ell}$ ( PRSR )	54.62	47.85	49.39	97.13

Table 4: Comparison of subspace removal and amplification using LLaMA-2-7B with PromptEOL. MTEB denotes MTEB (eng, v2), and

P_{\ell}=(h_{\ell}B_{\ell})B_{\ell}^{\top}

is the projection onto the estimated language-sensitive subspace. The best results are highlighted in bold .

Method	Mono. Avg.	Tatoeba	FLORES	BUCC.v2
LLaMA-2-7B
PromptEOL	43.80	12.95	25.43	44.25
w/ PRSR	51.09	47.85	49.39	97.13
Mistral-7B
PromptEOL	49.40	29.21	37.99	49.97
w/ PRSR	55.18	56.30	55.96	97.82

Table 5: Performance of PRSR across different LLM backbones with PromptEOL. Mono. Avg. denotes the average score over 208 monolingual evaluations spanning 18 languages. The best results for each backbone are highlighted in bold .

Figure 4: Effect of the subspace rank

q

on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL. Rank 0 denotes the baseline, and

K

is the number of languages used for paired-residual subspace construction.

Figure 5: Effect of the intervention starting block

\ell_{s}

on MTEB (eng, v2) with PromptEOL. Darker markers indicate the best-performing starting block for each LLM.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Code	Benchmark	# Datasets
English	en	MTEB (eng, v2)	41
Chinese	zh	MTEB (cmn, v1)	29
French	fr	MTEB (fra, v1)	24
Russian	ru	MTEB (rus, v1)	18
German	de	MTEB (deu, v1)	15
Japanese	ja	MTEB (jpn, v1)	13

Appendix

Table 6: Benchmark sources and dataset counts for monolingual evaluation.

Benchmark	# Languages	Direction
BUCC.v2	4	X $\rightarrow$ En
Tatoeba	112	X $\rightarrow$ En
FLORES	195	En $\rightarrow$ X / X $\rightarrow$ En

Appendix

Table 7: Cross-lingual bitext mining benchmarks. En and X denote English and a non-English language, respectively; # Languages indicates the number of evaluated non-English languages.

Measure	Baseline	w/ PRSR
NMI $\downarrow$	0.569	0.148
ARI $\downarrow$	0.396	0.108
Ordering violations (%) $\downarrow$	36.24	10.06
Distance ratio (%) $\downarrow$	85.56	50.98
Translation distance	0.203	0.254
Same-language negative distance	0.237	0.499

Appendix

Table 8: Effect of PRSR on the two motivating findings in Section 3 using LLaMA-2-7B with PromptEOL.

Figure 6: Effect of the number of parallel sentence groups used for subspace estimation on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL.

Figure 7: Effect of the subspace construction corpus and the number of construction languages on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL.

Method	Mono. Avg.	Tatoeba	FLORES	BUCC.v2
ECHO	41.51	1.44	1.63	0.50
w/ PRSR	44.51	20.93	19.56	91.79

Appendix

Table 9: Comparison of ECHO with and without PRSR under mean pooling using LLaMA-2-7B. Mono. Avg. denotes the average score across all

208

monolingual evaluations.

Language	#	Datasets
English (en)	41	ArguAna, ArXivHierarchicalClusteringP2P, ArXivHierarchicalClusteringS2S, AskUbuntuDupQuestions, Banking77Classification, BIOSSES, BiorxivClusteringP2P.v2, ClimateFEVERHardNegatives, CQADupstackGamingRetrieval, CQADupstackUnixRetrieval, FEVERHardNegatives, FiQA2018, HotpotQAHardNegatives, ImdbClassification, MassiveIntentClassification, MassiveScenarioClassification, MedrxivClusteringP2P.v2, MedrxivClusteringS2S.v2, MTOPDomainClassification, MindSmallReranking, SCIDOCS, SICK-R, SprintDuplicateQuestions, StackExchangeClusteringP2P.v2, StackExchangeClustering.v2, STS12, STS13, STS14, STS15, STS17, STS22.v2, STSBenchmark, TRECCOVID, Touche2020Retrieval.v3, ToxicConversationsClassification, TweetSentimentExtractionClassification, TwentyNewsgroupsClustering.v2, TwitterSemEval2015, TwitterURLCorpus, AmazonCounterfactualClassification, SummEvalSummarization.v2
Chinese (zh)	29	CMedQAv1-reranking, CMedQAv2-reranking, MMarcoReranking, T2Reranking, CLSClusteringS2S, CLSClusteringP2P, ThuNewsClusteringS2S, ThuNewsClusteringP2P, LCQMC, PAWSX, QBQTC, Waimai, OnlineShopping, JDReview, MultilingualSentiment, ATEC, BQ, STSB, AFQMC, Cmnli, Ocnli, CmedqaRetrieval, CovidRetrieval, DuRetrieval, EcomRetrieval, MMarcoRetrieval, MedicalRetrieval, T2Retrieval, VideoRetrieval
French (fr)	24	AmazonReviewsClassification, MasakhaNEWSClassification, MassiveIntentClassification, MassiveScenarioClassification, MTOPDomainClassification, MTOPIntentClassification, AlloProfClusteringS2S, AlloProfClusteringP2P, HALClusteringS2S, MasakhaNEWSClusteringP2P, MasakhaNEWSClusteringS2S, MLSUMClusteringP2P, MLSUMClusteringS2S, AlloprofReranking, SyntecReranking, AlloprofRetrieval, BSARDRetrieval, MintakaRetrieval, SyntecRetrieval, XPQARetrieval, SICKFr, STSBenchmarkMultilingualSTS, STS22, SummEvalFr
Russian (ru)	18	GeoreviewClassification, HeadlineClassification, InappropriatenessClassification, KinopoiskClassification, MassiveIntentClassification, MassiveScenarioClassification, RuReviewsClassification, RuSciBenchGRNTIClassification, RuSciBenchOECDClassification, GeoreviewClusteringP2P, RuSciBenchGRNTIClusteringP2P, RuSciBenchOECDClusteringP2P, RuBQReranking, MIRACLReranking, RuBQRetrieval, RUParaPhraserSTS, STS22, RuSTSBenchmarkSTS
German (de)	15	AmazonCounterfactualClassification, AmazonReviewsClassification, MTOPDomainClassification, MTOPIntentClassification, MassiveIntentClassification, MassiveScenarioClassification, BlurbsClusteringP2P, BlurbsClusteringS2S, TenKGnadClusteringP2P, TenKGnadClusteringS2S, GermanQuAD-Retrieval, XMarket, GerDaLIR, GermanSTSBenchmark, STS22
Japanese (ja)	13	LivedoorNewsClustering.v2, MewsC16JaClustering, AmazonReviewsClassification, AmazonCounterfactualClassification, MassiveIntentClassification, MassiveScenarioClassification, JSICK, JaqketRetrieval, JaGovFaqsRetrieval, NLPJournalAbsIntroRetrieval, NLPJournalTitleAbsRetrieval, NLPJournalTitleIntroRetrieval, ESCIReranking

Appendix

Table 10: Dataset composition of the monolingual evaluation across 18 languages.

Retrieval (10)
Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
ArguAna	27.08	41.25	30.51	44.67	26.18	42.61	26.92	42.60	22.22	39.81
ClimateFEVERHardNegatives	5.40	13.10	4.88	14.02	7.95	14.93	5.70	13.79	4.36	11.30
CQADupstackGamingRetrieval	13.92	32.20	18.73	32.06	17.55	31.15	16.42	33.28	17.66	35.29
CQADupstackUnixRetrieval	16.77	27.55	17.53	26.63	16.32	25.92	17.43	27.91	16.77	26.72

Appendix

Table 11: Detailed results of training-free baselines with and without PRSR on MTEB (eng, v2) using LLaMA-2-7B. For category-level and overall averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
ja	LivedoorNewsClustering.v2	25.76	32.58	26.70	32.99	19.73	31.36	28.49	34.74	25.36	34.09
	MewsC16JaClustering	45.46	48.54	46.25	47.28	41.21	43.67	44.62	50.19	43.93	46.73
	AmazonReviewsClassification	38.07	38.06	39.53	38.84	39.73	38.22	37.75	37.61	37.65	36.96
	AmazonCounterfactualClassification	70.63	70.37	70.66	70.94	72.28	72.77	70.36	70.12	69.95	70.40
	MassiveIntentClassification	70.08	71.63	70.26	71.44	70.58	71.81	70.37	71.96	70.53	71.76

Appendix

Table 12: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part I). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
zh	CMedQAv1-reranking	32.64	41.45	37.17	47.32	30.67	38.29	32.63	43.70	33.07	46.81
	CMedQAv2-reranking	36.68	49.75	39.76	50.65	34.88	47.12	37.04	51.06	37.07	51.28
	MMarcoReranking	14.19	15.01	10.07	10.89	13.30	14.45	13.05	15.24	14.16	13.68
	T2Reranking	62.99	64.29	62.91	64.11	63.04	63.80	62.68	64.19	62.29	64.47
	CLSClusteringS2S	28.75	27.98	28.56	27.74	26.28	25.79	30.18	28.45	28.01	27.50

Appendix

Table 13: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part II). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
sw	AfriSentiClassification	36.35	36.38	35.94	36.63	38.57	36.74	36.60	36.82	37.49	37.34
	MasakhaNEWSClassification	66.76	69.12	66.11	68.24	63.66	65.27	67.35	69.60	66.85	68.68
	MassiveIntentClassification	37.03	43.30	38.00	42.53	37.19	40.91	36.79	43.28	38.58	44.12
	MasakhaNEWSClusteringS2S	20.78	22.79	11.73	24.53	17.08	21.91	19.64	22.29	20.00	23.32
	XNLI	52.93	56.67	52.77	56.82	52.85	54.54	53.02	56.67	53.32	56.79

Appendix

Table 14: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part III). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 56 papers

ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs
Jun 4, 2026Ailiang Lin, Zhuoyun Li, Yusong Wang +3Text Embeddings Decoder-Only Language Models
Institute of Science Tokyo · Tencent
Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
Feb 25, 2026Barah Fazili, Koustava GoswamiCross-Lingual Representation Alignment Multilingual Language Models
IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India
F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World
Mar 19, 2026Ziyin Zhang, Zihan Liao, Hang Yu +2Text Embeddings Text Embedding Models
Ant Group · Shanghai Jiao Tong University

Jun 4, 2026J/K move · Enter open · S save

ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao+2

Institute of Science Tokyo · Tencent

Text Embeddings Decoder-Only Language Models Text Embedding Models

Open paper AskPDF arXiv

Recent advances in Large Language Models (LLMs) have opened new avenues for generating training-free text embeddings. However, the causal attention in decoder-only LLMs prevents earlier tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text. Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.

Abstract

Text Embeddings

Explore similar work

ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World