Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

Ailiang Lin; Zhuoyun Li; Yusong Wang; Keyu Mao; Kotaro Funakoshi; Manabu Okumura

cs.CLOct 16, 2026

Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

Authors: Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, Manabu Okumura

Organizations: Institute of Science Tokyo · Tencent

Abstract

While deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs. Our analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. We hypothesize that language-specific variation can be disentangled from semantic content to improve embedding quality. To this end, we propose Paired-Residual Subspace Removal (PRSR), a training-free inference-time intervention that steers LLM representations toward a more language-invariant semantic space. PRSR first constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. It then estimates layer-wise low-rank subspaces from these residuals and suppresses the corresponding components in the last-token hidden states. We conduct more than 200 monolingual evaluations across 18 languages using MTEB and MMTEB, along with cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.

Figures & tables

Figure 1: Illustration of PRSR . (a) Training-free text embeddings remain separated by languages despite shared semantics. (b) PRSR estimates language-sensitive subspaces from paired residuals of parallel sentences and removes the corresponding components within the frozen LLM to improve semantic alignment.

Figure 2: Language-conditioned geometry of training-free text embeddings on FLORES+. (a) t-SNE visualization showing language-specific clustering. (b) Cosine-similarity margins between anchor–translation and anchor–negative pairs; in 36.2% of comparisons, negatives are more similar to anchors than translations.

Figure 3: Overview of PRSR . (a) Paired-residual subspace construction. At each selected layer, we first construct paired residuals by subtracting the group mean from the last-token representations of parallel sentences. We then stack the residuals across groups and languages, and apply SVD to estimate a low-rank language-sensitive subspace. (b) Inference-time intervention. We remove the projections of last-token hidden states onto these subspaces at the corresponding layers of the frozen LLM, steering representations toward a more language-invariant semantic space.

Method	en	zh	fr	ru	de	ja †	it †	es	pt †	ar	id †	sw	vi	ko †	bn †	th	te †	ms †	AVG
PromptEOL	46.8	38.7	43.4	45.0	39.4	46.6	55.1	47.9	47.9	41.2	49.6	42.8	44.7	53.0	32.0	40.8	18.4	25.2	43.8
w/ PRSR	54.6	42.1	53.8	49.1	45.0	51.1	66.0	57.7	62.0	47.1	59.3	45.7	54.5	60.2	36.4	49.7	28.8	51.5	51.1 ↑7.3
Pretended CoT	49.0	40.0	44.6	46.7	40.3	47.2	58.6	49.5	51.9	41.9	53.0	40.9	46.0	49.3	32.2	42.1	19.6	34.0	45.3
w/ PRSR	54.6	42.5	53.3	50.4	45.8	51.2	67.5	59.1	63.4	47.9	59.2	45.7	54.7	56.6	39.2	50.0	28.1	52.7	51.5 ↑6.2
ECHO	48.4	38.4	44.3	44.2	38.4	47.5	56.5	48.3	47.1	42.4	49.0	41.9	40.5	47.5	25.8	39.9	20.7	21.5	43.7
w/ PRSR	54.1	40.7	52.3	47.4	42.6	50.7	66.9	57.6	60.6	48.5	55.3	43.9	51.1	55.2	36.0	46.9	23.7	41.2	49.7 ↑6.0

Table 1: Monolingual performance across 18 languages on MTEB and MMTEB. Each language column reports the average over its monolingual tasks, and AVG denotes the overall mean across all 208 evaluations. Languages marked with

\dagger

were not used for subspace estimation. The best results are highlighted in bold . PRSR significantly outperforms each corresponding baseline (paired

t

-test,

p<0.01

Method	FLORES			Tatoeba	BUCC.v2
Method	En $\rightarrow$ X	X $\rightarrow$ En	Mean	Tatoeba	BUCC.v2
PromptEOL	27.42	23.43	25.43	12.95	44.25
w/ PRSR	48.61	50.16	49.39	47.85	97.13
Pretended CoT	30.74	24.97	27.85	16.10	49.31
w/ PRSR	48.88	50.07	49.47	48.32	97.43
ECHO	34.42	31.13	32.78	24.60	66.20

Table 2: Cross-lingual bitext mining performance of training-free embedding methods on FLORES, Tatoeba, and BUCC.v2. For FLORES, we report English-to-X and X-to-English results together with their mean. The best results are highlighted in bold .

Method	MTEB	Tatoeba	FLORES	BUCC.v2
Baseline	46.82	12.95	25.43	44.25
Random	46.80	12.20	24.92	43.13
Language-Mean	50.00	42.95	40.90	90.91
Shuffled-Pair	52.34	43.90	44.33	95.73
PRSR	54.62	47.85	49.39	97.13

Table 3: Performance comparison of different subspace construction strategies using LLaMA-2-7B with PromptEOL. MTEB denotes MTEB (eng, v2). The best results are highlighted in bold .

Intervention	MTEB	Tatoeba	FLORES	BUCC.v2
$h_{\ell}$ (Baseline)	46.82	12.95	25.43	44.25
$h_{\ell}+P_{\ell}$	24.16	0.06	0.04	0.06
$h_{\ell}+2P_{\ell}$	23.77	0.05	0.03	0.04
$h_{\ell}+3P_{\ell}$	23.64	0.05	0.03	0.03
$h_{\ell}-P_{\ell}$ ( PRSR )	54.62	47.85	49.39	97.13

Table 4: Comparison of subspace removal and amplification using LLaMA-2-7B with PromptEOL. MTEB denotes MTEB (eng, v2), and

P_{\ell}=(h_{\ell}B_{\ell})B_{\ell}^{\top}

is the projection onto the estimated language-sensitive subspace. The best results are highlighted in bold .

Method	Mono. Avg.	Tatoeba	FLORES	BUCC.v2
LLaMA-2-7B
PromptEOL	43.80	12.95	25.43	44.25
w/ PRSR	51.09	47.85	49.39	97.13
Mistral-7B
PromptEOL	49.40	29.21	37.99	49.97
w/ PRSR	55.18	56.30	55.96	97.82

Table 5: Performance of PRSR across different LLM backbones with PromptEOL. Mono. Avg. denotes the average score over 208 monolingual evaluations spanning 18 languages. The best results for each backbone are highlighted in bold .

Figure 4: Effect of the subspace rank

q

on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL. Rank 0 denotes the baseline, and

K

is the number of languages used for paired-residual subspace construction.

Figure 5: Effect of the intervention starting block

\ell_{s}

on MTEB (eng, v2) with PromptEOL. Darker markers indicate the best-performing starting block for each LLM.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Code	Benchmark	# Datasets
English	en	MTEB (eng, v2)	41
Chinese	zh	MTEB (cmn, v1)	29
French	fr	MTEB (fra, v1)	24
Russian	ru	MTEB (rus, v1)	18
German	de	MTEB (deu, v1)	15
Japanese	ja	MTEB (jpn, v1)	13

Appendix

Table 6: Benchmark sources and dataset counts for monolingual evaluation.

Benchmark	# Languages	Direction
BUCC.v2	4	X $\rightarrow$ En
Tatoeba	112	X $\rightarrow$ En
FLORES	195	En $\rightarrow$ X / X $\rightarrow$ En

Appendix

Table 7: Cross-lingual bitext mining benchmarks. En and X denote English and a non-English language, respectively; # Languages indicates the number of evaluated non-English languages.

Measure	Baseline	w/ PRSR
NMI $\downarrow$	0.569	0.148
ARI $\downarrow$	0.396	0.108
Ordering violations (%) $\downarrow$	36.24	10.06
Distance ratio (%) $\downarrow$	85.56	50.98
Translation distance	0.203	0.254
Same-language negative distance	0.237	0.499

Appendix

Table 8: Effect of PRSR on the two motivating findings in Section 3 using LLaMA-2-7B with PromptEOL.

Figure 6: Effect of the number of parallel sentence groups used for subspace estimation on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL.

Figure 7: Effect of the subspace construction corpus and the number of construction languages on MTEB (eng, v2) using LLaMA-2-7B with PromptEOL.

Method	Mono. Avg.	Tatoeba	FLORES	BUCC.v2
ECHO	41.51	1.44	1.63	0.50
w/ PRSR	44.51	20.93	19.56	91.79

Appendix

Table 9: Comparison of ECHO with and without PRSR under mean pooling using LLaMA-2-7B. Mono. Avg. denotes the average score across all

208

monolingual evaluations.

Language	#	Datasets
English (en)	41	ArguAna, ArXivHierarchicalClusteringP2P, ArXivHierarchicalClusteringS2S, AskUbuntuDupQuestions, Banking77Classification, BIOSSES, BiorxivClusteringP2P.v2, ClimateFEVERHardNegatives, CQADupstackGamingRetrieval, CQADupstackUnixRetrieval, FEVERHardNegatives, FiQA2018, HotpotQAHardNegatives, ImdbClassification, MassiveIntentClassification, MassiveScenarioClassification, MedrxivClusteringP2P.v2, MedrxivClusteringS2S.v2, MTOPDomainClassification, MindSmallReranking, SCIDOCS, SICK-R, SprintDuplicateQuestions, StackExchangeClusteringP2P.v2, StackExchangeClustering.v2, STS12, STS13, STS14, STS15, STS17, STS22.v2, STSBenchmark, TRECCOVID, Touche2020Retrieval.v3, ToxicConversationsClassification, TweetSentimentExtractionClassification, TwentyNewsgroupsClustering.v2, TwitterSemEval2015, TwitterURLCorpus, AmazonCounterfactualClassification, SummEvalSummarization.v2
Chinese (zh)	29	CMedQAv1-reranking, CMedQAv2-reranking, MMarcoReranking, T2Reranking, CLSClusteringS2S, CLSClusteringP2P, ThuNewsClusteringS2S, ThuNewsClusteringP2P, LCQMC, PAWSX, QBQTC, Waimai, OnlineShopping, JDReview, MultilingualSentiment, ATEC, BQ, STSB, AFQMC, Cmnli, Ocnli, CmedqaRetrieval, CovidRetrieval, DuRetrieval, EcomRetrieval, MMarcoRetrieval, MedicalRetrieval, T2Retrieval, VideoRetrieval
French (fr)	24	AmazonReviewsClassification, MasakhaNEWSClassification, MassiveIntentClassification, MassiveScenarioClassification, MTOPDomainClassification, MTOPIntentClassification, AlloProfClusteringS2S, AlloProfClusteringP2P, HALClusteringS2S, MasakhaNEWSClusteringP2P, MasakhaNEWSClusteringS2S, MLSUMClusteringP2P, MLSUMClusteringS2S, AlloprofReranking, SyntecReranking, AlloprofRetrieval, BSARDRetrieval, MintakaRetrieval, SyntecRetrieval, XPQARetrieval, SICKFr, STSBenchmarkMultilingualSTS, STS22, SummEvalFr
Russian (ru)	18	GeoreviewClassification, HeadlineClassification, InappropriatenessClassification, KinopoiskClassification, MassiveIntentClassification, MassiveScenarioClassification, RuReviewsClassification, RuSciBenchGRNTIClassification, RuSciBenchOECDClassification, GeoreviewClusteringP2P, RuSciBenchGRNTIClusteringP2P, RuSciBenchOECDClusteringP2P, RuBQReranking, MIRACLReranking, RuBQRetrieval, RUParaPhraserSTS, STS22, RuSTSBenchmarkSTS
German (de)	15	AmazonCounterfactualClassification, AmazonReviewsClassification, MTOPDomainClassification, MTOPIntentClassification, MassiveIntentClassification, MassiveScenarioClassification, BlurbsClusteringP2P, BlurbsClusteringS2S, TenKGnadClusteringP2P, TenKGnadClusteringS2S, GermanQuAD-Retrieval, XMarket, GerDaLIR, GermanSTSBenchmark, STS22
Japanese (ja)	13	LivedoorNewsClustering.v2, MewsC16JaClustering, AmazonReviewsClassification, AmazonCounterfactualClassification, MassiveIntentClassification, MassiveScenarioClassification, JSICK, JaqketRetrieval, JaGovFaqsRetrieval, NLPJournalAbsIntroRetrieval, NLPJournalTitleAbsRetrieval, NLPJournalTitleIntroRetrieval, ESCIReranking

Appendix

Table 10: Dataset composition of the monolingual evaluation across 18 languages.

Retrieval (10)
Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
ArguAna	27.08	41.25	30.51	44.67	26.18	42.61	26.92	42.60	22.22	39.81
ClimateFEVERHardNegatives	5.40	13.10	4.88	14.02	7.95	14.93	5.70	13.79	4.36	11.30
CQADupstackGamingRetrieval	13.92	32.20	18.73	32.06	17.55	31.15	16.42	33.28	17.66	35.29
CQADupstackUnixRetrieval	16.77	27.55	17.53	26.63	16.32	25.92	17.43	27.91	16.77	26.72

Appendix

Table 11: Detailed results of training-free baselines with and without PRSR on MTEB (eng, v2) using LLaMA-2-7B. For category-level and overall averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
ja	LivedoorNewsClustering.v2	25.76	32.58	26.70	32.99	19.73	31.36	28.49	34.74	25.36	34.09
	MewsC16JaClustering	45.46	48.54	46.25	47.28	41.21	43.67	44.62	50.19	43.93	46.73
	AmazonReviewsClassification	38.07	38.06	39.53	38.84	39.73	38.22	37.75	37.61	37.65	36.96
	AmazonCounterfactualClassification	70.63	70.37	70.66	70.94	72.28	72.77	70.36	70.12	69.95	70.40
	MassiveIntentClassification	70.08	71.63	70.26	71.44	70.58	71.81	70.37	71.96	70.53	71.76

Appendix

Table 12: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part I). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
zh	CMedQAv1-reranking	32.64	41.45	37.17	47.32	30.67	38.29	32.63	43.70	33.07	46.81
	CMedQAv2-reranking	36.68	49.75	39.76	50.65	34.88	47.12	37.04	51.06	37.07	51.28
	MMarcoReranking	14.19	15.01	10.07	10.89	13.30	14.45	13.05	15.24	14.16	13.68
	T2Reranking	62.99	64.29	62.91	64.11	63.04	63.80	62.68	64.19	62.29	64.47
	CLSClusteringS2S	28.75	27.98	28.56	27.74	26.28	25.79	30.18	28.45	28.01	27.50

Appendix

Table 13: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part II). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Lang.	Dataset	PromptEOL		Pretended CoT		ECHO		CP		TP
		Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR	Base	w/ PRSR
sw	AfriSentiClassification	36.35	36.38	35.94	36.63	38.57	36.74	36.60	36.82	37.49	37.34
	MasakhaNEWSClassification	66.76	69.12	66.11	68.24	63.66	65.27	67.35	69.60	66.85	68.68
	MassiveIntentClassification	37.03	43.30	38.00	42.53	37.19	40.91	36.79	43.28	38.58	44.12
	MasakhaNEWSClusteringS2S	20.78	22.79	11.73	24.53	17.08	21.91	19.64	22.29	20.00	23.32
	XNLI	52.93	56.67	52.77	56.82	52.85	54.54	53.02	56.67	53.32	56.79

Appendix

Table 14: Detailed monolingual results of training-free baselines with and without PRSR across

17

non-English languages using LLaMA-2-7B (Part III). For language-level averages, the better result between each baseline and its PRSR -enhanced version is highlighted in bold .

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 56 papers

Jun 4, 2026

ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

Recent advances in Large Language Models (LLMs) have opened new avenues for generating training-free text embeddings. However, the causal attention in decoder-only LLMs prevents earlier tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text. Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.

Ailiang Lin, Zhuoyun Li, Yusong Wang +3

Institute of Science Tokyo · Tencent

Text Embeddings Decoder-Only Language Models Text Embedding Models

Ask PDF

Feb 25, 2026

Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.

Barah Fazili, Koustava Goswami

IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India

Cross-Lingual Representation Alignment Multilingual Language Models Sentence Embeddings

Ask PDF

Mar 19, 2026

F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World

We present F2LLM-v2, a new family of general-purpose, multilingual embedding models in 8 distinct sizes ranging from 80M to 14B. Trained on a newly curated composite of 60 million publicly available high-quality data samples, F2LLM-v2 supports more than 200 languages, with a particular emphasis on previously underserved mid- and low-resource languages. By integrating a two-stage LLM-based embedding training pipeline with matryoshka learning, model pruning, and knowledge distillation techniques, we present models that are far more efficient than previous LLM-based embedding models while retaining competitive performances. Extensive evaluations confirm that F2LLM-v2-14B ranks first on 11 MTEB benchmarks, while the smaller models in the family also set a new state of the art for resource-constrained applications. To facilitate open-source embedding model research, we release all models, data, code, and intermediate checkpoints.

Ziyin Zhang, Zihan Liao, Hang Yu +2

Ant Group · Shanghai Jiao Tong University

Text Embeddings Text Embedding Models

Ask PDF

Abstract

Text Embeddings

Explore similar work

ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World