Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs

Zifeng Cheng; Lingyun Qian; Zhiwei Jiang; Cong Wang; Yafeng Yin; Fei Shen; Ao Zhou; Qing Gu

cs.CLSep 26, 2026

Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs

Authors: Zifeng Cheng, Lingyun Qian, Zhiwei Jiang, Cong Wang, Yafeng Yin, Fei Shen, Ao Zhou, Qing Gu

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore

Abstract

Extracting conditional text embeddings from large language models (LLMs) is a promising paradigm, as it requires neither additional data nor fine-tuning. Existing methods incorporate conditions into prompts to guide LLMs to focus on specific aspects and elicit conditional text embeddings. However, relying solely on prompts often fails to produce high-quality conditional text embeddings, as they remain entangled with general text embeddings, ultimately degrading their quality. To this end, we propose an inference-time, plug-and-play Self-Contrastive Steering (SCS) method that constructs unconditional general text embeddings and uses them to refine conditional text embeddings, making them more focused on the target condition. Specifically, we modify the attention mask and positional encodings to mask the condition, thereby obtaining unconditional text embeddings and intervening in the multi-head self-attention computation process. Notably, our method is highly efficient, requiring only a single additional multi-head self-attention computation at inference time. Extensive experiments on clustering, Semantic Textual Similarity, and triplet alignment datasets demonstrate that our method can seamlessly improve the performance of existing prompt-based methods across different LLMs in a training-free and plug-and-play manner. Our code will be released at https://github.com/zifengcheng/SCS

Figures & tables

Figure 1: An example of the conditional text embedding task. Notably, compared to labels, conditions often correspond to coarser-grained information.

Figure 2: Illustration of the self-contrastive steering method. Our method additionally generates an unconditional text embedding through condition-masked embedding generation and introduces two activation steering strategies.

Model	Clustering (V-measure $\uparrow$ )			STS (Spearman & Pearson $\uparrow$ )		Triplet Alignment (Acc. $\uparrow$ )
Model	TweetEmo	Amazon-R	Amazon-C	I.STSB	C-STS	Toxic	AGNews	Mean $\uparrow$
Supervised Contrastive Training
sup-SimCSE ${}_{\text{RoBERTa-large}}$	29.4	22.4	19.5	-	4.1/3.4	-	-	-
E5 ${}_{\text{Mistral-7B-Inst}}$	41.3	37.6	37.4	-	34.8/34.6	-	-	-
GTE ${}_{\text{Qwen2-7B-Inst}}$	36.8	36.8	38.3	-	33.5/33.9	-	-	-
Without Contrastive Training

Table 1: Results on three downstream tasks across seven datasets using LLaMA3-8B-Inst. Since PonTE does not open-source code,

{\dagger}

denotes our reproduced results. The best results are in bold , and the second-best are underlined .

Figure 3: Effects of intervention layer, scaling factor, and output layer on SCS-DE. (a) Effects of the intervention layer. (b) Effects of the scaling factor. (c) The effects of the output layer.

Method	TweetEmo	Amazon-R
PonTE	70ms	79ms
PonTE + SCS-DE ( Ours )	69ms	80ms

Table 2: The runtime per sample on two datasets using LLaMA3-8B-Inst.

Position	I.STSB	TweetEmo	Amazon-R
Head	39.9/37.7 $\uparrow$ 6.0/8.2	45.5 $\uparrow$ 5.6	37.0 $\uparrow$ 2.2
FFN	40.8/39.1 $\uparrow$ 6.9/9.6	44.2 $\uparrow$ 4.3	35.7 $\uparrow$ 0.9
Hidden	39.1/37.2 $\uparrow$ 5.2/7.7	46.2 $\uparrow$ 6.3	35.1 $\uparrow$ 0.3

Table 3: Effects of intervention position on three datasets. FFN denotes the output of FFN block, and Hidden denotes the output of the Transformer layer.

Method	Backbone	I.STSB	TweetEmo	Amazon-R	Mean
PonTE	LLaMA3-8B	10.2/9.5	26.5	23.4	17.4
PonTE + SCS-DE ( Ours )	LLaMA3-8B	25.9/24.2 $\uparrow$ 15.7/14.7	34.8 $\uparrow$ 8.3	26.9 $\uparrow$ 3.5	28.0 $\uparrow$ 10.6
PonTE	LLaMA3-8B-Inst	33.9/29.5	39.9	34.8	34.5
PonTE + SCS-DE ( Ours )	LLaMA3-8B-Inst	39.9/37.7 $\uparrow$ 6.0/8.2	45.5 $\uparrow$ 5.6	37.0 $\uparrow$ 2.2	40.0 $\uparrow$ 5.5
PonTE	LLaMA3.1-8B	11.3/11.0	26.2	23.1	17.9
PonTE + SCS-DE ( Ours )	LLaMA3.1-8B	25.5/23.1 $\uparrow$ 14.2/12.1	38.7 $\uparrow$ 12.5	25.9 $\uparrow$ 2.8	28.3 $\uparrow$ 10.4

Table 4: Results on three datasets using different backbones.

Prompt	I.STSB	TweetEmo	Amazon-R
Express this text “ [TEXT] ” in one word in terms of condition: “	39.9/37.7 $\uparrow$ 6.0/8.2	45.5 $\uparrow$ 5.6	37.0 $\uparrow$ 6.2
Express this text “ [TEXT] ” in one word with respect to condition: “	39.8/38.0 $\uparrow$ 6.8/8.3	44.6 $\uparrow$ 1.9	35.8 $\uparrow$ 4.5
Express this text “ [TEXT] ” in terms of condition in one word: “	37.1/33.1 $\uparrow$ 7.8/8.9	44.2 $\uparrow$ 2.2	37.2 $\uparrow$ 0.4
This text “ [TEXT] ” means in one word in terms of condition: “	36.5/31.5 $\uparrow$ 6.1/7.3	42.5 $\uparrow$ 0.8	37.6 $\uparrow$ 0.9
This text “ [TEXT] ” means in one word with respect to condition: “	31.6/28.3 $\uparrow$ 5.0/6.8	42.8 $\uparrow$ 2.6	37.2 $\uparrow$ 0.4
This text “ [TEXT] ” means in terms of condition in one word: “	34.1/29.2 $\uparrow$ 4.2/5.8	43.2 $\uparrow$ 1.6	37.7 $\uparrow$ 0.1

Table 5: Generalization across different prompts using LLaMA3-8B-Inst as the backbone on three datasets.

Dataset	Condition	V-measure
TweetEmo	the emotion	44.8 $\uparrow$ 3.9
	the feeling	42.4 $\uparrow$ 1.2
	the sentiment	43.5 $\uparrow$ 4.0
Amazon-R	the star	32.4 $\uparrow$ 4.7
	the rating	35.7 $\uparrow$ 5.0
	the star rating	37.0 $\uparrow$ 2.2

Table 6: Generalization across different conditions using LLaMA3-8B-Inst on two datasets.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	German	Spanish
PonTE	33.0	29.0
PonTE + SCS-DE ( Ours )	35.8 $\uparrow$ 2.8	31.6 $\uparrow$ 2.6

Appendix

Table 7: Results on the two languages of the Amazon-R dataset.

Dataset	Before Steering	After Steering
I.STSB	0.95	-0.17
TweetEmo	0.94	-0.52
Amazon-R	0.96	-0.27

Appendix

Table 8: Similarity before and after steering on three datasets.

Layer Index	TweetEmo	Amazon-R
Intervening on a single layer:
1	43.6	34.5
2	43.9	33.8
3	45.5	35.4
4	43.3	35.7
5	43.9	36.5

Appendix

Table 9: Results of applying multi-layer interventions on the two datasets.

Method	Backbone	I.STSB	TweetEmo
PonTE	Qwen2.5-14B	13.5/10.9	17.4
PonTE + SCS-DE ( Ours )	Qwen2.5-14B	20.2/16.2 $\uparrow$ 6.6/5.3	27.0 $\uparrow$ 9.6
PonTE	Qwen2.5-14B-Instruct	24.7/21.3	22.5
PonTE + SCS-DE ( Ours )	Qwen2.5-14B-Instruct	28.9/27.5 $\uparrow$ 4.2/5.2	43.1 $\uparrow$ 20.6

Appendix

Table 10: Results on three datasets using different backbones.

Text	Condition	Method	Top-predicted Tokens and Probability
A man and two children lounge in the living area by a sliding glass door.	the number of people	PonTE	Three (0.482), Four (0.054), three (0.046), I (0.038),
			TH (0.032), Family (0.03), T (0.029), Two (0.029)
		PonTE+SCS-DE	Three (0.550), three (0.056), Four (0.049), Two (0.037)
			TH (0.028), 3 (0.028), T (0.026), Tri (0.024)
A row of green donuts, a row of glazed donuts, and a row of chocolate donuts.	the rows	PonTE	Rows (0.131), Tr (0.107), Tri (0.105), tr (0.067),
			Don (0.056), T (0.052), rows (0.05), TR (0.043)

Appendix

Table 11: Top-8 tokens predicted by different methods using LLaMA3-8B-Inst.

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 56 papers

Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
May 2, 2026Ailiang Lin, Zhuoyun Li, Keyu Mao +2Text Embeddings In-Context Learning
Institute of Science Tokyo · Tencent
LLM2Vec-Gen: Generative Embeddings from Large Language Models
Mar 11, 2026Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt +3Text Embedding Models Self-Supervised Pre-Training
McGill University · Mila–Quebec AI Institute · ServiceNow Research +3
On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
Jun 10, 2026Iuri Macocco, Pau Rodríguez, Arno Blaas +3LLM Fine-Tuning Activation Steering
Universitat Pompeu Fabra · Apple · Universitat Pompeu Fabra / ICREA

May 2, 2026J/K move · Enter open · S save

Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

Ailiang Lin, Zhuoyun Li, Keyu Mao, Kotaro Funakoshi+1

Institute of Science Tokyo · Tencent

Text Embeddings In-Context Learning Text Embedding Models

Open paper AskPDF arXiv

Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.

Abstract

Text Embeddings

Explore similar work

Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

LLM2Vec-Gen: Generative Embeddings from Large Language Models

On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study