Extracting conditional text embeddings from large language models (LLMs) is a promising paradigm, as it requires neither additional data nor fine-tuning. Existing methods incorporate conditions into prompts to guide LLMs to focus on specific aspects and elicit conditional text embeddings. However, relying solely on prompts often fails to produce high-quality conditional text embeddings, as they remain entangled with general text embeddings, ultimately degrading their quality. To this end, we propose an inference-time, plug-and-play Self-Contrastive Steering (SCS) method that constructs unconditional general text embeddings and uses them to refine conditional text embeddings, making them more focused on the target condition. Specifically, we modify the attention mask and positional encodings to mask the condition, thereby obtaining unconditional text embeddings and intervening in the multi-head self-attention computation process. Notably, our method is highly efficient, requiring only a single additional multi-head self-attention computation at inference time. Extensive experiments on clustering, Semantic Textual Similarity, and triplet alignment datasets demonstrate that our method can seamlessly improve the performance of existing prompt-based methods across different LLMs in a training-free and plug-and-play manner. Our code will be released at https://github.com/zifengcheng/SCS
Figures & tables
Figure 1: An example of the conditional text embedding task. Notably, compared to labels, conditions often correspond to coarser-grained information.
Figure 2: Illustration of the self-contrastive steering method. Our method additionally generates an unconditional text embedding through condition-masked embedding generation and introduces two activation steering strategies.
Model
Clustering (V-measure ↑ )
STS (Spearman & Pearson ↑ )
Triplet Alignment (Acc. ↑ )
TweetEmo
Amazon-R
Amazon-C
I.STSB
C-STS
Toxic
AGNews
Mean ↑
Supervised Contrastive Training
sup-SimCSE RoBERTa-large
29.4
22.4
19.5
-
4.1/3.4
-
-
-
E5 Mistral-7B-Inst
41.3
37.6
37.4
-
34.8/34.6
-
-
-
GTE Qwen2-7B-Inst
36.8
36.8
38.3
-
33.5/33.9
-
-
-
Without Contrastive Training
Table 1: Results on three downstream tasks across seven datasets using LLaMA3-8B-Inst. Since PonTE does not open-source code, † denotes our reproduced results. The best results are in bold , and the second-best are underlined .
Figure 3: Effects of intervention layer, scaling factor, and output layer on SCS-DE. (a) Effects of the intervention layer. (b) Effects of the scaling factor. (c) The effects of the output layer.
Method
TweetEmo
Amazon-R
PonTE
70ms
79ms
PonTE + SCS-DE ( Ours )
69ms
80ms
Table 2: The runtime per sample on two datasets using LLaMA3-8B-Inst.
Position
I.STSB
TweetEmo
Amazon-R
Head
39.9/37.7 ↑ 6.0/8.2
45.5 ↑ 5.6
37.0 ↑ 2.2
FFN
40.8/39.1 ↑ 6.9/9.6
44.2 ↑ 4.3
35.7 ↑ 0.9
Hidden
39.1/37.2 ↑ 5.2/7.7
46.2 ↑ 6.3
35.1 ↑ 0.3
Table 3: Effects of intervention position on three datasets. FFN denotes the output of FFN block, and Hidden denotes the output of the Transformer layer.
Method
Backbone
I.STSB
TweetEmo
Amazon-R
Mean
PonTE
LLaMA3-8B
10.2/9.5
26.5
23.4
17.4
PonTE + SCS-DE ( Ours )
LLaMA3-8B
25.9/24.2 ↑ 15.7/14.7
34.8 ↑ 8.3
26.9 ↑ 3.5
28.0 ↑ 10.6
PonTE
LLaMA3-8B-Inst
33.9/29.5
39.9
34.8
34.5
PonTE + SCS-DE ( Ours )
LLaMA3-8B-Inst
39.9/37.7 ↑ 6.0/8.2
45.5 ↑ 5.6
37.0 ↑ 2.2
40.0 ↑ 5.5
PonTE
LLaMA3.1-8B
11.3/11.0
26.2
23.1
17.9
PonTE + SCS-DE ( Ours )
LLaMA3.1-8B
25.5/23.1 ↑ 14.2/12.1
38.7 ↑ 12.5
25.9 ↑ 2.8
28.3 ↑ 10.4
Table 4: Results on three datasets using different backbones.
Prompt
I.STSB
TweetEmo
Amazon-R
Express this text “ [TEXT] ” in one word in terms of condition: “
39.9/37.7 ↑ 6.0/8.2
45.5 ↑ 5.6
37.0 ↑ 6.2
Express this text “ [TEXT] ” in one word with respect to condition: “
39.8/38.0 ↑ 6.8/8.3
44.6 ↑ 1.9
35.8 ↑ 4.5
Express this text “ [TEXT] ” in terms of condition in one word: “
37.1/33.1 ↑ 7.8/8.9
44.2 ↑ 2.2
37.2 ↑ 0.4
This text “ [TEXT] ” means in one word in terms of condition: “
36.5/31.5 ↑ 6.1/7.3
42.5 ↑ 0.8
37.6 ↑ 0.9
This text “ [TEXT] ” means in one word with respect to condition: “
31.6/28.3 ↑ 5.0/6.8
42.8 ↑ 2.6
37.2 ↑ 0.4
This text “ [TEXT] ” means in terms of condition in one word: “
34.1/29.2 ↑ 4.2/5.8
43.2 ↑ 1.6
37.7 ↑ 0.1
Table 5: Generalization across different prompts using LLaMA3-8B-Inst as the backbone on three datasets.
Dataset
Condition
V-measure
TweetEmo
the emotion
44.8 ↑ 3.9
the feeling
42.4 ↑ 1.2
the sentiment
43.5 ↑ 4.0
Amazon-R
the star
32.4 ↑ 4.7
the rating
35.7 ↑ 5.0
the star rating
37.0 ↑ 2.2
Table 6: Generalization across different conditions using LLaMA3-8B-Inst on two datasets.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
German
Spanish
PonTE
33.0
29.0
PonTE + SCS-DE ( Ours )
35.8 ↑ 2.8
31.6 ↑ 2.6
Appendix
Table 7: Results on the two languages of the Amazon-R dataset.
Dataset
Before Steering
After Steering
I.STSB
0.95
-0.17
TweetEmo
0.94
-0.52
Amazon-R
0.96
-0.27
Appendix
Table 8: Similarity before and after steering on three datasets.
Layer Index
TweetEmo
Amazon-R
Intervening on a single layer:
1
43.6
34.5
2
43.9
33.8
3
45.5
35.4
4
43.3
35.7
5
43.9
36.5
Appendix
Table 9: Results of applying multi-layer interventions on the two datasets.
Method
Backbone
I.STSB
TweetEmo
PonTE
Qwen2.5-14B
13.5/10.9
17.4
PonTE + SCS-DE ( Ours )
Qwen2.5-14B
20.2/16.2 ↑ 6.6/5.3
27.0 ↑ 9.6
PonTE
Qwen2.5-14B-Instruct
24.7/21.3
22.5
PonTE + SCS-DE ( Ours )
Qwen2.5-14B-Instruct
28.9/27.5 ↑ 4.2/5.2
43.1 ↑ 20.6
Appendix
Table 10: Results on three datasets using different backbones.
Text
Condition
Method
Top-predicted Tokens and Probability
A man and two children lounge in the living area by a sliding glass door.
the number of people
PonTE
Three (0.482), Four (0.054), three (0.046), I (0.038),
TH (0.032), Family (0.03), T (0.029), Two (0.029)
PonTE+SCS-DE
Three (0.550), three (0.056), Four (0.049), Two (0.037)
TH (0.028), 3 (0.028), T (0.026), Tri (0.024)
A row of green donuts, a row of glazed donuts, and a row of chocolate donuts.
the rows
PonTE
Rows (0.131), Tr (0.107), Tri (0.105), tr (0.067),
Don (0.056), T (0.052), rows (0.05), TR (0.043)
Appendix
Table 11: Top-8 tokens predicted by different methods using LLaMA3-8B-Inst.
Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.
Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.
Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt +3
McGill University · Mila–Quebec AI Institute · ServiceNow Research +3
Controlling the output of Large Language Models (LLMs) is a central challenge for their reliable deployment, yet a clear understanding of the involved trade-offs remains elusive. Current approaches to conditioning are often evaluated with a narrow focus on their effectiveness at injecting or removing a target concept, neglecting generation quality. We systematically investigate a range of conditioning methods in both injection and removal scenarios. We find that efficient steering methods frequently achieve conditioning at a steep cost to fluency. Furthermore, we identify a critical yet previously overlooked interaction with the training paradigm: activation steering methods are far less effective on instruction-tuned models than on their base counterparts. Simple prompting and full-fledged supervised fine-tuning, on the other hand, are viable options for concept injection, but are not as good at concept removal. Finally, cheaply computed textual metrics highly correlate to costly LLM-as-judge scores, and provide insights on the behavior of conditioning methods.