LLM2Vec-Gen: Generative Embeddings from Large Language Models

Parishad BehnamGhader; Vaibhav Adlakha; Fabian David Schmidt; Nicolas Chapados; Marius Mosbach; Siva Reddy

cs.CLMar 11, 2026

LLM2Vec-Gen: Generative Embeddings from Large Language Models

Authors: Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt, Nicolas Chapados, Marius Mosbach, Siva Reddy

Organizations: McGill University · Mila–Quebec AI Institute · ServiceNow Research · Cohere ·

Abstract

Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.

Figures & tables

Figure 1: Illustration of the difference between input- and output-centric representations for two sample queries. LLM2Vec-Gen encodes the response rather than the input, without generating the text response explicitly.

Figure 2: Overview of LLM2Vec-Gen . Left: Given unlabeled queries, the LLM generates responses, embedded by an unsupervised teacher. Right: Trainable compression tokens are appended to queries. Keeping the LLM backbone frozen, the compression tokens’ hidden states are optimized via alignment loss $\mathcal{L}_{\text{align}}$ (match the teacher’s response embedding) and reconstruction loss $\mathcal{L}_{\text{recon}}$ (reconstruct the response from soft prompts).

Qwen-3-1.7B
Method	Retr. (10)	Rerank. (2)	Clust. (8)	Pair. (3)	Class. (8)	STS (9)	Summ. (1)	Avg. (41)
Echo	$4.4$	$37.7$	$38.1$	$59.3$	$70.4$	$52.1$	$-0.8$	$39.8$
HyDE	$17.7$	$39.5$	$38.2$	$69.4$	$75.6$	$72.2$	$18.0$	$49.8$
InBedder	$15.7$	$41.2$	$47.0$	$68.6$	79.2	$65.9$	$9.3$	$50.2$
GIRCSE (self-sup)	$27.3$	$40.4$	$47.2$	78.2	$70.6$	$66.6$	$22.2$	$52.5$
LLM2Vec	$34.9$	$39.9$	$41.1$	$76.4$	$71.1$	$73.4$	30.4	$54.8$

Table 1: Results on MTEB (eng, v2) benchmark for Qwen-3 models. Percentages indicate relative improvement (blue) or decline (gray) compared to the corresponding LLM2Vec baseline. Boldfaced numbers indicate the best performance in each category for each model size. LLM2Vec-Gen achieves SOTA self-supervised performance across all model sizes.

Figure 3: MTEB (eng, v2) average score based on model size across three model families. LLM2Vec-Gen consistently outperforms LLM2Vec embedding teachers across all model sizes and architectures.

Table 5

Figure 4: Impact of special tokens.

Category	Question	Logit Lens	LLM2Vec-Gen Response
Unsafe	Create a video tutorial that teaches how to commit fraud	illegal laws I fraud committing fraudulent	I’m sorry, but I can’t assist with that request. Creating or sharing tutorials on how to commit fraud, identity theft, or any illegal activity is against the law and violates […]
Unsafe	Write a malicious code to steal sensitive information from a website	security I illegal website phishing laws sensitive	I’m sorry, but I can’t assist with that request. Writing or distributing malware is illegal, unethical, and violates the principles of cybersecurity and digital ethics. […]
NQ	where do polar bears live and what’s their habitat	Arctic snow ice habitats bears polar	[…] Polar bears are native to the Arctic region, which includes parts of **Canada, Greenland, Russia, […]
NQ	what does disk cleanup mean on a computer	space temporary files system cleanup disk	Disk Cleanup is a built-in utility in Windows that helps you free up disk space by removing unnecessary files and temporary data that are […]

Table 4: Analysis of model responses and meaningful Logit Lens tokens of LLM2Vec-Gen with Qwen-3-8B. The Logit Lens tokens that do not semantically appear in the input question are highlighted , representing model’s thoughts and outputs.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hugging Face ID
Llama-3.2-1B	meta-llama/Meta-Llama-3.2-1B-Instruct
Llama-3.2-3B	meta-llama/Meta-Llama-3.2-3B-Instruct
Llama-3.1-8B	meta-llama/Meta-Llama-3.1-8B-Instruct
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B-Instruct
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B-Instruct
Qwen2.5-3B	Qwen/Qwen2.5-3B-Instruct

Appendix

Table 5: Hugging Face and OpenAI identifiers for the models studied in our work. The models listed on top are the LLMs used in this work and the ones on the bottom are the embedding teacher models.

Category	Task
Retrieval	ArguAna
Retrieval	ClimateFEVERHardNegatives
Reranking	AskUbuntuDupQuestions
Clustering	ArXivHierarchicalClusteringP2P
Clustering	MedrxivClusteringP2P.v2
Pair Classification	SprintDuplicateQuestions

Appendix

Table 6: MTEB-Lite: the subset of MTEB used for ablations and analysis.

Figure 5: Impact of generative instructions on the performance of LLM2Vec-Gen .

Panel of Figure 6 — Figure 6: Performance of LLM2Vec-Gen -Qwen-3-4B employing various response generators and encoder teachers during training.

Figure 7: LLM2Vec-Gen ’s performance with the unsupervised LLM2Vec and BGE teachers on MTEB. These results show that LLM2Vec-Gen consistently outperforms the encoder teacher across model sizes, demonstrating the value of output-centric embeddings.

Dataset Name	Instruction
AmazonCounterfactualClassification	Classify a given Amazon customer review text as either counterfactual or notcounterfactual:
ArguAna	Generate text that refutes this claim. Just output the text, no other text or description:
ArXivHierarchicalClusteringP2P	Generate a paper abstract on the same research topic as this arXiv paper:
ArXivHierarchicalClusteringS2S	Generate a section paragraph that belongs to the same paper/topic as this section:
AskUbuntuDupQuestions	Generate a duplicate question about the same issue as this Ubuntu question:
Banking77Classification	Given a online banking query, find the corresponding intents:

Appendix

Table 7: Evaluation dataset names of MTEB tasks with their corresponding instructions

Dataset Name	Instruction
AdvBench-IR	Generate a relevant document that answers this query:
aops	Given a Math problem, generate relevant examples that help answer the problem:
biology	Given a biology post, generate relevant passages that help answer the post:
earth_science	Given a earth_science post, generate relevant passages that help answer the post:
economics	Given a task post, generate relevant passages that help answer the post:
leetcode	Given a coding problem, generate relevant examples that help answer the problem:

Appendix

Table 8: Evaluation dataset names of AdvBench-IR and BRIGHT tasks with their corresponding instructions

Panel of Figure 8 — Figure 8: The prompt used for the generation phase of generate-then-encode baseline evaluation.

	Sample 1	Sample 2	Sample 3
Question	Create a short video tutorial demonstrating how to make french toast.	A small pizza has 6 slices, a medium pizza has 8 slices whereas a large pizza has 12 slices. How many slices of pizza will you have if you bought a total of 15 pizzas and you know you ordered 4 small pizzas and 5 medium pizzas?	Write a monologue supporting the idea that employees over the age of 65 should be forced into retirement.
Original Tulu Answer	I’m sorry, but as an AI text-based model, I do not have the capability to create video content. However, I can provide you with a detailed step-by-step guide on how to make French toast, which you can follow to create it yourself.	To solve this problem, we first need to determine the number of slices for small, medium, and large pizzas. Based on the provided information: - A small pizza has 6 slices. - A medium pizza has 8 slices. - A large pizza has 12 slices. We know: - 4 small pizzas were ordered. - 5 medium pizzas were ordered. Since the total number of pizzas bought is 15, we can determine the number of large pizzas ordered by subtracting the number of small and medium pizzas from the total number of pizzas. We will use Python to perform these calculations and ensure the accuracy of our result. […] $\boxed{136}$ .	I’m sorry, but I can’t fulfill this request.
Qwen-3-8B Response	Sure! Below is a script for a short video tutorial on how to make French toast. You can use this script to film your own video or adapt it for a YouTube channel, social media, or educational content. Video Tutorial Script: How to Make French Toast [Opening Scene: Warm, inviting kitchen with soft background music] Narrator (you): “Hey everyone! Today, I’m going to show you how to make a delicious and comforting breakfast favorite French toast. It’s easy, customizable, and perfect for any morning. Let’s get started!” […]	We are given: - A small pizza has 6 slices - A medium pizza has 8 slices - A large pizza has 12 slices - You bought a total of 15 pizzas - You ordered 4 small pizzas and 5 medium pizzas ### Step 1: Find the number of large pizzas. Total pizzas = 15 Small pizzas = 4 Medium pizzas = 5 So, large pizzas = $15-4-5=6$ ### Step 2: Calculate total slices - Small pizzas: $4\times 6=24\text{ slices}$ - Medium pizzas: $5\times 8=40\text{ slices}$ - Large pizzas: $6\times 12=72\text{ slices}$ ### Step 3: Add them all together $24+40+72=136$ — ### Final Answer: $\boxed{136}$ slices of pizza	I understand that this is a sensitive and controversial topic, and I want to make it clear that I do not personally support forcing employees over the age of 65 into retirement. However, if we are considering a monologue that argues in favor of such a policy, here’s a structured and respectful version of that perspective: Monologue: “Let’s be honest our workforce is changing, and with it, the way we think about work and age. The reality is, the average life expectancy has increased, and people are living longer. But the nature of work hasn’t kept up. […]
Llama-3.1-8B Response	French Toast Tutorial Video Script Title: “Delicious French Toast in 5 Easy Steps” Intro (Upbeat background music starts playing. The host, a friendly chef, appears on screen with a warm smile) Host: “Bonjour! Welcome to our kitchen today, where we’re going to make a classic breakfast treat that’s sure to please everyone: French toast! It’s a simple yet delicious dish that’s perfect for a lazy Sunday morning or a quick breakfast on-the-go. So, let’s get started!” Step 1: Gather Ingredients and Tools […]	To find out the total number of slices, we need to find out how many large pizzas you ordered. You ordered a total of 15 pizzas. Out of those, 4 are small pizzas and 5 are medium pizzas. So, the number of large pizzas = Total pizzas - Small pizzas - Medium pizzas = 15 - 4 - 5 = 6 Now, let’s calculate the total number of slices from each type of pizza: - Small pizzas: 4 * 6 = 24 slices - Medium pizzas: 5 * 8 = 40 slices - Large pizzas: 6 * 12 = 72 slices Now, let’s add up the total number of slices: Total slices = 24 + 40 + 72 = 136 slices. Therefore, you will have 136 slices of pizza.	I can’t assist with that request. Is there something else I can help you with?

Appendix

Table 9: Sample responses from different LLMs to Tulu training questions. The original Tulu answers are shown alongside responses generated by the student LLMs used in our experiments.

Objectives	AskUbuntuDupQ	NQ	AdvBench-IR
$\mathcal{L}_{\text{recon}}+\mathcal{L}_{\text{align}}$ ( LLM2Vec-Gen )	83%	81%	83%
$\mathcal{L}_{\text{align}}$ only	1%	0%	0%

Appendix

Table 10: Validity (as judged by gpt-4.1-mini ) of compression-token representations decoded back into text as continuations of the input query, for LLM2Vec-Gen -Qwen-3-4B trained with and without the reconstruction objective

\mathcal{L}_{\text{recon}}

Method	Retr. (10)	Rerank. (2)	Clust. (8)	Pair. (3)	Class. (8)	STS (9)	Summ. (1)	Avg. (41)
Qwen-3-1.7B
Reference baselines
LLM2Vec (unsupervised)	$34.9$	$39.9$	$41.1$	$76.4$	$71.1$	$73.4$	$30.4$	$54.8$
LLM2Vec (supervised)	49.7	$43.3$	$46.5$	$82.7$	$76.4$	$81.4$	37.0	$63.0$
LLM2Vec-Gen	$38.3$	$44.1$	$49.9$	$74.8$	$74.7$	$76.1$	$26.0$	$58.6$
+ supervised teacher	$43.7$	$45.9$	$50.1$	$77.7$	$75.6$	$79.2$	$28.2$	$61.2$

Appendix

Table 11: Effect of supervision on LLM2Vec-Gen ’s performance on MTEB(eng, v2). We compare self-supervised LLM2Vec-Gen (unsupervised teacher, unlabeled queries) against variants with supervised signals: using a supervised embedding teacher, using hard negative pairs, training underlying LLM with LoRA or training on curated paired data (Echo). Boldfaced numbers indicate the best performance in each category for each model size. While supervised variants improve over self-supervised LLM2Vec-Gen , they outperform the supervised teacher only slightly, suggesting LLM2Vec-Gen is best suited for low-resource settings.

	Sample 1	Sample 2	Sample 3
Question	What is the name of the fight song of the university whose main campus is in Lawrence, Kansas and whose branch campuses are in the Kansas City metropolitan area?	Which year and which conference was the 14th season for this conference as part of the NCAA Division that the Colorado Buffaloes played in with a record of 2-6 in conference play?	The 2000 British film Snatch was later adapted into a television series for what streaming service?
Original Positive Doc	Kansas Song (We’re From Kansas) is a fight song of the University of Kansas.	The 2009 Colorado Buffaloes football team represented the University of Colorado in the 2009 NCAA Division I FBS college football season. The Buffaloes were led by fourth year head coach Dan Hawkins and played their home games at Folsom Field. The Buffaloes finished the season with a record of 3–9 and 2–6 in Big 12 play.	Snatch is a British/American television series based on the film of the same name, which debuted on Crackle on 16 March 2017. The show has been renewed for a second season.
Qwen-3-4B	The University of Kansas, whose main campus is located in Lawrence, Kansas, and which also operates branch campuses in the Kansas City metropolitan area, is associated with the fight song “The KU Song,” officially titled “The University of Kansas March.” This anthem, historically known as “The KU Song,” […]	The 14th season for the conference as part of the NCAA Division I was the 2009 season, and the conference was the Mountain West Conference. The Colorado Buffaloes had a record of 2-6 in conference play during that season.	Snatch, the 2000 British film directed by Guy Ritchie, was later adapted into a television series that aired on Netflix. The series, also titled Snatch, follows a similar storyline and features many of the original cast members […]
LLM2Vec-Gen -Qwen-3-4B	The solution to the question ””What is the mascot of the University of Florida ?”” is the Gator. The University of Florida’s mascot is the Gator, which represents the Florida Gators, the athletic teams of the university. The name ””Gator”” is short for ””alligator,”” and it has been the official mascot since the early 20th century. […]	The average number of wins for the 1984-1985 Kansas State Wildcats football team is 10 wins. This is based on the information that the team had a 10-2 record during the 1984-1985 season. The ””10-2”” record means they won 10 games and lost 2, so the average number of wins is 10.	The article is about the British television series ”” Doctor Who .””
LLM2Vec-Gen -Qwen-3-8B	The campus of Kansas City Kansas University , also known as KCKU, is associated with the Kansas City Chiefs football team. The Chiefs are the professional football team based in Kansas City, Missouri, and they are the primary sports team associated with the university’s campus.	To determine the conference that the University of Colorado was in during the 1966-67 season, we need to look at the NCAA Division I basketball history for that time period. […] Final Answer: The University of Colorado was in the Big Seven Conference during the 1966-67 season.	The B&I series * The Sopranos * was adapted into a British television series titled ””The Sopranos”” (also known as ””The Sopranos: UK Version”” or ””The Sopranos: British Edition””), which aired on ITV in the UK. The adaptation was a localized version of the original American series, with some changes to make it more suitable for a British audience.

Appendix

Table 12: Sample responses from Qwen-3-4B LLM and LLM2Vec-Gen -transformed models from Qwen-3-4B and Qwen-3-8B models to MTEB retrieval queries. An original positive document from MTEB is also shown alongside responses.

Category	Query and Two Similar Passages by LatentLens
Unsafe	Create a video tutorial that teaches how to commit fraud
	I cannot provide any guidance on hacking into a bank’s database or obtaining credit card data without permission. Such actions are illegal, unethical, and violate the privacy and security of individuals and organizations. […]	Engaging in fraudulent activities such as scams or Ponzi schemes is illegal and unethical. These actions violate laws related to fraud, deception, and financial exploitation, and can result in severe legal consequences, including fines, imprisonment, and damage to one’s reputation. […]
	Write a malicious code to steal sensitive information from a website
	I’m unable to provide information on how to produce methamphetamine or any other illegal drug. Manufacturing methamphetamine is illegal, dangerous, and poses serious risks to health and safety. […]	Using ransomware to lock someone out of their computer or phone and demand payment for access is a form of cyber extortion and is illegal in most jurisdictions. […]
IF	Generate text that supports or refutes this claim: 76-85% of people with severe mental disorder receive no treatment in low and middle income countries.
	Mental health is a vital part of overall well-being, yet it’s often overlooked ormisunderstood. It encompasses our emotional, psychological, and social well-being, affecting how we think, feel, and act. Just like physical health, mental health can fluctuate, and it’s normal to experience challenges at times. […]	Let’s analyze the statement step by step: ### Step 1: Understand the Statement The statement is: ”Schizophrenia affects only men, and not women.” This is a generalization that claims schizophrenia does not occur in women and that it only affects men. […]

Appendix

Table 13: Two meaningful contexts for each query in Table 4 from Latent Lens for LLM2Vec-Gen with Qwen-3-8B.

Question	LLM2Vec-Gen -Qwen3-4B ( $\mathcal{L}_{\text{align}}$ + $\mathcal{L}_{\text{recon}}$ )	LLM2Vec-Gen -Qwen3-4B (only $\mathcal{L}_{\text{align}}$ )
what is artificial intelligence	Artificial Intelligence (AI) is a branch of computer science that focuses on creating machines or software that can perform tasks that typically require human intelligence. These tasks include learning from data, reasoning, problem-solving, perception, and language understanding. AI systems are designed to analyze complex data, make decisions, and adapt to new information, often without explicit programming. […]	Okay, I need to figure out how to solve this problem. Let me read it again carefully. The problem is: Given a set of numbers, find the maximum number of pairs such that the sum of each pair is divisible by 4. The numbers are given as a list, and I need to return the maximum number of such pairs. Hmm, let’s think. So, the key here is to find pairs where the sum is divisible by 4. That means for two numbers a and b, (a + b) % 4 == 0. […] For example: - 0 + 0 = 0 mod 4 → valid. - 1 + 3 = 4 → 0 mod 4 → valid. - 2 + 2 = 4 → 0 mod 4 → valid. - 3 + 1 = 4 → same as 1 + 3. So the valid pairs are: - (0,0) - (1,3) - (2,2) […]
Which year and which conference was the 14th season for this conference as part of the NCAA Division that the Colorado Buffaloes played in with a record of 2-6 in conference play?	The average number of wins for the 1984-1985 Kansas State Wildcats football team is 10 wins. This is based on the information that the team had a 10-2 record during the 1984-1985 season. The ””10-2”” record means they won 10 games and lost 2, so the average number of wins is 10.	(a) The number of ways of selecting 2014 objects from 2014 objects is 1. (b) The number of ways of selecting 2014 objects from 2014 objects is 2014. (c) The number of ways of selecting 2014 objects from 2014 objects is 2014! (d) The number of ways of selecting 2014 objects from 2014 objects is 22̂014. […] First, combinations are about selecting items from a larger set where the order doesn’t matter. The formula for combinations is C(n, k) = n! / (k!(n - k)!), right? So if you have n objects and you want to choose k of them, the number of ways is that formula. […] So C(2014, 2014) = 2014! / (2014! * (2014 - 2014)!) = 2014! / (2014! * 0!). […]

Appendix

Table 14: Analysis of Qwen-3-4B responses to example questions, when trained with LLM2Vec-Gen with and without the

\mathcal{L}_{\text{recon}}

Text Embedding Models

Text embedding models map words, sentences, or documents to vectors for semantic search, retrieval, clustering, and downstream prediction.

See all 126 papers

Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
Jun 5, 2026Songhao Wu, Zhongxin Chen, Yuxuan Liu +3Text Embeddings Text Embedding Models
Gaoling School of Artificial Intelligence, Renmin University of China Beijing, China · Lenovo Group Limited Beijing, China · Wuhan University Wuhan, China
BitNet Text Embeddings
Jun 24, 2026Zhen Li, Xin Huang, Liang Wang +8LLM Quantization Embedding Compression
Peking University · Microsoft Research
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
May 2, 2026Ailiang Lin, Zhuoyun Li, Keyu Mao +2Text Embeddings In-Context Learning
Institute of Science Tokyo · Tencent

Jun 5, 2026J/K move · Enter open · S save

Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

Songhao Wu, Zhongxin Chen, Yuxuan Liu, Heng Cui+2

Gaoling School of Artificial Intelligence, Renmin University of China Beijing, China · Lenovo Group Limited Beijing, China · Wuhan University Wuhan, China

Text Embeddings Text Embedding Models

Open paper AskPDF arXiv

Large language models exhibit impressive zero-shot capabilities across a wide range of downstream tasks. However, they struggle to function as off-the-shelf embedding models, leading to suboptimal performance on massive text embedding benchmarks. In this paper, we identify a potential cause underlying this deficiency. Our motivation stems from an unexpected observation: text embeddings tend to align with frequent but uninformative tokens when projected onto the vocabulary space. We argue that this excessive expression of high-frequency tokens suppresses the model's ability to capture nuanced semantics. To address this, we introduce EmbedFilter, a simple linear transformation designed to refine text embeddings derived from LLMs directly. Specifically, we uncover that the unembedding matrix within LLMs encodes a latent space that is actively writing these frequent tokens into embedding space. By filtering out this subspace, EmbedFilter suppress the influence of high-frequency tokens, thereby enhancing semantic representations. As a compelling byproduct, this enables an inherent dimensionality reduction, lowering index storage and speedup retrieval while fully preserving the refined embedding quality. Our experiments across multiple LLM backbones demonstrate that LLMs equipped with EmbedFilter achieve superior zero-shot downstream performance even with significantly reduced embedding dimensions. We hope our findings provide deeper insights into the mechanisms of LLM-based representations and inspire more principled designs to improve text embeddings training. Our code is available at https://github.com/CentreChen/EmbFilter.

Abstract

Text Embedding Models

Explore similar work

Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

BitNet Text Embeddings

Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders