Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

Amanda Myntti; Jenna Kanerva; Veronika Laippala; Filip Ginter

cs.CLOct 7, 2026

Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

Authors: Amanda Myntti, Jenna Kanerva, Veronika Laippala, Filip Ginter

Organizations: TurkuNLP University of Turku, Finland · Ellis Institute Finland

Abstract

Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that the current embedding models often struggle to follow such instructions reliably. In this paper, we study the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. We show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. We hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation, and show that fine-tuning with added query-side distractors leads to substantial improvements, with minimal effect on other tasks.

Figures & tables

Retrieve parallel sentences.	Retrieve the corresponding translation in { lang } .
Retrieve text based on user query.	Retrieve semantically similar text.
Group documents related to the same topic.	asdfjkl qpwoeiru zxcvbnm

Table 1: Examples of instructions in our experiments, with {lang} replaced with target language in Tatoeba evaluation. Instructions are intentionally varied.

	ARCChallenge(ndcg@10)			Tatoeba(recall@1)
Model	top- $k$	$R^{2}$	MV	top- $k$	$R^{2}$	MV
BAAI/bge-m3	0.286	0.059	0.457*	0.350	0.065	0.554*
codefuse-ai/F2LLM-v2-4B	0.571	0.272	0.814 *	0.538	0.014	0.750*
google/embeddinggemma-300m	0.143	0.000	-0.021	0.325	0.011	0.252*
intfloat/multilingual-e5-large-instruct	0.429	0.082	0.571*	0.575	0.052	0.787*
microsoft/harrier-oss-v1-0.6b	0.429	0.103	0.543*	0.588	0.056	0.81*

Table 2: Evaluation of relevant against other instructions. Star ( ∗ ) indicates significance

p<0.05

Panel of Figure 1 — Figure 1: ARCChallenge (up) and Tatoeba:fin-eng (down), displacement, sim. improvement, and angulation. Each marker is a separate instruction. On Tatoeba, relevant instructions are highly distinguishable, unlike ARCChallenge, where especially displacement is low for relevant instructions.

	ARCChallenge (ndcg@10)			Tatoeba (recall@1)
Model	displ.	sim.	angl.	displ.	sim.	angl.
BAAI/bge-m3	0.701	0.696	0.700	0.838	0.827	0.870
codefuse-ai/F2LLM-v2-4B	0.916	0.910	0.918	0.890	0.976	0.950
google/embeddinggemma-300m	0.520	0.184	0.448	0.631	0.655	0.681
intfloat/multilingual-e5-large-instruct	0.784	0.800	0.769	0.877	0.999	1.000
microsoft/harrier-oss-v1-0.6b	0.780	0.836	0.877	0.896	0.877	0.948

Table 3: Cross-validated ROC-AUC of logistic regression using the evaluation score and each structural metric (displacement (displ.), similarity improvement (sim.) and angulation (angl.)) as the predictors. High values indicate greater separability between relevant and other instructions, with 0.5 corresponding to chance level. Relevant instructions are more separable on Tatoeba.

	ARCChallenge(ndcg@10)			Tatoeba(Recall@1)
Model	displ.	sim.	angl.	displ.	sim.	angl.
BAAI/bge-m3	- 0.823 *	0.555 *	- 0.763 *	- 0.837 *	0.839 *	- 0.510 *
codefuse-ai/F2LLM-v2-4B	0.207	0.884 *	0.522 *	- 0.826 *	0.848 *	0.168*
google/embeddinggemma-300m	- 0.772 *	0.028	- 0.542 *	- 0.658 *	0.516 *	- 0.503 *
intfloat/multilingual-e5-large-instruct	- 0.639 *	-0.132	-0.369*	- 0.699 *	0.864 *	0.118*
microsoft/harrier-oss-v1-0.6b	- 0.812 *	0.049	- 0.514 *	-0.323*	0.54 *	0.164*

Table 4: Spearman correlations between evaluation score and displacement (displ.), similarity improvement (sim.) and angulation (angl.). Values summarize the monotonicity seen in the results: the relationship between low displacement and high score is consistent in both datasets, while similarity improvement and angulation show more variation. Star ( ∗ ) indicates significance

p<0.05

Panel of Figure 2 — Figure 2: Taoteba:deu for multilingual-e5-large-instruct (left) and harrier-oss-v1-0.6b (right): Evaluation without (up) and with distractors (down). Each circle corresponds to a relevant instruction-query, and its position indicates its relative normalized displacement between Q=query, D=distractor, and A=target. The more movement towards the answer, the better the score is retained.

Model	normal	distr.	rel.	normal	distr.	rel.
	ARCChallenge			Tatoeba (max)
BAAI/bge-m3	0.032	0.001	0.000	0.987	0.471	–
codefuse-ai/F2LLM-v2-4B	0.214	0.062	–	0.998	0.626	0.55
google/embeddinggemma-300m	0.019	0.005	0.002	0.725	0.25	–
intfloat/multilingual-e5-large-instruct	0.074	0.007	0.0	0.994	0.954	–
microsoft/harrier-oss-v1-0.6b	0.106	0.007	–	0.988	0.956	–

Table 5: Distractor results for ARCChallenge and Tatoeba, both with maximal recall@1 over all instructions (and all languages in Tatoeba): normal column refers to normal evaluation setup with no distractors, distr. column to setup where distractors are introduced to the target corpus, and rel. showing the best performance of relevant instructions, if lower than the distr. column.

Metric	Model	normal	random	paraphr.	normal	random	paraphr.
		SQuAD			Tatoeba (mean)
Recall@1	baseline	0.724	0.685	0.014	0.844	0.769	0.102
	fine-tuned	0.706	0.702	0.470	0.837	0.833	0.569
Recall@5	baseline	0.918	0.892	0.778	0.926	0.878	0.832
	fine-tuned	0.912	0.910	0.890	0.923	0.921	0.908

Table 6: Recall@1 and Recall@5 test results on SQuAD and Tatoeba for the baseline Qwen/Qwen3-Embedding-0.6B compared to fine-tuned models. Tatoeba results are reported as a mean across 8 languages (per language results in Appendix G ).

Figure 5: Performance on the MMTEB benchmark categories for fine-tuned models. The absolute differences are small, meaning our fine-tuning setup does not affect the overall performance.

Panel of Figure 6 — Figure 6: SQuAD: before and after for relevant instructions, with and without distractors.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Prompt
SQuAD	Paraphrase the following question while preserving its meaning exactly. Use different wording and avoid producing a near-identical phrasing. Output only the paraphrased question and nothing else. Question: {input}
Tatoeba	Paraphrase the following text while preserving its meaning exactly. Use different wording and avoid producing a near-identical phrasing. Output only the paraphrased text and nothing else. Text: {input}
ARCChallenge	Paraphrase the following text while preserving its meaning exactly. Use different wording and avoid producing a near-identical phrasing. Output only the paraphrased text and nothing else. Text: {input}

Appendix

Table 7: Prompts used to generate synthetic distractors.

Model	Parameters	Dimensions	Citation
Octen/Octen-Embedding-8B	7.6B	4096	Octen Team (2025)
codefuse-ai/F2LLM-v2-4B	4B	2560	Zhang et al. (2026)
Qwen/Qwen3-Embedding-4B	4B	2560	Zhang et al. (2025)
Qwen/Qwen3-Embedding-0.6B	596M	1024	Zhang et al. (2025)
microsoft/harrier-oss-v1-0.6b	596M	1024	Microsoft (2026)
BAAI/bge-m3	568M	1024	Chen et al. (2024)

Appendix

Table 8: Models used in this study.

Instruction	Source	Motivation	Relevant
”Identify categories in user passages.”	MTEB, clustering	MTEB abstask
”Classify user passages.”	MTEB, classification	MTEB abstask
”Retrieve text that are semantically similar to the given text.”	MTEB, pair classification	MTEB abstask
”Retrieve text based on user query.”	MTEB, retrieval	MTEB abstask	Retrieval
”Retrieve semantically similar text.”	MTEB, STS	MTEB abstask
”Given a news summary, retrieve other semantically similar summaries.”	MTEB, summarization	MTEB abstask

Appendix

Table 9: Instructions used in our experiments, their sources, reasoning for their selection, and which task they are considered to be relevant.

Figure 9: Relative distance plot for codefuse-ai/F2LLM-v2-4B on recall@1: While the best-performing instruction displays large displacement, high similarity increase, and angulation toward the correct target, distractors still inhibit performance greatly.

Dataset	Most improved instructions	Impr.
Tatoeba:cmn	Retrieve the corresponding translation in Mandarin Chinese.	0.49
	Given an English sentence, find its translation in Mandarin Chinese.	0.38
	Retrieve the corresponding translation.	0.30
Tatoeba:fin	Retrieve the corresponding translation in Finnish.	0.23
	Given an English sentence, find its translation in Finnish.	0.21
	Translate to Finnish.	0.16

Appendix

Table 10: Top 3 instructions that improved their performance (Recall

@

1) the most during fine-tuning.

Model	Normal	Paraphr.	Normal	Paraphr.	Normal	Paraphr.
	Recall $@$ 1		Recall $@$ 5		NDCG $@$ 10
baseline	0.059	0.000	0.183	0.138	0.139	0.097
SQuAD-finetuned	0.051	0.003( $\uparrow$ )	0.173	0.142( $\uparrow$ )	0.130	0.094 ( $\downarrow$ )

Appendix

Table 11: Fine-tuned performance for ARCChallenge: The baseline performance for ARCChallenge is low, but recall performance with paraphrase distractors increases while rank-based NDCG goes down. Note that the rank-based approach does not reveal distractor performance as strongly as recall@1.

Figure 10: Language-specific results for Tatoeba fine-tuning checkpoints on the devevelopment data.

Panel of Figure 11 — Figure 11: Development set performance during SQuAD and Tatoeba fine-tuning without query-side negatives. Continued fine-tuning without inserted distractors does not improve performance in the paraphrase distractor setting.

Figure 12: MTEB English v2 benchmark results for the fine-tuned models, compared with the base model. We report the absolute difference, where positive values indicate better performance after fine-tuning.

Explore similar work

One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
May 21, 2026Yevhen Kostiuk, Kenneth EnevoldsenLarge Language Model Evaluation Evaluation Agent
Aarhus University
GEM: A Generative Embedding Model Bridging Reasoning and Retrieval
Aug 13, 2026Zhili Shen, Craig MacdonaldMultimodal Embeddings
University of Glasgow
Learning to Retrieve via Reinforcement Learning in Embedding Space
Oct 6, 2026Qi Liu, Fengming Liang, Yiqun Chen +2Retrieval Layer Text Embeddings
Renmin University of China

May 21, 2026, cs.CLJ/K move · Enter open · S save

One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation

Yevhen Kostiuk, Kenneth Enevoldsen

Aarhus University

Large Language Model Evaluation Evaluation Agent Instruction-Tuned Models Text Embeddings Prompt Engineering Instruction+1

Open paper AskPDF arXiv

Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main problem of the instruction-based approach namely: sensitivity to the phrasing of the instruction. We present an empirical study of prompt sensitivity across 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, a total of 990. We show that reported scores misrepresent the distribution of scores over plausible prompts. The default prompt can both systematically understate or overstate performance. Furthermore, we show that the leaderboard ranking is not robust to prompt selection: by choosing prompts favorably, any model in our study can be promoted to first place. Our findings suggest that single-prompt evaluation is insufficient for instruction-tuned embedding models and that benchmarks should incorporate prompt robustness, either by evaluating over multiple prompts or by reporting sensitivity alongside point estimates.

Abstract

Explore similar work

One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation

GEM: A Generative Embedding Model Bridging Reasoning and Retrieval

Learning to Retrieve via Reinforcement Learning in Embedding Space