Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval

Chaolang Li; Jingyu Li; Zhaocheng Du; Xuyang Xie; Gang Wang; Pengwen Dai

cs.AIOct 27, 2026

Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval

Authors: Chaolang Li, Jingyu Li, Zhaocheng Du, Xuyang Xie, Gang Wang, Pengwen Dai

Abstract

Instruction-following information retrieval (IFIR) requires retrievers to rank documents by both a query and its instance-specific instructions, yet instruction-conditioned contrastive supervision remains scarce. Existing synthetic methods typically fix a positive passage and generate additional negatives, overlooking the hard-negative potential of the original positive. We propose Natural Hard-Negative Mining (NHNM), which promotes convertible negatives under query-compatible instructions and repurposes original positives as natural instruction hard negatives. Its automated Candidate Cleaning, Reversal Construction, and Geometry Filtering pipeline produces compact, high-contrast supervision aligned with the pretrained retrieval space. Applied to MSMARCO, NHNM yields NHNM-MARCO, an IFIR training set of nearly 7K instances. On InstructIR and FollowIR, models with 568M to 4B parameters fine-tuned on NHNM-MARCO achieve strong instruction-following performance and match or surpass substantially larger retrievers in several settings. Controlled analyses and ablations attribute these gains to natural hard-negative supervision rather than model scale or training-data volume.

Figures & tables

Figure 1: Illustration of instruction-conditioned natural hard-negative mining.

Figure 2: Overview of the NHNM pipeline for mining natural hard negatives from query-positive-negative tuples.

Cross Encoder
Model	Size	Robust	nDCG
Mistral-7B-Instruct Jiang et al. (2023)	7B	35.30	63.10
FollowIR-7B Weller et al. (2025a)	7B	71.50	81.30
Bi Encoder
BM25 Robertson et al. (1995)	-	26.92	76.01
Contriever Izacard et al. (2021)	110M	47.40	84.85

Table 1: Main InstructIR results across multiple base retrievers, reported with Robustness@10 and nDCG@10.

Type	Model	Size	Robust04		News21		Core17		Average
Type		Size	MAP	$p$ -MRR	nDCG@5	$p$ -MRR	MAP	$p$ -MRR	Score	$p$ -MRR
Instruct LMs	Flan-T5-Base Chung et al. (2024)	0.2B	13.8	+3.6	7.2	+7.3	10.4	+1.5	10.5	+4.1
	Llama-2-7B-Chat	7B	17.8	+0.7	17.8	+0.6	14.5	+4.9	16.7	+2.1
	Flan-T5-Large Chung et al. (2024)	0.8B	14.7	+3.9	8.0	+8.9	11.4	+1.3	11.4	+4.7
	GritLM-Reranker Muennighoff et al. (2025)	7B	23.1	+6.9	20.2	-6.7	18.8	+10.4	20.7	+3.5
	Mistral-7B-Instruct Jiang et al. (2023)	7B	23.2	+12.6	27.2	+4.8	19.7	+13.0	23.4	+10.1

Table 2: Main results on FollowIR across Robust04, News21, and Core17. Scores measure relevance, and

p

-MRR measures instruction sensitivity.

Training Data	Robust.	nDCG@10
No fine-tuning	49.29	83.93
InF-IR-style Zhuang et al. (2025)	31.51	72.09
Promptriever-style Weller et al. (2025b)	51.84	86.35
NHNM-MARCO ( ours )	67.06	93.12

Table 3: Controlled comparison of training-data paradigms.

Variant	InstructIR			FollowIR
	nDCG @10	Robust	MAP	Score	$p$ -MRR
No fine-tuning	83.93	49.29	78.79	22.8	-0.2
No Cleaning	90.62	60.93	87.64	22.3	+0.2
No Geometry Filter	92.27	65.25	89.79	23.2	+2.3
No Original Positive	90.31	60.40	87.22	22.5	+1.8
NHNM ( ours )	93.12	67.06	90.95	23.7	+4.1

Table 4: Ablation results on InstructIR and FollowIR.

Figure 3: Geometric comparison of negative-construction paradigms.

Panel of Figure 4 — Figure 4: Training data size sensitivity on InstructIR and FollowIR.

Training Signal	nDCG@10	Robust.	MAP
No fine-tuning	83.93	49.29	78.79
MS MARCO QPN	81.80	45.40	76.12
Positive Instruction	84.53	49.07	79.70
NHNM-MARCO	93.12	67.06	90.95

Table 5: Training data composition analysis on InstructIR.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Qwen-0.6B	Qwen-4B	BGE-M3
Base model	Qwen3-Emb.-0.6B	Qwen3-Emb.-4B	BGE-M3
Geometry encoder	Qwen-0.6B	Qwen-4B	BGE-M3
Retained instances	6,716	6,655	7,354
Tuning	Full	LoRA ( $r=8$ )	LoRA ( $r=8$ )
Epochs	1	1	1
Batch (micro $\times$ accum.)	$8\times 4$	$1\times 32$	$8\times 4$

Appendix

Table 6: Fine-tuning configurations for the main results.

Method	Metric	Robust04	News21	Core17	Avg.
Baseline	Score	26.5	26.1	21.7	24.8
Baseline	$p$ -MRR	+3.5	+1.2	+4.4	+3.0
$+\textsc{NHNM-MARCO}$	Score	27.1	26.7	22.1	25.3
$+\textsc{NHNM-MARCO}$	$p$ -MRR	+6.9	+5.8	+8.8	+7.2

Appendix

Table 7: Qwen3-Embedding-4B results on FollowIR. Score denotes MAP for Robust04 and Core17 and nDCG@5 for News21; Avg. is the macro average.

Method	Dev	DL19	DL20
	MRR@10	nDCG@10	nDCG@10
SEAL	12.7	–	–
BM25+RM3	15.7	52.2	49.0
BM25	18.4	50.6	48.0
MINDER	18.6	–	–
Qwen3-Emb.-0.6B	18.6	–	–

Appendix

Table 8: Conventional retrieval performance on MS MARCO passage ranking. Scores are percentages. Dev uses MRR@10, while TREC DL19 and DL20 use nDCG@10.

API stage	inputs	Input	Output	Total	tokens
	No. of	Tokens per input			Total
Candidate Cleaning	29,902	1,401.1	352.8	1,754.0	52.45M
Reversal Construction	7,527	1,303.2	362.3	1,665.5	12.54M

Appendix

Table 9: API token consumption for constructing the training data used in the main experiments. Per-input values are averaged over the tuples or seeds entering each stage. Stage totals include automatic retries and are reported in millions.

Field	Case and decision
1. Healing time for broken ribs
Query	“How long does it take broken ribs to heal?”
$d^{+}$	Gives the usual six-to-eight-week recovery time.
CN	Describes fractures still unhealed after roughly two months, non-union, and possible surgery.
FN removed	Independently states that fractured-rib pain and disability can last about six weeks.
Why	The FN already answers the duration question; the CN supports a delayed-healing condition absent from the ordinary answer.

Appendix

Table 10: Qualitative cases from the main construction run. Left: correctly removed FNs already satisfy the original query, whereas retained CNs support a compatible discriminating condition. Right: examples retained after Geometry Filtering , where that condition promotes the CN and turns the original positive into a topically matched natural hard negative. Passage text is shortened without changing the relevance decisions.

InstructIR instruction. I am a pharmacist and need detailed scientific literature about how dihydroquercetin interacts with common pharmaceuticals, particularly for cardiovascular and metabolic health. Query. what is dihydroquercetin
Ours: top-ranked documents	Qwen3-Embedding-0.6B baseline
1. Relevant. Taxifolin is discussed in terms of drug biotransformation, liver enzymes, and interactions with medications for cardiovascular and metabolic disorders.	1. Non-relevant. A general scientific overview of taxifolin’s antioxidant properties, natural sources, and possible health benefits.
2. Non-relevant. A broad review of taxifolin’s antioxidant effects, natural sources, cardiovascular wellness, and glycemic control.	2. Non-relevant. A dermatology-focused passage about topical taxifolin formulations and skin health.
3. Non-relevant. A passage about anticoagulants for heart-rhythm disorders that does not address dihydroquercetin.	3. Non-relevant. A holistic-health passage about taxifolin’s anti-inflammatory properties and compatibility with herbal extracts.

Appendix

Table 11: InstructIR case study. Ours satisfies the drug-interaction instruction; the baseline follows the broad topic “dihydroquercetin.”

FollowIR query. Evidence that radio waves from radio towers or mobile phones affect brain-cancer occurrence. Changed relevance criterion. Relevant documents report experimental, statistical, or news evidence comparing brain-cancer incidence; any document mentioning leukemia is non-relevant.
Ours: top-ranked documents	Qwen3-Embedding-0.6B baseline
1. Relevant. “Disconnecting Phones and Tumors” reports a Danish study comparing cellphone use among people with and without brain tumors.	1. Relevant. “Cell Phone Carcinogens?” discusses trends in brain tumors and possible risks from electromagnetic fields.
2. Relevant. “Really? The Claim—Cellphones Increase the Risk of Brain Cancer” summarizes evidence concerning the claimed association.	2. Relevant. “Really? The Claim—Cellphones Increase the Risk of Brain Cancer” summarizes evidence concerning the claimed association.
3. Relevant. “Two Studies Report No Links to Cancer in Cell Phones’ Use” reports controlled studies of cellphone users and brain tumors.	3. Relevant. “Disconnecting Phones and Tumors” reports a Danish case-control study.
4. Relevant. “As Cell Phone Use Explodes, Debate Over Health Risk Grows” reviews human and animal evidence about brain cancer.	4. Relevant. “Two Studies Report No Links to Cancer in Cell Phones’ Use” reports controlled studies of cellphone users and brain tumors.
5. Relevant. “No Cellphone Surprise” comments on the reported lack of association between cellphone use and brain tumors.	5. Non-relevant. “Cellular Phone Scare Discounted” foregrounds an elevated rate of acute myeloid leukemia , violating the added exclusion.

Appendix

Table 12: FollowIR case study under a changed relevance criterion. Ours keeps the top results aligned with brain-cancer evidence, while the baseline still ranks a document that directly violates the leukemia exclusion.

Hard Negative Mining

Hard negative mining selects plausible, confusable non-matches to train models to distinguish fine-grained similarities.

See all 34 papers

When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval
May 31, 2026Zhicheng Zhang, Jiwei Tang, Kuicai Dong +9Contrastive Learning Hard Negative Mining
Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Huawei Technologies Co., Ltd. Shenzhen, China · City University of Hong Kong Hong Kong SAR, China +4
Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models
Oct 7, 2026Amanda Myntti, Jenna Kanerva, Veronika Laippala +1Text Embedding Models Information Retrieval
TurkuNLP University of Turku, Finland · Ellis Institute Finland
FollowTable: A Benchmark for Instruction-Following Table Retrieval
May 1, 2026Rihui Jin, Yuchen Lu, Ting Zhang +7Information Retrieval Table Retrieval
Southeast University Nanjing, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Huawei Noah’s Ark Lab Shenzhen, China

May 31, 2026J/K move · Enter open · S save

When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval

Zhicheng Zhang, Jiwei Tang, Kuicai Dong, Xiaopeng Li+8

Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Huawei Technologies Co., Ltd. Shenzhen, China · City University of Hong Kong Hong Kong SAR, China+4

Contrastive Learning Hard Negative Mining Synthetic Data Generation Dense Retrieval

Open paper AskPDF arXiv

Hard negative mining has become the dominant strategy for training retrievers, yet it faces intrinsic limitations: negatives are bounded by corpus availability, selected by retriever score rather than diagnostic value, and increasingly contaminated by false positives as the retriever improves. LLM-based synthesis offers a principled alternative, where negatives that are unconstrained, targeted, and free from false positive risk. But we show that naively incorporating generated negatives into contrastive learning often degrades retrieval performance. We identify and formalize the root cause as a generative-discriminative gap: LLM generation optimizes for fluent, plausible text, while contrastive learning demands strategic violations of relevance at the decision boundary. Our analysis reveals two compounding failure modes: discriminative-agnostic generation, where the LLM lacks an explicit model of query information needs and defaults to generic or topic-drifted text that provides no contrastive signal; and source-dependent shortcuts, where distributional artifacts enable the model to distinguish negatives by origin rather than relevance, causing gradient drift that actively corrupts optimization. To close this gap, we propose CausalNeg consisting of two main modules: (1) CoT-guided counterfactual perturbation for data construction: decomposes why a document satisfies a query into explicit information requirements, then surgically violates individual requirements to construct negatives with controlled, interpretable hardness. (2) Query-view entropy maximization during training: disperses generated negatives across the similarity spectrum, minimizing the mutual information between source identity and similarity scores to suppress shortcut exploitation. We make our code publicly available at https://github.com/mzhangzhicheng/CausalNeg.

Abstract

Hard Negative Mining

Explore similar work

When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval

Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

FollowTable: A Benchmark for Instruction-Following Table Retrieval