AI agents require specialised retrieval across domains and tasks, but adapting embedding models is constrained by costly manual training-data design. Here we introduce SYNTRA, a self-improving agentic framework that turns a few retrieval examples into reusable synthesis instructions. Given an unlabelled corpus and a few relevant query-document pairs, SYNTRA infers implicit information needs and relevance criteria, then uses feedback on trial outputs to refine instructions for query generation and relevance assessment. Experiments across 24 datasets spanning 15 domains demonstrate that the same synthesis workflow produces effective training data, benefiting even strong embedding models with and without existing labelled training sets. Self-improvement strengthens this supervision on all six evaluated tasks, yielding better retrieval than human calibration on four. The training value persists at larger scales: on MS MARCO, retrieval gains continue across a 32-fold range of synthetic training-data sizes. SYNTRA thus enables scalable embedding-model adaptation with minimal human intervention.
Figures & tables
Figure 1: Constructing task-specific training data for specialised retrieval. a , A coding agent searches a repository and retrieves the implementation that satisfies a stated constraint. A research agent retrieves supporting studies from a paper corpus. b , Documents differ across domains in terminology, structure and knowledge. Within a single domain, the user’s objective changes which documents are relevant. For the same query, similar-argument and counterargument retrieval rank the same two arguments in opposite orders. c , SYNTRA receives an unlabelled corpus and a small set of few-shot examples, each pairing a query with a positive document. Few-shot-guided self-improvement turns a shared synthesis workflow into a task-specific one. This workflow then constructs (query, positive, negative) training records at scale. SYNTRA is run separately for each retrieval task using the same framework. The examples shown span software, scientific literature and case law. Within a domain, the same two arguments exchange positive and hard-negative roles under different retrieval needs. Pos., positive document; Hard neg., hard-negative document. All examples are schematic and abbreviated, not generated samples or measured retrieval results.
Figure 2: Task preparation and adaptation in SYNTRA. a , SYNTRA receives an unlabelled corpus and a small set of few-shot examples pairing queries with positive documents. Corpus preprocessing builds a full-corpus index and selects diverse source documents. The task definition is inferred from the few-shot examples; optional user review is recommended. Point groups schematically illustrate semantic diversity, not clustering outputs. b , Small-sample generate–evaluate–refine cycles establish reusable task-adapted instructions before production. Each test loop ends when verification passes or its iteration limit is reached. The shared instructions then remain fixed throughout production.
Figure 3: Production-scale synthesis in SYNTRA. Task-adapted instructions established in test mode remain fixed during production. a , For each selected document, an LLM jointly extracts semantic identifiers and constructs query attributes with dimensions and options. b , Query synthesis forms plans that pair one semantic identifier with one option from each query-attribute dimension. It then writes one query per plan using the same task-adapted instruction for both steps. c , Relevance-label synthesis retrieves candidates from the full corpus and labels them with the task-adapted relevance-label instruction. The labelled candidates are assembled into (query, positive, negative) training records. Document content, plans, queries, candidates and labels are illustrative.
Figure 4: Cross-task applicability, self-improvement and quality of SYNTRA-generated supervision. a , Coverage of 24 datasets spanning 15 domains, with relative retrieval gains for the 23 task-specific comparisons against the MS MARCO-trained baseline. Numeric labels report percentage gains; radial extensions use a nonlinear scale. The CodeTrans-DL gain averages three reported variants. MS MARCO is included as the common training corpus without a task-adaptation gain. b , Retrieval performance of the MS MARCO-trained baseline and models trained on data generated without self-improvement, with automatic self-improvement or with human calibration. The six tasks were TheoremQA (theorems), ArguAna, NFCorpus, FiQA-2018 (FiQA), SCIDOCS and Text2SQL. c , Mean nDCG@10 on BEIR and NanoBEIR for models trained with original or SYNTRA-generated queries under matched conditions. d , Agreement between Qwen3-30B-A3B relevance labels and reference labels derived from votes by seven human annotators. Binary and three-class agreement were evaluated on 150 query–document pairs, with 50 pairs from each of three tasks. Binary evaluation distinguishes relevant from non-relevant documents; three-class evaluation distinguishes positive, hard-negative and easy-negative labels. Training-label precision reports exact-category matches among the 87 evaluation pairs labelled positive or hard negative by the LLM. Error bars indicate 95% confidence intervals from 10,000 bootstrap resamples stratified by dataset. The 95% confidence intervals were 61.3–76.0% for three-class agreement, 86.7–95.3% for binary agreement and 74.7–90.2% for training-label precision. Four pairs with tied highest three-class vote counts were assigned hard-negative reference labels.
Figure 5: Few-shot-guided self-improvement on FiQA. a , Refinement of query-generation instructions. The source excerpt and queries are verbatim production examples generated in separate runs with instructions corresponding to the first and third refinement rounds. Attribute plans were not held fixed between these runs. Feedback summarizes the original refinement history. b , Refinement of relevance-annotation instructions using a known-positive few-shot example. Feedback clarifies that explaining why a plan may fail can help answer the user’s question. Text in this panel provides an illustrative summary of the recorded calibration episode. Bottom, FiQA retrieval performance without and with self-improvement, reflecting joint refinement of query-generation and relevance-annotation instructions.
Figure 6: Model adaptation and ablations of SYNTRA. a,b , BGE-M3 ( a ) and Qwen3-Embedding-8B ( b ) results with and without existing task-specific labelled training data. Ori denotes models fine-tuned on existing task-specific labelled data. Dashed lines separate the two settings; right-hand values report gains over Ori or the corresponding pretrained baseline. c , NanoBEIR ablations of the core synthesis components. d , Comparison of the shared MS MARCO baseline, a task-specific reference and SYNTRA. Reference training data were constructed using ReasonEmbed queries for the two TheoremQA tasks and official queries for CosQA and Synthetic Text2SQL. e , MS MARCO nDCG@10 as SYNTRA-generated training data increase from 10,000 to 320,000 examples. f , Retrieval performance as the number of queries generated per document increases on CodeTrans-DL and MedicalQA. Dashed lines show the MS MARCO-trained reference scores.
Task
Training records
Production time (h)
MS MARCO (original queries)
9,546
4.444
MS MARCO (SYNTRA queries)
9,486
5.789
LeCaRDv2 Li et al. (2024)
8,278
63.786
FiQA
9,360
6.489
SCIDOCS
9,742
7.197
SciRepEval Search
9,302
7.040
Extended Data Table 1: Production-stage runtimes and finalised training-data sizes across 24 retrieval tasks. Production timings are reported for one batch per listed task and query-source condition; when multiple batches were considered, the shorter-running batch was retained. Training records are valid entries in a finalised JSONL (JSON Lines) training-data file. Each batch has two equally sized training-file variants. Counts are reported per variant, without summing the variants or multiplying by training epochs. Timings include identifier extraction and attribute construction, query generation, candidate retrieval and final relevance annotation. Calibration, prebuilt indexing, failed attempts, waiting between resumed runs, final training-format conversion, and model training and evaluation are excluded. For resumed runs, elapsed times for the relevant completed stages are summed. For MS MARCO original queries, the reported runtime covers retrieval and annotation only. The measurements describe selected batches rather than task-average runtimes or total experimental costs. Data production used a system equipped with six NVIDIA GeForce RTX 5090 GPUs. Four hosted the generative model, and the other two were used only during embedding-model operations. Times are elapsed production-stage times, rather than GPU-hours or sums of per-GPU runtimes.
Scope
Label space
n
Matches
Agreement
κ
Macro-P
Macro-R
Macro-F1
All samples
Three class
150
103
0.687
0.510
0.713
0.749
0.699
MS MARCO
Three class
50
37
0.740
0.600
0.756
0.764
0.746
Text2SQL
Three class
50
32
0.640
0.444
0.742
0.824
0.704
TheoremQA (theorems)
Three class
50
34
0.680
0.484
0.670
0.725
0.675
All samples
Binary
150
137
0.913
0.719
0.875
0.846
0.859
MS MARCO
Binary
50
44
0.880
0.705
0.893
0.829
0.851
Extended Data Table 2: Agreement between LLM-generated relevance labels and human reference labels overall and by dataset. Human reference labels were derived from seven annotators as described in Methods. Agreement is the fraction of exact matches, and κ denotes Cohen’s κ . Three-class evaluation distinguishes positives, hard negatives and easy negatives; binary evaluation merges the two negative classes. Macro-P, macro-R and macro-F1 denote class-averaged precision, recall and F1, respectively.
a. Three-class confusion matrix
Human reference
LLM positive
LLM hard negative
LLM easy negative
Positive
22
8
0
Hard negative
5
50
32
Easy negative
0
2
31
Extended Data Table 3: Confusion matrix and class-wise performance of LLM-generated relevance labels. Human reference labels were derived from seven annotators as described in Methods. a , Three-class confusion matrix. Rows denote human reference categories and columns denote LLM-assigned categories. b , Class-wise performance. Human n and LLM n give the number of evaluation pairs assigned to each category. Among the 87 evaluation pairs labelled positive or hard negative by the LLM, 72 matched the human reference category, yielding a pooled precision of 82.8%. The pooled precision combines the counts from both categories rather than averaging their precisions. These are the categories retained in the hard-negative-only training export.
a. Query and training-data sizes
Task
Query source
Query entries
Records per variant
Combined input rows
TheoremQA (questions)
ReasonEmbed
7,184
5,873
11,746
SYNTRA
9,852
9,006
18,012
TheoremQA (theorems)
ReasonEmbed
8,861
6,947
13,894
SYNTRA
9,880
9,517
19,034
CosQA
Official
9,020
6,599
13,198
Extended Data Table 4: Data sizes and retrieval performance for the task-specific query-source comparisons. a , Nonempty query entries submitted for relevance annotation, valid training records per positive-selection variant, and combined input rows supplied to training. Query entries include repeated query strings. Each valid record contains at least one positive and one hard negative. The two equally sized variants differ in positive-document selection as described in Methods and are concatenated without deduplication; combined input rows therefore equal twice the per-variant count. Input rows count supplied records before batching, rather than distinct queries or examples consumed over training epochs. b , nDCG@10 on a 0–100 scale, corresponding to Fig. 6 d. The MS MARCO-trained model provides the shared baseline. Reference queries are from ReasonEmbed for the two TheoremQA tasks and official training pairs for CosQA and Synthetic Text2SQL. All task-specific query sources undergo SYNTRA relevance annotation and the shared training procedure.
Model family
Model
BEIR
NanoBEIR
BEIR rank
NanoBEIR rank
Encoder-only
snowflake-arctic-embed-l
55.980
65.360
1
2
snowflake-arctic-embed-m
54.900
64.420
2
3
snowflake-arctic-embed-xs
50.150
61.170
8
7
bge-large-en-v1.5
54.340
65.910
3
1
bge-base-en-v1.5
53.230
63.760
4
4
bge-small-en-v1.5
51.660
62.650
5
5
Extended Data Table 5: Paired BEIR and NanoBEIR results for the 18 models used in the consistency analysis. Values are suite-level mean nDCG@10 scores on a 0–100 scale. Ranks are computed separately within each model family; rank 1 denotes the highest score.
Model family
n
Spearman’s ρ
P value
Within one rank
Encoder-only
9
0.933
2.0×10−4
8/9
Decoder-only
9
0.817
7.0×10−3
6/9
Extended Data Table 6: Rank consistency between NanoBEIR and BEIR within embedding-model families. Spearman’s ρ was computed from paired suite-level mean nDCG@10 scores. Ranks were assigned separately within each family, with rank 1 denoting the highest score. P values are two-sided.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Setting
Initialization
Qwen3-0.6B-auto-eos
Pooling
Last token
Embedding normalization
L2 normalization
LoRA configuration
Rank 32, alpha 64, dropout 0.1
Learning rate
10−4
Global query batch size
96
Appendix
Supplementary Table 1: Shared training settings for the self-improvement comparison, component ablations and MS MARCO query-source comparison.
Parameter
BGE-M3
Qwen3-Embedding-8B
Qwen3-8B-auto-eos (scaling)
Learning rate
10−5
10−4
10−4
Training epochs
1; 5 for SpartQA
1
1
Query batch size per GPU
2; 16 for SpartQA
16
16
Pooling
CLS token
Last token
Last token
Embedding normalization
L2
L2
L2
Maximum query / passage length (tokens)
512 / 512
512 / 512
512 / 512
Appendix
Supplementary Table 2: Training settings for adaptation of strong embedding models and the MS MARCO data-scaling experiment. The strong-model settings apply to LeCaRDv2, EvidenceBench, SciRepEval Search, TheoremQA questions, SpartQA and WinoGrande.
Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.
Data quality remains a critical bottleneck in developing capable, competitive models. Researchers have explored many ways to generate top quality samples. Some works rely on rejection sampling: generating lots of synthetic samples and filtering out low-quality samples. Other works rely on larger or closed-source models to extract model weaknesses, necessary skills, or a curriculum off of which to base data generation. These works have one common limitation: there is no quantitative approach to measure the impact of the generated samples on the downstream learner. Active learning literature provides exactly this, in the form of acquisition functions. Acquisition functions measure the informativeness and/or influence of data, providing interpretable, model-centric signals. Inspired by this, we propose AcquisitionSynthesis: using acquisition functions as reward models to train language models to generate higher-quality synthetic data. We conduct experiments on classic verifiable tasks of math, medical question-answering, and coding. Our experimental results indicate that (1) student models trained with AcquisitionSynthesis data achieve good performance on in-distribution tasks (2-7% gain) and is more robust to catastrophic forgetting, and (2) AcquisitionSynthesis models can generate data for other models and for low-to-high resource training paradigms. By leveraging acquisition rewards, we seek to demonstrate a principled path toward model-aware self-improvement that surpasses static datasets.
Ishika Agarwal, Sofia Stoica, Emre Can Acikgoz +4
University of Illinois Urbana-Champaign1 · Amazon2
High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.
Ru Peng, Tianyu Zhao, Xijun Gu +9
Zhejiang University · Inclusion AI, Ant Group · Ant Group +2