cs.IROct 10, 2026
SaveSelf-improving data synthesis enables scalable adaptation of embedding models
Organizations: University of Science and Technology of China, Hefei, China. · State Key Laboratory of Cognitive Intelligence, Hefei, China.
Abstract
AI agents require specialised retrieval across domains and tasks, but adapting embedding models is constrained by costly manual training-data design. Here we introduce SYNTRA, a self-improving agentic framework that turns a few retrieval examples into reusable synthesis instructions. Given an unlabelled corpus and a few relevant query-document pairs, SYNTRA infers implicit information needs and relevance criteria, then uses feedback on trial outputs to refine instructions for query generation and relevance assessment. Experiments across 24 datasets spanning 15 domains demonstrate that the same synthesis workflow produces effective training data, benefiting even strong embedding models with and without existing labelled training sets. Self-improvement strengthens this supervision on all six evaluated tasks, yielding better retrieval than human calibration on four. The training value persists at larger scales: on MS MARCO, retrieval gains continue across a 32-fold range of synthetic training-data sizes. SYNTRA thus enables scalable embedding-model adaptation with minimal human intervention.
Figures & tables
Figure 1: Constructing task-specific training data for specialised retrieval. a , A coding agent searches a repository and retrieves the implementation that satisfies a stated constraint. A research agent retrieves supporting studies from a paper corpus. b , Documents differ across domains in terminology, structure and knowledge. Within a single domain, the user’s objective changes which documents are relevant. For the same query, similar-argument and counterargument retrieval rank the same two arguments in opposite orders. c , SYNTRA receives an unlabelled corpus and a small set of few-shot examples, each pairing a query with a positive document. Few-shot-guided self-improvement turns a shared synthesis workflow into a task-specific one. This workflow then constructs (query, positive, negative) training records at scale. SYNTRA is run separately for each retrieval task using the same framework. The examples shown span software, scientific literature and case law. Within a domain, the same two arguments exchange positive and hard-negative roles under different retrieval needs. Pos., positive document; Hard neg., hard-negative document. All examples are schematic and abbreviated, not generated samples or measured retrieval results.
Figure 2: Task preparation and adaptation in SYNTRA. a , SYNTRA receives an unlabelled corpus and a small set of few-shot examples pairing queries with positive documents. Corpus preprocessing builds a full-corpus index and selects diverse source documents. The task definition is inferred from the few-shot examples; optional user review is recommended. Point groups schematically illustrate semantic diversity, not clustering outputs. b , Small-sample generate–evaluate–refine cycles establish reusable task-adapted instructions before production. Each test loop ends when verification passes or its iteration limit is reached. The shared instructions then remain fixed throughout production.
Figure 3: Production-scale synthesis in SYNTRA. Task-adapted instructions established in test mode remain fixed during production. a , For each selected document, an LLM jointly extracts semantic identifiers and constructs query attributes with dimensions and options. b , Query synthesis forms plans that pair one semantic identifier with one option from each query-attribute dimension. It then writes one query per plan using the same task-adapted instruction for both steps. c , Relevance-label synthesis retrieves candidates from the full corpus and labels them with the task-adapted relevance-label instruction. The labelled candidates are assembled into (query, positive, negative) training records. Document content, plans, queries, candidates and labels are illustrative.
Figure 4: Cross-task applicability, self-improvement and quality of SYNTRA-generated supervision. a , Coverage of 24 datasets spanning 15 domains, with relative retrieval gains for the 23 task-specific comparisons against the MS MARCO-trained baseline. Numeric labels report percentage gains; radial extensions use a nonlinear scale. The CodeTrans-DL gain averages three reported variants. MS MARCO is included as the common training corpus without a task-adaptation gain. b , Retrieval performance of the MS MARCO-trained baseline and models trained on data generated without self-improvement, with automatic self-improvement or with human calibration. The six tasks were TheoremQA (theorems), ArguAna, NFCorpus, FiQA-2018 (FiQA), SCIDOCS and Text2SQL. c , Mean nDCG@10 on BEIR and NanoBEIR for models trained with original or SYNTRA-generated queries under matched conditions. d , Agreement between Qwen3-30B-A3B relevance labels and reference labels derived from votes by seven human annotators. Binary and three-class agreement were evaluated on 150 query–document pairs, with 50 pairs from each of three tasks. Binary evaluation distinguishes relevant from non-relevant documents; three-class evaluation distinguishes positive, hard-negative and easy-negative labels. Training-label precision reports exact-category matches among the 87 evaluation pairs labelled positive or hard negative by the LLM. Error bars indicate 95% confidence intervals from 10,000 bootstrap resamples stratified by dataset. The 95% confidence intervals were 61.3–76.0% for three-class agreement, 86.7–95.3% for binary agreement and 74.7–90.2% for training-label precision. Four pairs with tied highest three-class vote counts were assigned hard-negative reference labels.
Figure 5: Few-shot-guided self-improvement on FiQA. a , Refinement of query-generation instructions. The source excerpt and queries are verbatim production examples generated in separate runs with instructions corresponding to the first and third refinement rounds. Attribute plans were not held fixed between these runs. Feedback summarizes the original refinement history. b , Refinement of relevance-annotation instructions using a known-positive few-shot example. Feedback clarifies that explaining why a plan may fail can help answer the user’s question. Text in this panel provides an illustrative summary of the recorded calibration episode. Bottom, FiQA retrieval performance without and with self-improvement, reflecting joint refinement of query-generation and relevance-annotation instructions.
Figure 6: Model adaptation and ablations of SYNTRA. a,b , BGE-M3 ( a ) and Qwen3-Embedding-8B ( b ) results with and without existing task-specific labelled training data. Ori denotes models fine-tuned on existing task-specific labelled data. Dashed lines separate the two settings; right-hand values report gains over Ori or the corresponding pretrained baseline. c , NanoBEIR ablations of the core synthesis components. d , Comparison of the shared MS MARCO baseline, a task-specific reference and SYNTRA. Reference training data were constructed using ReasonEmbed queries for the two TheoremQA tasks and official queries for CosQA and Synthetic Text2SQL. e , MS MARCO nDCG@10 as SYNTRA-generated training data increase from 10,000 to 320,000 examples. f , Retrieval performance as the number of queries generated per document increases on CodeTrans-DL and MedicalQA. Dashed lines show the MS MARCO-trained reference scores.
| Task | Training records | Production time (h) |
|---|---|---|
| MS MARCO (original queries) | 9,546 | 4.444 |
| MS MARCO (SYNTRA queries) | 9,486 | 5.789 |
| LeCaRDv2 Li et al. (2024) | 8,278 | 63.786 |
| FiQA | 9,360 | 6.489 |
| SCIDOCS | 9,742 | 7.197 |
| SciRepEval Search | 9,302 | 7.040 |
Extended Data Table 1: Production-stage runtimes and finalised training-data sizes across 24 retrieval tasks. Production timings are reported for one batch per listed task and query-source condition; when multiple batches were considered, the shorter-running batch was retained. Training records are valid entries in a finalised JSONL (JSON Lines) training-data file. Each batch has two equally sized training-file variants. Counts are reported per variant, without summing the variants or multiplying by training epochs. Timings include identifier extraction and attribute construction, query generation, candidate retrieval and final relevance annotation. Calibration, prebuilt indexing, failed attempts, waiting between resumed runs, final training-format conversion, and model training and evaluation are excluded. For resumed runs, elapsed times for the relevant completed stages are summed. For MS MARCO original queries, the reported runtime covers retrieval and annotation only. The measurements describe selected batches rather than task-average runtimes or total experimental costs. Data production used a system equipped with six NVIDIA GeForce RTX 5090 GPUs. Four hosted the generative model, and the other two were used only during embedding-model operations. Times are elapsed production-stage times, rather than GPU-hours or sums of per-GPU runtimes.
| Scope | Label space | Matches | Agreement | Macro-P | Macro-R | Macro-F1 | ||
|---|---|---|---|---|---|---|---|---|
| All samples | Three class | 150 | 103 | 0.687 | 0.510 | 0.713 | 0.749 | 0.699 |
| MS MARCO | Three class | 50 | 37 | 0.740 | 0.600 | 0.756 | 0.764 | 0.746 |
| Text2SQL | Three class | 50 | 32 | 0.640 | 0.444 | 0.742 | 0.824 | 0.704 |
| TheoremQA (theorems) | Three class | 50 | 34 | 0.680 | 0.484 | 0.670 | 0.725 | 0.675 |
| All samples | Binary | 150 | 137 | 0.913 | 0.719 | 0.875 | 0.846 | 0.859 |
| MS MARCO | Binary | 50 | 44 | 0.880 | 0.705 | 0.893 | 0.829 | 0.851 |
Extended Data Table 2: Agreement between LLM-generated relevance labels and human reference labels overall and by dataset. Human reference labels were derived from seven annotators as described in Methods. Agreement is the fraction of exact matches, and denotes Cohen’s . Three-class evaluation distinguishes positives, hard negatives and easy negatives; binary evaluation merges the two negative classes. Macro-P, macro-R and macro-F1 denote class-averaged precision, recall and F1, respectively.
| a. Three-class confusion matrix | |||
|---|---|---|---|
| Human reference | LLM positive | LLM hard negative | LLM easy negative |
| Positive | 22 | 8 | 0 |
| Hard negative | 5 | 50 | 32 |
| Easy negative | 0 | 2 | 31 |
Extended Data Table 3: Confusion matrix and class-wise performance of LLM-generated relevance labels. Human reference labels were derived from seven annotators as described in Methods. a , Three-class confusion matrix. Rows denote human reference categories and columns denote LLM-assigned categories. b , Class-wise performance. Human and LLM give the number of evaluation pairs assigned to each category. Among the 87 evaluation pairs labelled positive or hard negative by the LLM, 72 matched the human reference category, yielding a pooled precision of 82.8%. The pooled precision combines the counts from both categories rather than averaging their precisions. These are the categories retained in the hard-negative-only training export.
| a. Query and training-data sizes | ||||
|---|---|---|---|---|
| Task | Query source | Query entries | Records per variant | Combined input rows |
| TheoremQA (questions) | ReasonEmbed | 7,184 | 5,873 | 11,746 |
| SYNTRA | 9,852 | 9,006 | 18,012 | |
| TheoremQA (theorems) | ReasonEmbed | 8,861 | 6,947 | 13,894 |
| SYNTRA | 9,880 | 9,517 | 19,034 | |
| CosQA | Official | 9,020 | 6,599 | 13,198 |
Extended Data Table 4: Data sizes and retrieval performance for the task-specific query-source comparisons. a , Nonempty query entries submitted for relevance annotation, valid training records per positive-selection variant, and combined input rows supplied to training. Query entries include repeated query strings. Each valid record contains at least one positive and one hard negative. The two equally sized variants differ in positive-document selection as described in Methods and are concatenated without deduplication; combined input rows therefore equal twice the per-variant count. Input rows count supplied records before batching, rather than distinct queries or examples consumed over training epochs. b , nDCG@10 on a 0–100 scale, corresponding to Fig. 6 d. The MS MARCO-trained model provides the shared baseline. Reference queries are from ReasonEmbed for the two TheoremQA tasks and official training pairs for CosQA and Synthetic Text2SQL. All task-specific query sources undergo SYNTRA relevance annotation and the shared training procedure.
| Model family | Model | BEIR | NanoBEIR | BEIR rank | NanoBEIR rank |
|---|---|---|---|---|---|
| Encoder-only | snowflake-arctic-embed-l | 55.980 | 65.360 | 1 | 2 |
| snowflake-arctic-embed-m | 54.900 | 64.420 | 2 | 3 | |
| snowflake-arctic-embed-xs | 50.150 | 61.170 | 8 | 7 | |
| bge-large-en-v1.5 | 54.340 | 65.910 | 3 | 1 | |
| bge-base-en-v1.5 | 53.230 | 63.760 | 4 | 4 | |
| bge-small-en-v1.5 | 51.660 | 62.650 | 5 | 5 |
Extended Data Table 5: Paired BEIR and NanoBEIR results for the 18 models used in the consistency analysis. Values are suite-level mean nDCG@10 scores on a 0–100 scale. Ranks are computed separately within each model family; rank 1 denotes the highest score.
| Model family | Spearman’s | value | Within one rank | |
|---|---|---|---|---|
| Encoder-only | 9 | 0.933 | 8/9 | |
| Decoder-only | 9 | 0.817 | 6/9 |
Extended Data Table 6: Rank consistency between NanoBEIR and BEIR within embedding-model families. Spearman’s was computed from paired suite-level mean nDCG@10 scores. Ranks were assigned separately within each family, with rank 1 denoting the highest score. values are two-sided.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Setting |
|---|---|
| Initialization | Qwen3-0.6B-auto-eos |
| Pooling | Last token |
| Embedding normalization | L2 normalization |
| LoRA configuration | Rank 32, alpha 64, dropout 0.1 |
| Learning rate | |
| Global query batch size | 96 |
Supplementary Table 1: Shared training settings for the self-improvement comparison, component ablations and MS MARCO query-source comparison.
| Parameter | BGE-M3 | Qwen3-Embedding-8B | Qwen3-8B-auto-eos (scaling) |
|---|---|---|---|
| Learning rate | |||
| Training epochs | 1; 5 for SpartQA | 1 | 1 |
| Query batch size per GPU | 2; 16 for SpartQA | 16 | 16 |
| Pooling | CLS token | Last token | Last token |
| Embedding normalization | L2 | L2 | L2 |
| Maximum query / passage length (tokens) | 512 / 512 | 512 / 512 | 512 / 512 |
Supplementary Table 2: Training settings for adaptation of strong embedding models and the MS MARCO data-scaling experiment. The strong-model settings apply to LeCaRDv2, EvidenceBench, SciRepEval Search, TheoremQA questions, SpartQA and WinoGrande.
More in this topic
Synthetic Data Generation
Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.
See all 625 papers