Finding training data in million-scale machine learning repositories is a Web search problem where text relevance is not enough. A query couples a semantic training goal with constraints such as task, license, and training stage, and the metadata that would verify them are often missing. Text retrieval ranks dataset cards by similarity but cannot enforce hard constraints or follow the relations that reach a sparsely described dataset. Free-form tool-calling agents are more flexible, but the same model that picks a dataset also declares it a match, and nothing checks that the query's constraints are actually met. Neither approach establishes that the data it returns will improve model training. To address it, we propose AgenticDS, a contract-governed agentic framework in which every language model decision instantiates a typed object before deterministic code consumes it. The framework compiles a query into a typed search plan, executes it through admissible deterministic operators over hybrid retrieval and bounded graph expansion, and verifies the provenance-grounded evidence before accepting a ranking or emitting a typed diagnosis that permits only local repairs. On HuDS-Bench, a new benchmark of 200 queries with 24,908 graded judgments pooled over 15 systems, AgenticDS improves NDCG@10 by 25.7% over the strongest dense--sparse hybrid and by 18.9% over a commercial LLM agent. A controlled search-to-train study further shows that the datasets it retrieves give the best downstream accuracy in mathematics, code, and medicine. Code and data are publicly available at https://github.com/nju-websoft/AgenticDS.
Figures & tables
Figure 1. Overview of AgenticDS. A Planner constructs complementary query branches. An Engineer allocates bounded resources and deterministically executes typed retrieval, directional expansion, branch fusion, and Critic-pool construction. The Critic independently scores evidence and diagnoses local repairs.
Query type
Count
Primary grounding
Typical requirement
General
90
Metadata + task tags
Discover data satisfying task, language, modality, or scale constraints
Fine-tuning
40
Metadata + training stage
Find instruction, dialogue, preference, or domain-adaptation data
Paper reference
40
Paper/model relations
Locate data associated with a specific paper or training lineage
Entity neighbor
30
Dataset/collection relations
Find datasets reachable through multi-hop collection, model, or task paths
Table 1. Composition of HuDS-Bench.
NDCG
MAP
Method
@5
@10
@20
@5
@10
@20
Platform search
GoogleDS
.0301 †
.0217 †
.0165 †
.0262 †
.0146 †
.0091 †
HFFullText
.2786 †
.2438 †
.2100 †
.2989 †
.2301 †
.1661 †
Sparse
BM25
.4719 †
.4717 †
.4797 †
.5912 †
.5575 †
.5252 †
Table 2. Retrieval quality on HuDS-Bench. Bold marks the best system and underline the second best. † marks a cell where AgenticDS (Qwen3-Emb) is significantly higher than that system ( p<0.05 ).
Figure 2. Search-pipeline ablation. Each stem is a configuration’s distance from the full system (vertical line). Lollipop chart of ten ablated configurations, each a horizontal stem running back to a vertical reference line at the full system's 0.6148 NDCG@10, with single-module removals in blue and two-module removals in orange, ranging from -0.017 for sample evidence to -0.156 for removing both the adaptive budget and the Critic.
Math (EM)
Code (pass@1)
Medicine (acc.)
Acquisition method
GSM8K
MATH
AMC23
OCWCourses
Avg
HumanEval
BigCodeBench
Avg
MedQA
MMLU-Med
Avg
All
No fine-tuning (zero-shot)
54.54
42.91
27.50
16.42
35.34
55.08
29.04
42.06
51.40
54.84
53.12
43.51
Qwen3-Emb+BM25
79.78
52.73
34.17
17.16
45.96
66.06
32.37
49.21
49.36
58.82
54.09
49.75
monoT5 (OOD)+BM25
69.34
48.87
29.17
17.15
41.13
57.93
31.78
44.85
52.58
63.66
58.12
48.03
Claude Code
77.05
45.25
30.00
17.16
42.37
64.02
29.97
47.00
55.77
61.25
58.51
49.29
Codex
65.56
50.82
30.83
17.16
41.09
62.60
29.73
46.17
53.44
61.86
57.65
48.30
Table 3. Training utility on eight benchmarks from three domains. Each block averages the three base models. Avg is the within-domain mean and All the mean of the three domain averages. Bold marks the best system.
Search
Training
Sign
Component removed
Δ NDCG@10
mean
s.e.
agrees
Critic
− 0.149
− 1.85
1.36
yes
Planner
− 0.086
− 0.60
0.92
yes
KG expansion
− 0.084
+ 0.10
0.56
no
Feedback loop
− 0.036
—
—
—
Adaptive budget
− 0.031
+ 0.43
0.85
no
Table 4. Search-end and training-end ablation compared. Training entries are means over the nine (domain, base model) blocks, and s.e. is the standard error of that mean.
Figure 3. Cost over the 200 HuDS-Bench queries. (a) billed tokens by type, (b) cost in USD against mean latency, and (c) AgenticDS token and latency share by stage. Three panels. Left: horizontal stacked bars of billed tokens for AgenticDS (12.3M), Claude Code (111.4M, 87 percent cached) and Codex (280.9M, 92 percent cached). Middle: a scatter of cost in US dollars against mean seconds per query, with AgenticDS cheapest at 19.86 dollars and Codex both slowest and most expensive at 112.40 dollars. Right: two stacked strips over the same five pipeline stages, one for token share and one for latency share, showing the Critic at 83 percent of tokens but 28 percent of time, the two Planner calls at 13 percent of tokens but 50 percent of time, and the Engineer's deterministic execution at 0 percent of tokens and 7 percent of time.
Failure code/severity; affected intents; evidence and confidence; permitted local plan repair within protocol limits.
Appendix
Table 5. Typed objects exchanged between the three agents.
Query type
Pairs
Label 0
Label 1
Label 2
General
10,730
2,990
6,388
1,352
Fine-tuning
4,968
1,134
3,210
624
Paper reference
5,049
4,659
1
389
Entity neighbor
4,161
3,842
164
155
Total
24,908
12,625
9,763
2,520
Appendix
Table 6. Graded relevance judgments by query type.
Answer reachable from
Query type
Queries
Δ NDCG@10
Card text
General
90
+ 0.075
Fine-tuning
40
+ 0.105
Relations only
Paper reference
40
+ 0.206
Entity neighbor
30
+ 0.197
Appendix
Table 7. NDCG@10 gain over the Qwen3-Emb+BM25 hybrid by query type. All four survive Holm correction.
Backbone
NDCG@5
NDCG@10
MAP@10
GLM-5.2
0.6297
0.6148
0.6973
Qwen3.5-35B-A3B
0.5417
0.5244
0.5702
GLM-4-9B
0.4809
0.4706
0.5750
Appendix
Table 8. Sensitivity to the agent backbone.
Domain
Benchmark
Items
Metric
Math
GSM8K
1,319
exact match
MATH
5,000
exact match
AMC23
40
exact match
OCWCourses
272
exact match
Code
HumanEval
164
pass@1
BigCodeBench
1,140
pass@1
Appendix
Table 9. Evaluation split of every benchmark.
Figure 4. Component ablation under the search-to-train protocol, all base models and domains. Each block is the change in domain accuracy when a component is removed. The full system’s score is printed beneath each column. Heatmap of five ablation rows by nine base-model-and-domain columns, coloured on a blue-to-red diverging scale centred at zero, with the numeric delta printed in every block and a separated nine-block mean column at the right.
Domain
Model
Qwen3-Emb+BM25
monoT5 (OOD)
Claude Code
Codex
AgenticDS
Math
Qwen
49.63
46.40
39.60
45.84
56.90
Llama
29.34
21.79
29.20
19.45
32.64
Olmo
58.90
55.21
58.30
57.98
59.93
Code
Qwen
56.58
52.79
55.48
53.56
56.80
Llama
36.86
36.69
34.32
32.23
37.55
Olmo
54.20
45.09
51.19
52.71
54.50
Appendix
Table 10. Per-model domain scores. Bold: best per row.
Billed tokens (M)
Cost (USD)
System
s/query
in, unc.
in, cached
output
total
per query
AgenticDS (Qwen3-Emb)
150.5
10.6
—
1.7
19.86
0.099
AgenticDS (bge-m3)
212.7
13.1
—
2.1
24.35
0.122
Claude Code
139.3
13.1
97.2
1.0
48.78
0.244
Codex
868.5
20.0
258.1
2.8
112.40
0.562
Appendix
Table 11. End-to-end cost of the 200-query evaluation. A dash marks no cached-prompt reuse, so AgenticDS is billed entirely at the uncached rate and its cost is an upper bound.
Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts such as SWE-Smith, SERA, and Nemotron-Terminal typically target a single benchmark, leaving open the question of how to train models that generalize across diverse agentic tasks. The OpenThoughts-Agent (OT-Agent) project addresses this gap with a fully open data curation pipeline for training agentic models. We conduct more than 100 controlled ablation experiments to systematically investigate each stage of the pipeline, yielding insights on the importance of task sources and diversity. We then assemble a training set of 100K examples from our pipeline and fine-tune Qwen3-32B on this dataset, which yields an average accuracy of 44.8% across seven agentic benchmarks and a 3.9 percentage point improvement over the strongest existing open data agentic model (Nemotron-Terminal-32B, 40.9%). Moreover, our training data exhibits strong scaling properties, outperforming alternative open datasets at every training set size in compute-controlled comparisons. We publicly release our training sets, data pipeline, experimental data, and models at openthoughts.ai to support future open research on agentic model training.
Negin Raoof, Richard Zhuang, Marianna Nezhurina +47
Ad hoc dataset search requires matching underspecified natural-language queries against sparse, heterogeneous metadata records, a task where typical lexical or dense retrieval alone falls short. We reposition dataset search as a software-architecture problem and propose a bounded, auditable reference architecture for agentic hybrid retrieval that combines BM25 lexical search with dense-embedding retrieval via reciprocal rank fusion (RRF), orchestrated by a large language model (LLM) agent that repeatedly plans queries, evaluates the sufficiency of results, and reranks candidates. To reduce the vocabulary mismatch between user intent and provider-authored metadata, we introduce an offline metadata augmentation step in which an LLM generates pseudo-queries for each dataset record, augmenting both retrieval indexes before query time. Two architectural styles are examined: a single ReAct agent and a multi-agent horizontal architecture with Feedback Control. Their quality-attribute tradeoffs are analyzed with respect to modifiability, observability, performance, and governance. An evaluation framework comprising seven system variants is defined to isolate the contribution of each architectural decision. The architecture is presented as an extensible reference design for the software architecture community, incorporating explicit governance tactics to bound and audit nondeterministic LLM components.
Riccardo Terrenzi, Phongsakon Mark Konrad, Tim Lukas Adam +1
Centre for Industrial Software University of Southern Denmark Alsion 2, Sønderborg, 6400, Denmark
Search agents powered by large language models can autonomously decompose queries, retrieve information, and synthesize answers through multi-step reasoning. However, the rapid growth of training methods has outpaced controlled comparison: existing works differ in retrieval corpora, reward designs, and training protocols, making it unclear what actually drives improvements. We present a controlled empirical study that isolates three under-explored dimensions of search agent training. First, we identify a critical data-coverage issue in the widely used Wikipedia 2018 corpus and show that correcting it alone yields larger gains than the differences between training algorithms. Second, we systematically compare outcome-based and process-based reward methods across three base models, finding that the simplest outcome-based approach achieves competitive or superior performance in most settings, and that process-level credit assignment can over-correct agent behavior. Third, we analyze training data diversity, off-policy data utilization, and search budget scaling, distilling practical guidelines for training effective search agents. Our code is available at https://github.com/YiboZhao624/SearchAgentReview.
Yibo Zhao, Zichen Ding, Jiayi Wu +2
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory