AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation

Yunxiao Shi; Wujiang Xu; Tingwei Chen; Haoning Shang; Ling Yang; Yunfeng Wan; Zhuo Cao; Xing Zi

cs.AIMar 4, 2026

AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation

Authors: Yunxiao Shi, Wujiang Xu, Tingwei Chen, Haoning Shang, Ling Yang, Yunfeng Wan, Zhuo Cao, Xing Zi,

Organizations: University of Technology Sydney · Rutgers University · University of New South Wales · University of Sydney ·

Abstract

LLM agents are rapidly becoming the practical interface for task automation, yet the ecosystem lacks a principled way to choose among an exploding space of deployable configurations. Existing LLM leaderboards and tool/agent benchmarks evaluate components in isolation and remain fragmented across tasks, metrics, and candidate pools, leaving a critical research gap: there is little query-conditioned supervision for learning to recommend end-to-end agent configurations that couple a backbone model with a toolkit. We address this gap with AgentSelect, a benchmark that reframes agent selection as narrative query-to-agent recommendation over capability profiles and systematically converts heterogeneous evaluation artifacts into unified, positive-only interaction data. AgentSelectcomprises 111,179 queries, 107,721 deployable agents, and 251,103 interaction records aggregated from 40+ sources, spanning LLM-only, toolkit-only, and compositional agents. Our analyses reveal a regime shift from dense head reuse to long-tail, near one-off supervision, where popularity-based CF/GNN methods become fragile and content-aware capability matching is essential. We further show that Part~III synthesized compositional interactions are learnable, induce capability-sensitive behavior under controlled counterfactual edits, and improve coverage over realistic compositions; models trained on AgentSelect also transfer to a public agent marketplace (MuleRun), yielding consistent gains on an unseen catalog. Overall, AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation, which establishes a reproducible foundation to study and accelerate the emerging agent ecosystem.

Figures & tables

Figure 1: Overview of AgentSelect . We construct three benchmark parts—LLM-only (Part I), toolkit-only (Part II), and compositional agents (Part III)—and use the resulting interactions to train an agent recommender for natural-language queries. Arrows show the flow; icons indicate backbone LLMs, tools, and composed agents.

Field	Value
Backbone LLM $M$	Name: Qwen2.5 Instruct-72B Description: Released in September 2024, Qwen2.5 pushes the context window to 128k and can generate passages up to 8k tokens. Relative to Qwen2 it shows …
Toolkit $T$	Name: family_relation_finder Description: A tool designed to find and analyze familial relationships… Name: genealogy_query Description: Focusing on identifying various familial connections of historical or contemporary figures. Name: extract_last_name Description: Extracting the last name from a full name string… Name: advanced_character_counter Description: Counting occurrences of specified characters in given strings…
Configuration $C$	Session History: add_history_to_messages: True read_chat_history: True read_tool_call_history: True Memory Management: enable_agentic_memory: True enable_user_memories: True enable_session_summaries: True Knowledge Integration: knowledge_base: Some Built Vector Databases

Table 1: Example of Agent Configuration.

Figure 2: Compositional Agent Construction and Query-Agent Interaction Simulation Pipeline for Part III.

Figure 3: Overview of Benchmark Characteristics and Distribution.

Method	Part I					Part II					Part III
Method	Prec.	Rec.	F1	nDCG	MRR	Prec.	Rec.	F1	nDCG	MRR	Prec.	Rec.	F1	nDCG	MRR
MF	$0.9200$	$0.9298$	$0.9237$	$0.9339$	$0.9631$	$0.0141$	$0.1406$	$0.0256$	$0.0660$	$0.0434$	$0.0000$	$0.0000$	$0.0000$	$0.0000$	$0.0000$
LightFM	$0.4679$	$0.4731$	$0.4698$	$0.5269$	$0.8010$	$0.0056$	$0.0563$	$0.0102$	$0.0365$	$0.0305$	$0.0032$	$0.0065$	$0.0043$	$0.0052$	$0.0099$
DNN (TFIDF)	$0.2971$	$0.3029$	$0.2993$	$0.3193$	$0.5743$	$0.0845$	$0.8453$	$0.1537$	$0.5425$	$0.4478$	$0.1774$	$0.3548$	$0.2365$	$0.2912$	$0.3075$
DNN (Bert)	$0.7257$	$0.7345$	$0.7290$	$0.7469$	$0.8787$	$0.0854$	$0.8536$	$0.1552$	$0.6682$	$0.6094$	$0.2429$	$0.4858$	$0.3239$	$0.3488$	$0.3037$
DNN (Bert ∗ )	$0.7336$	$0.7424$	$0.7369$	$0.7550$	$0.8889$	$0.0926$	$0.9261$	$0.1684$	$0.6835$	$0.6054$	$0.3209$	$0.6418$	$0.4279$	$0.4789$	$0.4106$

Table 2: Leaderboard results for query-to-agent recommendation on Parts I–III. Language embedding models marked with ∗ are trained with in-domain supervision (fine-tuned). All reported metrics are calculated based on the top-10 recommendations.

Figure 4: Long-tailed agent popularity in our benchmark. Left: Pareto curve over all parts. Right: popularity-by-rank curves, with Part I/II/III overlaid. We report two definitions: Unnormalized popularity counts reuse by agent IDs, while Normalized popularity is derived from content components (LLM and tools) and scaled to $[0,1]$ , mitigating near-unique agent IDs under one-off supervision.

	Features				Validation (Part III / Overall)
Setting	LLM content	Tool content	LLM ID	Tool ID	P	R	F1	Hit	nDCG	MRR
All modalities (content + IDs)	✓	✓	✓	✓	0.4753	0.9506	0.6337	0.9830	0.9344	0.9453
No LLM ID	✓	✓	✗	✓	0.4728	0.9456	0.6304	0.9811	0.9284	0.9393
No Tool ID	✓	✓	✓	✗	0.4743	0.9485	0.6324	0.9850	0.9287	0.9447
No IDs (LLM ID + Tool ID removed)	✓	✓	✗	✗	0.4722	0.9444	0.6296	0.9826	0.9260	0.9436
IDs only (no content)	✗	✗	✓	✓	0.4659	0.9317	0.6211	0.9749	0.9127	0.9277

Table 3: Modality attribution via feature ablations (Top@10). Query text is always included; we ablate agent-side features. “LLM/Tool content” refers to the textual descriptions used to represent the backbone model and toolkit. “LLM/Tool ID” are discrete identifiers. All results are obtained with the TwoTower TF-IDF model trained on Part III and evaluated on the held-out Part III split.

Intervention ( $A_{\text{cf}}$ )	$\mathbb{E}[\Delta s]\uparrow$	$\mathbb{E}[\Delta r]\uparrow$	Consistency $\uparrow$
Remove key tool	$0.0538\pm 0.0335^{*}$	$1.76\pm 0.84^{*}$	$64.0\pm 16.7\%$
Remove secondary tool	$0.0602\pm 0.0186^{**}$	$1.72\pm 0.77^{*}$	$64.0\pm 16.7\%$
Add irrelevant tool	$0.0825\pm 0.0203^{**}$	$2.48\pm 0.67^{**}$	$84.0\pm 16.7\%^{*}$
Add redundant tool	$-0.0287\pm 0.0387$	$-0.08\pm 0.92$	$44.0\pm 26.1\%$
Swap backbone (rank 2–5)	$0.0235\pm 0.0043^{**}$	$1.28\pm 0.58^{*}$	$92.0\pm 11.0\%^{**}$
Mean $\pm$ std over 5 runs. ${}^{}/^{*}$ denote Holm-corrected $p<0.05/0.01$ .

Table 4: Counterfactual capability sensitivity on

N{=}100

queries. Starting from

A_{\text{full}}

, we apply a single controlled intervention to form

A_{\text{cf}}

. A capability-sensitive ranker should show positive score/rank drops and high consistency. We aggregate three metrics: score drop

\Delta s

, rank degradation

\Delta r

, and consistency

\mathbb{I}[s(Q,A_{\text{full}})>s(Q,A_{\text{cf}})]

Training Data	Eval on Part I		Eval on Part II		Eval on Part III
Training Data	nDCG@10	$\Delta$ %	nDCG@10	$\Delta$ %	nDCG@10	$\Delta$ %
Train I+II	0.9457	–	0.9562	–	0.4907	–
Train I+II+III	0.7111	-23.46	0.9632	+00.73	0.8341	+69.98
Train III	0.5297	-25.51	0.6217	-35.45	0.9410	+12.82

Table 5: Learnability attribution of Part III with TwoTower (TF-IDF). We train on Parts I+II, Parts I+II+III, and Part III, and evaluate on the held-out splits of Parts I/II/III.

\Delta

(%) denotes the relative change (each row is compared against the row above).

Cutoff	Method	P	R	F1	nDCG	MRR
@1	EasyRec	$0.3556$	$0.3556$	$0.3556$	$0.3556$	$0.3556$
@1	EasyRec	0.4000	0.4000	0.4000	0.4000	0.4000
@5	EasyRec	$0.1304$	$0.6519$	$0.2173$	$0.5109$	$0.4642$
@5	EasyRec ∗	0.1360	0.6800	0.2267	0.5465	0.5023
@10	EasyRec	$0.0769$	$0.7685$	$0.1397$	$0.5491$	$0.4802$
@10	EasyRec ∗	0.0800	0.8000	0.1455	0.5870	0.5200

Table 6: Real-world transfer on MuleRun: EasyRec ∗ (tuned on AgentSelect ) vs. original EasyRec across different cutoffs.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Rec vs E2E-Label	Random vs E2E-Label	Gain over Random
Top-1 match $\uparrow$	0.364 [0.236, 0.491]	0.236 [0.127, 0.345]	+53.9%
Spearman $\rho$ $\uparrow$	0.225 [0.111, 0.336]	$-$ 0.011 [ $-$ 0.140, 0.109]	+0.236
Kendall $\tau$ $\uparrow$	0.185 [0.095, 0.276]	$-$ 0.007 [ $-$ 0.109, 0.087]	+0.193
nDCG@1 $\uparrow$	0.695 [0.632, 0.759]	0.608 [0.548, 0.672]	+14.2%
nDCG@3 $\uparrow$	0.819 [0.787, 0.850]	0.765 [0.731, 0.798]	+7.1%
nDCG@5 $\uparrow$	0.907 [0.890, 0.923]	0.880 [0.864, 0.897]	+3.0%

Appendix

Table 7: End-to-end rank alignment on deployed agents (Agno + MirrorAPI ). We compare the recommender’s predicted ordering against the end-to-end judged ordering over executed top-5 candidates, and include a random-permutation baseline over the same set. Values are mean with bootstrap 95% CI; we retain only queries where all five candidates yield valid, comparable outcomes. “Gain over Random” is reported as relative improvement for Top-1 and nDCG (%):

100\times(\textsc{Rec}-\textsc{Rand})/\textsc{Rand}

; for correlation metrics we report absolute differences since the random baseline is near zero and percentage gains are numerically unstable.

Figure 5: Online deployment of the recommended agent configuration, including the final runtime policy $C$ finalize and glue code for Agno integration.

Figure 6: Coverage-Balanced Prototype-Based Data Selection Distribution Selected VS Total Samples.

Agent Benchmarks

Agent benchmarks measure how well AI systems plan, use tools and memory, and complete tasks in interactive environments.

See all 116 papers

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
The Amazing Agent Race: Strong Tool Users, Weak Navigators
Apr 11, 2026Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2LLM Agent Evaluation AI Agent Benchmarks
University of Minnesota Twin Cities · Yonsei University · Grammarly
AgentSearchBench: A Benchmark for AI Agent Search in the Wild
Apr 24, 2026Bin Wu, Arastun Mammadli, Xiaoyu Zhang +1Agentic Search AI Agent Evaluation
Centre for Artificial Intelligence, University College London

Apr 12, 2026J/K move · Enter open · S save

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Wanyi Chen, Xiao Yang, Xu Yang, Tianming Sha+6

Soochow University · Microsoft Research Asia · Stony Brook University+2

Agentic RL AI Agent Benchmarks LLM Post-Training LLM Agent Evaluation

Open paper AskPDF arXiv

We introduce Agent2 RL-Bench, a compact diagnostic benchmark for evaluating agentic RL post-training, which tests whether LLM agents can autonomously design, implement, debug, and execute post-training pipelines that improve foundation models. RL post-training increasingly drives model alignment and specialization, yet existing benchmarks are largely static, rewarding supervised fine-tuning or script generation without assessing an agent's ability to close an interactive RL loop. Agent2 RL-Bench provides a unified agent-facing interface: each run starts from an isolated workspace containing a base model, task data, instructions, and a grading API, and agents must iterate within a fixed budget by training models and submitting artifacts for evaluation. The benchmark spans six tasks across three levels, from static rule-based training to judge-based optimization and closed-loop online RL with trajectory collection. Two diagnostic skills, namely runtime recording and post-hoc summarization, enable structured analysis of agent behavior, facilitating smooth and effective iteration of the benchmark's evaluation framework. Across five agent systems and six driver LLMs, agents show intelligent behavior but clear limitations: one RL-oriented run improves ALFWorld from 4.85 to 93.28 via SFT warm-up and GRPO with online rollouts, yet DeepSearchQA remains difficult, most successful routes rely on supervised pipelines, and interactive outcomes show large single-run differences across agent stacks. Overall, Agent2 RL-Bench shows that current agents can sometimes engineer online RL, but stable agent-driven RL post-training remains rare under fixed budgets. It also demonstrates that our benchmark provides a strong and effective evaluation framework for future research in this direction. Code is available at https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/rl/autorl_bench/README.md

Abstract

Agent Benchmarks

Explore similar work

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

The Amazing Agent Race: Strong Tool Users, Weak Navigators

AgentSearchBench: A Benchmark for AI Agent Search in the Wild