Agentic AI envisions ecosystems of intelligent agents collaboratively solving complex tasks with minimal human intervention. In such ecosystems, each agent possesses specialized expertise, making effective expert selection central to overall system performance. While most current approaches assume a small number of well-documented models, real-world expertise is far more diverse and cannot be adequately captured through static metadata or hand-written descriptions. We anticipate a future with millions of specialized language models (LLMs), each excelling in different domains or problem types. Rather than relying on predefined capability statements, we propose a retrieval-based paradigm in which an assistant agent infers expertise dynamically by examining models' observable behavior. Upon receiving a user query, the assistant ranks candidate LLMs based on demonstrated competence, enabling efficient and adaptive expert selection. The TREC Million LLM Track operationalizes this paradigm by shifting the retrieval target from documents to expert LLMs. Participants are given a discovery set consisting of queries, answers, and log-probabilities from more than one thousand LLMs and are challenged to infer meaningful expertise representations for each model. Given an unseen test query, systems must then rank the LLMs according to their expected performance, providing the first large-scale benchmark for expertise retrieval in agentic AI.
Figures & tables
Figure 1: Using RAG-based LLMs to simulate expert models. Each expert consists of a shared LLM and retriever, but is restricted to a distinct document collection—typically a structured KB—that defines its domain expertise.
Figure 2: Distribution of cluster sizes across all document clusters. The y-axis is shown on a logarithmic scale.
Figure 3: Distribution of relevant documents across clusters. The y-axis is shown on a logarithmic scale.
System
NDCG@10
External Models
Devset Training
submission_4
0.17447
no
no
infolab_UD_run2
0.16913
no
no
r-ef-rm3-1000-ss
0.16801
no
no
r-f-rm3-1000-rr-ss
0.16081
no
no
r-f-rm3-1000-ss
0.15446
no
no
infolab_UD_run4
0.15341
no
no
Table 1: NDCG@10 scores with external model usage and devset training information.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.