$τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

Quan Shi; Alexandra Zytek; Pedram Razavi; Karthik Narasimhan; Victor Barres

cs.AIMar 4, 2026

$τ$ -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

Authors: Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres

Organizations: Sierra · Princeton University

Abstract

Conversational agents are increasingly deployed in knowledge-intensive settings, where correct behavior depends on retrieving and applying domain-specific knowledge from large, proprietary, and unstructured corpora during live interactions with users. Yet most existing benchmarks evaluate retrieval or tool use independently of each other, creating a gap in realistic, fully agentic evaluation over unstructured data in long-horizon interactions. We introduce $τ$ -Knowledge, an extension of $τ$ -Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs to produce verifiable, policy-compliant state changes. Our new domain, $τ$ -Banking, models realistic fintech customer support workflows in which agents must navigate roughly 700 interconnected knowledge documents while executing tool-mediated account updates. Across embedding-based retrieval and terminal-based search, even frontier models with high reasoning budgets achieve only $\sim$ 25.5% pass^1, with reliability degrading sharply over repeated trials. Agents struggle to retrieve the correct documents from densely interlinked knowledge bases and to reason accurately over complex internal policies. Overall, $τ$ -Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments.

Figures & tables

Figure 1: Overview of the $\tau$ -Banking domain. Agents must interact with a knowledge base to acquire procedural knowledge, policies, tools, and business offerings in order to resolve complex user requests by invoking discovered tools that modify underlying databases. The example on the right illustrates an agent assisting a user who has lost a wallet containing bank cards: although the user initially requests to freeze the card, card-specific policies and transaction-history constraints require the agent to instead cancel the card.

Statistic	$\tau$ -Banking
Knowledge Base
Total Tokens (cl100k_base)	194,562
# Documents	698
Avg. Tokens / Document	278.7
# Knowledge Categories	21
# Discoverable Tools	51

Table 1: Summary statistics of the new

\tau

-Banking domain.

Figure 2: Knowledge-base construction pipeline for $\tau$ -Banking. First, a large-language model (LLM) expands high-level product/category lists into a structured schema of offerings and typed attributes (e.g., fees, bonuses, limits). Then, the structured records are transformed into natural-language documents (e.g., FAQs and policy articles) that distribute the underlying variables across documents. Finally, during task creation, humans and an LLM review, edit, link, and de-duplicate content to produce the final corpus.

			Embeddings
Model	Reasoning	Gold	text-emb-3-large	Qwen3-emb-8b	BM25	Terminal
GPT-5.2	High	32.73	23.45 (-9.3)	24.74 (-8.0)	24.48 (-8.2)	25.52 (-7.2)
GPT-5.2	None	15.72	8.25 (-7.5)	12.37 (-3.4)	9.54 (-6.2)	11.60 (-4.1)
Claude-4.5-Opus	High	39.69	18.30 (-21.4)	19.59 (-20.1)	17.78 (-21.9)	24.74 (-14.9)
Claude-4.5-Sonnet	High	33.76	17.53 (-16.2)	17.78 (-16.0)	16.75 (-17.0)	22.42 (-11.3)
Gemini-3-Pro	High	33.25	12.89 (-20.4)	12.89 (-20.4)	13.66 (-19.6)	15.72 (-17.5)

Table 2: Main results on

\tau

-Knowledge (passˆ1, in %). Columns indicate retrieval configurations: text-embedding-3-large and Qwen3-emb-8b use dense retrieval with the respective embedding models; BM25 represents sparse lexical retrieval; Terminal provides shell access to the knowledge base as files; Gold places ground-truth documents directly in context, removing retrieval from the evaluation. Parenthetical values indicate the difference from the Gold setting for the row.

Figure 3: (Left) passˆ1–4 reliability for the best-performing configuration of each model declines sharply with increasing

k

, with substantial variation in reliability across systems (e.g., Gemini-3-flash vs GPT-5.2 (high)). (Right) Pareto frontier of average duration per task versus passˆ1 performance highlights substantial solution efficiency differences.

Figure 4: Representative agent failure modes in $\tau$ -Bench, grouped into four categories: (1) complex product interdependencies causing incorrect recommendations, (2) violations of required task ordering due to insufficient planning, (3) missing or unverified actions caused by over-trusting user statements, and (4) search inefficiency and unwarranted assumptions during retrieval-driven decision-making.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

		Impact of adding reranker
Retriever	Configuration	$\Delta$ pass ˆ 1	$\Delta$ Action Recall
text-embedding-3-large	Grep	$+0.013$ (n.s.)	$+0.026$ (n.s.)
text-embedding-3-large	No grep	$+0.016$ (n.s.)	$+0.015$ (n.s.)
Qwen3-embedding-8B	Grep	$-0.016$ (n.s.)	$-0.027$ (n.s.)
Qwen3-embedding-8B	No grep	$+0.011$ (n.s.)	$+0.055$ (sig.)
BM25	Grep	$+0.010$ (n.s.)	$+0.023$ (n.s.)

Appendix

Table 3: Effect of adding a reranker within each retriever family, stratified by grep setting.

		Impact of adding grep
Retriever	Configuration	$\Delta$ pass ˆ 1	$\Delta$ Action Recall
text-embedding-3-large	No reranker	$+0.054$ (n.s.)	$+0.035$ (n.s.)
text-embedding-3-large	Reranker	$+0.008$ (n.s.)	$+0.018$ (n.s.)
Qwen3-embedding-8b	No reranker	$+0.021$ (n.s.)	$+0.019$ (n.s.)
Qwen3-embedding-8b	Reranker	$+0.039$ (n.s.)	$+0.098$ (sig.)
BM25	No reranker	$0.000$ (n.s.)	$+0.029$ (n.s.)

Appendix

Table 4: Effect of adding grep within each retriever family, stratified by reranker setting.

Retriever	Configuration	$\Delta$ pass ˆ 1	$\Delta$ Action Recall
text-emb-3-large	k=10 vs k=5	-0.010 (n.s.)	+0.013 (n.s.)
text-emb-3-large	k=20 vs k=10	0.000 (n.s.)	+0.063 (n.s.)
Qwen3-emb-8b	k=10 vs k=5	-0.100 (n.s.)	+0.043 (n.s.)
Qwen3-emb-8b	k=20 vs k=10	-0.030 (n.s.)	+0.109 (n.s.)
BM25	k=10 vs k=5	+0.019 (n.s.)	+0.048 (n.s.)
BM25	k=20 vs k=10	+0.041 (n.s.)	+0.030 (n.s.)

Appendix

Table 5: Impact of increasing retrieval set size

k

(reranker + no-grep). Deltas computed as (right – left).

	No Knowledge		Long Context
Model	pass ˆ 1 (%)	pass ˆ 2 (%)	pass ˆ 1 (%)	pass ˆ 2 (%)
Claude-4.5-Opus (high)	2.32	1.55	–	–
GPT-5.2 (high)	1.03	0.20	11.86	2.23
Gemini-3-flash (high)	1.29	0.34	8.76	2.06
Gemini-3-pro (high)	0.77	0.17	11.08	2.58

Appendix

Table 6: Model performance in additional configurations. No knowledge means the agent does not have access to information in the unstructured knowledge base at all; Long Context means the entire knowledge base is put in the system prompt (not applicable to Opus due to an insufficient context window size).

	Gold	Terminal	Qwen3-emb-8b	BM25	text-emb-3-large
Gold	—	$<$ .001*	$<$ .001*	$<$ .001*	$<$ .001*
Terminal	$<$ .001	—	0.005*	0.013*	0.001*
Qwen3-emb-8b	$<$ .001	0.005	—	0.480	0.072
BM25	$<$ .001	0.013	0.480	—	0.128
text-emb-3-large	$<$ .001	$<$ 0.001	0.072	0.128	—

Appendix

Table 7: Pairwise significance for retrieval configurations (passˆ1, paired bootstrap). Each cell shows the p-value for whether the row outperforms the column. Asterisks (*) indicate statistical significance at

p<0.05

for the row outperforming the column

	Gold	Terminal	Qwen3-emb-8b	BM25	text-emb-3-large
Gold	—	$<$ .001*	$<$ .001*	$<$ .001*	$<$ .001*
Terminal	$<$ .001	—	0.007*	$<$ .001*	$<$ .001*
Qwen3-emb-8b	$<$ .001	0.001	—	0.474	0.002
BM25	$<$ .001	$<$ .001	0.474	—	0.014
text-emb-3-large	$<$ .001	$<$ .001	0.002	0.014	—

Appendix

Table 8: Pairwise significance for retrieval configurations (action recall, paired bootstrap). Each cell shows the p-value for whether the row outperforms the column. Asterisks (*) indicate statistical significance at

p<0.05

for the row outperforming the column

	GPT-5.2 (high)	Opus (high)	Gemini-Flash (high)	Sonnet (high)	Gemini-Pro (high)	GPT-5.2 (none)
GPT-5.2 (high)	—	0.072	0.004*	0.001*	$<$ .001*	$<$ .001*
Opus (high)	0.072	—	0.148	0.014*	$<$ .001*	$<$ .001*
Gemini-Flash (high)	0.004	0.148	—	0.274	$<$ .001*	$<$ .001*
Sonnet (high)	0.001	0.014	0.274	—	$<$ .001*	$<$ .001*
Gemini-Pro (high)	$<$ .001	$<$ .001	$<$ .001	$<$ .001	—	$<$ .001*
GPT-5.2 (none)	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	—

Appendix

Table 9: Pairwise significance for models (passˆ1, paired bootstrap). Each cell shows the p-value for whether the row outperforms the column. Asterisks (*) indicate statistical significance at

p<0.05

for the row outperforming the column

Model	Reasoning	Retrieval Config	pass ˆ 1	pass ˆ 2	pass ˆ 3	pass ˆ 4	Doc. Recall
Claude-4.5-Opus	High	Terminal	24.74	14.95	11.34	9.28	62.38
Claude-4.5-Opus	High	BM25	17.78	10.65	7.73	6.19	59.04
Claude-4.5-Opus	High	text-emb-3-large	18.30	10.82	7.47	6.19	57.07
Claude-4.5-Opus	High	Qwen3-emb-8b	19.59	12.03	9.54	8.25	58.28
Claude-4.5-Opus	High	Gold	39.69	31.62	28.35	26.80	N/A
Claude-4.5-Sonnet	High	Terminal	22.42	14.09	11.86	10.31	59.66

Appendix

Table 10: Task success results (passˆk and document recall) for all model and retrieval configuration combinations. Doc. recall is the document recall, computed as coverage of gold documents in the context window (see Appendix E ). Retrieval configurations: text-emb-3-large = dense retrieval with OpenAI text-embedding-3-large embeddings; Qwen3-emb-8b = dense retrieval with Qwen3-embedding-8b embeddings; BM25 = sparse lexical retrieval; Terminal = terminal use; Gold = required (ground-truth) documents in context. Bolded values represent 1) best configuration overall and 2) the best non-gold configuration for each column.

Model	Reasoning	Retrieval Config	$/Task	Dur. (s)	Med. Turn (s)
Claude-4.5-Opus	High	Terminal	4.36	177.1	21.1
Claude-4.5-Opus	High	BM25	4.20	160.3	19.3
Claude-4.5-Opus	High	text-emb-3-large	3.42	159.2	19.2
Claude-4.5-Opus	High	Qwen3-emb-8b	4.19	178.5	19.8
Claude-4.5-Opus	High	Gold	1.05	132.2	15.8
Claude-4.5-Sonnet	High	Terminal	4.05	223.7	24.9

Appendix

Table 11: Efficiency results for all model and retrieval configuration combinations. Cost is average dollars per task of all agent input/output tokens. Duration (Dur.) is the total wall-clock time from start of task to end. Median turn time (Med. Turn) is the median wall-clock time for a single agent turn (from user message to final agent response before the next user message). Bolded values represent 1) best configuration overall and 2) the best non-gold configuration for each column.

		Embeddings
Model	Gold	text-emb-3-large	Qwen-Emb	BM25	Terminal
GPT-4.1	20.6	10.1 (-10.5)	9.5 (-11.1)	9.3 (-11.3)	9.8 (-10.8)
GPT-4o	15.7	7.2 (-8.5)	8.2 (-7.5)	9.0 (-6.7)	5.2 (-10.5)

Appendix

Table 12: passˆ1 scores for older GPT models. Columns indicate retrieval configurations: text-embedding-3-large and Qwen-Emb use dense retrieval with the respective embedding models; BM25 represents sparse lexical retrieval; Terminal provides shell access to the knowledge base as files; Gold places ground-truth documents directly in context, removing retrieval from the evaluation. Parenthetical values indicate the difference from the Gold setting for the row.

Terminal Use (Shell Commands)
Model	grep	cat	sed	ls	Other
Claude-4.5-Opus (high)	10.85	9.74	0.00	0.34	0.20
Claude-4.5-Sonnet (high)	12.55	13.54	0.00	1.58	0.46
GPT-5.2 (high)	29.38	11.65	6.07	1.49	0.54
GPT-5.2 (none)	6.68	6.73	0.11	1.27	0.10
Gemini-3-Flash (high)	22.51	12.66	0.01	1.71	0.16

Appendix

Table 13: Average knowledge tool calls per task by condition and model. The top section shows the number of specific shell commands each agent made in the terminal use condition. The bottom section shows the number of KB_search tool calls made in each sparse/dense retrieval configuration.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

$τ$ -Rec: A Verifiable Benchmark for Agentic Recommender Systems
Jun 8, 2026Bharath Sivaram Narasimhan, Karthik R NarasimhanAI Agent Reliability AI Agent Benchmarks
Independent Researcher Mountain View, CA, USA · Princeton University Princeton, NJ, USA
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Sep 24, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Tool-Augmented Language Model Agents LLM Agent Evaluation
Eon
$OneMillion-Bench: How Far are Language Agents from Human Experts?
Mar 9, 2026Yang Liu, Jiaqi Li, Jun Bai +20LLM Agent Evaluation AI Agent Benchmarks
BIGAI · Humanlaya · xbench +1

Jun 8, 2026J/K move · Enter open · S save

$τ$ -Rec: A Verifiable Benchmark for Agentic Recommender Systems

Bharath Sivaram Narasimhan, Karthik R Narasimhan

Independent Researcher Mountain View, CA, USA · Princeton University Princeton, NJ, USA

AI Agent Reliability AI Agent Benchmarks Multi-Turn Dialogue Evaluation Recommender Systems LLM Agent Reliability

Open paper AskPDF arXiv

As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a-judge" evaluations, which introduce subjectivity, high costs and inconsistency. We present

τ

-Rec, a benchmark for agentic recommender systems that replaces subjective evaluation with verifiable rewards and a reveal-tagged elicitation (RTE) mechanism that controls how task constraints surface during dialogue. By testing agents against structured catalog predicates and employing a pass^k reliability metric,

τ

-Rec provides a systematic test for consistent reasoning. Our evaluation of nine configurations across five model families -- GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B and GPT-5 mini -- reveals a steep reliability cliff, where even the best model achieves only ~57% at pass^1 and ~35% at pass^4, highlighting a critical gap in current conversational agent deployment. All code and data are publicly available at https://github.com/nbharaths/tau-rec.

Abstract

LLM Agent Evaluation

Explore similar work

τττ-Rec: A Verifiable Benchmark for Agentic Recommender Systems

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

$OneMillion-Bench: How Far are Language Agents from Human Experts?

$τ$ -Rec: A Verifiable Benchmark for Agentic Recommender Systems