While conversational recommendation systems provide a more natural user interface, they face the challenge of progressively discovering a user's needs before they can recommend an item that satisfies them. To evaluate the ability of AI agents at this elicitation process, we introduce τ-Rec-Books, a benchmark that holds each user request fixed while varying only when the user reveals their requirements: all at the start (volunteer), one at the start and the rest when asked (mixed), or some never (hidden). A hidden requirement can be found only by proposing books and inferring it from the user's accept or reject verdicts. We generate 450 tasks over a catalog of 20,000 books from Wikidata, develop a rule-based simulated user to answer the agent, and score recommendations against catalog metadata without an LLM judge. Across four agents and two search interfaces, success rates fall for every agent and interface, on average from 0.975 (volunteer) to 0.756 (mixed) and 0.283 (hidden). Mixed tasks fail mainly because agents do not ask, and when told to ask, every agent reaches at least 0.931. On hidden tasks, most failures come from agents recommending a book they never proposed to the user. When told to propose first, agents only match a simple baseline that proposes popular books, and stay well below one that uses rejection to rule out possible requirements. We release the catalog pipeline, task generator, simulated user, validators, reference agents, and all agent traces at https://github.com/nbharaths/tau-rec/tree/main/data/tracks/books-wikidata-v2.
Figures & tables
Agent turn
Rule-based user’s reply
Field question
Its on-ask facts; “I’d rather not say (more) about” a field holding a hidden fact; “No preference on” any other field
“Anything else?”
One undisclosed on-ask fact; once none remains, “That’s everything I can share”
One tagged book
###ACCEPTED### if it meets every predicate, is unread, and exists in any format the user named; else ###REJECTED### , never with a reason
Several tags, or none
No verdict; “Please choose one catalog title …”, or, if it recognizes no question, “Please ask about anything you need …”
Prompt
Agent instruction
Elicitation
“Engage the user in natural conversation to understand their preferences …”
Table 1: The rule-based user’s replies and the agents’ system prompt (verbatim excerpts).
Block
Agents
Interfaces
Conditions
Families, trials/task
RQ
Panel, other
all four
both
all three
33, 4
1, 2, 4
Reliability block
all four
both
all three
18, 8
1, 2, 4
Scale study (5k, 10k, 20k)
open-weight
both
not hidden
135, 4
3, 4
LLM users
DeepSeek
structured
all three
18, 4 × 2 users
4
Ask prompt
all four
structured
all three
18, 8
1
LLM users
GPT-6.1 Sol
structured
all three
18, 4 × 2 users
4
Table 2: Experimental blocks; the panel is the reliability block plus Panel, other . Blocks use the rule-based user and the 20k catalog unless a row says otherwise; 32,976 trials in all.
Figure 2: (a, b) The panel’s 45 satisfiable families, agents pooled over interfaces (95% family-bootstrap intervals), and the scripted agents on structured search; (b) counts a hidden success only within r rejections. (c) Hidden satisfiable success on the reliability block by prompt (structured search; prompts in Table 1 ).
Satisfiable (45 families)
NVR (6)
All (153 tasks)
Mixed (51)
Agent
Interface
Volunteer
Mixed
Hidden
Hidden
pass^1
pass^4
Elic.
DeepSeek
keyword
0.989
0.872
0.297
0.000
0.691
0.609
0.576
structured
1.000
0.967
0.358
0.000
0.760
0.666
0.859
GLM
keyword
0.978
0.869
0.325
0.167
0.714
0.574
0.569
structured
1.000
0.947
0.367
0.042
0.756
0.627
0.641
Qwen
keyword
0.842
0.569
0.294
0.167
0.550
0.381
0.377
Table 3: pass^k on the panel at 20k. Elic. : share of mixed trials eliciting every askable fact. LLM average: mean of the eight LLM rows. Bold: best LLM agent per pass^k column.
Families
Templates
Contrast
Δpass^1
95% CI
p
p
Tested alone
Interface, model-averaged
+7.1
[+3.0,+11.4]
0.0005
0.15
Holm-corrected (11 tests)
Interface, DeepSeek
+6.9
[+2.5,+11.6]
0.0091
0.38
Interface, GLM
+4.2
[−0.7,+9.5]
0.288
0.69
Table 4: Paired pass^1 contrasts in points, with 95% family-bootstrap intervals. Interface rows are structured minus keyword at 20k; the interaction is that difference at 5k minus at 20k. p values come from the family bootstrap and an exact sign-flip test over templates (9 for the LLM-user rows, 13 otherwise); ≤0.0011 is the smallest Holm-adjusted value 10,000 resamples allow.
Figure 3: How failed trials end on the panel at 20k, as shares of each agent’s failures over both interfaces ( n at the bar end; All pools the four agents). Missed: keyword matching missed a field question that LLM labels found; untested: never proposed for a verdict.
As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a-judge" evaluations, which introduce subjectivity, high costs and inconsistency. We present τ-Rec, a benchmark for agentic recommender systems that replaces subjective evaluation with verifiable rewards and a reveal-tagged elicitation (RTE) mechanism that controls how task constraints surface during dialogue. By testing agents against structured catalog predicates and employing a pass^k reliability metric, τ-Rec provides a systematic test for consistent reasoning. Our evaluation of nine configurations across five model families -- GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B and GPT-5 mini -- reveals a steep reliability cliff, where even the best model achieves only ~57% at pass^1 and ~35% at pass^4, highlighting a critical gap in current conversational agent deployment. All code and data are publicly available at https://github.com/nbharaths/tau-rec.
Bharath Sivaram Narasimhan, Karthik R Narasimhan
Independent Researcher Mountain View, CA, USA · Princeton University Princeton, NJ, USA
Agents typically assume an expert user -- one with well-formed preferences about what they want -- and default to clarifying questions whenever the task is underspecified. We argue this assumption is unrealistic. Users often lack the domain knowledge to have completely specified preferences; if asked about their preference on some feature, the user may be unable to answer without the agent helping the user to learn some domain knowledge needed to form a preference for that feature, e.g., via examples or explanations. To formalize these principles, we draw on the Search-Experience-Credence framework from Information Economics to introduce CoPref, a model of how users construct preferences based on agent dialog actions. We then study these ideas concretely in agentic recommender systems, proposing CoShop, an interactive benchmark. In CoShop, an agent converses with and makes recommendations for a CoPref user. The agent's performance depends on whether it can help the user gain the knowledge needed to specify the task well. Evaluating five frontier models, we find that no agent exceeds 56% accuracy on CoShop despite five turns of interaction. Failures stem not from agents' ability to find items, but from how little the interaction expands what users know about what they want.
LLM-based recommendation agents can search catalogs, inspect products, ask users, and revise rejected recommendations. However, these behaviors are often governed by hand-designed workflows, while existing training methods optimize isolated components such as query rewriting or list ranking rather than the interaction policy as a whole. We present SimRec, a reinforcement learning framework that trains the full interaction policy of a tool-integrated recommendation agent. SimRec introduces two components. A task-anchored user simulator provides feedback on every recommendation: conditioned on the current task need rather than on user profile, and it can be an external LLM or the recommender itself under self-play. A task-aligned reward optimizes the whole trajectory with GRPO, crediting natural retrieval quality and valid tool use alongside the final hit, with training-only target injection to handle sparse hits over large catalogs. On ESCI and Amazon C4, SimRec improves the NDCG@100 of a Qwen3-4B agent from 32.23% to 35.42% and from 22.94% to 24.72%}, outperforming training-free agents on far larger backbones and task-aligned baselines on NDCG. Analysis shows that the trained agent verifies candidates, uses preferences, and recovers from rejected recommendations far more often than its untrained counterpart. Our code is available at https://github.com/ZJU-REAL/SimRec .
Yaxin Zhang, Xingyu Wu, Yuchen Yan +6
Zhejiang University · Tsinghua University · Hikvision Research Institute