While conversational recommendation systems provide a more natural user interface, they face the challenge of progressively discovering a user's needs before they can recommend an item that satisfies them. To evaluate the ability of AI agents at this elicitation process, we introduce τ-Rec-Books, a benchmark that holds each user request fixed while varying only when the user reveals their requirements: all at the start (volunteer), one at the start and the rest when asked (mixed), or some never (hidden). A hidden requirement can be found only by proposing books and inferring it from the user's accept or reject verdicts. We generate 450 tasks over a catalog of 20,000 books from Wikidata, develop a rule-based simulated user to answer the agent, and score recommendations against catalog metadata without an LLM judge. Across four agents and two search interfaces, success rates fall for every agent and interface, on average from 0.975 (volunteer) to 0.756 (mixed) and 0.283 (hidden). Mixed tasks fail mainly because agents do not ask, and when told to ask, every agent reaches at least 0.931. On hidden tasks, most failures come from agents recommending a book they never proposed to the user. When told to propose first, agents only match a simple baseline that proposes popular books, and stay well below one that uses rejection to rule out possible requirements. We release the catalog pipeline, task generator, simulated user, validators, reference agents, and all agent traces at https://github.com/nbharaths/tau-rec/tree/main/data/tracks/books-wikidata-v2.
Figures & tables
Agent turn
Rule-based user’s reply
Field question
Its on-ask facts; “I’d rather not say (more) about” a field holding a hidden fact; “No preference on” any other field
“Anything else?”
One undisclosed on-ask fact; once none remains, “That’s everything I can share”
One tagged book
###ACCEPTED### if it meets every predicate, is unread, and exists in any format the user named; else ###REJECTED### , never with a reason
Several tags, or none
No verdict; “Please choose one catalog title …”, or, if it recognizes no question, “Please ask about anything you need …”
Prompt
Agent instruction
Elicitation
“Engage the user in natural conversation to understand their preferences …”
Table 1: The rule-based user’s replies and the agents’ system prompt (verbatim excerpts).
Block
Agents
Interfaces
Conditions
Families, trials/task
RQ
Panel, other
all four
both
all three
33, 4
1, 2, 4
Reliability block
all four
both
all three
18, 8
1, 2, 4
Scale study (5k, 10k, 20k)
open-weight
both
not hidden
135, 4
3, 4
LLM users
DeepSeek
structured
all three
18, 4 × 2 users
4
Ask prompt
all four
structured
all three
18, 8
1
LLM users
GPT-6.1 Sol
structured
all three
18, 4 × 2 users
4
Table 2: Experimental blocks; the panel is the reliability block plus Panel, other . Blocks use the rule-based user and the 20k catalog unless a row says otherwise; 32,976 trials in all.
Figure 2: (a, b) The panel’s 45 satisfiable families, agents pooled over interfaces (95% family-bootstrap intervals), and the scripted agents on structured search; (b) counts a hidden success only within r rejections. (c) Hidden satisfiable success on the reliability block by prompt (structured search; prompts in Table 1 ).
Satisfiable (45 families)
NVR (6)
All (153 tasks)
Mixed (51)
Agent
Interface
Volunteer
Mixed
Hidden
Hidden
pass^1
pass^4
Elic.
DeepSeek
keyword
0.989
0.872
0.297
0.000
0.691
0.609
0.576
structured
1.000
0.967
0.358
0.000
0.760
0.666
0.859
GLM
keyword
0.978
0.869
0.325
0.167
0.714
0.574
0.569
structured
1.000
0.947
0.367
0.042
0.756
0.627
0.641
Qwen
keyword
0.842
0.569
0.294
0.167
0.550
0.381
0.377
Table 3: pass^k on the panel at 20k. Elic. : share of mixed trials eliciting every askable fact. LLM average: mean of the eight LLM rows. Bold: best LLM agent per pass^k column.
Families
Templates
Contrast
Δpass^1
95% CI
p
p
Tested alone
Interface, model-averaged
+7.1
[+3.0,+11.4]
0.0005
0.15
Holm-corrected (11 tests)
Interface, DeepSeek
+6.9
[+2.5,+11.6]
0.0091
0.38
Interface, GLM
+4.2
[−0.7,+9.5]
0.288
0.69
Table 4: Paired pass^1 contrasts in points, with 95% family-bootstrap intervals. Interface rows are structured minus keyword at 20k; the interaction is that difference at 5k minus at 20k. p values come from the family bootstrap and an exact sign-flip test over templates (9 for the LLM-user rows, 13 otherwise); ≤0.0011 is the smallest Holm-adjusted value 10,000 resamples allow.
Figure 3: How failed trials end on the panel at 20k, as shares of each agent’s failures over both interfaces ( n at the bar end; All pools the four agents). Missed: keyword matching missed a field question that LLM labels found; untested: never proposed for a verdict.