Organizations: The Chinese University of Hong Kong Hong Kong, China · Macao Polytechnic University Macao, China · The Hong Kong Polytechnic University Hong Kong, China
AI assistants are increasingly asked to reason over a user's own web traces: group chats, community threads, activity logs and revision histories. On CL-bench Life, a benchmark of such real-life context learning, frontier LLMs solve fewer than one task in five. We find that the request is not the specification: over 80% of the benchmark's rubrics share less than 60% of their content words with the final request, as for a rule set two turns earlier or every record that meets the user's criterion, and LLMs answering directly pass only 67% of these implicit rubrics against 81% of the others. We propose QERA, a Question-driven Evidence Reasoning Architecture that asks before it answers. QERA first turns the request into a requirement checklist and typed information needs, filtered by code for faithfulness to the user's words; it then answers from the whole conversation, asks the same questions again with quotes verified by code, and applies a guarded local repair. It needs no training or rubrics and makes about four LLM calls per task. On all 405 tasks with four frontier LLMs, QERA attains the highest solving rate on every LLM (21.0% on average, against 18.2% for the strongest of three inference-time baselines and 14.6% for direct answering) and leads every baseline significantly when pooled across LLMs, while using fewer tokens than PSCI and 4-5% of the tokens of Ctx2Skill. Ablations on GPT-6-Luna show that the gain is settled before the answer, by a fixed answer protocol (+3.8 rubric points) and the request-specific questions (+1.6), and that it is concentrated on implicit rubrics, with checking afterwards refining mainly numeric requirements.
Figures & tables
Figure 1. A CL-bench Life task (Game Logs). The user’s first turn rules out chess notation; the request two turns later does not repeat it. Answering directly, all four LLMs write chess notation. QERA’s checklist carries the rule forward, and QERA solves the task with Kimi K3 and Grok 4.7 (both replies shown are from Kimi K3). Top strip: answering directly sends the conversation through one LLM call to a reply that misses unstated requirements, with 81 percent pass rate on explicit and 67 percent on implicit rubrics. Bottom panel: a left-to-right pipeline from the conversation x through four numbered LLM stages, Ask, Answer, Check and Repair, with the artifacts Q, the draft, the audited evidence and the reply between them; dashed code boxes below each stage show the filters, the answer protocol, the measure-and-audit step and the guard; a blue line carries Q to Check and Repair.
Figure 2. Top: answering directly misses what a request leaves unsaid; the four LLMs of our study, answering directly, pass 81% of the rubrics whose wording the request largely contains but only 67% of the others. Bottom: QERA runs Eq. ( 2 ) from left to right. Ask writes the checklist O and the information needs N , which the filters Φ turn into the questions Q ; Answer drafts y~ under Q and the protocol π ; Check answers Q again with quotes, audited by code into E† ; Repair patches only what E† flags, and the guard Γ keeps the patch y^ or the draft. The blue line carries the same Q to Check and Repair . Two panels. Left: answering directly, a conversation goes through one LLM call to a reply that misses unstated requirements, with bars showing 81 percent pass rate on explicit and 67 percent on implicit rubrics. Right: QERA's four stages Ask, Answer, Check and Repair, each with an icon and its question; below them the checklist and needs, the code filters producing the questions Q, the draft, the evidence, the measure-and-audit step, the guard and the final reply; arrows show Q feeding the draft and, dashed, the check.
Question
Asks for
Check returns
Code verifies
requirement ok
what the reply must deliver
verdict and a quote of the draft
measured lengths and counts; required strings; exclusions
lookup (28%)
a specific fact
fact, quote, location
quote in x ; found values in y~
enumerate (47%)
every item meeting the user’s criterion
one finding per item
items of a need called correct that y~ never names
count (3%)
a number over items
the items and the count
count = number of items
compute (4%)
arithmetic on stated values
every input and the result
inputs sum or average to the result
compare (5%)
a comparison
facts on each side
quotes in x ; found values in y~
Table 1. The questions Q : what each asks for, what Check returns for it, and what code then verifies. Findings whose quote is not in x are flagged for the repair and ignored by the audit. Shares are of all needs over the four LLMs.
Communication
Behavioral
Fragmented
Overall
Model
Method
Solv.
Rub.
Solv.
Rub.
Solv.
Rub.
Solv.
Rub.
Tokens
Kimi K3
Vanilla
20.7
76.6
19.3
70.9
17.0
76.1
19.0
74.6
24k
CCA
24.4
78.5
17.0
72.1
20.7
78.9
20.7
76.7
72k
PSCI
28.1
80.7
23.7
77.5
13.3
78.3
21.7
78.9
148k
Ctx2Skill
26.7
79.4
20.7
74.4
20.0
76.8
22.5
76.9
2.54M
QERA
28.1
81.1
24.4
78.3
21.5
82.0
24.7
80.5
128k
Table 2. Results on all 405 CL-bench Life tasks (135 per category), graded by GPT-5.1: solving rate (Solv.) and rubric pass rate (Rub.) in %, for the three context categories (Communication & Social Interactions, Behavioral Records & Activity Trails, Fragmented Information & Revisions) and overall, and API tokens (input + output) per task. Within each block, bold is the best and underlined the second best value. ∗ On GPT-6-Luna, Ctx2Skill reuses the skills selected by its Grok 4.7 self-play (§ 4.1 ); its tokens are that self-play’s (3.10M Grok 4.7 tokens per task) plus GPT-6-Luna’s inference (28k).
QERA −
Δ Solving rate
Δ Rubric pass rate
Vanilla
+6.4 [+4.3, +8.5]
+5.6 [+4.5, +6.8]
CCA
+2.9 [+0.9, +4.9]
+2.1 [+1.1, +3.1]
PSCI
+2.8 [+0.7, +4.9]
+2.4 [+1.4, +3.3]
Ctx2Skill
+3.5 [+1.5, +5.6]
+3.1 [+2.2, +4.0]
Table 3. QERA minus each baseline, pooled over 405 tasks × 4 LLMs, in percentage points of unrounded rates, with paired 95% bootstrap intervals (tasks resampled with their four LLM results). Ctx2Skill includes Ctx2Skill ∗ on GPT-6-Luna. All eight differences remain significant after Holm correction (adjusted p≤0.014 ). Per-LLM tests: Table 6 .
Figure 3. Solving rate against API tokens per task (log scale) on each LLM; the dashed line is the accuracy–cost Pareto front. Ctx2Skill ∗ on GPT-6-Luna is drawn at its full cost: GPT-6-Luna inference plus the Grok 4.7 self-play whose skills it reuses. Four scatter plots, one per LLM, of solving rate versus tokens per task. QERA, marked by a star, is the highest point in every panel and lies on the Pareto front; Ctx2Skill is far to the right at about three million tokens.
Configuration
Solv.
Rub.
Impl.
Δ Rub. vs. row above
Tok.
Vanilla
12.1
67.1
64.1
–
24k
Before the answer
+ answer protocol
17.0
70.9
68.4
+3.8 [+2.0, +5.7]
27k
+ requirement checklist
16.8
72.0
70.0
+1.1 [ − 0.3, +2.6]
55k
+ information needs (draft)
18.8
72.5
70.5
+0.5 [ − 1.0, +2.0]
56k
After the answer
Table 4. Cumulative ablation on GPT-6-Luna (405 tasks). Impl.: pass rate on implicit rubrics. Δ Rub.: paired difference of unrounded rates to the row above ( † : to QERA), with 95% bootstrap interval. Tok.: API tokens per task of the calls the configuration makes. The fourth row is QERA’s draft (details in Appendix E ).
Figure 4. Rubric pass rate on GPT-6-Luna as QERA’s stages are added, with tokens per task. Blue steps act before the answer and the orange step after it; diamonds give the pass rate on implicit rubrics. A waterfall chart from Vanilla at 67.1 to QERA at 73.2. The answer protocol adds 3.8 points, the checklist 1.1, the information needs 0.5 and check and repair 0.7; the token cost rises from 24k to 123k, mostly in the last step.
Figure 5. (a) Rubric pass rate on explicit and implicit rubrics, pooled over the four LLMs (rubric–LLM pairs in parentheses), with QERA minus the best baseline. (b) Rubric pass rate by context-length quartile (characters). Left: grouped bars; on explicit rubrics all methods are near 82 percent, on implicit rubrics QERA reaches 74 percent, 2.5 points above the best baseline. Right: lines over four context-length quartiles; QERA is the top line in every quartile.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Solving rate (%)
Rubric pass rate (%)
Best Rub.
Subcategory
Vanilla
CCA
PSCI
Ctx2Skill
QERA
Vanilla
CCA
PSCI
Ctx2Skill
QERA
K / G / Z / L
Communication & Social Interactions
Community Interactions
12.2
15.6
18.9
17.8
17.2
71.5
74.3
75.6
72.9
74.6
P / C / Q / S
Group Conversations & Meeting Transcripts
19.4
23.3
21.7
23.9
27.2
76.5
78.7
75.4
78.9
81.2
Q / S / Q / Q
Private Conversations
12.2
21.7
20.0
16.7
21.7
67.7
70.7
69.0
71.0
74.8
Q / C / P / Q
Behavioral Records & Activity Trails
Appendix
Table 5. Results by subcategory (45 tasks each), averaged over the four LLMs. Bold: best; underlined: second best.
QERA −
Kimi K3
Grok 4.7
GLM-5
GPT-6-Luna
Pooled
Vanilla
Solv.
+5.7 [+1.5, +9.9]
+10.9 [+6.7, +15.3]
+4.4 [+1.2, +7.9]
+4.4 [+1.0, +7.9]
+6.4 [+4.3, +8.5]
Rub.
+5.9 [+4.0, +7.7]
+4.7 [+3.0, +6.5]
+5.8 [+3.8, +7.9]
+6.1 [+3.9, +8.3]
+5.6 [+4.5, +6.8]
CCA
Solv.
+4.0 [0.0, +7.9]
+2.5 [ − 1.7, +6.7]
+2.7 [ − 0.7, +6.2]
+2.5 [ − 1.5, +6.4]
+2.9 [+0.9, +4.9]
Rub.
+3.9 [+2.0, +5.7]
− 2.1 [ − 3.8, − 0.4]
+2.9 [+0.8, +4.9]
+3.6 [+1.7, +5.6]
+2.1 [+1.1, +3.1]
PSCI
Solv.
+3.0 [ − 1.2, +7.2]
+4.2 [+0.2, +8.1]
+0.2 [ − 3.2, +3.7]
+3.7 [0.0, +7.4]
+2.8 [+0.7, +4.9]
Rub.
+1.7 [ − 0.2, +3.4]
+0.9 [ − 0.9, +2.6]
+0.5 [ − 1.5, +2.5]
+6.4 [+4.3, +8.7]
+2.4 [+1.4, +3.3]
Appendix
Table 6. QERA minus each baseline per LLM and pooled, in percentage points of unrounded rates, with paired 95% bootstrap intervals over tasks; grey: the interval contains zero. Ctx2Skill on GPT-6-Luna is Ctx2Skill ∗ .
Kimi K3
Grok 4.7
GLM-5
GPT-6-Luna
Method
Tok.
Calls
Tok.
Calls
Tok.
Calls
Tok.
Calls
Vanilla
24k
1.0
28k
1.0
25k
1.0
24k
1.0
CCA
72k
4.4
169k
4.7
63k
3.1
70k
4.0
PSCI
148k
5.0
177k
5.0
139k
5.1
137k
5.0
Ctx2Skill
2.54M
163
3.14M
153
2.37M
158
3.13M ∗
153 ∗
QERA (ours)
128k
3.9
131k
3.8
112k
4.0
123k
4.0
Appendix
Table 7. API tokens (input + output, reasoning included) and LLM calls per task. ∗ GPT-6-Luna inference plus the reused Grok 4.7 skill discovery.
LLM
Draft S/R
Final S/R
Δ Solv.
Δ Rub.
Tokens
Repairs ‡
Kimi K3
25.2/80.2
24.7/80.5
− 0.5 [ − 3.0, +2.0]
+0.4 [ − 0.9, +1.4]
56k/128k
334/15/8/48
Grok 4.7
27.7/78.8
28.4/78.7
+0.7 [ − 1.7, +3.2]
0.0 [ − 0.9, +0.8]
64k/131k
307/0/0/98
GLM-5
14.6/69.7
14.3/69.8
− 0.2 [ − 2.7, +2.2]
+0.1 [ − 1.0, +1.2]
54k/112k
336/2/0/67
GPT-6-Luna
18.8/72.5
16.5/73.2
− 2.2 [ − 5.4, +0.7]
+0.7 [ − 0.6, +2.0]
56k/123k
381/2/0/19
Pooled
21.5/75.3
21.0/75.6
− 0.6 [ − 1.9, +0.7]
+0.3 [ − 0.3, +0.8]
57k/123k
1358/19/8/232
Appendix
Table 8. QERA’s draft (Ask + Answer) vs. final reply: differences of unrounded rates with paired 95% intervals; tokens per task (draft/all); ‡ repairs kept/guard-rejected/invalid/nothing to fix.
We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.
Neha Srikanth, Jordan Boyd-Graber, Rachel Rudinger
Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.
Large language models consistently fail the "car wash problem," a viral reasoning benchmark requiring implicit physical constraint inference. We present a variable isolation study (n=20 per condition, 6 conditions, 120 total trials) examining which prompt architecture layers in a production system enable correct reasoning. Using Claude 3.5 Sonnet with controlled hyperparameters (temperature 0.7, top_p 1.0), we find that the STAR (Situation-Task-Action-Result) reasoning framework alone raises accuracy from 0% to 85% (p=0.001, Fisher's exact test, odds ratio 13.22). Adding user profile context via vector database retrieval provides a further 10 percentage point gain, while RAG context contributes an additional 5 percentage points, achieving 100% accuracy in the full-stack condition. These results suggest that structured reasoning scaffolds -- specifically, forced goal articulation before inference -- matter substantially more than context injection for implicit constraint reasoning tasks.