When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that assesses model responses along three criteria grounded in cooperative response theory: (a) presence and (b) emphasis on pragmatic solutions, and (c) identification of misconceptions. Our experiments show that even the strongest LLMs predominantly answer the literal request (0.75--0.92) and far less often the intended one (0.33--0.71), while substantially lagging behind humans at identifying misconceptions (at most 63% vs. 79--90%). Further, models overwhelmingly prefer pragmatic responses in a multiple choice setting yet consistently fail to generate them. Oracle ablation experiments show that providing explicit user intent at generation time helps; however a large gap remains, suggesting pragmatic redirection is a fundamentally underdeveloped capability in current LLMs.
Figures & tables
Figure 1: Users often ask queries laden with misconceptions in both technical and non-technical domains. A user’s stated request of using regex to parse XML implicitly reflects an underlying goal of extracting structured data. A truly cooperative system must: (1) address the stated request by providing a working regex, if feasible, (2) identify the misconception that regex is unavoidably fragile for nested or complex XML, and (3) suggest an XML parser to better serve the user’s intended goal.
Figure 2: XYBench construction pipeline. Queries are curated from three sources: technical queries are drawn from real Stack Overflow and Stack Exchange threads tagged as XY problems; a manually-curated set of in-the-wild XY queries; and further everyday queries are synthetically constructed from WikiHow guides by injecting likely user mistakes into procedural steps. All pipelines extract or synthesize stated ( S- ) and intended ( I- ) requests and solutions, along with misconceptions arising from stated-intended requests ( S-I-misconception-identification ). Queries and associated metadata are rigorously filtered for quality and validated by human annotators to produce the final benchmark.
Source
Domain
# Queries
Stack Overflow
programming
1557
Stack Exchange
electronics
61
superuser
36
unix
36
askubuntu
19
+11 others
32
Table 1: XYBench statistics across data sources.
Solution
Emphasis
S-I-Miscon.
Fully
Fully
(at least once)
avg. orientation of responses
identif.
Prag.
Coop.
Source
Model
S-soln.
I-soln.
S-soln
Balanced
I-soln
Stack Overflow
Claude Sonnet 4.6
0.82
0.59
0.57
0.08
0.36
0.47
0.25
0.20
GPT-5
0.83
0.71
0.47
0.12
0.41
0.60
0.31
0.28
Gemini 3.1 Pro
0.79
0.62
0.51
0.08
0.40
0.54
0.29
0.22
Qwen3.5-9B
0.75
0.64
0.53
0.06
0.41
0.60
0.29
0.21
Table 2: Evaluation of model and human responses on XYBench . Models consistently address the stated request but rarely prioritize the implicit goal ( high S-soln, lower I-soln presence ). Models also consistently fall behind humans in identifying misconceptions and responding to what users actually need . Within each source block, color gradation is normalized, and the maximum value in each column is bolded.
Figure 3: Oracle upper-bound experiment where S-request and I-request are given at generation time along with the query. Solid bars are query-only; pale segments show the change under the oracle setting, with hatching marking declines. Human bars are query-only. Despite overall improvement in model performance, they still lag behind humans, indicating that intent inference and misconception detection are not the only bottlenecks.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Example query from Stack Overflow: Human Reference Answer, and Model Responses from Claude Sonnet 4.6 and Gemini 3.1 Pro are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution
Figure 5: Example query from Stack Exchange: Human Reference Answer, and Model Responses from Claude Sonnet 4.6 and GPT 4.1 are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution
Figure 6: Example query from Manually-Curated: Model Responses from Gemini 3.1 Pro and Qwen3.5-9B are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution
Figure 7: Example query from WikiHow: Model Responses from Claude Sonnet 4.6 and GPT 4.1 are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution
Figure 8: Prompt used to extract S-request and I-request from real conversation threads on SO and SE
Figure 9: Prompt used to extract S-solution , I-solution and S-I-misconception-identification from real conversation threads on SO and SE, given the query and previously extracted S-request and I-request
Figure 10: Prompt used to filter queries which are better suited with a S-solution , i.e., they are less misconceived. Two response choices are provided (order shuffled): one with S-solution and one with I-solution . Models must choose the response option that might be most helpful to the user.
Pipeline Accuracy
Strict
Lenient
MCQ filtering agreement
0.920
0.940
Appendix
Table 3: Human validation results for MCQ filtering over n=100 sampled instances (balanced retained/excluded). Annotators judge each filtering decision as agree , borderline , or disagree . Strict pipeline accuracy counts agree only; lenient counts borderline as correct.
Dimension
Exact
Kripp. α
Ann1
AnnAK
Strict
Lenient
S-request
1.000
1.000
0.950
0.950
0.950
0.990
I-request
0.970
0.687
0.950
0.950
0.940
1.000
S-solution
1.000
1.000
0.930
0.930
0.930
0.930
I-solution
0.970
0.715
0.950
0.940
0.930
0.960
S-I-misconception-identification
0.970
0.798
0.910
0.930
0.910
0.930
All fields correct
—
—
0.840
0.840
0.820
0.860
Appendix
Table 4: Human validation results on Stack Overflow data over n=100 sampled instances across five dimensions. Strict accuracy counts only correct / yes ; lenient counts partial as correct.
Figure 11: Prompt used to generate WikiHow data items: initial query, S- and I- request and solution, S-I-misconception-identification
Figure 12: Prompt used to check if I-request is identifiable in the user query by a domain expert, rewrite if needed
Figure 13: Prompt used to check query has obvious clues, rewrite if needed
Figure 14: Prompt used to filter and regenerate query data: Given a query, we filter and rewrite queries to again remove obvious clues, as well rewrite to add psychologically grounded explanation to motivate S-request
Figure 15: Prompt used to check S-I request gap quality
Dimension
Exact
Kripp. α
Ann1
AnnAK
Strict
Lenient
S-request
0.995
0.988
0.723
0.725
0.723
0.953
I-request
0.995
0.986
0.775
0.777
0.775
0.970
query.plausible
1.000
1.000
0.863
0.863
0.863
—
S-solution
0.995
0.987
0.747
0.747
0.744
0.749
S-I-misconception-identification
0.995
0.983
0.818
0.818
0.815
0.820
I-solution
0.995
0.980
0.858
0.853
0.853
0.858
Appendix
Table 5: Human validation results over n=400 sampled instances of WikiHow constructed data across six dimensions. Strict accuracy counts only correct / yes ; lenient counts partial as correct.
SO
SE
Misconception Type
%
%
Approach Fixation
42.8
59.2
Missing Knowledge
31.5
25.0
Tool & Syntax Fixation
15.5
9.8
False Belief
10.1
6.0
Appendix
Table 6: Distribution of misconception types across Stack Overflow and Stack Exchange
Human–Human
Human–Model
Stack Overflow
WikiHow
Stack Overflow
WikiHow
Dimension
Exact
α
Exact
α
Exact
α
Exact
α
S-solution Presence
1.000
1.000
1.000
1.000
1.000
1.000
0.978
0.915
I-solution Presence
1.000
1.000
1.000
1.000
1.000
1.000
0.975
0.940
Emphasis
1.000
1.000
1.000
1.000
0.742
0.600
0.798
0.587
S-I-misconception-identification
0.955
0.910
0.991
0.979
0.933
0.866
0.846
0.634
Appendix
Table 7: Inter-annotator agreement (human–human) and judge–human agreement (human–model, averaged across annotators) for Stack Overflow (SO, n=100 ) and WikiHow ( n=400 ). Model judge is Claude Sonnet 4.5.
Domain
Field
Human–Claude-Sonnet-4.5 κ
Human–gpt-4.1 κ
Human–gpt-5-mini κ
Stack Overflow ( n=100 )
presence.direct
1.000
0.710
0.665
presence.alternate
1.000
0.764
0.734
emphasis.stance
0.600
0.560
0.605
mismatch.verdict
0.865
0.865
0.864
WikiHow ( n=150 )
presence.direct
0.945
0.525
0.384
presence.alternate
0.959
0.505
0.515
Appendix
Table 8: Cohen’s κ between each alternate judge (gpt-4.1, gpt-5-mini) and the human annotations (averaged over the two annotators). Human labels are tied to Claude-Sonnet-4.5’s unit segmentation, so each judge’s presence call is compared against the human-confirmed ground truth keyed by item.
WikiHow ( n=150 )
Stack Overflow ( n=100 )
Field
Comparison
κ
Exact
κ
Exact
presence.alternate ( I-solution )
Claude–gpt-4.1
0.51
0.77
0.76
0.90
Claude–gpt-5-mini
0.52
0.78
0.73
0.89
mismatch.verdict (mismatch identification)
Claude–gpt-4.1
0.45
0.80
0.71
0.85
Claude–gpt-5-mini
0.29
0.62
0.71
0.85
Appendix
Table 9: Agreement between Claude-Sonnet-4.5 and each alternative judge
Dataset
Model
P( I-solution ∣ ID)
P( I-solution ∣ not ID)
Gap
ϕ
StackOverflow
GPT-5
84.8%
51.5%
+33.3%
0.361***
GPT-4.1
79.2%
38.2%
+40.9%
0.417***
Gemini 3.1 Pro
79.2%
42.3%
+36.9%
0.380***
Claude Sonnet 4.6
78.0%
42.8%
+35.2%
0.358***
DeepSeek V4 Pro
80.5%
49.0%
+31.5%
0.331***
Qwen 3.5-9B
73.9%
48.5%
+25.4%
0.259***
Appendix
Table 10: Conditional probability of I-solution mention given whether the model identified the misconception (ID) or not. Gap = P( I-solution ∣ ID) − P( I-solution ∣ not ID). ϕ = phi coefficient; *** p<0.001 , ** p<0.01 , * p<0.05 , ns = not significant. Manually-curated ( n=102 ) is a smaller set, so two models fall short of significance there.
Figure 16: Prompt used for multiple choice discriminative evaluation
Figure 17: Multiple choice discriminative evaluation results on Stack Exchange. Models overwhelmingly choose the pragmatic response as the most helpful one.
Figure 18: Multiple choice discriminative evaluation results on Stack Overflow. Models overwhelmingly choose the pragmatic response as the most helpful one.
Figure 19: Multiple choice discriminative evaluation results on Manually-curated. Models largely choose the pragmatic response as the most helpful one.
Figure 20: Multiple choice discriminative evaluation results on WikiHow. Models largely choose the pragmatic response as the most helpful one.
Solution
Emphasis
S-I-Miscon.
Fully
Fully
(at least once)
(resp. orientation)
identif.
Prag.
Coop.
Source
Model
S-soln.
I-soln.
S-soln
Balanced
I-soln
SO
Claude Sonnet 4.6
0.79 -4%
0.67 +14%
0.47 -18%
0.10 +25%
0.44 +22%
0.34 -28%
0.18 -28%
0.15 -25%
GPT-4.1
0.81 -1%
0.68 +11%
0.53 -15%
0.10 +25%
0.38 +27%
0.30 -46%
0.14 -39%
0.11 -42%
GPT-5
0.83 +0%
0.78 +10%
0.43 -9%
0.13 +8%
0.45 +10%
0.34 -43%
0.17 -45%
0.17 -39%
Gemini 3.1 Pro
0.76 -4%
0.77 +24%
0.32 -37%
0.09 +12%
0.54 +35%
0.58 +7%
0.38 +31%
0.30 +36%
Appendix
Table 11: Oracle ablation: models prompted with S-request and I-request alongside the query. Cell values show oracle score with positive / negative percentage change relative to query-only (Table 2 ). Bold marks the highest value per column within each source block. Shading is normalized per source block (min–max across all rows shown, oracle models and query-only human rows together), with one color family per block. Balanced uses a shared gray scale. Rows without a percentage are the corresponding query-only human response rows from Table 2 , shown for comparison. Overall Takeaway : Model responses are much more pragmatically aligned, yet still lag behind humans!
Solution
Emphasis
S-I-Miscon.
Fully
Fully
(at least once)
(resp. orientation)
identif.
Prag.
Coop.
Source
Model
S-soln.
I-soln.
S-soln
Balanced
I-soln
SO
Claude Sonnet 4.6
0.79 -4%
0.66 +11%
0.50 -12%
0.09 +11%
0.41 +14%
0.69 +47%
0.35 +39%
0.27 +37%
GPT-4.1
0.80 -2%
0.64 +5%
0.57 -8%
0.09 +17%
0.33 +11%
0.69 +22%
0.27 +18%
0.23 +23%
GPT-5
0.83 +0%
0.79 +11%
0.41 -13%
0.12 +2%
0.47 +14%
0.74 +24%
0.38 +23%
0.37 +31%
Qwen3.5-9B
0.71 -5%
0.70 +9%
0.46 -14%
0.06 +2%
0.48 +18%
0.78 +29%
0.40 +37%
0.27 +28%
Appendix
Table 12: Ablation with added user prompt "please correct the request if there are any misconception". Cell values show the ablation score with positive / negative percentage change relative to query-only (Table 2 ). Bold marks the highest value per column within each source block. Shading is normalized per source block (min–max across all rows shown, ablation models and query-only human rows together), with one color family per block. Balanced uses a shared gray scale. Rows without a percentage are the corresponding query-only human response rows from Table 2 , shown for comparison. Overall Takeaway : Model responses identify the misconception much more, however I-solution presence and emphasis do not improve as much, and overall models still lag behind humans!
Figure 21: Annotation for correctness of extracted Stated and Intended Requests
Figure 22: Annotation for correctness of extracted Response Dimensions: S-solution , I-solution and S-I-misconception-identification
Figure 23: Annotation for correctness of evaluated I-solution and S-solution presence
Figure 24: Annotation for correctness of evaluated response emphasis orientation
Figure 25: Annotation for correctness of evaluated misconception identification
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.
Are large language models (LLMs) bad at capturing human judgment? Two commonly stated limitations are that LLMs fail to capture full distributions of responses, and that their judgments are unstable across wording variations. We demonstrate simple prompting strategies that mitigate these limitations. Across two datasets--a U.S.-representative set of 144 moral scenarios and 38 moral beliefs from the International Social Survey Programme's Family and Changing Gender Roles module covering 32 countries--we show how simple elicitation techniques help improve AI-human alignment. First, prompting models to report standard deviations and response proportions recovers the full range of human responses better than common strategies. Second, ensuring scenarios are clear to human participants--as reflected in human confusion ratings--boosts model alignment, and LLMs can track human confusion ratings. At the same time, we find that LLMs' estimates of their own error are poorly calibrated, though they can predict human variability relatively well. These results suggest that asking better questions to LLMs can yield better answers.
Danica Dillion, Chen Cecilia Liu, Baihui Wang +5
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2
We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.
Neha Srikanth, Jordan Boyd-Graber, Rachel Rudinger