XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

Akhila Yerukola; Jena D. Hwang; Mingqian Zheng; Jenna Godsey; Hyunwoo Kim; Valentina Pyatkin; Jennifer Hu; Maarten Sap

cs.CLSep 6, 2026

XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

Authors: Akhila Yerukola, Jena D. Hwang, Mingqian Zheng, Jenna Godsey, Hyunwoo Kim, Valentina Pyatkin, Jennifer Hu, Maarten Sap

Organizations: Carnegie Mellon University · Allen Institute for AI · NVIDIA · Johns Hopkins University

Abstract

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that assesses model responses along three criteria grounded in cooperative response theory: (a) presence and (b) emphasis on pragmatic solutions, and (c) identification of misconceptions. Our experiments show that even the strongest LLMs predominantly answer the literal request (0.75--0.92) and far less often the intended one (0.33--0.71), while substantially lagging behind humans at identifying misconceptions (at most 63% vs. 79--90%). Further, models overwhelmingly prefer pragmatic responses in a multiple choice setting yet consistently fail to generate them. Oracle ablation experiments show that providing explicit user intent at generation time helps; however a large gap remains, suggesting pragmatic redirection is a fundamentally underdeveloped capability in current LLMs.

Figures & tables

Figure 1: Users often ask queries laden with misconceptions in both technical and non-technical domains. A user’s stated request of using regex to parse XML implicitly reflects an underlying goal of extracting structured data. A truly cooperative system must: (1) address the stated request by providing a working regex, if feasible, (2) identify the misconception that regex is unavoidably fragile for nested or complex XML, and (3) suggest an XML parser to better serve the user’s intended goal.

Figure 2: XYBench construction pipeline. Queries are curated from three sources: technical queries are drawn from real Stack Overflow and Stack Exchange threads tagged as XY problems; a manually-curated set of in-the-wild XY queries; and further everyday queries are synthetically constructed from WikiHow guides by injecting likely user mistakes into procedural steps. All pipelines extract or synthesize stated ( S- ) and intended ( I- ) requests and solutions, along with misconceptions arising from stated-intended requests ( S-I-misconception-identification ). Queries and associated metadata are rigorously filtered for quality and validated by human annotators to produce the final benchmark.

Source	Domain	# Queries
Stack Overflow	programming	1557
Stack Exchange	electronics	61
	superuser	36
	unix	36
	askubuntu	19
	+11 others	32

Table 1: XYBench statistics across data sources.

		Solution		Emphasis			S-I-Miscon.	Fully	Fully
		(at least once)		avg. orientation of responses			identif.	Prag.	Coop.
Source	Model	S-soln.	I-soln.	S-soln	Balanced	I-soln
Stack Overflow	Claude Sonnet 4.6	0.82	0.59	0.57	0.08	0.36	0.47	0.25	0.20
	GPT-5	0.83	0.71	0.47	0.12	0.41	0.60	0.31	0.28
	Gemini 3.1 Pro	0.79	0.62	0.51	0.08	0.40	0.54	0.29	0.22
	Qwen3.5-9B	0.75	0.64	0.53	0.06	0.41	0.60	0.29	0.21

Table 2: Evaluation of model and human responses on XYBench . Models consistently address the stated request but rarely prioritize the implicit goal ( high S-soln, lower I-soln presence ). Models also consistently fall behind humans in identifying misconceptions and responding to what users actually need . Within each source block, color gradation is normalized, and the maximum value in each column is bolded.

Panel of Figure 3 — Figure 3: Oracle upper-bound experiment where S-request and I-request are given at generation time along with the query. Solid bars are query-only; pale segments show the change under the oracle setting, with hatching marking declines. Human bars are query-only. Despite overall improvement in model performance, they still lag behind humans, indicating that intent inference and misconception detection are not the only bottlenecks.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Example query from Stack Overflow: Human Reference Answer, and Model Responses from Claude Sonnet 4.6 and Gemini 3.1 Pro are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution

Figure 5: Example query from Stack Exchange: Human Reference Answer, and Model Responses from Claude Sonnet 4.6 and GPT 4.1 are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution

Figure 6: Example query from Manually-Curated: Model Responses from Gemini 3.1 Pro and Qwen3.5-9B are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution

Figure 7: Example query from WikiHow: Model Responses from Claude Sonnet 4.6 and GPT 4.1 are shown. Highlighted are S-solution , S-I-misconception-identification , I-solution

Panel of Figure 8 — Figure 8: Prompt used to extract S-request and I-request from real conversation threads on SO and SE

Panel of Figure 9 — Figure 9: Prompt used to extract S-solution , I-solution and S-I-misconception-identification from real conversation threads on SO and SE, given the query and previously extracted S-request and I-request

Panel of Figure 10 — Figure 10: Prompt used to filter queries which are better suited with a S-solution , i.e., they are less misconceived. Two response choices are provided (order shuffled): one with S-solution and one with I-solution . Models must choose the response option that might be most helpful to the user.

Pipeline Accuracy	Strict	Lenient
MCQ filtering agreement	0.920	0.940

Appendix

Table 3: Human validation results for MCQ filtering over

n=100

sampled instances (balanced retained/excluded). Annotators judge each filtering decision as agree , borderline , or disagree . Strict pipeline accuracy counts agree only; lenient counts borderline as correct.

Dimension	Exact	Kripp. $\alpha$	Ann1	AnnAK	Strict	Lenient
S-request	1.000	1.000	0.950	0.950	0.950	0.990
I-request	0.970	0.687	0.950	0.950	0.940	1.000
S-solution	1.000	1.000	0.930	0.930	0.930	0.930
I-solution	0.970	0.715	0.950	0.940	0.930	0.960
S-I-misconception-identification	0.970	0.798	0.910	0.930	0.910	0.930
All fields correct	—	—	0.840	0.840	0.820	0.860

Appendix

Table 4: Human validation results on Stack Overflow data over

n=100

sampled instances across five dimensions. Strict accuracy counts only correct / yes ; lenient counts partial as correct.

Panel of Figure 11 — Figure 11: Prompt used to generate WikiHow data items: initial query, S- and I- request and solution, S-I-misconception-identification

Panel of Figure 12 — Figure 12: Prompt used to check if I-request is identifiable in the user query by a domain expert, rewrite if needed

Panel of Figure 13 — Figure 13: Prompt used to check query has obvious clues, rewrite if needed

Panel of Figure 14 — Figure 14: Prompt used to filter and regenerate query data: Given a query, we filter and rewrite queries to again remove obvious clues, as well rewrite to add psychologically grounded explanation to motivate S-request

Panel of Figure 15 — Figure 15: Prompt used to check S-I request gap quality

Dimension	Exact	Kripp. $\alpha$	Ann1	AnnAK	Strict	Lenient
S-request	0.995	0.988	0.723	0.725	0.723	0.953
I-request	0.995	0.986	0.775	0.777	0.775	0.970
query.plausible	1.000	1.000	0.863	0.863	0.863	—
S-solution	0.995	0.987	0.747	0.747	0.744	0.749
S-I-misconception-identification	0.995	0.983	0.818	0.818	0.815	0.820
I-solution	0.995	0.980	0.858	0.853	0.853	0.858

Appendix

Table 5: Human validation results over

n=400

sampled instances of WikiHow constructed data across six dimensions. Strict accuracy counts only correct / yes ; lenient counts partial as correct.

Misconception Type	%	%
	SO	SE
Approach Fixation	42.8	59.2
Missing Knowledge	31.5	25.0
Tool & Syntax Fixation	15.5	9.8
False Belief	10.1	6.0

Appendix

Table 6: Distribution of misconception types across Stack Overflow and Stack Exchange

	Human–Human				Human–Model
	Stack Overflow		WikiHow		Stack Overflow		WikiHow
Dimension	Exact	$\alpha$	Exact	$\alpha$	Exact	$\alpha$	Exact	$\alpha$
S-solution Presence	1.000	1.000	1.000	1.000	1.000	1.000	0.978	0.915
I-solution Presence	1.000	1.000	1.000	1.000	1.000	1.000	0.975	0.940
Emphasis	1.000	1.000	1.000	1.000	0.742	0.600	0.798	0.587
S-I-misconception-identification	0.955	0.910	0.991	0.979	0.933	0.866	0.846	0.634

Appendix

Table 7: Inter-annotator agreement (human–human) and judge–human agreement (human–model, averaged across annotators) for Stack Overflow (SO,

n=100

) and WikiHow (

n=400

). Model judge is Claude Sonnet 4.5.

Domain	Field	Human–Claude-Sonnet-4.5 $\kappa$	Human–gpt-4.1 $\kappa$	Human–gpt-5-mini $\kappa$
Stack Overflow ( $n{=}100$ )	presence.direct	1.000	0.710	0.665
	presence.alternate	1.000	0.764	0.734
	emphasis.stance	0.600	0.560	0.605
	mismatch.verdict	0.865	0.865	0.864
WikiHow ( $n{=}150$ )	presence.direct	0.945	0.525	0.384
	presence.alternate	0.959	0.505	0.515

Appendix

Table 8: Cohen’s

\kappa

between each alternate judge (gpt-4.1, gpt-5-mini) and the human annotations (averaged over the two annotators). Human labels are tied to Claude-Sonnet-4.5’s unit segmentation, so each judge’s presence call is compared against the human-confirmed ground truth keyed by item.

		WikiHow ( $n{=}150$ )		Stack Overflow ( $n{=}100$ )
Field	Comparison	$\kappa$	Exact	$\kappa$	Exact
presence.alternate ( I-solution )	Claude–gpt-4.1	0.51	0.77	0.76	0.90
presence.alternate ( I-solution )	Claude–gpt-5-mini	0.52	0.78	0.73	0.89
mismatch.verdict (mismatch identification)	Claude–gpt-4.1	0.45	0.80	0.71	0.85
mismatch.verdict (mismatch identification)	Claude–gpt-5-mini	0.29	0.62	0.71	0.85

Appendix

Table 9: Agreement between Claude-Sonnet-4.5 and each alternative judge

Dataset	Model	P( I-solution $\|$ ID)	P( I-solution $\|$ not ID)	Gap	$\phi$
StackOverflow	GPT-5	84.8%	51.5%	+33.3%	0.361***
	GPT-4.1	79.2%	38.2%	+40.9%	0.417***
	Gemini 3.1 Pro	79.2%	42.3%	+36.9%	0.380***
	Claude Sonnet 4.6	78.0%	42.8%	+35.2%	0.358***
	DeepSeek V4 Pro	80.5%	49.0%	+31.5%	0.331***
	Qwen 3.5-9B	73.9%	48.5%	+25.4%	0.259***

Appendix

Table 10: Conditional probability of I-solution mention given whether the model identified the misconception (ID) or not. Gap = P( I-solution

|

ID)

-

P( I-solution

|

not ID).

\phi

= phi coefficient; ***

p<0.001

, **

p<0.01

, *

p<0.05

, ns = not significant. Manually-curated (

n=102

) is a smaller set, so two models fall short of significance there.

Panel of Figure 16 — Figure 16: Prompt used for multiple choice discriminative evaluation

Figure 17: Multiple choice discriminative evaluation results on Stack Exchange. Models overwhelmingly choose the pragmatic response as the most helpful one.

Figure 18: Multiple choice discriminative evaluation results on Stack Overflow. Models overwhelmingly choose the pragmatic response as the most helpful one.

Figure 19: Multiple choice discriminative evaluation results on Manually-curated. Models largely choose the pragmatic response as the most helpful one.

Figure 20: Multiple choice discriminative evaluation results on WikiHow. Models largely choose the pragmatic response as the most helpful one.

		Solution		Emphasis			S-I-Miscon.	Fully	Fully
		(at least once)		(resp. orientation)			identif.	Prag.	Coop.
Source	Model	S-soln.	I-soln.	S-soln	Balanced	I-soln
SO	Claude Sonnet 4.6	0.79 -4%	0.67 +14%	0.47 -18%	0.10 +25%	0.44 +22%	0.34 -28%	0.18 -28%	0.15 -25%
	GPT-4.1	0.81 -1%	0.68 +11%	0.53 -15%	0.10 +25%	0.38 +27%	0.30 -46%	0.14 -39%	0.11 -42%
	GPT-5	0.83 +0%	0.78 +10%	0.43 -9%	0.13 +8%	0.45 +10%	0.34 -43%	0.17 -45%	0.17 -39%
	Gemini 3.1 Pro	0.76 -4%	0.77 +24%	0.32 -37%	0.09 +12%	0.54 +35%	0.58 +7%	0.38 +31%	0.30 +36%

Appendix

Table 11: Oracle ablation: models prompted with S-request and I-request alongside the query. Cell values show oracle score with positive / negative percentage change relative to query-only (Table 2 ). Bold marks the highest value per column within each source block. Shading is normalized per source block (min–max across all rows shown, oracle models and query-only human rows together), with one color family per block. Balanced uses a shared gray scale. Rows without a percentage are the corresponding query-only human response rows from Table 2 , shown for comparison. Overall Takeaway : Model responses are much more pragmatically aligned, yet still lag behind humans!

		Solution		Emphasis			S-I-Miscon.	Fully	Fully
		(at least once)		(resp. orientation)			identif.	Prag.	Coop.
Source	Model	S-soln.	I-soln.	S-soln	Balanced	I-soln
SO	Claude Sonnet 4.6	0.79 -4%	0.66 +11%	0.50 -12%	0.09 +11%	0.41 +14%	0.69 +47%	0.35 +39%	0.27 +37%
	GPT-4.1	0.80 -2%	0.64 +5%	0.57 -8%	0.09 +17%	0.33 +11%	0.69 +22%	0.27 +18%	0.23 +23%
	GPT-5	0.83 +0%	0.79 +11%	0.41 -13%	0.12 +2%	0.47 +14%	0.74 +24%	0.38 +23%	0.37 +31%
	Qwen3.5-9B	0.71 -5%	0.70 +9%	0.46 -14%	0.06 +2%	0.48 +18%	0.78 +29%	0.40 +37%	0.27 +28%

Appendix

Table 12: Ablation with added user prompt "please correct the request if there are any misconception". Cell values show the ablation score with positive / negative percentage change relative to query-only (Table 2 ). Bold marks the highest value per column within each source block. Shading is normalized per source block (min–max across all rows shown, ablation models and query-only human rows together), with one color family per block. Balanced uses a shared gray scale. Rows without a percentage are the corresponding query-only human response rows from Table 2 , shown for comparison. Overall Takeaway : Model responses identify the misconception much more, however I-solution presence and emphasis do not improve as much, and overall models still lag behind humans!

Figure 21: Annotation for correctness of extracted Stated and Intended Requests

Figure 22: Annotation for correctness of extracted Response Dimensions: S-solution , I-solution and S-I-misconception-identification

Figure 23: Annotation for correctness of evaluated I-solution and S-solution presence

Figure 24: Annotation for correctness of evaluated response emphasis orientation

Figure 25: Annotation for correctness of evaluated misconception identification

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
Sep 7, 2026Yongchang Peng, Qingshui Gu, Liya Zhu +31Pragmatic Reasoning in Language Models LLM Agent Evaluation
ByteDance Seed · M-A-P
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Preference Evaluation LLM Alignment
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2
DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
Apr 16, 2026Neha Srikanth, Jordan Boyd-Graber, Rachel RudingerQuestion Answering
University of Maryland

Sep 7, 2026J/K move · Enter open · S save

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

Yongchang Peng, Qingshui Gu, Liya Zhu, Ge Zhang+30

ByteDance Seed · M-A-P

Pragmatic Reasoning in Language Models LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

LLMs Can Better Capture Human Judgments--With the Right Prompts

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering