Investigating Human--AI Discrepancies via Multiple-Solution Problems

Zihao Wang; Francesco Insulla; Andrea Montanari

cs.CYSep 28, 2026

Investigating Human--AI Discrepancies via Multiple-Solution Problems

Authors: Zihao Wang, Francesco Insulla, Andrea Montanari

Organizations: Department of Mathematics, Stanford University · Institute of Computational and Mathematical Engineering, Stanford University · Department of Statistics and Department of Mathematics, Stanford University

Abstract

Frontier artificial intelligence (AI) models are benchmarked on whether they reach a correct answer. Yet many problems admit several correct answers and repeated attempts, by different people or by the same model resampled, trace out a distribution over them. In this work, we ask whether human and model reasoning lead to different distributions over valid solutions. Our testbed comprises 270 reasoning puzzles across five puzzle families. These multiple-solution puzzles each have 3 to 8 valid solutions and are simple enough that humans and models can solve them reliably. The resulting distributions differ markedly: models differ from one another, yet resemble each other far more than they resemble humans. Model distributions are, moreover, within every puzzle family, less diverse than human ones. We compare these discrepancies across puzzle categories, and trace how they respond to reasoning-effort settings, to prompting, and to perturbations of the puzzle that leave its solutions unchanged. Together, these results point at significant differences between human and AI problem-solving processes, and their choice among equally defensible solutions. As progressive deployment of AI systems in society comes into focus, evaluating such differences (beyond one-dimensional accuracy metrics) is increasingly important. Data and code are available at https://hai-discrepancies.github.io/

Figures & tables

Figure 1: General experimental procedure. Puzzles with multiple valid solutions are presented to humans and models, and their answer distributions are compared.

Figure 2: Total-variation distances and source and family geometry. a , Equal-family mean TV distances among Uniform, Human and the three AI models, averaged over puzzle family (we report everywhere 95% confidence intervals from 5,000 whole-puzzle bootstrap resamples within family.) b , TV distances by puzzle family, averaged across 20 puzzles per family. c , Two-dimensional solver embeddings from multidimensional scaling of mean TV across all puzzle families. d , Centered kernel alignment (CKA) of full positive-semidefinite five-source kernels built from each family’s mean TV matrix across 20 puzzles. Percentages on the axes give the fraction of explained distance. Estimates use 100 main module puzzles (20 per family), conditional on correct human or valid model answers; model conditions are low effort and plain prompts.

	Mean normalized entropy [95% CI]
Puzzle family	Human	GPT	Claude	Gemini
Arithmetic	$0.68[0.65,0.72]$	$0.45[0.36,0.55]$	$0.58[0.52,0.64]$	$0.63[0.59,0.68]$
Maze	$0.68[0.62,0.74]$	$0.48[0.38,0.59]$	$0.39[0.27,0.52]$	$0.49[0.38,0.61]$
Rooks	$0.91[0.88,0.94]$	$0.28[0.18,0.39]$	$0.47[0.35,0.59]$	$0.42[0.31,0.53]$
Minesweeper	$0.93[0.91,0.95]$	$0.21[0.11,0.32]$	$0.54[0.43,0.65]$	$0.33[0.21,0.45]$
Sudoku	$0.91[0.88,0.94]$	$0.02[0.01,0.04]$	$0.42[0.34,0.50]$	$0.39[0.29,0.51]$

Table 1: Normalized Shannon entropy by puzzle family. Same data pipeline as for Fig. 2 .

Figure 3: Entropy across puzzle families. a , Mean normalized Shannon entropy across 20 puzzles per family and source. b , Histograms of normalized entropy for each source over the 100 main puzzles (20 per family; bins of width 0.05, y axis in percent of puzzles); ticks below the axis mark the individual puzzles. Same data pipeline as for Fig. 2 .

Figure 4: Effects of presentation and immediate context. a , TV from reference. (As a reference distribution: Preceding-puzzle context uses the product of its observed answer marginals for related puzzle pairs; other modules use the unperturbed answer distribution, after alignment for Spatial reflection.) b , Normalized probability change for Highlighting, Preceding-puzzle context and Strategy primer (defined in Section 4.2 ). For Preceding-puzzle context, the target group consists of answer pairs in which the target-puzzle answer is the mapped counterpart of the preceding-puzzle answer. c , Mutual information (MI), in bits, in related and unrelated context. Context estimates include only pairs in which both puzzles were answered correctly.

Figure 5: Relative difficulty across sources. Relative-difficulty scores for 100 main puzzles (20 per family), standardized within each source to mean zero and sample standard deviation one. Diagonal panels show kernel density estimates of each source’s scores; ticks mark the individual puzzles.

Figure 6: Relative-difficulty profile decomposition. a , Family coefficients of the first two singular directions. b , Corresponding source scores. Percentages indicate the share of variation in the centered family profiles captured by each direction.

Figure 7: Distributional distances across reasoning-effort settings and prompts. a , Model conditions projected onto the fixed MDS axes from Figure 2 c, using distances to Human, Uniform and the three Low plain model distributions, see Section 4.2 . b , Mean TV from Human. As a reference, distance from Uniform is shown by a dashed line and band. c , Mean TV from each model’s Low plain distribution to its other conditions. In b and c , estimates average 100 main puzzles (20 per family); bars and the Uniform band are 95% whole-puzzle bootstrap intervals from 5,000 resamples within family. Circles indicate the low reasoning-effort setting and squares the medium setting; filled markers indicate plain prompts and hollow markers human-persona prompts.

Figure 8: Examples from the five puzzle families. One example each of Arithmetic, Maze, Rooks, Minesweeper and Sudoku.

Figure S1: Highlighting : one example puzzle per family. The two highlighting variants outline disjoint cells and are the only perturbation to the puzzle.

Figure S2: Spatial reflection : a board and three transformed versions, preserving solution identities under the corresponding coordinate map.

Figure S3: Number ordering : the same four numbers displayed in a different order.

Figure S4: Preceding-puzzle context : the target puzzle B alone, or displayed together with an unrelated or a related puzzle A.

Figure S5: Strategy primer : the multi-solution target puzzle B alone, or with a single-solution primer puzzle A that can be solved by a designated local rule. The primer is an ordinary puzzle, not a worked demonstration. Humans see A before B; models receive both panels in one image.

Arithmetic
task	Make exactly 24 using the four displayed numbers.
rules	Use each number exactly once. Allowed operations: +, -, *, /. Parentheses are allowed. Do not use other numbers. Only one correct answer is needed.
output	Return exactly one line: ANSWER: expression=<expression>. Replace placeholders with your answer.
Maze
task	Find one shortest path from S to G.
rules	Move only U/D/L/R. White cells are open; black cells are walls. S and G are open. The path must be shortest. Only one correct answer is needed.

Figure S6: Exact single-puzzle instructions. Text displayed above the boards, reproduced word for word from the rendered stimuli.

Two puzzles: Minesweeper
task	Solve Puzzle A and Puzzle B.
rules	Use the rule statement shown above each Minesweeper board. Only one correct answer is needed for each puzzle.
output	Return exactly two lines and nothing else. Replace placeholders with your answers.
line 1	A: cell=<row,column>
line 2	B: cell=<row,column>
Above each board, under its Puzzle A / Puzzle B label:

Figure S7: Exact two-puzzle instructions. Text displayed above the boards, reproduced word for word from the rendered stimuli.

Plain prompt, single-puzzle trials
	Solve the puzzle shown in the image.
	Return exactly one final answer line and nothing else.
	Your final line must exactly match the answer format shown in the image, including labels, punctuation, separators, and capitalization.
	Replace placeholders with your answer.
	Do not include any explanation, reasoning, preface, markdown, extra whitespace, or additional lines.
	Give the first correct answer that comes to mind.

Figure S8: Exact model prompts. The four prompts are reproduced word for word. Every request carries one of them plus the rendered image.

	GPT	Claude	Gemini
Model identifier	gpt-5.6-sol	claude-opus-4-8	gemini-3.5-flash
API	Responses	Messages	GenerateContent
Reasoning control	reasoning effort	adaptive thinking	thinking level
Settings	low, medium	low, medium	low, medium
Output token cap	8,192	8,192	8,192
Text verbosity	low	—	—

Table S1: Model sampling configuration, as frozen in the run manifests.

Source	Responses	Correct/valid	Collected
Human, main battery (104 participants)	10,400	90.5%	2026-06
Human, modules (417 participants, 2 waves)	25,647	92.3%	2026-06
GPT, main battery	40,000	98.0–99.4%	2026-07
GPT, modules	68,000	99.8–99.9%	2026-07
Claude, main battery	40,000	99.6–99.8%	2026-06
Claude, modules	68,000	99.3–99.4%	2026-06

Table S2: Data underlying the analyses. Human counts are retained participants and attempted responses; model counts are scored response rows over four conditions (two reasoning-effort settings

\times

two prompts) at 100 samples per puzzle and condition. These are collection totals; primary analyses use the 100 low-effort, plain-prompt samples for each model and puzzle. Rates are correct for humans and valid for models.

		95% bootstrap CI
Comparison	Mean TV	Puzzles	Trials	Both
Uniform vs Human	$0.30$	$[0.29,0.32]$	$[0.30,0.33]$	$[0.29,0.34]$
Uniform vs GPT	$0.65$	$[0.63,0.68]$	$[0.65,0.66]$	$[0.63,0.68]$
Uniform vs Claude	$0.55$	$[0.52,0.58]$	$[0.54,0.56]$	$[0.52,0.58]$
Uniform vs Gemini	$0.56$	$[0.53,0.59]$	$[0.56,0.57]$	$[0.53,0.59]$
Human vs GPT	$0.54$	$[0.51,0.57]$	$[0.53,0.56]$	$[0.51,0.58]$

Table S3: Pairwise mean total-variation distance across the 100-puzzle battery. Uniform assigns equal mass to a puzzle’s valid solution classes.

Figure S9: Per-puzzle normalized-entropy relationships. Each panel compares one source pair over the 100-puzzle main battery. Diagonal panels show kernel density estimates of each source’s normalized entropy (bandwidth 0.075, reflected at 0 and 1); ticks mark the individual puzzles.

	Pearson correlation [95% CI]
Source pair	Mean	Arithmetic	Maze	Rooks	Minesweeper	Sudoku
Human vs GPT	$0.59[0.47,0.71]$	$0.71[0.54,0.85]$	$0.51[0.20,0.82]$	$0.85[0.74,0.92]$	$0.72[0.48,0.87]$	$0.17[-0.32,0.58]$
Human vs Claude	$0.72[0.60,0.80]$	$0.56[0.20,0.81]$	$0.72[0.43,0.92]$	$0.77[0.60,0.89]$	$0.78[0.62,0.90]$	$0.76[0.47,0.91]$
Human vs Gemini	$0.62[0.48,0.72]$	$0.15[-0.37,0.52]$	$0.74[0.50,0.89]$	$0.86[0.76,0.94]$	$0.73[0.55,0.86]$	$0.62[0.29,0.84]$
GPT vs Claude	$0.71[0.56,0.81]$	$0.68[0.50,0.85]$	$0.87[0.67,0.96]$	$0.90[0.81,0.97]$	$0.75[0.46,0.93]$	$0.32[-0.32,0.75]$
GPT vs Gemini	$0.60[0.44,0.74]$	$0.27[-0.32,0.67]$	$0.76[0.51,0.89]$	$0.92[0.82,0.97]$	$0.80[0.60,0.93]$	$0.24[-0.28,0.69]$

Table S6: Pearson correlation of per-puzzle relative-difficulty scores by puzzle family. The Mean column weights the five family correlations equally. Brackets are 95% paired-puzzle bootstrap intervals.

Figure S10: Relative difficulty by puzzle family. Points are family means. Bars show pointwise 95% intervals from 5,000 whole-puzzle bootstrap resamples within family, with each source’s mean recomputed in every resample.

GPT
Contrast	Mean	Arithmetic	Maze	Rooks	Minesweeper	Sudoku
Persona $-$ plain (low effort)	$0.82$ [1pt] $[-0.35,1.95]$	$1.53$ [1pt] $[-1.51,4.67]$	$1.14$ [1pt] $[-1.98,4.35]$	$1.17$ [1pt] $[-0.95,3.49]$	$0.26$ [1pt] $[-2.43,2.99]$	$0.00$ [1pt] $[-0.50,0.50]$
Persona $-$ plain (medium effort)	$\mathbf{-1.08}$ [1pt] $[-1.98,-0.21]$	$-1.42$ [1pt] $[-4.03,1.11]$	$0.03$ [1pt] $[-2.27,2.19]$	$\mathbf{-2.55}$ [1pt] $[-4.90,-0.57]$	$\mathbf{-1.86}$ [1pt] $[-4.15,-0.14]$	$0.40$ [1pt] $[0.00,0.85]$
Medium $-$ low (plain prompt)	$1.11$ [1pt] $[-0.07,2.30]$	$1.95$ [1pt] $[-1.18,5.13]$	$-0.27$ [1pt] $[-3.97,2.99]$	$\mathbf{2.91}$ [1pt] $[0.75,5.43]$	$1.04$ [1pt] $[-1.99,3.92]$	$-0.05$ [1pt] $[-0.60,0.50]$
Medium $-$ low (persona prompt)	$-0.78$ [1pt] $[-2.08,0.52]$	$-1.00$ [1pt] $[-3.77,1.69]$	$-1.38$ [1pt] $[-5.76,2.88]$	$-0.81$ [1pt] $[-3.57,1.89]$	$-1.09$ [1pt] $[-4.12,1.47]$	$\mathbf{0.35}$ [1pt] $[0.10,0.70]$
Claude

Table S7: Condition contrasts in model–Human mean TV. Estimates and 95% paired-puzzle bootstrap intervals are multiplied by 100 (Section 4.2 ). Positive values indicate greater distance from Human; Mean weights the five families equally. Bold estimates have intervals excluding zero. Intervals are pointwise and unadjusted for multiple comparisons.

a Selected-solution prediction
Family	Human	GPT	Claude	Gemini
Arithmetic	$\bm{0.79}[0.62,0.92]$	$0.20[-0.10,0.57]$	$0.18[-0.03,0.42]$	$0.34[-0.01,0.61]$
Maze	$\bm{0.67}[0.49,0.88]$	$\bm{0.57}[0.30,0.78]$	$\bm{0.61}[0.39,0.91]$	$\bm{0.65}[0.41,0.90]$
Rooks	$0.20[-0.17,0.49]$	$-0.03[-0.16,0.44]$	$0.25[-0.05,0.54]$	$0.01[-0.15,0.41]$
Minesweeper	$0.33[-0.31,0.57]$	$0.01[-0.37,0.34]$	$0.25[-0.23,0.46]$	$0.24[-0.27,0.47]$
Sudoku	$0.26[-0.09,0.49]$	$0.10[-0.25,0.29]$	$-0.21[-0.27,0.31]$	$0.06[-0.33,0.28]$

Table S8: Normalized held-out prediction accuracy. Accuracy is normalized within each held-out puzzle before averaging over 20 puzzles per family (Section S4.1 ). Brackets give 95% intervals from 2,000 whole-puzzle bootstrap refits. Model responses use low effort and the plain prompt.

Source	$\widehat{\theta}_{\mathrm{turn}}$	$\widehat{\theta}_{\mathrm{edge}}$	$\widehat{\theta}_{\mathrm{rel}}$	Acc.
Human	$\bm{-1.23}[-1.84,-0.76]$	$\bm{-1.19}[-1.74,-0.58]$	$-0.32[-1.01,0.30]$	$0.41[0.32,0.52]$
GPT	$\bm{-1.65}[-3.12,-0.61]$	$\bm{-1.66}[-2.60,-1.00]$	$\bm{-0.98}[-2.23,-0.04]$	$0.46[0.32,0.60]$
Claude	$\bm{-2.47}[-3.93,-1.51]$	$\bm{-2.01}[-3.16,-1.05]$	$-0.84[-1.87,0.13]$	$0.54[0.39,0.75]$
Gemini	$\bm{-1.60}[-2.97,-0.61]$	$\bm{-1.81}[-2.56,-1.01]$	$0.08[-0.99,0.90]$	$0.54[0.40,0.71]$

Table S9: Selected-solution Maze regressions.

Model/Human	$\widehat{\beta}_{\mathrm{turn}}$	$\widehat{\beta}_{\mathrm{edge}}$	$\widehat{\beta}_{\mathrm{rel}}$	BA
GPT	$-0.55[-1.44,0.55]$	$-0.41[-1.18,0.11]$	$-0.71[-1.52,0.28]$	$0.46[0.39,0.58]$
Claude	$\bm{-1.08}[-2.19,-0.25]$	$-0.61[-1.44,0.11]$	$-0.58[-1.44,0.44]$	$0.57[0.47,0.63]$
Gemini	$-0.25[-1.14,0.52]$	$\bm{-0.46}[-0.99,-0.03]$	$0.27[-0.42,1.07]$	$0.55[0.50,0.60]$

Table S10: Source classification for Maze.

Source	$\widehat{\theta}_{\mathrm{div}}$	$\widehat{\theta}_{\mathrm{bal}}$	$\widehat{\theta}_{\mathrm{inv}}$	Acc.
Human	$\bm{-0.64}[-0.89,-0.34]$	$\bm{1.72}[1.49,1.96]$	$\bm{-0.94}[-1.37,-0.58]$	$0.40[0.33,0.46]$
GPT	$-0.62[-1.54,0.05]$	$\bm{1.41}[0.90,2.08]$	$-0.21[-1.12,0.69]$	$0.23[0.08,0.47]$
Claude	$-0.08[-0.82,0.41]$	$\bm{1.26}[0.68,1.91]$	$-0.49[-1.09,0.09]$	$0.21[0.13,0.31]$
Gemini	$0.14[-0.62,0.76]$	$\bm{0.81}[0.44,1.17]$	$\bm{0.60}[0.04,1.19]$	$0.28[0.13,0.42]$

Table S11: Selected-solution Arithmetic regressions.

Model/Human	$\widehat{\beta}_{\mathrm{div}}$	$\widehat{\beta}_{\mathrm{bal}}$	$\widehat{\beta}_{\mathrm{inv}}$	BA
GPT	$0.07[-0.83,0.69]$	$-0.32[-0.70,0.14]$	$\bm{0.66}[0.05,1.31]$	$0.55[0.46,0.61]$
Claude	$0.59[-0.14,1.12]$	$-0.48[-0.97,0.09]$	$\bm{0.49}[0.04,1.03]$	$0.55[0.48,0.61]$
Gemini	$\bm{0.90}[0.18,1.47]$	$\bm{-0.96}[-1.43,-0.53]$	$\bm{1.49}[0.87,2.21]$	$0.65[0.60,0.70]$

Table S12: Source classification for Arithmetic.

Source	$\widehat{\theta}_{\mathrm{rarity}}$	$\widehat{\theta}_{\mathrm{mono}}$	$\widehat{\theta}_{\mathrm{change}}$	Acc.
Human	$0.05[-0.21,0.33]$	$-0.33[-0.83,0.07]$	$-0.28[-0.72,0.22]$	$0.22[0.15,0.29]$
GPT	$\bm{-1.78}[-3.33,-0.94]$	$0.15[-1.29,1.38]$	$-0.24[-1.63,1.25]$	$0.16[0.08,0.47]$
Claude	$\bm{-2.31}[-3.21,-1.70]$	$-0.29[-1.36,0.70]$	$-0.01[-0.93,0.84]$	$0.33[0.17,0.51]$
Gemini	$\bm{-1.86}[-2.91,-1.29]$	$0.26[-0.92,1.39]$	$-0.15[-1.28,0.88]$	$0.21[0.13,0.42]$

Table S13: Selected-solution Rooks regressions.

Model/Human	$\widehat{\beta}_{\mathrm{rarity}}$	$\widehat{\beta}_{\mathrm{mono}}$	$\widehat{\beta}_{\mathrm{change}}$	BA
GPT	$\bm{-1.67}[-2.77,-0.93]$	$0.32[-0.75,1.18]$	$-0.05[-0.89,0.83]$	$0.61[0.50,0.70]$
Claude	$\bm{-2.10}[-2.71,-1.63]$	$0.02[-0.79,0.79]$	$0.19[-0.41,0.72]$	$0.65[0.56,0.72]$
Gemini	$\bm{-1.73}[-2.53,-1.21]$	$0.42[-0.39,1.18]$	$0.08[-0.47,0.59]$	$0.62[0.54,0.70]$

Table S14: Source classification for Rooks.

Source	$\widehat{\theta}_{\mathrm{size}}$	$\widehat{\theta}_{\mathrm{scope}}$	Acc.
Human	$\bm{-0.48}[-0.87,-0.09]$	$-0.17[-0.58,0.18]$	$0.29[0.21,0.34]$
GPT	$0.34[-1.08,2.19]$	$-0.06[-1.98,1.36]$	$0.21[0.04,0.40]$
Claude	$-0.66[-2.32,1.09]$	$-0.59[-2.15,0.45]$	$0.32[0.17,0.40]$
Gemini	$-1.22[-3.53,0.95]$	$-0.46[-2.40,0.66]$	$0.34[0.12,0.47]$

Table S15: Selected-solution Minesweeper regressions.

Model/Human	$\widehat{\beta}_{\mathrm{size}}$	$\widehat{\beta}_{\mathrm{scope}}$	BA
GPT	$0.74[-0.22,2.25]$	$0.12[-1.35,1.01]$	$0.56[0.38,0.64]$
Claude	$-0.17[-1.35,0.98]$	$-0.36[-1.38,0.35]$	$0.53[0.43,0.57]$
Gemini	$-0.66[-2.10,0.85]$	$-0.25[-1.68,0.42]$	$0.52[0.44,0.59]$

Table S16: Source classification for Minesweeper.

Source	$\widehat{\theta}_{\mathrm{domain}}$	$\widehat{\theta}_{\mathrm{location}}$	Acc.
Human	$-0.20[-0.53,0.08]$	$\bm{-0.89}[-1.12,-0.68]$	$0.28[0.21,0.33]$
GPT	$-0.40[-1.86,0.97]$	$\bm{-1.06}[-4.98,-0.27]$	$0.28[0.03,0.45]$
Claude	$0.11[-0.97,1.26]$	$\bm{-0.81}[-1.83,-0.16]$	$0.10[0.06,0.41]$
Gemini	$-0.42[-1.32,0.37]$	$\bm{-1.21}[-3.20,-0.61]$	$0.28[0.08,0.39]$

Table S17: Selected-solution Sudoku regressions.

Model/Human	$\widehat{\beta}_{\mathrm{domain}}$	$\widehat{\beta}_{\mathrm{location}}$	BA
GPT	$-0.16[-1.39,0.84]$	$-0.11[-1.55,0.43]$	$0.49[0.33,0.61]$
Claude	$0.25[-0.55,1.07]$	$0.06[-0.64,0.49]$	$0.51[0.37,0.58]$
Gemini	$-0.19[-0.76,0.30]$	$-0.22[-1.43,0.15]$	$0.53[0.42,0.58]$

Table S18: Source classification for Sudoku.

Module	Human	GPT	Claude	Gemini
Highlighting	0.0917 (10/20)	0.3034 (10/20)	0.0146 (13/20)	0.0935 (13/20)
Related context	0.0010 (9/10)	0.8383 (1/10)	0.3363 (3/10)	0.1878 (2/10)
Strategy primer	0.2549 (0/10)	0.6074 (2/10)	0.3044 (3/10)	0.2272 (3/10)
Spatial reflection	0.1103 (6/30)	0.0337 (28/30)	0.0133 (28/30)	0.0255 (28/30)
Number ordering	0.2332 (1/10)	0.1525 (5/10)	0.0424 (4/10)	0.1556 (2/10)

Table S19: TV permutation tests. Cells report mean unadjusted

p

, followed by the number of comparisons with Holm-adjusted

p\leq 0.05

out of the total.

Figure S11: Differences in mutual information across contexts and sources. a , Related minus unrelated MI for each source. b , Human minus model MI in each context. Points are equal-family mean differences over five Minesweeper and five Sudoku puzzles; bars are pointwise 95% bootstrap intervals (Section S5 ). Estimates use pairs with two valid answers; models use low effort and plain prompts.

	Related		Unrelated
Source	Mean $p$	$p<0.05$	Mean $p$	$p<0.05$
Human	0.0086	9/10	0.0083	10/10
GPT	0.8381	1/10	0.8002	2/10
Claude	0.3297	6/10	0.5391	2/10
Gemini	0.2254	6/10	0.3830	5/10

Table S23: Individual-puzzle MI permutation tests. Mean

p

averages ten unadjusted

p

-values per source and context (five puzzles per family); counts give the number below 0.05. Mean

p

is descriptive, not a combined

p

-value or a test of mean MI. Models use low effort and plain prompts; tests are described in Section S5 .

Explore similar work

Jun 25, 2026cs.CL

The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans

Humans flexibly adapt their reasoning strategies to the requirements of a given problem. Large language models (LLMs) have performed well on many cognitive tasks, however, it is unclear whether this accuracy is a result of pattern matching from training data or flexible reasoning. Here, we introduce a novel paradigm to test this question: the riddle riddle paradigm. Riddle riddles are word problems written to mimic popular riddles, but altered so their answers only require literal interpretations. Identifying correct answers requires looking past the structure of each question and flexibly apply different reasoning strategies based on the content. If LLMs respond to surface features, such as form, a riddle-like structure should cause models to use an inventive reasoning strategy even when a literal interpretation suffices. Alternatively, if LLMs reason based on content, they should flexibly switch strategies when appropriate. Across two experiments with nine state-of-the-art LLMs and 100 human participants, we show humans and LLMs fail on this paradigm in opposite directions. LLMs were far more accurate on genuine riddles than on riddle riddles (84.9% vs. 50.7%); whereas humans showed the reverse effect (50.5% vs. 80.5%). Error analysis shows that 90.8% of LLM errors on riddle riddles (the condition where they show diminished performance) were due to inappropriate use of inventive reasoning while only 57.6% of human errors on genuine riddles were due to overextending literal reasoning. Thus, while both groups make mistakes, reasoning mistakes are made more often by LLMs than by humans. Overall, LLMs' strong performance on genuine riddles may reflect memory retrieval rather than flexible strategy selection, and without stimuli designed to elicit this contrast, it becomes easy to conflate LLM-generated outputs that look like reasoning with genuine reasoning.

Bella Fascendini, Kathryn McGregor, Max D. Gupta +1

Department of Psychology, Princeton University · Department of Computer Science, Princeton University

LLM Reasoning Strategies Reasoning Skills Reasoning Errors Interpretation

Ask PDF

Nov 5, 2025cs.CL

MultiZebraLogic: A Multilingual Logical Reasoning Benchmark

We create high-quality datasets for LLM evaluation of logical reasoning skills across nine different languages, which have been manually checked by fluent speakers. The datasets consist of so-called zebra puzzles, and we analyse different ways of tuning the difficulty of the puzzles to fit modern LLMs. This includes the size of the puzzle (number of objects and number of clues), as well as a novel addition of red herring clues containing only irrelevant information. We show that presence of red herrings indeed makes the puzzles significantly harder for the models, and we find puzzle sizes 2x3 and 4x5 are sufficiently challenging for GPT-4o mini (a non-reasoning model) and o3-mini (a reasoning model), respectively. We analyse whether LLM performance of these are sensitive to the language, the cultural sensitivity of the puzzle theme, and the choice of clue types. These analyses are conducted with English and Danish, where we show that there is no significant difference for either of these three aspects, at least for the OpenAI models GPT-4o mini and o3-mini, chosen as representative non-reasoning and reasoning models, respectively. We publish the datasets for each of the nine languages for the identified sizes 2x3 and 4x5. We also publish the code used to generate the puzzles, which can be used to extend the benchmark into more languages.

Sofie Helene Bruun, Dan Saattrup Smart

The Alexandra Institute Rued Langgaards Vej 7, 5D, 2300 Copenhagen S, Denmark

Reasoning Benchmark Puzzle Reasoning Skills Large Language Model Evaluation Openai

Ask PDF

Sep 15, 2026cs.CY

AI and Human Approaches to Mathematical Problem Solving

AI systems have begun to report solutions, disproofs, and substantive advances on long-standing mathematical problems, raising questions about whether they approach research in the same way as mathematicians. This study compares public AI research accounts with the human literature on 11 such problems. The human corpus contains 58 papers that directly addressed the same mathematical targets later reported by AI sources as resolved, disproved, or substantially advanced; 31 within-problem comparisons were constructed from these materials. Six validated text-based measures capture problem resolution, method articulation, uncertainty and boundary specification, successor-question generation, generality, and cross-disciplinary integration. AI accounts place greater emphasis on resolving the focal problem and connecting ideas across fields. Human papers devote significantly more attention to explaining methods, specifying assumptions and limitations, and identifying questions for subsequent research. No precise difference is detected in generality. The estimated directions remain unchanged when each mathematical problem is removed in turn. The findings reveal two distinct research profiles: AI accounts concentrate on closing and recombining problems, whereas mathematical papers more extensively document the procedures, limits, and research opportunities through which results become cumulative knowledge. Evaluating research AI therefore requires attention to the organization of inquiry, not only whether a target is solved.

Yang Ding

University of Edinburgh Business School, Edinburgh EH8 9JS, UK. · Centre for Technomoral Futures, Edinburgh Futures Institute, University of Edinburgh, Edinburgh EH3 9EF, UK. · Manchester Institute of Innovation Research, Alliance Manchester Business School, University of Manchester, Manchester M13 9PL, UK.

Research-Level Mathematics Artificial Intelligence Research Open Problems Interdisciplinary Research Human-Ai Collaboration Human Agency+1

Ask PDF