Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require multi-step decisions to reach a solution. However, existing puzzle benchmarks typically focus on narrow environments with rigid mechanics and tightly constrained solution targets. To bridge this gap, we introduce the Active Probing Benchmark (APB), which generalizes puzzle-solving by casting concept identification as an interactive reasoning task that integrates specialized knowledge assessment into the puzzle setting. Specifically, APB evaluates whether models can acquire information via multi-turn active probing and inference to pinpoint 960 target concepts of specialized difficulty across eight diverse academic domains. Our evaluation reveals a counter-intuitive phenomenon: most open-source reasoning models significantly underperform their standard instruction counterparts. This finding suggests that in interactive settings, these models struggle to leverage procedural reasoning to reliably access and apply specialized domain knowledge. Beyond these empirical insights, APB provides a scalable pipeline for generating benchmarks that probe knowledge reasoning under interaction across diverse settings.
Figures & tables
Figure 1: Overview of the Active Probing Benchmark (APB) and representative performance results. (Left) Data construction : each puzzle specifies a target concept Ptarget drawn from a generalized knowledge source (e.g., Wikipedia or domain documents) along with a reference document Iref . (Middle) Multi turn active probing : the model under test iteratively asks multiple-choice queries Qt , receives feedback At from a knowledge oracle, and updates its hypotheses to reduce the candidate space until the target is identified or a turn budget is exhausted. (Right) Performance comparison : average success rates show that several open source “Thinking” models underperform their “No-Thinking” counterparts. Full results are reported in Table 2 .
Figure 2: Example episode from the Human Disease domain in APB, where the agent successfully identifies the target concept “Reticular dysgenesis”. Across turns, the agent acquires diagnostic evidence through queries and feedback, progressively narrowing the hypothesis set until the target is determined. The resulting interaction trace provides process-level evidence of how the model accesses and applies specialized knowledge.
Figure 3: The multi-stage, coarse-to-fine data filtering pipeline used to construct the Active Probing Benchmark from the English Wikipedia dump.
Group
N
Mean
Med.
Type
G1: Rejected (Stage 1)
1k
379.2
500 +
Common
G2: Rejected (Stage 4)
1k
86.7
19.0
Specialized
G3: Final (APB)
1k
59.3
16.0
Deep/Edge
Table 1: Topological Analysis of Concept Difficulty. In-link counts drop drastically, confirming the selection of “edge” knowledge.
Figure 4: Hierarchical distribution of puzzle targets across domains and sub-disciplines. Eight distinct color families represent the primary domains, while individual segments denote specific sub-topics. The area of each segment is proportional to the number of questions in that category. Exact distributions are provided in Appendix D .
Model
Bio
Dis
Chem
Phys
Math
CS
Econ
Phil
Avg.
GPT-5
34.2
37.5
33.3
34.2
34.2
25.0
39.2
30.0
33.4
Claude Opus 4
27.5
40.8
29.2
26.7
18.3
19.2
31.7
27.5
27.6
Claude Sonnet 4
29.2
35.0
23.3
23.3
17.5
18.3
42.5
21.7
26.4
o3
28.3
33.3
21.7
29.2
20.0
10.0
35.0
25.0
25.3
GPT-4o
26.7
24.2
20.0
17.5
11.7
10.8
25.0
20.8
19.6
Grok-4
14.2
30.8
16.7
13.3
10.0
13.3
20.8
21.7
17.6
Table 2: Success Rate (%) for various models across eight domains. The best performance in each domain is highlighted in bold . The bottom row, ’Domain Avg.’, summarizes the average performance for each domain.
Table 3: Detailed behavioral analysis metrics. PL : Probing Length, IG : Incorrect Guess rounds, IO : Incorrect Option rounds, IS : Incorrect Streak rounds, SR : Success Rounds, TR : Total Rounds. Blue bars visualize the normalized magnitude of each metric within its column to highlight behavioral patterns.
Figure 5: DeepSeek R1 reasoning trace for the target ’Reticular dysgenesis’ (Round 5). Top: The internal reasoning content shows the model brainstorming specific disease candidates following an incorrect guess. Middle: The extracted hypotheses visualize the diverse set of conditions considered during this step. Bottom: The generated follow-up question focuses on differentiating clinical features.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Synopsys
Cadence Innovus
Model
ICC
ICC2
Common
Legacy
Avg.
Claude Opus 4
27.5
14.2
16.7
22.5
20.2
GPT-5
25.8
15.0
14.2
20.0
18.8
grok-4
15.0
10.8
18.3
24.2
17.1
o3
20.0
15.0
12.5
12.5
15.0
Claude Sonnet 4
14.2
10.8
9.2
18.3
13.1
Appendix
Table 4: Success rates (%) on the IC Back-End Puzzle Benchmark. Tools are grouped by vendor: Synopsys (ICC, ICC2) and Cadence Innovus (Common, Legacy). “NT” denotes “No-Thinking”. Qwen3 models refer to the Qwen3-235B-A22B variants. Best results in each category are bolded .
Figure 6: Ranking consistency analysis (DeepSeek V3 vs. Claude Sonnet 4) across eight domains. Each point represents a model’s performance in a specific domain. The distinct clustering of model colors and the high Spearman’s rank correlation ( ρ ) within each domain confirm that relative model rankings remain robust despite the absolute score shift caused by the stricter judge.
Model
Configuration
Success Rate
Tier
Effect of Temperature (Open-Source Reasoning)
DeepSeek R1
Temp = 0.2 (Default)
6.7%
Bottom
Temp = 1.0 (Recommended)
8.3%
Bottom
Qwen3 Thinking
Temp = 0.2 (Default)
11.7%
Low
Temp = 0.6
10.8%
Low
Effect of Reasoning Effort (GPT-5)
Appendix
Table 5: Ablation results on sampling temperature and reasoning effort ( N=120 ). The “Tier” column reflects the relative ranking within the full benchmark context.
Model
QA
QA
QA Total
KO Comp.
Prompt
Comp.
Tok.
Tok.
(Best Performance)
GPT-5
9,821
24,301
34,122
430
(Most Verbosity)
GPT-OSS-120B
14,034
58,471
72,505
473
Qwen3 235B Think.
33,414
24,170
57,584
473
Appendix
Table 6: A summary of token usage for representative models. ”Comp.” stands for Completion, ”Tok.” for Tokens. KO output is low due to fixed formatting.
We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles, a family of constraint-satisfaction problems closely related to NP-complete problems, with deterministic, step-level verification. From a database of 62,231 puzzles across 94 varieties with verified unique solutions, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers in two modes: direct ask (single-shot) and agentic (multi-turn with iterative verification). A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints, localizing errors to the exact rule violated, providing the infrastructure for dense, per-move reward signals for process supervision and reinforcement learning. Our evaluation reveals two distinct axes of capability: (1) reasoning effort scaling, where GPT-5.2 improves 81x from no reasoning to maximum effort; and (2) agentic iteration, where Claude Opus 4.6 rises from 0.3% to 30.0% through iterative checking, while GPT-5.2@xhigh improves from 20.2% to 56.0%. Agentic attempts span a median of 29 turns over 17 minutes, with the longest exceeding 1,221 turns and 14.3 hours - a demanding test of long-context utilization, not just reasoning.
Contemporary large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these reasoning skills to more general and broader contexts--often termed general reasoning--remains under-explored. Unlike domain-specific reasoning, general reasoning relies less on expert knowledge but still presents formidable reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks. These results suggest that the reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code, Dataset, and Leaderboard: https://general365.github.io
Junlin Liu, Shengnan An, Shuang Zhou +10
University of Chinese Academy of Sciences · Meituan
Large language models perform well on many reasoning tasks, yet they often lack awareness of whether their current knowledge or reasoning state is complete. In non-interactive puzzle settings, the narrative is fixed and the underlying structure is hidden; once a model forms an early hypothesis under incomplete premises, it can propagate that error throughout the reasoning process, leading to unstable conclusions. To address this issue, we propose SABA, a reasoning framework that explicitly introduces self-awareness of missing premises before making the final decision. SABA formulates reasoning as a recursive process that alternates between structured state construction and obstacle resolution: it first applies Information Fusion to consolidate the narrative into a verifiable base state, and then uses Query-driven Structured Reasoning to identify and resolve missing or underspecified premises by turning them into queries and progressively completing the reasoning state through hypothesis construction and state refinement. Across multiple evaluation metrics, SABA achieves the best performance on all three difficulty splits of the non-interactive Detective Puzzle benchmark, and it also maintains leading results on multiple public benchmarks.
Fulong Fan, Peilin Liu, Fengzhe Liu +2
School of Software, Jilin University · School of Computer Science, Jilin University