Contemporary large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these reasoning skills to more general and broader contexts--often termed general reasoning--remains under-explored. Unlike domain-specific reasoning, general reasoning relies less on expert knowledge but still presents formidable reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks. These results suggest that the reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code, Dataset, and Leaderboard: https://general365.github.io
Figures & tables
Figure 1: Performance of various LLMs on General365 . Gemini-3-Pro achieves state-of-the-art performance with 62.8%, while the majority of models fail to reach the 60% passing standard.
Figure 2: The construction pipeline of General365 .
Figure 3: Statistical Overview of General365 . (a) Distribution of challenge categories, highlighting the diversity of problems within the benchmark. (b) Distribution of multi-label problems, showcasing the complexity in task assignments.
Model
Complex Constraints
Branching & Enumeration
Spatial & Temporal
Recursive & Backtracking
Semantic Interference
Implicit Information
Optimal Strategy
Probability & Uncertainty
Overall
Reasoning Models
Gemini-3-Pro
65.3%
64.3%
57.6%
66.8%
55.7%
68.8%
50.8%
54.5%
62.8%
Gemini-3-Flash
62.9%
64.5%
56.5%
69.3%
55.7%
62.7%
48.3%
60.2%
60.8%
GLM-5-Thinking
63.5%
69.1%
52.9%
68.2%
49.2%
55.4%
51.7%
56.8%
59.9%
GPT-5-Thinking
61.7%
65.4%
48.6%
65.2%
57.2%
58.5%
47.5%
59.1%
58.6%
GPT-5.1-Thinking
61.1%
62.9%
48.6%
64.4%
53.4%
61.5%
47.5%
47.7%
58.2%
Table 1: Accuracy of General365 across various challenge categories for different models.
Figure 4: The relationship between accuracy and average output tokens for various LLMs, highlighting the reasoning efficiency on the General365 benchmark. Gemini-3-Pro exhibits superior reasoning efficiency compared to other frontier models.
Figure 5: Radar chart of various LLMs series across eight challenge categories.
Figure 6: T-SNE visualization of query embeddings for General365 and comparative benchmarks. (a) General365 (1460 instances) shows a highly dispersed and uniform distribution, highlighting the semantic independence achieved through our seed problems. (b) BBH (6511 instances) and (c) BBEH (4520 instances) exhibit significant "local collapse" with isolated, high-density clusters.
Figure 7: The distribution of reasoning similarity scores across benchmarks shows a clear right-skewed distribution for BBH and BBEH, while General365 exhibits a left-skewed distribution.
Figure 8: Comparative performance across benchmarks, highlighting the significantly higher difficulty floor of General365 .
Figure 9: The relationship between accuracy and average output length varies significantly across benchmarks: the fact that models yield substantially longer outputs on General365 despite lower accuracy underscores the benchmark’s rigorous demand for deep reasoning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model Name
Acc (%)
Acc (Public Subset) (%)
Gemini-3-Pro
62.8
61.3
Gemini-3-Flash
60.8
59.7
Gemini-2.5-Pro
48.7
50.0
Gemini-2.5-Flash
39.6
38.2
GPT-5.1-Thinking
58.2
57.1
GPT-5-Thinking
58.6
56.0
Appendix
Table 2: Performance of various LLMs on the complete General365 benchmark and its publicly accessible subset. The closely aligned scores demonstrate that the public subset reliably reflects models’ overall capabilities on the full benchmark.
Model Name
Rep 1
Rep 2
Rep 3
Rep 4 (Acc)
Δ
DeepSeek-V3.2-Thinking
53.4
54.9
55.1
54.9
1.7
Kimi-K2-Thinking
51.5
53.2
53.0
53.0
1.7
GLM-4.7-Thinking
57.0
55.4
57.4
57.4
2.0
LongCat-Flash-Thinking-2601
50.7
50.9
51.6
50.8
0.9
LongCat-Flash-2512
40.4
43.0
41.0
41.7
2.6
Appendix
Table 3: Stability analysis of representative models on the General365 across four independent evaluation runs. We report the accuracy (%) for individual repetitions and the maximum deviation ( Δ = max-min) between runs.
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.
Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.
Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing benchmarks are limited by synthetic settings and data contamination. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning built from recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns. The benchmark introduces a thirteen-category logical taxonomy of theorem types (e.g., implication, equivalence, existence, uniqueness), enabling fine-grained evaluation across reasoning forms. It employs a proof-sketch-guided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices reflecting misleading proof directions, increasing sensitivity to genuine understanding over surface-level matching. We also introduce a substitution-resistant mechanism to distinguish answer recognition from substantive reasoning. Evaluation shows the benchmark is far from saturated: Gemini-3.1-pro-preview, the best model, achieves only 43.5%. Under substitution-resistant evaluation, accuracy drops sharply: GPT-5.4 scores highest at 30.6%, while Gemini-3.1-pro-preview falls to 17.6%, below the 20% random baseline. A dual-mode protocol reveals that proof-sketch access yields consistent accuracy gains, suggesting models can leverage high-level proof strategies for reasoning. Overall, LiveMathematicianBench offers a scalable, contamination-resistant testbed for studying research-level mathematical reasoning in LLMs.
Linyang He, Qiyao Yu, Hanze Dong +5
Columbia University · Microsoft Research · University of Amsterdam