Language Model Reasoning Evaluation

Follow this topic Ask about this topic

Latest papers

128 papers66 papers in the last four weeks, up 2100% on the four weeks before. 0.4% of all new papers.

Diagnostic Analysis for Large Language Model Reasoning
Oct 19, 2026Xinyu Liu, Dongqi Fu, Jun WuLanguage Model Reasoning Evaluation LLM Reasoning with Graphs
Michigan State University · Meta AI
Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization
Oct 19, 2026Yupeng Chang, Linyuan Gao, Zhiyuan Li +2Efficient Language Model Inference Early Stopping
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Understanding When Reflective Prompt and Skill Optimization Helps on Locally Served Reasoning Models: A Two-Model Study of GEPA and SkillOpt
Oct 19, 2026Andrea Giummarra, Carmelo Baglieri, Antonio Di Maria +3LLM Input Optimization Large Language Model-Guided Optimization
Evision srl, Ragusa, Italy · Department of Clinical and Experimental Medicine, University of Catania, Catania, Italy
ABACUS: Program-Guided Multistep Reasoning over Frozen Judgment Models
Oct 19, 2026Yuhao Wang, Zelin ZangStructured Reasoning in Language Models Efficient Language Model Reasoning
Westlake University · Hong Kong Institute of Science & Innovation, CAS · Tsientang Institute for Advanced Study
Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers
Oct 19, 2026Zhongyin Wang, Zhaoqian Xue, Junting Mu +4LLM Decision-Making Efficient Language Model Reasoning
University of Illinois Urbana-Champaign · University of Arizona · University of California, Irvine +3
Which Answers Agree? Evaluating Self-Consistency from Incomplete Logs
Oct 19, 2026Zhengtao Gui, Zexi Chen, Weining ShenSelf-Consistency in Language Models Language Model Reasoning Evaluation
University of California, Irvine
Assertion and Belief: Moore's Paradox in Language Models
Oct 18, 2026Anshun Asher Zheng, Weili Cao, Fangcong Yin +2Pragmatic Reasoning in Language Models Language Model Reasoning Evaluation
The University of Texas at Austin · Duke University · New York University +1
Stepwise Reasoning Verification: Don't Generate, Classify!
Oct 18, 2026Yifei Shi, Michalis Korakakis, Rami Aly +3Reasoning Verification Language Model Reasoning Evaluation
University of Cambridge · Alan Turing Institute · University of Edinburgh
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Oct 18, 2026Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1Retrieval-Augmented Reasoning LLM Reasoning
Aarhus University · Microsoft Research · Technische Universität Darmstadt
MAMath-Bench: Process-Aware and Robust Evaluation of Multimodal Multi-Agent Mathematical Reasoning
Oct 18, 2026Xinping Wang, Jie Ma, Rui Xing +6Mathematical Reasoning Benchmarks Language Model Reasoning Evaluation
School of Cyber Science and Engineering, Xi’an Jiaotong University · MOE KLINNS Lab, Xi’an Jiaotong University · School of Computer Science and Technology, Xi’an Jiaotong University +3
What Can't Jev Decide? Probing the Limits of One-Pass Decision Models
Oct 18, 2026Wang Yang, Xinpeng Li, Yanyan Zhang +8State Tracking in Language Models LLM Decision-Making
Case Western Reserve University · NII LLMC (Japan) · Georgia Institute of Technology +1
Verifying Human-Written Reasoning References
Oct 17, 2026Ali Ansari, Farhana Ferdousi LizaReasoning Verification Language Model Reasoning Evaluation
School of Computing Sciences, University of East Anglia
What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make
Oct 17, 2026Siyu Zhang, Yuquan Lu, Yoshua Y. Li +7RL for Language Model Reasoning LLM Reasoning
University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen
CoTs remain monitorable when being unfaithful requires computation: Replicating and Extending the Necessity Effect Across Eleven Models
Oct 17, 2026Arav Dhoot, Yixiong Hao, Zephaniah RoeFaithfulness of Language Model Reasoning Faithfulness of Language Model Explanations
Columbia University · Second Look Research
Distillation, Self-Training, and the Fragility of Learned Chemical Reasoning
Oct 16, 2026Anagha Aneesh, Sebastian Ibarraran, Frank Hu +2Chemical Reasoning in LLMs Chemical Reaction Prediction
Stanford University
Reasoning Models Can Control Their Chains of Thought When They Tell Themselves To
Oct 16, 2026Uriel Dolev, Idan Schwartz, Yoav GoldbergLanguage Model Reasoning Evaluation LLM Prompting
Bar-Ilan University · Allen Institute for AI
Not All Test-Time Compute Is Evidence: Model-Specific Bias in LLM Reasoning
Oct 16, 2026Ming Shen, Tianjun YaoTest-Time Scaling Efficient Language Model Reasoning
Shanghai Jiao Tong University · Shenzhen Loop Area Institute (SLAI)
Do Knowledge and Reasoning Oriented Benchmarks Induce Stable Representation Signatures in LLMs?
Oct 16, 2026Parsa Rahimi, Azim Dehghani AmirabadLLM Interpretability Language Model Probing
EPFL, Idiap Research Institute · Independent Researcher
Are Large Reasoning Models Good Everyday Reasoners? A Comparison with Non-Reasoning Models on Logical Reasoning Tasks
Oct 15, 2026Kentaro Ozeki, Risako Ando, Takanobu Morishita +3Language Model Reasoning Evaluation LLM Reasoning
Keio University
RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfer
Oct 14, 2026Yuqi Zhang, Di ZhangLanguage Model Reasoning Evaluation LLM Reasoning
Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
Sep 24, 2026Zeyan Li, Siyuan Qiu, Jianfeng XuVLM Evaluation Language Model Generation Evaluation
Shanghai Jiao Tong University
LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator
Sep 19, 2026Yiming Feng, Naihao Deng, Yulong Chen +1Reasoning Evaluation Language Model Reasoning Evaluation
University of Michigan · University of Aberdeen · University of Cambridge
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Sep 8, 2026Jaewoo Lim, Sungbok Shin, Sanghyun HongLLM Evaluation LLM Interpretability
Oregon State University, Corvallis USA · Sogang University, Seoul, South Korea
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Aug 31, 2026Qiyao Yan, Chenpeng Wang, Liangming PanLanguage Modeling Language Model Calibration
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
Aug 30, 2026Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5Implicit Reasoning in Language Models Language Model Probing
1Stanford University · 2York University · 3Facebook
ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning
Aug 18, 2026Stefano GoriaLLM Reasoning with Graphs Language Model Reasoning Evaluation
AIM Research Lab
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
Aug 18, 2026Ayoub Kirouane, Christos PetrocheilosFine-Tuning Language Models Structured Reasoning in Language Models
Sophea AI, KIEFER SA, Athens, Greece
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM Evaluation LLM Auditing
School of Statistics and Data Science, Southwestern University of Finance and Economics
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM Evaluation Language Model Generation Evaluation
Federal University of Santa Catarina (UFSC) Brazil
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University

Oct 19, 2026J/K move · Enter open · S save

Diagnostic Analysis for Large Language Model Reasoning

Xinyu Liu, Dongqi Fu, Jun Wu

Michigan State University · Meta AI

Language Model Reasoning Evaluation LLM Reasoning with Graphs Multi-Hop QA

Open paper AskPDF arXiv

Reasoning structures (e.g., chains and graphs) are increasingly used for probabilistic inference, factuality verification, and uncertainty estimation, but their structural reliability is rarely evaluated directly. We test whether this reliability holds through a proposed two-stage diagnostic evaluation in open-ended multi-hop QA. Stage 1 tests different capabilities required to construct reasoning graphs, showing that graph reliability varies with topology, prompt design, and model family. Stage 2 uses uncertainty estimation as a downstream test case. Generated-edge experiments show that graph-based computation is sensitive to upstream graph errors, with performance dropping relative to the gold-graph setting. Gold-graph experiments further show that graph structure provides little benefit when used only as prompt context, but improves the separation of correct from incorrect answers by confidence score and enables node-level diagnostics.