Language Model Reasoning Evaluation

Latest papers

128 papers66 papers in the last four weeks, up 2100% on the four weeks before. 0.4% of all new papers.

Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model
Nov 6, 2026Mohammed Nayeem Khan, Logesh LInput Order Sensitivity in Language Models Multiple-Choice Question Answering
Intel Corporation
KLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation
Nov 6, 2026Seoyeon Kim, Jeonghyun Kang, Hamin Koo +3LLM Evaluation Language Model Reasoning Evaluation
Yonsei University · DATUMO INC.
Does Reasoning Improve Psychological Depth in Large Language Models?
Nov 6, 2026Ruichen Zheng, Yihe Wang, Fabrice Harel-Canada +4Text Generation Language Model Reasoning Evaluation
University of California, Los Angeles · Now at Arizona State University. · Cornell University +2
Aspect-wise Pairwise Tournaments for Test-Time Selection
Nov 5, 2026Hannah Guan, Yilun DuAutomated Evaluation Language Model Reasoning Evaluation
Harvard University
Benchmarking Large Language Models on Heterogeneous Cognitive and Biomedical Knowledge Graphs: The Role of Graph Grounding in Multi-Hop Reasoning and Hallucination Mitigation
Nov 5, 2026Ali Sarabadani, Kheirolah Rahsepar Fard, Tahereh JeiraniLLM Reasoning with Graphs Language Model Reasoning Evaluation
Department of Computer Engineering and Information Technology, University of Qom, Qom, Iran. · Master of Science in Positive psychology, Qom, Iran
TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research
Nov 5, 2026Yifan Luo, Zhengying Liu, Yang Yuan +2Language Model Reasoning Evaluation AI Agent Benchmarks
Tsinghua University · Moonshot AI · Shanghai Qizhi Institute
Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge
Nov 4, 2026Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor +1Language Model Reasoning Evaluation Multilingual Language Model Evaluation
Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University · Universitat Pompeu Fabra
Reasoning-Trace Overflow in Code Generation: An Empirical Study of Explicit Reasoning Models
Nov 4, 2026Lukas Twist, Helen Yannakoudakis, Jie M. ZhangLLM Code Generation Efficient Language Model Reasoning
King’s College London, London, UK
When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing
Nov 3, 2026Teng-Ruei ChenLLM Routing Language Model Reasoning Evaluation
Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Krixvon, Taipei, Taiwan
Beyond Answers, Traces, and Efficiency: Evidence for Adaptive Reasoning
Nov 2, 2026Sejin Kim, Sundong KimLanguage Model Reasoning Evaluation Adaptive Reasoning in Language Models
Chungnam National University · Gwangju Institute of Science and Technology
Chain of Faith: Policy-Internal Rewards for Reasoning Reliance
Nov 2, 2026Fanyu meng, Zhaodan Kong, Xin LiuFaithfulness of Language Model Reasoning RL for Language Model Reasoning
University of California, Davis
Solver-Like Behavior in Large Language Models as Formalizers
Nov 2, 2026Rikhil Amonkar, Stuti Mohan, Kevin Nguyen +3Language Model Reasoning Evaluation Constraint-Guided Reasoning in Language Models
MemShift: Separating Detection from Repair in LLM-Derived Personal Memory
Nov 2, 2026Zichuan Xu, Haochen Zhang, Yuyang Dong +2Memory-Augmented Language Models Language Model Error Detection
Osaka University · Initial S · Nagoya University
Reasoning is topologically simple
Nov 2, 2026Chris Kapulkin, Nathan KershawLanguage Model Reasoning Evaluation AI Safety Evaluation
College of Connected Computing & Department of Mathematics, Vanderbilt University · Department of Mathematics, University of Western Ontario
Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation
Nov 1, 2026Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh +2Language Model Self-Assessment Confidence Estimation in Language Models
AWS Generative AI Innovation Center
Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models
Oct 31, 2026Zhengshuai Yang, Changyun LiAutomatic Prompt Optimization Large Reasoning Models
College of Information and Intelligent Science and Technology, Hunan Agricultural University, Changsha 410128, China
Rethinking LLM Pruning via Test-Time Scaling
Oct 31, 2026Mingyang Song, Mao ZhengLLM Pruning Efficient Language Model Reasoning
Foundation Model Department, Tencent, China
CogRSI: Evolving and Auditing Cognitive Reasoning Strategies
Oct 31, 2026Yandan zheng, Haoran Luo, Xinliang Zhou +1Adaptive Reasoning in Language Models Theory of Mind
Nanyang Technological University, Singapore · Stanford University · VinUniversity, Vietnam
Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark
Oct 31, 2026Emma Graham, Richard GrangerLanguage Model Reasoning Evaluation Test-Time Scaling
Dartmouth College Hanover, NH 03755
Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning
Oct 31, 2026Charles Chang, Yuen Yuen Ang, Bolun SunPolicy Evaluation Language Model Reasoning Evaluation
Duke Kunshan University · Johns Hopkins University · Northwestern University
GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs
Oct 30, 2026Minh-Nam Tran, Abulhair SaparovLong-Context Reasoning in Language Models Language Model Reasoning Evaluation
Department of Computer Science, Purdue University, West Lafayette, IN 47907, USA
Early Stopping in Large Reasoning Models: A Survey and Empirical Audit of Efficiency and Reliability
Oct 30, 2026Quang Hung Nguyen, Hanh Nguyen, Thanh Tam Nguyen +3Early Stopping for Language Model Reasoning Efficient Language Model Reasoning
VinUniversity, Vietnam · Griffith University, Australia · VinUni Bigdata Research Institute, VinUniversity, Vietnam
ImplicitHop: Evaluating knowledge integration across context and memory
Oct 30, 2026Alexander Detkov, Dashiel Karish, Matthew Oleynikov +1Knowledge Retrieval in Language Models Language Model Reasoning Evaluation
California Institute of Technology
CIRCE Divines LLM Task Complexity from Constraint Structure
Oct 30, 2026Chenhao Hong, Jiakai Xu, Kaushiki Singh +2Language Model Performance Prediction Language Model Reasoning Evaluation
Columbia University
Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures
Oct 29, 2026Helen Jenne, Davis Brown, Jesse He +1LLM Mathematical Reasoning AI-Assisted Mathematical Discovery
Pacific Northwest National Laboratory · University of Pennsylvania · University of California San Diego +1
H $^2$ R: Hierarchical Hint-Based Reward for Mathematical Reasoning via Reinforcement Learning
Oct 29, 2026He Tu, Siye Li, Hongyi WenLLM Mathematical Reasoning Reward Shaping
New York University Shanghai
RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories
Oct 29, 2026Tidor-Vlad PricopeLanguage Model Reasoning Evaluation Mathematical Reasoning Benchmarks
Poindexter Labs
TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics
Oct 28, 2026Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad +1LLM Evaluation Language Model Reasoning Evaluation
Department of Mechanical Engineering Boston University Boston, MA 02215 · Department of Mechanical Engineering Division of Systems Engineering Boston University Boston, MA 02215
Deogen: Automatic Benchmark Generation for Reasoning over Policies
Oct 28, 2026Sam Kouteili, William Liang, Keeran Murray +4Synthetic Benchmark Generation Language Model Reasoning Evaluation
Yale University · Leibniz AI, Inc. · Yale Law School
Does Chain-of-Thought Reliance Follow a Scaling Law?Larger Models Follow Their Steps More, Independent of Post-Training
Oct 28, 2026Esther XinFaithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Independent Researcher

Nov 6, 2026J/K move · Enter open · S save

Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model

Mohammed Nayeem Khan, Logesh L

Intel Corporation

Input Order Sensitivity in Language Models Multiple-Choice Question Answering Language Model Bias Evaluation Language Model Reasoning Evaluation Non-Autoregressive Text Generation

Open paper AskPDF arXiv

Laya answers typed multiple-choice questions in one forward pass by scoring a hidden state at one [MASK] marker per option. We re-engineer the checkpoint (421M parameters; a 2-layer option head with no positional tensor; a documented but unused parallel layout) and test whether the answer depends on where an option is listed. Presenting 951 Qwen-verified questions under every cyclic rotation of their options (or six random permutations for 12 options), the released checkpoint changes its answer for 25.8% [22.9-28.7] of questions (19.7% on the 605 English-language items alone); accuracy with the gold option first versus last differs by 10.1 points, and the flips concentrate where the model's own confidence is low (54% versus 8% across confidence terciles). Fine-tuning with Laya's trainer on 2,818 teacher-verified rows with option shuffling cuts the flip rate to 7.2%; the identical rows without shuffling give 11.4%, so shuffling is responsible for 4.2 points of the improvement. A confidence-weighted vote over orderings gives 69.7% against 69.1% for a single base pass at 4.1x the compute. Labels are LLM-made and the test sets are teacher-filtered. We also fine-tune with the authors' parallel layout on the same rows: flip rate 0.5%.