Scientific Reasoning in Language Models

Follow this topic Ask about this topic

The field studies whether language models can do more than recall scientific facts: interpret data, connect evidence, design tests, explain mechanisms, and generate useful hypotheses. Common approaches include chain-of-thought and structured reasoning traces, retrieval and tool use, multi-agent collaboration, and training with expert examples, simulators, or reinforcement learning with verifiable rewards. Evaluation spans scientific question answering, mathematics and physics problems, papers and tables, multimodal data, and interactive experimental workflows. Models are improving on specialized tasks, but reliable evidence use, robust execution, and mechanistic understanding remain difficult, and correct answers do not always show that a model reasoned scientifically.

Latest papers

144 papers40 papers in the last four weeks, up 233% on the four weeks before. 0.2% of all new papers.

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Aug 25, 2026Ruoran Xu, Wending Gao, Liyunfeng Chen +5Physical Reasoning Scientific Reasoning in Language Models
Xi’an Jiaotong-Liverpool University
EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
Aug 25, 2026Mingxu Zhang, Ying Sun, Yuhan Li +4LLM Evaluation Scientific Reasoning in Language Models
The Hong Kong University of Science and Technology (Guangzhou) · The 63rd Research Institute, National University of Defense Technology · Nanjing University of Aeronautics and Astronautics
AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions
Aug 25, 2026Manmeet Singh, Somnath Luitel, Prabhjot Singh +3Forecasting Benchmarks Language Model Generation Evaluation
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.
BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification
Aug 24, 2026Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone +1LLM Evaluation Benchmark Design
Benchling
Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking
Aug 24, 2026Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang HouAutomated Fact-Checking Scientific Reasoning in Language Models
Interdisciplinary Transformation University Austria (ITU)
What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation
Aug 24, 2026Ziyue Wang, Aomufei Yuan, Yiran Yao +10LLM Evaluation Language Model Generation Evaluation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Peking University · Tianjin University +2
Ethical LLM-Assisted Research: A Framework for Responsible Delegation, Verification, and Epistemic Value
Aug 24, 2026Kalin Stoyanov, Nina IlievaAI-Assisted Scientific Research Human Oversight of AI
Department of Automation, University of Chemical Technology and Metallurgy
Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses
Aug 18, 2026Alona Strugatski, Licol Zeinfeld, Jason Cooper +3LLM Evaluation LLM Interpretability
Weizmann Institute of Science
SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models
Aug 18, 2026Sarvesh Gharat, Junpei KomiyamaAI Agents for Scientific Discovery Research Idea Generation
C-MInDS, IIT Bombay · Machine Learning Department, MBZUAI
Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
Aug 17, 2026Shaolong Chen, Yanlin Fei, Nazhou Liu +7Scientific Reasoning in Language Models Large Language Model-Guided Scientific Discovery
Prentis AI, San Francisco, CA · Stanford University, Stanford, CA · Titan Holdings, San Francisco, CA
LLMs Don't Pay for the Jump
Aug 14, 2026Paras Balani, Subhrakanta PandaAbductive Reasoning in Language Models Abductive Reasoning
Department of Mathematics and Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India · Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
Aug 14, 2026Jennifer D'Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade +9Scientific Figure Understanding Multimodal Model Evaluation
Data Science and Digital Libraries, TIB Leibniz Information Centre for Science and Technology, Hannover, Germany · Department of Applied Physics and Science Education, Eindhoven University of Technology, Eindhoven, Netherlands · Department of Biology, Chemistry, and Pharmacy, Freie Universität Berlin, Berlin, Germany +5
TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
Aug 11, 2026Valentin Rodionov, Shamil AssylbekovLLM Evaluation LLM Reliability
Intellicat Cleveland, OH 44106
Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Aug 7, 2026Taolin Han, Yuchen Zhang, Jinghang Wang +22Scientific Reasoning Scientific Discovery
1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3* +28
Divergence Decoding: Training-Free Capability Fusion
Jul 28, 2026Yimi Wang, Hao Li, Shuo Yang +6LLM Routing Scientific Reasoning in Language Models
Peking University · International Digital Economy Academy (IDEA) · The University of Hong Kong
Chemical Chain-of-Thought Functions as a Hallucination-Prone Molecular Scratchpad
Jul 23, 2026Jiatong Li, Yuxuan Ren, Weida Wang +2CoT Faithfulness Chemical Reasoning in LLMs
Blue Whale Lab, National University of Singapore · Fudan University · Hong Kong Polytechnic University
Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Model
Jul 22, 2026Markus J. BuehlerMechanistic Interpretability Scientific Reasoning in Language Models
Department of Civil and Environmental Engineering · Department of Mechanical Engineering · Schwarzman College of Computing +1
Though Language Models Err While They Strive: Conformal Prediction for Self-Correcting Scientific Generation
Jul 18, 2026Mingqiao Mo, Yunlong Tan, Hao ZhangLLM Self-Correction Scientific Reasoning
University of Chinese Academy of Sciences
Consensus as Privileged Context for Label-Free Self-Distillation
Jul 15, 2026John Gkountouras, Josip Jukić, Ivan TitovSelf-Training for Language Models Language Model Distillation
1ILLC, University of Amsterdam · 2ILCC, University of Edinburgh
Evidence-Informed LLM Beliefs for Continual Scientific Discovery
Jun 28, 2026Dhruv Agarwal, Reece Adamson, Andrew McCallum +3Scientific Hypothesis Generation Belief Updating in LLMs
University of Massachusetts Amherst · Allen Institute for AI
SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks
Jun 14, 2026Jingru Guo, Xiangyuan Xue, Lian Zhang +6LLM Agent Orchestration Multimodal Reasoning
1Imperial College London · 2The Chinese University of Hong Kong · University of Illinois Urbana-Champaign +4
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2
DN-Hypo-Pipeline: An AI-Driven Workflow for Generating Hypotheses using Large Language Models and Scientific Explanations
Jun 7, 2026Lei Lin, Ronghao Wang, Chunbao Zhou +2Scientific Reasoning Scientific Hypothesis Generation
Computer Network Information Center, Chinese Academy of Sciences, China
FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
Jun 3, 2026Leonardo Bertolazzi, Katya Tentori, Raffaella BernardiLLM Evaluation AI Agent Benchmarks
University of Trento · Free University of Bozen-Bolzano
SocraticPO: Policy Optimization via Interactive Guidance
Jun 3, 2026Zirui Liu, Jie Ouyang, Qi Liu +8RL for Language Model Reasoning LLM-Guided RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd
Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches
May 31, 2026Teddy Ferdinan, Bartłomiej Koptyra, Mikołaj Langner +42AI for Science Scientific Reasoning
ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
May 28, 2026A. J. Lew, Y. Cao, M. J. BuehlerLLM Evaluation Scientific Reasoning
Unreasonable Labs · Unreasonable Labs Mountain View, CA 94043, USA
In-Context Examples Suppress Scientific Knowledge Recall in LLMs
Apr 30, 2026Chaemin Jang, Woojin Park, Hyeok Yun +2Knowledge Conflicts in Language Models Scientific Reasoning
Korea Advanced Institute of Science and Technology · Shanghai Jiao Tong University
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models
Apr 20, 2026Seyedali Mohammadi, Manas Gaur, Francis FerraroLLM Evaluation Scientific Reasoning in Language Models
University of Maryland, Baltimore County, MD, USA

Aug 25, 2026J/K move · Enter open · S save

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi+4

Xi’an Jiaotong-Liverpool University

Physical Reasoning Scientific Reasoning in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.