Scientific Reasoning in Language Models

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 34

All topics

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
May 28, 2026A. J. Lew, Y. Cao, M. J. BuehlerLLM Evaluation Scientific Reasoning
Unreasonable Labs · Unreasonable Labs Mountain View, CA 94043, USA
In-Context Examples Suppress Scientific Knowledge Recall in LLMs
Apr 30, 2026Chaemin Jang, Woojin Park, Hyeok Yun +2Knowledge Conflicts in Language Models Scientific Reasoning
Korea Advanced Institute of Science and Technology · Shanghai Jiao Tong University
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models
Apr 20, 2026Seyedali Mohammadi, Manas Gaur, Francis FerraroLLM Evaluation Scientific Reasoning in Language Models
University of Maryland, Baltimore County, MD, USA

May 28, 2026, cs.AIJ/K move · Enter open · S save

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

A. J. Lew, Y. Cao, M. J. Buehler

Unreasonable Labs · Unreasonable Labs Mountain View, CA 94043, USA

LLM Evaluation Scientific Reasoning Scientific Hypothesis Generation Research Idea Generation Scientific Reasoning in Language Models Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Scientific discovery is an inherently creative and uncertain process, requiring reasoning beyond the recall of known knowledge. While many benchmarks have been proposed to evaluate large language model (LLM) performance on deep research tasks via multi-hop retrieval, their innovative reasoning abilities essential for true scientific discovery remain largely untested. We introduce a benchmark framework for evaluating model performance in scientific discovery and reasoning, building up from a raw problem to the classical null hypothesis test. In our framework, models initially receive only the topic and research question from a recent paper, with technical details progressively revealed. At each stage of information disclosure, the model is tasked with generating hypotheses that address the research question, which is compared with the conclusions from the original paper and evaluated via automated semantic similarity of constituent atomic claims. This progressive evaluation of semantic divergence from ground-truth conclusions enables assessment of a model's innovativeness (under minimal information) to grounded reasoning capabilities (under full experimental details), both critical for using LLMs for scientific discovery purposes. Our framework provides a foundation for systematically evaluating scientific reasoning and discovery capabilities in LLMs, crucial for advancing the development of next-generation AI scientist/co-scientist systems. Specifically, here we evaluate GPT-5, GPT-5.4, Gemini 2.5 pro, and Gemini 3.1 pro preview across 45 papers spanning bioactive materials, mechanical materials, and nanomaterials. We find that GPT-5.4 and Gemini 3.1 pro outperform their previous generation counterparts as expected, and GPT-5.4 in particular maintains 0.7 F1 score alignment with ground truth conclusions even under minimal context.

Momentum

Related topics

Latest papers 34

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

In-Context Examples Suppress Scientific Knowledge Recall in LLMs

AI scientists produce results without reasoning scientifically

Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models