cs.CLNov 17, 2026

Language Models Make New Research Sound Old

Authors: Guy Zyskind

Organizations: University of Miami

Abstract

Researchers are increasingly using large language models (LLMs) to validate research questions. We find that LLMs make open research questions sound partly studied and overstate what related papers establish. Models recognize most of these overstatements when asked to check the papers' full texts separately. On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers. All six frontier systems show reporting gaps of 30-90 percentage points. Across 22 models given identical papers, higher capability is associated with more accurate judgments and larger reporting gaps. This reporting failure is asymmetric: models correctly identify studied questions in 88-100% of ordinary answers when we supply the answering papers. To correct this, we build a fixed answer from each model's judgment and separate checks of cited papers. This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems. Finally, we predict reviewers' novelty grades by combining several models' written ratings with a rule learned from graded examples, reaching the performance range reported by Think-Probe-Respond. Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods

    Jun 15, 2026Francesca Carlon, Brecht Verbeken, Vincent Ginis +1LLM EvaluationAI-Assisted Scientific Research

  2. What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation

    Aug 24, 2026Ziyue Wang, Aomufei Yuan, Yiran Yao +10LLM EvaluationLanguage Model Generation Evaluation