cs.AIMar 17, 2026

LLMs learn scientific taste from institutional traces across the social sciences

Authors: Ziqin Gong, Ning Li, Huaikang Zhou

Organizations: School of Economics and Management, Tsinghua University, Beijing, China

Abstract

Reinforcement-learned reasoning has powered recent AI leaps on verifiable tasks, including mathematics, code, and structure prediction. The harder bottleneck is evaluative judgment in low-verifiability domains, where no oracle anchors reward and the core question is which untested ideas deserve attention. We test whether institutional traces, the record of what fields published, where, and at which tier, can serve as a training signal for AI evaluators. Across eight social science disciplines (psychology, economics, communication, sociology, political science, management, business and finance, public administration), we built held-out four-tier research-pitch benchmarks and supervised-fine-tuned (SFT) LLMs on field-specific publication outcomes. The fine-tuned models cleared the 25 percent chance baseline and exceeded frontier-model performance by wide margins, with best single-model accuracy ranging from 55.0 percent in public administration to 85.5 percent in psychology. In management, evaluated against 48 expert gatekeepers, 174 junior researchers, and 11 frontier reasoning models, the best single fine-tuned model (Qwen3-4B) reached 59.2 percent, 17.6 percentage points above expert majority vote (41.6 percent, non-tied) and 28.1 percentage points above the frontier mean (31.1 percent). The fine-tuned models also showed calibrated confidence: confidence rose when predictions were correct and fell when wrong, mirroring how a skilled reviewer can say "I'm sure" versus "I'm guessing." Selective triage on this signal reached very high accuracy on the highest-confidence subsets in every field. Institutional traces, we conclude, encode a scalable training signal for the low-verifiability judgment on which science depends.

Explore similar work

CardsList
  1. When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

    Sep 17, 2026Sy-Tuyen Ho, Minghui Liu, Hieu Tran +1Automated Peer ReviewLLM Reliability

  2. Contemporary AI lacks the imagination to diverge or negate in science

    Jun 6, 2026Honglin Bao, Siyang Wu, Xiao Liu +3LLM-as-a-JudgeScientific Reasoning

  3. SciΠΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

    Oct 15, 2026Youran Sun, Xingyu Ren, Chugang Yi +15LLM Agent EvaluationAI Agent Benchmarks