cs.LGOct 28, 2026

Automatic Construction of Strong Simple Baselines

Authors: Caleb N. Ellington, Jiaqi Wang, Sohan Addagudi, Elijah Cole

Organizations: Carnegie Mellon University · Sunday Scientific · University of Washington

Abstract

To rigorously establish the value of a novel deep learning method, it must be compared against appropriate baselines. As deep learning has become increasingly common in the physical and life sciences, there have been numerous cases in which apparent progress has turned out to be an artifact of omitted or missing baselines. In this work, we investigate whether this problem can be mitigated by using large language models to automate the construction of baselines. We assemble 36 supervised classification and regression tasks from peer-reviewed studies in Nature, Science, and Cell-family journals that investigate novel deep learning methods for scientific applications. We then use a language model to construct simple baselines for each task. These automatically constructed baselines are strong, recovering a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the score of the deep learning model. The median simple baseline matches or exceeds the performance of the deep learning model in 12 studies. We further examine the variation across scientific domains and publication venues and we characterize the effects of test-time compute and access to different levels of study context. As human peer review has struggled to keep up with the rapidly increasing volume of research articles in recent years, our results suggest that automated baseline construction is a practical and scalable check on deep learning contributions in the physical and life sciences.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

    May 28, 2026Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +6AI Agent BenchmarksLLM Evaluation

  2. SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

    May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for ScienceLLM Evaluation

  3. LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

    Jun 23, 2026Thi Huyen Nguyen, Zahra AhmadiScholarly Peer ReviewLLM Security