Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

Hui Mao

cs.CEAug 17, 2026

Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

Authors: Hui Mao

Organizations: Independent Researcher

Abstract

Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future corpus then determines whether each question was subsequently answered, partially addressed, independently posed, or ignored, and whether its underlying premise was supported or refuted. The protocol is model-agnostic: any system that emits frozen questions can be scored. We release reproducible astronomy instances with temporally isolated corpora, frozen questions, auditable labels, four reference baselines, and a submission interface. Two findings result. First, evidence-structure-first generation outperforms LLM-only prompting: across a generator decomposition crossed with a four-cutoff stress test (2010-2024, 798 judged questions) whose last window postdates model training, LLM-only generation shows memorized relevance without specific foresight, while a generator using no model weights at all finds questions whose premises the future refutes in every era. Second, a seven-rater agreement study (two blinded human annotators, five judge models, 90 items) indicts the outcome taxonomy rather than the judge: two careful humans agree at kappa = 0.17, every judge model agrees with the professional annotator as well or better (0.17-0.26), and frontier models agree with one another at 0.60 -- certifying an LLM judge by model-model agreement would have overstated its reliability threefold. A prospective instance -- 200 questions frozen 2026-08-17, scored 2027-2030 -- is released so the central claims become contamination-free tests that time itself will grade.

Figures & tables

Panel of Figure 1 — Figure 1: The historical backtesting protocol. No question generator, evidence graph, or particular LLM appears in the loop: the protocol evaluates frozen question lists , whatever produced them.

Dimension	Label	Meaning
outcome	answered	Future evidence substantially answers the question (including by refuting its premise)
	partially_addressed	Important, directly relevant progress; core question unresolved
	posed_but_open	The community independently poses essentially the same question without resolving it
	not_addressed	No meaningful follow-up in the future corpus
premise_status	supported	Future evidence confirms the underlying premise
	refuted	Future evidence falsifies the underlying premise

Table 1: Outcome taxonomy v1.0. The two dimensions are labeled independently.

Metric	Definition
Coverage (future attention rate)	$\frac{1}{n}\,\|\{q:o(q)\neq\texttt{not\_addressed}\}\|$ — did future science engage the question at all?
Answer rate	$\frac{1}{n}\,\|\{q:o(q)=\texttt{answered}\}\|$
Partial rate	$\frac{1}{n}\,\|\{q:o(q)=\texttt{partially\_addressed}\}\|$
Open rate	$\frac{1}{n}\,\|\{q:o(q)=\texttt{posed\_but\_open}\}\|$ — the community independently recognized the question
Premise refutation rate	$\frac{1}{n}\,\|\{q:p(q)=\texttt{refuted}\}\|$ — questions that led to an established conclusion being overturned
Evidence strength	mean $\|E(q)\|$ over engaged questions, and the multi-source rate $\frac{1}{n}\,\|\{q:\|E(q)\|\geq 2\}\|$ — is the label supported by multiple independent papers?

Table 2: Benchmark metrics v1.0. All are computed by benchmark/metrics.py from the released annotation records.

Historical cutoff	2020-12-31
Domain scope	atmospheric composition in transmission spectra; cloud/haze degeneracies; instrument systematics
Corpus A (past)	2,512 deduplicated papers, 2015–2020 (NASA ADS; 500-paper full-text core)
Corpus B (future)	1,891 unique papers, 2021–2026, temporally isolated
Questions	10, frozen, ranked, with pre-cutoff source evidence
Retrieval	text-embedding-3-small , cosine, top-8; records released
Judge	gpt-4.1 , temperature 0, citation-constrained; human-adjudicated

Table 3: Astronomy v1 at a glance. Corpus manifests, frozen questions, retrieval records, and adjudicated labels are all released.

System	Eval	Cov.	Ans.	Part.	Open	Ref.	$s_{1}$
evidence_graph_v1 ( Scientific Question Discovery project, 2026 )	adjud.	100%	20%	70%	10%	10%	0.666
evidence_graph_v1 (rerun)	judge	90%	10%	80%	0%	10%	0.668
B4 citation leaders	judge	100%	30%	70%	0%	0%	0.632
B3 review future work	judge	100%	10%	70%	20%	0%	0.593
B2 direct LLM	judge	90%	0%	90%	0%	0%	0.732
B1 random claims	judge	70%	0%	70%	0%	0%	0.611

Table 4: Astronomy v1 leaderboard (

n=10

questions per system). Adjud. = human-adjudicated labels; judge-only rows are directly comparable to each other. Cov. = future-attention rate; Ans. = answered; Part. = partially addressed; Open = posed but open; Ref. = premise refuted;

s_{1}

= mean top-1 retrieval similarity. Lead time is null for all systems until community first-posed dates are annotated (Section 4 ).

Rank	ID	Outcome	Premise	$\|E\|$	Year
1	q_002	posed_but_open	still_plausible	1	2022
2	q_001	partially_addressed	supported	3	2021
3	q_004	partially_addressed	supported	1	2021
4	q_008	answered	refuted	3	2025
5	q_007	partially_addressed	still_plausible	3	2024
6	q_010	partially_addressed	still_plausible	1	2022

Table 5: Per-question outcomes for evidence_graph_v1 on Astronomy v1 (system rank order;

|E|

= independent supporting papers; year = earliest cited engagement).

Panel of Figure 2 — Figure 2: Timeline of q_008. The question was frozen from pre-2021 evidence; by 2025 three independent analyses had performed the test it specified and refuted its premise—the strongly subsolar water abundance was a retrieval artifact.

System	$n$	Eng.	Ans.	Ref.	$s_{1}$	lag
evidence_graph (anchor)	10	80% [44,97]	30% [7,65]	10% [0,45]	0.682	3.5
B2 direct LLM	125	96% [91,99]	15% [9,23]	0% [0,3]	0.722	1.9
B4 citation leaders	125	87% [80,93]	26% [18,34]	3.2% [1,8]	0.632	2.5
B3 review future work	49	92% [80,98]	22% [12,37]	0% [0,7]	0.567	2.4
B1 random claims	125	73% [64,80]	11% [6,18]	0% [0,3]	0.622	2.8

Table 6: Astronomy v1L results (judge-only). Brackets are exact 95% Clopper–Pearson intervals. Eng. = future-attention rate; Ans. = answered; Ref. = premise refuted;

s_{1}

= mean top-1 retrieval similarity; lag = mean years to earliest cited engagement. The evidence-graph row is the ten v1 questions re-evaluated on the v1L corpus (anchor), not a scaled submission.

Pipeline	Eng.	Ans.	Ref.	$s_{1}$	$s_{\mathrm{obj}}$
A: LLM only	96% [91,99]	15% [9,23]	0% [0,3]	0.722	5%
B: structure $\to$ LLM	74% [65,81]	39% [31,48]	13% [8,20]	0.694	95%
C: structure only	77% [68,84]	25% [17,33]	2.4% [0,7]	0.721	100%

Table 7: The A/B/C decomposition on astronomy v1L (judge-only,

n=125

each; brackets are 95% Clopper–Pearson intervals).

s_{\mathrm{obj}}

= share of questions naming a specific catalogued object (deterministic rubric, Section 9.3 ).

		c2010	c2015	c2020	c2024
A: LLM only	Ans.	22%	12%	10%	16%
	Ref.	2	0	0	0
B: structure $\to$ LLM	Ans.	31%	26%	34%	32%
	Ref.	6	3	6	5
C: structure only	Ans.	10%	6%	16%	4%
	Ref.	0	0	0	0

Table 8: Temporal contamination stress test (judge-only,

n=50

per cell; c2010 tension cells have

n=48

—the 855-paper 2005–2010 corpus yields only 48 detectable tension pairs). Cutoffs 2010–2020 lie inside the LLM’s training data; the 2024 cutoff’s future window (2025–2026-06, censored) postdates it. Ans. = answered; Ref. = refuted count.

Conclusion	gpt-4.1	gpt-4o	gpt-4-turbo
structure $\to$ LLM answers more than LLM-only	✓	✓	✓
LLM-only engages more than random templates	✓	✓	✓
structure $\to$ LLM refutes more than LLM-only	✓	✓	$\times$
structure-only answers more than LLM-only	✓	$\times$	$\times$

Table 9: Conclusion robustness under three judges, unstratified sample (

n=40

per system). ✓ = the stated ordering holds;

\times

= it does not. Every

\times

is a tie at zero, where the judge assigns the label to no system; no judge reverses any conclusion.

	H2	4.1	4o	4-t	fable-5	5.6-sol
H1 (non-expert)	0.17	0.10	$-$ 0.03	$-$ 0.00	0.02	0.02
H2 (professional)		0.26	0.17	0.20	0.21	0.21
gpt-4.1			0.34	0.20	0.47	0.31
gpt-4o				0.71	0.47	0.58
gpt-4-turbo					0.32	0.51
claude-fable-5						0.60

Table 10: Pairwise Cohen’s

\kappa

on outcome, all seven raters,

n=90

. H1 = non-expert human; H2 = professional annotation team. Read against the human–human cell (0.17): no model reaches agreement with a human that could pass for reliability, and every model–model pair agrees more strongly than any human–model pair.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Jul 6, 2026

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.

Yufeng Wang

Scientific Hypothesis Generation Large Language Model-Guided Scientific Discovery

Ask PDF

Jul 29, 2026

Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting

Current AI systems are optimized for answering questions; the scientific enterprise is bottlenecked earlier, at discovering the questions worth investigating. We present a framework that turns a traceable, reproducible, scope controlled research corpus into ranked, falsifiable research questions: evidence is represented as provenance carrying claims; cross paper tensions are detected, typed, and human adjudicated; surviving signals are refined into questions and ranked by a two stage protocol separating scientific priority from execution priority. We instantiate the framework on exoplanet atmospheres, a domain that uniquely combines literature, structured catalogs, and space telescope archives. In a historical backtest, all questions generated from evidence available before 2021 were substantively engaged by the 2021 to 2026 literature the sys?tem never saw: two were answered, including one whose premise the community later explicitly refuted and the top ranked question is independently posed and still open. These results sug?gest that systematic question discovery from evidence tensions surfaces the questions working scientists subsequently invest in.

Hui Mao

Independent Researcher

Evidence-Grounded Reasoning Scientific Discovery

Ask PDF

Jun 10, 2026

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.

Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria

DeCLaRe Lab, Nanyang Technological University

AI for Science LLM-as-a-Judge Research Idea Generation

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting

On the Limits of LLM-as-Judge for Scientific Novelty Assessment