HindSight: Evaluating LLM-Generated Research Ideas via Future Impact

Bo Jiang

cs.CLMar 16, 2026

HindSight: Evaluating LLM-Generated Research Ideas via Future Impact

Authors: Bo Jiang

Organizations: Temple University

Abstract

Evaluating AI-generated research ideas typically relies on LLM judges or human panels -- both subjective and disconnected from actual research impact. We introduce HindSight, a time-split evaluation framework that measures idea quality by matching generated ideas against real future publications and scoring them by citation impact and venue acceptance. Using a temporal cutoff~ $T$ , we restrict an idea generation system to pre- $T$ literature, then evaluate its outputs against papers published in the subsequent 30 months. Experiments across 10 AI/ML research topics reveal a striking disconnect: LLM-as-Judge finds no significant difference between retrieval-augmented and vanilla idea generation ( $p{=}0.584$ ), while HindSight shows the retrieval-augmented system produces 2.5 $\times$ higher-scoring ideas ( $p{<}0.001$ ). Moreover, HindSight scores are negatively correlated with LLM-judged novelty ( $ρ{=}{-}0.29$ , $p{<}0.01$ ), suggesting that LLMs systematically overvalue novel-sounding ideas that never materialize in real research.

Figures & tables

Figure 1: The HindSight framework. An idea generation system accesses only pre-

T

literature to produce research ideas. These are encoded alongside post-

T

papers using SPECTER2, matched via FAISS, and scored by the matched papers’ real-world citation impact and venue prestige.

HindSight evaluation
Metric	RA	BL	$\boldsymbol{\Delta}$	$\boldsymbol{p}$
Score (mean)	0.297	0.119	+0.178	$<$ 0.001
Score (median)	0.403	0.000	+0.403	—
Match rate	81%	42%	+39%	—
Avg. matches	9.0	3.3	+5.7	—
LLM-as-Judge evaluation

Table 1: System comparison (RA = ResearchAgent, BL = baseline). Statistical tests: Mann–Whitney

U

. HindSight sharply distinguishes the two systems, while LLM-as-Judge Overall does not.

Figure 2: Score distributions for both evaluation methods. (a) HindSight clearly separates the two systems, with the baseline clustering at zero. (b) LLM-as-Judge Overall scores are nearly identical (

p{=}0.584

). Diamond markers show means.

	ResearchAgent		Baseline
Dimension	$\rho$	$p$	$\rho$	$p$
Novelty	$-$ 0.291	0.003	$-$ 0.140	0.164
Feasibility	+0.252	0.012	+0.006	0.951
Impact	$-$ 0.225	0.025	$-$ 0.150	0.135
Overall	$-$ 0.075	0.457	$-$ 0.124	0.219

Table 2: Spearman

\rho

between HindSight and LLM-Judge dimensions. For ResearchAgent, HindSight is negatively correlated with novelty and impact but positively correlated with feasibility. All baseline correlations are non-significant.

Figure 3: Threshold sensitivity. (a) At lenient thresholds (

\theta{\leq}0.93

) nearly all ideas match, reducing discriminative power. (b) The ratio of RA to BL mean HindSight scores grows monotonically from 1.1

\times

to 3.8

\times

\theta

increases, confirming that the advantage is robust and amplified at stricter thresholds. Dotted lines mark

\theta{=}0.96

Figure 4: Each idea plotted by LLM-Judge Overall (

x

) and HindSight score (

y

). Dashed lines mark the medians used for quadrant classification. Retrieval-augmented ideas (blue) concentrate in the upper quadrants; baseline ideas (orange) cluster at

y{=}0

Category	$n$	%	$n$	%
	RA		BL
True Positive	17	17	6	6
Hidden Gem	23	23	13	13
Overhyped	26	26	34	34
True Negative	34	34	47	47

Table 3: Quadrant distribution. True Positive : high on both. Hidden Gem : high HindSight , low Judge (LLM underrates). Overhyped : low HindSight , high Judge (LLM overrates). True Negative : low on both.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Spearman $\rho$ between HindSight and LLM-Judge dimensions. Stars denote significance: * $p{<}0.05$ , ** $p{<}0.01$ , *** $p{<}0.001$ .

Scientific Idea Evaluation

See all 15 papers

Measuring the Gap Between Human and LLM Research Ideas
Jul 1, 2026Ziyu Chen, Yilun Zhao, Arman CohanLLM Evaluation Scientific Hypothesis Generation
University of Chicago · Yale University
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

Jul 1, 2026J/K move · Enter open · S save

Measuring the Gap Between Human and LLM Research Ideas

Ziyu Chen, Yilun Zhao, Arman Cohan

University of Chicago · Yale University

LLM Evaluation Scientific Hypothesis Generation Research Idea Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

Abstract

Scientific Idea Evaluation

Explore similar work

Measuring the Gap Between Human and LLM Research Ideas

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation