Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Srujan P Mule; Aniketh Garikaparthi; Manasi Patwardhan

cs.LGApr 6, 2026

Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Authors: Srujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan

Organizations: IISER Pune · TCS Research

Abstract

As language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI-generated ideas without exhaustive experimentation. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run. We study comparative empirical forecasting: given a benchmark-specific research goal and two candidate ideas, predict which will achieve better benchmark performance. We construct a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode. While off-the-shelf 8B-parameter models struggle (30% acc.), SFT dramatically boosts performance to 77.1%, outperforming GPT-5 (61.1%). By framing evaluation as a reasoning task via Reinforcement Learning with Verifiable Rewards (RLVR), we train models to discover latent reasoning paths, achieving 71.35% acc. with interpretable justifications. Through additional ablations and out-of-distribution tests, we show robustness to surface-level heuristics and transfer to both a cross-domain time-split test set and an independently constructed test set. Our results demonstrate that compute-efficient small language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery.

Figures & tables

Figure 1: We explore various methods to fine-tune 8B Parameter Language Models using our constructed dataset. The figure illustrates the potential of our fine-tuned model to accurately judge the quality of scientific ideas. For the two given benchmarks and an overlapping method, our model robustly predicts which idea will perform better in context of the benchmark, while providing insightful reasoning.

Figure 2: Dataset Construction Pipeline We use raw entries from 1,918 NLP leaderboards to construct a statistically grounded idea pairs with a benchmark specific research goal, while difficulty stratification ensures robust evaluation across diverse research goals.

$\sigma$ -Category	Train	/w Reasoning.	Test	Total
1-sigma	6120	120	494	6614
2-sigma	3461	45	284	3745
3-sigma	1038	5	91	1129
Total	10619	170	869	11488

Table 1: Stratified Dataset Statistics of Train, subset of train with reasoning traces and Test Idea pairs

Model / Method	1- $\sigma$	2- $\sigma$	3- $\sigma$	Overall	CD Test
Qwen3
Base	18.42	26.05	10.99	20.14	3.55
Base (Reasoning)	15.38	27.11	26.11	25.31	12.62
Direct-SFT	70.85	85.56	84.62	77.10	45.67
Reason-SFT	35.32	38.90	45.05	37.51	29.31
Reason-DAPO	69.43	75.00	83.52	72.73	45.96

Table 2: Accuracy (%) breakdown across different difficulty subsets i.e. (

\sigma

)-categories, on CD test set and models. Bold : Best, Underline : Second Best

Model	Accuracy (%)
Qwen3
Base	2.69
Base (Reasoning)	20.06
Direct-SFT	63.43
Reason-DAPO	65.94
Synthetic-Reason-SFT-DAPO	56.46

Table 3: Accuracy (%) on the Wen et al. (2025) independently constructed test set. Our fine-tuned 8B models are evaluated zero-shot (no retraining). GPT-4.1 result from Wen et al. (2025) .

Figure 3: Differential Analysis ( $\Delta$ based) with Bootstrap statistical significance tests across different Difficulty Subsets ( $\sigma$ ) and Overall Performance. ∗∗ : $p<0.01$ ; ∗ : $p<0.05$ .

Figure 4: Consistency (%) across different stages and training paradigm of Qwen3 Model.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Total Pairs
Code Generation On Mbpp	864
Common Sense Reasoning On Winogrande	454
Question Answering On Copa	381
Named Entity Recognition Ner On Conll 2003	372
Question Answering On Boolq	363
Question Answering On Piqa	331

Appendix

Table 4: Top 20 Benchmarks by Total Pairs (includes train and test)

	Train Set				Test Set
Category	Original	Excluded	Final	Augmented	Original	Excluded	Final	Augmented
$1\sigma$	8,881	2,761	6,120	12,240	687	193	494	988
$2\sigma$	4,827	1,366	3,461	6,922	401	117	284	568
$3\sigma$	1,436	398	1,038	2,076	169	78	91	182
Total	15,144	4,525	10,619	21,238	1,257	388	869	1,738

Appendix

Table 5: Dataset distribution across sigma categories. Pairs classified as “Excluded” were removed due to the lack of a valid research goal.

Figure 5: Top 20 Tasks based on the total pairs (includes both train and test)

Figure 6: Distribution of the ideas/methods across the years with $\sigma$ -wise breakdown in test

Figure 7: Distribution of the ideas/methods across the years with $\sigma$ -wise breakdown in train

Model	Rank=Metric	Rank $\neq$ Metric	Overall	2025-Only
	(1,308)	(102)	(1,410)	(104)
Qwen3
Base	3.67	1.96	3.55	3.85
Base (Reasoning)	13.14	5.88	12.62	17.31
Direct-SFT	46.64	33.33	45.67	53.85
Reason-DAPO	46.64	37.25	45.96	51.92

Appendix

Table 6: Full Cross-Domain (CD) test set results. Rank=Metric : pairs where the leaderboard rank order agrees with the individual metric order. Rank

\neq

Metric : disagreement subset. 2025-Only : subset of 104 pairs (52 pre-augmentation) with all papers dated

\geq

2025. Consistency-aware accuracy throughout. Bold : best per column among trained models.

Figure 8: The average rewards through the training iterations of Reason-DAPO

Figure 9: The mean output length through the training iterations of Reason-DAPO

Figure 10: Overall Accuracy (%) Vs Mean Number of tokens generated during reasoning.

		Recency (%)			Length (%)
Data Split	Total	Newer	Older	Same	Shorter	Longer	Equal
Full Train Set	19,113	57.08	17.73	25.19	47.97	51.83	0.20
Breakdown by Sigma ( $\sigma$ )
$\sigma=1$	11,016	52.35	19.64	28.01	48.04	51.82	0.15
$\sigma=2$	6,229	62.93	15.49	21.58	48.40	51.31	0.29
$\sigma=3$	1,868	65.42	13.97	20.61	46.09	53.69	0.21

Appendix

Table 7: Distribution of dataset preferences in percentages in Train. The Total column indicates the number of samples, while other columns show the percentage breakdown of Recency and Length preferences within each split.

Model	Length		Recency			Paraphrasing
Model	Longer	Shorter	Newer	Older	Same	Original	Para.
1- $\sigma$ ( $N$ )	516	466	480	212	296	988	988
Direct-SFT	$72.9$	$69.1$	$79.6$	$62.3$	$62.8$	$70.9$	$75.5$
Reason-DAPO	$79.1$	$59.2$	$76.3$	$64.2$	$62.2$	$69.4$	$72.1$
Reason-SFT-DAPO	$71.3$	$57.1$	$70.4$	$50.9$	$64.9$	$64.6$	$61.5$
Reason-SFT-DrGRPO	$70.9$	$61.8$	$70.4$	$69.8$	$56.8$	$66.2$	$65.6$

Appendix

Table 8: Detailed Robustness Statistics: Accuracy (%) and Total Sample Count (

N

). The sample counts for each category (e.g., Longer, Newer) remain same across models for the same

\sigma

-subset.

Model	Tier	Length $\Delta$ [95% CI]	Recency $\Delta$ [95% CI]	Paraphrase $\Delta$ [95% CI]
Direct-SFT	Overall	$+$ 2.7 [ $-$ 2.8, 8.3]	$+$ 17.5 ∗∗ [11.8, 23.2]	$+$ 3.3 [ $-$ 0.6, 7.1]
	1- $\sigma$	$+$ 3.8 [ $-$ 4.4, 12.0]	$+$ 17.1 ∗∗ [9.3, 24.8]	$+$ 4.6 [ $-$ 0.8, 10.1]
	2- $\sigma$	$+$ 0.1 [ $-$ 8.3, 8.5]	$+$ 12.3 ∗∗ [3.7, 21.3]	$+$ 1.4 [ $-$ 4.2, 7.0]
	3- $\sigma$	$+$ 7.2 [ $-$ 8.1, 23.2]	$+$ 22.5 ∗ [0.2, 46.0]	$+$ 2.2 [ $-$ 7.7, 12.1]
Reason-DAPO	Overall	$+$ 12.3 ∗∗ [6.5, 18.2]	$+$ 10.9 ∗∗ [4.9, 16.9]	$+$ 2.2 [ $-$ 2.0, 6.3]
	1- $\sigma$	$+$ 19.8 ∗∗ [11.7, 27.7]	$+$ 13.3 ∗∗ [5.2, 21.2]	$+$ 2.6 [ $-$ 3.0, 8.3]

Appendix

Table 9: Bootstrapped bias significance tests (

B=10{,}000

\Delta

in percentage points with 95% CIs. ∗∗ :

p<0.01

; ∗ :

p<0.05

Model	Zero-shot	3-shot
GPT-5 Reasoning (Low)	57.65	57.89
GPT-5 Reasoning (Med)	59.61	59.03
GPT-5 Reasoning (High)	61.10	60.41

Appendix

Table 10: Zero-shot vs. 3-shot GPT-5 accuracy (%) on our in-domain test set. Few-shot examples marginally improve low-reasoning performance but slightly degrade medium and high.

Model / Method	1- $\sigma$ (988)	2- $\sigma$ (568)	3- $\sigma$ (182)	Overall (1738)
Qwen3
Base	69.47 (262)	72.55 (204)	50.00 (40)	69.17 (506)
Base (Reasoning)	63.86 (404)	64.71 (238)	46.67 (60)	62.68 (702)
Direct-SFT	75.92 (922)	88.69(548)	90.59 (170)	81.71 (1640)
Reason-DAPO	80.90 (848)	85.54 (498)	86.36 (176)	83.05 (1522)
Reason-SFT-DAPO	75.59 (844)	87.55 (514)	87.36 (174)	80.94 (1532)

Appendix

Table 11: Conditional Accuracy (%) breakdown across different

\sigma

categories. Values in parentheses denote the total number of samples (

N

) for that category that were consistent. They are no directly comparable since N changes.

Model	In-Domain (ID)			Cross-Domain (CD)
Model	Con. (%) $\uparrow$	Top-1 (%) $\uparrow$	RMSE $\downarrow$	Con. (%) $\uparrow$	Top-1 (%) $\uparrow$	RMSE $\downarrow$
Qwen3
Base	29.55	40.00	1.87	5.35	38.46	2.45
Base (Reason)	40.09	31.43	1.73	19.34	28.21	1.87
Direct-SFT	90.71	44.76	1.22	79.84	31.82	1.96
Reason-SFT-DAPO	83.96	42.86	1.12	77.37	28.89	1.73

Appendix

Table 12: Performance comparison on In-Domain (ID) and Cross-Domain (CD) test sets. Metrics reported are Overall Consistency Rate (Con.), Top-1 Accuracy, and Median RMSE. Bold : Best, Underline : Second Best within each domain. (↓) lower is better; (↑) higher is better.

Family	Model	DB $\downarrow$	DE $\downarrow$	DM $\downarrow$
Qwen	Base	0.2507	0.0937	0.2832
	Base (Reason)	0.2635	0.1392	0.3276
	Direct-SFT	0.1676	0.1390	0.1915
	Reason-DAPO	0.1755	0.1584	0.5475
	Reason-SFT-DAPO	0.1941	0.1688	0.3112
	Reason-SFT-DrGRPO	0.1970	0.1734	0.3641

Appendix

Table 13: Calibration metrics on the in-domain test set. DB : Debiased Brier, DE : Debiased ECE, DM : Debiased MCE. Bold : best within family, Underline : second best within family.

Scientific Idea Evaluation

See all 15 papers

Can Large Language Models Forecast What Researchers Study Next?
Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2LLM Evaluation Forecasting Benchmarks
University of Illinois Urbana-Champaign
Idea2Plan: Exploring AI-Powered Research Planning
Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM Evaluation LLM-as-a-Judge
University of Michigan · Microsoft Research
Learning to Predict Future-Aligned Research Proposals with Language Models
Mar 28, 2026Heng Wang, Pengcheng Jiang, Jiashuo Sun +4LLM Evaluation Research Idea Generation
University of Illinois Urbana-Champaign

Sep 1, 2026J/K move · Enter open · S save

Can Large Language Models Forecast What Researchers Study Next?

Fenghai Li, Zihan Tang, Haofei Yu, Yining Zhao+1

University of Illinois Urbana-Champaign

LLM Evaluation Forecasting Benchmarks Research Idea Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.

Abstract

Scientific Idea Evaluation

Explore similar work

Can Large Language Models Forecast What Researchers Study Next?

Idea2Plan: Exploring AI-Powered Research Planning

Learning to Predict Future-Aligned Research Proposals with Language Models