BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction

Xiao Zhou; Yilun Zhao; Owen Jiang; Tiansheng Hu; Cai Xu; Manasi Patwardhan; Arman Cohan

cs.AISep 21, 2026

BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction

Authors: Xiao Zhou, Yilun Zhao, Owen Jiang, Tiansheng Hu, Cai Xu, Manasi Patwardhan, Arman Cohan

Organizations: Yale NLP Lab · New York University · TCS Research

Abstract

Scientific weak signals are early, low-visibility research directions that later become central to mature scientific topics, yet existing resources such as trend tracking, citation forecasting, and foresight reports rarely provide validated reference sets that link concrete early precursors to later paradigms. We introduce BackTrend, a retrospective benchmark in which, given a mature target topic and a temporal evidence constraint, systems must recover two types of precursors: problem-space signals, underrecognized research problems, and solution-space signals, emerging methods for known problems. BackTrend contains 25 mature target topics in artificial intelligence and machine learning and 66 human-validated weak signals, reconstructed from large-scale literature by grounding each candidate in its 2019-2024 publication-frequency trajectory. We evaluate frontier LLMs, RAG systems, and agentic research systems using semantic matching and coverage-based metrics. Current systems often generate plausible but misaligned precursors, exhibiting topic drift, granularity mismatch, near-miss matching, and incomplete coverage; the strongest system achieves only 10.1% F1, while Coverage10 reaches at most 18.5% of the reference signals. Our budget analyses show that additional retrieval and web-search evidence can improve performance up to a moderate budget, but does not by itself close the substantial performance gap.

Figures & tables

Figure 1: BackTrend overview for the 2024 mature target topic large language models . Top: the benchmark-construction pipeline (§ 3.2 ), in which candidate problem-space and solution-space precursors are mined from the topic’s 2019–2023 papers, scored by their 2019–2024 frequency trajectory, gated, and expert-validated. Lower-left: two of the topic’s eleven human-validated weak signals (the top-ranked problem- and solution-space precursor), each with its 2019–2024 frequency trajectory, which shows the low-then-rising signature that defines a weak signal. Lower-right: four real GPT-5.4 predictions, verbatim from the prediction outputs and produced under a pre-2024 evidence cutoff, each listed with its outcome and with the reference weak signal it was scored against. The full error taxonomy, which also covers granularity mismatch and coverage failure, is defined in § 5.2 .

Figure 2: Overview of the BackTrend construction pipeline. We first compile 25 mature target topics from the Artificial Intelligence and Machine Learning domain of a 2024 JRC report ( Eulaerts et al., 2025 ) and retrieve their 2019–2024 papers from Semantic Scholar. We then mine problem-space and solution-space candidate topics from the 2019–2023 abstracts, consolidate them by embedding clustering, and score each candidate by its 2019–2024 publication frequency, keeping only candidates that pass all four frequency gates. Finally, two AI/ML researchers among the authors validate the surviving candidates for topical relevance, correct problem/solution categorization, temporal consistency, and evidence faithfulness to produce the final BackTrend benchmark.

Mature Target Topic	# Papers	# P-WS	# S-WS	Mature Target Topic	# Papers	# P-WS	# S-WS
Artificial Intelligence of Things	25.8K	1	0	Machine Unlearning	1.9K	2	2
Asynchronous Federated Learning	3.0K	2	1	Masked Face Recognition	1.9K	1	1
Attention Mechanisms in CNN	29.5K	3	5	Masked Language Model	5.4K	0	3
Decentralized Federated Learning	4.3K	1	1	Multimodal AI	8.6K	0	0
Epistemic AI	1.2K	0	0	Multimodal Hate Speech	0.1K	0	0
Evolutionary Neural Arch. Search	1.1K	1	0	Privacy-Preserving Machine Learning	11.3K	1	0

Table 1: Data statistics of BackTrend. All 25 mature target topics fall in the Artificial Intelligence and Machine Learning domain. # Papers : unique non-empty-abstract papers retrieved for the topic and its paraphrases over 2019–2024 (thousands; entries are rounded to 0.1K and need not sum exactly to the total). # P-WS / # S-WS : number of human-validated problem- / solution-space weak signals. Seven topics yield no validated signal: five produce no gate-passing candidate, and for two, expert validation removes every gate-passing survivor (§ A.7 ).

Model	Set LLM			Signal LLM			Cov@10			Human
Model	Problem	Solution	All	Problem	Solution	All	Problem	Solution	All	Human
Frontier LLMs
GPT-5.4	7.4	3.6	5.7	7.0	1.6	4.5	14.3	14.5	14.4	6.0
Qwen3.5-397B	6.5	4.0	5.3	4.7	1.1	3.0	7.5	9.5	8.4	5.3
DeepSeek-R1-0528	14.1	5.4	10.1	9.8	4.1	7.1	23.2	13.1	18.5	9.0
Retrieval-augmented LLMs

Table 2: Main results over the 30 topic-direction pairs with validated weak signals, contributed by 18 of the 25 AI/ML target topics. We report set-level, signal-level LLM-judge F1 (Claude-Opus-4.8, 3 runs) and Coverage@10 (Cov@10), broken down by signal direction (Problem / Solution / All), together with human-judged F1 (Human) from our annotation interface. The highest score in each column is in bold.

Model	Depth	Set F1	Signal F1	Cov@10
Qwen3-8B	$k{=}10$	2.7	2.5	9.3
	$k{=}30$	4.0	3.1	9.8
	$k{=}50$	4.4	4.0	12.7
Qwen3-30B	$k{=}10$	5.2	4.1	10.4
	$k{=}30$	3.5	3.5	9.4
	$k{=}50$	6.6	5.0	9.8

Table 3: Retrieval-budget ablation (set-level / signal-level LLM-judge F1 and Coverage@

10

, Claude-Opus-4.8, 3 runs; percentages). The

k{=}50

rows match the Qwen3 (RAG) rows of Table 2 ; best per model and metric in bold.

Search budget	Set F1	Signal F1	Cov@10
$b{=}0$	4.9	2.7	6.6
$b{=}3$	5.7	2.8	16.6
$b{=}8$	7.9	4.4	18.2
$b{=}\infty$	5.6	4.3	10.4

Table 4: Web-search-round ablation for Tongyi-DR-30B-A3B, where

b

is the number of search rounds the agent may issue;

b{=}\infty

is the default policy of the main results, so its row equals the Tongyi row of Table 2 . Set-level / signal-level LLM-judge F1 and Coverage@

10

(Claude-Opus-4.8, 3 runs; percentages); best per metric in bold.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Expert	Level	Domain	Contribution
Expert 1	Postdoctoral researcher	AI & ML	Weak-signal validation
Expert 2	4th-year PhD student	AI & ML	Weak-signal validation
Expert 3	2nd-year PhD student	AI & ML	Human evaluation of model predictions
Expert 4	2nd-year MS student	AI & ML	Human evaluation of model predictions

Appendix

Table 5: Experts involved in BackTrend. Experts 1–2 independently validate the gate-passing weak-signal candidates (§ 3.2 ); Experts 3–4 independently annotate all model predictions in the human evaluation ( Table 2 ). All 25 target topics fall in the Artificial Intelligence and Machine Learning domain, and all experts are AI/ML researchers among the authors; the two weak-signal validators (Experts 1–2) are the senior members of this group (§ 3.2 ).

Panel of Figure 3 — Figure 3: Why seven mature topics carry no certifiable weak signal: absence is decided by the same 2024-frequency gate that certifies the present topics. For each of the 25 target topics we plot the largest 2024-frequency lift achieved by any candidate passing the low-visibility and exponential-growth gates ( $g_{1}\wedge g_{4}$ ); a validated weak signal requires this lift to clear gate $g_{2}$ ( $\lambda=1.2$ , dashed line). $\bullet$ All 18 signal-bearing topics clear the line ( $1.60$ – $28.7$ ). $\bullet$ Five absent topics fall below it (all $\leq 1.05$ , even though a per-topic maximum over noisy candidate ratios is biased upward): point estimates decline for four and are flat for Federated RL ( $1.05$ , CI $[0.38,2.92]$ ); § A.7 additionally audits every candidate above the line that the other gates excluded. $\blacktriangle$ Two absent topics clear the line, but expert validation removes every surviving candidate: none of their year-by-year traces is a genuine exponential rise (§ A.7 ). Topic names are abbreviated on the axis; Table 1 lists them in full.

Absent topic	# cand.	best lift	Outcome	Dominant cause
Trustworthy ML	16776	2.00	6 survivors, 0 kept	All 6 removed because their frequency trajectories are not exponential rises: each rests on 1–2 papers per nonzero year with multi-year gaps (e.g. $1,0,0,0,1$ ), visibly noise. Nothing concentrates under the umbrella vocabulary.
Federated DL	1781	7.35	3 survivors, 0 kept	All 3 removed because their frequency trajectories are not exponential rises: erratic ( $0,0,1,0,3$ ) or frequency-flat; the one genuine precursor ( prototype-based FL ) rises monotonically, and is kept, in the broader Federated ML corpus.
Multimodal AI	4469	0.56	$0$ pass $g_{1}\!\cdots g_{4}$	Late-consolidating container: corpus quadruples 2019–2023, and its 2024 growth is carried by candidates with single-year in-corpus records (25 of its 29 risers, mostly 2023-only); the one already-visible riser ( multimodal reasoning ) is rejected by $g_{4}$ , and its rise is uncertifiable (CI $[0.57,3.28]$ ).
Self-Supervised CNN	1960	0.60	$0$ pass $g_{1}\!\cdots g_{4}$	Scope migration: precursors matured onto transformer backbones; their in-corpus collapse is statistically certified ( contrastive pretraining $65\!\to\!7$ papers, lift $0.10$ , CI $[0.05,0.21]$ ).
Federated RL	539	1.05	$0$ pass $g_{1}\!\cdots g_{4}$	Sparse niche (69–381 papers/yr, 2019–23): paraphrase expansion adds $<\!1\%$ papers; best lift statistically indistinguishable from flat (CI $[0.38,2.92]$ ).
Epistemic AI	948	0.38	$0$ pass $g_{1}\!\cdots g_{4}$	Sparse (53–270 papers/yr, 2019–23), fragmented across unrelated senses of “epistemic”; strongest candidate’s decline is statistically certified (lift $0.38$ , CI $[0.24,0.60]$ ).

Appendix

Table 6: The seven mature topics with no validated weak signal. # cand. : candidate topics mined; best lift : largest 2024-frequency lift among low-visibility precursors (

g_{1}\!\wedge\!g_{4}

), which gate

g_{2}

requires to reach

\lambda\!=\!1.2

; Outcome : result of the automatic gates and expert validation. Full analysis in § A.7 .

Panel of Figure 4 — Figure 4: The weak-signal signature and three absent-topic failure modes. Each candidate’s yearly frequency is normalized to its own 2019–2023 peak $F_{\max}$ , so every trajectory reaches $1.0$ in its early window; the 2024 point is the citation-based 2024 frequency, and entering the shaded band ( $>\!\lambda F_{\max}$ , $\lambda=1.2$ ) is exactly the gate $g_{2}$ . Present signals rise from low visibility and then leap into the band in 2024. The three absent -topic candidates each fail differently: contrastive pretraining rose from low visibility to $11\%$ of the CNN-scoped 2023 corpus, then its frequency collapses in 2024 as the method matured onto transformers (lift $0.10$ , CI $[0.05,0.21]$ : a statistically certified failure of the outcome condition); multimodal reasoning was never low-visibility, starting at $61\%$ of its own peak, so $g_{4}$ rejects it, and its 2024 rise is itself uncertifiable (CI $[0.57,3.28]$ ); hateful-meme detection spikes on a handful of papers in a tiny corpus and collapses, noise with no positive growth fit for $g_{1}$ . None completes the low-visibility, in-window-rise, 2024-confirmation signature that defines a weak signal.

Mature Target Topic	#	Gate-passing candidates removed by expert validation
Artificial Intelligence of Things	4	(P) cyber-physical production systems interoperability; heterogeneity in federated learning for IoT; intrusion detection in Internet of Vehicles. (S) AI-assisted network slicing
Attention Mechanisms in CNN	2	(P) local feature modeling in vision transformers. (S) deformable self-attention
Evolutionary Neural Arch. Search	2	(P) neural architecture size–performance tradeoffs. (S) weight inheritance in evolutionary neural architecture search
Explainable AI	6	(P) XAI deployment challenges; actionability of AI explanations; explainable fake news detection; explanation effectiveness. (S) explanation method benchmarking; layer-wise relevance propagation
Federated Deep Learning †	3	(P) client drift in federated learning. (S) batch normalization in federated learning; prototype-based federated learning
Federated Machine Learning	1	(S) vehicular federated learning

Appendix

Table 7: Gate-passing candidates removed at expert validation, by mature target topic and signal space ( P : problem space, S : solution space). The four automatic gates of § 3.2 pass 125 candidates across the 25 target topics; two senior AI/ML researchers independently review all of them and retain 66, so the 59 listed here are the removals. Eight target topics do not appear: five produce no gate-passing candidate at all (§ A.7 ), and for the other three, Asynchronous Federated Learning , Decentralized Federated Learning , and Human-Centric AI , every gate-passing candidate is retained. † The two topics whose every survivor is removed, leaving an empty signal set (§ A.7 ). ‡ The single removal on ground (i), naming a physical object rather than a research direction; every other removal is on ground (ii), a trajectory that lacks the low-then-exponentially-rising shape the definition requires (§ 3.1 ).

Model	Set-level BERTScore F1			Signal-level BERTScore F1
Model	Prob	Sol	All	Prob	Sol	All
GPT-5.4	81.7	82.9	82.3	86.9	88.3	87.6
Qwen3.5-397B	82.3	82.5	82.4	87.0	87.1	87.0
DeepSeek-R1-0528	82.3	82.9	82.6	87.1	87.7	87.4
Tongyi-DR-30B-A3B	83.1	82.9	83.0	87.5	87.4	87.4
DR-Tulu-8B	80.9	80.8	80.8	86.1	86.0	86.1

Appendix

Table 8: Appendix BERTScore results on the current AI/ML benchmark, broken down by signal direction ( Prob lem / Sol ution / All ) for the set-level and signal-level settings. All values are percentages (%). Higher is better; the best score in each column is in bold. All systems cluster within a few points, confirming that BERTScore is poorly discriminative for weak-signal matching (cf. the LLM-judge scores in Table 2 ). The spread across all seven systems is 2.5 points at the set level and 1.7 at the signal level, an order of magnitude narrower than the LLM-judge spread on the same predictions.

Reference weak signal	Prediction BERTScore ranks highest	F1	Why it is not a match
parameter-efficient tuning	Parameter-efficient finetuning	92.0	(the correct match, shown for reference)
long-context understanding in large language models	Interpretability and mechanistic understanding deficits	91.2	interpretability $\neq$ long context, and it outranks the correct Long-context dependence and context window limitations (90.1)
retrieval-augmented language models	Program-aided and code-interpreted reasoning	89.8	code execution $\neq$ retrieval, and it outranks the correct Retrieval-augmented generation (89.6)
post-training quantization for large language models	Tool use via language-model orchestration	89.3	model compression $\neq$ agentic tool use
privacy leakage in NLP models	Benchmark contamination and data leakage	88.9	test-set contamination $\neq$ leaking training data; only the word “leakage” overlaps
machine-generated text detection	In-context generalization without reliable evaluation	87.6	detecting generated text $\neq$ few-shot generalization

Appendix

Table 9: BERTScore false matches on the large language models topic (GPT-5.4 predictions, both signal directions). For each reference weak signal we list the prediction that signal-level BERTScore scores highest and its F1 (%). The first row is the genuine match, included as a baseline. Every other row is a different research direction that the LLM judge rejects, yet all of them score between

86.7

and

91.2

, indistinguishable from that baseline. In two cases the top-scoring prediction is not merely wrong but outranks the correct one: the judge accepts Retrieval-augmented generation and Long-context dependence and context window limitations , and BERTScore places a rejected prediction above each. BERTScore rewards shared domain vocabulary and phrase templates rather than same-topic identity.

Topic	Direction	Reference signal(s)	GPT-5.4 prediction(s) (rank)	Error interpretation
Trustworthy AI	Problem	AI trust measurement; appropriate reliance on AI; AI trust calibration; ethical AI in chatbot systems	Robustness under distribution shift (1); uncertainty calibration (2); out-of-distribution detection (3); spurious correlation dependence (4)	Topic drift from human-facing trust calibration and measurement to generic model-robustness and safety problems. Uncertainty calibration is the nearest miss, but calibrating a model’s confidence is not calibrating a user’s trust; none of the 13 ranked predictions matches.
Attention Mechanisms in CNN	Solution	Linear-complexity self-attention; sparse attention in vision transformers; large-kernel convolutions; locality-enhanced transformer architectures; hierarchical vision transformers	Lightweight channel–spatial gating modules (1); multi-scale feature reweighting (2); dynamic convolution via input-conditioned kernel weighting (3); efficient attention approximations for vision backbones (9)	Granularity mismatch: the top of the ranking is fine-grained attention-block variants rather than the benchmark’s architecture-family solutions. The single match, efficient attention approximations , surfaces only at rank 9.
Masked Language Model	Solution	Vision-language models; prompt tuning for pretrained language models; masked video modeling	Span corruption denoising objectives (1); replaced-token detection objectives (2); dynamic masking during pretraining (3); multimodal masking objectives (12)	Lexical near-miss: all 14 predictions stay inside masked-LM pretraining objectives. The two that do point outward, multimodal masking objectives and retrieval-augmented encoder representations , sit at ranks 12 and 14 and still miss the downstream paradigms the benchmark tracks.
Vertical Federated Learning	Problem	Straggler problem; privacy leakage; feature selection; unaligned data (all in vertical FL)	Privacy-preserving entity resolution across siloed parties (1); partial-overlap sample populations (3); leakage risk from intermediate predictions and gradients (5); label leakage from party roles and training interactions (10)	Coverage failure on the highest-scoring pair in the benchmark (F1 $41.8$ ): unaligned data and privacy leakage are matched at ranks 3 and 5, yet the top-ranked prediction reframes alignment as entity resolution and feature selection is absent from all 13.
Machine Unlearning	Problem	Federated unlearning; graph unlearning	Certified removal of data influence (2); retraining cost after data deletion (3); cross-silo deletion in federated or distributed training (9)	Coverage failure: the federated setting is recovered only at rank 9, graph unlearning is absent from all 12 predictions, and the top ranks are generic erasure-compliance concerns.
Large Language Models	Problem	Privacy leakage in NLP models; long-context understanding; LLM alignment with human feedback; machine-generated text detection; pre-trained LM compression	Hallucination and factual unreliability (2); benchmark contamination and data leakage (3); long-context dependence and context window limitations (10); memorization, privacy leakage, and unintended training-data reproduction (11)	Coverage failure: two of the five references are recovered, but only at ranks 10–11; text detection and model compression are absent from all 16 predictions, and the first nine ranks are evaluation and reliability concerns outside the reference set.

Appendix

Table 10: Representative benchmark–prediction comparisons for GPT-5.4 predictions on the AI/ML target topics. Predictions are quoted from the ranked lists scored in Table 2 , with each one’s rank in the system’s own confidence ordering in parentheses. The examples illustrate that many low-scoring outputs are semantically plausible near-misses that diverge in framing, abstraction level, or benchmark coverage. The ranks also show where the misalignment sits: when a reference signal is recovered at all, it is usually recovered late.

Panel of Figure 5 — Figure 5: Construction prompt used to extract problem- and solution-space candidate topics from each retrieved paper abstract, reproduced verbatim from the released pipeline code; braces mark the template slots filled in at run time: the full system prompt, and the user prompt up to the specificity guidance. The remainder is shown in Figure 6 .

Panel of Figure 6 — Figure 6: Continuation of the candidate-topic extraction user prompt ( Figure 5 ): bad-topic examples, output requirements, and the response schema. Extracted candidates are then deduplicated, embedding-clustered, and scored by early frequency and 2024 reference adoption before expert validation (§ 3.2 ).

Panel of Figure 7 — Figure 7: Prediction prompt template for problem-space weak signals, reproduced verbatim from the released pipeline code; braces mark the template slots filled in at run time. domain is Artificial Intelligence and Machine Learning, mainframe_topic the mature target topic $M$ , and year_range the 2019–2023 prediction window. Every evaluated system receives this as a single user message with no system prompt; the RAG systems additionally append the retrieved evidence block and the instruction “Use this evidence when generating the weak signals.” (§ 4.1 ).

Panel of Figure 8 — Figure 8: Prediction prompt template for solution-space weak signals, reproduced verbatim from the released pipeline code; braces mark the template slots filled in at run time. domain is Artificial Intelligence and Machine Learning, mainframe_topic the mature target topic $M$ , and year_range the 2019–2023 prediction window. Every evaluated system receives this as a single user message with no system prompt; the RAG systems additionally append the retrieved evidence block and the instruction “Use this evidence when generating the weak signals.” (§ 4.1 ).

Panel of Figure 9 — Figure 9: Set-level LLM-as-a-judge prompt, reproduced verbatim from the released pipeline code; braces mark the template slots filled in at run time. Both judges share the same system prompt and differ only in the user message. gt_block/pred_block are the reference and prediction lists, rendered as 0-based numbered items. A match requires the two topics to denote the same specific research topic, not merely a related one (§ 4.2 ).

Panel of Figure 10 — Figure 10: Signal-level LLM-as-a-judge prompt, reproduced verbatim from the released pipeline code; braces mark the template slots filled in at run time. Both judges share the same system prompt and differ only in the user message. ref_block/cand_block are the reference and prediction lists, rendered as 0-based numbered items. A match requires the two topics to denote the same specific research topic, not merely a related one (§ 4.2 ).

Forecasting Benchmarks

Forecasting Benchmarks are datasets and evaluation protocols for comparing predictions of future time-series values, events, and outcomes.

See all 227 papers

Feb 24, 2026

PreScience: A Dataset and Benchmark for Scientific Forecasting

Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a dataset and benchmark for scientific forecasting built around 98K recent AI research papers, together with companion papers covering author publication histories and citation links, yielding 502K papers in total. The resulting paper records include titles, abstracts, disambiguated author identities, influential references, topic labels, citation trajectories, and metadata snapshotted to respect temporal cutoffs. We instantiate seven exemplar tasks: five paper-anchored tasks -- contribution generation, collaborator prediction, prior work selection, citation count prediction, and future combination prediction -- and two aggregate topic trend forecasting variants. We develop baselines ranging from simple heuristics and embedding methods to frontier language models and agentic systems, and introduce LACER, an LLM-based metric for evaluating similarity of generated contribution descriptions that agrees better with human judgments than existing metrics. Finally, we compose task models to generate a 12-month synthetic corpus and find that the resulting papers are systematically less diverse and less novel than human-authored research from the same period. We release the PreScience dataset (https://huggingface.co/datasets/allenai/prescience) and code (https://github.com/allenai/prescience).

Anirudh Ajith, Amanpreet Singh, Jay DeYoung +7

Allen Institute for Artificial Intelligence, Seattle, WA, USA · Knowledge Lab, University of Chicago, Chicago, IL, USA · School of Computer Science and Engineering, Hebrew University of Jerusalem, Israel +1

AI for Science Forecasting Benchmarks

Ask PDF

Sep 9, 2026

RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate because reviews and research ideas lack uniquely verifiable outcomes. We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes. At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions. Search generally helps, but all four diagnostic models perform worse than an exact-count exponentially weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks. Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models; frozen-evidence replay links a shared component of this reversal to Forecast-oriented policies retrieving a smaller share of recent evidence. Even with exact historical activity, future-specific updating remains limited, with only GPT-5.5 plus reopened Search slightly surpassing EWMA. Fine-tuning on realised outcomes improves Qwen3-4B's forecast Spearman correlation by 0.105 on held-out fields at later origins, with gains also on change-rich episodes.

Yingqian Wu, Jingcong Liang, Siyuan Wang +4

Fudan University · The Chinese University of Hong Kong · University of Oxford +1

LLM Evaluation

Ask PDF

May 30, 2026

ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems.

Qiuyu Tian, Haojie Yin, Yingce Xia +2

1Southeast University, Nanjing, China · 3Duke Kunshan University, Kunshan, China · 2Beijing Zhongguancun Academy, Beijing, China

LLM Agent Evaluation AI Agent Benchmarks LLM Decision-Making

Ask PDF

Abstract

Forecasting Benchmarks

Explore similar work

PreScience: A Dataset and Benchmark for Scientific Forecasting

RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment