In pairwise comparisons where each judge chooses between an idea generated by its corresponding TIGER-8B policy and one generated by GPT-4.1, the three LLM researchers select the TIGER-8B idea in 56.6%-68.4% of 76 comparisons per researcher. PhD1 and PhD2 select the TIGER-8B idea in 19 of 30 pairs (63.3%) and 17 of 30 pairs (56.7%), respectively. All six directed cross-policy comparisons likewise favor the judge's matched TIGER-8B policy over an off-target policy, supporting the use of individual preference feedback for tailored research ideation.
Figures & tables
Figure 1: TIGER training pipeline, repeated for each researcher. (I) A researcher’s ratings of candidate ideas generated from research requests train a reward model. (II) A policy initialized from Qwen3-8B samples eight ideas per request. Post-processing uses a frozen neutral rewriter to supply the RM input; a frozen quality judge scores five quality dimensions, with format and length checked separately. The taste score and penalties form the reward in Eq. 1 , which drives GRPO updates on original-response tokens. Only the policy is updated; the RM, rewriter, and quality judge remain frozen.
Figure 2: Research areas covered by the 400 requests, using LLM-assisted labels. Queries may belong to multiple areas. HCI denotes human–computer interaction.
Source of RM training annotations
Target researcher
Terra
Gemini 3.7
Haiku
PhD 1
PhD 2
Terra
77.35
76.01
76.81
67.29
59.92
Gemini 3.7
83.55
91.90
91.26
71.08
60.28
Haiku
82.71
90.71
91.74
71.74
60.00
PhD 1
67.11
60.49
54.44
76.56
74.67
PhD 2
61.96
52.97
45.81
74.64
79.96
Table 1: Strict pair accuracy (%) of matched and off-target RMs, all initialized from Qwen3.8-27B. Rows identify the researcher whose labels are used for evaluation; columns identify whose annotations trained the RM. Bold diagonal entries are matched RMs.
Judge
TIGER wins
GPT wins
TIGER (%)
Terra
46
30
60.5
Gemini 3.7
52
24
68.4
Haiku
43
33
56.6
PhD 1
19
11
63.3
PhD 2
17
13
56.7
Table 2: Policy preference evaluation. (a) Pairwise preferences for TIGER-8B versus GPT-4.1. Both PhD researchers provide direct human judgments. (b) Off-target policy win rate (%, ↓ ), based on 30 judgments per direction on shared requests. Each cell is the column policy’s win rate against the row judge’s matched policy; lower values indicate stronger preference for the matched policy. Opposite directions use different judges, and dashes indicate self-comparisons. (c) Terra preferences for TIGER-8B versus direct generation and personalized-RM best-of-8 selection from base Qwen3-8B.
Variant
Taste reward
Quality judge
Rewrite before RM
Full TIGER
Yes
Yes
Yes
Without RM
No
Yes
N/A
Without quality judge
Yes
No
Yes
Without rewriting
Yes
Yes
No
Table 3: RL ablations for Terra. (a) Training components; all variants retain format and length penalties. (b) Pairwise Terra preferences between each ablation and full TIGER. Each row contains 76 judgments; full preference is full TIGER wins divided by 76.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Policy backbone
Qwen3-8B
Rollouts per request
8
Rollout temperature
1.0
Request groups per update
48
Optimization epochs per update
1
LoRA rank
32
Appendix
Table 4: Policy optimization settings for all five researchers.
Figure 4: Agreement among the five researchers in the earlier exploratory study (pooled within-query Spearman ρ ). (a) Agreement among Terra, Gemini 3.7, and Haiku. (b) Each PhD researcher’s agreement with the three LLMs; parentheses show rated ideas/queries. These exploratory pools precede the current five-backbone dataset.
Figure 5: Overview of TIGER. An ideation agent generates candidate ideas from research requests. Two PhD researchers and three LLM researchers provide ratings, which train personalized reward models and guide the corresponding 8B ideation policies. Section 3.1 describes the candidate pools used by these researchers.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.
Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26% of the human level on citation matching and only 18% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.
Shanshan Zhong, Chen Xu, Jingjie Ning +3
Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence
Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.
Zhiyu Chen, Keyu Zhao, Jigao Fu +8
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1