Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. We introduce Re-Search, which adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution. Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions, which largely decide how the game ends; after that, most models plateau. Through controlled interventions on the candidate questions and the hypothesis space, we find that the bottleneck is generation, not judgment: models recognize a good idea or a good question when one is offered, but often cannot produce it on their own. We further analyze the ideation traces and find that models fail by either over-exploiting one direction or over-exploring too many. These findings point to concrete design principles for autoresearch systems: since models can judge candidates they cannot produce, enriching the search with externally sourced ideas and questions may be more effective than improving any single model's generation ability.
Figures & tables
Figure 1: DeepSeek-V4-Flash and GPT-5.4 play Re-Search on the same paper. Height reflects recovery of the paper’s idea. Green marks matching terms; red marks mismatches. While DeepSeek-V4-Flash ends up circling in a basin, GPT-5.4 recovers from a dip and reaches the summit.
Figure 2: Part of the GPT-5.4 game in Figure 1 : given the research problem p , the inquisitor {\color[rgb]{0,0.4453,0.6992}M_{\mathrm{i}}} asks yes-or-no questions, the answerer {\color[rgb]{0.6055,0.418,0}M_{\mathrm{a}}} replies based on the full paper, and the judges {\color[rgb]{0.6328,0.375,0.5195}M_{\mathrm{j}}} score the idea card.
Figure 3: Recovery score of the games over 100 rounds, averaged over 200 papers and three games per paper. The patterns of models are different: three models gain little after the first 30 questions, Gemma and Sonnet slowly climb, while GPT-5.4 has consistent improvement. (§ 4.1 )
Figure 4: An example survey under Cfull . Left: the Cfull survey given before the game, which names the core idea as one option, with no extra hint. Right: how the four context conditions differ. Cdecoy replaces only that phrase with a plausible decoy, Cdummy also replaces the other lines that point to the paper, and Cnone gives no survey. Because Cfull and Cdecoy differ in a single phrase, any difference between them only comes from whether the core idea is in the survey. (§ 5.3 )
Model
k
Gap at k
Gap at 100
Ahead (%)
10
1.59
1.16
63
Gemma-4-31B
20
1.53
1.64
70
30
2.28
2.30
78
10
1.03
1.00
64
Qwen3.6-35B-A3B
20
1.21
1.41
69
30
1.86
1.70
73
Table 1: Prefix-fork experiment: we truncate the best and worst game at round k and continue each three times. Gap at k means score difference between the best and worst prefix at round k. Ahead : share of papers where the best prefix still leads at question 100. The gap largely persists across models. (§ 4.2 )
Figure 5: Share of questions (%) each inquisitor (row) chooses from each proposer (column) in the four-model pool on 200 papers. All models favor questions from GPT-5.4 and themselves. While the proportions differ across inquisitors, the final scores are close. All four score between 5.19 and 5.34, with no significant difference. (§ 5.2 )
Figure 6: Recovery curves under the four context conditions of Figure 4 (200 ICML 2026 papers, three games per paper). Naming the idea ( Cfull ) lifts the curves of most models clearly above Cdecoy and Cdummy ; for GPT-5.4, all four conditions end close together. (§ 5.3 )
Inquisitor
Score
Gain over alone
GPT-OSS-120B
5.20
+1.04 [ +0.74 , +1.34 ]
Qwen3.6-35B-A3B
5.34
+1.09 [ +0.87 , +1.39 ]
DeepSeek-V4-Flash
5.24
+1.07 [ +0.79 , +1.36 ]
GPT-5.4
5.19
−0.59 [ −0.90 , −0.28 ]
Random pick
4.43
—
Table 2: Score after 30 questions for each inquisitor on the same four-model pool on 200 papers. The last column gives the difference from each model’s solo baseline, with 95% confidence intervals. (§ 5.2 )
Figure 7: Top: score after 100 questions under Cdecoy and Cfull for each inquisitor. Bottom: gain in that score from naming the idea ( Cfull−Cdecoy , with 95% confidence intervals), against the score under Cdecoy ; the dashed line is a least-squares fit (Pearson r=−0.74 across the six inquisitors). Naming the idea helps most the models that score lowest without it. (§ 5.3 )
Figure 8: Three patterns of traces in games. The grey dot is the research problem, and each color is one top-level branch. Below each tree: mean top-level branches per game on 200 papers, as a range over the models with that pattern (per-model means in Figure 11 ). See more examples in § G .
Figure 9: Search breadth and final recovery across 1,200 games. Shading separates the three patterns in § 6.1 . Left. After the inquisitor receives a Yes ( ▲ ) or a No ( △ ), how often its next question is a child of the answered question ( goes to a child ) or moves to another top-level branch ( leaves the branch ). In particular, over-exploitation models rarely leave the branch, even after a No, while the over-exploration model goes to a child more often after a No than after a Yes, and leaves the branch about 30% of the time after either answer. Right. Each point groups games by their number of top-level branches. The y -axis shows how much higher or lower these games score at question 100 than an average game, once differences between models and between papers are taken out. The fitted curve peaks at about eight branches (dotted line): games with a moderate number of top-level branches score the highest.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Knowledge cutoff
GPT-OSS-120B
June 2024
DeepSeek-V4-Flash
May 2025 †
Claude Sonnet 4.5
July 2025
GPT-5.4
August 2025
Gemma-4-31B
January 2025
Qwen3.6-35B-A3B
December 2024 ‡
Appendix
Table 3: Knowledge or training-data cutoffs of the inquisitor models, as stated in each provider’s model card or documentation. † DeepSeek’s public summary of training content states that some of the data were collected no later than about May 2025 ( DeepSeek-AI, 2026b ) ; a behavioral audit estimates an effective cutoff of July 2025 ( Jia, 2026 ) . ‡ Not reported by the provider; effective cutoff estimated by the same audit ( Jia, 2026 ) . (§ A )
Judge
Own family
Others
Difference
Qwen3.6-35B-A3B
−0.71
−0.88
+0.17
GPT-OSS-120B
−0.13
−0.46
+0.33
Gemma-4-31B
+1.51
+1.21
+0.30
Appendix
Table 4: Rating of each judge minus the mean rating of the other two judges, for the inquisitor of the judge’s own family and for the other inquisitors (survey experiment, all conditions, question 100). (§ B )
Inquisitor
All judges
Same family left out
DeepSeek-V4-Flash
+1.25
+1.25
GPT-OSS-120B
+1.08
+1.16
Claude Sonnet 4.5
+0.90
+0.90
Qwen3.6-35B-A3B
+0.60
+0.61
Gemma-4-31B
+0.36
+0.38
GPT-5.4
+0.16
+0.11
Appendix
Table 5: Gain in score at question 100 from naming the idea ( Cfull−Cdecoy , paired by paper), with all three judges and with the judge of the inquisitor’s own family left out. (§ B )
Inquisitor
Qwen
GPT-OSS
Gemma
Panel
GPT-5.4
1
1
1
1
Gemma-4-31B
3
2
2
2
Claude Sonnet 4.5
2
3
3
3
GPT-OSS-120B
4
4
4
4
Qwen3.6-35B-A3B
6
5
6
5
DeepSeek-V4-Flash
5
6
5
6
Appendix
Table 6: Rank of each inquisitor at question 100, scored by each judge alone and by the three-judge panel. (§ B )
Inquisitor
Cdummy
Cdecoy
Cfull
DeepSeek-V4-Flash
1.05
1.04
1.10
GPT-OSS-120B
2.80
2.82
2.80
Qwen3.6-35B-A3B
1.32
1.22
1.21
Claude Sonnet 4.5
1.98
2.06
2.04
Gemma-4-31B
2.46
2.42
2.42
GPT-5.4
1.74
1.74
1.74
Appendix
Table 7: Score of the idea card before any question ( c0 ) under each survey condition. (§ C )
Cfull
Inquisitor
Cdecoy
Not asked
Asked, Yes
DeepSeek-V4-Flash
4.93
4.73
6.92
GPT-OSS-120B
5.15
4.16
7.37
Qwen3.6-35B-A3B
4.94
3.78
6.27
Claude Sonnet 4.5
5.44
4.29
7.50
Gemma-4-31B
5.59
3.98
6.80
Appendix
Table 8: Score at question 100 under Cdecoy , and under Cfull split by whether the game asked about the core idea and got a Yes . (§ C )
All
Yes
No
Unk.
Yes / No
n
795
354
279
162
633
Gemma-4-31B
94.0
98.9
93.2
84.6
96.4
GPT-OSS-120B
86.2
96.0
92.5
53.7
94.5
Qwen3.6-35B-A3B
78.4
95.2
88.9
23.5
92.4
Appendix
Table 9: Accuracy (%) of each answerer against the human majority label, overall and by the human label. The last column pools the Yes and No items. (§ D )
Pair
Agree
κ
Yes / No flip
Gemma vs. GPT-OSS
78.4
0.66
6.5
Gemma vs. Qwen
74.3
0.59
7.0
GPT-OSS vs. Qwen
77.7
0.62
10.8
Appendix
Table 10: Pairwise agreement (%), Cohen’s κ , and share of questions (%) answered Yes by one answerer and No by the other, on 9,774 questions. (§ D )
Figure 10: How the search changes over the 100 questions of a game (1,200 games on 200 papers). Questions are grouped into bins of ten. Depth of the node : mean depth of the questions in the bin. Goes to a child and leaves the branch : how often the next question is a child of the current question, or lies in another top-level branch, as in Figure 9 (left) but over all answers. Share of Yes answers : share of the questions answered Yes.
Figure 11: Top-level branches per game for each model on 200 papers. The brackets group the models by search pattern, as in Figure 8 .
Figure 12: Search trees of the six inquisitors on “Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning”.
Figure 13: Search trees of the six inquisitors on “A Direct Second-Order Method for Solving Two-Player Zero-Sum Games”.
Figure 14: Search trees of the six inquisitors on “Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs”.
Figure 15: Search trees of the six inquisitors on “MAD: Manifold Attracted Diffusion”.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.