We evaluate language models' capability to generate novel research ideas through the lens of compression. We aim to lower-bound the potentially tiny probability that a language model generates the essence of a future research idea without any hints. Rather than estimate this probability through expensive repeated sampling, our Priced Guidance framework measures the compression cost: how many additional bits of information are needed to guide the model to recover the target idea. We prove that if the model can recover the target idea with at most K bits of guidance in expectation, then it can generate the idea without any guidance with probability at least 2−K. In our framework, the language model, called the generator, can pose a sequence of multiple-choice questions and specify a probability distribution over possible answers. A guide, which is a language model with access to the target idea, selects answers. If a selected answer has prior probability p, the generator pays −log2p bits. The generator aims to produce an idea that matches the essence of the target idea with minimal cumulative cost. This cumulative cost equals, up to an additive constant, the number of bits of information sent by the guide. Using this methodology, we evaluate five generator LLMs (Opus 5, Fable 5.1, GPT-6 Astra, GPT-5.6 Sol, and GLM 5.3) on the core ideas in 87 recent high-quality deep learning papers and we use LLM as judge to determine whether the generated idea matches the target in terms of the central research object and defining mechanism. Fable 5.1 achieves the lowest median compression cost at 69.9 bits, substantially lower than gzip's median of 5,712 bits for losslessly compressing the summary of the target idea. A uniform ensemble of Fable 5.1, Opus 5, and Astra further reduces the median compression cost to 55.8 bits and improves the generation probability lower bound by 18,000 times.
Figures & tables
Figure 1: A top-down view of the Priced Guidance protocol as tree search, illustrated with the ResNet idea. At each bid, the generator asks a question and assigns probabilities to possible answers, defining branches of the tree. The guide uses privileged access to the target paper to select an answer. Orange arrows show the information sent from the guide to the generator, labeled by its cost in bits; blue highlights the resulting trajectory. For a fixed generator realization and a judge that accepts this idea with probability one, 2−K lower-bounds the probability of generating an equivalent idea without guidance. Probabilities and questions are stylized here.
Figure 2: Test-set results under the Essence judge on 87 papers (judge criteria in Section 3.1 ). (a) Coverage versus compression cost. Curves show the percentage of papers successfully compressed at each compression cost for each generator model. The black ensemble uniformly mixes Fable 5.1, Opus 5, and Astra. (b) Median (P50) and (c) 80th-percentile (P80) compression cost. Here we define the q th-percentile compression cost as the minimal compression cost required to output an idea that passes the judge on q% of the test papers. GLM 5.3 is omitted from the P80 panel as it did not pass the essence judge on 80% of the papers within the guide’s API budget. The guide model is fixed to be Fable 5 here.
Figure 3: Illustration of the generator scaffold. (a–b) Excerpts of the research taxonomy of deep learning, with 3,035 nodes, and of the list of 11 contribution categories. GPT-5.6 Sol generated both once. They are fixed across all target papers and given to every generator model in its prompt. The taxonomy bids walk down the taxonomy one edge at a time, offering each child and an option to stop with equal probability, so the guide chooses one taxonomy node (not necessarily a leaf). The category bid offers the 11 categories with equal probability, and the guide chooses one contribution type. (c) A simplified meta question on the SkillNet example. At the start of a turn, the generator prepares both strategies’ questions, answer options, and probabilities. The guide selects the multiple-choice strategy and then the answer. These two selections contribute −log2(.60)−log2(.25)=2.74 bits. Answer sets and probabilities are illustrative.
Figure 4: Information acquired along one successful Fable 5 trajectory for SkillNet. Filled nodes identify paper-specific content; hollow nodes mark meta questions about strategies. Blue denotes the Directional stage. Orange denotes the additional 15.9 bits after promotion to Essence, culminating in identification of the skill ontology and a pass at 64.1 bits.
Figure 5: SkillNet (arXiv:2603.04448). (a) Gold summary of the target idea. (b) Final generator proposal along the trajectory in Figure 4 , with revisions from the Directional proposal in purple; […] marks an omitted field. Appendix E gives the proposals and the gold summary in full.
Figure 6: Test-set results under the Directional judge on the same 87 papers as Figure 2 . (a) Coverage versus compression cost; (b) median (P50) and (c) 80th-percentile (P80) compression cost. The q th-percentile cost is the minimum compression cost needed to produce an idea that passes the judge on q% of the test papers. The black ensemble uniformly mixes Fable 5.1, Opus 5, and Astra; the guide is fixed to Fable 5. GLM 5.3 is omitted from the P80 panel for consistency with the Essence figure; its Directional P80 is 108.3 bits.
Compression cost (bits)
Directional usage
Essence usage
Component
Directional
Essence
Δ
Calls
Mean
Calls
Mean
Field taxonomy
11.61
11.61
0.00
270
3.65
–
–
Contribution category
3.46
3.46
0.00
74
1.00
–
–
Multiple choice
3.21
4.57
1.36
57
0.77
21
0.28
Keyword
33.29
43.70
10.41
134
1.81
56
0.76
Without letter hints
16.06
23.95
7.90
103
1.39
49
0.66
Table 1: Compression cost and scaffold usage for Opus 5. Opus 5 high on the same 74 test papers for which the generated idea passes the Essence judge. We show the compression costs and also the number of times different components are called in the final successful trajectory. Δ is the Essence cost minus the Directional cost. Keyword costs include semantic-category and strategy selections; the letter-hint row includes the full hint path. Promotion is the cost of switching from the Directional to the Essence stage (Appendix A.5.4 ).
Figure 7: Directional compression cost on the development set. (a) Comparing GLM 5.2 and 5.3 shows the effect of general post-training of the generator on this benchmark. (b) Guide choice for Opus 5 ablations; both runs use the Opus 5 generator. (c,d) Opus 5 and Fable 5.1 reasoning effort ablations. Legends report median compression cost in bits. The two-run ensemble in (c) averages the two members’ observed scores as in Equation 26 . Panel (d) is indicative because high and max effort use Fable 5.1 and Fable 5 guides, respectively.
Figure 8: Run-to-run variation and ensembling for Opus 5 on the development set. (a,c) Paired Directional ( n=38 ) and Essence ( n=36 ) costs. Dashed diagonals denote equal cost; r and ρ are Pearson and Spearman correlations. (b,d) Corresponding coverage for both independent runs and their uniformly weighted ensemble, with effective cost given by Equation 26 . While each paper may have stochastic compression cost, the median stays surprisingly stable. The randomness in the independent runs also explains why ensembling improves the median compression cost.
Figure 9: Astra candidate-count and context-management comparisons on the development set. (a,b) Native Codex with one (gray dashed) or eight (blue solid) Directional candidates, evaluated by the Directional and Essence judges, respectively; legend values report median compression cost in bits. (c) Directional compression cost with one candidate in both configurations: Codex with reasoning history (blue) and our earlier scaffold without reasoning history (orange). (d) Essence compression cost with eight Directional candidates using custom context management (blue) or native Codex (gray). Essence costs include inherited Directional cost. Details are deferred to Appendix C.14 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Promotion cost pays for choices inserted into the retained path. Each panel compares the choices retained after recovery (top) with their replay (bottom). Blue boxes marked + are inserted choices and contribute to cprom ; gray boxes are already accounted for in Kπ . (a) Reusing the submission requires a second Submit for Essence evaluation. (b) Resuming at the promoting question requires restoring the removed Submit before taking the retained Essence answer. (c) Resuming elsewhere requires a switch before that answer. The question and its bid probabilities are preserved across the switch. Dashed arrows omit intervening Essence choices. Earlier stay choices within each Directional prefix are omitted; each also contributes to cprom .
Figure 11: Monthly title-only memorization scores for Claude Opus 5 on 849 Top-1 selections from Hugging Face Daily Papers, judged by GPT-5.6 Terra. We can observe a clear transition near the knowledge cutoff. The solid orange curve is the monthly mean; the dashed and dotted curves show the 90th and 10th percentiles, with their interval shaded. Each month contains 15–25 papers.
Metric
Fable 5
Opus 5
GPT-5.6 Sol
GLM-5.3
Astra
Development set (40 papers)
Mean score
22.30
24.78
28.05
9.93
2.50
Score ≥80
0
0
0
0
0
Test set (87 papers)
Mean score
23.37
22.70
27.33
13.40
1.51
Score ≥80
0
0
0
0
0
Appendix
Table 2: Title-only recall on the 40-paper development set and 87-paper test set, scored from 0 to 100 by GPT-5.6 Terra. All five models are evaluated on the same papers within each set. Means are computed after cohort selection. Score ≥80 counts papers meeting the memorization threshold.
Figure 12: Ensemble compression cost versus paper date. Black points show Directional (left) and Essence (right) costs for the uniform mixture of Fable 5.1, Opus 5, and Astra on the 87-paper test set (Appendix A.5.3 ). Dates are Hugging Face Daily Papers listing dates from May 1 to September 4, 2026, spanning approximately four months. Both correlations are close to zero over this period.
Experiment
Ideas
Reasoning history
Post-training (Section 5 )
1
Not retained
guide choice (Section 5 )
1
Not retained
Run-to-run randomness (Section 5 )
1
Not retained
Candidate count: Directional (Section 5 )
1 vs. 8
Retained
Candidate count: Essence (Section 5 )
1†
Retained
Reasoning effort (Section 5 )
1
Not retained
Appendix
Table 3: Scaffold settings in Section 5 . Reasoning history refers to earlier generator calls; explicit working state is retained in every configuration. † The initial Essence evaluation inherits the one- or eight-idea Directional submission; subsequent new submissions contain one idea.
Paper
Fable 5.1
Opus 5 high
GPT-6 Astra
GPT-5.6 Sol
GLM-5.3
Heterogeneous Scientific Foundation Model Collaboration 2604.27351
67.9
45.1
45.9
56.6
94.0
UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors 2605.00658
22.6
21.9
24.7
64.2
45.0
MolmoAct2: Action Reasoning Models for Real-world Deployment 2605.02881
∞
59.6
72.2
82.4
48.0
ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration 2605.03042
∞
34.3
42.8
46.5
41.7
Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation 2605.03849
51.6
65.8
47.8
90.9
108.3
Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction 2605.05242
40.1
49.3
95.0
49.7
67.8
Appendix
Table 4: Directional compression cost by paper and generator on the 87-paper test set. Costs are in bits; ∞ denotes that no validated idea passing the Directional judge was obtained within the experiment budget.
Paper
Fable 5.1
Opus 5 high
GPT-6 Astra
GPT-5.6 Sol
GLM-5.3
Heterogeneous Scientific Foundation Model Collaboration 2604.27351
71.4
79.8
99.5
60.0
∞
UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors 2605.00658
61.1
62.1
60.4
70.4
149.6
MolmoAct2: Action Reasoning Models for Real-world Deployment 2605.02881
∞
61.8
74.8
84.7
79.4
ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration 2605.03042
∞
48.5
45.5
50.2
96.2
Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation 2605.03849
93.2
70.4
70.7
95.7
122.2
Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction 2605.05242
42.4
51.4
98.0
51.3
69.9
Appendix
Table 5: Essence compression cost by paper and generator on the 87-paper test set. Costs are in bits; ∞ denotes that no validated idea passing the Essence judge was obtained within the experiment budget. Essence costs include the Directional stage and the promotion cost (Appendix A.5.4 ).
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.
As language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI-generated ideas without exhaustive experimentation. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run. We study comparative empirical forecasting: given a benchmark-specific research goal and two candidate ideas, predict which will achieve better benchmark performance. We construct a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode. While off-the-shelf 8B-parameter models struggle (30% acc.), SFT dramatically boosts performance to 77.1%, outperforming GPT-5 (61.1%). By framing evaluation as a reasoning task via Reinforcement Learning with Verifiable Rewards (RLVR), we train models to discover latent reasoning paths, achieving 71.35% acc. with interpretable justifications. Through additional ablations and out-of-distribution tests, we show robustness to surface-level heuristics and transfer to both a cross-domain time-split test set and an independently constructed test set. Our results demonstrate that compute-efficient small language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery.
Srujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan
Large language models (LLMs) are increasingly used to assist ideation in research, but evaluating the quality of LLM-generated research proposals remains difficult: novelty and soundness are hard to measure automatically, and large-scale human evaluation is costly. We propose a verifiable alternative by reframing proposal generation as a time-sliced scientific forecasting problem. Given a research question and inspiring papers available before a cutoff time, the model generates a structured proposal and is evaluated by whether it anticipates research directions that appear in papers published after the time. We operationalize this objective with the Future Alignment Score (FAS), computed via retrieval and LLM-based semantic scoring against a held-out future corpus. To train models, we build a time-consistent dataset of 21,835 paper occurrences across 3,642 instances from targets and their pre-cutoff citations, and synthesize reasoning traces that teach gap identification and inspiration borrowing. Across Llama-3.1 and Qwen2.5 models, future-aligned tuning improves future alignment over unaligned baselines (up to +10.6% overall FAS), and domain-expert human evaluation corroborates improved proposal quality. Finally, we demonstrate practical impact by implementing two model-generated proposals with a code agent, obtaining 4.17% accuracy gain on MATH from a new prompting strategy and consistent improvements for a novel model-merging method. Our code and data are publicly available at https://github.com/Arthur-Heng/future-aligned-proposals.