cs.CLOct 28, 2026
SaveTracing the Influence of Retrieved Literature on Scientific Ideation in LLMs
Abstract
Large language models (LLMs) are increasingly used to generate research ideas from scientific literature. Existing systems combine literature retrieval with search, generation, and refinement, making it difficult to determine how the retrieval strategy itself shapes the ideas generated. We isolate this contribution by comparing five retrieval strategies and a No Literature baseline under a shared generation protocol, using 45,000 ideas across 15 machine learning topics and ten open-weight models. Relative to no literature basline, all five retrieval strategies reduce diversity among ideas generated from the same set of papers by --, while most increase diversity by -- when ideas from different sets are pooled. Effects on idea quality and novelty are modest on average but vary across topics. We further apply a JSD-based attribution method to trace the influence of individual input papers on each idea's motivation, core idea, approach, and validation. In of ideas, only one or two distinct papers receive the highest attribution scores across these four parts. These leading sources often recur across ideas generated from the same set of papers, and groups with more varied source attribution contain more diverse ideas (). These findings motivate various sets of literature and the use of attribution to guide the choice of sources for subsequent ideas.
Figures & tables
Figure 1: How do retrieval strategies shape scientific exploration? (a) Assessing individual ideas does not reveal how retrieval shapes the range of ideas explored. (b) We compare retrieval strategies under a shared generation protocol, assess idea sets, and trace literature influence on idea components.
Figure 2: An example of generated idea and attribution computation. (a) R4 retrieves a core paper (P1) and its references; (b) The generated idea is shown through condensed component text. (c) Four pairs of curves illustrate distribution changes for the four components under leave-one-paper-out scoring of the generated idea.
Figure 3: Retrieval changes the range of scientific exploration more than quality and novelty. (a) Mean cosine distance among ideas generated from the same literature set (open circles) and across all sets under each retrieval strategy (green). “Gap” denotes overall minus within-set diversity; for No Literature the two coincide. (b) Mean quality with 95% confidence intervals; the shaded region marks the 95% confidence interval of No Literature . (c) Mean local and global novelty with 95% confidence intervals.
Figure 4: Ideas spread more widely with retrieval in the PCA visualization. Each panel shows 50 Qwen3 ideas and 2,000 corpus papers for the topic Vision, Language, Audio, and Multimodal Applications .
Figure 5: Leading-source reuse and idea diversity. (a) In of ideas, at most two papers lead the four components. (b) Within a fixed literature set, – of idea pairs share the same leading paper for a given component, over three times the baseline under independent, uniform source selection. (c) Across groups of 20 ideas, greater source variation is associated with higher idea diversity (Pearson ).
Figure 6: Different leading sources within one generated idea. P1 leads the motivation; P7 leads the core idea, approach, and validation. Bars show the top three shares of mean token-level leave-one-out JSD, normalized over all eight input papers per component. Colored accents identify each component’s leading paper.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Group | Model | Parameters |
|---|---|---|
| 2024 | Qwen2.5 | 32B |
| Gemma 2 | 27B | |
| 2025 | Qwen3-2507 | 30B |
| Gemma 3 | 27B | |
| Nemotron 3 Nano | 30B | |
| GLM-4 | 32B |
Table 1: Models used in the study. Open-weight models are grouped by release year.
| Research topic | Constituent areas |
|---|---|
| Representation, Transfer, Meta, and Continual Learning | Representation learning; transfer, meta, and lifelong learning |
| Foundation and Generative Models | Foundation/frontier models; generative models |
| Reinforcement Learning and Decision Making | Reinforcement learning |
| Probabilistic and Causal Learning | Probabilistic methods; causal reasoning |
| Optimization and Learning Theory | Optimization; learning theory |
| Graph and Geometric Learning | Graphs, geometries, and topologies |
Table 2: The 15 research topics and their constituent ICLR 2026 areas. Area names are abbreviated.
| Analysis | Coverage |
|---|---|
| Generation and embeddings | 45,000 ideas across ten open-weight models |
| Quality | Five ideas per model, topic, and condition; 4,500 ideas in total |
| Local novelty | All 37,500 ideas generated with retrieved literature |
| Search-assisted global novelty | One idea per model, topic, and condition; 900 ideas in total |
| Attribution | Ten open-weight models; expanded groups contain 20 ideas from the same model and literature set |
Table 3: Evaluation coverage for the open-weight models. A condition is one of the five retrieval strategies or No Literature .
Figure 7: Quality evaluation prompt. The judge scores validity, feasibility, significance, and clarity on a 1–10 scale.
Figure 8: Local novelty evaluation prompt. The judge scores the idea’s novelty relative to the supplied papers on a 1–5 scale.
Figure 9: Global novelty evaluation prompt (part 1 of 2). The judge assesses the idea’s novelty against related literature found through web search.
Figure 10: Global novelty evaluation prompt (part 2 of 2).
Figure 11: Human annotation instructions and interfaces for (a) quality, (b) local novelty, (c) global novelty, and (d) literature attribution.
| Metric | Human–Judge | Human–Human |
|---|---|---|
| Quality | 0.307 | 0.501 |
| Local novelty | 0.680 | 0.420 |
| Global novelty | 0.504 | 0.143 |
Table 4: Spearman correlations on 20 NLP ideas. Human–Judge compares the mean rating from three annotators with the primary LLM judge’s score. Human–Human averages correlations across the three annotator pairs.
| Score | ICC(A,1) | ICC(C,1) | |||
|---|---|---|---|---|---|
| Validity | 0.212 | 0.395 | 0.223 | 0.416 | 0.158 |
| Feasibility | 0.574 | 0.616 | 0.546 | 0.545 | 0.616 |
| Significance | 0.103 | 0.479 | 0.401 | 0.068 | 0.083 |
| Clarity | 0.221 | 0.605 | 0.285 | 0.385 | 0.140 |
| Overall quality | 0.196 | 0.549 | 0.338 | 0.218 | 0.139 |
Table 5: Agreement on quality ( ). S, G, and C denote Sol, Gemini Pro, and Claude Opus. All values use quadratic weights. Overall quality scores retain their quarter-point spacing.
| ICC(A,1) | 95% CI | |||
|---|---|---|---|---|
| 0.654 | [0.607, 0.689] | 0.721 | 0.607 | 0.620 |
Table 6: Agreement on local novelty ( ). L, G, and H denote Luna, Gemini Flash, and Claude Haiku. The ICC confidence interval uses 2,000 bootstrap resamples of research topics.
| Model | Reasoning | Accuracy | Macro-F1 | MAE | |
|---|---|---|---|---|---|
| DeepSeek-R1 | – | 0.307 | 0.123 | 0.993 | 0.087 |
| Llama-3.1-8B | – | 0.289 | 0.146 | 1.004 | -0.014 |
| Llama-3.3-70B | – | 0.289 | 0.095 | 1.043 | 0.032 |
| Llama-4-Scout | – | 0.289 | 0.130 | 1.014 | 0.076 |
| GPT-5 | – | 0.289 | 0.172 | 0.928 | 0.140 |
| GPT-OSS-120B | – | 0.296 | 0.148 | 0.964 | 0.117 |
Table 7: Novelty prediction on RINoBench ( ). All models are evaluated against the same reference scores. The upper block uses archived benchmark predictions; the lower block evaluates GPT-5.6 models with the benchmark prompt. MAE denotes mean absolute error, and denotes Spearman’s rank correlation. Bold values indicate the best result in each column.
| Evaluation | Estimated cost (USD) |
|---|---|
| Quality | 500.96 |
| Local novelty | 139.17 |
| Global novelty | 180.39 |
| RINoBench evaluation | 37.41 |
| Total | 857.93 |
Table 8: Estimated API costs for LLM-based evaluation, including development and validation runs. Global novelty includes web-search costs.
| Retrieval effect | Change | ||||
| Metric | 2024 | 2025 | 2026 | 2024–2025 | 2025–2026 |
| Gemma and Qwen | |||||
| Diversity | |||||
| Quality | |||||
| Gemma, Qwen, GLM, and Nemotron | |||||
| Diversity | — | — | |||
Table 9: Retrieval effects across model releases, averaged equally over R1–R5, the listed families, and 15 topics. Effects are retrieval minus No Literature; changes are later minus earlier. Asterisks mark changes with Benjamini–Hochberg adjusted .
Figure 12: Comparison of matched 2025 and 2026 models. (a,b) Changes in diversity, quality, and novelty by retrieval strategy and model family. (c) Mean number of distinct leading papers across an idea’s four components. (d) Change in leading-paper reuse for each component, measured among idea pairs generated from the same literature set, in percentage points. Heatmaps show 2026 minus 2025; positive values in (d) indicate greater reuse. Asterisks mark statistically significant changes.
Figure 13: Semantic diversity across idea components, measured by mean pairwise cosine distance. Open circles show within-set diversity among ideas generated from the same papers. Filled circles show overall diversity among ideas from all literature sets for the same strategy, model, and topic. Panel (a) uses the title and core idea together; panels (b)–(f) use individual components. All panels share the same distance scale.
Figure 14: Quality scores by dimension. The composite is the mean of validity, feasibility, significance, and clarity. Points show mean scores, with 95% confidence intervals obtained by bootstrapping topics. Shaded bands and dashed lines indicate the No Literature intervals and means, respectively. Horizontal axis ranges differ across panels.
Figure 15: Semantic distance from ideas to their input literature. Panel (a) shows cosine distance to the centroid of the eight input-paper embeddings. Panel (b) shows distance to the nearest paper (circles) and mean distance across the input papers (diamonds). Points show means, with 95% confidence intervals obtained by bootstrapping topics. Dashes denote undefined distances for No Literature .
| Quality | Diversity | Novelty | |||||||
| Strategy | V | F | S | C | Avg. | W | O | L | G |
| GPT-5.6 Luna | |||||||||
| no-lit | 6.43 | 6.32 | 7.97 | 7.97 | 7.17 | 0.273 | 0.273 | — | 2.67 |
| R1 | 6.41 | 5.80 | 8.04 | 7.87 | 7.03 | 0.230 | 0.328 | 3.65 | 2.53 |
| R2 | 6.40 | 5.97 | 8.01 | 7.97 | 7.09 | 0.229 | 0.339 | 3.79 | 2.67 |
| R3 | 6.33 | 5.84 | 8.05 | 7.97 | 7.05 | 0.223 | 0.224 | 3.84 | 2.87 |
Table 10: Detailed results for the three closed-source models in the 2026 evaluation. V: Validity; F: Feasibility; S: Significance; C: Clarity; Avg.: aggregate quality. W: within-set diversity; O: overall diversity. L: local novelty; G: global novelty. Bold marks the highest displayed value within each model and column. Local novelty summarizes 75 ideas per model and retrieval strategy; it is not reported for no-lit. † Global novelty averages 14 topics for this entry (15 otherwise).
| Analysis | Topics | Samples/cell | Total/mode |
|---|---|---|---|
| Diversity | 15 | 50 | 18,000 |
| Quality | 15 | 5 | 1,800 |
| Local novelty | 15 | 10 | 3,000 |
| Global novelty | 2 | 4 | 192 |
Table 11: Evaluation samples per mode. A cell is a model–topic–condition combination. Local novelty covers R1–R5; the other analyses also include No Literature.
Figure 16: Reasoning-enabled ideation across four models. (a) Diversity within literature sets and overall, using title and core-idea embeddings. Gap denotes overall minus within-set diversity. (b) Quality; the shaded band shows the No-Lit 95% confidence interval. (c) Local novelty across 15 topics and global novelty across two topics. Quality and local-novelty error bars are 95% topic-bootstrap confidence intervals.
| Quality (1–10) | Local novelty (1–5) | Global novelty (1–5) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Off | On | Off | On | Off | On | |||
| Qwen3.6 | 5.792 | 6.225 | 3.264 | 3.541 | 2.208 | 2.458 | |||
| Gemma 4 | 6.280 | 6.448 | 3.235 | 3.513 | 2.250 | 1.979 | |||
| Nemotron Cascade 2 | 5.498 | 5.690 | 2.929 | 3.043 | 2.292 | 2.188 | |||
| GLM-4.7-Flash | 5.272 | 5.790 | 3.229 | 3.397 | 2.396 | 2.229 | |||
| Macro-average | 5.711 | 6.038 | 3.164 | 3.374 | 2.286 | 2.214 | |||
Table 12: Quality and novelty with reasoning disabled (Off) and enabled (On); . Each model contributes 450 quality, 750 local-novelty, and 48 global-novelty paired samples. Quality and global novelty average over six conditions; local novelty over R1–R5. Global novelty covers two topics; the other scores cover 15.
Figure 17: Topic-level scores for No Literature and the five retrieval strategies, averaged across the four 2025 models. Cell numbers show mean scores; colors indicate ranks within each topic, with lower ranks indicating higher scores. Bold outlines mark the highest means, including ties. Rankings use unrounded scores and average ranks for ties. Gray No Literature cells in panel (c) indicate that local novelty is undefined without reference literature.
Figure 18: Topic-level changes in diversity, quality, and novelty, averaged across the four 2025 models. Values show differences from No Literature in panels (a), (b), and (d), and from R1 in panel (c). Positive values indicate increases. Each panel uses a separate color scale symmetric around zero.
| Sources requested | This idea’s report | Another idea’s report |
|---|---|---|
| After generation | 77.58% | 52.97% |
| During generation | 71.89% | 48.36% |
Table 13: Percentage of idea parts whose highest-scoring paper appears in the source report. Each row covers 11,200 idea parts. The comparison with another idea averages over the corresponding part of the other 19 ideas generated from the same inputs. Empty reports count as no match. In 20,000 shuffles of whole source reports among ideas within each setting, none reached the observed correspondence in either condition (Holm-adjusted across the two conditions).
Figure 19: Two examples of disagreement between attribution and reported sources: (a) a relevant method paper is omitted; (b) one of two almost equally scored papers is reported. Percentages show attribution scores normalized across eight papers. Quotes reproduce saved text; paper numbers are local to each case.
| Sources requested | ||
|---|---|---|
| Measure | After generation | During generation |
| Distinct highest-scoring papers per idea | 1.750 | 2.014 |
| Average distance between ideas | 0.3505 | 0.3154 |
Table 14: Comparison of 2,800 ideas per condition. The first row counts distinct papers ranked first across an idea’s four parts. The second reports the average cosine distance between embeddings of titles and core ideas generated from the same inputs; lower values indicate more similar ideas.
Figure 20: Component-level attribution for ideas generated from the same literature set. (a) Attribution shares for Idea 3’s four components across all eight papers. Color encodes attribution share; thicker outlines mark the largest share in each component. (b) Core-idea attribution shares across five ideas, with LangFlow in purple, TraceRouter in green, and the other six papers aggregated in gray. TraceRouter leads Ideas 1–4, while LangFlow leads Idea 5. P1 and P2 denote LangFlow and TraceRouter in the input order.
Figure 21: Idea 3 with manual highlights. Purple passages mark correspondences with LangFlow’s continuous-generation setting and evaluation metrics. Green passages mark correspondences with TraceRouter’s path-level intervention and feature analysis using sparse autoencoders.
Figure 22: Idea 5 with manual highlights. Purple passages mark correspondences with LangFlow’s continuous-generation setting and perplexity-based evaluation.
Figure 23: LangFlow (P1): title and abstract supplied to the model. Purple highlights mark passages on continuous diffusion, flow matching, ODE-based evaluation, and perplexity metrics.
Figure 24: TraceRouter (P2): title and abstract supplied to the model. Green highlights mark passages on path-level intervention, feature analysis using sparse autoencoders, and selective suppression of harmful causal chains.
More in this topic
Research Idea Generation
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
See all 62 papers