The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions
Authors: Alejandro H. Artiles, Martin Weiss, Levin Brinkmann, Iyad Rahwan, Bernhard Schölkopf, Christopher Pal, Hugo Larochelle, Anirudh Goyal, Nasim Rahaman
Organizations: Tiptree Systems · Max Planck Institute for Human Development, Berlin · Max Planck Institute for Intelligent Systems, Tübingen · ELLIS Institute Tübingen · Polytechnique Montreal · Canada CIFAR AI Chair · Mila – Quebec AI Institute
Scientific discovery is constrained not only by what is true, but by what is cognitively available to researchers. Many directions are coherent in light of the literature yet unlikely to be proposed because no existing community combines the relevant concepts, methods, and intuitions. Modern language models inherit this bias, recombining high-density regions of the literature when prompted for novel ideas. We target the complementary region, the alien space of science: directions that are scientifically plausible but unlikely under the distribution of existing researchers. Our method decomposes papers into conceptual units and clusters them into a vocabulary of idea atoms. A coherence model scores whether atom combinations form viable research directions, while an availability model scores whether any existing author community is positioned to produce them. We rank combinations to maximize coherence while minimizing availability. On 16,068 peer-reviewed LLM papers from NeurIPS, ICLR, ICML, and major NLP venues, the sampler produces more diverse research portfolios than frontier LLM baselines, including batch-prompted variants that see their entire output, while matching or exceeding their per-idea quality under blind human, LLM, and downstream experimental evaluation. In temporal validation, all components use only pre-2025 data. Selecting less than 1% of the candidate space recovers 2025 combinations at 35.9 times chance; weighting unavailability increases recovery of strictly novel combinations to 3.24 times chance. By separating scientific plausibility from community availability, our framework points toward AI ideation that complements rather than merely accelerates human science, expanding exploration into coherent directions the community may overlook.
Figures & tables
Figure 1: Overview of the Alien Science Sampling pipeline. Papers are distilled into conceptual units, which are clustered into a shared vocabulary of idea atoms. A coherence model learns which atom combinations form viable research directions, while an availability model estimates which combinations existing author communities are positioned to produce. Alien directions are sampled by maximizing coherence while minimizing availability.
Figure 2: Two-encoder contrastive training for cognitive availability. A held-out query subset Qi is encoded by ϕ ; the complement Ri∖Qi and other authors’ complements are encoded by ψ . Disjoint views of the same author are pulled together, while views from different authors are pushed apart.
Figure 3: Frozen temporal validation at K=10,000 (0.875% of 1,143,135 candidates). Left: Recovery relative to uniform random ranking for all 2,424 targets and the 1,411 strictly novel targets (log scale; dashed line: chance). A triple is strictly novel if its atoms never co-occurred in any pre-2025 corpus paper. Right: Strictly novel targets as a percentage of all recovered targets; labels give novel/total hits. At β=0.9 , no targets are recovered, so zero enrichment and undefined novelty share are omitted.
Portfolio diversity (300 ideas)
Diversity–quality composite ↑
Method
Atoms
Eff. atoms ↑
Vendi ↑
Rep. pairs ↓
Near-dup. ↓
Human
Tournament
Autoresearch
Alien (ours)
251
198.0
225.2
106
51.3%
0.906
0.917
0.977
Claude, batch ∗
235
191.7
197.9
244
81.7%
0.708
0.049
0.506
Gemini, batch ∗
120
118.1
182.7
314
94.0%
−0.467
0.489
0.030
Gemini, independent
118
56.2
97.4
482
94.0%
−1.084
−0.129
−0.588
Claude, independent
85
28.7
56.5
614
96.3%
−0.316
−0.975
−0.519
Table 1: Portfolio diversity of each method’s 300 ideas, and the equal-weight composite 21[z(Vendi)+z(quality)] using three evaluation measures (Section 5.4 ). Bold marks the best non-random method. Uniform random is the diversity ceiling but optimizes neither coherence nor availability. ∗ Batch variants were not separately rated and inherit the quality of the same model’s independent run.
Figure 4: Generated ideas in embedding and score space. Left: UMAP embedding of method digests from the 16,068-paper corpus and decoded ideas from Claude, Gemini, Alien, and Random samples. Claude and Gemini concentrate around sparse-autoencoder and interpretability concepts, while Alien samples occupy a broader region. Right: Coherence–availability scores of generated three-atom combinations; higher coherence and lower availability are preferred. Alien concentrates in the intended high-coherence, low-availability region, LLM baselines stay in more available regions, and random sampling loses coherence.
Figure 5: Interpretability motifs in 10 initial ideas and final reports (after autoresearch loop) per method. (Appendix E.4 ).
Table 3: Distribution of LLM-filtered papers by venue and year. Zero entries indicate years with no papers from that venue in the filtered database.
Metric
Value
Input conceptual units
82,255
Initial HDBSCAN clusters
273
Initial noise units
62.8%
Selected coverage after reassignment
80%
Mean atoms per paper before reassignment
1.65
Mean atoms per paper at selected coverage
3.38
Appendix
Table 9
Figure 6: Distribution of papers per initial atom cluster, before noise reassignment. Most atom clusters are supported by tens of papers, while a long tail of broad, cross-cutting clusters pulls the mean (81.1) above the median (44).
Figure 7: Author connectivity is coupled to atom coverage. Each point is an author node, positioned by unique atom coverage and distinct coauthor degree; color encodes log(1+publications) , and red points mark the top ten authors by atom coverage. The association shows that broad author profiles are also socially well connected, so individual prolific authors can bridge distant regions of the atom space.
Quantity
Coherence data
Availability data
Training objective
Next-atom prediction
Author–set contrastive retrieval
Training records used
136,827
43,077,647
Validation records used
15,203
4,786,404
Source entities
15,203 papers
44,809 eligible authors
Positive input length
Mean 3.50; median 3; p90 5; max 8
Mean 3.80; median 4; p90 4; max 4
Author-tower input length
–
Mean 27.66; median 24; p90 48; max 143
Appendix
Table 4: Training-record characteristics for the coherence and availability models. Lengths count atom IDs, excluding special sequence tokens for coherence. Availability statistics are computed after dropping empty-complement records, matching the records consumed by the dual-encoder loader.
Parameter
Value
Epochs
30
Batch size
64
Gradient accumulation steps
2
Effective batch size
128
Learning rate
1×10−4
Warmup steps
1,000
Appendix
Table 5: Run settings and architecture details for the coherence transformer. Architecture values not passed explicitly in the command use the training script defaults. Vocabulary size is 273 atom tokens plus a shared BOS/EOS token.
Figure 8: Order sensitivity of coherence scores. Points show the mean coherence score for random, real-paper coherent, and pairwise-disjoint incoherent atom sets; horizontal bars show ±1 standard deviation across permutations of the same atom set. Scores are −NLL , so higher is more coherent. Although scores vary with atom order, the permutation-induced variation is small relative to the separation between real-paper coherent sets and the random/incoherent baselines.
Parameter
Value
Epochs
100
Optimizer
AdamW
Global batch / per GPU
4,096 / 1,024
Gradient accumulation steps
1
Peak learning rate
5×10−4
Learning-rate schedule
2,000-step linear warmup; cosine decay
Appendix
Table 6: Training settings for the dual-encoder availability model. The 100-epoch run took 13 h 42 min.
Model
Conditioning
Paper > zero
Author-only > zero
Paper > author-only
Density estimator
Random negatives
0.683
0.708
0.445
Dual encoder
Author–set compatibility
0.996
0.792
0.950
Appendix
Table 7: AUC-ROC over all 50,036 paper-supported, 2,636,735 author-only and 667,125 zero-support triplets. Both models score exactly the same sets and support labels.
Model
Community > prolific
Paper > prolific
Prolific in top quartile
Density estimator
0.1975
0.4293
21.0%
Dual encoder
0.9999
1.0000
0.0%
Appendix
Table 8: Prolific-author diagnostic on five matched 500-triplet pools. The top quartile contains the 625 highest-scoring sets across all 2,500 candidates; ties use fixed candidate order. Prolific-only controls have no paper support.
Method
Comm. > prolific
Comm. > zero
Comm. > random
Control share top Q
Dual encoder
0.9999
0.9999
0.9689
6.4%
Atom-author W2V, α=1.0 (default)
0.9525
0.9597
0.9100
24.6%
Atom-author W2V, α=0.5
0.8436
0.8397
0.7572
50.6%
Atom-author W2V, α=0.1
0.6095
0.5794
0.5026
76.6%
Appendix
Table 9: Availability diagnostics on four matched pools of 500 triplets: community-supported without paper support, prolific-only, zero-support, and random. The final column denotes the fraction of the highest-scoring quartile that is prolific-only, zero-support, or random.
Figure 9: Availability distributions on 1,500 matched triplets: 500 with highest author support, 500 zero-support sets, and 500 random controls. These histogram pools allow paper support and differ from the no-paper community pools in Table 9 . The same sets are used for all four methods.
Figure 10: Availability on the four matched support pools of Table 9 . Panel A shows per-method standardized means over the 2,000 candidates, with 1.96 standard-error bars. Panel B gives community-versus-control AUCs. Hypergraph α=1 balances author and atom exploration; lower values increasingly weight author transitions.
Figure 11: Model-disagreement quadrants on the same 2,000 support-controlled triplets. Dashed lines mark each method's 75th percentile. The hypergraph-high/dual-low quadrant contains 131 sets, of which 80.9% belong to control pools; the dual-high/hypergraph-low quadrant also contains 131 sets, with 11.5% controls.
Figure 12: Sensitivity to author/atom walk balance. The balanced atom-author hypergraph default ( α=1 ) is the strongest setting. Panel A reports availability-only AUCs for community-supported triplets against three control classes. Panel B shows that lower alpha values increase the fraction of prolific-only, zero-support, and random candidates in the hypergraph high-scoring region.
Figure 13: Dense author-mediated paths collapse the atom space. In the shared AI hypergraph, publication incidence already reaches all atoms by two steps, and adding author nodes reaches the full atom universe immediately under the semantic-hop convention. This explains why availability is difficult to infer from graph reachability alone.
Figure 14: Availability over all 3,353,896 three-atom combinations. Lower scores are less available. The shaded region is the bottom 10%; vertical lines mark its boundary and the means of the fixed random sample and the numerical Alien top 300 at β=0.7 .
Figure 15: Reconstruction-model capability across beta values. The same beta-selected atom combinations are reconstructed with different decoders. Coherence and absolute originality are rated independently per sample; dashed lines show the random baseline. Stronger reconstructors extract more signal from the same recombinations, but the coherence–originality trend as β changes is similar across decoder capacity.
Figure 16: Pairwise originality comparison between reconstructors while holding atom sets fixed. Each cell is the focal reconstructor’s win rate against the opponent. Claude Opus 4.7 is the strongest of the three decoders compared here, so the main beta choice is made in the Claude reconstruction setting.
Figure 17: Beta selection for the final k=3 Claude Opus 4.7 setting. Pairwise originality is obtained from beta-versus-beta originality comparisons; coherence is rated independently per reconstructed sample. The selection score combines the two standardized signals, and is maximized at β=0.7 .
β
Pairwise originality z
Coherence z
Coherence mean
Trade-off
0.6
0.36
0.21
3.38
0.31
0.7
0.86
0.49
3.44
0.74
0.8
1.00
−0.55
3.22
0.49
0.9
0.95
−0.91
3.14
0.34
1.0
0.95
−2.60
2.78
−0.22
Appendix
Table 10: Beta-selection neighborhood for the final k=3 Claude Opus 4.7 reconstruction setting. Pairwise originality comes from beta-versus-beta originality comparisons; coherence is rated independently per sample. Both are standardized within the beta sweep. The trade-off column is 0.67⋅zpairwise+0.33⋅zcoherence ; β=0.7 is best under this criterion. The random baseline’s raw coherence mean in this run is 3.32.
Figure 18: Decoder stability under repeated reconstruction. For each method, five atom combinations were reconstructed five times with Claude Opus 4.7. Panel A shows within-idea pairwise cosine similarities between reconstructions, and Panel B shows atom-combination means with SEM across ideas. The decoder is generally stable, but occasional lower-similarity pairs indicate that a fixed atom combination can admit multiple distinct reconstructions.
Rank
Decoder
Open
Parameters
BT score
Win share
Mean words
1
Kimi K3
yes
2.8T (104B active)
4.50
0.976
1,340
2
Claude Opus 4.7
no
—
2.38
0.809
1,086
3
Qwen 3.6 27B
yes
27B
0.93
0.623
1,079
4
GLM 5.2
yes
744B (40B active)
0.46
0.549
862
5
Gemini 3.1 Pro
no
—
−0.42
0.404
786
6
Ministral 3 8B
yes
8.4B
−0.95
0.320
1,429
Appendix
Table 11: Decoder ablation on 50 fixed Alien combinations. Parameter counts are from the official model cards (not disclosed for proprietary models). Bradley–Terry (BT) scores are position-adjusted; win share counts ties as half wins; mean words is the average length of each decoder’s reconstructions.
Figure 19: Clustering method and hyperparameter selection. Top: quality as a function of the number of clusters across candidate methods, with the selected UMAP+HDBSCAN base configuration marked. Bottom left: best non-degenerate configuration by method. Bottom right: UMAP+HDBSCAN quality across HDBSCAN min_cluster_size and min_samples . Quality is Q=(overlapARI+intruderaccuracy)/2 .
Figure 20: Parameter sensitivity for UMAP+HDBSCAN and Leiden candidates. Lines show mean LLM-judged quality as one parameter varies, with bands indicating variability across the remaining parameters. The selected UMAP+HDBSCAN configuration lies in a broad high-quality region rather than depending on a single narrow parameter setting.
Figure 21: Clustering quality as noise units are reassigned to increase atom coverage. Higher coverage increases the average number of atoms per paper, which is the key source of training signal for the downstream co-occurrence models, but it also makes clusters less semantically pure. We choose 80% coverage: it increases the average from 1.65 to 3.38 atoms per paper while keeping the minimum reassignment similarity above 0.8 and preserving reasonable LLM-judged cluster separability.
Figure 22: Direct diagnostics of the learned coherence model for k=3 (a) and k=4 (b), and the availability model for k=3 (c) and k=4 (d). Panels (a)–(b) evaluate the coherence model on paper matches, random combinations, and pairwise-disjoint combinations, while panels (c)–(d) evaluate the availability model on combinations with high author support, random support, and zero author support. Higher scores indicate greater coherence in panels (a)–(b) and greater cognitive availability in panels (c)–(d).
Top 1,000
Top 10,000
β
All targets
Exact-new
All targets
Exact-new
Exact-new, least available third
0.0
160 (75.4 × )
18 (9.7 × )
762 (35.9 × )
463 (24.9 × )
5 (0.8 × )
0.1
148 (69.8 × )
10 (5.4 × )
763 (36.0 × )
464 (25.0 × )
10 (1.6 × )
0.2
129 (60.8 × )
7 (3.8 × )
733 (34.6 × )
434 (23.3 × )
13 (2.1 × )
0.3
109 (51.4 × )
3 (1.6 × )
663 (31.3 × )
368 (19.8 × )
32 (5.2 × )
0.4
77 (36.3 × )
3 (1.6 × )
453 (21.4 × )
194 (10.4 × )
43 (6.9 × )
Appendix
Table 12: Exact recovery of 2025 three-atom sets with every component frozen at 2024: hits among the top K of all 1,143,135 candidates, with enrichment over the random expectation. Exact-new targets exclude exact pre-2025 paper representations but may occur within larger earlier atom sets; they differ from the strictly novel targets in Figure 3 . For β≥0.8 , at most four targets are recovered.
Protocol
n
AUC, author → target
AUC, target → author
MRR / random
Never-published combination
2,779
0.656 [0.646, 0.666]
0.673 [0.663, 0.683]
4.2 × / 4.7 ×
Withheld real paper
1,983
0.551 [0.539, 0.564]
0.563 [0.550, 0.576]
1.7 × / 1.6 ×
Strict disjoint papers
762
0.592 [0.572, 0.612]
0.606 [0.586, 0.625]
1.9 × / 1.8 ×
Atom-overlap baselines
0.445–0.451
0.445–0.451
0.2–0.3 ×
Context size only
0.500
0.500
—
Appendix
Table 13: Held-out-author test with 95% bootstrap confidence intervals. Chance AUC is 0.5. All model AUCs exceed chance with Wilcoxon p<10−14 . Overlap baselines (atom Jaccard, binary atom cosine, overlap count) fall below chance because the target shares no atoms with the visible repertoire by construction.
Method
Atoms used
Unique triples
Top-10 concentration ↓
Vendi ↑
Alien ( β=0.7 )
251
300
34.3%
225.2
Claude Opus 4.7, batch
235
277
32.3%
197.9
Gemini 3.1 Pro, batch
120
294
28.0%
182.7
Gemini 3.1 Pro, independent
118
234
76.7%
97.4
Claude Opus 4.7, independent
85
190
95.7%
56.5
Uniform random
266
300
25.0%
250.4
Appendix
Table 14: Additional diversity statistics of each method’s 300 combinations.
Figure 23: Diversity accumulation for the same 300-combination portfolios as Table 14 : top-10 atom-slot share, effective atoms, and unique-pair and unique-triple fractions.
Method
Distinct pairs / 900 occurrences
Distinct triples / 300 samples
Claude independent
286
190
Gemini independent
418
234
Claude batch
656
277
Gemini batch
586
294
Alien ( β=0.7 )
794
300
Random
888
300
Appendix
Table 15: Distinct unordered atom pairs and triples in each 300-sample generation condition. Duplicates are retained in the sample budget but counted once in the overlap analysis.
Comparison
Two-atom pairs
Three-atom combinations
Claude independent ↔ Gemini independent
73 (11.57%)
20 (4.95%)
Claude batch ↔ Gemini batch
101 (8.85%)
9 (1.60%)
Claude independent ↔ Claude batch
44 (4.90%)
2 (0.43%)
Gemini independent ↔ Gemini batch
40 (4.15%)
1 (0.19%)
Claude independent ↔ Gemini batch
29 (3.44%)
2 (0.41%)
Gemini independent ↔ Claude batch
59 (5.81%)
4 (0.79%)
Appendix
Table 16: Shared count (Jaccard percentage) for distinct unordered pairs and triples. Independent denotes one combination per call; Alien uses β=0.7 . All conditions contain 300 samples.
Method
Age index
First year
Emergence
Papers/atom
Papers/top-10 atom
Alien ( β=0.7 )
0.96
2018.68
2022.55
131.2
108.6
Claude, independent
1.01
2018.51
2022.49
141.2
114.6
Gemini, independent
1.00
2018.48
2022.40
154.2
141.5
Claude, batch
0.99
2018.04
2022.48
210.4
244.8
Gemini, batch
0.95
2018.22
2022.56
210.2
128.0
Uniform random
0.98
2018.26
2022.50
197.6
174.4
Appendix
Table 17: Publication age and popularity of selected atoms. Values are averaged over the 900 atom occurrences in each method’s 300 combinations, so repeatedly selected atoms count repeatedly. The final column averages the number of papers containing each of the method’s ten most-selected atoms. Across the full vocabulary, the average is 198.2 papers per atom.
Method
Full 50
Top 25
Top 10
Random
41.1%
41.1%
41.7%
Alien sampler ( β=0.7 )
55.6%
72.2%
69.4%
Gemini 3.1 Pro
54.4%
58.9%
58.3%
Claude Opus 4.7
48.9%
27.8%
30.6%
Appendix
Table 18: Raw method win rates in the coherent-unavailability pairwise judge. Each entry is the fraction of all matches involving that method that the method won in the corresponding round.
Method
Full 50
Top 25
Top 10
Random
0.763
0.749
0.766
Alien sampler ( β=0.7 )
1.184
2.109
1.888
Gemini 3.1 Pro
1.144
1.336
1.305
Claude Opus 4.7
0.967
0.474
0.530
Appendix
Table 19: Bradley–Terry method abilities fit separately within each judge round and normalized so that the method mean is 1. Scores above 1 indicate above-average pairwise preference under the coherent-unavailability judge for that round.
Figure 24: Bradley–Terry method abilities across elimination rounds. Alien strengthens sharply after selecting the top half of each method’s outputs, while the random baseline remains flat and Claude drops under the coherent-unavailability criterion.
Figure 25: Pairwise win-rate matrices for the same judge rounds. Each cell is the row method’s win rate against the column method; the diagonal is omitted. The black outline highlights the Alien sampler row.
Method
n
Coherence
Feasibility
Novelty
Promise
Overall
Alien
10
4.20±0.20
3.90±0.23
3.90±0.28
4.10±0.18
4.03±0.11
Claude
10
4.30±0.30
3.60±0.22
3.80±0.29
4.40±0.27
4.03±0.23
Gemini
10
4.00±0.33
3.20±0.29
3.70±0.37
3.80±0.39
3.68±0.30
Random
10
3.70±0.21
3.70±0.21
3.60±0.22
4.00±0.15
3.75±0.04
Appendix
Table 20: Human ratings of generated idea descriptions. Entries are means ± standard errors over clean reviews.
Figure 26: Human survey landing page. Reviewers entered a study-issued panelist ID and read brief instructions before starting. Each session covered four idea blogposts, presented one at a time, with the source method of each idea hidden.
Figure 27: Idea reading view. Each blogpost was rendered in full (with the title and word count shown above the body) and only the current idea was visible at any time. A progress indicator on the top right tracked completion across the four ideas.
Figure 28: Read gate. The “Write review” button was disabled for the first 45 seconds on each idea, enforcing a minimum reading window before any rating could be entered.
Figure 29: Rating form. For each idea, reviewers wrote a brief free-text summary, rated five ordinal dimensions on five-point scales—coherence (“How coherent is this as a research direction?”), feasibility (“How likely is this idea to work?”), obviousness (“How likely is it that a typical active AI/ML researcher would independently propose this idea?”), novelty (“How novel is this idea?”), and worthwhileness (“How worthwhile would this be for a researcher or lab to pursue?”)—and provided a short free-text justification.
Method
Avg. rank percentile
Reports
Top 5
Top 10
Median rank
Alien
0.633
10
3
4
14.5
Claude
0.574
10
1
4
14.0
Gemini
0.495
10
1
2
24.0
Random
0.295
8
0
0
28.0
Appendix
Table 21: LLM-judge rankings of completed reports. Each method attempted 10 ideas; mean scores and median ranks include the two Random failures as described above.
Method
Original ideas in cluster
Final reports in cluster
Alien
0/10
0/10
Claude
2/10
8/10
Gemini
4/10
5/10
Random
0/10
1/8
Appendix
Table 22: Concentration around the SAE / feature-level mechanistic-interpretability cluster before and after autoresearch. Claude and Gemini reports are much more likely to route through this toolkit than Alien reports.
Method
Final reports
Own nearest
Own in top 3
Mean own rank
Alien
10
7
8
2.40
Claude
10
7
9
2.30
Gemini
10
8
9
2.60
Random
8
3
6
2.88
Appendix
Table 23: Title-level identity preservation through the autoresearch pipeline. Most final reports remain closest, or close, to their own starting idea title.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers often rate LLM-generated ideas as more novel than those from human experts, raising hope that AI can accelerate scientific discovery. Yet it is unclear whether these models can do more than extend existing knowledge. Can LLMs drive genuine innovation by revealing new research directions, or are they limited to repeating and recombining what science already knows? This unresolved question lies at the heart of discussions about AI and the future of science. To address this, we distinguish two types of novelty: distance, which measures how far a hypothesis lies from existing work, and direction, which captures whether the hypothesis pursues new lines of inquiry beyond those already present. These two measures diverge sharply for LLM-generated work. We represent scientific papers in a shared conceptual space using a pretrained scientific-paper embedding model. Within this space, we embed 26.9 million human-authored papers across 1,325 research topics in six natural-science fields and 524,308 LLM-generated hypotheses from eight models. Distance is measured using existing metrics. To quantify direction, we introduce the orthogonal novelty score (ONS). LLMs consistently generate ideas more distant from existing work than human-authored papers, up to 16% further when the LLM is given the papers farthest from the centre of the topic. However, LLM hypotheses have a 15--23% lower median ONS than published papers from the same topic. Thus, LLMs move further from what is already known, but remain within established directions. This pattern persists across changes in various configurations. As a result, LLMs excel at extending the frontier but the creation of new directions is still led by human scientists.
Zhihua Liu, Kurt Butler, Damian Machlanski +2
Causality in Healthcare AI Hub (CHAI) School of Engineering, University of Edinburgh
AI research agents now support large-scale AI-assisted scientific discovery. We examine whether AI-generated ideas broaden scientific exploration or primarily reinforce existing work. Using five agent frameworks and five large language models, we generate 219,655 ideas for different scientific fields. Across experiments, four consistent patterns emerge. First, AI-generated ideas are more concentrated than human-authored papers within the same research area. Second, they remain much closer to starting literature than later human follow-on work does. Third, AI-generated ideas align less with future human research. Last, AI-generated ideas are located in lower-impact regions of the historical scientific landscape. Overall, current AI research agents appear better suited to local elaboration than to broadening scientific exploration.
Yixuan Tang, Yi Yang
1*ISOM, The Hong Kong University of Science and Technology.
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.