The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions

Alejandro H. Artiles; Martin Weiss; Levin Brinkmann; Iyad Rahwan; Bernhard Schölkopf; Christopher Pal; Hugo Larochelle; Anirudh Goyal

cs.AIMar 1, 2026

The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions

Authors: Alejandro H. Artiles, Martin Weiss, Levin Brinkmann, Iyad Rahwan, Bernhard Schölkopf, Christopher Pal, Hugo Larochelle, Anirudh Goyal,

Organizations: Tiptree Systems · Max Planck Institute for Human Development, Berlin · Max Planck Institute for Intelligent Systems, Tübingen · ELLIS Institute Tübingen ·

Abstract

Scientific discovery is constrained not only by what is true, but by what is cognitively available to researchers. Many directions are coherent in light of the literature yet unlikely to be proposed because no existing community combines the relevant concepts, methods, and intuitions. Modern language models inherit this bias, recombining high-density regions of the literature when prompted for novel ideas. We target the complementary region, the alien space of science: directions that are scientifically plausible but unlikely under the distribution of existing researchers. Our method decomposes papers into conceptual units and clusters them into a vocabulary of idea atoms. A coherence model scores whether atom combinations form viable research directions, while an availability model scores whether any existing author community is positioned to produce them. We rank combinations to maximize coherence while minimizing availability. On 16,068 peer-reviewed LLM papers from NeurIPS, ICLR, ICML, and major NLP venues, the sampler produces more diverse research portfolios than frontier LLM baselines, including batch-prompted variants that see their entire output, while matching or exceeding their per-idea quality under blind human, LLM, and downstream experimental evaluation. In temporal validation, all components use only pre-2025 data. Selecting less than 1% of the candidate space recovers 2025 combinations at 35.9 times chance; weighting unavailability increases recovery of strictly novel combinations to 3.24 times chance. By separating scientific plausibility from community availability, our framework points toward AI ideation that complements rather than merely accelerates human science, expanding exploration into coherent directions the community may overlook.

Figures & tables

Figure 1: Overview of the Alien Science Sampling pipeline. Papers are distilled into conceptual units, which are clustered into a shared vocabulary of idea atoms. A coherence model learns which atom combinations form viable research directions, while an availability model estimates which combinations existing author communities are positioned to produce. Alien directions are sampled by maximizing coherence while minimizing availability.

Panel of Figure 2 — Figure 2: Two-encoder contrastive training for cognitive availability. A held-out query subset $Q_{i}$ is encoded by $\phi$ ; the complement $R_{i}\!\setminus\!Q_{i}$ and other authors’ complements are encoded by $\psi$ . Disjoint views of the same author are pulled together, while views from different authors are pushed apart.

Figure 3: Frozen temporal validation at

K=10{,}000

(0.875% of 1,143,135 candidates). Left: Recovery relative to uniform random ranking for all 2,424 targets and the 1,411 strictly novel targets (log scale; dashed line: chance). A triple is strictly novel if its atoms never co-occurred in any pre-2025 corpus paper. Right: Strictly novel targets as a percentage of all recovered targets; labels give novel/total hits. At

\beta=0.9

, no targets are recovered, so zero enrichment and undefined novelty share are omitted.

	Portfolio diversity (300 ideas)					Diversity–quality composite $\uparrow$
Method	Atoms	Eff. atoms $\uparrow$	Vendi $\uparrow$	Rep. pairs $\downarrow$	Near-dup. $\downarrow$	Human	Tournament	Autoresearch
Alien (ours)	251	198.0	225.2	106	51.3%	0.906	0.917	0.977
Claude, batch ∗	235	191.7	197.9	244	81.7%	0.708	0.049	0.506
Gemini, batch ∗	120	118.1	182.7	314	94.0%	$-0.467$	0.489	0.030
Gemini, independent	118	56.2	97.4	482	94.0%	$-1.084$	$-0.129$	$-0.588$
Claude, independent	85	28.7	56.5	614	96.3%	$-0.316$	$-0.975$	$-0.519$

Table 1: Portfolio diversity of each method’s 300 ideas, and the equal-weight composite

\tfrac{1}{2}[z(\text{Vendi})+z(\text{quality})]

using three evaluation measures (Section 5.4 ). Bold marks the best non-random method. Uniform random is the diversity ceiling but optimizes neither coherence nor availability. ∗ Batch variants were not separately rated and inherit the quality of the same model’s independent run.

Panel of Figure 4 — Figure 4: Generated ideas in embedding and score space. Left: UMAP embedding of method digests from the 16,068-paper corpus and decoded ideas from Claude, Gemini, Alien, and Random samples. Claude and Gemini concentrate around sparse-autoencoder and interpretability concepts, while Alien samples occupy a broader region. Right: Coherence–availability scores of generated three-atom combinations; higher coherence and lower availability are preferred. Alien concentrates in the intended high-coherence, low-availability region, LLM baselines stay in more available regions, and random sampling loses coherence.

Figure 5: Interpretability motifs in 10 initial ideas and final reports (after autoresearch loop) per method. (Appendix E.4 ).

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Value
Metadata sources	OpenReview, DBLP, Semantic Scholar
Selection filter	LLM match in title, abstract, or keywords
Venues	NeurIPS, EMNLP, ACL, ICLR, ICML, AAAI, COLM, NAACL, IJCAI
Years	2017–2025
Total papers	16,068

Appendix

Table 2: Corpus construction summary.

Venue	2017	2018	2019	2020	2021	2022	2023	2024	2025	Total
NeurIPS	0	3	12	19	50	115	348	955	1,720	3,222
EMNLP	12	32	0	0	0	272	500	954	1,253	3,023
ACL	9	18	47	106	0	173	413	636	1,221	2,623
ICLR	16	27	19	55	85	108	285	559	1,395	2,549
ICML	2	4	6	8	15	32	110	536	956	1,669
AAAI	2	13	12	40	52	66	95	307	758	1,345

Appendix

Table 3: Distribution of LLM-filtered papers by venue and year. Zero entries indicate years with no papers from that venue in the filtered database.

Metric	Value
Input conceptual units	82,255
Initial HDBSCAN clusters	273
Initial noise units	62.8%
Selected coverage after reassignment	80%
Mean atoms per paper before reassignment	1.65
Mean atoms per paper at selected coverage	3.38

Appendix

Table 9

Figure 6: Distribution of papers per initial atom cluster, before noise reassignment. Most atom clusters are supported by tens of papers, while a long tail of broad, cross-cutting clusters pulls the mean (81.1) above the median (44).

Figure 7: Author connectivity is coupled to atom coverage. Each point is an author node, positioned by unique atom coverage and distinct coauthor degree; color encodes $\log(1+\mathrm{publications})$ , and red points mark the top ten authors by atom coverage. The association shows that broad author profiles are also socially well connected, so individual prolific authors can bridge distant regions of the atom space.

Quantity	Coherence data	Availability data
Training objective	Next-atom prediction	Author–set contrastive retrieval
Training records used	136,827	43,077,647
Validation records used	15,203	4,786,404
Source entities	15,203 papers	44,809 eligible authors
Positive input length	Mean 3.50; median 3; p90 5; max 8	Mean 3.80; median 4; p90 4; max 4
Author-tower input length	–	Mean 27.66; median 24; p90 48; max 143

Appendix

Table 4: Training-record characteristics for the coherence and availability models. Lengths count atom IDs, excluding special sequence tokens for coherence. Availability statistics are computed after dropping empty-complement records, matching the records consumed by the dual-encoder loader.

Parameter	Value
Epochs	30
Batch size	64
Gradient accumulation steps	2
Effective batch size	128
Learning rate	$1\times 10^{-4}$
Warmup steps	1,000

Appendix

Table 5: Run settings and architecture details for the coherence transformer. Architecture values not passed explicitly in the command use the training script defaults. Vocabulary size is 273 atom tokens plus a shared BOS/EOS token.

Figure 8: Order sensitivity of coherence scores. Points show the mean coherence score for random, real-paper coherent, and pairwise-disjoint incoherent atom sets; horizontal bars show

\pm 1

standard deviation across permutations of the same atom set. Scores are

-\mathrm{NLL}

, so higher is more coherent. Although scores vary with atom order, the permutation-induced variation is small relative to the separation between real-paper coherent sets and the random/incoherent baselines.

Parameter	Value
Epochs	100
Optimizer	AdamW
Global batch / per GPU	4,096 / 1,024
Gradient accumulation steps	1
Peak learning rate	$5\times 10^{-4}$
Learning-rate schedule	2,000-step linear warmup; cosine decay

Appendix

Table 6: Training settings for the dual-encoder availability model. The 100-epoch run took 13 h 42 min.

Model	Conditioning	Paper $>$ zero	Author-only $>$ zero	Paper $>$ author-only
Density estimator	Random negatives	0.683	0.708	0.445
Dual encoder	Author–set compatibility	0.996	0.792	0.950

Appendix

Table 7: AUC-ROC over all 50,036 paper-supported, 2,636,735 author-only and 667,125 zero-support triplets. Both models score exactly the same sets and support labels.

Model	Community $>$ prolific	Paper $>$ prolific	Prolific in top quartile
Density estimator	0.1975	0.4293	21.0%
Dual encoder	0.9999	1.0000	0.0%

Appendix

Table 8: Prolific-author diagnostic on five matched 500-triplet pools. The top quartile contains the 625 highest-scoring sets across all 2,500 candidates; ties use fixed candidate order. Prolific-only controls have no paper support.

Method	Comm. $>$ prolific	Comm. $>$ zero	Comm. $>$ random	Control share top Q
Dual encoder	0.9999	0.9999	0.9689	6.4%
Atom-author W2V, $\alpha=1.0$ (default)	0.9525	0.9597	0.9100	24.6%
Atom-author W2V, $\alpha=0.5$	0.8436	0.8397	0.7572	50.6%
Atom-author W2V, $\alpha=0.1$	0.6095	0.5794	0.5026	76.6%

Appendix

Table 9: Availability diagnostics on four matched pools of 500 triplets: community-supported without paper support, prolific-only, zero-support, and random. The final column denotes the fraction of the highest-scoring quartile that is prolific-only, zero-support, or random.

Figure 9: Availability distributions on 1,500 matched triplets: 500 with highest author support, 500 zero-support sets, and 500 random controls. These histogram pools allow paper support and differ from the no-paper community pools in Table 9 . The same sets are used for all four methods.

Figure 10: Availability on the four matched support pools of Table 9 . Panel A shows per-method standardized means over the 2,000 candidates, with

1.96

standard-error bars. Panel B gives community-versus-control AUCs. Hypergraph

\alpha=1

balances author and atom exploration; lower values increasingly weight author transitions.

Figure 11: Model-disagreement quadrants on the same 2,000 support-controlled triplets. Dashed lines mark each method's 75th percentile. The hypergraph-high/dual-low quadrant contains 131 sets, of which 80.9% belong to control pools; the dual-high/hypergraph-low quadrant also contains 131 sets, with 11.5% controls.

Figure 12: Sensitivity to author/atom walk balance. The balanced atom-author hypergraph default (

\alpha=1

) is the strongest setting. Panel A reports availability-only AUCs for community-supported triplets against three control classes. Panel B shows that lower alpha values increase the fraction of prolific-only, zero-support, and random candidates in the hypergraph high-scoring region.

Figure 13: Dense author-mediated paths collapse the atom space. In the shared AI hypergraph, publication incidence already reaches all atoms by two steps, and adding author nodes reaches the full atom universe immediately under the semantic-hop convention. This explains why availability is difficult to infer from graph reachability alone.

Figure 14: Availability over all 3,353,896 three-atom combinations. Lower scores are less available. The shaded region is the bottom 10%; vertical lines mark its boundary and the means of the fixed random sample and the numerical Alien top 300 at

\beta=0.7

Figure 15: Reconstruction-model capability across beta values. The same beta-selected atom combinations are reconstructed with different decoders. Coherence and absolute originality are rated independently per sample; dashed lines show the random baseline. Stronger reconstructors extract more signal from the same recombinations, but the coherence–originality trend as

\beta

changes is similar across decoder capacity.

Figure 16: Pairwise originality comparison between reconstructors while holding atom sets fixed. Each cell is the focal reconstructor’s win rate against the opponent. Claude Opus 4.7 is the strongest of the three decoders compared here, so the main beta choice is made in the Claude reconstruction setting.

Figure 17: Beta selection for the final

k=3

Claude Opus 4.7 setting. Pairwise originality is obtained from beta-versus-beta originality comparisons; coherence is rated independently per reconstructed sample. The selection score combines the two standardized signals, and is maximized at

\beta=0.7

$\beta$	Pairwise originality $z$	Coherence $z$	Coherence mean	Trade-off
0.6	0.36	0.21	3.38	0.31
0.7	0.86	0.49	3.44	0.74
0.8	1.00	$-0.55$	3.22	0.49
0.9	0.95	$-0.91$	3.14	0.34
1.0	0.95	$-2.60$	2.78	$-0.22$

Appendix

Table 10: Beta-selection neighborhood for the final

k=3

Claude Opus 4.7 reconstruction setting. Pairwise originality comes from beta-versus-beta originality comparisons; coherence is rated independently per sample. Both are standardized within the beta sweep. The trade-off column is

0.67\cdot z_{\mathrm{pairwise}}+0.33\cdot z_{\mathrm{coherence}}

;

\beta=0.7

is best under this criterion. The random baseline’s raw coherence mean in this run is 3.32.

Figure 18: Decoder stability under repeated reconstruction. For each method, five atom combinations were reconstructed five times with Claude Opus 4.7. Panel A shows within-idea pairwise cosine similarities between reconstructions, and Panel B shows atom-combination means with SEM across ideas. The decoder is generally stable, but occasional lower-similarity pairs indicate that a fixed atom combination can admit multiple distinct reconstructions.

Rank	Decoder	Open	Parameters	BT score	Win share	Mean words
1	Kimi K3	yes	2.8T (104B active)	4.50	0.976	1,340
2	Claude Opus 4.7	no	—	2.38	0.809	1,086
3	Qwen 3.6 27B	yes	27B	0.93	0.623	1,079
4	GLM 5.2	yes	744B (40B active)	0.46	0.549	862
5	Gemini 3.1 Pro	no	—	$-0.42$	0.404	786
6	Ministral 3 8B	yes	8.4B	$-0.95$	0.320	1,429

Appendix

Table 11: Decoder ablation on 50 fixed Alien combinations. Parameter counts are from the official model cards (not disclosed for proprietary models). Bradley–Terry (BT) scores are position-adjusted; win share counts ties as half wins; mean words is the average length of each decoder’s reconstructions.

Figure 19: Clustering method and hyperparameter selection. Top: quality as a function of the number of clusters across candidate methods, with the selected UMAP+HDBSCAN base configuration marked. Bottom left: best non-degenerate configuration by method. Bottom right: UMAP+HDBSCAN quality across HDBSCAN min_cluster_size and min_samples . Quality is $Q=(\mathrm{overlap\ ARI}+\mathrm{intruder\ accuracy})/2$ .

Figure 20: Parameter sensitivity for UMAP+HDBSCAN and Leiden candidates. Lines show mean LLM-judged quality as one parameter varies, with bands indicating variability across the remaining parameters. The selected UMAP+HDBSCAN configuration lies in a broad high-quality region rather than depending on a single narrow parameter setting.

Figure 21: Clustering quality as noise units are reassigned to increase atom coverage. Higher coverage increases the average number of atoms per paper, which is the key source of training signal for the downstream co-occurrence models, but it also makes clusters less semantically pure. We choose 80% coverage: it increases the average from 1.65 to 3.38 atoms per paper while keeping the minimum reassignment similarity above 0.8 and preserving reasonable LLM-judged cluster separability.

Figure 22: Direct diagnostics of the learned coherence model for

k=3

(a) and

k=4

(b), and the availability model for

k=3

k=4

(d). Panels (a)–(b) evaluate the coherence model on paper matches, random combinations, and pairwise-disjoint combinations, while panels (c)–(d) evaluate the availability model on combinations with high author support, random support, and zero author support. Higher scores indicate greater coherence in panels (a)–(b) and greater cognitive availability in panels (c)–(d).

	Top 1,000		Top 10,000
$\beta$	All targets	Exact-new	All targets	Exact-new	Exact-new, least available third
0.0	160 (75.4 $\times$ )	18 (9.7 $\times$ )	762 (35.9 $\times$ )	463 (24.9 $\times$ )	5 (0.8 $\times$ )
0.1	148 (69.8 $\times$ )	10 (5.4 $\times$ )	763 (36.0 $\times$ )	464 (25.0 $\times$ )	10 (1.6 $\times$ )
0.2	129 (60.8 $\times$ )	7 (3.8 $\times$ )	733 (34.6 $\times$ )	434 (23.3 $\times$ )	13 (2.1 $\times$ )
0.3	109 (51.4 $\times$ )	3 (1.6 $\times$ )	663 (31.3 $\times$ )	368 (19.8 $\times$ )	32 (5.2 $\times$ )
0.4	77 (36.3 $\times$ )	3 (1.6 $\times$ )	453 (21.4 $\times$ )	194 (10.4 $\times$ )	43 (6.9 $\times$ )

Appendix

Table 12: Exact recovery of 2025 three-atom sets with every component frozen at 2024: hits among the top

K

of all 1,143,135 candidates, with enrichment over the random expectation. Exact-new targets exclude exact pre-2025 paper representations but may occur within larger earlier atom sets; they differ from the strictly novel targets in Figure 3 . For

\beta\geq 0.8

, at most four targets are recovered.

Protocol	$n$	AUC, author $\rightarrow$ target	AUC, target $\rightarrow$ author	MRR / random
Never-published combination	2,779	0.656 [0.646, 0.666]	0.673 [0.663, 0.683]	4.2 $\times$ / 4.7 $\times$
Withheld real paper	1,983	0.551 [0.539, 0.564]	0.563 [0.550, 0.576]	1.7 $\times$ / 1.6 $\times$
Strict disjoint papers	762	0.592 [0.572, 0.612]	0.606 [0.586, 0.625]	1.9 $\times$ / 1.8 $\times$
Atom-overlap baselines		0.445–0.451	0.445–0.451	0.2–0.3 $\times$
Context size only		0.500	0.500	—

Appendix

Table 13: Held-out-author test with 95% bootstrap confidence intervals. Chance AUC is 0.5. All model AUCs exceed chance with Wilcoxon

p<10^{-14}

. Overlap baselines (atom Jaccard, binary atom cosine, overlap count) fall below chance because the target shares no atoms with the visible repertoire by construction.

Method	Atoms used	Unique triples	Top-10 concentration $\downarrow$	Vendi $\uparrow$
Alien ( $\beta=0.7$ )	251	300	34.3%	225.2
Claude Opus 4.7, batch	235	277	32.3%	197.9
Gemini 3.1 Pro, batch	120	294	28.0%	182.7
Gemini 3.1 Pro, independent	118	234	76.7%	97.4
Claude Opus 4.7, independent	85	190	95.7%	56.5
Uniform random	266	300	25.0%	250.4

Appendix

Table 14: Additional diversity statistics of each method’s 300 combinations.

Figure 23: Diversity accumulation for the same 300-combination portfolios as Table 14 : top-10 atom-slot share, effective atoms, and unique-pair and unique-triple fractions.

Method	Distinct pairs / 900 occurrences	Distinct triples / 300 samples
Claude independent	286	190
Gemini independent	418	234
Claude batch	656	277
Gemini batch	586	294
Alien ( $\beta=0.7$ )	794	300
Random	888	300

Appendix

Table 15: Distinct unordered atom pairs and triples in each 300-sample generation condition. Duplicates are retained in the sample budget but counted once in the overlap analysis.

Comparison	Two-atom pairs	Three-atom combinations
Claude independent $\leftrightarrow$ Gemini independent	73 (11.57%)	20 (4.95%)
Claude batch $\leftrightarrow$ Gemini batch	101 (8.85%)	9 (1.60%)
Claude independent $\leftrightarrow$ Claude batch	44 (4.90%)	2 (0.43%)
Gemini independent $\leftrightarrow$ Gemini batch	40 (4.15%)	1 (0.19%)
Claude independent $\leftrightarrow$ Gemini batch	29 (3.44%)	2 (0.41%)
Gemini independent $\leftrightarrow$ Claude batch	59 (5.81%)	4 (0.79%)

Appendix

Table 16: Shared count (Jaccard percentage) for distinct unordered pairs and triples. Independent denotes one combination per call; Alien uses

\beta=0.7

. All conditions contain 300 samples.

Method	Age index	First year	Emergence	Papers/atom	Papers/top-10 atom
Alien ( $\beta=0.7$ )	0.96	2018.68	2022.55	131.2	108.6
Claude, independent	1.01	2018.51	2022.49	141.2	114.6
Gemini, independent	1.00	2018.48	2022.40	154.2	141.5
Claude, batch	0.99	2018.04	2022.48	210.4	244.8
Gemini, batch	0.95	2018.22	2022.56	210.2	128.0
Uniform random	0.98	2018.26	2022.50	197.6	174.4

Appendix

Table 17: Publication age and popularity of selected atoms. Values are averaged over the 900 atom occurrences in each method’s 300 combinations, so repeatedly selected atoms count repeatedly. The final column averages the number of papers containing each of the method’s ten most-selected atoms. Across the full vocabulary, the average is 198.2 papers per atom.

Method	Full 50	Top 25	Top 10
Random	41.1%	41.1%	41.7%
Alien sampler ( $\beta=0.7$ )	55.6%	72.2%	69.4%
Gemini 3.1 Pro	54.4%	58.9%	58.3%
Claude Opus 4.7	48.9%	27.8%	30.6%

Appendix

Table 18: Raw method win rates in the coherent-unavailability pairwise judge. Each entry is the fraction of all matches involving that method that the method won in the corresponding round.

Method	Full 50	Top 25	Top 10
Random	0.763	0.749	0.766
Alien sampler ( $\beta=0.7$ )	1.184	2.109	1.888
Gemini 3.1 Pro	1.144	1.336	1.305
Claude Opus 4.7	0.967	0.474	0.530

Appendix

Table 19: Bradley–Terry method abilities fit separately within each judge round and normalized so that the method mean is 1. Scores above 1 indicate above-average pairwise preference under the coherent-unavailability judge for that round.

Figure 24: Bradley–Terry method abilities across elimination rounds. Alien strengthens sharply after selecting the top half of each method’s outputs, while the random baseline remains flat and Claude drops under the coherent-unavailability criterion.

Figure 25: Pairwise win-rate matrices for the same judge rounds. Each cell is the row method’s win rate against the column method; the diagonal is omitted. The black outline highlights the Alien sampler row.

Method	$n$	Coherence	Feasibility	Novelty	Promise	Overall
Alien	10	$4.20\pm 0.20$	$\mathbf{3.90\pm 0.23}$	$\mathbf{3.90\pm 0.28}$	$4.10\pm 0.18$	$4.03\pm 0.11$
Claude	10	$\mathbf{4.30\pm 0.30}$	$3.60\pm 0.22$	$3.80\pm 0.29$	$\mathbf{4.40\pm 0.27}$	$4.03\pm 0.23$
Gemini	10	$4.00\pm 0.33$	$3.20\pm 0.29$	$3.70\pm 0.37$	$3.80\pm 0.39$	$3.68\pm 0.30$
Random	10	$3.70\pm 0.21$	$3.70\pm 0.21$	$3.60\pm 0.22$	$4.00\pm 0.15$	$3.75\pm 0.04$

Appendix

Table 20: Human ratings of generated idea descriptions. Entries are means

\pm

standard errors over clean reviews.

Figure 26: Human survey landing page. Reviewers entered a study-issued panelist ID and read brief instructions before starting. Each session covered four idea blogposts, presented one at a time, with the source method of each idea hidden.

Figure 27: Idea reading view. Each blogpost was rendered in full (with the title and word count shown above the body) and only the current idea was visible at any time. A progress indicator on the top right tracked completion across the four ideas.

Figure 28: Read gate. The “Write review” button was disabled for the first 45 seconds on each idea, enforcing a minimum reading window before any rating could be entered.

Figure 29: Rating form. For each idea, reviewers wrote a brief free-text summary, rated five ordinal dimensions on five-point scales—coherence (“How coherent is this as a research direction?”), feasibility (“How likely is this idea to work?”), obviousness (“How likely is it that a typical active AI/ML researcher would independently propose this idea?”), novelty (“How novel is this idea?”), and worthwhileness (“How worthwhile would this be for a researcher or lab to pursue?”)—and provided a short free-text justification.

Method	Avg. rank percentile	Reports	Top 5	Top 10	Median rank
Alien	0.633	10	3	4	14.5
Claude	0.574	10	1	4	14.0
Gemini	0.495	10	1	2	24.0
Random	0.295	8	0	0	28.0

Appendix

Table 21: LLM-judge rankings of completed reports. Each method attempted 10 ideas; mean scores and median ranks include the two Random failures as described above.

Method	Original ideas in cluster	Final reports in cluster
Alien	0/10	0/10
Claude	2/10	8/10
Gemini	4/10	5/10
Random	0/10	1/8

Appendix

Table 22: Concentration around the SAE / feature-level mechanistic-interpretability cluster before and after autoresearch. Claude and Gemini reports are much more likely to route through this toolkit than Alien reports.

Method	Final reports	Own nearest	Own in top 3	Mean own rank
Alien	10	7	8	2.40
Claude	10	7	9	2.30
Gemini	10	8	9	2.60
Random	8	3	6	2.88

Appendix

Table 23: Title-level identity preservation through the autoresearch pipeline. Most final reports remain closest, or close, to their own starting idea title.

Research Idea Generation

Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.

See all 60 papers

Large Language Models Extend Known Research But Do Not Create New Research Directions
Oct 12, 2026Zhihua Liu, Kurt Butler, Damian Machlanski +2Large Language Model-Guided Scientific Discovery Scientific Hypothesis Generation
Causality in Healthcare AI Hub (CHAI) School of Engineering, University of Edinburgh
AI Research Agents Narrow Scientific Exploration
May 27, 2026Yixuan Tang, Yi YangAI Agents for Scientific Discovery AI Agent Evaluation
1*ISOM, The Hong Kong University of Science and Technology.
Measuring the Gap Between Human and LLM Research Ideas
Jul 1, 2026Ziyu Chen, Yilun Zhao, Arman CohanLLM Evaluation Scientific Hypothesis Generation
University of Chicago · Yale University

Oct 12, 2026J/K move · Enter open · S save

Large Language Models Extend Known Research But Do Not Create New Research Directions

Zhihua Liu, Kurt Butler, Damian Machlanski, Danilo Bueno+1

Causality in Healthcare AI Hub (CHAI) School of Engineering, University of Edinburgh

Large Language Model-Guided Scientific Discovery Scientific Hypothesis Generation Research Idea Generation Scientific Reasoning in Language Models

Open paper AskPDF arXiv

The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers often rate LLM-generated ideas as more novel than those from human experts, raising hope that AI can accelerate scientific discovery. Yet it is unclear whether these models can do more than extend existing knowledge. Can LLMs drive genuine innovation by revealing new research directions, or are they limited to repeating and recombining what science already knows? This unresolved question lies at the heart of discussions about AI and the future of science. To address this, we distinguish two types of novelty: distance, which measures how far a hypothesis lies from existing work, and direction, which captures whether the hypothesis pursues new lines of inquiry beyond those already present. These two measures diverge sharply for LLM-generated work. We represent scientific papers in a shared conceptual space using a pretrained scientific-paper embedding model. Within this space, we embed 26.9 million human-authored papers across 1,325 research topics in six natural-science fields and 524,308 LLM-generated hypotheses from eight models. Distance is measured using existing metrics. To quantify direction, we introduce the orthogonal novelty score (ONS). LLMs consistently generate ideas more distant from existing work than human-authored papers, up to 16% further when the LLM is given the papers farthest from the centre of the topic. However, LLM hypotheses have a 15--23% lower median ONS than published papers from the same topic. Thus, LLMs move further from what is already known, but remain within established directions. This pattern persists across changes in various configurations. As a result, LLMs excel at extending the frontier but the creation of new directions is still led by human scientists.

Abstract

Research Idea Generation

Explore similar work

Large Language Models Extend Known Research But Do Not Create New Research Directions

AI Research Agents Narrow Scientific Exploration

Measuring the Gap Between Human and LLM Research Ideas