Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

Yujian Liu; Li An; Zelin He; Jacob Helwig; Jiabao Ji; Rohit Jain; Siyu Zhu; Shiyu Chang

cs.CLOct 20, 2026

Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

Authors: Yujian Liu, Li An, Zelin He, Jacob Helwig, Jiabao Ji, Rohit Jain, Siyu Zhu, Shiyu Chang

Organizations: UC Santa Barbara · Pennsylvania State University · LinkedIn

Abstract

Autonomous agents can increasingly propose ideas, modify code, and run experiments, creating an opportunity to accelerate scientific discovery using teams of collaborative AI researchers. To escape local minima in the discovery, we propose to encourage orthogonal exploration by constructing the team using agents with varying expertise. However, realizing this design requires more than diverse backgrounds. When experiments are costly, a team can test only a limited number of ideas, thereby creating the need to balance exploration and exploitation when selecting a subset of researchers to propose and execute new experiments. Furthermore, each result should shape what the team tries next. We therefore ask how AI researchers with different scientific backgrounds can build on one another's findings and sustain progress under a limited experimental budget. To address these challenges, we build a team of persistent AI researchers, each grounded in a different discipline, who keep private research histories, share experimental evidence, and receive research opportunities through adaptive allocation. We demonstrate the effectiveness of this design by deploying the team to search for linear attention architectures, where it outperforms GDN2 at 1.3B scale on language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks. In controlled comparisons, removing any single organizational component from our design yields markedly weaker discoveries: without disciplinary backgrounds, the search plateaus early; without private histories, researchers make less progress despite identical shared evidence; and uniform selection falls short of adaptive allocation within the same budget. These results suggest that sustained autonomous discovery depends not only on stronger agents, but also on who explores, what they remember, and how research effort is allocated.

Figures & tables

Figure 1: Illustration of collaborative autoresearch. Sixteen research agents with diverse scientific backgrounds collaborate on linear attention architecture search. Starting from Gated DeltaNet-2 (GDN2), the team discovers Context-First DeltaNet (CFDN), which improves performance across language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks.

Figure 2: Organization of the autonomous research team. (a) Sixteen researchers have distinct scientific backgrounds and persistent individual histories. Adaptive selection balances measured contributions with opportunities for less-tested researchers. (b) A selected researcher develops one proposal through review and, if approved, implementation and experimentation, then reflects on its outcome. (c) Proposals, outcomes, and their relationships form a shared body of knowledge through which researchers build on one another’s work. Circled numbers identify the four organizational factors studied in our comparisons.

		Search	Full evaluation
Model	Tokens	Utility	Wiki.	LMB.	CSR	NIAH
Model	(B)	$U\uparrow$	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$
GDN	15.00	-1.29	35.15	64.03	42.88	16.94
KDA	14.82	-0.59	34.63	56.64	41.94	22.03
GDN2	13.53	0.00	34.93	54.74	43.60	25.79
CFDN	13.53	1.04	33.84	52.33	43.29	29.76

Table 1: 150M comparison.

U

is the aggregated search utility; CSR is the mean of eight common-sense reasoning tasks; NIAH is the mean of four synthetic retrieval tasks. Full results are in Appendix B .

Model	Tokens	Wiki.	LMB.	CSR	NIAH	RWR
Model	(B)	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
GDN	100.00	16.27	12.04	53.73	54.99	33.17
KDA	99.24	16.05	11.66	53.90	59.11	32.47
GDN2	87.18	16.05	11.22	52.65	66.86	32.52
CFDN	87.17	15.74	11.04	54.53	67.29	33.41

Table 2: Full evaluation at 1.3B. CSR is the mean of eight common-sense reasoning tasks; NIAH is the mean of four synthetic retrieval tasks; RWR is the mean of six in-context real-world retrieval tasks. Full results are in Appendix B .

Figure 3: Candidate utility throughout the search. Points are evaluated candidates; lines are the best found candidates up to an iteration.

Search objective	Wiki.	LMB.	CSR	NIAH
Search objective	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$
LM only	34.59	46.40	42.25	26.73
LM $+$ CSR $+$ NIAH	33.84	52.33	43.29	29.76

Table 3: Effect of the search objective. Full evaluation of the model discovered at 150M scale.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Search scale	Large-scale comparison
Blocks	8	18
Residual / FFN width	1,024 / 4,096	2,304 / 6,208
GDN2 recurrent heads	$8\times 128$	$16\times 128$
GDN2 non-embedding parameters	155,322,432	1,302,640,416
GDN2 total parameters	220,858,432	1,450,096,416
Output gate	Sigmoid	Sigmoid

Appendix

Table 4: Model configurations. Recurrent-head geometry describes GDN2; the outer backbone is fixed within each scale. Scale labels exclude the input embedding and output vocabulary projection. Both remain trainable, and the output projection is included in reference FLOPs.

Hyperparameter	Search scale	1.3B scale
Optimizer	AdamW	AdamW
Peak learning rate	$4\times 10^{-4}$	$4\times 10^{-4}$
Minimum learning rate	$4\times 10^{-5}$	$4\times 10^{-5}$
Adam $\beta_{1},\beta_{2}$	0.9, 0.95	0.9, 0.95
Weight decay	0.1	0.1
Gradient clipping norm	1.0	1.0

Appendix

Table 5: Training hyperparameters. The optimizer settings and global batch size are shared by all architectures. Microbatch size denotes the sequences processed per device in one forward/backward pass.

Dataset / metric	GDN	KDA	GDN2	CFDN
WikiText-2 token PPL	14.39	14.23	14.32	13.99
LAMBADA token PPL	11.39	10.61	10.40	10.13
PIQA	63.28	61.33	63.28	63.28
HellaSwag (normalized)	34.77	34.38	34.38	36.33
WinoGrande	53.91	52.73	50.78	53.12
ARC-Easy	50.39	55.08	52.73	53.12

Appendix

Table 6: Search-scale LM and CSR feedback. CSR uses 256 fixed examples per task; accuracies are percentages. These reduced-suite scores determine the utilities in Table 1 , rather than its full-evaluation columns.

Task	Context	GDN	KDA	GDN2	CFDN
S-NIAH-1	1K	82.81	56.25	68.75	100.00
	2K	42.19	42.19	50.00	98.44
	4K	21.88	9.38	25.00	98.44
S-NIAH-2	1K	42.19	62.50	82.81	96.88
	2K	10.94	39.06	40.62	92.19
	4K	1.56	7.81	15.62	25.00

Appendix

Table 7: Search retrieval feedback (%). Each cell uses 64 fixed examples and exact teacher-forced continuation scoring, giving 768 requests overall.

Dataset / metric	GDN	KDA	GDN2	CFDN	LM only
WikiText-2 word PPL	35.15	34.63	34.93	33.84	34.59
LAMBADA PPL	64.03	56.64	54.74	52.33	46.40
LAMBADA accuracy	26.28	27.15	27.61	28.35	29.92
PIQA	63.60	63.76	64.53	64.64	63.17
HellaSwag (normalized)	34.25	34.94	34.57	35.22	34.47
WinoGrande	51.62	51.07	53.04	51.22	52.17

Appendix

Table 8: Complete 150M LM and CSR results for Tables 1 and 3 . The last column is the strongest model selected by LM-only search. Accuracies are percentages; LAMBADA accuracy is excluded from the eight-task CSR mean.

Task	Context	GDN	KDA	GDN2	CFDN	LM only
S-NIAH-1	1K	88.40	63.00	87.60	89.40	94.20
	2K	42.40	38.60	49.80	33.20	73.40
	4K	14.20	10.60	19.20	1.00	33.40
	8K	6.40	3.60	8.40	0.40	12.40
S-NIAH-2	1K	39.20	69.20	82.40	89.60	56.40
	2K	5.00	42.00	45.60	81.00	39.20

Appendix

Table 9: Complete 150M generated-answer NIAH results (%) for Tables 1 and 3 . Each cell contains 500 frozen requests; the mean weights all 14 cells equally.

Dataset / metric	GDN	KDA	GDN2	CFDN
WikiText-2 word PPL	16.27	16.05	16.05	15.74
LAMBADA PPL	12.04	11.66	11.22	11.04
LAMBADA accuracy	46.61	47.76	48.81	48.77
PIQA	73.01	73.01	72.69	73.78
HellaSwag (normalized)	57.47	57.63	57.78	58.53
WinoGrande	57.46	58.80	57.77	58.48

Appendix

Table 10: Complete 1.3B LM and CSR results for Table 2 . Accuracies are percentages. LAMBADA accuracy is reported separately and excluded from the eight-task CSR mean.

Task	Context	GDN	KDA	GDN2	CFDN
S-NIAH-1	1K	100.00	100.00	100.00	100.00
	2K	88.20	100.00	100.00	100.00
	4K	40.00	100.00	100.00	99.60
	8K	18.60	99.60	98.60	94.20
S-NIAH-2	1K	100.00	98.40	99.80	100.00
	2K	100.00	100.00	99.80	100.00

Appendix

Table 11: Complete generated-answer NIAH results at 1.3B (%). Each of the 14 cells contains 500 frozen requests shared across models. These scores use answer generation, rather than the search-time teacher-forced proxy.

Dataset	GDN	KDA	GDN2	CFDN
SWDE	31.96	26.62	26.15	27.55
SQuAD	37.42	36.21	38.97	35.98
FDA	27.16	22.80	25.98	31.06
TriviaQA	60.31	63.68	60.78	61.08
Natural Questions	21.79	23.03	22.27	21.44
DROP	20.41	22.47	20.99	23.33

Appendix

Table 12: Full real-world retrieval results at 1.3B (%). The mean weights the six datasets equally. Each model uses the same checkpoint as the LM, CSR, and NIAH results in Table 2 . Best scores are bolded.

Scientific perspective	Organizing question or distinction
Conditional routing and adaptive computation	When should state change, which path should act, and when is further computation worthwhile?
Multiscale and hierarchical recurrence	How should clocks, temporal boundaries, and communication represent phenomena at different rates?
Stable, switched, and non-normal dynamics	Which distinctions survive input-driven transitions, and when do perturbations amplify?
Multiplicative operators and automata	What computations can input-selected operators express through composition?
Parallel and I/O-aware recurrent design	Which algebraic structure permits efficient execution once state traffic and differentiation are counted?
Streaming sketches and mergeable summaries	Which queries must bounded state support, and how does error compose across updates and merges?

Appendix

Table 13: The 16 scientific perspectives. Each row summarizes one profile’s organizing question; it is not a prescribed candidate mechanism or an exclusive research territory.

Condition	Completed evaluations	Higher utility than parent	Cross-researcher parent links
Main	63	9/41	41/54
No background	67	12/42	41/57
Fresh memory	65	5/32	36/44
Uniform selection	68	15/46	38/57

Appendix

Table 14: Completed evaluations and declared parent relationships. Each row covers 80 iterations. A fraction reports the number satisfying the criterion over the number eligible. The fresh-memory and uniform-selection rows include the common first 16 iterations and their 11 completed evaluations.

Autonomous Scientific Discovery

Autonomous scientific discovery develops AI systems that generate, test, and refine scientific hypotheses using data, computation, and experiments.

See all 128 papers

May 27, 2026

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

Scientific research proceeds through iterative cycles of hypothesis generation, experiment design, execution, and revision. AI agents can automate parts of this process, but existing approaches typically follow a single research trajectory or coordinate through a central planner with fixed objectives. As a result, they struggle to sustain parallel exploration, adapt as experimental evidence changes, or preserve knowledge of failed directions over long-running experiments. We introduce AutoScientists, a decentralized team of AI agents for long-running computational scientific experimentation. Agents interpret a shared experimental state, self-organize into teams around promising hypotheses, critique proposals before using experimental compute, and share successes and failures to reduce redundant exploration. Under matched experimental budgets, AutoScientists improves over prior AI agents across biomedical machine learning, language-model training optimization, and protein fitness prediction. On BioML-Bench, spanning biomedical imaging, protein engineering, single-cell omics, and drug discovery, AutoScientists achieves a mean leaderboard percentile of 74.4% across 24 tasks, improving over the strongest AI agent by +8.33%. On GPT training optimization, AutoScientists reaches a target validation bits-per-byte 1.9x faster than Autoresearch and continues discovering improvements from a starting champion where the single-agent approach finds none (7 vs. 0 accepted improvements). On ProteinGym fitness prediction, AutoScientists discovers a method for ACE2-Spike binding that improves over the current state-of-the-art model by +12.5% in Spearman correlation. Applied without modification across all 217 ProteinGym assays, the same method improves over the prior state of the art by +6.5% (Spearman correlation).

Shanghua Gao, Ada Fang, Marinka Zitnik

Harvard University

Protein Fitness Prediction Multi-Agent Collaboration Multi-Agent Systems Autonomous Scientific Discovery Autonomous Experimentation

Ask PDF

Oct 14, 2026

Scaling Discovery Requires Forgetting

Automated research with LLM agents has shown promising results in various domains. Through iterative refinement and scored verifiers, LLM agents are capable of discovering new ideas. There are different ways to scale automated research: let one agent iterate for longer, run several agents in parallel, or further let the parallel agents communicate and build on one another's candidates. However, we find that all three strategies eventually plateau as the agents accumulate history. Each agent becomes anchored to its own past attempts, making fewer and smaller edits. Beyond this point, additional budget brings little further improvement. In this paper, we show that the key to scaling discovery further is to forget. Periodically discarding the search state, including every agent's session and the candidate history, and restarting from the best candidate found so far, unblocks the search and pushes the same budget further. We implement forgetting in a team of agents with communication and evaluate it on four tasks spanning heuristic programming, kernel optimization and model training. The results show up to 15% improvement compared to parallel agents, and up to 39% compared to a single agent. Our results suggest that, for open-ended discovery, what an agent is allowed to remember matters as much as how much compute it is given.

Shangyin Tan, Ben Zhou, Lakshya A Agrawal +8

UC Berkeley · Stanford · Bespoke Labs

AI Agents for Scientific Discovery LLM Agent Memory Automated Algorithm Discovery Multi-Agent LLMs LLM Context Management

Ask PDF

Jul 24, 2025

Neural Architecture Discovery via Autonomous Evolution

Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for AI-driven AI research that autonomously conducts neural architecture research through a closed-loop research-experiment-analyze-update process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.

Weixian Xu, Yixiu Liu, Yang Nan +4

Shanghai Jiao Tong University · SII · GAIR +1

Autonomous Research Agents AI Agents for Scientific Discovery Neural Architecture Search Large Language Model-Guided NAS Linear Attention

Ask PDF

Abstract

Autonomous Scientific Discovery

Explore similar work

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

Scaling Discovery Requires Forgetting

Neural Architecture Discovery via Autonomous Evolution