When AI Designs AI: Innovation or Imitation?

Yikang Yang; Zhengxin Yang; Luzhou Peng; Minghao Luo; Yanqi Kan; Wanling Gao; Jianfeng Zhan

cs.AIAug 18, 2026

When AI Designs AI: Innovation or Imitation?

Authors: Yikang Yang, Zhengxin Yang, Luzhou Peng, Minghao Luo, Yanqi Kan, Wanling Gao, Jianfeng Zhan

Organizations: State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing 100190, China · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · BenchCouncil (International Open Benchmark Council) ·

Abstract

Recent advances in LLM agents have made them increasingly capable of designing methods for complex AI tasks. This raises two central questions about agent-designed methods relative to human-designed methods: how well they perform, and how different their algorithmic designs are. To study these questions, this paper introduces an analysis that derives task-specific algorithmic design spaces from human-designed methods, maps both human- and agent-designed methods into these spaces, and quantifies their algorithmic differences at the module level. Widely used LLM agents are evaluated on a suite of representative, open-ended AI tasks spanning multiple modalities, and the methods they design are analyzed in terms of both task performance and algorithmic differences from human-designed methods. Experimental results show that current agents can occasionally match or surpass human state-of-the-art (SOTA) performance (10/72 configurations), but such success does not generalize reliably across tasks or agents. Moreover, 96.8% of agent-designed methods fall within human-derived algorithmic design spaces, largely recombining algorithmic choices found in human-designed methods, while nearly half exactly match an existing human algorithmic design. Taken together, these findings suggest that although current agents can occasionally match or surpass human SOTA performance, their algorithmic designs remain within human-derived algorithmic design spaces, reflecting the reuse and recombination of algorithmic choices.

Figures & tables

Figure 1: Overview of the analysis methodology. An LLM agent develops and executes code for an AI task, producing submissions that are evaluated by a task-specific evaluator to measure performance. The code is mapped into an algorithmic design space constructed from human reference methods. Human- and agent-designed methods are represented in the same task-specific space, enabling their algorithmic differences to be quantified at the module level.

Figure 2: Human-in-the-loop construction of a task-specific algorithmic design space. LLM agents summarize human methods into structured method descriptions and use these descriptions to produce an initial version of the task-specific algorithmic design space. Human experts review the proposed modules, values, and dependencies, and LLM agents revise the design space based on their feedback until the revised design space is accepted. Each human method is then mapped to an algorithmic coordinate in the accepted design space.

Task	Domain	Dataset	Metric	SOTA year	References
Image classification	Vision	CUB-200-2011 [ 28 ]	Accuracy $\uparrow$	2023	44
Emotion classification	NLP	GoEmotions [ 6 ]	Macro-F1 $\uparrow$	2026	24
Node classification	Graph	ogbn-arxiv [ 13 ]	Accuracy $\uparrow$	2023	81
Link prediction	Graph	ogbl-ppa [ 13 ]	Hits@100 $\uparrow$	2025	27
Time-series forecasting	Time series	ETTh1 [ 43 ]	MSE $\downarrow$	2026	76
Time-series forecasting	Time series	Weather [ 34 ]	MSE $\downarrow$	2025	75

Table 1: Overview of the task collection.

Agent	Ref.	Performance on Each Task						Overall
Agent	Ref.	CUB200 $\uparrow$	GoEmotions $\uparrow$	ogbn-arxiv $\uparrow$	ogbl-ppa $\uparrow$	ETTh1 $\downarrow$	Weather $\downarrow$	$\overline{\mathrm{Rank}}$
Human SOTA	–	0.9310	0.5518	0.7803	0.7655	0.3748	0.2170	2.67
CC-Opus	w/o	0.9355	0.5553	0.7458	0.6488	0.5047	0.2518	5.17
CC-Opus	w/	0.9241	0.5635	0.7709	0.6288	0.5266	0.2436	5.00
Codex	w/o	0.8234	0.5304	0.7431	0.1824	0.5281	0.2296	10.00
Codex	w/	0.9089	0.5411	0.7413	0.5049	0.5437	0.2399	9.83

Table 2: Best performance achieved within the evaluation budget for each agent, task, and reference condition. The Ref. column indicates whether prepared reference papers and corresponding code are provided (w/) or withheld (w/o). CC-Opus, CC-DeepSeek, and CC-GLM denote Claude Code using Claude Opus 4.8, DeepSeek 4 Pro, and GLM 5.2, respectively. Codex and MLEvolve use GPT 5.5, while Gemini CLI uses Gemini Flash 3.5. The Human SOTA row reports the best collected human result for each task. Upward and downward arrows indicate whether higher or lower values are better.

\overline{\mathrm{Rank}}

is the mean task-wise rank across Human SOTA and the twelve agent configurations. Bold values indicate the best result in each column, while underlined agent results meet or exceed Human SOTA.

Panel of Figure 3 — Figure 3: Distributions of the minimum algorithmic distance from agent-designed methods to collected human methods, group separately by agent and task.

Figure 4: Concentration of agent-designed methods on the most frequently visited algorithmic coordinates within each task. Rank 1 denotes the most frequent coordinate, while Ranks 2–3 and Ranks 4–5 denote the corresponding groups of coordinates. Other includes less frequent coordinates and methods outside the constructed design spaces. Overall reports the mean distribution across the six tasks.

Figure 5: Task performance across algorithmic distance categories. Boxes show the performance distributions for categories containing at least eight solutions, while individual points are shown for smaller categories. Dashed lines indicate Human SOTA, while the regions labeled Human IQR span the 25th to 75th percentiles of the collected human performance results. OOS denotes solutions outside the constructed design spaces. Open triangles indicate results beyond the displayed vertical range, with numbers reporting their counts when greater than one.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 702 papers

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks
Aug 30, 2026Shitanshu Bhushan, Yunxiang Zhang, Lu WangCreativity Assessment LLM Agent Evaluation
Computer Science and Engineering University of Michigan Ann Arbor, MI, USA
Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design
Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4AI for Science Benchmark Design
Westlake University · Zhejiang University · Nanjing University of Science and Technology +1
Agentic Neural Architecture Search
Jul 8, 2026Seokhoon Jeong, Mijung Kim, Taehwan KimLarge Language Model-Guided NAS Large Language Model-Guided Optimization
Artificial Intelligence Graduate School, Ulsan National Institute of Science and Engineering, Ulsan, Republic of Korea · Department of Computer Science and Engineering, Ulsan National Institute of Science and Engineering, Ulsan, Republic of Korea

Aug 30, 2026J/K move · Enter open · S save

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Shitanshu Bhushan, Yunxiang Zhang, Lu Wang

Computer Science and Engineering University of Michigan Ann Arbor, MI, USA

Creativity Assessment LLM Agent Evaluation Computational Creativity

Open paper AskPDF arXiv

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.

Abstract

LLM Agents

Explore similar work

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

Agentic Neural Architecture Search