cs.CLOct 4, 2026

Building LLM Agent Systems the Deep Learning Way: From Modular Design to Architecture Search

Authors: Tao Feng, Pengrui Han, Zhongjie Dai, Jiaxuan You

Organizations: University of Illinois Urbana-Champaign

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) have revolutionized AI research and enabled exciting agent systems. To build a complex LLM agent system, most existing research relies on insights from other domains or heuristics to manually build the agent system. However, this approach often requires heavy hand-engineering and fails to fully optimize for the downstream task of interest. Inspired by the tremendous success of deep learning, we propose to construct LLM agent systems in a modular manner, similar to building a deep neural network. Our key insight is to make analogies between LLM building blocks, such as retrievals, memories, and prompting strategies, and the successful deep learning modules, such as MLPs, attention, and recurrent modules. We further design forward inference and feedback mechanisms for LLMs, where prompts in LLMs are considered as the weights in deep models, and the prompt optimization from feedback is analogous to the back-propagation algorithm. We additionally leverage a search algorithm to search for the best configuration of LLM agent systems, similar to the neural architecture search (NAS) in deep learning research. Comprehensive experimental results demonstrate that the proposed deep learning recipe for LLM agent systems is highly effective, in particular: (1) Organizing LLM modules into deep-learning-style architectures yields noticeable performance gain; (2) Automatic prompt optimization, equivalent to backpropagation, is efficient in incorporating feedback from the task of interest and achieves at least 5% performance improvement; (3) NAS equivalent algorithm works well for further optimizing the LLM agent system architecture with 11% performance gain compared with randomly designed architectures. Overall, our research demonstrates the exciting opportunity of transferring the success of deep learning to building LLM agent systems.

Figures & tables

Figure 1: Basic Building Blocks of LLM agent systems . (a) A vanilla LLM can be translated into a single layer MLP, which receives a query qq and outputs a response rr ; (b) Memory processor can be regarded as RNN since they both utilize historical data for future planning; (c) Retrieval is analogous to the attention network of DL because they both extract the importance information of the corpus context; (d) Prompting strategies, such as Tree/Graph of thoughts, are translated to GNNs since they decompose query qq into interconnected thoughts, which are connected and related to each other.
Figure 2: Forward and Back Propagation Algorithms for Large Language Model . In this process, meta prompt is regarded as trainable parameters in DL. For the forward inference, the input query goes through the composition of LLM building blocks to obtain the response, which is guided through the meta prompt. For the feedback propagation, the feedback from the self-reflection updates the parameters (meta prompt) of LLM.
Figure 3: Build a Cross-attention Transformer with LLM Building Blocks . The multi-query retrieval is equivalent to the cross-attention network, and the LLM is translated to MLP in DL. Additionally, the process of appending the raw query to the generated multiple queries is equivalent to the residual.
Figure 4: An example of the meta prompt used for RNN-MLP on the GSM8K dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Table 5
Figure 5: Optimization on transformer to maximize F1.
Figure 7
 074378
F1 Score0.5410.6850.6940.712
Table 3: As the number of LLM architecture searches increases, the LLM agent system is getting better.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

LLM agent systemsDeep Learning Architecture
Vanilla LLMSingle layer MLP
Memory processorRecurrent neural network
RetrievalAttention Network
Prompting strategies (ToT & GoT)Graph neural network
PromptsTraining parameters
Max tokens of each LLMSize of MLP
Appendix
Table 4: Deep Learning Recipe for LLM Agent Systems.
Figure 8: An example of the meta prompt used for RNN-MLP on the TSP dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 9: An example of the meta prompt used for Tree-MLP/GNN-MLP on the TSP dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 10: An example of the meta prompt used for RNN-MLP on GSM8K dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 11: An example of the meta prompt used for Tree-MLP/GNN-MLP on GSM8K dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 12: An example of the meta prompt used for RNN-Att-MLP on StrategeyQA dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 13: An example of the meta prompt used for Tree-Att-MLP/GNN-Att-MLP on StrategeyQA dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 14: An example of the meta prompt used for RNN-Att-MLP on Related-multi dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
Figure 15: An example of the meta prompt used for Tree-Att-MLP/GNN-Att-MLP on Related-multi dataset. The blue text contains input questions and previous responses/evaluations. The purple text describes the output format instructions.
 Range
Output Size m1m_{1}[64, 128, 192, 256, 320]
Output Size m2m_{2}[64, 128, 192, 256, 320]
Output Size m3m_{3}[64, 128, 192, 256, 320]
Head Num hh[1, 2, 3, 4, 5]
Retrieve Num nn[1, 2, 3, 4, 5]
Appendix
Table 5: Optimization range of each hyperparameter.
Figure 16: The optimization process on transformer . The x-axis represents the number of trials. For each trial, we will get a set of pr,F1pr,F1 ; the y-axis represents the F1F1 value.
DatasetTask TypeCases
Without External Knowledge
TSPOptimization1500
GSM8KMath Problem1,319
With External Knowledge
StrategyQACommonsense Reasoning2,290
Related-multiSummary1000
Appendix
Table 6: Overview of Datasets.
More in this topic

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

Explore similar work

CardsList

  1. TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

    Apr 15, 2026Zerun Ma, Guoqiang Wang, Xinchen Xie +7LLM Fine-TuningLLM Agent Training

    Shanghai AI Laboratory · Fudan University

  2. Evolving Excellence: Automated Optimization of LLM-based Agents

    Dec 9, 2025Paul Brookes, Vardan Voskanyan, Rafail Giavrimis +18Large Language Model-Guided OptimizationLLM Agent Workflow Optimization

    TurinTech AI, London, UK · University of Surrey, Guildford, United Kingdom · University of Leeds, Leeds, UK +4

  3. Self-Optimizing Multi-Agent Systems for Deep Research

    Apr 3, 2026Arthur Câmara, Vincent Slot, Jakub ZavrelDeep Research AgentsMulti-Agent LLM Systems

    Zeta Alpha, Amsterdam, The Netherlands