A Multi-Agent LLM System for Automated Tuning of Specialized Neural Network Models
Authors: Ivan Moloshnikov, Dmitry Kunitsyn, Alexander Sboev, Maksim Skorokhodov, Roman Rybka, Yuanchao Liu
Organizations: National Research Centre “Kurchatov Institute”, 1 Akademika Kurchatova Square, Moscow, 123182, Moscow, Russia · National Research Nuclear University MEPhI, 31 Kashirskoye Shosse, Moscow, 115409, Moscow, Russia · Moscow Institute of Physics and Technology, 9 Institutskiy Lane, Dolgoprudny, 141701, Moscow, Russia · Harbin Institute of Technology, No. 92 West Dazhi Street, Nangang District, Harbin, 150001, Heilongjiang, China
Many machine learning approaches are two-level: an expert selects a class of methods, within this class the choice of key components is critical and usually manual. This paper addresses the automatic tuning of these components and proposes a framework transferable across models of different nature. The multi-agent large language model (LLM) framework has three roles---analyst, coder, and recorder~---with an experiment log and a sandboxed execution environment. Unlike systems that optimize the entire ML script or search architectures in a discrete space, the framework operates over an open but semantically narrow space: the agent generates code only for the specified components, leaving the overall network structure unchanged. The approach is validated in two directions: tuning a classification spiking neural network with correlation learning, and solving high-order differential equations with Physics-Informed Neural Networks (PINN). Across five tasks---spiking classification on MNIST and Digits, and boundary-value problems for the generalized nonlinear Schrödinger equation (GNLSE) and Kuramoto--Sivashinsky (KS) equations---the agent outperforms the baselines: classification accuracy is 13 percentage points (MNIST) and 6 percentage points (Digits) higher; the error against the analytical single-soliton solution is reduced approximately 2491-fold; the invariant-preservation error drops approximately 325-fold in the selected GNLSE run with a Gaussian perturbation and by more than 100-fold on average over repeated initializations; and the mass drift drops 14-fold for the KS problem with a Gaussian perturbation. The framework thus tunes key components on tasks of different nature without changing its internal structure.
Figures & tables
Figure 1: Architecture of the multi-agent framework. The Human Researcher configures the task, agent-modifiable block(s), target metric, and software contract once. At each iteration, the LLM Analyst reads the experiment log and produces a technical specification, which the LLM Coder converts into contract-constrained code. The generated module is first functionally tested and then evaluated in a Docker sandbox; runtime errors are returned to the Coder, while a successful evaluation is passed to the LLM Recorder. The Recorder appends a structured JSON record to the experiment log, which serves as the external memory for the next iteration. The detailed-prompt mode additionally provides the Analyst with a method-class taxonomy and a dynamically updated coverage map. Only task-specific prompts, the software contract, the editable model block, and the target metric change between tasks; the agent roles and loop structure remain unchanged.
Figure 2: SNN pipeline. The input encoding, the SNN trained with STDP, and the output activity vector r (left and middle) are fixed. Only the decoder module Decoder(r) (right) is generated and optimized by the framework.
Figure 3: Reference solutions of the two GNLSE boundary-value problems, shown as a surface (top) and a space–time heat map (bottom). Left: the analytical single soliton. Right: the segmented PINN solution of the soliton with a Gaussian perturbation from Zavertyaev et al. (2024) (Figure 5). The Kuramoto–Sivashinsky problem is not shown because it is solved without a reference solution.
Figure 4: PINN architecture. The coordinates (x,t) pass through the transformation layer FI , the fully connected network, and the transformation layer FO , producing predictions u^,v^ . Automatic differentiation computes the derivatives for the loss function. The loss function ( L ) consists of the residual in the differential equation ( Lpde ) and the errors in the initial ( Lic ) and boundary conditions ( Lbc ). The optimizer updates the network weights.
Task (metric)
Baseline (single run)
Best agent result (best in series)
Improvement over baseline (pp / factor)
SNN, MNIST (Acc)
80 %
93 %
+13 pp
SNN, scikit-learn digits (Acc)
91 %
97 %
+6 pp
PINN, GNLSE, single soliton ( Relh )
5.51⋅10−2
2.21⋅10−5
2491×
PINN, GNLSE, soliton + Gaussian ( Reli )
2.15⋅10−1
6.61⋅10−4
325×
PINN, KS, traveling wave + Gaussian ( Δm(u) )
9.31⋅10−2
6.56⋅10−3
14×
Table 1: Baseline and best agent results across five tasks.
Model
Prompt
Mean maximum accuracy ± std, %
Best accuracy, %
Runs with accuracy ≥90%
Claude Opus 4.6
simple
91.13±0.44
91.65
10 out of 10
Claude Opus 4.6
detailed
92.15±0.62
92.83
10 out of 10
Claude Opus 4.6
detailed without coverage map
91.02±0.25
91.45
10 out of 10
Perplexity Sonar
simple
79.88±0.00
79.88
0 out of 10
Perplexity Sonar
detailed
91.76±0.56
92.70
10 out of 10
Perplexity Sonar
detailed without coverage map
82.34±0.70
83.18
0 out of 10
Table 2: Comparison of analyst models and prompt variants on MNIST. For each configuration, the table reports the number of complete agent runs, the mean and standard deviation of the maximum accuracy achieved within an individual run, the highest accuracy across all runs, and the number of runs in which the 90 % threshold was reached. Ten matched runs were performed for every configuration.
Method
Mode
Accmin , %
Accmax , %
kmin
kmax
TPOT
classic
89.60
90.15
201
243
TPOT
sequential
89.38
89.93
67
210
Random Choice
LLM sampler
89.48
89.93
-
-
Qwen3-30B- A3B-Instruct-2507
detailed
82.28
89.16
15
209
Perplexity Sonar
detailed
90.93
92.70
8
147
Claude Opus 4.6
detailed
91.22
92.83
2
43
Table 3: Comparison of the agent system, TPOT evolutionary search ( classic and sequential modes), and the Random Choice baseline for the SNN decoder on MNIST with a budget of 250 evaluated models.
Metric
Claude det.
Sonar det.
Qwen det.
Random Choice
Performance
Accmin , %
91.22
90.93
82.28
89.48
Accmax , %
92.83
92.70
89.16
89.93
kmin
2
8
15
—
kmax
43
147
209
—
Resources per run, tokens in millions (M)
Table 4: Performance, resources, and cost of the agent framework and the Random Choice baseline on MNIST (detailed prompt, budget of 250 evaluated decoders per run). The performance block is taken from Table 3 .
Model
Baseline Relh
Best Relh
Change relative to baseline
sonar-pro
4.69⋅10−2
1.97⋅10−4
238× better
anthropic-claude-opus-4.6
5.51⋅10−2
2.21⋅10−5
2491× better
qwen-qwen3-30b-a3b-instruct-2507
4.85⋅10−2
8.86⋅10−3
5.5× better
Table 5: PINN results for the single soliton ( Relh metric; problem with an analytical solution). Each value is the mean over three internal seeds. Because baseline variation is non-negligible, the change for each analyst is calculated relative to the identity baseline from the same agent run.
Model
Result
Reli
Resbvp
Change in Reli relative to baseline
Human-designed PINN from prior work Zavertyaev et al. (2024)
Reported result
9.87⋅10−4
9.11⋅10−5
130× better
Identity baseline (best across runs)
1.28⋅10−1
1.04⋅10−3
—
sonar-pro
Best Reli
6.35⋅10−3
2.37⋅10−2
20× better
anthropic-claude-opus-4.6
Best Reli
6.61⋅10−4
3.95⋅10−5
194× better
anthropic-claude-opus-4.6
Best Resbvp
1.40⋅10−3
2.08⋅10−5
91× better
qwen-qwen3-30b-a3b-instruct-2507
Best Reli
9.43⋅10−2
1.38⋅10−3
1.36× better
Table 6: Single-run PINN results for the soliton with a Gaussian perturbation (problem without an analytical solution). The shared identity-baseline row contains the best value of each metric across the analyst runs; these values may therefore come from different runs. Improvements in Reli are computed relative to this shared baseline.
Configuration
Resbvp
Δm(u)
Hypothesis
Baseline ( FI,FO identity)
1.41⋅10−3
9.31⋅10−2
—
Best Resbvp ( 9× better)
1.51⋅10−4
3.47⋅10−2
MultiplicativeIC_DualAdditive_DualZoom
Best Δm(u) ( 14× better)
1.70⋅10−4
6.56⋅10−3
BoundedMultAdd_SechZoom_DualPhase
Table 7: Best single-run PINN results for the Kuramoto–Sivashinsky traveling wave with a Gaussian perturbation (Claude Opus 4.6, one seed per hypothesis). Resbvp is the metric the analyst optimized; Δm(u) is an independent physical diagnostic. The two columns report the best value of each metric over the run; they are attained by different hypotheses, which are named in the last column.
Figure 5: Predicted solution of the Kuramoto–Sivashinsky problem at t=0,6,10 for the baseline (top) and for the hypothesis with the lowest mass drift, BoundedMultAdd_SechZoom_DualPhase (bottom). No numerical reference is available for this problem, so only the PINN prediction is plotted; the initial condition is shown in the left panels for comparison. The horizontal axis is the coordinate x , the vertical axis is u . The baseline pulse loses roughly half of its amplitude by t=10 and develops spurious low-amplitude oscillations across the domain, whereas the agent hypothesis keeps a localized traveling pulse whose amplitude stays close to the initial value of 1/2 .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Single soliton
Soliton + Gaussian
Number of hidden layers
2
4
Neurons per layer
32
64
Activation (hidden layers)
sin
sin
Activation (output layer)
linear
linear
Optimizer
Adam
Adam
Learning rate
10−3
10−3
Appendix
Table 8: Hyperparameters of the baseline PINN for the two GNLSE configurations. The Kuramoto–Sivashinsky configuration is given separately in Table 9 .
Parameter
Value
Number of hidden layers
4
Neurons per layer
32
Network output
u (one real scalar)
Activation (hidden layers)
sin
Activation (output layer)
linear
Optimizer
Adam, clipnorm=1
Appendix
Table 9: Hyperparameters of the baseline PINN for the Kuramoto–Sivashinsky traveling-wave problem with a Gaussian perturbation.
Parameter
Value
Vrest (resting potential)
−70 mV
Vreset (post-spike potential)
−70 mV
Vth (threshold potential)
−56 mV
τm (membrane time constant)
103 ms
τs (teacher-spike shift)
0.1 ms
Nref (spacing between spikes in S0 )
3
Appendix
Table 10: Hyperparameters of the baseline SNN and its training, common to all configurations.
Configuration
Mean ± std
Median
Min–max
CV
Reduction factor in Reli relative to baseline mean
Baseline ( n=10 )
(1.65±0.18)⋅10−1
1.66⋅10−1
1.29⋅10−1 – 1.94⋅10−1
11.1%
1×
Residual-selected configuration ( n=10 )
(1.42±0.59)⋅10−3
1.50⋅10−3
5.40⋅10−4 – 2.41⋅10−3
41.8%
116.1×
Invariant-selected configuration ( n=10 )
(1.60±0.81)⋅10−3
1.59⋅10−3
7.57⋅10−4 – 3.31⋅10−3
50.4%
103.3×
Appendix
Table 11: Repeated runs for the GNLSE soliton problem with a Gaussian perturbation. The table reports the mean and sample standard deviation, median, range, and coefficient of variation (CV) of Reli .
Method class
Methods
Best acc, %
Status
Ensembles
154
92.70
explored
LDA / QDA
32
87.90
explored
Softmax / calibration
13
82.32
explored
k -nearest neighbors
11
88.56
explored
Naive Bayes
11
83.73
explored
Mahalanobis
9
86.05
limited data
Appendix
Table 12: Coverage map of decoder method classes for one 250-iteration run (Perplexity Sonar, detailed prompt, MNIST).
Pair no.
Simple, %
Detailed, %
Δ , pp
1
79.88
82.28
2.40
2
79.88
83.35
3.47
3
79.88
88.57
8.69
4
79.88
86.48
6.60
5
79.88
89.16
9.28
6
79.88
84.10
4.22
Appendix
Table 13: Maximum accuracy in ten matched pairs of complete agent-system runs with the simple and detailed prompts for Qwen3-30B-A3B-Instruct-2507 on MNIST. Within each pair, the same trained SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy for the detailed prompt minus the accuracy for the simple prompt.
Pair no.
With map, %
Without map, %
Δ , pp
1
82.28
79.88
2.40
2
83.35
80.42
2.93
3
88.57
79.88
8.69
4
86.48
81.15
5.33
5
89.16
79.88
9.28
6
84.10
80.73
3.37
Appendix
Table 14: Comparison of maximum accuracy in ten matched Qwen3-30B-A3B-Instruct-2507 runs on MNIST for the detailed prompt with and without the coverage map. Within each pair, the same SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy with the coverage map minus the accuracy without the coverage map.
Pair no.
Simple, %
Detailed, %
Δ , pp
1
79.88
92.70
12.82
2
79.88
92.57
12.69
3
79.88
91.95
12.07
4
79.88
91.82
11.94
5
79.88
91.77
11.89
6
79.88
91.74
11.86
Appendix
Table 15: Maximum accuracy in ten matched pairs of complete agent-system runs with the simple and detailed prompts for Perplexity Sonar on MNIST. Within each pair, the same trained SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy for the detailed prompt minus the accuracy for the simple prompt.
Pair no.
With map, %
Without map, %
Δ , pp
1
92.70
83.18
9.52
2
92.57
82.92
9.65
3
91.95
82.77
9.18
4
91.82
82.66
9.16
5
91.77
82.65
9.12
6
91.74
82.47
9.27
Appendix
Table 16: Comparison of maximum accuracy in ten matched Perplexity Sonar runs on MNIST for the detailed prompt with and without the coverage map. Within each pair, the same SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy with the coverage map minus the accuracy without the coverage map.
Pair no.
Simple, %
Detailed, %
Δ , pp
1
90.20
92.83
2.63
2
91.15
91.88
0.73
3
91.65
91.58
-0.07
4
91.60
92.49
0.89
5
91.33
91.22
-0.11
6
90.71
92.35
1.64
Appendix
Table 17: Maximum accuracy in ten matched pairs of complete agent-system runs with the simple and detailed prompts for Claude Opus 4.6 on MNIST. Within each pair, the same trained SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy for the detailed prompt minus the accuracy for the simple prompt.
Pair no.
With map, %
Without map, %
Δ , pp
1
91.22
90.66
0.56
2
91.24
90.73
0.51
3
91.58
90.89
0.69
4
91.88
90.91
0.97
5
92.35
90.95
1.40
6
92.49
91.01
1.48
Appendix
Table 18: Comparison of maximum accuracy in ten matched Claude Opus 4.6 runs on MNIST for the detailed prompt with and without the coverage map. Within each pair, the same SNN instance and identical output spike data were used. The difference Δ is calculated as the accuracy with the coverage map minus the accuracy without the coverage map.
Model
Prompt
n
Mean maximum accuracy ± std, %
Best accuracy, %
Runs with accuracy ≥93%
Claude Opus 4.6
simple
10
95.07±0.52
95.56
10 out of 10
Claude Opus 4.6
detailed
10
96.12±0.66
96.67
10 out of 10
Perplexity Sonar
simple
10
93.44±0.57
93.89
9 out of 10
Perplexity Sonar
detailed
10
93.31±0.93
94.17
8 out of 10
Qwen3-30B-A3B-Instruct-2507
simple
10
91.63±0.74
92.78
0 out of 10
Qwen3-30B-A3B-Instruct-2507
detailed
10
93.66±0.79
94.17
7 out of 10
Appendix
Table 19: Comparison of analyst models and prompt variants on the digits 8×8 dataset. For each configuration, the table reports the number of complete agent runs, the mean and standard deviation of the maximum accuracy achieved within an individual run, the highest accuracy across all runs, and the number of runs in which the 93 % threshold was reached. Ten matched runs of 250 iterations were performed for every configuration; the baseline (argmax decoder) gives an accuracy of 90.83 %.
Figure 6: Search dynamics for the decoding module on the MNIST dataset. The blue line shows the accuracy of the solution obtained at each iteration, the red dashed line shows the best result achieved, and the gray horizontal line shows the baseline accuracy of 79.88 %. The columns correspond to analyst models, and the upper and lower rows correspond to the simple and detailed prompts, respectively.
Figure 7: Search dynamics for the decoding module on the digits dataset. The blue line shows the accuracy of the solution obtained at each iteration, the red dashed line shows the best result achieved, and the gray horizontal line shows the baseline accuracy of 90.83 %. The columns correspond to analyst models, and the upper and lower rows correspond to the simple and detailed prompts, respectively.
Method class
Catalog operator
Implementation
Prototypes and centroids
NearestCentroid
Classification by distance to the class centroid
k -nearest neighbors
KNeighborsClassifier
The number of neighbors, weighting rule, and metric are selected from the grid
Correlation, cosine, and local-density rules
KNeighborsClassifier
Implemented using different local-proximity metrics
While Large Language Models (LLMs) have empowered AI research agents to perform isolated scientific tasks, automating complex, real-world workflows, such as LLM training, remains a significant challenge. In this paper, we introduce TREX, a multi-agent system that automates the entire LLM training life-cycle. By orchestrating collaboration between two core modules-the Researcher and the Executor-the system seamlessly performs requirement analysis, open-domain literature and data research, formulation of training strategies, preparation of data recipes, and model training and evaluation. The multi-round experimental process is modeled as a search tree, enabling the system to efficiently plan exploration paths, reuse historical results, and distill high-level insights from iterative trials. To evaluate the capability of automated LLM training, we construct FT-Bench, a benchmark comprising 10 tasks derived from real-world scenarios, ranging from optimizing fundamental model capabilities to enhancing performance on domain-specific tasks. Experimental results demonstrate that the TREX agent consistently optimizes model performance on target tasks.
Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies. We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications. We evaluate ARTEMIS on four representative agent systems: the ALE Agent for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{13.6% improvement} in acceptance rate; the Mini-SWE Agent for code optimization on SWE-Perf, with a statistically significant 10.1% performance gain; and the CrewAI Agent for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{36.9% reduction} in the number of tokens required for evaluation. We also evaluate the MathTales-Teacher Agent powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a 22% accuracy improvement and demonstrating that ARTEMIS can optimize agents based on both commercial and local models.
Paul Brookes, Vardan Voskanyan, Rafail Giavrimis +18
TurinTech AI, London, UK · University of Surrey, Guildford, United Kingdom · University of Leeds, Leeds, UK +4
Producing task-specific large language models requires discovering effective training strategies through experimentation. Automated fine-tuning systems have made this experimentation feasible with far less manual effort. However, these systems are stateless: each search discards its discovered strategies, dataset insights, and hyperparameter findings once it ends. Every new task must then repeat this costly search from a cold start. To address this, we propose Strategy Accumulation and Guided Execution (SAGE), a two-stage framework that makes automated fine-tuning search cumulative. In the first stage, a multi-agent pipeline performs Monte Carlo Tree Search-based exploration. A parallel Distillation Agent extracts task-specific exploration records and confidence-scored cross-task insights, which together constitute a structured experience repository. In the second stage, SAGE retrieves relevant experience from this repository and selects what applies to guide training on the new task. We evaluate SAGE on nine unseen tasks spanning both single- and cross-category settings. In single-round execution, SAGE's accumulated experience raises the average relative improvement over baseline from 3.2% to 15.6%, a 12.4-percentage-point gain over the same pipeline without it. These results show that persistent strategy experience provides effective guidance for automated fine-tuning on unseen tasks.
Haoran Zhao, Wei Du, Dingwen Yang +7
Fudan University · Ant Group · Shanghai Jiaotong University