Industrial process control demands policies that are interpretable and auditable, requirements that black-box neural policies struggle to meet. LLM-driven heuristic synthesis offers an alternative: a language model iteratively proposes and refines human-readable Python controllers using rich behavioral feedback from a simulator. We study this form of programmatic policy search on hot steel rolling, a physics-simulated, multi-pass control problem with competing objectives and hard equipment constraints, and address three questions that arise once the search is run at scale: how to split a fixed budget of LLM iterations between restarting and refining, how to deploy the resulting pool of heuristics, and how to verify it. Treating each search run as a randomized algorithm, we find across 52 independent runs that runs of moderate length match the hindsight-optimal budget allocation, and that Luby-style universal restarts reach this regime without problem-specific tuning. Because each heuristic is cheap to simulate, a small portfolio selected per job outperforms the best single heuristic, also under cross-validation and on out-of-distribution operating conditions on which the best heuristic stalls. Finally, an automated audit pipeline combining static analysis, Z3 proofs, and property-based testing proves key safety properties for all 174 synthesized heuristics; the stall, which neither the search reward nor these properties reveal, shows that the specifications must also cover progress.
Figures & tables
Figure 1
Figure 3: Expected best reward (median) as a function of total iteration budget for uniform strategies ( L=1,5,10,20 ) and the optimal mixed allocation. Labels on the optimal mix curve show the allocation chosen.
Table 3Figure 4
Figure 6: Greedy portfolio construction: ad-hoc (18 heuristics) vs. Luby (156 heuristics), evaluated on the 81-case in-distribution grid. Solid: portfolios selected and scored on all 81 cases; dashed: selected on a random half of the cases and scored on the other half (mean over 200 splits). Dotted lines mark the K at which 90% of the full-pool gain is captured.
Pool
Best single
K=4
Oracle
Median
Ad-hoc (18)
41.8 (37)
58.7 (41)
63.6 (42)
39.1
Luby (156)
51.2 (42)
59.9 (42)
67.3 (42)
36.1
Table 3: Mean episode return over the 42 out-of-distribution cases (completed cases in parentheses). Best single and K=4 portfolio are selected on the in-distribution grid; the oracle keeps the best heuristic of the pool per case; Median is the median heuristic. The hand-coded baseline scores 37.2 and completes 40 cases.
Layer
Checks
Pass
Notes
AST (8 categories)
29
27/29
2 warn
Interval analysis
9
8/9
1 warn
Z3 SMT prover
10
10/10
3 proved ∀ , 7 deferred
Property testing
17
17/17
207 inputs, 0 failures
Total: 65 checks
62/65
0 errors
Table 4: Audit results for the best heuristic (reward 55.5). The pipeline produces 65 checks with 0 errors. Z3 proves safety specifications for all valid inputs; remaining specs verified via randomized property testing.
Figure 8
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Category
Property
Sev.
Meth.
001
Safety
HR ≤ thickness − target
E
Z3
002
Safety
HR/10 ≤ hr_limit
E
Z3
003
Safety
HR ≥0
E
Z3
004
Monoton.
Thicker ⇒ larger HR
W
H
005
Monoton.
Smaller grain ⇒ longer wait
I
H
006
Monoton.
Higher force ⇒ lower velocity
I
H
Appendix
Table 6: Complete domain specification catalog. Severity: E = Error (safety-critical), W = Warning, I = Info. Method: Z3 = formally proved; H = property testing.
Category
Chk
What it catches
Structural integrity
4
Wrong signature, missing returns, global state, class definitions
Table 7: AST-based check categories. Each category targets a specific failure mode of LLM-generated code.
Variable
Min
Max
Unit
current_thickness
5.0
110.0
mm
target_thickness
5.0
15.0
mm
hr_limit
20.0
50.0
mm
stock_temperature
800
1523
K
target_temperature
1073
1273
K
current_grain_size
5
500
μ m
Appendix
Table 8: Input value ranges for interval analysis and Z3 constraints.
Figure 9: Population audit of the 174 synthesized heuristics. Left: share of heuristics failing each of the most frequent checks, all at warning level. Right: warning-level findings against search reward for the ad-hoc (squares) and Luby (dots) pools, with vertical jitter. Warnings increase slightly with reward (Spearman ρ=0.16 ), so they are not a quality signal.
Figure 10: Expected marginal gain in best reward from continuing a run for one more iteration versus starting a fresh run, as a function of the current iteration. Restarting pays more during the first few iterations; the curves meet around iteration 4.
T=20 : optimal allocation minus
T=40
Luby
Runs
n
one run of 20
four runs of 5
L=1
optimal
percentile
All
52
+0.1[−2.5,+2.1]
+1.6[−1.1,+5.2]
36.4
45.4
39
Gemini 2.5 Pro
49
+0.1[−2.5,+2.3]
+1.4[−1.1,+5.1]
36.4
45.4
38
Final settings
27
+1.2[−1.4,+3.6]
+0.8[−2.3,+5.6]
37.0
46.8
28
Reached planned length
17
+2.1[−0.8,+4.2]
+0.9[−3.2,+6.6]
36.2
47.2
24
Appendix
Table 9: Restart analysis on subsets of the ad-hoc runs: paired bootstrap differences at T=20 between the optimal allocation (10+10 in every subset) and two uniform allocations (median and 95% CI), expected best reward at T=40 of one-iteration sampling and of the optimal allocation, and the percentile of the observed Luby campaign among 20,000 simulated ones.
Figure 15
Figure 13: Restart analysis at fixed reward thresholds τ . Left: empirical probability that a run’s best-so-far reward after L iterations is at least τ . Right: expected iterations to reach τ when restarting every L iterations, E[min(X,L)]/P(X≤L) , in the same colors; dots mark the optimal cutoff L∗ , which depends on the target.
Parameter
Value
LLM model
Gemini 2.5 Pro (49), 2.5 Flash (3)
Temperature
0.7 (18), 1.0 (10), 1.5 (2),
not recorded (22)
Outer iterations per run
1–30
Independent runs
52
Ideas per iteration
3 (43), 2 (7), 4 (2)
Appendix
Table 10: Search configuration of the 52 ad-hoc runs. For settings that changed between groups of runs, the number of runs is given in parentheses (Table 11 ).
Runs started
n
Model
T
Ideas
Stag.
Memory
19 21:11–23:14
4
Pro
–
2
–
compaction 20/10
19 23:23–20 00:03
3
Pro
–
3
–
compaction 20/10
20 00:58–03:07
3
Pro
–
3
8
compaction 20/10
20 01:19–02:21
3
Pro
–
2
8
compaction 20/10
20 02:24–02:58
2
Pro
–
4
8
compaction 20/10
20 03:52–09:27
4
Pro
–
3
6
compaction 25/10
Appendix
Table 11: Groups of ad-hoc runs with identical settings, in order of their first run (all in February 2026). T: sampling temperature; Stag.: stagnation threshold; compaction a/b : compact at a messages, keep b .
Figure 14: Best-so-far reward by memory strategy. Synthesis resets the conversation each iteration and provides the LLM with a structured ∼ 800-word knowledge summary synthesised by a narrator; Compaction preserves the full conversation and summarises old messages when the context grows too large. Solid lines show medians; shaded bands show IQR. Labels give the number of runs contributing at each iteration, which decreases as shorter runs drop out. The synthesis strategy shows higher median reward in early iterations, though sample sizes are modest (16 vs. 36 runs) and the two groups were run sequentially, so this comparison should be interpreted cautiously.
Iter
LLM’s stated strategy
Reward
0
“Adaptive Two-Phase: bulk reduction then precision finishing”
23.5
1
“Fix catastrophic temperature failure: every scenario loses ∼ 2.0 pts”
23.3
3
“Proactive heat conservation via interpass time modulation”
33.8
9
“Temperature excels but core reduction strategy has systemic flaws”
33.1
10
“Radical departure: the conflict between thermal and metallurgical efficiency requires a fundamentally new approach”
16.6
14
“Fuse strain-based HR with re-tuned thermal response—hybrid synthesis”
38.0
Appendix
Table 12: Strategy evolution over a 20-iteration search, in the LLM’s own words. Reward is the mean across 8 scenarios.
Family (cases)
Baseline
Best single
Portfolio K=4
Pool oracle
Thicker slabs, 130–150 mm (6)
38.6
56.9 / 60.5
62.4 / 68.1
63.9 / 68.2
Thin targets, 2–4 mm (6)
33.1
− 17.2 / 43.0
49.1 / 50.4
49.1 / 62.8
Thick targets, 20–40 mm (6)
20.8
27.9 / 28.3
38.7 / 31.9
46.0 / 47.0
Tight mills, HR limit 10–15 mm (6)
32.5
84.5 / 79.3
91.4 / 85.6
91.9 / 91.8
Loose mills, HR limit 55–60 mm (4)
13.2
34.7 / 8.7
34.8 / 39.1
35.1 / 40.3
Off-grid temperature pairs (4)
40.9
62.8 / 53.4
64.5 / 63.1
66.1 / 64.1
Appendix
Table 13: Mean episode return per out-of-distribution family (ad-hoc pool / Luby pool). The best single heuristic and the K=4 portfolio are selected on the in-distribution grid; the pool oracle keeps the best heuristic of the whole pool per case.
Figure 15: Mean episode return under multiplicative Gaussian observation noise for the best single heuristic and the K=4 portfolio of each pool, both selected on the in-distribution grid, and for the hand-coded baseline (three scenarios × three seeds per noise level; the noise-free values are for the same three scenarios).
Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.
Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1
Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition
We present a structured large-language-model-driven workflow for automated multi-variable control design from dynamic process models. The workflow decomposes the design task into constrained code-generation steps: plant-interface construction, normalization, manipulated-variable controlled-variable (MV-CV) pairing, controller specification, closed loop simulation, scenario generation, performance evaluation and Bayesian-optimization (BO) based tuning. Generated artifacts are executed and validated before downstream tasks proceed, and failed artifacts are repaired using validation feedback. The proposed approach is demonstrated on a nonlinear gas-preheater benchmark with coupled pressure and temperature dynamics. The generated workflow produces a physically consistent decentralized PI (proportional-integral) feedback-feedforward control structure and an executable tuning environment. Bayesian optimization reduces the closed loop performance objective, which aggregates set-point tracking and disturbance-rejection errors for the controlled variables, by approximately 26.5% relative to the initial controller generated by the workflow, mainly through improved pressure-loop transient performance. This figure quantifies the automated tuning stage rather than a comparison against a manually designed controller. The results demonstrate the feasibility of using structured large-language-model-based code generation to construct executable control-design workflows, while also highlighting the need for broader validation on larger plantwide-control benchmarks.
Ari Luna Rueda, Eike Cramer, Klaus Hellgardt +1
Department of Chemical Engineering Imperial College London London, UK · Department of Chemical Engineering University College London London, UK
The execution harness is a critical component of LLM agents, shaping how the agent interacts with its environment and organizes multi-step execution. Prior work has shown that improving tools, workflows, and other harness components can substantially improve agent performance. This suggests that the harness can be treated as a learnable control layer that adapts execution decisions from experience, rather than as a predetermined execution configuration. In this work, we introduce HarnessMDP, a learning-based framework that makes harness control itself learnable. HarnessMDP formulates agent execution as a finite-horizon Markov decision process, where the harness learns to select structural execution actions from the evolving agent state. The harness policy is learned offline from collected trajectories using task performance and deployed online to adaptively control new executions. Extensive experiments on coding tasks, τ-bench, AgentBench, and other agent evaluation settings show that HarnessMDP consistently improves execution behavior and final task performance over baseline harnesses. These results demonstrate the effectiveness of HarnessMDP and establish learnable harness control as a practical mechanism for improving LLM agents through better execution. Our code is available at https://github.com/Hik289/Agentic-RL-harness.git.