From Language Models to Control Policies: Search, Selection, and Verification

Nima H. Siboni; Seyedreza Kiamousavi; Vahid Rostami; Emad Scharifi

cs.AIMar 20, 2026

From Language Models to Control Policies: Search, Selection, and Verification

Authors: Nima H. Siboni, Seyedreza Kiamousavi, Vahid Rostami, Emad Scharifi

Organizations: Juna.ai, Kastanienallee 32, 10435 Berlin · Institute of Metal Forming, RWTH Aachen, Intzestr. 10, 52072 Aachen

Abstract

Industrial process control demands policies that are interpretable and auditable, requirements that black-box neural policies struggle to meet. LLM-driven heuristic synthesis offers an alternative: a language model iteratively proposes and refines human-readable Python controllers using rich behavioral feedback from a simulator. We study this form of programmatic policy search on hot steel rolling, a physics-simulated, multi-pass control problem with competing objectives and hard equipment constraints, and address three questions that arise once the search is run at scale: how to split a fixed budget of LLM iterations between restarting and refining, how to deploy the resulting pool of heuristics, and how to verify it. Treating each search run as a randomized algorithm, we find across 52 independent runs that runs of moderate length match the hindsight-optimal budget allocation, and that Luby-style universal restarts reach this regime without problem-specific tuning. Because each heuristic is cheap to simulate, a small portfolio selected per job outperforms the best single heuristic, also under cross-validation and on out-of-distribution operating conditions on which the best heuristic stalls. Finally, an automated audit pipeline combining static analysis, Z3 proofs, and property-based testing proves key safety properties for all 174 synthesized heuristics; the stall, which neither the search reward nor these properties reveal, shows that the specifications must also cover progress.

Figures & tables

Figure 3: Expected best reward (median) as a function of total iteration budget for uniform strategies (

L{=}1,5,10,20

) and the optimal mixed allocation. Labels on the optimal mix curve show the allocation chosen.

Table 3

Figure 6: Greedy portfolio construction: ad-hoc (18 heuristics) vs. Luby (156 heuristics), evaluated on the 81-case in-distribution grid. Solid: portfolios selected and scored on all 81 cases; dashed: selected on a random half of the cases and scored on the other half (mean over 200 splits). Dotted lines mark the

K

at which 90% of the full-pool gain is captured.

Pool	Best single	$K{=}4$	Oracle	Median
Ad-hoc (18)	41.8 (37)	58.7 (41)	63.6 (42)	39.1
Luby (156)	51.2 (42)	59.9 (42)	67.3 (42)	36.1

Table 3: Mean episode return over the 42 out-of-distribution cases (completed cases in parentheses). Best single and

K{=}4

portfolio are selected on the in-distribution grid; the oracle keeps the best heuristic of the pool per case; Median is the median heuristic. The hand-coded baseline scores 37.2 and completes 40 cases.

Layer	Checks	Pass	Notes
AST (8 categories)	29	27/29	2 warn
Interval analysis	9	8/9	1 warn
Z3 SMT prover	10	10/10	3 proved $\forall$ , 7 deferred
Property testing	17	17/17	207 inputs, 0 failures
Total: 65 checks		62/65	0 errors

Table 4: Audit results for the best heuristic (reward 55.5). The pipeline produces 65 checks with 0 errors. Z3 proves safety specifications for all valid inputs; remaining specs verified via randomized property testing.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	Property	Sev.	Meth.
001	Safety	HR $\leq$ thickness $-$ target	E	Z3
002	Safety	HR/10 $\leq$ hr_limit	E	Z3
003	Safety	HR $\geq 0$	E	Z3
004	Monoton.	Thicker $\Rightarrow$ larger HR	W	H
005	Monoton.	Smaller grain $\Rightarrow$ longer wait	I	H
006	Monoton.	Higher force $\Rightarrow$ lower velocity	I	H

Appendix

Table 6: Complete domain specification catalog. Severity: E = Error (safety-critical), W = Warning, I = Info. Method: Z3 = formally proved; H = property testing.

Category	Chk	What it catches
Structural integrity	4	Wrong signature, missing returns, global state, class definitions
Security / sandbox	6	Imports, exec/eval, file I/O, dunder access, forbidden modules
Action mask compliance	3	Mask not referenced, not checked, no fallback
Bounds & clipping	3	Outputs outside $[0,500]$ , $[1,120]$ , $[1,6]$
Division safety	2	Unguarded divisions, NaN/Inf risk
Info dict usage	4	Missing critical keys, force unawareness

Appendix

Table 7: AST-based check categories. Each category targets a specific failure mode of LLM-generated code.

Variable	Min	Max	Unit
current_thickness	5.0	110.0	mm
target_thickness	5.0	15.0	mm
hr_limit	20.0	50.0	mm
stock_temperature	800	1523	K
target_temperature	1073	1273	K
current_grain_size	5	500	$\mu$ m

Appendix

Table 8: Input value ranges for interval analysis and Z3 constraints.

Figure 9: Population audit of the 174 synthesized heuristics. Left: share of heuristics failing each of the most frequent checks, all at warning level. Right: warning-level findings against search reward for the ad-hoc (squares) and Luby (dots) pools, with vertical jitter. Warnings increase slightly with reward (Spearman

\rho=0.16

), so they are not a quality signal.

Figure 10: Expected marginal gain in best reward from continuing a run for one more iteration versus starting a fresh run, as a function of the current iteration. Restarting pays more during the first few iterations; the curves meet around iteration 4.

Runs	$n$	one run of 20	four runs of 5	$L{=}1$	optimal	percentile
		$T{=}20$ : optimal allocation minus		$T{=}40$		Luby
All	52	$+0.1$ $[-2.5,+2.1]$	$+1.6$ $[-1.1,+5.2]$	36.4	45.4	39
Gemini 2.5 Pro	49	$+0.1$ $[-2.5,+2.3]$	$+1.4$ $[-1.1,+5.1]$	36.4	45.4	38
Final settings	27	$+1.2$ $[-1.4,+3.6]$	$+0.8$ $[-2.3,+5.6]$	37.0	46.8	28
Reached planned length	17	$+2.1$ $[-0.8,+4.2]$	$+0.9$ $[-3.2,+6.6]$	36.2	47.2	24

Appendix

Table 9: Restart analysis on subsets of the ad-hoc runs: paired bootstrap differences at

T{=}20

between the optimal allocation (10+10 in every subset) and two uniform allocations (median and 95% CI), expected best reward at

T{=}40

of one-iteration sampling and of the optimal allocation, and the percentile of the observed Luby campaign among 20,000 simulated ones.

Figure 13: Restart analysis at fixed reward thresholds

\tau

. Left: empirical probability that a run’s best-so-far reward after

L

iterations is at least

\tau

. Right: expected iterations to reach

\tau

when restarting every

L

iterations,

\mathbb{E}[\min(X,L)]/P(X\leq L)

, in the same colors; dots mark the optimal cutoff

L^{*}

, which depends on the target.

Parameter	Value
LLM model	Gemini 2.5 Pro (49), 2.5 Flash (3)
Temperature	0.7 (18), 1.0 (10), 1.5 (2),
	not recorded (22)
Outer iterations per run	1–30
Independent runs	52
Ideas per iteration	3 (43), 2 (7), 4 (2)

Appendix

Table 10: Search configuration of the 52 ad-hoc runs. For settings that changed between groups of runs, the number of runs is given in parentheses (Table 11 ).

Runs started	$n$	Model	T	Ideas	Stag.	Memory
19 21:11–23:14	4	Pro	–	2	–	compaction 20/10
19 23:23–20 00:03	3	Pro	–	3	–	compaction 20/10
20 00:58–03:07	3	Pro	–	3	8	compaction 20/10
20 01:19–02:21	3	Pro	–	2	8	compaction 20/10
20 02:24–02:58	2	Pro	–	4	8	compaction 20/10
20 03:52–09:27	4	Pro	–	3	6	compaction 25/10

Appendix

Table 11: Groups of ad-hoc runs with identical settings, in order of their first run (all in February 2026). T: sampling temperature; Stag.: stagnation threshold; compaction

a/b

: compact at

a

messages, keep

b

Figure 14: Best-so-far reward by memory strategy. Synthesis resets the conversation each iteration and provides the LLM with a structured

{\sim}

800-word knowledge summary synthesised by a narrator; Compaction preserves the full conversation and summarises old messages when the context grows too large. Solid lines show medians; shaded bands show IQR. Labels give the number of runs contributing at each iteration, which decreases as shorter runs drop out. The synthesis strategy shows higher median reward in early iterations, though sample sizes are modest (16 vs. 36 runs) and the two groups were run sequentially, so this comparison should be interpreted cautiously.

Iter	LLM’s stated strategy	Reward
0	“Adaptive Two-Phase: bulk reduction then precision finishing”	23.5
1	“Fix catastrophic temperature failure: every scenario loses ${\sim}$ 2.0 pts”	23.3
3	“Proactive heat conservation via interpass time modulation”	33.8
9	“Temperature excels but core reduction strategy has systemic flaws”	33.1
10	“Radical departure: the conflict between thermal and metallurgical efficiency requires a fundamentally new approach”	16.6
14	“Fuse strain-based HR with re-tuned thermal response—hybrid synthesis”	38.0

Appendix

Table 12: Strategy evolution over a 20-iteration search, in the LLM’s own words. Reward is the mean across 8 scenarios.

Family (cases)	Baseline	Best single	Portfolio $K{=}4$	Pool oracle
Thicker slabs, 130–150 mm (6)	38.6	56.9 / 60.5	62.4 / 68.1	63.9 / 68.2
Thin targets, 2–4 mm (6)	33.1	$-$ 17.2 / 43.0	49.1 / 50.4	49.1 / 62.8
Thick targets, 20–40 mm (6)	20.8	27.9 / 28.3	38.7 / 31.9	46.0 / 47.0
Tight mills, HR limit 10–15 mm (6)	32.5	84.5 / 79.3	91.4 / 85.6	91.9 / 91.8
Loose mills, HR limit 55–60 mm (4)	13.2	34.7 / 8.7	34.8 / 39.1	35.1 / 40.3
Off-grid temperature pairs (4)	40.9	62.8 / 53.4	64.5 / 63.1	66.1 / 64.1

Appendix

Table 13: Mean episode return per out-of-distribution family (ad-hoc pool / Luby pool). The best single heuristic and the

K{=}4

portfolio are selected on the in-distribution grid; the pool oracle keeps the best heuristic of the whole pool per case.

Figure 15: Mean episode return under multiplicative Gaussian observation noise for the best single heuristic and the

K{=}4

portfolio of each pool, both selected on the in-distribution grid, and for the hand-coded baseline (three scenarios

\times

three seeds per noise level; the noise-free values are for the same three scenarios).

Automated Heuristic Design

Automated heuristic design creates or adapts problem-solving rules for optimization, planning, scheduling, and related search tasks.

See all 84 papers

Code to Control: Synthesizing Parameterized Reactive Controllers
Sep 30, 2026Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1RL Control Language Model-Based Control
Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition
An LLM-Driven Workflow for Automated Process Control Strategy Generation and Tuning from Dynamic Process Models
Jul 23, 2026Ari Luna Rueda, Eike Cramer, Klaus Hellgardt +1Language Model-Based Control Feedback Control
Department of Chemical Engineering Imperial College London London, UK · Department of Chemical Engineering University College London London, UK
HarnessMDP: Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
Jul 5, 2026Haiwen Yi, Xinyuan SongOffline RL LLM Agent Harnesses
University of Toronto · Emory University

Sep 30, 2026J/K move · Enter open · S save

Code to Control: Synthesizing Parameterized Reactive Controllers

Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates, Samuel J. Gershman

Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

RL Control Language Model-Based Control LLM-Based Program Synthesis Derivative-Free Optimization

Open paper AskPDF arXiv

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.

Abstract

Automated Heuristic Design

Explore similar work

Code to Control: Synthesizing Parameterized Reactive Controllers

An LLM-Driven Workflow for Automated Process Control Strategy Generation and Tuning from Dynamic Process Models

HarnessMDP: Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning