NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents

Lijie Ding; Changwoo Do

cs.AIOct 2, 2026

NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents

Authors: Lijie Ding, Changwoo Do

Organizations: Neutron Scattering Division, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA

Abstract

Designing a scientific instrument tests whether language-model agents can do physics rather than recall it, provided the grading cannot be argued with. We introduce NeutronGym, to our knowledge the first executable environment for neutron instrument design: agents build instruments through validating tools, McStas ray-traces what they build, and a level-resolved ladder grades syntax, runtime, structure and science with no LLM judge. Procedural families supply unlimited instances of a fixed layout whose design parameters the agent must set, with held-out parameter regimes; a curated slice, McStasBench, adds 16 tasks from published instruments behind memorization probes and a sandbox. Seven models reproduce at most 7 of the 16, none retrieves a reference, and none meets an improvement target. The environment also trains. Reinforcement learning on its reward takes Qwen3-8B from 11% to 77% of held-out instances of a family whose targets come from a hidden design (69% at a second seed), past an untrained Qwen3-32B, and the recipe holds, at one seed each, on three further gated families. The analysis says what that gain is. Without the ladder's partial credit it collapses by 60 points. From reward alone the trained model reaches what a classical optimizer reaches, at the agent's simulation budget, only when handed the closed-form physics (77% against 81%, a gap that does not separate at this size), while frontier models still solve 98-99%. Getting a trustworthy result meant failing four task designs that no-model baselines could solve, and we release the probes that found them.

Figures & tables

Figure 1: NeutronGym. (a) Tasks come from procedural families or from McStasBench, the curated slice; (b) the agent works only through validated tools in a sandbox, and (c) the environment runs what it builds and grades it on a level-resolved ladder. (d) One episode: the instrument the agent arrived at, and the beam profile (horizontal position) and divergence (horizontal angle) McStas returns for it (solid) against the baseline (dashed) and a hidden design’s targets, shaded at

\pm 5\%

;

m

is the supermirror coating index.

Model	Loop	1-shot	Loop	1-shot	Dominant failure
	Seen (of 16)		Held-out (of 2)
Claude Sonnet 5	5	7	2	1	L1 does not compile
Claude Haiku 4.5	4	4	2	0	L4 wrong physics
Gemini 3.6 Flash	4	6	2	1	L0 never finished
Gemini 3.5 Flash-Lite	2	1	1	0	L0 never finished
Llama 4 Maverick	0	2	0	0	L0 never finished

Table 1: McStasBench results, one episode per task; no per-model difference is significant at this size. “Dominant failure” is where most failed loop episodes stop on the ladder. Maverick’s held-out episodes ran on two serving routes (Appendix A ) and all four failed.

Figure 2: Pass rates on held-out instances (

n=300

;

150

for the no-model probe) with Wilson 95% intervals, covering instance sampling for a fixed policy, not training-seed variation (8 points) or grading-seed fragility (§ 5.3 ). (a) The trained 8B passes the untrained 32B on all four gated families and no fixed-answer or lookup baseline comes close; ∗∗∗ :

p<10^{-3}

, paired exact McNemar against the untrained 32B (discordant 208 vs. 12, 89 vs. 43, 116 vs. 12, 125 vs. 29). Guide match is replicated at a second seed; the other three are single runs. (b) Guide match under a tightening tolerance: the trained policy’s designs sit inside the tolerance, the untrained one’s on its edge.

	Family / slice	Pass rate	Read against	$\Delta$
Something simpler did it
Search from the baseline, agent’s budget	guide match	13.3%	76.7% RL	$-63$
Search from a random start, agent’s budget	guide match	14.3%	76.7% RL	$-62$
Search from the baseline, $3\times$ the budget	guide match	73.7%	76.7% RL	$-3$
Hand-coded physics rule, no search	guide match	27.7%	76.7% RL	$-49$
Physics rule, then search, agent’s budget	guide match	81.0%	76.7% RL	$+4$

Table 2: Attempts to explain the result away. On guide match the RL-trained 8B passes 76.7% of 300 held-out instances; each row is a candidate account of that number, read against the reference in its own row (

\Delta

in points). All search rows are coordinate descent, the best of the three optimizers we tried (Nelder–Mead, random search, coordinate descent) at both budgets, and differ only in where the search starts; the seeded search’s 4-point lead does not separate at this size (§ 5.3 ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Access	Serving	Notes
Claude Sonnet 5	OpenRouter	provider-pinned (Google)
Claude Haiku 4.5	OpenRouter	provider-pinned (Google)
Gemini 3.6 Flash	OpenRouter	provider-pinned (Google)
Gemini 3.5 Flash-Lite	OpenRouter	provider-pinned (Google)
Llama 4 Maverick	OpenRouter	DigitalOcean ‡	see § 4.2
Qwen3-32B	local vLLM	4 GPUs, YaRN $\times$ 4, 98,304 ctx	thinking off

Appendix

Table 3: Models and serving routes. The eighth row ran one seen-tier episode ( HZB_FLEX , passed) before being cut on cost and is excluded from every reported number.

Introspection: the 374 shipped components
Tool	Purpose
list_components	Browse components by category or search term
describe_component	Full parameter table for one component: type, unit, default, required, documentation
list_examples	Browse the shipped example instruments
get_example	Full source of one shipped example
Construction: validated at call time

Appendix

Table 4: The tool surface. Agents never write files or run a shell; these calls are the whole action space.

Task	Class	Site	Mon/obs	Note
Seen T1: 12 scored
ESS_IN5_reprate	ToF spectrometer	ESS	3 / 8
HZB_NEAT	ToF spectrometer	HZB	3 / 8
ILL_IN6	ToF spectrometer	ILL	3 / 8
HZB_FLEX	Triple-axis	HZB	2 / 6
ILL_H10_IN8	Triple-axis	ILL	2 / 6

Appendix

Table 5: Every McStasBench task. Seen T1 references are McStas shipped examples; held-out references we authored from published designs ( Wang et al., 2025 ; Bilheux et al., 2015 ) and never released.

#	Where	What went wrong	Fix
1	T2 calibration	Unconstrained flux maximization collected 672 n/s of direct beam past the beamstop; an upper limit on beam width cannot catch this, because leakage narrows the pattern.	Two-sided band constraints relative to the baseline pattern.
2	T2 calibration	The simulator’s Nelder–Mead optimizer ignored parameter bounds (a width escaped to 2.1 m).	Classical baselines are bounds- and constraint-filtered ensembles.
3	T2 calibration	Winner’s curse: the best of 30 noisy evaluations fell below its own target at a fresh seed.	Re-verify any selected optimum.
4	Agent shakedown	Matching monitors by event count graded an agent’s diagnostic monitor as its detector; correct instruments from two models failed 3/6.	Match monitors by role and position.
5	Phase-space check	A legitimate action read $1.14\times$ the bound on a 250-event monitor; at $10\times$ the rays it read 0.99.	Evaluate the bound only above the statistics floor.
6	SANS family	Maximal pinholes, with no model, passed 8/25 held-out instances, one at 11,992 $\times$ the target: unscattered beam leaked around the beamstop and counted as signal. 22 of 35 cached optima were such leaks.	Geometric direct-beam check at L1.

Appendix

Table 6: The nine reward failures, where each was found, and its fix.

Family	Objective	Fixed / lookup	Readout rule	Verdict
Guide divergence	maximize	13.3%	96.3%	rejected
SANS collimation	maximize	14.0%	56.0%	evaluation only
Bender, first version	match	10.7%	—	rejected (concentration)
Guide match	match	0 4.0%	27.7% †	admitted
SANS match	match	12.7%	0 3.3% †	admitted
TOF chopper	match	0 2.7%	0 0.0%	admitted

Appendix

Table 7: Probes per family, on 150 held-out instances unless noted. Four designs were discarded: an undersized-baseline guide version that predates the gate (§ 3 , not shown), the guide family below, where a rule copying two limits printed in the prompt passed 96.3% of the 300 instances used for training evaluation (98.0% on the 150-instance probe), the SANS collimation family, kept for evaluation only, and the first bender family, where the gate passed but a quarter of instances had a hidden design that filtered nothing, visible only as concentration.

Family	Free parameters	Objective / specification	Held-out regime
Guide (rejected)	entrance and exit width $\in[1,9]$ cm; $m\in[1,3]$	maximize flux within the divergence limit $m\cdot 0.099^{\circ}\!/\text{\AA}\cdot\lambda\leq\theta_{\max}$ ; exit $\leq$ sample width	guide 12.5–16 m (train 6–12)
SANS (evaluation only)	two pinhole radii $\in[1,20]$ mm	maximize scattered intensity; beam inside the beamstop, $q_{\min}\leq 1/R_{\text{sphere}}$ , beam fits the sample	collimation 4.5–6 m (train 2.5–4)
Guide match (admitted)	as guide, without the divergence-limit check; sample $0.1$ – $1.5$ m after the exit	match spot size and divergence within $\pm 5\%$	guide 12.5–16 m (train 6–12)
SANS match (admitted)	two pinhole radii $\in[1,20]$ mm	match beam width at the sample and at the beamstop within $\pm 1.5\%$	collimation 4.5–6 m (train 2.5–4)
TOF chopper (admitted)	chopper frequency 20–60 Hz, phase, opening angle	match mean wavelength and spread at the sample within $\pm 5\%$	chopper separation 9.5–12 m (train below)
Bender (admitted)	radius of curvature $40$ – $550$ m; aperture width $2$ – $8$ cm; $m\in[1.5,4]$	match the transmitted band’s mean wavelength within $\pm 0.25\%$ and its spread within $\pm 1\%$ ; the hidden design must cut 25–85% of the incident band	bender 23–30 m (train 12–22)

Appendix

Table 8: Procedural families and their status. Specification checks run at L1 before simulation and band constraints at L3, on the two maximization families only; the matching families carry neither and are graded on hidden-design targets. Tolerances were fixed before training by measuring the untrained 8B (for SANS match the sweep read 45.3% at

\pm 3\%

, 24.7% at

\pm 1.5\%

and 18.0% at

\pm 1\%

, and

\pm 1.5\%

was adopted); the sweeps are in the release.

Run	Family	Pass rate	Read against	$\Delta$
Imitation of all turns (pre-registered)	guide, pre-repair	31.3%	40.3% untrained	$-9$
Imitation of passing turns	guide, pre-repair	52.3%	52.3% by one fixed action	$0$
Imitation of passing turns	SANS, pre-repair	27.0%	31.0% untrained	$-4$
Same, harder targets	SANS, pre-repair	0 1.0%	0 8.3% untrained	$-7$

Appendix

Table 9: Supervised imitation runs, each read against its own untrained baseline on the pre-repair task version it ran on. All four predate the 2026-09-16 feedback-label fix (§ 5.3 ) and none is comparable with the RL rows of Table 2 .

	Passes		Loop failures					One-shot failures
Model	Loop	1-shot	L4	L3	L2	L1	L0	L4	L3	L2	L1	L0
Claude Sonnet 5	5	7	1	0	0	9	1	0	1	2	6	0
Claude Haiku 4.5	4	4	4	2	2	4	0	0	0	0	12	0
Gemini 3.6 Flash	4	6	0	1	0	1	10	1	1	2	6	0
Gemini 3.5 Flash-Lite	2	1	1	0	1	5	7	0	0	0	15	0
Llama 4 Maverick	0	2	0	2	0	1	13	0	0	0	14	0

Appendix

Table 10: Seen-tier results, 16 scored tasks per cell. Failures are counted by the deepest level reached; L0 means the episode ended without a finished instrument (loop) or without a parseable file (one-shot). Levels are read mechanically from the grading report. Over both arms the 185 seen-tier failures are 89 at L0, 73 at L1, 8 at L2, 8 at L3 and 7 at L4.

Run	Pooled	Guide match	SANS match	TOF chopper	Bender
Specialists, one per family	—	76.7%	51.0% ∗	46.7%	75.7%
joint3	3 families	67.7%	24.0%	55.0%	—
joint4	4 families	63.7%	43.3%	56.3%	55.0%
joint3rep	3, new seed	34.0%	19.3%	0 2.3%	—
joint3rep , stage A		—	—	11.0%	—
Untrained 8B	—	11.3%	22.3%	0 2.0%	23.0%

Appendix

Table 11: Joint runs, all evaluated on held-out instances 300–599 at 10 turns and temperature 0. joint3rep repeats joint3 exactly (same families, same 8,592 states, same budget and hyperparameters) with only the seed changed, and lands at the untrained rate on TOF chopper. Its stage-A checkpoint was already behind (11.0% against joint3 ’s eventual 55.0%), so the two runs diverged early. Only this three-family configuration was ever run twice: there is no evidence that four-family pooling is the stable one. ∗ The SANS-match specialist’s 51.0% is read on the slice its checkpoint was selected on; on a slice chosen for nothing the same checkpoint gives 44.3%, the number Figure 2 reports; the untrained-8B row is on the 300–599 slice, where SANS match reads 22.3%, against 26.0% on the selection-free slice Figure 2 draws, so the 18-point SANS-match gain quoted in § 5.2 is the figure’s (

44.3-26.0

), not this table’s. The other three specialist cells are like-for-like with the joint runs.

Policy	Passed	Note
Untrained Qwen3-8B	34/300 0 (11.3%)	exhausts the 10-turn cap
Untrained Qwen3-32B	34/300 0 (11.3%)	scale buys nothing here
Best fixed design or lookup	00/300 (4.0%)	incl. other instances’ solutions
Nelder–Mead, 10 simulations	8/300 00 (2.7%)	40.0% at 30 simulations
Random search, 10 simulations	22/300 0 (7.3%)	20.0% at 30
Coordinate descent, 10 simulations	40/300 (13.3%)	73.7% at 30

Appendix

Table 12: Guide match, held-out instances 300–599, a slice used for nothing but this measurement. Agents get 10 turns at temperature 0; classical optimizers get the same simulation budget and act on the parametrization directly, which the agent reads from its task sheet and reaches only through tool calls. Italic rows are no-model baselines. Frontier models were run on the first 100 instances of the slice, where the trained model scores 76 of 100.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 149 papers

Compositional Machine Design as Program Synthesis with LLMs
Oct 16, 2025Wenqian Zhang, Yangyi Huang, Weiyang Liu +1LLM-Based Program Synthesis Program Synthesis
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong
Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
May 25, 2026Matt L. Wiemann, Lindsay M. Smith, Peter Melchior +4Physical Reasoning Scientific Discovery
Princeton University · Work done during internship at NYU/Polymathic AI. · Boston University +3

Oct 16, 2025J/K move · Enter open · S save

Compositional Machine Design as Program Synthesis with LLMs

Wenqian Zhang, Yangyi Huang, Weiyang Liu, Zhen Liu

The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong

LLM-Based Program Synthesis Program Synthesis AI Agent Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong abilities in writing and revising programs, yet many program-synthesis benchmarks still evaluate programs in symbolic or digital environments. We introduce compositional machine design, a physically grounded form of program synthesis where machines are written as programs that compose standardized parts, and success is determined by simulated physical behavior. To study this problem, we present BesiegeField, a testbed built on the machine-building game Besiege. In BesiegeField, LLM agents generate machine programs from textual functional demands, execute the resulting machines in simulation, and receive rewards and state feedback. We benchmark LLM agents across representative machine-design tasks under single-agent generation, iterative editing, and hierarchical workflows. Strong models recover task-relevant structures and sometimes achieve nontrivial physical performance, but often struggle with spatially precise assembly, mechanism-level planning, and translating feedback into useful structural edits. We further finetune Qwen2.5-14B, an open-source LLM, with reinforcement learning from simulation-derived rewards. We find that, under a fixed generation budget, RL improves the best machine discovered. We additionally evaluate human performance to provide a reference point for task difficulty. These results establish compositional machine design as a testbed for studying LLM agents that synthesize executable machine programs and improve them through physical feedback.

Abstract

Scientific Reasoning in Language Models

Explore similar work

Compositional Machine Design as Program Synthesis with LLMs

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking