LLM Agents as Suppliers of Physical System Models: Calibration and Acceptance in Modelica

Jörg Frochte; Herbert Schmidt

cs.LGOct 18, 2026

LLM Agents as Suppliers of Physical System Models: Calibration and Acceptance in Modelica

Authors: Jörg Frochte, Herbert Schmidt

Organizations: Bochum University of Applied Sciences

Abstract

To procure a model of a device, a client hands a supplier a description, sometimes with recorded data or a target response, and accepts the model against this handover. We let LLM coding agents act as the supplier on handovers with data, building Modelica models from library components for four simulated systems and a real switching magnet, and ask what acceptance certifies. Several agents reach the noise floor, but a handover admits a set of models: the description leaves parts of the structure open, and the data narrow it only to the agreed accuracy. We therefore accept by membership in this set, not by matching one reference, and add layers above behaviour: a physical invariant exposes an energy-creating model at the noise floor. Interpretability becomes a property of the delivered model, tested on a variant, read where none exists and shaped through the handover. Most failures are fits delivered with a systematic residual on their own fitting data, where the handover left open how close is close enough. In a follow-up, stating a threshold makes the agents compare against it, and more deliveries pass. The argument applies wherever a handover underspecifies a model meant to be read, not only run.

Figures & tables

Figure 1: The electromechanical coupling of task 4 in two delivered models that both pass conformance and behaviour (Section 4 ): (a) the library component, (b) the coupling rebuilt from signal voltage, torque source and two gains. Components and connections as delivered, straightened, on one scale; signal connections dashed, power connections solid, electrical blue, mechanical grey; power ratios from simulations of the delivered models.

Figure 2: The acceptance scale: for each layer, what it certifies, the instrument, and a failure from this study that the layer catches. Each layer catches failures that the layers below it can pass. Only conformance is a precondition, and the other layers are graded independently. The reading was applied to seven magnet models; the check of the file against its report runs alongside the layers.

Figure 3: The runs of the base matrix for the nine LLMs that reach the noise floor at least once, one square per run in the order the runs were made. On tasks 1 to 4 the colour gives the group into which the precedence rule of this section sorts the run, and on task 5 whether the run delivered a conformant model and met every feature of the frozen list. A white dot marks a run that passes the parameter layer

V_{2}

; runs without a dot fail it or offer nothing to compare with the reference. Right: median cost of a run that delivered a gradable model on tasks 1 to 4, in US dollars at the list prices of the pinned endpoints; the smallest LLM is cheaper per token, but its runs are several times longer; each run is a single attempt without feedback. Sizes are the total parameters of the open-weight LLMs.

Task	Model	Fit	Validation	Test	$\times 2$	$\times 3$	Variants
1	delivered	1.02	1.02–1.24	1.05–1.06	1.04–1.18	1.06–1.33	1.05–1.11
	linear	1.02	1.15	1.06	1.18	1.29	274–387
2	delivered	1.01	1.03–1.23	1.05–31	1.03–11	1.01–1035	1.01–4.9
	linear	56	406	293	724	1154	392–1073
	network	12–22	178–216	222–324	618–758	1023–1199	473–1474
3	delivered	1.06–1.07	1.02–1.27	1.05–2.26	1.05–4.21	1.03–6.28	1.06–1.17

Table 1: Worst ratio of residual to noise per group of experiments. Delivered : range over the runs that pass the behaviour layer (19, 19 and 20). Linear : output-error state-space model. Network : the linear model with a neural correction, chosen like it by leaving one fitting experiment out, range over three seeds; on task 1 the selection returns the linear model. The columns

\times 2

and

\times 3

scale the inputs of the test trapezoid, chirp and random steps by two and three, and that of the first fitting experiment by three. Variants raise one parameter by half, in the device and in the delivered model, and the column gives the range over both variants. The networks are simulated in Python with the input as applied, all other values with the test bench of the acceptance. Task 4 has neither variants nor black boxes.

Figure 4: Every conformant graded run of the nine LLMs on the first four tasks: worst residual on the fitting experiments against worst residual on the validation experiment, in units of the disclosed noise. The gate of Eq. ( 3 ) divides the plane, and the dashed diagonal marks equal residuals on both. Colour gives the task; crosses mark the two files, both on task 1, that do not carry their calibration. Points beyond 300 sit at the edge. The five runs without a gradable model are not shown.

Base handover
Configuration	Runs	Features missed	Worst residual
gpt-5.6-sol	5	0, 1, 1, 5, 6	1.6 – 2.2
Opus 5	5	0, 0, 0, 0, 1	1.7 – 3.5
gpt-5.6-terra	3	0, 0, 0	1.5 – 4.3
Sonnet 5	3	4, 6, 6	3.9 – 11.0
DeepSeek V4 Pro (1.6 T)	3	2, 4, 6	1.9 – 6.1

Table 2: The magnet by configuration: graded runs, features missed per run against the frozen list, and the range of the worst-output residual in units of the stated scale of Section 4 . Shown are the base handover, the builder’s briefing for the four configurations it was issued to, and the smallest configuration under the two controls and the harness check. The briefing cell of Kimi K3 has three runs although they disagree: its diverging run misses the same two features as that of gpt-5.6-sol, so further runs would have enlarged the sample without settling the question. The OpenCode row of the controls pools five runs at the harness defaults with three in which the context length was declared (Appendix E ).

Figure 5: The decisive pair on the magnet. (a) The builder’s cutaway drawing from the briefing, scaled by its one stated dimension, with the pole region enlarged; below it the pole faces as measured and as the two delivered models carry them, and the builder’s verdict on reuse (Table 8 ). (b) The validation experiment, current-controlled at 8 A: the measurement and both delivered models, simulated as the evaluator does with the measured coil voltage as input. By the builder’s rating of their behaviour the two models cannot be told apart; the builder would adapt one of them to a new geometry and refit the other.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Reference	Values	Experiments	Noise	$V_{2}$
1	Force on Mass , SpringDamper to Fixed ; position and speed sensors	$m=2.0$ kg, $c=8.0$ N/m, $d=0.8$ N s/m	fit: pulse of $5$ N for $0.5$ s; step of $2$ N. Validation: sine of $3$ N at $0.3$ Hz. $25$ s at $0.05$ s, seed 42	$y_{s}$ : $3$ mm; $y_{v}$ : $10$ mm/s	1 % on $m$ , $c$ , $d$
2	Force on MassWithStopAndFriction , Spring to Fixed ; position sensor	$m=1.5$ kg, $c=5.0$ N/m, $F_{C}=0.6$ N, $F_{\mathrm{prop}}=0.3$ N s/m; no Stribeck term, stops at $\pm 100$ m	fit: pulse of $8$ N for $0.5$ s; steps of $0.5$ and $3$ N. Validation: sine of $4$ N at $0.25$ Hz. $25$ s at $0.05$ s, seed 4242	$y_{s}$ : $2$ mm	5 % on $m$ , $c$ , $F_{C}$ , $F_{\mathrm{prop}}$
3	SignalVoltage , Resistor , Inductor , RotationalEMF in series; Inertia with Damper to Fixed ; current and speed sensors	$R=2.0\,\Omega$ , $L=1.0$ mH, $k=0.1$ N m/A, $J=2\cdot 10^{-3}$ kg m 2 , $d=5\cdot 10^{-4}$ N m s/rad	fit: step of $12$ V at $1.013$ s; step of $6$ V at $1.013$ s, off at $12.017$ s; sine of $8$ V at $0.2$ Hz. Validation: sine of $10$ V at $1$ Hz. $25$ s at $0.05$ s, seed 333	$y_{i}$ : $0.01$ A; $y_{w}$ : $0.05$ rad/s	2 % on $k$ , $J$ , $d$ ; 5 % on $R$ ; factor 3 on $L$
4	SignalVoltage from a CombiTable1Ds over the integrated state of charge; two Resistor s, Inductor , RotationalEMF ; rotor Inertia and Damper , IdealGear , load Inertia and Damper , Torque as brake; current and output-speed sensors	$R_{i}=0.14\,\Omega$ , $R_{a}=0.36\,\Omega$ , $L_{a}=0.042$ H, $k=1.15$ V s/rad, $J_{\mathrm{rot}}=0.014$ kg m 2 , $d_{\mathrm{rot}}=0.0025$ N m s/rad, ratio $5.2$ , $J_{\mathrm{load}}=4.8$ kg m 2 , $d_{\mathrm{load}}=0.35$ N m s/rad, $Q_{\max}=2\cdot 10^{5}$ C from $0.85$ ; open-circuit voltage $198$ , $216$ , $226$ , $236$ , $246$ , $253$ , $260$ V at state of charge $0$ , $0.1$ , $0.25$ , $0.5$ , $0.75$ , $0.9$ , $1$	fit: brake torque of $50$ N m over $20$ s at $0.02$ s; staircase of $100$ , $300$ , $500$ , $200$ N m, $30$ s each, at $0.05$ s; $400\pm 200$ N m sine of period $300$ s over $1800$ s at $0.5$ s. Validation: staircase of $250$ , $550$ , $150$ , $450$ N m with steps at $150$ , $400$ , $650$ s, over $900$ s at $0.5$ s. Seed 2606	$y_{i}$ : $0.5$ A; $y_{w}$ : $0.2$ rad/s	5 % on $R_{i}+R_{a}$ , $L_{a}$ , $k$ , $J_{\mathrm{eq}}$ , $d_{\mathrm{eq}}$

Appendix

Table 3: The four synthetic tasks. Inputs act from

t=1

s on tasks 1 to 3 unless a time is given, and from

t=0

on task 4. On task 4,

J_{\mathrm{eq}}

and

d_{\mathrm{eq}}

refer rotor and load to the motor shaft through the ratio. The last column gives the quantities that

V_{2}

compares and their bands.

Feature	Experiment	Measured	Tolerance
Closing time	voltage-controlled, $6$ V	$3.90$ ms	$\pm 20$ %
Closing time	current-controlled, $4$ A	$2.64$ ms	$\pm 20$ %
Closing time	current-controlled, $8$ A	$1.06$ ms	$\pm 20$ %
Release time	current-controlled, $4$ A	$1.68$ ms	$\pm 25$ %
Release time	current-controlled, $8$ A	$1.76$ ms	$\pm 25$ %
Plateau current	voltage-controlled, $6$ V	$6.82$ A	$\pm 5$ %

Appendix

Table 4: The nine graded features of task 5 with their values on the measurement and the tolerances fixed after the first run of the base cell and before any further run. The third experiment is the validation experiment. Times are counted from switch-on or switch-off as defined in the text.

Task 1
Perturbed quantity	exp1	exp2	exp3	validation
mass $m$	1.88	1.22	–	12.88
stiffness $c$	1.91	1.48	–	14.19
damping $d$	1.04	1.03	–	2.82
Task 2
mass $m$	2.71	1.01	1.09	28.35

Appendix

Table 5: Sensitivity of the four synthetic tasks to one parameter at a time: residual in units of the noise when one reference quantity is perturbed by one per cent, the larger of the two signs, per experiment. The other quantities stay at their reference values, and of two outputs the worse is taken. A value near one means that the experiment does not see the perturbation. Task 1 has two fitting experiments. The last row of task 3 perturbs the inductance by the factor of three that

V_{2}

admits.

Task	Solved	Assemblies
1	19	Mass , SpringDamper
		Damper , Mass , Spring
2	19	MassWithStopAndFriction , Spring
		MassWithStopAndFriction , SpringDamper
		Damper , MassWithStopAndFriction , Spring
		Mass , SpringDamper , SupportFriction

Appendix

Table 6: Component assemblies among the runs that reach the noise floor. Sensors, sources, ground and fixed reference points are left out because they are measurement rather than mechanism, and signal-path arithmetic (gains, sums, constants) because it is notation rather than mechanism. Two runs that differ only in how they read out or scale a signal therefore count as one assembly, and a converter rebuilt from controlled sources and gains appears without a converter component. Solved is the number of runs at the noise floor that the assemblies are drawn from.

Configuration	Harness	Runs
Opus 5	Claude Code 2.1.251–2.1.281	22
Sonnet 5	Claude Code 2.1.251–2.1.266	17
DeepSeek V4 Pro 0813	OpenCode 1.18.19–1.18.25	19
gpt-5.6-luna	Codex 0.147.0	21
gpt-5.6-sol	Codex 0.147.0–0.154.0	22
gpt-5.6-terra	Codex 0.147.0	19

Appendix

Table 7: Configurations of the nine LLMs that reach the noise floor at least once, as recorded in the run manifests: harness and its versions and number of runs, without the follow-up on the acceptance criterion. Every configuration ran at the highest reasoning setting its provider offers.

Run	Magnetic network, assessment condensed	Reuse	Usable
Opus 5, 5	Yoke, poles and armature as geometric flux tubes; pole radii $5.35$ and $11.67$ mm; eddy path in the inner pole; two leakage paths around the poles and one between them. Moving mass $21.18$ g, coil $0.786$ $\Omega$ . Corner treatment inconsistent between inner and outer pole; outer fringing radius $8.6$ mm larger than the geometry supports. “Would pass with good marks.”	yes	yes
Opus 5, 1	The network of run 5 element for element, in almost the same arrangement; pole radii $5.77$ and $11.72$ mm; yoke bottom fitted at $1.05$ mm where run 5 fixed it at $3$ ; one fringing radius of $1.4$ mm for both poles. Moving mass $19.60$ g, coil $0.793$ $\Omega$ . A damper of $0.20$ Ns/m on the armature; stiffness and damping set separately for the two end stops, $278$ and $83$ kN/mm. “Comparable to run 5, if anything more clearly organised.”	yes	yes
Kimi K3, 1	Two lumped iron elements with fitted length and area; one working gap of radius $4.17$ mm, the outer pole absent; fringing width $68$ mm without geometric meaning. Remanence as a magnetomotive force of $35.6$ A. Series reluctance equivalent to $102\,\mu$ m of air beside a mechanical residual gap of $5\,\mu$ m: the same effect twice. “Clearly inferior” as a network.	no	yes
Hy4, 1	Two working gaps with geometric radii $5.44$ and $11.25$ mm; one leakage permeance. Iron as a self-written tabulated flux tube with $\mu_{r}=1+\|B\|/(\mu_{0}H)$ , the constant being an error; no iron in the armature; no eddy path. Diagram “dramatically bad”.	partly	no
sol, 1	Coil, magnetic circuit and armature as separate classes. One working gap from a fitted area of $43.8$ mm 2 without physical meaning; fringing dimensions fitted, not derived. Parasitic inductance of $47\,\mu$ H on the electrical side. Stribeck friction, alone among the seven. Its stop parameters are round numbers and its two stops share a stiffness, which the reviewer reads as a structure left uncalibrated.	no	yes
DeepSeek, 3	One working gap, one iron element of length $38.8$ mm and area $69.9$ mm 2 , close to the device; no fringing, no leakage, no eddy path. A mechanical damper of $49.7$ Ns/m stands in for the eddy losses, as the agent’s comments state. “Simple and easy to understand.”	no	yes

Appendix

Table 8: Task 5, seven delivered models read by the builder. Tier is the builder’s overall rating of the run’s standard chart: good, reasonable, not good, not viable. Reuse is whether the builder would adapt the model to a geometry variant rather than refit it; Usable is whether the diagram as delivered could be used without redrawing it; rearranging its elements is allowed, which is why five are usable while only one could be worked with without any rearrangement.

Model Calibration

Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.

See all 218 papers

Aug 24, 2026

Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.

Zizhe Wang

LLM Agent Harnesses AI Agent Benchmarks

Ask PDF

May 22, 2026

LLM-driven design of physics-constrained constitutive models: two agents are better than one

Developing constitutive models that capture how materials deform under load traditionally requires years of specialized expertise in continuum mechanics, machine learning, and scientific programming. Large language models (LLMs) have recently been shown to lower this barrier by generating constitutive models on demand, but existing single-agent pipelines lack systematic checks that the resulting models respect fundamental physical laws. To close this gap, we introduce the first multi-agent LLM-driven approach for constitutive model generation: a Creator agent proposes a model tailored to the data, while an Inspector agent critically audits each proposal against nine physical constraints and returns it for refinement whenever a violation is detected. We demonstrate this concept with constitutive artificial neural networks (CANNs) and benchmark it on brain tissue and rubber as isotropic materials, and on porcine skin tissue as a transversely isotropic material with a preferred fiber direction, using two different LLM backbones (Claude Opus 4.7 and Kimi K2.5). Whether a generated model satisfies the physical constraints is assessed numerically, by probing each constraint across a broad sample of deformation states, rotations, and perturbation directions. Adding the Inspector raises the share of exported models that pass all these checks from 90% to 95% for Opus and from 47% to 60% for Kimi. In addition, the generated models are on par with or even surpass expert-designed models in accuracy, extrapolate reliably beyond the training data, and generalize remarkably well to unseen loading paths. Separating generation from inspection thus turns LLM-driven constitutive modeling into a substantially more trustworthy process. The paradigm is deliberately technique-agnostic...

Marius Tacke, Matthias Busch, Kian Abdolazizi +4

1Helmholtz-Zentrum Hereon, Geesthacht, Germany · 2Hamburg University of Technology, Hamburg, Germany · 3RWTH Aachen University, Aachen, Germany +2

Neural Surrogate Modeling Multi-Agent LLM Systems Generative Modeling Physics-Informed ML

Ask PDF

Oct 26, 2026

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents

Language model agents often abandon correct work when a user challenges it, yet simply telling them to stand firm makes them ignore real mistakes. Once an agent reads a complaint, it tends to discount its own earlier checks in order to agree with the user. We introduce Thinking-in-Bets, a training-free method in which an agent writes a short private record of its main claims, the checks supporting them, and its confidence before reading user feedback. When challenged, the agent consults this Claim Checklist to keep verified work intact while investigating uncertain or newly challenged parts. We evaluate five Gemini models (3.5 Flash-Lite, 3.5 Flash, 3.6 Flash, 3.8 Flash, and 4.0 Argon) across eight benchmarks: SWE-Together, SWE-Interact, and DeepSWE for software engineering, Agents' Last Exam for professional reports, Humanity's Last Exam and SycEval for academic reasoning, Vals Finance for financial analysis, and Terminal-Bench for command-line tasks. Recording claims before criticism nearly eliminates false reversals in patch review and protects passing report drafts under mistaken pushback, whereas writing the same record after reading the complaint recovers very little of the gain. In multi-turn coding, pre-reply claim records substantially reduce unnecessary file edits after mistaken complaints, and gating the record so it is written only when the agent is uncertain lowers token overhead while fixing more real bugs than the baseline. Even in non-interactive tasks without user feedback, listing the key claims in a draft and re-checking the weakest one before submission improves both answer accuracy and confidence estimates across all tested models.

Nghi D. Q. Bui

Google

LLM Sycophancy Behavioral Consistency of LLM Agents Self-Reflection in Language Model Agents

Ask PDF

Abstract

Model Calibration

Explore similar work

Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

LLM-driven design of physics-constrained constitutive models: two agents are better than one

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents