Diagnosing Faults in Reinforcement Learning Simulators with Canonical Polynomial Invariants

Tesfay Zemuy Gebrekidan; Hadush Hailu Gebrerufael

cs.LGAug 12, 2026

Diagnosing Faults in Reinforcement Learning Simulators with Canonical Polynomial Invariants

Authors: Tesfay Zemuy Gebrekidan, Hadush Hailu Gebrerufael

Organizations: Computer Science, MIU, Iowa, USA

Abstract

A large literature incorporates physical structure into learned dynamics on the premise that models respecting the underlying physics should predict better. We test this premise using exact polynomial invariants recovered from trajectories and represented canonically as reduced Gröbner bases over Q. We find that exact polynomial invariants provide little additional benefit for residual-based downstream uses. On Gymnasium's Acrobot, consistency regularisation reduces algebraic residuals without improving long-horizon rollout fidelity, while one-step validation error is more strongly associated with long-horizon accuracy; similarly, shaping potentials recovered from systems with mass errors of up to 100% achieve similar learning performance to the physically correct potential. The main benefit observed here instead appears in diagnosis, where canonical representations permit exact ideal-equality and membership decisions, and where a reference generating set supplies a physical vocabulary that generic two-sample tests do not have. We develop a two-level diagnostic comprising screening, which identifies the violated physical constraint, and attribution, which recovers the faulty invariant and identifies the responsible physical parameter. Normal-form deflation removes algebraically trivial multiples, while quotient-space recovery avoids tolerance-based nullspace selection. Across fifteen injected faults, screening localises all fifteen when no observation noise is added, with no observed false alarms. Attribution identifies the responsible parameter in all seven single-parameter faults. Paired difference tests detect all faults but provide no constraint-level localisation, showing that the advantage is localisation and not detection. In conclusion, exactness is more useful for diagnosing violations of physical constraints than for improving prediction.

Figures & tables

Fig. 1: The diagnostic. Invariant recovery (bottom) turns trajectories from the system under test into a candidate invariant

\hat{p}

(Sections IV – IV-B ). Screening (top left) tests each reference generator’s residual and reports which

p_{i}\in G_{\rm ref}

is violated; attribution (top right) fits

\hat{p}

against a parametric template one parameter at a time and returns a verdict.

Regime, noise	Method	Faults detected	False alarms	Localised	Attributed
unpaired, $\sigma\!=\!0$	MMD permutation	$12/15$	$1/2$	n/a	n/a
	KS $+$ Bonferroni	$10/15$	$1/2$	n/a	n/a
	MLP discriminator	$12/15$	$1/2$	n/a	n/a
	paired difference	n/a	n/a	n/a	n/a
	algebraic, per block	$\mathbf{15/15}$	$\mathbf{0/2}$	$\mathbf{15/15}$	$7/7$
	algebraic, per trajectory	$\mathbf{15/15}$	$\mathbf{0/2}$	$\mathbf{15/15}$	$7/7$

TABLE I: Detection, localisation and attribution over fifteen faults and two healthy controls, one draw per cell. Localisation is scored as exact set agreement with the fault’s known constraint locus, and n/a marks a method with no constraint to name or, for the paired difference test, a regime it cannot run in. The screen appears at two units of replication, blocks cut from each trajectory and whole trajectories under an exact permutation null. The paired noiseless control column is an identity check rather than a false-alarm rate; Fig. 3 measures the rate.

Fig. 2: Acrobot swing-up, unshaped environment return, mean over five seeds. (a) The discovered energy leads unshaped PPO for the whole of the first

40

k steps. The two random configurations, of the same degree and matched in scale two different ways, each contain ten independent polynomial draws and five PPO seeds per draw; all remain behind the discovered potential. (b) Each faulty potential is plotted against the correct one so that the comparison is against seed-to-seed spread and not against the axis. Mass errors of

10\%

30\%

and

100\%

stay inside the band; only

300\%

leaves it, and it leaves it downwards for most of training.

Fig. 3: Measured family-wise false-alarm rate over

500

healthy-versus-healthy comparisons per setting, against a nominal

\alpha=0.01

, with Wilson

95\%

intervals. Both configurations are the same model differing only by seed, so every rejection is a false alarm by construction. Three intervals lie entirely to the right of the nominal line and all three are block-level and on Acrobot; every trajectory-level interval covers it or sits below.

Fig. 4: Separation between two shipped releases running identical model files, from a state both were handed identically. Colour is the environment and line style the release pair. The chaotic environment climbs fourteen decades at a constant rate from one unit in the last place, which is the signature of an amplified rounding difference and not of a changed model; the non-chaotic one, over the same release pairs and the same horizon, does not move. The lower Reacher pair is bitwise identical at every step and is drawn on the axis floor.

$\Delta t$	reference energy drift	LINK_MASS_2	LINK_LENGTH_1
$0.2$ (shipped)	$3.55\times 10^{-3}$	none	none
$0.05$	$4.19\times 10^{-6}$	$\leq 10^{-3}$	$\leq 10^{-3}$
$0.01$	$1.52\times 10^{-8}$	$\leq 10^{-3}$	$\leq 10^{-3}$

TABLE II: Release audit. The reference system’s energy drift sets the floor, and the smallest relative parameter error the screen localises tracks it. The first row is the shipped configuration. Both parameters have spec value

1.0

, and

\leq 10^{-3}

means every magnitude down to the bottom of the sweep was localised, and “none” means no swept magnitude was. At the shipped step size the energy’s own drift floor is a larger violation of conservation than a

30\%

parameter error, which is Remark 2 in the supplementary material stated as a measurement.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$n$	$d{=}1$	$d{=}2$	$d{=}3$	$d{=}4$
$4$	$5$	$15$	$35$	$70$
$6$	$7$	$28$	$84$	$210$
$8$	$9$	$45$	$165$	$495$
$11$	$12$	$78$	$364$	$1365$
$17$	$18$	$171$	$1140$	$5985$

Appendix

Table S1: Monomial dictionary size

M=\binom{n+d}{d}

. The support search scales with

M

, not with

n

; degree is as important as dimension. A degree-

1

(linear) invariant class such as stoichiometric conservation is tractable at any

n

encountered in practice.

$d$	multiples of $g$ present	projection removes	normal form removes
$2$	$1$	$1$	$1$
$3$	$5$	$1$	$5$
$4$	$15$	$1$	$15$

Appendix

Table S2: Deflating a single degree-

2

invariant

g

out of the degree-

\leq d

nullspace,

n=4

. Orthogonal projection removes one direction at every degree; normal-form reduction removes the entire filtered piece

\langle g\rangle\cap\mathcal{P}_{d}

. The two agree only at

d=\deg g=2

$\Delta t$	tol	raw $d_{\mathrm{null}}$	after projection	after normal form	residual	outcome
$0.2$	$10^{-6}$	$14$	$13$	$\mathbf{0}$	–	energy below tolerance
$0.2$	$10^{-4}$	$15$	$14$	$\mathbf{1}$	$6.2\times 10^{-1}$	survivor is not the energy
$0.05$	$10^{-4}$	$16$	$15$	$2$	–	spurious direction admitted
$0.01$	$10^{-6}$	$15$	$14$	$\mathbf{1}$	$1.6\times 10^{-7}$	energy isolated exactly
$0.002$	$10^{-6}$	$15$	$14$	$\mathbf{1}$	$4.9\times 10^{-10}$	energy isolated exactly

Appendix

Table S3: Nullspace deflation on Acrobot-v1, degree

\leq 3

n=6

M=84

monomials, single-energy passive trajectory, measured on sampled data and not on the exact ideal. The

14

trivial multiples vanish to machine precision at every step size, being exact algebra; the energy direction vanishes only to the integrator’s drift floor, so admitting it requires a nullspace tolerance above that floor. Sequential orthogonal projection of

g_{1}

then

g_{2}

removes one direction in total, not two, for the reason given in Section IV-A of the main paper. The last column is the residual of the single surviving direction outside

\mathrm{span}\{E,1\}

, and it is the column that matters: at

\Delta t=0.2

the count collapses to

1

just as it does at

\Delta t=0.002

, but the survivor is not the energy.

(a) Perturbed coefficients, $\sigma=0$
measured deviation	detected	localised	false alarms	$\langle G_{\epsilon}\rangle=\langle G\rangle$
$0$	$15/15$	$15/15$	$0/4$	yes
$\leq 9.8\times 10^{-5}$ (6 decades)	$14$ – $15/15$	$14$ – $15/15$	$0/120$	no
$9.8\times 10^{-4}$	$13$ – $14/15$	$13$ – $14/15$	$0/20$	no
$9.9\times 10^{-3}$	$9$ – $13/15$	$9$ – $13/15$	$0/20$	no
$1.0\times 10^{-1}$	$4$ – $8/15$	$3$ – $5/15$	$0/20$	no

Appendix

Table S4: An approximate reference set in place of the exact one, over the fifteen catalogue faults and two healthy controls. (a) The exact set with every coefficient perturbed, at

\sigma=0

; ranges are over five sign patterns and both regimes, and the last column is the exact ideal-equality decision on the same perturbed set. (b) The set a numerical recovery returns from healthy reference data, screened at the observation noise it was recovered at, against the exact set on the same data; forced removes the spectral-gap test so a generator is returned whatever the conditioning. Deviations are relative and gauge-fitted.

Environment	$n$	deg	Invariants
Cartesian pendulum (DAE)	$4$	$2$	$x^{2}{+}y^{2}{-}L^{2}$ ; $x\dot{x}{+}y\dot{y}$ ; energy (passive)
Acrobot-v1	$6$	$2$ – $3$	$c_{1}^{2}{+}s_{1}^{2}{-}1$ ; $c_{2}^{2}{+}s_{2}^{2}{-}1$ ; energy (passive) ; balance law
Reacher-v4	$\mathbf{11}$	$1$ – $2$	2 unit-norm; $d_{z}{=}0$ ; 2 forward-kinematics
Free rigid body / attitude	$7$	$2$	$\\|q\\|^{2}{-}1$ ; $\sum I_{i}\omega_{i}^{2}$ , $\sum I_{i}^{2}\omega_{i}^{2}$ (torque-free)
Planar Kepler	$4$	$2$	$x\dot{y}{-}y\dot{x}$ ; energy after lifting $r$
Reaction network	$8$ – $15$	$\mathbf{1}$	moiety conservation, integer coefficients

Appendix

Table S5: Tier-A environments. All invariants listed are exact and hold under arbitrary actuation unless marked (passive) . The two environments in bold are the ones every experiment in this paper runs on; the remaining four are stated to delimit the class the construction applies to and carry no results here.

fault	category	description	must break
acrobot_m2_1.3	parameter	second link mass $1.3$ not $1.0$	energy
acrobot_l1_1.2	parameter	first link length $1.2$ not $1.0$	energy
acrobot_lc2_0.6	parameter	second CoM at $0.6$ not $0.5$	energy
acrobot_I2_1.5	parameter	second inertia $1.5$ not $1.0$	energy
acrobot_g_9.81	parameter	gravity $9.81$ not $9.8$	energy
reacher_l2_0.13	parameter	$\ell_{2}=0.13$ not $0.11$	fk- $x$ , fk- $y$

Appendix

Table S6: The fifteen faults of Section VII-A of the main paper, plus two healthy controls. Every one is a mistake a real simulator or robot description has plausibly carried. The nips Coriolis entry is not hypothetical but a variant Gymnasium ships, so it is a structural fault with known ground truth that no one wrote for this evaluation.

setting	value
Acrobot trajectories	$8$ trajectories, $400$ steps, $\Delta t=0.005$
Reacher samples	$N=4000$ poses, lengths scaled by $\ell_{1}$
significance level $\alpha$	$0.01$ , Bonferroni corrected over $G_{\rm ref}$
screen blocks	$32$ contiguous, $4$ per trajectory
screen trajectory units	$8$ per configuration, exact permutation null over $\binom{16}{8}$ splits
screen residual floor $\rho_{\min}$	$10^{-8}$

Appendix

Table S7: Fault-catalogue protocol. Every setting is fixed across all methods and all faults. The baselines subsample because their cost is superlinear in the sample count, whereas the screen’s is not. The MMD permutation budget sets that test’s smallest attainable

p

1/10001

, two orders below

\alpha

, so the budget cannot be what decides its verdict; the kernel is built once and permuted by index, so that budget is affordable.

$\sigma$	lag $1$	lag $10$	lag $50$	lag $100$
$0$	$7$	$7$	$7$	$7$
$10^{-8}$	$6$	$6$	$\mathbf{7}$	$6$
$10^{-7}$	$2$	$\mathbf{6}$	$\mathbf{6}$	$\mathbf{6}$
$10^{-6}$	$2$	$2$	$\mathbf{4}$	$2$
$10^{-5}$	$2$	$2$	$2$	$2$
$10^{-4}$	$2$	$2$	$2$	$2$

Appendix

Table S8: Attribution under measurement noise: correct parameter named and within

5\%

, out of seven single-parameter faults. The lag is the difference stride of Section IV-B of the main paper. Reacher’s two faults are kinematic and use no difference dictionary, which is why the count floors at two, not zero.

noise	constant	MMD	KS	discriminator	screen
$\sigma=0$	Acrobot $m_{2}$	$0.3$	$0.3$	$0.3$	$\mathbf{\leq 10^{-4}}$
	Acrobot $g$	$0.3$	$0.3$	$0.3$	$\mathbf{\leq 10^{-4}}$
	Acrobot $l_{c2}$	$0.3$	$0.3$	$0.3$	$\mathbf{\leq 10^{-4}}$
	Reacher $\ell_{2}$	none	none	none	$\mathbf{\leq 10^{-4}}$
$\sigma=10^{-3}$	Acrobot $m_{2}$	$0.3$	$0.3$	$0.3$	$\mathbf{0.1}$
	Acrobot $g$	$0.3$	$0.3$	$0.3$	$\mathbf{0.03}$

Appendix

Table S9: Smallest relative parameter error still rejected at

\alpha=0.01

, swept over

\{10^{-4},3\times 10^{-4},\dots,0.3\}

. “none” means no swept magnitude was rejected. The screen is at the bottom of the sweep on all four constants without noise; under noise it separates by generator kind, staying near there on Reacher’s vanishing constraints and rising by two to three orders on Acrobot’s conserved one.

noise	$\alpha$	MMD	KS	discriminator	screen
$\sigma=0$	$10^{-12}$	$0.00^{*}$	$0.13$	$0.53$	$0.00^{*}$
	$10^{-6}$	$0.00^{*}$	$0.20$	$0.53$	$0.00^{*}$
	$10^{-4}$	$0.33$	$0.20$	$0.53$	$0.00^{*}$
	$10^{-3}$	$0.33$	$0.20$	$0.53$	$\mathbf{1.00}$
	$10^{-2}$	$0.40$	$0.40$	$0.53$	$\mathbf{1.00}$
	$0.05$	$0.53$	$0.40$	$0.53$	$\mathbf{1.00}$

Appendix

Table S10: True-positive rate over the fifteen catalogue faults as the significance level sweeps twelve decades, scored at the trajectory unit. An asterisk marks a level below the method’s

p

-value floor, where a rate of zero is arithmetic. The false-positive rate is

0.00

over the two healthy controls for every method and every level shown, which two controls bound only at

[0,0.66]

; the rate itself is measured in Fig. 3 of the main paper.

$\lambda$	val MSE	algebraic residual	horizon	$95\%$ interval
$10^{-3}$	$1.000$	$0.997$	$-0.1$	$[-0.5,+0.2]$
$10^{-2}$	$1.008$	$1.000$	$-1.3$	$[-2.5,-0.1]$
$10^{-1}$	$1.036$	$0.993$	$-2.1$	$[-6.0,+1.7]$
$1$	$1.574$	$0.834$	$-4.0$	$[-8.2,+0.5]$
$10$	$7.540$	$0.874$	$-20.0$	$[-24.4,-15.9]$
$100$	$64.16$	$0.787$	$-36.8$	$[-44.7,-29.0]$

Appendix

Table S11: Consistency weight against what it moves, paired within architecture, data budget and seed,

12

pairs per row against

\lambda=0

. Ratios are geometric means. The interval is a percentile bootstrap on the mean paired difference in divergence horizon.

condition	AUC	final return	spread	step	$\Delta$ vs. discovered	$p$
none (PPO)	$-147.8\pm 28.4$	$-84.4\pm 4.1$			$-56.5$ $[-81.6,-31.9]$
rand. matched range	$-195.3$ $[-443.6,-126.7]$	$-132.7$ $[-416.5,-85.7]$	$5.75$	$4.05$	$-104.0$ $[-352.3,-35.4]$
rand. matched step	$-162.2$ $[-214.0,-135.1]$	$-99.1$ $[-169.6,-83.5]$	$1.06$	$0.75$	$-70.9$ $[-122.7,-43.8]$
RND [ Burda et al., 2019 ]	$-118.1\pm 8.2$	$-85.8\pm 3.1$			$-26.7$ $[-33.7,-19.0]$	$0.22$
discovered	$\mathbf{-91.3\pm 2.4}$	$-84.5\pm 3.1$	$5.75$	$0.75$		$0.0079$
faulty, $m_{2}$ $+10\%$	$-90.8\pm 1.3$	$-83.7\pm 2.2$	$6.07$	$0.79$	$+0.5$ $[-1.5,+3.1]$	$0.0079$

Appendix

Table S12: Acrobot swing-up, PPO, five seeds,

120{,}000

steps, unshaped environment return, higher better. AUC is the mean return over a common step grid. Spread is the standard deviation of the deployed potential over states a random policy visits, and step the standard deviation of the

\gamma\Phi(s^{\prime})-\Phi(s)

it hands the agent there, against an environment reward of

-1

per step.

\Delta

is the AUC difference from the discovered potential with a percentile bootstrap interval. The

p

column flags detection against none only, uncorrected across conditions; five seeds against five bottom out at

0.0079

where ties do not force the asymptotic form. Random rows give the mean and range of ten draw means, not a pooled seed-level interval or

p

-value. RND runs at a single un-swept bonus coefficient. Every faulty potential is recovered from a faulty system and gauged entirely within it.

Fault Diagnosis

Fault diagnosis identifies, localizes, and explains failures in engineered and computational systems using sensor data, logs, models, and domain knowledge.

See all 149 papers

Sep 9, 2026

TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards

Reinforcement learning with verifiable rewards (RLVR) has advanced language-model reasoning in domains such as mathematics and code, where objective answers are inexpensive to check. Diagnostic reasoning over complex data lacks this advantage: establishing the true cause of an anomaly often requires costly expert investigation and may remain ambiguous after the fact. We ask whether this asymmetry of verification can instead be engineered. We sample an intervention, inject it into a controlled simulator, and generate the observations it would produce. The hidden intervention provides an oracle label and objective reward, while the agent must still investigate noisy, confounded, and distributed evidence. We instantiate this approach in TRACE, a digital-advertising diagnostic environment with 12 root causes and fine-grained segment attribution. Agents investigate each episode using Python and SQL and must identify both the root cause and, when applicable, the affected segment assignment. On a held-out 235-episode test set, the strongest prompted baseline, Claude Opus 5, reaches 0.686 FullAttr@1. Supervised fine-tuning raises Qwen3.5-35B-A3B from 0.159 to 0.637, and subsequent RL with synthesized rewards reaches 0.757, outperforming all evaluated prompted baselines, including frontier closed-source models and a prompted Qwen3.5-122B-A10B model. The resulting policy also uses substantially fewer tool calls than the prompted 35B base. These results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone. More broadly, simulation-based verification can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.

Rui Sun, Zhan Shi, Bing He

Independent Researchers

RL for Language Model Reasoning Causal Reasoning in Language Models Reinforcement Learning with Verifiable Rewards Root Cause Analysis

Ask PDF

May 25, 2026

Semigroup Consistency as a Diagnostic for Learned Physics Simulators

Learned physics simulators are often evaluated by one-step or short-horizon prediction error, but these metrics can miss failures in temporal composition and long-horizon rollout. For autonomous, state-complete systems, exact solution maps satisfy a semigroup law: direct evolution over

s+t

should agree with evolution over

s

followed by

t

. We propose normalized semigroup error as a post hoc, model-agnostic diagnostic comparing these direct and composed learned predictions. On one-dimensional heat and Burgers dynamics with time-conditioned ConvNet and FNO baselines, semigroup error is positively associated with rollout degradation, with trajectory-level Spearman correlation

ρ= 0.635

and

95%

[0.621, 0.649]

. Semigroup regularization has mixed effects, supporting semigroup consistency primarily as an evaluation diagnostic rather than a universally beneficial training objective.

Lennon J. Shikhman

Georgia Institute of Technology

Neural Surrogate Modeling PDE Operator Learning

Ask PDF

Oct 1, 2026

My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning

Agentic reinforcement learning (RL) has emerged as a powerful approach for training large language model agents on multi-step tasks, yet reliance on terminal outcome rewards creates two credit-assignment problems, particularly in long-horizon tasks. First, same-outcome rollout groups provide no learning signal from terminal rewards. Second, terminal rewards provide only trajectory-wide feedback, making it difficult to identify which decisions caused a failure. Recent work supplements terminal rewards with finer-grained information from trajectory analysis, such as natural-language reflections on intermediate decisions and errors. However, natural-language diagnoses are difficult to use directly for credit assignment: their error claims may be unreliable, and they do not quantify how much each error should affect learning. We propose Self-Diagnosis-guided Terminal Credit Redistribution (FAULT), which turns diagnosed errors into explicit step-level credit anchored by terminal outcomes. FAULT checks diagnostic evidence and learns relative error costs from task outcomes. During training, the policy and self-diagnoser co-evolve, while error costs are updated online from recent outcomes. On ALFWorld, FAULT recovers learning signals from same-outcome groups, reaching 95% signal coverage versus 41% for GRPO and 72% for GiGPO, while better localizing credit to specific error steps. Across two model scales, FAULT delivers strong. improvements on the long-horizon ALFWorld and WebShop tasks while remaining competitive on short-horizon Search-based QA.

Yihua Zhu, Qianying Liu, Weixu Qiao +10

Alibaba Group · Kyoto University · NII LLMC +6

Agentic RL Credit Assignment in RL Step-Level Credit Assignment in RL Credit Assignment Fine-Grained Credit Assignment RL for LLM Agents

Ask PDF

Abstract

Fault Diagnosis

Explore similar work

TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards

Semigroup Consistency as a Diagnostic for Learned Physics Simulators

My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning