TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution

Tommaso Bendinelli; Artur Dox; Christian Holz

cs.LGAug 27, 2026

TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution

Authors: Tommaso Bendinelli, Artur Dox, Christian Holz

Organizations: Department of Computer Science ETH Zürich, Switzerland · CSEM SA, Switzerland · Independent Researcher

Abstract

LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however, their performance on these tasks has not been systematically evaluated under controlled conditions. We introduce TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks. In each generated task, an agent receives time-series observations produced by simulating a physical dynamical system and must determine whether a system parameter was altered during the simulation and, if so, which one. Using TraceBench, we generate tasks from three interpretable mechanical systems and systematically evaluate four LLM agents across controlled experimental conditions, yielding new insights into how these agents analyze time-series observations from dynamical systems. Our results show that agents benefit substantially from domain context and explore data primarily through numerical console output rather than visualizations. We also find that agents generally perform worse when required to produce a Python script that maps each time-series sample to a predicted root-cause label than when they submit predictions directly. We release our datasets, agent trajectories, experimental results, and a leaderboard on our website, tracebench.github.io.

Figures & tables

Figure 1. Overview of the TraceBench framework using a bouncing-ball simulator. In this example, the black and orange segments show the position observation under the low-noise setting before and after an intervention at $t_{\mathrm{int}}=4.0$ . The intervention increases gravitational acceleration, and the dashed line marks the intervention time. At test time, the agent determines from the observed time-series channels whether an intervention occurred and, if so, which parameter changed. \name{} pipeline: a physical simulator produces multivariate time series before and after a parameter intervention; the low-noise BallDrop position observation is black before and orange after a dashed gravity-intervention marker; the agent identifies the changed root-cause parameter.

Figure 2. The four controllable axes in TraceBench. Four cards summarize the independently controlled \name{} task axes: domain context, observation noise, labeled examples, and direct or programmatic submission.

Figure 3. Representative noise-free signals from one observed channel for each simulator. Black and orange segments show the trajectory before and after the intervention time, respectively, which is marked by a dashed vertical line. The panels show position for BallDrop and BounceBall , and velocity for MassSlide . The interventions increase the coefficient of restitution in BallDrop and the inclination angle in BounceBall , and decrease the plane inclination in MassSlide . Additional examples with observation noise and all observed channels are provided in Appendix A . Three horizontally arranged time-series plots showing position for BallDrop and BounceBall and velocity for MassSlide. Each plot marks the intervention time with a dashed vertical line.

	Low Noise			High Noise
	$\mathcal{S}_{\mathrm{direct}}$	$\mathcal{S}_{\mathrm{prog}}$		$\mathcal{S}_{\mathrm{direct}}$	$\mathcal{S}_{\mathrm{prog}}$
	Episode	Episode	Held-out	Episode	Episode	Held-out
	Accuracy
gpt-5.5	$0.933\pm 0.056$	$0.853\pm 0.143$	$0.671\pm 0.096$	$0.833\pm 0.194$	$0.733\pm 0.227$	$0.547\pm 0.138$
gemini-3.1-pro	$0.760\pm 0.115$	$0.733\pm 0.153$	$0.604\pm 0.128$	$0.560\pm 0.221$	$0.547\pm 0.169$	$0.440\pm 0.085$
claude-opus-4.6	$0.853\pm 0.115$	$0.687\pm 0.122$	$0.614\pm 0.084$	$0.593\pm 0.170$	$0.593\pm 0.190$	$0.499\pm 0.129$

Table 1. Mean accuracy and exploration metrics per episode for the main experiment, aggregated across simulators. Accuracy values are reported as the mean

\pm

within-simulator standard deviation; random-chance accuracy is 0.189. Python calls indicate the total number of Python interpreter invocations, while Python statements count statements across generated Python source versions. Plot counts report the number of generated

\mid

inspected plots. Console-output entries report the number of numeric values printed

\mid

the estimated percentage of tokens attributable to console output. See Appendix G for computation details.

Figure 4. Accuracy as a function of token usage, evaluation cost, episode time, and tool calls. Each point aggregates all episodes for one agent-system configuration across the four main-experiment conditions, three simulators, and five repetition seeds. Four scatter plots compare model accuracy with token usage, evaluation cost, episode time, and tool calls. Each point represents one agent-system configuration aggregated over all episodes, with error bars showing population standard deviations.

	$\mathcal{S}_{\mathrm{direct}}$	$\mathcal{S}_{\mathrm{prog}}$
	Episode	Episode	Held-out
	Accuracy
gpt-5.5 (No labeled examples)	$0.887\pm 0.125$ ( +5.3 )	$0.847\pm 0.116$ ( +11.3 )	$0.506\pm 0.070$ ( -4.1 )
gemini-3.1-pro (No labeled examples)	$0.647\pm 0.159$ ( +8.7 )	$0.653\pm 0.176$ ( +10.7 )	$0.444\pm 0.113$ ( +0.4 )
gpt-5.5 (No domain context)	$0.513\pm 0.202$ ( -32.0 )	$0.453\pm 0.209$ ( -28.0 )	$0.316\pm 0.092$ ( -23.2 )
gemini-3.1-pro (No domain context)	$0.313\pm 0.135$ ( -24.7 )	$0.347\pm 0.154$ ( -20.0 )	$0.328\pm 0.043$ ( -11.3 )

Table 2. Accuracy and token usage for the two high-noise ablation conditions, aggregated across simulators. Entries report the mean

\pm

within-simulator standard deviation, followed in parentheses by the signed change relative to the corresponding non-ablated condition with three labeled examples per class and domain context. Accuracy changes are in percentage points; token values and changes are in millions.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Simulation End	10 s
Sampling Frequency	400 Hz
Number of Rows	3999

Appendix

Table 3. BallDrop simulation settings used by the released trajectories.

Applies to	Rule
All interventions	The first apex after the first detected divergence must exceed $0.1\,\mathrm{m}$ in at least one of the intervention and baseline traces, and each trace must contain fewer than 25 qualifying rebounds.
Qualifying rebound	The ground-contact signal must exceed $1$ , the incoming speed must exceed $0.5\,\mathrm{m\,s^{-1}}$ , and the rebound speed must exceed $0.1\,\mathrm{m\,s^{-1}}$ ; events separated by less than $0.03\,\mathrm{s}$ are merged.
Drag, mass, and restitution	A qualifying rebound must occur both before and after the first detected divergence.
Drag	The normalized-DTW velocity distance over the following $0.2\,\mathrm{s}$ must exceed $0.1$ , or the terminal-velocity rule must hold.
Terminal-velocity rule	Accept a near-terminal descent after a rebound or one that ends above $0.1\,\mathrm{m}$ before the trace ends.

Appendix

Table 4. Simulator-specific rejection rules for BallDrop.

variable	min	median	max
initial_height	0.19122	3.0483	5.769
initial_velocity	-9.7276	-4.8861	1.1469
mass	0.5	3.2	19.434
drag_coeff	0	0.5	6
restitution	0.377	0.93	0.99
gravity	2	5	10

Appendix

Table 5. Empirical BallDrop retained-pool variable summaries.

Coefficient	$\kappa_{\mathrm{ref}}$
force_base_sigma_scale	0.002
force_event_sigma_scale	0.002
force_hetero_sigma_scale	0.002
position_base_sigma_scale	0.002
position_drift_sigma_scale	0.002
position_event_sigma_scale	0.002

Appendix

Table 6. Exact BallDrop observation-noise coefficients.

Panel of Figure 5 — Figure 5. Position, velocity, and ground-contact impulse (Ns) channels from BallDrop under (a) low and (b) high observation noise. Two side-by-side sets of three aligned BallDrop time-series panels showing position, velocity, and ground-contact impulse (Ns) before and after a gravity intervention. The left set uses low observation noise and the right set uses high observation noise.

Channel	Low Noise	High Noise
Impulse	$21.07\pm 2.55$	$9.03\pm 2.55$
Position	$42.41\pm 3.19$	$30.58\pm 3.14$
Velocity	$36.58\pm 2.60$	$24.54\pm 2.60$

Appendix

Table 7. BallDrop released-trajectory SNR in dB as the mean and sample standard deviation.

Simulation End	60 s
Sampling Frequency	50 Hz
Number of Rows	2999

Appendix

Table 8. BounceBall simulation settings used by the released trajectories.

Applies to	Rule
All interventions	Each usable intervention and baseline trace must contain fewer than 20 impacts. Impacts are counted from velocity reversals whose incoming speed exceeds $0.1\,\mathrm{m\,s^{-1}}$ ; when velocity reversals cannot be computed, samples with absolute right-contact force above $10^{-9}\,\mathrm{N}$ are used.
Mass and viscous damping	In the intervention trace, a qualifying velocity reversal must occur both before and after the intervention time, and right-contact force must exceed $500\,\mathrm{N}$ on both sides of that time.

Appendix

Table 9. Simulator-specific rejection rules for BounceBall.

variable	min	median	max
initial_position	3	6.2	9.3
initial_velocity	-17.01	-8.2	10
mass	0.29498	15.435	99.287
viscous_damping	0.001	0.055	1.32
restitution_right	0.44813	0.74813	0.9999
restitution_left	0.25	0.79684	0.9999

Appendix

Table 10. Empirical BounceBall retained-pool variable summaries.

Coefficient	$\kappa_{\mathrm{ref}}$
position_base_sigma_scale	0.0175
position_drift_sigma_scale	0.004
position_event_sigma_scale	0.04
position_quant_step_floor	0.0005
position_quant_step_scale	0.0025
right_force_base_sigma_scale	0.0001

Appendix

Table 11. Exact BounceBall observation-noise coefficients.

Panel of Figure 6 — Figure 6. Position, velocity, and right-wall force channels from BounceBall under (a) low and (b) high observation noise. Two side-by-side sets of three aligned BounceBall time-series panels showing position, velocity, and right-wall force before and after an inclination-angle intervention. The left set uses low observation noise and the right set uses high observation noise.

Channel	Low Noise	High Noise
Position	$30.02\pm 1.91$	$17.97\pm 1.91$
Velocity	$40.98\pm 2.46$	$28.94\pm 2.46$
Force	$39.09\pm 2.45$	$27.05\pm 2.45$

Appendix

Table 12. BounceBall released-trajectory SNR in dB as the mean and sample standard deviation.

Simulation End	10 s
Sampling Frequency	50 Hz
Number of Rows	499

Appendix

Table 13. MassSlide simulation settings used by the released trajectories.

Applies to	Rule
Breakaway friction	At least one common non-time channel must have symmetric relative difference $2\|y-y_{0}\|/(\|y\|+\|y_{0}\|+\epsilon_{\mathrm{SRD}})>0.2$ for five consecutive samples.
Gravity	At least one of the intervention and baseline traces must contain five finite, nonzero velocity samples before and five after the reference change time, taken as the intervention time or, if unavailable, the earliest detected divergence.
All remaining classes	No additional simulator-specific rule is applied.

Appendix

Table 14. Simulator-specific rejection rules for MassSlide.

variable	min	median	max
initial_velocity	-5	1	12
Amplitude	-352.94	-27.652	360
Period	0.68753	1.7104	2.8184
mass	0.5	2.2361	10
gravity_constant	4	6	18.432
plane_inclination	5	20.945	65.938

Appendix

Table 15. Empirical MassSlide retained-pool variable summaries.

Coefficient	$\kappa_{\mathrm{ref}}$
friction_base_sigma_scale	0.001
friction_bias_sigma_scale	0.002
friction_drift_sigma_scale	0.001
friction_event_sigma_scale	0.02
normal_base_sigma_scale	0.001
normal_drift_sigma_scale	0.0002

Appendix

Table 16. Exact MassSlide observation-noise coefficients.

Panel of Figure 7 — Figure 7. Velocity, friction-force, and normal-force channels from MassSlide under (a) low and (b) high observation noise. Two side-by-side sets of three aligned MassSlide time-series panels showing velocity, friction force, and normal force before and after a Coulomb-friction-coefficient intervention. The left set uses low observation noise and the right set uses high observation noise.

Channel	Low Noise	High Noise
friction_force	$42.03\pm 9.75$	$29.99\pm 9.75$
mass_velocity	$35.64\pm 6.58$	$23.60\pm 6.58$
normal_force	$59.90\pm 0.40$	$47.86\pm 0.40$

Appendix

Table 17. MassSlide released-trajectory SNR in dB as the mean and sample standard deviation.

Platform	Pinned npm package
codex	@openai/[email protected]
gemini-cli	@google/[email protected]
claude-code	@anthropic-ai/[email protected]
opencode	[email protected]

Appendix

Table 18. Available platforms and their installation requirements.

Agent	Platform	Reasoning Effort	Img.
claude-opus-4.6	claude-code	high	Y
gemini-3.1-pro	gemini-cli	high	Y
gpt-5.5	codex	high	Y
minimax-m2.7	opencode	–	N

Appendix

Table 19. Agent profile catalog.

Package	Version
contourpy	1.3.3
cycler	0.12.1
fonttools	4.61.1
iniconfig	2.3.0
joblib	1.5.3
kiwisolver	1.4.9

Appendix

Table 20. Python packages installed in the agent runtime.

Agent	Input	Cached	Output
claude-opus-4.6	5	0.5	25
gemini-3.1-pro	2	0.2	12
gpt-5.5	5	0.5	30
minimax-m2.7	0.3	0.059	1.20

Appendix

Table 21. Agent prices used to compute evaluation costs, in USD per 1M tokens.

Figure 8. Episode top-1 accuracy versus mean evaluation cost for BallDrop under the four main-experiment conditions. Scatter plot of episode top-1 accuracy versus evaluation cost for BallDrop across models and experimental conditions.

Figure 9. Episode top-1 accuracy versus mean evaluation cost for BounceBall under the four main-experiment conditions. Scatter plot of episode top-1 accuracy versus evaluation cost for BounceBall across models and experimental conditions.

Figure 10. Episode top-1 accuracy versus mean evaluation cost for MassSlide under the four main-experiment conditions. Scatter plot of episode top-1 accuracy versus evaluation cost for MassSlide across models and experimental conditions.

Figure 11. BallDrop confusion matrices for $\mathcal{S}_{\mathrm{direct}}$ , reported separately for every agent under $\kappa_{\mathrm{low}}$ and $\kappa_{\mathrm{high}}$ .

Figure 12. BounceBall confusion matrices for $\mathcal{S}_{\mathrm{direct}}$ , reported separately for every agent under $\kappa_{\mathrm{low}}$ and $\kappa_{\mathrm{high}}$ . Eight BounceBall confusion matrices arranged in four agent rows and two noise-level columns. Rows within each matrix are ground-truth labels, columns are predicted labels, cells contain raw counts, and color encodes the row-normalized fraction.

Figure 13. MassSlide confusion matrices for $\mathcal{S}_{\mathrm{direct}}$ , reported separately for every agent under $\kappa_{\mathrm{low}}$ and $\kappa_{\mathrm{high}}$ . Eight MassSlide confusion matrices arranged in four agent rows and two noise-level columns. Rows within each matrix are ground-truth labels, columns are predicted labels, cells contain raw counts, and color encodes the row-normalized fraction.

Contrast	Effect [95% CI]	W/T/L
Agent comparisons (low/high noise averaged)
gpt-5.5 - gemini-3.1-pro, $\mathcal{S}_{\mathrm{direct}}$ , episode	22.3 [10.7, 34.0]	12/0/3
gpt-5.5 - gemini-3.1-pro, $\mathcal{S}_{\mathrm{prog}}$ , episode	15.3 [6.3, 24.3]	11/1/3
gpt-5.5 - gemini-3.1-pro, $\mathcal{S}_{\mathrm{prog}}$ , held-out	8.7 [1.9, 15.5]	12/0/3
gpt-5.5 - claude-opus-4.6, $\mathcal{S}_{\mathrm{direct}}$ , episode	16.0 [6.7, 25.0]	12/0/3
gpt-5.5 - claude-opus-4.6, $\mathcal{S}_{\mathrm{prog}}$ , episode	15.3 [6.9, 24.0]	13/1/1

Appendix

Table 23. Complete paired episode-level differences, reported in percentage points with confidence intervals obtained by exhaustive sign-flip inversion. For agent comparisons, differences are calculated within each simulator–seed pair and then averaged across the low- and high-noise conditions. For condition comparisons, differences are averaged across the three closed-weight agents; minimax-m2.7 is excluded. W/T/L (win/tie/loss) is determined from these paired differences.

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$1.000\pm 0.000$	$0.840\pm 0.089$	$0.960\pm 0.055$
gemini-3.1-pro	$0.680\pm 0.179$	$0.920\pm 0.084$	$0.680\pm 0.084$
claude-opus-4.6	$0.820\pm 0.130$	$0.920\pm 0.130$	$0.820\pm 0.110$
minimax-m2.7	$0.160\pm 0.114$	$0.180\pm 0.164$	$0.540\pm 0.152$

Appendix

Table 24. Per- simulator results for the main experiment under

\kappa_{\mathrm{low}}

and submission mode

\mathcal{S}_{\mathrm{direct}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.820\pm 0.239$	$0.860\pm 0.089$	$0.880\pm 0.084$
gemini-3.1-pro	$0.720\pm 0.217$	$0.720\pm 0.164$	$0.760\pm 0.089$
claude-opus-4.6	$0.620\pm 0.164$	$0.680\pm 0.148$	$0.760\pm 0.055$
minimax-m2.7	$0.180\pm 0.130$	$0.180\pm 0.084$	$0.480\pm 0.179$
Held-out Accuracy

Appendix

Table 25. Per- simulator results for the main experiment under

\kappa_{\mathrm{low}}

and submission mode

\mathcal{S}_{\mathrm{prog}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.840\pm 0.261$	$0.780\pm 0.179$	$0.880\pm 0.179$
gemini-3.1-pro	$0.440\pm 0.230$	$0.500\pm 0.324$	$0.740\pm 0.114$
claude-opus-4.6	$0.500\pm 0.187$	$0.440\pm 0.219$	$0.840\pm 0.134$
minimax-m2.7	$0.200\pm 0.158$	$0.200\pm 0.122$	$0.500\pm 0.235$

Appendix

Table 26. Per- simulator results for the main experiment under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{direct}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.760\pm 0.207$	$0.640\pm 0.321$	$0.800\pm 0.187$
gemini-3.1-pro	$0.460\pm 0.152$	$0.500\pm 0.255$	$0.680\pm 0.110$
claude-opus-4.6	$0.520\pm 0.130$	$0.440\pm 0.305$	$0.820\pm 0.130$
minimax-m2.7	$0.340\pm 0.167$	$0.200\pm 0.071$	$0.520\pm 0.249$
Held-out Accuracy

Appendix

Table 27. Per- simulator results for the main experiment under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{prog}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.900\pm 0.122$	$0.820\pm 0.148$	$0.940\pm 0.134$
gemini-3.1-pro	$0.760\pm 0.134$	$0.440\pm 0.167$	$0.740\pm 0.207$

Appendix

Table 28. Per- simulator results for the ablation without labeled examples under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{direct}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.880\pm 0.045$	$0.720\pm 0.164$	$0.940\pm 0.134$
gemini-3.1-pro	$0.640\pm 0.114$	$0.680\pm 0.228$	$0.640\pm 0.207$
Held-out Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.629\pm 0.042$	$0.411\pm 0.083$	$0.478\pm 0.092$

Appendix

Table 29. Per- simulator results for the ablation without labeled examples under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{prog}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.600\pm 0.235$	$0.400\pm 0.187$	$0.540\pm 0.230$
gemini-3.1-pro	$0.220\pm 0.045$	$0.200\pm 0.071$	$0.520\pm 0.239$
claude-opus-4.6	-	-	-
minimax-m2.7	-	-	-

Appendix

Table 30. Per- simulator results for the ablation without domain context under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{direct}}

Episode Accuracy
	BallDrop	BounceBall	MassSlide
gpt-5.5	$0.340\pm 0.230$	$0.440\pm 0.241$	$0.580\pm 0.205$
gemini-3.1-pro	$0.240\pm 0.114$	$0.300\pm 0.212$	$0.500\pm 0.158$
claude-opus-4.6	-	-	-
minimax-m2.7	-	-	-
Held-out Accuracy

Appendix

Table 31. Per- simulator results for the ablation without domain context under

\kappa_{\mathrm{high}}

and submission mode

\mathcal{S}_{\mathrm{prog}}

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Aug 30, 2026

Cascade Failure Attribution for Multi-Agent Systems

Large language model (LLM)-based agents have shown strong potential in solving complex tasks through multi-step reasoning, yet they remain vulnerable to execution failures. Accurate failure attribution is therefore critical for improving agent reliability. Existing topology- and spectrum-based methods exploit trajectory structures but often overlook fine-grained semantics, while LLM-based attribution methods capture semantic cues but suffer from long-context degradation over lengthy trajectories. To address these challenges, we propose DuoTrace, a plug-and-play detection filter for LLM-based failure attribution. DuoTrace follows a detect-before-attribute paradigm: it first detects anomalous executions and then supplies focused trajectory evidence to downstream LLM-based attribution methods. For effective VAE-based anomaly detection on agent trajectories, DuoTrace integrates dual-view semantic-structural node representations, a Tree-LSTM-based trajectory encoder, and data augmentation through outcome-verified LLM replay and root-to-node path extraction to handle heterogeneous nodes, hierarchical execution structures, and limited normal execution data. Experiments with six LLM-based attribution baselines show that DuoTrace improves agent-level and step-level attribution accuracy by 11.2% and 7.7%,respectively.

Jiayi Zhang, Zexin Wang, Degang Sun +4

Computer Network Information Center, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences

Agent Failure Analysis Multi-Agent LLM Systems Failure Attribution for LLM Agents LLM Agent Reliability

Ask PDF

Jun 25, 2026

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

Aoyang Fang, Yifan Yang, Jin'ao Shang +7

The Chinese University of Hong Kong, Shenzhen · Lero · The Chinese University of Hong Kong

LLM Agent Evaluation Causal Attribution AI Agent Benchmarks Causal Reasoning in Language Models Process Supervision Root Cause Analysis

Ask PDF

Aug 24, 2026

Automata from Agent Traces: Failure and Next-Step Prediction

LLM-based agents execute multi-step tasks, but their behavioral structure remains opaque: long unstructured traces resist the safety auditing and runtime monitoring that deployment requires. Existing approaches operate per-trace or success-only, so they miss the cross-run topology that links next-step and failure prediction. To recover that shared structure, we collapse an entire trace corpus into a single, compact finite-state machine (FSM) that serves as a structural substrate for the otherwise unpredictable behavior of LLM agents. Across twelve public datasets, the FSMs are compact (7-43 states), replay held-out data at >=0.997 fitness with near-identical topology across splits, and build in milliseconds. This substrate addresses both prediction goals. For next-step prediction, FSM-state context outperforms Agent Workflow Memory on every ground-truth-matched dataset. For failure prediction, per-state behavioral features reach held-out AUROC up to 0.94, and an online monitor ranks failing runs above passing ones from a partial trace, triggering early stopping well before completion. Behavioral topology thus appears shaped more by the deployment harness than by the LLM, providing a model-agnostic structural primitive for safety auditing and runtime monitoring.

Seonglae Cho, Franklin Cardenoso Fernandez, Umar Mohammed +4

Holistic AI · PUC-Rio · University College London

AI Agent Monitoring AI Agent Auditing LLM Agent Reliability Next-Action Prediction Finite-State Machine

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

Cascade Failure Attribution for Multi-Agent Systems

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Automata from Agent Traces: Failure and Next-Step Prediction