Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems

Boxuan Zhang; jianing Zhu; Xiaotian Han; Ruixiang Tang

cs.LGOct 12, 2026

Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems

Authors: Boxuan Zhang, jianing Zhu, Xiaotian Han, Ruixiang Tang

Organizations: Rutgers University · The University of Texas at Austin

Abstract

Reliable deployment of LLM-based multi-agent systems calls for runtime auditors that predict terminal outcomes from observed execution prefixes. These auditors typically learn from a limited corpus of trajectories with verified outcomes. Expanding this corpus requires costly execution and verification, yet broader failure coverage alone provides no explicit criterion for identifying prefixes that the current auditor finds difficult to classify. To this end, we propose Dream2Discern, a runtime auditor trained through implicit augmentation guided by its own prediction loss. During Dream, we select perturbations of the auditor's representation operators to increase prediction loss on recorded prefixes, yielding hard latent variants. During Discern, the auditor learns to predict execution outcomes from these variants using the original labels. The variants are regenerated as the auditor learns, turning a limited corpus into a renewable source of hard examples and requiring no additional trajectory collection or outcome verification. Extensive evaluations on AFTraj-2K and StepShield show that Dream2Discern outperforms competitive failure auditors and achieves over 90% trajectory-level weighted accuracy on both benchmarks while balancing prediction accuracy and timeliness. Project page: https://zbox1005.github.io/dream2discern-project/

Figures & tables

Figure 1: Motivation of Dream2Discern . (a) A runtime auditor predicts terminal outcomes from observed prefixes but learns from a limited labeled corpus. (b) Trajectory augmentation seeks broader failure coverage, but collection is costly and editing can invalidate outcome labels (right). Broader coverage alone does not identify prefixes that the current auditor finds hard to classify. (c) We construct hard latent variants through operator perturbations chosen to increase batch prediction loss, then train on these variants while keeping recorded prefixes and labels unchanged.

Panel of Figure 2 — Figure 2: (a) Original and edited prefixes overlap in a shared PCA projection. (b) Training with edits does not reduce held-out class-balanced BCE. A temporary Dream perturbation selected on training prefixes raises this loss with the Original head fixed.

Figure 3: Overview of Dream2Discern . ① Labeled prefixes . We form a class-balanced batch from recorded success and failure prefixes, keeping their inputs and outcome labels unchanged throughout the training steps. ② Dream . With the auditor fixed, we search for shared, bounded, low-rank perturbations to its query and value projection weights that increase batch prediction loss. ③ Discern . Holding the selected perturbation fixed, we update the LoRA adapters and prediction head to minimize the loss on the resulting hard latent variants. ④ Runtime auditing . We discard the perturbation and apply a calibrated threshold to each prefix score to predict failure during execution.

		AFTraj- $2$ K				StepShield
	Method	AUROC $\uparrow$	FPR95 $\downarrow$	wACC $\uparrow$	TWA $\uparrow$	AUROC $\uparrow$	FPR95 $\downarrow$	wACC $\uparrow$	TWA $\uparrow$
Open.	Qwen3-4B	$69.63$	$86.39$	$55.12$	$51.65$	$81.07$	$82.41$	$50.46$	$49.12$
	Qwen3-8B	$67.18$	$80.47$	$56.93$	$52.83$	$84.95$	$88.89$	$62.96$	$53.08$
	Qwen3-32B	$69.79$	$82.25$	$56.33$	$52.70$	$90.70$	$70.37$	$58.80$	$52.31$
	Llama3.2-3B-it	$59.60$	$90.53$	$54.22$	$51.21$	$78.88$	$86.11$	$50.93$	$50.28$
	Llama3.1-8B-it	$58.02$	$94.08$	$53.01$	$51.49$	$83.49$	$76.85$	$50.00$	$49.56$

Table 1: Main comparison on AFTraj-

2

K ( Zhang et al., 2026 ) and StepShield ( Felicia et al., 2026 ) . The best and second-best results in each column are shown in bold and underline .

Table 5

Figure 6: Case study of runtime failure prediction on a held-out coding trajectory from AFTraj-2K .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	AFTraj-2K	StepShield
Available training trajectories	1,944	1,278
Trajectories in the training index	1,821	1,253
Training-index rows	8,454	7,828
Test trajectories	332	216
Test trajectories with $Y_{i}=1$	163	108
Test trajectories with $Y_{i}=0$	169	108

Appendix

Table 4: Dataset statistics. Training-index rows count prefixes; test counts refer to trajectories.

Setting	Value
Optimizer	AdamW
Adapter and head learning rates	$10^{-4}$
Weight decay	$0$
Gradient clipping norm	$1$
Logical batch size	32 prefixes, 16 per outcome class
Microbatch size / accumulation steps	$8$ / $4$

Appendix

Table 5: Training settings for Dream2Discern .

Auditor backbone	$\gamma$
Qwen3-4B	$3\times 10^{-4}$
Qwen3-8B	$3\times 10^{-3}$
Llama-3.2-3B-Instruct	$3\times 10^{-3}$
Llama-3.1-8B-Instruct	$3\times 10^{-4}$

Appendix

Table 6: Backbone-specific perturbation radii. Each value is used on both benchmarks.

		AFTraj-2K			StepShield
Backbone	Readout	AUROC $\uparrow$	wACC $\uparrow$	TWA $\uparrow$	AUROC $\uparrow$	wACC $\uparrow$	TWA $\uparrow$
Qwen3-4B	Last token (default)	$95.31$	$90.36$	$72.28$	$92.91$	$90.28$	$57.48$
Qwen3-4B	Mean pooling	$91.44$	$89.76$	$69.56$	$87.27$	$79.17$	$47.61$
Qwen3-8B	Last token (default)	$96.21$	$91.57$	$73.70$	$94.59$	$88.43$	$56.42$
Qwen3-8B	Mean pooling	$92.95$	$92.17$	$74.56$	$91.98$	$87.04$	$57.06$

Appendix

Table 7: Prefix-readout comparison with training seed 0. All metrics are percentages.

Backbone	$\gamma$	AFTraj-2K		StepShield
Backbone	$\gamma$	wACC $\uparrow$	TWA $\uparrow$	wACC $\uparrow$	TWA $\uparrow$
Qwen3-4B	$10^{-4}$	$91.27$	$72.17$	$87.50$	$55.57$
	$3\times 10^{-4}$ $\dagger$	$90.36$	$72.28$	$90.28$	$57.48$
	$10^{-3}$	$91.27$	$73.58$	$86.11$	$53.72$
	$3\times 10^{-3}$	$91.87$	$73.45$	$80.56$	$54.42$
	$10^{-2}$	$89.76$	$67.66$	$78.70$	$55.82$

Appendix

Table 8: Perturbation-radius sensitivity with training seed 0. Metrics are percentages;

\dagger

marks the radius used in the main experiments.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 261 papers

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems
May 9, 2026Boxuan Zhang, Jianing Zhu, Zeru Shi +2Multi-Agent LLM Systems AI Agent Monitoring
Rutgers University · The University of Texas at Austin · Purdue University
Cascade Failure Attribution for Multi-Agent Systems
Aug 30, 2026Jiayi Zhang, Zexin Wang, Degang Sun +4Agent Failure Analysis Multi-Agent LLM Systems
Computer Network Information Center, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Automata from Agent Traces: Failure and Next-Step Prediction
Aug 24, 2026Seonglae Cho, Franklin Cardenoso Fernandez, Umar Mohammed +4AI Agent Monitoring AI Agent Auditing
Holistic AI · PUC-Rio · University College London

May 9, 2026J/K move · Enter open · S save

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu+1

Rutgers University · The University of Texas at Austin · Purdue University

Multi-Agent LLM Systems AI Agent Monitoring

Open paper AskPDF arXiv

LLM-based multi-agent systems are increasingly deployed on long-horizon tasks, but a single decisive error is often accepted by downstream agents and cascades into trajectory-level failure. Existing work frames this as post-hoc failure attribution, diagnosing the responsible agent and step after the trajectory has ended. However, this paradigm forfeits any opportunity to intervene while trajectory is still unfolding. In this work, we introduce AgentForesight, a framework that reframes this problem as online auditing: at each step of an unfolding trajectory, an auditor observes only the current prefix and must either continue the run or alarm at the earliest decisive error, without access to future steps. To this end, we curate AFTraj-2K, a corpus of agentic trajectories across Coding, Math, and Agentic domains, in which safe trajectories are retained under a strict curation pipeline and unsafe trajectories are annotated at the step of their decisive error via consensus among multiple LLM judges. Built on that, we develop AgentForesight-7B, a compact online auditor trained with a coarse-to-fine reinforcement learning recipe that first equips it with a risk-anticipation prior at the failure boundary on adjacent safe/unsafe prefix pairs, then sharpens this prior into precise step-level localization under a three-axis reward jointly targeting the what, where, and who of an audit verdict. Across AFTraj-2K and an external Who&When benchmark, AgentForesight-7B outperforms leading proprietary models, including GPT-4.1 and DeepSeek-V4-Pro, achieving up to +19.9% performance gain and 3

\times

lower step localization error, opening the loop from post-hoc failures detection to enabling deployment-time intervention. Project page: https://zbox1005.github.io/agent-foresight/

Abstract

AI Agent Auditing

Explore similar work

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

Cascade Failure Attribution for Multi-Agent Systems

Automata from Agent Traces: Failure and Next-Step Prediction