AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation

Congjing Zhang; Siyuan Xu; Yunfei Bai; Ke Ren; Xiaoning Qian; Shuai Huang

cs.MAOct 17, 2026

AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation

Authors: Congjing Zhang, Siyuan Xu, Yunfei Bai, Ke Ren, Xiaoning Qian, Shuai Huang

Organizations: Department of Industrial & Systems Engineering, University of Washington · Department of Electrical & Computer Engineering, Texas A&M University · Amazon

Abstract

Large language model (LLM) multi-agent systems (MASs) solve complex tasks through interacting agents, yet their reliability is difficult to analyze because failures depend on both individual agents and how information and errors propagate across the system. Existing approaches largely treat this as post-hoc failure attribution, but do not provide a common model across heterogeneous MAS architectures or support broader analysis of information influence and observability. We introduce AgentSoV, a virtual-sensing framework that models MAS execution as a stochastic dynamic system through Stream of Variation (SoV). AgentSoV maps heterogeneous trajectories into diagnostic stages, constructs virtual measurements from observable outputs, and learns cross-stage variation propagation with a state-space model. Its representation is architecture-aware in stage dependencies yet architecture-agnostic in analytical language, supporting chain-structured, orchestration-based, and directed acyclic graph (DAG)-structured MASs. Without re-executing the MAS, AgentSoV supports three system-level analyses: failure attribution identifies where abnormal variation enters; stage-input influence quantifies how strongly intermediate inputs affect downstream outcomes; and virtual-sensor placement selects complementary observations under a monitoring budget. Experiments across multiple MAS architectures demonstrate a unified approach to fault localization, information influence, and observability. AgentSoV reframes agentic execution as a dynamic system to model rather than merely text to inspect, providing insights for MAS structure and component design.

Figures & tables

Figure 1: Schematic overview of AgentSoV. The upper panel shows the observed LLM MAS execution trajectory $\mathcal{T}$ . AgentSoV maps $\mathcal{T}$ into diagnostic stages $\mathcal{S}$ , constructs stage inputs ${\bm{u}}$ and virtual measurements ${\bm{y}}$ , and models cross-stage variation through SoV dynamics. The diagnostic mapping need not preserve the original execution one-to-one: multiple execution steps $\tau$ may be grouped into a stage $s$ according to their diagnostic role. The resulting representation supports three system-level analyses through different model quantities: disturbances ${\bm{z}}$ for failure attribution, stage inputs ${\bm{u}}$ for stage-input influence, and virtual measurements ${\bm{y}}$ for virtual-sensor placement.

Dataset	Method	Step Accuracy	Top-3 Step Accuracy	MRR	Agent Accuracy
BFCL	Uniform Random	$11.7\%$	$35.2\%$	$31.7\%$	$11.7\%$
	GPT-5.5 (All-at-Once)	$16.5\%\pm 1.2\%$	$58.3\%\pm 1.2\%$	$39.4\%\pm 1.4\%$	$16.5\%\pm 1.2\%$
	GPT-5.5 (Step-by-Step)	$9.7\%\pm 0.9\%$	$49.8\%\pm 2.8\%$	$24.9\%\pm 1.1\%$	$9.7\%\pm 0.9\%$
	GPT-5.5 (Binary Search)	$2.9\%\pm 1.1\%$	$48.2\%\pm 3.0\%$	$23.7\%\pm 1.4\%$	$2.9\%\pm 1.1\%$
	MASC	$5.2\%\pm 1.0\%$	$33.8\%\pm 1.4\%$	$29.5\%\pm 0.4\%$	$5.2\%\pm 1.0\%$
	OAT	$8.4\%\pm 0.5\%$	$29.5\%\pm 1.3\%$	$27.9\%\pm 0.3\%$	$8.4\%\pm 0.5\%$

Table 1: Failure attribution performance on BFCL and Magentic-One.

Method	Exact-Set Accuracy	Top-1 Accuracy	Top-3 Accuracy	MRR
Uniform Random	$9.0\%$	$44.9\%$	$91.7\%$	$67.6\%$
GPT-5.5 (All-at-Once)	$16.5\%\pm 1.1\%$	$84.2\%\pm 0.8\%$	$99.1\%\pm 0.3\%$	$91.5\%\pm 0.3\%$
GPT-5.5 (Step-by-Step)	$8.3\%\pm 2.2\%$	$67.9\%\pm 0.5\%$	$99.8\%\pm 0.3\%$	$83.0\%\pm 0.5\%$
GPT-5.5 (Binary Search)	$12.2\%\pm 2.6\%$	$86.8\%\pm 0.6\%$	$99.6\%\pm 0.3\%$	$92.9\%\pm 0.4\%$
MASC	$0.1\%\pm 0.3\%$	$76.7\%\pm 2.6\%$	$95.5\%\pm 0.0\%$	$86.53\%\pm 1.4\%$
OAT	$4.9\%\pm 1.1\%$	$41.7\%\pm 1.0\%$	$87.5\%\pm 1.2\%$	$62.5\%\pm 0.6\%$

Table 2: Failure attribution performance on AEGIS.

Method	$>0\%$	$\geq 10\%$	$\geq 20\%$
Uniform Random	$65.5\%$	$45.1\%$	$24.8\%$
MASC	$67.1\%$	$67.1\%$	$60.8\%$
OAT	$48.1\%$	$21.5\%$	$8.9\%$
AgentSoV	$\mathbf{74.7\%}$	$\mathbf{68.4\%}$	$\mathbf{63.3\%}$

Table 3: Recall of repaired AEGIS executions under different anomaly-reduction thresholds.

Method	Training Time (s)	Inference Time (s)	LLM Tokens (In/Out)
GPT-5.5 (All-at-Once)	0	5.3427	618/440
GPT-5.5 (Step-by-Step)	0	16.4219	2,586/1,026
GPT-5.5 (Binary Search)	0	2.6835	1,221/27
MASC	23.86	0.0054	5,102/0
OAT	7.82	0.3282	4,937/0
AgentSoV-S	10.92	0.0001	–

Table 4: Computational costs of BFCL.

Panel of Figure 2 — Figure 2: Stage-input influence on Magentic-One.

Panel of Figure 3 — Figure 3: Virtual-sensor placement and ${\bm{y}}$ -dimension selection under different observation budgets: (a & c) BFCL and (b & d) Magentic-One (MO).

Panel of Figure 4 — Figure 4: Agent-masking effects on failure-attribution accuracy relative to full observation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Step Accuracy	Top-3 Step Accuracy	MRR	Agent Accuracy
BFCL	AgentSoV-S	$\mathbf{37.7\%\pm 1.3\%}$	$\mathbf{65.2\%\pm 2.7\%}$	$\mathbf{56.1\%\pm 1.4\%}$	$\mathbf{37.7\%\pm 1.3\%}$
BFCL	AgentSoV	$37.5\%\pm 1.0\%$	$63.1\%\pm 1.7\%$	$54.6\%\pm 0.5\%$	$37.5\%\pm 1.0\%$
Magentic-One	AgentSoV-S	$32.2\%\pm 1.9\%$	$58.5\%\pm 1.5\%$	$46.4\%\pm 1.0\%$	$54.8\%\pm 5.1\%$
Magentic-One	AgentSoV	$\mathbf{37.8\%\pm 2.1\%}$	$\mathbf{61.5\%\pm 1.8\%}$	$\mathbf{51.0\%\pm 1.5\%}$	$\mathbf{62.6\%\pm 0.8\%}$

Appendix

Table 5: Failure attribution on BFCL and Magentic-One using a shared transition matrix

{\bm{F}}

across all stages, i.e.,

{\bm{F}}_{k}={\bm{F}}

for all stage

k

Method	Exact-Set Accuracy	Top-1 Accuracy	Top-3 Accuracy	MRR
AgentSoV-S	$38.5\%\pm 0.0\%$	$77.6\%\pm 0.3\%$	$98.2\%\pm 0.3\%$	$87.4\%\pm 0.3\%$
AgentSoV	$\mathbf{46.2\%\pm 1.3\%}$	$\mathbf{91.7\%\pm 0.5\%}$	$100.0\%\pm 0.0\%$	$95.4\%\pm 0.3\%$

Appendix

Table 6: Failure attribution on AEGIS using a shared transition matrix

{\bm{F}}

across all stages, i.e.,

{\bm{F}}_{k}={\bm{F}}

for all stage

k

Method	Accuracy	$\Delta$ vs. Failure-Specific ${\bm{H}}$
AgentSoV ( ${\bm{F}}{=}\mathbf{0}$ , ${\bm{H}}{=}{\bm{I}}$ )	$26.2\%\pm 1.4\%$	$-1.4\%$
AgentSoV ( ${\bm{H}}{=}{\bm{I}}$ )	$35.3\%\pm 2.0\%$	$-2.2\%$

Appendix

Table 7: Failure attribution accuracy on BFCL when replacing the failure-aware disturbance matrix

{\bm{H}}

with

{\bm{H}}{=}{\bm{I}}

\Delta

denotes the resulting change in accuracy.

Panel of Figure 5 — Figure 5: Stage-input influence on BFCL.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,001 papers

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
May 17, 2026Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2Agent Failure Analysis Multi-Agent LLM Systems
Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
Jul 8, 2026Yufei Xia, Anjun Gao, Yueyang Quan +2Agent Failure Analysis Multi-Agent LLM Systems
University of Louisville · University of North Texas
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Failure Attribution for LLM Agents Agent Failure Analysis
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

May 17, 2026J/K move · Enter open · S save

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

Hezhe Qiao, Hanghang Tong, Ee-Peng Lim, Bing Liu+1

Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago

Agent Failure Analysis Multi-Agent LLM Systems Multi-Agent Systems LLM Agent Reliability

Open paper AskPDF arXiv

Large language model-driven multi-agent systems (LLM-MAS) excel at complex tasks, yet unreliable agents remain a key bottleneck to system-level reliability. Automatic failure attribution is therefore critical, but existing approaches, such as direct prediction of agent-error pairs and agent-first failure attribution, rely on local logs of agents and miss global failures that only manifest over full interaction trajectories, such as cross-step inconsistencies and inter-agent coordination errors. Moreover, directly predicting failures induces a large combinatorial search space, hindering fine-grained attribution. To address these challenges, we propose VerifyMAS, a hypothesis verification framework for agent failure attribution. Instead of directly predicting faulty agents and error types, VerifyMAS formulates and verifies failure hypotheses against full trajectories. This verification-based approach decomposes attribution into trajectory-level error validation and fine-grained agent localization, providing an error-first attribution approach that captures global failure patterns while substantially reducing the search space. We further introduce a hypothesis-based data construction strategy grounded in a structured error taxonomy and fine-tune a specialized LLM verifier model for trajectory-level failure verification and agent attribution. Experiments on Aegis-Bench and Who&When show that VerifyMAS consistently improves diverse backbone models, including open-source Qwen and API-based GPT models, outperforming prior methods without sacrificing inference efficiency for long multi-agent trajectories.

Abstract

Multi-Agent LLM Systems

Explore similar work

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems