AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation

Congjing Zhang; Siyuan Xu; Yunfei Bai; Ke Ren; Xiaoning Qian; Shuai Huang

cs.MAOct 17, 2026

AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation

Authors: Congjing Zhang, Siyuan Xu, Yunfei Bai, Ke Ren, Xiaoning Qian, Shuai Huang

Organizations: Department of Industrial & Systems Engineering, University of Washington · Department of Electrical & Computer Engineering, Texas A&M University · Amazon

Abstract

Large language model (LLM) multi-agent systems (MASs) solve complex tasks through interacting agents, yet their reliability is difficult to analyze because failures depend on both individual agents and how information and errors propagate across the system. Existing approaches largely treat this as post-hoc failure attribution, but do not provide a common model across heterogeneous MAS architectures or support broader analysis of information influence and observability. We introduce AgentSoV, a virtual-sensing framework that models MAS execution as a stochastic dynamic system through Stream of Variation (SoV). AgentSoV maps heterogeneous trajectories into diagnostic stages, constructs virtual measurements from observable outputs, and learns cross-stage variation propagation with a state-space model. Its representation is architecture-aware in stage dependencies yet architecture-agnostic in analytical language, supporting chain-structured, orchestration-based, and directed acyclic graph (DAG)-structured MASs. Without re-executing the MAS, AgentSoV supports three system-level analyses: failure attribution identifies where abnormal variation enters; stage-input influence quantifies how strongly intermediate inputs affect downstream outcomes; and virtual-sensor placement selects complementary observations under a monitoring budget. Experiments across multiple MAS architectures demonstrate a unified approach to fault localization, information influence, and observability. AgentSoV reframes agentic execution as a dynamic system to model rather than merely text to inspect, providing insights for MAS structure and component design.

Figures & tables

Figure 1: Schematic overview of AgentSoV. The upper panel shows the observed LLM MAS execution trajectory $\mathcal{T}$ . AgentSoV maps $\mathcal{T}$ into diagnostic stages $\mathcal{S}$ , constructs stage inputs ${\bm{u}}$ and virtual measurements ${\bm{y}}$ , and models cross-stage variation through SoV dynamics. The diagnostic mapping need not preserve the original execution one-to-one: multiple execution steps $\tau$ may be grouped into a stage $s$ according to their diagnostic role. The resulting representation supports three system-level analyses through different model quantities: disturbances ${\bm{z}}$ for failure attribution, stage inputs ${\bm{u}}$ for stage-input influence, and virtual measurements ${\bm{y}}$ for virtual-sensor placement.

Dataset	Method	Step Accuracy	Top-3 Step Accuracy	MRR	Agent Accuracy
BFCL	Uniform Random	$11.7\%$	$35.2\%$	$31.7\%$	$11.7\%$
	GPT-5.5 (All-at-Once)	$16.5\%\pm 1.2\%$	$58.3\%\pm 1.2\%$	$39.4\%\pm 1.4\%$	$16.5\%\pm 1.2\%$
	GPT-5.5 (Step-by-Step)	$9.7\%\pm 0.9\%$	$49.8\%\pm 2.8\%$	$24.9\%\pm 1.1\%$	$9.7\%\pm 0.9\%$
	GPT-5.5 (Binary Search)	$2.9\%\pm 1.1\%$	$48.2\%\pm 3.0\%$	$23.7\%\pm 1.4\%$	$2.9\%\pm 1.1\%$
	MASC	$5.2\%\pm 1.0\%$	$33.8\%\pm 1.4\%$	$29.5\%\pm 0.4\%$	$5.2\%\pm 1.0\%$
	OAT	$8.4\%\pm 0.5\%$	$29.5\%\pm 1.3\%$	$27.9\%\pm 0.3\%$	$8.4\%\pm 0.5\%$

Table 1: Failure attribution performance on BFCL and Magentic-One.

Method	Exact-Set Accuracy	Top-1 Accuracy	Top-3 Accuracy	MRR
Uniform Random	$9.0\%$	$44.9\%$	$91.7\%$	$67.6\%$
GPT-5.5 (All-at-Once)	$16.5\%\pm 1.1\%$	$84.2\%\pm 0.8\%$	$99.1\%\pm 0.3\%$	$91.5\%\pm 0.3\%$
GPT-5.5 (Step-by-Step)	$8.3\%\pm 2.2\%$	$67.9\%\pm 0.5\%$	$99.8\%\pm 0.3\%$	$83.0\%\pm 0.5\%$
GPT-5.5 (Binary Search)	$12.2\%\pm 2.6\%$	$86.8\%\pm 0.6\%$	$99.6\%\pm 0.3\%$	$92.9\%\pm 0.4\%$
MASC	$0.1\%\pm 0.3\%$	$76.7\%\pm 2.6\%$	$95.5\%\pm 0.0\%$	$86.53\%\pm 1.4\%$
OAT	$4.9\%\pm 1.1\%$	$41.7\%\pm 1.0\%$	$87.5\%\pm 1.2\%$	$62.5\%\pm 0.6\%$

Table 2: Failure attribution performance on AEGIS.

Method	$>0\%$	$\geq 10\%$	$\geq 20\%$
Uniform Random	$65.5\%$	$45.1\%$	$24.8\%$
MASC	$67.1\%$	$67.1\%$	$60.8\%$
OAT	$48.1\%$	$21.5\%$	$8.9\%$
AgentSoV	$\mathbf{74.7\%}$	$\mathbf{68.4\%}$	$\mathbf{63.3\%}$

Table 3: Recall of repaired AEGIS executions under different anomaly-reduction thresholds.

Method	Training Time (s)	Inference Time (s)	LLM Tokens (In/Out)
GPT-5.5 (All-at-Once)	0	5.3427	618/440
GPT-5.5 (Step-by-Step)	0	16.4219	2,586/1,026
GPT-5.5 (Binary Search)	0	2.6835	1,221/27
MASC	23.86	0.0054	5,102/0
OAT	7.82	0.3282	4,937/0
AgentSoV-S	10.92	0.0001	–

Table 4: Computational costs of BFCL.

Panel of Figure 2 — Figure 2: Stage-input influence on Magentic-One.

Panel of Figure 3 — Figure 3: Virtual-sensor placement and ${\bm{y}}$ -dimension selection under different observation budgets: (a & c) BFCL and (b & d) Magentic-One (MO).

Panel of Figure 4 — Figure 4: Agent-masking effects on failure-attribution accuracy relative to full observation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Step Accuracy	Top-3 Step Accuracy	MRR	Agent Accuracy
BFCL	AgentSoV-S	$\mathbf{37.7\%\pm 1.3\%}$	$\mathbf{65.2\%\pm 2.7\%}$	$\mathbf{56.1\%\pm 1.4\%}$	$\mathbf{37.7\%\pm 1.3\%}$
BFCL	AgentSoV	$37.5\%\pm 1.0\%$	$63.1\%\pm 1.7\%$	$54.6\%\pm 0.5\%$	$37.5\%\pm 1.0\%$
Magentic-One	AgentSoV-S	$32.2\%\pm 1.9\%$	$58.5\%\pm 1.5\%$	$46.4\%\pm 1.0\%$	$54.8\%\pm 5.1\%$
Magentic-One	AgentSoV	$\mathbf{37.8\%\pm 2.1\%}$	$\mathbf{61.5\%\pm 1.8\%}$	$\mathbf{51.0\%\pm 1.5\%}$	$\mathbf{62.6\%\pm 0.8\%}$

Appendix

Table 5: Failure attribution on BFCL and Magentic-One using a shared transition matrix

{\bm{F}}

across all stages, i.e.,

{\bm{F}}_{k}={\bm{F}}

for all stage

k

Method	Exact-Set Accuracy	Top-1 Accuracy	Top-3 Accuracy	MRR
AgentSoV-S	$38.5\%\pm 0.0\%$	$77.6\%\pm 0.3\%$	$98.2\%\pm 0.3\%$	$87.4\%\pm 0.3\%$
AgentSoV	$\mathbf{46.2\%\pm 1.3\%}$	$\mathbf{91.7\%\pm 0.5\%}$	$100.0\%\pm 0.0\%$	$95.4\%\pm 0.3\%$

Appendix

Table 6: Failure attribution on AEGIS using a shared transition matrix

{\bm{F}}

across all stages, i.e.,

{\bm{F}}_{k}={\bm{F}}

for all stage

k

Method	Accuracy	$\Delta$ vs. Failure-Specific ${\bm{H}}$
AgentSoV ( ${\bm{F}}{=}\mathbf{0}$ , ${\bm{H}}{=}{\bm{I}}$ )	$26.2\%\pm 1.4\%$	$-1.4\%$
AgentSoV ( ${\bm{H}}{=}{\bm{I}}$ )	$35.3\%\pm 2.0\%$	$-2.2\%$

Appendix

Table 7: Failure attribution accuracy on BFCL when replacing the failure-aware disturbance matrix

{\bm{H}}

with

{\bm{H}}{=}{\bm{I}}

\Delta

denotes the resulting change in accuracy.

Panel of Figure 5 — Figure 5: Stage-input influence on BFCL.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,001 papers

May 17, 2026

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

Large language model-driven multi-agent systems (LLM-MAS) excel at complex tasks, yet unreliable agents remain a key bottleneck to system-level reliability. Automatic failure attribution is therefore critical, but existing approaches, such as direct prediction of agent-error pairs and agent-first failure attribution, rely on local logs of agents and miss global failures that only manifest over full interaction trajectories, such as cross-step inconsistencies and inter-agent coordination errors. Moreover, directly predicting failures induces a large combinatorial search space, hindering fine-grained attribution. To address these challenges, we propose VerifyMAS, a hypothesis verification framework for agent failure attribution. Instead of directly predicting faulty agents and error types, VerifyMAS formulates and verifies failure hypotheses against full trajectories. This verification-based approach decomposes attribution into trajectory-level error validation and fine-grained agent localization, providing an error-first attribution approach that captures global failure patterns while substantially reducing the search space. We further introduce a hypothesis-based data construction strategy grounded in a structured error taxonomy and fine-tune a specialized LLM verifier model for trajectory-level failure verification and agent attribution. Experiments on Aegis-Bench and Who&When show that VerifyMAS consistently improves diverse backbone models, including open-source Qwen and API-based GPT models, outperforming prior methods without sacrificing inference efficiency for long multi-agent trajectories.

Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2

Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago

Agent Failure Analysis Multi-Agent LLM Systems Multi-Agent Systems LLM Agent Reliability

Ask PDF

Jul 8, 2026

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Yufei Xia, Anjun Gao, Yueyang Quan +2

University of Louisville · University of North Texas

Agent Failure Analysis Multi-Agent LLM Systems

Ask PDF

Oct 8, 2026

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose Error-Propagation Modeling for Failure Attribution (EMFA). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5

College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Failure Attribution for LLM Agents Agent Failure Analysis Multi-Agent LLM Systems Multi-Agent Systems

Ask PDF

Abstract

Multi-Agent LLM Systems

Explore similar work

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems