MASIF: Implicit Failures for Process Observability in Multi-Agent Systems

Yanwei Huang; Arpit Narechania; Atanu Sinha

cs.AIOct 19, 2026

MASIF: Implicit Failures for Process Observability in Multi-Agent Systems

Authors: Yanwei Huang, Arpit Narechania, Atanu Sinha

Organizations: HKUST · Adobe Research

Abstract

Multi-agent systems (MAS) can complete a task despite local execution errors that are corrected, bypassed, or carried forward without individually determining its outcome. We call these implicit failures (IFs) and study them as signals for process observability beyond final task outcome. Across 1,828 traces from nine MAS-benchmark configurations, IFs occur in 54.3% of successful and 75.2% of failed traces. We develop taxonomies of error types and subsequent absorption mechanisms, supported by human and cross-model annotation audits. Counterfactual replay further probes how repairing an IF changes the final outcome. To examine the practical value of IFs, we evaluate IF-based best-of-N trace selection against existing process reward models, and introduce a novel RL algorithm, IF-GRPO, that uses IF costs to refine preferences among successful trajectories in GRPO. Experiments show that IF-GRPO exhibits some gains in accuracy and process metrics. Importantly, it exposes trade-offs between task performance, IF count and cost, and token-normalized IF density. Overall, our results show that IFs provide complementary process observability into how MAS reach their outcomes, including error recovery, propagation, and IF-based process differences that final task success alone does not expose.

Figures & tables

Figure 1: Example MAS traces for five absorption patterns. IFs are marked with crosses, with their resolution steps highlighted. Self-correction : WebWalker corrects its own navigation error to find the target. Self-workaround : The Assistant switches from flawed algebra to code to solve MathProxyAgent’s problem. Peer-repair : SimpleReviewer fixes a code error from SimpleTester. Orchestrator intervention : The Orchestrator redirects Assistant’s failed download to WebSurfer for scraping. Unresolved propagation : Browser’s wrong claim propagates downstream.

MAS	Benchmark	Underlying LLM	# Traces	# Success	# Failure	# IFs
Magentic-One	GAIA	GPT-4o	165	52	113	383
WebAgent (WebWalker)	WebWalkerQA	GPT-4o-mini	680	71	609	1090
MrlX	BrowseComp-Plus	Qwen3-8b	100	4	96	119
AG2 (MathChat)	OlympiadBench	GPT-4o	206	29	177	178
AG2 (MathChat)	GSMPlus	Claude-3.7-Sonnet & GPT-4o	223	44	179	214
AG2 (MathChat)	MMLU	GPT-4o	168	27	141	241

Table 1: Detailed profiles of the configurations in our study corpus.

Type	Description	Example	Share
Failed Tool Use	A tool call fails, is blocked, times out, cannot execute, or returns unusable output because of tool or environment issues.	FileSurfer attempts to read an MP3 audio file as text and receives a conversion exception.	24.5%
Wrong Action	The agent takes an action that is inappropriate for the current task state or unlikely to make progress.	WebSurfer clicks page text rather than a valid link or play button, leaving the browser on the same page.	23.0%
Unnecessary Action	The agent performs redundant, low-value, or repetitive actions that consume steps without advancing the task.	WebSurfer handles a cookie prompt instead of following the already identified target page.	16.4%
Hallucination	The agent fabricates information, tool behavior, source content, or intermediate evidence not supported by the trace.	The agent invents a ranked list item that is not present in the retrieved source.	15.3%
Incomplete Action	The agent attempts an action but omits required details, formatting, inputs, or follow-through.	FileSurfer repeatedly returns the first PDF page instead of searching or navigating to the relevant pages.	9.1%
Reasoning Error	The agent makes an incorrect inference, introduces or propagates incorrect information, repeats a known-bad strategy, misses relevant evidence, or terminates from insufficient reasoning.	The assistant swaps document statuses across several listed books, or the orchestrator identifies the wrong single from an album despite available evidence.	8.6%

Table 2: Taxonomy of IF Types and their share in the benchmark.

Absorption Mechanism	Success	Failure	Total	Share
Self-correction	34	585	619	20.3%
Trivial issue	77	1,037	1,114	36.6%
Orchestrator intervention	36	262	298	9.8%
Peer repair	4	38	42	1.4%
Self-workaround	23	412	435	14.3%
Unresolved propagation	51	485	536	17.6%

Table 3: Distribution of IF absorption mechanisms across the corpus, separated by trace outcome.

MAS	Benchmark	Traces	Unique Tasks	IFs	SPR (%)	RR (%)	$\boldsymbol{\Delta RR}$ (pp) [95% CI]
AG2 (MathChat)	MMLU	100	100	138	$100.0\to 99.0$	$1.0\to 42.0$	$+41.0\;[28.6,53.8]$
ChatDev	MMLU	100	100	176	$96.3\to 96.7$	$8.0\to 35.8$	$+27.8\;[15.8,39.9]$
MetaGPT	MMLU	100	90	145	$98.0\to 91.7$	$1.0\to 47.2$	$+46.2\;[33.0,60.0]$
AG2 (MathChat)	OlympiadBench	100	98	104	$82.0\to 78.0$	$28.0\to 51.0$	$+23.0\;[7.9,38.4]$
AG2 (MathChat)	GSMPlus	100	82	164	$89.3\to 90.5$	$42.1\to 49.4$	$+7.3\;[-3.0,17.7]$
MagenticOne	GAIA	100	81	206	$36.7\to 40.4$	$2.7\to 7.6$	$+4.9\;[-2.9,12.8]$

Table 4: Matched no-op and IF-repair replay across seven MAS–benchmark configurations. Arrows indicate no-op

\rightarrow

IF repair. SPR measures success preservation and RR measures failure rescue.

\Delta

RR is the additional rescue rate under IF repair (percentage points), with 95% paired task-cluster bootstrap confidence intervals (5,000 resamples; unadjusted for multiple comparisons). Each configuration contains 50 originally successful and 50 failed traces. IF repair increases failure rescue by 23.0–46.2 pp beyond no-op replay in four configurations, demonstrating the potential of targeted IF interventions to recover failed executions.

Configuration	N	Random	PRM	IF	PRM+IF
MagenticOne / GAIA / PRInTS	4	19.3	18.2	25.0	23.5
	8	19.8	21.2	24.2	23.5
	16	17.7	19.7	26.5	25.0
AG2 / GSMPlus / Qwen2.5-Math-PRM	4	80.1	79.7	81.3	81.3
	8	79.1	78.1	82.8	81.3
	16	78.8	78.1	82.8	78.1

Table 5: Best-of-

N

task accuracy (%) across 3 configurations (MAS, benchmark, PRM) and

N\in\{4,8,16\}

. IF-based selection outperforms the evaluated PRMs in all nine comparisons, while PRM-only selection generally performs close to random, highlighting the additional process-quality information captured by IFs.

MAS	Benchmark	Accuracy (%)	IF Cost / Task	# IF / Task	IF Cost / 1K Tokens	# IF / 1K Tokens
AG2 (MathChat)	MMLU-Pro	$54.98\to 57.06$	$1.42\to 1.51$	$1.14\to 1.41$	$4.37\to 2.39$	$3.49\to 2.22$
	MMLU-Pro	+2.08 pp	+6.33%	+23.84%	-45.38%	-36.49%
	GSMPlus	$71.75\to 72.21$	$0.49\to 0.34$	$0.49\to 0.30$	$2.33\to 1.54$	$2.31\to 1.33$
	GSMPlus	+0.46 pp	-30.83%	-39.66%	-33.92%	-42.37%
	OlympiadBench	$37.69\to 40.00$	$2.35\to 2.32$	$3.37\to 2.63$	$2.40\to 1.64$	$3.45\to 1.86$
	OlympiadBench	+2.31 pp	-0.98%	-21.91%	-31.71%	-46.18%

Table 6: GRPO versus IF-GRPO across seven MAS–benchmark configurations. Each configuration reports GRPO

\to

IF-GRPO (first row) and its change (second row). “# IF” means the number of IFs. Green / red indicate improvement/deterioration. Bold indicates Holm-adjusted

p<0.05

(two-sided paired permutation tests). IF-GRPO improves accuracy in six configurations, but per-task IF burden and token-normalized IF density can change in opposing directions, revealing process-level trade-offs invisible to task accuracy alone.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Error Type	$w_{type}$
Hallucination	1.30
Reasoning Error	1.20
Wrong Action	1.10
Misunderstanding	1.10
Failed Tool Use	1.00
Malformed Output	1.00

Appendix

Table 7: The heuristic values of

w_{type}

for each error type.

Absorption Pattern	$w_{abs}$
Trivial Issue	0.35
Self-correction	0.55
Self-workaround	0.65
Peer Repair	0.75
Orchestrator Intervention	0.80
Unresolved Propagation	1.40

Appendix

Table 8: The heuristic values of

w_{abs}

for each absorption pattern.

Weights	Accuracy	IF Cost / Task	# IF / Task	IF Cost / 1K Tokens	# IF / 1K Tokens
Original	72.21	0.34	0.30	1.54	1.33
Uniform absorption	72.42	0.36	0.45	0.84	1.04
Uniform type	71.96	0.57	0.95	1.07	1.79
Perturbed	73.29	0.46	0.60	1.04	1.37

Appendix

Table 9: Sensitivity to IF severity weights on AG2/GSMPlus. Uniform sets the corresponding weights to one. Perturbed changes the original weights by up to 10%. All metrics are measured on the same evaluation benchmark.

Multi-Agent Systems

Multi-Agent Systems studies how autonomous agents coordinate, compete, communicate, and learn in shared or interdependent tasks.

See all 651 papers

MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics
Feb 26, 2026Yutong Wang, Siyuan Xiong, Xuebo Liu +4Agent Failure Analysis Multi-Agent Systems
Harbin Institute of Technology, Shenzhen · Alibaba Group
Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search
Jul 31, 2026Hanxiao Lu, Tianyi ZhangMonte Carlo Tree Search Multi-Agent Systems
Purdue University
Conformal Agent Error Attribution
May 7, 2026Naihe Feng, Yi Sui, Shiyi Hou +2Agent Failure Analysis AI Agent Reliability
Dalhousie University · Halifax, NS, Canada · Layer 6 AI +1

Feb 26, 2026J/K move · Enter open · S save

MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics

Yutong Wang, Siyuan Xiong, Xuebo Liu, Wenkang Zhou+3

Harbin Institute of Technology, Shenzhen · Alibaba Group

Agent Failure Analysis Multi-Agent Systems Rubric-Based Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

While multi-agent systems (MAS) excel at complex reasoning, they are vulnerable to errors that intermediate agents introduce and downstream agents build upon. Auditing intermediate messages before they propagate requires an explicit standard, yet evaluation rubrics are typically authored by domain experts or written against a reference answer, neither of which is available for an unseen message at test time. We present MASRubric, a MAS information flow auditing framework with failure-distilled pitfall rubrics. Offline, trajectories on which the MAS has failed are automatically distilled into a reusable bank of pitfall criteria, each describing a recurrent error by its underlying misconception, the reasoning situations in which it arises, and the check that would expose it. Online, the criteria applicable to each intermediate message are retrieved from this off-the-shelf bank and checked one by one, and the resulting satisfaction rate decides whether the message is broadcast, returned to its author with diagnostic feedback for revision, or withheld. Empirical results demonstrate that MASRubric enhances MAS performance on both fixed and dynamic frameworks, achieving average accuracy gains of up to 2.83 points on math reasoning benchmarks and 1.74 points on code generation benchmarks. Further analysis shows that the retrieved criteria vary systematically with task types, and that the audit effort tracks task difficulty. Moreover, the bank transfers without re-mining to a system with a stronger backbone, which makes more adaptive and more efficient use of it. Our code and dataset are released at https://github.com/TonySY2/MASRubric.

Abstract

Multi-Agent Systems

Explore similar work

MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics

Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search

Conformal Agent Error Attribution