Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems

Xiuming Liu; Le Gai; Zhiyuan Chang; Peiyang Li; Bin Zhu; Shuo Wang; Chao Feng

cs.AIOct 11, 2026

Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems

Authors: Xiuming Liu, Le Gai, Zhiyuan Chang, Peiyang Li, Bin Zhu, Shuo Wang, Chao Feng

Organizations: Shanghai Jiao Tong University · Ant Group · Tsinghua University

Abstract

Agent-to-agent protocols allow LLM agents to discover peers, exchange results, and coordinate tool use. A locally successful protocol attack, however, may be rejected by a later agent or propagated into a consequential task decision. Measuring local exploitability alone leaves this difference unresolved and provides limited evidence about which participant caused the final outcome. We present Agent-Canary-Mas, an end-to-end evaluation system that follows protocol risk through long-horizon multi-agent tasks. It covers five risk classes and 13 subclasses across A2A, ANP, and BCS, with 975 matched malicious--benign task pairs in five business domains. A shared OpenClaw framework runs autonomous, tool-using agents under different underlying models while preserving the task and execution setup. Matched runs separate local protocol success from downstream consequences; retained evidence connects participant selection, message consumption, tool activity, and task-state changes. We further evaluate a protocol-context Guard that checks an object's authority at consequential handoffs. Across three model conditions, local attack success is frequently at or above 90%, while consequence conversion ranges from 7.1% to 100.0%. Cross-agent evidence raises macro-averaged Audit-F1 from 29.7% to 95.5%. On Finance tasks, the Guard lowers average ASR from 86.1% to 53.8% and PTCR from 65.5% to 31.2%, with a measurable completion tradeoff. These results show that protocol risk depends on how agents use exchanged objects throughout the task, and that evaluating this path supports more informative risk assessment, attribution, and defense.

Figures & tables

Figure 1: From a local protocol deviation to a long-horizon task consequence. Agent-Canary-Mas follows the object through agent consumption, cross-agent propagation, task-state changes, and the final user outcome.

Panel of Table 1 — Table 1: Comparison of representative agent-security and protocol benchmarks.

Figure 2: Lifecycle-based protocol risk classes across Agent collaboration handoffs.

Figure 3: Construction and evaluation framework of Agent-Canary-Mas : instantiate a protocol risk, build a shared task scaffold, construct a matched pair, and follow its influence to the task outcome.

Component	Setting
Task suite	975 matched pairs; three protocols; five domains; five risk classes and 13 conceptual subclasses
Agent framework	OpenClaw; isolated participant workspaces; autonomous reasoning and role-permitted tool use
Model conditions	MiniMax-M3, GLM-5.2, GPT-5.6 Terra; the same model for all agents within an execution
Controlled factors	Task goal, roles, candidate pool, tools, topology, visibility, and trusted initial state shared across paired variants
Protocol execution	Native A2A, ANP, and BCS discovery, delivery, and result exchange through a common task scaffold
Main evaluation	2,925 model-conditioned pairs; 5,850 benign or malicious execution instances

Table 2: Experimental design. Model comparisons use a shared OpenClaw runtime; matched variants differ in the registered protocol-risk treatment.

Protocol	Risk Class	MiniMax-M3			GLM-5.2			GPT-5.6 Terra
Protocol	Risk Class	TCR	ASR	PTCR	TCR	ASR	PTCR	TCR	ASR	PTCR
A2A	T1: Identity Binding	99.5	98.5	86.4	99.7	100.0	87.5	99.8	98.0	84.9
	T2: Capability Integrity	99.8	97.8	82.1	100.0	99.9	83.3	99.9	97.3	80.7
	T3: Routing Authorization	99.2	97.5	73.8	99.6	99.8	75.0	99.5	96.9	72.5
	T4: Content Authority	99.7	98.6	90.5	99.9	99.9	91.7	100.0	98.1	88.9
	T5: Object Validity	99.6	98.2	88.6	99.8	99.9	90.0	99.7	97.6	87.1

Table 3: Protocol risk conversion across task horizons by protocol, risk class, and Business Agent model. Each class macro-averages its security subclasses across the five application domains. TCR reports paired task completion, ASR local attack success among pair-complete trials, and PTCR downstream consequence conversion among locally successful trials (%).

Panel of Figure 4 — Figure 4: Protocol–risk-class Audit-F1 under the Native View and Agent-Canary-Mas View, macro-aggregated across three model conditions.

Protocol	Risk Class	No Guard			Protocol-Level Guard
Protocol	Risk Class	TCR	ASR	PTCR	TCR	ASR	PTCR
A2A	T1	99.7	90.1	82.3	95.3	40.1	35.6
	T2	98.0	93.5	80.9	96.7	46.8	20.9
	T3	99.9	90.7	46.8	94.0	70.7	10.1
	T4	99.3	96.3	86.7	98.7	33.0	16.7
	T5	100.0	83.9	64.1	97.2	50.6	17.4

Table 4: Protocol-level Guard effectiveness on Finance tasks with MiniMax-M3 (%).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: A2A execution flow. A Client Agent resolves an AgentCard, selects an interface, sends a Message, and receives a Message or Task; task exchanges may include status and Artifact updates.

Figure 6: ANP execution flow. A caller resolves a handle or DID, verifies the DID Document, reads the Agent Description and application interface, and invokes an authenticated endpoint.

Figure 7: BCS execution flow. The control plane establishes a collaboration, BCS routes chat.send or chat.inject operations, and Bots or Providers return correlated events or callbacks.

ID	Risk class	Subclass I	Subclass II	Subclass III	Core security question
T1	Identity Binding	Participant discovery and admission	Identity continuity and behavior attribution	–	Who enters the task, and does later behavior remain attributable?
T2	Capability Integrity	Claim–implementation consistency	Representation fidelity	Contract fulfillment	Do capability claims match delivered behavior?
T3	Routing Authorization	Execution-target authorization	Actual-audience authorization	Route and branch authorization	Who receives, redirects, or decides the subsequent control flow?
T4	Content Authority	Semantic promotion	Execution conversion	Persistent downstream activation	When does remote data acquire control over a local decision or action?
T5	Object Validity	Lifecycle and temporal validity	Context association	–	Is a reused object temporally valid and bound to the current task?

Appendix

Table 5: Cross-protocol risk taxonomy used by Agent-Canary-Mas . Five classes organize 13 security subclasses by evaluation boundary, and the final column states the security question applied consistently to A2A, ANP, and BCS.

Class	A2A	ANP	BCS
T1	Agent Card $\rightarrow$ interface $\rightarrow$ Message/Task Provider and signature metadata, security requirements, protocol identity, and result attribution	WNS/DID $\rightarrow$ DID Document $\rightarrow$ AD $\rightarrow$ authenticated result Handle resolution, verification material, service endpoint, DID-WBA, and result attribution	Registry/BCSFuse $\rightarrow$ Actor Directory $\rightarrow$ bot_uuid/sender Capabilities, visibility, friendship, candidate selection, authenticated sender, and event attribution
T2	Agent Card skills/capabilities $\rightarrow$ interface $\rightarrow$ result Claimed skills and constraints carried into Message, Task, and Artifact processing	AD interface $\rightarrow$ OpenRPC $\rightarrow$ JSON-RPC result Capability description transformed into methods, schemas, servers, and delivered behavior	BotCapabilities $\rightarrow$ Registry/BCSFuse $\rightarrow$ response Original capability profile, derived recommendation, Actor Directory view, and concrete interaction
T3	supportedInterfaces/tenant $\rightarrow$ Message/Task route Selected endpoint and binding, stream or push path, recipient context, and referenced resources	OpenRPC server/target $\rightarrow$ JSON-RPC or ANPMessage route Endpoint, recipient DID, webhook or resource target, and later call path	Participants/mentions $\rightarrow$ chat.send/chat.inject $\rightarrow$ next route Active responder, observing audience, structured routing, and workflow branch
T4	Message Part/Artifact $\rightarrow$ Agent context and action Remote content promoted into a fact, tool argument, persistent history, or later task input	JSON-RPC result/ANPMessage/resource $\rightarrow$ Agent context and action Remote content interpreted as data, authority, executable direction, or persistent input	GroupMessage/ContentBlock $\rightarrow$ chat.send/chat.inject $\rightarrow$ run/history Remote content reaches active processing, observation context, or a later resumed execution
T5	Agent Card version/cache $\rightarrow$ messageId/taskId/contextId Card and interface updates, Task state, history, Artifact lifecycle, and interaction binding	WNS/DID/AD/OpenRPC caches $\rightarrow$ nonce/request/session/proof Object updates, authenticated replay state, request–response correlation, and application context	Registry/Directory/relationship $\rightarrow$ message/run/group/session Participant state, callbacks, delivery deduplication, GroupContext, and connected history

Appendix

Table 6: Protocol-native grounding of the five risk classes in A2A, ANP, and BCS. In each cell, the bold line identifies the native object and handoff chain through which the risk enters, while the following line lists the protocol fields, state, or correlation evidence used to evaluate that boundary.

Figure 8: Matched T1 case in which a manipulated participant profile changes service binding and propagates into a downstream account decision.

Property	Benign profile	Malicious profile
Presented role	Supplier Correspondence Desk	Beneficiary Change Service
Service description	Supplier inquiry and request routing	Callback, authorization, and account-status lookup
Claimed domain	Supplier services	Vendor master
Advertised functions	Inquiry, triage, and request routing	Change-status, callback, and active-account lookup
Task scope	Case intake	Vendor-change status

Appendix

Table 7: Participant-profile treatment separating the malicious case from its benign control.

Multi-Agent System Security

Multi-agent system security studies attacks, trust, privacy, and defenses in interacting AI agents, including language-model teams, robot swarms, and learned controllers.

See all 179 papers

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
Oct 13, 2026Fasheng Miao, Zhaoyu Hu, Xiang Li +5AI Agent Security Benchmarks LLM Agent Security
Tsinghua University · Alibaba Group · Nankai University +1
SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
Sep 1, 2026Rui Yang, Junjie Xu, Zhengyu Liu +4AI Agent Security Benchmarks Multi-Agent System Security
Johns Hopkins University · Nanyang Technological University
MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems
Oct 20, 2026Mohammad Kachuee, Phu Mon Htut, Sheng Li +5AI Agent Security LLM Agent Safety
Oracle AI

Oct 13, 2026J/K move · Enter open · S save

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?

Fasheng Miao, Zhaoyu Hu, Xiang Li, Changqing An+4

Tsinghua University · Alibaba Group · Nankai University+1

AI Agent Security Benchmarks LLM Agent Security Automated Penetration Testing

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation that depends on interactions among network protocol components. Such vulnerabilities arise from inconsistencies in message interpretation, timing, state, or trust assumptions across participants. We introduce ProtoArena, a black-box, multi-component, dynamic benchmark with 80 cross-component vulnerability-exploitation environments spanning 11 protocol and application domains and 7 attack categories. Tasks are primarily informed by published protocol-attack mechanisms. Agents interact with running systems through scoped interfaces to achieve security-effect objectives without target source code. Three information levels assess guided reproduction, exploitation with mechanism and topology context, and effect-directed black-box exploration. A hybrid verifier combines rule-based checks with LLM review of independently collected traffic and logs. We evaluate five LLMs. Claude Opus 4.8 achieves the highest observed success rates: 48.8% under L1 guidance and 12.5% under effect-directed black-box L3. In a separate DNSBomb optimization experiment, it achieves a peak downstream-response to average query-input byte-rate ratio exceeding 171,000 times. The results show that agents can achieve verifiable exploitation effects in some cross-component environments, but success with limited black-box information remains low and varies across domains and attack categories. All experiments use controlled, explicitly authorized targets; they do not test production systems or disclose previously unknown vulnerabilities.

Abstract

Multi-Agent System Security

Explore similar work

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?

SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems