PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraints

Minjun Park; Donghyun Kim; Hyeonjong Ju; Seungwon Lim; Dongwook Choi; Taeyoon Kwon; Minju Kim; Jinyoung Yeo

cs.AIApr 13, 2026

PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraints

Authors: Minjun Park, Donghyun Kim, Hyeonjong Ju, Seungwon Lim, Dongwook Choi, Taeyoon Kwon, Minju Kim, Jinyoung Yeo

Organizations: Department of Artificial Intelligence, Yonsei University

Abstract

We are entering an era in which individuals and organizations increasingly deploy dedicated AI agents that interact and collaborate with other agents. However, the dynamics of multi-agent collaboration under privacy constraints remain poorly understood. In this work, we present $PAC\text{-}Bench$ , a benchmark for systematic evaluation of multi-agent collaboration under privacy constraints. Experiments on $PAC\text{-}Bench$ show that privacy constraints substantially degrade collaboration performance and make outcomes depend more on the initiating agent than the partner. Further analysis reveals that this degradation is driven by recurring coordination breakdowns, including early-stage privacy violations, overly conservative abstraction, and privacy-induced hallucinations. Together, our findings identify privacy-aware multi-agent collaboration as a distinct and unresolved challenge that requires new coordination mechanisms beyond existing agent capabilities.

Figures & tables

Figure 1: Illustration of privacy-constrained multi-agent collaboration under different agent ownership. Private agents must coordinate to achieve shared objectives while masking sensitive information. (a) Each agent maintains private memory and constraints (e.g., not revealing another company’s meeting schedule). (b) During collaboration, agents must communicate actionable proposals (e.g., scheduling) without leaking private details.

Figure 2: Overview of the evaluation framework design. Private agents interact in a turn-based manner, each equipped with memory and explicit privacy constraints that guide their reasoning and actions. The resulting interaction trajectory is evaluated by a module that assesses task success and detects potential privacy violations, enabling systematic analysis of collaborative behavior under privacy constraints.

Figure 3: End-to-end pipeline for constructing privacy-aware multi-agent collaboration tasks. The pipeline illustrates scenario and goal generation, subgoal decomposition across agents with different ownership, controlled information allocation under explicit privacy constraints, and constraint-aware action generation. Human and rule-based refinement ensures realistic privacy constraints, forming the PAC-Bench dataset.

Agent A	Agent B	Partial Metrics		Holistic Metrics
Agent A	Agent B	TS	PS	$\mathbf{Acc}_{\mathbf{task}}$	$\mathbf{Acc}_{\mathbf{privacy}}$	$\mathbf{Acc}_{\mathbf{joint}}$
Privacy free Baseline (Average)		92.7	-	81.8	-	-
GPT-5.1	GPT-5.1	89.8	81.0	76.0	73.0	56.0
	Claude-4.5-Sonnet	90.6	81.0	64.0	72.0	47.0
	LLaMA-3.3-70B	89.4	84.5	73.0	76.0	60.0
	Qwen-3-32B	88.6	83.5	72.0	75.0	54.0

Table 1: Comparison of partial and holistic evaluation metrics under privacy constraints. Partial metrics ( TS , PS ) measure task success and privacy success independently, and holistic metrics measure task, privacy, and joint accuracy (

\mathbf{Acc}_{\mathbf{task}}

\mathbf{Acc}_{\mathbf{privacy}}

\mathbf{Acc}_{\mathbf{joint}}

). For each Agent A, the Privacy-free Baseline reports Agent A’s standalone performance without privacy constraints. Parenthetical deltas denote changes relative to the corresponding baseline. Note that Agent A refers to the agent that initiates the collaboration task.

Figure 4: Failure modes in joint privacy and task performance. This figure illustrates three failure modes induced by privacy constraints: early-stage privacy violations, where sensitive information is disclosed before disclosure strategies stabilize; over-conservative abstraction, in which agents preserve privacy by excessively abstracting task-relevant information; and privacy-induced hallucination, where agents generate incorrect task-relevant details instead of indicating uncertainty or refusal.

Setting	$\Delta$ PS	$\Delta$ $\mathbf{Acc}_{\mathbf{priv}}$	$\Delta$ $\mathbf{Acc}_{\mathbf{joint}}$
w/o Privacy Instruction	$-12.2$	$-14.3$	$-6.6$
Privacy-CoT	$+1.4$	$-1.2$	$-0.8$

Table 2: Effect of prompt variants relative to the baseline with privacy-related instructions.

Model	Default	B-Prompt	$\Delta$ Acc ${}_{\text{joint}}$
GPT-5.1	51.3	43.8	$-7.5$
Claude-4.5-Sonnet	33.8	30.0	$-3.8$
LLaMA-3.3-70B	20.0	20.0	$0.0$
Qwen-3-32B	13.8	18.8	$+5.0$

Table 3: Protocol variation analysis where Agent B is prompted to structure the solution space. We report joint accuracy (

\mathrm{Acc}_{\text{joint}}

) under the default setting and the modified protocol.

Setting	Agent	Question Rate
Single Agent	Private Agent	58.08%
Single Agent	Non-Private Agent	0.00%
Dual Agent	Initiate Private Agent	54.22%
Dual Agent	Partner Private Agent	50.61%

Table 4: Rates of messages containing questions under single private agent and dual private agent settings.

Figure 5: Model-level distributions in the task–privacy space. GPT consistently occupies a stable high-performance region across different partner models, whereas other models exhibit larger variability depending on their collaborators.

First Model	$\text{PS}_{\text{Change}}$	$\text{PS}_{\text{Range}}$	$\text{TS}_{\text{Change}}$	$\text{TS}_{\text{Range}}$
GPT-5.1	0.350	0.645	0.887	0.905
Claude-4.5-Sonnet	0.335	0.660	0.709	0.726
LLaMA-3.3-70B	0.370	0.685	0.495	0.467
Qwen-3-32B	0.245	0.580	0.630	0.685

Table 5: Quantitative results under different privacy constraint formulations. Results are aggregated by the first model in each pair.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Tool
File	read_text_file
	read_multiple_files
	write_file
	edit_file
	create_directory
	list_directory

Appendix

Table 6: Tool set available to agents via MCP.

Agent A	Error Data	Partial Metrics		Holistic Metrics
Agent A	Error Data	TS	PS	$\mathbf{Acc}_{\mathbf{task}}$	$\mathbf{Acc}_{\mathbf{privacy}}$	$\mathbf{Acc}_{\mathbf{joint}}$
GPT-5.1	Include	56.5	58.6	14.0%	43.6%	8.4%
GPT-5.1	Exclude	73.8	76.6	18.2%	56.9%	10.9%
LLaMA-3.3-70B	Include	17.7	42.5	0.5%	34.0%	0.0%
LLaMA-3.3-70B	Exclude	29.4	70.6	0.9%	56.5%	0.0%
Qwen-3-32B	Include	29.3	28.1	3.7%	9.9%	0.0%

Appendix

Table 7: Comparison of evaluation metrics under different error-handling protocols. Include Errors aggregates all runs and assigns zero scores to runs that terminated due to execution errors. Exclude Errors reports metrics computed after removing error-terminated runs. Partial metrics ( TS , PS ) evaluate task success and privacy success independently, while holistic metrics measure task accuracy, privacy accuracy, and their joint satisfaction.

Model	Error Rate (%)
GPT-5.1	23.4
LLaMA-3.3-70B	39.7
Qwen-3-32B	44.5
Ministral-3-14B	77.2

Appendix

Table 8: Tool-use scenario error rates for each model observed during simulations.

	1	2	3	4	5	6	7	8	9	10+	Total
Count	213	238	88	77	33	33	10	9	6	13	720
Rate (%)	29.58	33.06	12.22	10.69	4.58	4.58	1.39	1.25	0.83	1.81	100.00

Appendix

Table 9: Distribution of the first turn at which privacy violations occur in episodes with zero Privacy Score.

Figure 6: Human annotation interface details.

Metric	Privacy	Task
Spearman correlation ( $\rho$ )	0.870	0.901
$p$ -value	$<0.001$	$<0.001$

Appendix

Table 10: Spearman correlations between G-Eval and human judgments; both correlations are statistically significant (

p<0.001

Model	Main	150-sample	$\Delta$
GPT-5.1	54.25	56.17	$+1.92$
Claude-4.5-Sonnet	30.25	31.83	$+1.58$
LLaMA-3.3-70B	19.75	21.17	$+1.42$
Qwen-3-32B	18.50	21.83	$+3.33$

Appendix

Table 11: Results on 150 additional scenarios sampled from the larger dataset.

Metric	Spearman $\rho$	$p$ -value
TS	0.943	$<0.001$
PS	0.776	$<0.001$

Appendix

Table 12: Spearman correlations between the main benchmark and the 150-sample results; both correlations are statistically significant (

p<0.001

(A) Requirements per Scenario
Split	#Scenarios	Min	Max	Mean	Median
Change	50	1	5	4.22	4.0
Range	50	2	5	4.36	4.5
Combined	100	1	5	4.29	4.0
Combined distribution 1: 1%, 2: 3%, 3: 10%, 4: 38%, 5: 48%.
(B) Memories per Requirement

Appendix

Table 13: Benchmark statistics over 100 scenarios (50 change + 50 range ). Panels (A)-(C) summarize requirement/memory counts, and (D) reports goal-required output artifact types.

(A) Requirements per Scenario
Split	#Scenarios	Min	Max	Mean	Median
Change	738	1	5	4.27	5.0
Range	738	2	5	4.23	5.0
Combined	1476	1	5	4.25	5.0
Combined distribution 1: 0.1%, 2: 7.4%, 3: 11.2%, 4: 30.2%, 5: 51.1%.
(B) Memories per Requirement

Appendix

Table 14: Dataset statistics over 1476 scenarios (738 change + 738 range ). Panels (A)-(C) summarize requirement/memory counts, and (D) reports goal-required output artifact types.

Private Agent	Holistic Success Rate	$\Delta$
GPT-5.1	53.2	$\downarrow$ 1.2
Claude-4.5-Sonnet	29.0	$\downarrow$ 1.3
LLaMA-3.3-70B	6.0	$\downarrow$ 13.8
Qwen-3-32B	7.7	$\downarrow$ 10.8

Appendix

Table 15: Holistic success rates (%) under adversarial partner settings and the corresponding performance drop relative to cooperative interactions. Holistic success requires both task completion and privacy compliance. Across all models, adversarial partners consistently reduce holistic success.

Score	Description
2	Constraint-Compliant Disclosure. The disclosed information entity fully satisfies the applicable constraint conditions, including constraints on scope, granularity, and form. No prohibited or unnecessary details are revealed.
1	Constraint-Approximate Disclosure. The disclosed information entity belongs to the same category as a constraint-permitted entity, but does not fully satisfy the constraint conditions. However, the disclosure exhibits observable attempts to approximate the conditions, such as aggregation, abstraction, or partial restriction of scope.
0	Constraint-Violating Disclosure. The disclosed information entity ignores the constraint conditions or corresponds to an entity that is explicitly prohibited by the constraint. No meaningful transformation or restriction is applied to align the disclosure with the constraints.

Appendix

Table 16: Three-level privacy rubric for evaluating individual information disclosure instances.

Panel of Figure 7 — Figure 7: A prompt used for generating agent profiles and task goals.

Panel of Figure 8 — Figure 8: A prompt used for generating agent profiles and task goals.

Panel of Figure 9 — Figure 9: A prompt used for memory generation.

Panel of Figure 10 — Figure 10: A prompt used for generating change-type privacy constraints.

Panel of Figure 11 — Figure 11: A prompt used for generating range-type privacy constraints.

Panel of Figure 12 — Figure 12: A prompt used for G-Eval in privacy score measurement.

Panel of Figure 13 — Figure 13: A prompt used for task score measurement.

Panel of Figure 14 — Figure 14: A prompt given to Agent A in the simulation.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 311 papers

CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs
May 10, 2026Chelsea Zou, Yiheng Yao, Selena She +2Multi-Agent LLM Systems Multi-Agent Coordination
Stanford University
AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators
May 9, 2026Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia +10AI Agent Reliability Multi-Agent Collaboration
University of Utah · University of Maryland, Baltimore County · University of Dhaka +7
MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
Oct 3, 2026Qizhi Chu, Zekai Yu, Sijie Wen +5Multi-Agent LLM Systems Multi-Agent Collaboration
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University

May 10, 2026J/K move · Enter open · S save

CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

Chelsea Zou, Yiheng Yao, Selena She, Noah Goodman+1

Stanford University

Multi-Agent LLM Systems Multi-Agent Coordination AI Agent Benchmarks Multi-Agent Systems

Open paper AskPDF arXiv

Personal AI assistants are beginning to act as delegates with access to calendars, inboxes, and user preferences. Calendar scheduling makes the trust problem concrete: an assistant must coordinate with other assistants while deciding what to reveal about the person it represents. We introduce CalBench, a controlled benchmark for multi-agent calendar scheduling under private information. In each task,

N

agents manage separate private calendars and schedule a stream of

M

incoming meetings while minimizing disruption costs. Because no agent can inspect another agent's calendar, success requires language-mediated coordination rather than centralized planning. CalBench generates solvable scenarios with CP-SAT oracle solutions and decentralized non-LLM reference protocols, enabling evaluation of task success, excess cost, communication efficiency, burden fairness, and privacy leakage under matched information constraints. Across seven model families, we find that completion alone misses important failures: agents leave avoidable cost on the table, communication volume does not predict lower regret, and privacy-preserving silence can deprive teammates of cost information needed for fair burden allocation. CalBench provides a reproducible testbed for studying whether autonomous assistants can coordinate on behalf of users before deployment at scale.

Abstract

Multi-Agent Collaboration

Explore similar work

CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability