Organizations: Knight Foundation School of Computing and Information Sciences, Florida International University · SRI International · Department of Computer & Information Science & Engineering, University of Florida · Oak Ridge National Laboratory · Army Cyber Institute, United States Military Academy · Department of Computer Science and Engineering, Oakland University
Multi-agent LLM systems can improve reasoning by pooling diverse perspectives, but their effectiveness depends on coordinating communication, particularly in hidden-profile settings where each agent holds only part of the evidence required for a correct decision. Existing protocols, including fixed schedules, round-robin exchange, and unstructured debate, provide no guarantee that a conversational action is appropriate. We propose Consilience, an inference-time orchestration framework that both steers and certifies multi-agent communication under distributed private information. At each turn, Consilience summarizes the discussion using a compact state capturing uncertainty, disagreement, evidence gain, redundancy, and premature consensus, then selects both a communication intervention (challenge, clarify, seek evidence, or route) and an appropriate speaker. Its central contribution is a round-wise conformal calibration procedure that provides a distribution-free, finite-sample guarantee: at each discussion round, conditional on reaching that round, the one-step regret of a controller's proposed action is bounded by a calibrated threshold with marginal probability at least 1 - alpha; an acceptance mechanism enforces the same guarantee for the executed action by replacing inadmissible proposals. On HiddenBench-style hidden-profile tasks spanning 12 open and closed weight language models, Consilience improves decision accuracy and communication efficiency over fixed and unstructured discussion protocols, sometimes surpassing a full-information baseline where every agent observes all evidence. These results demonstrate that certified adaptive communication control can be more valuable than increasing information availability, providing a practical mechanism for reliable multi-agent LLM coordination.
Figures & tables
Figure 1: Proposed framework Consilience : (1) Each agent receives shared task information and a private clue. (2) Agents report beliefs conditioned on the public transcript, which Consilience aggregates with transcript-level signals into the collective state st . (3) A controller proposes a communication action, and round-wise conformal acceptance retains or replaces it using calibrated one-step regret. (4) An action-conditional router selects the speaker, whose message updates the transcript and agent beliefs. The loop repeats until termination, followed by plurality voting with confidence-based tie-breaking.
Method
Task Accuracy
Vote Accuracy
Hidden Pre
0.2537
0.2475
Hidden Post
0.4982
0.4925
Full Info
0.3657
0.3358
Rules (Ours)
0.6544
0.6542
MLP (Ours)
0.5071
0.5025
LLM (Ours)
0.6150
0.6125
Table 2: Task accuracy and vote accuracy for the benchmark generated from GroupTravelBench Cheng et al. (2026) averaged across 12 models.
Figure 2: Three agents choose between Apartments A and B. Shared evidence initially favors Apartment A, but complementary private clues collectively identify Apartment B as the correct choice. Round-robin discussion repeats common evidence further suppresses support for B (as Agent 1 has no clue about any utility surcharges for Apartment B in Round 1), and the final decision converges on A. Consilience instead adaptively selects interventions and speakers to surface unshared evidence, reverse premature consensus, and recover B.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Features retained
Success ↑
Communication cost ↓
Final objective ↓
All features
82.22%
0.5228
0.0239
Information only
82.22%
0.5254
0.2277
Belief + information
77.78%
0.5285
0.3509
Belief + cost
77.78%
0.5253
0.1582
Belief only
75.56%
0.4625
0.2212
Appendix
Table 6: State-feature ablation for Qwen3-8B. Each row retrains the MLP using only the retained feature groups. Higher success is better; lower communication cost and final objective are better.
Configuration
Rules
MLP
LLM
LLM+Vote
Average
Full
92.3
87.7
89.2
83.1
88.1
– Challenge
90.8 ( −1.5 )
86.2 ( −1.5 )
87.7 ( −1.5 )
83.1 (0.0)
86.9 ( −1.2 )
– Clarify
93.8 (+1.5)
87.7 (0.0)
89.2 (0.0)
81.5 ( −1.5 )
88.1 (0.0)
– New Evidence
92.3 (0.0)
92.3 (+4.6)
81.5 ( −7.7 )
70.8 ( −12.3 )
84.2 ( −3.8 )
– Routing
87.7 ( −4.6 )
89.2 (+1.5)
87.7 ( −1.5 )
69.2 ( −13.8 )
83.5 ( −4.6 )
– Voting
92.3 (0.0)
NA
NA
83.1 (0.0)
87.7 (0.0)
Appendix
Table 7: Action ablation study on HiddenBench. Each row removes one communication action from the full controller. Entries report task accuracy (%), with the change relative to the corresponding full action set shown in parentheses. Negative values indicate performance degradation.
Routing policy
Mean ΔJ↓
Normalized regret ↓
Consilience (action-conditional)
-0.0079
0.4868
Random
0.0145
0.4930
Round-robin
-0.0017
0.4839
Most-disagreeing
0.1205
0.5735
Most-spoken
0.2815
0.7184
Appendix
Table 8: Counterfactual speaker-routing ablation with Qwen3-8B. The controller-selected action is held fixed and only the speaker-selection policy is changed. Lower objective change is better, while lower normalized regret indicates routing decisions closer to the counterfactual optimum.
Method
Mean Success (%)
Mean Rounds/Trial
LLM
82.75
9.54
MLP
82.75
9.87
Rules
64.50
3.24
LLM+Voting
63.00
5.43
Appendix
Table 9: Comparison of controller methods over 400 trials. Using Qwen3-8B model.
Multi-agent communication studies how autonomous agents exchange information to coordinate decisions, share knowledge, and act together under constraints.