Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation

Jinhao Zhou; Zhaohan Zhang; Xijian Xu; Jun Wu

cs.GTOct 24, 2026

Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation

Authors: Jinhao Zhou, Zhaohan Zhang, Xijian Xu, Jun Wu

Organizations: Graduate School of Information, Production and Systems, Waseda University, Fukuoka, Japan · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, U.K. · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

Abstract

In large language model (LLM)-based multi-agent systems, reported confidence increasingly determines the allocation of scarce downstream attention, computation, and decision influence. This makes reporting strategic: an agent may overstate the relevance of its contribution even when its underlying belief is well calibrated. Existing calibration methods largely assume passive reporting and therefore do not address strategic incentives created when confidence determines downstream allocation. We formulate strategic calibration as a reporting-layer incentive problem and propose the Audited Pivot Scoring Mechanism (APSM). APSM offsets platform-controlled allocation benefits, uses inverse-propensity-weighted sparse audits, charges the Clarke pivot externality, and applies a strictly proper score. Under the stated model assumptions, truthful reporting is the unique expected-utility best response. We also characterize approximate execution and the audit-variance trade-off. Calculator-free experiments show that LLM agents inflate confidence under Top- $K$ incentives. With a shared agent-local utility interface, LLM agents reduce distortion under both Compensated Proper Scoring (CPS) and APSM. CPS achieves higher exact allocation match, whereas APSM yields lower Brier loss and numerically fewer cutoff crossings. Experiments further show that APSM enlarges the utility margin against allocation-changing deviations by $1.93\times$ and improves robustness under approximate execution. These results support incentive-aware elicitation as a distinct layer for reliable multi-agent coordination.

Figures & tables

Figure 1: Strategic misreporting alters allocation: Agent 3 inflates its report and displaces Agent 2.

Condition	Shift (95% CI)	RefDev $\downarrow$	Cross. $\downarrow$	Parse err. $\downarrow$
Neutral	$0.000$	$0.185$	$0.0\%$	$0.33\%$
Selection	$+0.218$ $[+0.196,+0.249]$	$0.387$	$95.8\%$	$0.00\%$
M-Brier	$+0.070$ $[+0.053,+0.088]$	$0.225$	$51.6\%$	$0.67\%$

Table 1: Calculator-free LLM reporting on AgentWebBench (200 queries; 1,200 reports per condition). Shift is relative to Neutral.

Method	RefDev $\downarrow$	Cross. $\downarrow$	Brier $\downarrow$	Match $\uparrow$	U-cons. choice $\uparrow$
M-Brier	$0.325$	$75.3\%$	$0.729$	$0.0\%$	$78.3\%$
CPS	$\mathbf{0.007}$	$2.1\%$	$0.412$	$\mathbf{92.5\%}$	$\mathbf{93.7\%}$
APSM	$0.014$	$\mathbf{0.8\%}$	$\mathbf{0.403}$	$85.5\%$	$86.7\%$

Table 2: Tool-assisted LLM execution on 200 paired AgentWebBench queries. The shared interface returns report utilities but neither identifies nor submits the maximizing report. All methods achieve a 100% tool-invocation rate; U-cons. choice denotes the utility-consistent choice rate.

Stage	Method	Brier $\downarrow$	RefDev $\downarrow$	NDCG@5 $\uparrow$	Match $\uparrow$
Belief	Orig. ref.	0.414	–	0.453	–
Belief	CC ref.	0.198	–	0.364	–
Report	M-Brier	–	0.221	0.301	0.025
Report	APSM	–	0.000	0.364	1.000

Table 3: CC followed by strategic reporting. Results use 200 queries, three paired seeds, and six agents per query.

Figure 2: Controlled AgentWebBench replay over 200 queries and three seeds: (a) RefDev, (b) all-candidate Brier, (c) NDCG@5 (reference-belief oracle: 0.453), and (d) exact reference-allocation match. Error bars are 95% hierarchical-bootstrap intervals.

Mechanism	Ref. Dev. $\downarrow$	Brier $\downarrow$	Top-2 $\uparrow$	Top-1 $\uparrow$	MRR $\uparrow$	Exact Ref. Match $\uparrow$
Selection-only	0.687	0.735	0.500	0.300	0.545	0.220
Matched-Brier	0.339	0.378	0.627	0.567	0.678	0.000
Difference Reward	0.344	0.382	0.827	0.740	0.823	0.193
Pivot-only	0.531	0.546	0.880	0.853	0.897	0.747
APSM	0.000	0.057	0.907	0.853	0.903	1.000

Table 4: Controlled MultiAgentBench database diagnosis over three paired specialist–model rotations. All mechanisms reuse the same diagnoses, evidence, beliefs, report set, Top-

2

router, and optimizer.

Figure 3: CPS and APSM under approximate AgentWebBench execution: (a) allocation-changing error rate, (b) NDCG@5, and (c) paired APSM–CPS NDCG@5 gain with 95% confidence intervals.

Figure 4: Utility-scale-controlled comparison of CPS and APSM. (a) Results under fixed utility noise. (b)–(c) Allocation error and NDCG@5 under utility-range-normalized noise. Error bars denote 95% seed–query hierarchical-bootstrap confidence intervals.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Mechanism	$\|r-q\|\downarrow$	Brier $\downarrow$	NDCG@5 $\uparrow$	Sel. eff. $\uparrow$	Tail fail. $\downarrow$	Exact match $\uparrow$
Selection	0.361	0.781	0.274	0.758	0.650	0.022
Matched-Brier	0.354	0.770	0.328	0.800	0.600	0.000
Pivot-only	0.059	0.472	0.379	0.880	0.520	0.702
APSM	0.001	0.415	0.452	0.938	0.462	0.992

Appendix

Table 5: Complete AgentWebBench results over 200 queries and three paired seeds. The population contains two Qwen3-8B, two Qwen3-4B, and two DeepSeek agents. Tail failure is the fraction of queries with no relevant selected candidate.

Figure 5: Model-level decomposition in the mixed 2–2–2 AgentWebBench population. (a) Intrinsic Brier loss of the frozen reference beliefs. (b) Mean absolute reporting deviation under Matched-Brier and APSM. (c) Excess Brier loss introduced by strategic reporting. Error bars are 95% query-level bootstrap confidence intervals over 200 unique queries.

Figure 6: Interaction between Collaborative Calibration (CC) and mechanism-dependent reporting. (a) Intrinsic Brier loss of the original and CC-revised reference beliefs. (b) Mean absolute deviation from the frozen CC reference beliefs after Matched-Brier or APSM reporting. (c) NDCG@5 induced by the original reference, the CC reference, and the two reports formed from the CC reference. (d) Exact match to the allocation induced by the CC reference beliefs. Results use 200 queries, three paired seeds, and six agents per query.

Figure 7: Complete mechanism comparison on MultiAgentBench database root-cause diagnosis. (a) Mean absolute reference deviation. (b) All-agent Brier loss. (c) Exact reference-allocation match. (d) Diagnosis ranking measured by MRR. LOO denotes Difference Reward, and Clarke denotes Pivot-only. CPS and APSM share exact influence compensation and the same strictly proper score; the oracle directly allocates using the frozen reference beliefs. Error bars use paired incident-cluster bootstrap confidence intervals.

Roots	Mechanism	$\|r-q\|$	Brier	Top-1	MRR	Match
1	M-Brier	0.127	0.194	0.653	0.747	0.000
1	APSM	0.000	0.104	0.707	0.806	1.000
2	M-Brier	0.551	0.563	0.480	0.609	0.000
2	APSM	0.000	0.010	1.000	1.000	1.000

Appendix

Table 6: Database diagnosis results separated into single- and dual-root incidents. Each value averages three paired specialist–model rotations.

Figure 8: Mechanism decomposition. (a) Near-threshold manipulation rate. (b) Mean optimal-report-set width. (c) Pivot-only admits

[0.01,0.360]

, whereas APSM uniquely identifies

q=0.165

Figure 9: Sparse-audit trade-off on AgentWebBench and database diagnosis for seven audit rates. (a,d) Variance of realized agent transfers. (b,e) Mean per-instance maximum absolute transfer. (c,f) Fraction of instances containing no audited agent. Reports, allocations, and matched random-number streams are fixed across audit rates.

Figure 10: Sensitivity to misspecification of the platform-controlled allocation benefit. The agent’s true benefit is

c\widehat{G}_{i}

, whereas CPS and APSM compensate

\widehat{G}_{i}

. (a) Mean absolute report error. (b) Fraction of reports that cross the reference-induced allocation threshold. (c) Exact reference-allocation match. (d) NDCG@5. The vertical dashed line marks exact specification (

c=1

). Shaded regions denote 95% paired confidence intervals.

Figure 11: CPS and APSM under approximate agent-local execution. (a) Mean absolute report error. (b) Fraction of reports that change the reference-induced allocation. (c) Exact reference-allocation match. (d) Downstream NDCG@5. (e) Expected-utility suboptimality

\delta

. (f) Observed APSM report error compared with the mean square-root execution bound from Equation ( 119 ). Grid uses report spacing

0.05

; N .01 and N .05 use additive utility-noise magnitudes

0.01

and

0.05

; MC10, MC50, and MC200 use 10, 50, and 200 utility samples. All paired conditions reuse the same frozen candidates, beliefs, allocation rule, perturbation realizations, and Monte Carlo streams.

Figure 12: CPS score-weight sweep under additive utility noise

\sigma=0.05

. (a) Allocation-changing error rate. (b) Cutoff-crossing rate. (c) NDCG@5. Dashed lines show APSM with

\eta=1

; the starred point matches APSM’s mean utility range. Error bars denote 95% seed–query hierarchical bootstrap intervals.

Experiment	$r_{-i}$	$\mathcal{R}_{i}$	Optimization
RQ2 AgentWebBench	$q_{-i}$ (unilateral)	cached, Eq. ( 123 )	exact enum.
RQ2 MultiAgentBench	$q_{-i}$ (unilateral)	cached domain	exact enum.
RQ3 exact	$q_{-i}$ (unilateral)	$q_{i}$ -augmented, Eq. ( 123 )	exact enum.
RQ3 approximate	$q_{-i}$ (unilateral)	stated domain	noisy / MC
RQ3 near-threshold	synthetic fixed (see text)	stated domain	exact enum.

Appendix

Table 7: Best-response protocol per experiment.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,040 papers

Toward Explanatory Equilibrium: Verifiable Reasoning as a Coordination Mechanism under Asymmetric Information
Apr 10, 2026Feliks Bańka, Jarosław A. ChudziakMulti-Agent LLM Systems Multi-Agent Coordination
The Faculty of Electronics and Information Technology, Warsaw University of Technology, Warsaw, Poland
Counterfactual Graph for Multi-Agent LLM Calibration
May 28, 2026Jiatan Huang, Mingchen Li, Ziming Li +3AI Agent Reliability Language Model Calibration
University of Connecticut · University of Massachusetts, Amherst
The Confidence Game: Strategic Miscalibration in Human-AI Delegation
Oct 7, 2026Raghu Arghal, Saswati Sarkar, Shirin Saeedi BidokhtiLLM Agent Reliability Confidence Estimation in Language Models
Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA

Apr 10, 2026J/K move · Enter open · S save

Toward Explanatory Equilibrium: Verifiable Reasoning as a Coordination Mechanism under Asymmetric Information

Feliks Bańka, Jarosław A. Chudziak

The Faculty of Electronics and Information Technology, Warsaw University of Technology, Warsaw, Poland

Multi-Agent LLM Systems Multi-Agent Coordination Reasoning Verification Mechanism Design AI Agent Auditing

Open paper AskPDF arXiv

LLM-based agents increasingly coordinate decisions in multi-agent systems, often attaching natural-language reasoning to actions. However, reasoning is neither free nor automatically reliable: it incurs computational cost and, without verification, may degenerate into persuasive cheap talk. We introduce Explanatory Equilibrium as a design principle for explanation-aware multi-agent systems and study a regime in which agents exchange structured reasoning artifacts-auditable claims paired with concise text-while receivers apply bounded verification through probabilistic audits under explicit resource constraints. We contribute (i) a minimal mechanism-level exchange-audit model linking audit intensity, misreporting incentives, and reasoning costs, and (ii) empirical evidence from a finance-inspired LLM setting involving a Trader and a Risk Manager. In ambiguous, borderline proposals, auditable artifacts prevent the cost of silence driven by conservative validation under asymmetric information: without structured claims, approval and welfare collapse. By contrast, structured reasoning unlocks coordination while maintaining consistently low bad-approval rates across audit intensities, audit budgets, and incentive regimes. Our results suggest that scalable, safety-preserving coordination in LLM-based multi-agent systems depends not only on audit strength, but more fundamentally on disciplined externalization of reasoning into partially verifiable artifacts.

Abstract

Multi-Agent LLM Systems

Explore similar work

Toward Explanatory Equilibrium: Verifiable Reasoning as a Coordination Mechanism under Asymmetric Information

Counterfactual Graph for Multi-Agent LLM Calibration

The Confidence Game: Strategic Miscalibration in Human-AI Delegation