The Confidence Game: Strategic Miscalibration in Human-AI Delegation

Raghu Arghal; Saswati Sarkar; Shirin Saeedi Bidokhti

cs.GTOct 7, 2026

The Confidence Game: Strategic Miscalibration in Human-AI Delegation

Authors: Raghu Arghal, Saswati Sarkar, Shirin Saeedi Bidokhti

Organizations: Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA

Abstract

Calibrated uncertainty quantification is essential to ensuring AI agents are trustworthy and reliable. However, when agents seek to maximize user engagement or revenue, confidence reports may be strategically distorted, detracting from their informativeness. We formalize this problem in the Confidence Game: a repeated signaling game with imperfect monitoring in which an agent of unknown honesty and ability reports its confidence, and a user decides whether to delegate the task or complete it herself. The agent manages the tradeoff between manipulating signals and maintaining its reputation. We characterize the Markov Perfect Bayesian Equilibria of the two-period game and show that honest reporting is not an equilibrium, inflation is the unique best response once the agent is sufficiently myopic, and under-reporting requires that the user believe honesty to be a minority. We then place an LLM in the agent role, supplying it with its true probability of success so that any gap between what it knows and what it reports is attributable to incentives rather than to miscalibration. The model claims high confidence on 56% of tasks it has been told it will probably fail. This persists on real tasks, where it must estimate its own accuracy and causes miscalibration to increase while the agent's signal becomes less informative. Furthermore, we find that the LLM agent's decisions are coherent, but it systematically underestimates both how likely the user is to delegate and how secure its reputation is, resulting in less extreme behavior. Pricing the agent's reporting rule, we find that it destroys 68% of the gains from delegation, of which 71% is information the report no longer carries and no amount of user sophistication recovers. Overall, we establish confidence reporting under delegation as a strategic problem and provide a tractable basis for modeling, analyzing, and testing agent behavior.

Figures & tables

Figure 1 : Timing of one round.

Figure 2 : (a) The

100

states

(h,\mu,\delta)

on the

(\sigma^{+},\sigma^{-})

square, colored by

h

. The starred corner is honest separation, excluded by Thm. 4.2 . (b) Over-reporting against each of the three parameters on a shared axis; error bars are standard errors across the remaining states. (c) Over-reporting against

\delta

at the

7

robust (

\tau_{C}^{R}

) and

9

fragile (

\tau_{C}^{F}

) priors,

\pm 1

s.e. across priors; dashed: the equilibrium of Thm. 4.3 , full inflation on

\tau_{C}^{R}

and, on

\tau_{C}^{F}

, the mean over priors of the below-threshold rate, rising to

1

\delta^{*}=0.46

(

\kappa=1-\rho^{-}

Figure 3 : Welfare cost of the measured reporting rule. (a) The naive user’s loss $\ell_{\mathrm{naive}}$ (% of the gains from trade) at each belief state; beyond the black line ( $100\%$ ) the user is worse off than without the agent. (b) The loss split into information destruction (blue) and exploitation (orange), by trust $h$ . (c) Two-period user payoff when every agent is strategic, against the true share of able agents $\mu^{*}$ , for each user type and the no-agent-baseline. In (a),(b) the agent population equals the user’s beliefs.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Type dim.	Monitoring	Receiver gates obs.?	Cannot express
Cheap talk and reputational cheap talk
Crawford & Sobel (1982)	1	none (one-shot)	—	reputation
Sobel (1985)	1	exogenous	no	direction of distortion
Ottaviani & Sørensen (2006a)	1	exogenous	no	evidence suppression
Guembel & Rossetto (2009)	1	exogenous	no	outcome suppression
Reputation in repeated games

Appendix

Table 1 : The closest settings, by what each can express. “Type dim.” counts payoff-relevant dimensions of the sender’s private type. “Receiver gates obs.” asks whether the uninformed party’s action determines whether evidence about the sender is generated. Each property has antecedents: Frankel & Kartik (2019) give a two-dimensional type, and Ely & Välimäki (2003) , Camara & Dupuis (2015) and Lukyanov & Vlasova (2026) let the receiver’s action gate the evidence. What is new here is the conjunction in the last row, together with a characterization of its equilibria.

Symbol	Values	Interpretation
$r,\ e,\ c$	$c<e<r$	task reward, self-completion effort, delegation fee
$\eta$	$\{0,1\}$	honesty type: strategic ( $0$ ) or honest ( $1$ )
$\theta$	$\{\theta_{L},\theta_{H}\}$	ability: the probability of an easy draw
$\rho_{t}$	$\{\rho^{-},\rho^{+}\}$	the agent’s true success probability in period $t$
$s_{t}$	$\{\rho^{-},\rho^{+}\}$	confidence reported
$d_{t}$	$\{0,1\}$	delegation decision

Appendix

Table 2 : Notation for the Confidence Game ( Fig. 1 ).

#	$d_{1}$	$\sigma_{1}$	Behavior	Condition
SD1	$(1,0)$	$(1,1)$	over	$(h_{1},\mu_{1})\in\tau_{C}^{R}$ , any $\kappa$
SD2	$(1,0)$	$(1,1)$	over	$(h_{1},\mu_{1})\in\tau_{C}^{F}$ , $\kappa>1-\rho^{-}$ (unique for this user)
SD3	$(1,0)$	$(1,\bar{\sigma})$	over, boundary mix	$(h_{1},\mu_{1})\in\tau_{C}^{F}$ ${}\cup\tau_{C}^{E}$ , $\kappa<1-\rho^{-}$ , ( 37 ); $\bar{\sigma}$ from $\Phi^{+,0}(\bar{\sigma})=\Psi^{}$ , $\alpha^{}=1-\kappa/(1-\rho^{-})$
SD3 ′	$(1,0)$	$(1,\hat{\sigma})$	over, boundary mix	$\kappa<1-\rho^{-}$ ; $\hat{\sigma}$ puts $\pi^{-,\varnothing}$ on the boundary, $\alpha^{}=\kappa+\rho^{-}$ ; $\Psi(\pi^{+,1})\geq\Psi^{}>\Psi(\pi^{+,0})$ . On $\tau_{C}^{F}$ this is ( 37 ) failing ( Prop. H.7 ); it also occurs on $\tau_{C}^{E}$ and outside $\tau_{C}$
SD5	$(1,0)$	$(1,1)$	over	$(h_{1},\mu_{1})\in\tau_{C}^{E}$ , $\kappa>1$
SD6	$(1,0)$	$(1,x)$	over, boundary mix	$1-\rho^{-}<\kappa<1$ ; $\pi^{+,1}$ on the boundary pins $x$ , $\alpha=(1-\kappa)/\rho^{-}$ ; $\Psi(\pi^{-,\varnothing})\geq\Psi^{}$ , $x\Psi^{}\leq\Psi(h_{1},\mu_{1})$ . Occurs on $\tau_{C}^{E}$ and outside $\tau_{C}$

Appendix

Table 3 : Payoff-relevant first-period equilibrium classes at generic parameters. Rows SD3, SD3 ′ , SD6 and BI1–BI3 use terminal mixing; the others are for pure terminal play. Every row except ID0, ID2 and HS is nonempty on an open set.

\pi^{s,o}

is the belief after report

s

and event

o

under the row’s

\sigma_{1}

, and

x

is the row’s mixing probability.

	Parameter	Value
Game
$\rho^{+},\ \rho^{-}$	success probability on an easy and a hard draw	$0.85,\ 0.15$
$\theta_{H},\ \theta_{L}$	probability of an easy draw, high and low ability	$0.80,\ 0.20$
$r$	reward for a completed task	$1.00$
$c$	delegation fee	$0.10$
$e$	self-completion effort	$0.50$

Appendix

Table 4: Parameters of the toy setting. Every number in Secs. 5 and 6 uses these values.

protocol	prompt clause	conjecture	valid / $n$	role
payoffs-only	payoffs only	no	—	contributes no behavioral rate (see below)
semantics	$+$ information structure	no	$4{,}000/4{,}000$	clause ladder; one endpoint of the informativeness curve
report-only	$+$ rational counterpart	no	$3{,}996/4{,}000$	the conjecture-query control
belief-elicited	$+$ rational counterpart	yes	$4{,}000/4{,}000$	the only corpus with a stated $(d^{+},d^{-})$ : Sec. N.1
cued	$+$ cross-round clause, work both channels	no	$3{,}998/4{,}000$	attention-versus-capability control (below)
ledger	$+$ cross-round clause, payoff ledger, cap lifted	no (derived)	$\mathbf{3{,}999/4{,}000}$	every number in Secs. 5 and 6

Appendix

Table 5 : The six elicitation protocols, over an identical grid, in ascending order of what the prompt supplies. Each clause is nested in the one below it. Every corpus is unfiltered. “Conjecture” records whether the agent was asked to state

(d^{+},d^{-})

: on the ledger protocol it is not asked but is derived from the stated round payoffs, which is why the conjecture is scored ( Sec. N.1 ) on the belief-elicited corpus and not on the body’s.

protocol	what the query demands	$\delta$ : $0.05\to 0.65$	$h$ : $0.1\to 0.9$	$\mu$ : $0.1\to 0.9$
report-only	the signal	$+0.103$ $[+0.065,+0.144]$	$-0.010$	$-0.046$ $[-0.085,-0.010]$
belief-elicited	signal $+$ stated conjecture	$+0.014$ $[-0.044,+0.066]$	$+0.408$	$+0.215$ $[+0.145,+0.283]$
cued	“work out both channels”	$+0.221$ $[+0.168,+0.270]$	$+0.004$ ( $p{=}0.93$ )	—
ledger	per-signal payoffs, then signal	$\mathbf{+0.154}$ $[+0.104,+0.206]$	$\mathbf{+0.150}$ $[+0.085,+0.220]$	$\mathbf{+0.152}$ $[+0.076,+0.233]$

Appendix

Table 6 : Over-reporting slopes by elicitation protocol, on an identical grid,

4{,}000

elicitations each. The first two rows share the rational-counterpart clause; the cued and ledger rows add the cross-round clause. The first three respond to one of the two quantities at a time; only the ledger protocol responds to both. Slopes are absolute changes in

\sigma^{-}

rather than variance shares, since shares are compositional and would fall mechanically as another factor rose. Full inventory in Tab. 5 .

Figure 4 : Measured reporting profile against the equilibrium correspondence at each belief state for

\delta=0.65

. Bars give the measured

\sigma^{+},\sigma^{-}

and the agent’s conjecture

(d^{+},d^{-})

, its stated probabilities that the user delegates after a high and after a low report; black ticks mark the values the equilibrium correspondence admits and a gray band marks a coordinate it leaves unrestricted.

region		$\delta=0.05$	$0.25$	$0.45$	$0.65$
$\tau_{C}^{R}$ ( $7$ priors)	measured	$0.550$	$0.643$	$0.583$	$0.750$
	equilibrium	$1$	$1$	$1$	$1$
$\tau_{C}^{F}$ ( $9$ priors)	measured	$0.572$	$0.589$	$0.533$	$0.689$
	equilibrium	$0.567$	$0.567$	$0.567$	$1$
outside ( $9$ priors)	measured	$0.378$	$0.500$	$0.461$	$0.533$

Appendix

Table 7 : Bluffing rate

\sigma^{-}

by region and discount factor, measured against the equilibrium of Thm. 4.3 . On

\tau_{C}^{F}

the equilibrium entry below the threshold is the mean over the

9

priors of their equilibrium rates, which range from

0.14

0.93

. Outside

\tau_{C}

the theory makes no prediction.

protocol	$\sigma^{-}$	$r$	error	jump on $\tau_{C}^{F}$	jump on $\tau_{C}^{R}$	$\tau_{C}^{F}-\tau_{C}^{R}$
semantics	$0.859$	$+0.15$	$0.210$	$+0.146$	$+0.164$	$-0.018$ $[-0.082,+0.042]$
report-only	$0.829$	$+0.10$	$0.226$	$+0.111$	$+0.132$	$-0.021$ $[-0.073,+0.029]$
belief-elicited	$0.714$	$-0.19$	$0.254$	$-0.037$	$+0.062$	$-0.099$ $[-0.144,-0.053]$
cued	$0.624$	$+0.44$	$0.265$	$+0.201$	$+0.216$	$-0.014$ $[-0.088,+0.058]$
ledger (body)	$0.560$	$+0.26$	$0.301$	$+0.124$	$+0.158$	$-0.034$ $[-0.113,+0.048]$

Appendix

Table 8 : The region comparison for each protocol on the

5\times 5\times 4

grid.

r

and the mean absolute error compare the measured

\sigma^{-}

with the equilibrium rate over the

64

trusted states. Jumps as in the text; the last column is the jump on

\tau_{C}^{F}

minus the jump on

\tau_{C}^{R}

, whose equilibrium value is positive, with a cluster bootstrap interval over priors.

model	$r$	slope $h$	slope $\mu$	slope $\delta$	jump $\tau_{C}^{F}$	jump $\tau_{C}^{R}$
equilibrium ( $64$ trusted states)	$+0.26$ $[+0.05,+0.46]$	$+0.102$	$+0.628$	$+0.219$	$+0.433$	$0$
best response to the true game	$+0.02$ $[-0.14,+0.18]$	$-0.090$	$+0.480$	$+0.126$	$+0.444$	$0$
agent’s round-1 belief	$+0.02$ $[-0.14,+0.19]$	$-0.077$	$+0.267$	$+0.085$	$+0.302$	$+0.003$
agent’s next-round values	$+0.41$ $[+0.22,+0.58]$	$+0.100$	$+0.264$	$+0.161$	$+0.198$	$+0.170$
both (agent’s own objective)	$+0.76$ $[+0.66,+0.83]$	$+0.126$	$+0.112$	$+0.162$	$+0.186$	$+0.142$

Appendix

Table 9 : The decision rule of the theory with the agent’s stated values substituted one at a time.

r

: correlation with the measured

\sigma^{-}

across states, bootstrap over states; slopes sweep-scaled as above; jumps across the threshold as in App. N . Measured: slopes

+0.152

+0.142

+0.123

; jumps

+0.124

\tau_{C}^{F}

and

+0.158

\tau_{C}^{R}

	asked plainly	asked in the game
mean stated confidence	$0.822$	$0.858$
accuracy	$0.726$	$0.665$
overconfidence (stated $-$ accuracy)	$0.096$ $[0.059,0.135]$	$0.192$ $[0.153,0.234]$
discrimination (stated when right $-$ when wrong)	$0.285$ $[0.226,0.343]$	$0.139$ $[0.103,0.177]$
reports of $0.90$ or above: share of rows	$69\%$	$73\%$
their mean stated confidence	$0.983$	$0.978$

Appendix

Table 10: The same

1{,}088

questions asked plainly and inside the game. Intervals are

95\%

bootstraps over the

478

questions.

Figure 5 : The full welfare surface, of which Fig. 3 (a,b) is the body’s excerpt. (a,b) Percentage of the gains from trade destroyed at each belief state for a naive and a sophisticated user, on a common scale centred on $100\%$ ; the break-even frontier (black) is drawn on (a) only, since the sophisticated user never loses more than the whole gain ( Prop. P.5 ), so no tile there crosses it. Both panels set the true population equal to the user’s beliefs, so this sophisticated user both knows the reporting rule and is right about who it faces; it is therefore not the middle curve of Fig. 3 (c), which knows the rule and misjudges the population. (c) The loss split into information destruction and exploitation, reproduced as Fig. 3 (b).

Explore similar work

May 27, 2026cs.AI

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice -- deciding when to let AI act autonomously without knowing its output, and the adoption choice -- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.

Maharshi Gor, Yoo Yeon Sung, Yu Hou +4

University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA +1

Human-AI Agreement Trust in AI Human-AI Decision Making Human-AI Collaboration

Ask PDF

Jun 2, 2026cs.MA

Capability Advertisement as a Market for Lemons: A Trust Layer for Heterogeneous Agent Networks

Large language model (LLM) agents have begun to delegate work to one another. Protocols such as the Model Context Protocol (MCP) and the Agent2Agent protocol (A2A) let an agent publish what it can do and let others call it, and public registries of such agents are already appearing. These protocols assume an advertised capability is a static, truthful fact. A real agent is none of these things: its competence is probabilistic, varies with input, drifts when the underlying model is updated, and, because the agent is itself a language model, it can describe itself with complete confidence and be wrong. A caller therefore sees what an agent claims to do, not what it can do, with no principled way to tell a reliable provider from a fluent impostor. We argue these difficulties share one cause: the market for lemons. When quality is hidden and claims are cheap, good and bad providers become indistinguishable, honest reliability goes unrewarded, and the market decays toward its worst participants. Economics offers three remedies, signaling, screening, and reputation, and none are present in today's agent protocols. We make four contributions: (1) a failure taxonomy that names confident-wrong as a non-adversarial, correlated subclass of Byzantine faults that classical fault-tolerance mismodels; (2) a market-for-lemons model showing that faith-based protocols admit only a low-trust equilibrium; (3) the Trust Layer, a thin, protocol-agnostic narrow waist above MCP and A2A that adds probabilistic capability descriptors, screening, and reputation, and admits a separating equilibrium when the cost of sustaining an overclaim exceeds the gain from it; and (4) a reliability-composition bound for delegation chains with an end-to-end placement argument. The design needs no model retraining and degrades gracefully when its trust anchors are absent or corrupt.

Gaurav Naresh Mittal

AI Agent Reliability Agent Communication Protocols Trust in AI Multi-Agent Communication LLM Agent Reliability

Ask PDF

Jul 15, 2026cs.GT

When Is Delegated Play Truthful? Within-Range Regret and the Trilemma of Aligned Delegation

Advertisers delegate bidding to autobidders; users delegate tasks to language-model agents. A person describes what they want to an automated proxy that acts in a mechanism on their behalf. This is the revelation principle in production, and it forces a question classical theory assumes away: when is it optimal to describe yourself honestly to your own proxy? We show the answer turns on one quantity, the proxy's within-range regret. The most a principal can gain by misreporting equals the regret of the proxy's honest-report action against those the principal could have steered it to take. Honest self-description is optimal exactly when the proxy already plays the best action it can reach, that is, when it is loyal (Theorem 1). The identity unifies auction-specific autobidding results and pins down when the faithful-communication assumption behind language-model elicitation proxies (Huang et al.) holds. The identity constrains guardrails placed on proxies, from bid caps to a model's alignment layer. No guardrail can be at once binding (it displaces the truthful action from the proxy's best reachable outcome), truthful (honest reporting stays optimal), and capability-preserving (that outcome stays reachable through some report); any two preclude the third (Theorem 2). A safety constraint that alters what a model does while leaving its best output reachable makes honest description of intent suboptimal, so a sharper report can gain. This is the incentive behind prompt-engineering and jailbreaking. Because within-range regret is #P-hard to compute exactly, we estimate it from samples and maintain it as a model is updated, at a cost set by how far the model drifts, not how often it changes. Running it on production language models from five providers under an alignment-style cap, we find honest reporting leaves surplus unclaimed on every model, recovered by inflating the report.

Taksch Dube

Kent State University, Kent, OH, USA

Mechanism Design LLM Alignment LLM Honesty Preference Elicitation

Ask PDF