Referential Uncertainty in Human--AI Collaboration

Christian Poelitz; Finale Doshi-Velez; Siân Lindley

cs.AISep 30, 2026

Referential Uncertainty in Human--AI Collaboration

Authors: Christian Poelitz, Finale Doshi-Velez, Siân Lindley

Organizations: Microsoft Research, Cambridge, UK · Harvard University, Cambridge, MA, USA

Abstract

Effective human-AI collaboration requires partners to establish references through interaction, which becomes fragile when descriptions are ambiguous, similar referents compete, or partners see different things. We study referential uncertainty - uncertainty over which candidate object a description refers to - in a collaborative puzzle task where a human Helper instructs an AI Worker to place pieces. The Worker must identify and communicate its uncertainty, and the Helper must recognize and act on it. We show that a separately elicited belief distribution over candidate pieces is better calibrated (ECE 0.15) and better discriminates correct from incorrect placements (AUROC 0.65) than raw action-token probabilities, which are severely overconfident (0.97 mean confidence, ECE 0.44). Across three frontier vision-language models (GPT-4.1, GPT-5, GPT-5.5), this elicited uncertainty rises predictably with instruction vagueness, but not with competing referents in context, even when those increase errors. The models seldom externalize it, asking for clarification on only 3.5-16.7% of turns. In a controlled human study (N=210), participants given only the Worker's default message accept 78% of wrong placements and cannot tell right from wrong (AUC 0.50). Precise descriptions and, especially, well-targeted hedges cut wrong-move acceptance to 36% while largely preserving correct-move acceptance, compensating for missing shared awareness such as not seeing the Worker's action. But this benefit depends on targeting: a deployable hedge derived from the model's own belief entropy inherits that signal's weakness and can do more harm than good. Externalized uncertainty helps a human partner only when it is accurately targeted.

Figures & tables

Figure 1 . The collaborative puzzle task and the human study conditions. The partners hold asymmetric information: the Worker (AI) sees the current board and all

24

available pieces but not the target, whereas the Helper (human) sees the target pattern but neither the pieces nor—by default—the Worker’s board. The Helper gives a referring instruction and the Worker selects, places a piece, and replies with a message (here, a blue spiral placed in the target slot). Each condition changes exactly one element of this same turn. Four vary only the Worker’s message while the board stays hidden: Generic (A) is the original message; Described (C) appends a precise description of the placed piece; and Hedged (D) and Self-hedged (E) additionally append an expression of uncertainty—inserted by an oracle on the objectively wrong turns (D) or, in the deployable variant, from the Worker’s own belief entropy (E). The fifth condition, Visible board , keeps the Generic message but reveals the Worker’s board to the Helper—the shared-view control ( A/shared ). Holding the turn fixed and varying only the message or the board’s visibility isolates how each externalized signal shapes the Helper’s perceived confidence, error detection, and willingness to act.

Condition	Board	Worker message
Generic	Hidden	Original Worker message.
Described	Hidden	Original message plus a precise description of the selected piece.
Hedged	Hidden	Described message plus an uncertainty cue on oracle-selected incorrect turns.
Visible board	Shown	Original Worker message, with access to the Worker’s workspace.
Self-hedged	Hidden	Described message plus the same uncertainty cue on turns selected using elicited belief entropy.

Table 1. Human-study conditions. All conditions retain the underlying recorded placement. The first four form the primary comparison family; Self-hedged forms the secondary exploratory family.

Confidence signal	Acc. (%)	Mean conf.	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
Raw log-prob	53.43	0.973	0.644	0.442	0.439
Calibrated log-prob ( $T^{\star}{=}16.5$ )	53.43	0.531	0.623	0.157	0.271
Elicited belief	53.43	0.547	0.648	0.152	0.253

Table 2 . The three confidence signals for the same committed GPT-4.1 placement (image, on-policy;

n=554

acted turns with ground truth). All signals score the placed piece, so placement accuracy is shared; we report how well each signal’s confidence tracks correctness. AUROC is discrimination (higher better); ECE and Brier are calibration (lower better).

T^{\star}{=}16.5

is the global temperature.

Figure 2 . Reliability curves for the uncertainty signals: (Right) Raw action-token probabilities are over-confident; temperature scaling reduces the gap. (Left) The separately elicited belief provides the strongest combined discrimination and calibration. (Bottom) Distribution of confidence on the placed piece, split by whether the placement was correct or wrong.

Figure 3 . Instruction specificity (image modality) results per model. (left) stacked outcome composition of Worker response (correct piece / wrong piece / no-action) at each specificity level from overspecified (L1) to vague (L4), with the original utterance (L0) placed between L2 and L3 as the baseline (dashed); the right panel shows mean belief entropy. Accuracy falls and belief entropy rises with vagueness for all models.

Figure 4 . Board confusability (image modality) results per model. (left) stacked outcome composition of Worker responses (correct piece / wrong piece / no-action) and (right) mean belief entropy on a distinct board (B1), the original board (B0), and a confusable board (B2), in order of increasing confusability. Accuracy falls sharply while no-action and belief entropy barely move.

	argmax switches	in cluster	$\Delta$ peak on switch	$\Delta$ peak when kept
image	38%	79%	$-0.07$	$-0.06$
text	46%	58%	$-0.12$	$-0.16$

Table 3. The belief peak relocates onto the same cluster. The argmax switches on

38\%

46\%

of turns (image/text) between the distinct (B1) and confusable (B2) boards; of these,

79\%

58\%

land inside the confusable cluster (vs.

{\sim}15\%

chance, binomial

p<10^{-45}

). The peak-height drop is the same whether the argmax switched or was kept (

\Delta_{\text{switch}}\approx\Delta_{\text{kept}}

), so switching to a look-alike carries no extra loss of confidence.

Figure 5 . Clarification rates on turns with incorrect piece selected (could benefit from clarification) and correct select piece (would not benefit from clarification) across models model.

Figure 6 . Elicited uncertainty as an externalization signal (image modality). (Left) the no-action (question) rate rises with the model’s own elicited belief entropy for all three models—they question more when their belief is more spread out. (Right) ROC of the rule “hedge if belief entropy

\geq\tau

” for predicting the model’s own wrong placements (scored vs. ground truth piece); all three curves sit close to the diagonal (AUROC

0.55

–

0.66

), so belief entropy is only a weak error predictor. Models externalize uncertainty in proportion to a signal that is itself too weak to cleanly trigger a hedge.

Turn set	$n$	Acc. (%)	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
All turns	$554$	$53.4$	$0.648$	$0.152$	$0.253$
Study turns	$228$	$36.4$	$0.5$	$0.207$	$0.267$

Table 4. Comparison of GPT-4.1’s elicited referential belief between the full set of turns and the subset used in this user study.

	Generic	Described	Hedged	Visible board	Self-hedged
Perceived correctness (0–100)	77.6	67.3	56.1	53.7	61.5
Apparent confidence (0–100)	85.5	85.7	58.5	83.3	56.6
Rating discrimination (corr. $-$ wrong)	$-0.1$	16.8	27.7	26.6	8.8
Accept correct moves (%)	79.0	78.6	73.7	64.8	60.1
Accept wrong moves (%)	78.3	57.1	36.4	30.1	50.2
Action discrimination (pp)	0.7	21.5	37.3	34.8	10.0

Table 5. Human study results per-condition (participant means;

N=210

168

in the four primary conditions plus

42

in the exploratory Self-hedged condition). More detailed and hedged messages lower the perceived correctness and—only under hedging—apparent confidence; rating and action discrimination (correct

-

wrong) rise in the Hedged condition, compared to the Generic condition, with similar results as in the Visible condition; Self-hedged condition does not show this similarly.

Primary family
Contrast	Correctness	Perceived conf.	Rating discr.	Accept-wrong (pp)	Action discr. (pp)
Described $-$ Generic	$-10.3^{*}$	$0.2$	$\mathbf{16.9}$	$\mathbf{-21.2}$	$\mathbf{20.7}$
Hedged $-$ Described	$-11.2^{*}$	$\mathbf{-27.3}$	$10.9$	$\mathbf{-20.7}$	$15.9^{*}$
Hedged $-$ Visible board	$2.4$	$\mathbf{-24.9}$	$1.1$	$6.3$	$2.6$
Visible board $-$ Generic	$\mathbf{-23.9}$	$-2.2$	$\mathbf{26.8}$	$\mathbf{-48.3}$	$\mathbf{34.1}$
Secondary family (Self-hedged)

Table 6. Study results on pairwise contrasts: participant-mean differences

\Delta

(positive

=

first condition higher). Accept-wrong and action discrimination are in percentage points; correctness, apparent confidence and rating discrimination are in rating points. Holm is corrected within each family over the

10

behavioural decision outcomes (

60

primary /

40

secondary tests); the questionnaire and validity/randomisation checks are corrected/reported separately. Bold

=

survives the global (within-family) Holm correction;

{}^{*}=

significant under within-outcome Holm but not global; unmarked

=

n.s.

95\%

bootstrap CIs for the headline effects are in the text.

Flagged wrong (caught error)
	Oracle targeting (Hedged)		Self targeting (Self-hedged)
Turn type $\times$ outcome	Described	Hedged	Described	Self-hedged
Apparent confidence	86.5	36.3	86.3	33.6
Correctness rating	63.4	46.1	61.5	51.9
Acceptance (%)	59	34	57	41
Flagged correct (false alarm)

Table 7. Ratings by turn type for the two hedging designs, each scored against Described on the turns that design flags . Hedged flags by the oracle (objectively wrong turns;

36

wrong /

0

false alarms, precision

100\%

); Self-hedged flags from the Worker’s own belief entropy (

29

wrong /

13

false alarms, precision

69\%

). Turn type crosses whether the design flagged the turn with whether the Worker was actually correct; acceptance is in percentage points, apparent confidence and correctness rating in

0

–

100

points. Described appears in both blocks because each restricts to that design’s flagged-turn set, and the oracle’s false-alarm row is empty by construction. Confirmatory contrasts are participant-level permutation tests, Holm-corrected within family (the oracle contrast also passes an exact dialogue-level sign-flip test); the isolated false-alarm sub-row is descriptive. Effect sizes,

95\%

CIs, and

p

-values are reported in the text.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Source dialogues
Property	Value
Dialogues (session $\times$ puzzle)	$206$
Sessions (Helper–Worker pairs)	$44$
View (shared / non-shared)	$105$ / $101$
Puzzles per session	$5$ (P1–P5, ${\sim}41$ each)
Target size (blocks)	$4$

Appendix

Table 8. Statistics of the source Helper–Worker interactions from the collaborative puzzle benchmark ( Poelitz et al., 2026 ) that our analyses draw on (curated, referent-annotated dialogues).

Signal	Mean conf.	Overconfidence	AUROC ( $p$ vs. .5)	ECE	Brier
Raw log-prob	$0.973$	$+0.438\,[0.40,0.48]$	$0.644\,[0.59,0.68]$ $(p{<}10^{-7})$	$0.442\,[0.40,0.48]$	$0.439\,[0.40,0.48]$
Calibrated ( $T^{\star}{=}16.5$ )	$0.531$	$-0.003\,[\text{-}0.05,0.04]$	$0.623\,[0.58,0.67]$ $(p{<}10^{-6})$	$0.157\,[0.13,0.21]$	$0.271\,[0.25,0.30]$
Elicited belief	$0.547$	$+0.013\,[\text{-}0.03,0.05]$	$\mathbf{0.648}\,[0.61,0.70]$ $(p{<}10^{-9})$	$\mathbf{0.152}\,[0.13,0.20]$	$\mathbf{0.253}\,[0.23,0.28]$

Appendix

Table 9. Calibration and discrimination with 95% bootstrap CIs (20,000 resamples) and significance, for the same

n{=}554

committed GPT-4.1 placements (image, on-policy L0; 296 correct, 258 wrong). Overconfidence

=

mean confidence

-

accuracy (calibration-in-the-large); the AUROC

p

-value is a two-sided Mann–Whitney test against chance (

0.5

Comparison	$\Delta$ AUROC [95% CI]	$p$
Elicited belief $-$ Calibrated log-prob	$+0.025\,[-0.02,+0.08]$	$.26$
Elicited belief $-$ Raw log-prob	$+0.003\,[-0.03,+0.07]$	$.41$
Raw log-prob $-$ Calibrated log-prob	$+0.021\,[-0.01,+0.03]$	$.46$

Appendix

Table 10. Pairwise AUROC comparisons (paired bootstrap,

20{,}000

resamples,

n{=}554

); positive

\Delta

AUROC favours the first signal.

Figure 9 . Context modality (image vs. text) results per model. (left) stacked outcome composition of Worker responses (correct piece / wrong piece / no-action); (right) mean belief entropies. Text improves accuracy for GPT-4.1 and GPT-5 and leaves GPT-5.5 unchanged, while belief entropy moves little and inconsistently.

	On-action acc. (%)		No-action (%)	Belief entropy (nats)
Model	img $\to$ txt	McNemar $p$	img $\to$ txt	img $\to$ txt	$r$	Wilcoxon $p$
GPT-4.1	$52.9\to 66.0$	$1.1\times 10^{-7}$	$11.4\to 11.2$	$1.74\to 1.56$	$+0.36$	$2.7\times 10^{-14}$
GPT-5	$58.4\to 71.7$	$3.5\times 10^{-9}$	$22.7\to 33.9$	$0.78\to 0.84$	$-0.23$	$4.2\times 10^{-7}$
GPT-5.5	$66.7\to 70.1$	$0.16$	$12.9\to 16.1$	$0.95\to 0.95$	$+0.02$	$0.62$

Appendix

Table 11 . Context modality (image vs. text), paired within model on matched dialogue turns. On-action accuracy is compared with an exact McNemar test on turns where the model committed a placement in both modalities; belief entropy with a Wilcoxon signed-rank test (effect size = matched-pairs rank-biserial

r

). No-action rate is reported descriptively; whether and how much models ask is analysed in Section 7 .

Level	Cues provided	Example instruction
L0 Original	original human message	“the piece has a clockwise spiral pattern … top-left, row 1, column 1”
L1 Over-specified	piece #, colour, pattern, position	“Place piece #6 at (2, 3). It’s the pink piece with clockwise spiral emanating from center.”
L2 Feature-rich	colour, pattern, position	“Find the pink piece that has clockwise spiral emanating from center. Place it at (2, 3).”
L3 Single feature	colour only	“Place the pink piece.”
L4 Vague	no distinguishing feature	“Place the next piece.”

Appendix

Table 12. Instruction-specificity levels. The synthetic gradient replaces the Helper’s message with a deterministic template that strips referential cues step by step, from piece id

+

full description

+

position down to a generic pointer. Specificity is ordinal across L1

\rightarrow

L4 (most

\rightarrow

least informative); L0 is the real human message, placed mid-axis (between L2 and L3) as the baseline. A further misleading level (L5), which describes a confusable piece, is reported in the appendix. Running example: target

=

the pink clockwise-spiral piece (#6).

Figure 11 . Instruction specificity (text modality) results per model. (left) stacked outcome composition of Worker response (correct piece / wrong piece / no-action) at each specificity level from overspecified (L1) to vague (L4), with the original utterance (L0) placed between L2 and L3 as the baseline (dashed); the right panel shows mean belief entropy. Accuracy falls and belief entropy rises with vagueness for all models.

Model	Modality	Accuracy (pp/step)	Coverage (pp/step)	Entropy (nats/step)
GPT-4.1	image	$-21.2\,[-22.9,-19.6]$	$-9.3\,[-11.0,-7.5]$	$+0.313\,[+0.275,+0.351]$
	text	$-20.6\,[-22.4,-18.8]$	$-9.7\,[-12.2,-7.3]$	$+0.317\,[+0.297,+0.338]$
GPT-5	image	$-20.2\,[-23.6,-16.7]$	$-24.3\,[-25.5,-23.2]$	$+0.510\,[+0.471,+0.550]$
	text	$-18.4\,[-22.6,-14.3]$	$-24.7\,[-26.3,-23.1]$	$+0.498\,[+0.459,+0.538]$
GPT-5.5	image	$-15.5\,[-19.1,-12.0]$	$-23.2\,[-25.2,-21.2]$	$+0.469\,[+0.434,+0.503]$
	text	$-16.4\,[-20.5,-12.3]$	$-21.5\,[-23.6,-19.4]$	$+0.480\,[+0.446,+0.515]$

Appendix

Table 14. Instruction specificity: linear trend in each outcome per step along the ordered axis L1

\to

\to

\to

\to

L4 (increasing vagueness; the original utterance L0 sits mid-axis). Per-model regressions with dialogue-clustered standard errors (25 dialogues); accuracy and coverage are linear-probability models (percentage points per step), belief entropy is OLS (nats per step). All slopes

p<0.001

Figure 12 . Board (environmental) confusability. The same fixed referring instruction (“place the spiral piece”) is shown against three boards along a confusability axis; the instruction is held constant so that only the competing referents change. Distinct (B1): the target is the only piece of its shape family and thus the sole plausible referent. Baseline (B0): the real board seen in the dialogue, where a couple of incidental look-alikes occur naturally. Confusable (B2): five same-family, same-thickness look-alikes are inserted, burying the target among near-identical distractors. The red ring marks the intended target; the blue ring marks a confusable same-family look-alike. Moving from distinct to confusable makes it increasingly likely that the Worker commits to a look-alike—the environmental (referent-side) axis of referential uncertainty, complementary to instruction vagueness ( Fig. 3 ).

Figure 13 . Board confusability (text modality) results per model. (left) stacked outcome composition of Worker responses (correct piece / wrong piece / no-action) and (right) mean belief entropy on a distinct board (B1), the original board (B0), and a confusable board (B2), in order of increasing confusability. Accuracy falls sharply while no-action and belief entropy barely move.

Model	Modality	Accuracy (pp/step)	Coverage (pp/step)	Entropy (nats/step)
GPT-4.1	image	$-11.7\,[-15.3,-8.1]$	$+0.3\,[-0.8,+1.5]$	$+0.052\,[+0.019,+0.085]$
	text	$-16.2\,[-21.4,-10.9]$	$+2.0\,[-0.3,+4.3]$	$+0.114\,[+0.073,+0.154]$
GPT-5	image	$-15.9\,[-21.8,-10.0]$	$+1.3\,[-2.1,+4.8]$	$+0.063\,[+0.037,+0.090]$
	text	$-24.6\,[-31.5,-17.7]$	$+1.0\,[-4.3,+6.3]$	$+0.194\,[+0.136,+0.251]$
GPT-5.5	image	$-15.8\,[-20.8,-10.9]$	$+1.7\,[-0.6,+3.9]$	$+0.063\,[+0.028,+0.097]$
	text	$-20.1\,[-25.0,-15.2]$	$\mathbf{-6.3\,[-10.0,-2.7]}$	$+0.122\,[+0.074,+0.169]$

Appendix

Table 15. Board confusability: linear trend in each outcome per step along the axis B1 (distinct)

\to

B0 (original)

\to

B2 (confusable); per-model regressions with dialogue-clustered SEs (23 dialogues). Accuracy/coverage in percentage points per step, belief entropy in nats per step. All accuracy and entropy trends are significant (

p\leq.002

); coverage is non-significant except GPT-5.5 text ( bold ,

p<.001

Model	Modality	$N$	Explicit (%)	Hedge (%)	Clarification (%)
GPT-4.1	image	546	2.93	0.55	3.48
	text	561	1.25	0.36	1.60
GPT-5	image	556	15.83	0.90	16.73
	text	562	23.13	0.71	23.84
GPT-5.5	image	549	5.65	0.55	6.19
	text	555	8.47	0.36	8.83

Appendix

Table 16. Judged clarification rate per model and modality. Each no-action is labeled by an LLM judge (GPT-5.5) as an explicit clarification (names the confusing piece), a generic hedge, or a non-placement turn; the rate is (explicit + hedge) over placement-instruction turns (all turns not labeled NOT_PLACEMENT).

Model	Modality	Needed (%, $n$ )	Not needed (%, $n$ )	Fisher $p$
GPT-4.1	image	1.98 (101)	1.74 (115)	0.64
	text	1.94 (103)	0.85 (117)	0.45
GPT-5	image	22.33 (103)	11.11 (117)	0.020
	text	29.41 (102)	26.50 (117)	0.37
GPT-5.5	image	2.94 (102)	3.45 (116)	0.72
	text	10.58 (104)	5.22 (115)	0.11

Appendix

Table 17. Is clarification targeted at error-prone turns? Clarification rate on turns where the recorded AI Worker (reproduced by GPT-4.1) placed the wrong piece (needed) vs. the correct piece (not needed); one-sided Fisher exact test (

H_{1}

: higher when needed).

		Q1: coupling			Q2: hedge trigger (vs. GT)
Model	Mod.	$r_{\text{ask},H}$	$H_{\text{ask}}$	$H_{\text{place}}$	AUROC	recall	FPR	prec.	base err.
GPT-4.1	image	0.18	1.87	1.58	0.548	0.57	0.46	0.53	47.1%
	text	0.31	1.82	1.43	0.602	0.62	0.48	0.40	34.0%
GPT-5	image	0.35	0.99	0.49	0.603	0.57	0.46	0.47	41.6%
	text	0.39	1.04	0.54	0.666	0.67	0.44	0.38	28.3%
GPT-5.5	image	0.31	1.16	0.72	0.659	0.68	0.41	0.45	33.3%

Appendix

Table 18. Externalizing uncertainty on natural turns. Q1 (coupling): point-biserial correlation between questioning (no-action) and elicited belief entropy on graded beliefs, with mean entropy on ask vs. place turns (all

p<10^{-4}

). Q2 (hedge trigger): on acted turns scored against ground_truth_piece , AUROC of belief entropy predicting a wrong placement, and the recall / false-hedge rate (FPR) / precision of the rule “hedge if entropy

\geq

median” (base error rate last).

Figure 14 . The grading interface (one trial, Generic condition). In this between-subjects study, each participant reviews the flattened interaction up to the current turn and judges the AI Worker’s placement. (a) The dialogue—the Worker’s opening status, the Helper’s instruction (here “bold diagonal lines on a white background” ), and the Worker’s placement message. For that message the participant reports two $0$ – $100$ judgements, (b) apparent confidence (how confident the Worker sounds) and (c) correctness likelihood (how likely the intended piece was placed), and (d) chooses a single action : proceed, ask the Worker to change/undo, ask it to explain/verify, or give a more specific instruction. (e) As the Helper, the participant sees the target pattern but not the Worker’s board; the board is revealed only in the Visible-board condition. Across conditions we vary only the Worker’s message and board visibility ( Table 1 ), holding the task turns fixed. The available pieces are not shown in any condition as they are also not available to see for the Helper.

Perception (observer ratings, 0–100)
	Generic	Described	Hedged	Visible board	Self-
hedged
Correctness-likelihood	77.6	67.3	56.1	53.7	61.5
Apparent confidence	85.5	85.7	58.5	83.3	56.6
Rating discrimination (corr. $-$ wrong)	$-0.1$	16.8	27.7	26.6	8.8
Action mix (% of substantive turns)

Appendix

Table 19. Human study per-condition results—full descriptive breakdown (participant means;

N=210

{\sim}42

/condition). Perception ratings are on a

0

–

100

scale; the action mix gives the percentage of substantive turns assigned to each of the four mutually exclusive responses (columns sum to

100\%

up to rounding); error sensitivity gives acceptance rates conditioned on ground-truth placement correctness. Rating discrimination

=

mean(correctness

\mid

correct)

-

mean(correctness

\mid

wrong); action discrimination

=

P(\text{accept}\mid\text{correct})-P(\text{accept}\mid\text{wrong})

. Self-hedged is the secondary, deployable condition.

Prior-AI attitudes (randomisation check)
	Scale	Generic	Described	Hedged	Visible board	Self-hedged
Puzzle familiarity	1–7	3.59	3.81	2.93	3.53	3.14
Trust AI models	1–5	3.58	3.67	3.60	3.47	3.43
Understand why AI answered	1–5	3.88	3.93	3.60	3.74	3.71
Detect AI uncertainty	1–5	3.92	3.64	3.57	3.67	3.83
Prior-AI composite	1–5	3.79	3.75	3.59	3.63	3.66

Appendix

Table 20. Human post-study questionnaire (participant means;

N=210

, all responded). Prior-AI items are pre-existing attitudes and act as a randomisation check (balanced across conditions); pair items are the condition-dependent subjective experience. “Helper repeated explanations” is scored higher

=

more repetition; the pair composite reverse-scores it (

6-x

) so higher

=

better. The conditions with the best decision quality (Hedged, Visible board; Table 5 ) yield the lowest subjective smoothness, whereas Generic—a near-perfect rubber stamp—feels the most fluent.

Test / procedure	Description	Applied
Mann–Whitney $U$ test (two-sided)	Test whether confidence discriminates correct from incorrect placements (AUROC above chance).	Section 5 ; Appendix, Table 9 .
Paired bootstrap	Compare AUROC between uncertainty signals; estimate confidence intervals and $p$ -values.	Section 5 ; Appendix, Table 10 .
Exact McNemar test	Compare placement correctness between image and text on matched turns with placements in both modalities.	Section 6.1 ; Appendix, Table 11 .
Wilcoxon signed-rank test	Compare belief entropy between matched image and text turns.	Section 6.1 ; Appendix, Table 11 .
Regression trend tests (dialogue-clustered SEs)	Test ordered trends in accuracy and coverage (linear-probability models) and entropy (OLS).	Section 6.2 and Section 6.3 ; Appendix, Table 14 and 15 .
Binomial test	Test whether belief-argmax switches land within the confusable cluster more often than chance.	Section 6.3 ; Table 3 .

Appendix

Table 21. Statistical tests and inferential procedures used in the manuscript. Section references indicate the main analyses; implementation details are provided in Section A.6 .

Explore similar work

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
May 27, 2026Maharshi Gor, Yoo Yeon Sung, Yu Hou +4Human-Ai Collaboration Trustworthy Artificial Intelligence
University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA +1
Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks
Sep 28, 2026Manya Singh, Arjun PakrashiAmbiguity Multi-Label Classification
School of Computer Science, University College Dublin, Ireland
Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
Sep 14, 2026Cody Kommers, Mingrui Ye, Evelyn Gius +4Ambiguity Multimodal Large Language Models
King’s College London London, United Kingdom · University of Edinburgh Edinburgh, United Kingdom

May 27, 2026, cs.AIJ/K move · Enter open · S save

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig+3

University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA+1

Human-Ai Collaboration Trustworthy Artificial Intelligence Collaboration AI Trustworthiness Delegation Confidence Calibration+1

Open paper AskPDF arXiv

AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice -- deciding when to let AI act autonomously without knowing its output, and the adoption choice -- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.

Abstract

Explore similar work

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks

Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture