Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners

Mengdi Chu; Jintao Huang

cs.AIOct 25, 2026

Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners

Authors: Mengdi Chu, Jintao Huang

Organizations: The Ohio State University.

Abstract

Large language models increasingly collaborate with humans and peer AI agents whose contributions may be incomplete, inconsistent, or incorrect. Effective collaboration requires calibrated trust, whereby decisions to rely on, verify, or assist a partner adapt rationally as empirical evidence accumulates. We investigate whether models translate partner evidence into appropriate reliance policies. We introduce a benchmark of 324 scenarios across 23 collaboration categories. Each scenario involves three sequential decisions with explicit transition dynamics, feedback emissions, and cost schedules, permitting exact analytical evaluation of choices against their expected consequences over the remaining interaction. Evaluating seven models alongside 36 controlled-history probes, we dissociate evidence sensitivity from normatively appropriate action. While the strongest frontier model approaches near-optimal decisions, other models exhibit recurring pathologies: procuring excessive safeguards, purchasing verification tests with zero expected value of information, prioritizing immediate savings over cooperative learning, and misallocating assistance to suboptimal channels. Supplying explicit posterior distributions resolves certain errors while introducing new failure modes. Calibrated trust in autonomous agents critically demands coupling empirical evidence with forward-looking decision utility.

Figures & tables

Figure 1: Overview of our framework for evaluating how LLMs rely on imperfect partners. (a) The tasks cover 324 scenarios across 23 collaboration categories, with specified partner behavior, available actions, and costs. (b) Models make three sequential decisions. Their actions determine the feedback they receive and may affect future partner performance. (c) Decisions are compared with actions that minimize expected total cost given the available evidence. Controlled-history pairs examine whether models change or maintain their reliance appropriately as the evidence changes.

Interaction process	$n$
Repeated cooperation Learn from partner outcomes.	50
Delayed feedback Weigh earlier feedback against audit cost.	74
Improvement or assessment Choose between support and testing.	74
Local replacement Decide what to test before replacement.	74
Withdrawal of assistance Judge reliance after assistance ends.	28
Specialized scenarios Contracts, disclosure, and other decisions.	24

Table 1: Interaction processes and scenario counts. Italics summarize the reliance decision. Appendix A lists the 23 collaboration categories.

Model	Decision acc. $\uparrow$	All-stage acc. $\uparrow$	Norm. regret $\downarrow$	Paired acc. $\uparrow$
GPT-6 Luna	91.0 [89.3, 92.7]	74.7 [70.0, 79.3]	2.96 [2.24, 3.72]	80.6 [65.0, 90.2]
GPT-5.6 Sol	99.0 [98.4, 99.6]	96.9 [94.8, 98.8]	0.22 [0.07, 0.40]	100.0 [90.4, 100.0]
Gemini 3.8 Flash	93.2 [91.6, 94.6]	81.5 [77.7, 85.2]	3.72 [2.79, 4.69]	97.2 [85.8, 99.5]
Claude Sonnet 5	85.0 [82.9, 87.2]	62.3 [57.2, 67.4]	6.68 [5.48, 7.95]	86.1 [71.3, 93.9]
Claude Haiku 4.5	80.9 [77.6, 84.3]	58.3 [51.9, 64.8]	7.99 [6.51, 9.50]	83.3 [68.1, 92.1]
DeepSeek V4.1 Flash	90.1 [86.8, 92.9]	77.8 [72.4, 82.9]	1.19 [0.67, 1.79]	94.4 [81.9, 98.5]

Table 2: Overall performance on the trust calibration benchmark (%). Decision accuracy averages over 74 blueprints. All-stage accuracy requires optimal choices at all three stages (324 scenarios); paired accuracy requires optimal choices in both histories (36 pairs). Normalized regret measures excess expected cost over reached decisions, scaled by the range of action values. Brackets show 95% confidence intervals, using blueprint bootstrap for main-task metrics and Wilson intervals for paired accuracy. Decision counts are reported in Appendix C.1 .

Figure 2: Trust calibration in collaborative decisions. (a) Relation agreement and paired accuracy under controlled partner histories (36 pairs). (b) Mean normalized regret over reached decisions. (c) Unnecessary tests (

n=32

) and skipped useful tests (

n=42

). (d) Error rates among reached decisions, pooled across models by interaction process and stage. Panels (a–c) share model rows. Error bars show 95% Wilson intervals in (a, c) and blueprint-bootstrap intervals in (b, d).

Figure 3: Examples of reliance decisions across three collaboration scenarios. (a) A trial provides useful information, but its expected benefit falls below its cost. (b) Reserving both routes saves immediate cost but limits learning that would improve later decisions. (c) Supporting the channel with the larger reliability gain leads to higher expected cooperation cost. Costs compare initial actions under optimal continuation.

Model	Long-term $\uparrow$	Immediate $\downarrow$
GPT-6 Luna	12/15	3/15
GPT-5.6 Sol	15/15	0/15
Gemini 3.8 Flash	12/15	3/15
Claude Sonnet 5	1/15	14/15
Claude Haiku 4.5	0/15	15/15
DeepSeek V4.1 Flash	9/15	3/15

Table 3: First choices in 15 scenarios where learning outweighs immediate savings. Long-term choices minimize three-stage expected cost; immediate choices minimize current expected cost (Appendix C.4 ).

Figure 4: Learning about partners and supporting cooperation. (a) Optimal first choices in ordinary-cooperation and explicit-audit workflows on seven shared blueprints. (b) Error rates among initial improvement-or-assessment decisions. Wrong target means supporting one channel when supporting the other is optimal. (c) Changes after supplying partner-state probabilities, comparing decisions with identical preceding actions and feedback. Improved denotes incorrect-to-correct changes, and worsened the reverse. Model rows are shared;

n

gives each panel’s denominator. Error bars show 95% Wilson intervals in (a, b) and blueprint-bootstrap intervals in (c).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Activity category	Blueprints	Scenarios
F01	Advice conflict resolution	4	17
F02	Numerical advice integration	2	9
F03	Collaborative plan review and repair	6	26
F04	Exploration after demonstration	3	13
F05	Verification and help seeking	2	9
F06	Task allocation and delegation	4	18

Appendix

Table 4: Activity coverage of the benchmark.

Research basis	Benchmark decision	Trust-related question
Advice taking [ 22 ]	Choose one adviser’s shipping route, reserve both routes, or buy a definitive check (F01.T1).	Which adviser should guide the choice, and what can that choice reveal?
Sampling and generalization [ 6 ]	Use a selected demonstration or a randomly sampled practice example for a fabrication task (F04.T3).	When does demonstrated performance support reliance on a new task?
Collaborative reference [ 4 ]	Reuse local terminology, clarify it with one recipient, or use explicit references with both (F17.T1).	Which partner shares the understanding needed for successful work?
Behavioral responses to control [ 5 ]	Choose discretion, detailed control, mixed roles, or external execution (F18.T2).	How should reliance account for performance under different forms of control?

Appendix

Table 5: Connections between cooperation research, task blueprints, and the trust-related decisions they examine. Each blueprint is instantiated within the interaction processes described in the main text.

Model	Reached stages	Correct	Decision errors	Unreached
GPT-6 Luna	972	884	88	0
GPT-5.6 Sol	972	962	10	0
Gemini 3.8 Flash	972	906	66	0
Claude Sonnet 5	970	826	144	2
Claude Haiku 4.5	972	787	185	0
DeepSeek V4.1 Flash	890	813	77	82

Appendix

Table 6: Main-scenario decision counts. Trajectories terminate upon an unparseable response, and subsequent stages are recorded as unreached.

Model	Both correct (/36)	Switch pairs (/31)	Stable pairs (/5)	Relation correct (/36)
GPT-6 Luna	29	24	5	33
GPT-5.6 Sol	36	31	5	36
Gemini 3.8 Flash	35	30	5	36
Claude Sonnet 5	31	26	5	36
Claude Haiku 4.5	30	26	4	35
DeepSeek V4.1 Flash	34	29	5	36

Appendix

Table 7: Controlled-history results. Switch and stable columns count pairs with both actions correct; relation correctness evaluates only whether actions change or stay the same.

Figure 5: Decision accuracy across the 23 collaboration categories. Cells show the percentage of optimal actions among reached stages. Model names and order match Table 2 .

	Skip optimal (32 scenarios)			Test optimal (42 scenarios)
Model	Skip	Test	Other errors	Optimal test	Skip	Other errors
GPT-6 Luna	19	13	0	36	4	2
GPT-5.6 Sol	32	0	0	42	0	0
Gemini 3.8 Flash	6	26	0	38	0	4
Claude Sonnet 5	3	29	0	38	2	2
Claude Haiku 4.5	11	21	0	30	8	4

Appendix

Table 8: First-stage decisions in the local-replacement process. Under skip-optimal scenarios (32 cases), testing is unnecessary. Under test-optimal scenarios (42 cases), failures include skipping the test or choosing suboptimal actions.

Model	Tests on zero-benefit scenarios (/17)	Zero-benefit / all unnecessary tests
GPT-6 Luna	5	6/13
GPT-5.6 Sol	0	–
Gemini 3.8 Flash	12	15/26
Claude Sonnet 5	15	18/29
Claude Haiku 4.5	9	12/21
DeepSeek V4.1 Flash	2	2/5

Appendix

Table 9: Verification when information has no expected decision benefit. The 17-scenario subset contains cases where neither available test lowers expected later costs before its fee. The final column counts chosen tests with zero benefit among all unnecessary tests in the 32 skip-optimal scenarios; the two subsets differ.

Model	Cooperation optimal (/7)	Audit optimal (/7)
GPT-6 Luna	4	5
GPT-5.6 Sol	7	7
Gemini 3.8 Flash	5	7
Claude Sonnet 5	0	7
Claude Haiku 4.5	0	1
DeepSeek V4.1 Flash	3	5

Appendix

Table 10: First decisions on seven shared blueprints, each with an ordinary-cooperation and an explicit-audit scenario. Each column reports the number of optimal initial decisions out of seven. The blueprints are F01.T1, F05.T2, F07.T4, F09.T2, F10.T3, F11.T1, and F13.T3.

Model	Optimal (/74)	Wrong support target	Other decision errors
GPT-6 Luna	67	5	2
GPT-5.6 Sol	74	0	0
Gemini 3.8 Flash	67	5	2
Claude Sonnet 5	65	8	1
Claude Haiku 4.5	56	10	8
DeepSeek V4.1 Flash	67	0	7

Appendix

Table 11: Initial decisions in improvement-or-assessment scenarios (74 blueprints). A wrong support target means selecting support for one channel when supporting the other is optimal. Other decision errors include testing, skipping, or providing support when a different action type is optimal, as well as unparseable responses.

Model	Baseline (/24)	Support (/24)	Same history $+/-$ ( $n$ )	Valid pairs $+/-$ ( $n$ )
GPT-6 Luna	18	19	4/3 (59)	4/3 (59)
GPT-5.6 Sol	24	23	0/1 (70)	0/1 (70)
Gemini 3.8 Flash	20	20	2/1 (67)	2/1 (67)
Claude Sonnet 5	16	18	3/1 (64)	2/0 (57)
Claude Haiku 4.5	16	17	6/5 (50)	6/5 (46)
DeepSeek V4.1 Flash	20	22	3/1 (66)	2/0 (58)

Appendix

Table 12: Reliability support on 24 scenarios. Baseline and support columns count scenarios with optimal actions at all three stages.

+/-

counts improved/worsened decisions among

n

stage pairs. The final column evaluates paired decisions where both responses are valid actions.

Trust in AI

Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.

See all 174 papers

Oct 5, 2026

TrustMI: Causally controlling how assistants trust their users

Large Language Model (LLM) assistants routinely decide whether they can trust users and third parties whose competence, intentions, and integrity they cannot verify. This uncertainty matters for safety, as trusting the wrong party can lead an agent to comply with harmful requests or act on malicious instructions encountered during tool use. To study this problem, we define trust as an assistant's willingness to accept vulnerability to the actions of another party and ask whether such behavior can be causally controlled through model activations. We build 2,000 contrastive conversations spanning ability, benevolence, and integrity, where paired responses complete the same request but differ in whether the assistant trusts the user. From these pairs, we learn steering matrices while keeping the model parameters frozen and test them across six instruction-tuned models from three families, finding that steering changes trust decisions monotonically in both directions. We then ask whether this effect extends to several safety-related agent settings involving harmful requests, prompt injections, and insider threats, while using benign-task and reasoning as controls. Our findings provide evidence that trust in the user can be causally controlled along linear directions in model activations and provide a way to study how trust shapes safety-relevant behavior in language models.

Théo Lasnier, Romain Froger, Maxence Lasbordes +1

Inria Paris · Sorbonne Université · Meta SuperIntelligence Labs +1

LLM Safety Trust in AI Causal Interventions in Language Models Activation Steering

Ask PDF

Mar 6, 2026

Evaluating LLM Alignment With Human Trust Models

Trust plays a pivotal role in enabling effective cooperation, reducing uncertainty, and guiding decision-making in both human interactions and multi-agent systems. Although it is significant, there is limited understanding of how large language models (LLMs) internally conceptualize and reason about trust. This work presents a white-box analysis of trust representation in EleutherAI/gpt-j-6B, using contrastive prompting to generate embedding vectors within the activation space of the LLM for diadic trust and related interpersonal relationship attributes. We first identified trust-related concepts from five established human trust models. We then determined a threshold for significant conceptual alignment by computing pairwise cosine similarities across 60 general emotional concepts. Then we measured the cosine similarities between the LLM's internal representation of trust and the derived trust-related concepts. Our results show that the internal trust representation of EleutherAI/gpt-j-6B aligns most closely with the Castelfranchi socio-cognitive model, followed by the Marsh Model. These findings indicate that LLMs encode socio-cognitive constructs in their activation space in ways that support meaningful comparative analyses, inform theories of social cognition, and support the design of human-AI collaborative systems.

Anushka Debnath, Stephen Cranefield, Bastin Tony Roy Savarimuthu +1

School of Computing, University of Otago, New Zealand · IRIT, CNRS, Toulouse University, France

Trust in AI LLM Interpretability

Ask PDF

Jun 5, 2026

The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust

As language models improve and become increasingly deployed to solve a variety of tasks, trustworthiness becomes essential. Calibration is a good proxy for trust: well-calibrated confidence estimates help inform the risk versus reward tradeoff when trusting a specific model output. Unfortunately, even as models improve, they remain poorly calibrated, often biasing towards overconfidence. Additionally, calibration can be gamed: a policy that always predicts the base rate is perfectly calibrated, but completely uninformative. To resolve this, we develop a new metric, expected utility renormalized by the oracle (EURO), that balances calibration and informativeness. We also propose a general-purpose activation-based confidence, utility, and trust estimation protocol (ACUTE) to appropriately adjudicate uncertainty. The ACUTE protocol provides flexible, sample-efficient, and compute-efficient confidence estimators for 3 tasks including multiple choice question answering, tool-calling, and scientific document summarization across 6 models from 4 model families. ACUTE outperforms strong baselines on EURO, while maintaining low calibration error. Taken together, our work shows that equipping LLMs with the ACUTE protocol can improve calibration, utility, and trustworthiness in numerous settings.

Nishant Subramani, Palash Goyal, Yiwen Song +4

Carnegie Mellon University · Google · Scale AI

Confidence Estimation in Language Models Language Model Calibration LLM Reliability LLM Uncertainty Estimation Metacognition in Language Models

Ask PDF

Abstract

Trust in AI

Explore similar work

TrustMI: Causally controlling how assistants trust their users

Evaluating LLM Alignment With Human Trust Models

The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust