Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking

Zhexi Feng; Wuxi Chen; Bingrui Zhang

cs.CLAug 26, 2026

Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking

Authors: Zhexi Feng, Wuxi Chen, Bingrui Zhang

Organizations: Department of Electrical and Computer Engineering University of California San Diego, La Jolla, CA, USA

Abstract

Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references. A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection on fixed outputs. Holding 259 beliefs and paired scores fixed, reference recoding lowers weighted prevalence from 0.783 to 0.295 and reverses strictly proper Brier risk: a frozen source-prior rule leads native confidence by 0.227 under reference labels and trails by 0.152 under blinded adjudication, in all six authored scenarios. A reference-only Platt recalibrator reverses further. An ICE-specific reversal appears in a released 301-question NQ-open DPR-BERT pipeline: its average-confidence baseline improves instance-level calibration error by 0.045 under exact match but worsens it by 0.074 under human correctness, with both intervals excluding zero. On independently authored OpenToM narratives, 90-96% of audited unmatched beliefs are literally true and the paired direction again reverses. An exact decomposition attributes the distortion to omitted truths, and a closed-form criterion correctly classifies comparisons from twelve released systems. Frozen-audit retrospective replay shows 50 attempted annotations recover ranking direction with probability at least 0.996. TriSource-Restore anchors full-frame reference labels and frozen automatic judgments to a probability-sampled human pilot, maintains at least nominal coverage, narrows intervals, and repairs confidence subject to a base-rate deployment gate.

Figures & tables

Figure 1: Mechanism, audit, and repair. (A) Fixed outputs receive finite-reference labels $Z$ or blinded full-context truth labels $Y$ . (B) In the observed weak-discrimination regime—DSC is the CORP discrimination component ( Dimitriadis, Gneiting, and Jordan 2021 ) —prevalence crosses the ranking boundary at $p^{\ast}$ , with omitted truths dominating distortion. (C) A stratified pilot estimates the paired gap and escalates if its interval remains open. (D) Human-anchored full-frame $Z$ and frozen $Q$ repair confidence without reference-only recalibration’s compounding error. Algorithm 1 adds a deployment gate; values are population-weighted and belief strings illustrative.

Rule minus native	Exact-match $\Delta$ ICE	Human $\Delta$ ICE
Temperature	$-.090\ [-.121,-.058]$	$+.005\ [-.028,+.037]$
Scale	$-.063\ [-.100,-.026]$	$+.047\ [+.009,+.083]$
Average	$-.045\ [-.087,-.004]$	$+.074\ [+.034,+.114]$

Table 1: A real NQ-open calibration pipeline changes conclusion with the label source (

n=301

; 95% paired bootstrap intervals).

Finite reference : $\Delta_{\mathrm{EG-N}}=-.227\ [-.264,-.188]$
Rule	$E_{w}[Y]/E_{w}[C]$	Brier $\downarrow$	ECE	CIL/rem.
Native	.295/.767	.446	.493	.472/.021
EG	.295/.392	.220	.127	.097/.030
Adjudicated truth : $\Delta_{\mathrm{EG-N}}=+.152\ [+.115,+.193]$
Native	.783/.767	.178	.058	.017/.041
EG	.783/.392	.330	.392	.392/.000

Table 2: Finite references reverse proper-score risk on the same 259 beliefs.

E_{w}[Y]/E_{w}[C]

gives prevalence/mean confidence; CIL/rem. splits ECE into

|E_{w}[C]-E_{w}[Y]|

/within-bin residual, and

\Delta_{\mathrm{EG-N}}

is the paired Brier gap with a 30-store interval.

Protocol	$n$		Nat.	EG	Nat.	EG	Selects
			Brier $\downarrow$		ECE $\downarrow$
Unmatched-as-false	$259$	bel.	$.446$	$.220$	$.493$	$.127$	EG
Matched-only	$1{,}017$	bel.	$.080$	$.354$	$.203$	$.572$	native
Closed-world	$1{,}855$	prop.	$.427$	$.546$	$.477$	$.650$	native
Adjudicated truth	$259$	bel.	$.178$	$.330$	$.058$	$.392$	native

Table 3: Fixed rules, different protocols. Rows one and four share the 259-belief frame; middle rows use distinct scoring frames as protocol controls. Matched-only is IR-style condensed-list scoring ( Sakai and Kando 2008 ) .

Unit	$T_{\rm omit}$ $T_{\rm fp}$	Total $q^{*}$ [CI]
Local	$-.397$ [ $-.439,-.355$ ] $+.019$ [ $+.004,+.034$ ]	$-.379$ [ $-.427,-.330$ ] $.570$ [ $.494,.652$ ]
OpenToM Vanilla	$-.652$ [ $-.686,-.612$ ] $+.000$ [ $+.000,+.000$ ]	$-.652$ [ $-.686,-.612$ ] $.585$ [ $.551,.629$ ]
OpenToM Conservative	$-.657$ [ $-.736,-.575$ ] $+.000$ [ $+.000,+.000$ ]	$-.657$ [ $-.736,-.575$ ] $.654$ [ $.605,.723$ ]

Table 4: Omitted truths dominate label-source distortion. Pairs give

T_{\rm omit}/T_{\rm fp}

and Total/

q^{*}

[CI];

q^{*}

is the audit-conditioned restoration fraction at a tie. Intervals resample stores or narratives.

Unit	RMSE H/T	95% width H/T	Cov.	Width reduction
Local	$.0483/.0309$	$.2146/.1356$	$.959$	$36.8\%$
OpenToM-V	$.0134/.0113$	$.0888/.0833$	$.996$	$6.1\%$
OpenToM-C	$.0212/.0188$	$.1510/.1009$	$.961$	$33.2\%$

Table 5: TriSource-Restore restores rankings and confidence. Panel A reports human-only/TriSource (H/T) RMSE and 95% CI width, TriSource coverage, and width reduction at 50 usable truth labels. Panel B reports held-out Brier risk. V/C denote OpenToM Vanilla/Conservative; full-training-

Y

is an empirical comparator, not an optimization bound.

Probability Calibration

Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.

See all 135 papers

Prefix-Safe Bayesian Belief Tracking for LLM Reasoning Reliability:Separating Calibration from Ranking
May 26, 2026Zhenghan Song, Yunyi Li, Yulong LiuBayesian Filtering Language Model Calibration
Cornell University · Columbia University
Diagnosing the Explicit-to-Applied Theory of Mind Gap in LLMs
Oct 22, 2026Prajvi Saxena, Sabine Janzen, Wolfgang MaassTheory of Mind Language Model Reasoning Evaluation
German Research Center for Artificial Intelligence (DFKI) · Saarland University
Who Knows What - and What Does It Support? Perspective-Bounded Evidence Calibration for Multimodal Theory of Mind
Oct 27, 2026Ziqi Liao, Jiaqi Shen, Yiming Na +3Theory of Mind MLLM Reasoning
Jilin University · National Taiwan University

May 26, 2026J/K move · Enter open · S save

Prefix-Safe Bayesian Belief Tracking for LLM Reasoning Reliability:Separating Calibration from Ranking

Zhenghan Song, Yunyi Li, Yulong Liu

Cornell University · Columbia University

Bayesian Filtering Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Long reasoning traces need reliability estimates before final answers are known. We study prefix-conditioned eventual-success estimation,

P(y=1 \mid o_{1:t})

, using prefix-safe observations. Sequential Bayesian Belief Tracking (SBBT) calibrates observation likelihoods and recursively updates a two-state belief, providing a common tracker for scalar scores, text and self-verification markers, hidden clusters, token-pooling probes, and latent-trajectory features. Across generated open-weight traces on MATH-500, GSM8K, AIME 2025, and RIMO-N, probability quality and ranking separate: score-only SBBT often improves Brier, while AUROC gains require structure-aware evidence beyond strong prefix-safe baselines. In the strongest hard math setting, structure-aware observations reach +0.110 AUROC against standard prefix-safe baselines. Under a same-prefix classifier audit, MATH-500 text markers and RIMO-N self-verification signals remain positive. Together, these findings support SBBT as a calibration-aware online inference framework and expose an evidence regime: scalar scores mainly support probability quality, while structure-aware prefix signals support ranking only when strong prefix-safe baselines have not already absorbed the rank evidence.

Abstract

Probability Calibration

Explore similar work

Prefix-Safe Bayesian Belief Tracking for LLM Reasoning Reliability:Separating Calibration from Ranking

Diagnosing the Explicit-to-Applied Theory of Mind Gap in LLMs

Who Knows What - and What Does It Support? Perspective-Bounded Evidence Calibration for Multimodal Theory of Mind