Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

Shashwat Pandey; Satwik Pandey; Suresh Raghu

cs.SEAug 24, 2026

Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

Authors: Shashwat Pandey, Satwik Pandey, Suresh Raghu

Organizations: University of California, Santa Cruz · Independent Researcher

Abstract

Aligning deployed language models requires knowing when their outputs can be trusted, yet on-device models now ship to hundreds of millions of devices with no server-side moderation, and the configuration developers can actually deploy is rarely audited independently. We present a reproducible reliability audit of the developer-accessible on-device foundation model, framed as an oversight question: can a user or a resource-constrained developer tell when the model is wrong? Red-teaming it on calibration, confident confabulation on false-premise questions, and over-refusal of benign prompts, we find a task-asymmetric miscalibration: its guardrails fail in opposite directions across tasks (confabulating on 69% of false premises while refusing 18% of entirely benign inputs), atop a self-reported confidence that is saturated and non-discriminative (AUROC 0.47; ECE 70, worst among comparable small models). Crucially, confident-correct and confident-wrong outputs are surface-indistinguishable: a classifier over 15 user-visible features separates them at AUROC only 0.55 (equivalence-confirmed), leaving no signal for oversight at inference time. No cheap single-generation signal flags these failures ( $\le$ 0.68 AUROC), whereas a black-box consistency wrapper requiring no model access recovers reliability (confident confabulation 75% $\to$ 3%; selective accuracy 43% $\to$ 83%) at a tunable cost. We contribute a model-agnostic audit protocol, a surface-indistinguishability test, and released code and frozen evaluation items as reusable infrastructure for auditing deployed models.

Figures & tables

Metric	Apple	Gemma	Llama	Minis.
Confidence AUROC $\uparrow$	0.47	0.47	0.52	0.50
ECE $\downarrow$	70.1	64.6	54.2	57.1
Confident halluc. (%) $\downarrow$	67.9	66.0	35.7	54.3
FP confabulation (%) $\downarrow$	69.1	26.4	12.7	7.3
Over-refusal, benign (%) $\downarrow$	18.4	0.0	0.0	0.0
Grounded (%) $\uparrow$	86.2	81.3	74.3	68.1

Table 1: Reliability comparison against comparably sized small models on identical items with one grader. The on-device model is worst-in-class on calibration, confident confabulation, and over-refusal, yet most grounded on its intended task: the task-asymmetric pattern. Bold marks the worst (best for grounded). Peer comparisons are across each model’s standard serving configuration (on-device guided decoding vs. hosted API for peers); serving-pipeline differences are a confound, so cross-model superlatives should be read as contextual. Our core claims (Sections 5 – 7 ) are within-model and do not depend on this contrast.

Figure 1: Reliability diagram on factual QA. The on-device model’s confidence sits near

100\%

almost regardless of accuracy, far from the diagonal, yielding the worst ECE among comparable small models.

Figure 2: Task-asymmetric miscalibration: the on-device model is high on both opposite failures, confidently confabulating on false premises (under-abstention) and refusing benign inputs (over-abstention), while peers are high on neither.

Figure 3: Confident-correct vs. confident-wrong outputs are surface-indistinguishable: per-feature effect sizes fall inside the negligibility band (

|\delta|<0.147

), and a classifier using only user-visible features reaches AUROC

0.55

(equivalent to chance by TOST).

$O(1)$ single-generation
Signal	TriviaQA	GSM8K
Verbalized confidence	0.529	0.519
HVR	0.543	0.590
Fused (HVR+conf)	0.555	0.598
Response length	0.513	0.680
$O(N)$ sampling

Table 2: Correct-vs-incorrect discrimination (AUROC, free-form correctness label). No

O(1)

signal is robust and task-general; only

O(N)

consistency beats the strongest cheap baseline on both (DeLong

p<0.003

). Full 95% CIs in the supplementary material.

Figure 4: Black-box SelfCheck (

k{=}5

) recovery: confident confabulation and hallucination collapse while selective accuracy nearly doubles, with no model access.

Figure 5: Consistency-gated abstention: discrimination and calibration improve with the number of samples

k

and are largely captured by

k{=}5

(marked), recovering reliability without model access.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language Models Language Model Calibration
Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models
Jun 8, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan SinghLLM Evaluation Language Model Error Detection
The Educational Approaches to Virtual Reality Technologies Lab, University of Ioannina, Greece · Meta Platforms
Cross-Model Disagreement as a Label-Free Correctness Signal
Mar 26, 2026Matt Gorbett, Suman JanaLanguage Model Error Detection Confidence Estimation in Language Models
Independent Researcher · Department of Computer Science, Columbia University, New York, NY 10027, USA

Oct 19, 2026J/K move · Enter open · S save

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

Debojeet Ghosh, Shreyansh Kumar, Ayushman Garg

Confidence Estimation in Language Models Language Model Calibration Verbalized Confidence Estimation Language Model Probing Semantic Entropy

Open paper AskPDF arXiv

Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.

Abstract

LLM Auditing

Explore similar work

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

Cross-Model Disagreement as a Label-Free Correctness Signal