From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence

Jiahong Qian

cs.CLOct 14, 2026

From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence

Authors: Jiahong Qian

Organizations: Shanghai Jian Qiao University, Shanghai, China

Abstract

Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures actually consist of, and whether confidence signals can turn unreliable output into usable output. We run a preregistered, fully frozen evaluation of two new-generation low-cost LLMs (DeepSeek-V4-Flash, GLM-5.3-Flash) with a flagship-tier control (GPT-6.1-Sol) on a public corpus of 354 incident campaigns (315 unique reports, 630 model-report evaluations), scoring extraction quality against gold, confidence calibration, and robustness to single-sentence perturbations. Three findings emerge. (1) Unreliability persists across tiers: micro precision is 0.03-0.34 and verbalized calibration error (ECE) 0.53-0.94 across both splits; the gap does not close at the flagship tier. (2) The dominant failure is not hallucination: a full mention census plus a three-tier annotation of 360 sampled candidates (two independent LLM annotators from vendors outside the evaluated set, Cohen's kappa = 0.79 on the preregistered acceptance line, fully disclosed) back-projects to 65.6% background mentions and 19.9% gold-scope gaps, with only 1.2% unsupported candidates. (3) Confidence ranks but does not verify: zero-cost self-consistency-frequency tiering passes a preregistered top-30% precision-gain bar in all six test-split model-type cells (4/6 on train), the zero-cost text-frequency x SC frequency combination wins 3, ties 2, loses 1 across cells, yet relation-level counterfactuals on all 45 targets show that a single misattribution sentence flips high-confidence judgments in 95.6-100% of cases, and even a background sentence explicitly marked as unrelated still binds a fake vulnerability in up to 26.7% of them. We conclude with a deployment boundary: confidence tiering may set review priorities, never truth.

Figures & tables

Model	Type	Micro P	Micro R	Micro F1	Macro F1	95% CI (macro F1)
GLM-5.3-Flash (test, 93)	APT	0.284	0.978	0.441	0.645	[0.576, 0.714]
	CVE	0.069	0.884	0.129	0.225	[0.156, 0.295]
	AV	0.174	0.806	0.287	0.309	[0.248, 0.373]
DeepSeek-V4-Flash (test, 93)	APT	0.336	0.957	0.497	0.675	[0.602, 0.743]
	CVE	0.062	0.855	0.115	0.224	[0.154, 0.292]
	AV	0.297	0.806	0.434	0.391	[0.314, 0.469]

Table 1: Multiset entity-extraction PRF, test split (train-split rows are analogous; appendix). High recall with low precision is structural: the models find most gold entities but drown them in extra candidates (GLM-5.3-Flash test emits 15.1 CVE strings per report on average against at most 2 gold primary CVEs).

Failure mode	$n$	$n$ %	w%
Background mention (L2 $=$ no)	134	37.2	65.6
Event-relevant, not in gold	120	33.3	19.9
TP (relevant, bound, in gold)	52	14.4	10.0
No textual support (lower bound)	37	10.3	1.2
Gold-matched, not event-relevant	11	3.1	2.7
Unresolved borderline	5	1.4	0.4

Table 2: Three-tier failure-mode distribution on the 360-row stratified subsample, all three entity types (w% = back-projected to the full FP composition via sampling weights; the top rows are FP-composition shares). Relation misbinding is a 1-row residual in natural output.

Figure 1: Coverage–accuracy curves of confidence tiering (test split): rows are GLM-5.3-Flash and DeepSeek-V4-Flash; columns are APT / CVE / AV. Orange circles: verbalized self-report; blue squares: SC frequency; gray dotted: overall accuracy. SC-frequency tiering is steep and monotone (e.g. DeepSeek-V4-Flash APT precision 0.500 at 30% coverage vs overall 0.321); verbalized is nearly flat (97%+ of self-reports pile into 80–100).

Model	Type	rand.	text_freq	verb.	sc_freq
DeepSeek-V4-Flash	APT	$-.004{\pm}.045$	+0.367	+0.042	+0.179
	CVE	$\pm$ .011	+0.084	+0.063	+0.118
	AV	$\pm$ .037	$-$ 0.034	+0.066	+0.153
GLM-5.3-Flash	APT	$\pm$ .039	+0.340	+0.319	+0.225
	CVE	$\pm$ .013	+0.082	+0.109	+0.131
	AV	$\pm$ .024	$-$ 0.063	+0.076	+0.157

Table 3: Unified-candidate-set ranking baselines, test split: top-30% precision gain against the preregistered

+0.10

bar (random = 200 seeds

\pm

sd). The zero-cost combination textfreq_sc (omitted column; Fig. 2 ) wins 3, ties 2, loses 1 of 6 cells; the flagship tier confirms the mechanism at textfreq_sc

+0.454

on APT, the largest value across all 9 combinations.

Figure 2: Ranking baselines including the zero-cost combination textfreq_sc : on actor entities mention strength measures event centrality; on techniques text frequency is negative while SC works. Red dashed line: preregistered $+0.10$ bar.

Perturbation	Outcome	DeepSeek-V4-Flash	GLM-5.3-Flash
neg (retraction)	true CVE abandoned	34.1%	76.2%
bg (unrelated)	fake CVE bound to event	26.7%	4.5%
mis (attribution)	injected CVE bound	95.6%	100%
mis (attribution)	injected org. accepted	73.3%	95.6%
Control: assertion (v1), fake CVE bound		86.7%	97.8%

Table 4: Relation-level counterfactuals: one injected sentence per report (all 45 targets, both models). Retained high confidence after binding: verbalized 58–98, SC frequency 0.50–0.90.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Batch	Reports	Used calls	Input tok	Output tok
GLM-5.3-Flash train	222	2,202	17.10M	2.99M
GLM-5.3-Flash test	93	822	5.64M	3.04M
DeepSeek-V4-Flash train	222	2,202	17.24M	5.81M
DeepSeek-V4-Flash test	93	822	5.64M	4.31M
ext main run (30 $\times$ 2)	60	258	2.05M	1.25M
ext counterfactual (15 $\times$ 2 $\times$ 2)	60	60	0.46M	0.29M

Appendix

Table 5: Usage and reproducibility metadata. Per-report call metadata (models/gateways/usage_by_gateway) is archived per batch; deterministic task ids guarantee idempotent resume without double billing; ext batches are physically separated under their own task-id prefix. Revision-round additions: relation-level counterfactuals (1,080 planned calls each for DeepSeek-V4-Flash and GLM-5.3-Flash), verification baselines (DeepSeek-V4-Flash 1,221 / GLM-5.3-Flash 1,200 calls), and the flagship control (822 calls, list-price

\approx

$28.5).

Model	Type	vECE	sECE	vGain	sGain
GLM-5.3-Flash train	APT	0.534	0.259	+0.328	+0.298
	CVE	0.851	0.165	+0.041	+0.063
	AV	0.694	0.244	+0.120	+0.151
DeepSeek-V4-Flash train	APT	0.652	0.253	+0.168	+0.405
	CVE	0.938	0.158	+0.022	+0.055
	AV	0.675	0.329	+0.018	+0.221

Appendix

Table 6: Train-split calibration (full). CVE verbalized ECE train 0.94 vs. test 0.837 (GLM-5.3-Flash) / 0.877 (DeepSeek-V4-Flash): caused by the higher FP density and lower correctness of train-split CVEs; the direction is consistent.

#	Campaign	Blocked at
1	APT41__2016-07	rag / cot
2	APT41__2017-07	rag / cot / rag_cot
3	APT41__2018-07	rag / cot
4	CopyKittens__2017-03	rag / cot
5	CopyKittens__2017-04	rag / cot
6	Deep Panda__2014-04	cot

Appendix

Table 7: Content-filtering gap in the ablations (15 reports, one vendor channel; finish_reason = sensitive / content_filter ). Neither the DeepSeek-V4-Flash channel nor the GLM-5.3-Flash main-run channel saw any blocking; ablation figures use the joint 84-report scope.

Model	Config	APT macro-F1 ( $\Delta$ )	CVE ( $\Delta$ )	AV ( $\Delta$ )	mean out tok
DeepSeek-V4-Flash	base@90	0.679 [.603,.747]	0.219 [.156,.286]	0.393 [.316,.474]	464
DeepSeek-V4-Flash	rag_k3	0.711 (+0.032) [.631,.775]	0.226 (+0.007)	0.343 ( $-$ 0.051) [.262,.425]	543
DeepSeek-V4-Flash	rag_k6	0.704 (+0.025) [.626,.777]	0.226 (+0.007)	0.378 ( $-$ 0.016) [.292,.465]	566
DeepSeek-V4-Flash	rag_k12	0.703 (+0.025)	0.227 (+0.008)	0.383 ( $-$ 0.011)	—
DeepSeek-V4-Flash	cot3	0.691 (+0.013) [.613,.770]	0.233 (+0.014)	0.311 ( $-$ 0.083) [.239,.396]	430
DeepSeek-V4-Flash	cot5	0.678 (+0.000) [.597,.762]	0.220 (+0.001)	0.281 ( $-$ 0.112) [.210,.356]	481

Appendix

Table 8: Second-stage ablation:

\Delta

macro-F1 / absolute values vs. the same model’s base (@90-report intersection; 95% bootstrap CI where shown; omitted CIs are of the same magnitude as adjacent same-model rows, full CIs in ablation_summary.json ). Token column is DeepSeek-V4-Flash-side measured; GLM-5.3-Flash-side thinking cannot be disabled (output tokens 5–6

\times

). schema_noev reports only PRF. Flagship scope: ablations were not run on GPT-6.1-Sol (repair-path conclusions scoped to the flash tier).

Hypothesis	Status	Evidence
H1 three-tier annotation	completed & frozen	§ 4.2 , Table 2
H2 SC increment (ctrl. text freq.)	completed	§ 4.4
H3 review budget + audit-label recalib.	completed	§ 4.4
H4 gold-bearing extrapolation	completed	§ 4.6
H5 review simulation vs. human effort	compl. w/ scope limits	Limitations
H6 third-party calibrated verifier	completed	§ 4.5

Appendix

Table 9: Preregistered hypothesis register (H1–H6), final status. All acceptance lines and mechanisms were registered before evaluation; failing a registered bar must be restated as a negative result.

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 238 papers

Jun 23, 2026

Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.

Nitesh Kumar

1Perfios Software Solutions Pvt. Ltd.

Selective Prediction Document Information Extraction LLM Reliability LLM Uncertainty Estimation

Ask PDF

Feb 18, 2026

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58% of retrieval operations to recover 95% of the maximum achievable accuracy gain on TriviaQA

Xie Xiaohu, Liu Xiaohu, Yao Benjamin

Alexa AI, Amazon, Seattle, WA, USA

Language Model Error Detection Confidence Estimation in Language Models LLM Reliability Language Model Calibration

Ask PDF

Oct 25, 2026

Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation

Tool-using large language models (LLMs) must decide whether external information warrants action. We investigate whether their expressed credibility judgments reliably indicate behavioral resistance to misinformation. We evaluate five LLMs on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted to simulated tool use. When asked to act on the constructed notices, models issue harmful tool calls in 27%--99% of cases they have rated untrustworthy. Removing the preliminary rating step increases harmful execution by 15 percentage points on average, showing that credibility elicitation itself changes the behavior being evaluated. Interventions that improve credibility discrimination do not consistently yield safer actions. A simple forewarning significantly reduces harmful execution in four of five models, but neither tested generated-example prebunk outperforms it in the main evaluation. These reductions can also compromise legitimate task completion. Supervised fine-tuning on generated false notices reduces harmful execution on held-out archived attacks under the rating-first protocol, with model-dependent utility costs and weaker protection without rating elicitation. Our findings motivate evaluating misinformation defenses through both harmful actions and legitimate task completion, while accounting for the behavioral effects of the evaluation protocol.

Emran Kanaan, Jiho Lee, Ibrahim Odat +2

Oakland University · Dartmouth College

Language Model Safety Evaluation Language Model Robustness LLM Agents Evaluation Awareness in Language Models

Ask PDF

Abstract

LLM Information Extraction

Explore similar work

Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation