Reliable Financial Named Entity Recognition Under Domain Shift: Confidence Estimation and Selective Prediction

Zihao Zheng; Baichuan Li; Junyi Yao; Jiayu Long

cs.CLAug 20, 2026

Reliable Financial Named Entity Recognition Under Domain Shift: Confidence Estimation and Selective Prediction

Authors: Zihao Zheng, Baichuan Li, Junyi Yao, Jiayu Long

Organizations: Washington University in St. Louis, St. Louis, Missouri, USA · Southern Methodist University, Dallas, Texas, USA

Abstract

Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, and standard F1 scores do not indicate which predictions remain safe to automate when that input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition, evaluating a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models with five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.

Figures & tables

Split	Domain	Sent.	Ent.	PER	ORG	LOC
Train	FIN (filings) [ 16 ]	1014	980	648	175	157
Valid	FIN (filings) [ 16 ]	150	177	97	68	12
Test	FIN (filings) [ 16 ]	299	295	201	56	38
Test	FiNER-ORD (news) [ 17 ]	300	322	78	151	93
Test	TweetNER7 (tweets) [ 18 ]	300	619	381	135	103

TABLE I: Dataset statistics after harmonization and filtering.

Model	Domain	P	R	F1	Halluc.%	ECE
BERT-base	FIN	66.7 $\pm$ 2.3	74.2 $\pm$ 1.4	70.3 $\pm$ 1.5	–	0.102 $\pm$ 0.016
BERT-base	FiNER-ORD	39.1 $\pm$ 0.3	37.6 $\pm$ 2.3	38.3 $\pm$ 1.1	–	0.071 $\pm$ 0.017
BERT-base	TweetNER7	35.2 $\pm$ 5.2	18.4 $\pm$ 1.7	24.1 $\pm$ 2.5	–	0.074 $\pm$ 0.007
Qwen-0.5B	FIN	40.5 $\pm$ 0.4	34.7 $\pm$ 2.1	37.4 $\pm$ 1.3	3.6 $\pm$ 2.4	0.362 $\pm$ 0.009
Qwen-0.5B	FiNER-ORD	30.7 $\pm$ 1.3	18.0 $\pm$ 4.4	22.6 $\pm$ 3.8	8.2 $\pm$ 0.7	0.407 $\pm$ 0.005
Qwen-0.5B	TweetNER7	30.8 $\pm$ 0.7	23.7 $\pm$ 4.2	26.7 $\pm$ 2.8	5.6 $\pm$ 1.1	0.370 $\pm$ 0.007

TABLE II: Main results (%; mean

\pm

std over 3 seeds; Qwen2.5-1.5B is a single-seed scale check).

Signal	FIN		FiNER-ORD		TweetNER7
	AUROC	ECE	AUROC	ECE	AUROC	ECE
Sequence prob.	0.839 $\pm$ 0.014	0.512 $\pm$ 0.005	0.661 $\pm$ 0.015	0.593 $\pm$ 0.011	0.626 $\pm$ 0.019	0.586 $\pm$ 0.006
Token prob.	0.823 $\pm$ 0.012	0.531 $\pm$ 0.004	0.666 $\pm$ 0.019	0.615 $\pm$ 0.013	0.605 $\pm$ 0.018	0.611 $\pm$ 0.006
Span prob.	0.801 $\pm$ 0.004	0.362 $\pm$ 0.009	0.690 $\pm$ 0.030	0.407 $\pm$ 0.005	0.721 $\pm$ 0.011	0.370 $\pm$ 0.007
Type prob.	0.492 $\pm$ 0.019	0.506 $\pm$ 0.004	0.536 $\pm$ 0.022	0.533 $\pm$ 0.019	0.511 $\pm$ 0.010	0.573 $\pm$ 0.002
Self-consistency	0.690 $\pm$ 0.019	0.116 $\pm$ 0.029	0.671 $\pm$ 0.048	0.120 $\pm$ 0.015	0.682 $\pm$ 0.017	0.100 $\pm$ 0.008

TABLE III: Error-detection AUROC and ECE per confidence signal (Qwen2.5-0.5B; mean

\pm

std over 3 seeds), by test domain.

Fig. 1: Self-consistency reliability (0.5B, pooled seeds). The highest-confidence bin inverts under both shifts.

Signal	0.5B (3 seeds)			1.5B (1 seed)
	FIN	FiNER	Tweet	FIN	FiNER	Tweet
Sequence prob.	0.839	0.661	0.626	0.764	0.720	0.637
Span prob.	0.801	0.690	0.721	0.819	0.729	0.638
Self-consistency	0.690	0.671	0.682	0.735	0.744	0.663

TABLE IV: Scale check: error-detection AUROC by signal and domain for Qwen2.5-0.5B (3-seed mean) vs. Qwen2.5-1.5B (single seed).

Fig. 2: Sentence-level risk–coverage for Qwen2.5-0.5B (3-seed mean and range; lower is better).

Domain	Signal	P@100%	P@80%	P@60%
FIN	Seq. prob.	40.5 $\pm$ 0.4	49.0 $\pm$ 0.8	58.4 $\pm$ 0.6
FIN	Span prob.	40.5 $\pm$ 0.4	49.2 $\pm$ 0.3	56.4 $\pm$ 0.6
FIN	Self-consistency	40.5 $\pm$ 0.4	47.7 $\pm$ 0.4	52.8 $\pm$ 1.0
FiNER	Seq. prob.	30.7 $\pm$ 1.3	34.9 $\pm$ 3.0	40.1 $\pm$ 1.3
FiNER	Span prob.	30.7 $\pm$ 1.3	35.3 $\pm$ 1.7	40.8 $\pm$ 2.6
FiNER	Self-consist.	30.7 $\pm$ 1.3	34.9 $\pm$ 3.0	41.4 $\pm$ 2.3

TABLE V: Entity-level selective precision (%) at 100/80/60% coverage (Qwen2.5-0.5B; mean

\pm

std over 3 seeds).

	Filtered	Relaxed
Sentences dropped (of 2807)	2179	0
Evaluated (cap 300)	300	300
Gold PER/ORG/LOC	619	443
Ignored entities	0	488
Encoder F1	24.1 $\pm$ 2.5	19.5 $\pm$ 1.0
Encoder MSP AUROC	0.615 $\pm$ 0.054	0.650 $\pm$ 0.047

TABLE VI: TweetNER7 filtering sensitivity, main (drop-sentence) vs. relaxed (ignore-entity) protocol. Encoder: 3-seed mean; Qwen: seed 42.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 110 papers

Instruction Finetuning DeepSeek-R1-8B Model Using LoRA and NEFTune
Jun 9, 2026Wu Yuerong, Mingni LuoNamed Entity Recognition Fine-Tuning
University of Hong Kong, Hong Kong, China · Northeastern University, Seattle, United States
Noise-Robust Financial Numerical Entity Attribute Tagging
May 24, 2026Sheng-Hua Tsai, Chen-Yang Lai, Yi-Jhen Li +2Document Information Extraction Noisy-Label Learning
National Taiwan University · National Central University
From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models
Aug 4, 2026Fusheng LuoLLM Evaluation Quantitative Finance
Johns Hopkins University Baltimore, Maryland, USA

Jun 9, 2026J/K move · Enter open · S save

Instruction Finetuning DeepSeek-R1-8B Model Using LoRA and NEFTune

Wu Yuerong, Mingni Luo

University of Hong Kong, Hong Kong, China · Northeastern University, Seattle, United States

Named Entity Recognition Fine-Tuning LLM Fine-Tuning Finance

Open paper AskPDF arXiv

Financial named-entity recognition (NER) is essential for translating unstructured financial reports and news into structured knowledge graphs. However, general-purpose large language models (LLMs) often misclassify financial entities or ignore domain-specific patterns. This paper investigates the use of DeepSeek-R1-8B, a recent open-source large language model, combined with Low-Rank Adaptation (LoRA) and Noisy Embedding Fine-Tuning (NEFTune) for financial NER. Each annotated sentence in our corpus of 1693 samples is converted into an instruction-input-output triple. We insert lightweight LoRA matrices into the Transformer layers and apply NEFTune to improve generalisation by adding uniform noise to embedding vectors during training. Experiments show that the LoRA-adapted DeepSeek-R1-8B achieves a micro-F1 of 0.901 on seven entity types (Company, Date, Location, Money, Person, Product and Quantity), and adding NEFTune further boosts the micro-F1 to 0.912, outperforming Llama3-8B, Qwen3-8B, Baichuan2-7B, T5 and BERT-Base baselines.

Abstract

Named Entity Recognition

Explore similar work

Instruction Finetuning DeepSeek-R1-8B Model Using LoRA and NEFTune

Noise-Robust Financial Numerical Entity Attribute Tagging

From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models