Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

Chenhao Xue; Raslen Guesmi; Siwei Feng; Yucheng Gong; Jacob Xavier Sundram; Jordan Pang; Lan Wang; Julian Kaljuvee

cs.CLAug 18, 2026

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

Authors: Chenhao Xue, Raslen Guesmi, Siwei Feng, Yucheng Gong, Jacob Xavier Sundram, Jordan Pang, Lan Wang, Julian Kaljuvee

Organizations: Predictive Labs Ltd · University of Oxford · Imperial College London · Independent Researcher

Abstract

Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train-test split is chronological or random, i.e., on temporal leakage. We audit this dependence on a 49,799-article corpus across 16 feature-model combinations spanning TF-IDF, MiniLM, FinBERT, and fine-tuned RoBERTa-large / DeBERTa-v3-large, plus separate zero/few-shot and LoRA probes of Llama-3 and Qwen2.5 LLMs: random splits inflate MCC by $1.1\times$ to $6.5\times$ , tracking model capacity and feature richness, and end-to-end FinBERT fine-tuning re-amplifies rather than closes the gap (size-matched ratio $1.75\times$ ). Conditioning on event type, mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation (TF-IDF MCC $= 0.138$ train-only, $0.068$ under train $\cup$ val refit; 10,000-permutation $p < 10^{-3}$ ); the signal does not transfer to FNSPID's 2009-2020 U.S. corpus, localising the headline to our 2024-2025 European-tilted M&A semantics rather than a universal predictor. Three independent role labellers converge on acquirer-tagged articles as the signal locus, a power-limited qualitative convergence rather than a hypothesis-tested asymmetry. Chronological splitting plays for financial NLP the role characteristics-purging plays for asset pricing: it strips the predictable, stale component of news and leaves a residual that is small, event-localized, and lexically shallow. We advocate leakage audits as a required disclosure for financial-NLP benchmarks.

Figures & tables

Statistic	Value
Total articles	56,409
Date range	2020-05–2025-08
Articles from 2025	81%
Exchanges covered	64
Event types	203
After neutral removal	49,799

Table 1: Dataset overview.

Figure 1: Multi-architecture leakage audit (

n_{\text{train}}{=}21{,}654

n_{\text{test}}{=}17{,}279

). Each row pairs temporal-split MCC (blue) with 10-seed mean random-split MCC (orange,

\pm

1 SD). Annotated values are the audit ratio

\rho

of Eq. ( 3 ); for end-to-end fine-tuned FinBERT (Table 2 , last block) the size-matched ratio is

1.75\times

(App. D.1 ).

Features	Model	Random MCC ‡	Temporal MCC	Ratio
TF-IDF title	LR	$0.022\pm 0.007$	$0.013$	$1.6\times$
TF-IDF title	RF	$0.045\pm 0.005$	$0.024$	$1.9\times$
TF-IDF title	GB	$0.026\pm 0.009$	$0.029$	$0.9\times$
TF-IDF title+content	LR	$0.039\pm 0.009$	$0.017$	$2.3\times$
TF-IDF title+content	RF	$0.106\pm 0.009$	$0.032$	$3.3\times$
TF-IDF title+content	GB	$0.036\pm 0.008$	$0.007$	$5.0\times$

Table 2: Multi-architecture leakage audit (proprietary;

n_{\text{tr}}{=}21{,}654

n_{\text{te}}{=}17{,}279

). FinBERT[CLS]+LR has the smallest leakage ratio (

1.1\times

); fine-tuning the same encoder lifts it to

2.7\times

( ¶ size-matched

1.75\times

, App. D.1 ). ‡ 10-seed mean

\pm

std (5-seed for FT rows); temporal column deterministic for frozen rows. § Fine-tuned on GPU: DeBERTa-v3-large uses class-balanced sampling ( He et al., 2021 ) ; RoBERTa-large ( Liu et al., 2019 ) collapsed to predict-all-UP. Bi-LSTM: 2-layer, hidden=128.

Model	Random	Temporal	Ratio
TF-IDF title $+$ LR	$0.025\pm 0.008$	$0.014$	$1.7\times$
TF-IDF title $+$ RF	$0.029\pm 0.005$	$0.001$	$28.9\times^{\dagger}$
TF-IDF title $+$ GB	$0.020\pm 0.007$	$0.007$	$2.8\times$

Table 3: Cross-corpus audit on EDT ( Zhou et al., 2021 ) (

n=106{,}619

, 2020–2021, 70/15/15 chronological vs 5-seed random). The directional pattern of Table 2 replicates on a different corpus/market/regime. † Near-zero temporal denominator amplifies

\rho

; see ratio-interpretation caveat below.

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Eight-month rolling MCC under chronological validation (cf. § 7 ). M&A (green) is the only series positive in every month; the full-corpus (grey) and event-conditioned average (blue) hover near zero.

Figure 3: Permutation null (10,000 label permutations, grey) vs observed M&A test MCC (blue);

z=3.81

, two-sided

p<10^{-3}

Figure 4: Stability of the M&A specialist over the 360-cell HP grid. The original-paper sub-grid (grey, 20 cells without max_features=100 or ngram_range variation) misses the true optimum at validation MCC

=0.228

; the extended grid (blue, top-15 cells) reliably surfaces it. The locked-test MCC at the selected cell is

0.138

(Figure 3 ).

$k$	max_f	$C$	sub	min_df	ngram	val MCC
1	100	5.0	F	2	(1,1)	0.228
2	50	5.0	F	2	(1,3)	0.222
3	50	5.0	F	1	(1,3)	0.222
4	50	5.0	T	2	(1,2)	0.221
5	50	5.0	F	2	(1,2)	0.218
6	100	5.0	F	1	(1,1)	0.218

Appendix

Table 4: Top-15 of 360 validation hyperparameter cells for the M&A specialist. sub is sublinear_tf . The cells differ by

\Delta\text{MCC}<0.03

, showing a broad plateau rather than a single lucky cell.

Quantity	Headline	Extended
	(Jun–Aug 2025)	(Mar–Aug 2025)
$n_{\text{train}}$	$731$	$611$
$n_{\text{test}}$	$786$	$1275$
Test pred-UP rate	$0.586$	$0.544$
Test label-UP rate	$0.594$	$0.590$
Test MCC	$\mathbf{+0.138}$	$\mathbf{+0.133}$

Appendix

Table 5: Headline vs. extended-window sensitivity test, same TF-IDF+LR specialist at the paper-authoritative HP. The extended window covers a

62\%

larger test set; the pooled MCC is within one bootstrap std of the headline. The wider bootstrap CI (lower bound

+0.040

vs.

+0.066

) reflects the inclusion of March 2025 as a single negative month (see Table 6 ).

Month	$n$	MCC	pred-UP	label-UP
2025-03	$120$	$-0.067$	$0.567$	$0.558$
2025-04	$202$	$+0.178$	$0.510$	$0.594$
2025-05	$167$	$+0.295$	$0.593$	$0.587$
2025-06	$206$	$+0.125$	$0.563$	$0.549$
2025-07	$405$	$+0.124$	$0.546$	$0.615$
2025-08	$175$	$+0.112$	$0.497$	$0.600$

Appendix

Table 6: Per-month MCC for the extended-window sensitivity test. The 2025-04 and 2025-05 columns were the original validation period (used for HP selection in Section 7.2 ); the 2025-06/07/08 columns were the original locked test. Holding HP fixed and folding the validation period into the test partition leaves five of six months positive and shifts the pooled MCC by only

0.005

relative to the headline (Table 5 ).

Protocol	$n_{\text{tr}}$	$n_{\text{te}}$	MCC
Train-only $\to$ locked test ( headline ; § 7.2 )	$731$	$786$	$\mathbf{+0.138}$
Train $\cup$ val refit $\to$ locked test (deployment-style; App. H.1 )	$1{,}100$	$786$	$+0.068$
Extended-window 6-month (App. B.3 )	$611$	$1{,}275$	$+0.133$

Appendix

Table 7: Three proprietary-corpus M&A protocols, same TF-IDF+LogReg specialist at paper-authoritative HP ( max_features=100, C=5.0, sublinear_tf=False, min_df=2, ngram_range=(1,1) ). Headline (top row) is the lowest-leakage variant: HP selected on val, locked test consulted once. The train

\cup

val refit protocol re-incorporates the validation period into training without re-tuning HP; it is closer to a deployed model and gives a more conservative MCC. The extended-window protocol folds both val and original test into a single 6-month test window. The MCCs are within one bootstrap std of each other. EDT and FNSPID rows are reserved for the cross-corpus diagnostic (Table 3 , App. J.2 ) and intentionally excluded here.

Figure 5: B1 – Locked M&A test (

n=786

): (a) ROC, (b) precision-recall, (c) reliability (calibration) curves with 10 quantile bins for TF-IDF

+

LR and FinBERT [CLS]

+

LR specialists.

Bucket	$n$	MCC	True UP
Head ( $\geq 10$ train articles)	42	$-0.121$	$0.548$
Torso ( $3$ – $9$ train articles)	128	$+0.097$	$0.539$
Tail-seen ( $1$ – $2$ train articles)	180	$+0.143$	$0.628$
Unseen at train time	436	$+0.043$	$0.601$

Appendix

Table 8: B2 – Per-ticker fairness: locked M&A test partitioned by train-side ticker frequency.

Variant	Locked-test MCC
Baseline (no removal)	$0.0683$
Top-5 tokens removed	$0.0503$
Top-10 tokens removed	$0.0728$
Top-20 tokens removed	$0.0570$
Top-40 tokens removed	$0.0008$

Appendix

Table 9: B3 – Counterfactual lexical-cue ablation. Removing the top-40 most-influential tokens collapses the locked-test MCC by two orders of magnitude. Note: baseline differs from the body

0.138

because the body headline (§ 7.2 ; Table 7 ) uses the train-only protocol, while this ablation uses the train

+

val deployment-style protocol to maximise sample size for cue-importance ranking stability.

Setup	$n_{\text{tr}}$ / $n_{\text{te}}$	MCC
Cross-year (train 2020, test 2021)	$1181$ / $1372$	$+0.008$
Within-2021 random (mean of 5 seeds)	$915$ / $457$	$-0.010\pm 0.018$

Appendix

Table 10: B4 – EDT M&A cross-year robustness. Both the cross-year and within-year settings produce MCC

\approx 0

, in contrast to the

0.138

obtained on our 2025 proprietary corpus. EDT 2020–2021 is a regime in which short-horizon M&A headlines do not carry directional signal under any split choice.

Variant	Locked-test MCC
Original titles	$0.0683$
ORG-tokens $\rightarrow$ [ORG]	$0.0730$
$\Delta$ (masked $-$ baseline)	$\mathbf{+0.0047}$

Appendix

Table 11: B5 – ORG-token masking ablation. Replacing capitalised firm-like tokens with [ORG] placeholders does not hurt the locked-test MCC (it improves marginally). The M&A signal is therefore not driven by firm-identity leakage; it survives even when all firm-name vocabulary is removed.

Threshold rule	Test MCC	Pred-UP rate
Default ( $0.50$ )	$0.1378$	$0.466$
Val-optimal ( $0.58$ )	$\mathbf{0.1443}$	$0.319$
$\Delta$ (val-optimal $-$ default)	$\mathbf{+0.0065}$	—

Appendix

Table 12: B7 – Decision-threshold sweep on M&A locked test. The validation-optimal threshold (

t=0.58

) shifts the model from

0.466\to 0.319

pred-UP rate (under true-UP rate

0.594

) and lifts the test MCC by

+0.007

. The headline result is stable to threshold choice.

Month	$n$	MCC	True UP	Pred UP
2025-06	$206$	$+0.079$	$0.549$	$0.549$
2025-07	$405$	$\mathbf{+0.112}$	$0.615$	$0.595$
2025-08	$175$	$-0.052$	$0.600$	$0.554$
Pooled	$786$	$+0.068$	$0.594$	$0.574$

Appendix

Table 13: B8 – Per-month rolling MCC on the locked M&A test (TF-IDF specialist, train

+

val merge to test).

Condition	Test MCC	Pred-UP rate
Original	$0.1378$	$0.593$
ORG masked	$0.1344$	$0.557$
ORG $+$ PERSON masked	$0.1271$	$0.650$
Full mask (six entity types)	$0.1238$	$0.573$
$\Delta$ ORG-mask $-$ original	$-0.0034$	—
$\Delta$ full-mask $-$ original	$-0.0140$	—

Appendix

Table 14: B9 – spaCy NER entity blanking on the M&A specialist. Replacing all spaCy-identified ORG spans with the [ORG] placeholder costs only

0.003

MCC; replacing six entity types in total costs

0.014

. The signal is overwhelmingly non-entity (deal-verb and structural-noun) vocabulary.

Publisher bucket	$n$	# pubs	Test MCC
Mid ( $200$ – $999$ train articles)	$457$	$2$	$+0.028$
Tail ( $<50$ train articles)	$108$	$10$	$\mathbf{+0.137}$
Unseen at train time	$221$	$55$	$\mathbf{+0.128}$

Appendix

Table 15: B11 – M&A locked test partitioned by train-side publisher frequency. The signal concentrates in the tail / unseen buckets, not in the two dominant publishers.

Split	Test MCC (mean $\pm$ std)	Test $n$
Temporal	$0.0639\pm 0.0066$	17,279
Random (70/15/15)	$0.1740\pm 0.0091$	7,470
Random (size-matched $n_{\text{te}}=$ 17,279) ♭	$0.1120\pm 0.0113$	17,279
Random 70/15/15 / Temporal ratio ( unequal $n_{\text{te}}$ )		$2.72\times$
Random ${}_{\text{size-matched}}$ / Temporal ratio ( matched $n_{\text{te}}$ )		$\mathbf{1.75\times}$

Appendix

Table 16: FinBERT-tone fine-tune: random split vs. temporal split, 5 seeds each. The original 70/15/15 random protocol gives test

n=7{,}470

, smaller than the temporal-split test

n=17{,}279

, so the random/temporal ratio of

2.72\times

partly reflects small-sample variance. ♭ Adding a size-matched random protocol (per-seed stratified random sample with exactly

n_{\text{tr}}=21{,}654

n_{\text{val}}=10{,}866

n_{\text{te}}=17{,}279

to match the temporal split) gives random MCC

0.1120\pm 0.0113

and a leakage ratio of

1.75\times

. The size-matched ratio is the apples-to-apples comparison; the headline finding survives: random splits still inflate MCC by

1.75\times

over the chronological split at identical test size, with no overlap between the random and temporal 95% CIs (

[0.090,0.134]

vs.

[0.054,0.074]

). The unequal-

n

ratio of

2.72\times

is retained for continuity with Table 2 and earlier audit conventions, but the size-matched

1.75\times

is the protocol-controlled estimate.

Figure 6: Frozen vs. fine-tuned FinBERT under random vs. temporal splits on the full proprietary corpus. Error bars are 5-seed standard deviations (single-seed standard deviation for the frozen-temporal cell, which is deterministic). Ratios printed above each pair.

Model	Seeds	MCC mean $\pm$ std	Range
FinBERT-tone	5	$0.050\pm 0.030$	$[0.012,0.083]$
DeBERTa-v3-large balanced	5	$0.085\pm 0.044$	$[0.034,0.135]$
Multi-task FinBERT	3	$0.067\pm 0.007$	$[0.061,0.074]$
TF-IDF specialist (1 cell)	1	$0.138$	—

Appendix

Table 17: Per-seed locked-test MCC for deep M&A specialists. Each row holds hyperparameters fixed (FinBERT 6 epochs/lr

2{\times}10^{-5}

; DeBERTa-v3-large 8 epochs/lr

6{\times}10^{-6}

/warm-up

10\%

/class-balanced WeightedRandomSampler; multi-task FinBERT

\alpha{=}0.7

direction-loss weight, M&A 4

\times

upweight). Even the best deep specialist (DeBERTa-v3-large with balanced sampling) trails the shallow TF-IDF baseline by

0.05

MCC absolute; class-balanced sampling is necessary but not sufficient to recover the M&A signal at

n_{\text{tr}}=731

Figure 7: Per-seed M&A locked-test MCC for three deep specialists. Dots are individual seeds; horizontal bars are the per-model means. Dashed line is the TF-IDF specialist locked-test MCC (

0.138

Model (5 seeds unless noted)	MCC	Wall
TF-IDF $+$ LR specialist (1 seed)	$\mathbf{+0.138}$	$<1$ s
FinBERT-tone LoRA (v2)	$+0.050\pm 0.030$	90s
FinBERT-tone full SFT (v4)	$+0.034\pm 0.035$	22s
DeBERTa-v3-large bal. (v3)	$+0.085\pm 0.044$	113s
DeBERTa-v3-large bal. (v4)	$+0.038\pm 0.021$	114s
pooled 10-seed	$+0.061\pm 0.040$	—

Appendix

Table 18: All large-model audits on locked M&A test (

n=786

). The TF-IDF specialist is the only model above MCC

=0.10

on the locked test; every transformer and LLM variant lags the shallow lexical baseline.

Internal ID	Public API	Snapshot	Access date	Pipe
Claude Sonnet 4.5	claude-sonnet-4.5	2026-04-29	2026-05-08	v8
Claude Opus 4.7	claude-opus-4.7 (TBD)	2026-05-15	2026-05-16	v8
GPT-5.4	gpt-5.4 (TBD)	2026-04-17	2026-05-09	v8
Qwen2.5-7B	Qwen/Qwen2.5-7B-Instruct	2024-09-19	2026-05-12	v9
Llama-3-8B	meta-llama/Meta-Llama-3-8B-Instruct	2024-04-18	2026-05-12	v9

Appendix

Table 19: LLM snapshot table. Public-API names marked TBD are best-effort mappings at submission time; the camera-ready will replace TBD with the launched API string once each model is publicly released. Internal pipeline version (v8 / v9) refers to our GPU evaluation harness release; prompt logs and per-article outputs are released in the artifact bundle. Inference parameters : all closed-source models were called with temperature=0.0 (deterministic; top_p not set). GPT-5.4 is a reasoning model and rejects the temperature parameter, so it was called with API defaults and a deterministic seed. Open-weight models (Qwen2.5-7B, Llama-3-8B) were run with do_sample=False (greedy decoding, equivalent to temperature=0.0 ); CoT and template-sensitivity ablations sweeping temperature

\in\{0.0,0.3,0.7\}

are reported in App. D .

Setup	Prompt	$n_{\mathrm{valid}}$	MCC
Claude Sonnet 4.5, global sample
single	title-only	500	0.077
single	title+event	476	0.026
single	title+content	475	0.051
single	CoT	500	0.055
Sonnet/Opus/GPT consensus, global sample

Appendix

Table 20: Zero-shot LLM results by prompt template and aggregation. Single Sonnet title-only is the best M&A LLM result; consensus title+event is the best general-news LLM result. Adding structured information beyond a single sentence often harms performance, suggesting LLMs over-attend to template scaffolding. None of these match supervised TF-IDF on M&A (test MCC

=0.138

Setup	Test MCC	Pred-UP rate
Llama-3-8B zero-shot	$0.036$	$0.691$
Llama-3-8B few-shot $k=3$	$0.032$	$0.729$
Llama-3-8B few-shot $k=5$	$0.022$	$0.733$
Llama-3-8B LoRA	$0.000^{\ddagger}$	$1.000$
Qwen2.5-7B zero-shot	$\mathbf{0.115}$	$0.355$
Qwen2.5-7B few-shot $k=3$	$0.046$	$0.607$

Appendix

Table 21: Open-LLM M&A locked-test results. True UP rate

=0.594

. ‡ Llama-3-8B-LoRA degenerates to predict-all-UP. The strongest open-LLM result, Qwen2.5-7B zero-shot, slightly under-predicts UP (

0.355

) yet still trails the supervised TF-IDF specialist by

17\%

relative.

Figure 8: Open-LLM M&A benchmark (locked test,

n_{te}=786

). Dashed black line is the TF-IDF specialist reference (

0.138

). Qwen2.5-7B zero-shot is the only LLM within

0.025

MCC of supervised TF-IDF; fine-tuning interventions hurt or collapse.

Subset	$n_{\text{tr}}$	$n_{\text{te}}$	Test MCC
ACQUIRER specialist FinBERT	113	125	$\mathbf{0.195}$
TARGET specialist FinBERT	35	85	$0.017$

Appendix

Table 22: Dedicated FinBERT-tone fine-tunes on regex-ACQUIRER and regex-TARGET subsets. Despite the much smaller training set (

n_{\text{tr}}=113

vs. the overall

n_{\text{tr}}=731

), the ACQUIRER specialist exceeds every deep specialist trained on the full M&A subset (Table 17 ). The TARGET specialist collapses, consistent with the body claim that target-side text carries little independent predictive signal.

Model	Val MCC	Test MCC
TF-IDF $+$ LR (title)	$+0.025$	$+0.013$
TF-IDF $+$ num $+$ RF	$+0.042$	$+0.032$
MiniLM (frozen) $+$ LR	$+0.041$	$+0.037$
FinBERT [CLS] (frozen) $+$ LR	$+0.058$	$\mathbf{+0.060}$
Multi-LLM consensus, title+event	—	$\mathbf{+0.108}$

Appendix

Table 23: General-news locked test (

n=17{,}279

). Multi-LLM consensus is Sonnet

+

Opus

+

GPT under the title+event prompt of App. E.2 .

Model	Val MCC
M&A text model (TF-IDF, paper HP)	$+0.228$
M&A metadata only (exchange)	$+0.037$
M&A metadata aggregate †	$-0.020$

Appendix

Table 24: Non-text controls vs the M&A text specialist (paper HP). † exchange

+

day-of-week

+

event subtype

+

numerical features.

Cost	All-trade		Top-25%-conf
(bps/side)	Sharpe	Ret%	Sharpe	Ret%
0	$+0.52$	$+2.86$	$+2.90$	—
2	$+0.29$	$+0.66$	$+2.85$	—
5	$-0.04$	$-2.55$	$+2.76$	—
10	$-0.60$	$-7.68$	$+2.62$	—
20	$-1.71$	$-17.16$	$+2.33$	—

Appendix

Table 25: Cost-aware backtest of the paper-authoritative M&A specialist (reproducing locked-test MCC

=0.138

). All-trade Sharpe crosses zero near

\approx 5

bps/side; top-quartile-confidence (

n=197

) yields Sharpe

+2.62

at 10 bps/side. High-confidence subset is the actionable signal.

Event	$n_{\text{te}}$	Val MCC	Test MCC
Mergers & acquisitions	786	$+0.232$	$\mathbf{+0.123}$
Earnings releases	4138	$+0.018$	$-0.039$
Financial results	1815	$+0.044$	$+0.052$
Changes in companys own shares	1547	$-0.010$	$-0.022$
Partnerships	712	$+0.069$	$-0.015$
Regulatory filings	588	$+0.057$	$+0.044$

Appendix

Table 26: Per-event TF-IDF

+

LR specialist (val-then-test, 100-cell HP grid per event). Only M&A produces substantial positive test MCC; earnings and management-changes flip sign val

\to

test, illustrating the cost of selection on validation alone.

Event	Train end	Test start	$n_{\textrm{tr}}$	$n_{\textrm{vl}}$	$n_{\textrm{te}}$
M&A	2025-04-01	2025-06-01	731	369	786
Clinical Study	2025-04-01	2025-06-01	1,168	344	482
Legal Issues	2025-06-01	2025-07-15	121	34	1,017
Earnings	2025-04-01	2025-06-01	1,870	508	1,067

Appendix

Table 27: Cross-event cutoffs and split sizes.

Event	val-best HP	Chrono	Random (5s)	Ratio
M&A	mf 100 , $C_{5}$	$+0.228$	$+0.173\pm 0.020$	$\mathbf{0.76\times}$
Clinical Study	mf 500 , $C_{5}$	$+0.036$	$+0.151\pm 0.032$	$\mathbf{4.20\times}$
Legal Issues	mf 300 , $C_{5}$	$+0.212$	$+0.175\pm 0.177$	$0.83\times$
Earnings	mf 100 , $C_{5}$	$+0.107$	$+0.077\pm 0.045$	$\mathbf{0.72\times}$

Appendix

Table 28: Per-event audit ratio. Random MCC is 5-seed mean

\pm

std with val size matched to the event’s chronological val. Legal Issues’ large random-MCC std reflects

n_{\textrm{vl}}=34

. Earnings shares M&A’s low audit-ratio signature (no leakage) but, in contrast to M&A, has no locked-test signal (see Table 29 ).

Event	$n$	MCC	bal-acc	perm $p_{2}$	95% CI
M&A	786	$\mathbf{+0.138}$	$0.569$	$\mathbf{<10^{-3}}$	$[+0.066,+0.205]$
Clinical Study	482	$-0.049$	$0.481$	$0.32$	$[-0.124,+0.070]$
Legal Issues	1,017	$+0.022$	$0.510$	$0.50$	$[-0.006,+0.031]$
Earnings	1,067	$-0.007$	$0.497$	$0.86$	$[-0.084,+0.090]$

Appendix

Table 29: Locked-test results at the paper-authoritative M&A HP applied identically to each event. M&A is the only event whose locked-test MCC excludes zero at

p<10^{-3}

. perm

p_{2}

: 10K-permutation two-sided

p

. 95% CI : weekly block-bootstrap.

Event	Month 1	Month 2	Month 3
M&A	$+0.126\,_{n=206}$	$+0.135\,_{n=405}$	$+0.171\,_{n=175}$
Clinical Study	$+0.089\,_{n=135}$	$-0.070\,_{n=146}$	$-0.085\,_{n=201}$
Legal Issues	—	$-0.016\,_{n=341}$	$+0.025\,_{n=676}$
Earnings	$+0.190\,_{n=47}$	$-0.055\,_{n=580}$	$+0.039\,_{n=440}$

Appendix

Table 30: Per-month MCC. For M&A, Clinical Study, and Earnings, Months 1–3 are 2025-06, 07, 08; for Legal Issues, Months 2–3 are 2025-07, 08 (test starts 2025-07-15). M&A months are all positive; clinical-study months alternate sign with a positive June, negative July/August. Earnings has a tiny June bucket (

n=47

) followed by a near-zero, sign-flipping July/August.

Event	MCC	ROC-AUC	PR-AUC	Brier
M&A	$+0.068$	$0.567$	$0.655$	$0.256$
Clinical Study	$-0.050$	$0.458$	$0.601$	$0.265$
Legal Issues	$-0.060$	$0.506$	$0.522$	$0.304$
Earnings	$-0.012$	$0.487$	$0.510$	$0.274$

Appendix

Table 31: Cross-event B1: locked-test calibration and discrimination at the paper-authoritative TF-IDF

+

LR specialist trained on train

+

val pooled (matching the original M&A B1 protocol of Appendix C.1 ). ROC-AUC below

0.5

on CLN signals anti -discrimination (the model’s probability ranking inverts on the test window); LGL’s

0.506

and ERN’s

0.487

are effectively chance.

Event	Baseline	ORG-masked	$\Delta$ (base $-$ masked)
M&A	$+0.138$	$+0.093$	$\mathbf{+0.045}$
Clinical Study	$-0.049$	$+0.007$	$\mathbf{-0.056}$
Legal Issues	$+0.022$	$+0.055$	$\mathbf{-0.033}$
Earnings	$-0.007$	$+0.033$	$\mathbf{-0.040}$

Appendix

Table 32: Cross-event B5: regex-TitleCase ORG-token ablation at the train-only paper-HP protocol (matching the headline M&A setup). A positive

\Delta

means ORGs carry signal; a negative

\Delta

means ORGs are noise/leakage that hurts generalisation. M&A is the only event where ORG tokens carry transferable signal ; on CLN, LGL, and ERN, masking firm-identity tokens improves the locked-test MCC. This is the single most diagnostic cross-event finding: residual non-zero MCC on non-M&A events traces to firm-name memorisation that does not transfer across chronological splits, even on ERN where

n_{\textrm{tr}}=1{,}870

makes the negative

\Delta

comfortably out of the noise floor.

Event	orig	mask ORG	mask ORG $+$ PER	mask ALL
M&A	$+0.138$	$+0.134$	$+0.127$	$+0.114$
Clinical Study	$-0.049$	$+0.071$	$+0.059$	$+0.031$
Legal Issues	$+0.022$	$+0.022$	$-0.012$	$-0.038$
Earnings	$-0.007$	$-0.007$	$-0.009$	$-0.014$

Appendix

Table 33: Cross-event B9: spaCy true-NER entity blanking under four progressive masking schemes (ALL

=\{

ORG, PERSON, MONEY, GPE, PERCENT, CARDINAL

\}

). Train-only paper-HP protocol. The pattern of B5 (Table 32 ) is confirmed with true NER: M&A is robust to entity masking (the lexical signal is in deal-related verbs/nouns, not names); on CLN, ORG-masking lifts MCC by

+0.12

; on LGL, progressive masking strictly degrades from

+0.022\to-0.038

; on ERN, MCC stays near zero throughout, consistent with the absence-of-signal interpretation of Table 29 .

Event	$n_{\textrm{tr}}$	Default $t{=}0.50$	Val-opt $t^{*}$ $\to$ test MCC
M&A	731	$+0.138$	$t^{*}{=}0.58\to+0.144$
Clinical Study	1,168	$-0.049$	$t^{*}{=}0.30\to+0.048$
Legal Issues	121	$+0.022$	$t^{*}{=}0.46\to+0.027$
Earnings	1,870	$-0.007$	$t^{*}{=}0.50\to-0.007$

Appendix

Table 34: Cross-event B7: decision-threshold sweep on val; locked-test MCC at default and val-optimal thresholds. M&A is threshold-robust (

+0.138\to+0.144

\Delta=+0.006

); CLN’s apparent

+0.048

t^{*}{=}0.30

is a

20

-point threshold shift from the default and should be read as val-set overfitting on a small effective sample, not as recovered signal. On ERN the val-optimal threshold coincides with the default (

t^{*}{=}0.50

), so the locked-test MCC is unchanged: the model is already at its decision boundary and there is no threshold slack to exploit, consistent with the genuine-null interpretation.

Event	Pooled	Month 1	Month 2	Month 3
M&A	$+0.068$	$+0.079\,_{n=206}$	$+0.112\,_{n=405}$	$-0.052\,_{n=175}$
Clinical Study	$-0.050$	$+0.167\,_{n=135}$	$-0.072\,_{n=146}$	$-0.160\,_{n=201}$
Legal Issues	$-0.060$	—	$-0.106\,_{n=341}$	$-0.034\,_{n=676}$
Earnings	$-0.012$	$+0.148\,_{n=47}$	$-0.058\,_{n=580}$	$+0.035\,_{n=440}$

Appendix

Table 35: Cross-event B8: pooled and per-month locked-test MCC at the train

+

val-combined protocol (this drops M&A from the headline

+0.138

+0.068

because adding the val-period articles to training shifts the decision boundary; see Limitations Limitations ). M&A is positive in two of three months. CLN shows sign-inverting drift (

+0.167\to-0.072\to-0.160

); LGL is negative in both observed months; ERN’s first month is a

n=47

sliver (the bulk of earnings articles arrives in 2025-07–08), and its larger July/August buckets straddle zero. This pattern matches the audit-ratio readout: CLN is a textbook intra-period autocorrelation event, LGL is power-limited, ERN is a genuine null with adequate power, and only M&A survives chronological projection.

Event	Pooled MCC	Publisher buckets present
M&A	$+0.068$	head, mid, tail, unseen
Clinical Study	$-0.050$	spans $[-0.131,+0.167]$ across buckets
Legal Issues	$-0.060$	$993/1{,}017$ articles from unseen publishers ( $n_{\textrm{tr}}{=}121$ )
Earnings	$-0.012$	$651/1{,}067$ unseen-publisher articles; spans $[-0.077,+0.127]$ across buckets

Appendix

Table 36: Cross-event B11: publisher-level fairness audit (head/torso/tail/unseen at-train bucketing). LGL is essentially an unseen-publisher test (

98\%

of locked-test articles come from publishers not seen at train), reinforcing the power-limited interpretation. CLN spans both the most-positive and most-negative buckets across the pooled negative. ERN has high publisher churn (

61\%

unseen) but bucket-level MCCs straddle zero symmetrically: no single publisher source carries the genuine-null pooled MCC.

Model	M&A	CLN	LGL	ERN
FinBERT-tone (full SFT)	$0.050{\pm}0.030$	$0.055{\pm}0.050$	$0.014{\pm}0.028$	$-0.001{\pm}0.027$
DeBERTa-v3-large bal.	$0.085{\pm}0.044$	$0.085{\pm}0.028$	$0.008{\pm}0.017$	$-0.029{\pm}0.011$
XLM-R-large	— †	$0.000{\pm}0.000^{\ddagger}$	$-0.012{\pm}0.022$	$-0.001{\pm}0.003$
TF-IDF reference	$\mathbf{0.138}$	$-0.049$	$0.022$	$-0.007$

Appendix

Table 37: Cross-event deep specialists, 5-seed mean

\pm

std locked-test MCC at the per-event train-only protocol of Table 29 . The deep specialists do not recover a positive locked-test MCC on any non-M&A event : every cell is at most

+0.085

, and three of nine non-M&A cells are negative-in-mean. † XLM-R-large on M&A was not in the original Appendix D.2 run (gap; included on CLN/LGL/ERN here for symmetry with the new events). ‡ XLM-R on CLN collapsed to predict-all-UP across all 5 seeds (pred_up_rate

=1.0

, balacc

=0.5

, ROC-AUC mean

0.55

), giving MCC

=0

by definition; we report the degenerate value transparently rather than dropping the row. JSONs at paper/fin_nlp_gpu_v{5,6}_results.zip .

Multi-seed (5 seeds $\{42,0,1,2,3\}$ , $T=0.7$ , top- $p=0.95$ ; mean $\pm$ std)
LLM setting	M&A	CLN	LGL	ERN
Qwen-2.5-7B zero-shot	$-0.022\pm 0.010$	$0.126\pm 0.026$	$0.040\pm 0.006$	$0.068\pm 0.012$
Llama-3-8B zero-shot	$0.015\pm 0.026$	$0.080\pm 0.044$	$0.050\pm 0.025$	$0.032\pm 0.023$
Qwen-2.5-7B CoT v2 strict §	$0.011\pm 0.030$	$0.061\pm 0.023$	$0.058\pm 0.009$	$0.034\pm 0.034$
parse-fail rate (mean)	$17.3\%$	$25.7\%$	$32.5\%$	$26.8\%$
Single-seed reference (seed=42, deterministic $T=0.0$ )

Appendix

Table 38: Cross-event open-LLM locked-test MCC under a multi-seed protocol. All three multi-seed rows use a single uniform zero-shot/CoT prompt across the 4 events and 5 random seeds at

T=0.7

(sampling); the CoT-v2 structured row is retained as a deterministic (

T=0.0

) single-seed reference. pf

=

parse-fail rate. § CoT-strict

=

free-form CoT trace with regex parsing of FINAL_ANSWER: ; CoT-structured

=

same prompt instructing the LLM to emit only the structured answer line, no chain of thought. The deterministic

T=0.0

reference for each multi-seed row stays within

\pm 0.04

MCC of the row’s mean for both zero-shot models across all 4 events; for Qwen CoT-v2 strict the largest deterministic-vs-mean gap is

0.19

MCC on CLN (deterministic

-0.130

, sampling mean

+0.061\pm 0.023

), illustrating that single-seed deterministic CoT-strict numbers can swing sharply because of seed-dependent parse-fail subsets and the resulting non-random evaluable mask; this is the strongest single argument in our paper for the multi-seed reporting requirement on LLM CoT. The single-seed M&A-specific-prompt Qwen-zs value of

+0.115

in Appendix E.4 uses a different prompt template that biases the prediction distribution toward DOWN (predicted-UP rate

0.355

vs.

0.77

under the common cross-event prompt here); both are honestly reported and the disagreement quantifies LLM prompt-sensitivity on this task.

Figure 9: EDT (2020–2021) within-quarter audit. Each panel shows temporal (blue) vs. 5-seed random (orange) MCC for four chronological 70/15/15 quarters. Within-quarter the inflation ratios shrink to LR

\approx 15.7\times

, RF

\approx 3.2\times

, GB

\approx 1\times

, separating leakage from regime-shift (compare to the full-corpus headline ratios

1.7/28.9/2.8\times

in Table 3 ).

Protocol	$n_{\text{train}}$	$n_{\text{test}}$	MCC	$p_{\text{two}}$
A. proprietary $\to$ proprietary (sanity)	731	786	$+0.147$	$0.0001$
B. proprietary $\to$ FNSPID (cross)	731	4,235	$-0.016$	$0.30$
C. FNSPID $\to$ FNSPID (within)	30,070	4,235	$-0.011$	$0.46$
D. FNSPID $\to$ proprietary (reverse)	30,070	786	$-0.088$	$0.016$
E. joint $\to$ joint	30,801	5,021	$-0.017$	$0.24$

Appendix

Table 39: FNSPID cross-corpus 5-protocol matrix, all at paper-authoritative HP and 10K-permutation

p_{\text{two}}

, executed on the GPU v8 environment (Python 3.11, scikit-learn 1.5). Protocol A reproduces the paper headline (0.138 in Sec. 7.2 ) to within 0.009 (one-sided

p_{\text{perm}}=0.0001

in both runs; minor drift reflects the cross-version sklearn solver). Protocol B (the headline cross-corpus probe) is a clean null. Protocol C is also null. Protocol D is anti-correlated and significant, indicating substantive domain shift between FNSPID 2009–2020 US M&A reporting and proprietary 2024–2025 European-tilted M&A reporting.

Protocol	$n_{\text{tr}}$	$n_{\text{te}}$	MCC (5-seed)
C. FNSPID $\to$ FNSPID (within)	30,070	4,235	$+0.007\pm 0.023$
B. proprietary $\to$ FNSPID (forward)	731	4,235	$+0.000\pm 0.017$
D. FNSPID $\to$ proprietary (reverse)	30,070	786	$+0.045\pm 0.018$
E. joint $\to$ FNSPID (joint)	30,801	4,235	$-0.011\pm 0.018$
E. joint $\to$ proprietary (joint)	30,801	786	$+0.070\pm 0.036$

Appendix

Table 40: FinBERT-tone full-parameter FT cross-corpus matrix, 5 seeds at the same HP as the within-FNSPID corroboration. Protocol C (within-FNSPID) reproduces the TF-IDF null; Protocol B (forward) is a clean null consistent with TF-IDF; Protocol D (reverse) flips the sign of the TF-IDF cell (

+0.045

deep vs.

-0.088

TF-IDF,

\Delta=0.133

) and Protocol E (joint) recovers about half the in-domain proprietary signal (

+0.070

vs. headline

+0.138

n_{\text{te}}=786

). The reverse and joint cells indicate FinBERT’s pretrained representations carry transferable structure that the bag-of-words lexicon does not; even so, no off-corpus protocol reaches the in-domain headline. References: v8 script 47 for C; v9 scripts 49–51 for B, D, E.

Temporal Data Leakage

Temporal data leakage occurs when models use information unavailable at prediction time, making performance estimates and forecasts misleading.

See all 35 papers

Beyond Sentiment: Structured Information Extraction from Financial News
Jul 30, 2026Daohan Zhu, Sitong Ge, Ruofei Wang +4Financial Sentiment Analysis Document Information Extraction
School of ASEE, Beihang University, Beijing, China · School of BME, Beihang University, Beijing, China · CAIR and CECS, VinUniversity, Hanoi, Vietnam
Can News Predict the Market? Limits of Zero-Shot Financial NLP and the Role of Explainable AI
Jun 10, 2026Ali M Karaoglu, Shreyank N GowdaFinancial Forecasting Zero-Shot Text Classification
School of Computer Science, University of Nottingham, Nottingham, United Kingdom
Reading the News Without Hindsight: NoLBERT Topics and the Cross-Section of Stock Returns
Oct 15, 2026Kevin Foley, Jonathan Hartadi, Shivesh Prakash +2Financial Forecasting Stock Price Forecasting
University of California, Berkeley, Berkeley, California, USA

Jul 30, 2026J/K move · Enter open · S save

Beyond Sentiment: Structured Information Extraction from Financial News

Daohan Zhu, Sitong Ge, Ruofei Wang, Honggu Chen+3

School of ASEE, Beihang University, Beijing, China · School of BME, Beihang University, Beijing, China · CAIR and CECS, VinUniversity, Hanoi, Vietnam

Financial Sentiment Analysis Document Information Extraction LLM Information Extraction Finance

Open paper AskPDF arXiv

Financial sentiment analysis has become a standard component in news-driven stock prediction, yet it reduces rich, multi-dimensional news articles to a single polarity score. We hypothesize that financial news encodes multiple orthogonal information dimensions---event type, impact scope, temporal horizon, and semantic confidence---that sentiment alone cannot capture, and that these dimensions carry independent predictive value. To test this hypothesis, we propose a structured information extraction framework that leverages LLaMA-3.1-70B to extract six semantic dimensions from financial news. Through large-scale experiments on 41,618 news--stock pairs from the FNSPID dataset, we find that (i) FinBERT sentiment features exhibit strong predictive power under nonlinear models (F1=0.576) but substantially weaker performance under linear models (F1=0.230), revealing a highly nonlinear sentiment--return relationship; (ii) LLM-extracted structured features, while individually weaker, capture information orthogonal to sentiment, as evidenced by a 53.5% systematic disagreement rate between the two approaches; and (iii) combining both signal sources yields F1=0.600, significantly outperforming either alone (

p < 0.0001

), with consistent improvements across all seven event types. Ablation experiments confirm that non-sentiment structural dimensions (event type, impact subject, time horizon, confidence) independently contribute

Δ\text{F1} = +0.019

beyond FinBERT alone. Feature importance analysis reveals balanced contributions from all six extracted dimensions (14--21%), demonstrating that compressing news into a single sentiment score incurs substantial information loss. Our results suggest that the sentiment--semantics decoupling in financial text is systematic and exploitable, opening a new direction for multi-dimensional financial NLP.

Abstract

Temporal Data Leakage

Explore similar work

Beyond Sentiment: Structured Information Extraction from Financial News

Can News Predict the Market? Limits of Zero-Shot Financial NLP and the Role of Explainable AI

Reading the News Without Hindsight: NoLBERT Topics and the Cross-Section of Stock Returns