Damage Predicts Recovery: When Calibration Data Matters in Compressing Financial LLMs

Junyi Ye; Mengjia Yu; Debapriya Hazra; Guiling Wang

cs.CLAug 12, 2026

Damage Predicts Recovery: When Calibration Data Matters in Compressing Financial LLMs

Authors: Junyi Ye, Mengjia Yu, Debapriya Hazra, Guiling Wang

Organizations: School of Computing, Montclair State University Montclair, New Jersey, USA · Department of Computer Science, New Jersey Institute of Technology Newark, New Jersey, USA

Abstract

Post-training quantization and pruning rely on a small calibration corpus. Whether specialized domains such as finance require domain-matched calibration data remains unsettled. We argue that the answer depends on the task-level damage caused by compression rather than on domain mismatch. If compression preserves the target capability, changing the calibration corpus has little effect. If compression causes large losses, task-formatted calibration can recover part of the loss. We test this hypothesis across two model families, six compression configurations, three token-matched calibration corpora, and ten financial classification and numerical question-answering tasks. The results support this hypothesis. Quantization largely preserves task performance, and calibration choice has little effect in this case. Pruning reduces numerical QA accuracy by over 40 points. In these damaged settings, another generic corpus does not help, while FinMix, a mixture of financial task examples, recovers a large part of the loss. The link between damage and recovery holds across model families and scales. These findings support a practical rule. Measure task-specific compression damage first, and construct specialized calibration data only when the damage is large.

Figures & tables

Corpus	Content	Role	Seqs	Tokens
WikiText	encyclopedia text	reference	128	262k
C4	web text	generic control	128	262k
FinMix	financial tasks	matched corpus	2042	260k

Table 1. Calibration corpora used in the experiments. All calibration-based methods consume each corpus in full.

Task	Type	$n$	Balance	Num.	Metric
FPB ( 9 )	3-cls sentiment	970	59/29/12	none	wF1
FOMC ( 11 )	3-cls stance	496	49/26/25	none	wF1
Headlines ( 13 )	binary attribute	1500	67/33	low	wF1
FinRED ( 12 )	29-cls relation	1500	max 13	none	wF1
FinCausal ( 10 )	2-cls causality	1500	94/6	low	mF1
BigData22 ( 14 )	2-cls movement	1472	55/45	low	wF1

Table 2. Evaluation tasks.

n

= canonical test-set size. Balance = class distribution (%); for FinRED (29 relation types) we report only the largest class share, and the three free-form QA tasks have no fixed label set (–). Num. = degree of numerical dependence. Classification tasks report weighted F1, except FinCausal, which reports macro-F1. QA tasks report accuracy (judge-verified for the generative tasks, marked

\dagger

Calibration: WikiText			Classification							Numerical QA
	Method	PPL $\downarrow$	FPB $\uparrow$	FOMC $\uparrow$	Headlines $\uparrow$	FinRED $\uparrow$	FinCausal $\uparrow$	BigData22 $\uparrow$	Avg. $\uparrow$	CFA $\uparrow$	FinQA $\uparrow$	ConvFinQA $\uparrow$	TAT-QA $\uparrow$	Avg. $\uparrow$
	BF16	7.22	79.0	51.3	67.9	75.2	53.5	49.6	65.4	68.1	43.9	64.7	74.1	62.7
Quantization	RTN	7.96	78.7	50.4	58.1 ∗	73.8	47.4 ∗	43.4 ∗	61.7 ( $\Delta$ 3.7)	62.4 ∗	39.6 ∗	59.1 ∗	69.5 ∗	57.6 ( $\Delta$ 5.1)
	GPTQ	7.49	77.9	51.9	68.0	72.0 ∗	53.8	58.3 ∗	64.7 ( $\Delta$ 0.7)	62.8 ∗	36.9 ∗	58.6 ∗	70.5 ∗	57.2 ( $\Delta$ 5.5)
	AWQ	7.62	76.4 ∗	49.2	68.2	74.0	52.7	49.3	64.1 ( $\Delta$ 1.3)	64.8 ∗	45.8	59.8 ∗	73.5	61.0 ( $\Delta$ 1.7)
	SmoothQuant	7.33	79.0	53.1 ∗	67.7	73.7 ∗	54.3	50.5	65.6 ( $\Delta$ $-0.2$ )	66.8	44.8	63.1	73.0	61.9 ( $\Delta$ 0.8)

Table 3. Performance of Llama-3.1-8B under WikiText calibration, with BF16 as the reference (RTN is calibration-free). Higher task scores are better. Lower PPL is better. Avg. is the unweighted mean within each task family. The parenthetical

\Delta

gives the damage relative to BF16 (BF16

-

compressed), so positive values indicate a loss. The classification mean excludes BigData22. Stars denote significant differences from BF16 (

{}^{*}p{<}.05

Figure 1. Compression damage under WikiText calibration (left), and the change after replacing WikiText with C4 (middle) or FinMix (right). Positive values indicate damage in the left panel and recovery in the other two panels. Stars indicate paired-test significance ( ${}^{*}p{<}.05$ , ${}^{**}p{<}.01$ , ${}^{***}p{<}.001$ ).

Method	Classification	Numerical QA	Overall
SparseGPT	8/9	9/9	17/18
Wanda	9/9	9/9	18/18

Table 4. Pruning recovery across three WikiText draws. Entries give significantly positive comparisons over all task–seed comparisons (

p{<}.05

). All 36 recovery values are positive.

Method	BF16 correct	BF16 wrong	OR [95% CI]
	FinMix repair rate (%)
GPTQ	62.1	15.2	9.08 [6.95, 11.86]
SparseGPT	48.0	16.1	4.62 [3.69, 5.80]
Overall	53.2	15.6	6.07 [5.11, 7.22]

Table 5. FinMix repair rates among errors under WikiText calibration, grouped by whether BF16 answers the same sample correctly. The higher rate for BF16-correct samples indicates recovery of performance lost during compression.

	GPTQ		SparseGPT		Overall
Error type	Errors	Repair (%)	Errors	Repair (%)	Errors	Repair (%)
reasoning	184	63	249	51	433	56
unit/scale	106	66	106	48	212	57
arithmetic	69	52	256	43	325	45
extraction	32	62	89	48	121	52
format	23	61	9	89	32	69

Table 6. Counts and FinMix repair rates for manually categorized, compression-introduced errors across the two numerical tasks. The Errors columns give category counts, and the Repair columns give the percentage corrected by FinMix . Overall combines GPTQ and SparseGPT.

Figure 2. Compression damage and recovery across model families and scales. Panel (a) fits Llama-3.1-8B, panel (b) tests held-out Qwen3-8B, and panel (c) checks Llama-3.1-70B. The black dashed line and shaded area show the fit and its 95% prediction band. The gray dotted line marks full repair.

Classification
Method	Llama-3.1-8B	Qwen3-8B	Llama-3.1-70B
BF16 score	65.4	66.8	70.7
GPTQ	$+0.7/+0.9$	$-0.3/-0.4$	$+0.8/+0.6$
AWQ	$+1.3/+0.3$	$-0.1/+0.3$	$+0.2/-0.5$
SmoothQuant	$-0.2/+0.1$	$+0.1/0.0$	$+0.6/+0.2$
SparseGPT	$+13.6/+11.9$	$+3.5/+3.3$	$+1.0/-0.4$

Table 7. Average BF16 performance and compression damage/recovery (D / R, pp) by method and model. Higher BF16 scores are better.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 366 papers

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization
Mar 17, 2026Francesco Pio Monaco, Elia Cunegatti, Flavio Vella +1LLM Compression LLM Pruning
University of Trento
Scaling Laws for Task-Specific LLM Distillation
Jun 23, 2026Lavinia Ghita, Dhruv Desai, Ioana BoierLLM Pruning LLM Compression
NVIDIA
Capability Scaling-Down Laws for LLM Compression
Oct 1, 2026Xueqi Cheng, Liang Wu, Kelly Wan +2LLM Compression LLM Pruning
Florida State University · Nokia Applied Research

Mar 17, 2026J/K move · Enter open · S save

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

Francesco Pio Monaco, Elia Cunegatti, Flavio Vella, Giovanni Iacca

University of Trento

LLM Compression LLM Pruning LLM Quantization Post-Training Quantization

Open paper AskPDF arXiv

Post-training model compression is essential for enhancing the portability of Large Language Models (LLMs) while preserving their performance. While several compression approaches have been proposed, less emphasis has been placed on selecting the most suitable set of data (the so-called calibration data) for finding the compressed model configuration. The choice of calibration data is a critical step in preserving model capabilities both intra- and inter-tasks. In this work, we address the challenge of identifying high-performance calibration sets for both pruning and quantization by analyzing intrinsic data properties rather than model-specific signals. We introduce ZipCal, a model-agnostic data curation strategy that maximizes lexical diversity based on Zipfian power laws. Experiments demonstrate that our method outperforms standard uniform random sampling across various pruning benchmarks. Notably, it also performs on par, in terms of downstream performance, with a state-of-the-art method that relies on model perplexity. The latter becomes prohibitively expensive for large-scale models and datasets, while ZipCal is on average

\sim

240

\times

faster due to its tractable linear complexity. We make the code and the experiments available at https://github.com/FrancescoMonaco/ZipCal.

Abstract

Language Model Calibration

Explore similar work

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

Scaling Laws for Task-Specific LLM Distillation

Capability Scaling-Down Laws for LLM Compression