Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English

Jophin John; Michael Hoffmann; Piroska Lendvai; Florent Dufour

cs.CLOct 13, 2026

Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English

Authors: Jophin John, Michael Hoffmann, Piroska Lendvai, Florent Dufour

Organizations: Leibniz Supercomputing Centre (LRZ), Garching, Germany · Bavarian Academy of Sciences and Humanities, Munich, Germany · TUM University Hospital and Technical University of Munich, Munich, Germany

Abstract

Post-training quantisation reduces the memory cost of large language models, but it remains unclear whether compression preserves language skills of regional varieties as reliably as their standard-language counterparts. We study this question for Bavarian, Standard German, and English using BAVGROUND, a multiple-choice benchmark, where 206 source questions are aligned across all three languages. Across 14 open-weight 7B--10B instruction models, we compare BF16 with 8-bit GPTQ and two 4-bit settings, GPTQ and AWQ. Eight-bit GPTQ remains close to BF16 and largely preserves the relative behaviour of the three languages, while 4-bit quantisation was found to be much less stable. The two 4-bit methods reduce average accuracy by about 1.4--1.5 percentage points and change 11.9--16.0% of BF16 decisions, with changes concentrated on examples where the BF16 model is already uncertain. This instability does not translate into a consistent Bavarian disadvantage. Although Bavarian, a low-resource dialect, has lower baseline accuracy than German and English, the language most affected at 4 bits varies across models and quantisation methods. Calibration further shapes these cross-language effects: the relative Bavarian--German effect changes with calibration language, while changing the calibration source has little average effect on GPTQ-4 but can affect AWQ-4 much more strongly. Free-form evaluation on five models further shows that 4-bit compression can degrade generated responses even in settings where multiple-choice accuracy changes only modestly, while 8-bit GPTQ remains comparatively stable. Overall, our results show that quantisation robustness for regional language varieties depends on the model, quantisation method, calibration data, and evaluation setting, and cannot be measured by aggregate multilingual accuracy alone.

Figures & tables

Model	BF16	GPTQ-8	GPTQ-4	AWQ-4	Best
Aya Expanse 8B	58.7	58.4 (-0.3)	53.2 (-5.5)	58.7 (+0.0)	BF16
EuroLLM 9B	68.9	69.4 (+0.5)	65.2 (-3.7)	64.7 (-4.2)	GPTQ-8
GENBA-10B	57.6	57.8 (+0.2)	51.0 (-6.6)	56.5 (-1.1)	GPTQ-8
Gemma 2 9B	55.2	55.5 (+0.3)	55.8 (+0.6)	58.4 (+3.2)	AWQ-4
Granite 3.0 8B	62.8	62.6 (-0.2)	62.5 (-0.3)	57.1 (-5.7)	BF16
LLaMmlein 7B	57.1	57.8 (+0.6)	57.0 (-0.2)	58.7 (+1.6)	AWQ-4

Table 1: Primary BavGround accuracy under post-training quantisation. Scores are percentages under option-text likelihood evaluation. Values in parentheses are absolute changes in percentage points relative to the same model in BF16. The Best column denotes the highest-accuracy precision setting for each model, including BF16.

Setting	$\Delta$ BAR	$\Delta$ DE	$\Delta$ EN	$A_{\mathrm{BAR-DE}}$	$A_{\mathrm{BAR-EN}}$
GPTQ-8	-0.1	0.0	+0.2	-0.1	-0.3
GPTQ-4	-1.9	-0.5	-1.9	-1.4	0.0
AWQ-4	-1.1	-2.1	-1.1	+1.0	0.0

Table 2: Language-specific accuracy changes from BF16, macro-averaged across the 14 models. Values are percentage points. BAR–DE and BAR–EN compare the Bavarian change with German and English, respectively; negative values indicate greater degradation in Bavarian.

	GPTQ-4		AWQ-4
Calibration	$\Delta$ BAR	BAR–DE	$\Delta$ BAR	BAR–DE
Balanced	-1.82	-0.85	+0.97	+4.25
English only	-2.55	-1.09	+1.09	+3.88
German only	-3.52	-3.16	-1.33	+3.28
Bavarian only	-0.36	+0.12	-0.36	+1.82

Table 3: Calibration-language sensitivity in the focused four-model ablation. Values are mean percentage-point changes from BF16. Negative BAR–DE values indicate greater Bavarian degradation. Full results are reported in Appendix A.8 .

	Low margin (D1)		High margin (D10)
Setting	Change	Loss	Change	Loss
GPTQ-8	10.16	3.23	0.00	0.00
GPTQ-4	39.95	15.94	1.62	0.69
AWQ-4	43.42	16.28	1.27	0.23

Table 4: Prediction instability by BF16 selected-answer margin. D1 and D10 denote the lowest- and highest-margin deciles, respectively. “Change” is the percentage of paired predictions that differ from BF16, while “Loss” is the percentage of all paired rows that are BF16-correct and become incorrect after quantisation. Mean BF16 margins are 0.025 in D1 and 3.338 in D10.

Setting	$N$	BF16 win	Quant. win	Tie	Not worse	Net quant.
GPTQ-8	3047	13.1	10.8	76.1	86.9	-2.3
GPTQ-4	3049	36.2	29.8	33.9	63.7	-6.4
AWQ-4	3048	45.7	28.1	26.1	54.2	-17.6

Table 5: Blinded pairwise comparison of BF16 and quantised free-form responses across five models and all BavGround instances.

N

is the number of successfully parsed comparisons. “Not worse” combines quantised wins and ties; net preference is the quantised-win rate minus the BF16-win rate.

Figure 1: Quality–memory frontier across completed model and precision settings. Points show BavGround multiple-choice accuracy against peak reserved GPU memory.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$\Delta$ vs. BF16	95% CI	Source clusters	Paired rows
GPTQ-8	+0.03	[-0.14, +0.21]	206	8,652
GPTQ-4	-1.47	[-2.21, -0.72]	206	8,652
AWQ-4	-1.42	[-2.21, -0.61]	206	8,652

Appendix

Table 6: Aggregate accuracy change under quantisation, measured in percentage points relative to each model’s BF16 baseline. Confidence intervals are 95% percentile intervals from 5,000 bootstrap samples clustered by source question.

Model	BF16	Best quantised	$\Delta$
Qwen2 7B Instruct	63.8	63.6 (GPTQ-8)	-0.2
Qwen2.5 3B Instruct	60.7	60.4 (GPTQ-8)	-0.3
Qwen2.5 7B Instruct	63.1	65.4 (GPTQ-4)	+2.3
Qwen3 4B	51.9	51.6 (GPTQ-8)	-0.3
Qwen3 8B	48.1	56.6 (GPTQ-4)	+8.6

Appendix

Table 7: Focused Qwen-family results on BavGround . Scores are overall multiple-choice accuracy (%). “Best quantised” selects the highest observed quantised configuration for each checkpoint.

Setting	Contrast	Estimate	90% CI	95% CI	Equivalent
GPTQ-8	BAR–DE	-0.07	[-0.73, +0.59]	[-0.83, +0.69]	Yes
GPTQ-8	BAR–EN	-0.35	[-1.04, +0.31]	[-1.21, +0.45]	Yes
GPTQ-4	BAR–DE	-1.42	[-3.43, +0.59]	[-3.81, +0.90]	No
GPTQ-4	BAR–EN	+0.00	[-2.08, +2.18]	[-2.46, +2.67]	No
AWQ-4	BAR–DE	+1.01	[-1.35, +3.40]	[-1.94, +3.88]	No
AWQ-4	BAR–EN	+0.03	[-2.32, +2.50]	[-2.77, +3.02]	No

Appendix

Table 8: Dialect-gap amplification under a crossed bootstrap over models and source questions. Values are percentage points. Negative estimates indicate greater Bavarian degradation. Practical equivalence requires the 90% confidence interval to lie entirely within the

\pm 2

percentage-point region.

Setting	Contrast	$\pm 1$ pp	$\pm 2$ pp	$\pm 3$ pp
GPTQ-8	BAR–DE	Yes	Yes	Yes
GPTQ-8	BAR–EN	No	Yes	Yes
GPTQ-4	BAR–DE	No	No	No
GPTQ-4	BAR–EN	No	No	Yes
AWQ-4	BAR–DE	No	No	No
AWQ-4	BAR–EN	No	No	Yes

Appendix

Table 9: Sensitivity of practical-equivalence conclusions to alternative equivalence margins. Entries indicate whether the 90% crossed-bootstrap confidence interval lies fully within the stated percentage-point region.

Setting	BAR $<$ DE	Similar	BAR $>$ DE
GPTQ-8	6	2	6
GPTQ-4	9	1	4
AWQ-4	6	1	7

Appendix

Table 10: Direction of the per-model Bavarian–German quantisation effect. BAR

<

DE indicates greater Bavarian degradation, while BAR

>

DE indicates greater German degradation. Values within

\pm 0.25

percentage points are grouped as similar.

Model	Quantiser	$\Delta$ BAR	$\Delta$ DE	$\Delta$ EN	BAR-DE	BAR-EN
Aya Expanse 8B	GPTQ-8	+0.0	-0.5	-0.5	+0.5	+0.5
Aya Expanse 8B	GPTQ-4	-7.8	+0.0	-8.7	-7.8	+1.0
Aya Expanse 8B	AWQ-4	+0.0	+2.4	-2.4	-2.4	+2.4
EuroLLM 9B	GPTQ-8	+1.0	+0.5	+0.0	+0.5	+1.0
EuroLLM 9B	GPTQ-4	-4.9	-1.9	-4.4	-2.9	-0.5
EuroLLM 9B	AWQ-4	-1.5	-8.3	-2.9	+6.8	+1.5

Appendix

Table 11: Per-model language-specific accuracy changes under balanced trilingual calibration.

\Delta

BAR,

\Delta

DE, and

\Delta

EN denote percentage-point changes from each model’s BF16 accuracy. BAR–DE and BAR–EN compare these changes across languages; negative values indicate greater Bavarian degradation.

Domain	Prompt BAR/DE	Prompt BAR/EN	Prompt DE/EN	Gold BAR/DE	Gold BAR/EN
Building & Sacred Heritage	1.04	1.18	1.14	1.09	1.30
Language	1.00	1.10	1.09	0.94	1.06
Politics	1.03	1.17	1.14	1.01	1.31
Historical	1.05	1.17	1.12	1.07	1.20
Landscape	1.03	1.14	1.11	1.05	1.22
Living Traditions & Customs	1.03	1.18	1.15	1.03	1.38

Appendix

Table 12: Relative tokenisation length across aligned language versions, computed from BF16 runs and paired by model and source question. Ratios above 1 indicate that the numerator language requires more tokens. Gold denotes the gold answer option.

Subset	BF16	$\Delta$ GPTQ-8	$\Delta$ GPTQ-4	$\Delta$ AWQ-4
GEN	63.4	+0.2	-1.4	-1.4
GRD	57.9	-0.1	-1.5	-1.4

Appendix

Table 13: Performance on general ( gen ) and source-grounded ( grd ) BavGround questions, macro-averaged across the 14 models. BF16 reports absolute option-text accuracy (%); quantised columns report the mean absolute change from each model’s BF16 score in percentage points.

Domain	BF16	$\Delta$ GPTQ-8	$\Delta$ GPTQ-4	$\Delta$ AWQ-4
Language	40.5	-0.4	-2.2	-1.9
Building & Sacred Heritage	57.1	+0.0	-2.4	-3.1
Politics	61.7	-0.3	+0.4	-0.1
Arts & Identity	62.7	+0.2	-1.6	-0.9
Culinary	62.8	+0.3	+0.0	+0.1
Landscape	63.4	+0.3	-2.4	-1.6

Appendix

Table 14: Domain-level BavGround performance averaged across the 14 models. BF16 reports absolute option-text accuracy (%); quantised columns report absolute percentage-point changes from each model’s BF16 score before averaging.

Model	Checkpoint identifier
GENBA-10B	GENBA-10B-instruct Hoffmann et al. (2025)
Llama 3.1 Instruct	meta-llama/Llama-3.1-8B-Instruct
EuroLLM Instruct	utter-project/EuroLLM-9B-Instruct
Qwen2.5 Instruct	Qwen/Qwen2.5-7B-Instruct
Mistral Instruct v0.3	mistralai/Mistral-7B-Instruct-v0.3
Gemma 2 IT	google/gemma-2-9b-it

Appendix

Table 15: Exact checkpoint identifiers used in the quantisation study.

Calibration	Documents	EN tokens	DE tokens	BAR tokens
Balanced	1,383	83,416	83,423	83,607
EN-only	1,369	250,238	–	–
DE-only	1,387	–	250,291	–
BAR-only	1,363	–	–	250,106

Appendix

Table 16: Calibration corpus statistics for seed 42. Token counts use whitespace-delimited tokens rather than model-specific tokenisers. Language pools are sampled independently, without aligned source-example IDs. Exact normalised-text checks found no overlap with BavGround questions, answer options, or recorded source references.

Setting	Calibration	$\Delta$ EN	$\Delta$ DE	$\Delta$ BAR	BAR–DE	BAR–EN
GPTQ-4	Balanced	-2.91	-0.97	-1.82	-0.85	+1.09
GPTQ-4	English only	-2.79	-1.46	-2.55	-1.09	+0.24
GPTQ-4	German only	-0.12	-0.36	-3.52	-3.16	-3.40
GPTQ-4	Bavarian only	-0.85	-0.49	-0.36	+0.12	+0.49
AWQ-4	Balanced	-1.70	-3.28	+0.97	+4.25	+2.67
AWQ-4	English only	-1.33	-2.79	+1.09	+3.88	+2.43

Appendix

Table 17: Full calibration-language ablation. All values are mean accuracy changes in percentage points relative to BF16. Positive BAR–DE indicates that German degrades more than Bavarian; negative values indicate greater Bavarian degradation.

Setting	Calibration	Neg.	Similar	Pos.	Range
GPTQ-4	Balanced	3	0	1	$[-2.9,+1.9]$
GPTQ-4	English only	2	0	2	$[-3.9,+1.0]$
GPTQ-4	German only	4	0	0	$[-6.8,-0.5]$
GPTQ-4	Bavarian only	1	1	2	$[-1.5,+1.0]$
AWQ-4	Balanced	0	1	3	$[0.0,+6.8]$
AWQ-4	English only	0	1	3	$[0.0,+6.3]$

Appendix

Table 18: Direction of the per-model Bavarian–German quantisation effect in the four-model calibration ablation. Negative values indicate greater Bavarian degradation and positive values greater German degradation. Values within

\pm 0.25

percentage points are grouped as similar; the final column reports the per-model BAR–DE range.

Model	Quantiser	Balanced	EN-only	DE-only	BAR-only
EuroLLM 9B	GPTQ-4	-2.91	-2.43	-6.80	+0.97
GENBA-10B	GPTQ-4	+1.94	+0.97	-0.97	+0.97
Llama 3.1 8B	GPTQ-4	-0.49	+0.97	-0.49	0.00
Qwen2.5 7B	GPTQ-4	-1.94	-3.88	-4.37	-1.46
EuroLLM 9B	AWQ-4	+6.80	+5.34	+8.74	+5.83
GENBA-10B	AWQ-4	+6.31	+6.31	+2.91	+3.40

Appendix

Table 19: Per-model Bavarian–German dialect-gap amplification (in percentage points) under different calibration-language conditions at seed 42. Values are computed as

\Delta_{\mathrm{BAR}}-\Delta_{\mathrm{DE}}

relative to each model’s BF16 baseline. Negative values indicate greater Bavarian degradation, while positive values indicate greater German degradation. Results are exploratory because calibration corpora were independently sampled and are not content-aligned.

Focused: 4 models, 3 seeds, all calibration conditions
Setting	Instruction	Wikimedia	$\Delta$
GPTQ-4	62.25	62.43	+0.18
AWQ-4	61.86	63.11	+1.25
Overall	62.06	62.77	+0.72
Finite paired runs: balanced calibration, seed 42
GPTQ-4 (14 models)	58.55	58.16	-0.39

Appendix

Table 20: BavGround accuracy under translated instruction-style and natural Wikimedia calibration. The focused comparison covers four models, four calibration-language conditions, two quantisers, and three seeds. The broader comparison uses balanced calibration at seed 42; the invalid Gemma 2 AWQ-4 Wikimedia run is excluded from both paired AWQ conditions.

\Delta

denotes Wikimedia minus instruction-style accuracy in percentage points.

Predictor	Log-odds	OR	$z$
BF16 selected-answer margin	-3.263	0.038	-28.18
BF16 correct-answer margin	-0.040	0.961	-0.92
Prompt tokens	+0.243	1.275	+1.87
Gold-answer tokens	-0.549	0.577	-3.80
German vs. Bavarian	-0.065	0.938	-1.13
English vs. Bavarian	+0.002	1.002	+0.02

Appendix

Table 21: Item-level model predicting whether quantisation changes the selected answer. Continuous predictors are standardised. Bavarian and AWQ-4 are the categorical reference levels. The model includes model and source-question effects. OR denotes odds ratio.

Domain	Setting	$\Delta$ Acc.	Loss	Gain	Change
Building & Sacred Heritage	GPTQ-8	+0.0	0.3	0.3	1.0
Building & Sacred Heritage	GPTQ-4	-2.4	5.7	3.3	12.6
Building & Sacred Heritage	AWQ-4	-3.1	7.5	4.4	18.0
Language	GPTQ-8	-0.4	0.5	0.1	1.0
Language	GPTQ-4	-2.2	5.6	3.4	15.2
Language	AWQ-4	-1.9	7.3	5.4	20.7

Appendix

Table 22: Item-level prediction changes relative to BF16, aggregated by domain.

\Delta

Acc. is the mean accuracy change in percentage points. Loss and Gain are correct-to-incorrect and incorrect-to-correct transition rates (%), respectively; Change is the overall answer-change rate (%), including correctness-preserving answer changes.

Item	Lang.	Domain	Losses	Gains	Loss Rate	Change Rate
politics_012	BAR	Politics	9	2	21.4	26.2
building_and_sacred_heritage_025	BAR	Building/Sacred Heritage	9	0	21.4	23.8
culinary_002	EN	Culinary	8	3	19.0	28.6
culinary_022	EN	Culinary	8	0	19.0	21.4
language_024	DE	Language	8	1	19.0	21.4
historical_007	BAR	Historical	7	5	16.7	35.7

Appendix

Table 23: Ten of the most quantisation-sensitive source-item/language slices, ranked by loss rate and then answer-change rate across paired quantised comparisons. Losses count BF16-correct

\rightarrow

quantised-incorrect transitions; gains count the reverse. Rates are percentages across all paired model–quantisation comparisons for that item-language slice.

Language	Setting	$N$	Quant. not worse	Net quant.
BAR	GPTQ-8	997	86.1	-2.5
BAR	GPTQ-4	999	59.8	-7.8
BAR	AWQ-4	998	51.2	-18.8
DE	GPTQ-8	1021	86.8	-1.3
DE	GPTQ-4	1022	59.7	-13.7
DE	AWQ-4	1021	52.2	-21.7

Appendix

Table 24: Full-benchmark free-form preservation by language and quantiser, using primary comparisons only. Reversed-order position-audit judgments are excluded. “Quant. not worse” is the quantised-win plus tie rate; net quantised preference is the quantised-win rate minus the BF16-win rate. Values except

N

are percentages. The 9,144 primary comparisons agree with Tables 5 and 25 .

Model	$N$	Quant. not worse	Net quant.
EuroLLM 9B	1850	77.2	-5.8
Mistral 7B	1824	75.3	-5.9
Llama 3.1 8B	1798	64.3	-6.9
Qwen2.5 7B	1835	58.1	-12.1
GENBA-10B	1837	66.3	-13.0

Appendix

Table 25: Model-level free-form quantisation retention, pooled over GPTQ-8, GPTQ-4, and AWQ-4. Values except

N

are percentages.

Model	Setting	Acc.	Mem.	Red.	Items/s
EuroLLM 9B	BF16	68.9	23.2	0	2.42
EuroLLM 9B	GPTQ-8	69.4	14.2	38	0.98
EuroLLM 9B	GPTQ-4	65.2	10.2	56	1.31
Mistral 7B v0.3	BF16	66.8	19.2	0	3.11
Mistral 7B v0.3	GPTQ-8	66.3	11.4	41	1.69
Mistral 7B v0.3	GPTQ-4	67.2	7.9	59	1.70

Appendix

Table 26: Representative quality, memory, and throughput measurements. Acc. is BavGround option-text accuracy (%); Mem. is peak reserved GPU memory in GB; Red. is the reduction relative to the BF16 version of the same model; and Items/s is batch-size-one evaluation throughput on the GH200/Transformers inference setup.

Criterion	Configurations
$\geq 50\%$ memory reduction, within 1 pp of BF16	11
$\geq 50\%$ memory reduction, within 2 pp of BF16	15
Quantised accuracy $\geq$ BF16 accuracy	19
GPTQ-4 accuracy $<$ GPTQ-8	9/14
GPTQ-4 accuracy $\geq$ GPTQ-8	5/14

Appendix

Table 27: Summary of quality–memory trade-offs across the profiled configurations. Accuracy comparisons are descriptive; “within 1 pp” and “within 2 pp” refer to absolute percentage-point differences from each model’s BF16 accuracy.

Component	Captured setting
Accelerator	NVIDIA GH200 reported by nvidia-smi ; 97,871 MiB device memory (approximately 95.6 GiB), driver 580.95.05, and 900 W power limit.
Operating system	Linux on aarch64 ; kernel 6.2.0-1015-nvidia-64k .
Python	Python 3.10.12.
Core packages	torch 2.13.0, transformers 5.14.1, accelerate 1.14.0, GPTQModel 7.3.2, bitsandbytes 0.50.0, numpy 2.2.6, pandas 2.3.3, scipy 1.15.3, nvidia-ml-py 13.610.43.
Inference backend	Hugging Face Transformers with automatic device mapping. Quantised checkpoints are loaded as Hugging Face-compatible GPTQ or AWQ checkpoints.

Appendix

Table 28: Hardware and software details captured in run metadata.

Aspect	Setting
Evaluation config	configs/evaluation/bavground_full.yaml ; languages en , de , and bar ; protocols option_text_avg , letter , and generate_letter_and_text .
Runtime parameters	Batch size 1, maximum sequence length 4096, maximum generation length 24 tokens, random seed 42, prompt saving enabled, and resumable runs enabled.
Quantisation settings	BF16 reference plus GPTQ-8, GPTQ-4, and AWQ-4. Quantised runs use group size 128, asymmetric zero-point quantisation where supported, and source precision for embeddings and the language-model head.
Calibration	Balanced English–German–Bavarian calibration condition, seed 42, target size approximately 250k tokens, and calibration sequence length 2048. No exact normalised-text overlap was detected with BavGround questions, answer options, or recorded source references.
Runtime telemetry	Run metadata records model-loading duration, peak allocated and reserved GPU memory after loading and inference, average and maximum power draw, total energy, phase-level energy for model loading and inference, and nvidia-smi snapshots before and after execution.

Appendix

Table 29: Common evaluation, calibration, and runtime settings.

Parameter	Main-matrix setting
GPTQ weight precision	8-bit or 4-bit
AWQ weight precision	4-bit
Group size	128
Calibration budget	$\sim$ 250k tokens
Calibration sequence length	2048
Calibration seed	42

Appendix

Table 30: Common quantisation settings used in the main evaluation matrix. Backend-specific saved checkpoint settings and any architecture-dependent exceptions are reported in the released run manifest.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 362 papers

Aug 25, 2026

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its effects, however, comes from English benchmarks. It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla, and this gap is what we address here. We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) across five Bangla natural language understanding benchmarks (Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN), using zero-shot evaluation through lm-evaluation-harness. To our knowledge this is the first controlled comparison of quantization formats on Bangla NLU. The three families do not respond the same way: GPT-OSS loses up to 57.35% accuracy on reasoning-heavy tasks under GGUF-W8A16, while Qwen and LLaMA hold steady under GPTQ, and in a few cases the quantized version edges out the full-precision one. BoolQ-BN, a comprehension task, stays stable across all three families regardless of format. Taken together, these results suggest quantization can work well for Bangla deployment, but the choice of architecture and quantization method matters more than the bit width alone. We discuss what this means for practitioners choosing a model to run on constrained hardware.

Ismail Hossain, Nafi Ullah Shafin, Mohammad Abdullah Al Mumin

Institute of Information and Communication Technology (IICT) Shahjalal University of Science and Technology (SUST), Sylhet, Bangladesh

LLM Quantization Post-Training Quantization LLM Evaluation Bengali NLP Multilingual Language Model Evaluation

Ask PDF

Aug 12, 2026

Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages

Aggressive quantization disproportionately harms multilingual capability: in the sub-4B INT3 GPTQ regime, we measure 2-4x larger perplexity degradation on non-English languages than on English. We propose Language-Conditional Dequantization (LCD), a post-hoc method that attaches per-language rank-2 LoRA corrections to the linear layers of an already-quantized model, adding 0.12% parameters per language and training in under 20 minutes on a single GPU. Across Qwen2.5-3B and Llama-3.2-3B, LCD recovers 70-83% of the perplexity gap for non-Latin script languages and 17-28% of the GlobalMMLU accuracy gap, outperforming a language-agnostic correction of equal capacity by 3-9 points on typologically distant languages and a data-free low-rank baseline (LQER) by an order of magnitude. We further identify a perplexity-accuracy disconnect and trace it to where quantization concentrates damage: early-depth errors (Llama) propagate downstream and resist local correction, while late-depth errors (Qwen) do not. A layer-restricted variant of LCD validates this mechanism directly.

Nirmal Thomas

Prathama International

LLM Quantization Post-Training Quantization

Ask PDF

Jul 23, 2026

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked. We find its principal side effect is increased bias that standard safety evaluation misses. Holding the model, its training, and the prompts fixed, a quantized model still refuses harmful requests, still avoids over-refusing benign prompts, and still selects the unbiased multiple-choice answer. Yet asked an open-ended question, the same model volunteers stereotypes in all eight languages we probe, in roughly one in four open-ended answers under an independent judge (~24% to ~27% across the compression ladder): it passes every standard check and still reaches users measurably more biased. The selective gap is a robust finding; whether open-ended bias further increases with compression is less certain, sensitive to the judge that scores it. We address both with QuantiBias, a benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates. Across two backbone models (Qwen and Gemma), a five-family screen, and eight benchmarks, quantizers allocate their extra precision by capability data that carries no bias-prevention signal, and reasoning before answering roughly halves the effect on some families while doing nothing on others. A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.

Emilio Ferrara

Thomas Lord Department of Computer Science University of Southern California

Open-Ended Generation LLM Quantization Language Model Generation Evaluation Social Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Abstract

LLM Quantization

Explore similar work

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs