Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit

Jakub Prejzner

cs.CLSep 25, 2026

Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit

Authors: Jakub Prejzner

Organizations: Independent Researcher

Abstract

We ask how GGUF quantization affects agentic tool use in Polish and whether the effects generalize across models. We introduce PolAgentBench, a deterministic benchmark with Polish prompts and English tool schemas: a 67-task main suite (15 adversarial probes, 52 hard-tier tasks) and a 46-task arithmetic isolation ladder. Three models span two axes of variation: Bielik-11B-v3.0 and its pruned, distilled child Bielik-Minitron-7B-v3.0 isolate model compression, and Llama-PLLuM-8B adds a change of pretraining family. Each is measured at six precisions, Q8_0 to Q2_K. Only the collapse threshold replicates. (1) All three models fall off a cliff between 3-bit and 2-bit (11B 0.716 to 0.045, 7B 0.463 to 0.149, PLLuM 0.224 to 0.015; paired McNemar p < 0.001 in each), across a fourfold capability spread and both axes. (2) Failure modes do not replicate: at 2-bit the 7B fails long (median 9.1k tokens, 4 steps) while the 11B mostly answers at the first step with a confabulated final answer (37 of 64 failures); PLLuM fails on content across precisions (71.8-92.0% of steps parse). (3) On the arithmetic ladder the unscaffolded rung is a floor, left standing by a rerun that states the no-tool rule; four explicit calls lift the 8-bit 11B from 1/10 to 9/10 and the 7B from 0/10 to 7/10 after format-only failures with the gold value are forgiven, an exploratory effect with eight distinct baseline inputs that does not survive multiplicity correction, while the order-trap arm separates the models at 8-bit (11B 6/6, 7B 0/6). (4) The Polish-versus-English gap is associated with degradation or with task family. We document four artifacts that shaped our conclusions (rounding-hostile gold values, strict answer typing, a no-tool rule the prompt never stated, priority-ordered failure labels), report affected results in strict and corrected form, and release the benchmark, trajectories and commit-stamped artifacts.

Figures & tables

		Q8_0	Q6_K	Q5_K_M	Q4_K_M	Q3_K_M	Q2_K
7B	failures	21	21	21	21	20	25
7B	single category	1	0	4	5	13	7
11B	failures	11	10	11	17	10	25
11B	single category	3	3	4	2	8	3
PLLuM	failures	23	23	24	23	22	25
PLLuM	single category	6	9	6	4	9	5

Table 1: Label determinacy on the 25 arithmetic-graded tasks, repair = off : failures whose tags all fall within one taxonomy class. Full co-occurrence counts in Appendix C .

	Q8_0	Q6_K	Q5_K_M	Q4_K_M	Q3_K_M	Q2_K
7B	0.448 / 0.567	0.463 / 0.567	0.522 / 0.597	0.507 / 0.597	0.463 / 0.463	0.149 / 0.179
11B	0.806 / 0.806	0.836 / 0.851	0.746 / 0.761	0.537 / 0.597	0.716 / 0.716	0.045 / 0.075
PLLuM	0.194 / 0.224	0.179 / 0.209	0.104 / 0.134	0.224 / 0.254	0.224 / 0.269	0.015 / 0.015

Table 2: All-task pass rate (67 tasks) by model and precision, repair = off and on . All three models fall off a cliff between Q3_K_M and Q2_K. Tier-resolved counts in Appendix B .

Panel of Figure 1 — Figure 1: Quantization curves, repair = off . Three very different curves, one shared terminus: every model ends in the same cliff between 3-bit and 2-bit.

	Q8_0	Q6_K	Q5_K_M	Q4_K_M	Q3_K_M	Q2_K
7B	21 / 16 / 4 / 1	21 / 12 / 9 / 0	21 / 12 / 6 / 2	21 / 7 / 13 / 1	20 / 5 / 4 / 11	25 / 6 / 15 / 4
11B	11 / 5 / 4 / 2	10 / 3 / 5 / 2	11 / 4 / 6 / 1	17 / 9 / 7 / 1	10 / 0 / 7 / 3	25 / 5 / 19 / 1
PLLuM	23 / 0 / 23 / 0	23 / 0 / 22 / 1	24 / 4 / 20 / 0	23 / 1 / 22 / 0	22 / 1 / 20 / 1	25 / 0 / 23 / 2

Table 3: Failure modes on the 25 arithmetic-graded tasks, repair = off : total failures, then protocol shape , genuine arithmetic , tool fixation . The rounding artifact fires exactly once across the whole grid, at the 7B’s Q5_K_M point, and is zero in every other cell; that one case is why the three mode counts in that cell sum to 20 rather than 21 (Appendix C ). Labels on the 7B’s 8-bit to 5-bit plateau, at 2-bit everywhere, and throughout the PLLuM row are priority-dominated and indicative only (Section 3.7 ).

		median tokens		median steps
model	precision	pass	fail	pass	fail
7B	Q8_0	3504	12298	3.0	8.0
	Q6_K	3515	12384	3.0	8.0
	Q5_K_M	3577	14065.5	3.0	8.0
	Q4_K_M	3501	10793	3.0	6.0
	Q3_K_M	3509	26833	3.0	8.0

Table 4: Median tokens and steps per trajectory, split by outcome, Bielik pair, repair = off . Tokens are prompt plus completion tokens summed over all completion calls of a trajectory, so the conversation context is counted again at every step. Step budgets are per task, 4 to 10, and are 8 on 29 of the 67 tasks. Full distributions in Appendix F .

		L0 (0)	L0e (0, explicit)	L1 (1)	L2 (2)	L3T (4, trap)	L3N (4, no graph)	total /46
11B	Q8_0	0.00	0.10	0.10	0.40	1.00	0.20 ( 0.90 )	13 (20)
	Q4_K_M	0.00	0.00	0.00	0.10	0.00	0.00 (0.40)	1 (5)
	Q3_K_M	0.00	–	1.00	1.00	0.83	0.70	32
	Q2_K	0.00	–	0.00	0.00	0.00	0.00	0
7B	Q8_0	0.00	0.00	0.00	0.20	0.00	0.00 ( 0.70 )	2 (9)
	Q4_K_M	0.00	0.00	0.00	0.00	0.00	0.10 (0.30)	1 (3)

Table 5: Extended ladder, pass rate per rung, repair = off . L0, L0e, L1, L2, L3N have

n=10

; L3T has

n=6

. L0e is the 2026-09 rerun of L0 with an explicit no-tool sentence in the prompt, run at Q8_0 and Q4_K_M only (identical values at repair = on ); a dash marks cells not rerun. Corrected values (format-only failures carrying the gold value forgiven, Section 3.6 ) in parentheses where they differ; no other cell changes. L0 is 0.00 in all eight Bielik runs at both rates; L0e is 0.10 in one cell and 0.00 in the other three. The total column sums the five original rungs and excludes L0e; the 11B at Q8_0 is therefore 13 strict without it.

model	precision	contrast	rate	$n_{11}$	$n_{10}$	$n_{01}$	$p$ / Holm
11B	Q8_0	L0e vs L3N ( $n=10$ )	strict	0	1	2	1.000 / 1.000
11B	Q8_0	L0e vs L3N ( $n=10$ )	corrected	1	0	8	0.0078 * / 0.094
11B	Q8_0	L0e vs L3T ( $n=6$ )	strict	0	0	6	0.031 * / 0.31
11B	Q4_K_M	L0e vs L3N ( $n=10$ )	strict	0	0	0	1.000 / 1.000
11B	Q4_K_M	L0e vs L3N ( $n=10$ )	corrected	0	0	4	0.125 / 1.000
11B	Q4_K_M	L0e vs L3T ( $n=6$ )	strict	0	0	0	1.000 / 1.000

Table 6: Paired exact McNemar on the ladder with the explicit-prohibition rerun L0e as the baseline, instance slot as pairing unit, repair = off ; L0e has ten slots but eight distinct inputs ( a = e , c = f ). The twelve-row family was fixed at the analysis stage, after the L0e rerun: L0e against L3N at the strict and at the corrected rate, and against L3T, for both Bieliks at Q8_0 and Q4_K_M.

n_{01}

counts instances failed at L0e and passed at the scaffolded rung,

n_{10}

the reverse; the last column gives the nominal

p

and the Holm-adjusted

p

within the family. Asterisks mark nominal

p<0.05

; no contrast survives correction. The original L0-based contrasts, including those at Q3_K_M, and the sensitivity excluding duplicate baseline inputs from both rungs are in Appendix J .

model	precision	structure	with value	difference
7B	Q8_0	0.200	0.200	$0.00$
	Q4_K_M	0.400	0.200	$-0.20$
	Q3_K_M	0.400	0.200	$-0.20$
	Q2_K	0.100	0.000	$-0.10$
11B	Q8_0	1.000	0.800	$-0.20$
	Q6_K	1.000	0.900	$-0.10$

Table 7: Matched twin pairs (

n=10

), repair = off : structure-only pass rate against structure-plus-value. Differences mix a genuine value penalty on coinciding trajectories with disagreements caused by divergent greedy decodes (Section 3.6 ); the

+0.10

at 11B Q4_K_M and the

-0.20

at 11B Q8_0 are entirely of the second kind. The 7B rows are the four precisions of the original grid; at Q6_K and Q5_K_M the 7B scores

0.200/0.200

and

0.400/0.200

, both Q5_K_M discordances being of the second kind.

model	bucket ( $n$ )	Q8_0	Q6_K	Q5_K_M	Q4_K_M	Q3_K_M	Q2_K
7B	short, PL (9)	0.78	0.89	0.89	1.00	1.00	0.11
	short, EN (9)	0.33	0.33	0.33	0.56	0.44	0.11
	long, PL (13)	0.08	0.08	0.31	0.15	0.15	0.00
	long, EN (9)	0.56	0.56	0.78	0.33	0.22	0.00
11B	short, PL (9)	1.00	1.00	1.00	0.89	0.78	0.00
	short, EN (9)	0.78	0.89	0.67	0.78	0.89	0.00

Table 8: Pass rate by chain length and interface language (

n=40

chains), repair = off . For the 11B, Polish matches or beats English in the long-chain bucket at every healthy precision except the 4-bit dip (

0.31

against

0.44

); on short chains the single inversion is at Q3_K_M (

0.78

against

0.89

). For the 7B, the long PL_EN cell is a floor at every precision.

precision	seed 1	seed 2	seed 3
Q8_0	8	49	48
Q3_K_M	13	43	51

Table 9: 11B main suite at

T=0.7

, repair = off , passes of 67. The greedy (

T=0

) reference is 54 (Q8_0) and 48 (Q3_K_M).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

		7B				11B
task	probe	Q8	Q4	Q3	Q2	Q8	Q6	Q5	Q4	Q3	Q2
adv_001	language leakage	P	P	P	F	P	P	P	P	P	F
adv_002	language leakage	P	P	P	P	P	P	P	P	P	F
adv_003	inflection mismatch	P	P	P	P	P	P	P	P	P	F
adv_004a	diacritic corruption	P	P	P	P	P	P	P	P	P	F
adv_004b	final-answer shape	F	P	P	F	P	P	F	P	F	P

Appendix

Table 10: Easy tier, repair = off . P denotes pass, F fail. PLLuM easy-tier totals appear in Appendix B .

precision	repair	easy /15	hard /52	all /67	easy rate	hard rate	all rate
Q8_0	off	14	16	30	0.933	0.308	0.448
Q8_0	on	14	24	38	0.933	0.462	0.567
Q6_K	off	14	17	31	0.933	0.327	0.463
Q6_K	on	14	24	38	0.933	0.462	0.567
Q5_K_M	off	13	22	35	0.867	0.423	0.522
Q5_K_M	on	13	27	40	0.867	0.519	0.597

Appendix

Table 11: 7B tier-resolved pass counts. Repair acts on the hard tier from 8-bit down to 4-bit (

+8

+7

+5

+6

tasks), does nothing on either tier at 3-bit, and at 2-bit acts on the easy tier only (

+2

	Q8_0	Q6_K	Q5_K_M	Q4_K_M	Q3_K_M	Q2_K
11B easy /15	14 / 14	15 / 15	13 / 13	13 / 13	12 / 12	2 / 2
11B hard /52	40 / 40	41 / 42	37 / 38	23 / 27	36 / 36	1 / 3
11B all /67	54 / 54	56 / 57	50 / 51	36 / 40	48 / 48	3 / 5
PLLuM easy /15	6 / 7	5 / 7	4 / 6	8 / 10	9 / 11	1 / 1
PLLuM hard /52	7 / 8	7 / 7	3 / 3	7 / 7	6 / 7	0 / 0
PLLuM all /67	13 / 15	12 / 14	7 / 9	15 / 17	15 / 18	1 / 1

Appendix

Table 12: 11B and PLLuM tier-resolved pass counts. Each cell reads repair = off / on , the same convention as Table 2 , whose second number each all /67 row reproduces. Every task repair adds to the 11B lands on the hard tier, at most 4 of them (at 4-bit) and none at all at 8-bit or 3-bit; for PLLuM the opposite holds, 9 of its 11 added tasks being easy-tier.

model	precision	failures	arith+protocol	arith+fixation	protocol+fixation	single category
7B	Q8_0	21	3	3	18	1
	Q6_K	21	7	5	15	0
	Q5_K_M	21	3	4	14	4
	Q4_K_M	21	9	6	13	5
	Q3_K_M	20	0	2	5	13
	Q2_K	25	8	10	12	7

Appendix

Table 13: Tag co-occurrence among failures on the 25 arithmetic-graded tasks, Bielik pair, repair = off . Every co-occurring case is resolved by rule order rather than by evidence. The three pair columns count inclusively : a failure carrying tags from all three classes is counted in all three columns, so the columns overlap and need not sum to the failure total. The same inclusive rule produces the PLLuM counts quoted in Section 3.7 , Section 6 , and the paragraph below; counting exclusively, meaning a class set of exactly two, would lower PLLuM’s arithmetic-plus-fixation figure from 90 to 76.

pair	tier	$n_{11}$	$n_{10}$	$n_{01}$	$n_{00}$	$p$
Q8 vs Q4	all	26	4	8	29	0.388
Q8 vs Q3	all	23	7	8	29	1.000
Q8 vs Q6	all	30	0	1	36	1.000
Q5 vs Q4	all	28	7	6	26	1.000
Q4 vs Q3	all	27	7	4	29	0.549
Q8 vs Q2	hard	1	15	1	35	0.0005

Appendix

Table 14: 7B, repair = off . The plateau Q8_0 through Q3_K_M shows no significant contrast; every contrast against Q2_K is significant.

pair	tier	$n_{11}$	$n_{10}$	$n_{01}$	$n_{00}$	$p$
Q8 vs Q6	all	52	2	4	9	0.688
Q8 vs Q5	all	49	5	1	12	0.219
Q8 vs Q4	all	35	19	1	12	$<$ 0.0001
Q8 vs Q3	all	42	12	6	7	0.238
Q8 vs Q2	all	3	51	0	13	$<$ 0.0001
Q6 vs Q5	all	49	7	1	10	0.070

Appendix

Table 15: 11B, repair = off . Q4_K_M is nominally significant against both neighbours (uncorrected, twins counted); every contrast against Q2_K is significant. Fourteen of the fifteen pairwise contrasts form the analysis family; the omitted one, Q6 vs Q3, is

44/12/4/7

p=0.077

pair	$n_{11}$	$n_{10}$	$n_{01}$	$n_{00}$	$p$
Q3 vs Q2	1	14	0	52	0.00012

Appendix

Table 16: PLLuM, decisive contrast, all tasks, repair = off .

model	precision	passes /67	rate	95% interval
7B	Q8_0	30	0.448	$[0.328,\ 0.567]$
	Q6_K	31	0.463	$[0.343,\ 0.582]$
	Q5_K_M	35	0.522	$[0.403,\ 0.642]$
	Q4_K_M	34	0.507	$[0.388,\ 0.627]$
	Q3_K_M	31	0.463	$[0.343,\ 0.582]$
	Q2_K	10	0.149	$[0.075,\ 0.239]$

Appendix

Table 17: All-task point estimates with 95% intervals, repair = off . Every interval here and in Table 18 is reported at a fixed resampling seed of 42. Recomputing under a second seed (20260823) leaves 16 of the 18 cells here identical to three decimals and moves a single endpoint by about

0.015

, one task in 67, in the other two: PLLuM Q8_0 upper

0.299\to 0.284

and PLLuM Q5_K_M lower

0.045\to 0.030

. One endpoint moves the same way in Table 18 , the 7B Q3-versus-Q2 upper bound

+0.433\to+0.418

. No interval changes sign or crosses zero.

model	contrast	difference	95% interval	excludes zero
7B	Q4 vs Q3	$+0.045$	$[-0.045,\ +0.149]$	no
7B	Q3 vs Q2	$+0.313$	$[+0.209,\ +0.433]$	yes
11B	Q4 vs Q3	$-0.179$	$[-0.328,\ -0.030]$	yes
11B	Q3 vs Q2	$+0.672$	$[+0.552,\ +0.791]$	yes
PLLuM	Q3 vs Q2	$+0.209$	$[+0.119,\ +0.313]$	yes

Appendix

Table 18: Selected paired differences on all 67 tasks, repair = off .

model	precision	subset	$n$	mean tok.	median tok.	p90 tok.	mean steps	median steps
7B	Q8_0	pass	30	4540	3504	8645	3.30	3.0
	Q8_0	fail	37	12346	12298	17219	7.27	8.0
	Q6_K	pass	31	4280	3515	8589	3.29	3.0
	Q6_K	fail	36	11141	12384	14820	7.03	8.0
	Q5_K_M	pass	35	6674	3577	13752	3.74	3.0
	Q5_K_M	fail	32	15840	14065.5	29083	6.47	8.0

Appendix

Table 19: Token and step distributions by outcome, repair = off . Step budgets are per task, 4 to 10, and are 8 on 29 of the 67 tasks. Token counts, defined as in Table 4 , are comparable only within the Bielik pair (shared tokenizer).

tools	language	$n$	Q8_0	Q4_K_M	Q3_K_M	Q2_K
2	PL_EN	9	7/9	9/9	9/9	1/9
2	EN_EN	3	3/3	3/3	3/3	1/3
3	EN_EN	6	0/6	2/6	1/6	0/6
4	PL_EN	11	0/11	1/11	2/11	0/11
4	EN_EN	1	0/1	1/1	0/1	0/1
5	PL_EN	2	1/2	1/2	0/2	0/2

Appendix

Table 20: 7B hard-tier chains by exact chain length and interface language, repair = off . The 4-tool PL_EN family, the order-trap chains, is the model’s persistent failure independent of precision.

model	precision	contrast	rate	$n_{01}$	$p$
11B	Q8_0	L0 vs L3T ( $n=6$ )	strict	6	0.031
11B	Q8_0	L0 vs L3N ( $n=10$ )	strict	2	0.500
11B	Q8_0	L0 vs L3N ( $n=10$ )	corrected	9	0.004
11B	Q8_0	L0 vs L2 ( $n=10$ )	strict	4	0.125
11B	Q4_K_M	L0 vs L3N ( $n=10$ )	corrected	4	0.125
11B	Q3_K_M	L0 vs L2 ( $n=10$ )	strict	10	0.002

Appendix

Table 21: Paired McNemar on the ladder with the original L0 as the baseline, as run in 2026-07, selected contrasts, instance as pairing unit, repair = off .

n_{01}

counts instances failed at L0 and passed at the scaffolded rung;

n_{10}=0

in every row. Holm over these eleven rows leaves 11B Q8_0 corrected (

0.039

) and 11B Q3_K_M L0 vs L2 (

0.021

) below

0.05

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 289 papers

Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs
Oct 6, 2026Yuhe HuTool-Augmented Language Model Agents LLM Agent Evaluation
Duke University
Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
Jul 29, 2026Jiwon Jang, Kisu Yang, Heuiseok Lim +1Post-Training Quantization AI Agent Reliability
VAIV Company · Seoul National University · Korea University
Which Decisions Low-Bit Quantization Breaks, and How to Predict Them
Aug 6, 2026Zekun Wu, Swati Dhiman, Adriano KoshiyamaPost-Training Quantization Low-Bit Quantization
Holistic AI · UCL Centre for Artificial Intelligence, University College London

Oct 6, 2026J/K move · Enter open · S save

Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs

Yuhe Hu

Duke University

Tool-Augmented Language Model Agents LLM Agent Evaluation Post-Training Quantization LLM Agent Reliability

Open paper AskPDF arXiv

Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on how recovery is evaluated. We compare 8-bit and 4-bit variants of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic tool-use tasks and five prompts. The 8-bit-4-bit recovery comparison changes direction across prompts and evaluation targets. On tasks that both variants complete without faults under the same prompt, the difference ranges from 0 to +20.2 percentage points for Llama and from -50.0 to +35.0 points for Qwen. Full-pipeline point estimates favor 8-bit Llama under all five prompts, whereas the Qwen comparison changes direction across prompts. The evaluation target can also reverse the result. For Llama under one prompt, scoring each variant only on its own clean-passing tasks favors 4-bit by 17.5 points; scoring the same tasks for both variants gives no difference, while scoring the full pipeline favors 8-bit by 28.3 points. Executor leniency is a third such choice. Rescoring the same logs with strict output parsing, which 8-bit Llama violates far more often than 4-bit Llama under that prompt, turns that +28.3 into -15.0 while leaving Qwen essentially unchanged. These findings show that one prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness. Evaluations should compare variants on matched tasks, report full-pipeline success for deployment decisions, state the scoring policy, and quantify uncertainty across tasks rather than injected fault sites.

Abstract

LLM Tool Use

Explore similar work

Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Which Decisions Low-Bit Quantization Breaks, and How to Predict Them