Beyond Accuracy: How Quantization Affects Coding Agents

Areej Mehboob; Muhammad Ali Shafique; Muhammad Usman Qadeer; Hamza Farooq

cs.CLNov 2, 2026

Beyond Accuracy: How Quantization Affects Coding Agents

Authors: Areej Mehboob, Muhammad Ali Shafique, Muhammad Usman Qadeer, Hamza Farooq

Organizations: Traversaal AI · Kansas State University

Abstract

Agents are increasingly used for coding tasks, but the inference cost of large language models (LLMs) limits where they can run. Quantization reduces model size, but its effects on coding agents that execute and revise programs remain underexplored. We compare BF16 with 4-bit AWQ and GPTQ for Qwen2.5-Coder-Instruct (7B, 14B, and 32B) and Devstral-Small-2507 (24B) on OJBench coding problems across three seeds. We detect no statistically significant accuracy loss for any of the coding agents. However, final accuracy can conceal substantial behavioral changes: AWQ reduces Qwen-7B one-shot solves from 17 to 0 while increasing loop-based solves from 10 to 25. All Qwen-32B successes involve tool use, whereas Devstral rarely makes a valid executor call. Failure profiles also differ: Qwen-32B produces far more compilation errors than the smaller Qwen models, rising slightly under quantization, while Devstral's failures shift significantly from runtime errors toward wrong answers and timeouts. These findings indicate that quantization robustness in coding agents should be assessed through solution trajectories, tool use, and failure modes, in addition to final accuracy.

Figures & tables

Model	Precision	Easy (%)	Medium (%)	Overall (%)
Qwen-2.5-Coder-7B	BF16	$21.29_{\pm 1.6}$	$1.69_{\pm 0.7}$	$7.83_{\pm 0.9}$
	AWQ	$20.37_{\pm 1.6}$	$1.27_{\pm 0.0}$	$7.25_{\pm 0.5}$
	GPTQ	$18.52_{\pm 3.2}$	$1.27_{\pm 0.0}$	$6.67_{\pm 1.0}$
Qwen-2.5-Coder-14B	BF16	$30.56_{\pm 2.8}$	$2.53_{\pm 1.3}$	$11.30_{\pm 0.0}$
	AWQ	$26.85_{\pm 1.6}$	$1.27_{\pm 0.0}$	$9.28_{\pm 0.5}$
	GPTQ	$26.85_{\pm 8.9}$	$2.11_{\pm 0.7}$	$9.86_{\pm 2.7}$

Table 1: Performance of Qwen-2.5-Coder and Devstral-Small-2507 agents (vLLM, OpenAI-compatible tool schema) on OJBench, reported as mean ±std over 3 seeds. Darker shading marks the best overall accuracy per model family, lighter the second best. Overall accuracy is over the Easy and Medium splits (115 problems); the Hard split is excluded because all models scored 0%.

Panel of Figure 1 — Figure 1: Distribution of failure categories across BF16, AWQ, and GPTQ for each agent. Categories: IR (Invalid Return), WA (Wrong Answer), TLE (Time Limit Exceeded), CE (Compilation Error), MLE (Memory Limit Exceeded). Counts are per-seed averages over the Easy+Medium splits (115 problems, 3 seeds).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Precision	IR	WA	TLE	CE	MLE	$p$	$p_{\mathrm{Holm}}$
Qwen-2.5-Coder-7B	BF16	21.7	65.0	17.0	1.3	0.7	–	–
	AWQ	29.0	59.7	15.0	2.3	0.7	0.17	0.84
	GPTQ	23.3	66.0	15.0	2.0	1.0	0.81	1.00
Qwen-2.5-Coder-14B	BF16	15.3	66.3	18.0	2.0	0.3	–	–
	AWQ	13.0	69.0	17.0	4.3	1.0	0.34	1.00
	GPTQ	21.0	61.3	18.7	2.0	0.7	0.35	1.00

Appendix

Table 2: Distribution of Failure Types in Agent Submissions. IR: Invalid Return, WA: Wrong Answer, TLE: Time Limit Exceeded, CE: Compilation Error, MLE: Memory Limit Exceeded. Counts are per-seed averages over the Easy+Medium splits (115 problems, 3 seeds).

p

is a chi-square test of whether the failure distribution differs from BF16, computed on totals over the three seeds; MLE is excluded from the test because its counts are very small.

p_{\mathrm{Holm}}

applies a Holm correction across the eight comparisons.

Model	Scheme	$b$	$c$	$p$	$p_{\mathrm{Holm}}$	$\Delta$	95% CI	$p_{\mathrm{prob}}$
		Paired (345 pairs)				Problem level (115 problems)
Qwen-2.5-Coder-7B	AWQ	8	6	0.79	1.00	$0.58$	$[-1.74,3.19]$	0.82
Qwen-2.5-Coder-7B	GPTQ	8	4	0.39	1.00	$1.16$	$[-0.87,3.19]$	0.42
Qwen-2.5-Coder-14B	AWQ	9	2	0.065	0.46	$2.03$	$[0.00,4.06]$	0.094
Qwen-2.5-Coder-14B	GPTQ	9	4	0.27	1.00	$1.45$	$[-0.58,3.48]$	0.27
Qwen-2.5-Coder-32B	AWQ	16	5	0.027	0.21	$3.19$	$[0.58,6.09]$	0.051

Appendix

Table 3: Exact two-sided McNemar tests, BF16 versus quantized, over paired per-problem outcomes (same problem, same seed; 345 pairs per comparison).

b

= problems solved only by BF16;

c

= problems solved only by the quantized model.

p_{\mathrm{Holm}}

applies a Holm correction across the eight comparisons. The 32B/AWQ comparison is significant before correction (

p=0.027

, bold) but not after it; no comparison is significant after correction. The last three columns treat each problem as one unit:

\Delta

is BF16 minus quantized accuracy (points), the 95% CI comes from a cluster bootstrap over the 115 problems, and

p_{\mathrm{prob}}

from a within-problem permutation test. The problem-level analysis gives the same conclusion: no comparison is significant at

\alpha=0.05

, although the 32B/AWQ result is borderline (its CI just excludes zero,

p=0.051

Model	Precision	tool (%)	tool (%)	(%)	calls	solves	solves
		Used	Tried	Malformed	Mean	One-shot	Loop
Qwen-2.5-Coder-7B	BF16	66.7	70.1	3.8	2.26	17	10
	AWQ	73.9	82.9 †	5.9	2.29	0	25
	GPTQ	69.0	71.9	3.4	1.90	3	20
Qwen-2.5-Coder-14B	BF16	31.6	31.6	0.0	1.21	21	18
	AWQ	26.7	26.7	0.0	1.01	17	15

Appendix

Table 4: Tool use and solve path per agent, totals over 3 seeds (345 attempts). “Used tool” is the share of problems with at least one valid executor call. “Tried tool” also counts attempts in which the model issued a tool call that did not follow the provided tool schema (e.g., broken JSON or the tool written as a Python function inside its code). “Malformed” is the share of all tool calls that did not follow the schema (mean over seeds).

\dagger

marks tried-tool rates that differ significantly from BF16 after Holm correction. “One-shot” solves used zero tool calls; “loop” solves followed at least one execute-and-check round.

		Solve path		Tool use
Model	Scheme	$p$	$p_{\mathrm{Holm}}$	$p$	$p_{\mathrm{Holm}}$
Qwen-2.5-Coder-7B	AWQ	$<$ 0.001	$<$ 0.001	0.045	0.29
Qwen-2.5-Coder-7B	GPTQ	$<$ 0.001	0.003	0.57	1.00
Qwen-2.5-Coder-14B	AWQ	1.00	1.00	0.18	0.90
Qwen-2.5-Coder-14B	GPTQ	0.82	1.00	0.42	1.00
Qwen-2.5-Coder-32B	AWQ	1.00	1.00	0.042	0.29

Appendix

Table 5: Fisher’s exact tests on the behavioral measures in Table 4 , BF16 versus quantized. Solve path tests whether the one-shot/loop split among solved problems differs from BF16; tool use tests whether the share of problems with at least one executor call differs from BF16.

p_{\mathrm{Holm}}

applies a Holm correction across the eight comparisons within each measure. Significant results after correction are in bold. The tests treat runs as independent, so

p

-values are approximate. At the problem level, the Qwen-7B solve-path shift is significant under AWQ (

p=0.005

) and GPTQ (

p=0.022

). On problems solved by both precisions, every BF16 one-shot solve becomes a loop solve under AWQ (8 of 8 problems,

p=0.008

), and 7 of 8 under GPTQ (

p=0.070

), which does not reach significance on its own.

Panel of Figure 2 — Figure 2: Overall agent accuracy on OJBench (Easy+Medium, 115 problems), mean over 3 seeds with one standard deviation. The 32B agent under AWQ is the only comparison that differs from BF16 before correction (paired McNemar, $\ast$ ); after Holm correction across the eight comparisons, no gap is statistically significant.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 337 papers

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
Jul 29, 2026Jiwon Jang, Kisu Yang, Heuiseok Lim +1Post-Training Quantization AI Agent Reliability
VAIV Company · Seoul National University · Korea University
Quantize with Confidence? An Empirical Study of Quantization for Code Generation
Jul 15, 2026Saima Afrin, Md. Zahidul Haque, Antonio MastropaoloLLM Quantization Code Generation Evaluation
AURA Lab · Department of Computer Science · William & Mary +1
Qwen3-Coder-Next Technical Report
Feb 28, 2026Ruisheng Cao, Mouxiang Chen, Jiawei Chen +17Code Language Models AI Coding Agents

Jul 29, 2026J/K move · Enter open · S save

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

VAIV Company · Seoul National University · Korea University

Post-Training Quantization AI Agent Reliability Benchmark Design 4-Bit Quantization LLM Agent Reliability AI Agent Benchmarks+3

Open paper AskPDF arXiv

Post-training quantization to 4-bit weights is widely reported to be nearly lossless. We test this claim for multi-turn, tool-calling agents, where it now matters most. On

τ^2

-bench, across two open-weight model families in dense and MoE variants and two domains (eight cells, 456 episodes each, at 16-, 8-, and 4-bit weights), quantization indeed looks free on the standard metric. No cell shows a score change that survives multiple-comparison correction, and in the cell that carries the largest process damage, equivalence testing bounds the change within

\pm

7.5 points. The process tells a different story. Quantization amplifies the failure the model already exhibits at full precision (tool-name hallucination in telecom, with the same directional trend in retail entity errors) by up to 2.5

\times

in volume (+17.6 points per task), while creating essentially no new failures. The failure set is the same at every precision (rank correlation

\geq

0.94, 0.18% novel events). The score stays flat because the benchmark's ten-error budget absorbs the extra failures. Shrinking the budget to two errors re-exposes a score gap of 17 points, and it does so only in the one cell where quantization added error volume, exactly as the masking account predicts. A targeted error-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives. Both diagnostics, the per-channel error rate and success under a shrinking budget, come from logs benchmarks already collect; we suggest reporting them alongside task reward.

Abstract

AI Coding Agents

Explore similar work

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Quantize with Confidence? An Empirical Study of Quantization for Code Generation

Qwen3-Coder-Next Technical Report