cs.LGOct 17, 2026

Measuring the Task Value of Precision Upgrades in Closed-Loop Agents

Authors: Sicheng Lyu, Xinyu Wang, Yu Gu, Lei Ding, Ziyang Song, Xiao-Wen Chang

Organizations: McGill University · University of Manitoba · Ohio University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A low-precision agent can call its full-precision (BF16) model at selected decisions as a fallback. A learned precision router chooses where to call it from the disagreement between the two models and is judged by end-to-end success. An inserted BF16 action, however, can rescue a failing task or damage a succeeding one. We argue that a fallback's value is the number of tasks it saves minus the number it spoils, measured for the intervention that will actually run. From restored states, we run paired branches that cross which model supplies the current action with which model continues, separating replacement of one action, takeover of later execution, and both. On Qwen3 agents with simulated 4-bit weights in tau2-bench, replacing the first action where the two models differed in tool use changed many task outcomes, but rescues and damages nearly cancelled, and on held-out telecom tasks the action-only estimate was negative. Replacing both the action and the continuation had a positive net gain in two of five exploratory lower-bit stress settings. Two references changed outcomes at fixed bit width: replacing an action with that of a second 4-bit checkpoint changed about as many outcomes as replacing it with the BF16 action, and reordering tool-argument keys changed outcomes while the executed tool call stayed identical. At common states, the additional value of the BF16 action over that checkpoint remained unresolved. A frozen uncertainty trigger found more outcome changes than random selection, while its net benefit stayed within the range of random selection. We recommend evaluating each fallback by its signed task benefit under the intended continuation, against same-precision references, before attributing gains to more bits.

Figures & tables

Figure 1: Fallback interventions and state sources. Q is the quantized model and B its BF16 version. Branch XY lets X supply the current action and Y continue, and YXYY_{\mathrm{XY}} is its task success. (a) Four branches from one restored airline state (Qwen3-4B with 4-bit weights quantized by AWQ-style scaling, task 31, decision 1 of 12), crossing the action source (rows) with the continuing model (columns). The box below defines the upgrades GG , HH and FF against QQ and gives their values here. (b) Schematic of the three state sources defined in Section 3.2 .
Figure 2: Rescues, damages and net gain of action replacement. (a,b) Each row is one configuration or pool, in percentage of task–block units. Blue bars to the right show rescues, hatched orange bars to the left damages, and diamonds the full-pool net gain GallG_{\mathrm{all}} with its interval. The right column gives turnover, rescues plus damages. (a) Development panel, new-task 95% intervals. (b) Pre-registered telecom holdout, crossed 90% intervals, with grey shading marking the registered ±\pm 3 pp small-effect region. Bold rows are pools or the registered estimate. (c) Net gain on divergent states for action replacement ( GG ), takeover ( HH ) and joint replacement ( FF ), with new-task 95% intervals and the point estimate printed at the right. AWQ denotes AWQ-style scaling. Further intervals: Appendix B .
Stress settingGGHHFF [95% CI]
4B GPTQ-W3−1.3-1.3+4.3+4.3+1.3+1.3 [ −6.8-6.8 , +9.4+9.4 ]
4B KV4-rot+5.2+5.2+9.4+9.4+12.4+12.4 [ +3.0+3.0 , +21.6+21.6 ]
8B GPTQ-W3+2.1+2.1+0.4+0.4+2.1+2.1 [ −6.6-6.6 , +11.8+11.8 ]
8B KV4-rot+3.3+3.3−0.8-0.8+4.6+4.6 [ −6.6-6.6 , +15.9+15.9 ]
8B RTN-W3−0.4-0.4+15.3+15.3+18.4+18.4 [ +9.4+9.4 , +28.2+28.2 ]
Table 1: Net gain (pp) on divergent states in the five exploratory stress settings, Qwen3-4B (4B) and Qwen3-8B (8B) on retail and airline, three blocks each: action replacement GG , takeover HH and joint replacement FF . Only FF is shown with its unadjusted new-task 95% confidence interval (CI), and bold marks an interval above zero. Intervals for GG and HH : Table 6 .
Figure 3: Same-precision references. Rows give rescues, damages and turnover in percentage of units or states, with point estimates and 95% intervals. Circles mark the BF16 action, squares the second W4 checkpoint (Q ′ ) and triangles key-order reversal. Below the dashed line, Δ\Delta turnover is BF16 minus the reference in percentage points. (a) BF16 and the second checkpoint, each substituted at its own first divergence on 1020 task–block units, with new-task intervals. (b) BF16 action replacement and key-order reversal of the quantized tool call on the same 300 eligible states, with the wider of block and new-task intervals. The common-state comparison and all other intervals appear in Appendix E.2 and Table 14 .
Figure 4: Additional value of the BF16 action at 510 common Qwen3-8B states: success with the BF16 action minus success with the action of the second checkpoint (Q ′ ), for each continuing model. Thick lines are 97% tt intervals over block means and thin lines 97% bootstrap intervals, both adjusted for the two looks described in Appendix E.2 , and the grey band is the ±\pm 3 pp equivalence margin.
Figure 5: Frozen margin trigger on 1020 validation states from quantized histories, by nominal rate with actual selected counts. (a) Rescues (up) and damages (down) for the trigger (solid) and expected under matched-count random selection (hatched). (b) Rescues minus damages for the trigger (circles) and random selection (squares). The grey strip gives the matched-count hindsight oracle, which selects rescues by construction. Unadjusted exact one-sided pp -values compare each panel with random selection. Full values: Table 16 .
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

ConfigurationQ / BF16 (%)Q −- BF16 [95% CI]NI 3 ppTOST ±\pm 3 pp
4B GPTQ-W440.0 / 35.8+4.2+4.2 [ −2.4-2.4 , +10.5+10.5 ]passfail
4B AWQ-style W436.8 / 35.8+0.9+0.9 [ −5.8-5.8 , +7.5+7.5 ]failfail
8B GPTQ-W430.1 / 30.4−0.2-0.2 [ −7.7-7.7 , +6.8+6.8 ]failfail
8B AWQ-style W431.5 / 30.4+1.2+1.2 [ −5.3-5.3 , +7.6+7.6 ]failfail
pool, all four +1.5+1.5 [ −2.7-2.7 , +5.6+5.6 ]passfail
Appendix
Table 2: End-to-end closed-loop success of Qwen3-4B (4B) and Qwen3-8B (8B) configurations. Q / BF16: success rates of the quantized and BF16 models. Q −- BF16: difference in pp with a crossed task ×\times block bootstrap 95% confidence interval (CI), units paired on prompt block and the bootstrap drawn jointly across configurations. NI: non-inferiority of the quantized model at a 3 pp margin. TOST: equivalence by two one-sided tests at ±\pm 3 pp.
Configurationτ2\tau^{2} -bench retail (first div.)τ2\tau^{2} -bench airline (first div.)telecom dev (first div.)Ordinary (retail / airline)BFCL / ALFWorld
4B GPTQ-W458 t / 727 st 20/20 blocks27 t / 375 st 20/20 blocksholdout only3 blocks 393 / 177 rows788 items ⋅\cdot 573 st / –
4B AWQ-style W458 t / 986 st 20/20 blocks27 t / 453 st 20/20 blocks–3 blocks 393 / 177 rows788 items ⋅\cdot 642 st / –
4B KV858 t / 412 st 22/22 blocks27 t / 181 st 22/22 blocks–3 blocks 393 / 177 rows– / –
8B GPTQ-W458 t / 1017 st 20/20 blocks27 t / 456 st 20/20 blocks80 t / 477 st 6/6 blocks ; holdout3 blocks 393 / 177 rows788 ⋅\cdot 533 st / 134 ⋅\cdot 115 st
8B AWQ-style W458 t / 1087 st 20/20 blocks27 t / 475 st 20/20 blocks80 t / 480 st 6/6 blocks ; holdout3 blocks 393 / 177 rows788 ⋅\cdot 548 st / 134 ⋅\cdot 119 st
8B KV858 t / 585 st 22/22 blocks27 t / 270 st 22/22 blocks–3 blocks 393 / 177 rows– / –
Appendix
Table 3: Completed coverage of the crossed interventions for Qwen3-4B (4B), Qwen3-8B (8B) and Qwen3-14B (14B), counted after the strict admission rule. First div.: first divergent states. t = tasks, st = states, blocks = complete/found blocks. Ordinary: blocks completed and state rows per domain, with the first three blocks analysed. BFCL and ALFWorld: items ⋅\cdot states, one run each. “holdout”: the configuration is in the telecom holdout (Appendix C ), and the telecom development results (Appendix C.1 ) use the six blocks listed. Settings below the first rule are stress settings. † The QQ no-op rule (Appendix A ) dropped the BFCL base category for 4B GPTQ-W3.
Config. (models)Method, weightsGroup, gridAct-orderCalibrationKV cache / activations / rotation
GPTQ-W4 Q (4B, 8B)GPTQ, W4; damp 0.01, block 128, true-sequential128, asym.yesdraw c1: 57 BF16 agent-trajectory states (39 retail, 18 airline calibration tasks, disjoint from the 85 dev tasks), one state per task, full context (median 6041 / 6310 tok.); 64,000 activation rows (1122–1123 per state); draw seed 20260729KV BF16, act. BF16, no rotation
GPTQ-W4 Q ′ (4B, 8B)as Q128, asym.yesdraw c2, seed 20260730: same 57 tasks and trajectories, re-drawn decision state and rows (5/57 states shared with c1); not disjointas Q
AWQ-style W4 (4B, 8B)AWQ-style: per-linear scale ∣x∣ˉα\bar{|x|}^{\alpha} , 20-point α\alpha grid, scale folded into weight, then RTN; no clip search128, asym.n/a (GPTQ only)draw c1 as GPTQ-W4 Q; α\alpha fit on a 4096-row sample (code default; per-run value not recorded)as Q
GPTQ-W3 s Q / Q ′ (4B, 8B)GPTQ, W3128, asym.yesc1 / c2 as W4as Q
RTN-W3 s (4B, 8B)RTN, W3128, asym.n/anoneas Q
KV8 (4B, 8B)BF16 weights––noneINT8 K and V, asym. min–max per (token, KV head) over head_dim, no clip, no rotation; fake quant in fp32
Appendix
Table 4: Quantization configuration of every checkpoint, from the checkpoint metadata. 4B and 8B denote Qwen3-4B and Qwen3-8B. W nn : nn -bit weights, KV nn : nn -bit KV cache, INT8/INT4: 8-/4-bit integers, RTN: round-to-nearest. AWQ-style W4 folds the scale into the weight, then applies RTN without clip search. Q: the quantized model. Q ′ : the second checkpoint, which re-draws the decision state and activation rows from the same 57 calibration trajectories. One Qwen3-8B calibration context in draw c1 is marked truncated. All checkpoints are fake-quantized and stored in BF16, with embeddings, lm_head and norms kept in BF16. s Stress configuration.
Configurationunitsdiv.rrddMMGallG_{\mathrm{all}} [fixed pool][new tasks]
4B GPTQ-W485 ×\times 2011026.88.715.5−1.9-1.9 [ −3.7-3.7 , −0.1-0.1 ][ −6.8-6.8 , +2.7+2.7 ]
4B AWQ-style W485 ×\times 2014399.810.119.9−0.3-0.3 [ −2.2-2.2 , +1.6+1.6 ][ −5.4-5.4 , +5.2+5.2 ]
8B GPTQ-W485 ×\times 20147310.47.517.9+2.9+2.9 [ +1.4+1.4 , +4.4+4.4 ][ −2.9-2.9 , +9.2+9.2 ]
8B AWQ-style W485 ×\times 20156211.49.520.9+1.9+1.9 [ −0.6-0.6 , +4.3+4.3 ][ −3.9-3.9 , +8.1+8.1 ]
W4 pool85 ×\times 20 9.69.018.6+0.7+0.7 [ −0.3-0.3 , +1.6+1.6 ][ −2.1-2.1 , +3.4+3.4 ]
4B KV885 ×\times 225933.13.46.5−0.3-0.3 [ −1.7-1.7 , +1.0+1.0 ][ −3.0-3.0 , +2.3+2.3 ]
Appendix
Table 5: Action-only replacement at the quantized model’s first divergent state with the quantized model continuing, τ2\tau^{2} -bench retail and airline development panel (pp of units). Units: tasks ×\times complete blocks. Div.: number of first divergent states. rr , dd , MM : rescue, damage and turnover rates over all units, with units lacking a divergent state counted as zero. Gall=r−dG_{\mathrm{all}}=r-d . Fixed-pool 95% intervals use a block bootstrap ( ≥\geq 10 blocks) or a tt interval over block means. New-task 95% intervals use a crossed task ×\times block bootstrap. W3: 3-bit weights. KV8: INT8 KV cache. KV4-rot: rotated 4-bit KV cache. RTN: round-to-nearest. s Stress configuration, three blocks.
ConfigurationstatesGG (action)HH (continuation)FF (joint)G−HG-H
4B GPTQ-W41102−2.9-2.9 [ −5.7-5.7 , −0.2-0.2 ] [ −10.3-10.3 , +4.2+4.2 ]+0.7+0.7 [ −1.6-1.6 , +3.0+3.0 ] [ −4.0-4.0 , +6.0+6.0 ]−2.5-2.5 [ −5.3-5.3 , +0.2+0.2 ] [ −9.9-9.9 , +4.7+4.7 ]−3.6-3.6 [ −6.6-6.6 , −0.6-0.6 ] [ −11.2-11.2 , +3.6+3.6 ]
4B AWQ-style W41439−0.3-0.3 [ −2.6-2.6 , +1.9+1.9 ] [ −6.5-6.5 , +6.1+6.1 ]+0.9+0.9 [ −1.1-1.1 , +2.9+2.9 ] [ −3.7-3.7 , +5.5+5.5 ]−0.8-0.8 [ −3.3-3.3 , +1.6+1.6 ] [ −7.4-7.4 , +5.8+5.8 ]−1.3-1.3 [ −3.0-3.0 , +0.6+0.6 ] [ −7.7-7.7 , +5.1+5.1 ]
8B GPTQ-W41473+3.3+3.3 [ +1.6+1.6 , +5.0+5.0 ] [ −3.3-3.3 , +10.5+10.5 ]+0.4+0.4 [ −1.2-1.2 , +1.9+1.9 ] [ −3.8-3.8 , +4.9+4.9 ]+5.4+5.4 [ +3.5+3.5 , +7.2+7.2 ] [ −2.0-2.0 , +13.0+13.0 ]+2.9+2.9 [ +1.3+1.3 , +4.7+4.7 ] [ −3.2-3.2 , +9.3+9.3 ]
8B AWQ-style W41562+2.1+2.1 [ −0.6-0.6 , +4.7+4.7 ] [ −4.1-4.1 , +8.7+8.7 ]−1.3-1.3 [ −3.5-3.5 , +0.9+0.9 ] [ −7.1-7.1 , +4.6+4.6 ]+0.6+0.6 [ −1.5-1.5 , +2.7+2.7 ] [ −6.2-6.2 , +7.3+7.3 ]+3.4+3.4 [ +1.5+1.5 , +5.2+5.2 ] [ −1.6-1.6 , +8.6+8.6 ]
W4 pool5576+0.5+0.5 [ −0.7-0.7 , +1.7+1.7 ] [ −2.9-2.9 , +3.8+3.8 ]+0.2+0.2 [ −1.3-1.3 , +1.6+1.6 ] [ −2.7-2.7 , +2.8+2.8 ]+0.7+0.7 [ −0.6-0.6 , +2.1+2.1 ] [ −3.1-3.1 , +4.4+4.4 ]+0.4+0.4 [ −0.8-0.8 , +1.7+1.7 ] [ −3.1-3.1 , +3.7+3.7 ]
4B KV8593−1.0-1.0 [ −5.3-5.3 , +3.2+3.2 ] [ −9.4-9.4 , +7.3+7.3 ]−2.5-2.5 [ −4.5-4.5 , −0.5-0.5 ] [ −6.4-6.4 , +1.1+1.1 ]−1.2-1.2 [ −4.7-4.7 , +2.3+2.3 ] [ −9.3-9.3 , +7.7+7.7 ]+1.5+1.5 [ −3.1-3.1 , +6.0+6.0 ] [ −7.1-7.1 , +9.7+9.7 ]
Appendix
Table 6: Net gain of the three upgrade modes on first divergent states (pp). GG , HH and FF compare action-only (BQ), continuation-only (QB) and joint (BB) replacement with QQ, and G−HG-H compares the first two. In each two-line cell, the first line gives the estimate and its fixed-pool 95% interval and the second line the new-task 95% interval (crossed task ×\times block bootstrap). Fixed-pool intervals use a block bootstrap over 20–22 blocks, or a tt interval over three blocks for the stress configurations ( s ). Pools average the configuration estimates with equal weight, and stress configurations are reported separately. 4B and 8B denote Qwen3-4B and Qwen3-8B.
ConfigurationF−GF-GF−HF-Hoff-diag. %rev. %mean II
4B GPTQ-W4+0.5+0.5 [ −1.2-1.2 , +2.2+2.2 ] [ −4.7-4.7 , +6.1+6.1 ]−3.2-3.2 [ −6.4-6.4 , +0.2+0.2 ] [ −11.1-11.1 , +4.4+4.4 ]25.02.6+0.3+0.3
4B AWQ-style W4−0.4-0.4 [ −2.4-2.4 , +1.5+1.5 ] [ −6.2-6.2 , +5.4+5.4 ]−1.7-1.7 [ −3.6-3.6 , +0.4+0.4 ] [ −8.7-8.7 , +4.7+4.7 ]29.33.1+1.3+1.3
8B GPTQ-W4+2.1+2.1 [ +0.2+0.2 , +4.0+4.0 ] [ −3.0-3.0 , +7.1+7.1 ]+5.0+5.0 [ +3.2+3.2 , +6.8+6.8 ] [ −2.0-2.0 , +12.5+12.5 ]22.92.7−1.7-1.7
8B AWQ-style W4−1.5-1.5 [ −4.2-4.2 , +0.8+0.8 ] [ −7.6-7.6 , +4.2+4.2 ]+1.9+1.9 [ −0.1-0.1 , +3.8+3.8 ] [ −3.8-3.8 , +7.5+7.5 ]30.54.2+0.3+0.3
W4 pool+0.2+0.2 [ −1.0-1.0 , +1.2+1.2 ] [ −2.8-2.8 , +3.1+3.1 ]+0.5+0.5 [ −1.0-1.0 , +2.1+2.1 ] [ −3.5-3.5 , +4.5+4.5 ]26.93.2+0.0+0.0
4B KV8−0.2-0.2 [ −3.5-3.5 , +3.7+3.7 ] [ −5.7-5.7 , +5.3+5.3 ]+1.3+1.3 [ −2.5-2.5 , +4.9+4.9 ] [ −6.7-6.7 , +9.5+9.5 ]15.71.2−2.4-2.4
Appendix
Table 7: Paired mode contrasts and outcome patterns on the states of Table 6 . Two-line cells give the estimate with its fixed-pool 95% interval on the first line and the new-task 95% interval on the second, both unadjusted (pp). ΔQ=YBQ−YQQ\Delta_{\mathrm{Q}}=Y_{\mathrm{BQ}}-Y_{\mathrm{QQ}} and ΔB=YBB−YQB\Delta_{\mathrm{B}}=Y_{\mathrm{BB}}-Y_{\mathrm{QB}} are the effects of swapping in the BF16 action with the quantized model and with the BF16 model continuing. Off-diag.: % of states with ΔQ≠ΔB\Delta_{\mathrm{Q}}\neq\Delta_{\mathrm{B}} . Rev.: % of states with ΔQ=−ΔB≠0\Delta_{\mathrm{Q}}=-\Delta_{\mathrm{B}}\neq 0 . Mean II : mean of the interaction I=ΔQ−ΔBI=\Delta_{\mathrm{Q}}-\Delta_{\mathrm{B}} . G−HG-H compares the two upgrades with each other, and each upgrade’s value against QQ appears in Table 6 .
A. Joint rescue
Groupstatesobs.ref.obs. −- ref. [fixed pool][new tasks]
τ2\tau^{2} -bench W4 pool55763.963.48+0.47+0.47 [ +0.06+0.06 , +0.87+0.87 ][ −0.33-0.33 , +1.37+1.37 ]
4B W4 configurations25413.223.56−0.34-0.34 [ −0.83-0.83 , +0.13+0.13 ][ −1.46-1.46 , +0.88+0.88 ]
8B W4 configurations30354.693.40+1.29+1.29 [ +0.79+0.79 , +1.78+1.78 ][ −0.07-0.07 , +2.74+2.74 ]
τ2\tau^{2} -bench KV8 pool14482.992.17+0.81+0.81 [ +0.10+0.10 , +1.52+1.52 ][ −0.51-0.51 , +2.19+2.19 ]
Telecom 8B W4 pool9573.243.94−0.70-0.70 [ −1.46-1.46 , +0.05+0.05 ][ −2.47-2.47 , +1.15+1.15 ]
Appendix
Table 8: Joint rescue and rescue undone against the label-permutation reference that keeps the number kk of successful branches per state (% of divergent states). Obs.: observed share. Ref.: reference share. Fixed-pool 95% intervals use a block bootstrap for the 20- and 22-block τ2\tau^{2} -bench pools and a tt interval over per-block ratios for pools with fewer than 10 blocks (six, or five for Mistral). New-task 95% intervals use a crossed task ×\times block bootstrap. The same-precision rows each fork at their own first divergence. m States at which the Mistral actions differ on BF16 trajectories, so BB is the BF16 original, five blocks. These rows are kept separate from the first-divergent-state rows.
Configuration (weight)unitsexcl.GallG_{\mathrm{all}}rrddρc\rho_{c}
4B GPTQ-W4 (1/2)3961−3.03-3.03 [ −7.86-7.86 , +1.80+1.80 ]+5.05+5.05+8.08+8.080.668
8B GPTQ-W4 (1/4)4080−5.64-5.64 [ −9.33-9.33 , −1.94-1.94 ]+4.41+4.41+10.05+10.050.374
8B AWQ-style W4 (1/4)3961−6.57-6.57 [ −11.98-11.98 , −1.15-1.15 ]+6.31+6.31+12.88+12.880.644
Weighted ( GallG_{\mathrm{all}} )  −4.57-4.57 [ −7.17-7.17 , −1.96-1.96 ]+5.21+5.21+9.77+9.770.588
Appendix
Table 9: Telecom holdout, 34 tasks ×\times 12 blocks (pp of units, crossed 90% intervals). Units: task–block units analysed. Excl.: tasks structurally excluded because the decision count and the trajectory count disagreed at state search. GallG_{\mathrm{all}} : full-pool net gain of action-only replacement. rr , dd : rescue and damage rates. ρc\rho_{c} : correlation of per-task turnover between the two six-block halves, the statistic of the repeatability check. The weighted row is the registered main estimate.
FamilyLevelnnGallG_{\mathrm{all}} [crossed 90%]ppHolm pp
step budget aT1138/129/110−0.32-0.32 [ −1.57-1.57 , +0.94+0.94 ].6771.6771
 T2124/126/162−2.30-2.30 [ −3.88-3.88 , −0.72-0.72 ].0165.0495
 T3123/144/124−1.94-1.94 [ −3.35-3.35 , −0.54-0.54 ].0229.0495
personaHard11−5.99-5.99 [ −12.23-12.23 , +0.26+0.26 ].1147.3355
 None12−3.65-3.65 [ −7.42-7.42 , +0.13+0.13 ].1118.3355
 Easy11−4.36-4.36 [ −9.19-9.19 , +0.48+0.48 ].1385.3355
Appendix
Table 10: Telecom holdout strata: the six registered families, with Holm correction within each family over its three levels (pp). T1–T3 are tertiles from lowest to highest, with cut points fixed on telecom development tasks. For task-level families, nn is the number of tasks and GallG_{\mathrm{all}} is computed over the tasks in the level. a The step budget (remaining steps at the divergence) is a unit-level family: nn gives the number of divergent units per configuration (4B GPTQ-W4/8B GPTQ-W4/8B AWQ-style W4), and GallG_{\mathrm{all}} is the level’s contribution to the overall GallG_{\mathrm{all}} , so the three levels sum to GallG_{\mathrm{all}} .
SettingStatesNet gain [CI]
τ2\tau^{2} -bench retail, airlineall units+0.7+0.7 [ −2.1-2.1 , +3.4+3.4 ]
τ2\tau^{2} -bench telecom holdoutall units−4.6-4.6 [ −7.2-7.2 , −2.0-2.0 ]
BFCL multi-turnall items+0.1+0.1 [ −1.0-1.0 , +1.2+1.2 ]
ALFWorld (8B)all items−3.7-3.7 [ −9.7-9.7 , +1.9+1.9 ]
Mistral GPTQ-W4ordinary−0.8-0.8 [ −5.2-5.2 , +3.6+3.6 ]
Mistral RTN-W4ordinary+0.7+0.7 [ −2.7-2.7 , +4.2+4.2 ]
Appendix
Table 11: Action-only net gain (pp) across settings. Qwen3 rows are W4 pools over all units or items, with units without a divergence counted as zero. Mistral rows are GG at its own ordinary states on 83 retail and airline tasks. Intervals are new-task 95% (retail, airline), crossed 90% (holdout), item-bootstrap 95% (BFCL, ALFWorld) and fixed-pool 95% (Mistral). Per-configuration values: Appendix D.3 and D.2 .
ConfigurationblocksstatesGGHHFF
GPTQ-W46498−0.8-0.8 [ −5.2-5.2 , +3.6+3.6 ]+1.4+1.4 [ −3.0-3.0 , +5.8+5.8 ]+0.0+0.0 [ −4.2-4.2 , +4.2+4.2 ]
   [ −5.8-5.8 , +4.2+4.2 ][ −4.0-4.0 , +6.8+6.8 ][ −6.0-6.0 , +6.0+6.0 ]
RTN-W45415+0.7+0.7 [ −2.7-2.7 , +4.2+4.2 ]+1.9+1.9 [ −3.8-3.8 , +7.7+7.7 ]+2.4+2.4 [ −2.2-2.2 , +7.0+7.0 ]
   [ −3.6-3.6 , +5.5+5.5 ][ −4.6-4.6 , +8.0+8.0 ][ −3.9-3.9 , +8.4+8.4 ]
Appendix
Table 12: Mistral-Small-3.2-24B on its fixed 83-task conditional population of ordinary states sampled from BF16 histories. Effects in pp. Each entry gives the estimate with its fixed-pool 95% tt interval, then the crossed task ×\times block 95% interval. RTN-W4’s fourth block is excluded because of missing argument-JSON errors, while model-produced invalid tool names are scored as failures under the dated amendment (Appendix D.2 ).
A. Full item pool
ConfigurationitemsrrddMMGallG_{\mathrm{all}}
BFCL 4B GPTQ-W47886.34.310.7+2.0+2.0 [ −0.3-0.3 , +4.3+4.3 ]
BFCL 4B AWQ-style W47885.79.515.2−3.8-3.8 [ −6.5-6.5 , −1.3-1.3 ]
BFCL 8B GPTQ-W47882.92.35.2+0.6+0.6 [ −0.9-0.9 , +2.3+2.3 ]
BFCL 8B AWQ-style W47884.12.76.7+1.4+1.4 [ −0.3-0.3 , +3.3+3.3 ]
BFCL W4 pool    +0.1+0.1 [ −1.0-1.0 , +1.2+1.2 ]
Appendix
Table 13: Replacement and four-branch quantities on BFCL multi-turn and ALFWorld (no LLM user simulator, one run per item, 95% item-bootstrap intervals). 4B and 8B denote Qwen3-4B and Qwen3-8B. Panel A, all items: rescue rr , damage dd , turnover MM (% of all items) and net gain GallG_{\mathrm{all}} (pp), non-divergent items counting as zero. Panel B, divergent items (post hoc, pp): action GG , continuation HH , joint FF , and the off-diagonal share, the percentage of states with ΔQ≠ΔB\Delta_{\mathrm{Q}}\neq\Delta_{\mathrm{B}} . Pools weight configurations equally. s Stress and unrotated KV4 configurations. In unrotated KV4 rows, YQQ=0Y_{\mathrm{QQ}}=0 on every state, so every positive gain is a rescue.
ReferenceStates (denominator)Action replacing Q’s (continuation)Turn- overRescues / damagesNetPaired comparison [interval; 95% unless marked]
Second W4 checkpoint Q ′Own first divergences; Qwen3-4B and Qwen3-8B GPTQ-W4; 1020 task–block units (85 tasks ×\times 6 blocks per model)B (Q)16.88.0 / 8.7−0.7-0.7ΔM=−1.0\Delta M=-1.0 ; fixed [ −3.9-3.9 , +1.9+1.9 ]; new [ −4.5-4.5 , +2.7+2.7 ]
  Q ′ (Q)17.79.3 / 8.4+0.9+0.9 
Common states510 BF16-sourced Qwen3-8B states; 85 tasks ×\times 6 blocks; GPTQ-W4 substitutesB (Q)7.14.1 / 2.9+1.2+1.2K(Q)=+1.0K(\mathrm{Q})=+1.0 ; two-look 97% tt [ −4.2-4.2 , +6.2+6.2 ]; bootstrap [ −1.4-1.4 , +3.5+3.5 ]
  Q ′ (Q)8.04.1 / 3.9+0.2+0.2 
  B (B; vs. QB)9.05.3 / 3.7+1.6+1.6K(B)=+0.8K(\mathrm{B})=+0.8 ; two-look 97% tt [ −2.2-2.2 , +3.8+3.8 ]; bootstrap [ −2.0-2.0 , +3.7+3.7 ]
  Q ′ (B; vs. QB)8.64.7 / 3.9+0.8+0.8 
Appendix
Table 14: Reference comparisons behind Figure 3 and the common-state paragraph of § 4.3 , in percentage points (pp). Each row group has its own state source and denominator, and comparisons are paired only within a group. In the table, B denotes the BF16 model, Q the quantized model and Q ′ the second checkpoint. The references are reported beside the BF16 turnover and left unsubtracted. Turnover, rescues, damages and net gain compare the replacing action with the quantized action under the continuation shown in parentheses: against QQ when the quantized model continues, and against the quantized action followed by the BF16 model (QB) when the BF16 model continues. ΔM\Delta M is the BF16 turnover minus the second checkpoint’s, and K(c)K(c) is success with the BF16 action minus success with the second checkpoint’s action under continuation cc . Intervals follow each protocol. For the second checkpoint, “fixed” is a tt interval over 6 block means and “new” a crossed new-task bootstrap. For common states, a reading needs both the tt interval over block means and the task-cluster bootstrap, shown here under the two-look (Pocock) reading that the protocol designates confirmatory. For key order, the interval is the wider of block and crossed bootstrap. The key-order control is exploratory, its analysis code was frozen before outcomes were read, and it set no equivalence margin. Per-arm intervals appear in Appendices E.1 and E.4 , and unadjusted common-state intervals and the extension timeline in Appendix E.2 .
Benefit comparisonEstimate [interval] (pp)
W4 action-only vs. QQ+0.7+0.7 [ −2.1-2.1 , +3.4+3.4 ]
BF16 vs. Q ′ , Q continues+1.0+1.0 [ −4.2-4.2 , +6.2+6.2 ]
BF16 vs. Q ′ , B continues+0.8+0.8 [ −2.2-2.2 , +3.8+3.8 ]
Appendix
Table 15: Intervals of three benefit comparisons, in percentage points (pp). In row labels, B is the BF16 model, Q the quantized model and Q ′ the second checkpoint. W4 row: net gain of the BF16 action with the quantized model continuing, against QQ, with the full-pool new-task 95% interval. Common-state rows: K(c)K(c) with the two-look 97% tt interval over block means. Bootstrap intervals appear in Appendix E.2 . Rows differ in population and confidence level.
A. Frozen trigger and matched hindsight oracle
Ratesel.oracle r/d/net (pp)frozen r/d/net (pp)net ratio
5%5034/0/ +34+34 ( +3.33+3.33 )6/2/ +4+4 ( +0.39+0.39 )4/34
10%11438/0/ +38+38 ( +3.73+3.73 )7/5/ +2+2 ( +0.20+0.20 )2/38
20%22238/0/ +38+38 ( +3.73+3.73 )16/11/ +5+5 ( +0.49+0.49 )5/38
Appendix
Table 16: Frozen trigger, which reads only the quantized model, against a matched action-replacement oracle (hindsight, same per-configuration selection counts) and matched-count random selection, on the 1020 validation rows from the quantized model’s own histories. Rate: nominal selection rate. sel.: number of selected states. r/d/net: rescues, damages and net gain, with the net gain in percentage points (pp) of all rows in parentheses. Net ratio: frozen net gain over oracle net gain. Random: matched-count per-configuration random-selection expectation. pp : the exact one-sided random-selection tail defined below. Holm correction across the three rates applies to the net tests only.
More in this topic

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 838 papers

Explore similar work

CardsList

  1. Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

    Jul 29, 2026Jiwon Jang, Kisu Yang, Heuiseok Lim +1Post-Training QuantizationAI Agent Reliability

    VAIV Company · Seoul National University · Korea University

  2. Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit

    Sep 25, 2026Jakub PrejznerLLM Tool UseLLM Quantization

    Independent Researcher

  3. Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

    Aug 6, 2026Zekun Wu, Swati Dhiman, Adriano KoshiyamaPost-Training QuantizationLow-Bit Quantization

    Holistic AI · UCL Centre for Artificial Intelligence, University College London