Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

Janghoon Lee

cs.CLAug 14, 2026

Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

Authors: Janghoon Lee

Organizations: Redrob

Abstract

A tool-calling router has to pick the right tool when one applies and decline when none does. Restricting the decoder to a grammar over the tool names is the standard remedy for the first, and on small models it buys a large accuracy gain. Recent work separates the loss caused by asking for a format from the loss caused by enforcing it at decode time. The second is small, which made enforcement look nearly free. The same work declines to extend that to function calling, where a constraint decides which answers exist rather than how one is written. Declining to call anything is the answer it most easily removes, and the one a router can least afford to lose. A grammar decides which tokens may be emitted and where generation stops, and a two-condition design charges both to the restriction. We therefore run three conditions over one prompt: free generation, generation stopped at the first line, and both applied together. We evaluate open-weight models from 0.6B to 4B on the same items in English and Korean, comparing the languages item by item. The two-condition contrast is negative on abstention accuracy in four of six cells with intervals excluding zero, and positive in none, costing -29.5 points at worst. On the smallest model in Korean the stop costs -20.0 points, the restriction returns +19.5, and together they leave -0.5. What the restriction gives back is readable output, not judgment. Of the 698 abstentions it repairs, 545 had no readable answer at all and 0 were correct decisions the scoring rule rejected. On items that do need a tool the contrast is positive throughout, and abstention is reported first because it is the registered measure. Both preregistered claims about language fail: Korean does not lose more of the abstentions it holds without the constraint, and the removed mass does not explain what does.

Figures & tables

Model	Lang	Free abstain	Free tool sel.	Short abstain	Short tool sel.	Constrained abstain	Constrained tool sel.
Qwen3-0.6B	English	65.0 [58.5, 71.5]	26.0 [20.0, 32.5]	59.0 [52.0, 66.0]	16.5 [11.5, 22.0]	35.5 [28.5, 42.5]	43.0 [36.0, 50.0]
	Korean	64.0 [57.0, 70.5]	18.5 [13.5, 24.0]	44.0 [37.0, 51.0]	6.0 [3.0, 9.5]	63.5 [57.0, 70.0]	38.5 [32.0, 45.5]
Qwen3-1.7B	English	77.0 [71.0, 82.6]	33.8 [27.5, 40.5]	74.5 [68.0, 80.5]	34.0 [27.5, 40.5]	50.5 [43.5, 57.5]	45.0 [38.0, 52.0]
	Korean	73.0 [66.9, 78.9]	30.9 [24.7, 37.2]	61.5 [54.5, 68.0]	30.5 [24.5, 37.0]	48.5 [41.5, 55.0]	41.0 [34.0, 48.0]
Qwen3-4B	English	57.0 [50.1, 63.8]	12.9 [8.6, 17.5]	57.0 [50.1, 63.8]	66.6 [60.0, 73.1]	48.2 [41.2, 55.0]	72.5 [66.0, 78.5]
	Korean	58.2 [51.3, 64.9]	6.0 [3.1, 9.6]	57.7 [50.8, 64.4]	55.2 [48.1, 62.1]	59.7 [53.0, 66.4]	68.7 [62.2, 75.0]

Table 1: Abstention accuracy and tool-selection accuracy (%) with 95% bootstrap confidence intervals. Intervals resample items and seeds jointly. Absolute values are reported for context only; the design licenses claims about the differences in Table 3 , not about these levels.

Model	$\|S\|$	KO / EN only	$D_{\text{paired}}$	Short English	Short Korean	$\Delta$ (KO) $-$ $\Delta$ (EN)
Qwen3-0.6B	375	15 / 100	-22.7 [-34.2, -11.4]	59.0 [52.0, 66.0]	44.0 [37.0, 51.0]	+43.0 [+33.5, +52.5]
Qwen3-1.7B	550	65 / 50	+2.7 [-5.7, +11.3]	74.5 [68.0, 80.5]	61.5 [54.5, 68.0]	+11.0 [+2.5, +19.5]
Qwen3-4B	480	4 / 59	-11.5 [-18.5, -4.9]	57.0 [50.1, 63.8]	57.7 [50.8, 64.4]	+10.8 [+4.8, +16.8]

Table 2: The registered primary statistic and the two secondary measures of the same effect.

D_{\text{paired}}

is the McNemar contrast on the items both languages abstained on correctly without the constraint: the share only Korean lost minus the share only English lost. Bold marks an interval excluding zero.

Model	Metric	Contrast	$\Delta$ English	$\Delta$ Korean	$\Delta$ (KO) $-$ $\Delta$ (EN)
Qwen3-0.6B	Abstention	Masking (constrained $-$ short)	-23.5 [-32.0, -15.5]	+19.5 [+10.5, +28.5]	+43.0 [+33.5, +52.5]
Qwen3-0.6B	Abstention	Length (short $-$ free)	-6.0 [-9.5, -3.0]	-20.0 [-25.5, -14.5]	-14.0 [-20.5, -7.5]
Qwen3-0.6B	Abstention	Both (constrained $-$ free)	-29.5 [-37.5, -21.5]	-0.5 [-9.5, +8.0]	+29.0 [+19.0, +39.0]
Qwen3-0.6B	Tool selection	Masking (constrained $-$ short)	+26.5 [+19.0, +34.0]	+32.5 [+25.5, +39.5]	+6.0 [-1.5, +13.5]
Qwen3-0.6B	Tool selection	Length (short $-$ free)	-9.5 [-16.5, -3.0]	-12.5 [-18.0, -7.5]	-3.0 [-11.5, +5.5]
Qwen3-0.6B	Tool selection	Both (constrained $-$ free)	+17.0 [+9.5, +24.5]	+20.0 [+12.0, +28.0]	+3.0 [-6.0, +12.0]

Table 3: Each difference the three conditions make available, in accuracy points. Negative values mean the change hurt. Bold marks an interval that excludes zero.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Effect
make check	SSH reachability and GPU inventory
make smoke	10 pairs, both languages, both conditions
make dataset	manifest, translation draft, review queue
make run	full sweep, resumable
make analyze	five tables, three figures
make paper	this PDF

Appendix

Table 4: Build targets.

Item	English	Korean
pair_0322	I need a car park with available lots near Marina Bay Sands.	[Korean text; build with xelatex to typeset it]
pair_0249	Can you please provide me with the current time in Paris, France accurately?	[Korean text; build with xelatex to typeset it]
pair_0732	Can you update me on the current status of my portfolio returns through Magnetis?	[Korean text; build with xelatex to typeset it]
pair_0538	Hello, Sudoku pro! … Can you provide me with a Sudoku puzzle to tackle?	[Korean text; build with xelatex to typeset it]
pair_0035	Search for Twitter keywords associated with the entertainment industry in Japan.	[Korean text; build with xelatex to typeset it]
pair_0796	Can you save my information so you can give me personalized assistance?	[Korean text; build with xelatex to typeset it]

Appendix

Table 5: Seven of the 200 items, English against Korean. Only the query is translated; the ten candidate names, their descriptions, the instruction block and the few-shot examples around it are the English original, byte for byte.

Model	Lang	Abst. free	Abst. short	Abst. constr.	Tool free	Tool short	Tool constr.	Ret.
Qwen3-0.6B	English	65.0	59.0	35.5	26.0	16.5	43.0	60.2
	Korean	64.0	44.0	63.5	18.5	6.0	38.5	144.3
Qwen3-1.7B	English	77.0	74.5	50.5	33.8	34.0	45.0	67.8
	Korean	73.0	61.5	48.5	30.9	30.5	41.0	78.9
Qwen3-4B	English	57.0	57.0	48.2	12.9	66.6	72.5	84.6
	Korean	58.2	57.7	59.7	6.0	55.2	68.7	103.5

Appendix

Table 6: Abstention accuracy ( Abst. ) and tool-selection accuracy ( Tool ), in percent, in each of the three conditions, with the retention ratio: constrained abstention accuracy over contract-short , which is the denominator throughout and is not the same baseline as contract-free .

Model	Lang	chat	compl.	difference	chat	compl.	chat	compl.
Qwen3-0.6B	English	0.950	0.714	+0.236 [+0.209, +0.269]	12.2	44.0	1.8	48.5
Qwen3-0.6B	Korean	0.852	0.801	+0.051 [+0.017, +0.091]	18.5	30.8	1.5	65.2
Qwen3-1.7B	English	1.000	0.333	+0.667 [+0.629, +0.707]	16.5	80.0	0.2	15.0
Qwen3-1.7B	Korean	1.000	0.661	+0.339 [+0.315, +0.363]	3.0	56.2	0.0	31.2
Qwen3-4B	English	0.991	0.268	+0.723 [+0.633, +0.771]	17.5	73.0	0.5	9.2
Qwen3-4B	Korean	1.000	0.504	+0.496 [+0.395, +0.604]	3.2	55.0	0.0	18.8

Appendix

Table 7: Appendix. The chat template against raw completions on the same items: median masked first-token mass, the share of first tokens that begin a candidate name (%), and the share of outputs with no readable answer (%). Same three models, both languages, the unconstrained condition, one seed, enable_thinking false.

Lang	$n$	Mean	Median	10th pct.	$\geq 0.99$
English	1200	0.831	0.986	0.282	46%
Korean	1200	0.893	0.997	0.592	57%

Appendix

Table 8: First-token probability mass the enum grammar would remove when the same prompts are issued through the chat template, unconstrained.

Model	mass alone	mass $\|$ lang	$a$	$a\times b$ indirect	$c^{\prime}$ direct	$c$ total
Qwen3-0.6B	-2.37 [-5.55, +0.37]	-1.79 [-5.15, +1.21]	+0.05 [+0.02, +0.08]	-0.09 [-0.26, +0.07]	-0.89 [-1.44, -0.41]	-0.97 [-1.55, -0.48]
Qwen3-1.7B	+2.37 [+1.27, +3.58]	+5.04 [+2.80, +8.12]	+0.34 [+0.31, +0.37]	+1.70 [+0.96, +2.78]	-1.64 [-2.88, -0.72]	+0.11 [-0.23, +0.45]
Qwen3-4B	+8.39 [+4.48, +14.41]	+9.95 [+5.67, +19.43]	+0.01 [-0.01, +0.03]	+0.10 [-0.10, +0.33]	-2.09 [-5.79, -1.00]	-1.53 [-3.39, -0.74]

Appendix

Table 9: Exploratory. Suppression and the path decomposition of the language effect on flipping.

Model	Model form	$n$	masked_mass	ko	difficulty
Qwen3-0.6B	recover $\sim$ mass + lang	970 / 410	+3.69 [+1.59, +6.44]	+1.70 [+1.15, +2.41]	–
Qwen3-0.6B	flip $\sim$ mass + lang + difficulty	750	-1.91 [-5.33, +1.12]	-0.91 [-1.47, -0.39]	+0.57 [-0.85, +0.82]
Qwen3-1.7B	recover $\sim$ mass + lang	640 / 205	+1.23 [-0.13, +3.08]	+0.30 [-0.42, +1.10]	–
Qwen3-1.7B	flip $\sim$ mass + lang + difficulty	1100	+5.09 [+2.74, +7.88]	-1.64 [-2.95, -0.76]	+0.25 [-0.74, +0.73]
Qwen3-4B	recover $\sim$ mass + lang	853 / 83	+2.09 [+0.33, +4.25]	+1.08 [+0.01, +2.82]	–
Qwen3-4B	flip $\sim$ mass + lang + difficulty	960	+10.01 [+5.79, +19.54]	-1.95 [-5.47, -0.97]	-0.60 [-1.17, +1.15]

Appendix

Table 10: Exploratory. Recovery as its own outcome, and the flip model with the item’s unconstrained difficulty as a covariate.

Model	Mass band	$n$ EN	flip % EN	$n$ KO	flip % KO	KO $-$ EN
Qwen3-0.6B	0.20–0.33	50	40.0	25	60.0	+20.0
Qwen3-0.6B	0.33–0.44	35	42.9	40	25.0	-17.9
Qwen3-0.6B	0.44–0.46	40	62.5	35	28.6	-33.9
Qwen3-0.6B	0.46–0.52	40	100.0	35	14.3	-85.7
Qwen3-0.6B	0.52–0.54	45	55.6	30	50.0	-5.6
Qwen3-0.6B	0.54–0.57	50	60.0	25	0.0	-60.0

Appendix

Table 11: Exploratory. Flip rate within bands of masked first-token mass, by language.

Terms	cells	Intercept (pp)	Baseline	masked_mass	$R^{2}$	Zero crossing (%)
baseline	6	–	–	–	–	–
baseline + mass	6	–	–	–	–	–

Appendix

Table 12: Exploratory. The masking effect regressed on the unconstrained abstention accuracy it is taken from, one point per (model, language) cell. Every coefficient is withheld: this fit is over the three registered models, and the guard asks for five cells and five models per predictor before a coefficient is reported, so the row is shown empty rather than filled from six cells sharing one family.

Figure 1: Exploratory. The masking effect against the contract-short abstention accuracy it is taken from, one point per (model, language) cell. Bars are the spread across seeds. The dotted line, when the fit is reported, marks where the fitted line crosses zero.

Cells fitted	$n$	D0 $-$ nonD0 [95% CI]
all cells	32	+0.083 [-0.186, +0.300]
without tier B	26	+0.063 [-0.418, +0.396]
without tier E	26	+0.087 [-0.203, +0.353]
without tier B and E	20	+0.181 [-0.293, +0.684]

Appendix

Table 13: The difference between the two coefficients of the decomposition, refitted on subsets of the model set. Every subset is reported rather than a chosen one, and the interval covers zero in all of them, so the verdict does not depend on which tiers are included. Exploratory.

Model	Term	Coefficient [95% CI]	$R^{2}$	AIC
baseline	baseline	-0.581 [-0.788, -0.395]	0.523	-108.6
D0	D0	+0.454 [+0.215, +0.662]	0.222	-93.0
nonD0	nonD0	+0.357 [+0.080, +0.632]	0.146	-90.0
D0 + nonD0	D0	+0.624 [+0.378, +0.857]	0.528	-106.9
	nonD0	+0.542 [+0.298, +0.780]

Appendix

Table 14: The unconstrained baseline’s error split into the share of items with no readable answer ( D0 ) and the share with a readable but wrong one ( nonD0 ); the two sum to one minus the baseline by construction. Bold marks an interval that excludes zero. Exploratory.

Model	Lang	Condition	ok
Qwen3-0.6B	English	Constrained	2000 (100.0%)
Qwen3-0.6B	English	Free	2000 (100.0%)
Qwen3-0.6B	English	Short	2000 (100.0%)
Qwen3-0.6B	Korean	Constrained	2000 (100.0%)
Qwen3-0.6B	Korean	Free	2000 (100.0%)
Qwen3-0.6B	Korean	Short	2000 (100.0%)

Appendix

Table 15: Request outcomes by cell. Non- ok requests are excluded from accuracy rather than counted as errors, so that infrastructure failure is not attributed to the model.

Field	Value
vllm	0.27.1
torch	2.13.0
cuda	13.0
gpu	NVIDIA L4
driver	595.71.05
api_path	structured_outputs

Appendix

Table 16: Execution environment of the recorded results.

	Zhou (2026)	This paper
Task	Self-critique, open-ended	Tool abstention, closed choice
Models	One, 8B	Three registered,
sixteen in total, from 0.6B
Languages	English	English and Korean
Constraint	Applied whole	Stop and enum separated
Evidence	Failure mode named	Item-level transition counts

Appendix

Table 17: The nearest antecedent to the second contribution.

	Li et al. (2026)	This paper
Constraint	JSON Schema, on an axis separate from the tool call	Enum over the answer set itself
Correct token	Unreachable under the mask	Present in the enum by construction
Result	Uniform across models	$\Delta_{\text{mask}}$ changes sign between cells
Measures	A property of the decoder	A property of the model’s judgment

Appendix

Table 18: Two constraint taxes with the same name and different subjects.

Cell	Abstention	Tool selection	Pooled
Qwen3-0.6B EN	-29.5	+17.0	-6.2 [-9.4, -3.2]
Qwen3-0.6B KO	-0.5	+20.0	+9.8 [+6.7, +12.8]
Qwen3-1.7B EN	-26.5	+11.2	-7.7 [-10.7, -4.6]
Qwen3-1.7B KO	-24.5	+10.1	-7.2 [-10.2, -4.0]
Qwen3-4B EN	-8.8	+59.6	+25.4 [+22.4, +28.3]
Qwen3-4B KO	+1.5	+62.7	+32.1 [+29.1, +35.0]

Appendix

Table 19:

\Delta_{\text{total}}

, constrained minus contract-free, in accuracy points: on abstention items, on tool-needed items, and pooled over both. The two item types are equinumerous by construction, so the pooled column is an unweighted mean over 400 items and needs no weighting choice.

Model	Terms	$n$	flips	masked_mass	ko
Qwen3-0.6B	mass	750	295	-2.37 [-5.55, +0.37] $p=0.089$	–
Qwen3-0.6B	mass + lang	750	295	-1.79 [-5.15, +1.21] $p=0.246$	-0.89 [-1.48, -0.40] $p<0.001$
Qwen3-1.7B	mass	1100	455	+2.37 [+1.27, +3.58] $p<0.001$	–
Qwen3-1.7B	mass + lang	1100	455	+5.04 [+2.80, +8.12] $p<0.001$	-1.64 [-2.92, -0.74] $p<0.001$
Qwen3-4B	mass	960	91	+8.39 [+4.48, +14.41] $p<0.001$	–
Qwen3-4B	mass + lang	960	91	+9.95 [+5.67, +19.43] $p<0.001$	-2.09 [-5.52, -1.02] $p<0.001$

Appendix

Table 20: Logistic regression of flip — an abstention held without the constraint and lost with it — on the first-token probability mass the grammar removes, with and without a language term.

Model	Lang	Mean mass	Median mass	$n$	Slope	$R^{2}$
Qwen3-0.6B	English	0.685	0.695	200	+1.069	0.059
Qwen3-0.6B	Korean	0.754	0.771	200	+2.272	0.163
Qwen3-1.7B	English	0.386	0.382	200	+0.439	0.027
Qwen3-1.7B	Korean	0.634	0.661	200	+0.889	0.057
Qwen3-4B	English	0.375	0.365	200	+0.121	0.006
Qwen3-4B	Korean	0.516	0.520	200	+0.283	0.030

Appendix

Table 21: Masked first-token probability mass and its relation to the abstention penalty.

Figure 2: Per-item abstention penalty against the first-token probability mass the enum grammar removes, by language, with least-squares fits.

Figure 3: Distribution of masked first-token probability mass, English against Korean.

Figure 4: Abstention penalty against model size.

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 295 papers

Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions
Sep 16, 2026Janghoon LeeTool-Augmented Language Model Agents On-Device Language Model Inference
Redrob
Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance
Oct 26, 2026Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan +1Constrained Decoding Constraint-Guided Reasoning in Language Models
Télécom Paris, Institut Polytechnique de Paris, France · Nokia Bell Labs
Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints
Jun 24, 2026Fangzheng Li, Aimin Zhang, Chen LvConstrained Decoding LLM Tool Use
1Focus AI Center, Focus Technology Co., Ltd. · 2Nanjing University of Science and Technology

Sep 16, 2026J/K move · Enter open · S save

Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions

Janghoon Lee

Redrob

Tool-Augmented Language Model Agents On-Device Language Model Inference LLM Abstention Tool Use LLM Agent Routing

Open paper AskPDF arXiv

An AI assistant that calls tools makes two decisions on every request: which tool to invoke, and whether any available tool applies. In the usual design a single language model makes both, by emitting a call or by declining to emit one. On a device that has to answer without a server, the language model is what makes that design expensive, dominating both the latency and the memory of the router. The common alternative is to remove the model completely and rank the catalog of local actions with a retriever instead. That substitution is not symmetric across the two decisions. A retriever returns its highest-scoring candidate for every input and cannot signal that the catalog holds no valid action. Our earlier study found that constraining a decoder to a tool grammar repairs malformed output without improving the choice. What the substitution costs in each decision has not been measured. We evaluate the two decisions separately over 600 Korean and English requests and a catalog of 70 local actions. The router may also ask for a missing slot, reply, or delegate. Half the in-catalog requests reuse catalog vocabulary and half paraphrase it, separating lexical overlap from the action requested. Character 3-gram BM25 selects 162 of 164 lexically matched requests and 85 of 166 paraphrases. Restricting the candidate set to seven raises the paraphrase figure to a mean of 0.825 over five trials. No classifier over its score features separates in-catalog from out-of-catalog above 0.697 area under the curve, where the frozen encoder multilingual-e5-base reaches 0.806. Using that encoder for abstention alone keeps 376 of the requests local and misroutes 9 of the 150 needing delegation. Abstention, not selection, is where a neural component is required. A neural ranker improves every quality metric and is rejected on latency and memory rather than accuracy.

Abstract

LLM Tool Use

Explore similar work

Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints