Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Ngoc Trinh Hung Nguyen; Alonso Silva; Armen Aghasaryan; Mehwish Alam

cs.CLOct 26, 2026

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Authors: Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan, Mehwish Alam

Organizations: Télécom Paris, Institut Polytechnique de Paris, France · Nokia Bell Labs

Abstract

Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning. We examine this assumption through a taxonomy of three constraint types: format constraints, which enforce output structure; validity constraints, which mask invalid answer; and reasoning constraints, which restrict reasoning trajectories. We theoretically show that appropriate constraints can preserve or improve performance by eliminating invalid paths. Across four model families and eleven open-source models, causal ablations on diverse benchmarks using state-of-the-art constrained decoding strategies reveal that apparent performance degradation largely stems from non-equivalent prompting and output formats rather than from decoder-level masking. Meanwhile, validity constraints improve reliability and extractability by eliminating invalid answers, while reasoning constraints can improve reasoning and, consequently, accuracy when they enforce beneficial trajectories. These findings show that the impact of constrained decoding depends on the type of constraint and the output space it removes. Our code is available.

Figures & tables

Model	NL (Baseline)	JSON +FC+VC	NL-to-Format Unconstr. (plain-text)	+FC+VC
Qwen3-1.7B	33.6 $\pm$ 21.9	45.4 $\pm$ 19.1	60.7 $\pm$ 4.3	66.1 $\pm$ 4.8
Qwen3-4B	32.1 $\pm$ 26.2	41.9 $\pm$ 21.6	57.5 $\pm$ 7.5	60.4 $\pm$ 7.1
Qwen3-8B	63.8 $\pm$ 17.4	78.4 $\pm$ 7.9	69.4 $\pm$ 11.4	72.4 $\pm$ 10.6
Qwen3.5-2B	22.4 $\pm$ 29.9	69.1 $\pm$ 2.2	62.9 $\pm$ 5.6	68.7 $\pm$ 5.2
Qwen3.5-4B	81.6 $\pm$ 1.6	71.6 $\pm$ 28	84.1 $\pm$ 2.3	86.9 $\pm$ 2.6
Qwen3.5-9B	85.9 $\pm$ 1.2	77 $\pm$ 27	86.8 $\pm$ 1.2	89.9 $\pm$ 0.5

Table 1: LastLetter Task: comparison of unconstrained and combined Format and Validity Constraints (FC+VC) for plain-text answers using the NL-to-Format decoding strategy with the same prompt and reasoning traces. Italicized results are reported by Nguyen et al. (2026) , while Bold values indicate the highest accuracy.

Model	NL-to-Format Unconstr. (plain-text)	+FC+VC	NL-to-Format Unconstr. (JSON)	+FC+VC
Qwen3-1.7B	60.7 $\pm$ 4.3	66.1 $\pm$ 4.8	67.3 $\pm$ 4.1	68.9 $\pm$ 4.6
Qwen3-4B	57.5 $\pm$ 7.5	60.4 $\pm$ 7.1	60.3 $\pm$ 6.4	61.4 $\pm$ 6.5
Qwen3-8B	69.4 $\pm$ 11.4	72.4 $\pm$ 10.6	72.1 $\pm$ 10.5	73.5 $\pm$ 10.6
Qwen3.5-2B	62.9 $\pm$ 5.6	68.7 $\pm$ 5.2	68.6 $\pm$ 5.9	71.3 $\pm$ 5.4
Qwen3.5-4B	84.1 $\pm$ 2.3	86.9 $\pm$ 2.6	87.7 $\pm$ 1.5	88.8 $\pm$ 1.5
Qwen3.5-9B	86.8 $\pm$ 1.2	89.9 $\pm$ 0.5	89.6 $\pm$ 1.0	91.6 $\pm$ 1.0

Table 2: LastLetter Task: comparison of plain-text and JSON answer formats using the NL-to-Format decoding strategy with the same free-reasoning (NL) traces, under unconstrained decoding and combined Format and Validity Constraints (FC+VC). Bold values indicate the highest accuracy.

Figure 1: Comparison of the performance gains from Format Constraint (FC) and Validity Constraint (VC) on the LastLetter and LastLetter-Extended tasks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment Type	Parameter	Value	Note
Standard Setting	Temperature	0
	Sampling	False
	Max Reasoning Tokens	600
	Thinking Mode	Off
	Random Seed	1
Extended-Reasoning Setting (AIME-2026 only)	Temperature	Default

Appendix

Table 3: Experimental configurations for the two experimental settings.

Model	Unconstrained Baseline	Label- Constrained	Alternative- Constrained
Qwen3-1.7B	81.6	81.6	76.0
Qwen3-4B	92.6	92.6	83.4
Qwen3.5-2B	89.6	89.6	80.8
Qwen3.5-4B	94.6	94.8	85.8

Appendix

Table 4: IMDb sentiment classification under different decoding constraints, comparing the Unconstrained Baseline, Label-Constrained Decoding, and Alternative-Constrained Decoding.

$n_{\mathrm{shots}}$	Model	NL (Baseline)	JSON +FC+VC	NL-to-Format Unconstr. (plain-text)	+FC+VC
0	Qwen3-1.7B	33.6 $\pm$ 21.9	45.4 $\pm$ 19.1	60.7 $\pm$ 4.3	66.1 $\pm$ 4.8
	Qwen3-4B	32.1 $\pm$ 26.2	41.9 $\pm$ 21.6	57.5 $\pm$ 7.5	60.4 $\pm$ 7.1
	Qwen3-8B	63.8 $\pm$ 17.4	78.4 $\pm$ 7.9	69.4 $\pm$ 11.4	72.4 $\pm$ 10.6
	Qwen3.5-2B	22.4 $\pm$ 29.9	69.1 $\pm$ 2.2	62.9 $\pm$ 5.6	68.7 $\pm$ 5.2
	Qwen3.5-4B	81.6 $\pm$ 1.6	71.6 $\pm$ 28	84.1 $\pm$ 2.3	86.9 $\pm$ 2.6
	Qwen3.5-9B	85.9 $\pm$ 1.2	77 $\pm$ 27	86.8 $\pm$ 1.2	89.9 $\pm$ 0.5

Appendix

Table 5: LastLetter Task: comparison of unconstrained and combined Format and Validity Constraints (FC+VC) for plain-text answers using the NL-to-Format decoding strategy with the same prompt and reasoning traces. Bold values indicate the highest accuracy.

Model	JSON Unconstr.	+FC +VC	InWriting Unconstr.	+FC +VC	NL-to-Format Unconstr.	+FC +VC
Llama3-8B	42.1 $\pm$ 11.8	71.8 $\pm$ 1.6	74.7 $\pm$ 1.3	77.8 $\pm$ 1.2	76.4 $\pm$ 1.1	77.1 $\pm$ 1.2
Gemma2-9B	0 $\pm$ 0	86.7 $\pm$ 0.3	82.9 $\pm$ 1.4	86.7 $\pm$ 0.6	83.6 $\pm$ 1.6	84.4 $\pm$ 1.3
Qwen3-1.7B	17.6 $\pm$ 24.8	63.4 $\pm$ 0.5	68.8 $\pm$ 8.5	75.8 $\pm$ 0.8	72.9 $\pm$ 1.1	74.8 $\pm$ 1.1
Qwen3-4B	85.3 $\pm$ 1.2	88.3 $\pm$ 0.2	85.8 $\pm$ 3.0	89.7 $\pm$ 0.4	86.0 $\pm$ 1.1	87.4 $\pm$ 0.8
Qwen3-8B	56.7 $\pm$ 22.4	90.9 $\pm$ 0.4	89.4 $\pm$ 0.9	91.4 $\pm$ 0.5	88.8 $\pm$ 1.6	90.1 $\pm$ 1.2
Qwen3.5-2B	61 $\pm$ 1.2	69.9 $\pm$ 0.9	47.1 $\pm$ 28.4	74.9 $\pm$ 2.2	69.0 $\pm$ 4.6	72.1 $\pm$ 4.0

Appendix

Table 6: GSM8K: comparison of unconstrained (Unconstr.) and combined Format and Validity Constraints (FC+VC) across JSON, InWriting, and NL-to-Format decoding strategies. Bold values indicate the highest accuracy.

Model	JSON Unconstr.	+FC+VC	InWriting Unconstr.	+FC+VC	NL-to-Format Unconstr.	+FC+VC
Llama3-8B	0 $\pm$ 0	2.0 $\pm$ 1.4	53.6 $\pm$ 5.4	71.0 $\pm$ 3.8	68.7 $\pm$ 4.4	71.3 $\pm$ 4.0
Gemma2-9B	0 $\pm$ 0	43.8 $\pm$ 2.2	42.5 $\pm$ 4.1	56.6 $\pm$ 4.4	51 $\pm$ 5.8	54.1 $\pm$ 5.0
Qwen3-1.7B	0 $\pm$ 0	9.8 $\pm$ 1.1	54.2 $\pm$ 4.8	65.2 $\pm$ 5.1	67.3 $\pm$ 4.1	68.9 $\pm$ 4.6
Qwen3-4B	55.6 $\pm$ 5.7	56.7 $\pm$ 5.5	55.2 $\pm$ 8.0	59.6 $\pm$ 7.3	60.3 $\pm$ 6.4	61.4 $\pm$ 6.5
Qwen3-8B	75.6 $\pm$ 2.8	79.6 $\pm$ 2.8	72.0 $\pm$ 4.9	76.4 $\pm$ 5.6	72.1 $\pm$ 10.5	73.5 $\pm$ 10.6
Qwen3.5-2B	60.9 $\pm$ 1.7	69.1 $\pm$ 2.8	50.1 $\pm$ 21.9	75.5 $\pm$ 2.7	68.6 $\pm$ 5.9	71.3 $\pm$ 5.4

Appendix

Table 7: LastLetter Task: comparison of unconstrained (Unconstr.) and combined Format and Validity Constraints (FC+VC) across JSON, InWriting, and NL-to-Format decoding strategies. Bold values indicate the highest accuracy. X indicates that the corresponding experiment is unavailable.

Model	JSON Unconstr.	+FC+VC	InWriting Unconstr.	+FC+VC	NL-to-Format Unconstr.	+FC+VC
Llama3-8B	0 $\pm$ 0	2.4 $\pm$ 3.5	15.5 $\pm$ 3.6	28.7 $\pm$ 5.8	28.8 $\pm$ 9.6	31.9 $\pm$ 10.4
Gemma2-9B	0 $\pm$ 0	17.6 $\pm$ 0.3	14.0 $\pm$ 1.8	23.9 $\pm$ 4.1	24.0 $\pm$ 3.1	25.6 $\pm$ 3.0
Qwen3-1.7B	0 $\pm$ 0	0 $\pm$ 0	13.5 $\pm$ 4.8	23.8 $\pm$ 6.9	27.0 $\pm$ 5.6	28.4 $\pm$ 5.3
Qwen3-4B	15.3 $\pm$ 0.0	19.1 $\pm$ 0.6	22.4 $\pm$ 6.4	33.4 $\pm$ 7.8	26.1 $\pm$ 7.7	27.5 $\pm$ 7.3
Qwen3-8B	34.4 $\pm$ 2.7	39.6 $\pm$ 3.5	38.7 $\pm$ 5.4	47.3 $\pm$ 6.2	42.1 $\pm$ 7.2	43.5 $\pm$ 6.9
Qwen3.5-2B	15.1 $\pm$ 4.4	20.4 $\pm$ 5.4	18.1 $\pm$ 9.3	38.4 $\pm$ 8.2	34.4 $\pm$ 11.3	36.5 $\pm$ 10.0

Appendix

Table 8: LastLetter-Extended Task: comparison of unconstrained (Unconstr.) and combined Format and Validity Constraints (FC+VC) across JSON, InWriting, and NL-to-Format decoding strategies. Bold values indicate the highest accuracy. X indicates that the corresponding experiment is unavailable.

Model	JSON Unconstr.	+FC	+FC+VC	InWriting Unconstr.	+FC	+FC+VC
Llama3-8B	0 $\pm$ 0	1.8 $\pm$ 1.6	2.0 $\pm$ 1.4	53.6 $\pm$ 5.4	53.6 $\pm$ 5.4	71.0 $\pm$ 3.8
Gemma2-9B	0 $\pm$ 0	35.8 $\pm$ 2.7	43.8 $\pm$ 2.2	42.5 $\pm$ 4.1	42.6 $\pm$ 4.0	56.6 $\pm$ 4.4
Qwen3-1.7B	0 $\pm$ 0	8.4 $\pm$ 1.4	9.8 $\pm$ 1.1	54.2 $\pm$ 4.8	55.0 $\pm$ 5.4	65.2 $\pm$ 5.1
Qwen3-4B	55.6 $\pm$ 5.7	55.8 $\pm$ 5.6	56.7 $\pm$ 5.5	55.2 $\pm$ 8.0	56.4 $\pm$ 6.9	59.6 $\pm$ 7.3
Qwen3-8B	75.6 $\pm$ 2.8	75.6 $\pm$ 2.8	79.6 $\pm$ 2.8	72.0 $\pm$ 4.9	72.0 $\pm$ 4.9	76.4 $\pm$ 5.6
Qwen3.5-2B	60.9 $\pm$ 1.7	61.6 $\pm$ 2.7	69.1 $\pm$ 2.8	50.1 $\pm$ 21.9	63.0 $\pm$ 2.2	75.5 $\pm$ 2.7

Appendix

Table 9: LastLetter Task: decomposing constrained decoding gains relative to unconstrained (Unconstr.) into Format Constraint (FC) and Validity Constraint (VC) across JSON and InWriting decoding strategies.

Model	JSON Unconstr.	+FC	+FC+VC	InWriting Unconstr.	+FC	+FC+VC
Llama3-8B	0 $\pm$ 0	1.8 $\pm$ 2.5	2.4 $\pm$ 3.5	15.5 $\pm$ 3.6	15.5 $\pm$ 3.6	28.7 $\pm$ 5.8
Gemma2-9B	0 $\pm$ 0	15.6 $\pm$ 0.6	17.6 $\pm$ 0.3	14.0 $\pm$ 1.8	14.0 $\pm$ 1.8	23.9 $\pm$ 4.1
Qwen3-1.7B	0 $\pm$ 0	0 $\pm$ 0	0 $\pm$ 0	13.5 $\pm$ 4.8	13.8 $\pm$ 4.9	23.8 $\pm$ 6.9
Qwen3-4B	15.3 $\pm$ 0.0	15.8 $\pm$ 0.3	19.1 $\pm$ 0.6	22.4 $\pm$ 6.4	22.8 $\pm$ 6.5	33.4 $\pm$ 7.8
Qwen3-8B	34.4 $\pm$ 2.7	34.4 $\pm$ 2.7	39.6 $\pm$ 3.5	38.7 $\pm$ 5.4	39.0 $\pm$ 5.4	47.3 $\pm$ 6.2
Qwen3.5-2B	15.1 $\pm$ 4.4	15.1 $\pm$ 4.4	20.4 $\pm$ 5.4	18.1 $\pm$ 9.3	27.2 $\pm$ 6.6	38.4 $\pm$ 8.2

Appendix

Table 10: LastLetter-Extended Task: decomposing constrained decoding gains relative to unconstrained (Unconstr.) into Format Constraint (FC) and Validity Constraint (VC) across JSON and InWriting decoding strategies.

Model	JSON Unconstr.	+FC	InWriting Unconstr.	+FC	NL-to-Format Unconstr.	+FC
Llama3-8B	58.7	99.9	99.4	100	99.3	100
Gemma2-9B	0.2	100	99.8	100	99.7	100
Qwen3-1.7B	28.5	97.3	90.7	100	97.9	100
Qwen3-4B	96.7	99.8	97.3	100	98.2	100
Qwen3-8B	62.6	99.8	99.7	100	98.6	100
Qwen3.5-2B	82.2	88.4	63.1	99.6	95.9	100

Appendix

Table 11: GSM8K: Parse sucess rate analysis across JSON, InWriting, and NL-to-Format decoding strategies, comparing unconstrained with Format Constraints (FC).

Model	JSON Unconstr.	+FC	InWriting Unconstr.	+FC	NL-to-Format Unconstr.	+FC
Llama3-8B	0	100	99.8	100	99.6	100
Gemma2-9B	0	100	100	100	100	100
Qwen3-1.7B	0.7	96.2	97.4	100	99.6	100
Qwen3-4B	100	100	99.7	100	100	100
Qwen3-8B	100	100	99.5	100	99.6	100
Qwen3.5-2B	99.6	99.6	67.9	99.6	99.6	100

Appendix

Table 12: LastLetter-Extended: Parse sucess rate analysis across JSON, InWriting, and NL-to-Format decoding strategies, comparing unconstrained with Format Constraints (FC). X indicates that the corresponding experiment is unavailable.

Tasks	Model	JSON Unconstr.	+FC +VC	InWriting Unconstr.	+FC +VC	NL-to-Format Unconstr.	+FC +VC
GSM8K	Qwen3-1.7B	221.1	129.2	189.5	201.1	190.4	190.2
	Qwen3-4B	135.8	135.0	164.6	172.8	161.4	161.3
	Qwen3-8B	223.0	159.0	176.4	188.2	174.4	174.3
	Qwen3.5-2B	242.2	242.8	278.4	286.9	304.2	302.2
	Qwen3.5-4B	164.2	163.5	272.8	280.6	269.8	269.7
	Qwen3.5-9B	184.7	183.7	277.5	285.2	259.5	259.4

Appendix

Table 13: Comparison of generated output token counts for JSON, InWriting, and NL-to-Format decoding strategies, under unconstrained and combined Format and Validity Constraints (FC+VC).

Model	+FC+VC	+FC+VC +RC
Qwen3-1.7B	75.8 $\pm$ 0.8	77.7 $\pm$ 0.9
Qwen3-4B	89.7 $\pm$ 0.4	89.6 $\pm$ 0.3
Qwen3-8B	91.4 $\pm$ 0.5	91.9 $\pm$ 0.6
Qwen3.5-2B	74.9 $\pm$ 2.2	77.4 $\pm$ 0.5
Qwen3.5-4B	91.8 $\pm$ 0.4	91.5 $\pm$ 0.5
Qwen3.5-9B	93.9 $\pm$ 0.3	93.9 $\pm$ 0.3

Appendix

Table 14: GSM8K performance under the InWriting decoding strategy: comparison of Format and Validity Constraints (FC+VC) with FC+VC combined with Reasoning Constraints (FC+VC+RC). Bold values indicate the highest accuracy in each row.

Model	+FC+VC	+FC+VC +RC
Qwen3-4B	14.8 $\pm$ 4.5	16.7 $\pm$ 2.7
Qwen3-8B	12.2 $\pm$ 2.2	13.3 $\pm$ 3.1
Qwen3.5-4B	62.2 $\pm$ 3.1	55.6 $\pm$ 5.7
Qwen3.5-27B	81.9 $\pm$ 3.9	84.3 $\pm$ 5.6

Appendix

Table 15: AIME2026 performance under the InWriting decoding strategy: comparison of Format and Validity Constraints (FC+VC) with FC+VC combined with Reasoning Constraints (FC+VC+RC). Bold values indicate the highest accuracy in each row.

Constrained Decoding

Constrained decoding steers language model outputs to satisfy formal or task-specific rules while balancing validity, quality, and faithful probabilities.

See all 66 papers

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
Feb 8, 2026Avinash Reddy, Thayne T. Walker, James S. Ide +1Constrained Generative Modeling Constrained Decoding
Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US
Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
Sep 20, 2026Akash ChavanConstrained Decoding LLM Evaluation
Grammar-Aligned Decoding
May 31, 2024Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick +2Constrained Generative Modeling Constrained Decoding
University of Wisconsin-Madison · University of California San Diego

Feb 8, 2026J/K move · Enter open · S save

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Avinash Reddy, Thayne T. Walker, James S. Ide, Amrit Singh Bedi

Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US

Constrained Generative Modeling Constrained Decoding LLM Inference Structured Output Generation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose Draft-Conditioned Constrained Decoding (DCCD), a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints, with an optional best-of-

K

draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2% to 39.0% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Abstract

Constrained Decoding

Explore similar work

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap

Grammar-Aligned Decoding