The Format Tax

Ivan Yee Lee; Loris D'Antoni; Taylor Berg-Kirkpatrick

cs.CLApr 4, 2026

The Format Tax

Authors: Ivan Yee Lee, Loris D'Antoni, Taylor Berg-Kirkpatrick

Organizations: UC San Diego

Abstract

Asking a large language model to respond in JSON should be a formatting choice, not a capability tax. Yet we find that structured output requirements -- JSON, XML, LaTeX, Markdown -- substantially degrade reasoning and writing performance across open-weight models. The research response has focused on constrained decoding, but sampling bias accounts for only a fraction of the degradation. The dominant cost enters at the prompt: format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. This diagnosis points to a simple principle: decouple reasoning from formatting. Whether by generating freeform first and reformatting in a second pass, or by enabling extended thinking within a single generation, separating the two concerns substantially recovers lost accuracy. Across six open-weight models, four API models, four formats, and tasks spanning math, science, logic, and writing, decoupling recovers most lost accuracy. Notably, most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close. Code is available at https://github.com/ivnle/the-format-tax.

Figures & tables

Freeform
··T·	$x\sim p(x\mid\texttt{\textperiodcentered\textperiodcentered T})$
1-Turn		1-Turn + Think
GET·	$x\sim p(x\mid\texttt{GET})$	GET·	$r\sim p(r\mid\texttt{GET})$ ; $x\sim p(x\mid r,\texttt{GET})$
G·T·	$x\sim p(x\mid\texttt{G\textperiodcentered T})$	G·T·	$r\sim p(r\mid\texttt{G\textperiodcentered T})$ ; $x\sim p(x\mid r,\texttt{G\textperiodcentered T})$
·ET·	$x\sim p(x\mid\texttt{\textperiodcentered ET})$	·ET·	$r\sim p(r\mid\texttt{\textperiodcentered ET})$ ; $x\sim p(x\mid r,\texttt{\textperiodcentered ET})$
GETC	$x\sim p_{\text{GCD}}(x\mid\texttt{GET})$	GETC	$r\sim p(r\mid\texttt{GET})$ ; $x\sim p_{\text{GCD}}(x\mid r,\texttt{GET})$

Table 1: Experimental configurations organized by generation strategy and prompt variation. GETC notation: G rammar, E xamples, T ask, C onstrained decoding. G, E, and T vary prompt contents; C activates grammar enforcement during decoding. Dots indicate absence.

p

samples without constraints;

p_{\text{GCD}}

masks tokens that violate the grammar at each decoding step. Thinking variants first sample reasoning

r

, then formatted output

x

. For 2-Turn, Turn 1 produces answer

a

(task only) while Turn 2 formats.

	No Thinking					With Thinking
		1-Turn		2-Turn			1-Turn		2-Turn
Model	Free	LTX	Prose	LTX	Prose	Free	LTX	Prose	LTX	Prose
nemotron3-nano	62.3	-15.7	-16.1	-7.0	-6.9	66.4	-18.6	-17.4	-7.1	-6.4
qwen3-32b	56.7	-6.8	-7.8	-0.5	-0.9	56.6	-8.4	-9.4	-2.6	-2.6
qwen3-8b	51.5	-13.7	-10.1	-6.1	-2.5	54.2	-11.9	-9.0	-4.1	-3.4
smollm3-3b	32.4	-6.8	-5.6	-4.2	-4.2	35.6	-6.5	-6.4	-12.9	-12.9

Table 2: LaTeX formatting degrades writing quality—but 2-Turn recovers most of the loss. Freeform shows absolute quality (%); other columns show change vs. freeform. 1-Turn ( GET· ): model generates LaTeX directly with grammar and examples in the prompt, no constrained decoding. 2-Turn ( ··T·

\to

GET· ): model generates freeform, then reformats to LaTeX. Column labels: LTX = judge evaluates document as-is; Prose = markup stripped, evaluating content only.

	GPQA				Math				Zebra
Model	JSON	XML	MD	LTX	JSON	XML	MD	LTX	JSON	XML	MD	LTX	Avg
qwen3-32b	52.0				84.4				64.8				67.1
+ GCD	-7.4	-7.9	-6.9	-5.9	-12.7	-10.9	-10.0	-2.8	-7.1	-7.3	-10.1	-6.9	-8.0
no GCD	-7.1	-7.4	-3.7	-5.1	-6.8	-6.5	-4.2	-4.0	-6.6	-7.3	-5.8	-5.3	-5.8
qwen3-8b	40.6				84.0				63.1				62.5
+ GCD	-10.3	-6.4	-3.0	-1.5	-15.6	-9.3	-8.2	-5.3	-10.4	-7.9	-8.8	-17.5	-8.7

Table 3: Format tax across open-weight models: accuracy change from freeform baseline (%). + GCD averages over constrained configurations ( GETC , G·TC , ·ETC ); no GCD averages over unconstrained configurations ( GET· , G·T· , ·ET· ). All results: 1-Turn, thinking off.

	GPQA				Math				Zebra
Model	JSON	XML	MD	LTX	JSON	XML	MD	LTX	JSON	XML	MD	LTX	Avg
gpt-5-nano	47.0				88.2				50.4				61.9
+ GCD	-1.5	-10.6	-11.1	-11.1	+0.7	-1.6	+3.3	-5.8	-9.2	-16.2	-13.8	-19.2	-8.0
no GCD	-4.9	+0.5	-2.5	-4.2	-1.5	-1.9	+0.3	-3.7	-11.3	-13.9	-14.7	-4.2	-5.2
gpt-5.4-nano	58.1				88.9				65.4				70.8
+ GCD	-2.0	-16.2	-12.1	-12.1	+2.2	-59.4	-8.9	-28.7	+4.4	-24.6	-14.8	-18.2	-15.9

Table 4: Format tax across closed-weight models: accuracy change from freeform baseline (%). + GCD averages over constrained configurations ( GETC , G·TC , ·ETC ); no GCD averages over unconstrained configurations ( GET· , G·T· , ·ET· ). For non-OpenAI API models, constrained decoding is available only for JSON. All results: 1-Turn, thinking off.

Figure 1: Source attribution of the format tax. (a) Each point is a (model, task, format) combination; the x-axis shows accuracy drop from the format-requesting prompt alone ( GET· ), while the y-axis adds grammar-constrained decoding ( GETC ). Points near the diagonal indicate GCD adds little beyond the prompt effect. (b) Parallel McNemar tests (

p<0.05

) on 72 cells from open-weight models classify each by whether degradation is already present under the format-requesting prompt alone, added by GCD, or both.

No Thinking

With Thinking

GPQA

Math

Zebra

GPQA

Math

Zebra

Model

qwen3-32b

-6

-7

-1

-0

+10

qwen3-8b

-5

-8

-1

-11

+10

olmo3.1-32b

-2

-5

+10

-2

-9

-1

+10

olmo3-7b

-5

-7

-4

+15

-0

-1

+14

Table 5: Decoupling reverses the format tax. Each cell shows the average accuracy change (%) from the matched freeform baseline, averaged across four formats and all prompt configurations. 1T = 1-Turn (single pass), 2T = 2-Turn (freeform then reformat). Non-thinking columns use freeform as baseline; thinking columns use freeform + thinking.

Figure 2: Per-question flip rate (freeform-correct $\to$ structured-incorrect) by token delta decile. The U-shaped pattern is nearly identical with and without GCD.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Per-question thinking tokens under freeform vs. structured output. Each point is one question; pairs where either side hits the 8,192-token ceiling are excluded. The strong correlation ( $r=0.72$ , $n=28{,}000$ +) indicates that thinking effort is driven by question difficulty, not format constraints.

	Structured	Freeform	Net
	(%)	(%)	(pp)
qwen3-32b	0.7	0.7	+0.0
qwen3-8b	0.7	0.8	-0.0
olmo3.1-32b	3.0	0.8	+2.2
olmo3-7b	2.0	0.7	+1.3
smollm3-3b	0.6	0.6	-0.0

Appendix

Table 6: Format leakage into thinking blocks. Fraction of sentences in <think> blocks that mention format-related keywords (JSON, XML, format, structure, etc.). Freeform: same metric as a false-positive control (these keywords appear incidentally in math/logic reasoning). Net: structured minus freeform. Despite this, thinking nearly eliminates the format tax (Table 5 ).

	No GCD		+GCD
	Valid	Acc	Valid	Acc
JSON	65.0	56.4	96.2	54.3
XML	60.7	57.4	97.0	54.5
Markdown	46.5	58.6	94.3	57.1
LaTeX	50.5	56.8	81.4	57.0
qwen3-32b	76.4	61.3	98.9	59.1

Appendix

Table 7: Format validity and lenient accuracy (correct answer regardless of format compliance) in % by decoding method. +GCD dramatically improves format compliance (55.7%

\to

92.2%) but does not improve reasoning accuracy, which remains 5–7 pp below the freeform baseline (61.5%) regardless of compliance. All values averaged across 1-turn, thinking-off experiments on three reasoning benchmarks.

Figure 4: Per-model flip rate by token delta decile. The U-shaped pattern from Figure 2 holds across all six models: questions where structured output most compresses or expands token count relative to freeform show the highest flip rates.

Format	$N$ judged	Both Correct	Both Incorrect	Det. ✗ / Judge ✓	Det. ✓ / Judge ✗	Agreement
Freeform	4970	3986 (80.2%)	814 (16.4%)	164 (3.3%)	6 (0.1%)	4800 (96.6%)
JSON	4981	3700 (74.3%)	1056 (21.2%)	220 (4.4%)	5 (0.1%)	4756 (95.5%)
XML	4973	3713 (74.7%)	1012 (20.3%)	242 (4.9%)	6 (0.1%)	4725 (95.0%)
Markdown	4965	3565 (71.8%)	872 (17.6%)	522 (10.5%)	6 (0.1%)	4437 (89.4%)
LaTeX	4976	3672 (73.8%)	1012 (20.3%)	288 (5.8%)	4 (0.1%)	4684 (94.1%)

Appendix

Table 8: Agreement between the deterministic MATH-500 scorer and the LLM judge on the canonical ten-model, no-GCD slice. Counts and percentages are computed over judged rows only. Divergence is concentrated in deterministic failures that the judge recovers, while reverse disagreements remain rare.

Format	Model Response	Gold	Det.	Judge
JSON	10%	$10$	✗	✓
Markdown	$x=3,\;x=5,\;x=7$	$3,5,7$	✗	✓
JSON	quotient $=x^{5}-x^{4}+x^{3}-x^{2}+x-1$ ; remainder $={-}2$	$x^{5}-x^{4}+\cdots+x-1$	✗	✓
LaTeX	$11\sqrt{2}$	$11\sqrt{2}$	✗	✓
XML	$864$	$864\ \text{in}^{2}$	✗	✓
Markdown	Final answer: $1$ (also mentions $-2$ earlier)	$-2$	✓	✗

Appendix

Table 9: Reviewed real MATH-500 rows illustrating the appendix audit. The disagreements are concentrated in extraction, parse, and normalization failures of the deterministic scorer; the reverse-disagreement example shows the judge rejecting a response whose final answer is actually wrong.

		1-Turn			1-Turn + Think			2-Turn			2-Turn + Think
Config	Format	GPQA	Math	Zebra	GPQA	Math	Zebra	GPQA	Math	Zebra	GPQA	Math	Zebra
qwen3-32b		52.0	84.4	64.8	52.2	93.7	77.0	52.0	84.4	64.8	52.2	93.7	77.0
G·T·	JSON	-4.0	-6.4	-3.6	6.9	1.3	2.2	1.0	2.2	1.0	7.9	1.9	10.2
	XML	-5.6	-6.4	-6.8	0.3	0.7	1.4	0.5	1.8	1.8	9.9	1.5	7.8
	Markdown	-1.5	-3.7	-8.0	-1.7	0.3	1.0	1.0	2.3	1.0	11.4	2.7	9.6
	LaTeX	-3.5	-4.7	-4.4	-6.2	-0.1	-0.4	1.0	1.9	0.2	7.4	1.9	10.2

Appendix

Table 10: Full experimental results for all models, configurations, and thinking modes. Shaded rows show freeform baseline accuracy (%); unshaded rows show change from baseline (negative = degradation). Accuracy is lenient (answer correct; format validity not required). Config notation: G = grammar in prompt, E = few-shot examples, T = task description, C = constrained decoding.

Parameter	Value	Notes
Generation Parameters
Temperature	1.0
Top- $p$	1.0	No nucleus filtering; applied to vLLM only (not supported by OpenAI/Anthropic APIs)
Max output tokens	8,192	Per generation turn
Samples per question ( $n$ )	1	Single sample per question
Random seed	42	Fixed for vLLM experiments; API providers use server-side randomness

Appendix

Table 11: Default hyperparameters used across all experiments unless otherwise noted.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 429 papers

Structure Tax: How Structured Output affects LLMs Performance
Oct 8, 2026Vineet Kumar, Kanishka, Bhuvanesh MandoraStructured Output Generation Structured Reasoning in Language Models
PayPal Artificial Intelligence PayPal, Bengaluru India
Capacity, Not Format: Rethinking Structured Reasoning Failures
Jun 8, 2026Hengxin FanLLM Evaluation Structured Prompting
Tianjin Normal University
Structured Output Collapses Answer Diversity Across 44 Language Models
Jul 20, 2026Tapan ParikhLLM Evaluation Structured Prompting
Cornell Tech

Oct 8, 2026J/K move · Enter open · S save

Structure Tax: How Structured Output affects LLMs Performance

Vineet Kumar, Kanishka, Bhuvanesh Mandora

PayPal Artificial Intelligence PayPal, Bengaluru India

Structured Output Generation Structured Reasoning in Language Models LLM Reliability

Open paper AskPDF arXiv

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent

structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from

whether to structure' to `how to structure' for optimal reasoning preservation.

Abstract

LLM Prompting

Explore similar work

Structure Tax: How Structured Output affects LLMs Performance

Capacity, Not Format: Rethinking Structured Reasoning Failures

Structured Output Collapses Answer Diversity Across 44 Language Models