Structure Tax: How Structured Output affects LLMs Performance

Vineet Kumar; Kanishka; Bhuvanesh Mandora

cs.AIOct 8, 2026

Structure Tax: How Structured Output affects LLMs Performance

Authors: Vineet Kumar, Kanishka, Bhuvanesh Mandora

Organizations: PayPal Artificial Intelligence PayPal, Bengaluru India

Abstract

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from whether to structure' to `how to structure' for optimal reasoning preservation.

Figures & tables

Figure 1: Three response formats elicited from GPT-4o on the same MMLU Anatomy question. (Left) Free-form : natural-language reasoning arrives at the correct answer D . (Middle) Answer-first JSON : the model commits to "answer": "A" before generating a post-hoc rationale that cites only one of three valid targets, yielding an incorrect answer. (Right) Reasoning-first XML : rationale unrolled inside <reasoning> precedes the commitment, recovering the correct <answer>D</answer> .

			Answer-first (A,R)		Reasoning-first (R,A)
Dataset	Model	Free Form	JSON	XML	JSON	XML
GSM8K	GPT-4o	88.78 $\pm$ 1.02	55.34 $\pm$ 1.58	56.86 $\pm$ 2.71	95.60 $\pm$ 1.01	95.60 $\pm$ 0.17
	Gemini-Flash-2.0	79.38 $\pm$ 0.28	63.08 $\pm$ 0.53	54.90 $\pm$ 3.64	95.53 $\pm$ 0.59	95.45 $\pm$ 0.60
	Claude-Sonnet-4.5	86.50 $\pm$ 1.13	96.13 $\pm$ 0.88	62.35 $\pm$ 2.29	87.79 $\pm$ 0.98	97.65 $\pm$ 0.39
	Llama-3.1-8B	79.98 $\pm$ 1.10	31.01 $\pm$ 1.58	10.98 $\pm$ 2.66	80.21 $\pm$ 0.45	83.53 $\pm$ 2.58
	Gemma3-4B	78.82 $\pm$ 1.46	27.45 $\pm$ 4.70	10.98 $\pm$ 2.13	83.53 $\pm$ 1.96	87.06 $\pm$ 2.34

Table 1: Impact of output structure constraints on accuracy across benchmark tasks. We compare Answer-first (answer before reasoning) and Reasoning-first (reasoning before answer) orderings under JSON and XML formats, against an unconstrained Free Form baseline. Values are mean accuracy over 5 independent runs (standard deviation in gray). Best format per model–dataset pair is bolded and highlighted.

Figure 2: ECE increase under freeform relative to XML ( $\Delta\text{ECE}=\text{ECE}_{\text{freeform}}-\text{ECE}_{\text{XML}}$ ). Positive values (red) indicate freeform is worse calibrated; negative (green) indicates the reverse. LogicBench shows the largest format sensitivity; GSM8K is format-invariant.

Dataset	Model	ECE $\downarrow$	OvConf $\downarrow$	Corr ( $\rho$ ) $\uparrow$
GSM8K	GPT-4o	0.050 $\pm$ 0.005	0.051 $\pm$ 0.005	0.169 $\pm$ 0.028
	Gemini-Flash-2.0	0.042 $\pm$ 0.002	0.043 $\pm$ 0.002	0.134 $\pm$ 0.084
	Claude-Sonnet-4.5	0.020 $\pm$ 0.004	0.022 $\pm$ 0.004	0.264 $\pm$ 0.063
	Llama-3.1-8B	0.136 $\pm$ 0.008	0.138 $\pm$ 0.009	0.189 $\pm$ 0.053
	Gemma3-4B	0.109 $\pm$ 0.028	0.111 $\pm$ 0.029	0.350 $\pm$ 0.146
LogicBench	GPT-4o	0.097 $\pm$ 0.018	0.197 $\pm$ 0.018	0.253 $\pm$ 0.078

Table 2: Calibration under XML-structured output. ECE and OvConf are lower-is-better; Corr (

\rho

) is higher-is-better. Values are mean over 5 runs (std. in gray). Best model per metric–dataset pair is bolded .

Panel of Figure 3 — Figure 3: Internal representations for Llama-3.1-8B on MMLU and LogicBench. Left : Cross-group CKA under XML (R,A) (solid), XML (A,R) (dash-dot), and Freeform (dashed) across four benchmarks. Higher CKA = weaker separation between correct and incorrect representations; R,A yields the lowest mid-layer CKA on reasoning tasks while Spider reverses the pattern. Right : Linear probe balanced accuracy decoding the predicted answer label from hidden states at each layer. Solid = (R,A); dashed = (A,R); dotted = chance; gap panel shows the (R,A) $-$ (A,R) advantage with peak layer annotated. Gemma3-4B replicates both patterns at reduced magnitude (Appendix D.4 ).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Empty Extraction Rate
MMLU	29.8%
LogicBench	45.2%

Appendix

Table 3: Free-form empty-extraction rates caused by the Gemma3-4B parser bug.

Dataset	Split	Examples	Task Type	Answer Format
GSM8K	Test	1,319	Math reasoning	Numeric
LogicBench	Eval	500	Logical inference	MCQ
MMLU	Validation	1,531	Knowledge QA	MCQ
Spider	Validation	1,030	Text-to-SQL	SQL query

Appendix

Table 4: Dataset statistics and properties. All splits are held-out evaluation sets; no training data is used. Spider is evaluated via execution accuracy; all other datasets use exact-match accuracy.

Dataset	Model	Abs.	Rel.	$\Delta^{\dagger}$	Abs.	Rel.	Entropy $\downarrow$	Brier $\downarrow$
		ECE $\downarrow$			AvgConf		Entropy $\downarrow$	Brier $\downarrow$
LogicBench	GPT-4o	0.072 $\pm$ .010	0.084 $\pm$ .017	$-0.012$	0.883 $\pm$ .006	0.741 $\pm$ .005	1.141 $\pm$ .015	0.297 $\pm$ .023
	Gemini-Flash-2.0	0.094 $\pm$ .012	0.052 $\pm$ .005	$+0.042$	0.941 $\pm$ .004	0.798 $\pm$ .004	0.930 $\pm$ .027	0.244 $\pm$ .014
	Claude-Sonnet-4.5	0.045 $\pm$ .007	0.066 $\pm$ .010	$-0.021$	0.879 $\pm$ .007	0.834 $\pm$ .003	0.759 $\pm$ .014	0.200 $\pm$ .016
	Llama-3.1-8B	0.166 $\pm$ .015	0.161 $\pm$ .015	$+0.005$	0.914 $\pm$ .006	0.878 $\pm$ .007	0.496 $\pm$ .019	0.452 $\pm$ .036
	Gemma3-4B	0.290 $\pm$ .012	0.152 $\pm$ .017	$\mathbf{+0.138}$	0.938 $\pm$ .002	0.775 $\pm$ .002	1.046 $\pm$ .010	0.530 $\pm$ .019

Appendix

Table 5: Absolute vs. relative confidence elicitation on MCQ datasets.

\dagger

\Delta=\text{ECE}_{\text{Abs}}-\text{ECE}_{\text{Rel}}

; positive = relative prompting improves calibration. Entropy and Brier defined only under relative prompting. Highlighted

\Delta

values mark the largest per-column improvements. Values are mean over 5 runs (std. in gray).

		Correlation & Divergence			ECE $\downarrow$		Brier Score $\downarrow$
Dataset	Model	Spearman $\rho$	Pearson $r$	Mean KL	Logit	Verbal	Logit	Verbal
MMLU	GPT-4o	$0.439$	$0.128$	$0.500$	$0.117$	$0.116$	$0.276$	${0.247}$
	Llama-3.1-8B	$-0.049^{\dagger}$	$-0.040^{\dagger}$	$3.617$	${0.156}$	$0.191$	${0.496}$	$0.582$
	Gemma3-4B	$0.024^{\dagger}$	$0.002^{\dagger}$	$1.676$	$0.402$	$0.212$	$0.813$	$0.602$
LogicBench	GPT-4o	$0.673$	$0.312$	$0.423$	$0.122$	${0.084}$	${0.274}$	$0.305$
	Llama-3.1-8B	$0.282$	$0.230$	$2.304$	${0.141}$	$0.162$	${0.379}$	$0.453$

Appendix

Table 6: Logit vs. verbal confidence comparison. Spearman

\rho

and Pearson

r

measure scalar agreement on the chosen-option score; Mean KL measures full distributional divergence. ECE and Brier assess calibration against correctness. †

p>0.05

Figure 4: Accuracy vs. confidence scatter for all 5 models across 4 datasets under freeform (squares) and XML (circles) prompting. All models express high confidence (

>85\%

) regardless of format, while accuracy varies substantially. The gap between confidence and the diagonal is consistently larger under freeform, most dramatically for Gemma3-4B on MMLU and LogicBench (

<

26% accuracy,

\sim

100% confidence) and Llama-3.1-8B on LogicBench (23% accuracy, 93% confidence). XML structured output reduces overconfidence across all models, bringing points closer to the calibration diagonal.

Figure 5: Reliability diagrams comparing freeform and XML calibration across 20 model–dataset combinations (5 models × 4 datasets, each shown in freeform and XML format). XML bars align closer to the diagonal in most configurations, with the largest improvements on LogicBench where freeform produces severe overconfidence (ECE up to 0.703 for Llama-3.1-8B and 0.645 for Gemma3-4B ). Gemma3-4B shows the highest overall freeform ECE, particularly on MMLU (0.721). GSM8K remains comparatively well-calibrated under both formats across all models.

Panel of Figure 6 — Figure 6: Llama-3.1-8B within-group CKA heatmaps for (a) GSM8K and (b) MMLU. Each panel shows XML (R,A) (top row) and Freeform (bottom row) across three views: correct answers only, incorrect answers only, and their difference (correct $-$ incorrect). See § D.2 for construction and reading guide.

Panel of Figure 7 — Figure 7: Llama-3.1-8B within-group CKA heatmaps for (a) LogicBench and (b) Spider. Same layout as Figure 6 ; see § D.2 . Spider shows weak differentiation between outcome groups under both formats, consistent with the high-structure task pattern.

Panel of Figure 8 — Figure 8: Gemma3-4B within-group CKA heatmaps for (a) GSM8K and (b) MMLU. Each panel shows XML (R,A) (top row) and Freeform (bottom row) across correct answers, incorrect answers, and their difference (correct $-$ incorrect). See § D.2 for construction and reading guide.

Panel of Figure 9 — Figure 9: Gemma3-4B within-group CKA heatmaps for (a) LogicBench and (b) Spider. Same layout as Figure 8 ; see § D.2 . Spider shows weak differentiation between outcome groups under both formats, consistent with the high-structure task pattern.

Ordering	Probe Token	Offset	Preceding Context
(R,A)	> in <answer>	$-1$	Full reasoning block
(A,R)	> in <answer>	$-1$	No reasoning; answer is the first output field

Appendix

Table 7: Probe location relative to the answer token under different field orderings.

Panel of Figure 10 — Figure 10: Internal representations for Gemma3-4B , same layout as Figure 3 . Left : Cross-group CKA preserves the inverse-structure gradient of Llama-3.1-8B MMLU and LogicBench show lower mid-layer CKA under XML (R,A) than freeform; Spider reverses the pattern at reduced overall magnitude. Right : Linear probe balanced accuracy on MMLU and LogicBench. The (R,A) advantage is directionally consistent with Llama but smaller in magnitude (peak gap $+0.27$ on MMLU at L17; $+0.26$ on LogicBench at L13), in line with Gemma’s smaller behavioural gap under R,A in Table 1 .

Dataset	XML (R,A)	XML (A,R)	JSON (R,A)	JSON (A,R)
GSM8K	126	127	108	123
MMLU	121	99	121	33
LogicBench	114	92	113	96
Spider	54	58	62	64

Appendix

Table 8: Median reasoning-field word count per format–ordering, aggregated over 5 models. Bold = shortest per dataset. JSON (A,R) collapses to 33 words on MMLU vs. 121 under XML (R,A); GSM8K is flat across all variants.

By dataset			By format–ordering
Dataset	$r$	$n$	Format	$r$	$n$
GSM8K	$-0.11$	20	XML (R,A)	$+0.42$	20
MMLU	$+0.11$	20	XML (A,R)	$+0.28$	20
LogicBench	$+0.06$	20	JSON (R,A)	$+0.25$	20
Spider	$+0.16$	20	JSON (A,R)	$-0.73$ ∗∗	20
Overall	$+0.04$	80

Appendix

Table 9: Pearson

r

between median reasoning length and accuracy across 80 cells (5 models

\times

4 datasets

\times

4 format-orderings).

{}^{**}p{<}0.01

. Conditioning on dataset yields no signal; conditioning on format-ordering reveals JSON (A,R) as the sole significant predictor and it is negative.

Model	Format	Base	V1	V2	V3
Llama-3.1-8B	Free-form	77.5	81.0	78.5	80.0
	JSON (A,R)	32.0	35.0	34.5	36.0
	JSON (R,A)	81.5	86.0	74.5	77.0
	XML (A,R)	10.5	11.5	11.0	11.5
	XML (R,A)	84.0	87.5	82.0	82.0
GPT-4o	Free-form	90.0	86.0	90.0	86.5

Appendix

Table 10: GSM8K accuracy (%) dataset under three prompt-perturbation variants. Base = original prompt used in the main results. Within each (model, format-family) pair, the (R,A) row consistently exceeds the (A,R) row across all four conditions.

Figure 11: GSM8K accuracy under prompt perturbations. Each group of bars is one (model, format) cell; within-group order is baseline, V1, V2, V3. The (R,A)

-

(A,R) gap (annotated

\Delta

over each XML / JSON family) is preserved across all perturbations: the sign of the field-ordering effect is invariant to surface prompt changes , even though absolute accuracy shifts by a few points.

Explore similar work

Jun 8, 2026cs.AI

Capacity, Not Format: Rethinking Structured Reasoning Failures

Prior work treats structured output as a reasoning tax, but this framing is incomplete: the cost of formatting depends strongly on a model's spare capacity. Using information-matched prose controls and a four-level schema complexity gradient, we separate format-specific effects from prompt-length confounds across 4 models and 5 benchmarks with 0% parse failures on successfully generated responses. We find that structured formats are capacity-dependent. Models with sufficient headroom absorb JSON constraints without degradation (Sonnet:

88.7\pm4.0

% JSON vs.

89.3\pm1.7

% CoT on MATH-Hard). In contrast, formats severely degrade models operating near their limits through two distinct mechanisms. First, under standard token budgets, Haiku drops 36.2pp (

p < 0.0001

) largely due to truncation. Second, even with extended budgets eliminating truncation, GPT-4o-mini drops 28.0pp (

p < 0.001

), revealing pure capacity competition independent of token exhaustion. This format penalty scales with schema complexity (McNemar

p < 0.0001

) and cannot be explained by prompt length alone. Furthermore, these results qualify claims of frontier model immunity: on AIME competition math, Opus 4.7 drops from 96.2% to 91.0% under JSON (

-5.3

pp; the displayed percentages are independently rounded, exact difference is

7/133 = 5.26

\approx 5.3

pp). A delayed-structure ablation -- reasoning freely before formatting -- recovers most of the lost accuracy (3-run mean: 80--87%), supporting the capacity competition mechanism. The practical implication is not to avoid structured output, but to match it to capacity: when a model is near its limits, think first, format later.

Hengxin Fan

Tianjin Normal University

LLM Evaluation Structured Prompting Structured Reasoning Structured Output Generation LLM Reasoning

Ask PDF

May 20, 2026cs.LG

The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models

Production LLM systems increasingly require machine-readable outputs: JSON objects, typed traces, regex-constrained fields, and tool-call schemas. This paper targets on-device and low-cost small language model (SLM) deployments, where sub-3B models are attractive for privacy, latency, and commodity hardware but have limited capacity to satisfy schemas while solving tasks. The usual engineering assumption is that hard output constraints improve reliability without changing the underlying answer. We show that this assumption is unsafe for small models. We introduce \emph{constraint tax}, a measurement protocol for isolating the answer and executable-accuracy loss caused by structured-output constraints at fixed model, fixed task distribution, and fixed problem instances. Across 15,000 commodity-GPU generations with Qwen2.5-0.5B, Qwen2.5-1.5B, and SmolLM2-1.7B, hard answer-only schema decoding raises schema validity from 61.5% to 100.0%, but lowers answer accuracy from 19.7% to 11.0% and increases wrong-valid-schema outputs from 49.5% to 88.9%. The strongest industry analogue is a deterministic calendar tool-call task: Qwen2.5-1.5B achieves 91.5% executable accuracy with prompt-only JSON but only 48.0% under the same hard tool-call schema, while both modes are 100.0% schema-valid. The error is semantic, not structural. We also show that the 3B boundary still pays a direct-schema tax and that delayed packaging supports a constructive design pattern: reason free, constrain late. The practical conclusion is direct: production systems should report schema validity, answer accuracy, executable accuracy, and wrong-valid-schema rate separately.

Jaideep Ray

Constrained Decoding Small Language Models Automated Evaluation Structured Output Generation

Ask PDF

Jul 20, 2026cs.CL

Structured Output Collapses Answer Diversity Across 44 Language Models

When a language model must choose one answer from a large space of equally valid options, a format clause -- "Reply with JSON only" -- changes which answer it chooses. We re-run the One-Word Census (arXiv:2607.12796): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in JSON -- no schema enforcement, no constrained decoding, only the request. Convergence deepens sharply: on the unconstrained "Pick a word" prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36; mean answer-choice surprisal drops from 1.80 to 1.58 bits. The tax is progressive: six of 44 models move individually (BH-FDR q=.10), all toward the mode, led by the most distinctive models, while the conformist floor is immobile. It is a sharpener, not a re-indexer -- the plain-chat modal answer survives in 28 of 31 categories. Defaults are register-indexed: a within-run re-sample (n=20) finds JSON shifts 53% of a model's stable chat defaults, mostly back to the crowd, and installs defaults absent from chat (Claude Fable 5 answers "cerulean" for colour 0% of the time in chat, 100% in JSON). Full-battery controls reveal a register gradient: compression is significant and specific to the answer-delivery formats models are trained to speak (JSON -0.22 bits, p=.0002; XML -0.19, p=.002), absent for YAML and CSV, and reversed for an arbitrary bracket wrapper (+0.13, p=.009) -- weighing the mechanism toward tool-use post-training. Enforcing the schema at the decoder (response_format) compresses no further than the request (-0.03 bits): the collapse lives in the model's response to the register, not the decoder. Structured output is how software consumes language models, and that surface is served by a measurably more homogeneous model than the chat surface on which models are evaluated, compared, and chosen.

Tapan Parikh

Cornell Tech

LLM Evaluation Structured Prompting LLM Prompting Diverse Text Generation Structured Output Generation

Ask PDF