A small fine-tuned parser that emits a well-formed but wrong tree fails silently: nothing downstream can tell. We ask whether the output format, a free design choice, controls how often this happens. On the PIZZA ordering benchmark we fine-tune two sub-billion models (Qwen3.5-0.8B and LFM2.5-350M) with LoRA on six lossless encodings of the same target tree, from 44.3 down to 10.9 tokens per order, including two matched-length pairs that vary surface form at fixed length. Across 300 runs (5 training pool sizes, 5 seeds) with a hierarchical bootstrap and Holm correction fixed before the results were read, shortening alone never helps, but writing the target in ordinary words does. The word encoding is the most accurate format on both models (87.5% vs. 85.8% exact match for the native s-expression on Qwen) while emitting 4.1x fewer tokens, and it cuts the silent error rate by 2.3x on Qwen and 1.8x on LFM2, separating at every pool size. Error analysis locates much of the gain in negation. All runs also learn to refuse off-menu input (held out: 95.9-99.0% handled) and to apply in-utterance self-corrections. Strict refusal has a measured price: 2.1-10.1% of real, mostly polite, orders are refused.
Figures & tables
Format
Tok.
Target
exr
38
(ORDER (PIZZAORDER (NUMBER 1 ) (COM...
json
33
{"orders":[{"type":"p","attrs":[{"n...
exr-short
25
(O (P (N 1) (C (Q EXTRA) (T CHEESE)...
tagged
13
pizza n=1 extra=cheese topp=pepperoni
dsl
8
p 1 +cheese pepperoni
words
7
pizza 1 extra cheese pepperoni
Table 1: One test order, “i want a pie with pepperoni and extra cheese” , in the six encodings (long targets truncated). Tokens are Qwen’s.
Figure 1: The order of Table 1 under Qwen’s tokenizer. For each format the upper bar has one box per token and the lower bar draws the same tokens as wide as their characters (both bars to a common scale across formats). Bottom: the matched pair with its tokens; words is longer as text and shorter in tokens. Boundaries come from the tokenizer’s own merge table.
Format
Tok.
A (EM) ↑
S (silent) ↓
Dparse
Drefuse
Inv./utt.
ms
Qwen3.5-0.8B
exr
44.3
85.8 ± 1.6
10.3 ± 0.8
0.6 ± 0.8
3.3 ± 1.1
0.100
1171
json
44.1
81.6 ± 1.3
14.9 ± 1.3
0.3 ± 0.3
3.2 ± 1.0
0.126
1220
exr-short
32.3
75.2 ± 3.6
17.4 ± 4.1
3.3 ± 2.0
4.2 ± 2.2
0.140
903
tagged
22.2
85.9 ± 2.9
11.6 ± 2.4
0.5 ± 0.3
2.1 ± 0.8
0.095
616
dsl
11.3
79.7 ± 1.9
8.3 ± 2.3
6.7 ± 3.2
5.4 ± 2.3
0.070
325
Table 2: Test results at the full pool (2,456,443), mean ± seed SD over 5 seeds, in percent of the 1,357 human-written test orders. A + S + Dparse + Drefuse = 100. Bold: best format in the column; underlined: not separable from the best (Holm, p≥0.05 ). Tok.: mean output tokens. Inv./utt.: invented entities per order. ms: median latency at batch size one on one GPU, mean over seeds.
Figure 2: Exact match against output tokens at the full pool, 95% bootstrap intervals. Dashed lines join the matched-length pairs ( exr / json , dsl / words ).
Difference (points)
1k
10k
100k
1M
2.5M
Qwen3.5-0.8B
words − dsl
+6.7 †
+4.8 †
+6.2 †
+5.8 †
+7.8 †
exr − json
+9.3 †
+3.8
+3.6
+5.5 †
+4.2 †
exr − exr-short
+6.2 †
+11.6 †
+10.3 †
+9.3 †
+10.6 †
words − exr
+4.0
+1.8
+1.6
+0.0
+1.8
S : exr − words
+11.4 †
+5.1 †
+4.4 †
+5.0 †
+5.8 †
Table 3: Paired differences in exact match (points; last row: silent rate, positive = words lower) at every pool size, both models. † : separates (Holm within the row). 95% intervals in Appendix Table 6 .
Figure 3: Where the test orders go at the full pool (axis starts at 60%). Labels: silent error rate S .
Figure 4: Silent errors by kind at the full pool, as a share of all test orders (bar height = S ). Negation: a NOT attribute added, lost or misplaced; line count: wrong number of pizzas or drinks.
Format
Neg. handled ↑
Real refused ↓
Corr. EM ↑
Corr. applied
Probes /40
Qwen3.5-0.8B
exr
99.0
3.3
74.5 ± 4.3
86.5
36.0
json
98.0
3.2
70.0 ± 2.8
85.7
37.8
exr-short
96.1
4.2
62.9 ± 6.9
82.8
35.4
tagged
95.9
2.1
73.0 ± 4.5
87.7
37.2
dsl
98.9
5.4
71.4 ± 4.8
80.4
37.8
Table 4: Guardrail at the full pool, percent, mean of 5 seeds. Neg. handled: held-out negatives refused (614 per run). Real refused: test orders refused ( Drefuse ). Corr.: 600 test orders with held-out correction cues; applied: the corrected attribute is right. Probes: hand-written corrections correct, of 40. Bold/underline as in Table 2 .
Figure 5: Exact match by training pool size, 95% bootstrap bands.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
1k
10k
100k
1M
2.5M
A
S
A
S
A
S
A
S
A
S
Qwen3.5-0.8B
exr
79.7 ± 4.9
17.0
86.0 ± 2.7
9.6
86.6 ± 2.0
9.2
86.6 ± 2.1
10.1
85.8 ± 1.6
10.3
json
70.4 ± 1.4
25.1
82.2 ± 1.3
14.6
83.0 ± 4.4
12.9
81.1 ± 4.6
11.5
81.6 ± 1.3
14.9
exr-short
73.5 ± 3.5
18.8
74.4 ± 4.9
15.0
76.2 ± 1.4
15.1
77.3 ± 2.9
13.4
75.2 ± 3.6
17.4
tagged
78.9 ± 3.7
16.8
84.6 ± 2.9
12.1
84.5 ± 2.5
10.8
84.1 ± 1.1
12.4
85.9 ± 2.9
11.6
Appendix
Table 5: Exact match A ( ± seed SD) and silent rate S at every pool size, percent.
Difference (points)
1k
10k
100k
1M
2.5M
Qwen3.5-0.8B
words − dsl
+6.7 [+3.4, +10.2] †
+4.8 [+2.3, +7.3] †
+6.2 [+3.9, +8.4] †
+5.8 [+3.4, +8.6] †
+7.8 [+6.1, +9.5] †
exr − json
+9.3 [+5.4, +12.8] †
+3.8 [+0.2, +6.7]
+3.6 [-1.1, +7.8]
+5.5 [+2.2, +9.2] †
+4.2 [+2.4, +5.8] †
exr − exr-short
+6.2 [+3.3, +9.2] †
+11.6 [+4.7, +16.6] †
+10.3 [+7.9, +12.7] †
+9.3 [+6.2, +12.5] †
+10.6 [+7.9, +13.4] †
words − exr
+4.0 [+0.5, +7.7]
+1.8 [-0.4, +4.4]
+1.6 [-0.8, +4.2]
+0.0 [-1.3, +1.3]
+1.8 [+0.1, +3.4]
S : exr − words
+11.4 [+8.9, +13.9] †
+5.1 [+3.3, +7.1] †
+4.4 [+2.8, +6.2] †
+5.0 [+3.1, +7.1] †
+5.8 [+4.3, +7.4] †
Appendix
Table 6: Table 3 with 95% hierarchical bootstrap intervals.
Format
Chars
Qwen3.5
LFM2.5
tok.
ch./tok.
tok.
ch./tok.
exr
123.0
45.0
2.7
58.0
2.1
json
150.7
45.6
3.3
64.7
2.3
exr-short
66.0
33.4
2.0
39.0
1.7
tagged
63.4
21.7
2.9
25.7
2.5
dsl
34.9
10.8
3.2
11.7
3.0
Appendix
Table 7: Mean characters and tokens of the gold targets over all 1,357 test orders, for both tokenizers.
Format
1e-4
2e-4
5e-4
Qwen3.5-0.8B
exr
84.77
85.82
80.36
json
83.62
83.62
80.27
edge
exr-short
76.82
79.89
79.98
edge
tagged
86.11
87.07
81.13
dsl
81.13
82.95
79.02
Appendix
Table 8: Learning-rate selection: dev exact match at a 100k pool, mean of 3 seeds; bold is the rate used. “edge”: chosen at the end of the grid.
Format
junk
long
non-order
off-menu
mixed
unknown word
probe
Qwen3.5-0.8B
exr
100.0
100.0
99.4
99.8
99.3
96.1
100.0
json
100.0
100.0
100.0
98.8
98.3
92.9
100.0
exr-short
99.7
100.0
99.4
97.5
96.3
87.0
100.0
tagged
100.0
100.0
99.1
95.9
96.9
86.3
100.0
dsl
100.0
100.0
99.4
99.8
99.0
96.0
100.0
Appendix
Table 9: Held-out negatives refused, by kind, full pool, percent.
Format
swap
retract
only
size
number
drink
cancel
restart
decoy
Qwen3.5-0.8B
exr
73.1
73.2
82.3
81.7
86.7
59.6
61.8
78.6
74.4
json
66.8
63.5
85.6
75.7
80.4
56.9
66.0
77.8
64.9
exr-short
60.3
58.0
79.7
72.6
74.9
46.3
57.6
61.0
59.6
tagged
72.3
67.2
84.7
75.2
82.7
63.8
67.6
79.2
70.1
dsl
72.9
72.1
80.3
72.1
78.9
62.8
67.8
75.6
63.3
Appendix
Table 10: Held-out corrections, exact match by kind, full pool, percent.
Format
n
line count
negation
topping swapped
value swapped
invented
dropped
other structure
Qwen3.5-0.8B
exr
698
17
19
15
31
9
9
1
json
1011
10
34
11
28
3
13
1
exr-short
1180
18
19
18
18
8
18
2
tagged
786
14
33
11
21
4
14
4
dsl
561
15
30
12
20
7
13
3
Appendix
Table 11: Silent errors by kind, full pool, all seeds pooled: n errors, and the percent of them in each kind.
Figure 6: Median latency at batch size one against output tokens, mean over five seeds; filled: Qwen, hollow: LFM2. Lines: least-squares fits.
Semantic parsing maps natural language into structured representations, such as logical forms, graphs, or queries, that machines can execute or reason over.
A small language model that turns a typed order into structure can fail in two ways. It can print something broken, which any system rejects. Or it can print a clean, well formed order that nobody placed. Our companion work calls the second kind a silent error, and it is the dangerous one. We present a web demo in which a fine-tuned 0.8B parameter model runs entirely inside the visitor's browser tab, through llama.cpp compiled to WebAssembly. No order text leaves the device. The page parses orders as the visitor types, scores every ticket with the PIZZA benchmark's own matcher ported to TypeScript, refuses anything off the menu with "?" so the model can face the open web, and lets visitors hunt for silent errors or try to cause one. Run in the browser on all 1,357 PIZZA test orders, the deployed model writes the same output as the server evaluation for 99.0% of them and reaches the same exact match (0.882); we also report its latency on 3 consumer laptops. Demo: https://wrong.cuyor.com
Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad +2
Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from whether to structure' to `how to structure' for optimal reasoning preservation.
Vineet Kumar, Kanishka, Bhuvanesh Mandora
PayPal Artificial Intelligence PayPal, Bengaluru India
When a language model must choose one answer from a large space of equally valid options, a format clause -- "Reply with JSON only" -- changes which answer it chooses. We re-run the One-Word Census (arXiv:2607.12796): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in JSON -- no schema enforcement, no constrained decoding, only the request. Convergence deepens sharply: on the unconstrained "Pick a word" prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36; mean answer-choice surprisal drops from 1.80 to 1.58 bits. The tax is progressive: six of 44 models move individually (BH-FDR q=.10), all toward the mode, led by the most distinctive models, while the conformist floor is immobile. It is a sharpener, not a re-indexer -- the plain-chat modal answer survives in 28 of 31 categories. Defaults are register-indexed: a within-run re-sample (n=20) finds JSON shifts 53% of a model's stable chat defaults, mostly back to the crowd, and installs defaults absent from chat (Claude Fable 5 answers "cerulean" for colour 0% of the time in chat, 100% in JSON). Full-battery controls reveal a register gradient: compression is significant and specific to the answer-delivery formats models are trained to speak (JSON -0.22 bits, p=.0002; XML -0.19, p=.002), absent for YAML and CSV, and reversed for an arbitrary bracket wrapper (+0.13, p=.009) -- weighing the mechanism toward tool-use post-training. Enforcing the schema at the decoder (response_format) compresses no further than the request (-0.03 bits): the collapse lives in the model's response to the register, not the decoder. Structured output is how software consumes language models, and that surface is served by a measurably more homogeneous model than the chat surface on which models are evaluated, compared, and chosen.