Large language models can generate valid JSON that preserves field values yet binds them to the wrong records or hierarchy levels. We introduce JSONPlacement, a controlled task supplying all values and assignment rules for transforming flat records into hierarchical JSON. Across six open-weight models, we identify two error types: record-assignment errors and hierarchy-realization errors. Whole-record moves account for 98.6% of strictly traceable synthetic misplacements. We predict that aligning input with target traversal should reduce assignment errors, while replacing nested output with record-level paths should reduce hierarchy errors. With predictions fixed before generating new instances, we test both interventions separately and together. On synthetic instances, each nearly eliminates its targeted error while leaving the other. Combined, they achieve exact matches for three 8-12B models on all new synthetic instances and a real-data validation set. These findings motivate evaluating record placement explicitly and jointly designing input organization and output representation to preserve value-path bindings.
Figures & tables
Figure 1: JSONPlacement and two error types. (a) Correct transformation. (b) Record-assignment error. (c) Hierarchy-realization error. Each four-cell color group represents one record. All three JSON structures are valid and preserve every value, but only (a) places all values correctly.
Structure
Levels
Branching factors
Records
Original
5
2×3×2×2×2
48
Shallow–wide
2
6×8
48
Deep–narrow
5
2×2×2×2×3
48
Table 1: Synthetic target structures. Levels count grouping fields rather than full JSON path depth.
Original
Shallow–wide
Deep–narrow
Model
VP (%)
VPA (%)
EM
VP (%)
VPA (%)
EM
VP (%)
VPA (%)
EM
Qwen3.5-4B
85.96
53.08
0/24
97.83
97.14
9/24
76.06
37.04
0/24
Qwen3.5-9B
92.62
83.51
0/24
99.48
99.39
18/24
78.99
62.50
0/24
Qwen3.5-27B
99.74
99.57
20/24
100.00
100.00
24/24
98.26
98.00
12/24
Gemma 4 12B IT
97.22
53.39
1/24
100.00
99.74
22/24
87.50
70.20
0/24
Gemma 4 31B IT
99.91
99.91
23/24
100.00
100.00
24/24
98.35
98.00
12/24
Table 2: Baseline performance across six models and three target structures.
Figure 2: Baseline VP and VPA on Original as records per instance increase. Red bars mark values that are preserved but misplaced; numbers above the axis give exact matches out of 24. Each record count uses separately generated instances.
Original
Deep–narrow
Record errors
Record errors
Model
Order
Path
EM
VPA (%)
Other
Outside
Omitted
EM
VPA (%)
Other
Outside
Omitted
Qwen3.5-9B
–
–
0/24
80.82
111
0
110
0/24
60.33
121
3
286
✓
–
24/24
100.00
0
0
0
23/24
99.48
6
0
0
–
✓
0/24
80.42
84
0
104
0/24
72.14
107
0
103
✓
✓
24/24
100.00
0
0
0
24/24
100.00
0
0
0
Table 3: Validation on new instances for the two interventions and their combination.
Baseline errors
EM
Model
Oth.
Out.
Omit.
Base
Order
Path
Both
Synthetic, 27–31B (48/64/96 records, 24 each)
Qwen3.5-27B
9
0
16
60
72
53
72
Gemma 4 31B
1
0
17
56
72
62
72
PDB, 8–12B (48 instances)
Qwen3.5-9B
8
14
22
18
48
11
48
Table 4: Gains of each intervention for larger models and real records.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Definition
VP
Target leaves matched one-to-one to output values, ignoring paths / all target leaves
VPA
Target leaves with the correct value and type at the target path / all target leaves
Output precision
Correctly bound target leaves / all raw output leaf entries, including duplicates
Whole-instance accuracy
Entirely correct outputs / instances with model outputs; no missing or extra paths, duplicate declarations, or conflicts
Appendix
Table 5: Evaluation metrics.
Structure
Logistics
School
Original
tenant, order, shipment, package, role (sender/recipient)
school, campus, class, student, role (student/guardian)
Shallow–wide
tenant, order
school, student
Deep–narrow
tenant, order, shipment, package, address
school, campus, grade, class, student
Appendix
Table 6: Grouping fields by structure and domain, from the top level down. Each field has a corresponding plural container, such as tenants .
Condition
Input
IDs
Output
Baseline
Shuffled
Repeated
Nested
Aligned
Target
Repeated
Nested
Unique IDs
Shuffled
Unique
Nested
Leaf-level paths
Shuffled
Repeated
Leaf paths
Record-level paths
Shuffled
Repeated
Record paths
Appendix
Table 7: What each condition changes. All conditions keep the records and tree shape; only unique identifiers change the target paths.
Original
Shallow + Deep
Model
Logistics
School
Logistics
School
Qwen 9B
88/96
84/96
176/192
180/192
Gemma 12B
94/96
91/96
181/192
184/192
Ministral 8B
95/96
95/96
188/192
–
OLMo 7B
–
–
73/192
–
Total
1,529/1,728
Appendix
Table 8: Scorable outputs in the pooled audit. Each cell covers four conditions with 24 instances per structure.
Model
Cond.
Scor.
VPA (%)
EM
Whole
Qwen 9B
96×2
15/24
62.43
0/24
67/67
48×4
24/24
64.08
0/24
114/114
24×8
24/24
71.68
0/24
82/83
General
66/72
63.40
0/72
310/311
Path
64/72
66.92
0/72
252/252
Gemma 12B
96×2
24/24
67.06
0/24
189/189
Appendix
Table 9: Field-count and checking controls. r×f : records per instance × fields per record. Whole: records moved whole / records with at least one misplaced field. General and Path are the checking instructions, pooled over the three configurations.
Path similarity
Input position
Occupied
One ID (%)
Loss (%)
Ratio
Qwen
2,122/2,340
86.1 / 11.8
60.2 / 14.1
4.3 [3.7, 5.0]
Gemma
787/1,616
91.2 / 10.6
30.0 / 10.3
2.9 [2.5, 3.4]
Appendix
Table 10: Path similarity and input position. Occupied: absent records whose target is taken by another record. One ID: occupants differing from the absent record by one identifier, observed / reference, over 3,136 (Qwen 9B) and 1,850 (Gemma 12B) pairs. Loss: absence rate of the first 20% of input records / the rest, with the ratio and 95% interval.
Record errors
Struct.
IDs
EM
VPA (%)
Other
Outside
Omitted
Qwen3.5-9B
Original
Base
0/24
80.82
111
0
110
Unique
3/24
89.93
8
81
27
Deep
Base
0/24
60.33
121
3
286
Unique
0/24
90.08
0
64
36
Appendix
Table 11: Unique-identifier diagnostic on new instances. Columns as in Table 3 ; Base rows repeat its baseline. Deep: Deep–narrow.
Structure
Cond.
EM
PC
UC
Omit.
Outside
Gemma 4 12B IT
Original
U
10/24
14
3
0
336
J
8/24
16
4
0
384
T
11/24
0
0
13
0
T+C
24/24
0
0
0
0
Deep–narrow
U
1/24
21
7
0
504
Appendix
Table 12: Constrained-decoding diagnostic. U: unconstrained; J: JSON only; T: tree shape; T+C: tree shape plus tenant count. PC, UC: outputs flagged for premature closure and under-closure. Omit.: outputs missing a whole tenant. Outside: records outside the target tree.
Table 13: Intervention cohorts and exact reporting scope. The appendix keeps distinct validation purposes and reused contents explicit.
Model
Structure
Arm
EM
S
I/T
VP
VPA
Prec.
Tokens
Qwen 9B
Original
Base
0/24
24
0/0
90.45
80.82
81.02
3480
Order
24/24
24
0/0
100.00
100.00
100.00
3480
Path
0/24
20
0/4
89.17
80.42
84.70
3565
Both
24/24
24
0/0
100.00
100.00
100.00
3640
Qwen 9B
Deep–narrow
Base
0/24
23
0/1
75.36
60.33
41.56
6416
Order
23/24
24
0/0
100.00
99.48
99.48
3732
Appendix
Table 14: Fresh 2 × 2 validation: accuracy, scoring coverage, and output length. Each row attempts 24 outputs from 48-record instances.
Model
Structure
Arm
N
Other
Outside
Absent
Conflict
Copies
Qwen 9B
Original
Base
1152
111
0
110
0
107
Order
1152
0
0
0
0
0
Path
960
84
0
104
0
60
Both
1152
0
0
0
0
0
Qwen 9B
Deep–narrow
Base
1104
121
3
286
28
1085
Order
1152
6
0
0
0
0
Appendix
Table 15: Fresh 2 × 2 validation: complete record diagnostics. The three record-error columns in the main-text table are supplemented by target conflicts and extra copies.
Model
Order
EM
VP
VPA
Prec.
Other
Outside
Absent
Copies
Tokens
Qwen 9B
Shuffled
0/24
92.27
83.85
83.85
97
0
89
89
3482
Local
10/24
98.18
96.79
97.14
16
0
21
17
3481
Order
24/24
100.00
100.00
100.00
0
0
0
0
3481
Gemma 12B
Shuffled
2/24
97.31
55.73
56.98
4
475
31
4
3484
Local
3/24
99.91
56.16
56.21
0
504
1
0
3540
Order
7/24
100.00
64.58
64.58
0
408
0
0
3541
Appendix
Table 16: Local-shuffle check on reused Original instances. Local permutes records within contiguous target-rank blocks of eight.
Model
Records
Arm
EM
S
I/T
VP
VPA
Prec.
Tokens
Qwen 27B
48
Base
23/24
24
0/0
99.91
99.91
99.91
3480
Order
24/24
24
0/0
100.00
100.00
100.00
3480
Path
17/24
24
0/0
99.39
99.22
99.74
3638
Both
24/24
24
0/0
100.00
100.00
100.00
3640
Qwen 27B
64
Base
18/24
24
0/0
99.35
99.15
99.74
4635
Order
24/24
24
0/0
100.00
100.00
100.00
4635
Appendix
Table 17: Large-model record-count gradient: all four conditions. Each record contains four payload fields.
Model
Records
Arm
N
Other
Outside
Absent
Conflict
Copies
Qwen 27B
48
Base
1152
0
0
1
0
1
Path
1152
2
0
7
0
1
Qwen 27B
64
Base
1536
3
0
10
0
1
Path
1536
3
0
5
0
0
Qwen 27B
96
Base
2304
6
0
5
0
1
Path
2304
3
0
14
0
3
Appendix
Table 18: Large-model record diagnostics by difficulty level. Base and Path are displayed; the two uniformly correct arms are specified in the notes.
Model
Arm
EM
S
I/T
VP
VPA
Prec.
Tokens
Qwen 9B
Base
18/48
46
0/2
98.22
96.52
92.68
6216
Order
48/48
48
0/0
100.00
100.00
100.00
5986
Path
11/48
40
0/8
98.81
97.36
96.95
6644
Both
48/48
48
0/0
100.00
100.00
100.00
6620
Gemma 12B
Base
22/48
47
1/0
99.02
97.68
98.44
6482
Order
48/48
48
0/0
100.00
100.00
100.00
6452
Appendix
Table 19: PDB validation: all model–condition results. Each source entry supplies 48–66 atom records.
Model
Arm
N
Other
Outside
Absent
Conflict
Copies
Qwen 9B
Base
2467
8
14
22
56
221
Order
2574
0
0
0
0
0
Path
2139
14
10
25
17
40
Both
2574
0
0
0
0
0
Gemma 12B
Base
2522
0
29
26
2
7
Order
2574
0
0
0
0
0
Appendix
Table 20: PDB record diagnostics. N explicitly records the denominator lost through invalid or truncated outputs.
Trip IDs
Model
Arm
EM
S
I/T
VP
VPA
Prec.
Tokens
Native
Qwen 9B
Base
8/24
23
0/1
99.55
98.53
92.90
3466
Order
24/24
24
0/0
100.00
100.00
100.00
3346
Gemma 12B
Base
24/24
24
0/0
100.00
100.00
100.00
3443
Order
24/24
24
0/0
100.00
100.00
100.00
3443
Local trip
Qwen 9B
Base
3/24
24
0/0
98.31
95.42
86.02
3432
Order
24/24
24
0/0
100.00
100.00
100.00
3298
Appendix
Table 21: MBTA ordering validation with native and local trip IDs. Only Base and Order belong to this comparison.
Trip IDs
Model
Arm
N
Other
Outside
Absent
Conflict
Copies
Native
Qwen 9B
Base
1104
9
4
2
8
77
Order
1152
0
0
0
0
0
Gemma 12B
Base
1152
0
0
0
0
0
Order
1152
0
0
0
0
0
Local trip
Qwen 9B
Base
1152
3
23
23
10
261
Order
1152
0
0
0
0
0
Appendix
Table 22: MBTA record diagnostics. Copies remain visible even when almost every target value is correctly bound.
Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from whether to structure' to `how to structure' for optimal reasoning preservation.
Vineet Kumar, Kanishka, Bhuvanesh Mandora
PayPal Artificial Intelligence PayPal, Bengaluru India
Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, function calling, data extraction), yet little is known about how constrained decoding (CD) interacts with model scale in this regime. We benchmark five models from three families across 14 structured-output tasks under three decoding conditions (native, Outlines, XGrammar). We introduce a two-axis evaluation that separates structural correctness (schema validity) from semantic correctness (content accuracy). We find that CD eliminates all structural failures across all models (schema validity: 78.6-92.9% to 100%), but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures (e.g., multi-step function calling) remain CD-resistant. Schema conformance is necessary but not sufficient for semantic correctness; CD's reach ends exactly where schema conformance ends.
Structured Outputs from current LLMs exhibit sporadic errors, hindering enterprise AI deployment. We present CONSTRUCT, a real-time uncertainty estimator that scores the trustworthiness of LLM Structured Outputs. Lower-scoring outputs are more likely to contain errors, enabling automatic prioritization of limited human review bandwidth. CONSTRUCT additionally scores the trustworthiness of each field within a Structured Output, helping reviewers quickly identify which parts of the output are incorrect. Our method is suitable for any LLM (including black-box LLM APIs without logprobs), does not require labeled training data or custom model deployment, and supports complex Structured Outputs with heterogeneous fields and nested JSON schemas. We also introduce one of the first public LLM Structured Output benchmarks with reliable ground-truth values. Over this four-dataset benchmark, CONSTRUCT detects errors in outputs from various LLMs (including Gemini 3 and GPT-5) with significantly higher precision/recall than existing techniques.