LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Sönke Tenckhoff; Mario Koddenbrock; Erik Rodner

cs.CLFeb 16, 2026

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Authors: Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner

Organizations: KI-Werkstatt / FB2, University of Applied Sciences Berlin

Abstract

We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.

Figures & tables

Fig. 1: Schematic of the LLMStructBench inference-time evaluation setup. Each test case provides a natural-language message, a corresponding json schema, and an example input-output pair as input to the llm . The model’s generated json object is then evaluated against the gt for both syntactic validity and semantic accuracy.

			json
Use case	Description	Count	Keys	Leaves	Depth
Support tickets	Processing of IT support ticket requests	$199$	$5$	$5$	$2$
Sick leave	Processing of sick-leave requests	$199$	$7$	$5$	$3$
Project extension	Processing of project extension requests	$199$	$9$	$6$	$3$
Conference registration	Processing of conference registration requests	$199$	$9$	$7$	$3$
Loan request	Processing of equipment loan requests	$199$	$10.04$ *	$9.04$ *	$4$

TABLE I: Overview of the LLMStructBench dataset scenarios (use cases): Each scenario includes 199 unique, validated tests, detailed json complexity specifications, and a brief description.

Tag	Description	”format” parameter (API- json mode)	System-prompt
J	Only set the ”format” parameter during the request to ”json”	"json" (str) as request parameter	No mentioning about structure
P	Demand json return in prompts with Schema and example object	Not set	Include json schema and example object
PJ	Set ”format” parameter to ”json” and prompt for json in prompt	"json" (str) as request parameter	Include json schema and example object
J+	Set Schema into ”format” parameter during request	json -Schema (obj) as request parameter	No mentioning about structure
PJ+	Set ”format” parameter and prompt with Schema and example object	json -Schema (obj) as request parameter	Include json schema and example json object

TABLE II: Overview of the five json prompting strategies evaluated in this study. Strategies vary based on the explicit ‘format‘ parameter setting in the API request and the inclusion of schema/example object guidance in the system prompt.

Fig. 2: Parsing success rates for the Phi3 - 3.8b model across different prompting strategies. Bars represent the count of test cases resulting in a perfectly parsed json (green), successfully extracted json from surrounding text (yellow), or a complete parsing failure (red).

Fig. 3: Distribution of error types for the Phi3 - 3.8b model’s json outputs, categorized by prompting strategy. Categories include perfect matches to gt (green), semantic mistakes (orange), critical errors (gray), and complete parsing failures (red).

Strategy	Correct	Mistakes	Errors	Failed
J	$2,271$	$45,951$	$28,797$	$726$
P	$\mathbf{7,791}$	$\mathbf{14,171}$	$2,879$	$1,597$
PJ	$6,800$	$15,792$	$13,506$	$445$
J+	$6,129$	$28,903$	$1,242$	$135$
PJ+	$7,646$	$20,652$	$\mathbf{1,169}$	$\mathbf{18}$

TABLE III: Aggregated performance across all models for each prompting strategy, categorized by outcome. Metrics include total counts for perfectly correct json generations, semantic mistakes, critical errors ( mk / mv ), and complete parsing failures.

Strategy	J	P	PJ	J+	PJ+
Count	$0$	11	$3$	$0$	8

TABLE IV: Ranking of prompting strategies based on their overall effectiveness, showing the count of models for which each strategy achieved the highest performance score.

Fig. 4: Performance breakdown by model size for the Gemma3 family using the P -prompting strategy. The stacked bars illustrate the counts of perfectly correct (green), mistaken (orange), erroneous (gray), and failed (red) json extractions across model variants.

Fig. 5: Composite scores (

F1_{\text{micro}}

and

DOC_{\text{micro}}

) for the Gemma3 model family, demonstrating performance trends across different model sizes under the P -prompting strategy.

Fig. 6: Distribution of

F1_{\text{micro}}

(blue boxes) and

DOC_{\text{micro}}

(purple boxes) scores for the Deepseek-R1 model family, illustrating performance across different model sizes using the P -prompting strategy.

Fig. 7: Distribution of

F1_{\text{micro}}

(blue boxes) and

DOC_{\text{micro}}

(purple boxes) scores for the Deepseek-R1 model family, illustrating performance across different model sizes using the PJ+ -prompting strategy.

Model	Size	$F1_{\text{micro}}$	doc ${}_{\text{micro}}$	Score
GPT-4o	$1.8T$ *	$0.96$	$0.52$	$0.74$
Gemma3	$27$ B	$\mathbf{0.96}$	$\mathbf{0.52}$	$\mathbf{0.74}$
Llama3.1	$70$ B	$0.95$	$0.51$	$0.73$
Gemma3	$12$ B	$0.95$	$0.49$	$0.72$
Deepseek-R1	$70$ B	$0.95$	$0.45$	$0.70$
Llama3.3	$70$ B	$0.95$	$0.45$	$0.70$

TABLE V: Model performance leaderboard displaying

F1_{\text{micro}}

(token-level accuracy),

DOC_{\text{micro}}

(document-level validity), and their weighted composite score for all evaluated models using the P -prompting strategy. GPT-4o is included as a closed-weight reference.

Fig. 8: Scatter plot illustrating

F1_{\text{micro}}

and

DOC_{\text{micro}}

scores for all models as a function of their logarithmic parameter size, evaluated under the P -prompting strategy.

Fig. 9: Scatter plot illustrating

F1_{\text{micro}}

and

DOC_{\text{micro}}

scores for all models as a function of their logarithmic parameter size, evaluated under the PJ+ -prompting strategy.

Fig. 10: Venn diagrams illustrating the distribution and overlap of specific error types ( mk ( mk ), wv ( wv ), mv ( mv )) for the Deepseek-R1 model family under the P -prompting strategy. Each segment indicates the count of test cases exhibiting a single error type or a combination of types.

Fig. 11: Venn diagrams illustrating the distribution and overlap of specific error types ( mk ( mk ), wv ( wv ), mv ( mv )) for the Deepseek-R1 model family under the PJ+ -prompting strategy. Each segment indicates the count of test cases exhibiting a single error type or a combination of types.

	P				PJ+
		$\%$				$\%$
Model	$N_{\text{err}}$	mk	mv	wv	$N_{\text{err}}$	mk	mv	wv
$1.5$ B	$790$	$36.8$	$18.9$	$76.7$	$605$	$0.0$	$0.3$	$99.7$
$7$ B	$601$	$1.2$	$16.0$	$91.5$	$616$	$0.0$	$1.6$	$100.0$
$8$ B	$611$	$0.0$	$5.7$	$97.2$	$658$	$0.0$	$0.0$	$100.0$
$14$ B	$561$	$0.2$	$7.7$	$96.4$	$631$	$0.0$	$0.5$	$100.0$

TABLE VI: Absolute number of erroneous test cases (

N_{\text{err}}

) and the percentage incidence of mk , mv , and wv errors for the Deepseek-R1 model family, across both P and PJ+ prompting strategies. Percentages are calculated relative to

N_{\text{err}}

and can sum to more than 100% due to overlapping error types.

	P				PJ+
		$\%$				$\%$
Model	$N_{\text{err}}$	mk	mv	wv	$N_{\text{err}}$	mk	mv	wv
GPT-4o	$473$	$0.0$	$9.3$	$96.4$	$555$	$0.0$	$5.0$	$98.0$
Gemma3 - 27b	$478$	$0.0$	$6.7$	$97.3$	$559$	$0.0$	$0.0$	$100.0$

TABLE VII: Absolute number of erroneous test cases (

N_{\text{err}}

) and the percentage incidence of mk , mv , and wv errors for the top-performing models ( GPT-4o and Gemma3 - 27b ) across both P and PJ+ prompting strategies. Percentages are calculated relative to

N_{\text{err}}

and can sum to more than 100% due to overlapping error types.

Model	Size	Strategy	Score (P)	Score (PJ+)	Dominant Error(s)	Notable Observations
Gemma3	1B	PJ+	0.49	$\uparrow_{DOC_{\text{micro}}}+0.2$	mk , mv	Complexity penalty under P ; PJ+ improves structure
	12B	P	0.72	$\approx$	wv	Outperforms some 70B models; big jump from 1B
	27B	P	0.74	$\approx$	wv	Marginal gain over 12B ; structural errors rare
LLaMA3	70B (3.1/3.3)	P	0.70–0.73	$\approx$	wv	High accuracy; $DOC_{\text{micro}}$ variance increases with size
Qwen3	0.6B	PJ+	0.61	$\uparrow$	mk , wv	Low $DOC_{\text{micro}}$ with P ; PJ+ removes structural errors
	1.7B	P	0.67	$\approx$	wv	Mid-table despite small size

TABLE VIII: Comprehensive summary of model performance, showing the best-performing prompting strategy for each model family and size variant. The table includes composite scores under both P and PJ+ prompting strategy, identifies dominant error types, and provides key observations regarding model-specific strengths and weaknesses.

Explore similar work

The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM Evaluation LLM Information Extraction
1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India
Structure Tax: How Structured Output affects LLMs Performance
Oct 8, 2026Vineet Kumar, Kanishka, Bhuvanesh MandoraStructured Output Generation LLM Reliability
PayPal Artificial Intelligence PayPal, Bengaluru India
Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction
Feb 24, 2026Hui Wen Goh, Jonas MuellerLLM Answer Verification LLM Uncertainty Estimation
Cleanlab

Apr 28, 2026J/K move · Enter open · S save

The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani, Vineet Agarwal

1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India

LLM Evaluation LLM Information Extraction Automated Evaluation Structured Output Generation

Open paper AskPDF arXiv

Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.

Abstract

Explore similar work

The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

Structure Tax: How Structured Output affects LLMs Performance

Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction