From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

Uddipan Basu Bir; Vincent Christlein; Andreas Maier; Mathias Zinnen

cs.CVOct 8, 2026

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

Authors: Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen

Organizations: Friedrich-Alexander-Universität Erlangen–Nürnberg, Germany

Abstract

While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial APIs limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (up to 7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JSON, enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS*), and mean Average Precision F1 (mAP-F1). Against a fine-tuning baseline, we further test the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and CLAHE), and (iii) multi-stage training. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted VLMs with up to 7B parameters can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction.

Figures & tables

Figure 1: Document example for the Erlangen-Prehistoric dataset (left) and its target JSON structure (right). Image from the “Inventarbuch der Anthropologisch Prähistorischen Sammlung,” courtesy of the Sammlung für Ur- und Frühgeschichte, FAU Erlangen-Nürnberg. Reproduced with permission.

Dataset	Total	Train	Val	Test	Text Type	Schema Complexity
Erlangen-Prehistoric	0, 308	216	0 46	0 46	Handwritten	Medium (8 fields)
Pforzheim Jewelry	0, 590	413	0 88	0 89	Mostly printed	Medium (17 fields)
Regensburg Scalalogy	0, 164	114	0 25	0 25	Mixed	High (nested, booleans)
Total	1,062	743	159	160	—	—

Table 1: Dataset statistics and splits. All datasets are split into 70 % training, 15 % validation, and 15 % test sets.

Figure 2: Representative document example for the jewelry dataset (left) and its target JSON structure (right). Image courtesy of Pforzheim Jewellery Museum and HS Pforzheim (KUPFER Projekt). Reproduced with permission.

Figure 3: Representative document example for the Staircase dataset (left) and its extensive target JSON structure (right). Image courtesy of the Friedrich-Mielke-Institut für Scalalogie, OTH Regensburg, Dossier “Stadtpfarrkirche Bad Königshofen im Grabfeld” . Reproduced with permission.

Model	Architecture Type	Parameters
Donut-Base $\dagger$ [ 17 ]	Encoder-decoder	200M
Donut-CORD $\dagger$ [ 17 ]	Encoder-decoder	200M
Florence-2-Large $\dagger$ [ 31 ]	Unified vision-language	0.77B
Nanonets-OCR-s [ 25 ]	OCR-specialized VLMs	4B
PaddleOCR-VL $\dagger$ [ 10 ]	OCR-specialized VLMs	0.9B
Gemma-3-4B [ 13 ]	Instruction-tuned VLMs	4B

Table 2: Overview of the eight VLMs evaluated. Models marked with

\dagger

only accept fixed OCR-style prompts and do not support few-shot demonstrations.

Model	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Donut-Base †	93.4	0 0.0	0 0.0	97.2	0 0.0	0 0.0	90.1	0 0.0	0 0.0
Donut-CORD †	78.8	0 0.0	0 0.0	91.9	0 0.0	0 0.0	93.1	0 0.0	0 0.0
Florence-2-Large †	45.6	50.4	0 0.0	33.6	66.4	0 0.0	75.6	0 0.0	0 0.0
Nanonets-OCR-s	25.7	49.4	28.5	0 1.9	86.5	86.5	39.3	22.8	15.2
PaddleOCR-VL †	69.2	20.1	0 0.0	51.3	57.0	0 0.0	78.1	0 0.0	0 0.0

Table 3: Zero-shot results for all eight models across three datasets. All metrics are reported as percentages.

\downarrow

: lower is better;

\uparrow

: higher is better.

Model	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Nanonets-OCR-s	61.9	53.3	44.7	0 1.7	93.1	90.7	33.3	44.5	44.9
Gemma-3-4B	26.8	61.7	53.6	0 2.0	97.3	92.1	34.9	33.8	20.2
Phi-3.5-Vision	59.3	40.7	38.9	0 7.2	85.9	71.1	52.7	27.5	23.1
Qwen2.5-VL-7B	18.9	79.0	74.7	0 0.9	98.9	99.4	24.1	55.6	30.7

Table 4: Few-shot results (1–2 examples) for models that support custom prompting and in-context demonstrations. Models restricted to fixed OCR prompts (Donut-Base, Donut-CORD, Florence-2-Large, PaddleOCR-VL) are excluded. All metrics are reported as percentages.

Model	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Donut-Base	16.4	78.0	0 0.0	0 6.3	93.6	0 0.0	59.5	0 0.0	0 0.0
Donut-CORD	20.6	75.0	0 0.0	0 6.9	92.9	0 0.0	66.0	0 0.0	0 0.0
Florence-2-Large	10.7	75.5	45.4	11.1	89.9	0 0.0	57.4	0 4.3	0 0.0
Nanonets-OCR-s	23.0	82.5	81.2	0 0.5	96.7	87.2	20.2	79.9	50.7
PaddleOCR-VL	13.7	90.1	85.3	0 1.1	98.7	84.7	65.1	0 0.0	0 0.0

Table 5: Fine-tuning results for all eight models across three datasets. All metrics are reported as percentages.

Model / Setting	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model / Setting	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Gemma-3-4B (Default)	10.4	88.6	83.1	0.9	99.0	99.4	14.6	73.8	46.9
Gemma-3-4B (HPO)	0 9.3	88.8	65.4	0.4	99.2	99.5	0 3.4	92.3	77.9
Phi-3.5-Vision (Default)	17.9	86.5	82.2	0.3	99.4	99.5	0 6.3	90.1	72.0
Phi-3.5-Vision (HPO)	17.7	84.0	81.2	0.3	99.5	99.7	0 3.2	91.8	73.8
Qwen2.5-VL-7B (Default)	0 5.8	91.0	90.0	0.5	99.2	99.4	0 7.6	87.0	62.8

Table 6: Effect of hyperparameter optimization on the selected top-3 models across three datasets. Scores are reported after fine-tuning. All metrics are reported as percentages.

Model	Preproc.	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	Preproc.	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Gemma-3-4B		10.4	88.6	83.1	0.9	99.0	99.4	14.6	73.8	46.9
Gemma-3-4B	$\checkmark$	10.4	88.8	85.3	1.2	97.6	97.7	16.4	69.7	43.6
Phi-3.5-Vision		17.9	86.5	82.2	0.3	99.4	99.5	0 6.3	90.1	72.0
Phi-3.5-Vision	$\checkmark$	19.0	80.4	76.5	0.3	99.2	99.3	0 5.5	89.5	61.1
Qwen2.5-VL-7B		0 5.8	91.0	90.0	0.5	99.2	99.4	0 7.6	87.0	62.8

Table 7: Effect of image preprocessing on the selected top-3 models across three datasets. Scores are reported after fine-tuning. All metrics are reported as percentages.

Model	Stages	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	Stages	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Gemma-3-4B	single	10.4	88.6	83.1	0.9	99.0	99.4	14.6	73.8	46.9
Gemma-3-4B	multi	0 8.1	89.9	86.5	0.4	99.3	99.5	10.3	80.2	63.8
Phi-3.5-Vision	single	17.9	86.5	82.2	0.3	99.4	99.5	0 6.3	90.1	72.0
Phi-3.5-Vision	multi	18.0	86.7	83.0	0.3	99.5	99.7	0 4.9	90.5	72.5
Qwen2.5-VL-7B	single	0 5.8	91.0	90.0	0.5	99.2	99.4	0 7.6	87.0	62.8

Table 8: Effect of multi-stage training on the selected top-3 models across three datasets. All metrics are reported as percentages.

Model	Erlangen-Prehistoric			Pforzheim Jewelry			Regensburg Scalalogy
Model	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$	CER $\downarrow$	ANLS* $\uparrow$	mAP-F1 $\uparrow$
Gemma-3-4B (Single)	10.4	88.6	83.1	0.9	99.0	99.4	14.6	73.8	46.9
Gemma-3-4B (Multi)	0 8.7	90.6	87.9	0.3	99.4	99.7	0 9.1	81.5	68.0
Phi-3.5-Vision (Single)	17.9	86.5	82.2	0.3	99.4	99.5	0 6.3	90.1	72.0
Phi-3.5-Vision (Multi)	19.7	84.0	45.4	0.5	99.2	99.4	0 7.4	84.3	54.1
Qwen2.5-VL-7B (Single)	0 5.8	91.0	90.0	0.5	99.2	99.4	0 7.6	87.0	62.8

Table 9: Single-dataset vs. multi-dataset fine-tuning for the selected top-3 models. “Single” denotes a separate checkpoint fine-tuned per dataset; “Multi” denotes one checkpoint fine-tuned on the union of all training data and evaluated per dataset test set. All metrics are reported as percentages.

\downarrow

: lower is better;

\uparrow

: higher is better.

Explore similar work

May 26, 2026cs.CL

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.

Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1

Inria, Paris, France

VLM Evaluation Document Understanding Historical Document Analysis VLM Hallucination Vision-Language Grounding Optical Character Recognition

Ask PDF

Sep 14, 2026cs.AI

Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extraction

Vision-language models (VLMs) are increasingly used to extract structured fields from business documents, yet most evaluations report accuracy on clean benchmarks and offer little guidance to practitioners choosing an approach for a given task complexity. We address this gap with a measurement-grounded study and an open-source release. Across eleven systems (three commercial, two reasoning, five open-source VLMs in pretrained and fine-tuned form, and a non-LLM OCR->regex floor) scored on a 750-document held-out pool of synthetic checks, fine-tuning on 3K samples lifts the best open-source VLMs above F1 0.98-above every zero-shot commercial system on this task-while GPT-5 leads the commercial pool on F1 and Claude Sonnet 4.5 collapses on Date. To turn these measurements into actionable choices, we introduce a practitioner-oriented selection framework that maps a task profile (quality, latency, governance, volume) to a recommended approach via filtering and total-cost minimization, illustrated on a hypothetical mid-volume document-extraction scenario.

Kushal Patel, Pushkal Shrivastava, Mackenzie Lees +4

John Hancock, 200 Berkeley St, Boston, MA 02116, USA

VLM Evaluation VLM Robustness Document Information Extraction

Ask PDF

Jun 11, 2026cs.CV

PP-OCRv6: From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Tasks

Vision-Language Models (VLMs) have achieved impressive results on general vision-language tasks, yet they suffer from hallucination, imprecise localization, and prohibitive computational cost when applied to dedicated OCR scenarios. This paper presents PP-OCRv6, a lightweight OCR system that combines architectural innovation with data-centric optimization. PP-OCRv6 redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization, decoupling spatial token mixing from channel mixing and supporting both tasks through task-specific stride configurations. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge. On our in-house benchmarks, PP-OCRv6_medium achieves 83.2% recognition accuracy and 86.2% detection Hmean, outperforming PP-OCRv5_server by +5.1% and +4.6% respectively while surpassing Qwen3-VL-235B, GPT-5.5, and Gemini-3.1-Pro with orders of magnitude fewer parameters. The tiny tier achieves 3.9

\times

faster inference than PP-OCRv5_mobile on Intel Xeon CPU while maintaining comparable accuracy.

Yubo Zhang, Xueqing Wang, Manhui Lin +13

PaddlePaddle Team, Baidu Inc.

Optical Character Recognition

Ask PDF