Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images

A. Said Gurbuz; Ahmed Nassar; Christoph Auer; Maksym Lysak; Lucas Morin; Matteo Omenetti; Tim Strohmeyer; Panagiotis Vagenas

cs.CVAug 21, 2026

Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images

Authors: A. Said Gurbuz, Ahmed Nassar, Christoph Auer, Maksym Lysak, Lucas Morin, Matteo Omenetti, Tim Strohmeyer, Panagiotis Vagenas, +3 more

Organizations: IBM Research Zurich · ETH Zurich · IBM Research Zurich, Rüschlikon, Switzerland · ETH Zurich, Zurich, Switzerland

Abstract

Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.

Figures & tables

Panel of Figure 1 — Figure 1: Key-value extraction examples: (a) visualization with keys (blue), values (green), and links (red arrows); (b) complex many-to-many relationships forming a directed graph.

Paradigm	Model	OCR-free	Loc.	Struct.	Params
Multimodal enc.	LayoutLMv3 [ 13 ]	$\times$	$\times$	$\times$	133M/368M
	LayoutXLM [ 38 ]	$\times$	$\times$	$\times$	345M/625M
	LiLT [ 33 ]	$\times$	$\times$	$\times$	131M
Structure-aware	FormNet [ 18 ]	$\times$	$\times$	$\times$	217M
	GeoLayoutLM [ 24 ]	$\times$	$\times$	$\times$	399M
	XFormParser [ 7 ]	$\times$	$\times$	$\times$	370M

Table 1: Overview of key-value extraction models. OCR-free = no external OCR needed; Loc. = predicts bounding boxes; Struct. = structured output format.

Panel of Figure 2 — Figure 2: DocTags representation: a document image (left) and its corresponding structured markup (right) encoding key-value content, bounding boxes, and links.

Figure 3: Model architecture [ 27 ] : a SigLIP vision encoder (93M) produces patch embeddings that are projected and combined with text embeddings, then fed to a SmolLM2 decoder (135M) to generate DocTags output.

Dataset	Samples			Annotations
Dataset	Train	Val	Test	Entities	Links
DocLayNetV2	16,229	2,032	2,085	429,595	241,469
XFUND	1,043	175	175	69,974	51,146
FUNSD	149	25	25	8,362	5,621

Table 2: Dataset statistics. Entities and Links columns report totals for all splits.

Panel of Figure 4 — Figure 4: Data augmentation pipeline used for training: (a) synthetic filling of empty form fields and (b) graph-based crop generation preserving complete key-value subgraphs.

Model	Input	FUNSD		XFUND
Model	Input	SER	RE	SER	RE
BERT ${}_{\text{base}}$ [ 10 ]	T	0.610	0.277	–	–
XLM-R ${}_{\text{base}}$ [ 9 ]	T	0.667	0.266	0.342	0.222
XLM-R ${}_{\text{large}}$ [ 9 ]	T	0.707	0.347	0.429	0.266
InfoXLM ${}_{\text{base}}$ [ 8 ]	T	0.685	0.292	0.373	0.235
InfoXLM ${}_{\text{large}}$ [ 8 ]	T	0.733	0.368	0.448	0.302

Table 3: Contextual OCR/text-layout baselines on FUNSD and XFUND. These models consume OCR text and/or layout as input and are not directly comparable under our layout-aware metric.

Model	Size	Loc.	FUNSD		XFUND
Model	Size	Loc.	SER	RE	SER	RE
Llama-3.2 [ 11 ]	11B	$\times$	0.389	0.186	–	–
GPT-4o [ 28 ]	–	$\times$	0.554	0.274	–	–
Qwen2.5-VL [ 3 ]	7B	✓	0.273	0.144	0.285	0.123
Ours (SmolDocling)	256M	✓	0.514	0.275	0.412	0.206

Table 4: VLM comparison on FUNSD and XFUND. ✓ /

\times

indicates layout-aware (bounding box) evaluation. All baselines are evaluated zero-shot.

Model	Size	DocLayNetV2		FUNSD		XFUND
Model	Size	SER	RE	SER	RE	SER	RE
Qwen2.5-VL [ 3 ]	7B	0.179	0.074	0.273	0.144	0.285	0.123
Ours (SmolDocling)	256M	0.485	0.321	0.514	0.275	0.412	0.206

Table 5: Layout-aware comparison with Qwen2.5-VL across all three datasets.

Variant	DLNv2		FUNSD		XFUND
Variant	SER	RE	SER	RE	SER	RE
DLNv2 crops + aug.	0.401	0.287	0.479	0.275	0.321	0.122
All crops	0.425	0.304	0.521	0.261	0.361	0.166
All crops + aug.	0.416	0.296	0.514	0.275	0.348	0.172
All crops + aug. + unfilled	0.485	0.321	0.523	0.239	0.376	0.171
All crops + aug. + synth. filled †	0.459	0.301	0.543	0.261	0.412	0.206

Table 6: Data pipeline ablation (layout-aware evaluation). DLNv2 = DocLayNetV2.

Config	DLNv2		FUNSD		XFUND
Config	SER	RE	SER	RE	SER	RE
Link-in-key †	0.459	0.301	0.543	0.261	0.412	0.206
Spatial + link-in-key	0.504	0.282	0.557	0.251	0.441	0.194
Spatial + link-in-value	0.506	0.267	0.551	0.235	0.448	0.175
Spatial + link-in-value + wt. loss	0.509	0.269	0.510	0.197	0.444	0.178

Table 7: Model configuration ablation (layout-aware evaluation). DLNv2 = DocLayNetV2. All variants use all crops + aug. + synth. filled.

Panel of Figure 5 — Figure 5: Predictions on FUNSD and XFUND test documents. The model correctly identifies keys (blue), values (green), bounding boxes, and linking relationships (yellow arrows) across English, Spanish, and Japanese forms.

Panel of Figure 6 — Figure 6: Predictions on DocLayNetV2 test documents. The model generalizes across diverse document types: financial reports (a), structured forms (b), and report-style layouts (c).

Explore similar work

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence
Mar 11, 2026Daxiang Dong, Mingming Zheng, Dong Xu +17Document Image Understanding Optical Character Recognition
HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better
Jul 6, 2026Gengluo Li, Xingyu Wan, Shangpin Peng +20Vision-Language Models Efficient VLM Inference
Institute of Information Engineering, Chinese Academy of Sciences · Large Language Model Department, Tencent · Nankai University
From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings
Jul 14, 2026Zahra AnvariLLM Evaluation Document Information Extraction
Independent AI Researcher.

Mar 11, 2026J/K move · Enter open · S save

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

Daxiang Dong, Mingming Zheng, Dong Xu, Chunhua Luo+16

Document Image Understanding Optical Character Recognition Document Information Extraction Vision-Language Models Document Layout Analysis

Open paper AskPDF arXiv

We present Qianfan-OCR, a 4B-parameter end-to-end vision-language model that unifies document parsing, layout analysis, and document understanding within a single architecture. It performs direct image-to-Markdown conversion and supports diverse prompt-driven tasks including table extraction, chart understanding, document QA, and key information extraction. To address the loss of explicit layout analysis in end-to-end OCR, we propose Layout-as-Thought, an optional thinking phase triggered by special think tokens that generates structured layout representations -- bounding boxes, element types, and reading order -- before producing final outputs, recovering layout grounding capabilities while improving accuracy on complex layouts. Qianfan-OCR ranks first among end-to-end models on OmniDocBench v1.5 (93.12) and OlmOCR Bench (79.8), achieves competitive results on OCRBench, CCOCR, DocVQA, and ChartQA against general VLMs of comparable scale, and attains the highest average score on public key information extraction benchmarks, surpassing Gemini-3.1-Pro, Seed-2.0, and Qwen3-VL-235B. The model is publicly accessible via the Baidu AI Cloud Qianfan platform.

Abstract

Explore similar work

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings