cs.LGFeb 25, 2026

DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion

Authors: Marcel Lamott, Saifullah Saifullah, Nauman Riaz, Yves-Noel Weweler, Tobias Alt-Veit, Ahmad Sarmad Ali, Muhammad Armaghan Shakir, Adrian Kalwa, 

Organizations: RheinMain University of Applied Sciences, Wiesbaden, Germany · German Research Center for Artificial Intelligence, Kaiserslautern, Germany · DeepReader GmbH, Kaiserslautern, Germany · Insiders Technologies GmbH, Kaiserslautern, Germany · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Effective document intelligence models rely on large amounts of annotated training data. However, procuring sufficient and high-quality data poses significant challenges due to the labor-intensive and costly nature of data acquisition. Additionally, leveraging language models to annotate real documents raises concerns about data privacy. Synthetic document generation has emerged as a promising, privacy-preserving alternative. We propose DocDjinn, a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs) that produces annotated documents from unlabeled seed samples. Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset through clustering-based seed selection with parametrized sampling. By enriching documents with realistic diffusion-based handwriting and contextual visual elements via semantic-visual decoupling, we generate diverse, high-quality annotated synthetic documents. We evaluate across eleven benchmarks spanning key information extraction, question answering, document classification, and document layout analysis. To our knowledge, this is the first work demonstrating that VLMs can generate faithful annotated document datasets at scale from unlabeled seeds that can effectively enrich or approximate real, manually annotated data for diverse document understanding tasks. We show that with only 100 real training samples, our framework achieves on average 87%87\% of the performance of the full real-world dataset. We publicly release our code and 140k+ synthetic document samples.

Figures & tables

Figure 1: Examples of synthetically generated documents across diverse domains and tasks. Our framework produces documents with realistic layouts, VLM-generated content, diffusion-based handwriting, and contextual visual elements.
YearNameModelTargetSourceTextHWVEU. GT.TasksMax Res.Dyn. Spec.MLOSEditable
2017DocCreator [ 31 ]manualfull documentimages✓✗✗✓OCR / DLA-✗✓✓✗
2019Bui et al. [ 8 ]GANdocument imagetext✓✗✗✗OCR512×512512\times 512✗✗✗✗
2021Genalog [ 19 ]templatesdocument imagetext✓✗✗✗NERUnspec.✗✗✓✗
2021DocSynth [ 5 ]GANdocument imagelayout✗✗✓✗✗128×128128\times 128✗✗✓✓
2021Raman et al. [ 47 ]samplingfull documentlayout✓✗✓✓DLAUnspec.✗✓✗✗
2022SynthDoG [ 33 ]samplingfull document-✓✗✗✓DLA2560×19202560\times 1920✗✓✓✗
Table 1: Overview of recent synthetic document generation approaches compared to ours. Target specifies the generated modality, and Source the conditioning input. Text , HW , and VE indicate explicit support for readable text, readable handwritten text, and visual elements during generation. U. GT. represents the capability to automatically generate task annotations in an unsupervised manner. Max Res. denotes the maximum possible generation resolution, whereas Dyn. Spec. represents whether the generation process can be controlled via natural language. ML indicates multilingual generation ability, and OS indicates whether the framework is open source. Finally, Editable indicates whether generated documents are produced in an editable format (e.g., structured text) rather than as rasterized images.
Figure 2: Overview of DocDjinn for synthetic document generation. After selecting representative seeds from a source dataset, a VLM generates an HTML representation of the document, along with multi-task ground truth information. This representation is enhanced with diffusion-generated handwriting and further visual elements. Finally, the ground truth is updated with bounding boxes and verified.
Figure 3: Baseline alignment for sentence-level handwritten text. Top, left to right: input image, word segmentation, lowest-ink pixel per column (in red), and computed baseline via percentile (in blue). Bottom: example sentence-level handwritten text after baseline alignment (red).
DatasetSmplα=0.50\alpha=0.50α=0.75\alpha=0.75α=1.00\alpha=1.00
Score ( ↑\uparrow )LFID ( ↓\downarrow )Score ( ↑\uparrow )LFID ( ↓\downarrow )Score ( ↑\uparrow )LFID ( ↓\downarrow )
DocVQACC\mathsf{CC}62.467.6061.517.4963.167.52
DocVQAIC\mathsf{IC}63.956.8864.276.8763.646.96
CORDCC\mathsf{CC}55.6237.3157.0336.9557.7436.75
CORDIC\mathsf{IC}57.5136.9256.5637.0158.4036.46
RVL-CDIPCC\mathsf{CC}43.6511.1545.4912.4545.7411.06
Table 2: Layout-FID (LFID) [ 21 ] and performance comparison for CC\mathsf{CC} and IC\mathsf{IC} sampling across different α\alpha values on LayoutLMv3 (VQA, KIE, CLS) and Faster R-CNN (DLA). Results averaged over 5 seeds (std <0.04<0.04 ).
 Full\mathbf{Full}FewA\mathbf{Few_{A}}FewB\mathbf{Few_{B}}Δ\mathbf{\Delta}
ModelDatasetTaskMetric ( ↑\uparrow )R\mathbf{R}S\mathbf{S}R+S\mathbf{R+S}R\mathbf{R}R+S\mathbf{R+S}R\mathbf{R}R+S\mathbf{R+S}Full−FewB\mathbf{Full{-}Few_{B}}R−S\mathbf{R{-}S}
BERTDocVQAVQAANLS57.9752.9261.3346.6355.4525.6154.043.935.05
LiLTDocVQAVQAANLS70.5064.3472.5358.9966.9538.5964.116.396.16
LayoutLMv3DocVQAVQAANLS71.4566.0373.2662.9168.0443.6165.825.635.42
BERTDocVQA-HWVQAANLS48.2442.2651.1940.9444.0726.4742.725.515.97
LiLTDocVQA-HWVQAANLS58.9450.2658.2450.2150.9535.8850.788.168.67
Table 3: Performance across datasets and tasks with three training settings: (1) Full\mathrm{Full} : models trained on all real (R), all synthetic (S), or both (R+S); (2) FewA\mathrm{Few}_{A} : 300300 – 10001000 real samples with/without synthetic augmentation; (3) FewB\mathrm{Few}_{B} : 100 real samples with/without synthetic augmentation. Gap columns ( Δ\Delta ) report: RFull−(R+S)FewB\mathrm{R_{Full}}-(\mathrm{R{+}S})_{\mathrm{Few}_{B}} (how close 100 augmented samples approaches full training) and RFull−SFull\mathrm{R_{Full}}-\mathrm{S_{Full}} (real vs synthetic quality). Bold indicates best per setting (std <0.02<0.02 ). Synthetic augmentation achieves +12.38 average improvement in FewB\mathrm{Few}_{B} scenarios.
DatasetMethodTaskFID ( ↓\downarrow )LayoutFID ( ↓\downarrow )
DocVQAOursVQA41.366.96
WTQOursVQA52.433.13
 DocGenie [ 21 ]KIE155.3431.30
CORDOursKIE139.5236.46
FUNSDOursKIE44.579.60
KLCOursKIE26.987.98
Table 4: FID [ 24 ] and Layout-FID [ 21 ] scores comparing our method to prior work. Lower scores indicate better distributional similarity to real documents. Resolution annotations indicate synthesis resolution before upscaling for evaluation.
Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

   Pipeline Success (%)Quality
DatasetTaskVLMGTSPVisFID ↓\downarrowLFID ↓\downarrowN
  Claude92.692.295.945.97.06864
DocVQAVQAGemma69.480.273.4773.99.46864
  Claude95.595.593.551.29.6140
FUNSDKIEGemma71.90.071.183.211.2134
  Claude91.697.198.693.510.6850
Appendix
Table 5: Claude vs. Gemma 3 27B comparison. Pipeline Success : GT = valid annotations, SP = single-page renders, Vis = valid visual element and handwriting definitions (documents without handwriting/visual elements are counted as valid). Quality : FID/LFID (Layout-FID) computed on N samples (lower = better).
DatasetTrainValidationTestTrain (Synth)
DocVQA [ 38 ]10194128612878082
DocVQA-HWN/AN/A103N/A
WTQ [ 43 ]13503374211479
SROIE [ 29 ]626N/A3471008
FUNSD [ 30 ]149N/A50259
CORD [ 42 ]8001001001182
Appendix
Table 6: Train/validation/test splits and synthetic training sets for all datasets used in our experiments. For datasets with no validation set, we use 5% of the training set as the validation set.
Embeddingκ\kappaRank Score ↑\uparrow
combined1074
combined572
sentence565
clip556
clip1055
sentence1051
Appendix
Table 7: Clustering configurations ranked using cumulative position scores over our base datasets.
DatasetEmbeddingκ\kappaNum ClustersSilhouette Score ↑\uparrowNorm. Entropy ↑\uparrowFinal Score ↑\uparrow
SROIEcombined10140.640.940.79
ICDAR2019clip590.640.820.73
WTQcombined5500.410.950.68
CORDcombined10220.390.960.68
Tobacco3482combined10310.420.930.67
DocLayNetcombined10480.510.820.66
Appendix
Table 8: Metrics of the selected clusterings for all datasets (top) and listing of all clusterings for DocVQA (bottom), where we used combined embeddings and κ=10\kappa=10 for our experiments.
Figure 4: Overview of our used clusters for all datasets. Each clustering lists embedding type, HDBSCAN [ 10 ] minimum cluster size κ\kappa and number of resulting clusters.
Figure 5: Clustering results across different embeddings and HDBSCAN [ 10 ] minimum cluster sizes κ\kappa for DocVQA [ 38 ] .
Figure 6: Diffusion-based handwriting generation and baseline alignment. (a) The model synthesizes the full sentence “Quick Brown Fox Jumps Over 342 Lazy Dogs” with a coherent global baseline. (b) For each word segment, we estimate a robust baseline which is then used to compose the globally aligned line.
Figure 7: Baseline-aligned handwriting synthesis. The figure shows multiple handwriting segments generated by our latent diffusion model. Individually synthesized words, each with an automatically estimated baseline and composition of these word segments into globally baseline-aligned sentences, demonstrating consistent geometric structure across words and lines.
Figure 8: Image bank for visual elements of type figure , generated with Gemini 2.5 Flash [ 11 ] .
Figure 9: Image bank for visual elements of type logo , generated with Gemini 2.5 Flash [ 11 ] .
Figure 10: Image bank for visual elements of type photo , generated with Gemini 2.5 Flash [ 11 ] (top). Synthetic faces (bottom) generated with StyleGAN2 [ 32 ] via ThisPersonDoesNotExist.com.
Figure 11: DocVQA [ 38 ] synthetic dataset samples for VQA task.
Figure 12: WTQ [ 43 ] synthetic dataset samples for VQA task.
Figure 13: CORD [ 42 ] synthetic dataset samples for KIE task.
Figure 14: FUNSD [ 30 ] synthetic dataset samples for KIE task.
Figure 15: KLC [ 55 ] synthetic dataset samples for VQA task.
Figure 16: SROIE [ 29 ] syntheitc dataset samples for KIE task.
Figure 17: DocLayNet-CLS [ 44 ] synthetic dataset samples for CLS task.
Figure 18: RVL-CDIP [ 22 ] synthetic dataset samples for CLS task.
Figure 19: Tobacco3482 [ 34 ] synthetic dataset samples for CLS task.
Figure 20: DocLayNet-DLA [ 44 ] synthetic dataset samples for DLA task.
Figure 21: ICDAR2019 [ 18 ] synthetic dataset samples for DLA task.
Figure 22: PubLayNet [ 60 ] synthetic dataset samples for DLA task.
  Embedding TypeMin. Cluster SizeTotal SamplesTotal Valid SamplesInput Tokens (M)Output Tokens (M)Cost (USD)Avg. WordsAvg. HW ElemsAvg. Visual Elems
TaskDataset Name          
CLASSIFICATIONDocLayNet-CLS [ 44 ]comb10449439784.087.481244070.0520.334
RVL-CDIP [ 22 ]comb10444138235.165.34962061.4660.371
Tobacco3482 [ 34 ]comb10529240926.246.491162512.0420.435
DLADocLayNet-DLA [ 44 ]comb10450237322.957.281143720.0610.301
ICDAR2019 [ 18 ]img5159415152.012.89492970.0040.048
Appendix
Table 9: Overview of the final synthesized datasets using sampling strategy (v2) and sampling strength ( α=1\alpha=1 ) in our experiments. For each dataset, we report the embedding type (image-only or image+text), clustering and sampling parameters, total and valid sample counts, token usage, estimated generation cost, and average text/visual content statistics.
  Embedding TypeMin. Cluster SizeSample StrategySample AlphaTotal SamplesTotal Valid SamplesInput Tokens (M)Output Tokens (M)Cost (USD)Avg. WordsAvg. HW ElemsAvg. Visual Elems
TaskDataset Name            
CLASSIFICATIONRVL-CDIP [ 22 ]comb10v11450038535.045.28952111.3050.397
RVL-CDIP [ 22 ]comb10v10.75430638604.825.08912121.2570.415
RVL-CDIP [ 22 ]comb10v10.5449139975.035.32962091.240.414
RVL-CDIP [ 22 ]comb10v21444138235.165.34962061.4660.371
RVL-CDIP [ 22 ]comb10v20.75432938915.045.25942101.4460.411
Appendix
Table 10: Ablation variants generated by varying the sampling strategy (v1/v2) and sampling strength ( α\alpha ). The table reports the same statistics as Table 9 , enabling comparison of how sampling choices impact dataset size, token usage, and content composition.
Figure 23: Comparison between real and synthetic data for DocVQA [ 38 ] (a-b) and WTQ [ 43 ] (c-d). UMAP embeddings (a, c) show semantic distributions of questions, while bar charts (b, d) compare question type frequencies across datasets.
Figure 24: Comparisons of class distributions between real and synthetic data for RVL-CDIP [ 22 ] (a), Tobacco3482 [ 34 ] (b) and DocLayNet-CLS [ 44 ] (c).
Figure 25: Heatmaps comparing entity placement by type between real and synthetic for CORD [ 42 ] .
Figure 26: Heatmaps comparing entity placement by type between real and synthetic data for FUNSD [ 30 ] (a) and SROIE [ 29 ] (b).
Figure 27: Comparison of entity type distributions between real and synthetic KIE datasets.
Figure 28: DocLayNet-DLA [ 44 ] GT (left) compared to predictions of Faster R-CNN [ 49 ] trained on purely synthetic data (right). Poor performance likely caused by definitional mismatches in list-item classification (distance-based vs. semantic).
Figure 29: DocLayNet-DLA [ 44 ] GT (left) compared to predictions of Faster R-CNN [ 49 ] trained on purely synthetic data (right). Poor performance likely caused by insufficient variety in visual elements to capture complex images with embedded text.
Figure 30: DocLayNet-DLA [ 44 ] GT (left) compared to predictions of Faster R-CNN [ 49 ] trained on purely synthetic data (right). Poor performance likely caused by limited table diversity in size and structure in our synthetic data. Our model fails to annotate the region as a single table.
Figure 31: DocLayNet-DLA [ 44 ] GT (left) compared to predictions of Faster R-CNN [ 49 ] trained on purely synthetic data (right). Poor performance likely caused by insufficient detailed GT specification. Real GT annotates the formulas as "Picture", while our model annotates them as "Formula".
Figure 32: Comparison of annotation spatial heatmaps and class distributions for real and synthetic versions of ICDAR2019 [ 18 ] (a, b) and PubLayNet [ 60 ] (c, d).
Figure 33: Comparison of annotation spatial heatmaps (top) and class distributions (bottom) for real and synthetic versions of DocLayNet-DLA [ 44 ] .
TaskModelModalityLearning RateBatch SizeEpochsOptimizerWeight DecayMomentumWarmup RatioDropoutSegment-Level LayoutEarly StoppingMixed PrecisionImage Size
 BERT [ 13 ]T1.00E-053250Adam0.01N/A0.10.1✗✓✗N/A
 LiLT [ 58 ]T+L1.00E-053250Adam0.01N/A0.10.1✗✓✗N/A
CLSLayoutLMv3 [ 28 ]T+L+I1.00E-053250Adam0.01N/A0.10.1✓✓✗224×224
 BERT [ 13 ]T2.00E-0516100AdamW0.01N/A0.10.1✗✗✗N/A
 LiLT [ 58 ]T+L2.00E-0516100AdamW0.01N/A0.10.1✗✗✗N/A
KIELayoutLMv3 [ 28 ]T+L+I2.00E-0516100AdamW0.01N/A0.10.1✓✗✗224×224
Appendix
Table 11: Default training hyperparameters for all models and tasks in our experiments. For each task (CLS, KIE, VQA, DLA) and model combination, we report learning rate, batch size, number of epochs, optimizer settings, and other training parameters. These settings were chosen based on preliminary experiments and common practices for document understanding tasks.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

Explore similar work

CardsList
Mar 26, 2026

AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization

Document generation has gained growing attention in the field of AI-driven content creation. In this work, we push its boundaries by introducing AnyDoc, a framework capable of handling multiple generation tasks across a wide spectrum of document categories, all represented in a unified HTML/CSS format. To overcome the limited coverage and scale of existing human-crafted document datasets, AnyDoc first establishes a scalable data synthesis pipeline to automatically generate documents in HTML/CSS form. This pipeline yields DocHTML, a large-scale dataset containing 265,206 document samples, while spanning 111 categories and 32 distinct styles. Additionally, all documents are equipped with comprehensive metadata, including design intentions, HTML/CSS source code, visual assets, and rendered screenshots. Building on the curated dataset, AnyDoc fine-tunes multi-modal large language models (MLLMs) to achieve three practical document generation tasks: intention-to-document, document derendering, and element-to-document. To address the content overflow issue observed during fine-tuning, AnyDoc further incorporates a height-aware reinforcement learning (HARL) post-training procedure. By defining a reward function based on the difference between predicted and target document heights, overflow is penalized and gradually mitigated during HARL, thereby enhancing overall performance. Qualitative and quantitative experiments demonstrate that AnyDoc outperforms both general-purpose MLLMs and task-specific baselines across all three tasks.

Jiawei Lin, Wanrong Zhu, Vlad I Morariu +1

Xi’an Jiaotong University · Adobe Research

Text GenerationSynthetic Data Generation for Language ModelsLLM Fine-TuningMultimodal Large Language ModelsRL for Generative Models
AskPDF
Apr 8, 2026

Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation

Large language models (LLMs) have emerged as a powerful tool for synthetic data generation. A particularly important use case is producing synthetic replicas of private text, which requires carefully balancing privacy and utility. We propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and integrates privacy-preserving strategies, including a formal differential privacy (DP) mechanism in the candidate selection, to generate realistic synthetic data. Comprehensive experiments against state-of-the-art private synthetic data generation methods demonstrate that RPSG achieves high fidelity to private data while providing strong privacy protection.

Qian Ma, Sarah Rajtmajer

Information Sciences and Technology, The Pennsylvania State University

Privacy-Preserving Language ModelsSynthetic Data GenerationSynthetic Data Generation for Language ModelsDifferentially Private Synthetic Data Generation
AskPDF
Apr 15, 2026

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.

Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9

Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

Synthetic Data PretrainingSynthetic Data Generation for Language ModelsSynthetic Data SelectionPrompt Engineering
AskPDF