cs.LGMar 2, 2026

Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Data

Authors: Thomas Rückstieß, Robin Vujanic

Organizations: Independent Researcher, Melbourne, Australia · MongoDB, Sydney, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Synthetic data generation is an important capability for privacy-preserving data sharing, system benchmarking and test data provisioning. For mixed-type data, existing synthesizers largely target dense, fixed-schema tables, but many modern data systems store and exchange sparse, semi-structured JSON with nested objects, variable-length arrays and optional keys. Applying tabular synthesizers to such data requires flattening records into wide, sparse tables, turning nested structure and arrays into column-layout artifacts. We present ORiGAMi, an autoregressive transformer architecture for modeling and synthesizing semi-structured records without flattening. ORiGAMi serializes JSON records into key, value, and structural tokens, and encodes token positions by their path in the document tree. Grammar and schema constraints enforce syntactically valid JSON and dataset-consistent structure. We evaluate ORiGAMi against VAE, GAN, diffusion, and autoregressive baselines that operate on flattened representations across six datasets ranging from dense tabular benchmarks to large-scale semi-structured collections. Across fidelity, detection, and utility metrics, ORiGAMi achieves the best score in 17 of 18 benchmark comparisons, while maintaining high privacy scores above 96% across all settings. These results establish native record modeling as a strong alternative to tabular synthesis pipelines, preserving structure while achieving state-of-the-art benchmark performance.

Figures & tables

Figure 1. Tokenization of an example movies record.
Figure 2. origami dual-head model architecture with grammar and schema constraints imposed on the discrete head.
titlegenres.0genres.1genres.2awards.winsawards.nominations
Flash GordonActionAdventureSci-Fi38
TronActionSci-FiNaN“unknown”NaN
Figure 3. Flattening and type separation of two movie records. Nested objects and arrays are mapped to dot-separated columns; variable-length arrays and absent keys produce NaN . Mixed-type columns ( awards.wins : integer vs. string) and partially-present columns are expanded into a type indicator ( .dtype ) and per-type value columns. Homogeneous, fully-present columns ( title , genres.0 , genres.1 ) pass through unchanged.
DatasetRecordsTotal cols.Num.Discr.Sparsity
Adult48,8421561040.0%
Diabetes81,41337132,2610.0%
Electric Vehicles210,0111865,86011.1%
Yelp150,346142626,81277.8%
DDXPlus1,160,131100506,52767.1%
GitHub Issues642,099461187,52393.0%
Table 1. Dataset characteristics after flattening. Sparsity is measured before type separation as the fraction of structurally absent flattened key-path cells.
ModelParamsTime/EpochEpochsTotal
REalTabFormer59.4M2,160 sec40>> 24h
TabDiff25.8M187 sec461>> 24h
TabularARGN9.0M295 sec776.3h
Origami1.7M239 sec20013.3h
Table 2. Training cost comparison on the Yelp dataset.
  TabbyTVAECTGANREaLTabFormerTabularARGNTabDifforigami (ours)
AdultOverall score0.938{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.893{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.876{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.960{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.979{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.990{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}\underline{\mathbf{0.992}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
Shapes0.946{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.905{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.871{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.967{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.985{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.994{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.996{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
Trends0.930{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.011}0.881{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.882{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.954{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.973{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.985{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.989{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
DiabetesOverall scoreOOM0.860{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.927{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.965{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.982{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.982{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}\underline{\mathbf{0.992}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
Shapes–0.890{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.943{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.973{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.988{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.988{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.996{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
Trends–0.830{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.911{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.957{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.976{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.975{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.989{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
Table 3. Fidelity metrics across datasets (mean ±\pm std over 3 replicates). Higher is better.
  TabbyTVAECTGANREaLTabFormerTabularARGNTabDifforigami (ours)
AdultOverall score0.948{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.950{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.893{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.994{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.983{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.982{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}\underline{\mathbf{0.994}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}
TSTR F1F_{1}0.821{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.822{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.773{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.860{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.851{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.850{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.861{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}
DiabetesOverall scoreOOM0.954{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.011}0.926{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.973{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.973{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.976{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}\underline{\mathbf{0.988}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}
TSTR F1F_{1}–0.592{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.007}0.574{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.603{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.602{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.605{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.612{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
Electric VehiclesOverall scoreOOMOOMOOM0.955{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.009}0.983{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.978{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}\underline{\mathbf{0.999}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
TSTR F1F_{1}–––0.923{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.009}0.950{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.945{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.966{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
Table 4. Utility metrics across datasets (mean ±\pm std over 3 replicates). Higher is better. Overall utility normalizes TSTR F1F_{1} by the corresponding real-data baseline.
  TabbyTVAECTGANREaLTabFormerTabularARGNTabDifforigami (ours)
AdultOverall score ↑\uparrow0.587{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.218{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.112{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.807{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.007}0.866{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.967{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}\underline{\mathbf{0.979}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}
ROC AUC ↓\downarrow0.707{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.891{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.944{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.596{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.567{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.517{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.511{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
DiabetesOverall score ↑\uparrowOOM0.045{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.411{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.696{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.896{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.885{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}\underline{\mathbf{1.000}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
ROC AUC ↓\downarrow–0.978{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.794{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}0.652{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.552{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.557{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.485{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
Electric VehiclesOverall score ↑\uparrowOOMOOMOOM0.417{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.640{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.009}0.937{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}\underline{\mathbf{1.000}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
ROC AUC ↓\downarrow–––0.791{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.680{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.531{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.497{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.002}
Table 5. Detection metrics across datasets (mean ±\pm std over 3 replicates). Detection score: higher means harder to detect (better). XGBoost classifier ROC AUC: lower means harder to distinguish from real data (better).
  TabbyTVAECTGANREaLTabFormerTabularARGNTabDifforigami (ours)
AdultOverall score ↑\uparrow\underline{\mathbf{1.000}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.967{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}\underline{\mathbf{1.000}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.915{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.004}0.981{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.831{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.984{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}
DCR ↓\downarrow49.204{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.420}51.663{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.321}49.648{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.276}54.231{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.182}50.938{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.314}58.471{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.070}50.805{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.025}
DiabetesOverall score ↑\uparrowOOM0.967{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.009}\underline{\mathbf{1.000}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.000}0.874{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.963{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.003}0.941{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.006}0.976{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
DCR ↓\downarrow–51.643{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.456}49.504{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.151}56.293{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.068}51.869{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.154}52.936{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.310}51.224{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.054}
Electric VehiclesOverall score ↑\uparrowOOMOOMOOM0.161{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}0.958{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.005}0.938{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}\underline{\mathbf{0.965}}{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.001}
DCR ↓\downarrow–––91.951{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.071}52.081{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.273}53.125{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.056}51.770{\color[rgb]{0.5,0.5,0.5}\scriptstyle\,\pm\,0.072}
Table 6. Privacy metrics across datasets (mean ±\pm std over 3 replicates). Privacy score: higher is better. DCR score ≤\leq 50 indicates no memorization. Exact-match counts are discussed in the text.
Figure 4. KDE visualizations of sparse numeric columns on the Electric Vehicles dataset.
Figure 5. Wasserstein distance of length distributions between real and synthetic data on the Yelp “ categories ”, DDXPlus “ EVIDENCES ” and Github Issues “ issue.labels ” arrays. Mean and variances of 3 seeds (lower is better).
Figure 6. KVPE vs. sequential position encoding on a synthetic nested JSON dataset. KVPE accurately recovers path-specific marginals while sequential PE collapses to uniform ( ≈0.5\approx 0.5 ). Results over 3 seeds.
Figure 7. Training and validation loss curves on Adult with and without key shuffling. The unshuffled model starts to overfit at epoch 25 while the validation loss of the shuffled model remains stable. The higher loss baseline for the shuffled model is due to the unpredictability of randomized key tokens.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 620 papers

Explore similar work

CardsList

  1. TDGT: A Tabular Data Generation Toolkit supporting adaptive GPU-accelerated Bayesian mixture models, diffusion-based models, and latent-space generative modeling

    Jun 30, 2026Vasileios C. Pezoulas, Nikolaos S. Tachos, Eleni Georga +3Bayesian Mixture ModelsBayesian Inference

    SYNTHAINA AI, Ioannina, Greece

  2. Amalgam: Hybrid LLM-PGM Synthesis Algorithm for Accuracy and Realism

    Mar 28, 2026Antheas Kapenekakis, Bent Thomsen, Katja Hose +1Synthetic Tabular Data GenerationGenerative Modeling

    Aalborg University, Aalborg, Denmark · TU Wien, Vienna, Austria

  3. Synthesis Without Training: An Inference-Only Pipeline for Tabular, Temporal, and Relational Synthetic Data

    Sep 29, 2026Zilong Zhao, Abdul Raheem, Jiayu Li +5Constrained Generative ModelingSynthetic Data Generation

    Betterdata AI · National University of Singapore · University of Illinois Urbana-Champaign +1