Conditional generation is a key capability to be supported by Large Tabular Models, enabling a number of applications. Accurate conditional generation has to contend with two key challenges: (i) Data comprise a mixture of numerical and categorical entries, the latter often with high cardinality and heavily unbalanced proportions; (ii) Conditional generation becomes increasingly difficult for rare sub-populations. The best models in this class use a hybrid of continuous and discrete diffusions. In this paper, we design a continuous diffusion model with strong generation quality. An important element is the use of cross-entropy loss for categoricals, for which we provide a first theoretical justification. We introduce metrics that focus on rare subpopulations. Further, we generalize conditional generation techniques to enable new types of conditioning. Finally, we show that our model (trained on labor data) can capture classical insights from econometrics. Code is available at https://github.com/granica-ai/DiffLTM
Figures & tables
Figure 1: Conditional Shape score for conditional generation of ACS Labour data ( n\mboxtrain≈4.8 M). For each condition C , scores are evaluated on matched synthetic and test samples of size n\mboxsynth=n\mboxtest=min(n\mboxtestC,100,000) where n\mboxtestC denotes the number of rows in the test set that satisfy C . DiffLTM -CFG achieves accuracy very close to Rejection sampling (ideal conditioning using samples from DiffLTM ) and to Ideal , outperforming DiffLTM -SMC and Harpoon (Section 4 ; Appendix C.3 ).
Figure 2: Evolution of 10th percentile hourly wage in Arizona and Pennsylvania: observed wages display the effect of the Arizona minimum wage increase post-2016. Comparison of counterfactuals based on synthetic controls and conditional generation. Trained without each state’s post-2016 rows (shaded), DiffLTM -CFG predicts the held-out Pennsylvania wages (placebo, right), and in Arizona (left) produces a no-policy counterfactual well below the observed post-2017 wages. (Section 5 ; Appendix D ).
Dataset
Architecture
Shape
Trend
CShape
MLE
ACSLabour
MSE+Repr
89.39 (0.18)
79.61 (0.31)
72.36 (0.18)
91.22 (1.21)
MSE+Logits
93.70 (0.48)
87.40 (0.80)
81.40 (0.91)
97.33 (0.43)
CE+Logits
98.72 (0.06)
96.87 (0.10)
94.87 (0.07)
99.10 (0.02)
CE+Logits+Learnt
99.04 (0.04)
97.66 (0.04)
95.37 (0.04)
99.41 (0.05)
ACSPubCov
MSE+Repr
98.61 (0.35)
97.32 (0.63)
95.56 (1.73)
98.65 (0.69)
MSE+Logits
98.84 (0.10)
97.80 (0.15)
97.01 (1.06)
99.17 (0.28)
Table 2: Comparison across architectural variants, including MSE versus CE loss, representation (Repr) versus logits parameterization, and the effect of learned embeddings. Higher values indicate better performance for all metrics (mean (SD) over three training seeds) If multiple approaches are within 0.3% of the best mean, all are underlined and none is bold; otherwise the best is bold and the second-best underlined.
Figure 3: The counterfactual effect of Proposition 206 tracks each occupation’s exposure to the minimum wage. The gap between dashed curves (i.e. true 2019 values and DiffLTM predicted counterfactual) is large for cleaning and sales workers, whose observed wages pile up at $11, and negligible for healthcare workers, whose wages already sat above it (see Appendix D.4 for details).
Figure 4: Counterfactual analysis using synthetic control and conditional generation (based on DiffLTM ) for data generated according to a structural equations model. The prediction for f(y)=y , logy , y(maxy−y) .
Figure 5: Squared error between Mincer coefficients fitted on conditionally generated sub-populations and real data slices. Here we used n\mboxsynth=n\mboxtest≈min(n\mboxtestC,100,000) , where n\mboxtestC denotes the number of eligible test rows satisfying C .
Figure 6: Returns to education (China-born, Female, Private Industry). For China-born female private-sector workers, both methods reproduce the education ordering and the concave age–earnings profile of the real data.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Column
Type
Description
AGEP
Numerical
Age in completed years.
WKHP
Numerical
Usual hours worked per week during the previous 12 months.
log_wage
Numerical
Regression target: natural logarithm of annual wage and salary income in constant 2018 dollars, computed as log(WAGP×ADJINC/106) .
ST
Categorical
State or territory of residence.
PUMA
Categorical
Public Use Microdata Area of residence; codes are unique only within state.
SEX
Categorical
Sex.
Appendix
Table 3: Modeled columns in ACSLabour. Categorical variables retain their original ACS PUMS codes; code definitions and semantic labels are provided in the ACS 2014–2018 PUMS Data Dictionary [ 37 ] .
Dataset
Rows
Numerical
Categorical
Target
Task
ACSLabour
6,892,317
3
15
log_wage
Regression
ACSPubCov [ 13 ]
5,916,565
2
18
PUBCOV
Classification
ACSMobility [ 13 ]
2,041,325
4
17
MIG
Classification
Appendix
Table 4: Summary of the ACS-derived datasets
Figure 7: The DiffLTM conditional denoiser differs from the unconditional one only at the input. For each column, the noisy state is concatenated with the parameters of every condition involving that column (equality value, range bounds, or linear weight), each paired with a mask indicating whether the condition is present. A linear condition additionally contributes its bounds and its value st at the current noisy state. The concatenated input is then projected and processed by the same time-conditioned MLP and output heads as the unconditional model, and setting all masks to zero recovers the unconditional model (Section 4.1 ; Appendix C.1 ).
Figure 8: Shape score for conditional generation of ACS Labour data ( n\mboxtrain≈4.8 M) with Harpoon implemented on same base model as CFG and SMC. The generation and evaluation protocol is otherwise identical to that followed in Figure 1 .
Figure 9: DiffLTM generates the full counterfactual wage distribution for any subgroup. Arizona wage CDFs (lowest 30%); 2016 seen in training, 2019 held out. Dotted: $11 minimum set by Proposition 206. The policy effect, the gap between dashed curves, is large for high-school workers and small for bachelor’s-degree workers relative to their 2016 gap (Section 5 ; Appendix D.4 ).
Fidelity Metric
Dataset
Figure
Shape
ACSLabour
Figure 10
Trend
ACSLabour
Figure 11
Conditional Shape (Rare)
ACSLabour
Figure 12
Shape
ACSPubCov
Figure 13
Trend
ACSPubCov
Figure 14
Conditional Shape
ACSPubCov
Figure 15
Appendix
Table 5: Additional fidelity vs. rarity results across fidelity metrics and datasets.
Figure 10: Shape vs rarity on ACSLabour Dataset.
Figure 11: Trend vs rarity on ACSLabour Dataset.
Figure 12: Conditional Shape (Rare) vs rarity on ACSLabour Dataset.
Figure 13: Shape vs rarity on ACSPubCov Dataset.
Figure 14: Trend vs rarity on ACSPubCov Dataset.
Figure 15: Conditional Shape vs rarity on ACSPubCov Dataset.
Figure 16: Conditional Shape (Rare) vs rarity on ACSPubCov Dataset.
Figure 17: Shape vs rarity on ACSMobility Dataset.
Figure 18: Trend vs rarity on ACSMobility Dataset.
Figure 19: Conditional Shape vs rarity on ACSMobility Dataset.
Figure 20: Conditional Shape (Rare) vs rarity on ACSMobility Dataset.
Slice
Rarity (%)
Shape ×100 (SMC)
Shape ×100 (Ideal)
(10AGEP−43)2+(0.5log_wage−10.6)2≤0.64
9.48
97.56
99.09
0.25≤(10AGEP−43)2+(0.5log_wage−10.6)2≤0.64
5.91
97.32
98.97
(AGEP−43)(WKHP−40)<−200
6.90
96.75
99.08
60≤52WKHPexp(log_wage)≤90
3.63
96.70
98.73
Appendix
Table 6: Conditional generation performance of SMC for arbitrary (nonlinear) numerical conditions on the ACSLabour dataset. Shape values displayed are means over three training seeds. Details on implementation and generation provided in Appendix C.2 .
Synthetic tabular data generation has attracted growing attention due to its importance for data augmentation, foundation models, and privacy. However, real-world tabular datasets increasingly contain free-form text fields (e.g., reviews or clinical notes) alongside structured numerical and categorical attributes. Generating such heterogeneous tables with joint modeling of different modalities remains challenging. Existing approaches broadly fall into two categories: diffusion-based methods and LLM-based methods. Diffusion models can capture complex dependencies over numerical and categorical features in continuous or discrete spaces, but extending them to open-ended text is nontrivial and often leads to degraded text quality. In contrast, LLM-based generators naturally produce fluent text, yet their discrete tokenization can distort precise or wide-range numerical values, hindering accurate modeling of both numbers and language. In this work, we propose TabDLM, a unified framework for free-form tabular data generation via a joint numerical-language diffusion model built on masked diffusion language models (MDLMs). TabDLM models textual and categorical features through masked diffusion, while modeling numerical features with a continuous diffusion process through learned specialized numeric tokens embedding; bidirectional attention then captures cross-modality interactions within a single model. Extensive experiments on diverse benchmarks demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines.
Donghong Cai, Jiarui Feng, Yanbo Wang +3
Washington University in St. Louis · Peking University · Ant Group
Generative models for tabular data are typically trained separately for each dataset, limiting knowledge transfer and requiring the storage of many specialized models. In this paper, we introduce CDMD, a tabular diffusion model trained jointly across heterogeneous datasets with different schemas and variable numbers of numerical and categorical features. Unlike existing cross-dataset tabular diffusion models that operate in continuous representation spaces, CDMD defines diffusion directly over the mixed-type feature space and is trained end-to-end. To accommodate heterogeneous categorical domains, we introduce a schema-restricted reverse-process parameterization for masked diffusion models, in which the output space dynamically adapts to each feature's vocabulary. We then compose numerical and categorical feature-level diffusion processes into a schema-dependent row-level process. A shared schema-aware Transformer denoiser captures dependencies between features and parameterizes the reverse process across varying schemas. On seven real-world datasets, a single jointly trained CDMD achieves the highest average generation quality among strong single-dataset and cross-dataset baselines, while using substantially fewer total parameters than the collection of separately trained models. Furthermore, pre-training on a corpus of 337 datasets improves generation on previously unseen datasets under both limited target data and limited adaptation epochs. These results demonstrate the potential of direct mixed-type diffusion for shared and transferable tabular data generation. Our code is available at https://github.com/ketatam/cdmd.
Mohamed Amine Ketata, Maximilian Schambach, Stephan Günnemann
Munich Data Science Institute, Technical University of Munich, Germany · SAP SE, Germany
Generative modeling for tabular data has advanced to the point where models can synthesize realistic records, recover missing values, and generate samples conditioned on class labels or observed attributes. Despite recent advances, existing approaches either require table-specific training or restrict in-context learning to single-cell prediction. In this work, we formulate these tabular generative tasks as a single problem: modeling the joint conditional distribution of an arbitrary set of target cells given the observed cells of an unseen table. To this end, we turn to diffusion, whose denoising objective is defined jointly over all target cells rather than one cell at a time. Under the pretraining distribution, this objective is minimized by the posterior predictive distribution over the target set. Building on this, we propose TabMosaic, a unified in-context diffusion model for generating mixed-type target cells in unseen, potentially incomplete tables without table-specific training. We further show that its pretraining objective bounds the conditional negative log-likelihood and the divergence from the posterior predictive for arbitrary target sets, extending the Bayesian account of in-context learning from a single-value prediction to joint multi-cell generation. Across 48 real-world tables, TabMosaic ranks first on all evaluated generative tasks against 20 baseline approaches