cs.LGSep 28, 2026

From Data to Program: Fast & Direct Generative Program Inference from Empirical Data

Authors: Simon Klüttermann, Xueying Ding, Leman Akoglu

Organizations: Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Estimating probability densities from a finite set of samples typically requires dataset-specific model fitting. We introduce PRODiGI, a pretrained data-to-program model that infers an explicit, executable generative program in a single forward pass. Pretrained on synthetic datasets paired with their ground-truth programs, PRODiGI accommodates diverse generative families and data dimensionalities through template prediction and non-autoregressive program parameter decoding. Its inferred programs support direct sampling, density and score evaluation, and inspection independently of the pretrained model. We further introduce program-space fine-tuning, which refines differentiable program parameters by matching generated and empirical samples while keeping model parameters intact. Experiments show that PRODiGI achieves lower average density and score MAE than existing pretrained models, while offering multi-fold speedups over its closest competitors. Program-space fine-tuning further reduces generation MMD by 84%. By turning empirical data into explicit, reusable programs, PRODiGI introduces a new direction for fast, interpretable tabular generative modeling.

Figures & tables

Figure 1: PRODiGI overview: (a) Pretraining : Data sampled from normalized, serialized programs across diverse priors is encoded for template prediction and non-autoregressive slot filling; (b) Inference : A single forward pass maps input data to a standalone generative program; (c) Fine-tuning : Inferred program’s differentiable parameters are refined, model parameters remain fixed.
PropertyKDEGMMNDEsTabPFNDiScoDDESNONSNNKDEOurs
Zero-shot/No per-dataset fit✗✗✗✓✓✓✓✓✓✓
Explicit structure and param.s✓✓✗✗✗✗✗✗✓✓
Input-free upon construction✗✓✓✗✗✗✓✓✗✓
Pretrn.-model-free upon constr.✓✓✓✗✗✗✗✗✓✓
Multiple output families✗✗✗✗✗✗✗✗✗✓
Direct generation✓✓△\triangle✓ ∗✗✗✗✓✓✓
Table 1: Comparing classical, neural, and pretrained approaches to density and generative modeling. NDEs: class of Neural Density Estimators; DDE: data-driven Deep Density Estimation ( 27 ) ; SNO: Score Neural Operator ( 21 ) ; NS: Neural Statistician ( 9 ) ; NNKDE: pretrained adaptive KDE ( 37 ) . Last three rows describe capabilities after estimator construction; where direct implies no additional estimation procedure, such as an external sampler, numerical marginalization, etc.
Figure 2: Example input data vs. baseline generations, with corresponding marginals. Appdx. F presents further examples and case studies.
Figure 3: Performance-Inference time trade-off on Generation, Density, and Score estimation tasks on synthetic (all priors) and real-world data. PRODiGI-FT performs best on density and scoring, while PRODiGI offers the fastest generation, remaining competitive on real-world datasets .
 MetricPRODiGI-FTDiscoFormer-FTNNKDE-FTTabPFNKDEGMM
GenerationMMD ( ↓\downarrow ) ×103\times 10^{3}5.46 ± 1.450.89 ± 0.1941.37 ± 8.2067.58 ± 16.5119.51 ± 2.7710.98 ± 1.960.35 ± 0.05 ( ↓\downarrow )5.24 ± 0.950.78 ± 0.30
Rank ( ↓\downarrow )4.29 ± 0.172.92 ± 0.138.05 ± 0.117.93 ± 0.136.73 ± 0.136.14 ± 0.172.39 ± 0.154.33 ± 0.162.23 ± 0.10
% pp << 0.050.05 ( ↓\downarrow )57.3%39.6%99.0%97.9%92.7%80.2%30.2%61.5%31.3%
Time ( ↓\downarrow )0.06 ± 0.0112.90 ± 1.2992.70 ± 0.1493.27 ± 0.140.46 ± 0.261.20 ± 0.0278.09 ± 7.520.79 ± 0.0637.24 ± 2.74
Rank ( ↓\downarrow )1.04 ± 0.024.98 ± 0.037.60 ± 0.058.60 ± 0.052.15 ± 0.063.86 ± 0.057.41 ± 0.143.02 ± 0.066.33 ± 0.05
DensityMAE ( ↓\downarrow )7.58 ± 1.122.19 ± 0.445.10 ± 0.604.17 ± 0.506.44 ± 0.806.25 ± 0.802.31 ± 0.177.14 ± 1.073.46 ± 0.60
Table 2: Performance comparison across tasks and metrics over all prior families a{}^{\text{a}} : (1) Generation distribution match via MMD ( ↓\downarrow ) where %p\%p << 0.050.05 ( ↓\downarrow ) denotes the proportion of datasets where the null hypothesis ( H0H_{0} : generated and real distributions are identical) is rejected; (2) Density estimation via point-wise MAE ( ↓\downarrow ), population-wide Spearman correlation ( ↑\uparrow ), and NLL of generated data ( ↓\downarrow ); (3) Score estimation via MAE ( ↓\downarrow ) and cosine similarity ( ↑\uparrow ). Average rank ( ↓\downarrow ) per metric is across methods over all datasets. Time per task (sec.) is averaged over datasets.
Figure 4: Example real-world data vs. baseline generations, with corresponding marginals for (top) FinancialBeneficiary and (bottom) FinancialTransaction datasets (also see Fig. 11 ).
Figure 5: Ablations of PRODiGI (on 20-dimensional data); left to right: (1) N-AR vs AR decoding validity and speed, (2) generation MMD w/ and w/out normalization, (3) decoding loss w/ and w/out canonical program serialization.
Figure 6: (left) Two-sample rejection rate vs. input dimensionality dd ; (middle) Running time vs. dd ; (right) Average MMD per prior under d≤10d\leq 10 vs. d>10d>10 as the number of context samples is increased.
Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Method / TaskGenerationDensityScore vector
Neural Statistician [ 9 ]Direct conditional generationNumerical latent marginalizationNumerical marginal-score est.
DDE [ 27 ]External samplerDirect predictionAdditional estimation
Score Neural Operator [ 21 ]Iterative samplingAdditional likelihood computationTime-conditioned score prediction b
Adaptive NNKDE [ 37 ]Direct mixture samplingExplicit kernel densityAnalytic score
TabPFN [ 13 ]Autoregressive samplingAutoregressive product of conditional densitiesNot a native output a
DiScoFormer [ 16 ]External samplerDirect predictionDirect prediction
Appendix
Table 3: Capabilities relevant to generation/samling, density estimation, and score-vector estimation. Additional procedures are distinguished from direct/native model outputs.
HyperparameterValuesDescription
mm[1,10][1,10]Number of mixture components
dd[1,50][1,50]Dimensionality of data
μj(k)\bm{\mu}_{j}^{(k)}∼N(0,5)\sim\mathcal{N}(0,5)Component means
Σjj(k)\mathbf{\Sigma}_{jj}^{(k)}(0.5,2](0.5,2]Diagonal variances
ω\bm{\omega}(0.01,1.0−0.01⋅m⋅(m+1)2](0.01,1.0-0.01\cdot\tfrac{m\cdot(m+1)}{2}]Component weights
Appendix
Table 4: Hyperparameters for Gaussian Mixture Model (GMM) Data Prior
HyperparameterValuesDescription
MD[1,5][1,5]Max. In Degree, Max. number of nodes that
  directly influence any node
MP[1,5][1,5]Max. Path Length, Depth of the graph
dd[1,50][1,50]Dimensionality of data/Number of nodes
wi,jw_{i,j}[0.5,1.5]∪[−1.5,0.5][0.5,1.5]\cup[-1.5,0.5]Edge factors
ηi\eta_{i}[0.1,1.0][0.1,1.0]Gaussian noise factor besides causal relation.
Appendix
Table 5: Hyperparameters for Structural Causal Model (SCM) based Data Priors
Figure 7: SCM activation functions used in this paper. We limit our choice to three functions to make sure their behavior is seperable. Odd represents the Cauchy score functions x1+x2\tfrac{x}{1+x^{2}} and Even represents the Cauchy kernel ( 11+x2\tfrac{1}{1+x^{2}} ).
HyperparameterValuesDescription
CCGaussianCopula parameter family
dd[1,50][1,50]Dimensionality of data/Number of marginals
Marginal{Uniform, Normal, Exponential, \{\text{Uniform, Normal, Exponential, }Marginal family for each feature
 Student t, Kumaraswamy}\text{Student t, Kumaraswamy}\} 
μ\mu / σ\sigma[−5,5][-5,5] / [0.5,2.0][0.5,2.0]Mean/Variance for each marginal
ν\nu[2.5,10][2.5,10]Degrees of freedom for Student’s tt distribution
Appendix
Table 6: Hyperparameters for Copula Data Prior
Figure 8: Examples of each copula marginal used in this paper.
Figure 9: Example inferred programs and generations by PRODiGI for each prior family: (a) A 2-d GMM with 3 components; (b) A 2-d Copula with a uniform and a long-tail marginal, respectively; (c) A 5-d SCM with structural mechanisms odd , even , tanh with (right) ground-truth and (left) inferred parents per node/feature in parentheses (empty implies root variable).
Figure 10: Predicting the family of the program to be reconstructed is a learned task, PRODiGI may occasionally misclassify the data prior. Such confusion is rare, however, with only 1/7441/744 programs misclassified (left). Despite the misclassification, the model continues to fit the data well (right), suggesting that these errors arise from overlapping data priors rather than priors that are inherently difficult for PRODiGI .
Dim=1\text{Dim}=1Dim=2\text{Dim}=2Dim=5\text{Dim}=5
DifferentialExpressionAmusementParkSafetyStudentProgrammePerformance
TrafficIncidentMunicipalExpenditureAbandonedMineLand
HeartMurmurDetectionPlanetaryEmissionsAutoencoderAnomaly
AgilePriorityRxJavaPackageRestaurantViolation
SpeciesValidationFinancialBeneficiaryMaritimeVessel
VotingRecallServiceErrorLocationBusStopPrior
Appendix
Table 7: OddBench datasets used as Real-World Datasets
Figure 11: Example real-world datasets and generations by all methods, including fine-tuned variants. The first six examples are two-dimensional, while the last six are PCA representations of one 5-, 10-, 20-, 30-, 40-, and 50-dimensional dataset each. Real-world manifolds challenge all methods, where PRODiGI-FT notably boosts PRODiGI .
Figure 12: Performance–inference time trade-off for Generation, Density, and Score Estimation on real-world datasets. PRODiGI ranks among the fastest methods across all tasks, while PRODiGI-FT consistently improves performance on each task. However, the limited coverage of prior families constrains performance on datasets that cannot be adequately described by the available priors.
 MetricPRODiGI-FTDiscoFormer-FTNNKDE-FTTabPFNKDEGMM
GenerationMMD ( ↓\downarrow ) ×103\times 10^{3}70.34 ± 5.8636.75 ± 4.56253.32 ± 23.04269.06 ± 22.7680.67 ± 9.2730.95 ± 6.1845.15 ± 10.9469.36 ± 13.2915.59 ± 3.27
Rank ( ↓\downarrow )5.95 ± 0.174.02 ± 0.177.85 ± 0.128.22 ± 0.116.30 ± 0.133.33 ± 0.193.28 ± 0.233.95 ± 0.212.11 ± 0.10
% pp << 0.050.05 ( ↓\downarrow )100%90.2%100%100%100%83.7%84.8%80.4%78.3%
Time ( ↓\downarrow )0.06 ± 0.019.97 ± 1.0393.39 ± 0.1096.05 ± 1.890.10 ± 0.011.16 ± 0.0283.39 ± 7.861.35 ± 0.1117.84 ± 1.14
Rank ( ↓\downarrow )1.13 ± 0.045.07 ± 0.047.61 ± 0.058.62 ± 0.051.89 ± 0.043.60 ± 0.067.52 ± 0.133.41 ± 0.066.15 ± 0.06
DensityMAE ( ↓\downarrow )322 ± 239322 ± 239320 ± 239320 ± 239320 ± 239307 ± 237322 ± 239323 ± 238284 ± 210
Appendix
Table 8: Performance comparison across tasks and metrics on real-world datasets a{}^{\text{a}} : (1) Generation distribution match via MMD ( ↓\downarrow ) where %p\%p << 0.050.05 ( ↓\downarrow ) denotes the proportion of datasets where the null hypothesis ( H0H_{0} : generated and real distributions are identical) is rejected; (2) Density estimation via point-wise MAE ( ↓\downarrow ), population-wide Spearman correlation ( ↑\uparrow ), and NLL of generated data ( ↓\downarrow ); (3) Score estimation via MAE ( ↓\downarrow ) and cosine similarity ( ↑\uparrow ). Average rank ( ↓\downarrow ) per metric is across methods over all datasets. Time ( ↓\downarrow ) per task (s) is averaged over datasets.
Figure 13: Distribution of predicted families on Real-World examples. Each Family is predicted between 14 and 46 times, showing that each of our priors helps to model Real-World data more accurately .
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MMD  (×103)\mathrm{MMD}\;(\times 10^{3})2.81 ± 1.370.38 ± 0.1891.35 ± 42.99276.79 ± 98.848.34 ± 1.250.53 ± 0.280.41 ± 0.130.20 ± 0.130.35 ± 0.18
Rank4.75 ± 0.553.08 ± 0.347.96 ± 0.438.33 ± 0.187.00 ± 0.293.92 ± 0.474.21 ± 0.482.79 ± 0.472.96 ± 0.34
Time0.03 ± 0.001.47 ± 0.1693.28 ± 0.6193.80 ± 0.610.38 ± 0.191.17 ± 0.011.52 ± 0.020.11 ± 0.008.71 ± 0.18
Time Rank1.00 ± 0.004.83 ± 0.238.00 ± 0.009.00 ± 0.003.25 ± 0.244.25 ± 0.175.67 ± 0.142.00 ± 0.007.00 ± 0.00
Dim 2MMD  (×103)\mathrm{MMD}\;(\times 10^{3})1.38 ± 0.900.48 ± 0.2512.26 ± 3.7918.70 ± 8.4766.38 ± 8.588.47 ± 2.720.15 ± 0.070.45 ± 0.330.09 ± 0.08
Rank3.29 ± 0.383.67 ± 0.437.42 ± 0.256.79 ± 0.488.83 ± 0.116.50 ± 0.223.21 ± 0.392.75 ± 0.252.54 ± 0.29
Appendix
Table 9: Distribution match performance via MMD ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families.
DimensionPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 141.7%0.0%91.7%100.0%91.7%25.0%8.3%8.3%8.3%
Dim 216.7%16.7%100.0%83.3%100.0%91.7%16.7%8.3%0.0%
Dim 566.7%41.7%100.0%100.0%91.7%83.3%33.3%75.0%33.3%
Dim 1075.0%50.0%100.0%100.0%100.0%83.3%25.0%66.7%41.7%
Dim 2083.3%58.3%100.0%100.0%91.7%91.7%41.7%83.3%16.7%
Dim 3066.7%58.3%100.0%100.0%100.0%100.0%41.7%91.7%50.0%
Appendix
Table 10: Rejection rate of two-sample tests for distribution match w.r.t. MMD over test datasets from ALL prior families. Tests under generated data that successfully match the input data distribution fail to reject the null (i.e., p>0.05p>0.05 ) that the two samples come from the same distribution.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MMD  (×103)\mathrm{MMD}\;(\times 10^{3})5.31 ± 2.700.45 ± 0.3959.23 ± 41.1376.56 ± 59.006.77 ± 0.951.22 ± 0.710.23 ± 0.110.00 ± 0.000.66 ± 0.45
Rank6.50 ± 0.252.38 ± 0.328.25 ± 0.418.25 ± 0.227.00 ± 0.614.50 ± 0.433.00 ± 0.351.88 ± 0.373.25 ± 0.41
Time0.03 ± 0.001.12 ± 0.0192.61 ± 0.0793.18 ± 0.080.18 ± 0.011.16 ± 0.001.56 ± 0.040.11 ± 0.008.30 ± 0.05
Time Rank1.00 ± 0.004.00 ± 0.008.00 ± 0.009.00 ± 0.003.00 ± 0.005.00 ± 0.006.00 ± 0.002.00 ± 0.007.00 ± 0.00
Dim 2MMD  (×103)\mathrm{MMD}\;(\times 10^{3})4.02 ± 2.150.41 ± 0.3415.23 ± 2.698.24 ± 2.4352.62 ± 17.988.83 ± 2.490.22 ± 0.190.28 ± 0.240.00 ± 0.00
Rank4.62 ± 0.653.38 ± 0.608.00 ± 0.356.75 ± 0.548.75 ± 0.226.25 ± 0.222.75 ± 0.382.50 ± 0.182.00 ± 0.31
Appendix
Table 11: Distribution match performance via MMD ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MMD  (×103)\mathrm{MMD}\;(\times 10^{3})3.07 ± 2.470.36 ± 0.30179.43 ± 108.64538.42 ± 175.539.01 ± 3.470.00 ± 0.000.41 ± 0.220.24 ± 0.170.00 ± 0.00
Rank5.25 ± 0.653.00 ± 0.718.25 ± 0.228.50 ± 0.436.50 ± 0.252.75 ± 0.725.00 ± 0.943.50 ± 0.752.25 ± 0.38
Time0.03 ± 0.001.55 ± 0.0592.65 ± 0.1593.13 ± 0.090.17 ± 0.011.18 ± 0.021.51 ± 0.030.10 ± 0.009.31 ± 0.35
Time Rank1.00 ± 0.005.50 ± 0.258.00 ± 0.009.00 ± 0.003.00 ± 0.004.00 ± 0.005.50 ± 0.252.00 ± 0.007.00 ± 0.00
Dim 2MMD  (×103)\mathrm{MMD}\;(\times 10^{3})0.02 ± 0.020.74 ± 0.647.50 ± 2.7825.72 ± 17.6087.66 ± 4.735.60 ± 3.880.22 ± 0.070.04 ± 0.030.01 ± 0.01
Rank2.62 ± 0.212.88 ± 0.727.25 ± 0.417.25 ± 0.419.00 ± 0.006.25 ± 0.414.50 ± 0.562.88 ± 0.372.38 ± 0.21
Appendix
Table 12: Distribution match performance via MMD ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the SCM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MMD  (×103)\mathrm{MMD}\;(\times 10^{3})0.06 ± 0.030.34 ± 0.2035.40 ± 12.62215.40 ± 159.879.24 ± 0.440.38 ± 0.190.60 ± 0.270.38 ± 0.330.40 ± 0.19
Rank2.50 ± 0.403.88 ± 0.377.38 ± 1.148.25 ± 0.227.50 ± 0.434.50 ± 0.904.62 ± 0.693.00 ± 0.983.38 ± 0.74
Time0.02 ± 0.001.75 ± 0.4394.58 ± 1.6495.07 ± 1.650.78 ± 0.521.16 ± 0.001.50 ± 0.030.11 ± 0.008.53 ± 0.10
Time Rank1.00 ± 0.005.00 ± 0.358.00 ± 0.009.00 ± 0.003.75 ± 0.653.75 ± 0.225.50 ± 0.252.00 ± 0.007.00 ± 0.00
Dim 2MMD  (×103)\mathrm{MMD}\;(\times 10^{3})0.09 ± 0.050.28 ± 0.1614.03 ± 10.2922.14 ± 16.9758.87 ± 11.9610.98 ± 6.460.00 ± 0.001.02 ± 0.890.25 ± 0.22
Rank2.62 ± 0.414.75 ± 0.547.00 ± 0.356.38 ± 1.248.75 ± 0.227.00 ± 0.352.38 ± 0.542.88 ± 0.623.25 ± 0.65
Appendix
Table 13: Distribution match performance via MMD ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE0.34 ± 0.150.10 ± 0.030.49 ± 0.050.43 ± 0.060.21 ± 0.030.19 ± 0.012.38 ± 0.010.15 ± 0.030.12 ± 0.02
Rank4.25 ± 0.841.88 ± 0.367.75 ± 0.237.00 ± 0.004.75 ± 0.154.50 ± 0.409.00 ± 0.003.00 ± 0.712.88 ± 0.28
Time0.29 ± 0.001.48 ± 0.250.03 ± 0.020.64 ± 0.060.17 ± 0.001.15 ± 0.003.59 ± 1.480.14 ± 0.008.42 ± 0.07
Time Rank4.00 ± 0.007.38 ± 0.251.25 ± 0.235.00 ± 0.002.75 ± 0.156.62 ± 0.177.12 ± 0.412.00 ± 0.188.88 ± 0.12
Dim 2MAE0.61 ± 0.160.24 ± 0.080.48 ± 0.070.57 ± 0.070.65 ± 0.050.66 ± 0.021.47 ± 0.190.24 ± 0.020.19 ± 0.04
Rank5.25 ± 1.061.88 ± 0.284.62 ± 0.175.88 ± 0.217.00 ± 0.256.50 ± 0.568.88 ± 0.122.75 ± 0.232.25 ± 0.46
Appendix
Table 14: Density estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE0.64 ± 0.230.16 ± 0.050.54 ± 0.080.51 ± 0.070.23 ± 0.050.18 ± 0.012.38 ± 0.010.09 ± 0.010.12 ± 0.01
Rank6.50 ± 0.432.50 ± 0.567.50 ± 0.437.00 ± 0.004.75 ± 0.224.00 ± 0.359.00 ± 0.001.25 ± 0.222.50 ± 0.25
Time0.30 ± 0.001.16 ± 0.020.01 ± 0.000.56 ± 0.000.18 ± 0.011.16 ± 0.005.20 ± 2.610.13 ± 0.008.30 ± 0.05
Time Rank4.00 ± 0.007.00 ± 0.351.00 ± 0.005.00 ± 0.003.00 ± 0.006.50 ± 0.257.75 ± 0.542.00 ± 0.008.75 ± 0.22
Dim 2MAE0.79 ± 0.230.22 ± 0.120.44 ± 0.090.54 ± 0.110.61 ± 0.060.63 ± 0.031.42 ± 0.230.19 ± 0.030.10 ± 0.03
Rank6.50 ± 1.602.25 ± 0.224.50 ± 0.255.75 ± 0.416.50 ± 0.256.25 ± 0.748.75 ± 0.223.00 ± 0.351.50 ± 0.43
Appendix
Table 15: Density estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE0.05 ± 0.020.04 ± 0.010.45 ± 0.050.34 ± 0.060.18 ± 0.020.19 ± 0.012.39 ± 0.020.20 ± 0.050.13 ± 0.03
Rank2.00 ± 0.311.25 ± 0.128.00 ± 0.007.00 ± 0.004.75 ± 0.225.00 ± 0.619.00 ± 0.004.75 ± 0.653.25 ± 0.41
Time0.29 ± 0.001.79 ± 0.450.06 ± 0.040.73 ± 0.100.17 ± 0.001.15 ± 0.001.97 ± 0.840.15 ± 0.008.54 ± 0.10
Time Rank4.00 ± 0.007.75 ± 0.221.50 ± 0.435.00 ± 0.002.50 ± 0.256.75 ± 0.226.50 ± 0.432.00 ± 0.359.00 ± 0.00
Dim 2MAE0.42 ± 0.160.27 ± 0.110.52 ± 0.090.59 ± 0.080.70 ± 0.080.68 ± 0.021.53 ± 0.290.29 ± 0.020.29 ± 0.02
Rank4.00 ± 1.061.50 ± 0.434.75 ± 0.226.00 ± 0.007.50 ± 0.256.75 ± 0.829.00 ± 0.002.50 ± 0.253.00 ± 0.61
Appendix
Table 16: Density estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Corr0.82 ± 0.060.95 ± 0.030.60 ± 0.080.74 ± 0.090.90 ± 0.020.88 ± 0.030.87 ± 0.030.94 ± 0.020.95 ± 0.02
Rank5.21 ± 1.032.79 ± 0.639.00 ± 0.006.43 ± 0.604.57 ± 0.835.43 ± 0.675.86 ± 0.472.71 ± 0.483.00 ± 0.61
Time0.29 ± 0.001.50 ± 0.290.03 ± 0.030.65 ± 0.070.17 ± 0.001.15 ± 0.003.40 ± 1.690.14 ± 0.008.44 ± 0.07
Time Rank4.00 ± 0.007.43 ± 0.281.29 ± 0.265.00 ± 0.002.86 ± 0.136.71 ± 0.177.00 ± 0.451.86 ± 0.138.86 ± 0.13
Dim 2Corr0.76 ± 0.080.91 ± 0.040.80 ± 0.040.80 ± 0.050.92 ± 0.020.79 ± 0.030.73 ± 0.070.95 ± 0.010.93 ± 0.03
Rank5.69 ± 1.112.81 ± 0.617.00 ± 0.436.25 ± 0.584.00 ± 0.316.50 ± 0.617.12 ± 0.822.62 ± 0.433.00 ± 0.71
Appendix
Table 17: Density estimation performance via Spearman correlation between the overall ranking by estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Corr0.68 ± 0.040.91 ± 0.040.50 ± 0.110.64 ± 0.140.92 ± 0.030.90 ± 0.020.87 ± 0.040.96 ± 0.010.95 ± 0.02
Rank7.50 ± 0.253.25 ± 0.969.00 ± 0.007.00 ± 0.613.00 ± 0.795.00 ± 0.615.50 ± 0.562.00 ± 0.352.75 ± 0.54
Time0.30 ± 0.001.16 ± 0.020.01 ± 0.000.56 ± 0.000.18 ± 0.011.16 ± 0.005.20 ± 2.610.13 ± 0.008.30 ± 0.05
Time Rank4.00 ± 0.007.00 ± 0.351.00 ± 0.005.00 ± 0.003.00 ± 0.006.50 ± 0.257.75 ± 0.542.00 ± 0.008.75 ± 0.22
Dim 2Corr0.67 ± 0.140.92 ± 0.060.89 ± 0.050.86 ± 0.070.95 ± 0.020.82 ± 0.040.75 ± 0.070.98 ± 0.000.99 ± 0.00
Rank7.12 ± 1.623.12 ± 0.876.00 ± 0.355.75 ± 0.824.25 ± 0.416.25 ± 0.898.00 ± 0.353.00 ± 0.501.50 ± 0.43
Appendix
Table 18: Density estimation performance via Spearman correlation between the overall ranking by estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Corr1.00 ± 0.001.00 ± 0.000.73 ± 0.050.88 ± 0.060.88 ± 0.030.87 ± 0.050.88 ± 0.020.92 ± 0.040.96 ± 0.03
Rank2.17 ± 0.542.17 ± 0.549.00 ± 0.005.67 ± 0.986.67 ± 0.276.00 ± 1.256.33 ± 0.723.67 ± 0.723.33 ± 1.19
Time0.29 ± 0.001.95 ± 0.570.07 ± 0.050.78 ± 0.120.17 ± 0.001.15 ± 0.001.00 ± 0.040.14 ± 0.008.64 ± 0.05
Time Rank4.00 ± 0.008.00 ± 0.001.67 ± 0.545.00 ± 0.002.67 ± 0.277.00 ± 0.006.00 ± 0.001.67 ± 0.279.00 ± 0.00
Dim 2Corr0.86 ± 0.080.90 ± 0.070.72 ± 0.040.75 ± 0.050.89 ± 0.020.75 ± 0.030.71 ± 0.110.92 ± 0.020.86 ± 0.02
Rank4.25 ± 1.142.50 ± 0.838.00 ± 0.356.75 ± 0.743.75 ± 0.416.75 ± 0.826.25 ± 1.472.25 ± 0.654.50 ± 0.83
Appendix
Table 19: Density estimation performance via Spearman correlation between the overall ranking by estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior.
DimensionMetricPRODiGIPRODiGI-FTNNKDENNKDE-FTKDEGMM
Dim 1NLL0.63 ± 0.260.58 ± 0.260.66 ± 0.270.58 ± 0.260.61 ± 0.250.58 ± 0.25
Rank3.81 ± 0.572.69 ± 0.354.75 ± 0.413.62 ± 0.233.25 ± 0.572.88 ± 0.36
Time0.29 ± 0.001.48 ± 0.250.17 ± 0.001.15 ± 0.000.14 ± 0.008.42 ± 0.07
Time Rank3.00 ± 0.004.88 ± 0.121.88 ± 0.124.12 ± 0.121.12 ± 0.126.00 ± 0.00
Dim 2NLL1.46 ± 0.601.32 ± 0.571.55 ± 0.651.50 ± 0.641.31 ± 0.581.29 ± 0.58
Rank3.62 ± 0.572.62 ± 0.374.50 ± 0.374.00 ± 0.223.38 ± 0.212.88 ± 0.30
Appendix
Table 20: Density estimation performance via NLL ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families. We do not report NLL’s for TabPFN and Disco variants, as their predicted densities do not integrate to one.
DimensionMetricPRODiGIPRODiGI-FTNNKDENNKDE-FTKDEGMM
Dim 1NLL0.96 ± 0.450.86 ± 0.460.91 ± 0.490.84 ± 0.460.83 ± 0.450.83 ± 0.46
Rank5.12 ± 0.513.38 ± 0.214.38 ± 0.603.38 ± 0.212.12 ± 0.572.62 ± 0.32
Time0.30 ± 0.001.16 ± 0.020.18 ± 0.011.16 ± 0.000.13 ± 0.008.30 ± 0.05
Time Rank3.00 ± 0.004.75 ± 0.222.00 ± 0.004.25 ± 0.221.00 ± 0.006.00 ± 0.00
Dim 2NLL2.35 ± 0.882.08 ± 0.882.41 ± 0.972.32 ± 0.972.02 ± 0.912.00 ± 0.91
Rank4.12 ± 1.042.38 ± 0.484.88 ± 0.454.12 ± 0.273.12 ± 0.372.38 ± 0.48
Appendix
Table 21: Density estimation performance via NLL ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior. We do not report NLL’s for TabPFN and Disco variants, as their predicted densities do not integrate to one.
DimensionMetricPRODiGIPRODiGI-FTNNKDENNKDE-FTKDEGMM
Dim 1NLL0.30 ± 0.120.30 ± 0.120.41 ± 0.150.33 ± 0.130.39 ± 0.130.32 ± 0.13
Rank2.50 ± 0.402.00 ± 0.475.12 ± 0.513.88 ± 0.374.38 ± 0.603.12 ± 0.62
Time0.29 ± 0.001.79 ± 0.450.17 ± 0.001.15 ± 0.000.15 ± 0.008.54 ± 0.10
Time Rank3.00 ± 0.005.00 ± 0.001.75 ± 0.224.00 ± 0.001.25 ± 0.226.00 ± 0.00
Dim 2NLL0.57 ± 0.490.57 ± 0.490.70 ± 0.610.69 ± 0.600.60 ± 0.520.58 ± 0.50
Rank3.12 ± 0.322.88 ± 0.544.12 ± 0.543.88 ± 0.323.62 ± 0.113.38 ± 0.11
Appendix
Table 22: Density estimation performance via NLL ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior. We do not report NLL’s for TabPFN and Disco variants, as their predicted densities do not integrate to one.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE4.66 ± 2.991.00 ± 0.383.15 ± 1.113.33 ± 1.181.86 ± 0.4518.61 ± 4.976.48 ± 1.421.84 ± 0.311.90 ± 0.57
Rank4.69 ± 1.022.44 ± 0.475.25 ± 0.235.88 ± 0.413.38 ± 0.359.00 ± 0.007.25 ± 0.583.62 ± 0.713.50 ± 0.79
Time0.29 ± 0.001.48 ± 0.250.03 ± 0.020.64 ± 0.060.17 ± 0.001.15 ± 0.002.05 ± 0.630.15 ± 0.008.42 ± 0.07
Time Rank4.00 ± 0.007.38 ± 0.251.25 ± 0.235.00 ± 0.002.75 ± 0.156.50 ± 0.187.12 ± 0.332.00 ± 0.189.00 ± 0.00
Dim 2MAE19.34 ± 7.7916.26 ± 7.2028.45 ± 13.8527.94 ± 13.0625.51 ± 12.48102.51 ± 53.9753.68 ± 33.2925.84 ± 12.8728.64 ± 15.15
Rank5.19 ± 0.982.94 ± 0.704.38 ± 0.355.12 ± 0.453.88 ± 0.679.00 ± 0.007.38 ± 0.353.62 ± 0.473.50 ± 0.98
Appendix
Table 23: Score estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE9.05 ± 5.111.83 ± 0.464.65 ± 1.944.98 ± 2.032.46 ± 0.7722.28 ± 9.126.89 ± 2.761.82 ± 0.571.69 ± 0.50
Rank7.50 ± 0.253.25 ± 0.745.25 ± 0.226.25 ± 0.223.00 ± 0.509.00 ± 0.006.50 ± 1.031.75 ± 0.412.50 ± 0.56
Time0.30 ± 0.001.16 ± 0.020.01 ± 0.000.56 ± 0.000.18 ± 0.011.16 ± 0.002.73 ± 1.140.14 ± 0.008.30 ± 0.05
Time Rank4.00 ± 0.007.00 ± 0.351.00 ± 0.005.00 ± 0.003.00 ± 0.006.50 ± 0.257.50 ± 0.432.00 ± 0.009.00 ± 0.00
Dim 2MAE5.94 ± 2.952.52 ± 1.293.42 ± 1.683.99 ± 1.913.03 ± 1.4018.47 ± 7.874.72 ± 1.952.54 ± 1.241.51 ± 0.76
Rank6.12 ± 1.352.88 ± 0.914.00 ± 0.355.75 ± 0.544.75 ± 0.749.00 ± 0.007.00 ± 0.614.00 ± 0.791.50 ± 0.43
Appendix
Table 24: Score estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE0.26 ± 0.160.17 ± 0.091.64 ± 0.181.68 ± 0.191.26 ± 0.1914.94 ± 2.956.08 ± 0.641.85 ± 0.272.10 ± 1.00
Rank1.88 ± 0.321.62 ± 0.115.25 ± 0.415.50 ± 0.753.75 ± 0.419.00 ± 0.008.00 ± 0.005.50 ± 0.254.50 ± 1.30
Time0.29 ± 0.001.79 ± 0.450.06 ± 0.040.73 ± 0.100.17 ± 0.001.15 ± 0.001.38 ± 0.220.15 ± 0.008.54 ± 0.10
Time Rank4.00 ± 0.007.75 ± 0.221.50 ± 0.435.00 ± 0.002.50 ± 0.256.50 ± 0.256.75 ± 0.412.00 ± 0.359.00 ± 0.00
Dim 2MAE32.74 ± 12.0029.99 ± 10.5453.48 ± 21.2451.88 ± 19.8047.98 ± 19.20186.55 ± 89.77102.64 ± 56.8449.14 ± 19.7455.76 ± 23.44
Rank4.25 ± 1.243.00 ± 1.064.75 ± 0.544.50 ± 0.563.00 ± 0.949.00 ± 0.007.75 ± 0.223.25 ± 0.415.50 ± 1.30
Appendix
Table 25: Score estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Cosine0.54 ± 0.130.68 ± 0.130.46 ± 0.110.49 ± 0.120.59 ± 0.100.14 ± 0.020.39 ± 0.080.65 ± 0.100.65 ± 0.12
Rank4.94 ± 0.883.12 ± 0.676.25 ± 0.465.50 ± 0.774.38 ± 0.588.31 ± 0.466.12 ± 0.483.38 ± 0.643.00 ± 0.47
Time0.29 ± 0.001.48 ± 0.250.03 ± 0.020.64 ± 0.060.17 ± 0.001.15 ± 0.002.05 ± 0.630.15 ± 0.008.42 ± 0.07
Time Rank4.00 ± 0.007.38 ± 0.251.25 ± 0.235.00 ± 0.002.75 ± 0.156.50 ± 0.187.12 ± 0.332.00 ± 0.189.00 ± 0.00
Dim 2Cosine0.59 ± 0.110.80 ± 0.080.66 ± 0.090.63 ± 0.090.75 ± 0.060.22 ± 0.020.54 ± 0.050.76 ± 0.060.77 ± 0.08
Rank5.69 ± 1.103.31 ± 0.825.38 ± 0.396.00 ± 0.533.75 ± 0.668.75 ± 0.155.88 ± 0.623.50 ± 0.402.75 ± 0.81
Appendix
Table 26: Score estimation performance via cosine similarity between score vector estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from ALL prior families.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Cosine0.35 ± 0.080.62 ± 0.150.45 ± 0.100.45 ± 0.100.70 ± 0.100.15 ± 0.020.51 ± 0.090.76 ± 0.060.73 ± 0.08
Rank6.88 ± 0.873.25 ± 1.146.25 ± 0.416.75 ± 0.653.00 ± 0.008.88 ± 0.115.50 ± 0.562.00 ± 0.612.50 ± 0.43
Time0.30 ± 0.001.16 ± 0.020.01 ± 0.000.56 ± 0.000.18 ± 0.011.16 ± 0.002.73 ± 1.140.14 ± 0.008.30 ± 0.05
Time Rank4.00 ± 0.007.00 ± 0.351.00 ± 0.005.00 ± 0.003.00 ± 0.006.50 ± 0.257.50 ± 0.432.00 ± 0.009.00 ± 0.00
Dim 2Cosine0.50 ± 0.160.87 ± 0.090.86 ± 0.050.81 ± 0.070.87 ± 0.020.26 ± 0.020.61 ± 0.070.90 ± 0.010.97 ± 0.01
Rank6.88 ± 1.563.38 ± 1.084.50 ± 0.255.25 ± 0.654.50 ± 1.038.50 ± 0.256.50 ± 0.754.00 ± 0.611.50 ± 0.43
Appendix
Table 27: Score estimation performance via cosine similarity between score vector estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the GMM prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Cosine0.73 ± 0.210.73 ± 0.210.47 ± 0.200.54 ± 0.210.48 ± 0.150.14 ± 0.040.27 ± 0.080.53 ± 0.170.57 ± 0.21
Rank3.00 ± 0.683.00 ± 0.686.25 ± 0.824.25 ± 1.085.75 ± 0.657.75 ± 0.826.75 ± 0.654.75 ± 0.543.50 ± 0.75
Time0.29 ± 0.001.79 ± 0.450.06 ± 0.040.73 ± 0.100.17 ± 0.001.15 ± 0.001.38 ± 0.220.15 ± 0.008.54 ± 0.10
Time Rank4.00 ± 0.007.75 ± 0.221.50 ± 0.435.00 ± 0.002.50 ± 0.256.50 ± 0.256.75 ± 0.412.00 ± 0.359.00 ± 0.00
Dim 2Cosine0.69 ± 0.130.73 ± 0.120.46 ± 0.100.45 ± 0.110.62 ± 0.070.18 ± 0.030.47 ± 0.060.62 ± 0.070.57 ± 0.09
Rank4.50 ± 1.303.25 ± 1.246.25 ± 0.416.75 ± 0.653.00 ± 0.619.00 ± 0.005.25 ± 0.893.00 ± 0.354.00 ± 1.27
Appendix
Table 28: Score estimation performance via cosine similarity between score vector estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) is across methods over test datasets from the Copula prior.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MMD  (×103)\mathrm{MMD}\;(\times 10^{3})41.79 ± 15.476.39 ± 2.24184.48 ± 66.92213.03 ± 81.6619.72 ± 3.366.30 ± 3.407.59 ± 3.515.03 ± 2.525.93 ± 3.09
Rank5.90 ± 0.623.85 ± 0.688.60 ± 0.158.30 ± 0.205.80 ± 0.283.10 ± 0.543.70 ± 0.552.70 ± 0.213.05 ± 0.34
Time0.04 ± 0.011.22 ± 0.0693.20 ± 0.1493.87 ± 0.150.08 ± 0.001.13 ± 0.002.12 ± 0.230.22 ± 0.028.34 ± 0.10
Time Rank1.10 ± 0.094.70 ± 0.148.00 ± 0.009.00 ± 0.002.00 ± 0.144.30 ± 0.146.00 ± 0.002.90 ± 0.097.00 ± 0.00
Dim 2MMD  (×103)\mathrm{MMD}\;(\times 10^{3})65.60 ± 12.6818.19 ± 8.0474.69 ± 16.1288.65 ± 23.06165.03 ± 39.658.53 ± 2.754.86 ± 1.488.73 ± 5.712.94 ± 0.84
Rank6.42 ± 0.634.42 ± 0.536.92 ± 0.307.17 ± 0.428.25 ± 0.293.46 ± 0.513.04 ± 0.302.38 ± 0.502.96 ± 0.34
Appendix
Table 29: Distribution match performance via MMD ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets.
DimensionPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1100.0%60.0%100.0%100.0%100.0%50.0%60.0%50.0%40.0%
Dim 2100.0%66.7%100.0%100.0%100.0%58.3%66.7%41.7%50.0%
Dim 5100.0%90.9%100.0%100.0%100.0%81.8%72.7%63.6%63.6%
Dim 10100.0%100.0%100.0%100.0%100.0%72.7%90.9%81.8%72.7%
Dim 20100.0%100.0%100.0%100.0%100.0%100.0%83.3%100.0%100.0%
Dim 30100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%91.7%
Appendix
Table 30: Rejection rate of two-sample tests for distribution match w.r.t. MMD over test datasets over real-world test datasets.. Tests under generated data that successfully match the input data distribution fail to reject the null (i.e., p>0.05p>0.05 ) that the two samples come from the same distribution.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE1035 ± 7911035 ± 7911037 ± 7911037 ± 7911037 ± 791991 ± 7891037 ± 7901045 ± 789933 ± 695
Rank6.10 ± 0.684.30 ± 0.755.70 ± 0.606.30 ± 0.455.80 ± 0.633.70 ± 0.726.20 ± 0.884.50 ± 0.932.40 ± 0.62
Time0.04 ± 0.011.22 ± 0.060.02 ± 0.010.68 ± 0.060.08 ± 0.001.13 ± 0.001.07 ± 0.020.27 ± 0.028.34 ± 0.10
Time Rank2.00 ± 0.147.40 ± 0.321.30 ± 0.285.30 ± 0.282.80 ± 0.137.00 ± 0.206.30 ± 0.143.90 ± 0.099.00 ± 0.00
Dim 2MAE5 ± 13 ± 13 ± 13 ± 12 ± 13 ± 13 ± 13 ± 12 ± 1
Rank6.78 ± 0.865.00 ± 0.834.67 ± 0.475.33 ± 0.425.56 ± 0.694.56 ± 0.827.22 ± 0.684.44 ± 0.861.44 ± 0.28
Appendix
Table 31: Density estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Corr0.24 ± 0.090.41 ± 0.100.31 ± 0.090.37 ± 0.110.51 ± 0.110.74 ± 0.090.43 ± 0.110.60 ± 0.130.49 ± 0.12
Rank6.65 ± 0.675.20 ± 0.646.75 ± 0.546.15 ± 0.674.60 ± 0.382.80 ± 0.565.30 ± 0.723.05 ± 0.704.50 ± 0.74
Time0.04 ± 0.011.22 ± 0.060.02 ± 0.010.68 ± 0.060.08 ± 0.001.13 ± 0.001.07 ± 0.020.27 ± 0.028.34 ± 0.10
Time Rank2.00 ± 0.147.40 ± 0.321.30 ± 0.285.30 ± 0.282.80 ± 0.137.00 ± 0.206.30 ± 0.143.90 ± 0.099.00 ± 0.00
Dim 2Corr0.40 ± 0.080.59 ± 0.080.54 ± 0.060.53 ± 0.060.54 ± 0.060.57 ± 0.060.54 ± 0.090.64 ± 0.050.76 ± 0.07
Rank7.50 ± 0.435.22 ± 0.735.67 ± 0.655.44 ± 0.744.78 ± 0.805.67 ± 0.655.72 ± 0.933.33 ± 0.741.67 ± 0.27
Appendix
Table 32: Density estimation performance via Spearman correlation between the overall ranking by estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets.
DimensionMetricPRODiGIPRODiGI-FTNNKDENNKDE-FTKDEGMM
Dim 1NLL1.26 ± 0.121.29 ± 0.201.14 ± 0.0949.30 ± 28.9811.37 ± 8.79105.05 ± 96.95
Rank3.50 ± 0.552.40 ± 0.403.80 ± 0.564.40 ± 0.432.80 ± 0.564.10 ± 0.41
Time0.04 ± 0.011.22 ± 0.060.08 ± 0.001.13 ± 0.000.27 ± 0.028.34 ± 0.10
Time Rank1.10 ± 0.094.70 ± 0.141.90 ± 0.094.30 ± 0.143.00 ± 0.006.00 ± 0.00
Dim 2NLL4.02 ± 0.812.38 ± 0.491.90 ± 0.361.53 ± 0.391.51 ± 0.341.48 ± 0.39
Rank5.00 ± 0.474.11 ± 0.533.78 ± 0.472.89 ± 0.372.67 ± 0.352.56 ± 0.45
Appendix
Table 33: Density estimation performance via NLL ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets. We do not report NLL’s for TabPFN and Disco variants, as their predicted densities do not integrate to one.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1MAE14362 ± 987214361 ± 987114361 ± 987214361 ± 987214361 ± 987213251 ± 974014358 ± 986914344 ± 985911859 ± 7643
Rank6.30 ± 0.454.90 ± 0.505.70 ± 0.695.70 ± 0.474.40 ± 0.867.00 ± 0.994.70 ± 0.573.70 ± 0.852.60 ± 0.79
Time0.04 ± 0.011.22 ± 0.060.02 ± 0.010.68 ± 0.060.08 ± 0.001.13 ± 0.001.27 ± 0.060.31 ± 0.028.35 ± 0.10
Time Rank2.00 ± 0.147.00 ± 0.321.30 ± 0.285.20 ± 0.192.80 ± 0.136.40 ± 0.257.40 ± 0.153.90 ± 0.099.00 ± 0.00
Dim 2MAE162 ± 98170 ± 98511 ± 291320 ± 140240 ± 1055297 ± 3254164 ± 99162 ± 9946 ± 16
Rank6.11 ± 0.714.78 ± 0.835.56 ± 0.525.33 ± 0.524.22 ± 0.498.22 ± 0.734.78 ± 0.913.89 ± 0.672.11 ± 0.62
Appendix
Table 34: Score estimation performance via MAE ( ↓\downarrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets.
DimensionMetricPRODiGIPRODiGI-FTDiscoFormerDiscoFormer-FTNNKDENNKDE-FTTabPFNKDEGMM
Dim 1Cosine0.04 ± 0.030.17 ± 0.070.13 ± 0.050.04 ± 0.020.24 ± 0.070.37 ± 0.120.29 ± 0.100.49 ± 0.100.42 ± 0.11
Rank7.05 ± 0.506.05 ± 0.466.35 ± 0.367.35 ± 0.324.70 ± 0.654.05 ± 0.854.90 ± 0.672.00 ± 0.602.55 ± 0.36
Time0.04 ± 0.011.22 ± 0.060.02 ± 0.010.68 ± 0.060.08 ± 0.001.13 ± 0.001.27 ± 0.060.31 ± 0.028.35 ± 0.10
Time Rank2.00 ± 0.147.00 ± 0.321.30 ± 0.285.20 ± 0.192.80 ± 0.136.40 ± 0.257.40 ± 0.153.90 ± 0.099.00 ± 0.00
Dim 2Cosine0.10 ± 0.040.28 ± 0.050.26 ± 0.080.25 ± 0.070.27 ± 0.070.37 ± 0.080.23 ± 0.060.35 ± 0.070.61 ± 0.10
Rank7.28 ± 0.654.67 ± 0.725.33 ± 0.725.50 ± 0.665.06 ± 0.864.67 ± 0.966.72 ± 0.643.67 ± 0.542.11 ± 0.55
Appendix
Table 35: Score estimation performance via cosine similarity between score vector estimates and ground-truth ( ↑\uparrow ) and rank, as well as runtime in seconds ( ↓\downarrow ) and runtime rank for varying data dimensions. Average rank per metric (lower is better) across methods over real-world test datasets.

Explore similar work

CardsList
Apr 19, 2026cs.CL

Probabilistic Programs of Thought

LLMs are widely used for code generation and mathematical reasoning tasks where they are required to generate structured output. They either need to reason about code, generate code for a given specification, or reason using programs of thought. The typical approach to code generation is to prompt the model and generate samples until an appropriate program is obtained. Within this process, sampling nn programs from the language model requires nn GPU compute-intensive generations which becomes prohibitively expensive for larger values of nn. In this work, we address this limitation by exposing the LLM's distribution within the generated programs themselves. We propose a novel test-time framework we dub probabilistic programs of thought to obtain more samples from the model with fewer LLM generations. Given a program generated by a model and the associated next-token probabilities, we build a probabilistic program that compactly represents exponentially many deterministic programs. Since performing probabilistic reasoning in this probabilistic program is much cheaper, our approach allows sampling new programs without any additional GPU compute and little CPU overhead. We instantiate our approach on benchmarks for code generation, code understanding and mathematical reasoning and report improvements in performance with fewer generations from the LLM.

Poorva Garg, Renato Lui Geh, Daniel Israel +3

University of California Los Angeles · Allen Institute for AI

Code GenerationLLM Reasoning StrategiesLarge Language Model GenerationThoughts
AskPDF
Sep 29, 2026cs.LG

Synthesis Without Training: An Inference-Only Pipeline for Tabular, Temporal, and Relational Synthetic Data

Synthetic data generation is dominated by the fit-then-sample paradigm: a generative model is trained on a private dataset and then sampled from. Despite its widespread adoption, this paradigm faces three challenges: (1) a new training run is required for every dataset; (2) different data modalities, such as single tables, time series, and relational databases, require task-specific models and feature engineering; and (3) the resulting model is opaque, making its behavior under data constraints difficult to inspect. We propose GENSCRIPT, an inference-only pipeline that eliminates model training. GENSCRIPT computes a deterministic statistical profile of the source data (column types, ranges, missingness, categories, correlations, etc.) and passes it--rather than raw rows--to a language model to infer field semantics and cross-column integrity constraints. A coding agent then compiles the profile and constraints into an executable, auditable sampler. This unified approach supports single-table, temporal, and relational data without task-specific modeling. Across four single-table benchmarks, GENSCRIPT builds generators in 2 minutes and samples 50k rows within 6 seconds, while remaining within a few points of leading methods in marginal fidelity. Notably, it is the only method that perfectly preserves a 1-to-1 mapping between columns in the Adult dataset. On a smart-building dataset, it produces conditional time series that more closely match the real distribution than two baselines and perfectly preserves primary- and foreign-key relationships in the corresponding relational database.

Zilong Zhao, Abdul Raheem, Jiayu Li +5

Betterdata AI · National University of Singapore · University of Illinois Urbana-Champaign +1

Synthetic Tabular DataSynthetic DataGenerative ModelsSynthesisTabular Data
AskPDF
Jul 29, 2026cs.LG

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

Alexi Gladstone, Heng Ji, Yilun Du

1UIUC · 2Harvard

Generative ModelsPretrainingDeep LearningScaling
AskPDF