We introduce Exponential Family Discriminant Analysis (EFDA), a unified generative framework that extends classical Linear Discriminant Analysis (LDA) beyond the Gaussian setting to any member of the exponential family. Under the assumption that each class-conditional density belongs to a common exponential family, EFDA derives closed-form maximum-likelihood estimators for all natural parameters and yields a decision rule that is linear in the sufficient statistic, recovering LDA as a special case and capturing nonlinear decision boundaries in the original feature space. We prove that EFDA is asymptotically calibrated and statistically efficient under correct specification, and we generalise it to K≥2 classes and multivariate data. Through extensive simulation across five exponential-family distributions (Weibull, Gamma, Exponential, Poisson, Negative Binomial), EFDA matches the classification accuracy of LDA, QDA, and logistic regression while reducing Expected Calibration Error (ECE) by 2-6×, a gap that is structural: it persists for all n and across all class-imbalance levels, because misspecified models remain asymptotically miscalibrated. We further prove and empirically confirm that EFDA's log-odds estimator approaches the Cramér-Rao bound under correct specification, and is the only estimator in our comparison whose mean squared error converges to zero. Complete derivations are provided for nine distributions. Finally, we formally verify all four theoretical propositions in Lean 4, using Aristotle (Harmonic) and OpenGauss (Math, Inc.) as proof generators, with all outputs independently machine-checked by AXLE (Axiom).
Figure 1: ECE (%) by distribution and method ( n=1,000 , M=100 trials). EFDA achieves the lowest ECE in every distribution; QDA is dramatically miscalibrated on heavy-tailed data (Exponential, Gamma).
Figure 4: Left: mean variance of ℓ^(x0) (log x -axis, linear y ). All methods’ variances decay to zero; EFDA tracks the CR bound. Right: mean MSE (log-log), revealing the misspecification residual of Proposition 4 . LDA, LR, and QDA plateau as their variance vanishes but their squared bias remains; only EFDA’s MSE continues toward zero. ( M=1,000 trials, Weibull shape k′=3 , λ0=4 , λ1=2 ; x0 grid of 100 points sampled from both class-conditional distributions; N0=⌊n(1−α)⌋ , N1=⌊nα⌋ fixed per trial.)
Variance
MSE
n
EFDA
LR
LDA
QDA
CR
EFDA
LR
LDA
QDA
102
0.145
0.183
0.548
0.269
0.084
0.146
0.431
1.543
0.304
103
0.014
0.026
0.044
0.021
0.008
0.014
0.311
0.834
0.039
104
0.001
0.003
0.004
0.002
0.001
0.001
0.301
0.794
0.019
105
0.000
0.000
0.000
0.000
0.000
0.000
0.299
0.789
0.017
Table 4: Mean variance and MSE of ℓ^(x0) averaged over x0 grid ( M=1,000 trials, Weibull shape k′=3 , N0=⌊n(1−α)⌋ , N1=⌊nα⌋ fixed per trial). The CR bound is the theoretical minimum variance for correctly specified EFDA.
Proposition
Aristotle
OpenGauss
1. Consistency
✓
✓
2. Calibration
✓†
✓†
3. MLE Efficiency
✓‡
✓\lx@sectionsign
4. Asymptotic MSE
✓
✓
Table 5: AXLE-verified proof results for the four EFDA propositions. ✓ = sorry -free proof verified by AXLE.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Accuracy vs. n for EFDA (known and estimated k ), LDA, and LR on Weibull data with true k=3 ( M=100 trials). Estimating k incurs <1% accuracy loss and stabilises rapidly by n≈250 .
This tutorial explains Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) as two fundamental classification methods in statistical and probabilistic learning. We start with the optimization of decision boundary on which the posteriors are equal. Then, LDA and QDA are derived for binary and multiple classes. The estimation of parameters in LDA and QDA are also covered. Then, we explain how LDA and QDA are related to metric learning, kernel principal component analysis, Mahalanobis distance, logistic regression, Bayes optimal classifier, Gaussian naive Bayes, and likelihood ratio test. We also prove that LDA and Fisher discriminant analysis are equivalent. We finally clarify some of the theoretical concepts with simulations we provide.
Benyamin Ghojogh, Mark Crowley
Department of Electrical and Computer Engineering, Machine Learning Laboratory, University of Waterloo, Waterloo, ON, Canada
Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.
We develop exponential family synthetic controls (EFSC), a distributional version of synthetic controls for a panel of datasets. Each cell of the panel corresponds to a dataset drawn from an exponential family whose natural parameters factorize probabilistically across units and times. We estimate the latent factors using black-box variational inference. This replaces the usual weighted-average view of synthetic controls with a flexible probabilistic model that operates on full distributions. We propose causal estimands based on divergences between pre- and post-intervention distributions induced by the posterior of the natural parameters, together with distributional placebo tests to support causal inference and assess the significance of the estimated effects. We validate the proposed framework on synthetic and real data. Across a variety of exponential-family distributions, EFSC accurately recovers causal effects induced by exponential tilts, together with the corresponding divergences between treated and counterfactual distributions. The framework also captures effects induced by structural perturbations of the latent factors and by heavy-tailed noise contamination. Finally, we apply EFSC to study the expansion of Medicaid under the Affordable Care Act (ACA) and its impact on the distribution of health insurance coverage across U.S. states. Code is available at https://github.com/blei-lab/efsc.
Hector Rodriguez-Deniz, David M. Blei
Data Science Institute Columbia University New York, NY 10027, USA · Department of Computer Science and Department of Statistics Columbia University New York, NY 10027, USA