Defensive Generation

Gabriele Farina; Juan Carlos Perdomo

cs.LGFeb 24, 2026

Defensive Generation

Authors: Gabriele Farina, Juan Carlos Perdomo

Organizations: MIT · NYU

Abstract

We study the problem of efficiently producing, in an online fashion, generative models of scalar, multiclass, and vector-valued outcomes that cannot be falsified on the basis of the observed data and a pre-specified collection of computational tests. Our contributions are twofold. First, we expand on connections between online high-dimensional multicalibration with respect to an RKHS and recent advances in expected variational inequality problems, enabling efficient algorithms for the former. We then apply this algorithmic machinery to the problem of outcome indistinguishability. Our procedure, Defensive Generation, is the first to efficiently produce online outcome indistinguishable generative models of non-Bernoulli outcomes that are unfalsifiable with respect to infinite classes of tests, including those that examine higher-order moments of the generated distributions. Furthermore, our method runs in near-linear time in the number of samples and achieves the optimal, vanishing T^{-1/2} rate for generation error.

Figures & tables

\displaystyle\quad S_{t}(p)

\displaystyle=\Phi(x_{t},p)^{\top}\sum_{\tau=1}^{t-1}\mathbb{E}_{p_{\tau}\sim\mathcal{D}_{\tau}}\Phi(x_{\tau},p_{\tau})(z_{\tau}-p_{\tau})=\!\sum_{\tau=1}^{t-1}\mathbb{E}_{p_{\tau}\sim\mathcal{D}_{\tau}}\Gamma((x_{t},p),(x_{\tau},p_{\tau}))(z_{\tau}-p_{\tau})

Algorithm 1 Efficient High-Dimensional Multicalibration via Expected VIs

y_{i}^{+}=v+t_{i}^{+}u_{i},\;y_{i}^{-}=v+t_{i}^{-}u_{i}\ \quad\text{ and }\quad\lambda_{i}^{+}=\frac{\sigma_{i}}{t_{i}^{+}(t_{i}^{+}-t_{i}^{-})},\lambda_{i}^{-}=\frac{\sigma_{i}}{t_{i}^{-}(t_{i}^{-}-t_{i}^{+})}

Algorithm 3 Backfitting a probability measure

\mu

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 493 papers

A Theoretical Framework for Statistical Evaluability of Generative Models
Apr 7, 2026Shashaank Aiyer, Yishay Mansour, Shay Moran +1Generative AI Evaluation Statistical Learning Theory
University of Maryland · Tel Aviv University and Google Research · Technion and Google Research
Counterfactual Generation via Flow Matching: Coupling-Sensitive End-to-End Rates
Oct 1, 2026Yunrui Guan, Krishnakumar Balasubramanian, Shiva Prasad KasiviswanathanFlow Matching Causal Counterfactual Generation
Department of Applied Mathematics and Statistics, Johns Hopkins University. · Amazon · Department of Statistics, University of California, Davis.
Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM
Apr 28, 2026Vinith M. Suriyakumar, Ayush Sekhari, Lena Stempfle +5LLM Auditing Language Model Probing
MIT · BU · Thorn

Apr 7, 2026J/K move · Enter open · S save

A Theoretical Framework for Statistical Evaluability of Generative Models

Shashaank Aiyer, Yishay Mansour, Shay Moran, Han Shao

University of Maryland · Tel Aviv University and Google Research · Technion and Google Research

Generative AI Evaluation Statistical Learning Theory Integral Probability Metrics

Open paper AskPDF arXiv

Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d. test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and Rényi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.

Abstract

Generative Modeling

Explore similar work

A Theoretical Framework for Statistical Evaluability of Generative Models

Counterfactual Generation via Flow Matching: Coupling-Sensitive End-to-End Rates

Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM