Parameter Estimation for Unnormalized Discrete Models via Empirically Localized Deformed Bregman Divergence

Takashi Takenouchi

stat.MLSep 25, 2026

Parameter Estimation for Unnormalized Discrete Models via Empirically Localized Deformed Bregman Divergence

Authors: Takashi Takenouchi

Organizations: National Graduate Institute for Policy Studies (GRIPS) and Riken AIP

Abstract

Estimation of parameter of probabilistic models is an important task in the field of machine learning.For models of discrete variables, calculation of the normalization constant of model is sometimes difficult and a lot of researches have been done to avoid the calculation of the normalization constant. In this paper, we tackle with the difficulty by combining a technique of empirical localization and a deformed Bregman divergence.The technique of empirical localization makes it possible to drastically reduce computational cost of the calculation of the normalization constant, and in addition, appropriate choice of the deformation for the Bregman divergence can invest the proposed estimator with various kinds of favorable statistical properties, such as efficiency or robustness against outlier noise.

Figures & tables

Figure 1: An information geometrical interpretation of the proposed estimator. Each point in the figure corresponds to a probability distribution. Note that the deformed Bregman divergence between two points on a gray curve is computationally tractable.

Panel of Figure 2 — Figure 2: Examples of Bayesian-like estimation for $c=0.2,0.5,1,1.5$ (upper left, upper right, lower left, lower right). A red point ( $\circ$ ), a green point ( $\triangle$ ), and a blue point ( $+$ ) are the empirical distribution, the function $\eta(x)$ representing prior information, and the proposed estimator, respectively. A black dashed line is a space of probability distribution on $\{+1,-1\}$ . A black line represents a manifold of Bernoulli distributions and a blue dashed line is the modified model $q_{\boldsymbol{\theta}}(x)-\eta(x)$ . A red dashed line is a feasible set constrained by $\eta(x)<q_{\boldsymbol{\theta}}(x)$ .

Figure 3: (a)Boxplot of MSEs of estimators in log-scale, against sample size

n

. (b)Boxplot of computational times of estimators in log-scale against sample size

n

$U$	Stat. ver. of $f$	Efficient ver. of $f$
$\exp(z)$	KL: $\log z$	KL: $\log z$
$z^{2}/2$	Hel: $z$	F-Hel: $2\sqrt{z}$
$-\log(-z)$	IS: $-1/z$	F-IS: $-\exp(2\sqrt{z})$

Table 1: Combination of

U

and

f

, and corresponded abbreviations.

Figure 4: (a)Boxplot of MSEs of estimators, in log-scale. Efficient version of

f

s are colored red. (b)Boxplot of computational times of estimators, in log-scale. Efficient version of

f

s are colored red.

Figure 5: (a)Boxplot of MSEs of MLE against

n

in log-scale, for datasets without outlier noise. (b)Boxplot of MSEs of proposed estimator against

n

in log-scale, for datasets without outlier noise.

Figure 6: (a)Boxplot of MSEs of MLE against

n

in log-scale, for datasets with outlier noise. (b)Boxplot of MSEs of proposed estimator against

n

in log-scale, for datasets with outlier noise.

Parameter Estimation

Parameter estimation studies how to infer unknown quantities in statistical, physical, and machine-learning models from observed data.

See all 144 papers

A Spectral Framework for Closed-Form Relative Density Estimation
May 11, 2026Francis BachDensity Ratio Estimation Spectral Methods
SIERRA · Inria - Ecole Normale Supérieure · PSL Research University
Uniform-in-time convergence bounds for Persistent Contrastive Divergence algorithms
Oct 2, 2025Paul Felix Valsecchi Oliva, O. Deniz Akyildiz, Andrew DuncanParameter Estimation Stochastic Approximation
Department of Mathematics, Imperial College London
Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation
May 9, 2026Zhijun Zeng, Yixuan Jiang, Pipi Hu +1Energy-Based Models Density Estimation
Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China

May 11, 2026J/K move · Enter open · S save

A Spectral Framework for Closed-Form Relative Density Estimation

Francis Bach

SIERRA · Inria - Ecole Normale Supérieure · PSL Research University

Density Ratio Estimation Spectral Methods Statistical Learning Theory Density Estimation Kullback-Leibler Divergence

Open paper AskPDF arXiv

We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models, including unnormalized and conditional models. This is achieved by representing the Kullback-Leibler (KL) divergence as an integral of weighted chi-squared divergences, converting KL estimation into a family of least-squares problems. We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features. The framework extends to a broad class of f-divergences and can be combined with kernelization or feature learning with neural networks. We prove convergence guarantees for the resulting estimators and empirically compare them on synthetic data with optimization-based variational formulations, including logistic and softmax regression for normalized conditional models.

Abstract

Parameter Estimation

Explore similar work

A Spectral Framework for Closed-Form Relative Density Estimation

Uniform-in-time convergence bounds for Persistent Contrastive Divergence algorithms

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation