Information Estimation with Discrete Diffusion

Alberto Foresti; Giulio Franzese; Pietro Michiardi

cs.LGOct 16, 2026

Information Estimation with Discrete Diffusion

Authors: Alberto Foresti, Giulio Franzese, Pietro Michiardi

Organizations: Department of Data Science, EURECOM, 450 Route des Chappes, 06410 Biot, France

Abstract

Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the "embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.

Figures & tables

	coh	con	flu	rel	ovr
info-sedd-c	0.209	0.740	0.679	0.411	0.568
info-sedd-j	-0.091	0.550	0.455	0.288	0.322
kl-dime	0.170	0.214	0.194	0.076	0.193
hd-dime	-0.243	0.331	0.281	-0.145	0.063
smile	-0.367	-0.074	-0.162	-0.149	-0.221

Table 2: Pearson (left) and Kendall’s Tau (right) correlation of human metrics and MI for info-sedd and selected competitors. The labels are shorthand notations for the human metrics: COH=coherence, CON=consistency, FLU=fluency, REL=relevance, OVR=overall. The reader is invited to check Section C.3 for correlations with other metrics offered by the SummEval framework.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Estimator	info-sedd	gan-dime	hd-dime	kl-dime	minde	mine	nwj	smile
MI=10, D=10	346.37	645.81	645.81	645.81	440.51	645.81	645.81	645.81
MI=20, D=20	687.22	1301.98	1301.98	1301.98	876.44	1301.98	1301.98	1301.98
MI=30, D=30	1005.66	1913.15	1913.15	1913.15	1286.35	1913.15	1913.15	1913.15
MI=40, D=40	1345.94	2564.33	2564.33	2564.33	1726.25	2564.32	2564.32	2564.32
MI=50, D=50	1666.30	3180.50	3180.50	3180.50	2138.41	3180.49	3180.49	3180.49

Appendix

Table 3: Average maximum memory usage over one epoch for different estimators over different MI estimation tasks. Results are in MB. Lowest memory usages are written in bold.

Estimator	info-sedd	gan-dime	hd-dime	kl-dime	minde	mine	nwj	smile
MI=10, D=10	2.40	2.89	2.90	2.86	2.16	2.89	2.96	2.94
MI=20, D=20	3.34	3.99	4.02	3.98	2.62	4.03	4.03	4.00
MI=30, D=30	4.51	5.56	5.54	5.57	3.37	5.57	5.57	5.61
MI=40, D=40	5.96	7.34	7.33	7.36	4.00	7.39	7.42	7.37
MI=50, D=50	7.06	8.71	8.69	8.71	4.68	8.76	8.73	8.72

Appendix

Table 4: Average runtime for one epoch for different estimators over different MI estimation tasks. Results are in seconds. Best runtimes are written in bold.

Figure 6: Empirical convergence of the benchmarked MI estimators.

N	$10^{2}$	$10^{3}$	$10^{4}$	$10^{5}$
MI=10, D=10	7.472	11.139	10.100	10.115
MI=20, D=20	7.772	22.533	20.728	19.922
MI=30, D=30	26.126	29.458	36.093	29.809
MI=40, D=40	28.517	38.512	35.450	38.758
MI=50, D=50	27.782	50.300	43.750	48.822

Appendix

Table 6: Empirical evaluation of sample complexity for info-sedd .

Method	Estimated MI
info-sedd	0.43
hd-dime	0.43
smile	0.67
gan-dime	0.51

Appendix

Table 8: Extreme dimensionality experiment with

D=256

and ground-truth MI

=0.5

Method	Estimated MI
info-sedd	0.53
hd-dime	0.00
smile	0.00
gan-dime	0.00

Appendix

Table 9: Extreme dimensionality experiment with

D=1024

and ground-truth MI

=0.5

Parameter	Value
Backbone	MDLM small [ Sahoo et al., 2024 ]
Optimizer	AdamW
AdamW - $\beta_{1}$	0.9
AdamW - $\beta_{2}$	0.999
Weight decay	0
Learning rate	3e-4

Appendix

Table 10: Training settings for one run for the model selection experiments and the consistency tests (one value of

\rho)

	gan-dime	hd-dime	info-sedd-c	info-sedd-j	kl-dime	minde	mine	nwj	smile
$\rho$
0.0	-	0.09	7.52	106.69	-0.01	365679296.00	-0.50	-0.01	0.00
0.1	-	0.20	34.02	157.90	0.01	198489552.00	-0.02	0.00	0.00
0.2	-	-0.01	61.25	197.18	0.48	347922112.00	0.02	-0.10	0.09
0.3	-	0.13	90.17	220.47	3.03	12828973.00	0.00	-0.24	0.89
0.4	-	2.41	124.48	244.53	3.21	33884712.00	0.84	-	2.06

Appendix

Table 11: Numerical results for Text summarization consistency tests

Figure 7: Comparison of human metrics against mutual information estimated by info-sedd - c

Figure 8: Comparison of human metrics against mutual information estimated by info-sedd -J

Figure 9: Kendall Tau correlation heatmap of MI estimates info-sedd , other text summarization metrics and human metrics. CID=Cider, BLE=bleu.

Figure 10: Pearson correlation heatmap of MI estimates info-sedd , other text summarization metrics and human metrics. CID=Cider, BLE=bleu.

Parameter	Value
Backbone	Caduceus-1K [ Schiff et al., 2024 ]
Optimizer	AdamW
AdamW - $\beta_{1}$	0.9
AdamW - $\beta_{2}$	0.999
Weight decay	0.1
Learning rate	0.001

Appendix

Table 12: Training settings for one run for the promoter experiments or the consistency tests (one value of

\rho)

	gan-dime	hd-dime	info-sedd-c	info-sedd-j	kl-dime	mine	nwj	smile
$\rho$
0.0	0.01	0.00	0.00	0.05	0.00	0.00	-0.02	0.00
0.1	0.00	0.01	0.00	0.07	0.00	0.00	-0.01	0.00
0.2	0.02	-0.01	0.00	0.06	0.00	0.00	-0.02	0.02
0.3	0.04	0.01	0.03	0.08	0.00	0.00	-0.01	0.02
0.4	0.00	0.07	0.07	0.06	-0.01	0.00	0.00	0.07

Appendix

Table 13: Numerical results for DNA consistency tests

\Delta E=2\,\sigma_{i}\sum_{j}J_{i,j}\,\sigma_{j}

Appendix

Algorithm 4 Metropolis Algorithm for 2D Ising Spin Glass

Figure 11: Ising model entropy estimates

Setting	Estimated MI
MI=10, D=10	11.56
MI=20, D=20	9.33
MI=30, D=30	10.55
MI=40, D=40	2.87
MI=50, D=50	4.18

Appendix

Table 14: Plug-in estimator ( Gao et al. [2017] ) on high-MI synthetic benchmarks with

D=\mathrm{MI}

MI	D	Estimated MI
0.5	256	14.17
0.5	1024	14.30

Appendix

Table 15: Plug-in estimator ( Gao et al. [2017] ) in extreme dimensionality regimes with ground-truth MI

=0.5

Mutual Information Estimation

Mutual information estimation measures statistical dependence between variables from data, supporting analysis, prediction, and information-aware learning.

See all 20 papers

Minimax Optimality of Score-Entropy Discrete Diffusion
Aug 21, 2026Cholyeon Cho, Yuchen WuDiffusion Models Minimax Estimation
Department of Computer Science, The University of Texas at Austin · School of Operations Research and Information Engineering, Cornell University
NMINE: Normalized Mutual Information Neural Estimation
Jul 30, 2026Petra Eerikinharju, Marko Tuononen, Ville HautamäkiMutual Information Estimation
University of Eastern Finland
DIPHINE: Diffusion-based $Φ$ -ID Neural Estimator
Jun 17, 2026Simon Pedro Galeano Munoz, Mustapha Bounoua, Giulio Franzese +2Mutual Information Estimation
KAUST, Saudi Arabia · EURECOM, France

Aug 21, 2026J/K move · Enter open · S save

Minimax Optimality of Score-Entropy Discrete Diffusion

Cholyeon Cho, Yuchen Wu

Department of Computer Science, The University of Texas at Austin · School of Operations Research and Information Engineering, Cornell University

Diffusion Models Minimax Estimation Masked Diffusion Models

Open paper AskPDF arXiv

Discrete diffusion models have demonstrated strong performance across a range of datasets, including natural language data and graph-structured data. Among many variants, score-entropy discrete diffusion (SEDD) has achieved particularly strong empirical results. In SEDD, new samples are generated by iteratively evaluating a sequence of concrete score functions, which are learned by minimizing a score-entropy loss. While much of the prior theoretical literature on discrete diffusion has focused on the sampling efficiency of SEDD under the assumption of small score estimation error, recent work has begun to investigate the finite-sample properties of score estimation itself. In this work, we take a different route by investigating the fundamental statistical limits of concrete score estimation. We focus on uniform and masking discrete diffusions, two of the most widely adopted discrete diffusion models. We establish a minimax lower bound under the score-entropy loss, and propose an MLE-based thresholding estimator that matches this lower bound up to constant and polylogarithmic factors that depend on neighboring density ratios. We further show that, for any target distribution, this density ratio is naturally controlled under both uniform and masking discrete diffusion models, yielding nearly matching minimax lower and upper bounds for the aggregated score estimation error. Our results imply that, with appropriate initialization and discretization, SEDD can achieve nearly optimal minimax sample complexity, as measured by the KL divergence between the target and generated distributions.

Abstract

Mutual Information Estimation

Explore similar work

Minimax Optimality of Score-Entropy Discrete Diffusion

NMINE: Normalized Mutual Information Neural Estimation

DIPHINE: Diffusion-based ΦΦΦ-ID Neural Estimator

DIPHINE: Diffusion-based $Φ$ -ID Neural Estimator