Latent State Utilization in Probabilistic Circuits

Zilei Shao; Anji Liu

cs.LGOct 16, 2026

Latent State Utilization in Probabilistic Circuits

Authors: Zilei Shao, Anji Liu

Organizations: University of California, Los Angeles · School of Computing, National University of Singapore

Abstract

Probabilistic circuits (PCs) are generative models that support efficient probabilistic inference. Adding latent states to a PC increases its representational capacity, but yields diminishing gains in likelihood. We use mutual information to examine how the dependence represented by a trained PC changes with latent size. Experiments with Hidden Chow-Liu Trees (HCLTs) on ImageNet32 show that learned dependence grows slowly and remains well below both the latent-state upper bound and estimates of dependence in the data. To understand how learning uses these states, we examine how strongly each sample favors particular latent states and how broadly states are used across the dataset. Tracking these measurements during HCLT training distinguishes rapid early posterior concentration from the more gradual stabilization of sample-dependent state preferences. Early measurements also guide initialization selection in HCLTs and hidden Markov models (HMMs), as well as tree selection in HCLTs. The resulting choices improve the ultimate performance across the tested latent sizes, providing a practical benefit from our study of latent-state utilization.

Figures & tables

Figure 1: Materializing a latent variable. (a) A sum node mixes two components, each factorized over

X

and

Y

. (b) Indicator inputs

\mathbb{I}[Z=1]

and

\mathbb{I}[Z=2]

make the selected component explicit. Marginalizing

Z

recovers the original circuit.

Figure 2: Increasing latent size yields diminishing likelihood gains and lower MI capacity utilization. HCLTs on ImageNet32 share a tree topology, initialization scheme, and 180-epoch Anemone training recipe. (a) PC validation log-likelihood in nats per image. (b) Mean learned MI (orange), the PC MI upper bound

\log_{2}h

(gray dashed), and the mean neural InfoNCE lower-bound estimate (blue dash-dot), in bits. The neural estimate uses

N_{c}=4096

and is independent of

h

. (c) MI capacity utilization,

I_{p_{\boldsymbol{\theta}}}(\boldsymbol{X};\mathbf{Y})/\log_{2}h

. All MI measurements use the same set of non-root latent-induced partitions. Boxes show medians and quartiles, with full-range whiskers.

Panel of Figure 3 — Figure 3: Posterior concentration and aggregate state usage during Anemone training ( Unif-pert(16) , $h=256$ ), evaluated on a fixed set of samples. Blue lines mark medians and dashed lines indicate the maximum entropy attained by uniform state probabilities.

Panel of Figure 4 — Figure 4: Stabilization of sample-dependent state preferences during Anemone training ( Unif-pert(16) , $h=256$ ). Both panels follow the same latent variable and sample set. Cell values are percentages.

Baseline tree ( $t_{0}$ )
	Latent states $h$
Initialization	16	32	64	128	256	512	1024
Equal weights	$-2753.29$	$-2656.26$	$-2589.63$	$-2542.23$	$-2506.22$	$-2480.20$	$-2467.50$
Unif-pert( $64$ )	$-2750.22$	$-2654.17$	$-2588.84$	$-2540.60$	$-2504.24$	$-2477.87$	$-2462.21$
Selected tree ( $t_{1}$ )
Equal weights	$-2749.93$	$-2652.63$	$-2585.93$	$-2538.00$	$-2501.83$	$-2475.86$	$-2464.01$

Table 1: Tree and initialization selection improve likelihood. Final validation LL (nats/image) for Anemone-trained HCLTs. Details about initialization are in Appendix A.3 . Tree gains compare matching initializations across trees. LVD and initialization gains are relative to equal weights on

t_{1}

, and combined gains to equal weights on

t_{0}

. Gains are calculated before rounding. Bold and underline mark the best and second-best.

Figure 5: Early flow entropy guides initialization selection. Flow entropy after two epochs is compared with final validation likelihood for HCLTs and epoch-200 likelihood for HMMs. Open markers fail the entropy-difference gate, and blue rings mark selected candidates. Boxes report Spearman correlation across each full pool and the selected candidate’s likelihood shortfall from the best candidate.

Figure 6: Initial flow entropy selects the best tree in this pool. Ten HCLT trees (

h=512

) share Unif-pert(

64

). Likelihood is averaged over the last three evaluations of 180-epoch Anemone training. The blue ring marks the selected tree. The box reports Spearman correlation and the selection gap.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: An HCLT and its latent-induced variable partition. (a) The circuit sums over latent states, with

h=2

. Dashed groups identify the state-specific components of each latent variable. (b) In the corresponding latent tree, conditioning on

Z_{1}

separates

\boldsymbol{X}=(B,C)

from

\mathbf{Y}=(A,D,E)

. Terminal latent variables have been absorbed into the input distributions.

Model / configuration	Optimizer	Bpd $\downarrow$
QT-CP-512, PC	Adam	$5.286\pm 0.001$
QG-CP-512, PC	Adam	$5.255\pm 0.001$
QT-CP-512, QPC	Adam	$5.196\pm 0.007$
QG-CP-512, QPC	Adam	$5.085\pm 0.001$
Shared HCLT patch product, $h=512$
$t_{0}$ + Unif-pert(64)	Anemone	$4.6482$

Appendix

Table 2: Full-image likelihood with training-side model selection. Lower bpd is better. Published width-512 results are from Gala et al. (2024) , Table D.5, using lossless preprocessing, with five-run means and uncertainties reproduced as reported. Gains are positive reductions in bpd from tree selection, subsequent initialization selection, and both together, calculated before rounding.

Figure 8: The dependence gap is larger for larger scopes. Points compare neural InfoNCE lower-bound estimates with learned HCLT MI across 563 subtree-versus-complement partitions on ImageNet32. Points are horizontally jittered, and lines connect medians in logarithmic scope-size bins. HCLTs are trained for 180 Anemone epochs. Dashed lines mark

\log_{2}h

, and the dotted line marks the

12

-bit InfoNCE ceiling (

N_{c}=4096

Panel of Figure 9 — Figure 9: Initialization changes the early entropy trajectories. HCLTs with $h=256$ are trained with Anemone. In each panel, the upper row uses uniform-noise perturbation with $\lambda=16$ , and the lower row uses LVD. Blue lines mark medians, dashed lines mark $\ln h$ , and annotations report validation log-likelihood in nats per image.

Panel of Figure 10 — Figure 10: Posterior reorganization and later stabilization under Dirichlet initialization. Results are for an HCLT with $h=256$ trained with Anemone using Dirichlet-initialized sum weights.

Figure 11: Aggregate usage first concentrates and then spreads across more states. The latent variable is the one used in the main-text flow-entropy histograms, with

h=256

and perturbation strength

\lambda=16

. States are sorted independently at each checkpoint by their average posterior probability on the probe set. The vertical axis shows

hF_{\phi}^{\mathcal{D}}(n_{z})

, so the dashed line at one denotes equal usage. Annotations report the effective number of states,

\exp(\widehat{H}_{\mathrm{usage}})

Figure 12: Direct entropy-difference ranking gives the same initialization selections as the main-text rule. The difference

\widehat{H}_{\mathrm{usage}}-\widehat{H}_{\mathrm{flow}}

is measured after two epochs. Panels show 19 HCLT initializations with

h=64

and 13 HMM initialization settings with

h=512

. Colors and shapes identify initialization families, and blue rings mark the largest difference. Boxes report Spearman correlation and the likelihood shortfall from the best candidate in the pool. HMM likelihoods are evaluated at epoch

200

, and statistics include all candidates.

Figure 13: The entropy difference and flow entropy give the same tree ranking in this pool. Initial

\widehat{H}_{\mathrm{usage}}-\widehat{H}_{\mathrm{flow}}

is compared with validation log-likelihood for ten HCLT trees with

h=512

and the same parameter initialization. Likelihood is averaged over the last three evaluations of 180-epoch Anemone training. The blue ring marks the largest difference, which selects the best-performing tree. The box reports Spearman correlation and the selection gap in nats per image.

Figure 14: Early validation likelihood can misrank initialization strategies. For the same 19 HCLT initializations with

h=64

, panels compare epoch-two entropy difference and validation likelihood with final likelihood after Anemone training. Solid blue and dashed orange rings identify the candidates selected by the entropy difference and early likelihood in both panels. Boxes report Spearman correlation and the selection gap in nats per image.

Figure 15: Initial flow entropy selects a better tree than initial likelihood in the additional

h=64

pool. Panels compare flow entropy and validation likelihood before training with final likelihood for 15 candidate trees under the same parameter initialization. Colors and shapes distinguish the two sum-node-group counts. Rings mark the minimum-flow-entropy and maximum-likelihood selections in both panels. Boxes report Spearman correlation and the selection gap in nats per image.

Probabilistic Circuits

Probabilistic circuits are structured generative models that enable tractable probabilistic inference while balancing expressivity and computational cost.

See all 18 papers

Probabilistic Circuits as Reasoning Machines in Artificial Intelligence (Part I)
Aug 17, 2026Robert PeharzProbabilistic Circuits Probabilistic Graphical Models
Institute of Machine Learning and Neural Computation Graz University of Technology
The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models
May 13, 2026Zhiyu Zhao, Xuejie Liu, Muhan Zhang +1Probabilistic Circuits Autoregressive Language Modeling
School of Computing, National University of Singapore · School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Building Expressive and Tractable Probabilistic Generative Models: A Review
Feb 1, 2024Sahil Sidheekh, Sriraam NatarajanProbabilistic Circuits Generative Modeling
The University of Texas at Dallas

Aug 17, 2026J/K move · Enter open · S save

Probabilistic Circuits as Reasoning Machines in Artificial Intelligence (Part I)

Robert Peharz

Institute of Machine Learning and Neural Computation Graz University of Technology

Probabilistic Circuits Probabilistic Graphical Models

Open paper AskPDF arXiv

This cumulative habilitation thesis studies probabilistic circuits (PCs) as a powerful and tractable framework for reasoning and learning under uncertainty in artificial intelligence (AI). It first advocates for probability as a core language for AI, emphasizing its connections to logic and information theory; the conceptual simplicity of probabilistic reasoning---based primarily on the sum and product rules; the parallels between probabilistic inference and human cognition; and the role of probability in optimal decision making. However, probability also faces significant computational challenges, as probabilistic inference is NP-hard in almost all probabilistic models. PCs address these challenges through structural constraints that ensure exact computation of a wide range of inference queries in polynomial time, such as marginals, conditionals, most probable explanations, expectations, and more advanced inference tasks. This thesis synthesizes a decade of research across foundations, algorithmic developments, and empirical validation of PCs. Key contributions highlighted in this work are foundational theory of PCs, Bayesian approaches for learning PCs, scalable implementations and integration with deep learning, hybrid models that combine PCs with intractable models, and connections with symbolic machine learning paradigms. This is the first part of my Habilitation Thesis. The second part is omitted, as it comprises the cumulative part of the thesis and has been published at various venues (see Chapter 5).

Abstract

Probabilistic Circuits

Explore similar work

Probabilistic Circuits as Reasoning Machines in Artificial Intelligence (Part I)

The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models

Building Expressive and Tractable Probabilistic Generative Models: A Review