Probabilistic circuits (PCs) are generative models that support efficient probabilistic inference. Adding latent states to a PC increases its representational capacity, but yields diminishing gains in likelihood. We use mutual information to examine how the dependence represented by a trained PC changes with latent size. Experiments with Hidden Chow-Liu Trees (HCLTs) on ImageNet32 show that learned dependence grows slowly and remains well below both the latent-state upper bound and estimates of dependence in the data. To understand how learning uses these states, we examine how strongly each sample favors particular latent states and how broadly states are used across the dataset. Tracking these measurements during HCLT training distinguishes rapid early posterior concentration from the more gradual stabilization of sample-dependent state preferences. Early measurements also guide initialization selection in HCLTs and hidden Markov models (HMMs), as well as tree selection in HCLTs. The resulting choices improve the ultimate performance across the tested latent sizes, providing a practical benefit from our study of latent-state utilization.
Figures & tables
Figure 1: Materializing a latent variable. (a) A sum node mixes two components, each factorized over X and Y . (b) Indicator inputs I[Z=1] and I[Z=2] make the selected component explicit. Marginalizing Z recovers the original circuit.
Figure 2: Increasing latent size yields diminishing likelihood gains and lower MI capacity utilization. HCLTs on ImageNet32 share a tree topology, initialization scheme, and 180-epoch Anemone training recipe. (a) PC validation log-likelihood in nats per image. (b) Mean learned MI (orange), the PC MI upper bound log2h (gray dashed), and the mean neural InfoNCE lower-bound estimate (blue dash-dot), in bits. The neural estimate uses Nc=4096 and is independent of h . (c) MI capacity utilization, Ipθ(X;Y)/log2h . All MI measurements use the same set of non-root latent-induced partitions. Boxes show medians and quartiles, with full-range whiskers.
Figure 3: Posterior concentration and aggregate state usage during Anemone training ( Unif-pert(16) , h=256 ), evaluated on a fixed set of samples. Blue lines mark medians and dashed lines indicate the maximum entropy attained by uniform state probabilities.
Figure 4: Stabilization of sample-dependent state preferences during Anemone training ( Unif-pert(16) , h=256 ). Both panels follow the same latent variable and sample set. Cell values are percentages.
Latent states h
Initialization
16
32
64
128
256
512
1024
Baseline tree ( t0 )
Equal weights
−2753.29
−2656.26
−2589.63
−2542.23
−2506.22
−2480.20
−2467.50
Unif-pert( 64 )
−2750.22
−2654.17
−2588.84
−2540.60
−2504.24
−2477.87
−2462.21
Selected tree ( t1 )
Equal weights
−2749.93
−2652.63
−2585.93
−2538.00
−2501.83
−2475.86
−2464.01
Table 1: Tree and initialization selection improve likelihood. Final validation LL (nats/image) for Anemone-trained HCLTs. Details about initialization are in Appendix A.3 . Tree gains compare matching initializations across trees. LVD and initialization gains are relative to equal weights on t1 , and combined gains to equal weights on t0 . Gains are calculated before rounding. Bold and underline mark the best and second-best.
Figure 5: Early flow entropy guides initialization selection. Flow entropy after two epochs is compared with final validation likelihood for HCLTs and epoch-200 likelihood for HMMs. Open markers fail the entropy-difference gate, and blue rings mark selected candidates. Boxes report Spearman correlation across each full pool and the selected candidate’s likelihood shortfall from the best candidate.
Figure 6: Initial flow entropy selects the best tree in this pool. Ten HCLT trees ( h=512 ) share Unif-pert( 64 ). Likelihood is averaged over the last three evaluations of 180-epoch Anemone training. The blue ring marks the selected tree. The box reports Spearman correlation and the selection gap.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: An HCLT and its latent-induced variable partition. (a) The circuit sums over latent states, with h=2 . Dashed groups identify the state-specific components of each latent variable. (b) In the corresponding latent tree, conditioning on Z1 separates X=(B,C) from Y=(A,D,E) . Terminal latent variables have been absorbed into the input distributions.
Model / configuration
Optimizer
Bpd ↓
QT-CP-512, PC
Adam
5.286±0.001
QG-CP-512, PC
Adam
5.255±0.001
QT-CP-512, QPC
Adam
5.196±0.007
QG-CP-512, QPC
Adam
5.085±0.001
Shared HCLT patch product, h=512
t0 + Unif-pert(64)
Anemone
4.6482
Appendix
Table 2: Full-image likelihood with training-side model selection. Lower bpd is better. Published width-512 results are from Gala et al. (2024) , Table D.5, using lossless preprocessing, with five-run means and uncertainties reproduced as reported. Gains are positive reductions in bpd from tree selection, subsequent initialization selection, and both together, calculated before rounding.
Figure 8: The dependence gap is larger for larger scopes. Points compare neural InfoNCE lower-bound estimates with learned HCLT MI across 563 subtree-versus-complement partitions on ImageNet32. Points are horizontally jittered, and lines connect medians in logarithmic scope-size bins. HCLTs are trained for 180 Anemone epochs. Dashed lines mark log2h , and the dotted line marks the 12 -bit InfoNCE ceiling ( Nc=4096 ).
Figure 9: Initialization changes the early entropy trajectories. HCLTs with h=256 are trained with Anemone. In each panel, the upper row uses uniform-noise perturbation with λ=16 , and the lower row uses LVD. Blue lines mark medians, dashed lines mark lnh , and annotations report validation log-likelihood in nats per image.
Figure 10: Posterior reorganization and later stabilization under Dirichlet initialization. Results are for an HCLT with h=256 trained with Anemone using Dirichlet-initialized sum weights.
Figure 11: Aggregate usage first concentrates and then spreads across more states. The latent variable is the one used in the main-text flow-entropy histograms, with h=256 and perturbation strength λ=16 . States are sorted independently at each checkpoint by their average posterior probability on the probe set. The vertical axis shows hFϕD(nz) , so the dashed line at one denotes equal usage. Annotations report the effective number of states, exp(Husage) .
Figure 12: Direct entropy-difference ranking gives the same initialization selections as the main-text rule. The difference Husage−Hflow is measured after two epochs. Panels show 19 HCLT initializations with h=64 and 13 HMM initialization settings with h=512 . Colors and shapes identify initialization families, and blue rings mark the largest difference. Boxes report Spearman correlation and the likelihood shortfall from the best candidate in the pool. HMM likelihoods are evaluated at epoch 200 , and statistics include all candidates.
Figure 13: The entropy difference and flow entropy give the same tree ranking in this pool. Initial Husage−Hflow is compared with validation log-likelihood for ten HCLT trees with h=512 and the same parameter initialization. Likelihood is averaged over the last three evaluations of 180-epoch Anemone training. The blue ring marks the largest difference, which selects the best-performing tree. The box reports Spearman correlation and the selection gap in nats per image.
Figure 14: Early validation likelihood can misrank initialization strategies. For the same 19 HCLT initializations with h=64 , panels compare epoch-two entropy difference and validation likelihood with final likelihood after Anemone training. Solid blue and dashed orange rings identify the candidates selected by the entropy difference and early likelihood in both panels. Boxes report Spearman correlation and the selection gap in nats per image.
Figure 15: Initial flow entropy selects a better tree than initial likelihood in the additional h=64 pool. Panels compare flow entropy and validation likelihood before training with final likelihood for 15 candidate trees under the same parameter initialization. Colors and shapes distinguish the two sum-node-group counts. Rings mark the minimum-flow-entropy and maximum-likelihood selections in both panels. Boxes report Spearman correlation and the selection gap in nats per image.
Probabilistic circuits are structured generative models that enable tractable probabilistic inference while balancing expressivity and computational cost.
This cumulative habilitation thesis studies probabilistic circuits (PCs) as a powerful and tractable framework for reasoning and learning under uncertainty in artificial intelligence (AI). It first advocates for probability as a core language for AI, emphasizing its connections to logic and information theory; the conceptual simplicity of probabilistic reasoning---based primarily on the sum and product rules; the parallels between probabilistic inference and human cognition; and the role of probability in optimal decision making. However, probability also faces significant computational challenges, as probabilistic inference is NP-hard in almost all probabilistic models. PCs address these challenges through structural constraints that ensure exact computation of a wide range of inference queries in polynomial time, such as marginals, conditionals, most probable explanations, expectations, and more advanced inference tasks. This thesis synthesizes a decade of research across foundations, algorithmic developments, and empirical validation of PCs. Key contributions highlighted in this work are foundational theory of PCs, Bayesian approaches for learning PCs, scalable implementations and integration with deep learning, hybrid models that combine PCs with intractable models, and connections with symbolic machine learning paradigms. This is the first part of my Habilitation Thesis. The second part is omitted, as it comprises the cumulative part of the thesis and has been published at various venues (see Chapter 5).
Robert Peharz
Institute of Machine Learning and Neural Computation Graz University of Technology
Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.
Zhiyu Zhao, Xuejie Liu, Muhan Zhang +1
School of Computing, National University of Singapore · School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.