Probabilistic circuits (PCs) have emerged as generative models guaranteeing tractable probabilistic inference. However, tractability comes at the price of larger model sizes than in other generative models. As such, to scale PCs to high-dimensional data such as images, previous work has devised several parameter sharing techniques as well as fast evaluation protocols that consider learning PCs on independent image patches. In this work, we revisit this practice to highlight a surprising learnability issue: models that assume independence counterintuitively can yield better likelihoods than models that do not. To break this apparent bottleneck and to scale PCs further, we introduce a recursive parameter sharing construction (RePA) that fosters efficiency without assuming independence. We show empirically that RePA can greatly reduce time and memory consumption while yielding competitive likelihoods. Inspired by these results, we further generalize the approach to data modalities that do not inherently exhibit patch-structure (NoPA), such as tabular data, showing similar efficiency improvements, with marginal changes in likelihoods and sample quality on seven tabular datasets. Code available at https://github.com/april-tools/RePA.
Figures & tables
Figure 1: RePA delivers faster, smaller and more accurate circuits compared to PCs that assume patch independence ( IPA ) . This is shown here in terms of average step time (seconds), GPU memory, FLOPs and bpd (i.e. normalized negative log-likelihoods) on ImageNet32 for several tensorized PC architectures. RePA markers (yellow) that have no corresponding IPA markers (blue) are those for which IPA could not be trained on 40GB GPUs in under 2 days. See Fig. 14 for similar trends on CelebA and ImageNet64 and the precise PC configuration associated to each marker.
Figure 2: Example of a RG and a PC for a 2×2 patch of an 8×8 image ( 2(a) ) where the RG is a binary quad tree ( 2(b) ) resulting in the overparameterized ( K=3 ) PC with CP-T layers ( 2(c) ). Abstracting away the units, we obtain a layer notation with input, Hadamard and sum layers ( 2(d) ). RG regions and partitions are mapped to the corresponding sum and product layers in the PC by color.
Figure 3: Training time for different RGs with K=512 on images of increasing size.
Figure 4: IPA PC c′ built on 2×2 independent patches ( L=16 ), where the same circuit c (e.g., like in Fig. 2 ) is shared across all patches. The final likelihood is obtained by multiplying all values together ( Eq. 1 ).
Figure 5: RePA PCs improve over IPA and FullPC while greatly alleviating the independent patch conundrum as measured in bits-per-dimension (bpd), for increasing patch sizes and QT RGs on ImageNet32 (left), ImageNet64 (center), and CelebA (right). More results in Fig. 15 .
Figure 6: RePA recursively splits the image into a hierarchy of patches. The patches at each layer i of the hierarchy (color-coded) share the same circuit ci , which is rescoped according to each patch it is being evaluated on.
Figure 7: NoPA shares parameters across the topological depth . The layers of an unfolded circuit (left) can be grouped by their topological ordering (center) to fold the circuit, making it more amenable to GPU accelerators. NoPA exploits this grouping by sharing parameters across sum layers at the same topological depth (right). Each color represents a different set of parameters, we highlight in red the three sum layers that share the same weight tensor, we use grey for layers without parameters.
Figure 8: Our NoPA ( Section 5 ) setting on the Adult dataset reduces training time and memory consumption with minimal increase in bpd. Each shape is a pair (layer type, K ).
Figure 9: Our NoPA ( Section 5 ) setting on the Adult (left), Beijing (center) and Default (right) datasets shows minimal decrease in sample quality. Each shape is a pair (layer type, K ).
IPA (bpd = 5.00)
Figure 10: RePA samples retrieve global patch dependencies and are closer to FullPC on ImageNet64 where we show IPA 8×8×3 and K=1024 (left), RePA (center) with K=512 and FullPC with K=128 (right). All PCs use a QG RG. More samples in Figs. 23 and 24 . We take modes of the input unit distributions rather than sampling from them, see details in Section B.3 .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Tree RG → Tensorized PC. The Chow-Liu tree algorithm [ 8 ] can be used to build a tree region graph [ 12 ] (a) which can then be used to build a tensorized PC parameterized with CP layers (b) or with alternative CP-T [ 32 , 40 ] or Monarch layers [ 65 ] .
Dataset
Class
RG
#CPUs
#GPU
GPU Types
RAM
ImageNet32
IPA
[CLT, QG, QT-2]
16
1
[L40s, H200]
128GB
ImageNet32
RePA
[QG, QT-2]
20
1
[A100]
120GB
ImageNet32
RePA
[CLT]
127
1
[RTX A6000]
504GB
ImageNet64
IPA
[CLT, QG, QT-2]
16
1
[L40s, H200]
128GB
ImageNet64
RePA
[QG, QT-2]
16
1
[L40s, H200]
128GB
ImageNet64
RePA
[CLT]
127
1
[RTX A6000]
504GB
Appendix
Table 1: Hardware used for training experiments detailed in Table 4 , Table 5 , Table 6 . We report either reserved resources for SLURM jobs or available resources when not using a SLURM cluster.
Dataset
Train
Validation
Test
ImageNet32
960,875
320,292
50,000
ImageNet64
1,217,108
64,059
50,000
CelebA
162,770
19,867
19,962
Appendix
Table 2: Train, validation and test splits for each dataset.
Dataset
Effective Batch Size
Early Stopping Delta
ImageNet32
20,000
10
ImageNet64
20,000
30
CelebA
5,000
10
Appendix
Table 3: Dataset-dependent hyperparameters. We use different effective batch size and early stopping delta values to balance quality and training time.
Class
Dataset
Layer type
K
RG
D
IPA
ImageNet32
[CP, CP-T]
[128,256,512,1024]
[QT-2, QG, CLT]
[2,4,8,16,32]
IPA
ImageNet64
[CP-T]
[128,256,512,1024]
[QT-2, QG, CLT]
[2,4,8,16,32,64]
IPA
CelebA
[CP-T]
[128,256,512,1024]
[QT-2, QG, CLT]
[2,4,8,16,32,64]
IPA
ImageNet32
[Mona]
[1024, 4096]
[QT-2, QG]
[8,16,32]
Appendix
Table 4: Hyperparameter grid used for IPA , values in brackets indicate each value that this parameter can take. We train the full grid of all possible combinations.
Class
Dataset
Layer type
K
RG
D
Dt
RePA
ImageNet32
[CP, CP-T]
[512,1024,2048]
[QT, QG]
[2]
[2,4,8,16,32]
RePA
ImageNet64
[CP-T]
[512,1024,2048,4096]
[QT, QG]
[2]
[2,4,8,16,32,64]
RePA
CelebA
[CP-T]
[512,1024,2048,4096]
[QT, QG]
[2]
[2,4,8,16,32,64]
RePA
ImageNet32
[Mona]
[1024, 4096]
[QT, QG]
[2]
[8,16,32]
Appendix
Table 5: Hyperparameter grid used for RePA . When Dt is smaller than the image S , t being the recursion level, we use IPA ( RePA ).
ci
Class
Dataset
Layer type
K
RG
D
Dt
c1
RePA
ImageNet32
[CP-T]
[512,1024]
[CLT]
4
4
c2
RePA
ImageNet32
[CP-T]
[512,1024]
[CLT]
4
16
c3
RePA
ImageNet32
[CP-T]
[512,1024]
[CLT]
2
32
c1
RePA
ImageNet64
[CP-T]
[512,1024]
[CLT]
4
4
c2
RePA
ImageNet64
[CP-T]
[512,1024]
[CLT]
4
16
c3
RePA
ImageNet64
[CP-T]
[512,1024]
[CLT]
4
64
Appendix
Table 6: Hyperparameter grid used for RePA on CLT. When Dt is smaller than the image S , t being the recursion level, we use IPA ( RePA ). Each row represents the configuration for one of the three circuits that constitute the final RePA .
Parameter
Possible values
Layer type
CP, CP-T
K
1024, 2048, 4096, 8192
Learning rate
0.1 , 0.25 , 0.5
Scheduler
ReduceLROnPlateau, None
Batch size
256, 512
Appendix
Table 7: Hyperparameter grid used for tabular data
class
test bpd (mean)
test bpd (std)
count
FullPC
5.347784
0.001272
6
IPA
5.200556
0.000694
6
RePA (K=1024)
5.053292
0.005864
6
Appendix
Table 8: Variation of our best models using a quad-graph over different random seeds on ImageNet32.
class
test bpd (mean)
test bpd (std)
count
FullPC
1.097372
0.001769
6
NoPA
1.099097
0.003449
6
Appendix
Table 9: Variation of our TabPC on different random seeds on the Adult dataset.
lr
0.0001
0.0005
0.001
0.0050
0.01
0.05
FullPC
Timed out (T/O)
T/O
5.70
5.57
5.50
5.35
IPA
6.89
5.72
5.66
5.49
5.38
5.20
RePA (K=1024)
T/O
T/O
T/O
5.21
5.15
5.04
Appendix
Table 10: Variation of our best models using a quad-graph over different learning rates on ImageNet32.
batch size
512
1024
8192
16384
FullPC (Adam)
5.40
5.40
5.37
5.35
FullPC (EM)
6.88
6.60
5.85
5.59
Appendix
Table 11: Variation of batch size on our best FullPC model while using Adam or EM. The model uses a quad-graph, CP-T layers and K=512 .
patch size
2
4
8
16
32
IPA
5.605210
5.328382
5.277076
5.293465
5.346852
IPA ( NoShare )
5.611599
5.334445
5.278990
5.293443
5.346852
Appendix
Table 12: Dropping parameter sharing for IPA does not improve test bpds , although it improves theoretical expressivity.
Figure 12: Sum of squares (SOS) PC and probabilistic integral circuits (PIC) also show signs of plateau. All PCs are built using the QT region graph.
Figure 13: Both Adam and EM can be affected by the independent patch conundrum when optimizing PCs over different patch sizes. We note that Adam provides as good a BPD as, or better than, EM when we use the same effective batch size and learning rate scheduler. Lines in the plot that do not have points for all patch sizes indicate that we could not fit the corresponding circuit in memory.
Figure 14: RePA models are more Pareto efficient for all datasets tested . We compare bpd with regards to training step time (right), GPU memory (center) and FLOPs (left) on ImageNet 32 (top), ImageNet 64 (middle) and CelebA (bottom).
Dataset
Class
K
Layer type
RG
D
Test bpd
Dataset
Class
K
Layer type
RG
D
Test bpd
CelebA
IPA
128
CP-T
CLT
2
4.70
Imagenet 32
IPA
512
CP-T
QT
16
5.39
CelebA
IPA
128
CP-T
CLT
4
4.19
Imagenet 32
IPA
512
CP-T
QT
32
5.44
CelebA
IPA
128
CP-T
CLT
8
4.06
Imagenet 32
IPA
1024
CP
CLT
2
5.33
CelebA
IPA
128
CP-T
CLT
16
4.02
Imagenet 32
IPA
1024
CP
CLT
4
4.95
CelebA
IPA
128
CP-T
CLT
32
3.99
Imagenet 32
IPA
1024
CP
CLT
8
4.92
CelebA
IPA
128
CP-T
QG
2
5.38
Imagenet 32
IPA
1024
CP
QG
2
5.64
Appendix
Table 13: Complete density estimation results for every model configuration presented in the paper. We exclude PICs and SOS as they are only presented in appendix.
Figure 15: RePA improves learnability for density estimation. We compare the bpd with respect to patch size for QT (left), QG (center), CLT (right) on ImageNet32 (top), ImageNet64 (middle) and CelebA (bottom)
Figure 16: RePA yields a better Pareto front on ImageNet32. We compare the bpd with respect to efficiency for QT (top), QG (middle), CLT (bottom) on ImageNet32. We do not distinguish different values of K
Figure 17: RePA yields a better Pareto front on ImageNet64. We compare the bpd with respect to efficiency for QT (top), QG (middle), CLT (bottom) on ImageNet64. We do not distinguish different values of K
Figure 18: RePA yields a better Pareto front on CelebA. We compare the bpd with respect to efficiency for QT (top), QG (middle), CLT (bottom) on CelebA. We do not distinguish different values of K
Figure 19: Our RePA construction with CLTs reduces training time and memory consumption for the ImageNet32 (gold border dots), ImageNet64 (silver) and CelebA (bronze) datasets.
Figure 20: Combining RePA with Monarch layers lets PCs scale even further as shown for QT and QG RGs and K=1024,4096 on ImageNet32.
Figure 21: NoPA yields more efficient models with little decrease in C2ST with XGBoost. Each row is a dataset in this order: Adult, Beijing, Default, Diabetes, Magic, News, Shoppers.
Figure 22: NoPA yields more efficient models with little increase in bpd. Each row is a dataset in this order: Adult, Beijing, Default, Diabetes, Magic, News, Shoppers.
IPA
Appendix
Figure 23: Samples for CelebA (top row), ImageNet32 (middle row) and ImageNet64 (bottom row) using IPA 8×8×3 and K=1024 (left), RePA (center) with K=1024 for CelebA and K=512 for ImageNet64 and FullPC with K=128 (right). All the PCs were built with a QG RG.
IPA
Appendix
Figure 24: Samples for CelebA (top row), ImageNet32 (middle row) and ImageNet64 (bottom row) using IPA 8×8×3 and K=1024 (left), RePA (center) with K=1024 for CelebA and K=512 for ImageNet64 and FullPC with K=128 (right). All the PCs were built with a QT RG.
Probabilistic circuits are structured generative models that enable tractable probabilistic inference while balancing expressivity and computational cost.
We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.
Probabilistic circuits (PCs) can model complex joint distributions while supporting exact and efficient computation of many inference queries. However, standard likelihood-based PC learning is vulnerable to overfitting and fragile generalization when confronted with data noise, small sample sizes, or distribution shifts. This can be mitigated using distributionally-robust optimization which consider worst-case distributions within a Wasserstein ball of the empirical distribution, but current methods are limited to training a model from scratch in this framework. Instead, we propose PeTeR: a novel, data-free post-training framework designed to robustify pre-trained PCs against distribution shifts without retraining from scratch. Empirical evaluations across multiple density estimation benchmarks demonstrate that PeTeR effectively robustifies baseline models against both random and adversarial perturbations, achieving competitive or superior performance to data-dependent robust learning baselines.
Adrian Ciotinga, Yeming Dai, YooJung Choi
School of Computing and Augmented Intelligence, Arizona State University
Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.
Zhiyu Zhao, Xuejie Liu, Muhan Zhang +1
School of Computing, National University of Singapore · School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University