Vectorized Bayesian Inference for Latent Dirichlet-Tree Allocation

Zheng Wang; Nizar Bouguila

cs.LGFeb 21, 2026

Vectorized Bayesian Inference for Latent Dirichlet-Tree Allocation

Authors: Zheng Wang, Nizar Bouguila

Organizations: Concordia Institute for Information Systems Engineering Concordia University Montreal, QC H3G 1M8, Canada

Abstract

Latent Dirichlet Allocation (LDA) is a foundational model for discovering latent thematic structure in discrete data, but its Dirichlet prior cannot represent the rich correlations and hierarchical relationships often present among topics. We introduce the framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution. LDTA preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions. To perform inference, we develop universal mean-field variational inference and Expectation Propagation, providing tractable updates for all DT. We reveal the vectorized nature of the two inference methods through theoretical development, and perform fully vectorized, GPU-accelerated implementations. The resulting framework substantially expands the modeling capacity of LDA while maintaining scalability and computational efficiency.

Figures & tables

Figure 1: Notation for a general tree structure

Figure 2: (a) Conventional Multinomial; (b) Hierarchical Multinomial; (c) Dirichlet-Tree

Figure 3: The document–topic–word counting tensor.

Figure 4: Graphical representation (plate notation) of LDTA model. The shaded circle represents the observed variable; the blank circles represent the latent variables and diamonds indicate hyper-parameters. The arrows imply dependent generative relationship and the rectangles with number at its right-lower corner represent repetitions.

Figure 5: Tensor expression of relationship of the variables

Figure 7: Expectation-Maximization iteration process of mean-field variational inference

Figure 8: Construction of approximate posterior distribution and its relationship with other parameters. The $(n_{v})_{V}$ and $(n_{k})_{K}$ represents observed count vector of words, and latent count vector of topics, respectively. In explicit posterior distribution, the normalizer, or the evidence $s(\xi,\boldsymbol{\varphi})$ is intractable, which is the reason why we have to use approximate method. Notice $\prod_{k}{\theta_{k}}^{\sum_{v}\phi_{kv}n_{v}}=\prod_{v}\left({\prod_{k}{\theta_{k}}^{\phi_{kv}}}\right)^{n_{v}}$ . $\xi^{\prime}$ is the revision of prior parameter $\xi$ which incorporates the latent count of topics $(n_{k})_{K}$ based on the property of conjugacy.

Figure 9: EP inference in each iteration. The statistical manifold is conceptually illustrated with three dimensions. The target, the approximator and the tilted distributions are all located on the statistical Riemann manifold thanks to the tractability of the approximating distribution we choose. The approximator is drawn to the target by the group of tilted distributions in each iteration.

Figure 10: Relationships of Cavity and Tilted distributions

Figure 11: Tensor expression of Expectation Propagation

Panel of Figure 12 — Figure 12: ELBO convergence with respect to iterations; iterations stop when ELBO change less than 0.0001

Category	# Documents
earn	3900
acq	2289
crude	370
trade	324
money-fx	306
interest	271

Table 1: Number of documents of chosen categories

Models	Accuracy
Models	K=10	K=20	K=30	K=40	K=50
Dir-VI	92.0%	93.2%	94.4%	95.6%	94.6%
Dir-EP	88.6%	91.6%	91.8%	93.5%	93.9%
BL-VI	93.3%	92.7%	94.9%	95.3%	95.0%
BL-EP	88.9%	92.3%	92.4%	92.5%	92.6%
GDir-VI	79.9%	94.2%	95.1%	93.9%	93.8%

Table 2: Average accuracy of classification

Panel of Figure 19 — Figure 19: Best confusion matrix for each model settings; the titles indicating the topic number

Panel of Figure 20 — Figure 20: Image classification accuracies

Panel of Figure 21 — Figure 21: UMAP of topic proportion of single-cells

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 22: (a) Classical Dirichlet; (b) Beta-Liouville; (c) Generalized Dirichlet

Explore similar work

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings
Oct 4, 2026Ryotaro Kobayashi, Yuri Murayama, Kiyoshi IzumiLatent Variable Models Topic Modeling
Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
Scalable Sparse Autoregressive Variational Inference for Non-Gaussian Spatial Fields
Oct 9, 2026Paul F. V. Wiemann, Matthias KatzfussVariational Inference
Department of Statistics, The Ohio State University, Columbus OH 43210, USA · Department of Statistics, University of Wisconsin–Madison, Madison WI 53706, USA
Amortized Factor Inference Networks for Posterior Inference
May 26, 2026Joohwan Ko, Justin DomkeAmortized Inference Bayesian Inference
Manning College of Information and Computer Sciences University of Massachusetts Amherst

Oct 4, 2026J/K move · Enter open · S save

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

Ryotaro Kobayashi, Yuri Murayama, Kiyoshi Izumi

Graduate School of Engineering, The University of Tokyo, Tokyo, Japan

Latent Variable Models Topic Modeling

Open paper AskPDF arXiv

Latent Dirichlet Allocation (LDA) and models derived from it remain widely used topic models. LDA observes each document as a bag-of-words and models each topic by a categorical distribution over the vocabulary, so that it uses neither the internal structure of the document nor the similarity in meaning between words. Earlier work has responded to this limitation in two ways: many models have introduced embeddings, and some have assigned topics to sentences rather than to words. Their combination, a topic model that observes sentence embeddings, remains little explored. We propose vMF Sentence LDA (vSLDA), which keeps the admixture structure of LDA, observes each sentence as its L2-normalized embedding and models each topic by a von Mises-Fisher (vMF) distribution, which matches the cosine geometry of sentence embeddings. Its per-topic parameter count and per-iteration cost are linear in the embedding dimension, versus quadratic for the full-covariance Gaussian distribution in the existing model over sentence embeddings. We evaluate vSLDA where the limitation is expected to matter most, among topics that share much of their vocabulary: the topics that subdivide the one subject of a collection, and the narrow topics that result when a corpus is divided into a large number of topics. On two corpora, vSLDA attains the best mean rank against eight baselines when the fine categories within each coarse category are classified from the document-topic distributions. On the whole corpus, its advantage appears or widens as the number of topics grows. Weighting the word frequencies of each sentence by its topic posterior yields expected topic-word counts of the same form as those of LDA, so that the standard topic coherence and diversity measures apply to models that assign topics to sentences. On their product, topic quality, vSLDA leads in most within-category conditions of both corpora.

Abstract

Explore similar work

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

Scalable Sparse Autoregressive Variational Inference for Non-Gaussian Spatial Fields

Amortized Factor Inference Networks for Posterior Inference