Abstract
Topic modeling is widely used in computational social sciences to identify latent themes in large text corpora. Traditional approaches rely on Bag-of-Words representations and generative models such as LDA, while recent methods like BERTopic operate on dense document embeddings. This paper introduces the Structural Contextual Probabilistic Topic Model (SCPTM), an architecture that incorporates syntactic dependency relations into topic inference. SCPTM represents a corpus as a heterogeneous graph of documents and words connected by lexical and syntactic edges, processed through a Graph Attention Network within a Variational Autoencoder to produce probabilistic, mixed-membership topic distributions. We evaluate seven topic modeling techniques (including four SCPTM ablations) across four corpora differing in register and discourse structure. Our framework combines coherence (C_V, C_NPMI), topic diversity, clustering-label alignment (NMI), and phrase-level diagnostics (complementarity and valence gap). Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines, but these gains are attributable to the variational encoder rather than to syntax. Syntax contributes to topic diversity, where graph-augmented variants outperform the no-graph baseline across all corpora, and to descriptor quality: dependency paths capture predicate-argument structures and stance in deliberative registers, while proving redundant in technical and institutional corpora. The valence gap is positive across all variants, but driven primarily by phrase grouping rather than syntactic filtering. We conclude that syntactic encoding matters conditionally: it benefits action-oriented, argumentative texts, but introduces noise in informational or administrative registers.
Explore similar work
Jul 31, 2025cs.CL
Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic steering vectors. To properly evaluate MTM topics against word list approaches, we propose \textit{topic judge}, an LLM-based pairwise comparison evaluation framework. Across eight datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering.
Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5
Sep 9, 2026cs.CL
Topic models summarize large text corpora, but top-ranked words often provide only a limited representation of topic semantics. Sparse autoencoders (SAEs) offer a way to move beyond word-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic-inference quality remains unclear. We introduce \textbf{MonoTM}, an interpretable topic modeling framework that decouples these roles. Across three benchmark corpora, we show that document--topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets. MonoTM estimates mixtures from the full SAE bag-of-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus-grounded semantic features. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis.
Una Joh, Bei Yu
May 21, 2026cs.CL
Topic modeling in applied psychology increasingly spans two methodological traditions: probabilistic bag-of-words models and newer embedding-based approaches. Yet many evaluations of these methods rely on longer and cleaner benchmark corpora, leaving less guidance for short, open-ended survey responses. This paper compares Structural Topic Models (STM), a probabilistic topic model, and BERTopic, an embedding-based model, for analyzing open-ended survey responses. We evaluated three STM conditions and five BERTopic conditions, varying typographical correction, stemming, embedding choice, and contextual augmentation, a strategy we introduced to provide additional semantic context for very short responses. Results indicate that BERTopic consistently produced higher topic coherence than STM, with contextual augmentation yielding the strongest performance gains. In contrast, higher-dimensional embeddings alone did not improve coherence and were associated with greater data loss. Qualitative evaluation showed that BERTopic generated more interpretable and stable topics, while STM topics were often broader and more mixed. However, STM provides stronger support for inferential covariate analysis, whereas BERTopic covariate comparisons are primarily descriptive. These findings suggest that STM and BERTopic offer complementary strengths. We conclude with practical guidance for selecting and combining topic modeling approaches in applied social science research.
Yan Jiang, Sihong Liu, Philip A. Fisher