We present a novel application of embedding-based dynamic topic modeling techniques to detect and quantify topic drift at the comment level in a massive corpus. By leveraging pretrained language models to generate contextualized semantic embeddings for short text, we analyzed 12.7 billion Reddit comments spanning 2006 to 2022. Using unsupervised methods on these embeddings, we identify dynamically evolving topic clusters over time. Our primary contribution is a methodology for analysis of semantic drift and discourse evolution in the embedding space itself. We also demonstrate modifications to existing methods that enable this analysis at scale, and we propose and demonstrate a null model comparison test to filter spurious dynamics. Key findings suggest that politically and socially contentious topics exhibit significant directional drift in embedding space, with inter-topic distances changing systematically over time beyond what the null model can explain, whereas domains such as music and sports remain comparatively stable.
Figures & tables
Figure 1: Clustering in embedded space yields time-local topics with contextualized semantic similarity over time. Aligning highly similar topic clusters over time may reveal topic drift in embedding space.
Figure 2: Visualization of topic clusters for two example time periods (June 2007 (a) and 2008 (b) ), with each dot representing a single embedded comment. Only the top 200 closest to the centroid are shown for visual clarity; text label is the top scoring word by TF-IDF. We note trends of changing focus: increased presidential voting discussion, diversification of several topics (e.g., immigration), and increase in sexually-oriented commenting.
Figure 3: Aligned topic groups over all time periods of the Reddit corpus (2006-2022) of 12.7 billion comments, using comment-level embeddings, with each dot representing just the topic centroid of a particular time period; labeling is the top shared word across multiple topic groups (using TF-IDF). Visualization uses a projection to 2-D using UMAP.
Topic k
δk
Λk
pk
“conspiracy”
0.387
0.277
0.285
“oil”
0.374
0.794
0.295
top
“scientific”
0.354
8.656
0.003
“government”
0.353
2.776
0.004
“republican”
0.351
2.525
0.119
“woman”
0.032
0.142
0.901
Table 1: Top and bottom-ranked aligned topic groups ordered by total displacement ( δk ) in embedding space; log-likelihood ratio ( Λk ) and computed p -value ( pk ) of nonzero mean drift are also presented. We see movement in inherently controversial topics like conspiracy theories or politics, while sports or music discussion remains surprisingly stable.
Figure 4: (a) Visualization of two regimes of topic groups: those with a significant level of drift (in color) and those without (in gray). (b) A topic group related to “laughing” exhibiting static movement around a center, with non-significant levels of drift ( Λ=3.74 , p=0.15 ). (c) A “religion” topic group with significant levels of drift ( Λ=8.29 , p=0.001 ). All plots are presented in their 2-D UMAP projection.
Figure 5: Average annual change in centroid distance between a selected topic and other nearby topics, annotated by the topic group’s top shared keyword across all time periods. Negative values (blue) mean the topic group is growing closer to the selected topic in embedding space, implying a change in context and semantics.
Figure 6: Visualization of several aligned topic groups in a select region of embedding space. We observe 3-4 topic groups that appear to drift toward a central shared area of embedding space while others move away.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Average within-cluster sum-of-squares (WCSS) for a grid of k=30,50,70,90 across all time periods (excluding 2014-2015).
Figure 8: Average within-cluster sum-of-squares (WCSS) for a grid of 1≤k≤220 on a single demonstration time period (April 2017).
Figure 9: Aligned topic groups for the same resampled time period, using a single shared centroid initialization in each permutation (a) and different initialization (b) . Despite variation in centroids between permutations, the method recovers groups that are nearly an exact match for the original clusters, and inspection of the random walk test under both initialization schemes shows no groups have p -values below 0.01 .
Dynamic Embedded Topic Models (D-ETM) provide an interpretable framework for modeling temporal semantic evolution, but cross-corpus comparison remains difficult because topics are often learned independently and aligned only after training, a process that does not guarantee stable topic correspondence across corpora and time. To address this problem, we propose a D-ETM framework that first learns a common dynamic topic space over a merged multi-corpus collection, which we call the shared backbone, then introduces corpus-specific residual adaptation around the frozen backbone without creating separate latent topic spaces. This design preserves a shared topic index for cross-corpus comparison while allowing each corpus to specialize lexically. We evaluate the framework on three temporally structured corpora spanning 97 years: the Corpus of Historical American English, Harvard Business Review, and International Labour Review. Residual adaptation improves corpus-specific fit relative to the shared backbone while preserving the same-index cross-corpus topic trajectories, achieving substantially stronger alignment than full fine-tuning from the same backbone, with 97.5±0.7% versus 17.9±1.1% trajectory Retrieval@1, as well as stronger alignment than independent training with post-hoc Hungarian matching. These results suggest that incorporating topic alignment into the model can support more stable over-time cross-corpus comparisons while retaining corpus-specific lexical variation.
Ruoxuan Li, Bruce Kogut
Columbia University New York, NY, USA · Department of Sociology and Columbia Business School Columbia University New York, NY, USA
Research on conspiracy theories has largely focused on belief formation, exposure, and diffusion, while paying less attention to how their meanings change over time. This gap persists partly because conspiracy-related terms are often treated as stable lexical markers, making it difficult to separate genuine semantic changes from surface-level vocabulary changes. In this paper, we measure the semantic structure and evolution of conspiracy theories in online political discourse. Using 169.9M comments from Reddit's r/politics subreddit spanning 2012--2022, we first demonstrate that conspiracy-related language forms coherent and semantically distinguishable regions of language space, allowing conspiracy theories to be treated as semantic objects. We then track how these objects evolve over time using aligned word embeddings, enabling comparisons of semantic neighborhoods across periods. Our analysis reveals that conspiracy theories evolve non-uniformly, exhibiting patterns of semantic stability, expansion, contraction, and replacement that are not captured by keyword-based approaches alone.
Political polarization has become a defining feature of online discourse, yet its long-term evolution remains poorly understood. We present a longitudinal analysis of ideological polarization in Reddit discussions by measuring semantic differences in the language used by opposing political communities. We construct temporally aligned community-specific word embeddings and quantify ideological polarization as the semantic divergence of political concepts over time. Our analysis shows that ideological polarization has increased substantially during the study period, both at the concept- and topic-level. Unlike prior computational work, which has largely focused on cross-sectional analyses or affective dimensions of polarization at a single point at time, our approach captures the evolution of ideological differences in semantic framing. The proposed framework provides a scalable method for studying the temporal dynamics of ideological polarization in large-scale social media discourse.