We present a novel application of embedding-based dynamic topic modeling techniques to detect and quantify topic drift at the comment level in a massive corpus. By leveraging pretrained language models to generate contextualized semantic embeddings for short text, we analyzed 12.7 billion Reddit comments spanning 2006 to 2022. Using unsupervised methods on these embeddings, we identify dynamically evolving topic clusters over time. Our primary contribution is a methodology for analysis of semantic drift and discourse evolution in the embedding space itself. We also demonstrate modifications to existing methods that enable this analysis at scale, and we propose and demonstrate a null model comparison test to filter spurious dynamics. Key findings suggest that politically and socially contentious topics exhibit significant directional drift in embedding space, with inter-topic distances changing systematically over time beyond what the null model can explain, whereas domains such as music and sports remain comparatively stable.
Figures & tables
Figure 1: Clustering in embedded space yields time-local topics with contextualized semantic similarity over time. Aligning highly similar topic clusters over time may reveal topic drift in embedding space.
Figure 2: Visualization of topic clusters for two example time periods (June 2007 (a) and 2008 (b) ), with each dot representing a single embedded comment. Only the top 200 closest to the centroid are shown for visual clarity; text label is the top scoring word by TF-IDF. We note trends of changing focus: increased presidential voting discussion, diversification of several topics (e.g., immigration), and increase in sexually-oriented commenting.
Figure 3: Aligned topic groups over all time periods of the Reddit corpus (2006-2022) of 12.7 billion comments, using comment-level embeddings, with each dot representing just the topic centroid of a particular time period; labeling is the top shared word across multiple topic groups (using TF-IDF). Visualization uses a projection to 2-D using UMAP.
Topic k
δk
Λk
pk
“conspiracy”
0.387
0.277
0.285
“oil”
0.374
0.794
0.295
top
“scientific”
0.354
8.656
0.003
“government”
0.353
2.776
0.004
“republican”
0.351
2.525
0.119
“woman”
0.032
0.142
0.901
Table 1: Top and bottom-ranked aligned topic groups ordered by total displacement ( δk ) in embedding space; log-likelihood ratio ( Λk ) and computed p -value ( pk ) of nonzero mean drift are also presented. We see movement in inherently controversial topics like conspiracy theories or politics, while sports or music discussion remains surprisingly stable.
Figure 4: (a) Visualization of two regimes of topic groups: those with a significant level of drift (in color) and those without (in gray). (b) A topic group related to “laughing” exhibiting static movement around a center, with non-significant levels of drift ( Λ=3.74 , p=0.15 ). (c) A “religion” topic group with significant levels of drift ( Λ=8.29 , p=0.001 ). All plots are presented in their 2-D UMAP projection.
Figure 5: Average annual change in centroid distance between a selected topic and other nearby topics, annotated by the topic group’s top shared keyword across all time periods. Negative values (blue) mean the topic group is growing closer to the selected topic in embedding space, implying a change in context and semantics.
Figure 6: Visualization of several aligned topic groups in a select region of embedding space. We observe 3-4 topic groups that appear to drift toward a central shared area of embedding space while others move away.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Average within-cluster sum-of-squares (WCSS) for a grid of k=30,50,70,90 across all time periods (excluding 2014-2015).
Figure 8: Average within-cluster sum-of-squares (WCSS) for a grid of 1≤k≤220 on a single demonstration time period (April 2017).
Figure 9: Aligned topic groups for the same resampled time period, using a single shared centroid initialization in each permutation (a) and different initialization (b) . Despite variation in centroids between permutations, the method recovers groups that are nearly an exact match for the original clusters, and inspection of the random walk test under both initialization schemes shows no groups have p -values below 0.01 .