Modeling Document and Section Hierarchies in Embedding Space
Organizations: Columbia University
Abstract
Embedded topic models integrate word embeddings into the topic generation process, producing coherent topics and robust performance on large vocabularies. However, each document is represented by a single topic distribution, ignoring its internal structure. Many real world collections, including legal documents, company and government reports, and online discussion forums, exhibit hierarchical organization in which documents consist of multiple sections that often address distinct subtopics. We introduce the Hierarchical Embedded Topic Model (hETM), a structured topic model that couples the topic distribution of a document and those of its sections through a hierarchical prior in the embedding space. We derive an amortized variational inference procedure for hETM and evaluate it on structured collections from each of these domains. Experimental results show that hETM achieves the lowest or near-lowest held-out perplexity among flat and hierarchical topic model baselines, along with comparable or higher topic quality. It also performs strongly on a downstream clustering task, achieving the best clustering performance on most of the collections.
Figures & tables
| Model | U.S. Trade Statutes | Edgar | EUR-Lex |
|---|---|---|---|
| hETM | 0.190 20 | 0.334 10 | 0.174 20 |
| SawETM | 0.141 50 | 0.163 10 | 0.250 10 |
| TSNTM | 0.158 10 | 0.332 100 | 0.162 50 |
| mlhLDA | 0.037 10 | 0.326 50 | 0.152 100 |
| ETM(D) | 0.060 10 | 0.289 100 | 0.103 100 |
| ETM(S) | 0.072 50 | 0.242 100 | 0.145 100 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | U.S. Trade Statutes | Edgar | EUR-Lex |
|---|---|---|---|
| hETM | 0.582 20 | 0.571 10 | 0.512 20 |
| SawETM | 0.546 50 | 0.446 10 | 0.528 10 |
| TSNTM | 0.500 10 | 0.564 100 | 0.472 50 |
| mlhLDA | 0.409 50 | 0.524 50 | 0.521 20 |
| ETM(D) | 0.473 20 | 0.557 100 | 0.474 100 |
| ETM(S) | 0.470 30 | 0.512 100 | 0.449 100 |
| Model | U.S. Trade Statutes | Edgar | EUR-Lex |
|---|---|---|---|
| hETM | 0.236 20 | 0.402 100 | 0.190 20 |
| SawETM | 0.222 10 | 0.362 10 | 0.258 10 |
| TSNTM | 0.186 50 | 0.387 100 | 0.158 30 |
| mlhLDA | 0.206 20 | 0.320 10 | 0.234 50 |
| ETM(D) | 0.219 30 | 0.324 100 | 0.185 100 |
| ETM(S) | 0.181 10 | 0.245 100 | 0.104 30 |
| Corpus | Best embedding for hETM |
|---|---|
| U.S. Trade Statutes | Learned |
| Edgar | Learned |
| EUR-Lex | LegalBERT-base |
| 20 Newsgroups | Learned |
Topic Modeling
Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.
See all 63 papers