Sentence-Level Topic Modeling with Lexical and Contextual Evidence

Federica Granese; Serena Villata; Charles Bouveyron

cs.CLOct 16, 2026

Sentence-Level Topic Modeling with Lexical and Contextual Evidence

Authors: Federica Granese, Serena Villata, Charles Bouveyron

Organizations: Universit´e Cˆote d’Azur, CNRS, I3S, Inria · Universit´e Cˆote d’Azur, CNRS, LJAD, Inria

Abstract

Topic modeling faces a fundamental granularity dilemma: probabilistic topic models assign topics at the word level, whereas transformer-based methods operate at the document level. The latter overlooks that real-world documents typically cover multiple topics, whereas the former relies on individual words, which rarely provide sufficient context to identify them. We introduce SentenceETM, a neural topic model that assigns a topic to each sentence by jointly using lexical and contextual evidence. Each topic is described both by a distribution over the vocabulary, defined through word embeddings, and by a Gaussian over sentence embeddings concentrated on a topic-specific low-dimensional subspace. The topic of a sentence generates both its words and its embedding and is marginalized exactly, so that its posterior combines the two sources of evidence. Document proportions are inferred with amortized variational inference, while the Gaussian parameters are re-estimated in closed form. Experiments show that SentenceETM consistently improves sentence-level topic assignment over state-of-the-art approaches, and a case study of a presidential debate shows how sentence-level topics reveal how candidates handle the same subjects.

Figures & tables

Figure 1: Overview of SentenceETM . Document-level topic proportions (green) are combined with the contextual and lexical evidence of each sentence (red) into the posterior $\gamma_{mk}$ , which re-estimates each topic’s subspace Gaussian in closed form (dashed arrow).

Figure 3: Ablation on BBC News (

K=5

, all-mpnet-base-v2 ). ARI and AMI as a function of

c

, for the common subspace dimensions

h_{k}=h\in\{4,16,32,64\}

. Lexical evidence only in gray; diamonds denote contextual-only models.

Figure 4: Sentence-level topic assignments (Wikipedia, all-mpnet-base-v2 , $K=9$ ). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel.

Panel of Figure 5 — Figure 5: SentenceETM per-subset gain, all-mpnet-base-v2 . Each point corresponds to a subset of (K) ground-truth classes, with ( $\Delta$ ) measured relative to the best-performing baseline for that subset (SentenceLDA) and averaged over seeds.

Panel of Figure 6 — Figure 6: ARI and AMI as function of the number of topics $K$ , all-mpnet-base-v2 . Markers indicate the median across runs, while bars show the interquartile range. Higher is better.

Figure 7: Difference in probabilities of topic use between Trump and Biden across debate segments. $\Delta P=P(T\mid S,\text{Trump})-P(T\mid S,\text{Biden})$ .

Figure 8: New topics per additional sentence within speaking turns , by segment and candidate, as a difference from each candidate’s average. Positive values: more topics than usual; negative values: fewer.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Synthetic corpora generation. Example with $K=3$ , $L=10$ and $p=0.6$ .

$K$	Run	Seed	Topics
2	0	6142	entertainment, politics
2	1	6142	business, entertainment
2	2	6142	politics, tech
3	0	6142	entertainment, politics, tech
3	1	6142	business, entertainment, tech
3	2	6142	entertainment, sport, tech

Appendix

Table 1: BBC News. For each number of topics

K

and each repetition, the classes were sampled from BBC News to compose the synthetic documents. Every document draws its dominant topic uniformly among the

K

listed. When

K=K_{\max}

, a single subset exists, so repetitions vary the corpus seed instead of the classes.

$K$	Run	Seed	Topics
2	0	6142	rec.motorcycles, sci.crypt
2	1	6142	misc.forsale, sci.crypt
2	2	6142	alt.atheism, rec.autos
3	0	6142	rec.autos, sci.crypt, talk.religion.misc
3	1	6142	alt.atheism, comp.sys.mac.hardware, rec.autos
3	2	6142	rec.motorcycles, soc.religion.christian, talk.politics.mideast

Appendix

Table 2: 20 Newsgroups. For each number of topics

K

and each repetition, the classes were sampled from 20 Newsgroups to compose the synthetic documents. Every document draws its dominant topic uniformly among the

K

listed. When

K=K_{\max}

, a single subset exists, so repetitions vary the corpus seed instead of the classes.

$K$	Run	Seed	Topics
2	0	6142	Mathematics; Music
2	1	6142	Language and literature; Music
2	2	6142	Agriculture, food, and drink; Language and literature
3	0	6142	Language and literature; Music; Warfare
3	1	6142	Agriculture, food, and drink; Engineering and technology; Language and literature
3	2	6142	Language and literature; Social sciences and society; Video games

Appendix

Table 3: Wikipedia. For each number of topics

K

and each repetition, the classes were sampled from Wikipedia to compose the synthetic documents. Every document draws its dominant topic uniformly among the

K

listed. When

K=K_{\max}

, a single subset exists, so repetitions vary the corpus seed instead of the classes.

		Sentence-level				Document-level
Dataset	$K$	SentenceETM	SentenceLDA	senLDA	SenClu	BERTopic	ETM	LDA
BBC News	2	281	1946	86	1.6	1.6	4.3	26
	3	328	1391	87	1.9	1.6	4.4	23
	5	410	1815	88	2.3	1.7	4.6	24
	Avg.	340	1717	87	1.9	1.6	4.4	25
20 Newsgroups	2	210	1913	81	1.2	1.5	4.0	22

Appendix

Table 4: Training time in seconds (median over 9 runs; 700 documents,

14{,}000

sentences), excluding corpus construction and sentence encoding. All experiments were run on NVIDIA H100 GPUs. Neural models (SentenceETM, ETM, BERTopic, SenClu) run on a GPU; LDA, senLDA, and SentenceLDA, whose released implementations are CPU-only, run on CPU.

Panel of Figure 10 — Figure 10: Sentence-level topic assignments ( all-mpnet-base-v2 ). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel. Wikipedia is in Figure 4 .

		ARI			AMI
	Model	$K=2$	$K=3$	$K=5$	$K=2$	$K=3$	$K=5$
mpnet	SentenceETM	0.76 $\pm$ 0.04	0.69 $\pm$ 0.03	0.63 $\pm$ 0.01	0.67 $\pm$ 0.04	0.62 $\pm$ 0.03	0.61 $\pm$ 0.01
	SentenceLDA	0.52 $\pm$ 0.12	0.58 $\pm$ 0.03	0.54 $\pm$ 0.07	0.50 $\pm$ 0.08	0.54 $\pm$ 0.02	0.52 $\pm$ 0.05
	senLDA	0.52 $\pm$ 0.08	0.43 $\pm$ 0.11	0.46 $\pm$ 0.07	0.45 $\pm$ 0.06	0.39 $\pm$ 0.10	0.45 $\pm$ 0.05
	SenClu	0.26 $\pm$ 0.22	0.25 $\pm$ 0.12	0.23 $\pm$ 0.04	0.25 $\pm$ 0.20	0.26 $\pm$ 0.11	0.26 $\pm$ 0.04
	BERTopic	0.00 $\pm$ 0.00	0.01 $\pm$ 0.02	0.08 $\pm$ 0.01	0.00 $\pm$ 0.00	0.01 $\pm$ 0.01	0.11 $\pm$ 0.01

Appendix

Table 5: BBC News: ARI and AMI. Results are reported as mean

\pm

standard deviation over

3\times 3

runs. Bold the best results within each sentence encoder; underlined the overall best results.

		ARI				AMI
	Model	$K=2$	$K=3$	$K=5$	$K=9$	$K=2$	$K=3$	$K=5$	$K=9$
mpnet	SentenceETM	0.56 $\pm$ 0.05	0.46 $\pm$ 0.03	0.35 $\pm$ 0.04	0.27 $\pm$ 0.02	0.48 $\pm$ 0.03	0.44 $\pm$ 0.02	0.41 $\pm$ 0.05	0.37 $\pm$ 0.04
	SentenceLDA	0.16 $\pm$ 0.04	0.23 $\pm$ 0.02	0.30 $\pm$ 0.07	0.25 $\pm$ 0.03	0.26 $\pm$ 0.03	0.37 $\pm$ 0.01	0.39 $\pm$ 0.05	0.33 $\pm$ 0.03
	senLDA	0.22 $\pm$ 0.13	0.30 $\pm$ 0.02	0.21 $\pm$ 0.03	0.17 $\pm$ 0.05	0.18 $\pm$ 0.08	0.26 $\pm$ 0.01	0.22 $\pm$ 0.03	0.21 $\pm$ 0.05
	SenClu	0.36 $\pm$ 0.19	0.35 $\pm$ 0.07	0.28 $\pm$ 0.03	0.23 $\pm$ 0.04	0.32 $\pm$ 0.16	0.33 $\pm$ 0.06	0.33 $\pm$ 0.03	0.31 $\pm$ 0.05
	BERTopic	0.00 $\pm$ 0.00	0.02 $\pm$ 0.03	0.09 $\pm$ 0.02	0.06 $\pm$ 0.08	0.00 $\pm$ 0.00	0.03 $\pm$ 0.03	0.12 $\pm$ 0.02	0.10 $\pm$ 0.10

Appendix

Table 6: 20 Newsgroup: ARI and AMI . Results are reported as mean

\pm

standard deviation over

3\times 3

runs. Bold the best results within each sentence encoder; underlined the overall best results.

		ARI				AMI
	Model	$K=2$	$K=3$	$K=5$	$K=9$	$K=2$	$K=3$	$K=5$	$K=9$
mpnet	SentenceETM	0.69 $\pm$ 0.13	0.57 $\pm$ 0.08	0.45 $\pm$ 0.15	0.47 $\pm$ 0.04	0.61 $\pm$ 0.15	0.55 $\pm$ 0.09	0.47 $\pm$ 0.15	0.55 $\pm$ 0.02
	SentenceLDA	0.44 $\pm$ 0.11	0.41 $\pm$ 0.07	0.38 $\pm$ 0.15	0.45 $\pm$ 0.05	0.45 $\pm$ 0.09	0.48 $\pm$ 0.09	0.42 $\pm$ 0.15	0.50 $\pm$ 0.04
	senLDA	0.53 $\pm$ 0.19	0.47 $\pm$ 0.08	0.31 $\pm$ 0.09	0.29 $\pm$ 0.04	0.46 $\pm$ 0.18	0.43 $\pm$ 0.08	0.34 $\pm$ 0.09	0.38 $\pm$ 0.03
	SenClu	0.37 $\pm$ 0.32	0.40 $\pm$ 0.16	0.22 $\pm$ 0.15	0.27 $\pm$ 0.04	0.34 $\pm$ 0.28	0.38 $\pm$ 0.15	0.23 $\pm$ 0.15	0.34 $\pm$ 0.03
	BERTopic	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.04 $\pm$ 0.02	0.11 $\pm$ 0.05	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.08 $\pm$ 0.01	0.18 $\pm$ 0.03

Appendix

Table 7: Wikipedia: ARI and AMI. Results are reported as mean

\pm

standard deviation over

3\times 3

runs. Bold the best results within each sentence encoder; underlined the overall best results.

Figure 11: Sensitivity to document length (Wikipedia,

K=9

, all-mpnet-base-v2 ). ARI (left) and AMI (right) as a function of the number of sentences per document

M^{(d)}

(

3\times 3

runs).

Figure 12: Topic coherence (TC) and topic diversity (TD) on the original corpora. Topics are those learned in the synthetic-corpus experiments, while TC is evaluated using word co-occurrences from the corresponding full original dataset. TD is computed directly from the learned top words. Higher is better.

BBC News
True $\rightarrow$ Predicted	Sentence
Entertainment $\rightarrow$ Sport	There were no specific details on the injuries .
Business $\rightarrow$ Tech	It will do so at a presentation in New York on 26 January .
Business $\rightarrow$ Politics	“I will give up my role as soon as the board of directors asks me to do so,” he said .
Business $\rightarrow$ Politics	In both cases, the justification has been national security, amid allegations that the Turkish firms are too close to Israel .
Business $\rightarrow$ Politics	“First and foremost this is a human tragedy.”

Appendix

Table 8: Examples of topic disagreements with the source labels. We report the topic inherited by the sentence from its source document and the predicted topic (SentenceETM). The misassignments are on generic sentences, cases where the predicted topic better matches the sentence content, and overlaps between categories.

Topic	Label	Top words
0	Lockdowns	shut, close, country, open, allow, state, democrats, destroy, wrong, throw
1	Accusations	man, plan, lie, fraud, office, name, liar, run, wait, acknowledge
2	Mail-in voting	ballot, vote, count, send, poll, mail, solicit, state, open, director
3	Electric cars	car, cheap, expensive, electric, safe, cost, standard, automobile, tiny, least
4	Conversational	guess, wrong, hit, honest, suppose, hey, excuse, surprised, feel, plenty
5	Achievements	job, win, top, thank, proud, testify, set, every, happy, administration

Appendix

Table 9: Topics learned on the presidential debate. For each topic, the ten words with the highest probability under the learned topic–word distribution, and a short label assigned by inspecting them.

Topic Modeling

Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.

See all 58 papers

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings
Oct 4, 2026Ryotaro Kobayashi, Yuri Murayama, Kiyoshi IzumiLatent Variable Models Topic Modeling
Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
From Documents to Segments: A Contextual Reformulation for Topic Assignment
May 18, 2026Hoonsang Yoon, Takyoung Kim, Wonkee Lee +3Topic Modeling
1LG AI Research · University of Illinois Urbana-Champaign
Improving Topic Modeling by Distilling Soft Labels from Language Models
Feb 20, 2026Raymond Li, Amirhossein Abaskohi, Chuyuan Li +2Topic Modeling
University of British Columbia, Vancouver, Canada.

Oct 4, 2026J/K move · Enter open · S save

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

Ryotaro Kobayashi, Yuri Murayama, Kiyoshi Izumi

Graduate School of Engineering, The University of Tokyo, Tokyo, Japan

Latent Variable Models Topic Modeling

Open paper AskPDF arXiv

Latent Dirichlet Allocation (LDA) and models derived from it remain widely used topic models. LDA observes each document as a bag-of-words and models each topic by a categorical distribution over the vocabulary, so that it uses neither the internal structure of the document nor the similarity in meaning between words. Earlier work has responded to this limitation in two ways: many models have introduced embeddings, and some have assigned topics to sentences rather than to words. Their combination, a topic model that observes sentence embeddings, remains little explored. We propose vMF Sentence LDA (vSLDA), which keeps the admixture structure of LDA, observes each sentence as its L2-normalized embedding and models each topic by a von Mises-Fisher (vMF) distribution, which matches the cosine geometry of sentence embeddings. Its per-topic parameter count and per-iteration cost are linear in the embedding dimension, versus quadratic for the full-covariance Gaussian distribution in the existing model over sentence embeddings. We evaluate vSLDA where the limitation is expected to matter most, among topics that share much of their vocabulary: the topics that subdivide the one subject of a collection, and the narrow topics that result when a corpus is divided into a large number of topics. On two corpora, vSLDA attains the best mean rank against eight baselines when the fine categories within each coarse category are classified from the document-topic distributions. On the whole corpus, its advantage appears or widens as the number of topics grows. Weighting the word frequencies of each sentence by its topic posterior yields expected topic-word counts of the same form as those of LDA, so that the standard topic coherence and diversity measures apply to models that assign topics to sentences. On their product, topic quality, vSLDA leads in most within-category conditions of both corpora.

Abstract

Topic Modeling

Explore similar work

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

From Documents to Segments: A Contextual Reformulation for Topic Assignment

Improving Topic Modeling by Distilling Soft Labels from Language Models