Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic
Organizations: University of York
Abstract
Quantitative Discourse Analysis has seen growing adoption with the rise of Large Language Models and computational tools. However, reliance on black box software such as MAXQDA and NVivo risks undermining methodological transparency and alignment with research goals. This paper presents a hybrid, transparent framework for QDA that combines lexical and semantic methods to enable triangulation, reproducibility, and interpretability. Drawing from a case study in historical political discourse, we demonstrate how custom Python pipelines using NLTK, spaCy, and Sentence Transformers allow fine-grained control over preprocessing, lemmatisation, and embedding generation. We further detail our iterative BERTopic modelling process, incorporating UMAP dimensionality reduction, HDBSCAN clustering, and c-TF-IDF keyword extraction, optimised through parameter tuning and multiple runs to enhance topic coherence and coverage. By juxtaposing precise lexical searches with context-aware semantic clustering, we argue for a multi-layered approach that mitigates the limitations of either method in isolation. Our workflow underscores the importance of code-level transparency, researcher agency, and methodological triangulation in computational discourse studies. Code and supplementary materials are available via GitHub.
Figures & tables
| Bigram | Count |
|---|---|
| (“technical”, “education”) | 23 |
| (“secondary”, “education”) | 19 |
| (“board”, “education”) | 16 |
| (“education”, “committee”) | 13 |
| (“education”, “authority”) | 12 |
| Step | Process Type | Description |
|---|---|---|
| Pre-processing | Manual | Split corpus into sentences of even length |
| Embedding | Manual | Use all-mpnet-base-v2 to generate 768D vectors |
| Dimensionality Reduction | Internal | Apply UMAP to reduce to 2D |
| Clustering | Internal | HDBSCAN groups similar sentences |
| Keyword Extraction | Internal | c-TF-IDF retrieves top 10 terms per topic |
| Topic Refinement | Manual | Merge, rename, select 15 final topics |
| Corpus | Total Sentences | Avg Length | Min Length | Max Length | ¡5 Words | ¿25 Words |
|---|---|---|---|---|---|---|
| B&S | 95,557 | 15.22 | 5 | 270 | 0 | 4,763 |
| Model | Outliers | Topics | N-gram | Gini | Coherence | Silhouette | Time |
|---|---|---|---|---|---|---|---|
| (n) | Score | Score | (C_V) | (Avg) | (min) | ||
| all-MiniLM-L6-v2 | 60,277 | 520 | 0.16 | 0.529 | 0.060 | 0.000 | 10.96 |
| all-mpnet-base-v2 | 62,729 | 584 | 0.16 | 0.516 | 0.090 | 0.000 | 10.22 |
| distilroberta-base | 54,131 | 922 | 0.19 | 0.391 | 0.060 | 0.000 | 11.63 |
| bge-small-en-v1.5 | 59,266 | 59 | 0.13 | 0.867 | NaN | -0.080 | 10.91 |
| mpnet-distilled | 63,041 | 384 | 0.16 | 0.706 | 0.260 | -0.040 | 10.84 |
Topic Modeling
Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.
See all 62 papers