We introduce CALLIOPE, a top-down framework for discovering topic hierarchies in large corpora of unstructured data (e.g., LLM conversations), while supporting differential privacy (DP). Unlike traditional and even recent LLM-augmented approaches that ultimately rely on clustering vector embeddings, our framework leverages LLM capabilities directly for the entire pipeline. Experiments demonstrate that this approach provides better utility over existing (embedding-based) methods in the DP setting, especially in the high privacy regime. We evaluate the quality of CALLIOPE outputs using a variety of traditionally studied clustering metrics, as well as some novel utility metrics that we introduce.
Figures & tables
Figure 1: Side-by-side comparison by human raters showing Win / Tie / Loss (%) on Banking77 , CLINC-OOS , and Massive . Bold values indicate statistically significant preference ( p<0.05 , two-sided sign test on non-tied comparisons); right-hand labels indicate (d,k) configurations.
Figure 2: Results of the comparison on “Guess-Odd-One” test.
Figure 3: Results of the comparison on “Guess-Cluster”.
Figure 4: Visualization of BuildSchemaAndParts stage of the Calliope framework ( Algorithm 1 ); note that the Abstraction stage is not repeated for each level, but we include it here for clarity.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Venn diagram visualizing the clusters and document summaries presented for side by side comparison.
Figure 6: Silhouette Scores
Figure 7: Caliński & Harabasz Scores
Figure 8: Davies & Bouldin Scores
Figure 9: Comparison of Normalized Mutual Information.
Figure 10: Comparison of Adjusted Rand Index.
Figure 11: Side-by-side comparison across all six datasets using four LLM judges (Gemini-3.5-Flash, DeepSeek, Phi-4, and Qwen), grouped by algorithm pair and privacy budget ε . Each bar shows Win (green), Tie (blue), and Loss (red) percentages; the central dashed line marks the 50% equal-preference threshold, while the lighter dashed lines at 40% and 60% mark the thresholds for statistically significant preference ( p<0.05 , two-sided sign test, also indicated by bold numbers). Right-hand labels show the (d,k) configurations.
Dataset
Urania Tokens
Calliope Tokens
Banking77
2.22 M
7.98 M
CLINC-OOS
3.28 M
8.99 M
MASSIVE
123 M
576 M
Reuters
20.9 M
55.3 M
Emotion
3.81 M
14.5 M
IMDB
25.4 M
58.1 M
Appendix
Table 1: Number of tokens used by Urania vs Calliope (in non-private regime with maximal considered depth for gemini-3.5-flash ).
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3