PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement

Yongkang Guo; Zhihuan Huang; Yuqing Kong

cs.CLMar 14, 2026

PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement

Authors: Yongkang Guo, Zhihuan Huang, Yuqing Kong

Organizations: CFCS, School of Computer Science, Peking University, China

Abstract

High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.

Figures & tables

Figure 1: Flow of PMIScore .

Figure 2: PMIScore vs. baselines on synthetic distributions. The figure shows the Spearman correlation (left) and mean squared error (right) of different methods ( PMIScore , MINE, InfoNCE, and KDE) on three synthetic dependency structures ( Diagonal , Block , and Independent ). Higher Spearman values indicate better alignment with the true PMI ranking and lower MSE means better approximation of the accurate PMI. PMIScore achieves consistently highest rank correlation and lowest MSE across almost all settings.

Figure 3: Consistency of PMI estimation for different methods. Each panel compares the estimated versus ground-truth pointwise mutual information (PMI) for four estimation methods— PMIScore , MINE, InfoNCE, and KDE—on the Block synthetic dataset with the Qwen3-4B embeddings. To illustrate the magnitude of estimation error, 1,000 representative samples are plotted in each figure. The x-axis is the ground-truth of PMI and the y-axis is the estimated value. The dashed line means that the estimated PMI perfectly matchs the ground truth. While PMIScore aligns closely with the identity line, indicating minimal estimation bias, alternative estimators display noticeable deviations or slope distortions. Similar patterns are also observed across other embedding models and synthetic datasets.

Figure 4: PMIScore vs. baselines on empirical dialogue datasets. Bars show mean across different llms for English ( dstc_en ) and Chinese ( dstc_zh ). Left: ROC-AUC on val/test for the response-ranking task. Right: Spearman correlation with human relevance on the dev sets. Higher ROC-AUC indicates that the score can better distinguish the positive and negative pairs. PMIScore closely tracks InfoNCE on AUC and yields the best human correlation on English while remaining competitive on Chinese; both substantially outperform KDE .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

$f$ -Divergence	$f(t)$	$\partial{f}(D)$	$f^{\star}(\partial{f}(D)$ )
Total Variation Distance	$\|t-1\|$	$\sign(\log D)$	$\sign(\log D)$
KL-Divergence	$t\log t$	$\log D+1$	$D$
Pearson $\chi^{2}$	$(t-1)^{2}$	$2(D-1)$	$D^{2}-1$
Jensen-Shannon	$-(t+1)\log{\frac{t+1}{2}}+t\log t$	$\log{\frac{2D}{1+D}}$	$-\log(\frac{2}{1+D})$
Squared Hellinger	$(\sqrt{t}-1)^{2}$	$1-\sqrt{\frac{1}{D}}$	$\sqrt{D}-1$

Appendix

Table 1: Reference Table for

f,f^{\star}

( Kong and Schoenebeck, 2019 )

Dataset	Model	PMIScore		MINE		InfoNCE		KDE		MEEP
Dataset	Model	$\rho$	MSE	$\rho$	MSE	$\rho$	MSE	$\rho$	MSE	$\rho$	MSE
Block	Phi-4-mini-instruct	0.802 (0.017)	0.350 (0.070)	0.754 (0.023)	2.520 (3.499)	0.662 (0.042)	1.686 (1.042)	0.702 (0.016)	1.923 (0.036)	0.124 (0.021)	2.902 (0.038)
	Qwen3-0.6B	0.470 (0.019)	1.287 (0.424)	0.533 (0.012)	14.253 (16.473)	0.370 (0.016)	12.089 (13.000)	0.390 (0.009)	4.268 (0.019)	-0.002 (0.043)	3.119 (0.079)
	Qwen3-1.7B	0.649 (0.038)	0.692 (0.331)	0.578 (0.019)	12.457 (8.151)	0.464 (0.027)	5.630 (3.639)	0.550 (0.021)	2.677 (0.028)	0.037 (0.042)	3.058 (0.069)
	Qwen3-4B	0.777 (0.017)	0.371 (0.064)	0.742 (0.029)	2.080 (2.841)	0.639 (0.022)	0.850 (0.053)	0.638 (0.009)	2.059 (0.061)	0.096 (0.045)	2.964 (0.075)
	Qwen3-8B	0.811 (0.010)	0.337 (0.047)	0.808 (0.019)	0.398 (0.075)	0.750 (0.011)	0.702 (0.262)	0.632 (0.024)	2.533 (0.104)	0.191 (0.035)	2.789 (0.077)

Appendix

Table 2: Performance on Synthetic datasets. Best Pearson

\rho

(higher) and MSE (lower) per model are bolded .

Dataset	Model	PMIScore		MINE		InfoNCE		KDE		MEEP
Dataset	Model	AUC	$\rho$	AUC	$\rho$	AUC	$\rho$	AUC	$\rho$	AUC	$\rho$
English	Phi-4-mini-instruct	0.922 (0.002)	0.407 (0.096)	0.910 (0.005)	0.379 (0.107)	0.919 (0.002)	0.436 (0.075)	0.881 (0.002)	0.482 (0.052)	0.673 (0.010)	0.335 (0.085)
	Qwen3-0.6B	0.858 (0.006)	0.286 (0.075)	0.829 (0.026)	0.258 (0.081)	0.830 (0.009)	0.175 (0.069)	0.781 (0.005)	0.214 (0.092)	0.531 (0.013)	0.129 (0.127)
	Qwen3-1.7B	0.903 (0.004)	0.433 (0.055)	0.892 (0.003)	0.406 (0.092)	0.885 (0.006)	0.321 (0.078)	0.793 (0.004)	0.180 (0.060)	0.593 (0.018)	-0.018 (0.082)
	Qwen3-4B	0.941 (0.002)	0.475 (0.028)	0.933 (0.003)	0.445 (0.024)	0.936 (0.006)	0.351 (0.090)	0.914 (0.003)	0.381 (0.076)	0.670 (0.016)	0.038 (0.110)
	Qwen3-8B	0.952 (0.003)	0.467 (0.032)	0.947 (0.003)	0.462 (0.023)	0.952 (0.003)	0.414 (0.045)	0.926 (0.004)	0.365 (0.074)	0.798 (0.008)	0.173 (0.153)

Appendix

Table 3: Performance on Empirical datasets. Best AUC (higher) and Spearman

\rho

(higher) per model are bolded.

Multi-Turn Dialogue Evaluation

Multi-turn dialogue evaluation measures whether AI systems remain accurate, coherent, safe, and useful as conversations evolve.

See all 202 papers

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
Jun 10, 2026Paul He, Shiva Kasiviswanathan, Dominik JanzingLLM Evaluation Multi-Turn Dialogue Evaluation
NTU Singapore · Amazon Research, Tübingen, Germany · Amazon
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement
Sep 16, 2026Xinglang Zhang, Yuanmeng Xiang, Yunyao Zhang +3LLM Evaluation Preference Alignment
Huazhong University of Science and Technology

Jun 10, 2026J/K move · Enter open · S save

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

Paul He, Shiva Kasiviswanathan, Dominik Janzing

NTU Singapore · Amazon Research, Tübingen, Germany · Amazon

LLM Evaluation Multi-Turn Dialogue Evaluation Automated Evaluation

Open paper AskPDF arXiv

Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension of information-seeking dialogue: semantic progress, defined as the accumulation of new, question-relevant, and non-redundant information over the course of a conversation. We formalize semantic progress as question-conditioned uncertainty reduction and introduce an information-theoretic metric that approximates it in embedding space. Our main estimator uses a tractable Gaussian formulation with closed-form updates, while a complementary maximum-entropy argument shows why log-determinant structure arises more broadly when only second-order embedding information is retained. This formulation yields desirable theoretical properties, including monotonicity, additive decomposition of total information gain across turns, and diminishing returns for redundant evidence. Unlike LLM-as-a-judge approaches, our metric requires no autoregressive inference at evaluation time and is fully reproducible for a fixed embedding model. Experiments on MT-Bench, Chatbot Arena, and UltraFeedback show that the proposed metric achieves competitive agreement with human judgments despite targeting only semantic progress, with improved alignment on MT-Bench and UltraFeedback compared to several LLM-based judges. Notably, the method remains effective with lightweight embedding models under CPU-only execution, indicating that semantic progress can be captured without reliance on large model capacity.

Abstract

Multi-Turn Dialogue Evaluation

Explore similar work

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement