Clarify User Expertise: Towards Proactive Conversational Agents Tailoring Responses to User Proficiency
Authors: Zhihong Cao, Chen Huang
Organizations: School of Computing and Data Science, The University of Hong Kong · Institute of Data Science, National University of Singapore · College of Computer Science, Sichuan University
In the context of information seeking, conversational agents are undergoing an evolution from reactive tools to proactive, personalized assistants. A critical aspect of this evolution is the ability to tailor strategic interactions to a user's unique needs and expectations. Unlike existing studies that focus on proactively clarifying query ambiguities, we center on clarifying the user's expertise in order to tailor responses for better user comprehension. We find that existing agents struggle to determine user expertise from queries alone, a limitation that prevents them from dynamically adapting their responses. To address this gap, we introduce PASSING to empower the agent to proactively clarify a user's expertise through targeted inquiries. This is achieved by our What-to-ask and How-to-ask strategies, induced by LLM self-play. Our extensive experiments also show our superiority. We believe that PASSING represents a crucial step towards creating more human-centric conversational agents.
Figures & tables
Figure 1: User expertise can vary depending on the specific query. LLM agent should tailor its response based on user expertise.
Method
MMLU-Pro
ARC-MCAS
Kappa
Unk
Kappa
Unk
Deterministic
100.0
0.00
100.0
0.00
Fully Random
0.00
16.67
0.00
16.67
Direct Prompt
41.25
28.00
30.64
82.80
CoT
28.47
26.40
70.34
89.60
Self-consistency
85.12
32.00
75.35
86.80
Table 1: Prediction stability and unknown rate of existing methods for query-specific user expertise estimation.
Figure 2: Overview of Passing . It utilizes LLM-induced strategies (i.e., What-to-Ask and How-to-Ask), obtained through self-play simulation.
Aspects
Descriptions
Knowledge Scope (What they know)
The extent, type, and organization of the knowledge of the person: knowing “facts” (explicit knowledge), knowing when to apply it (applied knowledge), or knowing “how” intuitively (tacit knowledge)
Components (What they perceive)
What types of knowledge or information the person relies on: context-free (rules, abstract principles), rule-based (context-aware rule selection), or situational (contextual, experiential patterns)
Perspective (How they organize info)
How the person filters and prioritizes information: overwhelmed easily, making a deliberate plan to filter noise, or instinctively seeing important information without trying
Action (How they decide)
How decisions are made: analytic (step-by-step reasoning) or intuitive (immediate, holistic response)
Commitment (Emotional connection)
The level of personal agency and emotional investment in the outcome: detached (disengaged, rule-following) or involved (personally invested, responsible for outcome)
Table 2: Structured schema used for estimating user expertise based on the Dreyfus Model.
LLM Backbone
Method
MMLU-Pro
ARC-MCAS
Acc. ↑
Kappa ↑
FA ↑
Comp. ↑
US ↑
Unk. ↓
Acc. ↑
Kappa ↑
FA ↑
Comp. ↑
US ↑
Unk. ↓
Evaluation with LLM Simulators
GPT-4o
IDL ( the best baseline )
20.0
40.31
3.75
3.62
-
23.79
20.0
15.00
4.59
3.42
-
4.40
Direct Prompt
16.0
41.25
3.84
3.69
-
28.00
20.0
30.64
4.59
3.67
-
82.80
Passing
77.1
64.90
3.85
3.83
4.01
0.0
87.4
71.28
4.59
3.87
4.18
0.0
Passing w/o What-to-ask
68.6
64.43
3.81
3.87
3.96
0.0
76.8
67.26
4.45
3.86
4.08
0.0
Table 3: Main results. Best results are in bold, and second-best results are underlined.
Figure 3: Pairwise similarity among induced strategies (13 What-to-ask and 8 How-to-ask strategies in total)
Figure 4: Effect of the number of probing strategies on expertise estimation accuracy and user satisfaction.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Value
What-to-Ask intra-class similarity
0.608
How-to-Ask intra-class similarity
0.604
Inter-class similarity
0.604
Diversity gap Δ
0.003
Silhouette score
0.006
Appendix
Table 4: Semantic similarity statistics for induced strategies.
Setting
Passing US (95% CI)
Difference (95% CI)
Holm p
GPT-4o / MMLU-Pro
4.580 [4.540, 4.621]
1.224 [1.073, 1.360]
0.007812
GPT-4o / ARC-MCAS
4.583 [4.573, 4.594]
1.028 [0.999, 1.055]
0.007812
DeepSeekV4-Pro / MMLU-Pro
4.423 [4.408, 4.436]
1.146 [1.056, 1.234]
0.007812
DeepSeekV4-Pro / ARC-MCAS
4.568 [4.558, 4.580]
1.280 [1.240, 1.321]
0.007812
Appendix
Table 5: Error Bars and Statistical Significance in Human Evaluation.
Dataset
Backbone
Gwet’s AC2 ↑
FA
Comp.
US
Human–Human
ARC-MCAS
DeepSeek-V4-Pro
0.7681
0.6996
0.6641
GPT-4o
0.6949
0.8488
0.7614
MMLU-Pro
DeepSeek-V4-Pro
0.8698
0.8121
0.7888
GPT-4o
0.6213
0.5153
0.6477
Appendix
Table 6: Inter-rater agreement (Gwet’s AC2) for human–human and human–LLM evaluation across datasets and backbones. FA: factual accuracy; Comp.: comprehensibility; US: user satisfaction.
GPT-4o / MMLU-Pro
GPT-4o / ARC-MCAS
DeepSeekV4-Pro / MMLU-Pro
DeepSeekV4-Pro / ARC-MCAS
N
AB
C
P
E
N
AB
C
P
E
N
AB
C
P
E
N
AB
C
P
E
N
14
0
0
0
0
N
18
1
0
0
0
N
14
0
0
0
0
N
18
1
0
0
0
AB
0
12
1
1
0
AB
2
17
0
0
0
AB
6
8
0
0
0
AB
4
15
0
0
0
C
0
1
13
0
0
C
0
1
18
0
0
C
0
2
9
3
0
C
0
3
13
3
0
P
0
0
0
14
0
P
0
0
0
19
0
P
0
1
0
13
0
P
0
0
1
16
2
E
0
1
5
7
1
E
0
0
2
6
11
E
0
0
0
4
10
E
0
0
0
6
13
Appendix
Table 7: Confusion matrices across different LLM backbones and datasets. Rows denote ground-truth Dreyfus levels and columns denote predicted levels in the order Novice, Advanced Beginner, Competent, Proficient, Expert.
Case
Domain
Expertise
IDL Pred.
Passing Pred.
IDL Error Pattern
Passing Evidence
Case 1
Computer Science
Competent
Advanced Beginner
Competent
Underestimates the user because closed-form QA history appears shallow.
Probes concept boundaries, mechanism understanding, and applied judgment.
Case 2
Chemistry
Competent
Proficient
Competent
Overestimates the user because advanced terminology appears expert-like.
Probes mechanism understanding, uncertainty, and guided reasoning.
Appendix
Table 8: Case-level comparison between IDL and Passing .