KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn

Yoonjoo Lee; Hyoungwook Jin; Tae Soo Kim; Shaoyang Zhang; Philippe Laban; Q. Vera Liao

cs.AIAug 17, 2026

KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn

Authors: Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim, Shaoyang Zhang, Philippe Laban, Q. Vera Liao

Organizations: University of Michigan · KAIST · Microsoft Research

Abstract

To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user's evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KNOWSIM, an evaluation framework built around a user simulator that maintains explicit knowledge states, represented as a graph of Information Units with prerequisite relationships, that evolve under update rules grounded in learning theory. KNOWSIM computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KNOWSIM against 705 human-AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73-74% sign agreement), outperforming three baseline simulators. Applied to 9 LLMs, KNOWSIM reveals that the best model shifts by user knowledge level, revealing aptitude-treatment interactions invisible to standard evaluation.

Figures & tables

Figure 1: KnowSim simulates two users with different knowledge levels asking the same medical question to the same assistant. The novice patient absorbs nothing in Turn 1 (cognitive load exceeds capacity), slowly picks up prerequisites, and ends with limited gain and high overload. The advanced doctor, whose prerequisites are already met, absorbs one concept per turn and reaches full mastery with low overload. Metrics (KG, DC, CO), computed directly from evolving knowledge states, explain why the same response helps one user but overwhelms another.

Arm	Task	Fixed	Comparison Target
Math $\times$ Strat.	MATH (L5)	Model	Strategy
Math $\times$ Model	MATH (L5)	Strategy	Model
Exp. $\times$ Strat.	ExpertQA	Model	Strategy
Exp. $\times$ Model	ExpertQA	Strategy	Model

Table 1: Study arms overview. Each arm recruits

{\sim}

60 participants (min: 50, max: 72).

Source	KG	DC	CO	IQ
KnowSim (ours)	✓	✓	✓	✓
Baselines	—	—	—	✓
Human	✓ ∗	✓	✓	✓

Table 2: Metric coverage. IQ is common across simulators; KG, DC, and CO are unique to KnowSim .

Evaluation Setting	Agree / HS	Rate	$\boldsymbol{p}$
MathQA–Strategy	13 / 17	76%	.025 ∗
MathQA–Model	14 / 20	70%	.058 †
ExpertQA–Strategy	16 / 20	80%	.006 ∗∗
ExpertQA–Model	10 / 15	67%	.151
MathQA (pooled)	27 / 37	73%	.003 ∗∗
ExpertQA (pooled)	26 / 35	74%	.003 ∗∗

Table 3: Sign agreement between KnowSim and human pairwise condition contrasts on high-signal cells (

|\delta_{\text{human}}|\geq 0.15

). Rate indicates the fraction of contrasts where the simulator’s direction matches the human Cliff’s

\delta

;

p

is from a one-sided binomial test against chance (50%).

Figure 2: Per-level sign agreement (%) between KnowSim and human rankings, by task (Strategy + Model arms pooled).

Figure 3: Per-metric sign agreement (%) between KnowSim and human rankings, by task (Strategy + Model arms pooled). KG is not measured on ExpertQA.

Figure 4: Cross-task IQ sign agreement (%) between each simulator and human rankings, pooled across both tasks and all four arms (22 signal pairs).

	Novice					Intermediate					Advanced
Model	KG	CO $\downarrow$	DC	IQ	$\bar{R}$	KG	CO $\downarrow$	DC	IQ	$\bar{R}$	KG	CO $\downarrow$	DC	IQ	$\bar{R}$
Gemini 3.1 Pro	13.2	0.875	0.086	8.39	3.25	5.4	0.750	0.088	8.35	4.75	0.7	0.873	0.122	7.96	1.75
Claude Opus 4.7	14.3	0.882	0.090	7.80	3.25	5.6	0.807	0.128	8.39	2.50	0.6	0.879	0.107	8.30	3.00
DeepSeek V4	15.5	0.945	0.092	6.50	4.00	5.7	0.835	0.121	7.53	4.25	0.5	0.890	0.109	7.74	4.25
Gemini 3.1 Flash	15.1	0.936	0.093	6.83	3.25	5.6	0.838	0.116	7.73	5.75	0.5	0.912	0.109	7.64	6.00
Claude Sonnet 4.6	12.9	0.877	0.079	7.06	4.75	5.4	0.791	0.098	7.87	5.00	0.6	0.866	0.100	8.00	3.50

Table 4: Aptitude–treatment interaction patterns: per-level metrics across 9 LLMs. Each cell shows the metric value;

\bar{R}

is the mean rank across the four metrics within that level. CO

\downarrow

: lower is better. Best value per column is bolded .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Level	knows_well	partial	struggling	unaware
novice	0.10	0.10	0.30	0.50
intermediate	0.55	0.15	0.20	0.10
advanced	0.80	0.10	0.10	0.00

Appendix

Table 5: Initial knowledge-state ratios

R_{\ell}

used to instantiate

s_{0}(\ell)

for each level.

Pre-turn state	$W_{\text{S}}$ (mention)	$W_{\text{R}}$ (reasoning)
unaware	1.75	1.75
struggling	1.00	1.40
partial_understanding	0.70	0.70
knows_well	0.35	0.35

Appendix

Table 6: State-weighted load coefficients.

W_{\text{S}}

weights mentioned IUs by their pre-turn state.

W_{\text{R}}

weights IUs the user reasoned about. Articulation is a flat

W_{\text{A}}=0.35

ID	Concept	Prereqs
1	Classical probability	—
2	Combination counting	—
3	Multiplication principle	—
4	Color arrangement model	—
5	Color arrangement count	2, 4
6	On/off selection model	—

Appendix

Table 7: Example IU graph for a lamp arrangement probability problem. An intermediate user might start with IU1–3 at knows_well , IU4, 6 at partial_understanding , and the rest unaware or struggling .

Figure 5: The interface used for the human data collection. Participants are asked to solve a problem by conversing with a chatbot via a chat interface. The figure shows the interface for the math domain. The expert domain interface is identical, except that it does not include pre- and post-tests.

Level	KG	DC	CO	IQ	Turns
Strategy arm
Novice ( $n{=}20$ )	0.12 (0.34)	3.35 (2.63)	3.87 (2.70)	4.15 (2.78)	13.1 (14.6)
Intermediate ( $n{=}19$ )	0.10 (0.37)	4.82 (2.16)	3.88 (2.27)	5.26 (2.27)	12.8 (11.8)
Advanced ( $n{=}28$ )	0.04 (0.34)	5.87 (2.41)	3.21 (2.80)	6.05 (2.50)	0 6.4 (5.1)
Model arm
Novice ( $n{=}21$ )	0.02 (0.35)	4.97 (2.76)	2.83 (2.25)	5.83 (2.28)	0 7.9 (4.9)

Appendix

Table 8: Per-level statistics for the MathQA arms. Cells are mean (SD) computed over conversations (three sessions per participant);

n

is the number of participants. Turns counts assistant–user exchanges in the human conversation.

Strategy arm
Level	DC	CO	IQ	Turns
Novice ( $n{=}11$ )	5.73 (2.70)	2.70 (2.65)	6.03 (2.66)	6.7 (4.7)
Intermediate ( $n{=}15$ )	5.69 (2.81)	1.07 (1.66)	5.96 (3.06)	6.0 (4.6)
Advanced ( $n{=}24$ )	5.78 (2.22)	2.32 (2.40)	5.78 (2.45)	5.0 (3.7)
Model arm
Novice ( $n{=}20$ )	6.38 (1.83)	3.37 (2.52)	6.72 (1.65)	4.8 (1.9)

Appendix

Table 9: Per-level statistics for the ExpertQA arms. Cells are mean (SD) computed over conversations (three sessions per participant);

n

is the number of participants. KG is not measured on ExpertQA, whose open-ended questions admit no pre/post-test knowledge-gain measure.

Arm	$H$	$p$	$\varepsilon^{2}$
MathQA–Strategy	59.14	${<}.001$	.893
MathQA–Model	50.31	${<}.001$	.895
ExpertQA–Strategy	47.44	${<}.001$	.967
ExpertQA–Model	54.47	${<}.001$	.905

Appendix

Table 10: Knowledge-level group separation. Kruskal-Wallis

H

tests whether the grouping variable differs across the three levels within each arm. Groups in both tasks are defined by the objective prescreening score.

Level	$n$	KG–DC	KG–CO	KG–IQ
Novice	41	$-$ .323 ∗	.261	$-$ .205
Intermediate	35	$-$ .250	$-$ .146	$-$ .358 ∗
Advanced	47	.187	$-$ .296 ∗	$-$ .063
Pooled	123	$-$ .185 ∗	$-$ .052	$-$ .268 ∗∗

Appendix

Table 11: Participant-level Spearman

\rho

between KG and three self-reported metrics on human MathQA data (both arms pooled). KG = (post-test

-

pre-test) / (max

-

pre-test); DC, CO, IQ are survey-based.

{}^{*}p{<}.05

{}^{**}p{<}.01

	MathQA			ExpertQA
	n	Turns	Words	n	Turns	Words
ZS	270	3.1	922	360	5.7	2640
ZS-CoT	270	3.1	841	360	4.5	1880
ZS-CoT-Prof	270	6.5	981	360	7.4	1791
KnowSim	270	9.1	1638	360	7.3	2735
Human	372	8.6	1469	333	5.4	1485

Appendix

Table 12: Conversation statistics pooled across arms and levels. Turns : virtual early-stop for KnowSim , judged termination for baselines, actual length for humans. Words : total up to that turn.

Method	Strat.	Model	Strat.	Model	All
	MathQA		ExpertQA
KnowSim	67%	83%	86%	67%	77% ∗∗
ZS	67%	50%	86%	67%	68% †
ZS-CoT	50%	50%	86%	67%	64%
ZS-CoT-Prof	67%	67%	86%	67%	73% ∗

Appendix

Table 13: Per-arm IQ sign agreement (%) between each simulator and human rankings on high-signal cells. Individual arms contain few signal pairs (3–7 per arm), so significance is reported only for the pooled column (

{}^{\dagger}p<.10

{}^{*}p<.05

{}^{**}p<.01

; one-sided binomial vs. 50%).

	User-Only		Full-Turn
Method	Mean	Std	Mean	Std
KnowSim	.479	.260	.614	.271
ZS	.487	.279	.596	.266
ZS-CoT	.482	.266	.555	.254
ZS-CoT-Prof	.410	.235	.536	.267

Appendix

Table 14: Concept coverage Jaccard similarity between human and simulated users (

N{=}202

matched pairs). User-only measures overlap in IUs referenced by the user; full-turn includes both user and assistant messages. Bold: highest mean.

Tier	Models
Frontier	GPT-5.4, Claude Opus 4.7, Gemini 3.1 Pro
Mid-tier	GPT-5.4 mini, Claude Sonnet 4.6, Gemini 3.1 Flash
Open-weight	DeepSeek V4, Llama-4-Maverick, Qwen-3.6-35B

Appendix

Table 15: Assistant models evaluated in the benchmark.

Model	KG	DC	CO $\downarrow$	IQ	Mean Rank
Claude Opus 4.7	6.8 (5)	0.108 (1)	0.856 (3)	8.16 (2)	2.75
Gemini 3.1 Pro	6.4 (6)	0.099 (6)	0.833 (1)	8.24 (1)	3.50
DeepSeek V4	7.2 (1)	0.108 (2)	0.890 (8)	7.26 (5)	4.00
Gemini 3.1 Flash	7.1 (2)	0.106 (3)	0.895 (9)	7.40 (4)	4.50
GPT-5.4	7.0 (3)	0.103 (5)	0.878 (5)	6.38 (8)	5.25
Claude Sonnet 4.6	6.3 (7)	0.092 (9)	0.845 (2)	7.65 (3)	5.25

Appendix

Table 16: Benchmarking 9 LLMs with KnowSim , averaged across knowledge levels and both tasks. Each cell shows the metric value with rank in parentheses. CO

\downarrow

: lower is better (less cognitive overload). Mean Rank averages the four per-metric ranks; bold marks the top model.

Figure 6: Failure mode distribution by knowledge level, pooled across all nine assistants (214,642 classified IUs). Novice users face over-reaching failures (prereq-blocked

+

overload); advanced users face over-repetition (redundant).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Training User Simulators with Latent Intrinsic Feedback
Oct 9, 2026Tarek Naous, Craig Boutilier, Wei Xu +1User Simulation Large Language Model-Based Role-Play Simulation
Google Research · Georgia Institute of Technology
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
May 10, 2026Joseph Suh, Ayush Raj, Minwoo Kang +1LLM Agent Evaluation Human-AI Interaction
UC Berkeley

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

LLM Evaluation

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Training User Simulators with Latent Intrinsic Feedback

Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants