HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation

Yilin Jiang; Fei Tan; Xuanyu Yin; Jing Leng; Aimin Zhou

cs.CLOct 14, 2026

HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation

Authors: Yilin Jiang, Fei Tan, Xuanyu Yin, Jing Leng, Aimin Zhou

Organizations: East China Normal University, Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shanghai Innovation Institute, Shanghai, China

Abstract

Simulated students let researchers test tutoring systems and teaching strategies before real learners are available, but only if the simulation reproduces where a particular learner succeeds and fails. The common approach asks a large language model to role-play a student, but a role does not model what this learner knows or will do next, so the model's own competence fills the gap. Simulated students perform at a level set by the model, differ little from one another, and seldom make the real learners' errors. We model the learner's behavior and recast student simulation as third-person prediction of the learner's next action. HACHIMI-II builds a student model from the learner's history, infers the learner's state for the current item or tutor turn, and compiles it into a behavior harness that constrains the next action. On three item-response datasets, the Wasserstein-1 distance between simulated and real student ability is 2.4--8.4 $\times$ smaller than that of the strongest baseline. On two dialogue datasets, HACHIMI-II reproduces real student errors with the same knowledge type, locus, and mechanism 1.8--5.1 $\times$ more often than the strongest baseline, and ablations trace the gains mainly to the explicit learner state. Predicting the learner makes capable models usable as simulated students.

Figures & tables

Figure 1: Role-prompted simulation (top left) and the four steps of HACHIMI-II on a linear equation.

State variable	Harness constraint on the next action
Reachable knowledge	Accessible knowledge $K_{t}$ and reasoning depth $R_{t}$ of the action
Active error pattern	Error preservation $E_{t}$ at the predicted step
Confidence state	Uncertainty $U_{t}$ of the answer
Local interaction move	Response mode $M_{t}$ and step budget $L_{t}$ of the turn

Table 1: Learner state variables and the harness constraints compiled from them, named as in Figure 1 .

Item. Two towns are 300 m apart. A student walks there at 10 m/min and back at 15 m/min. What is the average speed of the round trip in m/min? (Answer: 12)
Evidence	30 earlier items with accuracy 0.80, and 0.43 on word problems; accuracy drops on harder items
Student model	Grade 3; confident on routine arithmetic; low confidence and reliance on help on unfamiliar problems
Learner state	Prerequisites familiar; item partly within reach; active error pattern: averages the two speeds, $(10+15)/2=12.5$ ; low commitment; short explanation
Harness	One final answer after a single reasoning step, in at most three sentences
Action	Reasoning step: 30 minutes there, 20 minutes back, average of 10 and 15. Answer: 12.5
Outcome	The real learner answered incorrectly. Direct RP, Prompt-C, and Profile-C all answer 12.

Table 2: One XES3G5M item traced through the four steps of HACHIMI-II.

Figure 2: Real and simulated accuracy per student for Profile-C and HACHIMI-II. The dashed line is

y=x

, and the solid line is a linear fit.

Format	Dataset	Instances
Item response	XES3G5M ( Liu et al., 2023 )	60 $\times$ 96
Item response	Eedi ( Wang et al., 2021 )	480 $\times$ 107
Item response	ENEM 2024 Natural Sciences ( 15 )	7,200 $\times$ 24
Dialogue	Eedi QATD2k ( Zent et al., 2025 )	17,256
Dialogue	Khan/CoMTA ( Miller and DiCerbo, 2024 )	917

Table 3: Datasets. Item-response instances are students

\times

items, and dialogue instances are student turns.

Dataset	Method	Agree $\uparrow$	$W_{1}\downarrow$	$\rho\uparrow$
XES3G5M	Direct RP	0.534	0.327	0.006
	Prompt-C	0.533	0.332	$-$ 0.071
	Profile-C	0.526	0.340	0.220
	HACHIMI-II	0.752	0.039	0.691
Eedi	Direct RP	0.502	0.160	$-$ 0.057
	Prompt-C	0.509	0.155	0.311

Table 4: Item-response results. Best per dataset in bold.

Dimension	Labels	Human	Judge
Functional label	correct; incorrect; not applicable	0.84	0.83
Error indicator	yes; no	0.81	0.80
Knowledge type	factual; conceptual; procedural; metacognitive	0.73	0.69
Error locus	reading; comprehension; transformation; process skills; encoding	0.68	0.62
Error mechanism	language difficulty; spatial information difficulty; deficient prerequisites; incorrect associations; irrelevant rules	0.66	0.60

Table 5: Judge labels and agreement. Human is Fleiss’

\kappa

among three annotators, and Judge is Cohen’s

\kappa

between GPT-5.4 and their majority label.

	XES3G5M			Eedi			ENEM
	Low	Middle	High	Low	Middle	High	Low	Middle	High
Real students	0.74	0.85	0.93	0.34	0.58	0.81	0.22	0.39	0.61
Direct RP	0.50	0.50	0.50	0.59	0.59	0.59	0.83	0.83	0.83
Prompt-C	0.50	0.50	0.50	0.58	0.59	0.60	0.81	0.82	0.83
Profile-C	0.49	0.49	0.49	0.47	0.54	0.58	0.78	0.80	0.82
HACHIMI-II	0.74	0.84	0.92	0.42	0.61	0.75	0.15	0.39	0.68

Table 6: Mean per-student accuracy by third of real accuracy.

		Functional agreement		Error-state prediction			Error-structure fidelity
Dataset	Method	$\kappa_{\mathrm{func}}\uparrow$	$\mathrm{MCC}_{\mathrm{func}}\uparrow$	$\mathrm{BAcc}_{\mathrm{err}}\uparrow$	$\mathrm{MCC}_{\mathrm{err}}\uparrow$	$\mathrm{F1}_{\mathrm{err}}\uparrow$	$\mathrm{Rec}_{\mathrm{struct}}\uparrow$	$\mathrm{F1}_{\mathrm{struct}}\uparrow$
Eedi QATD2k	Direct RP	0.592	0.616	0.560	0.175	0.236	0.157	0.220
	Prompt-C	0.586	0.614	0.558	0.174	0.230	0.150	0.214
	Profile-C	0.595	0.615	0.565	0.179	0.248	0.172	0.232
	Agent-Mem	0.597	0.618	0.562	0.177	0.240	0.162	0.225
	HACHIMI-II	0.659	0.673	0.738	0.509	0.561	0.315	0.339

Table 7: Dialogue results. Best per dataset in bold.

	$W_{1}\downarrow$			$\rho\uparrow$			$\kappa_{\mathrm{func}}\uparrow$		$\mathrm{F1}_{\mathrm{struct}}\uparrow$
Variant	XES	Eedi	ENEM	XES	Eedi	ENEM	QATD2k	CoMTA	QATD2k	CoMTA
HACHIMI-II	0.039	0.046	0.169	0.691	0.703	0.668	0.659	0.370	0.339	0.346
w/o Prior	0.025	0.041	0.142	0.603	0.704	0.594	0.571	0.326	0.264	0.323
w/o State	0.074	0.116	0.409	0.664	0.742	0.539	0.741	0.382	0.136	0.119
w/o Harness	0.056	0.048	0.257	0.559	0.703	0.643	0.565	0.348	0.290	0.346

Table 8: Ablation results. Appendix G reports the remaining metrics.

Figure 3: Simulated outputs on turns where the real learner errs.

Question. A function machine subtracts 4 and then multiplies by 3. Leo writes $3n-12$ and Sophie writes $3(n-4)$ . Who is correct?
Tutor	What do you get if you multiply out the brackets?
Real learner	3n-4 (incorrect; procedural, process skills, deficient prerequisites)
Baselines	$3n-12$ (all four; correct)
HACHIMI-II	State. Weak self-monitoring, tentative, fragment only. Predicted error. $3n-4$ in place of $3n-12$ . Harness. One concrete error at the process-skills step, one sentence. Turn. i get 3n-4 (incorrect; same labels on all three dimensions)

Table 9: A real Eedi QATD2k turn (§ 5.2 ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Item-response simulation	Dialogue simulation
Input evidence	Learner ability, prior responses, guessing tendency, confidence calibration; item prerequisites, operation complexity, response format, guessability	Visible dialogue history, previous student turn, current tutor turn; type, strength, and target of the tutor’s support
Learner state	Prerequisite familiarity, whether the item is within reach, likely error path, commitment, depth of explanation	Whether the step is within reach, misconception status, self-monitoring, uptake of the tutor’s support, help-seeking, readiness to attempt, form of engagement, commitment, depth of response, predicted error (presence, locus, mechanism, faulty rule)
Behavior harness	Whether a full explanation is allowed, response-length limit, restriction on fully correct reasoning, error path to follow, answer commitment	Whether the answer may be stated directly, whether full reasoning is allowed, persistence of confusion and of the predicted misconception, whether help-seeking is allowed, number of solution steps

Appendix

Table 10: State and harness fields in each task format.

Dimension	Fleiss’ $\kappa$	Agreement (%)
Functional label	0.84	91.0
Error indicator	0.81	90.5
Knowledge type	0.73	82.5
Error locus	0.68	78.5
Error mechanism	0.66	77.0

Appendix

Table 11: Agreement among the three annotators. Agreement (%) is exact three-way agreement.

Judge	Func.	Err.	Type	Locus	Mech.	Mean
Gemini 3.5	0.79	0.74	0.64	0.58	0.55	0.660
Kimi 2.6	0.76	0.72	0.62	0.56	0.54	0.640
GLM 5.1	0.78	0.73	0.63	0.57	0.55	0.652
DeepSeek V4	0.80	0.77	0.68	0.64	0.62	0.702
GPT-5.4	0.83	0.80	0.69	0.62	0.60	0.708

Appendix

Table 12: Cohen’s

\kappa

between each candidate judge and the majority human labels.

Dataset	Method	KS $\downarrow$	Stu. $r\uparrow$	Item $\rho\uparrow$	Item $r\uparrow$
XES3G5M	Direct RP	1.000	$-$ 0.005	0.364	0.260
	Prompt-C	1.000	$-$ 0.040	0.366	0.295
	Profile-C	1.000	0.215	0.348	0.264
	HACHIMI-II	0.217	0.629	0.329	0.315
Eedi	Direct RP	0.448	$-$ 0.062	$-$ 0.128	$-$ 0.082
	Prompt-C	0.415	0.334	$-$ 0.085	$-$ 0.044

Appendix

Table 13: Additional item-response metrics.

Figure 4: Real and simulated accuracy per student for all methods and datasets. The dashed line is $y=x$ , and the solid line is a linear fit.

Dataset	Method	Error recall	Error precision	$\kappa_{\mathrm{err}}$	$\mathrm{Prec}_{\mathrm{struct}}$
Eedi QATD2k	Direct RP	0.168	0.393	0.156	0.368
	Prompt-C	0.161	0.404	0.153	0.375
	Profile-C	0.184	0.382	0.164	0.358
	Agent-Mem	0.173	0.392	0.160	0.368
	HACHIMI-II	0.521	0.608	0.507	0.368
Khan/CoMTA	Direct RP	0.084	0.333	0.039	0.316

Appendix

Table 14: Additional dialogue metrics.

					Sim. acc.
Dataset	Variant	Agree $\uparrow$	$W_{1}\downarrow$	$\rho\uparrow$	Mean	SD
XES3G5M	HACHIMI-II	0.752	0.039	0.691	0.828	0.132
	w/o Prior	0.741	0.025	0.603	0.817	0.116
	w/o State	0.784	0.074	0.664	0.905	0.066
	w/o Harness	0.760	0.056	0.559	0.846	0.134
Eedi	HACHIMI-II	0.606	0.046	0.703	0.593	0.211

Appendix

Table 15: Item-response ablation results. Sim. acc. gives the mean and the standard deviation of simulated per-student accuracy. The real means are 0.831 (XES3G5M), 0.574 (Eedi), and 0.394 (ENEM).

Dataset	Variant	$\kappa_{\mathrm{func}}\uparrow$	$\mathrm{MCC}_{\mathrm{func}}\uparrow$	$\mathrm{BAcc}_{\mathrm{err}}\uparrow$	$\mathrm{MCC}_{\mathrm{err}}\uparrow$	$\mathrm{F1}_{\mathrm{err}}\uparrow$	$\mathrm{Rec}_{\mathrm{struct}}\uparrow$	$\mathrm{Prec}_{\mathrm{struct}}\uparrow$	$\mathrm{F1}_{\mathrm{struct}}\uparrow$
Eedi QATD2k	HACHIMI-II	0.659	0.673	0.738	0.509	0.561	0.315	0.368	0.339
	w/o Prior	0.571	0.635	0.834	0.603	0.651	0.300	0.236	0.264
	w/o State	0.741	0.753	0.612	0.322	0.346	0.099	0.220	0.136
	w/o Harness	0.565	0.626	0.808	0.588	0.639	0.307	0.275	0.290
Khan/CoMTA	HACHIMI-II	0.370	0.390	0.660	0.267	0.464	0.565	0.249	0.346
	w/o Prior	0.326	0.359	0.618	0.200	0.426	0.578	0.224	0.323

Appendix

Table 16: Dialogue ablation results.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 127 papers

StudentSim: Training LLM-based Student Simulators
Sep 1, 2026Ke Yang, Chenglong Wang, Michel Galley +4Intelligent Tutoring Systems AI in Education
1Microsoft Research · University of Illinois Urbana-Champaign
INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators
Aug 11, 2026Rose Niousha, Minwoo Kang, Narges NorouziLLM Fine-Tuning CoT Reasoning
Department of Electrical Engineering and Computer Sciences University of California, Berkeley
Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues
May 28, 2026Zhangqi Duan, Shuyan Huang, Alexander Scarlatos +3Large Language Model-Based Role-Play Simulation Intelligent Tutoring Systems
University of Massachusetts Amherst · 2Eedi

Sep 1, 2026J/K move · Enter open · S save

StudentSim: Training LLM-based Student Simulators

Ke Yang, Chenglong Wang, Michel Galley, Chandan Singh+3

1Microsoft Research · University of Illinois Urbana-Champaign

Intelligent Tutoring Systems AI in Education User Simulation

Open paper AskPDF arXiv

AI tutors are most useful when they adapt to each student's strengths, weaknesses, and preferred guidance, but evidence about which guidance works for which student is sparse, slow, and costly to collect from real learners. Student simulators can provide this signal as a proxy, yet existing approaches are limited: state-tracking models fit student behavior but struggle to process explanations or corrections, while LLM role-play follows guidance fluently but does not reliably match the competence of the student being imitated. We present StudentSim, a training framework that turns sparse per-student data into individualized simulators through pooled training followed by per-student specialization. The resulting simulators both mirror a student's own responses and update them under tutor guidance. We also introduce StudentSimEval, a standardized protocol covering 60 students across chess, second-language English writing, and mathematics, using public learner datasets with de-identified records shared for research. StudentSimEval measures behavioral fidelity (F), or how well a simulator matches a student's responses, and guidance responsiveness (R), or how readily it updates under tutor guidance, with all methods fit and evaluated on the same records. Across all three domains, StudentSim outperforms GPT-5.4 on both metrics. In chess, StudentSim reaches F=0.51 and R=0.91, compared with 0.23 and 0.72 for GPT-5.4 and 0.45 and 0.27 for Maia2. As a proof of concept, using StudentSim as a reward model for tutor reinforcement learning produces a chess tutor that expert humans rate as more accurate, better-guided, and more personalized than a no-RL baseline and a tutor trained against a GPT-5.4 simulator reward. Code is available at https://github.com/microsoft/StudentSim.

Abstract

User Simulation

Explore similar work

StudentSim: Training LLM-based Student Simulators

INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators

Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues