From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation

Yuan An; Emily Wang; Benjamin Wang; Ruhma Hashmi

cs.AIAug 21, 2026

From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation

Authors: Yuan An, Emily Wang, Benjamin Wang, Ruhma Hashmi

Organizations: School of Computer and Information Sciences, Nick Howley College of Engineering and Computing Drexel University Philadelphia, PA 19104, USA

Abstract

Large language models (LLMs) are increasingly used to simulate students at different mastery levels to generate synthetic training data and stress-test tutoring systems. Existing approaches primarily leverage an LLM to answer questions or perform tests while instructing it to simulate various skill levels. However, LLMs tend to execute at their full built-in capability regardless of target skill levels. We demonstrate this limitation on SAT questions across five mastery levels, where three LLMs consistently achieve 96.8-100% accuracy. To faithfully simulate student behavior, we propose a Stochastic Student Knowledge Graph (SSKG)-based approach that leverages the strengths of LLM in different ways. Using human-in-the-loop workflows, LLMs extract a Curriculum Knowledge Graph (CKG) from course materials, decompose solution paths into CKG triples, and map incorrect distractors. A student's mastery profile is represented as an SSKG, assigning mastery probabilities to CKG triples. When solving a problem, the SSKG is sampled to determine the outcome, after which an LLM generates a rationale consistent with the sampled result. Compared to seven baselines which fail to separate profiles and maintain 86.5-100% accuracy, our SSKG-based approach achieves 44.1-85.2% accuracy from low to high-mastery profiles, creating a clear monotonic gradient. An ablation study demonstrates that taxonomy awareness contributes most significantly to these fidelity gains, particularly for lower-mastery profiles.

Figures & tables

Fig. 1: Traditional LLM-driven student simulation (a) versus SSKG-grounded simulation (b). In the traditional pipeline (a), a question item and a student profile are handed directly to an LLM, which decides both the simulated answer and its correctness. In our pipeline (b), a Curriculum Knowledge Graph (CKG) is built as a set of fine-grained triples extracted from the source curriculum. A student profile is represented as a Stochastic Student Knowledge Graph (SSKG), which assigns a mastery probability to each CKG triple at a time point. Each question is mapped to the chain of triples required to solve it. We simulate a student profile over an item by sampling over the item’s chain against the SSKG’s mastery probabilities to decide whether the response is correct. The resulting item paired with this correctness label is then passed to the LLM, which narrates a first-person rationale consistent with the outcome.

Question Stem	Alan drives an average of 100 miles each week. His car can travel an average of 25 miles per gallon of gasoline. Alan would like to reduce his weekly expenditure on gasoline by $5. Assuming gasoline costs$ 4 per gallon, which equation can Alan use to determine how many fewer average miles, $m$ , he should drive each week?
Choices	(A) $\tfrac{25}{4}m=95$ (B) $\tfrac{25}{4}m=5$ (C) $\tfrac{4}{25}m=95$ (D) $\tfrac{4}{25}m=5$
Correct answer	D
Skill	Linear equations in one variable
Difficulty	Hard (College-Board-calibrated)
Rationale	Choice D is correct. Since gasoline costs $4 per gallon, and since Alan’s car travels an average of 25 miles per gallon, the expression$ 4/25 $gives the cost, in dollars per mile, to drive the car. Multiplying by$ m $gives the cost for Alan to drive$ m $miles in his car. Alan wants to reduce his weekly spending by$ 5, so setting $4/25\,m$ equal to 5 gives the number of miles, $m$ , by which he must reduce his driving.

TABLE I: Illustrative example: An SAT Algebra Question.

Fig. 2: An excerpt of a CKG’s number-system connections

Id	Abbreviated triple ( subject , predicate , object )	Type
T19	( Variable , is , A letter representing a changeable number )	def.
T149	( Converting units , uses , The identity property of mult. )	rel.
T150	( Making unit conversions , has_step_1 , Multiply by a unit fraction )	proc.
T170	( Translating a sentence , has_step_1 , Locate the “equals” word )	proc.
T171	( Translating a sentence , has_step_2 , Translate the left side )	proc.
T172	( Translating a sentence , has_step_3 , Translate the right side )	proc.

TABLE II: Excerpt of the illustrative item’s triple chain.

Fig. 3: The illustrative item’s triple chain and distractor map: failures localized at T149/T150 and T170–T172 route to choices B and C, respectively.

Fig. 4: A procedure node: a precondition triple for the Finding the

x

-intercept of a line procedure.

Profile	Archetype	Mastery Probability Generation
P1	near-expert	$U(0.85,0.99)$ all triples
P2	average, inconsistent	$U(0.50,0.70)$ all triples
P3	struggling	$U(0.10,0.30)$ all triples
P4	prerequisite gap	$U(0.10,0.25)$ early, $U(0.60,0.80)$ later
P5	strong basics, weak advanced	$U(0.80,0.95)$ early, $U(0.15,0.35)$ later

TABLE III: Student profiles as SSKG snapshots: natural-language archetype and per-triple mastery generation.

Arm	Adds
a	Single draw $\mathrm{ok}_{i}\sim\mathrm{Bernoulli}(p_{i})$ per triple, raw taxonomy-blind $p_{i}$ , guess floor 0.25 on failure.
b	Retrieval/execution decomposition: $\mathrm{knows}_{i}\sim\mathrm{Bernoulli}(p_{i})$ , then $\mathrm{slips}_{i}\sim\mathrm{Bernoulli}(s{=}0.05)$ ; still taxonomy-blind $p_{i}$ .
c	Taxonomy awareness: definitional $p_{i}\to p_{\mathrm{def}}$ per Eq. ( 7 ) ( $\kappa_{\mathrm{def}}{=}0.3$ ); active set $C(q)$ in Eq. ( 6 ) drops to procedural/relational/gap steps (definitional failures are narration-only).
d	Distractor routing: an execution failure at a mapped triple selects that triple’s specific wrong-answer choice instead of a uniform-random one.

TABLE IV: The four-arm ablation ladder. Each arm adds one mechanism on top of the previous.

ID	Method	What controls ability
B1	Vanilla LLM, no student description	nothing
B2	Persona prompt [ 8 , 7 ]	the prompt
B3	Level prompt, 1 to 5 [ 6 ]	the prompt
B4a	Skill-list prompt, student voice [ 8 ]	the prompt
B4b	Skill-list prompt, teacher predicts student [ 8 ]	the prompt
B5	Persona plus chain-of-thought [ 28 ]	the prompt

TABLE V: Baselines for Comparison

College Board skill	Profile P4	Profile P5
Linear equations in one variable	not learned (W)	mastered (S)
Linear equations in two variables	shaky (S)	not learned (W)
Linear functions	shaky (S)	not learned (W)
Systems of two linear equations	shaky (S)	not learned (W)
Linear inequalities	shaky (S)	not learned (W)

TABLE VI: Weak Skills (W) and Strong Skills (S)

Method	P1	P2	P3	P4	P5
B1	1.000	1.000	1.000	1.000	1.000
B2	1.000	0.997	0.968	1.000	1.000
B3	0.987	0.984	0.897	n/a	n/a
B4a	1.000	0.997	0.968	0.997	0.992
B4b	1.000	0.894	0.865	0.912	0.976
B5	1.000	1.000	1.000	1.000	1.000

TABLE VII: Accuracy by method and profile (Eq. 8 )

Method	$M$	95% CI
B1	0.000	[0.000, 0.000]
B2	0.901	[0.866, 0.990]
B3	0.879	[0.659, 0.950]
B4a	0.901	[0.866, 0.961]
B4b	0.950	[0.700, 0.999]
B5	0.000	[0.000, 0.000]

TABLE VIII: M2, mastery ordering (Eq. 9 ), with bootstrap 95% CI.

Method	P4 gap	P5 gap
B1	+0.000	+0.000
B2	+0.000	+0.000
B3	n/a	n/a
B4a	$-$ 0.003	+0.010
B4b	+0.010	+0.030
B5	+0.000	+0.000

TABLE IX: M3, gap localization (Eq. 11 ). A positive gap is the expected direction.

Profile	Method	Early	Mixed	Later
P4 (weak early)	B2	100.0%	100.0%	100.0%
	Ours	39.3%	39.9%	75.0%
P5 (weak later)	B2	100.0%	100.0%	100.0%
	Ours	70.0%	48.3%	62.5%

TABLE X: M3 breakdown. Accuracy by chain footprint (140 early-only, 143 mixed, 96 later-only items).

Fig. 5: Accuracy by profile across the four arms. The b

\to

c step makes the largest change. arm-c and arm-d are identical in accuracy.

Step	Change	Main effect
a $\to$ b	$-9.5$ to $0.0$ pp	mostly P1
b $\to$ c	$+14.0$ to $+19.5$ pp	larger for lower mastery
c $\to$ d	$0.0$ pp	diagnosis only

TABLE XI: Change in accuracy at each step of the ablation.

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 111 papers

Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators
May 12, 2026Heejin Do, Shashank Sonkar, Mrinmaya SachanLLM Evaluation LLM Sycophancy
ETH Zürich, ETH AI Center · University of Central Florida · ETH Zürich
INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators
Aug 11, 2026Rose Niousha, Minwoo Kang, Narges NorouziLLM Fine-Tuning CoT Reasoning
Department of Electrical Engineering and Computer Sciences University of California, Berkeley
"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators
Aug 30, 2026Krisztian Balog, Arild Michel BakkenComputational Cognitive Modeling Human Behavior Simulation
University of Stavanger, Norway

May 12, 2026J/K move · Enter open · S save

Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators

Heejin Do, Shashank Sonkar, Mrinmaya Sachan

ETH Zürich, ETH AI Center · University of Central Florida · ETH Zürich

LLM Evaluation LLM Sycophancy Cognitive Modeling LLM Post-Training

Open paper AskPDF arXiv

Large language models (LLMs) can fluently generate student-like responses, making them attractive as simulated students for training and evaluating AI tutors and human educators. Yet such simulators are typically evaluated by output similarity to real students, not by whether they behave like students with coherent misconceptions during interaction. We introduce a controlled framework for evaluating misconception faithfulness, whether a simulator maintains a misconception-driven belief state and updates selectively when feedback addresses the underlying misconception. Central to our framework is a misconception-contrastive feedback protocol that compares targeted feedback against two controls: misaligned feedback (targeting a different but plausible misconception) and generic feedback (only identifying answer is wrong). We propose Selective Flip Score (SFS), which quantifies how much more often a simulator flips its answer under targeted feedback than under contrastive controls. Across seven LLMs (4B-120B), multiple datasets, and prompting strategies, simulators exhibit near-zero SFS, correcting their answers at similarly high rates regardless of feedback relevance. Further analyses reveal a sycophantic failure mode: models behave less like students with misconceptions but more like problem-solvers who treat any corrective signal as a cue to abandon the simulated belief and re-solve from internal knowledge. To address this, we develop a post-training pipeline spanning supervised fine-tuning (SFT), preference optimization, and reinforcement learning (RL) with an SFS-aligned reward; SFT yields notable gains up to +0.56, and SFS-aligned RL provides more consistent improvements than preference optimization. Our results establish misconception faithfulness as a challenging yet trainable property, motivating a shift from static output matching toward interactive, belief-aware student modeling.

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators

INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators

"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators