Modeling Student Sensemaking with LLMs and Knowledge-Graph-Guided Inference
Authors: Özge Alacam, Zübeyde Demet Kirbulut Güneş, Funda Ekici, Nurcan Turan-Oluk, Dilay Dinçdemir, Hakkı Kadayıfçı, Sevinç Nihal Yeşiloğlu, Burcu Işık, Halil Tümay, Sinem Gencer
Organizations: Center for Information and Language Processing, LMU München, Munich, Germany · Department of Chemistry Education, Gazi Faculty of Education, Gazi University, Ankara, Türkiye
Collaborative science learning requires nuanced interpretation of student dialogue to characterize how learners identify knowledge gaps, build explanations, and work toward resolution - a theory-driven analysis that is labor-intensive and difficult to scale. We investigate whether instruction-tuned large language models (LLMs) can support multidimensional analysis of collaborative sensemaking without task-specific training, and whether structured knowledge-state information improves model inference. We evaluate two mid-size LLMs on 23 richly annotated, expert-labeled episodes across prompting conditions that vary definitional scaffolding, reasoning mode, and turn structure. Without reasoning, models tend to overpredict successful sensemaking; reasoning-enabled prompting improves identification of unsuccessful cases. Knowledge-state diagnostics provide additional grounding, improving detection of unsuccessful sensemaking and increasing agreement with expert annotations. No single configuration performs best across all sensemaking dimensions, underscoring the multidimensional nature of the task.
Figures & tables
Figure 1: The proposed experimental pipeline for LLM-based sensemaking analysis.
Episode
Mean Text Length
Mean Turn Count
Count
(SD)
(SD)
SM
16
5863 words (3968)
44 turns (39)
USM
7
8726 words (6010)
46 turns (33)
Table 1: Descriptive statistics for SM and USM episodes (for episode length and number of turns).
Model Config
Best Prompt
F1-SM
F1-USM
Cohen κ
Gemma-3-27B No-Thinking
V2
81%
40%
0.22 (weak)
Gemma-3-27B Thinking
V1
84.9%
61.5 %
0.45 (moderate)
Qwen3-32B No-Thinking
V1-3
86.5%
44.4%
0.36 (fair)
Qwen3-32B Thinking
V3
88.9%
60%
0.51 (moderate)
Table 2: Sensemaking Classification (single-turn)
Model Config
Best Prompt
F1-SM
F1-USM
Cohen κ
No-Thinking, single-turn
V1, 3, 4
86.5%
44.4%
0.36 (fair)
No-Thinking, two-turn
V2
75.9%
58.8%
0.36 (fair)
Thinking, single-turn
V3
88.9%
60.%
0.51 (moderate)
Thinking, two-turn
V3
76.9%
70.%
0.50 (moderate)
KG-informed Variants
No-Thinking, single-turn
-
86.5%
44.%
0.36 (fair)
Table 3: Sensemaking Classification (Qwen3-32B)
Figure 2: Cosine Similarity between expert-produced and model-generated features.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Excerpts from a successful collaborative sensemaking episode illustrating the temporal unfolding of sensemaking across Steps 1–3, the focal knowledge gap, recurring VQs, CER components, and complementary emotional cues.
Figure 4: QWEN3-32B Models’ Emotion Distribution for SM & USM
Figure 5: Prompt Template 1a: Domain Ontology Construction (Entity Extraction)
Figure 6: Prompt Template 1b: Domain Ontology Construction (Relation Extraction)
Figure 7: Prompt Template: Classification with LLM from raw text)
Figure 8: Prompt Template: KG metrics interpretation guide)
Figure 9: Prompt Template: Classification with LLM from raw text (part2))
Figure 10: Student KGs from successful and unsuccessful sense-making episodes.
Model
V1
V2
V3
V4
Gemma No-Think
0.172
0.222
0.172
0.172
Gemma Thinking
0.455
0.371
0.416
0.432
Qwen3 No-Think
0.358
0.188
0.358
0.358
Qwen3 Thinking
0.416
0.021
0.511
0.358
Appendix
Table 4: Sensemaking variation prediction with prompt variants
School of Computing and Information Systems Singapore Management University · Department of Management Science and Information Systems Rutgers Business School
Apr 2, 2026·Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths +2Theory of Mind
Department of Psychology, Princeton University · Department of Psychology, New York University · Department of Computer Science, Department of Psychology, Princeton University +1