ClickSight: An LLM Framework for Interpreting Student Clickstreams Through Learning Strategies
Organizations: Machine Learning for Education Laboratory (ML4ED), School of Computer and Communication Sciences, EPFL, INF Building, Station 14, Lausanne, 1015, Switzerland · Max Planck Institute for Software Systems, Campus E1 5, Saarbrücken, 66123, Germany
Abstract
Digital learning environments produce rich clickstream data that can reveal how students approach learning tasks, yet interpreting these fine-grained traces often requires substantial expert effort. We introduce ClickSight, a framework for interpreting student clickstreams through learning strategies defined in the literature. Given a student clickstream and a predefined set of learning strategies, ClickSight uses a staged LLM-based pipeline to identify strategy-relevant evidence, assess adherence to each strategy, and generate an evidence-grounded interpretation of the student's behavior. We evaluate ClickSight across two learning environments, 365 students, and 12 model backbones using an expert-grounded quality rubric and an LLM judge calibrated with human-expert ratings. Our results show that ClickSight can generate high-quality strategy-based interpretations across different models and environments, with reasoning models performing more consistently than instruction-tuned counterparts. We further find that providing external clickstream-extracted guidance can improve interpretation quality for strategies that require precise quantitative information from the trace. By connecting fine-grained clickstreams with theoretically defined learning strategies and preserving the evidence underlying each judgment, ClickSight provides a flexible approach for studying student behavior in digital learning environments.
Figures & tables
| Criterion | Key Question | Condition for Score = 1 |
| Completeness | Does the interpretation provide an adherence assessment for ? | The interpretation contains an assessment of the student’s adherence to , regardless of correctness. |
| Correctness | Does the interpretation correctly assess adherence for ? | The stated adherence level is semantically consistent with the corresponding ground-truth label. |
| Justifiedness | Does the interpretation justify the stated adherence assessment for using evidence from the student’s clickstream? | The interpretation provides reasoning that supports the stated adherence assessment, and all referenced actions are present in the clickstream. |
| Comprehensibility | Is the interpretation written clearly and grammatically, with natural descriptions of actions and timing? | The interpretation is grammatically coherent, describes actions in natural language rather than technical action codes, and expresses temporal information using human-readable references such as interaction phases or halves. |
| Environment | Method | Completeness | Correctness | Justifiedness | Comprehensibility | Overall |
| PharmaSim | Zero-shot | 1.000 | 0.958 ( 0.008) | 0.977 ( 0.008) | 1.000 | 0.938 ( 0.011) |
| Chain-of-Thought | 1.000 | 0.959 ( 0.008) | 0.964 ( 0.012) | 1.000 | 0.924 ( 0.014) | |
| Chain-of-Prompts | 1.000 | 0.961 ( 0.008) | 0.984 ( 0.009) | 1.000 | 0.945 ( 0.011) | |
| Beer’s Law Lab | Zero-shot | 1.000 | 0.782 ( 0.014) | 0.851 ( 0.014) † | 1.000 | 0.704 ( 0.016) |
| Chain-of-Thought | 1.000 | 0.797 ( 0.012) | 0.830 ( 0.014) | 1.000 | 0.702 ( 0.015) | |
| Chain-of-Prompts | 1.000 | 0.815 ( 0.012) ∗ | 0.844 ( 0.014) | 1.000 | 0.735 ( 0.015) ∗† |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Instruction | Examples |
| Evidence Filtration | Filter the actions that ask questions about the patient. | [discuss(mother, symptoms), discuss(baby, symptoms), ...] [discuss(mother, symptoms), discuss(mother, diet)] |
| Evidence Analysis | For each piece of evidence (i.e., each question asked about the patient), check if it corresponds to one of the checklist topics. | [discuss(mother, symptoms), discuss(mother, diet)] [discuss(mother, symptoms) covers nature of symptoms, discuss(mother, diet) covers worsening factors] |
| Adherence Assessment | If the evidence shows that all LINDAFF topics were addressed, mark the adherence level as followed . Otherwise, mark it as not followed . Justify the adherence level for: - Not Followed : Clearly state which LINDAFF topics were missing, or list only the subset covered. - Followed : Clearly state that all components were addressed and support with examples. | [discuss(mother, symptoms) covers nature of symptoms, discuss(mother, diet) covers worsening factors] Not followed: The student did not cover intensity, location, duration, medical history, or relieving factors. |
| Interpretation Generation | Ensure the interpretation meets the required criteria and is supported with evidence… | [discuss(mother, symptoms), discuss(baby, symptoms), ...] The student did not follow the LINDAFF checklist because they did not address symptom intensity, location, duration, relevant medical history, or relieving factors. |
| Adherence Level | Condition for Score = 1 (Justified) |
| Not Followed | The interpretation explicitly states that the student did not explore the variable associated with the aspect at all. |
| Low / Medium / High | The interpretation specifies all of the following: 1. Number of explorations : how many explorations of the variable related to the aspect were performed. 2. Adherence breakdown : how many, or what percentage, of those explorations followed the strategy. 3. Half of the interaction : whether most explorations occurred in the first or second half of the interaction. |
| Environment | Criterion | Human–human ( ) | LLM– human |
| PharmaSim | Completeness | 1.00 (1.00) | 1.00 |
| Correctness | 1.00 (1.00) | 1.00 | |
| Justifiedness | 0.85 (0.75) | 0.70 | |
| Comprehensibility | 1.00 (1.00) | 1.00 | |
| Beer’s Law Lab | Completeness | 1.00 (1.00) | 1.00 |
| Correctness | 0.98 (0.82) | 0.95 |
| Criterion | Contrast | 95% CI | ||||
| PharmaSim | ||||||
| Correctness | Chain-of-Prompts Zero-shot | .732 | .999 | |||
| Chain-of-Prompts Chain-of-Thought | .853 | .999 | ||||
| Chain-of-Thought Zero-shot | .841 | .999 | ||||
| Justifiedness | Chain-of-Prompts Zero-shot | .534 | .785 | |||
| Chain-of-Prompts Chain-of-Thought | .262 | .785 | ||||
| Model | Config. | Complete | Correct | Justified | Compreh. | Overall |
| Qwen3-4B | Instruct | 1.000 | 0.512 ( 0.023) | 0.478 ( 0.020) | 0.974 ( 0.011) | 0.385 ( 0.021) |
| Qwen3-4B | Thinking | 1.000 | 0.839 ( 0.015) | 0.924 ( 0.010) | 1.000 | 0.793 ( 0.017) |
| Qwen3-30B | Instruct | 0.972 ( 0.008) | 0.489 ( 0.023) | 0.611 ( 0.022) | 0.964 ( 0.009) | 0.434 ( 0.022) |
| Qwen3-30B | Thinking | 1.000 | 0.896 ( 0.013) | 0.953 ( 0.008) | 0.985 ( 0.008) | 0.846 ( 0.016) |
| Qwen3-80B | Instruct | 1.000 | 0.593 ( 0.022) | 0.660 ( 0.019) | 0.997 ( 0.003) | 0.520 ( 0.022) |
| Qwen3-80B | Thinking | 0.997 ( 0.003) | 0.831 ( 0.016) | 0.922 ( 0.011) | 1.000 | 0.789 ( 0.018) |
| Model | Config. | Complete | Correct | Justified | Compreh. | Overall |
| Qwen3-4B | Instruct | 0.956 ( 0.008) | 0.596 ( 0.016) | 0.263 ( 0.015) | 0.987 ( 0.007) | 0.147 ( 0.013) |
| Qwen3-4B | Thinking | 0.988 ( 0.005) | 0.690 ( 0.016) | 0.410 ( 0.017) | 1.000 | 0.335 ( 0.017) |
| Qwen3-30B | Instruct | 0.890 ( 0.014) | 0.642 ( 0.015) | 0.307 ( 0.015) | 0.966 ( 0.012) | 0.182 ( 0.015) |
| Qwen3-30B | Thinking | 0.983 ( 0.004) | 0.791 ( 0.013) | 0.478 ( 0.018) | 0.988 ( 0.009) | 0.408 ( 0.018) |
| Qwen3-80B | Instruct | 0.989 ( 0.004) | 0.615 ( 0.016) | 0.337 ( 0.018) | 1.000 | 0.224 ( 0.018) |
| Qwen3-80B | Thinking | 0.999 ( 0.001) | 0.812 ( 0.012) | 0.512 ( 0.020) | 1.000 | 0.451 ( 0.019) |
| Contrast | [95% CI] | Correction family | |||
| Qwen3-4B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3-30B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3-80B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3 30B-Thinking 4B-Thinking | .005 | .012 | Qwen3 scale ( ) | ||
| Qwen3 80B-Thinking 30B-Thinking | .004 | .012 | Qwen3 scale ( ) | ||
| Qwen3 80B-Thinking 4B-Thinking | .883 | .883 | Qwen3 scale ( ) |
| Contrast | [95% CI] | Correction family | |||
| Qwen3-4B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3-30B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3-80B Thinking Instruct | .001 | .001 | thinking vs. instruct ( ) | ||
| Qwen3 30B-Thinking 4B-Thinking | .001 | .001 | Qwen3 scale ( ) | ||
| Qwen3 80B-Thinking 30B-Thinking | .066 | .066 | Qwen3 scale ( ) | ||
| Qwen3 80B-Thinking 4B-Thinking | .001 | .001 | Qwen3 scale ( ) |
| Model | Guidance | Complete | Correct | Justified | Compreh. | Overall |
| Qwen3-80B-Thinking | None (baseline) | 0.998 ( 0.002) | 0.849 ( 0.016) | 0.868 ( 0.013) | 1.000 | 0.756 ( 0.019) |
| Qwen3-80B-Thinking | Instructions | 1.000 | 0.836 ( 0.017) | 0.894 ( 0.013) | 1.000 | 0.763 ( 0.020) |
| Qwen3-80B-Thinking | Examples | 0.995 ( 0.005) | 0.828 ( 0.016) | 0.891 ( 0.013) | 0.995 ( 0.005) | 0.751 ( 0.018) |
| Qwen3-80B-Thinking | Action indices | 1.000 | 0.859 ( 0.016) | 0.904 ( 0.012) | 1.000 | 0.793 ( 0.018) |
| Qwen3-80B-Thinking | Action counts | 1.000 | 0.834 ( 0.016) | 0.885 ( 0.014) | 1.000 | 0.758 ( 0.018) |
| GPT-OSS-120B | None (baseline) | 1.000 | 0.928 ( 0.010) | 0.951 ( 0.009) | 1.000 | 0.889 ( 0.013) |
| Model | Guidance | Complete | Correct | Justified | Compreh. | Overall |
| Qwen3-80B-Thinking | None (baseline) | 0.999 ( 0.001) | 0.812 ( 0.012) | 0.512 ( 0.020) | 1.000 | 0.451 ( 0.019) |
| Qwen3-80B-Thinking | Instructions | 0.996 ( 0.003) | 0.811 ( 0.014) | 0.564 ( 0.019) | 1.000 | 0.476 ( 0.019) |
| Qwen3-80B-Thinking | Examples | 0.998 ( 0.002) | 0.838 ( 0.011) | 0.600 ( 0.020) | 1.000 | 0.517 ( 0.019) |
| Qwen3-80B-Thinking | Action indices | 0.997 ( 0.002) | 0.800 ( 0.012) | 0.735 ( 0.017) | 1.000 | 0.601 ( 0.018) |
| Qwen3-80B-Thinking | Action counts | 1.000 | 0.819 ( 0.012) | 0.830 ( 0.013) | 1.000 | 0.703 ( 0.015) |
| GPT-OSS-120B | None (baseline) | 1.000 | 0.767 ( 0.012) | 0.688 ( 0.017) | 1.000 | 0.574 ( 0.016) |
| Contrast | [95% CI] | Holm family | |||
| GPT-OSS-120B: instructions baseline | .836 | .999 | baseline ( ) | ||
| GPT-OSS-120B: examples baseline | .965 | .999 | baseline ( ) | ||
| GPT-OSS-120B: action indices baseline | .119 | .358 | baseline ( ) | ||
| GPT-OSS-120B: action counts baseline | .001 | .001 | baseline ( ) | ||
| Qwen3-80B-Thinking: instructions baseline | .606 | .999 | baseline ( ) | ||
| Qwen3-80B-Thinking: examples baseline | .476 | .999 | baseline ( ) |
| Contrast | [95% CI] | Holm family | |||
| GPT-OSS-120B: instructions baseline | .084 | .167 | baseline ( ) | ||
| GPT-OSS-120B: examples baseline | .837 | .837 | baseline ( ) | ||
| GPT-OSS-120B: action indices baseline | .001 | .001 | baseline ( ) | ||
| GPT-OSS-120B: action counts baseline | .001 | .001 | baseline ( ) | ||
| Qwen3-80B-Thinking: instructions baseline | .086 | .086 | baseline ( ) | ||
| Qwen3-80B-Thinking: examples baseline | .003 | .005 | baseline ( ) |
| Environment | Metric | Median | IQR | Min | Max | Skew | |
| Beer’s Law Lab | Event count | 160 | 38 | 32.5 | 5 | 217 | 1.95 |
| Tokens per event | 160 | 229 | 77.2 | 119 | 555 | 1.37 | |
| Formatted token count | 160 | 9,568 | 8,250 | 889 | 39,241 | 1.43 | |
| PharmaSim | Event count | 205 | 22 | 15 | 1 | 59 | 0.38 |
| Tokens per event | 205 | 51 | 2.6 | 44 | 78 | 3.77 | |
| Formatted token count | 205 | 1,108 | 745 | 50 | 2,817 | 0.37 |
| Comparison | Environment | Correctness | Justifiedness |
| Unrestricted | Beer’s Law Lab ( ratings, 160 students) | 0.798 ( 0.011) | 0.845 ( 0.012) |
| PharmaSim ( ratings, 205 students) | 0.959 ( 0.006) | 0.975 ( 0.006) | |
| Token-matched a | Beer’s Law Lab ( ratings, 14 students) | 0.905 ( 0.025) | 0.966 ( 0.011) |
| PharmaSim ( ratings, 94 students) | 0.959 ( 0.008) | 0.974 ( 0.008) | |
| PharmaSim heaviest tercile | long ( ratings, 69 students) | 0.940 ( 0.013) | 0.952 ( 0.014) |
| rest ( ratings, 136 students) | 0.969 ( 0.006) | 0.987 ( 0.005) |
Educational Data Mining
Educational Data Mining studies computational methods for analyzing learning data to understand learners, improve instruction, and support educational decisions.
See all 76 papers