Past research using log data has faced the "learning system wall," whereby few methods exist for generalizing models of student learning across platforms. Increasingly, online learning is captured by richer forms of data, including dialog and video, with new affordances. An example of this is remote tutoring programs, where human tutors support students who use learning systems while video conferencing. Toward better platform-general modeling of learning, we introduce an AI-driven multimodal transcription system that processes screen-recording videos into unified screenplay-style transcripts containing audio dialogue and annotated learning log actions. We describe a planned method for temporally aligning AI-generated multimodal transcripts with MATHia learning logs and for identifying and classifying student learning processes to align with MATHia logs. Lastly, we highlight challenges and potential solutions in capturing learning processes in one system, offering initial steps towards generalizing log data across diverse systems.
Figures & tables
Figure 1: Left: A screenshot of a student screen-share recording during remote tutoring. The student is engaging with the MATHia learning platform with the support of a remote human tutor. Right: A student is using an interactive whiteboard while receiving support from a remote human tutor.
Figure 2: AI-generated multimodal transcript development pipeline.
Figure 3: Temporally-merged multimodal transcript (audio, chat, and MATHia logs) development pipeline.
Time
AI-generated multimodal transcript (Fig. 2 )
Temporally-merged transcript (Fig. 3 )
13:21–13:27
Tutor : I think it will be easy for me to explain. So, yeah, I think, okay.
Tutor : Your screen might stop sharing, because I think it would be easy for me to explain.
13:28–14:11
[view change event: The tutor’s whiteboard appears on the screen.]
14:12–14:30
[drawing event: tool=pen, Tutor writes the equation ‘74 = x + x - 10’ on the whiteboard.]
14:33–14:39
Tutor : Now we have a simple equation and we just need to find the value of x .
Tutor: Now we have a simple equation and we just need to find the value of x .
14:41–14:48
Tutor : Um, I think even you can write. So if you want to write on the whiteboard itself, you can feel free to go ahead and write as well.
Tutor : I think even you can write. So if you want to write on the whiteboard itself, you can be free to go ahead and write as well.
15:44–15:47
[drawing event: tool=pen, The student starts to write ‘84’.]
Table 1: An example of AI-generated multimodal transcription (left) and a corresponding temporally-merged multimodal transcription of audio, chat, and MATHia log data (right). AI-generated transcriptions can capture pedagogical and mathematical events not captured in available data streams, such as a student using an interactive whiteboard outside of the math learning system (its absence is shown in grey boxes.)
Event Type
Input/AI-Generated Multimodal Transcription
Output/Ground Truth (MATHia)
attempt
Visual detection of keyboard input, mouse drags, or system feedback while using MATHia containing “type”, “types”, “plots”, “drags”, “selects”, etc. (e.g., “Student types 30 into the input box” ; “text ‘13’ into the reflection line value box” ). Correct : “green checkmark appears”, “page advances”, “next screen loads”; Incorrect : “Try again modal appears”, “Error message appears”, “Orange (or red) input highlights”, “try again”, “incorrect”, etc.
Action = “Attempt” contains input text data (e.g., {value = x}) and correctness data (e.g., OK or ERROR).
hint_request
Visual detection of initial hint interaction while using MATHia containing keywords such as “hint” or “hints” (e.g., “Student clicks Hints button” ; “Hint pop-up appears, showing Hint 1 of 3” ; “The next button hint appears” ). Distinguished from hint_level_change by being the first event in the hint sequence.
Action = “Hint Request”, just-in-time “JIT”, or outcome = “INITIAL_HINT”.
hint_level_change
Visual detection of advancing hint states (e.g., “Student clicks Next button in hint pop-up” ; “Hint pop-up changes to show Hint 2 of 3” ).
Action = “Hint Level Change” or outcome = “HINT_LEVEL_CHANGE”.
done
Visual detection of problem-level completion indicated by “done” or “I’m done” (e.g., “Student clicks the I’m Done button” ).
Action = “Attempt” with input = “Done” or step_name = “Done”.
Table 2: Input/output classification used to map AI-generated transcriptions and MATHia ground truth to event type.
There exist numerous tutor training platforms. However, few provide AI-driven training and evaluation for human tutors based on real-life performance. We present an AI-driven system that assesses both open responses during training and authentic real-life tutoring. Unlike platforms that only assess learning through online training or simulations, our system utilizes Generative AI (Gemini-2.5-pro) to analyze transcriptions of authentic tutoring, measuring the transfer of tutor skills to real-life application. Human tutors instructing students remotely in math (N=86) completed six scenario-based lessons, averaging a significant 7.4% learning gain. Using mixed-effects models across 405 session-to-lesson pairs, we found that training performance significantly predicted real-life transcript scores with an effect size of 0.25 SD. Model comparison (AIC/BIC) indicated averaging open response and multiple choice performance during training predicted real-life tutor performance best, although open responses were comparatively more predictive. Exploratory analysis showed that after training, tutors were significantly more likely to encounter pedagogical opportunities to apply their skills (61.1% to 68.9%) and demonstrated higher execution quality within those opportunities (65.5% to 68.1%). Interrupted time series analysis suggested that these tutor improvements were part of a gradual trend over time rather than an immediate intervention effect of training. We illustrate an AI-driven method to link tutor training with real-life assessment. In doing so, we contribute open datasets, AI prompts, and scoring rubrics to support transparency and reproducibility.
Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Clara Brandt +2
Carnegie Mellon University, Pittsburgh, PA, USA · Vanderbilt University, Nashville, TN, USA
Large language models are now widely used for everyday learning, but the underlying interactions are typically unstructured chats rather than following a curriculum. Unlike formal online learning systems, these interactions carry no prior record of the student, so any estimate of what the student already knows must be inferred from the dialogue itself. We show that this gap is not closed by scaling models alone. Frontier and education-tuned LLMs perform poorly when asked to tutor a student over an extended session, because doing so requires three things at once. The tutor must sequence a curriculum, conduct Socratic dialogue, and infer the student's knowledge state from that dialogue. We propose separating these responsibilities. Given a student query, our system constructs a prerequisite knowledge graph in which subtopics are nodes and dependencies are edges, and frames tutoring as deciding which node to teach next and how many dialogue turns to spend on it before moving on. A lightweight PPO policy handles this sequencing decision, while an LLM conducts the Socratic exchange at the chosen node and returns a signal of student progress. Across held-out STEM and non-STEM topics, our PPO-paired tutor outperforms heuristic baselines, frontier general-purpose models, and a model specialised for Socratic dialogue: on both the rate at which students reach full curriculum mastery and the number of turns required. Explicit curriculum structure delivers gains that scaling the underlying model does not.
Sidney Tio, Arunesh Sinha, Pradeep Varakantham
School of Computing and Information Systems Singapore Management University · Department of Management Science and Information Systems Rutgers Business School
Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutorial, a framework that transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials for teaching both humans and agents. Demo2Tutorial first collects human experience via a dedicated recorder, then parses raw experience using a multimodal Action Parser to reconstruct perception, action, and intent. A Step Planner then abstracts these steps into hierarchical task graphs representing goals and steps. Finally, a Tutorial Composer transforms the parsed experience into structured, reusable image-text instructions. We evaluate the tutorial generation quality on a new benchmark derived from official software documentation. We further demonstrate that this distilled representation benefits (i) human learning, by automatically generating multimodal tutorials, and (ii) agent learning, by improving downstream GUI-agent planning and generalization. Experiments show Demo2Tutorial produces high-quality tutorials that surpass human-authored ones and significantly outperform baseline methods, while enabling both faster human task completion and improved GUI agent planning, demonstrating that structured tutorials distilled from human experience can serve as effective knowledge representations for advancing both human learning and agent capabilities. Code and data will be available at https://github.com/showlab/Demo2Tutorial.