cs.CLMar 25, 2026

Representation Learning to Study Temporal Dynamics in Tutorial Scaffolding

Authors: Conrad Borchers, Jiayi Zhang, Ashish Gurung

Organizations: Carnegie Mellon University · Worcester Polytechnic Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adaptive scaffolding enhances learning, yet the field lacks robust methods for measuring it within authentic tutoring dialogue. This gap has become more pressing with the rise of remote human tutoring and large language model-based systems. We introduce an embedding-based approach that analyzes scaffolding dynamics by aligning the semantics of dialogue turns, problem statements, and correct solutions. Specifically, we operationalize alignment by computing cosine similarity between tutor and student contributions and task-relevant content. We apply this framework to 1,576 real-world mathematics tutoring dialogues from the Eedi Question Anchored Tutoring Dialogues dataset. The analysis reveals systematic differences in task alignment and distinct temporal patterns in how participants ground their contributions in problem and solution content. Further, mixed-effects models show that role-specific semantic alignment predicts tutorial progression beyond baseline features such as message order and length. Tutor contributions exhibited stronger grounding in problem content early in interactions. In contrast, student solution alignment was modestly positively associated with progression. These findings support scaffolding as a continuous, role-sensitive process grounded in task semantics. By capturing role-specific alignment over time, this approach provides a principled method for analyzing instructional dialogue and evaluating conversational tutoring systems.

Figures & tables

Figure 1: Density distributions of semantic alignment scores for tutor and student moves. (A) Alignment to the problem statement. (B) Alignment to the correct solution.
Figure 2: Temporal evolution of semantic alignment. (A) Alignment to the problem statement. (B) Alignment to the correct solution.
More in this topic

Instructional Scaffolding

See all 11 papers

Explore similar work

CardsList
Oct 14, 2026

A Unified Response Framework for Questions and Tutoring Dialogues

Existing student models in educational applications largely focus on student responses to questions. However, as tutoring becomes more accessible at large scale, it is important to analyze dialogues between tutors and students, which may reveal fine-grained evidence on student knowledge. In this work, we propose a unified framework that models both student responses to questions and tutor-posed subtasks during dialogues on a shared item response theory (IRT) scale. We first identify mathematical subtasks in tutoring dialogues, rewrite them into self-contained questions, and identify the correctness of student responses to these subtasks. We then develop an IRT framework that estimates question/subtask difficulty from text, tracks students' evolving abilities across question-answering and dialogue interactions, and a novel method to leverages tutor moves between consecutive subtasks to regularize their difficulty values. We evaluate our framework on two real-world math education datasets containing both human- and AI-tutored dialogues. Results show that our framework outperforms strong baselines on both question difficulty prediction and response correctness prediction.

Zhangqi Duan, Simon Woodhead, Maya Miller-Vedam +1

University of Massachusetts Amherst · Eedi · Khan Academy

Educational AssessmentIntelligent Tutoring SystemsUser ModelingItem Response Theory
AskPDF
Jun 14, 2026

Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution. Alignment and evaluation methods for embedding scaffolding behaviour into chatbots, however, rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation. To examine whether this assumption holds, we introduce an evaluation pipeline around two metrics - Chatbot Scaffolding and Student Uptake - and apply them across nine datasets of 9,490 chats, spanning AI tutor benchmarks and real-world deployments of educational chatbots. Our analysis reveals that while benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall - frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost. We argue that bypassing scaffolding is not necessarily detrimental; rather, it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals. To meaningfully evaluate the effectiveness of a chatbot's assistance, future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how these chatbots navigate diverse learning contexts and student-driven interaction patterns.

Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer +2

Department of Mechanical Engineering, Imperial College London, UK · Department of Electrical Engineering, Imperial College London, UK · Department of Earth Science and Engineering, Imperial College London, UK

Instructional ScaffoldingIntelligent Tutoring SystemsAI in EducationHuman-AI InteractionGenerative AI Evaluation
AskPDF
Apr 3, 2026

Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts

Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's κκ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines (κ=0.275κ= 0.275-0.4130.413 and 0.1600.160-0.4100.410). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.

Jinsook Lee, Kirk Vanacore, Zhuqian Zhou +2

Cornell University Ithaca, NY, 14850, USA

LLM-Assisted AnnotationRetrieval-Augmented ICLEducational Dialogue SystemsInformation Retrieval
AskPDF