Worked examples are an important part of introductory programming, but reading their expert explanations is passive. Self explanation, students explaining the problem and its solution to themselves with subgoal level analysis, converts passive reading into an active study of worked example, yet it is hard to scale because assessing free-text explanations and returning timely feedback has had no easy automated solution. We investigate whether a large language model (LLM) can fill that gap. We build a self-explanation tutor for introductory programming, ESSE, in which students explain lines of worked examples and receive immediate LLM feedback on the correctness and completeness of each explanation, and we pursue two goals. First, we ask whether the LLM judges student explanations well enough to serve as the engine of the tutor; we assess its judgments against two independent human reference standards of different kinds, a single domain expert and a crowd of non-expert raters, each with its own strengths and weaknesses, characterizing both where the LLM is reliable and the systematic tendencies in how it diverges. Second, we ask whether the LLM-based tutoring benefits students; deploying it in an introductory Java course, we find that its feedback leads students to persist and revise rather than abandon a line, that their explanations grow more complete and conceptually richer across attempts, and that students show evidence of learning. These indicate that LLM-based assessment is good enough to power a self-explanation tutor, and that the tutor positively shapes how students study worked examples.
Figures & tables
Figure 1. The ESSE self-explanation tutor. Students explain a selected line of a worked example in their own words and receive immediate LLM feedback on the correctness and completeness of the explanation.
Dataset
Source
Size
Used for
CS1 study
Deployment
N=8 ; 409 attempts
RQ1, RQ2
Crowd
MTurk
124 raters; 1,696 ratings
RQ1
Table 1. The two datasets and the research questions they support.
Current attempt
Attempts
Revise
by correctness
Incorrect
62
63%
Correct
296
35%
by completeness
≤25%
14
64%
25–50%
271
42%
Table 2. Share of attempts followed by another attempt (“revise”), by the correctness and completeness of the current attempt. Students revise more when the feedback flags a problem.
Statement (abbreviated)
A
N
D
1a
Understood “correct” definition
7
1
0
1b †
Did not understand “complete” definition
1
2
5
1c
Feedback helped write longer explanations
5
1
2
1d
Feedback helped recall key concepts
7
1
0
1e †
Feedback was irrelevant
1
1
6
1f †
Feedback was incorrect or misleading
0
2
6
Table 3. Survey responses ( N=8 ): counts agreeing (A; strongly agree + agree), neutral (N), and disagreeing (D; disagree + strongly disagree). † Items phrased so that disagreement is the favorable response.
Figure 3. Students’ attitudes toward using the ESSE system.
Worked examples are step-by-step solutions to problems in a specific domain, offered to students to acquire domain-specific problem-solving skills. The effectiveness of worked examples could be enhanced by combining them with self-explanations, which ask students to explain rather than passively study each problem-solving step. The main challenge of this approach is assessing the correctness of the student's explanations. In the prevailing approach, student explanations are judged by their semantic similarity to an instructor's or domain expert's explanation. Given recent advances in LLM-based automated scoring, it remains unclear whether semantic similarity methods are still the most effective technique to automatically score textual student responses like essays or code explanations. Comparing these methods also requires quality datasets that offer distinctive features such as balanced class distributions and domain-specific labeled data for automated scoring tasks. In this paper, we present a rigorous comparison between LLMs and semantic similarity used for automated scoring, framed as a binary classification task.
Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Peter Brusilovsky
We introduce ACE-TA, the Agentic Coding and Explanations Teaching Assistant framework, that autonomously routes conceptual queries drawn from programming course material to grounded Q&A, stepwise coding guidance, and automated quiz generation using pre-trained Large Language Models (LLMs). ACE-TA consists of three coordinated modules: a retrieval grounded conceptual Q&A system that provides precise, context-aligned explanations; a quiz generator that constructs adaptive, multi-topic assessments targeting higher-order understanding; and an interactive code tutor that guides students through step-by-step reasoning with sandboxed execution and iterative feedback.
University of Alabama, Tuscaloosa, Alabama, USA · Brown University, Providence, Rhode Island, USA · Meharry Medical College, Nashville, Tennessee, USA +1
Active learning is a general learning mechanism shared by artificial and human learners. Whether AI can teach humans such a strategy that transfers across domains is an open question. Ultra Strong Machine Learning (USML), a system whose explanations quantifiably improve human out-of-sample performance compared to self-learning, is uniquely positioned to answer this question. Prior USML work relied on hand-crafted explanation templates that require expert effort for each new domain and do not scale. We developed an explanation pipeline combining Inductive Logic Programming (ILP) with large language models (LLMs) to automate explanation generation and scoring. We tested whether these explanations achieve USML in a human trial teaching active learning strategies across three related domains. Our exploratory results show that concise, expert-written explanations benefit learners with higher initial performance, while pipeline-generated explanations provide no advantage over self-learning despite being rated as higher quality from an LLM-as-judge evaluation. This case study reveals a systematic gap that LLM quality metrics do not predict human learning outcomes. Our findings point to explanation complexity relative to task difficulty as a key factor, and call for explanation methods and evaluation criteria grounded in human cognitive constraints rather than LLM preference.
Lun Ai, Johannes Langer, Ute Schmid +1
Department of Computing, Imperial College London, UK · Faculty Information Systems and Applied Computer Science, University of Bamberg, Germany