Understanding Teacher Revisions of Large Language Model-Generated Feedback
Authors: Conrad Borchers, Luiz Rodrigues, Newarney Torrezão da Costa, Cleon Xavier, Rafael Ferreira Mello
Organizations: Carnegie Mellon University · Federal Technological University of Paraná – UTFPR · Instituto Federal Goiano – IF Goiano · Federal Rural University of Pernambuco – UFRPE
Large language models (LLMs) increasingly generate formative feedback for students, yet little is known about how teachers revise this feedback before it reaches learners. Teachers' revisions shape what students receive, making revision practices central to evaluating AI classroom tools. We analyze a dataset of 1,349 instances of AI-generated feedback and corresponding teacher-edited explanations from 117 teachers. We examine (i) textual characteristics associated with teacher revisions, (ii) whether revision decisions can be predicted from the AI feedback text, and (iii) how revisions change the pedagogical type of feedback delivered. First, we find that teachers accept AI feedback without modification in about 80% of cases, while edited feedback tends to be significantly longer and subsequently shortened by teachers. Editing behavior varies substantially across teachers: about 50% never edit AI feedback, and only about 10% edit more than two-thirds of feedback instances. Second, machine learning models trained only on the AI feedback text as input features, using sentence embeddings, achieve fair performance in identifying which feedback will be revised (AUC=0.75). Third, qualitative coding shows that when revisions occur, teachers often simplify AI-generated feedback, shifting it away from high-information explanations toward more concise, corrective forms. Together, these findings characterize how teachers engage with AI-generated feedback in practice and highlight opportunities to design feedback systems that better align with teacher priorities while reducing unnecessary editing effort.
Figures & tables
Figure 1: LLM-supported feedback generation. Screenshot of the teacher’s tag, evaluation, and feedback module. The red area shows the feedback generated after clicking the Generate Feedback button. Additionally, the screen displays the student’s response and two tags created by a teacher.
Subset
Similarity
AI Feedback Length
Final Feedback Length
All feedback
0.97 (0.09)
43.24 (19.60)
41.60 (19.81)
Edited feedback only
0.88 (0.17)
48.38 (18.09)
41.00 (19.97)
Table 1: Descriptive statistics for AI feedback and final teacher explanations. Values are reported as mean (SD).
Figure 2: Cumulative distribution of the percentage of messages changed by teachers. The x-axis shows the percentage of teachers (sorted by change rate), and the y-axis shows the percentage of messages modified. Dashed lines indicate teacher percentiles.
Model
ROC AUC
Balanced Accuracy
Accuracy
Cohen’s κ
Logistic Regression
0.698 [0.534, 0.715]
0.672 [0.510, 0.688]
0.733 [0.712, 0.748]
0.322 [0.015, 0.358]
MLP (1 layer)
0.738 [0.518, 0.781]
0.535 [0.515, 0.550]
0.757 [0.740, 0.844]
0.095 [0.036, 0.134]
Gradient-Boosted Trees
0.746 [0.539, 0.768]
0.651 [0.487, 0.668]
0.824 [0.807, 0.864]
0.387 [-0.039, 0.423]
Table 2: Holdout predictive performance for revision decision models. Values are reported as point estimate [95% group-bootstrap CI], computed with 10,000 resamples.
We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in terms of feedback focus and adaptivity. Previous evaluation efforts have examined feedback characteristics, its impact on learning, and its target, yet the focus of feedback and its adaptivity remains largely overlooked. To bridge this gap, we adopt and refine Narciss's taxonomy into seven feedback focus types to annotate teacher and LLM-generated feedback across three university writing courses. We release FeedType, a benchmark containing annotated teacher and LLM feedback from six LLMs under three prompting strategies. We assess the coverage and distribution of feedback focus types, and examine whether LLMs adapt their feedback across draft stages and student performance levels as an expert instructor does. Our findings show that while most LLMs cover most feedback focus types, they fail to reflect teacher feedback distributions and show varying levels of adaptivity, with none matching the teachers' adaptive behavior. We believe FeedType will support future research on pedagogical alignment in LLM feedback generation.
Despite growing interest in using LLMs to generate feedback on students' writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter - a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students' knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.
Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.
Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3
Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University +2