Automated feedback on classroom discourse improves instruction in professional and online teaching contexts, and recent work suggests that delivering similar feedback to learners may strengthen these effects by sharing responsibility for participation. Whether this holds in small-group volunteer tutoring has not been tested. We present results from a randomized controlled trial across two four-week SAT mathematics bootcamps on the Schoolhouse.world platform, involving 1,030 volunteer tutors, 7,443 learners and 6,508 recorded sessions. Tutors were randomly assigned to receive no feedback, automated feedback on their own discourse, or that feedback together with participation feedback delivered to their learners under one of two motivational framings. Feedback to tutors increased eliciting of learner ideas, uptake of learner contributions, and learner reasoning by 8-15%, and the pattern replicated across both bootcamps. However, tutor talk time, which averaged 91.5% in the control group, did not change in any condition, and learner-facing feedback did not further change discourse. Interviews with 17 tutors indicate that they adopted the moves the feedback recommended but were limited by learners who did not respond. Interviews with 13 learners suggest why the learner-facing feedback had little effect: it reported a group-level rather than a learner-level statistic, some learners preferred to listen, and learner-to-learner interaction was rare in these sessions. Automated feedback can change how tutors invite learner thinking, but shifting how much learners talk likely requires changes beyond feedback.
Figures & tables
Figure 1. The feedback shown to tutors and learners, from the platform’s design specification. (a) The primary tutor modal: summary statistics (1), highlighted moments from the tutor’s own transcript followed by a fixed example from another tutor (2), and suggestions generated by GPT-4 Turbo (3). The reflection prompt (4) is omitted here for space. (b) The learner modal, shown in place of the usual join-session prompt. Figures are mockups with placeholder text and values; the two condition messages are given verbatim in Section 4.3 .
Bootcamp 1 (Apr–May)
Bootcamp 2 (May)
Condition
Tutors
Sess.
Elicit
Uptake
Reason
Engaged
Talk
Tutors
Sess.
Elicit
Uptake
Reason
Engaged
Talk
Control
225
1190
17.7
6.1
1.63
0.81
0.917
174
948
16.1
5.5
1.58
0.79
0.914
TutorFeedback
227
1225
19.9
6.7
1.86
0.83
0.912
174
954
18.7
6.3
1.84
0.80
0.918
Tutor+GoalLearner
118
639
18.5
6.6
2.00
0.83
0.913
81
448
17.0
6.1
1.68
0.81
0.922
Tutor+SocialLearner
113
616
19.2
7.0
2.04
0.85
0.910
87
488
17.6
6.5
1.86
0.83
0.907
Table 1. Tutors, sessions and unadjusted means by condition, before listwise deletion on model covariates. Talk-move measures are counts per hour; Engaged is the proportion of attending learners who spoke at least once; Talk is the tutor’s share of speaking time, excluding the Zoom chat.
Pooled
By bootcamp
Outcome
Coef.
(SE)
p
BH p
Ctrl. mean
Effect ( σ )
B1
B2
Eliciting / hr
1.408 ∗∗
(0.500)
0.005
0.010
17.161
0.116
1.439 ∗
1.087
Uptake / hr
0.519 ∗∗
(0.181)
0.004
0.010
5.914
0.117
0.437 +
0.566 ∗
Learner reasoning / hr
0.238 ∗∗
(0.074)
0.001
0.008
1.637
0.123
0.255 ∗
0.196 +
Learner questions / hr
0.429 +
(0.252)
0.089
0.107
6.577
0.070
0.495
0.342
Tutor talk time (speaking time)
− 0.001
(0.004)
0.803
0.803
0.915
− 0.009
− 0.005
0.003
Table 2. Effect of automated feedback to tutors on instructional practice. Pooled model with bootcamp fixed effects ( n=5,653 sessions, 899 tutors); B1 and B2 are the same specification fit to each bootcamp separately ( n=3,170 / 597 and 2,483 / 455). Standard errors clustered on the tutor. Effect ( σ ) is the coefficient in control standard deviations. BH: Benjamini–Hochberg corrected within the six-outcome family; the two all-channel measures are alternative operationalizations of talk time and participation and are outside that family. +p<.10 , ∗p<.05 , ∗∗p<.01 .
Figure 2. Effects on instructional practice in standard deviation units, with 95% confidence intervals, for each bootcamp and pooled. The three targeted discourse measures increase in both bootcamps, while the balance of talk remains unchanged. Forest plot of standardized effects with 95 percent confidence intervals for each outcome, by bootcamp and pooled.
Outcome
Coef.
(SE)
p
BH p
Ctrl. mean
Effect ( σ )
B1
B2
Panel A. Cohort outcomes
Attendance (learners)
0.026
(0.071)
0.714
0.907
4.292
0.013
0.058
− 0.021
Cohort practice test
− 0.005
(0.005)
0.296
0.907
0.656
− 0.024
0.001
− 0.013 +
Cohort NPS
0.008
(0.065)
0.907
0.907
8.235
0.005
0.048
− 0.042
Cohort rating
0.007
(0.015)
0.655
0.907
2.383
0.020
0.023
− 0.011
Panel B. Learner outcomes
Table 3. Effect of automated feedback to tutors on cohort and learner outcomes. Panel A is at the session level (3,858–5,653 sessions), Panel B at the learner-session level (5,840–14,839); both cluster on the tutor. Effect ( σ ) is the coefficient in control standard deviations. BH: Benjamini–Hochberg corrected within panel. +p<.10 , ∗p<.05 .
Outcome
Coef.
(SE)
p
Ctrl. mean
Effect ( σ )
B1
B2
Panel A. Instructional practice
Eliciting / hr
− 0.731
(0.607)
0.228
17.161
− 0.060
− 1.009
− 0.069
Uptake / hr
0.232
(0.211)
0.272
5.914
0.052
0.199
0.320
Learner reasoning / hr
0.067
(0.091)
0.466
1.637
0.034
0.210 +
− 0.115
Learner questions / hr
− 0.050
(0.314)
0.874
6.577
− 0.008
− 0.140
0.123
Tutor talk time (speaking time)
− 0.000
(0.004)
0.952
0.915
− 0.002
− 0.000
− 0.000
Table 4. Learner feedback above tutor feedback alone: the contrast between the two joint conditions and TutorFeedback . Effect ( σ ) is the coefficient in control standard deviations. Sample sizes as in Tables 2 and 3 . p -values are uncorrected ; correcting across the four satisfaction measures would leave none below .05. +p<.10 , ∗p<.05 .
We evaluate GPTutor, an LLM-powered tutoring system for an undergraduate discrete mathematics course. It integrates two LLM-supported tools: a structured proof-review tool that provides embedded feedback on students' written proof attempts, and a chatbot for math questions. In a staggered-access study with 148 students, earlier access was associated with higher homework performance during the interval when only the experimental group could use the system, while we did not observe this performance increase transfer to exam scores. Usage logs show that students with lower self-efficacy and prior exam performance used both components more frequently. Session-level behavioral labels, produced by human coding and scaled using an automated classifier, characterize how students engaged with the chatbot (e.g., answer-seeking or help-seeking). In models controlling for prior performance and self-efficacy, higher chatbot usage and answer-seeking behavior were negatively associated with subsequent midterm performance, whereas proof-review usage showed no detectable independent association. Together, the findings suggest that chatbot-based support alone may not reliably support transfer to independent assessment of math proof-learning outcomes, whereas work-anchored, structured feedback appears less associated with reduced learning.
Eason Chen, Sophia Judicke, Kayla Beigh +18
Carnegie Mellon University, Pittsburgh, PA, USA · The University of Hong Kong, Hong Kong, China
We study when natural-language feedback produces improvement beyond the gains obtainable from repeated attempts alone. In multi-turn language agent setting, higher final accuracy can reflect useful feedback, but it can also arise from resampling, format correction, or additional test-time computation. To separate these effects, we introduce a controlled student-teacher protocol across Omni-MATH, Codeforces, BBEH Linguini, and ARC-AGI1, evaluating thirteen open-weight models in both student and teacher roles. We compare external feedback, self-feedback, and unguided self-refinement, while varying interaction history, task difficulty, and teacher access to privileged task information. Across settings, we find that multi-turn improvement is often not evidence of feedback use: self-generated feedback adds little beyond unguided self-refinement, whereas the strongest external teachers produce substantially larger feedback-specific gains, suggesting that useful feedback must provide guidance beyond generic retry. Dense student-teacher interaction matrices further show that interactive gains are driven more by the student's ability to use feedback than by the teacher's identity, although teacher choice remains important for a fixed student. These results suggest that feedback-based agents should be evaluated against repeated-attempt baselines, and that ability to act on feedback, not merely feedback availability, is a central bottleneck for interactive improvement. We release our controlled student-teacher evaluation framework at https://j-lojek.github.io/feedback-generation-is-a-bottleneck/.
Bartłomiej Cupiał, Jan Łojek, Mikołaj Garstecki +3
University of Warsaw · AKCES NCBR · Princeton University +2
Current Artificial Intelligence (AI)-based tutoring systems (AI tutors) are primarily evaluated based on the pedagogical quality of their feedback messages. While important, pedagogy alone is insufficient because it ignores a critical question: what do students actually do with the feedback they receive? We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data, which complements pedagogical assessment. We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback from an introductory undergraduate programming course to measure whether students act on tutor feedback and whether those actions are applied correctly. Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns that are not captured by pedagogy-only evaluation. Moreover, these engagement-based behavioral signals are more strongly associated with student perception of helpful feedback than pedagogical quality alone, providing a more complete and actionable picture of AI tutor performance.
Rose Niousha, Samantha Boatright Smith, Bita Akram +5
University of California, Berkeley, USA · North Carolina State University, USA · University of Pittsburgh, USA +1