Automated feedback on classroom discourse improves instruction in professional and online teaching contexts, and recent work suggests that delivering similar feedback to learners may strengthen these effects by sharing responsibility for participation. Whether this holds in small-group volunteer tutoring has not been tested. We present results from a randomized controlled trial across two four-week SAT mathematics bootcamps on the Schoolhouse.world platform, involving 1,030 volunteer tutors, 7,443 learners and 6,508 recorded sessions. Tutors were randomly assigned to receive no feedback, automated feedback on their own discourse, or that feedback together with participation feedback delivered to their learners under one of two motivational framings. Feedback to tutors increased eliciting of learner ideas, uptake of learner contributions, and learner reasoning by 8-15%, and the pattern replicated across both bootcamps. However, tutor talk time, which averaged 91.5% in the control group, did not change in any condition, and learner-facing feedback did not further change discourse. Interviews with 17 tutors indicate that they adopted the moves the feedback recommended but were limited by learners who did not respond. Interviews with 13 learners suggest why the learner-facing feedback had little effect: it reported a group-level rather than a learner-level statistic, some learners preferred to listen, and learner-to-learner interaction was rare in these sessions. Automated feedback can change how tutors invite learner thinking, but shifting how much learners talk likely requires changes beyond feedback.
Figures & tables
Figure 1. The feedback shown to tutors and learners, from the platform’s design specification. (a) The primary tutor modal: summary statistics (1), highlighted moments from the tutor’s own transcript followed by a fixed example from another tutor (2), and suggestions generated by GPT-4 Turbo (3). The reflection prompt (4) is omitted here for space. (b) The learner modal, shown in place of the usual join-session prompt. Figures are mockups with placeholder text and values; the two condition messages are given verbatim in Section 4.3 .
Bootcamp 1 (Apr–May)
Bootcamp 2 (May)
Condition
Tutors
Sess.
Elicit
Uptake
Reason
Engaged
Talk
Tutors
Sess.
Elicit
Uptake
Reason
Engaged
Talk
Control
225
1190
17.7
6.1
1.63
0.81
0.917
174
948
16.1
5.5
1.58
0.79
0.914
TutorFeedback
227
1225
19.9
6.7
1.86
0.83
0.912
174
954
18.7
6.3
1.84
0.80
0.918
Tutor+GoalLearner
118
639
18.5
6.6
2.00
0.83
0.913
81
448
17.0
6.1
1.68
0.81
0.922
Tutor+SocialLearner
113
616
19.2
7.0
2.04
0.85
0.910
87
488
17.6
6.5
1.86
0.83
0.907
Table 1. Tutors, sessions and unadjusted means by condition, before listwise deletion on model covariates. Talk-move measures are counts per hour; Engaged is the proportion of attending learners who spoke at least once; Talk is the tutor’s share of speaking time, excluding the Zoom chat.
Pooled
By bootcamp
Outcome
Coef.
(SE)
p
BH p
Ctrl. mean
Effect ( σ )
B1
B2
Eliciting / hr
1.408 ∗∗
(0.500)
0.005
0.010
17.161
0.116
1.439 ∗
1.087
Uptake / hr
0.519 ∗∗
(0.181)
0.004
0.010
5.914
0.117
0.437 +
0.566 ∗
Learner reasoning / hr
0.238 ∗∗
(0.074)
0.001
0.008
1.637
0.123
0.255 ∗
0.196 +
Learner questions / hr
0.429 +
(0.252)
0.089
0.107
6.577
0.070
0.495
0.342
Tutor talk time (speaking time)
− 0.001
(0.004)
0.803
0.803
0.915
− 0.009
− 0.005
0.003
Table 2. Effect of automated feedback to tutors on instructional practice. Pooled model with bootcamp fixed effects ( n=5,653 sessions, 899 tutors); B1 and B2 are the same specification fit to each bootcamp separately ( n=3,170 / 597 and 2,483 / 455). Standard errors clustered on the tutor. Effect ( σ ) is the coefficient in control standard deviations. BH: Benjamini–Hochberg corrected within the six-outcome family; the two all-channel measures are alternative operationalizations of talk time and participation and are outside that family. +p<.10 , ∗p<.05 , ∗∗p<.01 .
Figure 2. Effects on instructional practice in standard deviation units, with 95% confidence intervals, for each bootcamp and pooled. The three targeted discourse measures increase in both bootcamps, while the balance of talk remains unchanged. Forest plot of standardized effects with 95 percent confidence intervals for each outcome, by bootcamp and pooled.
Outcome
Coef.
(SE)
p
BH p
Ctrl. mean
Effect ( σ )
B1
B2
Panel A. Cohort outcomes
Attendance (learners)
0.026
(0.071)
0.714
0.907
4.292
0.013
0.058
− 0.021
Cohort practice test
− 0.005
(0.005)
0.296
0.907
0.656
− 0.024
0.001
− 0.013 +
Cohort NPS
0.008
(0.065)
0.907
0.907
8.235
0.005
0.048
− 0.042
Cohort rating
0.007
(0.015)
0.655
0.907
2.383
0.020
0.023
− 0.011
Panel B. Learner outcomes
Table 3. Effect of automated feedback to tutors on cohort and learner outcomes. Panel A is at the session level (3,858–5,653 sessions), Panel B at the learner-session level (5,840–14,839); both cluster on the tutor. Effect ( σ ) is the coefficient in control standard deviations. BH: Benjamini–Hochberg corrected within panel. +p<.10 , ∗p<.05 .
Outcome
Coef.
(SE)
p
Ctrl. mean
Effect ( σ )
B1
B2
Panel A. Instructional practice
Eliciting / hr
− 0.731
(0.607)
0.228
17.161
− 0.060
− 1.009
− 0.069
Uptake / hr
0.232
(0.211)
0.272
5.914
0.052
0.199
0.320
Learner reasoning / hr
0.067
(0.091)
0.466
1.637
0.034
0.210 +
− 0.115
Learner questions / hr
− 0.050
(0.314)
0.874
6.577
− 0.008
− 0.140
0.123
Tutor talk time (speaking time)
− 0.000
(0.004)
0.952
0.915
− 0.002
− 0.000
− 0.000
Table 4. Learner feedback above tutor feedback alone: the contrast between the two joint conditions and TutorFeedback . Effect ( σ ) is the coefficient in control standard deviations. Sample sizes as in Tables 2 and 3 . p -values are uncorrected ; correcting across the four satisfaction measures would leave none below .05. +p<.10 , ∗p<.05 .