AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

Xinyi Lu; Kexin Phyllis Ju; Mitchell Dudley; Larissa Sano; Xu Wang

Figure 1. Our findings suggest that when making it easy for TAs to review, adopt and edit AI feedback suggestions, AI-mediated feedback authored through FeedbackWriter has higher benefits compared to human-only feedback. Students who received AI-mediated feedback had higher quality revisions (Cohen’s d = 0.50), roughly equivalent to moving a student from the 50th to the 70th percentile. AI-mediated feedback demonstrated more desirable properties of effective feedback, such as actionability and promotion of independent learning.

Abbreviated Version of the Assignment Prompt	Example Rubric	Student Mistake	AI Feedback
Given the rise in preference to oat milk, analyze one of the two policies: either a price floor for almond milk to support the almond milk market, or a tax on agricultural water use to reduce water usage. Explain its expected effects, trade-offs, and impact on society.	Identify that almond milk and oat milk are substitutes in consumption.	Student only analyzed the influence of preference change on the oat milk market; did not discuss the almond milk market.	What is the relationship between almond milk and alternative milk in terms of consumer choice? How might this relationship affect market dynamics?
	Explain that both the consumer and the producer will bear the burden of the tax.	Student only analyzed the influence of the tax on consumers, but did not consider suppliers.	Who is bearing the burden of the tax? Think about how the tax affects both sides of the market.
Identify a recent news article that contains a market failure, explain the market failure within it, and propose a solution.	Propose and expand well on a thoughtful solution to the negative externality in the article.	The student did not expand on the policy requirement, such as the efficient size.	How can the efficient size of a tax be determined, and why is this important for addressing the negative externality?
	Before discussing the article, mention that market failure occurs when the market fails to maximize society’s total surplus.	Student did not differentiate between the private inefficiency and society’s total surplus.	You’ve discussed inefficiencies for consumers, but how does this relate to society’s total surplus?

Table 1. This table shows abbreviated versions of the assignment prompts, a sample of rubrics for each assignment, student mistakes, and AI-generated feedback. The writing assignments are knowledge-intensive essays, in which students are required to demonstrate an accurate understanding and application of economic concepts. The rubrics are specifically designed to capture knowledge deficiencies instead of writing prose.

Figure 2. The design of FeedbackWriter follows TAs’ natural workflow of feedback provision. AI suggestions are provided for each rubric and are surfaced through corresponding UI elements, including 1) identifying relevant sentences (sentence highlighting); 2) judging whether the rubric is adequately and accurately addressed (rubric judgments); and 3) constructing effective feedback (AI feedback and historic feedback). Three-row diagram showing the alignment of TAs’ cognitive process with an LLM pipeline and example UI. The top row shows three steps in "TA’s Cognitive Process" left to right: Identify relevant sentences → Judge adequacy and accuracy → Construct effective feedback. The middle row shows the "LLM Pipeline" mirrors these steps with modules: Sentence highlighting → Rubric judgment → AI Feedback / Historic Feedback. Between the top and middle rows, two-way arrows labeled "LLM suggest" and "TA inspect" indicate that TAs review the system’s suggestions at each stage. The bottom row shows the "UI Elements" examples for each step: highlighted text in a student response, a rubric checklist item (“List all 4 types of market failure”) in red background color with an unchecked checkbox, and side-by-side feedback panels labeled Historic Feedback and GPT Feedback with an “Add to feedback” option.

Figure 3. FeedbackWriter interface contains an essay panel (left) and a feedback panel (right). The essay panel displays the student essay, where the relevant sentences associated with each feedback are highlighted (A). The feedback panel displays AI-generated comments organized by rubrics. Each rubric item has its own feedback box, including the rubric, rubric judgment with color cues (B) and two feedback suggestions (C). Users can also manually add additional in-text comments (D). Annotated screenshot of a web-based interface. The screen is split into two panels. Left panel: the student’s essay text; a sentence is highlighted in orange and boxed. The box is labeled as A. Top-left toolbar: a "Highlight & Comment" button is boxed and labeled D. Right panel: multiple cards each containing a feedback item for a rubric. Each card contains a rubric item on top with a checkbox, a tool box, two feedback suggestions and a textarea with placeholder text "Add some feedback here…". Two rubrics are highlighted in red background color and labeled B. One rubric is highlighted in green with the checkbox checked. Feedback suggestions in one feedback box is labeled C, with side-by-side cards for Historic Feedback and GPT Feedback.

Figure 4. In FeedbackWriter, users can use and adjust all AI suggestions conveniently. Users can use (A) the "reposition" button to modify the highlight, click (B) to flip the judgment, and click (C) the "regenerate" button to get another suggestion. The feedback boxes are empty by default, and users can click (D) to adopt either feedback suggestion. Annotated close-up of a feedback box on the FeedbackWriter interface. At the top, the rubric is highlighted in a red background color, with a checkbox at the left of the rubric is boxed and labeled B. On the right side of the comment row, the tool box contains a rubric-category dropdown, an anchor icon, and a delete icon. The anchor icon is boxed and labeled A. Below, two side-by-side panels compare Historic Feedback (left) and GPT Feedback (right). The Historic Feedback panel says "Don’t forget to discuss the socially’s net gain." and the GPT Feedback panel provides a longer suggestion about connecting inefficiencies to society’s total surplus. Both cards has an "Add to feedback" button boxed on the right-top corner, boxed and labeled D. In the GPT Feedback area, an exchange icon is boxed and labeled C. A text field with placeholder text "Add some feedback here …" appears at the bottom.

Modifications	Original Rubrics	Updated Rubrics	Assignment
Include course and assignment context (e.g., abbreviations and examples discussed in class)	Discuss policy choice sets for externalities as discussed in class	List at least 2 possible solution policies, including legal standards, Pigouvian taxes, tradable permits, standards, and production quotas	Writing Assignment 2
Include economic term explanations	Point out why the public good in the article has free-riders	Point out why the public good in the article has free-riders, i.e., people benefit from the good without paying for it	Writing Assignment 2
Add localization terms to specify where the rubric applies	Identify the decision-makers in the example	When discussing the article, identify the decision-makers in the market who did not produce at the ideal social amount	Writing Assignment 2
Specify the expected depth of the explanation	Perform a thorough welfare analysis	Explain why deadweight loss exists, and mention that it is quite large given that the government purchased the excess	Writing Assignment 1
Include acceptable alternatives	Explain that farmers need water to produce	Demonstrate understanding that farmers demand water, or analyze the influence on farmers as consumers of water	Writing Assignment 1

Table 2. Example changes made to adapt the originally TA-facing rubrics for LLM.

Figure 5. FeedbackWriter further supports the TAs on grading the final drafts in two ways. (A) a collapsible "diff" panel that visualizes students’ additions and deletions between the first and final drafts; (B) feedback from the first draft on the same rubric item is shown in the feedback box. FeedbackWriter further supports the TAs on grading the final drafts in two ways. First, a collapsible "diff" panel is added to the left of the interface, which visualizes students' additions and deletions between the first and final drafts. Second, feedback from the first draft on the same rubric item is shown in each feedback box in the feedback panel.

Figure 6. Timeline for the study. For each assignment in ECON101, students submitted a first draft, received and incorporated TAs’ feedback to form a final draft. The conditions were counter-balanced for the assignments. Diagram showing the study timeline and counterbalanced conditions. Timeline (left to right): submit a first draft, receive feedback and revise, submit a final draft, post-test, receive feedback on final draft. Below, two groups swap feedback conditions across assignments: Student Group 1 has Assignment 1 human-only feedback (with baseline) and Assignment 2 AI-mediated feedback (FeedbackWriter); Student Group 2 has Assignment 1 AI-mediated feedback (FeedbackWriter) and Assignment 2 human-only feedback (with baseline).

Figure 7. The baseline condition is a visually matched, non-AI variant of the FeedbackWriter system. Users can create in-text comments and have easy access to the spreadsheet containing the rubrics and historic feedback, but there are no AI suggestions. Screenshot of the baseline interface. The main view is split into two panels: on the left, the student's essay text is displayed, with one sentence highlighted in orange. On the right, a "Feedback" panel shows only one comment box and a short draft feedback message beginning "Great job on …".

Independent Variable	Estimate	t value	p
(Intercept)	0.37	12.31	$<0.001^{***}$
Condition (AI)	0.05	5.02	$<0.001^{***}$
Assignment (Assignment 2)	4.87	11.07	$<0.001^{***}$
TA Score—Motivational Factor	0.07	1.35	$0.178$
First Draft Quality	0.56	10.99	$<0.001^{***}$

Table 3. Results from the mixed-effects linear regression model on the students’ final draft quality

Figure 8. Draft quality by conditions. Students receiving AI-mediated feedback produced higher quality revisions, with significantly higher improvements, as measured by score difference from first to the final drafts (

p<0.001^{***}

). Bar chart comparing average score percentage for first vs. final drafts in two conditions: FeedbackWriter (left) and Baseline (right). The y-axis is Score Percentage (0 to 1). Each condition has two bars with small error bars; the final-draft bars are hatched. In FeedbackWriter, the mean increases from 0.47 (first draft) to 0.69 (final draft), with an average improvement of 0.23 (SD = 0.16). In Baseline, the mean increases from 0.50 to 0.67, with an average improvement of 0.17 (SD = 0.16). A bracket across the two conditions with "***" indicates a statistically significant difference in improvement between conditions.

Independent Variable	Estimate	t value	p
(Intercept)	0.58	17.62	$<0.001^{***}$
Condition (AI)	-0.007	-1.02	$0.31$
Assignment (Assignment 2)	-0.35	-43.90	$<0.001^{***}$
TA Score—Motivational Factor	0.02	0.61	$0.54$
First Draft Quality	0.05	1.50	$0.13$
Final Draft Quality	0.09	3.60	$<0.001^{***}$

Table 4. Results from the second mixed-effects linear regression model on the students’ post-test performance

Figure 9. Students’ post-test score percentage by condition. There was no significant difference between conditions. Bar chart comparing mean score percentage for two conditions. The x-axis has FeedbackWriter and Baseline; the y-axis ranges from 0.0 to 0.8 (score percentage). FeedbackWriter has a mean score of 0.59, and Baseline has a mean score of 0.56. Both bars include small error bars, and FeedbackWriter is slightly higher than Baseline, but the difference is not significant.

Figure 10. TAs in the baseline had higher inter-rater variance, whereas TAs in FeedbackWriter were more consistent. AI scores had lower variances compared with TAs in the baseline. This is one of the reasons that we did not use the TA scores to indicate essay quality. Two-panel scatter plot with error bars comparing TAs' first-draft writing scores versus the AI-derived scores across 11 fellows for Writing Assignment 1 (left) and Writing Assignment 2 (right). In each panel, the x-axis is Fellow ID (1-11) and the y-axis is Score (ranging 0.3 to 1.0). Each plot include blue circles and orange squares showing the mean TA score per fellow and the corresponding mean AI score on the same submissions, with error bars. Small text labels next to points report per-fellow variance values. Across both assignments and conditions, TA mean scores are consistently higher than AI mean scores for the same fellow. Insets summarize the variance of the mean values: AI = 0.001 in both panels; TA variance is smaller in FeedbackWriter (0.002 in Assignment 1; 0.001 in Assignment 2) than in Baseline (0.009 in Assignment 1; 0.012 in Assignment 2).

Figure 11. AI-mediated feedback from FeedbackWriter has significant higher word counts (left) and coverage of the rubrics (right) on average than the baseline (

p<0.001^{***}

). Two-panel bar chart. The bar chart on the left compares mean feedback word count for two conditions (FeedbackWriter in blue, Baseline in orange). The x-axis lists the two conditions; the y-axis shows Mean Feedback Word Count (ranging 0 to 350). Mean values are labeled above bars: FeedbackWriter = 283.7 words, Baseline = 158.8 words, each with error bars. A bracket with “” above the two bars indicates a statistically significant difference, with FeedbackWriter higher. The bar chart on right compares rubric coverage for two conditions (FeedbackWriter in blue, Baseline in orange). The x-axis lists the two conditions; the y-axis shows Rubric Coverage (ranging 0 to 0.35). Mean values are labeled above bars: FeedbackWriter 0.20, Baseline 0.16, each with error bars. A bracket with “” above the two bars indicates a statistically significant difference, with FeedbackWriter higher.

Figure 12. Category distributions of the feedback by condition. The y-axis is the average number of each type per essay. AI-mediated feedback from FeedbackWriter condition had significantly more Summary and Solutions. Four groups of bar chart showing the mean count (with error bars) of four feedback categories for two conditions (FeedbackWriter in blue, Baseline in orange). The x-axis lists the four categories: Praise, Summary, Problem, and Solution. The y-axis shows the mean count (ranging 0 to 7). Mean values labeled above bars: For praise, FeedbackWriter 0.394, Baseline 0.360; For summary, FeedbackWriter 0.845, Baseline 0.151; For problem, FeedbackWriter 4.070, Baseline 3.610; For solution, FeedbackWriter 5.490, Baseline 4.820. Brackets with “***” indicate statistically significant differences for Summary and Solution, with FeedbackWriter higher in both.

Student mistake	AI-mediated feedback	Human feedback	Themes	Analysis
The student mentioned “substitutes” but not “substitutes in consumption” as required	You’ve identified the role of substitutes, but how might you further explore the relationship between almond milk and its alternatives in consumer choices? What kind of substitutes are they?	As well I would mention the idea of what alternatives are in the market.	Acknowledgement of progress	The AI-mediated feedback acknowledges what the student already did before suggesting the next step. The hint contains questions, which clearly direct how the student could improve, without revealing the answer.
The student didn’t analyze how the solution changes the artificially scarce good to a public good.	What would happen if insulin were priced at its marginal cost? Consider how this might change its classification and the potential issues that could arise.	Consider the solutions discussed in class. What would happen if the government required the good be sold at the efficient price?	More specific, personalized, and localized	The AI-mediated feedback gives a specific and clearer direction for critical thinking and deeper analysis, tailors to the essay topic, and localizes the feedback to where the student discussed the insulin price.
The student mentioned tax, but did not discuss the efficient size	How can the efficient size of a tax be determined, and why is this important for addressing the negative externality?	Remember to expand well on a thoughtful, well-reasoned solution to the negative externality you present in your article.	More tailored to progress	When the student partially meets the rubric item, the AI-mediated feedback is more tailored to the progress, while the human-only feedback remains generic to the rubric item.
Mistake demand vs. quantity demanded	Can you clarify the difference between a change in demand and a change in quantity demanded? Remember that stating a “decrease in demand” implies a demand shift, which does not embody the true economic impact of the tax. How does a price increase from the tax specifically affect quantity demanded? It may help to draw out the tax on the water market on a graph and mention the changes that you see to price and quantity.	Water demand would not decrease as a result of the policy, but there would be a decrease in quantity demanded.	More specific; not revealing the answer	The AI-mediated feedback doesn’t reveal the answer, but explains the mistake and suggests a concrete tool (graph) to approach the problem. The human-only feedback only points out the mistake without supporting the student to reflect, understand, or improve.

Table 5. Example pairs of feedback from FeedbackWriter (AI-mediated) and baseline (human-only) condition with analysis and themes. In the “AI-mediated feedback” column, the bolded texts visualize the original AI-generated suggestions adopted in the final feedback. In the “Human feedback” column, the bolded texts indicate the historic feedback adopted in the final feedback.

Figure 13. Mean scores on desirable properties of effective feedback by condition. The y-axis is the percentage of feedback comments that have this attribute. AI-midated feedback from FeedbackWriter were rated higher on all properties. Four groups of bar chart showing the mean score (with error bars) for four feedback quality characteristics for two conditions (FeedbackWriter in blue, Baseline in orange). The x-axis lists Actionability, Promotion of Independent Learning, Content Accuracy, and Tone and Supportiveness. The y-axis shows the mean score (ranging 0.6 to 1). Mean values labeled above bars: For Actionability, FeedbackWriter 0.89, Baseline 0.76; For Promotion of Independent Learning, FeedbackWriter 0.93, Baseline 0.82; For Content Accuracy, FeedbackWriter 0.98, Baseline 0.96; For Tone and Supportiveness, FeedbackWriter 0.97, Baseline 0.81. Brackets above each pair indicate statistically significant differences: “*” for Actionability, Promotion of Independent Learning, and Tone and Supportiveness, and “” for Content Accuracy, with FeedbackWriter higher than Baseline in all four characteristics.

	Modify judgments	Add historic feedback	Add constructive AI feedback	Add positive AI feedback	Additional feedback	Total feedback
Writing Assignment 1 first draft (n=196)	4.74	2.79	3.49	0.14	1.22	10.53
Writing Assignment 1 final draft (n=190)	4.38	0.07	4.03	0.14	0.55	5.26
Writing Assignment 2 first draft (n=158)	0.67	4.91	4.01	3.68	2.28	15.13
Writing Assignment 2 final draft (n=159)	0.77	0.08	1.99	0.01	1.17	4.21
All assignments	2.83	1.92	3.41	0.91	1.27	8.71

Table 6. In the FeedbackWriter Condition, TAs actively engaged with AI. The table displays the average count of actions for an individual essay.

Independent Variable	Estimate	t value	p
(Intercept)	0.36	10.7	$<0.001^{***}$
Condition (AI)	0.04	2.44	$0.015^{*}$
Assignment (Assignment 2)	0.13	9.53	$<0.001^{***}$
Add Historic Feedback	-0.003	-1.44	0.15
Add AI Positive Feedback	-0.0005	-0.23	0.81
Add AI Constructive Feedback	0.005	2.22	$0.027^{*}$

Table 7. Regression analysis of AI suggestion usage on the students’ final draft quality

Metric	Description	Positive Examples	Negative Examples
Content Accuracy	This metric measures whether the graders show strong subject-matter knowledge. Met: The grader gives feedback that correctly reflects economic principles/concepts based on the rubric. The grader demonstrates strong subject-matter knowledge. Not Met: The grader provides feedback that mostly or completely misrepresents economic principles/concepts based on the rubric. The grader lacks subject-matter knowledge.	“Quantity supplied and quantity demanded in the almond milk market will increase along their respective curves, not supply and demand.” “Remember to identify the consumers and producers of each market. Evaluate who drives the demand in the water market, address both consumers.”	“You say that ‘the demand curve for oat and soy milk will shift right’, but these goods are substitutes, so they should shift together.” (Note: The grader misidentifies the concept of substitutes. The demand for one good depends on the price of the other. For example, if the price of soy milk rises and its quantity demanded falls, the demand for oat milk should increase, shifting its curve to the right.)
Promotion of Independent Learning	This metric measures whether the graders design feedback that elicits a student’s independent thinking. Met: The grader gives a suggestion without revealing answers or concepts through indirect hints, suggestions, or guiding questions. Feedback is designed to help the student recognize and address issues independently. Not Met: The grader directly reveals answers, concepts, or mistakes without indirect suggestions, hints, or guiding questions to encourage reflection.	“Remember to discuss what external marginal cost is and how that relates to social marginal cost.” “Can you explore how the changes in demand for different types of milk might collectively impact water demand before the policy?” “Make sure you understand the difference between demand and quantity demanded when discussing adjustments in the market.”	“There will not be a shortage in the alternatives market, the market will just adjust to equilibrium.” “There are no comparisons to similar failures within the essay.” (Note: The grader only states the mistake without indirect suggestions to promote independent thinking.) “You should explicitly mention that almond milk requires a greater amount of water as an input than the alternatives.”
Actionability	This metric measures whether the feedback is clear and specific. Met: The grader consistently gives feedback that is specific and personalized. The feedback includes concrete questions, examples, explanations, or hints. The language is clear for students to understand next steps. Not Met: The feedback contains vague or unclear examples, explanations, or hints. The feedback is not personalized based on the student’s writing. The language lacks clarity for students to understand next steps.	“Remember to discuss what external marginal cost is and how that relates to social marginal cost.” “Quantity supplied and quantity demanded in the almond milk market will increase along their respective curves, not supply and demand.” “Can you explore how the changes in demand for different types of milk might collectively impact water demand before the policy?”	“You should list the possible solutions.” “How do you calculate the tax revenue the government generates?” (Note: This feedback asks an unclear guiding question, so students may not understand what “calculate the tax revenue” means.) “Be sure to explain the general idea of both positive and negative externalities.”
Tone and Supportiveness	This feedback measures whether the graders’ tone is appropriate. Met: The grader communicates feedback in a suggestive and respectful tone. The feedback has a good balance between acknowledging attempts, summarizing, explaining, and providing suggestions for improvement. Not Met: The feedback doesn’t contain any suggestions, explanations, and acknowledgement, or uses a condescending tone (e.g., too many negative comments). The feedback may discourage student’s motivation and confidence in learning economics.	“Make sure to properly define market failure and include ALL parts of the definition. I recommend going back to lecture slides for this.” “I appreciate your discussion of social marginal cost/benefit, but I encourage you to compare this to the private marginal cost/benefit instead of just mentioning that it ‘is changed’.”	“Common resources lead to overconsumption, not overproduction.” (Note: The grader only restates a definition without giving any suggestions or acknowledgement, which makes their tone cold.) “I noticed there weren’t any in-text citations in your essay, but there was a quote. The quote is not at all relevant though.”

AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

Abstract

Figures & tables

Appendix

Explore similar work

Understanding Teacher Revisions of Large Language Model-Generated Feedback

Making AI-Generated Feedback Matter: A Large-Scale Study of Feedback Workflows and Student Enactment

Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education