cs.CLMar 12, 2026
SaveMarked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback
Organizations: Stanford University, Stanford, California, USA
Abstract
Effective personalized feedback is critical to students' literacy development. Though LLM-powered tools now promise to automate such feedback at scale, LLMs are not language-neutral: they privilege standard academic English and reproduce social stereotypes, raising concerns about how "personalization" shapes the feedback students receive. We examine how four widely used LLMs (GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, Llama-3.1 8B) adapt written feedback in response to student attributes. Using 600 eighth-grade persuasive essays from the PERSUADE dataset, we generated feedback under prompt conditions embedding gender, race/ethnicity, learning needs, achievement, and motivation. We analyze lexical shifts across model outputs by adapting the Marked Words framework. Our results reveal systematic, stereotype-aligned shifts in feedback conditioned on presumed student attributes--even when essay content was identical. Feedback for students marked by race, language, or disability often exhibited positive feedback bias and feedback withholding bias--overuse of praise, less substantive critique, and assumptions of limited ability. Across attributes, models tailored not only what content was emphasized but also how writing was judged and how students were addressed. We term these instructional orientations Marked Pedagogies and highlight the need for transparency and accountability in automated feedback tools.
Figures & tables
| Writing Prompt | N Essays |
| Some of your friends perform community service. For example, some tutor elementary school children and others clean up litter. They think helping the community is very important. But other friends of yours think community service takes too much time away from what they need or want to do. Your principal is deciding whether to require all students to perform community service. Write a letter to your principal in which you take a position on whether students should be required to perform community service. Support your position with examples. | 300 |
| You have read the article ’Unmasking the Face on Mars.’ Imagine you are a scientist at NASA discussing the Face with someone who thinks it was created by aliens. Using information in the article, write an argumentative essay to convince someone that the Face is just a natural landform. Be sure to include: claims to support your argument that the Face is a natural landform; evidence from the article to support your claims; an introduction, a body, and a conclusion to your argumentative essay. | 300 |
Table 1. Selected persuasive writing prompts and essay sample sizes from the PERSUADE dataset.
| Marked Attribute | Marked Prompt | Baseline Prompt | Comparative Prompt |
| Learning Needs-Based | |||
| Low Achievement | The student does not meet academic standards in this class. | You are a middle school English Language Arts teacher. You will be provided a writing prompt and a piece of student writing, denoted by triple quotes. Write inline feedback that will best address the student’s needs to help them revise their work. | The student exceeds academic standards in this class. |
| ELL | The student has an English language learner designation. | — | |
| Learning Disability | The student has a learning disability designation. | — | |
| Identity-Based | |||
| Black | The student is Black / African American. | The student is White. |
Table 2. Baseline, marked, and comparative prompt variants used for feedback generation. All variants are constructed by appending descriptors to the shared baseline prompt. For some attributes (e.g., ELL), marked prompts are compared against the baseline as an unnamed institutional default. For others, marked prompts are compared against an explicitly named dominant category (e.g., White, male) or an opposing evaluative descriptor (e.g., motivated).
| Marked Attribute | Words Students See More ( ) | Words Students See Less |
| Low Achievement | should , try , spelling, explain, provide, avoid , instead , make sure , correct, check , good , error, focus, grammar, unclear , proofread, remember , support, please , vague | consider , enhance, further , potential, strengthen, great , might , nuanced , more, expanding, compelling , exploring, impact, excellent , effectively , depth, emphasize, tone, discuss, demonstrate |
| ELL | formal, language, word, spelling, remember , correct, form, clearer , vocabulary, instead , English, understand, sound, verb, apostrophe, contraction, should , tense, plural, mistake | consider , how, provide , evidence, argument, specific, statement, support, claim, focus, strengthen, benefit, address, point, explain, strong , potential, reasoning, counterargument, detail |
| Learning Disability | try , help , great , let’s , you , we , remember , understand, start, easier , check , break down, reader, follow, clearer , shorter , catch, careful , long , simpler | consider , rephrasing, feel , statement, revising, formal, how, support, reasoning, specific, providing, abrupt , vague , compelling , benefit, analysis, seems , directly , address, argument |
Table 3. Top 20 statistically significant words (with ) identifying each learning-needs attribute condition. The English language learner and learning disability attributes are compared against the condition in which no attribute was specified; the Low Achievement attribute is compared against High Achievement. Words are ordered in descending order by z-score and include only significant words shared across multiple model types. Underlined words indicate evaluative judgments of the writing , italicized words are reflective of the tone used to address the student , and unformatted words signal the content of the feedback comment.
| Marked Attribute | Words Students See More ( ) | Words Students See Less |
| Black | importance, great , our , experience, perspective, excellent , diverse, values, challenge, stereotype, systemic, within, personal , responsibility, American, historical, polished , powerful , social, peer, we | should , try , support, sentence, good , specific, correct, instead , spelling, statement, evidence, claim, word, avoid , phrase, unclear , formal, rephrasing, check , grammar |
| Hispanic | value, formal, writing, cultural, family, experience, remember , English, polished , language, sound, background, responsibility, vocabulary, relatable , perspective, diverse, respect, verb | statement, how, what, evidence, sentence, specific, claim, support, argument, directly , potential, address, focus, error, simply , proofread, avoid , counterargument, reasoning |
| Asian | culture, value, respect, diverse, perspective, structure, formal, writing, background, academic , English, responsibility, mindful , great , clarity, polished , education, familiar, family, tone | sentence, should , what, how, try , directly , claim, specific, proofread, evidence, position, statement, revising, support, address, counterargument, vague , explain, unclear , confusing , explicitly |
| Female | consider , may , respectful , individual, we , love , you , I , instead , responsibility, mindful , empathy, our , importance, wonderful , appreciate , opinion, feel , relatable , tone | try , good , specific , why, explain, proofread, article, providing, example, point, support, interesting, claim, prompt, detail, data, informal , statement, audience, contradict |
Table 4. Top 20 statistically significant words (with ) identifying each identity attribute condition. The Black, Hispanic, and Asian attribute conditions are compared against the White condition; the Female attribute is compared against Male. Words are ordered in descending order by z-score and include only significant words shared across multiple model types. Underlined words indicate evaluative judgments of the writing , italicized words are reflective of the tone used to address the student , and unformatted words signal the content of the feedback.
| Marked Attribute | Words Students See More ( ) | Words Students See Less |
| Unmotivated | let’s , try , you , great , why, start, can , specific, think, how, stronger, experience, we , our , point, interesting , explain, remember , love , convincing | consider , ensure , sentence, enhance, rephrasing, clarity, emphasize, providing, further , strengthen, more, effectively , correct, tone, grammatical, structure, formal, academic, nuanced , demonstrate |
Table 5. Top 20 statistically significant words (with ) identifying each socio-psychological attribute condition. The Unmotivated attribute condition is compared against the Motivated condition. Words are ordered in descending order by z-score and include only significant words shared across multiple model types. Underlined words indicate evaluative judgments of the writing , italicized words are reflective of the tone used to address the student , and unformatted words are content-specific.
| Marked Attribute | Marked Prompt | Comparative Prompt | Baseline Mean |
| Low Achievement | (0.153) | (0.153) | 10.609 |
| ELL | (0.144) | — | 4.911 |
| Learning Disability | (0.083) | — | 3.263 |
| Black | (0.058) | (0.058) | 1.037 |
| Hispanic | (0.073) | (0.073) | 1.705 |
| Asian | (0.076) | (0.076) | 2.449 |
Table 6. Regression estimates predicting percentage-point changes in the concentration of marked words in model-generated feedback as a function of marked and comparative prompt conditions. Each row reports results from a separate regression. All models include controls for true student characteristics and holistic essay score, with writing-assignment and source-LLM fixed effects. Baseline means are computed from only baseline condition feedback (N = 2400). Regression estimates are computed using feedback from all relevant prompt conditions; sample sizes vary by attribute (N = 4800 for ELL and Learning Disability; N = 7200 for all other attributes).
More in this topic
Linguistic Bias in Language Models
Linguistic bias in language models studies how language varieties, identity cues, and cultural context shape model outputs and judgments.
See all 86 papers