LLMs and the Homogenization of Teaching
Organizations: Stanford University Stanford, CA 94305, USA
Abstract
Large language models (LLMs) are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do. We study diversity in two educational generation tasks: math items and essay feedback. We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it. We evaluate direct prompting, two diversity prompting methods, and post-training. Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content--78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity--and represent a narrower range of instructional forms. Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire. Models inherit and amplify the most normative routines of human instruction. Overcoming mode collapse in education will require richer instructional data and investment in teachers' creative capacity and professional judgment.
Figures & tables
| Domain | Original Output | Masked Output |
|---|---|---|
| Math | Jada has a sleeping bag that is rated for 30 degrees F. Write an inequality that represents the outdoor temperature at which Jada will be able to stay warm… | [text_variable] has a sleeping bag that is rated for [whole_number] degrees F. This means that if the temperature outside is at least [whole_number] degrees F, [text_variable] will… |
| Essay | Check spelling: ‘assitance’ should be ‘assistance.’ | Check spelling: [quote] should be [rewrite] |
| Essay | Clarify this idea by saying something like, ‘One good thing about driverless cars is that they could help elderly people.’ | Clarify this idea by saying something like, [insertion] |
| Measure | Category | Example |
| Math Task Type | Compute | Solve for . |
| Construct | Make a dot plot for the given data. | |
| Interpret | If the 5 represents the number of people, what does the 4 represent? | |
| Analyze | Jana says . Explain the mistake. | |
| Connect | Match each equation to the graph that represents it. | |
| Evaluate | Are and equivalent for all values of ? |
| Math Items | Essay Feedback | |||||||
|---|---|---|---|---|---|---|---|---|
| Self-BLEU | Emb. Sim. | Self-BLEU | Emb. Sim. | |||||
| Generator | Unmask | Mask | Unmask | Mask | Unmask | Mask | Unmask | Mask |
| Human | 0.394 | 0.602 | 0.347 | 0.479 | 0.270 | 0.318 | 0.246 | 0.257 |
| GPT-5.2 | 0.852 | 0.879 | 0.651 | 0.693 | 0.460 | 0.527 | 0.369 | 0.409 |
| + Verbalized sampling | 0.516 | 0.716 | 0.391 | 0.528 | 0.358 | 0.433 | 0.316 | 0.351 |
| + Negative sampling | 0.555 | 0.693 | 0.403 | 0.532 | 0.358 | 0.430 | 0.317 | 0.350 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Unit ID | Unit Description |
|---|---|
| HS_2_6 (High School Geometry Unit 6) | This unit brings together students’ experience from previous years with their new understanding from this course for an in-depth study of coordinate geometry. Students encounter a new coordinate transformation notation such as , which connects transformations to functions. Then they use transformations and the Pythagorean Theorem to build equations of circles, parabolas, parallel lines, and perpendicular lines from definitions. Students apply these ideas to proofs, such as classifying quadrilaterals. Finally, students use weighted averages to partition segments, scale figures, and locate the intersection point of the medians of a triangle. |
| MS_1_4 (Grade 6 Unit 4) | In this unit, students examine how the relative sizes of numerator and denominator affect the size of their quotient when numerator or denominator (or both) is a fraction. They acquire the understanding that dividing by has the same outcome as multiplying by , then by . They compute quotients of fractions. They solve problems involving lengths and areas of figures with fractional side lengths and extend the formula for the volume of a right rectangular prism to prisms with fractional edge lengths and use it to solve problems. They use tape diagrams, equations, and expressions to represent situations involving partitive or quotitive interpretations of division with fractions. Given a multiplication or division equation or expression with fractions, they describe a situation that it could represent. They use tape diagrams and equations in reasoning about situations that involve multiplication and division of fractions. |
| MS_2_5 (Grade 7 Unit 5) | In this unit, students interpret signed numbers in contexts (e.g., temperature, elevation, deposit and withdrawal, position, direction, speed and velocity, percent change) together with their sums, differences, products, and quotients. (“Signed numbers” include all rational numbers, written as decimals or in the form .) Students use tables and number line diagrams to represent sums and differences of signed numbers or changes in quantities represented by signed numbers such as temperature or elevation, becoming more fluent in writing different numerical addition and subtraction equations that express the same relationship. They compute sums and differences of signed numbers. They plot points in the plane with signed number coordinates, representing and interpreting sums and differences of coordinates. They view situations in which objects are traveling at constant speed (familiar from previous units) as proportional relationships. For these situations, students use multiplication equations to represent changes in position on number line diagrams or distance traveled, and interpret positive and negative velocities in context. They become more fluent in writing different numerical multiplication and division equations for the same relationship. Students extend their use of the “next to” notation (which they used in expressions such as and in grade 6) to include negative numbers and products of numbers, e.g., writing and rather than and . They extend their use of the fraction bar to include variables as well as numbers, writing as well as . |
| Assignment Prompt | Student Essay |
|---|---|
| Read “Narciso Rodriguez” by Narciso Rodriguez. Describe the mood created by the author in the memoir. Support your answer with relevant and specific information from the memoir. | The mood in the memoir that the author creates is a very warm and relaxed sort of mood. The way the author tells you about his parents; the selflessness and the way they put their kids first before themselves really just makes you think about your own parents and how good they are to you and what they would do for you. The way the author talks about his home makes you think of all the colors in the house. I think that the house would be a golden red color yellow, white, brown and orange on the insides. It reminds me of the way they would have cooked. They would have cooked with colorful peppers, onions, tortillas and oils. It makes you think about what colors you have in your home and what you would cook with. The music sounds wonderful. Even though they don’t demonstrate what it actually sounds like, you can easily imagine it. I imagine it laying in the background while all the adults are cooking and the kids are playing in colorful rooms with the golden sunlight shining in through an open window. I love the mood that this memoir puts me in. |
| Your principal is reconsidering the school’s cell phone policy, choosing between two options. Policy 1: allow phones at school for use during lunch and free times, turned off during class. Policy 2: do not allow phones at school at all. Write a letter to your principal convincing her which policy is better. Support your position with specific reasons. | Dear Principal, I have heard about the new cell phone policies that you have been thinking of. I feel that policy number 2 is the better one of the 2 choices. hear are a couple of the reasons why I personally think this. Students enjoy using their phone every day. I feel that sometimes the can use it a little to much during school. I think that letting students use their phones during lunch and free time is an excellent idea. Also students wouldn’t be interrupting class every day with their cell phones. This would be a lot better than just getting rid of cell phones completely. As principal you might want to consider possibly figuring out a schedule were students have free time during a couple of their classes. Doing this would not only help the new policy, but keep it going throughout the school year. students would appreciate it if during a boring class they got a 5-15 minute break to relax and use their phones. But, there would be some rules behind the free time. You would have to think of them so that students wouldn’t be doing stuff that is against school rules. So principal please consider the second policy for cell phones at our school. students do want to use their cell phones and not keep them at home. I hope you liked the reasons why please se |
| Math Items | Essay Feedback | |||||
|---|---|---|---|---|---|---|
| Generator | Underst. | Solvable | Unit Rel. | Underst. | Accurate | Context Rel. |
| Human | 99.1 | 98.3 | 85.4 | 91.3 | 88.6 | 88.2 |
| GPT-5.2 | 99.9 | 99.9 | 99.9 | 100.0 | 100.0 | 100.0 |
| + Verbalized sampling | 100.0 | 99.9 | 99.6 | 100.0 | 100.0 | 100.0 |
| + Negative sampling | 99.9 | 99.9 | 99.2 | 100.0 | 100.0 | 100.0 |
| Gemini-3.5-Flash | 100.0 | 100.0 | 99.9 | 100.0 | 100.0 | 100.0 |
| Code | Definition | Example |
|---|---|---|
| Recall | Retrieving a fact, definition, formula, or convention. Nothing is produced nor examined. | What is the appropriate unit for volume: cm, cm 2 , or cm 3 ? |
| Interpret | Surfacing or reasoning about meaning: what a mathematical object, symbol, or quantity refers to or implies about a situation; showing conceptual understanding, not carrying out a procedure or producing a representation. | If the 5 represents the number of people, what does the 4 represent? |
| Compute | Finding a numerical, symbolic, or exact answer: calculating, simplifying, solving, or determining a value. | Solve for . A shirt costs $40 and is 25% off. Find the sale price. |
| Construct | Producing a mathematical representation, such as a graph, table, diagram, dot plot, number line, expression, equation, model, or visual display. Lower-level than Create, more routine and less inventive. | Make a dot plot for a given list of data values. Write an expression to represent the perimeter of this figure. |
| Create | Generating something new and open-ended where many results are valid: inventing an example, data set, problem, representation, or situation that meets given conditions. More inventive and less routine than Construct. | Create a data set with a mean of 10 and a median of 8. Create a problem that can be solved with . |
| Classify | Organizing objects by a property: sorting into groups or categories, labeling, or ordering along a dimension. | Group expressions according to whether they are linear, quadratic, or exponential. Order from smallest to largest: number of pennies in a stack 1 ft high, number of books in a stack 1 ft high, … |
| Code | Definition | Examples |
|---|---|---|
| Operate | The feedback asks the student to retrieve, add, correct, or execute something straightforward. The remedy is clear and does not require judgment, reconsideration, or new reasoning. | Fix the spelling of “necessary.” Add a citation for this statistic. Start a new paragraph here. |
| Elaborate | The feedback asks the student to add more content, detail, explanation, context, or development of a kind they already understand and could produce. The student elaborates without needing to reconsider their claim, belief, purpose, or rhetorical plan. | Can you give an example here? Add something to connect these paragraphs. Your introduction needs more context. Expand on this point. |
| Connect | The feedback asks the student to work out relationships between ideas, evidence, claims, structure, counterarguments, or rhetorical choices. The student must construct a connection that is missing, unclear, contradictory, or underdeveloped. | How does this evidence support this claim? You have two contradictory points here. How do they fit together? |
| Interpret | The feedback asks the student to develop an idea by explaining its meaning, significance, implication, or application. The student goes beyond adding more content to show what the idea means or why it matters. | What can we infer about her feelings from this quote? Where can we see this same phenomenon in action today? |
| Evaluate | The feedback asks the student to reconsider validity, credibility, appropriateness, effectiveness, or fit. The student must assess whether a claim, source, choice, stance, tone, word, or rhetorical move works. | What might a critic say in response to this argument? Is this the right tone for this audience? I’m not sure this is true. |
| Code | Definition | Examples |
|---|---|---|
| Correction | The teacher directly fixes something in the text, supplies replacement wording, or gives an explicit edit to the student’s text. | Comma splice. Should be “their” not “there.” Change X to Y. |
| Command | The teacher directs the student to take an action, without supplying the exact edit. This includes imperatives and softened directives like “try,” “consider,” or “you might.” | Explain why… Add a transition here. Consider adding more detail. Try to develop this argument further. |
| Criticism | The teacher evaluates the writing negatively or identifies a problem. The teacher is judging the writing, even if the criticism is hedged. | This is unclear. These arguments are not convincing. I’m not sure this argument works. |
| Praise | The teacher evaluates the writing positively. Includes simple praise and elaborated praise. | Good. Strong argument here. You’ve defined the issues clearly. |
| Interpretation | The teacher neutrally describes, summarizes, or plays back what the student did in the text, without evaluating it or asking for a change. | You’re arguing here that legalization would reduce crime. In your conclusion you return to the opening question. |
| Explanation | The teacher provides information, reasoning, instruction, a lesson, or a general principle. The comment explains why something matters or teaches the student something. | Transitions help readers follow the logic between your ideas. Direct quotes need quotation marks because readers need to know whose words these are. |
| Human | Gemini-3.5-Flash + Verbalized sampling |
| • Explain what the problem is that you’re saying needs to be fixed. Is it kids getting inappropriate materials in books or the books being censored in general? • I appreciate that you are creating a solution: that’s great, but make sure that you are also defining the issue and articulating what you believe about removing offensive materials before introducing your solution. • What are examples of things that might offend people and be in that room? • I appreciate that you’re trying to signpost for your audience here: let’s try to embed transition words and phrases rather than state them in sentences all on their own. • Always reread for grammar and mechanics • Okay, so hear me out: by including this line right after the idea that people would be old enough to handle the content, you’re undercutting your argument because you’re implying that actually age doesn’t matter since what matters is whether someone gets offended easily. • Seems like this is the point in your writing that you’re going to focus on your views on censorship • Rather than telling your audience that your idea is a good idea, you want to show them: what will your ideas enable? • I think your goal of persuasiveness is present throughout the text, but you can add more to your ideas | • The contraction ’Thats’ needs an apostrophe to become ’That’s’. • Keep an eye on spelling! The correct spelling of this word is ’separate’. • Your spelling of ’seperate’ needs a quick fix. Remember it is spelled ’separate’. • Spelling check: make sure to use an ’a’ instead of an ’e’ in ’separate’. • Correct ’Its’ to ’It’s’ and adjust the apostrophe location in ’would’nt’ to ’wouldn’t’. • Please add an apostrophe to ’dont’ to properly spell ’don’t’. Also, consider using a more formal tone by avoiding ’I don’t think’. • Review your spelling of ’Seperate’. The correct spelling is ’Separate’. Also, ’my views’ is a bit vague for a concluding summary. • The correct spelling of ’Seperate’ is ’Separate’. I encourage you to use more persuasive language than saying ’my views are all good ideas’. • Ensure correct word usage. ’Affective’ has to do with emotions, whereas ’effective’ denotes success at achieving a result. You want ’effective’ here. |
| Human | GPT-5.2 + Verbalized sampling |
| • full of them vs. full of “the” - be mindful of proofreading as you look through your essay. Sometimes, spell check will miss errors when words are spelled correctly, even if they are misplaced/misused. • Perhaps you could give a specific example of distractions that impact driving quality - ex. look into research about texting while driving to make a stronger overall claim. • You should lead into the quote so that you integrate it into the language of your own paragraph. This will result in a smoother reading experience for your audience. • Be mindful of citations! Cite using the author’s name, page number, etc. depending on the citation model you are using. • The above quotation does not speak about energy efficiency. • This offers stronger quotation integration in comparison to your first quotation. Good job! • The assignment instructions do not state it is necessary to include a counterpoint to your argument. What do you gain from the inclusion of this line? • Same comment as before about citations. • You have not yet dedicated enough time to the counterpoint you’ve raised! You should think about the ramifications of the question posed by the author of the original paper. • This is not a strong conclusion! Consider expanding the perspective you have on this issue by considering broader implications of this technology. | • Add a specific example of a human driving flaw (from the article if possible) to make this point more concrete. • Add a transition that connects this idea to your previous point about human flaws to improve flow. • Consider using an example from the article (if it mentions texting, fatigue, or other distractions) to support this. • Try to include a statistic or example from the article to make this claim more persuasive. • Consider specifying a timeframe if the article provides one (ex: “within the next decade”). • Provide a reason for this prediction (ex: testing, government regulations, insurance changes) or cite the article. • Clarify what “another thing” refers to-driving tasks, paying attention, reacting quickly-so the final point is specific. |
| Human | Llama-3.1-8B + SFT |
| • I like how you come out/on strong with your opinion, but your syntax creates confusion and detracts from the power of your claim. • Awk wording. Try: In paragraph four of the article, the author states, “….” • Remember to unpack this a little bit with some analysis before you move onto your next point of support. • Never a good idea to end a paragraph on someone else’s words. Remember the TEEAC or TIQA formula we studied in class. • A strong conclusion doesn’t need this phrase (this is more middle school level). | • I would combine these three sentences and rephrase the idea that there is no need for technology to detect students’ emotional expressions. • Great start! † • I love that you are taking a stance on this issue! I love that you are citing textual evidence. I love that you are using transition words. What I don’t love is that your evidence is not properly cited. Make sure you are citing the article correctly. • Good - maybe you can combine these lines to make one solid thesis? • This evidence also supports your argument. Great job! |
| Human | Llama-3.1-8B and Qwen-2.5-7B + SFT |
| • I like that you have a celar position on the topic in the introduction paragraph, and that you have set up three clearly stated reasons. • You have done a good job of referring to a specific detail from the text to show that these cars can keep drivers alert and ready to take the wheel. • This reasons seems a lot like the previous paragraph’s reason. Perhaps you can combine these two paragraphs into one. I like that you are using some of the text from the article as evidence and support. • Glad to see you are using a lot of text evidence. I would like to see you continue to work on integrating direct quotations into your own wrtiting. After a direct quotation, make sure you add your ideas and explanation. • As you come to the conclusion, make sure you restate your claim and summarize your main points. | Llama-3.1-8B • Very nice job here! You are bringing in great points and organizing them in a clear and concise manner. • How does this support your claim that the car is not dangerous? Qwen-2.5-7B • This is a strong sub-claim! • Strong evidence from the text! How does this connect to your claim? • How does this sentence connect to your claim? |
| Human | GPT-5.2 + Verbalized sampling |
| • Is the solution positive or negative? 2* x=6 -2* x=6.1 2.9 * x = -6.04 -2.473* x = -6.859 • For each equation, write two more equations using the same numbers that express the same relationship in a different way. 3 + 2 = 5 7.1 + 3.4 = 10.5 15 - 8 = 7 3/2 + 9/5 = 33/10 • In 2012, James Cameron descended to the bottom of Challenger Deep in the Marianas Trench; the deepest point in the ocean. The vessel he rode in was called DeepSea Challenger. Challenger Deep is 35,814 feet deep at its lowest point DeepSea Challenger’s descent was a change in depth of (-4) … • Write a sentence to answer each question: How much warmer is 82 than 40? How much warmer is 82 than -40? • A submarine is only allowed to change its depth by rising toward the surface in 60-meter stages. It starts off at -340 meters. At what depth is it after: 1 stage 2 stages 4 stages How many stages will it take to return to the surface? • Some boats were traveling up and down a river. A satellite recorded the movements of several boats. A motor boat traveled -3.4 miles per hour for 0.75 hours. How far did it go? A tugboat traveled -1.5 miles in 0.3 hours. What was its velocity? What do you think that … • Clare is cycling at a speed of 12 miles per hour. If she starts at a position chosen as zero, what will her position be after 45 minutes? Han is cycling at a speed of -8 miles per hour; if he starts at the same zero point, what will his … • Find the value of each expression. 16.2 + -8.4 2/5 - 3/5 -9.2 + -7 -4 3/8 - (-1 1/4) | • A scuba diver changes depth at a constant rate of -3 m/min for 7 min. What is the change in depth? • A cyclist’s change in position is -24 km after riding for 3 h at constant velocity. What was the cyclist’s velocity (km/h)? • Compute: (-1.5)(8) • Compute: (-7) + 12 • Compute: (-3/4) + (5/6) • Compute: (-9)(-5) • Compute: (-2/3)(9/4) • Compute: (-7/10) + (-9/5) |
| Human | Gemini-3.5-Flash + Verbalized sampling |
| • Solve each equation. 2(x-3) = 14 -5(x-1) = 40 12(x+10)=24 1/6(x+6)=11 5/7(x-9)=25 • Elena walked 20 minutes more than Lin. Jada walked twice as long as Elena. Jada walked for 90 minutes. The equation 2(x+20)=90 describes this situation. Match each expression with the statement in the story with the expression it represents. A: x B: x+20 C: 2(x+20) D: 90 1: The number … • Solve the inequality that represents each story. Then interpret what the solution means in the story. For every 42 from this month’s earnings. 7(9-x) >= … • Andre wants to buy a backpack. The normal price of the backpack is 2 from Clare, but still has less than 2 and has less than 20. D: … | • Solve the inequality -3y - 4 >= 8 and state whether the endpoint on the number line is open or closed. • A tape diagram is partitioned into 4 equal parts labeled x, with an additional part labeled 12. The entire length of the tape diagram is 40. Write an equation that represents this tape diagram. • A number puzzle says: ’Multiply a number x by 3, add 12, and the result is 27.’ Write an equation to represent this puzzle. • Find the value of m that makes the equation 1/4m - 5 = -2 true. • Write an equation that represents a balanced hanger diagram with 4 identical blocks of weight y and a block of weight 3 on one side, and a block of weight 15 on the other side. • Solve the inequality 2x + 5 < 13 and describe the solution on a number line. • A taxi service charges a flat fee of 2.25 per mile. Write and solve an equation to find the number of miles, m, a passenger traveled if the total fare was $26.00. • Solve the inequality -5(z + 2) < 15 and write the solution. |
| Human | Qwen-2.5-7B + SFT |
| • Sixth-grade students were asked, “What grade are you in?” Explain why this is not a statistical question. • Zookeepers recorded the ages, weights, genders, and heights of the 10 pandas at their zoo. Write two statistical questions that could be answered using these data sets. • Mai and Priya each played 10 games of bowling and recorded the scores. Mai’s median score was 120, and her IQR was 5. Priya’s median score was 118, and her IQR was 15. Whose scores probably had less variability? Explain … • Invent a data set with five numbers that has a mean of 10 and a median of 12. • Adult elephant seals generally weigh about 5,500 pounds. If you weighed 5 elephant seals, would you expect each seal to weigh exactly 5,500 pounds? Explain your reasoning. | • A survey was conducted at your school to find out how many books each student read over the summer. Here are the results: … Create a histogram to display this data. Then, describe the distribution of the number of books … • A local park has been tracking the number of visitors each day for a week. Here are the numbers: … 1. Make a histogram to represent this data. 2. Find the mean, median, and mode of the data. 3. Calculate … • A local park has been recording the number of visitors each day for a week. Here are the numbers: … Calculate the mean, median, and mode of the daily visitor counts. • A local pet store wants to know more about the types of pets their customers own. They decide to survey 50 customers to find out what kind of pets they have. Here are the results: … Create a bar graph … • A survey was conducted at a middle school to find out how many books each student read over the summer. Here are the results: … Create a histogram to represent this data. Then, calculate the mean and median number of … |