LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.
Figures & tables
Speaker
Line
Dialogue
Ximena
1
“But I subtract th-, that, that to that.”
Destiny
2
“No, you minus on these sides, right?”
Ximena
3
“Yeah.”
Destiny
4
“That’s how we learned yesterday, and then do the same thing on these numbers once you figure out this answer.”
Ximena
5
“Oh, oh it’s ninety one. That’s why.”
6
“This is ninety one.”
Table 1. Excerpt 1 from Classroom Conversation between Ximena and Destiny
Figure 1. When Youth Enter the Chat: Before (blue) and After (yellow) Involving Youth in the Knowledge Production Process student map
Code Type
Code
Definition
Example
Topic
Off-Task
Talk not pertaining to the assigned task and how to engage with it.
“I’m just gonna go back there and play Fortnite. I somehow know how to play Fortnite.”
Understanding
Discussing group or individual’s understanding of a mathematical idea, question, or problem. Disscussing whether or not or to the extent to which the group or individual feels like they understand or do not understand.
“Okay, ya lo entiendo.” (Okay, I understand it now)
Recording
Talk pertaining to the recording equipment or the university researchers using it to observe them.
“Estoy diciendo cosas y tengo el micrófono.” (I am saying things and I have the microphone.)
Talk Move
Question
Asks a question
“Why?”
Claim
A mathematical statement of fact or conjecture that can be proven false or not.
“Veinte por cuarenta es ochenta.” (Twenty times forty is eighty.)
Next-Step
Suggests the next step for the group or an individual to take.
“Try and see if eight can go into twenty-four on this.”
Table 2. Code Definitions and Examples
Feature
Old F1
New F1
New Precision
New Recall
Off-Task
0.764
0.817
0.734
0.921
Understanding
0.639
0.700
0.605
0.830
Recording
0.661
0.682
0.564
0.863
Question
0.717
0.947
0.955
0.938
Claim
0.653
0.757
0.894
0.656
Next-Step
0.586
0.631
0.515
0.815
Table 3. GPT-5.1 Scores by Feature Before and After Definition Update
Figure 2. Example of Language Mapping Activity student map
Figure 3. Layout of Ms. Richer’s classroom mapping where Luna and Malik sat during a class session. Diagram of a classroom layout with desks arranged in groups. L is labeled for student Luna and M is labeled for student Malik
Student
Off-Task
Understanding
Recording
Question
Claim
Next-Step
Disagree
Diego
0/2
0/6
–
0/10
5/12
0/1
1/1
Drake
1/5
1/4
–
0/6
0/6
1/1
0/1
Destiny
0/8
0/5
1/2
0/8
1/4
1/3
0/2
Ximena
5/8
0/4
–
1/4
3/17
0/1
–
Table 4. Disagreement Rates During Student Member Checks for Analytic Codes
Speaker
Line
Dialogue
Destiny
1
“Divided by six.”
2
“Negative one. Right? Am I, am I dumb or am I dumb?”
Ximena
3
“I don’t know see honey bunch, you’re asking the wrong person over here.”
4
“Let’s be for real. you’re asking the wrong person.”
Table 5. Excerpt 2 from Classroom Conversation between Ximena and Destiny
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Phase
Questions / Prompts
Participant Retrospection Introduction
I want to talk about some of the moments when you were talking in your math class and how you felt and what you were thinking during those moments. Let’s read this example together.
Retrospection Questions
“Can you tell me more about this and what you were doing here?"
“Could you tell me how you were feeling during this moment?”
Member Check Introduction
One of the things we want to be able to learn from looking at how students talk in math is how to help teachers notice the brilliant and interesting conversations you all are having. As part of our research to understand the ways students talk in math class we are using Large Language Models (think of something like ChatGPT) to see how they pick up on the way you all talk to each other. So we can give the model a written version of your conversation and ask it to look for when a student asked a question, added on to what someone else said, or shared an idea about math. Here are some examples of what this model picked up from your conversation. Thinking about what you remember happening in this moment, what do you think about what the model noticed?
Member Check Questions
“If you agree, why do you agree?"
"If you disagree, why do you disagree?”
Appendix
Table 6. Overview of Participant Retrospection and Member Check Protocol
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.