cs.CLAug 24, 2026

When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk

Authors: Liliana Santos-Deonizio, James Malamut, Ramón Antonio Martínez, Dorottya Demszky

Organizations: Stanford University, Stanford, CA, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.

Figures & tables

SpeakerLineDialogue
Ximena1“But I subtract th-, that, that to that.”
Destiny2“No, you minus on these sides, right?”
Ximena3“Yeah.”
Destiny4“That’s how we learned yesterday, and then do the same thing on these numbers once you figure out this answer.”
Ximena5“Oh, oh it’s ninety one. That’s why.”
 6“This is ninety one.”
Table 1. Excerpt 1 from Classroom Conversation between Ximena and Destiny
Figure 1. When Youth Enter the Chat: Before (blue) and After (yellow) Involving Youth in the Knowledge Production Process student map
Code TypeCodeDefinitionExample
TopicOff-TaskTalk not pertaining to the assigned task and how to engage with it.“I’m just gonna go back there and play Fortnite. I somehow know how to play Fortnite.”
 UnderstandingDiscussing group or individual’s understanding of a mathematical idea, question, or problem. Disscussing whether or not or to the extent to which the group or individual feels like they understand or do not understand.“Okay, ya lo entiendo.” (Okay, I understand it now)
 RecordingTalk pertaining to the recording equipment or the university researchers using it to observe them.“Estoy diciendo cosas y tengo el micrófono.” (I am saying things and I have the microphone.)
Talk MoveQuestionAsks a question“Why?”
 ClaimA mathematical statement of fact or conjecture that can be proven false or not.“Veinte por cuarenta es ochenta.” (Twenty times forty is eighty.)
 Next-StepSuggests the next step for the group or an individual to take.“Try and see if eight can go into twenty-four on this.”
Table 2. Code Definitions and Examples
FeatureOld F1New F1New PrecisionNew Recall
Off-Task0.7640.8170.7340.921
Understanding0.6390.7000.6050.830
Recording0.6610.6820.5640.863
Question0.7170.9470.9550.938
Claim0.6530.7570.8940.656
Next-Step0.5860.6310.5150.815
Table 3. GPT-5.1 Scores by Feature Before and After Definition Update
Figure 2. Example of Language Mapping Activity student map
Figure 3. Layout of Ms. Richer’s classroom mapping where Luna and Malik sat during a class session. Diagram of a classroom layout with desks arranged in groups. L is labeled for student Luna and M is labeled for student Malik
StudentOff-TaskUnderstandingRecordingQuestionClaimNext-StepDisagree
Diego0/20/6–0/105/120/11/1
Drake1/51/4–0/60/61/10/1
Destiny0/80/51/20/81/41/30/2
Ximena5/80/4–1/43/170/1–
Table 4. Disagreement Rates During Student Member Checks for Analytic Codes
SpeakerLineDialogue
Destiny1“Divided by six.”
 2“Negative one. Right? Am I, am I dumb or am I dumb?”
Ximena3“I don’t know see honey bunch, you’re asking the wrong person over here.”
 4“Let’s be for real. you’re asking the wrong person.”
Table 5. Excerpt 2 from Classroom Conversation between Ximena and Destiny
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

PhaseQuestions / Prompts
Participant Retrospection IntroductionI want to talk about some of the moments when you were talking in your math class and how you felt and what you were thinking during those moments. Let’s read this example together.
Retrospection Questions“Can you tell me more about this and what you were doing here?"
 “Could you tell me how you were feeling during this moment?”
Member Check IntroductionOne of the things we want to be able to learn from looking at how students talk in math is how to help teachers notice the brilliant and interesting conversations you all are having. As part of our research to understand the ways students talk in math class we are using Large Language Models (think of something like ChatGPT) to see how they pick up on the way you all talk to each other. So we can give the model a written version of your conversation and ask it to look for when a student asked a question, added on to what someone else said, or shared an idea about math. Here are some examples of what this model picked up from your conversation. Thinking about what you remember happening in this moment, what do you think about what the model noticed?
Member Check Questions“If you agree, why do you agree?"
 "If you disagree, why do you disagree?”
Appendix
Table 6. Overview of Participant Retrospection and Member Check Protocol
More in this topic

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 265 papers

Explore similar work

CardsList
Jul 13, 2026

The Paternalistic Filter: Epistemic Injustice and Differential Refusal in LLM-Mediated History Education for Marginalized Romanian Students

As Large Language Models (LLMs) are increasingly deployed as conversational tutors, they risk institutionalizing systemic inequalities. This study presents a systematic API audit of four LLMs acting as history tutors, evaluating 1,800 responses regarding the 1989 Romanian Revolution across five student personas varying by ethnicity and socio-economic tier. We uncover four interconnected patterns of epistemic paternalism: (1)Differential Refusal, where safety-aligned models block 76.7% of educational requests from low-tier students; (2)Epistemic Gatekeeping, evidenced by a 3×\times reduction in access to geopolitical complexity (e.g., the contested ``coup theory'') for marginalized learners; (3)Agency Theft, a lexical shift where models like LLaMA produce a 5×\times higher victimization-to-politics vocabulary ratio for Roma students compared to elite peers; and (4)Elite Hermeneutics, where AI tutors disproportionately withhold epistemic confidence and justification scores from low-resource demographic profiles. We argue that current safety alignment acts as a paternalistic filter, transforming conversational AI into agents of narrative segregation -- a manifestation of hermeneutical injustice in Fricker's~\cite{fricker2007} sense that demands urgent pedagogical auditing.

Alexis Popovici, Andrei Ionascu, Adrian-Marius Dumitran

Universitatea din Bucuresti, Bucharest, Romania

Language Model Safety EvaluationAI in EducationDemographic Bias in Language ModelsLanguage Model Auditing
AskPDF
Feb 20, 2026

"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations

Providing scaffolding through educational chatbots built on Large Language Models (LLM) has potential risks and benefits that remain an open area of research. When students navigate impasses, they ask for help by formulating impasse-driven questions. Within interactions with LLM chatbots, such questions shape the user prompts and drive the pedagogical effectiveness of the chatbot's response. This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework. We analysed 6,113 messages from both learning contexts, using 11 different LLMs and three human raters to classify student questions using four existing schemas. On the feasibility of using LLMs as raters, results showed moderate-to-good inter-rater reliability, with higher consistency than human raters. The data showed that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment. These results provide a basis on which to use LLMs for classification of student questions. However, we identify clear limitations in both the ability to classify with schemas and the value of doing so: schemas are limited and thus struggle to accommodate the semantic richness of composite prompts, offering only partial understanding the wider risks and benefits of chatbot integration. In the future, we recommend an analysis approach that captures the nuanced, multi-turn nature of conversation, for example, by applying methods from conversation analysis in discursive psychology.

Alexandra Neagu, Marcus Messer, Peter Johnson +1

Imperial College London, London, United Kingdom

Generative AI in EducationEducational Dialogue SystemsConversational AgentsText ClassificationLLM-as-a-JudgeLLM-Assisted Annotation
AskPDF
Oct 6, 2026

Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue

In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasingly relies on large language models (LLMs) to extract such information from dialogue for a variety of downstream tasks, including knowledge tracing, behavioral modeling, and diagnosis of student reasoning errors. However, the validity of these model-generated interpretations remains insufficiently understood. In this exploratory study, we examine the validity of LLM classifications of five student failure modes in mathematics tutoring dialogue using an operational diagnostic codebook: uncertainty, misattribution, operator selection, conceptual gap, and procedural slip. Across models, human-LLM agreement was moderate (kappa = .524-.597), while cross-model agreement was substantially higher (kappa = .755-.781; alpha = .769). These findings show that cross-model agreement can create a misleading appearance of correctness, challenging the assumption that consensus among LLMs constitutes evidence of valid learner interpretation. For learning analytics, the implication is clear: scalable labeling is useful only if the inferred constructs are valid, and model consensus cannot substitute for independent evidence of that validity.

Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5

College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.

LLM EvaluationLLM-Assisted AnnotationAI in EducationLLM Reliability
AskPDF