SLA-LENS: From Classroom Transcripts to CSLA-Informed and Evidence-Linked L2 Interaction Analytics

Nemika Tyagi; Olga Kellert; Phanos Kareen; Nelvin Licona-Guevara

cs.CLOct 16, 2026

SLA-LENS: From Classroom Transcripts to CSLA-Informed and Evidence-Linked L2 Interaction Analytics

Authors: Nemika Tyagi, Olga Kellert, Phanos Kareen, Nelvin Licona-Guevara

Organizations: Arizona State University

Abstract

What happens after a learner makes an error, and how does the instructor's response shape the next turn? Investigating these exchanges across classrooms requires context-sensitive annotation that is difficult to scale manually. We introduce SLA-LENS, an end-to-end LLM-assisted framework that connects classroom transcripts to structured pedagogical annotations and evidence-linked descriptive analytics. Grounded in interactionist Second Language Acquisition (SLA) theory, its annotation schema captures second language learner errors, pedagogical functions, and uptake or repair while accounting for speaker roles, conversational context, and hierarchical labels. We apply the framework to 40 authentic online L2 Spanish classroom transcripts comprising over 21,000 sentence-level instances. We evaluate proprietary and open-weight LLMs, including task-specific QLoRA adaptation, using expert-reviewed gold annotations, held-out silver data, and metrics sensitive to different types of annotation error. A deterministic reporting layer generates reproducible visualizations of participation, feedback, learner errors, uptake, and temporal interaction patterns, linked to supporting transcript evidence. By making these patterns accessible and inspectable, SLA-LENS supports systematic classroom discourse research, pedagogical reflection, and curriculum design.

Figures & tables

Figure 1: Overview of the SLA-LENS end-to-end framework. Classroom transcripts are structured and annotated using contextual LLMs guided by a theory-informed CSLA schema. Annotation quality is assessed through diagnostic-sensitive evaluation, while a deterministic analytics layer computes descriptive interactional measures and candidate event associations. SLA-LENS produces an annotated transcript and an evidence-linked analytical report with participant-level statistics, temporal visualizations, and supporting transcript examples.

Corpus statistic	Count
Transcripts	40
Sentence rows	21,835
Approximate words	106,336
Instructor rows	14,648
Learner rows	7,187
Single-label rows	21,106

Table 1: Overview of the SLA-LENS annotated corpus. Counts are derived from GPT-generated annotations rather than independently validated gold labels.

Family	Labels	Role	Function and example
Prompt	P:I, P:IQ, P:CQ	Instructor	Elicits learner production through instructions, information-seeking questions, or clarification questions. Example: “So, the first question for today is: ¿cuál es el tema de la Unidad 7?” [Eng: “What is the topic of Unit 7?”]
Prompt response	PR	Learner	Responds to the preceding prompt. Example: “El tema de la Unidad 7 es la justicia.” [Eng:“The topic of Unit 7 is justice.”]
Feedback	F:EC, F:R, F:CR, F:EL, F:MF, F:CC, F:Pr, F:Ack, F:NE	Instructor	Responds to learner production through correction, reformulation, elicitation, metalinguistic info, evaluation, or acknowledgment. Example following a lexical error: “Justicia means justice.”
Other instructor functions	A, RN, CM, CoM	Instructor	Answers learner questions, reacts to content, manages classroom logistics, or regulates conversational flow. Example after a learner describes a large family: “That is a big family!”
Uptake / repair outcome	R:S, R:IR, R:Ack, R:NR	Learner	Characterizes a learner response following feedback as successful repair, incomplete repair, acknowledgment, or no uptake. Example following correction: “Mi hermana tiene trece años.”[Eng: “My sister is thirteen years old.”]
Student prompt	SP, SP:I, SP:IQ, SP:CQ	Learner	Initiates or extends interaction through questions, instructions, or requests for clarification. Example: “¿Cómo se dice kite en español?”[Eng:“How do you say kite in Spanish?”]

Table 2: Condensed CSLA annotation schema used by SLA-LENS. Classroom-event and learner-error annotations are stored independently. Full definitions, decision rules, and examples are provided in Appendix A .

Analysis dimension	Descriptive measures
Participation	Instructor and learner contributions; individual learner production.
Learner errors	Frequency, type distribution, and learner-normalized error incidence.
Pedagogical events	Prompt and feedback subtype distributions; uptake/repair outcomes.
Temporal patterns	Distribution of annotated events across lesson intervals.
Error $\rightarrow$ Feedback	Candidate associations between error-marked learner production and subsequent feedback.
Feedback $\rightarrow$ Uptake	Candidate associations between instructor feedback and subsequent learner uptake/repair.

Table 3: Descriptive interactional measures supported by SLA-LENS. Sequence-based measures represent proximity-derived candidate associations rather than verified pedagogical episodes.

Prediction relation	Credit
Exact label	1.00
Correct family, wrong subtype	0.70
Correct role, wrong family	0.25
Role-incompatible prediction	0.00
Correct labels, incorrect order	$\times 0.85$

Table 4: Base credits for hierarchical annotation similarity. Multi-label scoring additionally accounts for partial matches and unsupported labels.

Model	Hard Acc.	Soft Score	Feedback F1	Uptake F1	Error F1	Error Hard Acc.
GPT-4.1	76.72	84.12	87.57	76.64	81.40	96.63
Gemma 4 12B	54.30	67.20	65.10	45.80	61.70	90.80
Llama-3.1-8B	58.70	70.90	69.80	50.60	65.40	91.90
Qwen3-8B	62.40	73.80	73.50	55.20	69.10	92.80
Qwen3-8B (QLoRA)	69.80	79.60	81.20	65.40	75.80	94.90

Table 5: Annotation performance against the expert-reviewed gold transcript. All values are percentages except Soft Score, which is on a 0-100 scale.

Model	Hard Acc.	Soft Score	Feedback F1	Uptake F1	Error F1	Error Hard Acc.
Gemma 4 12B	66.80	76.40	76.20	58.70	70.30	94.20
Llama-3.1-8B	70.20	79.10	79.80	63.40	74.10	95.30
Qwen3-8B	74.10	82.20	83.60	67.90	77.80	96.10
Qwen3-8B (QLoRA)	81.30	87.10	89.20	76.50	84.10	97.20

Table 6: Agreement with GPT-generated annotations on five held-out silver transcripts. Silver agreement measures reproduction of GPT annotations rather than agreement with expert judgments.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Definition
P:I	Instruction or task intended to elicit learner production.
P:IQ	Information-seeking question requesting new information.
P:CQ	Clarification question outside a corrective-feedback sequence.

Appendix

Table 7: Instructor prompt subtypes.

Label	Definition
F:EC	Explicit correction that overtly provides the target form.
F:R	Recast that reformulates learner production without explicitly identifying the error.
F:CR	Clarification request following problematic learner production.
F:EL	Elicitation prompting the learner to supply or complete the corrected form.
F:MF	Metalinguistic information about the linguistic problem without directly supplying the correction.
F:CC	Confirmation check of a proposed interpretation or corrected form.

Appendix

Table 8: Instructor feedback subtypes.

Label	Definition
A	Answers a learner question without functioning as corrective feedback.
RN	Reacts to the meaning, content, or emotional aspect of learner production without evaluating linguistic accuracy.
CM	Manages lesson organization, technology, participation, or logistics.
CoM	Regulates conversational flow through minimal expressions such as okay , mhm , or vale .

Appendix

Table 9: Other instructor functions.

Label	Definition
SP	General learner-initiated prompt when a specific subtype cannot be determined reliably.
SP:I	Directs another participant to perform an action.
SP:IQ	Requests new information.
SP:CQ	Requests clarification, repetition, explanation, or lexical assistance.

Appendix

Table 10: Learner-initiated prompt categories.

Label	Definition
R:S	Successful repair that resolves the problem targeted by feedback.
R:IR	Attempted repair that does not fully resolve the targeted problem.
R:Ack	Acknowledges feedback without producing the corrected form.
R:NR	No learner uptake before the interaction moves on.

Appendix

Table 11: Learner uptake and repair outcomes.

Label	Definition
Er:G	One grammatical error involving morphology, syntax, agreement, word order, or another grammatical feature.
Er:L	One incorrect, missing, malformed, or non-Spanish lexical item in otherwise Spanish production.
Er:M	More than one observable grammatical or lexical error.
Er:EU	Substantially English production when Spanish is contextually expected.

Appendix

Table 12: Learner-language error categories.

Distinction	Decision rule	Example
Prompt vs. Feedback	Use P when the instructor initiates new learner production. Use F when responding to, evaluating, or facilitating correction of learner production.	Di una oración. $\rightarrow$ P:I ; after an error, Di “tiene”. $\rightarrow$ F:EC .
PR vs. R	Use PR after an instructional prompt and an R subtype within a feedback sequence.	T: ¿Cómo estás? S: Bien. $\rightarrow$ PR ; T: Di “tiene”. S: Tiene. $\rightarrow$ R:S .
F:Pr vs. RN	Use F:Pr for positive evaluation of learner performance and RN for reactions to the content or situation described.	S: Estoy cansado. T: Lo siento. $\rightarrow$ RN .
F:Ack vs. CoM	Use F:Ack when acknowledging learner production as a pedagogical response. Use CoM when the expression primarily regulates conversational flow.	T: Muy bien. $\rightarrow$ F:Ack or CoM , depending on preceding learner production and context.
CM vs. CoM	Use CM for lesson logistics, technology, participation, or classroom organization. Use CoM for minimal discourse-management expressions.	Voy a compartir mi pantalla. $\rightarrow$ CM ; Okay. $\rightarrow$ CoM .
Er:L vs. Er:EU	Use Er:L for an isolated non-Spanish lexical item and Er:EU for substantially English production when Spanish is expected.	Mi mamá lives en California. $\rightarrow$ Er:L ; My mother lives in California. $\rightarrow$ Er:EU .

Appendix

Table 13: Key contextual distinctions in the CSLA annotation guidelines.

Gold	Prediction	Score
F:EC	F:EC	100.0
F:EC	F:R	70.0
F:EC	P:I	25.0
F:EC	PR	0.0
F:Pr, P:IQ	F:Pr	50.0
F:Pr, P:IQ	P:IQ, F:Pr	85.0

Appendix

Table 14: Illustrative hierarchical Soft Scores for instructor-role rows. Scores reflect label-family credit, role compatibility, multi-label coverage, and ordering penalties.

Figure 2: SLA-LENS analytical report overview showing anonymized participant information, session-level annotation counts, and guidelines for interpreting the descriptive outputs. All statistics are derived from the annotated transcript without additional LLM inference.

Figure 3: Learner-level descriptive analytics showing approximate word production (top) and error-labeled rows per 100 learner rows (bottom). These measures account for differences in participation but do not independently indicate learner proficiency or language development.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 197 papers

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration
Mar 8, 2026Bakhtawar Ahtisham, Kirk Vanacore, Rene F. KizilcecLLM-Assisted Annotation AI in Education
Cornell University, USA
EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues
Sep 22, 2026Julian Bernado, Ana Trindade Ribeiro, Xander Beberman +1LLM-Assisted Annotation AI in Education
SCALE Initiative, Stanford University
SLATE: Are AI-Generated Slides Educationally Effective? A Benchmark for Language Teaching Quality and Learner Knowledge Acquisition
Sep 5, 2026Jingzhuo Wu, Jiajun Zhang, Liu Yi +4Generative AI in Education Generative AI Evaluation
Beijing Normal University · University of Science and Technology of China · Institute of Automation, Chinese Academy of Sciences +2

Mar 8, 2026J/K move · Enter open · S save

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

Bakhtawar Ahtisham, Kirk Vanacore, Rene F. Kizilcec

Cornell University, USA

LLM-Assisted Annotation AI in Education LLM Reliability Multi-LLM Collaboration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements.

Abstract

LLM-Assisted Annotation

Explore similar work

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues

SLATE: Are AI-Generated Slides Educationally Effective? A Benchmark for Language Teaching Quality and Learner Knowledge Acquisition