Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts

Jinsook Lee; Kirk Vanacore; Zhuqian Zhou; Bakhtawar Ahtisham; Rene F. Kizilcec

cs.CLApr 3, 2026

Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts

Authors: Jinsook Lee, Kirk Vanacore, Zhuqian Zhou, Bakhtawar Ahtisham, Rene F. Kizilcec

Organizations: Cornell University Ithaca, NY, 14850, USA

Abstract

Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's $κ$ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines ( $κ= 0.275$ - $0.413$ and $0.160$ - $0.410$ ). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.

Figures & tables

Figure 1: Overview of the proposed RAG-based annotation pipeline.

Dataset	Sessions	Utterances	Avg. Utt	Annotated (%)	Train	Test
TalkMoves	61	31,171	511	8,407 (27%)	49	12
Eedi	1,965	46,147	23	30,001 (65%)	1,570	395

Table 1: Dataset statistics. Annotated counts and percentages reflect ground-truth labels across the full dataset (train and test combined). Avg. Utt is the mean utterances per session.

Condition	Chunking	Retrieval	Index	Returned
no_rag	None	None (codebook only)	–	–
RAG_no_finetune	Semantic (Cohere)	Cohere embed-english-v3	Chunk	Chunk
RAG_finetuned_chunk	Semantic (FT-BGE)	Fine-tuned BGE (MNRL)	Chunk	Chunk
RAG_finetuned_utt	Semantic (FT-BGE)	Fine-tuned BGE (MNRL)	Utt.	Chunk

Table 2: Experimental conditions. Index denotes the granularity of FAISS entries. Returned denotes what is passed to the LLM as a few-shot demonstration. RAG_finetuned_chunk and RAG_finetuned_utt share the same embedding model, isolating the effect of index granularity. Additional ablation conditions are reported in Appendix D .

Model	Condition	TM $\kappa$	TM Acc	Eedi $\kappa$	Eedi Acc
GPT-5.2	no_rag	0.315	0.500	0.351	0.575
GPT-5.2	RAG_no_finetune	0.479	0.639	0.505	0.709
GPT-5.2	RAG_finetuned_chunk	0.454	0.620	0.530	0.740
GPT-5.2	RAG_finetuned_utt	0.539	0.688	0.659	0.810
Sonnet 4.6	no_rag	0.413	0.578	0.410	0.714
Sonnet 4.6	RAG_no_finetune	0.498	0.653	0.632	0.797

Table 3: Cohen’s

\kappa

and accuracy across three LLM backbones on both test sets (

k=3

). Bold indicates the best result per metric. All RAG vs. no_rag differences are significant at

p<.001

(permutation test,

n=2{,}000

) except Qwen3-32b

\kappa

on TalkMoves for RAG_no_finetune (

p<.01

	TalkMoves		Eedi
Condition	Top-1	Any- $k$	Top-1	Any- $k$
RAG_no_finetune	39.7%	63.4%	52.9%	75.6%
RAG_finetuned_utt	62.0%	83.4%	73.1%	89.9%
$\Delta$	+22.3pp	+20.0pp	+20.2pp	+14.3pp

Table 4: Retrieval label match rate. Top-1 measures whether the nearest retrieved example shares the target’s gold label. Any-

k

(

k=3

) measures whether at least one of the top-

k

retrieved examples matches. Domain-adapted embeddings produce substantially more label-relevant demonstrations.

Figure 2: Normalized confusion matrices for TalkMoves test (top) and Eedi test (bottom) with Claude Sonnet 4.6. Domain-adapted retrieval ( RAG_finetuned_utt , right) concentrates mass along the diagonal for all labels. Without retrieval, REA is systematically over-predicted on Eedi.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	ACC	KET	REV	RES	REA	GSR
no_rag	0.250	0.472	0.205	0.198	0.232	0.367
RAG_no_finetune	0.473	0.562	0.362	0.386	0.416	0.459
RAG_finetuned_utt	0.552	0.591	0.442	0.479	0.445	0.540
Support ( $n$ )	1,921	1,829	367	288	198	145

Appendix

Table 5: Per-label Cohen’s

\kappa

on TalkMoves test (

k=3

, GPT-5.2).

	ACC	KET	REV	REA	GSR	RES
no_rag	0.344	0.584	0.334	0.104	0.106	0.006
RAG_no_finetune	0.551	0.671	0.263	0.250	0.213	0.020
RAG_finetuned_utt	0.662	0.739	0.642	0.380	0.418	0.444
Support ( $n$ )	5,286	2,779	1,531	157	81	9

Appendix

Table 6: Per-label Cohen’s

\kappa

on Eedi test (

k=3

, GPT-5.2). RES has only 9 support instances. RAG_no_finetune underperforms no_rag on REV (

0.263

vs.

0.334

	Precision	Recall	F1	Support
KET	0.68	0.66	0.67	1,829
GSR	0.55	0.29	0.38	145
RES	0.22	0.30	0.25	287
ACC	0.62	0.41	0.49	1,921
REV	0.56	0.14	0.23	366
REA	0.17	0.96	0.29	198

Appendix

Table 7: no_rag classification report on TalkMoves test (GPT-5.2).

	Precision	Recall	F1	Support
KET	0.77	0.66	0.71	701
GSR	0.70	0.36	0.47	73
RES	0.46	0.39	0.42	111
ACC	0.68	0.71	0.69	774
REV	0.61	0.29	0.39	147
REA	0.30	0.99	0.46	84

Appendix

Table 8: RAG_no_finetune classification report on TalkMoves test (GPT-5.2).

	Precision	Recall	F1	Support
KET	0.83	0.65	0.73	534
GSR	0.64	0.49	0.55	43
RES	0.53	0.48	0.50	69
ACC	0.72	0.78	0.75	589
REV	0.68	0.36	0.47	90
REA	0.32	0.97	0.48	62

Appendix

Table 9: RAG_finetuned_utt classification report on TalkMoves test (GPT-5.2).

	Precision	Recall	F1	Support
KET	0.79	0.59	0.67	2,335
GSR	0.21	0.07	0.11	81
RES	0.00	0.17	0.01	6
ACC	0.75	0.65	0.69	5,265
REV	0.95	0.23	0.37	1,334
REA	0.07	0.99	0.13	157

Appendix

Table 10: no_rag classification report on Eedi test (GPT-5.2).

	Precision	Recall	F1	Support
KET	0.86	0.67	0.75	1,788
GSR	0.40	0.15	0.22	54
RES	0.01	0.50	0.02	4
ACC	0.78	0.86	0.81	3,522
REV	0.94	0.17	0.29	842
REA	0.16	1.00	0.27	105

Appendix

Table 11: RAG_no_finetune classification report on Eedi test (GPT-5.2).

	Precision	Recall	F1	Support
KET	0.96	0.69	0.80	852
GSR	0.44	0.41	0.42	27
RES	0.29	1.00	0.44	2
ACC	0.82	0.93	0.87	1,753
REV	0.95	0.52	0.67	359
REA	0.25	1.00	0.40	52

Appendix

Table 12: RAG_finetuned_utt classification report on Eedi test (GPT-5.2).

Model	Condition	KET	GSR	RES	ACC	REV	REA
GPT-5.2	no_rag	0.472	0.367	0.198	0.250	0.205	0.232
GPT-5.2	RAG_no_finetune	0.562	0.459	0.386	0.473	0.362	0.416
GPT-5.2	RAG_finetuned_utt	0.591	0.540	0.479	0.552	0.442	0.445
Sonnet 4.6	no_rag	0.421	0.506	0.410	0.397	0.548	0.353
Sonnet 4.6	RAG_no_finetune	0.577	0.464	0.461	0.480	0.522	0.360
Sonnet 4.6	RAG_finetuned_utt	0.631	0.528	0.542	0.594	0.566	0.448

Appendix

Table 13: Per-label Cohen’s

\kappa

on TalkMoves test across three LLM backbones (

k=3

). Bold indicates the best RAG_finetuned_utt result per label.

Model	Condition	KET	GSR	RES	ACC	REV	REA
GPT-5.2	no_rag	0.584	0.106	0.006	0.344	0.334	0.104
GPT-5.2	RAG_no_finetune	0.671	0.213	0.020	0.551	0.263	0.250
GPT-5.2	RAG_finetuned_utt	0.739	0.418	0.444 †	0.662	0.642	0.380
Sonnet 4.6	no_rag	0.303	0.300	—	0.478	0.643	0.207
Sonnet 4.6	RAG_no_finetune	0.683	0.292	—	0.670	0.653	0.330
Sonnet 4.6	RAG_finetuned_utt	0.749	0.387	0.104	0.775	0.855	0.361

Appendix

Table 14: Per-label Cohen’s

\kappa

on Eedi test across three LLM backbones (

k=3

). Bold indicates the best RAG_finetuned_utt result per label. † RES has only 9 support instances

Condition	Chunking	Retrieval	TM $\kappa$	Eedi $\kappa$
no_rag	None	None	0.313	0.351
RAG_no_finetune	Semantic (Cohere)	Cohere	0.474	0.505
rag_fixed_w3	Fixed ( $w{=}3$ )	Cohere	0.500	0.571
rag_fixed_w5	Fixed ( $w{=}5$ )	Cohere	0.475	0.519
RAG_finetuned_utt	Fine-tuned BGE	Fine-tuned BGE	0.539	0.659

Appendix

Table 15: Fixed-window vs. semantic chunking ablation on test sets (

k=3

, GPT-5.2). All conditions except no_rag and RAG_finetuned_utt use Cohere for retrieval. Sample sizes differ across conditions due to index coverage.

	KET	GSR	RES	ACC	REV	REA
rag_fixed_w3	0.554	0.254	0.731	0.470	0.549	0.350
rag_fixed_w5	0.537	0.396	0.691	0.447	0.553	0.293
RAG_finetuned_utt	0.591	0.540	0.479	0.552	0.442	0.445

Appendix

Table 16: Per-label

\kappa

on TalkMoves test for fixed-window conditions vs. RAG_finetuned_utt .

	KET	GSR	RES	ACC	REV	REA
rag_fixed_w3	0.676	0.195	-0.000	0.599	0.534	0.256
rag_fixed_w5	0.607	0.069	0.166	0.552	0.592	0.187
RAG_finetuned_utt	0.739	0.418	0.444	0.662	0.642	0.380

Appendix

Table 17: Per-label

\kappa

on Eedi test for fixed-window conditions vs. RAG_finetuned_utt .

Figure 3: Confidence score distributions by model and retrieval condition on the combined test sets. Domain-adapted retrieval ( RAG_finetuned_utt ) concentrates prediction mass at higher confidence levels across all three models.

Model	Condition	$\kappa$ ( $\geq 0.9$ )	Coverage
GPT-5.2	no_rag	0.086	19.1%
GPT-5.2	RAG_no_finetune	0.412	23.8%
GPT-5.2	RAG_finetuned_utt	0.939	11.6%
Sonnet 4.6	no_rag	0.594	3.9%
Sonnet 4.6	RAG_no_finetune	0.838	24.3%
Sonnet 4.6	RAG_finetuned_utt	0.896	34.3%

Appendix

Table 18: Cohen’s

\kappa

and coverage for predictions with confidence

\geq 0.9

. Coverage is the fraction of test utterances where the model’s confidence exceeds the threshold. GPT-5.2 with RAG_finetuned_utt achieves near-perfect agreement (

\kappa=0.939

) on its high-confidence subset, though coverage is limited to 11.6%.

Figure 4: Cohen’s $\kappa$ as a function of retrieval depth $k$ across three LLM backbones on TalkMoves test (top) and Eedi test (bottom). Left panels show RAG_no_finetune , right panels show RAG_finetuned_utt . Domain-adapted retrieval consistently outperforms general-purpose retrieval at every $k$ for all models.

Model	Condition	$\kappa$	Emb $\Delta\kappa$	Gran $\Delta\kappa$
GPT-5.2	RAG_no_finetune	0.479
GPT-5.2	RAG_finetuned_chunk	0.454	$-0.025$
GPT-5.2	RAG_finetuned_utt	0.539		$+0.085$
Sonnet 4.6	RAG_no_finetune	0.498
Sonnet 4.6	RAG_finetuned_chunk	0.501	$+0.003$
Sonnet 4.6	RAG_finetuned_utt	0.580		$+0.079$

Appendix

Table 19: Confound isolation on TalkMoves test (

k=3

). Emb

\Delta\kappa

is the effect of switching embeddings (Cohere

\to

FT-BGE) at chunk-level granularity. Gran

\Delta\kappa

is the effect of switching granularity (chunk

\to

utterance) with FT-BGE.

Model	Condition	$\kappa$	Emb $\Delta\kappa$	Gran $\Delta\kappa$
GPT-5.2	RAG_no_finetune	0.505
GPT-5.2	RAG_finetuned_chunk	0.530	$+0.025$
GPT-5.2	RAG_finetuned_utt	0.659		$+0.129$
Sonnet 4.6	RAG_no_finetune	0.632
Sonnet 4.6	RAG_finetuned_chunk	0.632 †	$+0.000$
Sonnet 4.6	RAG_finetuned_utt	0.743		$+0.111$

Appendix

Table 20: Confound isolation on Eedi test (

k=3

). † Sonnet RAG_finetuned_chunk is based on an incomplete run (

\sim

253 records short). Embedding model effect (Emb

\Delta\kappa

) ranges from

-0.045

+0.025

. Granularity effect (Gran

\Delta\kappa

) ranges from

+0.079

+0.149

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 197 papers

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors
May 27, 2026Parth Bhalerao, Jeromy Chang, David Chou +1Intelligent Tutoring Systems Low-Rank Adaptation
Santa Clara University
The Assistance Dilemma: Learning to Teach via Multi-Turn Reinforcement Learning
Oct 5, 2026Jakub Macina, Manu Kapur, Mrinmaya SachanRL for Language Models Intelligent Tutoring Systems
ETH Zurich
TACT: Taxonomy-Aligned Post-Training for Pedagogically Adaptive English Tutoring
Aug 4, 2026Dongjie Yang, Siyan Lin, Leixian Shen +3Intelligent Tutoring Systems AI in Education

May 27, 2026J/K move · Enter open · S save

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors

Parth Bhalerao, Jeromy Chang, David Chou, Oana Ignat

Santa Clara University

Intelligent Tutoring Systems Low-Rank Adaptation Generative AI Evaluation Educational Dialogue Systems

Open paper AskPDF arXiv

Evaluating AI tutor responses requires more than factual correctness: tutors must identify mistakes, locate errors, provide guidance, and offer actionable next steps. We present GRADE, a systematic study of open-source models for pedagogical ability assessment in student-tutor dialogues. Building on the BEA 2025 TutorMind setting, we evaluate 120 configurations across five language models, zero-shot inference, LoRA fine-tuning, synthetic augmentation, CoT+Reasoning, and single-task versus multitask formulations. Gemma3-12B performs best for single-task evaluation, while Gemma3-27B in 8-bit precision is more reliable for multitask prediction. We find that augmentation helps models that struggle with the original data, verification adds limited gains despite higher cost, and CoT+Reasoning is more useful for synthetic data generation than direct classification. We further show that LoRA fine-tuning on structured classification objectives interferes with instruction-following behavior under thinking mode, redirecting generation away from the required evaluation format. Carbon analysis shows that model choice and reasoning mode substantially affect emissions. Overall, GRADE shows that carefully selected open-source LoRA pipelines can match or surpass proprietary and ensemble-based systems on key pedagogical dimensions, with code and data available at https://github.com/pvbgeek/GRADE.

Abstract

LLM-Assisted Annotation

Explore similar work

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors

The Assistance Dilemma: Learning to Teach via Multi-Turn Reinforcement Learning

TACT: Taxonomy-Aligned Post-Training for Pedagogically Adaptive English Tutoring