TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

David Barron; Xiaohang Tang; Rezky Dwisantika; Minsun Kim; David H. Smith; Jiaming Cui; Yan Chen

cs.AIAug 22, 2026

TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

Authors: David Barron, Xiaohang Tang, Rezky Dwisantika, Minsun Kim, David H. Smith, Jiaming Cui, Yan Chen

Organizations: Virginia Tech Blacksburg, Virginia, USA · Sepuluh Nopember Institute of Technology Surabaya, Jawa Timur, Indonesia

Abstract

AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners' needs. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners' behavioral context visible and computable in real time from low-level IDE telemetry. Across four deployments in two introductory Python courses (N=480), TutorTrace captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics. From this foundation, we derive a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session, enabling systems to respond not just to what learners say, but to what they have done leading up to the help-seeking moment. In a preliminary classroom evaluation, behavior-aware prompts were associated with a decrease in intervals between queries with no independent work from 50.0% to 20.7%. As an additional demonstration of downstream utility, we evaluate TutorTrace on two held-out prediction tasks: whether a learner will query within the next 60 seconds (AUROC=.726) and whether an upcoming query reflects guided or dependent help-seeking (AUROC=.717). Together, these findings show how behavioral context can enable adaptive AI tutoring at scale.

Figures & tables

Property	CSEDM	ProgSnap2 ∗	CodeAid	CodeWatcher †	TutorTrace (ours)
Real classroom deployment	✓	✓	✓	✗	✓
Longitudinal coverage (semester)	✓	✓	✓	✗	✗
Raw IDE telemetry	✗	✓	✗	✓	✓
AI query & response logs	✗	✗	✓	✗	✓
Task outcome data	✓	✓	✗	✗	✓
Behavioral classification (expert-validated)	✗	✗	✗	✗	✓

Table 1. Comparison of TutorTrace with existing CS education and AI-assisted programming datasets: the 2nd CSEDM Data Challenge ( CSEDM Data Challenge Organizers 2021 ; Edwards and Murali 2017 ) , ProgSnap2 ( Price et al. 2020 ) , CodeAid ( Kazemitabaar et al. 2024 ) , and CodeWatcher ( Basha et al. 2025 ) .

Figure 1. The student interface with task description, code editor, terminal, test cases, and AI chat window. Screenshot of the student IDE showing a task description and test cases on the left, a code editor and terminal in the center, and an AI chat window on the right.

Region	Example Events	Example Payload
Code Editor	TYPE, DELETE, PASTE,	Characters, cursor
(11 types)	COPY, CUT, UNDO,	position, selection
	REDO, SELECT, INDENT	range
Terminal	RUN, OUTPUT, ERROR,	stdout, error type
(6 types)	RESULT, SELECT, COPY	and message, test
		pass/fail counts

Table 2. Telemetry events captured by TutorTrace, grouped by source region.

Behavior	Subtype	Definition	Derived rule
Implementing	—	Writing new code	CODE_EVENT with no unresolved terminal error
Debugging	—	Fixing an error	CODE_EVENT s while an unresolved error or failed test exists
Testing $\dagger$	—	Executing tests	Automatically marked at each TERMINAL_RUN event
Thinking	Task	Reading the task	$\geq$ 3s inactivity before any CODE, TERMINAL, or CHAT EVENT
	Code	Reviewing code	$\geq$ 6s gap between code edit events
	Error	Reading an error	$\geq$ 3s inactivity after TERMINAL_ERROR

Table 3. Behavioral codebook used by domain-expert annotators to label learner sessions. The table lists each behavior, its subtypes and definition, and the corresponding automated segmentation and classification rule derived through annotator consensus except where noted.

	Self vs Expert		Expert vs Auto		Self vs Auto
Behavior	$\kappa$	Agr.	$\kappa$	Agr.	$\kappa$	Agr.
Implementing	0.93	97%	0.95	98%	0.98	99%
Debugging	0.76	96%	0.88	98%	0.71	94%
Seeking Help	0.90	98%	1.00	100%	0.90	98%
Thinking:
Task	0.92	96%	0.96	98%	0.95	98%

Table 4. Pairwise agreement among learner self-reports, expert annotations, and the automated classifier across six user-study sessions. Cohen’s

\kappa

( Cohen 1960 ) and raw percentage agreement are reported for each behavior and overall.

Observable Area	Metrics
Code activity	code_edits	code_edit_rate
	chars_inserted	chars_deleted
	code_deletes	net_code_growth
	delete_type_ratio	code_pastes †
Terminal activity	terminal_runs	terminal_errors
	max_consecutive_errors	mean_time_between_runs_s

Table 5. The 35 candidate observable metrics grouped by observable area, drawn from prior IDE-based learning analytics literature ( Blikstein 2011 ; Mozannar et al. 2024 ) . Metrics marked

\dagger

were excluded during pruning.

Figure 2. TutorTrace behavioral abstraction pipeline. Raw telemetry events (A) are aggregated into window-specific observable metrics (B) and auto-segmented into behavioral sequences (C). Observable metrics support assignment to the behavioral profiles derived in the Pipeline diagram. Telemetry from the student IDE streams into raw event tokens, which are aggregated into per-window observable metrics and auto-segmented into behavioral sequences. Metrics feed profile assignment in a unified Behavioral Profiles card listing three, four, and three profiles for Windows 1, 2, and 3, while the sequence feeds the prompt directly. An AI tutor prompt box shows the profile, session pattern, recent sequence, and intervention appended after the system line. TutorTrace Taxonomy (D), while behavioral sequences preserve recent ordered activity. These representations can be incorporated into an AI tutor prompt to support behavior-aware adaptation.

Deployment	1	2	3	4
Instructor	A	A	B	B
Time limit	15 min	15 min	10 min	10 min
Task	Playlist	Playlist	Grade Book	Grade Book
Total students	190	113	70	107
Used AI	94	90	48	85
AI Interactions	428	540	190	228

Table 6. Deployment context across four sessions.

Figure 3. The TutorTrace three-window behavioral taxonomy. Each window captures a distinct moment in the help-seeking cycle, characterized by different behavioral signals and requiring different intervention strategies. Table of the three-window taxonomy: each row shows a profile with an example behavioral sequence strip, its description, intervention, share of observations, descriptive statistics, and completion rate.

Prediction Task	Description	AUROC	$\Delta$	AUROC	$\Delta$	AUROC	$\Delta$
		Raw Telemetry		+Observable Metrics		+Behavioral Sequences
Query imminence	Will the learner submit an AI query within the next 60 seconds?	0.689	—	0.726	+0.037	0.719	+0.030
Help-seeking type	Will the learner’s upcoming query reflect guided or dependent help-seeking?	0.690	—	0.717	+0.027	0.705	+0.015

Table 7. Held-out AUROC for query imminence (query within 60 seconds) and help-seeking type (guided vs. dependent within 15 seconds) across nested feature representations. Models were trained on Deployment 1 and evaluated on Deployment 2 (

n{=}113

\Delta

is relative to raw telemetry; bold indicates the best result.

Window 1: Before the First Query
Profile	Labeled $n$	Guided	Dependent
Cold Start	30	3.3%	96.7%
Oriented	97	69.1%	30.9%
Struggling	21	71.4%	28.6%
Window 2: Between Queries
Passive	318	25.8%	74.2%

Table 8. Guided and dependent help-seeking by behavioral profile. Percentages are calculated within profiles.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. The Behavioral Labeling Interface used by expert labelers to annotate student sessions. The interface presents a session replay, a timeline for placing segment boundaries, and the behavioral codebook for classification. Screenshot of the labeling tool with a session replay, an annotatable timeline, and the behavioral codebook.

Date	Change
Feb 2	Initial codebook: Implementing, Debugging, Thinking (with subtypes: Task, Code, Error, LLM Response), Idle, Off-Topic, Unknown.
Feb 3	Added Seeking Help (“Typing a query or reading LLM response”).
Feb 9	Revised Seeking Help description to “Typing a query” only, removing “reading LLM response” to resolve overlap with Thinking: LLM Response subtype.
Feb 10	Added Testing (“Running code and reviewing results”).
Mar 11	Removed Testing from manual labeling codebook (captured by auto-segmentation rules). Added Thinking: Pre-query subtype (“Asking LLM”) to capture formulation pauses before chat queries.
Mar 11	Codebook finalized. All labelers confirmed agreement on category definitions and segmentation rules.

Appendix

Table 9. Codebook revision history across the pilot labeling phase, derived from platform configuration changes.

Event Type	Description	Example Payload
Code Editor (11 types)
CODE_TYPE	Typed characters	{code:"l", changes:[{to:0, from:0, text:"l"}], raw_action:"input.type"}
CODE_DELETE	Backspace/Delete	{code:"...grade_boo", changes:[{to:101, from:100, text:""}], raw_action:"delete.backward"}
CODE_DELETE_SEL.	Deleted selection	{code:"", changes:[{to:145, from:0, text:""}], raw_action:"delete.selection"}
CODE_PASTE	Pasted content	{code:"...append([78,84,91])", changes:[{to:75, from:75, text:"[78,84,91]"}], raw_action:"input.paste"}
CODE_COPY	Copied to clipboard	{selected_text:"grade_book = [[88,92,75],...]"}

Appendix

Table 10. Full telemetry event schema with example payloads from live deployments. Each event includes a millisecond timestamp, source region, and event-specific payload. Long event names and payloads are abbreviated for space.

Layer	Representation	Scale
A: Raw Telemetry	Timestamped IDE events	${\sim}$ 180K
B: Observable Metrics	Effort intensity per window	27 features
C: Behavioral Sequences	Auto-classified segments	13,633
D: Behavioral Profiles	Clustered profiles	10 profiles
Student sessions		480
AI interactions		1,386

Appendix

Table 11. Foundation dataset structure. Layers A–C comprise the TutorTrace foundation; Layer D is derived in the TutorTrace Taxonomy.

Task	Window	Test $n$	Positive	Raw	+Obs.	+Seq.
Query imminence	15 s	11,160	37.4%	0.657	0.690	0.707
	30 s	10,825	38.2%	0.689	0.726	0.719
	45 s	10,492	38.7%	0.701	0.734	0.720
	60 s	10,165	39.1%	0.700	0.730	0.713
Help-seeking type	15 s	499	33.1%	0.690	0.717	0.705
	30 s	499	33.1%	0.696	0.680	0.694

Appendix

Table 12. Held-out AUROC across observation-window sizes for both prediction tasks and all three feature layers. The 30-second query-imminence row and 15-second help-seeking-type row correspond to Table 7 ; bold marks the configuration reported in the subsequent tables.

Feature	Imp.	Feature	Imp.
metric__time_in_chat_s	0.1465	metric__error_to_edit_s	0.0060
metric__longest_idle_s	0.0761	raw__TERMINAL_OUTPUT	0.0058
raw__event_density_per_s	0.0575	raw__CODE_SELECT	0.0057
metric__time_in_editor_s	0.0573	metric__failed_test_to_edit_s	0.0053
raw__event_count	0.0573	raw__TEST_CASE_RESULT	0.0043
raw__MOUSE_MOVE	0.0525	metric__max_consecutive_errors	0.0040

Appendix

Table 13. Complete feature-contribution list (Random-Forest importances, all features with importance

>0

) for query imminence at its reported configuration (30 s window, observable layer).

Feature	Imp.	Feature	Imp.
metric__longest_idle_s	0.1442	raw__TERMINAL_OUTPUT	0.0064
metric__time_in_editor_s	0.0984	metric__max_consecutive_errors	0.0054
raw__event_count	0.0922	raw__WINDOW_RESIZE	0.0050
raw__event_density_per_s	0.0881	raw__TERMINAL_ERROR	0.0049
metric__time_in_chat_s	0.0868	raw__TERMINAL_SELECT	0.0048
raw__MOUSE_MOVE	0.0786	metric__terminal_errors	0.0045

Appendix

Table 14. Complete feature-contribution list (Random-Forest importances, all features with importance

>0

) for help-seeking type at its reported configuration (15 s window, observable layer).

Behavioral context	Windows	Share	Positive rate
W2: Iterating	1,837	17.0%	80.8%
After last query	1,787	16.5%	0.0%
No query this session	1,600	14.8%	0.0%
W2 context (invalid interval)	1,289	11.9%	21.5%
W1: Oriented	1,072	9.9%	52.6%
W2: Passive	1,061	9.8%	91.3%

Appendix

Table 15. Distribution of behavioral contexts among held-out query-imminence observation windows at the reported window (30 s), with each context’s positive rate. Profiles are evaluation strata from the deployment-1–2 taxonomy models.

Preceding profile	Queries	Share	Guided
W2: Passive	186	37.3%	22.6%
W2: Iterating	166	33.3%	33.7%
W1: Oriented	47	9.4%	72.3%
W2: Debugging	28	5.6%	35.7%
W2 (invalid interval)	25	5.0%	36.0%
W1: Cold Start	20	4.0%	0.0%

Appendix

Table 16. Distribution of behavioral profiles among held-out help-seeking-type queries at the reported window (15 s), with each profile’s guided rate.

Intelligent Tutoring Systems

Intelligent tutoring systems model learner knowledge and adapt instruction, practice, hints, and feedback to support learning.

See all 154 papers

The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness
May 7, 2026Rose Niousha, Samantha Boatright Smith, Bita Akram +5Computer Science Education Intelligent Tutoring Systems
University of California, Berkeley, USA · North Carolina State University, USA · University of Pittsburgh, USA +1
Decoding AI Tutor Effects for Educational Measurement: Temporal, Multi-Outcome, and Behavior-Cognitive Analysis
Mar 22, 2026Yiyao Yang, Yasemin GulbaharIntelligent Tutoring Systems User Modeling
Columbia University, New York, NY, United States
TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors
Oct 22, 2026Yeon Su Park, Chanyoung Park, Tae Soo Kim +5AI in Education LLM Evaluation
KAIST · Seoul National University · Korea Institute for Curriculum and Evaluation +2

May 7, 2026J/K move · Enter open · S save

The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

Rose Niousha, Samantha Boatright Smith, Bita Akram, Peter Brusilovsky+4

University of California, Berkeley, USA · North Carolina State University, USA · University of Pittsburgh, USA+1

Computer Science Education Intelligent Tutoring Systems

Open paper AskPDF arXiv

Current Artificial Intelligence (AI)-based tutoring systems (AI tutors) are primarily evaluated based on the pedagogical quality of their feedback messages. While important, pedagogy alone is insufficient because it ignores a critical question: what do students actually do with the feedback they receive? We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data, which complements pedagogical assessment. We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback from an introductory undergraduate programming course to measure whether students act on tutor feedback and whether those actions are applied correctly. Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns that are not captured by pedagogy-only evaluation. Moreover, these engagement-based behavioral signals are more strongly associated with student perception of helpful feedback than pedagogical quality alone, providing a more complete and actionable picture of AI tutor performance.

Abstract

Intelligent Tutoring Systems

Explore similar work

The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

Decoding AI Tutor Effects for Educational Measurement: Temporal, Multi-Outcome, and Behavior-Cognitive Analysis

TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors