Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

Daryl Hedley; Doug Pietrzak; Jorge Dias; Ian Burden; Bakhtawar Ahtisham; Zhuqian Zhou; Kirk Vanacore; Josh Marland

cs.HCMar 9, 2026

Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

Authors: Daryl Hedley, Doug Pietrzak, Jorge Dias, Ian Burden, Bakhtawar Ahtisham, Zhuqian Zhou, Kirk Vanacore, Josh Marland, +4 more

Organizations: National Tutoring Observatory · FreshCognate · Cornell University · Massachusetts Institute of Technology · Carnegie Mellon University

Abstract

Digital educational environments are expanding toward complex AI and human discourse, providing researchers with an abundance of data that offers deep insights into learning and instructional processes. However, traditional qualitative analysis remains a labor-intensive bottleneck, severely limiting the scale at which this research can be conducted. We present Sandpiper, a mixed-initiative system designed to serve as a bridge between high-volume conversational data and human qualitative expertise. By tightly coupling interactive researcher dashboards with agentic Large Language Model (LLM) engines, the platform enables scalable analysis without sacrificing methodological rigor. Sandpiper addresses critical barriers to AI adoption in education by implementing context-aware, automated de-identification workflows supported by secure, university-housed infrastructure to ensure data privacy. Furthermore, the system employs schema-constrained orchestration to eliminate LLM hallucinations and enforces strict adherence to qualitative codebooks. An integrated evaluations engine allows for the continuous benchmarking of AI performance against human labels, fostering an iterative approach to model refinement and validation. We propose a user study to evaluate the system's efficacy in improving research efficiency, inter-rater reliability, and researcher trust in AI-assisted qualitative workflows.

Figures & tables

Design Goal	Description	Solution
DG1: Privacy Protection and Data Security.	Guarantee the privacy of educational data through scalable methods.	Secure, university-housed LLM cluster use and data storage.
DG2: Schema-Constrained Reliability.	Prevent LLM hallucinations and malformed tags during the coding process.	LLM agentic orchestration to ensure rigorous enforcement of data structure and qualitative codebook schemas and annotation validation during inference.
DG3: Verification and Benchmarking.	Support iterative refinement and validation of automated annotation.	Dedicated evaluation engine assesses alignment and classification accuracy of AI and human labels.

Table 1: Detailed Mapping of Design Goals, Descriptions, and Proposed Solutions

Panel of Figure 1 — Figure 1: The Sandpiper Pipeline. A central mixed-initiative loop tightly integrates the researcher dashboard and the schema-constrained orchestrator. The backend API and datastore are hosted within Cornell secure servers, while external model access is mediated through Cornell’s AI gateway.

Figure 2: The Prompt Editor interface, illustrating version control for instructions and explicitly defined JSON coding schemas.

Figure 3: System mechanisms for ensuring inference output perfectly aligns with the target qualitative schema.

Figure 4: The Session Explorer interface detailing the Chat View and interconnected labeling panel.

Figure 5: The Evaluations Dashboard aggregating metrics for multiple Experimental Run-Sets.

Explore similar work

Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use
Jul 30, 2026Alex Liu, Min Sun, Lief Esbenshade +4LLM-Assisted Annotation Human-in-the-Loop Annotation
College of Education, University of Washington, Seattle, WA, United States · Colleague AI
Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration
Mar 8, 2026Bakhtawar Ahtisham, Kirk Vanacore, Rene F. KizilcecLLM-Assisted Annotation AI in Education
Cornell University, USA
Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research
Apr 4, 2026Max Hao Lu, Ryan Ellegood, Rony Rodriguez-Ramirez +1LLM-Assisted Annotation LLM Auditing
Harvard University / United States

Jul 30, 2026J/K move · Enter open · S save

Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use

Alex Liu, Min Sun, Lief Esbenshade, Michael Xiao+3

College of Education, University of Washington, Seattle, WA, United States · Colleague AI

LLM-Assisted Annotation Human-in-the-Loop Annotation Human-AI Collaboration

Open paper AskPDF arXiv

Qualitative researchers increasingly encounter interaction corpora whose scale exceeds what manual coding alone can address, and large language models (LLMs) are frequently proposed as analytic assistants. The open questions are not whether LLMs can participate in qualitative analysis but to what extent, in what phases, and under what safeguards. This article provides a detailed procedural account of a multi-phase human-LLM collaborative pipeline that adapted open, axial, and selective coding to develop a hierarchical codebook from 45,000 messages exchanged between K-12 educators and a generative AI platform. Across three phases, LLMs generated candidate labels and structured annotations at scale, while human researchers retained conceptual authority over category definitions, merging decisions, and interpretive frameworks. The resulting instrument was then tested through systematic human coding, in which three trained coders with educational domain expertise applied the codebook to an independent sample of 2,560 messages, established reliability through iterative calibration using set-valued agreement measures appropriate for multi-label annotation, and extended the instrument with five codes that the LLM-assisted phases had not surfaced. The final codebook comprises 72 items within 19 categories and six domains. We reflect on the methodological decisions the pipeline required, including the choice of a conversational unit of analysis, the treatment of the LLM as a labeling instrument rather than an interpretive agent, the measurement of intercoder agreement under multi-label coding, and the conditions under which human domain expertise remained decisive. The account is offered as an auditable template for qualitative researchers considering LLM assistance in codebook development while preserving human interpretive authority.

Abstract

Explore similar work

Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research