CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Jaspreet Ranjit; Swaminathan Chellappa; Augusto Rivas Costante; Chandan Manjunath; Laura Onasch-Vera; Hailey Winetrobe Nadel; Eric Rice; Swabha Swayamdipta

cs.CLOct 16, 2026

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Authors: Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath, Laura Onasch-Vera, Hailey Winetrobe Nadel, Eric Rice, Swabha Swayamdipta

Organizations: Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina

Abstract

We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstructured text with language models. Existing language model annotation systems primarily focus on downstream classification performance, while assuming the availability of high-quality annotation guidelines, or codebooks. In practice, developing these codebooks is time-consuming and challenging, particularly for concepts that require expert knowledge. CoDA addresses this gap by collaborating with domain experts to iteratively develop codebooks through targeted review of incorrect language model predictions and incorporating expert feedback into the evolving codebook. We demonstrate CoDA on two real-world case studies: extracting health indicators from homelessness case management notes and classifying sentiment of wildlife trafficking posts on social media. Across both domains, CoDA produces codebooks that achieve downstream classification performance comparable to expert codebooks while reducing codebook development time from several weeks to two hours and requiring 5X fewer manually annotated examples. Applying CoDA to generate annotations at scale, we identified 2,410 case notes that contained health discussions or referrals and 10,196 social media posts expressing negative sentiment that may warrant investigation for potential wildlife trafficking. By accelerating codebook development, CoDA helps practitioners use language models to extract domain-specific insights from unstructured text at scale.

Figures & tables

Figure 1: CoDA is a human-in-the-loop system for developing annotation guidelines (codebooks) and scaling domain-specific text annotation. Given a small labeled validation set ( $\mathcal{D}_{\text{val}}$ ) and an unlabeled dataset $\mathcal{D}_{\text{all}}$ , the Sampler selects representative samples from $\mathcal{D}_{\text{all}}$ using coverage based sampling. The Generator uses a language model ( $\mathcal{M}_{\text{{CoDA} }}$ ) to predict a label, supporting span, and reasoning. Experts provide feedback on incorrect predictions. The Synthesizer uses $\mathcal{M}_{\text{{CoDA} }}$ to incorporate expert feedback into an evolving codebook ( $\mathcal{G}^{t}_{\text{{CoDA} }}$ ), and the Evaluator prompts $\mathcal{M}_{\text{{CoDA} }}$ with $\mathcal{G}^{t}_{\text{{CoDA} }}$ to generate labels and measure Macro F 1 on $\mathcal{D}_{\text{val}}$ after each iteration. Once the Macro F 1 on $\mathcal{D}_{\text{val}}$ reaches an expert-defined threshold $m$ , or the expert has exhausted the budget $b$ , $\mathcal{G}^{t}_{\text{{CoDA} }}$ is saved at iteration $t$ for large scale annotation. For our evaluated case studies, CoDA reduces codebook development from several weeks to approximately two hours while requiring 5 $\times$ fewer expert-reviewed examples.

Figure 2: Setting up an annotation task in CoDA . (1) Users create a new task from the task library, where they can access existing tasks or initialize a new task for codebook development. (2) Users configure the annotation budget, sampling strategy, and language model, and upload the task metadata and datasets. Task metadata specifies the annotation task and label set; the labeled dataset serves as the validation set ( $\mathcal{D}_{\text{val}}$ ) for evaluating codebooks during development, while the unlabeled corpus ( $\mathcal{D}_{\text{all}}$ ) contains the examples to be annotated at scale.

Figure 3: CoDA ’s main codebook development interface. (1) During each iteration, the language model generates a predicted label, supporting text span, and chain-of-thought reasoning for a sampled example, which the user reviews. (2) For incorrect predictions, the user provides the corrected label and natural-language feedback, which CoDA synthesizes into the evolving codebook shown in the left panel. (3) Throughout codebook development, users can inspect and edit the evolving codebook and monitor its Macro F 1 performance on the held-out validation set $\mathcal{D}_{\text{val}}$ . (4) After completing codebook development, users can export the codebook and evaluation metrics and use the resulting codebook to annotate the full unlabeled corpus $\mathcal{D}_{\text{all}}$ at scale.

	My Friend’s Place			Wildlife Trafficking
Model	$\mathcal{G}^{0}$	$\mathcal{G}^{23}_{\text{{CoDA} }}$	$\mathcal{G}_{\text{expert}}$	$\mathcal{G}^{0}$	$\mathcal{G}^{20}_{\text{{CoDA} }}$	$\mathcal{G}_{\text{expert}}$
Llama-3-70B	0.236	0.418	0.334	0.388	0.556	0.547
Qwen2.5-32B	0.343	0.493	0.446	0.398	0.552	0.529
Qwen2.5-14B	0.200	0.410	0.137	0.429	0.560	0.519

Table 1: Macro F 1 of a baseline codebook (

\mathcal{G}^{0}

), the best synthesized codebook developed with CoDA (

\mathcal{G}^{t}_{\text{{CoDA} }}

), and the expert codebook (

\mathcal{G}_{\text{expert}}

) across both case studies. All CoDA codebooks were developed using Qwen1.5-32B. For each case study, we selected the best-performing CoDA codebook on

D_{\text{val}}

(iteration

t=23

for My Friend’s Place and

t=20

for Wildlife Trafficking) and used it to prompt each of the three language models on the held-out validation set. Across all evaluated models, CoDA codebooks consistently outperform the simple baseline codebook while achieving performance comparable to, and in several cases exceeding, expert codebooks despite requiring substantially less expert time and annotation effort.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Specification
Meta-Llama-3-70B-Instruct	70B parameters
Qwen2.5-32B-Instruct	32B parameters
Qwen2.5-14B-Instruct	14B parameters
GPU	NVIDIA RTX A6000
# GPUs	4 (inference)

Appendix

Table 4: Language models evaluated in this work. Qwen1.5-32B-Instruct was used for codebook development, while all three models were used for evaluation.

Component	Version / Specification
Frontend	React 19.2, TypeScript 5.9, Vite 7.2, Node.js 20 (LTS)
Backend	Express 5.2, MongoDB Node.js Driver 7.0, JWT ( jsonwebtoken 9.0), Argon2
Python Stack	Python 3.11, FastAPI, uvicorn (ASGI), PyTorch (CPU build)
Embeddings	all-mpnet-base-v2
Vector Search	FAISS (CPU)
LLM Serving	Ollama (local/development), OpenRouter API (production)

Appendix

Table 5: Software components and versions used to implement CoDA .

Panel of Figure 4 — Figure 4: Distribution of Llama-3-70B predicted sentiment labels of 237K social media posts on wildlife trafficking of pangolins (left) and Qwen2.5-32B referral related labels of 6K case management notes (right).

Statistic	$\mathbf{D^{30}_{\text{guide}}}$	$\mathbf{D_{\text{val}}}$	$\mathbf{D_{\text{all}}}$
Total Examples	150	585	237,977
Positive	36	128	122,160
Negative	44	251	10,196
Neutral	70	206	105,621

Appendix

Table 6: Dataset statistics and label distributions for the wildlife trafficking sentiment classification task.

Statistic	$\mathbf{D^{30}_{\text{guide}}}$	$\mathbf{D_{\text{val}}}$	$\mathbf{D_{\text{all}}}$
Total Examples	150	593	6,725
Not Relevant	114	416	4,315
Health Discussion	12	87	1,474
Received Referral	24	90	936

Appendix

Table 7: Dataset statistics and label distributions for the My Friend’s Place health referral case study.

Metric	MFP	Wildlife
Total Time (min)	114.0	55.3
LM Explanation Useful (%)	73.2	54.0
LM Span Useful (%)	71.8	96.7

Appendix

Table 8: Annotation efficiency and qualitative feedback collected with CoDA .

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

Sep 13, 2026

CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.

Boqin Yuan, Xiaoyi Gu, Fiona Li +3

UC San Diego · University of Southern California

LLM-Assisted Annotation Human-in-the-Loop Annotation Prompt Optimization

Ask PDF

Sep 22, 2026

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used to strategically target expert attention, shortening months of codebook revision to days without sacrificing labeling performance.

Zeyu He, Zhuqian Zhou, Kirk Vanacore +2

The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA

LLM-Assisted Annotation Human-in-the-Loop Annotation

Ask PDF

Aug 24, 2026

Observational Equivalence of LLM and Human Annotation

In this paper, we show that LLM and human coding are observationally equivalent in terms of annotation quality: recent LLMs agree with expert coders at rates comparable to those observed among experts themselves. We demonstrate this through replications of text-classification tasks from 14 peer-reviewed political science studies, in which ten LLMs, three human experts, and 165 crowdsourced workers independently classify the same texts using identical codebooks. We find that this equivalence is driven by ambiguity in the texts and coding rules. When LLMs disagree with experts, experts are also more likely to disagree with one another, and clarifying coding rules reduces disagreement among both experts and sufficiently capable LLMs. Thus, there is little empirical basis for preferring human coding on the basis of annotation quality alone, while LLMs offer substantial advantages in speed and cost. We therefore argue that the central challenge of text annotation is no longer choosing between human and machine coders, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains. To this end, we propose using disagreement across LLMs to identify difficult cases and refine codebooks, and we develop ambiguity-aware bounds for downstream inference when a unique annotation cannot be defined for every text.

Kentaro Nakamura, Jing Ling Tan, George Yean

Ph.D. student at John F. Kennedy School of Government, Harvard University · Ph.D. student at Department of Government, Harvard University

LLM-Assisted Annotation Annotator Disagreement LLM Evaluation Text Classification

Ask PDF

Abstract

LLM-Assisted Annotation

Explore similar work

CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Observational Equivalence of LLM and Human Annotation