CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Jaspreet Ranjit; Swaminathan Chellappa; Augusto Rivas Costante; Chandan Manjunath; Laura Onasch-Vera; Hailey Winetrobe Nadel; Eric Rice; Swabha Swayamdipta

cs.CLOct 16, 2026

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Authors: Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath, Laura Onasch-Vera, Hailey Winetrobe Nadel, Eric Rice, Swabha Swayamdipta

Organizations: Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina

Abstract

We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstructured text with language models. Existing language model annotation systems primarily focus on downstream classification performance, while assuming the availability of high-quality annotation guidelines, or codebooks. In practice, developing these codebooks is time-consuming and challenging, particularly for concepts that require expert knowledge. CoDA addresses this gap by collaborating with domain experts to iteratively develop codebooks through targeted review of incorrect language model predictions and incorporating expert feedback into the evolving codebook. We demonstrate CoDA on two real-world case studies: extracting health indicators from homelessness case management notes and classifying sentiment of wildlife trafficking posts on social media. Across both domains, CoDA produces codebooks that achieve downstream classification performance comparable to expert codebooks while reducing codebook development time from several weeks to two hours and requiring 5X fewer manually annotated examples. Applying CoDA to generate annotations at scale, we identified 2,410 case notes that contained health discussions or referrals and 10,196 social media posts expressing negative sentiment that may warrant investigation for potential wildlife trafficking. By accelerating codebook development, CoDA helps practitioners use language models to extract domain-specific insights from unstructured text at scale.

Figures & tables

Figure 1: CoDA is a human-in-the-loop system for developing annotation guidelines (codebooks) and scaling domain-specific text annotation. Given a small labeled validation set ( $\mathcal{D}_{\text{val}}$ ) and an unlabeled dataset $\mathcal{D}_{\text{all}}$ , the Sampler selects representative samples from $\mathcal{D}_{\text{all}}$ using coverage based sampling. The Generator uses a language model ( $\mathcal{M}_{\text{{CoDA} }}$ ) to predict a label, supporting span, and reasoning. Experts provide feedback on incorrect predictions. The Synthesizer uses $\mathcal{M}_{\text{{CoDA} }}$ to incorporate expert feedback into an evolving codebook ( $\mathcal{G}^{t}_{\text{{CoDA} }}$ ), and the Evaluator prompts $\mathcal{M}_{\text{{CoDA} }}$ with $\mathcal{G}^{t}_{\text{{CoDA} }}$ to generate labels and measure Macro F 1 on $\mathcal{D}_{\text{val}}$ after each iteration. Once the Macro F 1 on $\mathcal{D}_{\text{val}}$ reaches an expert-defined threshold $m$ , or the expert has exhausted the budget $b$ , $\mathcal{G}^{t}_{\text{{CoDA} }}$ is saved at iteration $t$ for large scale annotation. For our evaluated case studies, CoDA reduces codebook development from several weeks to approximately two hours while requiring 5 $\times$ fewer expert-reviewed examples.

Figure 2: Setting up an annotation task in CoDA . (1) Users create a new task from the task library, where they can access existing tasks or initialize a new task for codebook development. (2) Users configure the annotation budget, sampling strategy, and language model, and upload the task metadata and datasets. Task metadata specifies the annotation task and label set; the labeled dataset serves as the validation set ( $\mathcal{D}_{\text{val}}$ ) for evaluating codebooks during development, while the unlabeled corpus ( $\mathcal{D}_{\text{all}}$ ) contains the examples to be annotated at scale.

Figure 3: CoDA ’s main codebook development interface. (1) During each iteration, the language model generates a predicted label, supporting text span, and chain-of-thought reasoning for a sampled example, which the user reviews. (2) For incorrect predictions, the user provides the corrected label and natural-language feedback, which CoDA synthesizes into the evolving codebook shown in the left panel. (3) Throughout codebook development, users can inspect and edit the evolving codebook and monitor its Macro F 1 performance on the held-out validation set $\mathcal{D}_{\text{val}}$ . (4) After completing codebook development, users can export the codebook and evaluation metrics and use the resulting codebook to annotate the full unlabeled corpus $\mathcal{D}_{\text{all}}$ at scale.

	My Friend’s Place			Wildlife Trafficking
Model	$\mathcal{G}^{0}$	$\mathcal{G}^{23}_{\text{{CoDA} }}$	$\mathcal{G}_{\text{expert}}$	$\mathcal{G}^{0}$	$\mathcal{G}^{20}_{\text{{CoDA} }}$	$\mathcal{G}_{\text{expert}}$
Llama-3-70B	0.236	0.418	0.334	0.388	0.556	0.547
Qwen2.5-32B	0.343	0.493	0.446	0.398	0.552	0.529
Qwen2.5-14B	0.200	0.410	0.137	0.429	0.560	0.519

Table 1: Macro F 1 of a baseline codebook (

\mathcal{G}^{0}

), the best synthesized codebook developed with CoDA (

\mathcal{G}^{t}_{\text{{CoDA} }}

), and the expert codebook (

\mathcal{G}_{\text{expert}}

) across both case studies. All CoDA codebooks were developed using Qwen1.5-32B. For each case study, we selected the best-performing CoDA codebook on

D_{\text{val}}

(iteration

t=23

for My Friend’s Place and

t=20

for Wildlife Trafficking) and used it to prompt each of the three language models on the held-out validation set. Across all evaluated models, CoDA codebooks consistently outperform the simple baseline codebook while achieving performance comparable to, and in several cases exceeding, expert codebooks despite requiring substantially less expert time and annotation effort.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Specification
Meta-Llama-3-70B-Instruct	70B parameters
Qwen2.5-32B-Instruct	32B parameters
Qwen2.5-14B-Instruct	14B parameters
GPU	NVIDIA RTX A6000
# GPUs	4 (inference)

Appendix

Table 4: Language models evaluated in this work. Qwen1.5-32B-Instruct was used for codebook development, while all three models were used for evaluation.

Component	Version / Specification
Frontend	React 19.2, TypeScript 5.9, Vite 7.2, Node.js 20 (LTS)
Backend	Express 5.2, MongoDB Node.js Driver 7.0, JWT ( jsonwebtoken 9.0), Argon2
Python Stack	Python 3.11, FastAPI, uvicorn (ASGI), PyTorch (CPU build)
Embeddings	all-mpnet-base-v2
Vector Search	FAISS (CPU)
LLM Serving	Ollama (local/development), OpenRouter API (production)

Appendix

Table 5: Software components and versions used to implement CoDA .

Panel of Figure 4 — Figure 4: Distribution of Llama-3-70B predicted sentiment labels of 237K social media posts on wildlife trafficking of pangolins (left) and Qwen2.5-32B referral related labels of 6K case management notes (right).

Statistic	$\mathbf{D^{30}_{\text{guide}}}$	$\mathbf{D_{\text{val}}}$	$\mathbf{D_{\text{all}}}$
Total Examples	150	585	237,977
Positive	36	128	122,160
Negative	44	251	10,196
Neutral	70	206	105,621

Appendix

Table 6: Dataset statistics and label distributions for the wildlife trafficking sentiment classification task.

Statistic	$\mathbf{D^{30}_{\text{guide}}}$	$\mathbf{D_{\text{val}}}$	$\mathbf{D_{\text{all}}}$
Total Examples	150	593	6,725
Not Relevant	114	416	4,315
Health Discussion	12	87	1,474
Received Referral	24	90	936

Appendix

Table 7: Dataset statistics and label distributions for the My Friend’s Place health referral case study.

Metric	MFP	Wildlife
Total Time (min)	114.0	55.3
LM Explanation Useful (%)	73.2	54.0
LM Span Useful (%)	71.8	96.7

Appendix

Table 8: Annotation efficiency and qualitative feedback collected with CoDA .

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

CALICO: A Human-Centered, Codebook-Aligned System for Annotation
Sep 13, 2026Boqin Yuan, Xiaoyi Gu, Fiona Li +3LLM-Assisted Annotation Human-in-the-Loop Annotation
UC San Diego · University of Southern California
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted Annotation Human-in-the-Loop Annotation
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
Observational Equivalence of LLM and Human Annotation
Aug 24, 2026Kentaro Nakamura, Jing Ling Tan, George YeanLLM-Assisted Annotation Annotator Disagreement
Ph.D. student at John F. Kennedy School of Government, Harvard University · Ph.D. student at Department of Government, Harvard University

Sep 13, 2026J/K move · Enter open · S save

CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Boqin Yuan, Xiaoyi Gu, Fiona Li, Chang Wan+2

UC San Diego · University of Southern California

LLM-Assisted Annotation Human-in-the-Loop Annotation Prompt Optimization

Open paper AskPDF arXiv

Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.

Abstract

LLM-Assisted Annotation

Explore similar work

CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Observational Equivalence of LLM and Human Annotation