CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Boqin Yuan; Xiaoyi Gu; Fiona Li; Chang Wan; Angel Hsing-Chi Hwang; Jieyu Zhao

cs.HCSep 13, 2026

CALICO: A Human-Centered, Codebook-Aligned System for Annotation

Authors: Boqin Yuan, Xiaoyi Gu, Fiona Li, Chang Wan, Angel Hsing-Chi Hwang, Jieyu Zhao

Organizations: UC San Diego · University of Southern California

Abstract

Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.

Figures & tables

Figure 1: The CALICO workflow. Users create a project from a codebook and optional data, generate dimension-specific prompts, run annotation, inspect outputs, refine prompts when needed, and export final labels. Prompt versions are managed within the project so users can iterate before final annotation.

Figure 2: Feedback and prompt-update interface. Users provide natural language feedback and review the proposed prompt revision before applying it as a new prompt version.

	Coder A			Coder B
Dimension	Zero-shot	+ ReflectAgent	$\Delta$	Zero-shot	+ ReflectAgent	$\Delta$
Self-Disclosure (single-label, accuracy %)
Level of disclosure	$68.1_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.7}}$	$\mathbf{70.0}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 1.7}}$	$+1.9$	$61.7_{\,{\color[rgb]{0.5,0.5,0.5}\pm 0.6}}$	$\mathbf{76.6}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 5.1}}$	$+14.9$
Disclosure as confession	$52.6_{\,{\color[rgb]{0.5,0.5,0.5}\pm 1.2}}$	$\mathbf{82.0}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 8.7}}$	$+29.4$	$49.6_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.8}}$	$\mathbf{50.1}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 1.9}}$	$+0.5$
Depth of disclosure	$\mathbf{61.0}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 6.6}}$	$58.7_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$-2.3$	$58.2_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.5}}$	$\mathbf{61.2}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 7.2}}$	$+3.0$
Intimacy of self-disclosure	$52.2_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.2}}$	$\mathbf{75.5}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.1}}$	$+23.3$	$54.9_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.5}}$	$\mathbf{66.0}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.5}}$	$+11.1$

Table 1: Per-coder agreement with each coder’s held-out labels, comparing the zero-shot prompt with ReflectAgent . Scores are accuracy (%) for Self-Disclosure and micro-

F_{1}

(%) for AI Behavior. Results are reported as mean

\pm

standard deviation over three seeds. Bold indicates the better score between zero-shot and ReflectAgent ; green and red

\Delta

values indicate absolute gains and drops from zero-shot. For AI Behavior, we report results on the Listening strategy theme.

Dim.	ZS	OPRO	GEPA	MIPRO	RA
Level	68.1	$72.4_{\,{\color[rgb]{0.5,0.5,0.5}\pm 4.6}}$	$\mathbf{74.8}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 1.6}}$	$72.2_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.0}}$	$70.0_{\,{\color[rgb]{0.5,0.5,0.5}\pm 1.7}}$
Confession	52.6	$52.6_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.6}}$	$77.8_{\,{\color[rgb]{0.5,0.5,0.5}\pm 5.5}}$	$\mathbf{85.2}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 7.3}}$	$82.0_{\,{\color[rgb]{0.5,0.5,0.5}\pm 8.7}}$
Depth	61.0	$58.7_{\,{\color[rgb]{0.5,0.5,0.5}\pm 6.3}}$	$\mathbf{64.8}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.4}}$	$62.0_{\,{\color[rgb]{0.5,0.5,0.5}\pm 5.0}}$	$58.7_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$
Intimacy	52.2	$62.3_{\,{\color[rgb]{0.5,0.5,0.5}\pm 4.6}}$	$\mathbf{78.0}_{\,{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$76.1_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.6}}$	$75.5_{\,{\color[rgb]{0.5,0.5,0.5}\pm 3.1}}$
mean	58.5	61.5	73.8	73.9	71.5

Table 2: Comparison of prompt optimization backends on Coder A’s Self-Disclosure dimensions. Scores are accuracy (%); optimized methods report mean

\pm

standard deviation over three seeds. All methods start from the same zero-shot (ZS) prompt and use identical data splits. Bold indicates the best score in each row. RA denotes ReflectAgent .

Tuned on	Eval. vs Coder A	Eval. vs Coder B
Coder A labels	75.2	41.0
Coder B labels	51.4	71.4

Table 3: Coder-specificity analysis on the Intimacy dimension. Values are mean held-out accuracy (%) over three seeds on the shared held-out set (

n{=}35

). Rows indicate the coder used for tuning, and columns indicate the coder used for evaluation. Matched-coder prompts outperform swapped-coder prompts by

+23.8{\pm}7.5

pp for Coder A and

+30.5{\pm}10.8

pp for Coder B, suggesting that ReflectAgent captures coder-specific label boundaries.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Coder A		Coder B
Dimension	ZS	+RA	ZS	+RA
Level	0.581	0.598	0.659	0.858
Confession	0.482	0.686	0.457	0.529
Depth	0.569	0.549	0.581	0.609
Intimacy	0.525	0.643	0.532	0.611

Appendix

Table 4: Macro-

F_{1}

on the Self-Disclosure dimensions for one split seed. ZS denotes the zero-shot prompt, and +RA denotes the prompt after ReflectAgent . Bold indicates the better score between ZS and +RA for each coder and dimension.

Model	Zero-shot	+RA	$\Delta$
gpt-5.4-mini	52.2	75.5	+23.3
gpt-5.5	45.3	78.3	+33.0
gemini-3.1-pro	38.4	73.0	+34.6
Qwen3-8B	37.1	57.9	+20.8

Appendix

Table 5: Intimacy agreement with Coder A (accuracy %) across annotation models, zero-shot versus ReflectAgent (+RA), on identical stratified splits across three seeds. Qwen3-8B is self-hosted via vLLM.

Figure 4: Feedback and prompt-update interface. Users provide natural language feedback and review the proposed prompt revision before applying it as a new prompt version.

Figure 5: The label-supervised prompt improvement view in CALICO , for a ReflectAgent run on the Level of disclosure dimension.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation
Oct 16, 2026Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5LLM-Assisted Annotation Human-in-the-Loop Annotation
Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted Annotation Human-in-the-Loop Annotation
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinement
May 27, 2026Jyotirmoy Nath, Neeraj Kumar, Brejesh LallLLM Prompting Large Language Model-Guided Optimization
1IIT Delhi, India

Oct 16, 2026J/K move · Enter open · S save

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath+4

Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina

LLM-Assisted Annotation Human-in-the-Loop Annotation LLM Information Extraction

Open paper AskPDF arXiv

We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstructured text with language models. Existing language model annotation systems primarily focus on downstream classification performance, while assuming the availability of high-quality annotation guidelines, or codebooks. In practice, developing these codebooks is time-consuming and challenging, particularly for concepts that require expert knowledge. CoDA addresses this gap by collaborating with domain experts to iteratively develop codebooks through targeted review of incorrect language model predictions and incorporating expert feedback into the evolving codebook. We demonstrate CoDA on two real-world case studies: extracting health indicators from homelessness case management notes and classifying sentiment of wildlife trafficking posts on social media. Across both domains, CoDA produces codebooks that achieve downstream classification performance comparable to expert codebooks while reducing codebook development time from several weeks to two hours and requiring 5X fewer manually annotated examples. Applying CoDA to generate annotations at scale, we identified 2,410 case notes that contained health discussions or referrals and 10,196 social media posts expressing negative sentiment that may warrant investigation for potential wildlife trafficking. By accelerating codebook development, CoDA helps practitioners use language models to extract domain-specific insights from unstructured text at scale.

Abstract

LLM-Assisted Annotation

Explore similar work

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinement