AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

Karen Zhou; Chenhao Tan

cs.CLMar 7, 2026

AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

Authors: Karen Zhou, Chenhao Tan

Organizations: University of Chicago

Abstract

Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.

Figures & tables

Figure 1: In the Compare page of AutoChecklist ’s UI, users select methods and enter an input to generate side-by-side checklists, enabling direct comparison of generation strategies on the same task.

Generator	Level	Strategy	Pipelines
Direct	Instance	Single-step generation: given the input (and optionally a reference), directly prompt the LLM to produce a checklist	TICK ( Cook et al., 2024 ) , RLCF Direct ( Viswanathan et al., 2025 ) , RocketEval ( Wei et al., 2025 )
Contrastive	Instance	Counterfactual reasoning: generate candidate responses of varying quality, then derive criteria by contrasting good vs. bad responses	RLCF Candidate-based ( Viswanathan et al., 2025 ) , CRG ( Liu et al., 2026 )
Inductive	Corpus	Bottom-up induction: convert reviewer feedback or observations into general evaluation criteria, with built-in deduplication and selection	Zhou et al. (2025)
Deductive	Corpus	Top-down decomposition: convert expert-defined evaluation dimensions into specific checklist questions, with optional augmentation	Lee et al. (2025)
Interactive	Corpus	Starting with defined dimensions, extract criteria from human and LLM think-aloud evaluation sessions via a multi-stage pipeline of clustering and question generation	Chu et al. (2025)

Table 1: Five checklist generation abstractions in AutoChecklist . Each represents a distinct strategy for deriving evaluation criteria. The rightmost column lists built-in pipelines that instantiate each generator with paper-specific prompt templates and scoring strategies.

Figure 2: Instance-level generators produce one checklist per input; criteria are tailored to each specific task (target). Corpus-level generators produce one checklist for an entire dataset of targets, capturing general quality patterns derived from higher-level signals.

Pipeline	Generator	Score	Ref?
tick	Direct	pass	No
rocketeval	Direct	norm.	Yes
rlcf_direct	Direct	weight.	Yes
rlcf_cand.	Contrastive	weight.	Yes
rlcf_cand._only	Contrastive	weight.	No
or_pairwise	Contrastive	pass	No

Table 2: Built-in pipelines. Each pipeline pairs a generator with a default ChecklistScorer configuration and paper-specific prompt templates. Score metrics: pass rate, normalized, or weighted (see § 3 ).

Figure 3: The Batch results page showing per-item score breakdowns and aggregate statistics for a dataset evaluation run.

Pipeline	W/L/T	Mean $\Delta$	Cohen’s $d$	$t$ -test $p$	Wilcoxon $p$
tick	75 /10/15	+0.286	0.919	6.3e-15	4.3e-12
rlcf	70/16/14	+0.228	0.785	4.9e-12	1.1e-10

Table 3: Preference discrimination on RewardBench pairs using instance-level pipelines. Win = chosen scores higher than rejected, and W/L/T = ratio of Win/Loss/Tie scores.

Pipeline	Dimension	$\rho_{s}$	$r$	$\tau$	MAE
checkeval	Coherence	0.723	0.686	0.571	1.104
	Consistency	0.756	0.709	0.597	0.925
	Fluency	0.819	0.814	0.669	0.829
	Relevance	0.571	0.540	0.430	0.719
interacteval	Coherence	0.755	0.763	0.586	0.703
	Consistency	0.835	0.835	0.699	0.565

Table 4: Corpus-level checklist validation on SummEval.

\rho_{s}

: Spearman,

r

: Pearson,

\tau

: Kendall; MAE on rescaled 1–5 scores. All

\rho_{s}

r

\tau

are significant (

p<.001

	Rating	Acceptance		Rating Change
Generator	$r_{s}$	$\|d\|$	$\Delta\bar{x}$	$r_{pb}$	AUC
Direct	0.253 ∗	0.296 ∗	+0.124	0.143	0.587
Contrastive	0.219 ∗	0.270 ∗	+0.094	0.084	0.531
Deductive	0.267 ∗∗	0.282 ∗	+0.094	0.242 ∗	0.668
Inductive	0.192	0.240 ∗	+0.068	0.172	0.630

Table 5: Evaluation of checklist-based rebuttal scoring on ICLR 2019. Rating : Spearman

r_{s}

between pass rate and reviewer rating. Acceptance : rank-biserial effect size

|d|

and pass rate difference

\Delta\bar{x}=\bar{x}_{acc}-\bar{x}_{rej}

. Rating Change : point-biserial

r_{pb}

and ROC-AUC for predicting whether the reviewer changed their rating post-rebuttal.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Additional UI pages to support prompt iteration and dataset evaluation.

Panel of Figure 5 — Figure 5: Additional results from validation and case studies.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Towards a Reliable and Practical Eval Pipeline
Sep 1, 2026Emma Thuong Nguyen, Abhishek GhoseLLM Evaluation LLM-as-a-Judge
Salesforce
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Oct 18, 2026Saeid Asgari, Rui Ying, Sara Malvar +5Rubric-Based Evaluation LLM-as-a-Judge

Sep 1, 2026J/K move · Enter open · S save

Towards a Reliable and Practical Eval Pipeline

Emma Thuong Nguyen, Abhishek Ghose

Salesforce

LLM Evaluation LLM-as-a-Judge LLM Reliability Rubric-Based Evaluation Automated Evaluation

Open paper AskPDF arXiv

LLM-based software systems increasingly require effective "evals" as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.

Abstract

Automated Evaluation

Explore similar work

Towards a Reliable and Practical Eval Pipeline

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges