From DQI to C-DQI: Automatic Codebook Generation and Evaluation for Measuring Computational Discourse Quality Index
Authors: Esther Shizgal, Ron Keinan, Daniel Shtauber, Roee Tzvi Esquira, Ori Aronson, Matan Toledano, Or Rappel-Kroyzer, Liron Lavi, Renana Keydar, Ittai Bar-Siman-Tov, Reut Tsarfaty
Organizations: Department of Computer Science, Bar-Ilan University · Max Planck Center for Democracy, Security and Human Rights, Faculty of Law, Hebrew University · Faculty of Law, Bar-Ilan University · Digital Humanities, Hebrew University · Department of Political Studies, Bar-Ilan University · Faculty of Law, Hebrew University
Effective democratic deliberation is vital for civic engagement and the health of legislative institutions. However, assessing parliamentary discourse quality is traditionally done using expert-scored, theory-based codebooks (CBs), which are rigid, labor-intensive, and prone to biases. While Large Language Models (LLMs) offer a scalable alternative for both schema induction and discourse annotation, ensuring schema quality and mitigating models' inherent bias remains a clear challenge. In this work, we propose a complete protocol for LLM-based codebook generation and evaluation. Starting from the widely accepted discourse quality Index (DQI), we compare top-down (theory-based), bottom-up (data-driven), and hybrid approaches for generating context-adapted schemas. To evaluate these schemas, we define an extrinsic proxy measure, validated by manual annotations, alongside an intrinsic inter-model agreement (IMA) metric, which leverages differing model biases to signal overall reliability. Our experiments on 100 Israeli parliament transcripts reveal that inductive, data-driven schemas achieve higher IMA than top-down methods while maintaining validity, demonstrating their potential for reliable, large-scale analysis of political discourse.
Figures & tables
Figure 1: The four workflows for deriving CBs for deliberation quality assessment: (a) the standard DQI CB (baseline), and three LLM-generated CBs — (b) theory-led (top-down), (c) data-driven (bottom-up), and (d) a hybrid CB combining the two previous approaches.
Figure 2: The evaluation pipeline: each discussion is segmented and scored against a given CB by multiple models, whose outputs are averaged. Repeating this across discussions yields the CB’s reliability (inter-model agreement) and validity (rank correlation with proxy C-DQI).
Statistic
Mean
Median
Std.
Min
Max
Transcript length
9,393.7
8,414.5
6,545.9
170
27,113
Turns / transcript
365.4
267.5
337.8
5
1,736
Turn length
25.7
10.0
53.5
1
1,364
Speakers per transcript
16.9
17.0
7.9
2
39
Segments per transcript
7.0
6.0
4.5
1
19
Segment length
1,342.0
1,466.0
299.6
45
1,500
Table 1: Structural statistics of the 100-transcript meeting corpus. Length statistics are reported in words.
CB
Krippendorff’s α
Percent Agreement
Spearman’s ρ
Standardized α
Bottom-Up-Hebrew
0.58
0.20
0.75
0.78
Bottom-Up-English
0.57
0.20
0.72
0.73
Bottom-Up-Corrected
0.57
0.17
0.74
0.77
Top-Down
0.49
0.13
0.70
0.72
Hybrid
0.46
0.10
0.70
0.73
Vanilla DQI
0.40
0.20
0.54
0.55
Table 2: Inter-Model Agreement (IMA) across six LLMs over 100 transcripts, computed per category on transcript-level scores and averaged within each CB. Percent Agreement counts only exact matches, which are rare between averaged scores; Krippendorff’s α weights disagreements by size, Spearman’s ρ compares rankings, and Standardized α compares relative scores.
CB
GPT-6-luna ( ρ )
Top-Down
0.66
Bottom-Up-English
0.62
Bottom-Up-Corrected
0.61
Hybrid
0.61
Vanilla DQI
0.59
Bottom-Up-Hebrew
0.59
Table 3: Spearman correlation between each C-DQI CB’s discussion-level score and the proxy C-DQI computed by GPT-6-luna, with the default hyperparameters. CB scores are averaged over the six annotator LLMs per transcript; items are all test transcripts ( n=100 ). All correlations are significant at p<10−9 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: System prompt used by the LLM judge.
Figure 4: User prompt used by the LLM judge (continued on the next page).
Figure 5: User prompt used by the LLM judge (continued). Text in {blue} is filled in per exchange.
Most reliable feature
Least reliable feature
Codebook
Feature
α
Feature
α
Vanilla DQI
Respect toward Groups
0.60
Justification Level
0.08
Top-Down
Respect and Civility
0.79
Justification Level
0.20
Bottom-Up-English
Procedural Decorum
0.83
Emotional Regulation
0.15
Bottom-Up-Corrected
Procedural Decorum
0.80
Emotional Regulation
0.19
Bottom-Up-Hebrew
Decorum and Tone
0.80
Chair Management Efficiency
0.36
Appendix
Table 4: Most and least reliable features per C-DQI codebook, determined by highest and lowest Krippendorff’s α scores.
Figure 6: Distribution of the proxy C-DQI across the 100 evaluation transcripts, computed by averaging each transcript’s 3-turn exchange scores (36,341 in total), judged by GPT-6-luna.
CB
Claude
Gemma4
DictaLM
Qwen3.6
GPT-OSS
Mistral
Bottom-Up-Corrected
0.57
0.55
0.64
0.59
0.58
0.58
Bottom-Up-English
0.59
0.56
0.63
0.54
0.58
0.64
Bottom-Up-Hebrew
0.57
0.53
0.54
0.55
0.53
0.62
Top-Down
0.55
0.64
0.59
0.69
0.65
0.64
Hybrid
0.54
0.56
0.52
0.61
0.59
0.63
Vanilla DQI
0.48
0.53
0.41
0.55
0.49
0.53
Appendix
Table 5: Spearman’s ρ between each C-DQI CB’s discussion-level score and the proxy (GPT-6-luna), broken out by annotator model rather than averaged across models. All correlations significant at p<10−4 .