Confronting Label Indeterminacy in Automated Bail Decisions

Cor Steging; Tadeusz Zbiegień

cs.LGApr 13, 2026

Confronting Label Indeterminacy in Automated Bail Decisions

Authors: Cor Steging, Tadeusz Zbiegień

Organizations: Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen, The Netherlands · Department of Legal Theory, Jagiellonian University, Poland

Abstract

Bail decisions present a fundamental challenge for data-driven decision support systems. When bail is denied, the counterfactual outcome of whether the defendant would have appeared in court remains unobserved. As a result, historical bail data embed structural label indeterminacy: future decisions are influenced by past decisions whose outcomes are only partially knowable. Building automated systems on such data risks introducing bias and reinforcing feedback loops. This raises a core question for machine-learning systems intended to assist judicial actors: how should cases in which bail was denied be treated during model development? In a case study of bail decisions from the Unified Judicial System of Pennsylvania, we evaluate five contemporary approaches to handling label indeterminacy across three machine learning models, including a novel label imputation method motivated by the dynamics of bail decisions. Each method relies on unverifiable assumptions, yet all influence the models' predictive behaviour, sometimes even more so than the choice of model itself. Explainable AI analysis further reveals that these effects extend to the models' internal decision-making processes as well. Finally, we consider the notion of label indeterminacy from a legal perspective and assess the legitimacy of these approaches in the context of bail decision-making.

Figures & tables

Figure 1. Flowchart of the bail decision process illustrating how missing counterfactuals give rise to indeterminate labels.

	Bail Status
Determinate	Posted
	Forfeited
	Revoked (due to FTA)
Indeterminate	Denied
	Set
	Partial Posting

Table 1. Label indeterminacy in the dataset.

ID	Name	Description	Assumption
$corr$	Correct labels	Includes indeterminate cases without altering their labels.	Assumes defendants who were in pretrial detention would have appeared in court if they were not detained.
$daf$	Detention-as-failure	Includes indeterminate cases and sets their label to ’fail to appear’.	Assumes defendants who were in pretrial detention would not have appeared in court if they were not detained.
$obs$	Observed only	Includes only cases where bail was not denied and the outcome is observable.	Assumes missing at random; fails if the choice for the denial of bail is based on the content of the case.
$obs+ip$	Observed + IP	Trains only on non-denied cases and applies inverse propensity weights to correct for sample bias	Assumes that every case has a non-zero chance of being granted bail (positivity) and that all factors influencing whether bail is granted are captured in the data (no unobserved confounders)
$nn$	Nearest neighbor	Imputes the labels of bail-denied cases using the most similar cases from the bail-granted sample	Assumes there is a valid similarity metric between bail-denied and bail-granted cases.

Table 2. Overview of the label imputation methods used to account for indeterminate labels in the bail domain.

Determinate cases
Method	$corr$	$daf$	$obs$	$obs+ip$	$nn$
Logistic Regression	10.78 $\pm$ 0.44	10.42 $\pm$ 0.23	10.85 $\pm$ 0.21	10.89 $\pm$ 0.22	10.70 $\pm$ 0.20
Random Forest	11.37 $\pm$ 0.31	9.79 $\pm$ 0.26	11.13 $\pm$ 0.17	11.06 $\pm$ 0.21	10.53 $\pm$ 0.27
XGBoost	10.61 $\pm$ 0.56	10.56 $\pm$ 0.30	11.41 $\pm$ 0.33	11.45 $\pm$ 0.31	10.99 $\pm$ 0.32
Indeterminate cases
Logistic Regression	1.25 $\pm$ 0.58	1.00 $\pm$ 0.52	0.24 $\pm$ 0.50	0.60 $\pm$ 0.62	1.10 $\pm$ 0.59

Table 3. Mean MCC and standard deviation for each model across the 25 training sets on determinate and indeterminate cases of the test set. These values should be interpreted with caution: they both reflect performance on only a restricted, unrepresentative subset of the data, and the true labels of the indeterminate cases are considered indeterminate.

Panel of Figure 2 — Figure 2. Violin plots displaying the distribution of the predictions for the three models trained using different label imputation methods.

Panel of Figure 3 — Figure 3. The mean prediction of the XGBoost model for two specific cases using different label imputation methods.

Figure 4. Top 15 most important features of the XGBoost model per label imputation method, ranked by mean normalized gain, averaged across the 25 training subsets.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Logistic Regression
Maximum iterations	2000
Regularization	$\ell_{2}$
Random Forest
Number of trees	500
Maximum tree depth	8
Minimum samples per leaf	50

Appendix

Table 4. Hyperparameters used for each model

Selective Labels

Selective labels studies how to learn and evaluate decision systems when outcomes are observed only for selected people, cases, or actions.

See all 10 papers

Do Judges Behave Like Algorithms?
Aug 11, 2026Riya Manchanda, Eric Chen, Chloe Zhu +3Legal Judgment Prediction Algorithmic Fairness
Duke University · Sungkyunkwan University
Man and machine: artificial intelligence and judicial decision making
Mar 19, 2026Arthur Dyevre, Ahmad ShahvaroughiAI-Assisted Decision Making Human-AI Decision Making
Leuven Centre for Empirical Jurisprudence · Leuven Centre for Empirical Jurisprudence Leuven Laboratory for Experimental Social Psychology
Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions
May 12, 2026Gideon Popoola, John SheppardFeature Attribution Counterfactual Explanations
Gianforte School of Computing Montana State University Bozeman, MT, USA, 59715

Aug 11, 2026J/K move · Enter open · S save

Do Judges Behave Like Algorithms?

Riya Manchanda, Eric Chen, Chloe Zhu, Cynthia Rudin+2

Duke University · Sungkyunkwan University

Legal Judgment Prediction Algorithmic Fairness

Open paper AskPDF arXiv

What if judges already behave like algorithms? As artificial intelligence and algorithms are deployed in many settings, including the judicial system, many have debated whether judges should be allowed to rely on them. Instead, we ask whether judges follow predictable, algorithmic-like rules already. If judges already follow consistent, formula-like rules based on discrete and static factors such as criminal history, age, and charge type, then judicial behavior may be improved. However, if judges rely on individualized information that cannot be identified through court data, then standards-based decision-making may be more challenging to understand or improve. This work explores these questions by studying judicial decision-making in misdemeanor bail hearings in Harris County, Texas. Using available court data, we investigate whether magistrate judges follow what resembles an algorithm; whether they consider the same variables in their decision-making; and whether they are consistent with themselves and with each other. To do this, we train machine learning models for each judge, measure variable importance metrics to determine important variables for each judge's decision-making, and analyze outcomes of similar cases for judges. Our results reveal that these judges generally behave algorithmically: their decisions can be captured by small, interpretable formulas. However, in some cases, judges differ substantially, leading to surprising inconsistency and unequal treatment across similar defendants. Identifying cases where algorithms do not explain judicial decision-making can improve the justice system by focusing attention on decisions where individualized standards, rather than rules, better explains outcomes.

Abstract

Selective Labels

Explore similar work

Do Judges Behave Like Algorithms?

Man and machine: artificial intelligence and judicial decision making

Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions