Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review

Gaurab Chhetri; Anika Baitullah; Subasish Das

cs.AISep 19, 2026

Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review

Authors: Gaurab Chhetri, Anika Baitullah, Subasish Das

Organizations: Department of Computer Science Texas State University, USA · Ingram School of Engineering Texas State University, USA

Abstract

Public crash databases increasingly support automated safety analysis, but crash severity prediction remains difficult to translate into public-sector decision workflows when models are evaluated primarily as ordinary classifiers. This study reframes dementia-related crash severity modeling as a decision-aware triage problem in which a system must classify crashes into no-injury/property-damage-only (O), minor or moderate injury (BC), and fatal or severe injury (KA), while also controlling outcome leakage, reporting severe under-triage, calibrating confidence, and preserving every raw prediction for audit. Using 4,781 Texas crash records with structured fields and police narratives, we evaluate structured, narrative, fusion, calibrated fusion, BERT-family, and local large-language-model baselines under a stratified 70/15/15 split. In the reported split, leakage-controlled Gemma obtains the highest observed macro-F1 (0.545; 95% bootstrap CI [0.507, 0.583]). The best calibrated fusion model obtains macro-F1 of 0.522 and expected calibration error of 0.033. Selective deferral improves performance among cases retained for automatic classification. At 70% coverage, macro-F1 rises to 0.573 and severity cost falls to 0.577, while deferred cases are treated as candidates for a proposed human-review process and are not further evaluated in the present experiment. The study contributes a reproducible, leakage-controlled, and uncertainty-aware evaluation framework for crash AI systems, emphasizing auditability and selective deferral rather than accuracy alone.

Figures & tables

Figure 1: Leakage-controlled multimodal crash triage architecture.

Study	Topic	Method	Key finding
Carr et al. (2000)	Dementia	Crash-record case series	Alzheimer drivers over-represented in angle and lane-change crashes
Meuleners et al. (2016)	Dementia	Linked hospital–crash records, case-control	Dementia raises crash and hospitalization risk
Dzinyela et al. (2023)	Older drivers	Random-parameters logit, heterogeneity in means/variances	At-fault older-driver injury severity driven by speed and roadway type
Li et al. (2024)	Crash narratives	LDA topics + explainable XGBoost (SHAP)	Narrative topics shift predicted crash severity
Arteaga et al. (2020)	Crash narratives	TF-IDF + interpretable ML	Narrative terms predict injury severity with feature attribution
Liu et al. (2025)	Crash narratives	Multimodal deep fusion (text+geometry+records)	Adding text to structured data improves severity prediction

Table 1: Representative dementia, older-driver, and crash-narrative studies for severity analysis.

Figure 2: Annual severity distribution for dementia-related crash records.

Model name	Input	Training or decision design	Primary audit artifact
Majority baseline	None	Predicts the modal training class and anchors the imbalance problem.	Deterministic baseline predictions
Structured logistic regression	Structured fields	One-hot structured features with class-weighted multinomial logistic regression.	Coefficients and class probabilities
Structured CatBoost	Structured fields	Categorical boosting with fixed seed and explicit categorical feature handling.	Fitted booster and probabilities
Narrative logistic regression	Narrative	TF-IDF unigram/bigram text features with class-weighted logistic regression.	Vocabulary, coefficients, probabilities
Early fusion logistic regression	Structured + narrative	One classifier over concatenated one-hot structured features and TF-IDF text.	Joint feature probabilities
Late fusion	Structured + narrative	Validation-tuned probability average of the best structured model and narrative model.	Alpha search and fused probabilities

Table 2: Model families and audit design.

Setting	Model	Acc.	Macro-F1	KA Prec.	KA Rec.	KA Under	Cost	Brier	ECE
Full	Late fusion	0.599	0.539	0.295	0.425	0.575	0.652	0.534	0.112
Full	Calibrated late fusion	0.599	0.539	0.295	0.425	0.575	0.652	0.515	0.039
Full	Early fusion logistic regression	0.579	0.533	0.300	0.493	0.507	0.663	0.524	0.017
Full	Narrative logistic regression	0.588	0.516	0.282	0.329	0.671	0.673	0.535	0.084
Leakage	Gemma	0.578	0.545	0.315	0.630	0.370	0.659	0.607	0.192
Leakage	Early fusion logistic regression	0.577	0.522	0.278	0.438	0.562	0.678	0.526	0.029

Table 3: Main stratified test results.

Figure 3: Leakage-controlled model performance on the stratified test set.

Model	Full F1	Leak F1	F1 drop	Full KA Rec.	Leak KA Rec.	Interpretation
Structured CatBoost	0.452	0.452	0.000	0.562	0.562	Unchanged because the current structured inputs do not include direct severity fields.
Narrative logistic regression	0.516	0.502	0.014	0.329	0.274	Text-only learning loses outcome words and hospital/transport phrases.
Early fusion logistic regression	0.533	0.522	0.011	0.493	0.438	Joint multimodal features retain signal, but KA sensitivity still drops.
Late fusion	0.539	0.522	0.017	0.425	0.384	Separate modality probabilities remain useful, but masking reduces severe-case evidence.

Table 4: Leakage audit for models evaluated in both full and leakage-controlled settings.

Figure 4: Leakage sensitivity for models evaluated in both full and masked settings.

Figure 5: High-support dataset-derived leakage phrases discovered from crash narratives.

Figure 6: Leakage-controlled confusion matrix for Gemma, the strongest macro-F1 model.

Figure 7: Confidence-band audit for calibrated late fusion. Panel A splits test records into five equal-sized confidence bands and shows correct versus incorrect predictions, with annotations giving band accuracy. Panel B uses the same confidence bands to show true KA records split into detected and missed severe cases.

Coverage	Classified	Deferred	Deferred KA	Macro-F1	KA Rec.	KA Under	Cost	Operating interpretation
1.0	718	0	0	0.522	0.384	0.616	0.675	Fully automated classification; no review burden, but all uncertainty remains in the automatic labels.
0.9	646	72	7	0.544	0.379	0.621	0.635	Light review queue; removes the lowest-confidence decile and lowers expected severity cost.
0.8	574	144	16	0.560	0.368	0.632	0.601	Moderate review queue; best balance when an agency can inspect roughly one in five cases.
0.7	503	215	25	0.573	0.396	0.604	0.577	Conservative automation; strongest cost reduction, but requires review of nearly one-third of cases.

Table 5: Selective deferral operating points for calibrated late fusion.

Figure 8: Selective deferral operating points for calibrated late fusion. Panel A shows macro-F1 gain and severity-cost reduction relative to full automation as more low-confidence records are routed to review. Panel B uses the same review levels to show the deferred queue size and the number of true KA records in that queue.

Audit	Observed	Expected?	Interpretation
Early–late fusion label agreement	0.833	Yes	Similar inputs produce similar labels, but boundary probabilities still shift.
Early–late fusion probability distance	0.084	Partly	Late fusion changes uncertainty more than aggregate accuracy.
Late-fusion cost-rule agreement	0.689	No	Cost-sensitive prediction shifts many borderline records.
Calibrated fusion confidence gap	0.622 vs. 0.562	Yes	Correct predictions remain more confident than errors, supporting deferral.
80% coverage deferred KA records	16 of 73	Yes	Deferral captures a meaningful share of severe ambiguous cases.

Table 6: Behavioral audits from saved test predictions.

Figure 9: Predicted probability and confidence distributions for calibrated late fusion. Panel A shows macro-F1 gain and severity-cost reduction among retained automatic predictions as more low-confidence records are withheld from automatic classification. Panel B shows class-specific confidence ECDFs used to construct the selective deferral rule.

Human Oversight of AI

Human oversight of AI studies how people supervise, verify, and control AI decisions, especially as systems become more autonomous and consequential.

See all 184 papers

CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata
Sep 10, 2026Amir Rafe, Subasish DasDistribution Shift Robustness Group-Conditional Conformal Prediction
Civil Engineering, Texas State University, 601 University Drive, San Marcos, 78666 TX, USA
Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment
Mar 23, 2026Sheng Liu, Long Chen, Zeyun Zhao +16Clinical Risk Prediction Clinical Decision-Making
Department of Biomedical Data Science, Stanford University · Department of Radiation Oncology, Stanford University · Center for Data Science, New York University +13
Not All Synthetic Data Are Equal: Expert-Committee Audit Screening for Imbalanced Crash-Injury-Severity Prediction in Automated Driving Systems
Aug 30, 2026Zewei Li, Qiaoqiao Ren, Hang Yang +3Synthetic Data Selection Synthetic Data Evaluation
Faculty of Maritime and Transportation, Ningbo University, Ningbo, China · Centre for Transport Engineering and Modelling, Department of Civil and Environmental Engineering, Imperial College London, London, United Kingdom · Department of Civil Engineering, The University of Hong Kong, Hong Kong, China +1

Sep 10, 2026J/K move · Enter open · S save

CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata

Amir Rafe, Subasish Das

Civil Engineering, Texas State University, 601 University Drive, San Marcos, 78666 TX, USA

Distribution Shift Robustness Group-Conditional Conformal Prediction Conformal Prediction

Open paper AskPDF arXiv

Transportation agencies increasingly predict crash-injury severity with statistical and machine-learning models, but these models do not state how often their output contains the recorded injury level or for which groups of drivers it fails, a gap that matters most for motorcyclists and unrestrained drivers. This study develops and evaluates a certification layer that gives any fitted severity model a finite-sample, distribution-free coverage guarantee within prespecified safety strata. The layer, CHOIR (Conformal Heterogeneity-aware Ordinal Inference with Risk control), combines groupwise and weighted conformal prediction with conformal risk control to return contiguous KABCO intervals, and adds a declared sensitivity analysis for medically assessed injury and bounds on fatal omission. It is evaluated on 4.04 million Texas crashes from 2017-2023, one sampled driver per crash, with seven base models from the ordered logit to a tabular foundation model, and on held-out counties and later years. Under one pooled threshold every model reaches 0.90 coverage overall but covers motorcyclists or unrestrained drivers at 0.868 or lower, and class-balanced gradient boosting covers unrestrained drivers at only 0.374. Calibration within four safety strata places all 28 model-by-stratum estimates between 0.898 and 0.907, at the cost of sets spanning 3.6 to 4.6 of five categories for these groups, and the certified ordered logit is within 0.03 categories of the narrowest model. Injury-model coverage should therefore be certified within safety groups rather than on average, calibration rather than model complexity determines validity, and a statewide threshold should not be applied to small rural counties without local calibration data.

Abstract

Human Oversight of AI

Explore similar work

CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata

Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment

Not All Synthetic Data Are Equal: Expert-Committee Audit Screening for Imbalanced Crash-Injury-Severity Prediction in Automated Driving Systems