MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Neha Sharma; Ritesh Sharma

cs.LGAug 23, 2026

MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Authors: Neha Sharma, Ritesh Sharma

Organizations: Department of Computer Science Virginia Commonwealth University Richmond, Virginia, United States · Department of Electrical and Computer Engineering Virginia Commonwealth University Richmond, Virginia, United States

Abstract

Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.

Figures & tables

Source	$n$	Years	Median	Mean	SD
Kaggle	315	1966–2017	5.0	8.56	8.8
Mother Jones	157	1982–2026	10.0	13.80	12.3
Stanford MSA	334	1966–2016	5.0	7.53	7.8
Gun Violence Archive	6,162	2013–2026	4.0	5.07	2.9
Total (pooled)	6,968	1966–2026	4.0	5.79	4.5

TABLE I: The four sources in MASH-Bench (total_victims

\geq 3

; Las Vegas 2017 excluded from curated sources). SD = standard deviation of total_victims.

Source	Rule VeryHigh rate	Std VeryHigh rate
Kaggle	$2.2\%$ (7/315)	$13.7\%$ (43/315)
Mother Jones	$5.1\%$ (8/157)	$35.7\%$ (56/157)
Stanford MSA	$1.8\%$ (6/334)	$11.1\%$ (37/334)
GVA	$\mathbf{0.03\%}$ (2/6,162)	$\mathbf{0.18\%}$ (11/6,162)

TABLE II: VeryHigh-class base rates per source under the two stratification strategies. Rule uses fixed thresholds; std uses the training-fold

\mu+2\sigma

threshold under LODO.

Source A	Source B	Wasserstein	TV	MMD 2
GVA	Kaggle	7.77	0.69	0.138
GVA	Mother Jones	9.41	0.64	0.135
GVA	Stanford MSA	8.18	0.48	0.135
Kaggle	Mother Jones	5.01	0.42	0.243
Kaggle	Stanford MSA	1.66	0.26	0.169
Mother Jones	Stanford MSA	4.32	0.29	0.153

TABLE III: Distributional shift across the six source pairs. Wasserstein-1 (larger = greater numeric shift), total-variation distance (0 = identical, 1 = disjoint), MMD 2 (larger = greater joint-distribution shift). Bolded values: smallest per metric.

Classifier	Kaggle	Mother Jones	Stanford MSA	GVA
Random Forest	$0.68\pm 0.12$	$0.86\pm 0.18$	$0.84\pm 0.10$	$0.20\pm 0.45$
LightGBM	$0.68\pm 0.12$	$0.78\pm 0.21$	$0.89\pm 0.10$	$0.20\pm 0.45$
XGBoost	$0.69\pm 0.13$	$0.78\pm 0.22$	$0.88\pm 0.09$	$0.20\pm 0.45$

TABLE IV: Best observed LODO VeryHigh recall per (classifier, test source): the highest mean over 5 seeds across the evaluated (strategy, feature-set, threshold-mode) configurations. These maxima are descriptive and are not unbiased estimates of a prespecified deployment configuration.

Fig. 1: Per-source LODO VeryHigh recall for the best configuration of each modern classifier. All three collapse on GVA to $0.20\pm 0.45$ (precision $0.0004$ ) despite achieving 0.68–0.89 on the curated sources. Dashed line at 0.5.

Method	Kaggle	Mother Jones	Stanford MSA	GVA	Mean
Baseline XGB	0.79	0.95	0.84	0.00	0.65
CORAL	$-0.49$	$-0.68$	$-0.49$	$\phantom{+}0.00$	0.00
IW	$-0.33$	$-0.42$	$-0.41$	$\phantom{+}0.00$	0.13
DANN	+0.01	$-0.22$	$-0.02$	+0.28	0.43

TABLE V: Per-source mean VeryHigh recall for baseline XGBoost and three domain-adaptation methods, averaged across the DA experimental grid (strategy, features, seeds). Deltas (

\Delta

) are per-source change relative to baseline. Bold: only positive per-source delta on any source.

Fig. 2: Best observed VeryHigh recall per (test source, method) across the DA configurations; Table V reports the corresponding grid-averaged values. DANN is the only domain-adaptation method that produces any non-zero recall on GVA; CORAL and IW hurt every source with informative features. Baseline is XGBoost without adaptation.

Source	Baseline recall	Masked recall
Kaggle	$0.544\pm 0.035$	$0.000\pm 0.000$
Mother Jones	$0.643\pm 0.013$	$0.000\pm 0.000$
Stanford MSA	$0.795\pm 0.024$	$0.000\pm 0.000$
GVA	$0.000\pm 0.000$	$0.745\pm 0.297$ †

TABLE VI: Feature-masking ablation under the best baseline LODO configuration (XGBoost, std strategy, full features); mean VeryHigh recall

\pm

std over 5 seeds.

	Baseline XGB		Prior-shift
Source	Recall	Precision	Recall	Precision
Kaggle	$0.342$	$0.251$	$0.253$	$\mathbf{0.373}$
Mother Jones	$0.557$	$0.269$	$0.171$	$0.278$
Stanford MSA	$0.160$	$0.172$	$0.040$	$0.079$
GVA	$\mathbf{0.000}$	$\mathbf{0.000}$	$\mathbf{0.000}$	$\mathbf{0.000}$

TABLE VII: VeryHigh-class performance before and after oracle prior-shift recalibration; mean over 5 seeds and the same configuration grid used for the DA experiments.

Group	$n$	$n_{pos}$	Recall	Precision	TP / FP / FN
Yes	96	25	0.88	0.34	22 / 43 / 3
Unknown	145	5	0.60	0.08	3 / 34 / 2
No	93	7	0.43	0.08	3 / 37 / 4

TABLE VIII: Per-group VeryHigh recall and precision on Stanford MSA for the best modern LODO configuration (XGBoost + std + ctx + youden_val). “Yes” incidents receive substantially higher predicted-risk rates, but with precision collapse.

Fig. 3: Per-group VeryHigh recall by mental_health label across the three curated sources. Incidents flagged Yes on Stanford MSA and Kaggle receive substantially higher recall than No (indicating over-prediction); Mother Jones shows near-uniform high recall across groups. Stanford MSA lacks an Unclear category (only Yes/No/Unknown are recorded in its schema). Precision values are reported in Table IX . GVA is omitted (all participant fields are Unknown, precluding group-wise audit).

Source	Group	Recall (95% CI)	Precision (95% CI)
Stanford MSA	Yes	$0.88$ $[0.69,0.97]$	$0.34$ $[0.23,0.47]$
Stanford MSA	No	$0.43$ $[0.10,0.82]$	$0.07$ $[0.02,0.20]$
Stanford MSA	Unknown	$0.60$ $[0.15,0.95]$	$0.08$ $[0.02,0.22]$
Kaggle	Yes	$0.89$ $[0.72,0.98]$	$0.31$ $[0.21,0.42]$
Kaggle	No	$0.86$ $[0.42,1.00]$	$0.11$ $[0.04,0.22]$
Kaggle	Unknown	$1.00$ $[0.48,1.00]$	$0.17$ $[0.06,0.35]$

TABLE IX: Wilson

95\%

CIs on per-group VeryHigh recall and precision ( mental_health attribute).

Cross-Dataset Generalization

Cross-dataset generalization studies whether AI models retain accuracy, calibration, and usefulness on data from sources beyond their training set.

See all 142 papers

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
Jul 16, 2026Goktug OzkanSynthetic Benchmark Generation AI Safety Evaluation
Department of Internal Medicine, Kutahya Emet Dr. Fazil Dogan State Hospital, Kutahya, Turkiye
EasyClassifier: Honest, Reproducible Machine-Learning Classification for Researchers Who Do Not Program
Oct 3, 2026Ahmad B. A. Hassanat, Ghada A. AltarawnehML Reproducibility Classification
Faculty of Information Technology, Mutah University, Karak, Jordan · Department of Accounting, Mutah University, Karak, Jordan
MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events
Apr 16, 2026Raunak Agarwal, Markus Wenzel, Simon Baur +3Multi-Label Classification Healthcare
Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute Berlin, 10587, Germany

Jul 16, 2026J/K move · Enter open · S save

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

Goktug Ozkan

Department of Internal Medicine, Kutahya Emet Dr. Fazil Dogan State Hospital, Kutahya, Turkiye

Synthetic Benchmark Generation AI Safety Evaluation

Open paper AskPDF arXiv

Most medical AI benchmarks measure whether a model knows the correct answer. MedFailBench asks a different question: which safety boundary failed? We present a clinician-built synthetic benchmark and failure atlas that labels medical AI errors by severity (1--5) and safety gate type (missed urgent escalation, unsafe remote dosing, unsafe discharge reassurance, evidence fabrication, unsafe protocol execution, source support gap). The current public release (v0.2.1) contains 44 clinician-reviewed synthetic cases with severity annotations, a live HuggingFace leaderboard preview, a safety gate taxonomy, a clinical severity rubric, and an automated pipeline for archiving model-response screening runs. No patient data, clinical validation claims, or model rankings are included. MedFailBench is released under Apache-2.0 and CC-BY-4.0 and carries the Zenodo DOI 10.5281/zenodo.21205535.

Abstract

Cross-Dataset Generalization

Explore similar work

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

EasyClassifier: Honest, Reproducible Machine-Learning Classification for Researchers Who Do Not Program

MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events