MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Neha Sharma; Ritesh Sharma

cs.LGAug 23, 2026

MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Authors: Neha Sharma, Ritesh Sharma

Organizations: Department of Computer Science Virginia Commonwealth University Richmond, Virginia, United States · Department of Electrical and Computer Engineering Virginia Commonwealth University Richmond, Virginia, United States

Abstract

Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.

Figures & tables

Source	$n$	Years	Median	Mean	SD
Kaggle	315	1966–2017	5.0	8.56	8.8
Mother Jones	157	1982–2026	10.0	13.80	12.3
Stanford MSA	334	1966–2016	5.0	7.53	7.8
Gun Violence Archive	6,162	2013–2026	4.0	5.07	2.9
Total (pooled)	6,968	1966–2026	4.0	5.79	4.5

TABLE I: The four sources in MASH-Bench (total_victims

\geq 3

; Las Vegas 2017 excluded from curated sources). SD = standard deviation of total_victims.

Source	Rule VeryHigh rate	Std VeryHigh rate
Kaggle	$2.2\%$ (7/315)	$13.7\%$ (43/315)
Mother Jones	$5.1\%$ (8/157)	$35.7\%$ (56/157)
Stanford MSA	$1.8\%$ (6/334)	$11.1\%$ (37/334)
GVA	$\mathbf{0.03\%}$ (2/6,162)	$\mathbf{0.18\%}$ (11/6,162)

TABLE II: VeryHigh-class base rates per source under the two stratification strategies. Rule uses fixed thresholds; std uses the training-fold

\mu+2\sigma

threshold under LODO.

Source A	Source B	Wasserstein	TV	MMD 2
GVA	Kaggle	7.77	0.69	0.138
GVA	Mother Jones	9.41	0.64	0.135
GVA	Stanford MSA	8.18	0.48	0.135
Kaggle	Mother Jones	5.01	0.42	0.243
Kaggle	Stanford MSA	1.66	0.26	0.169
Mother Jones	Stanford MSA	4.32	0.29	0.153

TABLE III: Distributional shift across the six source pairs. Wasserstein-1 (larger = greater numeric shift), total-variation distance (0 = identical, 1 = disjoint), MMD 2 (larger = greater joint-distribution shift). Bolded values: smallest per metric.

Classifier	Kaggle	Mother Jones	Stanford MSA	GVA
Random Forest	$0.68\pm 0.12$	$0.86\pm 0.18$	$0.84\pm 0.10$	$0.20\pm 0.45$
LightGBM	$0.68\pm 0.12$	$0.78\pm 0.21$	$0.89\pm 0.10$	$0.20\pm 0.45$
XGBoost	$0.69\pm 0.13$	$0.78\pm 0.22$	$0.88\pm 0.09$	$0.20\pm 0.45$

TABLE IV: Best observed LODO VeryHigh recall per (classifier, test source): the highest mean over 5 seeds across the evaluated (strategy, feature-set, threshold-mode) configurations. These maxima are descriptive and are not unbiased estimates of a prespecified deployment configuration.

Fig. 1: Per-source LODO VeryHigh recall for the best configuration of each modern classifier. All three collapse on GVA to $0.20\pm 0.45$ (precision $0.0004$ ) despite achieving 0.68–0.89 on the curated sources. Dashed line at 0.5.

Method	Kaggle	Mother Jones	Stanford MSA	GVA	Mean
Baseline XGB	0.79	0.95	0.84	0.00	0.65
CORAL	$-0.49$	$-0.68$	$-0.49$	$\phantom{+}0.00$	0.00
IW	$-0.33$	$-0.42$	$-0.41$	$\phantom{+}0.00$	0.13
DANN	+0.01	$-0.22$	$-0.02$	+0.28	0.43

TABLE V: Per-source mean VeryHigh recall for baseline XGBoost and three domain-adaptation methods, averaged across the DA experimental grid (strategy, features, seeds). Deltas (

\Delta

) are per-source change relative to baseline. Bold: only positive per-source delta on any source.

Fig. 2: Best observed VeryHigh recall per (test source, method) across the DA configurations; Table V reports the corresponding grid-averaged values. DANN is the only domain-adaptation method that produces any non-zero recall on GVA; CORAL and IW hurt every source with informative features. Baseline is XGBoost without adaptation.

Source	Baseline recall	Masked recall
Kaggle	$0.544\pm 0.035$	$0.000\pm 0.000$
Mother Jones	$0.643\pm 0.013$	$0.000\pm 0.000$
Stanford MSA	$0.795\pm 0.024$	$0.000\pm 0.000$
GVA	$0.000\pm 0.000$	$0.745\pm 0.297$ †

TABLE VI: Feature-masking ablation under the best baseline LODO configuration (XGBoost, std strategy, full features); mean VeryHigh recall

\pm

std over 5 seeds.

	Baseline XGB		Prior-shift
Source	Recall	Precision	Recall	Precision
Kaggle	$0.342$	$0.251$	$0.253$	$\mathbf{0.373}$
Mother Jones	$0.557$	$0.269$	$0.171$	$0.278$
Stanford MSA	$0.160$	$0.172$	$0.040$	$0.079$
GVA	$\mathbf{0.000}$	$\mathbf{0.000}$	$\mathbf{0.000}$	$\mathbf{0.000}$

TABLE VII: VeryHigh-class performance before and after oracle prior-shift recalibration; mean over 5 seeds and the same configuration grid used for the DA experiments.

Group	$n$	$n_{pos}$	Recall	Precision	TP / FP / FN
Yes	96	25	0.88	0.34	22 / 43 / 3
Unknown	145	5	0.60	0.08	3 / 34 / 2
No	93	7	0.43	0.08	3 / 37 / 4

TABLE VIII: Per-group VeryHigh recall and precision on Stanford MSA for the best modern LODO configuration (XGBoost + std + ctx + youden_val). “Yes” incidents receive substantially higher predicted-risk rates, but with precision collapse.

Fig. 3: Per-group VeryHigh recall by mental_health label across the three curated sources. Incidents flagged Yes on Stanford MSA and Kaggle receive substantially higher recall than No (indicating over-prediction); Mother Jones shows near-uniform high recall across groups. Stanford MSA lacks an Unclear category (only Yes/No/Unknown are recorded in its schema). Precision values are reported in Table IX . GVA is omitted (all participant fields are Unknown, precluding group-wise audit).

Source	Group	Recall (95% CI)	Precision (95% CI)
Stanford MSA	Yes	$0.88$ $[0.69,0.97]$	$0.34$ $[0.23,0.47]$
Stanford MSA	No	$0.43$ $[0.10,0.82]$	$0.07$ $[0.02,0.20]$
Stanford MSA	Unknown	$0.60$ $[0.15,0.95]$	$0.08$ $[0.02,0.22]$
Kaggle	Yes	$0.89$ $[0.72,0.98]$	$0.31$ $[0.21,0.42]$
Kaggle	No	$0.86$ $[0.42,1.00]$	$0.11$ $[0.04,0.22]$
Kaggle	Unknown	$1.00$ $[0.48,1.00]$	$0.17$ $[0.06,0.35]$

TABLE IX: Wilson

95\%

CIs on per-group VeryHigh recall and precision ( mental_health attribute).

Cross-Dataset Generalization

Cross-dataset generalization studies whether AI models retain accuracy, calibration, and usefulness on data from sources beyond their training set.

See all 142 papers

Jul 16, 2026

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

Most medical AI benchmarks measure whether a model knows the correct answer. MedFailBench asks a different question: which safety boundary failed? We present a clinician-built synthetic benchmark and failure atlas that labels medical AI errors by severity (1--5) and safety gate type (missed urgent escalation, unsafe remote dosing, unsafe discharge reassurance, evidence fabrication, unsafe protocol execution, source support gap). The current public release (v0.2.1) contains 44 clinician-reviewed synthetic cases with severity annotations, a live HuggingFace leaderboard preview, a safety gate taxonomy, a clinical severity rubric, and an automated pipeline for archiving model-response screening runs. No patient data, clinical validation claims, or model rankings are included. MedFailBench is released under Apache-2.0 and CC-BY-4.0 and carries the Zenodo DOI 10.5281/zenodo.21205535.

Goktug Ozkan

Department of Internal Medicine, Kutahya Emet Dr. Fazil Dogan State Hospital, Kutahya, Turkiye

Synthetic Benchmark Generation AI Safety Evaluation

Ask PDF

Oct 3, 2026

EasyClassifier: Honest, Reproducible Machine-Learning Classification for Researchers Who Do Not Program

Machine-learning classification is now used across medicine, the social sciences, economics, education and engineering, very often by researchers who do not program. Two errors recur in such work: preprocessing learned on all rows before cross-validation (data leakage), and reporting the cross-validation score of the classifier that won a comparison (selection bias). Both make published scores optimistic. We present EasyClassifier, an open-source Python package that guides a user from a CSV or Excel file to a finished analysis through a sequence of plain-language, multiple-choice questions, with no code. It makes the correct procedure the default rather than an option: every preprocessing step is learned inside each training fold; the selected classifier receives a separate final score from nested cross-validation (up to 2,000 rows), or an untouched 20% test set; classifiers run with fixed default parameters and fixed random seeds; and every run writes a report with a ready-to-adapt Methods paragraph, the references to cite, and figures prepared for publication. On ten public datasets from seven fields and a random-label control, split in half so that one half served as an untouched external test, the common practice overstated balanced accuracy by 2.9 percentage points on average (up to 9.9 on small data), whereas the score EasyClassifier reports had a mean signed bias of +0.7 points and a mean absolute error of 2.3 points (one-sided Wilcoxon p = 3.2 x 10^{-4}, 30 runs). On random labels it reported 48.8% balanced accuracy against a chance level of 50%. EasyClassifier is available under the MIT license from PyPI (pip install easyclassifier), GitHub, and Zenodo.

Ahmad B. A. Hassanat, Ghada A. Altarawneh

Faculty of Information Technology, Mutah University, Karak, Jordan · Department of Accounting, Mutah University, Karak, Jordan

ML Reproducibility Classification Data Leakage Tabular ML Cross-Validation Selection Bias

Ask PDF

Apr 16, 2026

MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE, a living MLTC benchmark derived from {m}edical device {ad}verse {e}vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark.

Raunak Agarwal, Markus Wenzel, Simon Baur +3

Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute Berlin, 10587, Germany

Multi-Label Classification Healthcare Benchmark Design LLM Uncertainty Estimation

Ask PDF

Abstract

Cross-Dataset Generalization

Explore similar work

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

EasyClassifier: Honest, Reproducible Machine-Learning Classification for Researchers Who Do Not Program

MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events