Leakage-Controlled Structure-Preserving Augmentation and Template-Aware Evaluation for Bangla Symptom-Based Disease Prediction

Sabah Nushra; Shovan Samanta Turzo; Moinul Hossain; Md Kishor Morol

cs.LGOct 30, 2026

Leakage-Controlled Structure-Preserving Augmentation and Template-Aware Evaluation for Bangla Symptom-Based Disease Prediction

Authors: Sabah Nushra, Shovan Samanta Turzo, Moinul Hossain, Md Kishor Morol

Organizations: Islamic University of Technology, Gazipur, Bangladesh · Daffodil International University, Dhaka, Bangladesh · American International University–Bangladesh, Dhaka, Bangladesh · ELITE Research Lab, New York, USA

Abstract

Bangla symptom-based disease prediction is commonly evaluated on a small, imbalanced structured benchmark containing many profiles that differ by only one or two symptoms. We test whether train-only asymmetric Bernoulli perturbation improves classification using 725 unique profiles spanning 85 diseases and 172 binary symptoms. Experiments use ten repeated real-profile holdouts and parent-grouped inner cross-validation. The 2 x 2 design separates augmentation from hyperparameter tuning, with exact duplication, random oversampling, class weighting, and SMOTEN as balancing controls. For Bernoulli Naive Bayes (BNB), the gain over tuned raw training does not withstand Holm correction, although BNB attains the highest mean accuracy (97.50%). The boosted-tree models benefit substantially: XGBoost, LightGBM, and CatBoost improve by 18.10, 28.45, and 10.09 percentage points, respectively. A Hamming-distance audit places 91.0% of standard held-out profiles within one symptom bit of the training data. Accuracy decreases on a same-disease 1-bit-purged challenge; nevertheless, XGBoost and LightGBM retain large augmentation gains. We also evaluate robustness to missing and spurious symptoms and the faithfulness of BNB explanations. Overall, Bernoulli perturbation is useful for some model families, but benchmark accuracy should be interpreted together with template proximity.

Figures & tables

Panel of Figure 1 — Fig. 1: Experimental pipeline. Real profiles are deduplicated and split before any augmentation. Parent-grouped cross-validation, augmentation, and model fitting use only the training partition. The reserved real test partition bypasses model development and is used only once, for final evaluation.

Model	Raw U	Raw T	Bern U	Bern T	F1 R/B	$\Delta$ pp	Holm- $p$
BNB	60.78	96.12 $\pm$ 1.36	97.24	97.50 $\pm$ 0.49	93.4/95.7	+1.38	.1406
LR	93.10	96.72 $\pm$ 1.27	97.24	97.41 $\pm$ 0.91	94.5/95.5	+0.69	.5625
MLP	96.29	95.95 $\pm$ 1.00	97.07	97.16 $\pm$ 1.08	93.2/95.3	+1.21	.3750
Ens-Hard	94.40	96.38 $\pm$ 1.27	97.33	97.16 $\pm$ 1.00	93.8/95.2	+0.78	.5469
Ens-Weighted	96.47	96.64 $\pm$ 1.25	97.16	97.16 $\pm$ 0.91	94.4/95.2	+0.52	1.0000
Ens-Soft	96.03	96.47 $\pm$ 1.31	97.16	97.16 $\pm$ 0.91	94.1/95.2	+0.69	.7031

TABLE I: Controlled

2\times 2

experiment over ten repeated real-profile holdouts. Accuracy and macro-F1 are percentages; tuned accuracy columns also report SD. U/T denote untuned/tuned, and Bern denotes Bernoulli augmentation. “F1 R/B” is raw-tuned/Bernoulli-tuned macro-F1.

\Delta

and Holm-

p

compare Bernoulli-tuned with raw-tuned accuracy. Deltas are computed before rounding, so subtraction of displayed means can differ by 0.01 point.

Model	Raw	Bernoulli	$\Delta$ pp	Holm- $p$
BNB	93.63 $\pm$ 2.12	94.51 $\pm$ 1.17	+0.88	.2500
LR	92.48 $\pm$ 1.57	93.19 $\pm$ 1.32	+0.71	.2500
XGBoost	71.24 $\pm$ 1.88	87.26 $\pm$ 2.58	+16.02	.0078
LightGBM	56.81 $\pm$ 3.00	87.35 $\pm$ 2.09	+30.53	.0078

TABLE II: Same-disease 1-bit-purged challenge over ten seeds. Accuracy is given as mean

\pm

SD (%).

\Delta

compares Bernoulli-tuned and raw-tuned training on the same purged split; Holm-

p

refers to accuracy.

Fig. 2: Hamming distance to the nearest training profile for the 1,160 standard held-out evaluation instances.

Fig. 3: Bernoulli-tuned accuracy on the standard split and the same-disease 1-bit-purged challenge. Bars show mean accuracy over ten seeds, error bars denote

\pm

1 SD, and the labels above the bars report the corresponding mean values.

Model	Raw	Exact dup.	Removal	Addition	Full
BNB	96.12	95.26	97.67	93.71	97.67
LR	96.72	95.00	97.41	95.43	97.33
XGBoost	77.41	68.28	95.43	71.81	95.78
LightGBM	66.81	68.36	71.72	51.72	24.57
CatBoost	85.78	90.17	96.21	91.12	96.12

TABLE III: Mechanism ablation: mean accuracy (%) over ten seeds, with raw-tuned hyperparameters held fixed for all variants.

Fig. 4: Robustness of Bernoulli-tuned models to missing symptoms on seed 42. Curves give mean accuracy over 50 independent corruption draws, and bands give 95% empirical intervals across the draws.

Fig. 5: Robustness of Bernoulli-tuned models to spurious symptoms on seed 42, using the same 50-draw protocol. Bands give 95% empirical intervals across corruption draws.

$k$	Del. top	Del. rand.	Ins. top	Ins. rand.
1	.102	.065	.268	.193
2	.335	.201	.701	.634
3	.592	.431	.882	.862
5	.908	.784	.989	.961

TABLE IV: Faithfulness of BNB explanations on held-out profiles (seed 42). All values are means; Del. and Ins. denote deletion and insertion, and rand. denotes random selection.

Axis	Value	BNB	LR
$p$	.80	.971	.968
	.85	.974	.970
	.90	.976	.972
	.95	.972	.957
$q$	0	.974	.969
	.0025	.975	.971

TABLE V: Sensitivity analysis varying one factor at a time. Entries are mean inner-cross-validation macro-F1 over three seeds.

Clinical Prediction

Clinical prediction uses patient records, signals, and images to estimate diagnoses, risks, outcomes, and treatment responses for care.

See all 213 papers

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis
Oct 24, 2026Jian Yang, Peilin Liu, Xiaohan Li +4Clinical Language Model Evaluation LLM Auditing
The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.
Is a 9B Decision Foundation Model Needed for Symptom-to-Disease Classification? Fine-Tuning Clef-Flash Against TF-IDF Baselines
Oct 10, 2026S. M. Mahbub Uddin AhmedMedical Diagnosis Clinical Decision-Making
Department of Computer Science and Engineering, United International University
Quantifying Symptom Causality in Clinical Decision Making: An Exploration Using CausaLM
Mar 25, 2025Alexander Romanus, Mehul Shetty, Connor JordanClinical Decision-Making Causal Interpretability

Oct 24, 2026J/K move · Enter open · S save

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

Jian Yang, Peilin Liu, Xiaohan Li, Songda Li+3

The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.

Clinical Language Model Evaluation LLM Auditing Data Contamination in Language Models Rare Disease Diagnosis

Open paper AskPDF arXiv

Augmented large language model (LLM) systems have reported high benchmark accuracies in rare disease diagnosis, yet their real-world clinical utility remains unproven. A recent meta-analysis flagged augmented LLM diagnostic evaluations as having a high risk of bias due to suspected data leakage; however, the precise impact of these vulnerabilities on model inference has yet to be fully quantified. In this study, we present a comprehensive empirical audit to quantify evaluation artifacts across prevailing AI paradigms, using a repository of over 60,000 cases from diverse public and independent clinical cohorts. By reconstructing a state-of-the-art agentic framework, we demonstrate that candidate-space contamination artificially inflates performance: enforcing strict cross-dataset isolation resulted in a marked performance decline, with the cross-dataset mean Recall@1 decreasing from 54.3% (under conventional one-vs-rest evaluation) to 40.3% (Hedges' g = 0.94). Similar spatial overfitting in supervised classifiers caused diagnostic accuracy to drop from an internal mean of 42.9% to 16.7% across external cohorts (Cohen's d = 2.01). Furthermore, we quantify how LLMs infer diagnoses by exploiting retrospective therapeutic cues in clinical narratives, and how temporal misalignment in foundational knowledge bases leads to performance fluctuations. We conclude that prevailing evaluation protocols for augmented LLM systems inadvertently reflect data leakage rather than genuine medical reasoning. To address these systematic artifacts, we propose a rigorous evaluation framework - coupling strict data-isolation protocols with standardized benchmarking safeguards - to ensure future diagnostic AI for rare diseases is evaluated on actual clinical robustness.

Abstract

Clinical Prediction

Explore similar work

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

Is a 9B Decision Foundation Model Needed for Symptom-to-Disease Classification? Fine-Tuning Clef-Flash Against TF-IDF Baselines

Quantifying Symptom Causality in Clinical Decision Making: An Exploration Using CausaLM