cs.AIOct 1, 2026
SaveA Comparative Explainability Framework for DeBERTa-v3 in Zero-Shot Medical Abstract Classification
Organizations: Department of Technologies and Information Systems, Universidad de Castilla-La Mancha, Avenida Carlos III, s/n, Toledo, 45071, Spain · University of Murcia, Avenida Teniente Flomesta, 5, Murcia, 3003, Spain · Murcian Bio-Health Institute (IMIB-Arrixaca), Pabellón Docente del Hospital Clínico Universitario Virgen de la Arrixaca, Murcia, 3120, Spain
Abstract
A comparative explainability framework is presented to audit DeBERTa-v3 under zero-shot classification of medical abstracts. The work addresses the disagreement problem in Explainable Artificial Intelligence, where different attribution methods produce divergent explanations for the same input and prediction. A natural language inference engine is implemented over the Medical Abstracts corpus with five enriched hypotheses per diagnostic category and a balanced sample of one thousand texts per class. Five explanation methods are compared: SHAP and LIME as model-agnostic approaches, occlusion and Input x Gradient as deep-learning-specific approaches, and Attention x Gradient as a transformer-specific approach. Explanations are standardized through top-token attribution, and pairwise agreement is quantified using the Jaccard index. High predictive accuracy is achieved across well-defined clinical domains, whereas performance degrades under high semantic ambiguity. Explanatory stability directly mirrors predictive certainty, exhibiting strong convergence in univalent categories and a marked drop under diagnostic uncertainty. Furthermore, qualitative error auditing uncovers three systemic failure mechanisms: lexical hypersensitivity, semantic overlap, and loss of attribution coherence. The results support the combined use of several explanation methods and quantitative agreement metrics when auditing transformer-based models in medical text classification, and suggest prioritizing specific clinical ontologies over broad diagnostic labels.
Figures & tables
| Framework / toolkit | Primary purpose | Method families | DeBERTa-v3 | Eval metrics |
| Quantus ( Hedström and others, 2023 ) | Evaluation of explanations | Agnostic backends | Partial | Yes |
| BEExAI ( Sithakoul et al., 2024 ) | Benchmark of XAI quality | Agnostic backends | Partial | Yes |
| Captum ( Kokhlikyan and others, 2020 ) | PyTorch attribution API | Gradient, occlusion, LIME, SHAP | Partial | No |
| Inseq ( Sarti et al., 2023 ) | Sequence-model interpretability | Gradient, attention, perturbation | Partial | Basic |
| Ecco ( Alammar, 2021 ) | Visualization of transformers | Attention, gradient | No | No |
| BertViz ( Vig, 2019 ) | Attention visualization | Attention | No | No |
Table 1: Comparison of XAI frameworks and toolkits relevant to transformer attribution. Methods : coverage of attribution families. DeBERTa-v3 : native support for disentangled attention. Eval metrics : built-in quantitative evaluation.
Figure 1: Methodology overview. High-level workflow of the five sequential operational stages: corpus balancing, zero-shot NLI inference, XAI methods, Top-20 Jaccard consistency analysis, and SHAP-based error analysis.
| Category | Train | Test | Total |
| Neoplasms | 2,530 | 633 | 3,163 |
| Digestive | 1,195 | 299 | 1,494 |
| Nervous | 1,540 | 385 | 1,925 |
| Cardiovascular | 2,441 | 610 | 3,051 |
| General | 3,844 | 961 | 4,805 |
| Total | 11,550 | 2,888 | 14,438 |
Table 2: Original distribution of the Medical Abstracts corpus by clinical category.
| Category | Selected hypotheses |
| Neoplasms | Description of a neoplastic condition such as tumor, cancer, malignancy, carcinoma, sarcoma, lymphoma, leukemic process, or metastatic disease; abnormal cell growth consistent with an oncological process; diagnosis, progression, treatment, staging, or complications of a neoplastic disease; oncological process affecting tissues or organs, including primary or metastatic tumors; pathology related to cancer or tumor biology. |
| Digestive | Disorder of the digestive system affecting the gastrointestinal tract, liver, pancreas, or biliary system; gastrointestinal symptoms, inflammation, infection, obstruction, hepatobiliary disease, or pancreatic pathology; diseases such as gastritis, colitis, pancreatitis, hepatitis, cirrhosis, intestinal dysfunction, or gastrointestinal bleeding; gastroenterological pathology involving digestive organs; digestive disease affecting esophagus, stomach, intestines, colon, liver, or pancreas. |
| Nervous | Neurological disorder affecting the brain, spinal cord, or peripheral nerves; stroke, neuropathy, seizures, demyelination, meningitis, encephalopathy, or neurodegenerative diseases; neurological deficits, altered consciousness, seizures, focal findings, or neuroinflammatory pathology; pathology affecting neural tissue or circuits; diseases of the central or peripheral nervous system. |
| Cardiovascular | Cardiovascular disorder affecting the heart or blood vessels; myocardial infarction, heart failure, arrhythmia, hypertension, ischemia, or vascular pathology; cardiological symptoms, hemodynamic changes, perfusion abnormalities, or vascular obstruction; diseases of the heart muscle, coronary arteries, valves, or systemic circulation; cardiovascular pathology affecting cardiac or vascular structures. |
| General | Systemic or generalized medical condition affecting multiple organs; generalized pathological state with symptoms such as fever, malaise, fatigue, weakness, or systemic inflammation; disorder affecting the body globally rather than a single organ system; diffuse or nonspecific organ anomaly involving general physiological dysfunction; systemic medical problem without a clearly dominant organ system. |
Table 3: Enriched hypotheses per clinical category used by the natural language inference engine.
Figure 2: Confusion matrices for DeBERTa-v3 over the balanced sample.Left: counts. Right: normalized percentages. NEO: neoplasms; DIG: digestive; NER: nervous; CAR: cardiovascular; GEN: general.
Figure 3: Architecture for the comparison of explanation methods. Outputs of the five methods are standardized through Top-20 token extraction and compared with the Jaccard index.
| Category | Summary of the abstract |
| Neoplasms | Aggressive facial melanoma with neural invasion, successfully treated by surgical resection, radiotherapy, and reconstructive muscle flap. |
| Digestive | Optimized hepatic surgical technique using light compression bands to minimize tissue damage and hypoxia during vascular occlusion. |
| Nervous | Plasticity of the central nervous system in children (3–6 years) to coordinate oropharyngeal musculature after corrective speech surgery. |
| Cardiovascular | Long-term safety study of verapamil, highlighting its efficacy in blood pressure control and its positive effect on HDL cholesterol without altering the metabolic profile. |
| General pathological | Reinterpretation of mandibular inflammation as a muscular dysfunction caused by mechanical overload rather than infection, validated by improvement with relaxation techniques. |
Table 4: Correctly classified cases used in the explainability comparison. Each abstract is summarized by its dominant clinical content; the full texts are available in the released experimental configuration.
| Comparison | Jaccard |
| SHAP vs. LIME | 0.333 |
| SHAP vs. IxG | 0.300 |
| SHAP vs. Occlusion | 0.500 |
| SHAP vs. AxG | 0.333 |
| LIME vs. IxG | 0.185 |
| LIME vs. Occlusion | 0.200 |
Table 5: Pairwise Jaccard values for a correctly classified neoplasm case.
| Comparison | Jaccard |
| SHAP vs. LIME | 0.352 |
| SHAP vs. IxG | 0.200 |
| SHAP vs. Occlusion | 0.111 |
| SHAP vs. AxG | 0.157 |
| LIME vs. IxG | 0.318 |
| LIME vs. Occlusion | 0.315 |
Table 6: Pairwise Jaccard values for a correctly classified digestive case.
| Comparison | Jaccard |
| SHAP vs. LIME | 0.200 |
| SHAP vs. IxG | 0.368 |
| SHAP vs. Occlusion | 0.352 |
| SHAP vs. AxG | 0.111 |
| LIME vs. IxG | 0.333 |
| LIME vs. Occlusion | 0.380 |
Table 7: Pairwise Jaccard values for a correctly classified nervous system case.
| Comparison | Jaccard |
| SHAP vs. LIME | 0.529 |
| SHAP vs. IxG | 0.217 |
| SHAP vs. Occlusion | 0.250 |
| SHAP vs. AxG | 0.150 |
| LIME vs. IxG | 0.307 |
| LIME vs. Occlusion | 0.238 |
Table 8: Pairwise Jaccard values for a correctly classified cardiovascular case.
| Comparison | Jaccard |
| SHAP vs. LIME | 0.136 |
| SHAP vs. IxG | 0.227 |
| SHAP vs. Occlusion | 0.059 |
| SHAP vs. AxG | 0.188 |
| LIME vs. IxG | 0.267 |
| LIME vs. Occlusion | 0.036 |
Table 9: Pairwise Jaccard values for a general pathological case. No token is shared by all five methods.
| Comparison | NEO | DIG | NER | CAR | GEN | Mean SD |
| SHAP vs. LIME | 0.333 | 0.352 | 0.200 | 0.529 | 0.136 | 0.31 0.14 |
| SHAP vs. IxG | 0.300 | 0.200 | 0.368 | 0.217 | 0.227 | 0.26 0.06 |
| SHAP vs. Occlusion | 0.500 | 0.111 | 0.352 | 0.250 | 0.059 | 0.25 0.16 |
| SHAP vs. AxG | 0.333 | 0.157 | 0.111 | 0.150 | 0.188 | 0.19 0.08 |
| LIME vs. IxG | 0.185 | 0.318 | 0.333 | 0.307 | 0.267 | 0.28 0.05 |
| LIME vs. Occlusion | 0.200 | 0.315 | 0.380 | 0.238 | 0.036 | 0.23 0.12 |
Table 10: Pairwise Jaccard index per category and method pair. NEO: neoplasms; DIG: digestive; NER: nervous; CAR: cardiovascular; GEN: general.
Figure 4: Architecture for the error analysis protocol of the general pathological category. SHAP attributions are visualized through saliency maps and waterfall plots.
| Case | Predicted | Top driving tokens | Failure pattern |
| 1 | Cardiovascular | hemodynamic , atherosclerosis , glomerular | SO |
| 2 | Neoplasms | malignancy , tumor , carcinoma | LH |
| 3 | Cardiovascular | pressure , blood | SO |
| 4 | Neoplasms | it , diagnose (no robust descriptor) | LAC |
| 5 | Neoplasms | biopsy | LH |
Table 11: Summary of the five misclassified general pathological cases inspected with SHAP. Predicted : erroneous category assigned by DeBERTa-v3. Top driving tokens : tokens with the largest positive SHAP attribution per case. Failure pattern : LH lexical hypersensitivity, SO semantic overlap, LAC loss of attribution coherence (see text).
| Case | Predicted | Summary of the abstract |
| 1 | Cardiovascular | Pathophysiology of glomerular damage and scarring in chronic kidney disease, highlighting the interaction between genetic and metabolic factors. |
| 2 | Neoplasms | Experimental study on the malignant transformation of keratinocytes infected with HPV-18, linking tumour progression to chromosomal instability and cell-cycle failures. |
| 3 | Cardiovascular | Investigates platelet dysfunction in uraemic patients, identifying excess nitric oxide as the key mediator of bleeding and its therapeutic reversibility. |
| 4 | Neoplasms | Studies streptococcal pharyngitis, its treatment with penicillin, and the systemic risks derived from inadequate management of the bacterial infection. |
| 5 | Neoplasms | Safety evaluation of outpatient lung biopsy in 169 patients, showing a low incidence of severe complications and the feasibility of same-day discharge. |
Table 12: Clinical content of the five misclassified general pathological cases. Predicted : erroneous category assigned by DeBERTa-v3. The true category in all cases is General pathological conditions .
Figure 5: Error analysis of a general case misclassified as cardiovascular. Left: saliency map. Right: SHAP waterfall plot.
Figure 6: Error analysis of a general case misclassified as neoplasm. Left: saliency map. Right: SHAP waterfall plot.