MagViT: Interpretable Multi-Magnification Transformers with Patient-Level Model Selection for Breast Histopathology

Nabil Ashab; Soumit Kumar Kundu; Saif Mahmud Parvez; Shahadat Hossain Sohag; Bidhan Biswas; Nazmus Subha

eess.IVAug 16, 2026

MagViT: Interpretable Multi-Magnification Transformers with Patient-Level Model Selection for Breast Histopathology

Authors: Nabil Ashab, Soumit Kumar Kundu, Saif Mahmud Parvez, Shahadat Hossain Sohag, Bidhan Biswas, Nazmus Subha

Organizations: Department of Computer Science and Engineering, Dhaka International University, Dhaka, Bangladesh · Department of Microbiology, TMSS Medical College, Bogura, Bangladesh

Abstract

Breast cancer is one of the most common types of cancer among women around the world. Rapid detection and early treatment can hinder its progress to more complex stages and can impede its spread to other parts of the body. Histopathological image classification is the most common task in cancer detection due to its robustness in analyzing cellular data. Breast histopathology classification requires handling both multi-scale tissue morphology and clinically relevant generalization beyond the source domain. This paper presents MagViT, an interpretable multi-magnification transformer framework with scale-gated fusion and patient-level model selection. The model uses four BreakHis magnifications (40X, 100X, 200X, 400X) and extracts per-scale representations with a ViT backbone, and combines them via a learnable gate that masks missing scales. Patient-level five-fold cross-validation with a fixed seed has been run and compared with three architectural branches. The most accurate branch is then selected as the final model due to the strongest patient-level accuracy while retaining the simplest fusion pathway. On BreakHis, our architecture achieves a mean image accuracy of 0.9191, a mean patient accuracy of 0.9643, and a mean macro-F1 of 0.9042. External transfer experiments provide preliminary evidence of cross-dataset generalization under controlled adaptation settings on BUSI (image accuracy 0.8306, macro-F1 0.7480, patient accuracy 0.8291) and IDC (image accuracy 0.8577, macro-F1 0.8191, patient accuracy 0.8372). Grad-CAM visualization indicates that the model focuses on diagnostically significant and meaningful regions across magnifications. Relative to prior ViT-centered BreakHis work, this study emphasizes patient-level selection and cross-dataset robustness under a reproducible protocol.

Figures & tables

Figure 1: Proposed multi-magnification architecture with expanded streams and outputs

Figure 2: Branch-wise segregation and branch 1 selection for the main MagViT architecture

Step 1: Create multi-magnification samples as groups from dataset

Step 2: Ensure patient-wise disjoint splits across folds

Step 3: Initialize MagViT model with shared encoder

For each fold:

Apply identical augmentation across all magnifications

Extract features from available scales

Handle missing scales using availability mask

Table I: Full End-to-End Final Pipeline

Figure 3: In-domain binary diagnostics: ROC curve (left) and confusion matrix (right).

Setting	Img. Acc. Mean	Img. Acc. Std	Pat. Acc. Mean	Pat. Acc. Std
No augmentation	0.9015	0.0381	0.9357	0.0539
No color jitter	0.9103	0.0359	0.9589	0.0467
No class weights	0.9094	0.0354	0.9482	0.0511

Table II: Component-level ablation study (patient-level protocol retained).

Method	Patient Acc.	Notes
ResNet-18 (single-scale)	0.8571	Standard CNN baseline
ResNet-50 (single-scale)	0.8929	Deeper CNN variant
CNN multi-scale fusion	0.9286	Feature-level aggregation
AutoML (neural search)	0.9143	Architecture search baseline
Hybrid CNN ensemble	0.9357	Multi-backbone ensemble
MagViT (proposed)	0.9643	MMFF-ViT

Table III: Patient-level accuracy comparison on BreakHis (reported results from prior studies; not strictly comparable due to protocol differences).

Branch	Image Acc.	Patient Acc.	Macro-F1
Branch 1	0.9191 $\pm$ 0.0335	0.9643 $\pm$ 0.0357	0.9042 $\pm$ 0.0379
Branch 2	0.9238 $\pm$ 0.0378	0.9571 $\pm$ 0.0466	0.9090 $\pm$ 0.0482
Branch 3	0.8260 $\pm$ 0.0232	0.9000 $\pm$ 0.0299	0.8005 $\pm$ 0.0288

Table IV: Branch comparison on BreakHis (patient-level 5-fold CV).

Fold	Image Acc.	Patient Acc.	Macro-F1	Runtime (min)
1	0.9243	0.9286	0.9112	39.36
2	0.8959	0.8571	0.8728	40.89
3	0.9659	1.0000	0.9585	40.15
4	0.8813	0.8571	0.8645	40.26
5	0.9629	1.0000	0.9569	38.80

Table V: Fold-wise performance of selected full model (Branch-1-compatible final pipeline).

Magnification	Mean Image Acc.	Mean Patient Acc.	Mean Macro-F1
40X	0.9106	0.9786	0.8972
100X	0.9112	0.9643	0.8998
200X	0.9209	0.9643	0.9067
400X	0.8920	0.9357	0.8795

Table VI: Magnification-wise summary (mean across 5 folds).

Dataset	Setting	Image Acc.	Macro-F1	Patient Acc.	ROC-AUC
BUSI	Linear probe	0.7581	0.4312	0.7436	0.6908
BUSI	Few-shot	0.8306	0.7480	0.8291	0.8610
IDC	Linear probe	0.8503	0.8194	0.7907	0.9144
IDC	Few-shot	0.8577	0.8191	0.8372	0.9197

Table VII: Transfer performance on BUSI and IDC.

Figure 4: BUSI (left) and IDC (right) transfer diagnostics: ROC curve and confusion matrix.

Figure 5: Grad-CAM visualization for benign and malignant samples. Warmer colors indicate a higher contribution to the predicted class.

Explore similar work

Mar 15, 2026

Histo-MExNet: A Unified Framework for Real-World, Cross-Magnification, and Trustworthy Breast Cancer Histopathology

Accurate and reliable histopathological image classification is essential for breast cancer diagnosis. However, many deep learning models remain sensitive to magnification variability and lack interpretability. To address these challenges, we propose Histo-MExNet, a unified framework designed for scaleinvariant and uncertainty-aware classification. The model integrates DenseNet, ConvNeXt, and EfficientNet backbones within a gated multi-expert architecture, incorporates a prototype learning module for example-driven interpretability, and applies physics-informed regularization to enforce morphology preservation and spatial coherence during feature learning. Monte Carlo Dropout is used to quantify predictive uncertainty. On the BreaKHis dataset, Histo-MExNet achieves 96.97% accuracy under multi-magnification training and demonstrates improved generalization to unseen magnification levels compared to single-expert models, while uncertainty estimation helps identify out-of-distribution samples and reduce overconfident errors, supporting a balanced combination of accuracy, robustness, and interpretability for clinical decision support.

Enam Ahmed Taufika, Md Ahasanul Arafatha, Abhijit Kumar Ghoshb +2

European University of Bangladesh, Department of Computer Science and Engineering, Dhaka, Bangladesh · BRAC University, Department of Computer Science and Engineering, Dhaka, Bangladesh · Ochsner Center for Outcomes Research, Ochsner Research, Ochsner Clinic Foundation, New Orleans, LA, 70121, USA

Histopathology Image Classification Explainable Medical Image Analysis

Ask PDF

Mar 12, 2026

DeepHistoViT: An Interpretable Vision Transformer Framework for Histopathological Cancer Classification

Histopathology remains the gold standard for cancer diagnosis because it provides detailed cellular-level assessment of tissue morphology. However, manual histopathological examination is time-consuming, labour-intensive, and subject to inter-observer variability, creating a demand for reliable computer-assisted diagnostic tools. Recent advances in deep learning, particularly transformer-based architectures, have shown strong potential for modelling complex spatial dependencies in medical images. In this work, we propose DeepHistoViT, a transformer-based framework for automated classification of histopathological images. The model employs a customized Vision Transformer architecture with an integrated attention mechanism designed to capture fine-grained cellular structures while improving interpretability through attention-based localization of diagnostically relevant regions. The framework is evaluated on three publicly available histopathology datasets covering lung cancer, colon cancer, and acute lymphoblastic leukaemia. Experimental results demonstrate state-of-the-art performance across all datasets, with classification accuracy, precision, recall, F1-score, and ROC-AUC reaching 100 percent on the lung and colon cancer datasets, and 99.85 percent, 99.84 percent, 99.86 percent, 99.85 percent, and 99.99 percent respectively on the acute lymphoblastic leukaemia dataset. All performance metrics are reported with 95 percent confidence intervals. These results highlight the effectiveness of transformer-based architectures for histopathological image analysis and demonstrate the potential of DeepHistoViT as an interpretable computer-assisted diagnostic tool to support pathologists in clinical decision-making.

Ravi Mosalpuri, Mohammed Abdelsamea, Ahmed Karam Eldaly

Histopathology Image Classification Computational Pathology Vision Transformer Explainable Medical Image Analysis

Ask PDF

Apr 21, 2026

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

Vision Transformers

(\texttt{ViT})

have become the architecture of choice for many computer vision tasks, yet their performance in computer-aided diagnostics remains limited. Focusing on breast cancer detection from mammograms, we identify two main causes for this shortfall. First, medical images are high-resolution with small abnormalities, leading to an excessive number of tokens and making it difficult for the softmax-based attention to localize and attend to relevant regions. Second, medical image classification is inherently fine-grained, with low inter-class and high intra-class variability, where standard cross-entropy training is insufficient. To overcome these challenges, we propose a framework with three key components: (1) Region of interest

(\texttt{RoI})

based token reduction using an object detection model to guide attention; (2) contrastive learning between selected

\texttt{RoI}

to enhance fine-grained discrimination through hard-negative based training; and (3) a

\texttt{DINOv2}

pretrained

\texttt{ViT}

that captures localization-aware, fine-grained features instead of global

\texttt{CLIP}

representations. Experiments on public mammography datasets demonstrate that our method achieves superior performance over existing baselines, establishing its effectiveness and potential clinical utility for large-scale breast cancer screening. Our code is available for reproducibility here: https://aih-iitd.github.io/publications/attend-what-matters

Samyak Sanghvi, Piyush Miglani, Sarvesh Shashikumar +3

Department of Computer Science and Engineering, IIT Delhi, New Delhi, India · Yardi School of AI, IIT Delhi, New Delhi, India · Department of Radiodiagnosis, PGIMER Chandigarh, Chandigarh, India

Medical Imaging Vision Foundation Model Adaptation Medical Image Classification Mammography Oncology

Ask PDF