An Attention-Guided Global and Local Fusion Framework for Lesion-Focused Image Classification

Mst Shafia Tasnima; Md Samaun Elaheea; Tanjim Taharat Aurpab; Md Musfique Anwar

cs.CVSep 4, 2026

An Attention-Guided Global and Local Fusion Framework for Lesion-Focused Image Classification

Authors: Mst Shafia Tasnima, Md Samaun Elaheea, Tanjim Taharat Aurpab, Md Musfique Anwar

Organizations: Department of Computer Science and Engineering, Jahangirnagar University, Savar, Bangladesh · Department of Data Science and Engineering, University of Frontier Technology, Bangladesh (UFTB), Gazipur, Bangladesh

Abstract

Lesion-focused image classification presents a core analytical challenge, as discriminative signals are often sparse, spatially dispersed, and easily obscured by background noise, while conventional convolutional neural networks (CNNs) process entire images uniformly and may dilute signal relevance. This study hypothesizes that adaptive fusion of global contextual information and lesion-focused local information can improve classification performance compared with using either representation independently. We propose a three-branch, attention-guided deep learning framework built on Densely Connected Convolutional Network-121 (DenseNet-121) to improve feature attribution, interpretability, and classification reliability. The architecture consists of a global branch that learns representations from full images, followed by Gradient-weighted Class Activation Mapping (Grad-CAM) to generate attention maps that highlight prediction-relevant regions and produce masked inputs, and a local branch enhanced with a Convolutional Block Attention Module (CBAM) to extract refined spatial and channel-wise features from these focused regions. An adaptive fusion branch integrates global and local representations by learning instance-specific weights, allowing dynamic prioritization between contextual and localized information. The framework is evaluated on a synthetic Spot Pattern Dataset (SSPD) and three benchmark datasets, including skin lesion, guava leaf, and grape leaf image datasets, where the fusion branch outperformed the individual global and local branches, reaching 97.75% accuracy on the skin lesion dataset and 99.64% on the guava leaf dataset. The results highlight the value of attention-guided architectures in healthcare analytics by improving model transparency, strengthening feature relevance, and supporting more reliable data-driven decision-making in medical image analysis.

Figures & tables

Figure 1: System architecture of the proposed framework

Figure 2: Pie charts of dataset split and class distribution. All reported testing results and ablation analyses for all datasets were performed on their corresponding held-out test subsets.

Figure 3: Synthetic Spot Pattern Dataset (SSPD)

	Dataset 2			Dataset 3			Dataset 4 (Grape)
Backbone	Global	Local	Fusion	Global	Local	Fusion	Global	Local	Fusion
DenseNet-121	94.75%	94.25%	97.75%	98.76%	95.84%	99.64%	94.16%	89.44%	96.52%
DenseNet-169	95.75%	89.50%	91.75%	98.48%	94.84%	98.00%	92.92%	84.86%	93.89%
InceptionV3	96.25%	96.00%	97.00%	98.44%	93.32%	98.72%	93.33%	86.81%	92.64%
Xception	92.25%	91.25%	94.75%	99.00%	96.88%	99.20%	92.39%	88.75%	94.19%

Table 1: Performance comparison using multiple backbone architectures. DenseNet-121 delivered the best overall fusion accuracy across all three datasets.

Figure 4: Performance comparison across different Grad-CAM threshold values. Across all three datasets, a threshold of 0.5 consistently yielded the highest fusion accuracy. A lower threshold of 0.25 includes excessive background noise, whereas a higher threshold of 0.75 may mask out critical lesion features. Therefore, 0.5 was selected in this study, while the threshold remains adjustable based on dataset characteristics and performance.

Figure 5: Original images, Grad-CAM heatmaps, binary masks, and masked images for Datasets 2, 3, and 4.

Figure 6: Local branch with and without CBAM. Incorporating CBAM in the local branch consistently improves local and fusion accuracy across all datasets.

Figure 7: Comparison of model performance with and without global weight initialization. While global weight initialization slightly improved the local branch accuracy for Dataset 3, fusion performance benefited more from independently trained branches. Both local and fusion performance degraded for Datasets 2 and 4 under global weight initialization.

Figure 8: Comparison between simple feature concatenation and adaptive feature fusion across Datasets 2, 3, and 4.

Class	$\alpha_{g}$ Min–Max	$\alpha_{l}$ Min–Max
Canker	0.183–0.898	0.102–0.817
Dot	0.256–0.938	0.062–0.744
Healthy	0.197–0.986	0.014–0.803
Mummification	0.083–0.953	0.047–0.917
Rust	0.257–0.957	0.043–0.743

Table 2: Class-wise ranges of learned global and local fusion weights on the Guava dataset.

Figure 9: Fusion model training accuracy curves for Datasets 2, 3, and 4

Dataset	Branch	Seed 7	Seed 42	Seed 99	Seed 123	Mean $\pm$ SD (%)
Skin lesion	Global	95.50	94.75	96.00	94.25	95.13 $\pm$ 0.78
	Local	94.25	94.75	95.75	93.25	94.50 $\pm$ 1.04
	Fusion	98.00	97.25	97.75	97.25	97.56 $\pm$ 0.38
Grape dataset	Global	94.86	94.02	94.44	94.72	94.51 $\pm$ 0.37
	Local	88.19	89.44	89.10	90.13	89.22 $\pm$ 0.81
	Fusion	95.83	96.11	95.97	96.66	96.14 $\pm$ 0.36

Table 4: Seed-wise accuracy comparison for Datasets 2 and 4

Panel of Figure 10 — Figure 10: Class-wise F1-score for all four datasets

Measure	Guava Dataset
Global errors	31
Fusion errors	9
Wrong by both models	5
Global wrong $\rightarrow$ Fusion correct	26
Global correct $\rightarrow$ Fusion wrong	4
McNemar $p$ -value	$\mathbf{5.95\times 10^{-5}}$

Table 5: Paired statistical comparison of the Global and Fusion models on the Guava dataset.

Model Branch	Accuracy (%)
Global	79.11
Local	82.55
Fusion	83.83

Table 6: Source-held-out accuracy on the MSID dataset

Figure 12: Class-wise F1-score comparison of the global, local, and fusion branches in the source-held-out experiment.

Application Domain	Paper	Dataset	Classes	Method	Accuracy
Skin Lesion	[ 45 ]	Public dataset	2,4	Developed multiple deep neural network models, with a modified DenseNet201-based architecture as the main model. LIME and Grad-CAM were incorporated to provide visual explainability.	97.63% (2-class), 95.18% (4-class)
	[ 3 ]	MCVSLD dataset	6	Proposed a modified Xception-based deep learning architecture	96.36%, 97.01%
	[ 49 ]	MSLD v2.0	6	Evaluated transformer-based deep learning architectures, including ViT, MAE, DINO, and Swin Transformer. The study investigated the effects of self-supervised learning, self-distillation, and shifted-window techniques on classification performance	93.71%
	[ 28 ]	MSID	4	Proposed Mpox-XDE, an ensemble architecture combining modified Xception, DenseNet201, and EfficientNetB7 models for classification, with Grad-CAM applied for explainable visualization.	98.70%
	[ 10 ]	MSLD, MSID	2,4	Introduced MSCADMpox, a multi-stage classification framework combining generative augmentation, handcrafted feature extraction, and deep feature extraction. The method used ViT, VGG16, VGG19, ResNet50, MBGWO-based feature selection, and traditional classifiers for final decision-making.	90.54% (MSLD), 91.88% (MSID)
	Our Model	MSLD v2.0, MSID	4	Proposed global–local–fusion framework combining global image features, local lesion-focused features, and adaptive fusion for skin lesion classification	97.75%

Table 7: Contextual performance comparison with existing studies on skin lesion, guava leaf, and grape leaf disease datasets.

Figure 13: Comparative performance of the global–local–fusion branches under increased background complexity using Gaussian noise ( $\sigma=70.0$ ).

Explore similar work

Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification
Jul 12, 2026Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali +1Medical Image Classification Attention Mechanisms
Department of Computer Science, University of Virginia, Virginia, USA · Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Kahlert School of Computing, University of Utah, Utah, USA +1
Geometry-Aware Superpixel Graph Transformer with Metadata for Skin Lesion Classification
Jun 18, 2026Muhammad Azeem, Tanveer Hussain, Amr Ahmed +1Skin Lesion Classification Graph Representation Learning
Edge Hill University, Ormskirk, Lancashire, L39 4QP, United Kingdom
DCSNet: Multiscale Feature Aggregation for Small Medical Object Segmentation with Detection-guided Hierarchical Cropping
Jun 24, 2026Shanfeng Zhang, Bo Gou, Yue Cao +3Image Segmentation Multi-Scale Feature Fusion
College of Computer Science, Sichuan University, 610065, Chengdu, China · The First Affiliated Hospital of Chengdu Medical College, School of Clinical Medicine, Chengdu Medical College, 610065, Chengdu, China · School of Artificial Intelligence, Sichuan University, 610065, Chengdu, China

Jul 12, 2026J/K move · Enter open · S save

Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali, Miaomiao Zhang

Department of Computer Science, University of Virginia, Virginia, USA · Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Kahlert School of Computing, University of Utah, Utah, USA+1

Medical Image Classification Attention Mechanisms Explainable Medical Image Analysis

Open paper AskPDF arXiv

Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework, Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier's attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention.

Abstract

Explore similar work

Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

Geometry-Aware Superpixel Graph Transformer with Metadata for Skin Lesion Classification

DCSNet: Multiscale Feature Aggregation for Small Medical Object Segmentation with Detection-guided Hierarchical Cropping