An Attention-Guided Global and Local Fusion Framework for Lesion-Focused Image Classification

Mst Shafia Tasnima; Md Samaun Elaheea; Tanjim Taharat Aurpab; Md Musfique Anwar

cs.CVSep 4, 2026

An Attention-Guided Global and Local Fusion Framework for Lesion-Focused Image Classification

Authors: Mst Shafia Tasnima, Md Samaun Elaheea, Tanjim Taharat Aurpab, Md Musfique Anwar

Organizations: Department of Computer Science and Engineering, Jahangirnagar University, Savar, Bangladesh · Department of Data Science and Engineering, University of Frontier Technology, Bangladesh (UFTB), Gazipur, Bangladesh

Abstract

Lesion-focused image classification presents a core analytical challenge, as discriminative signals are often sparse, spatially dispersed, and easily obscured by background noise, while conventional convolutional neural networks (CNNs) process entire images uniformly and may dilute signal relevance. This study hypothesizes that adaptive fusion of global contextual information and lesion-focused local information can improve classification performance compared with using either representation independently. We propose a three-branch, attention-guided deep learning framework built on Densely Connected Convolutional Network-121 (DenseNet-121) to improve feature attribution, interpretability, and classification reliability. The architecture consists of a global branch that learns representations from full images, followed by Gradient-weighted Class Activation Mapping (Grad-CAM) to generate attention maps that highlight prediction-relevant regions and produce masked inputs, and a local branch enhanced with a Convolutional Block Attention Module (CBAM) to extract refined spatial and channel-wise features from these focused regions. An adaptive fusion branch integrates global and local representations by learning instance-specific weights, allowing dynamic prioritization between contextual and localized information. The framework is evaluated on a synthetic Spot Pattern Dataset (SSPD) and three benchmark datasets, including skin lesion, guava leaf, and grape leaf image datasets, where the fusion branch outperformed the individual global and local branches, reaching 97.75% accuracy on the skin lesion dataset and 99.64% on the guava leaf dataset. The results highlight the value of attention-guided architectures in healthcare analytics by improving model transparency, strengthening feature relevance, and supporting more reliable data-driven decision-making in medical image analysis.

Figures & tables

Figure 1: System architecture of the proposed framework

Figure 2: Pie charts of dataset split and class distribution. All reported testing results and ablation analyses for all datasets were performed on their corresponding held-out test subsets.

Figure 3: Synthetic Spot Pattern Dataset (SSPD)

	Dataset 2			Dataset 3			Dataset 4 (Grape)
Backbone	Global	Local	Fusion	Global	Local	Fusion	Global	Local	Fusion
DenseNet-121	94.75%	94.25%	97.75%	98.76%	95.84%	99.64%	94.16%	89.44%	96.52%
DenseNet-169	95.75%	89.50%	91.75%	98.48%	94.84%	98.00%	92.92%	84.86%	93.89%
InceptionV3	96.25%	96.00%	97.00%	98.44%	93.32%	98.72%	93.33%	86.81%	92.64%
Xception	92.25%	91.25%	94.75%	99.00%	96.88%	99.20%	92.39%	88.75%	94.19%

Table 1: Performance comparison using multiple backbone architectures. DenseNet-121 delivered the best overall fusion accuracy across all three datasets.

Figure 4: Performance comparison across different Grad-CAM threshold values. Across all three datasets, a threshold of 0.5 consistently yielded the highest fusion accuracy. A lower threshold of 0.25 includes excessive background noise, whereas a higher threshold of 0.75 may mask out critical lesion features. Therefore, 0.5 was selected in this study, while the threshold remains adjustable based on dataset characteristics and performance.

Figure 5: Original images, Grad-CAM heatmaps, binary masks, and masked images for Datasets 2, 3, and 4.

Figure 6: Local branch with and without CBAM. Incorporating CBAM in the local branch consistently improves local and fusion accuracy across all datasets.

Figure 7: Comparison of model performance with and without global weight initialization. While global weight initialization slightly improved the local branch accuracy for Dataset 3, fusion performance benefited more from independently trained branches. Both local and fusion performance degraded for Datasets 2 and 4 under global weight initialization.

Figure 8: Comparison between simple feature concatenation and adaptive feature fusion across Datasets 2, 3, and 4.

Class	$\alpha_{g}$ Min–Max	$\alpha_{l}$ Min–Max
Canker	0.183–0.898	0.102–0.817
Dot	0.256–0.938	0.062–0.744
Healthy	0.197–0.986	0.014–0.803
Mummification	0.083–0.953	0.047–0.917
Rust	0.257–0.957	0.043–0.743

Table 2: Class-wise ranges of learned global and local fusion weights on the Guava dataset.

Figure 9: Fusion model training accuracy curves for Datasets 2, 3, and 4

Dataset	Branch	Seed 7	Seed 42	Seed 99	Seed 123	Mean $\pm$ SD (%)
Skin lesion	Global	95.50	94.75	96.00	94.25	95.13 $\pm$ 0.78
	Local	94.25	94.75	95.75	93.25	94.50 $\pm$ 1.04
	Fusion	98.00	97.25	97.75	97.25	97.56 $\pm$ 0.38
Grape dataset	Global	94.86	94.02	94.44	94.72	94.51 $\pm$ 0.37
	Local	88.19	89.44	89.10	90.13	89.22 $\pm$ 0.81
	Fusion	95.83	96.11	95.97	96.66	96.14 $\pm$ 0.36

Table 4: Seed-wise accuracy comparison for Datasets 2 and 4

Panel of Figure 10 — Figure 10: Class-wise F1-score for all four datasets

Measure	Guava Dataset
Global errors	31
Fusion errors	9
Wrong by both models	5
Global wrong $\rightarrow$ Fusion correct	26
Global correct $\rightarrow$ Fusion wrong	4
McNemar $p$ -value	$\mathbf{5.95\times 10^{-5}}$

Table 5: Paired statistical comparison of the Global and Fusion models on the Guava dataset.

Model Branch	Accuracy (%)
Global	79.11
Local	82.55
Fusion	83.83

Table 6: Source-held-out accuracy on the MSID dataset

Figure 12: Class-wise F1-score comparison of the global, local, and fusion branches in the source-held-out experiment.

Application Domain	Paper	Dataset	Classes	Method	Accuracy
Skin Lesion	[ 45 ]	Public dataset	2,4	Developed multiple deep neural network models, with a modified DenseNet201-based architecture as the main model. LIME and Grad-CAM were incorporated to provide visual explainability.	97.63% (2-class), 95.18% (4-class)
	[ 3 ]	MCVSLD dataset	6	Proposed a modified Xception-based deep learning architecture	96.36%, 97.01%
	[ 49 ]	MSLD v2.0	6	Evaluated transformer-based deep learning architectures, including ViT, MAE, DINO, and Swin Transformer. The study investigated the effects of self-supervised learning, self-distillation, and shifted-window techniques on classification performance	93.71%
	[ 28 ]	MSID	4	Proposed Mpox-XDE, an ensemble architecture combining modified Xception, DenseNet201, and EfficientNetB7 models for classification, with Grad-CAM applied for explainable visualization.	98.70%
	[ 10 ]	MSLD, MSID	2,4	Introduced MSCADMpox, a multi-stage classification framework combining generative augmentation, handcrafted feature extraction, and deep feature extraction. The method used ViT, VGG16, VGG19, ResNet50, MBGWO-based feature selection, and traditional classifiers for final decision-making.	90.54% (MSLD), 91.88% (MSID)
	Our Model	MSLD v2.0, MSID	4	Proposed global–local–fusion framework combining global image features, local lesion-focused features, and adaptive fusion for skin lesion classification	97.75%

Table 7: Contextual performance comparison with existing studies on skin lesion, guava leaf, and grape leaf disease datasets.

Figure 13: Comparative performance of the global–local–fusion branches under increased background complexity using Gaussian noise ( $\sigma=70.0$ ).

Explore similar work

Jul 12, 2026

Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework, Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier's attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention.

Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali +1

Department of Computer Science, University of Virginia, Virginia, USA · Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Kahlert School of Computing, University of Utah, Utah, USA +1

Medical Image Classification Attention Mechanisms Explainable Medical Image Analysis

Ask PDF

Jun 18, 2026

Geometry-Aware Superpixel Graph Transformer with Metadata for Skin Lesion Classification

Automated skin cancer classification from dermoscopic images remains challenging due to heterogeneous lesion structure, strong intra-class variability, and subtle visual differences between benign and malignant cases. Existing CNN/ViT pipelines typically rely on global or patch-level features and often combine patient metadata via late fusion, which limits spatially grounded multimodal reasoning. We present a novel region-based graph learning framework that explicitly models lesions as graphs of spatially coherent superpixel regions represented as frozen CNN features. To capture fine-grained lesion arrangements, we encode inter-regional geometry as edge attributes and introduce a dedicated metadata context node connected to all regions, providing structured integration of demographic/clinical variables within the same relational space. Node representations are updated using our edge-aware graph transformer followed by attention-driven propagation, and a final graph-level embedding for benign-malignant classification. Experiments on four public benchmarks demonstrate that explicit region-level relational modeling and graph-native multimodal fusion yield consistent gains over the state-of-the-art. Consequently, we establish a new graph-centric perspective in which CNN features are modeled as relational nodes and improved through contextual integration, yielding more expressive and robust classifications.

Muhammad Azeem, Tanveer Hussain, Amr Ahmed +1

Edge Hill University, Ormskirk, Lancashire, L39 4QP, United Kingdom

Skin Lesion Classification Graph Representation Learning Multimodal Classification Medical Image Classification Graph Transformer

Ask PDF

Jun 24, 2026

DCSNet: Multiscale Feature Aggregation for Small Medical Object Segmentation with Detection-guided Hierarchical Cropping

Small object segmentation in medical imaging is primarily hindered by class imbalance and inherent boundary complexity. Consequently, conventional global networks frequently fail to detect sparse targets or suffer from severe edge degradation. To overcome these limitations, we propose the Detection-guided Cropping Segmentation Network (DCSNet), an end-to-end framework that transforms global dense prediction into a localized refinement process. This framework integrates two core components, namely Detection-guided Hierarchical Cropping (DGHC) and Multiscale Feature Aggregation (MSFA). The DGHC module leverages region proposals to dynamically extract object-centric features, effdataectively filtering out massive background interference to mitigate class imbalance. Subsequently, the MSFA module operates strictly within these purified regions, synergizing a Transformer encoder with a pixel-adaptive fusion strategy. This mechanism dynamically aggregates multiscale features to capture both semantic context and fine-grained details for sharp boundary delineation. Extensive experiments across three diverse medical datasets demonstrate that DCSNet significantly outperforms existing state-of-the-art methods, yielding substantial improvements in boundary precision and offering a highly robust solution for clinical micro-lesion segmentation.

Shanfeng Zhang, Bo Gou, Yue Cao +3

College of Computer Science, Sichuan University, 610065, Chengdu, China · The First Affiliated Hospital of Chengdu Medical College, School of Clinical Medicine, Chengdu Medical College, 610065, Chengdu, China · School of Artificial Intelligence, Sichuan University, 610065, Chengdu, China

Image Segmentation Multi-Scale Feature Fusion Medical Image Segmentation

Ask PDF