Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis

S. M. Nasif Uddin; Rusab Sarmun; Muhammad E. H. Chowdhury; Adam Mushtak; Israa Al-Hashimi; Sohaib Bassam Zoghoul

eess.IVSep 23, 2026

Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis

Authors: S. M. Nasif Uddin, Rusab Sarmun, Muhammad E. H. Chowdhury, Adam Mushtak, Israa Al-Hashimi, Sohaib Bassam Zoghoul

Organizations: Department of Electrical and Electronic Engineering, Ahsanullah University of Science and Technology, Bangladesh · Department of Electrical and Electronic Engineering, University of Dhaka, Bangladesh · Department of Electrical Engineering, Qatar University, Doha 2713, Doha, Qatar · Department of Radiology, Hamad Medical Corporation, Doha, Qatar

Abstract

Primary bone tumors are rare but clinically aggressive neoplasms whose diagnosis from radiographs is challenged by heterogeneous morphology, subtle lesion margins, and overlapping bone structures. To address the limitations of existing single-view models, we present a dual-input, multi-task learning framework that, to our knowledge, is the first to apply bidirectional cross-modal attention between a lesion crop and the full radiograph for joint segmentation and subtype classification. Using the multi-institutional Bone Tumor X-ray Radiograph Dataset (BTXRD, n=3,746), we employ a YOLO-based detector to generate regions of interest, which are paired with full images as inputs to a dual-stream DenseNet121 architecture. Features are integrated via a novel cross-modal attention fusion strategy, refined by Hierarchical Multi-scale Feature Fusion, effectively balancing fine-grained lesion detail with global anatomical context. Evaluated on a held-out patient-level test split, the model demonstrates superior performance over single-input baselines, achieving an overall Dice Similarity Coefficient of 0.896 and a macro-averaged classification F1-score of 0.928. Notably, the system exhibits exceptional sensitivity for malignant osteosarcoma (AUC 0.999), validating the potential of dual-stream context modeling to support radiologists in accurate, early decision-making.

Figures & tables

Figure 1: Class distribution of the BTXRD dataset

Parameter	Value
Model Variant	YOLOv11x
Input Resolution	$640\times 640$
Batch Size	8
Training Epochs	118
Optimizer	SGD
Initial Learning Rate	$1\times 10^{-3}$

Table 1: YOLOv11 training configuration.

Figure 2: Training-set class distribution before and after augmentation

Figure 3: Proposed dual-input multi-task OsteoHiFuse-Net architecture

Figure 4: Convolutional Block Attention Module (CBAM).

Figure 5: Cross-modal attention fusion module.

Figure 6: Hierarchical multi-scale feature fusion (HiFuse) module.

Hyperparameter	Value
Batch Size	16
Input Crop Resolution	$256\times 256$
Full Input Resolution	$1024\times 1024$
Maximum Epochs	20
Early Stopping Patience	15
Optimizer	AdamW

Table 2: Training hyperparameters for OsteoHiFuse-Net.

Metric	Value
Precision	0.764
Recall (Sensitivity)	0.725
mAP@0.5	0.755
mAP@0.5:0.95	0.412

Table 3: YOLOv11x detection performance on the validation set.

Tumor Type	Dice Score	IoU Score	Sensitivity	Specificity
Osteochondroma	0.882	0.800	0.915	0.916
Osteosarcoma	0.923	0.859	0.911	0.941
Other Benign	0.947	0.901	0.942	0.950
Overall (sample-weighted)	0.896	0.821	0.919	0.924

Table 4: Per-class and overall segmentation performance on the held-out test split.

Figure 7: Qualitative segmentation performance of the proposed model (green: false negative; red: false positive; yellow: true positive).

Figure 8: Normalized confusion matrix for the three tumor classes.

Tumor Type	Accuracy	Precision	Recall	F1-Score	Specificity	ROC-AUC
Osteochondroma	0.965	0.986	0.966	0.976	0.962	0.993
Osteosarcoma	0.985	0.957	0.917	0.936	0.994	0.999
Other Benign	0.960	0.818	0.931	0.871	0.965	0.987
Macro average*	0.955	0.920	0.938	0.928	0.973	0.993

Table 5: Per-class and averaged classification performance on the held-out test split.

Figure 9: Receiver operating characteristic curves for each tumor class (one-vs-rest).

Submodules				Dice	IoU	Accuracy	F1-Score
Full	Crop	CBAM	HiFuse	Score	Score
$\checkmark$				0.862	0.781	0.884	0.730
$\checkmark$	$\checkmark$			0.881	0.792	0.911	0.787
$\checkmark$	$\checkmark$	$\checkmark$		0.890	0.814	0.925	0.873
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	0.896	0.821	0.955	0.928

Table 6: Ablation study over the dual-input, attention and hierarchical-fusion components.

Reference	Dataset	Data modality	Classes
He et al. [ 8 ]	1356	X-Ray	3 class (benign / intermediate / malignant)
von Schacky et al. [ 26 ]	934	X-Ray	Binary
Liu et al. [ 17 ]	643	X-Ray + Clinical	3 class (benign / intermediate / malignant)
Song et al. [ 23 ]	1305	X-Ray + CT + Clinical	3 class (benign / intermediate / malignant)
OsteoHiFuse-Net (Ours)	1613	X-Ray	3 class (osteochondroma / osteosarcoma / other benign)

Table 7: Performance comparison with existing literature.

Explore similar work

Multimodal Brain Tumour Classification Using Feature Fusion
Jun 9, 2026Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan +1Brain Tumor Classification Radiomics
School of Physics, Engineering and Computer Science, University of Hertfordshire, UK
Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis
May 18, 2026Fengyi Zhang, Xujie Zeng, Mohan Liu +2Diagnosis Visible Image Fusion
Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification
Jul 7, 2026Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza MohammadiCross-View Fusion Mammography
School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran · School of Computer Engineering, Iran University of Science and technology, Tehran, Iran

Jun 9, 2026, eess.IVJ/K move · Enter open · S save

Multimodal Brain Tumour Classification Using Feature Fusion

Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan, Volker Steuber

School of Physics, Engineering and Computer Science, University of Hertfordshire, UK

Brain Tumor Classification Radiomics Multimodal Clinical Data Cross-Modal Attention Deep Learning

Open paper AskPDF arXiv

Clinicians diagnose brain tumors by synthesizing patient symptoms, medical history, and quantitative imaging data from modalities such as MRI and CT scans into a unified clinical judgement. However, most deep learning models rely on MRI/CT images alone, failing to replicate the clinicians multimodal reasoning. We explore a two-branch multimodal network combining raw MRI scans with 91 extracted radiomic features (intensity, texture, shape, and boundary descriptors) to classify brain tumors into glioma, meningioma, pituitary, and no-tumor. A pre-trained CNN backbone encodes the image stream, whereas a dedicated MLP encodes the radiomic stream. Both streams are fused via concatenation, gated, or bidirectional cross-modal attention strategies. Across nine experimental runs on a balanced 7,200 image dataset, all multimodal configurations outperform unimodal baselines with gated fusion achieving the best accuracy of 96.13%.

Abstract

Explore similar work

Multimodal Brain Tumour Classification Using Feature Fusion

Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification