Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Musa Tur Farazi; K G Subarno Bithi

eess.IVAug 21, 2026

Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Authors: Musa Tur Farazi, K G Subarno Bithi

Organizations: Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology, Dhaka 1000, Bangladesh · Department of Materials and Metallurgical Engineering, Bangladesh University of Engineering and Technology, Dhaka 1000, Bangladesh

Abstract

Background: Multimodal fracture classifiers may benefit from patient and anatomical metadata, but they can also become brittle when contextual information is missing or mismatched. Methods: We studied 1493 radiographs from the Bangladeshi OrthoFrac-XR dataset using leakage-safe age, sex, bone type, and laterality. A ConvNeXt image encoder was combined with a clinical multilayer perceptron through concatenation, late fusion, reliability-gated residual fusion, and a hierarchical state-location formulation. We additionally introduced an anatomy-consistency gate that attenuates metadata corrections when an image-side anatomical prediction disagrees with the reported bone type. Results: Across five folds and three seeds, hierarchical residual fusion achieved a macro-F1 of 0.6046 +/- 0.0279, compared with 0.5727 +/- 0.0270 for image-only learning, while improving the Brier score from 0.5239 to 0.4948. In a five-fold robustness experiment, anatomy-consistency fusion reduced the macro-F1 loss under shuffled metadata from 0.0567 to 0.0203 relative to ordinary residual fusion, although its clean-data macro-F1 was lower. Without bone type at inference, auxiliary anatomy supervision improved macro-F1 from 0.5620 +/- 0.0330 to 0.5899 +/- 0.0289. Conclusions: Structured context improves fracture classification, and consistency-aware gating limits harm from mismatched metadata. The observed clean-performance-robustness trade-off and the absence of patient-level identifiers motivate external and prospective validation.

Figures & tables

Class	Number of Radiographs	Percentage (%)
Non-fracture	576	38.6
Post-fracture	349	23.4
Distal fracture	314	21.0
Proximal fracture	254	17.0
Total	1493	100.0

Table 1: Distribution of the four-class fracture target.

Figure 1: Overview of the reliability- and anatomy-consistency-aware residual fusion architecture.

Figure 2: Metadata-only feature ablation across five folds. Bars show mean macro-F1; error bars show the standard deviation across folds.

Feature Set	Macro-F1	Balanced Accuracy	Macro-AUROC
Sex only	0.1391 $\pm$ 0.0055	0.2500	0.5167 $\pm$ 0.0135
Age + sex	0.3274 $\pm$ 0.0212	0.3300	0.6127 $\pm$ 0.0223
Age + sex + laterality	0.3254 $\pm$ 0.0231	0.3270	0.6146 $\pm$ 0.0219
Bone type only	0.4401 $\pm$ 0.0388	0.4510	0.6873 $\pm$ 0.0220
All permitted metadata	0.4984 $\pm$ 0.0413	0.5030	0.7498 $\pm$ 0.0132

Table 2: Best metadata-only results for each feature set using histogram gradient boosting (five folds).

Model	Accuracy	Balanced Acc.	Macro-F1	Macro-AUROC	Brier $\downarrow$	ECE $\downarrow$
Clinical MLP	0.4945	0.4978	0.4830	0.7436	0.6444	0.0730
Image only	0.5988	0.5735	0.5727	0.8331	0.5239	0.0698
Hierarchical image only	0.6048	0.5695	0.5731	0.8346	0.5189	0.0671
Late fusion	0.6115	0.5890	0.5873	0.8433	0.5219	0.0959
Concatenation	0.6249	0.6009	0.6015	0.8491	0.4976	0.0702
Residual Fusion	0.6289	0.6019	0.6027	0.8472	0.4969	0.0603

Table 3: Main performance across five folds and three random seeds. Values are means over 15 runs. Higher is better except for Brier score and ECE.

Figure 3: Macro-F1 comparison for the main five-fold, three-seed experiment. Error bars indicate standard deviation over 15 runs.

Feature Set	Model	Macro-F1	Macro-AUROC	Brier $\downarrow$	ECE $\downarrow$
All metadata	Image only	0.5577	0.8225	0.5361	0.0649
	Concatenation	0.6031	0.8503	0.4934	0.0703
	Residual Fusion	0.6012	0.8480	0.4993	0.0603
	Hierarchical Residual Fusion	0.5821	0.8476	0.5031	0.0700
	Anatomy-Consistency Fusion	0.5846	0.8441	0.5044	0.0571
	Hierarchical Anatomy-Consistency Fusion	0.5965	0.8429	0.5080	0.0578

Table 4: Mean clean performance across five folds in the one-seed anatomy-consistency experiment.

Figure 4: Macro-F1 under clean, missing, shuffled, and incorrect metadata in the five-fold targeted robustness experiment.

Model	Clean	All Missing	Shuffled	50% Wrong Bone	100% Wrong Bone
Residual Fusion	0.6012	0.5728 ( $-0.0284$ )	0.5445 ( $-0.0567$ )	0.5855 ( $-0.0158$ )	0.5678 ( $-0.0334$ )
Hierarchical Residual Fusion	0.5821	0.5505 ( $-0.0316$ )	0.5321 ( $-0.0500$ )	0.5632 ( $-0.0190$ )	0.5438 ( $-0.0383$ )
Anatomy-Consistency Fusion	0.5846	0.5661 ( $-0.0185$ )	0.5643 ( $-0.0203$ )	0.5767 ( $-0.0079$ )	0.5663 ( $-0.0183$ )
Hierarchical Anatomy-Consistency Fusion	0.5965	0.5640 ( $-0.0325$ )	0.5610 ( $-0.0355$ )	0.5798 ( $-0.0166$ )	0.5627 ( $-0.0338$ )

Table 5: Robustness of residual models using all permitted metadata. Values are mean macro-F1 over five folds; stochastic corruptions were repeated ten times per fold. Parentheses show the decrease from each model’s own clean performance.

Panel of Figure 5 — Figure 5: Representative diagnostics for concatenation in the targeted five-fold experiment: normalized confusion matrix (left) and reliability diagram after validation-fitted temperature scaling (right).

Explore similar work

Mar 10, 2026

CLoE: Expert Consistency Learning for Robust Missing Modality Segmentation

Multimodal medical image segmentation often faces missing modalities at inference, which induces disagreement among modality experts and makes fusion unstable, particularly on small foreground structures. We propose Consistency Learning of Experts (CLoE), a consistency-driven framework for missing-modality segmentation that preserves strong performance when all modalities are available. CLoE formulates robustness as decision-level expert consistency control and introduces a dual-branch Expert Consistency Learning objective. Modality Expert Consistency enforces global agreement among expert predictions to reduce case-wise drift under partial inputs, while Region Expert Consistency emphasizes agreement on clinically critical foreground regions to avoid background-dominated regularization. We further map consistency scores to modality reliability weights using a lightweight gating network, enabling reliability-aware feature recalibration before fusion. Extensive experiments on BraTS 2020 and MSD Prostate demonstrate that CLoE outperforms state-of-the-art methods in incomplete multimodal segmentation, while exhibiting strong cross-dataset generalization and improving robustness on clinically critical structures.

Xinyu Tong, Meihua Zhou, Bowu Fan +1

University of Chinese Academy Sciences · School of Medicine, Southeast University · Zhejiang University

Medical Image Segmentation Missing-Modality Learning Multimodal Robustness Multimodal Fusion Consistency Regularization

Ask PDF

Sep 23, 2026

Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis

Primary bone tumors are rare but clinically aggressive neoplasms whose diagnosis from radiographs is challenged by heterogeneous morphology, subtle lesion margins, and overlapping bone structures. To address the limitations of existing single-view models, we present a dual-input, multi-task learning framework that, to our knowledge, is the first to apply bidirectional cross-modal attention between a lesion crop and the full radiograph for joint segmentation and subtype classification. Using the multi-institutional Bone Tumor X-ray Radiograph Dataset (BTXRD, n=3,746), we employ a YOLO-based detector to generate regions of interest, which are paired with full images as inputs to a dual-stream DenseNet121 architecture. Features are integrated via a novel cross-modal attention fusion strategy, refined by Hierarchical Multi-scale Feature Fusion, effectively balancing fine-grained lesion detail with global anatomical context. Evaluated on a held-out patient-level test split, the model demonstrates superior performance over single-input baselines, achieving an overall Dice Similarity Coefficient of 0.896 and a macro-averaged classification F1-score of 0.928. Notably, the system exhibits exceptional sensitivity for malignant osteosarcoma (AUC 0.999), validating the potential of dual-stream context modeling to support radiologists in accurate, early decision-making.

S. M. Nasif Uddin, Rusab Sarmun, Muhammad E. H. Chowdhury +3

Department of Electrical and Electronic Engineering, Ahsanullah University of Science and Technology, Bangladesh · Department of Electrical and Electronic Engineering, University of Dhaka, Bangladesh · Department of Electrical Engineering, Qatar University, Doha 2713, Doha, Qatar +1

Cross-Attention Multi-Task Learning Medical Image Classification Medical Image Segmentation

Ask PDF

Sep 9, 2026

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation

Multi-modal medical image segmentation leverages complementary diagnostic information, yet fusion can underperform single-modality baselines when spatially aligned inputs differ in quality. Here, "corruption" primarily denotes resolution-induced degradation rather than misalignment or complete modality absence, while synthetic noise is evaluated only as an auxiliary setting. We identify a critical optimization-inference inconsistency: degraded modalities can receive weak training updates yet substantially affect predictions, indicating active interference with fusion. We attribute this failure to resampling-induced feature corruption and optimization bias, where noisy features propagate through skip connections and encourage unreliable modality selection. We therefore propose CoReFuse-Med, a Corruption-aware Rebalanced Fusion framework that suppresses corruption during feature transmission and rebalances modality contributions during high-level fusion. Experiments on EPVS, BraTS, and WMH, including multiple Z-axis slice-retention ratios and an auxiliary noise test, demonstrate improved accuracy and robustness under modality-quality discrepancies. Our code is available at https://github.com/lrever/CoReFuse.

Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5

Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2

Cross-Modality Medical Image Segmentation Multi-Source Image Fusion Multimodal Robustness Medical Image Segmentation Multimodal Fusion

Ask PDF