Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Musa Tur Farazi; K G Subarno Bithi

eess.IVAug 21, 2026

Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Authors: Musa Tur Farazi, K G Subarno Bithi

Organizations: Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology, Dhaka 1000, Bangladesh · Department of Materials and Metallurgical Engineering, Bangladesh University of Engineering and Technology, Dhaka 1000, Bangladesh

Abstract

Background: Multimodal fracture classifiers may benefit from patient and anatomical metadata, but they can also become brittle when contextual information is missing or mismatched. Methods: We studied 1493 radiographs from the Bangladeshi OrthoFrac-XR dataset using leakage-safe age, sex, bone type, and laterality. A ConvNeXt image encoder was combined with a clinical multilayer perceptron through concatenation, late fusion, reliability-gated residual fusion, and a hierarchical state-location formulation. We additionally introduced an anatomy-consistency gate that attenuates metadata corrections when an image-side anatomical prediction disagrees with the reported bone type. Results: Across five folds and three seeds, hierarchical residual fusion achieved a macro-F1 of 0.6046 +/- 0.0279, compared with 0.5727 +/- 0.0270 for image-only learning, while improving the Brier score from 0.5239 to 0.4948. In a five-fold robustness experiment, anatomy-consistency fusion reduced the macro-F1 loss under shuffled metadata from 0.0567 to 0.0203 relative to ordinary residual fusion, although its clean-data macro-F1 was lower. Without bone type at inference, auxiliary anatomy supervision improved macro-F1 from 0.5620 +/- 0.0330 to 0.5899 +/- 0.0289. Conclusions: Structured context improves fracture classification, and consistency-aware gating limits harm from mismatched metadata. The observed clean-performance-robustness trade-off and the absence of patient-level identifiers motivate external and prospective validation.

Figures & tables

Class	Number of Radiographs	Percentage (%)
Non-fracture	576	38.6
Post-fracture	349	23.4
Distal fracture	314	21.0
Proximal fracture	254	17.0
Total	1493	100.0

Table 1: Distribution of the four-class fracture target.

Figure 1: Overview of the reliability- and anatomy-consistency-aware residual fusion architecture.

Figure 2: Metadata-only feature ablation across five folds. Bars show mean macro-F1; error bars show the standard deviation across folds.

Feature Set	Macro-F1	Balanced Accuracy	Macro-AUROC
Sex only	0.1391 $\pm$ 0.0055	0.2500	0.5167 $\pm$ 0.0135
Age + sex	0.3274 $\pm$ 0.0212	0.3300	0.6127 $\pm$ 0.0223
Age + sex + laterality	0.3254 $\pm$ 0.0231	0.3270	0.6146 $\pm$ 0.0219
Bone type only	0.4401 $\pm$ 0.0388	0.4510	0.6873 $\pm$ 0.0220
All permitted metadata	0.4984 $\pm$ 0.0413	0.5030	0.7498 $\pm$ 0.0132

Table 2: Best metadata-only results for each feature set using histogram gradient boosting (five folds).

Model	Accuracy	Balanced Acc.	Macro-F1	Macro-AUROC	Brier $\downarrow$	ECE $\downarrow$
Clinical MLP	0.4945	0.4978	0.4830	0.7436	0.6444	0.0730
Image only	0.5988	0.5735	0.5727	0.8331	0.5239	0.0698
Hierarchical image only	0.6048	0.5695	0.5731	0.8346	0.5189	0.0671
Late fusion	0.6115	0.5890	0.5873	0.8433	0.5219	0.0959
Concatenation	0.6249	0.6009	0.6015	0.8491	0.4976	0.0702
Residual Fusion	0.6289	0.6019	0.6027	0.8472	0.4969	0.0603

Table 3: Main performance across five folds and three random seeds. Values are means over 15 runs. Higher is better except for Brier score and ECE.

Figure 3: Macro-F1 comparison for the main five-fold, three-seed experiment. Error bars indicate standard deviation over 15 runs.

Feature Set	Model	Macro-F1	Macro-AUROC	Brier $\downarrow$	ECE $\downarrow$
All metadata	Image only	0.5577	0.8225	0.5361	0.0649
	Concatenation	0.6031	0.8503	0.4934	0.0703
	Residual Fusion	0.6012	0.8480	0.4993	0.0603
	Hierarchical Residual Fusion	0.5821	0.8476	0.5031	0.0700
	Anatomy-Consistency Fusion	0.5846	0.8441	0.5044	0.0571
	Hierarchical Anatomy-Consistency Fusion	0.5965	0.8429	0.5080	0.0578

Table 4: Mean clean performance across five folds in the one-seed anatomy-consistency experiment.

Figure 4: Macro-F1 under clean, missing, shuffled, and incorrect metadata in the five-fold targeted robustness experiment.

Model	Clean	All Missing	Shuffled	50% Wrong Bone	100% Wrong Bone
Residual Fusion	0.6012	0.5728 ( $-0.0284$ )	0.5445 ( $-0.0567$ )	0.5855 ( $-0.0158$ )	0.5678 ( $-0.0334$ )
Hierarchical Residual Fusion	0.5821	0.5505 ( $-0.0316$ )	0.5321 ( $-0.0500$ )	0.5632 ( $-0.0190$ )	0.5438 ( $-0.0383$ )
Anatomy-Consistency Fusion	0.5846	0.5661 ( $-0.0185$ )	0.5643 ( $-0.0203$ )	0.5767 ( $-0.0079$ )	0.5663 ( $-0.0183$ )
Hierarchical Anatomy-Consistency Fusion	0.5965	0.5640 ( $-0.0325$ )	0.5610 ( $-0.0355$ )	0.5798 ( $-0.0166$ )	0.5627 ( $-0.0338$ )

Table 5: Robustness of residual models using all permitted metadata. Values are mean macro-F1 over five folds; stochastic corruptions were repeated ten times per fold. Parentheses show the decrease from each model’s own clean performance.

Panel of Figure 5 — Figure 5: Representative diagnostics for concatenation in the targeted five-fold experiment: normalized confusion matrix (left) and reliability diagram after validation-fitted temperature scaling (right).

Explore similar work

CLoE: Expert Consistency Learning for Robust Missing Modality Segmentation
Mar 10, 2026Xinyu Tong, Meihua Zhou, Bowu Fan +1Medical Image Segmentation Missing-Modality Learning
University of Chinese Academy Sciences · School of Medicine, Southeast University · Zhejiang University
Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis
Sep 23, 2026S. M. Nasif Uddin, Rusab Sarmun, Muhammad E. H. Chowdhury +3Cross-Attention Multi-Task Learning
Department of Electrical and Electronic Engineering, Ahsanullah University of Science and Technology, Bangladesh · Department of Electrical and Electronic Engineering, University of Dhaka, Bangladesh · Department of Electrical Engineering, Qatar University, Doha 2713, Doha, Qatar +1
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Sep 9, 2026Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5Cross-Modality Medical Image Segmentation Multi-Source Image Fusion
Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2

Mar 10, 2026J/K move · Enter open · S save

CLoE: Expert Consistency Learning for Robust Missing Modality Segmentation

Xinyu Tong, Meihua Zhou, Bowu Fan, Haitao Li

University of Chinese Academy Sciences · School of Medicine, Southeast University · Zhejiang University

Medical Image Segmentation Missing-Modality Learning Multimodal Robustness Multimodal Fusion Consistency Regularization

Open paper AskPDF arXiv

Multimodal medical image segmentation often faces missing modalities at inference, which induces disagreement among modality experts and makes fusion unstable, particularly on small foreground structures. We propose Consistency Learning of Experts (CLoE), a consistency-driven framework for missing-modality segmentation that preserves strong performance when all modalities are available. CLoE formulates robustness as decision-level expert consistency control and introduces a dual-branch Expert Consistency Learning objective. Modality Expert Consistency enforces global agreement among expert predictions to reduce case-wise drift under partial inputs, while Region Expert Consistency emphasizes agreement on clinically critical foreground regions to avoid background-dominated regularization. We further map consistency scores to modality reliability weights using a lightweight gating network, enabling reliability-aware feature recalibration before fusion. Extensive experiments on BraTS 2020 and MSD Prostate demonstrate that CLoE outperforms state-of-the-art methods in incomplete multimodal segmentation, while exhibiting strong cross-dataset generalization and improving robustness on clinically critical structures.

Abstract

Explore similar work

CLoE: Expert Consistency Learning for Robust Missing Modality Segmentation

Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation