TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring

Automated classification of pulmonary disease from chest radiographs is a widely studied application of machine learning in medical imaging. This paper presents a pilot study evaluating classical texture- and gradient-based feature representations for distinguishing COVID-19 from other forms of pneumonia using the publicly available COVID-19 Image Data Collection (668 posteroanterior/anteroposterior radiographs from 408 patients). Using histogram of oriented gradients (HOG) and gray-level co-occurrence matrix (GLCM) texture descriptors with classical classifiers (logistic regression, random forest, and support vector machine), evaluated under patient-level 5-fold stratified cross-validation to prevent data leakage, we obtain a best mean accuracy of 75.4% and AUC of 0.755, modestly exceeding the 71.6% majority-class baseline. We report these results transparently, including their limitations, and use them to motivate and scope a proposed multi-modal deep learning architecture -- combining convolutional and transformer-based encoders across imaging modalities -- as a direction for future work requiring access to larger, multi-institutional, ethically sourced datasets.

Deep-learning models can achieve strong chest X-ray (CXR) classification performance without establishing whether their predictions predominantly rely on pulmonary image content. This study evaluates pulmonary attribution containment as an anatomy-related reliability property distinct from diagnostic performance. We propose DBCA-SegNet-MGAP, a multi-task anatomy-guided CNN-Transformer framework that combines complementary feature representations through bidirectional cross-backbone attention, predicts a soft lung mask, and incorporates this anatomical prior directly into classification through Mask-Guided Adaptive Global Average Pooling (MGAP). Pulmonary attribution containment is quantified using the Anatomical Local Energy Ratio (ALR) and high-intensity cumulative ALR (cALR@0.9). Experiments were repeated across three training seeds using the COVID-19 Radiography Database for four-class internal testing and a locked Shenzhen-to-Montgomery protocol for zero-shot external tuberculosis testing. On COVID-19, the proposed model achieved a weighted F1 of $0.9615 \pm 0.0015$ and macro ROC-AUC of $0.9906 \pm 0.0007$. In an architecture-matched dual-bridge comparison, replacing conventional GAP with MGAP increased ALR from $0.3878 \pm 0.0098$ to $0.7086 \pm 0.0104$ and cALR@0.9 from $0.5265 \pm 0.0101$ to $0.9905 \pm 0.0018$, while weighted F1 remained essentially unchanged ($0.9618 \pm 0.0015$ vs. $0.9615 \pm 0.0015$). Under locked external transfer to Montgomery, ROC-AUC remained $0.9080 \pm 0.0043$ and pulmonary ALR remained $0.6466 \pm 0.0081$, whereas weighted F1 decreased to $0.7528 \pm 0.0080$ and ECE increased to $0.1683 \pm 0.0055$. These findings show that diagnostic discrimination, calibration, and pulmonary attribution containment are distinct model properties and support their joint evaluation under internal testing and external domain shift.

Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance.