Self-Supervised Learning for Knee Osteoarthritis: Diagnostic Limitations and Prognostic Value of Hospital Data
Authors: Haresh Rengaraj Rajamohan, Yuxuan Chen, Kyunghyun Cho, Cem M. Deniz
Organizations: Center for Data Science, New York University, New York, 10011, NY, United States · Perlmutter Cancer Center, New York University Langone Health, New York, 10016, NY, USA · Department of Radiology, New York University Langone Health, New York, 10016, NY, USA · Bernard and Irene Schwartz Center for Biomedical Imaging, New York University Langone Health, New York, 10016, NY, USA
This study assesses whether self-supervised learning (SSL) improves knee osteoarthritis (OA) modeling for diagnosis and prognosis relative to ImageNet-pretrained initialization. We compared (i) image-only SSL pretrained on knee radiographs from the OAI, MOST, and NYU cohorts, and (ii) multimodal image-text SSL pretrained on hospital knee radiographs paired with radiologist impressions. For diagnostic Kellgren-Lawrence (KL) grade prediction, SSL yielded mixed results. While image-only SSL improved accuracy during linear probing (frozen encoder), it did not outperform ImageNet pretraining during full fine-tuning. Similarly, multimodal SSL failed to improve grading performance. A likely explanation is mismatch between the hospital pretraining corpus and the downstream diagnostic task: the hospital image-text dataset was restricted to knees from patients with clinically identified OA in routine care, rather than a cohort spanning the full spectrum from normal to severe disease needed for balanced KL grading. In addition, radiology impressions do not explicitly encode KL grade, limiting supervision for learning KL-specific decision boundaries. In contrast, this same multimodal initialization significantly improved prognostic modeling. It outperformed ImageNet baselines in predicting 4-year structural incidence and progression, including on external validation (MOST AUROC: 0.701 vs. 0.599 at 10% labeled data). Overall, these results suggest that our hospital image-text data may be less effective for diagnostic grading when the pretraining cohort is limited to OA knees, but can provide a strong signal for prognostic modeling when the downstream task is better aligned with the pretraining data distribution.
Figures & tables
Men
Women
Dataset
Parameters
Patients
Controls
Patients
Controls
OAI
No. of patients
216
707
355
970
No. of scans
241
917
421
1348
Mean age (y)
62.3 ± 9.0
62.1 ± 9.3
63.7 ± 8.2
62.5 ± 8.9
Mean height (m)
1.8 ± 0.1
1.8 ± 0.1
1.6 ± 0.1
1.6 ± 0.1
Mean weight (kg)
93.1 ± 14.3
91.7 ± 14.4
80.8 ± 14.8
78.2 ± 14.5
Table 1: Patient Characteristics in Structural Progression Cohorts (OAI and MOST). Data are presented as mean ± standard deviation where applicable. BMI = Body Mass Index.
Men
Women
Dataset
Parameters
Patients
Controls
Patients
Controls
OAI
No. of patients
149
1103
271
1336
No. of scans
160
1737
296
2171
Mean age (y)
61.0 ± 8.7
59.6 ± 9.4
60.3 ± 8.6
60.5 ± 9.0
Mean height (m)
1.8 ± 0.1
1.8 ± 0.1
1.6 ± 0.1
1.6 ± 0.1
Mean weight (kg)
91.0 ± 14.9
91.0 ± 14.9
77.2 ± 14.2
71.3 ± 13.3
Table 2: Patient Characteristics in Structural Incidence Cohorts (OAI and MOST). Data are presented as mean ± standard deviation where applicable.
Figure 1: Overview of the Experimental Pipeline. The workflow consists of SSL pretraining followed by downstream fine-tuning. A critical feature of our protocol is the ”Sample partial data” step: when evaluating low-data regimes (e.g., 1% training data), we proportionally subsample both the training and validation sets for hyperparameter tuning. This ensures a realistic evaluation, contrasting with prior works that utilize full validation sets even when training data is scarce.
1% Data
5% Data
10% Data
Pretraining Approach
FT
LP
FT
LP
FT
LP
Random
0.397 (0.042)
0.200
0.555 (0.011)
0.200
0.619 (0.015)
0.200
Imgnet-pret
0.483 (0.040)
0.263
0.613 (0.014)
0.308
0.646 (0.009)
0.320
MoCo (Knee)
0.475 (0.050)
0.368
0.606 (0.010)
0.411
0.636 (0.012)
0.422
Barlow (Knee)
0.425 (0.063)
0.324
0.589 (0.018)
0.355
0.620 (0.010)
0.358
ViCReg (Knee)
0.471 (0.048)
0.336
0.585 (0.018)
0.374
0.635 (0.007)
0.373
Table 3: Image-Only SSL vs. ImageNet for Knee OA Diagnosis. KLG prediction performance (Balanced Accuracy) on OAI test set. Note that ImageNet pretraining (Imgnet-pret) consistently rivals or beats in-domain SSL under fine-tuning.
Figure 2: ImageNet scaling. Downstream KLG balanced accuracy on OAI versus ImageNet MoCo pretraining size (log-scale x-axis). Performance shows a plateau: reducing pretraining from 1.2M to 6k yields minimal changes, indicating that sheer natural-image scale does not resolve the medical task. Error bars denote reported standard deviations.
Figure 3: In-domain scaling (Fine-tuning). Downstream KLG balanced accuracy (FT) on OAI versus in-domain Knee MoCo pretraining size. Increasing in-domain pretraining data from 6k to 58k yields only marginal gains, consistent with a data-quality / signal bottleneck rather than data quantity. Error bars denote reported standard deviations.
Figure 4: Control Experiment: Imagenette Scaling. Unlike the Knee domain, SSL (MoCo) shows strong, consistent scaling on simpler object-centric images (Imagenette). This confirms the difficulty lies in the medical task complexity, not the SSL method.
Pretraining
1% (FT)
5% (FT)
10% (FT)
Imgnet-pret
0.619 (0.013)
0.699 (0.011)
0.732 (0.009)
Chest MoCo
0.572 (0.015)
0.646 (0.012)
0.680 (0.004)
Chest ViCReg
0.558 (0.018)
0.668 (0.006)
0.699 (0.004)
Table 4: Image-Only SSL Failure on Chest X-rays. Macro AUROC on NIH Chest X-ray 14. Similar to Knee OA, pixel-only pretraining does not consistently beat ImageNet.
1% Data
5% Data
10% Data
Approach
LP
FT
LP
FT
LP
FT
Imgnet
72.6 (0.8)
75.9 (1.8)
76.9 (0.9)
82.8 (1.8)
78.5 (0.7)
83.8 (1.4)
ConVIRT
83.5 (0.7)
79.5 (3.4)
85.1 (0.3)
85.7 (1.1)
85.5 (0.3)
86.2 (0.7)
Table 5: Multimodal SSL Success on Chest X-rays (Control Experiment). When paired with reports, SSL (ConVIRT) significantly outperforms ImageNet (macro AUROC), proving the pipeline works in a standard setting.
1% Data
5% Data
10% Data
Approach
FT
LP
FT
LP
FT
LP
Imgnet
0.483
0.268
0.613
0.308
0.646
0.320
ConVIRT
0.570
0.455
0.619
0.520
0.627
0.531
GLORIA
0.400
0.411
0.558
0.463
0.589
0.482
Table 6: Multimodal SSL Results for Knee Diagnosis. Balanced Accuracy on OAI. Unlike Chest, Multimodal SSL fails to improve Diagnosis due to the lack of healthy controls in the pretraining data.
Approach
5% Data
10% Data
25% Data
OAI
MOST
OAI
MOST
OAI
MOST
LP
FT
LP
FT
LP
FT
LP
FT
LP
FT
LP
FT
Imgnet
0.514
0.581
0.532
0.604
0.539
0.580
0.566
0.599
0.546
0.618
0.575
0.645
MoCo
0.572
0.577
0.583
0.601
0.590
0.603
0.612
0.636
0.620
0.633
0.661
0.679
Barlow
0.543
0.550
0.565
0.566
0.577
0.589
0.602
0.608
0.586
0.632
0.618
0.669
ViCReg
0.565
0.545
0.576
0.558
0.583
0.586
0.600
0.615
0.598
0.653
0.612
0.689
Table 7: Prognosis Results (Structural Progression and Incidence). AUROC on OAI (Internal) and MOST (External). Multimodal SSL (ConVIRT) consistently outperforms ImageNet and Image-only SSL (MoCo, Barlow, ViCReg) across all data fractions, approaching the Supervised Upper Bound (KL Pret).
Figure 5: Stratified prognostic performance by baseline severity (ImageNet vs. ConVIRT). Performance (AUROC) on the structural progression/incidence task stratified by baseline Kellgren-Lawrence (KL) grade in the external validation set (MOST). Multimodal pretraining (ConVIRT) consistently outperforms ImageNet across baseline severity strata, suggesting that the learned representation captures features relevant to future structural worsening across disease stages.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
1% Data
5% Data
10% Data
Pretraining Approach
FT
LP
FT
LP
FT
LP
Random
0.397 (0.042)
0.200
0.555 (0.011)
0.200
0.619 (0.015)
0.200
Imgnet-pret
0.483 (0.040)
0.263
0.613 (0.014)
0.308
0.646 (0.009)
0.320
In-Domain (Knee)
MoCo (Knee)
0.475 (0.050)
0.368
0.606 (0.010)
0.411
0.636 (0.012)
0.422
Barlow (Knee)
0.425 (0.063)
0.324
0.589 (0.018)
0.355
0.620 (0.010)
0.358
Appendix
Table A.1 : Extended KLG Diagnosis Results (OAI Test Set). Comparison of pretraining sources, including cross-domain (Chest) and mixed-domain (Knee+Chest) initialization. Adding Chest data to increase diversity (Knee+Chest) did not yield significant gains over ImageNet.
1% Data
5% Data
10% Data
Pretraining Approach
FT
LP
FT
LP
FT
LP
Imgnet-pret
0.619 (0.013)
0.583
0.699 (0.011)
0.614
0.732 (0.009)
0.633
In-Domain (Chest)
Chest MoCo
0.572 (0.015)
0.593
0.646 (0.012)
0.632
0.680 (0.004)
0.648
Chest Barlow
0.601 (0.018)
0.570
0.676 (0.006)
0.598
0.702 (0.006)
0.615
Cross-Domain (Knee)
Appendix
Table A.2 : Extended Chest Pathology Results (NIH Test Set). Cross-domain evaluation showing that Knee pretraining fails to transfer effectively to Chest X-ray tasks compared to ImageNet.
Figure B.1 : The Reconstruction Paradox (MAE Visualizations). Sample reconstructions with a masking rate of 0.6, showing the original input, masked view, model reconstruction, and overlay. The model demonstrates high fidelity in hallucinating anatomical structures from partial data, yet this structural understanding did not translate to improved KLG diagnostic performance.
1% Data
5% Data
10% Data
Pretraining Approach
FT
LP
FT
LP
FT
LP
Imgnet-pret
0.489 (0.052)
0.301 (0.017)
0.589 (0.013)
0.345 (0.013)
0.618 (0.013)
0.381 (0.010)
MAE 0.25
0.398 (0.098)
0.247 (0.011)
0.574 (0.012)
0.305 (0.005)
0.603 (0.011)
0.332 (0.005)
MAE 0.35
0.498 (0.024)
0.260 (0.011)
0.571 (0.012)
0.324 (0.006)
0.581 (0.021)
0.352 (0.005)
MAE 0.5
0.399 (0.095)
0.287 (0.013)
0.573 (0.007)
0.358 (0.012)
0.609 (0.009)
0.389 (0.012)
MAE 0.6
0.464 (0.066)
0.295 (0.012)
0.581 (0.017)
0.373 (0.015)
0.594 (0.012)
0.404 (0.017)
Appendix
Table B.1 : Masked Autoencoder (MAE) Pretraining Results. Comparison of ImageNet pretraining vs. MAE pretraining with varying masking rates. Performance is reported as Balanced Accuracy (mean (std)) on the OAI test set. Despite successful image reconstruction, diagnostic performance remains stagnant.
Knee osteoarthritis (KOA) is a degenerative joint disease that can lead to chronic pain, reduced mobility, and long-term disability. Automated severity grading from knee radiographs can support early assessment, but current methods heavily depend on large labeled datasets and remain sensitive to class imbalance, noisy samples, and variability in clinical annotations. To alleviate these limitations, we propose a Hierarchical fusion of Semi-Supervised framework with Self-Supervision (H-SemiS) for KOA severity grading in knee X-ray samples using limited annotated data. Rather than treating severity grading as a flat multi-class problem, H-SemiS decomposes the task into a sequence of binary sub-tasks within a semi-supervised teacher-student architecture, directly mitigating the impact of class imbalance. To further enhance feature learning from unlabeled data, the framework integrates an adversarial self-supervised reconstruction module that encourages the network to capture robust anatomical structures. In parallel, a teacher-student design with quantum-inspired feature mixing improves discrimination boundaries between adjacent grades when pseudo-labels are noisy. We comprehensively evaluate H-SemiS on two challenging multi-class datasets and assess its generalizability on two binary-class datasets. Our experimental results demonstrate the superiority of the proposed H-SemiS framework across multiple evaluation metrics, consistently outperforming several competing baselines and state-of-the-art methods. The code is publicly available at https://github.com/chandravardhan-singh-raghaw/H-SemiS.
Chandravardhan Singh Raghaw, Anushka Parwal, Shahid Shafi Dar +2
Department of Computer Science and Engineering, Indian Institute of Technology Indore, Indore 453552, India · Department of Computer Science and Engineering, National Institute of Technology Tiruchirappalli, Tiruchirappalli 620015, India
Knee osteoarthritis (OA) assessment involves a natural but often underused label hierarchy: a coarse binary OA decision and a fine-grained Kellgren--Lawrence (KL) severity grade. Existing deep learning studies commonly treat these targets as separate classification problems, either reducing OA assessment to disease presence or directly optimizing noisy ordinal KL labels. In this work, we ask whether this clinical hierarchy can serve as a representation-level supervisory prior. Rather than introducing a complex architecture, we use a deliberately simple dual-head model with a shared encoder and two task-specific heads as a probe of hierarchical supervision. We compare single-OA, single-KL, and dual-head training across multiple 3D backbones under the same test protocol. Beyond standard classification metrics, we perform paired statistical comparisons, analyze latent severity-axis geometry, and examine saliency overlap with cartilage regions. The results show that dual-head supervision produces backbone-dependent gains, with clear improvements in KL-related metrics for selected backbones. More importantly, the gains are accompanied by a more ordered coarse-to-fine latent organization and, for responsive backbones, stronger anatomical alignment of saliency with cartilage. These findings suggest that even simple hierarchical dual-head supervision can reshape disease representations under noisy coarse/fine labels, providing a useful inductive bias for OA diagnosis and severity grading.
Background and objective. Conventional Deep Learning (DL) approaches for Knee Osteoarthritis (KOA) grading rely on one-hot labels, which fail to capture both the ordinal uncertainty of Kellgren--Lawrence (KL) and Calcium Pyrophosphate Deposition Disease (CPPD) severity scores and the asymmetric relationship between the two scales observed in clinical practice. Methods. We retrospectively collected 2172 knee X-ray images, including 968 radiographs jointly annotated for KL and CPPD severity. An ordinal DL framework based on soft-labelling was developed for both tasks, replacing one-hot targets with unimodal probability distributions centred on the annotated grade. Four formulations were investigated: binomial, beta, triangular, and exponential. Results. All soft-labelling strategies consistently outperformed the nominal baseline. For CPPD grading, the triangular formulation achieved the highest Quadratic Weighted Kappa (QWK) and the lowest Mean Absolute Error (MAE) (QWK = 0.796; MAE = 0.438), while the beta formulation yielded the most balanced class-wise performance considering Average MAE (AMAE) and Maximum MAE (MMAE) across classes (AMAE = 0.458; MMAE = 0.573). For KL grading, the beta-based approach provided the best overall performance, achieving the highest QWK together with the lowest MAE and class-wise errors (QWK = 0.777; MAE = 0.529; AMAE = 0.523; MMAE = 0.775). Statistical analysis demonstrated significant improvements over conventional one-hot supervision (p < 0.001).
Francisco Bérchez-Moreno, Riccardo Rosati, Maria Chiara Fiorentino +6
Departamento de Ciencia de la Computación e Inteligencia Artificial, Universidad de Córdoba, Córdoba, España · Department of Political Science, Communication, and International Relations, University of Macerata, Macerata, Italy · Department of Innovative Technologies in Medicine & Dentistry, Università degli Studi "G. D’Annunzio" Chieti - Pescara, Chieti, Italy +5