Self-Supervised Learning for Knee Osteoarthritis: Diagnostic Limitations and Prognostic Value of Hospital Data

Haresh Rengaraj Rajamohan; Yuxuan Chen; Kyunghyun Cho; Cem M. Deniz

cs.CVMar 26, 2026

Self-Supervised Learning for Knee Osteoarthritis: Diagnostic Limitations and Prognostic Value of Hospital Data

Authors: Haresh Rengaraj Rajamohan, Yuxuan Chen, Kyunghyun Cho, Cem M. Deniz

Organizations: Center for Data Science, New York University, New York, 10011, NY, United States · Perlmutter Cancer Center, New York University Langone Health, New York, 10016, NY, USA · Department of Radiology, New York University Langone Health, New York, 10016, NY, USA · Bernard and Irene Schwartz Center for Biomedical Imaging, New York University Langone Health, New York, 10016, NY, USA

Abstract

This study assesses whether self-supervised learning (SSL) improves knee osteoarthritis (OA) modeling for diagnosis and prognosis relative to ImageNet-pretrained initialization. We compared (i) image-only SSL pretrained on knee radiographs from the OAI, MOST, and NYU cohorts, and (ii) multimodal image-text SSL pretrained on hospital knee radiographs paired with radiologist impressions. For diagnostic Kellgren-Lawrence (KL) grade prediction, SSL yielded mixed results. While image-only SSL improved accuracy during linear probing (frozen encoder), it did not outperform ImageNet pretraining during full fine-tuning. Similarly, multimodal SSL failed to improve grading performance. A likely explanation is mismatch between the hospital pretraining corpus and the downstream diagnostic task: the hospital image-text dataset was restricted to knees from patients with clinically identified OA in routine care, rather than a cohort spanning the full spectrum from normal to severe disease needed for balanced KL grading. In addition, radiology impressions do not explicitly encode KL grade, limiting supervision for learning KL-specific decision boundaries. In contrast, this same multimodal initialization significantly improved prognostic modeling. It outperformed ImageNet baselines in predicting 4-year structural incidence and progression, including on external validation (MOST AUROC: 0.701 vs. 0.599 at 10% labeled data). Overall, these results suggest that our hospital image-text data may be less effective for diagnostic grading when the pretraining cohort is limited to OA knees, but can provide a strong signal for prognostic modeling when the downstream task is better aligned with the pretraining data distribution.

Figures & tables

		Men		Women
Dataset	Parameters	Patients	Controls	Patients	Controls
OAI	No. of patients	216	707	355	970
	No. of scans	241	917	421	1348
	Mean age (y)	62.3 $\pm$ 9.0	62.1 $\pm$ 9.3	63.7 $\pm$ 8.2	62.5 $\pm$ 8.9
	Mean height (m)	1.8 $\pm$ 0.1	1.8 $\pm$ 0.1	1.6 $\pm$ 0.1	1.6 $\pm$ 0.1
	Mean weight (kg)	93.1 $\pm$ 14.3	91.7 $\pm$ 14.4	80.8 $\pm$ 14.8	78.2 $\pm$ 14.5

Table 1: Patient Characteristics in Structural Progression Cohorts (OAI and MOST). Data are presented as mean

\pm

standard deviation where applicable. BMI = Body Mass Index.

		Men		Women
Dataset	Parameters	Patients	Controls	Patients	Controls
OAI	No. of patients	149	1103	271	1336
	No. of scans	160	1737	296	2171
	Mean age (y)	61.0 $\pm$ 8.7	59.6 $\pm$ 9.4	60.3 $\pm$ 8.6	60.5 $\pm$ 9.0
	Mean height (m)	1.8 $\pm$ 0.1	1.8 $\pm$ 0.1	1.6 $\pm$ 0.1	1.6 $\pm$ 0.1
	Mean weight (kg)	91.0 $\pm$ 14.9	91.0 $\pm$ 14.9	77.2 $\pm$ 14.2	71.3 $\pm$ 13.3

Table 2: Patient Characteristics in Structural Incidence Cohorts (OAI and MOST). Data are presented as mean

\pm

standard deviation where applicable.

Figure 1: Overview of the Experimental Pipeline. The workflow consists of SSL pretraining followed by downstream fine-tuning. A critical feature of our protocol is the ”Sample partial data” step: when evaluating low-data regimes (e.g., 1% training data), we proportionally subsample both the training and validation sets for hyperparameter tuning. This ensures a realistic evaluation, contrasting with prior works that utilize full validation sets even when training data is scarce.

	1% Data		5% Data		10% Data
Pretraining Approach	FT	LP	FT	LP	FT	LP
Random	0.397 (0.042)	0.200	0.555 (0.011)	0.200	0.619 (0.015)	0.200
Imgnet-pret	0.483 (0.040)	0.263	0.613 (0.014)	0.308	0.646 (0.009)	0.320
MoCo (Knee)	0.475 (0.050)	0.368	0.606 (0.010)	0.411	0.636 (0.012)	0.422
Barlow (Knee)	0.425 (0.063)	0.324	0.589 (0.018)	0.355	0.620 (0.010)	0.358
ViCReg (Knee)	0.471 (0.048)	0.336	0.585 (0.018)	0.374	0.635 (0.007)	0.373

Table 3: Image-Only SSL vs. ImageNet for Knee OA Diagnosis. KLG prediction performance (Balanced Accuracy) on OAI test set. Note that ImageNet pretraining (Imgnet-pret) consistently rivals or beats in-domain SSL under fine-tuning.

Figure 2: ImageNet scaling. Downstream KLG balanced accuracy on OAI versus ImageNet MoCo pretraining size (log-scale x-axis). Performance shows a plateau: reducing pretraining from 1.2M to 6k yields minimal changes, indicating that sheer natural-image scale does not resolve the medical task. Error bars denote reported standard deviations.

Figure 3: In-domain scaling (Fine-tuning). Downstream KLG balanced accuracy (FT) on OAI versus in-domain Knee MoCo pretraining size. Increasing in-domain pretraining data from 6k to 58k yields only marginal gains, consistent with a data-quality / signal bottleneck rather than data quantity. Error bars denote reported standard deviations.

Figure 4: Control Experiment: Imagenette Scaling. Unlike the Knee domain, SSL (MoCo) shows strong, consistent scaling on simpler object-centric images (Imagenette). This confirms the difficulty lies in the medical task complexity, not the SSL method.

Pretraining	1% (FT)	5% (FT)	10% (FT)
Imgnet-pret	0.619 (0.013)	0.699 (0.011)	0.732 (0.009)
Chest MoCo	0.572 (0.015)	0.646 (0.012)	0.680 (0.004)
Chest ViCReg	0.558 (0.018)	0.668 (0.006)	0.699 (0.004)

Table 4: Image-Only SSL Failure on Chest X-rays. Macro AUROC on NIH Chest X-ray 14. Similar to Knee OA, pixel-only pretraining does not consistently beat ImageNet.

	1% Data		5% Data		10% Data
Approach	LP	FT	LP	FT	LP	FT
Imgnet	72.6 (0.8)	75.9 (1.8)	76.9 (0.9)	82.8 (1.8)	78.5 (0.7)	83.8 (1.4)
ConVIRT	83.5 (0.7)	79.5 (3.4)	85.1 (0.3)	85.7 (1.1)	85.5 (0.3)	86.2 (0.7)

Table 5: Multimodal SSL Success on Chest X-rays (Control Experiment). When paired with reports, SSL (ConVIRT) significantly outperforms ImageNet (macro AUROC), proving the pipeline works in a standard setting.

	1% Data		5% Data		10% Data
Approach	FT	LP	FT	LP	FT	LP
Imgnet	0.483	0.268	0.613	0.308	0.646	0.320
ConVIRT	0.570	0.455	0.619	0.520	0.627	0.531
GLORIA	0.400	0.411	0.558	0.463	0.589	0.482

Table 6: Multimodal SSL Results for Knee Diagnosis. Balanced Accuracy on OAI. Unlike Chest, Multimodal SSL fails to improve Diagnosis due to the lack of healthy controls in the pretraining data.

Approach	5% Data				10% Data				25% Data
	OAI		MOST		OAI		MOST		OAI		MOST
	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT
Imgnet	0.514	0.581	0.532	0.604	0.539	0.580	0.566	0.599	0.546	0.618	0.575	0.645
MoCo	0.572	0.577	0.583	0.601	0.590	0.603	0.612	0.636	0.620	0.633	0.661	0.679
Barlow	0.543	0.550	0.565	0.566	0.577	0.589	0.602	0.608	0.586	0.632	0.618	0.669
ViCReg	0.565	0.545	0.576	0.558	0.583	0.586	0.600	0.615	0.598	0.653	0.612	0.689

Table 7: Prognosis Results (Structural Progression and Incidence). AUROC on OAI (Internal) and MOST (External). Multimodal SSL (ConVIRT) consistently outperforms ImageNet and Image-only SSL (MoCo, Barlow, ViCReg) across all data fractions, approaching the Supervised Upper Bound (KL Pret).

Panel of Figure 5 — Figure 5: Stratified prognostic performance by baseline severity (ImageNet vs. ConVIRT). Performance (AUROC) on the structural progression/incidence task stratified by baseline Kellgren-Lawrence (KL) grade in the external validation set (MOST). Multimodal pretraining (ConVIRT) consistently outperforms ImageNet across baseline severity strata, suggesting that the learned representation captures features relevant to future structural worsening across disease stages.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	1% Data		5% Data		10% Data
Pretraining Approach	FT	LP	FT	LP	FT	LP
Random	0.397 (0.042)	0.200	0.555 (0.011)	0.200	0.619 (0.015)	0.200
Imgnet-pret	0.483 (0.040)	0.263	0.613 (0.014)	0.308	0.646 (0.009)	0.320
In-Domain (Knee)
MoCo (Knee)	0.475 (0.050)	0.368	0.606 (0.010)	0.411	0.636 (0.012)	0.422
Barlow (Knee)	0.425 (0.063)	0.324	0.589 (0.018)	0.355	0.620 (0.010)	0.358

Appendix

Table A.1 : Extended KLG Diagnosis Results (OAI Test Set). Comparison of pretraining sources, including cross-domain (Chest) and mixed-domain (Knee+Chest) initialization. Adding Chest data to increase diversity (Knee+Chest) did not yield significant gains over ImageNet.

	1% Data		5% Data		10% Data
Pretraining Approach	FT	LP	FT	LP	FT	LP
Imgnet-pret	0.619 (0.013)	0.583	0.699 (0.011)	0.614	0.732 (0.009)	0.633
In-Domain (Chest)
Chest MoCo	0.572 (0.015)	0.593	0.646 (0.012)	0.632	0.680 (0.004)	0.648
Chest Barlow	0.601 (0.018)	0.570	0.676 (0.006)	0.598	0.702 (0.006)	0.615
Cross-Domain (Knee)

Appendix

Table A.2 : Extended Chest Pathology Results (NIH Test Set). Cross-domain evaluation showing that Knee pretraining fails to transfer effectively to Chest X-ray tasks compared to ImageNet.

Figure B.1 : The Reconstruction Paradox (MAE Visualizations). Sample reconstructions with a masking rate of 0.6, showing the original input, masked view, model reconstruction, and overlay. The model demonstrates high fidelity in hallucinating anatomical structures from partial data, yet this structural understanding did not translate to improved KLG diagnostic performance.

	1% Data		5% Data		10% Data
Pretraining Approach	FT	LP	FT	LP	FT	LP
Imgnet-pret	0.489 (0.052)	0.301 (0.017)	0.589 (0.013)	0.345 (0.013)	0.618 (0.013)	0.381 (0.010)
MAE 0.25	0.398 (0.098)	0.247 (0.011)	0.574 (0.012)	0.305 (0.005)	0.603 (0.011)	0.332 (0.005)
MAE 0.35	0.498 (0.024)	0.260 (0.011)	0.571 (0.012)	0.324 (0.006)	0.581 (0.021)	0.352 (0.005)
MAE 0.5	0.399 (0.095)	0.287 (0.013)	0.573 (0.007)	0.358 (0.012)	0.609 (0.009)	0.389 (0.012)
MAE 0.6	0.464 (0.066)	0.295 (0.012)	0.581 (0.017)	0.373 (0.015)	0.594 (0.012)	0.404 (0.017)

Appendix

Table B.1 : Masked Autoencoder (MAE) Pretraining Results. Comparison of ImageNet pretraining vs. MAE pretraining with varying masking rates. Performance is reported as Balanced Accuracy (mean (std)) on the OAI test set. Despite successful image reconstruction, diagnostic performance remains stagnant.

Self-Supervised Learning

Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.

See all 258 papers

H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading
Apr 25, 2026Chandravardhan Singh Raghaw, Anushka Parwal, Shahid Shafi Dar +2Class-Imbalanced Learning Self-Supervised Learning
Department of Computer Science and Engineering, Indian Institute of Technology Indore, Indore 453552, India · Department of Computer Science and Engineering, National Institute of Technology Tiruchirappalli, Tiruchirappalli 620015, India
Learning Coarse-to-Fine Osteoarthritis Representations under Noisy Hierarchical Labels
May 1, 2026Tongxu ZhangHierarchical Classification Hierarchical Representation Learning
The Hong Kong Polytechnic University
From Kellgren-Lawrence to Calcium Pyrophosphate Crystal Deposition: A Soft-Labelling Framework for Knee Osteoarthritis Assessmen
May 27, 2026Francisco Bérchez-Moreno, Riccardo Rosati, Maria Chiara Fiorentino +6Label Distribution Learning Medical Image Classification
Departamento de Ciencia de la Computación e Inteligencia Artificial, Universidad de Córdoba, Córdoba, España · Department of Political Science, Communication, and International Relations, University of Macerata, Macerata, Italy · Department of Innovative Technologies in Medicine & Dentistry, Università degli Studi "G. D’Annunzio" Chieti - Pescara, Chieti, Italy +5

Apr 25, 2026J/K move · Enter open · S save

H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading

Chandravardhan Singh Raghaw, Anushka Parwal, Shahid Shafi Dar, Prajakta Darade+1

Department of Computer Science and Engineering, Indian Institute of Technology Indore, Indore 453552, India · Department of Computer Science and Engineering, National Institute of Technology Tiruchirappalli, Tiruchirappalli 620015, India

Class-Imbalanced Learning Self-Supervised Learning Medical Image Classification Semi-Supervised Learning

Open paper AskPDF arXiv

Knee osteoarthritis (KOA) is a degenerative joint disease that can lead to chronic pain, reduced mobility, and long-term disability. Automated severity grading from knee radiographs can support early assessment, but current methods heavily depend on large labeled datasets and remain sensitive to class imbalance, noisy samples, and variability in clinical annotations. To alleviate these limitations, we propose a Hierarchical fusion of Semi-Supervised framework with Self-Supervision (H-SemiS) for KOA severity grading in knee X-ray samples using limited annotated data. Rather than treating severity grading as a flat multi-class problem, H-SemiS decomposes the task into a sequence of binary sub-tasks within a semi-supervised teacher-student architecture, directly mitigating the impact of class imbalance. To further enhance feature learning from unlabeled data, the framework integrates an adversarial self-supervised reconstruction module that encourages the network to capture robust anatomical structures. In parallel, a teacher-student design with quantum-inspired feature mixing improves discrimination boundaries between adjacent grades when pseudo-labels are noisy. We comprehensively evaluate H-SemiS on two challenging multi-class datasets and assess its generalizability on two binary-class datasets. Our experimental results demonstrate the superiority of the proposed H-SemiS framework across multiple evaluation metrics, consistently outperforming several competing baselines and state-of-the-art methods. The code is publicly available at https://github.com/chandravardhan-singh-raghaw/H-SemiS.

Abstract

Self-Supervised Learning

Explore similar work

H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading

Learning Coarse-to-Fine Osteoarthritis Representations under Noisy Hierarchical Labels

From Kellgren-Lawrence to Calcium Pyrophosphate Crystal Deposition: A Soft-Labelling Framework for Knee Osteoarthritis Assessmen