Semantic Context Forests for Learning-Based Knee Cartilage Segmentation in 3D MR Images

Quan Wang; Dijia Wu; Le Lu; Meizhu Liu; Kim L. Boyer; Shaohua Kevin Zhou

cs.CVJul 11, 2013

Semantic Context Forests for Learning-Based Knee Cartilage Segmentation in 3D MR Images

Authors: Quan Wang, Dijia Wu, Le Lu, Meizhu Liu, Kim L. Boyer, Shaohua Kevin Zhou

Organizations: Siemens Corporate Research, Princeton, NJ 08540, USA · Rensselaer Polytechnic Institute, Troy, NY 12180, USA

Abstract

The automatic segmentation of human knee cartilage from 3D MR images is a useful yet challenging task due to the thin sheet structure of the cartilage with diffuse boundaries and inhomogeneous intensities. In this paper, we present an iterative multi-class learning method to segment the femoral, tibial and patellar cartilage simultaneously, which effectively exploits the spatial contextual constraints between bone and cartilage, and also between different cartilages. First, based on the fact that the cartilage grows in only certain area of the corresponding bone surface, we extract the distance features of not only to the surface of the bone, but more informatively, to the densely registered anatomical landmarks on the bone surface. Second, we introduce a set of iterative discriminative classifiers that at each iteration, probability comparison features are constructed from the class confidence maps derived by previously learned classifiers. These features automatically embed the semantic context information between different cartilages of interest. Validated on a total of 176 volumes from the Osteoarthritis Initiative (OAI) dataset, the proposed approach demonstrates high robustness and accuracy of segmentation in comparison with existing state-of-the-art MR cartilage segmentation methods. Code: https://github.com/wq2012/kneeseg Pretrained Models: https://huggingface.co/collections/wq2012/knee-mri-segmentation

Figures & tables

	Femur DSC (%)	Tibia DSC (%)	Patella DSC (%)
Before RW	$92.37\pm 1.58$	$94.64\pm 1.18$	$92.07\pm 1.47$
After RW	$94.86\pm 1.85$	$95.96\pm 1.64$	$94.31\pm 2.15$

Table 1: The Dice similarity coefficient (DSC) of bone segmentation results before and after random walks (3-fold cross validation on 176 OAI volumes).

Panel of Figure 1 — Figure 1: (a) The bone segmentation framework. (b) 3D anatomy of knee joint. (c) Example of a 2D MR slice [ 9 ] . (d) The semantic context forests diagram.

Panel of Figure 2 — Figure 2: Probability maps of femoral cartilage by semantic context forests. (a) Original image. (b) Prob. map of the 1st pass. (c) Prob. map of the 2nd pass. (d) Ground truth.

Panel of Figure 3 — Figure 3: (a) Distances to densely registered bone landmarks encode anatomical position of a voxel. (b) Feature scatter plot: intensity and distance features separate tibial cartilage from femoral and patellar cartilages. (c) Frequency of each feature selected by the classifier in the 2nd pass. (d) A comparison of segmentation performance (DSC): 1-pass/2-pass forests without using distance to landmark (LM) features; 1-pass/2-pass/3-pass forests using distance to landmark features; 2-pass forests with graph cuts optimization (3-fold cross validation).

Figure 4: Example bone segmentations. Each case has three views, from left to right: transversal plane, sagittal plane, coronal plane. Red: femur; green: tibia; blue: patella.

Figure 5: Examples cartilage segmentations compared with ground truth. Each case has six images: segmentation results in upper row, ground truth in lower row. Red: femoral cartilage; green: tibial cartilage; blue: patellar cartilage.

		Fem. Cart. DSC		Tib. Cart. DSC		Pat. Cart. DSC
Author	Dataset	Mean	Std.	Mean	Std.	Mean	Std.
Shan [ 16 ]	18 SPGR images	78.2%	5.2%	82.6%	3.8%	–	–
Folkesson [ 5 ]	139 Esaote C-Span images	77%	8.0%	81%	6.0%	–	–
Fripp [ 6 ]	20 FS SPGR images	84.8%	7.6%	82.6%	8.3%	83.3%	13.5%
Lee [ 9 ]	10 images in OAI	82.5%	–	80.8%	–	82.1%	–
Yin [ 21 ]	60 images in OAI	84%	4%	80%	4%	80%	4%

Table 2: Performance of our method compared with other state-of-the-art cartilage segmentation methods: mean DSC and standard deviation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Resource	Location / Details
GitHub Repository	https://github.com/wq2012/kneeseg
PyPI Package	kneeseg (v0.2.5) — https://pypi.org/project/kneeseg/
HF Model Collection	https://huggingface.co/collections/wq2012/knee-mri-segmentation
License	MIT License (source code and all five Hugging Face pretrained models)
Core Dependencies	numpy , scipy , scikit-learn , scikit-image , SimpleITK , ml_dtypes , pycpd , joblib , threadpoolctl , tqdm , jsonschema

Appendix

Table 3: Summary of open-source resources for kneeseg .

Pipeline Stage	Original 2013 Paper (Siemens C++)	Open-Source kneeseg (Python v0.2.5)
Bone Segmentation (Section 2 )	3-step mesh pipeline: Marginal Space Learning (MSL) pose estimation $(\hat{t},\hat{r},\hat{s})$ , Active Shape Model (ASM) mesh deformation with Coherent Point Drift (CPD) correspondences, and 3D Random Walks (RW) refinement.	Self-contained 2-Pass Physical-Space Auto-Context Forest ( bone_rf_p1 , bone_rf_p2 ) + Binary Subchondral Articular Surface Refiner ( RefBone V2 , blended with RefBone V1 on SKI10) with 1D cortical enclosure rays, 3D bone atlases, Patellofemoral Specialist, and shaft hole-filling cleanup. Standalone CPD and 3D RW modules also included.
Bone Distance Features ( $f_{3}$ – $f_{9}$ , Section 3 )	Signed Euclidean distances $d_{F},d_{T},d_{P}$ to refined bone meshes and sum/difference combinations $d_{F}\pm d_{T}$ , $d_{F}\pm d_{P}$ .	Exact physical-millimeter Signed Maurer Distance Transforms ( SimpleITK ) $d_{F},d_{T},d_{P}$ and linear combinations $d_{F}\pm d_{T}$ , $d_{F}\pm d_{P}$ computed directly from predicted 3D bone masks.
Anatomical Localization ( $f_{10}$ , Section 3 )	Euclidean distances $\\|\mathbf{x}-\mathbf{z}_{\zeta}\\|$ to densely registered CPD bone mesh points $\mathbf{z}_{\zeta}$ .	Deterministic quantile-based 18 articular surface landmarks per bone ( $3\times 3$ condylar/plateau grids) + multi-resolution 4D Articular Coordinate Probabilistic Atlases (8 tables on SKI10, 11 on OAI) built via Leave-One-Out (LOO) cross-validation.
Appearance & Context Features ( $f_{1},f_{2},f_{11}$ )	Raw intensity $I(\mathbf{x})$ , gradient magnitude $\\|\nabla I(\mathbf{x})\\|$ , and voxel-offset Random Shift Intensity Differences (RSID) $I(\mathbf{x}+\mathbf{u})-I(\mathbf{x})$ .	ROI interquartile ( $p_{10},p_{50},p_{90}$ ) normalized intensity, multi-scale physical-mm Gaussian, gradient, Laplacian, and structure-tensor features, physical-mm RSID, and 1D surface-normal ray profiles ( $s\in[-3.6,+5.6]\,\mathrm{mm}$ ) along $\nabla d_{\mathrm{bone}}$ .
Iterative Semantic Context Forests ( $f_{12}$ – $f_{17}$ , Section 3 )	2-pass custom C++ Random Forest (60 trees, depth 18, 1000 node candidates). Pass 2 extracts class probabilities $P_{F},P_{T},P_{P}$ and RSPD features from in-sample Pass-1 predictions.	4-Class Compartment-Specialist Ensemble ( ExtraTrees + HistGradientBoosting ) predicting { Background , Subchondral Bone/Meniscus , Target Cart. , Opposing Cart. }. Pass-2 RSPD and normal-ray probability profiles are trained via 4-fold Out-Of-Fold (OOF) stacking.
Post-Processing (Section 3 )	Multi-label Graph Cuts energy minimization $E(L)=\sum V_{\mathbf{x},\mathbf{y}}+\sum D_{\mathbf{x}}$ .	4-fold OOF Stage-3 Subchondral Boundary Refiner (multi-family 103–167 features on SKI10; 101-feature specialist on OAI) + OOF Ridge Threshold & Bone-Penalty Calibrator $(\mathrm{th},\mathrm{bp})$ + physical-space Gaussian smoothing + contact resolution and component filtering.

Appendix

Table 4: Comparison between the original 2013 paper implementation and the open-source kneeseg (v0.2.5) reproduction.

Module Path	Description
kneeseg/bone_rf.py	BoneClassifier implementing the 2-pass physical-space bone auto-context forest and binary RefBone V1 / V2 subchondral surface refiner for SKI10.
kneeseg/cartilage_v6.py	8-table 4D Articular Coordinate Probabilistic Atlases, 4-class femorotibial cartilage ensembles, multi-family (103–167 feature) Stage-3 boundary refiner, and OOF Ridge calibrator.
kneeseg/rf_seg.py	CartilageClassifier orchestrating 2-pass iterative semantic context cartilage training and inference across SKI10, OAI, and unified models.
kneeseg/oai_subchondral_atlas.py	6-structure subchondral atlas pipeline for OAI, Patellofemoral Specialist, C/Fortran layout standardization, and unified SKI10+OAI domain router.
kneeseg/features.py	Base intensity ( $f_{1}$ ), gradient magnitude ( $f_{2}$ ), signed Maurer distance transforms ( $f_{3}$ – $f_{9}$ ), landmark distances ( $f_{10}$ ), RSID ( $f_{11}$ ), and RSPD ( $f_{12}$ – $f_{17}$ ) extractors.
kneeseg/io.py	3D medical volume I/O ( load_volume , save_volume , OAIImageReader ) for MetaImage ( .mhd / .raw ), NIfTI, and OAI ( .hdr / .img ) formats.

Appendix

Table 5: Overview of primary modules in the kneeseg open-source repository.

Hugging Face Model ID	Architecture	Train Set	Size	Target Structures & Domain
wq2012/knee_3d_mri_segmentation_SKI10_subchondral_atlas	Subchondral Atlas + Stage-3 OOF	80 SKI10	1.56 GB	4 labels (Femur, Tibia, Fem. Cart., Tib. Cart.) on high-res SKI10 ( $\sim 1.5\times 0.4\times 0.4\,\mathrm{mm}$ ).
wq2012/knee_3d_mri_segmentation_OAI_downsampled_subchondral_atlas	Subchondral Atlas + RefBone V2	128 OAI	1.33 GB	6 labels (Femur, Tibia, Patella, Fem., Tib., and Pat. Cart.) on isotropic OAI ( $1.0\,\mathrm{mm}^{3}$ ).
wq2012/knee_3d_mri_segmentation_SKI10_OAI_subchondral_atlas	Unified Dual-Domain Cascade	80 SKI10 + 128 OAI	2.93 GB	Unified 4-label SKI10 and 6-label OAI segmentation with automatic domain routing and 208-scan atlas regularization.
wq2012/knee_3d_mri_segmentation_SKI10	Baseline v1 Cascade	100 SKI10	5.67 GB	4 labels on SKI10 (initial baseline without 4D articular atlases or OOF stacking).
wq2012/knee_3d_mri_segmentation_OAI_downsampled	Baseline v1 Cascade	128 OAI	6.44 GB	6 labels on downsampled OAI (initial baseline prior to layout standardization and 4D atlases).

Appendix

Table 6: Summary of the five open-source pretrained models hosted on Hugging Face ( wq2012/* , all under the MIT License).

Model Architecture / Checkpoint ( wq2012/* )	Femur	Tibia	Fem. Cart.	Tib. Cart.	Overall Mean
Baseline v1 Architecture ( ..._SKI10 , 80/20 split)	$90.46\pm 3.61$	$92.92\pm 2.60$	$67.67\pm 4.81$	$64.11\pm 5.40$	$78.79$
..._SKI10_subchondral_atlas	$\mathbf{96.89\pm 1.61}$	$\mathbf{97.42\pm 0.80}$	$\mathbf{80.26\pm 5.04}$	$\mathbf{80.01\pm 4.04}$	$\mathbf{88.65}$
..._SKI10_OAI_subchondral_atlas	$\mathbf{96.89\pm 1.61}$	$\mathbf{97.42\pm 0.80}$	$80.25\pm 5.05$	$79.97\pm 4.03$	$88.63$

Appendix

Table 7: Open-source segmentation performance (Mean DSC

\pm

Std [

\mathrm{ddof}=0

], %) on the 20 held-out SKI10 evaluation scans defined in ski10_full_split.json . For SKI10_subchondral_atlas , unbiased sample standard deviations (

\mathrm{ddof}=1

) are

\pm 1.65\%

(Femur),

\pm 0.82\%

(Tibia),

\pm 5.17\%

(Fem. Cart.), and

\pm 4.14\%

(Tib. Cart.), with 95% bootstrap CIs (

B=10{,}000

) of

[96.14,97.52]\%

[97.05,97.76]\%

[77.88,82.31]\%

, and

[78.16,81.67]\%

, respectively.

Method / Pretrained Model	Femur	Tibia	Patella	Fem. Cart.	Tib. Cart.	Pat. Cart.
Original Paper (Before RW, $0.37^{2}\times 0.7\,\mathrm{mm}$ )	$92.37\pm 1.58$	$94.64\pm 1.18$	$92.07\pm 1.47$	—	—	—
Original Paper (Full Pipeline, $0.37^{2}\times 0.7\,\mathrm{mm}$ )	$94.86\pm 1.85$	$95.96\pm 1.64$	$94.31\pm 2.15$	$84.96\pm 3.30$	$83.74\pm 4.00$	$79.16\pm 8.88$
..._OAI_downsampled (Baseline v1, $1.0\,\mathrm{mm}^{3}$ )	$71.30\pm 6.73$	$75.45\pm 5.98$	$52.09\pm 8.31$	$51.71\pm 7.16$	$41.34\pm 8.88$	$36.33\pm 14.06$
..._OAI_downsampled_subchondral_atlas ( $N=31$ all)	$94.08\pm 3.39$	$96.69\pm 1.29$	$\mathbf{96.39\pm 2.10}$	$\mathbf{88.11\pm 1.88}$	$86.98\pm 2.30$	$\mathbf{82.63\pm 6.77}$
$\hookrightarrow$ Seen-subject companion visits ( $N=28$ , $\mathrm{ddof}=1$ )	$93.78\pm 3.48$	$96.78\pm 1.35$	$96.40\pm 2.23$	$88.25\pm 1.91$	$87.05\pm 2.39$	$82.44\pm 7.20$
$\hookrightarrow$ Strictly unseen subjects ( $N=3$ , $\mathrm{ddof}=1$ )	$96.91\pm 1.30$	$95.85\pm 0.29$	$96.29\pm 1.23$	$86.79\pm 1.57$	$86.30\pm 2.05$	$84.35\pm 2.34$

Appendix

Table 8: Comparison of bone and cartilage segmentation performance (Mean DSC

\pm

Std, %) on the OAI dataset between the original 2013 paper (Table 1 and Table 2 , subject-grouped 3-fold CV on 176 full-resolution

0.365\times 0.365\times 0.7\,\mathrm{mm}^{3}

scans) and the open-source kneeseg models on the 31 held-out downsampled

1.0\times 1.0\times 1.0\,\mathrm{mm}^{3}

OAI test scans (

\mathrm{ddof}=0

). For OAI_downsampled_subchondral_atlas (

N=31

), unbiased

\mathrm{ddof}=1

standard deviations are

\pm 3.45\%

\pm 1.31\%

\pm 2.14\%

\pm 1.91\%

\pm 2.34\%

, and

\pm 6.88\%

, with 95% bootstrap CIs (

B=10{,}000

) of

[92.84,95.23]\%

(Femur),

[96.23,97.14]\%

(Tibia),

[95.61,97.09]\%

(Patella),

[87.46,88.77]\%

(Fem. Cart.),

[86.15,87.76]\%

(Tib. Cart.), and

[80.16,84.91]\%

(Pat. Cart.).

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Reducing Annotation Burden for Femoral Cartilage Segmentation in Knee MRI via Cross-Sequence Transfer Learning
May 9, 2026Francesco Chiumento, Gianluigi Crimi, Elisa Moretta +7Magnetic Resonance Imaging Transfer Learning
School of Electronic Engineering, Dublin City University, Dublin, Ireland · Bioengineering and Computing Laboratory, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy · Dipartimento di Scienze Mediche e Chirurgiche (DIMEC), Alma Mater Studiorum – Università di Bologna, Bologna, Italy +4
Multitask Conditional Generative Adversarial Network Enables Automatic Whole Knee Cartilage and Menisci Segmentation and Reliable $T_{1ρ}$ and $T_2$ Quantification Without High-Resolution Morphological Images
Oct 5, 2026Ahmed Tahseen Minhaz, Richard Lartey, Zhiyuan Zhang +8Multi-Task Learning Magnetic Resonance Imaging
Program of Advanced Musculoskeletal Imaging (PAMI), Cleveland Clinic · Department of Biomedical Engineering, Cleveland Clinic Research, Cleveland Clinic · Department of Mathematics, Case Western Reserve University +1
KneePreM: Towards 3D Knee MRI Foundation Models via Large-Scale Unlabeled Pretraining and Label-Efficient Fine-Tuning
Sep 25, 2026Xinxin Wang, Liam Hazan, Jing Li +3Medical Imaging Foundation Models Self-Supervised Pre-Training
Program of Advanced Musculoskeletal Imaging, Cleveland Clinic, 9500 Euclid Avenue, Cleveland, OH 44195, United States · Department of Biomedical Engineering, Lerner Research Institute, Cleveland Clinic, 9500 Euclid Avenue, Cleveland, OH 44195, United States · Department of Computer and Data Science, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH 44106, United States +2

May 9, 2026J/K move · Enter open · S save

Reducing Annotation Burden for Femoral Cartilage Segmentation in Knee MRI via Cross-Sequence Transfer Learning

Francesco Chiumento, Gianluigi Crimi, Elisa Moretta, Rocco Milieri+6

School of Electronic Engineering, Dublin City University, Dublin, Ireland · Bioengineering and Computing Laboratory, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy · Dipartimento di Scienze Mediche e Chirurgiche (DIMEC), Alma Mater Studiorum – Università di Bologna, Bologna, Italy+4

Magnetic Resonance Imaging Transfer Learning Medical Image Segmentation

Open paper AskPDF arXiv

Purpose: To develop and evaluate cross-sequence transfer learning for automatic femoral cartilage segmentation, testing bidirectional transfer between dual-echo steady-state (DESS) and sagittal proton density-weighted 3D fast spin-echo (Cube) sequences. Materials and Methods: We optimized a modified 2D U-Net on 507 DESS images from the Osteoarthritis Initiative (OAI). We then established same-sequence baselines using subject-level cross-validation on a subset of 44 OAI DESS images and 44 Cube images acquired at the Istituto Ortopedico Rizzoli, Bologna, Italy. Each subset included 22 non-lesioned and 22 lesioned subjects. Finally, we performed transfer learning across sequences by fine-tuning the pretrained models on the target sequence with increasing training set sizes to study convergence, while keeping validation and test sets fixed. Segmentations were evaluated using Dice similarity coefficient (DSC) and average surface distance (ASD). Lesion effects were assessed with two-sided Mann-Whitney U tests with Bonferroni correction. Results: Same-sequence training yielded higher accuracy on DESS than Cube (DSC,

0.900

0.830

;

P < .001

). Cube-to-DESS transfer matched DESS performance (DSC,

0.903 \pm 0.032

0.900 \pm 0.027

), reaching a performance plateau at 9 training subjects. DESS-to-Cube yielded a lower combined DSC (

0.802 \pm 0.049

0.830 \pm 0.042

), reaching a plateau at 24 training subjects. Lesions did not affect DESS (

P \ge .39

) but reduced Cube accuracy (DSC,

0.805

0.856

;

P < .001

). Conclusion: Transfer learning across sequences can substantially reduce target-sequence annotation requirements for femoral cartilage segmentation, but performance is direction- and sequence-dependent, and the effects of lesions on segmentation may vary across MRI sequences.

Abstract

Medical Image Segmentation

Explore similar work

Reducing Annotation Burden for Femoral Cartilage Segmentation in Knee MRI via Cross-Sequence Transfer Learning

Multitask Conditional Generative Adversarial Network Enables Automatic Whole Knee Cartilage and Menisci Segmentation and Reliable T1ρT_{1ρ}T1ρ​ and T2T_2T2​ Quantification Without High-Resolution Morphological Images

KneePreM: Towards 3D Knee MRI Foundation Models via Large-Scale Unlabeled Pretraining and Label-Efficient Fine-Tuning

Multitask Conditional Generative Adversarial Network Enables Automatic Whole Knee Cartilage and Menisci Segmentation and Reliable $T_{1ρ}$ and $T_2$ Quantification Without High-Resolution Morphological Images