Evaluation of the consistency of the MRI- based AI segmentation cartilage model using the natural tibial plateau cartilage

Knee osteoarthritis (KOA) is a prevalent type of degenerative joint disease with multiple contributing factors, potentially resulting in functional impairment [1]. Early detection and accurate measurement of cartilage morphology and composition changes are crucial for effectively treating KOA [2]. Magnetic resonance imaging (MRI) provides enhanced visualization of cartilage morphology, making it the preferred modality for assessing cartilage damage and accurately detecting morphological changes [3]. Numerous observational and interventional studies [4‐6] have used semi-quantitative and quantitative evaluations of cartilage morphology as an outcome measure because it is sensitive to OA-related change and therapy interventions [7]. Semi-quantitative morphometric assessments are carried out using scoring systems like the Knee Osteoarthritis Scoring Systems (KOSS) [8], the Boston Leeds Osteoarthritis Knee Score (BLOKS) [9], MRI Osteoarthritis Knee Score (MOAKS) [10], and the Whole-Organ MRI Score (WORMS) [11]. Quantitative morphometric techniques utilize three-dimensional MRI data to measure parameters of cartilage tissue, such as cartilage volume, cartilage surface area, total subchondral bone area, and cartilage thickness as biomarkers of osteoarthritis severity and progression [12].

Segmenting the articular cartilage is necessary for cartilage morphometry to compute quantitative parameters. To obtain the quantitative measures, a trained observer must segment the normal cartilage surface and the extent of cartilage damage. This segmentation process is assisted by specialized image analysis software, which is used to calculate the articular cartilage morphometric parameters [12]. However, regardless of the expertise of the trained readers or the capabilities of the segmentation software, the manual or semi-automatic segmentation technique is complex and time-consuming [12].

Recently, deep learning (DL) models of artificial intelligence (AI), specifically convolutional neural networks (CNNs), have emerged as a novel approach for knee cartilage segmentation and KOA diagnosis [13]. Unlike traditional hand-crafted strategies, DL models automatically learn features through multiple layers and numerous parameters [14]. Prasoon et al. [15] developed three 2D CNNs, known as multi-planar CNNs. Their study demonstrated that the learned features from CNNs outperformed hand-crafted features, achieving a volume overlap of 82.49% in DSC. Norman et al. [16] utilized the 2D U-Net model to segment various knee sub-compartments, such as articular cartilage and meniscus. The researchers achieved a mean validation DSC of 86.7% on the OAI dataset, which consisted of 37 subjects. Similarly, Liu et al. [17] employed the 2D SegNet model [18] to segment bone and cartilage in the SKI10 dataset. They also compared the performance of SegNet with U-Net and found that SegNet exhibited superior accuracy and computational efficiency in segmenting musculoskeletal images. In a study by Zhou et al. [19], the CNN model was integrated with a conditional random field with spatial proximity as a supplementary post-processing technique to refine the labeling process. The researchers observed that all cartilages achieved a DSC of over 80%. Ambellan et al. [20] embedded SSMs adjustment into 2D and 3D CNNs as a post-processing step. The authors reported the best results of all published work on the validation data of the SKI10 dataset, which achieved up to 88.3% of DSC on the OAI validation set.

We offered a deep learning-based, fully automatic method for morphological assessment of knee cartilage. It is a new artificial intelligence (AI) segmentation cartilage model based on knee joint thin-section MRI data, which can automatically calculate the cartilage [21]. We all know that the accuracy of AI segmentation is verified by deep learning developers during the AI system development process. AI learning is based on the recognition of imaging data by doctors, and the process of learning imaging data will be affected by the subjectivity of doctors. Therefore, the accuracy of AI recognition may be affected. Over the past years, many studies also have found that there is a certain error in predicting the extent of Osteochondral lesions with MRI [22‐24]. If machine learning relies on flawed imaging as the benchmark for accuracy, it will inevitably impact the precision of the machine learning outcomes. One possible approach to tackle these challenges is to employ a pre-trained CNN trained on natural images or diverse medical image modalities and fine-tune it using medical images [25]. In order to avoid this subjective influence and better evaluate and improve the accuracy of our AI segmentation model, our study used the natural tibial plateau cartilage resected during total knee arthroplasty (TKA) as the gold standard for the first time.

The two main objectives of this study were as follows: (i) To evaluate MRI-based AI cartilage's two-dimensional (2D) using the natural tibial plateau as gold standard (ii) To evaluate MRI-based AI cartilage's 2D and three-dimensional (3D) accuracy using the CT-based cartilage model as gold standard.

The AI segmentation cartilage model produced reasonably high 2D DSC. The average 2D DSC between MRI-based AI cartilage model and the tibial plateau cartilage is 0.83. The average 2D DSC between the AI segmentation cartilage model and the CT-based cartilage model is 0.82. As for 3D consistency, the average 3D DSC between MRI-based AI cartilage model and CT-based cartilage model is 0.52. However, the quantification of cartilage segmentation with the AI and CT-based models showed excellent correlation (r = 0.725; P values < 0.05). In order to more intuitively show the results of MR-based cartilage segmentation models in each case. Figure 10 visually displayed the 2D and 3D DSC in each case. Figure 11 visually displayed cartilage volume of each case between AI segmentation cartilage model and the CT-based cartilage model.

This is the first study to use the natural tibial plateau cartilage as the gold standard to evaluate the accuracy of the MRI-based AI cartilage segmentation model.

Compared with traditional machine learning that uses MRI as the gold standard, natural tibial plateau cartilage images were used as the gold standard for the first time in our study. We can avoid the subjective influence and better evaluate the accuracy of our AI segmentation model, making the learning performance of the AI model more accurate and objective. This is the main strength of our study.

Most recent investigations on cartilage morphometry use a slice thickness of 1.0 to 1.5 mm, which is within the range recommended by the Osteoarthritis Research Society International (OARSI) clinical trial guidelines [33]. It has been demonstrated that more minor precision errors have been linked to thinner slices [34]. The slice thickness of 0.7mm in our study was another main strength.

In this study, the natural tibial plateau cartilage was employed as the benchmark. The SPT was used to compare the MRI-based AI cartilage segmentation model with the natural tibial plateau cartilage. The mean 2D DSC between the AI reconstructed cartilage model and the natural tibial plateau cartilage was found to be 0.83, and the average 2D DSC between the AI segmentation cartilage model and the CT-based cartilage model is 0.82. which aligns with the findings of previous researches [17‐20, 31, 35]. In Zhou’s study [19], they evaluated a new segmentation method using deep convolutional neural network (CNN), 3D fully connected conditional random field (CRF), and 3D simplex deformable modeling to improve the efficiency and accuracy of knee joint tissue segmentation. There were 7 tissue types with mean DSC between 0.8 and 0.9 including the femoral cartilage tibial cartilage, patella, patellar cartilage, meniscus, quadriceps and patellar tendon, and infrapatellar fat pad. In Latif’s study [35], a method for automatic segmentation of the tibiofemoral joint using magnetic resonance imaging (MRI) is presented in their work. The proposed method utilizes a deeply supervised 2D-3D ensemble U-Net, which consists of foreground class oversampling, deep supervision loss branches, and Gaussian weighted softmax score aggregation. The mean DSC of femoral cartilage (90.3 ± 2.89%), and tibial cartilage (86.7 ± 4.07%) is achieved. This study means our MRI-based AI cartilage model can reliably extract morphologic features such as cartilage shape and defect location of the tibial plateau cartilage. This approach could potentially benefit clinical practices such as diagnosing OA.

In order to assess the 3D consistency of the MRI-based AI cartilage model, CT-based cartilage model was also used as gold standard to evaluate 3D consistency of the MRI-based AI cartilage model. The average 3D DSC between MRI-based AI cartilage model and CT-based cartilage model is 0.52. In terms of cartilage thickness and three-dimensional accuracy, MRI-based AI cartilage model underestimate the actual cartilage volume. However, the quantification of cartilage segmentation with the AI cartilage model and the CT-based model also showed excellent correlation. This may mean that our previous AI models based on MRI learning were not so accurate in identifying cartilage thickness. We believe that the reasons for the decline in accuracy of the AI model in predicting cartilage thickness are as follows. First, MRI itself has certain errors in predicting cartilage damage [22‐24]. If machine learning relies on these MRI as the benchmark for accuracy, it will inevitably impact the accuracy of the machine learning outcomes. Second, we lacked extensively annotated medical images, which have not yet been fully integrated into clinical practices in our previous AI model learning. Training CNNs from scratch with limited labeled images can result in overfitting [36]. Third, the process of AI learning imaging data will be affected by the subjectivity of doctors. Our results showed that the previous AI verification methods may not be completely accurate and should be verified with real cartilage images. Combining multiple verification methods will improve the accuracy of the AI model.

There are still some limitations in this study. First, the criteria for selecting cases were stringent in terms of inclusion and exclusion. All patients had severe osteoarthritis. There was a lack of comparative data on healthy patients and mild osteoarthritis. Second, Although AI segmentation cartilage model produced reasonably high 2D Dice coefficients. The average 3D DSC between MRI-based AI cartilage model and CT-based cartilage model is not high. MRI-based AI cartilage model underestimate the actual cartilage volume. Thus, additional research with a larger pool of patients on more natural images is necessary to increase cartilage thickness and 3D accuracy of the MRI-based AI cartilage model.

Our study demonstrated that our MRI-based AI cartilage model can reliably extract morphologic features such as cartilage shape and defect location of the tibial plateau cartilage. This approach could potentially benefit clinical practices such as diagnosing OA. However, in terms of cartilage thickness and three-dimensional accuracy, MRI-based AI cartilage model underestimate the actual cartilage volume. Our results showed that the previous AI verification methods may not be completely accurate and should be verified with real cartilage images. Combining multiple verification methods will improve the accuracy of the AI model. In the future, we will employ the pre-trained CNN trained on more natural images or diverse medical image modalities and fine-tune it to improve cartilage thickness and 3D accuracy of the MRI-based AI cartilage model.