Development and evaluation of two open-source nnU-Net models for automatic segmentation of lung tumors on PET and CT images with and without respiratory motion compensation

In this study, we present a completely automatic algorithm to segment lung cancer tumors on PET images, with and without respiratory motion compensation. The performance of the algorithm has been evaluated with patients of ten different centres and by comparison with respect to a previously evaluated semiautomatic algorithm and with respect to the variability observed across different experts. In order to facilitate its clinical implementation, an automatic method for tumor segmentation on CT images, with and without motion compensation, based on the same nnU-Net has been additionally developed. The convolutional–network employed by training [38] and the resulting models (GitHub_​PETCTLungSegment​ationModels_​Link) are open source.

One of the clinical applications for the resulting models would be their use for RT planning and monitoring in lung cancer. According to the ESTRO-ACROP guidelines for target volume (TV) definition in the treatment of locally advanced non-small cell lung cancer, both a contrast-enhanced diagnostic CT scan and whole-body diagnostic FDG-PET-CT are considered mandatory in preparation for TV delineation for curative RT or chemo-RT [38]. Typically, the definition of the target volume is performed on the CT, adding the information derived from PET. Assessment of respiratory motion on a respiratory-correlated 4D CT scan is also recommended. 4D-gated PET/CT in the treatment position is optional but may improve the sensitivity of nodal identification and provide additional valuable information to help differentiate between tumor extent and adjacent tissues [39]. By applying our resulting models, the information from both images could be automatically integrated. Furthermore, with the advantage of the enhanced accuracy and robustness provided by 4D-protocolos, these segmentations could be employed to improve the TV definition, optimising therefore treatment planning and monitoring.

Most of the previous convolutional-neural-network (CNN) approaches for lung tumor segmentation have been based on CT images [40‐42] or on MR images [43] with 330, 1210, 19 and 9 patients, respectively. The benefit of PET imaging for lung cancer patients has been already demonstrated and consequently, segmentation models based on PET/CT images have been developed [44, 45], with 84 and 32 patients, respectively. It is known that the performance of AI segmentation models depends on the availability of a large amount of data for its training step. In PET, it is usually challenging to gather such large cohorts of patients compared to other imaging modalities [29]. In our study, the size of the PET cohort (100 patients) is larger than the size of the cohorts commonly employed for PET segmentation model development and validation [46]. It is also known that segmentation CNN-based approaches are used to perform well during training but demonstrate reduced performance during the validation and test steps. Therefore, cross-validation and testing of the model performance have been strongly recommended [29]. Our initial training cohort of 56 patients has been 5-fold cross-validated in order to take more benefit from the limited size of the sample. The performance of the resulting model after cross-validation, has been tested with 25 patients prospectively recruited at the same centre. Because the level of performance of AI models, as well as their reproducibility and robustness, is sensitive to imaging protocols and devices, we additionally evaluated the performance of our model for 19 patients prospectively recruited at 9 other centres. Segmentation accuracy in terms of DSC values was comparable for both, internal and external cohorts, allowing concluding an acceptable generalizability of the model. In terms of the clinical acceptability, it is important to include a comparison of the algorithm performance with respect to the manual contours and the variability across experts, which is the standard method commonly employed in our clinical practice for tumor segmentation. In contrast to most of the previous publications, our study not only includes the comparison with respect to the established ground truth (consensus of two experts), but the analysis also involved the manual contours of three additional experts. From the results in this comparison, it could be concluded that the accuracy of the proposed model is promising in terms of robustness. In terms of the viability of its clinical implementation, a CT-based segmentation model has been developed with the same nnU-Net in order to facilitate the integration of PET segmentation model in the clinical workflow. Although the average values of DSC for the resulting CT model were not high, the performance of the algorithm was proved to be better than the variability observed among experts. Finally, previous literature has demonstrated that the compensation of respiratory motion improves accuracy and reproducibility for the biomarker distribution quantification [20, 23, 24]. However, 4D-PET imaging implementation in the clinical workflow is hindered by the time and expertise, required from the clinical site, for a proper analysis and integration of the functional information. The segmentation algorithm resulted from our study has been proven to accurately segment tumors with both imaging protocols, with (4D) and without (3D) motion compensation. To evaluate the impact of motion compensation on the performance of our model, images with and without motion compensation for the same patients should be compared. Unfortunately, only for 8 patients and only for CT images, both imaging protocols (3D and 4D) were available. For these patients, the better performance obtained for the model with non-compensated images (DSC(3D-CT) = 0.63 ± 0.34, improving 15% improvement relative to experts) than with motion-compensated images (DSC(4D-CT) = 0.61 ± 0.28, improving 4% relative to experts), could be justified by the fact that the model has been trained with non-compensated images. It is not the case for the manual contours, where the accuracy with motion-compensated images, DSC(4D-Expert) = 0.59 ± 0.24, was better than without motion compensation, DSC(3D-Expert) = 0.55 ± 0.11. However, the accuracy for the nnU-Net model is better than the variability observed across experts in both 3D and 4D images, supporting the applicability of the model independently of the imaging protocol. Therefore, this segmentation model could facilitate the clinical implementation of 4D-PET images for monitoring and for the development of prognosis models, where the quantification of PET images plays a crucial role in order to identify clinically relevant changes and where, consequently, the accuracy and reproducibility provided by these 4D-protocols are of special interest.

In order to quantify the performance of our models, 3D computation of DSC was employed, because it is the most common and standardised method for the comparison of segmentations in medical imaging [38]. However, a high DSC does not necessarily indicate good agreement. An example could be a prediction of a tumor similar in size and location to the ground truth, but with a distant small region incorrectly classified also as a tumor. This scenario is likely to be found in lung tumor segmentation on 4D-PET thorax images, because of the high uptake of the heart and the noise implied by respiratory frames. For these reasons, Hausdorff distance [39] was additionally computed, and the results allowed us to confirm the good performance of our model. In addition, a positive-predictive-value [36] permitted to quantify the qualitatively observed trend in CT model performance, i.e., a high positive-predictive-value demonstrated that the model is able to detect the lesion and its location, but underestimates its size.

The main advantages of nnU-Net are that a preprocessing of the input data is not required, the non-fixed parameters for the training process of the model are automatically configured based on the input image data and the resulting model is generated and optimised without user interaction. In addition, the good performance of the nnU-Net method has been proved in different data sets and the code can be free to download [37]. For these reasons, the whole process can be easily reproduced. Medical data sharing is one of the most important concerns in AI development. It facilitates the comparison of the performance for different algorithms, can be used for the improvement of the generalizability of the algorithms, and makes feasible for developers to test their new CNNs. The main limitation in our study was that the clinical data employed is not open access. There are different CT and MR image repositories for lung cancer. The Lung Image Database Consortium-Image Database Resource Initiative (LIDC) is the world’s largest publicly available database and contains 1018 CT scans of 1010 patients. The Automatic Nodule Detection 2009 (ANODE09) is another publicly available database consisting of 55 CT scans, annotated by two radiologists. Also, the TIME [47]; ELCAP [48] and LISS [49], are available in the public domain for lung nodule research. Future work should focus on PET image repositories that will also facilitate the development of PET AI tools for its use in clinical practice.