From pixels to patient care: deep learning-enabled pathomics signature offers precise outcome predictions for immunotherapy in esophageal squamous cell cancer

ESCC patients receiving PD-1 inhibitors between 1 January 2018 and 1 January 2023 at Shandong Cancer Hospital and institute were included in this study. The inclusion criteria were as follows: (1) pathologically and radiological diagnosed as esophageal squamous cancer patients; (2) receiving PD-1 inhibitors; (3) with access to survival follow-up data. Patients with another primary malignant neoplasms were excluded from further analysis. The baseline characteristics were collected, including age, gender, smoking history, drinking history, TNM stage, metastasis and radiotherapy. Besides, the H&E-stained histological specimens of included patients were collected. The WSI images of H&E-stained histological specimens from included patients were scanned using Pannoramic MIDI II, Pannoramic SCAN II scanner or Zeiss Axio Scan.Z1. All images were saved as TIF files in pathological dataset, and were randomly divided into training (70%) and validation (30%) sets. The detailed clinical characteristics and survival time of the cohort are also retrieved. Immunohistochemistry staining for PD-L1 was performed on FFPE tumour tissue using PD-L1(22C3) monoclonal antibodies. The expression of PD-L1 was measured by tumor proportion score (TPS), which is defined as the percentage of membrane-positive tumor cells in all tumor cells.

The primary endpoint of the study was PFS of included patients after receiving PD-1 inhibitors, which was defined as the time from the beginning of PD-1 inhibitors to the disease progression or death. And the secondary endpoint was overall survival (OS) of included patients after receiving PD-1 inhibitors, which was defined as the time from the beginning of PD-1 inhibitors to the death. The PFS time of the cohort was used as labels for further model training. In this study, the survival prediction from PD-1 inhibitors was formulated as a classification problem.

After digitization, the WSI images were pre-processed for further patch-level and patient-level model training. We split the WSI into small patches of 1024 × 1024 pixels at 20 × magnification. To eliminate unnecessary white background in the further process, we selected patches with variable of pixels more than 500. An additional challenge of these images was the stain color distribution differed from WSIs due to the complex staining process, which we chose to address slide level color normalization using Macenko method.

Firstly, the ViT_base_patch_16_384 architecture which has been pretrained in ImageNet dataset was used for patch-level model training. The input data were the patch images obtained from splitting the WSI, and the labels of patches were the same as the WSI image of respective patient. In order to reduce the influence of noise and prevent overfitting, symmetric cross-entropy was applied to calculate the loss. Adam optimizer was used as the optimizer algorithm with an initial learn rate of 0.0001, weight decay of 0.0001 and a 50-image batch size. The patch level model was trained for 50 epochs.

To obtain the patient-level probability distribution of patches, the softmax output vectors were used to train a linear classifier. All patches from the WSI image of a single patient were summed together, and projected into probabilities using the linear classifier. The patches were ranked by their prediction probabilities, and the top 100 patches with highest probabilities were selected and assigned to the patient. Then a feature extractor would be trained for patches selection.

The 40 most suspicious patches of each WSI image are sequentially passed to the RNN for features integration and final patient-level prediction. The cross-entropy was used to calculate the loss of RNN model, and Adam optimizer was used as the optimizer algorithm with a batch size of 2. The ESCC-pathomics signature (ESCC-PS) was constructed based on the patient-level ViT-RNN. All patients were assigned to three groups according to the ESCC-PS, and the accuracy rate were calculated to evaluate the performance in validation cohort. Univariate and multivariate survival analysis was performed to confirm the predictive effect of ESCC-PS in validation cohort. The whole process of this study was shown in Fig. 1.

The expression of PD-L1 was evaluated using immunohistochemical stains. And the cut-off values of PD-L1 was determined by receiver operating characteristic (ROC) curve in training cohort, in order to divide patients into high and low expression group. The predictive effect of PD-L1 was evaluated by cox regression analysis. ESCC-PS incorporating expression of PD-L1 for the outcome prediction of PD-1 inhibitors were applied to determine the incremental value of ESCC-PS. Patients in validation cohort were divided into low-risk, medium-risk and high-risk group based on the incorporation of ESCC-PS and PD-L1. The performance of PD-L1, ESCC-PS and incorporation signature were assessed and compared by C-index in validation cohort.

The construction of ESCC-PS was performed using Python 3.6. And SPSS 26 and R 4.3.1 were used to conduct the data analysis and visualization. Kaplan–Meier survival was carried out to verify the clinical significance of ESCC-PS. Multivariable analysis was conducted using Cox proportional hazards modeling to validate the predictive value of ESCC-PS. Interactions between ESCC-PS and patient characteristics were detected by χ2 test. ROC curve with area under the curve (AUC) was performed to compare the performance between PD-L1, ESCC-PS and incorporation signature. All tests were 2-sided, and P < 0.05 was considered to indicate statistical significance.

There were 163 ESCC patients receiving PD-1 inhibitors from Shandong Cancer Hospital with baseline data and known outcomes included in analysis. 324 WSI images of H&E-stained slides were retrieved from these patients, and were randomly divided into training cohort and validation cohort with the proportion of 7:3. The patches from a single patient were divided into the same cohort. There were 120 patients with 227 images in training cohort and 43 patients with 97 images in validation cohort. The clinical characteristics of patients in this cohort were illustrated in Table 1, and no significant difference was observed between training and validation cohort. After the median follow-up time of 14.2 months, the median PFS was 7.9 months for the whole patients. The optimal cut-off of PD-L1 expression was set as 57.5% with the AUC of 0.632 in training cohort.

There were 486,188 patches with 1024 × 1024 pixels after image pre-processing from 324 WSI images. Then subsampling was used to resize these patches into 384*384 pixels to adapt to pretrained ViT_base_patch_16_384. All patches were used as input to ViT for patch-level model training, then were projected into group probabilities. The 40 most predictive patches are sequentially passed to the ViT-RNN for final patient-level prediction and construction of ESCC-PS. Patients in validation cohort were projected into three groups based on ESCC-PS, and 19 patients were projected in ESCC-PS 1 group, 13 patients in ESCC-PS 2 group and 11 patients in ESCC-PS 3 group. As shown in Table 2, no potential interactions were found between ESCC-PS and patient characteristics in validation cohort.

The ViT-RNN based ESCC-PS achieved the accuracy of 92% in training cohort, and 84.5% in validation cohort. Four iterations of random partition of all patients were performed to investigate the stability of the model, and the accuracies of the model ranged from 82.7% to 95.1% in the validation cohort, indicated the stability of the model. The Kaplan Meier survival curves indicated the significant difference on the PFS (P < 0.001) with the median PFS of 2.7, 4.8 and 16.7 months respectively between ESCC-PS 1, ESCC-PS 2, and ESCC-PS 3 group (Fig. 2A). The superiority in OS (P < 0.001) was found for patients in ESCC-PS 3 group (Unreached) compared to ESCC-PS 1 (6.3 month) and 2 group (20.2 month) (Fig. 2B). And the predictive effect of ESCC-PS (P < 0.001) was also shown according to the univariate cox analysis, highlighting the predictive value of ViT-RNN based ESCC-PS. As shown in Table 3, the expression of PD-L1 was also associated with the PFS of ESCC patients receiving immunotherapy (HR = 0.486, P = 0.092). Multivariate cox analysis indicated both PD-L1 (HR = 0.231, P = 0.009) and ESCC-PS (P < 0.001) remained a significant prognostic indicator indicating the independent prognostic factors for ESCC patients receiving immunotherapy. The superior performance of ESCC-PS for the prediction of PFS (P < 0.001) and OS (P = 0.005) was also shown in the training cohort based on multivariate cox regression analysis (Additional file 1: Tables S1 and S2).

The incorporation of ESCC-PS and expression of PD-L1 was performed based on the multivariate cox regression analyses to elucidate the incremental value of the ESCC-PS added to the expression of PD-L1 for predicting the outcome of PD-1 inhibitors. As shown in Fig. 3, survival analysis indicated the incorporation signature could significantly distinguish patients into high-risk, medium-risk and low-risk, with the PFS of 2.6, 4.5, 12.9 months, and OS of 6.3, 20.2, 34.8 months, respectively (Table 4)

The C-index of ESCC-PS for PFS after receiving PD-1 inhibitors was 0.806, which was significantly higher than that of the expression of PD-L1 (0.601, P < 0.001). As shown in Table 5, the significantly incremental C-index was observed for incorporation of ESCC-PS and PD-L1 compared with PD-L1 alone (0.814 vs 0.601, P < 0.001). The comparison of ROC for prediction of PFS and OS between PD-L1, ESCC-PS and ESCC-PS + PD-L1 was shown in Fig. 4. The AUC of ESCC-PS + PD-L1 for 6 month- (0.904 vs 0.610, P < 0.001) and 12 month- PFS (0.868 vs 0.679, P = 0.099) prediction was higher than PD-L1. Similarly, ESCC-PS + PD-L1 also exhibited higher AUC for prediction of 12 month- (0.901 vs 0.643, P < 0.001) and 18 month- OS (0.883 vs 0.626, P < 0.001).