A novel explainable online calculator for contrast-induced AKI in diabetics: a multi-centre validation and prospective evaluation study

The study was divided into two steps. Firstly, we retrospectively reviewed the medical records from multi-center hospitals to build and validate the predictive model. The multi-centre hospitals included Affiliated Sir Run Run Hospital of Nanjing Medical University, Nanjing First Hospital, Affiliated Shu Yang Hospital of Nanjing University of Chinese Medicine and Xu Zhou Medical University Hospital. The study population included diabetic patients who underwent coronary angiography (CAG) and PCI between January 2014 and January 2020. We excluded patients based on the following criteria: (1) missing serum creatinine levels prior to and after CAG and PCI; (2) needing dialysis before CAG and PCI; (3) repeated hospitalization for PCI; and (4) acute kidney injury prior to CAG and PCI. Our research was carried out in respect to the Declaration of Helsinki. Due to its retrospective design, our hospitals gave their approval for the study and waived the need for informed consent.

Secondly, we conducted a prospective study in Affiliated Sir Run Run Hospital of Nanjing Medical University to determine early prediction of CIAKI with our CIAKI online calculator. The study population included adult diabetic patients that underwent CAG and PCI from June 2021 to April 2022. The Ethnic Committee approved this study (Ethics number: 2021-SR-041) and waived the requirement for informed permission to use identifiable data. We reported our work following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement guideline [14], Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement [15] and guidelines for ML predictive modeling [16].

CIAKI was the primary outcome of our study, based on the Contrast Media Safety Committee (CMSC), described as an increment of serum creatinine value at least 44.2 μmol/L (0.5 mg/dl) or 1.25 times comparing the baseline level within 72 h exposure to contrast agent, eliminating alternative causes of acute kidney injury. The baseline creatinine was the lowest serum creatinine level within 7 days before CAG. In the 72 h following CAG and PCI, all serum creatinine values were collected. Dialysis, stroke, length of in-hospital stay, the new-onset or recurrence of myocardial infarction and other adverse cardiovascular events such as worsening heart failure and death were also included as outcomes.

DM was defined if the patient’s treatment included dietary, oral, or insulin therapy or if patients’ fasting blood glucose value was 126 mg/dl based on the practice guidelines of the American Diabetes Association [17]. Congestive heart failure (CHF) was diagnosed if the patients were grouped into New York Heart Association (NYHA) class III or higher based on the categorization system of the NYHA or history of pulmonary edema. Clinicians comprehensively diagnosed acute coronary syndromes (ACS) according to the symptoms of myocardial ischemia, changes in electrocardiogram, and myocardial injury biomarkers [18]. According to the definition of chronic kidney disease (CKD), patients with proteinuria, estimated glomerular filtration rate (eGFR) < 60 ml/min/1.73m², or both on at least two occasions more than or equal to three months apart [19, 20].

In each institution, demographic data, preoperative medications, and laboratory tests were collected, including gender, age, pre-CAG blood pressure, body mass index (BMI), coronary artery disease, primary disease, contrast agents, and periprocedural biochemical markers. We removed characteristics absent in 11% or more of the samples. The abnormal value of variables were rechecked in electronic hospital records. Otherwise, they were treated as missing values. Categorical variables were processed with one-hot encoding and label encoding. One-hot encoding creates a separate binary feature for each category and is suitable for categorical variables without a specific order or hierarchy. For example, we converted the gender “male” or “female” to “female or not”. Label encoding assigns a unique numerical label to each category. Each category is mapped to a different integer value. Label encoding is suitable for categorical variables with a clear order or hierarchy, such as ordinal variables. For example, the variable “Diabetes history (yrs)” with categories “ < 1 year, 1–5 years, 5–10 years, 10–20 years, >  = 20 years” were converted to “1, 2, 3, 4, 5”. Variance inflation factor (VIF) and generalized variance inflation factor (GVIF) were used to deal with collinearity between continuous and categorical variables, respectively. The continuous variables with VIF > 10 were removed. For categorical variables, we set the category with the largest proportion in each categorical variable as the reference level and considered the categorical variables with GVIF^[1/(2 × Df)] > 10^(1/2) to have high collinearity and removed them, where Df refers to the degree of freedom. We divided the data into the cohort of training, internal validation and external validation. We randomly used 80% of the data from Nanjing First Hospital for model training, 20% from Nanjing First Hospital for model internal validation, and other centres for model external validation. We used the missForest method, which can handle missing values with a combination of continuous and categorical variables to fill each remaining measurement’s missing data in the three cohorts separately [21]. Meanwhile, variables were standardized before training and prediction by removing the mean and scaling to unit variance.

In the prospective design, we recorded the time of each variable in the CIAKI model and the time of clinical diagnosis of CIAKI to obtain the earliest time when the model predicted the occurrence of CIAKI. Because of the prospective design, none of the required variables had a missing value.

To solve the imbalance between positive and negative samples, we adopt a variety of balancing methods in the training set, including oversampling and undersampling. Oversampling includes Synthetic Minority Oversampling Technique (SMOTE), ADAptive SYNthetic (ADASYN) technique, and random oversampling. Undersampling includes random undersampling and TomekLinks (Additional file 1: Table S4). Finally, we found that each balancing method performed equally on the internal validation set, but TomekLinks performed better in the external validation set, so we chose to use TomekLinks. Specifically, TomekLinks focuses on neighboring pairs of samples, where one sample belongs to the minority class and the other belongs to the majority class. These sample pairs are close to each other and form links. These links are considered potential noise or outliers, which may have a negative impact on model training and performance. By identifying and addressing these links, we can reduce noise or outliers in the data and improve the performance of the classification model.

Mehran risk score [9] is calculated with 8 variables: hypotension, CHF, intra-arterial balloon pump (IABP), anemia, age, diabetes, contrast media volume, serum creatinine or eGFR. We calculated the total Mehran risk scores for each patient based on the sum of the scores corresponding to the 8 variables which were 5 points for hypotension (Systolic blood pressure is less than 80 mmHg for at least 1 h and inotropic assistance is required), 5 points for IABP (IABP is used), 5 points for CHF (NYHA classification III/IV or history of pulmonary edema), 4 points for age (more than 75 years old), 3 points for anemia (men’s hematocrit less than 39% while women’s less than 36%.), 3 points for diabetes, 1 point for contrast media volume per 100ml and 4 points for serum creatinine > 1.5mg/dl, or 2 points for eGFR 40–60 ml/min/1.73m², 4 points for eGFR 20–40 ml/min/1.73m², and 6 points for eGFR < 20 ml/min/1.73m².

Six ML models were constructed, including extreme gradient boosting trees (XGBT) model, random forest (RF) model, support vector machine (SVM) model, logistic regression (LR) model, least absolute shrinkage and selection operator (LASSO) with LR (Lasso + LR), and gradient boosted decision trees (GBDT) model. Additional file 1 included a full explanation of the six ML models.

ML models were also trained using ten-fold cross-validation (Additional file 1: Figure S1 and Additional file 1: Table S3). The initial samples were randomly split into ten equal-sized subsamples, one of which was used to validate the results and the other nine as training samples. For each model, in order to select the ideal hyperparameters, a grid search method with ten-fold cross validation was used. Furthermore, we constructed the SHapley Additive exPlanation (SHAP), which demonstrates each variable’s impact on the overall model as well as its contributions to the model. Additionally, the SHAP plot function was also used to reveal the XGBT model’s complicated link between factors and results. Finally, to forecast the risk of CIAKI in diabetics, we developed an explainable online web-based risk calculator.

All models were evaluated in internal as well as external validation sets. Each model’s area under the receiver operating characteristic curve (AUC), accuracy, positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity, and F₁ score were also compared. Additionally, we chose the CIAKI prediction threshold by maximizing the F₁ score in the training set. A 95% confidence interval (CI) was performed in 1000 iterations of bootstrap sampling with replacement. To examine the agreement between calculated likelihood and observed CIAKI prevalence in the population, a calibration curve was utilized. Moreover, the net benefit of each model was calculated using decision curve analysis (DCA) based on the difference between the predicted benefit and the expected risk associated with CIAKI.

From January 1, 2014 to January 30, 2020, a total of 5052 diabetic patients underwent CAG + PCI. Based on the inclusion and exclusion criteria, 3514 patients were included in the study. The characteristics of included patients and excluded patients were compared in Additional file 1: Table S1. Patients that were excluded had more CKD stage 4–5, higher blood urea nitrogen levels and more vessels of coronary artery disease. The main reason was that the exclusion criteria included patients who were on dialysis and had developed acute kidney injury before CAG + PCI which resulted in excluded patients having worse kidney function. Moreover, the exclusion criteria included repeated hospitalization and repeated CAG which suggested that the excluded patient’s coronary artery disease might be more serious like more vessels of coronary artery disease and required multiple CAG examinations. The remaining characteristics were not statistically different between the included and excluded patients. We randomly partitioned patients from Nanjing First Hospital into the cohort of training (80%, 2368 patients) and the cohort of internal validation (20%, 592 patients). Patients from the other three hospitals were used as the external validation cohorts (554 patients) (Fig. 1). The frequency of CIAKI in training, and internal and external validation sets was 447/2368 (18.9%), 107/592 (18.1%), and 80/554 (14.4%), respectively (Fig. 1). Additional file 1: Table S2 displayed the three cohorts’ initial characteristics. The median age in the cohort of training, internal validation and external validation was 67-year-old (interquartile range [IQR]: 60–74), 67-year-old (IQR: 59–73) and 65-year-old (IQR: 58–73), respectively. Patients with CIAKI were older, had worse heart dysfunction, higher frequency of CKD 3–4 stage, coronary artery disease and anemia, and higher uric acid, urine protein, and total cholesterol (Table 1).

634 patients (18.04%) developed CIAKI, and 50 (1.42%) occurred adverse cardiovascular events. Of them, 21 (0.60%) patients revealed worse heart failure, and 12 (0.34%) reoccurred or experienced a new-onset myocardial infarction. On top of that, we observed 7 (0.20%) patients developed stroke, and 11 (0.31%) death. CIAKI was linked to an increased risk of myocardial infarction (0.79% vs 0.24%, P = 0.033) and overall adverse cardiovascular events (2.5% vs 1.2%, P = 0.01) and increased hospitalization stay (9.23 ± 4.87 vs 7.38 ± 4.87, P < 0.001) (Table 2).

During the study, 61 essential characteristics from electronic medical records were chosen, including 5 demographic data, 13 medical history characteristics, 9 intraoperative indicators, 6 preoperative medications, 16 preoperative laboratory tests, and 6 postoperative serum creatinine and postoperative blood urea nitrogen at 24, 48, and 72 h. In addition, 6 clinical endpoint variables were included. After data cleaning, missing values greater than or equal to 11% (Killip classification, glycated hemoglobin and NYHA classification) were removed. At the same time, collinear features were removed, including height, weight, nonionic iso-osmolar and total cholesterol. Since this study used pre-CAG + PCI and intraoperative indicators to predict the occurrence of postoperative CIAKI, we removed postoperative serum creatinine and postoperative blood urea nitrogen at 24, 48, and 72 h. We retained preoperative serum creatinine and preoperative blood urea nitrogen. In addition, 6 clinical endpoint variables occurred postoperatively, and we did not include them as risk factors As a result, 37 features, including 20 categorical features and 17 continuous variables, were retained in the training cohort to establish ML models. Further, we screened 23 features by the LASSO + LR model for CIAKI (Additional file 1: Figure S2). We also showed the top 20 features for CIAKI in each ML model (Additional file 1: Figure S3). Figure 2a illustrated the scaled importance rank of all features in the six ML models for identifying CIAKI risk in diabetic patients. Figure 2b, a subset of Fig. 2a, showed the importance of the final 15 variables screened in the six ML models. Figure 2c showed the relative importance of the 15 variables in Fig. 2b in the six ML models. As shown in Figs. 2b and c, ACS presented the most crucial feature in all ML models.

In the internal validation cohort, all ML models achieved higher AUC than Mehran score, of which the XGBT model achieved the best AUC (0.816, 95% CI 0.777 to 0.853) (Fig. 3a). Additionally, all of the ML models outperformed the Mehran score in terms of F₁ score, accurary, sensitivity, spensitivity, PPV and NPV. (Table 3). The XGBT model demonstrated superior performance than the other 4 ML models. The calibration curves of the 5 ML models and Mehran score were shown in Fig. 3b. The DCA indicated that the risk threshold of the XGBT model ranged from 0.20 to 0.50, which was superior to the ranges associated with other models (Fig. 3c). Additionally, the net benefit of the XGBT, RF, LASSO + LR and LR were optimistic when the risk threshold was in the range of 0 to 0.55.

The AUC performance in the cohort of external validation was comparable to that in the cohort of internal validation. The better performance of the ML models than the Mehran score remained consistent. XGBT model achieved better AUC (0.816, 95% CI 0.770 to 0.861) than others (Fig. 3d). Furthermore, the calibration curves of all models still performed well (Fig. 3e). According to the DCA, the XGBT model provided a clinical net benefit when models ranged from 0.10 to 0.65 (Fig. 3f).

Among all the models, XGBT achieved the best AUC in both internal and external validation cohorts. In general, the AUC of the model will increase as more features are selected. Nevertheless, adding more features does not improve clinical practice. To discover the significant features, we sorted the importance of XGBT features in descent order in the training set. The performances of AUC, sensitivity, specificity and accuracy corresponding to the top n number of features in XGBT were shown in Fig. 4a1. When the number of features was increased to the top 15, the AUC rose to 0.883, and the accuracy and sensitivity had significant improvements. With the number of features growing to 22, the AUC steadily reached its maximum which was 0.896. However, the sensitivity had a small decrease from 0.644 to 0.611 when the number was from the top 15 to 22. The performance of AUC, specificity, sensitivity and accuracy tended to be stable and no longer changed after more than 27 features. Considering more features were not beneficial to clinical use and practice, and the AUCs after 15 variables were overall stable, to facilitate the application in clinical practice, we selected the top 15 critical variables as the brief CIAKI prediction model for diabetes, called the BCPMD model: ACS, Urine protein level, Diuretics, left ventricular ejection fraction (LVEF)(%), Hemoglobin, CHF, Stable Angina, Uric acid, Preoperative DBP, Contrast Volumes, Albumin, Baseline creatinine, Vessels of coronary artery disease, Glucose and Diabetes history (Fig. 4a1 and a2). The corresponding risk threshold of BCPMD was 0.3338 which was based on maximizing the F₁ score. Violin plots were analyzed to demonstrate the distribution of 8 continuous characteristics contained in BCPMD between CIAKI patients (n = 634) and non-CIAKI patients (n = 2880) (Fig. 2d and Additional file 1: Table S5). Also, relationships between 7 categorical features and CIAKI were observed in Fig. 2d. Besides, The AUCs of the BCPMD for CIAKI in the cohort of training, internal validation, and external validation were 0.883 (95% CI 0.867–0.898), 0.819 (95% CI 0.783–0.855), and 0.805 (95% CI 0.755–0.850), respectively (Fig. 4b). The expected calibration errors (ECE) in calibration curves of BCPMD were 0.073 for the cohort of internal validation and 0.135 for external validation (Fig. 4c and e). The net benefits of the BCPMD in the cohort of external validation were reduced than in the cohort of internal validation (Fig. 4d and f).

Table 4 displayed the BCPMD’s prospective predictive performance. AUC, accuracy, sensitivity, specificity, PPV, NPV, and F₁ scores of BCPMD were 0.801 (95% CI 0.688–0.887), 0.826 (95% CI 0.779–0.866), 0.684 (95% CI 0.500–0.846), 0.843 (95% CI 0.793–0.887), 0.351 (95% CI 0.219–0.485), 0.956 (95% CI 0.924–0.979), 0.464 (95% CI 0.311–0.586), respectively. Moreover, CIAKI occurred in 19/172 (11.0%) in the prospective cohort. Of the 19 patients with true CIAKI, BCPMD correctly predicted 13 patients. Additional file 1: Table S6 displayed the prospective validation cohort’s basic characteristics.

We explained the BCPMD through the SHAP diagram. After inputting each variable, the model’s positive or negative contribution could be observed (Fig. 5c). The SHAP summary plot demonstrated that ACS, hemoglobin, diuretics, LVEF (%) and uric acid (umol/L) ranked as the top 5 important features. Moreover, the SHAP plot revealed that ACS, lower hemoglobin (g/L), using diuretics, lower LVEF (%) and higher uric acid (umol/L) were correlated with a greater SHAP value generated in BCPMD, implying a higher risk of CIAKI. Figure 5a showed two cases (Patient No.2, No.17) by SHAP decision plot, which simulated the path of each feature decision. In addition, the different feature values of BCPMD represented different positive and negative contributions to the final SHAP value output. Red values represented higher risk factors, and blue values represented lower risk factors (Fig. 5b). It reflects the personalized interpretation function of SHAP and helps physicians make clinical decisions on the individual level.

Based on the BCPMD, we built a dynamic and explainable website to calculate the risk of CIAKI in diabetes. The URL is here: http://​49.​51.​70.​39/​. When a patient plans to undergo CAG and PCI, the physician can enter the associated risk factor values into the website, which will immediately produce the projected risk values for CIAKI. Furthermore, the risk of CIAKI was judged as negative or positive according to the threshold of 0.3338 on the platform. Moreover, we used the missForest method to impute the missing data to predict the risk of CIAKI even when features are unknown. Notably, we developed a dynamic and explainable waterfall diagram to show the positive or negative contribution of different risk factor values, in which red presents higher risk and blue presents lower risk. Figure 6 showed an example that our web calculator predicted CIAKI in a case within 1 h of admission.