Design, validation and implementation of an automated e-alert for acute kidney injury: 6-month pilot study shows increased awareness

We designed a rule-based algorithm for automated AKI diagnosis that evaluates the three criteria, regarding PCr, adapted from the KDIGO guidelines (Supplemental Materials Fig. 1). These criteria assess changes in PCr as a difference (delta) between a PCr measurement and a previous ‘baseline’ PCr. AKI is diagnosed if there is an increase in PCr of 26,5 µmol/L within 48h, or an increase of 1.5 times ‘baseline’, which is known or presumed to have occurred in the previous seven days. Any value in the previous seven days is defined as ‘baseline’ and used to compute the ratio with a newly measured PCr. If more than one PCr measurement in the past 7 days was available, the lowest value was used as baseline to determine the delta PCr. If a patient did not have a PCr measurement available in the last 7 days, the algorithm selected the most recent PCr before the seven days to a maximum of one year as baseline [8]. The other two AKI stages (AKI stages II and III) were not further defined by the algorithm.

For every new PCr measurement, the flowchart was evaluated by our Laboratory Information System (LIS) (GLIMS 9.9.6, Clinisys, Gent, Belgium). When the e-alert was triggered, the physician who requested the PCr was notified via two ways: (1) the e-alert outcome was sent to and presented in our EHR as an additional ‘measurement’ in the laboratory test results page, and (2) a laboratory technician phoned the physician to notify that the requested PCr measurement triggered the e-alert. The e-alert outcome shown in the EHR (HiX 6.1, Chipsoft B.V., The Netherlands, Amsterdam) for the AKI ‘measurement’ was either “positive” or “unable to compute”. No outcome was shown when the alert’s outcome was negative (no AKI). To make the e-alert as actionable as possible, a text box (memo) was shown when the physician hovered with the mouse cursor over the outcome in the EHR system (Box 1, Box 2) as shown in Supplementary Materials Figs. 2 and 3. The text was mainly adapted from the KDIGO guidelines and aligned with our in-house work philosophy [9]. In addition, the memo referred to the hospital-wide AKI protocol in our hospital’s protocol system ‘iProva’ where both AKI in general and the e-alert system are explained, as well the definition of nephrotoxic medication adapted from Ashley et al. (2015) as listed in Supplemental Materials Table 1 [10].

When a patient’s PCr measurements generated multiple e-alerts within one week, only the first AKI episode would be communicated by phone, yet all would be displayed in the EHR. If the patient developed an AKI episode one week after the first alert, a laboratory technician would contact the treating physician again. When a patient met the AKI criteria on day 1 and on day 3, with a PCr measurement on day 2 that did not trigger the e-alert, a laboratory technician would phone the physician both on day 1 and day 3. The e-alert was shown in the EHR system for patients older than 18 years that were treated by all departments, except for the dialysis department (as per request). In addition, all departments except for the dialysis, ICU or COVID-19 wards were phoned (as per request by these departments), as patients were already under high medical surveillance in these wards.

After we developed the e-alert system, we pitched our idea to the laboratory, internal medicine and nephrology departments to receive feedback, as well as to ask how they wanted to be part of the alerting system. We further collaborated with the specialty departments who would most likely be phoned for consult after the onset of an e-alert, as we mentioned them in the memo text: nephrology, internal medicine and the pharmacy. From these stakeholder sessions, we gathered input to finalize the design phase before starting the implementation phase.

As the result of the e-alert was part of diagnostic decision making, the e-alert classifies as a class B Software as a Medical Device (SaMD) according to the IVDR [11]. According to the regulation, developed in-house tests have to meet both safety and performance requirements, and specifically software has to meet development and manufacturing requirements. Our lab is compliant with the ISO 15189 quality management standard to ensure safety and performance. We further verified and validated our software in line with the IEC 62304 standard [12].

To validate the functioning of the flowchart, we performed a risk analysis where we applied the flowchart on various PCr measurements to analyse the risks in terms of false positives and false negatives, and adjusted the system accordingly. Subsequently, we verified the e-alert by evaluating whether the implemented code generated the expected results by creating ‘shadow’ alerts in the background of our LIS system. The complete system was tested by extracting all data of the shadow alerts from the LIS together with all PCr measurements of patients that had at least one e-alert during the validation phase.

We performed a 6-month before-after study to study the effect in awareness of the e-alert in our hospital. We defined two patient groups by selecting all patients six months before (before group 6th of April 2021 to 5th of October 2021) and six months after the implementation date of the e-alert (after group, 6th of October 2021 to 5th of April 2022). At the end of 2021, patients who required intensive treatment were dedicated to the COVID cohort. These patients were excluded from the analysis (N = 55) as they received different treatment as compared to other departments. From 2022 onwards, patients with COVID were admitted to “normal” wards, and thereby automatically included in our analysis.

As the final version of the e-alert was already implemented in the background of the LIS during the testing phase, we could easily identify the patients who triggered an e-alert both before the alert was introduced (shadow alerts), as well as after (real alerts). For these patients, we extracted demographic data, all PCr measurements and all administered nephrotoxic medication from the Utrecht Patient Oriented Database (UPOD) to assess if actions were taken by treating physicians. In brief, UPOD is an infrastructure of relational databases comprising data on patient characteristics, hospital discharge diagnoses, medical procedures, medication orders and laboratory tests for all patients treated at the UMCU since 2004 [13]. PCr was measured by an enzymatic isotope dilution mass spectrometry traceable assay on an Atellica CH analyzer (Siemens Healthcare Diagnostics Inc., Tarrytown, USA).

Treating physicians may receive multiple consecutive e-alerts for the same patient. Therefore, performing analyses on all e-alerts would not be a valid representation of the system as the physician is not phoned when a second alert is generated within one week, and may already have changed treatment after receiving the first alert. For this reason, we focused on the first e-alert and defined this as the start of an AKI episode. To investigate whether the introduction of the e-alert led to action and awareness, we used follow up measurements of PCr within 2 days and discontinuation of at least one nephrotoxic medication within 7 days after the start of AKI episode. Patients who did not use any nephrotoxic medication were excluded from the second analysis. We also assessed the awareness on different levels by stratifying all analyses according to sex of the treated patient, hospital department (emergency department, hospital ward and outpatient clinic) and specialty of the treating physician. All data pre-processing and analyses were performed using the R environment (version 4.2.0). p-values were computed with the Chi-Square test, to test between the proportion of AKI episodes that were followed by an action, between the before and after period.