Comparison of the predictive ability of clinical frailty scale and hospital frailty risk score to determine long-term survival in critically ill patients: a multicentre retrospective cohort study

This study was approved by The Research Governance of Peninsula Health Ethics Committee (reference number HREC/47502/PH-2018, DHHS/RQ907) with a waiver of informed consent.

We conducted a retrospective multicentre observational study from 1st April 2017 to 30th June 2018 including consecutive critically ill patients admitted to 16 public ICUs in the state of Victoria, Australia with a clinical documented CFS score. The censor date for survival follow-up was 31st July 2018 to ensure that there was at least one-month follow up for all patients. We only included the first hospital admission during the study period.

Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Database (APD) This bi-national clinical quality registry dataset collects de-identified information on all admissions to contributing adult ICUs in Australia and New Zealand. Trained staff working in each ICU collect this data. All ICUs within public hospitals in Victoria contributed throughout the study period. Apart from each patient’s demographic details, the data also captured their diagnostic, biochemical, physiological, and chronic health parameters from the first 24 h of ICU admission as required to calculate illness severity scores. The definitions are described in the ANZICS-APD data dictionary [18].

Victorian Death Index (VDI) This administrative dataset records the date and cause of all deaths that occur in Victoria, based on the issued death certificates. Any deaths that occur outside Victoria are not included in the registry. Information was available up to 31^st July 2018.

Probabilistic methods were used to match de-identified ICU admission episodes listed in the ANZICS registry to their equivalent administrative data by the Centre for Victorian Data Linkage.

The CFS is voluntarily collected as part of the ANZICS-APD at the time of ICU admission, based on the patient’s level of physical function in the two months before ICU admission by sixteen out of twenty-three hospitals. The CFS (range 1–8) categorises patients as non-frail (1 = very fit; 2 = well; 3 = managing well; 4 = vulnerable) or frail (5 = mildly frail; 6 = moderately frail; 7 = severely frail; 8 = very severely frail). Patients with a score ≥ 5 were considered frail. CFS was assigned by data collectors in participating ICUs based on the patient’s level of physical function in the two months preceding ICU admission [18].

The HFRS (range 0 to 45) was estimated using routine data based on the ICD-10 coding system obtained from the AR-DRG [13,19]. ICD-10 codes used to estimate HFRS, and the respective points awarded for each diagnosis are summarised in Additional file 1: Table S1. For this study, we categorised patients with HFRS score > 5 as frail and those with score < 5 as non-frail [13]. We only used the ICD-10 codes that represented pre-existing conditions at the time of index hospital admission, rather than those that were developed during the hospitalisation. A complete list of ICD-10 variables extracted from the VAED database is listed in Additional file 1: Table S2. VDI database is summarised in Additional file 1: Table S3. The patients were categorised as non-frail and frail for both CFS and HFRS.

The primary aim was to assess the use of HFRS as a frailty screening tool in ICU patients by comparing its performance with the CFS as a predictor of one-year survival following ICU admission. The secondary aims were to compare the performance of the CFS and the HFRS as predictors of ICU, hospital, 28-day, 90-day, 6-month and 1-year mortality. The pre-defined subgroup analyses included prediction of 1-year mortality for patients ≥ 75 years of age and those needing mechanical ventilation.

Categorical comparisons between frail and non-frail patients were performed using Chi-square, two-sample t-tests for normally distributed data and Wilcoxon rank-sum test otherwise, with results reported as counts (%), means [standard deviation (SD)] or median (interquartile range [IQR]), respectively. Correlation between the continuous CFS and HFRS was assessed using Spearman correlation coefficient and agreement using Kappa for dichotomous (not frail: CFS 1–4, HFRS 0–5; frail: CFS ≥ 5, HFRS > 5). Patient survival was compared using Cox proportional hazards regression adjusting for patient’s illness severity, and sex, with results reported as hazard ratios (HR, 95% CI). Time-dependent covariate analysis and log-minus-log plots were performed to assess proportionality assumptions. While HR were reported as the risk associated with a 1-unit increment for the CFS (range 1–8), to facilitate a more proportional comparison between the 2 frailty tools, HR for HFRS were reported as the risk associated with a 5-unit increase (range 0–45). The dichotomised Kaplan–Meier survival curves were performed for the two frailty measures. The performance of the CFS and the HFRS in predicting time-specific mortality rates was determined using logistic regression models with results reported as the area under the receiver operating characteristic (AUROC) plots with comparison using chi-square tests [20]. Multivariable logistic regression analysis was used to compare the performance of the CFS and the HFRS as predictors of ICU, hospital, 28-day, 90-day, 6-month and 1-year mortality. Illness severity was determined using the Australian and New Zealand Risk of death (ANZROD) which is a highly predictive mortality prediction model used for benchmarking ICU performance in Australia and New Zealand. ANZROD includes components of the APACHE III/IV scoring system, such as age, chronic illnesses, acute physiological disturbance and diagnosis, and the presence of treatment limitation on admission to ICU and provides an accurate estimate of the severity of illness in the first 24 h of ICU admission [21,22]. Hence, adjusting for sex was required as a separate variable, and not age. Additional sensitivity analyses were performed based on elective and non-elective admissions and those who were discharged home alive. The HFRS was quantified with the ICD-10 codes from the linked dataset using R software, version 3.5.0 (The R Foundation). The data analysis was performed using SPSS Version 27 (IBM). A two-sided p value of 0.05 was used to indicate statistical significance.

Patients ≥ 75 years of age There were 1,683 patients 75 years and over. The HFRS weakly correlated with the CFS (Spearman’s rho = 0.22 [95% CI 0.18–0.27]; p < 0.0001) and had a poor agreement (Kappa = 0.05; 95% CI 0.01–0.08; p = 0.004; Additional file 1: Table S5). Although the AUROC curves had moderate discrimination the scores were similar for both CFS and HFRS (Additional file 1: Figure S4). The HFRS (5-unit increment; HR 1.05, 95% CI 0.95–1.16), adjusted for ANZROD and sex, was not independently associated with 1-year survival for HFRS when compared with CFS (1-unit increment; HR 1.18, 95% CI 1.10–1.27; Additional file 1: Table S6). Although the HFRS was independently predictive of only 90-day, 6-month and 1-year mortalities (Additional file 1: Table S7), the magnitude of prediction was weaker than the CFS prediction.

Patients needing mechanical ventilation 3266 patients received mechanical ventilation. While the HFRS did not significantly correlate with the CFS (Spearman’s rho = 0.02 (95% CI − 0.05, 0.02; p = 0.40), the AUROC curves indicated moderate discrimination with CFS AUROC lower than HFRS (0.60 vs 0.63 p < 0.0001; Additional file 1: Figure S4). After adjustment, both CFS (1-unit increment; HR 1.14; 95% CI 1.08–1.20) and HFRS (5-unit increment; HR 1.17, 95% CI 1.08–1.28) were independently associated with survival up to one year (Additional file 1: Table S6). While the CFS was predictive of both short- and long-term mortalities, HFRS was not (Additional file 1: Table S7).

Elective versus non-elective admissions 1740 patients were electively admitted, while 5261 were non-elective admissions. There were no differences in correlation, agreement, or 1-year survival based on their admission type (Additional file 1: Tables S5 and S6). The HFRS was independently predictive only for 90-day mortality for electively admitted patients, while the CFS was independently predictive only for 6-month and 1-year mortalities (Additional file 1: Table S7). Contrarily, the HFRS was not predictive of both short- and long-term mortalities for non-elective admissions, whereas the CFS was independently predictive of both short- and long-term mortalities (Additional file 1: Table S7).

Outcomes of patients who survived to hospital discharge Of the 6359 patients who were discharged home alive, 364 patients (5.7%) subsequently died (13.8% [154/1114] were frail and 4.0% [210/5245] were non-frail, based on CFS). The HFRS weakly correlated with the CFS (Spearman’s rho = 0.12 [95% CI 0.09–0.15]; p < 0.001) and had a poor agreement (Kappa = 0.12; 95% CI 0.07–0.14; p < 0.001; Additional file 1: Table S5). Both HFRS (5-unit increment; HR 1.13, 95% CI 1.02–1.25) and CFS (1-unit increment; HR 1.51, 95% CI 1.42–1.61), adjusted for ANZROD and sex, were independently associated with 1-year survival, but the magnitude of prediction for HFRS was weaker than the CFS prediction (Additional file 1: Table S6). Although the HFRS was independently predictive of only 90-day, 6-month and 1-year mortalities (Additional file 1: Table S7), the magnitude of prediction was weaker than the CFS prediction.

This large multicentre retrospective cohort study demonstrated both CFS and HFRS independently predicted 1-year survival with moderate discrimination. The CFS had better prediction than the HFRS. Both frailty measures were independently predictive and moderately discriminatory in differentiating long-term survivors from patients who died up to one year after ICU admission. The use of CFS to categorise patients as frail resulted in greater separation of Kaplan Meier curves. Neither CFS nor HFRS provided additional discriminatory value over and above that provided by the acute assessment of illness severity measured using ANZROD, however, the increased area under the curve for the combination of the two measures suggested they may both be measuring different aspects of frailty. Finally, frailty diagnosed using the CFS was predictive of both short- and long-term mortality in mechanically ventilated patients and those who were admitted non-electively, but HFRS was not.

There is no gold standard for frailty measure in critically ill patients. Diagnostic confirmation of frailty requires a comprehensive assessment across a spectrum of contributing domains, which are impractical and challenging in ICU settings. The validated HFRS, on the contrary, is estimated using administrative data. Both CFS and HFRS are readily interpreted by non-geriatric specialists and hence well suited for screening at-risk patients with frailty [23]. However, there is controversy of the performance of these two frailty measures.

The lower prevalence of HFRS-categorized frailty in our study compares with the reported literature (58–67% in hospitalised patients [13,24], and 59% in an ICU cohort [25]). The higher published prevalence is likely because the HFRS heavily depends on simple counts of co-morbidities for frailty classification generally without being able to differentiate pre-existing conditions from those which develop or are identified de-novo during the hospitalisation. By only including the pre-existing ICD-10 codes to determine HFRS, we observed that the prevalence of frailty was comparable to CFS-categorized frailty (26.2% and 18.9%, respectively) in our study. This was also relative to the recent meta-analysis that pooled ten observational studies that estimated a frailty prevalence of 30% among patients in ICU settings.

Higher age, comorbidity burden and primary diagnosis have previously been related to decreased survival in all patient groups following ICU admission [26,27]. CFS has been validated to predict short-term and long-term mortality in critically ill patients [4,6,11,28‐31]. HFRS however relies on patient records. ICD-10 codes cannot reflect disease severity; they are normally used for reimbursement purposes. We observed that the CFS had greater predictive validity of long-term survival than the HFRS, further strengthening the argument that the CFS is a better and more reliable frailty screening measure in critically ill patients. The CFS meets all the criteria: i.e., being multi-dimensional, time-efficient, accurate and also simple. While the HFRS might be useful to assess morbidity, however, frailty is more than just the sum of comorbidities. Although the HFRS has been shown to predict ICU readmission [25,32], it relies on patient records, which are prone to be incomplete or possibly incorrect. As previously observed by Flaatten and colleagues [2], our study found that it significantly fell short in evaluating the true burden of frailty.

The HFRS, created for hospitalised but not ICU patients, was originally developed as low-risk, medium-risk, and high-risk (0–5, 5–15, > 15) and not dichotomised. The ICD-10 codes were arbitrarily selected. Furthermore, HFRS uses weights (i.e., points for each diagnosis) based on the coefficients of likelihood ratios in UK cohorts; thus, it may not be appropriate to use exactly the same weight in Australian ICU populations. For instance, 2.3 points for "Other disorders of fluid, electrolyte and acid–base balance" (observed in 33% of our cohort). Similarly, 3.2 points for "Delirium, not induced by alcohol and other psychoactive substances" (observed in 12% of our cohort). Future studies should focus on re-calculating the weight for each diagnosis using a larger Australian cohort.

Previous evidence has exhibited the challenges in comparing results of different frailty scales, with a 2017 review of 35 different frailty scores demonstrating significantly different degrees of agreement between scales in a longitudinal study of over 5000 older community-dwelling participants [33] however, HFRS was not included in that review. Our study demonstrated poor agreements between the CFS and the HFRS. The most likely explanation is that both measures estimate frailty based on different concepts of frailty.

Although the HFRS correlated weakly with poor agreements and a lower magnitude of independently predicting longer-term survival than the CFS, both frailty measures have some validity and may be beneficial in different situations. These two frailty tools measure different aspects: the CFS is a clinical assessment, that is more likely to be immediately available to clinicians and does have some validity for differentiating long-term survivors from those who die, even before the calculation of acute illness severity. On the other hand, the HFRS is likely to be available to administrators and health departments since it is collected in and calculable from routine coding with the data. The CFS and ANZROD are less likely to be available than administrators and health departments. However, by using only the ICD-10 codes from only the pre-existing conditions, the frailty status could be extrapolated as being recorded at the time of the indexed hospitalisation. This implies that an automated HFRS could be made available when the patient is hospitalised again or readmitted to ICU. Furthermore, the combined HFRS and CFS AUROC model demonstrated that both measures could explain differing proportions of variation associated with outcome, so both would be useful if ANZROD was not available. Furthermore, as ANZROD requires 24 h of data whereas HFRS and CFS do not, a prediction model for death using frailty could potentially be developed earlier.

The strengths of the study included a large patient dataset along with long-term outcomes. Unlike previous evidence for HFRS, to the best of our knowledge, this was the first study to use ICD-10 codes only from the pre-existing conditions and not illnesses that might have developed (and coded) during the index hospitalisation. A few limitations need to be acknowledged. Firstly, the CFS measures patients’ frailty in the 2 months before ICU admission, however, the derived HFRS used in our study for only pre-existing conditions may pre-date index hospitalisation, and hence comparable with CFS. Secondly, there could be a possibility of inappropriate categorisation, which may be a consequence of how the data is collected. For example, historically, elective surgical patients are more likely to have comorbidities listed because they often have had a preoperative workup, while the past medical histories for emergency patients were often incomplete [34]. Although the HFRS is calculated at the time of discharge, sensitivity analysis on elective and non-elective admissions suggested that under-coding of comorbidities happened with non-elective admissions. This could have affected the HFRS scores and therefore the frailty status of patients [35]. Thirdly, with the advent of population and disease registers, more studies rely on such registries for gathering vital information; hence inadvertently making them prone to information bias due to either inaccurate measurement or documentation [36]. Fourthly, we were unable to determine inter-rater reliability for CFS in our study. However, previous studies investigating the inter-rater reliability of the CFS in critically ill patients showed that the inter-rater reliability was strong when intensive care clinicians assessed the CFS [37]. However, we were unable to determine inter-rater reliability for CFS in our study. Fifthly, not all data from the ANZICS-APD with the CFS were linked to VAED and VDI datasets. Sixthly, there was a significant variation in the CFS documentation (ranging between 7 and 100%) in the 16 hospitals that reported on the CFS with documentation of the CFS increasing over time (17% at commencement, 85% at completion; Additional file 1: Table S8). However, given there was little clinical difference between included and missing patient data (n = 13,457), our cohort was broadly representative of the larger population. Finally, the lower ICU mortality rates and short ICU length of stays are real-world outcomes for ANZ ICUs and may not be applicable to other countries.