Best Practices for Identifying Hospitalized Lower Respiratory Tract Infections Using Administrative Data: A Systematic Literature Review of Validation Studies

In this systematic review of 26 validation studies, we found that the sensitivity of ICD code-based algorithms for identifying LRTI cases among hospitalized adults was below 80% in two-thirds of studies, while specificity exceeded 90% in the majority. For all-cause LRTI studies, algorithms that included only pneumonia codes in primary position had a particular low sensitivity, which increased (with minimal loss in specificity) with the addition of codes for severe pneumonia, other LRTI-related infections, other respiratory diseases and free text search. The influence of coding position on sensitivity and specificity was unclear because algorithms differed in other characteristics that may have confounded this analysis. In HAP and pneumococcal-specific pneumonia, algorithms containing only nosocomial-specific or pathogen-specific ICD codes had poor sensitivity, which improved when broader pneumonia codes were added, in particular codes for unspecified organisms. The comparison of algorithms performance allowed us to derive some best practices that are summarized in Table 4.

Our review found that a majority of ICD-based algorithms reached a low sensitivity to identify LRTI cases among hospitalized adults. This finding is in line with two previous systematic reviews on hospitalized pneumonia, including one published during our review process [10], and one on hospital-acquired infections that included pneumonia [37]. The determination of the ICD codes that would improve the sensitivity of an algorithm is complex. The classical pneumonia codes were included in all algorithms, except four HAP-specific algorithms, but achieved low sensitivities overall. The addition of pneumonia severity codes (sepsis or respiratory failure) in primary position when pneumonia was in secondary position increased sensitivity by 5–15% with minimal changes to specificity in two studies on any pneumonia or CAP comparing different algorithms [11, 15]. The addition of other infections and/or respiratory codes was only explored in one study [11], in which sensitivity increased by 13% with the addition of other specific infections and respiratory codes. The limited impact of adding those codes was explained by the few patients who were assigned such codes, as these tend to be underreported in administrative databases [11]. Four studies reported the frequency of specific codes within the pneumonia and influenza group and revealed that the majority of ICD codes are rarely reported in hospitalized adult patients [13, 21, 29, 32]. One exception is the code for “organism unspecified” pneumonia cases (ICD-9 486 or ICD-10 J18.9), which was reported in 51–65% and 39–92% of LRTI or pneumonia cases in primary or any position, respectively, likely due to lack of etiological diagnosis at the time of coding [13, 21, 29, 32]. Therefore, it is essential to include the “organism unspecified” pneumonia code in studies aiming to measure LRTI incidence rates.

The influence of code position on sensitivity and specificity is poorly documented. Only one study on all-cause CAP compared pneumonia codes in primary vs. any position and reported a mild (8%) increase in sensitivity, but specificity was not reported [32]. Interestingly, several studies stated that pneumonia codes in secondary position better identify patients with severe pneumonia and underlying diseases [11, 27, 32], as pneumonia may be reported in secondary position only when it occurs in patients with other diseases, such as congestive heart failure or chronic bronchitis [11]. While not a formal validation study, one recently presented analysis in the UK found that ICD code-based pneumonia incidence using the first five positions correlated highly with estimates from a concurrent prospective study, but agreement decreases with age [38]. This suggests that algorithms aiming at deriving incidence rates of community-acquired LRTI cases in elderly patients should include pneumonia codes in any position, as this age group has a higher probability of presenting underlying diseases and severe pneumonia.

The distinction between CAP and HAP cases based on ICD codes is complex. In studies using the full medical file, the distinction is usually based on a time criterion, i.e. HAP is often defined as pneumonia with onset reported 24 or 48 h after hospital admission to be distinguished from CAP. Studies based on ICD databases only cannot apply this criterion because time after admission is not translated into ICD codes. There is also a lack of a specific code for HAP in ICD-9 and ICD-10, though some countries such as Germany and Switzerland introduced their own code. The ICD studies used different criteria to distinguish HAP from CAP, such as code position, specific HAP code, or pneumonia code not present at admission, but the performance of these criteria was not measured in these studies. Three studies provided some data on misclassification: 24% of all ICD-coded pneumonia cases were found to be HAP after clinical review [15], HAP represented 50% of misclassified cases in one CAP study [18], and CAP represented 25% of misclassified cases in one HAP study [35]. This highlights the need to introduce clear HAP codes in future ICD coding systems. The ICD-11 version, which has come into effect in 2022, includes an additional code for nosocomial origin (i.e. XB25) but it remains to be seen if it will be used in practice and how it will perform.

The added value of including or excluding ICD codes for aspiration pneumonia in algorithms is unclear. Aspiration, which is the inhalation of oropharyngeal or gastric content into the lower respiratory tract, may lead to aspiration pneumonitis, which is a chemical injury, and to aspiration pneumonia, which accounts for 5–15% of CAP cases [11]. Although there is some overlap between these syndromes, they are distinct clinical entities [39] but with similar microbiologic etiology [40]. Aspiration pneumonitis is often difficult to distinguish from CAP [39]. The seven algorithms including aspiration pneumonia codes covered different types of LRTI (any LRTI, any pneumonia, CAP, and HAP), those excluding it were CAP algorithms, and the performance of including or excluding it has not been measured separately. Its use is also controversial in the literature [11, 21, 22]. On one hand, including aspiration pneumonia codes may add patients who have pneumonitis instead of lung infection. On the other hand, its exclusion in CAP studies risks eliminating true CAP cases in nursing home residents or other frail patients [11]. Studies excluding aspiration codes may thus exclude a particular spectrum of patients.

Our review highlights the lack of a standard for the reference used to confirm true positive and negative in cases detected by ICD codes, as illustrated by the wide variety of criteria and processes, i.e. manual review of medical charts, diagnosis of the treating physician, both with or without explicit clinical criteria and standard forms. Only ten out of 26 used explicit clinical criteria in the process. A previous SLR also pointed to the variety of criteria and, in particular, to the lack of reproducibility and reliability of using the physician diagnosis of LRTI noted in the medical chart by manual review [10]. This process may be prone to subjective interpretation based on unspecific symptoms and/or clinical, laboratory, and/or imaging findings. However, LRTIs can have atypical presentations in the frail and elderly development and such events may be missed by standardized clinical case definitions. Further, ICD code incidence studies primarily aim to capture diagnoses disease in line with current practice which involved physician judgment. Specifically for methodological studies, greater standardization of the reference process would be helpful for future validation studies of ICD code algorithms to allow a quantitative combination of results among studies.

Our review of 26 eligible studies and 18 studies with sensitivity and/or specificity measures is the first one to our knowledge to cover any LRTI, from community-acquired as well as hospital-acquired origin. Our added value is to compare performance of the different algorithms to inform best practices and recommendations for future protocols for ICD-based studies to derive LRTI incidence rates. A main limitation is the lack of reproducibility and reliability of the clinical review used as reference standard to confirm pneumonia in a majority of validation studies. Standardization of criteria as a minimum, and prospective studies using explicit criteria to confirm LRTI, including laboratory testing to confirm pathogen-specific CAP, would address that limitation. The diversity of study characteristics and heterogeneity across patients (and their expected prevalence of pneumonia) also limited the inference of our results to other populations and settings. We also expect that our findings are biased by the influence of reimbursement rules, as many countries base their hospital financing on Diagnosis Related Groups which include ICD codes, but we found no data to estimate the extent of this influence. The poor quality of some studies (41% in high risk of bias in at least one domain) limits the robustness of our conclusions. Lastly, many studies have identified that respiratory pathogens play a causative role in broader group of cardiopulmonary conditions than LRTI (e.g. chronic obstructive pulmonary disease and chronic heart failure) [41], and the recommendations here are only intended to capture the subset of respiratory pathogen illness recognized as LRTI by the treating clinicians; the extended impact of these would be best captured by time-series modelling studies [42, 43].