Large-scale analysis of interobserver agreement and reliability in cardiotocography interpretation during labor using an online tool

We performed a cross-sectional online survey inviting clinicians to predict the fetal outcome based on CTG signals and clinical variables. We have built a publicly available web tool written in Python and compatible with most modern web browsers [18]. At the start of the study period, we shared the website widely via email through medical associations, and directly to heads of obstetrics and gynecology departments. The survey took place from November 2022 to January 2023. When browsing the website for the first time, the participant was asked to create an account by providing the following information: an anonymous and unique identifier, age, profession (obstetrician, resident, midwife or student midwife), number of years of experience (since diploma for non-students), place of practice (university hospital or general hospital) and country. We have considered asking additional information to the participants (for example whether they did specific trainings on CTG interpretation, they use scalp samples in clinical practice, and which CTG interpretation guidelines they use), but we decided to limit the number of information requested to simplify the use of the tool.

One hundred cases were displayed successively to the participant for annotation (Figure S1). The available information was the FHR and UC signals in the last 45 minutes before delivery and some relevant clinical information (sex, term of gestation and neonatal weight). We choose to use minimal clinical information to simplify the annotation process and to have as much labelled cases as possible. The layout of the tool mimics a standard paper graph (standard scale of 1cm per minute). For every case, the user was asked to predict the fetal outcome (normal outcome or fetal hypoxia) and to draw the FHR baseline. Only the results concerning the fetal outcome prediction have been presented in the current paper. The cases were assigned to each participant in a pseudorandom order: the same batches of 10 cases (5 with a normal outcome and 5 with fetal hypoxia) were presented to all participants but with a random order inside each batch. This method ensured a higher proximity in the cases annotated by the different participants than drawing them randomly among the 100 available cases. The participants were free to label as many cases as they wanted until 100. Upon each validation, users were informed of the correctness of their fetal outcome prediction. They were able to log out of their account and to sign in again later using their identifier to continue annotating.

The 100 cases were selected from the CTU-UHB dataset [16], which contains CTG signals (FHR and UC) and clinical information of deliveries occurring between 2010 and 2012 at the University hospital in Brno, Czech Republic. FHR signals were obtained by external ultrasound transducer or by direct scalp electrode, depending on the cases. The database represented 506 vaginal deliveries (with 44 operative deliveries), including 89 cases with fetal cord sample pH lower than 7.15. The full description of the selection process and the maternal and neonatal characteristics have been previously published [16]. Briefly, it includes patients over 18 who delivered singleton term fetuses with a second stage of labor lasting less than 30 minutes. Fetuses with known intrauterine growth restriction, malformation or infection were excluded. To complete the database, all the cases were annotated by nine Czech obstetric experts (named “the CTU-UHB experts”, hereafter) who predicted the labor outcome (e.g. estimated pH result for neonatal hypoxia) based on CTG signals and some clinical characteristics [14]. We randomly chose 50 cases exhibiting a pH level below 7.15 (corresponding to a moderate fetal hypoxia) and 50 other cases with a pH level higher than 7.15 (corresponding to a normal outcome). The pH threshold of 7.15 was the one used to discriminate normal from abnormal neonatal outcome in the CTU-UHB database [14]. We have kept the same threshold to facilitate the comparisons with the CTU-UHB experts annotations and also because it has been previously shown that even in case of moderate hypoxia there is an increased risk of fetal complication [19].

Based on the sensitivity and specificity of the prediction of the CTU-UHB experts, we performed a sample size calculation to determine the number of FHR needed in our study. With a sensitivity of 0.45, a specificity of 0.67, a precision in 95% confidence interval of 0.14 and a pre-specified prevalence of mild hypoxia of 50%, we determined that 97 cases are needed to conclude [20]. We have rounded the number to 100 cases including 50 cases of mild fetal hypoxia.

The primary outcome was the accuracy of the participants prediction. The secondary outcome was the interobserver agreement and reliability.

First, we performed a descriptive analysis of the participants and their annotations. Continuous variables were described using median and quartiles [Q1-Q3] and qualitative variables using numbers and percentages.

We analyzed the accuracy of the predicted outcomes for each participant and case using success rate (defined as the ratio of correctly predicted cases to the total number of cases), sensitivity and specificity. All metrics were accompanied by their respective 95% confidence intervals, computed using the Wilson method, which is more reliable when dealing with extreme proportion or small sample sizes [21]. The evaluation was carried out comprehensively for all annotations and according to the participants’ profession (residents, obstetrician-gynecologists, midwives) and number of years of experience (0 - 2 years, 2 - 4 years, 4 - 8 years and >8 years). We considered that there was a significant difference in accuracy when the 95% confidence intervals of success rate, sensitivity or specificity overlapped. Additionally, to illustrate the relationship between the success rate and the pH value at birth, the average success rate was evaluated across distinct ranges of pH levels defined by the following thresholds: 6.90, 6.98, 7.05, 7.13, 7.20, 7.28, 7.35 and 7.43. The thresholds were selected to ensure a balanced distribution of cases across each range.

We have also made a graphical analysis of the participants’ performance and interobserver variability by plotting the true positive rate (TPR, or sensitivity) against the false positive rate (FPR, or 1-specificity) for each participant. The size of the point corresponding to a participant is proportional to the number of annotations for this participant. We only show the participants with more than 10 annotations on this plot. We have extracted the existing annotations of the 100 cases included in our study by the nine CTU-UHB experts [14], which enabled to position them on the plot.

We evaluated the agreement and reliability between the professions using the proportion of agreement (PA) [22, 23] and Cohen’s Kappa coefficient (κ) respectively [24], as recommended in the literature [17, 25]. Agreement measures whether users’ annotations are similar. Reliability, on the other hand, corresponds to the ratio of the variability between the annotations of the same cases to the total variability of all annotations. 95% confidence intervals (CI) on those metrics were calculated. PA is defined as the proportion of cases for which the participants agreed on, and the Altman classification system [26] was employed to categorize the findings as follows: 0.81 to 1.00 indicates very good, 0.61 to 0.80 indicates good, 0.41 to 0.60 is moderate, 0.21 to 0.40 is fair, and a value below 0.2 is considered as a poor interobserver agreement. Moreover, if the lower boundary of the 95% CI for PA fell below 0.50, the agreement level was also regarded as non-significant [22]. Kappa quantifies the similarity between two sets of categorical ratings, adjusting for the degree of overlap that could happen randomly. It ranges from -1 to 1 with the following predefined interpretations: a value below 0.20 indicates slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 signifies substantial, and a value above 0.80 as almost perfect reliability [23, 24]. As kappa is sensitive to the prevalence, a low prevalence of pathological cases in the cohort could lead to a kappa close to zero, even if there is a high observed PA between practitioners [17, 27]. In our analysis, the prevalence is 50% as we have drawn the same number of cases in both groups, which makes the two metrics comparable. To calculate these metrics, the participants were categorized by their declared professions. Within each profession, we considered for each case annotated by at least one participant the most frequent annotation as the profession's consensus. We then calculated the pairwise agreement and reliability between these consensus annotations, considering only the cases annotated by at least one participant in the two considered professions (overlapping cases). Finally, we derived an overall agreement measure by taking a weighted average of these pairwise values, where the weights reflected the number of overlapping cases for each pair of professions. This approach accounts for the varying contributions of different profession pairs to the overall agreement metric.

Statistical analyses were performed using Python along with its associated libraries (`scikit-learn` and `statsmodels` for statical analysis, `pandas` for data manipulation and `plotly` for visualization).

During the inclusion period there were 120 participants. Most of them were based in France (94), and 84% were affiliated with a university hospital. The participants were grouped per profession into 3 groups: residents [28], obstetrician-gynecologists (58) and midwives [22]. Only 2 participants identified as student midwives and they were aggregated with non-student midwives. A total of 2950 annotations were collected during the study, with a median of 11 annotations per participant (q1=5, q3=36). 12 participants annotated the 100 cases and 62 annotated more than 10 cases (Figure S2). The midwives exhibited the highest median participation with 16 annotations, followed by the obstetrician-gynecologists and residents with 9 annotations in both groups (Table 1).

The overall success rate, sensitivity and specificity were 0.58 (95% CI 0.56-0.60), 0.58 (95% CI 0.56-0.60) and 0.63 (95% CI 0.61-0.65) respectively (Table 2). The mean success rate in the different groups varied from 0.55 (obstetricians) to 0.61 (midwives), with no statistically significant difference observed between them. The success rate did not significantly depend on the number of years of experience (Figure S3): it was 0.59 (95% CI 0.55-0.64), 0.55 (95% CI 0.51-0.60), 0.58 (95% CI 0.54-0.62) and 0.55 (95% CI 0.51-0.60) in the 0 - 2 years, 2 – 4 years, 4 – 8 years and >8 years groups respectively. On the same 100 cases, the success rate of the nine CTU-UHB experts was 0.56 (95% CI 0.53-0.60), with no significant difference with our results. The sensitivity and specificity were 0.43 (95% CI 0.40-0.46) and 0.68 (95% CI 0.65-0.71) respectively, showing a lower specificity and higher specificity. Figure 1 illustrates those findings.

Notably, the lowest success rate was obtained on signals around our pH threshold at 7.15: 0.44 for recordings within the 7.05-7.13 pH range, and 0.51 for recordings within the 7.13-7.20 range (Fig. 2). The highest success rate (0.80) was obtained on signals exhibiting pH values below 6.98. For signals with a pH value above 7.20, the success rate varied between 0.63 and 0.68.

We found a good agreement between participants (PA=0.82, 95% IC 0.68-0.96). The agreements between our different groups based on profession were similar (between 0.79 and 0.85). We found a substantial reliability among professionals (kappa=0.63, 95% IC 0.50-0.76), with similar values when evaluated per profession (between 0.58 and 0.68) Table 3.