Jacques Balayla1, Guy Shrem2. 1. Department of Obstetrics and Gynecology, McGill University, Montreal, QC, Canada. jacques.balayla@mail.mcgill.ca. 2. Department of Obstetrics and Gynecology, McGill University, Montreal, QC, Canada.
Abstract
OBJECTIVES: To determine the degree of inter-rater reliability (IRR) between human and artificial intelligence (AI) interpretation of fetal heart rate tracings (FHR), and to determine whether AI-assisted electronic fetal monitoring interpretation improves neonatal outcomes amongst laboring women. DATA SOURCES: We searched Medline, EMBASE, Google Scholar, Scopus, ISI Web of Science and Cochrane database search, as well as PubMed ( www.pubmed.gov ) and RCT registry ( www.clinicaltrials.gov ) until the end of October 2018 to conduct a systematic review and meta-analysis comparing visual and AI interpretation of EFM in labor. Similarly, we sought out all studies evaluating the IRR between AI and expert interpretation of EFM. TABULATION, INTEGRATION AND RESULTS: Weighed mean Cohen's Kappa was calculated to assess the global IRR. Risk of bias was assessed using the Cochrane Handbook for Systematic Reviews of Interventions. We used relative risks (RR) and a random effects (RE) model to calculate weighted estimates. Statistical homogeneity was checked by the χ2 test and I2 using Review Manager 5.3.5 (The Cochrane Collaboration, 2014.) We obtained 201 records, of which 9 met inclusion criteria. Three RCT's were used to compare the neonatal outcomes and 6 cohort studies were used to establish the degree of IRR between both approaches of EFM evaluation. With regards to the neonatal outcomes, a total of 55,064 patients were included in the analysis. Relative to the use of clinical (visual) evaluation of the FHR, the use of AI did not change the incidence rates of neonatal acidosis, cord pH below < 7.20, 5-min APGAR scores < 7, mode of delivery, NICU admission, neonatal seizures, or perinatal death. With regards to the degrees of inter-rater reliability, a weighed mean Cohen's Kappa of 0.49 [0.32-0.66] indicates moderate agreement between expert observers and computerized systems. CONCLUSION: The use of AI and computer analysis for the interpretation of EFM during labor does not improve neonatal outcomes. Inter-rater reliability between experts and computer systems is moderate at best. Future studies should aim at further elucidating these findings.
OBJECTIVES: To determine the degree of inter-rater reliability (IRR) between human and artificial intelligence (AI) interpretation of fetal heart rate tracings (FHR), and to determine whether AI-assisted electronic fetal monitoring interpretation improves neonatal outcomes amongst laboring women. DATA SOURCES: We searched Medline, EMBASE, Google Scholar, Scopus, ISI Web of Science and Cochrane database search, as well as PubMed ( www.pubmed.gov ) and RCT registry ( www.clinicaltrials.gov ) until the end of October 2018 to conduct a systematic review and meta-analysis comparing visual and AI interpretation of EFM in labor. Similarly, we sought out all studies evaluating the IRR between AI and expert interpretation of EFM. TABULATION, INTEGRATION AND RESULTS: Weighed mean Cohen's Kappa was calculated to assess the global IRR. Risk of bias was assessed using the Cochrane Handbook for Systematic Reviews of Interventions. We used relative risks (RR) and a random effects (RE) model to calculate weighted estimates. Statistical homogeneity was checked by the χ2 test and I2 using Review Manager 5.3.5 (The Cochrane Collaboration, 2014.) We obtained 201 records, of which 9 met inclusion criteria. Three RCT's were used to compare the neonatal outcomes and 6 cohort studies were used to establish the degree of IRR between both approaches of EFM evaluation. With regards to the neonatal outcomes, a total of 55,064 patients were included in the analysis. Relative to the use of clinical (visual) evaluation of the FHR, the use of AI did not change the incidence rates of neonatal acidosis, cord pH below < 7.20, 5-min APGAR scores < 7, mode of delivery, NICU admission, neonatal seizures, or perinatal death. With regards to the degrees of inter-rater reliability, a weighed mean Cohen's Kappa of 0.49 [0.32-0.66] indicates moderate agreement between expert observers and computerized systems. CONCLUSION: The use of AI and computer analysis for the interpretation of EFM during labor does not improve neonatal outcomes. Inter-rater reliability between experts and computer systems is moderate at best. Future studies should aim at further elucidating these findings.