PURPOSE: The Quality Oncology Practice Initiative (QOPI) relies on the accuracy of manual abstraction of clinical data from paper-based and electronic medical records (EMRs). Although there is no "gold standard" to measure manual abstraction accuracy, measurement of inter-annotator agreement (IAA) is a commonly agreed-on surrogate. We quantified the IAA of QOPI abstractions on a cohort of cancer patients treated at Beth Israel Deaconess Medical Center. METHODS: The EMR charts of 49 patients (20 colorectal cancer; 18 breast cancer; 11 non-Hodgkin lymphoma) were abstracted by separate physician abstractors in the fall 2010 and fall 2011 QOPI abstraction rounds. Cohen's kappa (κ) was calculated for encoded data; raw levels of agreement and magnitude of discrepancies were calculated for numeric and dated data. RESULTS: One hundred two data elements with 2,035 paired entries were analyzed. Overall IAA for the 1,496 coded entries was κ = 0.75; median IAA for n = 85 individual coded elements was κ = 0.84 (interquartile range, 0.30 to 1.00). Overall IAA for the 421 dated entries was 73%; median IAA for n = 17 individual dated elements was 67% (interquartile range, 61% to 86%). CONCLUSION: This study establishes a baseline level of IAA for a complex medical abstraction task with clear relevance for the oncology community. Given that the observed κ is considered only fair IAA, and that the rate of date discrepancy is high, caution is necessary in interpreting the results of QOPI and other manual abstractions of clinical oncology data. The accuracy of automated data extraction efforts, possibly including a future evolution of QOPI, will also need to be carefully evaluated.
PURPOSE: The Quality Oncology Practice Initiative (QOPI) relies on the accuracy of manual abstraction of clinical data from paper-based and electronic medical records (EMRs). Although there is no "gold standard" to measure manual abstraction accuracy, measurement of inter-annotator agreement (IAA) is a commonly agreed-on surrogate. We quantified the IAA of QOPI abstractions on a cohort of cancerpatients treated at Beth Israel Deaconess Medical Center. METHODS: The EMR charts of 49 patients (20 colorectal cancer; 18 breast cancer; 11 non-Hodgkin lymphoma) were abstracted by separate physician abstractors in the fall 2010 and fall 2011 QOPI abstraction rounds. Cohen's kappa (κ) was calculated for encoded data; raw levels of agreement and magnitude of discrepancies were calculated for numeric and dated data. RESULTS: One hundred two data elements with 2,035 paired entries were analyzed. Overall IAA for the 1,496 coded entries was κ = 0.75; median IAA for n = 85 individual coded elements was κ = 0.84 (interquartile range, 0.30 to 1.00). Overall IAA for the 421 dated entries was 73%; median IAA for n = 17 individual dated elements was 67% (interquartile range, 61% to 86%). CONCLUSION: This study establishes a baseline level of IAA for a complex medical abstraction task with clear relevance for the oncology community. Given that the observed κ is considered only fair IAA, and that the rate of date discrepancy is high, caution is necessary in interpreting the results of QOPI and other manual abstractions of clinical oncology data. The accuracy of automated data extraction efforts, possibly including a future evolution of QOPI, will also need to be carefully evaluated.
Authors: Francis X Campion; Leanne R Larson; Pamela J Kadlubek; Craig C Earle; Michael N Neuss Journal: J Oncol Pract Date: 2011-05 Impact factor: 3.840
Authors: S Trent Rosenbloom; Joshua C Denny; Hua Xu; Nancy Lorenzi; William W Stead; Kevin B Johnson Journal: J Am Med Inform Assoc Date: 2011-01-12 Impact factor: 4.497
Authors: Bruce J Roth; Lada Krilov; Sylvia Adams; Carol A Aghajanian; Peter Bach; Fadi Braiteh; Marcia S Brose; Lee M Ellis; Harry Erba; Daniel J George; Mark R Gilbert; Joseph O Jacobson; Eric C Larsen; Stuart M Lichtman; Ann H Partridge; Jyoti D Patel; David I Quinn; Leslie L Robison; Jamie H von Roenn; Wolfram Samlowski; Gary K Schwartz; Nicholas J Vogelzang Journal: J Clin Oncol Date: 2012-12-03 Impact factor: 44.544
Authors: Michael N Neuss; Christopher E Desch; Kristen K McNiff; Peter D Eisenberg; Dean H Gesme; Joseph O Jacobson; Mohammad Jahanzeb; Jennifer J Padberg; John M Rainey; Jeff J Guo; Joseph V Simone Journal: J Clin Oncol Date: 2005-08-08 Impact factor: 44.544
Authors: Carol P Roth; Ian D Coulter; Lisa S Kraus; Gery W Ryan; Gary Jacob; Joyce S Marks; Eric L Hurwitz; Howard Vernon; Paul G Shekelle; Patricia M Herman Journal: J Manipulative Physiol Ther Date: 2019-06-27 Impact factor: 1.437
Authors: Tracy Spinks; Patricia A Ganz; George W Sledge; Laura Levit; James A Hayman; Timothy J Eberlein; Thomas W Feeley Journal: Healthc (Amst) Date: 2014-03-01
Authors: Fernando Suarez Saiz; Corey Sanders; Rick Stevens; Robert Nielsen; Michael Britt; Leemor Yuravlivker; Anita M Preininger; Gretchen P Jackson Journal: JCO Clin Cancer Inform Date: 2021-01