Wendy W Chapman1, John N Dowling, Michael M Wagner. 1. RODS Laboratory, Center for Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15213-2582, USA. chapman@cbmi.pitt.edu
Abstract
OBJECTIVE: To generate and measure the reliability for a reference standard set with representative cases from seven broad syndromic case definitions and several narrower syndromic definitions used for biosurveillance. DESIGN: From 527,228 eligible patients between 1990 and 2003, we generated a set of patients potentially positive for seven syndromes by classifying all eligible patients according to their ICD-9 primary discharge diagnoses. We selected a representative subset of the cases for chart review by physicians, who read emergency department reports and assigned values to 14 variables related to the seven syndromes. MEASUREMENTS: (1) Positive predictive value of the ICD-9 diagnoses; (2) prevalence of the syndromic definitions and related variables; (3) agreement between physician raters demonstrated by kappa, kappa corrected for bias and prevalence, and Finn's r; and (4) reliability of the reference standard classifications demonstrated by generalizability coefficients. RESULTS: Positive predictive value for ICD-9 classification ranged from 0.33 for botulinic to 0.86 for gastrointestinal. We generated between 80 and 566 positive cases for six of the seven syndromic definitions. Rash syndrome exhibited low prevalence (34 cases). Agreement between physician raters was high, with kappa > 0.70 for most variables. Ratings showed no bias. Finn's r was >0.70 for all variables. Generalizability coefficients were >0.70 for all variables but three. CONCLUSION: Of the 27 syndromes generated by the 14 variables, 21 showed high enough prevalence, agreement, and reliability to be used as reference standard definitions against which an automated syndromic classifier could be compared. Syndromic definitions that showed poor agreement or low prevalence include febrile botulinic syndrome, febrile and nonfebrile rash syndrome, respiratory syndrome explained by a nonrespiratory or noninfectious diagnosis, and febrile and nonfebrile gastrointestinal syndrome explained by a nongastrointestinal or noninfectious diagnosis.
OBJECTIVE: To generate and measure the reliability for a reference standard set with representative cases from seven broad syndromic case definitions and several narrower syndromic definitions used for biosurveillance. DESIGN: From 527,228 eligible patients between 1990 and 2003, we generated a set of patients potentially positive for seven syndromes by classifying all eligible patients according to their ICD-9 primary discharge diagnoses. We selected a representative subset of the cases for chart review by physicians, who read emergency department reports and assigned values to 14 variables related to the seven syndromes. MEASUREMENTS: (1) Positive predictive value of the ICD-9 diagnoses; (2) prevalence of the syndromic definitions and related variables; (3) agreement between physician raters demonstrated by kappa, kappa corrected for bias and prevalence, and Finn's r; and (4) reliability of the reference standard classifications demonstrated by generalizability coefficients. RESULTS: Positive predictive value for ICD-9 classification ranged from 0.33 for botulinic to 0.86 for gastrointestinal. We generated between 80 and 566 positive cases for six of the seven syndromic definitions. Rash syndrome exhibited low prevalence (34 cases). Agreement between physician raters was high, with kappa > 0.70 for most variables. Ratings showed no bias. Finn's r was >0.70 for all variables. Generalizability coefficients were >0.70 for all variables but three. CONCLUSION: Of the 27 syndromes generated by the 14 variables, 21 showed high enough prevalence, agreement, and reliability to be used as reference standard definitions against which an automated syndromic classifier could be compared. Syndromic definitions that showed poor agreement or low prevalence include febrile botulinic syndrome, febrile and nonfebrile rash syndrome, respiratory syndrome explained by a nonrespiratory or noninfectious diagnosis, and febrile and nonfebrile gastrointestinal syndrome explained by a nongastrointestinal or noninfectious diagnosis.
Authors: William B Lober; Bryant Thomas Karras; Michael M Wagner; J Marc Overhage; Arthur J Davidson; Hamish Fraser; Lisa J Trigg; Kenneth D Mandl; Jeremy U Espino; Fu-Chiang Tsui Journal: J Am Med Inform Assoc Date: 2002 Mar-Apr Impact factor: 4.497
Authors: M G Kortepeter; J A Pavlin; J C Gaydos; J R Rowe; P W Kelley; G Ludwig; K T McKee; E M Eitzen Journal: Mil Med Date: 2000-08 Impact factor: 1.437
Authors: Per H Gesteland; Michael M Wagner; Wendy W Chapman; Jeremy U Espino; Fu-Chiang Tsui; Reed M Gardner; Robert T Rolfs; Virginia Dato; Brent C James; Peter J Haug Journal: Proc AMIA Symp Date: 2002
Authors: Michael D Lewis; Julie A Pavlin; Jay L Mansfield; Sheilah O'Brien; Louis G Boomsma; Yevgeniy Elbert; Patrick W Kelley Journal: Am J Prev Med Date: 2002-10 Impact factor: 5.043
Authors: J A Jernigan; D S Stephens; D A Ashford; C Omenaca; M S Topiel; M Galbraith; M Tapper; T L Fisk; S Zaki; T Popovic; R F Meyer; C P Quinn; S A Harper; S K Fridkin; J J Sejvar; C W Shepard; M McConnell; J Guarner; W J Shieh; J M Malecki; J L Gerberding; J M Hughes; B A Perkins Journal: Emerg Infect Dis Date: 2001 Nov-Dec Impact factor: 6.883
Authors: Matthew J Scholer; George S Ghneim; Shiying Wu; Matt Westlake; Debbie A Travers; Anna E Waller; Anne-Lyne McCalla; Scott F Wetterhall Journal: AMIA Annu Symp Proc Date: 2007-10-11
Authors: Sylvain DeLisle; Brett South; Jill A Anthony; Ericka Kalp; Adi Gundlapallli; Frank C Curriero; Greg E Glass; Matthew Samore; Trish M Perl Journal: PLoS One Date: 2010-10-14 Impact factor: 3.240