OBJECTIVE: Few previous studies evaluating the benefits of diagnostic decision support systems have simultaneously measured changes in diagnostic quality and clinical management prompted by use of the system. This report describes a reliable and valid scoring technique to measure the quality of clinical decision plans in an acute medical setting, where diagnostic decision support tools might prove most useful. DESIGN: Sets of differential diagnoses and clinical management plans generated by 71 clinicians for six simulated cases, before and after decision support from a Web-based pediatric differential diagnostic tool (ISABEL), were used. MEASUREMENTS: A composite quality score was calculated separately for each diagnostic and management plan by considering the appropriateness value of each component diagnostic or management suggestion, a weighted sum of individual suggestion ratings, relevance of the entire plan, and its comprehensiveness. The reliability and validity (face, concurrent, construct, and content) of these two final scores were examined. RESULTS: Two hundred fifty-two diagnostic and 350 management suggestions were included in the interrater reliability analysis. There was good agreement between raters (intraclass correlation coefficient, 0.79 for diagnoses, and 0.72 for management). No counterintuitive scores were demonstrated on visual inspection of the sets. Content validity was verified by a consultation process with pediatricians. Both scores discriminated adequately between the plans of consultants and medical students and correlated well with clinicians' subjective opinions of overall plan quality (Spearman rho 0.65, p < 0.01). The diagnostic and management scores for each episode showed moderate correlation (r = 0.51). CONCLUSION: The scores described can be used as key outcome measures in a larger study to fully assess the value of diagnostic decision aids, such as the ISABEL system.
OBJECTIVE: Few previous studies evaluating the benefits of diagnostic decision support systems have simultaneously measured changes in diagnostic quality and clinical management prompted by use of the system. This report describes a reliable and valid scoring technique to measure the quality of clinical decision plans in an acute medical setting, where diagnostic decision support tools might prove most useful. DESIGN: Sets of differential diagnoses and clinical management plans generated by 71 clinicians for six simulated cases, before and after decision support from a Web-based pediatric differential diagnostic tool (ISABEL), were used. MEASUREMENTS: A composite quality score was calculated separately for each diagnostic and management plan by considering the appropriateness value of each component diagnostic or management suggestion, a weighted sum of individual suggestion ratings, relevance of the entire plan, and its comprehensiveness. The reliability and validity (face, concurrent, construct, and content) of these two final scores were examined. RESULTS: Two hundred fifty-two diagnostic and 350 management suggestions were included in the interrater reliability analysis. There was good agreement between raters (intraclass correlation coefficient, 0.79 for diagnoses, and 0.72 for management). No counterintuitive scores were demonstrated on visual inspection of the sets. Content validity was verified by a consultation process with pediatricians. Both scores discriminated adequately between the plans of consultants and medical students and correlated well with clinicians' subjective opinions of overall plan quality (Spearman rho 0.65, p < 0.01). The diagnostic and management scores for each episode showed moderate correlation (r = 0.51). CONCLUSION: The scores described can be used as key outcome measures in a larger study to fully assess the value of diagnostic decision aids, such as the ISABEL system.
Authors: I D Adams; M Chan; P C Clifford; W M Cooke; V Dallos; F T de Dombal; M H Edwards; D M Hancock; D J Hewett; N McIntyre Journal: Br Med J (Clin Res Ed) Date: 1986-09-27
Authors: S J Nelson; M S Blois; M S Tuttle; M Erlbaum; P Harrison; H Kim; B Winkelmann; D Yamashita Journal: J Med Syst Date: 1985-12 Impact factor: 4.460
Authors: Mitchell J Feldman; Edward P Hoffer; G Octo Barnett; Richard J Kim; Kathleen T Famiglietti; Henry C Chueh Journal: J Grad Med Educ Date: 2012-06
Authors: Padmanabhan Ramnarayan; Graham C Roberts; Michael Coren; Vasantha Nanduri; Amanda Tomlinson; Paul M Taylor; Jeremy C Wyatt; Joseph F Britto Journal: BMC Med Inform Decis Mak Date: 2006-04-28 Impact factor: 2.796
Authors: Padmanabhan Ramnarayan; Andrew Winrow; Michael Coren; Vasanta Nanduri; Roger Buchdahl; Benjamin Jacobs; Helen Fisher; Paul M Taylor; Jeremy C Wyatt; Joseph Britto Journal: BMC Med Inform Decis Mak Date: 2006-11-06 Impact factor: 2.796