Ambuj Kumar1, Branko Miladinovic2, Gordon H Guyatt3, Holger J Schünemann3, Benjamin Djulbegovic4. 1. USF Health Program for Comparative Effectiveness Research, Division of Evidence Based Medicine, Department of Internal Medicine, Morsani College of Medicine, Tampa, FL, USA; Moffitt Cancer Center, 12902 USF Magnolia Drive, Tampa, FL 33612, USA. 2. USF Health Program for Comparative Effectiveness Research, Division of Evidence Based Medicine, Department of Internal Medicine, Morsani College of Medicine, Tampa, FL, USA. 3. Department of Clinical Epidemiology & Biostatistics, McMaster University Health Sciences Centre, Room 2C16 1280 Main Street, West Hamilton, ON L8S 4K1, Canada. 4. USF Health Program for Comparative Effectiveness Research, Division of Evidence Based Medicine, Department of Internal Medicine, Morsani College of Medicine, Tampa, FL, USA; Moffitt Cancer Center, 12902 USF Magnolia Drive, Tampa, FL 33612, USA; Tampa General Hospital, 1 Tampa General Circle, Tampa, FL 33606, USA. Electronic address: bdjulbeg@health.usf.edu.
Abstract
OBJECTIVES: The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) is a widely used methodology for the development of clinical practice guideline. Although its reproducibility is good for evaluating the quality of evidence, it has not been tested in context of developing recommendations. The objective of this study was to assess the reproducibility of all GRADE factors that determine the direction and strength of a recommendation among the guideline panel members with limited exposure to GRADE methodology. STUDY DESIGN AND SETTING: The study was conducted as part of the clinical practice guideline development process of American Association of Blood Banking for the use of prophylactic vs. therapeutic platelet transfusion in patients with thrombocytopenia. The results from the systematic review and meta-analysis for each question were summarized as a GRADE evidence profile. Interrater agreement for all GRADE factors and strength of recommendations was summarized using a weighted kappa statistic with 95% confidence intervals (CI). RESULTS: Eighteen members of the panel participated in the deliberation of making recommendations and completed the online questionnaire. They were given two 1-hour lectures about GRADE. The agreement for all domains was better than chance. The interrater agreement for the domain of quality of evidence was good (kappa value: 0.68; 95% CIs: 0.54, 0.84), and fair for balance of benefit and harms (kappa value: 0.4; 95% CIs: 0.25, 0.57) and use of resources (kappa value: 0.28: 95% CIs: 0.12, 0.42). The interrater agreement was moderate for the GRADE domain of patients' values and preferences (kappa value: 0.44; 95% CI: 0.31, 0.56). The interrater agreement for making a for/against recommendation was good (kappa value: 0.74; 95% CIs: 0.33, 0.91) and fair for strong/weak recommendation (kappa value: 0.39; 95% CIs: 0.18, 0.68). CONCLUSIONS: Although not all elements of GRADE system had good agreement, the interrater agreement for assessing the quality of evidence and issuing a recommendation of for vs. against among panel members who had limited exposure to GRADE methodology was good. This is probably because GRADE has operationalized these two areas in more detail than other domains. Further operationalization of all GRADE domains such as with the GRADE evidence to decision frameworks would likely improve its reproducibility.
OBJECTIVES: The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) is a widely used methodology for the development of clinical practice guideline. Although its reproducibility is good for evaluating the quality of evidence, it has not been tested in context of developing recommendations. The objective of this study was to assess the reproducibility of all GRADE factors that determine the direction and strength of a recommendation among the guideline panel members with limited exposure to GRADE methodology. STUDY DESIGN AND SETTING: The study was conducted as part of the clinical practice guideline development process of American Association of Blood Banking for the use of prophylactic vs. therapeutic platelet transfusion in patients with thrombocytopenia. The results from the systematic review and meta-analysis for each question were summarized as a GRADE evidence profile. Interrater agreement for all GRADE factors and strength of recommendations was summarized using a weighted kappa statistic with 95% confidence intervals (CI). RESULTS: Eighteen members of the panel participated in the deliberation of making recommendations and completed the online questionnaire. They were given two 1-hour lectures about GRADE. The agreement for all domains was better than chance. The interrater agreement for the domain of quality of evidence was good (kappa value: 0.68; 95% CIs: 0.54, 0.84), and fair for balance of benefit and harms (kappa value: 0.4; 95% CIs: 0.25, 0.57) and use of resources (kappa value: 0.28: 95% CIs: 0.12, 0.42). The interrater agreement was moderate for the GRADE domain of patients' values and preferences (kappa value: 0.44; 95% CI: 0.31, 0.56). The interrater agreement for making a for/against recommendation was good (kappa value: 0.74; 95% CIs: 0.33, 0.91) and fair for strong/weak recommendation (kappa value: 0.39; 95% CIs: 0.18, 0.68). CONCLUSIONS: Although not all elements of GRADE system had good agreement, the interrater agreement for assessing the quality of evidence and issuing a recommendation of for vs. against among panel members who had limited exposure to GRADE methodology was good. This is probably because GRADE has operationalized these two areas in more detail than other domains. Further operationalization of all GRADE domains such as with the GRADE evidence to decision frameworks would likely improve its reproducibility.
Authors: Mohamed A Kharfan-Dabaja; Ambuj Kumar; Mehdi Hamadani; Stephan Stilgenbauer; Paolo Ghia; Claudio Anasetti; Peter Dreger; Emili Montserrat; Miguel-Angel Perales; Edwin P Alyea; Farrukh T Awan; Ernesto Ayala; Jacqueline C Barrientos; Jennifer R Brown; Januario E Castro; Richard R Furman; John Gribben; Brian T Hill; Mohamad Mohty; Carol Moreno; Susan O'Brien; Steven Z Pavletic; Javier Pinilla-Ibarz; Nishitha M Reddy; Mohamed Sorror; Christopher Bredeson; Paul Carpenter; Bipin N Savani Journal: Biol Blood Marrow Transplant Date: 2016-09-19 Impact factor: 5.742
Authors: Maria Cecilia Dedios; Alexo Esperato; Luz Maria De-Regil; Juan Pablo Peña-Rosas; Susan L Norris Journal: Implement Sci Date: 2017-03-21 Impact factor: 7.327