Gerald Gartlehner1, Isolde Sommer2, Tammeka Swinson Evans3, Kylie Thaler2, Kathleen N Lohr3. 1. Department for Clinical Epidemiology an Evidence-based Medicine, Austrian Cochrane Branch, Danube University Krems, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria; RTI-UNC Evidence-based Practice Center, Research Triangle Institute International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA. Electronic address: gerald.gartlehner@donau-uni.ac.at. 2. Department for Clinical Epidemiology an Evidence-based Medicine, Austrian Cochrane Branch, Danube University Krems, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria. 3. RTI-UNC Evidence-based Practice Center, Research Triangle Institute International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA.
Abstract
OBJECTIVES: We sought to determine whether producers or users of systematic reviews using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach or a close variation give the same meanings to terms intended to convey uncertainty about treatment effects when interpreting grades for the quality or strength of evidence. STUDY DESIGN AND SETTING: Following exploratory interviews with stakeholders and user testing, we conducted an international Web-based survey among producers and users of systematic reviews. For each quality grade (high, moderate, low, very low/insufficient), we asked participants to assign a minimum likelihood that treatment effects will not change substantially as new studies emerge. Using multivariate analysis of covariance, we tested whether the estimated likelihoods differed between producers and users. RESULTS: In all, 244 participants completed the survey. The associated minimum likelihoods that treatment effects will not change substantially for high, moderate, and low grades of quality of evidence (QOE) were 86% [95% confidence interval (CI): 85%, 87%], 61% (95% CI: 59%, 63%), and 34% (95% CI: 32%, 36%), respectively (very low was preset at 0%). Likelihoods for each grade were similar between producers and users of systematic reviews (P > 0.05 for all comparisons). CONCLUSION: GRADE is, in general, a suitable method to convey uncertainties for systematic review producers to users. The wide ranges of likelihoods associated with GRADE terms suggest that current definitions of levels of QOE that rely exclusively on qualitative certainty expressions should be augmented by numerical predictions once such data are available.
OBJECTIVES: We sought to determine whether producers or users of systematic reviews using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach or a close variation give the same meanings to terms intended to convey uncertainty about treatment effects when interpreting grades for the quality or strength of evidence. STUDY DESIGN AND SETTING: Following exploratory interviews with stakeholders and user testing, we conducted an international Web-based survey among producers and users of systematic reviews. For each quality grade (high, moderate, low, very low/insufficient), we asked participants to assign a minimum likelihood that treatment effects will not change substantially as new studies emerge. Using multivariate analysis of covariance, we tested whether the estimated likelihoods differed between producers and users. RESULTS: In all, 244 participants completed the survey. The associated minimum likelihoods that treatment effects will not change substantially for high, moderate, and low grades of quality of evidence (QOE) were 86% [95% confidence interval (CI): 85%, 87%], 61% (95% CI: 59%, 63%), and 34% (95% CI: 32%, 36%), respectively (very low was preset at 0%). Likelihoods for each grade were similar between producers and users of systematic reviews (P > 0.05 for all comparisons). CONCLUSION: GRADE is, in general, a suitable method to convey uncertainties for systematic review producers to users. The wide ranges of likelihoods associated with GRADE terms suggest that current definitions of levels of QOE that rely exclusively on qualitative certainty expressions should be augmented by numerical predictions once such data are available.
Authors: Benjamin Djulbegovic; Muhammad Muneeb Ahmed; Iztok Hozo; Despina Koletsi; Lars Hemkens; Amy Price; Rachel Riera; Paulo Nadanovsky; Ana Paula Pires Dos Santos; Daniela Melo; Ranjan Pathak; Rafael Leite Pacheco; Luis Eduardo Fontes; Enderson Miranda; David Nunan Journal: J Eval Clin Pract Date: 2022-01-28 Impact factor: 2.336