Sean Grant1, Marika Booth2, Dmitry Khodyakov2. 1. RAND Corporation, 1776 Main Street, Santa Monica, CA 90401, USA. Electronic address: sgrant@rand.org. 2. RAND Corporation, 1776 Main Street, Santa Monica, CA 90401, USA.
Abstract
OBJECTIVES: To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes. STUDY DESIGN AND SETTING: Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels. RESULTS: Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus. CONCLUSION: Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.
OBJECTIVES: To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes. STUDY DESIGN AND SETTING: Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels. RESULTS: Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus. CONCLUSION: Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.
Authors: William T Gattrell; Amrit Pali Hungin; Amy Price; Christopher C Winchester; David Tovey; Ellen L Hughes; Esther J van Zuuren; Keith Goldman; Patricia Logullo; Robert Matheis; Niall Harrison Journal: Res Integr Peer Rev Date: 2022-06-07
Authors: Mark E Haaksma; Jasper M Smit; Alain Boussuges; Alexandre Demoule; Martin Dres; Giovanni Ferrari; Paolo Formenti; Ewan C Goligher; Leo Heunks; Endry H T Lim; Lidwine B Mokkink; Eleni Soilemezi; Zhonghua Shi; Michele Umbrello; Luigi Vetrugno; Emmanuel Vivier; Lei Xu; Massimo Zambon; Pieter R Tuinman Journal: Crit Care Date: 2022-04-08 Impact factor: 9.097
Authors: Toni Lange; Christian Kopkow; Jörg Lützner; Klaus-Peter Günther; Sascha Gravius; Hanns-Peter Scharf; Johannes Stöve; Richard Wagner; Jochen Schmitt Journal: BMC Med Res Methodol Date: 2020-02-10 Impact factor: 4.615