Anneliese Arno1, Julian Elliott2, Byron Wallace3, Tari Turner2, James Thomas4. 1. EPPI-Centre, UCL Social Science Research Institute, University College London, London, UK. anneliese.arno.17@ucl.ac.uk. 2. School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia. 3. Khoury College of Computer Sciences, Northeastern University, Boston, USA. 4. EPPI-Centre, UCL Social Science Research Institute, University College London, London, UK.
Abstract
BACKGROUND: The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis-systematic reviews and health guidelines-to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore, their opinions on the potential use of automation are crucial. METHODS: The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage, Compatibility, Complexity, Trialability, and Observability. Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers' Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach. RESULTS: Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e., Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in the methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants. CONCLUSIONS: If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.
BACKGROUND: The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis-systematic reviews and health guidelines-to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore, their opinions on the potential use of automation are crucial. METHODS: The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage, Compatibility, Complexity, Trialability, and Observability. Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers' Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach. RESULTS:Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e., Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in the methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants. CONCLUSIONS: If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.
Authors: Kaveh G Shojania; Margaret Sampson; Mohammed T Ansari; Jun Ji; Steve Doucette; David Moher Journal: Ann Intern Med Date: 2007-07-16 Impact factor: 25.391
Authors: Annette M O'Connor; Guy Tsafnat; Stephen B Gilbert; Kristina A Thayer; Ian Shemilt; James Thomas; Paul Glasziou; Mary S Wolfe Journal: Syst Rev Date: 2019-02-20
Authors: Julian H Elliott; Tari Turner; Ornella Clavisi; James Thomas; Julian P T Higgins; Chris Mavergames; Russell L Gruen Journal: PLoS Med Date: 2014-02-18 Impact factor: 11.069
Authors: Patricia Sofia Jacobsen Jardim; Christopher James Rose; Heather Melanie Ames; Jose Francisco Meneses Echavez; Stijn Van de Velde; Ashley Elizabeth Muller Journal: BMC Med Res Methodol Date: 2022-06-08 Impact factor: 4.612
Authors: Candyce Hamel; Mona Hersi; Shannon E Kelly; Andrea C Tricco; Sharon Straus; George Wells; Ba' Pham; Brian Hutton Journal: BMC Med Res Methodol Date: 2021-12-20 Impact factor: 4.615