Ahmed Allam1, Peter J Schulz2, Michael Krauthammer3. 1. Department of Pathology, Yale University School of Medicine, New Haven, CT, USA. 2. Institute of Communication and Health, Faculty of Communication Sciences, University of Lugano (Università della Svizzera Italiana), Lugano, Switzerland. 3. Program for Computational Biology and Bioinformatics, Department of Pathology, Yale University School of Medicine, New Haven, CT, USA.
Abstract
BACKGROUND: As the Internet becomes the number one destination for obtaining health-related information, there is an increasing need to identify health Web pages that convey an accurate and current view of medical knowledge. In response, the research community has created multicriteria instruments for reliably assessing online medical information quality. One such instrument is DISCERN, which measures health Web page quality by assessing an array of features. In order to scale up use of the instrument, there is interest in automating the quality evaluation process by building machine learning (ML)-based DISCERN Web page classifiers. OBJECTIVE: The paper addresses 2 key issues that are essential before constructing automated DISCERN classifiers: (1) generation of a robust DISCERN training corpus useful for training classification algorithms, and (2) assessment of the usefulness of the current DISCERN scoring schema as a metric for evaluating the performance of these algorithms. METHODS: Using DISCERN, 272 Web pages discussing treatment options in breast cancer, arthritis, and depression were evaluated and rated by trained coders. First, different consensus models were compared to obtain a robust aggregated rating among the coders, suitable for a DISCERN ML training corpus. Second, a new DISCERN scoring criterion was proposed (features-based score) as an ML performance metric that is more reflective of the score distribution across different DISCERN quality criteria. RESULTS: First, we found that a probabilistic consensus model applied to the DISCERN instrument was robust against noise (random ratings) and superior to other approaches for building a training corpus. Second, we found that the established DISCERN scoring schema (overall score) is ill-suited to measure ML performance for automated classifiers. CONCLUSION: Use of a probabilistic consensus model is advantageous for building a training corpus for the DISCERN instrument, and use of a features-based score is an appropriate ML metric for automated DISCERN classifiers. AVAILABILITY: The code for the probabilistic consensus model is available at https://bitbucket.org/A_2/em_dawid/ .
BACKGROUND: As the Internet becomes the number one destination for obtaining health-related information, there is an increasing need to identify health Web pages that convey an accurate and current view of medical knowledge. In response, the research community has created multicriteria instruments for reliably assessing online medical information quality. One such instrument is DISCERN, which measures health Web page quality by assessing an array of features. In order to scale up use of the instrument, there is interest in automating the quality evaluation process by building machine learning (ML)-based DISCERN Web page classifiers. OBJECTIVE: The paper addresses 2 key issues that are essential before constructing automated DISCERN classifiers: (1) generation of a robust DISCERN training corpus useful for training classification algorithms, and (2) assessment of the usefulness of the current DISCERN scoring schema as a metric for evaluating the performance of these algorithms. METHODS: Using DISCERN, 272 Web pages discussing treatment options in breast cancer, arthritis, and depression were evaluated and rated by trained coders. First, different consensus models were compared to obtain a robust aggregated rating among the coders, suitable for a DISCERN ML training corpus. Second, a new DISCERN scoring criterion was proposed (features-based score) as an ML performance metric that is more reflective of the score distribution across different DISCERN quality criteria. RESULTS: First, we found that a probabilistic consensus model applied to the DISCERN instrument was robust against noise (random ratings) and superior to other approaches for building a training corpus. Second, we found that the established DISCERN scoring schema (overall score) is ill-suited to measure ML performance for automated classifiers. CONCLUSION: Use of a probabilistic consensus model is advantageous for building a training corpus for the DISCERN instrument, and use of a features-based score is an appropriate ML metric for automated DISCERN classifiers. AVAILABILITY: The code for the probabilistic consensus model is available at https://bitbucket.org/A_2/em_dawid/ .
Authors: G K Berland; M N Elliott; L S Morales; J I Algazy; R L Kravitz; M S Broder; D E Kanouse; J A Muñoz; J A Puyol; M Lara; K E Watkins; H Yang; E A McGlynn Journal: JAMA Date: 2001 May 23-30 Impact factor: 56.272
Authors: Bradford W Hesse; David E Nelson; Gary L Kreps; Robert T Croyle; Neeraj K Arora; Barbara K Rimer; Kasisomayajula Viswanath Journal: Arch Intern Med Date: 2005 Dec 12-26
Authors: Hege K Andreassen; Maria M Bujnowska-Fedak; Catherine E Chronaki; Roxana C Dumitru; Iveta Pudule; Silvina Santana; Henning Voss; Rolf Wynn Journal: BMC Public Health Date: 2007-04-10 Impact factor: 3.295
Authors: Elizabeth Murray; Bernard Lo; Lance Pollack; Karen Donelan; Joe Catania; Ken Lee; Kinga Zapert; Rachel Turner Journal: J Med Internet Res Date: 2003-08-29 Impact factor: 5.428
Authors: Simon M Mueller; Valentina N S Hongler; Pierre Jungo; Lucian Cajacob; Simon Schwegler; Esther H Steveling; Zita-Rose Manjaly Thomas; Oliver Fuchs; Alexander Navarini; Kathrin Scherer; Oliver Brandt Journal: J Med Internet Res Date: 2020-04-24 Impact factor: 5.428