Majed Al-Jefri1,2, Roger Evans3, Joon Lee2,4,5, Pietro Ghezzi6. 1. Department of Medicine, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. 2. Data Intelligence for Health Lab, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. 3. School of Computing, Engineering and Mathematics, University of Brighton, Brighton, United Kingdom. 4. Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. 5. Department of Cardiac Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. 6. Brighton & Sussex Medical School, Falmer, Brighton, United Kingdom.
Abstract
Objective: Many online and printed media publish health news of questionable trustworthiness and it may be difficult for laypersons to determine the information quality of such articles. The purpose of this work was to propose a methodology for the automatic assessment of the quality of health-related news stories using natural language processing and machine learning. Materials and Methods: We used a database from the website HealthNewsReview.org that aims to improve the public dialogue about health care. HealthNewsReview.org developed a set of criteria to critically analyze health care interventions' claims. In this work, we attempt to automate the evaluation process by identifying the indicators of those criteria using natural language processing-based machine learning on a corpus of more than 1,300 news stories. We explored features ranging from simple n-grams to more advanced linguistic features and optimized the feature selection for each task. Additionally, we experimented with the use of pre-trained natural language model BERT. Results: For some criteria, such as mention of costs, benefits, harms, and "disease-mongering," the evaluation results were promising with an F1 measure reaching 81.94%, while for others the results were less satisfactory due to the dataset size, the need of external knowledge, or the subjectivity in the evaluation process. Conclusion: These used criteria are more challenging than those addressed by previous work, and our aim was to investigate how much more difficult the machine learning task was, and how and why it varied between criteria. For some criteria, the obtained results were promising; however, automated evaluation of the other criteria may not yet replace the manual evaluation process where human experts interpret text senses and make use of external knowledge in their assessment.
Objective: Many online and printed media publish health news of questionable trustworthiness and it may be difficult for laypersons to determine the information quality of such articles. The purpose of this work was to propose a methodology for the automatic assessment of the quality of health-related news stories using natural language processing and machine learning. Materials and Methods: We used a database from the website HealthNewsReview.org that aims to improve the public dialogue about health care. HealthNewsReview.org developed a set of criteria to critically analyze health care interventions' claims. In this work, we attempt to automate the evaluation process by identifying the indicators of those criteria using natural language processing-based machine learning on a corpus of more than 1,300 news stories. We explored features ranging from simple n-grams to more advanced linguistic features and optimized the feature selection for each task. Additionally, we experimented with the use of pre-trained natural language model BERT. Results: For some criteria, such as mention of costs, benefits, harms, and "disease-mongering," the evaluation results were promising with an F1 measure reaching 81.94%, while for others the results were less satisfactory due to the dataset size, the need of external knowledge, or the subjectivity in the evaluation process. Conclusion: These used criteria are more challenging than those addressed by previous work, and our aim was to investigate how much more difficult the machine learning task was, and how and why it varied between criteria. For some criteria, the obtained results were promising; however, automated evaluation of the other criteria may not yet replace the manual evaluation process where human experts interpret text senses and make use of external knowledge in their assessment.
Authors: R Moynihan; L Bero; D Ross-Degnan; D Henry; K Lee; J Watkins; C Mah; S B Soumerai Journal: N Engl J Med Date: 2000-06-01 Impact factor: 91.245
Authors: Elizabeth S Chen; George Hripcsak; Hua Xu; Marianthi Markatou; Carol Friedman Journal: J Am Med Inform Assoc Date: 2007-10-18 Impact factor: 4.497
Authors: Romana Haneef; Amélie Yavchitz; Philippe Ravaud; Gabriel Baron; Ivan Oransky; Gary Schwitzer; Isabelle Boutron Journal: BMJ Open Date: 2017-11-17 Impact factor: 2.692