C Blair Burnette1, Jessica L Luzier2,3, Brooke L Bennett4, Chantel M Weisenmuller2,3, Patrick Kerr2,3, Shelby Martin1, Jillian Keener2,3, Lisa Calderwood2. 1. Charleston Area Medical Center, Charleston, West Virginia, USA. 2. Charleston Area Medical Center - Institute for Academic Medicine, Charleston, West Virginia, USA. 3. Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine-Charleston Division, Charleston, West Virginia, USA. 4. Yale University School of Medicine, Department of Psychiatry, New Haven, Connecticut, USA.
Abstract
OBJECTIVE: Our original aim was to validate and norm common eating disorder (ED) symptom measures in a large, representative community sample of transgender adults in the United States. We recruited via Amazon Mechanical Turk (MTurk), a popular online recruitment and data collection platform both within and outside of the ED field. We present an overview of our experience using MTurk. METHOD: Recruitment began in Spring 2020; our original target N was 2,250 transgender adults stratified evenly across the United States. Measures included a demographics questionnaire, the Eating Disorder Examination-Questionnaire, and the Eating Attitudes Test-26. Consistent with current literature recommendations, we implemented a comprehensive set of attention and validity measures to reduce and identify bot responding, data farming, and participant misrepresentation. RESULTS: Recommended validity and attention checks failed to identify the majority of likely invalid responses. Our collection of two similar ED measures, thorough weight history assessment, and gender identity experiences allowed us to examine response concordance and identify impossible and improbable responses, which revealed glaring discrepancies and invalid data. Furthermore, qualitative data (e.g., emails received from MTurk workers) raised concerns about economic conditions facing MTurk workers that could compel misrepresentation. DISCUSSION: Our results strongly suggest most of our data were invalid, and call into question results of recently published MTurk studies. We assert that caution and rigor must be applied when using MTurk as a recruitment tool for ED research, and offer several suggestions for ED researchers to mitigate and identify invalid data.
OBJECTIVE: Our original aim was to validate and norm common eating disorder (ED) symptom measures in a large, representative community sample of transgender adults in the United States. We recruited via Amazon Mechanical Turk (MTurk), a popular online recruitment and data collection platform both within and outside of the ED field. We present an overview of our experience using MTurk. METHOD: Recruitment began in Spring 2020; our original target N was 2,250 transgender adults stratified evenly across the United States. Measures included a demographics questionnaire, the Eating Disorder Examination-Questionnaire, and the Eating Attitudes Test-26. Consistent with current literature recommendations, we implemented a comprehensive set of attention and validity measures to reduce and identify bot responding, data farming, and participant misrepresentation. RESULTS: Recommended validity and attention checks failed to identify the majority of likely invalid responses. Our collection of two similar ED measures, thorough weight history assessment, and gender identity experiences allowed us to examine response concordance and identify impossible and improbable responses, which revealed glaring discrepancies and invalid data. Furthermore, qualitative data (e.g., emails received from MTurk workers) raised concerns about economic conditions facing MTurk workers that could compel misrepresentation. DISCUSSION: Our results strongly suggest most of our data were invalid, and call into question results of recently published MTurk studies. We assert that caution and rigor must be applied when using MTurk as a recruitment tool for ED research, and offer several suggestions for ED researchers to mitigate and identify invalid data.
Authors: Shannon M Christy; Mariana Arevalo; Naomi C Brownstein; Junmin Whiting; Cathy D Meade; Clement K Gwede; Susan T Vadaparampil; Kristin J Tillery; Jessica Y Islam; Anna R Giuliano Journal: JMIR Form Res Date: 2022-06-23
Authors: Michael V Bronstein; Jonas Everaert; Erich Kummerfeld; Ann F Haynos; Sophia Vinogradov Journal: Int J Eat Disord Date: 2022-02-07 Impact factor: 5.791
Authors: Naomi C Brownstein; Harika Reddy; Junmin Whiting; Monica L Kasting; Katharine J Head; Susan T Vadaparampil; Anna R Giuliano; Clement K Gwede; Cathy D Meade; Shannon M Christy Journal: Prev Med Date: 2022-04-07 Impact factor: 4.637