Rating the Valence of Media Content about Electronic Cigarettes Using Crowdsourcing: Testing Rater Instructions and Estimating the Optimal Number of Raters.

Electronic cigarettes (e-cigarettes) are a controversial public health topic due to their increasing popularity among youth and the uncertainty about their risks and benefits. Researchers have started to assess the valence of media content about e-cigarette use, mostly using expert coding. The current study aims to offer a methodological framework and guideline when using crowdsourcing to rate the valence of e-cigarette media content. Specifically, we present (1) an experiment to determine rating instructions that would result in reliable valence ratings and (2) an analysis to identify the optimal number of raters needed to replicate these ratings. Specifically, we compared ratings produced by crowdsourced raters instructed to rate from several different perspectives (e.g., objective vs. subjective) and determined the instructions that led to reliable ratings. We then used bootstrapping methods and a set of criteria to identify the minimum number of raters needed to replicate these ratings. Results suggested that when rating e-cigarette valence, instructing raters to rate from their own subjective perspective produced reliable results, and nine raters were deemed the optimal number of raters. We expect these findings to inform future content analyses of e-cigarette valence. The study procedures can be applied to crowdsourced content analyses of other health-related media content to determine appropriate rating instructions and the number of raters.