PURPOSE: Interrater disagreements in ratings of quality plague the study of voice. This study compared 2 methods for handling this variability. METHOD: Listeners provided multiple breathiness ratings for 2 sets of pathological voices, one including 20 male and 20 female voices unselected for quality and one including 20 breathy female voices. Ratings for each listener were averaged together, mean ratings were z transformed, and the likelihood that 2 listeners would agree exactly in their ratings was calculated as a function of averaging and standardizing condition. Data were also multidimensionally scaled to examine similarities among listeners in perceptual strategy. Results were compared with parallel analyses of existing breathiness ratings of the same voices gathered using a method-of-adjustment task. RESULTS: Three-way interactions between the mean rating for a voice, standardization condition, and the number of voices averaged together were observed, but no main effect of averaging condition emerged. Multidimensional scaling revealed significant residual differences in perceptual strategy across listeners after averaging and standardizing. Ratings from the method-of-adjustment task showed both high agreement levels and consistent perceptual strategies across listeners, as theoretically predicted. CONCLUSION: Averaging multiple ratings and standardizing the mean are inadequate in addressing variations in voice quality perception.
PURPOSE: Interrater disagreements in ratings of quality plague the study of voice. This study compared 2 methods for handling this variability. METHOD: Listeners provided multiple breathiness ratings for 2 sets of pathological voices, one including 20 male and 20 female voices unselected for quality and one including 20 breathy female voices. Ratings for each listener were averaged together, mean ratings were z transformed, and the likelihood that 2 listeners would agree exactly in their ratings was calculated as a function of averaging and standardizing condition. Data were also multidimensionally scaled to examine similarities among listeners in perceptual strategy. Results were compared with parallel analyses of existing breathiness ratings of the same voices gathered using a method-of-adjustment task. RESULTS: Three-way interactions between the mean rating for a voice, standardization condition, and the number of voices averaged together were observed, but no main effect of averaging condition emerged. Multidimensional scaling revealed significant residual differences in perceptual strategy across listeners after averaging and standardizing. Ratings from the method-of-adjustment task showed both high agreement levels and consistent perceptual strategies across listeners, as theoretically predicted. CONCLUSION: Averaging multiple ratings and standardizing the mean are inadequate in addressing variations in voice quality perception.
Authors: Attila Andics; James M McQueen; Karl Magnus Petersson; Viktor Gál; Gábor Rudas; Zoltán Vidnyánszky Journal: Neuroimage Date: 2010-05-27 Impact factor: 6.556
Authors: Raymond D Kent; Julie Eichhorn; Erin M Wilson; Youmi Suk; Daniel M Bolt; Houri K Vorperian Journal: J Speech Lang Hear Res Date: 2021-03-31 Impact factor: 2.297