The intrajudge reliability of the perceptual rating of cleft palate speech before and after pharyngeal flap surgery: the effect of judges and speech samples.

OBJECTIVE: In this pilot study, the reliabilities of the perceptual ratings of four types of speech samples by six judges, with and without expertise in evaluating cleft palate speech, were studied.
DESIGN: Pre- and postoperative tape recordings of 15 patients with cleft lip and palate who had undergone a superiorly based pharyngeal flap operation were selected. Five speech-language pathologists and one oral and maxillofacial surgeon perceptually rated the following variables on separate 100-mm visual analog scales: hypernasality, audible nasal emission, intelligibility, misarticulations associated with velopharyngeal insufficiency, voice quality, and the presence or absence of hyponasality. These six variables were rated in four types of speech samples: reading of three sentences, repeating after the speech pathologist of three sentences, 10 sentences containing the aforementioned material, and the same 10 sentences in paired comparison. All speech samples were rerated after 3 months by the same judges.
RESULTS: Judges differed largely in the range they used in their rating. Intrajudge reliability of .56 to .78 was found for ratings of hypernasality. No significant differences in intrajudge reliability were found for the ratings with the different types of speech samples. The intrajudge reliability of a judge with expertise was not necessarily higher than of a judge without this expertise.
CONCLUSIONS: The improvement in speech is most reliably assessed with speech samples in paired comparison. A speech-language pathologist with expertise in evaluating cleft palate speech does not guarantee a high intrajudge reliability of the rating.