Objective computer-based assessment of valleculae residue--is it useful?

There have been questions about the reliability of subjective rating scales used to assess valleculae residue from fluoroscopic images. The aim of this study was to assess interrater agreement on one such scale, and compare it with agreement using a new objective measurement scale. Five speech and language therapists rated 100 valleculae residue still images from 20 consecutive patients using standard clinical practice (i.e., subjective visual grading of the videofluoroscopy still and rating as none, mild, moderate, or severe). The images were rerated by the same clinicians using Picture Archiving Communication System measurement tools. The valleculae residue ratio relates the residue size to the size of an individual's valleculae. A valleculae residue ratio scale was devised using a linear classifier, which defines the cutoff between grades of valleculae residue (none, mild, moderate, and severe). The new method proved at least as reliable as the traditional method; for interrater reliability, kappa = 0.73 vs. 0.73; for intrarater reliability, kappa = 0.87 vs. 0.85. The valleculae residue ratio is proposed as a new quick reliable method of quantifying residue where the Picture Archiving Communication System is available. We now wish to test the impact of this method where poor inter- and intrarater reliability exists.