Robert H Lee1, Byron J Gajewski, Sarah Thompson. 1. Department of Health Policy and Management, University of Kansas Medical Center, Mail Stop 3044, 3901 Rainbow Boulevard, Kansas City, KS 66160, USA. rlee2@kumc.edu
Abstract
PURPOSE: We designed this study to examine the reliability of the nursing home survey process in the state of Kansas using regular and simultaneous survey teams. In particular, the study examined how two survey teams exposed to the same information at the same time differed in their interpretations. DESIGN AND METHODS: The protocol for simultaneous surveys consists of having one in-region and one out-of-region team survey a facility together. RESULTS: The regular and simultaneous survey teams generally agreed about the number of deficiencies. The intraclass correlation coefficient was 0.87 for total deficiencies and 0.76 for deficiencies with scores of G or higher. But in a substantial number of instances the teams did not agree about the scope and severity of the deficiency or about what regulation the nursing home had breached. IMPLICATIONS: The survey process is reliable when assessing aggregate results, but it is only moderately reliable when examining individual citations. Stakeholders (i.e., consumers, policy makers, nursing home administrators) should be aware of the limitations of the survey process. It needs to be modified to reduce variability.
PURPOSE: We designed this study to examine the reliability of the nursing home survey process in the state of Kansas using regular and simultaneous survey teams. In particular, the study examined how two survey teams exposed to the same information at the same time differed in their interpretations. DESIGN AND METHODS: The protocol for simultaneous surveys consists of having one in-region and one out-of-region team survey a facility together. RESULTS: The regular and simultaneous survey teams generally agreed about the number of deficiencies. The intraclass correlation coefficient was 0.87 for total deficiencies and 0.76 for deficiencies with scores of G or higher. But in a substantial number of instances the teams did not agree about the scope and severity of the deficiency or about what regulation the nursing home had breached. IMPLICATIONS: The survey process is reliable when assessing aggregate results, but it is only moderately reliable when examining individual citations. Stakeholders (i.e., consumers, policy makers, nursing home administrators) should be aware of the limitations of the survey process. It needs to be modified to reduce variability.