Shaun O'Leary1, Marte Lund2, Tore Johan Ytre-Hauge3, Sigrid Reiersen Holm4, Kaja Naess5, Lars Nagelstad Dalland6, Steven M McPhail7. 1. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Physiotherapy Department, Royal Brisbane and Women's Hospital, Queensland Health, Herston, Brisbane, QLD 4029, Australia. Electronic address: shaun_oleary@health.qld.gov.au. 2. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Norwegian Sports Medicine Clinic (NIMI), Oslo, Norway. Electronic address: marte.lund@nimi.no. 3. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Medi 3 Clinic, Aalesund, Norway. Electronic address: toreyh@hotmail.com. 4. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; University Hospital of Northern Norway, Tromsø, Norway. Electronic address: sigrid.r.holm@gmail.com. 5. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Hans & Olaf Physiotherapy Clinic, Oslo, Norway. Electronic address: kaja.nass@hof.nhn.no. 6. NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Eggedal Physiotherapy Clinic, Sigdal, Norway. Electronic address: larsdalland@hotmail.com. 7. Centre for Functioning and Health Research, Queensland Health, Cnr of Ipswich Road and Cornwall Street, Brisbane, Australia; School of Public Health and Institute of Health and Biomedical Innovation, Queensland University of Technology, Victoria Park Road, Brisbane, Australia. Electronic address: steven_mcphail@health.qld.gov.au.
Abstract
OBJECTIVE: To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN: Inter-therapist reliability study. SETTING: Research laboratory. PARTICIPANTS: 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES: Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS: The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS: Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.
OBJECTIVE: To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN: Inter-therapist reliability study. SETTING: Research laboratory. PARTICIPANTS: 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES: Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS: The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS: Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.
Authors: Tom J Blodgett; Sue E Gardner; Nicole P Blodgett; Lisa V Peterson; Melissa Pietraszak Journal: Clin Nurs Res Date: 2014-09-22 Impact factor: 2.075
Authors: Patricia C Valery; Christina M Bernardes; Katherine A Stuart; Gunter F Hartel; Steven M McPhail; Richard Skoien; Tony Rahman; Paul J Clark; Leigh U Horsfall; Kelly L Hayward; Rohit Gupta; Elizabeth E Powell Journal: Patient Prefer Adherence Date: 2020-03-18 Impact factor: 2.711