Jack C Collins1, Ming Yeung Chan2, Carl R Schneider3, Lam R Yan2, Rebekah J Moles3. 1. Sydney Pharmacy School, Faculty of Medicine and Health, The University of Sydney, Australia. Electronic address: jcol4724@uni.sydney.edu.au. 2. Sydney Pharmacy School, Faculty of Medicine and Health, The University of Sydney, Australia; School of Pharmacy, University of Nottingham, United Kingdom. 3. Sydney Pharmacy School, Faculty of Medicine and Health, The University of Sydney, Australia.
Abstract
BACKGROUND: The use of simulated patients (SPs) in pharmacy practice research has become an established method to observe practice. The reliability of data reported using this method in comparison to pharmacy staff self-reported behaviour has yet to be ascertained. OBJECTIVE: To compare the inter-rater agreement of pharmacy staff and SP-reported data to researcher-reported data from audio recordings of SP encounters. METHODS: A dataset of 352 audio-recorded SP encounters was generated in March-October 2015 by 61 undergraduate pharmacy students completing SP visits to 36 community pharmacies in Sydney, Australia. Post-visit scores were recorded on data collection forms by SPs. Staff completed self-assessments on identical forms immediately after the encounter. Two-hundred-and-seventy visits were randomly selected as the sample for this study, where the researcher independently scored encounters via audio recordings. Inter-rater agreement was calculated through intra-class correlation (ICC) and weighted kappa analyses. RESULTS: Analysis of staff scores returned ICC values of 0.48 (95% CI:0.38-0.56; p < 0.001) for information gathering and 0.63 (95% CI:0.55-0.70; p < 0.001) for total score. Weighted kappa for information rating was 0.30 (95% CI:0.21-0.38; p < 0.001) and 0.43 (95% CI:0.34-0.51; p < 0.001) for overall outcome. ICC values for SPs were 0.91 (95% CI:0.88-0.93; p < 0.001) and 0.90 (95% CI:0.87-0.92; p < 0.001) for information gathering and total scores respectively. Weighted kappa values were 0.44 (95% CI:0.37-0.52; p < 0.001) for information rating and 0.63 (95% CI:0.55-0.70; p < 0.001) for overall outcome. CONCLUSION: Pharmacy staff self-reported their behaviour with a poor degree of reliability. Conversely, SPs had a high level of agreement with the researcher scoring from audio recordings. Disagreement for both groups of raters was most apparent in rating the information provided and overall appropriateness of outcome. Future research should investigate this discrepancy between staff-reported behaviour and actual behaviour and consider the implications of this discrepancy in the interpretation of self-reported data.
BACKGROUND: The use of simulated patients (SPs) in pharmacy practice research has become an established method to observe practice. The reliability of data reported using this method in comparison to pharmacy staff self-reported behaviour has yet to be ascertained. OBJECTIVE: To compare the inter-rater agreement of pharmacy staff and SP-reported data to researcher-reported data from audio recordings of SP encounters. METHODS: A dataset of 352 audio-recorded SP encounters was generated in March-October 2015 by 61 undergraduate pharmacy students completing SP visits to 36 community pharmacies in Sydney, Australia. Post-visit scores were recorded on data collection forms by SPs. Staff completed self-assessments on identical forms immediately after the encounter. Two-hundred-and-seventy visits were randomly selected as the sample for this study, where the researcher independently scored encounters via audio recordings. Inter-rater agreement was calculated through intra-class correlation (ICC) and weighted kappa analyses. RESULTS: Analysis of staff scores returned ICC values of 0.48 (95% CI:0.38-0.56; p < 0.001) for information gathering and 0.63 (95% CI:0.55-0.70; p < 0.001) for total score. Weighted kappa for information rating was 0.30 (95% CI:0.21-0.38; p < 0.001) and 0.43 (95% CI:0.34-0.51; p < 0.001) for overall outcome. ICC values for SPs were 0.91 (95% CI:0.88-0.93; p < 0.001) and 0.90 (95% CI:0.87-0.92; p < 0.001) for information gathering and total scores respectively. Weighted kappa values were 0.44 (95% CI:0.37-0.52; p < 0.001) for information rating and 0.63 (95% CI:0.55-0.70; p < 0.001) for overall outcome. CONCLUSION: Pharmacy staff self-reported their behaviour with a poor degree of reliability. Conversely, SPs had a high level of agreement with the researcher scoring from audio recordings. Disagreement for both groups of raters was most apparent in rating the information provided and overall appropriateness of outcome. Future research should investigate this discrepancy between staff-reported behaviour and actual behaviour and consider the implications of this discrepancy in the interpretation of self-reported data.