H Stephen Injeyan1, Sheilah Hogg-Johnson2, Sean Abdulla3, Ngai Chow3, Jocelyn Cox3, Anthony Ridding3, Craig Jacobs3. 1. Research and Clinical Education, Canadian Memorial Chiropractic College, 6100 Leslie Street, Toronto, Ontario, M2H 3J1, Canada. sinjeyan@cmcc.ca. 2. Graduate Education and Research Programs, Canadian Memorial Chiropractic College, Toronto, Ontario, Canada. 3. Division of Clinical Education, Canadian Memorial Chiropractic College, Toronto, Ontario, Canada.
Abstract
BACKGROUND: There is a dearth of information about health education clinical file audits in the context of completeness of records and demonstrating program-wide competency achievement. We report on the reliability of an audit instrument used for electronic health record (EHR) audits in the clinics of a chiropractic college in Canada. METHODS: The instrument is a checklist built within an electronic software application designed to pull data automatically from the EHR. It consists of a combination of 61 objective (n = 20) and subjective (n = 41) elements, representing domains of standards of practice, accreditation and in-house educational standards. Trained auditors provide responses to the elements and the software yields scores indicating the quality of clinical record per file. A convenience sample of 24 files, drawn randomly from the roster of 22 clinicians, were divided into three groups of eight to be completed by one of three auditors in the span of 1 week, at the end of which they were transferred to another auditor. There were four audit cycles; audits from cycles 1 and 4 were used to assess intra-rater (test-retest) reliability and audits from cycles 1, 2 and 3 were used to assess inter-rater reliability. Percent agreement (PA) and Kappa statistics (K) were used as outcomes. Scatter plots and intraclass correlation (ICC) coefficients were used to assess standards of practice, accreditation, and overall audit scores. RESULTS: Across all 3 auditors test-retest reliability for objective items was PA 89% and K 0.75, and for subjective items PA 82% and K 0.63. In contrast, inter-rater reliability was moderate at PA 82% and K 0.59, and PA 70% and K 0.44 for objective and subjective items, respectively. Element analysis indicated a wide range of PA and K values inter-rater reliability of many elements being rated as poor. ICC coefficient calculations indicated moderate reliability for the domains of standards of practice, accreditation, and overall file scores. CONCLUSION: The file audit process has substantial test-retest reliability and moderate inter-rater reliability. Recommendations are made to improve reliability outcomes. These include modifying the audit checklist with a view of improving clarity of elements, and enhancing uniformity of auditor responses by increased training aided by preparation of an audit guidebook.
BACKGROUND: There is a dearth of information about health education clinical file audits in the context of completeness of records and demonstrating program-wide competency achievement. We report on the reliability of an audit instrument used for electronic health record (EHR) audits in the clinics of a chiropractic college in Canada. METHODS: The instrument is a checklist built within an electronic software application designed to pull data automatically from the EHR. It consists of a combination of 61 objective (n = 20) and subjective (n = 41) elements, representing domains of standards of practice, accreditation and in-house educational standards. Trained auditors provide responses to the elements and the software yields scores indicating the quality of clinical record per file. A convenience sample of 24 files, drawn randomly from the roster of 22 clinicians, were divided into three groups of eight to be completed by one of three auditors in the span of 1 week, at the end of which they were transferred to another auditor. There were four audit cycles; audits from cycles 1 and 4 were used to assess intra-rater (test-retest) reliability and audits from cycles 1, 2 and 3 were used to assess inter-rater reliability. Percent agreement (PA) and Kappa statistics (K) were used as outcomes. Scatter plots and intraclass correlation (ICC) coefficients were used to assess standards of practice, accreditation, and overall audit scores. RESULTS: Across all 3 auditors test-retest reliability for objective items was PA 89% and K 0.75, and for subjective items PA 82% and K 0.63. In contrast, inter-rater reliability was moderate at PA 82% and K 0.59, and PA 70% and K 0.44 for objective and subjective items, respectively. Element analysis indicated a wide range of PA and K values inter-rater reliability of many elements being rated as poor. ICC coefficient calculations indicated moderate reliability for the domains of standards of practice, accreditation, and overall file scores. CONCLUSION: The file audit process has substantial test-retest reliability and moderate inter-rater reliability. Recommendations are made to improve reliability outcomes. These include modifying the audit checklist with a view of improving clarity of elements, and enhancing uniformity of auditor responses by increased training aided by preparation of an audit guidebook.
Authors: Richard Lilford; Alex Edwards; Alan Girling; Timothy Hofer; Gian Luca Di Tanna; Jane Petty; Jon Nicholl Journal: J Health Serv Res Policy Date: 2007-07
Authors: Jarone Lee; J Matthew Austin; Jungyeon Kim; Paola D Miralles; Haytham M A Kaafarani; Peter J Pronovost; Vipra Ghimire; Sean M Berenholtz; Karen Donelan; Elizabeth Martinez Journal: Am J Med Qual Date: 2018-09-28 Impact factor: 1.852
Authors: Jennifer A Bierman; Kathryn Kinner Hufmeyer; David T Liss; A Charlotta Weaver; Heather L Heiman Journal: Teach Learn Med Date: 2017-05-12 Impact factor: 2.414