Experience with a Clinical Audit Requirement for Interns in the Emergency Department.

BACKGROUND: The clinical audit is an important evaluation tool to ensure quality assurance. A clinical audit requirement for interns during their emergency department (ED) term may be a valuable educational activity.
METHODS: The Emergency Audit Initiative (EAI) Program was initiated at Redcliffe Hospital in January 2018. Interns, who were paired with a staff specialist audit mentor, chose a topic of interest and carried out a clinical audit during their 10-week ED term. At the end of term, interns formally presented audit findings in a grand round setting. Interns and staff specialists were surveyed at the end of the intern year regarding aspects of the program. Surveys aimed to assess: (1) value of the program as an educational activity, (2) availability of time and resources to conduct the audit, and (3) perceived impact on practice.
RESULTS: During the first year of the program, 27 clinical audits were carried out. 16 interns (59%) and 8 staff specialists (57%) responded to the surveys. Interns and staff specialists reported that the audit was a valuable educational experience (88% and 100%). Interns also reported that they had adequate time (94%) and resources (81%) to conduct the audit. Interns and staff specialists however reported only a modest impact on clinical practice because of the audit program.
CONCLUSIONS: Our experience with the EAI program suggests that incorporating a clinical audit requirement into the ED term is possible. Interns and staff specialists reported it to be a beneficial educational and professional development activity.

Introduction

Appropriate care, as defined by care in line with evidence based or consensus-based guidelines, is essential for good medical practice. The clinical audit is a review of clinical performance against evidence-based standards which attempts to identify discrepancies between actual practice and standard in order to identify the changes needed to improve the quality of care. This is a vital process to ensure quality in healthcare by improving patient care, increasing efficiency, and promoting awareness of clinical standards.[3-5]

A growing body of literature suggests that conducting a clinical audit is a valuable educational and professional development experience for doctors in training.[1-7] In Australia, the Australian Curriculum Framework for Junior Doctors (ACFJD) makes quality improvement and clinical auditing a core element of all prevocational training programs. Professional bodies and specialist colleges have also recognised the value of the clinical audit, now requiring them as part of their registration and ongoing professional development programs.[2,9] The clinical audit process is often not taught in medical schools, and it is vital that doctors in training learn the skills to conduct quality improvement activities early on in their training in order to develop knowledge of standards of clinical care and to improve their own practice.[9-12]

In order to address this deficit in junior doctors’ medical education, the Emergency Audit Initiative (EAI) program was introduced at Redcliffe Hospital ED in Queensland, Australia in January 2018. The program is a local clinical audit requirement for interns rotating in the ED that came about in response to a perceived gap in interns’ knowledge of quality assurance processes as observed by supervising ED staff specialists. The goals for the program were that it would enhance interns’ understanding of the clinical audit process, serve as a professional development activity, and help to promote a culture of quality improvement in the ED. From a time and resource standpoint, we also wanted to see if it would be possible to incorporate a clinical audit requirement into interns’ busy clinical schedules.

This article describes our experience with the first year of the ED clinical audit requirement.

Methods

Program description and participants

This case report describes our experience with the first year of the Emergency Audit Initiative (EAI) program at the Redcliffe Hospital ED. The program is a mandatory requirement for interns to design, carry out, and formally present a standards-based clinical audit on an emergency medicine topic of their choosing during their ten-week ED term. Interns measure local ED performance and benchmark it against established clinical guidelines and national standards.

Redcliffe Hospital is an urban district hospital located north of Brisbane, Australia, with a mixed ED seeing approximately 65 000 presentations a year. The hospital is a public teaching hospital where all interns do at least one 10-week emergency medicine term during their intern year. The ED is also accredited by the Australasian College for Emergency Medicine (ACEM) for emergency medicine training.

Participants were a prospectively recruited, voluntary, convenience sample of interns and emergency staff specialists working in the Redcliffe ED between 17 January 2018 and 10 January 2019. Interns were all locally trained medical graduates from Queensland medical schools. Participants were asked at the end of the study period to complete a voluntary online program experience survey.

The audit requirement

During the 10-week term, each intern performed a standards-based audit where current practice in the ED was compared against hospital-based guidelines or national standards of care. During Week 1 of the term, interns were oriented to the EAI program and provided with a clinical audit packet of information that included a guide to topic selection, a recommended timeline, authorization forms, and medical record request forms. As training in the clinical audit process and research ethics are not part of the curriculum for Queensland medical schools, in addition to the clinical audit orientation and information packet, each intern was paired with an ED staff specialist who served as an audit mentor during the term. Interns also had access to the EAI program coordinator and the emergency department research manager during the term for assistance with their audit. To ensure clinical governance, all intern audit proposals were reviewed and approved by the ED’s director before the audit commenced. Intern audits needed to be clinically focused, relevant, and have the capacity to change practice and patient outcomes. They also had to be both objective and measurable against a current clinical standard to measure the department’s clinical performance.

The program was designed with reasonable and achievable expectations within the 10-week time frame for completing the audit (Figure 1). In order to ensure interns had both adequate time and resources, the number of cases for each intern to audit was limited to 10 to 15 charts. Upon submission and approval of the audit proposals, ED medical administration officers assisted with collecting patient details and medical records for the audits. Interns were also given 4 hours of paid protected off-the-floor time during their ED term to review the medical records for their audit within the department.

Figure 1.

Flowchart detailing stages of the EAI Program.

During the final week of each term, interns formally presented their audit findings at an open-invitation hospital grand round forum. Audit presentations were assessed at the forum by a panel of ED staff specialists and feedback was provided to each intern at that time. To raise awareness of the EAI program amongst the ED staff, interns made posters of their audit findings which were displayed in the ED at the end of each term. For any clinical audits where ED performance was found to fall short of clinical standards, these findings were noted by the department director at the time of the intern’s formal presentation and educational initiatives and staff awareness campaigns were undertaken by the department’s medical and nursing staff to improve compliance with standards.

Program experience survey

At the end of the intern year, all interns and ED staff specialists were sent via email an experience survey of the EAI program to complete voluntarily. These surveys were developed using a previously published local clinical audit satisfaction survey and the findings of an intradepartmental survey that identified perceived barriers to carrying out clinical audits within the Redcliffe ED. Surveys addressed three key areas: (1) value of the program as an educational activity, (2) availability of time and resources to conduct the audit, and (3) perceived impact on practice.

The intern experience survey included 19 items assessing intern satisfaction with the program in improving understanding of the clinical audit process, value as an educational and professional development tool, and whether they felt their audit impacted practice and quality assurance in the department. Time and resource limitations can often make conducting a clinical audit challenging. A previous intradepartmental survey had identified perceived barriers to carrying out clinical audits within the Redcliffe ED as time, resource, and departmental support. The intern experience survey also assessed these perceived barriers by surveying interns regarding time available to perform their audits, resource availability, and departmental support for their audits.

The ED staff specialist program experience survey included 11 items assessing staff specialists’ satisfaction with the educational value of the EAI program for interns along with the overall impact the program had on quality assurance in the ED.

All survey items were evaluated on a five-point Likert scale (1 Strongly Disagree, 3 Agree, and 5 Strongly Disagree) and examined using descriptive statistics. Scores of 3 or more were used to calculate % agreement. Mean and median scores for each survey question were also calculated. Surveys were administered using a commercially available web-based survey program (www.surveymonkey.com) that was emailed to all interns and ED staff specialists. Completion of the program experience surveys was voluntary and anonymous.

Ethics

This study was submitted to the Prince Charles Hospital Human Research Ethics Committee for review and granted an exemption from full ethical review on the basis that it was an audit / quality assurance project.

Results

Between January 17, 2018 and January 10, 2019, a period which corresponded to 5 intern terms, a total of 27 interns completed their emergency medicine term and clinical audit at the study-site hospital. Five grand round forums were also held throughout that period for the interns to present their audit results. The forums were attended by ED staff specialists, registrars, nurses, health and safety officers, medical/surgical department heads, and hospital executives. Of the 27 interns who were sent the end of year program experience survey, 16 responded (response rate 59%) (Table 1).

Table 1.

EAI intern feedback survey frequency of responses on a scale of 1 to 5, where 1 = strongly disagree and 5 = strongly agree.

Educational value	% Agreement	Mean	Median
1. Have you completed a clinical audit before?	38
2. The clinical audit was a worthwhile teaching tool	88	4.3	4
3. I was able to carry out a clinical audit in an area of interest	94	4.5	5
4. The audit project was a waste of time and effort	6	1.6	1
5. The clinical audit should continue to be a requirement for ED interns	75	4.0	4
6. I understand the clinical audit process better after completing the EAI program	88	4.1	4
7. Formally presenting my audit findings helped me to develop communication skills	88	4.1	4
8. The EAI program gave me the skills and knowledge to conduct an audit in the future	88	4.4	5
Resource availability
9. I had adequate time to complete the clinical audit	94	4.4	5
10. The departmental resources and support for the audit were adequate	81	4.4	5
11. The IT system and medical records were adequate to support my audit	81	4.1	4
Impact on care
12. I believe I contributed to health care delivery because of my audit	50	3.8	4
13. I feel the clinical audit is relevant and improves quality of care	75	4.1	4
14. I have a better understanding of clinical standards and performance indicators	81	4.2	4
15. The audit made me aware that clinical standards are frequently not met and impact patient care	63	3.9	4
16. The audit influenced my practice in ensuring I am complying with best practice standards	69	4.1	4
17. By conducting an audit, I gained a better idea how system issues impact health care delivery	75	4.1	4
18. The audit made me aware of the limitations of coding and importance of documentation which has positively impacted my own coding and documentation practices	75	4.0	4
19. Because of the EAI program, I am more interested in research and want to perform a more comprehensive audit to answer a clinical question	50	3.5	4

Of the 14 staff specialists who staff the hospital’s emergency department, 8 responded to the program experience survey (response rate 57%) (Table 2). Of the 8 respondents, 7 had served as supervisors for the interns’ audits.

Table 2.

EAI staff specialist feedback survey frequency of responses on a scale of 1 to 5, where 1 = strongly disagree and 5 = strongly agree.

Educational value	% Agreement	Mean	Median
1. Did you serve as an audit mentor for an intern?	88
2. As a result of the audit program, interns are better able to conduct a clinical audit on their own practice in the future	75	4.1	4
3. The clinical audit requirement should be continued for interns rotating through the ED	88	4.3	4
4. The clinical audit is a valuable educational activity and teaching tool for interns	100	4.1	4
5. Interns became more aware of quality issues in general because of the audit program	100	4.3	4
6. Interns conducting a clinical audit and presenting their findings is important for their professional development	88	4.3	4
Impact on care
7. I believe the clinical audit program is a valuable addition to the department in improving patient care and safety	75	3.9	4
8. Changes have been made to departmental practice because of findings of the intern audits	38	3.3	3
9. The EAI program made interns more work ready through a better understanding of the medical workplace and barriers effecting change	50	3.5	4
10. Completion of the audit project has influenced interns ongoing level of participation in quality improvement activities as junior doctors	63	3.9	4
11. Results of the interns’ audits has raised awareness of the unit’s performance and guidelines of best practice	75	3.9	4
12. The requirement for interns to formally present their audit findings to senior medical staff/administrators has encouraged best practice	38	3.5	3

Educational value

The majority of interns (62%) reported that this was their first exposure to the clinical audit process. Most agreed or strongly agreed that the clinical audit requirement was a worthwhile teaching tool (88%) and should continue to be an ongoing requirement for emergency medicine interns (75%). There was also strong agreement in the professional development value of the EAI program in terms of the communication skills developed by doing the formal presentation (88%). Most interns reported that because of the program they had the skills and knowledge to be able to conduct an audit in the future (88%).

Similarly, staff specialists agreed or strongly agreed that the EAI program was a valuable educational and professional development activity for interns (100%) and that the EAI program should be continued in the future (88%). They reported interns were more aware of quality issues in health care (100%) and were better able to conduct a clinical audit in the future because of the program (75%).

Availability of time and resources

Interns agreed or strongly agreed that they had adequate time to conduct the audit (94%). They also felt that they were supported throughout the activity by the department (81%) and had adequate resources to complete their audit (81%).

Impact on clinical practice

Interns agreed or strongly agreed that the audit led to a better understanding of clinical standards and quality assurance (81%), system issues that impact health care delivery (75%), and raised awareness of the limits of coding and documentation (75%). While interns felt the clinical audit was relevant and improved quality of care (75%), they however did not report that health care delivery in the department was impacted because of their audits (50%). They also reported only modest agreement that the audit influenced their practice by ensuring that they were complying with best practice standards (69%). Interns reported only modest interest in performing an audit or research project in the future because of their experience with the EAI program (50%).

Most staff specialists agreed or strongly agreed that the EAI program was a valuable addition to the emergency department in terms of raising awareness of the ED’s performance and guidelines of best practice (75%). There was however, less agreement amongst staff specialists regarding the impact of the interns’ audits on clinical practice with only 38% agreeing or strongly agreeing that changes had been made to departmental practice because of the audits. There was also modest agreement that participation in the EAI program would impact interns’ ongoing participation in quality improvement activities in the future (63%).

Discussion

Continuous improvement based on ongoing monitoring of clinical practice should be an integral part of every healthcare system. It is important that all health care workers develop an understanding of this early on in their careers. Due to time constraints, the clinical audit process is often not taught in medical school curriculums.[9-12] The intern year is often the first time junior doctors are directly involved in patient care and can begin to see the importance of quality assurance. Interns’ emergency medicine term is an ideal rotation to introduce the clinical audit process because there are well established clinical guidelines and national standards upon which to bench mark practice.

The EAI program appears to have a number of benefits for interns. The program experience surveys’ results suggest that there were both educational and professional development benefits to conducting a clinical audit as well as improvements in interns’ understanding of standards of care. This study, along with others, has shown that both participant and supervisor satisfaction with a clinical audit program as a medical educational activity is high.[7,11,12]

Despite interns having significant clinical responsibilities and time constraints, especially during their ED term, our experience with the EAI program suggests that integrating a clinical audit program into the ED term is feasible with adequate planning, dedicated departmental resources, and protected non-clinical time. Staff support is also critical for a successful audit program. A local champion who is a driver and promotor of the audit program, hospital managers who are supportive of resources (i.e. paid off-the floor time for both interns and staff specialists) being allocated to conducting clinical audits, and enthusiastic staff specialists with interest in quality assurance and teaching are all critical to a sustainable program. From our experience, interns reported that they felt well supported with their audits throughout their term and had enough time to complete their audits in addition to their ongoing clinical responsibilities. Our experience with the EAI program also suggests that incorporating a clinical audit program into an existing education curriculum may be readily transferable to other medical and nursing education curricula.

The perceived clinical impact of the EAI program was however less clear. While both interns and staff specialists agreed on the relevance of the clinical audit, there was only modest agreement that the results of the clinical audits had made an impact on the delivery of care in the department. This is despite the fact that staff educational initiatives such as the grand rounds forums, audit posters, and post-audit staff awareness initiatives occurred because of the audits. There was also only modest agreement amongst both interns and staff specialists that participation in the EAI program would result in ongoing involvement of interns in quality improvement activities in the future.

Limitations

Due to time constraints, interns were unable to implement changes and re-audit their clinical questions within the 10-week term which is part of the standard audit cycle. As a result of this, interns may not have gained a full appreciation for the role and impact of the clinical audit in terms of quality assurance in health care. Another consequence of the time constraints was that interns were only able to conduct small clinical audits which limited the generalizability of their findings.

The program experience surveys used to assess intern and staff specialist attitudes towards the EAI program also had limitations. As the EAI program was limited to a single site and involved a relatively small number of participants, conclusions drawn from surveys regarding the educational value of a clinical audit program are limited. While the response rate of the interns and staff specialists on the satisfaction surveys (59% and 57%) was reasonable, there may be a degree of response bias that could paint an incomplete picture of the overall satisfaction with the EAI program and its impact. Specifically, the majority of staff specialists who responded to the survey had served as audit mentors and may have a more positive view of the EAI program than staff specialists who were not as involved. Interns may have felt compelled to respond favourably when surveyed about their audit experience. We attempted to control for this by making completion of the surveys anonymous and voluntary. There may also be a degree of recall bias with our survey results as the experience survey was administered only once at the end of the intern year, rather than at the end of each term, which may have influenced the responses of interns who participated in the program earlier on in the year.

Carrying out a clinical audit program such as this, even on a relatively small scale, requires considerable resources to ensure that it is successful which may not be available in all clinical settings. Significant IT and administrative support are required for data archiving and collection of medical records. Protected non-clinical time is also required to review medical records and to support junior doctors with their audits.

Conclusions

The EAI program demonstrates that incorporating a clinical audit requirement into interns’ emergency medicine term is possible provided there are adequate resources and staff commitment. Our experience suggests that there is a high degree of satisfaction with a clinical audit program as an educational and professional development activity amongst both interns and ED staff specialists.