What drives the 'August effect'? A observational study of the effect of junior doctor changeover on out of hours work.

OBJECTIVE: To investigate whether measurements of junior doctor on-call workload and performance can clarify the mechanisms underlying the increase in morbidity and mortality seen after junior doctor changeover: the 'August effect'.
DESIGN: Quantitative retrospective observational study of routinely collected data on junior doctor workload.
SETTING: Two large teaching hospitals in England. PARTICIPANTS: Task level data from a wireless out of hours system (n = 29,885 requests) used by medical staff, nurses, and allied health professionals. MAIN OUTCOME MEASURES: Number and type of tasks requested by nurses, time to completion of tasks by junior doctors.
RESULTS: There was no overall change in the number of tasks requested by nurses out of hours around the August changeover (median requests per hour 15 before and 14 after, p = 0.46). However, the number of tasks classified as urgent was greater (p = 0.016) equating to five more urgent tasks per day. After changeover, doctors took less time to complete tasks overall due to a reduction in time taken for routine tasks (median 74 vs. 66 min; p = 3.9 × 10(-9)).
CONCLUSION: This study suggests that the 'August effect' is not due to new junior doctors completing tasks more slowly or having a greater workload. Further studies are required to investigate the causes of the increased number of urgent tasks seen, but likely factors are errors, omissions, and poor prioritization. Thus, improved training and quality control has the potential to address this increased duration of unresolved patient risk. The study also highlights the potential of newer technologies to facilitate quantitative study of clinical activity.

Introduction

In the United Kingdom, final year medical students become doctors and junior doctors become a grade more senior on the same day in early August. There is therefore an abrupt change to the provision of care by junior doctors who are inexperienced, or who are less experienced for their level of seniority. Consequently, there is clear potential for a reduction in the quality and safety of patient care: the ‘August effect’. This situation causes unease amongst the medical community,[1] especially in light of reports suggesting medical students are poorly prepared for their first post.[2,3]

A number of studies have investigated the effect of changeover on broad clinical endpoints with varying results. A recent systematic review of such studies suggested that mortality rates for patients admitted just after changeover were higher than for the remainder of the year, though most of the papers in this review were reported to be of low quality.[4] The experience in our local Trust is consistent with this: data from the independent body Dr Foster suggest that there are around six more in-hospital deaths in Nottingham in August than would be expected for the volume and case-mix of admissions. In contrast, there are around 14 fewer deaths than expected in the rest of the year.[5]

Few studies have attempted to provide insight into the mechanisms by which any change in mortality or length of stay might arise. Investigations of the changeover period involving patients with more complex medical and surgical problems tend to show greater effect size, suggesting that these individuals may be more susceptible to any adverse effects around the start of the academic year.[4,6,7]

The risk of adverse outcomes associated with the August changeover would be expected to be most pronounced when considering work undertaken outside normal working hours; during this time, juniors have less supervision,[3] may be exposed to greater stress,[8] and may perform sub-optimally because of greater shift length.[9,10] With the introduction of the Hospital at Night[11] system in response to the European Working Time Directive,[12] they may also work in unfamiliar environments and may receive incomplete information from colleagues when shifts change.[13] In keeping with these assertions, reports suggest that mortality is higher than should be expected at the weekend across secondary care in the United Kingdom.[14]

In Nottingham, we have introduced a wireless solution for the management of out of hours (OOH) workflow for over 1000 inpatients at two large teaching hospitals. We have shown this system to be easy and rapid to use, and associated with high user satisfaction scores and liberation of nursing time.[15] As the system logs task level information, we have a record of the type and number of nursing requests, and the time to task completion by junior doctors.

In this study, we used the wireless out of hours system to objectively investigate changes in out of hours work around the time of the August junior doctor changeover. We posed three questions that aim to highlight potential mechanisms for the ‘August effect’:

Is more Out of Hours work required after changeover? This would suggest inefficiency and omissions during daytime hours.

Is the Out of Hours work required more urgent? This would suggest failure to identify deteriorating patients or major omissions in-hours.

Do junior doctors take longer to complete Out of Hours tasks after changeover? This would suggest inefficiency and delays driving the effect.

Methods

Data capture

The settings for the study were the City Hospital and the Queen’s Medical Centre, both part of Nottingham University NHS Hospitals Trust. These sites are large university teaching hospitals with around 1700 beds in 87 wards spanning a wide variety of specialties and handling around 190,000 acute attendances per annum. Both sites are equipped with a medical grade wireless network (Cisco Systems Inc, USA) and out of hours care is provided using Hospital at Night teams (composition of teams in Supplementary Materials). In common with other teams born of this national initiative, the Hospital at Night team is in place during the out of hours period (5 p.m.–9 a.m., weekdays and through the weekend). As with most teams, those in Nottingham consist of senior nurse coordinator who receives requests for clinical review or intervention from wards and triages them to one of a small number of on-call junior doctors or a clinical support worker who undertakes a limited range of tasks (e.g. cannulation). However, rather than employ the usual standard of a pager and landline based system, all tasks for medical and surgical patients required outside the dedicated acute admission wards are requested electronically in a standardized SBAR (Situation, Background, Assessment, Recommendation) format.[16] These tasks are automatically triaged according to preset rules. A nurse coordinator carrying a tablet PC sends the task to the mobile phone of the most appropriate team member using Nervecentre software (Nervecentre Software, Wokingham, UK). At the Nottingham City site, additional data were available as the individual doctor accepts the request onto the task list on their phone and once the task has been undertaken, they indicate it to be complete: both of these operations require a single button press and are recorded centrally. We captured task request data at both sites and in the two months before the 2011 changeover (on 3 August) and the month thereafter.

Task urgency

Tasks are assigned to one of three categories by predefined criteria related to the urgency of the response required (see Supplementary Item One). In brief, red calls require urgent medical attention to prevent serious harm (e.g. patient fall with > 2 point reduction in Glasgow Coma Scale) and amber tasks require a rapid response (e.g. wound dehiscence). Green tasks are sufficiently urgent that they should be completed in the out of hours period but are of lower priority than red and amber requests (e.g. prescription of further intravenous fluids for someone nil by mouth but otherwise stable).

Task complexity

We specified calls to attend patients with an early warning score (EWS) of more than 4 (see Supplementary Item Two) who did not fall into another category (such as neutropenic sepsis or sudden onset of breathlessness) as a marker of tasks that would challenge the knowledge and decision making ability of junior doctors. We specified calls for chest pain as an example of an urgent task that would usually require the reproduction of a well-defined pathway of assessment and therapy for which the junior doctor would have been trained, and for which local guidelines exist.

Analysis

A combined analysis for task requested using data from both sites is presented, as handling the data for the Queen’s Medical Centre (large single building with an unselected medical take and emergency department) and the City Hospital (large widespread campus with specialty units) separately did not affect the direction of effect in the results. Data from the City site are used for time to complete analyses because (as noted above) the duration of task was not available for the QMC site during the study period. We compared the following before and after the August changeover: number of tasks requested per hour of shift, number of urgent tasks (‘red’ or ‘amber’) per hour of shift, the average time from acceptance to completion of routine and urgent tasks, and the average time from acceptance to completion of the specific tasks ‘EWS>4’, and ‘chest pain’. As the data were not normally distributed, Mann–Whitney tests were used and results are given as medians with interquartile ranges (IQR). P values are given to two significant figures. Analyses were undertaken using SPSS 19 (IBM, New York, USA).

Results

We included 29,885 tasks which were completed between 1 June and 31 August inclusive in our analyses. The volume of work requested of junior doctors was similar before (median 15 tasks requested per hour, IQR 13) and after (median 14, IQR 14) changeover (p = 0.46). The number of amber or red tasks requested each hour increased significantly (p = 0.016) after changeover. Although median number of urgent tasks requested per hour was 5, both pre-and post-changeover, the distribution of the volume of tasks requested differed with a greater number of periods of high request volume after changeover. This difference was more apparent when longer timeframes were considered: it equated to a median of 5 more urgent tasks requested each weekday and 13 each weekend day.

The 17,986 tasks requested at the City site were used for duration analyses. The time taken by the junior doctors to complete non-urgent (‘green’) tasks was significantly shorter after changeover (median 74 vs. 66 min; p = 3.9 × 10−9). The time taken to complete tasks categorized as red or amber was not different (median 75 vs. 76 min; p = 0.453). In terms of task complexity, there was no difference in the duration of our chosen urgent task with clear pathways of assessment and treatment (p = 0.277 for chest pain). The time taken to complete a call to a patient with EWS>4 increased from 64 to 73 min, but this was not statistically significant (p = 0.077). These findings and the incidence of the events are summarized in Table 1.

Table 1.

Time taken between task acceptance and completion before and after junior doctor changeover in August 2011 at Nottingham City Hospital.

Median (IQR) duration (min)
Task type	Number of occurrences	Pre- changeover	Post- changeover	P (Mann–Whitney test to 2 significant figures)
All green	11,479	74 (158)	66 (113)	3.9 × 10−9
All red and amber	5945	75 (109)	76 (116)	0.45
Chest pain	101	69 (121)	53 (50)	0.28
EWS > 4	571	64 (73)	73 (89)	0.077

IQR: interquartile range.

Discussion

The changeover of junior doctors on a single day has generated concerns over patient safety in secondary care. An increase in morbidity and mortality is described (the ‘August effect’), but it is unclear how any change in adverse outcomes arises. Potential explanations that could increase patient risk include junior doctors working more slowly, having a generally greater amount of work to do, or factors such as omissions and errors that create more urgent work. By taking advantage of a wireless system for processing out of hours workflow, we studied the actual work undertaken by junior doctors before and after the August changeover 2011. In this study, we found that the total amount of work requested by nurses for the Hospital at Night teams did not differ overall before and after the changeover, there was a significant increase in the proportion of tasks categorized as urgent by pre-specified criteria, and junior doctors took less time to complete non-urgent tasks.

Volume of tasks requested

There was no change in the overall volume of work requested of junior doctors during the study period. This might be expected as there were no clear influences on admission volume,[17] case-mix (e.g. influenza outbreak), facilities (e.g. clinical department relocation), or staffing during the study period. These data suggest that the ‘August effect’ is not driven by an increase in the volume of work, or because a greater number of routine in-hours tasks have been left incomplete by those working in the day.

We found a statistically significant increase in the volume of requests for more urgent problems equivalent to approximately five more red or amber task requests per weekday. As each task took a median of over 1 h to be completed, this amounts to a considerable cumulative duration of unresolved patient risk per month. This duration of increased risk may be sufficient to drive a proportion of the excess mortality and morbidity reported soon after changeover in several studies.[6,7,18,19] This increase in urgent tasks may arise from inexperienced doctors being less likely to recognize deterioration in their patients at an early stage,[20,21] or inadvertently increasing the probability of an emergency by day time errors (e.g. through their inexperience with medication prescription[22]). If these findings are confirmed and advanced by future studies, there would appear to be scope to augment the quality of in-hours care to reduce the number of urgent tasks and the associated patient risk. Certainly, the current approaches to readying students for their first day on the wards have been heavily criticized despite the increased use of shadowing periods.[23] Any errors or oversights of junior doctors have traditionally been ameliorated by senior clinicians,[24] but this effect is unpredictable as some choose to undertake more frequent or detailed rounds around changeover but others are on holiday. These initial data lend some support to adopting a more uniform and robust approach to ensure high levels of patient care immediately after changeover.

Time to complete tasks

We found that junior doctors completed routine tasks faster after changeover than before. This was a relatively small (though statistically robust) difference, but multiplied by the number of events, this constitutes hundreds of hours of junior doctor time. Previous reports have suggested that newly qualified doctors have a lower ability to prioritize their workload, and that this may be amenable to improvement using simulation studies.[25,26] New junior doctors may also be undertaking larger batches of non-urgent tasks in one location to reduce complex route planning around an unfamiliar hospital.[27] In our experience, as well as a broader understanding of the urgency with which tasks need to be completed, comes a degree of job fatigue in doctors coming to the end of placements. Such findings may also relate to the aforementioned suggestion that inexperienced doctors fail to perceive the seriousness or complexity of situations: they may label a task as ‘complete’ when more experienced hands would have undertaken more investigations or interventions.

It may be expected that junior doctors would take relatively longer to deal with more complex tasks after changeover, such situations not only test experience and knowledge, but they are also influenced by familiarity with hospital systems and protocols that may be required when faced with an unstable patient, such as requesting urgent imaging.[28] We found the median duration of such tasks to be greater post changeover, but this was not statistically significant given the broad variation in duration seen.

Strengths and limitations of the study

This study is the first to our knowledge to use objective measures of junior doctor activity to gain some insight into the mechanisms of the ‘August effect’. We used data on tens of thousands of tasks undertaken by a large number of junior doctors in one of the UK’s largest Trusts. Tasks were requested across all medical specialities. Data collection arises through a wireless working system as part of usual clinical work and as such is detailed and repeatable.

Although our results may have been influenced by the familiarity of doctors with the wireless system, we have previously found new users become proficient rapidly,[15] and the new intake of junior doctors had all shadowed their predecessors for at least one on-call shift prior to their commencement. Junior doctors also tend to be very familiar with the use of mobile technology such as the Blackberry phones (Research in Motion, Ontario, Canada) used for this application. Certainly, the wireless system allowed no clear scope for the loss or adulteration of information that is likely to have occurred with paper records.

Although it is possible that nursing staff have less confidence in new doctors and have been deliberately misreporting tasks to increase their apparent urgency, we feel this is unlikely given the standard requesting interface; moreover, categorization of urgency is automatically determined within the software.

It may also be that more senior staff took leave during August when compared to June or July. In the large teaching hospitals in Nottingham, however, consultants usually have periods on and off the wards so there would not have been a disruption in senior ward cover. We accept the limitation of the data from a single Trust over one changeover period, though the volume of tasks requested renders our findings sufficiently robust to inform further studies.

Potential

Potential future studies could take advantage of newer technologies to more fully record the actions and movements of junior doctors including route planning and the use of rest breaks. There is also clear potential to relate such findings to outcome data.

High reliability organizations have a sound understanding of the fine detail of what occurs on a daily basis and how variable this is, and do not simply investigate major adverse incidents.[29] The use of mobile technology in secondary care can provide information on the denominators for incidents and derive a benchmark of average activity, acceptable variation and usual outcome for particular tasks or situations. These data will be central to understanding the issues arising around traditional sources of concern such as junior doctor changeover and weekend working.[30]

Conclusion

In summary, we report that although total workload does not change greatly around junior changeover, the proportion of urgent tasks increases significantly. Time to complete non-urgent tasks fell after the changeover. These results suggest that the ‘August effect’ of increased morbidity and mortality is not driven by new junior doctors working more slowly or by an increased of overall workload. Although further research in this important area is certainly required, this pattern could have arisen through omissions, errors, failure to recognize deterioration, and poor task prioritization skills. These factors are amenable to improved training, supervision, and quality control.

This study lends empirical evidence to qualitative research investigating whether junior doctors are prepared for the practicalities and complexities of their first posting and how the ‘August effect’ may arise. It also highlights the potential of newer technologies to study in detail the actual work undertaken in hospitals.