Evaluation of an early career clinical academic training programme using the CIPP model.

OBJECTIVES: This study describes the successful implementation and outputs of a combined clinical academic training programme for doctors in their first postgraduate year in Ireland, the Academic Internship Track (AIT).
DESIGN: The AIT was evaluated using the Context, Input, Process and Product model. Literature reviews, meetings with key stakeholders, reviews of similar established programmes overseas, a survey of undergraduate medical students, exit survey, scientific outputs and career trajectory monitoring were all implemented in the programme evaluation.
SETTING: The AIT represents collaboration amongst all six intern training networks in Ireland.
RESULTS: Key stakeholders indicated support and significant interest in establishing the AIT. The input evaluation informed programme design which incorporates protected time to carry out a research project, a named supervisor, a bursary and access to dedicated study days. Since the programme's launch in 2017, there has been 100% uptake of posts and 0% attrition. Exiting participants indicate high levels of satisfaction with the programme; 92% reported having benefited from participation. Over 90% intend remaining in Ireland in both the immediate and longer terms. Fifty-seven per cent of participants in the first 3 years of the programme had succeeded in publishing a research article or review paper in a peer-reviewed journal.
CONCLUSIONS: Now in its fourth year, AIT remains a highly sought-after programme and is perceived to be beneficial to one's career. Participants in the programme have contributed significantly to their field of interest despite being in the earliest career stages. The programme has the potential to help retain medical talent in Ireland. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first study to describe the successful implementation of a combined clinical-academic training programme for junior doctors in Ireland in their first postgraduate year.

The Context, Input, Process and Product evaluation model was a useful and appropriate method to evaluate the training programme.

Equitable stakeholder input was key to the programme’s successful evaluation and implementation.

The search for research outputs was conducted by one researcher using one database (PubMed).

It was not possible to compare the scientific outputs of participants in the programme with a control group.

Background

Clinical academics (or clinician scientists) occupy a crucial role in healthcare systems, bridging a gap between research and patient care. Through their combined clinical and academic practice, they have the potential to identify research questions relevant to patient care and to translate research from bench to bedside.1–4 A strong research culture is associated with improved patient outcomes.5 Despite the importance of the role, there are concerns that it is under threat: in the USA, the clinical academic workforce is ageing, with the average age at which a first independent NIH grant is awarded approaching 43. A leaky pipeline compounds the problem: one-third of young investigators with mentored NIH grants never apply for independent R01 grants, and many more are lost at later career stages.6 Similar problems exist in Australia and New Zealand, where the ability to recruit and retain clinical academics is potentially outstripped by an increase in demand for undergraduate medical education.7 Action is required to continue to attract talented graduates to careers in clinical academic practice.8

International literature suggests that those who engage with research early in their clinical careers will continue to do so. Graduates of the UK’s Academic Foundation Programme (UKAFP) are over forty times more likely to progress to specialty academic training compared with graduates of the standard foundation programme (9.5% vs 0.2%).9 A US review of 25 years of National Institutes of Health (NIH)-sponsored Medical Student Research Fellowship programmes found that up to half of former participants (n=1000) considered themselves to be working in academic medicine, and the vast majority had conducted additional research after their medical student experience.10 Publication during residency is recognised as a determinant of the capacity to publish in a future career.11 12 In psychiatry, for example, the decision to pursue a research career is already well established by residency and very few who have less than the highest level of interest in research by that time eventually enrol in research career tracks. Together these indicate the crucial importance of early influences13.

Ireland’s context

Opportunities for clinical research in Ireland are increasing. There has been considerable investment in clinical research infrastructure over the last decade. There now exists seven Clinical Research Facilities nationwide, multiple Clinical Trial Networks and Science Foundation Ireland-funded research institutes across the university and healthcare sector. Ireland has retained its reputation and position in the top 10 most innovative countries in Europe.14 The Irish government has pledged to build a strong research and innovation base in Ireland with the aim of becoming a Global Innovation Leader.15 This has prompted expansion of the number of Health Research Board funded clinician scientist awards, as a result academic capacity has increased in cancer, immunology, personalised medicine, neuroscience, ageing, bioinformatics and medical devices.

If opportunities for clinical research are increasing, clinical training and education must provide a workforce that is able to avail of these opportunities. Prior to 2017, there was no combined clinical and academic training programme in Ireland for junior doctors at any level. In July 2017, the academic internship track (AIT) was launched, it provides a 1-year combined clinical and academic training programme to junior doctors in their first postgraduate year (interns). Currently in its fourth year, we offer an evaluation of the programme’s development and success.

Programme evaluation

Context, Input, Process and Product model of evaluation

The Context, Input, Process and Product (CIPP) model includes formative and summative assessments with an emphasis on not only proving that a programme works, but also seeking areas for improvement. Key to the process is equity, and engagement is sought with all stakeholders in a fair and balanced manner16 17 (table 1).

Table 1

Academic internship track evaluation model (adapted from Stufflebeam and Coryn16)

CIPP model of evaluation
Type of evaluation	Questions asked	Methods
Context	What educational need is being met?What goals should be pursued to meet the needs?	Literature reviewMeetings with key stakeholdersOnline survey of undergraduate medical students
Input	What are the most promising approaches to meeting the educational need?What might some of the barriers to successful implementation of the programme be?	Literature reviewCurriculum review of similar programmesMeeting with Director of UKAFP, an established programmeOnline survey of undergraduate medical students
Process	To what extent was the programme carried out as planned?	Review of recruitment to process: post uptake and attritionSurvey of exiting academic interns seeking feedback on programme
Product	To what extent did the programme effectively address the original objectives?	Review of scientific outputs of previous participants in the programmeFeedback from exiting interns on career intentionsMonitoring of career trajectory and retention in Ireland

CIPP, Context, Input, Process and Product; UKAFP, UK’s Academic Foundation Programme.

Context evaluation

A context evaluation is similar but more comprehensive than a needs assessment as it also incorporates an assessment of problems, assets and opportunities.16 A literature review identified a need to provide training pathways for clinical academics to build and sustain the workforce. During the early stages of programme development, meetings were held with three groups who were key stakeholders: the Intern Network Executive, the body responsible for delivering intern training and assessment; the Irish Medical Council’s Education Committee who oversee intern education and training, and the Health Service Executive’s National Doctors Training and Planning, the body responsible for ensuring the Irish Health Service is provided with the appropriate number of medical specialists. All stakeholders were in support of the development of an academic track for internship whose aims are to:

Provide opportunities for doctors at the beginning of their careers to continue or develop research, education and healthcare leadership and management skills.

Promote scientific discovery and sustained academic development within the context of contemporary clinical practice.

Retain medical talent in Ireland.

Funding was subsequently provided for 24 posts nationally, representing ~3% of all intern posts.

Medical students may also be considered key stakeholders, so an anonymous online survey was distributed to medical students in their fourth (penultimate) year of training; respondents indicated a high level of interest in a potential academic internship.18

Input evaluation

A literature review was undertaken to investigate approaches to combined clinical academic training programmes for junior doctors at the earliest career stages. The most similar programme to the Academic Intern Track in Ireland is the UK Academic Foundation Programme (UKAFP). The programme design and curriculum were examined, and a meeting was held with the programme director who provided further information and advice on programme design and implementation.

The literature review also revealed barriers to junior doctors’ participation in clinical research which include time constraints owing to clinical/workplace duties, lack of statistical knowledge and research training, and lack of supervisors.19–21

The survey of fourth-year medical students asked what elements of a clinical academic training programme would be of greatest importance: they agreed that protected time within the working week, a named supervisor and access to training on basic and advanced research skills would be important or very important.18

Collating the advice, evidence from the literature and feedback from students, the AIT programme was designed to incorporate: protected time within the working week, usually occurring during one 3-month rotation during which time is divided equally between clinical and academic work; a named academic supervisor; funding in the form of a €2000 bursary to cover research and travel costs for the year; and study days and seminars to provide training in research, education and leadership skills.

Process evaluation

Uptake of AIT posts

Now in its fourth year, the AIT has seen 72 academic interns complete the programme with a current cohort of 24 due to complete training in July 2021. Similar numbers of candidates apply to the academic track annually, with initial applications outnumbering posts by about 16 to 1. There has been 100% uptake of posts each year, and 0% attrition from the programme.

Feedback from exiting academic interns

An anonymous online survey was distributed to the exiting interns of the 2017/2018 and 2018/2019 cohort (n=48)(online supplemental file 1). Twelve interns out of 24 responded in 2018 and 21/24 in 2019 giving an overall response rate of 69% (n=33).

In the first year of the programme, 92% indicated that their experience of the clinical component of the year was good or excellent, and 83% indicated that the quality of experience of the academic part of the year was good or excellent. Of that first cohort, 42% had also applied for or considered applying for the UKAFP prior to accepting the post on the AIT. Fifty-eight per cent held another third level qualification.

Ninety-two per cent of respondents perceived a benefit from participating in the AIT. As well as learning new skills, producing papers for publication and learning good time-management skills, participants appreciated the opportunity to work closely with mentors and get a sense of life as a clinical academic. Several participants mentioned that they found the overall experience to be very fulfilling. Eighty-three per cent would recommend the academic track to a friend. The 2018/2019 survey showed very similar results.

Product evaluation

Early scientific outputs

An individual search on PubMed for each of the 72 AIT participants was conducted between 10 June 2020 and 20 June 2020. Only original, peer-reviewed research papers, essays or review articles published ≤5 years prior to graduation and any time postgraduation were included, with letters, commentary, abstracts, conference proceedings and editorials excluded.

One hundred and thirty-five articles which named participants in the AIT from 2017 to 2019 (n=72) as authors were identified. Of these, 50 were published in the five years prior to graduation and 85 either during or at 1–2 years postcompletion of the AIT (figure 1).

Figure 1

Peer-reviewed publications of first three cohorts of academic interns (n=72). AIT, Academic Internship Track.

Forty-one of the 72 former academic interns (56.94%) have achieved publication of a research paper/review article in a peer-reviewed journal to date; a substantial number of publications were open access. Removing one very prolific individual from the cohort still indicates a high level of publication, with 94 total publications, 33 prior to the academic track and 61 during or after, with a range of 0–8 publications. In this group, there was an average of 0.46 publications per intern prior to the academic track and 0.86 publications during or after the academic track, however, it must be noted that the duration of time since commencement of the academic track and hence the amount of data available varied for each cohort.

The quality of publications based on journal impact factor is also an indicator for success of the programme (table 2).

Table 2

Top five publications by journal impact factor (published after start of AIT)

Journal	No of publications	Impact factor*
British Journal of Anaesthesia	2	6.199
Free Radical Biology & Medicine	1	6.17
The American Journal of Sports Medicine	6	6.057
Journal of Clinical Investigation Insight	1	6.014
Annals of Medicine	1	5.435

*Impact factors recorded at time of database review, June 2020

AIT, Academic Internship Track.

Academic interns develop their own project proposal and identify an area of research that is of interest to them. On reviewing the project titles and primary supervisors’ occupation, we found a total of 28 specialties represented, with oncology, anaesthetics, obstetrics and gynaecology, orthopaedic surgery and paediatrics being the most represented specialties (figure 2). Analysis suggests interns publish predominantly in the specialty represented by their academic track projects: 64 of the 85 publications attributed to academic interns during and after the AIT are within the same field as their academic track project.

Figure 2

Specialties represented by academic track interns (n=72). ENT, Ear, Nose and Throat surgery. Med El, Medicine for the Elderly.

Feedback from exiting interns indicated outputs other than publication or presentation, these included outreach activities, gaining additional qualifications, achieving awards and accolades for their academic work, gaining teaching experience and learning new research skills.

Career trajectory and retention in Ireland

In 2017/2018 on formal reporting, 10/12 participants confirmed they were remaining in Ireland for their first Senior House Officer (SHO) year, with two planning to travel abroad. However, informally 22/24 (or 92%) planned to stay in Ireland and subsequently did. In 2018/2019, none of the respondents reported an intention to apply for a training scheme abroad and just 10% planned to take up non-training scheme work abroad. Of the remainder, most (42.6%) planned to take up a training post in Ireland; others planned to join a GP scheme, take a stand-alone post or undertake further study.

The first AIT cohort provided open-ended responses regarding 5-year career intentions, most reported that they were aiming for higher specialist training (HST) schemes without indicating a specific location; only one indicated a preference to train abroad. By the second cohort, 66.7% indicated an aspiration for HST in Ireland and the Irish Clinical Academic Training (ICAT) programme, a combined specialist training and PhD programme. Other preferences such as non-integrated research/academia are shown (figure 3).

Figure 3

Longer-term career plans.

With the programme just in its fourth year, it is too early to determine its effects on career trajectory, however, this will be monitored as a key outcome going forward.

Discussion

Evaluation of the first 4 years of the AIT in Ireland has shown three key findings: a demand to develop key research skills by early career junior doctors, the capacity to translate these skills and contribute to a chosen field of study, and that satisfying these needs may contribute to the retention of this valuable expertise.

The survey of undergraduate students indicated a significant interest in the programme, and this has been borne out in the recruitment process. Similar numbers apply for academic internship annually, with applications received far exceeding the places available. In 4 years, we have never had a post unfilled, and no-one has left the programme early. The level of interest in the programme has been sustained, even during the uncertainties of the COVID-19 pandemic, indicating the perceived quality of the programme and its benefit to one’s career. It is likely that there is great scope for expansion of the programme.

Even at the earliest career stage, participants in a combined clinical academic training programme have the potential to contribute substantially to their field of interest. The majority of participants in the first 3 years of the academic track have already contributed to publications as named authors and it is anticipated that this number will rise significantly given the lengthy process of manuscript preparation and publication. Publications either during or after the academic track outnumber publications in the 5 years before the academic track (when participants would have been undergraduate medical students). A substantial number of publications (38.8%) were open access, and many were published in high calibre journals.

Data from a similar programme run in the UK, the UKAFP, indicates that on exiting the 2-year programme, 33% of participants have submitted or prepared papers for publication.22 While it is not a direct comparison, 57% of AIT participants having achieved publication at 0, 1 and 2 years postacademic track would suggest that they are prolific compared with their peers.

Medical doctors at all levels, clinical academics or not, are thought to publish at a rate of 0.47 papers per year. Publication rate is much lower at the earliest career stages.23 Taking 1 year of the academic track, 2018/2019 as an example, the 24 participants succeeded in publishing at an average rate of 1.27 papers each per year in the 2 years since their academic internship start date, indicating a rate of publication over twice that of the average doctor, despite being at the earliest career stage.

There are some limitations of the database review: it was conducted by one researcher only, and only one database (PubMed) was searched. Therefore, it is possible that papers which should have been attributed to participants in the programme were missed, whereas others may have been misattributed. Further, it was not possible to compare the research outputs of participants with a control group, for example, a group of interns who did not participate in the academic track, due to a lack of other identifying factors such as field of interest or supervisor.

It is possible that creating such opportunities at an early career stage can help retain doctors in Ireland in both the immediate and longer term. Participants in the AIT tend to remain in Ireland after completion of the programme; in contrast, almost two thirds of interns who have completed the standard intern-training programme plan to emigrate abroad at least for a year.24 Ireland has ongoing problems recruiting and retaining health personnel and despite producing the most doctors among Organisation for Economic Co-operation and Development countries, it remains below average for practising doctors per 1000 population. A national doctor retention strategy has been in place since 2015 but has enjoyed limited success.25 Factors influencing the higher retention rate may include intrinsic factors such as a focused career plan. Alternatively, academic interns having fostered relationships with other researchers may wish to continue to build their networks, emphasising the importance of mentors to early career clinical academics.

A different perspective might suggest that the academic track creates greater job satisfaction and hence greater retention. Burn-out is high among Irish junior doctors, and that this contributes to the decision to emigrate.26 A highly demanding job combined with low or scarce resources can negatively impact on employee well-being, whereas a highly demanding job with high levels of support and resources can have a positive impact.27 Exiting academic interns express high levels of satisfaction with the programme. It’s possible provision of additional resources to achieve their academic goals (in the form of time, funding and mentorship) enhance the academic interns’ experiences of intern year, mitigate against burn-out and encourage their retention within the Irish healthcare system, however, further research would be required to investigate this hypothesis. It is likely multiple factors are at play.

The AIT is not without its challenges: a small number of posts nationally means many excellent candidates miss out on posts each year, limited funding for study days and other events can impact on their delivery, and the arrival of COVID-19 has necessitated a move to online educational sessions as well as a potential increase in conflict between clinical responsibilities and academic goals. Despite these challenges, it remains a training programme that is highly regarded among students and interns and is perceived to be of value to one’s career. While it is early days yet in terms of career trajectory, it is hoped that this positive early experience will provide these talented graduates with a stepping-stone towards further clinical academic training, launching their careers as clinical academics and helping to build and sustain this workforce for the future.