Longitudinal Outcome of Programmatic Assessment of International Medical Graduates.

INTRODUCTION: Australia depends on international medical graduates (IMGs) to meet workforce shortages. The current standard assessment for IMGs is by clinical examination in observed structured clinical encounter (OSCE) format lasting 200 minutes. There are concerns about adequateness of this assessment as it does not test the qualities required to practice in a new country. We introduced a programmatic performance-based assessment for IMGs to prepare them to meet these challenges. The workplace-based assessment (WBA) program involves six-month longitudinal programmatic assessments comprising of 12 mini-clinical evaluation exercises (Mini-CEX), five case-based discussions (CBD), two in-training assessments (ITAs) and two sets of multisource feedback (MSF) assessments. We assessed 254 IMGs since 2010. We conducted a survey to evaluate the satisfaction with the program and the outcomes of these doctors.
METHODS: We surveyed 254 candidates from 2010 to 2020. The survey used "SelectSurvey" tool with 12 questions and free-text comments. All candidates were sent the survey link to their last registered mobile phone using "Telstra Instant Messaging Service". We analysed the data using Microsoft "Excel".
RESULTS: We received 153 (60%) responses. Amongst them, 141 (92%) candidates did not require further supervised practice for general registration and 129 (84%) candidates hold general/specialist registration. The candidates found the program useful and felt well supported. They appreciated real patient encounters. The feedback with positive critiquing was helpful in improving their clinical practice. The negative themes were program costs and frustration with the length of the program.
CONCLUSION: Upon completion of the WBA program and obtaining the AMC certificate, most of the doctors were able to gain general registration. Seventy-eight (50%) candidates chose to continue their careers within the local area with 124 (80%) of them within the state. Our survey shows a comprehensive assessment program with immediate constructive feedback produces competent doctors to fill the medical workforce shortages.

Introduction

Australia, like many other developed countries, depends on international medical graduates (IMGs) to meet workforce shortages. The proportion of doctors in the Australian medical workforce who obtained their initial qualification overseas increased by three per cent from 2013 to 2016.1 In 2016, statistics from the Medical Board of Australia indicated 32.2% of all the registered and employed medical workforce obtained their initial qualification overseas.1 IMGs are a key part of Australian medical workforce, most notably in rural areas.1 The above statistics indicate that 44.9% of the medical workforce working in outer regional areas and 43.1% in remote areas obtained their initial qualification overseas.1 This dependency is expected to continue for many more years.

The March 2012 Australian Federal Government report titled “Lost In The Labyrinth: Report on the inquiry into registration processes and support for overseas trained doctors” by the House of Representatives Standing Committee for Health and Ageing points to many issues that needed rectification.2 It made 45 recommendations including Workplace-Based Assessment (WBA), cross cultural orientation, and clinical supervision both before and after placement, especially for IMGs in regional, rural and remote areas.2 Medical migration, is an international issue and many of the recommendations are also generalizable to other countries.

The Australian Medical Council (AMC) provides several pathways for IMGs to obtain registration with the Medical Board of Australia/Australian Health Practitioner Regulation Agency (MBA/AHPRA). The standard pathway of assessment for IMGs includes verification of qualifications as a doctor to practice medicine in their country of training, a MCQ delivered as a computer adaptive test followed by clinical skills assessment by an Observed Structured Clinical Examination (OSCE) format or Workplace-Based Assessment (WBA).3,4 The OSCE assesses the candidates, using reliable and valid methods broadly accepted in medical education. However, an OSCE is only able to assess a candidate at moment-in-time. It is conducted over 200 minutes in a testing centre with 16 stations.3 Most of the stations use role players/actors with occasional real patients and some simulation equipment.

WBA is a novel method of programmatic assessments that provides an alternative pathway for IMGs working in the Australian healthcare system.5 The WBA program in Newcastle was developed in 2010 with the aim of addressing the problems associated with assessing and integrating IMGs into the Australian healthcare system.6 This is a proactive approach. It was developed in consultation with the AMC, the University of Newcastle and the Hunter New England Local Health District (HNELHD). WBA programs have high levels of validity because they are conducted in the workplace and are executed as part of normal clinical practice with real patients.7–10

From our literature search, this is the first programmatic assessment for IMGs anywhere in the world. The aim of this study was to evaluate the long-term outcome of the candidates, their satisfaction with the program and find out their professional trajectory.

Methods

The WBA Program in Newcastle is a six-month longitudinal programmatic assessment process focusing on clinical performance. The aim of this comparatively long assessment process is to provide assessors with multiple opportunities to assess the performance of IMGs.6 This in turn, provides the IMGs with the opportunity to improve their performance. The assessment includes 12 Mini-Clinical Evaluation Exercises (Mini-CEXs), five Case-based Discussions (CBDs), two In-training Assessments (ITAs) and two sets of 360-degree Multisource Feedback (MSF) assessments (one formative and one summative) (see Table 1). The disciplines covered are medicine, surgery, emergency, paediatrics, mental health and obstetrics and gynaecology.6 Candidates who pass this assessment program are eligible for AMC certification.4 Since its commencement in 2010, this program has assessed over 250 candidates in six hospitals within the Hunter New England Local Health District (HNELHD).11 The WBA Program is supported by the International Medical Graduate Unit (IMG Unit), which provides targeted IMG orientation, ongoing education, mentoring and linguistic support. Thus, the IMG Unit helps IMGs assimilate into the local medical workforce. A previous study assessing the composite reliability of WBA, found that it is a reliable assessment.12

Table 1

Tools and Criteria for Pass in WBA (Have to Pass All 4 Segments)

Tools of WBA
Mini-clinical evaluation exercises
Complete all 12Pass 8 out of 12Pass 1 in each of 6 clinical disciplines
Case-based discussions (CBDs)
Complete 5Pass 4 out 5
360° assessments (multisource feedback)
Month 1 Formative: 3 medical and 3 non- medical colleagues ● Border line candidates counselled and remediated
Month 6 summative: 3 medial and 3 non- medical colleagues ● Had to be “satisfactory”
In-training assessments (ITAs) x 2
Complete 2 end-of-term ITAs within 6 months Assessed at “at expected” level (or equivalent minimum) on both ITAs

The WBA team developed a survey using the “SelectSurvey” tool, with 12 questions and a section for free-text comments (see Table 2). This survey was piloted and then all the 254 candidates who completed the program were sent the survey link to their last registered mobile phone using “Telstra Instant Messaging Service” (TIMS). Follow-up reminder messages were sent every two weeks during the survey period. The survey was open for six weeks.

Table 2

Twelve Survey Questions

Questions
What year did you successfully complete the WBA Program
What is your country of origin? (Optional)
How many years had you worked in Australia as a doctor before commencing the WBA Program?
Did you attempt the AMC clinical exam prior to commencing the WBA Program?
What is your Registration type with AHPRA? You can choose more than one if applicable.
Do you have any restrictions on your registration with AHPRA?
If your registration type is General or Specialist, what year did you receive it?
Were you required to do further supervised practice after completing your WBA Program to meet requirements for general or specialist registration?
What is your current position?
What is the geographical location of your current place of practice?
What is the nature of your current practice? You can choose more than one answer.
Would you be willing to be contacted for WBA research and/or resource development activities?
Do you have any comments/suggestions/feedback?

The responses were collected and collated in Microsoft “Excel”. The data was analysed using the statistics tools (pivot table function) available within Microsoft “Excel”. As there was limited qualitative data, we did not undertake qualitative data analysis. However, we have included some of the free-text comments.

Results

Two hundred and fifty-four WBA candidates were surveyed during the time period of the audit. We received 153 (60%) responses.

Positive Themes

The candidates highlighted the following ‘positive themes’ a) acceptability of the program, “Best program, best people I am very thankful”, “Excellent program to help doctors and to help health workforce”; b) educational value and the immediate feedback “Great program, very educational” “I think the program was great with every mini CEX and case based discussion the examiner provided great feedback and I think that has helped me improve.”; c) enabled better integration into the Australian Health Care System and made it a learning experience “WBA program worked as an educational experience and helped with better integration into Australian health system” and d) appreciated real patient encounters rather than simulations. “It feels more realistic for me to talk to an actual patient than a pretend one”, “Very helpful programme for IMG, better assessment process than AMC clinical exam”. (See Table 1).

Negative Themes

The major ‘negative themes’ were, a) costs of the program and b) frustration with the length of the longitudinal assessment. (See Table 3 and ).

Table 3

Candidates Free-Text Feedback

Candidate Comments

● I found the WBA team supportive and helpful. I have recommended this programme to all foreign doctors looking to get their general registration● I think the program was great with every mini CEX and case-based discussion the examiner provided great feedback and I think that has helped me improve● I always recommended WBA to my IMG colleagues.● The WBA program helped me very much. I felt supported and performed better in the assessments. Now, I can contribute to the Australian health system.● WBA was a great learning experience which helped me complete my supervised practise successfully and achieve specialist registration● WBA program worked as an educational experience and helped with better integration into Australian health system● WBA program is a true reflection of the knowledge and experience we gain in the Australian Health System.● The WBA program is an excellent clinical pathway. The clinical scenarios were patients in hospital and relevant to the clinical exam● Very helpful programme for IMG, better assessment process than AMC clinical exam● I am very thankful for getting into WBA program, it feels more realistic for me to talk to an actual patient than a pretend one● The program is valuable and one of the best available. My main issue surrounds the mandated 6 months of employment with HNE prior to be accepted.● Make the program accessible for doctors in the area health general practices who have cleared their part 1 AMC exam

During the decade of implementation of the WBA Program, candidates from 32 countries were assessed. Of these, 35 (22.6%) had under one year of experience working in an Australian medical system, 80 (51.6%) had between 1–3 years’ experience, 40 (25.8%) had more than 3 years’ experience working under provisional registration. (See and for further details).

Sixty-nine (45%) candidates who did WBA had previously attempted the Australian Medical Council (AMC) clinical exam [Figure 1]. Of these, 59% of the candidates had attempted the clinical exam more than once [Figure 2].

Figure 1

Previous attempt at AMC clinical exam.

Figure 2

Number of times AMC clinical exam was attempted.

After completion of the WBA program, 141 (92%) of the candidates did not require supervised practice to apply for general registration. One hundred and twenty-nine candidates (84%) currently hold general and/or specialist registration with AHPRA and 134 (87%) candidates did not have any restrictions/conditions on their registration.

Currently, 15 (10%) candidates are practising as specialists and 40 (26%) are in a specialist training program. Thirty-eight (25%) are general practitioners and 22 (14%) are in general practice training program.

After completion of the WBA program, 78 (50%) of the candidates continue to work in the local health district and a further 46 (30%) are working within the state [Figure 3].

Figure 3

Current location of practice.

Eighty-four (54%) of the candidates are working in a hospital setting, 46 (30%) are working in general practice and 12 (8%) are working in a hospital as well as general practice.

Discussion

This survey had a response rate of 60%. A sizeable number of the candidates who undertook WBA had previously attempted the AMC clinical exam. About 59% of them had multiple unsuccessful attempts at the AMC clinical exam. This seems to be consistent with a recent paper that showed that the pass rate in the AMC clinical exam was between 35% and 45% from 2010 to 2019.13 The AMC annual report in 2019 shows an AMC clinical examination pass rate of 27.1%.14 The pass rate of the HNELHD WBA program is 99%.

Given that six months of prior Australian medical workforce experience is a pre-requisite for enrolment in the program, most of the candidates did not require further supervised practice in order to be eligible for general/specialist registration. This is in contrast to the candidates who passed the AMC clinical examination, who are usually required to complete supervised practice before being eligible for general registration.3

Upon completion of the WBA program and acquisition of the AMC certificate, 90% of candidates were able to achieve general and/or specialist registration. The remaining 10% had limited registration and were in the process of completing their requirements for general/specialist registration as per the AHPRA regulations.

The survey indicates that 50% of the WBA candidates chose to continue their medical careers within the local area and a total 80% of the respondents are working in New South Wales. All the respondents are working either in a hospital setting or general practice providing medical care to the Australian community. Thus, these additional, well assessed and competent doctors are now filling the gaps in the health-care system in NSW.

A key principle of WBA is assessment of several domains by multiple different assessors with feedback requirement built into each assessment encounter.6,8 The qualitative comments indicated that this focused feedback with positive critiquing was found to be a crucial element that helped candidates improve their clinical and interpersonal skills. The candidates found the feedback to be relevant to working in the Australian medical workforce and helped them progress professionally. This has been consistent since the inception of the program in 2010.15

Limitations

A Workplace-based assessment program by its nature can only be offered to the doctors already employed in the Australian medical workforce. It requires an extensive administrative setup, multiple assessors in various disciplines to be calibrated/re-calibrated and be available to conduct these comprehensive assessments.6,7

The six-month program means a time commitment is required from the program coordinators, assessors and the candidates. Some other WBA programs in Australia extend up to or over 12 months.4,14 Most programs require six months of Australian medical experience prior to enrolment as well.4,6,14

The cost of the program was also thought to be one of its limitations. In a study in 2014, the actual cost (AUD $16,226) of the program outweighed the fees paid (AUD $6000) by the candidates.16 A large portion of this program works on the goodwill of the assessors. Currently, the HNELHD candidates pay $10,000 for a six-month program. The program is supported by the local health district, since it appreciates the benefit of well-trained and assessed IMGs to practice in the same local health district or within the region. The local health district recognises this to be a long-term investment for the welfare of the community.

The limitation of the survey was the inability to contact all the candidates. Some candidates had changed their contact information. The email addresses or mobile numbers of some others were not up to date. There were no in-depth interviews built in, and the researchers did not have the opportunity to further question the respondents.

Conclusion

The WBA program was found to be valid, reliable and acceptable to the learner from our previous work and this current evaluation. The nature of the program made it an educational experience as an “assessment for learning” as opposed to “assessment of learning”. Our literature review found this to be consistent with the experience in other countries/healthcare settings.14–16 Our survey shows that having such an assessment program produces competent doctors. The feedback from this survey and the success of the candidates indicate programmatic assessment is an authenticated way of IMG assessment, this will prepare them for the workplace for the next several decades to provide better health care for the population.