Workplace-based assessment; learner and assessor perspectives.

OBJECTIVE: To examine the acceptability and educational impact of the workplace-based assessment program for international medical graduates on candidates and assessors.
METHOD: A grounded theory-based qualitative analysis of the experiences of 17 candidates and eleven assessors using focus groups, interviews, and surveys.
RESULTS: Both candidates and assessors identified positive opportunities for improved performance of international medical graduates. Their integration into the workforce was facilitated by improved communication and peer acceptance, from ongoing multifaceted feedback and time to practice skills.
CONCLUSION: This study showed a high level of acceptability of the Newcastle workplace-based assessment program among candidates and assessors.

Introduction

In Australia, the traditional path for registering international medical graduates (IMGs) has been the province of the Australian Medical Council (AMC) and the Medical Board of Australia. They are responsible for the registration of all doctors in Australia by setting the relevant standards, codes, and guidelines.1 Doctors from the UK, USA, New Zealand, and Ireland can fast-track medical registration under a “Competent Authority Pathway”.1 However, doctors from other countries are required to follow a “Standard Pathway” of examinations in English, medical knowledge, and clinical skills.

The AMC Clinical Examination is a 16-component multistation assessment of clinical skills in medicine, surgery, obstetrics, gynecology, pediatrics, and psychiatry. Most stations use role players and the entire assessment lasts for 180 minutes.1 The “Standard Pathway” pass rate is approximately 50%, and this coupled with a substantial backlog of candidates means it can take years for IMGs to obtain registration.2 This deprives IMGs of valuable work experience and the Australian medical system of much needed practitioners. Additionally, there is evidence that this assessment may not be optimal as IMGs reportedly receive more official complaints and have more adverse findings made against them compared with non-IMG doctors.3

The Newcastle workplace-based assessment (WBA) program was developed in 2010 with the aim of addressing problems associated with assessing and integrating IMGs into the Australian medical system.4 The WBA is a novel method of IMG assessment. IMGs who have been granted limited medical registration and work in an accredited Australian hospital, due to a doctor shortage, are eligible for this assessment. Details of this program have been previously published.4,5 A recent Australian Parliament Committee into registration processes for overseas doctors has recommended WBA to be used nationally.2

The WBA program consists of a 6-month assessment process focusing on clinical performance. This comparatively long assessment period provides assessors with multiple opportunities to observe IMGs in the workplace. Candidates receive ongoing feedback and time to improve their performance. It also provides time for IMGs to become familiar with the Australian medical system and the needs and expectations of patients.

Van Der Vleuten’s utility formula for assessment is a validated equation which combines the elements of validity, reliability, educational impact, cost effectiveness, and acceptability (Table 1).6 The utility of the WBA as an effective evaluation tool has been assessed by addressing each of these elements. The utility formula is as follows: where R = reliability, V = validity, EI = educational impact, A = acceptability, and C = cost.

Table 1

Positive themes: assessors

• They [candidates] feel part of the whole team. Now I know who they are and wave to them in the corridors. I can’t remember their assessments.

• WBA is playing a role in bringing [doctors] here [to regional] hospitals.

• [It will] enhance our national and international reputation as a center for educational excellence.

Abbreviation: WBA, workplace-based assessment.

The face validity of the WBA is demonstrated through clinical skills assessment which includes detailed and repeated patient assessment, including history taking, examination, and planning management. Communication is an essential element of effective medicine which is adversely affected by language, cultural, and social barriers. The WBA uses 360° assessments or multisource feedback where members of the health team, including medical, nursing, and allied and administrative staff, provide feedback on the candidate’s communication, teamwork, and professional skills.4

The reliability of the assessment tools used in the WBA are well established and described elsewhere.4 Cost of the WBA is also detailed in another paper.5 In the present paper, we focus on the acceptability and educational impact of the WBA program by exploring the views and experiences of assessors and candidates.

Methods

A grounded theory qualitative methodology was used to explore the experiences and views of assessors and candidates in the WBA program.7

Participants

Candidates from five WBA cohorts (n=95) were invited by email to participate in one of two candidate focus groups. Interested candidates who were not able to attend the focus groups were offered telephone interviews. Sampling was purposive in that some candidates were included on the basis of their country of origin, sex, and area of work. All assessors in the WBA program (n=72) were invited by email to attend an assessor focus group or interview.

Data collection

Participants provided informed consent and the study was approved by the Hunter New England Ethics Committee on May 14, 2010. Interviews and focus groups were conducted by two trained researchers (UP, JS). Each session was audio recorded and the recordings transcribed for the purposes of analysis. Identifying information such as names and country of origin was removed from transcripts in order to preserve anonymity. Topic guides were used; however, the format of the interactions was informal and conversational, and participants were encouraged to express their views freely.

Data analysis

The transcribed text from the interviews and focus groups was entered into NVivo 10.0 (QSR NVivo version 10; QSR International Pty Ltd, Victoria, Australia). The source texts were closely read and reread multiple times, independently, by three researchers (UP, JS, AW). Themes were identified and coded for concepts, with coded segments then analyzed and categorized thematically. Coded segments of text were compared for similarities and differences and then categorized. One-third of all text was coded separately by two of the coders to ensure consistency in coding. Saturation of themes was reached after six interactions of focus groups and interviews with no new themes emerging.

Results

A total of 26 candidates and eleven assessors participated in the study. The candidates’ countries of origin included India, Pakistan, The Philippines, Malaysia, Pakistan, Egypt, Germany, Jordan, People’s Republic of China, Indonesia, South American countries, and South Africa. Their special-ties included medicine, psychiatry, oncology, emergency medicine, and general surgery. Assessors’ specialities were psychiatry, obstetrics and gynecology, medicine, pediatrics, general practice, surgery, and emergency medicine.

The major themes arising from the two groups were similar. Both assessors and candidates saw great potential in the use of WBA due to the in-depth and broad knowledge gained by the candidates as well as the multifaceted approach to assessments. Both groups like the concept of having collegial relationships and looked forward to future working relationships on this basis. However, the negative side of these relationships included difficulty in assessing someone who works as a registrar at an intern level. The themes have been divided into positive and negative and are discussed in terms of assessors and candidates.

Positive themes

Opportunities

The majority of candidates and assessors were positive about the program and the direct benefits to IMGs as well as the indirect benefits to the health service itself (Tables 1 and 2). Assessors felt WBA raised the reputation of Australia and the Hunter New England Local Health District in educational terms while increasing the numbers of doctors interested in working in regional Australia. Candidates felt welcomed and valued the advice and education they received during their training.

Table 2

Positive themes: candidates

• [W]hat actually hinders us from becoming good practitioners is actually communication. I think the idea of communication is very important.

• You know a lot more people. You have a good relationship with consultants.

• 360°s were a reassurance for me that I am liked by my colleagues. [Having] 12 people that say good things about me [is] definitely a confidence booster.

• [F]eedback from 20 specialists telling you one or two points each – they are very wise words. I received some excellent feedback and it really did affect my practise.

• I have found the documentation feedback good because [usually] no-one tells you how to do the notes. This is feedback that you normally don’t get.

• [Y]ou can plan your assignments and study. That is the best thing I have found.

• It was six months of seeing patients which was better than just the clinical exam.

Communication skills

Candidates and assessors found the ongoing revision of education and knowledge improved clinical and communication skills, as well as developing expertise and workplace excellence. Cultural and communication differences between their country of origin and Australia were greater than most candidates had expected. The appropriate way to address patients and colleagues, especially senior staff, was in contrast to most candidates’ expectations. Time was needed to understand and accept the use of humor and informality of Australian culture. Candidates knew it was essential to quickly gain the necessary communication skills to overcome these differences in order to pass registration and to practice effectively, and they felt the WBA was an excellent means of achieving this.

Acceptance

Candidates and assessors acknowledged the advantage of being accepted as part of the health team. They described how the WBA program increased their confidence, networking, and team building. Candidates were impressed and surprised at the accessibility of their assessors and felt included and supported in their work. Assessors felt less pressured knowing the candidate had time to improve their skills and felt candidates were open to constructive criticism.

Holistic education

Education through ongoing feedback provided candidates with important clinical and practical information. Candidates felt that having a variety of assessors providing multifaceted advice and expertise improved their knowledge, especially when the information was demonstrated on a real patient. Workplace-based training provided candidates with information they would not have otherwise received or known that they needed, such as administrative input regarding patient records. At no time before the WBA were candidates taught anything about medical records, paperwork, or how to access electronic patient data. The time requirement for WBA was viewed positively by most candidates who felt it provided time to prepare for assessments and more opportunity to gain clinical skills.

Negative themes

Inconsistency

There was a recurrent theme among candidates that assessors needed more consistency in terms of competence and objectivity during assessment procedures. While candidates should be examined at an intern level, many felt their assessments were harder because their assessors knew they were registrars. The assessors admitted feeling conflicted examining at an intern level if the doctor was working as a registrar or resident (Tables 3 and 4).

Table 3

Negative themes: candidates

• [C]onsistency amongst the assessors isn’t the same and should be improved

• I’ve heard one clinician say “You’ve [taken] the easy way out.”

• I had to do three assessments in one and a half weeks which was quite stressful at the time but I got through it.

• I was so stressed before, and I think the orientation was not enough, so I don’t know how it can be managed to be better, but I think it needs to be more detailed.

Table 4

Negative themes: assessors

• Candidates need to know that they can ask questions but they can’t challenge you to the degree where they try to take you on when they want you to change your mind. So you try to make sure you are in a safe place. And that has come up but to be honest, it’s usually males…

• Because I get CMOs, residents and the majority of them are registrars, so I always find it hard and probably pitch it [the assessment] a bit too high for the residents and a bit too low for the registrar.

Abbreviation: CMOs, Career Medical Officers.

Intimidation

Issues of intimidation, both psychological and physical, were raised by assessors. The most serious were experienced by female assessors providing negative feedback to male candidates. It was felt this situation could be addressed by having assessment sessions video recorded. There were also instances where candidates felt demeaned by consultants who referred to WBA as a “soft” or “easy” option as opposed to the AMC process. Some candidates said the assessment period was too long and created burdens, such as being away from family and the increased cost of living in maintaining homes in different cities.

Discussion

The WBA is a novel approach to assessing IMGs which acknowledges the complex mix of knowledge and skills that defines medicine. What is important for patients and the society is not competency (what the doctor can do) but performance (what the doctor does).8 WBA is a performance-based assessment and the candidates and assessors acknowledged and embraced this approach. Assessment drives learning and students will always focus on learning what will be assessed. However, WBA is assessment for learning.8 The longitudinal assessment by multiple examiners with feedback was built into each clinical assessment. This design provides the candidates with the opportunity to modify and improve their performance over time.

This study showed that candidates and assessors felt the WBA process was positive and presented good opportunities for performance improvement and integration of the IMG into the Australian health care environment. This was especially so in the areas of communication, peer acceptance, multifaceted feedback, and time to process and practice their skills.

A major response in this study from candidates and assessors was that assessments performed on real patients, in a workplace by practicing clinicians, had true validity. All the tools used in WBA are reliable from previous studies. We have discussed the cost of WBA in another paper5 and it is reassuring to note the cost for the Organisation is only $10,000 (per candidate), and this is a small price to pay for a long-term investment in the safety of the patients and future of the IMG.

Limitations

The WBA is currently only available for doctors already working in the health care system in accredited sites, and most candidates are not in this situation. This raises the possibility of selection bias since these doctors have gone through a recruitment process and have knowledge of the health care system. WBA is also a labor- and time-intensive process for candidates and assessors.

Implications

Given the supply shortage of doctors in Australia, the Newcastle WBA program is an innovative response to improve the pathway for IMG registration. This study showed the program is acceptable to both the candidates and assessors while reinforcing its educational impact and validity. It is cost effective and ensures the successful integration of the IMGs into our health care system and the community. It needs to be extended to more sites and IMGs. Moreover, many of the postgraduate training bodies are introducing WBA and the lessons learnt from this work can influence future assessments.