Doctors who pilot the GMC's Tests of Competence: who volunteers and why?

BACKGROUND: Doctors who are investigated by the General Medical Council for performance concerns may be required to take a Test of Competence (ToC). The tests are piloted on volunteer doctors before they are used in Fitness to Practise (FtP) investigations.
OBJECTIVES: To find out who volunteers to take a pilot ToC and why.
METHODS: This was a retrospective cohort study. Between February 2011 and October 2012 we asked doctors who volunteered for a test to complete a questionnaire about their reasons for volunteering and recruitment. We analysed the data using descriptive statistics and Pearson's χ(2) test.
RESULTS: 301 doctors completed the questionnaire. Doctors who took a ToC voluntarily were mostly women, of white ethnicity, of junior grades, working in general practice and who held a Primary Medical Qualification (PMQ) from the UK. This was a different population to doctors under investigation and all registered doctors in the UK. Most volunteers heard about the General Medical Council's pilot events through email from a colleague and used the experience to gain exam practice for forthcoming postgraduate exams.
CONCLUSIONS: The reference groups of volunteers are not representative of doctors under FtP investigation. Our findings will be used to inform future recruitment strategies with the aim to encourage better matching of groups who voluntarily pilot a ToC with those under FtP investigation. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

As part of the General Medical Council's (GMC) Fitness to Practise (FtP) procedures where concerns are raised in relation to a doctor's professional performance, a performance assessment may be carried out.1 2 This consists of a peer review and Test of Competence (ToC).2 3 In peer review, doctors are visited at their workplace by trained peer observers. In order to gain a sense of the doctor's overall practice, peer observers conduct a clinical record review, interviews with the doctor and colleagues, a case based discussion and direct observation of practice.2 3 To identify potential gaps in a doctor's knowledge base or their clinical skills, a ToC is given. This includes a written knowledge test and an Observed Structured Clinical Examination (OSCE) tailored to the doctor's grade, specialty and clinical work. There is no ‘pass mark’ for these tests; instead the marks achieved by the doctor under investigation are compared with the range of marks achieved by a cohort of control doctors who have volunteered to take a similar test. Trained investigators at the GMC then reach a judgement about a doctor's FtP.2–5

The GMC ensure that the ToC are fair and fit for purpose. This is achieved by evaluating the test material through regular piloting events organised and delivered by University College London (UCL) Medical School. Doctors with no known FtP concerns are recruited to volunteer to take a ToC in their relevant specialty. Volunteer doctors can also be reported to the GMC if they perform poorly on their pilot test, although this has been an extremely rare occurrence. The pilot events provide reference data by specialty that is used to compare with the performance of a doctor undergoing FtP investigation. UCL Medical School provide this reference data to the GMC but have no role in making a judgement about the performance of a doctor undergoing FtP investigation.

To review the validity and reliability of these tests, we studied a cohort of doctors who have volunteered to take a pilot ToC. Volunteers are self-selecting and not necessarily representative of the wider medical community in the UK. The aims of this study were to (1) compare the demographics of a group of ToC pilot volunteers with those under FtP investigation, as well as those on the UK medical register and (2) understand why they volunteered in spite of potential consequences if their ToC performance was poor.

Summary of GMC's ToC pilot events

Doctors with no known FtP concerns are recruited to volunteer to take a ToC in their relevant specialty.

Events are advertised on UCL Medical School website and flyers are distributed to royal colleges, deaneries and medical associations.

Participation criteria: doctors at a minimum of Foundation Year 2 (FY2) level, with at least 4 months work experience in the pilot event specialty within the last 12 months. They must not have piloted a ToC previously.

ToC pilot procedure: Over 1 day, volunteer doctors complete a machine marked written knowledge test and an OSCE in their relevant specialty.

Incentives to volunteer: £350 honorarium, paid travel expenses, feedback on performance, certificate of attendance to be included in portfolio.

Methods

Design and study population

This was a retrospective cohort study. All doctors who volunteered to pilot a ToC between February 2011 and October 2012 completed the research questionnaire. This included doctors between FY2 and consultant level who volunteered for a test in their current specialty. We compared their demographics with that of two other cohorts of doctors: all doctors under FtP investigation for performance concerns between the years 1996 and 2008 (n=822) and all doctors in the UK (n=245 903) recorded on the List of Registered Medical Practitioners (LRMP) as of 2011.6

Design of questionnaire

We designed a short questionnaire to find out about recruitment and reasons for volunteering. Initially the questionnaire was intended to be completed by volunteers before they attended a pilot event. Version 1 consisted of demographic questions and gave an opportunity to provide open-ended responses about their reasons for volunteering and how they heard about the pilot events. This questionnaire was distributed by email to doctors who were registered to attend a pilot event during 2010. We received 134 completed questionnaires. We analysed the qualitative data and used it to inform the final questionnaire format. We redesigned the questionnaire to make it easier for participants to complete by including appropriate closed questions, and to simplify data analysis.

Doctors who participated in the present study completed the revised questionnaire (see online supplementary appendix 1). These volunteer doctors attended a pilot event during 2011–2012 and therefore represent a different cohort to those who completed Version 1 of the questionnaire. They were able to select their reasons for volunteering from a list of seven options. These were based on seven common themes from the qualitative data analysis and included (1) exam practice, (2) interest in medical education, (3) financial incentive, (4) continuing professional development, (5) insight into own performance, (6) influence how GMC operates and (7) other.

Procedure

Doctors who volunteered to take a ToC between February 2011 and October 2012 were invited to participate. Once they had completed the knowledge test and OSCE, a facilitator distributed paper copies of the research questionnaire (see online supplementary appendix 1) and asked volunteers to complete it onsite before leaving. Doctors were briefed about the purpose of the study and how their data would be used. They were assured that the completion of the questionnaire was voluntary and would only take 5–10 min. It was explained that questionnaire responses would be identifiable in order to match their questionnaire responses with their demographic data that was given at registration point. However, they were assured that their individual questionnaire responses remained confidential and would be summarised in an anonymous and unattributable manner when disseminating the research.

Data analysis

The demographic data and reasons for volunteering are described using descriptive statistics. The demographics of the volunteer doctors were compared with those under investigation and those on the LRMP in 2011. Volunteers’ gender, ethnic background, qualification region, current grade and specialty were collected and stored in a database at registration. We accessed a GMC database to obtain the same demographics (except grade) for doctors under investigation due to concerns with their performance. We used a recent GMC report to obtain the gender, ethnicity and PMQ region of doctors on the LRMP in 2011.6 Reasons for volunteering to take a ToC were ranked in the questionnaire in order of importance, 1 being most important and 7 being least important. We calculated how frequently each reason was ranked as a first, second or third choice as well as the reason that was most commonly chosen as the primary motivation for volunteering. Data on how they had been recruited was recorded, that is, where they saw an advertisement for the GMC's pilot events and if applicable, who was the person that informed them of the events.

Ethical approval

We received written confirmation from University College London's Research Ethics Committee in October 2008 that the study was exempt from ethical approval. Doctors who participated explicitly consented to their anonymised data being used for research purposes.

Results

Demographics

Three hundred and one doctors completed the research questionnaire. Doctors who volunteered to pilot a ToC were more likely to be female, to be white and to have qualified in the UK. Whereas doctors under FtP investigation were mostly male, of white or Asian background and/or were International Medical Graduates (table 1). The LRMP in 2011 also consisted of a majority of male doctors, doctors of a white background and/or UK graduates (table 1). International Medical Graduates were particularly over-represented in the FtP investigations for performance concerns compared with the LRMP. Further, Asian doctors were over-represented in the FtP cohort and the reference group compared with the LRMP. These comparisons highlight that when compared with the wider medical community, the cohorts of volunteer and FtP doctors differ from each other according to gender, ethnicity and qualification region.

Table 1

Comparing demographics of volunteer doctors with FtP doctors and the total UK medical population

Variable	Levels	Doctors who volunteered between February 2011 and October 2012 (N=301)	Doctors under FtP investigation for performance concerns between 1996 and 2008 (N=822)	List of Registered Medical Practitioners in 2011 (N=245 903)
Gender	Male	138 (46%)	688 (84%)	141 369 (57%)
	Female	163 (54%)	132 (17%)	104 534 (43%)
Ethnicity	White	164 (55%)	324 (39%)	118 822 (48%)
	Black/Black British	10 (3%)	69 (8%)	6812 (3%)
	Asian/Asian British	91 (30%)	325 (40%)	46 664 (4%)
	Mixed	12 (4%)	8 (1%)	3643 (2%)
	Other ethnic groups	13 (4%)	83 (10%)	9002 (4%)
	Not stated (includes prefer not to say)	11 (3%)	2 (<1%)	60 960 (25%)
Primary Medical Qualification region	UK	240 (80%)	245 (30%)	155 264 (63%)
	EU country	12 (4%)	136 (17%)	24 031 (10%)
	Non-EU country	49 (16%)	439 (53%)	66 608 (27%)

EU, European Union; FtP, Fitness to Practise.

Doctors specialty and seniority

The clinical specialty that volunteer and FtP doctors were working in are summarised in table 2. The majority of doctors in both cohorts were working in general practice, although this specialty represented a larger majority among FtP doctors than volunteers (table 2). The majority of volunteer doctors were working in junior (43%) or middle grade specialty trainee positions (42%). Grades were not available for doctors under FtP investigation. However, the age differences between the two samples suggest that most FtP doctors were likely to have been working in senior positions at the time of investigation. The mean age of volunteer doctors was 32 whereas doctors under FtP investigation had a mean age of 52.

Table 2

Specialty of volunteer and FtP doctors

Doctors specialty	Volunteer doctors (N=301)	FtP doctors (N=822)
Acute care	57 (19%)	69 (8%)
General practice	68 (23%)	393 (48%)
Medicine	49 (16%)	68 (8%)
Psychiatry	20 (7%)	46 (6%)
Surgery	41 (14%)	152 (18%)
Other (paediatrics, radiology, pathology, unknown)	66 (22%)	94 (11%)

FtP, Fitness to Practise.

Reasons for volunteering to pilot a ToC

Insight into own performance was most frequently ranked within the top three most important reasons for volunteering to take a ToC (figure 1). However, a breakdown of the single most important reasons (figure 2) shows that exam practice was most common (33%), followed by financial incentive (19%) and an interest in medical education (18%). There were no significant differences by gender in terms of motivations for volunteering to take a ToC (χ2 (6)=4.20, p=0.65). There was a significant difference between doctors of different ethnic backgrounds and their primary reason for volunteering (χ2 (12)=22.38, p=0.03). White doctors were more likely to attend for exam practice or financial incentive. More of the non-white doctors had an interest in medical education and wished to contribute to the GMC's FtP process. The primary reason for volunteering differed significantly between UK-trained doctors and those who trained overseas (χ2 (12)=21.49, p=0.04). A greater number of UK-trained than non-UK trained doctors stated that exam practice or financial incentive or an interest in medical education was their primary reason for volunteering to take a ToC.

Figure 1

Frequency each reason was ranked as first, second or third choice. GMC, General Medical Council.

Figure 2

Primary reasons doctors gave for volunteering to pilot a Test of Competence (ToC). GMC, General Medical Council.

Recruitment to the GMC's ToC pilot events

We summarised how doctors became aware of the GMC's pilot events and, if applicable, who made them aware. Half of this cohort received an email about the events with an attached flyer, usually forwarded from a colleague who had previously volunteered (table 3). Word of mouth refers to those who contacted the events manager without accessing/receiving any formal communication about the GMC's pilot events. A quarter of participants did not disclose how they were recruited.

Table 3

How volunteer doctors became aware of the GMC's ToC pilot events

	Frequency
Mode of communication
Email	153 (50.8%)
Word of mouth	30 (10.00%)
Other (eg, BMJ, Facebook, e-Portfolio, leaflet)	27 (9.0%)
UCL Medical School website	9 (3.0%)
Not stated	82 (27.3%)
Who communicated
Colleague	88 (29.2%)
Friend	47 (15.6%)
Deanery	28 (9.3%)
Other (eg, specialist groups, relative, assessor)	24 (8.0%)
UCL	17 (5.6%)
Royal College	9 (3.0%)
Supervisor	8 (2.7%)
GMC	7 (2.3%)
Not stated	73 (24.3%)

GMC, General Medical Council; ToC, Test of Competence; UCL, University College London.

Discussion

The population of doctors who volunteered to take a ToC differed from those who took a test as part of their FtP investigation. Volunteer doctors were more likely to be white, female and/or UK qualified. They were mostly working in junior or middle grade positions, most often in general practice. The majority of doctors who were tested under FtP investigation were white or Asian men, and/or qualified overseas. They were substantially older than volunteer doctors with a large majority also working in general practice. The reference group of volunteers were also mainly working in general practice but differed by gender and ethnic background from that of the demographic profile of all doctors in the UK. The most common motivation given for volunteering was exam practice. The second most common reason was the financial incentive, followed by an interest in medical education. Most had been recruited via email advertisement that had been forwarded from a colleague.

The reference group is over-represented by young junior doctors and women compared with the doctors under FtP investigation. It is likely that their reasons for volunteering are associated with their demographics. Younger doctors working in junior positions use these events for exam practice in preparation for forthcoming postgraduate exams. Further, more women than men attended primarily for exam practice and this reason may explain the higher proportion of women volunteers. It may be linked to underconfidence as women perceive themselves and are perceived by others as less confident in clinical knowledge and skills.7 8 The financial incentive was also influential and is probably linked to the attraction of being paid to attend an event that can be used for exam practice. These findings are not surprising given the recent changes to doctors’ pay, study leave budget cuts and an increased interest in medical education research.9 Senior doctors may be discouraged from volunteering to take a ToC due to their busy schedules and/or lack of recent exam practice. The potential loss of face may also be off-putting in the event that they do not perform well on their test.

Most people heard about the GMC pilot events from colleagues and friends, usually via email. This was encouraging as it suggests that our requests for each cohort of volunteers to spread the word about the pilot events are an effective way of recruiting. Word of mouth has been highlighted as a particularly effective method of recruiting volunteers to research.10 11

Our findings raise the question that the reference group may not be a suitable comparison for doctors under FtP investigation. It is not necessarily a problem that most volunteer doctors are younger and more junior than doctors under FtP investigation. The GMC's ToC are designed to assess minimum competence, that is, the competence expected of an FY2 doctor. This means that a large number of junior doctors in the reference group are an appropriate comparison to those under FtP investigation. Senior doctors in theory should outperform their junior colleagues on the same test, and if this is not the case it suggests impairments in their clinical knowledge and/or practical skills. It is also important to highlight that the experts who are involved in making judgements and decisions about poorly performing doctors are aware of the limitations with the reference group data and take this into account. Doctors under investigation are not judged solely on their ToC performance. Rather their performance is viewed in the wider context of a comprehensive peer review that provides in-depth information about their actual clinical practice.

This study has identified groups of doctors that may be less likely to volunteer and provides us with better understanding of which recruitment strategies are effective. The results are limited because only a sample of the reference group participated in this study. Our data on doctors under FtP investigation is only until the year 2008 and it is likely that the demographics of this cohort have changed more recently. Further, the large percentage of doctors as recorded on the LRMP who chose not to declare their ethnicity means that we do not know the true ethnic spread of all UK doctors and how that compares with the reference group.

The GMC depends on doctors in good standing to voluntarily sit a ToC before the tests are implemented in FtP investigations. We aim to gather more information about what facilitates and hinders volunteering for these tests. Our findings will be used to trial new methods of recruitment that may help increase the representativeness of the reference group of volunteers. In particular we aim to increase participation from ethnic minorities, International Medical Graduates and senior doctors.

Our findings indicate that the reference group of volunteers mostly consists of women, of white ethnicity, of junior grade, working in general practice and who hold a Primary Medical Qualification from the UK.

The reference group differs from the population of doctors under Fitness to Practise (FtP) investigation by gender, ethnicity, seniority and qualification region.

Most doctors volunteer because they want exam practice in preparation for forthcoming postgraduate exams and are attracted by the financial incentive.

How do the demographics of the whole reference group compare with that of the List of Registered Medical Practitioners (LRMP) and doctors under FtP from 2008 onwards?

What facilitates or hinders volunteering to pilot a General Medical Council (GMC) Test of Competence (ToC)?

How can doctors who are currently under-represented in the reference group be encouraged to volunteer?

McManus I, Winder B, Gordon G. UK doctors' attitudes to the General Medical Council's performance procedures, 1997–99. Med Educ 2001;35:60–9.

Southgate L, Cox J, David T, et al. The assessment of poorly performing doctors: the development of the assessment programmes for the General Medical Council's Performance Procedures. Med Educ 2001;35(Suppl 1):2–8.

Sturrock A, Conlon L, Hatch D, et al. The General Medical Council's Tests of Competence—What's happening now? Clin Teach 2006;3:9–12.

Southgate L, Campbell M, Cox J, et al. The General Medical Council's Performance Procedures: the development and implementation of tests of competence with examples from general practice. Med Educ 2001;35(Suppl 1):20–8.

Southgate L, Cox J, David T, et al. The General Medical Council's Performance Procedures: peer review of performance in the workplace. Med Educ 2001;35(Suppl 1):9–19.