'10% of your medical students will cause 90% of your problems': a prospective correlational study.

OBJECTIVES: Our aim was to explore the relationship between medical student Conscientiousness Index scores and indicators of later clinical performance held in the UK Medical Education Database (UKMED). Objectives were to determine whether conscientiousness in first-year and second-year medical students predicts later performance in medical school and in early practice. Policy implications would permit targeted remediation where necessary or aid in selection.
DESIGN: A prospective correlational study.
SETTING: A single UK medical school and early years of practice, 2005-2018. PARTICIPANTS: The data were obtained from the UKMED on 858 students. Full outcome data was available for variable numbers of participants, as described in the text. MAIN OUTCOME MEASURES: Scores on the UK Foundation Programme Office's Situational Judgement Test (SJT) and Educational Performance Measure (EPM), the Prescribing Safety Assessment (PSA) and Annual Review of Competency Progression (ARCP) outcomes.
RESULTS: Linear regression analysis shows Conscientiousness Index scores significantly correlate with pregraduate and postgraduate performance variables: SJT scores (R=0.373, R2=0.139, B=0.066, p<0.001, n=539); PSA scores (R=0.249, R2=0.062, B=0.343, p<0.001, n=462); EPM decile scores for the first (lowest) decile are significantly lower than the remaining 90% (p=0.003, n=539), as are PSA scores (p<0.001, n=463), and ARCP year 2 scores (p=0.019, n=517). The OR that students in the first decile fail to achieve the optimum ARCP outcome is 1.6126 (CI: 1.1400 to 2.2809, p=0.0069, n=618).
CONCLUSIONS: Conscientiousness Index scores in years 1 and 2 of medical school have predictive value for later performance in knowledge, skills and clinical practice. This trait could be used either for selection or for targeted remediation to avoid potential problems in the future. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study was carried out using data on undergraduate students from a single medical school.

We have explored the impact of a single predictor variable—the underlying causative factor—on a number of dependent variables, and the data structure of the predictor variable is unlikely to be continuous.

The Educational Performance Measure decile ranking is calculated based on the assumption that all medical schools are equivalent, which we know not to be the case.

The Annual Review of Competency Progression data contains a very high proportion of outcome 1 candidates that reduces the discrimination.

Our measure of conscientiousness in routine tasks appear to be most valid as a predictor of professional outcomes in later academic and clinical practice at the lower end of the scale. Therefore, this method is most likely to be useful where there is a high applicant/placement ratio, such as during selection.

Introduction

In 2002, Wright and Tanner published an article in the BMJ indicating that students who failed to bring passport photographs as requested on induction were significantly more likely (48%, as opposed to 8% for those who brought a photograph) to fail second-year exams.1 This observation was greeted with wry amusement by many of those in close contact with medical students, who clearly recognised the general phenomenon corresponds with the folk wisdom in medical schools that ‘10% of students will cause 90% of your problems’.

In a rather more substantial study,2 Papadakis et al found that negative student evaluations by tutors predicted the likelihood of disciplinary action. However, they also found that written exam scores predicted the likelihood of later sanctions even though such sanctions are rarely directly related to skills or knowledge. Papadakis et al summarised this finding as ‘It’s good to be good, and it’s good to be smart’, though this seems to contradict common experience: we do not normally observe that virtue is directly related to intelligence. Nor is disciplinary censure normally simply related to lack of knowledge: rather, it seems to reflect much more complex underlying characteristics. We hypothesise that there is a common factor underlying both examination success and the probability of fitness to practice sanctions in later practice, namely, the trait of conscientiousness. Conscientiousness is one of the ‘Big 5’ personality factors,3 the others being openness to new experience, extraversion, agreeableness and neuroticism. The work psychology literature generally identifies conscientiousness as the biggest single predictor of work place performance.4

Between the years 2006 and 2014, we measured the conscientiousness in routine tasks of a number of cohorts of first-year and second-year UK medical students in a single UK medical school, as described in the Methods section. A ‘Conscientiousness Index’ (CI) score, based on many observations, was calculated for each student on this basis. We have previously shown that the CI correlates strongly with staff and student estimates of professionalism.5–8 However, the CI can now be related to data held in the UK Medical Education Database (UKMED), ‘a platform for collating data on the performance of UK medical students and trainee doctors across their education and future career’ (https://www.ukmed.ac.uk/), so that the subsequent performance of these students can be studied, and correlations between their earlier conscientiousness and their later performance on a number of measures can be explored.

Methods

Patient and public involvement

This was not a patient-related study; therefore, this research was done without patient involvement. This study involved collecting and collating data on medical students in a single medical school and relating it to later performance.

For our predictor variable, we calculated the CI for first-year and second-year undergraduate medical students.5 The Index included: having brought required ‘induction’ information (photographs, criminal records information and immunisation status), attendance at compulsory sessions (unless a good reason had been notified), submission of assignments on time, fulfilling essential administrative requirements (eg, attending base unit allocation meetings) and completion of course evaluations. One point was awarded for each positive activity fulfilled. Typically, well over 100 points could be awarded each year, but all results are recorded as percentages. Students were aware of the collection of the CI data. Typically, the CI distribution for a year is kurtotic, negatively skewed, with a long tail.

For outcome variables, we obtained anonymised data from the UKMED on:

The UK Foundation Programme Office (UKFPO) Situational Judgement Test (SJT) scores were used by the UKFPO9 in allocating graduating medical students to their foundation year 1 post. The SJT represents a 70-item selected-response test, which has predictive validity for post graduate performance.10 11 The content domains are coping with pressure, working effectively as part of a team, effective communication, problem solving and commitment to professionalism.12

The Educational Performance Measure (EPM) was also used by the UKFPO in allocating graduating medical students to their foundation year 1 post, in conjunction with the SJT. The EPM represents the decile each medical student is placed in, based on their academic performance over the first 4 years of their undergraduate medical programme.

Scores on the Prescribing Safety Assessment (PSA)13 are relative to the pass mark. The PSA is a 60-item written multi-format test on prescribing accuracy, required to be taken by all UK final-year medical students.

Annual Review of Competence Progression (ARCP) outcomes: these represent the considered judgement of a panel of experts on the readiness of trainee doctors to progress to the next level of training, on the basis of evidence provided by the trainee and other sources. A numeric score is used to describe the outcomes, as shown in table 1, for all the outcomes coded in our database extract.

Table 1

Annual Review of Competence Progression outcomes

Outcome	Meaning
1	Satisfactory progress. Competencies achieved as expected
2	May progress but requires specific targeted training to achieve certain competencies
3	Has not achieved competencies required to progress. Additional training required
4	Released from training with or without specific competencies
5	Incomplete evidence provided

Analysis

All statistical analyses were carried out securely within a ‘safe haven’ set up by UKMED, using SPSS V.25.

Since the relationship between CI scores and all of these outcomes is likely to be complex and possibly non-linear, we made no advance assumptions about the nature of this relationship. Instead, we inspected the data graphically prior to assessing what the nature of the relationships, if any, might be.

Results

As in a previous study,14 we observed that the CI is stable between years 1 and 2; analysis using a Pearson’s correlation test of the combined CI scores for 3 cohorts of students showed a high degree of correlation (p=0.001, with R=0.54), and we, therefore, used the average value of both years, so that observations were based on the maximum number of data points.

Our first observation was that the first decile of CI scorers appears markedly different from the other deciles. Figure 1 shows the spread of CI scores in each decile against the average score in that decile. One-way analysis of variance (ANOVA) indicates that the deciles do not all belong to the same group (F(9, 848)=935.66, p<0.001), and a post-hoc t-test reveals that the first decile differs from all other deciles (p<0.001, n=858).

Figure 1

The spread of CI scores in each decile against the average score in that decile. One-way analysis of variance (ANOVA) indicates that the deciles do not all belong to the same group (F(9, 848)=935.66, p<0.001), and a post-hoc t-test reveals that the first decile differs from all other deciles (p<0.001, n=858). CI, Conscientiousness Index.

This corresponds to a more general observation that in measurements of undergraduate student performance (for instance the UKFPO SJT), the distribution is kurtotic and negatively skewed, but with a long tail of low scorers.

Due to this initial observation (that the deciles do not all belong to the same group and that the first decile differs from all other deciles, the CI is also kurtotic and negatively skewed), then methods such as factor analysis were considered inappropriate.

Relationship of the CI with UKFPO SJT

Figure 2 shows the relationship between CI scores and the UKFPO SJT. Linear regression analysis shows a relationship between these two parameters (R=0.373, R2=0.139, B=0.066, p<0.001, n=539). T-test showed a statistically significant difference between SJT scores of students scoring in the first decile of the CI and the other nine deciles, (p<0.001).

Figure 2

Scatter plot of Conscientiousness Index scores against Foundation Programme SJT scores. Linear regression analysis shows a statistically significant positive relationship (R=0.373, R2=0.139, B=0.066, p<0.001, n=539). CI_AVG, average Conscientiousness Index score over years 1 and 2 of medical school; FP_SJT, Foundation Programme Situational Judgement Test.

The Educational Performance Measure (EPM)

Similarly, for the EPM, the difference between the first decile and the other nine deciles by t-test was calculated (p=0.003, n=539) (see figure 3).

Figure 3

The EPM decile scores for those in first decile of the CI, and the other nine deciles. Analysis by t-test shows the first decile is significantly different to the rest (p=0.003, n=539). CI, Conscientiousness Index; EPM, Educational Performance Measure.

It should be noted that the EPM decile ranking is calculated based on the assumption that all medical schools are equivalent, which we know not to be the case. This will be a significant contribution to error on the part of the EPM.

The Prescribing Safety Assessment (PSA)

Figure 4 shows the scatter plot for CI scores versus PSA scores relative to the pass mark. Linear regression analysis shows R=0.249, R2=0.062, B=0.343, p<0.001 and n=462. T-test showed a statistically significant difference between PSA scores of students scoring in the first decile of the CI and the other nine deciles (p<0.001, n=463).

Figure 4

Scatter plot of average CI scores in years 1 and 2 of medical school against PSA scores relative to the pass mark. Linear regression analysis shows a statistically significant positive relationship (R=0.249, R2=0.062, B=0.343, p<0.001, n=462). CI, Conscientiousness Index; PSA, Prescribing Safety Assessment.

The Annual Review of Competency Progression (ARCP)

ARCP scores are difficult to interpret.15 However, Tiffin et al16 demonstrated that the Professional and Linguistic Assessments Board test (PLAB) scores correlate with subsequent ARCP scores, and that the relationship is at least ordinal. We compared the number of candidates with an ARCP score of 1 (which indicates that they can progress to the subsequent year of training) in the first decile with all other categories. First decile candidates had a higher average score (indicating more outcomes other than 1), as shown by t-test in year 2 of training (p=0.019, n=517), but not in year 1.

Since the probability that a student in the first decile is likely to fail to achieve the optimum ARCP outcome is of key importance to the predictive validity of the CI, we calculated the OR for this outcome. Calculation of the OR in these circumstances is usual in studies of predictive validity.16 The OR that students in the first decile of the CI score failed to achieve the optimum ARCP outcome was 1.6126 (CI: 1.1400 to 2.2809, p=0.0069, n=618).

Discussion

We found that there is a relationship between conscientiousness as measured in a single UK medical school by the CI in an objective and scalar manner, and subsequent performance as measured by outcomes such as exam scores and Objective Structured Clinical Examination (OSCE) scores (contained in the calculation of the EPM), SJT performance and later clinical practice, including professionalism as measured by ARCP. The results show that those scoring in the lowest decile are more likely to perform low later in their education and in clinical practice. However, these results are tentative and further research is required to fully establish the nature of the relationships.

Although use of ARCP data as an outcome measure has been challenged,17 and it certainly contains a very high proportion of outcome 1 candidates that reduces the discrimination (and, therefore, may be seen as a limitation of this study), the fact that there is a relationship between the CI and ARCP outcomes (in the same way as a relationship between assessment data and ARCP was observed by Tiffin et al16) indicates that ARCP outcomes are non-random. We, therefore, consider that continued use of ARCP outcomes is justifiable.

The results show predictive validity for low performance later in education and as junior doctors but do not extend to later events such as sanctions by the General Medical Council (GMC). A limitation of this study is that it was necessarily carried out in a single medical school; however, we look forward to other colleagues generalising these approaches. Indeed, future studies on a larger data set will be able to indicate if the CI predicts Fitness to Practice events in the UK, in the way that Papadakis et al2 observed for exam scores.

A further limitation of this study is that it is possible that students were aware that a conscientiousness measure was being applied, and as a result of this, responded by changing their behaviour, however, we did not find any evidence of this.

Conclusion and implications for clinicians and policymakers

We have already demonstrated that the CI predicts staff ratings of student professionalism and the likelihood of them receiving an adverse ‘critical incident’ report.5 We have also demonstrated that the CI predicts estimates of professionalism by fellow students,6 that the CI predicts scores on knowledge tests18 and student performance in clinical settings.7 It is also a predictor of SJT performance, which is itself a predictor of later clinical performance.10 Here, we extend these findings to a wider range of settings, including, for the first time, postgraduate performance.

Why should conscientiousness as a student be predictive of later professionalism in clinical practice, both as senior students and as junior doctors? We postulate that this is through behaviour patterns such as good note and record keeping, good hand overs, following up patients, keeping up to date with developments and so on. Measurement of conscientiousness in early years will then identify candidates for targeted remediation, and, if this fails, may in the ultimate case be used as a deselection tool.