An Assessment of the Correlation between Tests of Clinical Competence and Tests of Cognitive Knowledge amongst Nigerian Resident Doctors in Surgery.

Background: Assessment of clinical competence involves the assessment of cognition and assessment of clinical performance (behaviour in practice). The limitations of the traditional long case examination (LCE) in the assessment of clinical performance led to its replacement with the objective structured clinical examination (OSCE) by many institutions. Aims: To determine and compare the abilities of the OSCE and LCE to predict candidates' performance in the tests of cognitive knowledge in the fellowship examination of the National Postgraduate Medical College of Nigeria in the Faculty of Surgery. Materials and
Methods: The results of the OSCE, LCE, written papers, picture tests (PTs), vivas, and the total clinical score (TCS) of surgical residents who took part in the fellowship examination over six consecutive examination periods were compared by using the Pearson's correlation coefficient. A P-value less than.01 was considered as significant.
Results: The OSCE had a weak but statistically significant positive correlation (.175) with the LCE. Both the OSCE and LCE had similar correlations with the total written papers (TWP) and PTs. The viva had a higher correlation with the OSCE than the LCE. The TCS when compared with either the OSCE or LCE alone had a higher correlation with most of the tests of cognitive knowledge.
Conclusion: Neither the OSCE nor the LCE showed any superiority over the other in terms of the ability to predict performance in the tests of cognition. The TCS appears superior to either the OSCE or the LCE as a predictor of the candidates' overall knowledge of surgery. Therefore, both the OSCE and the LCE should be retained as part of the examination. Copyright:

Introduction

Assessment of clinical competence involves the assessment of cognition and assessment of performance (behaviour in practice).[1] Presently, there appears to be no single test that can adequately assess all aspects of clinical competence. Assessing cognition deals with knowledge and its application, whereas the assessment of behaviour deals with the assessment, under controlled conditions, of competence in practice. The traditional written papers including the oral viva[12] assess cognitive knowledge, whereas the clinical examinations assess competence or performance.

Although there is little doubt that the tasks undertaken in a traditional long case examination (LCE) resemble real-life situations, its limitations are well known, especially its reported low reliability.[34] The objective structured clinical examination (OSCE) was introduced with the aim of overcoming these limitations. Since it was initiated by Harden, the OSCE has gained worldwide popularity with many institutions and examination bodies preferring it to and abandoning the LCE. Although examination bodies in North America and most of Europe have completely abandoned the LCE, there appears to be some reluctance in the West African subregion to do the same.[5] Studies on OSCE in our environment have been sparse and have mainly looked at the perception of students towards the examination.[67] Although different studies worldwide have tried to assess the correlation between the tests of clinical performance (OSCE or LCE) and other tests of clinical competence, most of these studies have had one major limitation—inability to compare the OSCE and LCE in the same set of candidates at the same examination sitting—because most training institutions and examination bodies simply abandoned the LCE and replaced it with the OSCE.[7891011] The Faculty of Surgery of the National Postgraduate Medical College of Nigeria was one of the earliest faculties in the college to fully introduce the OSCE into its Part I (intermediate) fellowship examinations in 2010. The faculty however retained the LCE while the OSCE replaced the short cases and one of the oral viva examinations, thereby giving the authors a unique opportunity to assess the OSCE and LCE in the same candidates in the same examination.

The objective of this study was to assess the ability of the candidates’ scores in the tests of clinical performance to predict their performance in the tests of cognitive knowledge by determining the correlation between the clinical examination (OSCE and LCE) scores and the scores of the different tests of cognitive knowledge in surgical residents who sat for the Part 1 examinations in surgery.

Materials and Methods

This was a retrospective study that involved surgical resident doctors who fully participated in the Part I examination of the National Postgraduate Medical College of Niger between November 2013 and May 2016 (six examinations). At each examination, there were 20 OSCE stations consisting of two history stations, three clinical examination stations, and at least one performance station where candidates were tasked with demonstrating a specific skill on a mannequin. Each history, clinical examination, and performance station was manned by a minimum of two examiners using a checklist to score the candidates, and was followed by another station where the candidates were required to answer open-ended questions related to the previous clinical activity. Each OSCE station lasted 5 min. The other OSCE stations included questions on surgical instruments and patient management problems. For the LCE, each candidate was assessed by a minimum of three examiners. Each candidate was given 30 min to clerk a real-life patient and 15 min to be assessed by the examiners. The non-clinical examination (tests of cognitive knowledge) included one paper of essays and multiple-choice questions (MCQs) on principles of surgery, a second paper of essays and MCQs on operative surgery, a 20-min oral viva, and a 30-min picture test (PT). The oral viva was divided into equal segments that consisted of operative surgery and pathology (10 min) and principles of surgery (10 min). Every candidate was expected to answer questions on at least two topics in each segment. In order to reduce bias on the choice of the topics, each candidate was required to pick the topic via a ballot system. In the PTs, the candidates were showed different pictures such as clinical lesions and radiographs followed by related open-ended questions. The PT spanned 30 min and involved 20 different pictures with the candidate having 90 s to review a picture and answer the accompanying questions.

The LCE, written papers, and oral viva were marked over 20 marks, each using a close marking system with the implications that the highest possible mark was 12, whereas the lowest was 8. The maximum marks for the OSCE and PT were 100 each.

After a formal application was made to the college registrar, the authors were granted access to the hard copies of the spreadsheets containing the marks of all the candidates who sat for the examinations. Relevant data were collected from the sheets.

The SPSS version 20 software was used in data analysis. The correlation between the OSCE, LCE, and the other different forms of the examination were compared using the Pearson’s correlation coefficient. A P-value less than.01 was considered as significant.

Results

A total of 420 candidates sat for the examination (55 candidates in 2013, 147 in 2014, 144 in 2015, and 74 in 2016), but two candidates who had some of their marks missing were excluded from the final analysis. Table 1 shows the correlation coefficients between OSCE, LCE, and the other forms of assessment. The OSCE and LCE had a correlation of.175 (P <.001) between them [Figure 1]. When compared with the LCE, the OSCE had a higher correlation with the written papers on surgical principles, whereas the LCE had a higher correlation with the operative surgery. Both the OSCE and LCE had similar correlations with the total written papers (TWP) and the PTs. The correlation between the oral viva and the OSCE was three times its correlation with the LCE. The total clinical score (TCS)—the sum of the marks in the LCE and OSCE—when compared with either the OSCE or LCE alone had a higher correlation with the principles paper, the TWP, and the PTs. The OSCE had a higher correlation than the TCS with the oral viva.

Table 1

Pearson’s correlation coefficients between the different forms of examination

	POS	OPS	TWP	PT	OV	LCE	OSCE
LCE
r	*.194	*.211	*.242	*.183	*.137	-	*.175
P-value	<.001	<.001	<.001	<.001	.005	-	<.001
OSCE
r	*.306	.087	.227	*.193	*.392	*.175
P-value	<.001	.075	<.001	<.001	<.001	<.001
TCS
r	*.330	*.280	*.299	*.233	*.314
P-value	<.001	<.001	<.001	<.001	<.001

POS: principles of surgery, OPS: operating surgery, TWP: total written papers, PT: picture test, OV: oral viva, LCE: long case examination, OSCE: objective structured clinical examination, TCS: total clinical score, r: Pearson’s correlation coefficient. *Statistically significant

Figure 1

Scatter plot of the OSCE and LCE scores

Discussion

The OSCE had a weak but statistically significant positive correlation (.175) with the LCE. Other studies have also found a weak (<.5) correlation between the two forms of clinical examination. Tijani et al. found a weak correlation of.37 between performance in the final year OSCE and the in-course assessment LCE in 612 undergraduate students in surgery.[12] Bakhsh et al. also found a correlation of.40 in 904 undergraduates in internal medicine.[13]

The reasons for the relatively lower correlation found in our study are unclear. The fact that these other studies were on undergraduate students, whereas the present study was on surgical residents with at least 3 years of postgraduate training may be contributory.

Moreover, in each of those two studies, even though the same set of students were involved, the interval between the LCE and the OSCE was between 12 and 24 months.[1213] The implication of this is that although the physical identity of each student would remain unchanged, it would be reasonable to assume that the level of knowledge and clinical competence of the individual student would have significantly changed during this period. Wallenstein et al. also found a strong correlation (.6) between the performance in the OSCE of new intake residents and their average performance in other tests of clinical competence 18 months later.[14] It is also not clear whether the system of marking in the LCE affected the strength of correlation with the OSCE. With close marking in LCE, the minimum score was 8 and the maximum 12, which meant that only five different marks were possible. As a result, most of the points on the scatter diagram overlapped as shown in Figure 1. It may be conjectured that if the LCE scoring were to be opened up, a clearer picture of the strength of correlation between the two could be obtained.

Both the OSCE and LCE had positive but weak correlations with all the other components of the examination. These correlations were however statistically significant. Kirton and Kravitz also found an average correlation of.6 between the OSCE and the written papers in undergraduate students, whereas Al Rushood and Al-Eisa also found a correlation of.44 in undergraduate students in paediatrics.[1516] Komasawa et al. however found no statistically significant relationship between the OSCE and TWP.[17] In the present study, the OSCE had a correlation of about.23 with the TWP, which was statistically significant. The OSCE (when compared with the LCE) had a stronger correlation with the written papers on surgical principles and the oral viva, whereas the LCE had a better correlation with the operative surgery component of the written papers. Both the OSCE and LCE had similar correlations with the TWP and PTs. Our study was therefore unable to demonstrate any superiority of the OSCE over the LCE or vice versa in terms of their correlation with the candidates’ performance in the non-clinical forms of assessment. These findings were similar to the findings of Johnson and Reynard who did not find any consistent correlation between the OSCE and other forms of (non-clinical) examination in their study on undergraduate students in emergency medicine.[18] However, they were in contrast with a study by Tijani et al. that reported a consistent superiority of the OSCE over the LCE in terms of its correlation with other forms of non-clinical assessment in the undergraduate examinations in surgery.[12]

The literature is sparse on studies comparing OSCE and LCE in the same set of candidates, as almost all undergraduate and postgraduate examination bodies that adopted the OSCE simply abandoned the LCE at the same time. Indeed LCE can be said to have a face validity and authenticity as it is difficult to contest the logical assumption that the clinical method rehearsed in the course of a long case is the way that a good clinician should practise. It is also most likely true that an experienced examiner could by active questioning during an LCE differentiate between a good and a bad candidate. The decision to replace the LCE with OSCE has however been critically challenged.[19] Wass et al. in the UK reported a study of 214 candidates, who all had two LCEs alongside a 20-station OCSE.[20] Their study reported reliability of.84 and.72 for LCE and OSCE. They concluded that given the same amount of time, the LCEs are in terms of reliability no worse and no better than OSCEs in assessing clinical competence. In the present study, the TCS (a combination of the scores of the LC and OSCE) when compared with LCE or OSCE alone had a stronger positive correlation with all the other components of the Part 1 examinations except the oral viva, which had a slightly higher correlation with the OSCE. The TCS therefore appears to be a better predictor of the candidates’ performance in the non-clinical tests, thereby making it a good predictor of the candidates’ overall knowledge of surgery.

The study had some limitations. First, it was retrospective. Second, it did not take into consideration variables like previous experience with the examination because this may have an effect on the performance of the candidates as some of them were repeating the examination.

In conclusion, none of the methods of assessing clinical competence (OSCE and LCE) was able to independently show any consistent superiority over the other in terms of correlation with the candidates’ performance in the tests of cognition. It is therefore recommended that the TCS should be retained as part of the summative assessment of resident doctors in surgery.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.