Objective structured practical examination as a tool for the formative assessment of practical skills of undergraduate students in pharmacology.

INTRODUCTION: Assessment for practical skills in medical education needs improvement from subjective methods to objective ones. An Objective Structured Practical Examination (OSPE) has been considered as one such method. This study is an attempt to evaluate the feasibility of using OSPE as a tool for the formative assessment of undergraduate medical education in pharmacology.
MATERIALS AND METHODS: Students of second year MBBS, at the end of the first term, were assessed by both the conventional practical examination and the Objective Structured Practical Examination (OSPE). A five-station OSPE was conducted one week after the conventional examination. The scores obtained in both were compared and a Bland Altman plot was also used for comparison of the two methods. Perceptions of students regarding the new method were obtained using a questionnaire.
RESULTS: There was no significant difference in the mean scores between the two methods (P = 0.44) using the unpaired t test. The Bland Altman plot comparing the CPE (conventional practical examination) with the OSPE showed that 96% of the differences in the scores between OSPE and CPE were within the acceptable limit of 1.96 SD. Regarding the students' perceptions of OSPE compared to CPE, 73% responded that OSPE could partially or completely replace CPE. OSPE was judged as an objective and unbiased test as compared to CPE, by 66.4% of the students.
CONCLUSION: Use of OSPE is feasible for formative assessment in the undergraduate pharmacology curriculum.

INTRODUCTION

Assessment of students in medicine has always remained a topic of debate. Student assessment is often described as ‘the tail that wags the dog’ of medical education. It is seen as the single strongest determinant of what students actually learn (as opposed to what they are taught), and is considered to be uniquely powerful as a tool for manipulating the whole education process.[1] There are continuous attempts to make assessment more objective and reliable rather than subjective. Traditional, age-old methods like essay/essay type questions, which suffer from lack of objectivity, are giving way to newer objective methods of assessment in the form of multiple choice questions, short answer questions, and such other tools, for assessment of cognitive domain.[2] As far as skills assessment is concerned the conventional methods are not only subjective in nature, but also lack scope for direct observation of the performance of skills by the assessor. Moreover the coverage of contents may be limited. Hence, attempts have been made to introduce methods that can overcome the above-mentioned limitations.

One step in this direction is the Objective Structured Clinical Examination (OSCE) described in 1975, by Harden et al., at the Dundee University, for assessment in clinical subjects, which has been a useful tool in this regard.[3] The OSCE had been introduced as a reliable approach to assess the basic clinical skills. It is a flexible test format based on a circuit of ‘stations’. At each station, a specific leaning objective is tested.[4] The OSCE has been widely used for formative and summative assessment in various medical disciplines worldwide, including the non-clinical disciplines.[5] For assessment in preclinical and paraclinical subjects, a modified version of the OSCE, the objective structured practical examination (OSPE) has been introduced.[6] In India, the use of OSPE for assessment of pharmacology skills has been reported from some institutes.[78]

However, a majority of institutes still follow the conventional method of assessment. In our institute also, the assessment of practical skills is carried out by conventional methods, as directed by the University. Hence, this study was planned to evaluate OSPE as a tool for term ending assessment of practical skills in the undergraduate pharmacology curriculum. The students’ perceptions regarding this new tool of assessment were also assessed.

Our study was aimed at evaluating OSPE as a method of formative assessment of practical skills in pharmacology at the undergraduate medical curriculum, so as to find out the feasibility and acceptability of this method of examination. The study also compares OSPE with conventional assessment.

Objective of the study

To plan and implement OSPE (Objective Structured Practical Examination) as a tool of internal assessment in the undergraduate pharmacology curriculum.

To compare the conventional practical examination (CPE) with OSPE (Objective Structured Practical Examination).

To obtain the students’ opinion regarding OSPE as a tool of assessment.

MATERIALS AND METHODS

The study was carried out in February 2010, at our institute, after obtaining the Institutional Ethics Committee approval. The second year MBBS curriculum is divided into three semesters (third, fourth, and fifth semester) as suggested by the Medical Council of India. At the end of each semester, the students are assessed in theory and practical. This study was undertaken at the end of the third semester (First term of Second MBBS) for a class of 137 students.

The conventional practical examination (CPE) in pharmacology (40 marks) consists of two components – Practical exercises (25 marks) and viva voce related to the theory topics covered during the term (15 marks). The practical exercises consist of questions in the form of short exercises related to prescription and its components, dosage forms, dose calculations, sources of drug information, drug selection for disease conditions, as well as, identification of adverse drug reactions and drug interactions.

As the OSPE was being conducted for the first time, the students were notified three weeks in advance regarding the plan for conducting the term ending practical assessment – by both the CPE and OSPE – to be held with an interval of one week between the two. Detailed instructions regarding the OSPE, number of stations, and marks for each station and conduct of examination were displayed. The OSPE was planned for 25 marks assigned for practical examination that is, excluding the viva voce of 15 marks. The OSPE was conducted in two batches, on two consecutive days, using different sets of questions for each day. To reduce the time for examination, three identical sets were planned. The OSPE examination of 25 marks consisted of five stations of five marks each and one rest station. The time allotted at each station was five minutes. Out of the five stations, one was the procedure station. Each station was designed along with the checklist, by the authors. The stations were selected to represent the learning objectives from the first term of the Pharmacology curriculum.

Care was taken to have items that were similar in terms of objectives to be assessed as also the difficulty level, for both the sets of stations. [Table 1] For the procedure station, each point on the check list was scored according to the binary system, that is, the ‘Yes/No’ scale, by the observer, and marks were given accordingly [Table 2]. The Pharmacology faculty acted as the observers at the procedure station along with the check list. Students required 50% marks to pass both types of tests.

Table 1

OSPE* stations grouped by the domain tested

Table 2

Sample OSPE Station with checklist

After the examination, feedback was obtained from the students with the help of a pre-validated questionnaire, in the first theory class that followed. Questions pertaining to the students’ perceptions regarding OSPE compared to CPE, the difficulties they faced, and their opinions regarding inclusion of OSPE as an assessment method in pharmacology, were included.

Statistical analysis

The data was analyzed using Microsoft excel 2007. The unpaired t-test was used to compare the marks obtained in CPE and OSPE. As the OSPE was held on two consecutive days, with different exercises, the marks obtained by the two batches were also compared using the unpaired t-test. The value of P < 0.05 was considered statistically significant. The Bland Altman plot was used for comparison of the two methods.

RESULTS

Out of 137 students in the second-year MBBS class of first term, 134 students took both the tests. – CPE and OSPE. The mean scores out of 25 were 12.82 ± 4.18 (Range: 3.5 to 21.5) and 13.16 ± 2.99 (Range: 6.75 to 19.75) for CPE and OSPE, respectively. There was no significant difference in the mean scores between the CPE and OSPE (P = 0.44). The mean score obtained in OSPE on day one and day two (13.39 ± 3.18 and 12.94 ± 2.8) showed no significant difference (P = 0.77).

The Bland Altman plot comparing the CPE with the OSPE showed that ~96% of the differences in the scores between OSPE and CPE were within the acceptable limit of 1.96 SD [Figure 1]. Only 2.9% of the students scored above the anticipated difference in the score, and the rest scored within the anticipated difference in the scores of OSPE and CPE. Thus, both the methods were comparable.

Figure 1

Bland Altman plot comparing conventional practical examination with objective structured practical examination

Students’ Perceptions of objective structured Practical examination compared to conventional Practical examination

Out of 134 students who took both the tests, 128 responded to the feedback questionnaire. In response to the question related to difficulty level, 63 (49.2%) rated OSPE easier than CPE, 43 (33.6%) stated that it was the same as CPE, 13 (10.2%) found OSPE more difficult, while eight (6.2%) were uncertain. Responding to the question about time spent for the examination, 125 (97.7%) felt that OSPE consumed less time as compared to CPE. Regarding coverage of the course, 66 (51.6%) felt that it was the same as for the conventional, 33 (25.8%) felt that OSPE covered less of the course, and 15 (11.7%) felt OSPE covered a wider course than CPE. When responding to the question regarding objectivity, 85 (66.4%) felt OSPE was objective and unbiased as compared to CPE, 18 (14%) felt it was not unbiased, while 20 (15.6%) were uncertain [Table 3]. Seventy three percent felt that it could partially or completely replace the conventional practical examination.

Table 3

Responses of students about OSPE as compared to CPE

Regarding the difficulties faced by the students, 18 students reported no difficulty, while 55 felt difficulty due to shortage of time at some stations, especially at the procedure station (specified by five students). Other reasons for difficulty included lack of proper information (29), poor organization (8), anxiety (21), and constant vigilance (5). Students’ opinions regarding the inclusion of OSPE in practical assessment, is depicted in Figure 2.

Figure 2

Students’ opinion regarding inclusion of objective structured practical examination in practical assessment

DISCUSSION

The Objective Structured Practical Examination has been advocated for the practical assessment of preclinical and paraclinical subjects, including Pharmacology. An attempt was made to test the feasibility and acceptability of implementing this method in the internal assessment by comparing it with CPE, and also by obtaining the students’ opinion, with the help of a feedback questionnaire.

Before this study was planned, two pilot studies were carried out in 2009, at the end of the first and second term of the second professional year of the MBBS program, to identify the possible issues related to the planning of OSPE. In the present study, to ensure that all the students took the OSPE, they were given the incentive of including the higher score of the two types of tests in their internal marks.

The results suggest that OSPE and CPE are in agreement as seen in the Bland Altman plot of the two methods, which shows that 96% of the values lie within the limits of the mean of ±1.96 SD. Thus OSPE can replace CPE in the formative assessment. These findings differ from the previous studies, which show a significant difference between the CPE and OSPE scores,[78] The reason for this is that our conventional examination already has 50-60% marks devoted to objective exercises, which are calculations and short answer questions with precise answers

As far as the students’ perceptions with regard to the difficulty level are concerned, only about 10% of the students perceived OSPE as more difficult than CPE, suggesting that it would be acceptable to a majority of students in case it replaces CPE. The time for conducting OSPE was also less, as perceived by a majority (95%) of the students. For the faculty also, the time for conducting OSPE was reduced compared to CPE. About 63% of the students felt that OSPE provided optimum coverage of the course. On the basis of the student responses to a questionnaire, it was clear that the students responded positively (66.4%) to the OSPE format and it was perceived to be more fair and objective than the conventional examination. Previous studies on the attitudes of students to OSPE revealed similar findings.[78]

The strength of the study was that all the students were exposed to both the types of examinations, CPE and OSPE, and were in position to give their opinion. We could also assess a large group of students with OSPE in a shorter time than with CPE.

Communication is one of the most important components of physician – patient management skills. The OSCE have been used extensively to assess communication skills.[9] In our study there was no station to evaluate communication skills. This was because of the limited course content of the first term. However, in the pilot study, which was undertaken in 2009, with half the batch, at the end of the second term, a station meant to evaluate the communication domain was included, in the form of giving verbal instructions for the use of oral contraceptive pills. Similar stations can be designed for instructions with regard to the use of glucocorticoids, insulin injections, oral hypoglycemic drugs, antihypertensive drugs, and so on.

We conducted OSPE with five stations, which could affect the reliability of the test.[3] As OSPE was planned for formative assessment, we planned only five stations of five minutes each. It is believed that with a single experience, with a limited number of stations, it is not possible to judge the difficulties and constraints of using OSPE as a method to assess the complete course on a regular basis[10] However, as we had conducted pilot studies earlier, we were aware of the possible difficulties. Moreover, a limited number of stations may not always be a constraint for formative assessment, as reported by Mathews et al.,[11] who found micro OSCE with two stations satisfactory, as a formative method of assessment in Pediatrics. This study definitely confirmed the feasibility and students’ acceptability of OSPE in evaluating the pharmacology skills in the undergraduate medical curriculum.

CONCLUSION

From the results of our study it can be concluded that the use of OSPE is feasible and acceptable to the students for the internal assessment of practical skills in undergraduate training in pharmacology.