Assessment for learning with Objectively Structured Practical Examination in Biochemistry.

CONTEXT: Despite a radical shift in assessment methodologies over the last decade, the majority of medical colleges still follow the Traditional Practical Examination (TPE). TPE raises concerns about examiner variability, standardization, and uniformity of assessment. To address these issues and in line with the notion of assessments as motivating what and how students learn, Objectively Structured Practical Examination (OSPE) was introduced, as an assessment modality. Despite its usefulness, awareness and motivation to use the same, still needs to be probed. AIMS: To implement OSPE in the assessment of practical skills in biochemistry, and to know student and faculty perspectives regarding OSPE. SETTINGS AND
DESIGN: OSPE was introduced at the stage of formative assessment of practical skills, for 94 year one MBBS students. SUBJECTS AND METHODS: Students were divided into two groups; the first group was evaluated by the traditional method and the second by OSPE. Students were crossed over on a second examination. The mean score obtained by both the methods was compared statistically. Students and faculty perspectives regarding OSPE were obtained by a questionnaire. Student performance was compared using "Bland-Altman technique," and Student's t-test.
RESULTS: The mean scores of students was found to be significantly higher (P < 0.0001) when assessed with OSPE as compared to TPE. Number of students achieving >70% marks was also significantly higher with OSPE. Validity was supported by a significant correlation coefficient of comparison of marks by the two methods. Feedback from students and faculty indicated that they endorsed OSPE.
CONCLUSIONS: This evaluation demonstrated the need for a structured approach to assessment. Going in line with the notion that assessment drives learning, introducing OSPE would help tailoring teaching-learning to optimize student satisfaction and learning.

INTRODUCTION

Competency in performing bedside laboratory tests is an important skill to be effectively carried out by the medical graduates in the healthcare system. The present system of practical examination in biochemistry is not structured and standardized to evaluate the ability of students to perform and interpret these tests objectively. This results in inability of future graduates to utilize these simple laboratory tests in clinical practice in resource poor settings.

As assessment drives learning, developing better assessment tools will ultimately improve learning and help achieve the objectives of medical education.

The Traditional Practical Examination (TPE) method of assessment of practical skills raises concerns about examiner variability, standardization, and uniformity of assessment. In the traditional method, the practical skills are not directly observed but are assessed based on questions asked at the end of the session. There is thus, a need of a structured approach for assessment of practical skills to provide strategies that faculty can use to enhance skill performance and increase training efficiency.

Keeping the above in mind, undergraduate medical education is currently undergoing extensive re-evaluation with core educational objectives being defined. It was felt that a single examination did not fulfill all the functions of assessment such as assessing knowledge, comprehension, psychomotor skills, and feedback.[1] Newer assessment methods have consequently been devised to address these problems, one of which is Objectively Structured Practical Examination (OSPE).

OSPE is a practical exam system where there is a series of stations at which students work through tasks designed to test various skills. It has been found to be objective, valid, and reliable and eliminates examiner bias. The most important aspect of this method of evaluation is that it has scope for improving the teaching-learning process in total, through feedback. It provides an opportunity to test a student's ability to integrate knowledge, clinical and practical skills that are a must for any student aspiring to become a successful clinician. It has been shown that such an assessment method can influence student learning.[2] Assessing the different components like performing laboratory tests, analyzing and interpreting laboratory data would drive the students to learn the same.

The aim of this study was to evaluate OSPE as an assessment tool compared to TPE, in the assessment of practical skills in biochemistry. The study also aimed to investigate the faculty and student perspectives regarding OSPE in an attempt to provide scope for refining the method of assessment in the department and making it more objective, reliable, and valid.

SUBJECTS AND METHODS

The study was undertaken as a part of the FAIMER fellowship program after approval from the Institutional Research and Ethics Committee.

For the practical examination as a part of the curriculum for biochemistry, each student has to perform the qualitative analysis of urine to test the abnormal constituents. This exercise is a mandatory part of the curriculum of biochemistry in medical colleges and was therefore proposed to be taken up as an exercise to compare the assessment by the two methods that is, traditional method and OSPE.

OSPE was introduced at the stage of formative assessment to test the practical skills in the analysis of abnormal constituents of the urine.

The objectives of practical examination in biochemistry are to assess:

Acquisition of appropriate practical skills

Reading of certain basic tests

Analysis and interpretation of test results and its clinical correlation.

OSPE stations for analysis of abnormal constituents of urine were designed to meet all the above objectives. Of total 18, 6 were performance stations to test practical skills, 3 observation stations, 8 response stations to test critical thinking and knowledge and 1 relax station.

The OSPE stations designed were peer reviewed, expert validated and pilot tested before administration. All the faculty and staff involved in the designing and conducting of the OSPE were trained. Peer agreed checklists for procedure stations and structured questions with key answers for the response stations were also prepared. Simultaneously questionnaires were prepared, peer reviewed, and expert validated for student and faculty feedback.

Since OSPE was being introduced for the 1st time, students were oriented toward it in advance. Ninety-four students from a batch of 100 students participated in this study. The students were divided into two groups of 47 students each. The first group was evaluated by the TPE, and the second group by the OSPE. Students were crossed over on a second examination. Each group undertook the same examination, and precautions were taken to ensure that students were unable to communicate during the examination. At the end of the day, feedback was given to the students by the observers regarding their performance at the procedure stations.

Students and faculty perspectives regarding usefulness, relevance, timing, and structure of OSPE were obtained by asking them to respond to a Likert scale questionnaire.

The mean score obtained by both the methods was compared statistically using Student's t-test. Student performance by the two methods was compared using “Bland–Altman technique.”

RESULTS

A total of 94 year one MBBS students participated in the study, where practical skills in the analysis of abnormal constituents of urine were assessed by both OSPE and TPE.

Analysis of mean scores obtained by students with the two methods suggested that students scored significantly higher marks when assessed with OSPE (mean score 24.38 ± 2.28) as compared to TPE (mean score 21.98 ± 2.91), the difference was found to be statistically significant (P < 0.001). The number of students achieving >70% marks was also significantly higher with OSPE. It was found that only one student got <50% marks when assessed with OSPE as compared to eight students with the traditional method as shown in Figure 1.

Figure 1

Frequency distribution of marks with Objectively Structured Practical Examination and Traditional Practical Examination in students of the study group

Figure 2 shows the correlation coefficient of comparison of marks by the two methods was found to be 0.54 with a P value of 0.0001 suggesting significance. Bland–Altman plot showed that approximately 96% students got marks within the limits of agreement, and there were only five outliers [Figure 3].

Figure 2

Scatter diagram and correlation coefficient “r” of scores of students by Objectively Structured Practical Examination and Traditional Practical Examination

Figure 3

Bland–Altman Plot of scores of students using the Objectively Structured Practical Examination and Traditional Practical Examination

Evaluation of feedback responses demonstrated that the participating students strongly endorsed OSPE and preferred it over TPE [Table 1]. Majority of the students felt that OSPE led to an improvement in their practical skills, satisfaction of assessment, and confidence in performing the skills, questions asked were relevant to judge practical skills and OSPE should continue as a method of assessment in biochemistry. Nevertheless, 64.9% students felt that the time given at performance stations was less. The students also appreciated the feedback given after OSPE. Responses of faculty reflected that the majority of them felt it was an objective, uniform method of assessment which eliminated examiner bias [Table 2]. Responses also indicated a high level of acceptability and motivation toward incorporation of OSPE.

Table 1

Students perspectives regarding OSPE

Table 2

Faculty perspectives regarding OSPE

DISCUSSION

It is a well-known fact that assessment drives learning. A single examination does not fulfill all the functions of assessment such as assessing knowledge, comprehension, skills, motivation, and feedback. The traditional method of assessment of practical skills raises concerns about examiner variability, standardization, and uniformity of assessment. In the traditional method followed, the practical skills are not directly observed but are assessed based on questions asked at the end of the session.

Structuring of questions and objective assessment has been emphasized and gained importance in the practical examination. The OSPE has over the years gained importance not merely as an evaluation tool, but as a teaching method as well. This has been attributed to the feedback that OSPE gives to both the students and the faculty.B

Several studies have proved that OSPE is a reliable assessment tool.[34] Feroze and Jacob conducted OSPE for practical assessment in pathology and found that OSPE was more objective and measured practical skills better as compared to the traditional method.[5] Studies forwarded by Rahman et al. and Menezes et al. also emphasize OSPE as a better assessment technique over the traditional method for measuring practical skills of MBBS students in physiology and forensic medicine, respectively.[67] Another study conducted by Yaqinuddin et al., also found OSPE to be an efficient tool to assess the practical aspects of knowledge in the subject of anatomy.[8] Studies have also reported that OSPE is an effective tool in discriminating between good and not so good performers.[910]

In the present study, a total of 94 students participated, and their practical skills in the analysis of abnormal constituents of urine were assessed both by OSPE and TPE. The mean score of students was found to be significantly higher (P < 0.0001) when assessed with OSPE as compared to the traditional method. Only one student got < 50% marks when assessed with OSPE as compared to eight students with TPE. The number of students achieving more than 70% marks was also significantly higher with OSPE. This, however, underscores the need for standard setting to decide the pass marks, as traditional 50% may not be suitable for use with OSPE. The scores obtained by the students in OSPE also showed a significant positive correlation with the scores obtained in TPE, undertaken concurrently in the same setting, supporting validity of the assessment.

The better performance in OSPE could be attributed to the fact that the scoring is objective as standards of competence are preset, and agreed checklists are used for scoring. Examiner variability is reduced which also affects the scoring in TPE. A wide range of skills can be assessed by OSPE as compared to TPE. In addition, OSPE also ensures integration of teaching and evaluation. The variety maintains student's interest, and a large number of students can be assessed in a short time.

The feedback from students in our study reflected that most of the students felt that OSPE led to an improvement in their practical skills, satisfaction of assessment and confidence in performing the skills. The students appreciated the feedback provided at the end of the OSPE and felt it to be an important factor in improving their learning. Feedback from the faculty provided an insight into their satisfaction and motivation to adopt OSPE as an assessment tool. Our findings regarding the attitude of students and faculty toward OSPE correlate with earlier findings. In a study conducted by Malik et al., OSPE was rated by students as an effective, useful, interesting, and challenging exam.[11] The faculty and students both favored OSPE in a study conducted by Kundu et al.[12]

CONCLUSIONS

OSPE not only improves assessment but also provides a forum for the improvement of both teaching and learning through the feedback. Participating students perceived it as an important addition to education. This evaluation demonstrated the need for a structured approach to assessment. Going in line with the notion that assessment drives learning, introducing OSPE would help tailoring teaching-learning to optimize student satisfaction and learning.