Comparing the Effects of Objective Structured Assessment of Technical Skills (OSATS) and Traditional Method on Learning of Students.

BACKGROUND: Despite the benefits of the objective structured assessment of technical skills (OSATS) and it appropriateness for evaluating clinical abilities of nursing students , few studies are available on the application of this method in nursing education.
OBJECTIVES: The purpose of this study was to compare the effect of using OSATS and traditional methods on the students' learning. We also aimed to signify students' views about these two methods and their views about the scores they received in these methods in a medical emergency course. PATIENTS AND METHODS: A quasi-experimental study was performed on 45 first semester students in nursing and medical emergencies passing a course on fundamentals of practice. The students were selected by a census method and evaluated by both the OSATS and traditional methods. Data collection was performed using checklists prepared based on the 'text book of nursing procedures checklists' published by Iranian nursing organization and a questionnaire containing learning rate and students' estimation of their received scores. Descriptive statistics as well as paired t-test and independent samples t-test were used in data analysis.
RESULTS: The mean of students' score in OSATS was significantly higher than their mean score in traditional method (P = 0.01). Moreover, the mean of self-evaluation score after the traditional method was relatively the same as the score the students received in the exam. However, the mean of self-evaluation score after the OSATS was relatively lower than the scores the students received in the OSATS exam. Most students believed that OSATS can evaluate a wide range of students' knowledge and skills compared to traditional method.
CONCLUSIONS: Results of this study indicated the better effect of OSATS on learning and its relative superiority in precise assessment of clinical skills compared with the traditional evaluation method. Therefore, we recommend using this method in evaluation of students in practical courses.

1. Background

Nowadays, the importance of nurses’ tasks and responsibilities in performing nursing cares attracted a lot of attention. It is necessary for nurses to make an evolution in their traditional roles to accommodate with the needs of the modern society (1). The nurses’ knowledge is visible during their work in clinical environment. Nursing education should prepare its students to enable them perform their professional responsibilities in a changing clinical setting (2-4).

Because of the importance of clinical training, appropriate clinical evaluation strategies should be adopted to determine the success rate of clinical training (5, 6). Such strategies are important in evaluating the quality of teaching and learning processes (7), screening students weaknesses, increasing their motivation and helping them increase their efforts in attainment of their educational goals and in assisting teachers to assess their activities (8-13).

Despite the importance of clinical evaluation, this task is still time-consuming and a baffling problem which relies mostly on teachers subjective judgments. Moreover, most of the trainees and students are not satisfied with the ways and the results of evaluation (5, 14, 15).

Evidence shows that most of the newly graduated nurses have sufficient theoretical knowledge but they are not clinically proficient (16). It has been shown that the objective evaluation methods are appropriate alternatives to traditional manners. These methods are especially effective in giving immediate feedbacks to students on their flaws (17). Although the objective methods of evaluation, such as objective structured practical examination (OSPE) and objective structured assessment of technical skills (OSATS) are time consuming and need more human and financial resources, but bring more satisfaction to both students and their teachers (12, 18).

OSATS is an examination for evaluating qualification in practical skills, which is organized objectively in different stations and students are asked to do special clinical tasks in each station. This method was firstly introduced in Toronto University by the department of surgery in 1990 (19). Despite the benefits of OSATS and it appropriateness for evaluating clinical abilities of nursing students (20), few studies are available on the application of this method in nursing education.

2. Objectives

The purpose of this study was to compare the effect of using OSATS and traditional methods on the students’ learning. We also aimed to assess students’ views about these two methods and their views about the scores they received in these methods in a medical emergency course.

3. Patients and Methods

This quasi-experimental study was performed in nursing and midwifery school of Gonabad University of Medical Sciences (GUMS) in 2013. The study sample included 45 first semester students in nursing and medical emergencies passing a course on fundamentals of practice. All students were selected by a census method. Inclusion criteria were willingness to participate in the study and having no excessive absences from the class sessions. An absence of more than three sessions was selected as the exclusion criteria.

For gathering data and evaluating students, checklists were prepared using the ‘text book of nursing procedures checklists’ (21). Ten faculty members in the nursing school of GUMS confirmed the appropriateness and content validity of checklists and its reliability was also confirmed using the Cronbach’s alpha coefficient (0.80). Checklists were a collection of questions, words or sentences related to each skill and assessors marked questions, expressions, or sentences that were more relevant to respondents performance. Moreover, we designed a questionnaire containing items about learning rate and students’ estimation of their received scores with respect to their evaluation. These items were responded on a four point Likert scale ranging from strongly agree (= 4) to completely disagree (= 1).

In this study, traditional exam was implemented at midterm and the OSATS at the end of the term. At the beginning of semester, the content of the course was divided into two parts, one to be taught in a half of the semester. Skills composed in each part of the content were matched regarding their levels of difficulty or easiness. Therefore, a half of the skills were taught at the first half of the semester and were then evaluated using traditional methods in midterm and the other half of skills taught at the second half of the semester were evaluated using the OSATS method at the end of the semester. In each exam (i.e. traditional of OSATS), every student was asked to select a skill randomly; then he or she was asked to prepare the needed tools and perform it on a mannequin. Scoring in traditional tests depends on the students’ ability to answer the oral questions, exact performance, having self-confidence and the speed of performance. Assessment was performed using prepared checklists and at the end, they provided a questionnaire to rate their learning, estimate their received scores with respect to their evaluation and express their opinions about this method.

After completion of the second half of the semester, the OSATA exam was performed. At this time, seven stations with trained examiners were prepared for testing. The examiners remained fixed to the stations for all the students. All examiners were faculty members with a Master of Science in nursing and were previously trained in the objective structural test of clinical skills, how to implement it, their responsibilities as examiners, the structure of the checklists, the structure and the time allocated to each station. At first, all students were gathered in a class to prevent information exchanging during the test. In each station, students read its guideline posted outside the station. The students were asked to perform what they were asked to do, leave the current station after hearing the bell voice and enter the next one and similarly turn in other stations too. Each station timed about 15 minutes. The examiner presented in each station rated the students’ performance using a specific checklists. After the last station, the students responded to the questionnaire about the evaluation method again and leaved the testing environment. Finally, an expert weighted the scores of the stations and summing up them to yield a total score for each student.

3.1. Ethical Considerations

The Ethics Committee of GUMS approved the study protocol. All ethical issues such as obtaining informed consent and avoiding plagiarism were followed. The respondents were assured of the confidentiality of their personal information in dissemination of the findings. All participants signed a written informed consent before participation in the study and were briefed about the purpose and methodology, including the benefits and the structure of OSATS.

3.2. Data Analysis

Data was analyzed using SPSS 13 (SPSS Inc., Chicago, IL, USA). Descriptive and inferential statistics were used. Moreover, paired t-test was used to compare the students’ mean scores on the two exams and independent samples t-test was used to compare the students’ mean scores for gender, field of study and marital status.

4. Results

The mean age of students was 20.70 ± 3.80 years with a range of 18 to 33 years. Sixty percent of students were nursing students and the remaining were studying medical emergencies. Most of the students were males (66.7%) and single (84.4%).

The mean of the students’ score in OSATS was significantly higher than their mean score in traditional method (P = 0.01). Moreover, the mean of self-evaluation score after the traditional method was relatively the same as the score the students received in the exam. However, the mean of self-evaluation score after the OSATS was relatively lower than the scores the students received in the OSATS exam (Table 1).

Table 1.

Comparing the Mean and Standard Deviation of Students’ Scores According to Objective Structural Assessments and Traditional Assessments [a]

Type of Exam	Mean ± SD	Range of Score	P Value [b]
Self-evaluating score in traditional methods	15.75 ± 2.38	8 - 20	0.07
Self-evaluating score in OSATS	16.81 ± 2.41	10 - 20
Final score of traditional exam	15.20 ± 1.01	13 - 17	0.01
Final score of OSATS exam	17.93 ± 0.96	16 - 20

a Abbreviation: OSATS, objective structured assessment of technical skills.

b Paired t-test.

As Table 2 shows, no significant differences were found between the students’ mean scores in OSATS or in traditional method for their gender, field of study and marital status.

Table 2.

Frequency Distribution and Mean Standard Deviation of Students’ Score for Their Gender, Field of Study and Marital Status [a]

Variable	Type of Exam
	No. (%)	OSATS		Traditional
		Mean ± SD	P Value	Mean ± SD	P Value
Gender			0.7		0.3
Female	15 (33.4)	17.93 ± 0.98		5.27 ± 1.10
Male	30 (66.6)	17.93 ± 0.96		15.17 ± 0.98
Field of study			0.3		0.8
Nursing	27 (60)	18.19 ± 0.87	15.15 ± 0.98	15.15 ± 0.98
Medical emergency	18 (40)	17.56 ± 0.98	15.28 ± 1.07	15.28 ± 1.07
Marital status			0.6		0.1
Single	38 (84.5)	18.03 ± 0.94		15.18 ± 1.06
Married	7 (15.5)	17.43 ± 0.97		15.29 ± 0.75

a Abbreviation: OSATS, objective structured assessment of technical skills.

Most students believed that OSATS can evaluate a wide range of students’ knowledge and skills compared to traditional method. Besides, they believed that this method provides better learning opportunities, but needs more practice to gain success. Moreover, 95.6% of students believed that the quality of OSATS was higher than traditional one and this test is more accurate in evaluating students’ clinical skills. In addition, 60% of them declared that the evaluation had an important effect on their learning. According to the students, if the students were aware of the evaluation method, they would try more to learn. The items “better evaluation of the student’s weakness” and “better evaluation of the students’ strengths” were selected more for the OSATS than the traditional method (Table 3). Moreover, 93.3% of the students were in agreement using OSATS in the next semesters.

Table 3.

Frequency Distribution of Students’ Opinions About the Evaluation Methods [a,b]

Items	Type of Answer
	Totally Agree		Agree		Disagree		Totally Disagree
	Traditional	OSATs	Traditional	OSATS	Traditional	OSATS	Traditional	OSATS
Evaluation of wide range of skills and practical methods	1 (2.2)	15 (33.3)	19 (42.2)	28 (62.2)	18 (40)	2 (4.4)	7 (15.6)	0
Increase in performance speed	2 (4.4)	12 (26.7)	15 (33.3)	30 (66.7)	19 (42.2)	3 (6.7)	9 (20)	0
Needed time to learning	2 (5)	11 (24.4)	7 (15)	29 (64.4)	27 (60)	5 (11.1)	9 (20)	0
Deeper learning compared to other methods	7 (15)	13 (28.9)	9 (20)	29 (64.4)	18 (40)	3 (6.7)	11 (25)	0
Considering more detailed points	10 (22.2)	20 (44.4)	23 (51.1)	24 (53.3)	9 (20)	1 (2.2)	3 (6.7)	0
Better evaluation of the student’s weaknesses	10 (22.2)	16 (35.6)	19 (42.2)	24 (53.3)	12 (26.7)	3 (6.7)	4 (8.9)	2 (4.4)
Better evaluation of the student’s strength	11 (25)	11 (24.4)	16 (35)	30 (66.7)	13 (30)	4 (8.9)	5 (10)	0
Need to more practice to earn success	5 (10)	22 (48.9)	7 (15)	19 (42.2)	22 (50)	4 (8.9)	11 (25)	0
More emphasis on practical skills	9 (20)	12 (26.7)	22 (48.9)	28 (62.2)	9 (20)	4 (8.9)	5 (11.1)	1 (2.2)
More precision in evaluating clinical skills	4 (8.9)	18 (40)	7 (15.6)	24 (53.3)	20 (44.4)	3 (6.7)	14 (31.1)	0
Requiring oral questions in traditional method and written questions in OSATS	6 (13.3)	16 (35.6)	24 (53.3)	20 (44.4)	9 (20)	8 (17.8)	6 (13.3)	1 (2.2)

a Abbreviation: OSATS, objective structured assessment of technical skills.

b All data presented as No. (%).

5. Discussion

The present study showed that the mean score of students in OSATS was significantly higher than the traditional method. Moreover, in this study, we could not find any significant differences between students’ mean scores regarding variables such as gender, field of study and marital status. Our finding on the difference between the two methods was consistent with the results of some previous studies (19, 22-25). For instance, Rahman et al. in a study on 400 physiology students compared the OSPE and traditional method and reported that the students’ score was significantly higher in OSPE than traditional evaluations (26). In contrast, Pishkar et al. compared the OSPE and traditional method and reported that students’ score in traditional evaluation method was higher than OSPE (22).

In the current study, 62.2% of students believed that OSATS could evaluate a wide range of their skills. Similar findings were reported in studies conducted by Imani et al. (27), Pierre et al. (28), and Menezes et al. (29) who investigated other objective evaluation methods such as Objective Structured clinical examination (OSCE) and OSPE.

Most students participated in the present study believed that the OSATS was superior to the traditional method for being more precise, better evaluation of students strengths and weaknesses and inducing deeper learning. Although at the time of this study, no similar studies were available on OSATS, our findings are consistent with results of Pierre et al. (28) and Schoonheim-Klein et al. (30) who studied the OSCE method, which is similar to the OSATS in its nature and structure. On the other hand, most students in a previous study reported that the objective assessment method used (i.e. OSCE) was not useful to them (31). As reported by Alinier et al. (32), the OSATS has some limitations, but these limitations should not be an obstacle in using this useful method.

In conclusion, the results of this study indicated the better effect of OSATS on learning and its relative superiority in precise assessment of clinical skills compared with the traditional evaluation method. Therefore, we recommend using this method in evaluation of students in practical courses.

This study had some limitations such as limited number of students who participated in the study, not having a control group, and not using a randomized design. Then the results may not fully be generalized to all students. Given the results of the present study, we suggest to conduct similar research with a control group and a larger sample size.