Comparison of midwifery students' satisfaction with direct observation of procedural skills and current methods in evaluation of procedural skills in Mashhad Nursing and Midwifery School.

BACKGROUND: The clinical evaluation, as one of the most important elements in medical education, must measure students' competencies and abilities. The implementation of any assessment tool is basically dependent on the acceptance of students. This study tried to assess midwifery students' satisfaction with Direct Observation of Procedural Skills (DOPS) and current clinical evaluation methods.
MATERIALS AND METHODS: This quasi-experimental study was conducted in the university hospitals affiliated to Mashhad University of Medical Sciences. The subjects comprised 67 undergraduate midwifery students selected by convenience sampling and allocated to control and intervention groups according to the training transposition. Current method was performed in the control group, and DOPS was conducted in the intervention group. The applied tools included DOPS rating scales, logbook, and satisfaction questionnaires with clinical evaluation methods. Validity and reliability of these tools were approved. At the end of training, students' satisfaction with the evaluation methods was assessed by the mentioned tools. The data were analyzed by descriptive and analytical statistics.
RESULTS: Satisfaction mean scores of midwifery students with DOPS and current methods were 76.7 ± 12.9 and 62.6 ± 14.7 (out of 100), respectively. DOPS students' satisfaction mean score was significantly higher than the score obtained in current method (P < 0.000). The most satisfactory domains in the current method were "consistence with learning objectives" (71.2 ± 14.9) and "objectiveness" in DOPS (87.9 ± 15.0). In contrast, the least satisfactory domains in the current method were "interested in applying the method" (57.8 ± 26.5) and "number of assessments for each skill" (58.8 ± 25.9) in DOPS method.
CONCLUSIONS: This study showed that DOPS method is associated with greater students' satisfaction. Since the students' satisfaction with the current method was also acceptable, we recommend combining this new clinical evaluation method with the current method, which covers its weaknesses, to promote the students' satisfaction with clinical evaluation methods in a perfect manner.

INTRODUCTION

The clinical evaluation, as one of the important elements in medical education, must measure students’ competencies and abilities in the student-patient encounter.[12] The commonly used slogan that “assessment drives learning,”[34] although certainly true, presents a rather limiting concept. It was therefore suggested that it should be replaced by an alternative motto: “Assessment expands professional horizons.”[5] This stresses an important role of assessment in developing multiple dimensions of medical profession.[6] However, unfortunately traditional and current methods of students’ clinical evaluation face some challenges such as large number of students, a too short time for evaluation, non-objective methods,[1] lack of students’ work supervision, unfair grades, lack of self-evaluation, failure to provide feedback, improper evaluation methods like multiple choice questions in clinical settings,[7] and forgetfulness of students’ activities.[8] All these deficiencies can lead to students’ dissatisfaction, so that 77% of students in Mashhad Nursing and Midwifery School demanded a revision on evaluation methods[7] and 52% of clinical instructors have expressed the lack of standardized evaluation tools as the most important issue of clinical evaluation.[9] Gerrow et al. (2004) stated that the potential causes of reliability and validity problems are a result of traditional methods of clinical evaluation.[10]

In recent two decades, we have observed rapid evolution of clinical assessment methods from traditional ones to more sophisticated evaluation strategies.[11] The introduction of these new assessment methods has had a system-wide effect on assessment and learning in developing multiple dimensions of medical education and the medical profession in general.[6] The Department of Health and the Foundation Committee of the Academy of the Medical Royal Colleges produced a document named the “foundation program” in 2005 setting out a new curriculum for medical graduates, with incorporation of modern assessment tools. The foundation program introduced standardized validated assessment tools including Multi-source Feedback (MSF), Direct Observation of Procedural Skills (DOPS), Case-based Discussions (CBD), and Mini-Clinical Evaluation Exercise (Mini-CEX).[12] The most important method of evaluation is direct observation of trainees performing the clinical skills.[13] Feedback is needed to optimize the effectiveness of experiential learning in the workplace.[14] In this regard, DOPS appears to provide this feature. DOPS is a method used specifically to assess practical skills and is designed to provide feedback.[15] Therefore, it has high face validity.[16] But it may have some limitations such as being time consuming.[15] Logbook, which is one of the new clinical evaluation methods, is the current method of midwifery students’ clinical evaluation in Mashhad Nursing and Midwifery School. It is used by the students to document all about their observations and performance information. Although this method helps to focus on learning objectives and acts as an assurance for students’ experiences equality, it may have some limitations such as insufficient accuracy of the students’ documentation, and therefore instructors’ scoring failure to provide a structured feedback to the students.[17] All these strengths and weaknesses can be more distinctive as being applied and assessed from the view point of those involved. The implementation of any assessment tool is basically dependent on the acceptance of students and instructors who are affected by it.[1819] The students’ attitudes and satisfaction with their major and educational activities such as evaluation can affect provocation and promoting of educational quality.[18] Since attitudes and satisfaction play an important role in learning and skill acquisition,[1820] we can reinforce positive factors and convert the negative ones of the current evaluation status by assessing students’ attitudes to promote their satisfaction more.[21] So, having knowledge about students’ satisfaction with educational issues is very important to help the university to achieve educational goals.[19] This evoked us to investigate the answer to the question, “How is midwifery students’ satisfaction of Mashhad Nursing and Midwifery School with DOPS in comparison with the current method?”

MATERIALS AND METHODS

This quasi-experimental study was conducted on 67 undergraduate midwifery students in the university hospitals affiliated to Mashhad University of Medical Sciences (MUMS), Iran, in 2010. The inclusion criteria were as follows: Students being trained in maternity training with a past history in this course, students in intervention group who were attending to either a meeting of introducing DOPS method before the intervention began or at the beginning of each training session. Students who were evaluated fewer than twice for each of the selected techniques (vaginal examination, Leopold maneuvers, Fetal Heart Rate (FHR) Auscultation) in DOPS method were excluded. To make a single-blind research, we also held a similar workshop and sessions at the beginning of the research and training for the control group about evaluation in general. In addition to this, researcher's attendance in control group beside the tutor helped to equalize the circumstances of the two groups. The study sample size to compare means was calculated to be 32 people in each group. Finally, 67 students were selected through a convenience sampling method; any of the students who met the inclusion criteria entered the study. Then, they were divided into intervention (DOPS with 33 students in addition to current method) and control (current method with 34 students) groups according to the training transposition. In other words, the students whose training was beforehand in time were included in the control group and the students in subsequent training were placed in the intervention group. This study was approved by the Vice Chancellery for Research of MUMS. Written informed consents were obtained from all participants.

In both control and intervention groups, the inclusion and exclusion criteria were assessed and the sample's individual characteristics form was completed by the students. The current method used in this study was logbook which consisted of four parts: 1) a documentation form for practicing skills by students; 2) a documentation and evaluation form for scientific discussions held by students; 3) a documentation form for management of high-risk maternity cases; and 4) a student evaluation form filled by the instructors. The student and instructor had a mutual interaction in students’ scoring. Logbook was assessed by the instructor at the end of training period. After students’ self-evaluation, if there were any amendable notes, the score would be modified. On the last day of training, the students completed self-evaluation column in “documentation form” and instructors completed “student evaluation form” in students’ presence, and eventually, the instructor provided the final score.

In the intervention group, a joint session for instructors and students and a workshop just for the instructors were held to introduce DOPS technique principles and tools. A 15-min session was also held as a reminder for the intervention group at the beginning of each training session by the research team. Students’ evaluation was conducted according to their request. The evaluation was processed in such a way that the researcher and the instructor were behind the student without any interference (except for necessary situations) to observe students’ mentioned procedural skills directly and grade them using reliable and valid rating scales simultaneously. Reliability of applied rating scales was calculated through inter-rater reliability and their validity was assessed by Content Validity Index (CVI) as follows: Vaginal examination rating scale (r = 1 and CVI 0.8), Leopold maneuvers and FHR auscultation rating scales (r = 0.9 and CVI = 0.9). Finally, verbal and written feedbacks were presented to the student immediately after the scoring finished so that the students could discuss about the awarded scores. Since the evaluation tools were identical for students in different semesters, the instructors considered expected competency levels in the evaluation of any special semester.

In both control and intervention groups, the questionnaires of satisfaction with clinical evaluation method were completed by the students, maximally 1 week after the determined training. Due to the presence of different instructors in different training groups, a “survey form of clinical students’ views toward the quality of instructors’ education” prepared by Education Development Office of Mashhad Faculty of Nursing and Midwifery was completed by the students in order to control the quality of instructors’ education in relation with students’ satisfaction. This form consisted of 20 statements with a 20-80 score range in a 1-4 score scale, which is approved by content validity and Cranach's alpha (a = 0.96). The questionnaires of satisfaction with DOPS and logbook methods, prepared by extensive literature review, were approved by CVI (CVIDOPS = 0.8, CVIlogbook = 0.9) and Cranach's alpha reliability (αDOPS = 0.94, αlogbook = 0.92). These questionnaires consisted of nine similar domains (fairness, consistence with learning objectives, suitability, adequate time, possibility of performance, promoting skills, objectiveness, stressfulness, and interest in evaluation method) and one specific domain for DOPS (adequate number of assessments for each skill) in a 1-5 score scale (completely disagree = 1 to completely agree = 5), as well as an open question to assess students’ additional opinions. But the score of domain “stressfulness” was calculated in a reverse manner (completely disagree = 5 to completely agree = 1) in the total score of evaluation. Minimum and maximum scores of satisfaction with DOPS questionnaire were 17 and 85, respectively, and the scores for logbook were 16 and 80, respectively. To provide comparison and a better understanding, total1* and domains’2* scores of satisfaction were calculated on a 100-score scale. Higher scores indicated greater satisfaction with the mentioned methods in all domains except “stressfulness” which was reverse.

Data were analyzed through descriptive (mean ± SD) and analytical [independent t-test and Wilcoxon test to compare satisfaction scores in two groups of students, two-way analysis of variance (ANOVA), and correlation coefficients] statistics by SPSS-11.5. P < 0.05 was considered significant.

RESULTS

All participants in the control and intervention groups were matched concerning their age, interest in the field, work experience, total educational mean, except for their training time (semester and training shift) and acquired skill scores in each clinical evaluation method.

Mean of students’ satisfaction with DOPS method was 76.7 ± 12.9 (out of 100), and with the current method, it was 62.6 ± 14.7 (out of 100). Total score of students’ satisfaction with DOPS was significantly higher than with the current method (independent t-test, P < 0.000). The most satisfactory domain of DOPS method was being “objective” (87.9 ± 15.0), and in the current method, it was “consistence with learning objectives” (71.2 ± 14.9). In contrast, the least satisfactory domain in DOPS method was “adequate number of assessments for each skill” (58.8 ± 25.9), and in the current method, it was “being interested in applying the clinical evaluation method” (57.8 ± 24.5) [Tables 1 and 2].

Table 1

Score mean of midwifery students’ satisfaction with domains of the current and DOPS clinical evaluation methods

Table 2

Score frequencies of midwifery students’ satisfaction with domains of the current and DOPS clinical evaluation methods

Two-way ANOVA showed that there was a significant difference in the mean of students’ satisfaction score concerning age and the group (P = 0.001). Age (P = 0.017) and group (P = 0.000) separately had a statistically significant effect on the satisfaction mean score of clinical evaluation method too. But none of the other mentioned variables, alone, had a statistically significant effect on the satisfaction scores. Although the acquired skill scores of the two different clinical evaluation methods were heterogeneous in the control and intervention groups, they did not have a significant effect on satisfaction scores [Table 3].

Table 3

Mean of midwifery students’ satisfaction total score according to some selected variables separately in two current and DOPS groups

According to Pearson correlation coefficient, training time (week of semester) had a direct linear relationship with the students’ satisfaction in all the students (P = 0.003, r = 0.357), but this relationship was inverse in the control group (P = 0.004, r = -0.483) and there were no relationship between the students’ satisfaction and training time in the intervention group (P = 0.356, r = 0.166).

Some of the students expressed their opinions about DOPS and the current methods in response to the open question of the satisfactory questionnaire. Their obtained opinions about DOPS were as follows: “This method is very useful in learning and enhances clinical skills and students’ attention, and it leads to a better outcome and survives learning for longer time.” They added: “This method makes the students discover their problems, so it is appropriate for problem solving. In this method, students are not judged according to the instructors’ previous assumptions; this can eliminate severe dissatisfaction of the students with the discrimination which is created by the instructors.” Just two students mentioned “DOPS makes more stress.” The positive point which was declared by the students of control group was “logbook is useful to remind the students and instructors of daily objectives,” but according to the students, this method has some disadvantages such as carelessness in completing the logbooks, inadequate opportunity to observe the students’ skills by the instructors, no feedback, paying more attention to assess the quantity than the quality of the skills, and also consideration of theoretical aspects more important in this method.

DISCUSSION

This study showed that DOPS method is accompanied with greater students’ satisfaction, compared to the current method. According to the results, midwifery students in the intervention group were relatively satisfied3* with DOPS, but the students in control group, although not in favor of logbook, were not opposing it.

In a review of previous studies, no study investigating students’ satisfaction with DOPS method quantitatively was found. So, we discuss our findings in comparison with the other new clinical evaluation methods such as Mini-CEX, which is similar to DOPS in all aspects except evaluation fields,[1722] and objective structured clinical examination (OSCE). Weller et al. (2009) reported trainees’ satisfaction with Mini-CEX as 7.3 out of 10, which is consistent with our study.[23] Aj (2006), comparing logbook with a checklist, showed that midwifery students’ satisfaction with logbook (85.7%) was greater than with the checklist (47.6%),[24] which is not in line with the findings of our study. The reasons for these inconsistent findings can be: 1) conducting these two methods simultaneously on the same group of students in Aj's study, possibly affecting the students psychologically as a new method; 2) assessment of students’ satisfaction with just one question which could have reduced the accuracy and precision of the participants’ response; and eventually 3) the participants’ education degree which was associate degree in Aj study, whereas it was bachelor's degree in the present study. On the other hand, different levels of carelessness in completing logbooks could have affected students’ satisfaction. As seen in Raghoebar-Krieger's study (2001) on reliability of logbooks, there was no consistency between documentation of diagnostic information of the students and the professor.[25]

To answer the question of “Which characteristics in DOPS method can lead to greater students’ satisfaction?,” we assessed statistically significant differences in the domains of satisfaction between the two groups. Findings showed that students’ satisfaction in all domains with DOPS was greater than with logbook, except three domains of “consistence with learning objectives,” “possibility of performance,” and “stressfulness.” So, we here discuss the possible causes of these statistically significant and non-significant differences in the present study and the related ones. Chehrzad et al. (2004) reported that 83.3% of nursing students with OSCE and 10% with traditional method had high satisfaction with fairness domain.[26] This satisfaction with OSCE is almost similar to that with DOPS (78.8%), but satisfaction with the current method in our study differs significantly from that in Chehrzad's traditional method. Since traditional method has not been clearly mentioned, the comparison between these two methods is impossible. However, some reasons like assessment of satisfaction through fewer questions with just a two-point option (high and low satisfaction) among students of different fields can legitimize these slight and significant differences in satisfaction in the two aforementioned studies although the key item is the method of evaluation. OSCE is performed in an artificial environment with models, and although evaluation items are the same for all examinees, it has some limitations such as evaluation in an unrealistic setting.[27]

One of the most important factors affecting the conduct of a suitable evaluation is to select a specific method for the assessment of special learning objectives.[15] In the present study, 87.7% of the subjects in DOPS and 47% in logbook group were satisfied with the suitability of these methods. These findings confirm the characteristic of DOPS which is specific to evaluate procedural skills. But it indicates that logbook in the current format does not fulfill this property.

An objective tool can eliminate subjective instructors’ judgments.[115] Unfortunately, one of the problems in evaluation process is lack of an objective tool.[28] Surprisingly, the most satisfactory domain in DOPS method was objectiveness. The best explanation for this can be students’ opinion which emphasizes this feature: “Students are not judged according to the instructors’ previous assumptions.” Additionally, determination of performance process, scoring, and results interpretation of DOPS lead to complete satisfaction for students with objectiveness domain.[29] But in the current method, although all necessary skills are listed, there is no position for evaluating skills’ components step by step. In contrast, the least satisfactory domain in DOPS method was related to “the adequacy of assessment numbers for each skill.” Most of the students suggested three times for each skill.

The greatest students’ satisfaction with the current method was related to “consistence with learning objectives.” This finding was emphasized by one of the students claiming that the current method reminds daily objectives to the students and instructors. This domain is a strong point for DOPS and a milestone of the current method because one reason for students’ dissatisfaction with evaluation methods is not evaluating the students according to learning objectives.[78]

About being interested in applying clinical evaluation method, 78.8% and 44.1% of the students, respectively, were interested in applying DOPS and the current methods. Although based on the change theory people resist against acceptance of new things,[30] it was surprising that students were more eager to apply DOPS method, since being interested in reusing a method may be an indicator of student's satisfaction with it. So, this finding approves the main results of our study concerning greater students’ satisfaction with DOPS.

But we can interpret domains of satisfaction with no statistically significant difference in two groups, for instance, in “possibility of performance,” due to no need for any special settings or tools in these methods. Based on this reason, lower possibility of doing OSCE in Chehrzad's study compared to DOPS in the present study was expected.[26] The other domain was “stressfulness” of these methods. The reason may be direct observation in DOPS method and students’ concern about whether they encounter the required clinical experiences in teaching program during their training in logbook.[25] Since a minimum number of procedures should be performed or seen in a logbook, it is often set arbitrarily and is not validated concerning future performances. So, they do not necessarily correlate with the competence achieved.[17] However, the stressfulness, caused by the evaluation, can be useful because if done in proper time, it can provide an opportunity for amendment.[22]

According to the results, the acquired skill scores had no statistically significant effect on the students’ satisfaction, but the reverse correlation between time and the satisfaction score in the control group indicates less satisfaction through time. It may be due to getting closer to final exams of the university, but surprisingly, students’ satisfaction in the intervention group was not affected by this variable although their training time was closer to final exams compared to that of the control group. With regard to other variables, age, which was initially matched in both groups, had a statistically significant difference with the students’ satisfaction so that younger students were more satisfied with the clinical evaluation methods, which is not consistent with the BC College and Institute Student Outcomes Survey. This article reported that older students were more satisfied with their learning experiences such as evaluation. This difference can be due to the limited numbers of elderly participants in our study.

The study findings can be affected by some inevitable limitations such as assessing the control group prior to intervention group to prevent the phenomenon of informatics dissemination and evaluator–student relationship. We have tried to minimize these limitations by applying identical experienced instructors in the control and intervention groups, holding introductory sessions, and using “survey forms of clinical students’ views toward the quality of instructors’ education.” We have also assessed the effects of these interventional variables using multi-variable statistical analysis.

Since the students’ satisfaction with the current method was also acceptable, we recommend combining such a new clinical evaluation method, which covers the current method's weaknesses, to promote the students’ satisfaction with clinical evaluation methods in a perfect manner. The evaluation methods which are accepted by students can lead to their more accurate implementation. This can lead to identify competent people, to improve educational programs and teaching methods in classroom and clinical settings, to manage facilities, and finally to promote clinical competency of midwifery graduates. The study findings can be used as a guideline for instructors, managers, and the university to plan more acceptable, proper, and objective clinical evaluation methods which can elevate students’ clinical skills.