Does attitude hinder or help selecting evaluation questions?

BACKGROUND: Positive attitude leads to a more successfully implementation of a change. We investigated the effect of attitudes of stakeholders toward a program on their prioritization of the program components for selecting the key question of a theory-driven evaluation with concept mapping method.
MATERIALS AND METHODS: During a brainstorming session, stated statements defined the program components. Then they were sorted and rated regarding the importance and feasibility of them. In addition, the attitudes of participants were assessed by a 30 items questionnaire extracted from a pool named as "50 reasons not to change." We determined and compared the consensus points of participants both with and without of considering their attitudes toward the program.
RESULTS: The participants were divided into two groups of high (45% - above the mean) and low (55% - below the mean) attitude. Brainstorming discussions generated a pool of almost 120 statements which were subsequently refined to 44 statements. Matching the rating scores between two attitude groups yielded a consensus at a higher priority than the other method.
CONCLUSION: In the concept mapping procedure, it is crucial to reach the consensus with respect to the participants' attitude, rather than the similarity of mean scores of feasibility and importance.

INTRODUCTION

Clarifying the main question(s) is the fundamental step of every evaluation practice. But, how can we achieve that? Whose query is the best choice? To answer, the terms divergence and convergence,[1] indicate two phases of identifying and selecting evaluation questions. Moreover, contribution of the different program stakeholders, according to Fitzpatrick et al.,[2] increases the validity of results:

“Involving stakeholders increases the chances that they will use the results because it helps to reduce their anxiety about the evaluation and improves their understanding of its purposes and intent, as well as ensuring that at least some of the evaluation questions addressed their concerns.”

In the light of foregoing, the approach involves a high degree of stakeholders’ participation which despite the advantage of enhancing the feeling of ownership among the participants raises the interference of their values,[3] about the program. Besides, we know that human values are highly related with their attitudes, and attitude is related to behaviors;[4] in other words, attitude has a crucial role in the formation of behaviors and reactions toward the change.[567] It is also believed that attitudes of stakeholders may either facilitate or inhibit the change processes.[8]

Nevertheless, and in spite of Jackson and Trochim[9] concerns about intergroup agreements and/or differences, the latter has not been addressed well enough in the evaluation studies. However, different other issues have been frequently examined such as being at doctoral or nondoctoral-level.[10] Therefore, the participants’ consensus regarding their points of view about the change program has still works to do, although in a study,[11] the rating results was compared to make consensus on key areas of importance and feasibility. However, shaping the evaluation criteria by using values as the determinant factors is to some extent rare.[12] Though, in a recent approach named as “values inquiry,”[13] stakeholders’ values about the evaluation (not the program being evaluated) were taken into account. And, in some cases the attitude was utilized as an external factor that may affect the accomplishment of evaluation intended effects.[14]

Given the foregoing, the aim of this article is to investigate whether or not the attitude of the participants toward the program influences the definition of evaluation questions. We will simply analyze the results using “ladder graph” and prioritization of statements that all compose the program components. It is done both with and without incorporating the participants’ attitude into the results of “rating the statements.”

The evaluated program

The medical curriculum at Isfahan University of Medical Sciences consists of four major phases: The first 2 years and a half dedicated to basic sciences. Then, 1-year for introduction to clinical medicine (physiopathology courses), 2 years for clinical exposure, named as “clinical clerkship,” and finally eighteen month, named as “internship,” for delegating the patient's responsibility.

The educational authorities introduced a change into the clinical clerkship phase by dividing it into two parts, known as “primary beginner” and “advanced beginner.” The first part remained unchanged and during which the students should act as observers, whereas in the second part, as an innovative program, clerkship students, mostly under close supervision, should practice on clinical skills, and spend a few nights on call to fulfill more hand on action practices especially in outpatient and emergency settings. However, the change has faced some challenges after about two years of its starting time (i.e., April, 2010), and there is a great controversy about whether to continue or to cease the program. That's why, after surfacing the explicit assumptions about the program, our subsequent effort was figuring out the evaluation key points.

Evaluators and their objectives

Education Development Office at Isfahan University of Medical Sciences, School of Medicine has five main committees named as: Curriculum Development, Faculty Development, Teacher Evaluation, Student Assessment, and Program Evaluation. For the purpose of assessing the merit and worth of the mentioned innovative program (i.e., advanced clinical clerkship), the Program Evaluation committee was assigned to design a comprehensive evaluation plan. The present article is the result of the first step of the master plan of that study.

MATERIALS AND METHODS

Relevant stakeholders including the administrative body of the medical school, university vice chancellor for education, and academics in charge of clinical clerkship in addition to another representative of each clinical department were invited to a meeting.

We used two separate instruments for the study. First and before the brainstorming session, we administered a researcher-developed attitude questionnaire. It was a short form of 50 prevailing excuses that people in organizations may use for pretending against a change.

Dealing with various aspects of the program, the items showed the stakeholders’ different concerns about it. For example the people may say: “It's not my job ”or “It won’t work in our organization.” The original forms of these statements are gathered in a diagram known as “50 reasons not to change.”[15] Though, for the purpose of the study we adjusted and integrated them into a 30 items questionnaire. Then we asked some expert bodies to validate the instrument and used their suggestions for improving the tool. For determining the reliability of the tool, we used the α-Chronbach index which was above the 0.75.

Ranging from “strongly disagree” to “strongly agree,” each item was scored from 1 to 5 and the sum showed the reaction (attitude) of each participant toward the “advanced clinical clerkship” program. In addition, we recorded the negative statements during the analysis. Therefore, each questionnaire could be score between 30 and 150. Finally, based on scoring above or below the mean score of participants’ attitudes, they were divided into two groups of high and low attitude groups (HAG and LAG).

Brainstorming session

As the divergence phase of determining the evaluation question, the participants were asked to generate some statements that best complete this query: “A good medical clerkship is the one that….” The generated statements were refined to a ball of 44 items which constituted the second instrument. Then, the participants rated all of the final statements based on their importance and feasibility. For importance, each statement could be rated as very important (5), important (4), no idea (3), low important (2), and very low important (1). Similarly, for feasibility, they were rated as very feasible (5), feasible (4), no idea (3), low feasible (2), and very low feasible (1).

For determining and selecting the key concerns of the participants, we employed ladder graphs on which the statements sorted by averages of importance and feasibility scores. Furthermore, for the ease of the work, we numbered each of the statements, and used the numbers, instead of the main statements, on the bars. Totally, we had four separate graphs:

Total importance versus total feasibility,

Feasibility of HAG versus feasibility of LAG,

Importance of HAG versus importance of LAG, and

Importance + feasibility of HAG versus importance + feasibility of LAG. In this way, we compared the two kinds of prioritization, considering or ignoring the attitude in a participatory selection of the main evaluation questions.

In fact, after sorting and matching the statements, reaching to horizontal bars of the ladders would indicate the consensus or starting point of evaluation.

Findings

Thirty persons (21 men and 9 women), attended the meeting. Their ages ranged from 32 to 60 years, with the average of 46 (6.5), and the average of experience of 15.5 (7.7) years. While the majority of participants (80%) were academic staff, only 30% of participants were the decision makers at either departmental, school, or higher levels.

Attitude survey

Ranging from 65 to 122, the mean score of attitudes was (88.28 ± 13.42). But because the number of nonacademic, females, and decision makers were statistically too tiny, demographic and contextual variables did not yield accurate differentiation results.

Anyway, as mentioned earlier in this paper, the participants were divided into two attitude groups. 45% and 55% of them were included in HAG and LAG, respectively.

The maximum of the mean opinion scores (4.24 ± 0.99, after recoding), was given to the statement: “There is no need to this change.” However, in this case, there was a significant difference between the two groups, with mean score of (4.64 ± 0.63) for HAG and (3.87 ± 1.13) for LAG groups (P = 0.02). On the other hand, the minimum mean attitude score, 1.36 (1.22), was given to the statement of “enough money has not been provided for it,” with a mean of (1.36 ± 1.39) for HAG and (1.36 ± 1.8) for LAG groups. There was no significant difference between the two groups with this item.

To other statements listed below, HAG had given more scores than LAG (P < 0.05):

I am not eager to participate in this change

We don’t have enough time for it

All aspects of the change were not considered

The administrator bodies do not share an agreed view point on it.

Ratings and interpretation

The most important statement was number 5: “Good clinical clerkship is a program which designs learning activities appropriate for curriculum as well as students’ levels so as to prepare students for internship.” And the least one was number 14:“ Good clinical clerkship integrates the basic and clinical sciences, both horizontally and vertically.” Moreover, the most feasible statement was number 22: “Good clinical clerkship informs the students at the beginning, about major components of the program.” And the least feasible one was number 28: “Good clinical clerkship adjusts the varieties of educational environments and field with what in which, students should work after graduation.”

There was a high correlation between total importance and total feasibility scores of all participants (r = 0.965, P = 000). Moreover, significant correlations existed between feasibility and importance of statements among both LAG and HAG [Table 1].

Table 1

Correlations between statements rating results

Nonetheless, as the main objective of our study, we were looking for the major concern of participants for the start point of the program evaluation. Furthermore, reaching the parity (i.e., consensus) on top scored statements on the ladder graphs was the selected procedure for defining the evaluation question. However, matching the total importance versus total feasibility scores did not yield any similarity. Thus, as we presumed, it might have occurred as the result of the diversity of the participants’ view points toward the program (i.e., their attitudes). Nevertheless, drawing a ladder graph for total mean of importance + feasibility of each statement with separating the two attitude groups did not alleviate the problem. No horizontal line was found. The graph of feasibility of the statements with each attitude group on each side also was not the solution. Eventually, developing a separate graph for importance of the statements and matching the two groups worked at a very high priority level. Matching the importance scores of LAG with HAG resulted in similarity of statements number 5 “good clinical clerkship is a program which designs learning activities appropriate for curriculum as well as students’ levels so as to prepare students for internship” and number 17, “good clinical clerkship is a program which is logistically supported” respectively at 2nd at 10th priority (out of 44) [Table 2].

Table 2

Matching importance scores of LAG & HAG

DISCUSSION

In the present study, we successfully prioritized the program elements by using a helpful criterion (i.e., importance), and by achieving consensus between participants’ subgroups in terms of their attitude. So, we reached a good start point for evaluating the program.

The evaluation start point was statement number 5: “Good clinical clerkship is a program which designs learning activities appropriate for curriculum as well as students’ levels so as to prepare students for internship.” This statement constitutes the essence of the program: “Preparing the students for internship;” What that has been mentioned in ample of publications.[161718192021222324] More or less all of the mentioned authors believe that the stage before internship should both prepare the students for beginning a less stressful internship phase of medical education and provide them with a well relevant curriculum. Therefore, we reached at a very decisive start point.

Although two rating scales such as importance and feasibility, or improvement etc.,[25] are recommended to be used, as we found here, in some cases there is not a well parity in such criteria. So, in this study, good attitude discrimination among participants, with clarifying the main concerns about the program helped us arriving a good start point for evaluation.

The high attitude group members had a more commitment and certainty about the “advanced clinical clerkship” than the ones in the other group. Furthermore, lack of eagerness, time constriction, and feeling a lack of agreement on the program among administrative bodies was the most acknowledged concerns among almost all of participants. Unpredictably, program stakeholders had some concerns about the program. Although they agreed that the change had been necessary, one of the best results in our attitude survey was the point that “the program suffers from poor planning.” In other words, concrete curriculum planning had been the most important concern of the stakeholders. This finding was consistent with a previous study on the same program.[26] Furthermore, it is apparent that lacking a good curriculum plan, we are far from advanced steps of education such as integration, what that was cited in the result of the statement saying “good clinical clerkship integrates the basic and clinical sciences, both horizontally and vertically.” Moreover, most of the participants believed that there were insufficient financial supports for the program.

CONCLUSION

Summing the results, although there was a high correlation between indicators, using the ladder graph better helped us arrive at a consensus point because from the point of view of participants, none of the statements (except for the selected one) had the same levels of both importance and feasibility. Therefore, it can be claimed that in spite of low participation number, we well benefited from attitude as a discriminative factor for the convergence phase of identifying evaluation questions.[1] And finally, “50 reasons not to change” provided us with a useful list of the stakeholders’ concerns about the program.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

AUTHOR'S CONTRIBUTION

MD contributed in the conception of the work, conducting the study, revising the draft, approval of the final version of the manuscript, and agreed for all aspects of the work. BS contributed in the conception of the work, conducting the study, revising the draft, approval of the final version of the manuscript, and agreed for all aspects of the work.