Development and validation of evaluation tools of nursing students' clinical pharmacology unit.

INTRODUCTION: The need for valid, reliable, and objective tools has always been emphasized in studies related to the clinical assessment of nursing students. The aims of this study were to develop and assess the validity and reliability of the tools used to evaluate the clinical pharmacology unit.
METHODS: This study was a methodological one, conducted in 2016. An item pool was developed based on the literature review and personal interviews with faculty members. The tool's validity was determined through assessment of face validity, content validity, and construct validity, using exploratory factor analysis on the data provided by 264 second- and third-semester nursing students of the Islamic Azad University of Babol University of Medical Sciences. Reliability was determined through internal and external consistency, using a Cronbach's coefficient of the correlation between classes.
RESULTS: Based on the exploratory factor analysis, all items with a special value of >1 were grouped into six factors: 1) professional behavior; 2) effective communication; 3) recognition of medical terminology; 4) nursing actions before administering medicine; 5) nursing actions while administering medicine; and 6) nursing actions after administering medicine. These factors explained 77% of the total variance of the concept of assessment of the clinical pharmacology unit. In this study, reliability was demonstrated by a Cronbach's alpha coefficient of 0.96; the correlation coefficient between floors for the total tool was 0.91, ranging from 0.64 to 0.89 in its dimensions.
CONCLUSION: The evaluation tool of the clinical pharmacology unit has an acceptable construct validity and satisfactory reliability and validity. Therefore, it can be used to evaluate the clinical pharmacology unit in the nursing education system in Iran.

Introduction

Owing to being practice based, as well as developments with regard to the information and caring techniques needed by nurses, education in nursing is being considered now more than ever.1 Improving the quality of students’ performance in the area of education is one of the most important objectives for educators. More than half of a nursing training episode is dedicated to clinical education. During this period, students learn and improve upon clinical training in real situations, to prepare themselves for work in a clinical environment.2

Clinical care can be regarded as being of a high quality when the assessment of nursing skills is highly favorable.3 Clinical education has always been an important part of nursing education4 and faces many challenges.5 In general, it can be said that the training received by nursing students is far from the reality of their professional goals. Students are often not familiar with the expectations of others and the evaluation processes.6

This hinders familiarity with the execution of the nursing process, leading to lack of experience in undertaking correct clinical nursing procedures. It is also important to note that human life depends on nursing services.7 Nursing students often have difficulty engaging in theoretical discussions and providing quality care.

This impedes their ability to practice and apply the principles of basic training in clinical education, which means that they cannot integrate what they have theoretically learned in clinical environments. Such application is outlined in the evaluation criteria. Evaluation is one of the important elements of nursing education. By observing evaluation principles, the deficiencies and problems of nursing education programs can be determined.8 For this reason, a clinical skills evaluation form for students, which is based on their educational goals and measures, seems very important.9

Clinical evaluation could be likened to the compilation of multiple images of the moment of clinical practice for students. The greater the number of images produced, the greater the understanding of the results of the subject of evaluation. Challenges and frustration have always formed part of evaluation, demonstrated by the fact that 41% of nursing and midwifery students have complained about the evaluation process. A majority of students believe that clinical evaluation cannot identify students’ knowledge of theory and practice.10 In relation to this, instructors are seeking a reliable and valid tool for the clinical evaluation of students.11

The clinical pharmacology unit was included in the nursing education curriculum in the 2014–2015 academic year. Along with the theory of nursing pharmacology in the second semester of nursing degree programs, this course is offered with the aim of creating a platform for the application of theoretical knowledge in the pharmacology field in real settings.

Owing to the lack of tools measuring students’ knowledge and application in practical units, a need was felt to develop a tool for this purpose. Therefore, this study aimed to develop a tool for the evaluation of the clinical pharmacology unit. Since students’ assessment is known to be one of the important aspects of clinical nursing education, the development and validation of a tool can give a better understanding of students’ strength in both theoretical and practical knowledge.

Methods

The present study used methodological research and was conducted in 2016 at Babol University of Medical Sciences.

Designing the questionnaire

At this stage, the development and validation of the tool were based on the stages proposed by Schwab.12

Development of items

This stage entailed the development of unique items. To develop the clinical pharmacology unit’s evaluation tool, a search was conducted on different databases, without a time limit, with the keywords “psychometric”, “tools”, “clinical education”, “pharmacology training”, and “nursing students”. The assessment tools found were in the area of the nursing management unit and other training units; no tools related to the clinical pharmacology unit were found. The review of articles indicated areas that needed to be addressed and special items that could help the present study. In addition to this, individual interviews were conducted with eight faculty members, to aid tool development.

Development of tools

At this stage, the researchers drafted the items. Then, a pool of items was selected (74 items were considered suitable for the item pool). Furthermore, items that could enable structural measuring were selected. At this stage, problems related to the design and structure of the tool were identified and corrected using a pre-test. During the pre-test of the items, the nature of the tool was tested, to ensure consistency with the goal.

The pre-test was conducted by having two experts in psychometrics and nursing education read the items a few times and review them. In the pre-test, the sample should ideally be representative of the population, comprising at least 300 subjects.13 In the pre-test, unclear items or those that are poorly worded were identified and corrected. Seventy items formed the primary pool of questions. Then, in the pretest, six items were removed because they were not consistent with the objectives of the study; 64 items remained.

Tool evaluation

At this stage, the tool was validated and the items analyzed. To validate the newly developed tool, face validity, content validity, and construct validity were measured.

Face validity

Face validity is determined using qualitative and quantitative methods. In terms of quality, face validity is concerned with the relevance of the tool, based on what it should seemingly determine. In terms of quantity, an impact score is used to determine face validity.14 In this study, to determine qualitative face validity, 10 nursing students were asked to comment about items’ level of difficulty and obscurity and the proportion of each item. Quantitative validity was determined by obtaining the impact score through surveys with the 10 students; the score indicated the importance of each of the questionnaire items. The subjects were asked to indicate the importance of each item on a 5-point Likert scale (from “very important” to “unimportant”).

Content validity

Content validity is the evaluation of the content of a test, considering the construct that it is supposed to measure.15 To determine content validity, qualitative and quantitative methods were used. The qualitative method entailed asking 10 experts in the field of tool making about the simplicity, clarity, and appropriateness of items, based on their location on the questionnaire. To evaluate content validity in a quantitative manner, a content validity ratio and an index were used. Then the experts were asked to give their opinions regarding the necessity and relevance of each item. To examine the validity of the two indices, the content validity ratio and the content validity index were used.

Content validity ratio

In this study, Lawshe’s model (1975) was used to determine content validity. At first, the tool was given to a panel of experts, to discuss its necessity. The experts’ answers were encoded as urgent, useful but non-essential, and non-essential. Then, the panel members’ votes were quantified through content validity. The values of the content validity equation ranged from -1 to +1.16 In this study, seven experienced professionals in the field of tool development were contacted by telephone or email. Then, the questionnaire was presented to them, so that they could assess the validity of the content. Items that were assigned scores of >0.42 by the experts of CVR assessment were retained as meaningful (p < 0.05).

Content validity index

The content validity index is the ratio of experts’ agreement about the relevance of each item, that is, the number of professionals who have assigned a score of 3 or 4 to each item divided by the total number of professionals. This essentially refers to the ratio of agreement on the relevance of each item.17 In this study, seven experienced professionals in the field of tool development were contacted by telephone or email. The tool was then sent to these professionals. According to Polit and Beck,18 a content validity index of 0.72 for five experts is good and that of 0.78 for six or more specialists is ideal.

Construct validity

Exploratory factor analysis was used to determine construct validity. In exploratory factor analysis, the researcher has no particular expectations about the number and nature of factors. Principal component analysis with orthogonal rotation is the most commonly used method to find factors. Eigenvalues that are greater than one standard deviation were used to extract the factors. In most studies, the appropriate factor loading has been set as 0.4. Nevertheless, items with a factor loading of at least 0.3 are maintained. Before factor extraction, to ensure that the tool’s items meet assumptions for the analysis of the main components, the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were conducted. The least proposed value of the KMO test was 0.6. Bartlett’s test of sphericity was especially used to determine if the correlations between test items equal zero.13

Reliability

Two methods, namely, internal consistency and test-retest reliability, were used to determine the questionnaire’s reliability. Internal consistency measures the extent to which individuals give responses that are stable over time. Cronbach’s alpha, with values ranging from 0 to 1, was used to assess internal consistency. For this purpose, 20 students were evaluated by their instructors during training, and three weeks later, the evaluation tool was completed on their behalf. Cronbach’s alpha was calculated for the overall utility and every aspect. In this study, 20 students were asked to complete the questionnaire on two occasions, to enable evaluation of consistency after 2 weeks and, therefore, completion of tool development. Then, intraclass correlation (ICC) was calculated for all domains and the whole questionnaire. The minimum acceptable value of ICC is 0.4.19

Study participants

In total, 264 second- and third-semester nursing students of the Islamic Azad University of Babol University of Medical Sciences participated in the study. The inclusion criteria included being a student and the willingness to participate in the study. For sampling, a census method was used in this study.

Moreover, the minimum sample size considered was five for each item. According to researchers, the sample size required for factor analysis, in order to determine construct validity, differs. The sample size recommended is five to ten samples for each tool.20

To collect data in this study, a form containing personal information related to age, gender, and level of education was used.

Ethical considerations

This study was approved by the ethics committee of the University of Medical Sciences of Babol (approval number: 3388). All participants verbally gave informed consent to participate in the study. The confidentiality of individuals’ personal information was considered at all stages.

Results

The average age of the students was 20.23 ± 0.2 years. Further, 69.3% were female and 65.2% were in the second academic semester. Seventy-five items were assessed for validity. In examining the face validity, 11 options were removed, due to an impact score of <1.5. The number of items was subsequently reduced from 64 to 60, and the 60 items were included in the next step. To determine the content validity ratio, the views of experts in the field of tool development were considered. According to Lawshe’s table (1975), if the content validity value ranges from 0 to 0.62, then items with a numerical average of ≥0.5 are to be maintained. Two options were subsequently removed. Then, a 58-item tool was included in the next stage, which determined the content validity index. In this study, the minimum acceptable value of content validity was 0.72. Therefore, three items with a content validity index of <0.72 were removed, and 55 items were included in the heuristic content analysis stage. The KMO index in this study was 0.896. This indicated that there were sufficient data for analysis. Bartlett’s test of sphericity, with 12906.244, was also significant (p < 0.000). This indicates that the correlation between the items is sufficient and warrants factor analysis. Then, for determining the number of tool-producing factors, a scree plot and eigenvalues were used. The scree-plot graph showed that six factors were enough to explain the factors of the assessment tool for the clinical pharmacology unit. A 5-point Likert scale (“always” to “never”) was used for scoring. The scores ranged from 55 to 275.

Next, the tool’s factor structure was extracted through principal component analysis, with orthogonal rotation and varimax rotation (eigenvalue < 1). The result of principal component analysis was a factor matrix in which a factor loading for each item was determined separately. In this matrix, items that were highly correlated with each other were placed within one category or factor. In this study, the minimum factor loading for each item in the factor matrix and the rotated matrix was 0.3. In general, six factors were determined for the questionnaire, explaining 77% of the variance. After the extraction of factors, each was named based on the corresponding items and the extent to which these factors and concepts matched each other. Moreover, aspects of the evaluation of the clinical pharmacology unit that were determined in this study were evaluated (Table 1).

Table 1

Factors extracted from factor analysis using varimax rotation and factor loadings of their items

Number of item	Items	Factor loading
		Factors
		1	2	3	4	5	6
	Professional behavior
1	Follows Islamic rules	0.564
2	Respects the rules and policies of the department	0.570
3	Respects personal hygiene (uniforms, shoes, and nails)	0.580
4	Is present at the ward and leaves it on time	0.573
5	Leaves the ward by informing the trainer and respects the remaining time	0.581
6	Knows the rules and follows them	0.550
7	Feels responsibility for the educational needs of the patient and his family	0.606
8	Feels responsibility for performance of educational duties	0.577
	Effective communication
9	Deals with patients and their families with friendliness and courtesy and respects them		0.620
10	Uses appropriate ways to communicate with staff		0.618
11	Creating opportunities to express feelings, ask questions, and administer medication, to reduce anxiety		0.621
12	Respects the instructors and other students and cooperates with them when necessary		0.632
13	Uses clear sentences when educating patients		0.634
14	Accepts the logical comments and critique of instructors and officials and cheerfully accepts and uses their guidance		0.625
15	Stops repeating behaviors that have been criticized		0.612
16	Adjusts to the new environment and controls one’s emotions		0.597
	Understanding medical terminology
17	Knows key terms used in the medication process			0.478
18	Reads doctor’s instructions correctly from patient records			0.660
19	Reads the medication orders in the report correctly			0.742
20	Is able to read drug labels, such as name, size, and amount of drug			0.470
	Nursing care before drug administration
21	Reads drug use history from pharmaceutical package correctly				0.629
22	Is aware of ways of maintaining medication in pharmaceutical package				0.742
23	Is aware of non-use of medication without name and labels				0.754
24	Is aware of various methods of drug administration				0.687
25	To prepare the medication, washes hands properly and correctly				0.725
26	Observes sterilization tips in opening the syringe				0.750
27	Observes sterilization tips on how to use syringe				0.767
28	Has general knowledge in the field of medicine tools such as the GERS				0.745
29	Is aware of the cleanliness of tools while picking them				0.746
30	Diligent drug maintenance				0.755
31	Works on the correct maintenance of medication (light, heat, sterility, and dissolution of color drug)				0.691
32	Can take actions to prevent the use of drugs by children or disabled patients				0.738
33	Knows the maintenance procedures for narcotics				0.660
34	Is aware of the use of similar drugs or placebos				0.674
35	Knows risky drugs such as potassium chloride, digoxin, calcium, and insulin				0.784
36	Knows the drugs that should not be given intravenously				0.634
	Nursing care during drug administration
37	Knows the correct time for provision of medicines					0.512
38	Offers the right medication to the patient					0.635
39	Offers medication to the right patient					0.683
40	Knows correct procedures for administration of medications					0.632
41	Administers the correct dose of the drug to the patient					0.666
42	Records the medicine report correctly					0.648
43	Is able to detect caution points in drug consumption					0.517
45	Is able to calculate the correct time of infusion through a micro set					0.677
46	Is able to detect drug interactions with certain foods					0.522
47	Knows the drugs that are prescribed as gavage					0.542
	Nursing care after drug administration
48	Can identify drug effects						0.712
49	Has sufficient knowledge of the positioning of patients after taking drugs						0.652
50	Has sufficient knowledge of laboratory changes caused by use of medicine						0.678
51	Is able to detect drug interactions						0.789
52	Knows the right conditions to maintain the drug dissolved and the ambient temperature, and knows the date						0.583
53	Pays attention to the things that should be included on the label of the dissolved drug, including, for example, a clean glass medicinal syrup, medication opening date, and the patient name						0.632
54	Gives innovative solutions to the patient and the family along the way, with regard to maintaining medicine						0.564
55	Gives solutions to the patient and family members regarding the recollection of medication guidelines						0.591

Abbreviation: GERS, Global Expense Reporting Solutions.

First factor

This factor has eight items and refers to items related to Islamic norms, ethics, punctuality, and accountability. In this factor, the greatest factor loading was for the item “feels responsibility towards the educational needs of the patient and family” and the lowest factor loading was related to the item “is familiar with the laws and regulations of the ward and respects them”. The proportion of variance calculated for this factor before rotation was 41.694 and after rotation was 24.762.

Second factor

This factor has eight items and notes points related to effective communication. In this factor, the greatest factor loading was for the item “in patient education uses clear sentences”. The lowest factor loading was related to the item “adjusts to the environment and new situations and controls one’s emotions”. The proportion of variance calculated for this factor before rotation was 18.647 and after rotation was 23.033.

Third factor

This factor includes four items and refers to issues related to familiarity with medical terminology. In this factor, the greatest factor loading was for the item “reads the medication orders in the medical instruction correctly” and the lowest factor loading was related to the item “is able to read the label drugs such as name, size and amount of drug”. The proportion of variance calculated for this factor before rotation was 7.698 and after rotation was 7.233.

Fourth factor

This factor consists of 16 items and is concerned with nursing care prior to drug administration. In this factor, the greatest factor loading was for the item “knows high-risk drugs such as potassium chloride, digoxin, calcium, and insulin” and the lowest factor loading was related to the item “reads the drug and uses the expiration date from the medicine pack appropriately”. The proportion of variance calculated for this factor before rotation was 3.856 and after rotation was 4.950.

Fifth factor

This factor contains 11 items and is concerned with nursing actions during drug administration. In this factor, the greatest factor loading was for the item “is able to calculate the correct time of infusion through the micro set” and the lowest factor loading was related to the item “provides medication at the right time”. The proportion of variance calculated for this factor before rotation was 2.995 and after rotation was 3.736.

Sixth factor

This factor has eight items and is concerned with nursing care after medication. In this factor, the greatest factor loading was for the item “is able to detect drug interactions” and the lowest factor loading was related to the item “knows the right conditions to maintain the dissolved drug regarding environment temperature and the use date”. The proportion of variance calculated for this factor before rotation was 2.423 and after rotation was 3.554.

In this study, after determining construct validity, Cronbach’s alpha coefficient was calculated using a sample of 264 nursing students. A value of 0.96 was obtained, indicating that the questionnaire has good internal consistency. Then, Cronbach’s alpha was calculated for each factor. To determine the stability of the questionnaire in relation to reproducibility, ICC was calculated for all aspects. Given that the ICC of the tool is 0.91 and that of aspects ranges from 0.64 to 0.89, the reliability of the questionnaire is satisfactory (Tables 2 and 3).

Table 2

The results of the ICC coefficient (n = 20)

Factor	ICC	95% confidence interval	p value
The overall tool	0.910	00.814–0.971	>0.001
Aspects
Professional behavior	0.818	0.868–0.805	>0.001
Effective communication	0.890	0.898–0.675	>0.001
Understanding medical terminology	0.801	0.828–0.775	>0.001
Nursing before drug administration	0.704	0.728–0.679	>0.001
Nursing procedures during medication	0.640	0.509–0.711	>0.001
Nursing interventions after medication	0.732	0.745–0.709	>0.001

Abbreviation: ICC, intraclass correlation.

Table 3

Results of internal consistency reliability (Cronbach’s alpha; n = 264)

Factor	Number of items	Cronbach’s alpha
The overall tool	55	0.96
Aspects
Professional behavior	8	0.86
Effective communication	8	0.75
Understanding medical terminology	4	0.68
Nursing before drug administration	16	0.80
Nursing procedures during medication administration	11	0.82
Nursing interventions after medication administration	8	0.79

Discussion

Evaluation is a process during which the skills and activities of students are evaluated. During this process, the strengths, weaknesses, and opportunities for improvement of skills and development are identified.21 The development of evaluation tools for the clinical pharmacology unit could highlight important issues. The present study presents the stages of the development and validation of the evaluation tool for the clinical pharmacology unit. The findings ultimately show that the instrument has satisfactory psychometric properties. Therefore, this tool can be used to evaluate the clinical pharmacology unit in the nursing education system. The tool’s items are scored using a 5-point Likert scale (“always” to “never”). The scores ranged from 55 to 275. In this study, in addition to the assessment of the quality of the content by panel members, content validity and the content validity index were calculated, to assess the tool’s content validity, which ultimately led to the removal of five items from the questionnaire. According to the results of exploratory factor analysis, the KMO index was found to be 0.896, and the higher it is, the better the factor analysis. In this study, given that the KMO value was higher than 0.80, it was considered good and the results were deemed favorable.22 Therefore, based on the results, the tool was classified into six domains or factors. Therefore, it seems that satisfactory results were as a result of careful selection of appropriate statements for the clinical pharmacology evaluation tool. With regard to the tool’s reliability, it should be noted that a reliable tool can increase the power of the study in the determination of associations and significant differences. In other words, reliability refers to an instrument’s stability. However, it should be noted that the reliability of a tool is closely related to its validity. In this study, in addition to appropriate calculation of the tool’s reliability separately, the final internal consistency of the tool was demonstrated by a Cronbach’s alpha coefficient of 0.96. Since an alpha coefficient of 30.7 is considered appropriate,23 the Cronbach’s alpha coefficient obtained showed the tool’s high internal consistency.

The tool also showed good reliability. The value of ICC coefficient was 0.91, which indicates good stability of the clinical pharmacology evaluation tool. In this regard, an ICC of ≥0.4 is considered satisfactory.24

The final version of the extracted tool contained 55 items and six factors, namely, professional behavior, effective communication, knowledge of medical terminology, nursing procedures before administering medicine, nursing procedures during medication administration, and nursing care after administering medication. Six domains accounted for 77% of the cumulative variance, with minimum eigenvalues. In the domain of professional behavior, aspects such as respecting regulations, timely performance of tasks, ethics, and the rights of clients formed part of self-promotional activities. In the study by Pazargadi et al,21 the professional behavior domain had 14 items. In the evaluation tool of the University of Manitoba, which included two main domains, 12 sub-domains, and 55 phrases, this point constitutes one of the main domains known as professionalism and accountability, in turn consisting of three sub-domains and 22 phrases.25

The effective communication aspect entailed items related to student’s interaction with patients, teachers, staff, and other students. In other studies, effective communication is one of the important factors considered in the evaluation of students. The provision of safe and quality care is dependent on nursing students’ ability to assess patients’ needs through the nursing process.26

Nursing care before administering medicine entails items such as having enough knowledge and respecting essential points in the administration of medication to patients. Preparation by nurses is essential for the provision of effective and quality care to patients.27

In nursing care during administration of medicine, emphasis is on points such as respecting six principles pertaining to the provision of drugs to patients (e.g., identifying drug interactions). Nurses’ attention at the time of administering drugs to patients is important for ensuring patient safety and yields positive outcomes of a treatment regimen.28

Nursing care after drug administration entails issues such as creativity and innovation. During clinical evaluation, teachers should refer to activities such as critical thinking in clinical settings. In addition, they should ensure that students can apply critical thinking in clinical settings.29

Knowing the side effects of drugs and their interactions are considered important points in the assessment of the clinical pharmacology unit. Lack of timely understanding of complications due to medication and ignoring laboratory changes associated with the diet treatment can threaten patient safety and disrupt quality care.

In general, it can be said that the clinical pharmacology unit evaluation tool is multi-dimensional and that the best tool for student evaluation is one that addresses various aspects. The development and validation of the clinical pharmacology assessment tool can be used to analyze the clinical operation of nursing students in educational systems.

Conclusion

In educational systems, the evaluation of students is an important measure of skill acquisition and provision of quality care to patients. In this study, along with the development of a valid tool to evaluate the clinical pharmacology unit, an attempt was made to assure the reader about the tool, by providing sufficient information about the process of evaluating the tool’s validity and reliability and the quality of its evaluation. For the first time in Iran, the development of this tool was based on the psychometric process and the views of nursing students and an impressive number and variety of specialists. The maintenance of simplicity and eloquence and consideration of brevity and the logical sequence of items are presumed to be the positive aspects of this tool. In this study, the tool has an acceptable factor structure and satisfactory reliability and validity. Therefore, it can be used in similar studies, in related topics, and with study populations in educational systems.