The Effect of Portfolio-Based Education and Evaluation on Clinical Competence of Nursing Students: A Pretest-Posttest Quasiexperimental Crossover Study.

PURPOSE: Currently, revisions in education and evaluation in the nursing domain are necessary to increase the clinical competence of graduates. The aim of this study was to compare nursing students' clinical competence using the portfolio method with conventional education and evaluation methods in clinical education.
METHODS: In this pretest-posttest quasiexperimental crossover study, all six-semester undergraduate nursing students of Ahvaz Jundishapur University of Medical Sciences were classified into four groups. They were selected based on the use of portfolio and conventional methods during their apprenticeship. Then, they were educated using conventional and portfolio methods. Their clinical competence was measured by a self-appraisal questionnaire. Data were statistically analyzed with independent and paired t-tests.
RESULTS: There were significant differences between average scores of clinical competence in all three domains before and after study in portfolio-based education method (P=0.0001). Furthermore, the average difference in scores before and after internship between the two educational groups in all three domains - cognitive (P=0.002), affective (P=0.0001), and psychomotor - (P=0001) was significant. Similarly, there was a significant difference in total average scores of clinical competence between the two methods (P=0.0001).
CONCLUSION: The results showed the effect of educational method on increasing the clinical competence of nursing students in a clinical unit. This new method can be used as a combination of education, evaluation and student-centered methods.

Introduction

Nowadays, due to the practical orientation of the nursing profession and the increased information and techniques of nurses, educational considerations in this domain have high importance.1–3 The goal of nursing education as a practical discipline is to increase students’ awareness at the highest level of clinical competence in the domain of nursing care.4–7

Competence in nursing has many definitions. For example, according to the World Health Organization, competence in nursing is defined as a level of performance that represents the effective use of knowledge, skills, and judgment and as the ultimate efficiency of any educational system encompassing a variety of learning domains.8 The UK Nursing and Midwifery Council uses the term “competence” simply to describe the skills and ability to practice safely and effectively without the need for direct supervision.9 Nurse competence is determined through an assessment of the nurse’s personal characteritics, clinical observations, and practical skills in contact with patients and colleagues and written examinations.10

Studies in different countries have indicated that impediments toclinical education can be lack of access to adequate clinical trainers and lack of proper coordination between clinical education in schools. Facilities in hospitals are mong the clinical education problems of students in achieving professional competence.2,11 According to the findings of some studies, nursing students’ clinical competence is moderate and poor.12

Based on studies in Iran, most nursing students believed that they could not learn the skills needed for employment in nursing during the courses.13 Furthermore, Imanipiur and Jalili showed that 79.6% of students believed that current evaluations indicated the personal opinion of teachers versus the clinical competence of students.2

Nowadays, expectations of care service-recipients of service-provider systems with experienced employees and high clinical competence have increased. This issue is important in the domain of health care, particularly in nursing education and management.14,15

In recent years, extensive studies have been performed to modify or replace clinical education methods. These methods have advantages and disadvantages. They require planning and preparation to be executable while study using the different systems affords different educational opportunities and reduces weaknesses related to the separate use of each educational method.16

One of the new combined methods of education and evaluation is the portfolio method, which has been presented in the domain of medical education during the last few years.11 Various studies and investigations on the impacts of the portfolio method on different aspects of nursing students’ education and evaluation have been carried out. For example, McCready and Pennbrant and Nunstedt indicated that clear guidelines for portfolio construction and qualitative assessment are important to competence in practice. The portfolio method can help nursing students to develop understanding of their future profession.17,18 Byrne et al indicated that the portfolio process is a validated method encompassing personal and professional aspects in interactive and multidimensional manner in nursing domains, and may be used for other organizations based on their own portfolio models.19 Buckly et al indicated that portfolios can improve theoretical and practical skills of students in different emotional situations and also their relationships with trainers.20 Halfling et al indicated that portfolios can improve self-awareness of feelings, attitudes, and concerns. Students’ reflections on their consultation skills and the importance of active listening in the patient–doctor relationship were presented.21

In Iran, Latify et al foun thatd critical thinking scores increased in both portfolio and conventional groups after intervention, but progress in critical thinking was lower in the portfolio group.1 Also in Iran, Habibzadeh et al found that students’ understanding using the portfolio method about clinical education was significantly higher than the conventional education method. In other study in Iran,22 Studies by Assadi-Hoveizian et al and Azadi et al in Iran indicated that satisfaction of students using the portfolio method was higher than in the conventional method and recommended it for use as a learning and assessment tool of nursing students in apprenticeship courses.23,24

The portfolio method has some important advantages. It creates a link between theory and the clinical circumstance through reflection on application of theory to clinical circumstance.s Students’ awareness of their strengths, weaknesses, and limitations is increased. It encourages learners to strengthen their modes of learning trend. A sense of responsibility and self-confidence is increased in the learners. With active participation of the educational trainer in the process of evaluation, feedback is presented.25–28

Despite the aforementioned advantages, the use of this method in nursing education is ambiguous, and studies have been incomplete and limited to few aspects of education or evaluation. For example, studies have shown that instructions for constructing the content of the portfolio are not clear and their structure and process still in development at universities and centers of clinical internship.29

The objective of the present study was to evaluate the effectiveness of education with the portfolio method on the clinical competence of nursing students.

Methods

Data Collection

The study was conducted on sixth-semester undergraduate nursing students during the internship course in the Nursing and Midwifery School, Ahvaz Jundishapur University of Medical Sciences. Internship is a course/program for bachelor of science nursing students during the 4 years of their academic career in Iran. During the first 3 years of their education, the students take an internship course in relation to other defined educational courses. For the last year (fourth year) of the program, the students spend the whole year in hospitals and medical centers for practical education and to gain different skills in the nursing domains.30

Sample size was calculated based on mean scores for clinical competence and portfolio (n=29). To prevent the reduction of deleted items, the sample was increased to 37 students. To justify the sample size, we used a crossover design, which is conventional in studies with small samples. Power analysis between the means of the two educational methods after intervention was calculated to be 0.87, which is higher than the suggested suitable range (0.80) for power in statistical analysis.

This was a quasiexperimental study to evaluate the association between an intervention (not randomly assigned) and an outcome. The pretest–posttest quasiexperimental crossover design used in this research compared groups and measured changes between pretest and posttest data among all groups.31

After the methods had been explained to them, 34 of the 37 students filled and signed the written consent. Then, students were divided into four groups of eight, nine, seven, and eight without any researcher interference. Two students who had not provided their first portfolio report were excluded from the study and passed their internship through the conventional method. For elimination, external factors, and the previous educational method, the crossover method was used to examine both educational methods in each group (see Table 1 for classification of studied groups).

Table 1

Classification of Groups Studied Using the Crossover Method

	First unit	Second unit
First group (eight students)	Women’s psychiatry	Adult hematology
Educational method	Conventional	portfolio
Second group (nine students)	Adult hematology	Women’s psychiatry
Educational method	portfolio method	Conventional
Third group (seven students)	Adult hematology	Neurology and neurosurgery
Educational method	Conventional	portfolio
Fourth group	Neurology and neurosurgery	Adult hematology
Educational method	portfolio	Conventional

The length of the study was two periods of internship for 20 days (each 10 days). At the beginning and end of internship programs, two groups of four students were educated in the conventional trend by the university’s coaches for 10 days and the other two educated by the researchers with the portfolio method for 10 days. Then, in the next 10 days, interns that had been educated with the portfolio method were taught by the college instructor with the conventional method. The groups that had been educated with conventional methodology were educated by the researchers with the portfolio method. Students in the portfolio groups were supervised by the same clinical trainer as the conventional group.

In the conventional group, clinical education was provided by the trainer on the first day of the internship in an introduction. Educational goals and course assignments were explained, and then lecture topics associated with that internship course were split between students to present conferences in classrooms in hospital departments. One hour per day was assigned for conference presentation, and then group discussions were organized to get feedback from the trainer. Furthermore, with the help of a trainer, students performed therapeutic procedures and nursing care for patients in the ward, and finally an evaluation was carried out in conventional form. At the end of the internship course, a clinical competence form was given to the students to assess themselves.

In the portfolio group on the first day of internship, a clinical course lecture plan, and a brochure describing the portfolio method was given to the students. This brochure contained information according to the lecture plans to determine their learning needs and educational and internship courses and effects on the practical skills. Additionally, the portfolio was reviewed every week by the researcher and returned to the students after feedback to address deficiencies.

In this type of portfolio, the learner presents his/her abilities and progress, and gives examples of the tasks performed during a certain period for evaluation by the trainer. This collection needed to include the title, contents, objectives, reasons for choosing each goal and how to achieve that goal, feedback from the trainer, self-evaluation at the end of each goal, valid scientific resources, and finally an overall evaluation of the educational course.26

Moreover, they were asked to write all activities and ways to achieve the aforementioned purposes and collect contents of selected topics related to theory according to the needs and objectives and with emphasis on the evaluation of nursing care through reviewing theoretical content presented in past semesters, as well as scientific literature and Internet resources, and present reports of the portfolio to their coaches, with finalized feedback delivered at the end of the first week of the internship.

At the end of the second week, students wrote a report about practical activities relating to their goals, with help from the trainer to improve defects based on the feedback. The nursing process was reported in the form of three cases of nursing care patient during the nursing. On the last day of the internship, all written material, diagnostic and therapeutic tests, and patient-related therapies were integrated in a booklet that included evaluation at the end of each topic, ultimately evaluating the entire course of the internship, whether the course had achieved its goal, and if not, what the reason for this was. They were asked to delivering a list of references to their coaches for evaluation and scoring. Then, a self-evaluation checklist of clinical competence was filled out by the students at the end of each programs and data analyzed. Evaluation was based on this checklist to compare clinical competence between the conventional and portfolio groups.

To improve the internal and external validity of this study, we assessed the groups at the beginning and end of the internship program to compare results and used crossover statistical methods. Clinical competence was calculated and compared before and after the study in each group. We also used an experienced trainer to deliver both conventional and portfolio education methods.

Data Analysis

Demographic data (containing questions about age, sex, semester, the average of the previous semester grades, location of internship, interest in nursing, and possible familiarity with the portfolio method) were collected and a self-assessment questionnaire on clinical competence delivered. This questionnaire was used in Hakimzadeh et al.32 Its content validity had been confirmed by ten professors in the Faculty of Nursing, University of Tarbiat-e-Modarres. Its reliability had been determined with Cronbach’s α of 0.94, which indicates high internal consistency and reliability.

The questionnaire has 31 items in three domains — cognitive (12) affective (ten) and psychomotor (nine) —set up to follow the steps of the nursing process. For score assessment, a Likert scale with five response options was used: very low (0–20%), low (21%–41%), high (41%–60%) and very high (81%–100%). The score of one and five were assigned to very low and very high skills, respectively.32

To teaching the portfolio, the researcher used educational steps based on prior research.2 For evaluation of student portfolios, a checklist developed by Imanipour and Jalili was used.2 For evaluation of the portfolio method, a checklist was provided based on three criteria: portfolio content (eg, knowledge and clinical skills in specialized subjects, ways to improve of behavioral skills, and patient physical and emotional state, social problems, cares, and drug use), portfolio structure (eg, report format), and students’ clinical skills (eg, attendance at clinical and practical workshops and relationships with the medical team and patients).

The data were analyzed using SPSS version 18. Independent t-tests were used to compare average scores before and after the courses. Paired t-tests were used to compare average scores before and after internship. Subjects in the conventional and portfolio groups numbered 34 and 32, respectively.

Results

Demographic variables and baseline characteristics of the students are presented in Table 2.

Table 2

Demographic Variables and Baseline Characteristics of the Students A Majority of Students (20 of 34), Considering that they had Already Completed Four Courses of Internship, did not Satisfy the Current Procedure in the Internship Program, and Most, ie, 30 (88.2%), were not Familiar with the Portfolio-Based Training Method

Age (%)		Sex (%)		Interest in Nursing (%)		Average Scores of Students in the Previous Semester (%)		Satisfaction Rate During the Internship Program (%)		Familiarity with the Portfolio Method (%)
20–22 years	Other	Female	Male	Interested	Not interested	≤17	>17	Satisfied	Dissatisfied	Not familiar	Familiar
43.9	56.1	79.4	20.6	70.5	29.5	33.3	66.7	58.8	41.2	88.2	11.8

Calculation of the clinical competence of students and comparison of differences between before and after study using the conventional and portfolio methods indicated that there were significant differences in all three domains: cognitive (P=0.0001), affective and psychomotor (P=0.0001), and clinical competence overall (P=0.0001; Tables 3–6). Scoresbefore and after internship also showed a significant differences in all three domains — cognitive (P=0.002), affective (P=0.0001), and psychomotor (P=0.0001) — and clinical competence overall (P=0.0001; Tables 3–table 6, table 4, table 5). Therefore, implementation of the portfolioprogram and evaluation procedures affected the clinical competenceof the students.

Table 3

Mean Scores of Clinical Competence Before and After Study using Conventional and Portfolio-Education Methods in the Psychomotor Domain. The Difference in Mean Scores for Clinical Competence in the Psychomotor Domain Before and After Internship in the Portfolio Group using Paired T-Tests was Statistically Significant (P=0.0001). Also, Clinical Competence Scores in the Field of Motor Psychology in the Two Groups Before and After the Internship Using Independent T-Tests Showed a Significant Difference (P=0.0001)

Clinical Competence	Before Internship	After Internship	Mean Difference	95% CI		P-value
				Lower	Upper
Conventional	28.8±4.7	28.8±5.4	0.4±5	−2.24	1.43	0.65
portfolio	28.3±4	33.7±4.4	5.5±4.3	−6.88	−3.87	0.0001*
Average difference between the two groups	0.18±5.9	5±6.8	5.1±7.1	2.6	7.7	0.0001*

Note: *Statistically significant.

Table 6

Mean Total Scores for Clinical Competence Before and After Study using Conventional and Portfolio-Education Methods. Mean Total Scores for Clinical Competence in the Portfolio Group Before and After the Internship Were Significantly Different (P=0.0001), but those of the Common Group Before and After the Internship Were Not (P=0.67)

Clinical Competence	Before Internship	After Internship	Mean Difference	95% CI		P-value
				Lower	Upper
Conventional	97.2±15.8	98.5±17.6	1.3±3	−7.62	4.99	0.67
portfolio	96.0±13	117.5±13.0	21.4±2.3	−26.23	−16.70	0.0001*
Average difference between the two groups	1.5±18.4	19.5±22.6	21.0±24.7	12.15	29.97	0.0001*

Note: *Statistically significant.

Table 4

Mean Scores for Clinical Competence Before and After Study Using Conventional and Portfolio-Education Methods in the Cognitive Domain. Mean Scores for Clinical Competence of Portfolio-Group Students in the Cognitive Domain Using Paired T-Tests Were statistically significant (P=0.0001). Also, the Difference in Mean Scores of Clinical Competence of the Two Groups Before and After the Internship Using an independent t-test showed a Significant Difference (P=0.002)

Clinical Competence	Before Internship	After Internship	Mean Difference	95% CI		P-value
				Lower	Upper
Conventional	35.05±6.4	37±7.5	2.06±8.7	−5.20	1.07	0.19
portfolio	35.6±5.5	44.4±5	9.18±4.9	−10.59	−7.05	0.0001*
Average difference between the two groups	0.4±8	7.5±9.8	7.1±11.3	2.93	11.31	0.002*

Note: *Statistically significant.

Table 5

Mean Scores for Clinical Competence Before and After Study Using Conventional and Portfolio-Education Methods in the Affective Domain. The Difference in Mean Scores for Clinical Competence in the Emotional Domain Before and After Internship in the Portfolio Group Using the Paired T-Test was Statistically Significant (P=0.0001). Also, Mean Scores for Clinical Competence in the Emotional Domain Before and After Internship in the Two Groups Showed a Statistically Significant Difference (P=0.0001)

Clinical Competence	Before Internship	After Internship	Mean Difference	95% CCI		P-value
				Lower	Upper
Conventional	33.7±6	32.5±5.9	1.1±1	−10.03	3.35	0.29
portfolio	32±4.6	39.3±4.4	7.2±5.5	−9.18	−5.34	0.0001*
Average difference between the two groups	1.7±1.6.9	7.0±7.3	8.7±8.8	5.55	11.94	0.0001*

Note: *Statistically significant.

Mean clinical competence scores before and after study using the conventional and portfolio programs showed significant differences in all three domains — cognitive (P=0.0001), affective, and psycho-motor (both P=0.0001) — and clinical competence overall (P=0.0001; Tables 3–6). Differences between the two groups before and after education showed significant differences in all three domains — cognitive (P=0.002), affective (P=0.0001), and psychomotor (P=0.0001) — and clinical competence overall (P=0.0001; Tables 3–6). Therefore, implementation of education and evaluation using the portfolio method had an impact on the clinical competence of the students.

Discussion

In this study, the students experienced the portfolio and conventional education methods together, and intragroup comparison was performed. The increased clinical competenc scores after using the portfolio method indicates the effect of this type of education in increasing the clinical competence of nursing students in a clinical unit.

Demographic profiles in the two groups were similar and had no effect on our study. On the other hand, the shift between the two groups in any of the previous internship was carried out for the elimination of the effects of the previous intervention.

Implementation of portfolio education and evaluation methods was effective in terms of the cognitive aspect of students’ clinical competence, consistent with,Valizadeh et al, which concluded that portfolio method promoted cognitive learning of students significantly more than the combined approach.13

Habibzadeh et al concluded that portfolio education enhanced students’ understanding of the clinical environment more than conventional education, consistent with our study.22 Latifi et al investigated the effects of portfolio evaluation on critical thinking of nursing students. Critical thinking scores for their conventional group whigher than the portfolio group, inconsistent with our study. However, the large number of students in each group, too many duties, and their exhaustion may may explain the lack of effect found in their portfolio group.1 In our study, the portfolio method had an effect on the affective dimension of clinical competence (P=0.0001), consistent with Buckley et al, showing that this method enhances theoretical power in combination with clinical circumstances. It also improves students’ relationships with the instructor, self-evaluation, and dealing with affective situations.20

Haffling et al investigated students’ reflections on the pilot portfolio program at a Swedish University and concluded that the portfolio had an effect on affective issues, especially self-awareness of emotions, attitudes, and concerns, and increased interview skills and communication between patient and teacher.21

Implementation of the portfolio method in our study was effective in psychomotor aspects of clinical competence (P=0.0001). In a qualitative study using the content-analysis method, Hekmatpou concluded that portfolio learning enhanced the psychomotor aspect, consistent with our study.26

The impact of the portfolio method on all dimensions of clinical competence indicated that it is effective in improving the clinical competence of students during the internship program. Bahraini et al found that the clinical competence of nurses increased using theportfolio method.11 Similarly, Haffling et al successfully introduced the portfolio method in clinical education that causes to incorporate the cognitive, affective and practical domains in the clinical circumstances.21

The limitation of this study was the intervention of the college’s education in the division of students according to the planning with the relevant trainers, which limited the possibility of a researcher’s change that was used to overcome this defect in the student’s cross-choice approach. However, for justifying the sample size, we used a crossover study design, which is conventional in smaller studies, and power analysis was strong scientifically, though the small sample may have influenced the final results.

Portfolio-based education and evaluation can be used at different scientific and academic levels and lectures for nursing students to provide and improve evaluation, assessment, education, and efficiency in nursing domains. With this method, it will possible to monitor nurses for professional development and scientific management. It can also improve clinical competence for nursing students, and we recommended its design and implementation in the nursing educational system.

The results of a new educational approach, changing the model of education from a deterministic model to one constructivism and ownership of learning from teacher to student, can be used as a basis for further research in the domain of teaching methods. The results of this research can also be exploited for later studies that explore the impact of the portfolio methodology.

Conclusion

Self-assessment of clinical competence of nursing students and student feedback on portfolio-based education and evaluation as a combined approach for internship and externship programs indicated that students identified their strengths and weaknesses. As such, they did not contest their evaluation scores. Furthermore, they also worked harder to obtain better results. Since the portfolio method is the application of a student-centered learning model, it enhances students’ appreciation of clinical circumstances and can be used as a beneficial and effective method to optimize the quality and quantity of clinical education and motivate students for self-directed learning.