Portfolio as a tool to evaluate clinical competences of traumatology in medical students.

This article investigates whether a reflexive portfolio is instrumental in determining the level of acquisition of clinical competences in traumatology, a subject in the 5th year of the degree of medicine. A total of 131 students used the portfolio during their clinical rotation of traumatology. The students' portfolios were blind evaluated by four professors who annotated the existence (yes/no) of 23 learning outcomes. The reliability of the portfolio was moderate, according to the kappa index (0.48), but the evaluation scores between evaluators were very similar. Considering the mean percentage, 59.8% of the students obtained all the competences established and only 13 of the 23 learning outcomes (56.5%) were fulfilled by >50% of the students. Our study suggests that the portfolio may be an important tool to quantitatively analyze the acquisition of traumatology competences of medical students, thus allowing the implementation of methods to improve its teaching.

Introduction

Since 2008, all medical schools in Spain have implemented new medical curricula that follow the requirements of the so-called Bologna process. These plans encourage a shift from a professor-centered classical scheme related mostly to the acquisition of theoretical knowledge to a competence-based process, in which competences form the basis of the teaching–learning equation.1,2

Competences are not easy to evaluate3,4 with traditional tools because these are not often designed to determine whether a competence has been acquired by students5 or designed to provide formative evaluation.6 One of the new tools used for this purpose is the portfolio,7 considered as a method that allows continuous and formative assessment, bringing critical reflexion8–11 and autoregulation.12,13 Many authors have described the benefits of using portfolios in medical education,14–18 but limitations regarding their use19–21 and reliability and validity have also been reported.22,23

Although portfolios have been used in different clinical medical subject areas,24–28 to the best of our knowledge, there are no reports on the reliability of portfolio to assess the acquisition of competences in surgical areas or subjects such as traumatology. Thus, in the current article, we report the application of a portfolio tool to assess the level of acquisition of clinical skills and competences in medical students of the Medical School of the University of Murcia (Spain), during the clinical rotations of traumatology and orthopedic surgery. An important part of the study is also devoted to the analysis of the reliability of the portfolio.29

Methods

A portfolio was designed to assess the level of acquisition of clinical competences, according to the competences and learning outcomes assigned to traumatology in the new study plan of the degree of medicine of the Medical School of the University of Murcia. The list of competences and the learning outcomes are shown in Table 1. This study was approved by the ethics committee of the Universidad de Murcia and written informed consent was obtained from all participants.

Table 1

Frequency and percentage of achievement of clinical competences

Competences	Learning outcomes	Frequency	Percentage
1. Recognize injuries, assessment, and consequences	1.1 Diagnosis of fracture	112	85.3
	1.2 Diagnosis of muscle lesions	93	71.3
	1.3 Assessment of postoperative patients	115	87.6
	Competence 1: average percentage ± SD	81.4±8.9
2. Identify lesions during physical examination (supervised)	2.1 Assessment of omalgia	56	42.7
	2.2 Assessment of gonalgia/coxalgia	95	72.8
	2.3 Assessment of cervical/back pain	45	34.4
	2.4 Examination of foot	70	53.3
	2.5 Examination of hand	72	55.1
	2.6 Examination of backbone	30	23.3
	2.7 Importance of examination	63	47.9
	Competence 2: average percentage ± SD	47.1±15.9
3. Recognize lesions through image techniques	3.1 Reading radiographies	111	84.9
	3.2 Reading magnetic resonance images	69	52.9
	Competence 3: average percentage ± SD	68.9±22.6
4. Orthopedic treatment of lesions (nonsurgical, supervised)	4.1 Evaluation of immobilizations	67	51.5
	4.2 Healing wounds	61	46.7
	4.3 Reduction of dislocations/fractures	56	43.1
	4.4 Perform articular infiltration (seen or performed under supervision)	85	64.7
	Competence 4: average percentage ± SD	51.5±9.5
5. Establish a treatment plan	5.1 Visit of the doctor to the hospitalized patients of traumatology and orthopedic surgery	114	87.4
	Competence 5: average percentage ± SD	–	–
6. Observe and assist in surgical treatments	6.1 Surgical washing (at least once)	47	35.7
	6.2 Suture of wounds in emergency unit (supervised)	42	32.4
	6.3 Attending operating room	130	99.2
	Competence 6: average percentage ± SD	55.8±37.6
7. Write reports to patients, families, and other professionals (supervised)	7.1 Report of patient discharge: fracture	8	6.3
	7.2 Report of patient discharge: orthopedics	9	6.7
	7.3 Importance of doctor–patient relationship	85	65, 1
	Competence 7: average percentage ± SD	26.0±33.8
	Global average percentage		59.7

Abbreviation: SD, standard deviation.

Academic structure

Traumatology is a compulsory subject in the 5th year of the degree, consisting of 55 hours of theoretical classes, of which 28 hours take place in the classroom, 14 hours are of a seminar type, 10 hours are about clinical cases, and 3 hours are about cooperative learning; 21 hours of classes on laboratory skills in seven sessions; and 2 weeks of hospital rotation (50 hours, 10 days). Of these 50 hours, students rotate in the traumatology service of the university hospitals between hospital ward, medical consultation, operating rooms, and emergency services. Only one student is assigned to one professor in the hospital. Table 2 shows the number of days students spent during these rotations.

Table 2

Days spent by students during the traumatology rotation

Days in	Range	Sum	Mean	SD
Operating room	1–7	335	2.6	1.3
Consultation	1–6	445	3.4	0.9
Hospital ward	1–6	179	1.4	0.9
Emergencies	1–5	235	1.8	0.9

Abbreviation: SD, standard deviation.

Portfolio structure

The portfolio was given to all the students during the course (n=131) in a booklet format. After the identification of data, the students read about legal and ethical issues in relation to the need for keeping medical secret and signed conformity. Each day the students were expected to fill up their reflections before leaving the hospital, writing about the clinical activities observed during the day. In addition, the portfolio had to be signed by the assigned professor every day. The students were given enough space to include everything needed, but there were three main paragraphs:

Annotate what you have seen today.

Comment the issues you have observed during the day (patient care, clinical signs, pathologies, diagnostic obtained, treatments performed, communication with patient, and family).

Write up the most important thing you have learned today.

Portfolio evaluation

The portfolios were evaluated by four professors, three of them professors of traumatology and one a colleague professor from a different area, that is, physiology; all of them were medical doctors. A checklist with 23 indicators was built to help evaluators to assign the activities written up by the students to a learning outcome. These learning outcomes were selected from a total of ten competences of the matter. All the portfolios underwent blind evaluation. All the students agreed and participated enthusiastically in the study.

Statistical analysis

Data analysis was performed using the Statistical Package for the Social Sciences, Version 19. The Cohen’s kappa coefficient was used as a statistical measure of interevaluator agreement for qualitative items.29 Frequencies, percentages, means, and standard deviations were also obtained to describe the results.

Results

A total of 138 comparisons were made to obtain the kappa index among the four evaluators for each of the 23 learning outcomes. The distribution of frequencies and percentages of agreement, according to the classification of Landis and Koch,29 are shown in Table 3. As it can be seen, out of 138 comparisons, 83 had kappa indexes >0.4 and 44 of them had values >0.6, which indicates that in a 31.9% of the occasions, the agreements between evaluators have been very substantial.

Table 3

Distribution of frequencies and percentages of kappa values in every category

Kappa	Agreement	Frequency	Percentage
<0.00	Poor	0	0
0.00–0.20	Slight	14	10.2
0.21–0.40	Fair	41	29.7
0.41–0.60	Moderate	39	28.3
0.61–0.80	Substantial	35	25.4
0.81–1.00	Almost perfect	9	6.5

Table 4 shows the mean and standard deviation of the kappa index of the four evaluators. It can be seen that the global kappa is 0.48, indicating that the portfolio has a moderate reliability, according to Landis and Koch.29 As an additional way of assessing the reliability of the portfolio, 6 months after the initial evaluation, two of the evaluators revised again the portfolios (second revision or reference). Table 5 shows the level of agreement obtained by each evaluator when both revisions were analyzed. Globally, the differences among evaluators are not very important because kappa indexes show that evaluators 1 and 4 have a substantial agreement in both revisions, whereas evaluators 2 and 3 have a moderate level of agreement.

Table 4

Mean of kappa indexes of the four evaluators

	n	Mean kappa	Standard deviation
Competence 1	3	0.38	0.12
Competence 2	7	0.47	0.09
Competence 3	2	0.50	0.17
Competence 4	4	0.48	0.14
Competence 5	1	0.28	–
Competence 6	4	0.65	0.30
Competence 7	3	0.36	0.09
Global kappa	23	0.48	0.17

Table 5

Agreement, obtained as the kappa index, between the original evaluator and the result obtained in the second revision (reference)

	C1	C2	C3	C4	C5	C6	C7	Global
Evaluator 1	0.52	0.62	0.46	0.52	0.49	0.83	0.69	0.60
Evaluator 2	0.38	0.52	0.66	0.50	0.63	0.83	0.48	0.55
Evaluator 3	0.19	0.50	0.63	0.50	0.63	0.87	0.54	0.53
Evaluator 4	0.51	0.62	0.46	0.52	0.49	0.83	0.63	0.60

Abbreviations: C1, competence 1; C2, competence 2; C3, competence 3; C4, competence 4; C5, competence 5; C6, competence 6; C7, competence 7.

The level of achievement of the traumatology competences is shown in Table 1. The data show the mean percentage of the four evaluations as well as the number of times the procedures were performed. A total of 59.8% of the students have acquired the competences as expected during their hospital stay. In all, 13 of the 23 learning outcomes (56.5%) have been acquired by >50% of the students. Competence 5 (87.4%) and competence 1 (81.4%) show the greater values, whereas the rest of them show an acceptable level, except competence 7, showing only a value of 26%.

Discussion

The current article shows the results obtained with the application of a portfolio to medical students as a tool to obtain information regarding the level of acquisition of the clinical competences of the topic traumatology. One of the important aspects of any portfolio is its reliability, the consistency and accuracy of the assessment tool in measuring students’ performance. Our results quite agree with those found by other authors;30,31 although in some studies of nurse students, higher numbers (0.8) have been described.32 To prove that the results were not obtained by chance, the second review of the portfolios took place 6 months after the initial evaluation in order to see whether the reliability index could be improved. As observed in Table 5, the maximum level of agreement among evaluators is found in competence 6 (observe and assist in surgical treatments). A moderate agreement exists in competences 2, 3, and 4, whereas competences 1, 5, and 7 show a low level of agreement among evaluators.

The results of Table 1 show important information regarding the level of acquisition of the competences analyzed. Two of them, competences 1 and 5, related to diagnostic and treatment obtained the greatest value, with an important frequency of occurrence. Next competence in terms of the level of acquisition is number 3 (70%), probably due to the high number of radiographic images viewed in contrast to magnetic resonance images, where it is likely that doctors rely more on the radiologist report, without actually interpreting them. Competence 6 has also a moderate level of acquisition (55.8%), in spite of the fact that almost all students (99.2%) report attendance to operating rooms (2.6 + 1.3 days). This is likely due to the fact that the surgeons do not invite all the students to wash and be prepared to help, an activity we recommend to all our professors. A similar reason may be behind the moderate level achieved in competences 2 (47.1%) and 4 (51.5%), showing that not all the students are able to explore or perform maneuvers with the patients and that this may be due to a time limitation problem while in the ward or consultation. We believe that this should be corrected and doctors should allow more time for the students to perform, under the appropriate supervision, exploration, or treatment of the patients. Finally, competence 7 has the lowest level. It is likely related to the fact that students are not allowed to sign discharge reports or other forms, although it is important that they learn from their professors how to do it. In any case, this is the first time that we obtained this quantitative information regarding the clinical stays of the medical students and this will help us to analyze our teaching results and to develop new strategies to enhance the level of clinical competence the students should reach.

A problem learned from the application of this portfolio is that many of the learning outcomes were checked without taking into account the number of times these were carried out. The rest of the learning outcomes were quantitative, so we could obtain the number of times the student performed them. For instance, they attended the operating rooms 5.5 + 3.3 times (range, 1–26), and viewed a mean of 2.3 magnetic resonance images (range, 1–14). We believe that it is important that the portfolios are designed to collect also the quantitative information, since both quantitative and qualitative information give the real essence of the learning process. A drawback we have seen with the application of the portfolio to clinical students is that there is a random factor affecting the acquisition of the competences. This is due to the fact that the students rotate between different hospitals and at different times of the year. Thus, it depends on the type of medical problems available on the days they are assigned to the hospital; thus they may not see all the problems selected in the portfolio. Clearly, the greater the number of days of hospital stay, the better it is. However, this is not always possible. A better selection of the learning outcomes will also be helpful. Moreover, the portfolio is designed to be an evaluation tool that should be used in conjunction with others (written or oral exams, laboratory evaluations) because it is not possible to find just one tool that is able to assess all the competences.33,34

Conclusion

In conclusion, the portfolio is an important tool that has allowed to obtain qualitative and quantitative data regarding the degree of acquisition of traumatology competences by medical students. Although some factors need to be improved, such as the inclusion of more quantitative elements, a more comprehensive selection of learning outcomes, and combination with other tools, we believe that the use of this kind of tool by clinical teachers will be useful to ascertain the degree of practical competence reached by their students.