INTRODUCTION: In recent years, the use of portfolios as learning and assessment tools has become more widespread across the range of health professions. Whilst a growing body of literature has accompanied these trends, there is no clear collated summary of the evidence for the educational effects of the use of portfolios in undergraduate education. This systematic review is the result of our work to provide such a summary. METHODS: We developed a protocol based on the recommendations of the Best Evidence Medical Education (BEME) collaboration. Citations retrieved by electronic searches of 10 databases were assessed against pre-defined inclusion/exclusion criteria by two independent reviewers and full texts of potentially relevant articles were obtained. Studies were identified for inclusion in the review by examination of full text articles by two independent reviewers. At all stages, discrepancies were resolved by consensus. Data relating to characteristics of the student population, intervention, outcome measures, student design and outcomes were collected using a piloted data extraction form. Each study was assessed against 11 quality indicators designed to provide information about how well it was designed and conducted; and against the Kirkpatrick hierarchy as modified for educational settings. Comparisons between different groups were carried out using the Kruskal-Wallis test (non-parametric ANOVA) or the Mann-Whitney U test as appropriate. RESULTS: Electronic searches yielded 2,348 citations. A further 23 citations were obtained by hand searching of reference lists. About 554 full articles were retrieved and assessed against our inclusion criteria. Of the 69 studies included in our review, 18 were from medicine, 32 from nursing and 19 from other allied health professions, including dentistry, physiotherapy and radiography. In all professional groups, portfolios were used mainly in the clinical setting, completion was compulsory, reflection required and assessment (either formative, summative or a combination of both) the norm. Three studies used electronic portfolios. Whilst many studies used a combination of data collection methods, over half of all included studies used questionnaires, a third used focus group interviews and another third used direct assessment of portfolios. Most studies assessed student or tutor perceptions of the effect of the use of portfolios on their learning. Five studies used a comparative design, one of which was a randomized controlled trial. Studies were most likely to meet the quality indicators relating to appropriateness of study subjects, clarity of research question and completeness of data. However, in many studies, methods were not reported in sufficient detail to allow a judgement to be made. About 19 of the 69 included studies (27%) met seven or more quality indicators. Across all professions, such 'higher quality' studies were more likely to have been published recently. The median 'quality score' (number of indicators met) rose from two for studies published in 2000 or earlier to seven for studies published in 2005 or later. Significant differences were observed between the quality scores for studies published in or before 2000 and those published between 2001 and 2004 (p = 0.027), those published in or before 2000 and those published in 2005 or later (p = 0.002) and between all studies (p = 0.004). Similar trends were seen in all professional groups. About 59 (85%) of the included studies were assessed at level 1 of the modified Kirkpatrick hierarchy (i.e. 'participation' effects, including 'post hoc' evaluations of student perceptions of the effects of keeping a portfolio on their learning). About 9 (13%) of the studies reported direct measurement of changes in student skills or attitudes and one study reported a change in student behaviour. The main effects of portfolio use identified by the included studies were: Improvement in student knowledge and understanding (28 studies, six at Kirkpatrick level 2 or above), greater self-awareness and encouragement to reflection (44 studies, seven at Kirkpatrick level 2 or above) and the ability to learn independently (10 studies, one at Kirkpatrick level 2). The findings of higher quality studies also identified benefits in these areas. They reported improved student knowledge and understanding, particularly the ability to integrate theory with practice, although a correlation with improved scores in other assessments was not always apparent. Greater self-awareness and engagement in reflection were also noted, although some studies questioned the quality of the reflection undertaken. Higher quality studies also suggest that use of portfolios improves feedback to students and gives tutors a greater awareness of students' needs, may help students to cope with uncertain or emotionally demanding situations and prepares students for postgraduate settings in which reflective practice is required. Time commitment required to collate a portfolio was the major drawback identified. In two of the studies, this was found to detract from other clinical learning. CONCLUSIONS: At present, the strength and extent of the evidence base for the educational effects of portfolios in the undergraduate setting is limited. However, there is evidence of an improving trend in the quality of reported studies. 'Higher quality' papers identify improvements in knowledge and understanding, increased self-awareness and engagement in reflection and improved student-tutor relationships as the main benefits of portfolio use. However, they also suggest that whilst portfolios encourage students to engage in reflection, the quality of those reflections cannot be assumed and that the time commitment required for portfolio completion may detract from other learning or deter students from engaging with the process unless required to do so by the demands of assessment. Further work is needed to strengthen the evidence base for portfolio use, particularly comparative studies which observe changes in student knowledge and abilities directly, rather than reporting on their perceptions once a portfolio has been completed.
INTRODUCTION: In recent years, the use of portfolios as learning and assessment tools has become more widespread across the range of health professions. Whilst a growing body of literature has accompanied these trends, there is no clear collated summary of the evidence for the educational effects of the use of portfolios in undergraduate education. This systematic review is the result of our work to provide such a summary. METHODS: We developed a protocol based on the recommendations of the Best Evidence Medical Education (BEME) collaboration. Citations retrieved by electronic searches of 10 databases were assessed against pre-defined inclusion/exclusion criteria by two independent reviewers and full texts of potentially relevant articles were obtained. Studies were identified for inclusion in the review by examination of full text articles by two independent reviewers. At all stages, discrepancies were resolved by consensus. Data relating to characteristics of the student population, intervention, outcome measures, student design and outcomes were collected using a piloted data extraction form. Each study was assessed against 11 quality indicators designed to provide information about how well it was designed and conducted; and against the Kirkpatrick hierarchy as modified for educational settings. Comparisons between different groups were carried out using the Kruskal-Wallis test (non-parametric ANOVA) or the Mann-Whitney U test as appropriate. RESULTS: Electronic searches yielded 2,348 citations. A further 23 citations were obtained by hand searching of reference lists. About 554 full articles were retrieved and assessed against our inclusion criteria. Of the 69 studies included in our review, 18 were from medicine, 32 from nursing and 19 from other allied health professions, including dentistry, physiotherapy and radiography. In all professional groups, portfolios were used mainly in the clinical setting, completion was compulsory, reflection required and assessment (either formative, summative or a combination of both) the norm. Three studies used electronic portfolios. Whilst many studies used a combination of data collection methods, over half of all included studies used questionnaires, a third used focus group interviews and another third used direct assessment of portfolios. Most studies assessed student or tutor perceptions of the effect of the use of portfolios on their learning. Five studies used a comparative design, one of which was a randomized controlled trial. Studies were most likely to meet the quality indicators relating to appropriateness of study subjects, clarity of research question and completeness of data. However, in many studies, methods were not reported in sufficient detail to allow a judgement to be made. About 19 of the 69 included studies (27%) met seven or more quality indicators. Across all professions, such 'higher quality' studies were more likely to have been published recently. The median 'quality score' (number of indicators met) rose from two for studies published in 2000 or earlier to seven for studies published in 2005 or later. Significant differences were observed between the quality scores for studies published in or before 2000 and those published between 2001 and 2004 (p = 0.027), those published in or before 2000 and those published in 2005 or later (p = 0.002) and between all studies (p = 0.004). Similar trends were seen in all professional groups. About 59 (85%) of the included studies were assessed at level 1 of the modified Kirkpatrick hierarchy (i.e. 'participation' effects, including 'post hoc' evaluations of student perceptions of the effects of keeping a portfolio on their learning). About 9 (13%) of the studies reported direct measurement of changes in student skills or attitudes and one study reported a change in student behaviour. The main effects of portfolio use identified by the included studies were: Improvement in student knowledge and understanding (28 studies, six at Kirkpatrick level 2 or above), greater self-awareness and encouragement to reflection (44 studies, seven at Kirkpatrick level 2 or above) and the ability to learn independently (10 studies, one at Kirkpatrick level 2). The findings of higher quality studies also identified benefits in these areas. They reported improved student knowledge and understanding, particularly the ability to integrate theory with practice, although a correlation with improved scores in other assessments was not always apparent. Greater self-awareness and engagement in reflection were also noted, although some studies questioned the quality of the reflection undertaken. Higher quality studies also suggest that use of portfolios improves feedback to students and gives tutors a greater awareness of students' needs, may help students to cope with uncertain or emotionally demanding situations and prepares students for postgraduate settings in which reflective practice is required. Time commitment required to collate a portfolio was the major drawback identified. In two of the studies, this was found to detract from other clinical learning. CONCLUSIONS: At present, the strength and extent of the evidence base for the educational effects of portfolios in the undergraduate setting is limited. However, there is evidence of an improving trend in the quality of reported studies. 'Higher quality' papers identify improvements in knowledge and understanding, increased self-awareness and engagement in reflection and improved student-tutor relationships as the main benefits of portfolio use. However, they also suggest that whilst portfolios encourage students to engage in reflection, the quality of those reflections cannot be assumed and that the time commitment required for portfolio completion may detract from other learning or deter students from engaging with the process unless required to do so by the demands of assessment. Further work is needed to strengthen the evidence base for portfolio use, particularly comparative studies which observe changes in student knowledge and abilities directly, rather than reporting on their perceptions once a portfolio has been completed.