"Portfolios" as a method of assessment in medical education.

Portfolios are increasingly used in postgraduate medical education and in gastroenterology training as an assessment tool, as documentation of competence, a database of procedure experience (for example endoscopy experience) and for revalidation purposes. In this paper the educational theory behind their use is described and the evidence for their use is discussed.

Introduction

The use of portfolios for assessment in the health care professions has developed as part of a move away from “snap shot” examinations, towards broader methods of assessment. This form of assessment is thought to encourage closer links between assessment and learning by improving learning outcomes using assessment and providing feedback. The use of a portfolio is also thought to enhance the assessment of areas that are difficult to assess by traditional methods in a wide range of clinical contexts. These areas include attitudes, personal attributes, reflection and professionalism (1).

Portfolios are being increasingly used in postgraduate medical education to assess ability to progress. In the UK, a huge emphasis is placed upon them in the foundation years of training (2) and later in speciality training (3). They also form the basis of the annual appraisal (ARCP) for trainee doctors in the UK (2, 3). They are used extensively in training in Gastroenterology as a way of documenting clinical competence and for recording endoscopy experience and competence (4). In 2001, the general medical council (5) identified a portfolio approach to revalidation for all doctors. Revalidation was introduced in the United Kingdom in 2012 and requires licenced doctors to keep a portfolio of evidence of their practice (6).

Portfolio contents

Several models of portfolios have been described. In Table 1, their use is described and the advantages and disadvantage of their use is outlined. The four models demonstrate how different the contents of a portfolio may be. The appropriateness of each model depends very much on its purpose and whether it will be used in an assessment process. (7).

Table 1

Models of portfolio

Portfolio	Description	Advantages	Disadvantages
Shopping trolley	Contains anything which has been produced or used during the learning process	Very inclusive	Difficult to assess. No analysis of contents
Toast rack	“Toast” for each period of learning	Corresponds with the curriculum Can be marked Includes reflection	Each item is discrete and does not provide overall assessment of learningNo overall reflection
Cake mix	Integration of the parts “Mixing” is reflection on the analytical components	Global assessment	Individual components may not be clear
Spinal column	Series of competency statements are the “vertebrae”Evidence is linked to the competencies	Each competency has its own evidence

Alignment and assessment

In order to demonstrate alignment, a portfolio structure should be decided based on the format the evidence is required in. The more prescriptive portfolio models should be easily aligned with the curriculum as the curriculum can form the “spinal column” or “toast rack”. The “shopping trolley” model is least likely to demonstrate alignment as this method relies heavily on the student including relevant documentation. Whichever method is chosen should provide ample opportunity to demonstrate learning in many different ways and should be a holistic record of learning.

Davis and Ponnamperuma (2006) propose five steps in the portfolio assessment process. They are as follows:

Collection of the evidence of learning.

Reflection of the learning.

Evaluation of evidence by assessors.

Defence of evidence by the individual being assessed.

An assessment decision. All the assessors involved, should make a decision based on pre-defined criteria. (1)

Educational impact

The different types of portfolio will have varying degrees of educational impact. A model, which purely records educational events but does not provide overall reflection (for example the “toast rack” may not foster learning and may be seen as a box ticking exercise. In contrast, the “cake mix” model that includes evidence and reflection may increase educational impact by enabling an individual to reflect on learning.

An advantage of using portfolios as a means of assessment is that gaps may be identified in training particularly if the “spinal column” or “toast rack” models are used. It is also possible to present evidence from other assessments within a portfolio. For example, in postgraduate medical education work place based assessments are used. Incorporating these elements will allow the assessment process to be greater than the sum of its parts.

Reliability

Roberts et al. (2002) conducted a systematic review of the evidence of the use of portfolios in the medical education literature (9). Two papers were found which provided data from small studies (10, 11). These studies focused on the inter-rater reliability of portfolios. Their results showed reliability falling below the level of 0.8 (9). Jenkins et al., (2013) tested the reliability of a portfolio used as an assessment tool for postgraduate family medicine training in South Africa and found the tool as a whole to be reliable with a co-efficient of 0.92. Individual components of the portfolio were not shown to be reliable, which lead the researchers to make recommendations as to how the portfolio could be improved (12).

The literature suggests that in order to achieve high levels of inter-rater reliability, portfolios should be introduced to well prepared students and should be of uniform content (13, 14). They should be marked by well trained, experienced assessors using clear criteria. Both assessors and assesses should have a shared understanding of the purpose of the assessment (9).

Portfolios may include other assessments within them for example in post-graduate education, work-based assessments such as DOPS (directly observed procedural skills), CBD (case based discussion) and multi-source feedback. Generalizability studies have shown that content specificity is a major contributor to unreliability (15). This means that large samples of performance need to be tested before a reliable judgement about ability can be made (9). This means that by including work based assessments in a portfolio, reliability is increased, as there will be more assessments. This allows individual competencies to be assessed in more than one way, and enables triangulation of evidence and therefore increases reliability.

Validity

Dreissen et al. (2006) identified a few studies in the literature, which lend some support to different aspects of validity. These include criterion and construct validity, predictive validity and content validity. (16–19).

Driessen (2006) looked in detail at which criteria affected an assessor's judgement of student's reflective skills and found that there was no impact of irrelevant criteria (such as hand writing) in assessment, and that the strongest predictor of variance was the “quality of reflection” (19). Driessen et al., (2006) suggested that portfolio standardisation and the use of analytical criteria with the aim of improving reliability may threaten validity by limiting room for the description of personal learning experiences in different authentic situations (19). They also suggested that including checklists and analytical assessment criteria might trivialise the assessment process (19). A study by Quinlan (2002), it was demonstrated that competence ratings were influenced by what assessors already knew about students, and therefore reducing validity of the assessment tool (20).

The use of portfolios in medical training to demonstrate clinical competence

Portfolios were initially used in medical education for formative purposes in order to encourage reflection. (21). More recently, the use of portfolios has been advocated for summative purposes (9, 22). The “eportfolio” now forms the basis for both foundation year training and speciality training in the United Kingdom. This is a spinal column type of portfolio and individual pieces of evidence are linked to predefined competencies (2, 3). For endoscopy training in the UK trainees are expected to document all their procedure data in one database, and to collect work based assessments during their training. A trainee is deemed competent when they have achieved certain predetermined criteria laid out in the eportfolio. This documentation is available for trainers to explore (3).

Millers pyramid

Millar's pyramid is often used as a model for the assessment of clinical competence. The pyramid has four tiers that represent does, shows how, knows how and knows (23). Table 2 demonstrates how each aspect of Millar's pyramid may be demonstrated using a portfolio.

Table 2

Components of Millar's pyramid (23)

Millar's pyramid	Element of the portfolio
Does	Components of a portfolio: work based assessments, reflective diaries, 360 degree appraisal
Shows how	Clinical examinations
Knows how	Some written examinations, e.g. clinical examinations.
Knows	Written Examinations

Portfolios are advantageous in that they are able to test areas difficult to assess such as professionalism, continuous professional development, attitudes and critical thinking. They may be thought of as encouraging more effective learning by encouraging reflection and allowing continuous assessment to occur. Work based assessments provide real life assessments of real life situations and by including different methods of assessment, an individual is assessed in many different ways. Problems should be able to be identified early and gaps in an individual's learning may be identified. Feedback can be provided following individual assessments and after reviewing the entire document. If structured appropriately it should be feasible and acceptable to be used in summative assessments (1).

However, the environment in which the portfolio is introduced may influence attitudes towards portfolios. There may also issues around the practical nature and feasibility of portfolio assessments. They are time consuming for both the assessor and the individual being assessed and they are difficult to difficult to mark (1). They require evidence, assessments and data to be collected prospectively and cannot be effectively competed retrospectively (1).

Conclusions

Portfolios may be used to teach and assess attitudes and professionalism that are difficult to assess by other means. They may be used to teach and learn a range of outcomes. If they incorporate assessments from a range of settings they should be able to assess all four levels of Millar's pyramid if results from work based assessments. They may provide feedback to both the individual being assessed and the assessor, and a holistic picture of the individual's fitness to practice (1). They are widely used in medical education to demonstrate competencies across a range of clinical and non-clinical skills and this use will improve the reliability and validity of the tools used. There are some disadvantages, which have also been described which make using a portfolio as a means of assessment difficult, but if completed appropriately they provide good evidence for summative assessments and for revalidation purposes.