Competency-based assessment of practice-based experiential learning in undergraduate pharmacy programmes.

Objective: To obtain feedback from experiential leaning (EL) leads about how competency-based assessments could be undertaken by EL facilitators, and to scope existing EL assessment structures in undergraduate Masters in Pharmacy (MPharm) programmes across the United Kingdom (UK).
Methods: A cross-sectional survey was conducted utilizing a nine-item on-line survey, consisting of five open-ended and four closed-ended question. All UK universities with MPharm programmes (n=30) were invited to participate in the survey. Variables of interest were perceptions on activities and competencies that could be assessed by EL facilitators. The survey utilised a 5-point Likert-type response ranging from strongly disagree to strongly agree. Other variables of interest were tools/methods that could be used to assess competency, and perceived advantages and disadvantages of the proposed methods, the latter two captured via open-ended questions.
Results: Of the 21 universities that responded (Response rate: 70%), 17 were included in the final analysis. Fourteen of the 17 (82.4%) offered the 4-year programme, while 3 (17.6%) offered both the 4-year and 5-year integrated programme. Assessments were mainly undertaken by university staff (59%), with minimal amounts undertaken during EL (39%). There was unanimous agreement (100%) that facilitators could assess students' communication skills and professionalism during EL. No consensus, however, was achieved with regard to the tool(s) or method(s) to be used to assess student's competencies. There were 13 responses to the open-ended comments. An advantage noted was that EL facilitator assessment of students would allow for more accurate evaluation of students in the practice setting, while acknowledging barriers such as the burden of time and the lack of consistency in marking. To address this lack of consistency, the majority highlighted the need for facilitator training.
Conclusion: Minimal assessments are currently undertaken during EL, with students predominantly assessed on return to the university. No consensus could be achieved with regard to the tool(s) or method(s) to be used to assess students' competencies, suggesting that perhaps there is no one-size-fits-all, and that the tools and methods used should be informed by the competencies being assessed. Copyright: © Pharmacy Practice.

INTRODUCTION

The Additional Cost of Teaching for Pharmacy (ACTp) funding, an initiative to support the expansion of practice based experiential learning (EL) for student pharmacists in Scotland, was launched by the Scottish Government in 2018.1 This is an innovation in funding of EL not seen in any other part of the United Kingdom (UK) where EL is either funded through University monies or was provided by pharmacists without payment. This lack of dedicated funding means that quality and quantity of EL for students could vary depending on the availability of free EL or the limited funding available from the University. ACTp funding in Scotland mirrors the situation in medical schools where there is Government funding for support for learning. The ACTp money allows protected time for pharmacists to undertake EL facilitator roles, as well as ensuring that they are trained by means of the Preparation for Facilitating Experiential Learning (PFEL) sessions conducted by NHS Education for Scotland (NES)which results in more structured supervision.2 In the UK, facilitators are registered, practising pharmacists who supervise pharmacy students during EL.3 Prior to ACTp, EL sites received no payment and EL facilitators were not formally trained. At the time of this research, Scotland was the only UK country to have funded, quality managed EL sites.

In January 2021 the General Pharmaceutical Council (GPhC)- the UK Pharmacy regulator- published new Initial Education and Training (IET) Standards for Pharmacists. These are ambitious standards that mandated from July 2026, all student pharmacists would progress into a pre-registration year (known as Foundation Training Year from August 2021) run by the Statutory Education Body (SEB of NES in Scotland) and successful completion would allow trainee pharmacists to register with the GPhC as a pharmacist prescriber. To facilitate this development, throughout the MPharm, the Standards specify that increasing quality assured EL should be implemented. Part of this quality process would be the introduction of on-site assessment and targeted feedback to the student pharmacist to ensure the learning outcomes are met.4

As the EL facilitators assume an increased role in developing student pharmacists’ competencies, there is a desire by the Schools of Pharmacy for EL facilitators to conduct competency-based assessments of student pharmacists during their EL. Competency-based assessments measure student pharmacists’ performance during EL undertaken in real-world settings, against pre-defined standards and competencies.5 The competencies required for registration as a pharmacist in the UK are defined in the GPhC IET standards.4 In the UK, EL occurs throughout the 4 years of the MPharm integrated with classroom learning. Each university may deliver their education is in a different way to meet the GPhC IET Standards. The current Standards do not define the number and types of placements, but are outcome-based.4 GPhC reaccreditation processes for the Schools of Pharmacy and the SEBs drive continual improvement and raise standards for IET in the UK.4 are defined in the General Pharmaceutical Council’s Standards for the Initial Education and Training of Pharmacists.4

The ACTp research Project was initiated in 2018, funded by the Scottish Government, and is a collaboration between the two pharmacy schools in Scotland, NES, and overseen by an external advisor.3 It was tasked with evaluating the changes that could be made to EL with regard to placement sites, facilitator roles and training, and the assessment of students. This includes a series of studies which will then inform the design of the structures and processes of assessment. The project adopted a 360-degree approach where feedback was sought from academics, EL facilitators, key stakeholders, and students; employing multiple methods. In this initial study, we sought to obtain feedback from EL members of staff in the university responsible for EL about how competency-based assessments could be undertaken by EL facilitators, and to scope existing structures associated with assessment in EL in undergraduate MPharm programmes across the UK.

METHODS

Study design

This was a cross-sectional survey of members of staff responsible for EL in MPharm programmes in all Schools of Pharmacy in UK universities: 30 UK universities were fully-accredited to offer an MPharm degree at the time of the study.6 Ethical approval was obtained from the the Strathclyde Institute of Pharmacy and Biomedical Sciences Departmental Ethics Committee. The survey was hosted on Qualtrics, an online platform. An invitation email along with the participant information sheet were sent members of staff responsible for EL in March, 2020, with an anonymous link to the survey appended. Where contact details of the person(s) responsible for EL were not available, invitations via email were then sent to heads of school/department of pharmacy asking them to forward the survey/email to the appropriate person. No financial incentives were offered, and reminder emails were sent one month and two months after the initial email.

The survey was a nine-item anonymous self-report, comprising five open-ended and four closed-ended questions. Current assessment practices of EL inMPharm programmes in the UK were explored. Respondents’ perceptions were sought on competencies and activities that could be assessed by EL facilitators along with tools/methods that could be used. Perceived advantages and disadvantages of the proposed methods were captured using open-ended questions, which respondents could abstain from answering if desired. Details on programmes offered and funding received for EL were also obtained. No demographic details were collected.

The survey was developed based on a review of the literature and the research objectives. Possible competencies that could be assessed by facilitators were compiled from the Standards for the Initial Education and Training of Pharmacists by the General Pharmaceutical Council (GPhC).4 This list was produced after a series of discussions between the researchers and guided by a review of the literature. Face and content validation was undertaken by members of the ACTp research team. The survey was pilot tested on one academic with experience in assessments and experiential learning - this response was not included in the final analysis. Following feedback from the pilot test, minor amendments were made to improve the technical aspects of the survey. The survey took approximately 15-20 minutes to complete.

Data analysis

To create a composite picture of what respondents disagreed and agreed on for questions employing the five-point Likert scale, responses were collapsed to a 3-point scale (agree, neutral, disagree) for ease of interpretation, where the scores for the first two columns (“strongly disagree” and “disagree”) were combined, and the scores for the last two columns (“agree” and “strongly agree”) were also totalled. All analyses were performed using Microsoft Excel and SPSS 24.0 statistical software (SPSS Inc, Chicago, IL, USA).Open-ended comments were analysed descriptively.

RESULTS

There were 21 responses (response rate: 70%), of which four only filled in details on programme characteristics. Therefore, only 17 responses were included in the final analysis. Fourteen of the 17 (82.4%) offered the 4-year programme, while 3 (17.6%) offered both the 4-year and 5-year integrated programme. Three (17.6%) received funding for their EL, two from the government, and one from joint funding between the department of health and universities.

The majority (n=47, 59%) of assessment of competencies was undertaken by university staff, and on return to the university (n=48, 61%). There was high level of agreement from respondents that facilitators could assess the competencies listed in Table 2, especially with regard to professionalism, communication skills, and clinical skills such as providing pharmaceutical care (Table 1).

Table 2

Competencies that can be assessed by facilitators (n=16)

Competencies	Agree, n(%)	Neutral, n(%)	Disagree, n(%)
a) Contributing as members of an interprofessional healthcare team	12 (75.0)	3 (18.8)	1 (6.3)
b) Counselling patients on their medicines	15 (93.8)	1 (6.3)	0 (0.0)
c) Communication skills	16 (100.0)	0 (0.0)	0 (0.0)
d) Analysing prescriptions for validity and clarity	16 (100.0)	0 (0.0)	0 (0.0)
e) Clinically evaluating the appropriateness of prescribed medicines	16 (100.0)	0 (0.0)	0 (0.0)
f) Obtaining and documenting patient data and consultation records	13 (81.3)	3 (18.8)	0 (0.0)
g) Safe and legal handling of medicines	15 (93.8)	1 (6.3)	0 (0.0)
h) Using pharmaceutical calculations to verify the safety of doses and administration rates	16 (100.0)	0 (0.0)	0 (0.0)
i) Demonstrating teamwork	13 (81.3)	3 (18.8)	0 (0.0)
j) Demonstrating the characteristics of a prospective professional pharmacist as set out in relevant codes of conduct and behaviour	16 (100.0)	0 (0.0)	0 (0.0)

Table 1

When assessments are undertaken and by whom (n=17)

	During EL, n(%)*	On return to the university, n(%)*	Total, n (%)
Facilitators/tutors/preceptors	10 (58.8)	3 (17.6)	13 (16.4)
Director of experiential learning[¥]	0 (0.0)	5 (29.4)	5 (6.3)
Academic staff directly involved in EL[#, ¥]	1 (54.9)	13 (76.5)	14 (17.7)
Academic staff not directly involved in EL[#, ¥]	0 (0.0)	8 (47.1)	8 (10.1)
Patients	2 (11.8)	2 (11.8)	4 (5.1)
Students (self-assessment)	6 (35.3)	4 (23.5)	10 (12.7)
Students (peer-review/assessment)	2 (11.8)	3 (17.6)	5 (6.3)
University-employed clinical facilitators (including teacher practitioners, university regional tutors)¥	10 (58.8)	10 (58.8)	20 (25.3)
Total, n(%)	31 (39.2)	48 (60.8)	79 (100)

Respondents were allowed to select more than one option; therefore, totals might exceed 100%.

Academic staff refer to faculty of the School of Pharmacy

University staff

Cumulative tabulation of assignments or activities that could be assessed by facilitators, ,revealed that respondents displayed a preference for diaries (38%), especially for assessing interprofessional learning (IPL) (92%) and teamwork skills (92%). When looking at the individual competencies, approximately 70% preferred handbooks/learning portfolios/workbooks particularly for assessing, counselling, and clinically evaluating the appropriateness of prescribed medicines. while more than 75% preferred it for assessing IPL. Cumulative tabulation also revealed that clinical documentation (8%) and written assignments/projects (10%) were the least preferred options for assessment by facilitators (Table 3).

Table 3

Assignment/activities that could be assessed by facilitators based on specific competencies (n=13)

Competencies	Assignments/activities, n(%)*
	Written answers (e.g. written assignments/ projects, written papers)	Presentations (e.g. individual oral, small-group, poster)	Handbook/ learning portfolios/workbook	Clinical documentation	Diaries [e.g. reflective diaries/journals, E-log/electronic diary/report (written evidence of performance during EL)]
Contributing as members of an interprofessional healthcare team	2 (15.4)	4 (30.8)	10 (76.9)	1 (7.7)	12 (92.3)
Counselling patients on their medicines	0 (0.0)	1 (7.7)	9 (69.2)	2 (15.4)	9 (69.2)
Communication skills	0 (0.0)	5 (38.5)	7 (53.8)	0 (0.0)	10 (76.9)
Analysing prescriptions for validity and clarity	4 (30.8)	3 (23.1)	8 (61.5)	4 (30.8)	5 (38.5)
Clinically evaluating the appropriateness of prescribed medicines	5 (38.5)	4 (30.8)	9 (69.2)	5 (38.5)	9 (69.2)
Obtaining and documenting patient data and consultation records	3 (23.1)	4 (30.8)	6 (46.2)	5 (38.5)	8 (61.5)
Safe and legal handling of medicines	2 (15.4)	1 (7.7)	7 (53.8)	2 (15.4)	10 (76.9)
Using pharmaceutical calculations to verify the safety of doses and administration rates	5 (38.5)	3 (23.1)	6 (46.2)	1 (7.7)	8 (61.5)
Demonstrating team work	1 (7.7)	4 (30.8)	7 (53.8)	0 (0.0)	12 (92.3)
Demonstrating the characteristics of a prospective professional pharmacist as set out in relevant codes of conduct and behaviour	2 (15.4)	2 (15.4)	8 (61.5)	0 (0.0)	11 (84.6)
Other: Clinical skills	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	1 (7.7)
Cumulative preference, n(%)	24 (9.7)	31 (12.5)	77 (31.0)	20 (8.1)	95 (38.3)

Respondents were allowed to select more than one option

With regard to tools/methods of assessment as shown in Table 4, cumulative tabulation revealed that some respondents agreed with using direct observation of students (33.7%) with unanimous agreement for assessing professionalism, teamwork, and IPL. Less than 5% of respondents agreed with using validated or established assessments tools. In the open-ended comments, three respondents suggested using the Medication Related Consultation Framework for assessing consultation skills.

Table 4

Tools/methods that could be used to undertake assessments (n=13)

Competencies	Tools/methods, n(%)*
	Student interviews/oral assessment	Validated/established assessment tool	Assessment tool designed internally	Marking rubric	Mini Clinical Evaluation Exercise (mini-CEX)	Formal (objective) evaluation sessions on site	Direct observation of student during EL
Contributing as members of an interprofessional healthcare team	3 (23.1)	1 (7.7)	3 (23.1)	3 (23.1)	4 (30.8)	3 (23.1)	13 (100.0)
Counselling patients on their medicines	3 (23.1)	2 (15.4)	4 (30.8)	3 (23.1)	3 (23.1)	8 (61.5)	12 (92.3)
Communication skills	3 (23.1)	3 (23.1)	4 (30.8)	3 (23.1)	4 (30.8)	7 (53.8)	12 (92.3)
Analysing prescriptions for validity and clarity	2 (15.4)	2 (15.4)	3 (23.1)	4 (30.8)	4 (30.8)	7 (53.8)	7 (53.8)
Clinically evaluating the appropriateness of prescribed medicines	3 (23.1)	1 (7.7)	4 (30.8)	3 (23.1)	9 (69.2)	6 (46.2)	7 (53.8)
Obtaining and documenting patient data and consultation records	2 (15.4)	2 (15.4)	3 (23.1)	3 (23.1)	6 (46.2)	7 (53.8)	10 (76.9)
Safe and legal handling of medicines	1 (7.7)	1 (7.7)	4 (30.8)	2 (15.4)	2 (15.4)	4 (30.8)	9 (69.2)
Using pharmaceutical calculations to verify the safety of doses and administration rates	3 (23.1)	1 (7.7)	5 (38.5)	5 (38.5)	5 (38.5)	5 (38.5)	7 (53.8)
Demonstrating team work	3 (23.1)	1 (7.7)	4 (30.8)	1 (7.7)	3 (23.1)	3 (23.1)	13 (100.0)
Demonstrating the characteristics of a prospective professional pharmacist as set out in relevant codes of conduct and behaviour	4 (30.8)	0 (0.0)	3 (23.1)	1 (7.69)	2 (15.4)	5 (38.5)	13 (100.0)
Other: Clinical skills	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	1 (7.7)	1 (7.7)
Cumulative preference, n(%)	27 (8.7)	14 (4.5)	37 (12.0)	28 (9.06)	42 (13.6)	56 (18.1)	104 (33.7)

Respondents were allowed to select more than one option

There were 13 (76.5%) responses to the open-ended comments. Respondents felt that one of the advantages of facilitators formally assessing students during EL was that it would allow a more authentic assessment of students in the real-world setting, compared to assessing their skills based on manufactured scenarios, with one reflecting, “Assessment and feedback in the work environment, most closely mirrors skills, knowledge and attitude required when qualified.” Other noted advantages were increased involvement of practising staff in education, a reduction in academic workload, and increased motivation for students to perform better during EL. The disadvantages noted were the burden of time and workload on facilitators, inconsistency in marking, multiple facilitator involvement, lack of facilitator knowledge of assessments, and limited EL duration. Potential challenges highlighted were the lack of money for quality assurance purposes, as well as resistance from pharmacists due to time constraints.

Facilitator training was highlighted as very important to ensure fairness and consistency. Respondents also stressed the importance of using tools and mechanisms that were simple and accessible. There were also calls for a change in the current culture, with one noting the following: “We really need the support of the regulator in this - if education and training was more a part of the role of practicing pharmacists, in the way it is for medics and nurses, it would be easier to change the culture and break the barriers currently in place”.

DISCUSSION

Findings suggest that MPharm students in the UK are largely assessed on skills gained during EL on return to the university by university staff, with fewer assessments undertaken at placement sites. Indeed, in a previous nationwide study involving the same cohort, the majority (87%) relied on students’ reflective diaries to assess what they have learnt during EL. Other methods of assessments were student handbooks (47%), Objective Structured Clinical Examinations (OSCEs) (40%), and examinations (33%). Less than 2% assessed students based on feedback from facilitators.7

There are, however, gaps in the current approaches or methods used. Reflective diaries are heavily relied upon, however for reflective learning to be effective, students must be self-directed, have an understanding of the educational value of their experiences, and are motivated and curious to learn more.8 Students have also noted that direct observation by facilitators would be a more accurate method of assessing their competencies as they would have had time to re-create and submit a well-written report which details their experience, but it is not necessarily an accurate reflection of the actual interaction with the patient.9 Assessments via OSCEs, which are simulated scenarios and undertaken after the placements have ended, also do not reflect the dynamic nature of real-world consultations or the real-world stress that students will face. In addition, OSCEs tend to assess competencies individually, rather than in an integrated format, which again does not give an accurate measure of students’ competencies in practice.10 A better option would be the use of electronic portfolios (e-portfolios). where students upload evidence against the competencies achieved such as case studies, care plans, and patient information leaflets. It will also include a reflective component.11 Indeed, e-portfolios have been shown to be a useful tool for students to demonstrate the achievement of competencies,11 and also allows for feedback to be received from multiple sources e.g. facilitators, university staff, or even peers.12

Competency-based education is an example of outcomes-based education, which is preferred compared to input-based education, due to growing expectations from employers and the government.13 Indeed, two main features of competency-based education are its emphasis on outcomes as well as the development of skills and abilities to prepare healthcare professionals to serve patients in an effective manner.14 During the pre-registration period, trainees are assessed on their competencies by pre-registration tutors and have to prove their competence to practice to qualify for full registration.13 As such, adoption of a similar method of assessment at an earlier stage i.e. undergraduate, will allow for a seamless transition into the pre-registration period.15

Potential barriers and challenges highlighted such as lack of consistency in marking and lack of facilitator knowledge of marking were highlighted, however in Scotland, modules addressing these can easily be adapted into the PEFL training.2 The lack of consensus suggests there is no one-size-fits all when it comes to choosing a specific tool or method to assess students’ competencies. Instead, these should be personalised according to the respective competencies being assessed and the EL sites.

LIMITATIONS

The anonymous nature of the survey did not allow for individual follow-up especially with the respondents. It is suggested that future qualitative research is undertaken involving the same respondents to elicit more in-depth feedback especially with regard to the open-ended comments. Inferential statistical analysis could not be undertaken as demographic data was not collected. The researchers selected key competencies which were thought to be appropriate to be assessed during EL from a longer list of learning outcomes listed in the GPhC IET Standards.4 Readers should therefore be mindful of this, and the fact that there may be other competencies which might also be considered appropriate to be assessed by EL facilitators. The perspectives presented here are only from the point of view of university staff and therefore might not reflect the opinions of facilitators who will be tasked with this job in the future. This will be the focus of current and future work, as mentioned below.

CURRENT AND FUTURE WORK

In Scotland, the authors suggest that work placed assessments suitable for student pharmacists should be co-produced with the Schools of pharmacy, NES and the wider NHS service (including community pharmacy). These assessments will relate to the learning objectives for each EL period and would require EL facilitators to be trained appropriately. An e-portfolio would allow EL facilitators and School of Pharmacy staff to access the results of assessments and student feedback.

Following the findings of this study, qualitative studies to determine the perception of key stakeholders involved in or impacted by the proposed new approach to assessments i.e. students, facilitators, academics, and staff from NES have been undertaken. Qualitative studies are also currently being undertaken involving key persons from other programmes which have well-established experience in competency-based assessments such as nursing, medicine and teaching. Future plans are to involve these key stakeholders in the design and development of the assessment process, which will include the tools and methods to be used, and competencies to be assessed based on specific learning objectives in specific EL sites. A remedial EL pathway is also being developed by Schools of Pharmacy and NES.

CONCLUSION

Findings from this study suggest that various methods and tools can be used to assess students on the skills gained during EL, as well as how students can be assessed. While the benefits of competency-based assessments by facilitators were acknowledged, there were concerns about the impact of this new approach on students as well as facilitators. If we are to prepare students for the high-level, face-paced, high-risk, high-intensity of pharmacy and to practise as prescribers, it is vital that we then ensure that appropriate assessment methods are adopted so students are competent to meet the needs of the patients. To ensure this, all key stakeholders should be involved in the design of the assessment process.

SOURCE OF FUNDING

This study was supported by an educational grant received from NHS Education for Scotland (NES).

CONFLICTS OF INTEREST

The authors report no conflicts of interest.

AUTHOR ROLES (CRediT)

Conceptualization: SAJ, ACB

Data curation: SAJ

Formal analysis: SAJ, ACB

Investigation: SAJ

Methodology: SAJ, AP, JP, TJ, SC, ACB

Project administration: ACB

Resources: Not applicable

Software: Not applicable.

Supervision: ACB

Validation: SAJ, AP, JP, TJ, SC, ACB

Visualization: Not applicable

Writing - original draft: SAJ, ACB

Writing -review & editing: SAJ, AP, JP, TJ, SC, ACB