Appraisal of the entrustable professional activities (EPAs) patient care provider domain by North Dakota pharmacists.

BACKGROUND: Entrustable Professional Activities (EPAs) are the latest addition to a list of professional competencies that pharmacy educational organizations support, and accreditation organizations require, for assessment by colleges and schools of pharmacy.
OBJECTIVE: The study's objective is to assess the use of Core EPAs in the patient care domain (by practice setting, position, and preceptor status) in contemporary pharmacy practice.
METHODS: This survey assessed the EPA activities of pharmacists practicing in North Dakota. The pharmacists were asked "how many times in the past 30 days have you delivered the following services in your practice setting?" Response options were: 0, 1, 2, 3, 4, and 5 or more times.
RESULTS: Of 990 potential respondents, 457 pharmacists (46.1%) returned a survey, and 107 (10.8%) answered every survey item in the patient care domain. Respondents reported that the highest rated activity items "Collect information to identify a patient's medication-related problems and health-related needs," and "Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs" were performed an average of 3.9 times per week (SD=1.8), and 3.8 times per week (SD=2.0), respectively. Both of these items, were reported for 70% of the respondents at 5 or more times per week. For these items, the highest reported practice setting was 'other' practice settings (e.g., long-term care, community health centers) followed by chains, hospitals, and independent pharmacies. By position, clinical pharmacists and preceptors reported the highest activity levels for most EPAs and supportive example tasks.
CONCLUSIONS: This study provides empirical evidence suggesting (but not proving) that EPAs have potential as a means to assess outcomes in pharmacy education and practice. Our study sets the stage for future work that further refines and assesses core EPA activities and supportive example tasks to measure the impact of how this process relates to outcomes of care. Copyright: © Pharmacy Practice.

INTRODUCTION

One of the most challenging areas of clinical education is ensuring that students successfully transition between didactic, experiential, and practice settings. Empirically characterizing and assessing these transitions are equally challenging endeavors. ten Cate and Scheele were among the first educators to investigate the process of creating and integrating practice proficiencies based on academic competencies.1 Their work, among those of other scholars, eventually led to the creation of Entrustable Professional Activities (EPAs).

EPAs, as used and assessed in pharmacy practice, are a competency system that is increasingly used in health professions education to evaluate healthcare professionals. The EPAs themselves are descriptions of work-related tasks performed by professionals practicing in a given field. Competence is assessed by asking a supervisor (usually a preceptor or professional supervisor) to rate the level of trust (using a standardized response scale) that the supervisor has in the student to complete the given task after acquiring the necessary knowledge, skills and abilities in a professional clinical education program. Each professional activity can be broken down into several specific sub-competencies.2 New graduates (especially those in pharmacy practice) should be able to complete assigned tasks with reactive supervision (i.e., a Level 3 competency), while experienced clinicians should be able to complete the same task independently (i.e., Level 4 supervision).3 EPAs are not meant to replace academic competencies, but rather to be used to incorporate them into clinical practice settings.4 EPAs were first used by medical residencies to translate statements of competency into achievable tasks that can be observed.5 In response to EPA formation in graduate medical education for family medicine and internal medicine, in 2014 the Association of American Medical Colleges created a list of 13 core EPAs that every medical resident, regardless of practice area, should be capable of performing their first day without supervision.6 Since this publication, specific EPAs for other areas of physician practice, including (but not limited to) pediatrics, have been created.7,8

Pharmacy education is another field in which use of EPAs may be beneficial. Like medical education, pharmacy education is based upon a competency-centered framework. Many pharmacy preceptors find the competency statements currently being used (which primarily reflect the needs of academic programs) to be abstract and difficult to employ in clinical settings.5 In theory, the incorporation of EPAs may provide a more effective and transparent framework for connecting didactic knowledge to experiential education. In doing so, EPAs facilitate assessment of students by preceptors, who tend to find EPAs conceptually easier to understand. They may also serve as an improved framework for students, who can observe these specific activities and emulate their preceptor as the preceptor displays these skills.3,9

EPAs domains were discussed for pharmacy curriculum by the 2015-2016 American Association of Colleges of Pharmacy (AACP) Academic Affairs Committee, which focused on converting the 2013 Center for Advancement of Pharmacy Education (CAPE) outcomes into distinct, measurable tasks.4,10 The initial pharmacy EPAs required the integration of tasks and competencies within several different areas, such as knowledge, information collection and communication.4,10 The 2015-2016 Academic Affairs Committee created six Core EPA domains for pharmacy: patient care provider, population health promoter, information master, practice manager, interprofessional team member, and self-developer.10 As noted earlier, the Committee‘s expectation was that all new graduates, and subsequent new pharmacists, should be able to perform all of these activities with limited direction or feedback from supervisors (Level 3 performance).5 Although these Core EPAs may be similar to those used in other health care professions, they have been adapted to emphasize the unique contributions of pharmacists to patient care, including interprofessional, team-based patient care.5,10,11

EPAs are designed to standardize and strengthen the transition between educational training, residency, and practice.5 However, as schools and colleges of pharmacy integrate EPAs into their curricula, it is important to determine the frequency in which pharmacists, and pharmacist preceptors, are completing each of the activities and tasks within each of the Core EPAs. It is thought that all the EPAs can be used in any pharmacy site (institutional, community, ambulatory care, etc.), but some will obviously better match with certain settings.12 When looking specifically at the patient care provider EPAs, they appear to be geared toward an institution-based clinical experience, where patient records and other health care professionals are easily accessible.10 Because of this, certain other settings, such as community pharmacies, may demonstrate these EPAs less often than other sites. If true, this could create difficulties for other types of practice settings to incorporate specific EPA activities and tasks into experiential education at those sites.

Prior to incorporating Core EPAs into pharmacy curricula and assessment practices, it is useful to assess how frequently pharmacists actually perform specific Core EPAs in their daily pharmacy practice. As a corollary, it is interesting to determine whether the frequency of use varies by practice setting, position, and preceptor status.

The study’s objective is to use survey methods to quantify the self-reported use of Core EPA activities and supporting tasks by pharmacists practicing in the state of North Dakota. Given the large number of Core EPAs, this study focuses only on EPA activities and supporting tasks categorized within the patient care provider domain. The survey responses were used to measure the extent to which pharmacists practicing in North Dakota perform them in various practice settings (hospitals, independent community pharmacies, chain community pharmacies, and other practice settings), by position (pharmacy manager, staff pharmacist, clinical pharmacist, all other roles), and by preceptor status (yes or no).

METHODS

The survey was designed using the North Dakota Pharmaceutical Care Survey as a template and employed standard survey research criteria.13-16 The survey incorporated the AACP’s six Core EPA domains, including those in the patient care provider domain, which is the focus of this manuscript. Within the patient care domain, there are five activity items, each of which is illustrated using multiple supportive example tasks. The five activities are: 1. Collect information to identify a patient’s medication-related problems and health-related needs (5 example tasks); 2. Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs (8 example tasks); 3. Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective (6 example tasks); 4. Implement a care plan in collaboration with the patient, caregivers, and other health professionals (4 example tasks); and, 5. Follow-up and monitor a care plan (4 example tasks).

For each item and example tasks, the pharmacists were asked to respond to the question, “How many times in the past 30 days have you delivered the following services in your practice setting?” As noted previously, it is unclear exactly how frequently each of these tasks was conducted in practice, given the respondent’s practice setting. This makes the creation of an appropriate response scale challenging, since the scale must allow for meaningful responses, yet properly account for the wide range of possible responses that may be reported by pharmacists. The researchers determined that, if a pharmacist performs a given task 5 or more times per month, then over time in the position, the pharmacist would perform the task with sufficient frequency to develop and maintain a high level of expertise in completing that task. Within the context of this study, this means that responses of 5 times per month convey much the same information, as in the case where the same task was performed, for example, 20 times per month. Therefore, the researchers chose to use a 6-point response scale with the following options: 0, 1, 2, 3, 4, and 5 or more times per month. The survey also contained demographic items, including gender, age, highest level of pharmacy education, primary pharmacy practice setting, primary position, population of the city in which your pharmacy is located, and whether (during the past year) the pharmacy serves as a preceptor for NDSU pharmacy students.

The first survey draft was designed based on the training and experience of the research team. The instrument (inclusive of the choice of the response scale) was refined by comparing survey items against other surveys in the pharmacy literature and subsequently pilot testing it on five pharmacists. The survey was revised based on these processes.14,15 All study procedures were reviewed and approved by the NDSU Institutional Review Board. Once approved, the research team obtained a list of pharmacist mailing and email addresses from the North Dakota Board of Pharmacy. From the mailing list, the investigators removed addresses outside the state of North Dakota, leaving 990 names in the final sample. The investigators used a modified Total Design Method, which has been used successfully in mail and internet surveys, to obtain a meaningful response rate.17,18 Using the email address list, the survey was emailed to all pharmacists registered and living in North Dakota in September 2017. Survey reminders were emailed to respondents 2, 4, 6, 8, 10, and 12 weeks after the initial survey emailing.

Data analysis. The Qualtrics Survey Software (www.qualtrics.com) was used to compile the data collected from the on-line survey. Respondents were not identified in the analysis. Demographic variables were summarized using means and standard deviations for non-discrete variables, and proportions for discrete variables. One statistical concern associated with the survey is the interpretation of the EPA data collected using its response scale. We interpret the response scale as an interval scale, and its data as interval data with possible truncation.19 More specifically, for those activities and supportive example tasks that most pharmacists frequently undertake, we expect a clustering of responses in the “5 or more times” category. This means that analyzing the responses with their exact responses, as provided, may generate descriptive statistics that understate the true mean and standard deviation for each survey item, especially if pharmacists complete each task many more than 5 times per month (instead of exactly 5 times per month). Concomitantly, for tasks that are performed infrequently, these descriptive statistics (inclusive of the response “5 or more times”) are likely to provide accurate and precise estimates of the true frequency with which pharmacists undertake these tasks.

Addressing this potential truncation requires a two-step analysis of responses that is both discrete (to distinguish the “5 or more times” from all other response options) and non-discrete (i.e., that uses the numerical value of the data, as they are reported). More specifically, we first report descriptive statistics (i.e., means and standard deviations) for each item over all possible responses. If the reported means are centered around the midpoint of the scale, then truncation is likely not a major concern (because there is little information loss that accrues from combining responses of 5 times per month with responses that exceed performing the task more than 5 times per month) and the descriptive statistics can be interpreted as reported. However, if the means are centered around the upper end of the scale, then truncation is likely to be an issue (because respondents are likely performing a task much more frequently than 5 times per month) and the basic descriptive statistics are not reliable. In such cases, an appropriate method of analysis is to change the nature of how the data are reported, by only summarizing information provided by the data that is not subject to possible truncation-related biases. More specifically, we report the proportion of responses for each item that report conducting the activity 5 or more times per month (versus the proportion of respondents that report conducting the activity less than 5 times), and the reader is encouraged to give greater weight to this measure, rather than the descriptive statistics for the entire scale.

When analyzing whether significant differences exist in the frequency of performing specific activities across different groups of pharmacy (i.e., by gender, practice settings, etc.) two different methods of analysis were used. Both methods operate under the null hypothesis of no relationship (or mean differences) between the frequency with which a task was completed and a specific pharmacist demographic. The chi-square test of homogeneity was used to assess possible differences in the frequency of respondents who undertook an activity 5 or more times, versus those who self-reported that they undertook the activity less than 5 times per month. The Kruskal-Wallis test (i.e., a nonparametric analog of analysis of variance assessing whether two or more groups of respondents exhibit similar distributions of responses) was used to assess whether specific pharmacist characteristics led to (self-reported) higher or lower frequencies of undertaking a specific task within a given 30 day window.19 We note in passing that the Kruskal-Wallis test was used instead of alternative parametric tests because it may provide results that (in certain situations) may be less severely distorted should the data exhibit truncation. All tests were conducted using the IBM SPSS version 24 and utilized 5 percent significance levels.

RESULTS

A total of 990 pharmacists were initially deemed eligible to participate in the study, of which 457 individuals responded to the survey (46.1%). Of those who responded, 355 (78%) were licensed pharmacists practicing in North Dakota. The remaining [not employed in the state (n=33), not employed in a patient care setting (n=27), not in a related pharmacy related career (n=7), retired (n=28), and not currently employed (n=7)] were removed from the final sample. After further eliminating individuals who failed to respond to each of the patient care provider domain items, we obtained a sample of 107 responses (10.8%).

Table 1 reports the demographic characteristics. Of the respondents, 67% were female. The average age was 43.5 years, with 68% of respondents reporting that they were less than 50 years of age. Most respondents had a Doctor of Pharmacy degree (67%) as their highest pharmacy degree, while 33% held a bachelor degree. The most common practice setting was independent community pharmacy (40%), followed by hospital (29%), all other practice setting (21%), and chain community pharmacy (10%). Thirty-six percent reported holding a pharmacy manager position, followed by staff pharmacists (32%) and clinical pharmacists (23%) positions. Sixty-eight percent of respondents served communities of 25,000 or more (e.g. urban) with the remaining 32% serving communities of 24,999 or fewer residents (e.g. rural). One-half of the respondents (50%) reported serving as an NDSU preceptor during the past year.

Table 1

Mean or percentage of the patient care provider domain respondents with each demographic characteristic (n=107)

Variable	Mean (SD) or Percentage
Gender
Female	70.0
Male	30.0
Age (mean, standard deviation)	43.5 (11.4)
Under 40 Years of Age	50.0
40-49 Years of Age	20.0
50-59 Years of Age	20.0
60 Years of Age or Older	10.0
Highest Pharmacy-Related Degree
Bachelor Degree	30.0
Doctor of Pharmacy	50.0
Post-Graduate Residency	10.0
Other Degree	10.0
Practice Setting
Hospital	30.0
Independent Community	40.0
Chain Community	10.0
All Other Practice Settings	20.0
Respondent’s Role
Pharmacy Manager	40.0
Staff Pharmacist	30.0
Clinical Pharmacist	20.0
All Other Roles	10.0
Population of Community Served
Under 5,000 Residents	10.0
5,000-24,999 Residents	20.0
25,000 or More Residents	70.0
Serve as an NDSU Pharmacy Preceptor
Yes, Serve as a Preceptor	50.0
No, Does not Serve as a Preceptor	50.0

Table 2 summarizes the descriptive statistics for the patient care provider EPA domain. The first patient care provider EPA activity “Collect information to identify a patient’s medication-related problems and health-related needs” was performed an average of 3.9 times per week (SD=1.8). Within this professional activity, the most commonly occurring EPA supporting example task was “Discuss a patient’s experience with medication” which was performed 3.8 times per week (SD=1.8). The next most commonly occurring EPA example task was “Determine a patient’s medication adherence” performed an average of 3.8 times per week (SD=1.9).

Table 2

Mean and proportion of Entrustable Professional Activities (EPAs) by patient care provider domain (n = 107)

Patient Care EPA description	Mean (SD)	Number (%) who perform the EPA 5 or more times per week
1. Collect information to identify a patient’s medication-related problems and health-related needs.	3.9 (1.8)	74 (69.2)
a. Collect a medical history from a patient or caregiver.	2.4 (2.3)	42 (39.3)
b. Collect a medication history from a patient or caregiver.	3.2 (2.1)	57 (53.3)
c. Discuss a patient’s experience with medication.	3.8 (1.8)	66 (61.7)
d. Determine a patient’s medication adherence.	3.8 (1.9)	68 (63.6)
e. Use health records to determine a patient’s health-related needs relevant to setting of care and the purpose of the encounter.	3.5 (2.1)	66 (61.7)
2. Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs.	3.8 (2.0)	74 (69.2)
a. Assess a patient’s signs and symptoms to determine whether the patient can be treated within the scope of practice or requires a referral.	2.2 (2.3)	39 (36.4)
b. Measure an adult patient’s vital signs and Interpret the results (e.g., body temperature, pulse rate, respiration rate, and blood pressure).	0.8 (1.7)	12 (11.2)
c. Interpret laboratory test results.	2.7 (2.3)	50 (46.7)
d. Identify drug interactions.	4.5 (1.3)	90 (84.1)
e. Perform a comprehensive medication review (CMR) for a patient.	2.6 (2.3)	49 (45.8)
f. Assess a patient’s health literacy using a validated screening tool.	0.4 (1.2)	6 (5.6)
g. Compile a prioritized health-related problem list for a patient.	1.0 (1.9)	17 (15.9)
h. Evaluate an existing drug therapy regimen.	3.2 (2.2)	61 (57)
3. Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective.	2.3 (2.4)	45 (42.1)
b. Develop a treatment plan with a patient.	1.4 (2.1)	22 (20.6)
c. Manage drug interactions.	3.6 (1.9)	65 (60.7)
d. Select monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	1.9 (2.3)	35 (32.7)
e. Determine the appropriate time interval(s) to collect monitoring data.	1.8 (2.3)	33 (30.8)
f. Create a patient-specific education plan.	1.4 (2.1)	22 (20.6)
4. Implement a care plan in collaboration with the patient, caregivers, and other health professionals.	2.1 (2.2)	35 (32.7)
a. Write a note that documents the findings, recommendations, and plan from a patient encounter.	2.2 (2.3)	42 (39.3)
b. Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test.	3.9 (1.9)	76 (71)
c. Educate a patient on the use of medication adherence aids.	2.2 (2.2)	35 (32.7)
d. Assist a patient with behavior change (e.g., use shared decision making and motivational strategies).	1.4 (1.9)	19 (17.8)
5. Follow-up and monitor a care plan.	2.3 (2.3)	43 (40.2)
a. Collect monitoring data at the appropriate time interval(s).	1.9 (2.3)	35 (32.7)
b. Evaluate the selected monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	2.1 (2.3)	39 (36.4)
c. Recommend modifications or adjustments to an existing medication therapy regimen based on patient response.	2.5 (2.2)	42 (39.3)
d. Present a patient case to a colleague during a handoff or transition of care.	1.8 (2.2)	31 (29)

“Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs”, was the second highest reported patient care provider EPA activity, at an average of 3.8 times per week (SD=2.0). Within this professional activity, “Identify drug interactions” was the most frequently performed supportive example task (mean=4.5 times per week, SD=1.3), followed by “managed drug interactions” (3.6 times per week, SD=1.9). As expected, for the majority of items with high means, a majority of respondents reported that they perform a task close to or at 5 or more times in a 30-day window.

Differences attributable to practice setting are reported in Table 3. Statistically significant differences were reported for “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs,” (Kruskal-Wallis p<0.01; chi-square p<0.01). Significant differences were also noted across practice settings for “Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective” (Kruskal-Wallis p=0.02; chi-square p<0.01); “Implement a care plan in collaboration with the patient, caregivers, and other health professionals” (Kruskal-Wallis p<0.01; chi-square p<0.01

Table 3

Entrustable Professional Activities (EPAs) of patient care provider domain by practice setting (n=107)

	Mean (SD)				p value
Patient care EPA description	Hospital [n = 31]	Independent community [n = 43]	Chain community [n = 11]	All Other Practices [n = 22]	ANOVA	Kruskal- Wallis	Chi-square homogeneity Test*
1. Collect information to identify a patient’s medication-related problems and health-related needs.	4.0 (1.8)	3.7 (1.8)	4.1 (1.7)	4.1 (1.9)	0.72	0.45	0.21
a. Collect a medical history from a patient or caregiver.	1.7 (2.3)	2.2 (2.1)	3.9 (2.0)	2.9 (2.3)	0.03	0.04	0.04
b. Collect a medication history from a patient or caregiver.	2.8 (2.4)	3.2 (1.9)	3.9 (2.0)	3.2 (2.3)	0.56	0.60	0.42
c. Discuss a patient’s experience with medication.	3.0 (2.0)	4.4 (1.2)	4.6 (1.0)	3.4 (2.2)	<0.01	0.01	0.01
d. Determine a patient’s medication adherence.	2.7 (2.3)	4.1 (1.5)	4.6 (0.9)	4.1 (1.7)	<0.01	0.01	0.07
e. Use health records to determine a patient’s health-related needs relevant to setting of care and the purpose of the encounter.	4.2 (1.8)	2.4 (2.2)	3.8 (2.1)	4.3 (1.6)	<0.01	<0.01	<0.01
2. Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs.	4.1 (1.9)	3.1 (2.1)	4.3 (1.3)	4.4 (1.7)	0.02	0.01	0.01
a. Assess a patient’s signs and symptoms to determine whether the patient can be treated within the scope of practice or requires a referral.	1.7 (2.2)	2.4 (2.2)	2.8 (2.3)	2.5 (2.4)	0.35	0.26	0.44
b. Measure an adult patient’s vital signs and Interpret the results (e.g., body temperature, pulse rate, respiration rate, and blood pressure).	0.5 (1.5)	1.1 (1.8)	1.3 (1.9)	0.6 (1.5)	0.35	0.03	0.87
c. Interpret laboratory test results.	4.5 (1.4)	1.2 (1.8)	1.5 (2.0)	3.8 (2.0)	<0.01	<0.01	<0.01
d. Identify drug interactions.	4.6 (1.3)	4.4 (1.3)	4.6 (1.2)	4.6 (1.2)	0.84	0.47	0.38
e. Perform a comprehensive medication review (CMR) for a patient.	3.3 (2.3)	1.9 (2.2)	3.0 (2.3)	2.9 (2.3)	0.06	0.06	0.05
f. Assess a patient’s health literacy using a validated screening tool.	0.5 (1.5)	0.5 (1.3)	0.2 (0.6)	0.2 (0.8)	0.80	0.91	0.38
g. Compile a prioritized health-related problem list for a patient.	1.0 (2.0)	0.6 (1.4)	1.3 (2.2)	1.6 (2.2)	0.16	0.2	0.17
h. Evaluate an existing drug therapy regimen.	3.7 (2.1)	2.4 (2.3)	4.0 (1.8)	3.6 (2.0)	0.03	0.03	0.03
3. Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective.	2.4 (2.5)	1.1 (1.9)	2.0 (2.3)	3.1 (2.3)	0.01	0.02	<0.01
a. Follow an evidence-based disease management protocol.	3.5 (2.3)	1.2 (2.0)	1.6 (2.1)	3.4 (2.3)	<0.01	<0.01	<0.01
b. Develop a treatment plan with a patient.	1.3 (2.1)	1.0 (1.8)	1.3 (2.0)	2.5 (2.3)	0.04	0.07	0.06
c. Manage drug interactions.	3.5 (2.2)	3.6 (1.9)	4.2 (1.2)	3.8 (2.0)	0.72	0.9	0.73
d. Select monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	3.0 (2.4)	0.9 (1.8)	1.5 (2.1)	2.6 (2.2)	<0.01	<0.01	<0.01
e. Determine the appropriate time interval(s) to collect monitoring data.	3.3 (2.3)	0.6 (1.5)	0.4 (1.2)	2.8 (2.2)	<.01	<0.01	<0.01
f. Create a patient-specific education plan.	1.4 (2.1)	1.0 (1.8)	1.2 (1.9)	2.4 (2.3)	0.06	0.12	0.14
4. Implement a care plan in collaboration with the patient, caregivers, and other health professionals.	2.7 (2.4)	0.9 (1.6)	1.3 (1.6)	3.7 (1.9)	<0.01	<0.01	<0.01
a. Write a note that documents the findings, recommendations, and plan from a patient encounter.	3.1 (2.4)	1.4 (2.1)	1.0 (1.5)	3.3 (2.2)	<0.01	<0.01	<0.01
b. Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test.	2.9 (2.1)	4.4 (1.4)	4.4 (1.4)	3.9 (2.1)	<0.01	<0.01	<0.01
c. Educate a patient on the use of medication adherence aids.	1.1 (1.9)	2.6 (2.1)	4.0 (1.8)	2.2 (2.2)	<0.01	<0.01	0.01
d. Assist a patient with behavior change (e.g., use shared decision making and motivational strategies).	0.8 (1.7)	1.5 (1.9)	2.1 (2.1)	1.8 (2.1)	0.12	0.05	0.66
5. Follow-up and monitor a care plan.	2.9 (2.5)	1.2 (1.9)	2.2 (2.3)	3.4 (2.1)	<0.01	<0.01	<0.01
a. Collect monitoring data at the appropriate time interval(s).	3.0 (2.5)	1.0 (1.8)	0.7 (1.3)	2.9 (2.4)	<0.01	<0.01	<0.01
b. Evaluate the selected monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	2.9 (2.4)	1.1 (2.0)	0.6 (1.3)	3.3 (2.2)	<0.01	<0.01	<0.01
c. Recommend modifications or adjustments to an existing medication therapy regimen based on patient response.	3.0 (2.3)	1.7 (2.1)	2.4 (1.7)	3.3 (2.3)	0.02	0.02	<0.01
d. Present a patient case to a colleague during a handoff or transition of care.	3.3 (2.3)	0.8 (1.7)	1.3 (2.0)	2.0 (2.2)	<0.01	<0.01	<0.01

(Less than 5 versus 5 or more times per week)

EPA activities and supporting example tasks are disaggregated by pharmacist position, and are reported in Table 4. Clinical pharmacists reported the highest levels followed by managers and staff pharmacists for all five activities: “Collect information to identify a patient’s medication-related problems and health-related needs;” “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs;” “Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective;” “Implement a care plan in collaboration with the patient, caregivers, and other health professionals;” and “Follow-up and monitor a care plan.”

Table 4

Entrustable Professional Activities (EPAs) of patient care provider domain by pharmacy position (n=107)

Patient Care EPA description	Mean (SD)				p value
	Manager [n = 38]	Staff pharmacist [n = 35]	Clinical pharmacist [n = 25]	All other positions [n = 10]	ANOVA	Kruskal- Wallis test	Chi-square homogeneity test*
1. Collect information to identify a patient’s medication-related problems and health-related needs.	4.0 (1.6)	3.3 (2.0)	4.9 (0.6)	2.8 (2.4)	<0.01	<.01	0.01
a. Collect a medical history from a patient or caregiver.	2.6 (2.3)	2.0 (2.2)	2.4 (2.3)	3.1 (2.5)	0.48	.48	0.35
b. Collect a medication history from a patient or caregiver.	3.3 (2.1)	2.6 (2.2)	3.7 (2.0)	3.2 (2.4)	0.25	.21	0.14
c. Discuss a patient’s experience with medication.	4.0 (1.7)	3.6 (1.8)	3.8 (1.9)	3.6 (1.9)	0.85	0.75	0.63
d. Determine a patient’s medication adherence.	3.7 (2.0)	3.6 (1.8)	3.9 (1.9)	4.3 (1.6)	0.73	0.75	0.70
e. Use health records to determine a patient’s health-related needs relevant to setting of care and the purpose of the encounter.	3.6 (2.0)	2.7 (2.3)	4.6 (1.3)	3.2 (2.4)	0.01	0.01	0.01
2. Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs.	3.6 (2.0)	3.1 (2.2)	4.6 (1.4)	4.3 (1.5)	0.03	0.02	0.01
a. Assess a patient’s signs and symptoms to determine whether the patient can be treated within the scope of practice or requires a referral.	2.5 (2.2)	1.8 (2.1)	2.7 (2.5)	1.4 (2.01	0.23	0.37	0.15
b. Measure an adult patient’s vital signs and Interpret the results (e.g., body temperature, pulse rate, respiration rate, and blood pressure).	1.3 (2.0)	0.6 (1.3)	0.6 (1.5)	0.5 (1.6)	0.17	0.11	0.10
c. Interpret laboratory test results.	1.9 (2.1)	1.9 (2.2)	5.0 (0.0)	2.5 (2.6)	<0.01	<0.01	<0.01
d. Identify drug interactions.	4.2 (1.6)	4.6 (1.1)	5.0 (0.0)	4.2 (1.7)	0.07	0.05	0.05
e. Perform a comprehensive medication review (CMR) for a patient.	2.9 (2.2)	1.6 (2.3)	4.1 (1.7)	1.4 (2.0)	<0.01	<0.01	<0.01
f. Assess a patient’s health literacy using a validated screening tool.	0.8 (1.8)	0.1 (0.5)	0.3 (1.1)	0.0 (0.0)	0.06	0.09	0.08
g. Compile a prioritized health-related problem list for a patient.	0.9 (1.8)	0.7 (1.7)	1.6 (2.3)	0.5 (1.6)	0.21	0.12	0.30
h. Evaluate an existing drug therapy regimen.	3.0 (2.1)	2.9 (2.4)	4.2 (1.7)	2.9 (2.3)	0.08	0.09	0.06
3. Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective.	1.6 (2.1)	1.4 (2.1)	3.8 (2.1)	0.5 (1.6)	<0.01	<0.01	<0.01
a. Follow an evidence-based disease management protocol.	1.9 (2.3)	1.7 (2.2)	3.9 (2.1)	2.5 (2.6)	<0.01	<0.01	<0.01
b. Develop a treatment plan with a patient.	1.4 (2.0)	0.9 (1.8)	2.4 (2.4)	0.5 (1.6)	0.02	0.02	0.02
c. Manage drug interactions.	3.6 (2.0)	3.3 (2.0)	4.0 (1.9)	4.0 (1.6)	0.51	0.42	0.34
d. Select monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	1.6 (2.1)	1.0 (1.9)	3.7 (2.2)	1.9 (2.3)	<0.01	<0.01	<0.01
e. Determine the appropriate time interval(s) to collect monitoring data.	1.4 (2.1)	0.8 (1.6)	3.7 (2.2)	2.3 (2.5)	<0.01	<0.01	<0.01
f. Create a patient-specific education plan.	1.5 (2.1)	0.7 (1.6)	2.6 (2.3)	0.5 (1.6)	<0.01	<0.01	0.03
4. Implement a care plan in collaboration with the patient, caregivers, and other health professionals.	1.6 (2.0)	1.3 (1.9)	3.8 (2.0)	2.2 (2.5)	<0.01	<0.01	<0.01
a. Write a note that documents the findings, recommendations, and plan from a patient encounter.	2.0 (2.1)	1.2 (2.0)	4.3 (1.7)	1.6 (2.4)	<0.01	<0.01	<0.01
b. Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test.	3.8 (2.0)	4.4 (1.4)	3.6 (2.0)	2.9 (2.3)	0.13	0.14	0.18
c. Educate a patient on the use of medication adherence aids.	2.8 (2.1)	2.2 (2.2)	1.6 (2.0)	1.8 (2.4)	0.15	0.10	0.34
d. Assist a patient with behavior change (e.g., use shared decision making and motivational strategies).	1.9 (1.9)	1.1 (1.8)	1.5 (2.1)	0.5 (1.6)	0.11	0.04	0.73
5. Follow-up and monitor a care plan.	2.0 (2.2)	1.5 (2.2)	3.9 (2.0)	1.7 (2.4)	<0.01	<0.01	<0.01
a. Collect monitoring data at the appropriate time interval(s).	1.6 (2.1)	0.9 (1.8)	3.9 (2.0)	1.8 (2.4)	<0.01	<0.01	<0.01
b. Evaluate the selected monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	1.7 (2.1)	1.1 (1.9)	3.9 (2.0)	2.0 (2.6)	<0.01	<0.01	<0.01
c. Recommend modifications or adjustments to an existing medication therapy regimen based on patient response.	2.1 (2.2)	1.8 (2.0)	3.8 (2.0)	2.7 (2.5)	<0.01	0.01	<0.01
d. Present a patient case to a colleague during a handoff or transition of care.	2.2 (2.3)	0.7 (1.6)	2.8 (2.2)	1.4 (2.3)	<0.01	<0.01	0.03

Less than 5 versus 5 or More Times per Week

With regard to EPAs and pharmacy position, there were significant differences in several supporting example tasks: “Use health records to determine a patient’s health-related needs relevant to setting of care and the purpose of the encounter” (Kruskal-Wallis p<0.01; chi-square p<0.01); “Interpret laboratory test results” (Kruskal-Wallis p<0.01; chi-square p<0.01); and “Perform a comprehensive medication review (CMR) for a patient” (Kruskal-Wallis p<0.01; chi-square p<0.01). Some supporting tasks were not significant, but were performed highly for all groups, including: “Identify drug interactions;” “Manage drug interactions;” and “Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test.”

Table 5 reports the patient care provider domain activities and supporting example tasks, disaggregated by preceptor status, defined by NDSU preceptor during the past year versus not. For preceptors, the highly rated activities that was statistically significant were: “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs” (Kruskal-Wallis p=0.03; chi-square p=0.03); followed by “Implement a care plan in collaboration with the patient, caregivers, and other health professionals” (Kruskal-Wallis p=0.02; chi-square p=0.06); and “Follow-up and monitor a care plan” (Kruskal-Wallis p=0.02; chi-square p=0.03). In each of these cases, preceptors reported that they performed the activities more frequently than non-preceptors.

Table 5

Entrustable Professional Activities (EPAs) of patient care provider domain by preceptor status (n=107)

Patient care EPA description	Mean (SD)		p value
	Do not precept NDSU students [n = 54]	Precept NDSU students [n = 53]	ANOVA	Kruskal- Wallis Test	Chi-square homogeneity Test*
1. Collect information to identify a patient’s medication-related problems and health-related needs.	3.6 (2.0)	4.2 (1.6)	0.11	0.12	0.16
a. Collect a medical history from a patient or caregiver.	2.0 (2.1)	2.8 (2.3)	0.06	0.07	0.01
b. Collect a medication history from a patient or caregiver.	2.7 (2.2)	3.6 (2.1)	0.03	0.02	0.01
c. Discuss a patient’s experience with medication.	3.7 (1.8)	3.9 (1.8)	0.64	0.62	0.60
d. Determine a patient’s medication adherence.	3.5 (1.8)	4.0 (1.9)	0.24	0.11	0.03
e. Use health records to determine a patient’s health-related needs relevant to setting of care and the purpose of the encounter.	3.1 (2.2)	3.8 (1.9)	0.08	0.07	0.09
2. Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs.	3.3 (2.1)	4.2 (1.7)	0.02	0.03	0.03
a. Assess a patient’s signs and symptoms to determine whether the patient can be treated within the scope of practice or requires a referral.	2.2 (2.3)	2.3 (2.2)	0.73	0.76	0.74
b. Measure an adult patient’s vital signs and Interpret the results (e.g., body temperature, pulse rate, respiration rate, and blood pressure).	0.5 (1.1)	1.2 (2.0)	0.02	0.09	<0.01
c. Interpret laboratory test results.	2.3 (2.4)	3.1 (2.2)	0.07	0.07	0.21
d. Identify drug interactions.	4.5 (1.3)	4.6 (1.2)	0.73	0.87	0.76
e. Perform a comprehensive medication review (CMR) for a patient.	2.0 (2.3)	3.3 (2.2)	<0.01	<0.01	0.01
f. Assess a patient’s health literacy using a validated screening tool.	0.3 (0.9)	0.5 (1.5)	0.26	0.47	0.09
g. Compile a prioritized health-related problem list for a patient.	0.5 (1.4)	1.4 (2.2)	0.01	0.02	0.02
h. Evaluate an existing drug therapy regimen.	2.7 (2.3)	3.7 (2.0)	0.02	0.03	0.06
3. Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective.	1.7 (2.3)	2.2 (2.3)	0.29	0.30	0.49
a. Follow an evidence-based disease management protocol.	1.9 (2.3)	2.8 (2.4)	0.04	0.05	0.03
b. Develop a treatment plan with a patient.	1.0 (1.9)	1.8 (2.2)	0.07	0.05	0.22
c. Manage drug interactions.	3.4 (2.0)	3.9 (1.8)	0.12	0.12	0.13
d. Select monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	1.4 (2.1)	2.5 (2.3)	0.01	<0.01	0.06
e. Determine the appropriate time interval(s) to collect monitoring data.	1.1 (2.0)	2.5 (2.4)	<0.01	<0.01	0.01
f. Create a patient-specific education plan.	0.8 (1.7)	2.0 (2.3)	<0.01	<0.01	0.02
4. Implement a care plan in collaboration with the patient, caregivers, and other health professionals.	1.6 (2.1)	2.5 (2.2)	0.04	0.02	0.06
a. Write a note that documents the findings, recommendations, and plan from a patient encounter.	1.6 (2.2)	2.9 (2.3)	<0.01	<0.01	0.01
b. Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test.	4.0 (1.7)	3.7 (2.0)	0.44	0.58	0.78
c. Educate a patient on the use of medication adherence aids.	2.2 (2.2)	2.3 (2.2)	0.96	0.92	0.89
d. Assist a patient with behavior change (e.g., use shared decision making and motivational strategies).	1.1 (1.7)	1.8 (2.1)	0.08	0.11	0.19
5. Follow-up and monitor a care plan.	1.7 (2.2)	2.8 (2.3)	0.01	0.02	0.03
a. Collect monitoring data at the appropriate time interval(s).	1.4 (2.2)	2.5 (2.3)	0.02	0.01	0.06
b. Evaluate the selected monitoring parameters to determine the therapeutic and adverse effects related to the treatment plan.	1.5 (2.2)	2.6 (2.3)	0.02	0.01	0.09
c. Recommend modifications or adjustments to an existing medication therapy regimen based on patient response.	2.0 (2.2)	2.9 (2.3)	0.04	0.05	0.04
d. Present a patient case to a colleague during a handoff or transition of care.	1.2 (2.0)	2.5 (2.3)	<0.01	<0.01	0.02

Less than 5 versus 5 or More Times per Week

NDSU = North Dakota State University

In Table 5, four supportive example tasks were statistically significant. They include: “Collect a medication history from a patient or caregiver” (Kruskal-Wallis p=0.02; chi-square p=0.01); “Perform a comprehensive medication review (CMR) for a patient” (Kruskal-Wallis p<0.01; chi-square p=0.01); “Follow an evidence-based disease management protocol” (Kruskal-Wallis p=0.05; chi-square p=0.03); and “Write a note that documents the findings, recommendations, and plan from a patient encounter” (Kruskal-Wallis p<0.01; chi-square p=0.01)]. Four of the supportive tasks rated highly, but not statistically significant across preceptors and non-preceptors were: “Discuss a patient’s experience with medication;” “Determine a patient’s medication adherence;” “Identify drug interactions;” and “Manage drug interactions.” Another item ranked highly by both preceptors and non-preceptors was “Educate a patient regarding the appropriate use of a new medication, device to administer a medication, or self-monitoring test”.

DISCUSSION

Development of Core EPA domains in pharmacy education is an attempt to create a common vocabulary describing those activities and tasks that a new pharmacist should be able to undertake independently, as they are understood by practicing pharmacists.4,10,11 In doing so, EPAs may also serve as a practice performance measure.1

There is a shortage of assessment data of the Core EPAs in the daily pharmacist practice in various practice settings. The frequency of performing Core EPAs by independent community pharmacy (40%) in North Dakota was followed by hospital (29%), all other settings (21%), and chain community pharmacy (10%). Pharmacists in “all other settings in pharmacy practice” (e.g., long-term care, ambulatory care such as health service, community health center, and medical homes) performed those EPA supportive example tasks more frequently than did hospital, independent, and chain community pharmacists. Perhaps pharmacists in these ‘other settings’ may have more opportunities for direct patient care versus traditional dispensing roles. Significant differences were noted among practice settings for 4 of 5 EPA activities: “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs;” “Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective;” “Implement a care plan in collaboration with the patient, caregivers, and other health professionals;” and “Follow-up and monitor a care plan.” These results are reasonable, since pharmacists practicing in “all other settings” have ample (and perhaps greater) opportunities to see complex patient cases, patients with lower health status, and patients experiencing acute symptoms that require pharmacist interventions that connect with these tasks.

North Dakota clinical pharmacists were more likely than other types of pharmacists (manager, staff) to report performing all five patient care provider EPA activities and supportive example tasks. For instance, “Collect information to identify a patient’s medication-related problems and health-related needs,” and within this activity, commonly occurring EPA supporting tasks were “Discuss a patient’s experience with medication,” and “Determine a patient’s medication adherence.” Similarly, clinical pharmacists in North Dakota were also more likely to report performing the other four activities than other types of pharmacists including: Activity #2 (4 examples), Activity #3 (6 examples), Activity #4 (1 example), and Activity #5 (4 examples), each of which was statistically significant. The findings show that clinical pharmacists were more likely than other pharmacists to perform the activities and supportive examples in the patient care domain. This is not surprising, since clinical pharmacists typically are pharmacy practice department faculty members who set-up their practice to role model advance practice skills and train pharmacy students in the provider care domain. Concomitantly, this finding also suggests that EPA activities and tasks in the patient care domain may be more difficult to characterize and assess in other practice settings.

In our study, the patient care provider EPA activities and supportive example tasks were disaggregated by preceptor status (NDSU preceptor and non-preceptor). Preceptors significantly performed the following more frequently than non-preceptors: “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs.”; “Establish patient-centered goals and create a care plan for a patient in collaboration with the patient, caregiver(s), and other health professionals that is evidence-based and cost-effective;” “Implement a care plan in collaboration with the patient, caregivers, and other health professionals;” and “Follow-up and monitor a care plan.” Preceptors were more likely to role model, teach and practice these activities at their practice sites than pharmacists in other settings, and thus may be more aware than non-preceptors that they perform these activities on a daily basis.

In an attempt to bridge the gap between the didactic and experiential curricula, and between specific AACP domains and EPAs, the EPAs have been mapped at this school of pharmacy to ability-based outcomes (representing AACP domains and CAPE outcomes). The school’s Introductory Pharmacy Practice Experience (IPPE) and Advanced Pharmacy Practice Experience (APPE) programs integrated EPAs into training and assessment of pharmacy students in experiential education. In summary, this study’s survey appraisal represents an early assessment of this process (pilot test). As this integration process continues, the occurrence of these activities and supportive examples are expected to increase.

Limitations

This study has several limitations. First, EPAs require a set of core competencies that are measurable units in the workplace. This study used the 2015-2016 AACP published list of EPA domains, activities and supportive tasks.10,12 Pharmacy academicians typically propose how practice should be in the idealistic world, and not in the real pharmacy world. Several EPAs and supportive examples were not highly, or consistently, reported by the North Dakota pharmacists. These findings suggest that the AACP core EPA list may lack validity for all practice areas of pharmacy. Additionally, it should be noted that the core EPA list were not intended to be assessed in all practice settings, they were intended to guide what a generalist pharmacist should look like upon graduation. For example, some core EPAs may only be practiced and assessed at one practice site. Further research by practice setting is necessary to further examine these issues.

A second limitation is that the 6-point response scale used in this survey may lead to truncated responses, and by extension, descriptive statistics (i.e., means and standard deviations) that under-report the true frequency with which pharmacists complete these activities and supporting examples. If the EPA activities and supportive example tasks are highly representative of all areas of pharmacy practice, a respondent likely performs those tasks on, a daily, rather than a monthly basis. This leads to a large proportion of survey responses that self-report that they perform the activity “5 or more times” in a 30 day period. While the current analysis accounts for this possibility using chi-square tests (should truncation occur) and Kruskal-Wallis tests (in the absence of truncation), such an approach is inferior to an analysis of data collected from a survey whose response scale is not subject to truncation. This is especially the case when the distribution of responses is such that the chi-square and Kruskal-Wallis tests report different outcomes (i.e., reject versus do not reject the null hypothesis). Future studies that examined a smaller range of EPA activities and supportive example tasks would allow for the creation of different response scales that are less likely to be truncated, which in turn would provide unbiased and more efficient statistical estimates.

A third limitation is the relatively low response rate for the survey. As noted earlier in this paper, 990 individuals were initially deemed eligible for inclusion in the survey. However, once responses were collected, 107 of the 457 responses actually meet the study’s inclusion criteria, suggesting that the study’s effective response rate is much higher than the stated 10.8%. Even with a reasonable adjustment to address inclusion criteria-related issues, the response rate for the survey remains low. Future replications of this study which achieve much higher response rates, are necessary to confirm or refute the main conclusions of the current study. Lastly, there could be some element of recall bias in the survey, especially if there were any part-time employee respondents that work too infrequently to accurately recall performed tasks.

A final limitation is that the study’s data were drawn from a single state (North Dakota) that serves a disproportionately rural population. Thus, the study’s finding may not be generalizable to other (especially urban) states. The data also represent a specific cross-section of practice and of time (4th quarter 2017 and 1st quarter 2018) in North Dakota, and as such may not precisely represent the practice of pharmacy in North Dakota over time. Taken cumulatively, the last limitation implies that the 2015-2016 AACP EPAs list potentially needs further refinement and assessment in other states, and across different pharmacy settings, pharmacist positions and preceptor status.

CONCLUSIONS

This study provides empirical evidence (albeit partial) suggesting that EPAs have potential as a useful means to assess outcomes in pharmacy education and practice. Our study sets the stage for future work that further refines and assesses core EPA activities and supportive example tasks to measure the impact of how this process relates to outcomes of care.