Entrustable Professional Activities (EPAs) for Global Health.

PURPOSE: As global health education and training shift toward competency-based approaches, academic institutions and organizations must define appropriate assessment strategies for use across health professions. The authors aim to develop entrustable professional activities (EPAs) for global health to apply across academic and workplace settings.
METHOD: In 2019, the authors invited 55 global health experts from medicine, nursing, pharmacy, and public health to participate in a multiround, online Delphi process; 30 (55%) agreed. Experts averaged 17 years of global health experience, and 12 (40%) were from low- to middle-income countries. In round one, participants listed essential global health activities. The authors used in vivo coding for round one responses to develop initial EPA statements. In subsequent rounds, participants used 5-point Likert-type scales to evaluate EPA statements for importance and relevance to global health across health professions. The authors elevated statements that were rated 4 (important/relevant to most) or 5 (very important/relevant to all) by a minimum of 70% of participants (decided a priori) to the final round, during which participants evaluated whether each statement represented an observable unit of work that could be assigned to a trainee. Descriptive statistics were used for quantitative data analysis. The authors used participant comments to categorize EPA statements into role domains.
RESULTS: Twenty-two EPA statements reached at least 70% consensus. The authors categorized these into 5 role domains: partnership developer, capacity builder, data analyzer, equity advocate, and health promoter. Statements in the equity advocate and partnership developer domains had the highest agreement for importance and relevance. Several statements achieved 100% agreement as a unit of work but achieved lower levels of agreement regarding their observability.
CONCLUSIONS: EPAs for global health may be useful to academic institutions and other organizations to guide the assessment of trainees within education and training programs across health professions.

Global health education and professional training programs are increasing and evolving worldwide. To illustrate, in the United States, more than a quarter of medical and dental students and a growing number of nursing and pharmacy students participate in a global health experience. [1-4] This growth in student interest has led to an increase in U.S. medical and pharmacy residency programs offering a global health rotation. [5,6] As popularity grows, public health programs are also defining competencies for graduates specializing in global heath while nursing programs worldwide are incorporating global health competencies into their curricula. [7,8] Outside of the United States, many academic institutions, including those in Australia, the United Kingdom, Russia, and Germany, are also incorporating global health into health profession curricula. [9-12] The expansion of global health training has led to the emergence of several global health competency frameworks by different professions, including medicine, dentistry, and nursing. [13-17] Recently, the Consortium of Universities for Global Health (CUGH) developed and published interprofessional global health competencies. This work has led to the creation of complementary competencies by the Association of Schools and Programs of Public Health (ASPPH) for graduates concentrating in global health. [18,19] As education and training programs continue to shift toward competency-based approaches, there is a need to determine the best approach for assessing these competencies across professions. [20]

Interprofessional education and collaborative practice in global health are recognized by the World Health Organization (WHO) and other stakeholders as essential for achieving the sustainable development goals. [21] Collaborative practice has been shown to improve health outcomes, improve patient and provider satisfaction, and reduce health care costs. [21] Interprofessional education is a prerequisite for collaborative practice but currently remains a significant gap in most global health education and training efforts. [22] A challenge in implementing and sustaining interprofessional initiatives may be a lack of common assessment methods that can be used by multiple professions. Despite these challenges, academic institutions are continuing to focus on interprofessional education and working to understand how to best assess and evaluate trainee experiences. [20]

Entrustable professional activities (EPAs), defined as discrete activities suitable for entrustment to a trainee after obtaining sufficient competence, were developed to operationalize the application and assessment of competency-based education and training. [23] Several health professions, including medicine, pharmacy, and nursing, as well as some medical subspecialties, have developed and adopted EPAs that describe the essential work of the profession. [24-28] Countries using EPAs for health professions education include Australia, Canada, India, the Netherlands, New Zealand, and the United States, among others. [29-31] As competency-based educational frameworks are adopted for global health education, EPAs for global health are a logical next step. These could help educators translate, use, and assess competencies—both within classroom settings for trainees and in the workplace for the purpose of continuing professional development. A lack of clarity regarding the essential elements of global health work may result in unclear performance expectations, which can have significant implications on the health needs of local communities. [32] EPAs for global health may also help ensure social accountability so that global health education is conducted in a collaborative, equitable manner that balances the priorities of the sending institution with local needs and customs. [33] The objective of this study was to develop a list of EPA statements describing global health work that can be applied across health professions in educational and workplace contexts.

Method

We conducted a four-round modified Delphi study online using the Qualtrics survey application (copyright 2020, 2019 versions; Qualtrics, Provo, Utah) from June through November 2019. The Delphi technique is a means of building consensus by administering a series of surveys to panelists who all have expertise within a particular topic area. [34] While the number of panelists can vary substantially, most Delphi studies involve 15–20 experts. [35] We endeavored to enroll 28–32 experts in this study to characterize the breadth of global health activities. We recruited leaders in global health education and practice from medicine, nursing, pharmacy, and public health. Invited participants met one or more of the following inclusion criteria:

had been recognized within their health profession as a global health expert,

had served as the senior or lead author on highly cited global health papers, and/or

had held a leadership role in a global health organization or academic institution with a global health program.

We identified a total of 111 global health experts across professions. From this list, we invited potential participants from each profession until at least 6 but not more than 8 participants from each profession accepted our invitation. We applied no exclusion criteria. We strove for diversity of geographic representation as well as strong representation from low- to middle-income countries (LMICs). Participants received a stipend of 250 U.S. dollars for participating in all rounds, and they had 2 weeks to respond to each survey.

In the initial round, we collected demographic information and asked participants to list essential global health work activities. We provided Koplan and colleagues’ definition of global health: “an area for study, research, and practice that places a priority on improving health and achieving equity in health for all people worldwide.” [36] Participants could also include activities that fell outside this definition. We defined EPAs as “tasks or responsibilities to be entrusted to the unsupervised execution by a trainee once he/she has attained sufficient competence,” and we reminded participants that these statements were intended to describe the essential work of professionals in the field. We provided example EPAs in medicine and pharmacy [24,25] as well.

We analyzed survey data at the end of each Delphi round. During round one, 2 investigators (D.R.S., S.A.D.) used a conventional content analysis approach [37]; that is, they independently coded the data in Microsoft Excel (version 16.0, 2016; Microsoft, Redmond, Washington) by, first, using in vivo coding, followed by code mapping, for qualitative data analysis. [38] Discrepancies were reconciled before discussing the findings with the entire research team. Coded participant responses formed the basis of initial draft EPA statements. Next, we cross-referenced these initial EPA statements with a literature review of global health competencies to determine whether the former aligned with established competencies. We used descriptive statistics for quantitative data analysis in subsequent rounds to determine consensus, defined a priori as participant agreement equal to or greater than 70% for all statement attributes. Participants received the individual and collective results after rounds two and three to consider when completing, respectively, rounds three and four.

During round two, participants rated each draft EPA on the importance of the activity to global health work and the acceptability of the wording. Participants rated importance on a 5-point Likert-type scale, and they rated wording as either “acceptable as written” or “needs improvement.” We provided a comment box for participants to suggest revisions and leave other comments. Only those EPA statements which were deemed “Very important” (4) or “Important” (5) by a minimum of 70% of the participants were elevated to round three. In round two, we also asked participants to provide their perspectives on 3 terms—“stakeholder,” “community,” and “local”—as these were used differently in source documents. We probed if “stakeholder” was the most appropriate term to describe the variety of individuals and organizations with which one may interface during global health activities. We also queried participants as to whether “local” or “community” best described activities carried out at the national or subnational level. Statements that achieved a consensus of 70% or higher for both importance and “acceptable as written” in round two moved forward. In round three, we presented consolidated, eliminated, or modified EPA statements; changes were all based on participant feedback received in round two.

During round three, panelists were asked to rate the revised EPA statements in terms of importance and wording. Statements that achieved equal to or greater than 70% consensus for both importance and “acceptable as written” in round three moved forward. In addition, participants rated all proposed EPA statements, including those achieving consensus in round two, for their relevance to global health work, regardless of health professional discipline. As with earlier rounds, participants rated relevance using a 5-point Likert-type scale. Only those statements rated “Relevant to all” (5) or “Relevant to most” (4) by a minimum of 70% of participants moved forward to round four.

In round four, participants evaluated each of the proposed EPA statements, using a yes/no response, on 2 additional attributes: (1) whether the EPA statement represented a unit of work that could be assigned to a trainee to perform and (2) whether it was observable. Only those statements which were considered by a minimum of 70% of the participants to be both a unit of work and observable were included in the final set of EPA statements.

At the end of all rounds, we categorized the final set of EPA statements into domains that aligned with descriptors from participant comments from the first and subsequent rounds. The survey instrument is available as Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B47. The Institutional Review Board of the University of North Carolina at Chapel Hill approved this study.

Results

Of the 55 global health experts who were invited, 30 (55%) agreed to participate, and of those 30, 24 (80%) participated in all 4 rounds of the Delphi survey. Those individuals represented the fields of medicine, nursing, pharmacy, and public health (Table 1). Participants had an average of 17 years of global health work experience and spent most of their time in practice and program delivery, followed by education/training and supervising others. Participants were from Australia (2), China (1), Costa Rica (1), Kenya (2), Malaysia (1), Mexico (3), Namibia (1), Rwanda (1), Switzerland (1), Tanzania (1), the United Kingdom (1), the United States (14), and Zambia (1). Twelve (40%) participants were from LMICs. In addition to affiliations with academic institutions, participants came from organizations including the Bill & Melinda Gates Foundation, Partners in Health, Public Health Institute, United States Agency for International Development, and WHO, among others.

Table 1

Demographics of 30 Experts Participating in a Delphi Process for Developing Entrustable Professional Activities (EPAs) for Global Health and Response Rate by Round, 2019

Each Delphi round had a 90% or greater participation rate. Feedback from round one included a total of 225 suggested essential global health work activities: 149 (66%) fit in the definition of Koplan et al [36] (provided above), and 76 (34%) fell outside of this definition. We consolidated this feedback, based on coding, into 29 EPA statements for consideration during round two. Major themes that formed the foundation of the initial EPA statements included the following: capacity building, program planning, cross-cultural communication, health equity and promotion, partnership development, data analysis, and strategic planning (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B47). We cross-referenced the initial EPA statements against 207 global health competencies derived from a comprehensive literature review, including the competency frameworks of CUGH, ASPPH, and other health professions.

Comments from rounds one and two indicated that all global health EPAs need to incorporate certain principles. These included the following: the need for mutual respect and collaboration, consideration of cultural context, ethical behavior, and cross-cultural communication. Participants favored the term “local” over “community” in EPA statements to reinforce that global health is relevant to a variety of settings, including those at the international, national, and subnational levels. A majority also agreed that “stakeholder” was the best descriptor of the personnel and programs one might encounter and work with during global health activities. When suggesting wording modifications, several participants noted that the language of the EPA statements should reinforce the importance of mutual collaboration and respect between partners from the global north and global south.

Twenty-two statements achieved consensus with 70% or more of participants agreeing with the final wording and that the statement represented a unit of work that is important, relevant, and observable (see Table 2). Regarding the attribute of importance, all EPA statements had greater than 80% agreement, and some statements achieved greater than 95% agreement. The statements with the highest agreement were as follows:

Table 2

List of Global Health EPA Statements and Panelist Agreement Rates by Attributea

“Introduce and strengthen workforce development strategies,”

“Analyze connections between local health issues and larger global health agendas,”

“Prioritize the needs and resources of local populations to address inequities,”

“Identify and address ethical dilemmas in global health activities,” and

“Apply discipline-specific knowledge and skills to improve health and prevent disease.”

For relevance to other health professions, a few statements were just above the ≥ 70% agreement threshold, including “Evaluate how to strengthen health systems through design and systems thinking.” Seven statements had 100% agreement for being units of work that could be assigned to a practitioner, but there were lower levels of agreement as to whether some statements were observable.

Discussion

To our knowledge, this is the first attempt to develop a set of EPA statements for global health. To date, many of the recommendations for improving global health education have focused on program content and delivery strategies. [22] Less is known about how assessment approaches, especially those that are interprofessional, can advance global health training. Competency-based assessments most commonly reported in global health include self-reflections, topic-specific presentations, and the use of pre–post quantitative self-evaluations. [32] Global health EPAs can serve as a practical, work-based means of assessing trainees, and supervisors can link the EPAs to established global health competencies and curricular milestones. The use of EPAs can also extend beyond educational programs into workplace professional development programs to strengthen the global health training continuum. [39] Academic institutions and organizations can use EPAs not only for the purpose of assessing individual performance over time but also for evaluating and designing training programs and program goals. [40]

The number of EPA statements reaching consensus in our study is similar to the number developed for medicine, pharmacy, and nursing. [24-28] To facilitate the use of the global health EPAs across professions, countries, and settings, we categorized the EPAs into 5 domains, all characterized as roles: partnership developer, capacity builder, data analyzer, equity advocate, and health promoter. These roles correspond with and encompass the domains of competence proposed by CUGH and ASPPH, among others. [18,19,41,42] Global health EPA statements describe activities that require overlapping areas of competence. With the alignment of global health EPA domain roles to previously established global health competency domains, academic institutions and organizations can use these EPA statements in conjunction with familiar global health competency frameworks.

EPA statements related to the global health domain roles of equity advocate and partnership developer had the highest level of agreement among participants in terms of importance and relevance to most or all professional disciplines. As a foundational principle at the center of global health work, equity may serve as a starting point of discussion across professions even before students and trainees engage in global health activities. While the desire to advance health equity is a primary reason to pursue a career in global health, training programs often do not focus on this important construct and thereby omit important conceptual, theoretical, and patient-centered aspects of health equity. [43,44] Interprofessional and multidisciplinary collaboration is required to effectively address the complex structural and social determinants of health that contribute to inequities in access, availability, and quality of care. [45] Collaboration is at the core of many of the global health EPAs, and stakeholders must discuss factors that influence it, such as purpose, resources, process/structure, and environment. [46] Although EPAs are often used in the educational context to promote individual entrustment, the collaborative nature of global health activities must not be forgotten. Equity and collaboration reflect the intentions behind the sustainable development goals (SDGs), so embedding these in the global health EPAs is critical. Global health work is driven by how countries choose to address the SDGs.

Although all the EPA statements developed in this study met the criteria for consensus, there was less agreement on whether the EPA statements represented observable units of work. Entrustment decisions require observation to evaluate the level of supervision needed for an activity; thus, determining whether a statement captures work a supervisor can observe is one of the factors in validating the quality of EPAs. [47] Direct observation of some of the global health EPAs may be difficult. Some of the EPAs may require indirect forms of evidence of achievement, including evidence produced by the trainee, such as reports, proposals, and presentations. Other indirect forms of evidence may involve self-assessment methods, including trainees soliciting the perspectives of others to reflect on how they would evaluate their own performance. [48] Gathering others’ views, coupled with reflective practice, may also be applied to faculty or supervisors to develop their observation and feedback skills. [49]

While EPAs typically describe clinical activities that supervisors entrust health professionals to perform, trust is a core concept in global health work activities as well. Health service and program delivery across the world relies on trust between individual and institutional stakeholders. [50] As trust requires time to develop, programs may have to make a structural shift—from providing short-term experiences to, instead, creating more longitudinal experiences—to foster the development of trust among the trainee, supervisor, and local stakeholders. [51]

Although we gathered input from panelists with diverse perspectives across countries and professions, the EPA statements may need to be adapted for different settings. Cultural context is an important consideration, especially when evaluating the global health competence of learners. [52] Some programs include local public health in their definition of global health, and others are exploring “global to local” opportunities, including interprofessional experiences, that help demonstrate the applicability of global health work in a local environment. [53,54] Local opportunities for global health work are an important consideration for those who pursue global health postgraduate training, most of whom receive the majority of their instruction at their home institution. [43] The proposed EPAs can be used as a framework to enhance both community and global engagement as trainees and professionals reflect on how global issues have local implications. [55] More research is needed to further validate the use of these global health EPA statements. Continuing this work will inform best practices, including how the global health EPAs translate across cultures and into educational and practice settings.

There is also a need to translate each global health EPA into a list of discrete, supportive tasks across different settings. This delineation could provide clarity regarding expectations for both the trainee and supervisor. Such tasks might be context specific. For example, statements of supportive tasks related to the EPA, “Identify available resources and support optimal utilization,” could specify how others define these terms (i.e., resources, utilization) within different settings. Testing and evaluating the use of the global health EPAs across settings will provide more guidance and further specificity on the components [39] of each statement. Mapping the global health EPAs to curricular milestones for learners at different levels of development is another consideration.

A key strength of this study is the diversity of participants and strong representation from individuals from LMICs. In addition, we garnered a 90% or greater response rate during each Delphi round. There are several study limitations to consider as well. Given the inherent limitations of EPA descriptors, not all essential global health activities may translate into EPA statements. While participants considered all 22 EPA statements as units of observable work, future research needs to evaluate how best to observe and measure them. Another limitation is the possibility that we did not capture the breadth of perspective needed to fully inform the development of global health EPAs. Further, we did not have representation from all the possible global health stakeholder groups, which may influence the applicability of the results to other professions. Nonresponse bias is another potential limitation as the study participants may be different from those who did not participate in some critical, but unknown ways. Of the 55 invited experts, 25 either did not respond to or declined our request to participate in the study. In some cases, it is likely that potential participants never received our email invitation. Nonetheless, the reasons why potential participants either ignored or declined our invitation are unknown.

Conclusions

Global health work is interprofessional and requires individuals with different knowledge and skills to solve complex problems. To this end, global health workers and learners must possess a common set of competencies. EPA statements are a means to describe the key responsibilities of global health workers and to operationalize competency assessment. The 22 EPA statements developed in this study have the potential to advance global health education and training across health professions and countries by providing a common language regarding the work that trainees must perform.

Acknowledgments:

The authors would like to acknowledge Olle ten Cate, PhD, Professor of Medical Education at University Medical Center Utrecht, the Netherlands, and Margaret Bentley, MA, PhD, Carla Smith Chamblee Distinguished Professor of Global Nutrition and Associate Dean of Global Health at the University of North Carolina at Chapel Hill Gillings School of Global Public Health, for their review and contributions. The authors would also like to thank the American Association of Colleges of Pharmacy (AACP) for providing funding for this study.