Training for Health System Improvement: Emerging Lessons from Canadian and US Approaches to Embedded Fellowships.

The benefits of supporting experiential learning for improved health and societal outcomes have been recognized in many countries. A number of funding organizations have developed competitive funding opportunities to support experiential learning in health system organizations outside of the traditional university setting. AcademyHealth in the US is an early innovator that pioneered the Delivery System Science Fellowship (DSSF) and inspired Canada's creation of the Health System Impact (HSI) Fellowship program. The DSSF and HSI Fellowship have similar objectives: to improve the career readiness of doctorally prepared graduates and to build research capacity within health system organizations. However, the programs have taken different approaches to achieve these objectives and operate in different healthcare systems. This paper outlines the two models of embedded fellowships, analyzes their commonalities and differences, discusses lessons learned and suggests future directions for health services and policy research training.

Introduction

Health services and policy research (HSPR) is the innovation engine of a healthcare system (Tamblyn et al. 2016a). The Canadian Institutes of Health Research Institute of Health Services and Policy Research (CIHR-IHSPR) defines it as the field of scientific investigation that generates evidence on how to invest in programs, services and technologies that maximize health and health system outcomes (CIHR-IHSPR 2016), and it is similarly defined by AcademyHealth and the Agency for Healthcare Research and Quality (AHRQ) in the US (Mor and Wallace 2018). Countries worldwide are grappling with rising expenditures, inefficient service delivery and suboptimal patient outcomes, and doctoral graduates in HSPR have the skills to tackle such challenges and the potential to make significant contributions to health system improvement (Bornstein et al. 2019; Brown and Nuti 2016; Tamblyn et al. 2016b). The contributions of doctorally trained health services and policy researchers can come through investigator-driven research in academic institutions and be published in peer-reviewed journals, as is typical in the basic sciences. But their contributions can also take the form of applied research driven by the health system's needs that is conducted in real-world settings such as healthcare delivery organizations, and their research can be used to inform the policies, programs and services that directly impact people and populations (Atkins 2018; Atkins et al. 2017; Boaz et al. 2015; Chafe and Dobrow 2008; CIHR 2013; Hanney et al. 2003; Lomas and Brown 2009; Marshall et al. 2016). To date, however, university-based doctoral training curricula and research funding programs in Canada and the US have tended to emphasize the former type of contribution by encouraging and rewarding careers within the academy. To ensure that HSPR drives tangible improvement on the ground, there is emerging consensus in both countries that doctoral training and research funding programs must modernize to prepare graduates for embedded research and leadership roles within health system organizations and emerging learning health systems (LHSs) (Forrest et al. 2018; Tamblyn et al. 2016b).

CIHR-IHSPR in Canada and AcademyHealth in the US are two organizations that are experimenting with new models of post-doctoral fellowships that train for real-world impact by embedding HSPR trainees directly within systems and organizations at the coalface of policy making and service delivery. The underlying program logic is twofold: (1) increased support and recognition for post-doctoral training that occurs within applied health system settings – such as within health policy and service delivery organizations – will improve the success and impact of these individuals in a broader range of employment roles and settings and (2) embedding research expertise directly within health system organizations to help tackle complex and high-priority challenges will, as postulated by theories on integrated knowledge translation and participatory research (CIHR 2012; Cornwall and Jewkes 1995; Lomas 2000), lead to the creation of more relevant, useful research findings that are more likely to inform policy and practice. In the UK, this embedded researcher model is referred to as the “researcher-in-residence” and is viewed there as a participatory research approach that can help bridge the “know-do gap” and increase the influence of HSPR on policy and service delivery (Marshall et al. 2014, 2016). Given the small number of embedded fellowship and applied research training programs that have been evaluated to date, which come primarily from Canada and the pre-LHS era (see, for example, CIHR [2016]; Martens [2008] and the other articles in the 2008 Healthcare Policy/Politiques de Santé special issue on Canada's Regional Training Centres; and Paradis et al. [2017]), it is useful to compare two embedded fellowship models: the recently created CIHR-IHSPR model and the AcademyHealth fellowships. These two models have similar objectives and share several common elements but also differ in important ways and operate in different countries and contexts. They provide a valuable opportunity to compare approaches and reflect on what appears to be working and why. The purpose of this commentary is to compare the objectives, attributes and outcomes of the CIHR-IHSPR and AcademyHealth embedded fellowship models, discuss the lessons learned and introduce future directions for HSPR training and LHSs.

Description of the Canadian and US Embedded Fellowship Programs

The Canadian Approach: The Health System Impact Fellowship

Core Objectives

The Health System Impact (HSI) Fellowship program was designed in 2016 to fill a gap in Canada's HSPR doctoral and post-doctoral training programs. The Training Modernization Working Group (TMWG) of the Canadian Health Services and Policy Research Alliance (CHSPRA) found that although a very high proportion of Canada's HSPR PhD graduates enter careers in research and administrative jobs in health system organizations rather than in professorial positions in university departments (see McMahon et al. [2019b] in this issue), universities are continuing to train them in traditional ways, with a focus on academically relevant skills and university careers. Accordingly, through consultations with the TMWG and key stakeholders across the country, CIHR-IHSPR designed a new, competitive national program of embedded fellowships that prepares PhD trainees and post-doctoral fellows for careers and impact in a diverse array of health system settings. By enhancing the employability and effectiveness of Canada's PhD graduates for careers within the health system, the HSI Fellowship program also seeks to make a longer-term contribution to the development of LHSs in Canada.

Innovative Features

The HSI Fellowship program has five distinctive features: (1) It involves a modernized, pan-Canadian curriculum focused on a set of enriched core competencies (e.g., leadership, change management) designed to prepare fellows for success in careers outside the university (see Bornstein et al. [2018] and McMahon et al. [2019b] in this issue), and they include a dedicated professional development allowance to support fellows' pursuit of these core competencies; (2) It includes substantial experiential learning placements that embed fellows in a “host partner organization” that also provides co-funding for a program of work designed to address a high-priority challenge (an “impact goal”) identified by the organization; (3) It involves co-supervision and mentorship by a health system leader and an academic leader; (4) It permits fellows to protect up to 30% of their time for academic research within the university in order to stay at the forefront of their field and bring the latest academic developments to bear on “real-world” challenges; and (5) It brings all the funded fellows, and their health system and academic mentors, together in a national cohort whose annual in-person meeting and quarterly webinar sessions contribute to training, peer-to-peer collaborations and cross-cohort learning. These innovative features come together in a two-year fellowship for post-doctoral fellows and a one-year fellowship for doctoral trainees that are jointly funded by federal and provincial funders and by the host partner organizations.

Partners Involved

In the program's first two years, 62 health system organizations and 23 academic training programs across the country have hosted fellows. The organizations involved have included healthcare delivery organizations, federal and provincial government agencies, provincial and regional health systems, public health offices, health charities, consulting firms, health professional associations and non-university research institutes. Their number and diversity are an encouraging indication of the demand for doctoral-level skills within Canada's health systems. Together with the funding partners, these organizations have, since 2017, funded 95 HSI Fellowship awards.

The US Approach: The Delivery System Science Fellowship

In contrast to the Canadian initiative that is jointly spearheaded by an alliance of funders, training programs and health system organizations, including the federal funder of HSPR (CIHR-IHSPR), the US program is led by AcademyHealth, a private, professional organization of health services and policy researchers in the US. AcademyHealth convened the Health Services Research Consortium over the course of several years to support dialogue between directors of HSPR training programs and employers of health services and policy researchers. Examining changing membership trends, AcademyHealth identified a shift in the employment settings of its researcher members from primarily academic jobs toward the inclusion of positions in the healthcare industry (Menachemi et al. 2018). At the same time, it also found that its employer members faced growing hiring challenges. Based on this work and the expressed needs of delivery systems, the Delivery System Science Fellowship (DSSF) was launched in 2012. Background for the program and examples of fellow experiences are described in detail by Kanani et al. (2017).

To date, 29 post-doctoral fellows have completed their training as DSSFs and five individuals started their fellowship in 2018 (AcademyHealth internal DSSF program records).There is an annual application process and review of these applications by a national advisory committee and participating host sites, which then reach out to those highly ranked applicants who best meet their needs. Key objectives of the DSSF have been to:

provide a real-world learning experience in host delivery systems;

generate insights by answering relevant, operational questions faced by delivery systems and developing new measures and methods to support embedded work; and

expand the available research workforce by eliminating the gap in available training experiences.

Mentorship of fellows is structured to include at least three components: an on-site mentor to provide guidance on program activities, an external individual with shared interests who can advise fellows on their career track and broader professional development issues and a peer-mentoring component through monthly fellow calls. Additionally, quarterly webinars and an annual in-person meeting promote shared learning and peer mentorship. Of the 29 graduate fellows, approximately one-third stayed in the fellowship more than a year and approximately half have been hired permanently by their host site.

This program is a joint venture executed by AcademyHealth and sponsored by delivery system partners. AcademyHealth markets the opportunity annually, manages the application review process, supports health systems and fellows during the hiring and onboarding, serves as the national home for convening DSSF holders (both virtually and in person) and provides a respected, nationwide community of fellows. Salaries, benefits, travel to AcademyHealth meetings, requisite work tools and mentoring are provided by the delivery system for a minimum of one year, although several health systems have extended fellows for a second year and/or eventually hired them full time. This arrangement delivers a win-win experience whereby a delivery system can assess whether or not the fellow is a good organizational fit and the fellow can make an informed decision on whether operationally relevant research is compatible with his or her career goals. Particularly noteworthy is that this program was launched seven years ago and has never received a grant or other support from the US government or philanthropic sources. It has been sustainable due to hosts' demand for fellows and their willingness to cover the fellowship costs.

The commitment of DSSF partners in terms of resources, access to data, mentoring and support for attendance at the AcademyHealth Annual Research Meeting requires health systems to make significant investments in an on-the-job training program. Fellows have now been placed in 21 delivery systems across the US that provide care to over 100 million people, including some of the most well-known delivery networks (e.g., Kaiser Permanente, Intermountain Healthcare and Geisinger). Some of the DSSF host sites are now funded by the AHRQ (a governmental agency) and the Patient-Centered Outcomes Research Institute (a non-governmental organization funded by public dollars) through a $40 million initiative over five years to support the training of clinician and research scientists to conduct patient-centred outcomes research within LHSs (AHRQ 2018).

Comparing the Approaches

Although the DSSF model directly inspired the design and creation of Canada's HSI Fellowship, the two fellowships differ in a number of key features, as shown in Table 1. The contexts – including the political, societal and economic contexts – in which the two fellowships emerged and are situated also differ, but an in-depth comparison of those is beyond the scope of the present paper and is recommended as an area for future research attention.

Table 1.

Comparison of key features of the Canadian and US embedded fellowships

Program feature	AcademyHealth Delivery System Science Fellowship	CIHR-IHSPR Health System Impact Fellowship
Eligibility	Post-doctoral – multiple disciplines (e.g., PhD, MD, DrPH)	Doctoral trainees and post-doctoral fellows (PhD only or MD + PhD)
Host site mandate and eligibility	Host sites are US health service delivery organizations and integrated delivery systems and other sites with a mandate to deliver care (e.g., healthcare, dental care), identified and deemed eligible by AcademyHealth	Canadian health system and related organizations (public, private, not-for-profit) with a health-related mandate that are not universities or university-based research institutes, identified by CIHR or the applicant
Examples of host sites	Atrius Health; Geisinger Health System; Kaiser Permanente Northwest; US Department of Veterans Affairs; additional examples available at: https://www.academyhealth.org/dssf.	Ministries of health; regional health authorities; hospitals; health charities; health-focused consulting firms; additional examples available at: http://www.cihr-irsc.gc.ca/e/50612.html.
Duration	1 year minimum (may be longer, as determined by host site)	Doctoral: 1 year Post-doctoral: 2 years
% time embedded	100%	Doctoral: 60% (minimum) Post-doctoral: 70% (minimum) The balance of time, if any, is protected for academic research with the fellow's academic supervisor.
Supervision	Senior leader of host health system organization	Co-supervised by a senior leader in host health system organization and an academic at a Canadian graduate training program
Mentorship	Formal mentoring and professional development training plan created jointly by supervisor and fellow, discussed and reviewed regularly	Formal mentoring and professional development training plan jointly created by co-supervisors and fellow, discussed and reviewed regularly
University involvement	Not required	Required. Fellow must have a formal affiliation (as a registered post-doctoral fellow with a Canadian university) and an academic supervisor who has an academic appointment with a university-based graduate training program and has expertise in HSPR or a related field. Fellows may protect up to 40% of their time for academic work.
Professional development	Program level (through AcademyHealth) and host site level	Program level (through CIHR), fellowship level (dedicated training allowance) and host site level
Program seminars	Annual meeting and AcademyHealth-organized webinars	Annual National Cohort Retreat and quarterly CIHR-organized webinars
Amount	Determined by host site but must meet the National Research Service Awards' stipend minimum (as of January 2019, min. US$50,004[§])	Doctoral: $45,000 stipend + $5,000 training allowance Post-doctoral: $70,000 stipend (per year) + $7,500 training allowance (per year)
Funder	Host site (100%)	Co-funded by CIHR and provincial funders (together provide 70% of each award's funds) + host partner organization (provides 30% of the award's funds)
Number of host sites*	21	62
Number of funded fellows*	34	95

as of December 2018

https://grants.nih.gov/grants/guide/notice-files/NOT-OD-19-036.html

Lessons Learned and Future Directions

The comparison of fellowship models (Table 1), a review of program evaluation reports and in-depth discussions between the heads of CIHR-IHSPR and AcademyHealth and their fellowship advisory committees have revealed four principal lessons learned thus far and four future directions for HSPR training and LHSs.

Lessons Learned

1. A Flexible Program Design Fosters Uptake and Allows Adaptability.

Every health system is a complex entity composed of a diverse mix of organizations. For an embedded fellowship program to gain traction and widespread uptake, it must have clear objectives but also a flexible design that permits adaptation to the local organizational context. AcademyHealth provides concrete guidance to its host partner organizations on what the fellowship objectives are but not on how to operationalize the objectives. Organizations, most of whom have a long-standing tradition of research and existing research capacity, can tailor the implementation and execution of the fellowship to their unique needs and the interests and skills of the trainee. Similarly, CIHR-IHSPR provides clear objectives, eligibility conditions and peer review criteria but allows flexibility to fellows and host organizations to tailor their projects, training plans and the allocation of their embedded and academic time to their unique goals and needs. Given the diversity of organizations involved in the HSI Fellowship program and differences in these organizations' previous experience with and capacity for research, additional guidance regarding research processes and outputs, strategies for dual health system and academic mentorship and models of successful partnerships with universities could be beneficial.

2. The Culture and Operations of University Institutions and Health System Organizations Differ, and Acculturation Takes Time and Attention.

It has been recognized in both Canada and the US that the work environments in health system organizations differ from the university settings in which doctoral students are currently trained and that the skill set required to contribute to and lead transformative change within these organizations differs from those emphasized in most doctoral programs (Bornstein et al. 2018; Hamelin and Paradis 2018; Kanani et al. 2017; Paradis et al. 2017; Reid 2016; Tamblyn et al. 2016b) and from those currently prioritized and rewarded within academic institutions (Hunter 2019; Marshall et al. 2016). Many PhD graduates have extensive academic training but lack experience working in other settings. It can take months to grasp the language and cadence of an organization and even longer to learn to work effectively in the operation and contribute value to it (Marshall et al. 2016; Sheps et al. 2008). A different set of competencies than those conventionally emphasized in university training programs is needed to prepare graduates for success in health system organizations (see Bornstein et al. 2018; Burgess et al. 2018; Forrest et al. 2018; Paradis et al. 2017; Reid 2016). CIHR-IHSPR experimented with one- and two-year post-doctoral awards and learned that one year is too short to make a real impact. In its subsequent rounds, it offered two-year fellowships only. DSSF host sites have had a similar experience, and, although not mandated by the program, some are now offering only two-year fellowships. Additional guidance to host sites regarding promising strategies for onboarding fellows and creating initial conditions for long-term success, to health system and academic supervisors regarding the importance of mentorship (Bornstein et al. 2019; Hamelin and Paradis 2018; Sambunjak et al. 2010) and to fellows regarding the policy- and decision-making process may accelerate acculturation.

3. The Cohort Effect Is Valuable and Worth Fostering.

The DSSF and HSI Fellowship have both deliberately fostered connectivity among fellows and the establishment of a community of peers. Both programs provide structured opportunities via annual meetings and regular webinars to help the fellows establish relationships and collaborations. This helps with acculturation and helps fellows learn from one another's experiences, strategies and advice. The efforts to establish these cohorts have proven valuable in both countries. In Canada, the structured opportunities, such as the National Cohort Retreat, have catalyzed collaborations among fellows to develop research studies (see, for example, the paper included in this special issue by Blanchette et al. [2019]; Sim et al. [2019]; Weijs et al. [2019]), to submit multi-authored abstracts for panels at national conferences and to organize informal regional and thematic communities of practice.

4. The Fellowship Programs Have Legitimized Career Pathways Within Health System Organizations.

There is considerable demand for doctoral-level talent within health system organizations and demand from trainees for real-world training opportunities. Traditionally, however, trainees have been encouraged to pursue academic careers within universities, and anything else has been perceived (within academic circles) as second class. Similarly, health system organizations have lacked structured opportunities to recruit high-calibre doctoral talent. National championship of the DSSF has legitimized the choice to work within the health system and recognized it as a credible career pathway within the broader HSPR enterprise. It is hoped that the HSI Fellowship is helping to accelerate a similar culture change in Canada, a shift that was sparked almost two decades ago with the creation of innovative training programs such as Quebec's Training Program in Transdisciplinary Research on Public Health Interventions (Hamelin and Paradis 2018; Paradis et al. 2017), the Capacity for Applied and Developmental Research and Evaluation (CADRE) Regional Training Centres (Conrad 2008; Martens 2008; Sheps et al. 2008) and others that made significant contributions to building capacity for applied and embedded research but that were grant-funded with fixed funding terms that have ended. The DSSF and HSIF Fellowship have provided a structure to meet, funnel and shape current demand from both health system organizations and doctoral trainees.

Future Directions for Health Services and Policy Research Training and Learning Health Systems

1. Evaluate Programs for Continuous Improvement.

There is strong enthusiasm across much of the Canadian health system for how HSPR training modernization can support emerging LHSs and overall health system improvement. However, major initiatives in human capital development, such as the DSSF and the HSI Fellowship, are far too rarely evaluated. There are notable exceptions upon which to learn and build, such as those identified earlier in this paper. So, to contribute to the currently limited evidence on impact-oriented HSPR training, it will be important to evaluate their impact at regular intervals, adjust their parameters for continued relevance to employment trends and health system needs and report on lessons learned. The papers in the present issue reflect initial efforts to learn from early experiences with the HSI Fellowship program, including whether and how fellows' enriched core competencies evolved over the course of the first year (see McMahon et al. 2019a), what contributions fellows made to their host partner organizations (see Blanchette et al. 2019) and the role and value of mentors in embedded research settings (see Bornstein et al. 2019). Over time, as the Canadian and US programs expand and evolve, it will be important to investigate questions such as the following: Does hosting a fellow generate spillover effects for an organization's internal culture of research and continuous learning? What are the impacts of the dual health system and academic supervisor approach on the quality of the research and training experience? Do embedded research programs successfully enhance career opportunities for fellows and increase their capacity to drive health system improvement? How can we increase the implementation of embedded research to inform organizational decision-making? Are the HSI Fellowship's enriched core competencies the right ones, given the emergence of LHSs, the explosion of big data and the importance of patient and citizen engagement?

2. Develop Consensus on the Concept of Embedded Research Impact.

Related to future direction #1 is the importance of developing consensus on the concept of embedded research impact – particularly from the perspective of the health system organizations that are embedding researchers. Recent efforts in Canada have focused on developing a framework for measuring the impact of HSPR on policy- and decision-making (CHSPRA, Impact Analysis Working Group 2018) and on understanding the contributions that HSI fellows have made within their host partner organizations (see Blanchette et al. 2019). In the US, the AcademyHealth team has engaged its host sites to better understand what impact looks like and means from their perspective. Developing an enhanced understanding of embedded research impact and moving toward a framework with meaningful and measurable indicators will support efforts to understand the value of embedded research and can help inform the design of embedded research funding programs and embedded research positions and units within delivery systems (Atkins et al. 2017). It may also help expand the emphasis beyond peer-reviewed publications and bibliometric citation indexes that prevail in many academic institutions, which has been identified as crucial to the long-term success of embedded research programs (Hunter 2019). Efforts to understand the full breadth of research impact, and to measure and communicate such impacts, are under way in other countries too, and it will be important that embedded fellowship programs continue to study and learn from innovative approaches such as the Research Excellence Framework in the UK (https://www.ref.ac.uk/).

3. Align Goals, Policies and Programs.

Prestigious national fellowship awards have helped accelerate a promising shift in training, but sustaining this shift and its impact will require alignment of the policies of health research funders (public and private), the curricula and reward schemes of universities and the commitment of health system organizations to embedded research and data-driven operation. Pioneering changes in the remit of funding agencies have increased investment in implementation science and knowledge translation to enable the benefits of research to be realized in practice (e.g., the Patient-Centred Outcomes Research Institute, the Strategy for Patient-Oriented Research, European Union TO-REACH). Similarly, the addition of new fellowship programs, such as the DSSF, the HSI Fellowship and others, such as the Mitacs Elevate (Mitacs 2019) and Accelerate programs in Canada, to the existing choices available, has increased investment in embedded research. This, in turn, has created new opportunities for organizations to harness research evidence to inform decisions and continuous improvement efforts. However, Canada's experience with innovative grant-funded training programs, such as the CADRE Regional Training Centres and the CIHR Strategic Training Initiative in Health Research (STIHR) training programs (both of which included elements such as field immersion and enriched competency development), suggests that although changes in grant funding can catalyze innovation (see, for example, Conrad 2008; Hamelin and Paradis 2018; Martens 2008; Paradis et al. 2017; Sheps et al. 2008; Straus et al. 2011), sustainability requires that university training programs also be engaged (CIHR 2016).

The ability of doctoral programs to adopt the required new competencies will likely depend on several factors: faculty who have ongoing partnerships with health system organizations for co-designing and collaborating on research (the extent to which funding programs require and value these partnerships will be important too); a fundamental shift in how academics value careers outside the academy; development of a co-ordinated national curriculum that can be modularized and adopted by existing training programs; recognition of the value of embedded research for healthcare organizations; programs that support the career trajectories of these embedded scientists from doctoral awards to early career investigator roles and senior leadership; and the accreditation of health system organizations to both develop and hold research funding to support innovation and transformation in healthcare.

4. Harness the Lessons Learned to Inspire Scale, Spread and Further Innovation.

Both the American and the Canadian fellowship programs have spawned imitation and emulation, and these should be encouraged. In the US, the Education Council of AcademyHealth has overseen a workforce initiative that has underscored the importance of embedded experiences and the growing role of health systems in health services research (Atkins 2018; Javadi et al. 2018; Menachemi et al. 2018; Mor and Wallace 2018; Rich and Collins 2018). Second, several other groups have become interested in sponsoring similar, real-world learning opportunities for post-doctoral fellows. Sanofi is co-sponsoring with AcademyHealth the first Real World Evidence Fellowship in 2018–2019, which is modelled on the successful DSSF, and the National Cancer Institute (NCI) partnered with AcademyHealth in 2017 to launch the NCI/AcademyHealth Healthcare Delivery Research Fellowship. Third, AcademyHealth helped to inform a community-wide interest in LHSs and is poised to serve as a home and convener for cross-program sharing among the new LHS training programs noted above through its establishment of a new interest group on LHSs. Lastly, the development of a comprehensive measure of impact for this work that assesses value from the host site perspective will be important to sustain these activities.

In Canada, the alliance of organizations (CHSPRA) that initially identified training modernization as a top priority is also spearheading efforts to move Canada along the path toward LHSs and sees the HSI fellows and their host partner organizations as pivotal enablers and partners in these efforts. An LHS working group, the co-chairs of which are both health system supervisors to HSI fellows, has been established, and a strategy that will prominently feature embedded research is in development.

Conclusion

The spread of LHSs presents an important opportunity to increase the health system's demand and capacity for doctorally trained talent, to improve the alignment between HSPR doctoral training and the employment trends and needs of the health system and, ultimately, to help ensure that investments in doctoral education and post-doctoral training generate concrete improvements in health and healthcare. Organizations such as CIHR-IHSPR and AcademyHealth that directly and indirectly support the training of a country's HSPR workforce can help catalyze change through leadership, collaboration, funding and well-designed incentives. Robust evaluation of the impacts of these and other embedded researcher models is needed to advance the science and impact of doctoral training and research funding design and to inform our understanding of the extent to which embedded researcher models can facilitate, enable and catalyze evidence-informed health system impact.