Advancing a conceptual model of evidence-based practice implementation in public service sectors.

Implementation science is a quickly growing discipline. Lessons learned from business and medical settings are being applied but it is unclear how well they translate to settings with different historical origins and customs (e.g., public mental health, social service, alcohol/drug sectors). The purpose of this paper is to propose a multi-level, four phase model of the implementation process (i.e., Exploration, Adoption/Preparation, Implementation, Sustainment), derived from extant literature, and apply it to public sector services. We highlight features of the model likely to be particularly important in each phase, while considering the outer and inner contexts (i.e., levels) of public sector service systems.

Introduction

It is increasingly recognized that improving services designed to support the mental health and well-being of children and families involved in public sector services is influenced as much by the process of implementing innovative practices as by the practices selected for implementation (Aarons and Palinkas 2007; Crea et al. 2008; Fixsen et al. 2009; Greenhalgh et al. 2004; Palinkas and Aarons 2009; Palinkas et al. 2008). While concern exists about the lag between development of innovative, empirically tested practices and their ultimate implementation, the policy and practice landscape is often fragmented and changing rapidly (Shonkoff and Phillips 2000). The last decade has seen expansion in a range of promising and proven practices (Center for the Study and Prevention of Violence 2010; Substance Abuse and Mental Health Services Administration [SAMHSA] 2010) and in demands for practice focused organizations to consider, implement, and utilize interventions identified as having the potential to improve children’s and families’ mental health. Expectations that research and service communities will work together effectively to address the challenges of translating scientific potential into public health impact are high (New Freedom Commission on Mental Health 2003; U.S. Department of Health and Human Services [DHHS] 2000).

Unfortunately, the process of implementing evidence-based practices is often complex and fraught with challenges (Backer 2000; Bond et al. 2009; Institute of Medicine [IOM] 2007). Many efforts to implement programs designed to improve the quality and outcomes of human services have not reached their full potential due to a variety of challenges inherent in the implementation process. Implementation of innovative human service technologies is generally considered to be more complex than implementation of other types of technology, due to the fact that human service technologies are delivered through the actions of individuals and organizations, which exist within complex, multi-layered social contexts (Fixsen et al. 2009; Glisson and Schoenwald 2005).

One important link in the connection between research and practice is the translation of evidence-based practices into broader application and impact. The challenges of effective implementation of evidence-based practices are recognized by the National Institutes of Health (National Institute of Mental Health [NIMH] 2001; National Institutes of Health [NIH] 2006). A growing, but still young, body of research and discussion has developed around the science of Dissemination and Implementation. Drawing upon research in the areas of health care (Wagner et al. 2001), public health (Glasgow et al. 1999), organizational development (Glisson and Schoenwald 2005), business (Frambach and Schillewaert 2002), and mental health (Mendel et al. 2008), a variety of different models have been proposed to summarize factors at multiple levels of the social and organizational context that potentially influence the process of translating research into effective improvements in practice. For example, some models emphasize the importance of developing alignments among stakeholders in the community around approaches to solving a targeted societal issue (e.g., delinquency) as well as improving the interorganizational environments of agencies preparing to implement related evidence-based practices (Glisson and Schoenwald 2005). Others emphasize the significance of ongoing and extensive partnerships with agencies, consumers, and other stakeholders in the community to shape the design and implementation of quality improvement initiatives (Mendel et al. 2008). Still others describe processes for embedding research and quality improvement initiatives deeply within the structure of large-scale service systems (Stetler et al. 2008).

In looking across multiple models, several researchers have developed relatively comprehensive catalogs of the factors that may affect the success of research to practice translation efforts (Damschroder et al. 2009; Feldstein and Glasgow 2008; Greenhalgh et al. 2004). More globally, however, a review of implementation models reveals several core themes that are relevant to the current state of implementation science. First, many models divide the process of implementation into several phases (e.g., pre-implementation, implementation, and maintenance/enhancement). There is general recognition that implementation may not always move linearly through such phases (California Institute for Mental Health [CIMH] 2006; Fixsen et al. 2009; Mendel et al. 2008). Second, there are many common components across implementation models, but different models clearly emphasize specific factors above others as signature characteristics. The focus on community-based Participatory Action Research (PAR) in the model articulated by Mendel et al. (2008) is an excellent example of such an emphasis. Variables at many contextual levels are identified in the model, but a guiding force through which quality improvement efforts are directed is the PAR framework. From one perspective, some of the implementation models can be viewed as constructivist in nature (Guba and Lincoln 1994), seeking to describe how use of specific organizing forces can create a conducive community context for implementation. The implementation model or approach becomes a part of the way the community in which implementation occurs understands the problem. A final emergent theme from across proposed implementation models is the relative lack of evidence yet available to clearly prioritize which variables are likely to play key roles in any given implementation effort or context. Consensus opinion exists around important key features, such as leadership in implementation efforts, the need for ongoing consultation and training, especially in early implementation phases, the need to address the impact of staff turnover, and other core implementation drivers, but the body of concrete research evidence for each core driver in many cases remains to be developed (Fixsen et al. 2009). Consensus also exists that multi-component implementation strategies are needed in order to address the challenges of effective implementation, as many different factors need to be addressed in sequence or in tandem for effective implementation that retains impact in community settings (Ferlie and Shortell 2001; Fixsen et al. 2009; Glisson and Schoenwald 2005; Grimshaw et al. 2001; Grol and Grimshaw 1999).

Like other models before it, this paper offers a framework that articulates variables hypothesized to play important roles in achieving effective implementation of EBPs. However, rather than attempt to replicate existing comprehensive literature reviews on the topic, this paper recognizes that implementation models arise through a lens that is shaped by the service contexts chosen for emphasis and by the contextual levels that serve as primary organizing arenas.

As shown in Fig. 1, this paper advances a conceptual model of factors that, in our opinion, are some of the most likely to have a strong influence on the implementation of EBPs in publicly funded settings serving children and families. While most of our focus is on implementation in child welfare settings, which has received sparse attention in the field of implementation science, we also include examples from specialty mental health service contexts. However, we also propose that many of the factors addressed in these two sectors are common to public sector services in general. In addition to adopting a focus that primarily emphasizes the role of service delivery organizations and the service systems within which they operate, we also divide our discussion into factors likely to have the greatest potential impact on implementation at each of four different implementation phases, including Exploration, Adoption/Preparation, Implementation, and Sustainment. As shown in Fig. 2, different aspects of the outer and inner context may be more prominent or manifest differently during different phases. Few implementation models explicitly recognize that different variables may play crucial roles at different points in the implementation process. In addition, once the sustainment phase is reached, the experiences and lessons learned may impact future implementation efforts (as indicated by the recursive arrows at the top of the Fig. 2). Our overall objective in this paper is to provide examples that illustrate why specific factors are likely to be important at each implementation phase, with sufficient specificity that the model could inform the further development of multi-component implementation strategies.

Fig. 1

Conceptual model of global factors affecting implementation in public service sectors

Fig. 2

Conceptual model of implementation phases and factors affecting implementation in public service sectors

Exploration Phase1

The Exploration Phase involves awareness of either an issue that needs attention or of an improved approach to an organizational challenge (Grol et al. 2007). Because the implementation literature has been developed in areas where the adoption of innovation is driven by profit or science, there is little attention to this first phase in the extant literature (Damanpour 1991). Further, adoption of innovation in human services organizations is fundamentally different than adoption of innovation in business or agriculture due to the nature of the innovations and the variability of the clients (Damanpour 1991). Across public sector human service organizations, those with a strong focus on development of new knowledge and understanding of best practices (i.e., “learning organizations,” (Senge 2006), as well as incentives to adopt best practices are likely well poised to explore appropriate practices and implementation strategies (Chadwick Center 2004; Hemsley-Brown and Sharp 2003; National Association of Public Child Welfare Administrators [NAPCWA] 2005).

The Outer Context

Sociopolitical/Funding

Beginning at the broadest level, the state and federal sociopolitical and funding contexts influence the exploration of innovative interventions or practices (Davies and Nutley 2008; Ganju 2003; Hoagwood 2003). Child Welfare, as a publically-funded sector charged with the protection of children, is especially sensitive to social and political forces. Child and Family Service Reviews mandated as part of the (1997) Adoption and Safe Families Act (PL 105-89) require child welfare agencies to monitor outcome indicators in the three key areas of safety, permanence, and child wellbeing. Although all states have participated in at least one review, no state has achieved excellence in all areas, prompting the development of Performance Improvement Plans (U.S. Department of Health and Human Services [DHHS] 2009), which may provide an impetus for state child welfare agencies to begin the exploration phase to identify EBPs that address areas requiring improvement. Federal initiatives can also take the form of special reports (e.g., From Neurons to Neighborhoods: The Science of Early Childhood Development, (2000) or specific funding for the implementation of EBPs, such as NIMH’s Dissemination and Implementation Research in Health program announcements (i.e., National Institutes of Health [NIH] 2006, 2009a, b).

State legislatures often demand practice change in response to both public concerns over issues in child welfare services, such as abuse while in state-mandated out-of-home care and consent decrees and settlements arising from class action suits. Similarly, legislators may use funding in the form of special allocations to encourage the use of innovations. Conversely, severe budget restrictions discourage the exploration of innovations since exploration demands staff time, which may be limited when work forces are cut due to funding restrictions (Greenhalgh et al. 2004).

States may employ a number of other strategies to encourage exploration of EBPs. In mental health, numerous states have set up offices for EBPs to serve as resources to agencies interested in practice change. In California, the California Institute for Mental Health was established in 1993 as a private, nonprofit public interest organization to promote excellence in mental health services through training, technical assistance, research and policy development (California Institute of Mental Health [CIMH] 2010). For Child Welfare, the California Evidence-Based Clearinghouse for Child-Welfare (CEBC), funded by the California Department of Social Services, identifies and disseminates information on EBPs relevant to child welfare (CEBC 2010).

Private foundations, professional organizations and educational reforms can also shape the context for exploration of EBPs. Two excellent examples are the Annie E. Casey Foundation Family to Family initiative started in 1992 (The Annie E. Casey Foundation [AECF] 1992) and the educational changes proposed by the April, 2008 meeting on Partnerships to Integrate Evidence-Based Mental Health Research into Social Work Education and Research sponsored by the National Institute of Mental Health (National Institute of Mental Health [NIMH] 2010). The Family to Family initiative operates in 17 states and advocates for children remaining with their own families through the development of neighborhood-based foster care and family supports (AECF 2008). The NIMH initiative was a joint effort with the Institute for the Advancement of Social Work Research (2007) and explored necessary changes in education to incorporate EBPs. Similarly, the 2006 University of Texas at Austin Symposium, “Improving the Teaching of Evidence-Based Practice” was also an early effort to explore EBPs as part of reforms in social work teaching and curricula (Jenson 2007).

Client Advocacy

As noted by Hoagwood (2003), clients can be powerful advocates for system change through both their demands on individual providers and through their advocacy efforts with legislators. Although consumer based organizations are not common in child welfare, client advocacy has taken the form of organizations started by concerned individuals who, most often, were not child welfare clients. The most famous of these organizations, the Children’s Defense Fund (CDF), was started to promote policies that protect children from abuse and neglect, ensure access to health care, access to quality education and move them out of poverty (CDF 2008). The CDF has inspired the development of individual state advocacy organizations such as Connecticut Voices for Children, an organization heavily involved in encouraging the State of Connecticut to adopt better practices in child welfare (Connecticut Voices for Children 2010). Other smaller local organizations such as the Family and Youth Roundtable in San Diego County, California provide education and training for families with children involved in the mental health, child welfare, and alcohol/drug sectors.

Interorganizational Networks

Much of what has been written on the implementation of EBPs focuses on the level of organizations or agencies because the bulk of health and welfare services are delivered by employees organized in groups or teams (Ferlie and Shortell 2001). A key extra-organizational feature that may encourage the implementation of EBPs is the network of organizations with which agencies are involved (Frambach and Schillewaert 2002). When agencies or organizations interact with other organizations that employ EBPs, this has the potential to increase their own likelihood of exploring or adopting EBPs. In child welfare, contracting with mental health or juvenile justice agencies that employ EBPs will both familiarize the child welfare agency with EBPs and set the example that practice change is achievable. The Community Development Team (CDT) model, an implementation strategy adopted by the California Institute for Mental Health that organizes agencies and specific EBP implementations, is a good example of the power of interorganizational networks (California Institute for Mental Health [CIMH] 2006; Chamberlain et al. 2008).

The Inner Context

Organizational Characteristics

Considerable information is available on the organizational characteristics that promote the exploration and eventual adoption of innovation (Greenhalgh et al. 2004; Rogers 2003) and EBPs in mental health and social service organizations (Glisson et al. 2010). Three broad areas appear to be especially important in the early stage of implementation: an organization’s absorptive capacity, readiness for change and receptive context. Absorptive capacity refers to an organization’s preexisting knowledge/skills, ability to use new knowledge, specialization and mechanisms to support knowledge sharing. Organizations that start with good knowledge/skills, can incorporate new knowledge, are highly specialized, and have mechanisms in place to spread knowledge throughout the organization, are much more likely to explore EBPs and eventually initiate them (Damanpour 1991; Ferlie and Shortell 2001; Greenhalgh et al. 2004; Grol et al. 2007). Public sector agencies including child welfare agencies may suffer from a number of deficits in this area. They often have a workforce with varied levels of education and considerable workloads, have multiple responsibilities ranging from investigations to direct delivery of services, and have few readily available venues for knowledge sharing (Yoo et al. 2007).

Organizational context has received the most attention in public sector services, due largely to the work of Glisson and colleagues in children’s services (Glisson and Hemmelgarn 1998; Glisson and James 2002; Glisson and Schoenwald 2005) and in substance abuse treatment settings as described by Simpson and Flynn (2007). As noted by Glisson and James (2002) both culture (the normative beliefs and shared expectations of the organization) and organizational climate (shared perceptions of the psychological impact of the work environment on the provider) can impact the quality of service delivery and the adoption of EBPs. Thus, individuals who do not view the climate of their organization as welcoming innovation and organizations whose cultures do not promote exploration of practices in response to challenges are highly unlikely to explore the use of EBPs (Damanpour 1991; Klein and Sorra 1996; Simpson 2002). In fact, to assist public sector agencies in changing the climate for innovation, Glisson and colleagues have developed an organizational and community intervention model (Glisson et al. 2006; Glisson and Schoenwald 2005). Additionally, leadership, clear goal setting and prior success in undertaking practice change all have been linked to the likelihood that an agency will explore the use of EBPs (Aarons 2006; Damanpour 1991; Greenhalgh et al. 2004).

Individual Adopter Characteristics

At the more micro level, characteristics of individual adopters are important determinants of which agencies will and will not explore or initiate the use of EBPs. Individual adopters can be at the system, organization, or provider level. In reviewing the literature, three features of individual adopters appear to be important: (1) values and goals, (2) social networks and (3) the perceived need for change. First, values and goals have received considerable attention in the research literature. Those who value innovation are often labeled as innovators and early adopters (Rogers 2003). These individuals are seen as trend setters—comfortable with novelty but also being highly professional. They engage in ongoing education, usually follow the professional literature in their fields and are well-networked (Berwick 2003; Grol 2001). Unfortunately, child welfare, unlike medicine, is only recently developing a strong research base for innovators to follow (National Association of Public Child Welfare Administrators [NAPCWA] 2005). Second, the discussion of the use of EBPs in child welfare is reasonably recent (Barth et al. 2005; Chadwick Center 2004; NAPCWA 2005) and traditional social work education has not focused on dissemination of knowledge about EBPs (Weissman et al. 2006), although there is a growing recognition of the importance to do so (Jenson 2007). As the knowledge base develops, state and county child welfare directors will increasingly have the opportunity interact and learn about EBPs from one another. However, there are some structural issues that do not support the spread of knowledge about EBPs through provider social or professional networks. For example, although psychologists and physicians are licensed in all states, licensure for social workers is less consistent especially for those workers often employed by child welfare agencies. According to data from the Association of Social Work Boards, nine states and Puerto Rico only license MSW or DSW level social workers leaving the bulk of the child welfare workforce unlicensed and imposing no continuing educational requirements on that workforce (Data Path Design 2010). This lack of continuing education requirement coupled with very minimal standards in some states (range = 0–48 h over 2 years) and lack of EBP training in most social work curricula means that child welfare workers are highly unlikely to be exposed to and communicate with each other about EBPs. Finally, attitudes and perceptions of the need to change are critical if EBPs are to be explored (Aarons 2005). Such perceptions on the part of the individual worker come from the ability to identify a problem and feeling empowered that change can occur.

As shown in Fig. 2, both outer and inner contextual factors potentially influence whether or not EBPs are explored by public sector agencies and organizations. Although a strong research agenda on the impact of organizational culture and climate exists in child welfare, there are few data supporting the importance of the other drivers of EBP exploration suggested by the literature. This is unfortunate since, as suggested by Chaffin and Friedrich (2004), two of the biggest obstacles in implementing EBPs in the public sector may be the ability of agencies and organizations to access easily understood research-based information on the appropriateness/effectiveness of EBPs and the level of involvement/comfort with the implementation process. However, even with this dearth of information, policy, funding, and competitive forces are moving public sector systems and organizations to sometimes move hastily into the Adoption/Preparation phase.

Adoption Decision/Preparation Phase

The adoption decision is often conceptualized as a one-time event while, in practice, organizations may experiment with an innovation, sometimes intermittently, prior to broader implementation. As noted above, public service sector organizations generally do not have strong expertise in gathering and weighing research evidence. Thus, the adoption of EBPs cannot easily be separated from the process of adopting new service models, whether or not they have a strong base of research evidence (Schoenwald et al. 2008). Therefore, we consider factors that affect the adoption decision more broadly. A number of approaches to addressing the exploration and adoption decision phases include academic-public partnerships, community-based participatory research, and action research (Leykum et al. 2009; Wallerstein and Duran 2010). In addition, some tools to facilitate the adoption decision are now in place. For example, for child welfare, the CEBC provides a searchable web resource with information on EBPs and implementation approaches. For mental health and alcohol/drug services, the Substance Abuse and Mental Health Services Administration’s National Registry of Effective and Promising Practices (NREPP) (Substance Abuse and Mental Health Services Administration [SAMHSA] 2010) is a resource to help in evaluating the evidence for mental health, substance abuse, and child-welfare practices. In regard to children’s mental health, the PracticeWise.com website (Practice Wise 2010) provides services to support better understanding and accessibility of the research literature regarding EPBs. While some of these tools are available no cost to the user (i.e., CEBC, NREPP), other resources come at a price or subscription fee (i.e., Practice Wise 2010). While forces that have the greatest influence on the adoption decision may vary, the decision is likely to be a confluence of factors at multiple levels.

Outer Context

Some impetus for the adoption decision is captured in major legislative landmarks. The Adoption Assistance and Child Welfare Act of 1980 (1980) and the Family Preservation and Support Services Program amendment of 1993 (1993) emerged at a time when there was significant concern about too many children being placed in foster care or other residential care settings (Behar 1985; Knitzer 1982) rather than about providing sufficient services to families to aid in prevention of out-of-home placement. These laws reflected a heightened awareness of the large and growing numbers of children in foster care (Wulczyn and Goerge 1992) and the lack of services to prevent out-of-home placements. Over the subsequent decade, decisions to adopt Family Preservation models occurred in child welfare systems around the country. Many such programs relied on only a small amount of research evidence and were largely influenced by the sociopolitical and funding contexts, both of which supported exploration of family preservation models and decisions to adopt intensive family preservation service approaches. While some family preservation models that were adopted have a strong evidence base (Aos 2006; Aos et al. 2004), many others that operated under the concept of family preservation had a weak base of evidence (Chaffin and Friedrich 2004; Heneghan et al. 1996; Littell and Schuerman 1995).

Client Advocacy

Client advocacy, and advocacy on behalf of clients, also plays a role in the ultimate decision to adopt innovative service models. At the national level, advocacy organizations, such as the Children’s Defense Fund, were highly influential in shaping the sociopolitical context (conceptualization and legislation) that led to an environment in which specific service models were widely adopted. Foundations, such as American Humane and the Annie E. Casey Foundation, which also assume client advocacy roles through efforts to shape policy, have had a strong influence on the adoption of service models such as Team Decision Making (Crampton 2007), which has become increasingly widespread in its application in child welfare settings (Crampton 2007). Team Decision Making is an example of a practice that fits with the values of the sociopolitical context—having a focus on making services for families in child welfare more family-centered, culturally sensitive, and individualized—as described in the previous section. This service also received strong and coordinated technical support from these national foundations, which supported adoption decisions by child welfare organizations around the country.

Client advocacy at the local level has traditionally been less organized and had less influence. However, many child welfare jurisdictions have had class action lawsuits resulting from advocacy groups taking action to ameliorate concerns with safety, service equity, or service availability. These types of lawsuits often lead to changes in local policy and practice.

Interorganizational forces, which focus on how individual organizations relate to, partner with, and compete with one another also play a potentially powerful role in adoption decisions. A number of these forces have received attention in process models that describe pathways for encouraging adoption, implementation, and effective delivery of EBPs (e.g., (Chamberlain et al. 2008; Glisson and Schoenwald 2005; Institute for Healthcare Improvement [IHI] 2003; Stetler et al. 2008). One aspect of interorganizational structure pertains to how individual organizations, and their leaders, are linked to one another. Information about innovations can be transmitted from one organization to another through interorganizational communication pathways (Rogers 2003). The direct influence of such pathways on an adoption decision may take the form of educating a potential decision maker about the costs and benefits of using an innovation, which may have the effect of lowering (or raising) perceptions of risk regarding costs and benefits of initiating an innovation. However, innovative tools for cost estimation are being developed (Chamberlain et al. 2010) and hold promise for reducing perceived risk because more accurate cost estimates could be obtained prior to the Active Implementation phase. Perceptions of risk, not surprisingly, play an important role in whether organizations choose to adopt innovative practices (Panzano and Roth 2006).

In some cases, interorganizational networks represent informal communication and referral patterns that arise in a dynamic agency environment (Palinkas et al. 2010). In other situations, organizations may join together to formalize interorganizational ties through interest or umbrella groups or through technical assistance organizations designed to serve members of the interorganizational network (e.g., (Chamberlain et al. 2008). Several states have formed such organizations to support provision of technical assistance around assessing and supporting the adoption of new practices, such as the California Institute for Mental Health and the Ohio Center for Innovative Practice. Such formal actions may increase the potential influence of interorganizational ties, especially when the role of the technical assistance organization is to advocate and support introduction of innovative and EBPs.

Similar concepts for formalizing interorganizational ties to foster experimentation with and adoption of best practices have been developed in process models of practice improvement. The Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI) describes a process for utilizing individual medical centers as drivers of change around specific service approaches in other VA centers (Bowman et al. 2008). The CDT model, articulated by the California Institute of Mental Health, describes how a technical assistance entity can use the shared interests of multiple organizations to overcome barriers to adoption, particularly by assisting in reducing perceptions of risk of adoption and by developing an approach whereby multiple organizations can support one another during implementation. The Institute for Healthcare Improvement (IHI) learning collaborative model links organizations with shared goals for quality improvement and implementation of innovations (IHI 2010). Glisson and Schoenwald (2005) describe how the development of interorganizational partnerships sharing common outcome goals can create an environment in which shared commitment to a service approach can increase the likelihood of adoption by creating an environment in which organizations perceive that adoption of a service approach will have positive impacts for clients and the organizations involved.

In addition to partnerships, funding opportunities that encourage competition also have the potential to influence adoption decisions. In circumstances where contracts can be specified to require use of a particular type of service approach, such as occurred with the adoption of the SafeCare® EBP (Lutzker et al. 1982) statewide in Oklahoma and countywide in San Diego County, California, this is a powerful incentive for agencies to position themselves to compete for and receive such contracts.

Inner Context

Many intra-organizational characteristics have the potential to increase the likelihood of adopting service innovations. Several variables can serve to illustrate the role of organizational characteristics. For example, organizational size may increase the likelihood of innovation adoption. Larger organizations may have greater resources to commit to evaluating and exploring the potential utility of different innovative practices. Organizational size was an important contributing factor in a meta-analysis of innovation adoption conducted by Damanpour (1991). Organizational size is likely a proxy for structural variables, such as role specialization and the existence of knowledge and skills within an organization to support adoption of innovations.

Organizational structure can also influence the fit between specific practices and the organization. Fit of a practice with the roles, structure, values, and authority of an organization may contribute to the likelihood that a particular practice is adopted or not. A review of evidence-based parenting programs used within child welfare settings revealed that, despite their relevance to changes in key parenting practices of interest to child welfare, few EBPs were being utilized with families involved with child welfare services (Hurlburt et al. 2007). In part, this may have to do with the fact that many child welfare organizations, despite referring many families to parent training services, do not have extensive influence over the specific services provided because they often rely on service providers in other sectors that also serve clients with other reasons for attending parenting programs. On the other hand, the SafeCare home visiting model, which includes a parent training component, fits better structurally with existing home visitation services over which child welfare organizations often exercise significant control (Edwards and Lutzker 2008). Differences in the fit of some innovations with the existing structure and responsibilities of service organizations may contribute directly to the likelihood of their adoption and implementation, with SafeCare an example of a model with good fit with child welfare organization services.

Leadership

Leadership is a crucial variable in both creating the organizational culture and climate conducive to adoption of service innovations and in taking ownership of the process of advancing a specific innovative practice. Leadership is associated with effective implementation of evidence-based medicine (Edmondson 2004) and with improving the organizational context for EBP implementation (Aarons 2006). Without an internal organizational champion, the probability that a practice will move past the Exploration and Adoption phases and into the Implementation Phase will likely be lower (Forgatch et al. 2005; Price et al. 1998; Rogers 2003).

Like the processes in the Exploration Phase, ultimate decisions to adopt innovations and prepare for active implementation can be influenced by factors at multiple levels. In any particular implementation, the forces that have the greatest influence on an adoption decision may vary. At the point of adoption great attention and resources must be given to preparing for active implementation through planning and development activities.

Active Implementation Phase

Although general principles apply to large and small implementation efforts, the scale of the implementation has implications for specific issues in both outer and inner contexts. Implementation factors range from large system issues, such as political and funding concerns (availability of startup funding, sustained funding in place), through clinician issues (fit with productivity and work demands), to consumer concerns (potential for stigma, applicability of practices for client needs and culture) (Aarons et al. 2009; Hurlburt and Knapp 2003).

Whether an EBP is part of new services, being integrated into existing services, or replacing existing funded services, fiscal resource availability is critical (Aarons et al. 2009). The costs of policy and contracting changes involve resources to support staff in leading or attending internal and external meetings, developing requests for proposals, and facilitating proposal review. Funding is necessary for staff training, computer systems, and other materials (e.g., binders, notebooks, manuals, DVDs, training tapes, etc.) (Simpson 2002).

Funds for targeted services may face competing priorities of legislatures that may favor funding to cover other increasing costs such as Medicaid and prisons (Domino et al. 2004). In one study, public sector agency executive directors and administrators cited uncertainty about future funding allocations as a reason not to invest in EBPs (Buchanan et al. 2005). Creative outer context solutions can be developed. For example, the Mental Health Services Act in California provides for a 1% tax on personal income over $1 million annually to be allocated for mental health care. Previous research in other organizational settings suggests that payers can improve EBP implementation by supporting initial costs such as time spent on training (Frambach and Schillewaert 2002). However, a one-time funding allocation, even if generous, is likely insufficient to support a sustainable EBP implementation.

Contracting

Contracting can be a powerful tool for driving implementation and public sector service systems commonly use contracts with community based organizations (CBOs) to support services that go beyond the purview of the work activities of system employees. In child welfare responsibility for child protective services (e.g., investigations, disposition) generally rests with state or county authorities while family preservation/family support services such as home visitation are generally provided through contracts with CBOs. In child welfare, mental health and alcohol/drug sector contracts are generally awarded through a competitive bidding process and provide the opportunity for specification regarding the activities to be carried out by CBO staff.

Networks can operate at the level of states, counties, organizations, and individuals. Professional organizations for service providers can serve as informal networks through which knowledge and information about EBPs, as well as the specific steps required for full implementation, are shared. The National Association of Public Child Welfare Administrators also supports networking, collaboration, and support for implementing effective programs as well as policies (NAPCWA 2010). The National Association of State Mental Health Directors (NASMHPD) represents and advocates for state mental health agencies and directors, provides information exchange through technical assistance, data analysis, and sponsors conferences and meetings with a focus on research findings and best practice (NASMHPD 2010). The National Association of State Alcohol/Drug Abuse Directors (NASADAD) provides similar services on a national level (NASADAD 2010). Similar organizations provide support for child welfare, mental health, and alcohol/drug programs at the state level and provide structure and process to support information sharing, interaction, and working toward common goals (e.g., supporting legislation to provide more effective services).

Interorganizational networks within a given service system can be critical when moving from policy decisions to policy implementation (Hanf and O’Toole 1992). For example, in child welfare systems, effective interorganizational networks can facilitate appropriate referrals, subcontracting arrangements, training opportunities, and knowledge about EBPs. The interorganizational networks of contractors and their subcontractors may also come into play such as when a particular lead agency may not have the workforce or geographic reach to effectively and efficiently implement and deliver services in particular catchment areas. Thus, interorganizational networks can facilitate information sharing, provision of implementation support, and can be used to drive the implementation process forward.

Information-related challenges to implementation include disconnection, missing and/or misinterpreted information, and varying interpretations based on differing world views (Sobo et al. 2008). Guidelines to improve cross-organizational communications suggested by these authors include: mapping the territory to ensure that all relevant stakeholders are included; charting differences in stakeholder positions; ensuring buy-in at multiple levels; establishing policies and procedures to address organizational facts-of-life (e.g., staff turnover), practicing (active) collaboration to support co-ownership; establishing clear roles and responsibilities; and engaging an objective outsider to assess progress and help problem solve (Sobo et al. 2008).

Intervention Developers

Intervention developers and their organizations and staff play a crucial role in the Active Implementation Phase. There is a great deal of variability in the degree to which intervention developers understand the challenges of implementation across service systems, organizations, and providers. Interventions that have been developed with a focus on one sector may be implemented in a completely different sector. For example, The Incredible Years (IY) (Webster-Stratton et al. 2005) was developed for group settings in small clinics. However, there have been recent efforts to implement IY in child welfare (Hurlburt et al. 2007) and residential substance abuse treatment settings (Aarons et al., in press). In addition, some intervention developers are willing and able to support developing local expertise in their model, while others prefer to retain the training and fidelity support processes (Henggeler et al. 1992; Henggeler et al. 1999; Szapocznik and Williams 2000).

Leadership across levels of the outer context is necessary but not sufficient to facilitate effective implementation. Leadership is critical in the effective and sustainable implementation of innovation (Edmondson et al. 2001; Rogers 2003). Improving leadership, developing readiness for innovation, and developing systems and organizations with a learning orientation will likely facilitate EBP implementation (Aarons and Palinkas 2007; Evans et al. 2003; Iles 2003). Leadership at all pertinent levels (e.g., executive director, middle manager, clinical supervisor) combined with organizational support promotes a positive implementation climate, attitudes, and readiness for implementation (Aarons 2006; Aarons and Palinkas 2007; Klein et al. 2001).

Organizational characteristics refer to structures and processes that exist and/or take place in organizations. It is important to consider how characteristics of organizations influence the process of implementation (Rosenheck 2001).

Structure

More centralized organizations may have an easier time implementing innovations relative to less centralized or more dispersed organizations (Zmud 1982). This can occur, in part, because of potential barriers to implementing change across dispersed organizational units (e.g., clinics in rural areas or teams with different leaders). However, dispersing expertise across organizational units may, in some cases, facilitate EBP uptake and sustainment (Bertram 2008; Hollenbeck et al. 1995). In addition, formalized policies supporting the use of EBP that are part of the core values, mission, and supports of an organization can lead to better adoption and implementation of EBPs (Aarons et al. 2009). Such policies are usually more easily communicated in more centralized organizations.

Priorities and goals

Setting clear priorities and goals is critical to an organization’s sense of mission and purpose. Communicating organizational priorities supportive of EBPs can guide employees toward a common purpose. However, specific actions can be taken to instantiate the mission and vision of organizations. For example, formal communications and policies can be set to support the organizational mission and the importance of particular practices (Schein 2004). Formalization of policies supporting the use of particular interventions for particular mental health problems is associated with greater openness to EBP (Aarons 2004). In addition to communicating organizational priorities, the presence of such policies is congruent with procedural specifications that are often required by EBPs.

Readiness for change

Conceptualizations, definitions, and measures of organizational readiness to change vary widely and range from organizations that are poised to change and innovate (Marsick and Watkins 2003) to organizations with high levels of inertia that may stifle innovation. Organization type is related to openness to innovation and private sector organizations tend to have more organizational supports for use of EBP (Aarons et al. 2009). A number of conceptual models and measures of organizational readiness exist. Studies of organizational readiness to change identify both structural (e.g., availability of computer resources) and process (e.g., cohesion, pressure for change) variables that may be related to adoption of EBPs (Lehman et al. 2002). In addition, readiness is also related to process factors such as organizational culture and climate and individual attitudes toward innovation (Aarons and Sawitzky 2006; Armenakis et al. 2007; Glisson and Schoenwald 2005).

Receptive Context

Receptive contexts are those with organizational cultures, climates, and communications that provide openness to change and minimize competing demands. Such contexts can be found in organizations characterized by support for creative innovation and new ideas, tolerance of differences, personal commitment, and psychological safety (Ash 1997; Siegel and Kaemmerer 1978). In addition, positive social influences within organizations can be facilitated through events, communications, ideas, objects, and behaviors that support positive attitudes, beliefs and behaviors relevant to EBP (Lewis and Seibold 1993). Four social norms in the workplace are positively associated with a receptive or open context: (1) support for creativity and risk taking (2) teamwork, (3) speed of action, and (4) tolerance of mistakes (O’Reilly and Caldwell 1985). Fostering receptive system and organizational norms can be influential in shaping desired attitudes and behaviors in which adherence to structure and procedure (e.g., EBP) provides social approval and rejection avoidance (Cialdini et al. 1999). It follows that agencies with social processes supporting openness and innovation would be more likely to have employees likely to accept variation in work routines tied to EBP.

Culture and Climate

Organizational culture and climate can have a profound influence on organizational functioning and effectiveness in public sector services (Glisson et al. 2006; Glisson and Green 2006). Organizational culture can be defined as the implicit norms and assumptions of a work unit that guide behaviors (Cooke and Rousseau 1988) and can impact how readily new technologies will be considered and adopted in practice (Caccia-Bava et al. 2006; Hemmelgarn et al. 2001). There is concern that public sector service organizations have cultures that are resistant to innovation and may provide lower support for EBP (Aarons et al. 2009). Lower levels of bureaucracy and being a private (vs. public) sector organization are associated with more positive staff attitudes toward adopting EBPs and private provider organizations are more likely to provide organizational support for EBP (Aarons 2004; Aarons et al. 2009; Backer et al. 1995). However, technology transfer may be facilitated by adjusting implementation plans to the culture of a human service agency (Keller and Galanter 1999) and, although a difficult task, some aspects of organizational culture can be changed (Schraeder et al. 2005).

Organizational climate refers to employees’ perceptions and affective responses to their work environment (Joyce and Slocum 1982; Srivastava and Bathla 1996). More general organizational climate includes perceptions of job characteristics and the work group (Glisson 1989). However, “strategic” climates for particular purposes can be developed. For example, climate for innovation is a factor in human service organizational openness to change (Anderson and West 1998; Klein and Sorra 1996) and implementation climate is a measureable factor that can be targeted in organizational change efforts (Klein et al. 2001).

Culture and climate are interrelated but should be considered both in their own right, and in interaction with each other (Aarons and Sawitzky 2006). While culture and climate are important considerations in regard to quality of services, they should also be a key consideration during the Active Implementation Phase (Hemmelgarn et al. 2006).

Innovation-Values Fit

Innovation-values fit posits that implementation of an innovation will be successful to the degree that the innovation matches the mission, values and service provider tasks and duties of the organization and individuals within that organization (Klein and Sorra 1996). For example, case management services for families involved with child welfare systems have common goals of child safety, avoiding removal from the home, minimizing placement changes, and reunification and permanence. There are a number of ways in which these goals can be accomplished. Interventions that are most effective in meeting goals and also fit with other administrative and practice needs such as record-keeping, and productivity requirements are most likely to facilitate effective implementation. When considering which practice to implement, innovation-values fit should be considered at the organizational, managerial, provider and consumer levels. The degree to which an EBP is congruent with system, organization, and provider goals and processes is likely to impact the ease with which implementation proceeds. However, the characteristics of the EBP may allow for more or less flexibility in intervention delivery while adhering to EBP core elements demonstrated to be responsible for treatment outcome effects (Chorpita and Daleiden 2009; Chorpita and Regan 2009).

There can be a great deal of variability in service provider characteristics and readiness to implement EBP and attitudes toward organizational change are important in the dynamics of innovation (Dunham et al. 1989). Personal characteristics of service staff include demographic factors such as age, race/ethnicity, level of education, training, primary discipline, and amount of professional experience, adaptability, personal values and goals, personal dispositional innovativeness, and attitudes toward innovation and EBP. Level of education, level of professional experience, primary discipline, and race/ethnicity are associated with openness toward adopting EBPs (Aarons 2004, 2005; Aarons et al. 2010). Locus of control and self-esteem are, likewise, related to acceptance of innovation (Korunka and Vitouch 1999). Individual provider attitudes toward adoption of EBPs vary with organizational characteristics as well as demographic predictors such as education level, professional status (Aarons 2004), and organizational culture and climate (Aarons and Sawitzky 2006). In addition, definitions of EBP may vary and one study found more favorable attitudes after training in modularized treatment where attitudes were measured relative to perceived evidence rather than manualization (Borntrager et al. 2009).

We have identified a number of factors important in the Active Implementation phase of EBP in public sectors. A broad consideration of system, organizational, and individual factors is warranted and will likely lead to more effective planning for EBP implementation. While there may be readiness to change at a given level in the system or organization, readiness to change may be more variable in other parts of the system (e.g., CBOs, providers, etc.) so ongoing attention to problem solving is a critical driver of the implementation process.

Sustainment Phase

While there is movement in EBP implementation, we have little systemic knowledge about what factors facilitate or limit sustainment of an EBP in a service setting. Federal and state agencies may fund implementation demonstration projects and research studies to facilitate the implementation of EBPs, however, there has been little empirical work examining factors that either facilitate or limit sustainability of EBPs in public sector social services (Greenhalgh et al. 2004). We use the term sustainment to denote the continued use of an innovation in practice. We currently lack comprehensive models of factors that support maintenance or sustainment of EBPs in public service sectors.

Models of Sustainment

Most of the commonly cited models of implementation invoke sustainability as a key component. Greenhalgh et al. (2004) note that sustainability is a key element of models of implementation and diffusion of health innovations. The RE-AIM model (Glasgow et al. 1999; Glasgow et al. 2006) explicitly identifies the “maintenance” phase of implementation as critical in sustaining effective health interventions. A model developed for substance abuse services is the Organizational Readiness for Change (ORC) model that focuses not only system and organizational issues in implementation but also what factors may facilitate the chances of sustained innovation implementation (Simpson and Flynn 2007). The ARC model developed by Charles Glisson invokes the concept of sustainment in the “self-regulation” phase of the implementation process (Glisson and Schoenwald 2005). While sustainment is a desired outcome of effective implementation (Damschroder et al. 2009) there has been little empirical work completed in this area—nor is there a comprehensive guiding conceptual model of sustainability. This phase can also include models for expansion and scaling-up within a service system beyond an initial implementation trial (Brown et al. 2009).

Sociopolitical

Leadership

In order for practices to be sustained across service systems, leadership at multiple levels must be supportive in word and deed. The policies and funding needed for sustained use of practices typically follow from leaders who develop initiatives or set visions that can be maintained over time (Hoagwood et al. 2006). However, strong charismatic leadership may be required to create the conditions for sustained use of EBP once the active implementation phase is complete (Edmondson 2003a).

Policy

Policies to support sustainment of practices can arise at the legislative, system, or organizational levels. At the legislative level, funds can be allocated for particular initiatives. For example, the Mental Health Services Act (Proposition 63) in California (State of California, 2004), provides funds not only for new services but also for sustaining evidence-based services (Cashin et al. 2008). New York State has established an Evidence-based Treatment Dissemination Center (EBTDC) to support training and year-long consultation to front-line clinicians (Bruns et al. 2008) and the State of Ohio has developed “Coordinating Centers of Excellence” to promote use of best-practices and EBPs (Ohio Department of Mental Health [ODMH] 2009). Finally, some federal grant mechanisms now require planning for sustainability after research funds are withdrawn. SAMHSA’s children’s mental health system of care request for proposals has such a requirement.

Service systems may adopt a policy of supporting evidence-based programs as they provide a higher likelihood of positive impacts for individuals served directly by the system. For example, after reviewing promising initial outcome and workforce results of a statewide randomized effectiveness study, the Oklahoma Office of Children’s Services (OCS) implemented and is sustaining an evidence-based home visitation program across the entire state. The OCS revised the bidding and contracting process to support services and additional staff to provide ongoing fidelity monitoring, coaching and support. In another example, the San Diego County Child Welfare system, in collaboration with the United Way, is implementing evidence-based home visitation and made recent changes to the competitive bid process, contracting language, and statements of work in order to support the implementation and sustainability of the practice.

Another important impetus in policy and practice change that can support sustainability in social service systems is the class action lawsuit and related consent decrees. Such lawsuits focus on a variety of system factors relevant to sustainability of EBPs including caseloads, sufficient training, and effective supervision for caseworkers (Korunka and Vitouch 1999). Policies stemming from legal actions can result in mandates for the use of particular service models and practices. This type of lawsuit often results in “consent decrees” that specify mandated changes to be undertaken by the service system under the decree such as increased caseworker training, foster parent training, and use of assessments (Korunka and Vitouch 1999; Robinson et al. 2002) that may be addressed, in part, through the use of EBPs.

Funding

Just as usual care services must be funded, sustaining EBP after initial implementation requires a commitment to ongoing funding. Funding can be dedicated to the support of the particular practice or can be allocated more generally for supporting the services—and associated organizations and staff—in the implementation setting. However, some additional costs are generally associated with EBP that go beyond usual services such as costs associated with monitoring the fidelity with which an intervention is delivered and then providing relevant feedback, coaching, or support to service providers. Additional costs are also associated with training new staff in the EBP and, in many cases, supporting staff until they are certified in the practice. This can be a particularly critical issue in public sector service organizations where turnover rates can range from 20% to 50% annually. A concern for many service systems is supporting EBPs—that are generally more costly to deliver than usual care services—while reaching clients in need. This translates into a challenge for service organizations that have to balance providing high quality care while meeting productivity requirements.

Public-Academic Collaboration

Many implementation efforts involve public academic collaborations. Such collaborations can be critical in the successful translation of research findings into clinical practice (Biegel et al. 2001; Quill and Aday 2000). Successful partnerships can take different forms but some key elements have been identified including empowering community participants, frank discussion of issues and concerns of all stakeholders, building trust, and identification and valuing of needs of all stakeholders (Maurana and Goldenberg 1996). Such partnerships can improve chances of sustainability by providing a mechanism for continued involvement of multiple stakeholders, troubleshooting, problem-solving, procurement of funds for sustainability, and ongoing technical support.

Leadership and Organizational Culture

Sustainment may be supported by a leader’s strategic decisions and plans for how service providers work in organizations. Strong leader support of EBPs can create a climate conducive to continued buy-in and use of specific practices. Sustainability is likely to be more successful when placed in an organizational culture that values EBP and has leadership that facilitates team participation and psychological safety (Edmondson 2003a; b). Schein (2004) identified a number of primary and secondary “culture embedding mechanisms” by which leaders and organizations can set the stage for instantiating organizational values and actions supportive of EBP. Primary embedding mechanisms include what leaders pay attention to, what they measure and control, resource allocation, rewards and status in the organization, and how employees are recruited, selected, and promoted. Secondary embedding mechanisms include organization systems and procedures, design of physical space (e.g. rooms with one-way mirrors for in vivo coaching), and formal statements of organizational philosophy. Use of these types of strategies can promote organizational ideologies that support continued use of EBPs with a high level of excellence.

Critical Mass of EBP Use

An organization in which all targeted employees use a given innovation consistently and well is likely to be effective in its implementation and sustainment relative to organizations in which only some of the targeted employees use the innovation consistently and well (Klein and Sorra 1996). This may be, in part, because during implementation providers may become more “skillful, consistent, and committed in their use of an innovation.” (p. 1057), and sustainability is enhanced when there is a critical mass and social network support for the EBP. However, building expertise across an entire service system may require collaboration and building expertise across and between organizations to instantiate and sustain an EBP (Bertram 2008; Edmondson and Roloff 2009).

Fidelity Monitoring and Support

In the sustainment phase continued fidelity monitoring and support are critical for continued EBP effectiveness (Schoenwald et al. in press). While this is positioned in this paper as an “inner context” issue, it really spans the outer and inner contexts. For example, training and fidelity support that is reliant on external expertise and processes can be considered outer context. However, where training and fidelity support are garnered locally through developing localized EBPs, it becomes an inner context issue. In both cases, having requirements and mechanisms for ongoing fidelity support communicates the need for targeted continuous quality improvement. This is congruent with having a high degree of procedural specificity in work activities. That is, once an EBP is implemented, system and/or organizational cultural norms and processes support providers’ understanding that they now have new set of skills to learn and are expected to work toward perfecting those skills. In addition, this approach supports greater role clarity where service providers know what is expected of them and how their job performance will be evaluated. Such role clarity is associated with better job performance and role efficacy (Bray and Brawley 2002). However, appropriate tools such as fidelity checklists and use of technology such as web-based remote observation, or remote coding of session audiotapes are necessary (Baer et al. 2007). In addition, pairing EBP with ongoing fidelity support in the form of supportive coaching is associated with lower staff turnover in child welfare (Aarons et al. 2009). This is likely related to the reduced staff emotional exhaustion found with some EBPs (Aarons et al. 2009). Supportive coaching and supervision should also take into account the professional development level of child-welfare service providers (Stoltenberg et al. 1998). Approaches such as these along with appropriate technological supports such as web-based or automated phone technology for fidelity monitoring and reporting may help to improve EBP adherence.

Staffing

Staffing for EBP has probably received less empirical attention than other sustainability concerns. Agencies and organizations have standard hiring procedures but these are frequently not geared to the needs of EBPs (Fixsen et al. 2005). Usually largely unstructured interviews are used in the selection process. However, the literature on personnel selection in organizations suggests that such interviews have very low validity in predicting job performance. This can be improved by adopting practices of personnel psychologists. For example, a thorough job analysis should be conducted in order to determine criteria for selecting the best candidates (Patterson et al. 2008). From the job analysis selection criteria can be developed. Selection criteria should include knowledge, skills, abilities, attitudes, and other characteristics important for employees in general, and for effective learning and delivery of EBP in particular. Staff retention and replacement must be considered and planned for in all phases of implementation; however, this concern becomes critical when considering sustainability since a well trained staff is critical to the continued delivery of EBPs.

We have identified a number of factors in the outer and inner contexts that warrant particular consideration during the sustainment phase in public sector services. Consideration of these factors as well as proactive problem solving should lead to more effective maintenance of EBPs within public sector systems.

Discussion and Conclusions

Taken together, the four phases provide a conceptual framework for considering challenges and opportunities in EBP implementation in public service sectors. The nature of the outer and inner contexts, in practice, varies depending on one’s perspective and implementation phase. Similarly, many of the factors discussed above are likely to have relatively more or less importance depending on the implementation phase and the dynamics of the particular service system, organizations, providers, and consumers involved (Aarons et al. 2009; Green and Aarons 2010).

While the model and stages that we have proposed provide heuristic value in understanding and navigating the implementation process, there are few, if any, good measures of the process or progression through the phases. For example, if a county is just beginning to consider whether or not to use or adopt an EBP it would be in the Exploration phase. However, there may be a continuum on which to measure how far the county is into a phase. Some existing measures while developed through work in a particular sector may have relevance to other sectors (Saldana et al. 2007). In addition, the heuristic of clinical readiness to change has also been applied to organizational change in health and public sector services (Cinite et al. 2009; Weiner 2009) although the utility of this approach has not been proven. However, process-based measures of implementation phase would be useful to implementation researchers in order to assess and compare implementation efforts in terms of current status, progression, attaining milestones within each phase, the speed with which each phase is traversed. Such measures could then be examined in relation to implementation effectiveness (Klein et al. 2001) and implementation outcomes (Proctor et al. 2009; Proctor et al. 2010).

Embedding a culture (Schein 2004) of EBP can begin early in the overall implementation process. For example, once the Adoption/Preparation phase is entered, plans and processes should be developed to support embedding the particular EBP in the system both ideologically and practically. Positive organizational culture is associated with EBP sustainment (Glisson et al. 2008). However, the value of EBP in general should be part of the ongoing system and/or organizational culture. Embedding strategies could include improving implementation climate that supports the EBP, carving out time for training and coaching that also support fidelity, and tailoring information systems to support EBP record keeping and reporting.

Consistent with the current NIMH Strategic Plan (National Institutes of Mental Health 2007), dissemination and implementation of effective health, mental health, and social service strategies should be a priority for service systems and organizations providing mental health and social services. While there are a number of effective EBPs tailored to the needs of various service systems and their clients and patients, the movement of such interventions has been slow and relatively ineffective (Balas and Boren 2000). Having a better understanding of the challenges likely to be present during implementation phases should help multiple stakeholders more effectively navigate the complex process of EBP implementation.