Beacon communities' public health initiatives: a case study analysis.

INTRODUCTION: The Beacon Communities for Public Health (BCPH) project was launched in 2011 to gain a better understanding of the range of activities currently being conducted in population- and public health by the Beacon Communities. The project highlighted the successes and challenges of these efforts with the aim of sharing this information broadly among the public health community.
BACKGROUND: The Beacon Community Program, designed to showcase technology-enabled, community-based initiatives to improve outcomes, focused on: building and strengthening health information technology (IT) infrastructure and exchange capabilities; translating investments in health IT to measureable improvements in cost, quality, and population health; and, developing innovative approaches to performance measurement, technology, and care delivery.
METHODS: Four multimethod case studies were conducted based on a modified sociotechnical framework to learn more about public health initiative implementation and use in the Beacon Communities. Our methodological approach included using document review and semistructured key informant interviews. NACCHO Model Practice Program criteria were used to select the public health initiatives included in the case studies.
FINDINGS: Despite differences among the case studies, common barriers and facilitators were found to be present in all areas of the sociotechnical framework application including structure, people, technology, tasks, overarching considerations, and sustainability. Overall, there were many more facilitators (range = 7-14) present for each Beacon compared to barriers (range = 4-6). DISCUSSION: Four influential promising practices were identified through the work: forging strong and sustainable partnerships; ensuring a good task-technology fit and a flexible and iterative design; fostering technology acceptance; and, providing education and demonstrating value.
CONCLUSIONS: A common weakness was the lack of a framework or model for the Beacon Communities evaluation work. Sharing a framework or approach to evaluation at the beginning of implementation made the work more effective. Supporting evaluation to inform future implementations is important.

Introduction

The Office of the National Coordinator for Health Information Technology (ONC) Beacon Community program provided $250 million over three years to 17 selected United States communities. ONC’s support has enabled these communities to build and strengthen their health information technology (health IT) infrastructure and exchange capabilities to improve care coordination, increase the quality of care, and slow the growth of health care spending. Within these 17 communities, a variety of projects were undertaken, most focused on clinical health IT. Nine projects, however, aimed to improve public- and population health.

Funded by the Centers for Disease Control and Prevention (CDC), the Beacon Communities for Public Health (BCPH) project was launched in 2011 to gain a better understanding of the range of activities conducted in population- and public health by the Beacon Communities. Researchers on this project evaluated these public health efforts using the sociotechnical framework with the aim of sharing this information broadly among the public health community. This particular study focuses on bringing to light facilitators and barriers identified through the process of evaluating the framework across the case studies with the goal of informing future initiatives in these areas.

We conducted four case studies to learn more about the work of the Beacon Communities pertaining to public health. These case studies spanned six sites because three of the Beacon Communities conducted the same public health initiative. Case studies were selected in consultation with ONC based on the impact to public health and the stage of the identified public health initiative. The case studies chosen were txt4health, a two-way texting application aimed at improving outcomes for prediabetic and diabetic patients in the Cincinnati, Detroit, and New Orleans Beacon Communities; an avatar used for intake of Women, Infant and Children (WIC) participants in the Southern Piedmont Beacon Community (NC); a portal to share asthma action plans in the Southeast Minnesota Beacon Community; and public health use of a robust health information exchange (HIE) in the Western New York Beacon Community.

The goals of the Beacon Communities support various public health activities. However, public- and population health were not specifically outlined in the Beacon Community goals, and the relevance for public health varied across Beacon Communities. The purpose of the BCPH project was to more fully understand these public health activities. This effort included developing an inventory of public health activities across the communities and conducting case studies of several initiatives. The purpose of the case studies was to more fully understand how the Beacon Communities affected public health and to glean lessons learned and promising practices from each so that other communities can learn from their work.

Background

BCPH was designed to showcase technology-enabled, community-based initiatives to improve health outcomes.1 In 2010, ONC awarded 17 grants to multi-stakeholder groups for the Beacon Community cooperative agreement program. Of the nine programs focused on public health initiatives, the four initiatives outlined below were selected for this case study evaluation.

The Importance of Information Exchange for Public Health

Public health activities rely heavily on information sharing and analysis.2 Thus, the goal of the Beacon Communities to improve access to information can support public health services such as surveillance and monitoring immunization coverage.3

As greater numbers of providers implement electronic health records (EHRs), more information is available in the aggregate for population health purposes. Meaningful use measures specific to public health including immunization registry reporting, electronic laboratory reporting, syndromic surveillance, and cancer registry reporting have led to greater inclusion of information relevant for public health in EHR implementations.4 Public health departments have been able to take advantage of increased information availability of this type in a variety of ways, including improvements in efficiency, timeliness, and completeness of reporting.5,6

Public health specific, information-exchange activities may be broad, as outlined in Shapiro (2006), or more specific, such as the BioSense Program7 or the Public Health Information Network.7 In addition to efforts that specifically focus on health IT, various community-based funding initiatives include information exchange, such as the Health Care Innovations Awards Program funded by the Centers for Medicare & Medicaid Services.8 New York State developed a program to help support regional HIE organizations.9 This program has helped provide infrastructure to overcome barriers to HIE, which include structures, formats, and vocabularies.

Efforts to improve infrastructure and data exchange have important implications for public health. Access to information about outbreaks can help public health officials identify and treat those with highly communicable diseases quickly, preventing the further spread of disease. Information exchange can also be used to support management of chronic diseases across multiple providers, such as diabetes and asthma. Better management of these conditions can improve population health as a whole. Public health officials can conduct their activities more efficiently and effectively through automated access to multiple sources of information.

Each of the selected initiatives for review represented an important contribution to public health through IT. Brief summaries of each case study and their public health improvement areas follow.

txt4health

txt4health is a mobile health information and patient engagement service that uses text messaging to help patients to better manage their health and to control or prevent diabetes for those at high risk. This two-way text messaging system involves text messages sent to participants with messages promoting behavior change and disease management. The Detroit, New Orleans, and Cincinnati Beacon Communities used a vendor to help implement the initiative. Each community was responsible for identifying its target audience and marketing to them. The initiative was designed for a four-month period. The communities’ evaluations included the effectiveness of message tailoring, of marketing and recruitment efforts, and of lessons learned about participant enrollment and engagement.

Women, Infants, and Children (WIC) Avatar

The Southern Piedmont Beacon Community used an avatar, an online interactive educator, to provide client education and administrative intake for the Women, Infants, and Children (WIC) Program. This initiative was designed to alleviate call volumes for WIC technical assistance and to provide asynchronous patient education opportunities. The community’s evaluation examined how successfully end users interacted with the avatar, how successfully they retained information, and whether they still needed human interaction to get their questions resolved. Evaluation results examined whether the WIC avatar was acceptable to clients and easy for them to understand.

Asthma Action Plan Portal

By means of a regional HIE organization, Southeastern Minnesota was exchanging asthma action plans and storing them on a county IT server. The aim was to ensure that school nurses have accurate diagnosis, management, and dosing information about all students with asthma in each school, and that they could access that data anywhere the student might be, such as traveling for sports to other schools. Evaluation questions included whether the number of asthma action plans increased; how the technology fit within the current workflow for school nurses; concerns regarding consent, data sharing, and student privacy among schools; and unintended consequences.

Health Information Exchange (HIE)

The Western New York community expanded an existing HIE infrastructure to support public health activities. The additional funding allowed HEALTHeLINK, the regional HIE, to develop more interfaces with health care providers and other data sources. Including more data sources meant that the HIE had more robust data. Because public health staff had access to a rich source of regional data, they could conduct quality improvement, surveillance, reporting, and investigative work more efficiently and effectively.

Methods

We conducted a series of multimethod case studies to learn more about initiative implementation and use in the selected Beacon Communities. Document review and semistructured key informant interviews were the primary methods used for all sites, and an economic analysis was conducted for one Beacon Community (the Western New York community). In this section we describe how research questions were applied to the case studies.

Sociotechnical Framework

The Beacon Community program emphasized multi-stakeholder efforts across communities. When implemented, these technical initiatives have broad impacts on organizational, social, and professional systems. Thus, an approach that includes the intersection of technology and the social context in which it is implemented was indicated.10,11 Some recent studies of the unintended consequences of health IT have found that the fit between IT and the clinical work system leads intended end users to accept or reject IT, to use it or misuse it, and to incorporate it into their routine or work around it.12 Cast in the context of the sociotechnical approach, Figure 1 depicts the relationships between the BCPH project evaluation and the systems under study.

Figure 1.

Relationships Between Promising Practice Areas and the Sociotechnical Framework

Social informatics research sheds light on these issues by using mixed methods to understand technological initiatives and their contexts, including social, environmental, and technical systems. The social system encompasses the people and the work they do. The environmental system includes organizational structures. The technological system includes the technology itself and the tasks it supports. Because the sociotechnical approach involves optimizing the initiative, sustainability—especially in the form of promising practices—is another important factor to consider.

Based on these factors, research questions were developed to guide information gathering and were tailored for each case study. Table 1 presents the research questions, organized by the main areas of the framework: the social and environmental system, and the technical system, with relevant subheadings.

Table 1.

Research Questions Presented with the Sociotechnical Framework

Social and Environmental System

Technical System

Structure What technologies were in place prior to deployment? To what extent do partnerships and collaboration with traditional and nontraditional partners impact initiative outcomes? How does the size and breadth of a partnership working on a Beacon Community initiative impact implementation outcomes? Does previous collaboration impact the ability of Beacon Communities to structure project teams with sufficient expertise and resources? What is the relationship between congruence of partnership and individual partner goals and the success of the initiative? Does the level of distribution of costs and implementation responsibilities across partners correlate to initiative success? How were responsibilities of the program allocated and managed prior to and after Information and Communications Technology (ICT) introduction? People Is previous experience of any partners with similar initiatives a predictor of initiative success? How does partner motivation for participation impact the success of the partnership and initiative? Is there a relationship between the method of target population selection and success of the initiative? How do perceived usefulness, perceived ease of use, social influence and norms, perceived behavioral control, and facilitating conditions influence adoption of the specific health IT?

Technology How did the process of choosing the initiative technology impact outcomes? Was there a good match among technology, staff capabilities, and tasks that needed to be completed? How was technology implementation helped or hampered by people and structure? Tasks How did the structure of the partnership and individuals involved in the partnership shape tasks included in the initiative? Overarching Considerations How did the interaction between the ICT and their users facilitate optimization of each system? How did the interaction between systems hinder optimization of each system? What factors need to exist in both systems to enable scalability and sustainability? Sustainability What challenges did the Beacon Community face before, during, and after the initiative? What best practices and lessons learned can be gleaned from the implementation? Does the Beacon Community plan to continue the initiative after the funding period? What factors influence sustainability of the initiative? Are there any improvements in cost, efficiency, or appropriate provision of services as a result of the initiative?

Case Study Selection

We reviewed publicly available Beacon Community information to identify public- and population health activities. Because ONC administered the Beacon Community Cooperative Agreement Program, ONC project officers were most informed about these activities and their progress. They initially identified seven communities that were engaged in public- or population health related activities. Two additional Beacon Communities self-identified to their project officers as having relevant public- or population health activities under way. The evaluation team conducted telephone interviews with Beacon Community staff in each of the nine identified communities. Prior to each Beacon Community interview, we held a preinterview call with the ONC Beacon Community project officer to identify potential community activities to discuss during the interview. With input from ONC and CDC, we developed a protocol to guide the interviews. During the telephone interviews, we asked communities to identify and describe their public health related, Beacon Community funded activities. The focus was on those activities that have an impact on state or local health department IT infrastructure or population health outcomes.

The selection criteria to prioritize and select initiatives for the research case studies were adapted from the National Association of County and City Health Officials Model Practice Program,13 a nationally recognized and vetted program that identifies and promotes excellence in public health. The following criteria were used to identify case study candidates:

Meaningful collaboration should be occurring between a public health agency and community partners through the Beacon Community for technology or program implementation.

The use of technology should be new to the public- and population health field or an inventive use of an existing technology.

The development or adoption of the technology should have been a result of a particular public- and population health program or concern.

Evaluation activities must be underway or planned before the case study is completed.

The use of technology to meet a public- and population health need must be appropriate for replication in other health departments and communities.

Based on these criteria and discussions with ONC about the progress achieved by the individual Beacon Communities, the Western New York community, Detroit, Cincinnati, New Orleans, Southeast Minnesota, and Southern Piedmont were chosen for in-depth review as Beacon Community case study sites. These sites met all of the selection criteria and had made sufficient progress with their initiatives to be eligible for the study. ONC’s involvement in the selection process was primarily to provide guidance with regard to a site’s degree of progress with the selected initiative.

Data Collection

Data collection consisted of document review, review of epidemiology reports (the Western New York community only), phone interviews, a site visit, and follow-up telephone discussions. Once a community was selected, the community activities and implementation status were discussed with representatives from ONC. We also held a planning call with representatives from each selected Beacon Community to discuss the purpose and goals of the case study. This call facilitated selection of key informants for interviews.

Document review included reports from the communities and ONC. Some communities shared additional documents at the outset. The team also reviewed descriptions of and background information about communities on their Web sites and any social media sites, such as Facebook.

The document review informed development of an interview guide, which was used in semistructured, in-depth, in-person interviews with key personnel including representatives of the Beacon Community, public health department, partners, evaluators, and vendors—at a minimum—in each community. Interviews with each stakeholder and group ranged from 30 to 120 minutes. Table 2 provides the number of people interviewed in each Beacon Community.

Table 2.

Participants Interviewed at Beacon Communities

Respondent Type	Detroit txt4health	New Orleans txt4health	Cincinnati txt4health	Across txt4health sites	Southeast Minnesota Asthma Portal	Western New York	Southern Piedmont WIC Avatar
Beacon Community	3	3	1		2	2	2
Public Health			1		3	6	5
Clinicians			2		6	4	1
Evaluators		1		1
Partners	1	4			6		1
Vendor/contractor	3	1		1		3	1
Total	7	9	4	2	17	15	10

Those interviewed represented a number of occupations and functions, including public health commissioners, disease intervention specialists, nurses, epidemiologists, WIC staff, vendors, evaluators, and IT specialists. After the site visits, we conducted additional follow-up interviews as needed to seek clarification or additional information.

Data Management and Analysis

The evaluation team uploaded the interview notes and documentary data into a qualitative analysis software program (NVivo 9.3) to organize the data. This program allows data to be categorized and sorted by relevant concepts or codes. Prior to interviewing, the team developed a coding dictionary with definitions and examples of each code. Two team members independently and concurrently coded a sample of the interviews to assess interrater reliability at the community level. Challenges and contributors to success were extrapolated from coding reports and analyzed using a deductive approach for common concerns, barriers, and lessons learned.

Identification of Facilitators and Barriers

We reviewed the results of the individual case study evaluations for themes that cut across the sites. In particular, we identified facilitators and barriers to the implementations reviewed for each case study. For each of the sociotechnical framework domains, a measure of the degree to which it was facilitated or represented a barrier to the intervention was developed from the interview responses.

Results and Findings

Cross-Case Studies Analysis Approach

While each case study was analyzed individually using this framework, this work reports on themes across the interventions that were gleaned from review of the four case studies. Output from individual analyses was used to develop these themes. For txt4health (a case study with three sites) we also applied this approach to the individual sites to provide a detailed understanding of how those sites differed from each other in these critical areas.

The following sections summarize the findings based on the responses from the interviews and documents each site provided. The majority of references to respondents’ statements are summarized, and illustrative quotations are provided to identify a typical response or highlight a particular insight. To maintain anonymity among respondents, no information or quotations are provided that could be directly linked to any respondents.

Facilitators

We looked across the four case studies to identify factors that were clear facilitators for public health projects in the Beacon Communities: the role of public health, partnerships, and core funding sources. In addition, the role of the public health initiative in addressing staffing issues, workflow issues, and technology capabilities was addressed. Lastly, for two of the cases, the extent to which policy considerations were integral to the public health initiative was considered. These dimensions were analyzed for each Beacon Community. The results of this analysis are summarized in Table 3 (attributes are not necessarily mutually exclusive). Bullets within each broad facilitator dimension emerged as themes from the analysis of the NVivo-coded interviews, and were determined to be the best way to make comparisons of dimensions across broad initiatives. Further detail about each of the elements is discussed below.

Table 3.

Implementation Facilitators Within Beacon Community

Facilitators	Detroit txt4health	New Orleans txt4health	Cincinnati txt4health	Southern Piedmont WIC Avatar	Southeast Minnesota Asthma Portal	Western New York
STRUCTURE: Public health
• Oversight	X	X	X	X	X	X
• Direct involvement		X	X	X	X
• Champion			X	X	X	X
STRUCTURE: Partnerships
• Strong and sustainable		X	X	X	X	X
• Technology	X	X	X	X	X	X
• In place prior to Beacon funding	X	X	X	X	X	X
PEOPLE: Staffing levels
• Part of project (to assist with staffing issues)				X
• Maintained levels		X		X	X	X
• Increased levels	X		X
TECHNOLOGY
• Iterative				X		X
• Flexible	X	X	X	X	X	X
• Participatory		X		X	X
TASKS: Workflow
• Part of project (to assist with workflow issues)				X	X	X
• Delivered with other interventions	X	X	X		X
OVERARCHING CONSIDERATIONS: Policy
• Integral to implementation					X	X
SUSTAINABILITY: Funding
• In place prior to Beacon funding						X
• Beacon funding	X	X	X	X	X	X
• In addition to Beacon funding					X	X

Structure

Public health involvement in the initiative was a key consideration for the BCPH project. Our analysis investigated whether public health authorities provided direct oversight, were directly involved in the project, or had a high-ranking person with authority to champion public health. Two cases had all of those levels of public health involvement. For the txt4health initiative, each site varied in public health involvement: Detroit had only public health oversight and New Orleans lacked a public health champion. In the Western New York community, public health involvement took place through participation on the Board of Directors of the HIE. The New York State Associate Commissioner of Health provided oversight and served as a champion for public health.

We observed a variety of partners and partnership structures and focused on three elements: whether the partnerships appeared to be strong and sustainable, whether partnerships were formed around the technology involved in the initiative, and whether the partnerships were in place prior to the receipt of Beacon funding. These partnership factors applied to all cases and sites, indicating that strong partnerships were in place for every initiative except for txt4health’s Detroit site (see discussion in Barriers section for more detail), which faced sustainability challenges.

The txt4health communities relied heavily on partnerships for recruitment with varying levels of success. Involvement of partners in txt4health recruitment efforts was viewed favorably. According to a spokesperson for one of the sites, “Of course, among the first people or groups we contacted were social organizations because, with our focus on public health, we had these established networks of social organizations that we had worked with on tobacco and obesity and other campaigns.” When recruitment numbers were low initially, the txt4health sites expanded partnerships to incorporate grassroots marketing efforts and improved recruitment rates.

People

Public health staffing levels either were maintained or temporarily increased across case studies. None of the sites added permanent staff. The txt4health sites used students, community workers, and other partners to provide temporary staff to assist with recruitment. The Western New York community, Southern Piedmont, and Southeastern Minnesota Beacon Communities demonstrated how use of technology supported staff in their work so they could spend less time on administrative tasks and more on providing core services.

Technology

Three case studies implemented a new technology, and one used an existing HIE to support public health. The three txt4health sites used an existing product provided by an external vendor, while the WIC avatar and the asthma portal were developed in conjunction with stakeholders. In the Western New York community, public health leveraged the information in HEALTHeLINK to support their operations.

We reviewed whether the technology was developed in an iterative manner, the degree to which it was flexible and scalable to the community’s needs, and whether primary stakeholders directly participated in the design of the technology. This approach, involving stakeholders’ participation, guided WIC avatar development. As one participant noted, “The WIC directors had a chance to meet with some Beacon staff initially and the health directors to really talk about where this technology would be most beneficial within the agencies.”

Tasks

Workflow considerations were also identified as a key facilitator. For some sites, addressing a workflow concern was also an integral component of the initiative; for others, workflow was not addressed. All but txt4health delivered initiatives that specifically targeted workflow issues.

The WIC avatar was a key element of the WIC intake interview workflow. Timing the implementation of the avatar correctly so that it worked smoothly required planning and flexibility. For the asthma portal, the workflow depended on the consent process. For the Western New York community, a huge improvement from the former workflow was the ability for public health authorities to access data that would have previously been obtained with more difficulty via phone calls or travel. Workflow concerns are also discussed in the section on Barriers.

Overarching Considerations

We investigated the role of policy considerations for each case study. For two cases, the asthma portal and the Western New York community, privacy and consent policy considerations were critical. In both cases, the multi-stakeholder involvement was necessary to come to resolution.

Sustainability

Prior funding and strategies for future funding were identified as facilitators. We reviewed whether any funding for the initiatives was in place prior to the receipt of Beacon funding, whether Beacon funding was the sole source of funding, and whether the site had received or was seeking funding beyond the Beacon funding. Only the asthma portal and Western New York case studies received funding support either prior to or in addition to the Beacon funding. With the exception of the Detroit site for txt4health, all case study sites were seeking additional funding to sustain the initiatives.

Barriers

Our analysis investigated barriers to implementation that were organizational, technical, administrative, operational, and policy related. We examined issues that may have impeded implementation. The themes that emerged are described in more detail in the following sections and are summarized in Table 4.

Table 4.

Implementation Barriers Within Beacon Communities

Barriers	Detroit txt4health	New Orleans txt4health	Cincinnati txt4health	Southern Piedmont WIC Avatar	Southeast Minnesota Asthma Portal	Western New York
STRUCTURE
• Partnership considerations	X
• Stakeholder considerations					X	X
• Cultural concerns				X
PEOPLE
• Lack of expertise/capacity	X	X	X
TECHNOLOGY
• Technical issues	X	X	X			X
TASKS
• Workflow concerns	X			X	X	X
• Issues in implementation	X	X	X	X	X	X
• Problems with recruitment	X	X	X		X
• Lack of flexibility		X
OVERARCHING CONSIDERATIONS
• Required changes					X	X
SUSTAINABILITY
• Evaluation funding sufficient				X	X	X

One organizational barrier was the maturity of the partnerships in place to support a given initiative. For txt4health in Cincinnati and New Orleans, partnerships were not a barrier; however, in Detroit, the initiative relied on providers for patient recruitment, which appears to have made recruitment more difficult. Detroit staff were unable to secure sufficient time with some physician practices to educate them on recruitment.

Stakeholder considerations may have presented additional barriers to implementation. For example, some asthma portal participants were unwilling to share information and provide consent out of concerns for privacy. For Western New York, access for public health—while considered at inception—was not addressed until the system was robust enough to provide meaningful data. For the WIC avatar, the primary organizational barrier was the need to adequately and effectively address a fairly large Spanish-speaking population of WIC participants.

For the txt4health sites, both staff and participants had a general lack of expertise with the application. In addition, successful implementation depended on mobile devices, which added to the educational burden of implementation. These factors made implementation difficult in some cases and limited enrollment as well.

Of the four case studies, txt4health experienced the most technical issues, which was a key structural barrier. The application was not faulty; rather, a lack of understanding about how to incorporate the new technology into existing workflows hindered implementation.

Other factors included underestimating the facility of the target population with mobile devices, misunderstanding the limitations of texting, and incorrectly assuming that the target population used mobile devices. For example, the HIE in Western New York was initially designed for payers and providers and did not include a defined role profile for public health workers. Thus an ad hoc workaround was needed to give public health employees access to the system until the issue was resolved.

From an operations perspective, all four case studies experienced both workflow and implementation issues. txt4health and the asthma portal also had to deal with recruitment issues. Extensive partnerships facilitated recruitment in Cincinnati, but only a limited number of people completed the program, and the initiative was not sustained.

Task adjustments were made in the second and third phases of the WIC avatar implementation to help address workflow issues. The location of the avatar equipment and the point in the workflow in which the avatar was introduced to clients were improved.

Other external factors affected these initiatives as well. For the asthma portal, one staff member commented, “The hard part was the health system went through an EMR change during their first part of Beacon.” This additional contextual factor might have been hard to anticipate. All three txt4health sites reported some degree of difficulty with recruitment, attributable in some part to the novelty of the technology.

The asthma portal experienced issues in obtaining consent from some parents who were concerned about privacy. Since consent is required annually, providing continuity for participants was difficult, especially when consent rates were low. For the Western New York community, a state law limiting access to health information for youth ages 10–18 hindered coordination of care among providers but did not limit access for public health.

All of the Beacon Communities included evaluation plans as part of their funding proposals. For Southern Piedmont, Southeastern Minnesota, and the Western New York community, Beacon funding was sufficient to conduct a planned evaluation. In the case of txt4health, Detroit and Cincinnati cited limited funding as a reason for planning for evaluation after implementation, and New Orleans sought funding for evaluation from a partner. Efforts made to plan for and fund evaluation early on seeded sustainability plans.

Discussion

For each area of the sociotechnical model examined, both facilitators and barriers were present within nearly all Beacon Communities, demonstrating the complexity and multifaceted nature of health IT initiatives. Individual topic areas within the model and the associated lines of inquiry provided for rich discussion among the stakeholders. Three cross-cutting themes emerged from the individual case studies as being relevant to all Beacon Communities studied.

The Tension Between Technology and Funding in Public Health

As public health departments across the nation continue to downsize, they increasingly look to technology to fill gaps left by staff. With the exception of short-term staff hired for marketing and advertising activities in the txt4health case studies, none of the case studies resulted in increased staffing. One goal of the WIC avatar project was to free up staff to provide services to more WIC clients and to deliver the WIC information more uniformly. These thoughtful approaches to the use of technology may help solve staffing issues, but they may not resolve the larger funding issues public health programs face.

Changing Routines

Workflow and existing routines had to be adjusted in every case study implementation. These adjustments sometimes interrupted an existing routine or necessitated a replacement routine or the introduction of a new one. To the extent that these barriers can be anticipated, measured, and accommodated, implementation will benefit. Capturing these effects in evaluation is critical to anticipating these barriers. Planning for, measuring, and accommodating the often inevitable changes to routines that result with IT initiatives is imperative. When time and other resources exist, the best practice is to optimize business processes in advance of introducing new technology, and to then design and implement the technology to support reengineered workflow.14,15

Unanticipated Outcomes

Each Beacon Community project resulted in unanticipated outcomes—accrued benefits that were not part of the program goals. For example, ongoing public health surveillance activities that resulted from the Western New York community’s project are a positive outgrowth of the original stakeholders’ vision to reduce cost via information sharing. The stakeholders involved in the Detroit txt4health initiative commented that although activities such as HIE may be hard for patients to understand, connecting commonplace technology such as texting with a chronic disease resonated with patients. This sentiment was echoed at other sites as well. Asthma portal team members in Southeastern Minnesota felt that an unanticipated outcome of health care reform was the recognition of unmet patient needs within the community. Said one team member, “If we didn’t have the Beacon project, I think it would have been really hard for our community because the providers would be doing what they think they need to do and that voice [the patient] would get lost quickly.”

Limitations

This project was limited by the need to balance the very busy activities of each of the Beacon Communities with the goal of better understanding and evaluating the work they were doing. This limitation affected all aspects of the study from the process of selecting the sites and the initiatives to selecting the interviewee pool at each site. As one example, an original goal of this work was to include evaluation data generated by each of the sites and incorporate it in the overall evaluation, but such inclusion was not possible given the different approaches taken to evaluation and the different stages of progress each project was at when our work concluded. Each site represented a unique approach to the delivery of their selected initiative, and making generalizations across sites is not only difficult but may be misleading in some cases. However, many of the themes that emerged in this work were consistent across sites and are expected to be relevant to other organizations with projects undertaking similar work.

Conclusions

Our central goal was to identify the promising practices that supported the development and implementation of health IT for public- and population health purposes within these Beacon Communities. Each case study differed along technological, geographic, and target population dimensions. Using a sociotechnical approach provided a common thread to understanding similarities and differences.

Core Promising Practices

Based on this analysis, the four most influential promising practices to facilitate initiatives that have an impact on public health are the following:

For every case study and site, this practice was identified as the most critical component of a successful initiative. These partnerships aid in recruitment, help ensure that users’ needs are met, and provide other avenues for funding.

To help ensure a successful implementation for an initiative, the technology must fit the task, must be flexible enough to adjust as barriers are encountered, and must be iterative so that changes can be made as the initiative is rolled out.

Involving all stakeholders in all phases of the project is essential to a successful initiative. This effort to build acceptance buoys the implementation of the initiative and helps sustain it throughout the implementation period.

Ensuring that the target population and relevant partners and stakeholders are well educated about the goals of the project is also highly recommended. Demonstrating value helps stakeholders identify with an initiative and helps ensure sustainable support.

Strengthening the IT Infrastructure

Each Beacon Community brought a different set of strengths to their public health initiatives, and focused on different populations, public health concerns, and technologies. All four of the initiatives reviewed in this work contributed to the goal of building and strengthening the health IT infrastructure,1 some more directly benefiting the exchange capabilities of these infrastructures than did others.

Many lessons can be learned from the impacts of their facilitators and barriers described above. Importantly, these core promising practices we identified are not novel or necessarily ground breaking but underscore the importance of established approaches to implementation and evaluation. While informatics is a young field within public health, much is already known about facilitators and barriers to successful implementations. The challenge often faced is getting project sponsors and stakeholders to agree to use these best practices.5

Translating Investments into Improvements

When translating investments in IT activities (such as funding provided to the Beacon Communities) into improvements in cost, quality, and public health, a common shared weakness was the lack of a framework or model for their evaluation work. Our assessment revealed that if the Beacon Communities had shared a framework or approach to evaluation for their implementations at the beginning, their work may have been even more effective and their impact more easily measured.

Each community developed an evaluation plan as part of the Beacon Community grant application. However, many of the case studies evolved from the time the applications were submitted, and the communities were not able to make corresponding changes to the evaluation plan because of resource constraints. According to a staff member from one of the txt4health communities, “I also want to point out that, from the outset and as all three Beacon Communities were planning their campaign, we were solely relying on the back end program data for evaluation purposes.” This issue highlights the need for better evaluation planning at the formative stages and for better data capture and analysis. The framework and associated questions used in this paper were found to be robust and worked well for a multisite evaluation. Supporting evaluation that can inform future implementations is an important area of opportunity for future work, and the framework and questions used in this paper can be a starting place for doing so.

Other benefits have resulted from the broader Beacon Community work. As part of the Beacon Nation, funded by the Hawai’i Island Beacon Community, an awardee of the ONC Beacon Community Program, six learning guides have been developed that highlight lessons learned from the 17 Beacon Communities.16 Additionally, lessons learned from the Beacon Communities have informed ONC’s Standards and Interoperability Framework initiative, designed to empower health care stakeholders to establish standards, specifications, and other implementation guidance that facilitate effective health care information exchange.17