Sustaining Research Networks: the Twenty-Year Experience of the HMO Research Network.

PURPOSE: As multi-institutional research networks assume a central role in clinical research, they must address the challenge of sustainability. Despite its importance, the concept of network sustainability has received little attention in the literature, and the sustainability strategies of durable scientific networks have not been described. INNOVATION: The Health Maintenance Organization Research Network (HMORN) is a consortium of 18 research departments in integrated health care delivery systems with over 15 million members in the United States and Israel. The HMORN has coordinated federally funded scientific networks and studies since 1994. This case study describes the HMORN approach to sustainability, proposes an operational definition of network sustainability, and identifies 10 essential elements that can enhance sustainability. CREDIBILITY: The sustainability framework proposed here is drawn from prior publications on organizational issues by HMORN investigators and from the experience of recent HMORN leaders and senior staff. CONCLUSION AND DISCUSSION: Network sustainability can be defined as (1) the development and enhancement of shared research assets to facilitate a sequence of research studies in a specific content area or multiple areas, and (2) a community of researchers and other stakeholders who reuse and develop those assets. Essential elements needed to develop the shared assets of a network include: network governance; trustworthy data and processes for sharing data; shared knowledge about research tools; administrative efficiency; physical infrastructure; and infrastructure funding. The community of researchers within a network is enhanced by: a clearly defined mission, vision and values; protection of human subjects; a culture of collaboration; and strong relationships with host organizations. While the importance of these elements varies based on the membership and goals of a network, this framework for sustainability can enhance strategic planning within the network and can guide relationships with external stakeholders.

Introduction

Since 1994, the Health Maintenance Organization Research Network (HMORN) has grown to include 18 clinical research centers in the United States and Israel that are embedded within integrated health care delivery systems. HMORN-based investigators have conducted multi-institutional epidemiological studies, comparative effectiveness research, randomized clinical trials, and health services research. Because of its longevity and productivity, the HMORN has been proposed as a model for other multi-institutional networks.1 Drawing on this 20-year experience, we propose a definition of sustainability for multi-institutional scientific networks and present a case study of the HMORN to illustrate 10 essential elements that can contribute to network sustainability.

Background and Context

Multi-institutional research networks are assuming a central role in clinical research in the United States.2,3 Studies from a single site are generally too small to identify differences in outcome across sociodemographic or clinical subgroups. The findings of single-site studies can also be affected by contextual factors such as the organization of care delivery, the characteristics of clinicians, or the quality and completeness of data. As a result, single-site clinical studies are increasingly viewed as pilot investigations to inform the design of definitive, multi-institutional research.

Academic researchers have developed multi-institutional scientific networks to recruit large cohorts of subjects and collect primary data for observational studies and intervention trials.4,5 Investigators in these networks typically share interest in a single disease or expertise in the design of epidemiological studies or Phase 3 clinical trials. Many of these networks lack organizational or financial expertise, particularly early in their development. As a result, essential research processes such as institutional review board (IRB) review, contracting, data sharing, and quality assurance often remain dependent on local norms. This lack of coordination too often leads to slow initiation of projects, failure to achieve timely recruitment targets, and prohibitive expense.1,2,4–6 Physical facilities, data infrastructure, and staff expertise are too seldom preserved or shared with other networks.

In response, scientific leaders have called for community-based, multi-institutional research networks that can study a broad array of health conditions, use electronic health records (EHRs) to identify and recruit subjects, incorporate data from clinical practice, and coordinate regulatory and administrative processes.4,7–9 Federally funded programs such as the Clinical Translational Science Award (CTSA) program,3 the Electronic Data Methods initiative from the Agency for Healthcare Quality and Research (AHRQ),10 the National Institutes of Health (NIH)-sponsored Health Care Systems Pragmatic Clinical Trials Collaboratory,9 and the National Patient-Centered Clinical Research Network (PCORnet)11 have supported this transformation.

These funders also encourage scientific networks to diversify their sources of support to enhance sustainability.12 Few case studies have described durable multi-institutional scientific networks or identified characteristics that enhance their sustainability, however. In this paper, we draw on the 20-year experience of the HMORN to propose an operational definition of sustainability and identify 10 essential elements that can contribute to network sustainability.

The HMORN: A Case Study of a Sustainable Research Network

The HMORN was founded in 1994 by 10 research departments within integrated delivery systems in the United States.13 Currently, the HMORN includes community-based research departments or institutes in 18 integrated delivery systems in the United States and Israel (Table 1). In 2012, these delivery systems maintained EHRs and administrative information for 15.7 million members, including large cohorts of individuals with common and rare diseases. For example, the HMORN-based SUPREME-DM network has access to EHRs and other data for 1.1 million individuals with diabetes.14 These systems are able to identify members who are eligible to receive services in the delivery system, and not just patients, the subset of members who use clinical services. This denominator strengthens epidemiological studies, clinical interventions, and health services research within the HMORN.

Table 1.

HMO Research Network Members - Health Systems and Affiliated Research Centers

Health System	Year Founded	Research Center	Year Founded
Essentia Health	1997	Essentia Institute of Rural Health	2010
Geisinger Health System	1915	Geisinger Center for Health Research	2003
Group Health Cooperative	1954	Group Health Research Institute	1983
Harvard Pilgrim Health Care	1969	Harvard Pilgrim Health Care Institute	1993
HealthPartners	1957	HealthPartners Institute for Education and Research	1990
Henry Ford Health System | Health Alliance Plan	1915	Henry Ford Health System Research Centers	Varies
Kaiser Foundation Hospitals – Hawaii	1958	The Center for Health Research - Hawaii	1999
Kaiser Permanente Colorado	1968	Institute for Health Research	1992
Kaiser Permanente Georgia	1985	The Center for Health Research – Southeast	1998
Kaiser Permanente Northern California	1945	Division of Research	1961
Kaiser Permanente Northwest	1945	The Center for Health Research – Northwest	1964
Kaiser Permanente Southern California	1953	Department of Research & Evaluation	1963
Maccabi Healthcare Services	1941	Maccabi Institute for Health Services Research	2004
Marshfield Clinic | Security Health Plan of Wisconsin	1916	Marshfield Clinic Research Foundation	1959
Mid-Atlantic Permanente Medical Group	1980	Mid-Atlantic Permanente Research Institute	1999
Palo Alto Medical Foundation	1930	Palo Alto Medical Foundation Research Institute	1950
Reliant Medical Group | Fallon Community Health Plan	1929	Meyers Primary Care Institute	1996
Scott & White Healthcare | Scott & White Health Plan	1897	Scott & White Center for Applied Health Research	2010

The HMORN has assessed modest membership dues since 2009, based on the size of the research department and its activity in the HMORN. These resources support part-time staff who manage network operations, finances, and communications, coordinate volunteer working groups that develop new data resources, and plan an annual HMORN national conference that convenes the Governing Board, scientific networks, programmers, research administrators, and IRB leadership.15 HMORN staff is distributed among member sites, and financial management is provided by the HealthPartners Institute for Education and Research. All HMORN members are expected to develop a virtual data warehouse (VDW) to facilitate collaboration across institutions, and to participate in network governance and scientific activities.

In aggregate, HMORN research centers employ over 1,450 scientists and staff. In 2012 these centers received over $340 million from federal agencies and other extramural sources, although not all that funding supported multi-institutional research. While the HMORN does not track the publications of all its members, investigators from Kaiser Permanente published over 1,000 scientific papers in peer-reviewed journals in 2012,16 and investigators from the Group Health Research Institute published over 350 papers in 2013.

Over its 20-year history, the HMORN has received substantial federal funding to support its scientific networks (Figure 1). Initial support was provided by the Vaccine Safety Datalink, funded by the Centers for Disease Control and Prevention in 1990.17 The Cancer Research Network (CRN), funded by the National Cancer Institute in 1999,18 and the Centers for Education and Research in Therapeutics (CERTs), funded by AHRQ in 2000,19 refined data models and governance approaches, and established new relationships among HMORN sites. Support from the National Cancer Institute has also allowed the CRN to maintain a secure document repository, to maintain data dictionaries, and to establish an “issue tracker” tool to chronicle data problems. Funding from the NIH Roadmap Initiative in 2004 helped the HMORN refine its approach to data governance, contracting and IRB review.20,21 Based on this infrastructure, HMORN investigators led 3 of 11 projects funded by AHRQ under its electronic data infrastructure and methods initiative from 2010–2013,14,22,23 and are leading or participating in 5 of 7 pragmatic clinical trials in the NIH Health Systems Collaboratory.9 A network of 9 HMORN sites (along with one safety-net delivery system) is also participating in PCORnet.

Figure 1.

Select Multi-institutional Research Projects of the HMO Research Network

Funder Acronyms:

AHRQ = Agency for Healthcare Research & Quality

CDC = Centers for Disease Control and Prevention

FDA = U.S. Food and Drug Administration

NHGRI = National Human Genome Research Institute

NHLBI = National Health, Lung and Blood Institute

NIH = National Institutes of Health

NIMH = National Institute of Mental Health

HMORN members have collaborated extensively with academic investigators. In 2012, 9 of the 17 HMORN sites in the United States partnered with academic centers through CTSA awards.24 HMORN investigators have participated in federally funded randomized clinical trials such as the Diabetes Prevention Program Research Group,25 the ACCORD Study Group,26 and the ALLHAT Trial.27 Several HMORN sites have developed biobanking resources that enhance collaboration with academic partners in genomics research.28,29

A Definition of Sustainability for Scientific Networks

Over this 20-year period, HMORN investigators have published papers that describe many aspects of network organization. To develop a definition of sustainability for scientific networks and identify essential components of these networks that might enhance sustainability, we reviewed these papers from a publicly available bibliography compiled by HMORN staff through 2012.30

Our operational definition of sustainability has two parts. First, sustainability requires the development and enhancement of shared research assets to facilitate a sequence of research studies in a specific content area or multiple areas. Second, sustainability requires a community of researchers and other stakeholders who reuse and develop those assets.

The first part of the definition focuses on the technical infrastructure of a scientific network, while the second part emphasizes its community of scientists and stakeholders.

Elements of Sustainability for Scientific Networks

Sustaining a robust multi-institutional research network requires attention to 10 specific areas that align with our definition of sustainability (Table 2): fair and transparent network governance; trustworthy data and strategies for data sharing; shared knowledge about research tools and approaches; administrative efficiency; adequate physical infrastructure; predictable infrastructure funding; a clearly defined mission, vision and values; protection of human subjects; a culture of collaboration; and strong relationships with host organizations.

Table 2.

A Definition of Sustainability for Multi-Institutional Research Networks and the Essential Elements to Achieve It

Definition/Essential Element
Definition: Development and enhancement of shared research assets to facilitate a sequence of research studies in a specific content area or in multiple areas.
Essential Elements: Clearly defined network governance Trustworthy data Shared knowledge about research tools and approaches Administrative efficiency Adequate physical infrastructure Predictable infrastructure funding
Definition: A group of researchers and other stakeholders willing to reuse and further develop those assets.
Essential Elements: Mission, vision and values Protection of human subjects A culture of collaboration Strong relationship with host organizations

Shared Research Assets

Network governance

The HMORN is a “network of networks,” with a central governance structure that supports topic-specific scientific networks. Each scientific network adapts the HMORN governance model to meet its internal needs.19,21,31–34 As stated in its bylaws, the HMORN is led by a Governing Board (GB) that comprises the director of each research center or their representative. The GB elects a chair, vice-chair, and Executive Committee to serve two-year terms. The GB is responsible for HMORN strategic planning, management of member dues, membership applications and evaluation, and internal and external communications. The GB may assist in the selection of leaders for scientific networks, but final authority resides with the investigators in that network. The GB delegates oversight of data resources, administrative and research tools, and approaches to human subjects review to an Asset Stewardship Committee that includes directors of scientific networks and other senior investigators and staff.

Data and data sharing

The data infrastructure of the HMORN is based on a common data model, standardized processes for improving data quality and validity, and governance of data sharing.

Data models from early scientific networks were unified into the HMORN virtual data warehouse (VDW) in 2004.31 Each HMORN site imports information from EHRs and other clinical and administrative sources into a set of VDW data tables with standardized variable names, labels, definitions, and coding. The VDW provides a common backbone for diverse studies, freeing individual projects to develop new variables to address specific scientific questions.32,34 As a result, the content and organization of the VDW continue to evolve.35

Since most data used in HMORN studies are derived from routine patient care and operations, researchers have little ability to improve data quality at the point of collection. In response to concerns about the validity of clinically derived data,36 the HMORN has developed approaches for understanding data provenance37 and improving data quality within the VDW.17,19,31,38–40 These approaches informed a recent conceptual model of iterative data quality assessment.41 Each HMORN organization first assesses the quality and completeness of its data using standardized approaches. When data from multiple sites are aggregated, site-level discrepancies that cannot be attributed to clinical variation prompt additional data quality improvement within sites. Data quality problems at either step can lead to a review of processes for extracting, translating, and loading (ETL) data into the research data set, and, where necessary, comparison with external sources such as medical records or tumor registries to assess the validity of specific data elements42–52 and to validate clinical phenotypes using combinations of administrative codes, laboratory findings, and medication fills.14,44,50,51,53

The HMORN uses a distributed model of data sharing in which primary data are retained within the host organizations. Only study-specific, de-identified, or limited data sets are shared for analysis.22,33,54–57 This distributed model protects data privacy, reduces proprietary concerns of host organizations, permits sites to authorize data queries, and engages local knowledge to facilitate data interpretation.58 In recent years, funding from AHRQ and the U.S. Food and Drug Administration (FDA) supported the development of query tools to facilitate study planning and data sharing for research and public health surveillance.22,34,56,57 Extensive tests of PopMedNet, the query tool for the FDA MiniSentinel Network, have shown that data queries received accurate responses within two days.57

Knowledge about research tools and approaches

The HMORN has developed useful research tools such as recruitment materials, computer code, surveys, and dissemination strategies.59–63 The HMORN has not developed a consistent strategy for sharing this knowledge, however. One group of HMORN investigators developed a web-based compendium of research resources in collaboration with several academic centers through the CTSA program, but dissemination was limited and the project was not sustained.59,64,65

Communication tools to connect researchers and disseminate opportunities for participation in new projects are another aspect of knowledge management. HMORN researchers are linked through a directory of HMORN investigators and scientific interest groups. External investigators can seek collaboration with HMORN scientists through the HMORN website. The HMORN website, monthly newsletter, listserv lists, and committees also disseminate knowledge and opportunities.

Administrative efficiency

Historically, business processes such as contracting, data sharing agreements, and financial closeout of studies differed between research institutes. Inefficiencies at the local level were compounded when these administrative activities were carried out across multiple organizations.34,59,64 In recent years, HMORN administrators have developed standard templates for contracts and data use agreements. While metrics for assessing administrative processes such as contracting are increasingly used, such information is not yet consistently collected or reported. As a result of these efforts, the time needed to execute contracts within Kaiser Permanente Colorado has decreased by almost 90 percent since 2010, for example.

Physical infrastructure

Observational epidemiology, comparative effectiveness and patient-centered outcomes research, pragmatic clinical trials, and health services research require little physical infrastructure beyond office space and computing resources. Some HMORN sites also have research clinics and staff to conduct Phase 3 clinical trials of new drugs and devices. Several HMORN sites have established biorepositories, but these efforts have not been coordinated across HMORN members to date.28,29

Infrastructure funding

Most HMORN assets were initially developed for studies in a specific content area. In recent years, member dues have enabled the HMORN to curate these assets and adapt them into broadly useful tools. Even though funding devoted to the HMORN is much less than the direct and indirect research funding of the individual research centers, these funds have proven essential to coordinate network growth and collaboration. Figure 1 shows that external support for the HMORN can be characterized as a “relay race,” in which different funders have supported scientific networks for variable periods of time. Even these network grants and the indirect costs associated with individual research awards such as R01s have not provided sufficient funding to sustain the Network, however. All HMORN sites receive variable amounts of direct or in-kind support from their host health systems, which underwrites activities such as management of tumor registries that satisfy regulatory requirements while providing high-quality data for research. Most HMORN members also evaluate internally funded programs developed by operational or clinical leaders to improve care delivery.66

Relationships Among Researchers and Stakeholders

The scientific community of the HMORN has been sustained through a shared mission, vision and values, common strategies to protect the privacy and security of members and their data, relationship-building among researchers, and engagement with the host delivery systems.

Mission, vision, and values

The 2012 update of the mission, vision and values of the HMORN is shown in Table 3. These core elements emphasize the importance of conducting public-domain research to benefit both the health plans and the broader community.

Table 3.

HMORN Mission, Mission and Values

Mission	To improve individual and population health through research that connects the resources and capabilities of learning health care systems.
Vision	The HMO Research Network is the nation’s preeminent source of population-based research that measurably improves health and health care.
Values	Scientific excellence Innovation and creativity Actionable research findings Collaboration and teamwork Transparency Efficiency

Protection of human subjects

Efficient and consistent IRB review across a network is an essential component of sustainability. A decade ago, HMORN investigators described problems in human subjects review that were similar to other multi-institutional studies or consortia. For example, Greene and colleagues described the variability in IRB reviews across six HMORN sites for a low-risk, mailed survey on cancer survivorship.67 Two to eight revisions were necessary to finalize the protocol, delaying initiation of the study and increasing costs. Although the HMORN does not use a central IRB, local IRB administrators convene on a regular basis to develop common approaches to oversight of multi-institutional studies. In general, the IRB for the lead research site serves as the primary IRB for data-only studies, and other participating sites cede oversight to that IRB. Local IRB oversight remains the norm for studies that involve patient contact, even if minimal risk.14,68,69 Differences in IRB management software across organizations continue to hamper efforts to develop common metrics to assess improvement in IRB efficiency.

A culture of collaboration

Although the importance of organizational culture is widely acknowledged, neither the HMORN nor other multi-institutional networks have systematically described their cultures or the effects of these on scientific productivity or administrative efficiency. The HMORN is a loose federation, since it is unincorporated and has no financial resources beyond membership dues. The leadership structure is consensus-based and lacks formal authority over the decisions of individual scientists or research centers. As in most organizations, many critical aspects of the culture of the HMORN are independent of its governance. For example, an important bond between HMORN investigators is their decision to base research careers within community-based delivery systems rather than academic medical centers. Collaboration among the highly trained and dedicated project managers, programmers, statisticians, and administrators in the HMORN is also critical to sustainability, since these individuals maintain the institutional memory of that network across projects and over time.

Relationships with host organizations

The health systems that host HMORN research departments differ substantially in their organizational support for research. Many HMORN-based researchers maintain clinical practices within their systems, allowing them to align research with the needs of the broader organization. Clinical trial units in many HMORN research departments offer opportunities for full-time clinicians to participate in research.

Surveys of primary care physicians provide one method for assessing organizational priorities, and have achieved response rates up to 91 percent.70 Four studies have assessed the interest and concerns of HMO leaders, clinicians, and members about traditional or cluster-randomized trials within the health plan.71–74 Clinicians were enthusiastic about participation as long as their time demands were not excessive and research staff could complete IRB applications and manage timelines and budgets on their behalf.71,73 Organizational leaders raised concerns about the financial impact of clinical trials, the alignment of trials with their organizational mission, and the impact on member satisfaction of informing members about their eligibility for trials. Health plan members worried that participating in trials might jeopardize their care, although these concerns could be alleviated by a trusted physician.74 In response to such operational concerns, successful clinical trials within the HMORN generally are designed to minimize the impact on routine clinical workflow.75

The conceptual model of the Learning Healthcare System emphasizes that ongoing organizational input promotes rapid dissemination of research findings into practice.76 Although examples of successful organizationally driven research within the HMORN have been published,66,77,78 the Network has not developed a systematic approach to identifying and reporting collaborative projects between researchers and clinical/operational leaders.

Discussion

Our proposed definition and essential elements of sustainability demonstrate that sustaining the HMORN has required attention to a broad array of technical and cultural issues. Even after 20 years, the HMORN has developed only provisional solutions to some of these concerns. We recognize that networks based in academic medical centers or community practices that have different organizational structures and incentives may need to address different elements of sustainability. Regardless of the structure of the network, the rapid pace of change in science and technology will require constant reevaluation of these fundamental issues.

A lack of information-sharing between networks is a barrier to sustainability in its own right. Few of the thousands of scientific papers published by HMORN investigators have addressed organizational issues.30 Other practice-based research networks, cooperative clinical trial groups, and academic medical centers (through Clinical Translational Science Awards) have also reported strategies to improve efficiency and governance.3,21,79–84 Areas of current emphasis in these settings include data models,85,86 approaches to querying multiple sites in planning for research studies,87 and strategies for accelerating human subjects review.88 The literature on network sustainability is limited in part because few journals solicit papers that describe the organization and management of research. Substantial publication bias is also likely, since unsustainable networks and ineffective strategies are unlikely to be reported. Finally, project managers, administrators, and IRB leaders who develop innovative solutions to network problems lack incentives to publish their work.

Metrics to assess the efficiency of conducting research are needed. For example, a common definition of the time necessary to complete an IRB review could generate measures that can be used internally to improve efficiency while providing external accountability to funders and other stakeholders. Measurement strategies for improving business processes in other fields have rarely been applied to multi-institutional health care research, but can also enhance internal and external credibility.81

The Patient-Centered Outcomes Research Institute (PCORI) and other funding agencies are demanding that researchers involve patients, clinicians, operational leaders, and community members in designing, conducting, interpreting, and disseminating research. Such stakeholders may also contribute to long-term network sustainability. Including a token number of patients on network advisory committees may no longer suffice in an era when delivery systems can rapidly conduct large surveys of their members, and virtual communities of patients and caregivers share information and identify topics for research.89 Closer relationships between research and operational activities such as quality improvement or business analytics may also help improve the sustainability of these networks, Ideally, the relationship between networks and their stakeholders can become a “virtuous circle” in which stakeholder input into research enhances the value of research findings, which promotes further stakeholder trust and engagement.

Conclusions and Next Steps

Ultimately, the decision to conduct research within a multi-institutional network is based on both pragmatism and trust. Pragmatically, researchers will join networks that meet their scientific needs and improve the efficiency of research. To do so, they must develop trust in the quality of data within and between sites, the fairness and transparency of network policies and dispute resolution, the willingness of other researchers to share resources and scientific credit, and the efficiency of scientific and regulatory activities. In a recent paper, Holve proposed four “pillars of sustainability” for networks conducting research and quality improvement.12 Drawing on the experience of networks funded through the AHRQ Electronic Data Methods Forum, she identified these foundational elements: trust and value, governance, management, and financial and administrative support. Her conceptual model and our operational view of sustainability are well aligned. Case studies of other scientific networks will be essential to confirm these models of sustainability and assess their applicability to networks that vary in size or membership. By developing a strategy for sustainability, multi-institutional scientific networks can assure that they operate efficiently, remain relevant to the needs of their stakeholders, and promote scientific excellence.