The National United States Center Data Repository: Core essential interprofessional practice & education data enabling triple aim analytics.

Understanding the impact that interprofessional education and collaborative practice (IPECP) might have on triple aim patient outcomes is of high interest to health care providers, educators, administrators, and policy makers. Before the work undertaken by the National Center for Interprofessional Practice and Education at the University of Minnesota, no standard mechanism to acquire and report outcome data related to interprofessional education and collaborative practice and its effect on triple aim outcomes existed. This article describes the development and adoption of the National Center Data Repository (NCDR) designed to capture data related to IPECP processes and outcomes to support analyses of the relationship of IPECP on the Triple Aim. The data collection methods, web-based survey design and implementation process are discussed. The implications of this informatics work to the field of IPECP and health care quality and safety include creating standardized capacity to describe interprofessional practice and measure outcomes connecting interprofessional education and collaborative practice to the triple aim within and across sites/settings, leveraging an accessible data collection process using user friendly web-based survey design to support large data scholarship and instrument testing, and establishing standardized data elements and variables that can potentially lead to enhancements to national/international information system and academic accreditation standards to further team-based, interprofessional, collaborative research in the field.

Introduction

Health care decision makers are calling for information and evidence clarifying the value-add of interprofessional education and care models (Institute of Medicine, 2015; Reeves, Perrier, Goldman, & Freeth, 2013; REFS). For example, in 2014, the Institute of Medicine charged a committee to “analyze the available data and information to determine the best methods for measuring the impact of interprofessional education (IPE) on specific aspects of health care delivery and health care systems functioning, such as IPE impacts on collaborative practice and patient outcomes (including safety and quality of care)” (pp. 23–34). The National Center for Interprofessional Practice and Education (hereafter the National Center) in the Academic Health Center at the University of Minnesota (www.nexusipe.org) is committed to providing the leadership, resources and evidence needed to inform interested parties on how interprofessional education and collaborative practice (IPECP) might enhance the experience of health care, improve population health, and reduce the overall cost of care. Many have noted that little rigorously generated evidence has been produced demonstrating the impact of interprofessional education (IPE) on collaborative practice (CP) that in turn positively impacts satisfaction with care received, care provision, population health, and per capita cost of care (e.g., Brandt, Lutfiyya, King, & Chioreso, 2014; Reeves et al., 2008, 2013; Zwarenstein, Goldman, & Reeves, 2009). To address this gap, the National Center has created a Nexus Innovation Network (NIN) to test the connections among IPE, CP, and health-related outcomes through comparative effectiveness research.

A number of well-designed interventions have been implemented and the relevant data is being collected in order to ascertain the effectiveness of the interventions. Two earlier papers (Lutfiyya, Brandt, Delaney, Pechacek, & Cerra, 2015; Pechacek, Cerra, Brandt, Lutfiyya, & Delany, 2015) have addressed the National Center’s Nexus Innovation Network (Pechacek et al., 2015) and the National Center’s research agenda (Lutfiyya et al., 2015). This paper describes the National Center’s Data Repository (NCDR) including the structure, databases and interface with the Nexus Innovation Network (NIN). The NCDR is built to capture and store the data related to IPECP processes and outcomes to support analyses of the impact of IPECP on the triple aim (Berwick, Nolan, & Whittington, 2008) outcomes. All members of the NIN participate in data collection on the outcomes related to their unique interventions. The structure of the NCDR and the essential interface with the NIN and a brief overview of the development and adoption of the NCDR are described in this paper.

Nexus innovation network

Earlier papers (Lutfiyya et al., 2015; Pechacek et al., 2015) describe the Nexus Innovation Network (NIN). As described by Lutfiyya et al., the NIN is a collaborative of higher education and health system partners in the United States committed to studying and advancing interprofessional practice and education together. Incubator members are testing new organizational, care delivery, and learning models in real-world settings; identifying and collecting data to create an evidence base for IPECP; identifying evidence-based models to educate health professions students and practitioners; and training faculty, students, clinicians and staff as teams to build leadership skills and develop the capacity for data collection and intervention research. All members of the NIN commit to participate in the NCDR data collection strategy. The landscape of the Incubator Network is constantly in flux as new sites and new intervention projects are added to the network. The process for becoming an incubator site is described elsewhere (Pechacek et al., 2015).

The National Center Data Repository

As an essential part of the evidence generating strategy of the National Center, development of the National Center Data Repository (NCDR) was a top priority, and initiated in the fall of 2013. Part of the early development entailed creating an advisory committee consist of recognized experts in the fields of clinical practice, health professions education, health services research and informatics The NCDR consists of four components: (1) NIN data, (2) tool testing, (3) third party data, and (4) partner sources data. These four discrete components of the database support the IPECP data needs for the National Center and IPECP scientists and researchers (Figure 1). The following section provides a description of each component.

Figure 1.

National Center Data Repository.

Data components

Standardized data are collected from each NIN site. Some of the data collected are common to all sites while some of the data collected are site specific. Data are entered into the NCDR via web-based surveys. Table I displays the data collected by the specific NCDR surveys. The NIN Data collection components consist of eight web-based surveys that support data collection, each survey is described in more detail below.

Table I.

National Center Data Repository Surveys*.

Survey	Respondents	Questions	Response type	Time to complete
Demographics survey	All	Six questions creating a personal profile	Multiple choice	<5 minutes
Education Content & Process survey	Lead of the Inter-professional education initiative (with input from associated educational units)	24 questions about the interprofessional education program (one survey/unique facility or site)	Multiple choice with ability to add open ended comments	15–20 minutes (following 1–2 collective hours data gathering with educational leaders)
Costs to design, implement and evaluate the Site Intervention Survey	Intervention PI/lead – with consultation from team of educational, clinical, finance, and administrative leaders engaged in implementing the intervention.	51 questions related to general financial data (one survey/unique facility or site)	Multiple choice with ability to add open ended comments	30 min (following 1–2 collective hours’ data gathering from finance, operations, HR, facilities, etc.)
User Perception Survey	All clinical and educational participants in the Intervention (e.g., clinicians, faculty)	32 questions related to interprofessional education and collaborative teamwork at the clinical intervention site	Likert Scale	20–30 min (following convening of all team members who will work on the project)
Student User Perception Survey	All student learner participants in the intervention	16 questions related to Interprofessional education and collaborative teamwork at the clinical intervention site	Likert Scale	20–30 minu (following convening of all team members who will work on the project)
Technology Survey	Tech expert (CIO/clinical analyst)	What and how technology is used in each facility (one survey/unique facility or site)	Open ended narrative and Multiple choice	30 min (following 1 – 2 collective hours’ data gathering)
Intervention Specific Survey	To be determined by intervention site team in consultation with the National Center	Standardized aggregate questions related to the processes, outputs and outcomes of the intervention as well as intervention specific data elements.	Varies	Determined by the survey designed for the individual project(s)
Critical Incidents/Ecological Survey	All clinical and educational participants in the intervention (e.g., clinicians, faculty) Complete only when a “critical incident” occurs	Five questions asking the who, what, where, when, how of the incident and your subsequent actions	Open-ended narrative	5–20 min depending on the extent of the issue.

*Copies of all of the surveys with the response categories can be found on the National Center’s website.

Demographics survey

This survey collects demographic data include the participants’ age, race, ethnicity, sex, education level, and professional title. Each person is uniquely identified in the NCDR with a unique identification number and a unique password. Each person indicates their role on the team, their education level and professional designation. Data access rights within the NCDR are determined by designated role of the person on the team.

Education content and process survey

This tool collects data on the current educational infrastructure at each site, the amount of education delivered, and the type of the education provided to the interprofessional team. A single survey is completed for each intervention project.

Costs to design, implement, and evaluate the site intervention survey

This survey collects data related to the costs of the intervention. The goal of this survey is to collect data to ascertain the actual costs of the intervention, where each cost is generated and in which phase of the intervention (design, implementation, evaluation, and sustainability) the cost is incurred. Data related to the people involved in the project, such as businesses leaders, information technology technicians, project leaders, lobbyists, administrators, and financial planners, are collected.

User perceptions survey

This tool focuses on education, training, teamwork, and health team socialization of the participants. This survey seeks to measure perception of the team members in the domains of changing knowledge, capacity building, health, health education, health sector policy and practice, social, and economics. Non-students involved in the intervention project complete this survey.

Student user perceptions survey

This tool focuses on education, training, teamwork, and health team socialization of the students involved in the site interventions. This survey also measures perception of the student team members in the domains of changing knowledge, capacity building, health, health education, health sector policy and practice, social, and economics.

Technology survey

This survey focuses on the technology readiness, electronic health record (EHR) type, and data use for each intervention. Additional data related to mobile devices, patient portal, and standard languages are collected. The Technology Survey is based on the Healthcare Information and Management Systems Society (HIMSS) EHR Adoption Model, an eight-step process that allows hospitals to track the progress against healthcare organizations across the country (HIMSS Analytics, 2013). The HIMSS Analytics EHR Adoption Model identifies the levels of EHR use, ranging from minimal EHR systems to a paperless EHR environment.

Intervention specific survey

This focuses on intervention specific demographics, process of care, and outcomes expected as a result of each specific intervention. There are two components to the Intervention Specific Survey (ISS) – a common component that collects data common to all the intervention sites, and an outcome intervention specific component. The second component – the outcome-specific component – collects data on the outcomes of each intervention. The ISS survey common component focuses on the site context, team make-up, process of care, satisfaction of patients and providers, return on investment, and quality elements related to interprofessional teamwork. The intervention-specific outcomes are the measured impacts the intervention has identified. A requirement of every intervention is that the outcomes of the project must relate to triple aim outcomes.

Critical incident/ecological survey

This collects data on factors that are external to the NIN and/or the intervention and may impact the ability of the team to accomplish their goals (i.e., decision making structures, departure of the project leader). This survey addresses the factors that the team encounters throughout the life cycle of the intervention. These factors can also include components of the environment that may affect the outcomes at a particular site – such as the specific patient population, type of clinical setting, or organizational structure. Critical incidents and ecological factors can also occur in the form of new national preventive screening guidelines, changes in leadership (Provost, CEO), and new policies such as scope of practice changes. Ecological factors are important in terms of transferability of the intervention to other sites and locales.

The NIN data component utilizes web-based surveys to collect standardized data. These surveys were developed by the National Center data team made up of people from the fields of medicine, nursing, informatics, epidemiology, health services research, program evaluation, and health professions education. The surveys were reviewed for content validity by both the National Center data team and the NCDR Advisory Council. Moreover, the NCDR Surveys were validated by an external review group comprised of experts from economics, sociology, education, clinical practice, epidemiology, health administration, methodologists and experts on queries and metrics.

Survey reliability was determined by administering the core NCDR surveys to two independent NIN teams. Twenty-three out of 40 surveys were completed. Respondents provided feedback to the NCDR data team on the questions, response categories, and concepts. The National Center data team revised the surveys based on the feedback. Additional feedback was obtained by one NIN team related to the User Perception Survey. It was determined that the questions asked of clinician users and student users were sufficiently different to warrant development of a separate survey for students.

The NCDR resides in the University of Minnesota Academic Health Center Information Systems infrastructure (AHC IS), a trusted, secure environment that supports the access, storage, and use of clinical data. The AHC IS infrastructure assures best practice informatics industry standards, policies, and procedures for privacy, access, and governance of data. The web-based application is written using ColdFusion programming language. All data collected within the NCDR are stored in an Oracle database. The permission level is determined by each respondent’s role on the intervention team. For example, intervention leaders have access to local administrator functions to add and delete users; the intervention users have access to the type of survey(s) that corresponds to their unique role on the team. All users are uniquely identified, and all have a unique password. All data are stored indefinitely.

End-to-end testing of the NCDR database included the examination of all the NCDR surveys and response categories. Survey data from the two independent NIN sites were entered into the NCDR, and subsequently downloaded to a character-separated file (CSV). The data strings were analyzed to assure the correct values were recorded. The data were further uploaded to both Statistical Package for the Social Sciences (IBM, Chicago, IL, SPSS version 22.0) and Statistical Analysis System (SAS Inc., version 8, Cary, NC) and tested for consistency and accuracy of the data elements.

Each intervention site completes the web-based NCDR surveys. These data are submitted at baseline (before initiation of an intervention) and annually unless it is determined that a site should submit data at shorter intervals (e.g., quarterly). This supports longitudinal measurement. The data are used to report on the status of successes, failures, mitigating factors, and performance improvement recommendations for each intervention.

The National Center and the intervention site leaders work together to develop work plans related to survey completion. All members of the interprofessional teams that participate in the interventions have an opportunity to share their perceptions with the National Center through the NCDR survey process. Clinician participants are notified by intervention leaders (health system/education system) about their relationship with the National Center. They are invited to participate in the NCDR data collection efforts and are given written information on how to access and respond to the surveys in the NCDR (see Table I for individual responsibility for completing each survey).

Each site is assigned an NCDR coach from the National Center. The coach works with each site individually in order to fully understand the planned interventions and the outcomes being examined or studied. The site intervention leader along with their team completes the Intervention Project Work Plan. This work plan is reviewed and must be approved by the National Center’s scientific review committee. Once approved, this document serves as a template of the intervention and guides the development of the outcomes part of the ISS. The NCDR coach and National Center scientific review team work with the leaders and data professionals for each intervention to define the outcome elements of the ISS ensuring standard measures are used whenever possible.

Tool testing components

The Tool testing component of the NCDR is reserved for IPE leaders and scientists developing measurement tools/instruments that they are interested in validating. Instrument developers work with the appropriate National Center staff to deploy the measurement tool through the secure NCDR platform. The measurement developers must identify the population about whom the instrument is being validated, provide email contact information for appropriate people to collect validation data from, and establish the necessary sample size through a power analysis. Once sufficient data are collected for validation by either exploratory or confirmatory factor analysis, National Center scientific staff will provide assistance with data analysis. It is up to the measurement developers to disseminate the validation findings.

Third party data component

Third Party Data Component is reserved for sites across the nation that are performing IPECP interventions and wish to have their data stored and analyzed in and with the National Center. The type of data submitted includes aggregate process of care and de-identified individual or aggregated patient outcomes data. This domain of the NCDR is active, collecting data and in the final stages of database construction.

Partner sources data component

The NCDR is also used to house standardized data from National Center partner sources. In these partnerships, interventions and outcomes for IPECP are co-created. The National Center, with the partner, analyzes the data and reports the findings. This domain of the NCDR is actively collecting data and in the final stages of database construction.

Analytics

Previous work has addressed the analytical approaches (Pechacek et. al., 2015) and research agenda (Lutfiyya et al., 2015) for the NCDR data and how the creation of the NCDR begins to address some of the gaps identified in the field of IPECP (Brandt et al., 2014). The NCDR Third Party and Partnership data will augment this prospective effort.

Discussion

The NCDR is one of three National Center strategies to generate standardized data that is analyzable and that can both produce new knowledge and test models regarding the possible connections between IPECP and triple aim outcomes. The other two strategies are establishing a research agenda (Lutfiyya et al., 2015) and articulating analytic approaches for rigorously generated data (Pechacek et. al., 2015). This data and informatics development work has a many decades long history (Brandt et al., 2014). Recent reviews and commentaries underscore the necessity and importance of a shared lexicon and concepts for the field of IPECP, which would build upon the discipline-specific and interprofessional work. For example, the NCDR work builds upon and can potentially enhance the National Library of Medicine’s Unified Medical Language System (UMLS) which integrates and distributes key terminology, classification and coding standards, and associated resources to promote creation of more effective and interoperable biomedical information systems and services, including electronic health records (http://www.nlm.nih.gov/research/umls/ http://www.nlm.nih.gov/research/umls/). SNOMED CT is another example; as Warren (2012) summarized in her 2012 presentation “Standardized Terminologies Used in Learning Health Systems, SNOMED CT is a patient-centric, interprofessional clinically focused terminology developed to be used in Electronic Health Records (EHRs) for data entry and retrieval, and is optimized for International Health Terminology Standards. (Development Organization: http://www.ihtsdo.org/snomed-ct)

Sound scientific analyses of rigorously generated and collected data regarding IPECP efforts and interventions can be further empowered through coupling with the data resources supported through the Clinical Awards Translational Science Awards (CTSA), a national program addressing the development and implementation of national standards and best practices for translation, from basic discovery to clinical and community-engaged research. The CTSA supports a national network of medical research institutions collaborating to transform how clinical and translational science is conducted nationwide (http://www.ncats.nih.gov/ctsa). The National Center and NCDR provide a rigorous addition to the potential collaboration with the CTSA interprofessional program. Moreover, the NCDR work can relate and contribute to the Patient-Centered Outcomes Research Institute (PCORI) dedicated to improving the quality and relevance of evidence available to help patients, caregivers, clinicians, employers, insurers, and policy makers make informed health decisions through a particular approach to clinical effectiveness research, Patient-Centered Outcomes Research (PCOR). This research addresses the questions and concerns most relevant to patients and involves patients, caregivers, clinicians, and other healthcare stakeholders, along with researchers, throughout the process. Potential collaboration between the NCDR and the PCORnet, a large, highly representative national network and resource of clinical data gathered in “real time” and in “real-world” settings, such as clinics would leverage multiple data and expert resources. PCORnet data is collected and stored in standardized, interoperable formats under rigorous security protocols, and data sharing across health data networks (N = 29) uses a variety of methods that ensure confidentiality by preventing patient identification. Potential collaboration with the NCDR to expand knowledge building and examine the relationship of IPECP in the Triple Aim is recommended (http://www.pcori.org/about-us).

As noted in the Introduction as a key driver for the development of the NCDR, IPECP currently lacks evidence of efficacy (Paradis & Whitehead, 2015) that others (Hadara & Lingard, 2013) have observed. The field not only lacks a cohesive and coherent language of concepts but that the field has different and somewhat conflicting discourses muddling understanding of what IPE and CP are as well as the impact might be of either. The previous identified initiatives in partnership with the National Center can promote a cohesive, coherent language, and collaborative research.

The Institute of Medicine (IOM) recent convening of a Committee on Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes, charging them with: analyzing available data and information to ascertain the best methods for measuring the impact of IPE on healthcare delivery and on functioning of healthcare systems is clear indication of the import of this interprofessional priority (Institute of Medicine, 2015). Among other charges, this committee’s will identify how IPE and performance in practice, including the impact of IPE on patient and population health and healthcare delivery system outcomes, are linked (Institute of Medicine, 2015). That this committee was created emphasizes the necessity and importance of the informatics and data work undertaken by the National Center on Interprofessional Practice and Education.

Concluding comments

The NCDR is a critical component of the National Center and provides continuously increasing access to data generated by the NIN and other sources. The NCDR provides the essential infrastructure and access to data to lead to better understanding of the impact interprofessional education and collaborative practice has on triple aim outcomes. The NCDR provides the essential infrastructure and access to data that would support build collaboration with other national initiatives such as CTSA and PCOR. This infrastructure and the data captured in the NCDR will serve to inform policy, clinical, and educational practice and academic decisions for generations to come.