Measurement of integrated healthcare delivery: a systematic review of methods and future research directions.

BACKGROUND: Integrated healthcare delivery is a policy goal of healthcare systems. There is no consensus on how to measure the concept, which makes it difficult to monitor progress.
PURPOSE: To identify the different types of methods used to measure integrated healthcare delivery with emphasis on structural, cultural and process aspects.
METHODS: Medline/Pubmed, EMBASE, Web of Science, Cochrane Library, WHOLIS, and conventional internet search engines were systematically searched for methods to measure integrated healthcare delivery (published - April 2008).
RESULTS: Twenty-four published scientific papers and documents met the inclusion criteria. In the 24 references we identified 24 different measurement methods; however, 5 methods shared theoretical framework. The methods can be categorized according to type of data source: a) questionnaire survey data, b) automated register data, or c) mixed data sources. The variety of concepts measured reflects the significant conceptual diversity within the field, and most methods lack information regarding validity and reliability.
CONCLUSION: Several methods have been developed to measure integrated healthcare delivery; 24 methods are available and some are highly developed. The objective governs the method best used. Criteria for sound measures are suggested and further developments should be based on an explicit conceptual framework and focus on simplifying and validating existing methods.

Introduction

Integrated healthcare delivery plays an increasingly important role in current healthcare reform efforts, not least in European countries, adapting US managed care concepts [1-3]. The demand for change and reforms are seemingly similar across nations [3]. Economic, political and socio-demographic forces have moved the modern healthcare system beyond the largely reactive acute care paradigm to a more holistic paradigm emphasizing optimization of the population’s health [4]. At the core of this shift is the movement away from episodic treatment of acute illness events to the provision of a coordinated continuum of services that will support those with chronic conditions and enhance the health status of defined populations [4, 5]. Many healthcare providers believe that integrated healthcare delivery will lead to higher quality care at a lower cost while maintaining or improving the recipients’ health and satisfaction. Consequently, policymakers and system planners are striving to build and manage healthcare systems that can accommodate delivery of coordinated care services, either through macro-level healthcare reform or through initiatives at organizational (meso) or physician–patient interaction (micro) level. However, monitoring the progress potentially associated with the efforts being made, and the gathering and dissemination of evidence-based knowledge is hampered by the lack of measurement methods in this area [6, 7]. The objective of this paper was therefore, to present a systematic review of methods used for measurement of integrated healthcare delivery.

Methods of the systematic review

Issues in measuring integration

The conceptual diversity used within the field of integrated healthcare delivery is vast [8]. This is understandable as integration in healthcare is a complex phenomenon. However, two distinct conceptual subcategories within the literature can be identified. They refer to integrated healthcare delivery as being either a) an organizational structure that primarily follows economic imperatives (e.g. that unites a financing group with all providers—from hospitals, clinics, and physicians to home care and long-term care facilities to pharmacies) or b) a way of organizing care delivery—by coordinating different activities to ensure harmonious functioning—ultimately to benefit the patients in terms of clinical outcome [3, 9]. In both sub-categories it is a key characteristic that the provider being assessed is not a single facility (e.g. a hospital, or general practice) but a network of providers accountable to its eligible patients [6]. Various definitions related to integrated care have been suggested [1, 3, 9–13]; however, due to the lack of consensus we decided for the purpose of this review to include a broad range of similar terms focusing on coordination and integration, all of which contain issues that can be relevant to measure in the effort to build and improve an integrated delivery of healthcare services.

Selection criteria

This review is based on a systematic approach in terms of selection criteria and a pre-planned search strategy. To be eligible for inclusion in the review the stated primary or secondary research objective should be the measurement of integrated healthcare delivery or an equivalent concept. According to Devers et al. (1994) [1] measures of integrated healthcare delivery can be grouped into three areas: 1) measures of precursors of integration (structural and cultural measures); 2) measures of intermediate outcomes or internal process variables, which assess the level of system-wide activity and which are the means to achieving ultimate goals (process measures); and 3) measures which assess the extent to which systems are fulfilling their ultimate purpose (outcome measures) [1]. Both patient and cost measures (mortality, morbidity, functional status, costs) are the end goal for improvements in care coordination [7]. Since others have identified such outcome measures within the field [14-16], we chose to focus on structural, cultural and process measures for the purpose of this review.

Patient reported perceptions of coordination have been used as a proxy measure for the overall coordination performance of providers [7]. This can be both a practical and useful approach, especially when emphasis is on the patient’s experience with a single provider or the journey through a system of providers—often referred to as continuity of care within the field [17-19]. However, the patient’s perspective gives limited insight into the many specific clinical activities coordinated into their care, and patients are unlikely to have insight into both system- and organizational-level integration activities. Furthermore, continuity of care is somewhat different from the other concepts used within the field because often it does not refer to an attribute of healthcare organizations but rather to the subjective perceptions of the patient experiencing coordinated services or integrated care [17-19]. We, therefore, decided to exclude studies specifically measuring continuity of care strictly on the basis of patient surveys. Substantial literature exists on interprofessional working and teams in health and social care and associated measurement methods [20, 21]. Although potentially relevant, these methods are outside the scope of the review. Finally, there are a number of intervention studies evaluating the effect of integrated care programmes versus a standard care programme [22]. Such studies were excluded from this review, unless the authors had clearly made an effort to measure the concept of integrated healthcare delivery.

Search strategy

The search was limited to the following bibliographic sources: Medline/PubMed (1960–April 2008), EMBASE (1966–April 2008), Web of Science (1945–April 2008), Cochrane Library (1898–April 2008) and the World Health Organization library and information networks for knowledge database (WHOLIS) (1948–April 2008). To allow for the identification of a wider range of perspectives the search was extended to include grey-zone literature such as academic working papers, ministerial reports and measures developed by consortiums or international institutions, with the use of the conventional internet search engines Google and Google Scholar (January 2008). Publications written in English, Danish, Swedish, Norwegian, and German were included. Studies written in other languages would have been included if they had had an English abstract, and if found relevant, these studies would have been translated. To systematize the search in Pubmed/Medline, terms derived from the literature were used and supplemented with relevant Medical Subject Headings (MeSH®) and limited to studies written in English, Danish, Swedish, Norwegian and German. The following MeSH® term and words were used [*denotes that different suffixes have been used]: ‘delivery of healthcare, integrated’ (MESH® term); ‘care pathway*’, ‘chains of care’, ‘care coordination’, ‘care transition’, ‘clinical integration’, ‘collaborative care’, ‘cooperative care’, ‘coordinated care’, ‘coordination of care’, ‘cross sectoral care’, ‘financial integration’, ‘functional integration’, ‘horizontal integration’, ‘integrated care’, ‘integrated service network*’, ‘integration of care’, ‘intersectoral care’, ‘intrasectoral care’, ‘linked care’, ‘physician system integration’, ‘provider system integration’, ‘seamless care’, ‘service network*’, ‘shared care’, ‘transitional care’, ‘transition of care’, ‘transmural care’, ‘vertical integration’, ‘virtual integration’, ‘whole system thinking’, ‘continuity of care’, ‘care continuity’. The search using these words resulted in 81,078 hits. When restricting the search to papers also including the term ‘measure*’, it resulted in 4515 hits in Pubmed/Medline.

The same keywords and combinations of keywords were used to search Web of Science (51 hits), Cochrane library (0 hits), WHOLIS (256 hits) and EMBASE (529 hits).

After the initial search, all title or keywords of the 5351 hits were reviewed by the investigator and a co-investigator, who applied the inclusion criteria to determine if the abstract and full paper were needed for further investigation. This process excluded 5194 papers and the remaining 157 papers were reviewed again in greater detail using a hardcopy of the full papers. In this phase a number of papers were excluded because they used patient-reported perceptions of coordination as a proxy for the overall coordination performance of providers. Reference lists of the selected publications were searched using a snowball sampling technique and any not previously discovered studies were included if found relevant; 17 scientific journal papers and one scientific working paper were kept.

To search the conventional internet search engines the search was restricted to use the phrases ‘integrated care’ and ‘measurement system’ to identify relevant publications. The search on Google resulted in 753 hits, and the search on Google Scholar resulted in 72 hits. All hits where checked for relevance by the investigator using the inclusion and exclusion criteria and 24 potentially relevant publications were identified. Of these, five were finally kept after a more detailed review of a hardcopy of the publications. Any hits linking to relevant scientific journal papers were checked to see if these papers had already been included. If this was not the case, the paper was included in the review. However, only one additional paper was identified this way. The progress through the systematic review can be seen in Figure 1. From the final set of 24 publications that met the inclusion criteria, study details were extracted using a standard form. Extracted data included: name of first author, year of publication, primary or secondary research objective, concept measured, type of data, and respondent groups if applicable.

Figure 1

Progress through the systematic review.

Analysis of measurement methods—criteria for a sound method

To analyse the identified measurement methods we used a set of criteria from classical test theory and the existing literature within the field of integrated care. The criteria used are written in italics. The first criterion is if the measurement method is based on a clear, theoretical model. The concept subject for measurement should also be clearly defined (concept defined). Since integrated healthcare delivery can be measured on different levels e.g. system, organization or person-to-person, the selected level of analysis should be made explicit (defined level of analysis). Comparing performance at different levels may reveal different results. It has been suggested to include both structural and behavioural aspects in a measure of integrated healthcare delivery [23-25]. We propose dividing the behavioural aspect into a cultural and a process aspect. A complete method for measurement of integrated healthcare delivery would, therefore, preferably include structural-, cultural- and process aspects. The structural aspects (what we have) specify the actual mechanisms (referrals, guidelines, chains of care, health information technology systems, network managers and pooled resources) that both are or should be in place within or between organizations to indicate the degree of integration that has been achieved [23, 24]. The measure should also include the cultural aspect (willingness to do) and the actual coordination processes taking place (what we do), since these are also important aspects of achieving an integrated delivery of healthcare services [11, 23, 24]. Furthermore, Ahgren and Axelsson [23] proposed that a method should measure integration relative to a perceived optimal integration target (relative measure), in order to avoid the normative implication of having higher ranks of integration always being better than lower ranks [23, 24]. For a method to be useful it is seen as important that the chosen method enables the evaluators to present quantifiable data (quantitative measure) and, preferably, it should be possible to calculate sums and mean ranks of integration [23, 24]. Internal validity of a given measure is important—the method should, for example, have high test–retest reliability and should be relatively sensitive to change. Face validity and content validity of the developed method and the results are important too. Test of validity across settings would be preferable. We used these criteria to assess each of the identified methods. We considered a criterion to be fulfilled if the criterion was explicitly described in the reference.

Results

We identified 18 scientific journal papers [1, 23, 26–40, 41], one scientific working paper [42] and five other publications [43-47] that fulfilled the selection criteria; they are presented in Tables 1 and 2. Table 1 provides a summary of measures, including the primary or secondary research objective, concept measured, type of data and respondent group if applicable. Table 2 provides an overview of methods developed by consortiums or international institutions. The identified methods were published from 1979 to 2007, with an overweight of newer studies, from 2000 and beyond. The methods clearly reflect the conceptual diversity used within the field. There is no consensus on which data sources best capture integrated healthcare delivery. The identified studies can be categorized according to type of data source: a) questionnaire survey data, b) automated register data, or c) mixed data sources. Questionnaire surveys are, however, the most widely used data source. Only two of the identified papers relied solely on automated register data. A special category is the methods developed by international consortiums (Table 2), which are mainly relevant for macro-level accreditation processes or international health system comparisons. It is unclear whether these methods are based on a theoretical framework or have been subject to scientific assessment processes. In the identified references we identified 24 different methods to measure integrated healthcare delivery. However, five methods shared theoretical framework. For each of the methods identified published in scientific journals (including the academic working paper) we assessed whether they fulfilled the aforementioned criteria. Table 3 provides an overview of criteria fulfilled for each of the identified research methods. Currently none of the identified measures fulfils all of the criteria (Table 3). Almost all methods are based on a theoretical model; some more rigorously than others, however. In most papers the concept being measured is clearly defined and all papers have described the level of analysis. Structural and process aspects are often included in the measurement methods, while cultural aspects are rarely part of the methods. Only one paper describes a method that measures integration relatively to a perceived optimal integration target. Almost all the identified methods allow evaluators to quantify their findings but only a few allow the evaluator to calculate sums and mean ranks of a combined measure of integration. While a test for some degree of internal validity has been described in 9 of 19 papers published in scientific journals (including the academic working paper), none has been thoroughly validated across different settings.

Table 1

Methods published in scientific journals or as academic working papers for measurement of integrated healthcare delivery or an equivalent concept

Reference	Primary or secondary research objective	Concept measured	Type of data	Respondent group
Ahgren and Axelsson, 2005 [23]	To “conceptualize and validate a model of measurement that can be used to evaluate the degree of integration in local healthcare and similar arrangements of integrated care”	Functional aspects of clinical integration	Self-assessment form	Healthcare unit managers as leaders of internal multi-professional meetings
Brazil et al., 2003 [26]	To “present an evaluative framework for defining and measuring service coordination”	Service coordination	Structured interviews, mailed questionnaires, semi-structured telephone interviews	Administrators and front-line staff
Browne et al., 2004 [27]	To “[introduce] a new measure of human service integration that quantifies the extent, scope and depth of the effort. It identifies which sectors, services or agencies are connected and are collaborating well with each other and which sectors and/or agencies in the network could enhance their collaborative effort”	Inter- and Intra-sectoral service integration	Telephone or in-person interviews, web-forms, or work-shop	Representatives or coordinators from each service from each sector
Burns et al., 2001 [28]	To “[investigate] the degree to which the processes of integrating physicians and hospitals are closely linked to the structure and context of integrated delivery systems”	Structural integrationProcess Integration	Qualitative interviews	Hospital executives
Devers et al., 1994 [1]	To “develop measures of three types of integration that occur in vertically integrated healthsystems”	Functional integrationPhysician–system integrationClinical integration	Questionnaire survey	Appropriate personnel in the system and operating unit offices
Fletcher et al., 1984 [41]	To “[measure] the integration of care for patients visiting a setting involving multiple providers”	Continuous careCoordinated careIntegrated care	Medical record	–
Gillies et al., 1993 [29]	To “[measure] perceived levels of clinical integration, physician–system integration, and functional integration along with the perceived effectiveness of these activities”	IntegrationFunctional integrationPhysician–system integrationClinical integration	Questionnaire survey	System and operating unit managers
Hébert and Veil, 2004 [30]	To “develop a method to measure the implementation of specific components of an Integrated Service Delivery system for the frail elderly”	Implementation of Integrated delivery system	Indicator system mixed interviews, focus groups, surveys, documentation analysis, participating observation, management data monitoring, clinical file analysis	Policymakers, managers, clinicians, clients and informal caregivers
Karmann et al., 2004 [31]	To “[assess] to what extent patient hospital careers take place in a coordinated way”	CoordinationIntra-sectoral integration	Register data on re-hospitalization rates of patients with cardiovascular diseases	–
Lukas et al., 2002 [32]	To “provide reliable indicators of system integration”	Integration structuresSystem integration	Questionnaire survey	Staff: managers, clinicians, general staff
Morrisey et al., 1999 [33]	To “develop empirical measures of physician–hospital and clinical integration”	Physician–hospital integrationClinical integration	ProPAC survey, AHA annual survey	Hospital management
Newhouse et al., 2003 [34]	To “examine the level of service integration within Maryland hospitals and service differentiation across the hospital system or network and its effect on heart-failure patients clinical and economic outcomes”	Service integrationService differentiation	Inpatient data, AHA annual survey	–
Reilly et al., 2003 [35]	To “[measure] the degree of integration of health and social service provision, as well as inter-professional team working”	Operational integration	Questionnaire survey	Consultants responsible for old age mental health services
Shukla, 1985 [36] [Revised Munson 1979]	To “develop an instrument to measure…integration and continuity of nursing care in team, modular and primary nursing units in a hospital”	Nursing care integrationCare management integrationPlan-do integration	Direct observation	–
Simoens et al., 2001 [42]	To “measure the extent to which GPs, and other health and social care professionals integrate with a local healthcare cooperative”	Local healthcare cooperatives integration	Questionnaire survey	General managers
Starfield et al., 1979 [37]	To “determine the extent to which the medical record contained evidence of coordination of care”	Coordination of care	Medical record, Direct observation	–
Wan et al., 2002 [38]	To “empirically explore whether integration mechanisms are related to an integrated healthcare delivery system's efficiency”	Informatic integrationCase managementHybrid physician–hospital integrationForward integrationBackward integrationService differentiation	Dorenfest's survey on information systems, AHA annual survey	–
Wan and Wang, 2003 [39]	To “[examine] the effects of integration on the performance ratings of the top 100 integrated healthcare networks in the United States”	Vertical [forward] integrationClinical integration via case managementInformation system integration	Mailed surveys and telephone interviews merged with Dorenfest Survey on Integrated Healthcare Systems	Chief information officers
Wang et al., 2001 [40]	To “observe the relationships of different types of vertical integration with hospital efficiency and financial performance”	Vertical integration	Register data	–

Table 2

Methods for measurement of integrated healthcare delivery or an equivalent concept developed by consortiums or international institutions

Reference	Concept measured	Type of data	Respondent group
Baldrige National Quality Program 2007 [43]	Coordination of care	Open-ended questionnaire survey	Healthcare experts
European Centre for Social Welfare Policy and Research 2003 [44]	Integrated health and social care	Individual interviewsFocus groupsHealth profile data	Service users and/or carers, key actors, key workers, and professional staff
The Joint Commission 2007 [47]	Care continuum	Staff Interview and review of documents	Leadership, staff and other stake-holders in healthcare
National Chronic Care Consortium 2001 [45]	Chronic Care Network infrastructure	Work sheet and Open-ended questionnaire survey	Healthcare staff
Organisation for Economic Cooperation and Development 2007 [46]	Coordination of care	Questionnaire survey	Ministry of Health

Table 3

Overview of criteria fulfilled for each of the methods measuring integrated healthcare delivery and/or equivalent concept, published in scientific journals or as academic working papers

Reference	Theoretical model	Defined concept	Defined level of analysis	Structural aspects	Cultural aspects	Process aspects	Relative measure*	Quantitative measure	Internal validity
Ahgren and Axelsson, 2005 [23]	√	√	√	√		√	√	√	√
Brazil et al., 2003 [26]	√	√	√	√		√
Browne et al., 2004† [27]	√	√	√	√				√	√
Burns et al., 2001 [28]	√		√	√		√		√
Devers et al., 1994 [1]	√	√	√	√	√	√		√
Fletcher et al., 1984 [41]		√	√	√				√
Gillies et al., 1993 [29]	√	√	√	√	√	√		√
Hébert and Veil, 2004 [30]	√		√	√	√	√		√	√
Karmann et al., 2004 [31]			√					√
Lukas et al., 2002 [32]	√		√	√	√	√		√	√
Morrisey et al., 1999 [33]	√	√	√	√		√		√
Newhouse et al.,2003 [34]	√	√	√					√	√
Reilly et al., 2003 [35]		√	√	√		√		√
Shukla, 1985 [36]		√	√			√		√
Simoens and Scott, 2001 [42]	√	√	√	√	√	√		√	√
Starfield et al., 1979 [37]		√	√			√		√
Wan et al., 2002 [38]	√	√	√	√				√	√
Wan and Wang et al., 2003 [39]	√	√	√	√				√	√
Wang et al., 2001 [40]	√	√	√	√				√	√

*Perceived optimal integration target included as part of the measure, to avoid the normative implications of having higher ranks of integration always being better than lower ranks.

†An updated version of this method has been described in an academic working paper in 2007. The update is based on a relative measure. The update is available from the author only (browneg@mcmaster.ca).

Discussion

The present research

When developing measures for assessment of integrated healthcare delivery we seek concrete tools for healthcare system management and research efforts. It is a challenging task since the nature and complexity of modern health systems needs to be understood in the quest to capture when integration occurs (or does not occur). The literature on measurement of integrated healthcare delivery reveals a diverse array of concepts and methods. Methods to measure integrated healthcare delivery are clearly emerging and some measures are readily applicable but resource-intensive. Nevertheless, more research is needed to ensure valid and reliable measures. It is positive that almost all methods are based in a theoretical model; however, this remains an area that needs to be developed. First and foremost theoretical models need to be empirically tested and more work is needed to clarify relationships between central concepts and interactions with patient and cost outcomes. At present, the method selected will depend on the conceptual or theoretical framework and the given objective; however, due to the relative newness of this area, established, off-the-shelf measures exactly suited to a given purpose may not be available.

Conducting a review study on methods for measurement of integrated healthcare delivery is a difficult task. The confusing terminology with interchangeable and often undefined use of terms is a challenge for the systematic approach needed when conducting literature reviews. Although we strove to identify a broad range of papers available in the literature, the interdisciplinary nature of the field makes it difficult to be sure that all relevant papers were identified. To our knowledge only two comparable reviews have been conducted. The first was a review published by Granner and Sharpe in 2004 and includes papers published in 2001 as the latest [48]. Their field of enquiry was, however, more related to the field of health education research and aimed to give a summary of measurement tools to facilitate community coalitions and partnerships in order to promote community health. However, their description of missing information regarding validity and reliability is a precise description of some of the available measurement methods within the corresponding field of integrated care [48]. In general, the field has developed since the review published by Granner and Sharpe in 2004 and the available methods have clearly been improved, especially by matching theory with conceptual and operational definitions. In 2007 a review on instruments and measures related to care coordination mechanisms or patient/family perceptions of coordination was published by McDonald et al. [7]. They identified 20 instruments and approaches; two of the identified methods were also included in our review [37, 41]. About half the instruments were targeted patients and/or family members and the rest measured aspects of interprofessional working and are, therefore, outside the scope of our review. However, the main findings are comparable to ours since they also describe a measurement field in the early phases of its development [7].

Implications for future research

In this paper a set of criteria for measurement methods has been proposed to guide future research. To measure integration relatively to a perceived optimal integration target as proposed by Ahgren et al. [23] is a real contribution to the field and this approach could with benefit be incorporated in existing methods. The criteria we used are the most relevant for methods based on quantitative data; however, criteria for methods based on qualitative data are also warranted in future studies. There is a need for both quantitative and qualitative measures. As all the methods identified are relatively resource-intensive, researchers should focus on simplifying existing methods. An ideal method should be practical and relatively simple to use, especially for non-scientific use e.g. on-going monitoring conducted by health system planners. More research is needed on the validity of the existing measures. In addition, new measures need to be developed for areas that are as yet covered inadequately.

Conclusion

Integrated healthcare delivery can be measured: 24 methods are available and some are highly developed. However, the method selected depends on the objective. Due to the relative newness of this area, established, off-the-shelf measures that suit any given purpose are not yet available. We consider this paper makes two clear and intertwined contributions. First, in terms of compiling current evidence, this kind of literature review is essential for creating a stepping-stone for further conceptual and methodological improvements. Second, in terms of practice, it provides direct research support to stakeholders in need of specific tools for evaluating system performance. Criteria for the development of existing and new measurement methods have been suggested. A range of broadly validated methods for a specific purpose could be useful to both evidence-based healthcare system management and continuous research efforts within the field of integrated care—ultimately to the benefit of the recipients of care.