Exploring the psychometric properties of the Rainbow Model of Integrated Care measurement tool for care providers in Australia.

OBJECTIVE: To assess the reliability and validity of a shortened version of the Rainbow Model of Integrated Care (RMIC) measurement tool (MT). The original version of the measurement tool has been modified (shortened) for the Australian context.
DESIGN: Validation of the psychometric properties of the RMIC-MT.
SETTING: Healthcare providers providing services to a geographically defined rural area in New South Wales (NSW), Australia. PARTICIPANTS: A sample of 56 healthcare providers providing mental and physical healthcare. MAIN OUTCOME MEASURES: The psychometric properties of the tool were tested using principal component analysis for validity and Cronbach's alpha for reliability.
RESULTS: The tool was shown to have good validity and reliability. The 35 items used in the shortened version of the tool were reduced to 29 items grouped into four dimensions: community-governance orientation, normative integration, functional integration and clinical-professional coordination.
CONCLUSIONS: The shortened version of the RMIC-MT is a valid and reliable tool that evaluates integrated care from a healthcare provider's perspective in NSW, Australia. In order to assess the tool's appropriateness in an international context, future studies should focus on validating the tool in other healthcare settings. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The Rainbow Model of Integrated Care (RMIC) measurement tool (MT) allows for an all-inclusive evaluation of integrated care across the micro, meso and macro levels of a health system.

The RMIC-MT is based on the theoretical foundation of a unified integrated care model (RMIC) that conceptualises the different domains of integrated care into a unified model.

This study provides a very detailed, comprehensive methodological layout that can be replicated in future validation studies not limited to the integrated care domain.

Integrated care was evaluated from the perspective of healthcare providers, however a patient perspective evaluation is missing.

Including patient perspectives would enable a more holistic assessment of integration.

A sample size larger than 56 participants is favoured to increase the robustness of the results.

Introduction

Several studies have indicated that integrated healthcare delivery improves the performance of health systems by improving the quality of care and patient satisfaction.1–3 Integration is achieved by a series of coordinated activities among healthcare professionals that aim to achieve continuity of care to address the complex needs of patients with chronic conditions.4 5 Accordingly, a health system is transformed from multiple separate entities to a collectivist mechanism of healthcare services working together. This reduces the duplication of services, provides better access to primary care and improves the quality of service being delivered.6

Despite the increasing need for integrated care across health systems, integrated care lacks clarity in its concepts and methodologies. This has resulted in a wide range of integrated care definitions and an absence of a universal framework.7 8 WHO describes person-focused integrated care as the management and delivery of health services so that patients receive a continuum of preventive and curative services, according to their needs during their treatment and across the different levels (eg, macro, meso, micro) of the health system.9 This reinforces the notion that integrated care requires collective integration efforts from different health service units at different operational levels.4 Therefore, in order to improve the coordinated delivery of services, it is important to understand the different types and domains of any multifaceted integrated care intervention (eg, professional, organisational, clinical).10 This is addressed by the Rainbow Model of Integrated Care (RMIC) shown in figure 1, which conceptualises the different domains of integrated care into a unified model.11–13

Figure 1

The Rainbow Model of Integrated Care. Schematic representation of value-based integrated care. Adapted with permission from Rainbow of Chaos: A study into the Theory and Practice by P.P. Valentijn, 2015, Ede, Print Service Ede. Copyright 2015 by Pim P. Valentijn.

The RMIC is based on primary care tenets and outlines four dimensions of integrated care that interconnect the macro (system integration), meso (organisational, professional) and micro level (clinical integration) of a health system, and two more dimensions (functional, normative) that integrate the different health systems and the RMIC dimensions (figure 1). Primary care is considered the foundation of the RMIC and is expressed as person-focused care and population-based care. The dimensions are: clinical integration (micro level)—the extent to which person-focused care is coordinated; professional integration (meso level)—the sharing of roles, competencies and responsibilities; organisational integration (meso level)—collaboration through contracting and alliance; system integration (macro level)—the linkage of healthcare services through rules and policies; functional integration (spanning micro-meso-macro levels)—the extent to which support activities (IT and communication service) enhance service delivery, normative integration (spanning micro-meso-macro levels)—the degree to which culture and values are shared. The role of functional and normative dimensions is to integrate the different health systems and the RMIC dimensions. Person-focused and population-based care highlight the relationship between health and social problems.

In order to facilitate and evaluate integrated care models, a valid and reliable tool that can measure the extent and type of integration is required.4 5 8 Several authors have presented different integration measurements, such as the network matrix integration measurement which measures the extent, scope and depth of integration,14 the three levels of integration between medical and social services (linkage, coordination and full integration),15 the micro assessment tool16 and many other measurement tools identified by systematic literature reviews.4 5 13 Altogether, these studies reinforce the affirmation that none of the proposed integrated care tools capture all elements of integration as described by the RMIC.13 More precisely, 209 index instruments were found to relatively measure integrated care, however, most lacked the domains of professional, organisational or functional integration, and non-measured system and normative integration. Furthermore, the psychometric properties (eg, reliability, validity) were of low-to-moderate quality for the measurement properties. The development of a standardised tool is also affected by a number of challenges such as the complexity of healthcare systems, an agreement on what needs to be integrated and measured and the number of stakeholders involved.8 Therefore, in order to develop the RMIC measurement tool (RMIC-MT), a literature review and two international Delphi studies were undertaken.17 18 The preliminary version of the RMIC-MT has been pilot tested in the Netherlands,19 Singapore20 and Australia.21 However, results of these pilot studies indicated that further research was required to improve the psychometric properties of the RMIC-MT for care providers.

With the release of the Integrated Care Strategy, the New South Wales (NSW) Ministry of Health has identified improving integration of care as a major priority22 and has funded a number of local health districts to develop local integrated care initiatives. The aim of this paper is to analyse the psychometric properties of a shortened version of the RMIC-MT for care providers that was used to measure integrated care within the Australian healthcare system after the implementation of the NSW integrated care initiatives. This paper comes from a larger study which used social network analysis to assess the pattern of professional relationships between providers, early in the integrated care initiative, in order to provide a baseline to evaluate subsequent changes.

Methods

Integrated care tool amendment

To develop a shortened version of the RMIC-MT provider version because of the reasons described below, we modified a preliminary version that had been used to evaluate whether Australian integrated care projects have successfully implemented key integration strategies across the macro, meso and micro levels of a health system.21 This preliminary Australian version of the measurement tool contained 50 items or strategies grouped into the RMIC dimensions shown in figure 1. The majority of the participants (114 respondents) stated that the measurement tool was easy to understand and answer, where 36 of the 50 strategies were frequently operationalised by more than half of the participants. The results showed that micro-level strategies were highly executed in 38 different integrated care projects with less attention given to macro-level and meso-level actions.

In this study, the 50 items were reviewed both from a statistical perspective (with regard to their importance in previous factor analyses),23 and in terms of relevance to the initiative and the target population. A number of amendments were made to ensure applicability to clinicians in the Australian healthcare system. For instance, the term ‘organisations’ was replaced by the term ‘services’ since the integration study focused on the integration of services and not on organisations. In addition, questions regarding funding for the services were also excluded because modifications to funding arrangements were considered outside the scope of the project and were given less priority in understanding the level of integration. Some items were removed as they were not found to be important in the previous pilot studies,23 while others were combined. For example, the item ‘clients are involved in shaping the design and operations of the initiative’ was combined with ‘local community members are involved in shaping the design and operations of the initiative’ to give ‘local community members (including clients) are involved in shaping the design and operations of our service’. Therefore, the 50 items in the earlier Australian version of the tool were reduced to 35 items grouped into the RMIC dimensions: organisational integration (six items), normative integration (eight items), functional integration (four items), professional integration (four items), clinical integration (three items), system integration (five items), person-focused and population-based care (four items) and one item that measured services’ current overall level of integration. The item response options used a 7-point Likert scale from always or almost always (7) to not applicable (1).

Before interviewing the participants, we conducted a pilot study with two medical students from the University of Sydney and with a practice nurse local to the area who had previously worked on other research projects that involved interviewing general practitioners. The purpose of the pilot study was to gauge if the tool could be easily understood by healthcare providers and to make any changes, if required, to the content.

Participant recruitment and data collection

All services providing mental and physical healthcare within the defined geographic area were identified (53 services in total). A representative from each service (a healthcare provider) was asked to complete the cross-sectional survey during a face-to-face interview with one of the research team. The survey collected information about each participants’ demographic and service characteristics, their own years of practice, network relationships and their perceived degree of integration. The respondents completed sections on service characteristics, their own practice and the integration questionnaire themselves, and the interviewer facilitated completion of the section on network relationships. All interviews were conducted during December 2016 and January 2017 and lasted between 15 and 90 min.

Ethics approval

Participation in this study was voluntary and participants were asked to sign a consent form following a detailed written and verbal explanation of the study.

Data analyses

The data collected were sorted and cleaned in Excel and imported into the Statistical Package for the Social Sciences V.23 for the psychometric analysis. The assessment of psychometric properties was carried out in two phases: validity and reliability testing. To validate the integrated care tool, initial checks were conducted using principal component analysis (PCA) (figure 2) to test whether the data collected is suitable for factor extraction.24 It is fundamental to have a sufficient number of correlation coefficients ≥0.3. Keiser-Meyer-Olkin (KMO) should be >0.5 in order to consider the sample size sufficient in relation to the number of item in the questionnaire. A significant result for the Barlett’s test (p<0.05) is mandatory to indicate that the variables of interest are correlated. Furthermore, the measure of sampling adequacy (MSA) for each item on the diagonal should be >0.5 in the anti-image correlation table, while having the negatives on the off diagonal relatively low.25 If all of the above conditions are met, then factor extraction can be conducted.

Figure 2

Flow chart of the psychometric assessment process. MSA, individual measures of sampling adequacy; PCA, principal component analysis; KMO, Kaiser-Meyer-Olkin measure.

The number of components to be extracted in phase II of PCA was determined by the results of total variance (eigenvalue), scree plot and component matrix. Components with eigenvalue >1 were maintained. The scree plot provided more insights into the number of components to be maintained, which are represented by the number of points above the breakpoint of the line.26 The component matrix decisively determined the number of factors based on where the majority of the items were loaded.

The following step involved running the analysis based on the number of factors identified using varimax rotation.20 If the results show that the components are highly correlated (correlations >0.3), then PCA has to be rerun again using direct oblimin instead of varimax rotation. In the final step of PCA, variables with factor loadings below 0.4 were excluded because they are considered to have weak loadings. To confirm internal consistency reliability, a Cronbach’s alpha >0.7 is expected for each component.

The 7-point Likert scale included a ‘do not know’ answer for participants to choose. If ‘do not know’ answers were to be considered as missing data as suggested by Nurjono et al,20 there would be a substantial loss of information. Therefore, ‘do not know’ answers were considered as the midpoint of the 7-point Likert scale which represented a neutral response.27 28

Patient and public involvement

Patients and the public were not involved in this study. The healthcare services and providers have not been provided with the results of this study since the validation of the instrument is not their main concern. Rather, they have been provided with the results of a larger study that involves evaluating service integration using social network analysis, in which they consented to participate in.

Results

Descriptive statistics

Of the 53 services that consented to participate, 49 services completed interviews, yielding a 92% completion rate. A total of 56 interviews were undertaken (in some services more than one provider was interviewed). The services were grouped into eight categories: community-based services (12 services, 13 interviews done); services that provided outreach services to the defined geographical area (9 services, 9 interviews done); hospital-based services (4 services; 6 interviews done); government Aboriginal health services (3 services; 3 interviews done); government mental health and alcohol and drug services (4 services, 5 interviews done); non-government mental and physical health services (9 services, 9 interviews done) and general practitioners (3 services, 6 interviews done) and private allied health (5 services, 5 interviews done) (online supplementary file). Over half of the participants were in the public sector (57%), 23% worked in the private sector, 11% worked in both private and public and 9% were not for profit services. The descriptive statistics for each integrated care dimension are shown in table 1.

Table 1

Descriptive statistics for integration dimension variables

RMIC dimensions	N	Minimum	Maximum	Mean	SD
Person-focused and population-based (PP)	56	0	6	3.59	1.57
Clinical integration	56	0	6	4.84	1.37
Professional integration	56	0	6	3.88	1.76
Organisational integration	56	0	6	3.35	1.36
Functional integration	56	1	6	3.4	1.59
Normative integration	56	0	6	3.47	1.43
System integration	56	0	6	3.56	1.38

Initial conditions and suitability of data

Inspection of the correlation matrix revealed the presence of a sufficient number of correlations of 0.3 and above. The KMO yielded a value of 0.6, exceeding the recommended value of 0.5, indicating that the sample size was sufficient relative to the number of items in the integrated care scale. The Bartlett’s test of sphericity reached statistical significance (approx. Χ2=11 131.83, df=595, p<0.01), supporting the factorability of the correlation matrix. Hence, the null hypothesis that correlation between the items was not present was rejected. The MSA revealed a significant number of strong correlations (>0.5) among the individual items on the diagonal. Some of the items, such as Q9PI, had a moderate coefficient (<0.5), but the absolute value of negative values/partial correlations was small (<0.2).

Component extraction analysis

The results showed that 10 components had an eigenvalue >1 with 73.5% of the total variance explained (online supplementary file). Moreover, all components were positive, indicating that the correlation matrix used for the correlation coefficient in the preliminary analysis is suitable for factor analysis. The scree plot test shown in figure 3 was examined with the component matrix to reach the final decision on how many components to be extracted. The scree plot test showed that the breakpoint of the curve of the line was at component 5. Therefore, four components were retained for further investigation.

Figure 3

Scree plot test. The breakpoint for which the curve increases suddenly is at component 4, indicating that four components should be extracted. Blue line donates the eigenvalue for each component.

Factor rotation

The results of the varimax rotation and the component transformation matrix showed that the four components were highly correlated with correlation coefficients >0.3. Therefore, the rotation method was rerun using direct oblimin which considered the components correlated. The results showed that the 35 items of the scale loaded on 4 components as presented in table 2. The items with factor loadings below 0.4 (items 29NI, 34SI, 17OI and 24NI) were excluded from the analysis. Item 25NI loaded on both components 1 and 2, however, it was considered among the items of component 2 because it had a stronger correlation (r=−0.778).

Table 2

Factor loadings on four components using direct oblimin rotation

Content*	Pattern matrix
	Scale items	Component
		1	2	3	4
Formal agreements	14OI	0.728
Integrated care initiatives	15OI	0.666
Health status data	2PP	0.658
Service contribution	16OI	0.642
Community member involvement	1PP	0.619
Information sharing	3PP	0.595
Incentives	9PI	0.523
Provider participation	13OI	0.498
Population health outcomes	4PP	0.462
Integrated care governance	12OI	0.461
Integration strategy	32SI	0.439
Stakeholder involvement	29NI†
Policy impact	34SI
Trustworthy environment	25NI	−0.518	−0.778
Collaborative culture	30SI		−0.777
Service awareness	22NI		−0.664
Integration outcomes	33SI		−0.649
Leadership	26NI		−0.644
Extent of integration	35 overall		−0.598
Stakeholder collaboration	31SI	0.405	−0.545
Integrated care norms	27NI		−0.518		0.418
Teamwork leadership	11PI		−0.473
Integration evaluation	28NI		−0.441
Cultural differences	23NI		−0.407
Clinical information	19FI			0.767
Use of IT tools	18FI			0.744
Management of medical records	20FI			0.685
Performance data	21FI			0.563
Service commitments	17OI
Good quality care	10PI				0.762
Care coordination formality	7CI				0.761
Guidelines and protocols	6PI				0.744
Formal procedures	8CI				0.651
Client follow-up	5CI				0.593
Working relationships	24NI

Extraction method: principal component analysis. Rotation method: oblimin with Kaiser normalisation.

*Items in bold type have loadings <0.4.

† A shortened form for each item is displayed.

CI, clinical integration; FI, functional integration; NI, normative Integration; OI, organisational integration; PI, professional integration; PP, person-focused and population-based; SI, system integration.

To further validate our decision regarding 25NI, an internal consistency check was conducted on component 1 while considering item 25NI among its items. The results showed that if 25NI was deleted, Cronbach’s alpha would increase from 0.82 to 0.853. This indicated that 25NI lowered the internal consistency of component 1; and therefore, should be considered with the items of component 2 for the reliability analysis. The final results showed that 30 items should be retained for the internal consistency check.

Internal consistency analysis

The results of the reliability tests showed that the Cronbach’s alpha of the four components were between 0.731 and 0.863, indicating good internal consistency among the items. Regarding component 1, item 9PI (incentives) was deleted because it increased Cronbach’s alpha from 0.854 to 0.863, had a relatively weak loading compared with the other items loaded on component 1 and had a correlation below 0.3 with the other items. The Cronbach’s alpha for components 2, 3 and 4 were 0.858, 0.731 and 0.806, respectively. All items of components 2, 3 and 4 were retained because their exclusion from the reliability test for each component would not increase the internal consistency. The final number of items retained after the psychometric assessment was 29. Based on the factor loading results shown in table 2, the four components were named as: community-governance orientation (component 1), normative integration (component 2), functional integration (component 3) and clinical-professional coordination (component 4).

Discussion

The authentication of the validity and the reliability of the shortened RMIC-MT demonstrated good psychometric properties. The shortened version of the RMIC-MT consisted of four domains which reliably measured integrated care among care providers. In this study, we managed to create and validate a comprehensive shortened version of an integrated care tool for use by clinicians providing primary care services, which has its theoretical constructs grounded in the RMIC. The RMIC-MT used in this study is available on request.

Although the RMIC dimensions are conceptually distinct, each measuring a different form of integration, current measurement tools lack this distinction and tend to only measure the clinical and professional integration domain.13 As shown by several systematic literature reviews, none of the integrated care tools present capture all the domains of integrated care as the RMIC-MT does.4 5 13 The PCA of this study showed that some of the dimensions overlapped in measuring a certain aspect of integration. For instance, in community governance orientation (component 1), the results showed that the items associated with organisational integration (OI) and person-focused and population-based approaches (PP) loaded together. This was a surprising result considering that these integration dimensions are two separate concepts in the RMIC. The PP dimension considers the health needs of the client and the population at the centre of services’ operations. On the other hand, the OI dimension measures organisational collaboration through a common governance mechanism that involves contracting and alliances, to address the health needs of a population. Hence, the merging of the items associated with each of PP and OI suggests that population-based care cannot be achieved without governance and organisational coordination, or what is referred to as organisational integration.29 30 This is further supported by Valentijn et al,11 who state that addressing the health-related and social-related needs of a targeted population requires collective action among organisations in the form of partnerships across the different levels of a healthcare system. Therefore, community governance orientation can be defined as ‘a governance mechanism and collective action among organisations in the form of partnerships that addressed the health needs of a population’. The findings also indicate that the items of each of person-focused care and population-based care have loaded together. This was anticipated since both dimensions are primary care health oriented, and therefore, are associated.23

Regarding component 2, the items associated with normative integration loaded with that of system integration. This finding can be anticipated because the linkage between system and normative integration is supported by theoretical foundations. Normative integration interconnects the macro level with the meso level and micro level in a health system, because the sharing of culture and values across various stakeholders facilitates integration at different health system levels.31 In addition, system integration is founded on both person-focused care and population-based care, which are considered social norms that define the behaviour of healthcare stakeholders.11 All items associated with functional integration loaded together on component 3. This reinforces the notion that there exists a functional integration dimension for integrated care.17

Regarding clinical-professional integration (component 4), clinical integration items showed good consistency and high factor loadings with professional integration items. This suggests a relationship between coordinated delivery of services at the patient level and interprofessional partnerships defined by shared roles, responsibilities and documented agreements. The findings also show that two professional integration items (items 9PI and 11PI) did not load on the clinical-professional integration dimension (table 2). The loading of item 11PI (collaborative leadership) on normative integration is not surprising because it has been shown that leadership facilitates an integrative culture,32 and therefore item 11PI overlaps with the cultural dimension of integration. Although item 9PI, which measures the use of incentives (financial or other) to encourage teamwork, loaded on the community-governance orientation integration dimension, it is reasonable to reconsider the inclusion of this item in the RMIC-MT because it reduces the reliability of the subscale and appears to have interpretation issues. Clinical-professional coordination refers to the interprofessional partnerships that involves a shared understanding of competencies and responsibilities that coordinate the delivery of services at the patient level.

The strength of the study is in the RMIC and the instrument used. Theoretically, the RMIC provides conceptual clarity to the complex nature of integrated care,12 13 because it rectifies the miscomprehensions present regarding the different types and domains of the multifaceted integrated care interventions. Practically, the RMIC-MT allows for an all-inclusive evaluation of a health system because it evaluates integrated care across the micro, meso and macro levels of a health system, and from the perspective of healthcare providers. A number of limitations have been identified. Integrated care was evaluated from the perspective of healthcare providers where a patient perspective evaluation is missing. A patient perspective of integration is required in conjunction with a healthcare provider perspective in order to give a more holistic and accurate measurement of integrated care. A sample size of 56 participants is acceptable to conduct the psychometric assessment, however, a larger sample size would increase the robustness of the results.

Implications for future research

This study provides insights into future work that could be conducted in integrated care research studies.33 We have provided a very detailed, comprehensive methodological layout (figure 2) on how to analyse the psychometric properties of any integrated care tool using relevant validity and reliability tests. This chronological methodology can be operationalised in other validation studies not restricted to integrated care. It would be beneficial to conduct further validation studies, not just in Australia, but in other countries with differing healthcare systems. This would add strength to the validity of the tool and ensure greater relevance internationally.

We believe that integrated care evaluation should be performed in the form of a longitudinal study conducted with a preimplementation and postimplementation phase. This will determine whether the levels of integration across all dimensions change over time. Since this validation study is a part of a social network project that aimed to capture the structure of healthcare stakeholder relationships, future studies will explore the association between perceived levels of integrated care across its dimensions and services’ social network attributes such as density and centrality. This will assess the value of network theory and methodology in understanding integrated care and in modelling how healthcare stakeholders integrate and engage after the implementation of integrated care initiatives.

The results of the study show that the RMIC-MT is valid and reliable. Therefore, the instrument can be used as an indicator to monitor the effectiveness of the NSW integrated care programme as the domains of the integrated care intervention measured on the instrument align with the NSW health system reformation. The instrument could be used to benchmark integrated care performance across different local health districts, identifying areas in the NSW region to be strengthened by scaling up effective models of care. Aside from the research team, the psychometric assessment and the evaluation of integrated care may be of most interest to managers and policy makers who procure, manage and make decisions about local health districts services. The results provide comprehensive information regarding the choice of integrated care strategies that can be executed by providers at each of the RMIC levels. This had not been possible before this study since the distinction between the integration domains and their relevant strategies lacked definition and clarity.

Conclusion

In conclusion, we have described the validation process of a shortened version of the RMIC measurement tool within the Australian context. The results show that the tool has good validity and reliability with the number of questionnaire items reduced from 50 to 29 items, grouped into four integration dimensions: community-governance orientation, normative integration, functional integration and clinical-professional coordination. This shortened version of the RMIC measurement tool can be used to measure integrated care in different health settings from the perspective of healthcare services. This study contributes towards achieving an integrated care tool that can be applicable on an international level.34