Evaluating intersectoral collaboration: a model for assessment by service users.

INTRODUCTION: DELTA was launched as a project in 1997 to improve intersectoral collaboration in the rehabilitation field. In 2005 DELTA was transformed into a local association for financial co-ordination between the institutions involved. Based on a study of the DELTA service users, the purpose of this article is to develop and to validate a model that can be used to assess the integration of welfare services from the perspective of the service users. THEORY: The foundation of integration is a well functioning structure of integration. Without such structural conditions, it is difficult to develop a process of integration that combines the resources and competences of the collaborating organisations to create services advantageous for the service users. In this way, both the structure and the process will contribute to the outcome of integration.
METHOD: The study was carried out as a retrospective cross-sectional survey during two weeks, including all the current service users of DELTA. The questionnaire contained 32 questions, which were derived from the theoretical framework and research on service users, capturing perceptions of integration structure, process and outcome. Ordinal scales and open questions where used for the assessment.
RESULTS: The survey had a response rate of 82% and no serious biases of the results were detected. The study shows that the users of the rehabilitation services perceived the services as well integrated, relevant and adapted to their needs. The assessment model was tested for reliability and validity and a few modifications were suggested. Some key measurement themes were derived from the study.
CONCLUSION: The model developed in this study is an important step towards an assessment of service integration from the perspective of the service users. It needs to be further refined, however, before it can be used in other evaluations of collaboration in the provision of integrated welfare services.

Introduction

During a ten-year period between 1993 and 2003, there were a number of experiments in Sweden with collaboration between different welfare institutions in the field of vocational rehabilitation. The experiments resulted in a new legislation, the 2003 Act on Financial Coordination of Rehabilitation Measures [1], making it possible for different institutions in the rehabilitation field to form local associations for financial co-ordination.

One of the experiments preceding the legislation was a project called DELTA1 on the island of Hisingen in Gothenburg. It was launched in 1997 to promote collaboration between different institutions in the rehabilitation field in order to provide services to people who have been ill or unemployed for a long time. Many people in such a state need support from different welfare institutions, but because of that they are often left in a no man’s land.

As in many other countries, the responsibility for vocational rehabilitation in Sweden is separated between different professions, organisations and sectors of the society. Rehabilitation may therefore be hampered by different priorities, lack of communication and insufficient joint planning. DELTA has been trying to improve intersectoral collaboration, including interprofessional as well as interorganisational collaboration, in order to meet rehabilitation needs in the working-age population more efficiently. Such needs are mainly related to mental health problems, musculo-skeletal disorders, complex social problems, or long-term work incapacity.

In accordance with the new legislation, the DELTA project was in 2005 transformed into a local association for financial co-ordination between the national employment service, the regional health authority, the municipal social service, and the national social insurance administration. The DELTA association is financed by funds made available by these participating institutions. The funds are pooled into a joint budget, which is allocated to different rehabilitation services provided by the association.

Today there are 69 local associations for financial co-ordination like DELTA in Sweden [2]. Beside these associations, there are also other Swedish models for integrated provision of welfare services, for example chains of care and local health care, which have become more and more common [3]. With this development, there have also been increasing demands for assessment tools and models for evaluation of the different collaborative arrangements.

Due to the complexity of intersectoral collaboration, there is not likely to be a single comprehensive model of evaluation that can be applied everywhere. Instead, research and development has concentrated on specific models evaluating different aspects of collaboration from different perspectives. There are, for example, models assessing structural integration [4, 5] as well as functional integration [6, 7]. There are also models assessing integration effects on outcomes [8, 9].

Most of these models are evaluating collaboration from the perspective of the professionals involved. For example, in Australia, a research team has developed a model to measure the attitudes of health professionals to integration of patient outcomes, resource use and professional relationships [10]. In Canada, Browne and her colleagues have developed a similar human services integration measure based on the perceptions of the professionals involved [11, 12].

There are, however, very few evaluations of intersectoral collaboration based on the perceptions of patients, clients or other service users in the research literature. Surveys directed to these groups have been used to monitor process quality [13], but they have not been used to assess collaboration in service provision.

Against this background, the purpose of this article is to develop a model that can be used to assess the integration of welfare services from the perspective of the users and also to evaluate the results of these services. The model is based on a pilot study of the DELTA association for financial co-ordination of vocational rehabilitation and it has also been applied in a subsequent study of the DELTA service users.

Theoretical framework

Collaboration has become an increasingly important feature of most welfare systems, because of the increasing specialisation of services and the increasing professionalisation of different occupational groups. Intersectoral collaboration is the most complex form of collaboration, since it includes interprofessional as well as interorganisational collaboration between different sectors of the society. It means that the organisations and professions involved arrange their different services to fulfil needs of integration, which may be through co-ordination and co-operation as well as collaboration [14]. Integration can in this context be defined as the extent to which different welfare services are combined in a way that is consistent with the needs and personal circumstances of the service users. This is far from a precise definition of integration, but it may serve as a point of departure.

There are many different aspects to consider when assessing the integration of welfare services. Following Donabedian’s [15] classical model for evaluating the quality of medical care, there are three main aspects that should be considered in such an evaluation: the structure, the process and the outcome of care. These general aspects can also be applied in the assessment of other related phenomena, like the integration of welfare services.

According to some researchers [6, 16, 17], a basic condition for service integration is well functioning structural arrangements. The structure of integration includes such things as the access to services, the available information about the service users and their needs, the resources required for provision of the services, the professional qualifications of the personnel, and the division of tasks and responsibilities between the professions and organisations involved. Without such structural conditions it is difficult to combine the services of the collaborating institutions in a way consistent with the needs and circumstances of the users.

Following Donabedian [15], the process of integration consists of all the different activities included in the services provided. Of particular interest in this connection is the continuity and co-ordination of the different activities, the communication and the relations between users and professionals, between different professionals and between different organisations leading to a joint provision of services advantageous for the users [16, 18].

The process of integration requires adequate structural conditions, and together the process and the structure contribute to the outcome of integration. In the integration of welfare services the most important outcome is the satisfaction of the needs of the service users, for example recovery or rehabilitation, but there are also other important outcomes of integration like improvements of capacity and collaborative skills among the professionals or financial and other effects on the organisations involved [19].

These aspects can be measured along different dimensions and from different perspectives. As mentioned before, service integration has been measured mostly from the perspective of the professionals. Ultimately, however, it is only the individual service users who can really assess the extent of the different forms of integration in the actual service provision [20]. Therefore, it is important to assess the integration of welfare services from the perspective of the service users. Based on the three main aspects of integration and research on service users [13, 20–22] the following dimensions for an assessment of service integration can be derived.

The dimensions for assessing the structure of integration are, from a service user perspective, the extent to which skills, competences and information from the different organisations involved are available for the provision of services and also accessible for the users.

The dimensions referring to the process of integration can be divided into interpersonal and interprofessional dimensions. The interpersonal dimensions are the extent of personal trust and responsibility that is established between the users and the providers of services, which may add value to their repeated contacts. The interprofessional dimensions are the extent to which different professions, usually from different institutions, are involved in the provision of services or are working together as a team to provide these services.

The dimensions for assessing the outcome of integration are, from a service user perspective, the perceived effects or needs satisfaction brought about by the services provided. In connection with welfare services, an important outcome is the accomplished level of financial and other forms of independence from the providers of service.

Together these dimensions can be used as a model to assess the integration of welfare services from the perspective of the service users. These dimensions may also supplement or be supplemented by other dimensions assessing service integration from the perspectives of the professionals, the managers or the organisations involved in intersectoral or other forms of collaboration.

Setting and methods

As mentioned before, DELTA is a local association for financial co-ordination between four different welfare institutions in the field of vocational rehabilitation. The association has a budget of around five million Euros, to which all the institutions involved are contributing. It has also a manager, appointed by the institutions, and a small administration.

The rehabilitation services provided by DELTA are divided into three main types of activities:

Preventive and promotional activities aiming to promote health and to prevent sickness absence and social exclusion.

Social-medical activities included in a treatment programme for early and co-ordinated rehabilitation.

Occupational activities aiming to get people back into work, or into a rehabilitation programme, as soon as possible.

These activities are carried out by multidisciplinary teams, consisting of professionals from the different sectors and institutions involved, for example physicians, nurses, physiotherapists, psychologists, economists, lawyers and social workers. The teams are supervised by co-ordinators appointed by the association.

Currently some 4000 inhabitants, have contacts with DELTA each year, which means about 5% of the working population in the area. On the average 67% of these contacts are shorter than six months. Furthermore, when the contacts are ended, 8 out of 10 formerly unemployed users of service are financially self-supported and 2 out of 3 are no longer sick-listed [23].

Guided by the theoretical framework and input from interviews with DELTA co-ordinators and service users, and also by discussions with the DELTA management, a draft questionnaire addressed to users was created for the assessment of service integration. This questionnaire was tested in a pilot survey in early autumn 2007. This pilot study was evaluated and adjustments were made together with the co-ordinators and the management before the questionnaire and the survey proceeding was finalised.

The final questionnaire contained altogether 32 structured and open questions, of which ten were about the personal background of the respondents, their sex, age, family circumstances, education, social situation with possible unemployment period included, how and why the contact with DELTA was established, former contacts with other DELTA activities, and the duration of the present contact with DELTA.

The structured questions that were used in the assessment of service integration from the perspective of the users are listed in Table 1. These questions were formulated as statements to be rated on different ordinal scales. In total, the questionnaire included 7 questions about the structural dimension, 11 questions about the process dimension, and 4 questions about the outcome dimension of integration.

Table 1

The structured questions of the questionnaire

Dimension	Structured questions
Structure of integration	• I have to repeat my problem history when I meet different professionals
	• When I meet different professionals, they are all informed about previous activities in my present contact with DELTA
	• The waiting time for starting different activities has sometimes been long
	• The capacity of the professionals involved to help me to improve my situation
	• The ability of the professionals to clarify my role in dealing with my situation
Process of integration	• I have in DELTA experienced a different handling of my situation
	• All professionals involved have the same view of the handling of my situation
	• The experienced waiting time for meetings with the professionals
	• The understanding of my situation shown by the professionals involved
	• The professionals have sufficient time to help me
	• The willingness of the professionals to help me improve my situation
	• The confidence in my ability to handle my situation shown by the professionals
Outcome of integration	• I am satisfied with the help I get from the DELTA professionals
	• I regard myself as being on the right way to improve my situation
	• I have trust in the present activities as a way to improve my situation
	• My ability to work has improved since my contact with DELTA

Some of the structured questions were qualified by supplementary open questions, for example: ‘If you answered no on question X, what is missing?’ Thus, although most of the questions in the questionnaire had fixed response options linked to an ordinal scale, there were also open questions that provided less structured data.

The main study was carried out as a retrospective cross-sectional survey [24] during two weeks in November 2007, including all the current service users at that time. As described in Appendix 1, the appointed contact persons in each team and the DELTA management had crucial roles in facilitating the study. The anonymity of the service users was secured in several ways by the study proceedings.

The data on the structured questions were processed and analysed with statistical methods, using the software included in the Access database and the statistical package SPSS, while content analysis [24] was used for the data on the open questions. The scales and the different dimensions were tested for reliability using Cronbach’s alpha, and their validity was reviewed statistically and estimated on basis of the outcomes of the pilot and the main study. The results were also discussed with the DELTA management and the team co-ordinators indicating the face validity of the assessment model. In addition, the results on integration outcomes were compared with official statistics from the different institutions involved.

The study fulfils the Swedish demands for good ethical standards in social science research [25]. The respondents were informed about the aim of the study and they gave their consent to participate. Furthermore, confidentiality was secured and the data were not used in non-research contexts.

Results

Survey results

The number of active users of rehabilitation services was estimated to be around 700 at the time of the survey. However, not all of these possible respondents had contacts with DELTA during the two weeks study period. Therefore, 552 individuals received the questionnaire and 454 of these individuals answered it. The response rate varied between the different DELTA activities: 72% in the preventive and promotional activities, 79% in the occupational activities, and 86% in the social-medical activities, which gave a total response rate of 82%. This response rate is calculated on the available number of respondents. It is, however, difficult to calculate the response rate as the size of the population for administrative and anonymity reasons can only be roughly estimated.

Sixty percent of the respondents were females and 40% were males. The mean age of the women was 41 (median=40) and of the men 39 (median=37). Sixty-one percent of the respondents were involved in social-medical activities. The remaining part of the respondents included two equal groups involved in preventive-promotional activities and occupational activities, respectively.

The structure of integration

Fifty-four percent of the respondents stated that they had met other professionals beside their main contact person. In this group of respondents, 45% declared that they ‘never’ or ‘seldom’ needed to repeat their problem history when they met different professionals, but 46% said that this was ‘often’ or ‘always’ the case. Sixty-one percent of the respondents with multiprofessional contacts stated that all the professionals involved were ‘always’ or ‘often’ informed about previous activities throughout the present period with DELTA. Only 20% of the respondents declared that this situation ‘seldom’ or ‘never’ occurred.

No noticeable problem with access to services was observed among the service users. Seventy-four percent of the respondents stated that the waiting time for a meeting with a professional ‘never’ or ‘seldom’ had been long. Similarly, 69% declared that the waiting time for the start of activities had ‘never’ or ‘seldom’ been long. In addition, all the respondents had a positive view of the capacity of the professionals involved to help the service users to improve their situation as well as their ability to clarify the role of the users.

The process of integration

The respondents gave high ranks (median=9 on a scale from 0 to 10) to the understanding shown by the professionals regarding their situation. However, the female respondents seemed in general to have a more favourable opinion about the understanding of the professionals. Similarly, the respondents had very positive views of the willingness of the professionals involved to help the service users to improve their situations (median=10) as well as the confidence shown by the professionals in the ability of the users to handle their own situation (median=9).

Fourty-six percent of the respondents perceived a different handling of their situation in DELTA compared with previous experiences. This ratio varied however between 40% and 52% for different lengths of the contact period with DELTA. In addition, the ratio of ‘I do not know’ was decreasing with increasing duration of the contact period. Instead, there was a tendency that the proportion of respondents not perceiving any difference was increasing.

In a supplementary open question, the respondents described that the main difference compared with previous experiences was that in DELTA they had been understood and well treated by the professionals involved, e.g. ‘I have been taken seriously’, ‘I get the attention I need’ and ‘they are listening to me’. Some respondents emphasised the skills of the professionals. A few comments were specifically about the experiences of collaboration, the importance of the team work, the advantages of being ‘included in a wider network’, the good communication between the institutions involved, etc.

On the question whether all the professionals involved had the same view of how their situation should be handled, 67% of the respondents stated that the professionals ‘always’ or ‘often’ shared the same view. No significant differences could be found between gender, age groups, team tasks, or other background variables. Relatively few respondents answered ‘seldom’ or ‘never’ on this question, but, when this occurred, physicians were the professional group regarded to occasionally have a different view. No differences could be found between the four financially co-ordinated institutions.

The outcome of integration

Eighty-two percent of the respondents were satisfied with the help they received from the professionals of their respective DELTA team. Ten percent answered “I do not know”, but most of them had only a short contact period with DELTA. Eight percent of the respondents were not satisfied with the help he or she had received.

Many respondents were to a high degree certain that they were on the right way to improve their situation. They gave high ranks to the contribution of the DELTA activities (median=8 on a scale 0–10). These respondents included a larger proportion of females. The ranking was not correlated to the length of the contact period, education or family circumstances. The question about the trust of the respondents in the activities improving their situation also gave a similar result (median=9).

In a supplementary open question, one out of three respondents commented on their choice of rank concerning the improvement of their situation. One group of comments was about experienced effects, e.g. “feeling energetic’, ‘coping more and more’, and ‘having developed self-confidence’. Another group of comments was about the hopes for the future, e.g. ‘developing in the right way’ and ‘confident in finding work’.

Reliability and validity

One of the first issues concerning the reliability of the assessment model was to have unambiguous survey questions. This is particularly important in an assessment by service users with different educational and social background. Therefore, different questions were tested on the management and professionals as well as service users of the DELTA association during preliminary meetings and interviews. The questionnaire was also tested for comprehensibility in a pilot study and adjusted before being employed in the main study.

The final questionnaire contained 16 structured questions linked to different ordinal scales operationalising the different dimensions of integration — structure (5), process (7) and outcome (4). The reliability of these measures was tested by computation of Cronbach´s alpha. The alpha for the scales measuring the structural dimension was 0.8 indicating a very high level of reliability. The scales measuring the process and outcome dimensions had Cronbach´s alpha 0.7, which can also be regarded as a high level of reliability [26]. These results are summarised in Table 2 together with the characteristics of the scales.

Table 2

Reliability test of scales

Dimension	Description	No. of scales	Cronbach alpha
Structure of integration	Available user information; user access to services; capacity and ability of the professionals	5	0.80
Process of integration	Experienced differences by users; shared view and mutual understanding among professionals; appropriate working routines; interpersonal conditions for improvement	7	0.73
Outcome of integration	User satisfaction; confidence in the future; improved user ability	4	0.67

No serious biases, in connection with variation of response rates or seasonal circumstances were detected in the results of the study. Biases that could be derived from dishonest answers due to interpersonal dependence between professionals and service user were eliminated through the study proceedings described in Appendix 1.

The validity of the assessment model can be reviewed in many different ways, but for the purposes of this study the main focus must be on construct validity [24]. The initial survey questions were based on a well-known theory of quality evaluation and relevant research on collaboration and service users, which gave them good construct validity. This validity was strengthened by successive refinements based on the initial interviews and the pilot study. Even so, there were still problems with the validity of the assessment model that were clearly seen in the results of the main study.

One problematic question in the assessment model was the one about repeating the problem history. Almost half of the respondents, who had contacts with two or more members of a DELTA team, said they often or always had to repeat their history when they met different professionals. If this was due to a lack of documentation and communication between the different professionals it could be an indication of an insufficient integration structure. However, it seems that service users repeating their history could also be interpreted as something positive. The professionals had time to and were interested in listening to the history of the service users, even if it was already known by them. This interpretation was partly confirmed by 44% of the respondents, who said that they had to repeat their history but also pointed out that the professionals were always informed about the previous activities throughout the present contact.

As the views of the service users concerning the DELTA collaboration were not known in advance, the construct validity cannot be tested for sensitivity or responsiveness. However, the information content of each question was reviewed by calculating the maximum response frequency, i.e. the response alternative that was most frequently chosen. In one case, the dichotomised question if the user was satisfied with the help received, one response option exceeded 80%, which implies that this question has limited information content due to one heavily dominating response alternative [27].

With the exception of the two above mentioned questions, the survey can be considered sensitive enough to discriminate between the different aspects and dimensions of intersectorial collaboration. Furthermore, the feedback from the DELTA management and co-ordinators on the results of the main study support these remarks on the reliability and validity of the assessment model and also gives good face validity to the model. The results on the outcome of integration were also supported by official statistics from the different institutions involved. After completing their contact with DELTA, 61% of the service users were financially self-supported [23].

Discussion

The results of the survey

The service user assessment of the service integration in the DELTA model of vocational rehabilitation was generally positive. For instance, the users gave high scores on the understanding of their situation shown by the professionals involved, their capacity as well as their willingness to help. Many respondents seemed to have a high confidence in the activities of the professionals they had contact with and they felt that they were on the way to an improvement of their situation. Thus, the DELTA rehabilitation services seem to have been well attuned to the needs of the service users.

Following the perceptions of the respondents, there seems to have been a high degree of consensus in the DELTA teams. All the professionals were perceived to have mostly the same view of the services needed. This state is often referred to as transdisciplinary teamwork [28, 29], which is a more elaborated form than when teams are interdisciplinary [29] or multidisciplinary [30]. The financial co-ordination with funds pooled in a joint budget, a supportive legislation and ten years of enriching experience are most likely explanations for this high degree of interprofessional integration [31].

As it has been shown, the number of users who perceive no different handling of their situation compared with previous experiences is increasing, while the number of users who do not know is decreasing as the contact period gets longer. This result may seem strange, but with a longer experience, the respondents would be more likely to have opinions and at the same time they might have more difficulties perceiving differences compared with previous experiences as their services become more of everyday actions.

Since the users of the occupational or preventive and promotional activities have lower response rates, it is possible that they may have biased the results. As it has been shown, some of the results also seem to have gender as a dependent variable. In addition, the occupational activities have more male users than an average DELTA team, while the preventive and promotional activities have a higher proportion of female users. These conditions may presumably have counterbalanced each other, and the variation in non-response rates therefore probably has had a limited impact on the results.

The evaluation was carried out as a cross-sectional study during two weeks, and the findings are therefore based on the views of all the available service users at that time, assuming them to be typical of the whole group. This assumption can be justified without seasonal variations. Seasonal unemployment can have an effect on the number of individuals involved in occupational activities. However, this group includes primarily individuals with longer periods of unemployment, who are mostly unaffected by seasonal variations.

The model of assessment

The study has shown that the different dimensions of integration provide a broad foundation for evaluation of intersectoral collaboration. A more limited or unbalanced view could easily be criticised for neglecting relevant aspects of integration. The theoretical interdependence between the different aspects is also reflected in the relation between the three user dimensions of integration. The structure of integration seems to be a condition for developing the interpersonal and interprofessional activities in the process of integration, and the synergetic effects of these activities seem to improve the outcome for the service users.

It is important to point out that a cross-sectional survey, like the one used in this study, cannot show the causality between the different aspects of integration. This would require deeper qualitative studies of service integration. On the other hand, a survey is relatively economical in the sense that data from a large number of respondents can be gathered quickly, which is important in the development of an assessment or evaluation tool [24].

The results indicate that most of the questions used in this study have been working well as measures of the different dimensions of integration. However, as it also has been shown, dichotomised variables should be avoided. Their constricted structures increase the risk of getting one dominating response alternative and for that reason limited information content. Furthermore, it is important that questions are understood in the same way by the respondents and the evaluators. The question about the users repeating their problem history illustrates this dilemma. These kinds of questions must be excluded when the model is further developed. There are also other possible developments, for example making a distinction between interpersonal and interprofessional processes of integration.

A further development of the model could be based on the key assessment themes derived from this study and presented in Table 3.

Table 3

Key assessment themes of the different integration dimensions

Dimension	Key assessment themes
Structure of integration	Accessibility of relevant information. Design of service provision
Interpersonal process of integration	Trust between professionals and users. Motivation of professionals and users
Interprofessional process of integration	Common holistic perspectives. Responsiveness among professionals
Outcome of integration	User satisfaction. Progress of improvement

In this study, the questions were answered by users of the DELTA rehabilitation services, but similar questions related to the key assessment themes could probably be used also for other service users, for instance patients or clients included in integrated care arrangements. Thus, this assessment model seems to have a potential to be used in most integrated care contexts, on condition that the respondents have the capacity to understand and to give written answers on the questions. Integrated care arrangements for individuals with serious somatic or psychiatric handicaps are therefore less suitable, unless adequate adjustments can be made.

Conclusions

In spite of an extensive variation of differences in the background conditions and needs, the study of the DELTA association for financial co-ordination has shown that the users of the rehabilitation services perceived that the services were well integrated and well adapted to their needs. In other words, these perceptions indicate that the DELTA teams are working transdisciplinary and in this way matching the multiple needs of the service users.

The main purpose of this study was to develop a model that can be used for the assessment of service integration by patients, clients and similar groups of service users and also to evaluate the results of these services. Guided by this comprehensive approach and a theoretical framework based on quality evaluation and research on collaboration and service users, three groups of dimensions for the assessment of integration were identified: the structure, the process and the outcome of integration.

An assessment model was tested for reliability and validity in the study of the DELTA association. As a result, some further refinements were suggested in the assessment dimensions and a refined model was indicated by a number of key assessment themes. Such a model may be used in continued assessments and evaluations of intersectoral or other forms of collaboration in the provision of integrated welfare services. It may also be combined with qualitative studies in order to understand the causalities involved in service integration.