Implementing and evaluating the first german young-carers project: intentions, pitfalls and the need for piloting complex interventions.

The aim of the study was to develop, implement and evaluate a concept for the first support program for young carers and their families in Germany. This paper intends to critically review the implementation of that study and describe the problems experienced by the research team, including: the complexity of the intervention itself, the difficulty of finding host organizations, the lack of infrastructure, different values and beliefs about the project aims held between the host organization and the research team, shortage of time, identifying and recruiting families among the hidden population of young carers. These initial problems led to the re-constructuring of the original research design. In order to evaluate factors that influenced these difficulties, the original research intentions, emerging problems and their consequences will be presented.

INTRODUCTION

It is well documented that the implementation of a complex intervention is most likely to face several problems [1-3]. When performing intermittent interventions in a newly discovered population (like young carers, who have not been a subject of research in Germany up to now) researchers must correctly identify and describe the problems and difficulties they face [4]. In our study on young carers we are delivering a support program which can be characterized as a socially complex intervention (see heading “intervention outline”). When delivering a novel intervention for a segment of social reality which up to now has been concealed and not discussed (i.e. young carers), it is imperative that an account of the process and the possible disturbances form part of the study.

BACKGROUND

Young people below the age of 18, whose lives are affected by looking after and caring for a relative with a long-term illness or disability, are called young carers [5, 6]. Researchers refer to the vulnerability of families concerned, and they predict that the child’s development will be affected if the families stay without support [5, 7-9]. Findings from a study in Germany [10, 11] identified that young carers suffer in the following ways:

having no one to talk to

living in secrecy

lack of free-time

social isolation and loss of childhood

problems in school and missing time in school

strong parental attachment

feelings of loneliness, sadness, fear and shame

physical and mental exhaustion

The aim of this current study was to develop a concept for the first young carers support service in Germany, to implement this project and to evaluate its effect [12]. The project is currently running in a large German city with the financial support of the German Federal Ministry of Education and Research (BMBF, Funding Reference Number 01GT0619). The project was due to start at the end of 2008, with an RCT (Fig. ) to gauge the project’s effectiveness planned to start in the second quarter and lasting until the end of 2009 (Trial Registration No: NCT00734942). Data analysis was scheduled for the first quarter of 2010. According to this, the study was funded until April 2010 (Fig. ).

The study faced several problems concerning the conduct and implementation of the project and recruitment for the study. This resulted in re-structuring the scientific design from an RCT to case analysis.

This paper explores these issues and the research team’s response to the emerging problems. First, a short overview of the intervention is presented. A more detailed outline of the study is described elsewhere [12].

INTERVENTION OUTLINE

According to the young carers literature [5, 6, 8, 10, 11, 13-16], nearly all areas of family life are affected. Thus a central concern of the concept is its family orientation. Different families share common needs, but, in their everyday life, individual families, as well as individual family members, have individual problems, needs and demands, and these require individual support.

To deal with these differences the support provided consists of several different modules (Table ), which are based on Metzing’s findings [10, 11], and which can be flexibly offered according to a family’s specific situation. In this way, the intervention may differ from person to person.

The intervention takes place in an independent youth center, where the participants meet for a few hours a week. Access to the intervention had no time limit and participating young carers were free to stay within the support service as long as they wished. The variety of intervention modules (Table ) required the input from a multi-professional team, including family health nurses (FHN) and social workers, with additional support from physicians, psychologists and volunteers. To capture the impact of these approaches it was planned to use an RCT with a waiting group design (Fig. ) using health-related quality of life (HRQOL) as an outcome criterion [17]. Group interviews were the proposed medium by which to gain the participants’ evaluation of the intervention (what they appreciated, what they missed and what should be modified).

INTENTIONS AND PROBLEMS

The study faced several problems from the very beginning. These will be described in chronological order (Fig. ).

2008 - Finding a Project Partner

It was planned to initiate the project in a city not far away from the research team (RT), which is located in one of the biggest metropolitan areas in Germany. As the BMBF’s grant was limited to financing the scientific staff (including development and evaluation of the concept as well as the scientific manpower), the RT sought a partner (PP) who was willing to implement and fund the practical part of this project. The RT presented its findings at several conferences and conducted a symposium where the concept was presented and discussed. The RT was to act as consultants to the practice partners who were to implement the intervention. The RT was also responsible for undertaking the evaluation. The Grounded Theory study [10, 11], on which this intervention had been founded, had a high profile in the media and attracted a lot of interest among local practice partners. Due to the upcoming economic recession, all potential PPs were unable to fund the project and finally withdrew their interest. It took the RT seven months to finally find a PP in another city about 400 km away. The official start of co-operation was December 2008.

The PP is an old, well-established, hierarchically structured organization, which is used to acting independently. The PP’s motivation to implement a young carers project was to realign their social engagement in their neighborhood, and they also found it attractive to be the first to take care of this vulnerable population in Germany. Until then, their main sphere of activity was traditional nursing care and nurse’s education.

The RT and PP agreed to work as a team with different responsibilities: the RT was responsible for scientific leadership and the PP for the practical implementation. Due to the distance of 400 km between the two partners, we decided to primarily communicate via email and telephone, with bi-monthly face-to-face meetings.

2009 - Problems Concerning Implementation and Recruitment

In 2009, the whole team was enthusiastic about starting the project. The PP decided to take six months for necessary preparatory work, like getting rooms which are suitable for children, recruiting staff, creating a logo, website and information flyers. Thus the official start was rescheduled - which led to a delay of six months compared to the original research schedule.

Cooperation Agreement

During this first phase of co-operation, the RT made several mistakes. There was no insistence on a written co-operation contract, listing all duties, responsibilities and rights. The RT gave hardly any ongoing instructions and guidance during the first phase of implementation, as the PP appeared to be in any case informed and competent to act independently. For the first six months, no formal minutes were taken during meetings, which led to misunderstandings concerning the state of implementation. As a result, the RT almost lost control of the implementation process.

The PP on the other hand concentrated predominantly on fitting the project into its Corporate Design (e.g. spent time on discussing a logo and organizing the web-page on a professional basis) rather than on the project’s content and startup.

Staff

There was no assining of jobs according to the concept’s job chart, which implied having at least a 1.5 whole time equivalent (wte) of FHN and social workers. At the beginning, the PP’s team consisted of the project’s leader with an 0.2 wte. Four months later (May 2009), a social worker with an 0.5 wte finally joined the team.

Public Relations

The recruitment process during the Grounded Theory study [10, 11] demonstrated the difficulty of reaching the hidden population of young carers and their families. However, families were waiting for the first support project to start and it was hoped that when they were made aware of the available support they would quickly enroll. After presenting the results of the Grounded Theory study, the topic “young carers” had a high profile in the German media, which was renewed whenever the RT updated their press releases. Therefore the RT intended to use the local media to launch the project and disseminate information about the center where young carers might find support. In addition, commercials and flyers were used to spread the information of the upcoming start within the quarter. Unfortunately, the local media totally ignored the press releases, thus the primary method of dissemination was not broadcasted. As a result, local awareness and knowledge of the project and its intention was very low.

Missing Infrastructure

It was planned to establish a network of doctors, nurses, youth projects and administrative offices who could refer young carers to the project. But the PP was unknown in the field of youth work and did not have an established network of referral agents. Thus, due to lack of contacts with important partners and to lack of experience with networking, some potential partners identified the project as competition or did not take it seriously. Therefore at the start of the project hardly any gatekeepers were available.

Ownership and Crisis

In June 2009, it became clear that each team understood the project to be more or less their project, with the other team being just an assistant. For instance, the RT was not involved in the decision-making processes on-site, whilst the RT expected the PP to follow the initial instructions rather than working out a joint implementation strategy. As a result, there was no jointly developed project work. Communicating via email and telephone was rather formal and this led to misunderstandings. The whole process resulted in frustration on both sides and subsequent conflicts. These were strengthened by the fact that the project was still not well known and therefore no children enrolled in the project by the time it started officially.

Finally, a crisis meeting took place at the end of July 2009 to resolve these controversial issues. After negotiating the issues, the two parties were able to work together more smoothly and it was hoped that the working morale would improve with recruitment of children for the project. Over the next few months, some mothers called the project leader. However this was not to get their kids into the project, but to get support for their family’s “everyday life”.

Sample-Size and Design

The funders (the BMBF) wanted strong measurable outcomes rather than action research with qualitative analysis only. It was therefore planned to conduct an RCT with N=150 children for the evaluation of the project’s effectiveness using HRQOL as an outcome criterion (17). The team assumed recruitment would not be problematic. However, after three months, no children had been enrolled in the project, thus the RCT design needed to be reconsidered. The RT took a pragmatic decision to switch to a Pre-Post design with N=30 children in order to meet the study’s funding deadline. At this time (October 2009), the original schedule was supposed to have recruited at least 50 children for the project. This failure to recruit meant the project was in danger of losing its funding.

Process Analysis

Three months after the project’s official start (October 2009), a FHN-in-training joined the team with an 0.5 wte. The PP was in contact with eight families, who according to the PP did not fit into the project.

In December 2009, the RT carried out interviews with the PP’s staff in order to reconstruct the implementation process, and to understand the project’s initial difficulties. Additionally, the RT got copies of all records about every family that had contacted the project so far. At the end of December 2009 the PP’s project leader resigned from the project for personal reasons. As no child was enrolled into the project by then, everyone’s morale was low.

2010 – Addressing Recruitment

The RT analyzed the interviews and case-files. The RT learned from the interviews that there was a lack of communication and guidance between the leader and two team members and that this had adversely affected recruitment for the study. The latter had poor knowledge about the concept which resulted in uncertainty concerning work assignments and unfocused project work. Case file analysis confirmed that two families, whose children fitted the criterion for inclusion into the project, had been sent to other social projects by the PP.

Missing Leadership and Crisis

In January 2010, a social worker with an 0.5 wte joined the PP’s team. After one year, the concept’s job chart was finally fulfilled. The team consisted of three members, each with an 0.5 wte position. But the ‘project memory’ (concept, research project) was in danger of being lost.

Without a project leader, the PP’s team began to rearrange the concept’s content according to their experiences hitherto within the project and relating to the feedback of some network partners. The content of this realignment was in any case not compatible with the RT’s understanding of a “young carers’ project”, as it put the focus mainly on lobbying rather than on getting into contact with young carers.

In February, the team was again divided into two opinions:

The PP felt convinced that the concept was not deliverable in their city, and that - because of problems with accessing young carers - the team should thus focus on the PP’s realignment suggestion. The RT argued to hold on to the original concept, as this was a) mainly based on the Grounded Theory study’s results, b) funded by the BMBF, and c) showed, as findings of interviews and case-file analysis revealed, potential room for improvement concerning the implementation strategy on-site.

The whole team’s mood hit rock bottom and the PP’s as well as the RT’s decision-makers thought about canceling the whole project. Finally, a second crisis meeting was held in February 2010 to discuss the controversial issues again. Both partners decided to give the project a last chance. While the social worker with the most seniority left the PP’s team for personal reasons one week after the crisis meeting, the other social worker - having started one month before - was nominated to be the PP’s new project leader. At this time, the PP’s team consisted of a social worker, who was now the new leader, and a FHN-in-training, each with an 0.5 wte.

Relaunch and Forecast

The situation changed completely after the crisis meeting and after the reconstruction of the PP’s team. The new leader adjusted the PP’s networking, with the result that the network was growing and functioning as intended. During the crisis meeting, a mutual consensus about the implementation strategies was reached. As a result, the whole team now acted in concert, with a good working atmosphere and amicable communication, and the implementation process was running - with a total delay of seven months - as intended. In May 2010, a new social worker with an 0.5 wte completed the PP’s team (Fig. ). Since August 2010, there are regular meetings between the RT and the PP about every third week.

Since with the new team constellation the project seemed to be on track again, the RT endeavored to get further funding. Finally, the BMBF awarded a grant extension until end of 2010.

After the PP’s adjustment of networking, some gatekeepers brought young carers in touch with the project. Summing up, it took seven months to get the first child into the project (March 2010).

Currently (November 2010, sixteen months after the official start), eight children have been enrolled.

The RT finally changed the research design into Case Analysis, and is optimistic about doing interviews with about 10 enrolled children by the end of 2010, focusing on the children’s experiences with the project.

DISCUSSION

In retrospective, it is astonishing that the RT wondered about the project’s initial difficulties at all. The problems during the implementation process can be assigned to the following four main shortcomings:

Accepting RCT design

No insistence on guidance

No pilot on feasibility

No pilot on recruitment

Accepting RCT Design

RCTs are still widely accepted as the gold standard for evaluating health-care interventions [18] and decision-makers are increasingly looking to the results of RCT studies as a practice guide. The project’s intervention is - as most nursing interventions are - a complex intervention (1; 19), and there is an ongoing discussion whether RCTs are preferable and appropriate for their evaluation at all [1, 3, 19]. Nevertheless, several guidelines have been published on how to develop, implement and evaluate complex nursing interventions [20, 21], and all of them state that the study design should be chosen on the basis of the study’s specific characteristics, RCTs included.

When the study was conceived, the RT based the whole process on the Utrecht Model [21, 22], which also states that an RCT should be conducted if necessary and ethical justifiable. However, the model also states that a field phase should be conducted prior to the RCT, which is predominantly linked to action research. This phase aims on piloting the intervention until no further modification is indicated.

When the study’s application form was sent to the BMBF, the RT proposed a qualitative approach, which combined both the field and evaluation phase. The BMBF advised to state precisely a specific outcome criterion which the intervention should focus on. In addition, the RT was advised to name an adequate design in order to evaluate this specific outcome criterion rather than conducting action research. In order to get funding for the study, the RT changed the design to an RCT despite better knowledge. The RT also skipped the field phase during the re-conception of the study’s schedule because of three reasons:

The generally granted running time was limited to three years, leaving the RT no time for an adequate field phase

The RT was confident that the concept was cogent enough to be tested directly. Minor adjustments were planned to be carried out after the RCT phase.

The RT feared the rejection of the proposal if it contained a good deal of action research.

No Insistence on Guidance

At the beginning, the RT had huge problems finding a PP. At the time when the current PP decided to co-operate, there was already a delay of three months within the granted research schedule, and the RT was to a certain degree reliant on the PP’s participation. For this reason, the RT let the PP act independently concerning decision-making on-site rather then insisting on adherence to the study’s strict schedule. For instance, the PP’s decision to set the official start at July 2009 led to a further delay in the RT’s study schedule.

The RT did not control the PP’s understanding and interpretation of the concept, as the PP appeared to be well-informed and able to act independently. Therefore the PP was more or less free in its decision-making concerning the preparation and management of the implementation process, and the RT failed to:

give overall guidance,

be involved in the decision-making process on-site,

have a say concerning application procedures of new team members,

instruct new team members, and to

have the final say.

Due to this delegation of responsibility the RT was slow to realize that the project implementation was not progressing in the intended direction. The spatial distance between the RT and the PP exacerbated this loss of control. The problems concerning the relationship between the PP and the RT were of an interpersonal nature, and thus they were not predictable. Nevertheless, these difficulties could have been identified in the early stage if the whole process had been subject to analysis and reflection while it was being carried out. For example, in qualitative research, it is good practice to have an external person, who is not involved in the project and who helps with reflection and analysis of the whole process during the study’s entire running time. Having such external supervision can prevent the genesis of interpersonal conflicts within empirical social research.

No Pilot on Feasibility

The RT did no feasibility piloting. As the project needed to be set up from scratch with a completely new team, there was no need to test whether the intervention would fit into existing work proceedings. What at least should have been piloted was how the project could be installed as a part of the local social and health-care services network. Due to inexperienced networking in 2009, the project had low awareness within this network, and some projects even identified it either as a rival or as superfluous. It was a fortunate coincidence that the PP’s new project leader not only had experience in networking but also had personal contacts to important established projects as well as to administrative offices.

No Pilot on Recruitment

The main aim of the project is of course the support of young carers and their families, but on the other hand, the RT was dependent on a high number of participants in order to carry out a RCT to evaluate the intervention’s effectiveness.

Therefore, the RT’s first priority was to recruit children for the project in order to reach the calculated sample size. It is known from the literature that recruitment often takes more time or is more difficult than expected, with many trials failing to reach their sample size within the granted timescale [23]. Although the RT had experience with the difficulties of reaching the very hidden population of young carers, they totally underestimated the obstacles of enlisting suitable children for the project. This was due to the results of the Grounded Theory study, which stated that the families concerned were waiting for the first support project to start. On the basis of these findings, RT predicted that children would readily enroll in the project.

One characteristic of the population under study is its concealment. During the Grounded Theory study’s interviews, the informants were in a protected and self-imposed conversation, and they became more and more familiar with the interviewer. It could be due to this confidence that nearly all of them stated they would enroll in a young carers’ project if it were available.

For a new project, it is far more difficult to earn confidence from a cautious population.

Therefore, as an overall strategy, support needs to be organized through an outreach program with confidence-building as the first priority in order to be able to reach those families in need. It is arguable whether the project’s recruitment strategies were in concert with an outreach strategy.

The label “out reach program” implicates that the project’s services need to be offered directly to the families concerned. Therefore, the project must try to reach them at places where they spent time socially.

With regard to the children, convenient places could be schools or youth centers. The project staff could have consultation hours at fixed dates, maintaining a presence and spreading information. This ongoing presence could increase confidence, and it might enhance the chance that a young carer calls on the project’s service.

With regard to the parents, convenient places could be hospitals or doctor’s practices. If a parent with a chronic illness is treated there, general information about the project could also be supplied. After giving the parents some time to reflect, an initial contact with the project could be offered. However, using the network as gatekeepers requires network partners who remember the project and who introduce its services to the clients at the right time.

Nevertheless, in pursuing these strategies, there is still a need for piloting recruitment in order to get an overview on how many families can be put in touch with the project in the required time-frame. This would answer the question of whether conducting an RCT with a given sample size in a given time is realistically feasible at all.

This emphasizes the necessity of conducting a field phase before running an RCT that already has a given sample size and time-frame, especially within a hidden population. It could have saved time and money if the RT would had followed all phases of the Utrecht Model, which puts emphasis on a field phase in which the intervention is tested and modified before running a quantitative evaluation study.

CONCLUSION

This paper presents a process evaluation that has focused on the problems that arose whilst implementing a complex social nursing intervention. It provides insight into the social reality of young carers and on the difficulties of doing research with this particular group. The experiences gained from the study lead to the following conclusions:

Health-Care Provider

It is difficult to persuade health-care providers to recognize new problems and it takes an even longer time until they realize they have a responsibility to act. The RT totally underestimated the persuasive powers which are necessary to find a suitable PP. Additionally, when an interested provider is found, one should at least keep in mind that the PPs are likely to have specific reasons, even a hidden agenda, for implementing new interventions in new areas, and these might be very different to the RT’s intentions.

Young Carers as a Hidden Population

The population of young carers is hidden, unknown and probably underestimated [8, 10, 24, 25]. Reaching this hidden group in one study (Grounded Theory study) does not necessarily mean that this will succeed again in another study or intervention (current study). The implication of this is that any project or research targeting such a group must add in extra time in their calculations for contacting and recruiting members of this population.

Guidelines

The RT chose the Utrecht Model as a guideline for the implementation and evaluation process. As described above, the RT decided to skip the model’s field phase despite better knowledge. However, a major part of the above described implementation problems are topics related to this specific phase. One could say that although the phase was skipped, it claims its time and space anyhow. The model’s authors apparently have good reasons for including such a phase in their guideline. If researchers choose such a model, they should adhere to it completely.

Research Funding

When the study was initially planned, it was intended to use a qualitative approach. Following the BMBF’s feedback, these plans had to be changed. Adhering to the paradigm of evidence-based health-care, the reviewers argued for the application of a “gold-standard” approach. Although this recommendation is in concert with the current economic zeitgeist concerning efficacy, it lacked the reflection of the study’s original interests.

Blumer (1998) states that “such conceptions are a travesty on methodology as the logical study of the principles underlying the conduct of scientific inquiry” [26, p. 24].

We would even add that following the evidence-based paradigm in an unreflective way is the end of any serious methodological legitimization.

However, due to the failure of recruiting the calculated sample size, the funded RCT study had to revert to a qualitative design in accordance with the BMBF.

Table 1.

Intervention Modules

A) Having Someone to Talk to	B) Support for Children	C) Support for Parents
Befriending	Leisure-time activities and “Young carers group”	Administrative support and counseling
Professional counseling	Information and education about illness and care	Flexible help with everyday-life activities
Family Conferences	First-aid course	Parents’ café
Self-help group	Schoolwork assistance