Acceptability of a German multicentre healthcare research study: a survey of research personnels' attitudes, experiences and work load.

OBJECTIVES: The DACAPO study as a multicentre nationwide observational healthcare research study investigates the influence of quality of care on the quality of life in patients with acute respiratory distress syndrome. The aim of this study was to investigate the acceptability to the participating research personnels by assessing attitudes, experiences and workload associated with the conduct of the DACAPO study. DESIGN, SETTING AND PARTICIPANTS: A prospective anonymous online survey was sent via email account to 169 participants in 65 study centres. The questionnaire included six different domains: (1) training for performing the study; (2) obtaining informed consent; (3) data collection; (4) data entry using the online documentation system; (5) opinion towards the study and (6) personal data. Descriptive data analysis was carried out.
RESULTS: A total of 78 participants took part (46%) in the survey, 75 questionnaires (44%) could be evaluated. 51% were senior medical specialists. 95% considered the time frame of the training as appropriate and the presentation was rated by 93% as good or very good. Time effort for obtaining consent, data collection and entry was considered by 41% as a burden. Support from the coordinating study centre was rated as good or very good by more than 90% of respondents. While the DACAPO study was seen as scientifically relevant by 81%, only 45% considered the study results valuable for improving patient care significantly.
CONCLUSION: Collecting feedback on the acceptability of a large multicentre healthcare research study provided important insights. Recruitment and data acquisition was mainly performed by physicians and often regarded as additional time burden in clinical practice. Reducing the amount of data collection and simplifying data entry could facilitate the conduct of healthcare research studies and could improve motivation of researchers in intensive care medicine. TRIAL REGISTRATION NUMBER: NCT02637011; Pre-results. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Participating researchers were from a variety of heterogeneous hospitals with differing workload and research experience and expertise.

Based on this online survey, problems with the data collection were disclosed and these data led to an open discussion with the participating study centres.

The results could be helpful for the design and set-up of future healthcare studies.

The results of the online survey only reflect the research personnels’ opinions towards a single-specific healthcare research study.

Researchers who are particularly motivated for supporting the study may be also motivated to complete the survey.

Introduction

The management of critical illness at the intensive care unit (ICU) is a resource-consuming process that requires a specially qualified team, a high technology standard and sophisticated treatment strategies to provide life-sustaining therapy. Healthcare research in the field of critical care medicine is relatively new, and data on the influence of organisational structures or processes of care on mortality or health-related quality of life (HRQoL) in ICU survivors are of growing interest.1 2 For the assessment of long-term outcomes in patients surviving critical illness, prospective multicentre observational study designs are required. One example is the DACAPO study3 which investigated the influence of quality of care and individual patient characteristics on HRQoL and return to work in survivors of the acute respiratory distress syndrome (ARDS). The conduct of multicentre observational studies is associated with a high workload of the researchers regarding recruitment, obtaining informed consent and capturing baseline and long-term follow-up data.4 5 One challenge in this respect is that patients need to be recruited in life-threatening situations which means that often they are not able to provide consent themselves. Researchers conducting studies in intensive care are often clinicians who recruit, obtain informed consent (predominantly with the families or next of kin for the unconscious patient) in routine clinical practice, and transmit the data to the leading study centre. Difficulties with recruitment of study patients are common both in randomised and observational studies,6–9 sometimes resulting in a selection bias and potentially the reduction of statistical power.10 For instance, it is conceivable that patients with more severe disease have a lower chance of being included. It must be noted that in the intensive care setting only around 10% of patients admitted to the ICU possess decisional capacity with a large gap for providing consent for research participation between patients and their substitute decision-makers.11 12 Strategies for improvement of recruitment activities have recently been investigated and discussed.5 Characteristics of study centres, research infrastructure like full-time research staff and study experience are important factors associated with successfully procuring informed consent to participate in trial.13 14 Furthermore, study set-up, the volume of data collection and training regarding data collection and study procedures have a profound influence on the study success.15 Case report forms (CRFs) are used for data collection in clinical research and the extent of required documentation in these CRFs affects study success.16 The use of electronic CRFs (eCRFs) is becoming more popular due to the advantages they offer such as improved data quality, online discrepancy management and faster database lock.17 Electronic data capture systems (EDCSs), which enable the collection of clinical data, typically use eCRFs.18 Acceptability of the EDCS is enhanced if the study participants find it convenient and easy to use.19 The training how to deal with eCRFs is a very important aspect; all participants should understand how to use the EDCS.20

The aim of this study was to investigate the acceptability of a multicentre healthcare research study (DACAPO study) in Germany by assessing research personnels’ attitudes, experiences and workload.

Methods

An account for the prospective anonymous online survey was sent by mail to all participants (physicians, study nurses), who recruited patients for the DACAPO study. The online survey was performed by the coordinating study centre (CSC) in Regensburg in the context of a large Germany-wide prospective cohort study.

All 65 participating study centres received initiation visits which included an extended presentation of the study’s aims and design, illustration of the data to be recorded and training how to use eCRFs with the freely available data management system chosen for the DACAPO study (OpenClinica, Waltham, Massachusetts, USA). In this context, all participants received instruction materials (two folders) for data collection and organisation. The first folder contained detailed explanations on the following aspects: general information about the study, overview and workflow of data acquisition, detailed user guide for OpenClinica, all eCRFs in paper size, explanation and materials for delirium monitoring, all contact options with the CSC (mail, telephone, 24 hours hotline). The second folder contained all materials for the extensive collection of patient data. This included the master data sheets for patients under study, documentation aids for data acquisition (referring clinic, interhospital transportation, internal clinical data, socioeconomic status, scoring of delirium, discharge from ICU), study information and consent form for patient and next of kin/caregivers/legal guardians, material for postal transport of consent form and 12-item Short Form Health Survey.21 Written informed consent for participation in the DACAPO study was obtained primarily from next of kin/caregivers/legal guardians of patients and additionally later from those patients who survived ICU and regained their decisional capacity. Requirement for data entry in OpenClinica is a Linux or Windows operating system and internet access. All participating study sites received the website link plus an individual password-protected access from the CSC after the initiation visit.

Due to the lack of established measurement instruments which capture the domains relevant to our study and the specifics of the DACAPO study procedures, it was decided to develop a novel questionnaire (online supplementary file: Questionnaire Online Survey Dacapo Study.pdf (V.1.0)). The 42 items of the online survey were designed on the basis of face validity. During this process, the research team which comprised psychologists, physicians and health services researchers subsequently modified the questionnaire based on repeated group discussions and the questionnaires were pilot tested on a group of clinical researchers. Domains of interest were usefulness of training of the study procedures, ease of obtaining informed consent, burden of and satisfaction with data collection, usability of the EDCS and attitudes of the involved personnel towards the study.

The survey was divided into six parts as listed below. Items regarding specific tasks within the study were preceded by a question assessing whether the respondent was actually involved in this task (for parts 1–4). If so, the respondents were asked about their attitudes, acceptability, feasibility, workload and possible improvements for the respective section. Otherwise the respondent was directed to the next part. Items were answered using a dichotomous response format or five-point Likert scale. After successful piloting by the Regensburg DACAPO team, the link for the voluntary online survey was sent out to all research staff involved in the DACAPO study (physicians, study nurses, documentarists, medical students). The survey was composed of six parts with the following items:

Training: participation at the training session (and if applicable reason for non-participation), comprehensibility of certain aspects of the training (three items), comprehensibility of instructional material (two items), satisfaction with time range (one item).

Informed consent: participation at obtaining, questions about and problems in clinical practice in context with next of kin and patient informed consent, support by CSC (two items).

Data collection: participation at (external/interhospital/internal) data collection (six items), handling of instructional material (two items), satisfaction with (external/interhospital/internal) data collection (five items), support by CSC (two items), time effort (one item).

EDCS: participation at and time range for data entry, comprehensibility of and satisfaction with EDCS (six items), support by CSC (two items).

Opinion towards the DACAPO study: scientific relevance and improvement of patient healthcare.

Personal data: professional qualification of the survey participants, experience with clinical studies and preferred contact with the study centre.

The initial training sessions were conducted between October 2014 and March 2015. At the time of the survey, all researchers had recruited participants for the study for at least 8 months. All researchers who were trained for DACAPO study got an invitation for participating in the survey and had exactly 8 weeks (from 7 December 2015 to 31 January 2016) to complete the questionnaire. After the initial invitation to participation, we sent out an email reminder at the beginning of the 6th and 8th week of the survey period. Reminder mails are an effective instrument to increase participation rate in online surveys.22 For data acquisition, the online platform SurveyMonkey was used. Descriptive statistics were calculated using IBM SPSS Statistics V.21.

Patient and public involvement

No patients and public persons were involved in this online survey.

Results

Seventy-eight out of 169 researchers (46.2%) responded to the survey. Finally, 75 questionnaires (44.4%) could be analysed, three responses were incomplete. The online survey response rate was 40% (54 of 135) for physicians and 58.1% (18 of 31) study nurses. The distribution of weekly participation in the survey is shown in figure 1.

Figure 1

Distribution of weekly survey participation (n=78).

Training

Eighty-five per cent of researchers (n=64) participated actively in the initial training session delivered by the clinical DACAPO team. Most of them who did not participate stated that they had no time or had not yet been involved in the study at this time. About 95% assessed the duration of the training session as ‘adequate’. The assessment of the intelligibility of various aspects of the training session is presented in table 1. About 90% of the respondents considered the intelligibility of the presentation of the study and the training material as ‘excellent’ or ‘good’. However, the intelligibility of specific aspects of the study (consent procedure, parameters and scores to assess, EDCS) was rated as only ‘partly good’ by up to 21% of participants.

Table 1

Assessment of the intelligibility of the instructions for the study (percentage of votes)

Item	Excellent (%)	Good (%)	Partly good (%)	Poor (%)	Very poor (%)	Total (n)
Presentation of the study by the leading investigator team	42.2	51.5	4.7	1.6	0	64
Written training material	28.1	61.0	10.9	0	0	64
Presentation of the consent procedure	18.8	56.3	18.7	6.2	0	64
Description of the parameters and scores	17.2	65.6	15.6	1.6	0	64
Instruction for the electronic data capturing system (OpenClinica)	11.3	59.7	21	4.8	3.2	62

Informed consent

A total of 72% (n=54) and 73% (n=55) of survey participants were involved with obtaining informed consent from next of kin/legal guardians or patients, respectively. A total of 78% and 89% of respondents indicated that next of kin and patients only rarely had questions, respectively. Forgetting to obtain informed consent was reported by 15% and 16% of researchers, respectively. Nearly half of the respondents (48%) had contacted the CSC for questions regarding the consent procedure at least once and the majority expressed a (very) high level of satisfaction (92%) with the support provided.

Data collection

Eighty-one per cent of respondents (n=61) actively collected data for the DACAPO study, thereof 77% (n=47) also documented data gathered by referring hospitals. Two-thirds agreed that the documentation aids facilitated data collection. One-third considered the data collection in general as at least sometimes distressing. More respondents considered data collection from external data sources (referring clinics, interhospital transport) as burdensome as compared with the collection of internal data. Two-thirds of the respondents stated that the assessment of the sociodemographic variables and the confusion assessment method (ICU) are feasible in clinical routine (table 2). More than half (57%) contacted the CSC at least once with questions regarding data collection. Of these, 94% judged the support as ‘excellent’ or ‘good’. The respondents contacted the CSC almost exclusively by phone or mail.

Table 2

Intelligibility of and satisfaction with the data collection

Item	Disagree (%)	Disagree rather (%)	Agree partly (%)	Agree (%)	Agree completely (%)	Total (n)
Patient folder is easy to use	0	4.9	16.4	50.8	27.9	61
Documentation aids facilitate data collection	8.2	3.3	19.7	50.8	16.4	61
The effort to collect data in my hospital is huge	1.7	34.5	36.2	10.3	17.3	58
The effort to collect data from external sources (referring hospital and interhospital transportation) is huge	0	4.2	27.7	27.7	40.4	47
Collection of sociodemographic data can be well integrated in clinical routine	3.6	14.6	18.2	49.1	14.5	55
CAM-ICU can be well integrated in clinical routine	5.9	9.8	19.6	47.1	17.6	51
Time effort for data collection is distressing	9.9	22.9	32.8	13.1	21.3	61

CAM-ICU, confusion assessment method for intensive care unit.

Electronic data capture system

Seventy-two per cent of respondents (n=54) reported having entered data in OpenClinica for the DACAPO study. The items of the DACAPO eCRFs are described in table 3. The items pertain to six sections: inclusion and exclusion criteria, time of admission on ICU and diagnosis of ARDS, informed consent of next of kin/caregivers, sociodemographic status, discharge of ICU/death of study patient and conclusion with informed consent of surviving study patients. Data regarding satisfaction with the web-based data capturing system are shown in detail in table 4. One-third of the survey participants had problems with the navigation and application of the eCRFs. A total of 46% and 24% of respondents reported that they needed 1–2 and 2–3 hours per study patient, respectively, for entering data into OpenClinica. The use of the EDCS was found ‘burdensome’ by 41% (n=22) of the respondents.

Table 3

Six sections of DACAPO-eCRFs with number of items for each study patient

Inclusion and exclusion criteria	8 items
Time of diagnosis of ARDS/admission on ICU	76–95 items (ARDS diagnosed in own hospital) 184–221 items (ARDS diagnosed in external hospital)
Informed consent of next of kin/caregivers	2 items
Sociodemographic status	10–11 items
Discharge of ICU/death of study patient	28–49 items (study patient alive) 12–16 items (study patient passed away)
Conclusion with informed consent of study patient	4 items

ARDS, acute respiratory distress syndrome; eCRFs, electronic case report forms; ICU, intensive care unit.

Table 4

Intelligibility of and satisfaction with the web-based data recording system

Item	Disagree (%)	Disagree rather (%)	Agree partly (%)	Agree (%)	Agree completely (%)	Total (n)
I had no problems with the navigation from the beginning	16.7	14.8	37.0	31.5	0	54
I had no problems to generate a new patient from the beginning	5.6	25.9	16.7	42.6	9.2	54
The construction of the electronic case report forms (eCRFs) is reasoned	7.4	22.2	27.8	38.9	3.7	54
I had no problems with the English version of the eCRFs	5.8	13.5	19.2	48.0	13.5	52
Electronic queries can be well edited	7.4	27.8	29.6	33.3	1.9	54
The workload of the electronic documentation is burdensome	11.3	18.9	28.3	24.5	17	53

Opinion towards the DACAPO study

The majority of respondents believed that the study had high scientific relevance (>80%, table 5), but only about 45% presumed that it will help to improve healthcare.

Table 5

Opinion towards the DACAPO study

Item	Disagree (%)	Disagree rather (%)	Agree partly (%)	Agree (%)	Agree completely (%)	Total (n)
The scientific relevance of the study is high	1.3	2.7	15.1	56.2	24.7	73
Healthcare of patients will improve through the study	2.8	17.8	34.2	34.2	11	73

Personal data

Eighty-seven per cent of respondents (n=65) stated that they were actively involved in several clinical studies. The professional qualification of the attending researchers is listed in table 6. Over 70% of the participants were physicians; thereof about 50% were senior medical specialists (anaesthesiology, internal medicine, surgery) and only 24% were study nurses.

Table 6

Professional qualification of the survey participants (n=75)

Profession	n, %
Senior medical specialist	38 (50.7)
Study nurse	18 (24)
Medical specialist	14 (18.7)
Resident physician	2 (2.7)
Medical documentarist	1 (1.3)
Qualified nurse	1 (1.3)
Medical student	1 (1.3)

Discussion

In the context of a multicentre nationwide healthcare research project on patients surviving ARDS, we sought to assess the attitudes of the involved researchers regarding acceptability, feasibility, experiences and workload. The main results are: (1) The presentation of the study during the training, the instructions for implementation and the quality of study material and support were assessed as good/excellent by 90% of the attendants; (2) The respondents were confident that the procedure for obtaining informed consent could be well implemented with few questions from next of kin and patients, but the half of them required support from the CSC; (3) Data collection was considered by one-third as (very) distressing, two-thirds agreed that documentation aids facilitated data collection; (4) The use of the web-based EDCS was perceived burdensome by one-third of the survey participants and (5) although the majority believed that the study had a high scientific relevance, only 50% presumed that it had a positive impact on healthcare.

In this online survey, the response rate was about 45%. This is comparable to response rates observed in other surveys addressed to health professionals. Cobanoglu et al conducted a fax, mail or web-based survey with 100 professors in each kind of survey on the subject ‘Education in Hospitals’ and obtained similar results. The highest participation rate was 44.2% in the web-based survey, compared with 26.3% in the mail or 17.0% in the facsimile-based survey.23 Another online survey of intensive care physicians from the UK focusing on perceptions of diagnosis and management of patients with ARDS achieved a lower response rate of 11%.24

Perceived quality of study presentation during the training, the instructions for implementation and the quality of study material and support was high. The high level of satisfaction with the initial training session can be attributed to different reasons. First, the briefing took place in small groups, was conducted interactively and provided sufficient opportunities for feedback. At the beginning, study objectives were explained in a brief presentation. The training has been flexibly adapted to the temporal and local needs of clinicians. Data entry was practically trained using case scenarios, the materials were distributed and discussed. Furthermore, aids to solve problems (contact options to the CSC for extra support, written training material, documentation aids) with the data collection and recording were presented. The training of recording data with eCRFs/EDCS is an important factor for the success of a study.20 Start-up meetings, personalised education and training visits assist in improving recruitment.25

In our study, the procedure applied to obtain informed consent resulted in no or only few questions from next of kin and patients, according to the participating respondents. This is remarkable as the procedure is quite complex (including much information and a variety of documents) and the next of kin/caregivers are in a distressing situation when witnessing the life-threatening disease of a family member. However, we do not know whether the lack of questions from next of kin was caused by the provision of sufficient and comprehensible information or by missing opportunities for asking questions. It must be noted, however, that the CSC provided support regarding the complexities of obtaining informed consent to many respondents. In a trinational study,26 respondents argued for modifications to consent approaches such as deferred consent, waived consent or consent from two physicians in the absence of a substitute decision-maker. On the other hand, the necessity of a careful and correct informed consent process was underlined by the respondents and beliefs about the feasibility and ethics of ‘simplified’ consent processes varied. The support of the CSC for questions regarding the informed consent procedure was used from almost 50% of respondents with high satisfaction. It became evident that such support is quintessential for study processes and success. One option to improve the process of obtaining informing consent might be to explain precisely the standardised informed consent procedure with the researchers during the initiation visit, and thus implement obtaining informed consent within routine work procedures in the participating hospitals.27 This applies particularly for situations where informed consent cannot be obtained directly from the patient (unconscious patients).28

Data collection for the DACAPO study was extensive and complex. Seventy per cent of respondents stated that they needed between 1 and 3 hours for data collection and documentation in eCRFs per study patient. The eCRFs are divided into six sections with standardised items (table 3). For patients in whom the ARDS was diagnosed in one of the study sites, 112 up to 169 items had to be completed in the eCRFs. However, when the diagnosis of ARDS was made in a referring hospital, the participating hospital had to complete between 220 and 295 items per study patient. Like in a multicentre study in Sweden, the main advantages of an internet-based data acquisition technique were automated data validation and standardisation, fast data transfer independent of geographical distance, user feedback, synchronisation of protocol updates and automatic data formatting facilitating statistical analyses.29 In another study, it was shown for web-based eCRFs that the integration of data from different domains is a difficult undertaking, and requires a proper infrastructure to acquire and disseminate data from multiple modalities.30 The advantages over paper-based data collection are explained below. For future studies, data collection should potentially be even more simplified by agreeing on the most necessary laboratory and clinical data in relation to the study’s objectives (core data sets). Conducting a feasibility study before the main study would have been beneficial for data acquisition,31 because problems with obtaining informed consent and data collection could be anticipated and prevented.

The use of EDCS for multicentre studies is widespread. In a Canadian survey between 2006 and 2007, an EDCS was used in 41% of 259 clinical trials.32 For the DACAPO study, the EDCS OpenClinica was used. This open source software offers comprehensive functionality for collection, management and storage of subject data in large multicentre clinical trials.33 34 OpenClinica is free of charge and can adapt to the personal focus, allows adjustments during ongoing study.35 Compared with paper-based data collection, such EDCSs reduce data error rates and costs.33 36 A qualitative evaluation of EDCS found that the ease of use and training materials were most important.34 In addition, the users of OpenClinica can buffer the recording of patient data and continue anytime as they wish. It must be noted, however, that despite all the positive aspects mentioned, up to one in three of participants in our survey reported problems with using OpenClinica and found its use burdensome. One reason accounting for the mixed evaluation of our EDCS could be that the navigation might have been perceived as not intuitive. In this context, EDCSs that allow better tailoring to a specific study might be an alternative.

Above and beyond the barriers encountered when recording data in EDCSs, the motivation of the individual researchers and clinicians involved in a study is of paramount importance. Orientation towards research, personal experience with science, participation of an academic research group and views about the special research topic were addressed as factors which can affect motivation, but no clear associations between one of these factors and the recruitment activity was identified.37 In other interview studies of trialists,38 39 it was found that payment of clinicians for research work was perceived to be less important compared with being convinced of the importance of research, the support by the principal research centre regarding salient questions and the reduction of study-associated bureaucracy. In our study, we found that the majority was convinced of the scientific importance of the DACAPO study. Of note, only 50% presumed that it would also have a positive impact on healthcare. Such mixed attitudes towards the influence of research on daily practice are commonly encountered. One explanatory approach is the different attitude of physicians and nurses towards research aspects and therapeutic measures in ICUs.40 The response rate in this survey was higher among study nurses than among physicians. Perhaps the method of a ‘realist synthesis’41 of evidence-based medicine may help to transfer ‘fatalistic’ or negative attitudes into an impact of high-quality research on daily routine and the use of guidelines for instance. In this context, further research should examine the impact of the practical aspects of carrying out a study on the motivation of researchers.

The increasing workload in intensive care medicine negatively influences the willingness to participate in clinical studies.5 42 In the systematic preparation of the DACAPO study, it was difficult to motivate clinical staff in different kinds of hospitals to take part in an observational study with a high documentation effort and low financial support. On average, only two to three contact persons for each DACAPO study centre were registered in the CSC. More than 70% of survey participants reported that physicians were responsible for data collection and documentation. Only 20 of 65 participating study centres (30.8%) of the DACAPO study permanently employed study nurses or clinical research assistants. This fact could explain the low proportion of study nurses in this survey and shows one option to improve the German research structure in the ICU setting.

Limitations

An ad hoc questionnaire was developed specifically for this survey. Face and content validity were ensured by repeated group discussions among the research team and a pilot test with a group of clinical researchers. Formal clinometric validation was not carried out. The anonymous online survey does not make it possible to calculate the participation rate in each study centre. The response rate of 45% could be a source of selection bias.

A further limitation is that the present study only reflects the researchers` opinions towards a single-specific study. However, participating researchers were from a variety of heterogeneous hospitals with differing workload and research experience and expertise. It would have been beneficial to include items on motivation in our survey as well. For future research standardised training formats would be useful with a view to maintaining high motivation and compliance among the study staff.

Conclusion

Collecting feedback on the acceptability of the conduct of a large multicentre healthcare research study provided important insights regarding training, obtaining informed consent, data collection, electronic data capturing and perceived relevance as well as importance of the study. Recruitment and data acquisition was mainly performed by physicians and often regarded as additional time burden in clinical practice. Reducing the amount of data collection and simplifying data entry could facilitate the conduct of healthcare research studies and could improve motivation of researchers in intensive care medicine. Optimised, tailored, flexible and usable study procedures are a necessary prerequisite for creating a culture of research.43