Implementation of a randomized controlled trial on an inpatient stroke rehabilitation unit: Lessons learned.

BACKGROUND/AIMS: The objective of this manuscript is to present challenges and solutions that arose during a mid-sized single-site RCT of a rehabilitation intervention performed in an inpatient stroke rehabilitation setting.
METHODS: Seventy-six participants from an inpatient stroke rehabilitation unit were randomized to experimental and control groups. All participants did 30-45 min of virtual reality (VR) daily for 10-12 sessions. The experimental group did VR targeting sitting balance while the control group did VR with limited arm movement. Challenges during the implementation of the RCT were documented and strategies to mitigate them were applied.
RESULTS: Challenges were placed into five categories:1. Recruitment. Our recruitment procedures required multiple steps prior to initiating direct patient contact; one solution would be to have patients consent to be approached about research upon admission to the inpatient unit.2. Patient-specific Issues. Fatigue, pain, vision problems and engagement were managed through scheduling, increasing the workload slowly and personalized modifications to the VR.3./4. Scheduling and Staffing. Recruitment and attendance at VR sessions were maximized through good communication, flexibility and cooperation, between research staff, clinical staff, volunteers, students and participants.5. Technology. Because hospital internet service was poor, a mobile internet data plan was purchased to ensure the system's reliability.
CONCLUSIONS: We have identified challenges in delivering a rehabilitation intervention on an inpatient stroke rehabilitation unit and some of the measures taken to surmount these challenges. Through good planning, flexibility and collaboration, almost all of the challenges were successfully addressed. CLINICAL TRIAL REGISTRATION NUMBER: URL: http://www.clinicaltrials.gov. Unique identifier: NCT02285933.

Introduction

Randomized controlled trials (RCT) are considered to be the “gold standard” of medical interventional research, including research focused on rehabilitation interventions [1]. The number of RCTs published in PubMed has been growing steadily over the past 50 years [2]. There are many challenges to implementing RCTs, and many are undocumented. Some are specific to a particular patient population, institutional situation or intervention. Rehabilitation interventions that are enmeshed within a multi-disciplinary care setting can be particularly problematic to research, since participants are undergoing many aspects of rehabilitation and care. One area that has seen somewhat greater discussion is the challenge of recruitment [3,4]. Tyson et al. (2015) [4] reported variable recruitment success at different sites during a stroke rehabilitation study, and found that “enthusiastic, regular and structured engagement with the entire stroke multidisciplinary team” was one key to successful recruiting.

It is helpful to share some of these challenges, and possible solutions, so that others can learn. While each research and clinical situation has its own unique challenges, we hope that by sharing our experiences others might avoid some of our pitfalls and thoughtfully consider how others might impact their research, timeline and budget. Therefore, the objective of this manuscript is to present challenges and solutions of study implementation that arose during a mid-sized single-site RCT of a rehabilitation intervention performed in an inpatient stroke rehabilitation setting.

These lessons were learned from the execution of a trial entitled “Does the addition of virtual reality training to a standard program of inpatient rehabilitation improve sitting balance ability and function after stroke?“ [5,6]. Approximately 25 500 Ontario residents have a stroke each year, and of those 50% are moderately or severely impaired [7]. For these people, sitting balance ability is a predictor of their level of functional mobility at discharge from inpatient rehabilitation [8]. Virtual reality (VR), the use of computer technology to allow someone to interact with a game or activity in a virtual environment, is a promising modality for stroke [9,10]. VR has been assessed for the rehabilitation of upper extremity function and standing balance and function (i.e. gait) post-stroke; however there has been no research on its use for the treatment of sitting balance. Therefore, the main objective of our RCT was to “determine if supplemental sitting balance exercises, administered via VR training, improve the control of sitting balance ability in stroke rehabilitation inpatients.”

Methods

The protocol for the RCT has been published previously [5] and the study received research ethics board approval. Seventy-six participants who could not stand independently for more than 1 min were recruited from a dedicated inpatient stroke rehabilitation unit. All participants provided informed consent. Participants were randomly allocated into one of two groups, experimental and control. Both groups performed 10 to 12 sessions of VR training (30–45 min each), in addition to their regularly scheduled therapies. Participants in the experimental group played five games that involved reaching and leaning movements, designed to challenge sitting balance (Fig. 1). Participants in the control group played five games that required only minimal upper extremity movement. Assessments of sitting balance and upper extremity function were performed by a research associate blinded to group allocation before and immediately after training and one month later. Of the 76 participants who began the study (38 in each group), 33 in the experimental group and 36 in the control group completed the VR training and post-assessments. Twenty seven in the experimental group and 26 in the control group completed the one-month post assessments.

Fig. 1

Participant playing a Ball Maze VR game. The Movavi screen capture, upper left, shows the VR game as presented on the screen. The webcam, upper right, shows the participant's movements. The CONFORMat image, lower right, shows the displacement of the centre of pressure (grey line) as the participant performs the game.

During the RCT, the research associate and VR trainer (LS) documented barriers to the research process. These barriers were addressed at monthly team meetings (which included the principal investigator, co-investigators, trainees/volunteers and research associate) and a plan was made to address each challenge. At subsequent monthly meetings the barriers and plans were reviewed and further actions were taken as required. Minutes of the monthly meetings were reviewed separately by the research associate and VR trainer in order to compile a list of themes, presented here with the challenges and solutions.

Results

Five themes into which the challenges could be placed were described: recruitment, patient-specific issues, scheduling, staffing, and technology. Challenges and solutions are presented under each theme in Table 1.

Table 1

Challenges and solutions discovered during the implementation of a randomized controlled trial in an inpatient rehabilitation setting. Challenges and solutions are presented within five categories: patient-specific, scheduling, staffing, technology and processes related to RCTs.

Issue	Challenges	Solutions
Recruitment
Recruitment	- Research ethics obligations required that potential participants had to be asked by a member of the patient's “circle of care” if they were willing to have our research associate talk to them about our project. This process was a potential barrier to timely and complete recruitment.	- A preferred solution, opt-in or opt-out, would be for patients to indicate at admission whether they were interested in hearing about research studies. We were not able to do this, due to ethical and management/logistical issues. Discussions regarding this issue are ongoing.
Patient-Specific Issues
Fatigue (9 out of 130 potential participants declined because of fatigue)	- Many participants needed planned breaks between treatments or an afternoon nap.- VR might fatigue a participant so that they would not participate fully in their therapy sessions.- A treatment session requiring 30 min of VR time could take up to 1 h, including breaks.	- Participant's schedules were carefully managed to ensure appropriate rest times.- VR sessions were often held later in the day, so that fatigue did not influence regular therapies, although this had to be balanced by the potential that a participant would be too tired to participate fully during the VR session.- Occasionally, participation in the study was delayed until fatigue had diminished.- VR sessions became longer as endurance increased.
Pain (2 out of 76 participants declined to continue because of pain)	- Pain of many types is a common sequela of stroke [11].- Some participants had pre-existing pain from conditions such as chronic rotator cuff tears or arthritis [12].- Some pain treatments (for example, certain medications, pain clinic interventions) were no longer available in the inpatient setting.	- VR sessions were shortened as required.- VR parameters were modified as required (while staying within the study goals).- Subluxed shoulders were supported at all times, especially during transfers.- VR sessions were scheduled appropriately with respect to pain medication schedules.- Research staff consulted with members of the participant's circle of care regarding ways to minimize a participant's discomfort.
Vision	- For many participants, their stroke had altered their vision.- Some individuals also had pre-existing conditions such as colour-blindness, cataracts or age-related macular degeneration.- Some elderly participants had a decreased ability to see objects that were green (green was a prominent colour on the VR screen).- Hemianopsia was frequently encountered [13].	- There was no need for participants to read, only look at a virtual object or avatar on a 40-inch television screen; therefore even individuals with poor vision could participate.- The TV screen was changed to black and white for those who could not see green. The VR manufacture also addressed this by changing some of the object avatars from green to yellow and black and increasing the contrast.- After these accommodations were incorporated, there were no further complaints of vision and poor vision did not appear to affect participation in VR.- We encouraged the participants to work on compensatory strategies for hemianopsia with the VR games, as appropriate [13]. Frequent cuing to look to the affected side was used as needed.
Engagement (2 out of 76 participants declined to continue because of disinterest)	- Some participants were initially nervous with the technology.- Some participants became less enthusiastic about attending VR sessions over time. This was more common in the control group, especially for participants with a very dense hemiparesis and minimal recovery of their upper extremity.	- The VR platform was user-friendly and easy to learn. Success could be achieved early and game parameters could be adjusted to increase the physical demands of the game or the potential for gaming success depending on the participant's motivations and the intervention goals for their group.- To increase engagement with the control group participants, when necessary, the VR trainer engaged the participant in conversation. Occasionally the participant's choice of music was played during the VR session. It was important to ensure that the music did not encourage motions like foot-tapping and dancing that might affect the intervention goals.
Scheduling
Appointment Scheduling	- It was sometimes challenging to complete 10–12 VR sessions and reassess before discharge. Patients attended inpatient rehabilitation for four to six weeks. There was often a delay of a week or more before consent could be obtained because potential participants needed time to carefully consider their participation and have their therapy assessments completed and schedules confirmed before signing. Then the pre-assessment had to be completed.- Completing 5 VR sessions a week was challenging. Participation in research was a lower priority than regular therapy and medical appointments. Participants received one to 5 h of therapy a day with family meetings, medical tests and appointments, and off-site appointments and visits scheduled as needed. Some participants needed scheduled rest periods. Some went home for regular extended weekends.- Communication regarding participant scheduling with the rehabilitation unit staff was difficult, complicated by the recent introduction of electronic patient records (EPR). All therapy and appointment scheduling was performed on the EPR but research staff were not allowed access.	- Most of the scheduling challenges were met by having a full-time VR trainer, which offered the flexibility to work around the participants' schedules.- Weekend appointments or two VR sessions a day were occasionally done, if time became short before discharge.- Communication boards hanging in each patient's room provided the best opportunity for the research staff to coordinate their participant's therapy schedule with VR appointments. The ward clerk, who had a master copy of the daily schedule, was another resource. However, there was still room for error as the unit staff tended to prefer the EPR.- A more permanent solution would be to have the VR trainer provided with access to the EPR.- Final result: experimental group participants performed 325 min of VR while control group participants performed 302 min of VR, which met the target of 30 min a day for 10–12 days.
VR Laboratory Scheduling	- Several research studies were held in the same VR laboratory.	- We procured room for a small second laboratory on the inpatient stroke unit. Because the VR equipment was portable, VR training could be done in this room if needed.- Researchers used a Google Calendar to schedule the laboratories.
Pre- and Post-assessment Scheduling	- Participants had to undergo pre- and post-assessments in a timely manner. The research associate who performed the outcome measures was available only three days a week.- A priori sample size calculations estimated that 31 participants per group would be required, and considering a 20% dropout rate, 38 per group should be recruited.	- The research associate's schedule, as well as holidays and the participant's schedule were considered carefully when booking assessment times to minimize the amount of delay between training and outcome measures.- Two casual research associates were hired and trained.- There was a range of VR sessions required for the study (10–12).
One-month Follow-up Scheduling	- The one-month post reassessment was challenging to schedule. Participants were generally discharged by then, to a variety of living situations. Four out of 69 were difficult to locate because their living situation changed after they were discharged. Four out of 69 lived or moved away from town and declined to return for the reassessment. Five out of 69 experienced health issues (subsequent stroke, seizures, other illnesses). Three out of 69 simply refused to be reassessed. These reasons lead to decreased rates of attendance for the one-month follow-up (53 total attended), resulting in an incomplete dataset.	- To mitigate these losses, we tried to accommodate participants' schedules, for example, scheduling the follow-up just after an outpatient therapy or physician follow-up appointment.- If a participant was only available for a few minutes, the laboratory-based outcome measures were performed at the hospital and the research associate performed the rest at the participant's home.- To avoid expenses for the participants, the research study paid for transportation or parking.- The longer the follow-up, the greater the attrition; therefore sample size should reflect this. A drop-out rate of 40% should be considered for a 1-month follow-up in this population.
Infectious Disease Outbreaks	- The physical layout on inpatient rehabilitation unit, which included shared rooms and bathrooms, along with staff treating multiple patients, led to a gastrointestinal illness outbreak during the study period which lasted three weeks. Patients who became ill were not able to attend their VR training for several days. Since a break in training could compromise the study objectives; these participants were removed from the study. Further, recruitment of new participants was paused because of the possibility that they might become ill.	- When planning the RCT, an extra 20% was added to the participant number calculations to allow for attrition due to all reasons, including infectious outbreaks.
Staffing
Personnel	- To maintain blinding, the VR trainer and outcomes assessor must be different people.- Beyond the paid personnel, the study required cooperation from all staff on the inpatient rehabilitation unit. Members of potential participants' “circle of care” must help to identify them and ask for their permission to be approached by the research staff. Therefore, the support and cooperation of all of the staff on the inpatient rehabilitation unit was essential.	- There were two primary paid staff members on the VR team. A full-time postdoctoral fellow (physiotherapist, PhD Rehabilitation Science) oversaw the study and did all of the VR training. A part-time research associate (physiotherapist, MSc Rehabilitation Science) performed the recruitment and outcome measures. Casual staff were available to cover VR training and outcome measures as needed.- The relationship between researchers and clinical staff was facilitated through open communication on an individual basis and at staff meetings. The main VR laboratory was located in a prominent room centrally-located on the unit and had a “doors-open” policy (except for training sessions), which enhanced visibility. Two open-houses, with VR demonstrations and food were held to increase the staff's awareness of VR and the research study. Colourful posters presenting VR research were prominently displayed throughout the unit.
Volunteers and Students	- At times additional help was required with the outcome measures, VR training, operating computers, inputting data, etc. This additional assistance was provided by volunteer students.	- Volunteer students were recruited from the local University and scheduled to attend 3 h once a week.- The volunteers were very helpful and in turn obtained valuable experience in research methods and participant/patient interaction. Some students took on complementary research projects such as reliability studies, in order to fulfill University requirements for research courses. Two full-time summer students also assisted for one summer; they were paid through separate, individual grants.
Vacations and Other Absences	- Staff were entitled to three weeks of vacation a year. Considering that each participant took at least 3 weeks to complete the entire study protocol, vacation and other time off presented a challenge.	- There were two strategies to deal with the VR trainer's vacation. For one summer, VR training coverage was provided by an experienced casual trainer, to create a seamless training protocol for the participants. For the other summer vacation period, the project was suspended. Recruitment ceased three weeks before the vacation started, to allow for the entire protocol to be completed with enrolled participants, and resumed a few days before the vacation ended. The VR trainer took all of her yearly vacation at once so only one suspension was required.- Shorter absences (due to conferences, illness) were covered by the casual VR trainer.- Two casual research associates (physiotherapists) were available to perform the outcome assessments during their lunch hour and after work, ensuring that pre- and post-testing were performed in a timely fashion.- Recruitment was put on hold during the research associate's vacation; however this did not generally impact the study.
Technology
Virtual Reality Equipment	- Game protocols for the study had to be designed to support the specific study objectives of the intervention and control groups and also allow for individualized progression of training.- Throughout the trial, it was important that concerns were addressed quickly, since an error or bug in the VR system could cause down time.- It was equally important that the VR platform did not change substantially over the course of the study, so that the intervention protocols remained consistent	- VR training was provided with Jintronix (Jintronix, Montreal, QC) software. Jintronix was very responsive to our needs and concerns.- Jintronix distributed regular software updates, which were intended to improve and expand their VR offerings and make the system more user-friendly. These updates did not always suit the research objectives; however Jintronix was very responsive to make changes according to our feedback.
Internet Connectivity	- The Jintronix VR system required constant high-speed internet connectivity to work. Because the computer running the Jintronix software was not a hospital-purchased system, it was not able to be connected to the general hospital internet service. Instead, a “guest” internet service was used. This service had limited bandwidth, which caused delays and loss of internet service. This was a significant and ongoing issue which at times threatened to shut down the study.	- Frequent discussions with the hospital information technology service did not provide significant relief until very late in the study.- The solution was to purchase a mobile internet key, accompanied by a data plan.
Monitoring Participants' Movements While Performing Virtual Reality	- We wished to ensure that the participants in the experimental group were challenging their sitting balance and the participants in the control group were not.	- A CONFORMat pressure mat (Tekscan, South Boston, MA) was used to monitor the displacement of the centre of pressure under the buttocks and thighs of the participants as they performed VR. The CONFORMat was ideal as it was comfortable to sit on during an entire VR session. An optional webcam fed into the software to take simultaneous video recordings.
Visual Record of Virtual Reality	- We wished to analyse quantitatively and qualitatively how the participant moved as they did VR training. Movements were to be compared over time and between intervention groups.	- Screen capture software (Movavi Video Suite 14, St. Louis, MI) was used to take videos of the television screen while doing VR. The same video suite software was used to make a four-part video of a participant performing VR, which included the CONFORMat image, the webcam video, the screen capture video and a title (Fig. 1). This was useful for qualitatively analysing participants' movements and for presentations.

Discussion

There were several challenges to the implementation of this RCT in the inpatient stroke rehabilitation unit, in part due to ethical and privacy obligations, the need for scientific rigor, and the fact that we were working with participants who had just experienced a life-changing event. Each participant had their own experience dealing with the effects of stroke, which impacted such things as fatigue and motivation. In the end, we were able to overcome almost all obstacles in order to maintain the integrity of the research study and treat each participant with respect while completing the RCT on time.

The greatest strength of our VR study was the people. The ideal situation is when clinicians and researchers work together and “side-by-side” with patients [14]. This is a long-term goal that takes time and effort to accomplish. Involvement in research benefits patients and clinicians by providing access to state-of-the-art equipment and treatments. Involvement of clinicians in research also benefits researchers since it provides a stream of potential participants, and collaborative discussions to determine research priorities. Our research team has been working with the clinicians on the stroke rehabilitation unit for almost 10 years. Over that time our relationship with the clinicians has evolved so that clinicians and researchers have become more of a team. Our early VR studies used a lab that was remote from the inpatient rehabilitation unit [15]. By moving the laboratory to a central location on the unit, and maintaining an open-doors policy, visibility was improved considerably, along with the opportunity to engage with clinicians, administrators, patients and families.

The greatest technical barrier to the implementation of this RCT was the availability of reliable internet service. While we expected reliable internet to be available in an urban, institutional setting, two issues prevented this. The first was the need for our VR computer to be on the “guest” internet system, since it did not have the firewalls and level of antivirus protection required to be on the institutional network. The second was that the “guest” system had relatively low bandwidth and the organization was not able to increase it or allow us to have greater access. Therefore the internet was frequently very slow and prone to interruptions. Purchasing a mobile internet key and data plan provided an economical and reliable way to mitigate this issue and offered flexibility to use the VR equipment in a variety of locals without reliable internet (i.e. for demonstrations and training). Fortunately, late in the project's recruitment the bandwidth issues were resolved, but this remains an important consideration for research or clinical practice using technology. Internet and information technology issues need to be considered in the planning and proposal stages of any clinical research project. The use of a data plan may need to be considered whenever uninterrupted internet is required.

We learned important lessons from our earlier VR research, which experienced significant barriers with the VR system. Levac et al. (2016) [15] trained outpatient therapists to use VR with their stroke patients and documented barriers to the implementation of VR in the clinic. Identified barriers included system usability as well as location. There were many hardware and software malfunctions, and obtaining solutions to these created many delays. Therapists found that it was time-consuming to leave the unit to accompany a patient to the VR laboratory and they were unable to use VR with only one patient at a time if other patients remained in the unit. These issues were addressed in the planning phases of the current RCT. Our VR laboratory was located centrally on the inpatient rehabilitation unit. While this central location was not essential for the research study, it would be for subsequent implementation by therapists. The Jintronix VR system was much more user-friendly than the system used by Levac et al. (2016) [15].

While scheduling participants' time for VR was at times difficult, there is considerable evidence that many patients in inpatient rehabilitation spend much of their time inactive and alone [16,17]. Canadian stroke best practice recommendations state that rehabilitation inpatients should have 3 h of direct, task-specific therapy five days a week [18]. In addition to being used by physiotherapists and occupational therapists as part of their one-on-one therapy, VR training might be able to complement traditional therapies and increase patients’ rehabilitation intensity. This could be accomplished by using VR as an adjunctive treatment, performed either during times when a patient is not otherwise scheduled for therapy, or in the evenings or on the weekends, when formal therapy is typically not available. A VR program could be developed for each individual patient by a therapist, but carried out under the supervision of an assistant or volunteer or even independently, as long as safety measures are in place (for example, doing only upper extremity exercises in sitting if balance is poor). The Jintronix VR system provides a simple-to-use user interface designed for a patient to access their personal exercise program.

The one-month follow-up assessment was completed by only 71% of the participants, for a variety of reasons (illness, moved, refused). The one-month follow-up is important to assess the sustained impact of a research intervention over a longer term. Because participants may have difficulty fulfilling their obligation for the final assessment, sample size determinations should take into account the expected numbers of participants at follow-up. Furthermore, only 74% of the one-month follow-ups occurred within one week of the one-month date, primarily due to scheduling issues between the research associate and the participant. The assessor(s) for RCTs must be very flexible and accommodating with respect to scheduling, in order to ensure that participants are assessed at the correct time. Providing in-home assessments might be preferable to requiring the participant to travel back to the hospital.

At the trial site, a member of a patient's “circle of care” must obtain permission from the patient for the research associate to provide information about the study. While it is understandable that staff with a primary interest in the health of their patients would be appropriate to pre-screen potential participants, this process appeared to be burdensome and surprisingly time-consuming for the clinicians. While this did not impede the overall study time-line, it was a source of frustration for the research associate. The trial site is currently exploring opt-in and opt-out options for study recruitment similar to the processes used in numerous other research hospitals. An alternative option would be to hire a member of the circle of care to perform the entire recruitment process.

Timely completion of research studies is important for researchers, funders, administrators and trainees. Funding typically extends for a defined period of time and continuation of a project past its deadline incurs unbudgeted costs and delays dissemination of the study findings. The academic progress of students involved in the study may also be delayed. We hope that our “lessons learned” helps other researchers implement clinical trials in very complex inpatient rehabilitation settings.

Grant support

The work was supported by a grant-in-aid from the (G-14-0005830), and by a generous personal donation from Tony & Elizabeth Graham. The funders accepted the proposed study design but did not have any role in the collection, analysis and interpretation of data nor in the writing of the report and decision to submit the manuscript for publication.