Implementation of a Virtual Reality recreation program in long-term care.

Introduction: This manuscript describes the implementation of a Virtual Reality (VR) recreation program at long-term care sites across Ontario, Canada, using the RE-AIM Framework to guide the implementation and its evaluation.
Methods: We developed a VR recreation program to enhance the lives of long-term care residents, through 3 sequential phases. In Phase 1, we learned about resident and staff needs through focus groups, staff surveys and observations. In Phase 2, we developed 10 VR experiences, based on the data from Phase 1. In Phase 3, we implemented the VR experiences and supporting manual and measured their implementation, using the RE-AIM Framework.
Results: We found the VR program to be highly (but not consistently) implementable across all sites. Factors that supported implementation were the following: resident interest in the content and technology, relative ease of use for staff to implement and formally integrating VR into the recreation calendar. Factors that impeded implementation were the following: the size of the headset, inability for the headset to cast given the sites' Information Technology infrastructure and some content that was not engaging. Conclusions: VR programs are highly implementable and this implementation is enhanced by integration of the program into existing recreational systems, ease of use and resident engagement.

Introduction

The proportion of our global population that is 65 and older is growing faster than ever before. According to the World Health Organization, by the year 2050 there will be approximately 2 billion older adults populating the globe. With this significant demographic shift occurring, individuals are spending longer in their senior years, and living with higher levels of chronic conditions associated with this stage of life. This drastic increase in the population of older adults necessitates changes to the delivery of flexible care services, community care and long-term care homes.

Older adults living in long-term care are amongst one of the most vulnerable populations, with high rates of cognitive impairment and physical impairment. Residents also face many challenges in relation to social isolation, such as feelings of lowered sense of belonging and less access to social supports in comparison to older individuals living in the community.[5,6] Furthermore, these challenges have significant implications on the mental health of residents, including high rates of depression and anxiety. To combat these challenges, as well as other difficulties, these facilities deliver a number of different programs made to promote the residents’ psychosocial and functional capacities, quality of life and happiness. Recreational programs in long-term care homes have therapeutic value to residents, especially when they are first adjusting to living within the facility. This value has been demonstrated through residents’ increased autonomy and independence, and improved mental alertness, as well as better physical performance. Research has shown that engagement in recreational programs offered in long-term care homes has a positive impact on the quality of life of older adults. Yet, studies conducted globally have also revealed that these recreational programs are typically limited (in terms of the amount and range of activities available), leaving residents with little to counter the various challenges such as loneliness and boredom.

Virtual Reality & Older Adults

Virtual reality (VR) is a technology that replaces one’s senses and physical environment with images and sounds produced by technology. Individuals become immersed within this new ‘world’ that replaces their lived reality and can also (with certain types of VR) interact with this digital environment.[5,15] Although it is commonly believed that VR is a new tool, some type of VR system or another has existed for decades.[15,16] With that being said, VR designed specifically for older adults is certainly in its early stages. While this area of research is rapidly advancing, there is much to learn about VR and its potential impact on older adults.

Much of the research on the potential uses of VR in older populations has focused on its potential as a therapeutic agent and as a source of recreation. The literature on VR as a therapeutic agent explores its potential to enhance physical wellbeing – from improving balance to reduce falls, to stroke rehabilitation, to improving pain management – and its potential to enhance cognition – especially among those with cognitive impairment. VR as a tool to enhance recreation programming in long-term care homes and the wellbeing of its residents has been studied, although these studies have primarily been at single study sites and in very small numbers of residents. The exception is Appel and colleagues, study, a feasibility study of the use of VR therapy for older adults that piloted VR in four locations, including one long-term care facility. The findings and approaches of these studies are detailed below.

The existing literature on effective therapeutic VR interventions among older adults includes studies where VR was utilized as a form of pain management, rehabilitative therapy, as well as for cognitive therapy. For example, Benham and colleagues conducted a pre–post study of the use of immersive VR among community-dwelling older adults (n = 12) to understand its impact on pain management and user acceptability. They found that this type of VR was well-received and entertaining, in addition to being an effective method of pain distraction for the participants. In Optale and colleagues’ randomized controlled trial which studied the impact of VR on memory function in older adults (n = 36), the results showed that VR memory training enhanced participants’ focused attention. This increased focused attention, coupled with the fact that the VR environment motivates selective attention, may improve older adults’ memory function.

Yeşilyaprak and associates conducted a randomized controlled trial (n = 18) that investigated the impact of VR-based balance exercises on the balance and fall risk of older adults living in nursing homes (the authors refer to long-term care homes as nursing homes) in Turkey. They found that VR-based balance exercises aided in the improvement of balance function, in a manner similar to conventional balance exercises. Another VR-based balance training program for older adults studied by Duque and colleagues in a randomized controlled trial (n = 60 community-dwelling older adults) demonstrated an increase in balance and a decrease in fear of falling and number of falls, and participants reported enjoying the intervention.

Finally, several scientific groups have investigated the impact of VR therapies on older adults living with dementia, and particularly the potential for VR to improve their mood. Appel and colleagues conducted a prospective, longitudinal pilot study of older adults (n = 10) with a diagnosis of dementia who were inpatients in an acute care hospital. They were shown nature scenes using a head-mounted display, and the research team found that their acceptance of the technology was high and the technology demonstrated a potential to manage mood in these individuals. Niki and colleagues investigated the impact of VR head-mounted display scenes on nursing home (the authors refer to long-term care homes as nursing homes) residents living with mild cognitive impairment or dementia, focusing on how using VR for reminiscence may impact anxiety. They conducted a pilot randomized cross-over study with 10 residents and found that anxiety decreased after exposure to the VR, and that minimal side effects were reported.

In a pilot qualitative study (n = 5 residents and n = 5 staff) of the experiences of VR among residents of a long-term care homes Baker and colleagues found that VR had the potential to give these residents many new opportunities, including; experiencing environments that were no longer geographically accessible to them, participating in different activities, as well as partaking in virtual meditative experiences. Baker and colleagues highlighted VR’s potential to address boredom in those who tend to socially isolate by providing immersive experiences which are a novel and accessible form of engagement. VR may be an effective tool in engaging individuals who typically self-isolate, and increasing engagement of residents with their family and friends. Baker and colleagues also emphasize that VR in long-term care could boost residents' mood by being a valuable source of entertainment. They assert that VR is a promising intervention for enhancing the lives of older adults who are often confined to their environments and this technology can possibly ‘expand their worlds’ beyond those walls. Similarly, in their acceptability study of VR, Huygelier and associates found that head-mounted VR was acceptable to their sample of 76 older adults, Acceptability levels increased after using the technology, and self-reported cybersickness was minimal. The multi-site study of the feasibility of VR therapy among older adults with physical and/or cognitive impairment conducted by Appel and colleagues found that the technology was feasible and safe, finding no negative side effects such as dizziness and disorientation. Furthermore, they found that over three quarters of their participants wanted to try it again, and overall positively responded to the experience. The authors did not conduct sub-analysis of the participants living in long-term care (one of the four sites included in the study), so it is unknown whether the cohort of long-term care residents had a slightly different experience.

Implementation of Effective Interventions

Executing interventions in long-term care can be challenging due to the heavy workload, high staff turnover and other competing priorities. Therefore, proper implementation of interventions in this setting, including those involving VR, can be a crucial component. An example of an intervention targeted at older adults within care environments that failed due to poor implementation is the video-call intervention known as Skype on Wheels (SoW). This intervention consisted of a device on wheels that held an iPad and a handset. Skype on Wheels gave residents opportunity to connect to their family and friends via Skype video-calls to target the loneliness and social isolation. Although results showed that this intervention was beneficial to the older adults who participated, it is important to note that implementation did not occur in half of the study’s settings as there were several barriers to the implementation of Skype on Wheels. These barriers included risk averseness, the physical design of the intervention, high staff turnover, staff attitudes towards the Skype on Wheels device and the lack of family commitment. Another example of an intervention that failed due to suboptimal implementation was the PACE Steps to Success Programme – an intervention targeted at staff members in long-term care environments to improve palliative care. The investigators evaluated its implementation across 37 long-term care homes and found that implementation varied considerably. There were three major categories of factors that affected this: (1) the programme itself, as well as its delivery, (2) the staff working with the programme and (3) the context in which the programme was implemented. The study’s investigators highlight the importance of close observation and monitoring of programs during their implementation to ensure that they are appropriately adjusted to fit within the necessary contexts. They also stress the importance of proper training to support staff implementation and appropriate modification of programs.

The RE-AIM Framework

The RE-AIM framework was developed to guide the evaluation of public health interventions. The RE-AIM framework can be used to identify interventions that are worth the investment of further time and resources, and those that will be effective in real-world environments, as opposed to highly controlled settings. It is one of the most commonly used frameworks to evaluate the implementation (or potential for implementation) of a program or intervention. This model conceptualizes the effect of an intervention along five different dimensions: reach, efficacy, adoption, implementation and maintenance. Reach assesses the individual level of participation within the intervention as well as their characteristics. This dimension encompasses all individuals who are affected by the intervention or program. Efficacy refers to both the positive and negative outcomes that occur due to the intervention. Adoption, as a dimension of the RE-AIM framework, looks at the ease (or difficulty) of the intervention being implemented within the chosen setting. Implementation measures intervention/program delivery and assesses whether it is accurate and as intended. This dimension can include both individual-level and program-level measures. The final dimension, Maintenance, refers to whether the intervention or program becomes a stable, long-term part of the community.

The RE-AIM model has not yet been used to understand the implementation of VR interventions for older adults. The success of a new technology is not simply based on its measured outcomes, but on how well it is implemented. As such, we have developed and tested VR programming for older adults in long-term care homes using this approach. The results of the impact (versus the implementation) of the program have been published in another manuscript 30 (supercript). The purpose of this paper is to examine the implementation of our head-mounted VR system in long-term care homes through the lens of the RE-AIM framework. Following the analysis, we list recommendations for improving the implementation of VR systems in long-term care homes.

Methods

This study was approved by the Sheridan College Research Ethics Board. We adopted a mixed-methods approach using both quantitative and qualitative methods. We partnered with a private company that manages over 30 long-term care (LTC) homes in the province of Ontario to develop and test a VR recreation program for their residents. Their corporate head office chose five LTC home sites to work with as data collection sites and collaborators. These sites were chosen to be (a) diverse geographically and (b) sites that often do not have access to innovation.

Design. The project was organized in three phases, spread out across 15 months. In Phase 1 (months 1–6), we collected data about each site’s recreational wants and needs and capacity to deliver a technology-based program. In Phase 2 (months 7–10), we filmed and edited VR footage based on data obtained in Phase 1. In Phase 3 (months 11–15), we implemented the VR programming across the pilot sites.

Phase 1: Site data collection

Participants: Staff and residents in five LTC homes

Procedures: We conducted a staff survey and ran focus groups in each location. We sent online surveys (which contained some closed, but primarily open-ended questions) to all staff members of the five pilot sites. These were used to collect preliminary information such as their position at the facility, comfort with technology, barriers and facilitators at their sites, preferences regarding how they receive training, and any previous experience facilitating or using VR. The data collected helped us better understand the staff, their knowledge of VR and their preferred method of training. After the survey data was collected, we visited each site to conduct semi-structured focus group sessions with staff and residents. The sessions lasted 30–60 min and a total of 39 staff-chosen residents across all five sites were involved in the activity. The staff selected residents who could participate in a focus group and they thought could benefit from expanded recreational programming. Staff specifically sought out residents who did not typically participate (or could not participate) in most recreational programming and whom they felt may benefit from additional programming and support. During our focus group sessions, we discussed what technology participants currently used, their interest in technology and encouraged them to imagine what types of virtual experiences they would like.

Materials: Survey Monkey™ was used to conduct the online survey

Data Analysis: The survey results were downloaded into a spreadsheet, which we used to summarize and organize the data. We audio recorded and transcribed all focus groups. Two team members independently thematically coded the transcripts. They discussed and resolved their differences to develop a final coding scheme, with input from other team members if resolution proved challenging.

Phase 2: Develop VR Experiences

Participants: Clients of an adult day centre

Procedures: We collaborated with an industry partner to facilitate the creation of video experiences using an InstaPro360 camera. We initially filmed four sites in Ontario, recording unfiltered video and audio. We were not given permission to film iconic but requested sites in the area (e.g. the Aquarium, the Skydome, etc.), which limited our films to public spaces. We experimented with different types of videos (stationary and moving), and decided to move forward with only stationary videos to minimize motion sickness. Our industry partners made a trip to Western Canada (Alberta and BC) to film popular outdoor locations (a total of nine). Due to constraints within the research program and personnel logistics, recordings in Eastern Canada were not feasible. To obtain early feedback, we created 3 prototype experiences with early local footage. We hosted two sets of user testing sessions with community partners (staff from a day program for people living with dementia) where we asked staff and clients to watch the prototype films. We chose to test the prototype films with this population because the organization that runs the adult day program was initially interested in also testing the VR experiences with their clients, but learned quickly that many of their clients living with advanced dementia did not accept the headset. As such, this group was no longer interested in implementing a VR program using the head-mounted display. However, they had a few clients who found the headset acceptable who were willing and interested in providing initial feedback. Additionally, the adult day program was adjacent to our labs, providing us with convenient access to regular feedback on our prototypes. Finally, we chose to test these prototype experiences with people living with dementia to help ensure that our experiences were acceptable to people of all cognitive status. When the user was comfortable with the headset (n = 1), we asked them to put it on and view it through that. When the user was not (n = 2), we showed them the 360 video on a desktop computer and invited them to explore it. This early data from the day program’s clients and staff helped us revise those experiences and build out another seven experiences. We developed a VR manual to help the site staff use the headsets and implement the VR recreational program.

Materials: InstaPro360 camera, Adobe Premiere Pro video editing software, Oculus Go VR headset

Phase 3: VR Program Implementation

Participants: Residents (n = 32) and Staff (n = 26) of four LTC homes

Procedures: Once the VR experiences were developed, the research team progressed to Phase 3, where they organized training sessions at four of the five pilot sites (the fifth site had an outbreak during the pilot testing phase, so we visited them afterwards to pilot test the manual, as the final implementation step). Two members of the research team visited each of the pilot sites twice – once to provide training and collect data from the residents who used the VR, and again 2–3 weeks later to collect feedback from staff and collect more resident VR data. During the first visit we ran a 60-minute training session with staff and recreation directors of each facility where we discussed the research program, its benefits and how to use and troubleshoot the equipment. The training demonstrated how to select an experience, how to mount the headset onto a resident and how to cast the experience onto a tablet (an optional, but helpful feature, which allows the staff member to see what the resident is watching). We then modelled how to use VR headsets with residents and collect data to capture its implementation and impact. We modeled how to use each of four data collection instruments: (1) the PAINAD to collect pain based on observation of the resident, (2) the Interact Short form to capture resident reaction to the VR, based again on observation, (3) the implementation sheet that we developed based on the RE-AIM framework that captured resident reactions to the visuals, audio, narration and headset and (4) a short qualitative interview questionnaire, which asked residents about their perceptions of the experience, whether they would like to use it again and its impact on any chronic pain they may be feeling (to allow us to capture both observed pain – using the PAINAD – and subjective pain). During the two-week pilot period, we asked staff to use the technology when and as often as they could, and administer the Interact Short form and the qualitative questionnaire (administering the VR program and all four data collection instruments would not have been possible for one person). Staff collected this data on residents who consented to the research project. Staff members were encouraged to use the technology with any interested resident, so as not to restrict its usage to those who provided consent. Staff recorded when they used the technology, but only used the data collection instruments with those residents who consented to be part of the study. After the pilot phase, the research team returned to the site to collect additional data on resident use of the VR and to learn how the technology was implemented at each site. We collected the data collection sheets that the staff completed and asked all staff to complete a feedback survey to provide further detail on their experience using the technology and suggestions for its further development and implementation.

Materials: We equipped each site with a VR toolkit that included an Oculus Go™ headset, the VR manual we developed, consent forms and data collection sheets. The data collection sheets included: the Interact Short form to measure resident reaction, the PAINAD (Pain Assessment in Advanced Dementia) scale to capture observed pain, a short qualitative questionnaire to learn about resident experiences, and a sheet that the research team developed to record implementation data based on the RE-AIM framework. All four instruments were used to collect data during the research team’s visit. Members of the research team were trained on how to use the instruments., and trained the recreational staff on how to use the Interact Short form and the short qualitative questionnaire.

Analysis: We created an Excel spreadsheet to analyze the quantitative data. We used descriptive statistical analysis to summarize resident engagement, impact of the VR program on pain and implementation of the VR program (using the RE-AIM framework). We analyzed the qualitative data using thematic analysis methods, with one research team member (LH) conducting the primary analysis and another (FC) validating the work. The purpose of the analysis was to identify barriers and facilitators to implementation, categorizing resident responses into each of the RE-AIM domains.

Results

Phase 1: A total of eight staff members (out of 18; 44%) completed the pre-visit survey; all subsequent analyses in this section reflect the responses of these eight staff. Overall, staff described themselves as relatively comfortable with technology, and had some familiarity with VR. Six respondents were ‘comfortable’ or ‘very comfortable’ using the technology, although two were ‘neutral’ or ‘very uncomfortable’ with them. Six staff members responded that residents only engage with technology provided to them ‘sometimes’. As for training, we found that a total of six staff members had received some sort of training during the implementation of any previous technology-focused programming (such as tablet games, an electronic whiteboard, etc.,) and two members had received no training at all. Although some sites have had technology-driven programs implemented in the past, only one site facilitated a program that involved using VR. However, six staff members had at least some knowledge of the platform. We also asked them what kind of environments or activities they think the residents would like and the top suggestions were visiting the residents’ hometowns and festivals/fairs. Finally, we asked them what kind of training they preferred while learning about new technology, and the most popular responses were hands-on training and workshops. This helped us fine-tune our approach to create an effective VR training program. We conducted five focus groups – one at each of the pilot locations. These focus groups included both residents and staff members. The focus group participants are described in Table 1.

Table 1.

Focus group participants.

Location	Resident count			Staff count	Average age of residents
	Male	Female	Total
Location A	5	5	8	5	81
Location B	3	3	6	3	77
Location C	2	3	5	3	74
Location D	4	6	10	2	71
Location E*	4	6	10	5	77

*Location E was not part of the pilot testing (they had an outbreak during that phase and thus we could not safely enter the home), and was later used to field test the VR manual, to ensure that it could support implementation of the VR program without a visit from the research team.

The focus groups taught us about resident comfort level with technology and what types of VR content they would be interested in. The residents’ comfort with technology varied – some used tablets and personal computers, while others used smart phones, and a substantial number did not regularly use any technology (save the landline telephone). In terms of locations to feature for the VR filming, residents offered a wide variety and number of locations. A summary of requested locations is summarized in Table 2. As we can see from Table 2, there are three categories of locations: Canadian, International and more general activities and locations (such as fishing and forests). For feasibility, we decided to focus on iconic Canadian locations (both ones people have seen and have always wanted to see) from Western Canada and Eastern Canada and some locations in Ontario, while trying to incorporate as many general locations and activities as possible in each of these target locations.

Table 2.

Summary of requested locations for VR filming.

Canadian locations	International locations	Activities and non-specific locations
Toronto	Switzerland	Countryside
Niagara Falls	Hawaii	Snorkeling
East coast	Alaska	Parasailing
Western Canada	Mexico	Sports games
Kelowna	England	Ocean
Calgary	Spain	Camping
Algonquin Par	Greece	Fishing
Kakabeka Falls	Venezuela	Campfire
—	Paris	Nature
—	Jamaica	Forest
—	Rome	—

In Phase 2, after obtaining local footage, we stitched and edited it, creating three VR experience prototypes. We tested these prototypes with n=3 members of an adult day program. From these user testing sessions, we gained valuable information on how we could improve our videos. The feedback included: limiting the length of the video to a maximum of 8–10 min, adding relaxing background music and adding narration to focus the attention of the user. We applied these learnings to the editing and development of the final set of VR experiences. When adding calming background music, we had to maintain a balance between music and ambient sounds as any noise-cancelling effect in the video would interrupt the experience of the participant and would reduce or even completely stop the VR immersion. A description of all VR experiences is listed in Table 3.

Table 3.

Descriptions of all VR experiences developed.

Name	Location	Description
Burlington Pier	Burlington, Ontario	A visit to the boardwalk and promenade of the Burlington Pier on a beautiful summer day
Hendrie Park	Royal Botanical Gardens, Burlington, Ontario	Beautiful cultivated garden with incredible flower beds and flowering trees
Niagara Falls	Niagara Falls, Ontario	Tour of the Canadian side of the Niagara Falls, with spectacular views and an outdoor concert
Victoria Docks	Victoria, BC	Tour of the popular tourist destination, including a water taxi ride
Butchart Gardens	Brentwood Bay, BC	Select footage of the famous gardens, focusing on the rose garden
Stanley Park and Granville Market	Vancouver, BC	Scenes of the vibrant Granville Market and the impressive natural landscape of Stanley Park
Capilano Suspension Bridge	North Vancouver, BC	Scenes near the 110 foot suspension footbridge, a popular tourist destination
Canadian Pacific Railway Last Spike	Craigllachie, BC	This spike (made in 1885) marked the completion of the Canadian Pacific Railway
Lake Louise	Banff National Park, Alberta	Dramatic scene of Emerald Lake, with mountain vistas and wildlife
Banff	Banff National Park, Alberta	Ride on the Gondola, with six stunning mountain ranges in site, to the top of Sulphur mountain

In Phase 3 we collected implementation data based on the RE-AIM framework. Below we describe each of the sites briefly and then detail the data collected by site in terms of each of the components of the RE-AIM framework.

Site Descriptions

These Site Descriptions are based on interviews with each location’s recreation manager, video and audio footage taken at each location and observational notes recorded during site visits. We have added these descriptions to help the reader better understand the context of each site implementation.

Location A

This facility, located in a small town in Northern Ontario, is a long-term care home that was built in 1972. It is currently occupied by 148 residents and eight recreation staff members (3 full-time and five part-time). It has over a hundred different recreation programs that cover social, spiritual, physical, emotional and cognitive domains. The home is divided into three areas or ‘neighbourhoods’ and each of them has its own monthly calendar for programs. The programs vary depending on the resident population and their therapeutic needs in that area and are extremely important for the satisfaction of both the residents and the facility administration.

Location B

The long-term care home was built in 2000 and is located in a rural area.The home houses 96 residents, with an average age well into the 80s, and has seven recreation staff members (2 full time, two part-time, one supervisor and two extra staff) to help run programs. All programming is planned based on the facility’s home areas, and since there are 3 home areas or ‘wings’, the staff members create 3 different recreation calendars, which they plan monthly. All programs offered at this facility are focused on five domains: physical, cognitive, emotional, social and spiritual. The facility sometimes assigns their specialized Dementia-focused program, which is sensory-based, to certain areas based on the cognitive capabilities of residents. According to the staff, the population at the home in recent times tends to turn over faster than it would have 15 years ago. As such, they have more tech-savvy residents in their facility who use computers, cell phones and iPads. This is the only location we visited that has trialed VR experiences in the past. Although the home has an abundance of common areas, the staff members usually conduct these sessions one-on-one with residents by their bedsides. These positive experiences with VR technology have led them to plan a future where the headsets are available throughout the whole building so that residents can use them on demand.

Location C

This urban facility has a total of 170 residents and seven recreation staff members (4 full time and 3 part-time). The building was built in 1950 but was converted into a long-term care home in 1960. The facility has a large population of non-native English speakers (with many Cantonese and Polish speakers). The facility is supported by many staff members who have worked there for over 30 years. The building is divided into multiple home areas and each has its own lounge and calendar of activities and special events. A variety of recreational programming has already been implemented, such as Bingo, bus outings, active games, mental aerobics and spiritual activities. The programs at this facility have evolved significantly over the past 5 years and the staff and administration are working hard to implement modern technology, such as iPads, Netflix, Apple TV and Google Home, in their programming.

Location D

This facility is located in a northern urban area. It was opened in 1972 and has 57 residents. They have several full-time and part-time recreation staff members (the exact number was not available, but estimated to be 4, considering the smaller size of the home) and conduct placements for college students to learn about recreational programming and long-term care. They have a busy recreational calendar and try to ensure that programming is available during different times of the day and on both weekdays and weekends. They do not use much technology currently in their recreational programming, but are interested in using technology to bring more options to residents.

Population

A total of 32 residents across all four sites used the VR programming, for a total of 102 experiences. Since we tested on participants during site visits at the commencement of the pilot and 2 weeks later, and since staff used the VR with residents for 2 weeks between the visits, some residents (n = 26) had multiple opportunities to use the technology. The participants are described in Table 4.

Table 4.

Pilot study participants.

Location	Resident count (consented)			Average age of participants	Age range of participants
	Male	Female	Total
Location A	2	5	7	70	50–80
Location B	3	1	4	74	71–81
Location C	7	5	12	81	67–91
Location D	4	5	9	78	60–100

Implementation data (using the RE-AIM framework)

To understand how implementable the program and technology were, we gathered implementation data using the RE-AIM framework. These data are described below. For 67 of the 102 times the VR technology was used, site staff collected data using the Interact Scale and the qualitative interview guide. In 35 of the 102 times the VR technology was used, research staff collected data using the Interact Scale, the PAINAD, the RE-AIM implementation sheet that we developed and tested and the qualitative interview guide.

Reach

Reach measures the individual level of participation, and also concerns itself with who gets access to the intervention and who does not. We measured aspects of reach during each site visit and during the 2-week phase where each site was left to implement the technology. The reach data collected, by site, is described in Table 5.

Table 5.

Reach data across sites.

Measure	Location A	Location B	Location C	Location D
Data collected during site visits
Number of male residents who used technology	2	3	4	4
Number of female residents who used technology	4	1	2	5
Total number of residents who used technology	6	4	6	9
Total number of experiences	9	6	9	10
Average age of participants	68.4	75.3	75.4	82.7
Data collected during 2-week pilot phase
Number of male residents who used technology	3	3	2	7
Number of female residents who used technology	5	1	3	5
Total number of residents who used technology	8	18	14	11
Total number of residents who consented to be in study	7	4	12	5
Total number of residents in location	148	96	170	57
% of residents who accessed technology at that location	5.4	18.8	8.2	19.3
All data collected from residents
Total number of experiences	37	11	31	23
Total number of residents using VR	7	4	12	9
Average experiences per resident	5.3	2.8	2.6	2.6
Data collected from post-intervention staff surveys
Number of staff who responded to the survey	5	3	5	3
Did they include the intervention in their ‘recreation calendar’?	No	Yes	No	Yes
What types of residents did the intervention work best with – according to staff?	Wheelchair users, those with memory loss	Everyone	Everyone	Those with better cognitive health
What types of residents did the intervention not work well with – according to staff?	Those who can get disoriented	None	Those who were not interested	Those who did not want to wear the headset

During the site visits we had an opportunity to learn about the type of residents who were provided access to the intervention. Across all sites, we saw that both men and women were provided with opportunities to access the intervention. Location A targeted a younger (age range: 50–80, average age: 70), more cognitively well audience. Location D targeted an older population (including one person 100 years of age). We saw that the intervention reached many residents with assistive devices, such as glasses, walkers and wheelchairs. The ‘other’ assistive devices included an oxygen tank and a communication board – both indicators that physical and communication ability did not seem to prevent staff from recruiting residents.

During the 2-week pilot phase, the recreation staff used the VR technology as often as they could and incorporated it into their programming either formally (by putting it into their recreation calendar) or informally (by individual recreation staff using the technology with residents on an ad hoc basis). We see that the locations that incorporated the intervention into their recreation calendars had the highest reach (18.8% and 19.3%). We also saw a range of staff observations about who the intervention does work well for and who it does not.

Efficacy

Efficacy relates to any positive or negative outcomes of an intervention. We measured efficacy at the individual level both during the site visits and the 2-week pilot period. All data collected using the Interact Short instrument, PAINAD instrument and qualitative interviews is reported elsewhere. Here we present the Interact Short and PAINAD data by site location, to understand differences in efficacy between sites. The scores in Figure 1 represent the percentage of experiences at each site that scored a 4 or 5 (out of 5) for each aspect of the Interact Short questionnaire. The scores are an indicator of frequency of the behaviour. A score of 1 equates to ‘not at all’, a score of 3 is ‘some of the time’ and a score of 5 is ‘nearly all of the time’ (Figure 1).

Figure 1.

INTERACT scores by location.

We can see that there is variation in resident reaction to the intervention, based on their observed behaviour and vocalizations. Sometimes residents were nervous or confused about the intervention, although very few. We also see that across all sites at least 60% of the time residents responded positively in some particular way or another – and sometimes in multiple ways. This demonstrates that there are multiple types of positive behavioural responses to VR, and that some individuals may find it relaxing, while others find it more invigorating. Location A tended to have the lowest scores in the positive elements (relaxed, happy/content, enjoying self, related well to staff). We hypothesize that this is due to the relatively younger cohort at location A (where residents were as young as 50 years old) and their collective request for more adventurous experiences. Because our experiences were not thrill-seeking (e.g. skydiving, swimming), we believe that some residents were somewhat disappointed with the experiences. In fact, one resident expressed disappointment in the slow pace and relaxing nature of the experiences.

Other VR interventions have noted that the technology can act as ‘distraction therapy’ and relieve the experience of pain for users. As such, we also collected observational pain data using the PAINAD instrument. Those data are described in Figure 2. These data demonstrate that overall, pain scores were low, but varied by location. This is not surprising, given the small sample sizes.

Figure 2.

Frequency of PAINAD score, by site location.

We tracked how many times residents refused the headset or described a negative experience. Out of the 102 total experiences, the resident refused the headset or asked to have it taken off prior to the end of the intervention only 2 times. Other ‘negative’ responses to the experience, as recorded during the post-experience interview were: could not see anything (n = 2), found the content scary (either due to heights or water) (n = 3), was bored (n = 1).

Adoption

Adoption refers to the ability of the care facility staff to adopt a new form of programming and predict how easily they will be implemented into their existing programs. Through direct observations of residents and information from staff surveys, we learned how comfortable the staff and residents were using the headset and used this to gauge their interest in our program (Table 6).

Table 6.

Adoption data across sites.

Measure	Location A	Location B	Location C	Location D
Number of staff members who had previous experience with the technology (n) (source: staff survey)	0	1	1	2
Residents who found it easy to adopt the technology (n/%) (source: INTERACT Short form), n (%)	7 (77.7)	5 (83.3)	9 (100)	6 (60)
Residents who faced some difficulty in adopting the technology (n/%) (source: INTERACT Short form), n (%)	2 (22.2)	1 (16.6)	0 (0)	2 (20)
Residents who found it very difficult to adopt the technology (n/%) (source: INTERACT Short form), n (%)	0 (0)	0 (0)	0 (0)	1 (10)
Residents who had no issues interacting with the headset (n/%) (SOURCE: RE-AIM observation sheet), n (%)	6 (66.6)	6 (100)	7 (77.8)	5 (50)
Residents who said they would like to try again, or might try again (n/%) (SOURCE: RE-AIM observation sheet), n (%)	7 (77.8)	5 (83.3)	2 (22.2)	3 (33.3)

Looking at these data (Table 6), we can see that the majority of residents found it easy to adopt the technology, despite the fact that it was new to most staff members and residents. Some faced minor difficulties in adopting it and just 2.9% of all residents found the headset very difficult to adopt. During our initial visit and staff training sessions, we found that most staff members were also excited about the potential of the technology and were comfortable adopting it once trained on it by the research team. Staff members from Location D were at first reluctant with the technology but gained confidence when they saw the residents use it. Residents at Location B had absolutely no issues interacting with the headset (likely due to their familiarity with the technology, having had prior exposure to VR). Location C had the greatest success in adopting the technology, while Location D faced the most difficulty adopting it. Judging by the percentage of residents who wanted to try more experiences, Location B (with 83% of residents willing to try it again) had the highest engagement with the headset. However, willingness to try again varied significantly between sites, from 22% at Location C to 83% at Location B. This variation could be related with familiarity with the technology. Location B had previous experience with VR programs, indicating that initial reluctance with the technology may be overcome with repetition and time. Overall, the data suggest that most residents were successful in adopting and interacting with the VR headset.

Implementation

Implementation pertains to the accuracy of the delivery of a program as it is intended. In this section, we describe the implementation of the technology at each site during its 2-week pilot phase, in addition to our observations of the implementation of the program during our site visits (Table 7). We used these data to identify bottlenecks and limitations of the hardware/program.

Table 7.

Implementation data by location.

Measure	Location A	Location B	Location C	Location D
Pilot phase data
Total experience count during 2-week pilot phase	30	23	25	16
Number of staff who used the headset with residents in the 2-week period (n)	1	3	3	3
Number of residents who participated in 2-week pilot phase (n)	8	18	14	11
% of entire resident population who participated in 2-week pilot phase	5.4	18.8	8.2	19.3
Site visit observation data
Experiences where the resident reacted to images (n/%), n (%)	9 (100)	4 (66.7)	8 (88.9)	10 (100)
Experiences where the resident reacted to the narrator (n/%), n (%)	4 (44.4)	6 (100)	7 (77.8)	7 (70)
Experiences where the resident reacted to the narrator’s questions (n/%), n (%)	2 (22.2)	5 (83.3)	6 (66.7)	8 (80)
Experiences where the resident reacted to music (n/%), n (%)	4 (44.4)	0 (0)	0 (0)	2 (20)

Staff members from all sites were very successful in implementing the headset. The technology was implemented at all four sites and at 3 of those sites, 3 different staff members used it. At the remaining site, only one staff member used the technology (even though they did not attend the in-person training), and they used it over 30 times. In terms of the experience, the staff observed that some residents moved around and tried to interact with their virtual surroundings and the staff assisted them in doing so safely. Two locations also made additions to the program in order to better implement it. Location D experimented with additional senses. For example, they brought out a bowl of water for the residents to dip their hands in while watching the Lake Louise experience to provide a more immersive experience. Location B tried other, more active video experiences such as piloting an airplane, using YouTube’s 360 video feature (all headsets have an option of accessing YouTube 360 videos, provided there is an internet connection). An issue that staff from all facilities encountered was with remote viewing (or casting). They were unable to project the experiences onto an iPad (because all of the facilities had restrictions on the ability of hardware to connect to their WiFi and interact with other hardware). In all locations, staff commented in the post-pilot survey that it would have been very helpful to be able to see what the resident was seeing, and thus help with any troubleshooting and to learn what types of things the resident enjoyed/reacted to. The casting issue was due to certain security measures being in place for network usage. Not being able to cast the video restricted the implementation of our program in its entirety.

Our observations of resident use of the program collected during site visits provided us with interesting insights about the implementation at each site. However, we should keep in mind that, since different research team members recorded this information, there could have been variation in recording between observers. Location B had residents who were more focused on the narrator (40%) than the audio or the visuals (0% and 26.6%, respectively). This site also had the highest number of residents who participated in this phase and they were the most engaged, when compared to the rest of the sites, when it came to listening to and answering the narrator’s questions and reacting to the images. Locations B and C were the only sites where the residents did not react to the music. In general, we can see that residents reacted more strongly to the visuals than the music, suggesting a high level of visual immersion in the virtual environment. It also suggests that the narration and music may be optional; however, further studies must specifically address this. These features were suggested during our initial piloting in Phase 2. However, not all residents were able to focus on the narration and seemed totally immersed in the visuals, in line with research which suggests that older adults have greater difficulty dividing their attention compared to younger adults. Future implementations ought to experiment with versions with and without narration. We do, however, believe that the addition of music was helpful even though residents did not respond greatly to it because it added to the mood of the experience. Finally, we learned from both our observations with residents during our site visits, and the post-pilot staff surveys, that in general, the experiences that had movement were the most enjoyable and popular. Two of the 10 experiences had segments where the camera was on a moving platform (a water taxi in one and a gondola in the other) and these were overwhelmingly the most enjoyable experiences.

Maintenance

Maintenance of a program or technology refers to whether and to what degree it becomes part of routine practice or institutional culture. In the case of our program, the senior management of the partner agency supported the initiative and requested the research team to send the manual and VR experiences, and conduct online training with all of their sites. Once each site receives approval to purchase its own headsets, the program will be implemented organization-wide. We have strong evidence to suggest that the program will be maintained (i.e. the headsets used, and the program incorporated into existing recreations programs). When the research team directly asked residents whether they would do it again, 72.7% (n = 16) said yes they would, 18.2% (n = 4) said maybe and 9.1% (n = 2) said they would not. We have no data on why that was the case. Neither of the participants who said they would not offered an explanation and neither appeared to have any difficulty with the headset or during the experience. This indicates a high potential re-use rate (defined as the percentage of users who use the intervention multiple times). Staff members from most sites were able to correctly show the residents how to use the technology and record their responses. We also learned from our post-pilot staff surveys that staff members were excited about its potential for residents who do not participate in many other recreation programs, either due to cognitive or physical impairment. This indicates that it may become a niche program, developed for those who need recreation and engagement the most. However, as suggested from our implementation data, it is likely that each site will use it differently, and this is one of its strengths. Having a flexible technology, where staff can use both specialized content and content that is available on the internet, aids implementation. Modern long-term care homes have residents with a wide range of backgrounds, interests and abilities. VR programs, such as ours, allow staff to offer person-centred programming to meet the needs of complex and diverse residents. In terms of feedback on how to improve the usefulness of the program, staff across all sites indicated that they would have liked the casting feature to have worked. This would have allowed them to watch on an iPad what the resident was seeing during the experience. This would have improved the ease of troubleshooting and amount of engagement with the resident.

Discussion

Our VR program proved to be highly implementable, based on the data we gathered using the RE-AIM framework as a guide. In this section, we describe what elements we think add to its implementability and how others can improve implementability of VR programs. We will describe these elements using the RE-AIM framework categories, followed by a summary of our study limitations in a separate section.

Our VR program reached men and women of a variety of ages (50–100) and physical difficulties (visual impairment, requirement of supplemental oxygen, requirement of mobility aids). All levels of dexterity, range of motion and strength were included in the current intervention. All residents were English speaking and our VR narration was in English.

Regarding cognitive impairment, through our exit surveys with staff, we learned that some residents with cognitive impairment enjoyed the VR, but the staff did not attempt to use the technology with those with more advanced cognitive impairment because they thought it would be disorienting. Increasing the reach to those with advanced cognitive impairment may be possible once smaller, more lightweight and less intimidating headsets become more available and more affordable. Our findings are similar to those of Baker’s, where they ‘call into question the use of HMD (head-mounted displays) with people living with dementia’. Moyle and colleagues avoided the use of the HMD altogether and displayed a scene of an animated VR forest to people living with dementia on a large projector screen. However, since we were able to conduct initial user testing on prototypes of our VR experiences with people living with cognitive impairment, it is clear that some people with cognitive impairment can tolerate and enjoy the experience. However, there is a need to design both headsets and experiences explicitly for people living with dementia.

Our VR program showed many positive results (enjoyment, engagement, relaxation, pain distraction), and very few negative results (refusal (n = 2), ending the intervention early (n = 1), expressions of fear (n = 1), expressions of sickness (n = 1)). As such, the technology itself has high efficacy. Our findings are similar to those of Baker and Moyle, who both documented high rates of enjoyment and engagement. However, we did note a variation in how residents responded to different experiences, which helps us understand how to maximize enjoyment and efficacy. Residents overwhelmingly preferred experiences that had a moving component. Since moving the camera itself can create motion sickness, putting the camera on a moving object (e.g., a water taxi or a gondola), allows the user to feel like they are ‘going somewhere’ without any adverse effects. In our study, we found that the recommended length of 8–10 min allowed users to experience immersion and maintain attention and focus. This result was similar to the Virtual Reality Forest intervention, but less than Baker’s VR intervention (about 30 min). We received almost no comments about it being too long or too short. The one exception was the particularly static ‘Canada Pacific Railway Last Spike’ experience, which all users thought was too long and uninteresting. As such, the optimal length of the experience may be a function of how interesting, engaging and varied the content is (and the cognitive health of the user).

In our intervention, adoption was fueled by recreation program staff and residents (and to some degree the organization’s leadership, who spearheaded the study). Locations varied by the number of staff who used the technology and how frequently each staff member did so. Our data do not provide clear insight into why this was the case. Staff at each site used the technology, and we hypothesize that this was the case because of its ease of use and staff motivation to use it. Recreation staff members were trained on how to use it and provided the necessary equipment and resources to troubleshoot (the manual).

Adoption of the program also depended on how receptive the residents were. Less than 2% of the time did the resident refuse the headset or end the experience early. Benham and Moyle had similarly high rates of acceptance. Across 3 of the locations (Locations A, C, D), 16–30% of residents found the technology difficult or very difficult to adopt (mostly regarding issues with the headset and headset fit), but none of the residents at the site that had already implemented a VR program (Location B) did. This tells us that adoption may increase over time as residents get more used to the programming and that some initial difficulty with the headset may be expected. We observed that describing the headset as ‘watching a movie, but with goggles on’ and similar metaphors helped residents understand the headset. We also observed that optimal headset fit was important and staff members should spend time to ensure that viewing and comfort are optimized for each resident. A lightweight headset may increase adoption.

A final, but important aspect of adoption is the safety of the headset in terms of hygiene. Since the headset could be a vector of bacteria or pathogens, we supplied each site with Virox™ wipes (antibacterial/antiviral wipes) to apply to the inner part of the headset between each use. We also purchased a leather interface cover to replace the foam interface that the headset comes with. This makes cleaning easier as the foam headset tended to retain moisture from the Virox™ wipes. Although we did not use a UV headset cleaner, this equipment is available and may be a good option for sites to ensure safety and hygiene.

Once the staff members were trained on how to use the technology, they were asked to implement it over a 2-week period. According to Fogg’s behaviour model, behaviour is dependent on ease of use and motivation, and exposure to a trigger (a reminder). According to our post-interview surveys, staff found the technology easy to use. We surmise that motivation for most staff members was high because the program helped them do their jobs (provide recreation to residents) and because the program was supported by upper management. Since ease of use and motivation were both high, the targeted behaviour (using the technology with residents) was achieved. To increase use of the VR program, sites can identify appropriate ‘triggers’ or reminders for staff. It could be putting VR on the recreation calendar (3 of the four sites did this), or leaving the headset in common areas to keep it on top of mind.

Most residents engaged with the technology, either responding to the images, music or narrator. Few residents responded to all 3, indicating that different ‘modes’ of engagement are helpful, to allow each resident to connect in their own way. Residents reacted the least to the music. During our initial user testing, users said that the experiences dragged without the music, so we opted for adding a soft music track to each experience. However, future experiences without the music track can easily be made, and it would be worthwhile to conduct A/B testing to determine whether (or for whom) the addition of music increases the enjoyment and engagement of the experience.

Maintenance is one of the least understood and studied aspects of implementation, and the most difficult to monitor. Although we did not learn about the long-term maintenance of the program, we did collect some early data to indicate how well the program could be maintained. Staff responded with excitement about the program, but all agreed that casting would have improved its value and usefulness. Casting can allow staff to see what the resident is seeing in the VR and engage with them. It can also allow other residents to watch while one resident is in the VR, transforming an individual experience into a group experience. Overall, residents were keen to continue to use the technology – on average over 70% said they would like to use it again. The ultimate success of the maintenance of the project lies with senior leadership. Because their staff and residents have embraced the program, they are keen to implement and support it across all sites. However, as with all new programming and technology, it is a matter of prioritization.

Study Limitations

One of our limitations is that we developed English narratives for the experiences and tested them on English speakers. However, it is very simple to create non-English narration tracks and replace the English ones. We did that for our partner’s final implementation, based on a request from recreation directors at other sites. Such modifications during the pilot would have likely increased reach. In addition, we did not determine what types of head mounted displays and VR experiences work best for people with cognitive impairment. There is a felt sense among staff that the VR experiences we developed were not appropriate for all people living with dementia or other forms of cognitive impairment, and that a specialized content and editing (and delivery – especially regarding the headset) would be appropriate for this population. A targeted approach would have been helpful to determine what good ‘design for dementia’ looks like in a VR development context and would be a valuable avenue for future work.

One of the limitations to enhancing the efficacy of the VR experiences was filming access. Many iconic scenes in Ontario (e.g. professional baseball games, the Zoo, the CN Tower) did not permit us to film in them. As such, residents were not able to revisit these memorable locations. Our focus groups taught us that residents were interested in visiting places they have always wanted to go to, and revisiting places of their youth. In practice, most of our VR experiences offered ‘new’ locations to residents and few revisits. Providing more known VR locations is an opportunity for further enhancement.

Another limitation of our program was that not all recreation staff members were formally trained on the VR technology because not all staff members were on shift when the research team visited the site for training. These staff members were often informally trained by other staff who did receive the training. One solution to this would have been to provide YouTube videos to train those staff who could not attend the in-person training.

We chose to add a low volume music track to the VR experiences, based on feedback from our initial user testing. However, future experiences without the music track can easily be made, and it would be worthwhile to conduct A/B testing to determine whether (or for whom) the addition of music increases the enjoyment and engagement of the experience.

The study sites were chosen by our partners, a company that operates over 30 long-term care homes in Ontario, Canada. This may have led to selection bias because the company may have unconsciously selected sites they thought would successfully implement the technology, and avoided ones they felt would not be able to do so. A follow-up study of the technology implementation at sites deemed least likely to successfully implement the program would further test the VR program’s overall implementabilty. Likewise, there may have been selection bias regarding the selection of the residents themselves. Each site was asked to identify residents that staff felt could use additional recreational programming. The staff may have chosen those residents in better physical and/or cognitive health, and excluded residents who were deemed unlikely to neither accept nor benefit from VR. In addition, since our purpose was to determine whether we could co-create and implement a VR recreation program with our partners, we did not collect detailed health or demographic data on the residents, which limits the ability of other sites to determine whether our findings are generalizable to their setting.

Finally, we were not able to measure long-term maintenance of the program. We hypothesize that ensuring that the casting feature is functioning would increase maintenance because it would allow staff to understand what each resident is responding to, and adjust the VR experience based on this knowledge and understanding. This would enhance staff’s ability to create ‘resident-centred’ experiences and increase continued enjoyment of residents and feedback to staff to encourage continued use.

Conclusion

VR recreational programming provides an opportunity for residents to visit locations they would not otherwise have access to, and to share these experiences with other residents. It is also an opportunity to be exposed to new technologies and to learn about them. In this study, we found that older adults living in long-term care are open to and curious about the technology.

Recreational VR experiences may be the preferred way to introduce VR to older adults in long-term care. We found that with exposure comes enhanced comfort with the technology. As such, introducing location-based VR experiences as an initial program before trying exergaming or VR educational programming may be a way of enhancing the acceptability of other more complex VR programs. We found that 8–10 min VR experiences with narration and music were highly implementable from both a staff and resident perspective. Other key implementation features included: explaining the technology in terms the residents could understand, ensuring good headset fit, providing in-person training and a training manual for staff and having an easy to use interface. The casting feature may increase implementability, as might a wider variety of familiar and meaningful experiences. Sites can enhance the implementability of a VR program by making it easier for staff to remember to use it (providing ‘triggers’ or reminders). Finally, as the headset technology advances and more lightweight and less obtrusive headsets become more affordable, the technology will become more and more acceptable to all residents.