Development and psychometric testing of the 5Senses screening tool for long-term care: a study protocol.

INTRODUCTION: As adults age, their senses tend to decline and a large portion of those most affected by sensory decline reside in long-term care. At present, the creation of a sense-sensitive environment in long-term care is a difficult task as there is minimal evidence or tools available to guide this process. The 5Senses screening tool was developed to measure the sense-sensitivity of a particular environment, with a focus on long-term care. The purpose of this paper is to describe a study protocol to assess the psychometric properties of the newly developed 5Senses screening tool. METHODS AND ANALYSIS: We will conduct a psychometric evaluation of the 5Senses screening tool in long-term care based on the Standards for Educational and Psychological Testing Framework. In phase I, we will seek input from international content experts (n=20) to assess the content validity of all sections of the tool. In phase II, we will invite auditors (n=3-9), residents (n=3-9) and staff (n=3-9) to partake in think-aloud sessions to assess response process validity. In phase III, we will conduct field testing of the revised 5Senses screening tool with auditors (n=100), residents (n=100) and staff (n=100) to evaluate additional measures including acceptability, inter-rater reliability, internal structure validity and internal consistency reliability, where possible. ETHICS AND DISSEMINATION: Ethical approval was obtained from the University of Ottawa Research Ethics Board. Findings will be disseminated through a peer-reviewed manuscript, through a dedicated website, through presentations in long-term care communities and through presentations at research conferences. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The psychometric methods used in this study will ensure valid and reliable research findings regarding the sense-sensitivity of long-term care facilities.

These methods will help further refine the screening tool and inform future studies to conduct further testing in other settings.

This study may allow researchers to study the impact of the sense-sensitive environment on resident outcomes.

The measure of ‘relation(s) to other variables’ will not be evaluated in this study because we found no other tools that measure the five senses constructs.

Very few items on our tool are supported by randomised controlled trials and therefore most items are based on untested empirical evidence. Further testing of these items are needed.

Background

The five senses (ie, hearing, vision, smell, taste and touch) are one of the key means by which individuals interpret and interact with the surrounding environment. Sensory abilities diminish with age. Around a third of those over the age of 65 have disabling hearing loss.1 Serious vision impairment is also more prevalent with age, with 81% of all cases seen in those 50 years or older.2 The sense of touch deteriorates at a rapid pace, with around 1% of tactile sensation lost per year from 18 years onward.3 Excluding the many cases of chemosensory disorders (eg, loss of taste or loss of smell) that go unreported due to the subtle nature of the problem, age-related sensory impairments are increasing globally.4 Considering that a large portion of those affected by sensory decline reside in long-term care (LTC) settings, the design of these facilities to accommodate these deficiencies is a topic worth exploring. At present, the creation of a sense-sensitive environment within LTC is a difficult task given that there is minimal evidence or tools available to guide this process.

A recent systematic review by Elf et al 5 found that the instruments (n=23) developed to assess the physical environment in healthcare varied in their comprehensiveness of psychometric testing. Although several instruments were developed to assess the physical environment in LTC facilities,5–9 none have looked comprehensively at the five senses. For example, both the Therapeutic Environment Screening Survey for Nursing Homes7 and the Environmental Audit Tool8 have some items related to the five senses (ie, odour, lighting), however, these instruments are observational tools and do not include an all-inclusive list of items related to each of the five senses. More recently, another tool called Dining Environment Audit Protocol9 incorporates some important items related to the five senses, however, it is specifically focused on the dinning environments in LTC facilities. With the increased need for an age friendly environment in LTC facilities, an in-depth focus on assessing the five senses is essential and this focus must include considerations about the environment, the perspectives of staff and most importantly, the perspectives of residents.

LTC settings that support and adopt interventions to enhance sensory capacities will excel in creating a stimulating and enriching environment for their residents. The 5Senses screening tool was designed to assess current sense-sensitive practices and also to serve as a tool to help identify opportunities for improvement in any LTC setting regardless of its current state. The purpose of this paper is to describe a study protocol to assess the psychometric properties of the newly developed 5Senses screening tool.

Development of the 5Senses screening tool

The development of the 5Senses screening tool was guided by ecological model of Lawton and Nahemow.10 Older adults are particularly sensitive to their environment as a result of the increasing number of sensory impairments that result from old age.10 According to the ecological model, in order to maintain independence and quality of life, an older adult’s capacity and demands on the environment must be in congruence.10 Lawton et al argue that the physical environment has the potential to enhance or constrain an individual as they age.11 Therefore, the physical environment must be adjusted to suit the needs of individuals in the later stages of their life.10

The 5Senses screening tool was developed to measure the sense-sensitivity of a particular environment, with a focus on LTC. The practices identified in the screening tool are based on the results of a systematic review.12 In this review, we collated all current knowledge examining the relationship between the sensory practices and the quality of life of residents living in LTC settings. The search strategy yielded 4166 articles of which 123 articles were eligible for full-text screening. A total of 38 studies were included in the final data extraction. The most common interventions reported were related to: touch (n=15), vision (n=10), smell (n=2), hearing (n=2) and more than one sense (n=9). No studies were found on taste (n=0). Examples of interventions reported in the included studies were: sound and music, multisensory environments, aromatherapy, therapeutic touch, pet therapy, massage, lighting therapy, gardens and use of art or pictures. The results of this systematic review were used to assist with the development of the 5Senses screening tool.

The key principles of the tool were that it had to (1) be comprehensive in scope, (2) provide simple ways to action and prioritise the gaps discovered and (3) be user friendly. The 5Senses screening tool assesses the sense-sensitivity of the environment, and captures the related organisational policies and procedures as well as resident and staff perspectives about the sense-sensitivity of the environment. The 5Senses screening tool guides the user through the following steps: a walkthrough of the environment (section 1—facility level), a review of the policies and procedures (section 2—facility level), an assessment of resident perspectives (section 3—resident version) and an assessment of staff perspectives (section 4—staff version). The results from these steps are then used to determine the level of sense-sensitivity of the environment.

An initial pretest of the 5Senses screening tool was conducted in two (n=2) LTC facilities in the USA. The purpose of the pretest was to determine preliminary content validity of the tool, to determine ease of administration, to identify any items that were not worded clearly, to determine adequacy of instructions and to determine the general flow of the tool. As part of the initial pretest, three (n=3) focus groups were conducted in each of the two (n=2) facilities. Participants were recruited from each of the LTC facilities for the pretest. The first being located in New Paltz, New York, and the second located in Collingswood, New Jersey. Focus groups were organised according to roles, with the first group consisting of administrators (including directors of care and facility managers), the second group consisting of care staff and support staff (including registered nurses, allied health professionals, personal support workers, activity organisers, catering providers, cleaning staff and maintenance workers) and the third group consisting of residents and families. A total of 45 participants contributed to the focus groups. No inclusion or exclusion criteria were applied for participation in the focus groups and involvement was voluntary. Participants worked in a variety of roles within the LTC facility and had worked in these roles for varying lengths of time, which enhanced the data collected during the pretest.

Participants were given the opportunity to review the tool in advance of the focus groups to allow them time to examine the document. The pretesting identified item wording and terminology that needed further clarification as well as items that should have been included in the tool, adequacy of instructions and the general flow of the tool. Participants were also asked to consider the logistics of completing the tool, including the time to perform the audit and who should be involved in the process. Based on feedback obtained in the pretest, revisions were made to the tool. The content and proposed scoring for each section of the tool are listed in table 1.

Table 1

5Senses screening tool sample content and proposed scoring

Domains	Section 1: walkthrough of the environment—items	Section 2: assessment of organisational policies, procedures and protocols—items	Scoring for sections 1 and 2	Section 3: assessment of resident perspectives—items	Section 4: assessment of staff perspectives—items	Scaling for section 3 and 4
Hearing	Quiet areas separated from noise	Evidence of staff training about hearing impairments	(0) No (1) Yes	Appropriate noise level during activities	Appropriate noise level during activities	Frequency scale: (1) Never (0) (2) Rarely (1–2 times) (3) Occasionally (3–4 times) (4) Frequently (5–6 times) (5) Almost always (7 times or more)
	Housekeeping schedule	Policies related to hearing (sound, noise, hearing impairment, communication with impaired residents)	(0) No (1) Yes	Appropriate noise level throughout the day	Appropriate noise level throughout the day	Frequency scale: (1) Never (0) (2) Rarely (1–2 times) (3) Occasionally (3–4 times) (4) Frequently (5–6 times) (5) Almost always (7 times or more)
Vision	Reflective materials easy to clean (y/n)	Staff training about visual impairments (y/n)	(0) No (1) Yes	Lighting levels appropriate	Lighting levels appropriate	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
	Minimum glare flooring (y/n)	Policies related to vision (lighting, visual impairments, communication with visual impaired residents) (y/n)	(0) No (1) Yes	Tidiness	Tidiness	Frequency scale: (1) Never (0) (2) Rarely (1–2 times) (3) Occasionally (3–4 times) (4) Frequently (5–6 times) (5) Almost always (7 times or more)
Smell	Containment of odours (y/n)	Staff training regarding odour control (y/n)	(0) No (1) Yes	Positive smells in the environment	Positive smells in the environment	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
	Management of unpleasant odours (y/n)	Policies related to management of odours (cleaning products, housekeeping, soiled linens and other item, garbage disposal) (y/n)	(0) No (1) Yes	Handling of negative odours	Handling of negative odours	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
Touch	Sensory rooms available (y/n)	Staff training regarding use of touch (y/n)	(0) No (1) Yes	Appropriate use of touch therapeutically (5-point scale)	Appropriate use of touch therapeutically (5-point scale)	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
	Tactile experiences available (y/n)	Policies related to touch (use of sensory rooms) (y/n)	(0) No (1) Yes	Feelings about touch use (5-point scale)	Feelings about touch use (5-point scale)	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
Taste	Variety of herbs and spices used to stimulate appetite (y/n)	Staff training to improve taste experience (y/n)	(0) No (1) Yes	Food tastes great	Resident enjoyment of food	Likert agreement: (1) Strongly disagree (2) Disagree (3) Neither agree or disagree (4) Agree (5) Strongly agree
	Kitchen or serving area kept open to allow residents to smell food and help stimulate appetite (y/n)	Policies related to taste (residents' food preferences, needs, cultural choices, oral health) (y/n)	(0) No (1) Yes	Dental/oral care needs are met	Dental/oral care needs are met	Frequency scale: (1) Never (0) (2) Rarely (1–2 times) (3) Occasionally (3–4 times) (4) Frequently (5–6 times) (5) Almost always (7 times or more)

Proposed scoring

In sections 1 and 2, each sensory item is scored in a binary fashion with ‘yes’ answers being assigned a value of ‘1’ and ‘no’ answers being assigned a value of ‘0’. In sections 3 and 4, each sensory item is scored using a Likert scale ranging from 1 to 5. For individual item scoring, we will calculate the mean for each item. For domain scoring (if feasible), we propose to calculate the ‘mean of means’ for each domain score. The proposed scoring will be further refined based on the psychometric findings (ie, factor analysis, item total) from this proposed study.

Methods

Beginning in March 2019, we will conduct psychometric testing of the 5Senses screening tool in three phases based on the Standards for Educational and Psychological Testing Framework.13–15 In phase I, we will assess the content validity of all sections of the tool (sections 1–4). In phase II, we will assess response process validity (sections 1–4). In phase III, we will evaluate acceptability, and inter-rater reliability (sections 1–2), and we will evaluate acceptability, internal structure validity and internal consistency reliability (sections 3–4).

Resident and public Involvement

Residents were not involved in the development of the protocol.

Phase I: content validity

Content validity refers to the degree to which the items on the tool represent the content of interest.16–18 International content experts from LTC will be invited to evaluate the 5Senses screening tool for relevance, clarity and missing items through an online survey.

Settings and participants

We will seek international representation from key experts (n=20) comprising the following: (1) administrators, directors of nursing, facilities, dietary services and researchers; (2) residents including advisors on resident and family councils and (3) staff working in LTC for >2 years. It is recommended to have a minimum of five experts for content validity testing.19

A list of experts obtained from existing contacts and from an internet search of international relevant organisations will be compiled including, but not limited to the Global Ageing Network, the International Longevity Centre Global and the WHO Department of Ageing and Life-Course. Participants will be invited by email to participate and will provide implied consent by completing and submitting the online survey.

Procedures

The experts will be asked through the online survey to evaluate and comment on the relevance of each item of the 5Senses screening tool in relation to their experiences of sense-sensitivity in LTC. Specifically, experts will be asked to rate the relevance of each item on a Likert scale: 1 (not relevant); 2 (item needs some revision); 3 (relevant but needs minor revision); and 4 (very relevant) as well as provide an explanation for their decision.16 Experts will also be able to add their suggestions on any missing items.

Data analysis

Each item rating on the tool will be averaged. We will also calculate the Scale-Content Validity Index for each domain. Based on the feedback from the experts, we will revise the tool as necessary, prior to phase II.

Phase II: response process validity

Response process validity involves verifying the fit between each question on the tool and the responses provided by the participants.16–18 We will conduct think-aloud sessions in an iterative fashion to obtain verbal feedback from participants on the proposed workflow and their experiences with the 5Senses screening tool.20 This approach will aid in understanding whether participants are interpreting the tool the way it was intended, which helps to ensure that participants interpret and respond to survey items in the manner intended by the researchers.

We will conduct two to three rounds of think-aloud sessions with LTC facilities in Ottawa, Canada using a convenience sample. Participants will consist of auditors/designated leads from the LTC communities (n=3-9) for sections 1 and 2, residents (n=3-9) for section 3 and staff (n=3-9) for section 4. We will recruit participants through our primary contact in each of the LTC facilities. The inclusion criteria will include: (1) auditors: administrators, directors of nursing, facilities and dietary services who work full-time or part-time at one of the facilities; (2) residents including advisors on resident and family councils and (3) staff working in LTC for >2 years. Participants will need to be fluent in English. The research assistant will provide information about the study and obtain informed consent from the potential participants.

Participants will be invited to participate in 60–90 min individual audio-recorded think-aloud sessions, which consists of going through the tool while explaining his or her interpretation of each question on the tool. We will make revisions following the first round of think-aloud sessions, and proceed to the second and/or third round in an iterative manner until no further changes are required.

The think-aloud sessions will be recorded and transcribed verbatim. Following each round, two researchers will independently analyse the transcripts using content analysis to provide a comprehensive and accurate descriptive summary of the participants’ perspectives.21 Discrepancies will be reviewed and resolved by a third researcher. Findings from the content analysis will then be used to revise the tool between each round and for the field testing.

Phase III: additional psychometric testing

We will conduct field testing to collect evidence on: (1) acceptability which refers to the ease of use of the tool17 (sections 1–4); (2) inter-rater reliability which refers to the likelihood that an audit conducted by two individuals will produce the same results (sections 1 and 2); (3) internal consistency reliability which refers to the correlation between the items that make up the overall score for that domain16–18 (sections 3 and 4) and (4) internal structure validity which examines the relationships between groups of items16–18 (sections 3 and 4).

Approximately 50 LTC communities will be invited by email to participate. We will select LTC communities of different sizes (small <25 beds, medium (26–100 beds) and large >100 beds), which provide 24 hours nursing care with the help of our international experts. From each of these LTC communities, we will recruit approximately two auditors (n=100) for sections 1 and 2, two residents (n=100) for section 3 and two staff (n=100) for section 4. For psychometric testing studies, it is recommended that approximately 5–10 participants be recruited per item on the tool22 23 with a minimum of 100 participants for best results.22

For this phase, the inclusion criteria will be the same as in phase II with the addition of participants being able to access the tool electronically through a weblink. Participants in each LTC community will be approached by the LTC administrator or delegate, and those who express interest in participating will be contacted by the research assistant to who will provide further information about the study and obtain informed consent.

For sections 1 and 2 of the 5Senses screening tool, a training manual and a set of training slides will be created and distributed to all study sites. The training will consist of viewing the training slides as well as a guided discussion on individual questions and response categories. All auditors will be trained prior to the start of data collection. Two auditors will independently and concurrently complete the tool in each LTC facility.

For sections 3 (residents) and 4 (staff), residents and staff will be provided with the weblink to complete their respective survey online. All data will be collected using survey software (SurveyMonkey, San Mateo, California, USA, http://www.surveymonkey.com). We will seek institutional review board approval at each site (if applicable).

Descriptive statistics (ie, frequency, mean, SD, maximum, minimum and mode) will be used. For acceptability, we will examine the frequency of missing data. We will also assess the time it takes to complete each component of the tool.16–18 For any missing data, we will also follow-up directly with the auditors to gain more insight into why items were not completed.

For reliability, we will use the following tests to examine inter-rater reliability for categorical items: (1) percentage of agreement and (2) weighted kappa.17 To assess internal consistency reliability, we will use the following coefficients, or estimates of between-score correlation: (1) Cronbach’s α for randomly equivalent measures, (2) Guttman split-half for parallel measures and (3) Spearman-Brown for any changes in the number of items on the tool.17 Internal consistency coefficients can range from 0 to 1. A coefficient of 0.70 is acceptable for newly developed scales such as the present tool under study.24

For internal structure validity, we will conduct an exploratory assessment rather than confirmatory in nature since this will be the first field assessment of the 5Senses screening tool. Therefore, to examine the underlying dimensional structure of the multi-item domains contained in the tool, we will conduct: (1) item to total correlations, (2) item-total statistics and (3) principal component analysis (PCA). Any missing values will be treated as such with no substitution or imputation. From the item to total correlations, any item that correlates with its scale (domain) score below 0.30 will be discussed and revised.25 From the item-total statistics, items that, if removed, cause a substantial change in the domain’s Cronbach’s α score will also be discussed and flagged for potential revision. For the PCA, a Varimax rotation with Kaiser normalisation will used to enhance its interpretability. Factors (domains) will be identified using the standard 1.0 eigenvalue cut-off rule and visualisation of scree plots. Items with factor loadings ≥0.35 will be retained.26 Items that cross load (ie, factor coefficients ≥0.35 on two or more factors) will be flagged for discussion and possible revision.

Ethics and dissemination

Ethical approval was obtained from the University of Ottawa Research Ethics Board. Findings will be disseminated through a peer-reviewed manuscript, through a dedicated website, through presentations in LTC communities and through presentations at research conferences.

Discussion

This paper presents the protocol for the psychometric testing of the newly developed 5Senses screening tool. The psychometric assessment of the 5Senses screening tool will help to ensure that it provides valid and reliable research findings regarding the sense-sensitivity of LTC facilities. The use of this tool will provide key information about the practices, or lack of practices that exist related to the five senses. The findings from the audits will highlight gaps in the sense-sensitive environment of LTC settings and will guide organisations to improve their sense-sensitivity. Further research on the effects of the five senses on quality of life is needed.

During our literature review, we found no other tools that measure the 5Senses environmental constructs, making it impossible to evaluate the tool on the basis of its ‘relation(s) to other variables’, as this metric is aimed at comparing the current 5Senses screening tool against a similar tool which has demonstrated reliable and valid results. Also, very few items on our tool are supported by randomised controlled trials and therefore most items are based on untested empirical evidence. Further testing of these items are needed. We anticipate that this tool will be useful as a stand-alone tool as well as in conjunction with other environmental assessments.

Furthermore, this screening tool is meant to provide a general measure of the sense-sensitivity in the environment, and may not consider every resident’s individual preferences in what they consider a stimulating environment. However, LTC communities can chose to administer the resident and staff sections of the tool to all or to a large sample of their residents and staff to obtain the overall perspectives on the sense-sensitivity of the environment. Additional qualitative information may also need to be gathered before any significant changes are made to the environment. Agitated people or people with advanced dementia may be better served in a peaceful environment with fewer stimuli. The resident and staff sections of the tool allow the possibility to gather data from all residents and all staff in the home in order to capture the different perspectives and needs for that particular LTC community.