Evaluating the validity, reliability and clinical utility of the Music therapy Sensory Instrument for Cognition, Consciousness and Awareness (MuSICCA): protocol of a validation study.

INTRODUCTION: A growing number of children and young people are surviving severe acquired brain injuries due to advances in healthcare. However, many fail to emerge from coma and continue to live with disorders of consciousness (DOC). Diagnostic, clinical and ethical challenges are prominent in this group. Misdiagnosis can have severe consequences for children and their families, including inadequate care, insufficient access to rehabilitation and stimulation, reduced accessibility to services and inappropriately limited opportunities for participation. The proposed project will develop and validate a diagnostic measure that supports detailed goal-planning-the Music therapy Sensory Instrument for Cognition, Consciousness and Awareness (MuSICCA). METHODS AND ANALYSIS: Face validity will be assessed using a short questionnaire and the MuSICCA will be amended if face validity is insufficient. Once face validity is sufficient, 80 participants with suspected DOC will be recruited from multiple sites around the UK, USA and Ireland.Validity will be assessed using external reference standards (Coma Recovery Scale-Revised, Coma Near-Coma Scale and Nociception Coma Scale). Intra-rater reliability will be established using repeated ratings of video recordings from the assessment sessions. Inter-rater reliability will be assessed through video ratings by a second blinded assessor. In addition to these analyses, the clinical utility of the MuSICCA will be evaluated using a questionnaire to be completed by clinicians and relatives of the participants following the completion of the MuSICCA assessment. ETHICS AND DISSEMINATION: Ethical approval has been obtained for this study from the Research Ethics Committee and Health Research Authority of the National Health Service of the UK (ID: 167534). Results will be presented at national and international conferences, published in scientific journals and disseminated to participant representatives, clinicians, educators and care providers. TRIAL REGISTRATION DETAILS: This study was registered at ClinicalTrials.gov Protocol Registration and Results System on 7th August 2019 (ID: NCT04050995); Pre-results. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study compares the Music therapy Sensory Instrument for Cognition, Consciousness and Awareness (MuSICCA) with three outcome measures, increasing its strength in determining validity.

The MuSICCA has been developed through consultation with psychologists, speech and language therapists, occupational therapists, music therapists and families of children and young people with disorders of consciousness (DOC). Thus, the measure incorporates multiple perspectives.

This study involves blinded raters.

Recruitment at multiple international sites will enhance generalisability.

There is no existing validated paediatric DOC measure for use as the external reference standard.

Introduction

Disorders of consciousness (DOC) include vegetative state (VS), also known as unresponsive wakefulness syndrome, and minimally conscious state (MCS).1 It is a complex population where measuring responsiveness is a topic of considerable interest, generating a range of behavioural measures that have been validated to assess awareness.2 Assessing DOC in children and young people is a particular challenge due to developmental factors, where the domains of language, cognition and motor ability vary substantially both within and across age groups thus complicating accurate measurement and appropriate outcomes. In addition, the reported incidence of visual impairment in children who have acquired brain injury is high.3 4 Consequently, there is a lack of research with this population,5 in part because no DOC measures have been validated specifically for use with children.6 Further research on this complex population is required to improve our understanding of prognosis and recovery, to address concerns regarding accurate diagnosis and to develop appropriate treatments.5

Language-based assessments and interventions with DOC are not the most appropriate for stimulating optimal responsiveness as the severity of brain damage incurred typically results in language disorders.7 This is a particular problem with children where language may not be fully developed. Music may provide an alternative medium for assessment and intervention with this population, given the fact that it does not rely on language and its ubiquitous use in education and leisure for children.8 Motivation in assessments and rehabilitation can be challenging when working with children and young people. Music seems to increase arousal9 and enhance cognition10 in adult patients with DOC. With its motivational elements and developmental relevance, music may provide an optimal medium for working with children with DOC who are minimally responsive. Although the evidence for music therapy with the DOC population is still limited,11 the development of psychometrically robust tools is an essential step in building a stronger evidence base. This will enable further understanding of the responses of paediatric DOC patients to music interventions.

The Music therapy Assessment Tool for Awareness in Disorders Of Consciousness (MATADOC) is a validated music-based assessment measure for use with adult DOC. Its principal subscale has demonstrated good inter-rater reliability (mean=0.83, SD=0.11), good test–retest reliability (mean=0.82, SD=0.05), with good internal consistency (α=0.76) and a strong first principal component.12 The MATADOC protocol uses a range of music stimuli that are presented live, with particular emphasis on emotionally salient stimuli such as singing the patient’s name as well as personally meaningful songs.13 It has shown diagnostic utility in comparison to another standardised DOC measure, the Sensory Modality Rehabilitation and Assessment Technique (SMART).12 Exploring its utility for children with DOC, a pilot study indicated that the MATADOC provided a useful protocol and measure for behavioural assessment and clinical treatment planning with paediatric DOC.6 Musical stimuli produced responses in the visual and auditory domains suggestive of higher-level functioning when compared with other DOC measures. This suggests that musical stimuli might be effective in maximising the potential to demonstrate awareness and assess consciousness in children and young people with DOC.

There is a significant need for useful protocols and a valid and reliable assessment measure for children with DOC.5 This is especially important because the consequences of misdiagnosis include insufficient care provision, unsuitable treatment programmes, poor identification of intentions to communicate and insufficient evidence for making difficult decisions around withdrawal of hydration and nutrition and instructions for ‘do not resuscitate’.5 14 The authors have developed a paediatric version of the MATADOC. This version is called the Music therapy Sensory Instrument for Cognition, Consciousness and Awareness (MuSICCA).

Expanding on the MATADOC protocol, the MuSICCA protocol involves a minimum of six tasks using live (and sometimes recorded) music with a pre-observation and post-observation period of 3 min each. The child is involved in listening and, where possible, vocalising and/or making sounds or choices of music to be played by the therapist. Responses are rated across the motor, visual, auditory, communication and arousal domains in 15 items. This behavioural assessment is intended to provide an indicative diagnosis for children with DOC and will offer detailed information about their responses to stimuli. This information will help to address the issues that result from misdiagnosis.

Objectives

This research study seeks to evaluate the validity, inter-rater and test–retest reliability and clinical utility of the MuSICCA. In order to test for validity, the MuSICCA will be compared with the Coma Recovery Scale-Revised (CRS-R)15 and the Coma Near-Coma Scale (CNC) (for convergent validity)16 and to the Nociception Coma Scale-Revised (NCS-R) (for discriminant validity).17 These comparisons will enable:

Comparison of diagnostic outcomes.

Comparison of diagnostic measures on subscale level (auditory, visual and other matching subscales). Each measure must be completed in its entirety to calculate a diagnostic result.

Additionally, the minimum number of sessions required for a diagnosis will be investigated by comparing outcomes from four sessions with those from two and three sessions. This will help to determine whether the MuSICCA could be used to efficiently and accurately track the changing level of awareness of a child or young person with a disorder of consciousness. Thus, the MuSICCA could be recommended for use in research studies and in a wider variety of clinical environments, including acute settings.

Hypotheses

This study intends to test the following hypotheses:

The MuSICCA will demonstrate acceptable inter-rater and test–retest reliability when used by trained assessors.

There will be a high level of agreement between diagnostic outcomes on the MuSICCA, the CRS-R and the CNC in paediatric patients with DOC.

The MuSICCA will demonstrate greater sensitivity than the CRS-R and CNC in detecting:

responses to visual stimuli,

responses to auditory stimuli,

responses to musical stimuli,

motor responses.

There will be no detectable correlation in this study between the NCS-R and the MuSICCA in paediatric patients with DOC, indicating discriminant validity.

The number of sessions required for an accurate diagnosis will be less than four.

Methods and analysis

A validation study will be conducted to evaluate the validity, reliability and clinical utility of the MuSICCA. It will involve five phases: face validity; participant enrolment; data collection for validity; data collection for reliability and clinical utility; and data analysis.

Phase 1: face validity

Face validity will be assessed by selecting assessors trained in the MATADOC who are experienced in work with children and young people with DOC (n=10), families of children and young people with DOC (n=5) and healthcare professionals (n=5). Families and other healthcare professionals will be recruited for this phase from a specialist paediatric rehabilitation centre in the UK. These various groups will be invited to review the MuSICCA video demonstration and provide feedback about its face validity using a short questionnaire. They will be asked to submit their answers via email, mail or face-to-face meeting. In each case the responses will be recorded using the Questionnaire for phase 1. If the results from the face validity analysis show that the MuSICCA has sufficient face validity (at least 75% agreement), then the study will progress into phase 2. If 75% agreement is not reached, adaptations will be made and the MuSICCA will be reassessed for face validity. This process will be repeated until the MuSICCA is found to reach sufficient agreement of its face validity before progressing to phase 2.

Phase 2: enrolment

Sample size

The statistical consequences of the sample size have been determined considering existing studies with DOC populations. The validation of the CRS-R, which is considered the gold standard for DOC, used a sample of n=80.15 The simulation assumes that 80 children are each seen four times by a single rater, who assigns a score derived from the diagnostic items of the MuSICCA. This research study is informed by a similar project with adults12 in which the two raters are assumed to have an (unknown) inter-rater reliability ICC coefficient of 0.83 when using this scale. Based on this scenario, the simulations suggest that the study design will be able to estimate this ICC value with 95% CIs having a width of about 0.079 on either side of the estimate. To this, it is then possible to apply a conservative design effect of approximately 1.2. This will allow for aspects of the study design that might reduce the efficiency of our estimation, for example, variance in assessor experience and variance in health status of the participants. Thus, the study will estimate the inter-rater reliability ICC coefficient to an accuracy represented by 95% CIs of 0.094 either side of the empirical estimate.

Participants and recruitment

The study will invite participants aged 2 to 18 years, who the medical team suspect of having a DOC, from various medical facilities in the UK, USA and Ireland. This convenience sample will be recruited over 3 years from September 2019.

All families of the children and young people will be informed about the research and consent will be sought from the children’s parents (or those with guardian status) by the principal investigator at each site to include them in the study. If the families give consent, this will be recorded on an approved consent form. The families will be given every opportunity to ask questions and will be informed about the withdrawal process and their right to withdraw at any time during the study.

Participants will be selected for inclusion in the study according to the following inclusion and exclusion criteria:

​Inclusion criteria:

Between 2 to 18 years of age.

Assessed by the interdisciplinary team as having a disorder of consciousness.

Be under active investigation for diagnosis of DOC.

​Exclusion criteria:

Known hearing impairments.

Known profound visual impairments.

Known musicogenic epilepsy.

Seizure disorders that cause frequent and/or prolonged seizures.

English not first language.

Phase 3: data collection – part 1

Data collection for phases 3 and 4 will occur over 3 years from September 2019. The following measures will be used:

MuSICCA: This will be conducted live by trained assessors over four sessions (30 min each) within a 10-day period for each participant. This involves presenting musical stimuli in three modalities (vision, auditory and tactile). The participants will be asked to respond to stimuli and commands. The MuSICCA uses a detailed protocol for its administration and scoring. Its outputs result in an indicated diagnosis of either VS, MCS or emergence from MCS.

CRS-R: This will be carried out by trained assessors over one session (25 min) for each participant. The CRS-R is comprised of six subscales addressing auditory, visual, motor, oromotor, communication and arousal functions. The CRS-R is the external reference standard and is considered to be the ‘gold standard’ in the absence of a more valid and reliable alternative. The CRS-R uses a detailed protocol for its administration and scoring, and results in diagnostic categories: VS, MCS and emergence from MCS.

The CNC: This will be carried out over one session (10 min) for each participant. It comprises of 11 items addressing auditory, command responsivity, visual, threat, olfactory, tactile, pain and vocalisation. This measure uses a simple protocol for administration and scoring. It results in five categories: no coma, near coma, moderate coma, marked coma and extreme coma.

The NCS-R: This is a measure of pain response and will be carried out during routine care duties. This measure has a simple protocol for administration and scoring and results in a total with a range of 0 to 9. A score of 4 to 9 suggests the presence of pain.

The CRS-R, CNC Scale and NCS-R will be conducted within the period of the MuSICCA assessment sessions. There will be a separate assessor for each measure, that is, one for the MuSICCA, one for the CRS-R, one for the CNC Scale and one for the NCS-R. Thus, four assessors from each site will be required to carry out the data collection in phase 3.

Assessors

The MuSICCA assessors will be trained and certified in the use of the MuSICCA. The assessors of the other measures (CRS-R, CNC Scale and NCS-R) will be provided with training in the use of those measures. Adherence to the protocols and standards in scoring will be checked at this stage.

Paper records will be used for all measures for data collection, and all MuSICCA sessions will be recorded using audio-visual equipment. (A secure, encrypted file repository will be used to store the video recordings for testing reliability in phase 4.) The assessors will conduct each assessment and record data according to the standard clinical assessment protocol set out in the manual for each assessment. Each assessor will be blind to the findings of the other assessors and they will need to submit their data for collation prior to clinical discussions about their findings. This is not expected to interfere with clinical practice and it is anticipated that the use of multiple measures will provide more accurate and comprehensive information about the patients’ awareness and consciousness. When the assessments in this phase have been completed, the MuSICCA assessors will share findings and recommendations with colleagues and families according to standard clinical practice. Then, the assessors and participant’s families will complete clinical utility questionnaires. The principal investigator at each site will then anonymise the data and submit de-identified copies of the assessment forms and questionnaires for data analysis. The principal investigators will be responsible for monitoring and auditing data collection at each site.

Phase 4: data collection – part 2

The same assessor will review and re-rate a single assessment session randomly selected from the four recordings of sessions that he/she conducted live, 4 to 6 weeks following the live sessions. The same audio-visual recording will be reviewed by a second (blinded) assessor. He/she will provide ratings of the recorded session according to the MuSICCA rating forms. This assessor will be blind to the patient’s diagnostic outcome and to each other’s scoring. These ratings will then be submitted for data analysis.

Phase 5: data analysis

This phase will involve assessing the validity, reliability and clinical utility of the MuSICCA.

Statistical methods will be used to test the hypotheses given earlier. When final results are compiled, they will be summarised with the tables of descriptive statistics and the graphical plots where useful. Numerical outcomes will be statistically analysed using Generalisability (G) theory18–20 to assess validity and reliability by quantifying all sources of variation. The specific statistical models underlying the application of G theory will include random effects to account for correlations due to repeated measures and clustering.

For data missing due to withdrawal, the missing at random (MAR) assumption will be assessed using a logistic regression model, which will compare missing versus observed cases on key baseline demographic variables. With the MAR assumption, mixed models can account for missing observations among repeated measures. Covariate values missing due to non-response will be handled using the technique of multiple imputation by chained equations.

Figure 1 illustrates the design of the study and shows the CONSORT (Consolidated Standards of Reporting Trials) flow chart of data and subjects through the various stages described above.

Figure 1

Flow diagram of study design. A1L1 represents data from assessor A1 conducting the first live assessment; A1L2 represents data from assessor A1 conducting a further live assessment (randomly selected from sessions 2, 3 or 4); A1V represents assessor A1 conducting video analysis of his/her own session after 4 to 6 weeks; A2V represents assessor A2 conducting video analysis of a MuSICCA session that was conducted by assessor A1. CRS-R, Coma Recovery Scale-Revised; DOC, disorders of consciousness;MuSICCA, Music therapy Sensory Instrument for Cognition, Consciousnessand Awareness; NCS-R, Nociception Coma Scale-Revised.

Data collection will be monitored for fidelity and completeness throughout the study by Dr Richard Siegert, Dr Wendy Magee and Dr Jonathan Pool. Access to the final trial data set will be limited exclusively to these researchers.

This proposal outlines the background and design of a validation study examining the validity, reliability and clinical utility of the MuSICCA for use with children and young people (aged 2 to 18 years) with disorders of consciousness. If the MuSICCA is assessed as valid, reliable and clinically useful, it will be possible to recommend this as validated for use with children and young people with DOC. It will then have the potential to provide benefits to this population and the families affected by DOC. The findings from the MuSICCA assessments will aid clinicians, medical personnel, educators and care teams in providing optimum support, care, education and therapy for these children. It will also inform medico-legal decisions about the hydration and nutrition of these children and young people.

This measure may offer a significant development for the care and rehabilitation of children and young people with DOC. Given the role of music in typical child development, the systematic use of music within a detailed assessment protocol, such as the MuSICCA, may improve the assessment, treatment and care of children with DOC.8

The Standards for Reporting of Diagnostic Accuracy Studies (STARD) checklist was used when writing this protocol report.21

Patient and public involvement

The development and design of the MuSICCA protocol and documentation for this study have involved informal consultation with families of children with DOC, and also formally with music therapists, physiotherapists, speech and language therapists, occupational therapists and psychologists working with this population. The families of children with DOC were introduced to the protocol and documentation during routine clinical duties. Their opinions and preferences shaped the MuSICCA through their verbal feedback. Patients were not involved in the design of the study. The results from this study will be disseminated to the participants through reports written in plain language.

Ethics and dissemination

Ethical approval has been granted from the Research Ethics Committee (REC) and Health Research Authority (HRA) of the National Health Service of the UK (ID: 167534). Protocol modifications will be communicated and approval requested for them through the REC and HRA.

Findings from this study will be disseminated through presentations nationally and internationally to participant groups and conference delegates, and through future publications in international peer-reviewed journals.

Data sharing

Following completion of the study, de-identified, anonymised data (with participant consent) will be available from the corresponding author. Data made available will include individual-level de-identified, anonymised participant data, excluding audio-visual recordings from the MuSICCA sessions.

Relevance and benefit to society

This proposal outlines the background and design of a validation study examining the validity, reliability and clinical utility of a new measure for use with children and young people (aged 2 to 18 years) with DOC. A validated tool of this type will inform the provision of care, therapy, rehabilitation, education and medical treatment for this population. The MuSICCA will be one of the first validated measures of arousal and awareness for use with children and young people with DOC. This may result in improved confidence for clinicians, care providers and families in setting goals and providing care and stimulation for this population.