Measuring Recovery in Deaf, Hard-of-Hearing, and Tinnitus Patients in a Mental Health Care Setting: Validation of the I.ROC.

This study was aimed at validating the Individual Recovery Outcomes Counter (I.ROC) for deaf, hard-of-hearing, and tinnitus patients in a mental health care setting. There is a need for an accessible instrument to monitor treatment effects in this population. The I.ROC measures recovery, seeing recovery as a process of experiencing a meaningful life, despite the limitations caused by illness or disability. A total of 84 adults referred to 2 specialist mental health centers for deaf, hard-of-hearing, and tinnitus adults in the Netherlands completed the Dutch version of I.ROC and 3 other instruments. A total of 25 patients refused or did not complete the instruments: 50% of patients using sign language and 18% of patients using spoken language. Participants completed the measures at intake and then every 3 months. In this sample I.ROC demonstrated good internal consistency and convergent validity. Sensitivity to change was good, especially over a period of 6 or 9 months. This study provides preliminary evidence that the I.ROC is a valid instrument measuring recovery for hard-of-hearing and tinnitus patients using spoken language. For deaf patients using sign language, specifically those with limited language skills in spoken and written Dutch, more research is needed.

Deaf and hard-of-hearing (DHH) mental health care patients are very diverse in audiological background, as well as mental health problems. Audiologically, patients can be early deaf, late or sudden deaf, (progressively) hard of hearing, having tinnitus (with or without hearing loss), hyperacusis, and/or Meniere’s disease.

Comorbid mental health issues cover the entire range of psychiatric disorders, including but not limited to anxiety and mood disorders, psychotic disorders, personality disorders, addiction, and developmental disorders. In general, higher rates of mental health issues are reported in the deaf population compared to hearing individuals (Fellinger, Holzinger, & Pollard, 2012). Additionally, there may be cognitive or learning disabilities and/or language and communication disorders. The onset and type of hearing loss is related to people’s language preference, language possibilities, and psychological well-being. In general, there is no specific psychopathology related to (early) deafness, but there seems to be a higher incidence of impulse control, behavior, and developmental disorders in deaf patients compared to hearing (Fellinger et al., 2012). Anxiety, depression, and grief are often associated with loss of hearing later in life or developing tinnitus.

The population of patients coming to Dutch specialized mental health care departments for DHH people can be roughly divided into two major groups depending on age of onset, referred to here as “Later Life” and “Early Onset”. The first group (Later Life) contains patients losing their hearing later in life, suddenly or progressively, and/or having tinnitus, hyperacusis, or Meniere’s disease. These patients experience their hearing loss or tinnitus as a handicap; they have problems coping with it and as a result, they experience psychological burden, have difficulties pursuing a meaningful life, and can lose hope for the future. In treatment, this requires a different approach than simply reducing symptoms of mental illness. The so-called third generation cognitive and behavior therapies (such as Acceptance and Commitment Therapy) focus on contextual and experiential change strategies in order to enhance people’s flexibility in coping with life, increasing acceptance, and gaining a purposeful life (Hayes, Follette, & Linehan, 2004). Several studies show this focus fits this “Later Life” patient group (Molander et al., 2018, Westin et al., 2011). The recovery vision used in this study (Leamy, Bird, Le Boutillier, Williams, & Slade, 2011) is in line with the theory of these third generation cognitive and behavior therapies.

The second group (Early Onset) contains patients who are deaf or severely hard of hearing, with an early onset and an impact on cognitive and emotional development, and specifically on development in the area of language and communication (Glickman, 2013). Early onset deaf is defined as congenital hearing loss or hearing loss incurred at or before the age of 4 (Seldran et al., 2011). People who are deaf early in life seem to be more vulnerable to psychological distress due to traumatic events and/or more repressive parenting styles, and traumatization and attachment problems occur more often in this population (Akram & Shafiq, 2014; Knutson, Johnson, & Sullivan, 2004; Schenkel et al., 2014; Schild & Dalenberg, 2012). The interaction of mental health issues and auditory background and their effect on fundamental areas of life, such as participation in society, is complex. As a result, their problems often meet the criteria of Severe Mental Illness (Ruggeri, Leese, Thornicroft, Bisoffi, & Tansella, 2000). Early onset DHH people often access specialized mental health care services for support with their specific communication and cultural needs. In mental health care for patients with severe mental illness, the personal recovery vision as used in this study is common practice (Villagonzalo et al., 2018).

Personal recovery is often described as “a deeply personal, unique process of changing one’s attitudes, values, feelings, goals, skills and/or roles … a way of living a satisfying, hopeful and contributing life even with the limitations caused by illness” (Anthony, 1993). In this view, recovery is seen as a subjective experience, a process in which the patient’s perspective is central (Anthony, 1993; Le Boutillier et al., 2011; Leamy et al., 2011; Slade et al., 2014). For many DHH people there is poor access to information and facilities, which can lead to reduced participation and exclusion. Social stigma can be internalized, reducing the ability of people to define a self, regardless of illness or disability (Cabral, Muhr, & Savageau, 2013; Jacobson & Greenly, 2001). On an individual level, these contextual aspects (i.e. reduced participation, exclusion, and social stigma) require additional (emotional, social, and daily-life) skills for people who are deaf and hard of hearing to achieve a fully fledged position within society and to recover hope, self-confidence, and self-control (Cabral et al., 2013).

Conceptualized as a personal instead of a medical process (Shanks et al., 2013), recovery is applicable to both the Early Onset and Later Life groups of DHH people, although for different reasons. This is relevant because both patient groups are treated in the same setting within the Dutch mental health care system. Application of the personal recovery paradigm within this mental health care system is shifting the focus of treatment away from “fixing the problem” toward learning how to lead a meaningful life without the dominance of symptoms. For Later Life DHH individuals, this means learning to live with the loss of hearing or with their tinnitus. For those in the Early Onset group, personal recovery provides a natural extension to the Deaf Empowerment movement, which has seen many people revolt against the hearing perspective on deafness as a medical issue that needs to be fixed (Andrews, Leigh, & Weiner, 2004; Power, 2005; Skanes, 2014). In the same way that Deaf people do not experience their deafness as something that needs to be cured, many people with psychiatric symptoms see the traditional (medical) view of “recovery” (cure) in mental health care as a paternalistic approach, having to meet the “healthy standard” that is determined by the professional instead of the patient (Le Boutillier et al., 2011; Leamy et al., 2011; Slade et al., 2014).

In mental health care, Routine Outcome Monitoring (ROM) is widely used and refers to regular measurements during treatment to monitor the patient’s response to the treatment. ROM can be used for shared decision-making (Duncan, Best, & Hagen, 2010; Metz et al., 2015) in preparing a treatment plan, adjusting the plan if necessary or deciding when to finish the therapy. Measuring outcome in DHH mental health care is obstructed for three reasons. The first reason applies to the whole population; the next two reasons specifically apply to the Early Onset deaf and severely hard-of-hearing group. The first obstruction is that a focus on symptom reduction in measuring outcome is not fitting the population as argued earlier in this section. Second, there is no instrument available that matches the vision of recovery expressed by DHH patients (Cabral et al., 2013). Third, available instruments are not fully accessible to the Early Onset group because of a greater risk of a language barrier due to limited language skills in spoken and written language.

Research by Walls, Hough, Arkle, and Tathata (2013) points out that mental health care institutes for DHH people have limited knowledge of this recovery vision that is strongly emerging in other fields of mental health care. There are no fully accessible instruments available for this population to measure recovery. Institutes using recovery tools claimed that existing instruments did not meet the needs of the users; therefore, questions were adjusted, pictures were added, or ad hoc translations in a sign language were given to improve understanding, which was only partially achieved. Also, the adjustments had negative consequences for reliability and validity (Walls et al., 2013).

The language barrier for many deaf and severely hard-of-hearing people in mental health care is large. Psychological symptoms mainly are expressed through communication, and assessment and treatment of psychological problems depends on communication (Pollard & Dean, 2003). Research by Crump and Hamerdinger (2017) points out that lack of knowledge about deafness and related problems in language dysfluency can lead to misdiagnosis and inappropriate treatment. When using instruments to collect data on treatment progression, accessible language and communication methods are equally vital.

As stated by Walls and colleagues (2013) in their research on recovery tools in mental health services for deaf people: “it is a misconception by many that the use of sign language is a mixture of signs and gestures created from spoken language. In fact, sign language is a language with its own vocabulary and grammatical structure” (Abstract, p. 12). Certain concepts and terminology may be unknown in Deaf culture. Deaf people may respond to key words and miss the meaning of the question because the grammatical structure of the complete sentence is not understood due to a lack of fluency in the spoken language of the larger culture. As a result, self-report using existing instruments created in written/spoken language and developed for a hearing population is complicated (Walls et al., 2013). Research finds that children with profound and severe hearing loss learn vocabulary at about half the rate of hearing children; as a result, their vocabulary in adulthood is about half that of hearing people, and reading comprehension scores are far below that of hearing people (Boothroyd, Geers, & Moog, 1991; Wauters, Van Bon, Tellings, 2006). This means many early deaf and severely hard-of-hearing people have serious language problems in spoken, written, and even signed languages (Fellinger et al., 2012; Walls et al., 2013).

Instruments measuring recovery that are widely used in the general population and already validated (such as the Recovery Assessment Scale, RAS; Giffort, Schmook, Woody, Vollendorf, & Gervain, 1995) are usually long, using difficult grammatical structures or abstract language, and so make greater demands on language skills. Creating an instrument in a sign language, or translating an existing instrument into a sign language, would seem to be a solution, provided that the sign language skills of the deaf patients using the instrument would be sufficient. As described above, this assumption cannot be made due to many factors influencing language and vocabulary development. In the Netherlands, there is also the situation of an oral history in deaf education with large regional differences in Dutch sign language. Also, research showed that there was a negative attitude to the use of signing in communication in the Dutch deaf population of the 1980’s (Tijsseling, 2014). Many deaf people use “old” signs and do not know standardized Dutch sign language.

Altogether, monitoring recovery in mental health care for DHH patients is obstructed by the lack of a valid self-report instrument that measures what DHH people see as recovery and that is sufficiently visual and fits their linguistic needs (Walls et al., 2013). The “Individual Recovery Outcomes Counter” (I.ROC; Monger, Ion, Henderson, Cumming, & Hardie, 2012) could qualify as a suitable instrument for this population, both for substantive reasons (measuring personal recovery) as for its language properties (using graphics and a concept list for each question). I.ROC is a recovery questionnaire developed by Scottish mental health charity Penumbra in 2007 to measure personal recovery (Ion, Monger, Hardie, Henderson, & Cumming, 2013; Monger, Hardie, Ion, Cumming, & Henderson, 2013). The instrument measures 4 domains, home, opportunity, people, and empowerment, and has 12 items. These domains seem to fit the recovery vision of DHH people. The instrument is visually constructed and contains pictures as well as a list of concepts that clarifies each question. The pictures and concept list are meant to guide the respondent in what aspects are related to the question at hand. This meets the need for certain background information when answering the questions, which deaf people who are minimally literate in the spoken language of the larger community do not have due to information deprivation. Although in this study the I.ROC was used as a self-report instrument (for methodological reasons), it is designed to have a focus on a conversational approach. Provided that communication between clinician and patient is optimal, this seems to be a way to improve accessibility of the instrument to a larger group of patients, with different levels and methods of communication.

The primary objective of this study was to validate the Dutch version of the I.ROC to measure recovery in a mental health care population with deaf, hard-of-hearing, and tinnitus patients. Internal consistency and convergent validity were measured with three other instruments. Also investigated was whether the I.ROC is sensitive to change in the extent of recovery. If the I.ROC proves to be a valid instrument to measure recovery in this population and if it is sensitive to change in the course of treatment, addition of this instrument to ROM could improve treatment monitoring and planning by making better use of the patient’s perspective on his recovery process.

Methods

Design

This study is a quantitative naturalistic cohort study (validation) among deaf, hard-of-hearing, and tinnitus patients in a mental health care setting.

Setting

Most patients included for this study were treated at a special department for deaf, hard-of-hearing and tinnitus patients of a large mental health care agency in the Netherlands (Setting 1). The department offers mental health care to patients of all ages with hearing loss and psychiatric problems. Three patients were included at an institute serving people with hearing loss and mental health or social problems (Setting 2). Setting 2 was added in order to obtain more deaf respondents in the study. All included patients from both centers were 18 years and older. No disorders were excluded (psychiatric or auditory). Patients could be deaf, hard of hearing, or (hard of) hearing with tinnitus. See Table 1 for demographic characteristics.

Table 1

Characteristics of responders and non-responders

	Responders	Non-responders	Subgroup differences
	N = 84 (77,1%)	N = 25 (22,9%)	Chi-squared (p valuea)
Mean I.ROC score at T0 (distribution)	40.8 (20–62)
Auditory status and first language
Early onset	n = 29 (35%)	n = 14 (56%)	3.70 (.148)
Later life hard-of-hearing	n = 8 (9,5%)	n = 2 (8%)
Later life tinnitus	n = 47 (56%)	n = 9 (36%)
Dutch Sign Language	n = 16 (19%)	n = 8 (32%)	2.90 (.235)
Spoken Dutch with signs	n = 6 (7%)	n = 3 (12%)
Spoken Dutch	n = 62 (74%)	n = 14 (56%)
Sex and age
Male	n = 46 (55%)	n = 12 (48%)	.35 (.552)
Female	n = 38 (45%)	n = 13 (52%)
Mean age	N = 84 (48,3)	N = 25 (54,2)	(.085)
Main diagnosis
Mood disorder	n = 16 (19%)	n = 8 (32%)	1.94 (.770)
Anxiety disorder	n = 12 (14%)	n = 3 (12%)
Personality disorder	n = 11 (13%)	n = 2 (8%)
Somatoform disorder	n = 25 (30%)	n = 7 (28%)
Other	n = 20 (24%)	n = 5 (20%)

aDifferences were tested using Chi-squared tests or, if cell counts less than 5, Fisher’s exact test. Mean age difference was tested using a t test.

Recruitment

All patients meeting the age criterion, who were referred to Setting 1 in the period of May 2016 until September 2017 were eligible. Early deaf and severely hard-of-hearing patients, meeting the age criterion, who were referred to Setting 2 in the period of May 2017 to September 2017 were eligible. Informed consent was requested at the intake. The information about the research, the researcher, and collection of data was provided in written language in a research folder, and in summary repeated on the consent form. A deaf communication specialist looked into the language used in the research information folder and in the consent form. For the informed consent procedure, a licensed Dutch sign language (Nederlandse Gebarentaal, NGT) interpreter was available, if needed, to provide a translation in Dutch sign language, adjusted to the participant’s communication skills to make sure the consent information was fully accessible and understood. In this way we tried to provide optimum information in accordance with the ethical standard for research with deaf people (Pollard, 1992).

Procedure

For consenting participants across both settings, baseline measurements were completed at intake. Within Setting 1, measurements were then repeated every 3 months until July 2018.

At every measurement moment four instruments were used: the Health of the Nation Outcome Scales (HoNOS) and the Functional Remission Scale (FRS) filled in by a trained mental health practitioner, the I.ROC, and the RAS as self-report instruments. The quantitative study design was chosen with the purpose of finding a self-report instrument suitable for the population at study. This implicated that we could not use the instruments in a conversational way. A licensed NGT interpreter was present when patients filled in the instruments to note particularities such as remarks about comprehension of the questions, long duration of completion, or premature termination of the measurement. This information was used to better understand the reasons for dropout after giving informed consent to participate. No ad hoc translations or additional explanation was allowed to make sure data collection was standardized. Data from the initial measurement moment were used for validation. Repeated measurements were used for testing sensitivity to change.

Measures

For descriptive statistics and for comparing subgroups on initial I.ROC scores, diagnostic information, auditory information, and demographic variables were collected. Main diagnoses and demographic variables were obtained from patients’ medical files. Patients were asked directly for information about their auditory status to keep the patient’s perspective central in the study instead of using a medical approach. Patients were asked if they considered themselves to be deaf, hard of hearing, or primarily have tinnitus, and what the age of onset of their hearing loss was. Patients were also asked for their preferred language, meaning Dutch sign language (NGT), spoken Dutch with signs (NmG, not being a language but a communication mode where spoken Dutch is supported by signs), or spoken Dutch.

The Individual Recovery Outcomes Counter

Developed by Penumbra in Scotland in 2007 (Monger et al., 2012), I.ROC was created to measure recovery, meaning “the realisation of a meaningful and fulfilling life in the presence or absence of any mental health problem” (Cumming, & Monger, 2011; Ion et al., 2013; Monger et al., 2013). I.ROC is designed to be completed as a guided self-report, in which the individual and mental health professional work together to explore the individual’s recovery. I.ROC is a 12-item questionnaire, which covers 12 indicators: Mental health, Physical health, Personal network, Participation and control, Life skills, Exercise and activity, Social network, Self-management, Safety and comfort, Purpose and direction, Valuing myself, and Hope for the future (Ion et al., 2013). Questions are answered on a 6-point Likert scale ranging from never to all the time. The 12 questions are divided into four clusters: Home, Opportunity, People, and Empowerment. In a first validation study, Monger and colleagues (2013) found that I.ROC is a valid and reliable instrument for measuring recovery, which can be used for monitoring outcome and as a tool for mental health care planning. Dickens, Rudd, Hallett, Ion, and Hardie (2017) examined the properties of the I.ROC in a larger sample and found that when the 6-point response structure was changed to a 4-point scale, the items measured a single, recovery-related construct with acceptable fit statistics. The I.ROC items covered around 75% of the recovery continuum; items did not adequately capture the people who reported the lowest levels of personal recovery.

The Recovery Assessment Scale (RAS)

Developed by Giffort and colleagues (1995), RAS was one of the earliest developed recovery-oriented scales and is now widely used in mental health care. RAS contains 41 items and is a self-report measure, whereby individuals respond to each item on a five-point Likert scale (1, strongly disagree; 5, strongly agree). RAS has good internal consistency (α = .93) and validity (Corrigan et al., 1999), now independently assessed in several studies (Chiba, Miyamoto, & Kawakami, 2010; Corrigan, Salzer, Ralph, Sangster, & Keck, 2004; McNaught, Caputi, Oades, & Deane, 2007). In the current study the factor distribution from research by McNaught, Caputi, Oades, and Deane (2007) was used, because of their solid arguments to replace two items to a different factor. In other respects, the factor distribution is equal to the original study of Corrigan et al. (1999). Factor 1: Personal confidence and hope. Factor 2: Goal and success orientation. Factor 3: willingness to ask for help. Factor 4: Reliance on others. Factor 5: No domination by symptoms. Because the RAS is a validated and widely used instrument to measure recovery, it is used as the main validating instrument in this study.

The Health of the Nation Outcome Scales

HoNOS (Wing, Curtis, & Beevor, 1996) was developed to routinely map the mental health condition and social functioning of psychiatric patients in a simple, reliable, and valid way (Mulder et al., 2004). Completed by a mental health care professional involved with the patient, HoNOS comprises 12 items, in four subscales: behavioral problems (Items 1–3), functioning (Items 4–5), symptoms (Items 6–8), and social problems (Items 9–12). Researchers (Wing et al., 1998) have concluded that HoNOS “is simple to use and generally clinically acceptable; it covers a broad range of clinical problems and social dysfunctions; it is sensitive to change or the lack of it over time; it has acceptable reliability and is compatible with longer and well-established instruments” (Discussion section, para. 1). Results of inter-assessor reliability and test–retest reliability testing vary (Pirkis et al., 2005), but Dutch research (Mulder et al., 2004) shows reasonable to good reliability: Cronbach’s alpha = .78; Intraclass Correlation Coefficient (ICC) = .92 and satisfactory convergent validity. Because the HoNOS is currently the only instrument used for ROM in the population under study, it was added to our research.

The Functional Remission Scale

Developed by a Dutch taskforce (Wiersma et al., 2015), the FRS was designed to measure social and societal functioning in a population with severe mental illness. A health care professional assesses functioning over the past 6 months in three areas: (I) living and personal care; (II) work, education, or aimed daily activities; and (III) social contacts. There are three score options: 0 (functioning is completely independent), 1 (clear problems in functioning independently), and 2 (a severe, manifest problem in functioning). The sum score can vary from 0 to 6; a sum score of 0 means complete remission (functioning is fully independent). Wiersma and colleagues (2015) found that internal consistency and (interrater) reliability is sufficient and acceptable. Discriminating validity of the instrument is evident; there is little overlap between symptomatic and functional remission. In fact, in a Dutch study, 50% of the patients with functional remission had no symptomatic remission (Wiersma et al., 2015). This is interesting for our study because of the distinction made between medical and personal recovery. This instrument was also added to cover some items that would be missed using only the RAS.

The Physical Activity Question

As none of the measures described above covered the I.ROC item on exercise and activity, an additional question was added, in which respondents were asked how many days per week they were physically active for at least 30 min a day. This question was derived from the PHAMOUS protocol (Bartels-Velthuis et al., 2018).

Analyses

Data were analyzed using SPSS (version 20.0). Chi-squared statistics (for auditory status, first language, and sex), Fisher’s exact tests (for main diagnoses), or t tests (for mean age difference) were used to analyze differences between responders and non-responders (see Table 1). Cronbach’s Alpha was used to assess the internal consistency of the I.ROC. One-way Analysis of Variance (ANOVA) analyses were executed on the I.ROC measurements at time 0 to see if I.ROC scores varied between the different subgroups (three subgroups for auditory status and five subgroups for main diagnosis).

For convergent validity, Pearson’s correlations were calculated between total scores of the I.ROC, RAS, and HoNOS. Hypotheses were formulated about expected correlations between the I.ROC items and sections of the HoNOS, factors of the RAS, items of the FRS, and the Physical activity question (see Table 2). Only these hypothesized correlations were tested. Correlations above .30 were considered medium and correlations above .50 were considered large (Cohen, 1988).

Table 2

Expected correlations between the I.ROC items and the HoNOS sections, the RAS factors, the FRS items, and the Physical activity question

I.ROC cluster	I.ROC item	HoNOS sectionsa	RAS factorsb	FRS itemsc	Physical activity question
Home	Mental health	Section C: −.376 **	Factor 1: .577 **
			Factor 5: .553 **
	Life skills	Section D: −.311 *		Item I: −.112
	Safety and comfort			Item I: −.277
Opportunity	Physical health	Section B: −.020			.358 **
	Exercise and activity				.620 **
	Purpose and direction	Section D: −.330 **	Factor 2: .392 **	Item II: −.339 *
People	Personal network	Section D: −.286*	Factor 4: .516 **	Item III: −.077
	Social network		Factor 4: −.212
	Valuing myself		Factor 1: .435 **
Empowerment	Participation and control		Factor 2: .470 **
	Self-management		Factor 1: .482 **
			Factor 5: .416 **
	Hope for the future		Factor 1: .692 **
			Factor 2: .508 **
Total score		Total score: −.346 **	Total score: .655 **	Total score: −.200

Note. I.ROC = Individual Recovery Outcomes Counter. HoNOS = Health of the Nation Outcome Scales. RAS = Recovery Assessment Scale. FRS = Functional Remission Scale. Medium (above .30) to high (above .50) correlations are in boldface.

aHoNOS sections: A, Behavioral problems; B, Impairment; C, Symptomatic problems; D, Social problems.

bRAS factors (McNaught et al., 2007): Factor 1, Personal confidence and hope; Factor 2, Goal and success orientation; Factor 3, willingness to ask for help; Factor 4, Reliance on others; and Factor 5, No domination by symptoms.

cFRS items: Item I, living and personal care; Item II, work, education, or aimed daily activities; and III, social contacts.

* p < .05

** p < .01

Sensitivity to change was assessed by calculating Cohen’s d effect size at different measuring moments of the I.ROC, RAS, and HoNOS. Cohen’s d of .20 is described as a small effect size; Cohen’s d of .50 is described as a medium effect size (Cohen, 1988). To make sure that measuring moments of the I.ROC and the RAS were in fixed and comparable time frames after the first measurement, the following rules were obeyed: first measurement is at time 0 (intake), second measurement needs to take place within 10–22 weeks after time 0, third measurement within 23–35 weeks after time 0, fourth measurement within 36–48 weeks after time 0, and fifth measurement within 49–61 weeks after time 0. Consecutive measurements were planned every 12 weeks; the minimum time between them was set at half of that time, namely 6 weeks. Because the HoNOS was separately filled in by an assessor, date of assessment could differ from the I.ROC/RAS assessment. In order to keep variability limited, the HoNOS should be administered within 1 month before or 1 month after the I.ROC and RAS. Measurements not fitting the stated rules were excluded from analysis.

Results

In the research period, 109 patients were newly referred to the department, 84 of them were included in the study. No significant differences were found concerning auditory status, first language, sex, age, and main diagnosis between the responders and non-responders (see Table 1), but the subgroups were small. In the early onset group, one third did not participate, specifically looking at the Dutch sign language users, one third did not participate, and the same can be said for patients with a mood disorder. In the group of “non-responders” six deaf Dutch sign language users and two hard-of-hearing patients using Dutch with signs dropped out after giving consent at first.

Internal consistency

A total of 84 valid I.ROC measurements at time 0 were used for Cronbach’s alpha analysis. Taking into account that the I.ROC is a short (12-item) questionnaire and a lower number of items has a lowering impact on Cronbach’s alpha, an alpha value of .80 is considered as a good goal to aim for (Gliem & Gliem, 2003). According to the convention of alpha values greater than .70 indicating acceptable reliability and alpha values greater than .80 indicating good reliability (Gliem & Gliem, 2003), the internal consistency of the I.ROC was good: .853. When tested only for the early onset group, internal consistency was slightly better: .884. Cronbach’s alpha of the RAS was .92 (N = 83, 41 items), of the HoNOS .57 (N = 81, 12 items), and of the FRS .67 (N = 49, 3 items) in this sample.

Subgroup differences in initial I.ROC score

The mean I.ROC score at baseline for the whole sample (n = 84) was 40.8 (s.d. 9.35, range 20–62). Two one-way ANOVA analyses showed that I.ROC score was not associated with auditory status and not associated with psychiatric diagnosis. Post hoc Bonferroni test showed no differences between specific samples.

Convergent validity

For convergent validity, correlations (Pearson’s r) were calculated between total scores of the I.ROC, RAS, and HoNOS. For these analyses, cases for which the two instruments at Time0 were not completed within 1 month from the date on which the I.ROC was completed were excluded. A total of 63 valid cases remained. The I.ROC scores were positively and highly correlated to the RAS total score (r = .655, p = .000). The I.ROC total score was negatively correlated to the HoNOS total score, on a medium level (r = −.346, p = .005). The FRS and I.ROC total score correlated negatively; this correlation was small (r = −.200, n.s.). Analyses of only the participants in the early onset group meeting the “within 1 month completion” criterion (N = 20) showed an equally high correlation between I.ROC and RAS total scores (.651 p = .002) and also a medium correlation between I.ROC and HoNOS total scores (−.415, n.s.).

In Table 2 all the correlations hypothesized to be medium to high are shown. If, in fact, the correlation turned out to be medium to high, this is printed in bold. To be complete, significance levels are also shown. In these analyses the items of the I.ROC and FRS, the sections of the HoNOS, and the factors of the RAS were used.

Sensitivity to change

The effect sizes (Cohen’s d) for three instruments at four measurement moments (M0–M3) were calculated. Because of the small patient numbers at M4, the effect sizes between M3 and M4 were excluded. See Table 3 for the results. The effect sizes for all three instruments at the separate moments can be considered as small. Across all three measures, score changes from M0 to M1 and from M1 to M2 reflected a positive change (increased recovery/improved functioning). From measurements 2 to 3, the RAS mean score went down (to the level at M0), and the HoNOS mean score went up, suggesting subjects experienced decline in recovery and more problems. This pattern, however, was not found with the I.ROC scores, which instead maintained the pattern seen within the earlier testing phases, with scores consistently increasing from measurement 0 to 3, indicating progress in recovery throughout the whole period.

Table 3

Effect sizes for three measurement periods for the I.ROC, the RAS, and the HoNOS

		N	M 0 (SD1)	M 1 (SD2)	M 2 (SD3)	M 3 (SD4)	ESa
			(T0)	(T10–22 weeks)	(T23–35 weeks)	(T36–48 weeks)
I.ROC	M1 – M0	41	39,59 (9,02)	43,71 (9,58)			.44
	M2 – M1	29		42,31 (9,98)	44,86 (10,03)		.25
	M3 – M2	15			42,93 (8,13)	45,13 (11,53)	.22
RAS	M1 – M0	41	135,34 (17,65)	142,49 (19,59)			.38
	M2 – M1	29		140,07 (20,72)	146,86 (20,68)		.33
	M3 – M2	15			141,27 (22,13)	135,40 (21,54)	−.27
HoNOS	M1 – M0	27	11.07 (4.01)	10.15 (4.78)			−.21
	M2 – M1	16		9.94 (4.16)	8.56 (4.08)		−.33
	M3 – M2	7			8.57 (3.41)	9.29 (3.50)	.21

Note. I.ROC = Individual Recovery Outcomes Counter. RAS = Recovery Assessment Scale. HoNOS = Health of the Nation Outcome Scales.

aES, effect size, M2–M1/SDpooled

In order to investigate the sensitivity to change of the I.ROC over a larger period of time, effect sizes over the periods M0–M2 and M0–M3 were calculated, and medium effect sizes were found (M0–M2, d = .53; M0–M3, d = .60). The following effect sizes were found for the RAS: M0–M2, .56 and M0–M3, .48; and the HoNOS: M0–M2, −.52 and M0–M3, −.68.

Discussion

The aim of the current study was to validate the I.ROC for early onset DHH and late onset deaf, hard-of-hearing, and tinnitus patients in a mental health care setting.

Internal consistency findings in this study are consistent with the preliminary validation study of Monger and colleagues (2013). Specifically looking at the early onset group, this conclusion stays intact. This suggests that, although our research is aimed at a different and varied patient population, the instrument remains internally consistent. Also comparable is the finding that internal consistency of the RAS is the highest. The literature about interpreting Cronbach’s alpha however suggests that alpha values above .90 could indicate redundancy (too many items measuring the same; Briggs and Cheek, 1986). The internal consistency of HoNOS in our study was poor and lower than found by Mulder and colleagues (2004).

The mean initial I.ROC score for all responders was comparable to the factor validation study of the I.ROC by Dickens et al. (2017). In this study, a varied patient population using different kinds of mental health services was used. This means that these populations, which are both very diverse but also different from each other, still seem comparable in terms of experienced degree of recovery at start of treatment. However, it would be even more interesting to know if variation in recovery patterns at baseline is seen for different subgroups, that is “early onset” deaf patients and “later in life” hard-of-hearing or tinnitus patients. Cabral and colleagues (2013) showed that deaf people define recovery as being more independent and taking care of themselves, which could be associated with the items “participation and control” and “self-management” (cluster Empowerment) and “life skills” (cluster Home). No data are available about tinnitus patients and people who acquire hearing loss later in life on this matter. One could theorize about items being especially relevant in these groups, such as “mental health” or “hope for the future”, as these groups often experience a loss in these areas. As hypothesized, a high correlation was found between the I.ROC and the RAS total score. The I.ROC and the RAS were both constructed to measure recovery and consequently should correlate highly. The correlation between the I.ROC and the HoNOS was on a medium level. The HoNOS was constructed to measure symptoms and problems, and as the vision on personal recovery states, recovery can be achieved even with the limitations of illness. Because of the lack of research data about the use of the HoNOS in this population, it is difficult to interpret this finding properly. When looking at these correlations specifically for the “early onset” group, the correlation between the I.ROC and the RAS remains high, and between the I.ROC and the HoNOS it remains on a medium level. Subsequently, of the expected correlations, only one medium correlation was found between I.ROC and FRS items and no medium or high correlations between I.ROC clusters and FRS items. To find an explanation for this lack of correlation, FRS scores were explored. Little variation in FRS scores was found in this patient group; many patients scored high on functional remission in all three areas, meaning that the population in our study generally functions independently in living, work, and social contacts. FRS is an instrument designed for patients with “severe mental illness”, which can be defined as a patient group with a duration of treatment of 2 years and more, and severe symptoms or severe difficulty in social, occupational, or school functioning (Ruggeri et al., 2000). The patient group in this study is much more varied in severity of mental illness and level of functioning, so the conclusion is drawn that this instrument does not fit the population in this study.

Sensitivity to change was good for the I.ROC. Comparing the I.ROC, RAS, and HoNOS, in the first 3-month period, I.ROC was most sensitive to change. Over a period of approximately 9 months, the I.ROC and HoNOS did best. Over a 6-month period, the three instruments were comparable. This would mean that the I.ROC is very suitable for treatment planning and adjusting, as it is sensitive for changes in the first period of treatment as well as later in a patient’s process. The finding that the HoNOS and RAS show an opposite trend in terms of recovery or problems between the 6-month and 9-month measurements is remarkable. The I.ROC here shows a steady pattern in the same direction; patients experience more recovery as treatment progresses. We found no explanation for this difference, and again subgroups were too small to be able to examine this in more detail.

Limitations

This research had some limitations. The variation in the studied population is high, because of different times of onset and types of hearing loss, with varying consequences for language, development, and psychological well-being, making it difficult to draw specific conclusions about subgroups. On the other hand, this aspect provides a strong naturalistic study, as it mirrors the level of existing variation in this field of mental health care. However, due to the small number of participants in this study reliable analyses for subgroups were incomplete.

The non-response rates among the patients in the early onset group and patients who indicate using Dutch sign language as their first language were relatively higher than in the late onset hard-of-hearing and tinnitus groups and people stating they prefer spoken Dutch supported by signs or spoken Dutch. About half of the early onset deaf patients in the non-response group dropped out because they were not able to complete the RAS (after agreeing to participate).

Study design may have contributed to this non-response bias, as instruments were presented in a static order with the RAS appearing first, followed by the I.ROC. In some cases, this led to termination of the measurement before completing the RAS. The reported reasons for termination were taking very long to complete the RAS and not understanding the questions. Consequently, these participants did not want to or were not able to proceed. In these cases, the I.ROC was skipped, and information about the ability of these participants to answer I.ROC questions was missed. For this reason, our hypothesis that the I.ROC would be more accessible and easier to understand could not be tested.

Connected to this subject, the very reason to perform this research, namely the lack of valid recovery instruments for this population, provided a big methodological challenge in this quantitative research. Using an existing and validated instrument meant using an instrument that theoretically is not accessible to part of our population. There seems to be no real solution in this case, but adding qualitative data could have strengthened our conclusions. Because the design of the quantitative study (in particular using the I.ROC strictly as a self-report instead of a conversational approach connected to treatment evaluations) made the collection of relevant qualitative data more difficult, this has not been done. One could state that establishment of criterion validity for the DHH subgroup with language difficulties is not possible without knowing whether their reading abilities were sufficient for understanding the instrument used as the criterion. Dropout rates suggest that RAS was indeed not an accessible instrument for this subgroup, and therefore the information about the I.ROC is insufficient.

Clinical Implications

There is a distinct need for accessible Routine Outcome Measurements for use with people experiencing both hearing-related difficulties and mental health concerns. While recovery-oriented practice, including the use of recovery-focused outcome measures, is burgeoning within many areas of mental health, it has yet to significantly impact upon services supporting DHH patients. DHH people with mental health problems view their recovery as being more independent, feeling better, being more active, and taking care of themselves (Cabral et al., 2013). Recovery-oriented measures offer one way of ensuring that such services remain focused on this personal process of recovery. Their use can help facilitate the embedding and evaluation of personalized, strengths-based approaches. The I.ROC seems to be an instrument that could fit the patient groups included in this study (namely the early onset deaf and severely hard-of-hearing patients in the response group and the later in life hard-of-hearing and tinnitus patients); it is independent of specific problems and symptoms, is short, contains questions in a clear language with keywords to support full understanding of the questions, and is visually supported by using colors and graphics. However, the high dropout rates in the early onset group and the lack of data on these specific patients make it impossible to fully test the I.ROC’s accessibility for these patients. It is hypothesized that these are the patients with limited language skills in spoken Dutch and further study with the I.ROC in the early onset deaf and severely hard-of-hearing group is needed. It is strongly recommended that the I.ROC is then used with a conversational approach providing optimal communication between the clinician and patient (Pollard, 2002). Assessment of Dutch reading abilities, as well as Dutch Sign Language skills of the participants (a communication assessment on language dysfluencies), could further strengthen the conclusions to be drawn (Crump & Hamerdinger, 2017). Qualitative data should be collected to ensure all relevant aspects of accessibility are covered.

Although this study provides preliminary evidence that the I.ROC is a valid instrument measuring recovery in the late onset hard-of-hearing and tinnitus patients and part of the early onset group, this has not been proven for the patients in the early onset deaf group and patients who prefer sign language and are minimally literate in spoken Dutch. Hopefully this study invites further research into the usefulness of the recovery vision in mental health care for DHH patients and into the accessibility of the I.ROC for the early onset deaf and severely hard-of-hearing group.