Development and usability testing of a web-based smoking cessation treatment for smokers with schizophrenia.

INTRODUCTION: Over half of people with schizophrenia and other psychotic disorders smoke tobacco. Web-based approaches to cessation have been effective for the general population, but are not usable by many with schizophrenia disorders due to cognitive impairments and low computer experience. We developed a prototype smoking cessation treatment website for this group of smokers with features to reduce cognitive load. Here we report results from initial office-based usability testing and home-based field testing.
METHOD: Five people were observed using the prototype website in the office with think-aloud cognitive interviewing. The website was modified based on these data. Six people then used the website on a home laptop after a single training session, with further coaching if needed.
RESULTS: Office-based testing showed that the website was usable, but required minor modifications. Further editing provided labels that were more explicit and concrete, limited the conceptual content on each page, and modified features of the support group forum. Home-based field-testing identified further functionality issues that were rapidly modified. Over half of users needed more than a single session of training to use the computer and website. Eighty three then used it independently and were very satisfied with the web-based program. Among the five smokers who field tested the prototype, 60% cut down and 20% had quit smoking after three weeks of home use.
CONCLUSION: The prototype website was usable and satisfactory. With training and support, home use of this cessation website appears to be feasible and promising for cessation among smokers with schizophrenia. Further research is needed to evaluate web-based cessation treatment in people with psychotic disorders.

Introduction

Although the prevalence of smoking has declined in the past 50 years (Escobedo and Peddicord, 1996, CDC, 2009), people with behavioral health disorders remain vulnerable to nicotine addiction. Nicotine dependence is particularly common among people with schizophrenia and other psychotic disorders (50–80% smoke) (de Leon et al., 1995, de Leon et al., 2002, Vanable et al., 2003, Kotov et al., 2010, de Leon and Diaz, 2005, Forchuk et al., 2002, Lasser et al., 2000, Herran et al., 2000, Lawrence et al., 2009, Hughes et al., 1986, Etter et al., 2004, Annonymous, 2008). The high rate of smoking in this group contributes to their premature morbidity and mortality (Kelly et al., 2011, Colton and Manderscheid, 2006, Mauer et al., 2006, Brown et al., 2000, Hoang et al., 2011, Chwastiak and Tek, 2009, Tiihonen et al., 2009). Fortunately, quitting at any age reduces morbidity and increases life expectancy (Doll et al., 2004), and, like the general population, people with schizophrenia can use cessation treatment to increase their likelihood of quitting (Evins et al., 2001, Evins et al., 2005, Evins et al., 2007, George et al., 2002, George et al., 2008, Baker et al., 2006, Weiner et al., 2011, Tsoi et al., 2010, Ferron et al., 2009, Williams et al., 2012).

Interactive and tailored website cessation programs facilitate cessation for the general population (Myung et al., 2009, Shahab and McEwen, 2009, Civljak et al., 2013), and they dramatically extend the reach of cessation treatment (An et al., 2010). Human-centered design processes can facilitate high quality designs (IDEO.org, 2015), but websites may still may not be usable by important subpopulations. Along these lines, currently available cessation websites are not usable by people with psychotic disorders (Brunette et al., 2011), due to lack of accommodations for the cognitive impairments commonly present in this group (Keefe and Eesley, 2006).

In order to develop websites that can be usable by people with psychotic disorders, we have used principles of design for people with disabilities (Anonymous, 2010) and iterative usability testing (Rotondi et al., 2007, Ferron et al., 2011) to establish designs that reduce cognitive load, enabling people with schizophrenia to effectively use them (Ferron et al., 2012). With such a design, Rotondi's group developed a psychoeducation website for people with schizophrenia with an online discussion forum that provided peer support. After one training session for use of this particular website, 100% of subjects with schizophrenia used the site and support group independently for an average of 46 h (and almost 3000 page views) over a year, resulting in improved symptoms and functioning (Rotondi et al., 2010).

Using similar design principles, we sought to develop a prototype of a tailored, easy-to-use smoking cessation website that included behavioral therapy modules, an information resource library, and an online support group for smokers with psychotic disorders who want to quit smoking, called Let's Talk About Quitting Smoking. We started with a prototype, as prototypes are a cost effective strategy to enable rapid evaluation of new technology tools or to test tools in new populations (IDEO.org, 2015, Hall, 2001). Usability testing is a key component of website development that can ensure that the intended users are able to benefit from the website (Stoddard et al., 2006). This report describes the website design and usability testing of this prototype website among 11 people with psychotic disorders.

Materials and methods

Website development

Website content

We used the Theory of Planned Behavior to guide our overall approach, in which we attend to attitudes, social norms, and perceived behavioral control for smoking and cessation treatment (Ajzen, 1991, Ajzen, 2006). We developed the website prototype to contain a) standard evidence-based behavioral techniques for smoking cessation (Association, A.L., 2015) adapted for people with psychotic disorders (Cather et al., 2007) in eight brief, linear, interactive modules, b) an online forum support group, and c) a “library” of educational materials for people with psychotic disorders who are quitting smoking.

The behavioral therapy modules

In this prototype engaged users to list reasons for quitting, track the circumstances associated with their smoking, learn and practice simple coping strategies to use when trying not to smoke. Additionally, users learned about why medications can help with quitting and how to talk to a doctor about getting medications. The content helped users set a quit date and make a detailed plan for activities during that day. Additional content that was planned for the final version of the website included helping the user develop skills for: managing stress without smoking, maintaining use of cessation medication to prevent relapse; and dealing with a slip without relapsing to daily smoking. The content was also designed to help the user maintain his or her motivation to quit smoking.

The modules were interactive and included video demonstrations, practice exercises, and questions to engage the user and reinforce learning (Binder, 1996). Since engagement promotes abstinence (Richardson et al., 2013), and contingent reinforcement can improve engagement (Carey and Carey, 1990, Helmus et al., 2003, Cahill and Perera, 2011), the completion of each module resulted in an automated email to the coordinator, who could reinforce use. In this study, completion of each module was also reinforced with $5 that was electronically placed on a personal gift card. This feature can be adapted to automate reinforcing emails or texts to the user rather than cash reinforcers.

The online forum support group

Was intended to help users connect with peers who were also learning skills to quit smoking. The support group used a bulletin board format. This allowed for asynchronous communication between members, which is generally more convenient than requiring all users to be on line at the same time. This arrangement required messages to be text-based, and in this application users typed messages using a keyboard. A moderator, who was a tobacco treatment specialist, made posts to engage users into the forum and into conversation with each other.

The online ‘library”

This feature consisted of a “library” of easy-to-read information sheets about smoking, quitting, and cessation treatment. We planned for the next version to contain a larger group of documents and videos presenting information relevant for smokers with psychotic disorders, including managing stress, relaxed breathing, etc.

Development and design considerations

The interface was designed to minimize reliance on cognitive functions (Mayer and Moreno, 2003) that are often impaired in schizophrenia, including working memory, spatial perception and abstraction (Keefe and Eesley, 2006). Key design features included a shallow hierarchy and explicit labels, which rendered a website with a flat, explicit, weak-modular design that has been shown to be highly usable for this group (see Table 1) (Rotondi et al., 2015). The eight linear modules, designed to be used in order, guided users to learn smoking cessation information and skills. This linear design guides the user to view the content from start to finish, increasing the likelihood of viewing all content and thus improving knowledge acquisition (Crutzen et al., 2012). To improve focus on and learning of the website content, pages were presented sequentially within each session rather than scrolling, which requires simultaneous assessment of need to scroll, physical implementation of scrolling, and continued refocusing on a new area to read. Further, the sequential page approach guides the user to complete the acquisition of one “chunk” of information before moving on to attend to the next “chunk” of information (Anonymous, 2010). We used large font, simple text or video, and large buttons to enhance user's ability to view, understand and interact with the content. Advertisements and moving content were not used, as they tend to distract learners. Text-to-speech software enabled audio presentation as the user reads the pages, a feature that also enhances learning and is preferred by this population (Ferron et al., 2012). Fig. 1 depicts an example of one of the program's simply designed pages that addresses triggers. In contrast, Fig. 2 is a screenshot of a currently available website page addressing the same issue. At the end of each module, the user was asked questions about the information and given feedback to reinforce learning.

Table 1

Description of shallow hierarchy website.

Level 1 (Homepage)	List of modules	Library list	Support group description
Level 2	Main module content	Information sheets	Scrollable online forum
Level 3	Selectable “More Information” text or video

Fig. 1

Let's Talk About Quitting Smoking website page.

Fig. 2

Freedom from smoking website page.

Because computers have large screens and keyboards as well as a mouse (which improve viewing and interactive capacity for those with cognition, dexterity and vision problems that are common in this group), we chose the laptop computer over phones and tablets to deliver treatments to this population. However, the website was programmed to allow for future use on phones or tablets. We used the Java programming language to implement the interactive portions of the website, including the modules, support group, and the user's preference for audio. For the website's text-to-speech capabilities, the Festival software package was used to generate spoken text on our web server.

Overview of study design

The study protocol was approved by the Dartmouth Committee for the Protection of Human Subjects. After consenting, subjects completed baseline assessments. They were then randomly assigned to either a) use the prototype website with a usability cognitive interview protocol in the office during the first 4 weeks (Group 1), or b) use the further modified prototype website independently at home during the second 4 weeks (Group 2). We enrolled two groups of five or six participants, as groups of this size are sufficient for identifying the vast majority of problems with the design of a product (Rubin and Chisnell, 2008, Nielsen and Landuar, 1993). All participants were assessed at baseline, week four and week eight.

Study participants

Participants were adults with documented clinic chart diagnoses of psychotic illnesses (schizophrenia, schizoaffective disorder) receiving outpatient treatment at the mental health center. They were daily smokers who stated that they were interested in quitting during the next month. They were fluent in English and willing and able to give informed consent. No computer experience was required. Two people completed baseline assessments but dropped out before being randomized to use the program and are not included in this report. One smoker in Group 2 quit smoking before he began home-based field testing in the 5th study week and is included because he participated in the home-based field testing. The demographics of the study group are described in Table 2.

Table 2

Participant baseline characteristics.

	Total group	Office users	In-home users
	n = 11	n = 5	n = 6
Mean years of age (SD)	49 (12)	48 (16)	49 (10)
Number female (%)	6 (55)	2 (40)	4 (67)
Number African American (%)	5 (46)	3 (60)	2 (33)
Number White (%)	4 (36)	1 (20)	3 (50)
Number single (%)	9 (82)	4 (80)	5 (83)
Mean years education (SD)	11.3 (2)	11.6 (1)	11 (3)
Number with schizophrenia (%)	10 (91)	5 (100)	5 (83)
Mean Colorado Symptom Index Score (SD)	13.5 (8.6)	15.6 (6.3)	11.7 (12.0)
Mean number lifetime psychiatric hospitalizations (SD)	49 (12)	48 (16)	6 (7)
Number used computer ≥ 5 times (%)	6 (55)	3 (60)	3 (50)
Mean number of smokes per day (SD)	13 (5)	13 (7)	13 (4)
Mean breath carbon monoxide (ppm; SD)	19 (14)	21 (16)	17 (12)
Stage of Change
Number planning to quit right now (%)	3 (27)	1 (20)	2 (33)
Number planning to quit in next month (%)	4 (36)	2 (40)	2 (33)
Number planning to quit but not in next month (%)	4 (36)	2 (40)	2 (33)
Mean BACS Total Z Score (SD)	− 3 (1)	− 2 (2)	− 3 (1)
Mean raw WRAT Reading Score (SD)	48 (7)	48 (9)	47 (6)
Mean raw WRAT Comprehension Score (SD)	31 (11)	31 (8)	32 (15)

SD = standard deviation.

BACS = Brief Assessment of Cognition in Schizophrenia.

WRAT = Wide Range Achievement Test 4.

Evaluation protocol

Office-based usability testing

This testing was designed to evaluate usability of the website design (Rubin and Chisnell, 2008) with “think aloud” style cognitive interviewing and observations of use (van Someren et al., 1994) as well as satisfaction questionnaires (Davis, 1989) to get user feedback on the content. During week 1, trained research staff met individually with Group 1 subjects in the office, provided brief computer skills training, and accessed the website to use the first 3 modules. Research staff observed and assisted participants who used one to three modules with a 15-min break in between the modules. At the end of each module, the computer assessed participant satisfaction via a brief questionnaire. The staff then showed the subject the online support group, helped him/her to log on, and provided brief instruction on its use. Staff observed and recorded the subject using the group. The group leader initiated the discussion; following users responded to the leader and to each other. Group 1 subjects returned to the clinic twice over the next two weeks to use the next two to four modules and online support group with the same observation procedure. In the fourth week, Group 1 subjects returned to the clinic for an exit interview, including smoking characteristics and satisfaction with the website.

Home-based field testing

After the four-week assessment for smoking characteristics, research staff then met with Group 2 subjects in their homes, set up the laptop computer, provided training on how to use the computer, access the website, and use it's components in the same manner as was used with the in-office procedure. They instructed subjects to use the website program approximately 3 times a week for 3 weeks. Research staff called subjects twice during the first week to remind them to use the program and to help the subject solve any technical problems. They were available to subjects by phone or in-person for computer coaching and support over the next 2 weeks. At the end of the third week of in-home website use, research staff came to participant's homes, observed them using a final module and returned the computer to the research clinic. All subjects were again interviewed at 8 weeks regarding smoking characteristics and satisfaction with the website.

Measures

Subjects were assessed at baseline for demographics, smoking characteristics (Fagerström, 1978, Weinberger et al., 2007, Tiffany and Drobes, 1991, Rohsenow et al., 2003, Tidey and Rohsenow, 2009), intention to quit (DiClemente et al., 1991), symptoms (Modified Colorado Symptom Index) (Conrad et al., 2001), cognitive function (Brief Assessment of Cognition in Schizophrenia) (Keefe et al., 2004) and reading comprehension (The Wide Range Achievement Test) (Wilkinson et al., 2006). Staff reviewed mental health center charts for psychiatric diagnoses. Qualitative observation and the “think-aloud” method to evaluate cognitive processes during website use enabled usability assessment: subjects were instructed to speak their thoughts as they read and navigated the website while a researcher observed their use (van Someren et al., 1994). With Camtasia software (Techsmith.com), we video-recorded users' activity on the computer screen, allowing researchers and programmers to observe use again after the sessions if needed. The Perceived Usefulness and Ease of Use Scale, an adapted brief semi-qualitative interview (Davis, 1989) gathered reactions to and satisfaction with website components. For Group 2, we also tracked number of logins, page views, and minutes of use of each website component during the three weeks of independent home use.

Statistical analysis

Descriptive statistics were used to describe the study groups as well as to present usability and outcomes data.

Results

Office based usability testing

The usability problems and resolutions addressed in this first phase of testing are listed in Table 3. The participants' ability to use the website varied. Three of five participants were able to complete the program with little difficulty and little aid after the first in-office training and use session. One participant needed extensive coaching and one person decided not to continue after completing the first session. Most participants reported they were highly satisfied with the website. Participants rated their satisfaction with modules as follows: 75% or more of users rated modules 2 and 4 through 8 “high” or “very high.” One half to two thirds of participants rated their satisfaction with Modules 1 and 3 “high or “very high.” The remaining users rated their satisfaction with the modules as “neutral,” e.g. neither satisfied nor dissatisfied. One young participant with high cognitive functioning and extensive computer experience found the program ‘too easy’ and ‘repetitive.’

Table 3

Usability problems and solutions.

Problems in phase 1	Solutions in phase 1
Some participants had difficulty learning computer mouse skills while using the first module	More intensive coaching was provided with practice on the first session pages
Function of some buttons was not clear - participants did not know what to do	Function of the buttons was explained with text in button
Participants had difficulty navigating the online support group, finding posts, making new posts	Changed the order of posts; adjusted spacing, font and bolding to clarify posts
Pages with two areas of content or function were confusing to users	Reduced per page content and function to contain single concept or function
Participants did not understand some phrases and words	Changed text and labels to be more concrete and explicit

Home based field testing

Additional usability problems and resolutions addressed in the home use phase are listed in Table 3. Tracking of website use showed that participants logged into the website an average of 7.5 (4–13) times over three weeks. They averaged about an hour per login session (27–123 min), and they spent an average of 36 min (16–63 min) on each module. Participants looked at an average of 1 document in the website library. They averaged 15 page views (0–47) in the support group online forum and posted an average of 2.3 posts (0–5) in the support group.

Similar to in-office use, each participant's ability to use the website independently at home varied, and is summarized in Table 4. All six users were able to learn to use the website within one to three training sessions, although they varied widely in how much coaching they needed to sustain their engagement with the program (ranging from a brief phone call to answer a question, to three sessions over three weeks with a phone call in between). One participant had difficulty attending to and comprehending the material in the program, even with the audio function of the program turned on. His reading comprehension score was 18, which is much lower than the group mean of 32. Another participant was concerned that the computer might be stolen from her apartment and wanted to have apartment building staff store it for her. She used the program in an office of the apartment building. All participants returned their computers fully intact.

Table 4

Vignettes of in-home website use.

1	This computer beginner learned to navigate but had trouble with clicking. Fearful of theft, she stored the computer in the staff office of her apartment building. She completed all modules in staff office. She did not change her smoking.
2	This intermediate computer user learned the program in the first session and used it independently without difficulty. She quit smoking.
3	This beginner used the computer with coaching. Did not attempt modules on his own. He used the website 2 more times with a coach, but was unable to focus. He cut his smoking in half.
4	This computer owner used the website with coaching in 1st session. He needed several phone calls for help for hardware issues. He completed all modules. He cut his smoking in half
5	This beginner user needed three coaching sessions and was then able to use the program independently. She cut her smoking in half.
6	This intermediate computer user learned to use the program easily with coaching. He did 2 modules independently, but did not want to continue using the program because he had quit smoking immediately after enrolling in the study. He remained abstinent.

All participants said they would recommend the program to a friend, and 83% said they were very satisfied with the program. When asked what could be improved about the program, a third said no improvements were needed, two people said they thought the support group part of the website was hard to use, two said they would like more time with the computer program than the three weeks allowed in this pilot study, and one said it should be mandatory for all smokers. Among the five smokers who used the website program on a home laptop, one (20%) quit smoking and three (60%) cut down on their smoking after using the program for three weeks.

Discussion

Using an evidence-based design to enhance usability (Rotondi et al., 2007, Ferron et al., 2011), we developed a prototype of a web-based smoking cessation intervention and conducted office usability testing to finalize the design and content. With home laptop testing, we found that three weeks of using this prototype cessation website was feasible for smokers with psychotic disorders, was highly satisfactory to the majority of users, and led to promising cessation outcomes.

With several sessions of training, practice and phone or in-person technical assistance, overall participation in the website was high. Interactions with the study coordinator, who coached participants in website use, clearly facilitated this process and may be an important component of technology-based interventions in people with severe mental illnesses (Ben-Zeev et al., 2015, Brunette et al., 2016, Gottlieb et al., 2013). Additionally, the five-dollar gift card reinforcement for using the program may have improved participation, as has been shown in other studies (Carey and Carey, 1990, Helmus et al., 2003).

This program for smokers with severe mental illness impairments (who have very high rates of smoking) would be scalable within settings where a technology coach could be available to provide brief training and support for website use, such as mental health clinics, primary care clinics, or peer support centers (Ben-Zeev et al., 2015, Brunette et al., 2016). A benefit of this web-based approach is that a tobacco specialist is not required to provide this comprehensive behavioral cessation treatment, an advantage to these clinics in which such clinicians are often not readily available. Whether some or all of the technology coaching could take place remotely via phone needs to be tested in the future.

Users of this website found the bulletin board style of the support group more difficult to use than in a previous study of a similarly designed website for people with schizophrenia (Rotondi et al., 2010). Observations of use of the site indicated that people were more interested in using the behavioral modules as compared to the support group, and therefore used it first. They may have become fatigued by spending time with the cessation module, reducing their ability to sustain involvement with the support group. Additionally the small number of people using the support group may have rendered it less interesting. Whether utilization of the support group would have improved with more time, more participants, or voucher reinforcement is unknown. Notably, recent small studies have shown that people with severe mental illnesses are able to use and like similarly designed Facebook user groups for information and support (Aschbrenner et al., 2015). Future work is needed to evaluate adaptations in technology-facilitated support group designs for web-based smoking cessation treatment among people with schizophrenia.

Although this prototype did not contain all of the smoking cessation content planned for the full website, we believe this prototype pilot demonstrated promising efficacy for this approach as well. Twenty percent of home users who smoked quit, and 60% cut down on their smoking after using the prototype for three weeks. These outcomes are in line with outcomes reported for website use in the general population of smokers (Civljak et al., 2013, Etter, 2006).

A key limitation of this developmental study was it's small size, with usability testing in two groups of five or six participants. Past research has indicated that using groups of five enables identification of most usability problems (Rubin and Chisnell, 2008, Nielsen and Landuar, 1993), but testing in larger groups would undoubtedly provide more information (Faulkner, 2003).

Conclusions

In conclusion, this preliminary work indicates that our design for web-based behavioral smoking cessation treatment and use on home laptop computers is feasible among people with schizophrenia and other psychotic disorders. Further study is needed to evaluate its potential for efficacy. Web-based behavioral treatments used in the home environment have the capacity to dramatically extend the reach of behavioral treatment for this population of smokers.

Funding acknowledgement

Funding for this research was provided via a pilot grant to Brunette from NIDA P30DA029926 (Marsh).