The feasibility of using electronic clinical outcome assessments in people with schizophrenia and their informal caregivers.

Many clinical outcome assessments (COAs) were originally developed for completion via pen and paper. However, in recent years there have been movements toward electronic capture of such data in an effort to reduce missing data, provide time-stamped records, minimize administrative burden, and avoid secondary data entry errors. Although established in many patient populations, the implications of using electronic COAs in schizophrenia are unknown. In accordance with International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force recommendations, in-depth cognitive debriefing and usability interviews were conducted with people with schizophrenia (n=12), their informal (unpaid) caregivers (n=12), and research support staff (n=6) to assess the suitability of administration of various electronic COA measures using an electronic tablet device. Minimal issues were encountered by participants when completing or administering the COAs in electronic format, with many finding it easier to complete instruments in this mode than by pen and paper. The majority of issues reported were specific to the device functionality rather than the electronic mode of administration. Findings support data collection via electronic tablet in people with schizophrenia and their caregivers. The appropriateness of other forms of electronic data capture (eg, smartphones, interactive voice response systems, etc) is a topic for future investigation.

Introduction

Clinical outcome assessments (COAs) measure a patient’s symptoms, overall mental state, or the effects of a disease or condition on how a patient functions and can be used to determine whether or not a health intervention has demonstrated a benefit (or otherwise) to recipients. This umbrella term encompasses many types of assessments including patient-reported outcomes (PROs), observer-reported outcomes (ObsROs), clinician-reported outcomes (ClinROs), and performance-related outcomes.

ClinROs and PROs are commonly used in schizophrenia to facilitate clinical diagnosis and evaluation of treatment benefit. There are a number of symptoms and impacts of schizophrenia which cannot be reliably reported by people with schizophrenia themselves, but may also not be observable to clinicians. Observer-reported outcomes are therefore often used to provide unique insights into observable signs and symptoms and levels of functioning in people with schizophrenia. Furthermore, the impact that caring for a person with schizophrenia can have on informal caregivers themselves is increasingly being recognized1 and assessed as a means to understand the wider burden of disease.

Traditionally, COAs have been developed and validated using pen and paper. However, as digital technology has evolved, the use of electronic COAs (eCOAs) has become more prevalent. The use of eCOAs is associated with a number of advantages over pen-and-paper methods, particularly in terms of reducing missing data, providing time-stamped records, minimizing administrative burden, and avoiding secondary data entry errors.2 When migrating an instrument from a paper to an electronic mode of administration, the resulting eCOA ought to produce data that are equivalent or superior to the data produced from the original paper version and this equivalency should be solely due to the change in mode.2 In addition, it is essential to demonstrate the “usability” of the eCOA (ie, that respondents are able to use the software and the device appropriately).2

Although eCOA has been implemented with success in many populations, there is limited evidence regarding the feasibility of using eCOAs with people with schizophrenia. It has been suggested that people with schizophrenia are less “digitally proficient” than healthy controls3,4 and are less likely to own a mobile device than the general population or people with other types of mental disorders.5 In addition, lower socioeconomic status6 and cognitive difficulties7 observed in schizophrenia populations are factors associated with less frequent use of computers.8,9 Importantly, the issues encountered in this population are likely to differ according to the type of electronic device (eg, mobile versus on-site assessments), study design (eg, frequency of assessments), and the content of the instruments used (eg, recall of past week versus in-the-moment ratings).

The aims of the present study were to provide a comprehensive assessment of the feasibility of people with schizophrenia completing a range of validated PROs using a tablet computer in the context of an on-site clinical visit. This is in contrast to existing research which has centered predominately upon ecological momentary assessment using mobile technologies.10–12 In addition, the present study also explores usability of eCOA devices for completion of a range of measures by caregivers of people with schizophrenia. Lastly, the ability of research support staff members to instruct and support participants when using an eCOA device is also explored.

Methods

Usability and cognitive debriefing interviews were performed with 12 people with schizophrenia, their 12 informal caregivers, and six members of research support staff. An informal caregiver was defined as “A person who has significant responsibility for managing the well-being of a person diagnosed with schizophrenia in an unpaid capacity”.13 Research support staff were defined as staff members within a physician’s office who would frequently interact with and administer questionnaires to patients.

A range of disease-specific and generic self-report instruments were debriefed among participants. Specifically, people with schizophrenia completed the following instruments on the electronic device:

Schizophrenia Quality of Life Scale (SQLS).14 A 30-item disease-specific instrument assessing the impact of a person with schizophrenia’s symptoms on their quality of life.

EuroQOL 5 Dimension (EQ-5D).15 A generic instrument comprising five items assessing mobility, self-care, usual activities, pain/discomfort, and anxiety/depression and one global item assessing the respondent’s perception of their health that day on a 0–100 visual analog scale (VAS).

Short-Form 36 ([SF-36] v 2.0).16 A generic 36-item instrument assessing general health status which comprises eight domains that broadly assess physical and mental functioning.

Caregivers completed three instruments on the device:

Schizophrenia Caregiver Questionnaire (SCQ).1,17 A 32-item disease-specific instrument assessing the impacts of caring for a person with schizophrenia on the informal caregiver’s quality of life.

Caregiver Global Impression scales (CaGI).1,17 Three single-item measures to assess severity of the person with schizophrenia’s symptoms, change in the person with schizophrenia’s symptoms, and change in the experience of caregiving on a six- or seven-point Likert scale.

SF-36 (v 2.0).16 As described above.

Instruments appeared on the device as one item per screen, with the exception of the EQ-5D, which had all five items on one screen and the VAS on a separate screen.

The changes from paper to electronic versions of the instruments used in this study were non-substantive (eg, small changes in instructions, changing to one item per screen) and no changes were made to item content or meaning. This level of modification is classed as minimal according to International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force guidelines and US Food and Drug Administration (FDA) guidance,2,18 and, accordingly, cognitive and usability testing interviews are considered appropriate for providing evidence of equivalence.

Ethics

The study was conducted in accordance with the Declaration of Helsinki and was approved by Copernicus Group, a centralized independent review board (IRB) in the US. (IRB number: ADE2-12-453). Study procedures ensured that written informed consent was obtained from people with schizophrenia, their caregivers, and the members of research support staff prior to the collection of any data. All participants received financial compensation for the time spent participating in this study.

Recruitment

Twelve people with schizophrenia and their 12 informal caregivers were recruited into the study via referrals from private physicians. In addition, six members of research support staff were recruited. The sample size is deemed adequate given that ISPOR electronic patient reported outcomes (ePRO) Task Force guidelines recommend five to ten participants for studies of this type.2 Participants were recruited from three sites in the US to avoid geographical bias: Baltimore, MD (eastern USA), St Louis, MO (central USA), and New Orleans, LA (southern USA).19,20 To be included in the study, participants had to meet a number of predefined inclusion and exclusion criteria (Table 1). A series of quotas were employed to ensure a representative sample (Table 1).

Table 1

Study recruitment strategy

Study inclusion criteria
People with schizophrenia	• A medically confirmed diagnosis of schizophrenia according to DSM-IV-TR classification criteria• Currently taking an atypical or typical antipsychotic• Aged ≥18 years• In good general health aside from their schizophrenia
Caregivers	• Identified as the primary informal caregiver of a person with schizophrenia (as reported by both the person with schizophrenia and the caregiver themselves)• Provides care to the person with schizophrenia for at least 4 hours per week• Aged ≥18 years
Support staff members	• Identified as the person at the site who would typically be responsible for administering questionnaires to patients
Recruitment quotas and justification
DSM-IV-TR schizophrenia subtype	Previous research implied that behaviors associated with certain subtypes of schizophrenia may have differential effects on the ability to complete instruments on an eCOA device.10,26 Efforts, therefore, were made to recruit a sample representative of different DSM-IV-TR subtypes and varying manifestations of positive and negative symptoms associated with schizophrenia
Schizophrenia severity	Efforts were made to recruit participants of varying severities of schizophrenia. A CGI-S scale was completed by recruiting physicians as a means of assessing symptom severity. Representation of people with schizophrenia with CGI scores from 3 (mildly ill) to 6 (severely ill) was targeted. Note: people with a CGI score of 7 (among the most extremely ill patients) were judged by the recruiting physicians as unsuitable for participation in this study due to their symptomology and the fact that they would likely be receiving formal or paid care
Caregiver age	A quota was implemented to recruit a representative sample of caregivers both over and under 60 years of age.
Caregiver education	It was ensured that at least 20% of the caregivers had high school as their highest level of education

Abbreviations: CGI, Clinical Global Impression; CGI-S, Clinical Global Impression of Severity; DSM-IV, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; DSM-IV-TR, DSM-IV Text Revision; eCOA, electronic clinical outcome assessment.

eCOA device

The eCOA device used in this study (SITEpro®; eResearch Technology Inc. Philadelphia, PA, USA) is a tablet with a 30.7 cm screen (Motion Computing J3500 Tablet PC with a Windows® 7 Professional Operating System). The device weighed approximately 900 g and moved between screens at a maximum of 200 ms. Text was 16 point throughout the device. The devices were locked to functionalities specific to this study in order to simplify their use. Tablet devices are more portable than a traditional desktop computer and lighter and easier to hold than a laptop computer. Such devices also have fewer functional limitations in terms of the display (font size and VAS selection) than alternative mobile devices such as a personal digital assistant or smartphone.2

Interview procedure

All interviews were conducted by a trained interviewer using a semi-structured interview guide. Prior to the conduct of interviews, study staff members were asked to prepare the device for use, following instructions in a study-specific user manual. Staff members also received training on how to assist people with schizophrenia and their caregivers to use the device. People with schizophrenia and their informal caregivers were then interviewed concurrently by two interviewers. In the event that participants encountered difficulties, staff members/interviewers were available to assist the person with schizophrenia or caregiver in completing the instruments using the electronic device (but as the instruments are developed for self-completion by the respondent, staff members/interviewers were not to provide interpretation of instrument content).

Once the instruments had been completed, participants took part in a semi-structured interview to assess usability. Questions were initially broad (eg, “What did you think about using the eCOA device?”) and became more focused to ensure that all areas of interest were covered (eg, “What did you think of the layout of the questions on the device?”). Participants were also debriefed about any particular items they found difficult to answer and about specific aspects of the instruments which were modified in migration to electronic form. Once each staff member had assisted one person with schizophrenia and one caregiver in completing the instruments, they were also interviewed about their opinion of the ease with which people with schizophrenia and their caregivers were able to complete measures using the eCOA device and their experience of assisting participants in doing so.

Analysis

All interviews were audiotaped and transcribed verbatim for the purpose of qualitative analysis. Written interview transcripts were then entered into a qualitative software package (Atlas.Ti), which was used to facilitate the analysis of interview transcripts. Codes were assigned to statements which were then categorized into relevant domains. Analysis was conducted according to participant group, ie, people with schizophrenia, caregivers, and support staff.

Results

Demographic and clinical characteristics

The average age of people with schizophrenia recruited to take part in the study was 47 years, and males and females were equally represented (n=6). In accordance with recruitment quotas, people with a range of schizophrenia subtypes and severities were recruited (Table 2). The majority of caregivers were female, but male caregivers were adequately represented in the study sample (25%), and the majority of caregivers (83%) reported living with the person with schizophrenia. Caregivers reported having spent an average of 14 years caring for the person with schizophrenia and spending, on average, 47 hours per week providing such care. All but one of the support staff members reported having prior experience of administering questionnaires to patients in electronic form.

Table 2

Demographic and clinical characteristics

Demographic characteristic	Person with schizophrenia	Caregiver	Research support staff member
Age (years)
Mean	47	57	44
Range	23–83	24–70	25–69
Sex, n (%)
Male	6 (50)	3 (25)	0
Female	6 (50)	9 (75)	6 (100)
Work status, n (%)
Working full or part time	4 (33)	6 (50)	6 (100)
Looking for work	1 (8)	0	0
Full-time homemaker	1 (8)	0	0
College or university student	0	0	0
Not working due to schizophrenia diagnosis	5 (42)	2 (17)	0
Retired	1 (8)	4 (33)	0
Other	0	0	0
Race, n (%)
Black/African American	9 (75)	9 (75)	5 (83)
White/Caucasian	3 (25)	3 (25)	1 (17)
Years since diagnosis, n (%)
0–10	5 (42)	N/A	N/A
11–20	2 (17)
20+	5 (42)
Range	2–33
Mean	14.8
Schizophrenia DSM-IV-TR subtype, n (%)
Paranoid	5 (42)	N/A	N/A
Disorganized	3 (25)
Undifferentiated	3 (25)
Paranoid, disorganized	1 (8)
Schizophrenia severity, n (%)
CGI-S score 3	4 (33)	N/A	N/A
CGI-S score 4	4 (33)
CGI-S score 5	3 (25)
CGI-S score 6	1 (8)
Medication, n (%)
Typical antipsychotic	2 (17)	N/A	N/A
Atypical antipsychotic	9 (75)
Antidepressant	6 (50)
Anti-anxiety	3 (25)
Other	3 (25)
Education, n (%)
High school diploma or GED	N/A	5 (42)	0
Some years of college		3 (25)	3 (50)
Certificate program		0	2 (33)
College or university degree (2- or 4-year)		3 (25)	0
Graduate or professional degree		1 (8)	1 (17)
Living with person with schizophrenia, n (%)
Yes	N/A	10 (83)	N/A
No		2 (7)
Relationship with person with schizophrenia, n (%)
Parent	N/A	4 (33)	N/A
Partner/spouse		2 (17)
Sibling		2 (17)
Son/daughter		2 (17)
Friend		1 (8)
Grandparent		1 (8)
Years spent caring for person with schizophrenia, n (%)
0–10	N/A	8 (67)	N/A
11–20		1 (8)
20+		3 (25)
Range		3–41
Mean		14
Hours per week spent caring for person with schizophrenia, n (%)
<20	N/A	5 (42)	N/A
21–40		3 (25)
40+		4 (33)
Range		5–40+
Mean		47
Years in current job
0–5	N/A	N/A	4 (67)
6–10			1 (17)
11–15			0
15+			1 (17)

Abbreviations: CGI-S, Clinical Global Impression of Severity; DSM-IV, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; DSM-IV-TR, DSM-IV Text Revision; N/A, not applicable or not asked of that population; GED, General Educational Development.

People with schizophrenia

Device use

All 12 of the people with schizophrenia reported a positive experience of completing the instruments on the eCOA device, with nine (75%) spontaneously commenting that it was easy to use and four (33.3%) stating that it was easier than using pen and paper despite some patients having little experience using computers, smartphones, and tablets. When probed, all 12 of the people with schizophrenia said that they found it easy to use the eCOA device. Seven of the people with schizophrenia specifically commented that they liked the size of the device as it made the text easy to read. However, two people did state that the device was too big and could be made smaller. All of the people with schizophrenia said that they found it easy to use the touchscreen to select answers and three people specifically commented that the ability to see/hear what they had selected on the screen was useful. The majority of people (n=10; 83.3%) said that the device always responded to their touch first time, which was consistent with interviewer observations. When asked, most people liked the speed at which the device responded, although one person said that the device responded to their inputs “too fast”. Table 3 presents the participants’ quotes.

Table 3

Participant comments on electronic clinical outcome assessment completion

Participant quotes
General thoughts	Person with schizophrenia• “It was easy. Um, it was easy. It was very simple.” (BM-P1-F-40-R)• “Because all you had to do was read it – read it and tap – and tap it at the bottom.” (NO-P3-M-42-P)• “I just think it was easier than dealing with a pen and paper … It was pretty simple and straightforward.” (SL-P5-M-47-U)
	Caregiver• “It’s very easy to use. It’s fast.” (NO-C2-F-57)• “The questions were clear and the answers were clear. And you just press the one that you want to.” (SL-C1-F-70)
	Staff member• “I think it was very easy – to maneuver through, very self-explanatory.” (BM-S2-F-35)• “It’s more clear-cut and simple.” (SL-S1-F-42)
Appearance and size	Person with schizophrenia• “Absolutely perfect. The writing is bold. Um, I put my glasses on, but I would’ve probably not really needed – you know, to have them on. The size of the letters and stuff like that. It just things was – it was just perfect.” (BM-P3-F-55-R)• “It’s a little big. I mean, you’re just answering questions on it. Uh, it doesn’t have to be that big.” “It could be smaller, though, because it would be easier to hold in your hand and what have you.” (BM-P1-F-40-R)
	Caregiver• “The screen is perfect. The size is really good.” (SL-C4-F-46)• “I – what I don’t like about it is like I wanted to kind of put it down when I did it, you know, like that … It’s so heavy.” (SL-C1-F-70).• “I would change the outer case and probably make it thinner.” (NO-C5-M-57)
	Staff member• “And it’s, uh, reliable – and durable. And I think that would be the most important parts.” (NO-S1-F-59)• “I think it’s a little big. Um, it could be smaller. Um – And it’s a – it’s a – the weight is a little heavy.” (NO-S3-F-35)
Touchscreen	Person with schizophrenia• “Because I just had to tap. I didn’t have to write.” (NO-P1-F-42-P)
	Caregiver• “Moving on to the next thing took maybe a little longer than I’m used to, but it wasn’t an impediment at all.” (SL-C3-F-69).
	Staff member• “And you don’t have to keep pressing it or maybe even put in a – trying to pluck it out – to – it’s just touch screen is just easier.” (NO-S1-F-59)
Question layout	Person with schizophrenia• “They were good. It was set up well.” “Q: The layout of it?” “Yeah, the layout. It was good, it was simple.” (BM-P2-F-53-R)
	Caregiver• “I thought the questions were very good and I thought they were worded good and the layout was very well – where you could just read it and it wasn’t hard to understand, because it’s pretty straightforward.” (SL-C4-F-46).• “And that top part, every time I went to – a question, that top part came back up. So why continue to, uh, include that? You know, I mean, let it be one time and just move on to the questions.” (NO-C4-F-63).
	Staff member• “It’s very clear. Very clear. If you read, uh, you know, the questions, they got straight to the point.” (NO-S2-F-25)
Text size	Person with schizophrenia• “It was an easily readable font.” (SL-P5-M-47-U)• “Well, or not necessarily make the text bigger or maybe give you an option that would allow – like if you realized that you couldn’t read it very well because the text was smaller maybe something that would enlarge it that way.” (SL-P5-M-47-U)
	Caregiver• “The size of the type and the, um the spacing … I thought it was clear.” (SL-C3-F-69)
	Staff member• “Um, it wasn’t too big. It wasn’t too bright.” (BM-S2-F-35)• “We had some elderly people, and, uh – would be easier – I changed the fonts – be able to change the font size. That’s the only thing, so.” (BM-S1-F-69)
Completion time	Person with schizophrenia• “It was pretty fast. It didn’t take a lot of time at all actually.” (SL-P1-M-31-P)
	Caregiver• “It didn’t take long at all. Before I knew it, I was at the end.” (SL-C4-F-46)
	Staff member• “Actually, I thought it was pretty fast. I thought they answered it, uh, pretty quickly.” (NO-S2-F-25)• “And a lot of times it depends on the patient. If the patient understands and how fast or slow they read.” (NO-S3-F-35)
Ability to go back	Person with schizophrenia• “I thought about it. I – I thought that maybe in a couple I should have – um, I didn’t – I didn’t answer, like, 100% truthfully, but – Not – not that it wasn’t 100% truthfully. It’s – just say I can’t – I couldn’t make up my mind on the answer.” (BM-P1-F-40-R)
	Caregiver• “I just wanted to check and make sure, like, it was accurate.” (NO-C2-F-57)
Number of items per screen	Person with schizophrenia• “See, sometimes things get complicated. That confuse somebody, or they get anxiety when they see all these questions on paper. Oh Lord, this is a test! You know? But here – I was in a good mode doing what I was doing when I focus.” (BM-P2-F-53-R)• “And I liked the way that it was one question per page, not a whole looming page of questions that makes it look much more intimidating.” (SL-P5-M-47-U)
	Caregiver• “You can focus on this one question – without drifting – on the other questions.” (NO-C5-M-57)
Moving to next questionnaire	Person with schizophrenia• “Maybe the first one should have been highlighted or had a check box next to it so you knew you always – you had already done that. Because whenever we got to – through like the third questionnaire and there was the fourth one there – I kind of paused going, OK, which one did I just come out of?” (SL-P5-M-47-U)
	Staff member• “He knew to go to the next one. So I didn’t even have to tell him to go.” (NO-S1-F-59)• “Yeah. I – I think probably it could have just simply said, eh, the word completed … And just say – and the next screen – and to complete it, tap. And then the next section could have said, start next section.” (BM-S1-F-69)
Repeated completion over 3 months	Person with schizophrenia• “ I think, if I knew that I was answering it on the same type of device I think I would probably agree to it faster than I would if it was paper and pencil.” (SL-P5-M-47-U)
	Caregiver• “That would be fine. It’s – it’s really quick and – easy, like – and I mean, like I said, for the amount of information I filled it out – filled out – it wasn’t – the timing wasn’t bad.” (BM-C2-F-24)
Assisting completion	Staff member• “The questions for the schizophrenia patients, um, were a little – a little bit easier than the caregiver questions. They were more easy for them to read, and it was a – a lot shorter, straight to the point questions. They were – to which the caregiver questions were a little more in – in dep – in detail.” (NO-S3-F-35)• “Yes. Yeah, because he – he caught on fast.” (NO-S1-F-59)

Notes: Subject IDs are provided in parenthesis after quotes. IDs refer to site, subject type, gender, age and DSM-IV-TR subtype.

Abbreviation: DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition.

Instrument completion

The majority of issues to arise during instrument completion related to the content of the instruments rather than the electronic format per se. However, although there were no issues with understanding of the content of the EQ-5D, there were some issues specific to the electronic format of the EQ-5D in this study. The EQ-5D items were all displayed on one screen, and three of the 12 people with schizophrenia needed to confirm with the staff member/interviewer how the items were separated. In addition, five of the 12 people with schizophrenia had trouble selecting their response on the 0–100 VAS, either with the stylus or their finger. This was most likely a result of the narrow width of the scale. There were no trends with regard to age, schizophrenia subtype, computer familiarity, or issues making selections on the device.

Responses to the item layout were generally positive (n=11) and the majority of people found the text easy to read (n=10), although three participants suggested either making the text bigger or providing an option to make the text bigger. Most had no problem with the time it took to complete the instruments, even though they were completing multiple instruments using a think-aloud technique, thus increasing completion time. Only one person with schizophrenia said it took a long time to complete the instruments and, interestingly, this person was observed to complete the instruments in a shorter amount of time than others. Only two of the people with schizophrenia needed to go back and change their answers within an instrument. Reasons for this included confusion over what the item was asking or uncertainty over their chosen response.

Generally, individual items were presented on one screen, unlike a paper instrument, which would contain all items and thereby make it possible for participants to read all the items before responding. Five of the 12 people with schizophrenia said that they would have answered differently if they had been able to see all of the items at once; however, all who were asked said they would prefer to see the items individually on separate screens. Nine people with schizophrenia said that they found it easy to move onto the next instrument; however, some people suggested the device should show more clearly which instruments had been completed. When asked, all of the people with schizophrenia said that they would be willing to complete instruments on an eCOA device every 3 months.

Table 3 presents the participants’ quotes.

Caregivers

Caregivers of people with schizophrenia experienced few difficulties completing instruments on the eCOA device. Caregivers had a varied level of experience using computers, smartphones, or tablets and ranging from never having used a computer to using one every day. All caregivers reported finding the device easy to use, with seven saying that they preferred using an eCOA device to pen and paper, and some caregivers also said that they liked the speed and size of the device. However, device-specific issues mentioned by caregivers were that it was too heavy or bulky (n=4), and that the speed of response was slow (n=3). The only issue caregivers had with instrument completion was a dislike of repetitive statements (n=3). For example, “The following question is about activities you might do during a typical day” which appears for every question in the electronic format of the SF-36 but just once in the paper format, and “Over the past four weeks” which appears in every question in both paper and electronic formats of the SCQ. For the SF-36, this instruction appears for every question on the electronic version but once in paper format. There did not appear to be any differences in the issues experienced based on the age or computer familiarity of caregivers. Table 3 presents the caregivers’ quotes.

Staff members

All staff members used computers daily at work; however, some had little experience of using smartphones or tablets. Generally, staff members found the device easy to set up and to use. The issues staff members experienced with the device itself were associated with the need to press an arrow symbol to move onto the next question (n=1) and the use of a stylus to select answers (n=1), the size of the device (n=2), the orientation of the screen (n=1), the speed of response (n=1), and the size of the text (n=1). None of these issues, however, were a barrier to preparing the device for use by study participants.

Staff members also found it easy to guide people with schizophrenia and their caregivers on how to complete instruments on the device. They reported that people with schizophrenia and their caregivers found it easy to use the device itself; only one staff member reported that some people with schizophrenia found it hard to move onto the next question using the arrow. Staff member suggestions for improvements to the device included an option to increase text size and putting the device on a stand to improve readability.

There were some concerns from staff members that people with schizophrenia were not using the correct recall period but were thinking back over a longer period of time, and this was also noted by interviewers. Specifically, only one person with schizophrenia used the correct recall period when completing the SQLS, and less than half of the caregivers used the correct recall period when completing the SCQ. This issue is not specific to an electronic mode of administration, and participant understanding and use of the recall period for each instrument, in both paper and electronic format, could be further investigated.

Table 3 presents the staff members’ quotes.

Discussion

Although there is evidence of the use of electronic data capture on mobile devices (eg, ecological momentary assessment) among people with schizophrenia,10–12,21 the feasibility of administrating eCOAs on-site in tablet mode has not been assessed. For many clinical research studies, collection direct from patients and their caregivers during scheduled site visits presents the most viable option of data collection (practical reasons, ensuring standardization of timing of assessments, etc). It is important, therefore, to assess the feasibility of data collection (and specifically the usability of electronic devices) used to collect data in such instances.

Findings from the current study provide evidence that people with schizophrenia and their caregivers are able to complete COA instruments using an electronic device, with many preferring this mode of completion to traditional pen-and-paper methods. These findings support those from previous studies where no differences in ease of device use between people with schizophrenia and healthy controls were observed.21,22 That there was no difference in ease of completion of eCOAs according to participants’ familiarity of using computers or similar electronic devices is also consistent with previous findings.12,22 Furthermore, no differences in ease of completion according to respondent age, symptom severity, and education were observed. Despite the small sample size included in this study, the finding that variables such as age, computer familiarity, and symptom severity did not affect ease of device use or eCOA completion may indicate that on-site eCOA could also be a feasible method of data collection in other mental illnesses or elderly populations (subject to further research).

Observations from this study highlight several considerations for future use of eCOAs. Although only a small number of difficulties were reported by participants, findings from the study highlight the importance of having a staff member available nearby to assist and resolve any issues encountered (specific to the electronic platform) if the eCOA is being completed in a clinical trial setting.

Repeated assessments are a requirement for all interventional studies that seek to demonstrate change over time, and there is a balancing act between the collection of information from participants and responder burden. In this study, participants indicated that they would not find completing repeated eCOAs over a period of time overly burdensome, suggesting that this should perhaps be less of a concern. This is supported by a study by Arbuckle et al, which demonstrated that study participants completing eCOAs at multiple time points could do so quickly and did not find it overly burdensome, suggesting concerns over respondent burden may often be unfounded.23

Participants preferred to see individual items on separate screens, ameliorating concerns raised by Gwaltney et al4 that having one item per screen removes the option for participants to review all of their answers. Participants struggled to select an answer on the EQ-5D VAS, likely because of the width of the scale on the device screen, and despite the selected value being displayed on the device, difficulties selecting an answer could lead to responses that are not reflective of true health status. The EQ-5D is a widely used measure, and relatively small between and within participant differences on the VAS (7–10 points) have been identified as minimal important differences in other populations.24 This is an important consideration, especially when participants are completing measures off-site (eg, at home) rather than on-site and help is less accessible.

Findings from the study suggest that repetition of recall period on each screen does not eliminate incorrect recall by participants. Based on these findings alone, it is difficult to determine whether this applies to all modes of administration or is specific to eCOA. To some extent, the design of the current study (ie, in-depth debriefing of measures following completion at a single time point in a research facility) may have influenced findings. However, incorrect recall has previously been demonstrated in both paper and electronically administered COAs.25 Regardless of mode of administration, incorrect use of recall presents challenges to evaluating changes on respondent-reported parameters over time and is something to consider when using instruments in this manner.

Feedback regarding the usability of eCOA was consistent among people with schizophrenia and their caregivers. Accordingly, none of the issues encountered in this study suggest that people with schizophrenia are any less able to use eCOA than healthy individuals; however, further investigation is required in this area. It is noted that the sample in this study is limited to the US and does not include people with residual schizophrenia or people with schizophrenia who cannot identify a formal caregiver. A conceivable limitation of the study is therefore that the findings may differ geographically and in people with residual schizophrenia or with no informal caregiver. This could be a topic for further investigation. Modern technology provides endless opportunities for real-world and real-time data capture and future research could investigate the feasibility of utilizing more advanced mobile technologies (eg, participants’ own personal tablets and smartphones) to collect COA data in people with schizophrenia and their caregivers on devices that are already familiar to them. Other uses of electronic devices (eg, assessing adherence and compliance to treatment, patient education, and tracking of symptoms) among people with schizophrenia and their caregivers could also be an interesting target for future research.

Conclusion

Providing that the instruments are laid out clearly, instructions are understood, and the device is easy to use, eCOA is a feasible method of on-site data collection in people with schizophrenia and their caregivers. This appears to be the case regardless of clinical or demographic characteristics of respondents (eg, schizophrenia severity, age, or computer familiarity). In addition, research support staff members can easily assist people with schizophrenia and their caregivers in completing assessments on an eCOA device.