Internet-delivered mental health treatment systems in Scandinavia - A usability evaluation.

Mental health problems are a major public health concern worldwide. Approximately 50% of the population will experience mental problems during their life. Traditional treatment is based on psychopharmacotherapy or psychotherapy, with face-to-face interaction between the patient and the therapist. New technologies such as Internet-delivered treatments are seen as an opportunity to offer more scalable and cost-efficient treatments in the field of mental health. Despite the growing interest and new evidence supporting the effect of Internet-delivered treatments is it remarkably little research on how the technology and the usability of Internet-delivered treatment programs affects the treatment. In this paper, we propose a set of evaluation criteria for evaluating the usability and the responsive design of Internet-delivered treatment systems. By our knowledge we are the first to include usability and universal design principles in the evaluation of Internet-delivered treatment systems. Our findings indicate that despite the good treatment results and proven clinical effects, the systems in general have several issues regarding usability, universal design and outdated technology. Based on our findings we propose that there should be established guidelines for testing the usability and technology of Internet-delivered treatment systems.

Introduction

Mental health problems are a major public health concern worldwide. Treatment is mainly delivered as psychopharmacotherapy or psychotherapy, with face-to-face interaction between the patient and the therapist. However, new technologies such as Internet-delivered treatments are seen as an opportunity to offer more scalable and cost efficient treatments in the field of mental health (Andersson and Cuijpers, 2018; Andersson et al., 2014; Titov et al., 2018a; Ebert et al., 2018). Internet-delivered psychological treatments often make use of the same therapeutic principles as evidence–based face-to-face psychological treatments. The difference is that the therapeutic content is provided via the Internet on computers, tablets and/or smart phones. The structure of the Internet-delivered treatments is similar to that of regular face-to-face treatments; content and assignments are dispatched weekly with the aim that patients will practice and apply the techniques in their daily lives. Most Internet-delivered treatments provide text-based information and are highly structured with systematically presented psycho-education, assignments (homework), symptom assessment through questionnaires and other resources (e.g. activity monitoring, electronic (thought) diaries) and are often supplemented with audio, video and animations. The treatments can be provided with some level of guidance of a clinician or coach via (asynchronous) email or a secure messaging system, or (synchronous) online chat, video-consultations or telephone calls. Internet-delivered treatments can also be entirely self-guided with no or automated feedback on (homework) assignments, symptom level etc. A considerable amount of research has shown that Internet-delivered CBT (ICBT) can be effective means in the treatment of common mental disorders (Andersson et al., 2014; Titov et al., 2018a; Ebert et al., 2018). When compared with traditional face-to-face treatment, clinician guided Internet interventions can even be as effective when treating depression and anxiety (Andersson et al., 2014). One important finding that emerges from earlier meta-analytic studies of Internet-delivered treatments is the positive correlation between time spent on the program and clinical effects (Farrer et al., 2014; Baumeister et al., 2014; Karyotaki et al., 2018). Reflecting the weight of this encouraging evidence regarding Internet-delivered treatments, several health care providers internationally now offer guided Internet-delivered treatment as part of routine care in i.e. Europe, North-America and Australia (Titov et al., 2018b). In Scandinavia, Sweden, Denmark, and Norway all have clinics which provide guided Internet-delivered treatment for depression as part of routine care (Folker et al., 2018). However, nowadays Internet-delivered treatment systems are often developed with limited resources, which often results in issues concerning usability, personalized treatment, interactive treatment, and adaptable treatment, both from the patient and the health service point of view. In order to increase the impact of Internet-delivered treatments in routine care, one should investigate how the technology and the usability of the system correspond with the treatment outcome. It should be systematically studied what treatment effect the following factors have: is the system in line with the user requirements, is the system adaptable to what patients want and to what patients need (e.g. based on the patients educational level, motivation and functional level). By meeting the needs of the patients, the motivation for and the adherence to the treatment might be changed (increased), this is important since adherence is linked to positive outcome in ICBT (Titov et al., 2018a). Simpson et al. (2012) published a study indicating that adherence was related to improved outcome for Obsessive compulsive disorder. However, we are not aware of any systematic research studying the relation between usability, engagement and adherence of Internet-delivered treatment. In this paper, we take a step in this direction by identifying a set of evaluation criteria and performing a systematic review of Internet-delivered treatment systems with emphasis on usability aspects based on the WCAG 2.0 framework.

More specifically, we study the usability of the system and how the patient is interacting with the therapist and the health care system, we also study the responsiveness of the system i.e. how well the system adapts to different platforms. For each system, we also give some brief background information about the clinical usage, including the methodology, and the inclusion criteria for the depression treatment. Five systems in Norway and Sweden for adults with depression, the most common mental health disorder, will be examined.

Method

Procedure

The assessment of the Scandinavian systems was conducted in the spring 2018 and was done by one ICT researcher taking the role as a patient. The researcher conducted a full intervention in one day in each Internet-delivered treatment system. The researcher could experience how good the usability and the accessibility were by taking the role as a patient on a PC, and on an android mobile phone to evaluate if the system uses responsive web design. We used two test tools to test whether the Internet-delivered treatment systems meet the success criteria from WCAG 2.0. The first tool, W3C checker1 was used to test whether the system meets the programmatic successes criteria, e.g. if there is a text alternative for a picture. The tool specifies which criteria are not met and where the source code error exists. The second tool, Color Contrast Analyser2 check whether the contrast satisfies the criteria according to WCAG 2.0. Some success criteria need to be verified manually, e.g. the reading level.

Assessment

Clinical usage

Target group

The inclusion criteria for the treatments are (with some local differences) being adults, diagnosed with a current major depression, stable medication and reading and writing the language of the treatment program. Exclusion criteria (with some local differences) are ongoing suicidal ideation, other severe disorders with an immediate need of treatment, no computer or Internet access.

Recruitment of patients

The ICBT treatment programs in the Scandinavia Internet clinics are through referrals from a General Practitioner (GP) and self-referrals from the general population.

Treatment model

All programs in the study are mainly based on the model of CBT.

Therapist support

Programs with and without asynchronous therapist guidance via the system is included.

Usability

Usability assess the quality of the user's experience when interacting with a product or system e.g. a user interface on a website, including challenges regarding universal design and user-friendliness according to the standard Web Content Accessibility Guidelines 2.0 (WCAG 2.0) (Caldwell et al., 2008). WCAG 2.0 is a standard for evaluating usability of software systems that takes into account new technologies, different user agents (any software that retrieves and presents web content for users) and universal design in sufficient degree (Caldwell et al., 2008). The standard consists of four general principles and 12 guidelines, including 61 testable requirements called success criteria on the following sale: A (lowest), AA and AAA (highest). To be qualified for a level, a system needs to fulfill all the success criteria for that level. The four main principles are as follows: 1) Perceivable: The users must be able to perceive the information being presented; 2) Operable: The users must be able to operate the interface; 3) Understandable: The users must be able to understand the information as well as the operation of the user interface; 4) Robust – The users must be able to access the content as technologies advance.

Interaction

To evaluate the user interaction of the Internet-delivered treatment systems we have built on Kraut et al. (Kraut et al., 2002) terminologies for affordance in communication media (interaction through the technology). The goal of evaluating the interaction with the system is to map the functionality they provide and evaluate how user-friendly they are. The four chosen terminologies to describe the interaction possibilities through the systems are: 1) Communication, communication mechanisms, functionality and Roles (Table 1).

Table 1

The evaluation criteria for interaction.

Criteria	Description
Communication	In which degree is there a direct interaction between the patient and therapist?Is the communication asynchronous or synchronous, automated feedback?
Communication mechanism	What kind of communication mechanisms are used?i) Video ii) Telephone iii) Emailiv) Chat/SMS
Functionality	What kind of functiionality is supported?i) Chat ii) Exercises iii) Forum iv) Questionnaires v) Diary vi) Video lectures vii) Audio lecture viii) Animation
Roles	Which roles do the systems support? i) Patient and therapist ii) Group therapy (many patient and one therapist)

The criteria used for evaluating responsive web design and compliance to web standards covers navigation, content, image, video, typography and touch controls (Table 2).

Table 2

Criteria for responsive web design (Bohyun, 2013; Marcotte, 2011; Institute of Electrical and Electronics Engineers (IEEE), 2012).

Criteria	Description
Navigation	Make sure the maximum display area is available for the main content.Check if they use a drop-down bar for small phones.Allow columns to reduce in width up to a point for tablets, but they should then collapse under each other, allowing the user on smart phones (small) and on smaller tablets to scroll through the content.
Content	Ensure any content that is not the primary point of the page is not surrounded by auxiliary content.The website has configured the viewport (the user's visible area) i.e. adapt the width of the page to the screen width of the device.
Image	The images should be resized to fit with the resolution of the screen without losing the fidelity of the images.
Video	Supporting newer HTML5 formats for web video like Web-M, Theora Ogg and H.264 instead of Flash player.
Typography	Typefaces should be carefully considered since clarity is important, especially when being read devices.
Touch controls	Touch control placement: options in the lower half of the screen will generally be easier to control with a single hand than in the upper portion of screen.It should be easy to click on links, buttons and other clickable items. The size of control target sizes should be minimum of 44px in width or height at 72 dpi.

Results

In this study we have investigated five Internet-delivered treatment systems and reviewed how they comply with universal design principles and responsive design. 1) eMeistring provides guided Internet-delivered treatment in secondary outpatient mental health clinics in Norway. The treatment has documented effects from routine care (Nordgreen et al., 2019; Nordgreen et al., 2018; Nordgreen et al., 2017). 2) Assistert Selvhjelp is an Internet-delivered treatment system targeting mild or moderate depression and anxiety in primary care where patients are self-referred, today in use by more than 30 Norwegian municipalities. 3) MoodGym is an Internet-delivered treatment system designed for patient to prevent and manage symptoms of depression and anxiety first introduced in Australia in 2001, today with more than 1 million users worldwide. The treatment has shown to have small but significant effect in community settings reported in a randomized controlled trial (Christensen and Griffiths, 2001). 4) Psyktools (renamed to YouWell after the study) is a Norwegian Internet-delivered treatment system designed for therapist and patients. The tool helps the therapist to manage and conduct digital self-management program for patients with mental health problems. The system is still under development and not applied for treatment yet. 5) Internetpsykiatri is a system to create, administer, and evaluate therapist-supported Internet-delivered psychological treatment in the Stockholm area (Sweden). The system has continuously been updated and used in research and regular care since 2003. The effects for the interventions for adults with i.e. social anxiety disorder (Furmark et al., 2009), depression, insomnia (Kaldo et al., 2015) and Irritable Bowel syndrome (Ljótsson et al., 2010) has proven to be large in randomized controlled trials and open effectiveness trials (Hedman et al., 2014).

A detailed summary of the main findings from “MoodGym 2.0” are presented below and in Table 3, the other systems are presented in Table 4, Table 5, Table 6, Table 7, Table 8.

Table 3

Evaluation of usability for MoodGym 2.0.

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image.
1.2.1 Audio-only and video-only (prerecorded)	A	✓
1.2.2. Captions (prerecorded)	A
1.2.3 Audio description or media alternative (prerecorded)	A
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA
1.2.6 Sign language (prerecorded)	AAA
1.2.7 Extended audio description (prerecorded)	AAA
1.2.8 Media alternative (prerecorded)	AAA
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✗	The tables are not coded like they look.
1.3.2 Meaningful sequence	A	✓
1.3.3 Sensory characteristics	A	✗	Using symbols only to remove rows in the tables.
1.4.1 Use of color	A	✗	Using color only to highlight important information in a text.
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 3,1:1.
1.4.4 Resize text	AA	✗	The text is unclear when zooming to 200% in certain text box.
1.4.5 Images of text	AA	✓
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 3,1:1.
1.4.7 Low or no background audio	AAA	✓
1.4.8 Visual presentation	AAA	✗	Cannot choose background color.
1.4.9 Images of text (no exception)	AAA	✓
2.1.1 Keyboard	A	✗	Cannot expand the text field by using keyboard only.
2.1.2 No keyboard trap	A	✓
2.1.3 Keyboard (no exception)	AAA	✗	Cannot expand the text field by using keyboard only.
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A	✗	Cannot pause, stop or hide the animation.
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA	✓
2.4.1 Bypass blocks	A	✓
2.4.2 Page titled	A	✓
2.4.3 Focus order	A	✓
2.4.4 Link purpose (in context)	A	✗	The system has “link” designed like links which it is not.
2.4.5 Multiple ways	AA	✓
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✓
2.4.8 Location	AAA	✓
2.4.9 Link purpose (link only)	AAA	✗	The link does not describe the purpose correctly.
2.4.10 Section headings	AAA	✓
3.1.1 Language of page	A	✓
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words like jargon words.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations, for example i.e.
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	There is no mechanism for supporting this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✓
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✓
3.3.1 Error identification	A	✗	In few web forms there not provided a text description of the error.
3.3.2 Labels or instructions	A
3.3.3 Error suggestion	AA	✗	The web form has not provided a suggestion to fix the error.
3.3.4 Error prevention (legal, financial, data)	AA	✗	The user did not have the chance to check or control the data before sending web form.
3.3.5 Help	AAA	✗	The website had not help documentation and specific instructions for the web forms.
3.3.6 Error prevention (all)	AAA	✗	The user did not have the chance to check or control the data before sending web form.
4.1.1 Parsing	A	✓
4.1.2 Name, role, value	A	✓

Table 4

Evaluation of usability for eMeistring.

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image (empty attribute alt).
1.2.1 Audio-only and video-only (prerecorded)	A
1.2.2. Captions (prerecorded)	A
1.2.3 Audio description or media alternative (prerecorded)	A
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA
1.2.6 Sign language (prerecorded)	AAA
1.2.7 Extended audio description (prerecorded)	AAA
1.2.8 Media alternative (prerecorded)	AAA
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✓
1.3.2 Meaningful sequence	A	✗	The text will not be read in right order by using assertive technology.
1.3.3 Sensory characteristics	A	✗	Using a sensory characteristics instruction: “You will find the tasks in the left menu”.
1.4.1 Use of color	A	✗	Using only green (to add row) and red (to delete row)..
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 2,8:1.
1.4.4 Resize text	AA	✓
1.4.5 Images of text	AA	✗	An image of “Example” instead of text.
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 2,8:1.
1.4.7 Low or no background audio	AAA
1.4.8 Visual presentation	AAA	✗	User can not choose background color.
1.4.9 Images of text (no exception)	AAA	✗	An image of “Example” instead of text.
2.1.1 Keyboard	A	✓
2.1.2 No keyboard trap	A	✓
2.1.3 Keyboard (no exception)	AAA	✓
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A	✓
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA	✓
2.4.1 Bypass blocks	A	✓
2.4.2 Page titled	A	✗	The page title is not relevant to current page
2.4.3 Focus order	A	✓
2.4.4 Link purpose (in context)	A	✗	The link does not describe the purpose - “click here”.
2.4.5 Multiple ways	AA	✗	Only one way to navigate to a specific side.
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✓
2.4.8 Location	AAA	✗	The system do not shows the location of current page.
2.4.9 Link purpose (link only)	AAA	✗	The link does not describe the purpose - “click here”.
2.4.10 Section headings	AAA	✗
3.1.1 Language of page	A	✗	The language of the text is not provided in the source code.
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words like jargon words.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations: e.g.
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	Not supported for this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✗	The data is stored while the user is filling the form.
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✓
3.3.1 Error identification	A	✓
3.3.2 Labels or instructions	A	✗	None instruction is provided.
3.3.3 Error suggestion	AA	✗	None suggestions are provided when the error is automatic found.
3.3.4 Error prevention (legal, financial, data)	AA	✓
3.3.5 Help	AAA	✗	No help documentation is provided.
3.3.6 Error prevention (all)	AAA	✓
4.1.1 Parsing	A	✗	The system has logic error: By using check box you can be men and woman.
4.1.2 Name, role, value	A	✓

Table 5

Evaluation of usability for Assistert Selvhjelp.

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image (empty attribute alt).
1.2.1 Audio-only and video-only (prerecorded)	A
1.2.2. Captions (prerecorded)	A	✗	Missing captions on video lectures.
1.2.3 Audio description or media alternative (prerecorded)	A	✗	The alternatives are not provided for any videos.
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA	✗	Audio descriptions are not provided for any videos.
1.2.6 Sign language (prerecorded)	AAA	✗	Sign language is not supported for any videos.
1.2.7 Extended audio description (prerecorded)	AAA	✗	Audio descriptions are not provided for any videos.
1.2.8 Media alternative (prerecorded)	AAA	✗	A text alternative that serves the equivalent purpose is not provided.
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✗	“Select element is missing an associated label”.
1.3.2 Meaningful sequence	A	✓
1.3.3 Sensory characteristics	A	✗	Instructions: “See her answers on the next page”.
1.4.1 Use of color	A	✓
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 3,0:1.
1.4.4 Resize text: AA 3
1.4.5 Images of text	AA	✗	Using image of text to explain human body reactions for the mental disorder.
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 3,0:1.
1.4.7 Low or no background audio	AAA
1.4.8 Visual presentation	AAA	✗	Impossible to choose background color for the website.
1.4.9 Images of text (no exception)	AAA	✗	Using image of text to explain human body reactions for the mental disorder.
2.1.1 Keyboard	A	✗	Cannot close dialogue box with keyboard.
2.1.2 No keyboard trap	A	✗	Cannot close dialogue box with keyboard which leads to a keyboard trap.
2.1.3 Keyboard (no exception)	AAA	✗	Cannot close dialogue box with keyboard which leads to a keyboard trap.
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA
2.4.1 Bypass blocks	A	✓
2.4.2 Page titled	A	✗	Web pages do not have titles that describe clearly the purpose.
2.4.3 Focus order	A	✓
2.4.4 Link purpose (in context)	A	✗	The link does not describe the purpose and missing underline.
2.4.5 Multiple ways	AA	✓
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✗	Focus indicator is not visible by using the keyboard.
2.4.8 Location	AAA	✗	The user does not know where web page is located in the website.
2.4.9 Link purpose (link only)	AAA	✗	The link does not describe the purpose correctly: “click here to download the form”.
2.4.10 Section headings	AAA	✓
3.1.1 Language of page	A	✓
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words like jargon words.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations e.g. PTSD.
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	There is no mechanism for supporting this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✓
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✗	Using pop-up forms to evaluate treatment at the end of the treatment.
3.3.1 Error identification	A	✓
3.3.2 Labels or instructions	A	✓
3.3.3 Error suggestion	AA	✓
3.3.4 Error prevention (legal, financial, data)	AA	✗	The system does not have a mechanism for reviewing data, confirming or correcting information.
3.3.5 Help	AAA	✗	There system does not provide a detail descriptions and instructions for all the web forms.
3.3.6 Error prevention (all)	AAA	✗	The system does not have a mechanism for reviewing data, confirming or correcting information.
4.1.1 Parsing	A	✓
4.1.2 Name, role, value	A	✓

Table 6

Evaluation of usability for MoodGym.

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image (empty attribute alt).
1.2.1 Audio-only and Video-only (prerecorded)	A
1.2.2. Captions (prerecorded)	A
1.2.3 Audio description or media alternative (prerecorded)	A
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA
1.2.6 Sign language (prerecorded)	AAA
1.2.7 Extended audio description (prerecorded)	AAA
1.2.8 Media alternative (prerecorded)	AAA
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✗	Field “password” missing associated label.
1.3.2 Meaningful sequence	A	✓
1.3.3 Sensory characteristics	A	✗	Instruction: “Please click on the yellow next button to left”.
1.4.1 Use of color	A	✗	Using color only to highlight important information.
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 3,1:1.
1.4.4 Resize text	AA	✗	The text is unclear when zooming to 200%.
1.4.5 Images of text	AA	✓
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 3,1:1.
1.4.7 Low or no background audio	AAA
1.4.8 Visual presentation	AAA	✗	Cannot choose background color.
1.4.9 Images of text (no exception)	AAA	✓
2.1.1 Keyboard	A	✓
2.1.2 No keyboard trap	A	✓
2.1.3 Keyboard (no exception)	AAA	✓
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A	✗	Cannot pause, stop or hide the animation.
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA	✓
2.4.1 Bypass blocks	A	✗	There is not provided a menu or shortcuts to skip to the main content.
2.4.2 Page titled	A	✓
2.4.3 Focus order	A	✗	The website has 319 questions on a single questionnaire, should split it up.
2.4.4 Link purpose (in context)	A	✗	The purpose of the link is not described: “click here”.
2.4.5 Multiple ways	AA	✗	There is no menu at the beginning of the website; buttons of “next” and “previous”.
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✓
2.4.8 Location	AAA	✗	Information about the user's location within a set of web pages is unavailable.
2.4.9 Link purpose (link only)	AAA	✗	The purpose of the link is not described: “click here”.
2.4.10 Section headings	AAA	✗	Sections are not used to organize the content.
3.1.1 Language of page	A	✗	The software code has an invalid language code.
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations, like “e.g.”.
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	There is no mechanism for supporting this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✓
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✓
3.3.1 Error identification	A	✗	The web form did not show where the error was detected.
3.3.2 Labels or instructions	A	✗	The text label was empty.
3.3.3 Error suggestion	AA	✗	The web form has not provided a suggestion for fix the error.
3.3.4 Error prevention (legal, financial, data)	AA	✗	The user did not have the chance to check or control the data before sending web form.
3.3.5 Help	AAA	✗	The website had not help documentation and specific instructions for the web forms.
3.3.6 Error prevention (all)	AAA	✗	The user did not have the chance to check or control the data before sending web form.
4.1.1 Parsing	A	✓
4.1.2 Name, role, value	A	✓

Table 7

Evaluation of usability for Psyktools (Youwell).

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image.
1.2.1 Audio-only and video-only (prerecorded)	A
1.2.2. Captions (prerecorded)	A	✗	Missing captions on video lectures.
1.2.3 Audio description or media alternative (prerecorded)	A	✗	The alternatives are not provided for any videos.
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA	✗	Audio descriptions are not provided for any videos.
1.2.6 Sign language (prerecorded)	AAA	✗	Sign language is not supported for any videos.
1.2.7 Extended audio description (prerecorded)	AAA	✗	Audio descriptions are not provided for any videos.
1.2.8 Media alternative (prerecorded)	AAA	✗	A text alternative that serves the equivalent purpose is not provided.
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✓
1.3.2 Meaningful sequence	A	✓
1.3.3 Sensory characteristics	A	✓
1.4.1 Use of color	A	✓
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 2,1:1.
1.4.4 Resize text	AA	✓
1.4.5 Images of text	AA	✓
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 2,1:1.
1.4.7 Low or no background audio	AAA
1.4.8 Visual presentation	AAA	✗	Impossible to choose foreground and background color.
1.4.9 Images of text (no exception)	AAA	✓
2.1.1 Keyboard	A	✓
2.1.2 No keyboard trap	A	✓
2.1.3 Keyboard (no exception)	AAA	✓
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA
2.4.1 Bypass blocks	A	✓
2.4.2 Page titled	A	✓
2.4.3 Focus order	A	✓
2.4.4 Link purpose (in context)	A	✓
2.4.5 Multiple ways	AA	✗	Only one way to navigate to a specific side.
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✗	Focus indicator is not visible by using the keyboard.
2.4.8 Location	AAA	✗	The user does not know where web page is located in the website.
2.4.9 Link purpose (link only)	AAA	✓
2.4.10 Section headings	AAA	✗	Sections are not always used to organize the content
3.1.1 Language of page	A	✗	The software code does not provide the language of the page.
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words, like “temporallappen”.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations, like “e.g.”
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	There is no mechanism for supporting this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✓
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✓
3.3.1 Error identification	A	✗	There is no mechanism for validating the input from the user.
3.3.2 Labels or instructions	A	✓
3.3.3 Error suggestion	AA
3.3.4 Error prevention (legal, financial, data)	AA	✗	The user did not have the chance to check or control the data before sending web form.
3.3.5 Help	AAA	✗	The website had not help documentation and specific instructions for the web forms.
3.3.6 Error prevention (all)	AAA	✗	The user did not have the chance to check or control the data before sending web form.
4.1.1 Parsing	A	✓
4.1.2 Name, role, value	A	✓

Table 8

Evaluation of usability for Internetpsykiatri.

Success criteria	Level	Result	Evaluation
1.1.1 Non-text content	A	✗	Missing the text for the image.
1.2.1 Audio-only and video-only (prerecorded)	A
1.2.2. Captions (prerecorded)	A
1.2.3 Audio description or media alternative (prerecorded)	A
1.2.4 Captions (live)	AA
1.2.5 Audio description (prerecorded)	AA
1.2.6 Sign language (prerecorded)	AAA
1.2.7 Extended audio description (prerecorded)	AAA
1.2.8 Media alternative (prerecorded)	AAA
1.2.9 Audio-only (live)	AAA
1.3.1 Info and relationships	A	✓
1.3.2 Meaningful sequence	A	✗
1.3.3 Sensory characteristics	A	✗	Instruction: “Print out at the top right of the screen”.
1.4.1 Use of color	A	✓
1.4.2 Audio control	A
1.4.3 Contrast (minimum)	AA	✗	The measured contrast is 2,9:1.
1.4.4 Resize text	AA	✓
1.4.5 Images of text	AA	✗	Tables are pasted as a picture.
1.4.6 Contrast (enhanced)	AAA	✗	The measured contrast is 2,9:1.
1.4.7 Low or no background audio	AAA
1.4.8 Visual presentation	AAA	✗	Cannot choose background color or foreground color.
1.4.9 Images of text (no exception)	AAA	✗	Tables are pasted as a picture.
2.1.1 Keyboard	A	✗	Cannot expand the text field by using keyboard only.
2.1.2 No keyboard trap	A	✓
2.1.3 Keyboard (no exception)	AAA	✗	Cannot expand the text field by using keyboard only.
2.2.1 Timing adjustable	A
2.2.2 Pause, stop, hide	A
2.2.3 No timing	AAA	✓
2.2.4 Interruptions	AAA
2.2.5 Re-authenticating	AAA
2.3.1 Three flashes or below threshold	A	✓
2.3.2 Three flashes	AAA
2.4.1 Bypass blocks	A	✓
2.4.2 Page titled	A	✗	Using the same page title on every single page.
2.4.3 Focus order	A	✓
2.4.4 Link purpose (in context)	A	✗	Purpose of link is not provided.
2.4.5 Multiple ways	AA	✓
2.4.6 Headings and labels	AA	✓
2.4.7 Focus visible	AA	✓
2.4.8 Location	AAA	✓
2.4.9 Link purpose (link only)	AAA	✗	Purpose of link is not provided.
2.4.10 Section headings	AAA	✓
3.1.1 Language of page	A	✗	According to ISO-639-1: Two letter language code is missing.
3.1.2 Language of parts	AA
3.1.3 Unusual words	AAA	✗	There is no mechanism for explaining unusual words, like “Behavioral activation”.
3.1.4 Abbreviations	AAA	✗	The expanded form of abbreviations was not provided for every abbreviations, like “e.g.”
3.1.5 Reading level	AAA
3.1.6 Pronunciation	AAA	✗	There is no mechanism for supporting this purpose.
3.2.1 On focus	A	✓
3.2.2 On input	A	✓
3.2.3 Consistent navigation	AA	✓
3.2.4 Consistent identification	AA	✓
3.2.5 Change on request	AAA	✓
3.3.1 Error identification	A	✗	Several webpages have not a mechanism for validating the input from the user.
3.3.2 Labels or instructions	A	✗	The activity form does not explain how to calculate the sleep effect.
3.3.3 Error suggestion	AA	✗	The forms do not indicate where the error is.
3.3.4 Error prevention (legal, financial, data)	AA	✗	The user did not have the chance to check or control the data before sending web form.
3.3.5 Help	AAA	✗	The website had not help documentation and specific instructions for the web forms.
3.3.6 Error prevention (all)	AAA	✗	The user did not have the chance to check or control the data before sending web form.
4.1.1 Parsing	A	✓
4.1.2 Name, role, value	A	✓

MoodGym (2.0)

MoodGym is an Internet-delivered treatment system designed for patient to prevent and manage symptoms of depression and anxiety. Moodgym 2.0 was released in 2017 and is a new version of MoodGym, using more modern technologies and is more user-friendly than the previous one (Table 3). The system offers a digital self-help interactive program first introduced in Australia in 2001, today it has more than 1 million users worldwide.

MoodGym is a treatment system targeting adolescents and adults with symptoms of depression. The treatment is based on Internet-delivered cognitive behavioral therapy and is implemented as a stand-alone intervention. There are no opportunity for communication between the patient and the therapist through the system, but therapist guidance may be delivered via telephone or face-to-face consultations. The patients are self-referred. The treatment has shown to have small but significant effect in community settings reported in a randomized controlled trial (Christensen and Griffiths, 2001). However, it has also been reported that MoodGym have no additional effect to a “treatment as usual” follow up by a General Practitioner.

The system partially satisfies the usability criteria. There are several improvements related to the usability of the system, especially considering the criteria about presentation of the content. In practice, this means that the system does not strictly follow the principles of WCAG 2.0 (Table 3). The system has only fulfilled few of the criteria that belongs to the first (perceivable) principle of WCAG 2.0. The system was using sensor characteristics to provide the information about how to remove rows in the table, by using only symbols. The use of symbols only makes it difficult for people with visual disability to locate and read the symbols. The system does not make the content accessible for people with reduced vision by using two different colors to highlight important content and using white text on grey background. The second principle of WCAG 2.0 was partially fulfilled by the system. It is visible where you are navigating on the website and you may reach all the functionalities by using keyboard only. The user must have the possibilities to pause, stop or hide animations that start automatically. By not having this opportunity the user may be distracted, which leads to poor user experience. In the system you may find blue underlined text that looks like links but does not behave like links. This is called false affordance in interaction design (Gaver, 1991). The system shows the user's location within a set of web pages for example: Home – Module 1 – Introduction. This will help users to keep track of their treatment progression.

The third principle of WCAG 2.0 has not been followed by the system with few exceptions. Every single form provides instructions about how to fill out the form and is providing error identification when an error is occurring by text description. However, the forms do not provide any error recovery suggestions while filling out the forms. A process for confirming the answers from questionnaires is required by the criteria, but is not supported by the system. The system has fulfilled every criterion that belongs to the fourth and last principle, robustness of WCAG 2.0.

Responsive web design

The system has a good solution for responsive web design, but still need some improvement to meet the criteria.

The system is not browser and device independent, hence it do not meet one of the seven criteria for responsive web design. MoodGym 2.0 is using a viewport that make sure that the content is not wider than the screen size of the device. This means that the images are scaled up or down relative to the screen size. The newest HTML5 technology is used to play video and animations, which support browser independence. Furthermore, the majority of the content is readable on small devices, but one may find it difficult to read the text on clickable buttons in the menu.

We also evaluated the systems for how they provide user interaction and their security and privacy mechanisms. Importantly, there was found only a few issues related to the security criteria that may concern the systems. Details about these evaluations are left out from the paper due to space limitations.

Discussion

The main result from the study shows that the systems mostly satisfy the evaluation criteria for interaction and security, but less for usability and responsive design. The systems only partially fulfill the universal design requirement from WCAG 2.0 (Table 9).

Table 9

Evaluation of Internet-delivered treatment systems according to WCAG 2.0)

Systems	Levels
	A		AA		AAA
	Number	Percentage	Number	Percentage	Number	Percentage
eMeistring	10/20	50.0%	6/10	60.0%	5/15	33.3%
Assistert Selvhjelp	12/21	57.1%	6/11	54.5%	2/17	11.8%
MoodGYM	9/20	45.0%	5/10	50.0%	5/15	33.3%
MoodGYM 2.0	13/21	61.9%	6/10	60.0%	6/16	37.5%
Psyktools	16/21	76.2%	5/10	50.0%	5/17	29.4%
Internetpsykiatri	11/19	57.9%	6/10	60.0%	4/14	28.6%

Despite the fact that Internet-delivered treatments are nowadays disseminated into regular mental health care, few of these systems have been released with consideration of their quality with regard to usability. It is well known that Internet and WEB technologies have been rapidly evolving during the last decade, but most Internet-delivered treatment systems in use today were developed with older technologies and have not been updated in accordance with the evolution of the technology. This was especially affecting the criteria on responsive web design. We observed several errors concerning browser and device independence, e.g. images and buttons were not adapted when the size of the screen was changed. Our findings indicate that the latest developed systems (MoodGym 2.0 and Psyktools) are using technologies that follow today's standard, however Psyktools was the only system that fully satisfies the responsive web design criteria.

On the positive side, our study indicates that the evaluated Internet-delivered mental health treatment systems have adequate solutions concerning security and privacy of patient data, which is of great importance to establish trust between the health care providers and the patients. All of the systems use the encrypted HTTPS communication protocol, hence they ensure secure data transfer between the patients and the systems. The authentication mechanism could be improved for some of the systems, but two of the systems (Internetpsykiatri and eMeistring) uses a robust two factor authentication mechanism making it difficult for intruders to login to the system. The scoring on the usability criteria was not satisfactory for any of the systems in the evaluation. All of the systems have issues regarding the universal design guideline WCAG 2.0. The following errors were observed: the navigation structure of the system is not adapted to the patients’ needs, use of color and contrast is making it difficult for people with visual disability such as color blindness, images and text was not scaling well when the resolution and size was changed. However, we see a tendency that the newer treatment systems score better on usability, but they still need to improve to meet the criteria. Especially were the results on the WCAG 2.0 evaluation below expectation, with an average score on 58% on level A, 56% on level AA and only 29% on level AAA.

A limitation of the present study was that we were not able to assess the architecture of the present systems. In an early stage of the study we tried to evaluate the architecture of the systems, but unfortunately, we did not have access to this part of the systems. Therefore, it could be interesting for further research to investigate if the systems are following the standards for architecture and interoperability like HL7-FHIR (HL7, 2019), LOINC (Regenstrief Institute, 2018), openEHR (openEHR Foundation, 2018) and ICD-11 (World Health Organization, 2018). One other interesting direction of further research will be to systematically study the possibilities Internet-delivered treatment systems has for integration with other health care system like Electronic health records and medical charts. One other limitation of the study is that we only reviewed the patient interface of the treatment systems. One can argue that if the therapist interface to the program is not good enough, the therapist might not be motivated to use the system, could make mistakes or could miss important information about the patient. We would recommend that a similar study for the usability of the therapist interfaces should be performed.

Future research

As described in the previous section are today's Internet-delivered treatment systems not following the state-of-the-art knowledge of usability and Internet technology. We propose that it should be conducted more research on the relationship between usability, technology and outcome of Internet-delivered treatment. We also advice to involve end-users in the design and continuous evaluation of treatment systems. Usability should preferably be considered from the start of system development, however few Internet-delivered treatment systems have undergone usability evaluation prior to dissemination. We propose that it should be developed benchmark tests for systematic evaluation of Internet-delivered treatment systems based on usability heuristics, universal design principles (WCAG 2.0) and security and privacy principle, to enhance technology acceptance by end-users and to guide system modification.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.