Evaluation of a telerehabilitation system for community-based rehabilitation.

The use of web-based portals, while increasing in popularity in the fields of medicine and research, are rarely reported on in community-based rehabilitation programs. A program within the Pennsylvania Office of Vocational Rehabilitation's Hiram G. Andrews Center, the Cognitive Skills Enhancement Program (CSEP), sought to enhance organization of program and participant information and communication between part- and full-time employees, supervisors and consultants. A telerehabilitation system was developed consisting of (1) a web-based portal to support a variety of clinical activities, and (2) the Versatile Integrated System for Telerehabilitation (VISYTER) video-conferencing system to support the collaboration and delivery of rehabilitation services remotely. This descriptive evaluation examines the usability of the telerehabilitation system incorporating both the portal and VISYTER. Telerehabilitation system users include CSEP staff members from three geographical locations and employed by two institutions. The IBM After-Scenario Questionnaire (ASQ) and Post-Study System Usability Questionnaire (PSSUQ), the Telehealth Usability Questionnaire (TUQ), and two demographic surveys were administered to gather both objective and subjective information. Results showed generally high levels of usability. Users commented that the telerehabilitation system improved communication, increased access to information, improved speed of completing tasks, and had an appealing interface. Areas where users would like to see improvements, including ease of accessing/editing documents and searching for information, are discussed.

Background

Telerehabilitation (TR) has been defined as the delivery of rehabilitation services via information and telecommunication technologies (i.e., assessment, monitoring, intervention, supervision, education, consultation, and counseling) (Brennan et al., 2010). TR creates opportunities for underserved regions (e.g., rural communities) to access advanced rehabilitation expertise and services that would be otherwise inaccessible. The advances of Internet technologies and the availability of broadband connections have expanded TR applications that were previously too complicated or expensive to deliver (Parmanto & Saptono, 2008).

The nature of the rehabilitation services determines the necessary telecommunication technology and informatics infrastructures. Community-based rehabilitation typically needs continuous monitoring of clients’ goals and progress, as well as effective and confidential ways of communicating this information to a team of professionals (Parmanto & Saptono, 2008). Clients with disabilities generally prefer this type of rehabilitation, and research has identified valuable evidence for the unique benefits for service providers to use TR in a client’s natural environment in order to maximize functioning (McCue, Fairman, & Pramuka, 2010; McCue, Schutte, & Sporner, 2011). Previous research has indicated that TR can play a vital role in the implementation and accessibility of naturalistic treatments such as in vivo exposure (McCue, Fairman, & Pramuka, 2010). In addition, many community-based programs would benefit from accessing the clinical knowledge of experts who are not full-time staff members due to distance or budgetary concerns.

Web-Based Portals

Web-based portals have primarily been used in the fields of medicine (i.e., e-health or telemedicine) (Brennan et al., 2008) and research. In health fields, on-line portals have been used to help patients manage chronic conditions, such as weight control (Peterson, 2008) and diabetes (Bryce et al., 2008), and as a method for patients to conduct essential medical tasks such as refilling prescriptions and scheduling appointments (Abbott, 2003). In clinical research, on-line portals have been developed to fulfill the need of researchers to access critical information at a moment’s notice and communicate it effectively with a variety of professionals and collaborators (Hume, 2001). The University of Pittsburgh, through a NIDRR funded Rehabilitation Engineering Research Center on Telerehabilitation (grant #H133E090002) has developed a portal to support multiple clinical activities (PITT Model) that will be more efficient and improve clinical practice (Schutte, Pramuka, McCue, Fairman, Saptono, & Parmanto, 2009).

Cognitive Skills Enhancement Program

The University of Pittsburgh’s Department of Rehabilitation Science and Technology (RST) conducts clinical, academic, and research activities in its Johnstown, Pennsylvania location at the Hiram G. Andrews Center (HGAC). HGAC is a state-operated vocational rehabilitation facility, primarily serving consumers of Pennsylvania’s Office of Vocational Rehabilitation (OVR). The Cognitive Skills Enhancement Program (CSEP) is operated by RST but is integrated into the larger service structure of HGAC. It is a stand-alone, 15-week, Monday through Friday, pre-vocational training program specifically designed for individuals with cognitive disabilities. Each term, CSEP accepts approximately 15 participants. The Center for Assistive Rehabilitative Technologies (CART), which provides evaluations for and training on assistive technology, including learning technologies and organization/memory aids, is another collaborative program between RST and HGAC.

In order to provide quality services to CSEP participants, CSEP staff members must have access to up-to-date information, including individual participants’ histories, goals, strategies, accommodations, assistive technology currently being used or evaluated, and current issues or difficulties. In addition, staff members need information on the structure and progress of the group as they move through various cognitive, social, and vocational sessions and activities. Regular staff meetings, trainings, and supervisions are also important to ensure the quality of services being provided. Coordinating and organizing services across two institutions (University of Pittsburgh and HGAC) and three geographical locations (Pittsburgh, PA; Johnstown, PA; and Virginia Beach, VA) makes communication a daunting task. One way of effectively managing information is through telerehabilitation (TR) technologies.

Challenges to effective and efficient clinical service at CSEP involve cumbersome paper documentation, (including client files that cannot be transported for review by specialists/consultants) and an unwieldy number of session and activity notes, directions, and handouts that are difficult to locate or update. There was also often a need for clinicians/consultants to contact other staff members or clients when separated by distance. A telerehabilitation system including a web-based portal and a videoconferencing system was developed in order to support a variety of clinical activities, to increase efficiency and to improve clinical practices.

CSEP Portal Features and Functions

The CSEP Portal was developed using the PITT model of Information Management Infrastructure, which was developed at the University of Pittsburgh’s Rehabilitation Engineering and Research Center on Telerehabilitation (RERC-TR). The infrastructure was developed through a design process that focused on accessibility, openness, extensibility, cost-effectiveness, and security (Parmanto, Saptono, Sugiantara, Brienza, & Nnaji, 2006). The PITT model was designed to support TR applications. Based on CSEP staff input, the PITT model was customized to address various TR needs specific to CSEP.

The portal includes methods of gathering and updating individual participants’ information and progress, planning and updating the group schedule of sessions and activities, documenting intake/referral screening information, storing program evaluation/research data, and improving methods of staff communication during meetings, trainings, and supervisions. It is accessible from any computer with an Internet connection. Following is a brief description of the most important portal features and functions.

Security. Critical to the development and implementation of the PITT model is the security of the infrastructure. The portal has been built to comply with standards for privacy and security of personal information (i.e., HIPAA) and employs common security policies, such as encryption of data, requiring access information, and limiting access to rightful persons. Staff members log in with a username and password (Parmanto, Saptono, Sugiantara, Brienza, & Nnaji, 2006).

Homepage. The Homepage is the first site viewed after logging in to the portal. It can be customized to display any combination of portal features. On the CSEP Portal, there is a left-hand quick launch bar that displays links to all site contents. Announcements, action items, shared documents, links, staff contact information, and a Really Simple Syndication (RSS) viewer displaying a web feed of Rehabilitation Counseling Bulletin current article titles are displayed.

Calendars. Calendars have monthly, weekly, and daily views. When staff clicks on an event in the calendar, a more detailed description is displayed, including location, start and end time, description, and attachments. The CSEP calendar has daily sessions and activities, which can be scheduled at any time, by any staff member. Part-time staff can view what was covered during days and sessions when they were not present; likewise, staff physically located at the University of Pittsburgh can monitor clinical activity. CART has a similar calendar for documenting staff availability and scheduling clients. Two additional calendars are used to schedule the two telerehabilitation offices that are equipped with VISYTER and can be used for videoconferencing.

Sessions and activities. The Sessions and Activities folder houses information for group sessions and activities organized by topic (e.g., memory, attention, problem solving). CSEP staff can access and edit descriptions, directions, and documents necessary to run sessions.

Participant information. Participant information is organized in the Tier I, Tier II, and Tier III folders. Documentation includes participant narratives, including demographic information, OVR counselor information, participant goals, diagnoses, medications, educational and work history, and intellectual assessment results; and evaluations, including weekly progress checklists, weekly reports to OVR counselors, and mentorship evaluations.

Referral screening. The Referral Screening folder includes information on the CSEP program for potential participants, protocol for interviewing a potential participant and reviewing his or her file, and a referral screening database where information regarding new screens is constantly updated.

Resources. The Program and Participant Resources folders house documents and information for CSEP staff, including the CSEP Program Manual and Rehabilitation Specialist Procedural Manual.

Discussions. Whereas the calendar documents the actions and plans of CSEP staff, the discussion board is where notes are kept for individual and group clinical interactions. Each participant has his or her own thread under which staff can note objective (e.g., performance on test or time needed to complete activity) and subjective (e.g., attention or cooperation level) performance. Primary clinicians can check the client’s thread to review progress and performance on days when the clinician was not present.

VISYTER Features and Functions

The Rehabilitation Engineering Research Center on Telerehabilitation (RERC-TR) developed the Versatile and Integrated System for Telerehabilitation (VISYTER). VISYTER is an integrated system that provides both real-time and asynchronous communication channels to support the collaboration and delivery of rehabilitation service remotely. VISYTER components include the videoconferencing system (which is primarily what is used in CSEP), integrated web portal, electronic health records, and a database for archiving the sessions.

In addition to the synchronized sound/picture communications available in popular videoconferencing systems, this multimodal system allows the user to present visual stimuli. Visual stimuli are presented in a window on the monitor. Responses are in the form of verbal communication and video or through the manipulation of visual stimuli. All real-time communication streams are encrypted to protect the security and confidentiality of on-line interactions.

VISYTER consists of computer stations at each site. At HGAC, four telerehabilitation offices are set up for VISYTER use – two conference rooms, and two private offices. At the University of Pittsburgh, several faculty associated with the RERC have VISYTER installed on their office computers. Graduate and doctoral students working at HGAC have access to a shared TR office as well. All VISYTER computer stations are connected by a broadband Internet connection, with a minimum connection of 3 Mbps/768 kbps (downstream/upstream). This connection is expected to provide optimal performance at an affordable cost.

Telerehabilitation System Implementation

The portal was implemented in February 2009. VISYTER was implemented later, in June 2010. It was anticipated that the telerehabilitation system would provide functional benefits, including improved communication and increased efficiency, compared to the old system of paper records and a written curriculum. For the purpose of this paper, the TR system is defined as an integrated web-based portal and a flexible integrated videoconferencing system (VISYTER). This paper evaluates the usability of the system, identifies strengths and weaknesses, and determines areas for improvement.

Method

All users of either the portal or VISYTER were invited to participate in evaluating the TR system. Nine staff members participated in phase 1 of the study (fall, 2009) and nineteen TR system users participated in the second phase of usability questionnaires (October, 2011). Because CSEP is a training opportunity for Master’s level students, many staff members are transitory. Of the initial 9 participants, 5 were also participants in phase 2. One system user accessed computer applications using assistive technology, specifically Dragon NaturallySpeaking 7 and sticky keys.

In phase 1, participants each completed two usability questionnaires, the IBM After-Scenario Questionnaire (ASQ) and the Post-Study System Usability Questionnaire (PSSUQ) (see below – Usability surveys). Questions were answered based on use of the portal only, since VISYTER was not yet implemented.

In phase 2, in addition to the ASQ and PSSUQ, participants were asked to complete demographic questionnaires as well as a telehealth usability questionnaire. Participants that used both the portal and VISYTER (n=14) completed a separate demographic questionnaire for each component of the TR system. Participants who only used VISYTER (n=5) completed only a demographic questionnaire about VISYTER. IRB approval was granted from the University of Pittsburgh (IRB PRO11010460). Table 1 represents the questionnaires completed in each phase.

Table 1

Phase 1 and 2 Survey Completion across TR System User Groups

	Phase 1	Phase 2
Survey	Portal Only	VISYTER only	Portal + VISYTER
ASQ	x		x
PSSUQ	x		x
TUQ		x	x
VISYTER Demographic		x	x
Portal Demographic			x

Usability Surveys

Two IBM computer usability satisfaction questionnaires were used to evaluate the CSEP Portal: The After-Scenario Questionnaire (ASQ) and The Post-Study System Usability Questionnaire (PSSUQ). These measures gather both subjective and objective information directly related to the users’ experience interacting with the system or program (Lewis, 1993).

The after- scenario questionnaire (ASQ). The ASQ is a scenario-based usability questionnaire. For evaluation of the CSEP Portal, four scenarios were created. The scenarios were devised to highlight some of the most often used features of the portal (i.e., homepage, calendars, resources, and discussions). See Table 2 for a description of the scenarios and the competencies required to complete them.

Table 2

Description of Four after Scenario Questionnaire (ASQ) Scenarios and their Rationale

	Scenario	Competencies
Calendar	On the CSEP Calendar, create a 2-hour session on Sunday, October 11, 2009. Attach any document.	Access CSEP (Tier I) CalendarAccess daily or weekly viewCreate an item on a specific dayEdit item lengthAttach a document
Action Items	Create a high priority action item to remind you to complete these questionnaires by Monday, Oct. 26 2009.	Access HomepageAccess Action ItemsCreate new action itemChange priority levelAssign action itemAdd a due date to an action item
Add Name	Add your name, the date, and time to the bottom of the HGAC program list.	Use Quick Launch to navigate to a documentAccess Resources, ProgramOpen a documentEdit a documentUpload an edited document
Discussion	Access the portal Usability discussion board and follow posted directions.	Access DiscussionsRead a postReply to a thread

After portal users completed the ASQ, the authors were able to go on the portal to verify that the task and all of its components were completed fully and correctly.

After completion of a scenario, the portal user responded to three statements: (1) Overall, I am satisfied with how easy it is to use this system; (2) It was simple to use this system; (3) I could effectively complete the tasks and scenarios using this system. Portal users responded to the questions in two ways, (1) on a Likert scale ranging from 1 (strongly agree) to 7 (strongly disagree), and (2) in written comments. Psychometric evaluation found the scale to be highly reliable, have high concurrent validity, and reasonably sensitive (Lewis, 1993).

The post- study system usability questionnaire (PSSUQ). The PSSUQ is a 19-item questionnaire designed to assess overall user satisfaction with system usability, (as opposed to the ASQ which is designed to assess satisfaction with usability of specific tasks). It provides an overall evaluation of the usability of a system. Like the ASQ, portal users responded to the questions both (1) on a Likert scale ranging from 1 (strongly agree) to 7 (strongly disagree), and (2) in written comments. The items on the scale were selected to evaluate system characteristics such as ease of use, ease of learning, simplicity, effectiveness, information, and the user interface (Lewis, 1993).

Scoring resulted in an overall score and three sub-scale scores. The subscales were System Usefulness (SYSUSE) (example item: “Overall, I am satisfied with how easy it is to use this system,”); Information Quality (INFOQUAL), (example item: “The system gave error messages that clearly told me how to fix problems,”); and Interface Quality (INTERQUAL), (example item: “The interface of this system was pleasant.”) Psychometric evaluation found the scale to be highly reliable, valid, and sensitive (Lewis, 1993).

Telehealth usability questionnaire (TUQ). Additionally, researchers at the RERC-TR have developed a satisfaction and usability tool based upon the PSSUQ but specifically designed for telerehabilitation systems (B. Parmanto, personal communication, November 23, 2011). Here usability is defined as how easy or convenient it is to use a telehealth system. For the purpose of this study telehealth system refers to VISYTER and the CSEP Portal combined, including audio and visual quality and how similar or different this was from receiving an in-person assessment. It is a 21-question survey that covers all factors of usability, usefulness, ease of use effectiveness, reliability, and satisfaction (Parmanto, 2011). Two questions were removed due to incompatibility with participants’ use of the TR system.

System users responded to questions on a Likert scale, ranging from 1 (complete disagreement) to 7 (complete agreement) or not applicable (N/A). After questions were completed participants were given the opportunity to provide comments about the telehealth system.

Demographic questionnaires. Two questionnaires were developed by the University of Pittsburgh telerehabilitation team to obtain additional demographic data on CSEP Portal and VISYTER users that was not collected during phase 1. VISYTER and portal questions were represented on two separate questionnaires. Both surveys asked questions related to system use, including:

“How long have you been using the system?”

“Approximately how often do you use the system?”

“How comfortable are you with the system?”

“How comfortable are you using technology in general?”

“Do you feel the usability of the system has increased since you began using it?”

Participants responded to questions in 1 of 3 ways, Likert scale, ordinal and nominal measurement, and written comments. Descriptive statistics were calculated using Statistical Package for the Social Sciences (SPSS).

Findings

Demographic Questionnaires

During phase 1, demographics questionnaires were not administered. Of the nineteen participants in phase 2, all participants completed the VISYTER demographics questionnaire (all TR system members use VISYTER to some extent). However, not all participants use the portal; therefore only fourteen completed the CSEP Portal questionnaire. See Table 3 for averages across questions of interest for both CSEP Portal and VISYTER users.

Table 3

Mean Demographic Scores for Participants Who Use the CSEP Portal and/or VISYTER

	# of users	How long (years)	How often (monthly)	Comfortable	ComfortTech	Usability
CSEP Portal	14	1.79	3.64	2.64	2.64	3.14
VISYTER	19	1.32	1.47	2.53	2.58	2.68

Note. Five participants used VISYTER only and not the portal. No one used only the portal.

Comfortable, ComfortTech, and Usability were rated on a scale of 1 to 7, with lower scores indicating increased comfort/usability.

Additionally, most VISYTER survey participants (16 of 19) reported using the teleconferencing system for team meetings and staffings. The second most common use of VISYTER was for supervision purposes; about 47 percent (nine participants) reported utilizing this function. CSEP Portal users expressed the desire for added features and capabilities to the portal, including cross- platform compatibility (i.e., Mac), a usable search function, and the ability to “drag and drop” reminders and appointments.

Five CSEP staff members completed both phase 1 and phase 2 of the TR system usability study. All felt the usability of the CSEP portal has increased since they began using the system and since they filled usability surveys out during phase 1. Specifically, participants noted that several “bugs” in the system have now been worked out and file sharing has become more accessible, but several also noted that the lack of a search feature continues to make finding things difficult in such a large system.

ASQ

According to the psychometric evaluation of the ASQ, it is reasonable to condense the three items into a single scale through averaging, allowing for easier interpretation of results (Lewis, 1993). Therefore, all telerehabilitation system user responses pertaining to each scenario were averaged to achieve an overall usability score per scenario. On the Likert scale, 1 is anchored by “Strongly Agree” and 7 by “Strongly Disagree,” so lower scores indicate higher usability. The ASQ was administered to nine participants in phase 1 and the nineteen participants in phase 2.

Phase 1. For the Calendar scenario, the average usability score was 1.67 (SD=1.01, Range=1–4). For the Action Item scenario, the average usability score was 2.22 (SD=1.04, Range=1–4). For the Add Name scenario, the average usability score was 2.74 (SD=1.38, Range=1–5). For the Discussion scenario, the average usability score was 1.33 (SD=0.44, Range=1–2). Results indicate the portal users found the portal highly usable regarding completion of the scenarios. Portal users found the discussion board to be the most usable feature, and editing and uploading a document to be the least usable. See Figure 1 for a graphical depiction of usability scores for both phases 1 and 2.

Figure 1.

Comparison of mean ASQ scores across scenarios for phase 1 and phase 2 data.

Note: Lower scores indicate higher usability.

The results of scenario completion accuracy check corresponded with the usability ratings. The Calendar and Discussion tasks were rated most usable, and 89% of users completed these tasks without error. For the least usable task, Add Name, only 67% of users completed the task successfully. While all users were able to create an action item, only 22% were able to complete all the secondary requirements of the task, such as assigning the task to themselves, making the task high priority, and setting a due date. The tasks that were completed successfully were generally rated as more user-friendly.

Phase 2. For the Calendar scenario, the average usability score was 1.95 (SD=1.37, Range=1–6). For the Action Item scenario, the average usability score was 2.52, (SD=1.99, Range=1–7). For the Add Name scenario, the average usability score was 2.57, (SD=1.90, Range=1–7). For the Discussion scenario, the average usability score was 1.55, (SD=1.10, Range=1–5). Similar to results from phase 1, overall CSEP staff found the portal highly usable across all scenarios, with the discussion task reported as the most usable and the add name task as the least usable. Figure 2 illustrates the averaged results for all four scenarios in phase 2.

Figure 2.

Comparison of mean PSSUQ scores across categories for phase 1 and phase 2 data.

Note: Lower scores indicate higher levels of usability and satisfaction.

Completion of all tasks was verified. Participant comments showed portal users found the Discussion task the most usable because the discussion board works for a variety of purposes. However, one participant did not find the Discussion task easy to complete using the portal. It is very likely this dissatisfaction is a result of the short amount of time the participant had been a user of the system (less than 6 months). Parallel to results from phase 1 ASQ data, the Add Name task proved to be the least usable task and produced the highest number of negative comments and need for troubleshooting. Specifically, approximately 29% of portal users rated the task a 4 or above on overall usability, indicating they disagreed that the system was easy to use, simple, and allowed them to effectively complete the task. The Action Item task was the second least usable task reported by participants. Of the 11 participants (79%) that rated the task high on usability (below 4 on the scale), only 6 (55%) actually completed the task fully. Common mistakes made included failing to set a due date for the action item, not setting the action item priority to high, or not completing the task at all. Many of the users who were unable to complete the task reported that (1) the process of creating a high priority action item was not compatible with a Mac computer (cross platform incapability), or (2) that finding the correct place on the portal to do the task was difficult.

PSSUQ

The PSSUQ is an evaluation of general user satisfaction with system usability. It is a 19-item questionnaire comprised of (1) a Likert scale ranging from 1 (strongly agree) to 7 (strongly disagree), and (2) written comments.

Phase 1. The overall average score for portal usability was 2.25 (SD=0.52, Range=1.47–3.16). The average score for System Usefulness was 1.95 (SD=0.68, Range=1.29–3.29). The average score for Information Quality was 3.15 (SD=1.13, Range=2.00–5.00). The average score for Interface Quality was 1.67 (SD=0.71, Range=1.00–3.00). See Figure 2 for a graphical depiction of usability scores across phase 1 and 2. Again, lower scores indicate higher usability, and again, results were positive overall. Of the subscales, portal users found the interface (defined on the measure as “those items you use to interact with the system”) to be the most usable, and the quality of information to be least usable.

Phase 2. The overall average score for portal usability for phase 2 data was 2.21 (SD=1.17, Range=0.79– 4.95). The average score for System Usefulness was 2.15 (SD=1.29, Range=1.00– 5.50). The average score for Information Quality was 2.25 (SD=1.70, Range=0.00–6.29). The average score for Interface Quality was 2.38 (SD=0.90, Range=1.00– 4.67). Figure 2 represents the averages for all four measures of usability and system satisfaction.

Unlike the variations in response for data obtained during phase 1, the averages across all four measures in phase 2, including overall usability were very stable with low variability. The subscale of system usefulness was rated by participants to have the highest usability. Additionally, although only by a small margin, interface quality was rated the lowest usability. This result contrasts with the high usability rating users gave this subset in phase 1.

Telehealth Usability Questionnaire

The Telehealth Usability Questionnaire was not developed during the initial data collection during phase 1; therefore this questionnaire was only administered to eighteen participants in phase 2 of the study (one user did not complete the questionnaire).

For the purpose of ensuring all TUQ questions were relevant to portal and VISYTER users, two questions were removed, leaving nineteen questions. Questions on the TUQ that were the main focal point were: (1) VISYTER saves me time traveling to a hospital or specialist clinic; (2) It was easy to learn to use VISYTER; (3) I can easily talk to the other site (clinician, patient); (4) I can hear the other side clearly; (5) I feel comfortable communicating with my VISYTER colleagues; and (6) Overall, I am satisfied with VISYTER. Averages across these questions of interest are represented in Figure 3.

Figure 3.

TUQ means for selected questions of interest in phase 2.

Note: Higher scores indicate higher levels of usability and satisfaction.

Overall, users rated VISYTER high on usability across all measures of interest, including overall satisfaction with the system and its capabilities. TUQ uses a Likert scale, ranging from 1 (complete disagreement) to 7 (complete agreement) or not applicable (N/A). The average scores were as follows: for VISYTER’s ability to save the user time completing tasks (6.18); can learn the system easily (5.82); can talk easily with the person on the other side (5.24), can hear the other side clearly (4.89); feel comfortable using VISYTER (6.06); and overall satisfaction with the teleconferencing system (5.50).

Written Comments

Qualitative information was collected in six areas: (1) Scenario 4, (Discussion) in which portal users were asked to post a response to the question, “Do you like the portal? Why or why not?”; (2) ASQ Comments (optional after every item); (3) PSSUQ comments (optional after every item); (4) TUQ general comments about the telerehabilitation system; and (5) Optional comments after each item in both demographic questionnaires for CSEP Portal and VISYTER users.

This section provides a summary of common positive and negative responses regarding portal and VISYTER features and usage.

TR system strengths. Comments regarding the positive impact of the TR system revolved around four main areas: (1) improved communication; (2) easier access to documents and information (i.e., all information in a central place that can be accessed from various locations); (3) faster access to documents due to remote access capabilities; (4) increased videoconferencing usability; and (5) an appealing interface.

A primary theme was that the system improved communication. One portal user noted, “I think it’s a good communication tool and offers flexibility to do documentation from home, especially since I’m only here once per week.” The portal provides a way to share both formal (e.g., progress reports) and informal information. Another portal user commented on the benefit of being able to share notes and thoughts about portal user progress with staff: “Overall, I like the discussion feature as a way to document observations.”

In addition to being able to send and receive information with others, portal users also commented on the benefits of increased access to important documents and information easily and quickly. Comments included: “[The portal] allows us to have all of the information we need in once central place that we can access from anywhere. This is especially helpful with all of the program activities,” and, “It seems to have decreased the ‘last minute’ scrambles in terms of locating information mostly because everything we use on a day-to-day basis can be accessed on the portal.” Regarding time saving benefits, portal users noted: “It saves time and allows access remotely,” and “We are getting tasks done much more quickly, such as scheduling, agendas, etc.”

Finally, portal users responded positively to the portal interface. One user stated, “The design of the site is very pleasant including the graphics, etc.” Regarding the Add Name task, another user commented, “The task was easy and intuitive. I have never gotten into the HGAC program list. It took me two clicks to find it.”

In terms of VISYTER, many users felt the usability of the videoconferencing system has increased since they first began using it. Specifically, one user reported improvements to VISYTER have been made in terms of “better audio integration and video quality, and the ability to capture screen shots.” Another user commented that “VISYTER has become a more reliable method for connecting with other staff members as bugs have been worked out.”

TR system weaknesses. Portal users also noted weaknesses of the TR system and offered suggestions for its improvement. These comments fell into seven main themes: (1) difficulty opening/uploading/editing documents; (2) lack of a search function; (3) difficulty with action items; (4) unhelpful error messages; (5) lack of alerts; (6) repeated requests for username and password; and (7) audio and video quality inconsistencies.

The difficulty with opening, uploading, and editing documents became apparent in the Add Name scenario. One portal user noted that the task was “fairly easy, but [I was] unable to save.” Another user noted, “I couldn’t complete all the tasks because I couldn’t save the work. I became somewhat frustrated.” The problem was not the same for every portal user. For one user, “The only problem that I have is opening documents. For some reason, I have to click ‘view properties’ on the drop down arrow, and then click the document.” For another user, “I cannot open documents directly within the portal, but have to download the document, edit it locally on my computer, and upload again, overwriting the existing document. I don’t think I can ‘check out’ documents in the intended way either.” The document files were also noted to be rather inflexible. Other user comments included: “[I] cannot move documents already on portal to new folder/location on portal without downloading and re-uploading to second location,” and “It would be nice if it was a little easier to rearrange documents and folders.”

The second issue that was almost unanimously identified was that the search function did not work. Comments included: “I initially did not know where the document was located and the search function did not return any results when I searched for various parts of the document title,” and “Organization is key to easily find things; search/find would be helpful.”

As previously mentioned, most portal users were not able to successfully complete the Action Item task. Several portal users noted difficulty assigning action items to themselves. In addition, when these problems were encountered, displayed error messages were either unavailable or unhelpful. “I’m not sure if there are error messages or ways to help us fix problems,” wrote one portal user. Another commented, “In the past when I have received error messages, I have been unable to fix the problem on my own.” Another portal user offered related input, stating “The error messages that I can recall getting have not been particularly informative.”

A common suggestion made by portal users was that “…when a new announcement/document/etc. is posted there is an option to email specific users to make them aware of this.” Users noted that an automatically generated email would help users to know where to look to find new or updated information.

Finally, users expressed frustration over the portal’s frequent requests for their username and password. One portal user wrote, “I had to enter my username and password three times to open [a] document to edit, and then another two times in order to save the document.” Reentering username and password repeatedly was time-consuming and frustrating for some users, but was not an issue for everyone.

Regarding VISYTER, several commented that the quality of the audio is heavily dependent on the type of microphone used, which is often a problem encountered when laptop computers are used to videoconference. Also, many users felt the lack of a troubleshooting menu decreased the ability to feel fully comfortable using the system without assistance, especially when issues with audio and video quality can be time consuming to address. However, many users indicated that when audio and video quality are working at the highest level, VISYTER is a highly versatile tool for providing healthcare services remotely and meeting with colleagues.

Discussion and Conclusion

The results of the telerehabilitation system evaluation were largely positive indicating that the system is highly usable. Implementation of the TR system has improved staff communication and made accessing documents and information faster and easier. Users also found the interface pleasant.

Accessibility was addressed in one respondent who accessed the portal and VISYTER by using assistive technology. Specifically, the tasks of each of the four ASQ scenarios were completed using Dragon NaturallySpeaking 7 and sticky keys.

Despite the strong positive results, it is important to acknowledge negative feedback. The process involved in opening, uploading, and editing documents, and creating and editing action items, should be made simpler and more effective. Automated alerts would further streamline daily interactions between staff, perhaps by reducing email composition, while keeping all staff members informed. The implementation of a usable search function would increase user speed and efficiency and reduce frustration. By reducing the number of times a username and password is required, users may increase speed of use. Finally, portal users would appreciate more helpful error messages.

Difficulties with tasks may have been due to portal functionality, differences between computers (e.g., Windows-based vs. Macintosh or Internet Explorer vs. Firefox) or lack of familiarity with the task. Regardless of the reason for frustration, it will be important to address negative components because the successful implementation of telerehabilitation applications often relies on staff acceptance. Barriers to acceptance may include “technophobia” (Schmeler, Schein, McCue, & Betz, 2008), accessibility, and ease of use (Pramuka & van Roosmalen, 2008).

Strengths and Limitations

The IBM Computer Usability Satisfaction Questionnaires allowed the authors to obtain both quantitative and qualitative information regarding portal usability. While the quantitative results offered strong measures of usability, the qualitative comments provided the input necessary to make improvements.

It is important to note that the results of this usability evaluation should not be generalized to other portal-based systems, or even to the entire CSEP Portal. While the scenarios covered the most important features of the portal, they did not address every possible task. The authors determined that a brief questionnaire would allow staff to demonstrate basic skills and provide input without detracting from daily programming and responsibilities. Though the PITT model may provide the groundwork for online telerehabilitation systems, portal-based systems and other telerehabilitation approaches should be developed based on agency needs and use (Pramuka & van Roosmalen, 2008).

Lastly, the authors did not follow-up with portal users to clarify any comments or responses on the ASQ and PSSUQ as suggested by Lewis (1993). The authors determined that this was not necessary because portal user comments were clearly written and understandable.

Future Directions

Because this evaluation elicited feedback from staff members who rely on the portal and VISYTER, the authors plan to discuss the results of the evaluation with them as well as the developers of the components of the TR system. By sharing recommendations with developers, there may be an opportunity to address user concerns by making technical improvements (e.g., operable search function).

Because user satisfaction may have been impacted by unfamiliarity with scenario tasks, the authors intend to provide training as needed on the commonly missed tasks. This training might also provide further insight into the origin of difficulty.

Although the portal seems to be compatible with speech-to-text software, further accessibility issues may be explored. For example, accessing portal information using text-to-speech software or screen reading software would allow users with visual impairments to access the portal. After user training and portal alterations, the authors may administer another usability questionnaire with different scenarios. The authors predict that as the usability of the TR system increases, clinical staff will use it more often and more efficiently, which will hopefully translate into more consistent and higher quality of service provided to clients.