Testing a mobile app for child abuse treatment: A mixed methods study.

OBJECTIVE: This study provides a preliminary evaluation of the usability and acceptability of a mobile application (sexual assault care algorithm, SACA).
METHODS: An explanatory sequential mixed methods research was used. A quantitative survey was followed up by a qualitative study. A convenience sample of participants (n = 4) was recruited. The research was conducted on a one-on-one basis. In the quantitative phase, a random assignment technique was used to divide four participants into two groups of two participants each. Post-Study System Usability Questionnaire(PSSUQ) and Acceptability e-Scale were used to collect quantitative data. In the qualitative phase, interview, observation, and documentation were used to collect qualitative data. Data were analyzed both quantitative and qualitatively. The qualitative data were linked with the initial quantitative data to determine how the follow-up qualitative data helped explain the initial quantitative results.
RESULTS: The quantitative results suggested that SACA has high usability (5.05 ± 1.83) and acceptability (3.81 ± 1.22). The qualitative results further indicate that the participants thought SACA was easy to use and useful, and most of them would recommend it to others. Areas of improvement include adding features that would calculate and validate the elapsed time since the sexual assault, adding explanations to some buttons, and providing training.
CONCLUSIONS: Our findings highlight the value of using a mixed methods research design to conduct a usability and acceptability test. Nurses are more likely to adopt a new technology for their evidence-based practice when the technology is easy to use and useful and requires less time to find the right piece of guideline evidence. Individualized training needs to be designed based on users' characteristics.

What is known?

Mixed methods research design can be used to increase the understanding of a research phenomenon; however, it is not always used in usability and acceptability testing.

What is new?

This article highlights the value of using a mixed methods research design to conduct a usability and acceptability test. The results provide important implications for nursing informatics.

Introduction

More than 334,626 children experience sexual abuse or neglect each year in the United States [1]. Although the use of clinical guidelines is improving victims’ outcomes, the complexity of the exams and procedures makes it hard for providers to seek the right information from the guidelines [2]. The use of mobile apps is a useful strategy to enhance providers’ guideline adherence. Mobile apps make it relatively easy for clinicians to remember and follow the guidelines’ requirements and suggestions. Health care professionals find mobile apps accessible, acceptable, and useable [3].

The authors developed a mobile application called sexual assault care algorithm (SACA) to aid in rapid decision-making and provide guidance for health care providers of child sexual abuse victims, especially those working in emergency rooms [4]. The contents and logical rules of SACA follow the paper-based Child Abuse Medical Provider (CHAMP) program’s guidelines. CHAMP is a New York State educational network of child abuse providers. It is funded by the New York State Department of Health.

The objective of this study is to provide a preliminary evaluation of the usability and acceptability of SACA. The research questions are:

What were the users’ experiences when using SACA?

How do developers optimize users’ experiences of using SACA?

Methods

Study design

This study employed an explanatory sequential mixed methods research design [5], which included an initial quantitative survey and a follow-up qualitative study. The survey recorded the participants’ usability and acceptability scores. The quantitative results were then used as a foundation upon which to build the second, qualitative phase. The emphasis of the study was on the qualitative strand, because the major design of the study was mainly associated with the qualitative tradition.

The participants’ individual experiences were analyzed independently to understand their specific complexities and uniqueness. A cross-participant theme analysis was then conducted to combine similar themes from different cases. Rich and complex data from multiple participants and resources illuminated important themes about participants’ experiences using SACA. The qualitative results helped explain and give insight into the initial survey results.

Participants

The inclusion criteria of the eligible participants included any potential users of SACA in the nursing profession. Following Institutional Review Board approval, a convenience sample of participants (n = 4) was recruited over 2 months. The participants were all pediatric trained Sexual Assault Nurse Examiners (SANE) who worked in an emergency department in the same institution. Virzi [6] suggested that four to five subjects could help identify about 80% of usability problems. Six and Macefield [7] suggested that three to 20 subjects are enough for the usability test to evaluate a novel user-interface design and to discover problems.

Setting

The research was conducted in a private conference room of a university on the east coast of the United States. The research was conducted on a one-on-one basis. Up to 60 min were allotted for each session. A numerical code was assigned to each participant as an identification number for use on all data collection forms.

App test link

The platform used to build SACA was Appery, a cloud-based platform [8]. Appery provides an online mobile simulator of an app through a test link (i.e. a website) for developers to test the app (for example, by collecting usability data). There is no difference in users’ experience when using either the simulator or the real app. Therefore, SACA could be tested in an environment similar to the one in which it is launched from the app store as a native app. An iPad was used by each participant to interact with the simulator of SACA in this study.

Research procedures

Before the study started, a 10-min orientation was given to the participants. Each participant was provided with a sheet that explained the project’s aim and procedure. It was explained that the aim of this study was solely research, not to test their clinical knowledge levels.

As indicated in Fig. 1, two clinical scenarios about child abuse were designed by one of the authors, who is a board-certified child abuse pediatrician and who led in the development of the CHAMP guidelines used for child sexual abuse testing and treatment. Three case presentations were provided, followed by three questions each. Participants were asked to identify the piece of CHAMP guideline information that provided direct guidance for testing and treatment of patients based on information provided in each question. The CHAMP guideline was either in the printed guideline or in SACA. Correct answers to all questions (the location of the correct piece of guideline) were written by the aforementioned expert.

Fig. 1

Research procedures.

All answers were about the location of the piece of the guideline (either in paper or SACA). To answer the questions, participants needed to know how to navigate the paper-based guidelines and SACA. Since it would be the participants’ first time interacting with SACA, the research team provided participants a brief overview of how to navigate SACA using an iPad. This same iPad was used by the participants in the study. No assistance was offered when the participants used SACA. After participants finished answering the questions following the guidelines, they were asked to complete a 33-question online survey and an interview.

Data collection

This study collected both quantitative and qualitative data (Table 1). Quantitative data were collected from a survey. After participants answered questions regarding two child abuse scenarios, they were asked to fill out an online survey. The survey link was opened in another iPad (which was different from the tablet used to test SACA).

Table 1

Quantitative and qualitative data collection tools and data collected.

Component	Survey	Interviews	Observations	Documentation
Tools	11 items about demographic information,16-item PSSUQ,6-item Acceptability e-Scale	Open-ended questions	Direct observation notes,Reflective notes,Transcript of participants’ think aloud,Video recorded from the camera	Screen shot of SACA,Highlighted paper-based guideline
Data Collected	Demographic information	Participants’ suggestions for SACA improvement	Time to complete each question	Accuracy rates of the answers
	Usability score	Participants’ perceptions about their use of SACA	Process of looking for the evidence in SACA	Location of the evidences in SACA
	Acceptance scores	Participants’ responses about whether they will use the SACA in the future	Process of looking for the evidence in the paper-based guideline	Location of the evidences in the paper-based guideline
		Participants’ perceptions of their understanding of the scenarios and questions	Participants’ interaction with the scenarios, information in the questions, and guidelines (either in paper or in SACA)

Note: PSSUQ= Post-Study System Usability Questionnaire; SACA = sexual assault care algorithm (app).

Qualitative data were collected from interviews, observations, and documentation. Semi-structured interviews were conducted immediately after participants completed the survey. The interview took place at the same location as the testing. During the interview, open-ended questions were mostly used. Specific topics based on observation of participant interaction with SACA were further probed.

Observation and documentation were conducted while participants interacted with the guidelines to answer questions. When participants followed the paper-based guideline, they were asked to highlight the related information in the paper. When they followed SACA, they were asked to take a screenshot of the page that provided the evidence needed to answer the questions. The paper-based guidelines, with highlights and app screenshots, were collected for further analysis. After participants identified the piece of guideline information, they were asked to share its location with us. The think-aloud method [9] was used. Participants were asked to state their thoughts and feelings aloud as they viewed the screens; give feedback regarding SACA’s content, presentation, and navigation; explain their hesitation at certain screens or skipped screens, and convey overall impressions. A camera was used to audio and video record participants’ finger interaction and their thoughts while they used SACA.

Data sources and measurement

The quantitative data source is from the survey which included three sections with a total of 33 questions: 11 items about demographic information, the 16-item Post-Study System Usability Questionnaire (PSSUQ), and the 6-item Acceptability e-Scale. PSSUQ has 16 items with a 7- point scale [10]. It is usually used to assess the quality of participants’ experiences when they interacted with the software interface. Cronbach’s α for the PSSUQ range from 0.91 to 0.96, which indicates excellent reliability [11]. Permission was obtained for the use and modification of PSSUQ. The quantitative outcomes of the study include participants’ demographic information, their rated usability scores, and rated acceptability scores.

In this study, the PSSUQ was modified from a 7- to a 5-point scale ranging from 1 meaning strongly disagree (i.e., low usability) to 5 meaning strongly agree (i.e., high usability). The purpose of this modification was to uniform questionnaires to have one type of option scale only. This uniformity is more effective in clarity for participants. However, the reliability and validity of the 5-point Likert scale PSSQU were not found in the literature. To ensure the reliability and validity of the findings, the data collected from the 5-point scale PSSUQ were converted to the 7-point scale following the data transforming procedure suggested by IBM [12] for further data analysis.

A six-item Acceptability e-Scale [13] was used to measure SACA’s acceptability. Permission was obtained for the use of this research instrument. This scale allows participants to evaluate the ease, helpfulness, and overall satisfaction of using SACA. Cronbach’s α coefficient is 0.72 [13]. Item responses range from 1 (low acceptability) to 5 (high acceptability). The word “system” in the original questionnaires was changed to be “app” for this project.

The data sources for qualitative data analysis include the interview transcript, the researcher’s observations, and reflective notes, videos of participants’ finger interaction with the app, and transcripts of participants’ thoughts. The qualitative outcomes of the study include the participants’ perceptions about SACA and their understanding of the scenarios and information in the questions; participants’ intent to use SACA in the future; participants’ suggestions for SACA improvement; participants’ interaction with the scenarios, questions, and guidelines (either on paper or in SACA); the evidence selected by the participants in SACA and in the paper-based guidelines and how it was selected; time to complete each question; and the accuracy rate of participants’ answers. Major themes for each participant’s use of SACA were identified.

Bias

To eliminate the fact that users might get familiar with the guideline contents from paper-based guidelines (the format they are familiar with) before they used SACA to seek the clinical evidence, a random assignment technique was used. Four participants were divided into two groups of two participants each (see Fig. 1). Participants who were assigned to group 1 assessed scenario 1 and answered questions 1–3 using the paper-based guideline. Then they assessed scenario 2 and answered questions 4–6 using SACA. Participants who were assigned to group A assessed scenario 1 and answered questions 1–3 using SACA. Then they assessed scenario 2 and answered questions 4–6 using the paper-based guideline.

Quantitative variables and statistical methods

The quantitative variables from the survey include participants’ rated usability scores and rated acceptability scores. The mean, standard deviation (SD), and rank of the scores were calculated and discussed in terms of whether the feedback was positive. The average usability score and acceptability score of each participant was compared to those of each other participant.

The quantitative variables from observation and documentation include the accuracy rate of the answers and time spent to answer questions. To calculate the accuracy rate of the answers, the following procedures were used: If a participant’s answer (the location of the right piece of guideline information) was the same as the correct answer, this question was evaluated as being accurate. Each participant’s percentage of correct answers was calculated as the accuracy rate. Each participant’s time spent to answer questions and the accuracy rate of the answers were compared when paper-based guidelines or SACA were used.

Results

Four nurses at a large university participated in the study. Each was a White female with at minimum an associate degree. These four nurses had a broad range of age levels (from 35 to 64 years). Their years of working experience varied significantly (from 3.5 to 20 years), as did the number of sexual assault examinations each had performed (from 40 to over 400) (Table 2).

Table 2

Demographic information of the participants.

Participant	Group	Professional title	Employment status	Number of sexual assault examinations previously performed	Years of work experience in the related field	Highest degree or level of education	Age (years)	Gender	Race/ethnicity (that best represents you)
P1	1	Registered nurse	Full time	40	3.5	Bachelor’s degree	35–44	Female	White
P2	1	Registered nurse	Full time	90	4	Graduate or professional degree	45–54	Female	White
P3	A	Registered nurse	Full time	100	5	Graduate or professional degree	25–34	Female	White
P4	A	Registered nurse	Full time	Over 400	20	Associate degree	55–64	Female	White

Note: SACA = sexual assault care algorithm (app).

Descriptive data

Table 3, Table 4 demonstrate the four participants’ mean rating scores and rank for each item of the PSSUQ and Acceptability e-Scale. The results indicated that the overall mean usability score of SACA was 5.05 ± 1.83 across items. Three of four participants’ average usability scores were above 5.0, and one was above 6.5. The overall mean acceptability score was 3.81 ± 1.22. Three of four participants’ average acceptability scores were above 4.0. Only participant 4 (P4) scored low (i.e. less than 2.5) on almost all items. The mean rating of higher than 4.0 using PSSUQ and higher than 3.0 using Acceptability e-Scale indicated participants’ agreement that they had positive experiences in using SACA, and therefore the usability of SACA was considered high.

Table 3

Participants’ rating scores for each item of the Post-Study System Usability Questionnaire.

Items	P1	P2	P3	P4	Mean	Rank
Overall, I am satisfied with how easy it is to use this app.	7.0	5.5	7.0	2.5	5.50	1
It was simple to use this app.	7.0	5.5	7.0	2.5	5.50	1
It was easy to learn to use this app.	7.0	5.5	7.0	2.5	5.50	1
I felt comfortable using this app.	7.0	5.5	7.0	2.5	5.50	1
The interface of this app was pleasant.	7.0	5.5	7.0	2.5	5.50	1
I liked using the interface of this app.	7.0	5.5	7.0	2.5	5.50	1
The information provided for the app was easy to understand.	7.0	5.5	7.0	2.5	5.50	1
This app has all the functions and capabilities I expected it to have.	7.0	5.5	5.5	2.5	5.13	2
I was able to efficiently complete the tasks and scenarios using this app.	5.5	5.5	5.5	4.0	5.13	2
The information provided for this app (online help, documentation) was clear.	7.0	4	7.0	2.5	5.13	2
The information provided for this app was effective in helping me complete the tasks and scenarios.	7.0	5.5	5.5	2.5	5.13	2
Overall, I am satisfied with this app.	7.0	5.5	4.0	2.5	4.75	3
The organization of information on the app screens was clear.	7.0	5.5	4.0	2.5	4.75	3
Whenever I made a mistake using the app, I could recover easily and quickly.	4.0	5.5	7.0	2.5	4.75	3
It was easy to find the information I needed.	7.0	5.5	4.0	2.5	4.75	3
The app gave error messages that clearly told me how to fix problems.	4.0	4.0	1.0	2.5	2.88	4
Mean ± SD	6.53 ± 1.06	5.31 ± 0.51	5.78 ± 1.75	2.59 ± 0.38	5.05 ± 1.83 (Overall)

Table 4

Participants’ rating scores for each item of the Acceptability e-Scale.

Items	P1	P2	P3	P4	Mean	Rank
How much did you enjoy using this app?	5.0	4.0	5.0	2.0	4.00	1
Was the amount of time it took to complete this app acceptable?	5.0	4.0	5.0	2.0	4.00	1
How helpful was this app in describing what you want to know when facing a client/patient?	5.0	4.0	4.0	2.0	3.75	2
How would you rate your overall satisfaction with this app?	5.0	4.0	3.0	2.0	3.50	3
Mean ± SD	5.00 ± 0.00	4.00 ± 0.00	4.25 ± 0.96	2.00 ± 0.00	3.81 ± 1.22 (Overall)

Seven items had the highest rank in the PSSUQ, and two items had the highest rank in the Acceptability e-Scale. The average time spent answering one question using SACA (2 min) was shorter than the average time using paper-based guidelines (2.58 min). The average accuracy rate was the same (67%) regardless of whether SACA or paper-based guidelines were used.

Qualitative results

The qualitative results (Table 5) were collected from interviews, observation, and documentation. Five themes were generalized based on 23 codes. These themes relate to participants’ interactions with scenarios, questions, paper-based guidelines, and SACA and to participants’ intent to use SACA in the future. These themes are discussed in detail in the following list:

Table 5

Results from observation and documentation of the participants.

Participant	Average time (minutes) per question using paper-based guidelines	Average time (minutes) per question using SACA	Overall average time (minutes) per question using guidelines	Accuracy rate using paper-based guidelines (%)	Accuracy rate using SACA (%)
P1	1.0	1.0	1.0	100.0	100.0
P2	5.3	2.3	3.8	100.0	66.7
P3	3.0	2.6	2.8	33.3	33.3
P4	1.0	2.0	1.5	33.3	66.7
Mean	2.58	2.00	2.29	66.65	66.68

Note: SACA = sexual assault care algorithm.

Participants’ interactions with scenarios and questions

All participants seemed focused during the test. Participant 1 and 2 (P1 and P2) understood the scenarios and questions quickly and without any problems. Participants 3 and 4 (P3 and P4) looked frustrated when trying to comprehend the scenarios and questions.

Participants’ interaction with paper-based guidelines

Two patterns were identified. One was a simple and efficient pattern exhibited by P1 and P4. A short average time (1 min) was used to locate the evidence in the guideline, which helps to make a treatment plan for each question. Only simple words were highlighted in the guideline. The other was a complicated pattern, exhibited by P2 and P3. Long average times (3.0 and 5.3 min) were used to answer each question. The entire paragraph and part of the table were highlighted in the guideline.

Participants’ observed interaction with SACA

Two patterns were identified. First, P1 and P4 started with “time elapsed.” On the start page, each chose the option “If you know the time elapsed since the abuse or assault, please click here.” The average time to answer each question using SACA was short (1.0 and 2.0 min). Second, P2 and P3 started with “type of algorithm.” On the start page, each chose the option “If you know the type of algorithm, please click here.” The average times (2.3 and 2.6 min) to answer each question using SACA were longer than those of the other two participants.

Participants’ perceived experiences of using SACA

All the participants said SACA was easy and useful. They commented that SACA is “easy to use,” “easy to understand” (except P4, who was confused by some information), has “user-friendly screens and features”, is easier than paper (to “target information specifically and not have to go through the entire paper”, and “easy to start by selecting the type of patients to get exact information needed”), and had a similar workflow to using other apps or paper-based guidelines.

Participants’ intention to use SACA in the future

Most participants would want to use SACA in the future and recommend it to others. P4 suggested novices to use SACA.

Mixed methods results

For an explanatory sequential mixed method, the qualitative data were linked with the initial quantitative data to determine how the follow-up qualitative data helped explain the initial quantitative results. The results were illustrated using a joint display table (Table 6). All the data from different sources for each participant were arrayed for comparison and explanation purposes. The results from two strands were categorized under the two main components of the technology acceptance model (TAM)—ease of use and usefulness [14]—to bring data together visually. TAM has been one of the most influential models of technology acceptance and dominates the field. It explains the main factors and mechanisms of technology use. The quantitative and qualitative results were compared and contrasted to see whether the two strands were convergent to draw out new insights. The joint display tables also serve as triangulation for the qualitative findings. Table 6 will be interpreted in the following section.

Table 6

Joint display table comparing the results from different sources.

Survey items	Rank	Qualitative themes & categories	Sample quotes from interviews	Notes from observation and documentation
Ease of use
Overall, I am satisfied with how easy it is to use this app.	First	Feeling easy and comfortable	The app is easy to use.	Only P4 looked frustrated when using app.
It was simple to use this app.			Easier than paper:“Targets information specifically and do not have to go through the entire paper”;“Easy to start by selecting the type of patients to get exact information needed.”
I liked using the interface of this app.			User-friendly screen and features
I felt comfortable using this app.			Similar to current workflow:“People are used to computerized things”;“As a sexual assault provider, we have a piece of paper. We check things off. The app is similar to that piece of paper.”
It was easy to learn to use this app.				Most of them learned how to use the app fast and asked no questions while interacting with the app.
The information provided for the app was easy to understand.			Easy to understand. However, P4 said she was confused by the questions asked in the app.
The interface of this app was pleasant.		Positive emotion		All participants were focused.
How much did you enjoy using this app?
Was the amount of time it took to complete this app acceptable?		Shorter time to complete app		They had shorter times with similar accuracy rates using app compared to using paper-based guidelines.
Usefulness
I was able to efficiently complete the tasks and scenarios using this app.	Second	Efficiently complete the tasks and scenarios		All of them completed the questions using the app.
How helpful was this app in describing what you want to know when facing a client/patient?		Helpful in front of patients	Handy (on the phone, not on paper).“The information is right around you when you need it. What applies to patients is in front of you and it guides you with the response to the treatment.”
		Helpful to teamwork	Helps working with team members:“Everyone would be on the same page in treatment response to the patients”;“With the app, I can show other providers what we should be doing based on the guidelines.”
This app has all the functions and capabilities I expected it to have.	Second	Functions and capabilities	Adding how to calculate and validate the elapsed time.
The information provided for this app was effective in helping me complete the tasks and scenarios.		Information	Need training on how to use this app based on scenarios, including how to calculate time; need explanation on some buttons for better future use.	Participants have different reading comprehension levels.P1 understood the questions very quickly without any problems.P2 verified the info, which she said is accurate.P3 and P4 had difficulty comprehending the scenarios and questions, including the elapsed time in the questions. They read slowly and repeatedly read the same questions several times. While reading, P3 highlighted some words of the questions to help her comprehension.
The information provided for this app (online help, documentation) was clear.			Explain some buttons, such as “follow up” to whom and “mandatory report” to whom.
The organization of information on the app screens was clear.	Third	Organization	“The app is organized.”	There were two ways they got started on the first page.P1 and P4 chose the option, “If you know the time elapsed since the abuse or assault, please click here.” The only difference is that P1 was confident in knowing where to go from there, while P4 looked frustrated and confused afterwards. Their average times to answer each question using the app were 1 and 2 minutes; the accuracy rates were 100% and 33%.P2 and P3 chose another option on the start page of the app: “If you know the type of algorithm, please click here.” Their average times to answer each question using the app were 2.3 and 2.6 minutes, which were longer than those of the other two participants; their accuracy rates were 67% and 33%, which were not as high as the other two.
It was easy to find the information I needed.			Faster than paper:“It will not take too much time trying to get all the details, especially when I am there with the patients.”
Whenever I made a mistake using the app, I could recover easily and quickly.		Error message		They looked around to find the right path.
The app gave error messages that clearly told me how to fix problems.	Fourth
Overall, I am satisfied with this app.	Third	Overall	All but P4 would use the app in the future.All participants would recommend the app to others. P1 would like to know more about the app before she recommends it to others. P4 only wants to recommend it to novices.All but P1 suggested training on how to use the app based on scenarios.
How would you rate your overall satisfaction with this app?

Discussion

Key results

High usability and acceptability

Altogether, in response to the first research question, the quantitative results from the survey suggested that SACA has high usability and acceptability. The qualitative results further indicate that the participants thought SACA was easy to use and useful, and most of them would recommend it to others. This is consistent with the TAM model [14], suggesting that when users perceive a new technology to be easy to use and useful, they are more likely to use it. Our discussion follows the two themes of TAM: ease of use and usefulness.

Ease of use

Feeling easy and comfortable. Participants rated highest the “ease of use” item in both questionnaires. Most of them strongly agreed that it was easy to learn SACA, easy to understand its information, and simple to use; they felt comfortable using SACA and liked using its interface. The qualitative results were consistent with the quantitative findings and provided more insights. Participants thought SACA was easier to use than paper-based guidelines because it starts with the specific patient’s condition and provides targeted information. Participants intuitively knew how to use SACA, probably because of the user-centered app design that is consistent with most other apps and needs minimum training for users to learn [15]. This finding was further explained by the results from the interviews: Participants thought SACA had a user-friendly screen and features, that using it was similar to using their other mobile apps, and SACA’s workflow of treating patients was similar to what the participants currently use—a checklist on a paper sheet in the emergency room. Connecting with people’s prior knowledge and experience helps make the new technologies meaningful to them and therefore easier for people to implement new technologies in the future [16].

Shorter time to complete the app. The Acceptability e-Scale survey results indicated participants strongly felt the amount of time it took to complete this app was acceptable. The average problem-solving time per question was faster using SACA (2 min) than paper-based guidelines (2.58 min), meaning SACA is more efficient than paper-based guidelines, although participants’ average accuracy rate was the same (67%). This result is consistent with another research finding that an electronic guideline-based decision support systems had high efficacy and efficiency compared to the paper-based guideline [3].

Positive emotion. Participants rated highest the survey items indicating emotion (i.e., “The interface of this app was pleasant,” and “How much did you enjoy using this app?”). All participants were focused when using SACA, although they looked either confident or frustrated. They were not asked about their perceived emotions during the interview, so it is unknown what exact emotions participants felt while using SACA.

Usefulness

Efficiently complete the tasks and scenarios. Giving the second-highest rating to this survey item, participants believed they were able to efficiently complete tasks and scenarios using this app. The qualitative results from the observation indicated that all participants completed the questions using SACA with an average time of 2 min per question and an average accuracy rate of 67%.

Helpful in front of patients. Participants also gave the second-highest rating to the item indicating that SACA helped describe what they wanted to know when facing a client/patient. During their interviews, they said SACA is easy to access when around patients; the information is organized; SACA is faster than paper; it provides useful and accurate information (e.g., a hyperlink to the Centers for Disease Control and Prevention) to solve the problem; and it helps providers to work independently.

Helpful to teamwork. Participants commented that SACA helps them work with team members by keeping everyone on the same page when treating patients. Inter-professional team collaboration improves patient outcomes and reduces health care costs [17]. Adherence to clinical guidelines is important for team collaboration, although it might be hard to achieve when professionals in various disciplines use paper-based guidelines. SACA could provide instant access to anyone with a smartphone or tablet device, allowing automatic clinical decisions that can be presented and shared with health care providers, customized information about patients’ situations, and recourse from other authorized websites. All these advantages may help professionals from different disciplines to be on the same page.

Optimize users’ experience

In answering the second research question, the quantitative results suggested the following areas of improvement:

Functions and capabilities

Participants rated second-highest the survey item, stating that SACA has all the functions and capabilities they expected. They also provided suggestions for developers to improve SACA, including adding features that would calculate and validate the elapsed time since the sexual assault, adding explanations to some buttons (e.g., “mandatory report” to whom), and providing training on how to use SACA based on real-world clinical scenarios.

Information

Participants also rated second-highest the survey item stating that the information provided by SACA was clear and effective in helping them complete tasks and scenarios. Qualitative results indicated that participants’ comprehension level of SACA’s information and their years of experience in a related field might affect their rating of those survey items. Although it was not required, P2 verified the information in SACA while working on the questions. She said the guideline in SACA was accurate. More explanations for some buttons were suggested to help users answer the questions well. For example, incident history may need to be added under medical history, and the terms consent and mandatory report need to be explained.

Future training

Survey results indicated that participants gave a third-level rating to the survey item related to the organization of the information in SACA. The qualitative results explained that participants’ strategies for using SACA might be associated with their rating of those survey items. Those who chose “time elapsed since the abuse or assault” on the SACA start page took less time to complete tasks. In future training, this effective strategy for using SACA might be recommended.

Participants rated lowest the survey item indicating there was no error message or reminder when they made a mistake. This is because SACA provides standard clinical guidelines based on an assumption that there is no wrong clinical encounter when using SACA; therefore, SACA does not have error message features. However, related training is necessary to help users become familiar with SACA, especially on how to find the right piece of guideline information and how to return to the previous page.

Participants did not give the highest rating to how they felt about SACA overall. P4 had the most complaints. Our observation indicated that she was frustrated while reading the scenarios and questions and when interacting with SACA. It was not surprising that she consistently gave scores lower than 3 (with the lowest mean scores 2.59 for usability and 2.00 for acceptability). P4 explained that SACA’s cognitive information process did not match hers, and she challenged the accuracy of the paper guidelines. She explained in her interview that SACA was not suitable for people like her with many years of experience in a related field. This is consistent with the finding that experts have their own cognitive information process, so it may be harder to accept other points of view [18]. She suggested that SACA is only good for novices and she had recommendations to improve SACA so that it would be more helpful to novices. The age might be another factor in her use of technologies. She is between 55 and 64 years old and has 20 years’ experience in a related field. This is consistent with research showing that age is one of the factors that influence the acceptance of new technologies [19] and that older adults have different technology-use patterns than younger adults [20]. Future training must be tailored for individual users based on characteristics such as age, work experience, reading comprehension level, and cognitive information process.

Limitations

A very small number of participants were evaluated. Future studies should evaluate SACA with a larger number to provide more information about its effects on providers’ treatment of sexually assaulted children. Because our sample lacked diversity in the participants’ background (e.g. gender, ethnic group, prior knowledge, professions), our findings cannot be generalized to other groups (e.g. males, students, residents, etc.). It is unknown if a newly graduated resident would find SACA as easy to use and useful as did the participants in our study. Participants’ digital literacy was not measured and thus it cannot be claimed that SACA would be acceptable or useable by those with low digital literacy. Participants were not asked about their perceived emotions during the interview and did not test the use of SACA in front of a real patient. Future studies are required to thoroughly evaluate the efficacy of SACA, including participants’ positive emotions from a more diverse sample. Also, future studies should be conducted in a real-world setting where users are facing actual patients.

Interpretation

The results provide important implications for nursing informatics. This research expands the use of information management systems to mobile apps that could be integrated into nursing science. Lessons learned about SACA design, revisions, and training are helpful to better integrate nursing science with the mobile app. The results reinforce that stakeholders of nursing informatics include not only nurses but also interprofessional health care teams [21].

The results help understand how nurses interacted with the information structures of a mobile app and how they processed the information supported by the app to make exam and treatment plans. SACA’s user-centered design makes app users not have to read the entire guidelines to look for the right evidence to answer the questions. SACA provides an automatic process for users to focus on the most important information. The process is under the user’s control and gets the user’s attention. New information is provided based on the user’s prior knowledge in a step-by-step manner. Users can start from what they already know––patients’ information and their clinic judgment––to make choices in SACA, and SACA leads them to the right evidence to solve the problem. All these features help decrease users’ cognitive resources so they can work faster with the same accuracy rate to provide the right exam, testing, and treatment for the patient compared with using paper-based guidelines.

The results further indicated that nurses are more likely to adopt a new technology for their evidence-based practice when the technology is easy to use and useful and requires less time to find the right piece of guideline evidence. Therefore, to increase the adoption of an app, the interface and structure of the app need to be easy to use and useful to nurses. To make the guideline information in the app useful for the decision-making process of nurses, it is critical that the information provided by the app is timely and relevant. The design of the information system should follow nurses’ cognitive information and workflow processes.

Information and explanations were already added based on the participants’ suggestions (Fig. 2). Individualized training will be designed based on users’ real-world workflow to give users a meaningful learning experience [16]. The training will consider users’ age, reading comprehension level, prior knowledge, work experience, and thinking process when using SACA. Appropriate strategies to navigate SACA will be taught to the users. They will be encouraged to choose “time elapsed since the abuse or assault” on the start page if they are not familiar with different types of algorithms. They will be convinced that they do not need to validate the information in SACA, because it has already been validated by a consultant [4]. Per the participants’ request, how to calculate patients’ elapsed time accurately will be taught to future users.

Fig. 2

Screen shot of sexual assault care algorithm.

Conclusions

Our findings highlight the value of using a mixed methods research design to conduct a usability and acceptability test. Our data provide preliminary evidence of high usability and acceptability of SACA in a sample of providers and suggest that the mobile app has the potential to improve provider’s compliance with the decision-making guidelines for testing and treatment plans for children who might be sexually abused or assaulted. The results provide important implications for nursing informatics. Information collected from this project was used to guide revisions to SACA for better use by nurses. Participants identified issues that will be addressed prior to the next iteration of testing SACA. Identification and correction of these issues are critical before making SACA available to a larger audience. Our findings also underline the necessity of tailoring the training for using SACA based on users’ characteristics such as age, work experience, reading comprehension level, and cognitive information process.

Funding

The development of the application for SACA was partially supported by funding from the Upstate Foundation, a 501(c)3 non-for-profit corporation (upstatefoundation.org).

Author statement

Shuhong Luo: Conceptualization, Methodology, Investigation, Data Curation, Formal Analysis, Writing- Original Draft Preparation, Visualization. Ann S. Botash: Resources, Methodology, Writing- Reviewing and Editing, Validation, Project Administration

Declaration of competing interest

No conflict of interest has been declared by the authors.