Designing a health referral mobile application for high-mobility end users in Indonesia.

OBJECTIVE: The purpose of this study was to design a suitable mobile application for high mobility medical staff to support the health referral system of Indonesian universal health coverage.
METHODS: Design science research (DSR) approach were conducted with two iterations. The first iteration was conducted by designing a high-fidelity mockup and evaluating the design using the System Usability Scale (SUS) framework to evaluate the overall usability of the system (n = 48). The second iteration was conducted by designing a running prototype and evaluating the design using the Post-Study System Usability Questionnaire (PSSUQ) framework to evaluate the system's usefulness, information quality, and interface quality (n = 31). For designing the user interface, the eight golden rules guided the process.
RESULTS: The results found that the design for the first iteration was good and the second iteration's design was satisfactory to the respondents. The information quality aspects underwent several changes such as the implementation of local language as the users deemed it to be important to help them easily understand and use the system and eventually support their mobility at the health facility.

Introduction

The development of Information Technology (IT) in Indonesia, which is growing rapidly, is being driven by the presence of the Internet. In 2017, the number of Internet users in Indonesia reached more than 143 million (~54.68% of the total population) [1]. This growth has been and continues to be influenced by the utilization of smartphones. Based on a 2017 survey by the Association of Internet Providers in Indonesia (APJII), 50% of Indonesians have smartphone devices, which they mainly use for Internet access. Smartphones significantly impact the number of Internet users in Indonesia because they provide wireless access [2]. In addition, smartphones have more capabilities such as more applications, high resolution camera and friendly screen interaction than did previous phone technology than did previous phone technology. Widespread connectivity to the Internet also provides more opportunities for application developers to create solutions to daily activity issues [3], which make smartphone utilization inherent in our daily lives [4]. These applications can solve transportation, logistics, financial management, business, and health-related activity problems. Various mobile health (m-health) applications have been created with specific objectives, such as monitoring health conditions and remote consultation.

The Indonesian Government has implemented several rules, programs, and health efforts, which were created by the Ministry of Health, to improve health services. One such effort is the health referral system, which became regulated by the Minister of Health Regulation, Republic of Indonesia Regulation 001 in 2012. This health referral system is carried out both vertically and horizontally. Vertical referrals start from primary health facilities (Fasilitas Kesehatan Tingkat Pertama: FKTP); therefore, every citizen first visits his/her assigned health facility when having a health problem. A designated health facility can refer a patient to a higher-level health facility (Fasilitas Kesehatan Rujukan Tingkat Lanjut: FKRTL) if they lack the appropriate medical staff and/or treatment options. However, patients with emergencies do not have to use the referral system. Per Regulation 001, participants of the National Health Insurance program (Jaminan Kesehatan Nasional: JKN) are required to follow the referral system. Participants with commercial health insurance must follow tiered services within the referral system and rules that are adjusted according to the provisions of their insurance policies.

To support the health referral process, the government has developed a web application, which is used by all health facilities in Indonesia to refer patients to other health facilities. Medical staff members are the main users of the system. Currently, the mobile version of this application (Mobile JKN) is only designed to help patients register themselves at designated primary health facilities; the feature that will allow medical staff to send referrals has yet to be implemented. The implementation of an m-health application for referrals is important for medical staff, given their high level of mobility. In addition, as mentioned earlier, most Indonesians are more accustomed to accessing the Internet using smartphones. Thus, the implementation of an m-health application for the referral system, especially the medical staff module, is expected to increase the productivity of medical staff, impact the provision of health care, and eventually improve the effectiveness and efficiency of the referral process. The Mobile JKN patient module still has some problems, as perceived by its users, which are resulting in unsatisfactory utilization of the system. These problems include usability, ease of use, and user interface [5]. This study was conducted to support medical staff practice in using an m-health referral system and to correct issues until the system met their expected usability. This system was chosen for study because of the large number of possible users (there are more than 200 million JKN participants) [6].

Previous studies on designing m-health applications have been carried out using a unified model language (UML) as the communication medium because it can be explained and understood easily by the end user [7, 8, 9]. This allows both the developer and end user to have the same perception of the application. However, we only found one study that explained what methodology would be suitable for designing an m-health application [9]: used the Design Science Research (DSR) method in one iteration to describe the design process, and the authors also evaluated the design's results via the Post-Study System Usability Questionnaire (PSSUQ) to measure the usefulness of the design.

In the current study, we used the DSR methodology with a UML to systematically design an m-health application for a referral system in Indonesia that specifically focused on the medical staff module. To do so, we conducted two iterations. In the first iteration, we used a high-fidelity mockup to test the first design draft. The evaluation of the first iteration was conducted using the System Usability Scale (SUS) model to measure the level of usability and learnability. The SUS is one of the most widely used and well-studied measurement models [10]. After an evaluation of the initial design, we improved the prototype and reevaluated it in the second iteration via the PSSUQ model to measure the levels of system usefulness, information quality, and interface quality. The PSSUQ was also chosen because it provides additional data regarding the design, such as ease of repair, assistance, and feedback on quality. Through this method, the results were expected to capture users' needs and expectations, produce the finest application design, and provide a good understanding of how to design an accurate health referral system.

Literature review

M-Health

M-health is defined as the use of mobile communication technology and networks for health-related activities [11]. Per the Global Observatory for eHealth, an organization that is owned by the World Health Organization (WHO), m-health is a form of public health and medical practice that is supported by mobile devices, such as mobile phones, surveillance devices, personal digital assistants, and other wireless devices [12]. M-health applications are innovative in that they can provide health services at any time and in any location; thus, they overcome existing restrictions, such as geographical boundaries [13]. The development of mobile-based communication is done through smart cellular devices that currently support 3G and 4G networks for exchanging information and data. M-health is a breakthrough in the health industry because it can help health workers provide optimal services via smart cellular device technology in any location.

M-health is a subclass of electronic health (e-health), with the difference being that m-health requires the use of a mobile device [14]. However, the two share the goal of increasing both the effectiveness and the efficiency of medical personnel in providing health services [15]. M-health applications should provide immediate information to support medical staff members' decisions regarding patient care both anytime and anywhere, making its purpose broader than that of e-health. M-health applications also seek to help patients improve their health and engage in preventative care by providing information regarding their current healthcondition.

M-Health implementation

The development of mobile technology for the promotion of health services can be partially linked to technological accessibility, the level of personalization, useful location-based services, and timely access to information [16]. Several studies have evaluated the use of m-health for chronic conditions, such as diabetes [17], chronic obstructive pulmonary disease [18], Alzheimer's disease [19], and osteoarthritis [20]. These studies found that the use of m-health applications can support success in the management of chronic diseases and healthy behavioral changes. However, reaching the objectives of m-health requires significant effort, and the process remains a challenge. Obstacles can include resistance to change, unreliable technology, integration issues, and lack of user education [21], and documentation of these obstacles regarding the design, implementation, and evaluation of m-health is still quite limited. Based on a survey conducted by the WHO, only 12% of member countries have reported the results of evaluations of m-health services, and only a few knew how to effectively evaluate solutions [22]. Hence, the implementation of successful m-health applications is difficult because many obstacles are encountered late in the process, and there is little reference material for guidance.

Health referrals in Indonesia

The referral system in Indonesia has been regulated by the Minister of Health's Regulation Number 001 since 2012. Generally, there are three types of health referrals: patient, specimen, and medical personnel [23]. When a patient is registered for the JKN, they will be assigned to the nearest FKTP. A patient's health referral begins with his/her admission to an FKTP. When that facility lacks the staff expertise, medical equipment, and/or services needed to treat the patient, he/she is referred to the next closest facility that is deemed capable of providing proper treatment (FKTRL). Such referrals aim to decrease the number of patients coming to hospitals when their conditions can be handled by either clinics or public health centers. However, in cases of emergency, patients should go to the nearest health facility. This referral system must be followed by all people who participate in the JKN and other insurance participants who make adjustments per the regulations of their insurance providers. In addition to regulating how the referral system in Indonesia runs and who is required to follow it, Regulation 001 also regulates the data and information that are needed and the procedures that must be carried out in the referral process.

The eight golden rules

The design stage of an application requires preparation of the user interface. To design a user-friendly interface, a designer might need to follow existing design rules, which can be used as directives to ensure ease of use of the application [24]. Of the available rules and directives for creating a good interface, many need to be simplified. Therefore, collections of rules that provide useful summaries of the design process were created. A popular guideline is Shneiderman's eight golden rules [25]:

Strive for consistency: All things that are displayed on the interface must be consistent in terms of both layout and terminology.

Enable frequent users to use shortcuts: These should be available for any important, recurring activity.

Provide informative feedback: This helps users complete actions.

Ensure dialogs to yield closure designs: Users should know that their activity has been completed.

Offer error prevention and simple error handling: The interface should have an optimal design to help users avoid mistakes and easily fix any that occur.

Provide easy reversal of actions: Make it easy to return to the previous page.

Offer support for the internal locus of control: Users should feel that they have control over the application and not vice versa

Reduce short-term memory load: The interface must make the application easy to use and not require the memorization of extensive information.

Previous studies

We searched the keywords “design m-health referral,” “m-health referral,” and “design m-health” in the online databases ScienceDirect, IEEE Xplore, Scopus, and Emerald Insight with a minimum publication year of 2014 (Table 1). We found limited number of papers on the development of mobile application design especially in health area. Most of the publications studied the acceptance, impact and conceptual design of the mobile health application. Before 2018, the studies regarding m-health design have not used a standardised methodology and the evaluation of the design also missing. Thus, this study will enrich the previous studies in the area of the design implementation of mobile health. Three of the most relevant previous studies are summarized in Table 1.

Table 1

Previous studies.

Attributes	Study 1	Study 2	Study 3
Title	Mobile-health application software design and development	Integrating m-health applications for self-management to enhance telecare systems	Design of m-health for Android applications
Journal	International Journal of Computer Science and Information Security	Telematics and Informatics	American Journal of Engineering Research
Authors	Ayangbekun & Kasali	Kao, Wei, Yu, Liang, Wu, & Wu	Rao & Krishna
Publication Year	2014	2018	2014
Keywords	electronic-health, healthcare, intranet, mobile health, patient, smartphone, socket	mobile health application, self-management, telecare, design, science, usability	e-health, mobile health, BMI, Android, mobile application
Purpose	Design and develop e-health and m-health applications with important functions, such as managing short messages and viewing prescription drugs, test results, health tips, and health documents, in a health facility.	Design an m-health application for telecare, where patients can autonomously carry out home-based health management.	Design an m-health application system that allows self-monitoring and self-prevention based on user food consumption data.
Framework Evaluation	Not mentioned	PSSUQ	Not mentioned
Methodology	Not mentioned	DSR	Not mentioned
Research Object	Medical staff and patient	Patient	Patient
Result	Both e-health and m-health applications can perform functions to manage health services in health facilities and connect services at health facilities to patients.	The final design of the m-health application helped the telecare process be carried out autonomously at home.	An m-health application mockup design for self-monitoring and self-prevention based on food consumed.

The noted previous studies did not define their complete methodology or their framework evaluation, except for the study by [9], which only evaluated one iteration. This made it difficult to evaluate the improvements that were made after the first user evaluation. In this study, we used the DSR approach in two iterations for the design and evaluation stages. We also used two different types of usability testing frameworks to measure different objectives. The first iteration would have resulted in high-fidelity mockups while, in the second iteration, the result would have been in the form of a prototype that could test features that could not be tested in the first iteration and was uploaded on Google Playstore. while, in the second iteration, the result would have been in the form of a prototype that could test features that could not be tested in the first iteration. Therefore, we used SUS testing for the first iteration and PSSUQ for the second iteration. This study also used UML diagrams to assist in designing the m-health applications, which was similar to the work of [7, 8, 9].

Methodology

Design Science Research

Information systems—an applied research discipline—uses several theories from other scientific disciplines, such as economics, computer science, and the social sciences. Results from the use of information systems are still under debate, and they cannot always be applied to problems in research and practice [26]. Therefore, DSR is considered relevant in information systems research because it can solve two key issues: the role of IT artefacts [27] and information systems research that is considered less relevant to professionals [28]. In general, information systems research that uses DSR is better suited to problem solving.

Notably, DSR differs from Professional Design regarding the problems faced and the solutions offered. Professional Design solves problems that already exist and can be solved using artefacts of best practices. By contrast, DSR solves an important new problem that must be solved in a unique, effective, and efficient way [29]. In the initial stages, every new artefact made by DSR for a discipline is an “experiment” that can still be questioned. Existing knowledge can be used at the appropriate time, even though it is often found that the required knowledge does not exist at the time that the research is conducted. Creativity and trial and error are characteristics of DSR efforts [29].

The activities of DSR projects are generally divided into five stages: awareness of problems, suggestions, development, evaluation, and conclusions [30]. However, this approach can be modified to suit the research [26]. In the model created by [26], they divided awareness of problems into two processes: identify the problem and motivate and define the objectives of a solution. They also united suggestions and development with the design and development process. In addition, they divided the evaluation process into demonstration and evaluation, and they changed conclusions to communications [26]. Considering the above, DSR clearly has a considerable impact on research in the discipline of information systems regarding problem solving.

Procedures

There were five stages carried out in this study: problem identification and objective definition, design and development, demonstration, evaluation and communication, and conclusions (Figure 1). The design and development, demonstration, and evaluation and communication processes were conducted in two iterations. In the first iteration, the design was made in the form of a mockup, where users could try the flow process and evaluate the design interface. In the second iteration, the design was made in the form of a prototype for use by participants with smartphones that had the Android operating system. The design success rate was measured using the SUS usability testing framework in the first iteration to discern the values of the dimensions of usability and learnability regarding the application's design. In the second iteration, the PSSUQ was used to see evaluate three dimensions: system usefulness, information quality, and interface quality.

Figure 1

Research procedure.

Research instrument

For the evaluation and communication processes, two measures were chosen. The first iteration used the SUS, and the second iteration used the PSSUQ. These measures were chosen because they are well-known usability testing frameworks that are widely used for testing and evaluation [10, 31]. As previously mentioned, the PSSUQ was used in the second iteration because it evaluated the ease of repair, assistance, and feedback regarding system quality that were created by the SUS. It was not possible to use the PSSUQ in the first iteration because the initial design was in the form of a high-fidelity mockup.

The instrument used in this study was a questionnaire that was divided into two parts. The first part was related to the respondents' demographics, the obstacles faced, and the expectations of the current referral process. The second part included a statement that was related to the research, both the SUS and PSSUQ usability testing frameworks, and criticisms and suggestions regarding the application's design. The research statement was answered via a 5-point Likert scale for the SUS and a 7-point Likert scale for the PSSUQ. Readability tests were conducted with five respondents on February 17, 2019 to ensure that all questions and statements could be understood. Feedback from this testing guided the revision of the questionnaire. Generally, improvements were made regarding word selection and highlighting keywords for each statement.

Population and data collection

Data collection in this study was conducted by distributing questionnaires offline to a sample of a predetermined population. The data collection was permitted and supported by Health Department of Tangerang Selatan City. Prior filling out the questionnaires and conducting the demonstration, all participants were asked to read the agreement at the beginning of the questionnaires. If the participants agreed, then the survey was carried out.

The application was designed for medical staff in health facilities whose tasks are related to patient health referrals. Purposive sampling, which is a method that is done randomly on a population with characteristics that are suitable for the research [32], was used. Data were collected by attending meetings of medical staff at a hospital, which allowed direct interaction between the respondents and the researchers.

Results and analysis

Problem identification and objective definition

As mentioned previously, the medical referral system in Indonesia was implemented in 2012 for JKN users and other insurance participants, which, when combined, exceeded 200 million people in 2018. Due to the great number of people involved, the system must be run effectively and efficiently. One way to ensure this is to use IT to assist in the referral process. Some health applications, such as SISRUTE and P-Care, have been made for several platforms. These web-based applications are used by medical personnel to conduct referrals, and Mobile JKN allows participants to register as members and view their history. However, there is currently no mobile-based application that was designed for use by medical staff to offer referrals. A mobile application for health referrals is important for medical staff due to their high mobility while completing tasks. Therefore, the objective of this study was to design an m-health application for the medical referral system in Indonesia that met users' expectations.

First Iteration

Design and development. The process at this stage started with creating a UML diagram. This study used two UMLs: the use case diagram (UCD) and the activity diagram. Both diagrams resulted from previous research [23] with a few additions. Activity diagrams a used to describe the activity or process of the current reference. The activity diagrams guided the design of this application and were used as a reference to ensure that all the referral processes could be implemented through the application. The results of the activity diagram supported the creation of the UCD (Figure 2). The use case that will be evaluated in this study is ‘Refer the Patient’. The flow of the use case is shown in Figure 3. The first step, user has to determine the referral type which are emergency or policlinic. The next steps (step 2 until step 5) steps are fill out the data including patient profile, referral schedule, referral data, ICD-10 and ICD-9 CM code. Then, user can submit the referral and receive a success message. The interface's design was conducted by following the eight golden rules. All but two rules were implemented. The offer error prevention and simple error handling rule was left out because users were not allowed to fill in the data. An example of the interface that was designed to refer patients can be seen in Figure 4.

Figure 2

Use case diagram.

Figure 3

Flowchart of the GUI

Figure 4

Example of a user interface for referring the patient (Step 1, Step 2 and Step 6).

Demonstration

For the first population, the evaluation of the first iteration was done offline by attending a meeting of medical personnel on February 26, 2019 at a district hospital. The number of respondents in the first iteration was 48 (n = 48) (Table 2). For the first demonstration, the respondents were asked to complete one use case which was to refer the patient following the flowchart in Figure 3. The mockup of the mobile application was built with marvelapp.com and available on bit.ly/rujukanPetugas. Respondents, with their own smartphones, were asked to access the mockup through the link.

Table 2

Respondents' profile for the medical staff module in the first iteration.

Variables		Total
Gender	Female	41 (85%)
	Male	7 (15%)
Age	24–30 years old	19 (40%)
	31–40 years old	23 (48%)
	>40 years old	6 (12%)
Occupation	Nurse	24 (50%)
	Doctor	23 (48%)
	Other Medical Staff	1 (2%)

Evaluation and communication

The results of the questionnaires after the demonstration were used to evaluate the design's draft. The data obtained by the questionnaire were then processed for interpretation. The calculation was made following the SUS evaluation method (Table 3).

Table 3

SUS value.

Design for module	SUS value (average)	Note
Medical staff	62.7	OK

Prior to evaluating the SUS value, reliability of the instrument was conducted to evaluate the Cronbach's Alpha (CA) and the Composite Reliability (CR). The values of the CA and the CR were 0.736 and 0.68, respectively. The SUS values were measured via a method developed by [33]. The measurement was done with words called “Adjective Ratings” instead of numbers. We chose this measurement so that the value that was produced by the SUS would be easier to understand. Based on the measurement, the application module designs were in the “OK” range, meaning that the results of the design were good, but many changes could be made to make it more fun to use and easier to learn. To determine what improvements should be made, we utilized an open-question questionnaire, which was designed to elicit criticism and suggestions. The suggestions were grouped to distinguish between improvements to usability and functionality, which are shown in Table 4. In addition, the grouping determined the priority of the design improvements, which were the usability groups items.

Table 4

Suggestions for the medical staff module in the first iteration.

No.	Usability	Number of Occurences	Functionality	Number of Occurences
1	Use of Indonesian in the ICD-9 and ICD-10 field	4	Provide reasons why a referral is rejected by health facilities	2
2	Display the number of empty rooms	5	Looking for another referral to a health facility if the first one is rejected	1
3	Indicate that the data are already registered in the system	2
4	Changes to the sequence of the field for emergency referrals	3
5	Display the rooms' amenities when making emergency referrals	2

Second iteration

Design and development

The design of this iteration focused on improving the initial design based on the results of the evaluations from the first iteration. Those results found that some information was not in accordance with the current conditions. The design of the second iteration was technically different from the first iteration because the design was built into a prototype, which is available in the Google Playstore. However, the design was still implemented by using the eight golden rules to assist in directing the design process. Table 5 shows the improvements that were made in the second iteration. Examples of the design improvements are shown in Figure 5. Notably, all fields on the referral form were updated to the local language.

Table 5

Second iteration improvements.

No	Improvement	Module	Related rule	Related use case	Action
1	Uses Indonesian in the ICD-9 and ICD-10 field	Medical staff	Reduce short-term memory load	Refer the patient	Changing the language
2	Displays the number of empty rooms	Medical staff	Reduce short-term memory load	Refer the patient	Update the interface to view the empty rooms' information
3	Indicates that the data are already registered in the system	Medical staff	Offer error prevention and simple error handling	Refer the patient	Adding information when the user has already completed the form
4	Changes to the sequence of the field for emergency referrals	Medical staff	Reduce short-term memory load	Refer the patient	Updating the order and position of the field on the patient referral form
5	Displays the rooms' amenities when making emergency referrals	Medical staff	Reduce short-term memory load	Refer the patient	Changing the placement and appearance of the rooms' amenities

Figure 5

Design improvements to the referral form in the second iteration. (Step 3, Step 4, Step 5).

Demonstration, evaluation, and communication

The second iteration's evaluation was carried out offline at a meeting with medical staff members. The number of respondents in this evaluation was 31 (n = 31) (Table 6). In the second iteration, the respondents were asked to download the application via Google Playstore in their own smartphone, and complete the use case to refer a patient with the task shown in Figure 3.

Table 6

Respondents' profile for the medical staff module in the second iteration.

Variables		Total
Gender	Female	27 (87%)
	Male	4 (13%)
Age	17–23 years old	2 (6%)
	24–30 years old	12 (39%)
	31–40 years old	14 (45%)
	>40 years old	3 (10%)
Occupation	Nurse	11 (35%)
	Doctor	11 (35%)
	Other Medical Staff	9 (30%)

The second iteration was evaluated using the PSSUQ usability testing framework, which was chosen for its ability to measure the task related to error handling, which was implemented in this iteration. The values of the CA and the CR for each variable are shown in Table 7. The maximum value of each dimension was 7.0. The average value for all the dimensions was 4.6, meaning that, overall, the respondents were satisfied with the results of the design. The values of system usefulness, information quality, and interface quality can be seen in Table 8.

Table 7

Cronbach's alpha and composite reliability.

Variables	Cronbach's Alpha	Composite Reliability
System Usefulness	0.937	0.951
Information Quality	0.925	0.942
Interface Quality	0.844	0.898

Table 8

PSSUQ values.

Design Module	System Usefulness	Information Quality	Interface Quality	Average Score	Note
Medical Staff	4.7	4.5	4.8	4.6	Above the median (4) is considered good

The communication process that was carried out in the second iteration was not significantly different from that of the first. Communications phase was also carried out during demonstrations and the implementation of the design evaluations. In addition, the final design of this study was published in the Playstore so that it can be tried by users.

Discussion

This study aimed to design an m-health application for a health referral system in Indonesia that could meet user expectations for usability. The application was designed to cover the scope of the system and for use by medical staff. The research process was carried via two iterations with different measurements. Respondent in the first and second iteration are different. This is because the demonstration was carried out on hospital staffs who were actively working in the different health facility, making it difficult to meet the same respondents in both iterations. However, in the both iterations there was no significant difference between the distribution of gender and age. But, the occupation of respondents is quite different. While in first iteration the percentage of doctors and nurses is 50% and 48% respectively, the second iteration has doctors and nurses each by 35% and the rest are other hospital staff. However, this might be seen as opportunity to evaluate the application by other staffs at hospital, because eventually, not only doctor and nurse, other staff such as admins would also use the mobile application.

The first evaluation used the SUS to measure the level of usability and learnability of the design, while the second iteration used the PSSUQ to measure system usefulness, information quality, and interface quality. The purpose of the first iteration evaluation was to ensure that the initial design was in accordance with the objectives of usability and learnability, which help users to complete tasks effectively and efficiently and satisfy their perceived utilization level [34]. Thus, the first iteration focused on the overall design, which was expected to help users carry out the intended functionalities.

The implementation of usability and learnability in the first iteration was accomplished by using the eight golden rules during the design process. For example, the strive for consistency rule was implemented by designing the main menu at the bottom of the display but having it disappear if the user enters something else. Another rule was enabling frequent users to use shortcuts, which was implemented by allowing users to easily access other features on the home menu display. To implement the informative feedback rule, the user will be sent information on actions that he/she has carried out. For the dialogs to yield closure rule, the display will tell the user whether a task has been successfully completed. For example, if a user has completed the referral process, then the system will convey that the information has been created and saved. The offer error prevention and simple error handling rule was only implemented in the second iteration because the user could not enter data in the first iteration's design. The result form the first iteration using SUS shows that overall value for the usability were OK with some change suggestion to be made. There are five improvements regarding the usability of the system. The improvement can be categorized as information quality aspect (implementation of local language, displaying rooms information, success message) and interface quality aspect (the sequence of information) that would be evaluated in second iteration using PSSUQ.

The second iteration was developed using the Android platform, which did allow the user to enter data. The application was designed to prompt an error message and to indicate which field will require correction. For easy reversal of actions, the design includes a return button for each view. To implement the support of internal locus of control, a reference history was made available to the user. In the second iteration, the search field was updated to increase user awareness regarding their power to navigate the application. Finally, for the reduce short-term memory load rule, the first evaluation found that the design was lacking in this area; a field had been implemented in a foreign language (ICD-9 and ICD-10 field). This was therefore corrected in the second iteration.

Based on the results of the first evaluation, reducing short-term memory load is quite important, especially for busy medical personnel. Improvements were made based on the results of the first iteration's evaluation, including changes in the use of language in the ICD-9/ICD-10 field, which also improved information quality. Per [35], information quality is defined as when information that is provided by a system can be easily understood and can effectively assist in the completion of tasks. The information quality dimension underwent many improvements based on the results of the first iteration evaluation, including changing the order of the data that refer to emergencies, adding information on actions that have been given, changing the language in the ICD-9/ICD-10 field, and adding information on patient data automatically if it has been registered in the system previously.

The interface quality dimension is concerned with whether a system is capable of satisfying its users [35]. From the first iteration, this dimension was also improved. For example, the display was changed to include certain information, such as amenities in rooms, amenities in health facilities, and the number of empty rooms, so that it can be more easily seen by users. The three dimensions of the PSSUQ were important for achieving usability that met user expectations.

Based on the results of our calculations, the application's designs obtained SUS values at the level of “OK” [33], meaning that the design results were acceptable, but improvements could be made. The PSSUQ value in the second iteration evaluation mimicked the first iteration, where users found the application's design satisfactory, with the lowest value found in the information quality dimension.

Implications

The theoretical implications of this research are the contributions to the designing of an m-health application, especially when considering that there is little research in this area. This study also provides more insight into the use of the DSR methodology. In the previous research of [9], the procedure was only carried out in one iteration; therefore, their evaluation results focused only on the value of the usability testing framework (PSSUQ). In this research, we used the SUS to evaluate the first iteration design and the PSSUQ to evaluate the second iteration design.

From a practical standpoint, this research is expected to give practical implications to relevant policy-making institutions, such as the Ministry of Health and the Health Insurance Institution, regarding information quality, interface quality, and system quality when implementing m-health applications. It will also support the referral process in Indonesia, given the high level of mobility of medical staff. In addition, this application is expected to be a benchmark in the process of designing and further development of m-health applications for the referral system in Indonesia.

Conclusion

This research was conducted to design an m-health application for a health referral system in Indonesia. This study used the DSR methodology in the design process, which was carried out in two iterations. The evaluation process used the SUS usability testing framework for the first iteration and the PSSUQ for the second iteration. In the first iteration, there were 48 respondents. The results from the SUS value in the first iteration placed the design in the category of “OK,” meaning that the results of the design were good, but many improvements could be made. Therefore, in the second iteration, improvements were made to meet the suggestions from the first iteration. There were 31 respondents in the second iteration. Based on the results of the evaluation, the final design produced system usefulness, information quality, and interface quality that satisfied the respondents.

Medical staff have the expectation of usability of the system's design, which requires local language for the needed forms (ICD-9 and ICD-10) to improve their effectiveness. In addition to system usefulness, information quality and interface quality must be implemented in the application. The respondents required improvements and additions in the data sequence when making emergency referrals and improvements in the appearance of the nearest health facilities. These three dimensions must have a minimum value above the middle Likert value (4) to obtain good results, which would place the design in accordance with the needs of its users. In addition, the values of usability and learnability that were obtained from the SUS must be at least 51 to place the design in accordance with the expectations of its users.

Declarations

Author contribution statement

A. A. Pinem: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

A. Yeskafauzan: Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

P. W. Handayani: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data.

F. Azzahro: Performed the experiments; Analyzed and interpreted the data.

A. N. Hidayanto, D. Ayuningtyas: Conceived and designed the experiments; Analyzed and interpreted the data.

Funding statement

This work was supported by the Directorate General of Higher Education for the Program Dasar Penelitian Unggulan Perguruan Tinggi (PDUPT) 2019, grant No. NKB-1495/UN2.R3.1/HKP05.00/2019, as well as by the Directorate of Research and Community Engagement at the Universitas Indonesia.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.