Critical success factors for mobile health implementation in Indonesia.

PURPOSE: Mobile-based technology health services in Indonesia are experiencing significant growth. However, the implementation of m-health in Indonesia is at a stand-still because the numbers of users or medical personnel who use m-health applications are still low. Given this fact, this study aims to identify critical success factors (CSFs) in the implementation of a mobile health applications in Indonesia. This research covers the following four dimensions: system quality, information quality, service quality, and organizational.
METHOD: This study uses a quantitative research approach with the entropy method to analyze system quality, information quality, and service quality, and it uses a qualitative approach for the organizational dimension. A total of 127 respondents completed the questionnaire (quantitative approach) and interviews were performed with three personnel from the Social Security Agency for Health (BPJS-K) (qualitative approach).
RESULTS: The highest weights for each dimension were as follows: ease of access for the system quality dimension; adequate and relevant information for the information quality dimension; user service convenience for the service quality dimension; and top management support for the organizational dimension.
CONCLUSIONS: Based on the results of this study, the regulator, health facilities management, and mobile health provider should ensure a long-term commitment to support the implementation of mobile health applications. In order to reach a wider market, all mobile health applications should be as user friendly as possible for the patients.

Introduction

In Indonesia, the number of smartphone users is estimated to be more than 100 million people. Technological trends have led 92 million users or about 32% of the population to use mobile health applications [1]. This has encouraged the Social Security Agency for Health (BPJS-K), a government entity responsible for providing health care insurance to all Indonesian citizen, to develop a mobile health application. In 2017, BPJS-K launched a mobile application called Mobile National Health Insurance (Mobile JKN), to facilitate the National Health System-Indonesian Health Card (NHS-IHC) participants. Indonesia is an island country with more than 13,000 islands, which can make it difficult for people outside of urban areas to access services. Since Mobile JKN can be accessed from anywhere, the NHS-IHC service is now available to residents in all regions and islands throughout the country [1]. The President of BPJS-K articulated during the launch of the Mobile JKN application in Jakarta that the application is a digital transformation of the BPJS-K business model, which have traditionally taken the form of administrative activities performed at branch offices or health facilities [1].

As one attempt to improve health services, Mobile JKN is expected to be used frequently by the people. With Mobile JKN, the public can find all information related to participant data including their medical history, billing information, availability of health care facilities, such as beds, and so on. As a result, it is important to include critical success factors (CSFs) to guarantee the successful implementation and development of the Mobile JKN application. One example of using CSFs in mobile health applications is the location-based, mobile, cardiac emergency application called iHeart [6]. The iHeart system allows patients to send their blood pressure and heart rate information to hospitals and emergency care units. Location-based services can be used with iHeart to make the healthcare systems more efficient and accurate [6]. This type of service can improve the quality of information, system quality, and ease of use in health care systems [7]. Finally, iHeart reduces the probability that a patient will develop a critical condition during an emergency, which suggests that the utilization of mobile health applications could save more lives [6].

A few studies have examined CSFs in mobile health applications to date [11, 24]. In addition, e-health literacy is one of the factors that affects the use of new systems, including mobile applications [13]. Based on these thought processes, the authors aim to explore the factors that influence critical success in the implementation of mobile health applications using the Mobile JKN application as a case study. Mobile JKN was developed by BPJS-K, which is the only government entity that provides health care insurance to citizens of Indonesia as part of the national security program. Mobile JKN can be accessed by all participants of JKN and is connected to more than 20,000 health facilities providers throughout Indonesia. This was our primary reason for using only one application in our research.

The findings of this research should be considered when developing future mobile health applications. This study also provides an assessment of CSFs in the Mobile JKN application and provides information for health facilities and health regulators in Indonesia (i.e., BPJS-K and the Ministry of Health) about the role of CSFs in mobile health applications more generally. Furthermore, health facilities and health regulators can discover the obstacles, disadvantages, and advantages that exist in mobile health applications.

This rest paper is organized into six sections: Section 2 comprises the literature review and Section 3 describes the methodology. Section 4 describes the results of this study, while Section 5 presents the discussion and implications of the findings. Finally, Sections 6 offers conclusions and future opportunities for research.

Theory

Critical success factors (CSFs)

Assessments of success factors or CSFs are systematic attempts to identify the feasibility, value, and quality of technologies, policies, and programs that use different methods and strategies [6]. The purpose of this type of assessment is to measure specific system issues in order to determine the success of the system. In order to understand the success of an information system, it is important to understand the accuracy and value of the system [12].

The research done conducted [8] on CSFs during the implementation of information technology shows that approximately 45% of companies in the United States (U.S.) use the Information Technology Service Management (ITSM) framework when developing an application. Furthermore [9], has shown that approximately 87% of companies use the ITSM guidelines as a reference for the development of information technology. Although ITSM may provide a competitive advantage, these projects have high failure rates. Therefore, because of these low success rates, it is necessary to have a comprehensive understanding of the success factors in ITSM [10]. In another paper that discusses CSFs [11], explains that many challenges have to be overcome in order to achieve the desired results in mobile health development. The most common challenge to mobile health solutions is acceptance and adoption. Applications should be fully integrated into the clinical workflow and add value to patient care, while offering easy administration and facilitating communication between health services. These challenges are the driving force behind pushing for progress in mobile health solutions [11]. In another study [12], proposes models for measuring the success of information systems based on previous empirical and theoretical studies. In the modified DeLone and McLean model, quality consists of three components: system quality, information quality, and service quality. Each must be controlled separately because it affects usability and user satisfaction, which support the successful implementation of information systems [12].

Mobile health (m-health)

The term m-health was invented by Robert Istepanian to describe the use of mobile communication technology and networks that function for health [14]. Another definition of m-health, created at the Summit Foundation organized by the National Institute of Health (FNIH), is the delivery of health services through mobile communication devices. The most common m-health application uses mobile technology and communication tools to educate users about existing health services in mobile healthcare [15]. Most view m-health as a type of technology that supports the function and delivery of healthcare services, while others consider m-health to provide direct access to healthcare services. In addition, the World Health Organization describes m-health as a medical and public health practice supported by mobile devices, such as cell phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices [16].

M-health also involves the use of mobile devices for voicemail or short message service (SMS), as well as more complex functions and applications, such as general packet radio service (GPRS), third- and fourth-generation mobile telecommunications (3G and 4G), and the global positioning system (GPS) [16]. M-health aims to integrate technology in the health sector because it has great potential to promote better health communication, which can help users achieve healthier lifestyles, while improving decision-making by health professionals (and patients). Furthermore, m-health can increase the quality of health services by enhancing access to medical and health information services and facilitate communication in places where this type of service was previously unlikely [17]. Therefore, increasing the use of m-health can help reduce healthcare costs by improving efficiency in healthcare systems, promoting prevention through mobile devices, and reducing information gaps through the use of m-health technology.

Mobile health application implementation in Indonesia

The Mobile JKN application was launched on November 15, 2017 (Fig. 1) [1]. The application is a digital transformation of the BPJS-K business model, which originally took the form of administrative activities performed at branch offices or health facilities. These physical spaces were transformed into an application that could be used by participants anywhere, anytime, without the limitation of normal business hours (via self-service). Currently, there are more than one million users of the Mobile JKN application on the Android platform and more than 2,000 users on the iOS platform [1].

Fig. 1

Screen shot of the Mobile JKN application.

The Mobile JKN application can be downloaded from the Google Play Store or Apple Store. Once the application is downloaded, the participant registers via the menu in the Mobile JKN application. When registration is finished, the participant can use the application to take advantage of all the features available. Other features available in Mobile JKN include user profiles, billing services, and a complaints service for BPJS-K patients/participants.

BPJS-K has been working with more than 20,000 employees from the first-level health facility (FKTP), which consists of 9,841 primary healthcare centers (Puskesmas or Pusat Kesehatan Masyarakat), 4,586 general practice physicians, 5,495 first-level clinics, 13 D-type hospitals, and 1,160 individual dentist practices [1]. Moreover, to support the Mobile JKN application, BPJS-K has been partnering with 5,566 advanced-level health facilities (FKRTL), which include 2,227 hospitals and main clinics, 2,332 pharmacies, and 1,007 opticians [1].

The number of reports that relate to gaps in the utilization of the Mobile JKN application during its implementation can be found on the Online Aspiration and Complaints Service (LAPOR!) website. LAPOR! is an easy-to-access, integrated forum for social media-based aspirations and complaints that is supported by 81 ministries/agencies, five local governments, and 44 state-owned enterprises in Indonesia [2]. LAPOR! was developed by the Office of the Presidential Staff to increase public participation in the supervision of the government's programs and to improve government performance during the implementation and development of public services, such as BPJS-K [2]. The most common complaints about the Mobile JKN application are related to the following issues: (1) slow online registration process for BPJS-K claims; (2) real-time data on the number of invalid bills involving user data and existing payment history are unavailable in the application; and (3) unclear follow-up processes for complaints [2]. Kompasiana, a journalist blog portal, reported that long queues to use BPJS-K triggered delays in healthcare services and additional costs for the community [3]. CNN Indonesia, a television station and news site jointly owned by Trans Media and Turner International, reported complaints against BPJS-K that arose from user ignorance of the procedures and measures that must be taken when using the service [4]. Other assessments show that the lack of technical regulations related to real-time data and information have led to problems with the availability of intensive care units and inpatient rooms [5].

Methods

This research uses a combined qualitative and quantitative approach. Interviews were used for the qualitative dimension to gain an understanding of the success factors in the development of the Mobile JKN application. Interviews were conducted to obtain more complete and in-depth information related to the development of Mobile JKN, particularly regarding the success factors in the organizational dimension. Qualitative data collection was carried out via structured interviews with BPJS-K employees in the IT Strategy, Planning and Security department. The authors prepared the question instrument for the interviews ahead of time and added additional questions when the interviews took place. The interviews were conducted for approximately one hour with each respondent in the BPJS-K office. The authors recorded the interviews using assisted voice recording. An interview and survey permit were obtained from BPJS-K through letter number 201B/UN2.F12.D/PDP.01KegiatanPerkuliahan/2018.

With regards to the quantitative approach, questionnaires were distributed to obtain research data and determine the success factors for the system quality, information quality, and service quality dimensions in Mobile JKN from the user application side. This was an easy and efficient task because this information is easily accessible via the Internet. Before distributing the questionnaires, a legibility test was performed to ensure the feasibility of the questionnaire in terms of writing and the use of words. A legibility test also ensured that the respondent understood the content and intent of the questionnaire. The authors performed a legibility test involving nine respondents who used Mobile JKN and understood the concepts in the questionnaire and spoke the Indonesian language. Finally, it was confirmed that the respondents who could fill out the questionnaire also used Mobile JKN.

The next stage, quantitative data collection, was carried out by distributing questionnaires online and offline. The population and samples used in this research were Mobile JKN users in Indonesia (purposive sampling). To collect the data, the authors will use the Universitas Indonesia survey services, and the questionnaire link will be distributed to Mobile JKN users on various social media platforms, such as WhatsApp, LINE, Facebook, Twitter, Instagram, and several online forums. These media were selected because it was possible to disseminate the questionnaire links quickly and easily. The respondents who prefer to use an offline questionnaire will be given one.

The first section of the questionnaire contained questions relating to the respondents' demographics and Mobile JKN usage data. The second section contained statements that represented the CSFs to be analyzed. The measurement scale used in this research was assessed based on an ordinal scale whereby respondents classified the answers into different categories. The ordinal scale used in this questionnaire was the level of approval of a statement (also known as the Likert scale) from one (strongly disagree) to five (strongly agree). The data were processed using the entropy method, which is a decision-making method that can determine the weights of a group of data [18]. The criteria with the highest value variations received the highest value weightings and were considered to represent most of the variance in the data group [18].

This research consisted of 28 CSFs, which were used as criteria (Fig. 2). Each CSF had a total occurrence of one to four times in the previous seven studies. These criteria were regrouped into four dimensions: system quality, information quality, service quality, and the organizational dimension. The grouping of dimensions was based on matching the same criteria and was also used to identify success factors when developing mobile health applications. Table 1 presents the definition of each CSF (i.e., the study criteria), which were translated into questions in the questionnaire.

Fig. 2

Critical success factors of m-health implementation.

Table 1

Critical success factor definitions and statements in the questionnaires.

Dimension	Criteria (CSF) Code	Criteria (CSF)	CSF description	Questionnaire statement
System Quality (SYQ)	SYQ1	Ease of monitoring activities	Ease of monitoring data on health service activities in mobile health applications to improve user satisfaction and comfort through health-system monitoring [6, 24]	I find it easy to monitor activity related to BPJS-K services using Mobile JKN.
	SYQ2	Ease of learning the system	Describe whether the system is easy to learn by new users [6, 23]	I feel Mobile JKN is easy to learn.
	SYQ3	Ease of user interface	The existing user interface makes it easy for users [20, 24]	Mobile JKN has an interface/display that is easy to use.
	SYQ4	Adequate system support	System support is an important factor that includes software and hardware support, which affects the use of information systems in terms of providing health services [24]	I feel that the software and hardware support available for Mobile JKN are sufficient to access the application at any time.
	SYQ5	System response time	The response time provided by the system when the user uses the application [6, 23]	I feel Mobile JKN provides a quick response time for its users.
	SYQ6	Supports decision making	Data-processing capabilities will affect system usage and user satisfaction to support decision making [6, 24]	Mobile JKN is able to support me in making more decisions via mobile devices (e.g., deciding to get a health screening).
	SYQ7	Easy access anywhere	The ability to integrate data effectively in different places and ability to access information anywhere [20, 24]	Mobile JKN can be used easily anywhere.
	SYQ8	Fast and efficient communication support	The system's ability to support fast and efficient two-way communication for every communication system especially in the health field [6]	I feel Mobile JKN supports fast and efficient communications technology.
	SYQ9	Security protection and system privacy	The ability to guarantee system resources is not used or modified, interrupted, or disturbed by unauthorized persons [20]	I feel Mobile JKN provides privacy protection for user's data.
Information Quality (INQ)	INQ1	Easy access to information	Ease of access to information using mobile devices is particularly suited for task performance that requires direct access to information [6, 20, 23, 24]	I feel Mobile JKN provides easy access to information related to BPJS-K services (e.g., billing or health facilities information).
	INQ2	Information is real-time	The timelier information generated by healthcare services will help to reduce uncertainty, improve decision-making, and reduce the risk of information errors [22]	I feel the information available on Mobile JKN is always in real-time/updated.
	INQ3	Sufficient and relevant information	A technology capability that allows users to always get relevant information from their company's existing databases and organizations to improve technology usage [6, 20, 22, 24]	I feel the information on Mobile JKN matches my physical document (for example, the number on my BPJS-K card).
	INQ4	Ease of obtaining factual information	The focus is on the experience of healthcare workers whether they can search for data and establish a patient picture based on available information [23]	I find it easy to get accurate information through Mobile JKN.
	INQ5	Ease of reading information	Observations show that user satisfaction with information quality has been enhanced by Information Technology (IT) departments by providing structured and easy-to-read information [23]	I feel the information available on Mobile JKN is easy to read and understand.
	INQ6	Accuracy of information	Accuracy as a significant dimension of information quality in which the accuracy, completeness, and conformity of information is an important factor in the health field [6]	I feel the accuracy of healthcare information available on Mobile JKN is important.
	INQ7	Usefulness of information	The information listed can be used in accordance with the needs of health workers covering almost all the functions required by health care professionals [22, 23]	Mobile JKN gives me the information I need.
	INQ8	Information is always updated	Observations show that user satisfaction is enhanced by the IT department's collaboration with system providers in terms of providing quick updates [23]	I feel the information available on Mobile JKN is always up to date.
Service Quality (SEQ)	SEQ1	User guide and help function	The user manual contains written guidelines or images related to the use of the application [23]	I am satisfied with the user manual and help functions available on Mobile JKN.
	SEQ2	Reliability in terms of completing the service	Service reliability provides satisfaction and availability of services to users who can assess the quality of their health services [6, 23, 24]	I feel Mobile JKN is able to finish every task related to an available service (e.g., when registering and making payments).
	SEQ3	Supports continuous improvement	Linear and continuous improvement is made on existing services [20]	I feel Mobile JKN offers continuous improvement of the services provided.
	SEQ4	Responsive system	Successful implementation and improvement of perceived user service systems, including higher system responsiveness that can reduce errors in system utilization [22]	Mobile JKN helps me because it is more responsive to my needs as they relate to the services provided.
	SEQ5	Trusted system for service	Trust requires certainty, security, comfort, and competence to gain the trust of healthcare users [6]	I believe Mobile JKN can monitor, help, and facilitate my needs.
	SEQ6	Ease of user service	Systems can create different opportunities to help organizations reduce costs and improve service levels for users [22]	Mobile JKN can facilitate the provision of services provided by BPJS-K.
	SEQ7	Consistent service availability	Consistent service for users and satisfaction with healthcare services to complete services without problems or damage [24]	Mobile JKN consistently provides services without system problems or damage.
Organizational (ORG)	ORG1	Ability to adjust the information system	The adjustment capabilities of the information system address the unique needs of the medical departments and health partners and is able to provide appropriate applications for its users [20, 26]	Mobile JKN has the ability to adjust the information system to address the unique needs of medical departments and health partners, as well as provide appropriate applications for its users.
	ORG2	Business process adjustment capability	Organizational capability requires cooperation with existing health facilities to adapt business processes in the development of a health system [10, 26]	Mobile JKN has organizational capabilities that require collaboration with existing health facilities in order to adapt business processes for the development of a health system.
	ORG3	Top management support	Top management support is a prominent success factor for technical and non-technical projects in the development of a health system [10, 20, 26]	Top management support is an important success factor for technical and non-technical projects for the development of a health system.
	ORG4	IT staff ability	For system development projects, organizations must have dedicated team members who have problem solving skills, relevant experience, and good communication [10, 20, 26]	For system development projects, organizations must have dedicated team members who have problem solving skills, relevant experience, and good communication.

Results

Respondent demographics

Data collection lasted seven weeks (from March 6 to April 20, 2018). Based on the information obtained from the Information Management and Documentation Officer of BPJS-K, data were collected from 127 respondents who represented Mobile JKN users and were equivalent to the NHS participants who used Mobile JKN. According to [28], 127 respondents exceeds the minimum threshold of 30 samples that must be processed in any given study. Table 2 provides a demographic summary of the respondents based on their domicile, gender, age, education level, and current job. Table 2 also shows how often Mobile JKN services were used within the last six months, as well as the frequency of health referrals in the last six months. It also shows frequently used features in Mobile JKN, referral health facilities, and the use of funds or personal insurance.

Table 2

Summary of respondent demographics.

Demographic variables		Number of respondents (percentage)
Domicile	Jabodetabek	103 (81%)
	Non-Jabodetabek in Java Island	18 (14%)
	Non-Java Island	6 (5%)
Gender	Male	60 (47%)
	Female	67 (53%)
Age	<20 years	16 (13%)
	20–30 years	85 (67%)
	31–40 years	14 (11%)
	>40 years	12 (9%)
Last education level	SD/SMP/SMA equivalent	77 (61%)
	Diploma	12 (9%)
	S1/Bachelor's degree	36 (28%)
	S2/Master's degree	2 (2%)
	S3/Doctorate	0 (0%)
Current job	Student	62 (49%)
	PNS/TNI/POLRI	16 (12%)
	Employee	25 (20%)
	Entrepreneur	11 (9%)
	Other	13 (10%)
Frequency of use of Mobile JKN services in the last six months	1–5 times	108 (85%)
	6–10 times	12 (9%)
	>10 times	7 (6%)
Frequency of health referrals in the last six months	1–5 times	120 (94%)
	6–10 times	5 (4%)
	>10 times	2 (2%)
Features that are often used on Mobile JKN	JKN Information	78 (61,42%)
	Location	61 (48,03%)
	Participant	59 (46,46%)
	Payment Dues	56 (44,09%)
	VA Check (Virtual Account)	23 (18,11%)
	Service	33 (25,98%)
	Health history screening	18 (14,17%)
	Complaint	20 (15,75%)
	Others	2 (1,57%)
Referral health facilities	Primary healthcare center	80 (63%)
	Hospital	47 (37%)
Use of funds or personal insurance in the context of referrals	Ever	41 (32%)
	Never	86 (68%)

With regard to problems encountered when using Mobile JKN, data processing was carried out with qualitative methods using the respondents' keywords. The most common mentioned keywords comprise: system malfunction, the user does not know the functionality of some features or services provided, technical issues such as system response time, non-updated information, and app display. A number of quotations are shown below for each of the problems encountered when using Mobile JKN:

“The application is still slow/often times out of the application and often an error”

“Still don't know the functionality of the application,” “Need to know more about usage in mobile app,” “At the beginning, I as a user [was] a little confused about how to use this application”

“Long time for app registration process,” “Sometimes took a long time to access,” “Long time to use feature”

“Info not updated,” “Do not feel it updates its info”

“Confused, not yet accustomed to using this application”. “User interface less familiar”

A summary of the problems encountered when using Mobile JKN can be seen in Table 3. 49 respondents said that there is no issue in using Mobile JKN. Table 3 shows that Mobile JKN needs to update its application so that it can be accessed anytime by the user. Information regarding health services must also be regularly updated by BPJS-K and health facilities connected to the JKN mobile application to ensure the users provided with updated information. Moreover, friendly use user interface is an important element that need to be embedded for future development of JKN mobile to enhance ease of experience for the users.

Table 3

Problems encountered while using Mobile JKN.

Problems	Number of Respondents	Percentage
Failure of system function	22	28.21%
Not knowing the functions and services	21	26.92%
System response time	19	24.36%
Information not updated	12	15.38%
User interface less familiar	4	5.13%
Total	78	100%

Data processing was also carried out using keywords in order to determine the desired expectations of Mobile JKN development. The most commonly mentioned keywords were: development features, application socialization, ease of use, updated information, display improvements, and fast access. The following are quotations representing the desired expectations of Mobile JKN:

“Always take care that the application will always be improved in accordance with the needs of society in the future,” “Features are maximized,” “Hopefully better and more interesting features can be added”

“There should be more information about the application,” “Give more information about application usability,” “Hopefully, Mobile JKN app can be used more evenly across the people of Indonesia”

“Easier to use,” “Hopefully easier to use,” “System in the hospital should not be difficult to access using JKN”

“Update the latest info from BPJS because with Mobile JKN, BPJS can be accessed anywhere and anytime,” “Always update the latest info,” “The more the participants know about the development of BPJS, the better”

“User interface is displayed to make it more interesting and easier to use,” “Look more beautiful again,” “Too stiff”

“Faster access,” “Give faster service,” “Faster service”

Table 4 presents the desired expectations for the development of Mobile JKN. Fourteen respondents stated that Mobile JKN can meet users' needs, while 113 respondents said they expect Mobile JKN to be upgraded, have more complete features, become more user friendly, and improve the user interface. All development that has been carried out in the JKN mobile application must be communicated by BPJS-K to the users, so they are aware and understand the improvements that have been made to the application. Finally, due to the limited capacity of the Internet connection in Indonesia, the JKN mobile user interface must be simplified (e.g., not displaying too much text or images on one page) and the users must be able to access the application quickly.

Table 4

Desired expectations of Mobile JKN development.

Desired expectations	Number of occurrences	Percentage
Development features	49	38.58%
Application socialization	17	13.39%
Ease of use	16	12.60%
Update information	16	12.60%
User interface improvements	9	7.09%
Fast access	6	4.72%
Total	113	100%

Calculation using entropy for system, information, and service quality dimensions

Before processing the data using Entropy, we validated the questionnaire by calculating the Cronbach Alpha (CA) and Composite Reliability (CR) values to measure the validity and reliability of the dimensions used in the questionnaire. Based on [27], a variable should have a CA and CR value >0.7. According to Table 5, all of the variables have CA and CR values >0.8, which indicates a high level of internal consistency for our scale.

Table 5

Result of Cronbach Alpha (CA) and Composite Reliability (CR).

Dimensions	CA	CR
INQ	0.858	0.890
SEQ	0.863	0.895
SYQ	0.863	0.892

Analysis of the CSFs in the development of Mobile JKN was carried out by weighting each criterion used in this research. Weighting was carried out on 24 criteria that fall within three dimensions: system, information, and service quality. The weighting was done using the entropy method and the weight difference obtained shows that criteria with higher weightings is a factor in successful development. The weighting consisted of three steps [19]:

Normalize the results of the questionnaire

The answers obtained from the questionnaire provided a number on a range of 1–5 for each criterion. The higher the value, the more successful the factor in the development of the mobile health application, according to the user. The normalization value for each criterion was calculated by combining all of the values with the highest value of the scale used. The results of the data normalization stage can be seen in Appendix 1.

The value obtained in Step 1 was divided by the total value of all criteria, using the formula:

For m > 1, i = 1, ..., n; j = 1, ..., m, where n is the number of decision makers and m is the number of criteria. The results of this stage can be seen in Appendix 2.

Calculate the value of entropy, dispersion, and weight of each criterion based on the results of Eq. 1, using the formula:

The dispersion of each criterion is calculated by the formula:

The weight of each criterion is calculated by the formula:

Appendix 3 presents the results of the weighting performed using Eqs. (2), (3) and (4). Table 6 presents the ranking of the dimension criteria based on the weighting in a scoring order from the highest to the lowest value. The different criteria weighting values in Table 6 also indicate that there are differences in the respondents' assessments for each criteria. In addition, according to Table 6, there was no significant difference between the weights of the factors identified (all of the results were within the range of 0.04); therefore, it can be summarized that all of the factors were proposed by the respondents. Fig. 3 describes the stages of running CSFs for mobile health implementation by ranking and classifying the weight values for each CSF.

Table 6

Criteria weight score.

Rank	Dimension criteria	Criteria's code	Criteria	Weight
1	System quality	SYQ7	Easy access anywhere	0.04256039
2	Information quality	INQ3	Sufficient and relevant information	0.04219069
3	Service quality	SEQ6	Ease of user service	0.04216464
4	Information quality	INQ6	Accuracy of information	0.04215937
5	Information quality	INQ5	Ease of reading information	0.04215792
6	System quality	SYQ1	Ease of monitoring activities	0.04212578
7	Information quality	INQ1	Easy access to information	0.04211588
8	System quality	SYQ8	Fast and efficient communication support	0.04210198
9	System quality	SYQ3	Ease of user interface	0.04200421
10	System quality	SYQ4	Adequate system support	0.04195183
11	System quality	SYQ2	Ease of learning the system	0.04188499
12	Service quality	SEQ5	Trusted system for service	0.04180447
13	Information quality	INQ4	Ease of obtaining factual information	0.041746
14	Information quality	INQ7	Usefulness of information	0.04172729
15	System quality	SYQ9	Security protection and system privacy	0.04170559
16	System quality	SYQ5	System response time	0.04126339
17	Service quality	SEQ4	Responsive system	0.04117002
18	Service quality	SEQ2	Reliability in completing the service	0.04109873
19	Service quality	SEQ7	Consistent service availability	0.04109075
20	Service quality	SEQ1	User guide and help function	0.04106886
21	System quality	SYQ6	Supports decision making	0.04102111
22	Service quality	SEQ3	Supports continuous improvement	0.04098643
23	Information quality	INQ2	Information is real-time	0.04098116
24	Information quality	INQ8	Information is always updated	0.04091851

Fig. 3

Roadmap in conducting the CSFs of m-health implementation.

The entropy weighting that resulted from the responses of the 127 respondents indicates that the key success factor in Mobile JKN development is easy access, which has the highest weight. This shows that Mobile JKN is capable of effectively integrating data in different places and can be accessed anywhere. However, based on Table 3, Mobile JKN still has problems related to system failure; thus, the application should increase reliability and scalability, so the application can be accessed quickly by the user. Regarding information quality, the users felt that the most important criterion was adequate and relevant information so that the information contained in the Mobile JKN would match their physical health documents (i.e., the user's profile, medical resume, etc.). This ensures that users always have relevant information from the BPJS-K database, as well as access to the health facilities involved in improving the technology. This result is consistent with Table 4 where the users expect sufficient and updated relevant information when using the application. For the service quality dimension, user convenience was selected as the most important criterion where the user believes Mobile JKN can facilitate service delivery by BPJS-K.

For system quality, the criterion with the lowest weight was decision support. This is because the data processing capabilities in Mobile JKN offer less support for and influence on system use and user satisfaction with decision making. Based on Table 4, many users still expect the applications to fulfill administrative functions in health services (e.g., for recording complete and integrated health data from diagnosis to storing image data) in addition to analytical features. For the information quality dimension, the criterion with the lowest weight is the fact that information was not updated continuously. According to Table 3, Mobile JKN provides information but is not updated or is not updated for long periods of time so that users feel the information provided is lacking. Finally, for the service quality dimension, the criterion with the lowest weight was support for continuous improvement where the user feels Mobile JKN has not improved on the services provided (Table 4).

Qualitative analysis for the organizational dimension

Two of the three interviewees voted for top management support as the first-order criterion in the organizational dimension where this was a prominent success factor for technical and non-technical projects in the development of a health system [20]. The sequence of criteria varies with each interviewee where Interviewee 1 gives a sequence related to the organizational dimension in accordance with the field of work involved in developing the Mobile JKN application. The next highest criteria were the ability of IT staff, the ability to adjust the information system, and the four business process adjustment capabilities. Although Interviewees 2 and 3 give a sequence for the 28 criteria of success factors, the sequence is based on the interviewees' experience during the development of Mobile JKN. For the sequence of the organizational dimension, the second interviewee selected business process adjustment capability in the first order on the organizational dimension, which supports top management support as the second success factor, followed by IT staff ability, and finally, the ability to adjust the information system. The third interviewee identified top management support as the first most important criterion, followed by the ability to adjust the information system, business process adjustment capabilities, and finally IT staff capabilities. Thus, sorting the above criteria varies according to the perspective of each interviewee. For the organizational dimension, one criterion is the most influential, namely top management support, which was chosen by two of the three interviewees.

Aside from the many obstacles and challenges associated with development, Mobile JKN can be considered successful because it can change the manual business process by utilizing technology. From the developer side, the implementation of Mobile JKN is capable of utilizing limited resources. The successful implementation of Mobile JKN in BPJS-K is inseparable from the success factors that affect it. Table 7 shows the data extracted from the three interviews that referenced the organizational dimension.

Table 7

Data extraction of interview results.

No.	Critical success factors	Is there any or not	Conditions at BPJS-K	Interviewees
				1	2	3
1	Ability to adjust the information system	✓	• Mobile JKN implementation aims to digitize BPJS-K services • Other objectives for planning in decision making • Mobile JKN connects participants with BPJS-K • The challenge is how Mobile JKN can be developed to facilitate NHS participants	✓	✓	✓
2	Business process adjustment capability	✓	• Mobile JKN helps business process needs in adjustment of the BPJS-K information system • Innovation of manual business processes becomes automated with Mobile JKN • Online queue for FKTP in collaboration with Mobile JKN • Unnecessary information and services sent to the branch office or center (self-service)	✓	✓	✓
3	Top management support	✓	• Commitment of the leader to support Mobile JKN • Involvement in Mobile JKN development	✓	✓	✓
4	IT staff ability	✓	• Mobile JKN in-house development of BPJS-K • General training to improve employee competence • The teams involved are organized based on planning, development, operation and business process optimization (BPO) • The challenge is with limited human resources in order to remain efficient and effective	✓	✓	✓
5	Another success factor	✓	Socialization	X	✓	X

Discussion

The results obtained using the entropy method show that the criterion with the highest weight in terms of critical success is easy access (weight 0.04256039) with system integration capabilities where systems are able to integrate data effectively in different places and can be accessed anywhere [20]. The fact that easy access has the highest weight shows that the quality of the system is most important compared to other criteria perceived by the user. This is in line with the transformation of BPJS-K services, which were originally carried out at the branch offices or health facilities, into Mobile JNK's digital services. These services can now be accessed by users anywhere, anytime without the limitation of normal business hours [1]. Access to health facilities with diverse geographic situations and conditions poses a significant challenge to the delivery of health services in Indonesia [21]. With the development of Mobile JKN, users can now access health services directly. The second highest ranking criterion when analyzing the CSFs in Mobile JKN development is adequate and relevant information (weight 0.042190686). This reflects the digital transformation of the BPJS-K's business model in terms of administrative activities [1], which allows users to have constant access to relevant information from existing databases in the Mobile JKN application [6]. The Minister of Health Regulation Number 1144/MENKES/PER/VII/2010 on the Organization and Working Arrangement of the Ministry of Health mandated that data centers and information be the executors of the duties of the Ministry of Health in the field of data and health information. The purpose of the Health Information System is to be able to transform the available data through routine and non-routine recordings [3]. An example in the Mobile JKN application is the exchange of appropriate information between the application and the physical documents that belong to the JKN participants.

Ease of user service (weight 0.04216464) is the third criterion through which the Mobile JKN application can support organizations in reducing costs and improving service [22]. Supported by Mobile JKN, the NHS-IHC program provides greater access to the public for ease of use and guaranteed access to health services [1]. The audited BPJS-K report for 2016 shows that there are 177.8 million visits to health facilities and that this number has increased by as much as 92.3 million since 2014, and by as much as 146.7 million since 2015. This confirms the importance of ease of use in healthcare services [1]. The fourth and fifth criteria are part of the quality of information dimension, which includes information accuracy (weight 0.04215937) and ease of reading information (weight 0.042157922) and are important for Mobile JKN application users. This is in line with accuracy, which is a quality-of-information dimension that is significant for accuracy, completeness, and conformity of information and is an important factor in the field of health [6]. According to research conducted by [23], user satisfaction with information quality is enhanced by the provision of structured and user-friendly information. Mobile JKN is able to provide accurate information, such as information related to user billing and the location of health facilities, which enhances accessibility because the information is easy to read.

Ease of monitoring activity (weight 0.042125781) is the next criterion. It is an effective factor for system quality that is defined as user satisfaction, comfort, and fun in health system monitoring [6]. The menu services available on Mobile JKN have features that make it possible to monitor users' activities. One example is the “Service History” feature where participants can see their records and health services history that have been received by NHS-IHC participants either in FKTP or FKRTL. Another feature is health screening, which is as important as other features in the Mobile JKN application. Health screening aims to detect symptoms of chronic diseases, such as diabetes mellitus, hypertension, chronic kidney disease, and coronary heart disease. To discover potential health risks, participants must first answer 47 questions in order to monitor their activities over the past year. The seventh highest ranking criterion is ease of access to information (weight 0.042115884). Using mobile devices is appropriate for tasks that require direct access to information [20], as in the Mobile JKN application, which provides access to information-related services provided by BPJS-K through its “Info” features. These features explain how to upgrade the application, as well as the rights and obligations of the participants.

Rankings 8 through 11 are part of the system quality dimension that consists of fast and efficient communication support (weight 0.042101984), user interface (weight 0.042004207), system support (weight 0.041951827), and ease of learning (weight 0.04188499). The system's ability to support fast and efficient two-way communication for every communication system, especially in the healthcare field [6], is implemented in the Mobile JKN application, which supports communication technology through a complaints service that can be connected to the BPJS-K Care Center. In addition, the existing user interface makes it easy for those who use mobile devices to monitor their health and gain satisfaction from the service usage [20]. The potential of Indonesia's digital economy and the commitment of the Ministry of Communications and Information Technology to the construction of networks and infrastructure has enabled BPJS-K to be optimistic about the utilization of the Mobile JKN application by the entire Indonesian population. This will make it easier for users to access NHS-IHC services because they will be available to residents in all parts of Indonesia. The interface must be understood by all users in order to support the implementation of Mobile JKN [1]. In terms of adequate system support, Mobile JKN is accessible at any time and has adequate software and hardware support for the services provided [24] because it can be downloaded in both Android and iOS versions. In terms of the ease of learning, the system shows whether or not Mobile JKN is easy for new users to learn, and it appears the Mobile JKN application is very easy to learn. Users need only download the app from the Google Play Store or Apple Store. Once the application is installed, the user can register via the menu available in the Mobile JKN application. Once registration is complete, the participants can enter the application and take advantage of all the features available [1].

Trust includes certainty, security, comfort, and service quality for all healthcare users [6]. The fact that quality of service ranks twelfth supports the success of Mobile JKN and shows that it is important to become a trusted system that can provide services (weight 0.041804472). Trust is an important criterion for healthcare users because if they do not believe in the application, it will become useless [6]. In the 13th and 14th rank, we find ease of obtaining factual information (weight 0.041745998) and the usefulness of information (weight 0.041727293), respectively. The Mobile JKN application focuses on the user experience by allowing users to log in and search for data, and set the picture based on the available information. The information listed can be used as needed. In the 15th rank, we find security protection and system privacy (weight 0.041705586). There are thousands of health-themed devices and online applications on the market, but there is rarely a guarantee that the users' data will be kept confidential [25]. is skeptical of new healthcare applications and devices with existing privacy security measures. This is why this criterion is ranked 15th. Mobile JKN can ensure that system resources are not used, modified, interrupted, or disturbed by unauthorized persons [20].

The 16th ranking is the system response time criterion (weight 0.041263393). In this case, users feel the response time provided by the Mobile JKN interface is still too slow when using the application service. The 17th to 20th ranks comprise the quality of service dimensions, including the responsiveness of the system (weight 0.041170019). Reliability (weight 0.041098734), consistent service availability (weight 0.041090748), and the user guide and help function (weight 0.041068863) complete the service dimensions. Users felt that Mobile JKN is not a very responsive system. In addition, system service improvements that include system responsiveness to reduce errors are also lacking. Providing satisfaction and availability of services to Mobile JKN users was also perceived as less successful by users who determined the quality of service assessment. This is because some of the processes in Mobile JKN still do not work properly, such as searching for inappropriate health facility locations, whereas consistent service availability is aimed towards healthcare services. The users did not give a high ranking to the completion of service activities without any problems or damage or to the user guide and help function criteria. There is no information about user manuals containing written guidelines or images related to application usage [24]. The absence of a Mobile JKN manual requires the user to dabble directly with the application before knowing the functionality of each feature on the Mobile JKN menu.

The lowest ranking criteria are decision support (weight 0.04102111), support for continuous improvement (weight 0.040986434), real-time information (weight 0.040981163), and information that is always updated (weight 0.040918507). Data processing capabilities affect system usage and user satisfaction in terms of stronger decision making [6]. Mobile JKN does not support decision making when the user is looking for a new health facility in the application. Furthermore, continuous improvement has not been implemented, as can be seen from unavailable services, such the lack of payment options. The two quality information criteria are ranked 23rd and 24th because the timelier information generated by Mobile JKN services will help reduce uncertainty, improve decision making, and reduce the risk of misinformation [22]. Finally, user satisfaction with the quality of information is influenced by the cooperation of service providers in terms of being able to update information quickly [23]. This is still lacking in Mobile JKN because the information available on the application tends not to be updated for long periods of time.

Based on the dimensions used in this research, the quantitative data criterion with the highest weight in the system quality dimension is ease of access (weight 0.04256039). For the information quality dimension, the criterion with the highest weight is sufficient and relevant information (weight 0.04219069). Finally, for the service quality dimension, the highest criterion is ease of user service (weight 0.04216464). In addition to the criteria with the highest weight, each dimension also has a lowest weighting criterion. For the system quality dimension, the criterion with the lowest weight is decision support (weight 0.04102111). For the service quality dimension, the criterion with the lowest weight is always updated information (weight 0.04091851). For the quality of service dimension, the criterion with the lowest weight is support for continuous improvement (weight 0.04098643). In the organizational dimension, the results of the qualitative data analysis show a difference of perspective for the most influential factor therefore, the sequencing of the existing criteria varies. However, the criteria with the highest weight in the organizational dimension is top management support, which was chosen by two of the three interviewees.

This research contributes to the field by demonstrating that the success factors in the development of mobile health applications consist of four dimensions, and each dimension consists of several criteria. This research also uses to entropy method to gather quantitative and qualitative data through questionnaires and interviews, respectively.

A practical implication of this research is that organizations that are supporting mobile health application providers can evaluate system, information, and service, as well as organizational dimensions using the criteria derived by this research. This will help the developers of mobile health applications and the organizations that use them to identify the advantages and disadvantages associated with the development of mobile health applications. Based on the results of the questionnaire, the five lowest criteria in the three dimensions of quality include user guide and help functions, support for decision making, support for continuous improvement, information in real-time, and information that is always updated. Data processing capabilities can also be improved by supporting decision-making options on Mobile JKN, such as health screenings, and by utilizing information already owned by the health facilities connected with BPJS-K. As for improvements, it is expected that organizations with mobile health applications should experience continuous improvement of the services provided. Finally, when the information on the mobile health applications is updated, it should be accessible in real-time.

Qualitative data obtained from open-ended questions and interviews with stakeholders also emphasize that system improvement is needed periodically. Information that helps users should continue to be given and updated. Although the interviews provided qualitative data, the results show that the Mobile JKN implementation of BPJS-K is continually developing its features to support the NHS program. It is expected that through this research, BPJS-K can increase the number of human resources to meet the needs of its IT division and continue to provide training to support their IT capacities. The authors have also found other features that could factor into the success of mobile health applications. These include services and features related to encouraging users to socialize and features that help users understand and use the application.

Conclusions

This research has identified CSFs in the development of mobile health applications in Indonesia. Success factors are arranged in order of highest to lowest weight: (1) easy access; (2) sufficient and relevant information; (3) ease of user service; (4) accuracy of information; (5) ease of reading; (6) ease of monitoring activities; (7) ease of access to information; (8) fast and efficient communication support; (9) ease of user interface; (10) adequate system support; (11) ease of learning the system; (12) trusted system for service; (13) ease of obtaining factual information; (14) usefulness of information; (15) security protection and system privacy; (16) system response time; (17) responsive system; (18) reliability to complete the service; (19) consistent service availability; (20) user guide and help functions; (21) support for decision making; (22) support for continuous improvement; (23) information in real-time; and (24) information always updated. In addition, top management support is essential to the successful development of mobile health applications.

The authors suggest that subsequent research should use different case studies in the development of mobile health applications in Indonesia in order to identify other success factors in the development of mobile health applications, as well as other criteria affecting system, information, and service, as well as organizational dimensions. Other work could map the CSF within the demographic variables to understand the factors that might influence certain groups of people. The expectations for BPJS-K on the Mobile JKN application include the development of features, socialization on the application, ease of use, the existence of information updates, improved display, and fast access. The optimal application would also shorten the queue of referrals and obtain information related to referrals and hospital options in order to make costs more affordable when making referrals.

Declarations

Author contribution statement

Putu W. Handayani, Dira A. Meigasari: Conceived and designed the experiments, Performed the experiments, Analyzed and interpreted the data, Contributed reagents, materials, analysis tools or data, Wrote the paper.

Ave A. Pinem: Conceived and designed the experiments, Performed the experiments, Analyzed and interpreted the data, Contributed reagents, materials, analysis tools or data.

Achmad N. Hidayanto, Dumilah Ayuningtyas: Analyzed and interpreted the data, Contributed reagents, materials, analysis tools or data.

Funding statement

This work was supported by the Directorate General of Higher Education for the Program Dasar Penelitian Unggulan Perguruan Tinggi (286/UN2.R3.1/HKP05.00/2018) and the Directorate of Research and Community Engagement, University of Indonesia.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.