Health workers' experience of a digital health intervention implemented in peri-urban communities in Karachi, Pakistan.

Objective: Digital health interventions (DHIs) have the potential to improve access and quality of care in low-middle-income countries. The aim of this study was to assess the acceptability, usability and aesthetics of a DHI by frontline workers in peri-urban community settings in Karachi, Pakistan.
Methods: A mixed-methods study was carried out in peri-urban field sites in Karachi, Pakistan, where maternal and childcare services are provided through front-line care providers using a DHI. These workers include community health workers, midwives, and physicians who were using the DHI for at least six months. For quantitative data, a questionnaire regarding the module design and interface, technical difficulty, and appropriate utilisation was assessed using a 5-point Likert scale. For qualitative data, focus group discussions (FGDs) based on experiences regarding operability, design, its effect on work efficiency and the provision of beneficial health services were conducted.
Results: There were 93 respondents for the quantitative questionnaire who reported high satisfaction (>85%) with the DHI in many themes including content quality, aesthetics and ease of use. Participants were least satisfied with service quality (45% satisfaction only) due to issues related to data sync and network connections in these areas. During the FGDs, the workers stated that the DHI helped them with accessing previous data and providing quality health care services to the community.
Conclusion: Although frontline workers reported a few technical difficulties while using the DHI, the majority reported that it was acceptable, had user-friendly features and was beneficial in their work processes.

Introduction

Gaps in health service delivery and lack of implementation of the policy have led to deteriorating maternal, newborn and child health (MNCH) indicators in low-middle income countries (LMICs). On the service delivery side, this problem is primarily driven by low MNCH service utilisation brought upon by contextual factors from both the patient and health system perspectives. Some of these factors include lack of infrastructure of health facilities, poor quality health records as well as inadequate coverage necessary for quality MNCH service delivery thus worsening MNCH outcomes. Digital health interventions (DHIs) may be one possible solution to reduce the disparities in geographical inaccessibility and inequity in LMICs. A DHI is defined by World Health Organisation (WHO) as “a discrete functionality of digital technology that is applied to achieve health objectives and is implemented within digital health applications and ICT systems, including communication channels such as text messages.” Digital technologies offer the potential to enhance the coverage and quality of health practices and services by improving effectiveness, efficiency, accessibility, safety and personalisation of care. Although DHI may not often find itself within communities that lack technological infrastructure, its potential of these has been consistently reinforced by global health institutions such as the WHO.

While still in its infancy, digital health has a promising future in Pakistan. The industry is steadily growing with the population's needs and limited health infrastructure representing essential catalysts for change. “Teeko,” a routine immunisation DHI, was rolled out in Pakistan to assist with the monitoring and evaluation of vaccination services as well as sending reminders to parents for routine immunisation visits. In a short span of 15 months, the implementation of this DHI led to a four-fold increase in vaccination coverage in the catchment area. This demonstrates the potential of digital health approaches to addressing child and maternal mortality in the country.

Building on this digital appetite locally, the study team developed a DHI for the peri-urban coastal sites in Karachi, Pakistan. These sites have high neonatal and child mortality as compared to national rates. Poor access to quality care is cited as one of the important factors for the poor maternal and child indicators reported from these sites. To improve the provision of maternal and newborn services in these sites, the study team developed a DHI to longitudinally capture MNCH encounters at evidence-based touch-points through the pregnancy and postnatal journey that enable front-line workers to provide high-quality services at the home and facility. The frontline workers including midwives, CHWs, and physicians collect all information using an android-based application installed on a tablet, which helps maintain an electronic health record.

A user-centric design is a critical ingredient for the successful integration and scale of a DHI and through understanding frontline worker perception, necessary and important changes can then be made. The DHI has been implemented around two years ago, therefore it is imperative for the study team to observe and evaluate its performance as a user-centric design. To comprehensively understand the end-user perspective, the aim of this study was to assess the acceptability, usability, service quality and aesthetics of this DHI by frontline workers in peri-urban settings.

Methods

This was a mixed-method study conducted from November 2021 to February 2022 at the peri-urban coastal field sites in Karachi, Pakistan, where this DHI is being implemented. These sites have quarterly MNCH household surveillance along with antenatal care, immunisation and physician services for children under the age of five years. This DHI consists of eight modules that allow health workers to register women and children, log events such as pregnancies, births, deaths, and keep a longitudinal record of the woman's antenatal visits (including laboratory investigations and ultrasound) and child's immunisation. All frontline workers who were DHI users for at least six months and had provided written informed consent were included in the study. These services are provided by frontline workers such as community health workers (CHWs), midwives, and physicians who have been using a DHI implemented since June 2020. The CHWs have at least 10 years of education and are mostly residents of the same communities where surveillance is taking place. They visit the households once every quarter to enquire regarding vital events such as pregnancies, births, maternal and child deaths, and migrations. The midwives provide ANC services at the primary health care (PHC) in the community. The physician services are available for the children of the community who are under the age of five years.

The Design and Evaluation of Digital Health Interventions (DEDHI) framework was adapted to assess the user experience in this study. This framework provides an overview of goals, evaluation criteria, technical maturity and implementation barriers to be considered during the four-phase (Preparation, Optimisation, Evaluation and Implementation) life cycle of DHIs. Given the recent establishment of the DHI, quantitative and qualitative assessments used in the study were created to incorporate characteristics present within the Preparation and Optimisation phases of this framework. The Preparation phase evaluated ease of use, adherence, and privacy and security as key themes while the Optimisation phase assessed content quality, perceived benefit, service quality, adherence and aesthetics of the DHI (refer to Table 1). The study team designed a questionnaire that consisted of questions which were used to assess these themes using a 5-point Likert scale. The survey was filled out as an online tool and was available in both English and Urdu (local language).

Table 1.

Definition of the themes for assessing user experience with the digital health intervention.

Theme	Definition
Content quality	Extent to which the DHI's content is accurate, relevant, timely, and encompasses all the appropriate needs of the patient.
Aesthetics	Extent to which the DHI is visually appealing and user-friendly, especially in terms of font sizes, colour, interface, and language.
Service quality	Technical difficulties encountered while using the DHI and the support available to resolve these.
Ease of use	Extent to which the DHI is easy to search and record information.
Privacy and security	Safety and protection of data in the DHI.
Perceived benefit	The impact of the DHI on the work processes.
Adherence	Extent to which the DHI is being utilised for the intention it was developed.

DHI: digital health intervention.

The qualitative data was collected using focus group discussions (FGDs) with a-priori themes described above. To assess the feedback of all different modules, each group comprised 8 participants (5 CHWs, 2 midwives, and 1 physician) who were selected using simple random sampling for the FGD. A total of 48 participants were included in the six FGDs. The sessions were conducted in Urdu by a trained research assistant who was independent of the DHI intervention team and was audio-recorded for transcription purposes.

The quantitative results were reported as frequencies and percentages where the options ‘agree’ and ‘strongly agree’ were represented as ‘agree’, ‘disagree’, and ‘strongly disagree’ were represented as ‘disagree’ while ‘neutral’ remained as it was. The FGDs were first transcribed by two investigators followed by a collaborative review to ensure accuracy and validity after each session. Since saturation was observed after the sixth session, the FGDs were stopped. The two investigators then analysed the transcriptions of all six sessions using MAXQDA, where a deductive coding method was used to go through each transcription and assign codes to a theme from the DEDHI framework. After a thorough discussion, the codes were inspected by the principal investigator to further ensure rigour. This study was approved by the Aga Khan University Ethical Review Committee (2021-6868-19847).

Results

Survey findings

There were 93 responses recorded for the questionnaire. There were 81 CHWs, nine midwives, and three physicians among the participants, 97% of whom were females with a mean age of 31.2 ± 7.0 years. The results of the survey have been reported according to the themes from the DEDHI framework mentioned in Table 1.

The participants were satisfied in terms of the content quality offered by the modules (refer to Figure 1). The majority of the participants (n = 87, 94%) agreed that the information provided in the module was relevant to their work while 88% (n = 82) believed that the information provided was accurate.

Figure 1.

Participants’ responses regarding aspects of content quality of the DHI.

When asked about the design interface, the majority of participants (n = 89, 96%) found the interface visually appealing in terms of font size, colour themes and icons used, and easy to follow in terms of the flow of information. Even though the DHI is in the English language, 96% (n = 89) participants believed that it was easy to understand (refer to Figure 2).

Figure 2.

Participants’ opinions regarding the aesthetics of the DHI.

Eighty-six percent (n = 80) participants reported that they felt that the DHI helped them provide beneficial MNCH services to the community. Majority of them (n = 88, 95%) reported that the DHI helped them access the relevant health information of the mother and/or child easily and hence decreased the time in providing these services (refer to Figure 3).

Figure 3.

Participants’ responses regarding the perceived benefit of the DHI.

The technical difficulty was reported by 73% (n = 68) of the study population. However, when asked about the availability of technical assistance, 77% (n = 72) agreed that it was readily available (Figure 4).

Figure 4.

Participants’ responses regarding the service quality of the DHI.

Ninety-one percent (n = 85) of the total study population found the features in the module easy to find while 88% (n = 82) used the module for its intended purpose. Those who disagreed did so because they felt the need to use paper-based forms at times when the modules proved insufficient to their needs, that is when the data was not syncing. Majority (n = 87, 94%) agreed that the patient information entered was secure within the platform.

Focus group discussions

The thematic analysis around the major themes has been presented below.

Ease of use

The frontline workers expressed satisfaction when asked about ease of searching and recording information in the DHI. The workers felt that storing data had become much easier as they were only required to carry a small tablet to acquire the information.

In terms of paperwork, we would need a lot of space to store these records. Whether the data is two or four years old, we can easily find it when we need it. If we need a print, then we can print too. It's made things a lot easier for us. We don’t have to wait much [for the information] and neither does the patient. (CHW, Rehri Goth)

Content quality

The frontline workers found the content in their respective modules in the DHI to be highly accurate and relevant to their work. Participants also reported that there were prompts in the DHI that would ensure that they fill all sections before moving on to the next one thus ensuring data capture.

Even if we forget to fill something and try to move forward, the form will not allow us to move ahead. It won’t move forward until we’ve clicked everything. This is good because sometimes we can forget, so it reminds us to fill it properly. (CHW, Rehri Goth)

While they are generally satisfied with the content quality aspect of the DHI, the participants emphasised the need for more features and options to be added to their respective modules including editing rights to correct names, dates, etc., and more auto-fill options.

Another problem is that when we enter the vaccines, we give multiple vaccines at the same time. OPV 1, ROTA 1, PENTA 1, and PCV 1 are all given at the same time, but we have to select and log dates for each of them separately. If all four are given on the same day, then there should be an option to auto-fill dates for the others when we enter it for one. It would save us time. (CHW, Ali Akbar Shah)

Aesthetics

The frontline workers all unanimously agreed that the modules of the DHI were visually appealing and easy to follow.

Our antenatal care modules are very easy [to understand]. We can easily teach an outsider to use the modules and they’ll understand it too … we just have to click on things. (Midwife, Rehri Goth)

In terms of colours and font size, everything is fine. Visually, everything is visible to us and is easily understandable. (CHW, Ibrahim Hyderi)

Even though the DHI was designed in English, the workers did not report that as a concern, in fact, they considered it a positive learning experience.

We should know this much [English]. Through these modules, we are able to learn more English, it's an improvement for us. (Physician, Bhains Colony)

Service quality

The participants raised concerns regarding technical issues such as data syncing or problems with the android device (e.g. charging the device adequately) which impacted their work efficiency.

We have issues with signals which lead to other problems. Yesterday, while we were working, our signals went down which stopped the data from uploading. It was not reaching the midwife, so the patient also had to suffer a bit [due to waiting]. (CHW, Rehri Goth)

Sometimes when we’re taking information from a family, the application suddenly stops working. If we have to enter any data, we would have to wait for it to work again, which takes time. It becomes difficult when the patient is in a hurry and says, ‘Could you please be a bit quicker?’. (CHW, Bhains Colony)

Although the workers faced many technical difficulties, they also received the relevant help needed to solve most of these issues.

Yes, we report these problems, and they get solved. The application has been improving because whenever we face issues, we report them, and we are provided with solutions. (Midwife, Rehri Goth)

Privacy and security

While elaborating on how helpful having a DHI is as compared to doing paperwork, the frontline workers mentioned how they felt the patients’ data was now in a more secure platform.

It gets difficult to carry around papers. In comparison, this application has been very helpful because papers can get misplaced, but the data can’t get misplaced in the tablet. (CHW, Ibrahim Hyderi Extension)

Perceived benefit

The DHI has helped the workers provide beneficial health services as it has increased accessibility to information, making their work easier and more efficient.

It has been very helpful. We can remind them of information they have given us before. It's also helpful because, for example, if we are talking to a pregnant woman, we can remind her of older complications she's had with past pregnancies that she should be wary of. (Midwife, Rehri Goth)

The participants also pointed out that due to the DHI, they have been able to establish a stronger patient–provider connection.

[While filling the application] we can maintain eye contact, advise them, comfort them, and brief them on other things like hygiene. If they ask a question, we can reply to them because we don’t have to pay attention [to filling the data] as much. (CHW, Bhains Colony)

The frontline workers shared experiences that the community acknowledges that the DHI helps keep a reliable record of the information and that they also have increased access to it now.

Sometimes if we have, for example, gone to a newborn's house, the mother will ask us what the baby's weight was last time and what it is right now as well. So, we open the older data and tell them and then they can compare and see if the baby's weight has improved. It makes the mothers happy. (CHW, Rehri Goth)

If their children are nearing the school-going age and they want to get their admission done, they contact us and ask, ‘When my child was born, you wrote down his date of birth, can you tell me what it is?’. We would immediately search it and give it to them, and they would compliment us saying ‘you people have made things better; we are able to access all the data from you now’. (CHW, Bhains Colony)

Discussion

The provision of DHIs allows for the transformation of existing health systems, improve access to care for rural and marginalised communities and address prevalent inequities. The results of the present study reported that health workers’ experience of the DHI implemented in peri-urban communities in Karachi, Pakistan has been favourable for the user as indicated by the themes of content and service quality, aesthetics, ease of use and perceived benefit to the communities they serve.

There are a few examples that currently exist in the literature that demonstrate the use of DHI in LMICs. In 2010, Burkina Faso introduced a DHI to improve the health of children across the country. Built on Dimagi's CommCare platform, the tablet-based application utilises the Integrated e-Diagnostic Approach (IeDA) with enhanced decision support for the Integrated Management of Childhood Illnesses (IMCI). The app has since scaled to more than 1700 rural PHC facilities, with 6300 health workers delivering more than 250,000 IMCI consultations per month. Evidence from independent evaluations indicated that in PHC facilities covered by IeDA, 97% of consultations for children under five were conducted with the app, leading to a 50% improvement in adherence to IMCI protocol.

As DHIs such as CommCare are rapidly gaining momentum, their success relies heavily upon the efficiency and effectiveness of targeted work processes along with a positive user experience. To integrate these DHIs into the health infrastructure, they need to be rigorously evaluated using common digital health frameworks. There are a few frameworks that have been developed to ensure systematic monitoring and evaluation of DHIs including the DEDHI and the WHO framework. Although the current study employed the DEDHI framework for evaluation of the DHI, the WHO framework has similar themes regarding usability, efficacy and feasibility at the initial stages of the DHI to check system functionality for further improvements.

The usability of the DHI focuses on the interaction between the user and technology. Schnall et al. reported that a key feature of a DHI should be a simple and easily understandable manner in which information is presented. The study participants felt that searching for patients and their information in the DHI was made considerably easier as compared to paper. Similar studies from low resource settings also highlight the positive impact of replacing paperwork with DHIs as they allow faster data collection and the removal of heavy registers, leading to increased efficiency while seeing patients. Lack of user-friendliness is stated as one of the most common reasons for negative perceptions regarding DHIs in front line workers and hence limit adoption of technology. The user interface of a DHI is hence an important feature that should be explored as part of the user experience. A recent systematic review reported that ease of navigation on the user screens, simple and consistent styles along with a bright colour scheme directly impact the user's experience and engagement with the DHI. The current study demonstrated that end users have high satisfaction with the aesthetics and ease of use of the DHI resulting in improved overall user engagement.

Content quality of DHIs is an essential item to assess as a lack of accurate data that can potentially lead to ineffective use of the application and has an undesirable health impact on the populations being served. However, the DHI users were overall satisfied with the content quality as it encompassed all the relevant information pertaining to their respective tasks for MNCH service delivery. The participants felt that the DHI made health information more accessible in a safe manner, thus building patient trust as well. Data privacy and confidentiality are of utmost importance when scaling DHIs. The study participants mentioned the need for additional options in the DHI, including editing rights and comment boxes. Granting these rights can potentially result in the unregulated tampering of data hence decreasing the reliability of the patients’ records. Similar to other studies, the availability of free text boxes has been restricted to avoid the possible inclusion of unrelated content in the system, which may affect the performance of the hardware and data quality. The DHI implementation team needs to explain these restrictions to the end users so that they are aware of the reasoning of such features. Further, refresher training should be conducted for the frontline workers to enhance their communication skills and understanding of the DHI as studies suggest that this would lead to increased confidence and information retention which would result in better work efficiency.

Technical difficulties may affect the quality and efficiency of work being done by the frontline workers. A significant number of the study participants reported frequent data sync and battery issues. This may be due to the poor infrastructure in peri-urban communities which has led to poor network connections and recurrent power outages. Similar studies discuss how the unreliability and low bandwidth of networks and unavailability of electricity to charge devices act as key barriers in hindering the effectiveness of DHIs in LMICs.[13,14] In contexts such as these, the responsiveness of the technical staff to address the users’ concerns is imperative for the success of a DHI.[16,18] The study participants reported that they were frequently met with prompt solution-based responses after reporting their technical difficulties, thus ensuring end-user support.

Besides the listed themes that were inquired about, the participants also indicated that the DHI improved the patient–provider relationship as they could be more attentive and maintain eye contact when interacting with patients. This is a significant impact as literature demonstrates that nonverbal communication helps further clarify and retain information, and generally builds a better bond between patients and providers.

Having a mixed-methods approach, this study was able to derive valuable insights to assess the user experience of the DHI holistically. It was also ensured that the study team was different from the implementing team to avoid any possibilities of biases that could influence the results. A limitation observed is that the study was conducted over a short time, during which the participants may have been experiencing more technical difficulties than usual. A mitigating strategy employed was to recruit health workers who have been using the DHI for at least six months to allow for overall rather than situational responses. Another possible limitation is that the study portrays the overall experiences of frontline workers from various backgrounds limiting external validity to a specific cadre. The adapted framework was also not validated in our context; however, the study team felt that it was the most appropriate evaluation framework in view of the study objectives.

Conclusion

DHIs can make a meaningful impact on underserved communities in LMICs. As a next step, the study team will work towards incorporating the suggestions received by the users. Periodic monitoring and evaluation of the DHI to assess the user experience, efficiency, and effectiveness of the application on process and outcome indicators in the MNCH journey has the potential to enhance quality health services to the wider community so as to reduce health disparities.

Click here for additional data file.

Supplemental material, sj-docx-1-dhj-10.1177_20552076221129076 for Health workers’ experience of a digital health intervention implemented in peri-urban communities in Karachi, Pakistan by Kaniz Amna Haider, Omar Gulam, Rehan Adamjee, Benazir Balouch and Zahra Hoodbhoy in Digital Health