Case study of Argus in Togo: An SMS and web-based application to support public health surveillance, results from 2016 to 2019.

INTRODUCTION: Argus is an open source electronic solution to facilitate the reporting and management of public health surveillance data. Its components include an Android-phone application, used by healthcare facilities to report results via SMS; and a central server located at the Ministry of Health, displaying aggregated results on a web platform for intermediate and central levels. This study describes the results of the use of Argus in two regions of Togo.
METHODS: Argus was used in 148 healthcare facilities from May 2016 to July 2018, expanding to 185 healthcare facilities from July 2018. Data from week 21 of 2016 to week 12 of 2019 was extracted from the Argus database and analysed. An assessment mission took place in August 2016 to collect users' satisfaction, to estimate the concordance of the received data with the collected data, and to estimate the time required to report data with Argus.
RESULTS: Overall completeness of data reporting was 76%, with 80% of reports from a given week being received before Tuesday 9PM. Concordance of data received from Argus and standard paper forms was 99.7%. Median time needed to send a report using Argus was 4 minutes. Overall completeness of data review at district, regional, and central levels were 89%, 68%, and 35% respectively. Implementation cost of Argus was 23 760 USD for 148 facilities.
CONCLUSIONS: The use of Argus in Togo enabled healthcare facilities to send weekly reports and alerts through SMS in a user-friendly, reliable and timely manner. Reengagement of surveillance officers at all levels, especially at the central level, enabled a dramatic increase in completeness and timeliness of data report and data review.

Introduction

Public health surveillance is an essential function of a health system, defined as “the systematic on-going collection, collation and analysis of data for public health purposes and the timely dissemination of public health information for assessment and public health response as necessary” [1]. Since 1998, the World Health Organization (WHO) has advocated for the Integrated Disease Surveillance and Response (IDSR) approach for public health surveillance in the WHO African region [2, 3] with specific roles defined at each level of the surveillance system. In resource-limited settings, paper-based transmission is the traditional way of reporting weekly public health surveillance data [3, 4]. Due to lack of resources and infrastructure, paper-based transmission may hamper full and timely data reporting and lead to difficulties in the routine analysis of the collected data, ultimately delaying the detection of a potential public health event such as an outbreak [3-8].

In the last few decades, the development of telecommunication has created an opportunity to enhance both the completeness and timeliness of data reporting. While information technology (IT) can facilitate the management and analysis of collected data [9], striking development of mobile phone infrastructure in resource-limited settings, including in hard-to-reach areas, along with internet access at the intermediate and central levels of the surveillance system offer new possibilities for IT tools to facilitate public health surveillance in compliance with the IDSR. Dedicated IT tools have already been used in several countries to facilitate reporting and management of public health surveillance data [7, 9–13]. In practice, many systems failed to allow remote healthcare facilities to report data due to lack of internet connectivity. Furthermore, we were unable to identify a single tool which would allow each level of the system to review and analyse data from its area in a workflow respecting the IDSR procedures.

WHO has therefore developed “Argus”, an IT tool to facilitate public health surveillance for early detection and response of events in compliance with IDSR procedures (http://www.argus.community/). It uses Short Message Service (SMS) technology for the transmission of aggregated weekly reports and alerts of serious or unexpected events between healthcare facilities and all levels of the public health surveillance system via a mobile phone application. A web platform complements the phone application for data management and analysis at each level of the system.

To finalize the development of Argus and assess if it could improve public health surveillance in Togo, Argus had been pilot tested in two regions of the country since May 2016. In this paper, we describe the results of the Argus pilot test in Togo in terms of data reporting by healthcare facilities, data review by surveillance officers, public health events identified, users’ satisfaction and costs.

Material and methods

Argus

Argus included several components (http://www.argus.community/):

A custom Argus Android Client application installed on Android mobile phones, Fig 1.

Fig 1

Argus IT solution for public health surveillance.

An Argus server located at the Togolese Ministry of Health in charge of data collection and management. It displayed an online web platform to review the collected data and identify potential acute public health events through weekly epidemiological summaries and dedicated dashboards, Fig 1. The Argus server was composed of three PHP applications exchanging information through XML files and web services, a web front end developed with the AngularJS framework, and an instance of the R statistical software.

A custom Argus Android Gateway application was installed on eight Android mobile phones at the Togolese Ministry of Health to act as a gateway between the Argus Android Clients and the Argus server.

Argus was released with a GNU AGPL3 license.

Pilot test in Togo

Argus was implemented in two regions of Togo (Lomé Commune and Savanes) in May 2016 to assess its usefulness in improving public health surveillance. Lomé Commune is the urban region capital of the country, located in the south. The other region, Savanes, is the northernmost region of the country, consisting mainly of rural and hard-to-reach areas.

Before implementing Argus, weekly public health surveillance from healthcare facilities was performed via paper transmission or phone calls to the district level. The district-aggregated forms were transmitted to the regions, who in-turn transmitted region-aggregated forms to the central level.

To make the best use of limited human and financial resources for public health surveillance, the list of events to be weekly reported should be kept to a minimum to ensure that it is manageable by the system and that adequate resources are available to carry out a response. For early detection of outbreaks via weekly aggregated reporting, it was recommended to keep the number of variables to a minimum, ideally reporting only number of cases and deaths, to avoid unnecessary burden on the healthcare facilities and maximize reporting efficiency. In the planning process of Argus, and prior to its implementation, the Ministry of Health revised the list of diseases and conditions to be reported on a weekly basis. It was reduced from 20 to 13, with acute flaccid paralysis, acute haemorrhagic fever, bloody diarrhoea, cholera, flu syndrome, human anthrax, human rabies, measles, meningitis, neonatal tetanus, other severe diarrhoea, severe acute respiratory syndrome and yellow fever. The variables to be collected were reduced from seventeen variables (mainly number of cases and deaths by age groups) to two: the weekly number of cases and weekly number of deaths. A new paper form was developed to support tallying of the weekly figures based on the healthcare facility registry.

Data was to be reported with Argus before the paper form was archived in the healthcare facility. Alert thresholds at healthcare facility level were defined for nine diseases and conditions (acute flaccid paralysis: 1, acute haemorrhagic fever: 1, cholera: 1, human anthrax: 1, human rabies: 1, measles: 3, meningitis: 2, neonatal tetanus: 1, yellow fever: 1). If the weekly number of cases reached an alert threshold, a warning message appeared on the Argus Android Client application at the healthcare facility level and the disease or condition was highlighted on the aggregated report displayed on the Argus web platform.

Argus duties at each level of the surveillance system were defined as follows, based upon the IDSR guidelines:

Healthcare facility: send the weekly aggregated report from the previous week on Monday before four PM. In case of occurrence of a serious or unexpected event, staff at healthcare facilities were required to immediately notify their supervisor, they could do so using Argus. Upon description of the serious or unexpected event on the Argus Android Client application, information was immediately forwarded to the personal mobile phone of pre-identified supervisors by SMS.

District level: review (i.e. validate or reject) healthcare facilities weekly reports and examine the weekly epidemiological summary to identify any potential public health event before Monday midnight.

Regional level: review districts aggregated weekly reports and examine the weekly epidemiological summary before Tuesday four PM.

Central level: review regions aggregated weekly reports and examine the weekly epidemiological summary before Tuesday midnight.

Seven training sessions were conducted between the 19th and 25th May 2016, attended by at least one staff from each healthcare facility, one surveillance officer from each district and region, and one surveillance officer from the central level. An Android phone equipped with Argus and a parental control application was provided to each healthcare facility along with brochures and posters detailing how to use it. Due to the parental control application, the phone could only be used to run the Argus application. Each training session was composed of:

A 1-hour plenary on the concepts and objectives of public health surveillance;

A 30-minute plenary on the Argus tool and its pilot in Togo;

A 45-minute practical exercise on sending a weekly report and an alert; and

A 1-hour session for intermediate and central level surveillance officers on the use of the central web platform to review reports, examine their weekly epidemiological summary and perform custom analyses.

Deployment of Argus took place in two phases. An initial phase was conducted from May 2016 to July 2018 and included 148 healthcare facilities (54 from the 5 districts of Lomé Commune and 94 from the 5 districts of Savanes). A reinforcement phase then started on July 2018, including 185 healthcare facilities (87 from Lomé Commune and 98 from Savanes). The implementation, operation and assessment of the pilot were conducted under the supervision of the Togolese Ministry of Health with the support of the WHO country office in Togo and WHO Headquarters Lyon office.

Prior to the reinforcement phase, the list of diseases and conditions to be reported and the alert thresholds were updated by the Ministry of Health. During this update, other severe diarrhoea was removed; dengue, neonatal deaths and maternal deaths were added; alert threshold for measles was reduced to 2 cases, and alert thresholds for bloody diarrhoea and flu syndrome were added with 5 and 20 cases respectively. Seven training sessions were conducted in July 2018 to update the Argus application on all healthcare facility phones and provide reinforcement training to all staff. A responsible focal point at the central level was designated to supervise the pilot of Argus during the reinforcement phase.

Outcome measures

An evaluation protocol detailing the indicators to be collected and their collection and analysis modalities was developed by the WHO and agreed upon with the Ministry of Health at the onset of the pilot test. The primary outcomes of interest were the fulfilment of the duties attributed to each level of the surveillance system, i.e. the completeness and timeliness of data reporting and data review. Additional indicators were added to assess: the quality of the reported data (i.e. concordance of the received data with the collected data); the public health events identified (i.e. number of alert thresholds reached and immediate notifications received); the users’ satisfaction regarding several characteristics of the system (i.e. appearance, documentation, usefulness, simplicity); the reliability of the system (i.e. percentage of uptime of the Argus web platform); and the costs of implementation and operation. The full list of indicators used to assess and describe the Argus pilot test results in Togo is presented in Table 1.

Table 1

Indicators used to describe the Argus pilot test results.

Type of outcome measure	Indicators and calculation method	Data source	Period
Data reporting by healthcare facilities	Evolution of completeness and timeliness of data reporting by healthcare facilities.	Argus database.	W21 2016 –W12 2019
	(No. of reports received from healthcare facilities / No. of reports expecteda from healthcare facilities)*100.
	(No. of reports received on time from healthcare facilities/ No. of reports expecteda from healthcare facilities)*100.
	Time required by a user at healthcare facilities to report data with Argus Android client.	Assessment mission.
	Concordance of the received data with the collected data.	Paper forms.	W21 2016 –W31 2016
	(No. of data having the same value in the system and on the paper forms) / (No. of data collected in the system)*100.	Argus database.
Data review by surveillance officers	% of uptime of the Argus web platform.	Monitoring serviceb.	30/05/2016–01/04/2019
	(duration of Argus web platform online and available to users / duration of Argus web platform offline)*100.
	Evolution of completeness and timeliness of data review at each level.	Argus database.	W21 2016 –W12 2019
	By level: (No. of validated or rejected reports / No. of reports received)*100.
	(No. of reports received and validated or rejected on time / No. of reports received on time)*100.
Public health events identified	Number of reached alert thresholds identified by the system.	Argus database.	W21 2016 –W12 2019
	Number of immediate notifications of serious or unexpected events received.	Argus database.	30/05/2016–01/04/2019
Users’ satisfaction	Opinion on:	Assessment mission.
	• general appearance of Argus Android Client at healthcare facilities;
	• available documentation for Argus Android Client used at healthcare facilities;
	• general appearance of Argus web platform at intermediate and central levels;
	• available documentation for Argus web platform at intermediate and central levels;
	• usefulness of Argus Android Client used for data reporting;
	• usefulness of Argus web platform for data review;
	• usefulness of Argus web platform for data analysis;
	• overall simplicity of use of Argus Android client;
	• simplicity of use of Argus Android client for data reporting;
	• overall simplicity of use of Argus web platform;
	• simplicity of use of Argus web platform for data review;
	• simplicity of use of Argus web platform to monitor the completeness and timeliness of data reporting.
	Sum of ratings assigned by respondents (from 1 worst to 5 best) / No. of respondents.
Costs	Cost of setting up Argus.	Invoices paid for the project.	18/05/2016–04/04/2019
	Cost of internet access to the central server.	Argus database
	Cost of SMS exchanges.

a One report expected each week from each healthcare facility included in the pilot.

b https://www.site24x7.com/

To assess data reporting and data review and to describe public health events identified by the system we extracted data from epidemiological week 21 of 2016 to epidemiological week 12 of 2019 from the Argus database. We monitored the percentage of uptime of the Argus web platform through an online monitoring service (https://www.site24x7.com). To describe the costs of the implementation and operation of the system we used the invoices paid for the project and extracted the number of SMS exchanged from the Argus database.

One assessment mission took place from the 4th to 12th August 2016 to collect users’ satisfaction, to estimate the concordance of the received data with the collected data, and to estimate the time required to report data with Argus. This was done through eight sessions organized with all users of the system, each session was composed of: a 20-minute presentation on the current use of Argus in Togo; a 40-minute slot for participants to fill in a user-satisfaction questionnaire (S1 Questionnaires); a 1-hour structured discussion with all participants; and a 30-minute structured discussion with surveillance officers at intermediate and central levels.

To estimate the concordance of the received data with the collected data, participants were asked to bring the paper forms archived at the healthcare facilities and used to tally the weekly figures. A copy of each form was collected and its data entered twice into a database using the Epidata software.

To estimate the time required to report data with Argus, participants in four of the eight evaluation sessions were provided with a mock paper tallying form and asked to report it using Argus. The amount of time it took each participant to enter the information and send the report was measured and registered by one surveyor.

All data analyses were performed using the R statistical software [14].

Results

Data reporting by healthcare facilities

Results of completeness and timeliness of data reporting by healthcare facilities are available Table 2.

Table 2

Results of completeness and timeliness of data reporting and data review.

Indicator	Overall period	Initial phase	Reinforcement phase
Level	(weeks 21 of 2016–12 of 2019)	(weeks 21 of 2016–29 of 2018)	(weeks 30 of 2018–12 of 2019)
Completeness of data reporting
% (No. of reports received; No. of expected reports)
Country	76% (17 406; 23 062)	69% (11 591; 16 724)	92% (5 815; 6 338)
Savanes region	81% (11 315; 13 979)	78% (8 240; 10 622)	92% (3 075; 3 357)
Lomé Communes region	67% (6 091; 9 083)	55% (3 351; 6 102)	92% (2 740; 2 981)
Timeliness of data reporting
% (No. of reports received before Monday 4PM; No. of expected reports)
Country	48% (10 987; 23 062)	39% (6 573; 16 724)	70% (4 414; 6 338)
Savanes region	52% (7 267, 13 979)	46% (4 846; 10 622)	72% (2 421; 3 357)
Lomé Communes region	41% (3 720, 9 083)	28% (1 727; 6 102)	67% (1 993; 2 981)
Median time for a weekly report to be received by the server
Time from the end of the reported week, Sunday midnight (No. of received reports; [10th-90th percentiles])
	11h (n = 17 221; [7h–10d])	13h (n = 11 406; [8h–14d])	9h (n = 5 815; [7h–7d])
Time when 80% of reports were received by the server.
Time from the end of the reported week, Sunday midnight
	2 d 21h (Wednesday 9PM)	3d 19h (Thursday 7PM)	1d 8h (Tuesday 8AM)
Completeness of data review
District level	89% (15 511; 17 406)	86% (9 442; 11 591)	96% (5 569; 5 815)
% (No. of reports validated or rejected; No. of reports received from healthcare facilities)
Regional level	68% (958; 1 404)	60% (643; 1 065)	93% (315; 339)
% (No. of reports validated or rejected; No. of district aggregated reports)
Central level	35% (101; 289)	16% (37; 225)	100% (64; 64)
% (No. of reports validated or rejected; No. of regional aggregated reports)
Timeliness of data review
District level	48% (5 279; 10 987)	31% (2 050; 6 573)	73% (3 229; 4 414)
% (No. of reports reviewed before Monday midnight; No. of reports received from healthcare facilities before Monday 4PM)
Regional level	41% (188; 459)	21% (46, 223)	60% (142; 236)
% (No. of reports reviewed before Tuesday 4PM; No. of district aggregated reports created before Monday midnight)
Central level	43% (29; 67)	7% (2; 27)	68% (27; 40)
% (No. of reports reviewed before Tuesday midnight; No. of regional aggregated reports created before Tuesday 4PM)
Median time for a district to review a report upon reception (hours, [10th-90th percentiles])
Time upon reception of the report; [10th-90th percentiles]
	1d 7h (n = 16 135; [3h–19d])	2d 5h (n = 10 511; [6h–25d])	8h (n = 5 624; [1h–7d])

The evolution of completeness of data reporting is presented in Fig 2, the distribution of weekly reports reception in Fig 3.

Fig 2

Evolution of completeness of data reporting by healthcare facilities from week 21 2016 to week 12 2019 (n = 23 062 expected reports).

Fig 3

Distribution of weekly reports reception time as from Sunday midnight (n = 17221 received reports).

Reception time axis: base 10 log scale.

By district, for the overall period, completeness of data reporting ranged from 63% to 90% and timeliness from 36% to 59%.

The median time required by a user at a healthcare facility to fill and send a weekly report (composed of the number of cases and deaths for 13 diseases and conditions) was 4 minutes (n = 73; 10th- 90th percentiles: [2 minutes; 7 minutes]).

One thousand seventy-one paper forms were collected from 144 healthcare facilities. When comparing reports received from Argus with the paper forms: 94% were identical (n = 1200); 4% had one error (n = 48); 2% had more than one error (n = 23). The overall percentage of consistent data between Argus and the paper forms was 99.7% (n = 32397 / 32506).

Data review by surveillance officers

From the 30th May 2016 to 1st April 2019, the Argus web platform was online and accessible to the users 94% of the time. The Argus web platform faced 61 offline periods: 11 lasted less than 10 minutes; 26 lasted between 10 minutes and one hour; 19 lasted between one hour and one day; five lasted more than a day (i.e. 2 days, 5 days, 15 days, 17 days, 20 days).

Results of completeness and timeliness of data review by surveillance officers are available Table 2. The evolution of completeness of data review is presented in Fig 4.

Fig 4

Evolution of completeness of data review from week 21 2016 to week 12 2019 by level.

The drop on week 30 of 2018 was linked to the update of the system to accommodate the new list of diseases to be reported.

By district, for the overall period, the completeness and timeliness of data review ranged from 82% to 96% and 22% to 68% for the completeness and timeliness respectively.

Public health events identified

From week 21 of 2016 to week 12 of 2019, alert thresholds were reached 934 times: flu syndrome (n = 321); yellow fever (n = 251); meningitis (n = 88); acute flaccid paralysis (n = 71); bloody diarrhoea (n = 69); human rabies (n = 50); acute haemorrhagic fever (n = 38); measles (n = 36); neonatal tetanus (n = 9); cholera (n = 1). Ninety-six healthcare facilities reported a weekly number of cases reaching an alert threshold at least once. Sixty-four immediate notification of serious or unexpected public health events were sent by a healthcare facility and forwarded to the personal mobile phone of their supervisor. Eighteen were related to a real alert (i.e. four for suspect cholera, three meningitis, two acute haemorrhagic fever, one acute flaccid paralysis, one avian flu, one bloody diarrhoea, one cluster of deaths, one food poisoning, one human anthrax, one human rabies, one measles, one yellow fever) and 46 were related to testing or manipulation error by the healthcare facility.

Users’ satisfaction

An evaluation questionnaire was filled by 144 healthcare staff and 20 surveillance officers at district, regional and central levels. Ninety nine percent of the healthcare facility staff reported owing a personal mobile phone (n = 140/142), with a median duration of ownership of 10 years (10th-90th percentiles: [7 years; 15 years]). Ninety percent of the surveillance officers reported owing a personal computer (n = 18/20), with a median duration of ownership of 4 years (10th-90th percentiles: [3 years; 14 years]). All surveillance officers reported accessing the Internet for their personal use: 45% daily (n = 9/20), 35% weekly (n = 7/20), 20% less than weekly (n = 4/20). Results of the questionnaire are displayed in Table 3.

Table 3

Results of the evaluation questionnaires.

Indicator	No. of respondents	Score (from 1 worst to 5 best)
		Median	10th—90th percentiles
Argus Android client used at healthcare facilities
General appearance	143	5.0	[4.0; 5.0]
Available documentation	142	5.0	[4.0; 5.0]
Usefulness for data reporting	143	5.0	[4.0; 5.0]
Overall simplicity of use	142	5.0	[4.0; 5.0]
Simplicity of use for data reporting	142	5.0	[4.0; 5.0]
Argus web platform used by surveillance officers
General appearance	14	4.5	[3.3; 5.0]
Available documentation	16	5.0	[4.0; 5.0]
Usefulness for data review	15	5.0	[4.0; 5.0]
Usefulness for data analysis	15	4.0	[2.4; 5.0]
Overall simplicity of use	16	5.0	[2.5; 5.0]
Simplicity of use for data review	15	5.0	[3.4; 5.0]
Simplicity of use to monitor completeness and timeliness of data reporting	16	5.0	[3.0; 5.0]

Costs

The sum of implementation costs was 23 760 USD, shared between: two servers (4 351 USD), 185 Android phones (10 770 USD), SIM cards (655 USD), brochures and posters printing (855 USD), training sessions (7 130 USD).

The internet access of the central server (for the Argus web platform to be online) costed 168 USD per month. From 18th May 2016 to 04th April 2019, 732 932 SMS were exchanged between the phones and the central server for an estimated overall cost of 31 870 USD (cost of 0.033 USD per SMS using network 1 and 0.05 USD per SMS using network 2). Considering 15 diseases and conditions to be reported, the cost to send a weekly report and receive feedback was 1 USD using network 1 and 1.5 USD using network 2.

Discussion

The use of Argus in Togo enabled healthcare facilities to send their weekly reports and alerts through SMS in a user-friendly, reliable and quick manner. During an almost three years period, the overall completeness of data reporting by healthcare facilities was 76%. Eighty percent of the reports of a given week were received by Tuesday evening of the following week. Automated feedback to the healthcare facilities on reached alert thresholds, automated highlights of these reached thresholds on the web platform, and automated data analyses facilitated quick identification of potential acute public health events.

Through the Argus web platform, each level could perform its respective duties. While 89% of reports where quickly reviewed by the surveillance officers at the district level, surveillance officers at the regional and central levels showed a lack of engagement with only 68% and 35% of their respective aggregated reports reviewed. Re-engagement of surveillance officers at all levels during the reinforcement phase enabled a dramatic increase in completeness and timeliness of data report and data review, especially at the central level.

With three years of operation in two regions of the country, our study was able to provide a description of the use of Argus for both an extended time-period and area [15, 16]. The lack of monitoring of the completeness and timeliness of weekly reporting by healthcare facilities in the paper-based system prevented temporal comparisons before and after the implementation of Argus and geographical comparisons with regions without Argus. Indeed, only figures of completeness and timeliness of data reporting from the district level were available in the paper-based system. This capacity of Argus to monitor the number of reporting healthcare facilities provided the capacity to display the representativeness of each aggregated report.

We believe the revision by the Ministry of Health of the aggregated data to be weekly reported, with a decrease in the number of diseases and conditions from 20 to 13 and in the number of variables from 17 to 2 was crucial in the adhesion of healthcare facility staff. Due to the study design, the effect of this reduction in the completeness and timeliness of data reporting can’t be independently assessed from the use of Argus, but is very likely important. The crucial role of supervision at district, regional and central level was highlighted by the relationship between completeness and timeliness of data reporting and data review. It showed that better performance could be achieved when each level took upon its duties in a timely manner. When a drop in the completeness of data review happened at the central and regional level, a drop was also identified in the completeness of data reporting. However, completeness of data reporting remained high in the mainly rural and hard-to-reach region of Savanes. This highlights the added value of data transmission by SMS in remote areas [4].

The main challenge faced during the study was the lack of involvement and ownership of the central level on a day to day basis, as already described in similar settings [5, 7]. Reinforcement training and designation of a responsible focal point at the central level at the onset of the reinforcement phase lead to increased performance at all levels.

Another common challenge was the lack of available IT competencies at the Ministry of Health [5]. The Argus web platform was offline for three extended periods due to configuration problems with the Internet service provider. These offline periods could have been reduced to a few hours had qualified IT staff at the Ministry of Health been available and responsible for the system. While some authors described network coverage and internet availability at district level as challenges [7], these did not significantly affect the use of Argus in Togo.

With an implementation cost of 23 760 USD for 148 facilities Argus was an affordable solution. Communication costs could easily be reduced through specific agreements with mobile service providers and through the decrease in the number of SMS used to report (currently one SMS is used for each disease). While we couldn’t compare the cost of using Argus with the cost of the previous paper-based reporting system, several authors reported fewer costs in a system using an IT tool mainly due to the removal of paper transportation and data entry costs [4, 13, 17, 18].

We believe providing a tool that complies with existing IDSR procedures in the country, especially its workflow trough the different levels of the surveillance system, is a prerequisite to its adoption and sustainability. Monitoring the work of surveillance officers at each level and displaying the results in a shared platform was an effective manner to make every actor accountable. Simplicity of use and limitation to features needed for each actor’s duties were at the core of Argus development. This simplicity by design may explain the limited time needed to fill a report (4 min) and the high accuracy of the reported data (99.7% consistency with paper forms).

A common criticism in the field of mHealth is the lack of interoperable and open sourced systems that are flexible enough to interface with existing infrastructure (e.g. existing health information management system) and to adapt to new requirements (e.g. new list of diseases to be reported) [5, 9, 19, 20]. Through its modular architecture and use of XML files to share information Argus can easily be used with other existing tools [21]. At the Ministry of Health, both the units in charge of the IDSR and of the Health Information Management System were involved in the pilot of Argus. The unit in charge of Health Information Management System was also piloting DHIS2 as a data warehouse and monthly data reporting tool. One option considered by the authorities was to automatize a weekly data export from the Argus instance to the DHIS2 instance.

Similarly, unlike elsewhere reported [5], Argus was conceived from the beginning to be highly adaptable to the specific procedures of each country. Its administrators can easily set up the number of levels of the system, the list of diseases and variables to be reported, or activate and deactivate reporting sites over time. Its free and open-source license enables Argus to be used free of charge and to be modified to best suit the needs of each country [18].

Argus has proved itself an affordable and effective solution for public health surveillance in the two regions of Togo evaluated in this study. Extension of Argus to the whole country is currently under consideration by the Togolese health authorities. Such extension should only be performed ensuring health authorities at each level of the system are fully engaged in the daily operation of the system and staff with IT skills is available as needed. When considering the use of telecommunication and IT tools to improve public health surveillance, countries must also consider other crucial factors such as the integration of data collection and reporting between parallel systems and the limitation in the amount of collected data. Indeed, while the burden put by health information systems on healthcare facility staff is often overwhelming [22-24], minimum data (i.e. weekly number of cases, or alert of an unexpected or serious event) is enough to trigger an investigation.

Evaluation questionnaires.

(PDF)

Click here for additional data file.

(ZIP)

Click here for additional data file.

13 Sep 2020

PONE-D-20-01670

Argus in Togo: an SMS and web-based application to support weekly public health surveillance, results from 2016 to 2019.

PLOS ONE

Dear Dr.Guerra

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Oct 28 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Luigi Lavorgna

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Our internal editors have looked over your manuscript and determined that it is within the scope of our Digital Health Technology Call for Papers. This collection of papers is headed by a team of Guest Editors for PLOS ONE: Eun Kyoung Choe (University of Maryland, College Park), Chelsea Dobbins (University of Queensland), Sunghoon Ivan Lee (University of Massachusetts, Amherst), and Claudia Pagliari (University of Edinburgh).

The Collection will encompass a diverse range of research articles on digital health technologies ranging from technology design to patient care and health systems management.  Additional information can be found on our announcement page: https://collections.plos.org/s/digital-health-tech.

If you would like your manuscript to be considered for this collection, please let us know in your cover letter and we will ensure that your paper is treated as if you were responding to this call. If you would prefer to remove your manuscript from collection consideration, please specify this in the cover letter.

3. In ethics statement in the manuscript and in the online submission form, please provide additional information about the database used in your retrospective study. Specifically, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have their data used in research, please include this information.

4. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The article presents data on the use of an SMS-based surveillance tool from 2016 thru 2019 in two regions within Togo. The goal of the system is to improve reporting of notifiable diseases to the district, regional, and central Ministry of Health in support of IDSR. The SMS tool supports the use of ISDR at all levels of public health. The tool is open source, works with an Android platform, analyzes data using R, and utilizes XML files.

The evaluation of the system is descriptive. The examination did not directly compare paper-based reporting to electronic-supported reporting. Furthermore, the examination did not compare the initial phase with the reinforcement phase. The paper largely presents a collection of data gathered by the team who worked on the implementation. The lack of rigorous methods and limited ability to generalize findings detracts from an interesting report of a novel tool developed for ISDR. While interesting and potentially useful, there are a number of items that need to be addressed before the paper can be further considered for publication.

MAJOR ITEMS

1. A major weakness of the paper is the lack of a clear study design. The paper was submitted as a Research Article. Yet the paper does not appear to be a research study. The paper should follow the STARE-HI guidelines for reporting evaluations of health informatics applications (https://www.imia-medinfo.org/new2/Stare-HI_as_published.pdf). Many aspects of STARE-HI are present in the paper. However, methodological details on study design and evaluation criteria are missing. The paper needs to make it clear in the ABSTRACT and METHODS sections that this is a descriptive study. It further needs to characterize the approach to evaluation. Why was the Evaluation mission so short in duration, even though the paper presents data on a longer time period?

With respect to Table 1 in the METHODS, a variety of time periods and measures were used. The table is a handy reference. Yet it is not clear why so many components were used. A clearer vision for the overall evaluation strategy would be helpful. Did the team use a logic model? Were there primary outcomes of interest? Which metrics were most important to the Ministry? How about the health care facilities? Districts? Were the various stakeholders consulted about the evaluation? Who organized and directed the evaluation?

2. With respect to development of the Argus tool, who commissioned the tool? WHO? Were the stakeholders like MOH and districts and health care facilities consulted about the development of the tool? Was there any kind of prototype development or usability testing done before implementing in Togo? Were best practices for e-health development utilized? Is the tool part of Togo’s e-Health Strategy as a country? It is not clear why the tool was tested in Togo first rather than another nation.

3. The paper needs a Limitations section where the authors make it clear that improvements in reporting might have been confounded by the fact that the Ministry constrained weekly reporting before the rollout of the tool. Dropping the number of diseases to be reported and reducing the fields to be reported likely facilitated better reporting even without the Argus tool. What variables were dropped from the old paper forms? Why did the ministry constrain reporting for Argus deployment? Did reporting change in the other regions? While a comparison with other regions might not be possible due to data limitations, the fact that close supervision at district, regional, and central levels was associated with better reporting suggests that the Argus tool might have little to do with the outcome. Instead if countries spent more time on these fundamental aspects of IDSR then we might have a more resilient surveillance infrastructure. This should be discussed.

4. The number of cases needed for an alert to occur was very small (primarily 1 case). Did an abundance of alerts (N=936 which is approximately 6 per week) contribute to the lackluster use of the tool at the Regional and Central levels? Syndromic surveillance tools have long suffered from “alert fatigue” in the same way as clinical decision support systems. What effect, if any, might this have had on the adoption and use (and enthusiasm) of the Argus tool outside of the health care facilities? This kind of discussion needs to be added to the DISCUSSION section.

5. The paper makes a point that Argus is open source and designed with interoperability in mind. Yet the paper does not discuss how the tool does or does not integrate with DHIS2, a system used in many countries to support IDSR. Does this tool interface with DHIS2? Are there plans to support use of Argus in conjunction with DHIS2 if some regions in a decentralized country want to use DHIS and others wish to use Argus?

MODERATE ITEMS

6. The drop in Week 30 2018 is linked to a system update that included new diseases. Did this correspond with additional training for facilities and ministry staff members? This is not discussed in the DISCUSSION section. It seemed to be a major driver of utilization.

7. Some of the detailed data presented in paragraphs in the RESULTS section could become tables that would more succinctly present the results, especially with respect to completeness and timeliness overall and stratified by district, region, and central levels. Furthermore, the differences before and after the reinforcement phase in a table aligned with the initial phase would strengthen comprehension and comparison of these data.

MINOR COMMENTS / SUGGESTIONS

8. The data on phone and computer ownership is not useful as presented. These results are not discussed, nor do these data seem to relate to completeness or timeliness. Consider removing these data.

9. It would be VERY helpful if the “Evaluation mission” as well as the reinforcement phase were highlighted in Figures 2, 3, and 4. Identifying when the re-training occurred would be helpful to see if this was associated with an uptick in completeness.

10. The images are quite grainy in Figure 1. As a result it is difficult to read the words in the images. Higher quality images should be provided for publication.

Reviewer #2: I found this paper and the described approach extremely interesting and complete. This kind of mHealth is even more actual in this pandemic phase, during which less progressed countries might have been poorly estimated due to underreporting of events.

What the authors clearly state is the limitation due to lack of engagement. Re-engagement of surveillance officers

enabled an increase in completeness of data report . It would be interesting to explain more extensively how re-engagement was performed and which strategies might be implemented to overcome lack of engagement.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Brian E. Dixon, PhD

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

12 Nov 2020

Please find in the attached file labeled "Response to Reviewers" detailed response to each point raised by the reviewers.

Submitted filename: response_reviewers.pdf

Click here for additional data file.

17 Nov 2020

Case study of Argus in Togo: an SMS and web-based application to support public health surveillance, results from 2016 to 2019.

PONE-D-20-01670R1

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Luigi Lavorgna

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

19 Nov 2020

PONE-D-20-01670R1

Case study of Argus in Togo: an SMS and web-based application to support public health surveillance, results from 2016 to 2019.

Dear Dr. Guerra:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Luigi Lavorgna

Academic Editor

PLOS ONE