Electronic medical records in humanitarian emergencies - the development of an Ebola clinical information and patient management system.

By November 2015, the West Africa Ebola epidemic had caused 28598 infections and 11299 deaths in the three countries most affected. The outbreak required rapid innovation and adaptation. Médecins sans Frontières (MSF) scaled up its usual 20-30 bed Ebola management centres (EMCs) to 100-300 beds with over 300 workers in some settings. This brought challenges in patient and clinical data management resulting from the difficulties of working safely with high numbers of Ebola patients. We describe a project MSF established with software developers and the Google Social Impact Team to develop context-adapted tools to address the challenges of recording Ebola clinical information. We share the outcomes and key lessons learned in innovating rapidly under pressure in difficult environmental conditions. Information on adoption, maintenance, and data quality was gathered through review of project documentation, discussions with field staff and key project stakeholders, and analysis of tablet data. In March 2015, a full prototype was deployed in Magburaka EMC, Sierra Leone. Inpatient data were captured on 204 clinical interactions with 34 patients from 5 March until 10 April 2015. Data continued to also be recorded on paper charts, creating theoretically identical record "pairs" on paper and tablet. 83 record pairs for 33 patients with 22 data items (temperature and symptoms) per pair were analysed. The overall Kappa coefficient for agreement between sources was 0.62, but reduced to 0.59 when rare bleeding symptoms were excluded, indicating moderate to good agreement. The time taken to deliver the product was more than that anticipated by MSF (7 months versus 6 weeks). Deployment of the tablet coincided with a dramatic drop in patient numbers and thus had little impact on patient care. We have identified lessons specific to humanitarian-technology collaborative projects and propose a framework for emergency humanitarian innovation. Time and effort is required to bridge differences in organisational culture between the technology and humanitarian worlds. This investment is essential for establishing a shared vision on deliverables, urgency, and ownership of product.

Background

By November 2015, the West Africa Ebola epidemic had caused 28598 infections and 11299 deaths in the three countries most affected [1]. In response, Médecins sans Frontières (MSF), drawing on over 20 years of experience in responding to Ebola outbreaks, had treated more than 7500 suspected Ebola cases, including more than 4700 confirmed cases. The outbreak, unprecedented in size, required rapid innovation and adaptation. To cope with the number of cases, MSF scaled up its usual 20–30 bed Ebola management centres (EMCs) to 100–300 beds with over 300 workers in some settings. This increased scale brought challenges in patient and clinical data management resulting from the difficulties of working safely with high numbers of Ebola patients ( Panel 1). Little had been published on managing clinical documentation and data transfer from filovirus wards, and there were no established, standardized, or widely used approaches [2]. Here we describe how MSF developed context-adapted tools to address the challenges of recording Ebola clinical information for in-patient management. We share the key issues and lessons learned in innovating rapidly under pressure in difficult environmental conditions and propose a framework for emergency humanitarian innovation.

Panel 1.

Challenges in patient and clinical data management in Ebola management centres.

• Staff working in the high-risk zone of an EMC (the area where there is high risk of Ebola infection) must wear PPE, which is heavy, limits vision severely, adversely affects manual dexterity, and is extremely hot. Clinicians have a strictly limited time in which they can safely remain in PPE without overheating and increasing infection risk during PPE removal. A lot of time was lost while clinicians in PPE tried to find patients inside the high risk zone of the EMC. Patients were moved through tents in the EMC representing different levels of risk during their treatment and could also move around within the EMC of their own volition (patients can become confused and delirious during Ebola infection), which made locating a specific patient in a large EMC challenging. • Clinicians lacked an efficient means to document patient history and record patients’ progress through the EMC. In some EMCs, clinicians in full PPE shouted data over the fence for someone outside the high-risk zone to record on ‘clean’ paper, or data were recorded in digital photographs, since nothing could be taken out of the high-risk zone without being disinfected in chlorine. The original paper records made in the high-risk zone were incinerated. As a result, adequate data to support good clinical care were rarely available. • Data collection processes varied widely between EMCs, resulting in little data sharing between them and limiting the potential for learning and improving care. There was no centralized organizing authority for data collection forms or formats. • Despite the efforts of the Emergency Telecommunications Cluster *, most EMCs had spotty and intermittent internet connectivity at best.

EMC=Ebola management centre. PPE=personal protective equipment.

*Information on the ETC is at: http://www.etcluster.org/about-etc

Methods

The project

In September 2014, MSF medical staff established a collaborative project with a small group of software developers and the Google Social Impact Team. The aim was to develop a tool to enable the collection, visualisation, and sharing of standardised information on Ebola patients and treatment programmes, and thereby make the most efficient use of the limited time that staff are able to remain in personal protective equipment (PPE) to treat patients in the high-risk zone of an EMC.

Informal discussions with current and returning MSF field staff enabled a first set of requirements to be drawn up. Initial scoping, conducted by Google over 2 days, showed that several data collection devices were already under development for Ebola, some of which promised to be rapidly deployable [3, 4]. However, it was determined that none of these would sufficiently meet the requirements that had been identified. These requirements were further refined through extensive user consultation including current MSF national and international field staff, operations managers, and medical specialists with experience in Ebola, as well as public health specialists from Harvard Medical School and The London School of Hygiene and Tropical Medicine. A product specification was generated to meet these requirements ( Panel 2) based on the following principles: the solution should be available quickly (within 6 weeks); it should be “just enough” to meet the requirements; it should be useable without programming knowledge; and both the product and its intellectual property (IP) should be freely or cheaply available.

Panel 2.

Core requirements and the initial proposed solution (the minimum viable product).

Requirements	Solution (minimum viable product)
Clinicians within the high-risk zone wearing PPE are able to quickly, easily, and safely record and visualise patient data, to enable appropriate clinical decision making.	Waterproof tablets with wireless charging, disinfected by dipping in 0.5% chlorine. A tablet app enabling rapid data entry in PPE (large buttons, no swipe-screen features), providing instant access to patient historical records, a current patient list which staff can fill during clinical rounds, and an interface for the creation of more fields as required.
Data transfer out of the high-risk zone is easy, safe, and secure, enabling aggregation and analysis of outcomes (programme monitoring).	A local server-side application to synchronise data between tablets, store it securely on a server outside the high-risk zone, and export anonymised patient data (to the field office and headquarters in Europe) when an internet connection is available.
System functions with unreliable power supply (up to 12 hours without electricity), poor or no internet connectivity, and no in-country IT support desk or readily available electronics supply.	Lightweight, plug-and-play, battery-powered server infrastructure, automatically backed up on every device in case some fail.
Clinicians within the high-risk zone able to locate patients quickly, easily, and safely.	Radio-frequency identification tags on wristbands to rapidly locate patients within the centre and confirm their identity; readers for the tags located on each tablet device.

PPE=personal protective equipment.

To speed up development and ensure that the final tools would be available free of charge to Ministries of Health or other humanitarian organisations, the development team used pre-existing open-source platforms where possible, rather than developing from scratch. The data model and database from OpenMRS (an open-source Medical Record System; platform v 1.10.1 on an Edison server running Yocto Linux + Debian GNU/Linux 7) were combined with data entry elements from OpenDataKit (open-source mobile data collection tools), together with a bespoke user interface (made with code forked from ODK Collect v1.4.4, running on Android 4.4.2 [KitKat]), designed for a high-risk zone. Panel 3 outlines the tool that was finally developed. A full, open-source, technical specification of the product is available [5].

The client-server architecture developed for this project involves a low-energy server implemented on a 36 × 25 × 4 mm Intel Edison computer, built into an enclosure full of batteries and a custom charging circuit to ensure all-hours reliability. Rather than relying on the internet for updates, backups, and maintenance, the backup and updating system relies solely on USB sticks. The client software installed on the tablets can be replaced with any other application, which can then benefit from the hardware and server capacity of the system. Servers can be deployed inside and outside the high risk zone, enabling safe and effortless transfer of data between locations.

Approximately US$1.9 million was spent on development, of which $1.8 million was provided by Google. This included a team of five full-time engineers for 6 months, as well as contractors and consultants. Approximately $500,000 was spent on custom manufacturing of hardware, often at above-market costs due to the urgency of the project. For instance, a factory in California was commissioned to run for 72 hours straight during the Christmas of 2014 to manufacture tablet enclosures. In all, the project took 7 months from concept to deployment.

Piloting and evaluation

In January 2015, an alpha version was field-piloted in the MSF EMC in Magburaka, Sierra Leone; feedback from the field team was incorporated and bugs were fixed. In March 2015, a full prototype was deployed in Magburaka. Inpatient data were captured on 204 clinical interactions with 34 patients from 5 March until 10 April 2015 (equating to 95% of those admitted in this relatively quiet period). In this initial deployment, for each clinician-patient interaction the routine paper record system was maintained in parallel; clinical observations and treatments were recorded on paper in the high risk zone by the clinician wearing PPE, then shouted over the fence to a colleague standing in the low-risk zone (an area of an EMC where there is low risk of Ebola transmission and staff do not wear full PPE) who transcribed them to ‘clean’ paper patient charts, and later entered into an Excel database by a data encoder.

Information on adoption, maintenance, and data quality was gathered through review of project documentation, discussions with field staff and key project stakeholders, and analysis of data from the tablets. We carried out a rapid informal mixed methods evaluation to look at adoption/acceptance by health workers, implementation and maintenance challenges, and data quality and usefulness. Observation of staff and six unstructured staff interviews were carried out by a member of the implementation team with experience of implementing and evaluating e-health innovations, and were recorded in note form. Six semi-structured interviews with key project stakeholders (Google, MSF, Harvard) were carried out using a topic guide by an MSF administrator with experience in qualitative research, who recorded and transcribed the interviews. Project documentation from September 2014 to April 2015, including situation reports, vision and requirements documents, were included as an additional data source. Thematic analysis of project documentation, observations and field interview notes, and stakeholder interview transcripts was performed by the administrator supported by a senior team member with experience in health-care evaluation.

Data from the tablets were analysed via a semi-automated match of record “pairs” (clinician-patient interactions where a record for the same interaction existed in both the tablet dataset and the paper→Excel dataset). For data items in both tablet and Excel datasets (temperature and symptoms), the simple data (raw data rather than OpenMRS codes) were extracted from the tablet dataset and multiple records for interactions within 30 minutes were manually merged. If a symptom was recorded as present in any record in a set of multiple records within 30 minutes, the symptom was marked as present. The Excel dataset was checked and corrected against scanned paper records. The aligned tablet and cleaned Excel datasets were combined into a single list, sorted by patient and date/time of interaction, such that the single daily record in Excel was paired with the first tablet record that day. This is a potential limitation, as it assumes that no observations would be recorded on the tablet prior to the principle morning clinical encounter, at which the single daily paper record was made. The record pairs were then recoded to collapse graded symptoms in the tablet record (e.g. 24 hour count of diarrhoea or vomiting episodes, extent of weakness) to only symptom present or absent, and some symptoms combined into a single symptom variable (e.g. abdominal pain was a single variable on paper but recorded separately for upper and lower on the tablet, so a combined variable was used for comparison). Temperature was recoded to <37.5°C or >=37.5°C, and also matched using precise value. Kappa coefficients were calculated using STATA v14.2 (StataCorp, College Station, Texas USA) for each symptom and for all records with an entry in both sources. Observations of vital signs were not part of the Excel dataset, but the presence of these items in the scanned paper records was documented and compared to whether these signs were also recorded in the tablet record. The quality of the scanned paper charts was subjectively assessed.

Ethics

This evaluation met the criteria of the MSF Ethics Review Board for exemption from ethics review. The product was discussed with and demonstrated to health authorities prior to implementation, who were supportive of this innovation and did not suggest additional ethical scrutiny.

Results/outcomes

The time taken to deliver the product was substantially more than that anticipated by MSF (7 months, as opposed to 6 weeks). The initial specification was largely but not fully delivered; a patient localising feature (radio-frequency identification tags for patients and a network of readers) was developed but never completed, as the declining patient numbers reduced the potential value of this feature. In addition, exported data require significant work to clean and analyse outside of the application interface. The database was not configured to create a single record for a patient encounter if the user moved in and out of the record, resulting in up to five partial records within a 10 minute period which all related to a single encounter, and thus required to be merged into a single record of that encounter. Records were then exported with content and order based on OpenMRS coding, resulting in 3 data fields per data element and an order that was non-intuitive and would require further development for the product to be usable in field conditions.

Due to the hasty decommissioning of the Google team at the end of the project, there was insufficient time for a comprehensive handover (from Google to MSF) of information and understanding required to operate and maintain the software and hardware that had been developed. In the case of the software and key hardware, MSF was able to hire an ex-Google engineer who worked on this project to further develop the software and complete the process of handover to MSF. However some parts of the hardware (e.g. polycarbonate casing) were deemed too expensive to warrant further investment by MSF, so little attempt was made to obtain the knowledge necessary to produce these.

Deployment of the tablet coincided with a dramatic drop in patient numbers. As a result, the full prototype had little impact on patient care.

The final product deployed in Magburaka was well received by medical staff, some of whom had no previous experience with computers or touch-screen devices. Staff described the system as intuitive and reliable for data entry and visualisation, even when power and internet connections were interrupted. Medical staff valued the ability to access more detailed, real-time, clinical data.

Tablets remained functional after frequent dipping in chlorine disinfectant, and the system remained active for periods of up to 24 hours without electricity.

83 record “pairs” for 33 patients with 22 data items (temperature and symptoms) per pair were available to be analysed.

The overall Kappa coefficient for agreement between both sources was 0.62, but reduced to 0.59 when rare bleeding symptoms were excluded, indicating moderate to good agreement. For individual symptoms it the Kappa coefficient ranged from 0.29 to 0.86, with chest pain, diarrhoea and sore throat having only fair agreement, but precise temperature very good agreement. The number of pairs with data available in both sources per symptom ranged from 78–81 of the 83 record pairs. Most patients were admitted for a short period and symptoms were recorded on paper only once per day, so 30/33 (91%) had 1–3 record pairs. One patient had 19 record pairs, but if all record pairs beyond 4 were excluded, the Kappa coefficient was still 0.59.

The tablet contained more unique patient encounters than the paper records – the paper chart usually showed one set of observations per day, while the tablet was used to record additional encounters that were (at best) otherwise written only in patient progress notes (which were not standardised and thus difficult for a data encoder to enter into a standardised database format). However, it is unknown if this recording would still occur if the EMC were very busy.

The tablet contained data fields for vital signs (BP, HR, RR) that were not always recorded on the paper vital signs charts, but rather only on the free-text clinician progress notes (which had not been entered into the Excel database at the time of matching). At least one of these vital signs was entered in the tablet for about 40% of 204 interactions, usually matching the paper record, and sometimes providing data that did not exist in the paper record. In a few instances (11/85 interaction pairs), one or more vital signs in the paper record were not in the tablet dataset.

The paper charts were sometimes hard to read, leading to errors in the Excel database such as a temperature recorded in the paper record that could be read as either 39.2 or 34.2 (the tablet record showed 34.2). In addition, there was variable use in the symptoms chart of symbols such as yes, y, no, n, ✓, ✘, -, |, ?. About 20% of paper-recorded encounters had some ambiguity, including no or only partial zero reporting (that is, specifying that symptoms were absent, rather than an assumption that any symptom not marked as present is absent as opposed to not assessed).

Interviews with field staff and key stakeholders revealed these common themes around challenges faced in the project.

Lack of agreed vision from start: The team’s haste to deploy something quickly that was “just enough” meant that insufficient time was taken to establish internal (MSF) project vision. Instead, the project team adopted a more perfection-driven Google vision (which extended the timeframe), and the user (MSF) questioned only late why something so complex had been made that was potentially useful in the long term, but not in this outbreak.

No explicit approach to project management: MSF project management structure, including governance and an organogram, was defined only when the project started to falter. As a result, the software developers were never sure who was the MSF lead, or who to go to when things went wrong, so at times asked the wrong question of the wrong person and lost focus on the minimum viable product requirements. At a late stage, efforts were made to develop an evaluation plan, but the numbers of cases dropped very substantially soon after.

Business pressures: the pressure to start meant that reflection on team composition happened too late. For example, the team approached numerous medical people with different backgrounds, experiences, and expectations for user input. Since the epidemic evolved rapidly, this focus on “the last medic returning” as the optimal source of information meant that the user stories became rapidly out of date. A consistent medical focal person was appointed to the team to provide ongoing guidance, but this was done too late. Likewise, Google’s need to move its developers onto other projects meant that there was insufficient time for effective knowledge transfer to a technologically-naïve partner.

Lack of understanding between the humanitarian and technology fields: MSF expected to be able to describe what was needed quickly and then leave the software developers to deliver, without realising the time commitment required from them to enable the developers to make the right product. In parallel, the developers expected that health programmes could be measured in terms of number of patients accessed, and struggled to incorporate a health outcome perspective in their objectives and notion of impact.

Discussion

The tablet facilitated more frequent and slightly more detailed data than the fence→ paper→Excel database routine, and there was high consistency between the results from the two systems. Given that the tablet data record is essentially real-time (in terms of data entry and opportunity for correction), it is likely to be the more accurate record. This assumption requires validation, but there is anecdotal evidence of the errors and uncertainties of the non-electronic data system from several EMCs.

To our knowledge, this is the first time a portable real-time electronic medical record (EMR) has been developed that specifically meets the needs of an extreme biohazard environment in a resource-limited setting with erratic power and internet supply. A client-server architecture normally necessitates a reliable, high-bandwidth internet connection, or a local server with very reliable electrical energy, IT support, and the availability of specialized parts. These requirements make a client-server architecture difficult to implement in rural sub-Saharan Africa.

MSF has gained experience and understanding of the process of technological innovation, including the limits and challenges of deploying new technology and working with technology partners [6]. The positive collaboration between MSF and Google has sparked interest in the potential for humanitarian-technology collaborations [7]. Successful deployment of reliable client-server architecture in a rural sub-Saharan African environment represents a proof of concept, such that it is already being deployed by other agencies (The Wellbody Alliance, in collaboration with Harvard Medical School).

However, there were major challenges with the project. Due to the length of time to deliver the product, in the context of an evolving Ebola outbreak, a product was delivered too late to have an impact on patient care or efficiency of EMC management. Lack of knowledge transfer meant that MSF had a product that they did not fully know how to support or adapt for future use.

The challenges we describe are echoed throughout the literature on humanitarian and technological innovation, such as best practice principles for humanitarian innovation defined by ALNAP and lessons learnt from IT innovation as outlined in the Chaos Manifesto [8, 9]. Our experience has led us to identify some concrete lessons specific to humanitarian-technology collaborative projects. Most importantly, time and effort is required to bridge differences in organisational culture between the technology and humanitarian worlds. This investment is essential for establishing a shared vision on deliverables, urgency, and ownership of product. To guide this process, there is a need for a framework for innovation in humanitarian emergencies. The overwhelming priority for medical teams is immediate delivery of care, and innovation projects must fit around this reality; careful consideration must be given to whether there are sufficient resources to deliver the project. Therefore, emergency innovation projects need to be agile, iterative, and free from heavy project management processes. Yet if we are serious about innovation, we need to push the boundaries to ensure it has impact not only on our current patients, but also for future patients. We have outlined the key components of such a framework in Panel 4. This framework should be used as a flexible and practical tool; we caution against adapting systems that could lead to stifling of innovation, especially in complex and challenging environments where fast solutions are needed.

Panel 4.

A framework for humanitarian-technology innovation projects

• Problem statement and project vision, with deliverables. • High-level requirements for the solution with early agreement on the minimum viable product. Users should be involved in defining requirements, and for each subsequent step. • Implementation requirements: clarifying who is involved and what they are responsible for; ensure they all have a common understanding of terms such as impact, and that the right parties are involved in decisions. • Implementation planning with timeline for key milestones (even if this is likely to evolve): planning, prototyping, piloting, roll out. Ensuring the time factored in for input and delivery is feasible. • Evaluation plan with expected impact and key performance indicators against which to measure success. • Communication and change management plan: how and who to handle internal updates and external communication; who needs to be consulted and who needs to be informed. • Governance: expert steering committee (or equivalent) which validates each of the above milestones; a mechanism for ethical oversight; clarity about who owns the IP that results.

An MSF team is in the process of adapting the software code-base of the Ebola data management solution to develop a generic emergency EMR that can be rapidly modified for different types of emergency. Our vision is to build a modifiable Open-MRS backbone for which new disease-specific apps can be developed within 5 days by a non-coder. The first ‘test-case’ is nutrition, for which we are working on apps for inpatient and ambulatory therapeutic feeding centres. In this project, having learnt from our experience with the Ebola EMR, we have taken the time to apply our framework ( Panel 4) from the start.

A complete kit of open-source software, hardware, and documentation for the Ebola EMR is available on-line and can be downloaded for free; the code is also freely available for developers to use [5]. Several of the developers involved have formed an open-source community that is supporting this code, and can help develop the software for new-use cases [10]. Harvard Medical School, MSF, and Google are all represented in this consortium. Harvard Medical School have carried out successful deployments of the hardware, which they have combined with new software for community surveillance, triage, and inpatient care. The consortium is focused upon new use-cases, the economics of the solution and driving economies of scale, the incorporation of additional software and hardware toolsets, and the enabling of clinical research during outbreak settings.

Conclusion

The fundamental innovation of our project in the end, was not the new technology that was developed, but rather the adaptation of existing technology so that it would work in an environment that is generally hostile for complex systems. We hope that the lessons learned and the tools developed in this collaboration will help others involved in innovation in humanitarian crises to gain the balance between speed, impact, and sustainability.

Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2017 Jobanputra K et al.

Data are deposited in a secure server in MSF Operational Centre Amsterdam and are available via a managed access process under the terms of the MSF Data Sharing Policy [11]. For contact details and explanation of process please visit http://fieldresearch.msf.org/msf/handle/10144/306501 or email data.sharing@msf.org.

We thank the authors for their clarification of all the points that we have queried and for their revisions of the text. We are content to approve the paper fully now and congratulate the authors on an interesting contribution to the scientific literature.

We note that the authors have used and interpreted the Kappa statistic as 'moderate to good' in the abstract and results, but called it 'high' consistency in the discussion. We think this apparent discrepancy is acceptable as for some elements, e.g. temperature the kappa statistic was very good.

We provide below a reference for the interpretation of the Kappa statistic for those who might require it. [1]

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thank you for giving us the opportunity to review the revised article and explanations from the authors. In general they have responded and given clarifications either in the revised article, or the covering letter, and sometimes both. However, there are a few outstanding points that have not been addressed and we would like to see responses to these before we would be ready to amend the current status of approved with reservations.

Our outstanding three comments are detailed below.

The authors have added a para about exporting data from the red zone, but it would still be good to have some simple text explaining exactly what staff needed to do to transfer the data, to help those readers who find ‘servers’ a technical term. We are still a bit puzzled about the reference to USB sticks for backup and updating and whether that refers only to outside the red zone back up, or internally.

As mentioned in our previous referee report:

“The results are generally clearly described, and suggest that the tablet may be at least as good as paper records, though the analysis currently does not appear to take into account the role of chance, and a proportion of records (17% ?) had to be excluded. The qualitative study is quite weak. It appears that different data were collected systematically using the two different systems, so only some of the data were collected using both methods.

Further comments on the quantitative study are:

If we understand correctly the agreement between the two methods was calculated by simple percentage agreement: would it be possible to perform a Kappa test to take into account chance?

What was the range of interaction pairs per patient? Was any difference seen in the recording by either method related to the severity of the patient’s illness, or the number of people admitted at the time?

Is it possible to state the percentage of missing data overall and/or by recording method and comment on the implications for the agreement results? On rough calculation overall it seems to be 17% (204-(85x2) = 34, 34/204) missing data in one or other system.

The agreement between data sources ranged between 69 and 95%. This is rated as ‘high consistency’ but I was not sure on what basis. It was not clear to me, which collection method was felt to be the gold standard, or even if this was determined before the comparison. I sense from the discussion, the tablet was thought to perform better than the paper method, but I am not convinced about this from the results.” [1]

These points are not addressed either in response or text – and the reference to high consistency remains (1 st para discussion)

Similarly, this point has also not been addressed:

“The authors claim that this is the first such device to be developed. How did they search for other evidence of similar projects? I heard anecdotally of other projects, but am not aware whether they reached fruition, or have ever been published in any form. It would be good to have a thorough systematic search to establish what else has been developed.” [1]

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The authors would like to thank the reviewer for the comments in their response letter, and have endeavoured to address the outstanding issues raised:

The authors have added a para about exporting data from the red zone, but it would still be good to have some simple text explaining exactly what staff needed to do to transfer the data, to help those readers who find ‘servers’ a technical term. We are still a bit puzzled about the reference to USB sticks for backup and updating and whether that refers only to outside the red zone back up, or internally.

We have addressed this in the text of the article (v3)

If we understand correctly the agreement between the two methods was calculated by simple percentage agreement: would it be possible to perform a Kappa test to take into account chance? What was the range of interaction pairs per patient? Was any difference seen in the recording by either method related to the severity of the patient’s illness, or the number of people admitted at the time? Is it possible to state the percentage of missing data overall and/or by recording method and comment on the implications for the agreement results? On rough calculation overall it seems to be 17% (204-(85x2) = 34, 34/204) missing data in one or other system. The agreement between data sources ranged between 69 and 95%. This is rated as ‘high consistency’ but I was not sure on what basis. It was not clear to me, which collection method was felt to be the gold standard, or even if this was determined before the comparison. I sense from the discussion, the tablet was thought to perform better than the paper method, but I am not convinced about this from the results.”

The methods have been edited and percentage agreement results have been replaced with Kappa coefficients (v3). Data on missing results and the number of records per patient has been added.  Relatively few patients were admitted during the period, and bleeding symptoms were rare, so attempting to account for these issues is unlikely to be valid with this dataset. Neither method was a gold standard.

Other comparable products: “The authors claim that this is the first such device to be developed. How did they search for other evidence of similar projects? I heard anecdotally of other projects, but am not aware whether they reached fruition, or have ever been published in any form. It would be good to have a thorough systematic search to establish what else has been developed.”

At the time of writing, we searched PubMed and Google using various search terms including “EMR”, “data collection” and “Ebola”, and found no description of similar products. Through our informal networks, we found out about two data collection devices that were developed for the ebola epidemic, but neither had EMR functionality. After submission of this article, we learnt about an ebola EMR developed by ThoughtWorks, but without the server infrastructure. Although a systematic review is beyond the scope of this article, we have add a reference to this other product in the text (v3).

Research and development of technology in the humanitarian arena is a rapidly growing field that encompasses many specialities. There are international working groups, peer reviewed journals and conferences dedicated to this field. This paper is a valuable addition to the body of evidence openly available to the academic, technician and aid worker. My review of this paper relies on my field experience of working in the humanitarian setting, particularly though the West African Ebola outbreak of 2014 till 2016.

This paper describes the process, implementation and challenges of introducing a novel way of recording patient data electronically in a high biohazard environment. The aim being to improve data collection and the efficiency of patient interactions and communication.

It is bold and brave of the authors to openly state the errors made through the design and implementation of this project. While there is substantial discussion around the lessons identified from their experience, there could have also been more highlighting some of the possible successes, not least that this project was rolled out at a time of unprecedented pressures on the organisation responsible, Médecins Sans Frontières (MSF).

In their peer review Whitworth and Bower (2016) have critically appraised the methodology and data analysis described in this paper. I am therefore not going to repeat what has already been expertly written, focussing instead on the patient care and field worker practicalities of this tool.

Involvement from the field is raised on several occasions in the article, however it remains unclear to what lengths the designers and implementers used the returnees’ knowledge on the field realities to make this tool suitable for the task in-hand, beyond “informal discussions”. All volunteers returning from Ebola missions were routinely interviewed in the European headquarters by the organisation, to what extent this was used as an opportunity by the developing team is unclear, if not utilised was this a missed opportunity that could be added to their list of “lessons identified”. It would be useful to know how many returnees were interviewed, the style of interview used and how the information gained guided the designers. Furthermore, given that all returnees were debriefed at head office it would be worth stating if the returnees were all systematically questioned in a standardised manner, or if specific persons were chosen and how this choice was made.

The field staff who used both the alpha version and full prototype were also interviewed, it would be informative to the reader to understand how many staff were involved in these interviews, whether they were expatriate or local staff and what their specific job role was. The experience of a European doctor may be different to that of a local nurse or health care assistant.

The authors rightly discuss the difficulty of working in the full Personal Protective Equipment (PPE) that is necessary to remain safe inside the “high risk” area of the Ebola Management Centre (EMC), it would be interesting to gain a deeper insight into how much this was factored into the overall program design. For example, visual difficulties and clumsiness were often described as part of the challenge of working in PPE. Were large buttons or colour coding on the tablet part of the design to assist the healthcare worker overcome these difficulties?

During very busy times at the peak of the outbreak a numbered system was used to quickly identify the patients with the most immediate needs, a score from 1 (very well) to 5 (critically ill) was assigned to each patient. It would have been informative to know if this or other field-designed approaches were considered or incorporated into the program. Whilst the focus of the tablet appeared to be for transfer of medical information, there is much communication that must also be made between the EMC workers. Identification and location of deceased patients must be efficiently passed to hygiene teams, or safety concerns to logistical teams, was inter-disciplinary communication considered as part of this program?

Similarly, whilst communication on patient symptomatology was traditionally shouted across the fence and then transcribed into folders outside the “high-risk” area, treatment decisions also needed to be remembered or shouted over. It is not entirely clear how much the tablet was utilised for recording treatments given, for example intravenous fluid administration, hygiene procedures or palliative care medications. This would be valuable from a patient care perspective and likely reduce prescribing errors, missing or duplication of doses. Retrospectively this could also have added to data on what treatment and care was being given and how often. Commonly a generic recipe of medications was administered to all patients from admission into the EMC, this could have therefore been pre-programmed into the tablets as an electronic prescription.

The method of shouting information across the fence in the EMC, whilst this was often a part of high to low risk communication, was not exclusive. Digital photography that could later be transcribed was often used, particularly during times of high activity or when transferring patients from one location to another. Similarly, only those patients that were bedbound or too unwell to speak with healthcare workers from across the fence required examination and questioning in full PPE. Hence for many inpatient interactions the “high-risk” tablet would not be required. Explanation of how these other interactions were incorporated into the data gathering would be useful, particularly for the completeness of the information generated. The benefits of being able to talk with a patient across the fence, where the clinician was recognisable (no PPE needed) and there were no time constraints due to heat stress, meant that this was a desirable way of working whenever possible. However, it would not negate the use of the tablet as information could still be recorded on the hand-held device.

The stated aim of making the most effective use of the limited time in PPE to treat patients has not been quantified. Did the tablet assist the healthcare worker in treating the patients, did the number of patients receiving clinical review or treatment change as a result of the tablet or were the health workers able to achieve the same quality of work in a shorter period of time? I think these would have been important questions to ask, rather than the focus being on the ease of data gathering and transfer, it could have been on quality and availability of patient care, perhaps including patient outcomes.

I found the concept of using radio-frequency identification tags particularly interesting, this would have had great value both for finding and identifying patients for examination, but also importantly for identification and tracking of the deceased. Perhaps because this intervention came during the tail-end of the outbreak there is little discussion of how the tablet could be used for the management of deceased patients, though this was a major issue particularly during the peak of the outbreak when there was a high mortality rate. Unfortunately, the radio-frequency tag aspect of this project did not come to fruition, however a discussion on how this could be moved forward would be worthwhile.

The potential benefits in the efficiency and quality of data collection gained through the use of the electronic tablet system are clear. How this translated into improving patient care seems less apparent. The balance between patient benefit and data for the sake of research and publication remained controversial throughout the Ebola outbreak. Whilst I don’t question the motives of the team behind this innovation, it would be interesting to know how they managed the balance of this innovation being primarily for patient rather than organisational benefit.

The honesty of the authors and implementing team is to be applauded, though deeper discussion on how to move forward would have been of value. This venture came at a high price, and though it was not the success they had hoped for it is certainly not an entirely failed project either. Whilst the authors have identified lessons, it is not clear how they have been learnt. The conclusion would benefit a wider discussion of what next, where this technology can continue to be of use and developed to meet its ultimate aim: saving lives.

The technology is presented as being developed for the specific situation of the EMC, however it would be easily transferable to many humanitarian settings where large numbers of patients or biohazard risks limit the accuracy and efficiency of traditional clinical record keeping. Measles, yellow fever, hepatitis E, cholera and Lassa fever outbreaks would all be well suited to the continued use and development of a potentially very useful field tool. The authors do refer to their vision of the future, and the plan to test a modifiable version of this technology in a nutrition project, a greater emphasis on this vison and how the lessons identified will be learned for the future could have been of use.

As previously stated, humanitarian technology is a rapidly growing and developing phenomenon. The challenges, lessons and future plans identified by the authors are a useful reminder of how best intentions can go askew without the correct planning and inter-agency communication. These lessons are unlikely be unique to this situation, being broadly adaptable. The honesty of the authors on the difficulties in collaborating with Google is commendable, the next step though is who and how they will work with next. MSF has an opportunity to work within and alongside others in the humanitarian technology field, perhaps partners with the experience to assist them in avoiding these common pitfalls again.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The authors would like to thank the reviewer for their thorough and constructive critique of this manuscript. We have organised our response by theme:

User involvement: extensive user consultation was carried out, as well as user testing with national staff, international staff returning from the field and clinicians who had worked in comparable settings. Several design features addressed the challenges of working in PPE, including large buttons. We have clarified this in the text.

RFID: Rapid identification of patients, including the sickest patients, was originally part of the original scope of the project; RFID identification would have helped address this. This was however dropped from the scope as the patient numbers declined and the priority shifted towards transfer of patient data outside the high risk area.

Treatment module: a module to record treatments administered was planned and worked on, but not completed and put on hold when it was clear that the outbreak was reducing in scale and thus the opportunity to field test the tablet was highly time sensitive.

Using these tools for research: The impetus to develop the EMR was to support improved patient care, including through more efficient use of staff time within the high-risk zone. There were no plans to use the tablet-based EMR for research, although this would be a potential application of this technology.

The authors present a honest evaluation of an operational research project to innovate a new tool for recording, visualising and sharing medical data in response to the challenging needs of a high risk infectious situation in a humanitarian emergency, namely the outbreak of Ebola in West Africa 2014-15.  This is an interesting and valuable paper about developing a clinical information and patient management system based on tablet computers that can be used in Ebola Treatment Centres and similar high risk environments. It is mainly a narrative paper with little in the way of quantitative results. The lessons learned are fairly generic to any project management challenge, and go beyond those of innovation in humanitarian settings.

They describe the process of developing a tablet and new software to respond to the complex requirements of high bio-hazard infection control, and compare this tool with standard practice in this setting. They note the urgency of the project, which was intended to improve clinical care, and by implication outcomes, for patients by helping medical staff make more efficient use of the limited time they could spend in the red zone due to the constraints of personal protective equipment.

The title is appropriate and the abstract is generally clear, although I felt that the description of the comparison of tablet and paper-based data was not clear. It was not just tablet data that were analysed, and what is meant by ‘record “pairs”’ is not clear just from reading the abstract.

The background and methods are mostly clear and appropriate. I am not an expert on computer hardware or software, so cannot comment on the appropriateness of the systems selected. The researchers used a mixed methods approach appropriate for the different aspects of the project they wished to evaluate: namely need identification, buy-in, implementation, tool functionality and maintenance, and a measurement of data quality. They have identified key issues that delayed the process of development and the limited testing of the tool they were able to achieve.

I am astonished at the cost of development of the system, which seems to have been driven largely by the salary costs and person-hours involved, as well as producing customised hardware.

It was not clear to me quite how the data collected in the high risk zone were exported for further analysis and storage. Was this by disinfecting the tablets in chlorine and physically removing them from the high risk zone, or was it through connection some form of local area network that allowed the data collected in the high risk zone to be electronically transmitted outside? This could be made clearer.

I wondered about issues of confidentiality of patient data being transferred from Sierra Leone to the MSF headquarters in Europe (panel 2). How were these addressed, approved and overseen?

I was surprised that no data were collected on clinical management according to panel 3. This seems like a missed opportunity, as further research into optimal management of cases of ebola is still required. There is comment about this at the end of the discussion (page7), but was this an oversight when the system was being developed?

Did this study undergo any ethical scrutiny in Sierra Leone? I would imagine that it should have done, but this is not mentioned.

Methods para 6:

Suggest rephrasing the second sentence to make it clear that this part of the paragraph refers to data extraction for the tablet, and the latter to the paper/excel database.

Is it a correct understanding that only the first tablet entry of the day was taken into account? If so, could you mention why it was not possible to take account of subsequent entries given the relatively small number of records under analysis

Did practitioners entering data in the high risk zone review their own entries once out? Our question refers to the comment in the first para of the discussion about opportunity for correction. It might be useful to include a paragraph on the procedure for using the tablet.

The results are very interesting, and it is good to see an honest description of the problems encountered with this project. Their results highlight areas of tension in the process and detail the successes (a functioning tool with good data capture) and failures (implementation too late to have impact on patient care, and lacking one of the initial requirement – a way to find patients in a large facility) of the project. From both they draw out lessons for future innovation between humanitarian and technological enterprises.  They also detail the technical specifications and overview costings for the tool.

I was not clear what was meant by ‘transfer of knowledge of the hardware from Google to MSF did not occur’ (page 5). Further on in the discussion section of the paper there is a comment about the transfer of knowledge, but this appears to be related to the software rather than the hardware. Indeed, it seems that this problem has been overcome and the software has been adapted for use in other emergencies (page 6).

I would have appreciated more information about the ‘significant work needed to clean and analyse the exported data’ from the tablet. Why was that necessary?

The results are generally clearly described, and suggest that the tablet may be at least as good as paper records, though the analysis currently does not appear to take into account the role of chance, and a proportion of records (17% ?) had to be excluded. The qualitative study is quite weak. It appears that different data were collected systematically using the two different systems, so only some of the data were collected using both methods.  Further comments on the quantitative study are:

If we understand correctly the agreement between the two methods was calculated by simple percentage agreement: would it be possible to perform a Kappa test to take into account chance?

What was the range of interaction pairs per patient?  Was any difference seen in the recording by either method related to the severity of the patient’s illness, or the number of people admitted at the time?

Is it possible to state the percentage of missing data overall and/or by recording method and comment on the implications for the agreement results? On rough calculation overall it seems to be 17% (204-(85x2) = 34, 34/204) missing data in one or other system.

The agreement between data sources ranged between 69 and 95%. This is rated as ‘high consistency’ but I was not sure on what basis. It was not clear to me, which collection method was felt to be the gold standard, or even if this was determined before the comparison. I sense from the discussion, the tablet was thought to perform better than the paper method, but I am not convinced about this from the results.

There was clearly a tension between MSF wanting a ‘good-enough‘ product, and Google being perfection-driven. I suspect this was not just a question of communication, there is likely to be some underlying differences in corporate culture, which might be hard to bridge even with all the time in the world.

What is meant by ’Google’s haste to move onto the next project’ (page 6). Was this ebola-related, or simply reflecting that this department at Google was busy with a range of different projects?

The discussion was well written and interesting. The authors claim that this is the first such device to be developed. How did they search for other evidence of similar projects? I heard anecdotally of other projects, but am not aware whether they reached fruition, or have ever been published in any form. It would be good to have a thorough systematic search to establish what else has been developed.

Is it correct to assume that the more frequent and detailed data available with the tablet related to the recording of vital signs, which were not entered from the paper record at the time of analysis?

Could you comment on the value gained from the additional encounters recorded in the tablet and whether users found recording real-time information was clinically useful?

It is not clear to what extent this system can be made available to others. The reference to the consortium being established is to a website that is being prepared.

What data are available from MSF in Amsterdam? Are they data related to the development of this tablet computer based system or the patient data from this Ebola Treatment Centre? This is not clear to me.

In our view, however, the authors present a well-targeted piece of operational research undertaken in challenging circumstances with transparent insights into where the development process fell short. They make it clear that their results are limited, but provide a foundation for moving forward.

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

The authors would like to thank the reviewer for their thorough and constructive critique of this manuscript. We have organised our response by theme:

Record pairs: All interactions continued to be captured on paper during the pilot of the tablet-based EMR, and the comparison included all those clinical entries that had been captured by both means (referred to as “record pairs’.

Data transfer: Data was transferred wirelessly to a server outside the high risk zone via a secure local area network, which indeed was one of the key advantages of this technology for Ebola Management Centres.

Confidentiality: Tablet data included only an ID code and approximate date of birth based on current age, so although data was linked to an individual, the individual was not identifiable.

Treatment module: A module to record treatments administered was planned and worked on, but not completed and put on hold when it was clear that the outbreak was reducing in scale and thus the opportunity to field test the tablet was highly time sensitive.

National oversight and ethics approval: This was not a study as such, but an ad hoc evaluation of an operational innovation that was introduced at the peak of the emergency. The evaluation had no impact on patient experience, since it did not involve subjecting patients to any process additional to that which they would already be undergoing; no additional patient data was collected for this evaluation. Interviews with staff were informal and focused on experience of using the tablets. As such, it met the MSF ERB criteria for exemption from formal ethics review and was approved by the MSF medical director. The tablet-based EMR was discussed with and demonstrated to health authorities prior to implementation, who were supportive of this innovation and did not suggest additional ethical scrutiny.

Data checking and validation: Practitioners were able to review the data at any time, including within the high-risk zone, so were able to check for consistency. Since the implementation team regularly checked for consistency between data entered into the tablet, and data on the server, it was not deemed necessary to ask the clinicians to do this themselves.

Why was knowledge transfer not successful?: Due to the hasty decommissioning of the team, there was insufficient time for a comprehensive handover (from Google to MSF) of information and understanding required to operate and maintain the software and hardware that had been developed. In the case of the software and key hardware, MSF was able to hire an ex-Google engineer who worked on this project to further develop the software and complete the process of handover to MSF. However some parts of the hardware (e.g. polycarbonate casing) were deemed too expensive to warrant further investment by MSF, so little attempt was made to obtain the knowledge necessary to produce these.

What was the 'significant work needed to clean the data'? The database was not configured to create a single record for a patient encounter if the user moved in and out of the record, resulting in upto five partial records within a 10 minute period which all related to a single encounter, and thus required to be merged into a single record of that encounter. In addition, new users entered a variety of practice data which was retained in the database, and needed to be identified and removed. Finally, records were exported with content and order based on OpenMRS coding, resulting in 3 data fields per data element and an order that was not linked to the logical grouping and order on the tablet forms.

How reliable was the comparison of tablet and paper data?: Of 204 tablet records, 119 did not have an equivalent record in the Excel database because only a single set of observations were recorded each day on paper whereas on the tablet additional observations were sometimes recorded later the same day. Therefore only one record pair per day was possible. Of 111 encounters recorded on paper, 28 were not matched to a tablet record.

The median number of record pairs per patient was 2.0, with a maximum of 19 for a patient who was admitted for 19 days. There were relatively few patients admitted during the implementation, many of whom were later confirmed as not having Ebola, so while the influence of severity of illness or patient load on the data quality are interesting in theory, it cannot be validly assessed.

To what extent can this system be made available to others? The software is available on-line and can be down-loaded for free; the code is also freely available for developers to use. Several of the developers involved have formed an open-source community that is supporting this code, which has replaced the consortium in this regard.

Panel 3.

Core components of the final Ebola data management solution.

Hardware Sony Android tablets (Sony Experia Z2 running Android 4.4.2 [KitKat]) in a custom-built polycarbonate casing containing a radio- frequency identification reader and rechargeable batteries, custom-built by Thinkify. The server (Edison server running Yocto Linux + Debian GNU/Linux) and local area network were based on a federated Edison server structure (housed in individual waterproof polycarbonate boxes developed by Thinkify) powered by rechargeable batteries linked to mains electricity. Software The tablet software is a custom-built Android app. The server-side application (back end) is OpenMRS (platform v 1.10.1) built on a Linux Operating System running on a low-energy server (Intel Edison), with a custom-built module for data import from the Android app (imported data is anonymised but linkable to an individual if required, using a patient ID code and approximate date of birth based on current age). Data content The following data fields were hard-coded:        • Identifiers: patient identification number, approximate date of birth, encounter time and date, date of EMC admission.        • Bleeding: blood in stool (black/red), red eyes, bleeding from eyes, nosebleed, haemoptysis, haematemesis, vaginal bleeding, haematuria, oral bleeding, bleeding NOS.        • Other symptoms: chest pain, muscle/joint pain, difficulty breathing, hiccups, heartburn, headache, cough, back pain, sore throat, abdominal pain (upper/lower), nausea.        • Signs: consciousness, hydration, diarrhoea (episodes/24 hours), mobility, vomiting (episodes/24 hours), blood pressure, heart rate, temperature (°C), weight (kg), weakness, respiratory rate, pregnant, diet, IV access present, pain level.        • Laboratory test results: Ebola NP gene PCR CT value, Ebola L gene PCR CT value.

EMC=Ebola management centre. NOS=not otherwise specified. IV=intravenous.