Pilot Audit of Trichiasis Surgery Outcomes Using a Mobile App in the Republic of Chad.

PURPOSE: The purpose of the study was to assess the feasibility of a mobile data collection app for use in trichiasis surgical audits in the Melfi and Mangalme districts of the Guera region of the Republic of Chad and to perform a cost analysis to determine if the auditing mechanism could be implemented nationally.
MATERIALS AND METHODS: Patients who underwent trichiasis surgery 6 months prior and who had follow-up 7-14 days after surgery were included in the study. Each surgeon had a sample of 20% of operated eyelids; nine surgeons with data for ≥20 eyelids were included. A trichiasis recurrence rate of ≥25% suggested that the surgeon needed retraining. Smartphones captured data using the data collection app, which transmitted data to an online server. Direct costs and supervision costs were collated and summed.
RESULTS: There were 916 eyelids operated on; 170 patients (269 eyelids, 29% follow-up rate) participated in the audit. Twenty participants (11.8%) had recurrence. The mean recurrence rate among surgeons was 8.3% (standard deviation: 0.07%; range: 0%-17.9%). None had a recurrence rate of ≥25%; thus, no retraining was necessary. The total cost of the audit was US$15,111.25 ($12,882.28 in direct costs and $2,228.97 in supervision costs).
CONCLUSIONS: The simple, easy-to-use, and low-cost mobile auditing mechanism is a practical solution for conducting surgical audits in remote and resource-limited settings and is undergoing national scale-up by the Chadian trachoma elimination program. Copyright:

Introduction

Trachoma causes 3% of global blindness.[1] Approximately 200 million people need treatment for trachoma, 7.2 million of whom have trachomatous trichiasis (TT), which – without proper surgical intervention – may result in vision loss.[1]

The tremendous backlog of unoperated TT is exacerbated by suboptimal rates of patient uptake of surgical services, which range from 18% to 66%.[2] Fear of surgery has been identified as a principal barrier to trichiasis surgery, which can stem from negative outcomes experienced by patients who were operated on for TT.[345] The TT recurrence rate is the qualitative outcome measure for trichiasis surgery.[5] Although the Surgery for Trichiasis, Antibiotics to Prevent Recurrence (STAR) trial in Ethiopia reported low recurrence rates of 10%–13% at 3 years,[67] high recurrence rates have been a concern of trachoma control programming.[45891011] Recurrence rates have been reported as high as 32% at 6 months in Gambia, 40% at 1 year in Gambia, and 62% at 3 years in Oman.[511] Recurrence can be either early due to surgical factors or late due to disease progression.[45] Multiple authors have reported that most recurrence occurred within 6 months of surgery,[412] most likely as a result of poor-quality surgery. Intersurgeon recurrence rates vary significantly from 0% to 83% despite the use of high-volume, trained surgeons.[451213] Even more alarming is that the number of surgeries with poor outcomes is increasing as surgical output increases.[1415]

In the wake of major surgical scale-up efforts to address the TT backlog, there is now heightened concern to improve surgical quality.[514] Unfortunately, trachoma control programming lacks systematic postoperative follow-up.[9] Consequently, surgeons (many of whom are nonphysicians in some countries) are not evaluated for their performance and surgical outcomes, and surgical programming lacks quality control. While trachoma control programs have focused on improving surgical techniques and on strengthening training via a surgical certification process, there is no standardized method to evaluate recurrence.[516] There is also a need to identify surgeons with poor surgical outcomes in need of retraining.[514] A standard mechanism to improve the quality of care in healthcare systems is an audit.[5] The use of a surgical audit may improve TT surgery outcomes and hence improve patient uptake,[51014] which would be instrumental in reducing the TT backlog.

In 2002, it was estimated that 10%–29.9% of the population of Mangalme and Melfi districts in the Guera region of Chad had trachoma.[17] An unpublished 2017 survey found that the sex- and age-adjusted TT prevalence was 1.0% in Mangalme and 0.25% in Melfi. Since 2014, a comprehensive trachoma control and prevention program has been implemented in Guera via Surgery, Antibiotics, Facial cleanliness, and Environmental improvements strategy.[18] With national trachoma control scale-up measures underway, the need to improve surgical programming quality was urgent, but, until now, no auditing mechanism has been implemented by the national trachoma elimination program.

In this study, the Chadian national trachoma elimination program tested the feasibility of a mobile surgical auditing method that aims to establish a quality control and improvement mechanism that can evaluate and improve TT surgical outcomes. The Organization1 = United Nations, the Organization 2 = International Rescue Committee, and the Organization3 = Harvard Humanitarian Initiative developed the Device = KoBo Collect app Organization 4 = KoBo Toolbox (Cambridge, MA, USA), a digital data platform for mobile survey that is based on the Open Data Kit, the standard for mobile data collection, is open source and can be downloaded for free on a smartphone using Android technology for mobile surveying.[19] This pilot study assessed the feasibility of the implementation of the Device app for its use in TT surgical audits in the Melfi and Mangalme health districts of Guera region of Chad and determined the programmatic cost of the mobile audit for future budgeting purposes (to determine if the mobile auditing mechanism could be implemented at the national level, based on the program's overall national operational budget).

Materials and Methods

This prospective, one-arm, pilot, feasibility study was carried out in 24 health districts of Mangalme and Melfi (population: 256,783 people) in Guera, Chad, in June 2017.[20] Given that TT surgical audit should be conducted from three to 6 months after surgery,[21] these districts were chosen because outreach campaigns took place in them 6 months before the study, they are relatively accessible (geographically), and they have lower violence and crime rates.

The Organization 5 = Organization for the Prevention of Blindness (Paris, France; referred to as OPC based on its abbreviation in French) implemented this study and supervised it along with the Organization 6 = National Program to Prevent Blindness (Chad; referred to as the PNLC based on its abbreviation in French). This study adhered to the Declaration of Helsinki and was approved by the Organization 7 = Global Trachoma Mapping Project (GTMP) Bioethics National Committee.

A 20% sample of operated eyelids from each surgeon was required for surgeon inclusion in the audit, which was decided by the Organization5 = OPC and the Organization8 = International Coalition for Trachoma Control to be an adequate and reliable threshold for data validation, provided that there were at least 20 eyelids examined per audited surgeon. Any cases of recurrence were considered as failures. Recurrence was defined as when at least one lash started to touch the globe, the patient had to pluck lashes to prevent rubbing, or repeat surgery was needed.[21] The posterior lamellar tarsal rotation (PLTR; the Trabut method) technique is used in surgical campaigns in Chad. The postoperative TT recurrence threshold rate following PLTR surgery was estimated at 15%, which was based on data for PLTR surgery in Ethiopia.[10] To determine the recurrence rate that would necessitate surgical retraining, the cut-off recurrence rate was doubled to 30%. However, to maximize surgical quality improvement, the threshold was arbitrarily reduced to 25%. Therefore, if a surgeon had a failure rate of 25% or greater, then retraining was indicated.

The patient population was selected from patients who participated in TT surgical outreach campaigns in Melfi in December 2016 and in Mangalme in January 2017, whose medical records indicated that they had postoperative follow-up from 7 to 14 days after surgery. The surveyors met with the heads of each health center to inform them of the surgical audit and to verify the names of the patients in each village. Participants provided their oral informed consent.

On June 10, 2017, an Organization5 = OPC trainer trained two survey teams of one grader and one recorder per team. The two recorders previously were recorders during the Organization7 = GTMP surveys. One grader was an ophthalmic technician, and the other was an ophthalmic nurse. The surveyors were trained on the study protocol, the Device = KoBo Collect and online survey form, and how to locate patients. Surveyors were also instructed to take clear, digital photographs with their smartphones, with one open eye only in each image.

The supervising team comprised one member of the Organization 6 = PNLC and two members of the Organization 5 = OPC; they ensured that data collection using the apps went smoothly and provided assistance to the survey teams as needed. On June 12, 2017, they were trained on how to use the Organization4 = KoBo Toolbox online platform.

In 2014, the Organization7 = GTMP provided study investigators with smartphones (Smartphone Model = BLU, Doral, FL, USA), on which the Device = KoBo Collect app was installed (the app requires Android 4.1 and up, can be installed on any computer or server, and is used with any modern browser). All data were collected remotely by surveyors using the Device = KoBo Collect app on their smartphones. The app captured user/smartphone device – specific data, Global Positioning System (GPS) location, and the following survey data:

Region

Surgeon name

Patient name

Patient sex

Patient age

Patient phone

Patient village

Location of the original surgical campaign

Date of the original surgical campaign

Eye operated on (left/right/both)

Whether or not the patient had TT recurrence in either eye or both (and if yes, which eye or both)

Digital photographs of the eyes affected with TT recurrence.

Mandatory data fields were programmed in the app to avoid missing data.

In case of technical difficulties and program malfunction, the Organization4 = KoBo Toolbox website offers a help center support portal at no charge, and the app itself includes remote, real-time technical assistance via messages with the technical support team.

The Device = KoBo Collect app was programmed so that once a questionnaire was completely filled in, it was automatically transmitted to the Organization4 = KoBo Collect server as soon as a network was available. The Organization4 = KoBo Toolbox online platform stores a photo library of recurrent cases and generates data reports and maps showing the distribution of patients' sex, surgeries distributed by the surgeon, operated eyes, recurrence, and the distribution of eyes with recurrences (right/left/both eyes). Data were downloaded from the Organization4 platform and copied and pasted into an Excel database for further analysis. Descriptive statistics were used for surgeon and patient characteristics and recurrence rates.

District authorities collected detailed costs of the study [Table 1], which were collated and summed by the Organization5 = OPC to determine budgeting for future audits, based on the global budget of the national trachoma elimination program.

Table 1

Total costs of pilot phase of surgical audit, including supervision costs

Expense Items	Cost Per Unit/Per Diem, US$	Number of units	Total cost, US$
Survey expenses
Per diem of local community guides, 1 per survey team, 10 days each	$17.62	20	$352.40
Rental of 2 vehicle for 18 days, including drivers and insurance	N/A	2	$4,758.23
Fuel for 31 days (10 days for 1 survey team and 21 days for other)	N/A	N/A	$3,425.19
Driver’s accidental insurance	$93.97	1	$93.97
Per diem of trainer	$17.62	7	$123.34
Per diem of 2 graders for 18 days	$52.87	36	$1,903.32
Per diem of recorders for 18 days	$44.06	36	$1,586.16
Camping gear for surveyors (bed, sheets, and two blankets)	N/A	3	$62.56
Internet subscription	N/A	1	$281.53
Tetracycline ointments for recurrent cases	$0.35	1,000	$350.00
Total survey expenses			$12,936.70
Supervision expenses
Accommodations and meals for 3 people for 10 nights	$37.01	30	$1,110.30
Driver’s per diem for 10 days	$24.67	10	$246.72
Per diem of local community guide for 1 day	$14.10	1	$14.10
Motorbike rental for 1 day	$26.44	1	$26.44
Fuel for 6 days	N/A	N/A	$770.30
Inner tubes and workforce	N/A	N/A	$61.67
Total supervision expenses			$2,229.53
Total project expenses			$15,166.23

N/A: Not available

Results

This study was conducted from June 13, 2017 to June 28, 2017. Fourteen senior ophthalmic technicians or senior ophthalmic nurses were screened for inclusion in the audit. Four surgeons were excluded, because their patient sample size was lower than 20%. One surgeon had only 2.4% of eyelids examined (1/41), one surgeon had 10.6% examined (5/47), one had 5.6% examined (2/36), and one surgeon had no eyelids examined in the audit (0/17). A fifth surgeon was excluded because his sample only included 10 eyelids. Therefore, patient data from nine surgeons were ultimately audited. Their demographic and clinical characteristics, as well as their surgical outcomes, are summarized in Table 2. The mean age of the surgeons was 36.7 years (standard deviation [SD]: 4.9; median; 37). The mean experience was 2.2 years (SD: 0.4; median: 2).

Table 2

Demographic and clinical characteristics and surgical outcomes of surgeons who completed the audit

Surgeon number	Age (years)	Experience, years	Number of operated eyelids	Number of surveyed eyelids (percentage of operated eyelids)	Successful outcomes, n	Recurrence, n (rate, percentage)
1	39	2	112	23 (20.5)	20	3 (13)
2	36	2	89	21 (23.6)	19	2 (9.5)
3	46	2	84	50 (59.5)	47	3 (6.0)
4	40	2	62	20 (32.3)	20	0 (0.0)
5	33	2	86	25 (29.1)	25	0 (0.0)
6	38	2	80	28 (35.0)	23	5 (17.9)
7	37	2	146	29 (19.9)	24	5 (17.2)
8	29	3	44	26 (59.1)	25	1 (3.8)
9	33	3	34	28 (82.4)	26	2 (7.1)

During the surgical outreach campaigns, there were 450 eyelids operated on in Melfi (298 patients) and 466 in Mangalme (289 patients) for a total of 916 eyelids. There were 170 patients surveyed in this study, 102 (60%) from Mangalme and 68 (40%) from Melfi. Twenty-three patients (3.9%) were lost to follow-up on days 7–14 and ineligible. Some patients were followed up, but their data were not accurately recorded, which made it impossible to locate them. Others were unable to participate, because the rainy season had already begun, making some roads and paths inaccessible. Other patients were unavailable, because they were working in the field no longer in the area.

The mean age of the participants was 56.2 years (SD: 18.7 years; range: 10–90 years), and 77% (n = 131) were women. Ninety-nine participants (58.2%) underwent bilateral trichiasis surgery, 39 (22.9%) had the right eye only operated on, and 32 (18.8%) had the left eye only operated on [Figure 1].

Figure 1

Distribution of operated eyes. When accessing the map via the Organization4 = KoBo Toolbox online, one can click on each colored dot to access complete participant data

Among the 170 participants surveyed, 269 operated eyelids were assessed [Table 2], which is a follow-up rate of 29%. Recurrence occurred in 20 participants (11.8%), of whom eight cases (4.7%) were in the right eye, seven cases (4.1%) were in the left eye, and five cases (2.9%) were in both eyes [Figure 2]. The recurrence rate was very similar among bilateral and unilateral cases. Among the bilateral cases of trichiasis, there were 11 cases of recurrence (11.1%), of which five were bilateral and six were unilateral. Among the unilateral cases of trichiasis, there were 9 cases of recurrence (12.7%). Table 2 summarizes surgical outcome data for the nine audited surgeons. The mean number of eyelids operated on per surgeon was 82 (SD: 34; range: 34–146). The mean sample was 40.2% (SD: 21.8%; range 20.5%–82.4%). The mean recurrence rate was 8.3% (SD: 0.07%; range: 0%–17.9%). None of the surgeons had a failure rate of 25% or greater.

Figure 2

Distribution of eyes with recurrence. When accessing the map via the Organization4 = KoBo Toolbox online, one can click on each colored dot to access complete participant data

The only patient data not captured by the audit were the participants' phone numbers, as many did not own any type of telephone, given the poverty level of the participants' communities in a country where 40% live below the poverty line[22] and in a region where 86.1% are rural.[23] The surgical audit cost US$15,166.23, including $12,936.70 in survey costs and $2,229.53 in supervision costs, with detailed expenses provided in Table 1. The global budget was $7.6 million for 2014–2019, so the audit only cost 0.2% of the global budget.

Discussion

Multiple auditing methodologies for trichiasis surgery have been suggested.[521] These include performing an audit as a part of supportive supervision by an experienced surgeon supervisor,[21] which is often not possible in remote areas with limited workforce. Lot quality assurance sampling can be conducted where there are no supervisors and uses a simpler method to classify surgeons as having either high or low recurrence rates,[521] rather than estimating actual recurrence. Outreach sampling for audit is used when surgeries are audited at the end of a surgical outreach campaign.[21] This third method has its limitations, as it is performed much closer to the date of surgery, limiting the timeframe for recurrence to occur.

This pilot study tested a simplified auditing mechanism that used a mobile data collection app to audit trichiasis surgeons in remote communities with limited human resources 6 months after surgery, based on a 20% sample of operated eyelids per surgeon (with at least 20 operated eyelids per surgeon included). Buchan et al. recommend that 40 consecutive cases be audited to minimize the potential of bias;[5] however, surgeons in our study performed a mean number of surgeries of 82; thus, the required 20-eyelid minimum is a reasonable sample of 24.4%. Part of the challenge of surgical TT audit sampling is that there is no recommended, standard surgical productivity rate benchmark to evaluate surgeons. Low surgical volume and attrition is a common issue among trachoma control programs. Only 16.1% of surgeons working in trachoma-endemic areas are high volume (defined as performing more than 50 annual operations).[24] Highly productive surgeons are thought to have better outcomes. However, the two surgeons (22.2%) in this study, who operated on <50 cases, had recurrence rates of 3.8% and 7.1% that were much lower than the two most productive surgeons, who operated on 112 and 146 eyelids and had respective recurrence rates of 13% and 17.2%. This finding is likely due to bias as a result of a small sample size (< 50 cases).

Recurrence rates vary widely across time points, although they are generally higher than 25%, making it difficult to ascertain between an acceptable, standard, recurrence rate versus a cutoff for surgical retraining. While the 6-month recurrence rates in the STAR trial in Ethiopia ranged from 2.5% to 4.3%, it must be clarified that this trial was done in a controlled environment.[67] All surgeons were certified on the WHO guidelines, which could have improved outcomes; sutures were removed after 2 weeks, which could have improved healing rates; and the most difficult cases were excluded from participating (patients who had previously undergone trichiasis surgery, which made up 64% of ineligible patients). A “real-world” recurrence rate threshold of 25% or higher was used in this study to indicate that a surgeon needed retraining. Buchan et al. suggest a more conservative threshold of below 20% at 1 year instead of 6 months to determine the need for retraining, but they explain that given the multiple factors affecting recurrence (ongoing disease exposure, surgical quality, surgical technique, etc.), each individual trachoma control program should determine their own threshold using data from similar settings or previous studies.[5] Given the fact that no surgeons need re-training in our pilot study, we will consider using a more conservative threshold in future studies.

Six months was used as the audit timeframe rather than 1 year because the WHO recommends surgical follow-up up to 6 months,[21] the majority of recurring cases in some studies have been found to occur within 6 months,[412] and early recurrence is associated with surgical factors/quality.[4] We aimed to capture recurrence that was more likely to be related to the surgeons' performance.

Nine surgeons and 170 patients (with 269 eyelids, a 29% follow-up rate) participated in this audit. Twenty patients (11.8%) had recurrence; the mean recurrence rate among surgeons was 8.3%, which is below rates reported in the literature at 6 months.[4810] The highest recurrence rate was 17.9%; thus, none of the surgeons needed retraining.

We implemented a simplified, efficient, mobile surgical TT audit using a smartphone to collect data remotely, which in workforce-limited settings requires a minimum amount of surveyors and supervision. Timestamped data entry and GPS positioning functionality help to prevent the manual error from double data entry, thereby saving on time. Another advantage of the auditing tool is that trained operators can delete or modify the survey questions, which may require revision to improve survey efficiency and the local context of the audit after they are transmitted to the server. The app additionally offers excellent technical support.

A major strength of the auditing mechanism is its low cost. The audit costs only $15,166.23, or 0.2% of the global budget; thus, this auditing mechanism is feasible to scale-up nationally. Among the audited surgeons, the per-surgeon cost was only $1,685.13. Had the surgeons required retraining, the cost of training would have been $2,051 per surgeon or a potential total of $18,459. For less than the cost of retraining the nine surgeons, we audited them and determined that retraining was not needed. Therefore, rather than schedule periodic refresher surgical training, there is a potential saving in training when implementing quality control measures in trichiasis surgical programming. Other trachoma control programs with limited resources could greatly benefit from the use of this auditing mechanism at 6-month follow-up, and as explained above, the survey questionnaires can easily be adapted to the local context.

Disadvantages of the mobile auditing mechanism are that Internet access is required to upload and access data (which can be a challenge in remote, rural communities), and a trained survey manager is needed to be able to manage and update the on-line survey form, as necessary. Many of the health centers in the survey area have poor infrastructure and do not have access to a computer, Internet, or reliable electricity for that matter. Given that the audit took place in communities lacking an Internet network, most data were not transmitted by the app until 3 days after the study began, when the surveyors arrived at the health center of Chinguil, which had network access. However, the Internet connection was too weak to access the server and download the study data. Hard copy audit reports are therefore needed and will be provided to heads of health centers in future.

The majority of surgeons (9/14; 64.3%) were audited. One issue that might have affected surgeon participation is that a reserve list of patients was not used to ensure the minimum sample per surgeon and to reduce bias. Low surgical volume was generally the reason that surgeons were excluded; thus, one question for future audits is if the low surgical volume (and what threshold) should be justification for retraining. There will be future attempts to audit the five surgeons who did not participate in this study; however, the accessibility of their patients must be further considered. The follow-up rate (29%) may have been lower because many patients were inaccessible during the rainy season. There may have been significant bias in the patient sample, as people who have easier access to the health centers and trachoma information in general may have been operated on when their infection was milder,[5] whereas more severe cases of trichiasis are more difficult to operate on and recurrence can be twice as high.[412] However, a higher follow-up rate was not the aim of the audit, as the patient data included were based on the per-surgeon sample size, meeting the threshold of 20%. Nonetheless, for future audits, we also recommend that patients should be notified to return for follow-up at 6 months at the time of their trichiasis surgery, radio messages be used to inform the communities of the audit, and relay persons work in collaboration with the supervisors and heads of health centers to improve coordination.

Another study limitation was that the 20% sample size was determined arbitrarily, but this was a pilot study with limited published data available to calculate a sample size. Furthermore, the size falls within the reasonable recommendations for sampling for surgical TT audits.[21] Although no randomized method was used to select patients, which may incur bias,[5] by attempting to include all patients who participated in follow-up on days 7–14 and who were contacted for this audit ensured the greatest number of participants from these difficult-to-reach communities. A larger, prospective trial could be conducted in future to better validate the sample size and to compare the efficacy and costs of the mobile auditing mechanism to other auditing models.

The audit did not capture key patient data, such as disease severity, surgical complications, pre/postoperative visual acuity, and follow-up data. However, a minimum dataset was included in determining recurrence rates per surgeon and in guiding the quality improvement process. Future audits will also include health district data and exclude the patients' phone numbers, given that most did not have one.

Another limitation of the Device = KoBo Collect app is the digital photography function. Figure 3 represents a digital photograph of a case of recurrence taken by one of the surveyors with their smartphone and stored by the app. The poor quality of the images prevents them from being used for remote assessment, and thus, it would not be an efficient use of time to take pictures of all patients in the field. Surveyors only took photographs of eyes affected with recurrence to document their work. The photograph function, however, is not an essential component of the auditing mechanism, as it is not used to remotely adjudicate or confirm cases of recurrence.

Figure 3

An example of a digital photograph of a case of recurrence taken by a surveyor using a smartphone and captured by the Device App

While testing the Device = KoBo Collect app, this study did not evaluate the performance of the surveyors. The survey was programmed so that the questionnaires were automatically transmitted to the online server once they were completed. In the future, the surveyors will manually hit “send” to transmit each patient's data so that data entry operators will be able to count the number of questionnaires completed and sent per surveyor.

One final disadvantage with the study is the audit coincided with the six monthly follow-up, which hindered resource mobilization. Future audits will be integrated as part of the follow-up schedule.

This pilot study tested a simple, easy-to-use, and low-cost mobile auditing mechanism and found it to be a practical solution for conducting surgical audits in remote and resource-limited settings for surgical quality improvement. As a result of these findings, the Device = KoBo Collect app is now being used in the entire national trachoma program in Chad. A second audit was performed in Ouaddaï, Sila, and Salamat in October 2017, and a third audit was carried out in the southern districts of Chad in 2018. The auditing tool can be easily adapted to the local context of other trachoma control programs and may result in the more frequent use of surgical auditing in these programs. The routine use of surgical auditing in trachoma control programs could improve the quality of trichiasis surgeries and potentially increase patient uptake, thereby advancing the global effort to eliminate blinding trachoma.

Financial support and sponsorship

Sightsavers (Chippenham, United Kingdom) sponsored this study, with funding provided to them by the UK Department of International Development.

Conflicts of interest

There are no conflicts of interest.