Ensuring quality in AFRINEST and SATT: clinical standardization and monitoring.

BACKGROUND: Three randomized open-label clinical trials [Simplified Antibiotic Therapy Trial (SATT) Bangladesh, SATT Pakistan and African Neonatal Sepsis Trial (AFRINEST)] were developed to test the equivalence of simplified antibiotic regimens compared with the standard regimen of 7 days of parenteral antibiotics. These trials were originally conceived and designed separately; subsequently, significant efforts were made to develop and implement a common protocol and approach. Previous articles in this supplement briefly describe the specific quality control methods used in the individual trials; this article presents additional information about the systematic approaches used to minimize threats to validity and ensure quality across the trials.
METHODS: A critical component of quality control for AFRINEST and SATT was striving to eliminate variation in clinical assessments and decisions regarding eligibility, enrollment and treatment outcomes. Ensuring appropriate and consistent clinical judgment was accomplished through standardized approaches applied across the trials, including training, assessment of clinical skills and refresher training. Standardized monitoring procedures were also applied across the trials, including routine (day-to-day) internal monitoring of performance and adherence to protocols, systematic external monitoring by funding agencies and external monitoring by experienced, independent trial monitors. A group of independent experts (Technical Steering Committee/Technical Advisory Group) provided regular monitoring and technical oversight for the trials.
CONCLUSIONS: Harmonization of AFRINEST and SATT have helped to ensure consistency and quality of implementation, both internally and across the trials as a whole, thereby minimizing potential threats to the validity of the trials' results.

To address the priority research needed to identify effective alternative, simplified antibiotic regimens for community-based management of newborn sepsis, 3 clinical trials were conceived and implemented, as previously described.[1] Each of these trials was designed as a randomized open-label trial to test the equivalence of simplified antibiotic regimens compared with the more complex standard regimen of 7 days of parenteral antibiotics. Trials known as the Simplified Antibiotic Therapy Trials (SATTs) were designed separately; through a 2008 consultation convened by the Bill and Melinda Gates Foundation (BMGF) and subsequent meetings, the separate study protocols were harmonized as a single, common protocol to be implemented in each country. Implementation of SATT started first in Bangladesh in July 2009 followed by Pakistan in January 2010. The African Neonatal Sepsis Trial (AFRINEST) was subsequently initiated in 2010 and was designed to use the same protocol (with additional arms) and outcomes as SATT in sub-Saharan Africa [the Democratic Republic of the Congo (DRC), Kenya and Nigeria].

The main objective of SATT and AFRINEST was to evaluate simplified alternative antibiotic regimens of antibiotics that could subsequently be recommended for community-based management of serious infections in newborns and young infants. Eligible participants include newborns and young infants (0–59 days old) with suspected serious infection, whose families do not accept or comply with referral for hospital-based treatment. The trial methods have been described in detail elsewhere.[2-5] Although these research trials were originally conceived and funded separately, subsequent efforts were undertaken to harmonize the trials to ensure consistency and quality, both internally and across the trials as a whole.[1] The Bangladesh SATT, developed by the Johns Hopkins University and Dhaka Shishu Hospital, was originally funded by United States Agency for International Development (USAID). The Pakistan SATT was developed by Aga Khan University (Karachi) and was funded by the Saving Newborn Lives initiative (SNL) of Save the Children with support from BMGF, with additional funding provided by World Health Organization (WHO) and USAID. AFRINEST was originally conceived and developed by WHO in consultation with SNL and USAID with funding from BMGF. Through a systematic proposal development process, principal investigators (PIs) of AFRINEST were selected from Nigeria (University of Ibadan, Obafemi Awolowo University and Ahmadu Bello University), DRC (Kinshasa School of Public Health) and Kenya (Moi University).

Previous articles in this supplement[1-5] have included information on the unique internal quality control processes that were used within each separate trial, as well as the development of a common protocol (including standardized definitions and terminologies) for the trials as a whole. This article focuses on the quality assurance: it presents the methods used for quality control across SATT and AFRINEST and additional information regarding efforts to harmonize the trial-specific internal monitoring procedures.

QUALITY CONTROL METHODS IN AFRINEST AND SATT

Potential Threats to Validity

During the planning and design stages of AFRINEST and SATT, study teams identified inherent design-related issues that had the potential to threaten the quality of the studies and validity of results. Significant efforts were made to develop systematic approaches to minimize these challenges and ensure quality of implementation. A primary concern was that the enrollment and treatment outcome criteria rely on clinical judgment, as opposed to objective measures, such as laboratory confirmation of severe infection or death as the primary outcome. Reliance on subjective determination of clinical signs of severe infection presents the potential for misidentification of clinical signs of severe infection and of enrolling patients with different types of illnesses and infections. Were this threat not effectively mitigated by quality control efforts, otherwise strong evidence of antibiotic regimen equivalence might be considered inconclusive. The determination of treatment failure, with the exception of death, also relies on clinical assessment; this poses a similar threat to conclusive evidence of equivalence. In addition, the trials are open-label, meaning that study personnel and participants are not masked to treatment assignment and thus know which therapeutic regimen an infant is receiving. The nature of these trials did not allow for masking the treatment regimen. Placebo injections would not be ethical, and sham injections would be potentially unethical and logistically problematic. The PIs, donors (BMGF and USAID) and trial technical steering committee (TSC) and technical advisory group (TAG) therefore undertook rigorous quality control methods to mitigate these and other potential threats, to internal validity within and across these trials.

Ensuring Quality

Development of SATT Quality Control Measures

Because the 2 Asia SATTs were separately funded and administered, special efforts were undertaken to coordinate the studies’ implementation, technical oversight, data management and quality control. SNL was specifically funded by BMGF to organize a mechanism of technical oversight and monitoring for the 2 SATTs in Asia, using independent technical experts with experience in design and implementation of clinical trials in developing countries, pediatrics, infectious diseases, ethics and biostatistics. This mechanism was mutually agreed to and planned by the PIs, funding agencies (SNL and USAID), WHO and BMGF; thus, a TSC was formed, consisting of independent experts, PIs, WHO, BMGF, USAID and SNL. A major initial concern was the need for standardized trial monitoring across the 2 SATTs to ensure uniform and high-quality implementation and internal monitoring procedures as much as possible. The TSC first considered whether an independent contract research organization (CRO) would be able to provide the necessary cross-site monitoring functions. The TSC held discussions with potential CROs and also specifically identified, together with the PIs, the particular issues that needed to be addressed to robustly mitigate the major threats to validity. The conclusion from these investigations and discussions was that CROs, in general, were not the most appropriate mechanism for monitoring in these trials. This was based on the limited capacity of CROs to specifically monitor the clinical standardization and quality control required for trials that rely on clinicians’ skills and judgment to determine enrollment eligibility and treatment outcomes. The SATT TSC identified the following as key to quality control within and across sites:

Development and use of standardized protocols for clinical assessments for enrollment eligibility and treatment outcomes.

Development and use of standardized internal study team monitoring procedures to routinely assess and refresh clinical skills and judgments of staff.

Frequent and standardized monitoring visits by TSC members, using WHO site visit monitoring guidelines and checklist.

The SATT TSC used an external clinical monitoring expert, who was a trained pediatrician, to independently review the sites and make specific recommendations to standardize and harmonize all aspects of study implementation, clinical skills and judgments, and quality control procedures. These visits resulted in modification of standard operating procedures (SOPs) in both SATTs, particularly standardization across sites of clinical assessment and internal quality control methods.

Development of AFRINEST Quality Control Measures

AFRINEST is a single trial funded entirely by BMGF, with 5 sites and PIs in 3 sub-Saharan African countries, led and coordinated by a technical team at WHO. From the outset, AFRINEST used the common protocol that was developed initially for SATT. Modifications in implementation were necessary due to contextual factors, including differences in health delivery platforms between SATT Asia sites and those in the AFRINEST locations. As noted in the methods of previous articles,[4,5] additional antibiotic regimens were tested in AFRINEST, including a simplified regimen for young infants with fast breathing only. The quality control measures instituted for AFRINEST were based on the development of SOPs, uniformly applied across all sites, including standardized training, WHO monitoring visits, assessment of clinical skills, refresher training and use of an external monitor for independent review.

SATT and AFRINEST Common Approaches to Quality Control

A critical component of quality control for AFRINEST and SATT was striving to eliminate variation in clinical assessments and decisions regarding eligibility, enrollment and treatment outcomes. While each of the trials adopted procedures specific for the setting, there was a uniform effort across the trials to assure that each “treatment failure” assignment was correct. In Bangladesh and Pakistan, the outcome assessor may know the infant’s treatment assignment. Infants who are identified as failing treatment (during routine follow-up) receive a second assessment by a different study physician; this second assessor is blinded to the infant’s treatment assignment and history. If the 2 assessments disagree, the decision is referred to a supervising senior physician who makes the final determination. The AFRINEST sites use a separate “independent outcome assessor:” a nurse who is responsible for performing the outcome assessment and who is blinded to the infant’s treatment assignment. Ensuring appropriate and consistent clinical judgment was accomplished through a combination of approaches, applied across the trials, including training standardization, routine (day-to-day) internal monitoring, systematic external monitoring by funding agencies and external monitoring by experienced, independent trial monitors.

Training and Standardization

AFRINEST and SATT use different cadres of health workers [including physicians, nurses and community health workers (CHWs)] and have implemented standard training protocols to ensure common understanding and clinical judgment among and across all cadres.

In AFRINEST, an initial training-of-trainers was conducted to train the PIs; the PIs then led trainings at each study site for local study staff. WHO technical monitors visited each site as soon as the training of health workers was underway to conduct a standardization exercise (Appendix 1) to check clinical assessment skills of the study staff. Those staff who did not meet preset criteria in the standardization exercise were immediately retrained; if they did not perform well after retraining, they were withdrawn from the study. Staff who met the performance criteria were assigned to roles within the study based on the knowledge and skills demonstrated during the standardization exercise (eg, the highest performing nurses were assigned to be independent outcome assessors). Regular refresher trainings are conducted by the PIs every 3–6 months; additional standardization exercises are held at 3- or 6-month intervals (depending on the type of study staff). Results of the standardization exercises are input into a spreadsheet (Microsoft Excel) and scores outside the standard range are marked; those staff who do not perform well (eg, not conducting the required number of follow-up visits or performing poorly on clinical assessments) are retrained on specific areas in which their performance is low and reassessed before they are allowed to continue.

In SATT, intensive training was also conducted at the start of the trials. Refresher training sessions and standardization exercises are held regularly to ensure consistency in eligibility and treatment outcome evaluation. In Bangladesh, refresher trainings and standardization exercises are conducted quarterly; weekly meetings of the study staff also provide opportunities for “continuing education” through review of randomly selected cases of severe infection and discussion of any discrepancies in the assessors’ identification of clinical signs (Appendix 2). In Pakistan, refresher trainings and standardization exercises are conducted on a monthly basis; the primary goal of these activities was to reinforce knowledge and skills related to clinical assessment (Appendix 3). Video recordings of selected infants with and without clinical signs of severe infection are used to assess and refresh skills.

Internal Monitoring

SOPs were developed for AFRINEST and SATT to guide the operational and clinical activities of the trials; this included SOPs for internal monitoring. The main objective of internal monitoring was to verify that the study protocol and SOPs are being followed correctly. As with other aspects of the trials, internal monitoring approaches were harmonized across all sites to ensure similar high levels of quality and safety. In all cases, feedback from the routine internal monitoring is presented and discussed with the appropriate members of the study team on an ongoing, real-time basis.

In AFRINEST, PIs conduct routine monitoring visits to study sites and use a standard checklist developed by WHO. These visits include checking study equipment and supplies (eg, weighing scales), as well as conducting supervision activities for study staff. Scheduled and unscheduled visits are performed by PIs and field supervisors; a standardized checklist is used to monitor the skills of CHWs, treatment nurses and outcome assessors regarding identification of danger signs, classification, eligibility criteria, randomization and consent procedures, treatment and injection technique. At a minimum, 5% of a nurse’s/CHW’s visits must be directly observed; outcome assessment nurses must be observed during at least 20% of their visits. If errors or gaps in skills are observed during a supervisory visit, immediate feedback is given to the study team member.

In the SATT Bangladesh trial, a SOP was developed to guide the operational and clinical activities of the trial; this included SOPs for internal monitoring. The main objective of internal monitoring was to verify that the study protocol and SOP were being followed correctly. As with other aspects of the trial, internal monitoring approaches were harmonized across all sites to ensure similar levels of quality and safety. In all cases, feedback from the routine internal monitoring was presented and discussed with the appropriate members of the study team on an ongoing, real-time basis.

Site visits were conducted by study coordinators and by investigators (and/or consultant experts) in the SATT Bangladesh trial. The study coordinator is allocated 2 days every month for each site for this purpose. One day is randomly selected for each site for the monitoring visit, and the visit is completed on the next day. The coordinator uses a standard monitoring checklist (developed by WHO) to guide the visit, which includes evaluating study management, reviewing subject records and observing key study procedures. Findings are shared with the local study team and with investigators. A trip report is submitted after each monitoring visit. One random day is selected per month for each site to be visited by study investigators/consultants for a monitoring and supervision visit. The first 5 cases on the selected day are selected for a study physician to assess (the clinical component only) and the investigators/consultants to observe. Only disagreements are recorded, and the reason for disagreements are immediately addressed and discussed. These random visits are planned to ensure that all study physicians are subjected to a validation exercise once in a 3-month period. For clinical signs about which the investigators disagreed with study physicians’ assessment, a database is kept; these signs are also emphasized during refresher training.

In SATT Pakistan, routine internal monitoring is conducted on a daily basis by 2 dedicated clinician supervisors who observe the screening and enrollment process, randomization and the delivery of therapy. They also observe approximately 20% of interviews conducted by study physicians, perform spot checks and formally reinterview 10% of participants interviewed by study physicians. Finally, supervisors check case files for errors, inconsistencies and missing values. Additional monitoring visits are conducted by an “internal auditor:” a physician and monitoring expert who is not associated with the trials. The auditor performs a random visit once a month to observe and review clinical practices in the study; a standardized checklist is used and feedback is provided directly to the PI, research supervisor and field team.

External Monitoring

Both AFRINEST and SATT receive structured external monitoring visits from WHO monitors and other members of their respective TAG (AFRINEST) or technical steering committee (SATT). WHO technical staff, following a standard WHO site monitoring guideline/checklist (Appendix 4), visit each AFRINEST site at least once every 6 months. A detailed structured review of study implementation is conducted at each visit, including recruitment rate, clinical practices (eg, identification of danger signs and critical signs, detection of treatment failure), data management and overall study procedures. All study documentation is checked, and a proportion of completed case report forms (including all reports of treatment failure and adverse events) is reviewed. The recommendations arising from the site visit are discussed with the PIs at the sites and followed up for actions taken and dates. At least once per year, the SATT Bangladesh and Pakistan sites receive structured external monitoring visits from a WHO technical expert, either alone or accompanied by SNL/SC technical staff and independent TSC member(s). Additional ad hoc monitoring visits by external TSC members are conducted for SATT Bangladesh, mostly in conjunction with TSC/Data Safety Monitoring Board (DSMB) meetings, which are frequently held in Dhaka. Additional ad hoc visits are intended for SATT Pakistan but are subject to security restrictions for non-Pakistanis. The format for SATT external monitoring visits is similar to AFRINEST: recommendations and written feedback are provided, and the PI subsequently updates the TSC on actions taken in response to each recommendation. The same WHO site monitoring checklist is used to structure the SATT monitoring process and to standardize across all sites.

In addition, all sites have been visited by an independent external trial monitor who examined study implementation and adherence to protocols and Good Clinical Practices standards. For SATT sites (Bangladesh and Pakistan), the external monitor conducted an independent assessment of clinical and quality control procedures and made recommendations on how to standardize or harmonize all such processes. PIs followed these recommendations to implement uniform internal monitoring and quality control procedures. In AFRINEST, an independent external monitor visited each site, provided an assessment using the same protocol as WHO monitors (Appendix 4), made recommendations for each separate sites that were implemented by PIs and then checked for by WHO during routine monitoring.

Technical Advisory Groups

Two external advisory groups were set up for the trials: AFRINEST formed a TAG and SATT formed a TSC to oversee the 2 trials in Bangladesh and Pakistan. These bodies are responsible for providing broad technical oversight for the trials. In the case of SATT, the TSC adds unique value by facilitating coordination and harmonization for the 2 separate studies. The SATT TSC has played an active role to provide coordinated oversight for the 2 separate trials, as explained above. This includes routine external monitoring visits, including visiting homes and clinics to observe enrollment and management of patients; the TSC then makes recommendations based on its observations and follow-up to ensure recommendations appropriately reviewed and implemented by each trial. The AFRINEST TAG is not needed to provide coordination of technical oversight, as this was done by WHO as the coordinating agency for the trial. The AFRINEST TAG has been convened for annual review meetings but has not engaged in direct field-level monitoring in the same way that it is done by the SATT TSC. Membership of the TAG and TSC includes experts from a variety of areas, including clinical pediatrics, neonatology and infectious diseases; global clinical trials (including ethics); epidemiology and biostatistics; and global health policy. There was an intentional overlap in the TAG and TSC membership; a number of individuals were selected to serve on both groups to facilitate greater harmonization between the trials. The SAT TSC holds quarterly and as-needed conference calls to review quarterly data and discuss technical issues, as well as annual in-person meetings. The AFRINEST TAG holds teleconferences on an as-needed basis throughout the year, as well as annual in-person meetings.

Data Safety Monitoring Boards

Two DSMBs were formed, 1 for AFRINEST and 1 for SATT (Bangladesh and Pakistan combined), each with 5 members: 1 epidemiologist, 1 statistician and 3 clinicians/researchers. Members of the DSMB are not involved in the design or conduct of the trials. These groups are tasked with monitoring and assessing the safety of the trials (based on reports); responsibilities include ensuring patient safety within each site, setting up “stop rules” based on ethical and public health considerations, reviewing interim analysis data and recommending appropriate actions to be taken by the PIs. The DSMBs meet annually with the PIs and TSC/TAG and communicate among members via e-mail and teleconferences as needed to review data and make recommendations to PIs.

Data Management

There are 3 levels of data management and quality assurance in AFRINEST and SATT: local data management and quality control practices at study sites and within the individual trials; technical oversight, review and quality control for data management from the TSC/TAG; and external monitoring visits and high-level data management, review and quality checks performed by the data management team at the London School of Hygiene and Tropical Medicine (LSHTM). Methods for site-level and individual trial-level data management have been described elsewhere.[2-5] The team at LSHTM performs a standard set of range and consistency checks for all data sent from the individual sites. Data are archived at LSHTM, and reports are generated as required by IRBs, the AFRINEST Coordinator (WHO) and TAG, and the SATT TSC.

Routine external monitoring visits include review of study data and data management procedures. Specifically, database and CRFs are reviewed for all death and treatment failure cases, and a random sample is taken of approximately 10–20% of eligible and enrolled cases. Finally, LSHTM provides data management and analysis support for both AFRINEST and SATT. AFRINEST submits data to LSHTM on a monthly basis; these data are shared with WHO monthly and with the TAG annually. The SATT trial sites submit their data to LSHTM on a quarterly basis, and the data are shared with the TSC quarterly for review. LSHTM also provides routine data summaries of deaths, treatment failures and severe adverse events for regular review by each DSMB.

Strategic Planning Committee

The Strategic Planning Committee is composed of AFRINEST and SATT funding agency representatives and chairs of the TSC and TAG. Its primary purpose was to review the trials’ progress and to provide strategic direction for planned study analyses to answer questions of global policy relevance. The Strategic Planning Committee provides general strategic direction to maximize policy impact, based on efforts to ensure sound evidence, relevant evidence from both trials.

CONCLUSIONS

To ensure quality of implementation and to minimize potential threats to validity, AFRINEST and SATT adopted a variety of purposeful quality control approaches; these were similar or the same across the trials. Key quality control processes have included standardization of training efforts and assessment of clinical skills, routine internal monitoring of study protocol adherence, and external monitoring and oversight of study implementation by funders and independent experts.