Data quality assurance and quality control measures in large multicenter stroke trials: the African-American Antiplatelet Stroke Prevention Study experience.

Data quality assurance and quality control are critical to the effective conduct of a clinical trial. In the present commentary, we discuss our experience in a large, multicenter stroke trial. In addition to standard data quality control techniques, we have developed novel methods to enhance the entire process. Central to our methods is the use of clinical monitors who are trained in the techniques of data monitoring.

Introduction

The clinical trial is the 'gold standard' for obtaining definitive information about medical interventions. Collecting accurate data on patients who are being followed under specific and controlled circumstances lies at the heart of the mission of a clinical trial. Ensuring that the accumulating data are as free of anomalies as possible is of utmost importance to the soundness of the process. Indeed, having no errors in the data and no occurrences of protocol violations is the goal, but the amount of resources required to accomplish such a goal is prohibitive. At the very least, effective procedures should be instituted in every trial to limit inadvertent errors, as well as fraudulent activity, even though it is believed that fraud occurs rarely [1]. Failure to plan and implement an effective quality assurance program within a trial can not only adversely affect the scientific impact of the trial itself, but can also affect public confidence in the reliability and effectiveness of clinical trials within medical research.

Gassman et al [2] and McFadden [3] provided comprehensive literature reviews and detailed discussions of the necessity and structure of effective data quality assurance and quality control programs within a clinical trial. In addition to standard data quality control methods [2], the African-American Antiplatelet Stroke Prevention Study (AAASPS) [4] developed unique methods for accomplishing the goals of these programs. The AAASPS is a two-arm, multicenter, randomized, double-blind clinical trial comparing ticlopidine and aspirin therapy in patients who experience an ischemic stroke within 90 days of randomization. This clinical trial has been enrolling and following patients since December 1995. The primary objective is to assess the efficacy and safety of the two treatments in the prevention of recurrent stroke, myocardial infarction, and vascular death in 1800 African-American patients treated for 2 years. Here we discuss some of the components of our quality assurance program that we have found to be particularly helpful.

A more comprehensive use of clinical monitors

In the AAASPS, a clinical monitor is assigned to each local site. At least one monitoring visit is scheduled per quarter. Monitors serve both clinical and data management functions. They provide information and guidance on the management of study patients, in accordance with the study protocol. They also review source materials that document patient eligibility and review study notebooks for each patient in order to ensure that the protocol is being followed properly. Any incomplete or questionable patient documentation is cited and reported by the monitor for review and investigation at the study management level.

The AAASPS protocol specifies 13 in-person visits and 15 telephone contacts over the 2-year follow-up period. More visits can be made if patients experience an outcome event (recurrent stroke, myocardial infarction, or death). At each visit or telephone contact, local site staff complete case report forms (CRFs) that are printed on three-part carbonless paper. At each site visit, the monitor reviews uncollected CRFs, related source documents, and all patient progress notes. Any discrepancies or errors discovered during this review are resolved with site personnel, if possible, during the same visit. CRFs are not collected from the site without review and approval of the designated clinical monitor. In fact, CRFs are not processed for data entry without monitor approval being indicated on each form. On approving the completed CRFs, the monitor separates and removes the original (for data entry) and one copy (for the Clinical Management Center) of each CRF, and personally delivers the original to the Data Management Center (DMC) for data entry.

It is difficult to gauge the increase in data quality and overall efficiency gained by having monitors check for errors at the site before CRFs are collected and processed. However, each monitor prepares a report subsequent to every site visit (which usually occurs once per quarter) summarizing significant discrepancies identified during the visit and corrective measures to be taken. Well over half of all these reports address significant breaches of the study protocol and/or outright data errors, such as the miscoding of dates and test results.

The monitors also hand-deliver any printed queries generated from the logical checks program that is used as part of the data entry process. All checks that fail are flagged for printing as queries to the local sites and then distributed to the monitors. The monitors review each query with site personnel and offer assistance with the development of a short written response to the query. If a query results in a change of previously submitted data, then the monitor must also verify that the change is made to the site's copy of the CRF. In this way, no changes to the CRFs are made without the monitor's approval (as indicated by the monitor's initials).

We have found that this personalized form of monitoring provides an effective additional level of quality assurance in both clinical and data management. In this way, data are evaluated and assessed at the source (the local site) before being collected or entered into the study database. Moreover, site personnel are able to interact, in person and on a regular basis, with a clinician who is trained in the details of the protocol. This form of monitoring often prevents errors occurring that may not be detected by the usual computer-based logical checking systems. We also have found it useful for study personnel to visit sites regularly and to assess the overall effectiveness of those sites over time. Such information enables us to react more proactively to problems that develop at local sites.

We emphasize that such use of monitors adds an additional level of quality control and increases efficiency through earlier error detection. However, this feature does not replace computer-based data checking. To illustrate this point, we note that approximately 20% of all outcome events forms (which provide key data describing the assumed occurrence of an outcome event - recurrent stroke, myocardial infarction, or death) have generated a computer query. Most of these queries involve key data (such as the date of the event) that were missing or incorrectly coded. Even though this percentage would have no doubt been much higher without the aid of our monitors, it is important to have employed multiple methods for maintaining quality control.

Audit of local sites

Clinical and data management personnel jointly conduct an intensive program of auditing local sites in the AAASPS. Sites are selected for an unannounced audit on the basis of criteria that include the number of patients randomized by the site, total number of data queries generated by their CRFs, total number of protocol violations and misrandomizations reported, and whether the site has been cited by federal regulators. Sites with large enrollments or showing evidence of having continuing difficulty have been audited more than once.

The audit visit team consists of at least three people, two of whom are representatives from the DMC. The others are clinical monitors appointed by the Clinical Management Center. The clinical monitor assigned to the local center is not allowed to be a part of the audit team, thus avoiding a potential conflict of interest. At least half of the members of the audit team are experienced, having participated in a previous audit. In this way, experience gained in previous audits is passed on.

During the audit, the site is assessed for general organization, security, and adherence to the study protocol. High priority is given to verification of patient eligibility and informed consent, assessing the quantity and quality of patient progress notes and source documentation, and checking for consistency between the study database and the CRFs at the site. An exhaustive review is made of the study materials for approximately 10% of the patients randomized to that site. These patients are chosen by the staff of the DMC, and are not revealed to site personnel before the visit.

We are careful to emphasize that the audit process is not meant to be hostile. Our purpose is to document a high quality of data collection and follow up of patients. To make the process as nonconfrontational and fair as possible, the site to be audited is contacted 14-30 days before the visit to arrange a mutually acceptable time. After the visit, the audit team writes a report of their findings. This report is then discussed with site personnel, and usually requires corrective actions to be completed.

Our goal is to conduct an unannounced audit of every site that randomizes 20 or more patients. Thus far, 28 sites have been audited and there are six to eight sites remaining. A significant part of the audit is one of three summary designations that is assigned (by the audit team) to the site: 'satisfactory', 'needs improvement', or 'unsatisfactory'. To date, no site has been given an 'unsatisfactory' designation, but three have been given a 'needs improvement' designation. These three sites had significant problems implementing the study protocol, and were eventually closed to patient recruitment.

Along with providing the usual quality control measures, the audits have been valuable as an educational tool, helping us to identify and address situations before they become serious problems. In addition, many site personnel view the audits as an opportunity to reinforce the information contained in the study procedure manual.

Preliminary close out of local sites

When a local site is closed to new patient recruitment and has completed follow up of all of its patients as stipulated by the study protocol, a preliminary close out visit is planned. The purpose of the close out visit is to ensure that all data have been properly reviewed, collected, and verified; to ensure that all unused study supplies have been properly processed and returned; and to verify that the investigator's files are complete and accurate. This visit is conducted by a team of clinical monitors led by the designated monitor for that site. During the visit, all site regulatory documents, study binders containing CRFs, patient progress notes and source documents, and remaining study supplies are reviewed and processed, if required. It is emphasized to site personnel that this close out process is preliminary, in that they continue to be responsible for adequately maintaining study records until the official conclusion of the study.

Once the close out visit is scheduled by the monitor, in co-operation with the site and DMC, the DMC prepares a data packet for the team's use. This packet contains the following, for each patient randomized to the site: a list of all study visits recorded in the database; all generated data queries, along with an indication of whether each query has been resolved; all recorded outcome events and whether they have been properly adjudicated; and any reported serious adverse events that have not been reported as resolved. In addition, the DMC reviews the data on each patient at that site and provides in the packet details regarding the status of each patient at their last study contact. Any questions the DMC has regarding patient follow up or data quality are also included. The typical close out visit takes 1-3 days, depending on the number of patients randomized to the site.

It is common for clinical trials to conduct site close out visits at the conclusion of the study, sometimes even after the final results paper is written. The problem with this approach is that the study staff may no longer be employed at the site, making resolution of outstanding issues difficult. Moreover, resolution of problems with patient follow up or data entries may be complicated (and impossible in some cases) by the passage of time. Some issues can only be handled if they are discovered and addressed in a timely manner before patients relocate or study staff forget the circumstances surrounding certain occurrences. To date, approximately one-third of the close out visits we conducted have resulted in the recovery of substantial amounts of patient data, such as the discovery of previously undocumented diagnostic examinations and adverse events. We also discovered a patient who was misrandomized (that is, who did not meet all entry criteria).

Conclusion

In the present commentary we discuss three components of the overall AAASPS quality assurance program that specifically deal with managing the quality of the study database. A key feature of this process is the heavy use of clinical staff who are trained in the rigors of the protocol and in data monitoring. Having trained study personnel visit sites on a regular and frequent basis affords the opportunity of detecting and preventing errors in a more timely manner. In addition, we found that the relationship developed with site staff provides a useful conduit for conveying information between the study managers and the sites.

Of course, this use of personnel comes at a price. Hiring, training, and underwriting travel expenses for such a trained group of clinicians and data management professionals carries considerable expense. Originally, we developed this strategy to support the significant number of community-based hospitals that were a part of our study. Such hospitals often are not experienced in the conduct of clinical trials, and do not have the resources to manage the rigors of data collection. However, we found the expense not to be so great as to outweigh the benefits. In fact, the cost has been minimal compared with the benefits realized in improved data quality.

Abbreviations

AAASPS = African-American Antiplatelet Stroke Prevention Study; CRF = case report form; DMC = Data Management Center.