Results of a quality control on non-interventional studies.

Non-interventional studies (NIS) have for decades been an established part of post-authorisation medicinal research. As early as the mid-nineties, there were at least rudimentary demands for controllable data quality. Beginning with the recommendations of the Federal Institute for Drugs and Medical Devices (BfArM) on the execution of non interventional (observational) studies of 1998 and finally with the guidelines and recommendations for ensuring Good Epidemiological Practice (GEP), with the VFA (Verband der forschenden Arzneimittelhersteller [German Association of Research-Based Pharmaceutical Companies]) - Recommendations for the Improvement of Quality and Transparency of NIS and the joint recommendations of BfArM and PEI (Paul-Ehrlich-Institut) on the execution of NIS, pharmaceutical companies are required to monitor and/or verify quality in the course of a project. According to a survey of pharmaceutical companies 2010, about one third of the companies surveyed to date carry out such quality controls on site, at participating study centres. This report deals with the results of such quality control measures in 4 completed projects. The control rates defined in the respective cohort study plans, the measures carried out on site and any consequent measures, such as adjustment of forms, reduction of consultation time and necessary organisational changes are described. A high level of agreement between the data collected and the original patient documents is found, comparable to that in clinical trials.

1 Introduction

Already in the nineties, it was pointed out that cohort studies were limited in their ability to provide useful information and that the quality of the data yielded was difficult to control. At the time, the quality criteria of a non interventional, observational study were met if observation and evaluation plans were available and the representativeness of the cohort had been investigated [1].

The recommendations of the Federal Institute for Drugs and Medical Devices (BfArM) on the execution of observational studies of November 1998 already included the usual quality requirements for epidemiological studies to ensure the completeness and validity of the data and be in a position to recognise and correct defects early [2].

Suitable quality assurance measures, especially for the verification of the data collected, are a part of the recommendations published in January 2007 by the VFA. According to these VFA recommendations [3], quality assurance systems should be used that ensure the validity and representativeness of the data collected.

The "Freiwillige Selbstkontrolle für die Arzneimittelindustrie e.V." [Voluntary Self-regulation for the Pharmaceutical Industry, Registered Association] (FSA), founded by member companies of the VFA in 2004, incorporated these recommendations in its codex in August 2008: in the planning, set-up and execution of non interventional observational studies (Anwendungsbeobachtungen – AWB), the recommendations and guidelines published by the BfArM are to be considered. The planning, management, evaluation and quality assurance of the NIS must take place within the company and fall within the responsibilities of the head of the medical department. The required processes and procedures, especially suitable quality assurance measures to verify the data collected, are to be defined in the company's Standard Operating Procedures (SOP) [4].

Especially with regard to the quality of data provided by the doctor, the quality standards must be assured with the objective of obtaining valid data, permitting a qualified evaluation [5]. This shall include the requirement for strict compliance with special SOP and the use of checklists of quality indicators in the quest for improved data quality and validity.

In the guidelines and recommendations for ensuring Good Epidemiological Practice (GEP) the objective is to establish a quality standard for epidemiological studies in order to obtain valid research results: internal quality assurance of all relevant instruments and procedures must thus be an indispensable part of every epidemiological study [6]. The quality assurance targets the rules governing the timing, organisation and technical aspects of the execution as set out in the study plan and/or quality audit concept.

According to the recommendations of the BfArM and Paul-Ehrlich-Institut (PEI) of July 2010 [7], the monitoring and evaluation plan must include a description of the measures for quality assurance "based on" suitable GCP-Standards. GCP is not, however, based on the european legislation applicable to pharmaco-epidemiological studies such as observational studies (EudraLex Vol. 9A, chapter 1.5 Company-Sponsored Post-Authorisation Safety Studies) [8].

In two surveys of the member companies of the VFA, the implementation status of the VFA recommendations was investigated in 2008 and 2010 and the results were compared [9], [10]. Independent NIS quality assurance measures, such as sampling type, systematic control measures by company internal, independent quality assurance units, ensure that studies are carried out correctly and in compliance with the rules in the majority of the companies surveyed.

Meanwhile, almost 1/3 of the companies surveyed carries out such quality assurance measures on site, which is also recommended by the user’s guide “Registries for Evaluating Patient Outcomes” of the Agency for Healthcare Research and Quality AHRQ – part of the U.S. Department of Health & Human Services and member of Guidelines International Network [11].

Quality assurance measures can be used in the planning, execution or evaluation phases. The alignment of the source data (quality control by Source Data Verification, SDV), described, amongst others, by Theobald et al. [12], is the subject of this publication, in which we report on 4 projects in which, finally, quality control measures were carried out and evaluated.

2 Methods

2.1 Quality control concept

In line with the VFA recommendations 2007 and the BfArM/PEI recommendations in their draft version of 09/05/07 [13] and taking account of an internal directive "Management of Observational Studies", in May 2009, a first quality control concept was developed.

To this end, project specific forms were developed: visit report (study site), patient record (see Table 1 (Tab. 1)) and a visit record. In the patient record, an extract of which is shown, taking an observational study in the field of diabetology as an example (Table 1 (Tab. 1)), in addition to the listed variables (at the different times at which documentation is required), there is a direct question as to the occurrence of serious adverse events (SAE) in order, if appropriate, to investigate whether this might conceal a serious adverse drug reaction (SADR) so that, if need be, a possibly overlooked SADR can immediately be notified.

Table 1

On-site quality control form (patient)

2.1.1 Execution through company based NIS Management

The participating practices/clinics at which the quality control is to be carried out are selected randomly (in this example, by means of the Microsoft Excel Analysis Tool Pak, MS Excel version ’97 or later).

The responsible doctor does not necessarily have to be present for the quality control; in many cases, it is the practice personnel or the study assistant who is also very well acquainted with the patient documentation. At the practice or clinic, after presentation of the signed patient's consent form and, where appropriate, the patient identification list, the items to be controlled are compared with the entries in the electronic or conventional medical records and the results are noted on the appropriate forms.

In the case of a register or epidemiological project "quality calls" can also be made during the carrying out phase of a project without problem; documentation errors at a particular centre can thus be discussed as soon as a project is completed. In non-interventional studies with a medicinal product (AWB), however, it is essential to respect the principle of non-intervention (lat. intervenire = come between/amongst). Any intervention in the ongoing process would be a deviation from the principle of pure observation and must be avoided in such studies. In post authorization observational studies to be carried out in accordance with German Drug Law (AMG) §67 (6) however, it is permissible in the execution phase to bring any documentation problems/errors to the attention of all participating doctors, for example, in the form of a newsletter; this can also be used to keep all participants abreast of the current status of the study (see Attachment 1 : QoLiTaX-Newsletter Nr. 2). Feed back in such a form in the course of a project shows the study sites the interest that the pharmaceutical company takes in the quality of the data documented and is a further motivation factor to continue to maintain the documentation correctly. Further, it is, in many (not all) cases, such information that first gives rise to a correct understanding for the control of the quality of the documented data.

In the closing discussion, it is therefore an advantage to involve the responsible doctor. A short report can be written of the measures carried out on site in which inconsistencies and more important abnormalities are recorded. These are to be drawn to the attention of the project leader.

Taking account of the BfArM/PEI recommendations that had meanwhile been published, the quality audit concept was further revised.

The early involvement of all company-internal collaborators in the communication process with the selected study site has been found to be very helpful.

2.1.2 Outsourcing of quality control measures

After exercising the quality control concept on 3 projects in the indications neurology, oncology and diabetology, a parallel concept was developed to be able to hand over these quality control measures, if required, to a Contract Research Organisation (CRO). It is recommended that it should first be carefully checked to what extent the CRO in question, in the event that a full service contract is awarded, might hand over individual tasks (such as this kind of quality control of an NIS) to sub-contractors [14].

The procedure to be followed here is set down in detail: for example, the CRO contracted receives a list of those centres selected randomly, in an internal process, for a quality control. Furthermore, specifications for precise recording will be made, similar to the internal quality control concept (see 2.1). The on-site visit must be carried out by a qualified employee of the contracted CRO; an audit of this employee must, in turn, always be a part of a qualification audit. Additional training of the CRO employee charged with carrying out these measures is to be advised to ensure that the party awarding the contract receives work of good quality. The evaluation, proposed by Theis and Hundt [15], of a CRO by the client's specialist departments, during and after carrying out of a project, opens up the possibility, where appropriate, of taking appropriate countermeasures in advance in the event of follow-up contracts.

2.2 Quality controlled projects

2.2.1 Project description

Since the development of the quality control concept described in section 2.1, quality measures were taken in 4 NIS projects over a period of about 20 months:

Use of a chemotherapy product in patients with hormone-refractory prostate carcinoma (quality audit project in oncology at clinics and/or large medical practices)

The quality control of this project took place between November 2008 and June 2009. For many sites, such measures in a NIS were a completely new experience; a great deal of detailed explanation was required, both in advance and in the meeting on site. The scepticism initially expressed in some cases gave way, however, to a readiness to allow the quality control to be carried out in the NIS/observational study field. At the time of the visit, all patient documentation had already been completed.

Registry "Work Invalidity in patients with Multiple Sclerosis" in neurological specialist practices

The experience gained in the first project regarding visit preparation, duration, agreement of deadlines and structuring of forms led to further procedural improvements in this registry. The visits took place in February 2010, at neurological specialist practices; here, too, the documentation of the patients had already been completed.

Non-interventional study to record quality of life for patients with solid tumours in clinics and/or specialist practice

At the time of the quality control in the clinics and specialist practices, the project was still in the execution phase. Sets of documents, some of which were completed and other not yet completed, were compared with the data in the medical records. In view of the 2,500 or so sets of patient documents to be expected, a total of 110 sets of documents were controlled in 3 runs between July 2009 and January 2010.

NIS of patients with type II diabetes mellitus being treated with insulin by specialists in internal or general medicine

This prospective observational study was carried out from March 2009 to March 2010 in internal and general medicine practices. The quality control measures took place from January to August 2010. Here, too, there were initially problems in actually being able to carry out the quality control in the practices foreseen in the study plan (see Results). Further participants had to be included in the process by random selection in order to reach the prespecified control rate of 3%.

3 Results

3.1 Control rates

According to surveys by the VFA, to date, quality control measures have been mostly carried out at 2–5% of participating sites for NIS [9], [10], [12]. The prescribed control rates specified in the study plans for the 4 projects described were all achieved. In the first project, the specified target rate was 10% of completed patients; in the second project, the control rate was 7.4% (specified between 5 and 10% of participating centres). In the third project the rate is currently 5.8% but should finally settle to 5% of patients as in this project, with a run time >24 months, a total of 2,500 patient document sets is expected. Here, it was permitted to control not only the already completed document sets but also those still in the documentation process at the study centres in order to disclose systematic errors in the documentation and be able to draw attention to these where appropriate. In this observational study, feedback to all the participating doctors took the form of newsletters (see Attachment 1; the third issue of this newsletter is now available).

3.2 Project overview and control variables

See Table 2 (Tab. 2).

Table 2

Quality control measures for NIS

3.3 Duration of measures

The evaluation of the 4 projects listed in Table 2 (Tab. 2) with regard to the duration of the visits shows that an duration of 30 min. up to 1 hour (depending on the number of patient document sets to be controlled) remains within reasonable bounds, including for the site organisation. Decisive for the smooth running of the visit is the preparation of the meeting, both in the pharmaceutical company, with the preparation/completion of the forms and at the site.

The duration of these on-site measures is to be taken into account in setting the fees being integrated into the contract with the site [16].

3.4 Degree of consistency

For each project, parameters relevant to the study and safety-related data are controlled. It is checked whether the value requested is consistent, inconsistent or unavailable. The evaluation yielded the picture shown in Table 3 (Tab. 3).

Table 3

Consistency

The high degree of consistency, 95%, for project 2 was achieved in neurological specialist practices; in the studies with internal and general medical practitioners, this was lower with an average of 85%.

If we compare project 3 (carried out mainly in clinics) with project 4 (practitioners), there was significantly "more experience in dealing with studies" in this study in oncology (see Table 4 (Tab. 4)).

Table 4

Study center details

The logistical effort (appointment requests and cancellations, notifications, further enquiries etc.) for the pending visit for the study carried out exclusively with practitioners is, compared with the project carried out mostly in clinics – more than twice as high: on average, 1.5 doctors had to be contacted in the later case, in order to be able to carry out a visit on time; in the NIS carried out in the practitioner sector, an average of 3.5 doctors had to be contacted to find a suitable appointment.

In all projects, the paper-based medical records were maintained alongside electronic records in about 1/3 of cases.

In all quality measures carried out on site, enquiry was made as to the occurrence of serious adverse events (SAE). In only 2 cases (out of a total of 302 checked) did the treating physician forget to document AE/SAE as required in the study plan. In both cases, the required notification was immediately made or repeated.

3.5 Consequences

For project No. 1 (cohort study in the indication prostate carcinoma), the quality control measures carried out were, erroneously, strongly GCP-oriented. The forms with the characteristics and variables to be checked were very comprehensive so that the visit in the participating centres took a great deal of time.

In the beginning obviously there were not given sufficient information to the participating doctors regarding their documentation duties and retention period. Contrary to the requirement to keep for 10 years specified in the study plan, a few doctors had apparently already destroyed the patient consent forms, as a result of which a quality control, i.e. the direct check of the medical records, could no longer be carried out on the grounds of data protection. In these cases the check was done by interview technique.

This experience was the basis for the optimisation in the subsequent projects 2, 3 and 4 regarding adaptation of forms and reduction of the visit duration. The lack of awareness of doctors regarding documentation requirements could, in the future, lead to a special training (for instance on-the-job training by eLearning). In this respect, it could be considered whether this should be integrated in the study preparation. Especially for project No. 4, which was carried out exclusively with practitioners, it was shown to be of value to inform the local employee responsible in good time of the pending quality measure in his area. In conferences, it should be standard practice for the project leader, when presenting a new NIS, to draw attention to this task of the employee.

4 Discussion

When carrying out quality measures for non-interventional studies (NIS) the non-intervention principle must always take precedence. In individual cases, participating doctors or their staff made proposals for the design of documentation questionnaires. In many cases there was additional positive feedback on the carrying out of such quality measures, concerning also non-interventional studies. Given the experience so far, however, it is advisable when planning visits, to take every opportunity to draw the attention of the site to be visited to the necessity of collecting the signed patient consent forms and having the medical records ready so that an on-site visit can be carried out efficiently, given that clinics and practices are always short of time. It should be explicitly drawn to the attention of the doctor participating in a NIS that his participation may also involve a quality control visit of the documented data and that this measure will require the devotion of some time in his centre. The time required for this quality measure is to be taken into account when considering fees [16].

To provide a proper understanding regarding the time for which study documents (e.g. patient consent forms) are to be kept and with regard to data protection, study-specific training courses regarding the necessity of documentation are proposed. Since, in many indications (especially in the field of oncology) clinics and practices participate not only in clinical trials but also in non-interventional studies, it is proposed to expand the established investigator seminars with the part "NIS" and thus save both costs and time for all concerned. For projects of a longer duration, the participating sites should be kept abreast of the project status and informed of any pending quality aspect, e.g. by means of a newsletter.

In only 2 cases (<1%), it was found when checking the medical records that a notification of serious adverse events had been missed by oversight. Thus, when carrying out a NIS, the doctors provide excellent quality concerning their professional obligation to notify side effects/unexpected events.

A greater experience in clinics and large specialist practices correlates with "more staff specifically for documentation".

From our experience so far we can conclude that the quality of data in non-interventional studies – even though only samples has been controlled – is close to that of data from clinical (interventional) trials with a degree of consistency up to 95% [17].

Using the quality control described in this publication, the quality assurance required in the GEP guidelines can be fulfilled completely and high quality standards thus are established for epidemiological studies [6].

Notes

This publication is partly based on the seminar paper (mentor K. Wörz) "Quality Assurance Measures in Non-interventional Studies by Sanofi-Aventis – Descriptive Evaluation of Quality Controls in 4 Projects" by Nadine Geisler, Medical Documentation course at the IB-GISmbH Medical Academy, Rostock.

Competing interests

The authors declare that they have no competing interests.