A comparative study to evaluate quality of data documentation between investigator-initiated and pharmaceutical industry-sponsored studies.

PURPOSE: In pharmaceutical industry funded clinical studies (PIS), there is rigorous monitoring to ensure adequate and accurate data documentation. In comparison, the investigator-initiated studies (IIS) often lack in resources and may not follow such quality checks. At present, very limited data on the existing deficiencies in documentation for IIS are available. Hence, the present study assessed data quality in IIS relative to those funded by the industry to identify and address issues in data documentation.
MATERIALS AND METHODS: We evaluated records of 1276 participants in 13 studies (5 - industry sponsored and 8 - investigator initiated) conducted during 2009-2015 using a prevalidated checklist. The percentage total scores for overall documentation and general trial-related and patient-specific documents were calculated. The percentage total scores within the patient-specific documents were also calculated and compared. Between-group score analysis was done by Student's t-test using GraphPad InStat version 5.0.
RESULTS: The mean (standard deviation [SD]) percentage total score for the IIS was 80.96 (13.26) and that for PIS was 98.77 (1.84) (P = 0.01). For IIS, the total percentage scores ranged from 63% to 100% while it was above 95% for all PIS. For general trial-related documents, the mean (SD) percentage total score for IIS was 90.39 (13.26) while that for PIS was 97.38 (0.92) (P > 0.05). In the patient-specific documents, IIS scored 80.52 [14.41] versus 98.95 (1.98) for PIS (P = 0.016). The scores for IIS versus PIS (70.22 [21.6] and 99.36 [1.43]) within patient-specific documents were significant only for admission criteria (P = 0.016).
CONCLUSION: Quality of IIS needs to be addressed by greater oversight and periodic quality control assessments. Copyright:

INTRODUCTION

The International Conference on Harmonization–Good Clinical Practices (ICH-GCP) recommends generation of credible data in clinical trials.[1] However, despite this, lack of reliable, accurate, and adequate source documentation is a common inspection finding at clinical trial sites.[23] Systemic deficiencies in documentation may lead health authorities to ask for exclusion of such data from final analyses and regulatory submissions. Identification of these deficiencies is largely seen in pharmaceutical industry-funded clinical studies (PIS) which follow a system of rigorous monitoring and audits. In comparison, investigator-initiated studies (IIS) lack in resources and may not have similar quality checks.[4] IIS are, however, equally important for the reason that they reassess safety and effectiveness of already approved drugs and explore the use of marketed drugs for new indications in different dosage regimens among others.[5]

Studies in literature have emphasized the need to improve methodological and reporting quality of both PIS and IIS.[46] An exhaustive literature search revealed scant evidence regarding comparison of data integrity of IIS vis-a-vis PIS. Hence, we conducted the present study with the objective of comparing the quality and completeness of documentation in IIS versus PIS done at a tertiary referral center.

MATERIALS AND METHODS

Ethics

The Institutional Ethics Committee (IEC) approval was taken before the study was conducted (EC/OA-08/2015). A consent waiver was obtained for the study from the IEC as this study was a retrospective audit. Confidentiality was maintained using unique identifiers.

Study selection criteria

The archival ledger of the department was retrieved and studies completed between the years 2009 and 2015 were considered for assessment. No studies were excluded.

Type of studies assessed and sample size

A total of n = 13 studies (5 – PIS and 8 – IIS) formed the study sample.

Pharmaceutical industry-sponsored studies

These were five in number and all five were interventional in nature. One was a Phase I and four were Phase II/III studies.

Investigator-initiated studies

These were eight in number – three were interventional in nature and the remaining five were observational. Of the interventional studies, one was a bioequivalence study and two were pharmacokinetic evaluations.

Study instrument

A prevalidated and published checklist[4] was used to assess quality and completeness of the documentation. We assessed general trial-related and patient-specific documents. For general trial-related documents, the following elements/subitems were assessed: presence of protocol, ethics committee correspondence, various study logs, and patient safety data. For patient-specific documents, we assessed inpatient information (source documents including outpatient and inpatient papers of the hospital), informed consent documents, source documents, case record forms (CRFs), and protocol and safety parameters.

Study procedure

Each study document was scored independently by authors BF and KS. These were verified by the senior authors NG and UT. Disagreements related to scoring were resolved through consensus.

Scoring

Each element/subitem in the checklist for both general and patient-specific documents was scored out of a total of “1.” When documents were present but were incomplete, a score of “0.5” was assigned. Zero was the score given when the document was missing.

General documents

Due to the variation in applicable subitems for the different studies, the total possible score (TPS) for each study was different, and thus, percentage scores were calculated. For example, the subitem “investigator brochure” was not applicable to some IIS studies.

Patient-specific documents

The number of patients for each study varied and the TPS for each subitem also differed for each study. The TPS for each subitem was given as TPS = (score given for the subitem present, incomplete, or absent) × (number of participants). For example, if a study had 22 participants, for the subitem “approved version of the informed consent,” the maximum possible score was 22. If the approved version of informed consent was present for all participants the total score would be 22. Similarly, total percentage scores were calculated for all the subitems. When marks were deducted, the reasons were noted.

The overall documentation for the studies was also assessed. This was done by adding the scores obtained for the general and patient-specific documents.

Outcomes of interest

Primary

These included (a) total scores for overall documentation, (b) general trial-related documents, and (c) patient-specific documents.

Secondary

These were individual elements within patient-specific documents such as admission criteria (inpatient history sheet and eligibility criteria), informed consent, CRF, source documentation (laboratory reports, patient history sheets, and details of vital parameters), and protocol and safety parameters (number of protocol deviations reported, documentation of serious adverse events (SAEs), and AEs). Furthermore, documentation deficiencies were collated.

Statistical analysis

Descriptive statistics

The demographic data for the number of participants in the studies were expressed in absolute numbers and percentage. Scores were assessed for normality using the Kolmogorov–Smirnov test. The percentage total scores were calculated and expressed as either mean (standard deviation [SD]) or median (range) based on whether scores were normally distributed or otherwise.

Inferential statistics

The percentage scores between the IIS and PIS were assessed using Mann–Whitney U-test or unpaired Student's t-test[7] based on the normality of the score. All the statistical analysis was performed using Graphpad Instat version 5.0 (Graphpad Software, La Jolla, California, USA, 2007) at 5% significance.

RESULTS

Demographics

In the 13 studies, a total of n = 1276 participants were enrolled. Of these, 959/1276 (75%) were enrolled in IIS and 317/1276 (25%) in PIS.

Overall percentage total scores [PTS]

The mean (SD) percentage total score for the IIS was 80.96 (13.26) and that for PIS was 98.77 (1.84) (P = 0.01). For IIS, the total percentage scores ranged from 63% to 100% while it was above 95% for all PIS. For general trial-related documents, the mean (SD) percentage total score for IIS was 90.39 (13.26) while that for PIS was 97.38 (0.92) NS. In the patient-specific documents, IIS scored 80.52 (14.41) versus 98.95 (1.98) for PIS (P = 0.016). The details of the scores are given in Tables 1 and 2.

Table 1

Summary of the overall scores obtained for the studies

IIS	Type of study	Score obtained	Total possible score	Percentage total score	PIS	Type of study	Score obtained	Total possible score	Percentage total score
IIS1	Observational	2667	3193	83.52	PIS 1	Interventional	2372	2385	99.45
IIS 2	Observational	1177	1455	80.89	PIS 2	Interventional	2173	2177	99.81
IIS 3	Observational	1416.5	2130	66.50	PIS 3	Interventional	481	482	99.79
IIS 4	Interventional	133	189	70.37	PIS 4	Interventional	1160.5	1215	95.51
IIS 5	Observational	1892	2076	91.13	PIS 5	Interventional	280	282	99.29
IIS 6	Observational	2075.5	3281	63.25
IIS 7	Interventional	391	391	100
IIS 8	Interventional	543	590	92.03
			Mean (SD)	80.96 (13.26)*				Mean (SD)	98.77 (1.84)*

*P=0.01 using unpaired t test. IIS=Investigator-initiated studies, PIS=Pharmaceutical industry-sponsored studies, SD=Standard deviation

Table 2

Summary of percentage total score obtained: general trial-related and patient-specific documents

IIS	Score obtained	Total possible score	Percentage total score (score obtained/total possible score × 100)	PIS	Score obtained	Total applicable score	Percentage total score (score obtained/total possible score × 100)	P
General trial-related documents
IIS1	31	32	96.87	PIS 1	47	48	97.9	0.27*
IIS 2	17	24	70.83	PIS 2	47	48	97.9
IIS 3	31	32	96.87	PIS 3	44	45	97.7
IIS 4	32	33	96.96	PIS 4	42	43	97.67
IIS 5	32	34	94.11	PIS 5	45	47	95.74
IIS 6	33	33	100
IIS 7	39	39	100
IIS 8	27	40	67.5
		Mean (SD)	90.39 (13.26)			Mean (SD)	97.38 (0.92)
Patient-specific documents
IIS1	2636	3116	84.6	PIS 1	2325	2337	99.48	0.016*
IIS 2	1160	1431	81.06	PIS 2	2126	2129	99.85
IIS 3	1385.5	2098	66.03	PIS 3	437	437	100
IIS 4	101	156	64.74	PIS 4	1118.5	1172	95.43
IIS 5	1860	2042	91.08	PIS 5	235	235	100
IIS 6	2042.5	3248	62.88
IIS 7	352	352	100
IIS 8	516	550	93.81
		Mean (SD)	80.52 (14.41)			Mean (SD)	98.95 (1.98)

*Significant using unpaired t-test. IIS=Investigator-initiated studies, PIS=Pharmaceutical industry-sponsored studies, SD=Standard deviation

Scores for individual elements within patient-specific documents

The scores for IIS versus PIS were 70.22 (21.6) and 99.36 (1.43) for admission criteria (P = 0.016); 85.26 (12.33) and 97.17 (6.31) for informed consent (P = 0.07); 93.87 (9.03) and 91.27 (19.17) for source documents (P = 0.74); and 60.34 (27.97) versus 90.81 (18.77) for CRF (P = 0.06), respectively [Figure 1]. All analysis done using unpaired t test.

Figure 1

Scores obtained within the patient-specific documents. *Denotes significance using unpaired ttest

Documentation deficiencies

Among IIS, 7/8 did not have all the amended versions of the protocol filed in the Trial Master File, the “final approved IEC version” of the protocol stamp was missing in 4/8 studies, reference ranges were not filed in 3/8 studies, and study logs (for example, screening, enrollment, and investigational product administration log) were missing in 7/8 studies. None of these deficiencies were seen in PIS.

DISCUSSION

The present study compared the quality of data documentation in IIS versus PIS and found that the overall quality of documentation was better in PIS (percentage total score: 98.77 vs. 80.96) relative to IIS, and this difference was statistically significant (P = 0.01). This was driven by the difference in the scores in the patient-specific documents between the two types of studies while scores for the general trial-related documents did not differ significantly.

The difference in overall quality of documentation between IIS and PIS could be attributed to the fact that IIS do not undergo the rigorous monitoring that PIS routinely undergo. The quality of documentation as measured by the percentage total score for general trial-related documents did not vary significantly between the PIS and IIS (97.38% vs. 90.39%). This observation is similar to a study by Patwardhan et al. who evaluated the quality and completeness of data between one industry-sponsored and one investigator-initiated studies.[4] The reason why there are no differences in the general trial documents is likely to be due to the fact that most of these documents such as study protocol, informed consent documents, letters of correspondence with the ethics committees, CRFs, study staff's curriculum vitae, and GCP certificates are maintained as an integral part of the IEC submissions and are driven by IEC standard operating procedures (SOPs), thus mandating compliance.

The quality of documentation for patient-specific documents was better in PIS relative to IIS (98.95 vs. 80.96%; P = 0.02). This has serious implications as the key mandates of GCP are data integrity and patient protections. Our results highlight the need to improve documentation and monitoring for IIS. In addition, investigator sites must have and implement site-specific SOPs to ensure better patient documentation and GCP compliance. Another way to do this is to allocate study-specific monitors even in IIS further emphasizing the need for fund allocation for this activity. IEC monitoring these studies regularly also would help ensure compliance and improve quality of documentation.

Within the patient-specific documents, we found that the scores were higher for PIS relative to IIS regardless of the nature of the document (admission criteria, informed consent, source documents, and CRF), though it was significant only for admission criteria (70.2% for IIS vs. 99.36% for PIS; P = 0.02). The score was also low (60.34% for IIS as against 90.3% for PIS) for CRFs but not statistically significant (P = 0.07).

The low scores seen for informed consent documents are a matter of concern since informed consent is an important document to protect the participant's safety and rights and is an index of the autonomy. Literature shows that deficiencies related to patient-specific documentation are seen in IIS as well as PIS. A review of the warning letters issued by the US Food and Drug Administration (FDA) to the investigators between the years 2005 and 2010 found that inconsistencies in documentation of informed consent were the third most common reason behind issuing of warning letters.[8] Similarly, source documentation errors were listed among the top ten findings by the European medicines Agency.[9] Another study that analyzed the warning letters issued to the clinical trial investigators by the US FDA found that violations related to informed consent and CRF documentation were among the main reasons for the issue of these letters.[10]

Our study is limited by the fact that it is restricted to research conducted at a single center and thus may have limited generalizability. Furthermore, there is a heterogeneity in the studies evaluated, especially among the IIS, where the 5/8 of the studies were observational in nature whereas all PIS were interventional.

The ICH-GCP recommends “accurate capture, storage and reporting of data” for all documents.[1] However, compliance with these recommendations requires significant investment in terms of resources. The IIS quite often lack adequate workforce to carry out intense reviews and arduous monitoring of the trials. However, despite this, the documentation deficiencies must be addressed as accurate and diligent data collection along with storage and reporting could impact analysis and interpretation of the results.[11]

CONCLUSION

Maintaining GCP is essentially an attitude, and therefore, regardless of the type of study, adequate steps must be taken to ensure quality data capture and documentation.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.