Critical appraisal of fundamental items in approved clinical trial research proposals in Mashhad University of Medical Sciences.

Background: Writing, designing, and conducting a clinical trial research proposal has an important role in achieving valid and reliable findings. Thus, this study aimed at critically appraising fundamental information in approved clinical trial research proposals in Mashhad University of Medical Sciences (MUMS) from 2008 to 2014.
Methods: This cross-sectional study was conducted on all 935 approved clinical trial research proposals in MUMS from 2008 to 2014. A valid and reliable as well as comprehensive, simple, and usable checklist in sessions with biostatisticians and methodologists, consisting of 11 main items as research tool, were used. Agreement rate between the reviewers of the proposals, who were responsible for data collection, was assessed during 3 sessions, and Kappa statistics was calculated at the last session as 97%.
Results: More than 60% of the research proposals had a methodologist consultant, moreover, type of study or study design had been specified in almost all of them (98%). Appropriateness of study aims with hypotheses was not observed in a significant number of research proposals (585 proposals, 62.6%). The required sample size for 66.8% of the approved proposals was based on a sample size formula; however, in 25% of the proposals, sample size formula was not in accordance with the study design. Data collection tool was not selected appropriately in 55.2% of the approved research proposals. Type and method of randomization were unknown in 21% of the proposals and dealing with missing data had not been described in most of them (98%). Inclusion and exclusion criteria were (92%) fully and adequately explained. Moreover, 44% and 31% of the research proposals were moderate and weak in rank, respectively, with respect to the correctness of the statistical analysis methods.
Conclusion: Findings of the present study revealed that a large portion of the approved proposals were highly biased or ambiguous with respect to randomization, blinding, dealing with missing data, data collection tool, sampling methods, and statistical analysis. Thus, it is essential to consult and collaborate with a methodologist in all parts of a proposal to control the possible and specific biases in clinical trials.

↑ What is “already known” in this topic:

Application of clinical trial results depends on the precise interpretation of the study findings. Moreover, when inclusive information is available to design a study, conducting the study and data analysis are possible.

→ What this article adds:

A significant number of approved proposals in Mashhad University of Medical Sciences during 2008 to 2014 were biased and ambiguous in main principles of clinical trial studies (random allocation, blinding) including dealing with missing data, sampling methods, and statistical analysis.

Introduction

Writing research proposals forces research providers to think about their research question and its scope to elucidate their purpose, design the investigation, and review and assess the appropriateness of their methodology. Furthermore, writing a research proposal is an opportunity for referees or reviewers to propose necessary improvements about the feasibility and reasonableness of the research in costs, equipment, materials needed, and the time required to finish the project (1, 2). Indeed, the idea that writing a research proposal is a futile chore is wrong, because it is similar to map of buildings and guides researchers to operate and implement the research. In other words, it is not hard to do the next steps if research proposals are carefully designed and well written, meaning that research will result in production of science (scientific article).

Clinical trials have been identified as the strongest type of research design to evaluate the effects of health-related interventions, and therefore these study designs would appear on top of the evidence pyramid in validity of the findings (3-5). Based on the evidence, designing and using clinical trials have increased significantly in the recent years (6). Application of clinical trial findings depends on the precise interpretation of the study findings and it is possible when comprehensive information is available on how to design and conduct the study and analyze the data.

Writing a clinical trial research proposal has an important role in achieving valid and reliable findings. There are several guidelines and standard forms for writing research proposals for clinical trial studies. Almost all of them are unanimous about what items should be included in a research proposal; the items are as follow: an outline of the research aims/hypotheses; a summary of preceding researches on the research question; detail of the suggested study design and methods; and timeline (7). Each of these items consists of key components such as having an appropriate comparison group, proper sampling method, randomization, allocation concealment, and blinding, which is highly critical to achieve valid and practical results (8-9).

Considering the above-mentioned and the importance of writing, designing, conducting, and publishing the results of clinical trials, the present study aimed at critically appraising fundamental information in approved clinical trial research proposals in Mashhad University of Medical Sciences from 2008 to 2014. We discussed significant issues that are the Achilles' heel of clinical trial studies. We also highlighted the most common factors and items which affect the approval of research proposals and validity, reliability, and usefulness of their findings.

Methods

At the first step, purposeful sessions were held with professors, specialists, and experts in the field of research, biostatistics, and methodology/epidemiology as well as those responsible for the retrieval, collection, and extraction of data. Results of these sessions specified an overall view on the study subject that was agreed upon by researchers and colleagues including the main research aims, methods of achieving them, documents, and selecting a useful tool for the study. In the next step, all documents and evidences related to critical appraisal, quality assessment, and standard reporting tools for clinical trials were examined to achieve a comprehensive, simple, and usable tool. These investigations resulted in specifying the fundamental items in writing clinical trial research proposals and designing a checklist as a research tool. To ensure the validity of the research tool, brainstorming sessions with experts and specialists in the field of research was done again. The validity of the research tool was determined by the content validity method. After corrections, the content validity index was estimated at 0.85 and resulted in the final tool, consisting of 11 main items.

To achieve a reliable research tool before the onset of the study, 3 reviewers, who were responsible for extracting and recording data were trained and differences or discrepancies between them were resolved by consulting with the team’s methodologist. Reliability of the research tool was assessed using agreement rate. Agreement rate between reviewers was assessed during 3 sessions, and Kappa statistics was calculated at the last session as 97%, showing a relatively good status for reliability. Further sessions were held for the continuation of the agreement between reviewers with randomly selecting some issues in the presence of a methodologist. Decision on consensus had been made in consultation with a methodologist about issues where reviewers disagreed.

In each main item, issues related to the research project specifications, research title, information related to executive research team, study design, aims and hypothesis, information on the research methodology, variables specification, data analysis and statistical methods, ethical considerations, timetable, and finally the cost of the research project were assessed, respectively.

Some of the key issues related to the main items that were assessed are as follow: registration in IRCT (Iranian Registry of Clinical Trials) and having a research council code; pointing to the name and type of study design; having a methodologist consultant; an applied research; appropriateness of the aims with hypothesis, sample size formula, and the required components to calculate it; sampling method; the accuracy of the data collection tools; follow- up; dealing with missing data; definition of the intervention and outcome; and issues related to improving the quality of clinical trials such as randomization, blinding, and having a control group.

The process of extracting data from the research proposals was conducted regardless of the researcher’s name and specifications and in compliance with the principle of trusteeship.

Results

The highest number of approved research proposals was found in 2013 (Fig. 1). Only the research proposals of the first 3 months of 2014 were assessed in this study. The end of the project time was determined in less than half of the approved proposals (32.5%). Type of study design had been stated in the title in a small percentage of research proposals (5.8%).

Fig. 1

Trend of Approved Clinical Trial Research Proposals from 2008-2014 (only the first 3 months of 2014)

A large part (67%) of the approved research proposals belonged to School of Medicine, and School of Nursing and Midwifery was second in rank (12.8%), with a significant gap. Based on the findings, School of public health had the least number of clinical trial proposals. Institutions outside the university participated in only 2% of the approved research proposals. More than 60% of the research proposals had a methodologist consultant, and type of study or study design had been specified in almost all of them (98%). Type of study (basic or applied) had been specified in most research proposals. Coordination and Appropriateness of study aims with hypotheses was not observed in a significant number of research proposals (585 proposals, 62.6%).

Sampling method was stated in 858 projects (92%), of which 178 (21%) used probability sampling and the others nonprobability sampling method. Simple random sampling was the most common probability sampling method (81%). Findings revealed that the required sample size for 66.8% of the approved proposals was based on a sample size formula; however, in 25% of them, sample size formula was not in accordance with the study design. The main source (pilot, similar studies) used to calculate the sample size was not mentioned in a significant number of research proposals (26.4%).

Results indicated that data collection tool was not selected correctly in 55.2% of the approved research proposals. Since clinical trials were considered in this study, assigning individuals into study groups was done by random allocation method/randomization only in 62% of research proposals. Type and method of randomization were unknown in 21% of the proposals, although 48% used simple randomization. Dealing with missing data had not been described in most proposals (98%). Type of intervention in most studies was pharmaceutical, surgical, and educational, respectively. Among the items that had been mentioned in almost all proposals, inclusion and exclusion criteria were (92%) fully and adequately explained.

Practical/applied definition of the variables was mentioned in nearly half of the research proposals. The role, type, and scale of variables were defined correctly in 72.4%, 72%, and 56% of proposals, respectively. In almost half of the approved research proposals, specific aims were in accordance with the table of variables. More than 90% of the research proposals explained about statistical analysis methods and merely 25% of them were good in accordance with the study design. In this case (correctness of the statistical analysis methods), 44% and 31% of the research proposals were moderate and weak in rank, respectively. SPSS, SAS, R, and MLwin statistical softwares had been listed in research proposals, and SPSS was the most popular among them. Informed consent form had been attached in 62.6% of the proposals (Table 1).

Table 1

Main Fundamental Items Assessed in the Approved Clinical Trial Research Proposals

Items and subitems	Yes, N (%)	No, N (%)
1. Project specifications
1-1. Registration in IRCT	3 (0.3)	932 (99.7)
1-2. Having research Code in Research Council of MUMS	935 (100)	0 (0.0)
2. Research title
Statement of study type in the research proposal title	54 (5.8)	881 (94.2)
3. Information related to investigators
The project has methodologist consultant	591 (63.3)	344 (36.7)
4. Type of study design
Specifying the type of the interventional study design	915 (97.9)	20 (2.1)
5. Aims and hypotheses
5-1. Statement of the main aims of the study	884 (94.5)	51 (5.5)
5-2. Appropriateness of the aims with hypothesis	35 (37.4)	585 (62.6)
6. Information on methods of study implementation
6-1. Having sampling method	858 (91.7)	77 (8.3)
6-2. Type of sampling method	Probability 178 (20.7)	Non-probability 680 (79.3)
6-3. Having sample size formula	625 (66.8)	310 (33.2)
6-4. Suitability and correctness of sample size formula	469 (74.7)	156 (25.3)
6-5. Using source, pilot, and indicators in sample size calculation	688 (73.6)	274 (26.4)
6-6. Describing the methodology of plan	856 (91.6)	79 (8.4)
6-7. Selecting the accurate and valid data collection tool	419 (44.8)	516 (55.2)
6-8. The full definition of the Intervention	777 (83.1)	158 (16.9)
6-9. The full definition of the outcome	738 (78.9)	197 (21.1)
6-10. Statement of the follow-up period	735 (78.6)	200 (21.4)
6-11. Statement of dealing with missing data	16 (1.7)	919 (98.3)
6-12. Considering the Wash-out period in crossover trials	16 (88.9)	2 (11.1)
6-13. Having blinding	185 (19.8)	750 (80.2)
6-14. Having randomization/random allocation	582 (62.2)	353 (37.8)
6-15. Randomization procedure	Simple 279 (47.9)	Blocked 32 (5.5)	Other 271 (46.6)
6-16. Statement of inclusion and exclusion criteria	860 (92.0)	75 (8.0)
7. Table of variables
7-1. The accurate definition of variable type	672 (71.9)	263 (28.1)
7-2. The accurate description of variable role	677 (72.4)	258 (27.6)
7-3. The accurate description of variable scale	523 (55.9)	412 (44.1)
7-4. Specific aims is in accordance with variables table	451 (48.2)	484 (51.8)
8. Statistical analysis methods
8-1. Having a statistical analysis method	850 (91.0)	85 (9.0)
8-2. Correctness of the statistical analysis method (categorized by statistician opinion)	Good 217 (25.6)	Moderate 369 (43.4)	Weak 264 (31.0)
8-3. Specify the statistical software	548 (58.6)	387 (41.4)
9. Ethical considerations
Taking into account the ethical considerations	836 (89.4)	99 (10.6)
10. Informed consent
Attachment of informed consent form	586 (62.6)	349 (37.4)
11. Timetable
The average time considered for the study (month)	Mean (SD): 13.8 (5.4)

Discussion

Bias in designing and conducting a study can greatly affect its quality, and ultimately making the results invalid (11). Selection bias is one of the most important biases and randomization is the best way to control it in clinical trials. In the present study, assigning individuals into study groups were done by random allocation only in 62% of the assessed research proposals. Also, method of randomization was unknown in quarter of them, although nearly half of them had used simple randomization. Proper randomization procedures in obtaining valid and reliable results is important to the extent that Moher et al. study results showed that studies that had used incorrect random sequence generation/random allocation, overestimated the estimated effect as much as 30% to 50%, compared with studies that used a correct method (12). In fact, it can be stated that low quality studies tend to exaggerate and overestimate their findings (13).

Blinding was not described in most of the approved proposals. In other words, 80% of the approved proposals, incorrectly, had not pointed to blinding of patients, researchers, or assessors (analyzers), while blinding is highly essential in clinical trials, as blinding can be helpful in reducing performance bias. It helps achieve real results and reduce adverse effects resulting from awareness and attitude of participants and researchers toward the study results. Also, it can prevent measurement bias in the study by blinding of the assessor as much as possible (14). Most researchers make a mistake between blinding and allocation concealment. Such mistakes show a lack of adequate knowledge of these 2 concepts (15). Pildad et al. study revealed that not using blinding in clinical trials lead to overestimating the estimated effect as much as 9% (16).

It seems that failure to register the proposals in IRCT, inappropriateness of the aims with hypothesis, invalid data collection tool, and weaknesses in randomization and blinding occurred due to the absence or insufficient presence of methodologists/epidemiologists in research proposals.

How to deal with missing data had not been described in 98% of approved proposals. It should be noted that to reduce biases in clinical trials, in addition to the foregoing methods, it is necessary to predict and anticipate efficient approach to reduce attrition bias. Missing data that arise due to individual attrition or withdrawal and exclusion from a study cause a bias in effect size estimate (16). In fact, the removal of the results of samples with missing data in the statistical analysis phase causes the results to be in favor of the intervention group falsely compared to when all the data were analyzed (17). Hence, all clinical trials should consider attrition, withdrawal, and exclusion of patients from the study and predict methods/approaches to control them.

According to weaknesses observed in the cases of sample size formula and its calculation procedure, sampling methods, dealing with missing data, description of study variables, and statistical analysis methods, it is recommended that all researchers, particularly young researchers, consult with statisticians. It is necessary that researchers use careful and complete statistician counseling at various stages of the study including writing, designing, conducting, and reporting findings, and not only in 2 stages of sample size calculation and data analysis, to help increase internal and external validity of the study.

Due to the necessity of registration of clinical trials in IRCT, it is clear that a large part of the methodological weaknesses of research proposals can be resolved. Perhaps, failure to register the clinical trial proposals in the IRCT was one of the factors that caused the observed weaknesses in these proposals. There was no particular problem about defining intervention and outcome, specifying the type of the interventional study design, describing inclusion and exclusion criteria, and taking into account the ethical considerations in approved research proposals.

The further limitation of this study was the lack of a national or international standard tool or checklist to evaluate critical appraisal or quality assessment of clinical trial research proposals. There was no other limitation with regards to the descriptive nature of the present study and access to all clinical trial proposals in the registration system (Pajouhan System).

Conclusion

According to the results of this study, it is suggest that researchers consider the highest standards in writing and designing clinical trial proposals. This will help the researchers to improve research and provide a good report of clinical trials. Approval of studies should depend on registration in IRCT or in one of the international registration centers of clinical trials. It is necessary to consult and collaborate with a methodologist to control the possible and specific biases in clinical trials.

Conflict of Interests

The authors declare that they have no competing interests.