Matching research design to clinical research questions.

The importance of randomized controlled trials (RCTs) versus observational studies has been debated for several years. However, the question is not whether RCTs are better than observational study designs. RCTs certainly provide the most unbiased answers in scenarios where it is logistically and ethically feasible to conduct both RCTs and observational studies. That is, study design is not a choice but a function of matching the research question to provide the most unbiased answers. The basic concept that underpins every clinical research project is the requirement of a clearly defined research question domain. Broadly, the clinical research question domain relates to prognosis, diagnostic accuracy, treatment or adverse events. While RCTs provide the most unbiased answers on questions related to the efficacy of treatments, other designs are better suited to answer questions related to prognosis or diagnostic accuracy of tests. In this paper, we illustrate the significance of matching study design to the research question domain while using clinical scenarios as an example. Although there are several other question domains that also concern the practice of medicine, we are only focusing on study designs concerning the issue of prognosis and diagnostic accuracy in this paper.

INTRODUCTION

The debate on the importance of randomized controlled trials (RCTs) versus observational studies has been ongoing for several decades.[1] Consequently, there are proponents and opponents of both study designs. However, in the debate of superiority of observational versus RCTs or vice versa, a key message is lost. The question is not of whether RCTs are better than observational study designs. The issue that needs to be emphasized is that while RCTs provide the most unbiased answers in scenarios where it is logistically and ethically feasible to conduct RCTs as opposed to observational studies. Nevertheless, it is not always the case to conduct all study designs for all clinical questions due to various ethical and logical reasons. That is, study design is not a choice but a function of matching the research question to the study design that will provide the most unbiased answers. The basic concept that underpins every clinical research project is the requirement of a clearly defined research question domain.[2] Broadly, the clinical research question domain relates either to prognosis, diagnostic accuracy, treatment, or adverse events. Research on the type of questions shows that a majority relates to diagnosis and treatment.[3] While RCTs provide the most unbiased answers on questions related to efficacy of compared treatments, other designs are better suited to answer questions related to prognosis or diagnostic accuracy of tests. In addition, rarely an answer to a clinical question is obtained from one study and accordingly results from systematic reviews and meta-analysis of several studies are considered more reliable than results from single studies.[1] Evidence-based medicine classifies different types of studies on the basis of research design as the criterion for hierarchical rankings.[4] RCTs or systematic reviews and meta-analysis of RCTs are at the top of the pyramid, while anecdotal evidence is at the bottom[5] [Figure 1]. However, the evidence pyramid does not provide the disclaimer that all study designs are not possible to conduct to answer all research questions.[6] The aim of this paper is to illustrate the importance of matching the study design to the research question domain while using clinical scenarios as an example. While there are several other question domains that also concern the practice of medicine, we are focusing only on study designs concerning the issue of prognosis and diagnostic accuracy in this paper.

Figure 1

Hierarchy of evidence pyramid

Overview of research study designs

Overview of research study designs is illustrated in Figure 2. Overall, the study design tree can be divided into two broad categories of descriptive and analytical studies.[7] While descriptive studies provide us with the information on the occurrence of certain outcomes over time, analytical studies quantify the association between a predictor and an outcome variable. Case reports, case series, and cross-sectional surveys are the most commonly used descriptive study designs and are helpful in generating a hypothesis.[8] A case report or a case study is the most basic descriptive study design and focuses on an individual with an unusual disease.[9] A sequence of case reports is called case- series which is when more than one case is reported in a certain time frame and studied in detail. Descriptive cross-sectional (survey based) are conducted to estimate the frequency and distribution of a disease. In these surveys, information is collected on several individuals at one point in time about their health status, behavior, or other risk factors. The key distinction between descriptive and analytic studies relates to the quantification of data. Descriptive studies provide the frequency of events without any statistical analyses.

Figure 2

Basic research designs

Analytical studies are divided into two main categories of observational and experimental designs, as shown in Figure 2. The key difference between observational and experimental study designs relates to the assignment of exposure. In experimental studies, researchers make a decision on the assignment of exposure or intervention, while in observational studies exposure is not assigned by the researcher.[68] Experimental study designs include randomized controlled trials (RCTs). Based on the study design, RCTs can be classified into cross-over, parallel, and factorial designs. RCTs allow the most unbiased comparison between the intervention group and the control group. Randomization is the only known mechanism to create balanced groups in all aspects except the exposure or intervention for known and unknown prognostic factors.[6] However, RCTs like any other designs are prone to biases if not implemented appropriately. Questions related to efficacy of competing treatments are best answered using RCTs. We are intentionally not providing details on conduct of RCTs as the focus of this paper is on observational study designs related to question domains of prognosis and diagnostic accuracy.

In observational studies, as the name suggests, a specific population is observed versus assigned and the information is collected on outcomes according to exposure. That is, unlike experimental study designs, the investigator does not control the assignment of exposure and is only involved passively in collecting data on exposure followed by outcomes assessments. The three types of observational study designs are cohort, cross-sectional, and case-control. While cohort and cross-sectional studies are prospective in nature, case-control studies are retrospective. That is, in the case of cohort and cross-sectional studies, exposure precedes the outcome, whereas in case-control studies, outcomes precede the exposure.

A classic prospective cohort study design examines the association between health exposure and resultant outcomes. For example, a cohort study is helpful in estimating the incidence and etiologies of diseases as well as questions on prognosis. Study subjects are grouped according to exposure status and followed over time to determine the incidence of outcome. The incidence of the disease in the exposed group is compared with the incidence of disease in the unexposed group. Properly designed and conducted cohort studies offer several advantages. A key advantage is showing an appropriate temporal sequence between exposure and outcome. Since exposure precedes outcome, it is easier to attribute the outcome to the exposure and is therefore best suited to assess an association between exposure and outcome. Furthermore, cohort studies allow for the direct calculation of incidence rates in the exposed and unexposed groups summarized as risk ratios or relative risk.[8] Cohort studies are essentially key to address research questions associated with rare diseases due to the fact that exposure is determined first. It can be determined a priori if there is an adequate number of exposed and unexposed subjects [Figure 3]. Some of the disadvantages of cohort studies include requirements of large sample size and longer time duration for follow-up. A cohort study is not feasible in cases of chronic diseases where the latency period is prolonged as well as in cases of rare diseases where the incidence is low.[9] It is also important to note that due to the long time needed to follow-up, reassessment of exposure status is required. This has the potential to change during the course of follow-up leading to association with significant costs and logistical issues. Due to the long course of follow-up, loss to follow-up is also a major challenge and a potential source of bias.

Figure 3

Prospective cohort study design

Analytical cross-sectional studies are useful to determine prevalence, which is important for clinical practice. Prevalence provides information on the likelihood of a diagnosis and the predictive value of a diagnostic test. While differentiating cohort and cross-sectional studies, we can use the analogy of a picture versus a video. Cohort studies are similar to a video which allows viewing the whole scenario over a period of time and permit multiple measurements unlike cross-sectional studies which are like a snapshot and permit measurement only at one time.[10] That is, cross-sectional study design involves observation of a subset of a population for studying exposure and outcome variables at one point in time without any follow-up. Cross-sectional study design is useful to assess the diagnostic accuracy of compared tests [Figure 4]. For example, the same study subject is given two diagnostic tests and associated observations with each test in terms of true positive, true negative versus false positive, or false negative are determined. In cohort or case control studies, there are two groups with different observations whereas in a cross-sectional study design, there are two observations from the same group. The most important advantage of cross-sectional studies is that they require no follow-up and therefore are relatively cheap to conduct requiring fewer resources. However, cross-sectional studies are not adequate in differentiating between cause and effect or the sequence of events.

Figure 4

Cross-sectional study design

In contrast to cohort and cross-sectional studies, case control studies are retrospective. A case control study design provides an estimate on the strength of association between each predictor variable and the presence or absence of a disease. Retrospectively, subjects with certain outcome or disease are selected and then matched with the normal controls. Information is obtained from these subjects to estimate the prevalence of exposure to risk factors [Figure 5]. Odds ratio is an appropriate measure to estimate the association between an exposure and an outcome. The major strength of a case control study is that it can provide rapid and useful information on a smaller sample size. There can be a problem of recall and selection bias in this type of design as an investigator relies on past information to determine the exposure status.[6] For rare diseases or the diseases that have a long latency period between an exposure and outcome, case control studies are often the only feasible choice.[9]

Figure 5

Case control study design

We will now discuss here two clinical scenarios to illustrate the importance of matching the study design with the research question.

CASE 1

Clinical scenario

A 30-year-old woman presents with progressive weakness of both legs for last 10 days. She has difficulty in walking, climbing stairs, and standing up from a sitting position. She has developed decreased sensation up to the level of umbilicus and urgency of micturition. All muscles of both lower limbs have decreased tone and power (grades 3 to 4/5) with flexors weaker than extensors and absence of involuntary movements. There is a graded (distal more than proximal) sensory loss involving all modalities of sensation with a sensory level of D10. All deep tendon reflexes are elicitable and plantars are extensor. Her abdominal reflexes are all absent and there is no spinal tenderness. An MRI of the dorsal spine rules out a diagnosis of dorsal transverse myelopathy. Subsequent MRI of the brain reveals seven white matter T2 hyperintensities of variable size and location. There is no enhancement with gadolinium and antimyelin antibodies are positive. A diagnosis is made of “demyelinating disease of the central nervous system” and intravenous methylprednisolone is prescribed. The patient is informed that her symptoms and test results will be monitored over time. She would like to know what her chances are of developing multiple sclerosis (MS) in the next few years.

As evident from the clinical scenario, the primary question or concern of the patient pertains to the domain of prognosis. That is, among patients (problem) with the described symptoms (intervention/exposure) and those who do not have these symptoms (control/no exposure), the question pertains to the incidence of MS (outcome) over time. It is obvious that in this case, an RCT would be impossible to conduct. Since the presence of symptoms cannot be manipulated by an investigator, an experimental design can be ruled out. In this case, investigators cannot randomize patients according to certain symptoms which brings observational study designs into consideration. Another possible observational design is the cross-sectional study, which examines both exposure and outcome at the same time in the same group of people. This is clearly not a relevant study design for this scenario since the aim is to measure the incidence of the outcome between two groups over time. The next possibility is a case-control design. In the case-control study design, participants could be separated into those with MS and those without and then go retrospectively to assess the differences in terms of presence or absence of risk factors between the two groups. While this study design is feasible, there are two limitations. First, because the primary interest is associated with several risk factors (i.e., composite outcome), this study could be subject to recall bias.[8] Second, incidence cannot be calculated in a case-control study as patients cannot be followed retrospectively. This leaves the cohort study as the best study design to answer the clinical research question on prognosis. In this scenario, subjects would be separated into those that have the described symptoms and those that do not. They would then be followed over time and the incidence of MS in the exposed group would be compared to the incidence of MS in the unexposed group. From this data, the relative risk would be calculated.

CASE 2

A 37-year-old pregnant woman comes to the clinic for a routine checkup. This is her second pregnancy. She had her first child when she was 35 years old and had an amniocentesis at 18 weeks to test for Down syndrome. The amniocentesis was negative. She is currently 8 weeks pregnant and would like to know as soon as possible of any abnormalities. An ultrasound can be done in the first trimester for diagnosing Down syndrome, but it is unknown if the results are as reliable as the conventional test of amniocentesis.

This scenario represents the domain of diagnostic accuracy. The question can be formulated as follows: Among pregnant women over the age of 35 (patient/problem), what is the accuracy of ultrasound (intervention/experimental test) compared to amniocentesis (control/conventional test) in detecting Down syndrome in fetus (outcome)? Again, the RCT can be eliminated because, in order to determine diagnostic accuracy in an unbiased manner, both tests need to be administered to the same subset of the population. In the case-control study, subjects with the outcome of interest are selected and then matched with healthy controls. Since we do not know whether or not the outcome of Down syndrome is present, a case-control study cannot be used. The cohort study group's subjects are selected based on exposure status followed over time and the incidence of the outcome is calculated. In this clinical scenario, no exposure is described and we are not interested in calculating the incidence of Down syndrome. Accordingly, the characteristics of the cross-sectional study make it the best choice to answer this clinical research question. A cross-sectional study would allow two observations from the same group to be determined. In this case, a sample of pregnant women over the age of 35 would be given both an ultrasound and amniocentesis. The results of the tests would be recorded and the number of true positives and negatives versus the false positives and negatives would be calculated along with positive and negative predictive values.

SUMMARY

In this paper, we primarily focus on the observational study designs of cohort, case-control, and cross-sectional studies. Through the overview of study designs and demonstration of case scenarios, it is clear that the choice of study design is a function of the clinical research question. While RCTs provide most unbiased answers to questions related to efficacy of competing interventions, they are not suited to answer questions related to prognosis or diagnostic accuracy issues. Therefore, investigators are encouraged to match the clinical research question to an appropriate study design according to the question domains. A thorough understanding of the research question is necessary in order to select the best study design.