Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review.

Clinical trials are time consuming, expensive, and often burdensome on patients. Clinical trials can fail for many reasons. This survey reviews many of these reasons and offers insights on opportunities for improving the likelihood of creating and executing successful clinical trials. Literature from the past 30 years was reviewed for relevant data. Common patterns in reported successful trials are identified, including factors regarding the study site, study coordinator/investigator, and the effects on participating patients. Specific instances where artificial intelligence can help improve clinical trials are identified.

Background

Clinical trials for pharmaceuticals and medical devices offer many opportunities for failure. Failures can arise from a lack of efficacy, issues with safety, or a lack of funding to complete a trial, as well as other factors such as failing to maintain good manufacturing protocols, failing to follow FDA guidance, or problems with patient recruitment, enrollment, and retention. Generating accurate and sufficient results to determine whether or not there is merit in continuing is important at each stage in the clinical trial process. The investments of resources, time, and funding grow with successive stages, from pre-clinical through phase 3. Thus, the cost of a failed phase 3 trial is not just the cost associated with the trial itself but the cost of all prior trials as well as the cost of lost time pursuing a potentially viable alternative.

It is important to maintain a philosophy of continual improvement with respect to clinical trials broadly and specifically with an aim towards optimizing every aspect of the research and development process. A comprehensive survey of all possible points of failure in clinical trials is beyond the scope of this publication. Still, there are many factors associated with failed trials that can be distilled with evidence, along with recommendations for improving the chances of success.

Failing to demonstrate efficacy or safety

The primary source of trial failure has been and remains an inability to demonstrate efficacy. Hwang et al. [58] assessed 640 phase 3 trials with novel therapeutics and found that 54% failed in clinical development, with 57% of those failing due to inadequate efficacy. There are many reasons that potentially efficacious drugs can still fail to demonstrate efficacy, including a flawed study design, an inappropriate statistical endpoint, or simply having an underpowered clinical trial (i.e., sample size too small to reject the null hypothesis), which may result from patient dropouts and insufficient enrollment.

Clinical trials also fail with respect to safety. Hwang et al. [58] found that 17% of the failed phase 3 trials examined were due to safety. Safety is addressed in every clinical trial in every phase, but issues with safety may only become apparent with the larger populations associated with phase 3 studies, or at post-approval (phase 4) or post-market [24]. Identifying safety issues is not always straightforward. Patients have individual concerns about various adverse events that may not match what physicians are concerned about. This can influence which adverse events are reported, particularly if they are mild to moderate in severity.

For example, Henon et al. [49] studied 27 phase 1 trials in diverse settings between 2014 and 2015. Prior to the start of these particular trials, patients most feared adverse events of hematuria, vomiting, and hyperglycemia, and after the trials they feared some of the same events, but also personality change, fever, and dizziness. The physicians in these trials were concerned instead with eye disorders, confusion, and blurred vision. People may have a greater propensity to present for care when they experience an adverse event that is of concern to them, and not necessarily when experiencing an adverse event of less concern to them but greater concern to the physician. Reminding patients of the importance of reporting adverse events, particularly events of special interest, is recommended for improving the likelihood of detecting safety issues earlier rather than later (e.g. [22]).

It is important also to recognize the desire for a sponsor to move a drug or device forward in the clinical trial process. Rushing studies into phase 3 after successful phase 2 trials may not provide time for reflection on how best to address safety in phase 3 [107]. Research also has identified that having higher-educated nurses is associated with lower risks of mortality and failure to rescue ([3,121,128]), which may be helpful as a factor to include in study site selection.

It is critical at each stage of clinical development to have safety be a primary concern even if it is not a primary objective. The cost of uncovering a safety issue increases at each stage, including post-approval [118].

Financial impact

Hwang et al. [58] noted that 22% of the failed phase 3 studies they examined failed due to lack of funding. The costs required to complete the entire development process from discovery to bringing a drug to market vary, and so do estimates of these costs; however, they have been reported in excess of $2.5 billion [34]. This includes $1.5 billion of hard dollar out-of-pocket costs with the remainder being lost opportunity of investment costs, but does not include additional post-approval clinical trials. Focusing on phase 3 trials, the Pharmaceutical Research and Manufacturers of America estimated the cost at $42,000 per patient in 2013, with $10 billion spent on 1680 phase 3 clinical trials comprising over 600,000 patients.

Certain studies present unique cost considerations. For example, in a study of hospital-acquired bacterial pneumonia, the cost of a 200-site, 1000-patient phase 3 study was $89,600 per patient [112], with screen failures being a principal driver of the cost. Pharmaceutical research and development is a costly endeavor. More generally, particularly in the United States, the cost of complying with an increasing regulatory burden is also impactful, necessitating more staff, storage, and financial outlay [43].

With such a large financial burden, many trials (in phase 3, but also earlier) are underfunded, and may not have any reasonable opportunity to generate a positive outcome (even if protocols are amended, at additional cost). This leads to ethical issues regarding patient involvement [127]. Patients generally have an expectation that their participation in a trial will lead to an advancement of knowledge based on the trial's successful completion [71]. Underfunded trials are by definition more likely to miss the enrollment needed to demonstrate statistical significance at a predefined level of efficacy.

Eligibility criteria

Ideally, inclusion/exclusion criteria should result in a population that matches statistically the intended general patient population [48,124]; however, study designers must account for additional concerns, including whether or not particular segments of a target population may have too many comorbities, leading to additional higher risk of withdrawal and adverse events. For example, Hill et al. [50] noted the heterogeneous nature of pulmonary arterial hypertension (PAH), for which clinical studies have had varied eligibility, but have tended to exclude patients with advanced conditions (New York Heart Association functional class III or IV) and older women, among other categories. The correspondence between the study population and the actual population of concern can become unclear [50].

Inclusion/exclusion criteria also must be chosen in light of the expected effect on recruitment. In the case of patients with PAH, Hill et al. [50] noted the availability of competing therapies, which can suppress enrollment in any one particular study, along with investigators being influenced to recruit patients who will be stable for the duration of the study (3–4 months). Investigators may look for patients who have been stable recently, thus restricting the available population in a way that does not match the general patient population.

Inclusion criteria may vary widely across studies in a specific area, providing little guidance to a prospective sponsor or investigator. For example, in heart failure, Luo et al. [78] reported that there are no uniform diagnostic criteria for heart failure with preserved ejection fraction (HFpEF), with approximately 55% of 121 trials using 50% as the cut-off value for diagnosing HFpEF, leaving 45% choosing another threshold.

Overly specific inclusion criteria can lead to problems in finding suitable participants. This is true particularly for conditions associated with small populations but also it applies generally. Many oncology studies, for example, have exclusions based on prior chemotherapy, having an advanced stage of disease, or not being newly diagnosed. Particularly in oncology, targeted treatments based on specific genetic markers [53] will exacerbate this issue as diagnostics screen out more individuals (hopefully with the benefit of improved efficacy). Making inclusion criteria too narrow may lead to longer recruitment times and also eventually to amending the study protocol in an attempt to recruit additional participants. Getz et al. [44] reported that 16% of protocol amendments are due to changes in inclusion/exclusion criteria, which can lead to differences in the patient populations before and after the amendment [76].

It is clear that the choice of inclusion and exclusion criteria can affect the duration and cost of a clinical trial [4], as well as the likelihood of the trial meeting desired enrollment levels and retaining sufficient participants to have an opportunity to meet a statistical endpoint. Getz et al. [44] noted that across 3400 clinical trials, more than 40% had amended protocols prior to the first subject visit, delaying trials by 4 months. Some protocol amendments cannot be avoided; however, the potential for amendments can be reduced with better planning and anticipation of the consequences from design choices.

Exclusion criteria are often presented without an explicit rationale [104]. Sometimes criteria can be put in place based on an expectation of excluding participants who may not show sufficient improvement against an endpoint, not because their health is too poor but because it is too good. For example, Hill et al. [50] reported on the endpoint of a 6-min walk test for patients with PAH. Patients who could walk more than 400 m prior to being included in trial might not be able to show much improvement (482 m in 6 min is already a 3 mph pace, which would be a moderate pace for a healthy individual). Thus, there would be pressure to exclude patients at this functional level in favor of those who could only walk between 100 and 150 m prior to inclusion. Without background knowledge, someone reviewing exclusion criteria for such a trial might not have explicit motivation to intuit the rationale for this sort of exclusion criterion.

Performing a requisite literature review for related studies remains a labor-intensive task requiring personnel with specific knowledge who can interpret the framework, criteria, and results of prior clinical trials. Future protocol development will benefit from the use of artificial intelligence tools, such as natural language processing [2,17,32,53], which will be able to extract meaningful information across published documents and present systematically organized data to the study designer for consideration. Still, the study designer must think through the implications of different inclusion/exclusion criteria (as well as objectives and endpoints) and the effects they will have on recruitment, enrollment, retention, and ultimately time and cost to completion.

Patient recruitment

Patients are often willing to consent to participation in a clinical trial if they believe that they have an opportunity to receive better treatment or if the results can help others [29,45,89]. Still, failing to enroll a sufficient number of subjects in a trial is a long-standing problem [82,101]. A study of 114 trials in the UK [10] indicated that only 31% met enrollment goals. In addition, Campbell et al. [15] reported that one-third of publicly funded trials required a time extension because they failed to meet initial recruitment goals.

Feller [39] reported that 25% of cancer trials failed to enroll a sufficient number of patients, and 18% of trials closed with less than half of the target (the number of enrollees divided by the number of subjects screened) number of participants after 3 or more years. Only a relatively small percentage (2%–5%) of adult cancer patients enroll in clinical trials [41]. Furthermore, enrollment fractions (the number of enrollees divided by the number of subjects screened) can be very low [50,116].

Stensland et al. [111] reported that, particularly for cancer trials, between 2005 and 2011, the cumulative incidence of trials failing to complete was about 20%, with almost 48,000 enrolled patients in these failed trials. An earlier report from the Institute of Medicine [60] indicated that 40% of National Cancer Institute-sponsored trials were not completed.

Some studies offer remuneration to patients, generally to cover the patients' time and expenses but also in the hope that recruitment will be improved. While logic suggests that trials that offer remuneration to patients should fare better with respect to recruitment than those that do not and, moreover, patients sometimes report this as being important to them [109], evidence supporting this has been generally inconclusive. Bryant and Powell [13] found no controlled studies aimed at testing the hypothesis that paying patients to participate in a trial generates superior recruitment or retention.

Several trials have reported no observed relationship between financial incentives and recruitment [31,52,98]. On the other hand, Edwards et al. [36] reported that monetary incentives increased participant response to postal and electronic questionnaires that were designed to improve retention. Martinson et al. [81] reported effective remuneration in a smoking trial, but the trial focused on adolescents, who may have a different reaction to receiving funds than adults. Surveys show, however, that a high remuneration is often associated in patients' minds as being associated with a perception of higher risk in the trial [25,109] and thus a reluctance to enroll. The effect and effectiveness of remuneration may depend on many factors and should remain an open area of research.

Additional costs associated with recruitment

Beyond remuneration, the additional costs associated with patient recruitment can be difficult to estimate and highly variable, even within the same investigative area [21]. For example, Okuyemi et al. [95] conducted a study to assess the efficacy of nicotine gum and counseling to help low-income African-American smokers to quit. They reported spending $156 per enrollee obtained via direct marketing, but $5040 per enrollee obtained via gas-pump advertising. This suggests that marketing strategies can play an important role in the financial viability of some trials, and by consequence with the ultimate outcome as well.1 The degree to which professional marketing expertise has been applied to help promote clinical trials is difficult to ascertain presently, but deserves specific research attention.

Healthcare providers can have a significant impact on patient recruitment and retention. Recruitment and retention can suffer when patients perceive support staff to be unavailable or uninterested, or if they have to interface routinely with new staff [33]. Encouraging patient trust in the clinical trial process may be expected to lead to better participation [125].

Incentivizing staff (providing funds for enrolling patients) has been shown to improve patient recruitment [33]. Using nurses instead of surgeons (physicians) to perform recruitment has not evidenced any difference in outcomes; however, cost savings have been realized [35,40] which may be important in supporting recruitment and retention, or other aspects of the clinical trial, indirectly.

Respecting the patient's concerns

Patient recruitment and retention is affected negatively when patients are concerned about being assigned to a control group rather than receiving active study drug. Part of this effect may be due to patients having poor knowledge about placebos [55] or what specific treatment is given in the control group. For patients with poor prognoses, the concern may center around not having effective treatment at all.

Ulrich et al. [122] addressed the burdens of patients participating in cancer clinical trials. Patients reported burdens including potential side effects from treatment, additional tests that would have to be endured, financial concerns (including loss of job support and work disruption), and a general worry about the unknown future, including whether or not the study drug assigned would be beneficial. Sometimes patients are not presented with a clear rationale for why their participation is important and receive minimal feedback. These concerns were echoed in Rosbach and Andersen [102], which reported on burdens on patients with multimorbidity.

In addition, scientific literacy in the general population is limited, leading to difficulty understanding information associated with a clinical trial [9,67]. Hadden et al. [46] reviewed more than 200 approved ICFs and found the mean readability was at the 10th grade level, similar to results in Paasche-Orlow et al. [97]. In a survey by Lopienski [75] of various completed trials, patients who dropped out of a trial early were twice as likely to say that it was difficult to understand the informed consent form (ICF) than were patients who completed the trial (35% vs. 16%, respectively). Moreover, even for those who completed a trial, 1 in 6 patients found the ICF vexing.

It is of interest to determine if artificial intelligence tools employing sentiment analysis could be useful in crafting the language of the ICF and other materials to provide a more compassionate tone and greater patient confidence [38,91,114] in addition to maintaining an appropriate reading level. Davis et al. [30] reported that patients prefer simplified ICFs and testing showed no lower level of patient comprehension of the details of the clinical trial when using a simplified ICF. There is an impetus for simplifying the ICF, as well as other associated written materials.

In addition, Sood et al. [110] and Cartmell et al. [18] reported that providing information regarding the clinical trial process was helpful in improving patient satisfaction. Even receiving a simple biosketch card of a healthcare provider has been associated with improved patient satisfaction [90]. Surveys of patient satisfaction conducted by a personalized health network suggest patients often have a poor experience [132]. Communicating with the patient is important at all stages of the clinical trial and supports recruitment, enrollment, and retention [61].

Poor recruitment, dropouts, and underpowered trials

A repeated problematic pattern in the literature is that study centers report fewer eligible patients than anticipated [6,33]. Study centers with a track record of successful performance are historically more likely to meet enrollment targets [43]. There is considerable literature reporting results from studies in which numerous study sites failed to meet enrollment, or failed to enroll any subject at all [64,72,73,105]. Levett et al. [70] identified several factors most associated with above-average recruitment rate: implementation of a clearly defined “system” of recruitment, engagement of other staff, time from ethics approval to first recruit, and the provision of a dedicated trial coordinator. A site that has historically little focus on clinical trials or presents other non-scientific impediments may lead to low investigator enthusiasm [61].

Enthusiasm from the lead investigator at a study site was the most important factor associated with positive recruitment across 60 study centers in trial assessing the management of local post-surgical pain [26,41,51]. Thoma et al. [117] noted the critical importance of a friendly and approachable study coordinator, as this can not only improve recruitment but also lower the study's dropout rate. Slow recruitment may come from an inadequate staff and a lack of prioritizing the clinical trial over day-to-day operations [117]. It can also come when the investigator has competing trials. In addition, retention can be improved by having dedicated recruiting/support people at each study center, as well as tailoring the protocol of the trial to conform to common practices at the study center [33].

When a trial suffers too many dropouts (either based on projected or actual enrollment), the trial may become underpowered. Underpowered clinical trials are problematic. The sponsor may adapt to low enrollment by expanding the number of sites (perhaps in additional countries, with corresponding costly protocol amendments and delays in further research), increasing funds allocated to the study in an effort to meet minimum enrollment. By consequence this sometimes necessitates eliminating certain planned tests in order to reallocate available funds. In turn, certain endpoints may have an insufficient sample size to detect an important result.

As an example [61], the STICH trial (surgical treatment for ischemic heart failure) that studied the effectiveness of coronary bypass surgery in patients with heart failure took place ultimately in 26 countries, in 127 study sites, and involved 2135 patients. Originally, the study was designed to cover 32 study sites in the USA and Canada, but low patient enrollment required expanding the study to 171 sites internationally. After the expansion, 44 sites that had been approved for the trial failed to enroll a patient. The cost per deactivated site was estimated at $10,000. The extra costs involved in expanding the trial to numerous sites in various countries meant that funds intended originally for imaging studies had to be diverted. The imaging studies were removed from the protocol, creating additional expenses for protocol amendments.

Importantly, and as mentioned briefly earlier, underpowered trials have also been described as unethical [47], even though some patients may benefit from the trial, because patients who volunteer to be in the trial are unlikely to know or appreciate that their results will not be likely to contribute to a statistically significant outcome. Carlisle et al. [16] studied 2579 trials from the National Library of Medicine clinical trial registry and found that 48,027 patients had enrolled in trials that closed in 2011 that were “unable to answer the primary research question meaningfully,” notably for failed accrual or less-than-expected enrollment. Thus, poor recruitment, enrollment, and retention remains a primary area of concern for multiple reasons.

Employing quantitative measures

Formulating a list of factors to consider when designing and executing a clinical trial can provide a foundation for better outcomes. However, not all factors are equally important. A well-structured mathematical framework (e.g., a Valuated State Space [87]) for trading off degrees of achievement in various parameters can offer a quantitative measure for comparing alternative choices.

For example, increasing the speed of enrollment leads to faster completion, and may be associated with fewer dropouts, better statistical power, and increased confidence in results. In contrast, consistently slow recruitment may suggest problems in inclusion/exclusion criteria [117], which should be addressed by amending a protocol. Enrollment can be accelerated by spending money on recruitment, both in advertising and in having available friendly personnel. Thus, there is a direct trade-off between the speed of enrollment and the cost of executing the trial.

Budgets are not unlimited, and therefore various trade-offs need to be considered, including not only the speed of enrollment, but the likelihood of meeting the enrollment goal. A more-effective study center with a long history of running clinical trials successfully and with a nearby population of prospective participants may be more expensive than another more remote site with less experience. But choosing the cheaper alternative may result in failing to meet recruitment. By consequence, this may necessitate spending more on additional study centers, which come with additional costs of evaluating, training, protocol amendments, and trial execution. Quantifying these trade-offs can assist with making better decisions.

Considering the patient's financial burden

Given the tremendous problem of clinical trials that fail to complete due to poor recruitment, enrollment, and retention, it's of primary importance in designing and executing clinical trials to consider the burden that each patient undergoes, with the belief that retention is correlated negatively with patient burden. All burdens to the patient should be given attention, but financial impacts deserve special consideration.

The financial impact to patients in clinical trials can be easily overlooked while focusing on the objectives, endpoints, and other aspects of a particular trial design. Patients may have out-of-pocket costs when participating in a clinical trial. These include the cost of transportation and lost work, but also medical costs for additional testing. Insurance may not cover medical care beyond that which is deemed routine. Even when it does, deductibles are often quite high and a given patient may not be able to afford to participate [92].

In addition, many trials require participants to travel to their specific study centers, even for tests or procedures that could be provided locally [92], or conducted at home. Patients may need to relocate close to a study center for some period of time [79]. The additional cost of participating under these circumstances biases participation to those in higher socioeconomic levels [5,103,115,119], particularly in oncology studies. Stump et al. [113] reported on a cancer study in which 99% of participants were insured and still greater than 30% reported concern about paying for treatment. Studies also show that the financial impact of some trials can adversely event patient adherence as well as retention [8,23,130].

Patient time investment

While some trial participants do need to relocate during a study, many are not willing to do so [19] and most participate in local trials. Patient recruitment and retention depends in part on the willingness of the participant to travel to and from the local study center [96]. Transportation is a long-standing particular challenge for elderly participants [77,83,96,99]. Regardless of patient age, long travel times, particularly in urban areas can dissuade participation.

Research that would provide a mathematical function describing the likelihood of patient recruitment or retention as a function of distance to a study center and other factors such as demographics appears missing currently. However, some related information can provide guidance in the absence of such specific research.

The Washington State Office of Financial Management [129] surveyed resident health care consumers. Based on more than 5000 responses, adults generally reported being willing to travel less than 30 min and less than 22 miles for urgent care, and about 10% less for routine care. Demographic analysis showed that males were willing to travel for longer duration (32.8 min), as were those in non-urban areas (34.8 min), and those who were uninsured (34.1 min), each being statistically significant at α = 0.05. Interestingly, those 65 years of age or older were willing to travel only 26.4 min, which was also statistically significant at α = 0.05.

These data suggest the importance of recruiting patients from proximate vicinities local to study centers when in urban settings. Moreover, when incorporating older patients it is important to assist in minimizing their total time investment as they may begin from a perspective of being less patient than the average participant. Proper site selection can help minimize long travel times.

In addition, selecting a study site with a nearby larger population pool has been correlated positively, as expected, with the likelihood of meeting recruitment targets [123]. van den Bor et al. [123] also reported statistically significant results pertaining to recruitment by geographic region. In their analyzed data, study centers in China and India were more likely to meet recruitment targets, with centers in certain locations in Western Europe and North America being least likely. It remains an open question as to whether study adherence is equivalent across these sites, and if other factors influencing positive recruitment might be associated with any operational issues.

Artificial intelligence applications offer promise in helping reducing patient time investment regardless of constraints on study site location [7,20]. In particular, evolutionary algorithms [11,66], which use computer simulations of nature's processes of variation and selection to solve problems, can assign the most appropriate study center for each prospective patient in a trial based on patient and study center availability. There is also the opportunity not only to schedule staff to support a clinical trial appropriately [12] but also to match staff with patients so that patients tend to see familiar faces at each visit (and could also request having alternative staff to interact with if desired).

In certain cases, it is possible to schedule study center visits to minimize other conflicts that a patient may have. For example, the burden on a single parent of elementary school children who must come to a study center at 10am is different than the burden for the same procedure scheduled at 4pm, after school has let out for the day. Artificial intelligence software can examine the profile of each study participant and impute the least burdensome times for appointments within the constraints of a study center's activity and the constraints of the protocol (e.g., pharmacokinetic time points). The same software can search for opportunities to reschedule patients adaptively when openings develop, making the most efficient use of the clinical trial's time.

Effective scheduling also should incorporate the patient's time spent waiting after checking in before being seen. Waiting time has been offered as being associated negatively with patient satisfaction and how patients feel about the quality of their health care [28,93,94].

Vitals Index reported 30% of US patients have left an appointment because of an excessive wait and 20% have changed doctors due to habitually long waits [69]. This same study identified superior patient satisfaction when the average wait time was just over 13 min. The lowest level of satisfaction was generated at an average wait time of just over 34 min. Long waiting times are a source of stress and can leave patients feeling disrespected, which intuitively would be associated with lower retention.

The patient's perception of a long waiting time can be reduced by assigning an additional person to facilitate interaction with the patient [108]. This additional person can also help to relieve the burden on other doctor-office staff who would be dividing their attention between patients receiving routine care and those participating in a trial.

Discussion

Each of the facets of protocol design, execution, and successive trial planning offers opportunities for trading off different concerns, as well as simply making inappropriate judgments leading to poor outcomes.

Study site selection is an important aspect of the clinical trial process. Poor choices can lead directly to study failure, or to a costly exercise of including additional study sites, amended protocols, and the potential for patient populations receiving different treatment regimens. When possible, having contingency plans to open additional sites, perform extra recruitment, and cover protocol amendments is recommended. The practicality of holding out reserve funds to covers these and other contingencies, however, is case specific.

Many factors for study site selection are study specific. Hurtago-Chong et al. [57] reported on a multi-step study site evaluation process, starting from a request for participation and extending through telephone follow-up, site selection questionnaires focusing on geriatric fracture management. It is straightforward to presume that many specific requirements for a study in this specific area would not carry over to a criteria for, say, a study on pediatric oncology. Still, there are many study-site-related factors that are common to successful trials.

Getz [42] cited research from pharmaceutical companies Lilly and Pfizer, suggesting a correlation between performing well on one trial and performing well on a subsequent trial, as well as the converse of performing poorly on one trial and performing poorly on a subsequent trial. Experience with clinical trials is also important, as experience facilitates effectiveness. A site that has conducted between 6 and 10 clinical trials has a greater probability of meeting enrollment within the required time than does a site with a history of fewer trials [42]. An additional indicator is time to enroll the first patient, which is correlated with better overall performance. Data in Ref. [123] corroborate that successful experience is a predictor of positive recruitment performance. In addition, as mentioned earlier, other positive factors include an enthusiastic investigator and experienced and involved staff.

A key item deserving more attention is the minimization of patient burden and maximizing patient appreciation. This encompasses: (1) providing materials that are easy to understand, (2) having empathetic and supportive staff, (3) leadership and enthusiasm from the principal investigator, (4) a schedule (time and events) that works in synergy with the patient's constraints rather than at odds with those constraints, (5) the opportunity to adaptively reschedule visits and assign appropriate personnel to support participants, (6) trial management software to send effective reminders about visits and protocol adherence via phone, text, or email, including supporting multiple languages in multi-lingual areas [14], and (7) understanding what the patient's day-to-day experience during the trial is likely to be. Even asking a simple question such as “are you planning on moving?” can help ensure that patients are more likely to remain involved in a study, particularly if the study is of longer duration [117].

Study support staff should be generally aware of how study participants are feeling during the trial, and seek to minimize patient stress. In a survey [75], patients who described site visits as stressful were more than twice as likely to drop out (38% of dropouts vs. 16% of those who completed). Hui et al. [56] reported that in one cancer study, 21% of patients who withdrew from the trial did so because of “symptom burden.” Some burdens cannot be avoided, depending on the therapy and testing involved. Patients are more likely to withdraw from a trial when they perceive their condition as not improving, even though this may be anticipated. Support staff, as well as the investigator, should seek to set patient expectations appropriately and provide appropriate empathy for any burdens that a patient is undergoing during a trial.

Patients deserve to have access to reports from studies in which they participated. Yet, Ziv [132] reported that, after completion, most studies are not available via open access. Thus, patients have to pay to be able to read a published study, even one in which they have participated. Having already given much of their time in support of a clinical trial, they may feel disrespected to have to pay to find out what information was discovered during the trial. It would be easy for a sponsor to take the position that the patient has nothing left to offer to the trial after the trial concludes and thus any additional funds required to provide article access or to provide a copy of a publication would be better allocated elsewhere. This misses the point, however, that by ensuring patient participation from start all the way through publication, the patient may feel more respected and be less likely to dropout.

Study designers should employ methods to ensure that study populations are relevant to the real-world population that is intended to benefit from treatment. Eligibility criteria should be reviewed carefully in this regard. Older patients may be viewed, correctly in some cases, as presenting more potential for comorbidities, propensity for adverse events, and for ultimately withdrawing from a trial. However, some research in breast cancer [63] reported that older patients (>65) were less likely to be eligible for breast cancer trials but when eligible their participation rate was not statistically different from the participation rate of younger eligible subjects.

Collecting and reporting data on participation and withdrawal should be more common place in order to assist with a better understanding of how to design trials so that they can complete with representative subpopulations. For example, Hui et al. [56] reported on data from palliative oncology trials. Data revealed that patients of Hispanic ethnicity (odds ratio [OR] = 1.87), those holding an advanced degree (OR ≈ 1.5), non-Christians (OR not provided), and those with higher levels of dyspnea (OR = 1.06) and fatigue (OR = 1.08) were associated with statistically significantly higher dropout rates prior to end of study. Determining the repeatability of these factors across different types of trials remains for future work.

Future efforts should also be directed toward improving the efficiency and effectiveness of clinical trials broadly (e.g., using adaptive designs) and also based on specific genetic markers or other personalized factors. Bringing eligibility to the level of the individual holds the promise for establishing greater study drug efficacy but also has the drawback of limiting the available sample size [129] to more rapidly direct the use of study drug to targets of opportunity.

Success depends crucially on identifying genetic features reliably [86,131], which could benefit from establishing collaborative databases for academic research. It will also be important to determine quality of life measures to better assess the cost effectiveness of these tailored trials. Current data do support the cost effectiveness (in terms of additional dollars spent per month of extended life) in the case of genetic markers for acute myeloid leukemia [52] but short-term extended life (cf. multiyear extension [100]) offers only a broad indication of the potential value of a treatment.

This review covers many aspects of clinical study design that can be affected positively by appropriate design considerations and trial execution. Study site selection and addressing patient concerns are two primary areas where the effectiveness of clinical trials can be affected positively. It is important to note that there are other factors that deserve attention, which arise even in “successful” trials, including (1) whether or not the appropriate outcome measure is chosen, particularly if it is a surrogate measure [27,48,68,120], (2) how missing data are handled, and whether values are imputed [59,65], (3) the use of subjective measures that are subject to observer bias [1,54], (4) defining what observation would constitute a clinically meaningful result rather than merely a statistically significant result [80,88], (5) the lack of long-term follow up [74], (6) minimizing protocol deviations [62,85], and (7) under reporting adverse events in peer-reviewed publications [106,126].

For convenience, Table 1 offers a summary of the factors associated with problems or challenges that occur when preparing for and executing clinical trials. Some of these issues may not lead directly to the failure of a trial; however, a series of issues can lead to a critical failure. The table presents the opportunities for improving the likelihood of success and the role that artificial intelligence may play in that improvement. Many of these factors are correlated or interrelated, thus the table is not a substitute for the greater detailed explanation found in the text.

Table 1

A list of factors associated with problems or challenges when preparing for or executing a clinical trial, along with the opportunities for artificial intelligence to help alleviate these issues. Abbreviation: NLP = natural language processing.

Factor	Opportunity	Role for Artificial Intelligence
Poor study design	More complete literature review	NLP of available literature, finding similar trials, trials addressing similar issues, or trials addressing different issues utilizing similar techniques, summarized for the study designer
	Appropriate endpoints	NLP of available literature, showing endpoints/measures used in other similar studies
	Inappropriate eligibility criteria	NLP assessment of similar published trials to determine suitability of eligibility criteria and any potentially important omissions.
	Appropriate statistical analysis	NLP of available literature, summarizing statistical methods and associating these methods with successful or failed outcomes.
	Determination of appropriate sample size	Nonlinear modeling, such as with neural networks, to predict patient drop-out rates and better estimate sample size to avoid becoming underpowered. Agent-based modeling to simulate trial before execution. Use of NLP to mine previously published trials to determine sample sizes used in successful trials
	Reducing likelihood of amendments	NLP and knowledge-based processing to present designer with pertinent information to consider.
	Inconsistencies in protocol	NLP (including table-based format) to check time and events schedule against text, as well as summary of changes for any amendments.
Ineffective site selection	Effective measurement of trade-offs for each site	Nonlinear modeling, such as with neural networks, to assess trade-offs site history, staff experience, investigator enthusiasm, available population, expected patient burden, and financial impact. Potential use of fuzzy logic to provide linguistic measurement descriptions.
Poor recruitment	Improved use of funds	Optimizing communication/advertising to maximize cost effectiveness. Targeting communication to meet patient profile, including sentiment analysis.
	Ensuring appropriate eligibility criteria	NLP on prior publications to identify suitable criteria, and also criteria associated with other trial failures.
	Facilitating locating eligible patients	Database coordination, prompting investigators and patients when appropriate trials are available for specific patients.
	Enrolling patients who are likely to complete the trial	NLP and machine learning to profile patients based on prior data on who is more likely to complete a trial, reducing drop-outs.
Patient burden/safety	Minimize travel and wait times	Adaptive patient scheduling, also potentially turn-by-turn driving instructions, using evolutionary algorithms. Incorporate patient profiles to tailor site assignment/schedules to patient constraints where possible. Adapt site visit schedule if possible.
	Minimize out-of-pocket expenses	Systematic review of all patient costs to identify opportunities to minimize impacts.
	Minimize possibility of contraindicated medicines/procedures	Automatic review of prior and concomitant medications for contraindications, protocol violations.
	Increase likelihood of feeling respected	Sentiment analysis and other NLP tools applied to all documents provided to patients. Prompts to interacting staff for personalizing interactions. Tailored messaging to participants to increase likelihood of retention.
Poor trial execution	Automating reporting of events	Automated prompting of events for patients and staff, reporting requirements, notes missed events, prompts for required reporting, including protocol deviations and adverse events.
	Preparing data and reporting for write-up	Automatic brand/generic conversion, skeletal form generation for narratives, table creation based on specified cut-offs.
	Lack of general awareness	Situation awareness provided to investigator/study coordinator monitoring study progress, patient progress, indicating interventions if needed.
Overall	Factor analysis to improve trade-offs based on budget and other constraints	Multicriteria decision making based on Pareto analysis or single aggregated evaluation function (Valuated State Space) to quantify and illuminate trade-offs.