Guarding the Gate: Remote Structured Assessments to Enhance Enrollment Precision in Depression Trials.

BACKGROUND: Failed treatment trials are common in major depressive disorder and treatment-resistant depression, and remotely performed multifaceted, centralized structured interviews can potentially enhance signal detection by ensuring that enrolled patients meet eligibility criteria.
METHODS: We assessed the use of a specific remote structured interview that validated the diagnosis, level of treatment resistance, and depression severity. The objectives were to (1) assess the rate at which patients who were deemed eligible for participation in trials by site investigators were ineligible, (2) assess the reasons for ineligibility, (3) compare rates of ineligibility between academic and nonacademic sites, (4) compare eligibility between US and non-US sites, and (5) report the placebo response rates in trials utilizing this quality assurance approach, comparing its placebo response rates with those reported in the literature. Methods included a pooled analysis of 9 studies that utilized this methodology (SAFER interviews).
RESULTS: Overall, 15.33% of patients who had been deemed eligible at research sites were not eligible after the structured interviews. The most common reason was that patients did not meet the study requirements for level of treatment resistance. Pass rates were significantly higher at non-US compared with US sites (94.6% vs 83.3%, respectively; P < 0.001). There was not a significant difference between academic and nonacademic sites (87.8% vs 82.4%; P = 0.08). Placebo response rates were 13.0% to 27.3%, below the 30% to 40% average in antidepressant clinical trials, suggesting a benefit of the quality assurance provided by these interviews.
CONCLUSIONS: The use of a remotely structured interview by experienced clinical researchers was feasible and possibly contributed to lower-than-average placebo response rates. The difference between US and non-US sites should be the subject of further research.

Novel treatments for major depressive disorder (MDD) are needed, considering the morbidity and mortality that currently exist despite currently available treatments. Drug development for MDD is presently hindered by high placebo response rates in clinical trials. Potentially efficacious new medications for MDD and treatment-resistant depression (TRD) may be derailed from further study and development if early trials do not demonstrate separation of drug from placebo response rates, and the elevation of placebo response rates across the years has made such signal detection increasingly difficult.[1,2]

One component that may contribute to higher placebo response rates is the enrollment of inappropriate subjects into depression studies, increasing the risk that nonspecific factors can influence study outcomes. Investigators often feel pressured to recruit and enroll participants into clinical trials as quickly as possible, because of financial and systemic pressures that occur within large-scale multisite clinical trials. These burdens on site investigators may influence the suitability and appropriateness of enrolled study participants.[3,4] This process is additionally complicated by the fact that some potential study subjects are motivated by factors such as remuneration (eg, “professional patients”) or other secondary gains from participation.[5] Therefore, quality assurance measures are considered a necessary element to enhance the precision of patient selection into randomized trials for the treatment of MDD and TRD.

The investigators developed and honed a multifaceted, centralized, remotely structured interview tool for application after in-person patient screening at individual research sites. The goal of the structured interview is to further refine the patient population to only include patients with the protocol-specified history, diagnosis, and disease severity. The SAFER structured interview, an abbreviation for state versus trait, assessability, face validity, ecological validity, and rule of the three Ps (see below for more details). The SAFER Interview and the Antidepressant Treatment Response Questionnaire (ATRQ) represent 2 clinical trial tools developed at Massachusetts General Hospital (MGH) to increase the precision associated with the diagnosis and course of MDD and of TRD. The SAFER and ATRQ provide researchers with tools that are well suited for use in a clinical setting, that enrich the qualitative assessment of MDD and TRD, and that when combined with a rating scale for symptom severity assessment provide a comprehensive quality assurance process. For simplicity, the entire remote interview is referred to as the “SAFER Interview,” although components of the ATRQ and depression severity are typically included and used together in studies referred to in this article.

The objectives of this study were (1) to assess the rate at which patients who are deemed eligible for enrollment into clinical trials of MDD and TRD by site investigators are deemed ineligible based on these tools, (2) to determine the reasons why these patients were deemed ineligible, (3) to compare rates of ineligibility between academic and nonacademic sites, (4) to compare eligibility between US and non-US sites, and (5) to report the placebo response rates in trials utilizing these tools, comparing them with placebo response rates reported in the literature.

METHODS

A review of independent remote SAFER Interviews conducted in 9 consecutive clinical drug trials in MDD and TRD was performed. In each trial, the interview occurred after each subject had passed screening procedures at the study site and was considered to be eligible for the trial by the site investigators.

Raters

The qualifications of the SAFER remote raters are as follows: (1) psychiatrist or psychologist, (2) at least 2 years of clinical research experience, (3) fluent in English and, if applicable, in the language of the patient being interviewed, and (4) completed training in the instruments used in the SAFER Interview. The North American interviews were performed by clinical researchers from MGH. Interviews outside North America were performed by independent contractors hired and trained by the MGH Clinical Trials Network and Institute (CTNI). The contractors met the same experience requirements and completed the same training as the MGH SAFER raters.

Rater training was composed of a didactic presentation of materials, observation of interviews, performance of mock interviews by the trainee, and observation of the trainee's study interviews by an experienced senior rater. Upon completion of training, each rater's summary of qualifications, experience, and training performance were reviewed by CTNI leadership before the rater started conducting interviews.

Procedures

The SAFER Interview was performed remotely by raters, who telephoned the patient directly at their home or on their cell phone. During the interview, the rater administered the SAFER Criteria Inventory (SCI), the MGH ATRQ, and a structured severity scale for MDD (either the Montgomery-Asberg Depression Rating Scale or the 17-item Hamilton Depression Rating Scale). After interviews where the eligibility of a patient was not clear, the process for adjudication entailed review with the MGH SAFER ratings team via a secure e-mail system. If additional information was needed to assess whether the patient was eligible, the rater contacted the clinical staff at the study site for additional information and/or pharmacy or medical records. The collection of additional information and good communication with sites were encouraged among raters to facilitate the accuracy and collegiality of the independent remote interview process.

SAFER Criteria Inventory

The SAFER Criteria Interview was created to assess the patient's presentation and history to determine whether the patient meets criteria for the disorder being studied as observed in real-world settings. This has been previously described in detail.[6,7] Originally developed for use in trials of MDD and TRD, the SCI has been adapted and utilized in clinical trials for other psychiatric disorders such as schizophrenia, bipolar depression, obsessive-compulsive disorder, and attention-deficit/hyperactivity disorder.

To summarize, the SCI is a 9-item clinician-performed structured interview based on the following criteria: (1) state versus trait: the identified symptoms must reflect the current state of illness and not longstanding traits that are unlikely to change across a treatment trial; (2) assessability: the patient's symptoms can be reliably assessed with standardized, reliable rating instruments; (3) face validity: the patient's presentation is consistent with our knowledge of the illness (their symptoms are specific and map onto the nosological entity, ie, the Diagnostic and Statistical Manual of Mental Disorders; there is clear change from previous level of function; the symptoms are similar to previous episodes if recurrent); (4) ecological validity: the patient's symptoms reflect the characteristics of the illness in a real-world setting (frequency, intensity, duration, course, and impact over at least 4 weeks); and (5) rule of the 3 Ps: identified symptoms must be pervasive, persistent, and pathological. The three Ps must interfere with function and quality of life.

Massachusetts General Hospital Antidepressant Treatment Response Questionnaire

The MGH ATRQ is a clinician-rated questionnaire that examines the patient's antidepressant treatment history, using specific anchor points to define the adequacy of both dose and duration of each antidepressant trial, and the degree of symptomatic improvement obtained with each trial. This questionnaire, which has been validated,[10] allows for the determination of treatment resistance in depression. The remote raters performed either the Montgomery-Asberg Depression Rating Scale or the 17-item Hamilton Depression Rating Scale as required by each protocol to verify severity of depression for eligibility.[11,12]

Statistical Methods

For continuous measures, descriptive statistics consisted of means and standard deviations, and for rates, descriptive statistics consisted of percentages or proportions. For continuous measures, statistical comparisons were conducted by independent t tests, and for rates were computed by chi-square tests or Fisher exact test.

RESULTS

Across 7 independent trials of MDD and 2 of TRD, 2734 independent remote SAFER Interviews were performed. In these 9 trials involving 2734 interviews of patients, 2315 patients (84.67%) were deemed eligible for continued screening (ie, “passed” the SAFER Interview). The results are summarized in Table 1. Among the excluded patients, 33 (1.21%) did not meet severity criteria only as specified by the depression rating scales in the study inclusion criteria, 73 (2.67%) did not meet SAFER criteria only, 195 (7.13%) did not meet ATRQ criteria for treatment resistance only, and 3 (0.11%) met additional protocol exclusion criteria revealed during the interview, outside the scope of the severity, SAFER, or ATRQ measures. The remaining 115 (4.21%) patients did not meet criteria based on a combination of 2 or more components of the remote interview. Therefore, a total of 419 (15.33%) patients were deemed ineligible upon completion of the interview.

TABLE 1

Overall MDD and TRD Trials Results

Academic Versus Nonacademic Study Sites

Results from SAFER Interviews were compared between academic study sites and nonacademic sites and are summarized in Table 2. All 9 of the independent trials of MDD and TRD utilized at least 1 academic site. Across the academic sites in these 9 trials, 370 independent remote SAFER Interviews were performed. Of the 370 interviews, 325 (87.84%) patients were deemed eligible for continued screening. Of the remaining patients, 3 (0.81%) did not meet severity criteria only, 17 (4.59%) did not meet SAFER criteria only, 15 (4.05%) did not meet ATRQ criteria for treatment resistance only, and 3 (0.81%) met additional protocol exclusion criteria, outside the scope of the severity, SAFER, or ATRQ measures. The remaining 7 (1.89%) patients did not meet criteria on a combination of 2 or more components of the remote interview. A total of 45 (12.16%) patients were deemed ineligible upon completion of the interview.

TABLE 2

Comparison of SAFER Interview Results at Academic Versus Nonacademic Sites

Across the nonacademic sites in these 9 trials, 2364 independent remote SAFER Interviews were performed. Of the 2364 interviews, 1990 (84.18%) patients were deemed eligible for continued screening. Of the remaining patients, 30 (1.27%) did not meet severity criteria only, 56 (2.37%) did not meet SAFER criteria only, 180 (7.61%) did not meet ATRQ criteria for treatment resistance only, and 0 (0.00%) met additional protocol exclusion criteria outside the scope of the severity, SAFER, or ATRQ measures. The remaining 108 (4.57%) patients did not meet criteria on a combination of 2 or more components of the remote interview. A total of 374 (15.82%) patients were deemed ineligible upon completion of the interview.

US Versus Non-US Study Sites

Results from SAFER Interviews were compared between study sites in the United States and sites outside the United States, which included Australia, Belgium, Bulgaria, Canada, Germany, Hungary, Moldova, Poland, Romania, Russia, and Ukraine. Of the 9 independent trials of MDD and TRD, 7 used at least some sites located in the United States. Across these 7 trials, 2397 independent remote SAFER Interviews were performed in the United States. Of the 2397 US interviews, 1996 (83.27%) patients were deemed eligible for continued screening. Of the remaining patients, 31 (1.29%) did not meet severity criteria, 69 (2.88%) did not meet SAFER criteria only, 187 (7.80%) did not meet ATRQ criteria for treatment resistance, and 0 (0.00%) met additional protocol exclusion criteria outside the scope of the severity, SAFER, or ATRQ measures. The remaining 114 (4.76%) patients did not meet criteria on a combination of 2 or more components of the remote interview. A total of 401 (16.73%) patients were deemed ineligible upon completion of the interview.

Of the 9 independent trials of MDD and TRD, 4 used sites located outside of the United States. Across these 4 trials, 337 independent remote SAFER Interviews were performed outside of the United States. Of the 337 interviews, 319 (94.66%) patients were deemed eligible for continued screening. Of the remaining patients, 2 (0.59%) did not meet severity criteria, 4 (1.19%) did not meet SAFER criteria only, 8 (2.37%) did not meet ATRQ criteria for treatment resistance, and 3 (0.89%) met additional protocol exclusion criteria outside the scope of the severity, SAFER, or ATRQ measures. The remaining 1 (0.30%) patient did not meet criteria on a combination of 2 or more components of the SAFER Interview. A total of 18 (5.34%) patients were deemed ineligible upon completion of the interview.

Because of the smaller number of non-US sites, independent sample t tests were conducted to compare interview results between the different types of sites. A significantly greater number of SAFER Interviews per site were performed at sites in the United States (P ≤ 0.001). The mean rate of interview passes per site was significantly higher at sites outside the United States (P = 0.010) (Table 3).

TABLE 3

Comparison of SAFER Interview Results at US Versus Non-US Sites

Placebo Response Rates

Placebo response rates provide a signal as to whether the SAFER Interviews added to the quality of the studies and potential for signal detection. We did not have a control group of studies conducted in parallel, but report the placebo response rates in Table 4.

TABLE 4

Studies Using the SAFER Interviews and Placebo Response Rates

In all of the studies, placebo response rates were within a range of 13.0% to 27.3%, below the 30% to 40% average reported in antidepressant clinical trials.[1,2,19]

DISCUSSION

As demonstrated by these findings, a substantial proportion of patients who were initially deemed eligible by research site-clinicians proved not eligible after a comprehensive remote evaluation by experienced interviewers. The remote interviews consisted of the SAFER tool, the ATRQ, and a measure of depression severity. Of the trials described here, overall 15.33% of the patients were determined not to meet enrollment criteria on at least 1 component. Thus, these patients would have been enrolled into the trials had there not been a remote interview. We found that the elements of the interview most likely to lead to ineligibility were the verification of treatment resistance (as determined by the ATRQ) and the SAFER tool. Interestingly, we found that failure rates were higher among US compared with non-US sites. We also found a trend for higher failure rates among nonacademic sites compared with academic sites. Finding that 1 in approximately 7 patients deemed eligible for enrollment by site investigators was observed to be ineligible through an independent interview is remarkable, as the sites were aware of the process in place and therefore perhaps more likely to avoid the enrollment of inappropriate patients.

Although remote interviews in psychiatric research are not new, this is a novel application of a multidimensional interview conducted by experienced clinical psychiatrists and psychologists for quality assurance in clinical trials. Previous studies have demonstrated that interviews can be conducted remotely in trials of depression and other psychiatric disorders with good precision with the use of severity measures, and good acceptability among participants.[20-22]

The implications for precision in clinical trials of MDD and TRD are far reaching and have major implications for drug development and public health. Certainly, there is a cost associated with these remote quality assurance interviews by MD- and PhD-level clinical researchers, in terms of financial costs, and the additional efforts required to incorporate this step in trial implementation. However, failed trials are much more costly financially and, more importantly, present barriers to advances in the care of patients with psychiatric disorders. In addition, because placebo rates have risen, the risk of obscuring positive findings has heightened, making drug development more expensive and reducing the incentive for drug development in psychiatric disorders.[1,2] Higher placebo response rates necessitate larger trials to achieve adequate statistical power, resulting in higher overall costs of clinical research, a cost ultimately passed along to patients who have to pay higher prices for medications that are eventually marketed.

It is widely appreciated that patients who will likely be placebo responders are problematic to include in clinical trials because high placebo response rates increase the risk of failed trials. McCann et al have described these participants as those “destined to succeed” and have modeled the robust “threat” that they pose to drug development, much more so than participants who are “destined to fail” or are unlikely to respond to any treatment condition. As they discuss, the complicated overlapping relationships between nonadherence, “professional subjects” motivated by financial gain, and placebo response rates are aspects of clinical research that need to be addressed concurrently. However, individuals who are unlikely to respond to any treatment, such as those living in situations of socioeconomic deprivation or who have different kinds of situational stresses, are also of concern in trial enrollment and may impact outcomes in studies with lower placebo response rates.[23] The implementation of a remote interview to pare out those “destined” for response or nonresponse, regardless of study arm, is an important tool for the conduction of trials that will further the field.

The significant differences found in SAFER Interview failure rates are intriguing. The US sites had a significantly greater rate of SAFER failures. Compared with most of the non-US countries in which these studies were conducted, the United States lacks a national systematic health care system. Therefore, there are broad disparities in access to mental health services in the United States, and those who enter MDD/TRD trials may live with financial or environmental stressors that diminish the observable impact of an efficacious biological treatment. Similarly, the financial incentives that are offered to trial participants are likely to provide a stronger motivation for participation for those in more dire socioeconomic situations. Although we do not have specific data for each site's practices for patient remuneration, from the knowledge we have about these studies, in most of the patients in the United States, trials were remunerated, and most of those outside the United States were not. It is possible—and probable—that financial payments to patients may influence the recruitment and enrollment of ineligible patients.

We also saw a slight nonsignificant trend for a higher failure rate at nonacademic sites. A major difference between academic and nonacademic sites is the volume of patients screened and enrolled in trials of depression.[23] Nonacademic sites generally recruit a higher number of patients in a shorter amount of time per trial. Recruitment strategies may also differ, with some sites using databases of patients from previous studies. This lack of a significant difference is consistent with a finding seen in a previous analysis of one of the studies included in these analyses.[24] The current findings build upon this finding with a larger volume of patients screened and studies represented.

The strengths of this study include the assessment of the impact of remote assessments upon quality assurance in clinical trials for MDD/TRD. These assessments address multiple aspects of patient eligibility, are conducted by highly experienced clinician/researchers who were trained in remote interviewing, and incorporate a systematic process for managing inconclusive eligibility determinations by individual raters. The study also had limitations. The interviews were conducted by MD- and PhD-level clinical researchers, and it is not known whether reinterview at the sites or interview by individuals of other disciplines would have yielded similar results. We were not able to have a “control” group of studies matched for study drug, indication, study sites, and study size without the remote interview to assess for differences in study outcomes such as drug and placebo response rates. However, that said, the placebo response rates observed in the 9 consecutive trials that we examined were in ranges (13.0%–27.3%) well below the 30% to 40% average reported in standard antidepressant clinical trials,[1,2,19] suggesting a robust clinical research utility.

In summary, the remote interview based on the SAFER, MGH ATRQ, and symptom severity instruments permits greater vigilance over patients enrolled into clinical trials and provides quality assurance rendered by experienced clinician researchers. These and other quality assurance measures should be implemented to ensure that the drug development process is precise and efficient and can ultimately bring effective drugs to market for patients who are not adequately treated with available options.