Physician-Reported Experience and Understanding of Adverse Event Attribution in Cancer Clinical Trials.

OBJECTIVES: To report the results of a survey conducted among Mayo Clinic medical oncologists, hematologists, and cancer prevention specialists to better understand the current practice of determining whether an adverse event that a patient experience in a clinical trial is related to the drug under investigation, a process commonly known as attribution, as well as to formulate recommendations for an improved system. PATIENTS AND METHODS: An electronic survey was developed and conducted (from August 2 through 29, 2017) among 165 medical oncologists, hematologists, and cancer prevention specialists at the 3 Mayo Clinic sites: Rochester, Minnesota; Scottsdale, Arizona; and Jacksonville, Florida. The survey included 21 items that queried clinicians about their clinical practice and trial experience, their training and process in adverse event attribution assignment, and their recommendations for improving the current attribution system.
RESULTS: Thirty-seven percent (61 of 165) of physicians responded to the survey. The median number of years in clinical practice was 15 (range, 1-64) and that of clinical trial experience 12. Eighty-nine percent (54 of 61) had served as a trial principal investigator. Only 15% (9 of 60) of responders reported having received any formal attribution training. Eighty percent (48 of 60) were confident about their ability to assign attribution. Seventy-five percent (45 of 60) consulted their colleagues or study chair when assigning attribution. Sixty-seven percent (40 of 60) recommended formal training to improve attribution accuracy.
CONCLUSION: Very few clinical trialists in our survey received any formal training for adverse event attribution, yet most identified formal training as effective means to improve attribution accuracy. These data underscore an unmet need of formal adverse event attribution training among clinical trialists.

The reporting of adverse events in clinical trials and assigning their relatedness to the drug under investigation, also known as attribution, are mandated by regulatory agencies, such as the National Cancer Institute (NCI) and the US Food and Drug Administration (FDA). Based on the NCI Cancer Therapy Evaluation Program guidelines, adverse event attribution is categorized as “not related,” “unlikely related,” “possibly related,” “probably related,” and “definitely related” to study treatment. Assignment of adverse event attribution is a time-consuming process that is required for every cancer drug under clinical investigation.

Mukherjee et al conducted 32 semi-structured interviews with medical oncologists and trial coordinators and found that the process is often performed without complete clinical or investigational data. At the same time, there is no clear guidance from the NCI or the FDA on how to undertake this task.1, 2, 4 A recent research statement issued by the American Society of Clinical Oncology–American Association of Cancer Institute’s Best Practices in Cancer Clinical Trials Initiative, which provided recommendations to ensure adverse event reporting is meaningful and informative, acknowledged the challenges of adverse event attribution yet offered no specific guidance for attribution assignment.

Previous research has found that adverse event attribution data can be unreliable. Data from 2 multicenter, randomized, double-blind, placebo-controlled drug trials that included 398 placebo-exposed patients indicated that approximately 50% of adverse events in the placebo (inert substance) arm were ascribed to the drug under investigation. A more recent pooled analysis using data from 9 randomized, double-blind, placebo-controlled trials in 3 settings—cancer treatment, symptom intervention, and cancer prevention—also revealed that across all trials and settings, a very high proportion of adverse events reported as related to an investigational drug were classified as possibly related, a substantial proportion of adverse events in the placebo arm were incorrectly reported as related to the investigational drug, and clinician-reported attribution overestimated the rate of adverse events related to an investigational drug. Fatigue, nausea, vomiting, diarrhea, constipation, and neurosensory symptoms were the most common adverse events that clinicians overreported as related to an investigational drug.

On the basis of these results, we recommended that the collection of adverse event attribution in randomized, double-blind, placebo-controlled trials be eliminated. In these trials, excess toxicity from the investigational drug can be reliably determined by comparing the toxicity rates between the investigational arm and the placebo arm. Even in randomized double-blind trials comparing a new therapy (or a new combination) with the standard of care, additional toxicity from the new therapy can also be estimated by comparing toxicity rate reported in the investigational arm with that from the standard-of-care arm, much like in placebo-controlled trials. Therefore, adverse event attribution can also be eliminated in this setting.

However, in nonrandomized trials or in single-arm trials, a trial design often used in very early drug development studies, such comparisons are not available. Physicians’ judgment of whether an adverse event is caused by the investigational drug is extremely important to determine whether the drug is safe for patients. Reliable attribution is especially important in early drug development settings. The importance of attribution accuracy is not limited to clinical trials setting, but has a larger impact on clinical practice setting because the ultimate goal of drug development is to market the drug to the broader patient population.

The objectives of this article are to report the results of a survey conducted at Mayo Clinic to better understand the factors that are part of the current adverse event attribution process and to help formulate recommendations, including educational recommendations, for an improved system. This is a first step toward an effort to ensure that new drugs are safe for patients and that the development of promising cancer drugs is not halted for unfounded safety concerns. In view of our institution’s 3-site national structure and its long-standing track record of practice-changing clinical trial research—for example, our institution served as the scientific and coordinating center for the first study to define a role for adjuvant chemotherapy for resected colon cancer and the first to report a role for trastuzumab as adjuvant therapy for breast cancer—we decided that a single-institution survey would provide highly efficient yet uniquely informative insight into this important drug development issue.

Participants and Methods

This study was reviewed and approved by the Mayo Clinic Institutional Review Board.

Survey Development

The novelty of the question being posed required the development of a new survey. The study team met to review the published data on the topic and to decide on survey subject matter. Demographic questions for survey respondents were carefully framed to ensure anonymity. A list of questions related specifically to adverse event drug attribution were formulated and then revised by members of the study team. The survey was reviewed and revised after consultation with survey experts from the Mayo Clinic Survey Research Center (an institutional core facility for conducting surveys). The survey was then tested on a separate group of investigators involved in clinical trials. These investigators were excluded from the recipient pool of the final survey. The survey was further revised on the basis of these investigators’ questions and comments. After further multiple iterations, the survey was deemed finalized. The final survey included 21 items that queried physicians about their experience in clinical practice and in clinical trials, their training in adverse event attribution assignment, and their recommendations for improving the current adverse event attribution system. Table 1 lists the domains and questions included in the survey.

Table 1

Survey Domains and Questions

Domain	Questions (response options)
Professional experience and demographic characteristics	• Years in clinical practice (numerical) • Areas of clinical specialty (solid tumors, hematologic malignant neoplasm, other – specify) • Areas of research interest (cancer therapeutics, symptom management and survivorship, cancer prevention, other – specify, none) • Age (≤35, 36-45, 46-55, 56-65, >65 y) • Sex (female, male)
Clinical trial experience	• Ever enrolled and/or assessed patients on a clinical trial (yes, no) • Years of clinical trial experience (numerical) • Experience as a principal investigator of clinical trials (yes, no) • Type of trials in which involved as an investigator (phase I, II, or III; single-center or multicenter; government funded or industry funded) • Number of trials involved in each type of trials or funding sources (1-5, 6-10, 11-20, >20)
Adverse event attribution process	• Previous training (formal, informal/on the job, none) • Confidence in adverse event attribution reported in the literature (very confident, somewhat confident, neither confident nor not confident, somewhat not confident, not confident at all) • Confidence in own ability to assign attribution (very confident, somewhat confident, neither confident nor not confident, somewhat not confident, not confident at all) • Second thoughts about adverse event attribution assignment (yes, no) • Team members consulted (nurse, pharmacist, clinical colleagues, study principal investigator, clinical research associate, patient, family/caregiver) • Information used to assign attribution (patient age, performance status, comorbidities, cancer, cancer stage, tumor burden, previous cancer treatment, baseline adverse events, adverse events from previous cycles, adverse event profile of the study intervention, concurrent medications/treatment, other) • Suggested effective training to improve the adverse event attribution process (formal, mentoring, informal on the job, regulatory guidance, other)

Survey Dissemination

The inclusion criteria for survey recipients were staff medical oncologists, hematologists, and cancer prevention specialists, who had an FDA 1572 form on file (a requirement to enroll patients onto clinical trials), at the 3 Mayo Clinic sites: Rochester, Minnesota; Scottsdale, Arizona; and Jacksonville, Florida. Recipients’ names and e-mail addresses were obtained from the department rosters. The final survey was distributed to physicians who met the inclusion criteria. To ensure that survey responses remained anonymous to the study investigators, the survey was conducted via the Mayo Clinic Survey Research Center, and the study investigators received only de-identified responses. The survey was conducted electronically from August 2 through August 29, 2017. An e-mail invitation with a survey link was sent to each physician, with a request to complete the survey within 2 weeks. After the 2-week period, the Mayo Clinic Survey Research Center sent an e-mail to remind physicians to respond to the survey if they had not done so. The survey was closed permanently after a total of 4 weeks.

Analyses

The survey responses were summarized using descriptive statistics with median (range) for continuous variables and frequency (percentage) for categorical variables. Bar graphs and box plots were used for data visualization. Because the goal of the survey was to understand and describe physicians’ experience rather than test specific hypotheses, no formal statistical tests were conducted.

Results

Demographic Characteristics

One hundred sixty-five medical oncologists, hematologists, and cancer prevention experts at the 3 Mayo Clinic national sites met the inclusion criteria. Sixty-one of 165 (37%) responded to the survey. The median survey completion time was 3.7 minutes, with 54 of 61 (89%) responders having completed the survey in less than 10 minutes. One responder started but did not complete the survey, leading to missing response for some questions. Twenty-six of 61 (43%) responders were younger than 45 years, and 17 of 61 (28%) were female. The median number of years in clinical practice was 15 (range, 1-64). Most responders (38 of 61 [62%]) specialized in solid tumors, and 50 of 61 (82%) had research interests in cancer therapeutics.

Clinical Trial Experience

All 61 survey responders had enrolled and/or assessed patients in 1 or more multicenter trials. The median years of experience in clinical trials was 12 (range, 2-45). Fifty-four of 61 (89%) had served as the principal investigator on 1 or more clinical trials. The number of trials on which they had enrolled and/or assessed patients is given in Table 2 according to the trial funding mechanism. Across all types of trials and funding sources, more than 50% had experience with at least 10 clinical trials.

Table 2

Frequency at Which Surveyed Physicians Enrolled and/or Assessed Patients in Various Types of Trials (N=60)

Experience	No. of trials
	≤10	11-20	>20
Multicenter trials	19 (32)	13 (22)	28 (47)
Trials conducted through the National Cancer Institute Clinical Trials Network	29 (48)	14 (23)	17 (28)
Investigator-initiated trials funded by industry	31 (52)	8 (13)	21 (35)
Industry-initiated trials	31 (52)	7 (12)	22 (36)

Data are presented as No. (percentage).

Adverse Event Attribution Experience

Only 9 of 60 (15%; 1 incomplete response) physicians reported having received any type of formal training for adverse event attribution, 42 (70%) reported having had some informal training, and 9 (15%) reported no training. Figure 1 presents the responders’ level of confidence (on a 5-point scale) in the accuracy of adverse event attribution as reported in published clinical trials and their ability to assign adverse event attribution. Thirty-six of 60 responders (60%) reported being “very” or “somewhat” confident about the accuracy of adverse event attribution reported in published clinical trials, while 48 (80%) were “very” or “somewhat” confident about their own ability to assign adverse event attribution (Figure 1). However, when asked if they ever had second thoughts about their own adverse event attribution assignment, 55 (92%) answered “yes.”

Figure 1

Confidence in accuracy of adverse event attribution (N=60).

Frequencies of consultation with other team members are presented in Figure 2A. Team members most often consulted when assigning attribution were clinical colleagues (n=45 [75%]) and the study principal investigator (n=44 [73%]). Only 23 responders (38%) reported consulting a clinical research associate, and 21 (35%) reported consulting with patients.

Figure 2

A, Team members consulted when assigning attribution (percent response of 60 responders; responders can select multiple responses). B, Information used to determine attribution (percent response of 60 responders; responders can select multiple responses). C, Suggested effective training for adverse event attribution (percent response of 60 responders; responders can select multiple responses).

Information used to determine attribution is summarized in Figure 2B. History of adverse events including baseline (87%) and from previous cycles (85%) was reported as most often considered when assigning attribution. The adverse event profile of the study intervention, concurrent medications, and comorbidity were the next set of factors being considered in the attribution process by 77%, 77%, and 73% of responders, respectively. Previous cancer treatment, cycle of current cancer treatment, and type of cancer were reported being considered by 62%, 62%, and 58% of responders, respectively.

When asked for their recommendation for effective training to improve adverse event attribution, 40 responders (67%) selected formal training while 26 (43%) suggested mentoring or informal training (Figure 2C). Exploratory analyses evaluating the association between a physician’s confidence in his or her own ability to assign attribution with experience and age revealed no clear trend of association (Supplemental Figure A and B, available online at http://mcpiqojournal.org).

Discussion

Adverse event attribution (assessment of whether an adverse event is related to the treatment) within the context of a clinical trial is challenging. Many patients have multiple comorbidities from their disease. Patients who had previous treatments may experience residual effects from their previous therapies. Concomitant medications that patients with cancer receive during a course of treatment can also cause adverse effects, making attribution more difficult. As a presumed result of all the above, our current system yields meaningless data, as we have previously shown.6, 7

However, particularly, in single-arm phase I or phase II trials, physicians’ determination of attribution is essential for patient safety and for ensuring that the development of novel cancer drugs proceeds with an in-depth understanding of drug outcomes. To our knowledge, the present study is the largest survey conducted to understand the current process and to solicit recommendations, directly from clinical trial investigators, to improve the current system to ensure that the drug development process proceeds with accuracy.

The present study was undertaken within 1 large 3-site comprehensive cancer center in which physicians have access to colleagues with extensive clinical trial experience. Our survey reported that very few clinical trialists received formal training for adverse event attribution while the majority received only informal/on-the-job training, and some never had any training at all. Importantly, clinical trialists who were surveyed have a large network of clinical colleagues whom they can draw on during the attribution process. Hence, clinical trialists in our survey reported confidence in their ability to accurately attribute adverse events.

Most experienced physicians and clinical trialists, who responded to our survey, still identified a need for formal training in adverse event attribution. Such training might come in the form of modules that could be appended to current training curricula for clinical trial investigators. It is important to note that not all investigators who enroll and/or assess patients in multicenter clinical trials have a similar experience or have a similar network of colleagues to consult. In fact, investigators in multicenter trials are typically from smaller institutions that likely enroll only a few patients to each trial. This lack of access to a large network of experienced clinical trialists combined with the lack of formal training may contribute to the inaccuracy of adverse event attribution in multicenter trials as previously reported. In this context, our survey findings might be underestimating the extent to which formal training in adverse event attribution assignment might be needed.

Conclusion

The biggest challenge with adverse event attribution is assessing the truth of whether an adverse event is caused by the experimental drug. However, formal training in addition to on-the-job mentoring will help establish a standardized process that will lead to better consistency of attribution across investigators. Although consistency alone does not guarantee accuracy, attribution consistency would be an achievable first step toward a more standardized process that, coupled with further understanding and improvement of the current system, could lead to better accuracy. Further research and close collaborations among stakeholders, including clinical trialists and representatives from regulatory agencies, are needed to develop optimal training modules that are effective but at the same time not too taxing to the current training curriculum.