Methods for Incorporating Stakeholder Engagement into Clinical Trial Design.

CONTEXT: Lack of engagement with healthcare stakeholders results in missed opportunities to understand translation of evidence into practice. CASE: Stakeholder engagement is a key component of the Comparing Outcomes of Drugs and Appendectomy (CODA) Study, a pragmatic clinical trial funded by PCORI to evaluate the effectiveness of antibiotics versus urgent appendectomy for acute uncomplicated appendicitis. We provide a framework for developing a stakeholder coordinating center (SCC) and describe two examples of how stakeholder engagement can inform study development.
FINDINGS: Coordinating engagement activities through the SCC established a commitment to the important partnership with stakeholders. It also facilitated communication and provided a central mechanism for obtaining input on key decisions such as development of patient-centered consent documents and appropriate stopping rules for a specific sub-population of patients with appendicitis. MAJOR THEMES: Translatable lessons include thoughtful planning for engagement, identifying stakeholders with a direct interest in the study conduct and findings, and integration of input received into the decisions that drive the conduct of the study.
CONCLUSIONS: Standards for conducting patient-centered research should address the ability to successfully engage patients by demonstrating the capacity to recruit study participants, engage them over the duration of the study, and disseminate findings that are congruent with stakeholder needs. The process of sharing important clinical research findings has improved patient care, and we believe that dissemination of novel engagement strategies can lead to increased success in study design and execution.

Introduction

The reference to “the ivory tower” in the context of academic research reflects the widely held belief that a disconnect exists between the traditional questions pursued and outcomes measured and those that are the most needed by patients, consumers, clinicians, policymakers and payers to make actual health care decisions. Lack of engagement may result in missed opportunities to understand the complexity of decision-making and the full impact of treatment options; thus, the translation of evidence into practice may not meet its full potential. Recognizing this divide, stakeholder engagement was specifically written as a founding principle of the Patient-Centered Outcomes Research Institute (PCORI) to better ensure that funded research addresses important and relevant evidence gaps. PCORI infuses stakeholder engagement throughout the work it conducts. Upon its establishment, the United States Government Accountability Office appointed a Board of Governors inclusive of diverse health care perspectives to guide all aspects of work. PCORI expects similar engagement in funded research. Submitted proposals for funding are evaluated on specific criteria addressing how patients and stakeholders are involved throughout the research process [1].

Evidence for effective stakeholder engagement in developing and conducting research is evolving. Current guidance on engagement practices presents myriad ways stakeholders can inform and collaborate to support research, without the assumption that one prescribed approach exists. A number of factors may influence stakeholder engagement strategies: desired perspectives, purpose of engagement, time frame, and available resources [2345]. The authors conclude that rather than promoting specific strategies, thoughtful consideration and planning should be devoted to engagement—to ensure that the subsequent selection of engagement activities represents the broad needs of stakeholders while achieving informed research [6]. While methods for stakeholder engagement continue to emerge, very little has been written about stakeholder engagement in the context of surgical research [7].

Stakeholder engagement is a key component of the Comparing Outcomes of Drugs and Appendectomy (CODA) study. CODA is a pragmatic clinical trial funded by PCORI to evaluate the effectiveness of antibiotics versus urgent appendectomy for acute uncomplicated appendicitis (AUA). For over 100 years, appendectomy has been the standard treatment for AUA but recent evidence suggests that nonoperative management is a safe alternative [89]. However, existing studies have not included patient-centered outcomes such as quality of life, decisional regret, anxiety, and other patient-reported outcomes that may be associated with treatment decision [9]. The CODA study aims to answer these important questions in order to inform patient-centered decision-making about treatment options.

We describe above methods for stakeholder engagement in research proposal development prior to the PCORI funding announcement; [10] we next describe methods for involving patients and other stakeholders in all aspects of research conduct and trial design. We provide a framework for developing a stakeholder coordinating center (SCC) and describe two examples of how stakeholder engagement informed study development.

Case Description

Development of a Stakeholder Coordinating Center

The organizational infrastructure for large clinical trials frequently includes data coordinating and clinical coordinating centers. While incorporation of advisory boards is not new, formal inclusion of a center to coordinate engagement activities is novel. Stakeholders play an integral role in the conduct of the CODA trial. We established the SCC in parallel to the other major study organizational components. The SCC is a recognized core within CODA’s organizational structure (Figure 1). It coordinates all engagement activities among a diverse group of stakeholders, including representatives from the patient population of interest (those at risk for or who have had AUA); clinicians (including surgeons, emergency physicians, nurses, and surgeons); leaders of professional societies (American College of Surgeons, American College of Emergency Physicians, and AcademyHealth); representatives of Accountable Care Organizations; policymakers; insurers and payers; researchers; and leaders of large, self-insured employers. The goal of the SCC is to inform, develop, and refine the research questions and protocol. Stakeholders participate via teleconferences, in-person meetings, and direct outreach activities.

Figure 1

Comparing Outcomes of Drugs and Appendectomy (CODA) Study Organizational Structure

The SCC works directly with the Clinical and Data Coordinating Centers to ensure stakeholders are kept up-to-date on study progress and findings, and inform key decisions as these pertain to study development and conduct primarily through established advisory groups. Advisory group membership, role, and activities are presented in Table 1.

Table 1

The Stakeholder Coordinating Center Manages Diverse Advisory Group Activities

ADVISORY GROUP	MEMBERSHIP	ROLE FOR ENGAGEMENT 1
Patients Advisory Group	Patients with a history of, or at risk for, appendicitis. Caregivers and family representatives.	Provide guidance about the patient and caregiver experience in research and health care, advise on protocol refinement and informed consent, assist with drafting patient materials and recruitment strategies, discuss challenges that arise during the study, assist with interpreting findings, advise on dissemination activities, and engage in co-learning and teaching with other advisors.
Clinician Advisory Group	Clinicians (emergency physicians, nurses, and general surgeons) from all CODA practice sites.	Advise on protocol refinement, additional scientific questions that emerge, and changes in technology or interventions. Identify and discuss challenges with study conduct, assist with interpreting findings, and advise on dissemination activities.
National Stakeholder Advisory Group	Leaders of professional societies, representatives of Accountable Care Organizations; policymakers; insurers and payers; researchers; and leaders from large, self-insured employers.	Support national outreach, dissemination, and implementation of the results by providing a national perspective on metrics, approaches, and materials and messages related to the study.

Stakeholder Engagement in the Research Development

Study protocol development relied on the coordinated input from SCC members, specifically in the development of educational materials related to informed consent and study safety rules for specific subpopulations. We describe how the SCC contributed to these important changes to the study.

Informed Consent Documentation: Occult Appendiceal Cancer

Following the PCORI funding announcement, patient stakeholders raised concern that nonoperative management of AUA may lead to missed cases of occult appendiceal cancer. Appendiceal cancer is a rare disease diagnosed in 0.7–1.7 percent of pathologic specimens of patients with appendicitis who undergo urgent appendectomy [11]. However the true prevalence of the disease among patients with AUA is unknown. Estimates from case series of patients initially treated with antibiotics for appendicitis who later had interval appendectomy are as high as 12–29 percent [1112]. Advocates from the appendiceal cancer community stated their concerns about the study both in online forums and through email communication with the study team. While we had a well-established advisory group of patient partners within the SCC, we were lacking representation from the broader appendiceal cancer community. A representative from this community agreed to join the research team as a patient partner to ensure that concerns regarding appendiceal cancer were addressed in the study design.

Through discussion between the study team and CODA patient partners, consensus was reached that study consent materials must directly address the potential risk of missed cancer in patients randomized to the antibiotics treatment group. The original consent documents did not include any mention of appendiceal cancer. After engagement with patient partners, consent documents were revised to include the following statement: “If we treat patients with antibiotics only (without surgery), we might miss the opportunity to see another problem that we did not expect to find (such as a small mass that was not seen on the computed tomography [CT] scan or ultrasound test you had in the diagnosis of appendicitis).” The final study protocol was revised to state that patients who were treated nonoperatively and had ongoing abdominal symptoms should be surveyed with a repeat CT scan to evaluate for changes that might be associated with appendiceal cancer

Trial Design and Stopping Rules Regarding Presence of Appendicolith

Stakeholder engagement is also critical when considering specific subpopulations who may have a higher risk of experiencing complications or treatment failure. The inclusion of diverse stakeholders provides an opportunity to discuss safety and study stopping rules more robustly, to determine appropriate safety end points. One specific subpopulation of interest in CODA includes patients with an appendicolith, or a small stone in the appendix. A previous randomized study suggests that patients who were found to have an appendicolith were more likely to have complicated appendicitis, and this group had a higher risk for failure of the antibiotic approach [13]. This observation prompted the question of whether patients with an appendicolith should be included in the trial, as complicated appendicitis is an exclusion criterion for the CODA trial. However surgeons do not agree on a standard definition for “complicated” appendicitis. Because of conflicting evidence regarding this issue, we questioned whether there should be a prespecified “stopping rule” to determine when and if we would need to exclude patients with appendicoliths for safety reasons. The specific question was, “How low of a success rate for antibiotics in people with stones should we tolerate before including patients with stones?”

We first invited SCC patient advisors to learn about this topic from study investigators. We then provided three ranges of potential success rates (1–25 percent, 26–50 percent, 51–75 percent) and asked them to vote on which would be the most appropriate to use as a safety endpoint. Given the conflicting evidence regarding whether the presence of appendicolith is associated with a greater risk of antibiotic failure, patient advisors were not comfortable making this decision alone and coordinated a brief survey to circulate to a broader group of stakeholders (patients, researchers, payers, and clinicians). Among 38 individuals who completed the survey, there was no clear consensus on the most appropriate stopping point. Approximately one-third of respondents said that they would tolerate a treatment success rate as low as 1–25 percent, while approximately one-quarter said they would tolerate a success rate of 26–50 percent. The remaining respondents said that they would require a success rate of at least 50–75 percent in this population.

Based on the information obtained from this survey no formal stopping rules were proposed because stakeholders determined that information for this substantial group of patients would be informative to decision makers even if only a small proportion (<25 percent) had successful treatment with antibiotics. However, the data safety monitoring board will review safety and efficacy data reports every six months for the duration of the trial, with planned interim analyses to determine safety and efficacy.

Major Themes

The SCC is demonstrating early effectiveness for supporting engagement activities and integration of diverse perspectives in key decisions. The intentional decision to establish an SCC was, from the outset, made recognizing that patients and clinicians may have different perspectives and needs with regards to evidence generation. Involvement of professional societies, insurers, payers, and employers through the National Advisory Group provides a unique opportunity to discuss the study conduct with a focus toward the future dissemination of research findings. The SCC facilitates communication, feedback, and diverse stakeholder input and promotes dedicated forums for these groups to openly discuss study questions and protocols. We recognize that the organization of the SCC was designed for the particular needs of this study and therefore might not be generalizable to other studies. Translatable lessons include thoughtful planning for engagement, identifying stakeholders with a direct interest in the study conduct and findings, and integrating input received into the decisions that drive the conduct of the study

We present two examples of how stakeholder engagement supports the conduct of a pragmatic clinical trial. In the first example, the guidance for handling the disclosure for appendiceal cancer in the consent process was clear In contrast, we did not obtain consensus on a stopping rule for appendicolith despite considerable time and effort. While no change in the study protocol occurred, it provided a valuable opportunity to discuss the state of uncertainty in the evidence and to vet our prior decision not to include prespecified stopping rules. Recognizing this has potential safety implications for patients participating in the study it was time well spent, although the lack of tangible change may make it appear as if valuable resources were expended to no avail. Stakeholder engagement requires dedicated time and resources. Identifying topics and issues where stakeholder input can best support research is a critical step, recognizing that not all outcomes from such efforts will yield change.

We also recognize that engagement needs may change during the study. Early phases may call for more intense engagement when determining study conduct and outcomes. In these examples we found that early efforts needed to focus on big picture issues that would affect all study participants, such as the informed consent process and the study’s organizational structure, and that required more time and effort. As recruitment and study processes are now underway activities focus more on updates and refinement of study processes.. Ongoing efforts will focus on sustaining engagement and communication with all stakeholders as well as planning for dissemination.

Conclusion

The research community is increasing efforts to meaningfully engage stakeholders throughout the research process. We describe a novel approach to engaging stakeholders in a PCORI-funded pragmatic clinical trial. While we have successfully recruited a diverse group of stakeholders through the SCC to inform the trial going forward, we will not fully know the success or failure of our work until the trial’s conclusion. Going forward, standards for conducting patient-centered research should evaluate the success of engagement efforts in relation to important markers of study success: recruitment and retention of study participants, timeliness of study completion, and dissemination of research findings congruent with stakeholder needs. The process of sharing important clinical research findings has improved patient care, and we believe that dissemination of novel engagement strategies can lead to increased success in study design and execution.