Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology.

INTRODUCTION: Orthopaedic oncology researchers face several obstacles in the design and execution of randomised controlled trials, including finite fiscal resources to support the rising costs of clinical research and insufficient patient volume at individual sites. As a result, high-quality research to guide clinical practice has lagged behind other surgical subspecialties. A focused approach is imperative to design a research programme that is economical, streamlined and addresses clinically relevant endpoints. The primary objective of this study will be to use a consensus-based approach to identify research priorities for international clinical trials in orthopaedic oncology. METHODS AND ANALYSIS: We will conduct a 3-phase modified Delphi method consisting of 2 sequential rounds of anonymous web-based questionnaires (phases I and II), and an in-person consensus meeting (phase III). Participants will suggest research questions that they believe are of particular importance to the field (phase I), and individually rate each proposed question on 5 criteria (phase II). Research questions that meet predetermined consensus thresholds will be brought forward to the consensus meeting (phase III) for discussion by an expert panel. Following these discussions, the expert panel will be asked to assign scores for each research question, and research questions meeting predetermined criteria will be brought forward for final ranking. The expert panel will then be asked to rank the top 3 research questions, and these 3 research questions will be distributed to the initial group of participants for validation. ETHICS AND DISSEMINATION: An ethics application is currently under review with the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada. The results of this initiative will be disseminated through peer-reviewed publications and conference presentations. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

The primary objective of this study will be to identify research priorities for international clinical trials in orthopaedic oncology.

The Delphi method has been used extensively in various fields of healthcare programme planning and research priority development, but this protocol represents the first application in the orthopaedic oncology field.

The anonymity of the Delphi method will minimise domination by individual participants and eliminate group pressures for conformity.

We will include participants and stakeholders from varying healthcare systems, organisations and career stages.

Purposive sampling will lead to the inclusion of participants interested in collaborative research, who may differ from those who decline participation.

Introduction

Barriers to conducting randomised controlled trials in orthopaedic oncology

Finite resources

Collaborative multicentre randomised controlled trials (RCTs) are considered the gold standard for the reliable evaluation of therapeutic interventions. However, large RCTs have become prohibitively expensive and increasingly complex, due, in part, to extensive regulations governing their conduct and finite fiscal resources.1 Surprisingly, most federal funding agencies typically spend only a small proportion of their budgets on RCTs. For example, the Canadian Institutes of Health Research allocates <5%, and the National Institutes of Health allocates <10%.2 Rigorous regulatory requirements, multicentre ethics approvals and institutional contracts further complicate the research process.3 A trial will succeed only if it can recruit the required sample size during a predetermined time period. Large trials require substantial resources to overcome these challenges and accomplish their recruitment goals.

Insufficient volume of rare cancers at individual centres

Sarcomas are rare forms of cancer and represent <1% of all malignancies. Bone sarcomas affect just four to five people per million persons each year.4 Insufficient numbers at individual clinical sites have historically challenged clinical studies of sarcoma patients. Therefore, multicentre collaboration is imperative to achieve statistical power in order to draw valid conclusions.5 As a result, the orthopaedic oncology community continues to face significant knowledge gaps in the surgical management of sarcoma patients, and many critical questions remain.

Rationale

Lack of evidence from randomised controlled trials

We previously conducted a systematic review to evaluate the quality of evidence available to guide the surgical management of lower extremity bone tumours.5 We identified a complete absence of collaborative multicentre RCTs in orthopaedic oncology, and found that retrospective observational studies remain the dominant source of knowledge. Our results highlight the lack of prospective collaborative research in this field and a dire need for innovation and focused effort on collaboration.6

A paradigm shift in the landscape of musculoskeletal oncology research

Orthopaedic oncologists have, for decades, discussed the need for multicentre RCTs, but few centres have the expertise and logistic support required to coordinate this type of research. The Prophylactic Antibiotic Regimens In Tumor surgerY (PARITY) trial (NCT01479283) is the first-ever multicentre surgical RCT in orthopaedic oncology, and it represents the first step in a paradigm shift toward evidence-based patient care in sarcoma surgery.7 The PARITY trial has the potential to significantly impact clinical practice well beyond its primary research objective because it is leading to the development of an international network of committed collaborators.8 Properly leveraged, this promising collaborative group could facilitate additional high-quality RCTs in the field.

Unmet needs in musculoskeletal oncology

We completed a qualitative study to explore and identify perceived facilitators and barriers to multicentre collaboration in the field. According to our focus group of 13 orthopaedic surgeons from seven countries, we determined that collaborators were enthusiastic to conduct further research, provided that the research questions are clinically important and feasible. We further identified shared attitudes valuing leadership, expertise, logistical support and centralised infrastructure. This study confirmed that further investigation into the unmet needs in orthopaedic oncology research is warranted, and supports our current protocol as the next logical step in the development of key strategic research priorities in the field.

Aims

Our primary objective will be to identify research priorities for international clinical trials in orthopaedic oncology. We will decide on three potential research questions that are clinically important and feasible. Our secondary objectives will be to address critical issues in sarcoma surgery research, understand the current state of evidence in order to inform study design, facilitate innovation and collaboration among stakeholders, explore possible funding mechanisms available to support our research and strategise how to engage relevant patient-advocacy groups for their input to develop and disseminate these strategic priorities.

Methods and design

Our planning initiative will employ a three-step modified Delphi method.9 The process will consist of a qualitative assessment (phase I), a ranking evaluation (phase II) and a consensus meeting (phase III) (see figure 1).

Figure 1

Planning initiative conduct procedure.

The Delphi method is a consensus-based technique that provides a systematic method of collecting and aggregating informed judgments from a group of experts via multiple iterations. Controlled feedback from sequential rounds encourages participants to reassess, alter and/or develop opinions. The Delphi method maximises the benefits of using an expert panel while minimising potential disadvantages by implementing anonymity.10 This method has been used extensively for programme planning and the development of research priorities in various areas of medicine, but this protocol represents the first application in orthopaedic oncology.11–15

Planning committee

We have formed a Planning Committee comprised of two orthopaedic oncologists, an orthopaedic Research Program Manager, the PARITY Project Manager, and a Health Research Methodology expert to oversee the design, execution and analysis of all phases of this study. Research methods were established in face-to-face meetings and email correspondence. Agreement was reached regarding participant selection, consensus thresholds, survey format and question structure, and analysis processes, according to the proposed quality indicators for a Delphi study.16

Participant selection

We will invite ∼200 clinician-scientists who are interested, or are actively participating in the PARITY trial, Musculoskeletal Tumor Society (MSTS) members, Connective Tissue Oncology Society (CTOS) members, International Society of Limb Salvage (ISOLS) members, and Orthopaedic Research and Education Foundation (OREF) partners to participate. We expect to include participants from North and South America, Europe, Asia, Africa and Australia, and we expect substantial variation in prior research experience, clinical focus and career stage.

Delphi procedure

Phase I—qualitative assessment

The first phase will use a web-based, open-ended questionnaire (see online supplementary appendix 1) asking participants to identify up to three research questions that they believe are of particular importance to the field of orthopaedic oncology. Participants will be invited to complete the questionnaire via email. Participants' responses will not be guided by accompanying facilitators or literature reviews. Instead, they will be asked to review the literature and consult with colleagues as they see fit prior to proposing their ideas. Participants will also be asked to provide some demographic information and to declare any potential financial or intellectual conflicts of interest. The questionnaires will remain online for up to 4 weeks, and reminder emails will be sent out approximately every 7 days after the initial invitation.

Once all the questionnaires have been received, we will compile the responses and review the proposals to generate a list of candidate research questions that will progress to phase II. Responses from phase I will be initially reviewed by a Planning Committee member who is familiar with orthopaedic oncology. Similar ideas will be clustered together into emerging research questions; duplicate responses and those deemed not applicable will then be removed. This review will be repeated independently by a Health Research Methodology expert. The two reviewers will then meet to discuss any differences produced from these independent analyses.

Phase II—ranking evaluation

During the second phase, participants will be emailed a questionnaire (see online supplementary appendix 2) asking them to rank each candidate research question individually on a 5-point Likert scale (see table 1) for five criteria: scientific merit, significance, innovation, relevance and feasibility. Participants will also be asked to further clarify or add to research questions, and comment on their rankings.

Table 1

Phase II 5-point Likert scale

1	2	3	4	5
Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree

Once all the questionnaires have been received, we will compile the rankings for each research question and any additional responses. Using predetermined consensus thresholds (see Table 2), we will decide which research questions will be brought forward to the consensus meeting for review. Research questions that meet the inclusion or non-consensus thresholds will progress to phase III for review. Research questions that meet the exclusion consensus threshold will not be brought forward for review. If none of the research questions meet the inclusion or non-consensus thresholds, the thresholds will be lowered until a critical mass of research questions are able to be brought forward to the next phase. We consider a priori that a critical mass of 10 priority research questions will suffice. However, Delphi processes tend to be flexible and, therefore, it is more likely that there will be a large number of questions brought forward for discussion.17 If this is the case, we will rank the questions in tiers and give priority to discussion of the highest ranking tier.

Table 2

Phase II consensus thresholds

Consensus thresholds
Inclusion	>75% of respondents provide a positive result (four or five) on the Likert scale for all criteria.
Exclusion	>75% of respondents provide a negative result (one or two) on the Likert scale for all criteria.
Non-consensus	When the proposed priority research question has met neither the inclusion nor exclusion consensus thresholds.

Phase III—in-person expert panel consensus meeting

We will hold an in-person expert panel consensus at the MSTS 2016 Annual Meeting. An experienced facilitator will moderate the meeting. We will invite all participants from phases I and II to participate, as well as representatives from patient advocacy groups, MSTS, CTOS, ISOLS and OREF. Should too many individuals express their interest in participating on the expert panel, we will select a diverse and balanced group comprised of 20 individuals, with at least one individual from each of the geographical, stakeholder and career stage groups. All individuals involved in phase III will be asked to explicitly identify any known biases prior to their participation in the consensus meeting (which will be documented using Conflict of Interest disclosure statements), and may be asked not to participate in certain parts of the discussion.18

We will implement Powercom anonymous electronic audience response devices to enable anonymous scoring throughout the meeting. A semistructured agenda will be provided to minimise time constraints, and to ensure that all individual participants are allowed a period of uninterrupted time to voice their opinions for each topic discussed (see figure 2). Each proposed research question will be individually discussed by the expert panel, thereby providing an opportunity for members to reconsider their initial ratings in light of other members' views. When differences in the ratings from phase II appear to have resulted due to ambiguity in the wording of the research question, the members will agree to a revised wording. Following these discussions, the members will be asked to anonymously assign a score from 1 to 9 for each research question (see table 3).

Figure 2

In-person expert panel consensus meeting agenda.

Table 3

Phase III scoring scale

1	2	3	4	5	6	7	8	9
Should not be studied	Lowest priority	Very low priority	Low priority	Medium priority	Slightly high priority	Moderately high priority	High priority	Highest priority

Once the scores have been compiled, those meeting one of the following predetermined criteria will be brought forward for final ranking:

100% of respondents scored the research question as either a seven, eight or nine; or

At least 10% of respondents scored the research question as a nine.

If none of the research questions meet these criteria, the top 10 scoring research questions will be brought forward for final ranking.

These highest ranked research questions will once again be discussed by the expert panel. The members will then be asked to rank their top three research questions. The three research questions with the highest mean scores (see table 4) will be validated by the initial group and recommended in the final report as priority questions for future research in the field.

Table 4

Phase III final ranking scoring system

Rank	1st	2nd	3rd
Points	3	2	1

We will also use the phase III consensus meeting to discuss critical issues in sarcoma surgery research, understand the current state of evidence in order to inform study design, facilitate innovation and collaboration among stakeholders, explore possible funding mechanisms available to support our research, and strategise how to engage relevant patient-advocacy groups for their input to develop and disseminate these strategic priorities.

Statistical analysis

We will report discrete variables as counts or proportions, normally distributed continuous variables as means with SDs , and skewed continuous variables as medians with IQRs. We will compare results across different stakeholder groups, regions and career stages using χ2 tests or ANOVA with individual pair-wise comparisons. Rank correlation will be determined using the Spearman coefficient. All tests of significance will be two-tailed, and p values of <0.05 will be considered significant.

Ethics

An ethics application is currently under review with the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada. Potential participants will be informed that responding to the questionnaires implies consent to participate in the initiative, and having their deidentified responses included in future iterations and any analyses. The consensus meeting facilitator will also clearly explain the purpose of the initiative to the expert panel participants and will obtain written informed consent prior to commencement of the meeting. All anonymised data will be kept on a password protected computer on a secure network and in a locked office, in accordance with local guidelines. The data will only be accessible to the Planning Committee, and will be destroyed after 10 years, as per local guidelines.

Discussion

The application of consensus methods is critical where unanimity of opinion does not exist due to a lack of or conflicting evidence on a given issue. These methods provide a means of synthesising the valuable insights of experts to either assess the extent of agreement (consensus measurement) or to resolve disagreement (consensus development).19 Accordingly, consensus methods are being increasingly employed to enhance the effective development of research priorities in medicine.11–13 15 Two consensus methods commonly used in medical research are the Nominal Group Technique (NGT) and the Delphi method.19

The NGT uses a structured meeting to facilitate the discussion and the collection of data for a particular issue.18 The direct interaction during the meeting allows participants to clarify and justify their points of view, which may help the group make more informed decisions.19 It is, however, often impractical to secure the participation of all key knowledgeable individuals.19 As previously described, the classical Delphi method is an iterative process used to solicit and distil the judgments of experts using a series of questionnaires alternated with feedback.10 This method allows for information to be obtained from a greater number of individuals and does not impose geographical constraints.10 However, a lack of face-to-face interaction prevents participants from exploring reasons for disagreements, thereby limiting the chance for opinions to be modified in response to such discussions.19 A hybrid of these two methods, the modified Delphi method, maximises the benefits of both consensus methods through the initial collection of information via questionnaires (consensus measurement) followed by a structured in-person meeting (consensus development).

The validity of several consensus methods has been questioned due to manipulation of the processes by organisers, domination of discussions by certain participants, and group pressure for conformity.19 To avoid imposing our views on participants and thereby introducing information error into the evaluation, we will begin with an open-ended questionnaire.9 Furthermore, in our modified Delphi approach, all responses will remain anonymous, and any discussion will be moderated by an experienced facilitator. The anonymity of phases I and II will ensure that all ratings are given the same importance in phase III.9 The discussion of phase III will provide an equal chance for each expert panel member to reassess their views through moderated discussion without compromising the anonymity of the scoring. These measures should ensure that various response biases are minimised, and that the final ranking does not merely reflect the perspectives of any single individual. Moreover, some individuals, especially those with minority views, may be more likely to drop out of a consensus process; this attrition can lead to overestimation of the final degree of consensus.

The purposive sampling employed by the Delphi method may lead to having only participants who are interested in collaborative research in the field.20 However, considering that our objective is to determine priority research questions in orthopaedic oncology in order to drive future multicentre prospective research, this bias is actually favourable, as participants will likely provide insightful ideas. Nevertheless, by including participants from different healthcare systems, organisations and career stages, we still plan on assembling a relatively diverse panel.

Dissemination plan

We expect that the results of this initiative will be published in a high-impact peer-reviewed journal, which will lend credibility to future grant applications and provide an opportunity for participants to publish as a collaborative group. Furthermore, the findings of this initiative will also be disseminated to regional, national and international audiences through presentations at relevant research conferences. Finally, the results of this study will be brought forward to initiate the next phase of collaborative research in orthopaedic oncology. Such collaborative research will be imperative to advance surgical care in patient with sarcoma.