Development of a core outcome set for medication review in older patients with multimorbidity and polypharmacy: a study protocol.

BACKGROUND: Medication review has been advocated to address the challenge of polypharmacy in older patients, yet there is no consensus on how best to evaluate its efficacy. Heterogeneity of outcomes reported in clinical trials can hinder the comparison of clinical trial findings in systematic reviews. Moreover, the outcomes that matter most to older patients might be under-reported or disregarded altogether. A core outcome set can address this issue as it defines a minimum set of outcomes that should be reported in all clinical trials in any particular field of research. As part of the European Commission-funded project, called OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly, this paper describes the methods used to develop a core outcome set for clinical trials of medication review in older patients with multimorbidity. METHODS/
DESIGN: The study was designed in several steps. First, a systematic review established which outcomes were measured in published and ongoing clinical trials of medication review in older patients. Second, we undertook semistructured interviews with older patients and carers aimed at identifying additional relevant outcomes. Then, a multilanguage European Delphi survey adapted to older patients was designed. The international Delphi survey was conducted with older patients, health care professionals, researchers, and clinical experts in geriatric pharmacotherapy to validate outcomes to be included in the core outcome set. Consensus meetings were conducted to validate the results. DISCUSSION: We present the method for developing a core outcome set for medication review in older patients with multimorbidity. This study protocol could be used as a basis to develop core outcome sets in other fields of geriatric research.

Introduction

Patients aged 65 years and older are often exposed to polypharmacy in the context of multimorbidity.1,2 This increases the risk of adverse drug reactions and the cost of medications.3–6 Structured medication review has been shown to be an efficient way to optimize the quality of prescriptions in older patients.7,8 Medication review has recently been defined by the guidelines of the National Institute for Health and Care Excellence as “a structured, critical examination of patient’s medications with the objective of reaching an agreement with the patient about treatment, optimising the impact of medications, minimising the number of medication-related problems and reducing waste”.9 A wide range of randomized controlled trials (RCTs) have been performed to evaluate the impact of medication review on clinical, patient-reported, and economic outcomes. Several systematic reviews and meta-analyses have, therefore, been conducted to summarize the effectiveness of medication review in various clinical settings.7,8,10–20 However, the heterogeneity of outcomes reported in the RCTs has limited the quality of the conclusions. Robust meta-analyses could be performed for only a few outcomes, including hospitalization and death.15–17,21 For other outcomes, results were essentially summarized in a descriptive way because of heterogeneity in the choice and definition of the outcome measures.17–20,22

Outcome reporting bias is an under-recognized problem that affects the conclusions in a substantial proportion of systematic reviews and meta-analyses.23–25 Moreover, outcomes that are considered highly relevant to older adults are often ignored in RCTs.26,27 The development of a core outcome set (COS) can address the challenge of obtaining a consensus on which outcomes are deemed essential for all stakeholders involved in the management of a given condition, including patients. A COS is an agreed standardized collection of outcome variables that should be measured and reported in all trials for a specific condition or clinical area.28 It has the potential to reduce heterogeneity between trials, lead to research that is more likely to measure relevant outcomes, and enhance the value of evidence synthesis by reducing the risk of outcome reporting bias.29 COS has been developed in various fields of medicine, such as nonspecific low back pain, breast cancer surgery, and acute diarrhoea.30–32 Currently no COS has been developed specifically for clinical trials of medication review in older patients with multimorbidity.

The European Commission-funded OPERAM (OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly) project will perform a multicenter RCT to assess the impact of an intervention to optimize pharmacotherapy and to enhance compliance in 2,000 multimorbid older patients. To tackle the challenge of measuring relevant outcomes for these patients, the development of a COS for clinical trials of medication review in older patients with multimorbidity was planned. The results of the COS will be implemented into the RCT.

The aim of this study protocol was to describe a method that could be used to develop a COS for future use in trials of older patients with multimorbidity. The specific scope of our COS was “Medication review among patients aged 65 years and older with polypharmacy (≥5 medications) and multimorbidity (≥2 chronic conditions/diseases)”.

Preliminary search

Before starting the development of a COS on medication review in older patients with multimorbidity, we checked that there was no existing or ongoing work on this subject. A systematic search was performed in the COMET (Core Outcome Measures in Effectiveness Trials) database which collects all details of ongoing and completed COS developments.33 We found one ongoing study of COS development on prescribing in older patients living in the nursing home setting. The primary investigator of this COS study was contacted, and after a discussion, the conclusion was that the two studies would be complementary with a slight overlap. Furthermore, an ongoing project attempting to develop a COS for the assessment and management of frail older patients was retrieved.34 The principal investigator was contacted for discussion and the reply was that the COS study had not yet started.

Overview of the work performed

The development of the COS was achieved through a four-step approach represented in Figure 1. The scope was defined by the OPERAM research team in order to best fit with the OPERAM project and to be of the highest relevance for future RCTs in this research topic. We followed the guidelines published for the development of a COS35,36 and the project was registered on the COMET database (http://www.comet-initiative.org/studies/details/806?result=true). The methodological aspects addressed in this study protocol are detailed in Table S1, following recommendations made by Sinha et al.37 A steering committee was set up with the members of the OPERAM board and the local principal investigators of the participating centers.

Figure 1

Representation of the four-step approach used to develop the core outcome set.

Abbreviation: COS, core outcome set.

Step 1: systematic review

Purpose

The objective of this systematic review was to identify all outcomes used or planned to be used in previous and ongoing studies of medication review in older patients. Owing to time and resource constraints, we performed an update of a recent systematic review on medication review published in 2014 by Lehnbom et al,10 combined with a systematic search in randomized clinical trial registries and in the Cochrane Database.

Study selection

We considered RCT, quasi-RCT, and other prospective interventional studies that investigated the effect of medication review performed in patients aged 65 years or older.38 The following studies were excluded: studies published before 2000; studies predominantly including patients younger than 65 years old; retrospective studies; medication reviews for a specific disease or condition (eg, chronic heart failure) or as part of a multifaceted approach, namely a complex intervention that contained interventions in addition to medication review (eg, physiotherapy, nutritional advice, and occupational therapy); no outcome reported; or a sample size that was lower than 50 participants.

Search strategy

We used the studies identified in a recent systematic review on medication review published in 2014 by Lehnbom et al,10 the aim of which was to examine the evidence regarding the effectiveness of medication review to improve clinical outcomes in hospitals, in the community, and in aged care facilities. Lehnbom et al identified 43 studies published before March 2014 that investigated the effectiveness of medication review.10 As the purpose of the present systematic review was different in nature, we used only the result of the search strategy, that is, the list of the published studies included in the systematic review. The queries developed by Lehnbom et al were used to identify all eligible studies published during the period between March 2014 and July 2015. Two reviewers (LP and J-BB) independently assessed the title, abstracts, and full texts of studies resulting from the searches. A third reviewer (AS) was called upon to resolve any divided opinion toward consensus, if needed.

In addition, we used the search terms from the systematic review to identify RCT protocols related to medication review on the following RCT websites: the World Health Organization (WHO) international clinical trials registry platform,39 the EU Clinical Trials Register,40 and the US Clinical Trials Register.41 BB and J-BB first assessed the titles and subsequently the summaries of the RCTs protocols identified by the queries. AS was called upon to resolve any divided opinion toward consensus, if needed. It was verified if protocols had matured to full report publications. If this was the case, the publication was evaluated as mentioned above and added to the set of published studies.

Finally, the search terms from the systematic review were also used to identify relevant Cochrane reviews. JBB identified the relevant Cochrane systematic reviews and then extracted the eligible original studies in the selected Cochrane systematic reviews. The selection process was checked by AS.

Data extraction

All data extractions on outcomes and outcome measurement instruments were performed by two independent reviewers (ST and J-BB for RCT protocols; LP and J-BB for published studies). Any disagreement was resolved by discussion and consensus. AS was called upon to resolve any divided opinion toward consensus, if needed. The characteristics of the RCT protocols and the published studies were extracted by JBB and included the following: setting (hospital setting, primary care, and nursing home); number of patients intended to be recruited (RCT protocols) or actually included (published studies); and mean age (published studies only).

Outcomes used to compare the two groups under investigation (in RCTs and prospective before/after studies) or to evaluate the medication review process were extracted: name of the outcome in free text (what was measured in published studies and what was planned to be measured in protocols); primary or secondary outcome. For each outcome, the following data about measurement instrument were extracted from published studies: which instrument was used to measure the outcome (free text); was the method of measurement clearly defined? (the reviewer answered “Yes” if he/she believed that another researcher could reproduce the procedure and its measurement with the explanations provided in the “Methods” section). As the data provided in RCT protocols are often less detailed than in published studies, the data on measurement instruments were extracted when available.

Classification of outcomes into health domains and subdomains

The classification of the outcomes extracted from the included studies was achieved in several steps, starting from identifying a first list of subdomains and matching outcomes to subdomains. The list was then refined by the research team and additional experts. The objective was to obtain a consensus on subdomain terms, to avoid major overlaps between subdomains, and to aggregate subdomains into health domains. The OMERACT filter 2.0 was used to organize this classification. A total of 57 subdomains were identified and grouped into 8 health domains.

Step 2: qualitative study

In the development of a COS, it is recommended to conduct a qualitative study before the Delphi survey to identify unknown and relevant outcomes that the population of interest notice and care about.36

Participants

The qualitative study was conducted in two Belgian teaching hospitals (the Cliniques Universitaires Saint-Luc, Université catholique de Louvain [UCL], Brussels, located in an urban area and the CHU UCL Namur Godinne, located in a rural area of Belgium). The study was approved by the local research ethics committee (“Comité d’Ethique hospitalo-facultaire des Cliniques universitaires Saint-Luc”). Patients and caregivers were included after a consent form was signed. Eligible patients were patients aged 65 and older, taking at least five daily prescription medications at home and hospitalized in specialist departments of Geriatric Medicine, Internal Medicine or Orthopaedic Surgery. The exclusion criteria were: patient’s refusal to participate and inability to give informed consent (eg, dementia). Eligible caregivers were carers providing assistance to patients aged 65 years or older taking at least five daily prescription medications at home. The only exclusion criterion here was the refusal of the caregiver.

Interviews

With the agreement of the physician in charge of the patient’s care, a pharmacist researcher (AS or OD) informed the patient or the caregiver about the study and its objectives. Participants provided informed consent prior to the one-to-one interview. A semistructured approach was used. The interviews took the form of a discussion about the patient’s medications, the concept of medication review, the perception of the patient of risks and benefits of his/her medication, and what he/she would expect from a medication review. A topic guide was developed, pilot-tested, and used by both investigators (AS and OD). Interviews were recorded and then transcribed verbatim.

Audio recordings of the semistructured interviews were analyzed using NVivo10® software. The analysis was conducted by two independent researchers (a physician and a psychologist) using an interpretative approach. Key issues were identified in the transcripts and then classified under more global themes. The analysis focused on generating a comprehensive list of outcomes that are important to older patients.

Step 3: preparation of a Delphi survey adapted to older patients

The purpose of the Delphi survey is to gather opinion and to reduce the number of outcomes to a priority list for consideration in future clinical trials on medication review in older patients with multimorbidity. Some important methodological aspects have to be defined before starting the Delphi survey to improve both the quality of the survey and its reporting, as detailed in Table S1.37

List of eligible outcomes to propose

The results from the systematic review and the qualitative research were merged into a unique list of outcomes through a consultation exercise with clinical experts and researchers. The selected outcomes were then classified into outcome domains and areas according to the OMERACT (Outcomes Measures in Rheumatology) classification.35 Definition from the OMERACT Filter 2.0 is given in supplementary data (Supplementary material).

The selected outcomes were then written in four languages (Dutch, English, French, and German) in medical terms and in plain language with an explanation if needed. The plain language terms and explanations were tested for understanding with older patients and improved with their suggestions. This procedure provided a final list of outcomes understandable to all Delphi participants, especially the patient group. This list of outcomes and the results of the systematic review and the qualitative study were not provided to participants before the first round of the Delphi survey.

Key stakeholders to recruit

There is currently no consensus regarding the appropriate sample size of a Delphi survey. In recent studies using a Delphi survey methodology to develop a COS, the number of participants varied widely from 13 to 222.37 The ratio between patients and health care providers is also unclear, but the patients should be sufficiently represented. On the basis of current recommendations and on discussions among research team members, we agreed on the following list of stakeholders to recruit:

Group 1 (35%): patients and family caregivers; patients aged 80 years and over should represent a significant part of this group.

Group 2 (35%): health care professionals including general practitioners (GPs), community and hospital pharmacists, geriatricians, specialists in internal medicine, and nurses. We defined that GPs should be the majority in this group (30%–40%), given their central role in medication review.

Group 3 (30%): researchers from the field of medication review and from other areas (eg, sociologists of ageing), representatives of scientific organizations, and policy makers.

A higher attrition rate was anticipated for the patients and GPs. The characteristics and the number of each category of participants to be recruited in each center were determined in order to ensure a balanced representation of the four countries involved.

Participants from groups 1 and 2 were all recruited locally by the primary investigator of each participating center (Berne, Brussels, Cork, and Utrecht). Researchers from nonmedical fields, representatives of scientific organizations, or policy makers were also recruited locally. Researchers from the field of medication review were recruited on the basis of their publication profile. A search performed in Scopus® identified researchers with a high number of publications since 2000 on the subject of medication review and inappropriate prescribing in older patients. The details of the composition of each group of stakeholders and the number of participants are presented in Table 1.

Table 1

Repartition of all stakeholder groups and subgroups with number planned to be recruited in each center

Participant groups	Characteristics	Place of recruitment
		Locally in each center*	International
Group 1: patients and carers		16
Patients	65–80 years old and living at home	6
	>80 years old and living at home	6
Carers	Living at home	2
Patient/carer	Nursing home	2
Group 2: health care professionals		13
Primary care	GP	6
	Pharmacists	1
	Nurses (1 at home; 1 in nursing home)	2
Secondary care	Physicians	2
	Clinical pharmacists	1
	Nurses	1
Group 3: scientifics and experts		5	20
Researchers specialized in medication review			20
Researchers in related fields (nonmedical)		2
Policy makers		2
Experts from professional associations		1
Total number to be recruited		136	20

Notes:

Four centers in total: University of Bern, Switzerland; University College Cork, Ireland; Université catholique de Louvain, Belgium; University Medical Centre Utrecht, Netherlands.

Abbreviation: GP, general practitioner.

The physicians, pharmacists, nurses, or researchers directly involved in the OPERAM project were not allowed to participate to the Delphi survey because of the potential conflict of interest.

Methods for the Delphi survey

Online questionnaires and anonymity

The Delphi survey was developed by a company (WorldAPP®) that specializes in online surveys in order to ensure anonymity of the answers, to allow personal anonymized feedback at Rounds 2 and 3, and to develop a flexible online solution adapted to both very old patients and highly graduated participants. The complete online solution proposed the following: 1) an online questionnaire for health care professionals and experts that included medical terms, plain language terms, and their explanations; 2) an online questionnaire for patients and carers that did not show the medical terms in order to avoid confusion; 3) multilanguage interface; 4) a short video that explained the concept of medication review and the aim of the study (one for patients, one for other stakeholders, both in four languages); and 5) a possibility to propose new outcomes or to make comments on the outcomes proposed. The online survey can be viewed and tested (version sent for Round 1) through the following website: http://app.keysurvey.fr/f/1038815/5ded/. In this online test version, the user must choose the language, the participant category, and the age group before getting the questionnaire. During the Delphi survey, these questions were prefilled and hidden, so the participants had direct access to the questionnaire in the right form and in adequate language.

A printable version of the online questionnaire was proposed for each round to older patients who did not want to fill the questionnaire online. It was also made possible to fill the questionnaire by an interviewer who helped older patients during the Delphi survey (see the section on “Maximizing response rate”).

WorldAPP ensured anonymity of the answers. The results database was sent in an anonymous form to UCL (coordinating research center) for analysis. The participants did not know the identity or the answers of any of the other participants.

Consensus

In each round of the Delphi survey, participants were asked to score each of the outcomes listed using the GRADE process, which suggests a scale of 1–9 to rank their importance.42 Ratings from 7 to 9 indicate outcomes of critical importance, ratings from 4 to 6 indicate outcomes that are important but not critical, and ratings from 1 to 3 indicate outcomes of limited importance.

The definition of consensus during the Delphi survey was similar to the definition used in previous COS studies or in study protocols.43 We considered that a consensus would be reached to include a given outcome in the COS (“consensus-in” rule) if most of the participants (75% or more) considered the outcome as critically important (ratings of 7–9) and very few participants (15% or less) considered it of limited importance (ratings of 1–3).

Step 4: running the Delphi survey

This Delphi survey was performed in four centers from four European countries: UCL, coordinating centre (Belgium), University of Bern (Switzerland), University College Cork (Ireland), and University Medical Centre Utrecht (Netherlands). Ethical approval was sought locally in each center and obtained for the project in Belgium (“Comité d’Ethique hospitalo-facultaire des Cliniques universitaires Saint-Luc”) and Ireland (“Clinical Research Ethics Committee Of The Cork Teaching Hospitals”). Participants signed a consent form to take part in the study in these two centers. In the Netherlands (“Medical Ethics Review Committee UMC Utrecht”) and Switzerland (“Kantonale Ethikkommission Bern [KEK]”), official ethical approval was not required as the research ethics committees confirmed that the Medical Research Involving Human Subjects Act was not applicable. Completion of the questionnaire deemed consent in these two centers.

Maximizing response rate

The health care professionals and experts received an email containing a personal link to the questionnaire. The email containing the personalized link to the survey for patients and carers was sent to the local principal investigator. It was expected that this survey would be difficult to answer for older patients. All patients were proposed to answer the questionnaire with the help of a local interviewer at their own home or during a consultation. When this was not necessary, the principal investigator forwarded the email to autonomous patients and carers. The questionnaire remained online for 3 weeks and automated reminder emails were sent every 7 days after the initial invitation. The list of the participants who had answered the questionnaire during a given round was sent to the local principal investigator of each center. He/she sent a personalized reminder to nonresponders after each automated reminder and before the end of each round.

Rounds 1–3

Round 1: Participants were asked to rate each outcome as described under Consensus section. Participants could propose to add additional outcomes and to comment on their ranking. Answers to the questionnaire were analyzed globally and by the stakeholder group. Any additional outcome identified by participants was checked by two researchers and discussed with the steering committee.

Round 2: Participants who did not participate in Round 1 were not invited for Round 2. Outcomes considered as “very important” (rating 7–9) by 75% or more participants in at least one stakeholder group were further presented in Round 2 with additional outcomes suggested by the participants, if relevant. In Round 2, the participants were presented with their own results and the results of each stakeholder group in Round 1. They were asked to rerate the importance of each of the outcomes.

Round 3: Participants who did not participate in Round 2 were not invited for Round 3. Outcomes considered as “critical” (rating 7–9) by 75% or more participants in at least one stakeholder group were further presented in Round 3. All participants were asked whether the outcomes should be systematically measured in all studies on medication review in the elderly (answer YES or NO). Participants were specifically encouraged not to rate YES for all presented outcomes.

Consensus meetings

Adaptations in the methods used to obtain consensus were required. The two main changes concerned the removal of outcomes that did not meet the consensus-in rule in any group (ie, the consensus-out rule was extended) and the change of the question in Round 3 (answer YES or NO instead of Likert scale). These changes were discussed and validated with the steering committee, the OPERAM research team, and two external experts in COS development. They were also discussed with participants during consensus meetings.

A face-to-face consensus meeting with all groups of stakeholders who participated in the Delphi survey was not feasible due to the number of participants, the age of the included patients, their diverse locations across Europe, the time constraint, and the various languages involved. We, therefore, proposed to perform adapted consensus meetings after each round.

After Rounds 1 and 2, two consensus meetings were performed by teleconference: the first one with health care professionals from UCL who participated in the Delphi survey and the second one with the steering committee. Patients and carers were not invited because of the difficulty of performing a long teleconference (over 1 hour) for old and very old patients. However, local investigators from each center had participated in interviews with old and/or very old participants and could provide an indirect feedback from these participants. Researchers specialized in medication review were not invited because of time constraint and high workload of these participants. However, they frequently made very clear comments on the online questionnaire that could be included during these consensus meetings.

After Round 3, three consensus meetings were held. The results of each round of the Delphi survey were presented at the meeting, with the consensus results from Round 3 analyses used as the starting point for discussion. The goal was to comprehensively address points for discussion and to validate the final COS. The first consensus meeting was a face-to-face one with patients and carers from UCL and was conducted in a location close to the homes of several participants. The second consensus meeting was a teleconference with health care professionals from UCL. The third was a teleconference with researchers specialized in medication review and the steering committee. The interaction between the participants of the three groups was thus indirect, with the feedback and opinions of each group being transmitted to the other group by the two coordinators of this study (J-BB and AS).

Discussion

Both the European Union Geriatric Medicine Society and the American Geriatrics Society have identified that defining outcomes that are relevant for older people are important for overcoming the recognized age discrimination in clinical trials.27 COS can contribute to meet this challenge. It is, therefore, important that clinicians and researchers in geriatric care are well-equipped to develop COS. This study protocol presents the methodology to develop a COS for trials of medication review in patients aged 65 years and older with multimorbidity. We demonstrated a feasible way to involve very old patients and other stakeholders in the process of selecting the most relevant outcomes for use in future trials. We have contributed significantly to the development of the European OPERAM RCT and future RCTs of medication review by improving the quality of outcome reporting.

Strengths and limitations

The strengths of this study include: 1) methods following the guidance of initiatives like COMET and OMERACT; 2) the involvement of a large number of older and very old patients; 3) the involvement of a large expert panel of stakeholders representing various disciplines and countries; and 4) multi-institutional contributions. Some limitations should be mentioned. The applicability of the process described in this manuscript may be limited because it included very old patients and required interviews of older patients – a process that is less standardized than questionnaire completion.

A COS will have an impact only if it is consistently implemented in most trials of relevance in any given area of research. Trialists, regulators, and those who fund and publish clinical trials should engage to ensure that the COS is used. Having a strong outcome data would then help clarify if and how medication review can be effective, in which population can it be useful, and which are the important contextual factors that support positive outcomes.

Supplementary materials

Definitions from OMERACT filter 2.01

Core area: An aspect of health or a health condition that needs to be measured to appropriately assess the effects of a health intervention. Core Areas are broad concepts consisting of a number of more specific concepts called Domains.

(Sub) Domain: Component of Core Area: a concept to be measured, a further specification of an aspect of health, categorized within a Core Area.

Outcome: Any identified result in a (Sub) Domain arising from exposure to a causal factor or a health intervention (Adapted from John Last, Dictionary of Epidemiology. Toronto: Oxford Press 1995).

Measurement instrument: A tool to measure a quality or quantity of a variable, in this context a (Sub) Domain or a contextual factor. The tool can be a single question, a questionnaire, a score obtained through physical examination, a laboratory measurement, a score obtained through the observation of an image, and so on.

Core Outcome set or Core Domain set: For studies of health interventions, the minimum set of Domains and Subdomains necessary to adequately cover all Core Areas, that is, adequately measure all relevant concepts of a specific health condition within a specified setting. Describes what to measure. Currently, the COMET initiative uses the term “Core Outcome Set” for this concept.

Table S1

Methodological aspects to be defined before starting the Delphi survey to improve both the quality of the survey and its reporting, as recommended by Sinha et al2 along with the paragraph of the study protocol that addressed each given issue

Size and composition of the panel
The total number of participants invited, and the number who completed the first round	Table 1
Whether the following types of participants were involved in the study: clinicians (and whether they were eligible on the basis of treating patients with the condition of interest, or whether clinical trial involvement was an additional requirement), patients or their families, researchers, biostatisticians, representatives from the pharmaceutical industry, representatives from drug regulatory authorities, or other types of participants	Table 1 and Key stakeholders to recruit section
The proportion of each type of participant described above	Table 1
How participants were identified/sampled	Key stakeholders to recruit section
Methodology of the Delphi process
Administration of questionnaires: postal, email, internet, in person (eg, at a clinic), or at a meeting	Maximizing response rate section
Information about outcomes, known to the facilitators before the study, which was provided to participants before the first round: for example, if the Delphi process followed a review of outcomes measured in clinical trials, were the results of the review shared with participants? Alternatively, if some work had been conducted prior to the Delphi survey (eg, workshop meeting, or focus groups among patients), were the results presented to the participants?	List of eligible outcomes to propose section
How outcomes were considered in the first questionnaire: were participants asked an open question, that is, no outcomes were initially listed, or were they asked to comment on a prespecified list? If the latter, was the source of the list identified? Where possible, the questions asked to participants should be described in the methods, or made available to the reader, as supplementary information	List of eligible outcomes to propose sectionOnline questionnaire and anonymity sectionRounds 1–3 section
What was asked in subsequent rounds: where possible, the questions asked to participants should be described in the methods, or made available to the reader, as supplementary information	Rounds 1–3 section
Feedback to participants after each round: if the results were not fed back, but only certain outcomes were carried forward to the next round (eg, only those suggested by at least 10% were carried forward), this should be clearly described	Rounds 1–3 section
Level of anonymity should be described: In order to be “fully anonymised”, participants should not know the identities of the other individuals in the group, nor should they know the specific answers that any other individual gave. In studies that are “quasi-anonymised”, the participants should know the identities of some or all of the other individuals, but should not know how they individually responded to any of the questions in any round. In studies that are not anonymized, participants must know the identity of some or all of the other individuals, and also know how some or all of them responded to any of the questions in any round	Online questionnaires and anonymity section
If a predetermined definition of consensus was used, this should be clearly described in the methods section of the study report	Consensus section
Were nonresponders invited to subsequent rounds, or were they excluded from the rest of the study? Were additional people invited as the Delphi survey progressed?	Rounds 1–3 section