Which outcomes should be used in future bronchiolitis trials? Developing a bronchiolitis core outcome set using a systematic review, Delphi survey and a consensus workshop.

OBJECTIVES: The objective of this study was to develop a core outcome set (COS) for use in future clinical trials in bronchiolitis. We wanted to find out which outcomes are important to healthcare professionals (HCPs) and to parents and which outcomes should be prioritised for use in future clinical trials. DESIGN AND
SETTING: The study used a systematic review, workshops and interviews, a Delphi survey and a final consensus workshop.
RESULTS: Thirteen parents and 45 HCPs took part in 5 workshops; 15 other parents were also separately interviewed. Fifty-six items were identified from the systematic review, workshops and interviews. Rounds one and two of the Delphi survey involved 299 and 194 participants, respectively. Sixteen outcomes met the criteria for inclusion within the COS. The consensus meeting was attended by 10 participants, with representation from all three stakeholder groups. Nine outcomes were added, totalling 25 outcomes to be included in the COS.
CONCLUSION: We have developed the first parent and HCP consensus on a COS for bronchiolitis in a hospital setting. The use of this COS will ensure outcomes in future bronchiolitis trials are important and relevant, and will enable the trial results to be compared and combined. TRIAL REGISTRATION NUMBER: ISRCTN75766048. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

With stakeholder input from medically qualified staff, nurses and other clinical staff, and parents of children with bronchiolitis, we have developed a preliminary core outcome set (COS) for paediatric bronchiolitis trials.

A mixed-method approach was used to determine which outcomes would be included in the COS.

This COS can be considered when designing paediatric bronchiolitis trials as a minimum for outcomes to be collected and reported.

While we followed guidance within the COMET (Core Outcome Measures in Effectiveness Trials) handbook in regards to Delphi processes and face-to-face consensus meeting format, one could argue that parental input during the consensus meeting had a relatively large effect on which items were included in the final COS.

Introduction

Background and objectives

Bronchiolitis, an acute viral lower respiratory tract infection which predominantly affects infants, is a major cause of morbidity and mortality worldwide.1 Typical clinical features include a coryzal prodrome lasting approximately 3 days, persistent cough, increased respiratory rate, chest recession and wheeze or crackles on auscultation.2 While most children with bronchiolitis have mild symptoms and can be managed at home,3 in the UK and USA approximately 3% are hospitalised, most commonly between 3–6 months of age.1 4

Although multiple therapeutic interventions for bronchiolitis have been assessed in clinical trials, treatment remains supportive.2 Oxygen therapy and the use of oximetry are the only interventions that have significantly impacted survival over the last 40 years, contributing to a reduction in mortality from approximately 20% in some studies to <1%.5 6 Interventional clinical trials in bronchiolitis have increased in number over the past decade, investigating novel ways of administering oxygen nasally and non-invasively with varying levels of positive airway pressure or flow,7–9 and novel antiviral medications against respiratory syncytial virus, the principal viral cause of bronchiolitis.10 11 More interventions are in the therapeutic pipeline.12

Randomised controlled trials (RCTs) are regarded as the gold standard for evaluating healthcare interventions.13 Although they produce high quality evidence that inform clinical care through practice guidelines, their clinical impact is often diminished by variations in outcome measurement and reporting. Systematic reviews in many different branches of medicine have consistently demonstrated the large number and heterogeneity of outcome reporting in trials and other research studies.14–16 This makes clinically relevant comparisons between trials and pooling of results in meta-analyses difficult. Furthermore, multiplicity of outcome measurement can lead to the selective reporting of significant findings, referred to as outcome reporting bias.17

A proposed solution is to develop and use a ‘Core Outcome Set’ (COS).18 This consists of a minimum set of outcomes that key stakeholders agree are important, the measurement of which should be considered for all trials in a particular field.18 This has the potential to improve the efficiency with which research can answer clinical questions. The benefits of COS have been embraced internationally by funding bodies,18 regulatory bodies19 20 and journal editors,21 all of which recommend their use where available. As a result, the development of COS is increasingly common. The COMET (Core Outcome Measures in Effectiveness Trials) initiative has recorded approximately 700 published or ongoing studies into COS in many branches of medicine. Until now there has been no COS for trials of children with bronchiolitis although the need for one has previously been identified.22

The aim of this study was to obtain consensus from key stakeholders on which outcomes should be included in a COS for use in future bronchiolitis trials and other studies. This study formed part of a larger research project (Non-Invasive Ventilation for the Management of Children with Bronchiolitis: a feasibility study (NOVEMBR)).23

Methods

The COS was developed in three phases using methods recommended by COMET and COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments).18 The aim of the first phase was to generate a list of initial outcomes for consideration within the COS; this comprised a systematic review, stakeholder workshops and interviews. The second phase was a two-round Delphi survey, and the third phase, a face-to-face consensus meeting where results from the Delphi were presented.

COS-STAR (Core Outcome Set-STAndards for Reporting) guidelines for reporting were followed.24

Patient and public involvement

Parents were involved in writing the original study protocol, information sheets and Delphi questionnaires. Discussion regarding non-invasive ventilation (NIV) modalities was part of a separate exercise designing a protocol of NIV for infants with bronchiolitis. A parent was also a member of the trial management group.

Phase I: systematic review of the literature and stakeholder perspectives

Lists of outcomes were generated separately by both systematic review of the literature and stakeholder workshops/interviews.

The systematic review identified outcomes to assess efficacy and safety of interventions (pharmacological and non-pharmacological) used to treat children with acute bronchiolitis in published clinical trials since 2000 (search strategy provided in online supplemental appendix A). Details of this systematic review are not contained within this publication and will be published at a later date.

Parents or guardians were eligible to participate in a workshop or interview if their child had been admitted to hospital with bronchiolitis within the previous year. Eligible parents were invited to participate by research nurses at seven study sites (Alder Hey Children’s NHS Foundation Trust (Liverpool); Wirral University Teaching NHS Foundation Trust; Countess of Chester NHS Foundation Trust; Derby Teaching Hospital NHS Foundation Trust; Darlington Memorial Hospital; Cambridge University Hospitals NHS Trust and Royal Alexandra Hospital, Brighton. The lead centre will be Alder Hey Children’s NHS Foundation Trust (Liverpool)), through advertisements on social media. Healthcare professionals (HCPs) were eligible to participate in a workshop or interview if they had at least 6 months experience in managing children with bronchiolitis. Study sites emailed invitations to eligible HCPs to register interest in a workshop. Parent and HCP selection aimed to ensure variance (eg, child’s age 0–24 months, gender and severity of illness and HCP role and hospital geographical location).

For both HCP and parent workshops, interview topic guides exploring aspects of trial design were developed based on previous research.25–27 To identify prioritised outcomes, participants were first asked to reflect on their personal experiences including what they would consider as a sign that a child was getting better. To inform discussion, they were then shown a list of 34 outcomes (online supplemental appendix B) extracted from Cochrane Systematic Reviews and National Institute for Health and Care Excellence (NICE) guidance on bronchiolitis,28 asked to consider which were important to them, and identify outcomes not included on the list. This process was adapted for parent interviews (eg, list of outcomes emailed to interview participants).

A professional transcription company (Voicescript, Bristol, UK) transcribed verbatim digital audio recordings. Transcripts were anonymised and checked for accuracy. NVivo V.10 software (QSR International, Melbourne, Australia) was used to assist in the organisation and coding of outcomes identified both through responses to direct questioning, and referred to by participants during interview and workshop discussions.

A conceptual outcomes framework was developed based on previous exploratory work in which domains and subdomains were defined and outcomes categorised accordingly.29 30 Similar outcomes identified by systematic literature review and workshops/interviews were collapsed together—following discussions between study management group members.

Phase II: Delphi process and outcome scoring

Participants

Key stakeholders who participated in the Delphi survey included parents/legal representatives of children hospitalised with bronchiolitis defined as per UK NICE Bronchiolitis Guidelines (2015).28 These were identified through Phase I workshops and interviews, and by the seven study sites by research nurses on the wards and asked whether they would like to participate, they were then contacted at a later date. Parents of children who had died during their hospital admission were not approached. Non-English speakers were not eligible to participate.

Also included in the Delphi survey were HCPs (and nurses/other clinical staff) with experience of caring for children with bronchiolitis. These were identified via emails sent to professional organisations, and distributed via global email address lists or associated social media sites. HCPs who had previously completed the NOVEMBR National Survey of Current Practice were also approached if they had expressed an interest in participating.31 Participants were also invited to pass on details of the study to any of their own contacts who met eligibility requirements. The Delphi process was conducted and managed by DelphiManager software (). Access to the Delphi survey was via a hyperlink distributed by email. In round one, participants confirm their eligibility, and assigned a score (using a Likert scale of 1–9) to each of 56 outcomes, listed alphabetically, based on their opinion of its importance in the management of children with bronchiolitis. The Grading of Recommendations, Assessment, Development and Evaluations (GRADE) guidelines scale was used for scoring: scores of 1–3 indicated that the outcome was ‘of low importance’, 4–6 indicated ‘equivocal—important but not critical for decision making’ and 7–9 indicated ‘critical for decision-making’.32 Participants were also given the option to respond, ‘do not know’. Lastly, there was the opportunity to suggest any additional outcomes along with a score for importance. All additional outcomes were reviewed by the Study Management Group (SMG) (which contained clinicians, research nurses, trials unit representatives and parents) and where applicable carried forward to round two.

Those who completed round one were invited to participate in round two. In round two participants were shown the distribution of scores given by each stakeholder group and then asked to review and re-score each outcome. At the end of the first-round participants were shown their own scores for and asked to re-score based on the same question as the first round.

Data were presented as counts and percentages for categorical data and mean and SD for continuous data.

Consensus for inclusion of an outcome in the COS was achieved if 70% or more participants gave the outcome a score of 7–9 and less than 15% gave a score of 1–3 in each stakeholder group for round two of the survey (table 1) in all three stakeholder groups. The rationale for these levels of agreement was based guidance in the published literature.30 33 Consensus for exclusion of an outcome followed if 50% or fewer participants scored 7–9 in all of the three stakeholder groups. Any other outcome was classified as ‘no consensus’. Reminder emails were sent to participants to encourage completion. Participants who completed both Delphi survey rounds received a certificate of completion and were entered into a prize draw to win an iPad. Completion of the surveys was deemed as consent to participate.

Table 1

Definition of consensus

Consensusclassification	Description	Definition
Consensus in	Consensus that outcome should be included in the core outcome set	70% or more participants scoring as 7–9 AND<15% participants scoring as 1–3 in each group
Consensus out	Consensus that outcome should not be included in the core outcomes set	≤50% of participants scoring as 7–9 in each group
No consensus	Uncertainty about importance of outcome	Anything else

Phase III: consensus meeting

The results from each round of the Delphi survey were presented at a face-to-face consensus meeting involving a representative group of stakeholders; members of this group were those who participated in the survey or workshops and expressed an interest in attending the consensus meeting. Participants were not specifically invited based on their views in the survey and were not required to have completed the Delphi survey in order to participate. An independent medical professional, with expertise in COS development and not a member of the SMG, chaired the meeting. Meeting attendees were asked to review the full list of outcomes and were given the opportunity to discuss whether they agreed with the outcome consensus classifications from the Delphi process in Phase II. They were then asked to discuss the outcomes classified as ‘no consensus’ from the Delphi process. The Chair ensured that all participants had equal opportunity to give their views on each of the outcomes prior to voting taking place. Voting was carried out with the same 9-point Likert scale used in the Delphi (described in Phase II) and conducted anonymously using TurningPoint software and handsets (Turning Technologies LLC, Youngstown, USA). At the end of the meeting the outcomes that met the criteria for ‘consensus in’ were presented to the stakeholders.

Registration

The study protocol was registered retrospectively with the COMET initiative and is available online,23 and on the ISRCTN Registry (18 December 2017).

Results

Phase I: literature review and stakeholder perspectives

The systematic review identified 154 studies for inclusion (online supplemental figure 1). Within these studies, 923 individual outcome measures were identified. These outcomes were grouped and tabulated under appropriate outcome domains using a predefined conceptual framework as described previously.29 30 This work is being written up as a separate manuscript.

Between April 2016 and March 2017, 13 parents and 45 practitioners took part in five workshops (three for HCPs and two for parents); 15 other parents were also interviewed by telephone. No outcomes were identified during these workshops and interviews that had also not been identified as part of the systematic review.

All outcomes identified from the systematic review, workshops and interviews were reviewed by for similarity. Outcomes considered sufficiently similar with regards to what they measured were collapsed and merged together. Furthermore, this process was discussed and reviewed with the SMG to agree the final list of 56 outcomes to be included into the Delphi survey (online supplemental table 1).

Phase II: Delphi process

Round one of the survey was conducted between 19 February 2018 and 23 March 2018. Round two was conducted between 29 March 2018 and 13 April 2018.

Online supplemental table 2 shows the breakdown of stakeholder group and their participation in both rounds. In total, 299 participants (from the UK) registered on the online system and 286 (96%) scored at least one outcome in round one. Sixty-eight per cent (194/286) were medically qualified, 28% (81/286) were nurses and other clinical staff, and the remaining 4% (11/286) were parents. Participants were invited to round two if they had scored at least one outcome in round one. Sixty-eight per cent (194/286) participated in round two by scoring at least one outcome. Mean round one scores for those who participated in both rounds and those who participated in round one only were similar (online supplemental table 3).

Following round one, one item was added to the list of outcomes for consideration in round two: length of time spent on oxygen.

Online supplemental table 4 shows the results from round two. Sixteen outcomes met the criteria for ‘consensus in’ for inclusion in the COS, 8 outcomes met the criteria for ‘consensus out’ and 32 outcomes were classified as ‘no consensus’.

The final set of 16 outcomes that were included from the Delphi process are included in table 2.

Table 2

Final outcomes to be included in a core outcome set from the Delphi process

Domain (subdomain)	Outcome
Physiological and clinical (general symptoms)	Appearance
	Level of consciousness
	Non-respiratory physiological parameters/vital signs
	Worsening illness
Physiological and clinical (feeding; nutrition and hydration)	Feeding
	Need for feeding tube
	Inhalation (breathing in) of milk; fluids or solids
Physiological and clinical (respiratory distress)	Apnoea
	Oxygen saturation
	Cyanosis
	Effort of breathing
	Paediatric Early Warning score
Physiological and clinical (respiratory interventions and support)	Need for respiratory support
Resource use (hospital related short term)	Critical care admission
Death	Death
Adverse events	Serious adverse events

Phase III: consensus meeting

The consensus meeting took place on the 14 June 2018 and was attended by 10 participants: 4 were medically qualified staff, 3 were parents and the remaining 3 were nurses and other clinical staff.

When round two classifications of ‘consensus in’ outcomes were reviewed, the inclusion of Paediatric Early Warning (PEW) score was discussed as not every hospital uses it and even in those that do, the composite measures can vary. However, participants agreed it should remain in the COS as a standardised, widely used objective PEW score has the potential to be a quick indicator of changing health and healthcare needs. No further comments were provided on ‘consensus in’ outcomes. Discussion around ‘consensus out’ outcomes was centred on the economic cost outcome; one participant believed it should be included since it impacts on whether the intervention is adopted by some healthcare funders. After some discussion, it was agreed that other core outcomes can be used for a health economic evaluation, without the need for a separate outcome.

Following discussion and re-vote, nine outcomes were added to the ‘consensus in’ list (table 3). The final COS contains 25 outcomes across eight domains (tables 2 and 3). In the Delphi, need for fluids given through a drip (intravenously) was scored 7–9 by 80%, 85% and 67% of medically qualified staff, nurses and parents, respectively. However, in the consensus meeting only 20% of meeting attendants scored it 7–9; discussions at the time centred on the relative subjectivity of the assessment of need for fluids through a drip and that it was not a critical outcome. Likewise, bronchiolitis severity score and additional chest infections/pneumonia were close to being included as ‘consensus in’ in the Delphi but were voted ‘consensus out’ during the consensus meeting. There were differing views in the room on whether or not a disease specific score should be included in the COS but after discussion, only 60% voted 7–9. Although all agreed additional chest infection/pneumonia was important, it was decided that it was not critical. A full report from the consensus meeting detailing discussion behind each outcome has been included in online supplemental appendix C. At the close of the meeting the final COS was agreed by all the participants.

Table 3

Outcomes that were added to the core outcome set during the consensus meeting

Domain (subdomain)	Outcome
Physiological and clinical (general symptoms)	Pain and discomfort
	Parent report of symptoms and/or resolution of illness
Physiological and clinical (feeding; nutrition and hydration)	Reduced urine output
Physiological and clinical (respiratory interventions and support)	Length of time spent on oxygen
Life impact (health-related quality of life)	Quality of life
Resource use (hospital related short term)	Hospital length of stay
	Time until ready for discharge from hospital
Adverse events	Adverse events
	Long-term effects of illness or treatment interventions

Protocol changes

The definition of ‘Consensus Out’ was changed from ‘70% or more participants scoring 1–3 and <15% of participants scoring 7–9 in each group’ to ‘≤50% of participants scoring 7–9 in each group’. This change was made for practical reasons only, in order to refine and reduce the list of outcomes to be discussed at the consensus meeting. Meeting attendees were also given the opportunity to comment on and discuss any of the ‘consensus out’ outcomes.

Discussion

We have developed a COS of 25 outcomes of importance to medically qualified staff, nurses and other clinical staff, and parents of children with bronchiolitis, for use in RCTs of interventions for children with a clinical diagnosis of bronchiolitis in a hospital setting.

On the COMET database, 10% of the approximate 700 listed published and ongoing COS are in child health with most being for chronic paediatric diseases. Those for acute conditions include sepsis, infantile colic, appendicitis, head injury, acute diarrhoea and asthma. Two other studies on the COMET database mention bronchiolitis, one a systematic review of outcome measures and measurement instruments used in bronchiolitis RCTs,34 and the other a European Respiratory Society guideline on ‘Endpoints in respiratory diseases’ based solely on expert views of HCPs and published in 2010.35 This is the only COS developed for use in for bronchiolitis, one of the most common acute causes of childhood admission to hospital, developed using recommended consensus based methods.18

One of the strengths of this study was the involvement of both parent and HCPs in each phase and the necessity for agreement by each stakeholder group before consensus could be reached; this ensured all views were incorporated throughout. For example, at the final consensus meeting, discussions were carefully ‘managed’ by an independent chair who ensured all participants could voice their opinions and have their opinions heard. The use of ‘anonymous’ TurningPoint software also allowed peer/social pressures to be minimised when it came to voting. This enabled the views of all participants to be heard, and particularly those of parents who might otherwise have been reluctant to contribute when surrounded by experienced HCPs. As a result, parental ‘championing’ of four ‘no consensus’ items led directly to their inclusion in the COS (pain and discomfort, parent report of symptoms and/or resolution of illness, quality of life and reduced urine output). Three of these are relatively subjective measures. For changes in urine output, parents argued that this provided a tangible way for them to know whether a treatment or intervention was working and their child was ‘on the mend’.

There were a number of limitations to this study. Since the Delphi survey was disseminated via mailing lists, it was not possible to ascertain the total number of individuals approached and therefore the proportion who took part. Also, although reminders for completion were circulated, not all participants who took part in round one also took part in round two. However, comparing scores for round one for those who completed both rounds with scores for those who completed round one only, did not highlight any attrition bias. Indeed, the overall number of participants who took part in both rounds of this study (194) was above the median of 111 from a recent review of 31 published and ongoing studies that used a Delphi to develop a COS.35 In our study, having fewer outcomes to assess may have increased the response rate from round one to round two.36

We would have liked to include a larger number of parents in the Delphi survey. Parents were approached during their child’s admission to hospital with bronchiolitis and many agreed to take part in the study and gave their contact details. However, only 11 of these subsequently completed the two surveys. We speculate that by the time parents of otherwise healthy infants and young children were contacted some months after the acute bronchiolitis episode, time pressures perhaps meant that their priorities lay elsewhere. Of note, 30% of attendees at the consensus meeting were parents, which ensured their voice was heard at this crucial part of the process and that the consensus meeting comprised a representative group of stakeholders.

Responders were primarily UK based which raises the question of whether the COS would be applicable to trials overseas. An extension of this work would be to repeat the process including international participants, to determine whether the choice of outcomes changes depending on location. It is likely that this would indeed be so, given that respiratory support or nasogastric feeding are not universally available in all settings. Future work on the COS could also include pharmaceutical representatives and regulators.

Burden of measurement should also be taken into consideration when developing a COS. Although the COS we have developed contains 25 outcomes, it can be seen that several of the outcomes are related, for example, adverse events and serious adverse events would both be captured when measuring safety. Similarly, feeding and need for feeding tube, hospital length of stay and time until ready for discharge from hospital are related and would require minimal time to capture these data.

We believe the COS should be used by future trialists as the minimum set of outcomes that should be collected in a bronchiolitis trial. Importantly, this COS provides an essential step towards increasing and improving much-needed evidence synthesis in this disease area.22 37 There was some discussion during the consensus meeting on how potential outcomes would be measured, for example appearance, however the chair reminded participants that the aim of the meeting was to agree on ‘what’ should be measured rather than ‘how’ to be measured. Now that we have established which outcomes should be reported, there is a need for future research to ascertain the best methods to operationalise these outcomes, particularly some of the more subjective ones such as pain and discomfort, parent report of symptoms and inhalation of milk/feeds.

Conclusions

This was an important study identifying the outcomes of importance to key stakeholder groups which will provide guidance to future trialists in paediatric bronchiolitis. This is particularly pertinent with anti-viral treatments in the therapeutic pipeline, and the need to clearly define how some existing interventions (such as high flow nasal cannula oxygen) are best used in clinical practice. We recommend researchers designing bronchiolitis trials consider this preliminary COS as a minimum for outcomes collected and reported in trials. Future work is now needed to test the validity and generalisability of the COS and ongoing work should include routine updating of the COS.