Comparing clinical quality indicators for asthma management in children with outcome measures used in randomised controlled trials: a protocol.

INTRODUCTION: Clinical quality indicators are necessary to monitor the performance of healthcare services. The development of indicators should, wherever possible, be based on research evidence to minimise the risk of bias which may be introduced during their development, because of logistic, ethical or financial constraints alone. The development of automated methods to identify the evidence base for candidate indicators should improve the process of indicator development. The objective of this study is to explore the relationship between clinical quality indicators for asthma management in children with outcome and process measurements extracted from randomised controlled clinical trial reports. METHODS AND ANALYSIS: National-level indicators for asthma management in children will be extracted from the National Quality Measures Clearinghouse (NQMC) database and the National Institute for Health and Care Excellence (NICE) quality standards. Outcome measures will be extracted from published English language randomised controlled trial (RCT) reports for asthma management in children aged below 12 years. The two sets of measures will be compared to assess any overlap. The study will provide insights into the relationship between clinical quality indicators and measurements in RCTs. This study will also yield a list of measurements used in RCTs for asthma management in children, and will find RCT evidence for indicators used in practice. ETHICS AND DISSEMINATION: Ethical approval is not necessary because this study will not include patient data. Findings will be disseminated through peer-reviewed publications. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Develop a novel method for comparing indicators with research evidence in clinical trial.

Provide a systematic and objective way to identify candidate indicators, identify research associated with current indicators, and identify potential inconsistencies between research evidence and clinical practice.

Generate a list of measurements used in randomised controlled trials of asthma management in children.

Introduction

The drive to improve the quality and safety of healthcare has resulted in the proliferation of clinical quality indicators. Indicators may be used to assess the quality of care, or to identify and prioritise areas for improvement.1 2 They describe “as much about a system as possible in as few points as possible”.3

A good indicator should be important, relevant, valid, reliable, meaningful and understandable.3 4 In addition, indicators should be easy to collect,5 and the costs of doing so should not outweigh the potential benefits.6 The development of indicators can be complex and resource intensive. Indicators can be developed using deductive approaches (moving from target concepts to supporting data), inductive approaches (using data to identify concepts) or both.7 Deductive approaches are more common.7 The steps to developing indicators deductively are: (1) selecting a topic, (2) forming a review team, (3) researching the evidence and practice behind candidate indicators, (4) selecting indicators, (5) designing their specifications, (6) pilot testing their collection and (7) implementing them.2 8

In practice, clinical indicators can be vulnerable to bias and lack rigour8 because of (sometimes necessary) logistic, ethical or financial constraints during their development. To minimise these risks, the development of indicators should, wherever possible, be based on research evidence.2 9 Different grades of evidence are available, and randomised controlled trials (RCT) are generally regarded as one of the highest levels of evidence.10 11

Asthma is a chronic respiratory disease that affects about 300 million people globally, with an estimated 250 000 deaths annually.12 It is the most common chronic disease among children. There is currently no cure for asthma, but it can generally be managed and controlled with appropriate care (care regarded as being in line with clinical guidelines13–15). Many quality indicators have been developed for asthma;16–20 most are based on clinical practice guidelines (CPGs). For example, the National Institute for Health and Care Excellence (NICE) quality standards for asthma are based on the British Thoracic Society/Scottish Intercollegiate Guidelines Network (BTS/SIGN) clinical guideline 101: British guidelines on the management of asthma;13 and the asthma clinical performance measurement tool promoted by the Physician Consortium for Performance Improvement is based on the National Asthma Education and Prevention Program Expert Panel Report 2.21

To better assess the level of evidence behind currently recommended indicators, this protocol describes a study of the relationship between outcome measurements used in RCTs and national level clinical quality indicators in the UK and the USA, using asthma management in children as a case study. The study will compare outcome terms extracted from RCT reports with clinical quality indicators by assigning concepts from the Unified Medical Language System (UMLS) using MetaMap22 to both sets of terms. This allows terms from different sources to be standardised for easy aggregation and comparison. MetaMap has been used for many tasks, for example, information retrieval23 and text mining,24 25 and has been shown to perform well in mapping biomedical concepts to text compared with human labelling.26

Methods and analysis

The study will examine asthma management in children aged less than 12 years. There are four basic steps: (1) identification and extraction of indicators, (2) identification and extraction of outcome measurements from RCTs, (3) mapping both sets of metrics to underlying concepts as defined in the UMLS using MetaMap and (4) matching and comparing clinical quality indicators and RCT measurements, with an evaluation of the basis for any differences.

Identification and extraction of indicators

National-level clinical quality indicators for asthma management in children from the USA and the UK will be collected. The National Quality Measures Clearinghouse (NQMC) is a database on healthcare quality measures sponsored by the US Department of Health and Human Services—Agency for Healthcare Research and Quality (AHRQ). The NQMC will be searched using the term ‘asthma’ within the respiratory tract diseases condition (50 measures in asthma of the 243). The NICE quality standard for asthma (11 asthma quality standards) will also be included. NICE plays a major role in commissioning and producing guidelines and developing quality standards for Britain's National Health Service (NHS).

The following inclusion criteria will be used to select indicators:

Any process or outcome indicator,

Any indicator of pharmacological or non-pharmacological management,

Applicable to children aged below 12 years,

Developed for national-level assessment.

Candidate indicators identified in the search will be independently assessed against inclusion criteria by two appraisers. Disagreements will be resolved by consensus.

The included indicators will then be normalised as follows:

Remove: Unit of analysis (eg, number of people/patients),

Remove: Time points/frames (eg, more than 24 h, within 2 days) for a process to occur,

Remove: The term “asthma” if it describes the characteristics of patients,

Extract: Noun terms describing the unit of analysis (examples in table 1).

Table 1

Examples of classification of extracted terms

Classification	Examples
Type of recommended care	Hospitalisation, asthma action plan
Medication	Steroids, inhaled corticosteroids, leukotriene inhibitors
Event	Exacerbation of asthma

Table 2 shows examples of possible terms that may be extracted from the indicator sources.

Table 2

Examples of terms extracted from clinical indicators

	Extracted terms	Classification
Paediatric inpatients with asthma who received relievers during hospitalisation	Relievers; hospitalisation	Medication; type of recommended care
Paediatric inpatient discharge, age 2 years through 17 years, with an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) Principal Diagnosis Code of asthma	Inpatient discharges	Type of recommended care
Number of patients with asthma who have an asthma discharge plan	Asthma discharge plan	Type of recommended care
Number of patients with asthma who were seen in an emergency department or hospitalised for asthma treatment	Emergency department; hospitalised	Type of recommended care; type of recommended care
The number of people in the denominator followed up by their own GP practice within 2 working days of treatment	GP practice	Type of recommended care
The number of people who received treatment in hospital or through out-of-hours services for an acute exacerbation of asthma	Hospital; out-of-hours services; acute exacerbation of asthma	Type of recommended care; type of recommended care; event

Bold typeface indicates ‘extracted terms’.

GP, general practitioner.

Identification and extraction of outcomes from RCTs

Types of studies

Only RCTs will be included. Secondary/post hoc analysis/protocol of RCTs will be excluded.

Types of participants

Children with asthma aged below 12 years will be included.

Types of interventions

Studies that involved any type of medical or non-medical management of asthma will be included.

Search strategy

We will search PubMed for RCTs on asthma management for children aged below 12 years. Studies will be restricted to English language articles published in the past 10 years (2005–2014).

The following search terms will be used to retrieve PubMed articles:

Asthma

(randomised controlled trial[Publication Type] OR (randomised[Title/Abstract] AND controlled[Title/Abstract] AND trial[Title/Abstract]))

#1 AND #2

#3 AND “English”[Language]

The second search term above is the specific/narrow therapy filter for clinical queries. PubMed filters of Species (‘Humans’) and Ages (‘Infant: birth-23 months’, ‘Preschool Child: 2–5 years’, and ‘Child: 6–12 years’) will be applied.

Study selection

The titles and abstracts of the resulting references will be screened independently by two reviewers against the following inclusion criteria:

Articles describing RCT, excluding secondary/post hoc analysis/protocol of RCTs;

The participants of the trial include children with asthma aged below 12 years even if the trial also includes other participants;

The trial's focus is on medical or non-medical management of asthma.

Any control arm is acceptable as well as any outcome measure. Disagreements will be resolved by consensus. Inter-rater agreement will be measured periodically on screening decisions and only one screener may be used for the remaining abstracts if agreement is high (ie, Cohen's κ>0.8).27

The full text of the potentially eligible studies will be retrieved and reassessed if the abstract does not contain enough information to make a decision.

Outcome measures extraction from clinical trials and ambiguities resolution

RCTs generally report using the PICO criteria:28 Population, Intervention, Comparison, and Outcome. Two independent annotators will be asked to independently extract the Population (ie, number of participants in the study) and Outcome (outcome names) from the full text of every included RCT. Outcome terms will be any phrase that describes a quantifiable measurement.

Outcome names identified by this process will be normalised using the following rules:

Outcome deduplication: Similar outcome terms mentioned multiple times in an RCT will only be considered as a single occurrence.

Generalisation relationships: The most specific term mentioned in an RCT will be preferred. For example, spirometry and peak flow measurement both describe lung function tests, but the latter is more specific.

Ambiguities: Any ambiguities will be resolved by consultation with a paediatric respiratory physician, paediatric nurse or other relevant clinician.

Inter-rater agreement will be measured periodically on extraction decisions and only one extractor may be used for the remaining extractions if agreement is high (ie, Cohen's κ>0.8).

UMLS concept mapping using MetaMap

The list of terms extracted from indicator sources and the list of outcome measures taken from RCTs will next both be mapped into standardised terms (UMLS concepts using MetaMap) to facilitate aggregation and comparison. The UMLS Metathesaurus, from the National Library of Medicine, comprises over 1 million biomedical concepts from over 100 source vocabularies. MetaMap22 is a tool for mapping biomedical terms in free text to the UMLS Metathesaurus. MetaMap will be provided with extracted terms from clinical indicators and RCTs and will return a list of ranked mapping options. We will use all concepts identified by MetaMap in this study. If no UMLS concept is found, we will retain the extracted term. Table 3 shows examples of phrases mapped to UMLS terms. As shown in table 3, ‘exacerbation of asthma’ will be mapped to ‘asthma exacerbation’ and ‘PEF’ will be mapped to ‘peak expiratory flow’ even if they are expressed differently.

Table 3

Examples of mapping to UMLS terms using MetaMap

Phrases	UMLS mapping by MetaMap
Exacerbation of asthma	C0349790:Exacerbation of asthma [Finding]C0004096:–Asthma (Asthma) [Disease or Syndrome]C2984299:Asthma (Asthma Pathway) [Functional Concept]
Asthma exacerbation	C0349790:asthma exacerbation (Exacerbation of asthma) [Finding]C0004096:–Asthma (Asthma) [Disease or Syndrome]C2984299:Asthma (Asthma Pathway) [Functional Concept]
PEF	C0030771:PEF (Pefloxacin) [Antibiotic,Organic Chemical]C1518922:PEF (Peak Expiratory Flow) [Laboratory Procedure]C1542834:PEF (Peak expiratory flow rate) [Finding]
Peak expiratory flow	C1518922:Peak Expiratory Flow [Laboratory Procedure]C0857465:peak flow [Finding]C0231800:Expiratory (Expiration, function) [Organ or Tissue Function]C0444505:Peak (Peak level) [Quantitative Concept]C0806140:Flow (flow) [Natural Phenomenon or Process]

UMLS, Unified Medical Language System.

Extracted term ranking

UMLS concepts will be ranked according to the total number of RCT participants that they were applied to. In most cases, this is the number of patients randomised or for whom there was an intention to treat.

Mapping and comparing clinical quality indicators and RCT measurements

The RCT outcomes and practice indicators will then be compared using UMLS concepts by two annotators. Terms in both lists will be mapped to each other if one or more UMLS concepts associated with them match. The mappings will be agreed by consensus and inter-rater reliability will be measured. Advice will be sought from appropriate clinicians if consensus cannot be reached.

The congruence between the mapped outcomes from RCTs and those from clinical indicators will be measured. Measured outcomes in RCT that are not used as clinical indicators may imply wastage either because metrics with research evidence behind them are not being used as indicators, or because clinical trials have measured outcomes that are not important in the assessment of routine care. Indicators that cannot be mapped to outcomes in RCTs will also be identified as they may have little or no research evidence to support their use.

Discussion

This protocol describes a novel method for comparing indicators with research evidence in clinical trials, and will help to determine if indicators associated with the management of asthma in children are supported by research evidence. This study will also generate a list of measurements used in RCTs of asthma management in children.

A limitation is that it will not examine other levels of evidence beyond level 1 (RCT) evidence. Some indicators may not be appropriate for use in RCTs and may appear in different study types, such as economic evaluations. Further, the protocol does not investigate if the RCTs support the use of indicators, and only looks for evidence of their use as a proxy indicator of utility. Some RCT outcomes may be important in early phases of the evaluation of an intervention, but will have limited application in routine practice. This study is also limited by searching only PubMed and assessing only English language RCTs.

However, it is important because it provides a systematic and objective way of finding candidate level 1 evidence for clinical indicators. It will assist in identifying candidate indicators, identifying research associated with current indicators, and identifying potential inconsistencies between research evidence and clinical practice.