CareTrack Kids-part 1. Assessing the appropriateness of healthcare delivered to Australian children: study protocol for clinical indicator development.

INTRODUCTION: Despite the widespread availability of clinical guidelines, considerable gaps remain between the care that is recommended (appropriate care) and the care provided. This protocol describes a research methodology to develop clinical indicators for appropriate care for common paediatric conditions. METHODS AND ANALYSIS: We will identify conditions amenable to population-level appropriateness of care research and develop clinical indicators for each condition. Candidate conditions have been identified from published research; burden of disease, prevalence and frequency of presentation data; and quality of care priority lists. Clinical indicators will be developed through searches of national and international guidelines, and formatted with explicit criteria for inclusion, exclusion, time frame and setting. Experts will review the indicators using a wiki-based approach and modified Delphi process. A formative evaluation of the wiki process will be undertaken. ETHICS AND DISSEMINATION: Human Research Ethics Committee approvals have been received from Sydney Children's Hospital Network, Children's Health Queensland Hospital and Health Service, and the Women's and Children's Health Network (South Australia). Applications are under review with Macquarie University and the Royal Australian College of General Practitioners. We will submit the results of the study to relevant journals and offer national and international presentations. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Using and evaluating a novel method for ratifying indicators of ‘appropriate care’ for 20 paediatric conditions.

Achieving consensus on clinical indicators of ‘appropriate care’ that may be used for point-of-care decision-making and benchmarking purposes.

The recruitment of experts for the review process may introduce selection biases.

Introduction

Australian paediatricians commonly see children with a diverse range of sometimes complex health conditions.1 Clinical practice guidelines (CPGs) are available to help healthcare providers deliver appropriate care (care in line with evidence-based or consensus-based guidelines).2–4 However, it is not always easy for healthcare providers to navigate their way through CPGs due to factors such as: lack of timely access, multiple CPG sources and hence a lack of consensus, and lengthy recommendations that may not be specific or practical for point-of-care decision-making.5–10 Definitions of objective or measurable compliance with processes and outcomes are often lacking.5–10

Research was undertaken in the USA between 1998 and 2000 to develop recommendations for a range of paediatric conditions, and to benchmark the quality of ambulatory care against these recommendations.11 However, no such study has been conducted in Australia or elsewhere. The overall objective of CareTrack Kids (CTK) is to determine the appropriateness and safety of healthcare for common conditions delivered to children in Australia. In order to achieve this, a set of measurable clinical indicators is required.5

The CTK project involves a suite of three separate but related studies: part 1 (this study)—developing a set of clinical ‘appropriateness’ indicators for common paediatric conditions; part 2—measuring the appropriateness of paediatric care in Australia against these clinical indicators (using an onsite retrospective review of medical records during 2012 and 2013);12 and part 3—collecting information regarding the prevalence and characteristics of adverse events in paediatric healthcare encounters during 2012 and 2013.13

This protocol describes the clinical indicator development process for part 1 of CTK. It extends the methods of two studies from the USA11 14 and the CareTrack Australia (CTA) study15 16 using a collaborative online approach (a ‘wiki’). The outcomes are consensus on a set of clinical indicators for appropriate care for a range of common paediatric conditions in Australia during 2012 and 2013, and an evaluation of the clinical indicator development process.

Methods and analysis

There are three key components for this study protocol: to identify candidate paediatric conditions, to develop clinical indicators for these conditions and to evaluate the wiki methodology. The methods for the other CTK studies will be presented in separate protocol papers.12 13

Component 1: identify candidate paediatric conditions

A scoping exercise was undertaken to identify common candidate conditions from published research,1 17 18 burden of disease (BOD),17 prevalence (Children's and General Hospital 2011–12 New South Wales. SNOMED-CT Emergency departments, personal correspondence; Bettering the Evaluation and Care of Health. Top 100 most frequently managed problems 2011–12 weighted data: Age 0–17 years, personal correspondence, 2012),19 frequency of presentation (Children's and General Hospital 2011–12 New South Wales, personal correspondence)19 and quality of care priority lists (table 1).20 Using these data, the CTK research team identified an initial list of 28 conditions common across a range of healthcare practice facilities, including primary care provided by general practitioners, secondary care provided by outpatient paediatricians and tertiary care in hospitals.

Table 1

Prevalent paediatric conditions managed in Australian health facilities 2011–2012

Condition	Paediatric outpatients/community1	General practitioners (Bettering the Evaluation and Care of Health, personal correspondence)19 (>2% conditions)	Hospital inpatients19(>400 NSW cases pa)	ED cases (Children's and General Hospital 2011–12 New South Wales, personal correspondence)19(>400 NSW presentations pa)
ADHD*†	•
Learning difficulty/disability/behaviour/language delay†	•
Eczema and dermatological/skin problems†	•	•
Asthma*†‡	•	•	•	•
Allergy (food other than cows’ milk)†	•
Newborn/neonatal care*	•
Anxiety*†‡	•
General check-up/preventive care		•
GORD	•		•
Autism*†	•
Intellectual disability	•
URTI		•	•	•
Fever		•		•
Acute gastroenteritis		•		•
Croup				•
Head injury				•
Acute bronchiolitis		•		•
Sore ear—otitis media*		•
Acute abdominal pain				•
Renal dialysis			•
Leukaemia			•
Poisoning‡			•
Red blood cell disorders			•
Surgical repair cleft palate/lip			•
Tonsillitis/adenoids		•	•	•
Cerebral palsy			•
Diabetes‡	•		•
Viral illness		•		•

*Leading cause of DALYs in 0–14-year olds by sex, Australia 2003.17

†Most frequently reported long-term conditions on children aged 0–14 years, 2007–08.18

‡NHPA.20

ADHD, attention deficit hyperactivity disorder; DALYs, disability adjusted life years; ED, emergency department; GORD, gastro-oesophageal reflux disease; NHPA, National health priority area; NSW, New South Wales; URTI, upper respiratory tract infection.

The CTK research team further refined the list of candidate conditions by evaluating them against three criteria to determine their eligibility for inclusion: high prevalence (Bettering the Evaluation and Care of Health, personal correspondence),1 high clinical impact20 and BOD.17 Two additional conditions were recognised by the research team as having significant clinical importance and were therefore recommended for inclusion: obesity (high prevalence, impact and BOD)20–22 and urinary tract infection (high impact).23 24 Using these criteria, a list of 20 paediatric conditions (table 2) has been selected for inclusion.

Table 2

List of paediatric conditions and the key factors for their inclusion

	Condition	Rationale
1	Acute abdominal pain	Prevalent >1 provider type, 4th highest presentation in ED, 33rd most frequent visit to GP
2	ADHD	Prevalent >1 provider type, NHPA/BOD, #1 highest new diagnosis and review of condition for paediatricians
3	Acute bronchiolitis	Prevalent >1 provider type, NHPA/BOD, 5th most frequent visit to GP
4	Acute gastroenteritis	Prevalent >1 provider type and NHPA/BOD, 10th most frequent visit to GP
5	Anxiety/depression	Prevalent >1 provider type, NHPA/BOD, #10 highest new diagnosis for paediatrician consults, depression is 26th most frequent GP visit while anxiety is 34th most frequent visit to GP
6	Asthma	Prevalent >1 provider type, NHPA/BOD, high prevalence 10% of children have asthma, #6 highest new diagnosis, #4 review of condition for paediatricians, 4th most frequent visit to GP
7	Autism	Prevalent >1 provider type, NHPA/BOD, #5 review of condition for paediatrician consults
8	Croup	7th highest presentation in ED
9	Diabetes	Prevalent >1 provider type and NHPA/BOD
10	Eczema	8th most frequently managed problem in GP, #3 highest new diagnosis, #9 review of condition for paediatricians
11	Fever	9th highest presentation in ED
12	GORD	Prevalent >1 provider type, NHPA/BOD, #10 review of condition for paediatricians
13	Head injury	Prevalent >1 provider type, NHPA/BOD, 6th highest presentation in ED
14	Obesity	NHPA and key national childhood indicator
15	Otitis media	Prevalent >1 provider type, NHPA/BOD, 3rd most frequent visit to GP
16	Preventive care (SNAP well childcare)	Can be done on all records and includes screening for multiple conditions, 2nd most frequent visit to GP
17	Seizures (status epilepticus)	NSW Health guidelines Epilepsy ranked 8th males, 6th females 0–14-year-old DALYS
18	Tonsillitis	Prevalent >1 provider type, 7th most frequent visit to GP
19	URTI	Prevalent >1 provider type, the most frequent visit to GP
20	UTI	14th most frequent visit to GP, high clinical impact to patient

ADHD, attention deficit hyperactivity disorder; BOD, burden of disease; DALYs, disability adjusted life years; ED, emergency department; GP; general practitioner; GORD, gastro-oesophageal reflux disease; NHPA, National health priority area; NSW, New South Wales; SNAP, Smoking, Nutrition, Alcohol, Physical activity; URTI, upper respiratory tract infection; UTI, urinary tract infection.

Component 2: develop clinical indicators

Clinical indicators for this study will be developed using a three-stage process: (1) search and source relevant CPGs; (2) select, draft and format proposed clinical indicators; and (3) subject the indicators to several rounds of internal and external review using a modified Delphi approach.

Stage 1: search and source relevant CPGs

All clinical indicators will be derived from published CPGs relevant for the years 2012 and 2013. A systematic search will be undertaken sequentially in order of priority for national-level CPGs from Australia (eg, for the National Health and Medical Research Council (NHMRC)),2 and internationally (eg, for the National Institute for Health and Care Excellence (NICE) in England,3 Scottish Intercollegiate Guidelines Network (SIGN)4 and the Agency for Healthcare Research and Quality's (AHRQ's) National Guideline Clearinghouse in the USA).25 In the absence of Australian national and international CPGs, the CPGs of relevant professional medical colleges and associations will also be searched, as well as state or professional level (eg, the New South Wales (NSW) Clinical Practice Guidelines for Paediatrics).26 Three research team members (LKM, TDH and PDH) will be primarily responsible for conducting the CPG searches and developing the proposed clinical indicators. Full details of the search strategy are provided in online supplementary appendix A.27 28

Stage 2: select, draft and format proposed indicators

Recommendations from each CPG will be collated and used to inform the content and format of the proposed clinical indicators. For the purposes of this study, a clinical indicator is defined as a “measurable component of a standard or guideline, with explicit criteria for inclusion, exclusion, time frame and setting”.5 Not all recommendations published in CPGs will become indicators for the CTK study. Recommendations will be excluded based on the following criteria (table 3):

Table 3

Example recommendations mapped against indicator eligibility criteria

Indicator eligibility criteria	Example exclusions	Rationale for exclusion
Strength of wording	Following multidisciplinary review, if moderate to severe depression in a child (5–11 years) is unresponsive to a specific psychological therapy after four to six sessions, the addition of fluoxetine should be cautiously considered, although the evidence for its effectiveness in this age group is not established29	Use of wording: ‘cautiously considered’ and ‘evidence for its effectiveness in this age group is not established’
Likelihood of documentation	In children aged 4 weeks to 5 years, healthcare professionals should measure body temperature by one of the following methods: Electronic thermometer in the axilla Chemical dot thermometer in the axilla Infrared tympanic thermometer30	The method used to measure body temperature is unlikely to be documented in a medical record
Guiding statement without recommended action	Observation of infants and young children (ie, aged under 5 years) is a difficult exercise and therefore should only be performed by units with staff experienced in the observation of infants and young children with a head injury. Infants and young children may be observed in normal paediatric observation settings, as long as staff have the appropriate experience31	Guiding statement with no specific actions/criteria able to be used to determine compliance

Strength of the wording of the recommendation (ie, ‘may’ and ‘could’ statements would be excluded; ‘should’ and ‘must’ statements would be included);

Low likelihood of information being documented in the medical record;

Guiding statements without recommended actions.

All clinical indicators will be written in a structured and standardised format (ie, starting with the inclusion criteria followed by the action). For example, the inclusion criteria will define the age (infant, child, adolescent), condition and the phase of care (at diagnosis/presentation or ‘with’, indicating the diagnosis is existing). Indicators will be arranged chronologically according to phases of care (ie, screening, diagnosis, treatment, ongoing management). Table 4 provides examples.

Table 4

Examples of possible condition indicators

Condition	Classification	Indicator
Obesity	Screening/diagnosis	Children aged between 2–16 years have their BMI measured and diagnosed (using a BMI percentile chart) as follows according to the result: BMI for age and sex in 85th–94th centile are diagnosed as overweight OR BMI for age and sex >95th centile are diagnosed as obese
URTI	Treatment	Children with URTI are NOT to be prescribed antibiotics
Diabetes	Ongoing management	Children and young people with type 1 diabetes and their families are informed that the target for long-term glycaemic control is an HbA1c of less than 7.5%

BMI, body mass index; HbA1c, glycated haemoglobin; URTI, upper respiratory tract infection.

Stage 3: subject the indicators to several rounds of internal and external review

There are two stages involved in the indicator review. The proposed clinical indicators will initially undergo an internal review (stage 3a), followed by an external wiki-style review using a modified Delphi process (stage 3b).32 This approach has been chosen to enhance methodological rigour and optimise the content and face validity of the final set of clinical indicators.

Stage 3a: internal review processes

Internal reviews will be conducted by paediatricians and general practitioners sourced from within the CTK research team and their professional networks. It is envisaged that these reviewers will be employed as the head or director of a relevant paediatric department in a large hospital, hold at least an adjunct academic appointment or be directly involved in clinical care (eg, clinical psychologist). The clinical indicators for each condition will be reviewed by a panel of at least three different reviewers who, depending on their self-reported scope of practice and expertise, are able to participate in review panels for more than one condition. The total number of invitations will therefore depend on the skill-mix of invited reviewers and overall recruitment rate. This selection strategy is well supported within the Delphi process literature.33 34

The internal review will consist of three rounds. In the first round, drafts of proposed clinical indicators for each condition, and the recommendations on which they are based, will be sent via email to a panel of at least three internal reviewers. The review criteria to be used are based on the methods from the previous US and Australian studies.11 14 15 Internal reviewers will be asked to: score each indicator against three key criteria: acceptability, feasibility and impact (online supplementary appendix B); to recommend the indicators for inclusion or exclusion; and provide any additional comments. In this round, reviewers will complete their assignments independently from one another to minimise bias from ‘group-think’.35 36 Three team members (LKM, TDH and PDH) will collate the feedback and revise the content, structure and format of each indicator. The refined set of indicators (including the original indicator and de-identified feedback) will be sent to the same internal reviewers for a second round of scoring. The same approach will be used in this round, with a request for more information and identification of indicators to be included or excluded. In the final round, the proposed set of indicators will be sent to one of the three existing reviewers to review and approve the indicators for the external wiki-based review process.

Stage 3b: external wiki-based review

External reviews will be conducted by invited paediatricians and general practitioners. Relevant medical colleges, professional associations and networks, such as the Royal Australasian College of Physicians, Australian Paediatric Research Network (APRN), Children's Healthcare Australasia (CHA), Australian Research Alliance for Children and Youth (ARACY), Bettering the Evaluation and Care of Health (BEACH) programme, New South Wales (NSW) Kids and Families, Sydney Children's Hospitals Network, Children's Health Queensland (QLD), Women's and Children's Health Network (South Australia), will be contacted, requesting assistance with the recruitment of clinical experts to register as external reviewers. Invitations will comprise email notifications to members, media releases and articles within newsletters. Clinical experts will self-nominate as reviewers for one or more of the CTK conditions based on their interest, scope of practice and clinical experience.33 34

The external review will involve a wiki-based process whereby indicators for each condition (from round 3 of the internal review) will be posted to an online wiki site. The wiki ‘live’ time will depend on the recruitment rate of experts and the progress of their reviews, but is anticipated to be no longer than 3 months. The aim is for each condition to be independently reviewed by a minimum of nine experts. In addition to the scoring criteria used in the internal review process, indicators will be scored on a nine-point Likert scale as representative of appropriate care delivered to children during 2012 and 2013.11 15 37 A clinical champion for each condition will follow-up and manage external reviewers’ responses, and to make final recommendations regarding the inclusion, content, structure and format of indicators. For most conditions, this role will be undertaken by one of the stage 3a internal reviewers as previously identified by the CTK research team. The final sets of clinical indicators for care during 2012 and 2013 will be considered representative of appropriate care for Australian children for the candidate conditions and will form the suite of indicators for the second (part 2) CTK study.12 These indicators will also be useful for other purposes, such as providing healthcare practitioners with succinct and measurable compliance points to assist with point-of-care decision-making.

Component 3: evaluate the ‘wiki’ methodology

A multimethods evaluation of the wiki's methodology will be undertaken, with the aim of assessing utilisation, accessibility and ease of use.38 39 Three data sources will be used to inform the evaluation: (A) utilisation statistics sourced from the wiki logs—these will include demographics of users and rates and times of use; (B) the nature and content of reviewers’ and clinical champion's comments (eg, the format and rationale of proposed changes to indicators, level of agreement between reviewers, and resulting changes to the indicators); and (C) user perspectives. Following completion of their clinical indicator reviews, external reviewers will be invited to participate in an online survey of their experiences and perspectives. Both quantitative and qualitative analyses of these three data sources will be undertaken including: descriptive statistics for the characteristics of wiki users, their patterns of use and scores for clinical indicators; and frequency counts and content analyses of ratings and free-text responses from the user perspectives survey. Using these data, recommendations will be developed regarding the overall feasibility of the wiki process for future indicator developments and potential changes for improvement.

Ethics and dissemination

Ethical approval

Applications are under review with the Royal Australian College of General Practitioners.

To minimise the risk of bias, all reviewers will be required to complete a Conflict of Interest (COI) declaration. The COI declarations will be recorded for each reviewer and managed according to the NHMRC protocol for conflict.40

Dissemination

We will submit the results of the study to relevant national and international journals with the intention of publishing the results widely. As well, we will make national and international oral presentations to stakeholder groups including those involving patients, researchers, clinicians, managers and policymakers.

Discussion

We recognise several potential limitations to our study. The approaches used to identify candidate paediatric conditions and develop clinical indicators are specific to the Australian healthcare setting for the years 2012 and 2013. Findings can therefore not be generalised beyond these contexts. Experts will be invited to participate and self-nominate to review conditions that are within their scope of practice and for which they have current clinical experience. This may introduce a selection bias, as reviews will be representative of the participating sample rather than the general population of Australian paediatricians and general practitioners. There is potential for the final sets of clinical indicators to be a function of the scope of practice, clinical experience, level of research participation and degree of literacy with information technology of a subset of healthcare practitioners.

The CTK part 1 study will use and evaluate a novel method for ratifying indicators of ‘appropriate care’ for 20 paediatric conditions. These indicators will form the criteria against which the CTK part 2 study can, for the first time in Australia, measure appropriateness of paediatric care in 2012 and 2013. This study will help to establish the feasibility of using wiki-based technology for the purposes of developing clinical indicators. This information can be used to inform the design of future studies using Delphi processes to establish consensus on recommended healthcare, and will be relevant for national and international researchers, policymakers, healthcare practitioners and patients.