Primary Care Severe Asthma Registry and Education Project (PCSAR-EDU): Phase 1 - an e-Delphi for registry definitions and indices of clinician behaviour.

INTRODUCTION: Although most asthma is mild to moderate, severe asthma accounts for disproportionate personal and societal costs. Poor co-ordination of care between primary care and specialist settings is recognised as a barrier to achieving optimal outcomes. The Primary Care Severe Asthma Registry and Education (PCSAR-EDU) project aims to address these gaps through the interdisciplinary development and evaluation of both a 'real-world' severe asthma registry and an educational programme for primary care providers. This manuscript describes phase 1 of PCSAR-EDU which involves establishing interdisciplinary consensus on criteria for the: (1) definition of severe asthma; (2) generation of a severe asthma registry and (3) definition of an electronic-medical record data-based Clinician Behaviour Index (CBI). METHODS AND ANALYSIS: In phase 1, a modified e-Delphi activity will be conducted. Delphi panellists (n≥13) will be invited to complete a 30 min online survey on three separate occasions (i.e., three separate e-Delphi 'rounds') over a 3-month period. Expert opinion will be collected via an open-ended survey ('Open' round 1) and 5-point Likert scale and ranking surveys ('Closed' round 2 and 3). A fourth and final Delphi round will occur via synchronous meeting, whereby panellists approve a finalised ideal 'core criteria list', CBI and corresponding item weighting. ETHICS AND DISSEMINATION: Ethical approval has been obtained for the activities involved in phase 1 from the University of Toronto's Human Research Ethics Programme (approval number 39695). Future ethics approvals will depend on information gathered in the proceeding phase; thus, ethical approval for phase 2 and 3 of this study will be sought sequentially. Findings will be disseminated through conference presentations, peer-reviewed publications and knowledge translation tools. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first primary care severe asthma registry with an embedded educational programme that aims to enhance clinician behaviour as measured by data captured in the electronic medical record (EMR).

The development of a Clinician Behaviour Index (CBI) through consensus will allow for an evaluation of how primary care providers manage severe asthma in keeping with guideline recommendations.

The development of a CBI will provide opportunities to evaluate how quality improvement initiatives influence data capture in the EMR over time and how this may inform about changes in severe asthma outcomes.

The Primary Care Severe Asthma Registry and Education project will provide opportunities for the development of infrastructure necessary to understand the severe asthma care pathway in the real world and how it may be influenced by additional variables, including comorbidities and other relevant determinants of health.

Since phase 1 of this project focusses primarily on registry development and validation patient outcomes are not directly addressed.

Introduction

The identification and management of patients with severe asthma, defined as requiring a step 4 or 5 treatment to maintain symptom control,1 is challenging in the primary care setting.1 2 The wide variety of health issues managed in primary care and undifferentiated presenting symptoms represent important barriers for early detection. Reports describe missed opportunities to reduce the risk of asthma exacerbations and death, including factors that limit the referral of patients for specialist assessment from primary care.3 4 Indeed, recent evidence from international severe asthma registries suggests a large majority (72%) of individuals with potential severe asthma had neither been referred nor received specialist care within the preceding year.4 Systematic strategies that strengthen interdisciplinary management and monitoring of patients with severe asthma across the disease life cycle are necessary to improve the quality of care received by these patients.4

Existing initiatives like the International Severe Asthma Registry5 capture important information regarding patients managed in tertiary care; yet, there is limited ‘real-world’ data to inform the role of the primary care practitioner in the detection, referral and co-management of patients with severe asthma. Although patients with severe asthma represent a small proportion of the overall asthma population6, many patients have a primary care provider such as their family physician or nurse practitioner, who will usually be the first point of contact. Primary care-based ‘real-world’ data are needed in order to describe factors that influence physician decision making, patient behaviour and disease outcomes.7 A severe asthma registry in primary care will define and quantify the population prevalence of severe asthma while identifying best practices at both the clinical practice and population levels.

To address these existing care gaps, we will develop a severe asthma registry in primary care (phase 1), followed by the implementation of an education-based and integrated support system (phases 2–4) that aims to facilitate (1) collaboration among primary and secondary care providers and allied healthcare providers; (2) awareness of severe asthma management strategies and (3) data capture in the electronic medical record (EMR) reflecting severe asthma management in primary care. This paper will detail the protocol for phase 1, which will develop interdisciplinary consensus criteria for the: (1) definition of severe asthma; (2) generation of a severe asthma registry and (3) definition of a Clinician Behaviour Index (CBI). Phase 1 results will contribute to the development of an education intervention to improve primary care-based severe asthma management.

Project setting

This project will be conducted in partnership with the University of Toronto Practice-Based Research Network (UTOPIAN), a network of over 1700 family physicians in practices within the 14 Department of Family and Community Medicine academic sites dispersed throughout the Greater Toronto Area and other parts of Ontario.8 Primary care providers and patients associated with UTOPIAN will serve as the populations studied for the purposes of this project. Currently, UTOPIAN maintains primary care EMR data on over 6 00 000 patients from Toronto and other parts of Ontario.8

Primary objectives

The overarching goals of Primary Care Severe Asthma Registry and Education (PCSAR-EDU) are twofold: (1) develop a PCSAR and (2) implement a patient-centred educational programme for primary care providers that supports the collaborative management of severe asthma. Phase 1 activities focus on the development, implementation and validation of a PCSAR and EMR indices of clinician behaviour (ie, the CBI) which will represent an iterative metric of adherence to guideline-based care. The purpose of the severe asthma registry is to aid in the early identification, management, and referral of patients with severe asthma when needed. Registry development will involve:

Using validated case verification approaches9 10 to identify paediatric and adult patients with asthma in the UTOPIAN database.

Establishing criteria through e-Delphi consensus that are relevant for the identification of severe asthma using EMR data.

Using validated database indices of asthma severity11 12 to identify patients with ‘suspected severe asthma’.

Manual clinician review of ‘suspected severe asthma’ cases to establish a reference standard.

Generating algorithms using established e-Delphi consensus criteria to identify the severe asthma population in the UTOPIAN EMR data.

Validating these algorithms against the reference standard (ie, manual clinician review).

Implementing a severe asthma registry within the UTOPIAN database.

Using the registry to estimate the prevalence of severe asthma in a primary care database.

Methods and analysis

Project overview

PCSAR-EDU is a 4-year project (2019–2022) with four phases. Phase 1 involves the interdisciplinary development and implementation of a severe asthma registry within primary care (figure 1). The remaining phases involve subjective and objective needs assessments (phase 2), educational programme development, implementation and evaluation (phase 3) and project refinements through ongoing quality improvement efforts (phase 4). Collectively, phases 2–4 comprise strategies designed to support an ongoing and iterative ‘Clinician Behaviour Modification Cycle’ (figure 2) that integrates continuing medical education, clinical research and quality improvement efforts that will evolve based on the extent of change in the CBI, new knowledge and the need for innovating current practices. As more individuals are entered into the registry, new research will be derived from this database to improve generalisability of research findings at the population level.

Figure 1

PCSAR-EDU project phase 1 overview. CBI, Clinician Behaviour Index; PCSAR-EDU, Primary Care Severe Asthma Registry and Education; SA, severe asthma.

Figure 2

PCSAR-EDU clinician behaviour modification cycle schematic representation. PCSAR-EDU, Primary Care Severe Asthma Registry and Education.

Project governance

The proposed project adopts an integrated knowledge translation approach,13 whereby a steering committee (table 1) consisting of thirteen relevant stakeholders (ie, primary and specialist care, pharmaceutical, patient perspectives, as well as educational expertise and representation from The Lung Health Foundation—Ontario) was established to inform all stages of the project and ensure the educational programme adequately targets the audience’s educational and practice-enhancing needs (table 1).

Table 1

Pproject governance structure

Committee	Role	Members
PCSAR-EDU Steering Committee (SC)*	Project direction and oversight. Democratic decisions on PCSAR-EDU-related research projects in partnership with UTOPIAN-SAC.	Scientific leads Co-investigators Advisors (specialist, education, patient representatives) Education scientist Industry members†
UTOPIAN Scientific Advisory Committee (SAC)	In partnership with PCSAR-EDU SC members, provides scientific decisions about projects, initiatives and site-specific project facilitation.	14 site representatives Key members of the DFCM programmes Patient & community members
PCSAR-EDU Patient Advisory Committee	Project advice on identifying unmet needs related to patient care and clinical research.	Patients and family members with experience of severe asthma
UTOPIAN Respiratory Health Working Group	Academic partner providing methodological advice and research priority oversight. Promotes primary care leadership in Respiratory Health. Promotes the use of big data for research.	Researchers, clinicians and graduate/medical trainees with a special interest in respiratory health
PCSAR-EDU Education, Quality Improvement and Implementation Committee	Provides guidance on the identification and implementation of site specific education-based QI initiatives.	Experts in quality improvement, education development and research

*All final decisions arising from the steering committee will be determined by voting members only.

†Non-voting members.

DFCM, Department of Family and Community Medicine; PCSAR-EDU, Primary Care Severe Asthma Registry and Education; QI, Quality Improvement; UTOPIAN, University of Toronto Practice-Based Research Network.

Patient involvement

Patients will be involved in the design and implementation of all phases of the project. In phase 1, patients involved in both the steering committee and patient advisory committee will guide project and registry priorities.

e-Delphi consensus activity design

A four-round modified e-Delphi activity will establish expert consensus on specific criteria for the: (1) definition of severe asthma; (2) entry of patients into the severe asthma registry and (3) CBI definition (figure 1).

Severe asthma definition consensus

We will consider two definitions of severe asthma as outlined by the GINA 2020 Strategy1 and the Canadian Thoracic Society Position Statement.14 Panellists will establish consensus on a specific definition of severe asthma which may reflect agreement with existing definitions (ie, GINA and/or CTS) or the development of consensus on a novel definition of severe asthma. Our mandate is to establish consensus on a definition of severe asthma that reflects the collaborative effort and expertise of primary, specialty and allied care.

Severe asthma registry entry criteria consensus

Panellists will establish the specific criteria used to (1) flag patients for entry and (2) confirm patient entry into the PCSAR. The agreed on definition of severe asthma will determine the type of data, including pharmacotherapy, that should be included in the database to flag a patient for potential registry entry. Additional EMR data criterion under consideration that will be used to flag and/or enter a patient into the PCSAR may include:

Objective documentation of asthma diagnosis.

Report of one or more exacerbation(s) requiring oral steroids per year in the community or hospital setting.

Report of symptoms several days during the week requiring rescue medication use.

Evidence of reduced lung function.

Other (as specified by participants in the open e-Delphi round).

CBI criteria consensus

E-Delphi panellists will also establish consensus on the components of the CBI. As previously stated, the CBI will include EMR data elements associated with guideline-based care that are deemed relevant by the expert panellists and will serve as an iterative metric of PCP adherence to guideline-based care. Panellists will vote on whether this metric is represented by a composite score of several CBI items or if there is a single CBI item that should serve as the primary outcome variable for the purposes of evaluating the impact of the proposed education programme on PCP adherence to guideline-based care. The agreed on CBI item(s) will be monitored via ongoing chart reviews and will track changes in PCP management of severe asthma and EMR documentation processes over time with a view to inform future education initiative objectives. The individual components of the CBI under evaluation are outlined in table 2.

Table 2

Components of the CBI under evaluation

ITEM	Description
1	A record of severe asthma diagnosis.
2	A record of instances when inflammatory markers are ordered to assess severe asthma.
3	A record of upward titration of asthma medications or loss of asthma control with downward titration.(ie, escalating from inhaled corticosteroid (ICS) monotherapy to ICS plus long acting bronchodilator-(B2-agonist (LABA) or antimuscarinic (LAMA)), or from ICS/LABA or LAMA to ICS/LABA/LAMA or any other form of escalation in keeping with Canadian/GINA asthma guideline/strategy).
4	A record of asthma control (ACT or ACQ), the need for emergency room care, hospitalisation, frequent visits to medical clinics, use of oral prednisone, and loss of productivity related to work or school and reduced lung function.
5	A record of variable airflow obstruction (ie, either simple spirometry or methacholine).
6	A record of personalised asthma action planning.
7	A record of exacerbation information (ie, frequency, duration).
8	A record of inhaler review/education.
9	A record of excluding incorrect diagnosis of asthma due to alternative conditions such as inducible laryngeal obstruction, cardiac failure or lack of fitness.
10	A record of excluding comorbidities and complicating conditions such as rhinosinusitis, gastro-oesophageal reflux and obstructive sleep apnoea.
11	A record of excluding ongoing exposure to sensitising or irritant agents.
12	A record of referral of patients to specialists (ie, respirologist/allergist/other) for suspected severe asthma
13	A record of primary care provider follow-up based on referral recommendations.
14	Other (as specified by participants in the open e-Delphi round).

CBI, Clinician Behaviour Index.

Selection of Delphi panellists

Our approach will involve purposeful sampling of individuals with knowledge of severe asthma identification and management, including those with clinical and/or research expertise in the area. Our goal is to ensure diverse perspectives are represented with regards to severe asthma management priorities and outcomes. Project collaborators including family physicians, paediatric and adult respirologists, allergists, respiratory therapists, and experts in respiratory research (n≥13) will be invited to complete the Delphi consensus activity.

Delphi panel sampling

Although no strict sample size requirements exist in the literature, our project will include a minimum of 13 panellists which is in keeping with prior recommendations.15–17 Each participant will be asked to create a unique identification code. Analysis of results will occur through the online survey platform Welphi (www.welphi.com) and via password-protected excel spreadsheets.

Delphi consensus methods

As previously stated, this project will adopt a four-round modified e-Delphi consensus approach (figure 1), whereby consensus will be established via soliciting anonymous, iterative rounds of feedback and opinion until consensus is achieved.

e-Delphi data collection and analysis

Delphi panellists will be invited to complete a 30 min online survey on three separate occasions (ie, three separate e-Delphi ‘rounds’) over a 3-month period.18 Panellist opinions will be collected via an open-ended survey (round 1; ie, ‘open-round’) and 5-point Likert scale and ranking surveys (round 2 and 3; ie, ‘closed-rounds’).

Round 1

In the open round (ie, round 1), participants will be presented with two widely recognised definitions of severe asthma as well as a draft list of criteria for the entry of patients into the severe asthma registry and the definition of a CBI. In addition to panellist demographic information, round 1 will solicit written opinions regarding the appropriateness of existing definitions of severe asthma as well as suggestions for additional criteria items that should be considered by the expert panel for registry entry and the CBI. The resulting list of items will be aggregated and analysed via qualitative content analysis methods19 and anonymously circulated within the second e-Delphi round for ranking (ie, closed-round). This round will be open for a period of approximately 1 month during June 2021. Reminder emails will be sent out twice throughout this period.

Round 2 and 3

In each closed round (ie, round 2 and 3), participants will be encouraged to provide opinions (via Likert scales; strongly disagree—strongly agree) and written comments regarding the requirement of each item. Participant responses will be analysed using frequency counts. Criteria items achieving less than 50% consensus (ie, 50% or more participants indicated ‘strongly disagree’ or ‘disagree’ with item being required) in the second e-Delphi round will not be included in the third e-Delphi round. All other criteria items and associated anonymous participant comments will be circulated for the third e-Delphi round. Criteria items achieving 80% agreement or more on ‘round 3’ will be compiled and will represent the ideal ‘core criteria list’ since all core criteria agreed on may not be implementable in every patient. Round 2 and 3 will be open for approximately 1 month, respectively. Two reminder emails will be sent during each period.

Round 4

A fourth and final Delphi round will occur via an in-person or virtual meeting, whereby panellists reflect on round 3 results and approve a finalised ideal ‘core criteria list’. Any outstanding disagreements will be documented and addressed where possible using a democratic approach. This fourth round will also involve determining the minimum and maximum number of criteria items to be included in the CBI, assigning weightings to the items and establishing a scoring framework. The resulting criteria list with associated weight and scoring will represent the CBI.

Severe asthma registry validation

This project will use EMR data from patients in the UTOPIAN database in Ontario, Canada.

Establishing the reference population

The reference population will be obtained from the UTOPIAN database. As previously stated, this database contains medical chart information from almost 600 000 patients which includes all age groups (53% female).20 Available data include the cumulative patient profile (CPP), demographics, progress notes, laboratory test results, allergies, medications, immunisation history and vital measurements.

Patients within UTOPIAN with an asthma diagnosis will be identified using a validated case definition.9 10 Inclusion criteria will be: age >6 years and use of EMR by the primary care provider for at least 2 years. Exclusion criteria includes: (1) greater than 10 pack-year smoker (ie, current or former smokers with a pack-year history of 10 or less would be included); (2) serum alpha 1 antitrypsin level <11 µmol and (3) other chronic lung disease (eg, Interstitial lung disease, Chronic obstructive pulmonary disease, bronchiectasis, cystic fibrosis).

To establish the reference population, a random subsample of patients with a high likelihood of diagnosis of asthma will be selected. Given the low prevalence of severe asthma in the general population, this project will select patients more likely to have severe asthma using validated database indices of asthma severity previously described11 12 to then undergo chart abstraction by trained abstractors. This approach will involve a search in patient medication history that categorise patients into ‘suspected severe asthma’ (ie, prescribed prednisone, moderate to high dose ICS/LABA, or reports of asthma-related hospitalisation or ER visit), which will undergo chart abstraction or ‘severe asthma unlikely’ (ie, not prescribed prednisone, ICS/LABA, no reports of asthma-related hospitalisation) whose charts will not undergo abstraction.

The charts of the patients categorised as ‘suspected severe asthma’ will be manually reviewed by a primary care physician using a standardised abstraction manual. Patients meeting criteria for a diagnosis of severe asthma will represent the reference standard. The EMR criteria for a diagnosis of severe asthma agreed on by Delphi Panellists will attempt to exclude patients with ‘difficult-to-treat asthma’.1 To determine intra-rater reliability, 25 randomly selected charts will be re-abstracted by the same abstractor. To assess inter-rater reliability, an additional 25 randomly selected charts will be abstracted by different abstractors (a respirologist, a primary care physician and an allergist). Uncertain categorisations will be resolved by a consensus committee consisting of a primary care provider and a respirologist. Kappa statistics will be used to measure inter-rater and intrarater reliability.

Variables for data algorithms

Registry algorithms will be based on items in EMR data that achieve expert consensus during the e-Delphi consensus activity. These EMR data will be captured through automated searches of free text within the CPP (eg, a record of asthma diagnosis; exacerbations requiring oral or systemic steroids; rescue medication use) and manual review of test results (eg, spirometry reports of reduced lung function) within the electronic medical chart.

Algorithm evaluation

Algorithm testing will involve searching the EMR for the registry entry criteria items that achieved e-Delphi consensus, as previously described. Combinations of e-Delphi consensus registry entry items will be evaluated to increase sensitivity and/or specificity. Algorithm examination will involve diagnostic accuracy assessments of specificity, positive predictive value and negative predictive value.

Clinician Behaviour Index validation

Since the CBI is a new assessment tool specific to this project, validity evidence relevant to research outcomes will be sought. Specifically, content validity, internal structure (ie, reliability) and response process will be evaluated according to the standards for validity.21

Content evidence for the CBI

The resulting CBI established by the Delphi panellists will develop validity evidence through a formal evaluation of content validity from an online questionnaire sent to a content evaluation panel. The content evaluation panel will involve purposeful sampling of 15 individuals according to their professional certifications, experience, accessibility and publications. The content evaluation panel will represent the perspectives of paediatric and adult respirologists, allergists, respiratory therapists and experts in respiratory research who are not otherwise involved in the project. The panel will score each CBI item in a four-point Likert scale in the following three domains: relevance, simplicity and clarity (from 1: not relevant, not simple, and not clear to 4: very relevant, very simple and very clear).22

Content validity will be assessed quantitatively through calculating the content validity ratio (CVR) and content validity index (CVI) for each item. The CVI will be calculated for all individual items (I-CVI) and the overall scale (S-CVI). For CVI, the content evaluation panel will rate each CBI item in terms of its relevance to the underlying construct (ie, adherence to severe asthma management guidelines). For each item, the I-CVI will be calculated as the number of panellists giving a rating of 3 or 4 divided by the total number of panellists. In keeping with existing recommendations, if I-CVI is >0.79, the item will be considered relevant; between 0.70 and 0.79, the item will be revised by the research team as per content evaluation panellist feedback; and if below 0.70 the item will be eliminated from the CBI.23 The S-CVI average, determined through calculating the average I-CVI of included items, will be computed to ensure the content validity of the entire scale (ie, the CBI). A minimum S-CVI of 0.8 is recommended.24

The use of Lawshe’s CVR will determine whether an item is necessary. The Content Evaluation Panel will be asked to score each item on a three-point scale ranging from 1=essential, 2=useful but not essential and 3=not necessary. The formula to calculate CVR = (Ne – N / 2) / (N / 2) where Ne is the number of panellists indicating ‘essential’ and N is the total number of panellists. Essential items will be determined according to Lawshe’s table, which states an acceptable CVR value for 15 experts is above 0.49.25

Internal structure

Two independent reviewers will score, through chart abstraction, the CBI of the same 50 patient charts. Cohen’s kappa will be assessed to determine the level of agreement between the two reviewers; an acceptable kappa will be >0.7. Additional analyses will explore the degree to which individual components of the CBI drive total score including intercomponent reliability and exploration of underlying factor structure, if appropriate.

Response process

The reviewers will report using a structured questionnaire on the process of deriving the CBI, issues of interpretation and quality control processes for recording CBIs.

Registry implementation and analysis

The validated severe asthma registry entry criteria will be used to develop a registry abstraction manual. This manual will describe in detail how to flag and enter individuals who meet the validated criteria into an ongoing PCSAR. This information will provide the basis for the calculation of a prevalence estimate based on the total number of individuals with severe asthma26 in the database between 1 September 2018 and 1 September 2020. The registry data will be managed and used based on objectives and priorities established by the PCSAR-EDU Steering Committee and UTOPIAN RHWG with oversight by the UTOPIAN Scientific Advisory Committee. PCSAR will be managed by UTOPIAN scientists and University of Toronto faculty involved in the project. Other stakeholders will be informed of the registry status and progress without compromising patient confidentiality. Clinician investigators that are affiliated with UTOPIAN will have access to deidentified registry data.

Ethics and dissemination

Ethical approval

Ethical approval is required for phases 1–3 of this project. Approval has been obtained for the activities involved in phase 1 from the University of Toronto’s Human Research Ethics Programme (approval number 39695) and will be sought for phase 2 and 3 activities. Future ethics approvals will depend on information gathered in the proceeding phase; thus, ethical approval for phase 2 and 3 of this study will be sought sequentially in order to address phase specific ethical considerations that may emerge.

Dissemination plan

Study findings will be shared and discussed at relevant project committee meetings to promote ongoing project evaluation and improvement. In addition, results will be disseminated through conference presentations, peer-reviewed journal publications and knowledge translation tools. With the launch of the validated severe asthma registry in primary care, PCSAR-EDU project collaborators will vote democratically on research proposals submitted to PCSAR-EDU, which will then require final approval from the UTOPIAN Scientific Advisory Committee.