Protocol for a Delphi consensus study to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia.

BACKGROUND: High-quality general practice has been demonstrated to provide cost-effective, equitable health care and improve health outcomes. Yet there is currently not a set of agreed comprehensive indicators in Australia. We have developed 79 evidence-based indicators and their corresponding 129 measures of high-quality general practice. This study aims to achieve consensus on relevant and feasible indicators and measures for the Australian context.
METHODS: This Delphi consensus study, approved by WSU Human Research Ethics Committee, consists of three rounds of online survey with general practice experts including general practitioners, practice nurses and primary health network staff. The identified indicators and measures are grouped under an attribute framework aligned with the Quadruple Aim, and further grouped under structures, processes and outcomes according to the Donabedian framework. Participants will rate each indicator and measure for relevance and feasibility, and provide comments and recommendations of additional indicators or measures. In the last round, participants will also be asked their views on the implementation of a quality indicator tool. Each indicator and measure will require ≥70% agreement in both relevance and feasibility to achieve consensus. Aggregated ratings will be statistically analysed for response rates, level of agreement, medians, interquartile ranges and group rankings. Qualitative responses will be analysed thematically using a mixed inductive and deductive approach. DISCUSSION: This protocol will add to the current knowledge of the translation of performance guidelines into quality practice across complex clinical settings and in a variety of different contexts in Australian general practice. The Delphi technique is appropriate to develop consensus between the diverse experts because of its ability to offer anonymity to other participants and minimise bias. Findings will contribute to the design of an assessment tool of high-quality general practice that would enable future primary health care reforms in Australia.

Introduction

The collection, robust analysis and appropriate use of general practice data are critical to informing continuous quality improvement and ensure high-quality primary health care (PHC). Many countries around the world collect PHC data for quality improvement purposes such as monitoring health and utilisation of health services like the Nivel’s Primary Care Database in The Netherlands, [1] and for research purpose such as the Clinical Practice Research Datalink (CPRD) in the UK [2].

Australian general practice care data is routinely collected by primary health networks (PHNs) for quality improvement purposes and more recently through the Practice Incentives Program Quality Improvement (PIP QI) initiative, launched in August 2019 by the Australian Government to improve patient outcomes [3-6]. Practices registered for PIP QI are required to regularly submit data collected against ten specified improvement measures to their local PHNs and commit to implementing continuous quality improvement activities in partnership with the PHNs [4]. The general practice data, known as the PIP Eligible Data Set, collected through this process is then uploaded to a national portal and the Australian Institute of Health and Welfare (AIHW) oversees access by researchers and other interested parties to the deidentified PIP Eligible Data Set [4]. The 31 PHNs were established in 2015 across Australia for supporting PHC to identify and meet local health needs, building PHC workforce capacity and the delivery of high-quality care, and integrating local health services to improve patient experience and better use of health resources [5]. They also collect and collate other general practice data from extraction of de-identified data from individual practice-based patient records [5]. However, there is a lack of consistency across PHNs in data content, variability in the quality of the data collected and also in the quality improvement outcomes achieved through this process [7].

A general practice indicator is “a measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in the quality, of care provided” [8]. A set of standardized and evidence-based indicators to measure and track high quality clinical performance and outcomes in general practice is necessary, not only for the profession’s accountability, but also for identifying population needs and gaps in the quality of care received by patients across Australia [7, 9]. Whilst the Royal Australian College of General Practitioners (RACGP) Standards for General Practices underpin accreditation, they are minimum standards for benchmarking purposes [4, 6]. Previous work to develop PHC quality indicators have focused on specific areas of interest, for instance, the Primary Care Practice Improvement Tool (PC-PIT) which is an organisational performance improvement tool focusing only on systems and processes [10]. Even though the collection of data is key to the evaluation of health care quality and services and clinical decision-making, there is currently not a set of universally agreed comprehensive high-quality indicators in Australia that would identify, measure and reward high-quality general practice.

In 2020, Western Sydney University, in partnership with PHNs in the western Sydney region, conducted a literature review to identify evidence-based indicators and measures, then assessed these in three workshops with general practitioners (GPs), practice managers, nurses, consumers and PHN staff in the western Sydney region [11]. A suite of 79 evidence-based indicators and their corresponding 129 measures of high-quality general practice was subsequently developed. The measures specifically included outcome measures as these are rarely addressed in frameworks of quality PHC [12, 13]. Key literature was also analysed to identify four attributes of high-quality general practice and construct a suitable framework for the indicators and measures [11]. The attributes are expressed as ‘accountabilities’: accountability to our patients; professionally accountable; accountability to the community and accountability to society [14, 15]. They align with the elements of the Quadruple Aim which states that effective healthcare improvement must take into account the care of individual patients, the health of populations, health care costs and the wellbeing of health care providers, [16] and is increasingly used to monitor and evaluate primary health system performance in Australia and countries like the UK and US [13, 17, 18]. The indicators and measures identified are further grouped under structures, processes and outcomes of high-quality general practice according to a Donabedian framework, [19, 20] and include some “blue sky” measures considered difficult to currently implement but are nonetheless important.

This study extends the previous work by Western Sydney University [11]. Wider consultations with Australian stakeholders will be conducted using a Delphi consensus study with experts to explore the relevance and feasibility of the identified suite of indicators and measures. Experts will include Australian general practices and PHNs involved in quality improvement initiatives. Consultations have been held with consumers, with regards to key patient-reported measures (PRMs). Aboriginal and Torres Strait Islander health and justice health sectors will also be consulted with regard to relevant indictors unique for those populations. These will be detailed elsewhere.

Materials and methods

Aim

The overall aim is to establish consensus with experts to contribute to the development of the first comprehensive, evidence-based, professionally endorsed tool for analysing and reporting across all components of high-quality general practice in Australia.

Study design

This study will use a survey to achieve consensus across an expert group of general practice and PHN staff. The Delphi technique has been selected due to its flexibility and anonymity provided to participants [21, 22]. The survey will consist of three rounds to obtain opinions on a suite of indicators and measures previously developed by the research team to reach consensus on a core set of relevant and feasible high-quality performance indicators [11].

Project governance

A Project Control Group has been established with the responsibility for overseeing the conduct of the project. The group consists of representatives from the Digital Health Cooperative Research Centre (CRC), and eight primary health organisations: Brisbane North PHN, Central and Eastern Sydney PHN, Nepean Blue Mountains PHN, North Western Melbourne PHN, South Western Sydney PHN, Western Sydney and then (WentWest), Western Australia Primary Health Alliance, and Western NSW PHN.

A Steering Committee, that meets more frequently, has also been established to provide strategic direction and advice to the research team on dissemination and collaboration with relevant stakeholder groups. This committee consists of the representatives of the primary health organisations and the RACGP, Australian College of Rural and Remote Medicine (ACRRM), Justice Health NSW (New South Wales) and SA (South Australia) Prison Health Service.

Setting

The study will be undertaken in four states in Australia across regions of the eight primary health organisations: seven PHNs and one primary health alliance comprising three PHNs which support primary care across a less populous state of Australia. These organisations cover a total area of 2,942,817km2 in metropolitan and rural Australia, and a diverse population of over 9.6 million with over 3,000 general practices. The characteristics of the PHNs, their geographical locations and the populations in their regions are summarised in Table 1.

Table 1

Characteristics of the primary health organisations involved.

Primary health organisation		Total area	Geographical location	Population in year	Number of general practices in year
Western Sydney PHN (WentWest) [34]		766 km2	Metropolitan	>1,000,000 in 2020	347 in 2020
Nepean Blue Mountains PHN [35]		9,063 km2	Metropolitan	>380,000 in 2020	138 in 2020
South Western Sydney PHN [36]		6,186 km2	Metropolitan	1,019,985 in 2020	429 in 2020
Central and Eastern Sydney PHN [37]		626 km2	Metropolitan	1,637,740 in 2018	608 in 2020
Western NSW (New South Wales) PHN [38]		433,379 km2	Rural	309,900 in 2020	110 in 2020
North Western Melbourne PHN [39]		3,212 km2	Metropolitan	1,640,000 in 2020	564 in 2020
Brisbane North PHN [40]		3,901 km2	Metropolitan	1,004,747 in 2017	341 in 2019
WA (Western Australia) Primary Health Alliance	Perth North PHN [41]	2,975 km2	Metropolitan	1,065,744 in 2016 [42]	248 in 2019
	Perth South PHN [43]	5,148 km2	Metropolitan	965,997 in 2016 [44]	250 in 2019
	Country WA PHN [45]	2,477,561km2	Rural	548,185 in 2016	Unavailable

Sample size

The study will aim to recruit a minimum of 80 participants. A minimum of 17 participants is the recommended minimum sample size for content validity in Delphi studies involving the selection of healthcare quality indicators [23]. In order for this Delphi study to meet the minimum sample size requirement, we must achieve a minimum of 47% retention rate in rounds 2 and 3.

Participants and recruitment

Participants will include GPs, practice nurses, practice managers and key PHN staff who are familiar with quality improvement initiatives in the context of Australian general practice. People under 18 years old will be excluded.

A purposive and convenience sampling approach will be used. Each of the eight primary health organisations will assist in recruiting eight to ten general practices in their region and nominate two to three key PHN staff. Practices will be purposively recruited to maximize diversity in regard to geographic location, practice size, and socio-economic status based on the Socio-Economic Index for Areas (SEIFA). An Invitation Pack containing an invitation letter, project information and consent form will be emailed by the PHN to their nominated staff and recruited practices. Each practice will nominate one to two practice staff to participate in the survey. All survey participants will be anonymised to their PHNs and other participants with allocation of a random identification number. A password protected file will be maintained by the research team with participants’ identifying information.

Criteria for the Delphi participants to consider

A total of 79 indicators with 129 measures that had been developed and finalised by the QUEST PHC team in 2020 [11] will be assessed by participants in the Delphi study. (Table 2) They are grouped under the four attributes of high-quality general practice framework aligned with the four elements of the Quadruple Aim [14-16]. Table 3 outlines the four high-quality general practice attributes, their definitions and alignment with the Quadruple Aim and the number of indicators and measures identified under each attribute.

Table 2

Indicators and measures for assessment by participants.

Indicators	Related measures
ATTRIBUTE 1: ACCOUNTABLE TO OUR PATIENTS
PERSON CENTRED CARE AND PATIENT-TEAM RELATIONSHIP
S1: Availability of information for patients	Written and electronic information in appropriate languages
P2: Patient input/feedback on health care delivery	Evidence of formal process to consider patient input and incorporate into practice care delivery
O3: Patient perceptions of care	Results of PREMs
O4: Patient activation	PAM® scores
O5: Strength of team- patient relationship	Results from using validated survey tool
EVIDENCE-BASED COMPREHENSIVE CARE: PREVENTIVE HEALTH CARE
P6: Risk factors recorded	% active patients ≥15 years with a BMI recorded who have weight classification (obese, overweight, healthy, underweight) in previous 12 months
	% active patients ≤ 15 years with height/length and weight recorded in previous 12 months
	% active patients ≥15 years with a smoking status recorded/updated (current, ex-smoker, never smoked) in previous 24 months
	% active patients ≥15 years with alcohol consumption status recorded in previous 24 months
	% active patients aged 14–19 years with other substance use recorded
	% active patients ≥18 years with BP recorded in previous 24 months
P7: Childhood adverse experiences recorded (blue sky)	% active patients aged 0–19 years screened for adverse childhood experiences in previous 12 months (blue sky)
P8: Early detection of cancer	% active patients aged 50–74 years with FOBT recorded in previous 24 months
	% active female patients aged 25–74 years without hysterectomy with up-to-date cervical screening
	% active female patients aged 50–74 years with no history of breast cancer screened with mammogram in previous 24 months (blue sky)
P9: Adult vaccination	% active patients ≥65 years immunised against influenza in previous 15 months
	% active patients with DM immunised against influenza in previous 15 months
	% active patients with COPD ≥15 years immunised against influenza in previous 15 months
	% active patients ≥70 years with one dose of pneumococcal immunisation recorded and for Aboriginal and Torres Strait Islander patients ≥50 years two doses at 5-year interval
	% active patients >70–79 years with shingles vaccination
P10: Childhood vaccination	% active patients ≥4 years who are fully immunised according to guidelines
P11: Aboriginal and Torres Strait Islander preventive health care	% active patients identified as Aboriginal and/or Torres Strait Islander with Aboriginal Health Check in previous 15 months
O12: Patient perceptions of preventive health discussion	PREMs to include patient report of discussion regarding the following health behaviours/risk factors: healthy eating, exercise/physical activity, risks of smoking/QUIT support, alcohol use, unintentional injuries (home risk factors), unsafe sexual practices, unmanaged psychosocial stress
EVIDENCE-BASED COMPREHENSIVE CARE: CHRONIC CARE
S13: Systems for management of chronic disease	Use of patient chronic disease registers
P14: Systems for management of chronic disease	Use of registers for patient follow up and recall
S15: Diabetes: known prevalence	% of active patients with diabetes coded in patient records
P16 Diabetes: monitoring CV risk	% active patients with DM and have their BP recorded in previous 6 months
	% active patients with DM and have their BMI recorded
	% active patients with T2DM and have their total Cholesterol, HDL, triglyceride and LDL levels recorded
P17: Diabetes: monitoring renal function	% active patients with DM and have their eGFR (estimated glomerular filtration rate) recorded in previous 12 months
	% active patients with DM and have their urine ACR recorded in previous 12 months
P18: Diabetes: managing risk	% active patients >60 years with T2DM prescribed a statin
P19: Diabetes care: managing complications	% active patients with DM and have their retinal screening performed in previous 24 months
	% active patients with DM and have their diabetic foot assessment in previous 12 months
P20: Diabetes: monitoring blood sugar control	% active patients with DM and have their HbA1c recorded in previous 12 months
O21: Diabetes: optimal management	% active patients with T2DM with HbA1c ≤8%
	% active patients with T2DM with BP <140/90 mmHg
O22: Diabetes: optimal risk management	% active patients with T2DM with lipids to target in previous 12 months
	% active patients with T2DM with microalbuminuria on ACE inhibitor or ARB
	% active patients >16 years with DM and not smoking
S23: Respiratory disease: known prevalence	% active patients with COPD coded in patient records
	% active patients with asthma coded in patient records
P24: Respiratory disease: use of spirometry record	% active patients with COPD and have spirometry
	% active patients with asthma and have their spirometry recorded in previous 24 months
P25: Respiratory disease: monitoring risk factors	% active patients with COPD and have their smoking status recorded
	% active patients >15 years with asthma and have their smoking status recorded
P26: Respiratory disease: planning care (blue sky)	% active patients with asthma with an asthma management plan (blue sky)
P27: Respiratory disease: Control (blue sky)	% active patients with COPD and have their COPD Assessment Test score recorded (blue sky)
	% active patients with asthma and have Asthma Control Questionnaire recorded (blue sky)
P28: Respiratory disease: appropriate use of medication	% active patients with COPD on LAMA
	% active patients ≥12 years with asthma on ICS containing preventer (blue sky)
O29: Respiratory disease: COPD control (blue sky)	% active patients with COPD and have been hospitalised in previous 6 months (blue sky)
S30: Cardiovascular disease: known prevalence	% active patients with CVD by category coded in patient records
P31: Cardiovascular disease: monitoring CVD risk	% active patients aged 45–74 years with the necessary risk factors assessed (smoking, diabetes, BP, Total Chol, HDL, age, gender) to enable CVD assessment in previous 24 months
	% active patients aged 45–75 years with no known CVD and with absolute CVD risk calculated in previous 24 months
	% active Aboriginal and/or Torres Strait Islander patients aged 35–75 years with no known CVD and with absolute CVD risk calculated in previous 24 months
P32: Cardiovascular disease: monitoring CVD	% active patients ≥18 years with hypertension and have BP recorded in the previous 6 months
P33: Cardiovascular disease: management of CVD	% active patients ≥18 years with CVD and have statin prescribed
34: Cardiovascular disease: Optimal outcome	% active patients with hypertension whose most recent BP is <140/90 mmHg
35: Renal disease: known prevalence	% active patients with renal disease coded in patient records
P36: Renal disease: screening for renal disease	% active patients with DM screened for nephropathy (eGFR and ACR) in previous 12 months
	% active patients coded in patient record as having hypertension screened for nephropathy (eGFR and ACR) in previous 12 months
	% active Aboriginal and/or Torres Strait Islander patients >30 years screened for nephropathy (eGFR and ACR) in previous 24 months
P37: Renal disease: monitoring renal disease	% active patients with renal disease and had their BP recorded in previous 12 months
	% active patients with renal disease and had their eGFR recorded in previous 12 months
	% active patients with renal disease and had their urine ACR recorded in previous 12 months
	% active patients with renal disease and had their chronic kidney disease stage recorded
O38: Renal disease: dialysis	% active patients with renal disease on dialysis
S39: Mental health: known prevalence of mental health conditions	% active patients with mental health conditions within each mental health category
S40: Mental health: known prevalence of co-morbidity	% active patients with mental health and also diagnosed with each of following: diabetes, CVD, respiratory and renal disease
P41: Mental health: treatment planning	% active patients with mental health with a GP Mental Health Treatment Plan (such as MBS item number 2715) in previous 12 months
P42: Mental health: management of patients with a mental health diagnosis documented	% active patients ≥15 years with a BMI recorded who have weight classification (obese, overweight, healthy, underweight) in previous 12 months
	% active patients ≥15 years with a smoking status recorded/ updated (current, ex-smoker, never smoked) in previous 24 months
	% active patients ≥15 years with alcohol consumption status recorded in previous 24 months
	% active patients with follow-up GP visit within 7–30 days of hospital discharge related to psychiatric condition (blue sky)
S43: Advance care planning (blue sky)	% active patients ≥75 years with discussions about advance care planning recorded on file (blue sky)
P44: Advance care planning (blue sky)	% active patients ≥75 years with Advance Care Plan uploaded to My Health Record (blue sky)
ACUTE CARE: PRESCRIBING SAFETY
S45: Safe prescribing of opioids and benzodiazepines	Practice has a policy on the safe prescription of opioids and BZDs
S46: Safe prescribing of opioids and benzodiazepines	Practice has a policy on discussing safe prescription of opioids and BZDs with all new prescribers
O47: Safe prescribing of opioids and benzodiazepines	% acute patients prescribed opioids who had discussion of risk of opioid use with prescriber
ATTRIBUTE 2: PROFESSIONALLY ACCOUNTABLE
MULTIDISCIPLINARY TEAM-BASED CONTINUING CARE THAT IS COORDINATED AND INTEGRATED WITH OTHER SERVICES AND THE MEDICAL NEIGHBOURHOOD
S48: Practice goal/mission	Defined practice mission/goal
	Mission/goal accessible to staff
	Mission/goal accessible to patients
S49: Practice profile	Total number of staff in each professional category including FTE
S50: Data sharing with local hospitals	Able to receive electronic discharge summary
	Able to receive data in real time e.g. shared EHR or real time electronic shared care plan (blue sky)
S51: Data sharing with other health care providers	Practice has GP system for notification of specialist and allied health care correspondence ((blue sky)
S52: Use of My Health Record	% of active patients with Shared Health summaries uploaded to My Health Record
P53: Team-based care	Regular clinical review meetings involving all team members
	Assigned care teams to coordinate care for individual patients (blue sky)
	Reports from each team member in patient file
P54: Care planning	% active patients with chronic disease who had a GP management plan in previous 12 months
	% active patients with chronic disease who had a medication management review (HMR) in previous 12 months
O55: GP and staff satisfaction	Survey measuring GP and staff satisfaction with: enjoyment of work, impact on local community health, safety in work, income from work, time with patients, work/life balance
O56: Patient experience of continuity of care (blue sky)	PREM questions on time taken for the notification of abnormal test results (blue sky)
O57: Care plan engages patient	PREM questions on experience with care planning
	PAM® scores
O58: Follow-up following hospital attendance (blue sky)	% of active patients reviewed following ED presentation within 7 days (blue sky)
	% of active patients reviewed following admission within 3 days (blue sky)
CLINICAL GOVERNANCE
S59: Clinical governance systems in place	Practice currently accredited according to RACGP or ACRRM standards
STAFF TRAINING
P60: Regular staff education undertaken	Number of meetings/attendances recorded
P61: Assessment of learning needs	Evidence of process for assessment of learning needs
DATA-ENABLED PRACTICE QUALITY IMPROVEMENT
S62: Data quality and completeness of demographic and key health data	% active patients with date of birth recorded
	% active patients with gender recorded
	% active patients with allergy or ‘nil known allergy’ coded in patient records
P63: Improving the quality of our practice	Evidence of work on data cleansing
	Data reports and date of most recent report
	Evidence of formal review of the collected data
O64: Consumer satisfaction with quality (blue sky)	Analysis of validated survey responses (blue sky)
EDUCATION, TRAINING AND RESEARCH TO SUPPORT QUALITY AND SUSTAINABILITY
S65: Registered for postgraduate GP training	Accredited as training practice with local RTO
P66: Engagement with student training	Number of medical, nursing and allied health students undertaking placements in previous 12 months
P67: Research activity (blue sky)	Evidence of engagement with research or PDSA activities (blue sky)
ATTRIBUTE 3: ACCOUNTABLE TO THE COMMUNITY
S68: Urgent access to care	Provides same day appointments
S69: Access to non-face-to-face care e.g. telephone, email	Process documented and advertised to patients for phone/email access
S70: Patient demographics recorded	% active patients with cultural and linguistic status recorded
	% active patients who identify as Aboriginal and/or Torres Strait Islander
	% active patients with Aboriginal and/or Torres Strait Islander status coded in patient records
	% active patients ≥16 years with Australian Government health care card
S71: Meets the needs of Aboriginal and/or Torres Strait Islander patients	Practice registered for PIP Indigenous Health Incentive
S72: Health related social needs assessed	% active patients with screening for health-related social needs recorded (blue sky)
S73: Community engagement	Practice has community/patient advisory structures
P74: Provides health care to vulnerable communities	Bulk billing for Australian Government health care card holders
P75: Meets the needs of CALD communities	Provides bilingual services as required
O76: Access to regular primary care provider (as measured in response to PREMs)	% active patients reporting they have a specific GP/ practice nurse/ care team (blue sky)
	% active patients reporting difficulties obtaining care in previous 12 months
	% active patients reporting same day response to phone call to GP/ nurse
O77: Access for low SES	Compare % active patients who are Australian Government health care card holders with % holding Australian Government health care cards in practice LGA (blue sky)
ATTRIBUTE 4: ACCOUNTABLE TO SOCIETY
O78: Avoidable hospital care (blue sky)	Use of linked data to measure potentially preventable hospital admissions (blue sky)
O79: Duplication of care (blue sky)	Use of linked data to measure duplication of pathology and radiology services (blue sky)

S = structural indicators measuring organisation factors that define the health system including material resources (e.g. facilities, equipment, money), human resources (e.g. number and qualifications of staff) and organisation structure (e.g. staff organisation, methods of reimbursement); P = process indicators measuring what is actually done in giving and receiving care and can also be thought of as activities; O = outcome indicators measuring the effect of care on populations and patients.

Table 3

High-quality general practice attribute framework, their alignment with the quadruple aim and the number of indicators and measures under each attribute.

Attribute	Definition [11]	Aligning with Quadruple Aim [16]	Number of indicators and measures
Attribute One: Accountability to our patients	High-quality general practice provides evidence-based, person-centred and comprehensive care (including preventive, chronic and acute care), with patient-general practice team partnerships as a key aim.	Improving the individual experience of care	47 indicators with 79 measures
Attribute Two: Professionally accountable	High-quality general practice is:◾high-functioning multidisciplinary teams engage in continuing care that is coordinated and integrated with other services and the medical neighbourhood;◾supported by clinical governance, staff training and data-enabled practice quality improvement;◾engaged with general practice education and/or research to provide a means of sustaining the quality of the health system.	Improving the work life of clinicians and staff	19 indicators with 31 measures
Attribute Three: Accountable to the community	High-quality general practice is accessible, responsive to population health needs and focussed on providing equitable care.	Improving the health of populations	10 indicators with 16 measures
Attribute Four: Accountable to society	High-quality general practice promotes efficient stewardship of health resources.	Reducing the per capita costs of care for populations	2 indicators with 2 measures

Survey format

Three rounds of online surveys will be administered using the Qualtrics platform. (Qualtrics, Provo, UT, USA. https://www.qualtrics.com). The online survey has been constructed and pilot-tested for comprehension and adequate functioning of the survey set up. Unique links to each round will be emailed to participants on the morning that it is officially opened. Each round will take around 20 to 30 minutes to complete and will remain opened for three weeks. Results will be analysed at least two weeks in between rounds. Participants will receive up to three email reminders to complete each round before it closes.

Rating process

Participants will be asked to rate each indicator and measure for relevance and feasibility in three rounds of the online survey. Relevance is defined as the value and appropriateness of an indicator/measure in Australian general practice. Participants will be asked to rate on a 4-point Likert scale: 1 irrelevant; 2 somewhat irrelevant; 3 somewhat relevant; 4 relevant. Feasibility is defined as the applicability and implementability of an indicator/measure in Australian general practice. Participants will be asked to rate each indicator/measure on a 4-point Likert scale– 1 infeasible; 2 somewhat infeasible; 3 somewhat feasible; 4 feasible. Text boxes will be available for participants to provide comments, including recommendations for additional indicators or measures, for each subgroup of indicators.

The flow of the Delphi study rating process is shown in Fig 1. In Round 1, participants will initially be asked to provide demographic information including their name, age, gender, job position, and number of years of experience. They will then be asked to rate the indicators and measures under Attribute 1. In subsequent rounds, only names will be requested to match participants’ responses in the various rounds. In Round 2, they will be presented with items that did not reach consensus in Round 1, and given the opportunity to change their previous responses if they wish to do so. They will then be asked to rate the indicators and measures under Attributes 2, 3 and 4 and to provide comments as before for each subgroup of indicators. In Round 3, they will similarly be presented with items that did not reach consensus in Round 2, and given the opportunity to change their previous responses if they wish to do so. In this last round, as the final list of indicators and measures emerges, participants will be presented with a summary of any suggestions or qualitative responses from rounds 1 and 2, and asked open questions regarding their views and suggestions on the implementation of a quality indicator tool in Australian general practice.

Fig 1

Delphi survey rating process.

The levels of consensus in the Delphi methodology vary depending on size of the expert panel and the aim of the research [24, 25]. Consensus target for this study are defined ‘a priori’ based on previous research experience [26]. Each indicator (average score of its measures) and measure will require a minimum of 70% agreement (combined scores of 3 and 4) in both relevance and feasibility to achieve consensus. We determined that this threshold target and approach to be pragmatic and reasonable for establishing consensus across diverse and complex general practice settings.

Data analysis

Quantitative data

Participants’ demographics will be analysed descriptively using Microsoft Excel software. The aggregate results of the participants’ responses will be analysed for percentage response rates, percentages for each level of agreement for each measure, medians, interquartile ranges and their associated group rankings [27].

A measure will require at least 70% in both relevance and feasibility to achieve consensus. Score of 1 and 2 will be collapsed as irrelevant or infeasible, and scores of 3 and 4 will be collapsed as relevant or feasible. If an indicator or measure achieves ≥70% in relevance but not feasibility, it will be included in a ‘blue skies’ category for future consideration. If an indicator or measure achieves ≥70% in feasibility but not in relevance, it will be discarded. Sub-analysis of the individual scores 1, 2, 3 and 4 will also be conducted to help us understand better the strength of the consensus.

Qualitative data

Participants’ responses in the text boxes will be analysed thematically. They will be imported into the NVivo analysis software and coded using a mix of inductive and deductive approaches [28, 29]. Patterns will then be identified from the codes and grouped according to the accountability attributes (deductive approach) as well as to elicit new themes (inductive approach). The research team will separately and collectively analyse the data and resolve any differences in interpretation.

Data management plans

The types of data that will be produced include demographic data collected on participant consent forms in MS Word/PDF format and electronic survey data. A MS Excel spreadsheet will be created in which participant names will be assigned a number. Participant numbers will be used in place of participant names in naming participant data files for the duration of the project. Survey files will be named using the participant’s number, the survey number and the date e.g. Participant1_survery1_190521.

Digital data will be stored on the Western Sydney University’s OneDrive system. PL is the administrator and the only person able to provide access to other team members. The only team members with access are PL, SR and JR.

Non-digital data, if any, will be scanned and stored with the digital data. The original hardcopy documents will be stored in a locked filing cabinet in a locked office at Western Sydney University Campbelltown Campus.

All research data and primary materials will be stored for 15 years and then destroyed in accordance to Western Sydney University protocols.

Potential risks and risk management

Potential risks related to this project include those that may be internal or external. Survey participants may feel inconvenienced by the process required in the study. This includes being required to read the project information, sign the consent form and complete three rounds of survey. To manage these risks the project aims and purpose will be clearly explained to the participants who are experts familiar with Australian general practice quality improvement initiatives. The study will be of inherent interest to them. It will also be styled to allow easy completion and participants will be able to save their responses and return to them later.

Some participants may be concerned about confidentiality. Although participants will be asked to provide demographic information, their identity and information will be blinded to other survey participants and the PHNs. As detailed above, they will be provided anonymity with a random participant project number that will only be able to be linked to their identifiable information by the research team.

External to the project, risks include the current COVID-19 pandemic and government restrictions on movements. These restrictions and the workload of vaccine roll-out may potentially affect recruitment and participation as PHNs and general practices are directly involved in pandemic prevention and control. The recruitment and survey timeline will be flexible to accommodate any unforeseen interruptions. Each round may also be opened for a longer period if necessary.

Ethical considerations

This research has ethics approval from Western Sydney University Human Research Ethics Committee (ID H14460). Participants will be required to provide written consent before round 1 of the survey.

Status and timeline

At the time of manuscript submission, the research has just commenced recruitment of participants. Tentative timeline is outlined in Table 4.

Table 4

Project tentative timeline.

26th October to 25th November 2021	Recruitment
26th November to 17th December 2021	Round 1
18th December 2021 to 13th January 2022	Holiday seasons break
14th January to 4th February 2022	Round 2
18th February to 11th March 2022	Round 3

Discussion

This study protocol describes the research design for a Delphi study to obtain opinions and reach consensus from experts on a core set of relevant and feasible high-quality performance indicators and measures from a suite of indicators and measures previously developed by the researchers in partnership with PHNs in Western Sydney [11]. This protocol will add to the current knowledge of the translation of performance guidelines into quality practice and how best to measure and promote high quality in Australian general practice.

Whilst many PHNs work with general practices to collect data for quality assurance purposes, there is no agreed comprehensive tool that could identify, measure and reward high-quality general practice. Some work has been done in PHNs supporting Patient-Centred Medical Home model of care, but the indicators that were used revolved around processes and system requirement for a team-based approach to deliver this model of care [30, 31]. Although very useful, these indicators are specific to the PCMH models and are dependent on the continuation of funding and evolving policies to support this model of care. Australian general practice requires practical and evidence-based indicators and measures of high quality if funding models move to incorporate payment for quality in addition to current throughput payment [32]. Findings from this Delphi consensus study will address the gaps in the literature around establishing consensus on high-quality structural, process and outcome indicators and measures for use across diverse and complex general practice settings, and contribute to the design of an assessment tool that would change how high-quality general practice can be measured and enable future PHC reforms in Australia.

The suite of 79 indicators and their corresponding 129 measures to be evaluated in this Delphi consensus study were derived from robust interrogation of existing literature and extensive consultations with key stakeholders [11]. They are focused on structures, processes and outcomes of care. This Delphi study will enable consideration of their relevance and feasibility within different general practice clinical settings where multi-morbidities and complex interventions are common and the constraints of providing health services are unique. Opinions from our participants will inform and guide the implementation of the developed tool in the real world.

There is growing interest in the processes required to establish assessment tools to identify high-quality health care and service performance. The Delphi technique is appropriate to develop consensus between the diverse stakeholders and experts in the Australian general practice setting because of its flexibility and ability to offer anonymity to participants. It has the benefit of being able to minimise bias from dominant experts compared to other consensus development methods. It provides a platform to canvass suggestions and opinions on implementation of the tool to measure improvement in individual practices and considerations required for specific contexts including cultural and socio-economic factors that may impact achievement of quality indicators. Additionally, the provision of opportunities for participants to review results from previous rounds and to revise their responses is a unique characteristic of the Delphi technique to enable the determination of consensus. A disadvantage of the Delphi technique, however, is that it does not involve direct interactions with the participants and may limit their ability to generate ideas during the consensus process [33]. Another limitation of this study is that it is designed specifically for the Australian context and may not represent the setting and conditions of other countries.

Using four high-quality general practice attributes that reflect the Quadruple Aim as a framework in this Delphi consensus study will help us to focus on the design of an assessment tool that will facilitate high-quality general practice delivery. The application of scoring criteria for approval for each consensus statement is also expected to ensure the relevance and feasibility of the final core set of indicators and measures.

Another strength of the study is the broad representation of Australian primary health organisations and diverse backgrounds of the participants involved. However, the diverse medical and non-medical participant populations with different perspectives and priorities may confound the results. If that is the case, we will be able to differentiate the stakeholder groups and analyse accordingly to identify and understand the different perspectives.

Although we have involved only PHN and general practice experts in this consensus development process, we plan to engage with primary health care consumers and Aboriginal and Torres Strait Islander health and justice health sectors separately in focus groups to explore their views on indicators and measures applicable to the final quality improvement tool. Through this Delphi consensus study, QUEST PHC will provide valuable information to guide future research and quality improvement activities in these diverse settings.

7 Mar 2022

29 Mar 2022

Dear Editor,

On behalf of my co-authors, I submit below our response to the reviewers. We thank both reviewers for their time. We greatly appreciate it.

Regards,

Phyllis Lau, 15th March 2022

Additional Journal Requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

AUTHORS' RESPONSE:

We have now adjusted our formatting to comply with PLOS ONE style.

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

The authors would like to acknowledge the funding body and the contribution of the Project Control Group: Digital Health CRC, Brisbane North PHN, Central and Eastern Sydney PHN, Nepean Blue Mountains PHN, North Western Melbourne PHN, South Western Sydney PHN, WentWest, Western Australia Primary Health Alliance, and Western NSW PHN.

Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

This study is funded by the Digital Health Cooperative Research Centre https://www.digitalhealthcrc.com/. The funding body is part of the Project Control Group which oversees the conduct of the study, including the design of the study and collection, analysis, and interpretation of data and the writing the manuscript.

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

AUTHORS' RESPONSE:

Funding information now deleted from the Acknowledgements section.

The reference to the funder now deleted from line 156.

Funding statement now deleted from the manuscript.

We have corrected a typo in the funding statement. A funding statement was already in the cover letter, but we have now amended to below in the cover letter so it is clearer – “This study is funded by the Digital Health Cooperative Research Centre https://www.digitalhealthcrc.com/. The funding body is part of the Project Control Group which oversees the conduct of the study, including the design of the study and collection, analysis, and interpretation of data and the writing of the manuscript.”

3. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

AUTHORS' RESPONSE:

We have reviewed our reference list and made appropriate edits. It is now complete and correct.

Reference 11 has been replaced – a peer reviewed paper describing findings reported in the previous reference has been accepted for publication.

Reference 23 has also been replaced with the reference suggested by reviewer 1 in point e. This reference better supports a more evidence-based recommendation of a minimum number of panel members in Delphi studies involving the selection of healthcare quality indicators.

Reviewer #1:

I read this manuscript about the protocol for a Delphi study to select indicators of high-quality general practice in Australia with great interest. I would like to thank the authors for addressing this important topic and the editors for giving me the opportunity to review this protocol.

There are many strong points to this study, but I will focus my comments on issues that I believe the authors should consider to possibly improve it further. They are presented in order of occurrence in the manuscript and not of importance. Even though they are numerous, I believe most are minor and easily fixable. I was generous in my comments to reflect my interest in the methodology adopted and the chosen topic.

AUTHORS' RESPONSE:

We are grateful and thank reviewer 1 for their comments and interest in our work.

*Regarding point 2 - Soundness of the protocol (Minor and major points):

a) The authors mention in the abstract and main text that they will compute weighted Kappas as part of their statistical analyses. However, they provide little detail as to how these values will be used to influence the study process. Weighted kappas are normally used to assess inter-rater agreement between two raters when the data has an order structure (such as in ordinal data). The reference cited in their manuscript (#27, Holey et al. 2007) used weighted kappas to assess intra-rater (within-subject) agreement as an indication of stability of a participant's responses between two rounds. I feel this would provide little added value in the present study. Because items will only ever be assessed over a maximum of two rounds, it won't be possible to determine whether stability increases or decreases over subsequent rounds as a criteria to stop the Delphi process. Furthermore, it is expected in Delphi studies that some respondents will change their views based on the feedback provided during the second round, so I wonder how the values would be interpreted meaningfully in this case. Finally, I also have reservations about such a use of the kappa statistic because it violates its assumption of independent ratings (ratings from the same individual are expected to be correlated to a certain degree). The authors should consider removing this statistic from their analyses or better explaining how it will be used/interpreted during the research process.

AUTHORS' RESPONSE:

We have considered reviewer 1’s comment about the Kappa statistics. We agree that there are questions about the value of these statistics. We have therefore removed them from line 60 in the Abstract and line 263.

b) I have not been able to access reference #11 (QUEST PHC Project Report) online. Perhaps providing more methodological details as to how the indicators and measures were produced/derived would be of interest to readers. This is briefly alluded to in the Discussion, but I think it deserves more attention either in the Introduction or in Materials and Methods.

AUTHORS' RESPONSE:

As indicated in our response to Additional Journal Requirement 3 above, we have replaced the report in reference 11 now with a recently accepted publication, currently in press.

We have added to the methodological details (as suggested by reviewer 1) in paragraph starting line 111 – “In 2020, Western Sydney University, in partnership with PHNs in the western Sydney region, conducted a literature review to identify evidence-based indicators and measures, then assessed these in three workshops with general practitioners (GPs), practice managers, nurses, consumers and PHN staff in the western Sydney region.[11]”

c) I also wonder whether all of the indicators in this study are brand new or if some of them are already in use in Australia within other measurement frameworks, such as the PCMH model (there is an indirect allusion to this around lines 330-334)? If so, then surely these indicators are "feasible" in an Australian context and participants could be prevented from needlessly assessing this dimension for them?

AUTHORS' RESPONSE:

Yes, some of the indicators in the QUEST PHC suite are already currently being collected. However, our assessment of ‘feasibility’ is not assessing whether the indicators and measures are possible to use, rather we are assessing the ‘applicability and implementability’ of an indicator/measure in Australian general practice. It is also worth noting that the PCMH model has largely been studied in the USA and there has been little evaluation conducted in an Australian context.

When considered in amongst a larger set of indicators and measures, the experts’ judgments on feasibility may differ in a context of high-quality practice and improvement. We think it is important to subject all the indicators and measures to the same assessment criteria.

d) The overall aim of this study, as stated in lines 137-139, mentions the development of a "professionally endorsed" tool. To me, these strong words do not resonate well with the consensus threshold (70%) that was selected. Close to a third of respondents could disagree with items that will be included in the tool. Aiming for a higher level of consensus may better convey the idea of professional endorsement. Perhaps the authors should consult with members of their target experts to determine the threshold needed to ensure the legitimacy of their tool? Or at least justify a bit more why this threshold was chosen in consideration with the context and aims of the study? See also point h) below for another issue related to the relatively weak consensus threshold adopted.

AUTHORS' RESPONSE:

Thank you for this comment. We consider the 70% threshold to be a reasonable target for establishing consensus across diverse and complex general practice settings. Furthermore, we are aiming for 70% agreement in both relevance and feasibility which is effectively consensus in two criteria, not just one.

We have added our justification to line 254 – “We determined that this threshold target and approach to be pragmatic and reasonable for establishing consensus across diverse and complex general practice settings.”

We have responded further on this issue in point h.

e) Regarding sample size (lines 174-178), I would like to point out that although eight participants can be enough for many Delphi studies, it is unlikely to be the case in this particular project given the aims (e.g., professional endorsement) and diversity of relevant perspectives involved in the subject matter (quality of primary care). A better reference number can be taken from a systematic review published in PLoS One, which found that the median number of panel members in Delphi studies involving the selection of healthcare quality indicators was 17 (see Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476).

Also, I believe there is an error in the calculation for the minimum response rate to achieve in each round to obtain 8 respondents from an initial pool of 80 recruited (line 178): the reported rate of 32% works for a two-round process, not three. For three rounds, a minimum response rate of 46% is required in each round (although presenting the information in this way is somewhat misleading in this study's case because not all items are assessed during all rounds). We agree with reviewer 1 about the sample size. We have amended the sentence in line 180 – “A minimum of 17 participants is the recommended minimum sample size for content validity in Delphi studies involving the selection of healthcare quality indicators.[23]” and have also replaced ref 23 with Boulkedid et al 2011.

AUTHORS' RESPONSE:

We thank reviewer 1 for taking the time to check our numbers. However, we believe our calculation was correct. The retaining of participants needed to occur only in rounds 2 and 3 ie only two rounds, not three. To achieve a final number of eight respondents and if we start off with 80 participants in round 1, we need to retain at least 32% (ie 25-26 participants) in round 2 and again 32% (ie 8-9 participants) in round 3.

However, since our target now is minimum of 17 participants, we will need to retain at least 47% (ie. 37-38 participants) in round 2 and and again 47% (ie. 17-18 participants) in round 3. We have therefore amended the sentence in line 182 – “In order for this Delphi study to meet the minimum sample size requirement, we must achieve a minimum of 47% retention rate in rounds 2 and 3.”

We do not agree with reviewer 1 that this is misleading as the final respondents retained in round 3 would have rated all 79 indicators with the corresponding 128 measures in the QUEST PHC suite.

f) Around line 212: It is not specified whether and how many reminders are planned to be sent to participants to maximize response rates. This can make a big difference as we often see surges of responses shortly after sending a reminder.

AUTHORS' RESPONSE:

We agree with reviewer 1 and the reminder process is actually in our ethics approval. We have now added to line 219 – “Participants will receive up to three email reminders to complete each round before it closes.”

g) Around lines 231-233: The authors should specify what information will be fed back to participants in subsequent rounds of rating. Feedback should not only include quantitative but also qualitative information. I have seen Delphi studies which only use quantitative feedback. In these studies, the only "reason" for participants to change their opinion is to conform to the majority, which goes against the Delphi principle of avoiding peer pressure.

AUTHORS' RESPONSE:

We were very clear in our manuscript that participants will be “presented with items that did not reach consensus” in previous rounds. We do not agree that “the only reason for participants to change their opinion is to confirm to the majority” if only the quantitative ratings were presented to them. Our participants are clinicians, managers and PHN staff considered ‘experts’ in general practice quality improvement. We rely on their expert opinions in this study. For someone to review and potentially revise their opinion, informed by the opinions of others, does not imply reacting to peer pressure. The Delphi process simply provides our experts the opportunity to re-evaluate each indicator/measure while considering their peers’ responses. That is the crux of the Delphi process.

In round 3, however, we do intend to present relevant qualitative feedback from the previous rounds. We have added to line 244 – “…as the final list of indicators and measures emerges, participants will be presented with a summary of any suggestions or qualitative responses from rounds 1 and 2, and asked open questions regarding their views and suggestions on the implementation of a quality indicator tool in Australian general practice.”

h) Line 245: I believe that collapsing scores of 3 and 4 for acceptance of items further weakens the impression of consensus that will emerge from this study --- a point also related to my point d) earlier. As currently planned, half of the response scale's categories (2 out of 4) would be viewed as providing support to an item. In comparison, typical Delphi studies with response scales of 9 categories only consider a third of them (range 7-9) as support for an item. I feel that a score of 3 reflects rather mild support vs. 4 which is stronger support. By combining these categories, items with large differences in their overall level of support are likely to be included in the tool as reaching consensus. Such combination also disregards the capacity of respondents to discriminate items that is implied by the full response scale. The authors may wish to reconsider this methodological choice, e.g. by using a dual threshold that includes a minimum proportion for scores of 4 only as well as a minimum proportion for combined scores of 3 or 4. For more consideration into this issue and its consequences on the study results, see De Meyer D, Kottner J, Beele H, Schmitt J, Lange T, Van Hecke A, et al. Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection. J Clin Epidemiol. 2019 Jul;111:23–31. Thank you for your comment.

AUTHORS' RESPONSE:

We have reviewed the reference suggested by reviewer 1. The De Meyer et al paper compared the inference of a 9-point and a 3-point rating scales on consensus development in one Delphi study. They concluded that “the format of rating scales in Delphi studies for core outcome set development and the definition of the consensus criteria influence outcome selection”.

When designing the Delphi study, we too considered the format of the rating scales om the context of the diverse and complex general practice settings as well as the diverse participants, including clinicians and non-clinicians, who we will be inviting. We also considered the burden of asking participants to rate 79 indicators and their corresponding 128 measures. As we alluded to in our response to point d, our approach is more pragmatic than theoretical. We consider the 70% threshold, the using of 4-point Likert scale and the combining of scores 3 and 4 to be realistic and reasonable. In our analysis, however, we also intend to differentiate the scores 1, 2, 3 and 4 to help us understand better the strength of the consensus.

We have added this to line 270 – “Sub-analysis of the individual scores 1, 2, 3 and 4 will also be conducted to help us understand better the strength of the consensus.”

i) Lines 251-252: Although they are frequently used in Delphi studies, means and standard deviations are generally not appropriate for ordinal data also unlikely to be normally distributed. This is especially the case with narrow response scales such as the one used in this study (wider scale can sometimes approximate interval data). Medians and interquartile ranges should be reported instead.

AUTHORS' RESPONSE:

Thank you for your comment. We agree with you. We have deleted ‘means’ and ‘standard deviations’ from line 59 in the Abstract and line 261, and added ‘interquartile ranges’ to the ‘median’ already mentioned.

j) Lines 375-377: I would remove the mention that diversity of perspectives is a potential risk to achieving consensus. It should be seen as a strength, given the complex and comprehensive nature of quality in primary care.

AUTHORS' RESPONSE:

We agree with reviewer 1 that the diversity of perspectives is not a risk. We have removed the word ‘risk’ from line 387 – “However, the diverse medical and non-medical participant populations with different perspectives and priorities may confound the results.”

Nevertheless, many of the indicators/measures are clinical, and clinicians will have very different views to non-clinicians. We will therefore retain the sentence following in line 389 that states our intention to conduct sub-analysis to help us understand the differences, if any.

* Regarding point 4 - Data availability (Minor points):

I am unsure whether the manuscript conforms to the PLOS Data policy. The authors wrote that data from the study planned in this protocol will be made available from the corresponding author on reasonable request and with permission of the study funder (Digital Health Cooperative Research Centre). However, PLOS Data policy states that "it is not acceptable for an author to be the sole named individual responsible for ensuring data access."

Furthermore, the authors do not specify how to obtain permission from the funder (contact information and criteria). However, I am unsure whether this is required for a protocol or only for full study reports.

AUTHORS' RESPONSE:

The corresponding author will not be the “sole named individual responsible for ensuring data access”. QUEST PHC has a project agreement with the Digital Health Cooperative Research Centre (DH CRC). The project IP will be owned by DHCRC legally; however, the authors will have non-exclusive royalty-free right to use the DHCRC IP for purposes including publication and conference presentations.

The role of the corresponding author, we thought, were to be a point of contact for the journal and any questions or requests from readers. Provided the requests for data do not compromise confidential information, are consistent with the purpose of the funded project, and do not prejudice the protection or utilisation of the DH CRC IP, they will be honoured in accordance with our project agreement with DH CRC.

* Regarding point 5 - Language considerations (Minor points):

a) I sugggest using the expression "Delphi study" or "Delphi process" rather than "Delphi survey" in the title and whenever the authors refer to the study type, since Delphi studies include multiple surveys and this could be confusing to some readers. E.g. Lines 2, 6, 198, 207.

AUTHORS' RESPONSE:

Amended references from ‘Delphi survey’ to ‘Delphi study’ wherever appropriate throughout the manuscript.

Please note that in line 213, we have retained the word ‘survey’ as the word is appropriate.

b) Line 14: "Translational Health Research Institute" appears twice in succession.

AUTHORS' RESPONSE:

Deleted repetition.

c) Line 78: First instance of the "PHN" abbreviation should be defined.

AUTHORS' RESPONSE:

‘PHN’ expanded.

d) Line 87: It looks like a verb is missing in this sentence? It should maybe read "The 31 PHNs [were] established in 2015 across Australia for supporting (...)"

AUTHORS' RESPONSE:

Amended as suggested.

e) Line 274: I think there is a typo in the e.g. of the survey file ("survery1" should probably read "survey").

AUTHORS' RESPONSE:

It was not a typo. This is the nomenclature we are using to name our files.

f) I don't know if this is a misunderstanding on my part or a typo, but in Table 2, the measure "PAM scores" is presented as blue sky for the indicator O4 but not for the indicator O57. How can it be considered difficult to implement for one and not the other?

AUTHORS' RESPONSE:

Thank you to reviewer 1 for the pickup. It is an error – the PAM scores should not be ‘blue sky’. This is now corrected in Table 2.

This manuscript has taken a long time to be reviewed, and data collection for the survey is already underway at the time of writing this response. We will note this error to participants in the final Round 3.

* ADDITIONAL COMMENTS FOR THE AUTHORS

Line 130: "Subsequent consultations will be held with consumers (...)". It is now increasingly common to seek involvement of consumers, patients, and communities as early as possible in the research process rather than merely at the later stages. There is evidence that professionals and patients have different priorities regarding quality improvement in primary care (e.g., see Boivin A, Lehoux P, Lacombe R, Burgers J, Grol R. Involving patients in setting priorities for healthcare improvement: a cluster randomized trial. Implement Sci. 2014 Feb 20;9:24. ). I encourage the authors to seek to involve them as early in their process as possible so that they have the opportunity to shape it just as much as professionals.

AUTHORS' RESPONSE:

We thank reviewer 1 for this comment. We agree that early engagement of consumers is very important. The QUEST PHC suite is a broad set of indicators that include both clinical and patient measures. We engaged with consumers at the outset of this work and they advised that whilst they understood and supported the need for clinical measures, they were most interested to be involved in the development of Patient reported measures. We have recently completed a literature review to identify patient-reported measures (PRMs) that are relevant to Australian general practice. At the time of writing this response, we have already conducted focus groups with consumers to explore their views on PRMs.

The sentence starting in line 135 has now been amended to “Consultations have been held with consumers, with regards to key patient-reported measures (PRMs).”

I fully understand that discussing the indicators and measures to be assessed in this study is outside the scope of my mandate as a reviewer of this protocol, but I found it unfortunate that, in Table 2, the only (two) indicators and measures for Attribute Four (Accountable to society) were labeled as blue sky. Could items related to reducing unnecessary care procedures (e.g., Choosing Wisely) have also been considered here? Would this deserve at least some discussion in the main text?

AUTHORS' RESPONSE:

The development of the indicators and measures were informed by extensive literature reviews and consultations with primary health networks and clinicians. We accept that not all important indicators are considered ‘practical’ by all stakeholders. The use of the ‘blue sky’ label is precisely to preserve such indicators until such time the environment is right for their use.

We have amended the sentence that mentions this in line 127 to be clearer – “…include some “blue sky” measures considered difficult to currently implement but are nonetheless important.”

Finally, around line 336, the authors indirectly imply that the indicators/measures achieving consensus may eventually be linked to payment in reformed funding models. Are Delphi participants made aware of this? Their judgments on relevance and feasibility may differ whether they consider that the items will be used reflexively in a context of continuous quality improvement or for external sanctioning from pay-for-quality/performance schemes.

AUTHORS' RESPONSE:

Yes, participants are made aware.

Our Participant Information Sheet clearly states that “This research will provide support for use of these measures to improve care in general practices and potentially inform funding models that would reward high quality care in Australian general practice.”.

In closing, I sincerely hope that at least some of my comments will help the authors improve their manuscript and make their already good study even more robust.

AUTHORS' RESPONSE:

We thank reviewer 1 and have no doubt that their very considered review will make our study more robust.

Reviewer #2:

This protocol paper reports the protocol for a Delphi consensus survey to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia.

The work is very timely and important for Australian Primary Health Care. It is also good to see outcome, as well as process, measures are being considered. Overall I strongly support the researchers' view that it "will add to the current knowledge of the translation of performance guidelines into quality practice and how best to measure and promote high quality in Australian general practice."

The protocol itself is well written and also aspects are presented appropriately. The Delphi process is an appropriate formal consensus method to use.

AUTHORS' RESPONSE:

We thank reviewer 2 for their encouraging and generous comments!

Submitted filename: Responses to Reviewers.docx

Click here for additional data file.

22 Apr 2022

Protocol for a Delphi consensus study to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia

PONE-D-21-34478R1

Dear Dr. Min-yu Lau ,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Marie-Pascale Pomey

Academic Editor

PLOS ONE

12 May 2022

PONE-D-21-34478R1

Protocol for a Delphi consensus study to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia

Dear Dr. Lau:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Marie-Pascale Pomey

Academic Editor

PLOS ONE