A process for developing standards to promote quality in general practice.

BACKGROUND: Since 1991, the Royal Australian College of General Practitioners' (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards are also linked to incentives for general practice remuneration. These Standards were revised in 2017.
OBJECTIVE: The objective of this study is to describe the process undertaken to develop the fifth edition Standards published in 2017 to inform future standards development both nationally and internationally.
METHOD: A modified Delphi process was deployed to develop the fifth edition Standards. Development was directed by the RACGP and led by an expert panel of GPs and representatives of stakeholder groups who were assisted and facilitated by a team from RACGP. Each draft was released for stakeholder feedback and tested twice before the final version was submitted for approval by the RACGP board.
RESULTS: Four rounds of consultation and two rounds of piloting were carried out over 32 months. The Standards were redrafted after each round. One hundred and fifty-two individuals and 225 stakeholder groups participated in the development of the Standards. Twenty-three new indicators were recommended and grouped into three sections in a new modular structure that was different from the previous edition.
CONCLUSION: The Standards represent the consensus view of national stakeholders on the indicators of quality and safety in Australian general practice and primary care.

Introduction

Quality and safety standards are used in health services across the world as a means of promoting excellence in patient care (1). Accreditation against such standards has been shown to promote leadership, enhance corporate culture and improve clinical performance (2). Since 1991, the Royal Australian College of General Practitioners’ (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards comprise a series of indicators grouped into sections that can be assessed.

To press, ~80% of Australian general practices are voluntarily accredited against the Standards (3). Funding of general practice services in Australia is based on a fee for service model administered by the government through Medicare. Accreditation is a prerequisite to access a portion of the payments administered separately as the Practice Incentives Program (PIP). PIP payments incentivize government priorities for general practice (4). Independent agencies accredit general practices against the RACGP Standards. Two surveyors, a GP and a non-GP surveyor with recent and relevant general practice experience and working for one of the agencies, undertake an onsite peer review and assessment of the practice against these standards.

All four previous editions of the Standards were developed by an expert committee in conjunction with the Standards Unit (SU), a business unit within the RACGP, and input via stakeholder engagement, consultation and piloting in general practice. In 2013, the expert committee reviewed the fourth edition Standards and identified areas where significant change was required to (i) maintain currency and reflect contemporary general practice, (ii) improve usability by providing flexibility for all general practices (regardless of location, size or patient cohort) and (iii) extend the framework for quality care and risk management to a variety of settings where primary care services are delivered. The fifth edition Standards were developed over 32 months using a modified Delphi process (5,6). The impact of accreditation against standards as a mechanism for promoting quality has been reported in the literature (7–10). However, there is a gap in the literature describing the process of developing such standards. This article details how the fifth edition Standards were developed in Australia.

Method

A modified Delphi process (5,6) was deployed to draft the fifth edition Standards (Fig. 1). It comprised four rounds of workshops to develop the final version. The process was led by the Standards committee (the expert panel) and the SU, a team employed by the RACGP.

Figure 1.

Modified Delphi process for the development of the fifth edition Standards for General Practices

The expert panel

Members were invited to nominate to the panel through a general call to the RACGP membership or were nominated by relevant peak bodies. A representative expert panel included 10 GPs (with clinical, research and academic experience), a practice nurse, a practice manager and a consumer. Five existing members had also been involved in the development of previous editions of the Standards. Panel members were from most Australian states and practice locations including urban, regional and rural sites. The clinical expertise of the panel included general practice, Indigenous health and emergency medicine. Five members of the panel were also accreditation surveyors with experience in the assessment of the Standards in practice.

The Standards Unit

The SU comprised a manager and project support staff. It supported the expert committee by drafting the Standards, sourcing relevant resources, and engaging with stakeholders including the accreditation agencies.

Modified Delphi process for the development of the 5th edition Standards

Preparation phase (September 2014–February 2015)

The preparation phase comprised five workshops involving the expert panel and the SU. Two of these workshops were facilitated. The workshops canvassed expert opinion on the future models of the Standards and accreditation system; developed the process that would be used to structure the fifth edition Standards; current issues affecting general practice; a review of the available evidence; and an initial analysis and mapping of the fourth edition indicators to the proposed fifth edition structure. The proposed fifth edition structure was available for feedback at the conclusion of this phase and included 124 indicators.

Stakeholder consultation and feedback (February–May 2015)

Consultation and feedback was sought on the existing fourth edition Standards—current issues impacting on the delivery of general practice and the proposed new structure for the fifth edition. This included not only informal commentary from stakeholders but also more formal discussion in the workshops held across Australia. The workshops were facilitated by the expert panel members supported by the SU and local RACGP faculties. The average number of participants per workshop was eight (range of 5–17).

Round 1 (June 2015–April 2016)

The stakeholder feedback received during the first consultation was reviewed by the SU using a traffic light system and presented to the expert panel. Suggestions for change were categorized as green if they were appropriate for inclusion, amber if items required further consideration, or red if the suggestion was deemed inappropriate for inclusion or out of scope. A total of eight workshops were held during Round 1. Where unanimous consensus could not be reached, an issue was put to the expert panel for a vote. This process was used throughout the rounds of stakeholder consultation and workshops.

During Round 1, an evidence review and benchmarking analysis of comparable national and international standards was undertaken by and presented to the expert panel. The SU reviewed accessible standards relevant to the Australian primary health care system from the UK, New Zealand, Canada, Denmark and Australia against the fourth edition Standards to identify gaps. From this process, the expert panel were asked to consider inclusions in the fifth edition Standards. At the conclusion of the workshops, the first draft of the fifth edition Standards was released to stakeholders for feedback.

This preliminary pilot of the draft Standards was conducted concurrently at the end of Round 1. Twelve practices and five surveyors were remunerated to participate in this phase. Although this sample had limited membership, the first pilot aimed to test the following measures for each indicator: feasibility, acceptability, achievability, applicability and ease of assessment. Thresholds were set for each measure as assessed by the surveyors and the practice representatives involved in the mock survey. Indicators needed to be assessed as ‘met’ by a minimum of 75% of the surveyors and practice representatives on feasible, acceptable and achievable measures using a rating of met, not met and partially met; a minimum 50% for ease of assessment using a rating of very difficult, difficult, easy, very easy and not sure. Three of the surveyors were members of the expert panel. The SU held two information sessions (n = 16 attendees) with the surveyors and practices to provide guidance on how the indicators could be assessed.

Round 2 (April–October 2016)

Stakeholder feedback from the first draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as described in Round 1. The expert panel further considered indicators assessed during the first pilot that did not meet the threshold measures. Qualitative feedback from surveyors and practices was also reviewed. Over two workshops, indicators were added, removed and modified to develop the second draft of the Standards. Stakeholders were then invited to provide formal and informal feedback on the second draft.

A larger pilot was then undertaken to field test the second draft using the same measures as Round 1. A workshop and webinar information session were held with the surveyors prior to the commencement of pilot visits. This second field test evaluated the Standards in a variety of general practice settings stratified by rural and urban location; solo, small and large practices based on full time equivalent of GPs; corporate and private business models; and Aboriginal Medical Services. For efficiency, the second pilot involved two concurrent processes: (i) dual-process pilot for practices also undergoing fourth edition accreditation. This process comprised accreditation against the fourth edition Standards and a review of new fifth edition indicators; and (ii) pure fifth edition pilot for practices who have recently completed fourth edition accreditation. This process comprised a mock survey visit and assessment based on the second draft fifth edition Standards. For both processes, practices were required to complete a self-assessment against the draft Standards using an RACGP-developed tool.

Round 3 (November 2016–January 2017)

Stakeholder feedback from the second draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as per Round 1. Indicators assessed during the pilot that did not meet the threshold measures were further considered by the expert panel. One workshop was held during this stage to edit the third draft. A final targeted review of the last 16 revised indicators was conducted. All surveyors and practices who participated in the pilots were invited to undertake the targeted review. For efficiency, a telephone survey was conducted between practices and surveyors. As well as including free text comments, practices and surveyors used the rating scale noted above when reviewing the revised indicators.

Round 4 (January–March 2017)

The feedback from the targeted review was reviewed by the SU and presented to the expert panel for consideration. Final edits were made to the third draft. One workshop between the SU and the expert panel was held during this stage.

Consensus (March–April 2017)

The final draft of the fifth edition Standards and supplementary resources approved by the expert panel was presented to the RACGP board for endorsement.

Results

The project was completed over 32 months.

Preparation phase

The revised structure of the fifth edition Standards was developed in the preparation phase (Fig. 1). A modular structure for the fifth edition Standards was adopted by expert panel consensus.

Round 1

A total of eight workshops were conducted (Fig. 1; Round 1). During these workshops, the expert panel reviewed all feedback presented by the SU. Consultation yielded feedback from 212 stakeholders comprising 106 questionnaire responses, 44 submissions, 58 workshop attendees and 4 verbal feedback responses. The expert panel developed 19 new indicators and amended 42 existing indicators with the intention to remove duplication and facilitate the move to outcomes-focused indicators (Table 1). The number of indicators that did not meet set thresholds in the first pilot is shown in Table 2 and was reviewed by the expert panel in the second round.

Table 1.

Summary of the development process of indicators for the fifth edition Standards for General Practices

Rounds	Changes to the standards
Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase.	Inclusions	Exclusions	Modified
1 n = 8 workshopsGeneration of first draft	adding new indicators in the fifth edition Standards (n = 19)	Removing duplication (n = 8)	Merging of fourth edition indicators that shared a similar theme (n = 42)
2 n = 2 workshopsGeneration of second draft	0 indicators added	0 indicators removed	4 indicators amended after consultation phase and first pilot
3 n = 1 workshopsGeneration of third draft	3 indicators added	0 indicators removed	1 indicator amended after consultation phase and second pilot
4 n = 1 workshopsGeneration of final draft	0 indicators added	0 indicators removed	The wording of 16 indicators refined after targeted review
Consensus	Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators.

Table 2.

Indicators that did not meet the thresholds in the first pilot

Threshold	Number of indicators	Percentage range
Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable	7	58–75
Indicators assessed as ≤75% met	3	58–75
Indicators that are ≥50% difficult to assess	4	50–58

Round 2

The results of the first pilot and consultation on the draft Standards were presented to the expert panel and considered over two workshops (Fig. 1; Round 2). Over 80 stakeholders had provided feedback on the first draft. In total, 330 separate items of feedback were received. The feedback was comprised of 25 submissions, 25 email responses, 22 workshop attendees and 5 other responses (including verbal communications). The feedback could be broadly categorized into expansion, deletion and amendments to the Standards and potential challenges to implementation. The SU analysed the feedback using the traffic light system and presented to the expert panel for review. The expert panel amended four existing indicators in response to piloting and stakeholder feedback (Table 1). The subsequent second draft was piloted in 66 practices, with 34 GP surveyors and 29 co-surveyors completing the second pilot visits. Indicators that did not meet set thresholds for the second pilot (Table 3) were reviewed in Round 3.

Table 3.

Feedback from GP surveyors and co-surveyors on indicators that did not meet thresholds in the second pilot

Threshold	Co-surveyors		GP surveyors
	Number of indicators	Percentage range	Number of indicators	Percentage range
Indicators assessed as ≤75% met	9	52.4–73.8	15	34–73.6
Indicators that are ≥50% difficult to assess	9	46.6–50	15	17–50

Round 3

As per Round 2, the SU presented feedback from the second pilot and consultation on the second draft to the expert panel. This comprised 650 individual items of feedback from over 56 stakeholders, including 25 organizations and 29 individuals and medico-legal commentators. Public consultation ceased at the end of Round 3. Evaluation of feedback by the expert panel resulted in the generation of the third draft of the Standards (Table 1). Indicators identified as not meeting threshold in the second pilot were evaluated further in a targeted review (Fig. 1; Round 3).

Round 4

A total of 34 responses were received from the targeted review. They comprised 5 telephone interviews, 9 email responses from surveyors and 20 email responses from practices. Of the 16 revised indicators, 6 required further consideration by the expert panel. All of these indicators were edited in the third draft.

Consensus

Edits of the third draft were conducted by the SU and expert panel to produce the final draft of the fifth edition Standards. Panel members were required to make a recommendation in the affirmative or negative and where consensus was not clear the decision was put to a vote. There were three occasions where issues were put to a vote following robust discussion and differences of opinion.

Discussion

The fifth edition Standards were endorsed by the RACGP board and at the time of writing have been adopted as the RACGP Standards for general practice and primary care in Australia. Whilst elements of the Standards development process have been captured in some of the literature, no comprehensive description of standards development in health care has been published to our knowledge (11–17). The fifth edition Standards were developed using the Delphi technique as deployed in other settings (18–20) with common elements including consensus, expert opinion, identifying opportunities and problem areas to create new standards, developing iterative drafts, piloting and testing (21–25). As described earlier, the approach is an iterative process, seeking and incorporating feedback and enhancing the scope for widespread consultation through as many channels as possible.

Limitations

Many challenges are known to be associated with the technique, including estimating the number of rounds of feedback, creating or maintaining channels for feedback and adhering to a timeframe for completion. However, Delphi techniques have been widely deployed to achieve consensus and can be modified to meet the needs of the project (26–29). It was challenging to facilitate stakeholder engagement to the point of saturation as well as managing multiple stakeholder expectations. Some stakeholders inappropriately expected that their suggestions would immediately be incorporated without further consideration. Maintaining the momentum of all participants for the duration of the project was a vital component of delivering a set of agreed standards on time and within budget. The project timeframe to develop the fifth edition Standards was significantly longer than that allocated to the fourth edition (3 years as opposed to 12 months). This meant developing the latest edition was more costly and labour intensive.

Often the same stakeholders provided feedback in each round. By the end of the process, a saturation of themes was evident and it was apparent that further rounds would not yield new information. At each stage, a summary of feedback was shared with stakeholders. While the heterogeneous composition of the expert panel enabled detailed discussions from varied perspectives, on only three occasions was there a call for a formal vote.

Strengths

Despite the challenges, the fifth edition Standards incorporated 17 fewer indicators than the previous edition. This was considered an improvement by reducing the burden on those who will ultimately be assessed against the Standards. A number of elements within the process were integral to its success. A consistent membership of the expert panel throughout the process ensured expertise and knowledge was retained. Momentum was maintained through effective leadership and maintaining the commitment of the expert panel and SU to achieve objectives. These elements were especially evident when there were differences of opinion on the indicators. Ongoing consistent support of the board for the duration was vital given the scope of the project as well as effective project management.

Conclusion

The production of the fifth edition Standards was led by an expert panel through multiple rounds of consultation and piloting. Through engagement nationwide with GPs and other key stakeholders, the fifth edition Standards were deemed feasible, acceptable to the profession and assessable by accreditation surveyors.

Declaration

Ethical approval: The RACGP National Research and Evaluation Ethics Committee (16-002) granted full ethics approval for this project.

Conflicts of interest: None declared.