Korean Clinical Practice Guidelines: Current Status of Adherence to the RIGHT Checklist.

BACKGROUND: The Korean Academy of Medical Sciences (KAMS) has been utilizing AGREE II to audit the quality of clinical practice guidelines (CPGs) developed in Korea. Monitoring the RIGHT Checklist adherence could help monitor the quality status and discover areas for improvement of CPG development.
METHODS: We included 129 CPGs from the past 5 years and assessed each item of the RIGHT Checklist. STATA version 15.0 was used for statistical analysis.
RESULTS: Among the seven sections of the RIGHT checklist, sections with a full compliance rate over 60% were 'basic information' (65%) and 'background' (66%). The other sections' mean full compliance rates were 'Evidence' 52%, 'Recommendation' 35%, 'Review and quality assurance' 25% and 'Funding, declaration and management of interest' 17%. Sections with a partial compliance rate over 60% were 'Recommendation' (60%) and 'Funding, declaration and management of interest' (70%). Non-compliance was highest in the 'Review and quality assurance' (17%) domain. In comparison between groups 1 (under median group) and 2 (over median group), group 2 showed a tendency to have multi-stakeholder involvement and present sufficient information on financial resources and conflict of interest declarations. For the CPGs developmental methodology aspect, group 2 provided more pertinent information than group 1 about supporting evidence-making and the process from evidence to recommendation.
CONCLUSION: This study evaluated adherence to the RIGHT Checklist of CPGs developed in Korea. It can provide helpful information to develop strategic plans for enhancing the capabilities of developing CPGs in Korea.

INTRODUCTION

The first CPG recorded in the Korean Medical Guideline Information Center (KoMGI), which is operated by the Korean Academy of Medical Sciences (KAMS), was the Korean Guideline for Asthma in 1994.1 However, the development of clinical practice guidelines (CPGs) in Western countries started in the 1980s, increased rapidly in the 1990s, and the number of publications related to CPGs exceeded 1,000 per year in the 2000s.2 As such, Korea has had less history and experiences related to CPG development than Western countries.

To overcome these weaknesses and quickly enhance the ability for development of CPGs by academic societies under the KAMS, the KAMS has formed the committee for guidelines development & management in 2008 and the committee was renamed the Executive Committee for CPGs (ECC). The ECC comprises multidisciplinary experts, including EBM methodology, medicine, nursing, public health, etc., and aims to improve the overall quality of CPG development related to clinical healthcare in Korea. ECC has provided irregular educational programs for CPG developers and disseminates information about the methodology used to develop CPGs to achieve the committee's purpose.3 From 2017, ECC launched multiple formal training courses for Korean CPG developers, and the reporting checklists for CPGs were included in the curriculum since 2018.

ECC developed the AGREE II scoring guide in 2012 and began education programs for AGREE II and its scoring guide from 2013.3 Besides, the ECC launched the KAMS Appraisal System for CPGs in 2013, and some of the medicine or EBM experts who receive AGREE II education each year are appointed as a KAMS certified appraiser after a series of follow-up training courses. The ECC has consistently sought to increase inter-rater reliability and to construct a consensus on the requirements required in each AGREE II item.4 The activities of the KAMS Appraisal System for CPGs was reflected in the development process of the CPG developers. As a result, the quality of the CPGs developed in Korea has been improved to some extent.5 KAMS utilizes AGREE II as an essential means of auditing the quality status of the domestically developed CPGs, reflecting the results observed in a year-long assessment in its KAMS training program for the following year (Fig. 1). With the growing interest in the development of CPGs and the quantitative expansion of published CPGs in Korea,6 the direction of ECC's efforts was not limited to the AGREE II instrument and has also gradually changed in the direction of strengthening the methodologies.

Fig. 1

Assessment-driven Improvement Strategy of the Executive Committee for Clinical Practice Guidelines of Korean Academy for CPGs Development in Korea.

CPG = clinical practice guideline, KAMS = The Korean Academy of Medical Sciences.

As reporting standards for CPGs, there are currently two checklists in the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network for newly developed CPGs: the AGREE (Appraisal of Guidelines, REsearch and Evaluation) Reporting Checklist and the Reporting Items for Practice Guidelines in Healthcare (RIGHT) Checklist.7 The AGREE Reporting Checklist was announced by the AGREE research team and each item on the checklist originated and reformatted from the AGREE II and intended to improve the comprehensiveness, completeness, and transparency of reporting in practice guidelines.8 The RIGHT Checklist developed by the RIGHT development group, which consisted of a multidisciplinary international team that included policymakers, methodologists, epidemiologists, clinicians, editors, and consumer representatives, had a purpose of assisting guideline developers when reporting their guidelines, supporting journal editors and peer reviewers when considering guideline reports, and helping healthcare practitioners understand and implement a guideline.9

The AGREE II has been utilized to audit the quality status of domestically developed CPGs. The requirements of the RIGHT and the AGREE Reporting Checklists have a lot in common, but they differ in some aspects. For example, the RIGHT Checklist requires specifying CPG in the title but not in the AGREE Reporting Checklist. Moreover, only the RIGHT Checklist requires descriptions related to the epidemiologic background regarding the health problem, how outcomes were selected and sorted regarding the healthcare questions, and the quality assurance process. Additionally, subgroup consideration of target population and developer's information should be provided more evidently in the RIGHT than the AGREE.10 With the reasons mentioned above, the ECC agreed that the RIGHT Checklist might be more transparent and explicit in some critical aspects for CPG developers. Monitoring the RIGHT Checklist adherence could be a useful means to monitor the quality status and discover areas of improvement of CPGs. This study aims to find less compliant areas to the RIGHT Checklist of guidelines developed in Korea from January 2014 to April 2019. ECC believes that assessing the current compliance status with the RIGHT Checklist can give feedback to the ECC's strategic plan and improve existing deficiencies.

METHODS

Inclusion of guidelines

The authors published an article entitled ‘Current Status of Clinical Practice Guidelines in Korea’ in the February issue of this journal.11 We have analysed the level of adherence with the RIGHT checklist for all 129 CPGs included in the last issue. To briefly describe the inclusion process, we collected Korean CPGs in the past five years (from 2014 to April 2019) through several electronic database searches (MEDLINE, EMBASE, and KoreaMed), hand searches, and surveys of academic society memberships from the KAMS. Three authors selected CPGs according to our inclusion/exclusion criteria. The inclusion criteria were as follows: 1) conforms to CPG definitions12 and 2) developed within the last five years (since January 2014). The exclusion criteria were as follows: 1) does not include recommendations, 2) nonmedical fields (dental, nursing, alternative medicine, etc.), 3) CPGs not for Koreans, and 4) if updated, the previous version.

Scoring compliance to the RIGHT Checklist

Three authors (MC, YKL, and SYK) independently assessed CPGs according to each item of the RIGHT Checklist. The RIGHT Checklist consists of 22 topics grouped in 7 sections: basic information, background, evidence, recommendations, review and quality assurance, funding, declaration and management of interest, and other information. Ten topics in the RIGHT Checklist have more than one item. For example, topic 1 (title/subtitle) was divided into 3 items (1a. identify the report as a guideline, 1b. describe the year of publication, 1c. describe the focus of the guideline) that were scored independently. Adherence levels were divided into three levels, which are full, partial or non-compliance, and scored 2, 1, or 0, respectively. Full compliance was defined if the relevant content is sufficiently described anywhere on the CPG, not considering which section should be attributed. Partial compliance was defined if some of the relevant content was described. Non-compliance was defined if the relevant content cannot be found. We did not consider weight difference by items because of the definition and importance of each item for development guideline process. Two assessors were assigned for each CPG and scored the technical precision of the RIGHT Checklist items based on the adherence levels of: 0 (non-compliance), 1 (partial-compliance), and 2 (full-compliance). They then went through the consensus process. If any CPG had been in full compliance for all 35 items, this CPG could score a total of 70 points.

Data analysis

The score was summarized and compliance rate of each items was calculated. The summarized score of guidelines did not showed normal distribution, therefore the median value was calculated. Based on the median value 49, we divided guidelines into two group. Group 1 was classified as under median value and Group 2 was classified as over median value. We compared the compliance rate of RIGHT Checklist between two groups by sections and each item.

Pearson χ2 test was used for comparison of groups, and STATA version 15.0 was used for statistical analysis.

RESULTS

Adherence to the RIGHT Checklist

The total score of each guideline was summarized and the median value was 49 (interquartile range: 43- 54). And the mean average scores by topics of RIGHT Checklist were calculated (Fig. 2). Among the seven topics of the RIGHT Checklist, topics with a full compliance rate over 60% were ‘basic information’ (65%), and ‘background’ (66%). The other topics’ mean full compliance rates were ‘Evidence’ 52%, ‘Recommendation’ 35%, ‘Review and quality assurance’ 25% and ‘Funding, declaration and management of interest’ 17%.

Fig. 2

Mean average score of RIGHT Checklist compliance by section.

Topics with a partial compliance rate over 60% were ‘Recommendation’ (60%), and ‘Funding, declaration and management of interest (70%). Non-compliance was highest in the ‘Review and quality assurance’ (17%) domain. Each topic’s detailed items and results are summarized in Table 1

Table 1

Summary of right checklist compliance rate (Total N = 129)

Variables			2 (Full compliance)	1 (Partial-compliance)	0 (Non-compliance)	Not applicable
Basic information
	Title/subtitle	1a (titlename)	125 (97)	4 (3)	0 (0)	0 (0)
		1b (year)	49 (38)	3 (2)	77 (60)	0 (0)
		1c (focus)	98 (76)	27 (21)	4 (3)	0 (0)
	Executive summary	2 (summary)	19 (15)	103 (80)	7 (5)	0 (0)
	Abbreviation/acronyms	3 (define)	83 (64)	46 (36)	0 (0)	0 (0)
	Corresponding developer	4 (contact)	126 (98)	3 (2)	0 (0)	0 (0)
	Section average		83.3 (65)	31 (24)	14.7 (11)	0 (0)
Background
	Brief description of health problem	5 (basic)	116 (90)	13 (10)	0 (0)	0 (0)
	Aims of guideline and specific objectives	6 (aim)	112 (87)	17 (13)	0 (0)	0 (0)
	Target population	7a (target)	106 (82)	23 (18)	0 (0)	0 (0)
		7b (subgroup)	102 (79)	25 (19)	2 (2)	0 (0)
	End-users and setting	8a (user)	92 (71)	36 (28)	1 (1)	0 (0)
		8b (setting)	72 (56)	54 (42)	3 (2)	0 (0)
	Guideline development group	9a (contributors)	47 (36)	79 (61)	3 (2)	0 (0)
		9b (list)	36 (28)	90 (70)	3 (2)	0 (0)
	Section average		85.4 (66)	42.1 (33)	1.5 (1)	0 (0)
Evidence
	Healthcare questions	10a (PICO)	89 (69)	37 (29)	3 (2)	0 (0)
		10b (outcomes)	59 (46)	67 (52)	3 (2)	0 (0)
	Systematic review	11a (new or adapt)	69 (53)	54 (42)	6 (5)	0 (0)
		11b (existing SR)	36 (28)	36 (28)	2 (2)	55 (45)
	Assessment of the certainty of the body of evidence	12 (evidence)	80 (62)	31 (24)	18 (14)	0 (0)
	Section average		66.6 (52)	45 (35)	6.4 (5)	11 (9)
Recommendations
	Recommendations	13a (clear recommendation)	94 (73)	35 (27)	0 (0)	0 (0)
		13b (subgroup recommendation)	76 (59)	52 (40)	1 (1)	0 (0)
		13c (strength recommendation)	80 (62)	30 (23)	19 (15)	0 (0)
	Rational/explanation	14a (value & preference)	9 (7)	118 (91)	2 (2)	0 (0)
		14b (cost & resource)	5 (4)	123 (95)	1 (1)	0 (0)
		14c (other)	3 (2)	122 (95)	4 (3)	0 (0)
	Evidence to decision process	15 (consensus)	49 (38)	66 (51)	14 (11)	0 (0)
	Section average		45.1 (35)	78 (60)	5.9 (5)	0 (0)
Review and quality assurance
	External review	16 (independent review)	41 (32)	67 (52)	21 (16)	0 (0)
	Quality assurance	17 (QA)	23 (18)	83 (64)	23 (18)	0 (0)
	Section average		32 (25)	75 (58)	22 (17)	0 (0)
Funding, declaration and management of interest
	Funding source and roles of the funder	18a (source)	31 (24)	89 (69)	9 (7)	0 (0)
		18b (roles of funder)	30 (23)	82 (64)	17 (13)	0 (0)
	Declaration and management of interest	19a (COI type)	18 (14)	93 (72)	18 (14)	0 (0)
		19b (how)	9 (7)	97 (75)	23 (18)	0 (0)
	Section average		22 (17)	90.3 (70)	16.75 (13)	0 (0)
Other information
	Access	20 (access where)	30 (23)	82 (64)	17 (13)	0 (0)
	Suggestions for further research	21 (gaps)	14 (11)	114 (88)	1 (1)	0 (0)
	Limitations of the guideline	22 (limitations)	25 (19)	97 (75)	7 (5)	0 (0)
	Section average		23 (18)	97.7 (76)	8.3 (6)	0 (0)

Values are presented as number (%).

QA = quality assurance, COI = conflict of interest.

Comparison between the two groups

Based on the median value of the summarized score, guidelines were classified as Group 1 (n = 62), which had a lower acquisition than median value 49, and Group 2 (n = 67), which had a higher score of the median value. Table 2 and Table 3 show a comparison of the characteristics of guidelines and methodology description.

Table 2

Difference in development process between groups

Description		Group 1 (Partial) (Total n = 62)	Group 2 (Well) (Total n = 67)	Pearson χ2 test (P)
Multidisciplinary in GDG (9a)
	Multidisciplinary (≥ 3)	20 (32.3)	44 (65.7)	14.382
	Single or two (1 or 2)	42 (67.7)	23 (34.3)	(*** P < 0.001)
Funding Source (18a)
	Reported	44 (71.0)	61 (91.0)	8.571
	Not reported	18 (29.0)	6 (9.0)	(** P = 0.003)
Declare of COI (19a)
	Yes	33 (53.2)	53 (79.1)	9.70
	No	29 (46.8)	14 (20.9)	(** P = 0.002)

Values are presented as number (%).

Group 1 (partial compliance group): Guidelines converted RIGHT scores were under median level 49, Group 2 (well compliance group): Guidelines converted RIGHT scores were over median level 49.

GDG = guideline development group, COI = conflict of interest.

*P < 0.05, **P < 0.01, ***P < 0.001.

Table 3

Difference in methodology description between groups

Description		Group 1 (Partial) (Total n = 62)	Group 2 (Well) (Total n = 67)	Pearson χ2 test (P)
Evidence level (12)
	Yes	38 (61.3)	66 (98.5)	28.55
	No	24 (38.7)	1 (1.5)	(*** P < 0.001)
Recommendation grading (13c)
	Yes	37 (59.7)	65 (97.0)	27.13
	No	25 (40.3)	2 (3.0)	(*** P < 0.001)
Consensus process (15)
	Reported	39 (62.9)	60 (89.6)	12.813
	Not reported	23 (37.1)	7 (10.4)	(*** P < 0.001)

Values are presented as number (%).

Group 1 (partial compliance group): Guidelines converted RIGHT scores were under median level 49,

Group 2 (well compliance group): Guidelines converted RIGHT scores were over median level 49.

*P < 0.05, **P < 0.01, ***P < 0.001.

Table 2 shows the difference in the development process between groups in terms of transparency in development. Firstly, regarding the multidisciplinary grouping of the guideline development group (GDG), we divided groups by the number of institutions or academic specialties that participated. Groups with over three institutions or specialties were considered as a multidisciplinary group. Group 2 showed a higher proportion of multidisciplinary than Group 1 (65.7% vs. 32.3%, group difference: P < 0.001). Ninety percent of guidelines in Group 2 mentioned the funding source. However, 70% of group 1 mentioned the funding sources, and 29.0% did not mention them. Disclosure of conflicts of interest were reported more in Group 2 guidelines than Group 1 (79.1% vs. 53.2%, group difference: P = 0.002).

Table 3 shows differences in methodology description between groups. Evidence level description rate was higher in Group 2 guidelines than Group 1 (98.5% vs. 61.3%, group difference: P < 0.001). Methodology for recommendation grading was well reported in Group 2 (97.0%), but in Group 1, the reported rate was 59.7%, and the not reported rate was 40.3%. Reporting the consensus process was also higher in Group 2 (89.6%); however, in Group 1, the reported rate was 62.9%, and not mentioned guidelines were 23 (37.1%).

In terms of CPGs developmental structure and process between groups 1 and 2, there were significant differences in the number of specialty groups involved, information for the funding source, declaration, and management of interest. Group 2 has shown a tendency to have multi-stakeholder involvement and present sufficient information on financial resources and conflict of interest declarations. For the CPGs developmental methodology aspect, group 2 provided more sufficient information compared to group 1 about the supporting evidence-making and the process from evidence to the recommendation.

DISCUSSION

To summarize the current status with RIGHT Checklist compliance of CPGs developed in Korea, topics in ‘basic information,’ ‘background,’ and ‘evidence’ showed full compliance in over 50% of CPGs. In contrast, topics in ‘review and quality assurance,’ ‘funding, declaration and management of interest,’ and ‘other information’ showed full compliance in 25%, 17%, and 18% of CPGs, respectively. This is thought to result from the relatively short history and less experience with CPG development, although guideline developers are well-trained in writing scientific research papers. Nevertheless, the RIGHT Checklist’s topic ‘funding, declaration and management of interest’ have been continuously emphasized in KAMS Appraisal System for CPGs. The corresponding assessment items of the AGREE II instrument are items 22 and 23, “The views of the funding body have not influenced the content of the guideline,” and “Competing interests of GDG members have been recorded and addressed,” respectively. However, the RIGHT Checklist guides CPG developers that they provide 1) funding source (18a) and 2) roles of funders (18b) for item 22, and 3) the types of conflict of interests (COI) (19a) and 4) how developers managed their COIs (19b) for item 23. As mentioned earlier, KAMS has continuously tried to educate and train on the AGREE II since 2013.3 Generally, in the assessment tool, related items presented a declarative sentence compared to the checklist. However, with the RIGHT Checklist, those comprehensive sentences are divided into four specific check items. So, CPG developers can remain aligned with the AGREE II item for related descriptions using the RIGHT Checklist. We believe this finding supports that the KAMS Appraisal System should consider introducing a self-check system using the RIGHT Checklist before submitting the CPG.

The RIGHT Checklist has been included in ECC’s education program from 2018. However, based on the current analysis, education for the RIGHT Checklist needs to be strengthened through diversified delivery methods, such as an interactive, hands-on workshops in addition to existing knowledge transfer methods. Even though 69% of CPGs have scored full compliance on item 10a, the authors all agreed that closed assistance from a methodology expert was necessary during the relevant CPG developmental stage. This decision reflected the experiences in consulting the individual GDG who converted clinical questions to PICO questions. The evidence and recommendation sections are considered to easily meet the RIGHT Checklist requirements if the GRADE Evidence to Decision framework (GRADE EtD) is used (Supplementary Table 1).

This study’s strengths are that 1) it is possible that data acquisition was more complete because KAMS implemented it (KAMS is the most authoritative organization for CPG development in Korea), and 2) the authors who assessed compliances for the RIGHT Checklists were experienced CPG development experts and AGREE II appraisal experts.

This study has a limitation that there is a possibility that there are some CPGs in Korea that are not included because the literature search or medical society survey was not perfect.

In conclusion, this study evaluated RIGHT Checklist compliance of CPGs developed in Korea over the past five years and derived the support measures for implementing each checklist item. There was no specific link between the level of compliance of items and the supporting measures agreed upon by the ECC. However, based on this study's results, the ECC expects that it can be used to develop strategic plans for enhancing the capabilities of developing CPGs in Korea.