Determinants Used to Justify the Strength of Recommendations among Korean Clinical Practice Guidelines.

BACKGROUND: A standardized systematic approach to grade evidence and the strength of recommendations is important for guideline users to minimize bias and help interpret the most suitable decisions at the point of care. The study aims to identify and classify determinants used to make judgement for the strength of recommendations among 56 Korean clinical practice guidelines (CPGs), and explore strong recommendations based on low quality of evidence.
METHODS: Determinants used in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach among 34 CPGs which have reported both strength of recommendations and level of evidence were reviewed.
RESULTS: Five of 34 CPGs (14.7%) considered quality of evidence, benefits and harms, patients' values and preferences, and costs. And 24 of 34 CPGs (70.6%) considered both magnitude of effect and feasibility as additional determinants. Judgement table was not widely provided for use to translate evidence into recommendations. Eighty-two of 121 recommendations (67.8%, ranged 20.0% to 100.0%) among 11 CPGs using the same judgement scheme showed 'strong' strength of recommendations based on low or very low quality of evidence. Among 5 paradigmatic situations that justify strong recommendations based on low or very low evidence, situation classified as 'potential equivalence, one option clearly less risky or costly' was 87.8% for 82 strong recommendations. Situation classified as 'uncertain benefit, certain harm' was 4.9%.
CONCLUSION: There is a need to introduce and systematize an evidence-based grading system. Using judgement table to justify the strength of recommendations and applying the 5 paradigmatic situations mentioned above is also recommended in the near future.

INTRODUCTION

To minimize bias and help interpret the most suitable decisions at the point of care for users of clinical practice guidelines (CPGs), a standardized systematic approach to grade evidence and the strength of recommendations is important.1 However, because several grading systems have shown inconsistency and wide variations to grade quality of evidence and recommendations,23 it is difficult for guideline users to understand the message. Different grading systems include Grading of Recommendations Assessment, Development and Evaluation (GRADE), National Institute for Health and Clinical Excellence (NICE), Scottish Intercollegiate Guideline Network (SIGN), and Centre for Evidence-Based Medicine, Oxford (CEBM) are used to give a clinical recommendation in a CPG which have an influence on the clinician's point of view and can significantly change clinician's decisions and clinical outcomes.1

The GRADE is an emerging consensus on rating quality of evidence and strength of recommendations. GRADE provides an explicit, comprehensive, transparent, and pragmatic approach456 and is increasingly being adopted by organizations worldwide.789 Separating the judgements regarding the quality of evidence from judgements about the strength of recommendations is a critical and defining feature of this new grading system.1011 The GRADE approach also provides a framework to move from evidence to the recommendation.12131415 The strength of recommendations161718 depends on estimates of magnitude of effect,192021 estimates of values and preference and their variability,22 confidence in each of these estimates, and resource use2324 considerations.

GRADE guidance allows five situations that justify strong recommendations in the face of low or very low confidence in estimates for critical outcomes.725 Despite GRADE guidance's warning against the transparent approach, World Health Organization (WHO) guidelines showed that approximately one-half of the recommendations were based on low or very low confidence.262728 It raises concerns about whether GRADE is being optimally applied in the WHO guideline development process.

In Korea, few CPGs are trying to adopt GRADE.2930313233 They considered little bit different components to grade recommendations and utilize varied grading systems. Some guidelines considered only an evidence and there was lack of information on the resource use and patients' value and preferences mostly. There has been no review of factors or grading systems used to grade recommendations in Korea. Therefore, it is necessary to obtain a clear rationale for the adoption of the GRADE approach which is currently accepted worldwide through the analysis of the current situation, particularly the situation where strong recommendations are based on low evidence.

The study aims to identify and classify determinants used to make judgement for the strength of recommendations among 56 Korean CPGs, and explore strong recommendations based on low quality of evidence.

METHODS

Data source

To identify determinants used to make judgement for the strength of recommendations in Korea, 56 CPGs developed based on the academic societies that are listed in the Korean Medical Guideline Information Center (http://www.guideline.or.kr/) were included. Of the 56 CPGs, 27 (48.2%) were developed between 2001 and 2010, and 29 (51.8%) were developed between 2011 and the end of June 2017. Fifty-six CPGs were developed in a total of 33 academic societies; 22 of 33 academic societies (66.7%) developed 1 CPG, 6 of 33 (18.2%) developed 2 CPGs, and 5 of 33 (15.1%) developed 3 or more CPGs.

Data abstraction process

We conducted 3-step process. First, 56 Korean CPGs were classified by reporting level of evidence and/or the strength of recommendations. Second, determinants used to make judgement for the strength of recommendations among 34 CPGs were identified (Appendix 1). The following determinants used GRADE approach worldwide is included: quality of evidence, balance between benefits and harms, patients' values and preferences, and costs (resource use). And additional determinants include absolute magnitude of effect and feasibility (Appendix 2). To collect data on feasibility, we considered two aspects, feasible to implement and whether to consider barriers. Third, to explore the strength of recommendations based on low or very low quality of evidence, 11 CPGs using the same judgement scheme (strong, weak) were reviewed.2930313233343536373839 We will compare a summary on the strength of recommendations by low or very low quality of evidence with WHO CPGs.726 Fourth, classification by 5 paradigmatic situations that justify strong recommendations based on low or very low certainty of the evidence in GRADE approach was performed. In case of ‘life-threatening situations and potential equivalence’ and ‘one option is clearly less risky or costly,’ the strength of the recommendation in favor of the intervention can be given a strong. In case of ‘uncertain benefit with certain harm,’ ‘potential catastrophic harm,’ and ‘high similar benefits, one option potentially more risk or costly,’ the strength of the recommendation against the intervention can be given a strong (Appendix 3).

RESULTS

Classification on reporting for the strength of recommendations and/or level of evidence among 56 Korean CPGs were as follows: 18 of 56 CPGs (32.1%) did not report both strength of recommendations and level of evidence, 2 CPG (3.6%) reported level of evidence only, 2 CPG (3.6%) reported strength of recommendations only, and 34 CPGs (60.7%) reported both strength of recommendations and level of evidence (Table 1).

Table 1

Classification of reporting for the strength of recommendations and/or level of evidence among 56 Korean CPGs

Reporting items	Number of CPGs (%)
Reported quality or level of evidence only	2 (3.6)
Reported strength of recommendations only	2 (3.6)
Reported neither strength of recommendations nor quality or level of evidence	18 (32.1)
Reported both strength of recommendations and quality or level of evidence	34 (60.7)
Total	56 (100.0)

CPG = clinical practice guideline.

Determinants used in the GRADE approach and judgement table used to justify the strength of recommendations among 34 CPGs which have reported both strength of recommendations and level of evidence were as follows: 12 of 34 CPGs (35.3%) considered only quality of evidence, and 12 CPGs (35.3%) considered quality of evidence and benefits and harms. Five of 34 CPGs (14.7%) considered quality of evidence, benefits and harms, patients' values and preferences, and costs such as GRADE approach. And 24 of 34 CPGs (70.6%) considered both magnitude of effect and feasibility as additional determinants (Table 2). Judgement table was not widely provided for use to translate evidence into recommendations. There was inconsistency between information in method part and information in background on the determinants for making judgement the strength of recommendations (Appendix 1).

Table 2

Determinants and judgement table used to justify the strength of recommendations

Considered determinants by GRADE approacha	Additional determinantsa	Number of CPGs (%)	Provide judgement table to justify the strength of recommendations
Quality or level of evidence only (n = 12)	None	1 (2.9)	No
	Magnitude of effect	6 (17.7)	No
	Magnitude of effect and feasibility	5 (14.7)	No
Quality or level of evidence and benefits and harms (n = 12)	Magnitude of effect	2 (5.9)	No
	Magnitude of effect and feasibility	10 (29.4)	No
Quality or level of evidence, benefits and harms, and patients' value and preferences (n = 3)	Magnitude of effect and feasibility	3 (8.9)	No
Quality or level of evidence, benefits and harms, and cost (n = 2)	Magnitude of effect	1 (2.9)	No
	Magnitude of effect and feasibility	1 (2.9)	No
Quality or level of evidence, benefits and harms, patients' value and preferences, and cost (n = 5)	Magnitude of effect and feasibility	5 (14.7)	No
Total		34 (100.0)

GRADE = Grading of Recommendations Assessment, Development and Evaluation, CPG = clinical practice guideline.

aSee Appendix 1.

Strength of recommendations based on low quality of evidence among 11 Korean CPGs using the same judgement scheme (strong, weak, or conditional) were as follows: 82 of 121 recommendations (67.8%, ranged 20.0% to 100.0%) showed ‘strong’ strength of recommendations based on low or very low quality of evidence. Six of 11 CPGs (54.5%) made judgement ‘strong’ for more than 70% of recommendations based on low or very low quality of evidence (Table 3). The determinants used to make judgement varied among 11 CPGs (Appendix 1). Table 4 showed comparison a summary on the strength of recommendations by low or very low quality of evidence. Sixty-seven point eight % of recommendations was given ‘strong’ recommendations based on low or very low quality of evidence in 11 Korean CPGs, follows 67.4% for 44 CPGs of 20 countries, 53.0% for 43 CPGs of WHO, and 48.4% for 8 CPGs (human immunodeficiency virus [HIV] and tuberculosis [TB] only) of WHO. It was ranged from 57.8% to 74.5% for low quality of evidence and ranged from 27.6% to 57.8% for very low quality of evidence.

Table 3

Strength of recommendations based on low or very low quality of evidence among 11 Korean CPGs using the same judgement scheme (strong, weak or conditional)

Guidelines (n = 11)	Strength of recommendations based on low or very low quality of evidence
	Strong	Weak (or conditional)	Total
CPG 1 (2010)29	7 (58.3)	5 (41.7)	12 (100.0)
CPG 2 (2011)34	20 (95.2)	1 (4.8)	21 (100.0)
CPG 3 (2012)35	3 (42.9)	4 (57.1)	7 (100.0)
CPG 4 (2012)36	3 (20.0)	12 (80.0)	15 (100.0)
CPG 5 (2012)30	13 (86.7)	2 (13.3)	15 (100.0)
CPG 6 (2012)31	10 (71.4)	4 (28.6)	14 (100.0)
CPG 7 (2013)37	7 (100.0)	0 (0.0)	7 (100.0)
CPG 8 (2014)32	11 (73.3)	4 (26.7)	15 (100.0)
CPG 9 (2014)33	1 (25.0)	3 (75.0)	4 (100.0)
CPG 10 (2014)38	2 (33.3)	4 (66.7)	6 (100.0)
CPG 11 (2015)39	5 (100.0)	0 (0.0)	5 (100.0)
Total	82 (67.8)	39 (32.2)	121 (100.0)

Values are presented as number (%).

CPG = clinical practice guideline.

Table 4

Comparison a summary on the strength of recommendations by low or very low quality of evidence

Nation	Number of guidelines	Quality of evidence	Strength of recommendations
			Strong	Weak (or conditional)	Total
Korea	11	Low	76 (74.5)	26 (25.5)	102 (100.0)
		Very low	6 (31.6)	13 (68.4)	19 (100.0)
		Total	82 (67.8)	39 (32.2)	121 (100.0)
20 countries7	44	Low	505 (74.4)	174 (25.6)	679 (100.0)
		Very low	283 (57.8)	207 (42.2)	490 (100.0)
		Total	788 (67.4)	381 (32.6)	1,169 (100.0)
WHO A26	43	Low	95 (60.1)	63 (39.9)	158 (100.0)
		Very low	65 (45.1)	79 (54.9)	144 (100.0)
		Total	160 (53.0)	142 (47.0)	302 (100.0)
WHO B7	8 (HIV/TB)	Low	37 (57.8)	27 (42.2)	64 (100.0)
		Very low	8 (27.6)	21 (74.4)	29 (100.0)
		Total	45 (48.4)	48 (51.6)	93 (100.0)

Values are presented as number (%).

WHO = World Health Organization, HIV = human immunodeficiency virus, TB = tuberculosis.

Classification by 5 paradigmatic situations that justify strong recommendations based on low or very low certainty of the evidence is shown in Table 5. Situation classified as ‘potential equivalence, one option clearly less risky or costly’ was 87.8% for 82 strong recommendations; 86.8% (66 of 76) in low quality of evidence and 100% (6 of 6) in very low quality of evidence. Situation classified as ‘uncertain benefit and certain harm’ was 4.9% (4 of 82 recommendations), and 7.3% (6 of 82) was classified as other situation including no intervention.

Table 5

Classification by 5 paradigmatic situations that justify strong recommendations based on low or very low certainty of the evidence in GRADE approacha

Quality of evidence	Strong recommendations in favor of the intervention		Strong recommendations against the intervention			Other situation (no intervention)	Total
	Life-threatening situationb	Potential equivalence, one option clearly less risky or costlyc	Uncertain benefit, certain harmd	High similar benefits, one option potentially more risky or costlye	Potential catastrophic harmf
Low	-	66 (86.8)	4 (5.3)	-	-	6 (7.9)	76 (100.0)
Very low	-	6 (100.0)	-	-	-	-	6 (100.0)
Total	0 (0.0)	72 (87.8)	4 (4.9)	0 (0.0)	0 (0.0)	6 (7.3)	82 (100.0)

Values are presented as number (%).

GRADE = Grading of Recommendations Assessment, Development and Evaluation.

aSee Appendix 3; bUncertain (low or very low) benefit, immaterial (very low to high) harm, high mortality, absence of effective alternative; cUncertain (low or very low) benefit, certain (high or moderate) harm, similar rates of complete response in comparison with the alternatives, but high confidence of less harm, morbidity, and cost; dUncertain (low or very low) benefit, certain (high or moderate) harm, intervention is associated with a substantial

established harm; eCertain (high/moderate) benefit, uncertain (low or very low) harm, potential greater harm; fImmaterial (very low to high) benefit, uncertain (low or very low) harm, potential important harm of the intervention.

DISCUSSION

Determinants used to justify the strength of recommendations among Korean CPGs were identified and classified. Five different grading systems each considered different factors; 1) only quality of evidence was considered, 2) quality of evidence and benefits and harms were considered, 3) quality of evidence, benefits and harms, and patients' values and preferences were considered, 4) quality of evidence, benefits and harms, and cost were considered, and 5) quality of evidence, benefits and harms, patients' values and preferences, and cost were considered. We also considered additional factors including absolute magnitude of effect and feasibility. GRADE approach has been adopted by 5 of 34 CPGs (14.7%). And 24 of 34 CPGs (70.6%) considered both magnitude of effect and feasibility as additional determinants (Table 2). However, we could not find judgement table with transparent explanation and evidence to justify the strength of recommendations among Korean CPGs. Template and judgement table by recommendation has been provided by WHO (Appendix 2). It is a summary table for translating evidence of effectiveness into recommendations. It will be helpful to the guideline end users at the point of care if guideline developers can present a judgement table to clearly explain what factors are considered important.

If CPGs do not provide a structured judgement table, it would require laborious review of the method and/or background for every recommendation listed to discover which determinants were considered. Also, there were some discrepancies between information in method part and information in background on the determinants for making judgement the strength of recommendations (Table 2). To overcome these shortcomings, we should adopt a systematic and transparent GRADE approach which is used by many organizations and associations worldwide. We also reviewed current situation regarding strong recommendations based on low or very low quality of evidence among Korean CPGs. Lower percentage of ‘strong’ recommendations in very low quality of evidence (31.6% vs. 57.8% in 20 countries) is shown in Table 4.

Alexander et al.2728 explained the reason why WHO guideline developers make strong recommendations inconsistent with GRADE guidance. The main reason was limitations in their understanding and optimal application of GRADE. To utilize GRADE approach consistent with GRADE guidance, it requires training of guideline developers or panelists and formal processes to maximize adherence to GRADE principles.7 GRADE guidance present five paradigmatic situations that justify strong recommendation based on low or very low certainty of the evidence (Appendix 3).2527 Among 5 paradigmatic situations, situation classified as ‘potential equivalence, one option clearly less risky or costly’ was 87.8% for 82 strong recommendations. Situation classified as ‘uncertain benefit and certain harm’ was 4.9% (4 of 82). There was no recommendation classified as ‘life-threatening situation’ on the strong recommendations in favor of the intervention, and there was no recommendation classified as ‘potential catastrophic harm,’ and ‘high similar benefits, one option potentially more risk or costly’ on the strong recommendations against the intervention. Especially, when guideline developers have to make a decision the strong recommendation based on low evidence, it will be better to consider the five paradigmatic situations mentioned above.

Limitations for carrying out this study are as follow: First, since only 56 CPGs in the Korean Medical Guideline Information Center are analyzed among all Korean CPGs, there was lack of representative. At the end of 2013, 115 CPGs were reported to have been developed by 45 societies and organizations.40 Second, 34 Korean CPGs were using different grading systems. Therefore, we performed data abstraction on the 11 CPGs using the same judgement scheme (strong, week or conditional), because we could not explore judgement of strong recommendation based on low or very low level of evidence for all CPGs included in the study. Third, when extracting the factors considered in the recommendation grading, we classified them based on the information provided in the method part. However, there were many cases in which the background part of the recommendation actually provided information that considers other factors. We summarized the two kinds of information in Appendix 1 because there are differences according to whether classification of factors considered in grading recommendation is based on information of method part or information of background part.

In conclusion, there were several grading systems with wide variations to grade quality of evidence and strength of recommendations, and strong recommendations based on low or very low quality of evidence were very common among Korean CPGs. There is a need to introduce and systematize an evidence-based grading system. It is also necessary to aggressively review, apply, and disseminate the worldwide GRADE approach that grades recommendations in consideration of important factors including quality of evidence, benefits and harms, patients' value and preferences, and cost in Korea. Before applying and disseminating the GRADE approach on rating quality of evidence and strength of recommendations in Korea, it is necessary to study what external and internal barriers are to use this grading system in advance. Judgement table was not widely provided for use to translate evidence into recommendations, and the five paradigmatic situations were not used against low evidences to decide strength of recommendations to be made.