Methodology for Developing Evidence-Based Clinical Imaging Guidelines: Joint Recommendations by Korean Society of Radiology and National Evidence-Based Healthcare Collaborating Agency.

This paper is a summary of the methodology including protocol used to develop evidence-based clinical imaging guidelines (CIGs) in Korea, led by the Korean Society of Radiology and the National Evidence-based Healthcare Collaborating Agency. This is the first protocol to reflect the process of developing diagnostic guidelines in Korea. The development protocol is largely divided into the following sections: set-up, process of adaptation, and finalization. The working group is composed of clinical imaging experts, and the developmental committee is composed of multidisciplinary experts to validate the methodology. The Korean CIGs will continue to develop based on this protocol, and these guidelines will act for decision supporting tools for clinicians as well as reduce medical radiation exposure.

INTRODUCTION

Evidence-based medicine proposes safer and more effective interventions for patients based on the integration of evidence from research, the knowledge of clinical experts, and the values of patients. One way of implementing evidence-based medicine is through the use of clinical guidelines. Clinical guidelines are the literature that describe optimal clinical practice and help clinicians and patients to make decisions. These guidelines are considered to be effective at maintaining continuity of patients' care and narrowing the gap between scientific evidence and treatment. As an effort to improve the quality of medical services in recent decades, the development and application of these guidelines has been steadily increasing worldwide. In Korea, many academic societies and organizations have also developed clinical guidelines, and have been publishing them since 1997 (1).

In the field of radiology, individual developed countries are utilizing evidence-based clinical imaging guidelines (CIGs) in order to augment clinical decision-making by physicians when requesting or prescribing a radiologic examination. Because radiologic examinations often involve exposure to medical radiation, the principle of justification must be abided by to protect patients from medical radiation exposure: the benefits must outweigh the potential harm in all circumstances where patients are to be exposed, and clinicians should perform only necessary examinations. The action plan (known as the ‘Three As’) for the principle justifying the use of radiation includes ‘awareness’ of medical radiation exposure, ensuring the ‘appropriateness’ of examinations and procedures, and a retrospective ‘audit’. As a means of ensuring appropriateness, guidelines to support clinical referrals and decisions are being developed and applied to clinical treatment (23).

Evidence-based CIGs have been developed in many countries depending on the local circumstances. The iRefer of the Royal College of Radiologists (RCR) in the United Kingdom, Appropriateness Criteria® of the American College of Radiology (ACR) in the United States, and Diagnostic Imaging Pathways (DIP) in Western Australia have each developed their own guidelines (456). These clinical referral guidelines provide the relative dose level of radiation in examinations for various clinical conditions, along with other recommendations and expert opinions. The guidelines are periodically revised (7), and in the case of the Appropriateness Criteria® of the ACR in the United States, a decision-making system reflecting the most recent guidelines has been made accessible online (8). The DIP of Western Australia also supports clinical decision-making through a diagram available on their website (6). The above medical guidelines were created de novo, meaning that they were newly developed through a direct search and review of the available evidence. In contrast, many other countries with similar medical environments often modify, adopt, or translate clinical radiation guidelines that were previously created by the RCR and ACR (7).

In Korea, the need for radiological guidelines led to the development of ‘the CT Examination and Repeat CT Examination Guideline’ by collaboration of Health Insurance Review & Assessment Service and Korean Society of Radiology (KSR) (9), ‘the Guideline for Diagnosis and Management of Hepatocellular Carcinoma’ by Korean Liver Cancer Study Group (10), ‘the Guideline for Ultrasonographic Diagnosis and Image-Based Management of Thyroid Nodules’ by Korean Society of Thyroid Radiology (11), ‘Korean Guideline for Interventional Recanalization of Lower Extremity Arteries’ by collaboration of multi-disciplinary societies including Korean Society of Interventional Radiology (12), and ‘the Guidelines for Appropriate Use of Cardiac CT in Heart Disease’ by the KSR, Korean Society of Cardiology, and National Strategic Coordinating Center of Clinical Research (1314). However, as there are still no CIGs that comprehensively support physicians to prescribe diagnostic and interventional radiologic procedures which are appropriately suited to the situation in Korea, the need for a Korean version of evidence-based clinical medical radiology guidelines has become paramount. Because of this, the development of the evidence-based CIGs has begun, led by the KSR and the National Evidence-Based Healthcare Collaborating Agency, along with the participation of clinical and methodology experts. They selected adaptation for the methodology of CIGs, which accepted developed guidelines and modified for a new guideline to be suited to the medical environment in Korea (15). As the previously existing guidelines in Korea focused mainly on therapeutic interventions, the practical development of manuals that instruct on developing guidelines for diagnostic examinations became a priority. For this reason, development manuals are being written alongside the guidelines (16).

Development of Korean Clinical Imaging Guidelines

Adaptation process of guidelines is based on internationally standardized adaptation methodology that is suggested by ADAPTE Collaboration (17). Then we modified adaptation process for working group's practice under consultation. Developing the Korean CIGs (K-CIGs) involved three stages, set-up (planning), adaptation process, and finalization (Fig. 1). The set-up stage (planning and composition) outlines a process to form committees and clarify their roles. The adaptation stage involves the stepwise development process to draft the guideline. The finalization stage includes the process of completing the recommendation document based on evidence, undergoing external review, and obtaining final approval. Also we developed a protocol for working group that including the process from adaptation to finalization as 8 stages for practical help (Table 1).

Fig. 1

Adaptation process for developing Korean clinical imaging guidelines.

Table 1

Development Protocol of Korean Clinical Imaging Guidelines

Stage	Content	Responsible Group
Stage 1	Define key questions	Working group
		Development committee
Stage 2	Search for guidelines	Development committee
Stage 3	Selection of searched guidelines	Working group
Stage 4	Quality appraisal of guideline	Development committee
		Working group
Stage 5	Grading recommendation and level of evidence	Working group
	Drafting recommendation document	Development committee
Stage 6	Agreement of recommendation and recommendation grading	Consensus group
		Working group
Stage 7	Finalizing recommendation document	Working group
		Development committee
Stage 8	External review	External experts
	Approval of clinical imaging guideline	Korean Academy of Medical Sciences

Planning of Guideline Development (Set-up)

Committee Composition

Two committees were involved in the development of the CIGs: the working group that writes the proposals, and the development committee, which is responsible for the overall planning and provides supports on research methodologies. The working group was composed of 3–4 clinical imaging experts from the KSR subspecial societies, including the cardiovascular imaging, thoracic radiology, interventional radiology, breast imaging, neuroradiology and head and neck radiology, abdominal radiology, uroradiology, musculoskeletal radiology, pediatric radiology, and thyroid radiology societies. The development committee is composed of clinical imaging experts, research methodology experts, and clinical guideline experts. Both committees contributed to improving the quality of the guidelines by providing their expertise at various steps in the development process and collaborating when needed.

In clinical imaging examination, there are end-users who refer and perform the examinations. Therefore, it is important to include their opinions into the development process. After drafting key questions, an official document was sent to related clinical academic societies, which are the expected end-users, asking members of a consensus group for their clinical advice and to review the draft. Finally a consensus group, consisting of 23 nominated members from the final 14 related societies, was composed. Members participated in the review of key questions at the set-up stage, drafting of the proposal, and expert panel based investigation using the Delphi method.

Radiation Level

The radiation level of different imaging examinations is currently included in most of CIGs. As existing guidelines, the relative radiation level (RRL) is organized based on effective dose, which represents the expected risk level of radiation exposure in an entire population for an imaging examination measured in mSv. However, it does not disclose differences in risk level based on age and gender. The RRL used in the K-CIG was finalized after reviewing the ACR Appropriateness Criteria®, iRefer 7th edition from the RCR, and DIP by Western Australia, as well as recent literature and irradiation dose study results from Korea. In the proposal, each imaging examination and the corresponding recommended radiation dose are indicated using symbols (Fig. 2).

Fig. 2

Relative radiation levels in Korean clinical imaging guidelines.

CTA = computed tomographic angiography, IVU = intravenous urography, PA = posteroanterior, RRL = relative radiation level, UGIS = upper gastrointestinal series

Adaptation Process of the Guideline

The adaptation process was divided into 5 stages, including selection of key questions to drafting of the guideline, which was belonged to the K-CIG development protocol (Table 1).

Stage 1: Defining Key Questions

The guideline is deduced based on key questions. Key questions are generated in the form of Population/patient, Intervention/index test, Comparator/control, and Outcome (PICO) questions by a working group composed of experts from specialized fields of radiology. The development committee generates key questions in structured sentenceform based on PICO questions, and discusses the feasibility of development. Especially, the “I” (intervention) of PICO is defined in detail, or it is preferred including all related radiologic methods at least if specific methods are not defined. Both the working and development groups make an effort together to finalize the key questions in structured form.

Stage 2: Search for Guidelines

The strategy for searching guidelines is to maximize the sensitivity of literature searching by utilizing only the “P” and “I” of PICO. The development committee systematically organizes the search strategy and performs the searches using both domestic and international databases. The databases used include international databases (Ovid-MEDLINE, Ovid-EMBASE, National Guideline Clearinghouse, Guideline International Network) and major domestic databases (KoreaMed, KMbase, KoMGI, KGC), so that the final draft successfully reflects the current healthcare environment in Korea. The working group reviews the search strategy and results, and then performs additional searches to ensure inclusion of any important omitted guidelines. The finalized search strategy is saved to ensure reproducibility.

Stage 3: Selection of Searched Guidelines

The selection process for the searched guidelines requires the knowledge of clinical experts; therefore the members of working group are responsible for the selection process. According to the selection criteria outlined in Table 2, two different individuals independently review the literature in a primary screening process and secondary selection process in order to ensure objectivity. Primary screening involves reviewing the title and abstract of an identified study or guideline. In the secondary selection process, the full-text of identified literature is reviewed, and the reason for any exclusion is noted if certain literature has been excluded. Disagreements between reviewers are resolved either by consultation between the reviewers or by obtaining inputs from a third reviewer.

Table 2

Literature Search Selection Criteria

Selection Criteria	1	Guidelines including PICO that match key questions
	2	Guidelines published in Korean, English, or Japanese
	3	Guidelines published after 2000
Exclusion Criteria	1
	2	I&C: key question-related imaging examination is not included
	3	Appropriate results (diagnostic accuracy, efficacy, safety, prognosis, and patients’ preference) were not reported Non-Clinical Practical Guidelines
	4	Non-Clinical Practical Guidelines
		- Reviews, independent clinical trials, critical pathways
		- Guidelines produced by single author without representation, etc.
	5	Recommendations were not suggested
	6	Guidelines were not produced via evidence-based method
	7	Guidelines reported in neither English nor Korean
	8	Overlapping publication
		- Same content in different journal or different publication type
	9	Full-text was not obtainable

PICO = Population/patient, Intervention/index test, Comparator/control, and Outcome, I&C = Intervention/Index & Comparator/Control

Stage 4: Quality Appraisal of the Guideline

The final selected guidelines undergo quality appraisal using the Korean Appraisal of Guidelines for Research & Evaluation II tool (18). At least 3 different appraisers independently assess the selected literature, and the appraisers are selected from the development committee. Each evaluation category is scored by the scale of one to seven points, and the reasoning behind the scores is noted to ensure clarity and reproducibility of the assessment results. If the differences in scores for any of the categories among the appraisers are greater than 4, the literature is reexamined. Basically, when guidelines that score 50 or above in ‘Rigour of development’ domain, they are considered as candidates for enrollment to make K-CIGs. Final appraisal results are provided to the working group, but they can decide to enroll the guideline when available guidelines to use adaptation are limited or the local guideline has been developed in Korea despite receiving low quality appraisal scores.

Stage 5: Grading the Level of Evidence and Drafting the Recommendation Document

Recommendations and evidence for the guidelines appraised by the working group are organized according to the key questions, and then the recommendation document is drafted. The tables for comparison of guidelines outline the details of the recommendation and its strength for each key question (Table 3). It also assesses whether the guideline is up-to-date, acceptability, and applicability. Moreover, if additional review of domestic evidence is needed, the guideline is ensured to be up-to-date by performing additional searches of other guidelines or studies.

Table 3

Example of Table for Comparison of Selected Guidelines

KQ. Which is appropriate examination to diagnose acute appendicitis in adults with acute RLQ pain?
Source of recommendation	ACR Appropriateness Criteria®	JSR guideline
AGREE II (rigour of development)	69	64
Recommendation	In patients with RLQ pain, fever, leukocytosis and typical signs of acute appendicitis, CT (including contrast and non-contrast CT) is generally more accurate than ultrasonography, and it is appropriate (usually appropriate)	In case of adult, CT has evidence which CT is more sensitive and specific for diagnosis of acute appendicitis than ultrasonography, and it is recommended
	Ultrasonography may be also appropriate in special situations
Grades of recommendation	1	B
Acceptability
Similarity of population (prevalence, incidence, etc.)	Yes	Yes
Similarity of value and preference	Yes	Yes
Similarity of benefit by recommendation	Yes	Yes
Generally, acceptable	Yes	Yes
Applicability	Yes	Yes
Applicability of intervention/instrument	Yes	Yes
Applicability of essential technique	Yes	Yes
No legal and institutional barriers	Yes	Yes
Generally, applicable	Yes	Yes

ACR = American College of Radiology, AGREE = Appraisal of Guidelines for Research & Evaluation, JRS = Japan Radiological Society, KQ = key question, RLQ = right lower quadrant

Evidence tables organize the literature related to each key question and provide the evidence level in this study. Basic bibliographic data, title, type of research, results, and quality of the evidence are included as categories. Guidelines from Japan Radiological Society (JRS) and ACR, based on standards from the Oxford Centre for Evidence-Based Medicine, are directly applied to evidence grading in this study. RCR does not provide evidence level for individual literature by open-source, and is therefore, only utilized in recommendations for key questions.

Some evidences outside of the existing CIGs (ACR of the USA, RCR of the UK or JRS of Japan) which do not provide the evidence level should undergo an additional review process about 5 categories (presence of reference standard, continuous recruitment of patients, double blinded trial, systematic literature review, and case-control study type) to determine the evidence level in K-CIG (Table 4). Overall level of evidence grading in K-CIG is merged with the evidence level of individual literature (Table 5). It is categorized as high (I), moderate (II), low (III), or very low (IV).

Table 4

Criteria for Evidence Level of Each Evidence Literatures

Level	Content
1	Research satisfying all of criteria following three
	Criteria 1. Good reference standard
	Criteria 2. Consecutive patients study
	Criteria 3. Blind interpretation
	Systematic review of level 1
	Randomized controlled trial or cross-sectional cohort study that compares index test to comparators
2	Research satisfying all of criteria following two
	Criteria 1. Good reference standard
	Criteria 2. Consecutive patients study or Blind interpretation
	Systematic review of level 2
	Observational studies that compares index test to comparators
3	Without consistently applied reference standards
4	Case-control study
	Poor or non-independent reference standard
5	Expert opinion

Table 5

Definition of Overall Evidence Level for Each Key Question

Overall Evidence Level	Definition	Deduced in Comparison to Recommended Content	Evidence Level of Individual Literature Confirmed
High (I)	Results are from appropriately designed experiments with low risk of bias	Category 1 in ACR	Evidence levels from studies suggesting important conclusions of recommendation are 1 or 2
		Level 1 in RCR
		Levels 1, 2 in Japanese guideline
Moderate (II)	Results are from appropriately designed experiments with intermediate risk of bias	Category 2 in ACR	Evidence levels from studies suggesting important conclusions of recommendation are 2 or 3
		Level 2 in RCR
		Level 3 in Japanese guideline
Low (III)	Results are from inappropriately designed experiments, or risk of bias is high	Category 3 in ACR	Evidence levels from studies suggesting important conclusions of recommendation are 3 or 4
		Level 3 in RCR
		Level 4 in Japanese guideline
Very low (IV)	Results are from inappropriately designed experiments, or risk of bias is high	Category 4 in ACR	Evidence levels from studies suggesting important conclusions of recommendation are 5
		Level 4, 5 in RCR
		Level 5 in Japanese guideline

ACR = American College of Radiology, RCR = Royal College of Radiologists

A draft of the recommendation document consists of recommendations with responsible for key questions, summary of the evidence, considerations for the recommendation, and references. Each recommendation document includes grade of recommendation and overall evidence level. The grade of recommendation for the K-CIG contains A, B, C, and I, indicating the direction of the recommendation (Table 6). The strength of the recommendation is represented by the evidence level. The benefits and harms, acceptability, and applicability in a situation of Korea are considered, and the RRL for different examinations is also included.

Table 6

Grades of K-CIG Recommendation

Grading	Content	Definition
A	Recommended	This intervention (examination) has enough evidence to support desired effect, and therefore, is recommended
B	(Conditional) recommended	This intervention (examination) has intermediate to enough level of evidence to support desired effect
		Provide intervention (examination) selectively, or for specific individuals based on expert’s judgment
C	Not recommended	This intervention (examination) has enough evidence to support non-desired effect, and therefore, is not recommended (use of this examination is not recommended)
D	No recommendation	This intervention (examination) does not have enough evidence to either support or reject effectiveness, and needs further research
		This intervention (examination) has very low level of certainty for desired effect, and decision based on recommendation grading has no meaning

K-CIG = Korean clinical imaging guidelines

Finalization of Guideline Development

The finalization, the last stages of developmental protocol of K-CIG, includes agreement of the recommendations, decision of recommendation grading, external review, and final approval.

Stage 6: Agreement of the Recommendation Grade

The draft version of the recommendation document prepared by the working group is used to decide the evidence level and grade of recommendation, after discussion with the development committee. The validity of the recommendation document is determined by ensuring agreement between the working group and development committees.

Stage 7: Finalizing the Recommendation Document

Agreement on the recommendation document is determined by forming a consensus group. The group is composed of clinical imaging experts, CIG-related societies (end-users), and research methodology experts. The agreement level for recommendation, recommendation grading, and evidence level range from strongly disagree (level 1) to strongly agree (level 9). Delphi method is used, and after two rounds of assessment, the recommendation document is finalized.

Stage 8: External Review and Approval of Clinical Guidelines

Review for the finalized recommendation document is done both internally by clinical imaging experts who did not participate in the development of the guideline and externally by related-society members (end-users of the guideline). Reviews are performed by different methods: posting on the official website of the KSR, sending official written document to related societies, and holding a seminar to hear directly from the users. After collecting opinions from the reviewers and modifying these opinions, the evidence-based guidelines were finalized after a final approval process by the Korean Academy of Medical Sciences.

The developed CIGs will be re-assessed annually, and developed recommendations may revise if new key evidence is presented. In order to make these guidelines useful for clinicians requesting imaging examinations, recommendations in the guidelines will be widely disseminated through diverse methods, such as academic presentations and public dissemination.

CONCLUSION

This paper is a summary of the methodology used in the development of evidence-based clinical radiology guidelines to support referral and clinical decisions, which strive to promote the use of only appropriate examinations and medical procedures, performed in optimal clinical conditions.

To develop evidence-based CIGs suitable for Korea, the “Methodology of K-CIG development” was established. The working group consisted of clinical imaging experts to ensure clinical expertise. The development committee consisted of multidisciplinary experts to ensure validity of the methodology and provide insight on the guideline development process. The development process was divided into several stages, and the two committees were able to collaborate with each other during various stages. In order to reflect the opinions of end-users, a consensus group composed of experts from medical societies was formed to accept their opinion during the development process.

This is the first evidence-based CIG in Korea, and it provides final recommendations and RRLs for radiologic examinations. Although not all decisions in different clinical settings can be made based on evidence, these guidelines will definitely assist in the decision-making process of clinicians in advance, who need to refer the patients for imaging examinations (19). Furthermore, this will justify the medical radiation exposure for patients, and eventually assist in ensuring safety from medical radiation exposure for the patients. Since the developmental process of K-CIGs was fully considered whether their recommendations are acceptable and applicable to the situation in Korea, we expect these guidelines will be widely used at real clinical fields.

Also, we hope to see this article will be utilized as a manual for the next development of CIGs in future, along with the ‘Methodology manual for development of K-CIG’ which is the first-ever in Korea to outline the development process of the guidelines about diagnostic examinations.