Applying the RE-AIM Framework to Evaluate the Dissemination and Implementation of Clinical Practice Guidelines for Sexually Transmitted Infections.

Clinical practice guidelines (CPG) are one of the most effective ways to translate evidence of medical improvement into everyday practice. This study evaluated the dissemination and implementation of the Sexually Transmitted Infections-Korean Guidelines (STIKG) by applying the reach, effectiveness, adoption, implementation and maintenance (RE-AIM) framework. A survey questionnaire was administered to clinicians via the internet. Among the 332 respondents, 190 (57.2%) stated that they were aware of STIKG and 107 (33.2%) implemented STIKG in their practice. The odds that a physician was exposed to STIKG (dissemination) were 2.61 times greater among physicians with previous training or education for any CPG than those who did not. Clinicians who indicated that STIKG were easy to understand were 4.88 times more likely to implement STIKG in their practice than those who found them not so easy. When a clinician's workplace had a supporting system for CPG use, the odds of implementation was 3.76 times higher. Perceived level of effectiveness of STIKG did not significantly influence their implementation. The findings of this study suggest that, ultimately, knowing how to engage clinicians in CPG implementation is as important as how to disseminate such guidelines; moreover, easy-to-use guidelines and institutional support are key factors.

INTRODUCTION

Clinical practice guidelines (CPG) have been one of the most effective ways of bringing evidence-based medicine to the frontline of medical care (1, 2, 3). In Korea, there were many studies on the evaluation and quality assurance of clinical practice guidelines. However, there are few studies on implementation (4, 5, 6). Moreover the actual utilization of these guidelines in clinical settings has occurred less frequently than expected (7, 8, 9, 10). McGlynn et al. (11), in their study on guideline adherence to 30 conditions in the United States, showed that, on average, only half of patients received the recommended care. Korea is no exception in such underutilization of CPG, as supported by several studies. Kim et al. (12) reported that 62.6% of the physicians working in private clinics mentioned that they needed updated medical knowledge, but the majority of clinicians did not gain this information from CPG. Yang et al. (13) investigated the use of CPG for depression and found that only 55.7% of respondents stated that they used the guidelines. Such underutilization of CPG is not only a waste of resources, but more importantly, a missed opportunity for patients who need the most up-to-date care.

On that point, guideline implementation, as much as its development, has been taking center stage. It is broadly acknowledged that the guideline-development process itself should take into account how to disseminate and get clinicians to implement the guidelines. Fortunately, this strong integration of guideline development and implementation is being tried by international movements, such as the Guideline Implementability Research and Application network (GIRAnet) (9) and the Guideline Implementability Appraisal (GLIA) (14). It is imperative for the Korean medical society to join such a movement and help clinicians better help their patients.

Sexually Transmitted Infections-Korean Guidelines (STIKG) were developed by the Korean Association of Urogenital Tract Infection and Inflammation (KAUTII) in April 2011; since then, they have been actively distributed through the Korea Center for Disease Control and Prevention (KCDC) to community health centers and clinics across the country. KAUTII also distributed the guidelines to all 88 teaching hospitals with departments of urology and has posted the guidelines on its website (15). However, no studies have examined the implementation of these guidelines.

The implementation of guidelines can be better understood with the help of approaches based on proven theories and frameworks (16). The Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework has been used in various fields including the evaluation of clinical guidelines implementation (17, 18, 19). The five domains are self-explanatory, and RE-AIM assumes the solid implementation of guidelines, requiring that the guidelines reach the user, the user acknowledge clinical effectiveness of the guidelines, the user or the workplace (institution) adopt the guidelines, the user implement the guidelines, and the user maintain the use (usually longer than 6 months).

This study aims to investigate the factors that affect the dissemination and implementation of the STIKG in Korea by applying a modified RE-AIM framework and providing suggestions to improve the guidelines' implementation. The method used in this study is also expected to be applied to help future studies on CPG implementation.

MATERIALS AND METHODS

Survey development based on the RE-AIM framework

A survey questionnaire was developed to evaluate each RE-AIM domain and its related factors. Table 1 lists the questions asked to measure the level of each RE-AIM domain in this study. The maintenance domain was excluded because the duration between STIKG distribution and survey administration for most respondents was shorter than 6 months, the period required to measure maintenance per RE-AIM developers' recommendation (20). In addition, as multiple studies suggest that the complexity (difficulty) of CPG is important in their implementation (1, 2, 8, 21), the item "STIKG is easy to understand" was asked. Except for the adoption domain, which asked whether the clinician's workplace has good supporting system for STIKG (institutional level), all questions asked how one perceives and behaves regarding the guidelines (individual level).

Table 1

RE-AIM domains and questions use in the survey

Dimension	Level	Survey items included in this study
Reach	Individual	I am aware of STIKG.
Effectiveness	Individual	I think STIKG are helpful for deciding how to treat patients with sexually transmitted infections.
Adoption (Institutional)	Institutional	My workplace has systems supporting the use of STIKG (e.g., Order Communication System [OCS] or Electronic Medical Record).
Implementation	Individual	I use STIKG frequently when I treat the patients.
Maintenance	N/A	N/A

N/A, Not Applicable in this study.

RE-AIM items were measured on a five-point Likert-type scale, ranging from strongly disagree to strongly agree. The survey questionnaire also asked for respondents' demographic information, such as age and sex. In addition, information on workplace locations (urban or rural), workplace settings (hospital or clinic), and years of practicing and past training experience on any clinical guidelines was requested.

Survey administration

The survey was administered from November 1 through November 30, 2012, to the members of four clinical associations treating most patients with sexually transmitted infections: the Association of Korean Clinical Urologists, Korean Society of Obstetrics and Gynecology, Korean Physicians' Association, and Korean Association of Public Health Doctors. A short email containing the link to the survey's website was sent out to 7,387 clinicians working at clinics and hospitals.

Data analysis

It is notable that the developers of RE-AIM suggest that not all domains should be addressed, offering a certain level of flexibility for researchers (17). Considering the clinical atmosphere of Korea, where STIKG are being used, RE-AIM domains can be realigned. To illustrate, even when a clinician does not acknowledge the effectiveness of STIKG, the clinician tends to use them, once the workplace (e.g., hospital) adopted an electronic medical record system with the guidelines already built in. The same is true the other way around. Therefore, it is reasonable to consider effectiveness and adoption as separate and independent predictors of implementation.

As such, we focused on two domains in implementing STIKG. First, a clinician must be exposed to the STIKG (reach). Second, the clinician must decide whether to use and implement the guidelines (implementation). We conducted two separate analyses for each of the two domains to identify the key factors influencing them.

Reach

A multiple logistic regression analysis was conducted with all respondents. The dichotomized status of the reach domain was the dependent variable. Respondents' demographic information, workplace setting, workplace location, years of practice, and previous CPG training experience were included in the model as independent variables.

Implementation

The analysis of the implementation domain was conducted among only those who answered yes to the reach domain question. The dichotomized status of implementation was included in the model as a dependent variable, and all the independent variables used in the reach domain analysis were plugged into the model. Along with them, the dichotomized effectiveness domain, adoption status, and response to "STIKG are easy to understand" were included in the model as explanatory variables under the assumption that the effectiveness and adoption domains could independently affect CPG's implementation.

Initially, separate analyses were conducted for responses from clinics and hospitals. However, no noticeable difference was found; thus, they were included in the models only as adjustment variables. Age was not included in the model due to collinearity with years of practicing.

All analyses were conducted using the IBM SPSS Statistics, version 20, software package.

RESULTS

Characteristics of respondents

A total of 392 surveys were returned. After excluding 60 incomplete ones, 332 surveys were ultimately included in the analysis. Table 2 describes respondents' characteristics. The respondents comprised 264 male (79.5%) and 68 female (20.5%) clinicians. Most respondents were in their thirties to fifties; clinicians in their twenties (4.2%) and those who were 60 or older (7.2%) accounted for only a small portion of the respondents. With regard to years of practice, 234 (70.5%) of the respondents had more than 10 yr of experience, followed by 60 respondents (18.1%) with 6-10 yr of experience, and 38 respondents (11.4%) with 1-5 yr of experience. The majority of respondents answered that they work at clinics (243 respondents, 73.2%); 89 respondents (26.8%) replied they work in a hospital setting. In addition, 195 respondents (58.7%) work in urban areas, compared to 137 (41.3%), who work in rural areas. Only 45 respondents (13.6%) said they had any training on any clinical guidelines.

Table 2

Characteristics of respondents

Category	No. of subjects	%
Sex
Male	264	79.5
Female	68	20.5
Age
20-29	14	4.2
30-39	115	34.6
40-49	101	30.4
50-59	78	23.5
60 or older	24	7.2
Years of practice
1-5	38	11.4
6-10	60	18.1
Over 10	234	70.5
Workplace type
Clinic	243	73.2
Hospital	89	26.8
Workplace location
Urban	195	58.7
Rural	137	41.3
CPG education
No	287	86.4
Yes	45	13.6
Total	332	100.0

Reach and implementation rates

Among the 332 survey respondents, 190 (57.2%) stated that they were aware of STIKG (reach); 107 (33.2%) indicated that they had implemented STIKG in their practice. Among the reach group, the proportion of clinicians who stated that they implemented STIKG was 56.3%, as seen in Fig. 1.

Fig. 1

Rates of reach and implementation.

Factors related to reach of STIKG to clinicians

A multiple logistic regression was conducted among all 332 respondents to investigate factors affecting the reach of STIKG to clinicians (Table 3). Workplace type, location, and respondents' sex were not significant. The odds that a physician was exposed to STIKG were 2.53 times greater among physicians with 6 to 10 yr of clinical experience than physicians with 1 to 5 yr of experience. Although not shown in the table, clinicians with 6 to 10 yr of experience showed 1.90 times higher odds of exposure to STIKG than those with more than 10 yr of experience (P<0.05). Clinicians who had training on any clinical guidelines showed significantly higher odds (OR=2.61) than those who did not.

Table 3

Logistic regression results of reach of STIKG to clinicians

Category (Reference)	Odds ratio	P value	95% CI
			Lower	Upper
Sex (Male)
Female	0.88	0.67	0.48	1.59
Years of practice (1,2,3,4,5)
6-10	2.53	0.03*	1.08	5.93
Over 10	1.33	0.45	0.64	2.75
Workplace type (clinic)
Hospital	0.90	0.71	0.53	1.54
Workplace location (urban)
Rural	1.15	0.63	0.72	1.84
Previous CPG training (no)
Yes	2.61	0.01†	1.26	5.41

Parentheses include reference category. *P<0.05; †P<0.01. CI, confidence interval.

Factors related to implementation of STIKG

To understand factors related to whether a clinician uses STIKG in their practice (implementation), a multiple logistic regression was conducted among the 190 respondents who stated they were aware of STIKG (reach). The results are shown in Table 4. Most of the items, even those significant for the reach domain, did not show any statistically significant results, although high odds ratios were observed for some variables. Yet two items showed exceptionally high odds ratios and statistical significance. The odds of clinicians utilizing STIKG in their practice were 4.88 times higher among clinicians who answered that "STIKG were easy to understand" than clinicians who did not. The clinicians who stated that their workplace has systems supporting the use of STIKG (adoption) were 3.76 times more likely to use them than those who did not.

Table 4

Factors Affecting the Implementation of STIKG

Category (Reference)	Odds ratio	P value	95% CI
			Lower	Upper
STIKG was easy to understand (no)
Yes	4.88	0.02*	1.24	19.21
Effectiveness (no)
Yes	0.87	0.88	0.15	5.16
Adoption-institutional (no)
Yes	3.76	0.00†	1.52	9.30
Sex (male)
Female	0.45	0.07	0.19	1.05
Years of practicing (1,2,3,4,5)
6-10	2.76	0.11	0.81	9.40
Over 10	1.80	0.31	0.58	5.59
Workplace type (clinic)
Hospital	1.49	0.33	0.67	3.30
Workplace (urban)
Rural	0.88	0.69	0.47	1.67
Previous CPG training (no)
Yes	1.40	0.43	0.61	3.23

Parentheses include reference category. *P<0.05; †P<0.01. CI, confidence interval.

DISCUSSION

Guidelines' implementation should be understood in the cultural context of the target population and its interaction with the contents of the guidelines (22). This study was, therefore, conducted to investigate the implementation process of specific CPG-namely, STIKG in Korean healthcare settings. This process starts by exposing the target population to the guidelines and eventually getting their buy-in, which leads them to implement CPG, to some degree. This approach resembles how businesses promote their products among users: define the market, develop penetration strategy, and execute the strategy according to users' characteristics (23, 24). Such strategies usually consist of multiple stages.

This study, therefore, utilized a stage-based approach-namely, the RE-AIM framework. RE-AIM offers several advantages over other stage-based models, such as the Transtheoretical model (TTM), so-called the Stages of Change model. RE-AIM clearly defines the reach stage when an individual first encounters a program or guidelines; the rest of the domains are also well-aligned and easy to measure (25). By focusing on the reach and implementation domains and including other components as explanatory variables, we were able to obtain richer outputs than previous studies using the RE-AIM framework.

The results of this current study have significant policy implications. First and foremost, previous training or education on any CPG, not limited to STIKG, positively influenced the reach of STIKG. As the survey of this study was conducted one time, this result can be interpreted in various ways. Clinicians with previous CPG education might have more opportunities or tendencies to seek out new CPG or might better remember STIKG than clinicians without previous CPG education. Whatever the explanation, as long as reach is the prerequisite of the entire CPG implementation process, the importance of CPG-related education and training cannot-and should not-be overlooked. This result concurs with the results of Banait et al. (26), who emphasized the importance of active outreach, including education sessions on CPG.

The finding that clinicians with 6 to 10 yr of years of practice showed a significantly higher reach status than the other groups is rather difficult to interpret with the given data in this study. However, this result suggests that the developer of STIKG should consider more actively reaching out to clinicians with fewer than 5 yr or more than 10 yr of practice experience.

In the second analysis in which we investigated key factors for implementation, the most notable result was that the item "STIKG are helpful for deciding how to treat patients (effectiveness)" was not significantly influential. This finding supports the analysis strategy of this study: The domains of RE-AIM are not necessarily aligned in sequence; rather, they might interact with each other, influencing the subsequent domain in the chain. Contrariwise, the item "STIKG are easy to understand" and the institutional adoption factor (i.e., the workplace has a support system for using STIKG) showed significant effects on clinicians' STIKG implementation. This finding concurs with the findings of many previous studies: The complexity of CPG is a crucial factor in CPG implementation (2, 8, 21, 27). Even in a study, physicians prefer guidelines that are shorter than two pages (1). In addition, the institutional adoption of a CPG support system can vary from simple guidance on doctors' computer screens to a decision supporting system tied up with electronic medical records and a computerized order entry system. Although this study did not measure the level of support, this finding corresponds with the results of a study by Kawamoto et al. (28), who found that the Clinical Decision Support System (CDS) improves clinical practice. Considering that Korea is one of the countries where almost all doctors use computerized systems, such CPG integration in healthcare organizations' IT system is a great option. The ease of understanding and institutional adoption, combined together, are well connected to the study of Haagen et al. (29), who found that self-efficacy in using guidelines was one of the key determinant factors. All in all, providing easy-to-use guidelines and well-suited support systems is the most important way to help clinicians implement guidelines.

Despite the analytical soundness and compelling results, this study has its own limitations. First, the response rate was not as high as we expected, and the representativeness of the analyzed sample was limited. Using step-wise analysis (as depicted in Fig. 1), we achieved a certain level of internal validity, but the matter of external validity remained. Second, all key variables, including implementation, were measured using a Likert-type five-point scale and then dichotomized to yes or no. Using this data process, misclassification bias might occur, leading to a nullification of the results by reducing the power of the analysis (30). If this was the case, we might not have captured all significant factors. However, this paradoxically gives more confidence in the key factors significantly affecting STIKG reach and implementation in this study. Third, this study did not dig deeply into the facilitators or barriers of implementation. There could be numerous factors for this outcome, such as attitude toward CPG in general or specific guidelines, peer pressure or support, and even work conditions such as the volume of patients. Although these items are beyond the scope of this study focusing on RE-AIM domains, a comprehensive understanding of clinicians' behavior will give much richer information. Further studies are needed to better understand such factors with help of other already established behavioral theories such as the Theory of Planned Behavior or and the Reasoned Action Approach (31, 32). In addition, any incentives or benefits from using STIKG besides their effectiveness were not investigated, which otherwise would help us better depict the dynamics of physicians' behavior (25). Finally, as the duration between the beginning of STIKG dissemination and the survey was less than six months, we were not able to measure the maintenance status of the STIKG use, one of the five domains of RE-AIM framework. To get the whole picture of guidelines' implementation with RE-AIM, a future study might be conducted with a longer timeline that can cover all the stages of guidelines' implementation for most clinicians.

In conclusion, this study captures the key factors to consider in developing and implementing clinical guidelines in Korea, using STIKG as an example. The findings and unique analytic methodologies of this study will help tightly align future guidelines, from development through implementation.