Evaluation of an Education Strategy versus Usual Care to Implement the STEADI Algorithm in Primary Care Clinics in an Academic Medical Center.

BACKGROUND: Although falls are the leading cause of morbidity and mortality in the US in the older adult population, there is little information regarding implementation of evidence-based fall prevention guidelines within primary care settings. The objective of this study was to address this gap in the literature by determining the effectiveness of the use of education and written materials as implementation strategies.
METHODS: Using a prospective, mixed methods, controlled before-and-after study design, we studied the effect of the Stopping Elderly Accidents, Deaths, and Injuries (STEADI) education and written materials on knowledge and intention to use in primary care clinics as well as test the screening, assessment, and intervention behaviors. This manuscript details the quantitative findings of the study, using STEADI Knowledge Test, Continuing Professional Development (CPD) Reaction Questionnaire, and EMR Reports. We compared data between the study arms (usual implementation versus education implementation) using descriptive statistics, paired t-tests, and factorial ANOVAs.
RESULTS: In total, data from 29 primary care staff, including physicians, APRNs, RNs, and medical assistants, were analyzed. Although we found a statistically significant difference within the education arm between immediate pretests and posttests/surveys mean scores, there was no statistically significant difference between the study arms' knowledge, intent to use STEADI, or use behaviors. The pre/immediate post education mean knowledge score increased by 1.19 (p= 0.02) and the pre/immediate post education intent to use mean increased by 0.64 (p 0.01). There was no statistically significant change between the study arms over time.
CONCLUSION: Educational strategies, particularly written materials and an online module, did not increase the long-term use of the STEADI toolkit. Implementation research is needed to identify the strategies that are most effective for promoting the adoption of STEADI in primary care.

Introduction

In the United States (US), falls are the leading cause of injury-related morbidity and mortality in the older adult population. There are more than 29 million falls annually.1 Fall-related injury and death are projected to increase with the aging of the US population.2 Not only are there a staggering number of injuries due to falls at over three million in 2015,3,4 but the cost of caring for these patients is high, predicted to exceed 100 billion dollars by 2030.5

In response to the devastating public health issue of falls and fall-related injuries, the American and British Geriatric Societies (AGS/BGS) established evidence-based clinical practice guidelines to screen, assess, and intervene on fall-risk factors.6 The CDC utilized these recommendations to create the Stopping Elderly Accidents, Deaths, and Injuries (STEADI) algorithm and toolkit. This algorithm and toolkit were designed for use by providers in primary care practices. The STEADI algorithm is comprised of three core elements: screening, assessment, and intervention recommendations.7 The screening element includes key fall-related questions and evaluation of gait, strength, and balance. The multifactorial assessment includes a falls history and physical exam. The recommended interventions are based on the findings of the assessment, but can include patient education, vitamin D or calcium prescriptions, or referral to PT or community fall prevention program.7

Although STEADI has been in place for several years, there are few studies that have examined implementation strategies designed to support STEADI’s uptake.8,9 Implementation strategies for evidence-based guidelines within the primary care clinics vary in number and the strategy itself. On average, four to five strategies are used when implementing a new guideline.10–17 Strategies for implementation commonly include education, audit and feedback, and a facilitator.10–17 The study results for successful implementation have been mixed, with some studies reporting success with guideline adherence and others with either mixed results or no increase in guideline adherence. Even fewer studies have been performed to measure the success of the implementation strategies, including workgroups, project champions, audit and feedback, training, coaching, and reminders, in fall prevention in primary care.8,18,19 Additionally, there is little information on sustainability. Although there are available evidence-based fall prevention guidelines and screening tools, there continues to be a lack of information regarding the effectiveness of different implementation strategies to implement STEADI8,20,21 The objective of this study was to address the gap in the literature by determining the effectiveness of implementation strategies, specifically education regarding the STEADI toolkit and dissemination of the STEADI written materials, on the implementation of the (STEADI) toolkit. We tested the effect of the STEADI education and written material implementation strategy on the primary care clinic staff’s knowledge and intention to implement the STEADI toolkit at the conclusion of the intervention follow-up compared to usual implementation. We also tested the effect of the STEADI education and written material implementation strategy on primary care clinic staff’s medical record documentation of screening, assessment, and appropriate interventions 30 days after completing the STEADI education implementation strategy compared to primary care clinic staff receiving usual implementation.

Materials and Methods

Study Design

A prospective, mixed methods study was used to test the effect of the STEADI toolkit education and written materials on knowledge and intention to use in primary care clinics as well as test the effect of the STEADI education and written materials on screening, assessment, and appropriate intervention behaviors compared to usual care. This article describes the quantitative information of the mixed methods study.

Study Populations

Data were obtained from the primary care clinics at a major medical center in the southern US using a convenience sample approach. The sample included primary clinic staff, licensed and unlicensed, since all assist in the STEADI screening, assessment, and intervention. The licensed staff included physicians, advanced practice registered nurses, and nurses. The unlicensed staff included medical assistants. Verbal informed consent was obtained for each participant as approved by the University of Arkansas for Medical Sciences’ IRB. The participants gave verbal informed consent prior to completion of the study documents.

The mixed methods study utilized two study arms, usual implementation (control) and STEADI education implementation strategy. Two clinics were enrolled in each of the study arms using simple randomization. The control arm used the facility’s usual implementation strategy. The data collection methods included the Continuing Professional Development (CPD) Reaction Questionnaire,22 STEADI Knowledge Test,23 and medical record data.

Apriori power analyses were performed prior to the beginning of the study to assess the potential effect sizes based on the anticipated sample size. G*power software was used to perform three priori power analyses due to the need for different calculations for the t-test, factorial ANOVA for knowledge and intervention, and factorial ANOVA for behaviors.24 The effect size (Cohen’s d) of 0.58 for the t-test falls into the medium effect size grouping with each arm containing 20 participants. The Cohen’s f for the factorial ANOVA for both the STEADI Knowledge and CPD Reaction Questionnaire was 0.29 with a sample size of 20 participants per study arm. Finally, the Cohen’s f for the behaviors was 0.3 with a sample size of 12 per study arm.

Implementation Strategies

The two implementation strategies compared in this study were usual implementation and the STEADI education implementation strategy. The usual implementation consisted of the facility’s typical process for implementing new evidence-based guidelines. This included the creation of a reminder in the electronic medical record (EMR) system known as a best practice advisory and a three-page tip sheet outlining the implementation steps. The tip sheet is generated when an older adult has an annual wellness visit. The tip sheet included an introduction of what patient population should be included in the new process, screenshots of the STEADI fall risk screening in the EMR, Staying Independent questions, Timed Up and Go (TUG) exam results, fall risk category, and a documentation tool titled “STEADI High Risk Statement for Providers.” This tip sheet was distributed to the primary care clinic staff by e-mail approximately one month prior to the STEADI go-live.

The STEADI education implementation strategy included written and online educational materials. The use of education and dissemination of written materials as an implementation strategy was chosen because the use of a multimodal approach is essential for effective implementation.25 The principal investigator proctored the online, one-hour, interactive education module. The module titled STEADI Older Adult Fall Prevention Online Course, was provided by the Center for Disease Control and Prevention (CDC).7 The online training included provider education regarding the STEADI algorithm, TUG administration, and description of the written materials included in the STEADI toolkit. The principal investigator administered the online module in person at each clinic to enable participants to ask any questions as they arose during the training.

Measures

CPD Reaction Questionnaire

The CPD Reaction Questionnaire was used to evaluate the participant’s intent to implement the STEADI toolkit. This questionnaire is comprised of 12 Likert-scale questions, with scores from 1 to 7. A score of one is less likely to occur and a score of seven is more likely to occur.26 This scale was used to evaluate the participant’s behavioral intention, social influence, beliefs regarding one’s capabilities, moral norm, and beliefs about consequences.26

STEADI Knowledge Test

The STEADI Knowledge Test, available on the CDC Train website, was used following approval from the CDC, to examine the primary care staff’s knowledge of fall risks and prevention. This was a 10 question, multiple choice test. An example of a question is “Which is not a key question when screening older adults for fall risk?”.23

Medical Records

The participant behaviors were examined utilizing a report generated by the EMR for each primary care clinic in the study. The EMR reports consisted of the primary care provider, annual wellness visit date, STEADI screening, screening results, assessment results, and interventions/recommendations. With regards to the STEADI screening performed, this was a “yes/no” answer. If the screening result was identified as “yes”, the assessment and interventions/recommendations were identified. Interventions/recommendations included medication changes, footwear recommendations, use of assistive device, or referrals for vision or physical therapy.

Data Analysis

The quantitative data analysis included descriptive statistics, paired t-test, and factorial analysis of variance (ANOVA). The data were analyzed using IBM SPSS 26 software.27 Statistical significance was determined at P<0.05. The descriptive statistics were utilized for demographic information to describe the sample population. The descriptive statistics were collected for the study overall and each study arm as a single entity. Categorical data was described using frequencies and percentages. The between-group analysis of the categorical characteristics was examined using a paired t-test and factorial analysis of variance (ANOVA). A paired t-test was used to compare pre-education and immediate post-education means of the knowledge test and CPD Reaction Questionnaire. The factorial ANOVA was used to test the effects of the independent variables, time (30-day interval between the data points) and treatment (education and written materials versus usual implementation), on the dependent variables (knowledge, intention to use, and use behaviors).

Procedures

Following approval from the Institutional Review Board of University of Arkansas for Medical Sciences (#228969), the primary investigator (PI) randomized four of the primary care clinics into two study arms, control (usual implementation) and intervention (STEADI education implementation strategy). The randomization was performed using an online software, Random Lists.28 An alternative clinic was used within the intervention arm because of the decision for one of the originally randomized clinics to withdraw from the study. The PI then collected baseline electronic medical record (EMR) data to measure the primary care clinic staff’s STEADI toolkit use behaviors, including two-question screening, STEADI screening, assessment, and appropriate interventions or recommendations.

Following collection of the baseline EMR data, the PI collected baseline STEADI knowledge and intention to use data for the control and intervention arms using the STEADI Knowledge Exam and CPD Reaction Questionnaire. The STEADI education module was then performed at the intervention clinics. Immediately following this education, the STEADI knowledge and intention to use information was collected again. The PI collected the screening, assessment, and appropriate intervention behaviors; STEADI knowledge, and intention to use data 30 days following the initial data collection in the control arm and 30 days following the education in the intervention arm. Once the data were collected, the PI performed data analysis.

Results

Participants

A total of 29 participants were included in the study, 13 in the control arm and 16 in the education intervention arm. Professions included physician (13.8%), APRN (6.9%), RN (24.1%), medical assistant (41.3%), and other licensed (13.8%). Figure 1 consists of a CONSORT diagram to describe the study arms. The length of time in the profession was diverse with 20 plus years (20.7%), 15–19 years (13.8%), 10–14 years (20.7%), 5–9 years (20.7%), 3–5 years (6.9%), and 1–3 years (17.2%). None of the participants had been in their profession for less than one year. Participants also had varying lengths of employment at their respective clinics. None of the participants were employed in the clinics 15 or more years. One employee has worked in the clinic 10–14 years (3.4%), two for 5–9 years (6.9%), nine for 3–5 years (31%), 11 for 1–3 years (37.9%), and six for less than one year (20.7%). Table 1 shows the information regarding participants, length of time in profession, and length of time employed in the clinics between the study arms. There were no significant differences between the study arms.

Figure 1

CONSORT diagram to show enrollment of participants in the study.

Table 1

Demographic Information

Profession Type	Control Group Percentage of Participants (N)	Intervention Group Percentage of Participants (N)	Study Population Percentage of Profession (N)
Physician	13.8% (4)	0	13.8% (4)
APRN	3.4% (1)	3.4% (1)	6.9% (2)
RN	6.9% (2)	17.2% (5)	24.1% (7)
MA	17.2% (5)	24.1% (7)	41.3% (12)
Other Licensed	3.4% (1)	10.3% (3)	13.8% (4)
Length of Time in Profession
20+ years	13.8% (4)	6.9% (2)	20.7% (6)
15–19 years	6.9% (2)	6.9% (2)	13.8% (4)
10–14 years	6.9% (2)	13.8% (4)	20.7% (6)
5–9 years	6.9% (2)	13.8% (4)	20.7% (6)
3–5 years	0	6.9% (2)	6.9% (2)
1–3 years	10.3% (3)	6.9% (2)	17.2% (5)
Less than 1 year	0	0	0
Length of Time Worked in Clinic
20+ years	0	0	0
15–19 years	0	0	0
10–14 years	3.4% (1)	0	3.4% (1)
5–9 years	0	6.9% (2)	6.9% (2)
3–5 years	24.1% (7)	6.9% (2)	31% (9)
1–3 years	17.2% (5)	20.7% (6)	37.9% (11)
Less than 1 year	0	20.7% (6)	20.7% (6)

Outcomes

STEADI Knowledge

In the terms of STEADI knowledge, over time, although not statistically significant, at the 0.05 significance level, there was an increase in mean scores between the study arms. A factorial ANOVA was conducted to compare the main effects of the STEADI implementation strategy, education, and the interaction between the education and time on the mean STEADI Knowledge test scores. Table 2 demonstrates the summary of the factorial ANOVA results for the main effects interaction. In the education intervention arm, there was a statistically significant increase in mean scores between the immediate pre-education and immediate post-education test scores. The mean (M) increased by 1.19 (pre-education M 4.31, post-education M 5.5), p 0.02.

Table 2

Factorial ANOVA Summary

Tests of Between-Subjects Effects (Education * Time)
Dependent Variable	F	P-value
STEADI Knowledge Test Mean Score	.899	.348
CPD Reaction Questionnaire Mean	1.790	.187
Fall Risk Screening	.418	.519
Positive Screenings	.232	.631
Fall Risk Assessment	7.494	.012
Fall Risk Recommendations	.277	.604

STEADI Intent to Use

The participants’ intention to use STEADI mean CPD scores were not statistically different. However, within the education intervention study arm, the mean increase immediately pre-education and post-education was statistically significant. The mean (M) increased by 0.64 (pre-education M 5.47, post-education 6.11), p. 0.01. A factorial ANOVA was conducted to compare the main effects of the STEADI implementation strategy, education, and the interaction between the education and time on the mean scores of the CPD Reaction Questionnaire. Table 2 shows the summary of the Factorial ANOVA results for the main effects interaction.

STEADI Use Behaviors

A factorial ANOVA was conducted to compare the main effects of the STEADI implementation strategy, education, and the interaction between the education and time on the participants’ behaviors of screenings, positive screenings, assessments, and appropriate recommendations. Table 3 outlines the overall behaviors measured and Table 2 shows the factorial ANOVA summary of results. Overall, there was not a difference with statistical significance in fall risk activities, including screening, positive screenings, and appropriate interventions or recommendations between the study arms; however, there was a statistical significance in the interaction between time and education on assessments, p 0.01.

Table 3

Primary Care Clinic Behaviors Summary

Clinic	Total Number of Patients	% Patients 65+ (N)	% 2 Question Screenings Performed (N)	% Positive Screenings (N)	% STEADI Screenings Performed (N)	% Assessments (N)	% Recommendations (N)
Control – Pre	369	14.4% (53)	69.8% (37)	16.2% (6)	7.5% (4)	33.3% (2)	16.7% (1)
Control – Post	457	16.8% (77)	72.7% (56)	19.6% (11)	3.9% (3)	81.8% (9)	18.2% (2)
Education – Pre	288	13.9% (40)	55.0% (22)	13.6% (3)	5.0% (2)	100% (3)	0
Education – Post	254	12.2% (31)	71.0% (22)	22.7% (5)	6.5% (2)	40% (2)	20.0% (1)

Discussion

Although there was a statistically significant difference in pre-education and immediate post-education STEADI Knowledge Test and CPD Reaction Questionnaire mean scores, there was no significant difference in the longer-term mean scores. There is not sufficient evidence to state that the STEADI education alone was adequate, over time, to increase the healthcare providers’ knowledge or intent to use regarding STEADI. Further studies using other facilitation strategies should be performed to see what the most efficient process for implementation of STEADI will be.

The STEADI Education implementation strategy may not be sufficient to create a significant change in the rate of screenings, positive screenings, assessments, and recommendations for interventions. Further studies using other facilitation strategies should be performed to see what the most efficient process for implementation will be. The individuals implementing the new process should perform a pre-implementation plan to better understand the organization/clinic’s culture and understanding of the use of the new guideline in order to develop the best facilitation strategies to be utilized.29–31

Even after the usual implementation of the STEADI in the EMR, the majority of the clinic staff continued to use the old, two-question fall-risk screening format. This may be in part because there were no de-implementation strategies used. De-implementation is the process of using strategies to prevent the continuation of utilizing old, non-evidence based practices rather than the current practice being implemented.32–34 Further studies should include the use of de-implementation strategies such as actively discouraging prior approaches and removing competing screening tools to determine efficacy in combination with implementation strategies.

The lack of sustainability in both knowledge and intention to use STEADI was identified in the STEADI education implementation strategy arm between the statistically significant increases in means immediately pre and post-education versus the 30-day post-strategy scores. This correlates with the findings in other studies that education alone may not be sufficient to promote long term, sustained change.31,35 Sustainment requires the integration of multiple domains, including political support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, and strategic planning. A Program Sustainability Assessment Tool for use by facilities has been created and tested.36 Additionally, the Exploration, Preparation, Implementation, Sustainment (EPIS) framework provides guidance on sustainable implementation. Future studies testing program sustainability can use this tool or framework. The EPIS framework takes into consideration elements that have already been identified as important, including buy-in from all levels of the organization, innovation factors including how the primary care staff will use the new EBP, and support from program champions.35

Lastly, a pre-implementation assessment can be utilized to determine the clinic’s interpretation of what their needs are about the program that is going to be implemented.37 This can aid in identifying any barriers that need to be overcome or facilitators that can be used for the implementation. A pre-implementation assessment was not performed for this study because this study was looking to compare the usual implementation process versus the addition of implementation strategies of education and dissemination of written materials.

Conclusion and Recommendations

Educational efforts, particularly the dissemination of written materials and use of an online education module, to increase the use of an evidence-based fall screening tool in primary care did not increase the overall knowledge, intention to use, or use of the STEADI toolkit. Therefore, we recommend that future efforts focus on additional facilitation techniques including feedback, facilitators, or a different evidence-based tool. Facilitators can include clinic-based super users that have received additional education. Lastly, the use of an evidence-based screening tool should incorporate the clinic’s electronic medical record and may automate a fall-risk score based on data that is inputted into the patient’s medical record. Lastly, prior to the implementation process beginning, a pre-implementation survey should be performed to gage the clinic’s barriers and facilitators for implementing the new EBP.

Limitations

The small sample size due to lack of participation by all primary care clinics in each study arm and use of a single study site could lead to a type II error. This could lead to a lack of the ability to generalize the results. Additionally, the PI did not collect data on the participants’ age or gender. This lack of demographic information could potentially lead to a confounder on the generational differences in the use of electronic medical records or change.