Use of a CPD Plan Template with SMART Goals as Part of a Diabetes Pharmacotherapy Module.

OBJECTIVE: To evaluate the effectiveness of a Continuing Professional Development (CPD) plan template used in Fall of 2017 on quality of SMART goal development and student quiz scores. INNOVATION: The gap in time from when pharmacology is taught and when it is applied has contributed to poor student retention and performance in the diabetes pharmacotherapy course. To address this gap, the diabetes pharmacotherapy learning sequence was redesigned and included a self-assessment (pre-test), and the completion of a "CPD plan template", which involved writing 1-3 SMART goals for each question missed on the pre-test. Following sequence completion, students took an identical post-quiz. Pre- and post-quiz scores were compared. Quality of CPD plan SMART goals was evaluated. KEY
FINDINGS: The CPD plan template was completed by 98% of students. The majority, 62.5% of students, wrote SMART goals at the intermediate or good level, while 37.5% were evaluated as needs improvement. The average pre-quiz score was 7.4 points and average post-quiz score was 17.1 points with an average improvement of 9.8 points (p<0.0001). There was a statistically significant improvement for top 25% post-quiz scoring students who wrote "good" SMART goals compared to those who wrote goals needing improvement (p= 0.002). For students scoring in the lowest 25%, students with goals needing improvement scored higher than those with intermediate quality goals (p< 0.04). NEXT STEPS: It may be beneficial to introduce CPD to students sooner, as well as teach students more intentionally how to create and use SMART goals to improve learning. Finally, instructor follow up with students regarding use of their plan during a learning sequence may have additional benefit. © Individual authors.

Description of the Problem

The Accreditation Council for Pharmacy Education (ACPE) outlines Continuing Professional Development (CPD) as a self-directed, ongoing, systematic and outcomes-focused approach to lifelong learning that is applied into practice. The CPD cycle has several components: Reflect, Plan, Learn, Evaluate, Apply, and Record/Review.[1] As stated in the ACPE Standards 2016, pharmacy schools must provide an environment and culture that encourages self-directed lifelong learning. The CPD process may inspire students to review and consider feedback from exams, quizzes, or reports during the reflective phase, leading to action being taken to address that feedback during the planning and implementation phases.[2] The necessity for CPD as a model for healthcare professionals in the United States to retain professional competence has been well documented, [3] however the adoption of CPD by pharmacists in the United States has been slow. While many reasons for this exist, one reason may be lack of introduction and immersion of pharmacy students in the model during their training.[4] At the Samford University McWhorter School of Pharmacy (McWhorter), students are taught the principles of CPD throughout the first year as part of the professional portfolio. However, components of CPD were not incorporated into specific curricular content, such as pharmacotherapy.

The pharmacotherapy of diabetes is taught over eight days (10 contact hours) in the third course of a four-course sequence in the Fall semester of the third professional year. Pharmacology of diabetes medications is taught at the end of the spring semester of the second professional year. This 4-month gap in time is thought to have contributed to poor student retention of expected medication knowledge and, subsequently, poor student performance in the pharmacotherapy course. To address this retention problem the pharmacotherapy learning sequence was redesigned from didactic lecture with practice cases to a multi-modal active learning model. Initially, students were asked to then write SMART one to three SMART goals for each question missed. The instructor provided an example of a good SMART goal and a poor one in an email. Faculty review found that only 6% of student plans were “good,” while 49% of student’s plans would have been rated as “needed improvement As a result, it was determined that students required more support in progressing through the CPD process and a template was devised. This report describes the effectiveness of a template used to guide student developed CPD plans as a means to support student use of a CPD approach to ensure retention of previously covered disease specific content in a pharmacotherapy course.

Description of the Innovation

Prior to exposure to any diabetes pharmacotherapy content, third-year students were administered a pre-quiz to establish their baseline knowledge of diabetes. The twenty-two-item multiple choice pre-quiz was administered via a learning management system (LMS), Moodle (Perth, Australia).

Questions were mapped to specific learning objectives. Upon completion of the quiz, students received feedback on correct and incorrect responses and the corresponding learning objectives. Students were not given answer keys. Questions evaluated both retention of pharmacology knowledge from the second-year pharmacology course, as well as establishing baseline understanding of new content yet to be taught. Questions were selected from a pool of previously used and validated exam questions. Table 1 delineates the diabetes series learning objectives assessed by each question, as well as question statistics.

Table 1.

Pre/Post-Quiz Statistics and Correlated Learning Objectives

Learning Objective	Number  of Questions	Pre-Quiz Average			Post-Quiz Average			New Content (N) or Retention (R)
		% Correct	Disc. Index	Pt. Biserial	% Correct	Disc. Index	Pt. Biserial
Compare and contrast, both within and between classes, the different oral and injectable agents used in the treatment of diabetes.	11	47	0.5	0.21	76	0.03	0.02	R
		52	0.38	0.12	50	0.41	0.30
		17	0.44	0.29	63	0.25	0.19
		32	0.38	0.13	48	0.59	0.42
		18	0.09	0.00	55	0.44	0.30
		30	0.16	0.02	69	0.25	0.25
		37	0.56	0.27	64	0.28	0.24
		49	0.69	0.34	52	0.41	0.21
		11	0.16	0.10	71	0.16	0.1
		37	0.22	0.09	50	0.59	0.42
		34	0.38	0.13	74	0.19	0.22
Compare and contrast the different insulins regarding onset, peak, duration, OTC status, and use in patients with diabetes.	2	72	0.22	0.02	60	0.31	0.13	R
		55	0.81	0.35	69	0.25	0.27
Design initial insulin regimens in type 1 and type 2 diabetes patients.	3	18	0.31	0.23	49	0.59	0.44	N
		23	0.34	0.14	56	0.43	0.33
		80	0.38	0.28	61	0.41	0.34
Individualize A1C goals, as appropriate, based on individual patient factors.	1	15	0.25	0.15	71	0.16	0.17	N
Appropriately adjust insulin regimens in type 1 and type 2 diabetes patients.	1	22	0.22	0.07	71	0.22	0.23	N
List and appropriately apply the screening, clinical presentation, and diagnostic criteria for diabetes.	1	20	0.38	0.24	67	0.25	0.22	N
Adjust and optimize a treatment plan for a patient with type 2 diabetes.	2	19	0.19	0.07	41	0.43	0.32	N
		11	0.06	0.00	51	0.41	0.34
List the major results of the ACCORD, ADVANCE, VADT, and UKPDS Follow Up, and DPP.	1	25	0.19	0.01	74	0.13	0.11	N

Following the pre-quiz, students were directed to review their results, and missed questions, as well as the corresponding learning objectives, and to use the CPD template. The CPD plan template was created because students in the Fall of 2016 demonstrated a general inability to self-assess and identify areas needing development, as well as the ability to write appropriate SMART goals. For each missed question, students were required to create one to three SMART (Specific, Measurable, Achievable, Relevant, and Time related) goals. The instructor provided both a good and a poor example of a SMART goal with associated rationales in an email to the students prior to opening the pre-quiz. Completion with submission of the CPD plan template was incentivized ($25 gift card drawing) and students had approximately one week to complete it and write their SMART goals. Completed CPD plan templates were downloaded into the LMS. (Figure 1). It was assumed that the course content would assist students in meeting any goals that they might individually articulate. Therefore, an individualized “plan” (e.g. desired learning activities, resources/support required) was not needed. It was intended that this self-assessment and the act of writing goals would aid students in preparing to watch for and attend to the upcoming content that was most important to their particular learning needs.

Figure 1.

CPD Plan Template

Evaluation of Effectiveness

One hundred and twenty-eight students were taught in the diabetes pharmacotherapy sequence. An assessment was submitted by one hundred and twenty-four students (96.9%); however, only ninety-six students (75%) created SMART goals. Ninety-nine students (77%) completed both the pre- and post-quiz. To evaluate effectiveness of the CPD plan template on student learning, the difference in scores from pre to post-quiz was examined. To be included in the analysis, students had to complete both the pre- and post-quizzes and have submitted their completed CPD plan template. Students completing only the pre- or only the post-quiz were excluded from the data analysis, as were students who did not submit a completed CPD plan template.

To assess the quality of student written CPD plan SMART goals, each students’ goals were categorized as Needs Improvement, Intermediate, or Good. “Needs Improvement” was assigned if no SMART goals were written, goals were not tied to specific learning objectives, or if goals had more than one error (i.e., the goal was not specific and measurable). “Intermediate” was used for those SMART goals tied to an objective, but had one error. “Good” was used for those SMART goals which were tied to specific learning objectives and had no errors. Upon assessment, 30 assignments (31.3%) were rated as “good”, 30 (31.3%) as “intermediate” and 36 (37.5%)as “needs improvement.” Difference in quiz scores between those who wrote SMART goals and those who didn’t, was assessed using a paired t-test. As good SMART goal quality is necessary, an assessment of SMART goal quality was undertaken by instructors. An evaluation of how SMART goal quality influenced post-quiz scores was also completed. Microsoft Excel (Redmond, WA) was used to conduct the t-tests. The study was approved by Samford University Institutional Review Board.

Critical Analysis

There was a mode of eight and a mean of 10.1 SMART goals written per student. The average pre-quiz score was 7.4 ± 2.5 points and average post-quiz score was 17.1 ± 3.4 points with an average improvement of 9.8 ± 5.7 points (p<0.0001; 95% CI 4 to 15.4). While student scores improved from pre- to post-quiz, there was no statistically significant difference in average quiz scores found between those students who wrote goals and students who did not for any learning objective.

To determine the association between SMART goal quality and quiz scores, student quiz score differences were evaluated by category. The overall average pre- to post-quiz score differences for Needs Improvement, Intermediate, and Good were 10.5, 9.7, and 8.5 points, respectively. This indicates that students with “Needs Improvement” SMART goals had the highest change in score while students with the “Good” SMART goals improved the least. With those who had “Good” SMART goals having the least improvement while “Needs Improvement” goals improving the most, it does not appear that SMART goal quality was associated with an improved quiz score from pre-to post-quiz.

To determine if goal quality impacted student overall knowledge of the topic at the end of the diabetes series, a post-quiz score analysis was completed. There was a statistically significant improvement for students scoring in the top 25% on the post-quiz who wrote “Good” SMART goals compared to students, also scoring in the top 25%, whose goals were rated as “Needs Improvement” (p= 0.002; 95% CI 21.2, 20.8 to 21.6). Therefore, for high performing students, preparation of good SMART goals may have improved learning of the content. In contrast, a statistically significant eight-point decrease was found between students in the bottom 25% who had SMART goals categorized as “Intermediate” compared to those also in the bottom 25% with goals of “Needs Improvement” (p= 0.04). Hence for the more poorly performing students, time taken to create SMART goals, which may not ultimately have been used effectively, may have contributed to less learning compared to their counterparts who did not take the time to try and write quality SMART goals.

Next Steps

The template assisted students in creating SMART goals, with 60 out of 96 (62.5%) of students writing intermediate or good SMART goals. Despite the improvement in student plan quality, there was not a difference in score between students who used the template and created SMART goals compared to students who did not. Additionally, comparisons between top and bottom performance by goal quality yielded mixed findings. There are several potential reasons for these results.

First, the CPD approach may need to be introduced earlier, and include more intentional reinforcement, as well as more direct instructor support on the use of the created plan.[5-7] Patterson examined student perceptions of the use of learning plan tools and found that most students thought they were successful to very successful in creating their learning plans with a tool to guide them.[5] In this study, the implementation of a CPD plan template tool resulted in a ten-fold increase in the number of students’ plans which met good to intermediate criteria (62.5%) compared to the previous year when the template was not used (6%). Other investigators have examined student goal development. In a two-year study, a pharmacy class was followed from first to second year and SMART goals were evaluated utilizing a rubric. the average score was 79.2% with the lowest scores observed in the ‘Measurable’ and ‘Specific’ ratings, with average scores of 44.8% and 68.8%, respectively.[6] In the second year, the creation of SMART goals was easier for the students, in part due to a change in teaching strategy which included examples of SMART goals and a thorough presentation on how to effectively write them.[7] Further presentation about how to create and use a CPD plan with SMART goals may be warranted.

Another factor, which may have had significance, is that the CPD plan was incorporated into a third professional year classroom-based diabetes learning sequence. A study by Tofade in 2012 noted faculty did not find the majority of P3 students saw the benefit of CPD.[7] The placement of the first CPD plan in the third course of a four-course sequence may also have impacted student acceptance of this activity. If students didn’t see the value in creating the plan, it would be unlikely they would utilize it. In addition, most students have developed their study habits and methods before encountering the diabetes content. While students created a plan and SMART goals, they may never have executed the plan or looked at it again once they had submitted it.

Participation in the pre- and post-quizzes and CPD plan creation were voluntary and incentivized as the course sequence evaluation model did not allow for points to be assigned to the work. Students may have taken the assignment more seriously if points had been assigned and their effective use of the CPD plan template was evaluated. Further, the lack of instructor follow-up with students related to how they should use the created plan may also have led to students viewing it as unimportant. Both issues could have skewed student perception, use of the plan, and therefore, the data presented in this report. In addition, this report provides no information on the time students needed to complete the plan as this was not assessed as part of the study. The amount of time students took to prepare their plan could also have impacted the results presented.

Conclusion

The CPD Plan template assisted students in evaluating their knowledge and building a CPD plan with SMART goals. Use of the template was associated with a ten-fold increase in the number of intermediate to good SMART goals; however, additional training may be needed as 32% of students’ goals were rated as needs improvement. The effects of the intervention on knowledge are mixed. Earlier and further instruction with examples are likely needed to assist students in seeing the value of and effectively executing CPD.