Training upcoming academicians through interviews of pharmacy resident teaching certificate leaders.

BACKGROUND: Discovering methods of Residency Teaching Certificate Programs (RTCPs) will allow for collaboration in developing best practices to ensure both high quality of programming and outcomes for participants.
OBJECTIVE: The primary objective of this project is to describe and compare how RTCPs are conducted in the state of Ohio. Secondarily, to identify current practices in assessing RTCPs in both programmatic effectiveness and individual resident teaching outcomes.
METHODS: The seven coordinators of the seven Ohio RTCPs (n=7) were contacted via email and asked to participate in an IRB-approved interview, either in-person or telephonically. Standardized questions were developed to inquire about six categories of interest: demographics/background, administration/logistics, content, assessment of the resident, program financing, and program continuous quality improvement (CQI). All seven programs participated in interviews. Data was coded by multiple members of the research team for presentation in aggregate form.
RESULTS: RTCPs include seminar days at the respective pharmacy colleges; however, the number, length, and content of seminars vary. The majority of programs (n=5) stated using inherited curriculum and materials passed down from previous coordinators. While each RTCP requires participants to submit a teaching portfolio, only three of seven programs assess the summative portfolios. All programs (n=7) award participants a certificate based on completion of requirements without a defined minimum performance standard. Two programs are collecting participant feedback after every session for CQI however no programs are completing an annual programmatic assessment of resident outcomes. The majority of coordinators (n=7) are interested in collaborating and sharing "best practices" between RTCPs in the state.
CONCLUSIONS: Although published and available resources exist surrounding the development and delivery of RTCPs, in Ohio, their use varies greatly. The most striking outcomes highlighted the lack of resident and program assessment of outcomes in RTCPs. The research has brought forth ideas of ways to improve these programs through resident assessment, program assessment and also leads to reflection and innovation around the best way to deliver these programs. Copyright: © Pharmacy Practice.

INTRODUCTION

As the number of pharmacy schools continues to rise, there has been an increasing need for qualified pharmacy faculty and, thus, programs to develop pharmacy educators.1 The American Society of Health-Systems Pharmacists (ASHP) accreditation standards for postgraduate year one (PGY1) pharmacy residency programs outline “teaching, education, and dissemination of knowledge” as a required competency area within programs; however, methods to develop competency in this area are not clearly defined.2,3 Coupled with the gap in qualified pharmacy educators, “teaching residents to teach” is a vital component of the residency experience. Resident teaching and learning curriculum (TLC), often referred to as resident teaching certificate programs (RTCPs), were developed to educate residents on academia related content and provide various teaching experiences, providing a structured approach to training future educators. Such programs have existed for twenty years and have been found to be effective in improving resident confidence in teaching, in addition to providing multiple other benefits both in service as preceptors and in traditional academia.4-6 Additionally, Gettig and colleagues found the majority of Residency Teaching Certificate Program (RTCP) participants felt the experience aided in obtaining their current position.7

Despite a need for well-prepared educators and the availability of RTCPs, there is no standard structure or assessment for such programs. A White Paper published by the American College of Clinical Pharmacy (ACCP) in 2013 outlined guidelines for residency programs to incorporate various academic experiences, including teaching certificate programs.8 In 2014, the Task Force on Student Engagement and Involvement of the American Association of Colleges of Pharmacy (AACP), in conjunction with ASHP, published 12 best practice recommendations to incorporate in postgraduate education experiences.9 Both of these publications provide guidance to teaching certificate programs without giving specific requirements or outcome criteria for residents or programs (Table 1).

Table 1

Best practice recommendations for teaching and learning curriculum programs

AACP/ASHP			ACCP White Paper
1. Systematic experiences in teaching and learning should collectively be termed Teaching and Learning Curriculum (TLC) programs.	7. Participants in a TLC program should be evaluated at regular intervals.		1. Pharmacy residency programs providing teaching certificate programs should be affiliated with a school/college of pharmacy or an academic institution.
2. TLC programs should be facilitated through a school/college of pharmacy or other postsecondary institution.	8. Participants should develop and maintain a teaching portfolio.		2. Participants should develop and/or revise a personal teaching philosophy as part of a teaching portfolio documenting all teaching experiences.
3. TLC program content should include a discussion of specified core topics at a minimum.	9. TLC programs should have adequate personnel and institutional resources.		3. Participants should be assigned a specific teaching mentor for guidance and evaluation of experiences.
4. TLC program content should be delivered using different teaching styles.	10. TLC programs should incorporate a system of assessment to enhance ongoing programmatic improvement.		4. Participants should actively participate in a series of core pedagogy seminars.
5. TLC program content should be delivered commensurate with the learning level of the participants.	11. TLC program offerings, features, and participant obligations should be clearly described in all recruitment materials.		5. Participants should have exposure to different teaching experiences (ex. formal lectures, experiential precepting, small group discussion, and patient case development).
6. Participants in a TLC program should be expected to gain experience in a variety of educational modalities.	12. A formal external validation process for TLC programs should be established.		6. Participant performance should be assessed and clear criteria should exist for successful completion of the program.

A study by Strang and Baia examined 19 publications that described 20 RTCPs and one faculty development program. The majority of programs (95.2%) included both didactic and experiential components. Only 12 programs described an evaluation of the program including outcome data. All of these programs collected information on participants’ reactions to the program, while only seven collected data on improved confidence in teaching and only one program collected information on knowledge improvement. However, nine programs collected information on participants’ behavior change, such as obtaining academic employment or demonstrating changes in teaching skills and behavior. While the study outlined several trends in the components of various programs, it also exposed several gaps in the literature related to RTCPs, particularly in regard to outcomes data for these programs.10

Despite guidance from multiple publications on thoughtful design of RTCPs, there is little indication that such guidance is routinely used as evidenced by Stang and Baia’s variance in results.8,9,11 A shared definition of successful RTCP completion is lacking and evaluation of outcomes related to teaching effectiveness are not universally collected or assessed. For example, residents can participate in and complete an RTCP simply by participating in certain activities, but without thoughtful evaluation of outcomes by program coordinators. This lack of assessment may perpetuate knowledge gaps within academic practice. Moreover, programmatic assessment to continually improve RTCPs is not standardized across programs, potentially leading to a lack of quality “control” for programs. Although RTCPs have the potential to fill a gap for qualified educators within the academy, thoughtful design, assessment, and programmatic evaluation is essential to ensure the integrity of such experience. Currently RTCPs themselves are not held to a shared national standard based on provided guidance. Cataloging current RTCP structure and assessment, identifying gaps in knowledge and achieved outcomes, and sharing best practices aids in characterizing the current landscape of programs. Understanding current RTCP offerings and potential areas for improvement will promote the development of effective experiences for residents and impactful preparation for future educators.

Study authors coordinate one RTCP in Ohio, which has been in existence for ten years. In order to assess and improve current practices, study authors wanted to determine alignment with the other six Ohio RTCPs as well as currently available literature and guidance. The primary objective of this project is to describe and compare how RTCPs are conducted in the state of Ohio. Secondarily, the authors hope to identify current practices in assessing RTCPs in both programmatic effectiveness and individual resident teaching outcomes. Discovering methods of other RTCPs will allow for collaboration in developing best practices to ensure both high quality of programming and outcomes for participants.

METHODS

This study received exempt review approval from the Ohio Northern University Institutional Review Board to conduct a standardized, question-based interview of Ohio RTCP coordinators. Contact information for each of the seven different coordinators from the seven different programs was obtained through university or college websites or department chairs. Either current or immediate past coordinators were contacted between November 2018 and January 2019 to gauge interest in participation. If interested, each coordinator was then contacted via email or in person again in February 2019 and asked to participate in an interview and identify a date and time to do so, either telephonically or in-person. Coordinators from all seven of the Ohio colleges of pharmacy were included, provided consent, and elected to participate. No programs were excluded. The individual interviews took place in February 2019, with each interview taking 30 - 60 minutes to complete. Coordinators were asked to answer the questions to the best of their ability. Six interviews took place via telephone and one interview was completed in person, as the coordinators for Ohio Northern University are a part of the research team for this project. At least two members of the research team were present for each telephonic interview. A list of standardized questions was developed to focus on six categories of interest: demographics/background, administration/logistics, content, assessment of the resident, program financing, and program continuous quality improvement (CQI). Use of a standardized question list ensured that each coordinator was asked the same questions. Telephonic interviews were audio recorded and researchers typed answers during the interview into an electronic Google Doc shared amongst the research team.

Following the interviews, one of the researchers reviewed the audio-recording and updated the electronic documentation of responses to ensure accuracy and completeness for each interview. Once reviewed, members of the research team organized the responses into a summary chart categorized by question and school. Utilizing the organized data for each interview question, members of the research team coded responses by similarities without identifying the individual RTCP. Data was compiled in an aggregate form and analyzed using descriptive statistics where appropriate.

RESULTS

Data regarding the length of program existence and current participating residents and residency programs can be found in Table 2. Most RTCP coordinators were unable to provide exact data for the number of residents that have completed the program thus far in program existence. Most RTCPs only have PGY-1 resident participation; however, there have been occasional PGY-2 residents, fellows, or preceptor participants. No programs have included or collaborated with other healthcare professionals outside of pharmacists, based on received responses.

Table 2

Current individual residency teaching certificate program information; n (%)

Length of RTCP Existence
0 - 5 years	1 (14.3)
6 - 10 years	3 (42.9)
11 - 15 years	2 (28.6)
> 15 years	1 (14.3)
Number of Participants Currently in Individual RTCPs
0 - 15 participants	3 (42.9)
16 - 30 participants	2 (28.6)
31 - 45 participants	0 (0)
46 - 60 participants	2 (28.6)
Number of Residency Programs Participating in Individual RTCPs
6 - 10 programs	4 (57.1)
11- 15 programs	1 (14.3)
> 15 programs	1 (14.3)
Length of RTCP Existence
0 - 5 years	1 (14.3)
6 - 10 years	3 (42.9)
11 - 15 years	2 (28.6)
> 15 programs	1 (14.3)

Data regarding the administration and logistics of the Ohio RTCPs can be found in Table 3 and Table 4. The number of faculty that direct and participate in RTCPs vary with each program. Generally, RTCPs were organized by one coordinator (n=4) and additional college of pharmacy faculty members, ranging from 1-12 people, serve as mentors or guest speakers (n=7). Most content is delivered by college of pharmacy faculty; however, some programs incorporate an administrative assistant to help with organization (n=2), an academic technology department (n=1), or faculty from other departments or colleges within their university (n=2). The faculty that participate in the program are not incentivized individually and most coordinators commented that their role is a voluntary position that is done “out of the goodness of their heart.”

Table 3

RTCP program logistics and financing; n (%)

Characteristic	Yes
Seminar days occur at respective college of pharmacy	7 (100)
Standardized program for all participants	4 (57.1)
RTCP requires submission of a teaching portfolio	7 (100)
RTCP pairs all residents with a teaching mentor	4 (57.1)
RTCP faculty provided incentive to participate in the RTCP	0 (0)
Financial assistance provided to RTCP from the residency program	1 (14.3)
Financial assistance provided to RTCP from the university or college	4 (57.1)

Table 4

RTCP seminar day logistics

Characteristic	Range
Number of seminar days	2 – 13 sessions
Length of seminar days	2 – 8 hours
Number of faculty who participate	1 – 12 faculty

The content, design, and logistics of Ohio RTCPs vary greatly. Generally, each Ohio RTCP provides content through seminar days. The majority of programs front-load the content in the earlier part of the residency year (n=6). In addition, some RTCPs meet quarterly throughout the residency year (n=2) while some programs meet every month (n=1). Most programs stated that they provide full seminar days in the earlier months of the residency year (n=5) and additional seminar days may consist of shorter meetings. Content delivered during seminar days was delivered through interactive discussions (n= 5), lectures (n=1) and online modules (n=1). The most common content topics were teaching philosophies (n=7), assessment of learners (n=6), lecture delivery methods (n=6), learning styles (n=3), and careers in academia (n=3). Programs delineated and distinguished topics differently so there were some inconsistencies in how seminar topics were reported and inconsistencies in the number of hours reported on each topic. For example, teaching methodologies time varied from 1 hour to 16+ hours of content for the residents.

All of the programs required resident completion of a teaching portfolio (n=7) but actual teaching components varied from program to program. Only 3 programs stated that the teaching portfolio was evaluated by a faculty member. Hours spent teaching in the respective college of pharmacy varied from a one-time 2 hour requirement to a semester long teaching requirement incorporating 16+ hours of classroom teaching. There was no consistency in the number of hours or classroom setting for resident teaching experiences. Residents also had a variety of other requirements for each program that included monthly case development, assessment of students in patient simulations, involvement in skills based assessments, and precepting experiential education students.

Not all RTCPs pair participants with a teaching mentor (Table 5). One program even shared receiving “push back” from other faculty members and department leaders, while another program shared that typically residents are paired with each other. All RTCP coordinators confirmed that some past participants have obtained a position related to academia, however none were able to provide an exact number. Most feel that the percentage is low, but acknowledge that not all who participate in the RTCP are seeking full time positions in academia.

Table 5

Teaching mentor pairing of participants by RTCP coordinators

Number of programs pairing participants with a teaching mentor	n (%)
All participants	4 (57.1)
Some participants	2 (28.6)
No participants	1 (14.3)

When interviewees were asked about resources used to create the program, apart from the ACCP white paper, results varied. The majority of programs (n=5) stated using inherited materials passed down from previous coordinators. One program referenced the AACP/ASHP recommendations for postgraduate pharmacy experiences in education and another program cited using previously published literature and available syllabi.

All programs were asked about financing related to the offering of their RTCP. Most programs (n=6) do not receive financial assistance or compensation from residency programs or participating residents. Only one program requires the participating resident to pay a fee; however, some programs noted that compensation comes in informal or non-monetary forms, such as teaching for the college or a resident “sponsor” that pays for the resident’s dinner during the final banquet. Similarly, programs were asked if the college or university provides financial support for the program or if there is a specific budget for the RTCP. Generally, most programs do not have a specific budget (n=5) or stated their budget was absorbed in the college’s budget as a line item without a set amount (n=2). It was implied that most colleges (n=6) did cover the cost of food and beverages for the seminar days, parking, room reservation fees, or copying costs. In summary, most RTCPs do not have a set budget or receive financial compensation from residents, residency programs, or their affiliated colleges for delivery of the program.

All programs noted that resident performance was evaluated, but the methods utilized varied among the programs. A summary of evaluation methods are summarized in Table 6. All interviewees stated that residents earned a certificate based upon the completion of program requirements, however, only 3 programs stated that the summative portfolios were evaluated. The majority of RTCP mentors reviewed residents’ teaching materials (n=5) but there were inconsistencies in the use of an evaluation tool for resident teaching materials (n=2) and resident teaching (n=3). As shown in Table 7, CQI is not standardized among the Ohio RTCPs. CQI of programs range from an informal annual review (n=3) to written feedback from the participants after each seminar (n=2). One program noted getting feedback from a focus group of Residency Program Directors (RPDs) as a means of program improvement. Of note, all programs noted that they did not perform an annual programmatic assessment of resident outcomes (n=7). Most programs in Ohio (n=5) indicated specific interest in collaborating within the state of Ohio to develop a more standardized RTCP. The interviewees with reservations in creating a statewide program noted concerns with the potential loss of RTCP individualization, creativity, and flexibility.

Table 6

Assessment of residents

Method	n (%)
Certificate earned based on completion of requirements	7 (100)
Review of teaching materials by mentor	5 (71.4)
Evaluation tool during teaching	3 (42.9)
Evaluation of summative portfolio	3 (42.9)
Evaluation tool of teaching materials	2 (28.6)
Review student feedback	2 (28.6)
Assessment of reflection materials	1 (14.3)
Exit survey	1 (14.3)

Table 7

Teaching certificate program continuous quality improvement

Type of CQI	n (%)
Informal annual review	3 (42.9)
Written feedback after each seminar	2 (28.6)
Grant funded one time programmatic review	1 (14.3)
No regular assessment	2 (28.6)
University level assessment	2 (28.6)
Exit Interview	1 (14.3)
Annual programmatic assessment of resident outcomes	0 (0)

DISCUSSION

This study sought to identify ways to improve Ohio Northern University’s RTCP to ensure the program is preparing residents to meet the educational needs of a dynamic pharmacy landscape in accordance with national recommendations. By describing how Ohio RTCPs are conducted and identifying current practices in RTCP and resident assessment, the authors hoped to discern ways in which RTCPs could collaborate more productively to demonstrate that these programs are useful and effective in training future educators. As previously described, there is a lack of standardization in the delivery, assessment, and outcomes of RTCPs on a national level. Not surprisingly, there were inconsistencies found among the 7 Ohio RTCPs in terms of administration, logistics, content, program financing, assessment of participants, and program CQI. Of the various topics investigated, two of the most compelling areas in need of immediate and further development focus around assessment. The first being a lack of defined outcome measures surrounding resident effectiveness as educators and the second centering around the assessment of RTCP effectiveness in the form of CQI.

In hopes of gaining an understanding of how Ohio RTCPs determine if their residents are successful educators, this study explored how programs evaluate their residents as competent educators. Interviewers identified that there is not a standard definition of what it means to have “successfully” completed an RTCP. All programs stated that if the participant had completed the checklist of requirements, then the participant would receive the certificate. There was not a minimum performance standard for any of these requirements and no interviewees referenced any of the available validated tools or assessment rubrics used. This highlights the need for some means of distinguishing and measuring resident effectiveness as an educator prior to being awarded the teaching certificate. This would hopefully add more meaning to actually having earned the certificate and assure that residents are adequately prepared to educate future pharmacists. RTCPs could potentially begin creating benchmarks that program participants must meet prior to being awarded a teaching certificate, instead of simply completing tasks or projects. Additional studies should be conducted to identify what outcomes should be measured to determine resident effectiveness or success in the different requirements of the RTCP (for example, in lecture delivery). By ensuring residents are adequately prepared as effective educators, it advances the knowledge, skills, and qualities of pharmacists that educate our future students in the traditional classroom setting and in experiential education. Locally, RTCPs are not currently evaluating resident outcomes on an individual level for the resident or even on a program level. Furthermore, RTCPs are also failing to self-assess in regards to their program’s utility and effectiveness as an educator preparation tool and offers an opportunity to innovate and expand in the area of resident assessment.

While the majority of Ohio RTCPs have been in existence for six years or longer, most have been historically handed down to current coordinators and have often lacked specified outcomes and a routine CQI process. This has, in part, led most delivered content to be derived from academic inertia rather than utilizing evidence-based recommendations or published consensus statements. It was hoped that more RTCPs would have evidence-based curriculum and program design, however, the opposite was found through the interviews. This research emphasized the need to revisit the existing published recommendations for program requirements and programmatic assessment. Among the Ohio RTCP leaders interviewed, CQI was consistently noted as an area of opportunity for further development. RTCPs may benefit from a standardized process of implementing annual CQI, as well as a toolkit of resources that may outline the most effective means to do so. Additionally, it would be helpful to evaluate and share the strengths and limitations of RTCPs at a program level. General trends may become apparent at a program level that could be addressed collaboratively with enhanced seminars or teaching experiences. Through collaborative efforts, perhaps program leaders could identify outcome measures to show an RTCP participant’s maturation as an educator while simultaneously improving the utility and functionality of the actual RTCP itself. Future research into effective means for CQI for RTCPs could be disseminated along with potential repositories of resources for all RTCPs to adopt and adapt for their institution. Hopefully in the future, this research coupled with more broad research into effective means for preparing future academicians, will lead to clearly defined and consistent outcomes for residents, greater recognition of programs, and greater collaboration among those who coordinate RTCPs. Ultimately, this should lead to better preparation of residents to tackle the challenges and opportunities in educating future pharmacists.

The biggest limitation of the study is that not all coordinators were able to provide concrete information regarding their RTCP, specifically relating to the historical elements such as number of total participants, collaboration efforts, and academia job placement after program completion. A few limitations exist in regards to the interview process. All coordinators did not receive a copy of the interview questions ahead of time, which prevented answering all questions during the phone call. Although means were provided for coordinators to provide additional responses via email, not many were received. Another limitation is that one telephonic interview was not recorded, which prevented the research team from reviewing the electronic responses for accuracy.

Due to the specific geographic focus on Ohio for this study, information provided within may not be generalizable to RTCPs outside of Ohio; however, these results do serve as a pilot study to investigate the dissemination of RTCPs nationally. These interviews serve as a starting point for subsequent work regarding how RTCPs are conducted outside of the state. Future studies should be conducted at a national level to review, compare, and contrast RTCP logistics, content inclusion and delivery, participant requirements, and assessment of both participants and the program itself.

CONCLUSIONS

Even within the same state, RTCPs vary widely in terms of logistics, content delivery, and assessment of participants and CQI of programs. This focused study helped to identify current similarities, but also highlighted the vast differences between programs in one state. The most striking and meaningful finding of the study surrounded the lack of both resident and program assessment found in RTCPs and points to a need for significant work in these areas. Moving forward there is opportunity to expand the way that programs are assessing resident success as well as program success. The research has brought forth ideas of ways to improve these programs through resident assessment, program assessment and also leads to reflection and innovation around the best way to deliver these programs. More research and collaboration in this area would lead to opportunities for more fruitful collaboration between RTCPs coordinators at a national level to help improve these programs in preparing future educators and also lead to potential innovation around these programs. Potential future research into more interprofessional delivery of the programs, varied tracks of the programs (ie experiential versus didactic teaching tracks) or even potential to move these programs prior to residency training. Further research and collaboration could improve these programs and ultimately better prepare future pharmacy educators.