Effect of the international pharmacy education programs: A pilot evaluation based on Kirkpatrick's model.

Globalization has attracted much attention to universities over the past decade. The aim of this study was to evaluate the effect of the United States-China international pharmacy education programs (IPEP) in China.Kirkpatrick's model of evaluation was used to evaluate IPEP from reaction and learning levels. In the reaction level, a questionnaire survey with a Likert scale was used. In the learning level, data from "Assessment Record of Advanced Clinical Pharmacy Practice of Peking University" were collected. Cronbach' α coefficient of reliability was calculated, principal component analysis and independent t-test were conducted.All of the students who attended IPEP (n = 36) completed the questionnaire survey. The scores of benefits were increased in 4 categories, including "Clinical practice competency improvement" (mean ranking [MR] = 3.11 points), "Understanding of doctor of Pharmacy education mode" (MR = 3.48 points), "English competency improvement" (MR = 3.64 points) and "International collaboration" (MR = 3.92 points). Meanwhile, the overall satisfaction was relatively high with the IPEP (MR = 4.22 points). In the learning level, a total of 22 records was obtained. Students who attended (n = 5) the IPEP achieved higher scores than those did not attend (n = 17) in the assessment records, although no statistical significant differences were observed. Personal in-depth interviews further supported the overall benefit of IPEP.The Kirkpatrick model of evaluation can be used for IPEP. The benefit and satisfaction of students attended the IPEP were high in the reaction level; even though no statistically significant difference was shown in the learning level, higher scores were still demonstrated.

Introduction

Globalization has attracted attention to universities over the past decade, with an increasing emphasis on global health programs and international experiences.[ The American Association of Colleges of Pharmacy identified “globalization of pharmacy education” as a major initiative in 2008 and required its members to play in the area of global health in 2009.[ Meanwhile, pharmacy schools around the world have been active in pursuing international programs and partnerships. Pharmacists are encouraged to take a more active role in global health care such as pursuing opportunities to participate in international experiences.[

Clinical pharmacy programs were established in China in 1989. There were 30, 44, and 5 pharmacy schools providing undergraduate programs, master's degree programs, and PhD program in clinical pharmacy, respectively, and roughly 610 students were enrolled in 2012 in total.[ Peking University started a master's degree in clinical pharmacy in 2005 and accepted master students major in clinical pharmacy from a 6-year pharmacy program (bachelor-to-master), 3-year master of pharmacy degree program, 3-year master of pharmaceutical science program, and 5-year master-to-doctor program and enrolled 13 students from the 3 programs in 2012.[ As a rotation site of Peking University, Peking University Third Hospital (PUTH) took the lead in exploring international pharmacy practice in China and provided serial clinical practice courses such as introductory pharmacy practice experience and advanced pharmacy practice experience (APPE).[ Since 2008, PUTH started international pharmacy education programs (IPEP), a type of cultural exchange curriculum.

The objectives of IPEP were the following: understanding of doctor of Pharmacy (Pharm.D.) education mode, improving English competency and clinical pharmacy competency, as well as promoting international collaboration. Currently, for Chinese students at PUTH, the IPEP include 2 curricula. The first one is to work as assistants for APPEs, in which Chinese students take part in APPEs with Pharm. D. students from the United States (US) and participate in multidisciplinary rounds, medication prescription reviews, patient education, and case reports. The second one is the “Global Classroom (GC)” program, in which students take courses taught by preceptors in both China and the US and participate in group studies with Pharm.D. students in the US via internet conference.[ For the GC program, students and preceptors from different locations in China and the US come together to discuss topic of interest. Topics included Health Care System and Pharmacy & Pharmacists’ role, Drug-induced diseases, Complementary and Alternative Medicine, and so on. Students who entered IPEP completed a 4 to 5-year undergraduate programs of pharmacy, and all of them were pursuing their master degree or master-to-doctor degree of pharmacy in China.

The development of IPEPs requires significant money, time, and effort to establish course objectives, content, and the organization implementation. Waterval et al conducted a qualitative study on 6 cross-border curriculum partnerships and found 4-fold challenges from differences in health care systems, legislation/political interference, teaching/learning environments, and partnership.[ There are currently few studies on the effect of IPEP. Cisneros et al conducted telephone interviews on IPEP at 20 colleges and schools of pharmacy in the US but did not evaluate the effect of teaching.[ Gourley et al described a twenty-year international exchange program and found 72% of respondents agreed or strongly agreed that this program made them more sensitive to the diverse needs of patients and pharmacy practitioners.[ There are no studies currently evaluating IPEP between China and the US, especially a systematic assessment on knowledge and skills. Thus, it remains unknown whether benefits outweigh difficulties for the current modes of IPEP at China.

The Kirkpatrick 4-level model of evaluation consists of reaction level, learning level, behavior level, and result level. This model was specifically designed for evaluating training programs and has been adapted for use in higher education and healthcare training programs.[ The Kirkpatrick model can provide a way to contextualize both short-term and long-term organizational outcomes.[ Dudas et al conducted an evaluation of a simulation-based pediatric clinical skills curriculum for medical students with the first 3 levels in the Kirkpatrick model.[ Lavender et al reported results on the first 2 stages of Kirkpatrick model in a pilot quasi-experimental study to determine the feasibility of implementing an e-learning tool for student midwife training in Nairobi.[ Dennis et al evaluated pharmacy student learning extent during APPE according to different levels by Kirkpatrick hierarchy level.[ Therefore, the Kirkpatrick model may be a useful evaluation tool for IPEP.

The objective of the study is to explore the applicability of the Kirkpatrick model in the IPEP evaluation and assess the quality of current IPEP at PUTH.

Method

Study design and participants

A comprehensive assessment was made at 2 levels, the reaction level and learning level of the Kirkpatrick model.

For the evaluation of reaction level, the questions in the questionnaire were designed according to “The University of Michigan College of Pharmacy Experiential Education Preceptor Manual” and “The University of Kentucky College of Pharmacy Advanced Pharmacy, Practice Experience, Preceptor Manual & Global Syllabus.” Pre-surveys were conducted before the pilot to assess the validity and reliability of the survey instruments. A total of 3 questionnaires were sent to 3 students who attended the IPEP from each year with 3 completed. Final survey questions were assessed and modified based on the analysis. Multiple-choice questions in the questionnaire were adjusted to single-choice questions, and important information of the questions were highlighted after pre-surveys. There were 19 single-choice questions in the survey which took 5 to 10 minutes to complete, including 11 items (10 closed questions and 1 open question for other benefits not listed in the above 10 items) for benefits of IPEP, 6 questions (5 closed questions and 1 open question for other difficulties not listed in the above 5 items) for difficulties of IPEP, and 2 items for overall evaluation. Questionnaire survey in Chinese was sent to Chinese students who attended IPEP during 2014 and 2016 in PUTH. The questionnaire was translated into English in Table 2 by the first author and another author who got her Pharm.D. degree in the US did a back translation to Chinese. The ordered variable items in the questionnaire were scored 1 to 5 according to the Likert scale. Higher score indicated higher degree of benefits or difficulties for students.

Table 2

Benefits, difficulties, and overall evaluation of IPEP in the reaction level (n = 36).

All attended the IPEP during 2014 and 2016 in PUTH (36 students) were invited to answer the anonymous questionnaire through a website linkage. The Cronbach α coefficient of the questionnaire was 0.938, indicating good internal consistency and good reliability as the evaluation index system in the survey. Findings from the pre-survey was consistent with the final results. Principal component analysis found 4 categories of the 10 items related to IPEP benefits in the questionnaire as following: “English competency improvement” category, “Understanding of Pharm.D. education mode” category, “Clinical practice competency improvement” category, and “International collaboration” category.

For the evaluation of the learning level, we collected data based on “Assessment Record of Advanced Clinical Pharmacy Practice of Peking University” from all students who attended IPEP during 2014 and 2016 in PUTH. Preceptors only filled in the assessment records for master of pharmacy students due to the teaching requirements of Peking University, so available data were restricted to master of pharmacy only. Data of all matched comparator students who entered into Peking University in the same year but did not participated in the IPEP were also collected to explore the differences among students.

Personal in-depth interviews were conducted to meet the need for further investigation of the reaction and learning levels. An outline of the interview was developed according to findings of the 2 assessment levels and revised after discussions among researchers and a statistical expert. Students participated in IPEP were randomly selected using a random number table from each year for individual interview in a secret meeting room for about 30 to 60 minutes until information saturation was reached. A 3-interviewer group conducted the interviewing: 1 student who participated in the courses, 1 teacher who was the teaching assistant, familiar with the courses but not the teacher of the courses, and 1 statistical expert who did not participated in the courses together. To minimize bias in the in-depth interviews, efforts were made by limiting the number of interviewers as well as by using a standardized script. All of interviews was digitally recorded and transcribed verbatim. Two interviewer transcribed interview transcripts separately and then compared the results. A pharmacy student who did not participated in the IPEP and the interviews analyzed the interview transcripts under the instruction of the statistical expert. Incongruent information was discussed among interviewers and the analyzer.

Data analysis

Cronbach α was adopted for the measurement of the reliability of the questionnaire. If the Cronbach α coefficient was above 0.8, we considered questionnaires as good internal consistency. Principal component analysis was done to assess categories of the questionnaires. The fundamental idea of principal component analysis was to examine the matrix of item correlations to reduce the information into a smaller set of components. In the presence of high inter-correlation, items were assumed to be measuring the same latent component.[

All collected data were coded and entered into a database using the SPSS (version 20.0, IBM Corporation, Armonk, NY, 2011). A Student t test for question items were computed. The level of statistical significance was set at P < .05 (2-tailed analysis). Continuous data were indicated by the mean ± standard deviation, and the categorical data were expressed by frequency and percentage. Qualitative data were transcribed and thematically analyzed.

This study was approved by the Peking University Third Hospital Medical Science Research Ethics Committee. The consent of interview participants was written.

Results

A total of 36 graduate students attended the IPEP during 2014 and 2016 in PUTH (Table 1), and all were invited to participate. All of the students consented to the study (100% response), and 36 questionnaires were completed (100% completion).

Table 1

Basic information for students attended the IPEP (n = 36).

Evaluation results of reaction level

All scores of 4 categories were above 3 points. The results of the survey showed that “International collaboration” category (mean ranking [MR] = 3.92 points) scored the highest amongst all categories, followed by “English competency improvement” category (MR = 3.64 points) and “Understanding of Pharm.D. education mode” category (MR = 3.48 points), “Clinical practice competency improvement” category (MR = 3.11 points) had the lowest score (Table 2).

Among the “International collaboration” category, 61.1% of students (41.7% strongly agreed and 19.4% agreed) confirmed that IPEP provided a good chance for international collaboration establishment. Among the “English competency improvement” category, students confirmed that “academic English improvement” gained a higher score (MR = 3.71 points, 22.2% strongly agreed and 41.7% agreed) than “oral English improvement” (MR = 3.56 points, 16.7% strongly agreed and 36.1% agreed). Among the “Understanding of Pharm.D. education mode” category, “understanding Pharm.D courses” gained the highest score (MR = 3.64 points, 22.2% strongly agreed and 36.1% agreed). Among the “Clinical practice competency improvement” category, “knowledge of drug utilization habits abroad” gained a relative higher score (MR = 3.31 points, 11.1% strongly agreed and 38.9% agreed).

In open questions beyond the 4 categories, students listed other benefits including knowing the relative high tuition fees of foreign Pharm.D. education program, pharmacists’ average income at other countries, broadening culture horizons, expanding the global view, and improving recognition of patient-centered care.

All of the difficulties were considered minor when compared to benefits, among which international network speed was rated as the most important one (MR = 3.06 points, 11.1% strongly agreed and 25.0% agreed), suggesting that hardware equipment could be upgraded. Students reported other difficulties in the open questions. For example, 1-hour course was too stressful due to a time difference between countries (eg, the “Global Classroom” program started 8 am in the US while it was 8 pm in China), and course content was not relatable to clinical practice in China (Table 2).

The 2 overall evaluation scores were high, with “worth attending IPEP” category scoring 4.25 points (52.8% strongly agreed and 25.0% agreed) and “recommend other students for IPEP” category scoring 4.19 points (50.0% strongly agreed and 25.0% agreed). This appeared as the highest among the items, indicating that students generally recognized the value of the IPEP and felt satisfied with the programs.

Evaluation results of learning level

Learning level evaluation was based on using students’ assessment records, which were submitted to the Peking University when students graduated and students received the final scores at that time. Among the 18 master of pharmacy students who participated in IPEP, only 5 student records were obtained due to 6 students records were lost and 7 students did not graduate. In order to compare the education effect, assessment records of 17 matched students who entered into the Peking University in the same year but did not attended IPEP were also collected. Therefore, baseline characteristics of the 2 groups were considered comparable.

Higher scores were found for most of the 8 items for students participated in IPEP, but there was no statistically significant difference in students’ scores between participation and nonparticipation of IPEP in the learning level assessment (all P value > .05) (Table 3).

Table 3

Assessment results of the learning level (n = 22).

Personal in-depth interviews

When the number of interviewed students reached 6, information saturation was achieved. Interview outline covered 4 items, and results confirmed a beneficial and unique opportunity of IPEP despite of a nonobvious impact on learning level, and some suggestions on equipment and curricula systems were provided (Table 4).

Table 4

Personal In-depth interview outlines and summarized results (n = 6).

Discussion

The study confirmed that benefits outweigh difficulties for IPEP for pharmacy students in China. The scores of overall IPEP evaluation were high, suggesting satisfaction with the international programs. During the IPEP, Chinese students learned together with foreign students and discussed pharmacy knowledge daily with each other in English. Thus, students were able to improve their English proficiency. As teaching assistants of the IPEP, students in China were able to learn more about the Pharm.D. curriculum. While providing pharmacy services for patients together with Pharm.D. students, students were able to learn the thought process and working habits from each other. In this program, students dedicated more time but also gained better understanding about pharmacy education in the US.

However, scores in the clinical practice competency improvement category demonstrated less benefit, and no statistically significant difference in the learning level was found. As over half of the students only took the GC program, most communications with Pharm.D. students were conducted with WeChat software or other tools in written English, which could have led to low participation from some students.

Studies have been published to determine the extent and characteristics of global pharmacy education programs. Academics at University College London School of Pharmacy have shown that international experience stimulates pharmacy students’ personal development and professional development.[ Woods et al illustrated that student perceptions of learning through an international comparison made it more effective in raising international perspectives awareness in the pharmacy curriculum.[ Waterval et al conducted a survey on cross-border curriculum of medical students and found that host students felt the partnership afforded opportunities to acquire unique academic competencies and boost their career.[ Our findings are consistent with existing studies. Meanwhile, due to the difficulties of expenses, foreign language, and living habits while travelling abroad, introduction of international programs at native countries can benefit students of both native and foreign countries.

Our study employed a comprehensive evaluation system based on the Kirkpatrick’ model and applied it to the evaluation of the effect of IPEP, which provided quantitative results of teaching efficiency and quality of IPEP. Additionally, our study is the first study to explore the effect of 2 IPEP and provided quantitative evidence to help support decision-making for future international pharmacy programs. Moreover, this study also provided qualitative research in additional to quantitative research, which further strengthen results of our study.[ Due to inherent limitation in the results of quantitative research, results from personal in-depth interviews can be combined with quantitative results to thoroughly reflect the teaching effect of IPEP.

Our study has several limitations. First, the study did not evaluate all 4 levels in the Kirkpatrick model; the lack of behavior and result levels made evaluation results less complete. Given the duration of this study and the fact that this was preliminary work, this evaluation will not reach behavior level and result level, and these levels could be assessed in future study. Second, the “Assessment Record of Advanced Clinical Pharmacy Practice of Peking University” was not designed specifically for this study.[ Thus, accurate depiction of the impact of IPEP was difficult. Additionally, differences in students’ baseline competency level were not accounted due to the lack of initial assessment of students’ baseline performance. Therefore, it is necessary to design a specific scale for the evaluation of IPEP effect, collect baseline scores for a comprehensive investigation and analysis from more categories. Third, the sample size of the study is relatively small and no subgroup analysis according to grades was done, which could be vulnerable to extreme values even though the study is a full sample research which accounted all samples that can be found in a limited number of IPEP in China. Larger sample size is needed in the future in order to get a more accurate and realistic conclusion of effect of IPEP in China.

The findings from this study provided a good reference for schools currently developing IPEP, exploring IPEP curricula, and an evaluation model. While receiving foreign students from abroad, teaching assistants training can also bring benefit to students along with more chances for international communication via online GC course. AlFaar et al supported telecommunication as a relatively inexpensive approach to improve pharmacy practices, especially in developing countries.[ Additionally, a bibliometric analysis of literature in pharmacy education also supported the utilization of social networks with advanced telecommunication technologies as an emerging new direction in pharmacy education.[ The confirmed benefits by the GC program via internet conference in our study may encourage similar courses around the world in the future.

Conclusions

This study demonstrates that the Kirkpatrick model can be used in the IPEP evaluation. Students showed a high overall satisfaction and benefit in the reaction level for the IPEP at PUTH, while the higher scores in the learning level did not reach statistical significance. Additional efforts to improve students’ clinical practice competency are needed in the future.

Acknowledgments

The authors wish to thank Zhi-Yan Liu, Ya-Jin An, Qiao-Nan Lin, Xiao-Yan Nie from the Department of Pharmacy Administration and Clinical Pharmacy, Peking University School of Pharmaceutical Science for help with information and data collection. The authors also thank Hua Zhang from the Research Center of Clinical Epidemiology, Peking University Third Hospital for consultation for data analysis.

Author contributions

All the authors contributed extensively to the work presented in this paper. ZMY led the study design. ZMY, LYZ, and LY collected data and conducted data analysis. LY, RSZ, and SDZ reviewed data analysis. All authors jointly contributed to result in interpretations and manuscript writing. All authors read and approved the final manuscript.