A multimodal international collaborative clinical research training program in China.

Background: There is a strong need to include training of research methods in training programs for physicians. International clinical research training programs (CRTP) that comprehensively introduce the methodology of clinical research and combined with practice should be a priority. However, few studies have reported a multimodal international CRTP that provides clinicians with an introduction to the quantitative and methodological principles of clinical research. Objective: This manuscript is intended to comprehensively describe the development process and the structure of this multimodal training program.
Methods: The CRTP was comprised of three distinct, sequential learning components: part 1 - a six-week online eLearning self-study; part 2 - a series of three weekly interactive synchronous webinars conducted between Durham, North Carolina, USA and Beijing, China; and part 3 - a five-day in-person workshop held at Beijing Friendship Hospital, Capital Medical University (BFH-CMU). Self-assessment quiz scores and participation rates were used to evaluate effectiveness of the training program. Participants' demographic characteristics, research experience, satisfaction and feedback on the program were collected using questionnaires.
Results: A total of 50 participants joined the CRTP. Forty-four participants (88%) completed the program satisfaction questionnaires. The average quiz score of the six eLearning units varied from 31% to 73%. Among the three components of the program, the online eLearning self-study was felt to be the most challenging. Thirty-nine (89%) of the surveyed respondents were satisfied with all components of the training program. Among the respondents, 43 (98%) felt the training was helpful in preparing them for future clinical research projects and expressed willingness to recommend the program to other colleagues. Conclusions: We established a multimodal international collaborative training program. The program demonstrated acceptable participation rates and high satisfaction among Chinese clinicians. It provides a model that may be used by others developing similar international clinical research training programs for physicians.

Introduction

With the increasing societal concern for population health, the education and training of clinical and translational researchers has been an important priority [1]. Reliable medical results are generated from high-quality studies with the understanding of epidemiology and biostatistics. Recently, there has been a growing desire among young clinicians for clinical research training, especially training in research methods [2,3]. Domestically within China, continuing education programs on clinical research are provided by the Chinese Medical Association and taught by Chinese professors [4]. Also, there is an International Research Camp for Clinical Research Design and Program Development (IRCCRDPD) program hosted by Peking University Clinical Research Institute [5]. These programs offer short-term training (usually less than one week) and primarily invite overseas professors to attend and give courses or lectures. Fogarty International Clinical Research Scholars and Fellows Program and Harvard Medical School’s Introduction to Clinical Research Training (ICRT)–Japan program are two well-known international training programs [1,6,7], however, they are not appropriate for Chinese clinicians or not available in China.

Beijing Friendship Hospital, Capital Medical University (BFH-CMU), is a tertiary hospital with a long history of graduate, collaborative training programs in partnership with many universities, hospitals, and institutes. BFH-CMU decided to conduct a high-level training program to improve the quality of clinical research skills among young and middle-aged Chinese physicians. Therefore, BFH-CMU and Duke University School of Medicine jointly organized a clinical research training certificate program (BFH-Duke CRTP), which provides rigorous training in the quantitative and methodological principles of clinical research.

This article aims to comprehensively describe the development process and the structure of the multimodal training program.

Materials and methods

The organizations of BFH-Duke CRTP

Beijing friendship hospital, capital medical university

BFH-CMU established in 1952, is a large-scale general hospital in Beijing. With strengths in multiple clinical areas, BFH-CMU is the home of the National Clinical Research Center for Digestive Diseases (NCRC-DD), jointly awarded in 2014 by the Ministry of Science and Technology, National Health and Family Planning Commission of People’s Republic of China, and medical authority of Logistics Support Department, People’s Liberation Army.

NCRC-DD is a state-level medical center. It has established a nationwide clinical research collaborative network involving multi-level hospitals in China. The Clinical Epidemiology and Evidence Based Medicine (EBM) Unit of BFH-CMU provides methodological support and clinical research training for the NCRC-DD collaborative network.

Duke university school of medicine

The Duke University School of Medicine, established in 1925, is among the top medical schools in the USA and has one of the largest biomedical research enterprises in the USA [8,9]. The Duke Clinical Research Training Program (CRTP), founded in 1986, is a 2-year academic degree program in the Duke University School of Medicine that provides academic training in the quantitative and methodological principles of clinical research. The program is primarily designed for clinical fellows who are training for academic careers and offers formal courses in research design, research management, medical genomics, and statistical analysis [10].

Program development

This specialized certificate-based program was designed and produced by Duke CRTP faculty in collaboration with an educational consulting firm experienced in the design, construction, testing, and delivery of eLearning. The primary components of the curriculum represent a core set of biostatistics and clinical research design principles and methods from the Duke CRTP degree program curriculum.

The certificate program is intended to provide clinicians and other health care professionals an introduction to the quantitative and methodological principles of clinical research. Key learning objectives of the program are to provide participants with the skills to (1) contribute to the design, implementation, analysis, and interpretation of clinical research studies; (2) critically evaluate research design and biostatistics aspects of research literatures published in the clinical research journals; (3) participate as collaborators in interdisciplinary clinical research teams.

Three experienced Duke CRTP faculty members led the BFH-Duke CRTP. All are collaborative clinical researchers and experienced teachers. Two have expertise in biostatistics and bioinformatics and one is a general internal medicine clinical faculty member. The key members of the Duke CRTP program and the methodology team of Clinical Epidemiology and EBM Unit, BFH-CMU were involved in the development of this collaborative program.

Participants

The BFH-Duke CRTP program is targeted to Chinese clinicians interested in improving their clinical research skills. Program recruitment information was advertised online. Program acceptance requirements were: 1) clinician with certain experience of clinical research; 2) less than 45 years old; 3) having a good skill in English reading and a certain communication skill in English; 4) available for five-day in-person workshop in Beijing.

Fifty clinicians were enrolled in the BFH-Duke CRTP on a first come enrollment basis. These participants were from nine cities or provinces of China, including Beijing, Hebei, Shanghai, Zhejiang, Fujian, Chongqing, Inner Mongolia, Jilin and Liaoning provinces (Figure 1). The clinical specialties of participants included digestive diseases, liver diseases, cardiology, neurology, general surgery, liver transplantation, vascular surgery, urinary surgery, ophthalmology, ultrasonography, obstetrics and gynecology and intensive care. All participants were clinicians with experience of participating in clinical studies, and without statistical background/computer expertise. Participants were divided into ten structured teams that worked together throughout the program to develop a study proposal.

Figure 1.

Geographic distribution of clinicians who attended the training program.

A total of 50 training program participants were from nine cities or provinces of China, including Beijing, Hebei, Shanghai, Zhejiang, Fujian, Chongqing, Inner Mongolia, Jilin and Liaoning provinces (Blue areas in the map). The in-person workshop was in Beijing (Red area in the map).

The structure of the training program

The BFH-Duke CRTP was comprised of three distinct, sequential learning components: Part 1 – a six-week online eLearning self-study, Part 2 – three weekly interactive synchronous webinars conducted between Durham, North Carolina and Beijing, China, and Part 3 – a five-day in-person workshop held at BFH-CMU in Beijing (Figure 2).

Figure 2.

Training model for CRTP Abbreviation: CRTP, clinical research training program.

Part1: online eLearning self-study

The objective of the online eLearning self-study was to become familiar with the basic principles of clinical research design and biostatistics. The eLearning modules were delivered via the TalentLMS () cloud-based learning management system (LMS). The eLearning curriculum was divided up into six on-demand weekly units, each requiring two to three hours to complete. Each unit addressed both research design and biostatistics topics that were delivered via multiple faculty narrated eLearning modules that included discussion of case studies from published research, primarily selected from studies conducted in China. The detailed topics included in the online eLearning structure are provided in supplementary Table 1.

At the end of each unit participants’ performance were measured by self-assessment multiple-choice quizzes as well as a written assignment that was submitted and reviewed by Duke faculty. Multiple choice questions were largely based on case-based excerpts from published open-access clinical research articles, the majority coming from articles publishing results of clinical research studies conducted in China. Assignments had a similar focus but required participants to read excerpts from articles (or articles in their entirety) and provide short written responses in English to questions about biostatistics or research design aspects of the studies in question.

Participants were required to view all eLearning modules and complete all quizzes and assignments before continuing to the next weekly unit. Participants had access to an online discussion forum as well as email support from program faculty. All eLearning modules were presented and narrated in English with English text and Chinese written translation provided side by side for better understanding of biostatistics and research design concepts.

Part 2: interactive synchronous webinars

Three weekly synchronous webinars were initiated after completion of the six online eLearning units in teleconference centers of Duke and BFH-CMU using CISCO hardware and the Webex platform (). The purpose of the webinars was twofold: to utilize case studies to apply principles learned during the eLearning self-study and to introduce the team-based project to develop a study proposal in response to one of two requests for proposals (RFPs). Lecture-demonstration by teacher and student-team reports were used as teaching methods in this part of the program. Each of the three webinars was offered three times during the week to accommodate the schedules of the participants and divide participants into smaller groups to facilitate more effective discussion. The webinar portion of the program had several team-based activities (e.g., pick a team name, choose which of several project topics to pursue) to build team rapport that participants completed but there were no quizzes, assignments, or other assessments. The webinars also discussed team-based research and fundamental principles in team interaction and dynamics. These activities were intended to give each 5-person team the opportunity to become acquainted and begin collaborating so that they would be ready for more team-based project activities during the workshop. Each webinar was approximately 75 minutes in length and held either in the early morning or late evening due to the 12-hour time difference between Durham and Beijing. Attendance was monitored and each webinar was recorded.

Part 3: five-day in-person workshop

The final part of the program was conducted in-person at BFH-CMU where the Duke faculty in partnership with five BFH-CMU teaching assistants (TAs) with expertise in Epidemiology and Biostatistics led a five-day workshop where participants worked on a team-based project to develop a clinical research study proposal. Task oriented discussion and presentations by student teams were used as teaching methods in this part of the program. The capstone activity of the workshop and program was development of a clinical research study proposal that included both written and oral presentation components. At the end of the workshop, each team submitted a written study proposal and delivered an oral presentation summarizing their proposal. The research proposal included the following components: research question, study outcomes, study design, data analysis plan, and study limitations. Program faculty supervised and participated in the discussion of team proposals and provided solutions to problems each team faced. Throughout the workshop, the TAs worked collaboratively with participant teams, providing feedback and guidance as they developed their study proposals.

Participants’ performance and participation evaluation

A survey was delivered to the participants at the end of the program using a survey delivered in an online questionnaire tool (Questionnaire Star) via WeChat messaging app (). Detailed information about development of the survey instrument can be found in the website (). The structure of the delivered questionnaire included demographic characteristics (e.g., gender, professional title, English fluency, and years of work experience), research productivity (e.g., years of clinical research experience, number of Chinese and English Science Citation Index (SCI) publications), program characteristics (complexity of materials, program satisfaction, and financial value of training) and a free text feedback question on the overall evaluation of the training program.

In the section on satisfaction, participants were asked to report agreement with each statement on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). The survey was developed and distributed in Chinese to facilitate understanding of the survey items. After data collection, the survey results were presented in English by the translation of the authors (supplementary Table 2). This program was approved by the Institutional Review Board of Beijing Friendship Hospital (No. 2018-P2-216-01).

The participants were divided into active and passive groups to account for differing levels of participation in the webinars and workshop. Active and passive participants were identified by the number of faculty interactions completed as a surrogate for participation in the classroom throughout the interactive synchronous webinars and workshop by TAs [11]. Only participants who interacted with faculty equal to and more than once in any of the synchronous webinars and equal to and more than four times in the workshop were identified as ‘Active’. Other participants were identified as ‘Passive’ if they had no interaction in synchronous webinars or less than four interactions in workshop.

Statistical analysis

The quiz scores were calculated by the proportion of correct answers and the total number of quizzes. We used median and 1st quartile (Q1) – 3rd quartile (Q3) to describe the central tendency and dispersion for quantitative variables, such as the age, working years, clinical research years, SCI published articles, Chinese published articles and study time per week. The frequency (N) with the percentage (%) was used to describe categorical variables, such as gender, professional title, impact factor, oversea experiences, English communication ability, difficulty and satisfaction of the program, willingness to recommend and pay. Univariate logistic regression models were used to analyze the associations between degree of participation (active/passive) and potential influential factors, including gender, professional title, working years, total SCI index and other indicators of the capacity for scientific research. Modeling results are summarized using odds ratios (OR), confidence intervals (CIs), and p-values. Computation of confidence intervals was based on the profile likelihood.

Results

The response of all the participants

Forty-four participants (88%) answered the survey. The median ages were 34.0 years old and 32.5 years old for responders and non-responders. The male proportions were 17% and 45% in responders and non-responders. There were no statistically significant differences in age and gender between responders and non-responders.

Demographic characteristics of surveyed respondents

There were 20 (45%) male and 24 (55%) female clinicians, aged 34.0 (30.0-40.8) years old (Table 1). Seventeen (39%) participants of these young and middle-aged clinicians were associate or chief physicians (Table 2). As shown in Table 1 and Table 2, the medians and IQR of their working years and clinical research years were 8 (Q1-Q3: 3–15) and 3 (Q1-Q3: 1–7), respectively. The medians of publications were 4 (Q1-Q3: 1–5) of SCI in English and 4 (Q1-Q3: 2–10) in Chinese. Thirty (68%) of the surveyed respondents had overseas experience, and 35 (80%) of the fellows could fluent or excellent state their opinion clearly in English (Table 2).

Table 1.

Quantitative data of demographic characteristics of surveyed respondents (N = 44).

Variables	Median (Q1-Q3)
Age	34.0 (30.0–40.8)
Working years	8 (3–15)
Clinical research years	3 (1–7)
SCI published articles	4 (1–5)
Chinese published articles	4 (2–10)
Study time per week	7 (5–10)

Abbreviations: Q1, first quartile. Q3, third quartile. SCI, Science Citation Index. SCI published articles: the articles cited in Science Citation Index. Chinese published articles: the articles published in Chinese.

Table 2.

Categorical data of demographic characteristics of surveyed respondents (N = 44).

Variables	N (%)
Gender
Male	20 (45%)
Female	24 (55%)
Clinical titles for doctors
Resident	16 (36%)
Attending	11 (25%)
Associate chief physician	12 (27%)
Chief physician	5 (11%)
Impact factor
0–3	13 (30%)
3.1–5.0	4 (9%)
5.1–10.0	12 (27%)
10.1–20.0	10 (23%)
>20.0	5 (11%)
Overseas experiences
0	14 (32%)
1–6 months	15 (34%)
7–12 months	5 (11%)
>12 months	10 (23%)
English communication
Capable	9 (20%)
Fluent	27 (61%)
Excellent	8 (18%)

Completion and participation rates

The average quiz scores in eLearning varied from 31% to 73% among the six units (Figure 3(a)). The completion rates for the first three units were 100%. However, the completion rates for unit 4 to unit 6 showed a decrement from 86% to 66%. The participation rates for the three webinars ranged from 64% to 75% (Figure 3(b)). All respondents attended the five-day face-to-face workshop. The factors that affected the active participation rates are shown in supplementary Table 3. Compared with participants who could only state their opinions with the help of translators, those with the ability to speak clear and fluent English had significantly higher participant activity (OR = 13.33, 95% CI: 1.41–321.37, p = 0.044).

Figure 3.

Performance of respondents in the online eLearning and webinar. a, quiz scores and completion rates in each of the six units in online eLearning. b, the participation rates and the active participation rates in each of the three webinars.

Difficulty and satisfaction reported form surveyed respondents

As shown in Tables 3, 5 surveyed respondents (11%) found the overall training courses as easy, 28 (64%) as neutral and 11 (25%) as difficult. Regarding the degree of difficulty of the three training components, there were 3, 14, and 11 respondents graded easy for online eLearning (7%), webinar (32%) and in-person workshop (25%), respectively. Around 50% of respondents graded neutral for webinar and in-person workshop, which was higher than 39% for online eLearning. Up to 55% of them deemed online eLearning to be difficult, which was higher than that of the synchronous webinars (18%) or in-person workshop (18%).

Table 3.

Surveyed respondents’ ratings of overall difficulty of training program (N = 44).

Difficulty for learning	EasyN (%)	NeutralN (%)	DifficultN (%)
Overall	5 (11%)	28 (64%)	11 (25%)
Online eLearning self-study	3 (7%)	17 (39%)	24 (55%)
Live webinar	14 (32%)	22 (50%)	8 (18%)
In-person workshop	11 (25%)	25 (57%)	8 (18%)

Thirty-nine (89%) surveyed respondents showed satisfaction with the program, with the aspects in practical orientation, facilitators, materials, group work, and workshop. However, 5 surveyed respondents (11%) had a neutral attitude toward the content and frontier coverage in this program (Table 4).

Table 4.

Surveyed respondents training program satisfaction results (N = 44).

Satisfaction	Strongly disagreeN (%)	DisagreeN (%)	NeutralN (%)	AgreeN (%)	Strongly agreeN (%)
Practical orientation	0	0	2	11	31
			(5%)	(25%)	(70%)
Substantial content and frontier coverage	0	0	5	13	26
			(11%)	(30%)	(59%)
The faculty was knowledgeable	0	0	0	11	33
				(25%)	(75%)
The faculty clearly explained the course material	0	0	2	11	31
			(5%)	(25%)	(70%)
Facilitators answered participants questions well	0	0	1	11	32
			(2%)	(25%)	(73%)
The visual materials (slides, participant guide, etc.) were effective in supporting the workshop	0	0	1	10	33
			(2%)	(23%)	(75%)
The group research and presentation prepared me for future work in clinical research	0	0	1	16	27
			(2%)	(36%)	(61%)
The workshop was relevant to my role	0	0	4	16	24
			(9%)	(36%)	(55%)

Forty-three surveyed respondents (98%) indicated that they would recommend the program to their colleagues. While only the small percentage of individuals (n = 5, 11%) indicated willingness to pay >15,000 CNY, all remaining respondents (n = 39, 89%) valued the program at a minimum of 10,000–15,000 CNY (Table 5).

Table 5.

Willingness to recommend and pay (N = 44).

Willingness to Recommend and Pay	N (%)
Willingness to recommend to a colleague
Not recommended	1 (2%)
Possibly recommended	7 (16%)
Recommended	36 (82%)
Willingness to Pay
10,000-15,000 CNY (1,463-2,195 USD)	39 (89%)
15,001-20,000 CNY (2,196-2,926 USD)	2 (5%)
20,001-25,000 CNY (2,927-3,658 USD)	2 (5%)
25,001-30,000 CNY (3,659-4,390 USD)	0
>30,000 CNY (4,390 USD)	1 (2%)

Abbreviations: CNY, Chinese yuan ($0.15 in US Dollars); USD, US Dollars.

Qualitative feedback from participants

The participants spoke highly of this training program. One participant evaluated this program as ‘Internet + interactive training’ and stated that offline face-to-face communication and operations helped provide him with a better understanding of study design. The fully prepared handouts and materials before the discussion, the importance of medical writing and effective presentation skills, and the well-prepared and engaged faculty made teaching and learning more substantive, beneficial and interesting.

There were 20 surveyed respondents (45%) providing 22 suggestions for further improvement of the course. The feedback from participants included three main aspects of suggestions to further improve the training courses (Table 6). Firstly, 7 (32%) commented on the degree of difficulty of the eLearning, suggesting that the content of this part could be further compressed for an introductory course and more practical examples could be added to better facilitate participant understanding. Secondly, 5 (23%) suggested on providing bilingual slides (narration scripts were provided in both English and Mandarin). Lastly, 5 (23%) suggested that the content of teaching should be more extensive and compatible with clinical practice. For example, they suggested expanding content related to meta-analysis and adding content related to procedures for registering clinical trials on .

Table 6.

Surveyed respondents feedback (N = 22).

Suggestions for further improvement	N (%)
More examples need providing in eLearning to help understanding biostatistics curriculum	7 (32%)
Bilingual subtitle need providing in PowerPoint slides in eLearning	5 (23%)
Teaching contents should be more compatible with clinical practice	5 (23%)
Arrange more time for teamwork/activities	3 (14%)
Others	2 (9%)

Discussion

The certificate-based international collaborative training program was innovative in its use of a blended learning approach that included an online eLearning self-study, synchronous webinars, and a five-day in-person workshop. To our knowledge, this is the first manuscript to comprehensively describe the development process and the structure of the multimodal training program.

The infrastructure of online eLearning and synchronous webinar modules of the program described in this study were similar to the previous ICRT-Japan program [1,12], but with a more detailed introduction on the curriculum design, delivery method, duration and teaching method. Furthermore, we adopted a different approach to mentorship in face-to-face workshops.

The three distinct, sequential learning components delivered by BFH-Duke CRTP worked well in China. The results in evaluating the difficulties among the multimodal program demonstrated that most of the participants felt that the webinars and workshop were less difficult than the eLearning. As suggested by the open-ended feedback, this may be related to the focus of the eLearning on the basic foundations of epidemiology and biostatistics, which are relatively unfamiliar topics to the clinicians. Additionally, unlike the eLearning which was completed individually, participants worked in teams during the webinars and workshop to complete a research proposal which is more familiar to them since all have at least some previous clinical research experience.

Online and Offline (OAO) teaching mode was performed in this international collaborative training program. Over the past decades, the interest in medical education and clinical research training has increased dramatically [13,14], particularly in demand for international training of clinical research [15,16]. Novel project-based learning and research models arising from online learning have emerged with the development of the Internet, such as massive open online courses (MOOCs), creating opportunities for global education and research to address real-world challenges [17-19]. Though online eLearning has its own advantages, we found that offline interaction could be another feasible mode in curriculum development.

Mentorship was an important part in the training program. As Rezhake et al [7] and Shah SK et al [20] previously referred, effective mentoring offers opportunities to guide careers of medical researchers. We also discussed and addressed this critical issue during program development, the three fixed Duke faculty helped refine research questions and impart knowledge of clinical research. Five fixed BFH-CMU TAs with expertise in epidemiology and biostatistics worked collaboratively with participant teams, providing feedback and guidance as they developed their study proposals. The mentorship from the faculty and TAs of this training program could suggest appropriate post-program research directions for participants.

In international medical education, a Four-Component Instructional Design model (4C/ID), which emphasize part-task practice has been widely used [21,22]. Skills and knowledge, although taught, faced barriers including the difficulty in transferring to clinical practice [23]. BFH-Duke CRTP included some illustrative examples of educational practices. However, participants’ feedback suggested more practical examples in study design and statistical analysis should be given priority.

There are several potential limitations to consider when interpreting our findings. First, as all participants had some experience with and interest in clinical research, selection bias may affect the generalizability of this program. Second, as this was a cross-sectional survey, we could not obtain objective performance indicators, such as oral presentations at meeting (national/international), papers (cumulative impact factor and total number of first author papers), implementation of research projects and grants in this program, so a follow-up longitudinal survey will be conducted to collect objective performance indicators [24]. Third, it might be an important consideration of language barriers in international collaborative training programs. Though all the participants in this program were able to read and write in English, the ability of English communication might still be a critical barrier for discussion in the interactive modules of this program. The inclusion of both faculty and TAs fluent in Mandarin was a deliberate program decision to address this concern. Fourth, the limited number of participants might influence substantial assessment of this program. Further studies are justified to generalize this mode to other international training programs.

Conclusions

BFH-Duke CRTP established a multimodal international collaborative training program. This program could serve as a model for other similar international training programs for physicians and health professionals seeking to improve their clinical research skills.