The Rare Disease Research Scholars Program: A training curriculum for clinical researchers with mixed methods evaluation study.

Rare disease clinician investigators are essential to ensure appropriate diagnosis, care, and treatment for the rapidly growing rare disease population. As these researchers are spread across many specialties, learning the unique skill set for rare disease research (RDR) can be a hurdle and may hinder progress in the field. The need for an RDR focused training program for investigators in many specialties and backgrounds was identified in a needs assessment of trainees in the NIH funded Rare Diseases Clinical Research Network. Based on this information, the Rare Disease Research Scholars Program (RDRSP) was developed. We describe the needs assessment, curriculum creation, scholar recruitment, and outcome evaluation based on four years of programmatic data (2015-2019). This one year-long RDRSP uses a blended approach that includes in-person, web-based, synchronous and asynchronous learning. We evaluated the RDRSP using quantitative and qualitative approaches. Quantitative measures included pre and post questionnaires about knowledge, self-efficacy, and intent to remain in RDR. Data were analyzed using descriptive statistics and a paired t-test. Qualitative semi-structured interviews explored the RDR scholars' perceptions of the RDRSP; thematic analysis examined the textual data. Quantitative pre- and post-measures were statistically significant in the following areas: 1) improved knowledge content in RDR, 2) enhanced self-efficacy in clinical research, and 3) intent to remain in the field of RDR. Qualitative data analysis found the program supported the development of the scholar's research skills as well as 'soft-skills'. By combining training of skills unique to RDR with the more general topics of leadership, mentorship and collaboration among participants in diverse specialties, we created a program that supports the development of the next generation of rare disease clinician investigators and serves as a model for training in other niche research areas.

Rare disease research (RDR) is a niche field that demands specific skills and interactions that are not consistently fostered in the current academic research setting. Skills such as small population statistics, collaboration among institutions, and interacting with family and patient organizations were identified as unique skills. The field was propelled forward by the passing of the Orphan Drug Act of 1983, PL 97-414. This landmark legislation denoted a rare disorder as affecting less than 200,000 persons in the U.S., and it has served to expand access to novel medications for hundreds of rare disorders [1]. Using the terminology in this law, over 7,000 disorders are designated as “rare”; and RDR spans multiple medical specialties and funding mechanisms. With the expansion of social media and patient advocacy organizations such as the National Organization for Rare Disorders (NORD), rare disease patients and their families are now able to rally behind their own rare disorder and work towards faster diagnosis, improved treatment, and even cures in collaboration with health care providers and researchers. With the ongoing increase in patient identification, there has been enhanced interest in RDR from industry, the NIH and investigators. This is accompanied by the need to train clinician investigators in performing clinical research studies in rare diseases. In 2009, Griggs et al. [2] reviewed the opportunities, challenges, and proposed solutions to optimize production and retention of clinical researchers in rare disease. In this seminal work, the need for niche field training was described and led to the concept of rare disease training for individuals who already had specialty training. Since that time, the demand for rare disease researchers has only expanded with the burgeoning of novel diagnostics, biomarkers and treatments. This has led to the need to create new pathways of training in rare diseases [3].

The NIH has provided significant support for RDR including the development of the Rare Disease Clinical Research Network (RDCRN) to promote rare diseases clinical research, perform natural history studies, bring novel products to market and train scientist within the novel specialty field of RDR (reviewed in Krischer [4]). The RDCRN is a network of 23 consortia (as of January 2020) focused on 190 rare disorders [4]. Within each consortium and across consortia, best practices are shared to move forward rare disease patient care and clinical research. While institutions in each consortium have training programs in clinical research, we identified the need to create a cross-consortia, multidisciplinary RDR training program. Additionally, with the understanding that the need for training was even greater outside of the consortia hubs, the program was designed to meet the needs of both consortia-based and external early career rare disease researchers.

In 2015, the Rare Disease Scholars Program (RDRSP) was launched by the Children’s National Rare Disease Institute, Children’s National Hospital, Washington, D.C., funded by an R25 curriculum grant from NICHD and related to the institution’s Rare Diseases Clinical Research Center in Urea Cycle Disorders. This year-long educational program incorporated survey data from early career rare disease researchers, input from an invested oversight committee (the RDCRN Education Committee), and involvement by medical educators with the knowledge base to create a multi-faceted blended training curriculum.

The program development used the Lave and Wenger Communities of Practice theory of education, whereby direct interaction with a community of experts and mentorship is the basis of a learning experience [5]. This design employed two intensive day long in-person meetings and twice a month web-based half-hour “touch points” throughout the year. Each web-based session met one of the following criteria: 1) it provided a need identified in the assessment survey; 2) it involved an interaction with a rare disease expert; and 3) it gave information that was immediately implementable to the Scholar’s research. Additional optional content was created in a web-portal for lower need topics identified on the survey.

In this paper we describe the development of this educational curriculum and the results of the mixed methods of evaluation, incorporating immediate, long term, and qualitative feedback mechanisms.

Methods

Needs Assessment

The RDCRN Education Committee performed an assessment of their trainees to identify specific needs of the cohort prior to developing the curriculum and recruiting the Scholar Trainees from the host institutions. A total of 159 trainees who had participated in the RDCRN training programs over the preceding grant period were emailed an anonymous Survey Monkey link. There were no incentives for completion of the survey. A total of 106 surveys (67%) were completed.

Creation of the Curriculum

The curriculum included an in-person day-long workshop at the beginning of the program, a capstone meeting at the end of the program, bi-monthly webinars, and asynchronous learning opportunities through a web portal. All webinars were recorded and uploaded onto LearnRD a Learning Management System (LMS), along with other supporting documentation (i.e. PowerPoint slides, quizzes, handouts) for those who needed to access them at differing times (http://childrensmedicaleducation.org/learnrd/). Webinars included both general and RDR specific topics. Examples included: time management, undiagnosed disease programs, grant writing, rare disease funding mechanisms, Curriculum Vitae and Biosketch optimization, creation of educational portfolios, single subject study design, interactions with industry, rare disease advocacy, creation of biorepositories, engaging in a natural history study, and navigating regulatory affairs. Each year, several new topics were selected for presentation in webinars or online modules based on a needs assessment obtained mid-year. In addition, each year the full curriculum was re-assessed and re-designed based on the previous year’s cohort feedback and end of year surveys. In addition, specific sessions on working with patient and family groups were included based on studies showing the overall improvement in clinical research when these populations are involved in trial design and oversight [6].

Creation of an Online Portal

In 2015, LearnRD.com was created by the authors to serve as an integral component of the RDRSP. LearnRD.com is a LMS software that is used to deliver education courses, training programs, and development activities by administering quizzes/exams, course surveys, confidence level tests, and more while serving as a resource hub for teachings and learning materials. The LearnRD.com uses Moodle software, a course management software that runs on an open-source learning platform. The site is designed with username/password protection that allows for recordings of webinar session to be posted and stored. Alumni are welcome to return to review content or watch new content.

Scholar Recruitment

Scholars were recruited using multiple modalities. Initially, recruitment was through the RDCRN Education Committee and their junior faculty connections. As a result, in the first year, 91% of participants were affiliated with the RDCRN. Over time advertising at Rare Disease meetings, word of mouth, and encouraging non-traditional applicants (i.e. research genetic counselors, pharmacists) led to the participant levels from RDCRN-affiliation decreasing to about 70% in years three and four of the program. Aggressive external advertising for year five led to more than half of enrollees having no affiliation with the RDCRN. A total of 20 Scholars were accepted in year 1 with a maximum of 32 in year 3.

For individuals not chosen for a cohort, the most common reason was the level of training (too early or too experienced). Scholars were accepted during their final year of fellowship training and up to 5 years after beginning a faculty position. For PhD lab-based trainees, pharmacists, and genetic counselors, the terminal degree must have been completed and a letter of recommendation provided indicating a unique RDR interest and/or experience. Through the R25 funding up to 20 Scholars were offered partial reimbursement to off-set travel costs. All applications underwent a standard review process and top scored applicants were offered travel cost support. In years where additional applicants were identified and were able to support their own travel costs, additional Scholars where included in the program. Thus, the class size was based on funding and access to travel expense cost funding.

Outcome evaluation

By enrolling each scholar in the online learning portal (LearnRD.com) and in ORCID, data collection throughout the course and in follow-up was possible. Scholars were asked to complete the following: 1) pre/post subject content test, 2) an inventory of topics deemed important for RDR success, 3) the CRAI-12 clinical research self-efficacy evaluation, and 4) a questionnaire asking their likelihood of being involved in RDR in 10 years. All the scores were normalized to a maximum value of 10 points, and a paired T-test analysis was performed to determine significance.

The subject content test involves 34 multiple choice questions related to the field of rare disease research. Topics that were included and the questions were reviewed by the RDCRN Education Committee.

A second assessment is 12 questions that measure their level of comfort with their skills in performing RDR. These questions were created by the first author and evaluated and edited by the RDCRN Education Committee prior to implementation. Each question offers the participant a 10-point Likert scale with 0 =  no expertise to 10 =  expert. These questions were used to assess the Scholars knowledge of RDR-specific topics, such as small cohort statistics and forming collaborations researchers at other centers.

The Clinical Research Appraisal Inventory (CRAI-12) was developed by the University of Wisconsin to “assess an individual’s perceived abilities to perform tasks and activities needed to conduct clinical research”[7]. The CRAI assesses several aspects of clinical research by prompting takers to indicate their ability to successfully perform a task by selecting a single number from 0 to 10 that best describes their level of confidence. The assessment is broken up into sections: (1) conceptualizing a study, (2) designing a study, (3) collaborating with others, (4) funding a study, (5) planning and managing a research study, (6) protecting research subjects and responsible conduct of research, (7) collecting, recording and analyzing data, (8) interpreting data, (9) reporting a study, and (10) presenting the study. The scholar’s self-efficacy with clinical research is rated by indicating confidence level (0 = No Confidence to 10 = Total Confidence) [7]. Self-efficacy is considered an important factor in making career decisions and in outcomes of the decision [8, 9]. Because self-efficacy can change over a period of time, the RDRSP administers the CRAI-12 of all scholars’ pre and post the program to identify the impact of the program in trainees’ confidence in their abilities.

Qualitative evaluation interviews and analysis

To further evaluate the long-term outcomes of the program, all former Scholars were offered the opportunity to be interviewed [10, 11]. The interview was conducted by an individual (author JW) who was not involved in the curriculum creation or implementation. The purpose of the interview was to understand the participant’s experiences in the program. A semi-structured interview guide was developed to identify content of the program that was perceived to be beneficial (or not) for their long-term career development in RDR skills. The RDRSP Alumni council reviewed the questions prior to the interviews. Interviews were conducted over the telephone and were designed to be thirty minutes. Detailed memos were written after each interview [10]. The interviews were summarized and de-identified, and only de-identified summaries and thematic information were provided to the full research team. Thematic analysis was conducted iteratively using NVivo11 Plus, and data collection continued until saturation was reached [12, 13].

Statistics

Each Scholar’s performance was compared before and after course completion. Paired T-test was performed using Excel to determine significance. Participants who did not complete both the pre and post evaluation were removed from the study, as was prescribed by the IRB (allowing individuals to remove their data from the data set by not completing the final evaluation). We recognize that this may have led to a skewed data set (i.e. lower scoring pre-evaluation participants not completing the final evaluation).

Ethics statement

The Children’s National Hospital Institutional Review Board determined that the evaluation of this educational program was a Research Quality Improvement Study and, as such, was exempt from IRB review. Scholars were free to stop participation in the program at any time. While completion of the post-test was required for graduation from the program, participants were offered to have their results removed from the overall data set if they preferred. All data is presented as averaged values to prevent identification of any individual’s scores.

Results

The RDCRN Education Committee performed an initial Needs Assessment from among 106 trainees in 100 host institutions. Sixty-eight percent of trainees (72 of 106) replied to the survey and were either fellows or junior faculty members from 22 medical specialties who intended to remain in academic medicine. Others did not intend to stay in academic medicine or had a non-MD degree. The type of RDR they were pursuing was fairly equally divided between clinical (58.5%) and basic/translational (41.5%). Twenty-six groups of rare diseases were represented, overlapping largely with the respective RDCRN consortium with which they were affiliated. Thirty-eight percent of trainees were engaged in NIH funded mentored career development awards (K awards). Specific RDR training at their host institution covered 1-2 years for 68% of trainees, with the remainder being involved in longer duration training programs. Thirty-two percent of trainees collaborated with institutions outside of their host institution. Eighty-three percent of trainees presented their research at national meetings; and 97% published their work in a peer reviewed journal. There was also a strong relationship with the NIH funded CTSA training programs at the trainee’s host institution. Mentorship was a strong part of the trainees’ experience, with over 96% indicating they had a mentor for their RDR project. Almost 80% reported that they have had sufficient time to accomplish their research goals, with most having 50–75% funded protected time. Forty percent had attended one of the RDCRN’s R13-supported training conferences. Almost 70% of trainees indicated that they plan to remain in the field of rare diseases, and over three-quarters planned to remain in academic medicine. These individuals also indicated that they planned on submitting an R or K award application in the 2 years following the survey.

We concluded that those individuals engaged in a RDCRN-based training program were receiving adequate clinical research support and mentorship. When asked what additional training was needed, there was strong support for a centralized training curriculum in RDR included multiple areas of content specific for RDR (Listed in Table 1). The top needs identified were in the areas of clinical research skills (i.e. biostatistics, grant/manuscript writing, genomics, creation of teams, networking), Laboratory and Research Management (i.e. budgets, working with advocacy groups, patient recruitment, personal mentoring, human resource training). This needs assessment provided us with a context for developing the R25 RDR curriculum application. While the surveyed individuals were all housed in RDCRN based programs, their needs became the basis for the RDRSP which was expanded to individuals without access to a RDCRN training program and its resources.

Table 1

Description of educational content for the Rare Disease Scholars Program as it iteratively expanded based on feedback from scholars

	Specific Sub-topics Identified	Needs Assessment	Incorporated Year 1	Incorporated Year 2–4
Clinical Research Skills	RDR Biostatistics	x	x	Expanded
	Grant Writing with small cohort	x	x
	Understanding genomics	x	x
	Development interdisciplinary teams	x	x
	Manuscript writing	x	x
	Networking with others in RDR	x	x	Expanded: team projects
	Exposure to RDR outside sub-specialty or home institution	x	x	Expanded: travel and team science projects
Laboratory &Research Management	Budgeting in RDR	x	x
	Working with patient advocacy groups	x	x
	Patient recruitment		x
	Personal mentoring			x
	Personal accountability		x	Expanded
	Time management			x
	Human Resources (hiring, firing)			x

Scholars Outcomes

Recruitment of Scholars to the program relied on the Education Committee identification of RDCRN participants, word of mouth, and presentations at rare disease-centric meetings (National Organization for Rare Disorders, RDCRN Rare Disease meeting, American College of Medical Genetics, and Society of Inherited Metabolic Disorders). Four cohorts were included in the data set (Fig. 1). The most common specialty noted by participants was neurology, followed by genetics and nephrology. The “other” category included the lab-based (PhD most often) participants without a clear sub-specialty association, pharmacists, urologists, general surgeons and other specialties for which there were fewer than 3 participants in the program. Most participants held an MD/DO degree. Sixty-three percent of participants were female, and 89% were affiliated with a laboratory in the U.S.; Canada was the highest international participant.

Fig. 1

Top Pie Chart: Degrees of the Scholars admitted to the program (n = 130). Bottom Pie Chart: Specialties of the Scholars (n = 130). Other was used for any specialty field with one or fewer participants.

Participants completed pre and post evaluations of an inventory of topics deemed important for RDR success, the CRAI-12 evaluation, and were asked their likelihood of being involved in RDR in 10 years. These pre vs. post evaluations showed a significant effect of the program based on paired T-test analysis (Fig. 2). Subject content knowledge showed an averaged increase each year; combined data from four years is presented in Fig. 2. Each item on the CRAI-12 ranges from 0 to 10, with a higher score indicating more confidence. Scores were normalized to a 10 point scale for graphical representation. Pretest score increased from 6.83/10 to 8.07/10 (Normalized to a 10 point scale, each question p < 0.001). Comfort with RDR methodology increased from 6/10 to 7.5/10 (p < 0.001). CRAI-12 scores increased by 30% (p < 0.001) with clinical research self-efficacy increasing from an initial mean score of 6.0/10 to 7.9/10 (p < 0.001). Likelihood of remaining in RDR for 10 or more years increased from 6.3/10 to 8.4/10 (p < 0.001).

Fig. 2

Scholar outcomes pre and post the year-long R25 program. Participants completed pre and post evaluations of an inventory of topics deemed important for rare disease research success, the CRAI-12 evaluation, and asked their likelihood of being involved in Rare Disease research in 10 years. All were significantly increased based on paired T-test analysis (T-test value p < 0.001). Only participants that completed both the pre and post-test were included in the statistical analysis. Including individuals that completed only one of the tests (pre or post) did not change the overall trend or statistical significance of the data set (n = 47–61, T-test p < 0.01).

Qualitative feedback evaluation

While the volunteer nature of providing qualitative feedback likely skewed our analysis to more positive results, the identification of themes that are important to reinforce and prioritize in future training programs was deemed important by the oversight team. The data supported the generation of a theme where “participants acknowledged importance of research skills and knowledge but also highly-valued the program for the development of their ‘soft-skills’ in research.” Primary categories supporting this theme were collaboration (n = 8), mentorship (n = 8), research skills (n = 9), and personal development (n = 9) (Table 2). Each category had multiple sub-categories identified that led to suggestions for interventions needed to promote success of the program. The combination of needing resources, support, and community-building was seen across multiple categories. We believe that this theme is applicable to all types of training programs working with early career individuals in the biomedical sciences.

Table 2

Qualitative feedback identified a theme supporting the need to develop soft-skills in rare disease research. Scholars were interviewed using a semi-structured template to explore areas of improvement and benefits of the Scholars Program. All Scholars from the 4 years were offered an interview. Interviews were conducted across all cohorts equally and continued until thematic saturation was accomplished

Theme: “Participants acknowledged importance of research skills and knowledge but also highly-valued the program for the development of their
‘soft-skills’ in research.”
Category	Description	Sub-Category	Examples of feedback from Scholars
Collaboration (n = 8)	Increased opportunities to network and breakdown silos	Networking within program	-I met a lot of people that can relate to working with rare diseases.
			-I have about three or four people I can get advice from. Just knowing that there are people.
		Engaging with the research system	-Gaining visibility is the best part about the program. I went to NIH, spoke with program officers, and gave presentations.
			-I presented at the RDCN steering committee
		Staying Connected	-I’d love to continue meeting with [program leader] monthly and staying in touch with the scholars.
			-We had one face to face and that was good. . . seeing them face to face helps you make connections beyond the online portal. You saw where they were coming from and you could make collaborations and even friends. It would be nice to do this twice.
			-If there was somebody who could keep the group continuously conversing that would be helpful.
Mentorship (n = 8)	Opportunities to find mentors and advisors to receive guidance from experts	Learning from those more senior	-It was good to learn from someone 5–10 years ahead of me. Life advice in research—it is so different from medical school.
		Outsider perspective from program leader	-Seeing her [program leader] leadership and professionalism was great for me. I know I have a good professional role model.
			-I have mentors at my home institution but to have an outside perspective. It was good to hear different institutional support.
Research Skills (n = 9)	Opportunities to learn the nuts and bolts of research	Funding &Grants	-The resources—being able to use them when you’re writing grants.
			-. . . tapping into unusual places like the Department of Defense
			-There were talks about searching for funding and knowing where to look.
		Reading &Writing	-I learned how to read articles more critically
			-Helped me to write an IRB while thinking about what I wanted to do now and in the future.
		More content	-As a clinician, I felt like I needed a 1:1 grant session
			-We need to know more about contracts, what makes it good or bad.
			-I would add more statistics, analysis. . . knowing how to get the best information from a small sample size.
Personal Development (n = 9)	Opportunities to learn how to balance multiple roles, time, and advance one’s career.	Career Advancement	-The program benefitted me [at work] because it elevated my career as I’m now the site leader. The program fueled my passion to engage more and find collaborations.
			-After the program, I was able to secure a new position as a post-doctoral researcher.
			-I was one of three abstracts selected for an oral presentation and this was a boost.
		Time Management	-I learned how to optimize my small amount of time to research
			-For time management there was a lot of opportunities to dig into so additional resources for other topics and easily digestible options like the short ted talks.
			-It was good to hear tips but also that others are struggling.
		Describing your research	-. . . o one learns this in books. I learned how to give an elevator pitch about my ideas.
			-It [guest lecture] inspired me to when I give results especially with complicated things to custom create a handout.

Discussion

RDR is in a sense a new discipline. Representing over 7,000 diseases, it crosses specialty boundaries in medicine and requires team science. The NIH recognized this with the establishment of the RDCRN in 2003. Yet, fellowship and career development training programs tend to be subspecialty or disease specific. So, while trainees and early career faculty have developed a knowledge base in their specialty, if they intend to pursue RDR they need an additional set of skills. This was identified by the needs assessment described in this paper.

In response to this identified need we developed the RDRSP. This one-year curriculum uses a blended learning approach that included in person meetings, webinars and a LMS. While we initially focused on teaching skills specific to RDR, such as single subject design, recruitment in vulnerable populations etc., we found that equally important were more general topics of how to prepare oneself for a successful academic career. This finding of the importance of combining specific skills with general skills training is likely applicable to all training programs. We also found that developing a community of scholars who were diverse but focused on a specific topic, RDR, yielded a cohesive cohort of learners. The total investment of time, less than 6 weeks of training over the year curriculum, was able to be managed by even the busiest specialty scholar. The effectiveness of the program was demonstrated by a significant increase in tested knowledge and self-efficacy, as well as a positive quantitative assessment of the course value.

Furthermore, by creating an agile curriculum with space for year-specific content, each cohort’s unique needs were addressed. For example, in 2020 when COVID-19 became a hurdle for early career researchers, sessions were executed to strategically support researchers working from home. Quickly, the need for psychosocial support led to several sessions with a “happy hour” theme that offered a community and caring for scholars from the front lines of New York City COVID wards, to those with young children sequestered at home, and those without local family struggling with loneliness. Thus, the agility of the curriculum to meet current needs created a program that could remain timely even during the 2020 pandemic or the next hurdle that will present itself to rare disease clinical researchers.

The unique approach to this program allowed for creation of a community of diverse rare disease researchers, leading to increased collaboration, support, and skills for the Scholars. We believe that basic career development skills and creating a network of like-minded individuals are essential to effectively equip and encourage early career researchers. Both the early surveys and the follow-up qualitative interviews identified the mentorship theme. While this is certainly not a novel concept in education [14], the importance of various types of mentorship and developing a broad mentorship team came to light. With progression of the program over time, additional training and mentorship in non-research-based topics was identified as a priority. From issues such as human resource management and time management to life/work balance, the topics most appreciated were often not scholarly, but more the soft skills of interpersonal relationships. Time spent on discussing these skills should be prioritized to ensure that these needs are being met.

In our ongoing work to keep the value of the program high to participants, we have attempted to decrease the amount of any “busy work” for participants. While other programs have pre-reading and post activities, which have been shown to increase retention of topics, we instead used the approach to familiarize the participants to the resources and then make them readily available when needed (adult learning theory). Thus, all topics are recorded and stored on LearnRD.com for participants to return to use. The portal access is allowed indefinitely after completion of the course and alumni Scholars are welcome to join future conversations or topics as they need additional resources or support. This approach has increased the interaction with the material without increasing the initial time commitment, which can lead to learner fatigue. The only “scut” work is the pre and post tests and creation of a poster for the final presentations. Due to the concerns that some did not want to create an additional poster, we now accept poster presentation from other meetings and review their posters remotely.

An alumni network was created in 2018 to connect scholars of different years with each other. This group now works with the primary team on recruitment, Scholar selection, and activity oversight. We continually collect updated CVs of former scholars to share and we also use ORCID IDs to keep track of new publications by our scholars. Future data collection will focus on participant publications, funding, and faculty positions as outcomes of the cohort. By monitoring long term implications of this program, we will best direct future training programs for clinical researchers. The goal is to optimize the training program to give maximal resource and skills with the minimal time and cost for busy early career individuals.

Outcomes of the blended learning RDRSP curriculum showed improved readiness for clinical research. Participant’s feedback shows increased expectation of staying in RDR, increased understanding of important concepts in rare disease research, and increased comfort with clinical research topics. We believe that a blended learning curriculum based on both subject specific and general career development topics may be applicable to other niche specialties in medicine.