Assessing publication rates from medical students' mandatory research projects in the Netherlands: a follow-up study of 10 cohorts of medical students.

OBJECTIVES: The medical field is facing a clinician-scientist shortage. Medical schools could foster the clinician-scientist workforce by offering students research opportunities. Most medical schools offer elective research programmes. Subsequently, a subset of doctors graduates without any research experience. Mandatory research projects may be more sufficient to develop clinician-scientist, but take more supervision and curricular time. There is limited insight in the scientific outcomes of mandatory research experiences. This study aims to examine publication rates of a mandatory research experience, identify factors associated with publication, and includes postgraduate research engagement. DESIGN AND
SETTING: Prospective follow-up study involving 10 cohorts of medical students' mandatory research projects from Leiden University Medical Center. PARTICIPANTS: All medical students who conducted their research project between 2008 and 2018 (n=2329) were included. MAIN OUTCOME MEASURE: Publication rates were defined as peer-reviewed scientific publications, including research papers, reviews, and published meeting abstracts. Postgraduate research engagement was defined as research participation and dissemination of research at scientific conferences or in journals.
RESULTS: In total, 644 (27.7%) of all mandatory research experiences resulted in publication, with students mainly as first (n=984, 42.5%) or second author (n=587, 25.3%) and above world average citation impact (mean normalised journal score 1.29, mean normalised citation score 1.23). Students who conducted their research in an academic centre (adjusted OR 2.82; 95% CI 2.10 to 3.77), extended their research (adjusted OR 1.73; 95% CI 1.35 to 2.20), were involved in an excellency track (adjusted OR 2.08; 95% CI 1.44 to 3.01), or conducted clinical (adjusted OR 2.08; 95% CI 1.15 to 3.74) or laboratory (adjusted OR 2.16; 95% CI 1.16 to 4.01) research published their research more often. Later as junior doctors, this group significantly more often disseminate their research results at scientific conferences (adjusted OR 1.89; 95% CI 1.11 to 3.23) or in journals (adjusted OR 1.98; 95% CI 1.14 to 3.43).
CONCLUSIONS: Our findings suggest that a significant subset of hands-on mandatory research projects with flexible learning pathways result in tangible research output with proper impact and that such successful experiences can be considered as diving board towards a research-oriented career. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first prospective cohort study that bibliometrically reports scientific outcomes (publications) of a hands-on mandatory research experience including postgraduate research engagement in 10 cohorts with over 2000 medical students in total.

Insight in scientific outcomes (publications) of mandatory research programmes fills a gap in the literature since previous studies mainly focus on elective research outcomes with a subset of students graduating without any research experience.

Our study identified student and project factors associated with publication of a mandatory research project, thereby providing insight how to reach high academic levels among medical students.

Insight in postgraduate research engagement is limited due to loss to follow-up and non-response.

Publication rate is subjected to an underestimation of actual published papers due to publication delay and false negative cases.

Introduction

All doctors should be able to critically appraise and use research in clinical practice to keep up to date and apply evidence-based practice within their field of expertise.1 2 Additionally, society needs doctors to conduct research and contribute to new developments and knowledge.3 Clinician-scientists, that is, doctors with research expertise and engagement, do not only conduct research, but also play significant roles in directly translating clinical observations to the bench and in moving research findings into everyday practice. Thereby they contribute importantly to the development of tomorrow’s healthcare as newly invented medical solutions and developments will reach patients sooner.4 5 The adoption of this scholarly competency in frameworks as the US Accreditation Council for Graduate Medical Education and the Canadian Medical Education Directives for Specialists reflects the importance of doctors who conduct research.6 7 Despite this recognition, the number of clinician-scientists globally declined over the past few decades resulting in a shortage.8–12

A solution to overcome the clinician-scientist shortage is to engage medical students in research endeavours during medical school. Efforts concentrated on the research engagement of medical students and consisted of extracurricular or intracurricular research activities, the latter either mandatory or elective research programmes.9 13–17 To date, most medical schools only offer elective or extracurricular research programmes, such as summer schools and scholarly concentration programmes, mostly aimed at excellent or highly motivated students.9 18–20

Several studies demonstrated that these undergraduate research experiences (voluntary as well as mandatory) enhance research skills such as searching and critically appraising evolving medical literature, designing research, data analysis, academic writing and presenting.9 16 17 19 21–23 Furthermore, they foster research self-efficacy, positive research perceptions, motivation for research,24–27 and, on the long term, the ambition to pursue an academic career.9 11 14 17 19 28–30 Even more, some research experiences result in peer-reviewed publications, often assumed as an objective measure and a proxy for the ultimate learning experience of research programmes, and suggested to be one of the factors related to persistence within academic medicine.31–33 Considering these positive effects, one may argue that every medical student should engage in hands-on research. However, as current research experiences are mostly voluntarily, about 30%–70% medical students graduate without any hands-on research experience.17 20 34 Some of these students initially may lacked interest and motivation, while others did not participate in research due to time pressure, a lack of supervision, and/or opportunities.35–37

It may well be that elective programmes involve above average motivated and committed students.37 As such, previously described beneficial outcomes of elective research experience may differ from mandatory research experiences. Furthermore, given the limited curricular time, the benefits of mandatory research projects must outweigh the efforts of compressing an already tight learning schedule. In addition, proper supervision of mandatory research projects may demand substantial efforts from scientists and faculty, which might be justified if these research projects result in at least some publications. To the best of our knowledge, however, a large cohort analysis of medical students’ mandatory research output has not yet been conducted. This may prove useful to medical schools with established mandatory research programmes or others considering the introduction of a mandatory research experience. It can provide insight into the effects of mandatory research and help to influence policy around the introduction of mandatory research experiences and the enhancement of research-oriented careers among medical students. Therefore, this 10-year cohort study aims to investigate the scientific output based on number of publications resulting from mandatory research projects and identify key factors associated with these publications. In addition, we explore scientific engagement after medical school including the residency period and early clinical careers.

Methods

Setting

In the Netherlands, all eight medical schools educational programmes are based on the Dutch National Blueprint for Medical Education. The programme consists of a 3-year bachelor programme and a 3-year master programme. Individual mandatory research projects are longstanding part of each Master of Medicine and were already incorporated in all Dutch medical curricula even before 1970. Students have 4 to 6 months for a full-time, authentic, and hands-on research experience. They go through the phases of the empirical cycle by conducting their own research and develop research skills such as searching and critically appraising literature, designing research, analysing and interpreting data, academic writing and presenting. During this project students have much autonomy, for example, in arranging their internship at a health institute and department of preference, in choosing a research domain (eg, laboratory research, clinical research, public health research). In addition, students are free to choose the timing to conduct their research (ie, before or after clerkships) and to extend their research project with 5 or 10 weeks. During the research project, students fulfil the role of the primary investigator and receive input from one or few supervisors. Supervision is carried out by faculty staff members, that is, (clinician-)scientists or PhD candidates. As final products, students write a research report and orally present their findings at the department.

Materials and definitions

Publication rates and factors associated with publication

This follow-up study included all medical students from Leiden University Medical Centre, who started their mandatory research internship between 1 January 2008 and 1 January 2018. The latter cut-off was to allow for lag time between project completion and peer-reviewed publication. We extracted names and initials of all students together with the name of the supervisor(s) from course registration systems, together with other student factors (eg, participation in an excellency track) and project factors (eg, planned duration of the research project). Scientific output is operationalised as peer-reviewed publication rates of research projects. We included the following publications: research articles, meeting abstracts, and reviews, as these are described as most common measures for research success.9 Letters to the editor, editorial materials, corrections and news items were excluded. Within the publications, we looked at author position of the student, year of publication, and impact. For the latter, we used the mean normalised citation score (MNCS) as impact ratio of research articles, compared with the world citations average in the subfields in which the research unit is active, as well as the mean normalised journal score (MNJS) as impact ratio of the journal in which a research unit has published (the research unit’s journal selection), compared with the world citations average in the subfields covered by these journals.38 39

Postgraduate research engagement

For postgraduate scientific engagement, we developed a questionnaire (online supplemental appendix A) regarding research activity after graduation (other than accomplishing publication(s) of the research project). We defined conducting research as postgraduate participation in research, whether or not in the form of a PhD programme, next to disseminating research results, that is, publishing articles in journals or provide oral presentations at scientific conferences. This questionnaire was part of an institutional questionnaire about different postgraduate career pathways. Those who graduated before May 2019 were sent a questionnaire for postgraduate (ie, after medical school) follow-up.

Procedure

To identify mandatory research projects that resulted in a peer-reviewed publication, we searched full names of the students and supervisor(s) together with filters based on department and year of research project using validated bibliometric methods. Bibliometric methods enables to track scientific output of individuals strengthened by mapping individual hits to larger sets of publications (ie, author clustering), with more robust bibliometric scores of citation impact as a result. Author clustering algorithms are more accurate when more information is available, as publications can be clustered even when the initials do not match exactly.40 41 Consequently, students are more susceptible for false positive results due to a minimal oeuvre compared with their prolific supervisor(s). Therefore, as a first step, between December 2019 and January 2020, we searched names of all supervisor(s) in the in-house database in of one of the most comprehensive and widely used publisher-independent global citation database, Web of Science (WoS), at the Centre for Science and Technology Studies using a validated algorithm. This bibliometric search resulted in a list of clustered oeuvres of the supervisors. Second, we searched publications that also included the students’ name they supervised and considered these papers as publication that resulted from the research project. Common problems in such searches are false positive or negative assignments of papers, due to common Dutch names, forgotten initials or spelling errors.42 This problem is applicable for the bibliometric search to identify the oeuvre of the supervisor, as well as searching students’ names within this oeuvre. Therefore, we checked all included publications to distinct if the published paper matched the topic of the research project, department, and institute. Some false negatives are inevitable as a subset of students published in journals that are not indexed in the WoS-database (eg, a Dutch-language journal or English-language journals not processed for WoS), or because of spelling errors, missing initials, changed names, or changed initials. Complementary to bibliometric analysis, we performed a sensitivity analysis by manual assignment on a random sample of 150 research projects. By searching key words based on research title, next to students’ together with supervisors’ names on Google Scholar, PubMed, LinkedIn, and ResearchGate, 12% (n=18) false negatives and no false positives were identified. We critically studied these publications to identify explanations for being false negative in order to improve our search and added the publications to our dataset.

After graduation, the Alumni Office registers medical graduates of whom 80% agreed to receive questionnaires. To identify long-term scientific engagement, we invited medical graduates from 2008 up until May 2019 by email with a link to the online questionnaire. Participants received information on the study and an informed consent form.

Analysis

We used descriptive statistics to describe demographic variables. We grouped the population into a publisher and non-publisher group to analyse factors associated with publication. An unpaired t-test was used to compare group differences (eg, age and gender) between the publishers and non-publishers. To identify what student and project factors are associated with publication, we used logistic regressions, both crude and adjusted for possible confounding variables. Additionally, regarding publications, impact score, author position and mean publication delay were analysed.

For sensitivity analysis to identify possible (non-)response bias, we performed unpaired t-tests to disclose any differences between features (eg, age, publication) of responders and non-responders of the alumni questionnaire. To identify postgraduate outcomes associated with publication of the research project as student, we used multiple logistic and linear regressions. We adjusted for age, gender, and previous participation in an excellency track (ie, Honours programme) as possible confounders.43

We used a 95% CI to determine statistical significance. We analysed our data using IBM SPSS Statistics V26.0.

Patient and public involvement

No patients involved.

Results

Publication rates and factors associated with publication

Between 2008 and 2018, 2329 medical students had started their research internship. These students were 20 to 39 years (M=24.3, SD=2.0). Of all 2329 students, 1561 (67.0%) were female. In total, 644 students (27.7%) had one or more publication(s) as a result of their research project. Within the group that had published their research project, 57% has published one article, 15% has published two articles, 8% has published three articles, and 20% has published four or more articles related to their research project. Publishers and non-publishers did not differ in gender. However, they did differ in age with a mean difference of 0.46 years (95% CI 0.29 to 0.63). Further demographics are shown in table 1.

Table 1

Demographics of student and project factors of mandatory research projects (n=2329)

Variable	Mean (SD) or no (%) of students
Student factors
Age at start of project (years)	24.3 (SD 2.0)
Female	1561 (67.0)
Participated in a Bachelor excellency track	125 (5.4)
Project factors
Timing before clerkship	1167 (40.9)
Extended duration using elective weeks
5 weeks   10 weeks	636 (27.3)523 (22.5)
Research type
Clinical research   Laboratory research   Public health research   Other	1547 (66.4)422 (18.1)259 (11.1)101 (4.3)
Academic medical centre	1731 (75.5)
Location abroad total, whereof	216 (9.3)
Low-income country   Middle-income country   High-income country	24 (11.1)25 (11.6)167 (77.3)

Students who (1) were involved in an excellency track, (2) voluntary extended their research project with 10 weeks, (3) conducted their research in an academic medical centre, or (4) conducted clinical or laboratory research published their research project more often (table 2). After adjustment for potential confounding variables, effects of timing of the research project and doing research abroad lost significance. When looking at research abroad more closely, 24 projects were conducted in a low-income country, whereof one was published (4.2%) and 25 projects conducted in middle-income countries had no associated publications. Projects conducted abroad in high-income countries (n=136) resulted in 31 publications (18.6%).

Table 2

Student and project factors associated with published research projects

Factors associated with publication	Crude OR	95% CI	Possible confounders adjusted for	Adjusted OR	95% CI
Participated in an excellency track
No Yes	1.002.31*	1.60 to 3.32	Age, gender	1.002.08*	1.44 to 3.01
Timing of research project
Before clerkships After clerkships	1.000.89	0.74 to 1.07	Age, gender, participation in an excellency track, type of institute, project duration, research type	1.001.16	0.92 to 1.46
Project duration
Not extended Extended with 5 weeks Extended with 10 weeks	1.001.201.47*	0.96 to 1.491.17 to 1.84	Age, gender, participation in an excellency track, type of institute, project duration, research type, timing of the research project	1.001.201.73*	0.95 to 1.511.35 to 2.20
Research type
Other Public health Clinical research Laboratory research	1.001.342.10*2.24*	0.72 to 2.471.21 to 3.611.26 to 3.97	Age, gender, participation in an excellency track, project duration, type of institute, timing of research project	1.001.352.08*2.16*	0.69 to 2.611.15 to 3.741.16 to 4.01
Type of institute
Non-academic centre Academic centre	1.002.60*	1.97 to 3.45	Participation in an excellency track, project duration, research type, timing of the research project	1.002.82*	2.10 to 3.77
Country
The Netherlands Abroad	1.000.41*	0.27 to 0.61	Age, gender, participation in an excellency track, type of institute, project duration, research type, timing of the research project	1.000.47	0.47 to 1.40

*P<0.05

Sensitivity analysis showed comparable results during the study period, publication rates excepted. The latter declined in the last 3 years (figure 1). Of all 2182 publications, 1451 (66.5%) were research papers, followed by 609 (27.9%) meeting abstracts and 122 (5.6%) reviews. Of all students who published their research project, almost half of them (46.0%) had at least two types of publications (eg, research paper and meeting abstract). When distinguishing research papers and reviews from meeting abstracts, over two-thirds (69.7%) of students with a meeting abstract had a research paper and/or review as well. Students were first author of 984 publications (42.5%), followed by second author of 587 publications (25.3%), third author of 349 publications (15.1%), and fourth author of 398 publications (17.2%). Publications were cited with an average of 17.8 citations per publication. The MNJS in which students published was 1.29, with an MNCS of 1.23. The average lag time between the research project and first publication of this project was 2.4 years.

Figure 1

Published research projects per year.

Postgraduate research engagement

In total, 250 alumni (11% of all included students) participated in the survey. Table 3 shows main findings.

Table 3

Postgraduate research engagement after publication of the undergraduate research project

Association between publication and postgraduate research engagement	Crude OR	Crude 95% CI	Adjusted OR*	Adjusted 95% CI
Postgraduate research participation	1.26	0.76 to 2.09	1.12	0.66 to 1.90
Postgraduate research publication(s)	2.11†	1.25 to 3.54	1.98†	1.14 to 3.43
Participation in a PhD programme	1.84†	1.10 to 3.08	1.69	0.98 to 2.90
Postgraduate research conference contribution	1.99†	1.19 to 3.34	1.89†	1.11 to 3.23

*Adjusted for the following confounders: age, gender, participation in an excellency track

†P<0.05

The mean time between graduation and participation was 5.1 years (SD 2.7, median 4.5 years). We found no significant differences between the responder and non-responder group in gender, Honours programme participation, timing of research project or year in which the research project was started. The groups significantly differed in publication rates, with more publications of the mandatory research project in the responder group (mean difference −0.18, 95% CI −0.25 to 0.12).

Students who had published their undergraduate research project were more likely to publish (adjusted OR 1.98, 95% CI 1.14 to 3.43) after medical school or to share their research at a scientific conference (adjusted OR 1.89, 95% CI 1.11 to 3.23). Logistic regression showed a crude association between publication of the student research project and later enrolment in a PhD programme as medical doctor (OR 1.95, 95% CI 1.16 to 3.29). After adjusting for participation in an excellency track as possible confounder, this effect became marginally smaller (OR 1.74, 95% CI 1.01 to 3.00), as shown in table 3.

Discussion

The integration of scientific research projects into medical school programmes to develop scholarly doctors or even clinician-scientists is a widely discussed topic. Our study revealed that more than one in four medical students publishes findings of their mandatory research project in a peer-reviewed paper, mainly as first or second author. These papers are apparently of good quality as they passed peer review procedures as well with impact scores above world citation average, even though these students can be considered as relatively young researchers. Students who were younger, participated in an excellency track, conducted their research in an academic medical centre, and voluntary extended their project with 10 weeks by using elective weeks for the research project were more likely to publish their undergraduate research project. Timing or type of research did not impact publication rates.

Only few other studies have focused on the scientific output (publications) of mandatory research experiences. Three of these studies were conducted in private schools, with small amounts of graduates every year, which limits the generalisability, usability and applicability for education systems of public.44–46 Two other studies conducted in public schools reported publication rates of 11% and 17%, however, these were outdated or included less than 230 research projects.47 48 One other study conducted at a Dutch single institute included 551 research projects and describes a publication rate of 27%, in line with our results.49 Studies on elective or extracurricular research experiences reported publication rates between 14% and 75%, with limited numbers of students included usually selected on excellence or prior research interest.9 31 44 50 Although publication rates vary, at best 75%, this concerns a subset of an already pre-selected group representing the minority of all students. From this perspective, the scientific output based on number of publications of mandatory research experiences found in this study is relatively high and indicates actual scholarly development of medical students when research is imposed on them. It is important to note that comparisons of measured output reported in other studies should be done carefully, as variability in publication rates in the literature is likely attributable to differences in objective output for example, including or excluding meeting abstracts or oral presentations (the number of confirmed publications is, as expected, lower). Moreover, our study shows that student and project factors (eg, duration) are associated with publication rates of mandatory research experiences and might vary between institutes.

Student and project factors associated with publication provide insight in how faculties can optimise research experiences to foster the future clinician-scientist workforce. In line with Möller and Shoshan, we have found no gender difference regarding publication rates.51 Other studies are inconclusive and reasons behind a potential gender difference regarding publication rates remain unclear.26 50 52 53 While timing of the research projects apparently does not affect publication rates, extended duration results in higher publication rates. Half of all students are motivated to spend their elective weeks on extension of their research project. More time for research evidently leads to more mature research products with increased publication rates, which is also described by Dyrbye et al.44 Lastly, this study showed that projects conducted in an academic medical centre more often resulted in a publication. This might be attributed to the supervisor. Perhaps, projects conducted in an academic medical centre are more ‘publishable’ than others, as they are supervised by (clinician-)scientists working in an academic environment. This academic environment is highly research oriented as it includes research departments (ie, department of statistics and department of epidemiology) and facilitates, for example, journal clubs and research courses. Another explanation could be that these supervisors are more experienced in publishing research, as most clinicians in academic hospitals are involved in academic activities next to clinical care. Indeed, Alamri et al found that students with academic supervisors publish more often than those with non-academic clinicians as supervisor.50

Previous research has not demonstrated that mandatory research in medical school leads to a more productive academic career.9 22 This study provides a first insight in scientific engagement in the first years as medical doctor. It seems that graduates after publication of their research project tend to be more often involved in research and doctoral programmes, but this was not significant. However, when they did, they had significantly more scientific impact as they were two times as likely to disseminate their knowledge via peer-reviewed publications or presentations at scientific conferences. Perhaps, as (pre-)resident, these doctors are more scientifically literate and conduct high(er) quality research, which in turn might lead to more accepted published papers and orals at conferences. Another explanation is the power of success experiences for self-efficacy levels.26 Published student research projects might comfort students about publication issues and the dissemination of scientific knowledge and fosters future publications.31 This is an interesting outcome, as dissemination of research findings is essential for translating scientific outcomes to clinical practice and enhance evidence based patient care, considered as the most important aspect of clinician-scientists.

Furthermore, there is also the aforementioned selection effect for research opportunities to preferably hiring medical graduates who have published before.54 As a result, we cannot firmly state that the association with postgraduate research engagement is regardless or if because they had a greater interest in research, and whether the publication of their scientific work had directly benefitted postgraduate research opportunities. At the same time, unknown makes unloved; one may argue that there is a subset of students who on beforehand do not have the ability to take on extracurricular activities next to the overcrowded formal curriculum, hold inaccurate perceptions, or, perhaps, even do not have initial interest in research at all.35 36 This seems undesirable, as other studies showed that a significant subset (30%–70%) of students graduates without any research experiences,17 20 34 next to the clinician-scientist shortage. A mandatory research experience can provide them with an opportunity to explore how much fun it is, and an experience of success when they successfully fulfil their own research project, or even publish their first paper. As this is assumable, but cannot be drawn from our data, it would be worthwhile to explore if undergraduate mandatory research experiences positively affect research motivation, perceptions, and self-efficacy, and thus can foster future clinician-scientists who perhaps would have missed out on future research engagement when a first research experience would not have been imposed on them.

Our study has several limitations. First, although bibliometric methods are widely accepted and used for large-scale analysis of scientific output, false positive and negative results might occur. Sensitivity analysis resulted in 12% suspected false negatives and 0% false positives, suggesting that the observed 27.7% publication rate may underestimate the actual rate. Further adaption of our bibliometric search strategy risks the inclusion of false positives. Additionally, it is likely that conference proceedings were under-recognised in our study, as we included conference presentations as evidenced by publication of the associated abstract; however, many conferences do not publish abstracts. Therefore, we have to accept that our result is subjected to an underestimation of the number of actual publications. This is further strengthened by publication delay. Most publications appeared in the literature 2.4 years after research completion, a lag-period that is in line with findings from other studies and is especially applicable to research projects that have started at the final phase of our inclusion period.47 50 55 This might explain the decrease observed in figure 1, when looking at the last years of the analysis, as papers might still be in the process of getting published (‘the pipeline’).

As second limitation, we conducted this study at a single institution. However, van Eyk et al showed very small differences between Dutch medical schools’ scientific training regarding timing, duration and European Credits, as well as students’ publication rates during medical school.56 Therefore, we assume that our results are representative for other medical schools with similar mandatory research training.

A third limitation is that postgraduate responses were voluntary and despite the exact response rate is not known, 11% of all students were included for long-term follow-up. Although this is low, it does not substantially deviate from response rates of medical education surveys elsewhere in the literature. As a result, response bias might occur, as perhaps ‘publishers’ are more motivated to participate in our survey.

Conclusions

To our best knowledge, this is the first study investigating objectively verified publications rates as a result of undergraduate mandatory research experiences, together with associated factors and postgraduate outcomes in over 2000 medical students. Next to all students having experienced an authentic hands-on research project before becoming a clinician, a significant proportion of authentic undergraduate mandatory research experiences have great scientific value, judged by an overall publication rate of at least 27.7% of all medical students, with mainly first or second author positions and above world average citation impact. Especially when medical schools provide the opportunity to conduct research in an academic environment and facilitates flexible pathways regarding the duration and curricular position with respect to clerkships, for those who are willing to invest more. After experiencing such high levels of scholarly achievement during medical school, as young doctor, this group also more often disseminates their scientific findings with the field, enhancing the translation of research to clinical care, considered as one of the unique and distinctive aspects of clinician-scientists. As such, mandatory research experiences not only equip all future doctors with basic research knowledge and skills, but can also serve as breeding ground for potential clinician-scientists and can be perceived as worth it when countering the current decline in clinician-scientists.