Medical Student Research: An Integrated Mixed-Methods Systematic Review and Meta-Analysis.

IMPORTANCE: Despite the rapidly declining number of physician-investigators, there is no consistent structure within medical education so far for involving medical students in research.
OBJECTIVE: To conduct an integrated mixed-methods systematic review and meta-analysis of published studies about medical students' participation in research, and to evaluate the evidence in order to guide policy decision-making regarding this issue. EVIDENCE REVIEW: We followed the PRISMA statement guidelines during the preparation of this review and meta-analysis. We searched various databases as well as the bibliographies of the included studies between March 2012 and September 2013. We identified all relevant quantitative and qualitative studies assessing the effect of medical student participation in research, without restrictions regarding study design or publication date. Prespecified outcome-specific quality criteria were used to judge the admission of each quantitative outcome into the meta-analysis. Initial screening of titles and abstracts resulted in the retrieval of 256 articles for full-text assessment. Eventually, 79 articles were included in our study, including eight qualitative studies. An integrated approach was used to combine quantitative and qualitative studies into a single synthesis. Once all included studies were identified, a data-driven thematic analysis was performed. FINDINGS AND
CONCLUSIONS: Medical student participation in research is associated with improved short- and long- term scientific productivity, more informed career choices and improved knowledge about-, interest in- and attitudes towards research. Financial worries, gender, having a higher degree (MSc or PhD) before matriculation and perceived competitiveness of the residency of choice are among the factors that affect the engagement of medical students in research and/or their scientific productivity. Intercalated BSc degrees, mandatory graduation theses and curricular research components may help in standardizing research education during medical school.

Introduction

The education of health professionals has seen two revolutions over the past century. The first revolution-marked by what is known as The Flexner Report in 1910- was the effective integration of basic sciences into health education. The second revolution, initiated by the Welch-Rose report in 1915, introduced the concept of problem-based learning into medical education. In 2010, a special report was published by a global commission, The Commission on Education of Health Professionals for the 21st Century, aimed at updating the standards of an ideal medical curriculum. The committee strongly recommended a new medical educational model that emphasized flexibility and adaptability of traditionally rigid curricula to local and community needs [1]. Despite these educational advances, there are certain aspects of medical education that remain unstructured and largely variant between medical schools; among these is medical student participation in research. Moreover, there is an alarming decline in the number of physician-scientists in the US, which threatens the progress of translational medicine in the upcoming era [2-4].

In the U.S., outstanding students willing to enter medical school may apply for the National Institute of Health (NIH) funded Medical Scientist Training Program (MSTP) [5]. This program offers students the opportunity to get a good feel for what a physician-scientist career entails through a funded MD/PhD. The value of those MD/PhD programs is well established; a 2010 study by Brass et al, investigating the outcomes of half of all NIH-funded MD/PhD programs (24 programs in total) found that these programs were very successful at reaching their goals of training future physician-scientists. In fact, 81% of MD/PhD graduates landed academic positions and 82% of them were actively engaged in research [6]. Nevertheless, due to limited funding, MD/PhD graduates only constitute 3% of the US medical student population, highlighting the value of alternative pipelines for the creation of research-active physicians [7]. Moreover, organizational and contextual factors might make the support of costly MD/PhD programs difficult to implement in other countries.

Several other programs have also been devised to offer medical and health sciences students the chance to participate in research [8-13]. One of the common forms of medical student research engagement is Intercalated Bachelor of Science (iBSc) degrees. These are particularly common in the UK, and are characterized by research time-out periods between the basic and clinical years of medical school. Students who take intercalated degrees graduate with an extra BSc beside their medical degree. The value of such short-term research placements should not be underestimated. In fact, the benefits of undergraduate research have been discussed richly in the literature, though there were relatively fewer papers focusing primarily on medical student research [14-16]. Unlike many other degrees, a medical degree is at the interface of science and social service. It is therefore expected that the benefits of, and motivations behind, medical student participation in research are different from those of non-medical students [17].

A 2005 systematic review of the literature by Straus et al investigated the factors that influence career choice in academic medicine among residents, fellows and staff physicians [18]. Their review found a positive effect of having dual degrees or fellowships beside the medical degree, and of publishing research conducted during medical school. Further, the review highlighted the role of mentorship and desire to teach. Despite the presence of a large body of evidence investigating the impact of, and factors related to, medical student research, a systematic analysis of this evidence is missing. This makes the data seem conflicting and disorganized, and undermines the apparent overall strength of evidence.

This paper is a mixed-methods systematic review and meta-analysis of published studies investigating various aspects of medical student research, including its impact on the development of research-active physicians, difficulties faced by medical students performing research and potential solutions to overcome these difficulties. Our hope is that this work serves to complement the review by Straus et al, and helps provide a thorough overview of the evidence needed for curricular and educational policy reforms [18].

We aimed to satisfy the following objectives in this review:

Primary Objectives: (a) To examine the short- and long- term influence of curricular and extracurricular undergraduate medical research on the scientific productivity of medical students, measured by the number of published manuscripts, research awards or attainment of faculty rank. (b) To describe the influence of curricular and extracurricular medical student research on the career choice of medical students.

Secondary Objectives: (a) To explore the current forms in which medical students are engaged in research projects, as well as the prevalence of non-mandatory research exposure among medical students. (b) To identify the factors related to medical student engagement in research projects. (c) To investigate miscellaneous issues of relevance, including the pros and cons of research time-out periods (with a focus on Intercalated Bachelor of Science degrees), differences between countries with developing and developed economies and gender equality in medical student research engagement, perceptions and productivity.

Developing economies were identified according to the International Monetary Fund's World Economic Outlook Report [19]. We counted as a "medical student" anyone who is enrolled in the core medical school program, regardless of program duration, and whose graduation would guarantee the degree Bachelor of Medicine, Bachelor of Surgery (MBBS) or its equivalent (MD, in the US, for example). It should be noted that in the US model of medical education, admission into medical school is on a graduate-entry basis by default, and the first medical degree earned is called the "MD". In the non-graduate entry model, on the on the other hand, the term "MD" is reserved for higher research degrees (postgraduate degrees) in clinical medical and surgical disciplines. Graduate-entry medical students were included, but not MD/PhD students, residents or postgraduate students. The reasons behind excluding studies focusing on MD/PhD students is that this sub-population is considered to be different from the general student population, especially that their enrollment in the medical program was–by definition- meant to prepare them for physician-scientist careers. It may be argued that graduate-entry medical students who had a higher degree (MSc or PhD) at the time of matriculation also constitute a separate sub-population. Hence, we addressed any reported differences between these sub-populations in our results. "Medical student research" was defined as any activity performed by medical students that is driven by inquiry or hypothesis and that legitimately incorporates basic principles of the scientific method. This includes original research, review articles, case reports etc.

Methods

We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) statement guidelines in this systematic review and meta-analysis, and the relevant checklist can be found as [20]. Between March 2012 and September 2013, periodic searches were performed in the following databases for potentially relevant studies: MEDLINE, Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Database of Systematic Reviews, Cochrane Methodology Register (CMR), Educational Resources Information Center (ERIC), Center for Reviews and Dissemination (CRD), ISI Web of Science and Google Scholar. Further, we searched the bibliographies of the included studies for other potential publications on the subject. Our search strategy included the following keywords in various combinations: medical student; medical students; undergraduate; medical; research; intercalated; bachelor; BSc; iBSc; theses; thesis; developing. The search strategy used for PubMed was as follows: ((((((medical student research) OR undergraduate research) OR medical thesis) OR intercalated bachelor) OR intercalated BSc) OR iBSc) OR undergraduate research developing.

Inclusion criteria: All study designs, including cross-sectional, prospective, retrospective and interventional studies, randomized controlled trials and qualitative studies.

Exclusion criteria: Studies containing inadequate information about the participants and type of study; studies in languages other than English; studies assessing outcomes unrelated to medical student research; theses or commentaries; studies aimed at postgraduates or undergraduates other than medical students; studies whose main population was MD/PhD students. Graduate-entry medical students, nonetheless, were not excluded from this review.

Two of the authors independently reviewed the studies that met these criteria and any disagreements were resolved by consensus. Basic data extraction tables were then used to extract the main finding and characteristics of each of the included studies. Quantitative studies (reporting odds ratios (OR's), p-values, percentages or other statistical measures) were separated from qualitative studies in order to improve the judgment of cumulative evidence.

Qualitative studies were included in order to help contextualize the quantitative outcomes and to provide insights and entry points for future research. Qualitative studies were defined as those studies which satisfied the following criteria: a) Their aims did not include the extraction of quantitative outcomes and thus did not perform any statistical analysis; b) They present original research with clearly-defined study populations; c) They utilize qualitative research methods, including semi-structured and unstructured interviews, open-ended survey questions, focus groups and examination of records and documents.

An integrated methodology was utilized to assimilate quantitative and qualitative outcomes into a single mixed-methods synthesis [21,22]. After relevant studies have been identified, a thematic analysis was performed. The literature search and article inclusion/exclusion strategy was aimed at retrieving all articles relevant to the subject of medical students' research, without prior conceptions or theories about expected outcomes. Hence, our thematic analysis was data-driven (as opposed to being theory-driven) [22]. Quantitative and qualitative outcomes were discussed together under relevant thematic subject headings.

Two types of quantitative outcomes were used for meta-analysis: percentages (for explorative outcomes) and odds ratios (for interventional/associative outcomes). Whenever relevant or needed, the corresponding authors (or, if unavailable, other authors) of included studies were contacted to get the raw data needed for meta-analysis. In some cases, other outcomes beside the ones mentioned in the original paper were identified in the raw data and used for the meta-analysis.

Further details about the methodology used in this paper, including outcome-specific quality assessment, statistical methods used and the strategy used to tackle study heterogeneity and potential publication bias can be found in our supporting information ().

Results and Discussion

Our search returned 31,367 records in the various databases. After reviewing the abstracts, 31,111 were excluded because they were either duplicates in various databases or satisfied one or more of the exclusion criteria mentioned earlier. 256 articles met (or were suspected to meet) our inclusion criteria upon reviewing their abstract and were thus retrieved for full-text assessment. Eventually 79 articles were found to match the selection criteria and were included in this review. More details about the article selection process can be seen in

Of the 79 articles retrieved, 71 were of quantitative nature, seven were of qualitative nature and one had both quantitative and qualitative components. Fifty-two articles were self-reported questionnaire studies with response rates ranging from 7.9% to 100%. Ten survey-based articles had response rates less than 60%. Twenty-three studies used a more objective research strategy that relied on searching institutional databases and records, two used both questionnaires and objective database searching and two had an unknown/undisclosed methodology. There were 47 cross-sectional studies, 25 retrospective studies, three prospective studies, three intervention studies and one study with an unknown/undisclosed design. Fifty-seven studies were performed in a single institution (including four qualitative study) and 22 studies involved multiple institutions (including four qualitative studies). Further, there were 14 studies that reported the effects of certain research programs or initiatives, whose study population might or might not be affiliated with multiple institutions. Sixteen studies assessed the value of intercalated BSc's (iBSc's) and 14 studies were carried out in developing countries.

After thematic analysis was performed, the resultant themes and sub-themes, outlined in , also served as the scaffold for writing this paper. The data extraction and quality assessment worksheet and the relevant sensitivity plots can also be found in the supporting information files (and , respectively) [7,8,10,11,23-90].

Assessing the current situation

We assessed the current state of medical student research by focusing on two main outcome measures: interest in- and exposure- to research among the medical student population. Both of these outcomes are explorative in nature (rely on proportions rather than odds ratios) and have been quantitatively pooled to yield a weighed estimate value. The results have been summarized in [7,10,26,28,32,47-49,52,54,55,58,63,67-69,71-75,80-82,85,90-92].

Assessing the current situation: Interest in- and exposure- to research among medical students.

Forest Plot symbols: * The axis, not the data, is shown in logit scale for aesthetic purposes. Table symbols: * Mandatory exposure (in the form of curricular components or graduation theses) was excluded from this analysis. Abbreviations used: D, developing countries; H, higher commitment to a research career; I, intercalated Bachelor of Science degree (iBSc). Dates are shown beside studies that may be confused with others referenced in this review having the same similar first-author names.

Interest in research among medical students

While the only reliable method for probing interest in medical research is assessing actual voluntary research involvement, survey data (self-reported interest) may provide insights into any discrepancies between interest and actual involvement. To avoid pooling survey data that are too heterogeneous, we made a distinction between survey questions that ask about general interest in research and those specifically asking medical students about their interest in making commitments to research during their future careers.

I1a: Interest in performing research: A pooled weighed estimate of 72% of medical students reported having interest in performing research (0.72, 0.57–0.83). One particularly high estimate was that reported by De Olivera and colleagues, which showed that 90% of its 1004 student sample had interest in performing research [74]. However, even when this study was excluded from the analysis as a possible exception, the pooled weighed estimate remained fairly high (0.67, 0.53–0.79) ().

I1b: Interest in a career involving research: The single best estimator of career intentions of US medical graduates is probably the Graduation Questionnaire (GQ), developed by The Association of American Medical Colleges (AAMC) in 1978 [7]. In 2013, 63% of the 13,180 respondents indicated intentions to become somewhat-to-exclusively involved in research during their medical careers, including 17% who planned "significant" or "exclusive" future involvement. This huge sample size approaches a true census, with 81.8% of the US fresh medical graduate population being covered.

Upon quantitative pooling of our included studies, we found that about 31% of medical students (0.31, 0.19–0.46) were interested in a career involving research, and 12% (0.12, 0.07–0.21) showed interest in "significant" (higher) commitment to research during their future careers. One particularly important, high-quality study was that of McManus and colleagues, showing that 6.9% of UK medical students planned to pursue academic careers (or found them to be very appealing) [85]. When we calculated the pooled outcome excluding MacManus et al or the AAMC data, the pooled proportion was not markedly changed ().

It should be noted that there is considerable variation in the proportions reported in our included studies. This may reflect inherent (true) variability in students' research interests due to diversity of settings and study populations (as has been discussed in ). We also believe that there are other potential contributors to this variability, most notably the ambiguity of wording of survey questions. For example, many studies did not make a clear distinction between interest in an academic (university faculty) medicine career, and interest in a career involving some research outside of academia.

I2. Medical students’ exposure to research

Even today there is no consistent way in which undergraduate medical students are incorporated into research. For example, students may be engaged in research through summer research electives [9,45], mandatory curricular study modules [90], extracurricular research activities [93], or they might decide to intercalate for one or more years to obtain a BSc beside their medical degree. In Germany, it is mandatory for medical students to submit a thesis outlining the results of a research project in order to graduate with the title "Doctor" [30]. This requirement has also been reported in Peru, Finland, France and some U.S. universities such as Yale [24,27,76,94]. The AAMC 2013 Graduation Questionnaire shows that 68.2% of US medical graduates participated in a research project with a faculty member on a mandatory or volunteer basis and 41.7% co-authored a research paper [7].

If we exclude papers describing medical schools asking for mandatory graduation theses or research modules, we find that a little less than one third of medical students participated in research projects (0.31, 0.22–0.41). The proportion exposed to “prolonged” periods of research (>6 weeks) is even less (0.22, 0.16–0.28).

In the U.S., different medical schools have different research expectations, and the exposure of medical students to non-mandatory research seems to be largely dependent on medical school influence. Duke University, for example, incorporates students into summer-long research projects [95]. On the other hand, Stanford University, the University of Pittsburg and Warren Alpert Medical Schools incorporate students into longitudinal research projects in parallel with their academic studies [95-97]. This longitudinal approach may help in solving some of the reported problems of time-out research, such as the reluctance of medical students towards detachment from their colleagues and financial worries about spending extra time in college. Indeed, the success of Stanford is particularly evident, with 90% of medical students participating in research projects [91].

We found that the pooled proportion of medical students reporting some interest in research is higher than that of students who were actually involved in research projects. This may be due to: a) self-reported interest may not necessarily reflect serious willingness to pursue research; or b) lack of opportunities to meet students’ interest due to lack of funding, supervision and encouragement or inflexible curricula that leave little or no time for research () [45,47-50,52,55,57,68,74].

II. Factors related to- or affecting medical student research

We identified four main factors affecting medical student research: previous research experience, academic success, having a higher degree (MSc or PhD) at the time of matriculation and financial factors. The effects of the first three factors were reported using odds ratios due to the presence of untreated groups () [32,47,52-55,58,62,63,67,79,81,92,98], while the fourth factor (financial influence) was pooled using proportions from survey studies () [55,57,59,67,82]. Moreover, we discuss the results of various studies reporting other relevant factors that could not be meta-analyzed, including the role of mentorship and competitive residencies in shaping medical students’ perceptions about- and attitudes towards research.

Factors related to- or affecting medical student research (i)–Effects of previous research experience, academic success and higher degree graduate-entry into medical school.

Forest Plot symbols: * The axis, not the data, is shown in log scale for aesthetic purposes. Abbreviations used: D, developing countries; I, intercalated Bachelor of Science degree (iBSc); M, motivation to perform research; K, research knowledge or skills; C, confidence in research competencies; In, interest in research. For some studies, odds ratios and 95% confidence interval values were reported, but not the raw numbers.

II1. Effect of previous research experience

Students who participated in research projects during medical school were over three times as likely to report interest in research involvement during their future careers (OR = 3.55, 1.84–6.83). Two studies [92,98], which were not included in the pooled weighed estimate, reported paired outcomes, with non-significant differences in research career interests after research exposure. Additionally, we found that medical school research involvement has no significant correlation with attitudes or motivation towards research (OR = 2.05, 0.99–4.24).

It is difficult to conclude that self-reported interest is a direct effect of exposure to research, since reverse causality cannot be excluded. That is, it is logical to assume that a fairly large proportion of students who had pre-existing interest in a career in research decide to participate in research projects. As a matter of fact, students in two of the included studies agreed that research participation strengthened pre-existing interest in a research career [90,91]. These findings also make sense in light of the fact that over half of all medical students reported having some interest in a career involving research (). Another possible explanation for the above results is that students who have had prior research experience have better research knowledge and skills, and are therefore more confident about their ability to succeed were they to undertake research projects during their future careers. Indeed, in a series of interviews conducted by Jones et al, students who undertook an intercalated BSc in primary healthcare reported a positive influence of the experience on their appreciation of the research process [99]. Similarly, a thematic analysis of 905 SSC (Student Selected Component) projects by Murdoch-Eaton et al provided by medical students at six UK medical schools revealed gain of various research-related skills [90]. These results are also supported by eleven quantitative studies, summarized in [11,37,39,40,46,47,55,64,82,89,91].

II2. Effect of having a higher degree (MSc or PhD) prior to medical school

II2a: Having a higher degree is associated with involvement in- (or planned involvement in-) research: Siemens et al report that medical students who had a higher degree prior to enrolment in medical school were almost four times more likely to perform research during medical school (OR = 3.95, 2.22–7.01) [52]. However, data provided by Cruser et al showed no significant difference between the two groups regarding their planned involvement in future research (OR = 1.01, 0.57–1.79) and Gerrard et al actually reported the reverse trend, with higher degree graduate-entry medical students actually being less likely to pursue an iBSc [54,81]. This is consistent with data we obtained from Mahesan et al, which shows that graduate-entry medical students (having any degree prior to matriculation) were almost ten times less likely to pursue an intercalated degree (OR = 0.01, 0.00–0.13) [62].

Since career progress (especially the pursuit of competitive residency) is a major motive behind medical student research, it may be argued that medical students with a higher degree view this aspect of their Curriculum Vitae (CV) as being “complete enough” and hence devalue the pursuit of another degree. In fact, to the medical student with a prior degree, an iBSc will almost always result in degree duplication, even if the skills and knowledge base of the iBSc course were completely different from those of the other degree already gained by the student.

II2b: Other advantages of having a higher degree (MSc. or PhD.): There is no significant correlation between having a higher degree prior to medical school enrolment and research interest or motivation. However, as might be expected, higher degree graduate-entry medical students were more knowledgeable about research, showed better research skills and had higher confidence in their research competencies (). This is expected, given that almost all higher degrees have a compulsory research component.

II3. Effect of academic success

II3a: Academic success is associated with attitudes towards basic medical sciences or medical research: The data we obtained from Hren et al shows an association between higher Grade Point Average (GPA) and attitudes towards research (OR = 1.83, 1.42–2.36) [79]. Cruser et al’s data, on the other hand, shows no significant difference between highest MCAT (Medical College Admission Test) scores and attitude scores [54]. Perhaps GPA during medical school, but not before admission, is a factor that influences attitudes. However, we believe the evidence in favor or against this hypothesis is weak and further investigation is needed in the future.

II3b: Academic success is associated with involvement in- (or planned involvement in-) research: The weighed pooled odds ratio from four included studies shows no association between academic success and involvement (or planned involvement) in research projects (OR = 1.00, 0.62–1.64). The only study showing a significant correlation was Brancati et al, which asserts that students who were academically successful (top third of their class) were more likely to choose an academic career (OR = 2.11, 1.30–3.42) compared to their less successful peers (lower third) [32]. However, this study investigates choice of an academic career rather than involvement (or planned involvement) in research during or right after medical school. Hence, it may be argued that this study should be excluded from the analysis as it measures a different outcome, in which case the pooled odds ratio remains non-significant (0.82, 0.59–1.15). We suggest further investigation into this issue using studies with more favorable, preferably prospective, designs.

II4. Financial factors affect the appeal of research to medical students

About half of medical students who chose not to get involved in research reported being deterred by financial factors (0.50, 0.46–0.54) () [55,57,59,67,82]. Nicholson et al and Stubbs et al both show that about half of medical students who choose not to intercalate do so for financial reasons [59,82]. In addition, Galletly et al also reported that about half (48%) of medical students asserted that perceived lower salaries of academicians was an important factor behind their decision not to pursue an academic career [55]. The consistency of the findings by the former two studies with the latter one suggests that it's not just the short-term financial burden of pursuing an intercalated degree that deters medical students from getting involved in research, but a general long-term financial concern. Financial worries, particularly the fear of running out of grant money and the financial stress of academic careers, were indeed cited by students interviewed by O'Sullivan et al among the deterrents to academic career pursuit [100].

Similarly, Yamazaki et al and Kumar et al both showed that a considerable fraction of the general medical student population displayed concerns about the financial stability of a research career (45% and 12%, respectively) [57,67].

II5. Career progression is a main motive behind performing research during medical school

The result from seven included studies indicate that career progression is a main motive (if not the main motive) behind performing research during medical school. These results indicate that in a large fraction of cases, medical students perform research for purely pragmatic reasons (related to their residencies or further post-graduate education), rather than pursuing research for the value it has in and of itself ( [48,49,52,54,55,82,86].

Four studies mentioned the role competitive residencies play in driving medical students to perform research, and in fact students in three of those studies believed that seeking competitive residency was–explicitly- the main motive to perform research during medical school. The results from a qualitative study by Shapiro et al support this conclusion by showing that the motives behind research participation include (but are not limited to) pragmatic targets such as improving the students' relationship with faculty [101].

These conclusions are consistent with other results reported here showing that: a) there is a discrepancy between interest in clinical practice and interest in a research career ( [45,51,56,57] and b) there is a correlation between interest in academia or basic medical sciences and interest in research ( [55-57].

Combined, these findings indicate that any policies aimed at boosting medical students’ engagement in research have to align research involvement with the career progress and success of students. In much the same way that peer-reviewed publications are a key competitive edge in academia and in competitive residency applications, it must become clear that research is more than just an accessory when it comes to ordinary clinical practice.

II6. Other factors related- to or affecting medical student research

As Reynolds has discussed, it is simply not enough to match students with professors in research projects, as good quality research requires real mentorship [102]. Research instructors also act as role models to encourage students to pursue careers in academic medicine. Further, finding the right mentor is important to ensure that students provide a working and intellectual input into the research projects, rather than simple assistantship in lab work or data collection ( [48,52,57,58,82].

This is not always going to be easy; the results from two qualitative studies show that the complexity of ethical approval procedures (whether in terms of time or paperwork) is a major difficulty facing supervisors and students alike [90,103]. Further, the absence of clear, well-structured research governance may result in some aversion to faculty-mentored student research. This was the case in two qualitative studies, where students cited problems with approachability of faculty members and expressed concerns about being used as "free labor" on research projects [90,101].

In fact, Murdoch-Eaton et al's aforementioned project content analysis, while revealing some gain in useful research skills, also highlighted the failed attainment of a balanced skill-set; the majority of student projects involved information gathering and data processing, while fewer projects involved actual student engagement in research methodology development or critical analysis of data [90].

It may be presumed that the relatively short duration of the undergraduate research experience could limit its publication or citation potential. Indeed, Dyrbye et al found that graduates with a 17–18 week-long research experience published significantly less papers in which they appeared as first authors than their peers who spent 21-weeks doing research [29]. Further, Fede et al showed that the annual Undergraduate Medical Congress of ABC foundation (COMUABC) had a smaller proportion of abstracts accepted for publication in peer-reviewed journals in comparison to conferences of practicing physicians [70]. Conversely, Van Eyk et al. reported that the average number of citations of Dutch medical student publications was actually higher than the average citations for papers in the same field. [41]

A number of studies investigated factors that prevent medical students from being involved in research. Poor mentorship, lack of role models and perceived lower salaries of academic physicians were among the key factors cited (. The previous findings were also supported by four qualitative studies ( [17,45,90,99-101,103,104].

In addition, institutional influence as well as the type and length of available research opportunities were found to be relevant factors in determining whether students choose to engage in research [51,53]. McLean and co-authors provided an excellent set of tips to bolster the involvement of students in academic medicine projects and potentially overcome some the aforementioned limitations [105].

. The importance of psycho-cognitive factors in determining medical students' motivation towards- and engagement in- research was also highlighted in the qualitative literature. One of the most important motives behind performing research is curiosity. Not only is curiosity a main motive behind pursuing research while in medical school (as has been shown by Shapiro et al [101]), it is one of the very early psycho-cognitive predictors of persistence into scientific or research disciplines even before enrolment into medical school [17,104]. Conversely, perceived lack of competence may deter medical students from pursuing research-active careers [45].

III) Assessing the impact and effect of medical student research

We assessed three main outcomes that reflect the short- and long- term impact of medical student research: 1) the proportion of research performed during medical school that culminates in a peer-reviewed journal publication, 2) the effect of medical school research on the career choice and future research involvement of medical students, and 3) the effect of medical student research on long- term success in academia. The first outcome has been summarized in [10,24,25,27,29-31,37,38,41,49,64,75,76,93,106] and the latter two are shown in [8,25,26,31,43,44,66,68,81,83,85,90].

The proportion of medical student research resulting in a peer-reviewed journal publication.

Since the duration of research exposure will almost always affect the publication outcome, it has been shown too. Forest Plot symbols: * The axis, not the data, is shown in logit scale for aesthetic purposes. Table symbols: * The duration is probably prolonged (possibly months long); ** 20–40 European medical school credits; || For published projects, the average duration was 18 months. D, developing countries; I, intercalated Bachelor of Science degree (iBSc); HQ, relatively high quality publication (indexed in Medline, Scopus or Medic), HF, first-author publication in a relatively high quality journal. Dates are shown beside studies that may be confused with others referenced in this review having the same similar first-author names.

The impact of medical student research–Impact of medical student research on career choice and long-term academic success.

Forest Plot symbols: * The axis, not the data, is shown in log scale for aesthetic purposes. Table symbols: * at least one first-author publication; ** at least one citation; || more than 20 citations. For some studies, odds ratios and 95% confidence interval values were reported, but not the raw numbers.

III1. Medical student research results in a publishable product

Peer-reviewed journal publications are generally considered to be the best indicator of research productivity, and it may be viewed as a major metric (though not the only one) of the “return on investment” in supporting and funding medical student research. An average of 30% (0.30, 0.19–0.44) of research performed by medical students resulted in a peer-reviewed journal publication. When only higher quality publications were included in the analysis (indexed in Medline, Scopus or Medic), the proportion remained more or less the same (0.31, 0.18–0.47). Subgroup analysis of studies investigating the research productivity of graduation theses revealed that 26% (0.26, 0.10–0.52) of graduation theses result in higher quality publications.

As expected, all studies reporting first-authored peer-reviewed publication by medical students described instances of prolonged research exposure. An average of 13% (0.13, 0.06–0.27) of medical student research resulted in a first-authored peer-reviewed publication. The pooled outcome remained the same when only higher quality publications (Medline-, Scopus- or Medic- indexed) were included in the analysis (0.13, 0.05–0.30).

A few initiatives, aimed at propping up medical student publication output, have gained popularity over the last few years. Those initiatives include a number of student-run journals and journal spaces dedicated solely for medical student research publications [107-110]. A subset of these journals is Medline-indexed and some even involve undergraduates in the peer-review process. Similarly, the Yale Journal of Biology and Medicine annually publishes Yale's student thesis abstracts [111]. These initiatives, we suppose, will help in promoting student participation in research and comfort students about publication issues. To our knowledge, there is no systematic investigation in the literature so far regarding the quality of research published in medical student research journals in comparison to field-specific journals. Hence, we would like to take a conservative stance whenever we see such hierarchical "segmentation" of the scientific enterprise; the stringency of research assessment, in our opinion, should be indiscriminant to the identity of the study authors.

It is important to note that the failure of publication of medical student research may be reflective of other factors beside the success and relative contribution of the student. For example, Weber et al showed that 55% of the papers submitted to a medical specialty conference did not reach the stage of publication five years later [112]. Similarly, Riveros et al found that half of the clinical trials reporting results in ClinicalTrials.gov had no corresponding journal publication [113]. Keeping this in mind, the results by Cursiefen et al should not be surprising; showing that medical students were among the authors of 28% of the papers produced by a German medical faculty, even though only 66% of medical student research resulted in a publication [30].

III2. Research during medical school is associated with later involvement in research projects

Students who took part in research projects during medical school were more likely to get involved in (or report planned involvement in-) research later in their careers (OR = 3.58, 1.82–7.04). When a subgroup analysis was performed to include only studies that explicitly refer to academic careers (as opposed to brief research encounters), students who performed research during medical school were over six times as likely to pursue academic careers (OR = 6.42, 1.37–29.98) than their “untreated” peers.

With one exception, none of the included studies had a prospective design; hence reverse causality cannot be excluded, and is in fact very likely (students planning academic medicine careers choosing to get involved in research during medical school). Indeed, the only prospective study included (McManus et al [85]) showed that at the time of application to medical school, students who later chose to take an intercalated degree were already significantly more likely to report definite or highly likely choice of academic medicine careers (OR = 1.37, 1.13–1.66). Just before graduation, however, this likelihood had a substantial increase (OR = 3.45, 2.27–5.24). Together, these results indicate that medical school research strengthens pre-existing interest in an academic career.

A qualitative study by O'sullivan et al emphasized the value of early research exposure in giving medical students the opportunity to entertain the thought of pursuing academic careers [114]. Such exposure, they concluded, may sometimes even discourage students from pursuing academia, but is necessary nonetheless given the lack of sufficient free time during post-graduation residency to experience research.

III3. Research during medical school is associated with long-term success in academia

Three studies showed that physicians who performed research during medical school were more likely to attain faculty rank long after graduation [8,32,66]. While this has implications on the decision of individual medical students to pursue research, we argue that it has little bearing on policy decision-making, since faculty positions are awarded on a competitive basis. Indeed, Brancati et al showed that this effect was dependent on the publication status of research performed during medical school [32]. In other words, students who did not publish their research were not significantly more likely to attain higher faculty rank on the long run. Hence, the fact that medical student research is associated with higher likelihood of attaining faculty positions has little implications regarding the systematic incorporation of research into medical curricula.

Students who performed research during medical school were more than twice as likely to author at least one peer-reviewed publication later in their career (OR = 2.31, 1.88–2.83). This remained true after the exclusion of Chusid et al [25] (which correlates successful publication of graduation theses with long-term publication success) from the analysis (OR = 2.26, 1.83–2.77). They were also twice as likely to acquire first-authorship (OR = 2.21, 1.56–3.13). The total number of publications and ability to secure grants, too, was reported to be significantly higher among students with medical school research experience [81]. Evered et al, on the other hand, found no significant difference in either of those measures between both groups [66]. Moreover, students who performed research during medical school were more likely to be cited at least once [66], had a higher total citation count [81], were more likely to be cited more than 20 times [66], and had higher odds of receiving awards [8,81] later in their careers.

While this data provides strong evidence of a correlation between medical school research and long-term success in academia, a causal relationship cannot be established since students who decide to perform research may already have a keen interest in research. Nonetheless, a causal relationship is quite likely since early research experience (especially if it culminates in a first-authored publication) would naturally enhance the career prospects and significantly improve the CV’s of early career medical graduates. Overall, we believe that the long-term impact of medical school research is inadequately assessed, and that further evidence is needed using prospective study designs with proper adjustment for baseline status.

III4. Research during medical school is correlated with career choice of- (or interest in a career in-) the same or related specialty as the research project

Three of the studies that met the broad inclusion criteria reported results from control or “untreated” groups. Other studies reported results only from treated groups and hence were excluded from the analysis. Overall, students are 2.7 times as likely to be interested in careers in the same (or related) clinical specialty as the research project they got involved in during medical school. As with many other conclusions in this review, a causal relationship cannot be determined from this apparent correlation. This is especially true in the case of competitive residencies (and is particularly relevant to US residencies), where research experience in the same specialty gives recent graduates a competitive edge over their peers without such experience.

The relationship between medical school research and clinical practice was also touched upon in two of the included qualitative studies. Shapiro et al showed that many faculty members mentored student research in family practice in order to attract students to the same specialty [101]. Indeed, students interviewed by Jones et al believed an iBSc in primary healthcare provided them with deeper insights into patient care and a more thorough understanding of evidence-based clinical practice [99].

IV) Miscellaneous topics related to medical student research

In the following section of this review we discuss a number of miscellaneous topics relevant to medical student research. Three of these topics were discussed in light of quantitative data, and are summarized in [28,29,47-49,53,54,58,59,62,63,67,71,79,81,83,88,89,92] and [24,27,33,37,38,50,64,70]. Though they did not pass our inclusion criteria, four of the citations screened were personal perspectives provided by medical students, and are worth mentioning for enriching the discussion. They discussed the importance of the research experience on their medical career [115,116], the importance of medical students' research in increasing national research output [117] and the relevance of lab research involving animals to appreciation of human anatomy and physiology [118].

Miscellaneous topics related to medical student research.

Forest Plot symbols: * The axis, not the data, is shown in log scale for aesthetic purposes. Abbreviations used: D, developing countries; I, intercalated Bachelor of Science degree (iBSc); FC, studies measuring final year academic performance and controlling for baseline performance. Dates are shown beside studies that may be confused with others referenced in this review having the same similar first-author names. For some studies, odds ratios and 95% confidence interval values were reported, but not the raw numbers.

IV1. Effect of prolonged research time-off on subsequent academic performance

One of the issues discussed in the literature is the effect of prolonged research time-off amid the medical program on subsequent clinical knowledge. This question has been assessed in the context of iBSc degrees in a recent review [119]. All but one of our included studies investigated the effect of taking an intercalated degree on subsequent academic performance. The results have been conflicting; two studies that either matched groups by previous performance or adjusted for pre-clinical scores found no evidence of improvement in scores [88,120]. All five other studies that met our inclusion criteria reported an improvement in academic performance.

Due to heterogeneity in academic assessment methods and high possibility of confounding, we only pooled the studies for which we could extract odds ratio values that: a) measure final year academic scores and b) control for previous academic performance. Three studies met these two inclusion criteria, all of which reported the effect of iBSc degrees. On average, students who took some time off to perform research were twice as likely to outperform their peers (OR = 1.99, 1.39–2.84), even after adjustment for previous academic performance. It is noteworthy that all pooled studies investigated research time-off that was around one year in duration (iBSc), and that the positive effect of research time-off on subsequent academic performance may actually be reversed if the research delays are prolonged. Dyrbye et al pinned down a critical period of three years, after which medical students start to lose clinical knowledge and skills by the time they return to the core medical program [28].

IV2. Gender equality in medical student research

There is no apparent gender difference regarding the following outcomes: Interest in a career in research; involvement in research during medical school; attitudes towards research; interest in- or motivation towards- performing research; research knowledge or skills. However, on average, males seem to be significantly more likely to publish (or submit for publication) the research they performed during medical school (OR = 1.59, 1.26–2.01). The reasons behind this gender gap in publication are unclear to us, and have been inadequately researched. Since there is no apparent gender difference in research perceptions, attitudes, motivations or knowledge, we suspect that the gender difference in publications is due to factors unrelated to research such as the overall academic environment or psychosocial factors. Indeed, these findings are consistent with a 2006 study by Jagsi et al showing a generalized gender gap in the authorship of academic medical articles in six major medical journals. Whatever the reasons behind gender differences in publication, they underlie a general issue not specific to medical school research [121].

IV3. Type and field of research performed by medical students

The majority of medical student research is original in nature (as opposed to literature reviews). We were interested in finding out what percentage of these research projects were in the basic sciences, since this issue is of particular relevance to translational research. We found that the proportion was highly variable between different studies. In four of the five included studies less than half of medical student research was lab-based basic research, and the pooled weighted estimate was 0.32, 0.14–0.49. Given the relevance of research to competitive residency applications, it should not come as a surprise that lab-based projects do not constitute the majority of medical student research. Nonetheless, these results indicate that efforts directed at increasing the number of physician scientists involved in translational research should not only be directed at bolstering research involvement, but also improving the appeal of basic lab-based research to medical students.

IV4. Compulsory vs. elective medical school research

The question of whether undergraduate medical research should be made compulsory or elective has been discussed in the literature, and is a matter of debate [37,97,122]. Arguments in favor of mandatory incorporation revolve around the ever-increasing importance of evidence-based clinical practice, while arguments against it revolve around the importance of focusing on clinical skills education. Diez et al. recommended against Germany's dissertation requirement, due to the steady decline in the number of successful dissertations [123]. Our results tell a similar story; the fraction of graduation theses resulting in a first-authored higher quality publication was smaller than the overall average (0.07, 0.03–0.14). At first, this may seem counterintuitive, as one may predict that the systematic incorporation of research as a necessary graduation requirement would raise the fraction culminating in a first-authored higher quality publication. However, one needs to bear in mind that since graduation theses are an obligatory requirement, a fraction of those students performing research may not be interested at all in what they are doing. Taking this into consideration, it should not come as a surprise that percentages as high as 34% (Cohen et al [38]) and 31% (Dyrbye et al [29]) of voluntary medical student research were reported to result in first-authored Medline-indexed publications. Weihrauch et al and Pabst et al, on the other hand, reported favorable results in terms of the personal and professional value of the German dissertation requirement [124,125].

IV5. The situation in countries with developing economies

We retrieved studies that were performed in India [67,72], Uganda [68], China [69], Brazil [70,74], UAE [71], Croatia [73,79], Pakistan [75,77,80,86], Peru [76], and Turkey [78].

The number of medical schools and the research budget in developing countries are alarmingly mismatched with their needs [1]. This disparity, we believe, reflects naturally on the status of medical student research. In fact, medical student research might be even more important in developing countries than in developed countries, due to the pressing need to adapt international standards to local community needs.

Medical students in developing countries arguably face a set of extra challenges and are influenced by a number of different factors in comparison to developed countries [126]. For example, the high student-to-teacher ratio makes it increasingly difficult for medical students to have mentors and role models. Even research based on statistical analysis of patient records is often difficult to perform in many medical schools, due to suboptimal Information and Communications Technology (ICT) infrastructure in hospitals and in teaching premises in countries with developing economies [127]. While excellent research may of course be performed in resource-poor countries, it is preferable that any reform in research funding is coupled with a well-developed educational and managerial infrastructure; otherwise the research output may be largely suboptimal [128]. Worryingly, an essay by Silva et al. reported a decrease in the ratio of Undergraduate Student Research Assistant Programs (USRA's) to the number of undergraduates in Brazil over the past years [129].

Students’ interest in research was higher in countries with developing economies than in developed countries (0.82, 0.67–0.91 vs. 0.47, 0.26–0.69). One possible explanation for this finding is that the lack of opportunities causes higher eagerness to perform research. Another, possibly more likely, explanation is higher career-related anxiety in lower-income settings, with a resultant boost in research interest. Indeed, students in developing countries were not significantly less exposed to research, a result which may be reflective of the higher interest rates, bolstering research engagement despite inadequacies in resources. These results are supported by the findings of Baig et al, showing that 40% of Pakistani medical students viewed research as a tool to secure competitive residencies in the US [86].

Conclusions and Future Directions

Overall, our review shows that there’s considerable variability in medical student research exposure, engagement and productivity among different medical schools. A large proportion of the medical student population is interested in research, but is deterred by practical difficulties, including the lack of opportunities and funding. The benefits of research exposure on the short- and long-term scientific productivity is well documented in the literature, and a clear correlation is identified between medical school research engagement and later engagement in research projects (including the choice of an academic career). However, the number of well-controlled, high-quality prospective studies on the topic is limited and it is difficult to exclude reverse-causality. Existing evidence suggests that medical school research does have a positive effect on the choice of an academic career, but it does so through strengthening pre-existing interest. Financial worries, gender, having a higher degree (MSc or PhD) before matriculation and perceived competitiveness of the residency of choice are among the factors that affect the engagement of medical students in research and their scientific productivity.

Another potential limitation of this review is publication bias. It is conceivable that medical schools where students had a positive experience with research rush to publish their results, whereas others with experiences that were not so positive blamed it on the design of the program without publishing their results. It is also clear that there are plenty of successful undergraduate research programs that do not publish their results.

We suggest that more studies are done to assess the different structural and managerial aspects of standardized undergraduate medical research, as well as the differences between compulsory research components, elective research components, intercalated BSc's and extracurricular research in terms of academic, professional and psycho-cognitive effects. Further, we recommend more investigation into the quality and citation potential of published medical student research in comparison to that of established researchers and physicians.

PRISMA guidelines checklist.

(PDF)

Click here for additional data file.

Supplementary methodology file.

(PDF)

Click here for additional data file.

Quality assessment and quantitative data extraction sheet.

Abbreviations used: D, developing countries; I, intercalated Bachelor of Science degree (iBSc); X, Cross-sectional; R, Retrospective; I, Interventional; Pro, Prospective; Q, questionnaire; DS, database search; IN, interview.

(XLSX)

Click here for additional data file.

Sensitivity plots for the pooled effect size values calculated.

(PDF)

Click here for additional data file.

Supplementary tables accompanying the main text.

(PDF)

Click here for additional data file.

Table 1

Career progression (eg securing competitive residency) is a main motive for medical students to perform research.

Study	Type	Design	Control group	Institutions	Lim.	Overall quality	N	Population	Outcome measure	Outcome
Siemens	R/X	Q	Y	M	Rp	Medium	327	Second- and Fourth- year medical students; Three Canadian Medical Schools.	Seeking competitive residency correlated with research activity	P<0.001
			N	M					Got involved in research to facilitate admission into residency	140 (42.8%)
Cruser	R/X	Q	Y	S		Medium	354	Incoming, first, and second year classes of osteopathic medical students at University of North Texas Health Science Center; Research competencies.	Agree that "to be accepted into competitive residency, I have to have some knowledge or experience in research"	226 (63.4%)
Griffin *†	X	Q	N	M		Medium	72	Students; seven medical schools in UK.	Were mainly motivated to publish for career progression.	37 (51.4%)
Remes **	X	Q	N	S	Rp.	Medium	91	Students; University of Helsinki	Securing residency main reason behind performing research	79 (87%)
Baig ‡	X	Q	N	M		Medium	398	Medical students who showed interest in medical research; Four Medical Schools, Karachi, Pakistan	Seeking residency in the US main motive to perform research	159 (39.9%)
									"Desire for a strong CV" main motive to perform research	275 (69.1%)
Stubbs **†	R	Q		M	Rp.	Medium	253	Students who chose to intercalate; Bristol and Sheffield Medical Schools, UK	Did an intercalated degree "to get the job they want"	176 (69.6%)
Galletly	X	Q	Y	S		Medium	98	Final year students; University of Adelaide.	Students seeking higher degree (MSc or PhD) more likely to perform research during medical school §	OR = 3.17 (1.09–9.18)
							100		Students seeking higher degree (MSc or PhD) more interested in research §	Or = 5.68 (1.2–27.0)

* Baseline population is students who submitted an article for publication

** Baseline population is students ho performed research

† Studies assessing the effect of an Intercalated Bachelor of Science (iBSc)

‡ Study performed in a developing country (Pakistan)

§ Adjusted for age, sex and interest in a career involving research (data obtained directly from authors and dichotomized).

Abbreviations used: X, Cross-sectional; R, Retrospective; Q, Questionnaire; Y, Yes; N, No; S, Single; M, Multiple; Lim, Other limitations; Rp, low response rate (<60%).

Table 2

Positive effect of mentorship or the presence of an academic role model.

Study	Type	Design	Control group	Inst.	Lim.	Overall quality	N	Population	Outcome measure	Outcome
Siemens	X/R	Q	Y	M	Rp	Medium	327	Second- and Fourth- year medical students; Three Canadian Medical Schools.	Mentorship bolsters interest in research	P = 0.05
									Role models drive interest in academia	P = 0.015
Stubbs *	R	Q	Y	M	Rp	Medium	1484	Students who chose to intercalate; Bristol and Sheffield Medical Schools, UK	Students with clinical academic supervisors gained significantly more publications	P<0.0001
									Students with clinical academic supervisors gained significantly more posters	P = 0.0002
									Students with clinical academic supervisors gained significantly more first-class honors	P = 0.055
Greenberg	R	Q	Y	M	Rp	Medium	228	Third and Fourth year students; Three US Medical Schools	Mentorship bolsters interest in a research-oriented career **	OR = 2.5 (1.39–4.51); P = 0.002
Yamazaki	X	Q	Y	S	Rp	Medium	267	Students; Juntendo University School of Medicine, Japan	Interest in basic sciences correlated with faculty's efforts to promote interest in research.	OR = 2.86(1.62–5.06); P = 0.0003
Griffin *	X	Q	N	M		Low	72	Students; seven medical schools in UK.	Were encouraged to submit paper by supervisor	7 (9.7%)

* Studies describing the effect of an intercalated Bachelor of Science (iBSc)

** Data obtained directly from authors and dichotomized.

Abbreviations used: X, Cross-sectional; R, Retrospective; Q, Questionnaire; Y, Yes; N, No; Inst., Number of institutions; S, Single; M, Multiple; Lim, Other limitations; Rp, low response rate (<60%).

Table 3

Studies with a qualita3tive component.

Study	Type	Design	Study population and setting	Respondents (Response rate)	Outcome measures	Outcomes
O'sullivan	X	IN	Students, residents and faculty members; University of California, San Francisco.	40 (11 medical students)	Factors related to pursuit of academic medicine career.	Early exposure to research, finding the right mentors and role models are among the most important factors. Sociocognitive factors such as financial worries play a role.
McGee	X	IN; GTM	Participants; SURF program at Mayo Clinic College of Medicine + Participants; IMSD program at Mayo Clinic; 1997–2000.	109 (20%)	Themes related to pursuing a PhD, MD/PhD or MD with a research intention.	Five major themes are relevant: Curiosity, Problem solving, independence, serving the world indirectly and a flexible perspective of one's own future. MD-bound students talked about a desire to help others directly through patient care, while MD/PhD or PhD-pursuing students wanted to help others indirectly through research.
Shapiro	Ret	GTM	Participant students and mentors; Summer research assistantship program in family medicine.	11 students 10 faculty mentors	Motives behind participation	Most students were driven by curiosity, the will to learn about research and to improve their relations with faculty. Faculty mentors wanted to be more involved with the students and to attract more of them into research.
Murdoch-Eaton	X	Mixed: SG; FG; PA.	Students; Hull, York, Leeds, Liverpool, Newcastle and Sheffield Medical Schools.	Focus groups: 5 groups (one per school) Study groups: 15 students (three per school). Projects: 905	Perceptions about undergraduate research and thematic analysis of Student Selected Component (SSC) projects.	Focus groups: students understood the benefits of research learning and skills, but mentioned practical difficulties. Study groups: confusion between research and clinical practice among students. Project analysis: Various skills research-related skills were gained by students.
Sanchez	X	FG	Students; Three national student conferences in the US: Interest in an academic medicine career.	73	Interest in academic medicine careers.	Lack of knowledge or competency cited as obstacles in pursuing academic medicine career. Mentorship and career development resources cited as potential improvement strategies. Higher involvement of ethnic minorities prompted.
Pacifi	X	Mixed: Q; IN; CS	Undergraduate Upper-level science major premed and non-premed students; Southeastern United States.	135 (7.9%): 55 Premed 80 non-premed. 11 interviews	Influences and experiences regarding undergraduate scientific research	Premeds: talked more about serving humanity and the empathic aspects of research, viewed it as a tool to augment their professional outlook but not as a career. Non-premeds: talked more about the joy of discovery and were enthusiastic about a career in research. Both groups: had similar expectations from research.
Jones	X	IN	Students who did an intercalated BSc in primary Healthcare.	24 (92%)	Perceptions and outcomes.	Greater awareness about research and critical appraisal. More informed career decisions. Deeper insights into the psycho-cognitive aspects of illness.
Robinson	X	IN	Academic supervisors and administrative staff; three UK medical schools.	12	Impact of research governance on research education.	Ethical approval bureaucracy cited as a main limiting factor. Supervisors tend to avoid them by modifying existing projects or abandoning supervision altogether.

Abbreviations: X, cross-sectional; Ret, retrospective; Q, questionnaire; IN, interview; GTM, grounded theory methodology; SG, student groups; FG, focus groups; PA, project analysis; CS, case study.