Short-term outcomes of a program developed to inculcate research essentials in undergraduate medical students.

BACKGROUND: Participation in research during undergraduate studies may increase students' interest in research and inculcate research essentials in them. AIMS: The purpose of this study was to evaluate the effectiveness of the mentored student project (MSP) program. SETTINGS AND
DESIGN: In the MSP program, students in groups (n = 3 to 5) undertook a research project, wrote a scholarly report, and presented the work as a poster presentation with the help of a faculty mentor. To begin with, the logic model of the program was developed to identify short-term outcomes of the program on students, mentors, and the institution. A quasi-experimental design was used to measure the outcomes.
MATERIALS AND METHODS: A mixed method evaluation was done using a newly-developed questionnaire to assess the impact of the MSP on students' attitude, a multiple-choice question (MCQs) test to find out the impact on students' knowledge and grading of students' project reports and posters along with a survey to check the impact on skills. Students' satisfaction regarding the program and mentors' perceptions were collected using questionnaires. Evidence for validity was collected for all the instruments used for the evaluation. STATISTICAL ANALYSIS: Non-parametric tests were used to analyze data. Based on the scores, project reports and posters were graded into A (>70% marks), B (60-69% marks), and C (<59% marks) categories. The number of MSPs that resulted in publications, conference presentation and departmental collaborations were taken as impact on the institution.
RESULTS: Students' response rate was 91.5%. The students' attitudes regarding research changed positively (P = 0.036) and score in the MCQ test improved (P < 0.001) after undertaking MSP. Majority of project reports and posters were of grade A category. The majority of the items related to skills gained and satisfaction had a median score of 4. The MSPs resulted in inter-departmental and inter-institutional collaborations, 14 publications and 15 conference presentations. An area for improvement noted was to have the MSP implemented in the curriculum without increasing students' overall workload and stress.
CONCLUSION: The study identified strengths and weaknesses of the MSP program. Our model of undergraduate research project may be incorporated in undergraduate medical programs to foster positive attitude and knowledge base about scientific research and to instil research skills among students.

Introduction

The evolution of the medical field requires that medical educators constantly and systematically look at better ways to prepare future physicians.[1] Despite the absence of definite data, it is believed that student participation in research during undergraduate studies during medical school leads to better clinicians or physician scientists who will be able to create and apply new knowledge related to biomedicine.[1] Studies have shown that research experience during medical school training increased students’ interest in research,[234] improved their confidence in conducting research,[4] influenced students’ choice of clinical specialty,[5678910] improved knowledge regarding research principles,[23611] fostered abilities to critically evaluate literature[71213] and skills to write scientific papers.[14] The knowledge and skills gained while undertaking research work may further increase students’ participation in research thereby increasing students’ publications and presentations.[151617]

Despite the above evidences on educational benefits of research projects in medical school, whether such programs should be required as a part of the medical school curriculum is still debatable.[1] Studies have shown that these activities caused stress on students.[2418] Their improper implementation in the curriculum may have several negative effects on students’ learning.[1] When educational interventions become an extra burden on students, it may adversely affect their performance in examinations by affecting their learning and distracting them from studying for examinations. Bierer and Chen (2010) in their systematic review on the impact of scholarly concentration on medical students, recommend the requirement of more rigorous evaluation designs to show the positive and negative effects of such programs.[19]

The aim of this study was to initiate program evaluation of an introduced curriculum change, mentored student project (MSP) at the institution. The primary objective was to measure the short-term outcomes of MSPs on students’ knowledge, attitude, and skills regarding research. This study also describes outcomes of MSP on faculty and the institution. During the study we also developed instruments to measure the same, as systematic instrument design and use of reliable instruments are vital if the results of education testing are to be considered valid.

Materials and Methods

Educational context and subjects

The MSP program was implemented in the curriculum in 2007 with the intention of developing research skills essential for the career development of medical students. In MSPs, medical students (n = 141) were required to undertake a research project involving in-depth study of an area of their interest, write a scholarly report, and present the work as a poster presentation in 6 month duration during the second year of the course. The MSP coordination committee with two faculty members as coordinators and Dean as the chairman functioned to monitor the smooth conduct of the MSP program. Faculty coordinators oriented students regarding the MSPs; research methodology; preparation of project report; and the poster preparation and presentation. The coordinators supervised student projects done under the guidance of faculty mentors and also ensured that the time line was followed.

Keeping in mind the cultural background and specific needs of the international students, our medical school took the initiative in developing a mentorship scheme, in which the mentors (faculty members who are basic scientists) were required to supervise the academic progress of those students who were assigned to them. Mentors must review and approve the project proposal and final report. Mentors guide the students during all phases of the project from research topic selection, ethical consideration, writing the proposal, data collection, analysis and interpretation, writing up of final report and designing poster for poster presentation. Feedback taken from faculty and students of the institution in 2010 indicated the need for evaluation of the MSP program. To begin with the logic model[20] of the MSP program was developed to identify short-term outcomes of the program [Figure 1].

Figure 1

Logic model of MSP program. ISO: International organization for standardization

The ethics committee of the institution approved present quasi-experimental study. The convenience sampling method was used. Written informed consent was obtained from the students and faculty before responding in instruments.

Assessing outcomes

All the instruments required for assessing the outcomes of the MSP were developed using standard protocols and involving four experts with the Foundation for Advancement of Medical Education and Research (FAIMER) fellowship degree. To develop questionnaires, experts checked content validity individually. The questionnaire was further refined using a Delphi process in a series of meetings with a panel having above experts. All closed-ended questions in questionnaires were in a 5-point Likert scale (1 = disagree and 5 = strongly agree).

Students’ knowledge

Fifteen multiple-choice questions (MCQs) were developed by coordinators based on standard literature on research methodology[2122] and standard processes to establish validity and reliability. To test students’ knowledge, the finalized 15 MCQ administered to students before (n = 153) and soon after (n = 140) completing MSP with poster presentation being the final requirement for the completion.

Students’ attitude

To test the impact of the MSPs on students’ attitude, a questionnaire was developed after a literature review.[19] To assess test retest reliability, students were asked to respond in the questionnaire twice with a week interval in between. Following this, items with low a Cronbach's α (CA) value were deleted. The final questionnaire with 13 items was used to assess students’ (n = 140) attitude by doing a retrospective pretest. Students’ responded in the questionnaire in a 5-point Likert scale.

Students’ skills

Quality of students’ project reports and posters along with students’ perception on skills gained gives evidence for the impact of MSP on research skills of students.

Students’ project report and poster evaluation instruments (with 35 and 20 criteria respectively) were developed based on Glassick's six standards for assessing scholarships[23] and review criteria for scholarly manuscript.[24] The questionnaire developed to assess students’ perceptions regarding skills gained had 15 items with a Likert scale. After the completion of the MSP, two evaluators independently evaluated students’ posters (n = 41) and project reports (n = 39). The percentage marks scored was used for grading the reports and posters as A (>70% marks), B (60-69% marks), and C (<59% marks) categories.

Impact on faculty

A questionnaire with 13 Likert-type statements and 4 open ended questions was developed. Faculty responded in this questionnaire after the completion of the MSP.

Impact on the institution

The number of research projects resulted in publications, conference presentations, and research collaborations were noted. A questionnaire with 17 Likert type statements was developed to measure students’ satisfaction regarding the program. Perceptions regarding the factors that facilitated and hindered the conduct of the MSPs and measures taken to overcome those obstacles were collected using open-ended questions.

Analysis

Data analysis of attitude test was done using SPSS version 16. Change in students’ total attitude score was done using Wilcoxon Signed Ranks test. The item wise responses were cross-tabulated and subjected to McNemar's test. The item response analysis for difficulty index and discrimination index of MCQs was done using Excel worksheets. Comparison of MCQs scores in pre- and post-test was done using the Independent Sample T test. Concurrent validity between the project report and poster evaluating criteria was assessed by Pearson's product-moment correlation coefficient (PPMCC). The students’ and faculty's perceptions collected using other questionnaires were expressed as the median and the interquartile range (IQR). The students’ and faculty responses to open-ended questions were tabulated in the decreasing order of frequency of their appearance. Later, two authors independently grouped the responses into three categories. Finally, the categories and responses were finalized by consensus among authors.

Results

The response rate was 91.5% (140/153) during post-test of knowledge and retrospective pre-test of attitude.

The CA value for test retest reliability of attitude questionnaire was 0.879. The average total median score in attitude scale was 37 (32,40), which was significantly increased to 38 (32,44) after undertaking MSP (P = 0.036). After undertaking MSPs, statistically significant and positive changes in attitude were observed regarding most of the items except items number 5 and 10, where the change was negative [Table 1].

Table 1

Impact of mentored student project on students’ attitude

Items	Mean with IQR and cumulative frequency of agree + strongly agree responses in %		P value
	Before MSP	After MSP
I am interested to participate in research	3 (2-3) and 21.3	3 (2,4) and 24.7	0.236
MSP stimulated my interest in research	3 (2,3) and 16.7	3(2,4) and 32.6	0.116
I feel that research experience enriches my undergraduate learning experiences	3 (2,4) and 29.3	3 (2,4) and 38.7	<0.003
I feel that research will play significant role in my future career	4 (3,4) and 48.6	4 (3,4) and 59.3	<0.001
I feel that research will help me in qualifying for competitive exams like USMLE	4 (3,4) and 48.7	3 (2,4) and 40.6	<0.001
I feel that research is relevant to my long-term career goals	3 (2,4) and 42.7	4 (3,4) and 51.0	0.009
I feel that I am confident of doing research on my own	3 (2,3) and 20.0	3 (2,4) and 38.7	<0.001
MSP should be a mandatory requirement for the completion of 2nd MBBS	3 (2,3) and 11.4	2 (1,3) and 18.0	0.019
I feel that my research interest will influence my selection of the particular speciality for post graduation	3 (2,3) and 11.3	3 (2,4) and 32.7	0.034
I feel that undertaking MSP is stressful	4 (3,5) and 56.6	4 (3,5) and 63.0	0.142
I am ready to accept research assignments	2 (1,3) and 8.7	2 (1,3) and 18.7	0.282
I feel that research should be an integral component of undergraduate medical curriculum	3 (2,3) and 29.0	3 (2,4) and 33.4	0.010
I feel that experiences gained through MSP will help in publishing my research work in future	3 (2,3) and 34.0	4 (3,4) and 52.0	<0.001

IQR – Interquartile range

The majority (87%) of MCQs was within the difficulty index of 20% to 80%, whereas 73% of MCQs had the discriminating index of ≥0.2. The students’ scores in MCQ test showed statistically significant improvements (P < 0.001, 95% CI -1.62 to -0.71) following the MSPs (pre-test: 6.95 ± 1.95; post-test: 8.04 ± 2.15).

The majority of posters (68%) and project reports (62%) were of A grade. However, 20% and 18% of project reports were of B and C grade, respectively, whereas 25 % and 7% of posters were of B and C grade respectively. The PPMCC between scores obtained in posters and project reports was 0.418.

The CA values for questionnaires related to satisfaction and skills gained were 0.747 and 0.738 respectively. Median scores for most of the items denoting students’ satisfaction and perceptions regarding skills gained were ≥3 [Tables 2 and 3].

Table 2

Students’ responses [Mean and interquartile range (IQR)] to items related to satisfaction

Items	Median	IQR
I would take MSP again in Stage 2b if given the opportunity	2	(1,3)
I received appropriate level of guidance from my mentor	4	(2,4)
My mentor gave me enough opportunity and time to discuss the project with him/her	4	(3,4)
My mentor had an adequate expertise to guide me in MSP	4	(3,4)
My mentor was enthusiastic in guiding MSP	4	(3,5)
The time allotted for the completion of MSP was adequate	4	(3,4)
MSP should be a mandatory requirement for the completion of 2nd MBBS	3	(1.5,4)
MSP orientations given by MSP coordinators were helpful	4	(3,4)
Guidance from MSP coordinators was helpful…
In selecting the topic	4	(3,4)
To narrow down the topic	4	(3,4)
In writing the project report	4	(4,4)
In poster preparation	4	(4,5)
In protocol writing	4	(4,4)
In understanding research methodology	4	(3,4)
Students should be given an opportunity to present their research work as a poster	4	(4,4)
MSP “Poster presentation” organized by MMMC was a good platform to present research work	4	(4,5)
I was left out without any guidance during MSP	2	(1,3)

Table 3

Students’ perceptions [Median and inter quartile range (IQR)] regarding skills gained after MSP

Items	Median	IQR
To write protocol for scientific study	4	(3,4)
To formulate goal/objectives	4	(3,4)
To conduct literature search (to find out appropriate references)	4	(3,4)
Independently administer the intervention (like doing experiments)	4	(3,4)
To collect data	4	(3,4)
To prepare a questionnaire	4	(3,4)
To prepare a performa	4	(3,4)
To critically evaluate literature (relevant article)	4	(3,4)
To evaluate the strengths and weakness of my own study	4	(3,4)
To write project report	4	(3,4)
To use computers	4	(3,4)
To use statistical software packages	4	(3,4)
To design a poster for presentation of my research	4	(3,4)
To share thoughts and opinions with peer(s)	4	(4,4)
To work in a group	4	(4,5)

Faculty response rate was 40%. All faculty who responded in the questionnaire had more than 7 years of teaching and research experiences, and had published their research work in scientific journals. Table 4 depicts faculty responses to closed ended questions asked in the questionnaire.

Table 4

Faculty perspectives [Mean and inter quartile range (IQR)] regarding MSP

Items	Median	IQR
My interest to undertake research is enhanced	3.5	(1.25,5)
It enriched my research experience	3.5	(1,5)
It enriched my knowledge about research methodology	3.5	(1,4)
My perception of research having important role in my future career is enhanced	2	(1,3)
MSP has increased my confident to undertake research	2	(1,3.75)
My confidence to guide a research project has increased	3.5	(1,5)
MSP is not relevant to undergraduate medical education	2	(1,4)
Guiding MSP is stressful	3	(1.25, 4.75)
I am ready to accept research assignments	5	(4,5)
I will be encouraging others to take up research assignments	5	(4,5)
Research should be an integral component of undergraduate medical curriculum	4	(4,5)
Knowledge gained through MSP will help in publishing my research work in future	2	(2,3)
I am able to publish more research papers	3	(2.25,4)

Majority of students felt that mentors’ support, team work and guidance from MSP coordinators facilitated conduct of MSP. Lack of adequate time was the major hindering factor [Table 5].

Table 5

Students’ responses to open-ended questions regarding the conduct of MSP

Students’ perspectives				Faculty perspectives
Facilitating factor	Hindering factors	Measures taken to overcome obstacles	Changes required in MSP program
Teamwork	Inadequate time	Time management	More time for MSP	Facilitating factors: Students’ interest, guidance from coordinators, cooperative faculty colleagues
Guidance from coordinators	Difficulty in finding topics	Frequent communication with mentor	More committed mentors	Hindering factors: Inadequate free time for guidance, no funds, less motivated students
Guidance from mentor	Inappropriate guidance from mentors	Frequent discussion among team members	Student assessment in MSP	Measures taken to overcome: Time management, frequent meeting with students, delegation of work among colleagues
Group members	Uncooperative group members	Help from coordinators
Guidance from other faulty members	Misunderstanding among group members	Help from experts
Interesting topic	No sound knowledge regarding research methodology
Previous MSP	Low available sample
Help from senior students	Ethical committee approval procedure
Cooperative study subjects
Ample time to complete the project

To this date, 14 MSPs resulted in publications in indexed journals and 15 resulted in conference presentations. Around 5 MSPs lead to inter-departmental and inter-institutional collaborations.

Discussion

It is essential to impart knowledge and skills and inculcate a positive attitude regarding research among students from the beginning of their medical career through programs like MSP. The study initiated an evaluation of MSP program by measuring its short-term outcomes. Through this study, we developed valid instruments for assessing students’ knowledge, attitude and skills regarding research, and satisfaction regarding the program. Furthermore, the study also led to the development of criteria for the assessment of posters and project reports. The instruments developed detected a positive change in students’ attitude and knowledge gain while experiencing research. The present study, in addition, provided evidences for effectiveness and defined the scope for further refinement of the MSP program.

Retrospective pre-test [Table 1] demonstrated that the MSP changed students’ attitude positively. The attitude test also showed that, after the MSP, students understood the importance of research in their career in a better way. Although there is a statistical significance, the post-intervention total attitude score was just 1 point more than the pre-intervention score. In the MSP, students completed a research project within a short span of 6 months simultaneously learning other subjects of the second year of MBBS course, including the preparation of project report and presentation of poster. It was unlikely that students had an opportunity to have an experience of doing a major project. Hence though the short-term research program like MSP inculcates the knowledge regarding research methodology, more intense and long-term research programs may be required see the major attitudinal change. The study also revealed that students perceived the MSP as stressful [Tables 1 and 2]. The MSPs were implemented in the curriculum without making any changes in the already existing one. Students were expected to work on projects while simultaneously learning other subjects of the MBBS course, so it was an extra burden on the part of students. Therefore, it is noteworthy to mention that any curriculum innovation has to be made along with pruning of some other material from the curriculum. However, it has to be remembered that removing any material may come at the expense of student learning.[19]

As the majority of project reports and posters were of A category, it could be assumed that participating in MSPs instilled research skills in students. The newly-developed project report and poster evaluation instruments could differentiate different grades of posters and project reports. There was a weak correlation between scores of project reports and posters. This could be due to inability of student to choose appropriate information from project report to prepare poster or students were more careful in designing project reports as they were aware that the poster would judged by external judges and the best posters would be identified and awarded.

This study demonstrated only a small change in students’ knowledge after doing research. Our aim was not to quantify the students’ knowledge on research but to test our hypothesis that MSPs improve students’ knowledge in research and our instrument was able to detect that change.

Faculty responses regarding MSP reveal that they are ready to accept assignments of guiding students in MSPs, though they had not perceived any benefits from MSPs [Table 4]. Probably it was a very early phase of the experience of conducting MSP, and faculty might understand the benefits of it to students and themselves as they gain more experience from it. The mentors who responded in the questionnaire had >7 years of teaching and research experiences. Hence they may not have gained any additional benefits from guiding MSP.

Though this was a quasi-experimental study, the causative conclusion regarding the extent to which the MSPs influenced the knowledge, attitude change in students could not be made as there could be many factors that influenced this positive change. Moreover, the research skills gained was assessed through poster, project reports and students’ perceptions, which does not tell actual skills gained.

Despite these limitations, this study acts as a road map to action plan for remedial measures like the introduction of MSPs without increasing students’ overall workload and stress. It will be worthwhile to study the influence of MSPs on students’ learning approaches. The long-term impact of MSP on students’ career interest, scholarly work, and patients can also be explored.

Our model of undergraduate research project was found to be fostering positive attitude and a knowledge base about scientific research in students. As the majority of project reports and posters were of A category, it can be assumed that participating in MSPs instilled research skills in students. This model is practical and feasible to be implemented in undergraduate medical education worldwide. Moreover, researchers may use various instruments developed in this study across the globe for evaluating undergraduate research programs at their institute.