STEERing an IDeA in Undergraduate Research at a Rural Research Intensive University.

This study documents outcomes, including student career choices, of the North Dakota Institutional Development Award Networks of Biomedical Research Excellence program that provides 10-week, summer undergraduate research experiences at the University of North Dakota School of Medicine and Health Sciences. Program evaluation initiated in 2008 and, to date, 335 students have completed the program. Of the 335, 214 students have successfully completed their bachelor's degree, 102 are still undergraduates, and 19 either did not complete a bachelor's degree or were lost to follow-up. The program was able to track 200 of the 214 students for education and career choices following graduation. Of these 200, 76% continued in postgraduate health-related education; 34.0% and 20.5% are enrolled in or have completed MD or PhD programs, respectively. Other postbaccalaureate pursuits included careers in pharmacy, optometry, dentistry, public health, physical therapy, nurse practitioner, and physician's assistant, accounting for an additional 21.5%. Most students electing to stop formal education at the bachelor's degree also entered fields related to health care or science, technology, engineering, and mathematics (19.5%), with only a small number of the 200 students tracked going into service or industries which lacked an association with the health-care workforce (4.5%). These student outcomes support the concept that participation in summer undergraduate research boosts efforts to populate the pipeline of future researchers and health professionals. It is also an indication that future researchers and health professionals will be able to communicate the value of research in their professional and social associations. The report also discusses best practices and issues in summer undergraduate research for students originating from rural environments.

Introduction

The Department of Pathology in the School of Medicine and Health Sciences (SMHS) at the University of North Dakota (UND) coordinates support for many summer undergraduate health-related research experiences on the UND campus through participation in the Institutional Development Award (IDeA) Networks of Biomedical Research Excellence (INBRE) program. The ND INBRE is part of the larger IDeA program of the Center for Research Capacity Building (CRCB) within the National Institute of General Medical Sciences of the National Institutes of Health (NIH). The ND INBRE program provides organizational support for summer undergraduate student researchers who are sponsored through a variety of programs. These include the INBRE program itself; research grants from the NIH, National Science Foundation (NSF), and the Health Research Services Administration; the NSF Research Experiences for Undergraduates (NSF REU) program in the Biological Sciences; the Dean of the UND SMHS; and when active, the American Recovery and Reinvestment Act of 2009. A particular goal of the ND INBRE is to prime the “pipeline” for undergraduate students to continue in health research careers within IDeA states and the nation. The above program for undergraduate research was started in the summer of 2008 and followed the traditional path of identifying principal investigators of investigator-initiated research grants willing to mentor students for a 10-week summer research fellowship. The programs provided undergraduate salary support and a small supply budget. One issue reported by students regarding program oversight was that faculty had limited time for mentoring and that the students were often assigned to work primarily with a postdoctoral fellow, graduate student, or undergraduate with laboratory experience. In response, the faculty identified the pressures of preparing grants and publications to assure continued funding as a factor preventing aggressive mentoring of the student researchers. This issue motivated the ND INBRE program to determine whether this concern altered the career success and overall satisfaction of the undergraduate researcher and whether an alternative strategy could be developed that might enhance active learning, peer mentoring, and faculty interaction.

The ND INBRE program was able to test, and then apply, an alternative strategy for undergraduate research through participation in the Short-Term Educational Experiences for Research (STEER) program administered through the National Institute of Environmental Health Sciences. The alternate strategy was developed alongside the traditional program and was designed to address several issues. The first was to provide an environment with increased peer and faculty mentoring. The strategy entailed placing students together with mentoring from a small cohort of faculty that had protected time dedicated to the training of undergraduate researchers. The second issue was how to optimize the value of the research performed during the summer to both the faculty member and student. The strategy tested was that faculty would be highly engaged with the students if the students’ characterized gene/s were germane to their research. The last issue addressed was the very low rate of student research experiences prior to, and following, admission to baccalaureate programs within the institutions of higher learning in ND. The strategy tested was to give priority for program acceptance to rising juniors and seniors without prior undergraduate research experience. The present report details the ND INBRE experience with both programs, including career outcomes of 200 students following completion of the bachelor’s degree.

Methods

Traditional Research Program Faculty Mentors

The traditional program matched qualified undergraduate students (see below) with faculty members who volunteered to participate in the program. The only requirement placed on volunteer faculty mentors was that they had sufficient funds to support the summer student, a history of peer-reviewed research publications, and agreed to encourage student participation in ND INBRE sponsored events (research symposium, career seminars, networking events, research talks).

Short Term Educational Experiences for Research Faculty Mentors

Faculty members participating in the STEER program agreed to be present for at least 9 out of the 10 weeks of the summer program. Faculty also agreed to be present for the initial 2 days and the last 4 days of the program. The faculty decide as a group the identity of the genes to be analyzed during the summer program. The faculty also are responsible for having RNA and protein samples for the students from previous studies performed in their laboratory.

Basic Outline of the Short Term Educational Experiences for Research Program

The initial laboratory format, which has evolved over time, is subdivided into three flexible time frames. The first time frame is designed for the student to become proficient in pipetting small volumes with an understanding of triplicate measurement, dilutions to generate a standard curve, standard deviation, and standard errors. The student is also shown by direct laboratory exercise how to determine individual optical densities, use of a plate reader to generate a standard curve, and the concept of saturation density in measurements. The laboratory exercises are coordinated with lectures that explain the basic concepts underlying the laboratory exercises. A student cannot advance further in the laboratory until they master the above techniques, which varies from 4 to 8 days depending on the student. Outside the laboratory, the students are given lectures on laboratory safety before entering the laboratory, initial lectures on the responsible conduct of research, and how to keep and maintain a laboratory notebook. Also in the first week, the students are provided the identity of their genes to be studied and challenged to begin the process of discovering what is currently known about the genes structure and function. The students are encouraged to interact with program faculty and others in the laboratory regarding their findings.

The second time frame involves an introduction to gene expression and real-time polymerase chain reaction (PCR) through hands-on demonstrations and lectures. The students are given preexisting total RNA samples to quantify for use in their initial determinations of messenger RNA (mRNA) expression for their genes of interest. The students are also given a demonstration of RNA preparation, with an opportunity for a hands-on experience. The analysis of the mRNA expression for their genes under various experimental conditions will take an additional 4 to 6 weeks depending on the student. Lectures and workshops are also provided so that the students understand the initial experimental protocols, in many instances, time courses of exposure, from which their samples originate. The laboratory-based lectures are complimented by lectures from individuals with highly successful biomedical careers, including the obstacles associated with career development. These lectures are designed to highlight diversity in science and the importance of inclusion in the career pipeline.

The final time frame, usually the last 2 weeks, is utilized for data analysis, poster preparation and practice, and workshops. The workshops include hands-on experiences in cell culture, laser capture microdissection, and confocal microscopy. Demonstration workshops include routine tissue histology, immunohistochemistry, and Western blotting.

Undergraduate Student Selection

Information collected on the students consists of standard information such as name, current address, and permanent address if different. Academic information included high school of graduation (location and dates); college(s) attended, current academic level; expected graduation date; overall grade point average (GPA), science-only GPA, science hours completed; last semester GPA; hours completed last semester; ACT/SAT score; and anticipated major and minor subject areas. Demographic information included citizenship, ethnicity/race, first-generation status (whether either parent completed at least a bachelor’s degree), date of birth, and number and age of dependents. The application process also included 2 letters of reference, an autobiographical sketch, and an unofficial college transcript. Applications are reviewed for acceptance in the program by faculty mentors.

Pre- and Postsurvey of Undergraduate Student Participation

The students accepted for the summer undergraduate program are given a survey at the beginning and end of the program. The presurvey contains largely background and demographic information similar to that obtained on the application form. The presurvey asks for parent/guardian or other contact information to assist in locating the student in the future, if necessary, to determine degree and career outcomes. The survey also contains open content boxes for future academic and career plans. The postsurvey is designed to determine the impact the ND INBRE research experience had on students and includes a 9-question survey based on a 1 to 5 scale. In addition, the survey has an open comment section, so students can give additional input regarding the impact/satisfaction of the program. All activities related to the gathering of information for undergraduate student participants in the ND INBRE program are approved by the institutional review board (IRB) at UND.

Student Graduation and Career Outcomes

Graduation data for the students’ completion of a bachelor’s or other degree(s) are collected from the news releases and graduation programs publically provided by the respective universities, by self-report in follow-up surveys, and through the National Student Clearing House which provides graduation, but not career data. This latter source, however, requires the birthdate of the student and this was not collected in the early years of the program. Career paths of students following graduation are collected from a variety of sources. A major resource for student career tracking is social media as many students maintain contact with faculty and program staff through a variety of social media platforms. In addition, many students maintain social media accounts, such as LinkedIn, Facebook, and Twitter, that can be routinely accessed by program staff. This resource has been especially prevalent in the later years of the program. We now make a focused effort to encourage students to create a LinkedIn account if they do not have one and seek to connect with them through that platform. The ND INBRE does not collect social security numbers. All student-related information is stored in a database on a secure server maintained by Information Technology at UND SMHS. All activities related to the gathering of information on past student participants in the ND INBRE program are approved by the IRB at UND.

Results

Research Participation by Summer Undergraduate Student Researchers

The summer undergraduate research program coordinated by the ND INBRE was initiated in 2008 and remains active to the present. The primary source of funding for students participating in the program is the ND INBRE, followed by the NSF REU and the Office of the Dean of the SMHS. Prior to the organizational support from the ND INBRE, no detailed records are available for the UND or UND SMHS campuses on the overall number of undergraduates participating in summer research, the extent of their participation, their production of scholarly products, or ultimate career outcomes. The administrative core of the ND INBRE began tracking these data in 2008 and has compiled yearly data on the number of program participants each year from 2008 to the present (Figure 1). The students participating in the 10-week summer program were required to be present for 40 hours each week and were compensated at between US$10 and US$17.50 per hour depending on the program and how compensation is calculated for each program. The early years of the program were mostly in the US$10 per hour range and the later current years closer to the US$17.50 range. It should be noted that the higher levels (US$17.50) of compensation includes the expectation that the student be responsible for their own room and board. Some granting agencies provide funding for room and board for nonlocal students. As we reached out to students at in-state community colleges, tribal colleges, and other North Dakota universities, we recognized that this is an obstacle for participation by many students and thus raised the level of compensation to address this. The program has provided support for 335 individual undergraduate student researchers who participated in 396 summer research program experiences (Figure 1). Forty-nine students participated in 2 summer research experiences and 6 students completed 3. Financial resources available for the program stabilized participation at between 40 and 50 students each summer (Figure 1).

Figure 1.

Summer research participation by undergraduate students at UND SMHS. The bar graph shows the total number of students participating each summer in a 10-week research experience at UND SMHS from 2008 through 2016. The red line graph shows the cumulative total of undergraduate student participation from 2008 through 2016. SMHS indicates School of Medicine and Health Sciences; UND, University of North Dakota.

The results shown here and below under the next 3 subject headings are for all student participants supported by the ND INBRE structure regardless of program or funding source. The results are presented in aggregate since the 2 programs, traditional and STEER, had vastly unequal participation rates from 2010 through 2016. The initial 4 years of the program enrolled only 7, 5, 6, and 11 students in the STEER program, while the traditional program enrollment was between 25 and 35 students per year. In 2014, this distribution reversed with STEER being the program with the largest enrollments. This unequal distribution prevents a yearly comparison of the programs.

Poster Presentations by Summer Undergraduate Student Researchers

The ND INBRE program uses poster presentations by the summer undergraduate researchers to provide the students with the opportunity to present, discuss, and defend their research results with faculty and peers, and as such, this is a measure of scholarly activity. The number of posters presented by undergraduate students is determined using a defined set of criteria. A poster is counted only one time even when presented at multiple meetings. The poster is also counted only once per title and presentation even when multiple undergraduate authors are present on a given poster. The administrative core of the ND INBRE could not ascertain an accurate account of undergraduate posters and presentations prior to the 2005 to 2006 academic year; thus, results are presented from 2006 to the present (Figure 2). The low rate of undergraduate student participation as presenters at local, regional, and national meeting in 2006 to 2008 motived the ND INBRE to establish its own 2 research symposia. The first, established in 2009, serves the entire ND INBRE network of institutions and is called the ND INBRE Undergraduate Research Symposium. It is designed to stimulate interaction among faculty and students of the network partners and to provide a forum for undergraduates to present their research results via poster presentations. The symposium is a full-day event where the morning is dedicated to oral presentations and the afternoon to a poster session and opportunities for networking. The symposium is open to all faculty, staff, graduate students, postdoctoral fellows, undergraduate students, and high school students throughout ND. The administrative core of the ND INBRE program provides food for all attendees. There are no registration fees associated with the symposium. The second conference coordinated by the ND INBRE, the Summer Undergraduate Research Symposium, was also established in 2009 and is a poster session dedicated to students completing the UND SMHS Summer Undergraduate Research Program. This is a half-day event held on the final day of the summer program and all faculty at UND and the UND SMHS are invited to attend. Undergraduate participants from the UND SMHS summer program have produced a total of 402 posters since 2006, and the number per year shows a large increase under the organizational structure and support of the ND INBRE starting in 2009 (Figure 2). An additional impact of the symposium is to increase the number of opportunities a student has to present and defend their research. Approximately 48% of the posters presented at one of the ND INBRE symposia are presented at one additional meeting. Many are presented at 2 or more meetings, but this is a rarer occurrence.

Figure 2.

Poster presentations by the summer undergraduate research participants at UND SMHS. The bar graph shows the total number of posters presented each year from students who participated in a research experience at UND SMHS from 2006 through 2016. The red line graph shows the cumulative total of undergraduate poster presentations from 2006 through 2016. SMHS indicates School of Medicine and Health Sciences; UND, University of North Dakota.

Career Outcomes of Summer Undergraduate Student Researchers

The administrative core of the ND INBRE began tracking the career outcomes of students following the 2008 summer undergraduate program. The first issue addressed was the number of students who successfully completed the bachelor’s degree. The program was able to determine that 214 out of 233 students identified through tracking, or 91.8%, successfully completed the bachelor’s degree. The 233 students do not include the 102 undergraduates documented to still be attending college. There were several factors identified for students not completing the bachelor’s degree. The major factor is that the program does admit students for summer undergraduate research from community colleges and tribal colleges. More of these students, compared to those already at UND, obtain an associate degree and join the workforce instead of pursuing the bachelor’s degree. A second factor, although less prevalent, are students not completing the bachelor’s due to illness or injuries from accidents. With the high completion rates for the bachelor’s degree, these 2 issues account for the majority of those students who do not obtain the bachelor’s degree.

The ND INBRE administrative core was also able to track 200 (93.5%) out of 214 students for their career choices following completion of the bachelor’s degree. This number does not include the 102 students still pursuing their undergraduate careers. The first set of results determined the career choices of students continuing their postbaccalaureate education. This analysis showed that 76.0% of the students obtaining a bachelor’s degree went on to postbaccalaureate educational programs. The career choices of these individuals are shown in Table 1. The categories include students who have completed or are enrolled in programs leading to the respective degrees. Noteworthy is that 34.0% of the students chose to pursue the MD degree and 20.5% chose to pursue the PhD degree based on the total of 200 students tracked. The remaining 8 categories varied between 0.5% and 6.5% of the student population. Due to their impact on health care, several nonterminal degrees were included, such as physician assistant, nurse practitioner, and master’s in public health, clinical psychology, or social work.

Table 1.

Postgraduate Education Career Outcomes of Summer Undergraduate Researchers.‡,¦

Year	MD	PhD	PharmD	OD	DDS	DPT	PA	MPH	NP	MS§
2008	5	2	1	1	–	–	–	–	1	3
2009	11	9	–	–	1	2	3	2	–	2
2010	8*	11†	2	–	–	–	3	1	–	1
2011	12	2	1	–	1	–	–	1	–	2
2012	15	5	–	–	1	–	–	1	–	2
2013	6	3	–	–	–	–	2	–	–	2
2014	6	4	–	–	–	–	2	–	1	1
2015	5	5	1	–	–	–	–	2	–	–
Total	68	41	5	1	3	2	10	7	2	13
%	34.0	20.5	2.5	0.5	1.5	1.0	5.0	3.5	1.0	6.5

*Includes 1 student attending an osteopathic medical school.

†Includes 1 student completing a MD-PhD dual degree program.

‡Includes graduates of, and those enrolled in, post-bachelor’s degree programs.

§MS includes counseling psychology, epidemiology, social worker, research associate, health-care software development, science museum programming, high school science education, computer science.

¦Total number of post-bachelor’s degree students tracked is 200.

The administrative core also determined the career choices of students not pursuing a postbaccalaureate degree program but electing to enter the workforce following graduation. This group comprised 48 students (24.0% of those tracked) and their career choices are shown in Table 2. Five career choices were easy identified: nurse, other health-care worker, high school or elementary teacher, research laboratory technician, and information technology. Noteworthy is that the majority (39 out of 48) of these students chose to pursue careers in health care or science, technology, engineering, and mathematics (STEM). The service industry category was highly variable and formed no specific categories, but none was related to health-care delivery.

Table 2.

Career Outcomes of Summer Undergraduate Researchers With Terminal Bachelor’s Degrees.*,†

Year	Nursing	Other Health	IT	Service	Teacher	Technician	Other
2008	–	3	–	1	2	–	–
2009	–	1	–	1	–	1	–
2010	1	2	1	1	–	–	–
2011	1	2	–	2	–	–	–
2012	1	1	3	2	–	2	–
2013	5	2	–	2	–	–	1
2014	2	1	1	–	1	3	–
2015	–	1	–	–	–	–	1
Total	10	13	5	9	3	6	2
%	5.0	6.5	2.5	4.5	1.5	3.0	1.0

*Total number of bachelor’s degree students tracked is 200.

†Nursing includes nurse and certified nurse assistant. Other health care includes health center management and treatment, paramedic, respiratory therapist. IT includes web designer, software, and electrical engineer. Service includes farming, editor, bar manager, postal worker, banking, swim instructor, and minister. Teaching includes both elementary and high school. Technician includes research lab tech/manager or EPA technician. Other includes applying to medical or graduate school.

Career Outcomes of First-Generation Students

Among other demographic information, the application for the summer undergraduate research program asks if a student is first generation, defined as neither parent having completed a bachelor’s degree. Providing the information is optional and not required for acceptance into the summer program. Of 335 students, 316 provided this information. Of the 316 students, 95 (30.1%) were first generation and 221 (69.9%) were not. For those 214 completing the bachelor’s degree from 2008 through 2016, 197 reported being first generation or not. Of these 197 students, 144 (73.1%) reported that they were not first generation and 53 (26.9%) reported that they were the first in their family to graduate from college. To determine whether first-generation status affected career outcomes, the number of students enrolled in or graduating from an MD or PhD program was compared between the 2 groups of students (Table 3). These 2 parameters were chosen because they were the 2 career outcomes that had high levels of student participation. The results demonstrated that 32.6% of non-first-generation graduates and 32.1% of first-generation graduates were enrolled in or had completed the MD degree. The results demonstrated that 18.1% of non-first-generation graduates and 20.8% of first-generation graduates were enrolled in or had completed programs leading to the PhD degree. The results suggest that family degree status did not affect the outcome of students pursuing postgraduate education in both MD and PhD granting programs.

Table 3.

Influence of First-Generation Status on Student Careers in MD and PhD Programs.

Total first-generation students completing a bachelor’s degree	53
First-generation students in MD programs*	17 (32.1%)
First-generation students in PhD programs	11 (20.8%)
Total non-first-generation students completing a bachelor’s degree	144
Non-first-generation students in MD programs	47 (32.6%)
Non-first-generation students in PhD Programs†	25 (18.1%)

*Includes 1 student attending an osteopathic medical school.

†Includes 1 student completing a MD-PhD dual degree program.

A total of 335 students reported their demographic details (see Figure 3). There were 201 females (60%) and 134 males (40%) who attended the summer program between 2008 and 2016 (Figure 1). Of these, 245 (73.1%) self-identified as white, 67 (20.0%) as American Indian or Alaskan Native, 9 (2.7%) as Asian, 7 (2.1%) as black or African American, 5 (1.5%) as Hispanic or Latino, and 2 (0.6%) as Native Hawaiian or Pacific Islander.

Figure 3.

Demographics of the summer undergraduate research participants at UND SMHS. The pie charts show the percentage of total undergraduate researchers by gender (left) and race/ethnicity (right). SMHS indicates School of Medicine and Health Sciences; UND, University of North Dakota.

Evaluation of the Traditional and Short-Term Educational Experiences for Research Program

The ND INBRE asked the students to complete an evaluation, which includes a 9-question survey, on the last day of the summer research experience (Table 4). Using a 5-point Likert scale, the first 7 questions have choices from 1 “not at all” to 5 “a great deal.” The final 2 questions have choices from 1 “strongly disagree” to 5 “strongly agree.” In general, the results of the survey showed a high level of satisfaction with the summer research program for students participating in both the traditional and STEER program. The only areas where there was a statistical difference between the programs was in learning new skills and forming new contacts through networking, where those students in STEER scored these areas higher. The survey also contained 2 open boxes for student comments. One of these was “What would you change about the summer research experience” and the other “Any additional comments.” The comments were overwhelmingly positive and added nothing to the evaluation over that of the above 9-question survey. The only response that was repeated often was that students wanted to remain busy and engaged without lapses in activity.

Table 4.

Summer Undergraduate Research Program Student Evaluations.*

Query	STEER	Traditional
Strengthened research skills	4.65 ± 0.61 (112)	4.61 ± 0.72 (152)
Learned new lab skills	4.68 ± 0.61 (114)†	4.44 ± 1.01 (149)†
Made new contacts (networking)	4.49 ± 0.69 (128)‡	4.23 ± 0.83 (148)‡
Increased self-confidence	4.18 ± 0.82 (120)	4.17 ± 0.93 (149)
Better understanding of literature	4.36 ± 0.78 (117)	4.35 ± 0.90 (149)
Learned to work independently	4.27 ± 0.80 (119)	4.33 ± 0.84 (145)
Increased understanding of scientific method	4.46 ± 0.74 (119)	4.42 ± 0.67 (149)
Felt ownership of research project	4.39 ± 0.72 (115)	4.33 ± 0.93 (147)
Level of faculty interaction was appropriate	4.24 ± 0.90 (113)	4.42 ± 0.90 (141)

Abbreviation: STEER, Short-Term Educational Experiences for Research.

*Statistics used independent 2-tailed t test.

† P = .035.

‡ P = .0019.

Discussion

There is general agreement that an undergraduate research experience is an important addition to the academic training of the next generation of research scientists and health professionals. The concept that the undergraduate research experience is of added value to undergraduate education is advanced by both undergraduate students and faculty members in the peer-reviewed literature.[1-10] This is also particularly true for undergraduate minority and other disadvantaged students.[11-14] The acceptance of the concept that undergraduate research adds value to education in the sciences is also evidenced by the willingness of the federal government to invest in undergraduate research. The many individual institutes within the NIH provide funds for undergraduate research through the R25 funding mechanism. Similarly, the NSF also funds various programs for undergraduate research. Both agencies also encourage diversity in the undergraduate student research population through programs targeted to increase inclusiveness in the Nation’s future workforce. Despite the wide acceptance of the value of undergraduate research, most of the evidence on its effectiveness is anecdotal and empirical in scope. This is likely to continue since it is very difficult to identify a valid control group for comparison to the students who have an undergraduate research opportunity. In most cases, this would entail denial of the opportunity of qualified students to participate in an undergraduate research program in order to have a matched group for comparative purposes within a defined time and location. As an alternative measure of value, the present study focused on determining the career outcomes of undergraduate students who participated in the UND SMHS summer undergraduate research program.

The ND INBRE administrative core has been tracking the career outcomes of undergraduate students who participated in the UND SMHS summer undergraduate research program since 2008. The vast majority (91.8%) of summer undergraduate researchers successfully completed the bachelor’s degree. The ND INBRE administrative core was able to successfully track 200 (93.5%) of 214 students for their career choices following completion of the bachelor’s degree. A very high percentage of these students (76.0%) continued in postgraduate health-related education following completion of the bachelor’s degree. Of the 200 tracked students completing the bachelor’s degree, 34.0% elected to pursue the MD degree and 20.5% chose to pursue the PhD degree. Other postbaccalaureate pursuits included careers in pharmacy, optometry, nurse practitioner, dentistry, public health, physical therapy, and physician’s assistant, accounting for an additional 21.5%. In addition, the majority of students electing to stop formal education at the bachelor’s degree also entered fields related to health care or STEM. Only 4.5% of students went into service or other industries. This level of student career advancement into the postbaccalaureate health professions is viewed as strongly supporting the perceived value of undergraduate research. The students also indicated in the evaluation survey that the summer undergraduate research experience substantially increased their understanding of the scientific method along with many other positive survey results. These are important indications that these future researchers and health professionals will be able to communicate the value of research in their future professional and social interactions. Overall, the analysis of the current study supports the concept that participation in summer undergraduate research assists in efforts to populate the pipeline of future researchers and health professionals.

The programs supporting undergraduate research all encourage, but most do not require, recipients to determine student outcomes as a measure of success. Similarly, many of the programs encourage the collection of student demographics to determine inclusiveness of the program to promote a diverse future workforce. Despite this desire for data related to undergraduate student research, most programs do not provide direct cost funding for the collection of such student data. In addition, most of the programs limit or have policies that minimize indirect cost support; the R25 program of the NIH is a good example with an indirect cost rate of 8%. These low indirect cost rates also discourage institutions from using this resource to support tracking of students who participated in undergraduate research. The INBRE program, supported through the CRCB within the National Institute of General Medical Sciences of the NIH, is an exception to this trend. The INBRE defines a funding level for program evaluation within the request for application guidelines. This has allowed the ND INBRE to qualify the time requirement for program evaluation and the tracking of student outcomes. The ND INBRE can estimate, based on experience since 2006, that evaluation and student tracking consumes approximately 50% of a full-time staff equivalent (FTE) once a program has students entering the professional workforce. This estimate includes meeting with the students to establish social network identity; gathering demographic and contact information at the start of the program; completion of pre- and postprogram surveys; conducting follow-up surveys at annual or longer intervals; determining graduation rates for the baccalaureate degree; determining postbaccalaureate career choices; identification of scholarly products; collecting data from all faculty mentors; and finally entering data, maintaining, and updating a database that contains all the student information. The expenditure of administrative effort on social media to identify and then contact students for career outcome information is highly time intensive. The above FTE estimate also includes a similar tracking of approximately 350 undergraduate researchers in a sister program sponsored by the ND INBRE at the 4 state support primarily undergraduate institutions in ND. Among students currently being tracked, Facebook usage is ubiquitous, with LinkedIn profiles the next most common. LinkedIn is designed for professional networking and thus contact through that platform is preferred. While a student is occasionally located on Twitter or Instagram, these platforms are not specifically utilized for tracking at this time. A caveat noted in tracking students is that the stronger the relationship between tracker and student while the program is ongoing, the more willing the student is to report future career directions.

An additional issue that the current study examines is the career success of students who are the first in the family to attend college, referred to as first-generation students. This is of particular concern in ND, since ND is a very low population, large land mass state, which results in a very rural population. The current population of ND is 757 952 individuals[15] within a land mass of over 68 976 square miles, equating to an occupancy rate of about 10 citizens/square mile. Thirty-six of ND’s 53 counties retain a Federal designation as “Frontier Counties” as they have a population of less than 6 individuals per square mile. The national average is 80 individuals per square mile. The ND INBRE was able to track 197 students from 2008 to 2015 that self-identified as non-first-generation students (144) or first-generation students (53). The number of students pursuing postbaccalaureate MD or PhD degrees was compared between the 2 groups. The results demonstrated that approximately equal percentages of both groups had enrolled in or completed, the MD and PhD degrees. The finding provides empirical evidence that first-generation students performing undergraduate research compete favorably for admission to medical and graduate school.

An important feature of the summer undergraduate research program was the establishment of a forum where the students could present posters describing their summer research, defend the results of that research to other students and faculty members, and network with faculty and admission officers of health-related programs. The ND INBRE established 2 symposia, one at the end of the summer program and a second larger symposium in October of each year for all undergraduate student researchers, graduate students, postdoctoral fellows, and faculty in ND. This was very important for undergraduate research students in ND due to the rural nature of the state. Prior to these 2 symposia, there were no local or regional meetings in ND that focused on the research produced by students performing undergraduate research. In addition, there exists no city large enough in ND to host the national meetings of major scientific and medical societies where students might present their research. Thus, it was important for the students to have a venue where they could experience ownership of their research, successfully present that research to numerous conference attendees, and build confidence in their ability to present at larger venues as their careers progress. The poster also provided an end result for the students to present at interviews as they applied for postbaccalaureate educational and employment opportunities.

A limitation of the present study is that the results were obtained from students who originate from a rural population that is located over a large geographical area. This is also a value of the present study since the rural students are viewed as a group whose participation is necessary for a fully inclusive future workforce. In addition, there are many areas of the country similar to ND where rural students remain far removed from easy access to highly populated metropolitan areas. The program discovered one very important metric that applied to many of the rural undergraduate students from ND. This was that the majority of rising juniors and seniors attending college who were contemplating a career in health care or research had no prior experience in a research laboratory. This included those planning to pursue the MD or PhD degrees. The students appeared to have limited information that a research experience might be insightful for their career choice or that many postbaccalaureate programs require or highly value a research experience as part of the admission process. This recognition, when present, appeared to be gained only in the later years of the undergraduate experience. This lack of knowledge regarding research most likely reflects the rural high school education. Many of the rural areas in ND have high schools with graduating classes in the low double digits, with classes of about 20 not being uncommon. Even in the more populated areas of ND, it is only recently that a priority has been placed on providing Advanced Placement courses in the curriculum. Recent aggressive efforts in ND are addressing this need through online courses in advanced placement subjects. However, advanced placement science requires a laboratory experience for the student and does not adapt as well to online education. Regardless of cause, the present study found that for many students the recognition of the value of, and need for, undergraduate research occurred late in the undergraduate time line.

The finding that ND students had limited prior student research experience was the motivation for the program to change to a more structured initial undergraduate research experience originally funded by STEER. This change was put into place gradually using a limited number of students to determine both feasibility and student satisfaction. The first indication that a structured program was needed was the introduction to the pipetting of small volumes, sample dilutions, generation of standard curves and saturation, and the concepts of accuracy and precision. A student could not progress to the next step of the summer research until they could accurately pipette small volumes. The results showed that a high percentage of students understood the concept but had limited experience in the actual use of equipment within the laboratory. All students successfully completed the process within the initial week of the program. The program learned over time that special emphasis needed to be placed on the concept that once the standard curve flattens, it is no longer useful. At this time, the students were also given the identity of 3 unique genes that they would study over the summer. The students were able to employ all available informational resources to quickly and efficiently discover the science underlying each of their genes, in most cases more efficiently than the faculty mentors. An unanticipated favorable outcome also occurred during this first week. The students quickly coalesced with one another and formed an active learning and peer mentoring environment. They communicated problems, solutions, and successes among themselves on machine operation, gene searches, data analysis, and many other issues. This also enhances faculty mentoring as central issues concerning the student researchers could be rapidly identified and addressed by the mentors.

The high level of active learning and peer mentoring continued among the students as they mastered real-time PCR and the analysis of the expression of their genes over the ensuing 5 to 6 weeks. The last weeks, and interspersed throughout the program, included workshops on DNA/RNA isolation, confocal microscopy, cell culture, tissue histology and immunofluorescence in normal and diseased tissue, Western blotting, and bioinformatics. The initial plan was to allow students to determine the protein expression using Western blotting of their respective genes, enforcing the concept of transcription and translation. This was abandoned due to the fact that far too many commercial antibodies did not perform as advertised by the commercial entity. This placed faculty mentors in a situation trying to explain why the student’s reagents were not reliable, even though students could find Western blots in the literature of their protein showing one band on a cut-out figure. Due to this, a workshop on Western blotting has been developed which illustrates valid antibody use in Western blotting and invalid Western blotting using antibodies that do not perform as advertised by the product profile or literature. The students learn the value of seeing an entire Western blot and not a cut-out area of a Western blot. The students initiate poster preparation near the end of week 8 and present their poster in a summer undergraduate research symposium the last day of the program. All students retain a pdf of their poster and a standing offer for the poster to be printed as needed by the student.

The end of the summer surveys completed by the students show a high rate of satisfaction for both the traditional and STEER programs (Table 4). The evaluation did show a statistical increase for the STEER students in learning new laboratory skills and an increase in network formation with peers and faculty. In written comments, the STEER students also indicated that they liked the fact that each student had unique genes to study, but that all participants used a similar set of techniques. The surveys, plus the observation that the STEER program promoted both peer mentoring and active learning, motivated the ND INBRE to shift the majority of the summer undergraduate research students to the STEER program. Overall, as noted above, the more structured STEER program appears to have several advantages over the traditional method of randomly assigning students to primarily R01 funded laboratories. The authors wish to stress that this advantage may be limited to those students experiencing undergraduate research for the first time. There was one noteworthy disappointment in the STEER program that may or may not be present in the traditional method. The analysis showed that an overwhelming majority of the STEER students did not choose to continue their projects into the next academic year. This is despite the fact that many projects by the end of the summer were close to meeting the requirements for a peer-reviewed publication. The exact reason for this is unknown, but in general, the students appeared to lack the concept that a publication would be an asset in admission to postbaccalaureate programs. This may simply be a reflection of the rural background of the students.

The program can also provide an estimate of the impact of the STEER program on faculty time and research resources. The current STEER program accepts between 32 and 40 students for the summer program and employs 4 faculty members as the primary mentors. These faculty interact in the laboratory with students on a daily basis and also present most of the lectures the students need to understand the theory behind the projects and the methods of analysis. The faculty are assisted by an experienced postdoctoral fellow and at least 1 senior undergraduate and/or graduate student per lab who is fully engaged in the project during the 10-week period. Assistance by other laboratory personnel, overall program administration by the administrative core, protected time, and student research highly relevant to the mentor’s has mitigated faculty burnout thus far. Accounting for preparation time prior to student arrival, if the 4 faculty and postdoctoral fellow were engaged full time, this would equate to approximately 2.5 calendar months for each individual or approximately 1 FTE. An impact on faculty time that was not initially anticipated was the need to write letters of recommendation for the student researchers as they apply for employment and postbaccalaureate academic programs. Since students often apply to more than 1 program, and each may have a different reference requirement, these requests can number over 20 reference letters a year for each faculty member. With this taken into account, the one FTE figure is a very conservative estimate of overall faculty and postdoctoral time. The cost of supplies for the program is very cost effective since all the students use a similar base of supplies and all are routinely used during the normal research programs of the mentors. This maximizes the use of a reagent, allows purchases in bulk, and decreases the number of reagents remaining unused at the end of the program. A conservative estimate of cost per student would be between US$1500 and US$2500. The authors could find no literature on this topic and how these estimates would compare to the cost per student when students are place randomly into funded laboratories. It is anticipated that a similar analysis in other undergraduate research programs may enhance the understanding of the cost to benefit ratio of undergraduate research.