Evaluating Psychosocial Mechanisms Underlying STEM Persistence in Undergraduates: Evidence of Impact from a Six-Day Pre-College Engagement STEM Academy Program.

The persistence of undergraduate students in science, technology, engineering, and mathematics (STEM) disciplines is a national issue based on STEM workforce projections. We implemented a weeklong pre-college engagement STEM Academy (SA) program aimed at addressing several areas related to STEM retention. We validated an instrument that was developed based on existing, validated measures and examined several psychosocial constructs related to STEM (science identity, self-efficacy, sense of belonging to the university and to STEM, career expectancies, and intention to leave STEM majors) before and after the program. We also compared students in the SA program with a matched comparison group of first-year students. Results show that SA students significantly increased in science identity and sense of belonging to STEM and to the university, all predictive of increased STEM retention and a primary aim of the program. Relative to the matched comparison group, SA students began their first semester with higher STEM self-efficacy, sense of belonging, and science identity, positive career expectancies, and lower intention to leave STEM. The SA cohort showed 98% first-year retention and 92% STEM major retention. The SA program serves as a model of a scalable, first-level, cocurricular engagement experience to enhance psychosocial factors that impact undergraduate persistence in STEM.

INTRODUCTION

The retention of undergraduates who enter science, technology, engineering, and mathematics (STEM) degree programs has been identified as an area of national concern. There is a predicted need for qualified individuals to fill nearly 1 million new STEM jobs over the next 10–20 years (Carnevale ; Hossain and Robinson, 2012), sparking national calls to action in advocating reform in STEM education (see, e.g., American Association for the Advancement of Science [AAAS], 2011; President’s Council of Advisors on Science and Technology [PCAST], 2012). For the United States to remain internationally competitive in terms of education and science and technology advancements, it is necessary to increase focus on attracting a diverse body of talent to STEM fields and ensuring that students persist in STEM from the point of initial interest, into college, and subsequently into STEM careers (National Research Council [NRC], 2012). In the current study, we present preliminary findings of a novel cost-effective engagement program for incoming freshman students before the start of their first semester. The initial findings suggest promising results in terms of persistence at the university and in STEM disciplines, as well as in related psychosocial constructs (e.g., sense of belonging, science identity). We also suggest that adopting an interdisciplinary approach to examining persistence that includes psychosocial constructs can benefit understanding of STEM education.

Efforts to Increase STEM Retention at Universities

Many universities have developed or adopted programs to improve STEM retention, especially for students who come from underrepresented populations (e.g., Kuh, 2008; Labov ; Estrada , 2016; Hernandez ; Light and Micari, 2013). These programs vary in their structures, and include short 1-week summer boot camps (e.g., the Biology Intensive Orientation for Students [BIOS] program; Wischusen and Wischusen, 2007; Wheeler and Wischusen, 2014); intensive summer bridge programs that combine courses, mentoring, and research and continue into the students’ first year of study (e.g., the Meyerhoff Scholarship Program; Pender ; Stolle-McAllister ); cocurricular support and mentorship to students throughout their college careers (e.g., Biology Scholars Program; Matsui ), or course-based undergraduate research experiences (CUREs [Auchincloss ]; e.g., the Freshman Research Initiative program [Rodenbusch ]; or the SEA-PHAGES program [Caruso ; Harrison ; Hanauer ]). Many of these programs emphasize early engagement in undergraduate research, promoting hands-on, inquiry-based experiences that aim to increase “thinking like a scientist,” exposure to the application of course content, and connection with peers as a part of a “community” of scholars (Seymour ; Laursen ; Estrada ; Egan ; Hernandez ).

Although many of these programs show gains in STEM persistence, they are often tailored to an institution’s own STEM population needs and may require a significant investment of fiscal and faculty resources to implement. We sought to design a scalable, cost-effective cocurricular program (the STEM Academy, or SA) that did not require curricular changes or immediate undergraduate research but incorporated several practices shown to be effective in enhancing student success, retention, or engagement in long-term classroom or research settings (see Table 1 below). The SA took place across 1 week before the start of students’ first semester. Individual SA program elements (described in more detail in the Supplemental Material) were created based on research findings showing that: hands-on activities predict student motivation and interest in science areas (VanMeter-Adams ); mentorship experiences enhance interest and achievement (Pfund ; Feldman ; Daniels ); undergraduate research impacts persistence and engagement (Seymour ; Kuh, 2008; Laursen ; Linn ); reflective writing has benefits across multiple disciplines (Wald and Reis, 2010); and, perhaps most importantly, a sense of belonging/community is critical for integration and success in the university setting (Tinto, 1993; Anderman and Freeman, 2004).

TABLE 1.

Selected high-impact practices in the SA program

Mentorship	Students are mentored by graduate students and peers throughout all program modules with unstructured time built in for peer socialization and engagement.
Collaboration	Students work in teams to solve puzzles, complete math problems, discuss action plans, reflect on their experiences, and strategize about careers.
Writing	Students keep a journal and complete reflective writing exercises to specific prompts each day.
Hands-on experiences	Students engage in activities, including analyzing a standard curve assay, in a laboratory.
Career planning	Students engage and network with professionals across allied health, medicine, business, engineering, and education in several unstructured sessions and participate in career-planning/building workshops.
Research	Students attend professional development workshops and develop an action plan to engage in undergraduate research, tour research labs, and network with research faculty and graduate students.
Math competencies	Students complete exercises reflective of what they will experience in their first math courses, including regression analysis of their standard curve assay and precalculus problems.

We also aimed to extend research on STEM persistence by examining individual-level psychosocial factors that may drive the decision to stay or leave the STEM discipline (Cromley ; Estrada ; Hernandez ; Hanauer ). Research in this area is warranted for several reasons. First, research in psychology has long recognized the importance of these and other factors that, directly or indirectly, contribute to achievement and adjustment. Second, pioneering work by Seymour and Hewitt (1997) shows that a significant proportion of students who leave STEM have a high level academic success before leaving the major and that factors other than academic talent are likely to contribute to retention issues. In the Seymour and Hewitt (1997) study, many of the students interviewed indicated that they felt unwanted in the STEM discipline and lacked a sense of belonging or connection to the major and university. Third, recent reports emphasize that educational research on psychosocial constructs is a critically important area to explore, with the potential to enhance understanding of the mechanisms that impact students’ decisions to remain in or leave STEM disciplines (NRC, 2012; Cromley ; Estrada ). Finally, participation in non–course based activities (e.g., undergraduate research) is related to increased confidence, science identity, and networking (Grunert and Bodner, 2011; Harrison ; Ovink and Veazey, 2011). Thus, it is reasonable to expect the SA, as a noncurricular engagement program, to show gains in these important psychosocial measures that may in turn have significant impacts in STEM retention with the added benefit of minimal resource investment relative to some experiences, for example, mentored undergraduate research.

Psychosocial Factors and STEM Retention

Of the rich and relevant theoretical frameworks in psychology, this program and evaluation were primarily built from the complementary social cognitive theory and self-determination theory. In response to a call to increase educational research on psychosocial constructs in student experiences (NRC, 2012; Trujillo and Tanner, 2014), we aimed to examine several interrelated factors tied to STEM achievement and retention (self-efficacy, sense of belonging, science identity, and career expectancies in the context of STEM, as well as intention to leave STEM; for a similar approach, see Hanauer and colleagues, 2016).

Social cognitive theory (Bandura, 1997) emphasizes the role of perceptions of competence and ability and outcome expectations in achievement of personal goals (Bandura, 1982, 1986) as well as outcome expectations, or anticipated consequences of one’s actions (Bandura, 1986). High self-efficacy is related to positive general academic measures (e.g., grade point average [GPA], persistence, and achievement), regardless of prior academic success (Multon ; Usher and Pajares, 2008; van der Hoeven Kraft ; Komarraju and Nadler, 2013; for a review, see Zimmerman, 2000). Self-efficacy beliefs specific to STEM drive achievement and retention in STEM (Chemers ; Estrada ; Hernandez ; Perez ). However, it is important that self-efficacy reflects, at least in part, one’s true ability, as overconfidence may negatively impact performance or undermine perceived effort needed to achieve a task (e.g., Jensen and Moore, 2008). Positive expected outcomes (e.g., obtainment and desirability of employment in STEM) in turn foster self-efficacy, and when specifically examined in terms of science and math, are related to interest and course selection (Lent ).

Because self-efficacy facilitates regulation of one’s own learning and goals, increasing self-efficacy may enhance persistence, especially in the face of challenging STEM course work.

A sense of belonging, reflecting a feeling of acceptance by or membership in a particular community or group, also helps foster self-efficacy (Bandura, 1997) and is tied to positive academic experiences. From a self-determination theory perspective, belonging (along with competence and autonomy) facilitates intrinsic motivation, which underlies positive educational outcomes. Students fare better when they experience positive social interactions and peer relationships (Fass and Tubman, 2002; Dennis ). Other influential models target social integration and belonging at institutions as key to student persistence and success (see Tinto, 1993; Anderman and Freeman, 2004; Strayhorn, 2012). Research on belonging specific to STEM fields finds that a sense of belonging in math is predictive of intention to remain in math (Good ), and belonging in physics predicts exam grades and perceived value of the subject (Lewis ). Further, belonging may be especially important for women and for ethnic or racial minorities (Hausmann ; Good ), who are historically underrepresented in STEM disciplines (National Science Foundation, 2006; American Association of University Women, 2010).

Finally, belonging is closely tied to group identity. Commitment to STEM or science identity is important for understanding motivations and decisions regarding academic careers (Brickhouse ) and is linked to greater rates of persistence and lower intentions to leave STEM (Chemers ; Perez ; Robnett ), as well as to career choices (e.g., students who report a strong physics identity are more likely to pursue physics as a career; Hazari ). Science identity may be increased by undergraduate research experiences (Aschbacher ; Estrada ; Hernandez ) and is thus likely to also change based on other STEM-based experiences outside of course work. In this study, we expected the interrelated self-efficacy, positive career expectancies, belonging, and science identity to increase based on participation in a 1-week SA engagement program.

Present Study

In this study, we introduce the SA program and present data from its pilot cohort year (2015). Specifically, we 1) establish the validity of our survey measures; 2) examine changes in self-efficacy, belonging, career expectancies, science identity, and intention to leave STEM from beginning to end of the program; 3) compare these constructs with a matched comparison group of students beginning their first semester; and 4) determine the first-year persistence rates of SA and matched comparison students.

METHODS

SA Program Development

The SA program is a 1-week intensive-engagement program for incoming STEM students occurring just before the start of their first semester. The SA program consists of 19 different modules that are designed to expose the students to elements of several of the high-impact practices known to increase retention and build community (Kuh, 2008). These include but are not limited to mentorship, collaboration, writing, hands-on exercises, career planning, undergraduate research, and targeting math competencies. Table 1 summarizes these practices and highlights SA student experiences with each. Notably, the majority of SA program activities are delivered in a small-group setting that requires collaboration and the completion of action plans. Thus, even though the SA program occurs over a short period, its structure is designed to build a community of scholars. A detailed SA program guide containing personnel needs, anticipated costs, student applications, management timelines, and program modules can be found in the Supplemental Material. In addition, photos and video can be accessed through the SA website (www.usf.edu/admissions/freshman/stem-academy).

Participants

All participants in this project were students at the University of South Florida (USF). USF is located in a metropolitan setting and is classified by Carnegie as very high research. For the purposes of this project, two separate samples were recruited. The first were first-time in college (FTIC) students who participated in the SA program. The second included a non-SA group of FTIC STEM major students. In the following sections, we describe both of the samples. Note, from the larger non-SA group sample (see Non-SA Group), we selected a matched comparison group for analysis of psychosocial measures and retention (procedures for matching are detailed in the respective sections). Thus, the term “non-SA group” refers to the larger sample of non-SA first-year students, whereas the term “matched comparison group” refers to the smaller group selected for comparison. All procedures were approved by the university’s institutional review board (IRB).

SA Participants.

All admitted FTIC students who selected STEM College of Arts and Sciences majors (Biology and Chemistry Department majors, Physics, Math, Statistics, and Geosciences) and had a quantitative math Scholastic Aptitude Test (SAT) score < 650 or ACT < 29 were invited to apply to the SA program through a secure online application. Approximately 1200 students were invited to apply, 172 completed applications, and 140 were provisionally accepted (20 students never completed the additional communication requirement and were placed on waitlist). Admission was primarily based on the sophistication of the personal statement, the student’s understanding of the scope of the program, and entry into precalculus as a first math course, as the program was not intended to recruit a population that generally enters into higher-level calculus courses. The SA application and details on the review and admission process are detailed in the Supplemental Material. The first SA cohort was capped at 120 and ultimately included 115 students. Five students never attended the university after accepting admittance to the SA program. Of the 115 students who did enroll, six completed the program but were removed from the study because they entered into non-STEM majors (nursing and public health). Thus, the final N of the SA was 109. Of the SA students, 99 agreed to complete the psychosocial survey measures at T1 (pre survey), and these students also completed the T2 survey (postprogram). There were also an additional eight students who did not complete the pre survey, but who opted to complete the post survey. Thus, only 99 students were included in the pre–post survey analysis, but 107 were included in the comparisons with our matched comparison group (detailed in Procedures for Creating a Matched Comparison Group). Retention data for the entire SA group are presented.

Non-SA Group.

Students enrolled in introductory chemistry courses were recruited to complete the same survey as the SA participants for extra credit in the course. Chemistry was selected because it is a requisite course for most STEM majors at the university. A total of 1411 students completed the survey, including 720 FTIC, nonengineering STEM majors (non-FTIC and non-STEM majors were omitted from the analysis). This sample included 273 (37.9%) females; 445 (61.8%) males (two participants [0.3%] elected not to indicate gender); 334 (46.4%) white, Caucasian; 63 (8.8%) Black/African American; 138 (19.2%) Hispanic; and 135 (18.8%) Asian/Pacific Islander students (6.8% were classified as other or elected not to answer).

Procedures for Creating a Matched Comparison Group.

To select a matched group via propensity scoring, we used an R plug-in for the Matchit program in SPSS (Ho ; Thoemmes, 2012). In this approach, we used a number of covariates that are related to STEM persistence in a logistic regression model to create a propensity score reflecting the probability for one to select into the treatment (i.e., program) condition. For a review and example of methodology underlying and utility of propensity scoring in education programs, see Lane and colleagues (2012) or Rodenbusch and colleagues (2016).

In our model, we used demographic and achievement data available to us and relevant to STEM persistence, which resulted in inclusion of 10 covariates reflecting aptitude (combined SAT score or ACT equivalent), high school achievement (GPA and number of Advanced Placement [AP] credits earned), semester entered college (Summer or Fall; binary coded), race (four binary-coded variables for white/Caucasian, African American/Black, Hispanic, and Asian/Pacific Islander), gender (male or female; binary coded), and eligibility for the Federal Work Study (FWS) program (yes/no; binary coded). We used a 1:1 nearest-neighbor without replacement matching procedure within a 0.25 caliper (see Rosenbaum and Rubin, 1984; Stuart, 2010; Thoemmes, 2012). This resulted in a matched group of 109 students from the larger non-SA group. (See Supplementary Material Figure S1 for the distributions of propensity scores across matched and unmatched samples; distributions between the matched samples show greater similarity than those in the unmatched samples.) To evaluate balance between the matched comparison group and SA samples, we first examined whether the continuous-score covariates used differed significantly between groups. There were no mean-level differences in the continuous covariate variables of SAT score, number of AP hours, or high school GPA (Fs = 0.13, 0.06, and 0.22, respectively, all nonsignificant). Second, we examined whether there were significant differences between the SA and the matched comparison groups in the distributions of the categorical variables (race, gender, FWS, entry semester). Chi-square analysis indicated that there were no significant differences when comparing program participation with proportion of females/males, white/Caucasian students, Black/African-American students, Hispanic students, Asian/Pacific Islander students, Summer/Fall entry semester, and FWS eligibility (χ2s = 2.08, 0.16, 1.79, 0.24, 1.10, 0.03, 0.00, respectively; ps all nonsignificant).

Demographics of STEM Majors, SA Participants, and Matched Comparison Group.

Table 2 shows the demographic (gender and race) and intended major information for the SA cohort, nonengineering STEM FTIC, and matched comparison group. It is noteworthy that the SA cohort had a higher percentage of females than the overall nonengineering STEM population, but was similar in terms of race/ethnicity. Further, the SA group had an average combined SAT score that was 42 points below the nonengineering STEM population. The lower SAT scores were expected, given that the program was targeted to incoming students who were mostly likely going to start their college career in precalculus and had a quantitative SAT score <650. The matched comparison group was nearly identical to the SA group in all demographics, with an average SAT score midway between the SA and full STEM FTIC group. Most students in the SA and matched comparison group were enrolled in biomedical science and biology majors, 83% and 89%, respectively, and this is consistent with the full nonengineering STEM FTIC entering these majors (87.5%). It is noted that both cohorts also contained a small number of students who were enrolled in engineering majors. This was not intended but was due to four admitted SA students who changed their majors to engineering after attending the SA program and having the opportunity to engage with engineering faculty.

TABLE 2.

Demographic information of total STEM FTIC, SA participants, and matched comparison groupa

	Total	Male	Female	Black	Black male	Black female	Hispanic	Hispanic male	Hispanic female	Asian/Pacific Isl.	Asian/Pacific Isl. male	Asian/Pacific Isl. female	Ave Q + M SAT
STEM FTIC (not including engineering)	1407	539	868	136	38	98	316	128	188	184	85	100	1196
Percent of total		38.3	61.7	9.7	27.9	72.1	22.5	40.5	59.5	13.1	46.2	54.3
STEM Academy FTIC	109	33	76	11	1	10	23	7	16	14	5	9	1154
Percent of total		30.3	69.7	10.1	9.1	90.9	21.1	30.4	69.6	12.8	35.7	64.3
Matched comparison group FTIC	109	35	74	9	1	8	26	6	20	17	6	11	1184
Percent of total		32.1	67.9	8.3	11.1	88.9	23.9	23.1	76.9	15.6	35.3	64.7

SA majors: biomedical sciences (62); biology majors (29); chemistry (2); physics (4); health sciences (6); environmental sciences (2); engineering (4). Matched comparison group majors: biomedical sciences (85); biology majors (12); chemistry (8); physics (2); engineering (2). FTIC, first-time in college.

Measures

All psychosocial measures included self-report scales rated on a Likert rating system (1–7). Individual anchors of each scale are described in their respective sections. Individual items for each scale may be found in Table 3.

TABLE 3.

Confirmatory factor analysis: factor loadings of the scale items of all measuresa

Measure	Item	Factor loading
Academic self-efficacy for STEM	I feel confident in my ability to learn the material in my STEM courses.	0.82
	I am capable of learning the material in my STEM courses.	0.86
	I am able to achieve my goals in my STEM courses.	0.92
	I feel able to meet the challenge of performing well in my STEM courses.	0.90
Science-task self-efficacy	Generate a research question to answer	0.69
	Use scientific literature to guide research	0.73
	Create explanations for the results of the study	0.83
	Develop theories by integrating results from multiple studies	0.82
	Relate results and explanations of a research project to the work of others	0.85
	Use scientific language and terminology	0.71
Science identity	In general, my interest in science important part of my self-image.	0.76
	My interest in science is an important reflection of who I am.	0.77
	I feel like I belong in the field of science.	0.87
	I have a strong sense of belonging to the community of scientists.	0.90
	I am a scientist.	0.67
Expectancy for STEM career	I would enjoy a career in STEM.	0.94
	I have good feelings about a career in STEM.	0.91
	Having a STEM career would be interesting.	0.94
	I would like to have a career in STEM.	0.93
Belonging to university	I can get what I need at this university.	0.63
	This university helps me fulfill my needs.	0.69
	I feel like a member of this university.	0.83
	I belong at this university.	0.82
	I have a say about what goes on at this university.	0.70
	People at this university are good at influencing each another.	0.67
	I feel connected to this university.	0.90
	I have a good bond with others at this university.	0.77
Belonging to STEM
Membership	I feel that I belong to the STEM community.	0.88
	I consider myself a member of the STEM community.	0.95
	I feel like I am part of the STEM community.	0.96
	I feel a connection with the STEM community.	0.90
Acceptance	I feel like an outsider. (−)	0.76
	I feel accepted.	0.88
	I feel respected.	0.80
	I feel disregarded. (−)	0.69
	I feel valued.	0.84
	I feel neglected. (−)	0.71
	I feel appreciated.	0.83
	I feel excluded. (−)	0.72
	I feel like I fit in.	0.82
	I feel insignificant. (−)	0.72
Affect	I feel at ease.	0.82
	I feel anxious. (−)	0.59
	I feel comfortable.	0.86
	I feel tense. (−)	0.66
	I feel nervous. (−)	0.53
	I feel content.	0.78
	I feel calm.	0.72
	I feel inadequate. (−)	0.67
Desire to fade	I wish I could fade into the background and not be noticed. (−)	0.84
	I try to say as little as possible. (−)	0.77
	I enjoy being an active participant. (−)	0.62
	I wish I were invisible. (−)	0.80
Intention to leave STEM	I am likely to switch to a major that is NOT in a STEM field.	0.63
	I am likely to remain in my STEM major through to graduation or completion of my program of study. (−)	0.82
	I intend to leave my science major or science related track.	0.64
	I do not intend to leave my STEM major before I graduate or complete my program of study. (−)	0.71
	I am not going to remain in a major or track in the STEM fields.	0.60
	I am going to remain in a major or track in the STEM fields. (−)	0.78
	I have considered switching majors to one that is NOT in a STEM field.	0.60

aReverse-scored items are indicated by parenthetical minus signs. Model fit: χ2(1655) = 4356.87; CFI = 0.928; RMSEA = 0.046. All ps < 0.001.

Self-Efficacy.

Self-efficacy was evaluated in terms of academic course work and STEM-related tasks. Four items based on the self-determination theory literature (Williams and Deci, 1996; Ryan and Deci, 2000) assessed self-efficacy for course work (e.g., “I feel confident in my ability to learn the material in my STEM courses”; “I am able to achieve my goals in my STEM courses”; seven-point scale from “not at all true” to “very true”). Non-STEM versions of these items have been used reliably in many studies across varying contexts. Self-efficacy for science-related tasks was examined using a six-item scale developed by Chemers and colleagues (2011). For this scale, we prompted students to think about a project that they were involved in or may get involved in, and indicate the extent to which they were confident they could complete several tasks (e.g., “use scientific literature to guide research”; “use scientific language and terminology”; seven-point scale from “not at all confident” to “very confident”). This measure has been used reliably in past research on undergraduates.

Science Identity.

Identity as a scientist was examined using five items developed by Chemers and colleagues (2011; e.g., “My interest in science is an important reflection of who I am”; “In general, my interest in science is an important part of my self-image”; seven-point scale from “strongly disagree” to “strongly agree”). This measure has been used reliably in past research.

Expectancy for Career in STEM.

Positive expectancies of self in a career in STEM were examined using four items from a measure by Stake and Mares (2001). Items were partially modified to be specific to “STEM” rather than “science.” Participants read the prompt “Please think about yourself and rate how true the following statements are” and then indicated the degree to which they agreed with each statement (“not at all true” to “very true”; e.g., “I would enjoy a career in STEM”). The original science context of this measure has been used reliably.

Sense of Belonging to University.

Participants completed the eight-item sense of community scale, adapted from Peterson , to reflect the university context. That is, participants read the prompt “Please think about yourself as a member of this university, and rate your level of agreement with each statement. If you are new to the university, please use your initial feelings,” and then rated their level of agreement with statements such as “I belong at this university” and “I can get what I need at this university” (seven-point scale from “strongly disagree” to “strongly agree”).

Sense of Belonging to STEM.

Belongingness to STEM was measured using an adapted version of a 30-item scale developed by Good and colleagues (2012). The original scale was modified only in that the original context of math settings/courses was edited to reflect “STEM” settings/courses. This scale includes several subscales (all measured on a seven-point scale from “strongly disagree” to “strongly agree”) assessing belongingness in terms of membership (e.g., “I consider myself a member of the STEM community”), acceptance (e.g., “Regarding the STEM community, I feel respected”), affect (e.g., “Regarding the STEM community, I feel anxious”), desire to fade (e.g., “Regarding the STEM community, I try to say as little as possible”), and trust in instructors (e.g., “I trust my instructors to be committed to helping me learn”). Given that this project entails a precollege program, we did not presently analyze the trust in instructors subscale, as these items assume that students have begun their course work. Note, “desire to fade” items were reverse scored in order to be included in the composite sense of belonging scale (i.e., low scores indicated greater sense of belonging); that is, higher scores indicated lower desire to fade (and higher belonging).

Participants first read the prompt: “Today we have some questions we would like you to answer about your experience in the STEM community at this university. When we mention the STEM community, we are referring to the broad group of people involved in these fields, including the students in your STEM courses. Given this broad definition of the STEM community, please respond to the following statements based on how you feel about your membership in it. There are no right or wrong answers to any of these statements; we are interested in your honest reactions and opinions. Please read each statement, and indicate the number that reflects your degree of agreement.” Then, before the membership subscale items, participants read “When I am in a STEM setting at (university)…,” and before the other subscale items “Regarding the STEM community, please indicate your degree of agreement with the following…,” and then rated their level of agreement with each statement (“strongly disagree” to “strongly agree”; seven-point scale).

Intention to Leave STEM.

Participants rated their level of agreement with seven items created by Perez ; e.g., “I am likely to switch to a major that is NOT in the STEM fields”). Item responses ranged from “strongly disagree” to “strongly agree” and were both positively and negatively worded.

Survey Data Collection

The survey was administered through Qualtrics. SA students were introduced to the survey through email communication, during their USF orientation sessions (June and July), and through postings to the SA Canvas organization. SA students completed the pre survey 4–8 weeks before the start of the SA. Post surveys were completed on the last day of the SA program before the start of the semester. The non-SA group was introduced to the survey 2 weeks into the semester and could complete it over a 2-week window.

STEM Retention

All students entering the university as FTIC in Summer/Fall 2015 were evaluated for declared major at matriculation. STEM majors included biomedical sciences, cellular and molecular biology (Biology Department), chemistry/biochemistry, environmental biology (Biology Department), environmental microbiology (Biology Department), environmental sciences, geology/geosciences, integrated animal biology (Biology Department), Marine Biology (Biology Department), Mathematics, Microbiology (Biology Department), physics, statistics, all engineering majors, and management information systems. The 2015 cohort was revaluated in October 2016 for enrollment and declared major. Students who were not enrolled in Fall 2016 were scored as “loss from the university.” Students changing out of one of the listed STEM majors to a non-STEM major were scored as “loss to non-STEM major.” Students who changed major and also did not re-enroll were only scored as “loss from the university.”

RESULTS

Psychometric Properties of the Psychosocial Measures

We completed a three-part validation of the measures used for our instrument. Development and selection of measures was driven by the theoretical foundation described earlier and followed a careful review of the literature. The constructs focused on in this study reflect the primary aims of the SA program. To assess the psychometric properties of our measures, we conducted a three-part analysis in which we 1) examined the factorial validity of the measures using confirmatory factor analysis (CFA), 2) examined the internal consistency (reliability) of the respective measures, and 3) examined whether the scale variables evidence criterion validity via significant associations with measures we expect them to correlate with, including intention to leave STEM. Given that the measures used in the present study were based on preestablished, validated, and reliable published measures, we expected our approach to sufficiently address our aims (for a similar strategy used to validate a new survey measure of networking in undergraduate research experiences, see Hanauer and Hatfull, 2015). To have the statistical power (i.e., with a large enough sample) to conduct a factor analysis, we examined psychometric properties of the measures using data combined from the initial SA (pre survey) and entire non-SA group samples. Note, comparisons between SA and non-SA students on the measures and on retention were conducted using only the matched comparison group (see Results).

CFAs.

We used Mplus, version 7.4 (Muthén and Muthén, 1998), to conduct CFA on the survey variables. Model fit was evaluated using comparative fit index (CFI; acceptable model fit = 0.90 or above; good model fit = 0.95 or above), and root-mean-square error of approximation (RMSEA; good model fit =.05 or below, with 0.07 as an upper limit of acceptability; see Hu and Bentler, 1999). We also report the chi-squared test of significance; however, due to the impact of sample size on chi-squared tests, the CFI and RMSEA criteria are preferable to evaluate model fit.

A model in which the items from each measure were set to load on their respective latent factor was estimated. This model included 10 latent factors based on the expected structure of the data (factors include academic self-efficacy, task-specific self-efficacy, science identity, expectancies for STEM career, belonging to university, intention to leave STEM, and the belonging to STEM subscales of membership, acceptance, affect, and desire to fade). Modification indices in Mplus suggest that model fit would be significantly improved by adding paths depicting several correlated residuals between items. These simply reflect that a portion of the error terms between specific items are related, and are frequently added to CFA models of self-report survey data. Notably, all modification indices suggested adding correlated residuals between items within the same scales, but not across scales. A summary of these paths is as follows: paths were added between science identity items 1 and 2; university belonging items 1 and 2, 3 and 4; STEM belonging acceptance subscale items 5 and 7, 6 and 8, 4 and 6, and 4 and 8; and STEM belonging affect subscale items 2 and 4, 2 and 5, and 4 and 5. This model fit the data well. See Table 3 for factor loadings and model fit.

A second CFA was conducted to examine whether the sense of belonging subscales (membership, acceptance, affect, and desire to fade) loaded onto a higher second-order factor of sense of belonging (in line with Good ). In this model, all individual items again loaded onto factors reflecting their respective subscales with factor loadings of 0.59 or higher. Further, the four subscales loaded onto the higher-order factor with high factor loadings (membership = 0.78; acceptance = 0.97; affect = 0.87; desire to fade = 0.63). Finally, this model also fit the data acceptably, χ2(288) = 1429.41; CFI = 0.932; RMSEA = 0.07. Thus, the data suggest that it is appropriate to use either the four subscales or a composite score for sense of belonging..

Reliability of the Survey Measures.

Cronbach’s alpha was used to evaluate the internal consistency (reliability) of the survey measures. We first assessed alpha in the overall sample (combining across SA and all non-SA group participants). We also tested the reliability of survey measures in a matched comparison group (the procedures for selecting the matched group are outlined in the Methods section). All measures were sufficiently reliable, with alphas higher than the widely used 0.70 cutoff criteria. See Table 4 for alpha coefficients for the combined sample and for alphas for the SA and matched comparison group separately.

TABLE 4.

Cronbach’s alpha of measures

	Cronbach's alpha
Measure	Total sample	SA (pre)	SA (post)	Non-SA group (all)	Comparison group (matched)
Academic self-efficacy	0.93	0.91	0.94	0.92	0.94
Science-task self-efficacy	0.89	0.91	0.84	0.89	0.91
Science identity	0.90	0.83	0.83	0.90	0.91
Expectancy for STEM career	0.96	0.95	0.95	0.96	0.97
Belonging to university	0.91	0.89	0.89	0.91	0.91
Belonging to STEM	0.96	0.96	0.93	0.96	0.96
Membership	0.96	0.95	0.97	0.96	0.96
Acceptance	0.94	0.96	0.92	0.94	0.94
Affect	0.90	0.85	0.89	0.90	0.91
Desire to fade	0.84	0.87	0.79	0.84	0.86
Intention to leave STEM	0.85	0.78	0.85	0.85	0.81

Criterion Validity: Associations among the Variables.

Table 5 shows the bivariate correlations among the survey variables. For the purposes of examining the psychometric properties of the survey measures, and because associations were not expected to differ significantly between the SA and control group, the correlations are presented as combined across the samples. Overall, the correlations were consistent with expectations and support the criterion validity of the scales. In particular, intention to leave STEM majors was significantly associated with all psychosocial variables in the expected directions. Intention to leave STEM was negatively related to academic and task-specific self-efficacy, science identity, positive expectancies for a career in STEM, and belonging to STEM (overall and all subscales) and the university community.

TABLE 5.

Bivariate correlations among the survey variables

Measuresa	1	2	3	4	5	6	6a	6b	6c	6d	7
1. SE-Acad	—
2. SE-task	0.54	—
3. Sci ident	0.48	0.47	—
4. Expect car	0.43	0.36	0.50	—
5. Belong univ	0.33	0.33	0.37	0.31	—
6. Belong STEM	0.50	0.47	0.46	0.40	0.61	—
6a. Mem	0.49	0.43	0.59	0.47	0.55	0.77	—
6b. Accept	0.39	0.40	0.36	0.31	0.60	0.93	0.68	—
6c. Affect	0.44	0.39	0.37	0.29	0.45	0.87	0.57	0.73	—
6d. Fade	0.36	0.38	0.30	0.31	0.41	0.69	0.43	0.57	0.51	—
7. Leave	−0.33	−0.25	−0.39	−0.50	−0.18	−0.34	−0.39	−0.27	−0.24	−0.31	—

a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

All correlations significant at the p < 0.01 level.

Changes from Pre- to Post-SA

To determine whether the variables of interest changed across the duration of the SA program, we assessed mean-level changes in the survey variables from pre- to post-SA by conducting paired t tests. Only the 99 students who participated in both the pre and post surveys were included in this portion of the analyses. The results are shown in Table 6. SA students exhibited increased belonging to the university, belonging to STEM (overall and across all subscales), and science identity across the duration of 1 week. Interestingly, self-efficacy for science tasks significantly decreased from beginning to end of the program. There were no differences in self-efficacy for academic courses, expectancies for STEM careers, or intention to leave STEM.

TABLE 6.

Mean-level differences in the study variables before and after STEM academy program

	Pre		Post
Measuresa	M	SD	M	SD	t	p
1. SE-Acad	6.33	0.67	6.25	0.75	1.07	ns
2. SE-task	5.87	0.90	5.69	0.74	2.19	<0.05
3. Sci ident	5.92	0.82	6.08	0.78	−2.16	<0.05
4. Expect car	6.60	0.63	6.59	0.61	0.13	ns
5. Belong univ	5.91	0.78	6.22	0.68	−4.12	<0.001
6. Belong STEM	5.50	0.85	5.84	0.71	−4.65	<0.001
6a. Mem	5.68	1.00	6.23	0.86	−5.24	<0.001
6b. Accept	5.66	0.99	6.17	0.77	−5.44	<0.001
6c. Affect	4.98	0.97	5.28	1.16	−3.01	<0.01
6d. Fade	5.90	1.08	6.10	1.02	−2.11	<0.05
7. Leave	1.94	1.33	1.93	1.04	0.09	ns

a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

Because there was no change in academic self-efficacy, and there was a negative change in self-efficacy for science tasks, we tested whether changes from pre to post differed depending on whether the participants entered with an already high level of academic self-efficacy. Because the sample size was low, we were unlikely to have sufficient statistical power to conduct true moderation tests; therefore, we simply classified participants as high, average, or low in academic self-efficacy by mean-centering the variable and then separating based on ±1 SD. Using this approach, we found that both measures of self-efficacy decreased only for those who scored high in academic self-efficacy in the preprogram survey (t = 4.21, p < 0.01; t = 2.31, p < 0.05).

Differences between SA Participants and Matched Comparison Group

Because students in the SA program showed increases in important measures such as belonging to the university, belonging to STEM (overall and across all subscales), and science identity, it was pertinent to determine whether the gains were also evident when the SA students were compared with a cohort of students who had not participated in the SA program.

To examine whether SA students differed from the matched comparison group at the start of their first year, we conducted one-way analysis of variance (ANOVA) to test for differences in the study variables between the matched comparison group (who completed the identity survey within the first 4 weeks of entering the university) and the post scores of the SA participants (who completed the survey several days before entering their first classes). Thus, this comparison should approximate the levels of the constructs that students had at the beginning of their first semester in college. Note: SA participants’ average scores on the post survey differ somewhat from the previous analysis of changes from pre- to postprogram due to the additional eight students who participated in the post survey (but not the pre survey). When compared with the matched comparison group, SA students scored significantly higher in academic self-efficacy, science identity, positive expectancies for STEM career, belonging to university, and belonging to STEM (overall and in each subscale), and significantly lower in intention to leave STEM (Table 7).

TABLE 7.

Mean-level differences between STEM academy and matched comparison group at start of first semester

	STEM Academy		Matched comparison group
Measuresa	M	SD	M	SD	F	p
1. SE-Acad	6.22	0.85	5.87	0.96	8.04	<0.01
2. SE-task	5.67	0.76	5.57	0.96	0.81	ns
3. Sci ident	6.09	0.80	5.48	1.14	20.92	<0.001
4. Expect car	6.60	0.60	6.23	1.08	9.42	<0.01
5. Belong univ.	6.20	0.68	5.76	0.86	17.25	<0.001
6. Belong STEM	5.83	0.76	5.07	0.93	43.44	<0.001
6a. Mem	6.24	0.89	5.09	1.22	61.62	<0.001
6b. Accept	6.15	0.82	5.22	1.05	52.69	<0.001
6c. Affect	5.27	1.20	4.65	1.16	15.02	<0.001
6d. Fade	6.07	1.02	5.57	1.13	11.84	<0.001
7. Leave	1.93	1.04	2.35	1.43	5.91	<0.05

a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

Although we observed significant differences between the SA (post survey) and matched comparison group, we also noted that there were several unexpected differences between the matched comparison group and SA pre survey scores. Specifically, SA students scored higher in their pre scores of academic self-efficacy (F = 15.77, p < 0.001), self-efficacy for science tasks (F = 5.19, p < 0.05), science identity (F = 9.96, p < 0.01), positive expectancies for STEM career (F = 7.02, p < 0.01), and belonging in STEM (F = 11.68, p < 0.01) and lower in intention to leave STEM (F = 4.70, p < 0.05) than matched comparison group students. There was no difference in belonging to university. Thus, there are clearly unobserved characteristics in the SA group that were not captured by the covariates used in the propensity-scoring procedure but may have impacted selection into the program (even though variables such as high school GPA and number of math and STEM courses were used).

Retention after the First Year

The final analysis was to evaluate STEM retention of the SA and matched comparison group in the context of the full FTIC STEM cohort. Two factors contribute to STEM retention: 1) loss of a STEM student from the university, and 2) students who persist at the university but change from a STEM to non-STEM major. Table 8 shows the results for the SA, matched comparison group, and full 2015 STEM FTIC evaluated for enrollment in Fall 2016 (first-year retention after completion of the Fall 2016 drop/add period) and current major. In addition, results are presented for the specific STEM majors within each cohort. The SA lost 1.8% of the cohort from the university compared with 4.6% for the matched comparison group and 9.5% from the 2015 STEM FTIC (not including the 109 SA students). Loss from STEM due to transfer to non-STEM majors was also reduced in the SA (6.4%) compared with the matched comparison group (8.3%) and the total FTIC (8.0%). Thus, total STEM retention for the SA was 91.7% compared with 87.2% for the matched control and 82.5% for the full STEM FTIC. Increased retention in the SA cohort in comparison with the matched comparison group is also reflected in the biomedical science (BMS) major, which has the highest number of incoming STEM students to the university. Thus, when the SA cohort is included in the total STEM FTIC, overall STEM retention at the university increased 0.5%. It is noteworthy that a portion of the SA programming discusses how students can obtain STEM competencies while enrolled in a non-STEM majors. To assess the SA retention results in more detail, we evaluated the transcripts from students in the SA and matched comparison group who left the STEM majors but were still enrolled at the university. Of the seven students in the SA who switched to non-STEM majors, five were still taking core STEM courses compared with only one of nine in the matched comparison group. Thus, 96% of the SA cohort was still involved in taking STEM core courses compared with 89% of the matched comparison group.

Table 8.

Retention data for STEM Academy, matched comparison group, and FTIC STEM cohorta

aSTEM FTIC (w/o SA) = all entering STEM majors without the 109 SA students. BMS = biomedical science majors; BIO = Department of Biology majors; other STEM majors = chemistry, physics, math, and geosciences majors. The last row: total STEM FTIC is equal to the 109 SA cohort + 2015 total STEM FTIC (w/o SA) cohort. Total heads lost from STEM = heads lost to non-STEM major changes + heads lost from the university. STEM retention = total heads lost from STEM/total heads entering in 2015.

DISCUSSION

In this study, we have introduced the early engagement SA program and examined psychosocial constructs significant to STEM persistence. The results suggest that the SA may be a promising cocurricular initiative for increasing measures of STEM retention at the undergraduate level that may be scaled to different universities and serve as a “first-pass” program to more intensive semester-long CURE initiatives. Further, results support the notion that considering individual-level and social factors of STEM persistence heightens understanding of a program’s effectiveness and may give educators a better idea of the areas that should be targets of reform efforts. In the following sections, we discuss our results first in terms of the efficacy of the program (survey results and first-year retention) and then consider our analysis of psychosocial constructs, as well as practical considerations, limitations, and future directions.

Efficacy of the SA Program: Psychosocial Constructs

As a whole, the first cohort of the SA program showed promising results in terms of psychosocial underpinnings of STEM success. Now, we consider results from the pre–post changes and comparisons with the matched comparison group separately for each construct.

SA participants increased in their sense of belonging to the university and to the STEM community across the duration of 1 week. This was true in terms of overall sense of belonging to STEM, as well as in several subscales: participants reported greater feelings of acceptance and membership, more positive emotions in the STEM context, and less desire to fade at the end of the program relative to the beginning. Also, relative to a matched comparison group, participants began their first semester in college with a greater sense of belonging to the university and to STEM (again, across all subscales). Thus, the data suggest preliminary support for the capacity of the SA program to build community and peer connections, identified as fundamental reasons for persisting in science disciplines both during college and into careers. This is especially true for women and underrepresented minorities, who often experience both direct and subtle hostility and bias from their majority counterparts (Seymour and Hewitt, 1997; Estrada ; Hernandez ).

Considerable psychological research supports the need to heighten sense of belonging in STEM disciplines. For instance, work in goal congruity theory (see Diekman ) suggests that individuals (most often women) who hold communal values are more likely to leave STEM, as they perceive these fields to be discrepant with their values. As feelings of belonging correspond with communal values, targeting this construct is highly important. Thus, the increase in belonging in SA students is perhaps the single most encouraging finding from this study. However, in addition to the increased belonging in STEM disciplines, SA students also showed increase in belonging to the university; thus, students in the SA appear primed to participate in other engagement initiatives and use resources (regardless of ultimate major), to have more positive overall university experiences.

Participants also increased in science identity from beginning to end of the SA. Broadly speaking, identity and self-concepts guide motivation and behavior in a variety of contexts. Thus, science identity likely impacts the degree to which one is willing to pursue and persist in STEM disciplines, even in the face of potential challenges or hardships. Previous research has found that commitment to a science identity is predictive of persistence at various levels of STEM education (e.g., Aschbacher ; Chemers ; Perez ; Robnett ) and that science identity may be increased via undergraduate experiences in research (Charney ). In this study, immersing students in settings that foster peer support and provide hands-on engagement opportunities and rigorous training in how to engage in undergraduate research in subsequent semesters likely facilitated greater identification with the values of a scientist. However, future research should disentangle which portions of the program may have driven the increases found.

Existing programs that include undergraduate research experiences/CUREs or hands-on activities (e.g., Charney ; VanMeter-Adams ; Wheeler and Wischusen, 2014) find meaningful effects on psychological and motivational constructs. However, our results suggest that similar goals may be achieved even before the students gain valuable research experience at the university, and this is beneficial for students who will enter into a longer-term program. Indeed, fostering feelings of belonging by engaging students in a peer group and with supportive mentors and staff likely helps prepare the students for the challenging experiences they will face (including undergraduate research) within the context of the STEM curriculum.

It is important to note that some of the results were more challenging to interpret but still shed light on students’ experiences in the SA program. First, academic self-efficacy did not increase from beginning to end of the program, but participants in the SA scored higher than those in the matched comparison group. Further, self-efficacy for science tasks decreased across the duration of the program, and SA participants did not differ from the matched comparison group in this variable. The decrease in science-task self-efficacy and lack of change in academic self-efficacy were unexpected, given that an aim of the program was to build students’ self-confidence for STEM-related course work and tasks. However, this may simply reflect an imposed sense of realism regarding the rigor necessary in STEM contexts, as students sometimes lack understanding of the effort required in a university setting (Ozga and Sukhnandan, 1998). Our exploratory analysis supports this idea: self-efficacy decreased only for those students who entered with very high levels. While perceived competence bolsters achievement (van der Hoeven Kraft ), an inflated sense of competence can undermine effort exerted to complete tasks (Usher and Pajares, 2008). Relatedly, researchers also found that the BIOS program (a “biology boot camp” of curricular exposure for first-year students), may decrease participants’ self-efficacy (considered evidence of positive impact; Wischusen and Wischusen, 2007; Wheeler and Wischusen, 2014).

Even with this explanation, the importance of modifying self-efficacy should not be downplayed, because a large body of evidence supports its importance in fostering self-regulated learning, academic achievement, and overall well-being (Zimmerman, 2000). As we presently consider self-efficacy as especially important to success in challenging STEM fields, some modifications to SA content may lead to more favorable outcomes. For instance, given some elements, like lab settings where students are required to solve several precalculus problems or workshops designed to correct misperceptions regarding medical school admissions, it may be possible to achieve the learning goals of these exercises in a more positive and empowering manner. The analysis of the 2016 and upcoming cohorts will hopefully provide additional insight into this area.

Career expectancies in STEM did not change from beginning to end of the program, but SA students reported more positive expectancies than the matched comparison group. The end session of the SA program highlights aspects of a variety of STEM and health careers (e.g., a “career speed-dating” session in which students interacted with professionals in various disciplines). However, the majority of the SA focused on experiences at the college level. Thus, it is not entirely surprising that expectancies of future careers did not seem to be affected, as these are more distal in students’ thoughts about their trajectories. Regarding intention to leave STEM majors, results were similar: SA participants did not change from beginning to end of the program, but scored more favorably than their matched comparison group counterparts. This likely reflects a floor effect in which students entering the SA began with such low levels that there was little room for change (although any variation should be captured by the analyses used). Importantly, intention to remain or leave STEM majors will ultimately be evidenced in actual persistence rates and longitudinal follow-ups later in students’ college experience. However, the analysis of first-year retention showed that the SA students were retained at higher levels both to the university and in STEM relative to the general STEM population and matched comparison group.

In summary, from the results presented, we conclude that, relative to the matched comparison group, SA students started their first year with a more favorable mind-set that should foster success in STEM and further engagement with the university community. This is presented with the caveat that evaluation of the SA scores on the pre survey also suggest that the SA students may have begun their semester more favorably even if they had not participated in the SA. It will be of interest to see whether this trend continues when the 2016 cohort of 222 students is evaluated and the total number of participants increases. Nonetheless, the SA did produce measurable changes to key attributes associated with STEM retention, and the difference in SA participants relative to the matched comparison group was amplified.

First-Year Retention and the SA Model

It is reported that only 40–50% of undergraduates who begin in a STEM degree program graduate with a STEM degree (e.g., AAAS, 2011; PCAST, 2012). Loss from STEM includes loss from the university and also students who start in STEM but change majors and graduate from non–STEM degree programs. The time frame of actual STEM loss is not readily available in the literature, but it is clear that the first year is critical when targeting overall persistence to the university (Cuseo, 2003; Hossain and Robinson, 2012). First-year retention is a key metric for many universities, although most do not publicly report STEM retention specifically. Outcomes of retention and graduation with a STEM degree, as the ultimate goals of the SA program, cannot be evaluated for several years. Yet evaluation of first-year retention of the cohort reveals evidence of program effectiveness in retaining students. During their first year, SA students were retained in STEM majors and left the university at a rate much lower relative to the matched comparison group or total STEM FTIC (91.7%, 87.2%, and 82.5%, respectively). Although the SA cohort was only 109 students, the impact of the population on university retention was 0.5%. This may appear to be a small percentage increase, but it can represent a significant number of students at a large public university. In addition, 100% of the SA students from underrepresented groups were retained and 94% remained in STEM disciplines (full demographic analysis of the SA will be the focus of future work). We consider these results to be very encouraging moving forward, and as the program increases in size, and we anticipate obtaining follow-up information from current students regarding their academic progression.

Cost and Scalability

We believe that the current study provides preliminary support for the SA model as an effective, entry-level engagement program to increase STEM persistence at the university. Because financial resources for education reform are often limited, the SA program costs were manageable, at approximately $390 per student including early move-in fees, or $260 if fees for housing and meals were deferred. The other SA costs went toward graduate student mentor assistantships ($3000 per mentor), peer mentor stipends ($450 per mentor), student supplies (e.g., SA-branded lab coat, lab supplies, resource notebook, and other expenses), and an opening reception (keynote speaker, meal). These costs were fully subsidized by the HHMI grant; SA students did not occur any costs to participate in the program. Notably, these costs will differ depending on the university, what program elements are included, and what staffing (and associated salaries) are required for implementation of the program (for the program described here, there were no costs associated with faculty or the SA director). These general values are provided simply to offer context to the scalability of the program and for comparison with other longer-term curricular interventions.

Integrating Psychology into Understanding STEM Persistence

We consider the interdisciplinary nature of this study to be a strength. Just as teaching in the hard sciences is informed by educational research, educational research in STEM contexts can and should be informed by psychological theory, studies, and methods. As outlined by Seymour and Hewitt (1997), students do not persist due to a variety of factors outside of competence and talent; however, these factors have only recently been considered in the STEM education literature. This study extends existing research on STEM education by directly assessing several relevant psychosocial correlates of academic and STEM-specific success. Our program and evaluation were framed in terms of prominent theoretical frameworks reviewed in the introduction. Accordingly, this study provides a model in terms of theory and method for future programs to consider adopting when attempting to understand precisely how the program affects students. In this study, we used a survey instrument that included partially modified preexisting and validated scales. This approach ensures the validity and reliability of constructs measured (as opposed to using piecemeal, single items or assuming effects on students). In fact, our analysis demonstrated reliability of the scales, construct validity (via a CFA), and criterion validity (via correlations among the study variables). Each of the psychosocial constructs was correlated with intention to leave STEM in the expected directions. One implication of this result is that educators with limited access to confidential student data may benefit by conducting time-efficient survey research in which students report their intentions (assuming that there is little motivation to inaccurately report intent to remain or leave majors). Finally, it is important to note that, although we present the constructs as correlates of STEM success, the developmental interplay of the constructs is more complex in nature and should be evaluated with longitudinal research. For instance, a sense of belonging may directly increase science identity, which in turn may drive motivation and persistence. Incorporating sophisticated longitudinal methodology from the psychological sciences allows for more detailed understanding of directional associations of multiple constructs, and as we obtain additional data from other cohorts, we will begin to assess such relationships (e.g., Robnett ).

Limitations and Future Directions

Despite the strengths and promising findings in our study, there were some limitations. First, a randomized control trial was not possible in the present context; thus, we were not able to establish true causal effects of the program. Although the study used a matched comparison group, allowing for stronger claims or inferences to be made, it does not fully eliminate the self-selection bias of the students in the program. However, this issue may lead to areas of future inquiry. For instance, it will be interesting to determine what traits drive students to self-select into such programs. As we controlled for high school achievement (GPA, number of AP courses, and SAT scores), and showed that the SA students’ average SAT score was lower than the general STEM student population, academic competence alone does not explain the self-selection. Rather, there are likely other personality or motivational characteristics that should be evaluated (and potentially targeted). These unmeasured characteristics may also partially explain the differences found between the matched comparison group and preprogram levels of the psychosocial constructs. Additionally, future programs could consider attempting to include a covariate that taps into prior participation in extracurricular activities (during high school), as students with previous experience may have been more inclined to apply to the program. Another potential issue with the matched comparison group is that these participants completed the survey once their semester had already begun (vs. 3 days before the start of the semester in the case of the SA students). Experiencing some college may have affected findings. It may be possible to address this in future cohorts of the program by attempting to solicit responses from a comparison group before the start of the semester.

Second, the 1-week SA program entailed many components, making it difficult to disentangle which portion(s) may have affected the psychosocial and retention outcomes. For instance, the mentorship and group work likely influenced sense of belonging, whereas the hands-on lab experiences may have been more related to self-efficacy. Future research could conduct more controlled, small experiments to evaluate each component separately. Further, more detailed comparisons in future studies with larger sample sizes may allow for greater understanding of differential program impact based on social identity characteristics of student gender and ethnicity (e.g., Eddy ).

In summary, this report reports the implementation of the SA program and our findings through 1 year with the first cohort; there are several anticipated works to come in the future, such as longitudinal follow-up of the first cohort of SA students including STEM persistence and graduation rates. In addition, we have collected survey data from nearly 200 students in the 2016 SA program and more than 1000 comparison group responses to determine whether the results reported here are also observed in this larger group. As our data sets grow, we anticipate using modeling techniques (e.g., longitudinal multigroup structural equation modeling) to examine the interplay of program participation, psychosocial factors, and retention outcomes across students’ time in college.