Integrating Critical Approaches into Quantitative STEM Equity Work.

The recent anti-racist movements in the United States have inspired a national call for more research on the experiences of racially marginalized and minoritized students in science, technology, engineering, and mathematics (STEM) fields. As researchers focused on promoting diversity, equity, and inclusion, we contend that STEM education must, as a discipline, grapple with how analytic approaches may not fully support equity efforts. We discuss how researchers and educational practitioners should more critically approach STEM equity analyses and why modifying our approaches matters for STEM equity goals. Engaging with equity as a process rather than a static goal, we provide a primer of reflective questions to assist researchers with framing, analysis, and interpretation of student-level data frequently used to identify disparities and assess course-level and programmatic interventions. This guidance can inform analyses conducted by campus units such as departments and programs, but also across universities and the scientific community to enhance how we understand and address systemic inequity in STEM fields.

INTRODUCTION

Over the past 2 years, the world watched as police in the United States killed Black, Hispanic and Latinx Americans (Egbuonu, 2020); hate crimes toward Asians, Asian Americans, and Pacific Islanders surged (Gover ); and disproportionate numbers of Black, Hispanic and Latinx, and Native American people fell victim to the COVID-19 pandemic (Alcendor, 2020). Questions arose across countries, institutions, and communities about how these devastating and brutal events reflect underlying systemic inequities that exist across the globe (e.g., Collins ). Higher education in particular witnessed another reinvestment in diversity, equity, and inclusion (DEI) efforts with reinvigorated commitments to faculty cluster hires, curricular revisions, and changes to admissions criteria (Heinecke and Beach 2020; Peoples and Dillard, 2020). History has shown, however, that diversity initiatives can serve as mechanisms for institutions to send out signals of advancement without actually translating to systemic changes (Ahmed, 2012; Ray, 2019a,b; Johnson, 2020; Thomas 2020).

Likewise, science, technology, engineering, and mathematics (STEM) fields are focusing significant attention toward improving DEI in reflection on student enrollment and success data marred by gendered, racialized, and classed patterns (Asai, 2020; Griffin ; McGee, 2020a,b). Quantitative analyses in and focused on STEM education that rely on commonly available demographic variables (herein we consider gender, race, income, and parental education) are ever more present (Li ). The key issue we address here with STEM equity analyses is that demographic variables are often used automatically in “assessing student success” without situating these student characteristics in relation to the overlapping structural inequities that shape students’ experiences and academic performance.

Because of this tendency, there are often mismatches between the outcomes of equity analyses and how they are interpreted to inform institutional efforts that promote DEI in STEM. By inadequately grappling with the theoretical framing of students’ identities in relation to inequities, current methodological approaches guiding equity analyses can contribute to individualizing inequality (Byrd, 2021). As a consequence, educational initiatives are developed that prioritize changing students and not the STEM environments that perpetuate such inequalities.

STEM equity analyses are conducted by researchers in many different settings across academia—for example, STEM departments and programs do assessment work, institutional research offices conduct internal studies, and even individual instructors may be equipped to investigate their own courses. These analyses are also conducted by federal agencies, policy institutes, and corporations. That is, whether explicitly or implicitly described as “equity analyses” within a department, program, or organization, STEM equity analyses are part and parcel of everyday assessment and evaluation in STEM. Therefore, our intention here is to provide information that is applicable for researchers across the broader STEM community. Fostering an authentic discussion of issues raised by these questions will lead to better analyses that support justice for students who have historically been excluded from and continue to experience marginalization in STEM fields.

The purpose of this essay is twofold. We first discuss the importance of integrating critical perspectives in STEM equity research that relies on quantitative analyses. Then, through a series of critical questions, we aim to engender reflection and conversation with researchers and practitioners who do STEM equity research so that we all can use quantitative data more responsibly and accurately. This discussion was sparked by our own challenges in working with institutional data to better represent and understand the experiences and outcomes of marginalized and minoritized students in STEM through the interdisciplinary Sloan Equity and Inclusion in STEM Introductory Courses (SEISMIC) collaboration. We (the authors) represent the fields of chemistry and biology education, physics and astronomy, mathematics and mathematics education, educational psychology, and sociology and engage in quantitative research regularly in our positions as faculty, staff, and graduate students. We identify mostly as cis-heterosexual men and women, and all of us, except for one author, identify as white. Nonetheless, we all identify with and embrace the need for our fields and institutions to continually improve research and decision making aimed at tackling campus inequalities and injustices.

WHAT ARE CRITICAL APPROACHES TO RESEARCH?

As researchers, the lens through which we view the world has implications for the research questions we seek to explore and our methodological and analytical choices. Those who receive their academic training in STEM fields are generally socialized to adopt a postpositivist lens (Guba and Lincoln, 1994; Harding, 2006). Postpositivists assume that objectivity is achievable, where truths about the world remain credible if those seeking it follow the scientific method. This rationale works when observing scientific and mathematical phenomena like gravity or volcanic eruptions but breaks down when studying human experiences. Humans are situated in social contexts that have been shaped by historical events, structural forces, and interaction with other human beings. Thus, humans are more than objects of inquiry; humans and their experiences are a by-product of the structures in which they are embedded (Horkheimer, 1972; Bohman, 2005; Devetak, 2005).

In contrast to postpositivism, a critical lens assumes that what can be known about the world is socially constructed. Critical theorists separate themselves from traditional theorists across fields (Bohman, 2005). Critical theories explore how historical events and society have shaped present-day experiences and understandings of how the world functions (Horkheimer, 1972). So, whereas traditional theoretical approaches place the phenomenon of interest at the center of analysis, critical theorists seek to place societal contexts that shape a phenomenon as the focal point. Currently, several critical perspectives—that is, feminist, race, queer, disability, and decolonial theories—are accepted in the critical canon (Bell, 1995; hooks, 2000; Watson, 2005; Siebers, 2008; Mignolo, 2012). The common theme in these theoretical perspectives is their assertion that society has produced oppressive structures (e.g., patriarchy, racism, sexism, colonialism, and ableism) that harm those who are not white, cisgendered, male, heterosexual, able-bodied, wealthy, and Western individuals. As a result, there exists the need for society and those in it to interrogate how those oppressive ideologies show up in ourselves and connect to oppressive structures in the world around us. From there, we can begin to imagine new strategies for how to make life better for those who find themselves on the margins (Bohman, 2005; Devetak, 2005; hooks, 2000).

WHY ARE CRITICAL APPROACHES NEEDED IN STEM EQUITY RESEARCH?

Before we can discuss why a critical approach is needed, we must first define the goal of conducting STEM equity research. Over the past 2 years, we have seen increases in the numbers of people wanting to participate in research practices that focus on equity efforts in STEM, many of whom aim to increase the representation of marginalized and minoritized students in STEM fields for the betterment of our institutions and the U.S. economy (President’s Council of Advisors on Science and Technology, 2020). However, placing the United States and our institutions as the motivations for our work and not the students, who often find themselves pushed out of the fields they once found joy in (McGee, 2020a), is a striking concern we must all sit with. We argue that engaging in critical research is an effort to re-center the students, to create safer and healthier environments for them to pursue their passions.

STEM equity researchers must grapple with the historical events that have shaped what STEM fields look like today. STEM environments have led to scientific discoveries and innovation, but these environments also have a history of reproducing systemic inequities that harm individuals (Graves, 2001; Roberts, 2011; Wilder, 2013; Gholson, 2016; Joseph ; Saini, 2019; Cech and Waidzunas, 2021; Reinholz and Ridgway, 2021). Additionally, it is important to acknowledge that STEM fields have a history of conducting research, creating theories, and making measurements that primarily centered white, cisgendered, male, heterosexual, able-bodied, wealthy individuals (Harding, 2006). Consequently, the prioritization of individuals from privileged groups in STEM has produced research and policies that are susceptible to the structural inequities and personal biases that have historically harmed and excluded marginalized and minoritized communities.

Thus, we argue that the necessity of integrating critical approaches is directly linked to the need for revision in STEM education. Kiese Laymon, a Black Mississippian writer and professor, notes that “revision is a dynamic practice of revisitation, premised on ethically reimagining the ingredients, scope, and primary audience of one’s initial vision” (Laymon, 2021, para. 16). Laymon (2021) also argues that the current racial inequities we see in American society are a product of America’s failure to wrestle with and acknowledge its history of “anti-black terror” (para. 28). These arguments apply to STEM equity research, in which our inability to confront our institutions’ historical legacy of slavery, indigenous erasure, and exclusion of those who were not white, cisgendered, male, heterosexual, able-bodied, wealthy in our research and institutions is tied to the lack of representation we see in STEM fields. We assert that STEM equity researchers must commit to an act of revision in which we reflect on our motivations, historical and societal influences, and research processes in the hopes that we can imagine and work toward a more equitable future.

WHAT DO CRITICAL APPROACHES LOOK LIKE IN QUANTITATIVE RESEARCH?

For the purposes of this essay, we outline how to incorporate critical approaches in quantitative STEM equity research. Quantitative analyses are pivotal tools for examining DEI in STEM, but it is imperative to understand that researchers’ positions in relation to race, gender, socioeconomic status, where they are located within universities, academic training, and other characteristics can shape how they approach data and analyses. Researchers employing quantitative methods within a postpositivist framework tend to eschew acknowledging how the positionality of researchers impacts analytic decision making (Zuberi and Bonilla Silva, 2008; Gillborn ; López ). Additionally, centering individual demographic variables (e.g., race, gender, and ability) instead of structural inequities positions marginalized and minoritized students as solely responsible for their lack of representation in STEM fields. These unrecognized beliefs can lead to misinterpretations of people’s experiences, which in turn negatively affects campus decision making and policies (Sultana, 2007).

Critical quantitative (QuantCrit) approaches are helpful for those interested in studying ways to improve the experiences of marginalized and minoritized students from a quantitative standpoint. The foundational elements of QuantCrit are tied to critical race theory but are also aligned with other perspectives of the critical canon. Critical race theory explores where and how racism prevents people of color from accessing social and economic opportunities (Bell, 1995; Ladson-Billings, 2009). Critical race theorists are also interested in subverting deficit-framing projections by documenting the ways that people of color actively resist and cultivate joy despite racist structures (Devetak, 2005; Harper, 2010; Delgado and Stefancic, 2017). Relying on critical race theory, QuantCrit theory allows researchers to revise traditional notions of viewing relationships among racial groups as causal, instead seeing them as a reflection of historical and existing structural racism that differentially affects racial and ethnic groups (Zuberi, 2001).

Scholars who use QuantCrit: 1) grapple with the historical and present-day reality of racism; 2) recognize how the practice of naively using statistics can uphold white supremacy (e.g., achievement gaps); 3) interrogate how social categorizations such as race and ethnicity are varied, contested, and fluid over time (Omi and Winant, 2015) and how these shifts can impact analyses and interpretations; 4) integrate the voices of racially marginalized and minoritized individuals through qualitative and mixed-methods approaches to account for limitations in quantitative interpretations; and 5) embrace research methods, including quantitative approaches, to pursue equity goals that align with a social justice liberatory agenda (see Gillborn ). Recently, those who employ QuantCrit approaches have begun to extend these tenets beyond focusing solely on racism, incorporating how individuals are impacted by overlapping structural inequities (Crenshaw, 1989; Collins, 2000; Jang, 2018). Accordingly, using QuantCrit approaches provides researchers with the ability to use statistical practices as an analytical tool for improving the social conditions of marginalized and minoritized populations.

CRITICAL QUESTIONS: INTEGRATING QUANTCRIT APPROACHES IN STEM EQUITY ANALYSES

As STEM equity researchers who were originally trained in a postpositivist paradigm, we understand firsthand that learning how to be a QuantCrit researcher is difficult. As described, STEM fields have a history of pushing out marginalized and minoritized students (Gholson, 2016; Joseph ; McGee, 2020a,b; Cech and Waidzunas, 2021; Reinholz and Ridgway, 2021) and using harmful statistical approaches that contribute to negative perceptions of students (Zuberi, 2001; Zuberi and Bonilla-Silva, 2008). Instead of repeating harm, we as STEM education researchers can revise our paradigm to reflect more equity-centered approaches. Relying on Gutiérrez’s (2013) sociopolitical framework and critical perspectives (Crenshaw, 1989; Bohman, 2005; Cooper, 2018), we define equity as the process of reckoning with how historical events have shaped and continue to reinforce unequal power imbalances in a given context and actively working to dismantle those power imbalances so that society can restructure itself to better sustain and empower all. Importantly, this definition emphasizes continual adaptation as a goal of equity in order to accommodate changing perspectives of how we understand power, inequality, and injustice in our work.

Gutiérrez (2002) similarly argues that equity is a process, rather than a static goal, reflective of individual, institutional, and societal processes. As institutions and fields evolve through space (i.e., geographic location, institution, and classroom) and time, there will always be a need to reimagine new equitable practices. Therefore, here, we use the structure of questions rather than asserting definitive guidelines to follow, reifying our commitment to equity as a process with no universal, one-size-fits-all approach to equity analyses. These questions can assist researchers in adjusting their methodological approaches to the contexts of the educational environments in their studies as well as in delineating which data are collected and selected for analyses. To this end, imperfection and improvement represent the norm of equity analyses and can provide clarifications with each iteration. As a result, we advise that researchers use the following eight questions (summarized in Table 1) as self-reflective tools, rather than as an exhaustive list of questions to consider with every analysis.

TABLE 1.

Critical questions: A guide to integrating critical approaches in STEM equity quantitative analyses

Question	Recommendation
How does lived experience affect how one approaches research?	Before beginning the research process, researchers should reflect on how their beliefs about the world, personal background, characteristics, and academic training influence their approaches to the study.
What theoretical assumptions are present in conceptualizations of equity practices?	Researchers should think about what equity model they are using for their analyses. For example, does equity mean students from various backgrounds are performing the same academically? Relying on Gutiérrez (2013), we argue that conceptualizations of equity should embrace how historical events and structural experiences shape present-day students’ experience, which means that strategies for empowerment will vary depending on the group of interest, location, and time period.
What analytical and interpretive choices can be made to focus on excellence?	Historically, achievement gaps have contributed to negative perceptions of students who come from minoritized backgrounds. We advocate for researchers to focus their efforts toward exploring where and how marginalized and minoritized students are excelling despite structural inequities and using that information as a guide for advancing equity.
What theoretical linkages exist between the constructs and demographic variables of interest?	Many of the constructs used in educational research were created using samples of students who are mostly white, cisgendered, male, heterosexual, able-bodied, wealthy students men from privileged institutions. As a result, the relationship between popular constructs of interest and minoritized students often include negative stereotypes. We encourage researchers to reflect on the constructs in a study and whether those constructs adequately reflect the lived experiences of the target population.
What should be considered when using standardized test (ACT/SAT) scores as a metric for “prior preparation”?	Standardized tests (ACT/SAT) have a history of being used to support racial discrimination and subordination but are commonly used in equity research. High school GPA and college course work are better indicators of a students’ prior academic preparation, especially for marginalized and minoritized students. Although subject to structural inequities, we recommend that these metrics be used instead.
What measures capture structural inequalities that exist in STEM higher education?	STEM equity researchers commonly use individual-level variables (race, gender, ability, etc.) to understand societal inequities. Although these variables capture variations that exist across groups, they do not capture the underlying mechanisms that reflect inequities. We recommend that researchers additionally incorporate structural variables into their analyses, such as campus and classroom climate measures, policies, and institutional characteristics (e.g., selectivity).
How do changes in institutional categories for demographic variables over time affect analyses?	When working with institutional data, researchers should explore whether and how institutional definitions for demographic characteristics have changed over time.
Are quantitative analyses the best tools for answering the proposed research questions?	Quantitative analyses do an adequate amount of explaining student experiences at the macro level. However, qualitative and mixed-methods research can sometimes better uncover the underlying mechanisms that contribute to student experiences. We recommend that researchers reflect on the goals of their work to see if quantitative analyses are appropriate.

How Does Lived Experience Affect How One Approaches Research?

The lenses through which researchers view the world are influenced by their lived experiences accumulated through a multitude of interpersonal interactions and exposure to and engagement with different research perspectives, methodologies, and theories. When using quantitative approaches, researchers often implicitly regard themselves as objective observers, with numbers viewed as neutral (Guba and Lincoln, 1994). This practice becomes especially concerning in social and educational research, as scholars have uncovered how numbers and data have been used to reinforce social inequities. For instance, Zuberi and Bonilla-Silva (2008) highlight how “statistical analysis was developed to explain the racial inferiority of colonial and second-class citizens in the new imperial era” (p. 5). These harmful statistical practices still permeate education research today, with researchers studying achievement gaps in higher education without adequate explanation of structural barriers (Ladson-Billings, 2006; Gutiérrez, 2008). It is important to realize that researchers are human beings who are situated in societal contexts that privilege some groups over others (Bohman, 2005). As a result, researchers are prone to have conscious and unconscious biases that influence the development of research questions and decision-making practices for measurement and analyses (Harding, 1992).

A common practice in qualitative research is to write positionality statements (Secules ). In these statements, researchers discuss how their backgrounds and experiences impacted their academic trajectories and relationships to research. Harding (1992) argues that being upfront about one’s biases, values, and experiences reflects “strong objectivity,” because it allows the audience to understand how a researcher’s lived experience and personal biases might impact the study. Given that positionality statements are not a common practice in quantitative research, we understand that researchers might be hesitant to include them in their work. However, a core component of being a critical scholar is constantly reflecting on how society influences one’s view of the world and in return how one chooses to do research (Guba and Lincoln, 1994). By grappling with their positionality, researchers can better understand the strengths and limitations of their lenses and thus their work. Before and throughout the research process, we advise that researchers take time to reflect on, and perhaps write about, their positionality. Additionally, researchers should assess whether their lived experiences or academic training prepared them to conduct their research.

What Theoretical Assumptions Are Present in Conceptualizations of Equity Practices?

Before analyzing data, researchers should first assess their definitions of equity. Rodriguez outlined, for example, three different equity models using the language of parity, fairness, and individuality. Equity models based on parity focus on getting minoritized students to obtain similar levels of academic success as majority groups members, and equity models based on fairness aim to get different groups of students to achieve similar levels of progress on tasks and assignments (Rodriguez ). An inherent problem in these models is the assumption that academic success is contingent upon the behaviors and beliefs of majority groups. The equity of parity model also does not account for the historical trauma and discrimination that has hindered marginalized and minoritized students’ academic success in STEM programs. Individual students have their own sets of privileges and disadvantages that influence their needs and experiences in STEM learning environments. Therefore, STEM equity models should accept that conceptualizations of equity will vary across groups and situations, and not neatly align with cut-and-dried societal hierarchies.

Rodriguez advocate for researchers to use equity models of individuals in which researchers attend to the factors that have harmed marginalized and minoritized students’ access to STEM fields and develop conceptualizations of success for each individual group. Going further, Gutiérrez (2013) argues that focusing on individual groups is not enough, rather that equitable practices within STEM contexts must contend with the ways that identity and power manifest in our courses and institutions. First, Gutiérrez (2013) describes how equity for an identity group can fluctuate depending on the context and time frame. As a result, researchers should unpack their justifications for focusing on a specific identity group when conceptualizing equity. For example, researchers will explain that they are studying an identity group (e.g., women or students of color) due to their lack of representation in STEM fields. However, each institution, department, and classroom has its own set of historical origins and structural factors that have shaped the present-day experiences for each identity group. Equity models should describe and embrace these variations (Hancock, 2007).

Additionally, Gutiérrez (2013) comments on how STEM skill sets (e.g., quantitative literacy) are perceived as necessary tools for professional and personal development, creating a system in which individuals who fail to adopt these skill sets are rendered less valuable. These ideological assumptions shape STEM learning spaces as sociopolitical institutions wherein marginalized and minoritized students are blamed for their lack of representation without addressing systemic inequities. Thus, when focusing on ways to center equity in STEM analyses, we suggest that researchers avoid using language that solely focuses on marginalized and minoritized students in relation to their academic outcomes and more so on how their educational experiences are shaped by history, power, and context.

What Analytical and Interpretive Choices Can Be Made to Focus on Excellence?

A popular practice in STEM equity research is to observe the achievement gaps existing between majority and marginalized and minoritized students (Gouvea, 2021). A wealth of research explores how students who belong to underserved racial, gender, ability, and socioeconomic groups underperform academically in comparison to their privileged counterparts (e.g., Bastedo and Jaquette, 2011; Matz ; Whitcomb and Singh, 2021). Focusing solely on gaps is harmful, because it centers students’ identities as the reasons behind their academic failures. Additionally, the research on achievement gaps over the years has not substantially improved the experiences of marginalized and minoritized students in STEM; in fact, research has shown that overreliance on the documentation of these gaps has contributed to negative societal constructions of the academic abilities of students from minoritized backgrounds (Gutiérrez, 2008; Martin, 2012). Embracing critical approaches in STEM equity research necessitates that researchers use proactive approaches wherein efforts are pushed toward addressing what institutions can do to better support students.

Rather than framing analyses with gaps, those conducting equity analyses should focus on how different factors positively relate to advancements, gains, and excellence of students (Gutiérrez, 2008; Harper, 2010). For example, research shows that LGBTQ+ students experience fewer stressors when they attend colleges that have academic studies, policies, and student clubs supportive of LGBTQ+ individuals (Woodford ). By shifting the onus to institutional components, Woodford showcase how university programs and policies are directly tied to the success of LGBTQ+ students. Practitioners can use these findings to create supportive institutional and classroom environments for LGBTQ+ students.

We further caution against excellence-based approaches that solely center grades or degree attainment. Despite increases in STEM degrees conferred to racially minoritized students, there still exists a lack of representation in STEM fields (Fry ). Furthermore, the financial and psychological burdens that racially marginalized and minoritized students report while enrolled in and after college suggests that “success” postgraduation is not equitable across all groups (Keels ; Davis ; McGee, 2020b). Students of color often conceptualize “success” as tied to their ability to give back to their communities, which is different from traditional conceptualizations of success (McGee and Martin, 2011; Pérez Huber ; Lopez, 2020; McGee, 2020a). As a result, we recommend that researchers adopt definitions of excellence within STEM contexts based on the conceptualizations of their population of interest and then use those definitions in analyses (Pérez Huber ; Weatherton and Schussler, 2021).

What Theoretical Linkages Exist between the Constructs and Demographic Variables of Interest?

Too often, quantitative STEM equity analyses are conducted with a “kitchen sink” approach in which full combinatorics are used to study intersections of student constructed identities. Including several independent demographic variables without adequately accounting for past research or theoretical linkages among them hinders the interpretation of research findings. How and why these demographic variables are used in analyses impacts conversations about what inequities look like, for whom, and what should be done.

Recently, scholars have begun to discuss the need for a “race re-imaging” wherein commonly used measures such as motivation or institutional support are re-evaluated and adapted to fit the lived experiences of racially marginalized and minoritized experiences historically left out of educational psychology research (DeCuir-Gunby and Schutz, 2014; Lopez, 2020; Matthews and López, 2020). Research indicates that racially marginalized and minoritized students’ value in STEM is strengthened through their ability to understand how STEM educational skill sets can uplift their communities (McGee and Bentley, 2017; Gray ), countering the individualistic culture of STEM learning environments (Battey and Leyva, 2016; Carter, 2017). Therefore, STEM utility and motivational measures that ignore social justice and community engagement may miss out on the ways that STEM and racial identities intersect (McGee, 2020a; Miller-Cotto and Lewis, 2020). As STEM equity researchers, we can apply these ideals to questions of pre-existing assumptions we may hold about the relationships between our constructs of interest and different demographic variables (i.e., gender, socioeconomic status, ability, and sexuality) selected for our studies before running analyses. Then, if deficit-based theoretical linkages emerge, we recommend researchers find outside studies promoting strength-based approaches or adopt qualitative or mixed approaches that can better speak to the associations between the demographic variables and constructs of interest.

What Should Be Considered when Using Standardized Test Scores as a Metric for “Prior Preparation”?

Standardized test scores (ACT/Scholastic Aptitude Test [SAT]) must be incorporated cautiously considering who is and has been most likely to do well on them given structural inequalities that privilege certain families over others (Rothstein, 2004; Soares, 2007; Zwick, 2013; Carnevale ). As researchers focused on equity, we must acknowledge the racist origins of standardized assessments. In the early 1900s, standardized assessments were intentionally used by eugenicists as justifications for racial purity in American educational systems (Lemann, 2000; Harris ; Soares, 2007). Today, standardized assessments are still used in admissions decisions and placement into undergraduate STEM courses, even though research has shown that they are weak and inadequate predictors of college retention for racially minoritized students (Sedlacek, 2004). Although many institutions have either modified admissions policies to be test optional or completely eliminated standardized tests in admissions review due to the impact of the COVID-19 pandemic, how long institutions will continue with such policies and what possible alternative assessments may be used in place of ACT and SAT test scores remains to be seen. Further, students who come from economically privileged families have access to high schools and test preparation resources that increase their chances of doing well on standardized assessments. The economic privileges continue once these students enter higher education (Borg ; Carnevale ). As a result, causal linkages between standardized assessments and degree attainment generally fail to account for wealth as a confounding variable. Therefore, we encourage the use of other metrics to capture the academic preparation of students.

College course grades and high school grade point average (GPA), while also imperfect measures, are stronger predictors for student adjustment and success in college over standardized test scores (Byrd ; Koester ; Galla ). Unlike standardized assessment scores, a student’s high school course grades and college course work better reflect the level of mastery for a given subject. Additionally, researchers could consider using Advancement Placement (AP) scores. The AP program provides high school students with the chance to engage in college-level curricula (Kolluri, 2018). Research has shown that passing AP tests is related to positive college outcomes across students from different racial and socioeconomic backgrounds (Dougherty ; Fischer ). It is important to note that even these metrics are not perfect indicators of academic preparation, given the intersecting inequalities that exist in K–12 educational systems (see Lewis and Diamond, 2015). Therefore, researchers should account for the current limitations that exist in how we assess students’ prior academic preparation.

What Measures Capture Structural Inequalities That Exist in STEM Higher Education?

Equity analyses that only use individual-level variables provide great insight into how academic outcomes vary across different social groups. However, interpretations that come from these types of analyses often place sole responsibility on minoritized and marginalized students to persevere through systemic barriers (McGee, 2020a). Using complex multilevel models, researchers have assessed the impacts of various structural components on student outcomes, such as campus and classroom climate, policies, and institutional characteristics (e.g., selectivity and public vs. private status; Espinosa, 2011; Leath and Chavous, 2018; Ohland et al., 2018). For example, Espinosa (2011) found that women of color who attended private colleges were more likely than their peers enrolled at public institutions to persist in their STEM programs. Espinosa (2011) attributes the positive effect of private institutions to the large amounts of educational resources available that counteract a lack of academic preparation among women of color. Espinosa (2011) further showcases that experiences for women of color vary based on the STEM contexts in which they are situated. In contrast, Leath and Chavous (2018) show that Black women enjoy college less when they feel like they must conceal their racial and ethnic identity. Leath and Chavous (2018) demonstrate how tumultuous racial climates contribute to Black women’s college experiences. These studies allow researchers to gain insight into the underlying mechanisms and structural components (e.g., type of college and racist campus climate) that contribute to student experiences, persistence, and success. Also, these researchers illustrate a story in which the institution is held accountable for variations in student outcomes (Hancock, 2007).

There are, however, limitations to these approaches. For one, institutional climate measures may aggregate students’ perceptions about how well students from different backgrounds get along with one another. Although these measures are reflective of structural components, they still rely on individual-level perceptions and do not generally account for different conceptualizations of climate across social groups. Second, multilevel models require large sample sizes at the individual level to maintain statistical power (Snijders, 2005), but researchers attempting to collect information from a diverse set of participants are often blocked by a lack of financial resources and time (Hancock, 2007). As a result, researchers may be forced to make difficult analytical decisions like aggregating multiple social groups together (e.g., combining non-white students) that gloss over the variation between different groups of students and hinder our understanding of how structural inequalities on campus, and in STEM programs specifically, differentially impact their experiences and outcomes. Additionally, because STEM equity researchers often find themselves working with institutional data, individual variables may be the only ones available, and as we describe in more detail later, may not include documentation about how these data were collected, which can influence modeling strategies and subsequent interpretations.

As a result, we emphasize that models based only on individual-level variables (e.g., race, gender, and ability) can only suggest variations across existing groups; they are not explanations for the underlying mechanisms that influence these variations. Ultimately, much work remains in delineating the best practices for integrating structural features into analyses and appropriately contextualizing them within STEM equity research. In the meantime, we recommend that researchers do their best to incorporate structural components in analyses wherever possible.

How Do Changes in Institutional Categories for Demographic Variables over Time Affect Analyses?

When working with institutional data, researchers may need to track how their institutions’ social categories have changed over time (Viano and Baker, 2020; Byrd, 2021). Categorizations like gender, race and ethnicity, income, and parental education are not fixed; these categories fluctuate over time, even if slowly. As a broad example, the race and ethnicity categories on the U.S. census that inform data collection across society have changed with every census administration (Brown, 2020). At one time, for example, Irish immigrants were not viewed as “white” due to a few factors including their socioeconomic position and religious beliefs, but as the Irish gained economic mobility in a deepening Jim Crow era, their ascension to whiteness was solidified in the United States (Omi and Winant, 2015). Here we see that race is not static, but a by-product of social and political change.

In addition to race and ethnicity, gender categorizations in the United States have also evolved, with social surveys moving beyond binary conceptualizations and shifting toward more gender-inclusive (i.e., transgender, gender non-conforming, nonbinary) categories (Westbrook and Saperstein, 2015). Indeed, such variation has always existed (D’Ignazio and Klein, 2020), and it is important to note that individual perceptions of social categorizations are also subject to change (Freeman ). Similarly, sometimes the same information is collected about students in multiple contexts (e.g., when both the financial aid and registrar’s office have information about students’ first- or continuing-generation status). Identifying areas of discordance from different data-collection mechanisms over time can more properly contextualize analyses, particularly when merging multiple data sets for the same students.

Consequently, when working with secondary data sources, we recommend that researchers seek to obtain information about how social categorizations were solicited and defined as well as how they may have changed over time. Including this information in studies, even if only as supplemental material, will help to produce research that is better contextualized. Researchers should also reach out to campus offices that maintain and analyze student-level data for additional student information that may not be included in existing data sets to improve clarity about how groups are constructed and how this might influence analyses and interpretations.

Are Quantitative Analyses the Best Tools for Answering the Proposed Research Questions?

Using qualitative and mixed-methods approaches, scholars have provided in-depth commentary on the ways that systemic inequities have shaped marginalized and minoritized students’ experiences in STEM contexts (McGee and Bentley, 2017; Allaire, 2019). At the same time, the rise of big data has encouraged educational systems to examine the experiences of marginalized and minoritized students at the macro level (Daniel, 2019). Although quantitative approaches offer unique benefits, some research questions require alternative approaches to better capture the lived experiences of minoritized students (Covarrubias, 2011). For example, Jack (2019) studied the experiences of low-income students at elite institutions, showing that those who graduated from private high schools were able to navigate elite institutions better than their low-income peers who attended public high schools in their communities. A common practice in educational research tends to clump the experiences of low-income students together when studying inequity. Jack (2019) demonstrates how qualitative research has the power to capture variations within groups not easily noticeable when groups are combined in quantitative analyses. Therefore, before conducting analyses, we recommend that researchers first identify the main goals of a research project and assess whether quantitative analyses are most applicable and viable given the data available or to be collected, regardless of sample size.

CONCLUSION

Although higher education has contributed to the advancement of society, our institutions have also participated in creating and reproducing systemic inequities (Patton, 2016). Our institutions, as well as the research community, can and should play a role in making the experiences of all students more equitable by first examining, with the students themselves, what those experiences are that can inform campus decision making. The misuse of quantitative data in STEM equity analyses can, even when unintended, reinforce deficit interpretations about marginalized and minoritized students and mask the role of systemic inequities. Integrating critical approaches in STEM equity analyses can provide insight into how institutions bear responsibility for the lack of diversity, representation, and differential experiences in STEM fields reflecting an unequal opportunity structure on our campuses.

In this essay, we aimed to inspire those conducting STEM equity research from a quantitative perspective to commit to an act of revision (Laymon, 2021). As researchers in STEM education, we understand that career success is often dependent upon one’s ability to adopt beliefs that research and numbers are objective. We also know that researchers are encouraged to search for “silver bullets” or universal approaches in their work. In fact, we still fail at upholding all of the recommendations we have offered. However, understanding the value that statistical practices have in equity policy initiatives, we are committed to working through present-day limitations that come with the quantification of human experiences. By being upfront where our work falls short, we get closer to discovering new analytical approaches that can be used for liberatory purposes. Finally, we hope to contribute to a critical discourse and prompt reflection in an effort to make a meaningful impact that ultimately promotes equity and inclusion on our campuses and in STEM fields.