Defining success in graduate school.

We agree with the author that a quantitative analysis of the predictive nature of the metrics used in graduate student admissions is a worthy pursuit and value the sincere intentions behind the UCSF Tetrad study. However, these types of analyses would benefit from the same rigorous approaches that we employ in our other research endeavors. As UCSF Tetrad graduates with diverse careers in academia, medicine, industry, and publishing, we hope that the definition of success in graduate school can be as thoughtfully and scientifically examined as the measurements used to select the next young people to follow in our footsteps.

To the Editor:

As graduates of the University of California–San Francisco (UCSF) Tetrad program from the matriculating classes of 1992–1996, we read with interest a recently published Perspective entitled “How Should We Be Selecting Our Graduate Students?,” written by fellow alumnus and classmate Orion Weiner (Weiner, 2014). The author, who is now a member of the UCSF Tetrad faculty and cochair of graduate admissions, reported that success in this graduate program over the past 20 years correlated with years of undergraduate research experience and Graduate Record Examination (GRE) subject scores but did not correlate with other commonly used admissions metrics, such as undergraduate grades, general GRE scores, or ranking of the undergraduate institution. We applaud the author for taking on the important challenge of examining predictors of success in graduate school in the life sciences. As alumni of the program, however, we wish to respond to a number of issues, including the design and execution of the study and the implications of its conclusions. Most importantly, we propose a new, broader definition of success in graduate school. Our aim is to promote a discussion about meaningful, reliable, and scientific ways to define, analyze, and evaluate success in bioscience graduate programs.

PROBLEMS WITH USING “SUCCESS” AS AN OUTCOME MEASURE

Graduate programs in the biosciences have used essentially the same metrics to evaluate applicants to graduate school for decades, and we fully support the author's intention to evaluate whether or not these metrics continue to serve as predictors of student success. However, establishing unbiased criteria for graduate student success is a challenging endeavor, as previously recognized (Hartnett and Willingham, 1980; Meade and Fetzer, 2009). A number of attempts have been made to establish and evaluate objective (grades, passing qualifying exams, time to doctorate, number of publications) and subjective (professors’ ratings of dissertations or predictions of future success) criteria (Stricker and Rock, 1993; Sternberg and Williams, 1997; Burmeister ). In this study of the UCSF Tetrad program, past students were classified as “successful” or “underperforming” by a group of 30 current faculty members. Neither the criteria by which students were assigned to these two categories nor the career stage at which the assessment was made were reported. Thus, “successful” students could be those who published multiple first-author papers, were remembered as having worked long hours in the lab, got along well with others in the lab, or went on to academic careers at high-profile institutions. A student identified as “underperforming” might be one who took a long time to graduate, switched labs, was perceived to have a combative personality, or left academia after graduation.

Thus, in the UCSF Tetrad study, the relationship between the subjective and undefined dependent variable (student success) and the quantitative independent variables (admissions metrics) cannot be reliably interpreted. For example, it could be—as suggested by the author—that students who did more research before graduate school were more likely to know what getting a graduate degree would entail and were therefore more likely to be considered successful. However, “successful” students might simply be those who met cultural expectations for graduate students—for example, they needed less instruction in bench skills, were more likely to listen to authority, or came from academic families or privileged economic backgrounds. These examples illustrate both how the binary outcome of “successful or underperforming” simplifies the assessment of a complex process and, furthermore, leaves unaddressed the possible transformative or educational value of the graduate program.

CONFOUNDING FACTORS

The UCSF Tetrad study is a post hoc analysis that relies on the recollections of 30 current faculty members, an approach that excludes the many faculty members who left over the past 20 years. Relying on individual memories is not only inaccurate (as in the well-documented phenomenon of recall bias [ Pannucci and Wilkins, 2010]), but also limits the scope of the analysis to students who had personalities strong enough to make them memorable to multiple faculty members or who remained professionally connected to faculty after graduation. In fact, faculty-based evaluations can be subject to a host of unconscious biases, as demonstrated recently in a high-profile study of gender bias among science faculty (Moss-Racusin ). Any of these biases may have been inadvertently in play here, and eliminating their effect on the results would require a different study design.

Students for whom there was no consensus among faculty regarding their assignment as either “successful” or “underperforming” were excluded from the analysis. While several hundred students must have gone through the UCSF Tetrad program during the past 20 years, only 52 were included in the analysis. It seems likely that analyzing only those students on whom there was complete agreement—while failing to provide an external definition of success—would have a flattening effect and allow only strongly conservative or traditional definitions of success (and underperformance) to operate within the study.

The author's personal and professional relationships with the students being evaluated, past or present, introduces the possibility of interviewer bias (Pannucci and Wilkins, 2010). Furthermore, Dr. Weiner himself, as well as a number of other current faculty members, could have been included in the student data set under analysis, creating the appearance of a conflict of interest.

IMPLICATIONS OF RECOMMENDING TWO YEARS PRIOR RESEARCH EXPERIENCE

The author concludes that of all admissions metrics used, only the number of years of prior research experience and subject GREs were strong discriminators between successful or underperforming students. Based on this finding, he argues that admitting only students with two years of prior research experience would enrich for successful students. However, many young people who are excited about research and who have the ability to contribute have not had the opportunity to do two full years of research before matriculation, because they did not attend a university with a research program, needed to work in a nonresearch position during school and summers, lacked an early awareness of undergraduate research opportunities, or changed course from other pre-professional studies. As a result, taking this report as a prescription for improved student performance might serve to further disenfranchise applicants already disadvantaged due to gender or economic, racial, or family educational background. Thus, improving the availability of opportunities for inquiry-based lab experiences during and after undergraduate education should go hand in hand with any admissions requirement for multiple years of research experience, lest we lose these students to postgraduate education and to the research enterprise as a whole.

As the author correctly notes, the results of this particular study may not apply to all institutions, and we hope that each graduate program will clearly define student outcome metrics before moving forward with a similar study. For graduate programs interested in producing leaders—not only in the advancement of academic science but also in the entrepreneurial world, biotechnology, the pharmaceutical industry, science policy, intellectual property law, liberal arts education, scientific communication, or in recruiting the next generation of scientists—students who are willing to take on a risky project, work on a team rather than an independent project, or pursue teaching and outreach activities in addition to their thesis projects may in fact be the best candidates. Whether or not these types of students would be identified as successful in the UCSF Tetrad study is not clear, but this example illustrates that admissions metrics should not be evaluated for their predictive value until the outcome measure of success is clearly defined.

TOWARD A NEW AND BROADER DEFINITION OF GRADUATE STUDENT SUCCESS

As suggested above, graduate programs in the biosciences may need to reassess their definition of graduate student success, as training excellent future academic faculty can no longer be the sole guiding principle. The stated mission of UCSF includes “improving health worldwide through innovation,” as well as “attracting and supporting the most talented and diverse trainees” (UCSF, 2014 [emphasis added]); the UCSF Tetrad program surely supports this goal. Currently, graduate programs in general are doing a questionable job of preparing students for many of the careers that will be available to them when they graduate. Longtime UCSF Tetrad faculty member and current vice chancellor for research Keith Yamamoto has argued that trainees spend too long trying to get the faculty jobs that account for only 15% of the careers of life sciences PhDs (Yamamoto, 2014). In his view, graduate programs should value a range of careers and train students to pursue them. This impetus to broaden the training focus of graduate programs is shared by funding agencies, judging by the recent announcement of the National Institute of Health's Broadening Experiences in Scientific Training initiative (NIH, 2013) and the National Science Foundation's research traineeships program (NSF, 2013).

All of these issues may be considered if we are to develop valid and meaningful measures of graduate school success. In our view, a successful graduate student is one who goes on to contribute relevant scientific expertise in his or her chosen context, whether that be academia, law, industry, publishing, business, teaching, politics, or another career. Contributions could be measured in a longitudinal study in a number of ways, including the self-reporting of graduates, evaluation by outside experts, or, in economic terms, as a return on investment as previously done with teacher development (Silverstein ). While “contribution” to a chosen field after graduation is still a subjective metric, and these methods of measuring it may be complex, it has the potential to be less biased, more meaningful, and more inclusive than one solely based on the recollections of faculty members.

PRIVACY CONCERNS

We would like to draw attention to one final issue regarding publication of the UCSF Tetrad study. We believe that the article would have benefited from a report of the precautions taken to ensure 1) the future confidentiality of the success rankings generated in the study, 2) the exemption by the UCSF Committee on Human Research office obtained by Dr. Weiner (Weiner and UCSF Committee on Human Research, personal communication), and 3) adherence to the standards of the Family Educational Rights and Privacy Act during the access of student records. While Molecular Biology of the Cell does not specifically instruct authors to describe institutional review board exemption or approval (), a growing number of journals do (Atlas, 2003), and the complex nature of the UCSF Tetrad graduate program study highlights the need for scientific journals to promote the rights of human subjects of experiments reported in their pages.

SUMMARY

We agree with the author that a quantitative analysis of the predictive nature of the metrics used in graduate student admissions is a worthy pursuit and value the sincere intentions behind the UCSF Tetrad study. However, these types of analyses would benefit from the same rigorous approaches that we employ in our other research endeavors. As UCSF Tetrad graduates with diverse careers in academia, medicine, industry, and publishing, we hope that the definition of success in graduate school can be as thoughtfully and scientifically examined as the measurements used to select the next young people to follow in our footsteps.