Facilitating Growth through Frustration: Using Genomics Research in a Course-Based Undergraduate Research Experience.

A hallmark of the research experience is encountering difficulty and working through those challenges to achieve success. This ability is essential to being a successful scientist, but replicating such challenges in a teaching setting can be difficult. The Genomics Education Partnership (GEP) is a consortium of faculty who engage their students in a genomics Course-Based Undergraduate Research Experience (CURE). Students participate in genome annotation, generating gene models using multiple lines of experimental evidence. Our observations suggested that the students' learning experience is continuous and recursive, frequently beginning with frustration but eventually leading to success as they come up with defendable gene models. In order to explore our "formative frustration" hypothesis, we gathered data from faculty via a survey, and from students via both a general survey and a set of student focus groups. Upon analyzing these data, we found that all three datasets mentioned frustration and struggle, as well as learning and better understanding of the scientific process. Bioinformatics projects are particularly well suited to the process of iteration and refinement because iterations can be performed quickly and are inexpensive in both time and money. Based on these findings, we suggest that a dynamic of "formative frustration" is an important aspect for a successful CURE. ©2020 Author(s). Published by the American Society for Microbiology.

INTRODUCTION

In How We Think, the philosopher John Dewey (1) said,

…(F)ailure is not mere failure. It is instructive. The person who really thinks learns quite as much from his failures as from his successes. For a failure indicates to the person whose thinking has been involved in it, and who has not come to it by mere blind chance, what further observations should be made. It suggests to him what modifications should be introduced in the hypothesis upon which he has been operating.

While Dewey’s quote highlights the benefits of failure, the potential costs of failure in the learning process are also important. Csikszentmihalyi said, “Enjoyment appears at the boundary between boredom and anxiety, when the challenges are just balanced with the person’s capacity to act” (2). Multiple authors have found frustration occurs when a stated goal is not (initially) achieved, and that it may increase the vigor of the response (3–5). Fast knowledge acquisition is often inversely proportional to long-term retention (6). If the long-term goal is flexible internal access to material, then “desirable difficulties” that slow encounters with new material are advantageous (7). Student attitudes and attributes also influence the ability to work through challenges. Epistemic affects, the expression of feelings and emotions associated with learning, also promote engagement (8, 9). Proficiency is accompanied by development of comfort with failure and a desire to succeed (10). Two major patterns are exhibited by students when encountering challenges: a maladaptive “helpless” response, characterized by avoidance of challenging tasks and deterioration in performance; and an adaptive “mastery-oriented” response, characterized by continuing to strive despite failures (11). These ideas parallel two mindsets: a “fixed” mindset, where abilities are perceived to be innate and unchangeable, compared with a “growth” mindset, where abilities are developed through practice (12). The benefits of frustration have been explored for students in other scientific disciplines but go by different names, including “productive failure” for high school physics (13, 14) and “productive struggle” for middle school mathematics (15). Despite evidence in the literature for a positive role for frustration in learning, no one to our knowledge has investigated this idea in a course-based undergraduate research experience (CURE) setting.

Engagement in authentic research is seen as a valuable, if not essential, part of science education. For undergraduates, participation in research promotes retention in science (16, 17), interest in pursuing higher education in STEM fields (18), and higher earning potential after graduation (19). Course-based undergraduate research experiences can overcome limitations associated with independent research by embedding research experiences into coursework, increasing the opportunities for students (20–23) and enabling more diverse students to engage in research (20, 24). For many CUREs, however, there remain significant financial, personnel, space, and time constraints that limit the number of students served. In contrast, a computer-based bioinformatics or genomics CURE, which requires relatively few resources for implementation (namely, a computer with an Internet connection), provides scalable opportunities for many students to become involved in research.

The Genomics Education Partnership (GEP, https://gep.wustl.edu), a consortium of more than 125 faculty members, involves undergraduates in genomics research (23, 25–27). Thousands of student investigators have annotated regions of previously unstudied Drosophila genomes, which are then used in meta-analyses to address a particular scientific question (28, 29). A similar approach can be used to investigate any organism for which an assembled genome exists (30), allowing undergraduates to contribute meaningfully to scientific knowledge (31). Thus, GEP provides a dynamic approach to resolving the dual challenges of engaging large numbers of students in research and analyzing large amounts of data (32). In the GEP workflow, described previously (26–29), students must test their proposed gene models. As often happens, the first model fails, requiring students to re-evaluate the data and refine their model. Such challenges expose students to some of the frustrations associated with research, but they occur in a structured, supportive environment. An important feature, however, is that students are able to repeat their experiments and learn from their mistakes relatively quickly.

Having observed student work in GEP projects, and in consultation with the literature, we developed the hypothesis that frustration aids in student learning in this CURE setting. To explore the idea that struggle is important for positive student outcomes, we designed and administered a “formative failure” survey to determine whether GEP faculty agreed that frustration was formative for student learning. We also evaluated qualitative feedback from students involved to better understand their experiences. A common theme emerged from these different data sources—students often struggle with gene annotation yet find value in the struggle. Our conclusions are robust because the findings were replicated in dozens of institutions with data gathered from multiple sources (student and faculty surveys and student focus groups) involving large numbers of students and faculty over time (33). Analyses of each of these datasets support our conclusion that formative frustration and iteration to achieve satisfactory outcomes contribute to student learning.

MATERIALS AND METHODS

Participants

Schools participating in the GEP are described in Appendix 1. The demographics of the students participating in the current study are also described in Appendix 1.

Faculty formative failure survey and data analysis

A faculty survey (Appendix 2) was developed by a small group of GEP faculty (summer 2017) to gather data to support or refute the hypothesis that students benefit from experience with the iterative rounds of investigation and analysis required to solve problems. All GEP faculty were invited to participate in the survey (fall 2017). Anonymity was maintained by removing identifiers before data analysis. The comments were assessed qualitatively via inductive analysis using NVivo software to minimize rater bias. Comments were initially open-coded to capture emergent themes (34, 35). The initial codes were arranged into categories by reiteratively comparing data within and between codes (35, 36) to create a codebook based on groupings. Because the comments were open-ended, each faculty member had the opportunity to include more than one thought in response to a question. Thus the number of comments was greater than the number of participants.

Analysis of student survey comments

Each term, students are asked to take a pre-course survey and knowledge quiz before using GEP materials, followed by a post-course survey and quiz. Results from these two years of data (2015–2017) were similar to those obtained previously (26, 27). Detailed analysis of the quiz and survey results for this population will be reported elsewhere (Lopatto et al., in preparation). Surveys were accessed through the GEP website, and students could opt out of any or all questions. Informed consent was obtained by asking students to read a preamble describing the quiz or survey, then clicking to consent and advance to the questions. Confidentiality was maintained, making it impossible to identify the student from the output. Responses were collected by Washington University in St. Louis Biology Department staff on a server distinct from the server students use for their annotation projects.

Comments were solicited on the post-course survey with a broad, open-ended prompt, “Please comment on the effort made to integrate research and teaching in genomics in this course. What were the strengths and weaknesses? What was of special value to you? Should this effort be continued?” In addition to the analysis of emergent themes by NVivo, we also used NVivo for quantitative keyword analysis, specifically looking for words and stems that indicated frustration, learning, and understanding of the scientific process. These results were then evaluated by human readers to determine the context in which these words appeared. Similar to the faculty survey responses, each student could make comments in more than one area, and so the number of comments exceeds the number of overall responses.

Analysis of student focus groups

Student focus groups were conducted in spring 2016 to investigate student attitudes about the GEP research project. Groups contained six to eight participants working with a facilitator using a common set of questions (Appendix 3). A consent document similar to that used for online assessments was provided. Focus groups were conducted at six schools: one research university, two state universities, and three four-year colleges; two were minority-serving institutions. Two focus groups were conducted at one institution. In each case, the school had a trained facilitator and appropriate IRB approval.

RESULTS

The GEP CURE framework

The GEP was formed to advance understanding of genome organization, structure, and function, as well as to provide research opportunities for undergraduates. Here we examine the concept of “formative frustration” through the lens of the GEP CURE, as the workflow of gene annotation requires students to sift through different sources of evidence, some of which might be contradictory. An example is provided in Figure 1, which indicates multiple tracks pointing to the existence of a protein coding gene in the genomic region shown. However, the numbers of exons predicted by the various gene prediction algorithms are at variance with one another, and some match poorly with the RNA-Seq evidence track. Therefore, it is up to the student to evaluate the available data to arrive at a defendable gene model. Should the initial model fail (as often occurs), students are able to quickly repeat their experiments. Common errors include choosing the wrong splice sites, ones that fail to adhere to the canonical signals at the start and end of introns (5′-GU…AG-3′). Here, we use data obtained from faculty and student surveys and student focus groups to assess the impact of formative frustration in a CURE from both the faculty and student perspective.

FIGURE 1

The GEP mirror of the UCSC Genome Browser for Drosophila mojavensis (Sept 2008 GEP/dot assembly), with lines of evidence supporting the presence of a protein-coding gene in this region. The genome sequence is shown in the top line (Improved Sequence), with multiple lines of evidence supporting the presence of a gene mapped against that sequence. There are apparent contradictions in these evidence tracks. The BLASTx alignment track indicates that the region at 93000–12000 of D. mojavensis shows significant similarity to protein sequences for two isoforms of the D. melanogaster gene Sox102F (Sequence Homology track). Computer-based gene predictors indicate a gene in this region (Gene Predictions tracks), but vary on the number, size, and location of predicted exons. RNA-Seq data appear to support the presence of three or four exons, yet TopHat and Cufflinks differ on the number and location of intron splice sites. The region from 7500 to 8000 might contain an exon of Sox102F (predicted by N-SCAN), or it might be a separate gene (predicted by Genscan and SGP) as there is some RNA-Seq data, but little or no conservation is indicated in this region. Students must reconcile these differences to generate the best-supported gene models for this region of the D. mojavensis genome.

Faculty observations of formative frustration: struggles are of benefit to students

The impact of student failure on persistence and eventual success has been widely discussed (37–41) and is given extensive treatment in the development of the growth mindset (12). Standard surveys of faculty work (e.g., the Faculty Survey of Student Engagement, or FSSE, http://fsse.indiana.edu/) do not, however, ask questions about student frustration. Therefore, we constructed our own survey (Appendix 2). Faculty responses to open-ended questions on this survey (Nos. 9, 10, and 11) were analyzed by term-based computer software (NVivo) to determine repeated themes.

Faculty were first asked to report a setback, obstacle, or failure encountered in their delivery of the project (Table 1). Course structure was mentioned by 45% (65 of 144 comments in response to that question). Frequently, comments about structure were associated with time issues (57%, 37 of 65 faculty who mentioned course structure), as faculty noted that tasks generally took longer to complete than the time allotted. Difficulties with technology (23%, 33 of 144) and student-related challenges (29%, 42 of 144), including a level of challenge too high for the student’s preparation, were also cited as causes of setbacks.

TABLE 1

Course setbacks have both positive and negative effects on learning.

Effect on Learning	Frequency	Examples
Positive	96
Insight into research process	26	I think that this gave them insight into the process of science that they might otherwise not have experienced. They had to struggle, but the realization that this is not something that can be predicted was something that they learned. I had told them that at the beginning of the semester, but only after experiencing it did they appreciate the lesson.
Greater understanding of subject	17	They then enthusiastically appreciated the whole process and the use of the GEP tools, both for finally understanding the basic biology of gene expression, and for gaining a sense of how computational tools help us to organize large masses of sequence data into biological sense.
Gained confidence	15	For most students, they worked through their anxiety and developed skills and confidence in their work.
Sense of accomplishment	10	Succeeding at a difficult task gave them a greater feeling of success and accomplishment.
Eye-opening, “wake-up call”	9	I think it was eye-opening for them to really see an example of where the computer couldn’t find the right answer.
Sense of ownership (embrace challenge)	9	Because of the limited amount of time, in many ways it forced the students to take more ownership of their research projects.
Sense of community	4	The resolution of the setback encouraged more interaction and camaraderie in the classroom, which was more conducive to learning than the race-to-get-done-without-necessarily-reflecting-and-learning that was happening the first time I taught the course.
Other positive effect	4	The “real world” functions the same way and these setbacks are a part of doing research, thus I feel learning was increased.
See the “bigger picture”	2	Students appreciate the realization that faculty also get caught unaware and need to regroup—and that research tools are constantly being modified.
Negative	46
Less understanding of subject	18	For students who struggled but didn’t seek additional help from me, they didn’t really appreciate the scientific goals behind the research project, or really what they were doing with different annotation methods
Less accomplished during project		Only one group finished their project before the semester ended, a few others completed their projects after finals were over.
Other negative effect	6	However this could be a problem for two reasons: (1) equity of access; (2) transition to mobile/tablet devices with unclear user infrastructure.
Experience less authentic	4	Again, without reinforcement in later weeks of the annotation skills students learned in the first assignment, I feel their genomics experience overall was quite superficial. In my view, it simply wasn’t enough time spent on the project to create a lasting appreciation for the annotation process and the first-hand knowledge of eukaryotic gene structure that is derived from that process.
Students out of sync (variation in learning)	3	The original setback was disruptive as I had a classroom with “finished” students resentful at being done and having “nothing to do” and students who were still working feeling awkward because they perceived they were “taking too long.”
Neutral	22	Based on exam questions, still learned the information that I wanted them to learn (gene structure and expression).

Comments from question 9, part D, of the Formative Failure faculty survey (Appendix 2) were analyzed using NVivo. One hundred four faculty took part in the survey. The frequency of responses in a given category is shown here and in the subsequent tables. Because the prompts were open-ended (see Appendix 2), each individual response often contained more than one thought. Representative comments are shown.

Question Text: A) Write about at least one setback, obstacle or failure that interfered with the planned or scheduled activities for your course (for example, computer failures, scheduling problems, misestimates of how long a given lesson would take). B) How did you and/or the students overcome the setback? C) How did the setback affect student behavior? D) Comment on your perception of how this setback affected student learning.

GEP = Genomics Education Partnership.

Faculty were asked to describe whether and how students attempted to overcome the issue. The most frequent tactic was for student groups to change their working style (42%, 89 of 214 comments), such as working outside of class, working together, or working with the instructor. Initial responses to the obstacle were more frequently negative than positive, including feelings of anxiety, frustration, and lost motivation. These negative reactions were offset by the perception that students increased their engagement with the material. In many cases (59%, 96 of 164 comments), the effect on student learning was described as positive, including greater insights into the research process, greater understanding of the subject, and a sense of accomplishment and greater confidence. Negative effects on student learning were reported less frequently (28%, 46 of 164 comments), and included reduced understanding of the subject, feeling less accomplished, and feeling the experience was less authentic. Together, these faculty observations suggest that students not only encounter setbacks in the GEP-curriculum, but when they do, they develop strategies that are mostly perceived as beneficial.

To assess the student response to struggles with the research process beyond course structure/design, faculty were asked to describe examples where students struggled with the standard annotation workflow. The most frequently reported problem was an interpretation of the data that did not conform to “the rules” of molecular biology (51%, 56 of 111 comments), such as the absence of canonical splice-sites. Under-utilization of all lines of evidence available and lack of understanding were also sources of struggle. When asked how students overcame these struggles, faculty indicated that students most frequently resorted to a more holistic evaluation of all available data and analysis tools provided. Instructors reported that they acted as guides in this process, rather than as an authority holding the “correct” answer. It is worth noting that these genes have not yet been studied in detail, so there is no established answer.

Instructors were then asked how overcoming individual struggles affected student learning. Most of the responses indicated faculty perceived there was a positive effect on learning (84%, 112 of 134 respondents) (Table 2; Fig. 2). Instructors commented that students acquired a greater understanding of the subject matter, gained insight into the research process, and learned to question assumptions. Less frequently (11%, 15 of 134 respondents), responses showed there were negative effects on learning, including excessive frustration and reduced feelings of accomplishment in instances where the student was not able to generate a solution—in this case, a defendable gene model. Thus, consistent with faculty responses about course setbacks, faculty also have an overall positive perception of the impact of individual setbacks in the research workflow and their effect on learning gains.

TABLE 2

Student setbacks have both positive and negative effects on learning.

Effect on Learning	Frequency	Examples
Positive	112
Greater understanding of subject	40	As my student and I were working through this setback, the student had an “aha” moment where they understood a really interesting biological situation, something that they hadn’t thought of before. I think that this student will remember this situation for much longer than if I just lectured about this topic in a classroom. It certainly increased their learning.
Insight into research process	27	In research projects things aren’t cookie cutter (like in the teaching labs) and there are real problems that you have to work through. It wasn’t always satisfying for the students, but it gave them a more realistic appreciation for how scientific research actually works.
Important for learning (generally positive)	16	I really believe this setback increased the students’ learning. They were able to carry out problem solving methods and use many pieces of data to come to a best conclusion to a complex problem. As both students are attending graduate school, I hope this type of thinking will be beneficial for them in the future.
Learn skepticism, how to question assumptions	9	I think this setback had a very positive impact on the student’s perception of research. The student learned to question the “rule/assumption,” given the weight of the evidences contrary to the rule.
Gain confidence	7	If anything, I think these challenges are beneficial for them. They gain confidence in tackling problems, and they gain greater insight into the project by virtue of having to work through difficult analyses.
Sense of accomplishment	7	Both the students and I felt that they had really achieved something when they eventually were able to get the gene feature correctly annotated. They expressed a strong sense of accomplishment when it finally “worked” and clearly stated that in the end it was “worth it.”
Led to future improvements in instruction	6	I’ve recognized that when one student is struggling, there’s a good chance that others have the same issues. Therefore, I’ve become better at answering questions in front of the entire lab section, because I recognize that it benefits more than one student.
Negative	15
Frustration has negative effects	7	While the students who eventually figured out the annotation “puzzle” were excited about it, some other students basically checked out after initial frustrations.
Less understanding of subject	6	If we can never locate the exon, or if we just pick a likely spot to call the exon with little or no evidence, I feel that it is not particularly helpful for student learning. In these cases, we are just trying to find something to put down on the report—we have not solved the problem.
Less accomplished	2	No, it just interfered with student progress and success.
Neutral	7	This particular student does very well with challenges, so this was not really a setback for her—just another puzzle to solve.

Comments from question 10, part D, of the Formative Failure faculty survey (Appendix 2) were analyzed using NVivo. The analysis and representative comments are shown.

Question Text: A) Describe an example of an annotation or finishing situation where the student struggled with the standard workflow (for example, where the student ran into a situation where the methodological “rules” were not followed, e.g., where there was a failure in logic when looking at the evidence in hand, or when evidence was overlooked or misinterpreted). B) Was the student able to overcome the problem? Describe how the struggle was resolved in as much detail as possible. If the problem was not overcome, comment on why not. C) Did the student struggles interfere with your plan for the course? How did you contribute to overcoming the obstacle, either by your prior planning or your actions at the time? D) Comment on your perception of how this setback affected student learning

FIGURE 2

Faculty observe that setbacks in the research process promote student learning. Results from NVivo analysis of comments from the Formative Failure faculty survey (Appendix 2) on the effects on student learning from course setbacks (survey question 9.D: problems affecting all students in the class, N=161) and student setbacks (survey question 10.D: problems encountered by individual students, N=134). The percent of faculty responses that were positive (blue), negative (orange), or neutral (no effect; grey) are shown. GEP = Genomics Education Partnership.

Finally, faculty were asked to consider their attitudes toward allowing students to fail in wet lab coursework, field coursework, wet lab research, and field research as compared with genomics research projects (Table 3). The degree of faculty acceptance of the risk of failure was evaluated as 1 (very likely), 2 (moderately likely), and 3 (not at all) (Fig. 3) and showed instructors were more likely to let students risk failure in GEP projects than in other settings. Because the stakes for reiteration in a genomics CURE are relatively low (time rather than time and materials), this suggests faculty are willing to structure the classroom experience such as to allow students to experience frustration, followed by success. Together, these survey results suggest that the majority of faculty found that setbacks can positively influence student learning and understanding of science. Computer-based genomics research allows the instructor to accommodate setbacks more readily than wet lab scenarios.

TABLE 3

Faculty are more likely to let their students fail in the GEP CURE than in their wet bench or field lab courses or research projects.

Reasoning for Responses	Frequency	Examples
Reasons to Tolerate Failure	84
Learn from failure	43	But I do recognize that some element of failure and risk is important in learning, so I allow students to take moderate risks for some lab experiences.
Failing is part of science, discovery	27	Trying and failing is part of science—it will happen frequently, even if you guard against it in your course design. You build in steps for quality control and reiteration as needed.
Skill development	14	Learning to distill value from “failure” is a critical characteristic of a productive scientist, arguably of a productive human being. It is a skill that does not come easily or naturally to many. But, like any worthwhile skill, it is honed and improved through practice.
Reasons to Avoid Failure	102
Money (cost of supplies)	39	This coupled with a limited lab budget generally leads to not allowing my students to take much risk.
Time	32	I am only moderately likely for wet lab coursework because of the time constraints put on this work—too many failures impact the ability to meet learning objectives.
Need quality results	17	Since my wet lab coursework and research require expensive reagents and include technically challenging procedures, I spend a lot of time training my students to perform their experiments and watch them very closely when they are performing them so that they don’t deviate much from our plan in hopes that they will generate interpretable and publishable data.
Safety	8	I would rather not have them take risks/fail because that might be very dangerous for them. I instead will ask each of them to strictly follow the given protocol and finish the experiments safely.
Other	6	I think it has to do with my level of comfort with the material. I feel confident in advising students with my wet lab research, and I come from a lab scientist background, so I know what it takes to learn. Bioinformatics is newer to me and I still feel woefully inadequate with it, so I’d much rather not have too many risks, as then I cannot help them as well.
Failure vs. Attitude	22
Too much failure leads to frustration, giving up, etc.	17	The majority of my students are intimidated by science and those that experience failures are often discouraged, give up, and/or think that they are not “smart enough” to be in science. Students struggle with the concept that failed experiments are a normal part of the science and instead blame themselves.
Respond positively to challenge	5	I don’t think that I would be in science if I hadn’t had the chance to design experiments and test my own hypothesis as an undergraduate.
Pedagogical and Institutional Concerns	53
Computer-based research has low-risk failure	39	With the GEP research projects, there is no problem having students fail and learn from their mistakes, as there is no financial incentive to have things work the first time like in my big laboratory courses or my wet lab, when supplies are expensive. The GEP research projects are perfect for giving the students extra rope to really get their feet wet for doing science. They can try, struggle, and fail and all it really costs is time and occasional irritation on the part of both the student and the instructor, but typically the student works through that irritation and comes out having learned both science and life skills from the process.
Learning process more important than results	9	I also make it clear to the students that in my class, getting it “right” means learning the skills and applying them to the research question rather than arriving at a predetermined answer, and I think that helps them relax about failing and taking risks too.
Limitations of institutional expectations	5	Because my institution is not a research institution, I can feel that I have some flexibility in allowing my research students to pursue projects that might be a little risky and may fail.

Comments from question 11 of the Formative Failure faculty survey (Appendix 2) focusing on faculty reasoning behind such decisions were analyzed using NVivo. The analysis and representative comments are shown.

Question Text: How likely are you to let your students take risks/fail in your wet lab coursework? How likely are you to let your students take risks/fail in your field coursework? How likely are you to let your students take risks/fail in your wet lab research? How likely are you to let your students take risks/fail in your field research? How likely are you to let your students take risks/fail in your GEP research projects? Explain your reasoning for the responses you chose above.

GEP = Genomics Education Partnership; CURE = course-based undergraduate research experiment.

FIGURE 3

Faculty are more likely to let their students risk failure in GEP research projects than in wet bench or field work lab courses and research projects. Faculty were asked how likely they were to let students fail in performance of wet bench lab work (coursework and research), field work (coursework or research), and GEP research activities. The degree of willingness to risk failure was evaluated as 1 (very likely) to 3 (not at all). Note that many GEP faculty members do not do field work, so the number of responses in that category is lower. (Percentages do not sum to 100%, as “not applicable” responses are not shown.) GEP = Genomics Education Partnership.

Student feedback: many responses mention both frustration and success

To explore the formative frustration hypothesis from the student perspective, we examined open-ended comments from student post-course surveys for general themes. Initial NVivo analysis of 647 student comments for emergent themes revealed both positive and negative comments. Emergent positive comments included ones about classroom atmosphere, motivation, learning, understanding science, and career clarification. Negative comments included ones about time issues, task difficulty, and uncertainty. Since we observed many comments with both positive and negative elements, we conducted a second NVivo analysis, searching for terms indicating frustration or struggle, as well as terms indicating knowledge/learning gains or appreciation of the scientific process (see Materials and Methods). In parallel with this computer-based analysis, we also performed a manual examination of student comments that considered the context of specific phrases. Based on the initial computer-based analysis, 243 “frustration,” 69 “scientific process,” and 794 “learning gains” comments were found. These 243 frustration words mapped to 29% of the individual comments (186 of 647 comments, with some comments containing more than one frustration word). Of the comments referencing frustration, 70% also included at least one term referring to learning and/or the scientific process (129 of 186). The subsequent manual analysis showed a correlation between student comments referencing frustration and learning gains/the scientific process. This analysis was more stringent because context of a word or stem was evaluated; only 20% of comments (126 of 647) were identified as mentioning struggle or frustration, yet over half of these comments (56%; 71 of 126) also mentioned success. Representative comments showing student struggles/frustrations with or without links to success are shown in Table 4. Together, these data are consistent with the hypothesis that students view frustration as a formative experience in the context of the genomics CURE.

TABLE 4

Results from examination of student survey comments.

Comment type	Examples
Frustration only (55/647 = 8.5%)	Learning how to use the online gene annotation resources was a little complicated and made much of this process difficult.
	It was extremely frustrating and confusing to understand.
Frustration and Success (71/647 = 11.0%)	HUGE learning curve, I almost gave up and then one day it clicked and I was able to finish the project. Looking at the material given and figuring out what the next step would be took me a little while to figure out. The whole project was pretty cool to be a part of. I would say to maybe add a learning section for the rules of Gene Model Checker and what the “fails” mean.
	Overall this class was challenging, but very rewarding once you had those “ahhh” moments.
	This project was challenging. Nevertheless, contributing to this gene annotation was a great experience, academically and for future science research. My main weaknesses were to learn from scratch how to annotate and all the vocabulary as well as the science behind this project. However, one of my strengths, which is perseverance, allowed me to continue and understand what I was doing.
	It was challenging because every day it was unknown what I would come to class to find. It was like starting fresh every lab period, which could be frustrating. However, I really enjoyed knowing that I did something that other people will look at. I learned to work through the frustration. At times, it would be difficult to think that what I was doing was helping me learn, but by the end of the semester I was confident that I knew what I was doing.

Student survey comments were examined in light of the number of comments that were coded as “mixed,” meaning they articulated both positive and negative sentiments. The entire dataset (N=647) was manually re-examined, specifically looking for comments indicating frustration and struggle coupled with perseverance and ultimate success. Representative comments are shown in the table. Of the total, approximately 20% of the comments mentioned frustration or struggle, but of those, more than half also mentioned success.

To further explore the value of formative frustration from the student perspective, we examined a total of 669 comments from student focus groups conducted at six different schools (Appendix 3). Questions addressed whether participating in the project fostered better understanding of science, clarified career interests, or led to personal gains. Transcripts were first analyzed using NVivo to identify emergent themes. While some students said they did not get much out of the project (44 comments), many students said that engagement in the project promoted learning, including understanding of the scientific process (59 comments), and that obstacles/frustrations in science need to be overcome to make progress (22 comments). Students also noted that scientists need to be creative and adaptable (16 comments).

To examine the connection between struggle/frustration and knowledge/learning gains, we performed a secondary NVivo analysis on the student focus group comments, followed by context-based, manual analysis for key words as was done for the student survey comments (see Materials and Methods). For these analyses, comments were analyzed for co-occurrence of “frustration” terms associated with knowledge, learning gains, and/or engagement in the scientific process. The secondary NVivo analysis indicated that struggles are frequently associated with learning gains. Indeed, from comments evaluated across all groups, 21% (142 of 669 statements) referenced frustration. Of these frustration comments, over half (58%; 82 of 142) also referenced learning/engagement gains. Similarly, manual analysis revealed that 11% of comments (75 of 669) included a keyword or phrase associated with struggle or frustration, and of these, 72% (54 of 75) also referenced learning gains or a better understanding of the scientific process (Table 5). Example focus group comments are provided in Table 6. Thus, consistent with the faculty and student surveys, analysis of student focus groups showed a link between frustration and learning gains or understanding of the scientific process.

TABLE 5

Results from manual re-examination of student focus group comments.

Focus Group	Total Number of Student Comments	Percentage of Student Comments Referencing Frustration	Percentage of Frustration Comments Referencing Learning or Scientific Process
1	81	14.8% (n=12)	91.7% (n=11)
2	48	16.7% (n=8)	87.5% (n=7)
3	119	7.6% (n=9)	100% (n=9)
4	79	15.2% (n=12)	91.7% (n=11)
5	131	13.0% (n=17)	5.9% (n=1)
6	110	10.0% (n=11)	90.9% (n=10)
7	101	5.9% (n=6)	83.3% (n=5)
Aggregate	669	11.2% (n=75)	72.0% (n=54)

Student comments from the focus groups were manually evaluated as to whether they contained a word or phrase signifying frustration and if they did so, whether they also contained a word or phrase denoting learning or better understanding of the scientific process.

TABLE 6

Representative comments from student focus groups highlighting the transition from challenges to benefits/successes.

If this class has taught me anything it’s that you can’t really prove something. You can only gather evidence to support or not support it.

It was discouraging at first, but as we started working through it and trying to solve the problem…. We were taught all of the tools and it was satisfying to me to pretty much discover a way (to annotate).

[W]e had these troubles…and we always like approached it from a different angle because you just have to…it wasn’t something that you would just give up on…. [I]t made me more interested in it and made me just, like, want to explore it more.

I made a lot (of mistakes), especially at the beginning when I was getting used to using the programs and analyzing the data, but the good thing is that we had a lot of support from the professor…. That way…even though it was really hard, we had a better idea and were more prepared for data that were going to appear that would be difficult and we were prepared with the programs that we had used so that we could overcome those obstacles.

I think that the best way to deal with setbacks was…talking among ourselves and with the professor. The work was individual but you get to the point at which something wasn’t working after making several attempts, you need to get help. It was collaborative work.

I really think that is the best way to learn how to do anything because you’re not just like passively observing the information; you’re working with it and you’re figuring out the reasons why it has to be the way it is through trial and error. And I think that’s a really valuable process.

[I]nitially the things we were doing didn’t really make complete sense. But as we worked…really moving forward, it started to make more sense and we got more used to…methods and annotation. So…you can overcome the challenges coming forward. It was a great learning experience when it comes to that part.

[T]his project taught us to confront problems and solve them…. [T]his is a tough course that requires effort from a student and it makes us realize that we achieved it, that we confronted problems, that we managed to solve them, that it isn’t so easy and it motivates us to keep trying other courses that are also challenging in the beginning.

Our analysis showed one focus group was markedly different from the others (group 5; Table 5). Similar to other groups, 13% (17 of 131) of comments contained a term associated with struggle or frustration. However, only one of the 17 comments also referenced gains in learning or understanding of science, whereas in other groups there was a strong relationship between frustration and learning gains (Table 5). We hypothesize the difference in this group resulted from the apparent lack of a consistent advisor, since students indicated they struggled with logistical details and problem resolution, and linked their struggles to lack of mentorship. Conversely, comments from students in other focus groups often pointed to the roles of TAs and instructors in problem solving. Together with data described earlier indicating that frustration can lead to knowledge and learning gains, observations from this outlier group highlight the importance of implementing a framework that helps students move past obstacles. While this wasn’t a hypothesis we sought to investigate at the outset, the importance of classroom environment emerged during the study.

DISCUSSION

Authentic research often demands evaluation of multiple lines of evidence, some of which may be contradictory. In this CURE, students are often required to revise their initial gene models to develop ones they can ultimately defend. In this way, the GEP project promotes both engagement in original research and Dewey’s reflective thinking (1). In the current study, we set out to investigate the formative frustration hypothesis more systematically. Faculty overwhelmingly reported intellectual struggles as positive for student learning. Both organizational obstacles (i.e., computer problems, time challenges, group dynamics) and the trials of individual projects could be overcome, leading to favorable outcomes. The student survey comments and student focus group transcripts provide evidence that these ideas resonated with the students. Student survey comments were collected as part of an online survey over two years, whereas the student focus groups were organized independently of the surveys. Yet, the analysis of both datasets converged on the idea of formative frustration. In short, although the process can generate frustration for the student, when issues are resolved, a sense of accomplishment ensues.

“Opportunity for iteration,” named as one of the five essential dimensions of a CURE (42), is an inherent feature of our project. Our results suggest opportunities for student iteration foster better understanding of the topic and the scientific process. This view is consistent with literature on the design of student research experiences. For example, Lopatto (43) recommends that undergraduate research projects should avoid “cookbook experiments” and provide “built-in difficulties” (see also [22, 44]). The value of reiteration can be lost when faculty provide pre-made materials to get students over difficulties in efforts to save time, as might occur when wet lab projects fail in class. Genomics research allows for quick repetition of investigative steps and requires no additional materials, minimizing the cost of iteration. Moreover, if further time is needed, students have around-the-clock access to the online tools and databases. Importantly, recent reports suggest that wet lab and computer research result in similar attitudinal gains by students with respect to interest and achievement in research (45), and that students who fail to achieve scientific goals may nevertheless experience positive outcomes (46).

Both faculty and student comments refer to the value of formative frustration, but in practice we have found that making the struggle productive requires support from faculty, TAs, and peers (referred to as the Learning Environment in Fig. 4). Providing such support is reflected in the broader educational literature as using a social constructivist approach, in which instructor and student are engaged in a two-way dialogue to support learning. “Communities of practice” and “cognitive apprenticeships” (47) describe familiar modes for research scientists that can be replicated in the CURE environment. A clear and authentic research objective (number 2 in Fig. 4) and a scaffolded curriculum (number 3 in Fig. 4; [23, 27]) prepare students to engage in experimentation. Ideas around scaffolding and a positive environment also appear in the literature on “productive failure” (13, 14) and “productive struggle” (15). Creation of a supportive environment that increases confidence is especially important for those students prone to respond with a “helpless” strategy (11). Effective instructors know when to intervene and when to permit students to experience the value of struggle (48).

FIGURE 4

Genomics-based CUREs can support an iterative learning process. A collaborative Learning Environment (1) supports the entire process, which includes a defined learning objective (2), a formative strategy (3), and iterative experimentation (4). Adapted with permission from (10).

Our observations point to a dynamic in which the student’s expected sources of authority, including computer software and course instructors, abdicate their traditional roles. Instead, the student is encouraged to develop a sense of personal agency for making discoveries and resolving problems. We find from the multiple data sources described here, which include observations from a group of over a hundred faculty and thousands of students over the course of several years, that formative frustration, if channeled appropriately, can be an important means for student growth and development. In summary, we suggest these findings may be broadly applicable to CUREs engaging students in genomics research, and potentially to other CUREs on a variety of topics.

Click here for additional data file.