Introductory Biology Students' Use of Enhanced Answer Keys and Reflection Questions to Engage in Metacognition and Enhance Understanding.

Providing feedback to students as they learn to integrate individual concepts into complex systems is an important way to help them to develop robust understanding, but it is challenging in large, undergraduate classes for instructors to provide feedback that is frequent and directed enough to help individual students. Various scaffolds can be used to help students engage in self-regulated learning and generate internal feedback to improve their learning. This study examined the use of enhanced answer keys with added reflection questions and instruction as scaffolds for engaging undergraduate students in self-regulated learning within an introductory biology course. Study findings show that both the enhanced answer keys and reflection questions helped students to engage in metacognition and develop greater understanding of biological concepts. Further, students who received additional instruction on the use of the scaffolds changed how they used them and, by the end of the semester, were using the scaffolds in significantly different ways and showed significantly higher learning gains than students who did not receive the instruction. These findings provide evidence for the benefit of designing scaffolds within biology courses that will support students in engaging in metacognition and enhancing their understanding of biological concepts.

INTRODUCTION

Undergraduate biology students need support to learn how to integrate individual concepts and facts into the complex organization required for robust understanding of biological processes (Wilson ). Students’ alternate conceptions may not be true misconceptions but rather a combination of confusion about the topic and difficulty with integrating independent ideas into dynamic systems (Lewis and Wood-Robinson, 2000; Marbach-Ad and Stavy, 2000; Smith and Knight, 2012). Feedback is an important part of students developing scientifically accurate understanding of complex biological concepts and has been shown to lead to improved learning, because it gives students information on what they have done well and what they still need to improve upon (Black and Wiliam, 1998). Unfortunately, in large, undergraduate science classes, it is difficult for instructors to provide feedback that is frequent and directed enough to help individual students (Wood, 2009). To balance these factors, innovative strategies must be implemented to provide the kind of feedback that can be directed toward individual students’ unique learning needs even in large-enrollment classes.

One approach to overcome this challenge is to support students to engage in self-regulation of their own learning by monitoring their own work, generating internal feedback, and using that feedback to make changes to their learning strategies (e.g., Sadler, 1989; Bell and Cowie, 2001; Nicol and Macfarlane-Dick, 2006; Wood, 2009). Research has shown that students can effectively be their own sources of feedback via self-assessment and self-regulation (Ross ; Boekaerts ; Zimmerman and Schunk, 2001; Nicol and Macfarlane-Dick, 2006; Andrade et al., 2008). Metacognition is one core component of self-regulated learning and involves awareness of one’s own learning process (Schraw ; Wood, 2009; Sinatra and Taasoobshirazi, 2011). By engaging in metacognition, students question what they know, what they need to improve on, and how they can apply the information (Wood, 2009). Instructors can support students in this process by providing them with opportunities to consider and refine their knowledge within scientific contexts (e.g., Handelsman ; Haak ; Freeman ; Dolan and Collins, 2015). However, while prompting students to engage in metacognition may be enough for some students, others will require additional support to successfully engage in the practice (Stanton ). Importantly, metacognitive skills develop gradually and are linked to both knowledge and experience; students must have some understanding of a disciplinary domain in order to engage in reflection of their understanding of its concepts (National Research Council [NRC], 2000). Much more remains to be discovered about how undergraduate students can integrate metacognitive skills to become more effective at complex biological reasoning.

One way to support students to engage in metacognition and self-regulated learning is to provide them with scaffolds to structure their reflection and consideration of their own ideas. In this study, we examined the combination of three scaffolds in a large introductory biology course: enhanced answers keys, added reflection questions, and added instruction on their use. The enhanced answer keys consisted of the answers to assignment and exam questions but also included additional explanatory information and details on how questions were scored. Reflection questions were added to the enhanced answer keys to further support students’ engagement in metacognition and consideration of their understanding of biological concepts. Individual instruction was also provided to a subset of students on using the enhanced answer keys and reflection questions to help students engage in the process of self-regulated learning.

Because the enhanced answer keys provided guidelines and feedback to students, and some questions contained scoring levels, they were similar in some aspects to rubrics. Past research as shown that rubrics can play an important role in helping students evaluate their work and their ideas and provide a source of feedback students can use to improve their learning (Reddy, 2007). Rubrics are frequently used by instructors as evaluation tools but can also be used as teaching tools, in that they can help students to “think, learn, and produce high quality work” (Andrade, 2005, p. 27), or as a source of feedback that can lead to improved learning (Reddy, 2007). Although some research has focused on how students perceive and use rubrics generally (e.g., Andrade and Du, 2005), little work has focused on the extent to which students use rubrics to actively engage in their own assessment (Reddy, 2007), particularly to reflect on their performance and consider how to improve their understanding. Further, students may need instruction on how to properly use and understand rubrics (Andrade, 2005), but little work has focused on the type of instruction that would effectively enhance students’ use of rubrics or answer keys for self-regulated learning. Finally, some past work examining the practice of posting answers after an exam found no relationship with students’ test performance (Lake and Chambers, 2009). However, this study was limited to students in dental school and found that the posted tests primarily consisted of questions at the recall level, rather than at the application level, and researchers did not investigate the extent to which students engaged with the posted exams (Lake and Chambers, 2009). Research is therefore needed to examine how these types of scaffolds may be an important tool that instructors could use to support students to engage in metacognition and generate feedback on their own progress, particularly in large undergraduate classes in which frequent, individual feedback from instructors may not be possible.

To begin to explore this further, we examined how students used the featured scaffolds to engage in metacognition in a large undergraduate introductory biology course and asked the following research questions:

How do undergraduate students use postassignment enhanced answer keys and added reflection questions within an introductory biology course?

To what extent does providing instruction on use of enhanced answer keys and reflection questions influence undergraduate biology students’ use of the scaffolds and course performance?

In what ways do undergraduate students engage in metacognition when they use enhanced answer keys and/or reflection questions within an introductory biology course?

THEORETICAL FRAMEWORK

Self-Regulated Learning

Self-regulated learning consists of “the active control by students of some aspects of their own learning” (Nicol and Milligan, 2006). Students can take a proactive role in understanding their own learning by monitoring their own work and generating feedback they can use to adjust their strategies (Sadler, 1989; Nicol and Macfarlane-Dick, 2006). Past research has shown that students can learn how to self-regulate their learning and that those who do so have greater success in classroom settings (Pintrich, 2000; Zimmerman and Schunk, 2001). However, the idea that students can be a primary source of their own feedback is a relatively new idea (Andrade, 2010), and more extensive examination on how to enhance self-regulation within science education is needed (Nicol and Macfarlane-Dick, 2006). Self-regulated learning is composed of three major components: cognition, metacognition, and motivation (e.g., Schraw ; Sinatra and Taasoobshirazi, 2011), and three phases that function in a cyclical manner: 1) forethought, 2) performance or volitional control, and 3) self-­reflection (Zimmerman, 2000).

Metacognition

Metacognition involves the awareness of one’s own learning process (Wood, 2009) and includes both metacognitive knowledge and metacognitive regulation (Lawson, 1984; Jacobs and Paris, 1987). Metacognitive knowledge is awareness of our own thinking, while metacognitive regulation is how we control that thinking in order to learn (Stanton ). Asking students to be metacognitive and assign plausibility or intelligibility to their own ideas is one way to help students change their alternative conceptions to more scientifically accurate conceptions (Tanner and Allen, 2005). Students who employ metacognition perform better (e.g., Anderson and Nashon, 2006; Young and Fry, 2008; Vukman and Licardo, 2009; Stanton ) and have increased learning outcomes (e.g., Baird, 1986; Tobias and Everson, 1996; Stanton ) and increased problem-solving ability (e.g., Rickey and Stacy, 2000; Sandi-Urena ). The foundation for the metacognition aspect of this study is a framework described by Grotzer and Mittlefehldt (2012) that includes three dimensions and related questions: 1) “Intelligibility: Does the explanation make sense to me?,” 2) “Plausibility: Do I think that the explanation is a possible explanation?,” and 3) “Wide-applicability: Can I apply the explanation beyond the contexts in which I have learned it?” (Grotzer and Mittlefehldt, 2012, p. 82).

Conceptual Framework

Combined, these ideas constitute a conceptual framework that integrates students’ idea development with principles of self-regulated learning and metacognition. Students begin with a pre-existing idea and progress to a new or revised idea with support from both internal and external factors (Figure 1). External factors include aspects of learning environments to which both students and instructors contribute—such as attention to prior ideas, engagement with content, student assessment, and responsive feedback. In the context of these factors, students can use internal factors such as the cycle of self-regulated learning (forethought, performance control, self-reflection) to contribute to idea development and metacognitive skills to consider the intelligibility, plausibility, and wide applicability of the ideas. Scaffolds, such as enhanced answer keys and reflection questions, can help students and instructors bridge the external and internal factors and support learning. Ideally, the end product of these inputs and student thought processes is a new or revised idea.

FIGURE 1.

Conceptual framework. Scaffolds, such as enhanced answer keys and reflection questions, can support students in bridging external factors related to their learning to internal factors, such as self-regulated learning and metacognition, to help them progress from a pre-existing idea to a new or revised idea.

DESIGN AND METHODS

Context and Participants

All student recruitment and data collection for this study was approved through the institutional review board. All students in one section of an introductory biology course at a large midwestern university were recruited, and 85 students (88% of the class) consented to participate. Students in the class were primarily sophomores taking the second of two required introductory biology classes toward a life sciences major. The course consisted of three 50-minute meetings per week throughout the Fall semester, and four units respectively focused on the genetic basis of evolution, phylogeny and speciation, form and function, and ecology. In each unit, students completed three assignments (15 points each) and one exam (150 points each) for a total of 12 assignments worth 18% of the final grade and four unit exams worth 55% of the final grade. The assignments consisted of open-response questions that asked students to take concepts discussed in class and use them to interpret data, engage in scientific practices such as creating graphs to present data or models to explain connections between concepts, and apply concepts to new situations as presented to students in case studies. Exams consisted of multiple true/false, short-­answer, and essay questions that all focused on students’ ability to integrate concepts and transfer information from in-class and assignment work. See the Supplemental Material (Appendix A) for an example comparing how the assignments and exams evaluated similar topics but with different scenarios and questions.

As a normal part of the course, students had access to an instructor-designed, postassignment, enhanced answer key that was specific to each assignment or exam (example included in Appendix B in the Supplemental Material). These enhanced answer keys were used by instructors and learning assistants to grade the assignments and as feedback and examples of ideal answers for students after the graded assignments were returned. Instructors and learning assistants added explanations to the answer keys as they graded the assignments or exams and saw common issues in students’ responses. They also added information on how particular questions were graded if partial credit was awarded. Students received the regular enhanced answer key for the assignments in the first unit. As a part of this study, students then received a modified enhanced answer key for all assignments and exams from the second unit through the end of the semester. The modified enhanced answer key consisted of all components of the regular enhanced answer key with added reflection questions intended to support students in using the enhanced answer key to engage in metacognition to evaluate their work and reflect on their understanding. The questions included prompts to help students think about what they understood about a concept, what they needed to know to understand it better, what they did not understand, and whether or not they understood the idea well enough to apply the information to a different situation (Wood, 2009). Further, the questions challenged students to consider the intelligibility, plausibility, and wide applicability of the concepts that were relevant to the assignments (Grotzer and Mittlefehldt, 2012) and thus were explicitly embedded in the conceptual framework underlying the study (Figure 1). The reflection questions are included in the Supplemental Material (Appendix C).

Students who consented to be contacted were invited to participate in two interviews each, and the first 20 students who responded were selected. The group of interviewed students was found to be equivalent to the group of noninterviewed students based on no significant differences in grade point average (GPA ; t(77) = 1.66, p = 0.17) and ACT composite scores (; t(69) = 1.67, p = 0.08). The interview participants consisted of 11 females and nine males. Twelve of the students were sophomores, four were juniors, and four were seniors or returning postbaccalaureate students. Students received a $10 gift card for each interview they completed. The 20 participants were randomly assigned to one of two groups of 10 each (see Table 1). Group 1 (10 students) received the regular enhanced answer key during the first interview. Group 2 (10 students) received the modified enhanced answer key with the added reflection questions and instruction on their use in the first interview; this was before any other students in the course had access to the reflection questions. Group 1 then received the reflection questions and instruction on their use in the second interview, at the same time those items were available to all other students in the course. For both groups, instruction on the enhanced answer key and reflection question use consisted of discussion of why the documents were created and the intention of their use for students’ reflection and consideration of their own ideas. The researcher conducting the interview then walked students through the various components of the enhanced answer keys and reflection questions to demonstrate how students might use them. Students were then asked to consider the ways in which the scaffolds might be useful in general and the ways in which they could see themselves using the scaffolds in practice.

TABLE 1.

Data-collection summary

Data source	Description	Number collected	Interview group 1 (n = 10)	Interview group 2 (n = 10)
Interview 1	The first interview was before the first exam and included a “think-aloud” task.	n = 20	Received regular enhanced answer key (rubric alone)	Received modified enhanced answer key (with added reflection questions)
Interview 2	The second interview was after the second exam.	n = 17	Received modified enhanced answer key (with added reflection questions)	Received modified enhanced answer key (with added reflection questions)
Assignment scores	All students completed three ­assignments per unit for a total of 12 assignments.	n = 79 for each of the 12 assignments
Exam scores	All students completed one exam after each unit for a total of four exams.	n = 79 for each of the four exams
Surveys	All students were asked to complete three surveys during the course of the semester.	n = 46 for survey 1 n = 50 for survey 2 n = 46 for survey 3

Data Collection

Data collection consisted of scores from all assignments and exams, three online surveys, and semistructured interviews (see Table 1). The assignments consisted of open-response questions that asked students to take concepts discussed in class and use them to interpret data, engage in scientific practices such as creating graphs or models, and apply concepts to new situations. Exams consisted of multiple true/false, short-­answer, and essay questions that focused on students’ abilities to integrate concepts and to transfer information from in-class and assignment work. The instructors and learning assistants graded the assignments and exams. Scores from all assignments and exams for the first two units were collected for research purposes.

All students in the course were asked to complete three surveys after the first, second, and third exams. The timing of the surveys allowed analysis of how students used the enhanced answer keys both before and after the reflection questions were added, and how they may have changed their usage of the enhanced answer keys and reflection questions over the course of the semester. The surveys consisted of 1) closed-response questions about enhanced answer key use; 2) Likert-style statements focused on how students used the enhanced answer keys and reflection questions and the extent to which they used metacognitive skills; and 3) open-response questions about studying, how they evaluated whether or not they understood a concept, and enhanced answer key and reflection question use. The surveys were conducted online through Qualtrics. Students were given 1 week to complete each survey and received a small amount of course credit for each survey. The survey questions are included in the Supplemental Material (Appendix D).

The interviews were semistructured (Miles ) and consisted of questions about metacognitive skills the students used and connections they made between the content on the assignments and exam. The first interview occurred in the first unit before the first exam (interview protocol in Appendix E in the Supplemental Material). The second interview occurred after the second exam (interview protocol in Appendix F in the Supplemental Material). Group 1 students (see Table 1) were asked about the regular enhanced answer key in the first interview and about the modified enhanced answer key in the second interview. Group 2 students (see Table 1) were asked about both regular and modified enhanced answer keys in the first interview and about the ways the modified enhanced answer key affected the way they thought about the content and studying in the second interview. In both groups, the first interview also included a think-aloud task (Gall ) that consisted of the students reading through the enhanced answer key and then using it to analyze their assignments. Following these tasks, students were asked follow-up questions, again in a semistructured format, to reflect on the metacognitive processes they used to complete the task. First interviews averaged 20 minutes long and ranged from 11 to 33 minutes. Second interviews averaged 17 minutes long and ranged from 11 to 34 minutes. All interviews were audio-recorded and transcribed. Interviewed students were given pseudonyms for dissemination of research results.

Data Analysis

The assignments and exams scores were used for statistical analysis. We used independent sample t tests and analysis of variance (ANOVA) to identify differences on mean assignment and exam scores between students who used the enhanced answer keys and those who did not, those who used the reflection questions and those who did not, and between students who received the directed instruction (interviewed students) and those who did not (all other students). We also used independent-sample t tests to examine GPA and ACT composite scores between the group of the students who were interviewed and the group of students who were not in order to determine the equivalency of the two groups. We found no significant difference between the scores in the two groups as reported above. The survey data were used to examine trends in how students reported using the enhanced answer keys and the extent to which they engaged in metacognitive strategies. The open-response survey questions were categorized into common categories and both open- and closed-response survey responses were examined for trends across the surveys based on scaffold use.

The transcribed interviews were imported into qualitative analysis software (QDAMiner). First-round analysis consisted of coding for discussion of 1) enhanced answer keys, 2) reflection questions, and 3) metacognition. All statements coded as metacognition then underwent second-round analysis, which consisted of coding for Grotzer and Mittlefehldt’s (2012) three dimensions of metacognition: 1) intelligibility, 2) plausibility, and 3) wide applicability (see Appendix G in the Supplemental Material for examples of codes). We used open coding to identify common segments of data and pattern coding to group segments into common themes across the data and between groups 1 and 2 (Miles ). To ensure reliability of the findings, we cocoded a sample of the data to establish interrater reliability. Initial agreement was ∼70%, and reached 100% following discussion. Students’ answers to interview questions, closed-­response survey questions, and open-response survey questions were used to triangulate the data sources to ensure the findings were corroborated across sources (Miles ).

RESULTS

Use of Enhanced Answer Keys and Reflection Questions

In the first research question we asked, “How do undergraduate students use postassignment enhanced answer keys and added reflection questions within an introductory biology course?” Overall, we found that 1) students reported using the enhanced answer keys and reflection questions to study and to enhance their understanding and 2) the different ways in which students reported using them increased over the semester.

Enhanced Answer Keys.

First, in answers to closed-response survey questions, the number of students who reported using the enhanced answer keys at all increased, but not significantly, from the first to the third survey (survey 1 = 83%, survey 3 = 87%); however, the number of students who reported using the enhanced answer keys specifically to help them study for exams increased significantly from 24% on the first survey to 52% on the third survey (t(90) = −2.89, p = 0.002, d = 0.6). This indicates that students saw the enhanced answer keys as useful tools to help their studying and more students began using them for that purpose by the end of the semester.

In the beginning of the semester, students typically discussed using the enhanced answer keys to primarily look at incorrect answers, because they thought a correct answer indicated that they already understood the material. For example, in the first interview, Josh indicated he saw no reason to go over a correct answer:

I feel like if I got it right I probably knew what was on the enhanced answer key and stuff so there’s not really a reason to go over it and then if, obviously if it’s incorrect and I don’t understand why, I like to go see what answer is and why that is the way it is. (Josh, interview 1)

However, as the semester progressed, students began to recognize that using the enhanced answer keys could help them increase their understanding and see where they had misunderstood a concept or made a mistake. For example, Ashley acknowledged that she may have been “a little iffy” and could use additional consideration of the topic, even if she got the answer correct:

I used them to check myself and to check that my answers are correct especially when I really thought one was correct, you know sometimes I’m a little iffy and so when I see one is wrong I’m like, yeah, I understand, I wasn’t even sure about that when I first put it, but then some of them I’m like, I really thought that was right, what am I doing wrong? Obviously I’m doing something wrong. So then I’ll really go back and look at those. (Ashley, interview 2)

Even among those students who had not previously used the enhanced answer keys, the students who were introduced to the enhanced answer keys in the interview indicated that they could see how the keys would help them to gauge their understanding or to reach greater understanding. For example, though Ashley had not yet used the enhanced answer keys, she indicated that she planned to so to understand answers she got wrong or to increase her understanding of certain concepts.

I haven’t [used the enhanced answer keys] yet, but I’m planning on doing that for studying to make sure.… Definitely if I get one wrong, I want to find out why I get it wrong. And then I’ll go and I want to find out what the right answer is and figure out why I didn’t put that down.… And also with the studying, I’ll make sure that what I have is sufficient for me to understand it. And if I’m not understanding a concept, then I’ll go back and see what they have [on the enhanced answer key] to walk me through it. (Ashley, interview 1)

Finally, using ANOVA with follow-up tests, preliminary analysis of grades and responses on the last survey from all students in the course indicated significant differences between students who used some enhanced answer keys and students who indicated they did not use the enhanced answer keys at all. Students who used the enhanced answer keys at least some of the time scored significantly higher than students who did not use them on both total assignment points (; F(2, 66) = 3.79, p = 0.002) and total course points (, ; F(2, 66) = 3.80, p = 0.002). This may indicate that using the enhanced answer keys helped the students adapt to expectations on the assignments and, therefore, achieve higher scores. However, not all students in the course responded to the surveys, and the number of students who indicated on the surveys that they used the enhanced answer keys on either 1) all assignments and exams or 2) not at all was very small compared with students who indicated they used the enhanced answer keys on some assignment and exams (on survey 3: nall = 5, nsome = 60, nnone = 4). Therefore, additional examination of these trends with larger numbers of students is needed. However, though these findings from all students in the course need further investigation, we found a much clearer relationship between course performance and whether or not students received instruction on the rubrics (see Instruction on Enhanced Answer Key and Reflection Question Use below).

Reflection Questions.

Most of the students who reported using the reflection questions were the interviewed students who had been introduced to the enhanced answer key addition during the interviews. The students who reported using the reflection questions saw them as a helpful tool to consider their understanding and began to incorporate reflection as a normal part of their studying.

First, the small number of survey takers who reported using the reflection questions (survey 2 = 9 out of 51, survey 3 = 10 out of 46) indicated they used them to better understand the material and to make connections. This was also the case in the interviews in which students recognized that using the reflection questions could help them determine what they did and did not understand and to reach greater understanding. For example, Nick indicated the reflection questions could help him to structure his evaluation of the questions and to better understand what he needed to focus on to understand the material:

[The reflection questions] provide structure to how you can look at the questions. So, instead of me saying, ok, I don’t understand this what am I supposed to do here, I can go to these and say, how can I look at that, what’s a good way to look back at it and make sure I understand the material? (Nick, interview 2)

In contrast, students who responded that they had not used the reflection questions reported reasons such as they already had a set way of studying, they were concerned only at seeing incorrect answers, they could not learn a concept from an answer key, they saw it as a waste of time, or they did not see how the answer keys could be useful. For example, one student said, “I did not [use the reflection questions] because in my opinion, it seems like a waste of time, and I would rather spend that time actually learning the material, rather than reflecting on my studying experience” (33, survey 3). Not surprisingly, this indicates that the students who did not see value in reflecting on their understanding also did not see value in using the reflection questions. This may indicate that students need additional instruction on the value of this reflection and how it might improve their ability to understand the material. This is further supported by the Likert-style survey statements. The mode for most of the Likert statements about reflection questions was a 2, which indicated that more students agreed than disagreed with statements about use of the reflection question (Table 2). However, the number of students who agreed was not substantially different than the number who disagreed in many cases (Table 2). It is also noteworthy that, for the statement that the reflection questions caused them to change the way they studied, the mode for the responses was a 3, which indicates more students disagreed than agreed with the statement. These responses were similar in both survey 2 and survey 3. This finding is different from what interviewed students related, indicating that the additional discussion about the reflection questions is important for students to see their value.

TABLE 2.

Descriptive statistics for selected survey 3 Likert-style reflection question statements

	Averagea	Modea	Agree	Disagree
The section of the enhanced answer key with the reflection questions helped me to consider my understanding of the topics.	2.37	2	26	20
The section of the enhanced answer key with the reflection questions caused me to make changes to the way I study.	2.46	3	20	26
The section of the enhanced answer key with the reflection questions caused me to change the way I thought about the topics.	2.15	2	31	15

a1 = strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree.

Second, by the third survey, many students reported they had begun to incorporate the reflection step outlined by the reflection questions into their normal studying; this was especially true for the students who received the reflection questions in the first interview. They did not necessarily use the reflection questions as written, but did take time to consider them generally or mentioned using them more specifically if they recognized the need to go more in depth into a topic to reach understanding. For example, Josh said,

I think these questions bring up a lot of good points that you should think about in general when studying for a test and when using assignments to study for a test. So, I mean, yeah, I probably won’t look specifically at the questions but now that I’ve thought about all these things, some of them are going to be, I’ll probably use more. (Josh, interview 2)

In this case, Josh was not using the reflection questions exactly as they were written but still considered their value in terms of taking the time for reflection and considering understanding before moving on.

Instruction on Enhanced Answer Key and Reflection Question Use

In the second research question, we asked, “To what extent does providing instruction on use of enhanced answer keys and reflection questions influence undergraduate biology students’ use of the scaffolds and course performance?” Overall, we found students who received the instruction (interviewed students) performed significantly better in the course and reported using the enhanced answer keys in significantly different ways than students who did not receive instruction (all other students in the course). This is noteworthy, given that we did not see significant differences between students who were interviewed and the rest of the class in either GPA or ACT composite scores (see data in Design and Methods). Further, almost all reflection question users were students who had received instruction on their use.

First, while interviewed and noninterviewed students did not significantly differ in their reported uses of the enhanced answer keys on either survey 1 or survey 2, by survey 3, the interviewed students (received instruction) reported using the enhanced answer keys in significantly different ways than noninterviewed students (did not receive instruction; t(29) = 3.92, p = 0.0002, d = 0.35). A greater percentage of interviewed students than noninterviewed students indicated they were using the enhanced answer keys for all assignments and exams and in ways that allowed them to understand biological concepts and engage in metacognition (Table 3). For example, 82% of interviewed students indicated they used the enhanced answer keys to study for an exam, while only 51% of noninterviewed students selected this response (Table 3). In addition, the interviewed students alone were using the enhanced answer keys in significantly different ways by survey 3 compared with survey 1 (t(29) = −2.06, p = 0.02). By survey 3, more interviewed students were using the enhanced answer keys to verify why an answer was correct (survey 1 = 33%, survey 3 = 45%), to better understand a question and answer (survey 1 = 67%, survey 3 = 73%), and to study for exams (survey 1 = 67%, survey 3 = 82%). This was not the case for noninterviewed students, who had no significant difference between the ways they used the enhanced answer keys on survey 1 compared with survey 3 (t(29) = 0.62, p = 0.27). Overall, this indicates that the interviewed students who received instruction on enhanced answer key use were using the answer keys in both more and different ways than the noninterviewed students who did not receive added instruction.

TABLE 3.

Percent of students agreeing with selected survey statements

Question	Response	Survey 3 interviewed	Survey 3 not interviewed
When have you used the assignments or exam answer keys?	All assignments	36%	20%
	All exams	27%	14%
How have you used the enhanced answer keys?	To better understand the question and answer	45%	34%
	To study for the exam	82%	51%
	To reflect on my understanding of the content	45%	14%

Second, interviewed students had significantly higher total course points ( = 859.7) than noninterviewed students ( = 806.55; t(78) = 2.56, p = 0.006, d = 0.76), and interviewed students also scored significantly higher on overall assignment totals ( = 151.7) than noninterviewed students ( = 139.2; t(78) = 2.28, p = 0.01, d = 0.66). Interviewed students did not score higher on the first assignment ( = 13.8) than noninterviewed students ( = 13.4), but this assignment was completed before students had seen any enhanced answer keys and before any had received instruction on enhanced answer key use during an interview (t(78) = 0.61, p = 0.27, d = 0.15). Interviewed students did score significantly higher on the first exam (; t(78) = 2.93, p = 0.002, d = 0.86). This suggests that the instruction the students received in the interviews may have had a positive effect on their scores as early as the first exam.

Third, while not all interviewed students were initially using the enhanced answer keys, once they were shown an example, all but one saw the usefulness of the keys and planned to use them in the future (Table 4). Many interviewed students mentioned they did not know the enhanced answer keys existed before the interview, though the entire class was alerted to their existence and location online in class each time they were posted. For example, Jessica said, “Well, I didn’t know about the [enhanced answer keys] before I came [to the first interview] so that was helpful … because then I also used the [enhanced answer keys] to study” (Jessica, interview 2). In this way, the individualized instruction helped the students to understand that these tools were available to them and to see how they might use them.

TABLE 4.

Interview summary: number of students using enhanced answer keys and reflection questions

	Interview 1a		Interview 2b
	Group 1	Group 2	Group 1	Group 2
Used enhanced answer key before	1/10	5/10	7/9	8/8
Planned to use enhanced answer keys in the future	9/10	10/10	9/9	8/8
First interview conversation influenced answer key use	—	—	5/9	6/8
Used reflection questions when studying	—	—	1/9	8/8
Engaged in metacognition during discussion of enhanced answer keys	8/10	10/10	8/9	5/8
Engaged in metacognition during discussion of reflection questions	—	10/10	8/9	6/8

a Group 1 = regular enhanced answer key; group 2 = modified enhanced answer key.

b Both groups = modified enhanced answer key.

By the second interview, all but two students reported using the enhanced answer keys and all planned to use the enhanced answer keys moving forward (Table 4). In addition, more than half of the interviewed students mentioned that the conversation in the first interview had influenced their use of the enhanced answer keys (Table 4). For example, Amanda said the conversation about the enhanced answer keys influenced the way she studied “in terms of making sure I move forward with correct knowledge instead of moving with nothing” (Amanda, interview 2). Thus, drawing students’ attention to the enhanced answer keys in the interviews had an impact on how they viewed the scaffolds as tools to help them study and check their understanding.

The majority of the students who reported using the reflection questions were those who had participated in the interviews and had received instruction on their use. Of the students who completed the surveys, only 10 students who had not been interviewed indicated they had used the reflection questions. Students who used the reflection questions had significantly higher overall total course points ( = 855.2) compared with those who did not use the reflection questions ( = 809.4; t(67) = 2.34, p = 0.01, d = 0.58). This difference was due to a significantly higher assignment total (; t(67) = 3.1, p = 0.001, d = 0.79). Those students who used the reflection questions may have scored higher in the course because of the reflection question use or were more likely to use the reflection questions because they were more highly motivated students. This distinction will need to be explored further in future studies. In either case, this indicates that directed instruction on using reflection questions is needed to demonstrate their value and benefit to a larger number of students.

Second, the survey responses indicated that those who did not use the enhanced answer key additions did not know they existed, forgot they existed, or did not take the time to use them. However, some of the nonusers mentioned within their responses that they planned to use them or would consider using them in the future (survey 2 = 14 out of 41 nonusers; survey 3 = 15 out of 36 nonusers). This indicates that simply making enhanced answer keys and questions available to students is not enough and that, without more directed instruction on the use and value of the enhanced answer keys and reflection questions, students will not seek them out or remember they exist. However, the fact that students indicated they saw utility in them after they were reminded of their existence suggests students do see some value in the reflection questions that directed instruction could enhance.

The benefit of instructing students on use of the reflection questions was most apparent in the interviews. In the first interview, all of the students who received the modified enhanced answer key planned to use the keys in the future, and by the second interview, all of the students who received the modified enhanced answer key with the reflection questions were using them. By the second interview, all who had received instruction on the reflection questions in the first interview were using them at least informally, meaning that some already considered the reflection step a normal part of their studies. In the second interview, students who had received the addition in the first interview indicated that the conversation had influenced their use of the reflection questions and that the discussion about the reflection questions had influenced their use of the enhanced answer keys. For example, after our discussion, Nick realized the enhanced answer keys could help him to reflect on his understanding rather than just to see whether he got the answer correct:

I had never considered; I probably would have just used the [enhanced answer keys] to see whether I got it right or not. I guess I didn’t know how much the assignments were going to be like the short answers on the exam so I would have taken it for granted a little bit and they would have come straight out of left field if we hadn’t really talked about that and talked about using it as a reflection piece [emphasis added] and looking back and making sure I understand the concepts from it. (Nick, interview 2)

Daniel also said the reflection questions and the conversation “definitely did influence the way I thought about the [enhanced answer key], just because this sort of gave me an idea of really the purpose of the [enhanced answer key], so knowing the purpose is really helpful in knowing how to interpret it” (Daniel, interview 2). In both of these examples, our conversation about the enhanced answer keys helped the students to recognize additional benefits of the scaffolds beyond their initial impression. In particular, discussing the intended purpose of the enhanced answer keys supported students to better understand how they could use the scaffolds.

The interviewed students who indicated they did not think the addition of the reflection questions to the enhanced answer key would help also indicated that it was because they were already doing those types of reflection. For example, Rachel said,

I looked at those questions at the end of them, the reflection questions, but I kind of feel like I use those when I’m studying anyway, so I’m going through and I’m asking myself why was that the answer or how does that relate back to this question so I feel like I already did it already. So, maybe if it was at the end of it, I just kind of looked at it and said, yeah, I did that already. (Rachel, interview 2)

However, Rachel did still find value in the conversation about the reflection questions because

I was more aware of it than if we hadn’t talked about it.… I feel like I did spend more time on the [enhanced answer keys] afterwards than before I had seen this, I do spend more time on them. But, it’s not necessarily that I’m going through each individual question. (Rachel, interview 2)

Therefore, the instruction on reflection question use is an important additional scaffold to help students engage in self-regulated learning and reflection of their understanding.

Engagement in Metacognition

In the third research question, we asked, “In what ways do undergraduate students engage in metacognition when they use enhanced answer keys and/or reflection questions within an introductory biology course?” Overall, we found that students did engage in some aspects of metacognition when they used the enhanced answer keys and reflection questions. However, the extent to which they discussed engaging in each of Grotzer and Mittlefehldt’s (2012) three dimensions of metacognition differed between their use of enhanced answer keys and reflection questions.

More students indicated in open-­response survey questions that they used the enhanced answer keys to see why an answer was incorrect or to see what was wanted in the answer than students who indicated they used them to engage in metacognition. However, qualitative analysis indicated that ∼20% of students who answered open-response questions were using the enhanced answer keys to engage in metacognition in such ways as looking for feedback, looking for more information on what they did not know, or indicating that they used the enhanced answer keys to help with their learning or understanding. For example, one student said, “The [enhanced answer keys] help me conceptualize the information and reach a deeper understanding” (Chris, survey 3). Another student said, “Even when my answers were marked correct, I was able to see how the concept was explained in another way, which was also helpful” (Morgan, survey 3). In these ways, the enhanced answer keys helped students to think about the concepts in a different way that helped them to reach greater understanding.

Though these findings were from only a small number of students who reported using the enhanced answer keys in these ways, the results were further supported by the responses to the Likert-style survey questions, in which the majority of students agreed to statements that indicated they went beyond simply looking at the correct answers on the enhanced answer keys to seeing their use for determining what they needed to study further or to help them better understand a concept (Table 5). Table 5 shows only survey 1 responses to simplify presentation of the data, but these types of responses remained similar across all three surveys.

TABLE 5.

Descriptive statistics for selected survey 1 Likert-style statements about enhanced answer keys

	Averagea	Modea	Agree	Disagree
I can see how using an enhanced answer key could help me to determine what I need to study further.	1.76	2	45	1
I can see how using an enhanced answer key could help me to better understand a concept.	1.87	2	42	4
When I compare my answer to the answer on the enhanced answer key, I reflect on the differences and how I could improve my answer.	1.93	2	41	5
If I were to use the enhanced answer key on a question I missed, I would use it to try to reach greater ­understanding of the topic before the test.	1.89	2	44	2

a1 = strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree.

Second, as with the surveys, students discussed some nonmetacognitive uses for the enhanced answer keys in the interviews, such as to see the correct answer, to see what was wanted for a complete answer, or to use the enhanced answer key while studying because it had the correct answers or used particular key words. However, all but two students engaged in metacognition during the first interview, in which they were asked to discuss the ways they had previously used the enhanced answer key, to engage in a think-aloud task and compare the answers on the enhanced answer key with their own assignment answers, and to consider ways they would use the enhanced answer keys in the future. The think-aloud task, in particular, helped the students to use the enhanced answer key to increase their understanding of a concept they previously misunderstood or to understand a mistake they had made on the assignment. For example, during the think-aloud portion of the first interview, in which Austin was comparing the answers on his assignment with the enhanced answer key, he said,

I mean, for this number 3, I just wrote down the ABAbaB and they have that there are gametes for two genes and each with two possible alleles so that could be helpful in more understanding it. And for this one, number eight, I got it wrong. See, I just explained Figure F instead of wrote the conclusion. I mean I kind of understood that, but reading from this saying that plants with a petal length approximately eight centimeters were the most frequent, I just explained the distribution of petal length with an average height of eight centimeters… I think it helps me to better understand the question, kind of. So to see what they wanted and better understand the material a little bit. Because that one was kind of a simple one, but the others definitely go into more depth about the answers and help me to understand the question more. (Austin, interview 1)

For Austin, having that opportunity to read through the enhanced answer key and compare it with his own answer during the think-aloud task helped him to reach greater understanding. By engaging in this task, students were able to see how the enhanced answer keys could help their understanding and, by the second interview, all but four students engaged in metacognition when they discussed their use of the enhanced answer key (Table 4).

Third, all but one student discussed at least one of Grotzer and Mittlefehldt’s (2012) three dimensions of metacognition: intelligibility, plausibility, and wide applicability within the interviews, but not all students discussed all three dimensions. In trends across both interviews, students more commonly discussed plausibility (47% of metacognitive statements) than intelligibility (34% of metacognitive statements) and only infrequently mentioned wide applicability (19% of metacognitive statements). See the Supplemental Material (Appendix G) for examples of quotes from all three dimensions. Overall, this shows that the students were using the enhanced answer keys to engage to at least some extent in all three dimensions of metacognition.

While students engaged in various levels of metacognition throughout the interviews, all engaged in metacognition while discussing the reflection questions in the first interview, and all but three engaged in metacognition when discussing their use of the reflection questions in the second interview (Table 4). Every single student had at least one metacognitive statement in the first interview when they were discussing the reflection questions; however, this was not true for the second interview, which suggests they may need more sustained support to continue to use the reflection questions to engage in metacognition.

Finally, as with their discussion of the enhanced answer keys, the group of interviewed students discussed all three dimensions of metacognition when they discussed using the reflection questions, though not all students individually discussed all three dimensions. And, as with the enhanced answer keys, the group of students more often mentioned plausibility (36% of metacognitive statements) than intelligibility (25% of metacognitive statements). However, in contrast to the enhanced answer keys, students most often mentioned wide applicability when they engaged in metacognition while discussing reflection question use (39% of metacognitive statements). See the Supplemental Material (Appendix G) for examples of quotes from all three dimensions. This increase in discussion of wide applicability with the reflection questions over the enhanced answer keys may be because the reflection questions included a prompt for students to specifically consider the wide applicability of the ideas. While the reflection questions also included prompts for intelligibility and plausibility, most students were already engaging in those dimensions to some degree, while many were not considering wide applicability before they saw the reflection questions. In particular, when students engaged in wide applicability, they often considered how topics might appear on a different question on an exam. This is noteworthy, because this was one of the reflection questions listed on the enhanced answer key addition. In this way, students were utilizing one of the specific ways the reflection questions were intended to support them in engaging in metacognition.

DISCUSSION

Past work has shown that students can 1) learn to self-regulate their own learning, engage in metacognition, and use their own self-generated feedback; and 2) that doing so leads to improved learning and greater success in school (Sadler, 1989; Black and Wiliam, 1998; Pintrich, 2000; Bell and Cowie, 2001; Zimmerman and Schunk, 2001; Nicol and Macfarlane-Dick, 2006; Wood, 2009). However, before this study, empirical research had not yet shown how students could use enhanced answer keys as scaffolds to support them in engaging in metacognition and generating feedback to enhance their understanding within biology courses, nor had previous research shown factors that might influence that use, such as added reflection questions or instruction on their use. Findings from this study begin to address how enhanced answer keys and reflection questions can support students to engage in metacognition and understand biological concepts.

First, study results show how the use of enhanced answer keys and reflection questions can support students in this endeavor. Previous work has suggested that students can generate their own feedback through self-regulated learning and self- assessment and that scaffolds, such as rubrics, can play a role in evaluation of ideas and generation of feedback (Ross ; Boekaerts ; Zimmerman and Schunk, 2001; Nicol and Macfarlane-Dick, 2006; Reddy, 2007; Andrade, 2010). This engagement may help students take active control of their own learning and may be particularly useful in the self-reflection phase of self-regulated learning (Zimmerman, 2000; Nicol and Milligan, 2006). Here, we have extended this past work to provide evidence that enhanced answer keys and reflection questions can also be used to support students in engaging in metacognition and considering their understanding of biological concepts in response to self-generated feedback. As the semester progressed, students began to see the enhanced answer keys as important tools to help with studying and began to incorporate reflection steps as a normal part of their studying. Therefore, undergraduate courses should include these types of scaffolds to support students in engaging in these processes. More work is needed to examine variations to scaffolds and the most effective means to provide this type of support to students in order to maximize this scaffold effect.

Second, study results show that providing directed instruction on the use of scaffolds can further assist students in seeing the value of the scaffolds and engaging with them productively. Not only did students who received instruction on enhanced answer key use perform better in the course than those who did not, the added instruction helped students to use the enhanced answer keys in different and more metacognitive ways and also helped them to see previous misunderstandings and take steps to increase their understanding. These two components of metacognition and understanding are closely related, in that students can use metacognition to become aware of their own learning processes, which in turn helps students to perform better, have greater learning gains, and greater problem-solving abilities (Baird, 1986; Tobias and Everson, 1996; Rickey and Stacy, 2000; Anderson and Nashon, 2006; Schraw ; Young and Fry, 2008; Vukman and Licardo, 2009; Sandi-Urena ; Sinatra and Taasoobshirazi, 2011; Stanton ). Findings from this study show how the use of these particular scaffolds can additionally support that process. Therefore, directed instruction within courses on how to use the scaffolds, and the importance of doing so, can lead to student engagement with their own ideas and improved learning gains. However, while all of the students who received the reflection questions in the first interview reported using them by the second interview, few students who did not participate in the interviews indicated using the reflection questions in any way. This indicates that simply including the reflection questions with the enhanced answer keys may not be sufficient and that more directed attention to, and instruction on, use of the reflection questions is needed for students to see their value and begin to use them consistently. Further, more work is needed to examine how to scale up the instruction to a whole-class setting and to determine whether additional instruction on metacognition and the value of self-generating feedback is also necessary to have the greatest effect on students’ performance and learning gains.

Third, study results show how the enhanced answer keys and reflection questions can support students to engage in metacognition and consider their understanding. Panadero et al. (2012) had previously examined the use of self-assessment scripts to support students in engaging in self-regulation. The present study extends this work by showing how enhanced answer keys and reflection questions can help students to increase their understanding of the content and engage in Grotzer and Mittlefehldt’s (2012) three dimensions of metacognition: intelligibility, plausibility, and wide applicability. When they discussed the enhanced answer keys, students most often considered the plausibility of ideas and least often considered the wide applicability of ideas. Therefore, it may be necessary to consider additional ways within the enhanced answer keys to support students in considering the intelligibility and wide applicability of their ideas. When they discussed the reflection questions, students most often engaged in considering the wide applicability of ideas and least often engaged in considering the intelligibility of ideas. The reflection questions may have supported this wide-applicability consideration more than the enhanced answer keys, as some of the questions specifically prompted students to consider all three dimensions. In this way, the reflection questions were an important addition to the enhanced answer keys to help students consider this metacognitive dimension further. However, because students showed some engagement in metacognitive dimensions that were specific to the reflection questions provided, examining variations to the reflection questions and how students used each of the various prompts differently could advance understanding of the ideal prompts and reflection question format to maximize their benefit.

Finally, though the enhanced answer keys and reflection questions were available for all students for most of the semester, not all students reported using them by the end of the semester. One reason for this may be that not all students received directed instruction on their use and so did not understand their value in extending their learning and understanding. However, it may also be necessary to consider different types of instruction for different types of students. This aligns with work from Stanton , who reported a continuum of four categories of student engagement in metacognition among introductory biology students. Therefore, more work is needed to examine how best to support these various types of students in engaging in metacognition. Some students may require more specific direction on scaffold use and some may require more instruction on what metacognition is and how it can be valuable to their success. This will require analysis of variations to the scaffolds themselves and to the instruction provided to students on their use. Further, metacognitive skills develop gradually (NRC, 2000), so it may be necessary for students to have a prolonged exposure to the scaffolds as they gain both the content knowledge and experience needed to reflect on their own understanding.

In this study, we have shown that instructor-provided enhanced answer keys and reflection questions can help students to engage in metacognition, consider their own understanding of biological concepts, and use their self-generated feedback to take steps to reach greater understanding. We have also shown that providing students with instruction in addition to these tools can further support them in using the scaffolds and engaging in self-regulated learning. This is especially important in undergraduate-level biology courses in which students may have confusion and difficulty connecting individual facts with complex, dynamic systems (Lewis and Wood-­Robinson, 2000; Marbach-Ad and Stavy, 2000; Wilson ; Smith and Knight, 2012). By taking time to consider their own understanding, students can change alternative conceptions into more scientifically accurate conceptions, and instructors can support this process by providing students with opportunities to consider and refine their own knowledge (Handelsman ; Tanner and Allen, 2005; Haak ; Freeman ; Dolan and Collins, 2015). As such, this study has potential to inform design of introductory biology courses to integrate support for students to engage in these processes and design of professional development for instructors to begin to support students in these ways.