Short lesson plan associated with increased acceptance of evolutionary theory and potential change in three alternate conceptions of macroevolution in undergraduate students.

Undergraduates commonly harbor alternate conceptions about evolutionary biology; these alternate conceptions often persist, even after intensive instruction, and may influence acceptance of evolution. We interviewed undergraduates to explore their alternate conceptions about macroevolutionary patterns and designed a 2-h lesson plan to present evidence that life has evolved. We identified three alternate conceptions during our interviews: that newly derived traits would be more widespread in extant species than would be ancestral traits, that evolution proceeds solely by anagenesis, and that lineages must become more complex over time. We also attempted to measure changes in the alternate conceptions and levels of acceptance of evolutionary theory in biology majors and nonmajors after exposure to the lesson plan. The instrument used to assess understanding had flaws, but our results are suggestive of mixed effects: we found a reduction in the first alternate conception, no change in the second, and reinforcement of the third. We found a small, but significant, increase in undergraduate acceptance of evolutionary theory in two trials of the lesson plan (Cohen's d effect sizes of 0.51 and 0.19). These mixed results offer guidance on how to improve the lesson and show the potential of instructional approaches for influencing acceptance of evolution.

INTRODUCTION

Undergraduate students commonly misunderstand many important biological concepts, even after intensive instruction (Brumby, 1984; Crawford ; Nehm and Reilly, 2007). Decades of research in teaching evolution, physiology, and physics (Bishop and Anderson, 1990; Hestenes ; Windschitl and Andre, 1998; Nehm and Reilly, 2007; Nehm and Schonfeld, 2007) have shown that simply using terms correctly is insufficient to displace alternate conceptions and that students must reflect on and compare both correct and alternative concepts (Posner ).

Student alternative conceptions are common throughout biology; however, few topics are as well explored as evolution. Research on topics such as natural selection (Bishop and Anderson, 1990; Settlage, 1994; Jensen and Finley, 1996; Ferrari and Chi, 1998; Nehm and Reilly, 2007, Abraham ), tree-thinking (Baum ; Meir ), and genetics (Lawson and Thompson, 1988; Smith ) helped document and quantify the prevalence of student confusions in evolution. Some also tested the efficacy of instructional interventions on student understanding of these concepts. For example, studies on student acceptance of natural selection before and after instructional interventions found that many of the common misconceptions, such as Lamarckian inheritance or “need”-based genetic change, can be corrected through targeted instruction (Bishop and Anderson, 1990; Nehm and Reilly, 2007; Abraham ).

Compounding these problems of understanding is one of acceptance. Acceptance is distinguished from belief in that acceptance is based primarily on an evaluation of evidence, whereas beliefs refer to intuition, faith, and personal principles (Smith, 1994; Southerland ). Although instructors may not be concerned with their students' beliefs, one would hope that students develop a better understanding of not only content but also how that content supports scientific theory. The importance of this evidence-based acceptance underlies biologists' frustrations not only with those who reject evolutionary theory based on beliefs, but also those students who blindly support it but are ignorant of the reasons why biologists accept it.

Evolutionary theory is one of the few areas of science in which there is both wide-scale rejection and acceptance based on nonscientific concerns. Many studies have sought to document levels of acceptance of evolutionary theory in undergraduates (Ingram and Nelson, 2006; Nadelson and Sinatra, 2010; Wiles and Alters, 2011), teachers (Ellis, 1983; Koevering and Stiehl, 1989; Osif, 1997; Rutledge and Mitchell, 2002; Nehm and Schonfeld, 2007; Nadelson and Sinatra, 2009, 2010), and the general public (Miller ). These studies have all shown relatively low acceptance of evolution in the United States, despite the overwhelming evidence for evolution gathered by scientists. For instance, Miller found that acceptance of evolution declined among U.S. adults from 45% to 40% over a 20-yr period. Even among biology majors, acceptance is surprisingly low. Ingram and Nelson (2006) found average acceptance of evolution to fluctuate around 64% in a population of junior and senior biology majors entering an evolutionary biology course.

Acceptance of evolution is variably correlated with understanding of evolutionary concepts, among other factors (Smith, 2010; Wiles and Alters, 2011). Some researchers found positive correlations between understanding of evolution and acceptance (Trani, 2004). Achievement in an evolutionary biology course was weakly related to acceptance of evolution in another study (Ingram and Nelson, 2006). In other studies, students showed learning gains in evolutionary topics, such as genetic drift and natural selection, without changing their level of acceptance (Bishop and Anderson, 1990; Sinatra ; Southerland and Sinatra, 2003). Some have proposed that this can be attributed to microevolution being relatively nonthreatening to the evolution denier, thereby failing to engage cognitive dissonance and reevaluation of his or her knowledge constructs (Catley, 2006; Nadelson and Southerland, 2010). Nadelson and Southerland (2010) did, however, find significant relationships between acceptance of evolution and performance on their assessment of macroevolutionary understanding, and suggested that instruction in macroevolution may be linked to greater acceptance of the scientific basis of evolution.

Student understanding of the nature of science (NOS) and the evidence for evolution may also be positively correlated with acceptance of evolution. Lombrozo found a significant correlation between NOS understanding and acceptance of evolution in a population of undergraduate students, even when controlling for previous instruction in science and interest in science. Wiles and Alters (2011) also measured acceptance of evolution before and after a short course for gifted high school students. This course addressed a suite of factors that might influence acceptance of evolution, including understanding of the NOS and knowledge of the evidence for evolution. They found gains in acceptance of evolution immediately and 1 yr after students completed the course. Another group of high school students, including those classified as acceptors and as rejectors of evolution, discussed the importance of teaching evolution in terms of evidence and theories; the authors of that study suggest that framing instruction in terms of the NOS and evidence may be a promising approach (Donnelly ).

In this paper, we report on the development of a lesson plan that includes instruction on macroevolutionary patterns, the testability of scientific theories, and some of the evidence for evolution. We also discuss potential alternate conceptions we found when interviewing major and nonmajor Boston-area undergraduate students during the lesson plan development (Table 1). In addition, we report on rates of acceptance of evolutionary theory before and after exposure to the lesson plan in major and nonmajor undergraduate students from institutions in the Pacific Northwest and the University of Wisconsin, La Crosse (UWL). Finally, we include results that are suggestive, but not conclusive, of mixed effects of the lesson plan on the prevalence of the identified alternate conceptions in major and nonmajor students at UWL.

Table 1.

A brief description of the student alternate conceptions identified in this study and mapped test items designed to measure those alternate conceptionsa

Alternate conceptions	Description	Test items
EC	Species arise in succession and replace their ancestors, suggesting a chain or ladder model of evolution (anagenesis). Students do not describe cladogenesis.	6, 9
LC	Lineages must increase in complexity over time; stasis in form is evidence against evolutionary theory.	3, 7
NTW	Newly derived traits are expected to be more widespread within a clade than are older derived traits, even in instances when the newer trait arises after lineage divergence.	10, 12, 13

aThe test items can be found in Supplemental Material A.

METHODS

This study comprised three distinct phases. In phase I, we conducted interviews with students to support the development of the lesson plan and to explore undergraduate student conceptions of macroevolution and the evidence in support of evolutionary theory. We also began development of test items to measure content understanding. In phase II, we tested different implementations of the lesson plan with undergraduate students, and collected information on student acceptance of evolutionary theory before and after exposure to the lesson plan. Finally, in phase III, we administered the lesson plan in large introductory courses for biology majors and nonmajors; students completed pre- and postinstructional exams on content understanding and acceptance of evolution. We discuss each of these phases below. We were granted institutional review board approval for this study by review boards at UWL and the Massachusetts Institute of Technology.

Phase I: Boston-Area Interviews

From September 2008 to August 2009, we conducted a series of interviews with 46 Boston-area undergraduate students from a range of institutions, including a 2-yr community college and 4-yr public and private universities. Subjects represented a range of backgrounds; the sample included major and nonmajors of varying academic levels, both genders, and a range of ethnicities (Table 2).

Table 2.

Demographics of study participants in phases II and IIIa

		Phase II						Phase III
		Total (n = 41)	Majors (n = 26)		Nonmajors (n = 15)		Total (n = 622)	Majors (n = 478)		Nonmajors (n = 144)
Category	Response	Number	Number	Percent	Number	Percent	Number	Number	Percent	Number	Percent
Gender	Female	27	18	69.23	9	60	392	315	65.90	77	53.47
	Male	14	8	30.77	6	40	229	162	33.89	67	46.53
Race/ethnicity	American Indian	0	0	0	0	0	6	6	1.26	0	0
	Asian	19	11	42.31	8	53.33	26	17	3.56	9	6.25
	Black	0	0	0	0	0	3	3	0.63	0	0
	Hispanic	4	3	11.54	1	6.67	19	15	3.14	4	2.78
	Other	1	1	3.85	0	0	5	5	1.05	0	0
	Pacific Islander	0	0	0	0	0	1	1	0.21	0	0
	White	17	11	42.31	6	40	557	428	89.54	129	89.58
Language other than English in household
	No	Data not collected					554	430	89.96	124	86.11
	Sometimes						38	32	6.69	6	4.17
	Always						27	14	2.93	13	9.03
College level	Beginner						607	467	97.70	140	97.22
	Advanced						14	10	2.1	4	2.78

aSubjects in both phases were asked to report gender and race/ethnicity. Subjects in phase III were also asked about language use in their household when growing up and the number of college-level biology courses they completed. We categorized those subjects who reported two or fewer college-level biology courses as “beginner” and those who reported three or more college-level biology courses as “advanced.” Some subjects did not respond to all of the demographic questions. Thus, subgroup totals do not always match the total number of study participants. Percentages are calculated from the total sample in majors or nonmajors.

We (J.K.A. and J.C.H.) developed oral and written interview questions to capture common student alternate conceptions around the evidence for evolution and related macroevolutionary concepts. The interview questions were constructed to help us explore subjects' ideas about the evidence for and against support of evolutionary theory, their interpretation of evolutionary relationships between organisms, and their knowledge of scientific inquiry approaches. An example of interview questions used in an early interview is included in Supplemental Material A. Although the interview protocol varied throughout phase I, we generally opened the interview by presenting students with written open-response or multiple-choice questions. We often presented students with a beta computer-based version of the lesson plan during the interviews. Throughout each interview, we asked subjects to revisit their answers to the questions and to verbally explain their responses.

The interviews provided insight into some potential alternate conceptions about the evidence for evolution and macroevolutionary topics; we chose these alternate conceptions, because they are related to the topics covered in the lesson plan. For instance, when asked for examples of evidence that would help refute evolutionary theory, some examples of typical student responses were:

Student 1: “Fossils of animals that looked exactly the same as those of animals living today”

Student 2: “Fossils that over time show no changes in organisms”

The idea that stasis in a lineage is evidence of a flaw in evolutionary theory was volunteered by several subjects. Morphological stasis is common in the fossil record; apparent complexity can remain stable or even decrease in lineages (Gould and Eldredge, 1977; Eldredge ). For instance, habitat tracking or a high degree of spatial structuring in widespread species could lead to morphological stasis in a lineage (Eldredge ). We interpreted these responses as an indication that those subjects thought that lineages must become more complex (LC) over time; this alternate conception appears to impact student interpretation of cladograms (Meir ).

Students were also asked to verbally explain their correct and incorrect written responses. For example, two student written responses and verbal explanations to an interview question are shown below:

Question: Among the features of the chimpanzee's skull are: Trait A) a postorbital plate that ventrally separates the orbit and temporal fossa; Trait B) enamel caps on the molars. Trait A is found in fossils as old as 60 million years, whereas Trait B first appears in fossils just 5 million years old. Which of these traits do you think is more widespread among other present-day primate species? Why?

Student 3 written response: “Trait A, because it has been around for a long time.”

Student 3 verbal explanation: “It is believed from the current evolutionary theory that we all diverged out of one common ancestor which means that everyone has something in common, we all have something in common with a common ancestor. So, if Trait A has been around for a longer time that means it has more time to spread throughout all different types of trees of animals and organisms, whereas Trait B hasn't been around for that many years, so it maybe is just contained within one species and it not able to spread out beyond that.”

Student 4 written response: “Trait B, because it appeared in fossils closer to the present day, it is more likely to be found in present day primate species.”

Student 4 verbal explanation: “Since Trait B appeared sooner to, closer to now, it would be present more now. Trait A was further back so it might have just died out. That is what I thought.”

Explanations such as that from student 4 suggest an alternate conception that newly derived traits would be more widespread (NTW) throughout a clade than ancestral traits. Responses such as this persisted into the final version of the content test, in which we added information on the estimated time of divergence between the lineages after the emergence of the postorbital trait.

Some students, such as student 4, described traits as if they had a life span. Other students often related the likelihood of a trait being widespread with the perceived functionality of the trait; for this reason, we altered item 12 in the content test (a modified version of the chimpanzee interview question above) to include information on the functionality of the two traits (Supplemental Material B). One student quote from phase III is particularly telling:

Student 5: “[I]n lab I learned that older traits would be more widespread, but I still think survival is more important.”

For this student, the functionality of the tooth enamel cap made it appear more likely that the trait would be shared with other taxa in the clade, even though the taxa split millions of years prior to the development of that trait. Since this alternate conception was not one that we had encountered before or in the literature, we included an open-response section on item 12 to continue to capture student written responses (Supplemental Material B).

We interpreted student perceptions of transitional fossils as evidence of an additional alternate conception. When asked to define the term “transitional species,” one student responded:

Student 6 verbal response: “My explanation would be like maybe a species that evolved so that another species could evolve from that, and maybe the transitional species was not successful, so that is why it evolved to another, it changed its form and died off, and the one it changed into survived. It was a transition between two times.”

Interviewer: “So, based on your definition, is it fair to say that a transitional species is the ancestor of a modern species?”

Student 6: “No, because I don't…I think there is something about the transitional species not being successful, but…I don't know, I'm not sure.”

Interviewer: “Then…would it be possible for a transitional species to coexist with the modern species?”

Student 6: “Possibly, but I don't think so. I think the transitional species is part of getting from A to B, so what happens in the middle is just how it gets there, so I wouldn't really think so. No.”

When asked the same question, another student replied:

Student 7 verbal response: “If I had to guess, I would probably say between the evolutionary stages of two animals…the fossil that came in between them.”

Interviewer: “What do you mean by evolutionary stages between two animals?”

Student 7: “If you believe in evolution, then like between a whale fossil and whatever land animal came after that. The transition in between, then…that animal's fossil.

Interviewer: “Say we had an animal living right now, and an animal living 60 million years ago. Where would the transitional fall into that timeline?”

Student 7: “An animal that went extinct, but leaded [sic] up to the animal we have now.”

Interviewer: “The transitional fossil…would that be an ancestor of what we have now?”

Student 7: “Yeah, I think.”

Students, such as the two shown above, commonly described transitional species as ancestors of extant species, or described one species changing into another, often due to a need. We interpreted these types of responses as an indication that students are solely referencing an evolutionary chain (EC) model, to the exclusion of an evolutionary branching tree model, when thinking about macroevolutionary patterns (Mead, 2009; Meikle and Scott, 2010). Transitional species need not be ancestors of extant species and could co-occur with more modern forms. A brief summary of the three alternate conceptions we identified through our interviews and the test items that match those alternate conceptions can be found below (Table 1).

When we began this study, we were unaware of any published instruments designed to measure the concepts covered in our lesson plan. Therefore, J.K.A. and J.C.H. created isomorphic multiple-choice and multiple true–false versions of some of the interview questions to use as pre- and postinstructional tests. We varied the internal ordering of the two multiple true–false item sets between tests; all other items were in the same order between versions. We also included one open-response item (item 12) so students could better express their conceptions of trait nesting. The version of the pretest used in phase III is included in Supplemental Material B.

Description of the “Evolutionary Evidence” Lesson Plan.

J.C.H. designed the Evolutionary Evidence lesson plan; we modified aspects of this lesson plan based on observations made during the interview process. This lesson plan covers some of the evidence for modern evolutionary theory and emphasizes the predictive nature of scientific theories. A detailed version of the original lesson plan is available in Supplemental Material C, with a summary given below.

The initial exercises in the lesson plan have students compare the patterns of relationships predicted from common descent with those expected from deliberate design. In the first exercise, students are asked to organize seven extant taxa (e.g., nudibranch, fish, bird) into groups based on whatever criteria they see fit. In the second exercise, students act as designers, and modify representative lizards from five allopatric populations by adding different traits (e.g., neck frills). It is important to note that the trait states are binary in nature. Students then organize their lizards based on shared traits and determine whether any pattern emerges. In the third exercise, the rules change, such that lizards retain their ancestral traits. Students again add traits to individual lizards and organize their lizards; at this stage, they are introduced to the idea of trait nesting, in which closely related lizards share more traits than do two distantly related lizards (Figure 1). Students are asked to compare the patterns that emerge from their simulation of deliberate design with their simulation of descent with modification, and to comment on the differences between the two scenarios.

Figure 1.

A screenshot from the third exercise of the computer-based implementation of the lesson plan. This exercise is intended to demonstrate that organisms that evolve via common descent form nested sets similar to those of living organisms. In this exercise, students cause evolution and migration of populations of lizards. The simulation allows generations to pass and traits to fix in populations. Students then arrange sample lizards from each population based on the shared morphological traits among lizards. They then circle the groups sharing similar morphological traits and label each trait. Students compare the pattern they observe in this exercise (nested sets of traits) with the patterns they observed in a previous exercise that simulated special creation.

The next series of exercises shifts from nested traits to prediction of the order in which traits would appear in the fossil record, assuming common descent. In the fourth exercise, students are provided with representative lizards from seven populations. Students are not provided with any information on the evolutionary relationships between these lizards. The students examine the populations, group representative lizards based on their shared traits, and make predictions about the order of appearance of some of those traits in a lizard's evolutionary history. Once students make their predictions, they “dig” for fossils through simulated soil to test their predictions. This same approach is repeated in the fifth exercise, in which students return to the original seven extant species, reorganize them based on their shared traits, and examine selected examples from the real fossil record, which are annotated to describe traits. The lesson plan links the students' work to published work on the relationship between predicted fossil appearance and actual measurements from the fossil record (Norell and Novacek, 1992; Benton and Hitchin, 1997; Benton, 1998). The lesson plan ends with a discussion of the significance of transitional fossils for inferring evolutionary relationships between different lineages.

Phase II: Pacific Northwest Pre/Post Testing

In this phase of the research study, we collected information on the efficacy of the lesson plan for teaching our target concepts to major and nonmajor undergraduate students. We also measured acceptance of evolutionary theory in those students before and after exposure to our lesson plan. Finally, we were interested in whether presentation mode (lecture-based vs. computer-based) impacted their understanding of the concepts.

We used the Measure of Acceptance of the Theory of Evolution (MATE) to assess acceptance of evolutionary theory. The MATE is a commonly used 20-item, Likert-scale instrument (Rutledge and Warden, 1999; Rutledge and Sadler, 2007). It should be noted that although the MATE is used extensively and has excellent internal and test–retest reliability, some have expressed concern about the actual construct measured by the MATE (for details, see Smith, 2010). Originally developed for use with high school biology teachers, the MATE has been used to study various other populations (Rutledge and Warden, 2000; Trani, 2004), including high school students (Cavallo and McCall, 2008), nonmajors biology students (Rutledge and Sadler, 2007; Moore ; Cotner ), and upper-level students in an evolutionary biology course (Ingram and Nelson, 2006).

We recruited 54 subjects from 2- and 4-yr institutions in the Pacific Northwest in March 2009 (Table 2). This phase was independent of any coursework the subjects were taking at the time. We paid subjects for their participation. We first administered paper-and-pencil versions of the content and MATE tests to subjects 1–2 wk before our instructional intervention. During these sessions, we collected demographic information from the subjects and asked them to complete the tests. We randomly assigned subjects to the lecture or computer-based instructional treatment at that time.

We held instructional interventions (either lecture or computer) 2 d apart in late March 2009. The lecture- and computer-based instruction followed the same lesson plan, but students in the lecture interacted with the lecturer and with one another. We administered paper-and-pencil versions of the posttests immediately following the instructional intervention. Forty-one subjects completed both the pre- and postinstructional test.

In both the pre- and postinstructional testing, subjects were allowed as much time as they needed to complete the exams. Data on completion times were not systematically collected, but subjects regularly completed the exams in 15–25 min. Since students were enrolled in different courses at their respective institutions and were at different academic levels, the only consistent instruction on evolutionary topics during that time period was the instructional intervention we administered.

After phase II, K.E.P. and N.D. reviewed the test items and offered suggestions for improvement. We reduced the length of the pre- and posttests and revised the language and format of several questions. While the results from the content tests are not included, we present student pre- and postinstruction MATE scores from phase II below in Figure 3A.

Figure 3.

Pre- and postinstruction MATE scores for all students who took both tests in (A) phase II (n = 41) and (B) phase III (n = 622). Student MATE scores increased significantly in both phases II and III after completion of the lesson plan. MATE categories from Rutledge and Sadler (2007).

Phase III: UWL, Pre/Post Testing

In this phase of the study, we collected pre- and postinstructional data from students enrolled in a majors or nonmajors introductory biology course in Fall 2009. Seven hundred twenty students signed the IRB-approved consent form and took both the pre- and posttests. Ninety-eight of those students were removed due to nonresponse to test items on the MATE or content tests, leaving a total of 622 students (144 nonbiology major [nonmajors] and 478 biology/pre--health professions students [majors]) in the study. Demographic data for these subjects are shown in Table 2.

Course Description.

General Biology is a 100-level course taken by biology and health professions majors at UWL. In Fall 2009, the class consisted of eight lecture sections with a maximum of 95 students each, with a total of 781 students enrolled in the course. The six course instructors met weekly and shared a unified set of teaching materials. Five of the instructors each taught one section and one instructor taught three sections of the class. The course included sections on ecology, cell biology, life cycles, genetics, and evolution, in that order. The textbook used was Biology (Campbell and Reece, 2007). The lecture material was arranged in a series of learning cycles, with short segments of lecture interspersed with problem-solving and clicker questions on basic concepts.

Introductory Biology is a 100-level course taken by nonmajors at UWL. The textbook used was Life on Earth (Audesirk ). This course (228 students) follows the same order of topics as General Biology, and was taught by three instructors, each teaching one section using a unified set of teaching materials. The topics and order of the two courses are relatively constrained, as they must match in sequence to the shared laboratory sequence.

The courses differ primarily in depth of content covered and emphasis on biological patterns (nonmajor) versus biological mechanisms and processes (major). Students enrolled in either majors or nonmajors biology share a single 2-h laboratory course. That semester, there were 11 nonmajors labs and 28 majors labs; each lab had a maximum of 24 students. A total of 16 different lab instructors, a mix of master's-level graduate students and instructional academic staff, taught in the lab sections. Lab content matched that of the lecture, with multi-week experimental labs on ecology, cell biology, life cycles, genetics, and a single lab on evolution (replaced with the lesson plan in this study).

Study Description.

In this phase, the lesson plan was administered solely as a computer-based virtual laboratory. Lab instructors performed the virtual lab activities and were instructed in administration of the lab by K.E.P. the week prior to teaching their lab sections. They were instructed to introduce the virtual lab with minimal background material, merely demonstrating to the students how to run the program. Students worked through the lab in their usual lab groups of up to four students and answered question prompts in the workbook. To ensure completion of the entire simulation activity, workbooks were turned in at the end of class and graded as a regular part of the laboratory grade.

Lab instructors administered the pretest (content questions and MATE) in the last 30 min of the regular laboratory period in lab the week before students performed the simulation lab. We gave students credit for completing the pretest, but they were not given a grade based on their performance. The following week, we administered the lab. We made the postinstructional test (content and MATE) available immediately after the last lab section performed the lab. We offered the postinstructional test as a quiz through the online course management software. Students in both classes had taken a minimum of 12 online quizzes through D2L over the course of the semester so they were quite familiar with its operation. To encourage participation, we gave the students five bonus points (out of 800 total course points) for completion of the online test.

Students who had lab on Monday had a 5- to 7-d interval between the pretest and posttest. Students with Friday labs had a 0- to 2-d interval between tests. Both major and nonmajor labs were spread throughout the week. Students were asked to complete the posttest within 45 min, but were given as much time as they needed. We do not have data on average pretest completion times in phase III; however, we do know that in the labs 15–30 min was available to take the pretest, as the quiz followed completion of that week's lab activity. The average completion time for the posttest was 30 min, including three students who left the test open for more than 20 h. The median completion time for the posttest was 14 min.

While we would have preferred to give a paper posttest to replicate the pretest conditions, there was not time in the 2-h lab period to take a posttest after completion of the simulation. More critically for this study, there was not time to administer a test during lab to all students before they began to discuss evolutionary concepts in their lecture class. In the interest of completely segregating the effects of the lesson plan from the effects of lecture-based instruction in evolution, we chose to use an online posttest. The test was low-stakes (i.e., responses were not graded) and more akin to a power test than a speeded test; students were familiar with the online testing system, so the potential test-mode effects may have been reduced (Mead and Drasgow, 1993, Puhan and Boughton, 2007). However, we acknowledge that the change in test medium could have affected our results in phase III.

Data Analysis.

We calculated the prevalence of each student's alternate conceptions in phase III by dividing the total number of appearances of each alternate conception by the number of questions in which that alternate conception could be selected from the options. Students who failed to answer all of the questions tied to a particular alternate conception were removed from the analysis. The alternate conception data were checked for normality with Shapiro-Wilk tests and were found to be nonnormal. We compared pre- and postinstruction use of alternate conceptions with a nonparametric two-tailed Wilcoxon signed-rank test for paired samples. Mean pretest alternate conception performance between major and nonmajor students was compared with two-tailed Wilcoxon rank-sum nonparametric tests.

We compared mean student scores on the pre- and postinstructional MATE tests in phase II using a two-tailed Wilcoxon signed-rank test after the data were found to be nonnormal. We did not find any difference in performance on the MATE between treatment groups (lecture vs. computer-based instruction), so data for the two groups are pooled.

We used a two-tailed paired t test to compare pre- and postinstructional MATE scores in phase III. We also used a two-tailed Wilcoxon rank-sum test to compare mean change in MATE scores between majors and nonmajors. We did all analyses for student performance and MATE scores in R version 2.7.1 (R Development Core Team, 2008) using the exactRankTest extension (Hothorn and Hornik, 2010).

RESULTS

Student Alternate Conceptions

The reliability of both the full test and subtests for alternate conceptions did not demonstrate acceptable values (Cronbach's alpha < 0.7) to provide clear results. This is likely due in part to our decision to use a truncated version of the test, which included many binary-response items. We made these modifications to develop a relatively short exam that could be used in a typical course, in which time is precious. Unfortunately, this decision may have negatively impacted our reliability values.

For this reason, we do not report on student pre/post gain on the full test, nor do we argue that we have provided concrete evidence of the efficacy or shortcomings of the lesson plan in terms of alternate conceptions. However, on the basis of our initial interviews and what can be gleaned from subject responses to the content items, we feel we still gained some interesting insight into student conceptions and potential areas for improvement of the lesson plan. In the analyses that follow, we report results from the sets of questions associated with each alternate conception, with the understanding that the conclusions that can be drawn from this study are limited.

We found low to intermediate levels of the identified alternate conceptions (LC, EC, NTW). Overall, students selected an LC option an average of 16.4% (± 0.01 SEM of the time, an EC option 46.7% (± 0.02 SEM) of the time, and an NTW option 33% (± 0.01 SEM) of the time (Figure 2). However, nonmajor students selected LC options to a significantly greater extent than did major students (W = 32008.5, p = 0.021), averaging 21.4% (± 0.03) versus major LC usage of 14.9% (± 0.01). Usage of EC and NTW alternate conceptions did not differ between major and nonmajor students. Students also provided written explanations for their answers to item 12:

Figure 2.

Student performance on pre- and postinstructional content tests. Students exhibited significantly more instances of the LC alternate conception. There was no difference in student usage of the EC alternate conception. Students exhibited significantly fewer instances of the NTW alternate conception. Error bars represent ±1 SE. * = p < 0.05, *** = p < 0.0005.

The order Primates includes lemurs, lorids, galagos, tarsiers, monkeys, apes, and humans. Apes and humans split from other primates as long as 35 million years ago. Many primates feed on hard food. Among the features of the chimpanzee's skull are:

Trait A: A postorbital plate that ventrally separates the orbit and temporal fossa (performs no known survival function, but does not cause any harm).

Trait B: Enamel caps on the molars (help protect the teeth from damage while eating hard foods).

Trait A is found in fossils as old as 60 million years, whereas Trait B first appears in fossils just 5 million years old.

Which of these traits do you think is more widespread among other present-day primate species?

A: Trait A

B: Trait B

Of those students who selected the incorrect response (option B; NTW alternate conception), 53% wrote that they chose the newer trait because it served a survival function and therefore would be more widespread.

We found significant changes in the usage of two of the three alternate conceptions postinstruction; there was no change in usage of EC, the most common alternate conception (Figure 2). Postinstruction LC usage significantly increased, on average, to 19.6% (± 0.01 SEM) (Wilcoxon signed-rank test statistic V = 11,196.5, p = 0.029), while average postinstruction NTW usage significantly decreased to 18.8% (± 0.01 SEM) (V = 47,118.5, p < 0.0001) (Figure 2). Changes in the appearance of the three alternate conceptions did not differ significantly between major and nonmajor students.

Acceptance of the Theory of Evolution

Student scores on the MATE increased significantly pre- to postinstruction in phase II (Figure 3A). Subjects averaged 84.79 (± 1.94 SEM) on the preinstruction MATE. Student scores increased significantly postinstruction (V = 503, p = 0.007) to an average of 90.79 (± 1.91 SEM). The Cohen's d effect size for instruction in phase II was 0.51.

Subjects in phase III scored lower on the MATE test than did subjects from phase II, averaging 72.5 (± 0.52 SEM) on the MATE test preinstruction. Nonmajors averaged lower on the preinstruction MATE than did majors, scoring an average of 70.8 (± 1.14 SEM) compared with 73 (± 0.58 SEM; W = 30602, p = 0.044). We found that overall student scores on the MATE test significantly increased (t = 7.48, p < 0.0001) to an average of 75 (± 0.52 SEM) after completing the Evolutionary Evidence lesson (Figure 3B). We found no significant difference in mean score change from pre- to postinstruction between majors and nonmajors. The Cohen's d effect size for instruction across students in phase III was 0.19.

DISCUSSION

In this study, we explored several alternate conceptions through student interviews, designed a lesson plan to help undergraduate students engage with some of the evidence for evolutionary theory, and attempted to measure pre- and postinstructional prevalence of alternate conceptions and student acceptance of evolutionary theory. In this section, we highlight the probable strengths and weaknesses of the lesson plan in terms of student alternate conceptions. We also discuss the observed gains in student acceptance of evolution associated with use of the lesson plan, as well as some ways in which our efforts could be improved in the future.

We observed a large and significant decrease from preinstruction (33%) to postinstruction (18.8%) in the prevalence of the alternate conception that newly derived traits would be more widespread across a clade than older traits (NTW). This is encouraging, given the prominence of trait nesting as a theme in our lesson plan and the clarity with which the NTW alternate conception appeared in the interview and testing. Because one item in this question set also asked subjects to explain their selection, we were able to capture additional information on this alternate conception. More than half of the subjects who selected an answer corresponding to the NTW alternate conception on that item offered an explanation based on the functionality of the trait in question. We speculate that this tendency to reference functionality may be linked to the common misconceptions that traits arise out of need and that disused traits quickly disappear from populations (Gregory, 2009), but have no data to support this idea.

In contrast, the alternate conception that lineages become more complex over time appeared to be reinforced by the lesson plan, increasing from 16.4% to 19.6% of students. This undesired impact, if real, may be due to a design decision in the lesson plan, in which lizards continually gain traits and extant organisms range from unicellular to higher organisms. Adding traits (e.g., no horns/horns) is an easy method of visually showing evolutionary change in populations. Indeed, organisms in other commonly used evolution labs, like the Caminalcules (Gendron, 2000) and the Great Clade Race (Goldsmith, 2003), primarily gain traits, with few or no traits becoming reduced. However, an alternative approach is to show trait swaps (e.g., gray feathers/white feathers), rather than only trait gains (e.g., Meir ); we believe this approach may help to avoid reinforcement of the problem. Including both trait losses and gains in the lesson plan may also influence the propensity of students to use alternate conceptions when explaining patterns initially (Nehm and Ha, 2011), thereby increasing the likelihood of cognitive dissonance as they complete the lesson. In addition, the introductory and concluding exercises may inadvertently emphasize complexity, since students are asked to arrange organisms ranging from amoebas to birds. More explicit discussion of evolutionary patterns of complexity may be helpful.

We found no indication that the third alternate conception, that species in a lineage arise in linear succession (EC), changed from pre- to postinstruction. This particular alternate conception is often associated with creationist arguments about the failings of evolutionary theory. A common creationist argument, “If people came from monkeys, why are there still monkeys?,” indicates an EC mental model focused exclusively on anagenesis, rather than one that also includes cladogenesis (Meikle and Scott, 2010). Following our phase I interviews, we believed that student assumptions about the relationship between transitional and extant species were indicative of an EC mental model. Although transitional fossils are not the primary focus of the lab, they are specifically addressed in the final exercise. We hoped that this treatment would be sufficient to spark change in the prevalence of EC. However, the links between confusion about transitional fossils and EC mental models are potentially less clear than we assumed, and our items were insufficient to measure this alternate conception. We hope to better explore this alternate conception through additional student interviews and improved testing in the future. In any case, confusion about the role of transitional fossils in evolutionary biology is common (Mead, 2009), so we plan to provide a more thorough treatment of them in future versions of the lesson plan.

Student Acceptance of Evolution

Acceptance of evolution as a scientific explanation for the origin of biodiversity is unacceptably low in the United States. Surveys of the general public indicate acceptance rates of between 40% and 50% (Pew Research Center, 2005; Miller ). Fuerst (1984) found on average 62% of college students stated that they “believe in evolutionary theory.” This number ranged from 50% in first-year nonmajors to 84% among biology graduate students. Bishop and Anderson (1990) found 59% of nonmajors responded positively to the question: “Do you believe the theory of evolution to be truthful?” In a study of college seniors enrolled in evolutionary biology classes, Ingram and Nelson (2006) found over three semesters that 61%, 62%, and 67% accepted evolution. These data provide the big picture of expected levels of belief/acceptance of evolution, but it is difficult to directly compare these data, since each was generated using unique survey methods.

A growing number of studies use the MATE as a standard test, which allows for direct comparison across populations. Rutledge and Warden (2000) surveyed Indiana public high school biology teachers and found a moderate level of acceptance of evolutionary theory, with an average score of 77.6 on the MATE. High school teachers in Oregon appear to have a higher level of acceptance, with an average of 85.9 on the MATE (Trani, 2004). Public high school students in Michigan have a lower average MATE score of 66.7 (Cavallo and McCall, 2008). Wiles and Alters (2011) recently measured pre- and postinstructional acceptance of evolution with the MATE in a special summer course for a gifted high school student population; students entered the course averaging 72.89 on the test. In comparison, our nonmajors and majors averaged 70.8 and 73.0 respectively on the MATE test preintervention in phase III. Undergraduates (majors and nonmajors) in phase II averaged a much higher score of 84.8.

Three other studies allow direct comparison with a single MATE question: “Evolution is a scientifically valid theory.” Moore and Cotner (2009) examined the effects of evolution and creationist instruction in high school biology curricula on MATE scores of nonmajor college students. They found the percentage of students who agreed or strongly agreed with this statement across treatments was 64.58%. Using a very similar question, which asked students whether they accepted, rejected, or conditionally accepted evolutionary theory as scientifically valid, Robbins and Roy (2007) found 59% of students agreed with this statement in a nonmajor course. These results are both much higher than UWL nonmajor preinstruction responses to the above question (50%) and similar to UWL major responses preinstruction (59%). Johnson and Peeples (1987) asked the same question as on the MATE test and found the average of responses, on a five-point scale, was 3.12; our preinstructional mean across all students on that question was 3.49. Thus, at the beginning of our study, both the nonmajor and major students seem to fall toward the lower end but are still in the middle of the range for acceptance of evolutionary theory as observed across the United States.

The change in MATE test scores observed in phases II and III were moderate or small, respectively, but highly significant. The magnitude of change in acceptance in our study compares well with those recorded in a longer intervention. The course-based intervention described in Wiles and Alters (2011) was associated with a gain of 13 points on the MATE, from 72.89 to 85.88 immediately following the intervention. Those subjects who responded a year later scored a mean of 83.92. In our study on the single intervention lesson plan, we saw gains of 5 and 2.5 points on the MATE in phases II and III, respectively.

Robbins and Roy (2007) tested a four-lesson, inquiry-based intervention with lab and lecture components on a nonmajors class in Ohio. They found an increase from 59% to 92% of nonmajors agreeing unconditionally with evolutionary theory, a gain of 33 percentage points pre- to postinstruction. In our study, 59% of majors and 50% of nonmajors indicated agreement or strong agreement with this MATE statement preinstruction; this increased to 72.1% of majors and 72.9% of nonmajors after instruction, representing gains of 13 percentage points in majors and 22.9 percentage points in nonmajors. If these survey questions are estimating roughly the same student attitudes, our data show lower but comparable gains in acceptance. It is, however, unclear how long-lasting these changes in acceptance were, since our study design only allowed for immediate posttesting of students (within 0–7 d).

Given the increased acceptance of evolution postinstruction in two independent student populations, our instructional intervention appears to be associated with increased student acceptance of evolution. That the gains observed in this study are smaller than those reported after course-level or multiple-period interventions is not surprising; that we found significant gains in test scores after a 2-h intervention, in implementations with two different student populations, is encouraging.

Study Limitations

Several design decisions and confounding factors limit the conclusions that can be drawn from this study. The content test would have benefited from more rigorous expert review. Four evolutionary biologists reviewed versions of the test, but were not asked to provide specific feedback on each item individually. The reliability index of the test was also lower than is generally acceptable for research purposes. Thus, the content test is not yet appropriate for general use, nor can conclusive claims be made about the impact of the lesson plan on student alternate conceptions. Additionally, our decision to switch from a paper-and-pencil to online format and the differences in allotted time to complete the pre- and postinstructional tests may have influenced our results on the change in alternate conceptions and MATE scores in phase III of the study. Nevertheless, we hope that our results will prove useful to others designing evolutionary biology curricula.

CONCLUSION

The ever-growing interest in student learning and conceptions in biology is yielding not only more information on student alternate conceptions, but also new insight into how instructional materials are impacting those conceptions. In this paper, we described several alternate conceptions about macroevolutionary patterns and began to document the prevalence of those alternate conceptions. We also presented information on undergraduate major and nonmajor biology student acceptance of evolution. Any assessment of instructional materials for diverse student groups is likely to find that some concepts are better addressed than others. In (hopefully) rare instances, such as in this study, some alternate conceptions may even be reinforced. Thus, this study can serve as a cautionary tale: even with the best instructional design intentions, you may encounter students coming away from instruction with unanticipated or unresolved alternate conceptions. Although the flaws in our content test limited our ability to measure student alternate conceptions, at best the lesson plan has inconsistent effects on student alternate conceptions and will require more assessment before it is clear to what degree it impacts those conceptions. Encouragingly, the lesson plan appears to be associated with increases in student acceptance of the accumulated evidence that life on earth has evolved. This suggests that even short interventions that explicitly teach evidence for evolutionary theory may influence student acceptance.