Exploring undergraduates' understanding of photosynthesis using diagnostic question clusters.

We present a diagnostic question cluster (DQC) that assesses undergraduates' thinking about photosynthesis. This assessment tool is not designed to identify individual misconceptions. Rather, it is focused on students' abilities to apply basic concepts about photosynthesis by reasoning with a coordinated set of practices based on a few scientific principles: conservation of matter, conservation of energy, and the hierarchical nature of biological systems. Data on students' responses to the cluster items and uses of some of the questions in multiple-choice, multiple-true/false, and essay formats are compared. A cross-over study indicates that the multiple-true/false format shows promise as a machine-gradable format that identifies students who have a mixture of accurate and inaccurate ideas. In addition, interviews with students about their choices on three multiple-choice questions reveal the fragility of students' understanding. Collectively, the data show that many undergraduates lack both a basic understanding of the role of photosynthesis in plant metabolism and the ability to reason with scientific principles when learning new content. Implications for instruction are discussed.

INTRODUCTION

The goal of the work presented here was to develop a diagnostic question cluster (DQC) that would yield information on undergraduates' thinking about photosynthesis to inform improvements in instruction and assessment. This assessment tool is diagnostic in the sense that it identifies patterns across students' responses to questions, revealing root problems that can be the focus of instructional change.

Our work indicates that in order to apply basic concepts about photosynthesis, students need to be able to engage in a coordinated set of practices based on a few scientific principles: conservation of matter, conservation of energy, and the hierarchical nature of biological systems. We work with clusters of questions, rather than individual questions, to assess students' abilities to do the coordinated practices and to see whether students' abilities to apply concepts are context specific.

BACKGROUND

Misconceptions about Photosynthesis

Misconceptions about photosynthesis are well documented (e.g., Eisen and Stavy, 1988; Amir and Tamir, 1994; Hazel and Prosser, 1994; Marmaroti and Galanopoulou, 2006; Yenilmez and Tekkaya, 2006; Köse, 2008). These are pervasive and persist throughout schooling, from primary to postsecondary education. Some of these misconceptions arise from direct experiences students have had observing plants. For example, the idea that plants obtain all of their nutrients from the soil matches everyday experience with plants, in which the only visible inputs are through the roots (Eisen and Stavy, 1988; Marmaroti and Galanopoulou, 2006; Köse, 2008). Other misconceptions are perpetuated by commonly used textbooks that contain misinformation (Storey, 1989).

A number of misconceptions involve confusion about the roles of the products, reactants, and sunlight in photosynthesis. For instance, students may believe that sunlight is a material that is somehow incorporated into the mass of the plant (as opposed to providing energy to drive the reaction; Eisen and Stavy, 1988). They may not recognize that carbon dioxide is the major contributor to plant mass (Eisen and Stavy, 1988), or they may not understand the dual role of glucose as a source of building blocks for cell growth and energy storage (Köse, 2008). There are misconceptions about the location of photosynthesis in the plant and the role of chlorophyll (Marmaroti and Galanopoulou, 2006; Köse, 2008). For example, some students believe that the pigment is a reactant or product of photosynthesis (Marmaroti and Galanopoulou, 2006). In addition, students may think that photosynthesis (or at least the “dark reactions”) continues in the absence of light (Eisen and Stavy, 1988). Confusion exists about what is meant by “primary producer.” Instead of understanding that plants are autotrophs that make their own food, many students take this to mean that plants are a source of oxygen or food for animals (e.g., by producing fruit for humans to consume; Köse, 2008).

A major source of confusion for students is the relationship between photosynthesis and cellular respiration. Many students believe that plants do not respire at all (Amir and Tamir, 1994; Köse, 2008), that photosynthesis is the means by which plants respire (Amir and Tamir, 1994; Köse, 2008), or that photosynthesis provides the plant with energy directly (Yenilmez and Tekkaya, 2006; Köse, 2008). Students appear to confuse respiration with breathing, and thus view the former solely as a gas-exchange event. Because they believe that photosynthesis is the opposite of cellular respiration, it is also viewed as gas exchange, or how plants “breathe” (Eisen and Stavy, 1988; Amir and Tamir, 1994). Respiration is often seen as the opposite of photosynthesis, because some reactants of photosynthesis, namely carbon dioxide and water, are the products of respiration, while oxygen, a reactant of respiration, is a product of photosynthesis. However, students do not seem to realize that there are differences between the processes in chemical pathways, location in the plant (Eisen and Stavy, 1988; Yenilmez and Tekkaya, 2006), and when they occur (e.g., many students believe that photosynthesis occurs in the presence of light and respiration in the dark; Marmaroti and Galanopoulou, 2006; Yenilmez and Tekkaya, 2006).

Principled Reasoning

Our work differs from other research on misconceptions in that we seek to develop an interpretative framework that looks for patterns across misconceptions. We suggest in this study that principled reasoning provides that framework and we provide a diagnostic question cluster (Supplemental Material A) that assesses students' principled reasoning about photosynthesis. We define principled reasoning as reasoning guided by basic scientific principles and habits of mind or practices that facilitate students' learning and understanding. The principles apply to multiple contexts and content areas and therefore promote learning across content areas. Once the principles are defined, they become key to organizing content into frameworks for instruction.

Three principles we have found useful are conservation of matter (Wilson ), conservation of energy, and the hierarchical nature of biological systems. All three of these principles are identified in the Vision and Change in Undergraduate Biology Education report (American Association for the Advancement of Science [AAAS], 2010) as being “core concepts” in biology education. These principles encompass such statements as:

During chemical reactions, intramolecular bonds are broken and atoms are rearranged to form molecules of new substances as new bonds form. No atoms are lost in the process.

Energy is used to break bonds, while energy is released when bonds form.

Biological systems are nested in scale, and the properties and functions of a particular scale emerge from the properties and functions of smaller scales.

Principled reasoning also involves using a coordinated set of practices related to the scientific principles. For photosynthesis, we have found three practices to be important. We present the content associated with photosynthesis organized around these three practices in Supplemental Material B.

The practice of tracing matter includes:

identifying the matter that changes, that is, the inputs and outputs of a system or the reactants and products of a reaction or set of reactions;

distinguishing matter from energy;

tracing atoms; and

conserving matter.

In photosynthesis, tracing matter includes knowing the overall reaction and tracing individual elements through the process to see, for example, that elemental oxygen produced does not come directly from carbon dioxide, as shown by the color-coding of oxygen in this reaction.

The practice of tracing energy includes:

identifying the energy that is transformed or transferred and the forms of energy involved

describing the nature of the transformations or transfers

conserving energy, and

identifying processes that transfer or transform information.

The energy transformations of photosynthesis include transforming sunlight to chemical potential energy in NADPH. That chemical potential energy is transferred from a proton gradient to chemical potential energy in ATP and finally to chemical potential energy in fixed carbon. This establishes that ATP production is not the end point of photosynthesis.

The practice of organizing systems and identifying scale includes:

Knowing the structure of the systems in which the relevant processes are taking place and how they facilitate the function.

Selecting the appropriate level/scale in which to reason. In biological systems, the explanations for, or mechanisms of, phenomena apparent at one scale often lie at a different scale. For example, a plant such as a maple tree (at the human scale) gains mass as it grows through the molecular/subcellular process of photosynthesis.

The need for the last principle is perhaps not as obvious as for the first two. However, problems of scale plague much science instruction, impacting discussions of large amounts of time (geology, biology, astronomy), large distances (astronomy), small amounts of time (physics, chemistry, biology), and small sizes (physics, chemistry, biology). In a study of high school students' understanding of the cell, cellular structures, and processes, Flores showed that many problems arise because students fail to distinguish between processes that happen at the organismal level versus the organ or cellular level. An example of this type of confusion can be seen when students confuse respiration and digestion (Songer and Mintzes, 1994). Ben-Zvi and Orion (2005) define understanding scale as one of several key components of a systems approach to learning science. They point out that this approach gives students a framework for addressing many topics. We focus on these three fundamental principles, which apply to many topics.

METHODS

Question Development

We developed the cluster of questions used in this study by asking open-ended versions of these questions to undergraduates in large introductory biology classes or smaller upper-level courses. Common inaccurate responses were noted, and these were used to develop distracters for diagnostic multiple-choice questions (Treagust, 1988; Sadler, 1998). A multi-departmental panel of biologists and science teacher educators reviewed the questions for content validity. Construct validity was checked by administering the multiple-choice items to groups of students and asking them to explain their answer choices in writing or through interviews. All revised multiple-choice items were administered postinstruction on standard course exams in an introductory biology course with enrollments of 263–449 students from 2004 to 2009 at a large midwestern university.

We asked two levels of questions. The lower-level questions asked students to identify the inputs and outputs of the light reactions and Calvin cycle or to trace elements or energy transformations through these reactions. These questions did not ask students to carry this information across scales. They asked directly about matter and energy transformations at the cellular or subcellular levels—the scales that are usually emphasized during instruction in an introductory cell biology course. In Bloom's taxonomy, these would be classified as comprehension questions (Bloom, 1956).

The higher-level questions asked students to apply (Bloom, 1956) what they know about the matter and energy transformations of photosynthesis to explain phenomena in plants. Thus, for these questions, students needed to understand how the whole organism used the cellular process of photosynthesis.

Question Format Comparison

To compare what can be learned from different question formats, we did cross-over experiments comparing multiple-choice with multiple-true/false format and multiple-choice with essay format. (In multiple-true/false format, distracters are presented as individual statements and students indicate whether each one is true or false without knowing how many are true.) Four forms of an exam were generated with different formats of four questions, as seen in Table 1. Students were randomly given one of the four test versions.

Table 1.

Questions as they appeared on various versions of the assessment

Question/test	A	B	C	D
Maple tree	MC	Essay	MC	Essay
Corn	Essay	MC	Essay	MC
Euglena	MC	MC	MT/F	MT/F
Geranium root	MT/F	MT/F	MC	MC

aMC: multiple choice; MT/F: multiple true/false; Essay: constructed response.

The maple tree and corn questions asked students about the source of mass in growing plants. In multiple-choice format, they had the same foils in the same order. The Euglena and geranium root questions asked students about sources of ATP for cells in photosynthetic organisms. The foils were not the same. (For the specific questions, see Supplemental Material A and Results.) The order of questions on all exams was: multiple-choice version of maple tree or corn question, Euglena question, geranium root question, and essay version of maple tree or corn question.

Essays were about the source of mass gain in growing plants (maple tree and corn) and were scored as correct if students mentioned photosynthesis and carbon dioxide. Inaccurate processes and inputs to photosynthesis were noted separately.

Interviews

In a different semester, we conducted interviews with student volunteers in order to gain richer insight into students' understanding of photosynthesis. A month after taking their hourly exam, students were asked about three of the exam questions. They were asked to explain mass gain in corn plants and radish seeds growing in light and the energy transformations in Euglena growing in light, in that order. Volunteers were sorted into three categories: those answering both mass-gain questions correctly on the exam, those with a mix of correct and incorrect answers, and those with no correct responses. Students were randomly chosen from each category. In total, 14 interviews were performed. During the interviews, the students were shown the stem (question without distracters) to the “radish seeds in light” question (question 7 in Supplemental Material A; Ebert-May ) and asked to explain the mass gain. They were then shown the distracters one by one and asked to explain which they would choose (or not choose) and why; the process was repeated with the corn and Euglena questions.

Students

All data were collected in a one-semester course on cell and molecular biology that is one of a two-semester series of courses in introductory biology. Table 2 shows the demographic data for the semester of students who took the exam described in Table 1. This group of students was representative of students in other semesters. This course serves a large number of majors from multiple colleges. Students came from 71 majors. The largest single major was prenursing. One semester of introductory chemistry is a course prerequisite, and sophomores therefore represent the largest population of students.

Table 2.

Demographics of students (n = 333)a

Gender	Female	61.9%
	Male	38.1%
Class standing	1	11.1%
	2	59.8%
	3	21.0%
	4	7.8%
	Post-BA, second degree	0.3%
Ethnicity	Caucasian	82.3%
	American Indian	0.3%
	Black	5.4%
	Hispanic	3.0%
	Asian	4.5%
	Other or not reported	4.5%
Major	Pre-health	40.5%
	Science	18.9%
	Engineering	8.7%
	Agriculture	7.8%
	Other	24.1%

aClass standing is based on number of course credits. Multiple majors are included in each category. The pre-health group of majors includes students identifying a major associated with health or medical professions, such as medical technology, pre-nursing, or human biology.

RESULTS

The DQC and Students' Responses

Cluster questions, along with data on students' responses, are shown in Supplemental Material A. All items were administered postinstruction on standard course exams in introductory biology courses with enrollments of 263–449 students. Thirteen questions are presented in multiple-choice format, but can be used in multiple-true/false format (see Comparison of Question Formats below). The stems from most of the questions can be used as prompts for essay questions. Altered stems are proposed for essay versions of the remaining questions.

Questions are categorized according to the practices demanded by the stem. They are presented in order of increasing complexity. The first four questions ask students to trace matter through the process of photosynthesis at the cellular level. Questions 5–9 involve tracing matter across scales, since the questions are posed about whole organisms, but the explanations lie at the cellular level. Of these questions, 5–7 address mass gain in plants, while questions 8 and 9 address mass loss in plants. The latter require that students understand that plants undergo both respiration and photosynthesis. Questions 10–12 ask students about energy sources for plants. In particular, questions 10 and 11 ask about energy sources for cells. However, question 12 asks about cells in different parts of a multicellular plant. These questions require students to trace energy through both photosynthesis and respiration. While the stems of questions can be fairly cleanly categorized in this way, the distracters often encompass inappropriate mixed practices, such as matter–energy conversions or scale mistakes. Thus, as a diagnostic, an item whose stem calls for tracing matter may diagnose incorrect tracing of energy, and so on.

Questions were administered postinstruction on an exam in an introductory biology course at least once between 2004 and 2009. Summary results are shown in Table 3. Results for individual questions are shown in Supplemental Material A.

Table 3.

Scores on cluster questions categorized by practice(s) required (n = 263–449a)

Practices demanded by question	Question numbers	Percent of students answering correctlyb
Tracing matter	1–4	34.2–75.9
Tracing matter and keeping track of scale
Mass gain in plants	5–7	48.6–80.1
Mass loss in plants	8–9	31.3–56.3
Tracing energy	10–11	15.4–46.5
Tracing energy and keeping track of scale	12	31.2

an varies depending upon the semester in which the questions were asked.

bThe percent correct varies by question; the ranges are for the questions in each subset.

Comparison of Question Formats

Multiple True/False.

Multiple-choice questions are a manageable assessment format for large-enrollment courses, which often require machine scoring. However, students approach them using various test-taking skills to identify the correct answer. These skills, such as length of a distracter, word recognition, and how scientific an answer sounds, can result in inaccurate estimates of students' understanding. In particular, we were concerned with students who might remember a correct response but still retain inaccurate ideas. Nehm and Schonfeld (2008) describe this as having mixed or heterogeneous understanding. To test the prevalence of this situation, we used a cross-over experimental design. As part of a standard exam (2009, n = 380), half of the students in an introductory biology course (see Methods) were given a question (the geranium root question) as a multiple-choice question, while the other half of the students were given the same question in a multiple-true/false format in which the distracters were presented as individual statements, and students had to indicate whether each one was true or false without knowing how many were true. Students were given a second question (the Euglena question) in the other format. The results are shown in Figures 1 and 2.

Figure 1.

Percent of students (n = 380) choosing specific multiple-choice distracters (blue) vs. percent of students indicating statement is true (red) in multiple-true/false version of same question. B is the correct answer.

Figure 2.

Percent of students (n = 380) choosing specific multiple-choice distracters (blue) vs. percent of students indicating statement is true (red) in multiple-true/false version of same question. C is the correct answer.

For the geranium root question, students selected the correct answer (B) most frequently, regardless of the format in which the question was delivered. However, in the multiple-true/false format, more than half of the students indicated that the incorrect choice (A) was also true, and at least one-fourth of the students indicated that each of the choices was true, implying that they simultaneously held accurate and inaccurate ideas.

With the Euglena question, students' mixed ideas about the source of ATP for cellular work are even more apparent. This was a difficult question for students. Regardless of question format, the most popular choice was incorrect—that Euglena use ATP made during photosynthesis to do cellular work. However, in the multiple-true/false format, more than half of the students indicated that four of the five choices were true.

Essay.

A crossover experiment was used to compare essay and multiple-choice formats in an introductory biology course (2009, n = 380). On an exam, half of the class was given one of two questions about the source of mass in plants (maple tree or corn, see questions 5 and 6 in Supplemental Material A) as an essay question, with the second question being a multiple-choice question. In the multiple-choice version, both questions had the same choices in the same order. Essay responses were scored as correct if students mentioned carbon dioxide as a source of mass for plant growth and named photosynthesis (or the Calvin cycle) as the process, regardless of the presence of incorrect sources of mass or process labels. In a separate round of scoring, students who included both correct and incorrect sources of mass were noted. The results are shown in Table 4.

Table 4.

Percent of students correctly answering the multiple-choice and essay versions of the corn and maple tree questions and percent of students including both correct and incorrect mass sources in their essay responsesa

	Students answering correctly		Students with both correct and incorrect sources of mass
Question	Multiple choice	Essay	Essay
Corn	61.4 ± 8.8%	51.0 ± 7.1%	17 ± 5.4%
	n = 186	n = 190	n = 190
Maple tree	57.4 ± 9.3%	47.2 ± 7.2%	17 ± 5.5%
	n = 190	n = 186	n = 186

aAll percentages are shown with 95% confidence intervals.

Fewer students answered the essay version of this question correctly, compared with the multiple-choice version, indicating that they could correctly identify from a list the main contributors to plant mass gain, but could not articulate these ideas on their own. It should be noted that the multiple-choice version of the question appeared first on the exam forms, with the essay version occurring six questions later. However, very few students used the wording of the multiple-choice distracters in their essay, suggesting little copying or transfer.

Like the multiple-true/false questions (see preceding section), the essay responses revealed that students' understanding included both accurate and inaccurate ideas. Seventeen percent of the students who identified accurate sources for plant mass gain (carbon dioxide and/or water) also mentioned inaccurate sources, such as fertilizer, minerals, or sunlight. (Students who mentioned fertilizer or minerals, but indicated that these were minor contributors, were not counted as having heterogeneous responses.)

Interviews.

To determine how students interpreted the multiple-choice questions and to see whether their exam responses were indicative of their thinking, 14 student volunteers were interviewed about their midterm exam responses to two questions about the sources of mass when plants (radish seeds and corn) grow and the source of ATP for cellular work in Euglena. They were asked to respond in turn to the stem and then the distracters to the radish seed question, the corn question, and the Euglena question.

Mass Gain in Plants—Correct Responders. Four of the 14 students responded correctly on the exam to both the radish seed and corn questions about the sources of mass gain during plant growth. Excerpts from their interview transcripts are shown in Table 5 (all names are pseudonyms).

Table 5.

Excerpts from the interviews of the students who answered both questions about the sources of plant mass gain during growth correctly on the exam

Burt in response to the radish seed stem: And then how this increase in mass biomass [radish seeds in light] occurred would obviously be not from water so it had to be from something else like some sort of glucose or something like that… It [glucose] is made of C6H12O6 and so it needs the CO2 to make for the carbon and it has water that uses H for the water from the water too.

Devin in response to the radish seed stem: All right. With these, with the light and water, it's so much because during photosynthesis, carbon dioxide is taken in from the outside and then, through a number of processes, it's released. But the carbon atoms add to the mass of the plants…

Devin in response to the radish seed distracters: I know that plants need soil to grow in and that some of the minerals in soil help it to just grow and prosper and do its day-to-day activities, but it's not a significant contributor to the mass of the plant. I wouldn't say that this [solar radiation] is a direct contribution, but the plants need solar energy to do photosynthesis, so the solar radiation doesn't cause the increase in mass. That's not the materials, but it's like the light coming down is probably the cause of the… it makes the ATP that is needed for photosynthesis.

Wendy in response to the radish seed stem: Okay, we spent like 20 minutes on this in class and so I can answer this without even thinking. The increase in biomass occurred from the intake of carbon dioxide and the expulsion of oxygen.

Wendy in response to the stem of corn question: Oh that is easy, it is just because on farms, obviously they use fertilizers which are high in nitrates which can help to increase energy levels and bio mass intake because then they also have the intake from the roots, there is more minerals in the soil, increasing the bio mass, obviously there is carbon dioxide all around.

Wendy in response to the distracters for the corn question: It is still the CO2 because that is the main one, the others are just added bonuses to help with different processes because I want to say that there is a cycle.

Olivia in response to the stem of the radish seed question: I just don't really remember, so I'm like, I know part of it and I understood … I got it when he was teaching it, but I haven't reviewed it, so I'm just not quite sure. I just know that there's something with carbons and that's why it has more mass… She chose carbon dioxide after seeing the distracters.

Burt is the only student who frames his response in terms of tracing matter. Devin and Wendy give evidence of understanding the relative contributions of different sources (with some inaccuracies or sloppy language). However, Wendy's opening sentence indicates that this might be a memorized response. Olivia's response is confused. Although she was able to identify carbon dioxide as the correct response to both questions, she could not describe photosynthesis.

Mass Gain in Plants—Incorrect Responders. Six interviewees (Wren, Ruth, Sherman, Phillipa, Winnona, and Ivy) answered both questions about the sources of mass gain in plants incorrectly on the exam. Summaries and excerpts from their interviews are shown in Table 6. Four of the six demonstrated a confused understanding of photosynthesis during the interview. Winonna gave accurate responses in the interview, but explained that on the exam she reasoned from common sense and had forgotten about photosynthesis. Ivy was the only volunteer to choose solar radiation on the exam. In the interview, she gave accurate responses. The interviewer did not ask her about the discrepancy in her answers.

Table 6.

Excerpts from the interviews of students who answered both questions about the sources of mass gain in plants during growth incorrectly on the exam.

In the interview, Wren floundered for an answer in response to the radish seed question stem. She mentioned pyruvate oxidation, fermentation, Kreb cycle, Calvin cycle, and carbon dioxide release before mentioning that the mass gain came from glucose produced during photosynthesis. She explained her troubles by saying, Biology's just really hard for me, and I probably didn't study enough as I should have, but I just have a hard time memorizing like—trying to figure out what exactly I have to memorize for—like the carbon dioxide thing. After seeing the distracters, she wavered between carbon dioxide and radiation, but chose radiation. When discussing the corn question, the interviewer asked her to rank the contributions of the many sources she named. She responded, Well, the last question was CO2, so we'll go with CO2 again on this one… And then I like the water followed by light, but that's very, very close… Together, because you have to have them both, but then I'd probably go with organic compounds and then minerals. When questioned about each of these choices, she wondered whether there was a formula for light, but then said that light is necessary but does not contribute to the biomass. The interviewer asked whether CO2 and water were, like minerals, helpers or the actual material that becomes part of the plant. She responded, They're all kind of the helpers because carbon dioxide gets taken in and then it gets thrown off, and so does water.

Ruth responded to the stem of the radish seed question saying, Okay, um all right, well I know that the light makes it grow which gives it like, nutrition, which it gets from the dirt and the water. And it takes in the nutrition into the, photosynthesis also adds to it [sounds unsure]. And it gives, I know it gives off CO2 and that releases off, but it doesn't really add to the weight… it gained in mass because it grew from the water and the soil and nutrition that it took up from the dirt through the water. She stuck to variations on this answer throughout the interview.

Sherman did not come up with a response until after seeing the distracters for the radish seed question. After seeing the correct response, he said, I forgot about the “CO2” for photosynthesis. That was one of the big things I think that they were talking about. That's one of the biggest contributors to making more …Yeah…so it pops the carbon off and then releases “O2.” That's how plants release “O2” in the atmosphere, it's through photosynthesis.” In response to the stem of the corn question, he said, “I'd probably say the majority [of mass], like 60 percent comes from “CO2” and maybe 20 percent of water and 20 percent from the light. That's my guess. He did eventually choose carbon dioxide as the answer.

Phillipa in response to the radish seed stem: I don't remember which one I chose. But the absorption of mineral substances coming through the roots… it makes sense, but… that it would absorb… it'd be heavier because of that, but I think it's more of the oxygen. After seeing the distracters, she changed oxygen to carbon dioxide, but eventually chose minerals.

Winnona chose organic substances on the exam, but carbon dioxide in the interview. She explained her change in responses, I know, like if I were to retake this test, I would have definitely put that [inaudible]. But um… well um… I don't know. I just… I thought it through logically and I thought well it's absorbing stuff, organic substances. I thought, oh man, that has mass, you know, it's absorbing stuff that has mass. But I imagine that would have an effect, but it's not nearly as significant. I didn't think of photosynthesis… I didn't think it through, I don't think.

Ivy in response to the radish seed stem: The source of the carbon is from carbon dioxide in the air—And that's how it—the plant fixes the carbon dioxide and releases the oxygen and uses that carbon to make sugars, or cellulose, or whatever they grow from.

Ivy in response to the distracters of the corn question: I said, “No [to radiation],” because that's not where the mass technically is coming from. That's where the energy is coming from to create the mass.

Ivy in response to the distracters of the corn question: Okay. Well, I would say A [minerals], still kind of a guess because I know it's necessary, but I wouldn't say that the most biomass comes from the absorption of the mineral substances. For B [organic substances], I would say, “No,” because I don't think—well I know, pretty much, I hope—That they don't take in carbon through their roots. CO2 gas, I would say that's definitely still my—one of my top priority candidates.

Mass Gain in Plants—Mixed Responders. Three students correctly answered the radish seed question, but not the corn question, on the exam. One student correctly answered the corn question, but not the radish seed question, on the exam. They chose the same responses during the interview.

Energy Sources for Plants. In the same interviews, students were asked about a third cluster question—the Euglena question. This question asks students about the source of ATP for cellular work in photosynthetic Euglena. Five of the 14 interviewees answered the question correctly on the exam and in the interview. Six said that Euglena use ATP made from sunlight both on the exam and in the interview. We thought that they might be interpreting the question to mean that sunlight is the ultimate source of energy for ATP synthesis, a correct statement. However, the interviews showed this not to be the case. Students indicated that ATP was a product of photosynthesis. Some mentioned the electron transport chain producing ATP in response to sunlight.

Three students chose a response in the interview that differed from their exam choice. On the exam, Bob chose the response that Euglena use sunlight to make ATP. Before seeing the distracters in the interview, he could only articulate that Euglena use photosynthesis. After looking at the distracters, he wavered between that response and the correct answer. At first he was not sure that sugars were made in the chloroplast, and then he questioned whether they could leave the chloroplast. He did eventually choose the correct response. Pomina's responses followed a similar pattern. In the interview, she indicated that ATP made from sunlight and sugars made in the chloroplasts were the best answers. She was unsure whether “photons” or sugars were the most important products of photosynthesis. She eventually chose sugars. Selena chose the correct answer on the exam. In the interview, she rejected this answer, because she remembered that organisms could use other types of molecules, such as lipids, to make ATP. Instead, she chose ATP made in the chloroplast, describing the electron transport chain.

Other Insights into Students' Thinking from Interviews. Interviews are useful for exploring students' understanding of words, in particular words that have precise meanings in biology, but broader meanings in common usage. Three students associated “solar radiation” with damage, ultraviolet light, or other nondescript negative connotations. An additional student was unsure of the meaning of radiation. Five students had confused or vague ideas of what mineral and organic substances are. Two students associated organic substances with healthy foods or “natural stuff.”

Finally, interviews yield clues about students' mental images of biological processes. In these interviews, students wondered about what could and could not leave chloroplasts and thylakoids, and what could enter roots. Some students were unsure of what was transported in plants (vs. what is made and used within a cell). This points to the difficulty students have with location and scale.

Students' Interpretation of Context.

The context of the question may affect students' responses. For example, Nehm and Ha (2011) found that students' explanations of evolutionary trait gains are less sensitive to context, while their explanations for trait loss are more sensitive to context. We explored the effects of context in the cross-over experiment by comparing students' responses to the corn and maple tree questions about the source of mass gain in growing plants. As seen in Table 4, in both the multiple-choice and essay formats, slightly more students answered the corn question correctly; however, the differences were not statistically significant. There was no difference between the two contexts in the number of students who included both correct and incorrect products on the essay.

We also looked for evidence of context effects in the interviews by comparing interviewees' responses about the sources of mass gain in radish seeds in a Petri dish and corn. In the radish seed question, eight students eliminated mineral and/or organic substances, because the seeds are in Petri dishes and given only water. Seven students mentioned that corn needs or gets more nutrients (often mentioning fertilizer) than the radish seeds or remarked on the added presence of soil. However, these considerations did not appear to affect their ultimate answer choice. One student stated that corn is not a green plant. He said, “See corn is different because it is obviously a yellowish brown color, so it is not…it still goes under [sic] all the processes, but I don't think the CO2 gas would have [sic] the most contributing factor.” A second student questioned whether corn was a green plant, but decided it was.

The interviewees were drawn from one of two cohorts of students who responded to both the radish seeds in the light and corn questions on an exam. Given that interviewees used cues from the question to eliminate organic substances and minerals as possible responses for the radish seed question, we looked to see whether, on the exams, more students correctly answered the radish seed question compared with the corn question. Table 7 shows the cross-tabulation of these results with 95% confidence intervals. There was no statistically significant difference between the fraction of students answering each question correctly. Nor was there a statistically significant difference between the fraction of students who answered the radish seed question correctly but not the corn question compared with the reverse. We conclude that students' understanding of the source of mass for growing plants is not significantly dependent on context. This study did not address context effects on students' understanding of other aspects of photosynthesis.

Table 7.

Two cohorts of student responses to the radish seeds in light and corn questionsa

	Corn
	Incorrect	Correct
Incorrect: radish seeds in light	30.1%	9.0%
Correct: radish seeds in light	11.1%	49.7%

aPercentages are shown with 95% confidence intervals. n = 720.

CONCLUSIONS

The Cluster

Most students chose the same response on the exam and in the interview despite an intervening month, indicating test–retest reliability. Those students whose responses differed had explanations for their changes. The exception was Ivy, who switched from incorrect answers on the exam to correct responses in the interview. She was not questioned about this change in the interview.

If we take the interviews to be the best measure of students' understanding, the interview results demonstrate the construct validity of the diagnostic question cluster. An incorrect choice on one of the multiple-choice cluster questions is an indicator that the student has difficulty performing the practice(s) required by the question. However, the converse is not true. The multiple-true/false questions, essays, and interviews all indicate that a student who makes a correct choice on a multiple-choice question may have: reasoned as expected, remembered a response without understanding, held a mixture of accurate and inaccurate ideas, or simply guessed. This is in line with what others have found. For example, Nehm and Schonfeld's (2008) study of students' understanding of natural selection found that an open-ended response instrument revealed that students who could identify correct responses in a multiple-choice format also held alternative conceptions.

Alternative Formats

While the essay questions and interviews revealed students with both accurate and inaccurate ideas, these formats are time-consuming to administer and grade. The results of the cross-over experiment comparing multiple-choice and multiple-true/false formats indicate that the latter can also identify students with heterogeneous understanding. This machine-gradable format holds promise as a diagnostic tool with two caveats. Because students need to respond thoughtfully to each statement, they will require somewhat more time to answer questions in multiple-true/false format. Also, if multiple-choice questions are adapted to multiple-true/false format without change, students are likely to realize that there is only one correct response among the choices. Therefore, we recommend including additional correct responses from time to time and providing students with practice using this format.

Essays and Interviews.

In addition to uncovering heterogeneous thinking, both essays and interviews uncovered the fragility of students' understanding and identified some of the sources of confusion. For example, the essays showed a number of students (17%) who were unsure when to invoke photosynthesis versus a variant of photosynthesis, such as photorespiration or CAM photosynthesis. The interviews revealed finer-grained problems, such as difficulty with basic vocabulary (e.g., “mineral and organic substances”). Perhaps more importantly, only the interviews catch students questioning their ideas, such as their concept of what actually leaves the chloroplast. This indicates that both open-ended formats have a role in understanding students' reasoning. At the same time, it is apparent that we should be exploring other open-ended formats that are easily gradable, such as box-and-arrow diagrams (e.g., Sibley ) or machine-learning approaches to essay analysis (Haudek ; Nehm ). In addition to being more effective than multiple choice as both formative and summative assessment tools, these could provide opportunities in which students could practice articulating their own ideas and/or receive feedback.

The data from the use of the diagnostic question cluster in all formats indicate that introductory biology students have difficulty reasoning in principled ways about photosynthesis. The first four questions asked students to trace matter through photosynthesis (see Supplemental Material A). More specifically, these questions asked students to trace elements (oxygen) through the equation for photosynthesis and to associate process names with inputs and outputs. These questions are lower-level (Bloom, 1956) questions that assess students' basic knowledge. Students often learn the equation for photosynthesis in high school. Tracing oxygen through the equation indicates that this is not a simple substitution reaction in which elemental oxygen is stripped from carbon dioxide, released, and replaced with hydrogen and hydroxyl groups. Identifying the inputs and outputs of the sets of reactions, such as the Calvin cycle, is a first step in understanding photosynthesis at a subcellular level. Keeping track of inputs and outputs to the different sets of reactions is also a useful way to organize the encyclopedic information presented in standard textbooks.

For the first two questions for which we have data from multiple years, we see variation in students' performance on these questions. We surmise that this is due to variation in what the instructor emphasized. The answers to these questions could be found explicitly in textbooks and could be memorized. Questions 3 and 4 require students to synthesize several pieces of basic knowledge. Fewer than half of the students answered these questions correctly, indicating they do not have this basic knowledge at their disposal.

Whereas questions 1–4 assessed students' understanding of basic knowledge of the flow of matter at the subcellular level, questions 5–9 on mass gain and loss in plants assessed students' understanding of photosynthesis (and respiration) at the organismal level. Again, students' performances varied from year to year, but at no time did students appear to have a sound understanding of the big picture. We often assume that students come with this high school–level understanding of the relationships among photosynthesis, respiration, and growth and only devote instructional time to elaborating on it without first clarifying it. For example, in Biology (Campbell and Reece, 2005), the chapter on photosynthesis is 19 pages long. The single page that explains the relationship between photosynthesis and respiration is in the preceding chapter, which is devoted to cellular respiration. The section on plant growth comes 500 pages later and refers to cell division, but to no other cellular functions. Thus, it appears that we are layering details on a weak foundation.

Student performance on the three questions about energy sources for plants is also alarmingly weak. Fewer than one-half of the students answer these questions correctly. This indicates that students' big-picture understanding of the role of photosynthesis in providing reduced-carbon compounds that can be used as energy sources is as weak as their comprehension of the use of reduced-carbon compounds in plant growth. Again, we are giving students molecular details about a process whose basic functions are unclear to them.

The interviews indicate that many students are trying to memorize rather than understand the information (see responses of Wendy, Olivia, Wren, and Sherman in Tables 5 and 6). Students also draw on personal experience and knowledge. For example, many students mentioned that farmers fertilize their corn crops and then tried to fit this piece of information into their answers. Only Burt appeared to reason systematically and with principles about his answers by indicating that if C6H12O6 is the product of photosynthesis, he must account for a source for each of the three elements.

Implications for Instruction

Taken collectively, these results indicate that many students lack 1) a big-picture understanding of the role of photosynthesis in plant growth and energy use and 2) the knowledge and/or inclination to use basic principles to organize the large amounts of information we want them to learn about these topics. This means that we are layering molecular details onto a weak or faulty foundation without giving students tools for shoring up their understanding. We are guilty of ignoring what is known about how people learn. Redish (1994) summarizes this in a set of principles. “Principle 1: People tend to organize their experiences and observations into patterns or mental models” (Redish, 1994, p. 798). “Principle 2: It is reasonably easy to learn something that matches or extends an existing mental model” (Redish, 1994, p. 801). “Corollary 2.1: It's hard to learn something we don't almost already know” (Redish, 1994, p. 801). One implication of this is that “new information should always be presented in a context that is familiar to the reader and that the context should be established first” (Redish, 1994, p. 801). More specifically, the implication for teaching photosynthesis is that instruction should start at the organismal scale, clarifying the role of photosynthesis in plant growth and energy use, and only then move to the cellular and subcellular levels. Put bluntly, this means that in a standard semester-long introductory cell and molecular biology course, there may not be time to look at all of the individual reactions of photosynthesis. Even if students could memorize all of the reactions, this knowledge would only paper over, rather than correct, their lack of understanding of the role of photosynthesis.

The data presented here indicate that many students cannot learn all of the individual reactions of photosynthesis. They are struggling with just the inputs and outputs of the light-dependent reactions and the Calvin cycle (see results for questions 1–4 in Supplemental Material A). Since memorization appears to be a common student approach to learning, this probably means that the volume of information is too big to manage. People can remember only a limited number of unrelated pieces of information (Miller, 1956; Redish, 1994; National Research Council, 2000). This implies that students do not have the ability to organize and collate the information by connecting the details to the underlying principles. These results support the argument that the standard photosynthesis content should be organized around a few themes such as matter, energy, and scale. In the language of Redish (1994, p. 802), these themes are equivalent to the “touchstone problems” of physics, problems “students will come back to over and over again in later training. Touchstone problems become the analogs on which they will build the more sophisticated elements of their mental models.”

Common Misconceptions and Principled Reasoning

Common misconceptions about photosynthesis represented by distracters in cluster questions can be categorized according to problems with particular practices associated with principled reasoning.

Gases, such as the CO2 used in photosynthesis, have little or no mass, are unimportant, or cannot account for the mass gain of photosynthetic organisms.

ATP for cellular use is a product of photosynthesis.

Atoms from CO2 end up in ATP.

Minerals taken up by the roots make a significant contribution to the mass of the plant.

ATP (from any source) is moved throughout a plant.

ATP made during photosynthesis circulates throughout the plant.

Sunlight is converted into sugar.

To produce ATP, plants use respiration when in the dark and photosynthesis when in the light.

ATP (from any source) is moved throughout a plant.

ATP (from any source) is moved throughout a plant.

That plants grow is a sufficient explanation for mass gain, without referencing the source of the matter, the source of the energy, or the processes of photosynthesis.

Note that some misconceptions are associated with problems with more than one practice. For example, “ATP is moved throughout the plant” is associated with all three practices. To understand that ATP is made and used locally, students need to understand the inputs and outputs of photosynthesis—phosphate and ADP are not inputs and ATP is not an output of the overall process. An energy lens also provides insights into why ATP is not transported through the plant. Relative to ADP and phosphate, ATP has a lot of chemical potential energy. Thus, hydrolysis of ATP releases that energy. In other words, ATP is unstable, too unstable to last while it is transported. Using a location lens, the ATP made in the chloroplast stays in the chloroplast and is not available for cellular work.

The implications of viewing students' conceptual barriers in this way are that, if we can provide students with opportunities to practice using a handful of practices and to become better principled reasoners, they will come to understand photosynthesis better. This will require not only making the principles and practices explicit during instruction, but also orienting all instruction around these ideas. The interviews show that many students approach learning as memorization. Learning to reason will require a whole new approach on their part. The instructor's role in this significant paradigm shift is to give students sufficient time and opportunities to practice reasoning. As a starting point for instruction focused on principled reasoning, we present the content associated with photosynthesis organized around the three practices (Supplemental Material B).

The Vision and Change report (AAAS, 2010) paints in broad strokes the consensus of the biology community that teaching of biology content should be organized around five powerful ideas. The work presented here examines undergraduates' understanding of two of those ideas (pathways and transformation of energy and matter; systems) in the context of photosynthesis. Data from the use of the diagnostic cluster questions in multiple formats indicate that very few students reason about photosynthesis in this principled way. The work presented here also illustrates why traditional course structures that focus on content coverage have not shown any big gains in student achievement. Without an understanding of, and ability to apply, these principles, students have no foundation on which to build more elaborate and detailed understanding.