The C.R.E.A.T.E. approach to primary literature shifts undergraduates' self-assessed ability to read and analyze journal articles, attitudes about science, and epistemological beliefs.

The C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) method uses intensive analysis of primary literature in the undergraduate classroom to demystify and humanize science. We have reported previously that the method improves students' critical thinking and content integration abilities, while at the same time enhancing their self-reported understanding of "who does science, and why." We report here the results of an assessment that addressed C.R.E.A.T.E. students' attitudes about the nature of science, beliefs about learning, and confidence in their ability to read, analyze, and explain research articles. Using a Likert-style survey administered pre- and postcourse, we found significant changes in students' confidence in their ability to read and analyze primary literature, self-assessed understanding of the nature of science, and epistemological beliefs (e.g., their sense of whether knowledge is certain and scientific talent innate). Thus, within a single semester, the inexpensive C.R.E.A.T.E. method can shift not just students' analytical abilities and understanding of scientists as people, but can also positively affect students' confidence with analysis of primary literature, their insight into the processes of science, and their beliefs about learning.

INTRODUCTION

As scientific information continues to accumulate at a rapid pace, there is a growing sense among science educators that long-established practices need to be reconsidered. Numerous 21st-century science reform documents (American Association for Higher Education, 2000; U.S. Department of Education, 2000; National Research Council [NRC], 2003; Malcom ; Alberts, 2005; NRC 2007, 2009; American Association for the Advancement of Science [AAAS], 2010) suggest focusing less on content coverage and more on approaches that reveal science to be an ongoing creative process. Ideally, such a change would help to stem the long-standing attrition of bright students from science majors and, by extension, science research careers (Seymour and Hewitt, 1997; Cech and Kennedy, 2005; DePass and Chubin, 2009). At the same time, student attitudes toward learning (student epistemologies; Schommer 1990, 1993), student self-efficacy (confidence in ability to work effectively in a particular context; Lawson ), and student attitudes about science (Osborne, 2003) have been demonstrated to be important factors affecting students’ success in the science classroom. Thus, both changes in how science is taught and consideration of factors influencing students’ ability to learn deserve focus in science education reform efforts.

A variety of new approaches that employ alternatives to lectures, including hands-on classroom activities and small-group work (Klionsky, 1998; Handelsman ; Allen and Tanner, 2005; Knight and Wood, 2005), highlighting controversy to stimulate student engagement (Seethaler, 2005; Campion ), student participation in ongoing grant-funded research projects (Hanauer ; Call ; Lopatto ; Clark ), case study approaches (Herreid 1994a; Chaplin, 2009), use of the popular press (Strauss, 2005; Hoskins, 2010), and analysis of primary literature (Herreid, 1994b; Janick-Buckner, 1997; Lynd-Balta, 2006; Kozeracki ; Hoskins ; Hoskins, 2008; Schinske ; Yarden, 2009), shift classroom focus from a teacher-centered situation in which students are largely passive, to a student-centered classroom (Freeman ; Klymkowsky, 2007; Armbruster ; Hoskins and Stevens, 2009) more supportive of cognitive activities associated with learning (Bloom and Krathwohl, 1956; Chickering and Gamson, 1987; Zull, 2002).

We have focused on primary literature as a portal into the scientific research process through the C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) method. C.R.E.A.T.E. uses intensive critical analysis of a series of papers generated sequentially from one lab, coupled with email interviews of paper authors via a survey of student-generated questions, to demystify and humanize science. The approach is an iterative method, whereby individual steps of the process (Figure 1 and Table 1) provide students an organized approach to individual journal articles that are “dissected” using a series of novel or adapted pedagogical tools in preparation for intensive class discussion. Papers from a single lab are read in series (Figure 1), allowing students to follow the arc of a research project as it actually progressed. Students are not provided with the full series of papers in advance, nor with the titles, authors, abstracts, or discussions of the papers under consideration. While students could Google the missing information, doing so is ultimately more a hindrance, blunting creative thought, than a help. We encourage students to instead treat the course as a process of discovery. By working with a suite of novel or adapted pedagogical tools to prepare for class, students are empowered to participate actively in the lab-meeting atmosphere of the class sessions, where figures and tables are examined individually, and the logic of the overall study is examined. Previous work has documented precourse versus postcourse shifts in C.R.E.A.T.E. students’ critical thinking and content integration abilities, as well as changes in self-assessed attitudes about science and scientists, as determined by postcourse interviews and the Student Assessed Learning Gains instrument (Hoskins ). We report here the results of a survey designed to examine additional aspects of students’ attitudes, beliefs, and self-assessed abilities, comparing responses pre- and post-C.R.E.A.T.E. course.

Figure 1.

Overview of the C.R.E.A.T.E process. Papers 1–4 form a “module”—a series published by the same lab group as they followed a particular question in sequential studies. See Table 1 for details of each step of the C.R.E.A.T.E analysis. Variations on this approach, for example, discussing papers from different lab groups with conflicting data, or using the method in shorter-term analysis of newspaper/Internet reports of science studies, are also effective (see Hoskins, 2008, 2010).

*Defining/discussing grant panel criteria is done during this iteration only.

Table 1.

Overview of C.R.E.A.T.E. steps and associated activities, many of which are carried out by students in preparation for classa

C.R.E.A.T.E. step	Student activities
Consider	Concept map paper introduction, note topics for review, define new issue(s) to be addressed, begin defining relevant variables and determining their relationships.
Read	Define unfamiliar words, annotate figures, create visual depictions (sketch “cartoons”) of the individual substudies that underlie each figure or table. Transform data presented in tables into a different format (graph or chart).
Elucidate hypotheses	For each figure, define the hypothesis being tested or question being addressed by the work that generated the data illustrated. Rewrite the title of each figure in your own words.
Analyze and interpret the data	Using the hypotheses, questions, cartoons, diagrams, and charts and/or graphs, determine what the data mean. Fill in a data analysis template for each figure to track the logic of each experiment and prepare for class discussion. After all figures and tables have been analyzed, create a concept map for the paper, using each illustration as a map node to reveal the logic of the study design.
Think of the next Experiment	Consider: “If I had carried out the studies described in this paper, how would I follow up?” Design two distinct studies, and cartoon one on a transparency for in-class discussion (see Student grant panels, below).
Additional C.R.E.A.T.E. classroom activities
Student grant panels	Students work in small groups first to define criteria panels “should” use in allocating funding. After these are discussed by the whole class, students view all of the student-designed experiments, then return to small groups to evaluate the proposed studies, with the goal of reaching consensus on the one that most merits funding.
Email surveys of authors of papers	Throughout the semester, students are encouraged to jot down questions that arise regarding “the research life” or the researchers themselves. Late in the semester, 10–12 of the questions are compiled into a single survey and emailed to each paper author. Responses from authors (60–75% response rate) reveal novel behind-the-scenes insights.

aModified from Hoskins (2010, Table 1); see Hoskins for additional details on each step and the overall process.

Students’ beliefs about learning and knowledge affect their ability to learn and their application of metacognitive strategies, including their integration of prior knowledge with the task at hand, studying for understanding rather than superficial recall, and assessing what they do and do not comprehend. Such approaches can significantly facilitate students’ understanding of science (Hartman, 2002; Schraw ; Pulmones, 2010). Students’ attitudes about the nature of knowledge, for example, whether knowledge can change over time, and their attitudes about intelligence, for example, whether it is innate and fixed or malleable, comprise a set of epistemological beliefs that affect learning and understanding (Schommer, 1990). Students’ epistemological understandings are typically less sophisticated than those of their professors (Hogan and Maglienti, 2001), and these views can affect study approaches, as well as the extent to which students persist in challenging tasks (Schommer, 1993, 1994; Hofer, 2004). Reasoning ability has also been linked to epistemological beliefs, with students whose epistemologies are more sophisticated showing enhanced skills (Zeineddin and Abd-El-Khalick, 2010).

As the constructivist C.R.E.A.T.E. method uses a number of activities and pedagogical tools (Table 1) designed to increase both student engagement and metacognition, we hypothesized that students’ attitudes toward primary literature, the practice of science, and the nature of learning might change during the semester. We developed a questionnaire aimed at assessing students’ self-assessed views about science, scientists, the research process, and aspects of learning, and administered it on the first and last days of the 14-wk semester. Analysis of student responses indicates that C.R.E.A.T.E. students shifted significantly in their understanding of numerous aspects of scientific research, their approach to reading scientific literature, their confidence in their ability to understand science, and, perhaps most interesting, their epistemological beliefs. Because naïve epistemological beliefs can affect students’ study approaches, learning gains, and ability to interpret complex scientific information (Schommer, 1990; Kardash and Scholes, 1996; Pulmones, 2010), shifting student epistemology can be a first step toward developing attitudes toward knowledge and learning more supportive of student success. In this regard, it is notable the changes reported here were achieved during a single semester in a course that essentially costs nothing to implement and does not involve a hands-on laboratory component.

METHODS

Participants in the study were students in an upper-level elective at the City College of New York (CCNY). The class met twice weekly for a total of 140 min (2005; three credit hours) or 200 min (2006–2009; four credit hours) per week. Class size averaged 27 students (range: 19–32). Seven iterations of the course are included in this study. Most students were junior or senior biology majors who had completed the course prerequisites at CCNY: a year of introductory biology, and one semester each of genetics and cell/molecular biology. A few students (<10% of each class) were participants in the CCNY post-baccalaureate program. These students had earned degrees in other fields and returned to college to complete premed requirements. In the seven classes represented, 65% of students were female and 61% were African American, Hispanic, or Native American, all groups currently underrepresented at all levels of academic science (National Science Foundation [NSF], 2002, 2008; Atwell, 2004).

Presurvey/Postsurvey of Student Attitudes and Self-Rated Abilities

On the first and last days of the semester, students filled out an anonymous Likert-style survey aimed at elucidating their degree of agreement/disagreement with a series of statements. They also answered several open-ended questions on the survey. The survey statements focused on attitudes and beliefs about issues the C.R.E.A.T.E. approach was designed to address, including students’ self-rated ability to understand and analyze primary literature; whether primary literature had influenced their understanding of science; students’ understanding of the scientific research process; and students’ self-rated science reading ability, confidence in their ability to “think like a scientist,” understanding of “scientists as people,” and sense of whether research science was an appealing career choice. We designed the survey based on our experiences teaching from primary literature in previous classes, focusing on a variety of issues we had determined to be problematic for previous students. Open-ended questions requiring written answers focused on students’ understanding of the activities undertaken by research scientists were set aside for later analysis. This survey was administered in each C.R.E.A.T.E. class (two sections per year in 2005 and 2008 and one section per year in 2006, 2007, and 2009). The 2005 cohort of students was included in a previous analysis of the effects of the C.R.E.A.T.E. class on student critical thinking, content integration, and attitudes toward science/scientists (Hoskins ).

Surveys were anonymous and coded with numbers known only to the students themselves, to allow alignment of data for within-subject statistical analysis. The survey included 52 statements to which students reacted by marking “I strongly agree,” “I agree,” “I’m not sure,” “I disagree,” or “I strongly disagree” on their survey sheets. Some sample statements were phrased positively (e.g., “I could make a simple diagram that provided an overview of an entire experiment”) and others negatively (e.g., “I do not have a good sense of what motivates people to go into research”).

Three additional propositions were aimed at eliciting students’ overall sense of their ability to read/analyze journal articles, their understanding of the nature of science, and the extent to which primary literature had helped them to understand the nature of science (A, B, C). Question C had four possible responses ranging from “no influence” to “major influence,” and all other questions had five possible responses, phrased in parallel to the question posed (e.g., for question B, on understanding of the nature of science, possible answers ranged from “no understanding” to “understand it very well”). Postcourse, all surveys for which both “pre” and “post” copies were available were scored on a five-point scale, with “strongly agree” = 5 and “strongly disagree” = 1. The additional questions were scored in a parallel way, with scores for C ranging from 1–4 rather than 1–5.

RESULTS

The C.R.E.A.T.E. Survey

The C.R.E.A.T.E. survey included a collection of seven summary items (Table 2) and 52 specific skill and attitude items deemed relevant to the course goals. Following the accrual of 140 cases for the summary items and 155 cases for the skill and attitude items, the data were used to improve the usefulness of the survey. Initial inspection of the 52 skill and attitude items revealed some items were repetitious and others were unrelated to other items (low communalities) or to summary items. These items were set aside, leaving 38 candidates for continued analysis. Of the 38 items, 13 were similar to items used in research on epistemological attitude (Schommer, 1990) and attitude toward science. These are described below (Tables 5, 6, and related text). The epistemological items were a sampling of items developed by Schommer (1990) as part of a much broader investigation of epistemological beliefs. The items here were drawn from across the factors derived from Schommer's survey, and so include items representing the belief that knowledge is certain (e.g., that different scientists will come to similar conclusions) and that ability is innate (e.g., that scientists were born with a special talent), as well as items assessing attitude toward science (e.g., science is a creative endeavor). Because Schommer's previous work showed that these epistemological beliefs do not constitute a single scale, we did not include them in exploratory factor analysis, but analyzed them separately. The remaining 25 items were analyzed by means of a principal component analysis (PCA) to explore underlying factors that might aid in the reduction of the 25 variables to a more manageable set. The PCA was performed on 25 skill and attitude items, with a varimax rotation to aid with interpretation. The resulting analysis yielded eight factors accounting for 64% of the variance in the data; however, some variables were “split” across factors, resulting in two factors that were uninterpretable. These were set aside. The remaining six factors, with their member items and factor loadings, are shown in Table 3. The first factor, which we name Decoding Primary Literature, includes items that refer to scientific language and scientific literature, and indicates the respondent's feelings about reading primary scientific literature. The second factor, Interpreting Data, includes items that have to do with data presented in tables and graphs, as well as with data transformations. The third factor, Active Reading, includes items about diagrams, displays, and method. Visualization, the fourth factor, includes visualizing the method of a study and interpreting graphs. The fifth factor is named Thinking Like a Scientist and includes items about explaining a scientific paper and thinking of experiments. The final factor is called Research in Context and includes items about animal models and controls in experiments.

Table 2.

Seven summary items used on the C.R.E.A.T.E. survey

On a scale of 1–5, rate your confidence in your ability to read and analyze science journal articles.a
On a scale of 1–5, rate your understanding of “the way scientific research is done” or “the scientific research process.”b
When was the last time that you read an article from the primary scientific literature (e.g., a journal article)?
How many articles from the primary scientific literature (e.g., journal articles) have you read?
How much influence have journal articles had on your understanding of science?
Outline the path from a scientist's initial thoughts to a completed research study in a published journal article. Please be as detailed and complete as you can.
Journal articles are (choose the single best answer) a) hard to read and not worth the effort, b) hard to read but worth the effort, c) easy to read but not worth reading, or d) easy to read and worth reading.

a For this item, 1 = zero confidence, 2 = slightly confident, 3 = confident, 4 = quite confident, and 5 = extremely confident.

b For this item, 1 = I don't understand it at all, 2 = I have a slight understanding, 3 = I have some understanding, 4 = I understand it well, and 5 = I understand it very well.

Table 5.

Items from the C.R.E.A.T.E. survey that measure epistemological beliefsa

Knowledge is certain.
If two different groups of scientists study the same question, they will come to similar conclusions. (R)
The data from a scientific experiment can only be interpreted in one way. (R)
Because scientific papers have been critically reviewed before being published, it is unlikely that there will be flaws in scientific papers. (R)
Because all scientific papers are reviewed by other scientists before they are published, the information in the papers must be true. (R)
Sometimes published papers must be reinterpreted when new data emerge years later.
Results that do not fit into the established theory are probably wrong. (R)
Ability is innate.
I think professionals carrying out scientific research were probably straight-A students as undergrads. (R)
You must have a special talent in order to do scientific research. (R)
Attitude toward science.
Science is a creative endeavor.
I have a good sense of what research scientists are like as people.
I do not have a good sense of what motivates people to go into research. (R)
Scientists usually know what the outcome of their experiments will be. (R)
Collaboration is an important aspect of scientific experimentation.

a Items followed by an (R) were reverse-scored for analysis.

Table 6.

The results of paired-difference t tests for items (certain knowledge, innate ability, and attitude toward science) in Table 5

Item	Pretest mean (SD)	Posttest mean (SD)	Statistical significance	Mean difference/SD of the differencea
Certain knowledge	19.7 (2.2)	20.7 (2.7)	p < 0.001	0.40
Innate ability	7.5 (1.7)	8.1 (1.5)	p < 0.001	0.36
Creativity	4.1 (0.85)	4.4 (0.73)	p < 0.001	0.30
Sense of scientists	3.1 (0.93)	3.8 (0.77)	p < 0.001	0.70
Sense of motives	3.6 (0.95)	4.0 (1.0)	p < 0.001	0.31
Known outcomes	4.0 (0.82)	4.3 (0.81)	p < 0.001	0.30
Collaboration	4.4 (0.73)	4.6 (.66)	p < 0.006	0.22

aEstimate of the magnitude of the effect.

Table 3.

Items from the C.R.E.A.T.E. survey arranged according to a PCA with varimax rotationa

Factor	Item	Factor loading	Cronbach's alpha
1	The scientific literature is difficult to understand (R).	0.776
Decoding Primary Literature	When I see scientific journal articles, it looks like a foreign language to me (R).	0.593
	I am not intimidated by the scientific language in journal articles.	0.558
	I am confident in my ability to critically review scientific literature.	0.500	0.71
	I am comfortable defending my ideas about experiments.	0.328
2 Interpreting Data	It is easy for me to transform data, like converting numbers from a table to percents.	0.796
	If I see data in a table, it is easy for me to understand what it means.	0.680
	If I am shown data (graphs, tables, charts), I am confident that I can figure out what it means.	0.622	0.72
	It is easy for me to relate the results of a single experiment to the big picture.	0.352
3 Active Reading	I could make a simple diagram that provides an overview of an entire experiment.	0.763
	If I am assigned to read a scientific paper, I typically look at the methods section to understand how the data were collected.	0.584
	I do not know how to design a good experiment (R).	0.522	0.63
	The way that you display your data can affect whether or not people believe it.	0.345
4 Visualization	When I read scientific information, I usually look carefully at the associated figures and tables.	0.694
	When I read scientific material it is easy for me to visualize the experiments that were done.	0.649	0.75
	If I look at data presented in a paper, I can visualize the method that produced the data.	0.592
	When I read a paper, I have a clear sense of what physically went on in a lab to produce the results and information I am reading.	0.584
5 Thinking Like a Scientist	After I read a scientific paper, I don't think I could explain it to somebody else (R).	0.735
	I am confident I could read a scientific paper and explain it to another person.	0.655
	I enjoy thinking of additional experiments when I read scientific papers.	0.394	0.59
	I accept the information about science presented in newspaper articles without challenging it (R).	0.231
6 Research in Context	Experiments in “model organisms” like the fruit fly have led to important advances in understanding human biology.	0.774
	Progress in curing diseases has been made as a result of experiments on lower organisms like worms and flies.	0.597	0.35
	I understand why experiments have controls.	0.540

a Items followed by an (R) are reverse-scored. Cronbach's alpha, an index of inter-item consistency, is also shown.

Pretest–Posttest Differences

Raw scores for the items in each factor were summed, resulting in six pretest scores and six posttest scores for each student respondent. A paired-difference t test was performed on each of the six factors. The results are shown in Table 4. Each pretest–posttest difference is highly significant in the expected direction of posttest gains. The magnitude of the change, estimated as standard deviations in the final column of Table 4, is medium to large (Cohen, 1988). This magnitude of change, as well as the stringent level for significance testing, argues against the presence of a type I error (spurious significant differences).

Table 4.

The results of paired-difference t tests for raw data totals for each of the six factors in Table 3

Factor	Pretest mean (SD)	Posttest mean (SD)	Statistical significance	Mean difference/SD of the differencea
1	15.5 (3.6)	19.2 (2.9)	p < 0.001	0.93
2	13.6 (2.5)	16.4 (2.1)	p < 0.001	1.00
3	13.6 (2.2)	16.2 (2.4)	p < 0.001	0.84
4	13.2 (2.5)	15.8 (2.3)	p < 0.001	0.96
5	13.5 (2.3)	16.2 (2.1)	p < 0.001	0.97
6	12.6 (1.7)	14.0 (1.3)	p < 0.001	0.74

aEstimate of the magnitude of the effect.

Epistemological Beliefs

Table 5 shows 13 items related either to Schommer's constructs of certain knowledge and innate ability or to a general attitude toward science. It would be expected that students with an insightful attitude toward science would believe that knowledge is not certain and unchangeable; that scientific ability is not fixed and innate, and that science is a creative and collaborative endeavor. To explore the change in these variables from pre- to posttest, the items under the certain knowledge heading were summed (Cronbach's alpha = 0.66 for this scale). Similarly, the two items for innate knowledge were summed. The attitude items were examined individually. The result of paired-difference t-tests for pre- versus postcourse data are shown in Table 6. There were significant positive gains on all the variables. The possibility of the presence of a type I error is reduced by the stringent level for significance and the moderate effect sizes.

Relationships to Summary Item of Reading Confidence

In an effort to synthesize the findings, we chose to ask whether any of the components of the C.R.E.A.T.E. survey identified so far relate to the student participant's overall view of his or her ability to read science articles. For this purpose, we set the data for the first summary item (rate your confidence in your ability to read and analyze science journal articles) as a dependent variable for multiple linear regression. Specifically, the postcourse confidence data were analyzed using a multiple regression analysis, with predictors including precourse responses to the same item (pretest reading confidence), the sums of the raw data scores indicated by the factor groupings in Table 3 (postcourse data), and the epistemological items in Table 5 (postcourse data). A stepwise multiple regression analysis indicated that a model with three significant predictors, the scores relating to Decoding Primary Literature, the first factor in Table 3 (standardized coefficient 0.33; p < 0.01); the scores relating to Thinking Like a Scientist, the fifth factor in Table 3 (standardized coefficient 0.27; p < 0.01); and the pretest reading confidence level (standardized coefficient 0.17; p < 0.02), significantly predicted posttest reading confidence (r = 0.58; df = 3, 126; p < 0.05).

DISCUSSION

Experiences That Shift Students’ Attitudes about Science and Their Own Scientific Abilities

The one-semester C.R.E.A.T.E. course enhanced students’ confidence in their ability to read, understand, and explain science, as well as to understand how research is carried out (Tables 3 and 4). Students changed significantly on summary variables that assessed self-rated ability to design experiments, visualize methods based on data, visualize lab activities based on the written account in the journal article, manipulate data, relate results of individual experiments to “the big picture,” critically review data, read science with appropriate skepticism, and explain results to others. These topic areas touch on a wide range of activities undertaken by working researchers.

Our survey findings are consistent with a model that predicts that students’ overall confidence in their ability to read and analyze journal articles is based largely on their sense that they can: 1) Decode Primary Literature (Table 3, factor 1) and 2) Think Like a Scientist (Table 3, factor 5). The gains in students’ confidence in these abilities are likely related to the design of the C.R.E.A.T.E. course. Class discussion and homework assignments challenge students to work through the experiments and interpret the findings of published studies as if they had done the work themselves. During the semester, students carry out numerous activities typical of working scientists. As such, the approach may promote more epistemological engagement than does a standard class, thus engendering changes in students’ attitudes and beliefs about science.

Overall, the approaches typical of the C.R.E.A.T.E. classroom are likely to encourage students to employ metacognitive strategies that help them interpret complex information (Hartman, 2002; Nordell, 2009). Gaining deeper understanding of challenging material may in turn help students overcome widely held misconceptions about research science, for example, that experiments are done to demonstrate concepts already known, and therefore research is not a creative activity, and that researchers are simply fact-gatherers (Sandoval, 2003). Students’ repeated opportunities to design and evaluate experiments and models throughout the C.R.E.A.T.E. semester (Figure 1 and Table 1) may contribute to the shifts we noted in their views about the creativity of science (Tables 5 and 6). Developing their own questions for paper authors (e.g., “Do you have to be a straight-A student to become a researcher?” “What would be your ‘dream discovery’?”) encourages students to think beyond the data of the papers and consider the overall process of becoming and being a scientist.

Currently, undergraduate research experiences (UREs) are considered to be one of the most important mechanisms for stimulating students’ interest in science careers. The effects of UREs have been investigated using surveys of students’ experiences (Rauckhorst ; Lopatto, 2004a, b, 2007, 2009; Russell, 2006; Russell ) and extended interviews (Hunter ). These studies have reported high student enthusiasm for UREs, as well as major benefits for multiple aspects of students’ understanding of science, their hands-on research skills, and their attitudes toward research careers. In a longitudinal ethnographic study of the effects of UREs on students at four liberal arts institutions, participants noted in interviews that UREs increased their research confidence and sense of “feeling like a scientist” (Hunter ). Students who participate in science UREs are likely to already be interested in scientific research, and UREs clearly reinforce the aspirations of these students. In this respect, C.R.E.A.T.E. may reach a broader group of students, including many who have not previously considered careers in science.

The concept of “nature of science” includes, for many science educators, the ideas that scientists build understanding on observations of nature, that explanations and understanding can change over time, and that creativity comes into play throughout the research process (Lederman, 1992; Karakas, 2009). While there is general agreement that students need to understand the nature and processes of science, or “science as a way of knowing” (AAAS, 1993, p. 2; see also AAAS 1989, 2010; NRC 2009), it is less clear how to accomplish this goal. Teaching approaches focused on inquiry have been suggested as a way to build student understanding of the nature of science (Aulls and Shore, 2008; Shore ) and enhance learning (Quitadamo ). “Inquiry” alone, however, may not be sufficient to shift students’ concepts of the nature of science (Lederman ; Sandoval, 2003; Schwartz ) or to encourage students to use metacognitive approaches when studying science (Butler ). Our study finds that, although the C.R.E.A.T.E. course does not include a laboratory component or independent research projects, students nevertheless report substantial changes in attitudes and beliefs about science during the semester. Being challenged to devise their own research questions, analyze and interpret data, design experiments, and carry out peer review of studies devised by other students may stimulate C.R.E.A.T.E. participants to examine their personal beliefs about science. In addition, student interview data suggest the email survey of authors plays a role in shifting students’ understanding of “who does science, and why?” (see Tables 1 and S1 in Hoskins ). In this context, it is notable that students participating in a novel Deconstructing Scientific Research course, which focuses on intensive analysis of an individual research seminar and also lacks a hands-on component, showed large gains in multiple categories addressed by the Survey of Undergraduate Research Experiences instrument (Lopatto, 2004a, 2007, 2009), including the ability to “understand how knowledge is constructed” (Clark ).

Epistemologies, Learning, and the Nature of Science

Our survey also addressed aspects of students’ epistemological beliefs. Schommer and colleagues (Schommer, 1990; Schommer ) have identified epistemological beliefs that moderate learning in a variety of intellectual domains. For example, the beliefs that knowledge is certain, that authority should be trusted, that learning is quick and simple, and that intellectual talent is innate can interfere with striving to learn. The C.R.E.A.T.E. program, by uncovering the process of scientific thinking and by providing contact between students and professionals, may influence these epistemological beliefs in a beneficial way. We found substantial changes during the semester in students’ views of “scientists”; moderate shifts in students’ sense of whether knowledge is certain and ability is innate, the creativity of science, or understanding of motives that drive scientists; and a small shift in students’ views of science as a collaborative activity (Tables 5 and 6).

Undergraduates’ epistemological beliefs shift during the college years from a sense that knowledge is certain, typical of freshmen, to a more nuanced view of the relative nature of knowledge, held by seniors (Perry, 1970), and a longitudinal study suggests such views continue to change postcollege (Baxter Magolda, 2004). Epistemological beliefs change slowly during the college years (Perry, 1970), and only a minority of students achieve mature epistemological understanding by senior year (Baxter Magolda, 1992). For both high school (Schommer, 1993) and college (Schommer, 1990, 1993; Hofer, 2000, 2004) students, the sophistication of their epistemological beliefs correlates with their reading comprehension and academic performance, with naïve beliefs linked to lesser achievement. Epistemological beliefs also affect student metacognition (Hartman, 2002; Schommer-Aikins, 2002; Hofer, 2004), ability to integrate information (Schommer, 1993), and persistence when confronted with a challenging task (Dweck and Leggett, 1988). The interrelationships among personal epistemologies, metacognition, and learning are complex, but there is general agreement that naïve epistemologies may interfere with learning (Hofer, 2004; Pulmones, 2010). Overall, students with naïve epistemologies employ fewer of the metacognitive strategies (e.g., setting goals, monitoring progress, self-questioning, and connecting new information to broader concepts) that support self-directed learning (Zimmerman, 1990; Hartman, 2002; Pieschel ; Strømsø ).

The C.R.E.A.T.E. method's combination of epistemically challenging approaches applied in an authentic context may underlie the changes we saw in students’ epistemological beliefs. Several aspects of the C.R.E.A.T.E. approach present students with novel cognitive challenges and associated epistemic load. C.R.E.A.T.E. students employ visualization when sketching cartoons that fill the gap between the methods section and the charts, graphs, blots, and/or photomicrographs presented. Integrating verbal information with visual information promotes integration of different modalities. Such integrative thinking, reinforced by C.R.E.A.T.E.'s repeated use of concept maps, both as a tool for review and a way to organize papers’ central themes, can facilitate learning (Novak, 1991; Van Meter and Garner, 2005; Schwartz and Heiser, 2006). Class discussion often focuses on a point of controversy, which can both increase student engagement (Bell and Linn, 2002) and stimulate students to “do the real intellectual ‘work’ of synthesizing ideas across subdomains” (Seethaler, 2005, p. 273). C.R.E.A.T.E.'s narrow focus on a few papers may encourage students to work toward deep rather than superficial understanding (Schwartz ) as they engage in cognitively stimulating activities corresponding to upper levels of Bloom's taxonomy (e.g., analysis, synthesis, evaluation: levels 4–6; Bloom and Krathwohl, 1956). Extended discussions involving scientific argumentation are rare in lecture-dominated classrooms (Osborne, 2010), but can be of substantial benefit, especially when students feel free to develop their understanding through discussion and to speculate aloud as they do so (Sawyer, 2006). C.R.E.A.T.E. “grant panel” activities encourage student reflection on the research process beyond the details of individual journal articles.

UREs might also be expected to have a strong effect on students’ epistemological beliefs. This has been seen in some cases (Rauckhorst ; Lopatto, 2004b; both studies include both science and nonscience URE participants), but in other studies epistemological beliefs appeared not to shift significantly during the URE, either as reported by student participants or their faculty mentors (Hunter ). A recent meta-analysis of independent research experiences in science (Sadler ) suggests that supplementing research experiences with specific additional activities, such as keeping a journal of reflections on the research experience (Rauckhorst , college students; Schwartz , high school teachers) or interacting with peers also involved in research apprenticeships (Grindstaff and Richmond, 2008; high school students), can expand gains made in UREs and enhance understanding of the nature of science. These researchers further note that developing a deeper understanding of the nature of science will probably require instructional approaches that ensure undergraduates’ participation in developing hypotheses and analyzing data, both considered “epistemically demanding practices” (Sadler and McKinney, 2010, p. 48). Other investigators have suggested that epistemological beliefs are more likely to change if students are trained to think critically in a context that encourages metacognition and includes controversy (Valanides and Angeli, 2005). In studies of high school students (Bell ) and undergraduates (Ryder ), changes in scientific thinking were seen in students who participated in projects that demanded substantial epistemic engagement. Conversely, classrooms lacking in authentic scientific inquiry activities can reinforce naïve epistemological beliefs, for example, that scientific logic is simple and conclusions certain (Chinn and Malhotra, 2002). Students’ ability to carry out research projects may be constrained by such beliefs (Ryder and Leach, 1999).

We consider it likely that the shifts we see in epistemological beliefs of C.R.E.A.T.E. students are attributable to students’ experiences in the C.R.E.A.T.E. course. We did not measure presemester/postsemester epistemological beliefs in an independent control group of students who did not take the course, as no such control group was available. We are not, however, aware of any studies that show that increased sophistication of epistemological beliefs results from mere maturation or passage of time during a single semester. The experiences of science students in UREs may provide some insight into the malleability of students’ epistemological beliefs. Science students’ UREs would be expected to support or enhance any shifts in their epistemological beliefs that occurred “maturationally” during an academic semester. Thus, the finding that epistemological beliefs tend to remain stable in science URE participants interviewed repeatedly over several years (Hunter ), suggests that the epistemological beliefs of undergraduate science students do not shift rapidly. We feel it is likely that the postsemester versus presemester changes we document were brought about by students’ experiences in the semester-long C.R.E.A.T.E. course.

Although the C.R.E.A.T.E. approach is not unique in focusing on primary literature, it is unusual in its combination of intensive analysis of a series of related publications with an email survey of their authors, and its concentration in the classroom on discussion and analysis aimed at simultaneously decoding the figures and tables, modeling the research process, and humanizing the scientists behind the papers. The C.R.E.A.T.E. teaching method encourages students to engage in conversations, debates, and creative thinking, which involve cognitive challenges that can help develop understanding of complex material (Driver ; Marbach-Ad and Sokolove, 2000; Bell and Linn, 2002; Seethaler, 2005; Campion ) and at the same time encourage creative approaches to such material (DeHaan, 2009). Overall, our findings indicate that the C.R.E.A.T.E. method increases students’ confidence in their ability to read and understand primary literature, improves their self-assessed understanding of the nature and processes of science, and encourages their development of more sophisticated epistemological beliefs. We suggest that complementing existing curricula with inexpensive C.R.E.A.T.E.-style courses could be an effective way to help students develop deeper insight into the nature and practices of science. Finally, students who recognize early in their college years that science is creative and open-ended might be more likely to take advantage of the UREs that can stimulate and reinforce interest in science research careers.