Using primary literature to teach science literacy to introductory biology students.

Undergraduate students struggle to read the scientific literature and educators have suggested that this may reflect deficiencies in their science literacy skills. In this two-year study we develop and test a strategy for using the scientific literature to teach science literacy skills to novice life science majors. The first year of the project served as a preliminary investigation in which we evaluated student science literacy skills, created a set of science literacy learning objectives aligned with Bloom's taxonomy, and developed a set of homework assignments that used peer-reviewed articles to teach science literacy. In the second year of the project the effectiveness of the assignments and the learning objectives were evaluated. Summative student learning was evaluated in the second year on a final exam. The mean score was 83.5% (±20.3%) and there were significant learning gains (p < 0.05) in seven of nine of science literacy skills. Project data indicated that even though students achieved course-targeted lower-order science literacy objectives, many were deficient in higher-order literacy skills. Results of this project suggest that building scientific literacy is a continuing process which begins in first-year science courses with a set of fundamental skills that can serve the progressive development of literacy skills throughout the undergraduate curriculum.

INTRODUCTION

Studies have shown that incorporating the scientific literature into the undergraduate curriculum advances critical thinking, enhances quantitative literacy (27), promotes more sophisticated analysis of data and improves students’ ability to develop rigorous arguments about science (9, 15). Exposing students to the scientific literature in their courses also helped them develop skills in accessing journal articles, extracting information from them, evaluating data, and developing experimental designs (23, 27, 30). Finally, undergraduate programs that incorporated literature-intensive courses in the curriculum were able to show long-term gains that facilitated undergraduate transition to graduate school (13). Students reported that their experience with the scientific literature helped them later as they researched topics in the literature (19), critiqued the literature, listened to scientific presentations, and developed questions about their own research (12, 18).

There are many reports describing assignments and strategies that use the scientific literature in the classroom (6, 7, 11, 17, 20, 21, 24, 28, 29). Almost all of these focus on teaching a set of fundamental skills: access, retrieve, analyze, and evaluate (23). However, few reports discuss assessments or evaluations of student learning achieved through using these strategies. Students, particularly novice science students, often struggle as they work with the scientific literature. Some students may miss the point of an article because they do not understand the actual practice of the scientific method. Others may struggle with the language of the article. Still, others cannot integrate the article in the broader context of the discipline and thus are unable to separate important information from tangential information. Finally, a number of students may not have the sophisticated critical thinking skills needed to interpret the results or conclusions (3, 9, 26). Thus learning to read the scientific literature can be a challenge. Some educators have suggested that students’ problems may be related to deficiencies in their science literacy skills (10).

Science literacy has been broadly defined as the ability to understand how science is practiced, how science knowledge is gained, and how science is distinguished from other kinds of knowledge (1, 16, 22). For the scientist-expert, however, the definition of science literacy is more rigorous and demands that a scientist be fluent enough in the discipline to gather and integrate information (9), skilled quantitatively and able to interpret numeric information and data (27), and able acquire and use information from primary sources to defend statements and conclusions (9). This expertise depends largely on Bloom’s higher cognitive domains, analyzing, evaluating, interpreting, and synthesizing (2, 4). Developing expertise in science literacy requires a strong foundation in lower-order cognitive domains of science literacy, knowledge, understanding, and application. Teaching science literacy, particularly to novices, can be challenging and takes a considerable investment of resources, especially time. A recent study suggested that while faculty strongly support teaching these processing skills to undergraduates, more than half felt conflicted about spending time teaching them, particularly at the expense of content (8).

In this two-year study we present a strategy for using the scientific literature to teach science literacy skills to novice students in a way that will serve as a foundation for developing higher-order literacy skills throughout the undergraduate curriculum. We test the hypothesis that novice students in an introductory biology course could develop literacy skills with repeated guided practice of reading peer-reviewed articles and answering homework questions that targeted science literacy learning objectives aligned with lower-order cognitive domains of science literacy.

METHODS

Project population and course organization

This study took place in an eight-week session of the second semester of Introductory Biology at a midsized midwestern state university (Youngstown State University, Youngstown, Ohio). The project spanned two consecutive academic years (Year 1 and Year 2). There were 148 students enrolled in the course in Year 1 (57.4% women, 42.6% men) and 127 students in Year 2 (61.4% women, 38.6% men). Students in the course were preparing for careers in biology, medicine, dentistry, veterinary medicine, pharmacy, physical therapy, clinical lab science, forensic science, and exercise science. Course prerequisites include algebra, trigonometry, and introductory chemistry although concurrent enrollment in introductory chemistry was permitted.

Experimental design

This manuscript presents a two-year study in which Year 1 (Y1) served as a preliminary investigation that guided Year 2 (Y2). During Y1 the project evaluated student science literacy skills, identified deficiencies, created a set of science literacy learning objectives, aligned the learning objectives with Bloom’s taxonomy and developed a teaching strategy to address them. In Y2 the project evaluated learning objectives and curricular protocol developed in Y1.

During the first year of the study students took two tests, an objective assessment, the TIPS II test (5), and a subjective assessment, Homework 1 (HWK1), to help the instructor identify their science literacy deficiencies. Results from both assessments were used to develop learning objectives for the project and a teaching strategy. Science literacy learning objectives were aligned with Bloom’s lower-order cognitive domains of knowledge, understanding, and application (2). A preliminary evaluation of the project was performed throughout Y1 using student performance on four homework assignments, three interim exams, and a summative final exam. Homework assignments consisted of short-essay questions based on journal article readings. Science literacy learning objectives were embedded in all homework assignments. Interim exams contained questions that enabled the instructor to compare science literacy learning to science content learning. Course grades were based on exams, homework, and attendance. Interim exams were worth 60 %, homework, 12%, attendance, 8%, and the final exam, 20% of the total grade. Project results were reviewed at the end of the year and the preliminary investigation completed.

In Y2 the project evaluated learning objectives and curricular protocols developed in Y1. The course structure in Y2 was similar to Y1 except for three changes that resulted from Y1 observations. First, science literacy questions on interim exams which compared science literacy learning to science content learning were discarded because Y1 results showed that there was no significant difference in content and science literacy learning. Second, HWK1 replaced the TIPS II test as a science literacy assessment tool to inform the instructor about how students used science literacy in the professional setting. Finally, student science literacy learning gains were evaluated by pre-/postcomparison of scores on HWK1 and the final exam.

Year 1 preliminary study

Characterizing science literacy skills of first year biology students

Student literacy skills were evaluated using two assessment tools, TIPS II and HWK1. The TIPS II test assessed five scientific processing skills: identifying variables, using operational definitions, stating hypotheses, graphing and interpreting data, and designing investigations (5). In some questions the instructor substituted experimental design or data questions of TIPS II with parallel design questions using data founded in biological science. However the structure and framework of the question was retained. Several questions that evaluated student understanding of the organization of the scientific literature and the peer-review process were added to the test. In the second assessment of student literacy skills, HWK1, students were asked to read a journal article and answer a set of 12 short-answer questions.

Using homework assignments to teach science literacy.

The results of TIPS II and HWK1 were used to develop science literacy learning objectives (Table 1) and a set of homework assignments (HWK 2, 3, 4) that embedded these skills. Learning objectives were aligned with Bloom’s lower-order cognitive domains of knowledge, understanding, and application. Before students started these assignments, the instructor gave a lecture that reviewed the structure and organization of the scientific literature and explained the rationale and content of each section of a journal article. The presentation was available on the course website for review.

TABLE 1

Learning objectives and sample homework questions.

Science Literacy Learning Objectives for Project	Sample Homework Questions
1. Extract information from journal articles using knowledge of organization and structure of scientific literature.	What section would you read if you wanted to quickly find out if a paper had information that was useful to you?
2. Understand how science is practiced and how scientific knowledge is generated.	Find the answers to the questions below and write down where you found them (section). ○ What organism was studied in this paper? Describe the experimental design (# subjects, what they did for how long, how many times, what was observed/measured).
3. Know and identify the components of the scientific method (introduction, hypothesis, methods, results, conclusions, control, variable, etc.) in journal articles.	What hypothesis was tested in this article? Where did you find this information? Are the results of this study in agreement with other studies reported in the scientific literature? Where did you find this information? In 3–4 sentences describe why the authors did this study. Where did you find this information?
4. Interpret and explain the basic construction of the experimental protocol, explain its alignment to the hypothesis.	What components of the diet were different in the control and hypercholesterol groups?
5. Identify and critique the controls of a study.	Describe the experimental design for this study. Be sure to include a discussion of the control group(s).
6. Interpret and explain the results presented in graph or text.	Make a bar graph of the basal metabolic rate (kJ/22 hrs) found in Table 3. Make a separate bar graph of the basal metabolic rate (kJ/22 hrs/kg) found in Table 3. Why can’t you put both sets of data on the same graph?
7.Make inferences or conclusions using statistics and hypothesis testing and justify them with data from the article.	Look at Figures 3 and 4 and compare the aortas of the low cholesterol/exercise animals and the high cholesterol/no exercise animals. Would you consider modifying your diet based on this evidence in rats?
8. Evaluate or critique the results or conclusion and justify the position.	Do you think that capillary density affects how well a cardiac muscle functions or contracts? Would hypertrophied heart muscle work better with high capillary density or low capillary density?
9. Build personal knowledge and understanding from journal articles.	Review the vocabulary list below. If you don’t know the definition of any of these terms, find a definition or description on the Internet and type them up in a vocabulary list. Anabolic steroids, myocyte, Dianabol.

TABLE 3

Generic rubrics with examples of sample questions.

Science Literacy Learning Objective	Sample Question	Generic Rubric
LO1. Understand the genre, organization, and style of articles in peer-reviewed scientific literature.	What section would you read if you wanted to quickly find out if a paper had information that was useful to you?	Identify appropriate section. Explain why it was chosen.
LO2. Know and identify the components of the scientific method in journal articles.	What hypothesis/hypotheses was/were tested in this article? Where did you find this information?	Locate the hypothesis in the article. Demonstrate understanding of the function of the hypothesis in the protocol. Explain the relevance and application of the hypothesis to the article.
LO3. Apply knowledge of the organization and structure of scientific literature to extract information from journal articles.	Find the answers to the questions below and write down where you found them (section). What organism was studied in this paper? Describe the experimental design (# subjects, what they did for how long, how many times, what was observed/measured).	Identify appropriate section. Provide answer to question. Justify why answer was chosen.
LO4. Interpret and explain the basic construction of the experimental protocol, explain its alignment to the hypothesis.	What components of the diet were different in the control and hypercholesterol groups?	Describe the experimental protocol. Answer question and explain how answer (hyper-cholesterol) fits into experimental protocol. Describe how answer (hypercholesterol) aligns with hypothesis.
LO5. Identify and critique the controls of a study.	How did this study control for the possible weight difference between exercising animals and sedentary animals?	Identify controls in study. Demonstrate understanding of the general concept and role of controls in an experimental protocol. Explain relevance of specific controls to protocol.
LO6. Interpret and explain the results presented in graph or text.	Make a bar graph of the basal metabolic rate (kJ/22 hrs) found in Table 3.	Provide graph in appropriate format (bar, scatter, etc.). Provide graph with appropriately labeled axes, appropriately placed data values, a title, a legend when necessary, error bars, p values.
LO7. Make inferences or conclusions and justify them with data from the article.	Look at Figures 3 and 4 and compare the aortas of the low cholesterol/exercise animals and the high cholesterol/no exercise animals. Would you consider modifying your diet based on this evidence in rats?	Provide statement. Rationalize or justify statement using evidence requested.
LO8. Construct an evaluation or critique of the results or conclusion and justify the position.	Could drinking super oxygenated water improve your workouts?	Define position with a clear statement. Present statement of position that is consistent with data and significance presented in article. Use evidence from article to support position.
LO9. Build personal knowledge and understanding from journal articles.	Review the vocabulary list below: anabolic steroids, myocyte, Dianabol. If you don’t know the definition of any of these terms, find a definition or description on the Internet and type them up in a vocabulary list.	Provide a definition of term. When multiple definitions are available for term, select definition that is relevant to article. Explain application of term in article.

Students were asked to read a peer-reviewed article related to a topic covered in a lecture and answer 10–12 short-essay questions for all homework assignments. Each homework question embedded the learning objectives in Table 1. Journal articles (Table 2) were selected based upon their alignment with course goals, course content, and learning objectives (21). Articles were also evaluated for accessibility, how easily students could understand the premise of the article, and whether the relationship between the data and conclusion was simple and direct enough to be understood by novices (26). Before each assignment the instructor explained how the journal article related to lecture topics, described the learning targets of the assignment and, if necessary, provided students with the background to understand the experimental problem and methodology. Homework questions were scored using a rubric that was framed by the science literacy objective (LO) that it addressed. When questions with a common learning objective were used on more than one homework assignment, a generic rubric was developed and then adapted to the specifics of the journal article in the homework assignment. Examples of rubrics are shown in Table 3. Rubrics for LO6, quantitative literacy, LO7, making inferences with data, and LO8, constructing and justifying evaluations or critiques, were shaped by the literature (9, 23, 28). Assignments were graded by the instructor and returned with comments prior to the submission of the next assignment. Scores for each question were recorded, the percentage of students with correct answers calculated, and commonly missed aspects of the literacy skill noted. The instructor discussed homework answers, addressed misconceptions observed in homework answers, and explained the answers to commonly missed questions in class.

TABLE 2

Examples of homework assignment readings.

Reading Assignment
“Effect of Energy Restriction on Total Heat Production, Basal Metabolic Rate, and Specific Dynamic Action of Food in Rats.” (1981) Forsum E, Hillman PE, and Nesheim MC. J Nutr 111:1691–1697.
“Anabolic steroids impair the exercise-induced growth of the cardiac capillary bed.” (2000) Tagaralos CV, Bloch W, Hollmann W, and Addicks K. Int J Sports Med 6:412–418.
“Maximum cardiac performance and adrenergic sensitivity of the sea bass Dicentrarchus labrax at high temperatures.” (2007) Farrell AP, Axelsson M, Altimiras J, Sandblom E, and Claireaux G. J Exp Biol 210:1216–1224.
“Exercise Attenuates Diet-Induced Arteriosclerosis in the Adult Rat.” (1987) Hasler CM, Rothenbacker H, Mela DJ, Kris-Etherton PM. J Nutr 117:986–993.

Using interim exams to assess science literacy learning.

In Y1, interim multiple-choice exams framed by Bloom’s taxonomy tested student knowledge, understanding, application, analysis, and evaluation of lecture content and science literacy skills. Exams were scored by computer and student performance on lecture content questions was calculated separately from science literacy questions. Student scores for the two areas were compared to see if an objective assessment could measure a difference in learning. Student score distribution, class mean, and standard deviation were calculated for each exam and differences in exam means were determined using a t-test.

Using the final exam as a summative evaluation.

The final exam was composed of two parts: 1) a cumulative multiple-choice exam covering lecture material and 2) a cumulative essay exam of science literacy skills. Cumulative multiple-choice exams were scored by computer. For the science literacy portion of the exam, students were given a peer-reviewed journal article to read one week before the final exam and told that they would be tested on the article with questions similar to those on their homework. On the day of the exam students were provided with a copy of the article and 11 short-answer questions similar in style and construction to the homework questions. Exam questions assessed the science literacy skills identified in Table 1. The final exam was graded similarly to homework assignments and all grading was performed by the instructor. The learning objective and literacy skill of each question was ascertained, the appropriate rubric identified and modified to suit the journal article, and the question graded using this rubric. Scores for each question were recorded and analyzed for percentage of students with the correct answer.

Year 2

Course organization in Y2 was similar to Y1. Students were given four homework assignments, three interim exams, and a summative final exam. Course grades were based on exams, homework, and attendance and calculated as in Y1. Exams in Y2 were similar to Y1 in structure and content except that neither interim nor final exams contained science literacy questions. Homework assignments consisted of short-essay questions based on journal article readings and science literacy learning objectives were embedded in all homework assignments. At the beginning of Y2, baseline literacy skills were evaluated with a homework assignment (HWK1). After this assignment was completed the instructor gave a lecture that reviewed the organization of the scientific literature. The presentation was available on the course website for review. All homework assignments were graded by the instructor using the generic rubrics developed in Y1. Science literacy learning gains were evaluated by comparing scores on HWK1 and the final exam.

Youngstown State University Institutional Review Board approval for these studies was granted to protocol HSRC 161-09 in July 2009.

RESULTS

Year 1 – Characterizing science literacy skills of first year biology students using TIPS II test

TIPS II was used as an objective assessment of student knowledge, understanding, and application of the scientific method and scientific processing skills. The mean score of the TIPS II test was 56.7% +/−13.7 (n = 110). Overall the results revealed that students had knowledge of the terms and concepts associated with the scientific method but did not completely understand them or were unable to apply their knowledge in a practical way to research questions (Table 4). For example most students knew the operational definition of peer-reviewed literature, hypothesis, control, mean and standard deviation, and variable, but they were unable to apply their knowledge in a practical way to an experiment. TIPS II scores on science processing skills, especially data analysis skills, reflected the complexity of the tasks. While most students were able to evaluate and extract information from simple graphs, fewer could extract or evaluate information from complex graphs and tables similar to those prevalent in the literature. Finally, critiquing and evaluating experimental design, a sophisticated skill associated with higher-order cognitive domains, was a difficult task for many students. Thirty-nine percent of the students could not select protocols appropriate for graphed data.

TABLE 4

Results of TIPS II exam – baseline science literacy skills.

Term or Concept	Objective of Question	% Students (n = 110)
Hypothesis	Understand definition.	97%
	Apply understanding
	• Pick out appropriate hypothesis for experimental protocol.	72%
	• Identify appropriate hypothesis for multivariable experiment.	68%
Control	Understand definition.	88%
	Apply understanding
	• Identify appropriate control for multivariable experiment.	44%
Mean, standard deviation	Understand definition.	100%
	Apply understanding
	• Use mean and standard deviation to identify most reliable data on graph.	28%
Variable	Understand definition.	Not asked
	Apply understanding
	• Identify critical variable in an experiment.	12%49%, 35%a
	• Discriminate between dependent and independent variables in a protocol.
Peer review	Understand definition.	100%
	Apply understanding
	• Select peer-reviewed literature over non–peer-reviewed literature appropriately.
Graph analysis skills	Extract information from simple line graphs.	95%, 90%
	Apply understanding
	• Interpret or extract information from complex graphs with multiple plots.	32%, 40%a

There was more than one question on the exam addressing this concept. Each percentage represents class performance on a specific question on this term/concept.

Year 1 – Characterizing science literacy skills of first year biology students as they read the literature (HWK1)

In the second assessment of baseline student literacy skills, HWK1, students were asked to read a journal article and answer a set of 12 short-essay questions. This assessment tool was used to evaluate how students applied their literacy skills in situations encountered by professionals, such as reading journal articles, interpreting graphs, and developing conclusions. The results of HWK1 were consistent with the outcome of the TIPS II test. They revealed that while students knew the terms and concepts associated with the scientific method they did not completely understand them or were unable to apply them to evaluate and critique the scientific literature (Table 5).

TABLE 5

Results of Year 1 HWK 1 – baseline science literacy skills.

Term or Concept	Objective of Question	% Students with Correct Answer (n = 107)
Purpose of experiment	Locate information.	100%
	Explain and support statement with citations of previous studies.	29.5%
Hypothesis	Locate information.	93.5%
	Explain details.	23.4%
Experimental method	Locate information.	100%
	Interpret the purpose of selected sections of the protocol.	12%
	Explain how the methods could support or negate the hypothesis.	31.8%
Justify a conclusion with evidence	Locate information.	5%
	Interpret and present evidence.	5%
Graphic analysis	Locate information.	94.4%
	Interpret graphed data.	21.5%

Students did well on those questions where they were asked to locate information in the article. Most students were able to locate the abstract, study objective, hypothesis, methods, results, and conclusions in order to answer questions about them. However, students were much less successful at interpreting, inferring, or evaluating material from the article. The majority of students could not explain the reason for doing the study, provide details of the hypothesis, interpret the purpose of selected sections of the protocol, or explain how the methods would support or negate the hypothesis. Students continued to have difficulty with graphed data. In one question, most of the students (78.5%) could not interpret a graph that reported results that were not statistically significant. Finally, when students were asked for a conclusion, opinion, or interpretation of results, most provided a statement without substantiation or justification from the article.

The results of TIPS II and HWK1 were used to develop the nine science literacy learning objectives (LO) that are found in Table 1. These learning objectives were aligned with Bloom’s cognitive domains and embedded in all future assignments and assessments.

Year 1 – Interim exam assessment of science literacy

Student learning was evaluated throughout Y1 via multiple-choice questions on interim exams. The interim exams contained questions that evaluated both science literacy and lecture content, allowing the instructor to compare literacy learning with lecture content learning. Student performance on the science literacy questions (SCI LIT) was calculated separately from and compared to performance on lecture (LECT) content questions. The mean scores for SCI LIT and LECT questions on these exams are shown in Table 6. A comparison of Y1 scores showed that there was no significant difference between LECT and SCI LIT scores (p > 0.05) on any exam.

TABLE 6

Year 1 interim exam scores.

	EX2	EX3	EX4
LECTa	70.37%	72.71%	66.30%
SD	7.42	9.20	9.22
SCI LITb	69.73%	74.34%	63.22%
SD	13.22	19.12	31.72
n	142	139	130

LECT, mean score on questions associated with lecture portion of the course.

SCI LIT, mean score on questions associated with science literacy assignments and skills.

Year 1 – Summative assessment on final exam

Summative learning for Y1 was evaluated on a final exam composed of two parts: 1) a cumulative multiple-choice exam covering lecture material and 2) a cumulative essay exam of science literacy skills. The cumulative multiple-choice exam contained only questions covering lecture material. The cumulative essay exam of science literacy skills was based upon the peer-reviewed journal article, “Enhanced metabolic responses to caffeine in exercise-trained human subjects” (19). A week before the final exam students were given the article to read and told that they would be tested on the article with questions similar to those on their homework. On the day of the exam the students were provided with a “clean” copy of the article and 11 short-answer questions similar in style and construction to the homework questions. Student performance on the cumulative multiple-choice exam questions (LECT) was calculated and compared to performance on SCI LIT essay questions (Table 7). Results of the final exam suggested that students had made significant gains in their ability to read and understand the scientific literature. The mean score on the SCI LIT essay questions was significantly higher than the mean score on LECT exam questions (p < 0.001, t-test). In addition, the SCI LIT score distribution on the final exam showed that 75.2% of the students received scores ≥ 70% in the SCI LIT essay questions while only 51.5% of the students received scores ≥ 70% in the LECT questions.

TABLE 7

Year 1 and Year 2 final exam scores.

	YEAR 1		YEAR 2
	LECTa	SCI LITb	LECT	SCI LIT
Mean	67.5%c	78.3%	64.5% c	83.5%
SD	12.17	20.12	15.4	20.3
n	117	117	108	108

LECT, mean score on questions associated with lecture portion of the course.

SCI LIT, mean score on questions associated with science literacy assignments and skills.

p < 0.001 LECT vs. SCI LIT.

Analysis of the short-answer questions (Table 8) on the SCI LIT final exam showed that over half of the class demonstrated proficiency in six of the nine science literacy skills. The majority understood the organization of a journal article and were able to use that knowledge to locate information needed to answer exam questions about the object of the experiment, experimental design, controls, the hypothesis, data, and graphs. Final exam results showed that student understanding and application of science literacy concepts improved. When asked to identify the purpose of the experiment, 65% of the students answered correctly by identifying it and supporting their statement with citations of previous studies. In contrast 29.5% answered correctly on a similar question on HWK1. When asked to identify the experimental control and explain why it was needed 62.4% responded correctly on the final exam while 44% responded correctly on a similar question on TIPS II.

TABLE 8

Year 1 student performance on science literacy section of final exam.

Science Literacy Learning Objective	Assessment Question	% Students with Correct Answer (n = 134)
1. Extract information from journal articles using knowledge of organization and structure of scientific literature.	What was the object of this study?	65.8%
2. Understand how science is practiced and how scientific knowledge is generated.	What was the object of this study?	65.8%
	How long did the experiment last (start from the time that any data were collected from subjects)?	28.4%
3. Know and identify the components of the scientific method in journal articles.	What hypothesis was tested in this article?	45.3%
	Where did you find this information?
4. Interpret and explain the basic construction of the experimental protocol; explain its alignment to the hypothesis.	Why did the authors look at the effect of caffeine on free fatty acid mobilization during exercise?	65.0%
	One of the things measured in this study was oxygen consumption. How did the researchers ensure that they could accurately measure the volume of oxygen that the experimental subjects inhaled?	82.1%
	How long did the subjects exercise in these experiments?	49.6%
5. Identify and critique the controls of a study.	Researchers observed oxygen consumption (and CO2 exhaled) for 30 minutes before the experiment. Why did they do this?	62.4%
6. Interpret and explain the results presented in graph or text.	Figure 1 shows how caffeine affected RMR in trained subjects versus sedentary subjects. What did the researchers measure to get RMR?	58.9%
7. Make inferences or conclusions using statistics and hypothesis testing and justify them with data from the article.	Figure 1 shows how caffeine affected RMR in trained subjects versus sedentary subjects. Did caffeine make a difference in the RMR of trained subjects?	67.5%
8. Evaluate or critique the results or conclusion and justify the position.	How did caffeine affect the glucose levels of the subjects in these experiments?	68.4%
9. Build personal knowledge and understanding from journal articles.	What is the “respiratory quotient”?	8.5%

Student ability to interpret experimental design from the methods section of a journal article also improved but performance still varied with the complexity of the question. When asked to interpret the purpose of a selected section of the protocol 68.4% responded correctly by explaining how the methodology influenced the accuracy of measurements (Table 8) while 12% answered correctly on a similar question on HWK1. In a more complex methods question that required students to read carefully to identify a variable in the experiment protocol, 49.6% of the students answered correctly on the final exam.

Students’ quantitative and graphic skills as reflected by their ability to report and interpret data improved but still varied with the complexity of the question or graph. In an exam question asking students to interpret a graph of data that were not significantly different, 67.5% answered correctly (Table 8) while 21.5% answered correctly on a similar question in HWK1. When asked to interpret results of a multi-plot graph, 58.9% of the students answered correctly in contrast to 32% of the students answering correctly on a similar to a question in TIPS II.

Finally, Y1 final exam results showed that reading the scientific literature did not promote building a personal knowledge base in first year biology students. Only 8.5% of the students correctly answered questions that required them to learn and understand a term from the article.

Course organization in Y2 was similar to Y1 except that neither interim nor final exams contained science literacy questions. Science literacy learning objectives developed in Y1 (Table 1) were embedded in all assignments and assessments. Baseline science literacy skills of students in Y2 were defined by an initial homework assignment (HWK1) that consisted of questions that were aligned with the science literacy learning objectives and homologous to those in Y1 assignments (Table 9). Analysis of HWK1 answers showed that less than half of the class answered questions correctly on seven of the nine literacy objectives.

TABLE 9

Year 2 comparison of student performance on HWK 1 and science literacy section of final exam.

Science Literacy Learning Objective	Assessment	Assessment Question	% Students with Correct Answer (n = 108)
1. Extract information from journal articles using knowledge of organization and structure of scientific literature.	Homework 1	What section would you read if you wanted to quickly find out if oxygenated water affected exercise performance or recovery?	48%
	Final exam	What was the purpose of this experiment? Where did you find this information?	48.6%
2. Understand how science is practiced and how scientific knowledge is generated.	Homework 1	In 3–4 sentences describe why the authors did this study. Where did you find this information?	36%
		Are the results of this study in agreement with other studies reported in the scientific literature?Where did you find this information?	15%
	Final exam	What was the purpose of this experiment?	48.6%
3. Know and identify the components of the scientific method in journal articles.	Homework 1	What hypothesis was tested in this article?Where did you find this information?	14%
	Final exam	How did this study control for the possible weight difference between exercising animals and sedentary animals?	61.5%
4. Interpret and explain the basic construction of the experimental protocol; explain its alignment to the hypothesis.	Homework 1	What was the heart rate in stage 1 and stage 3 for subjects who drank super oxygenated water? Explain why these numbers are different.	61.5%
	Final exam	The authors made a change in their experimental protocol at six months into the experiment. How did they change it and why did they do it?	76.1%
5. Identify and critique the controls of a study.	Homework 1	Identify the controls and discuss why they are relevant to the protocol.	14.8%
	Final exam	How did this study control for the possible weight difference between exercising animals and sedentary animals?	61.5%
6. Interpret and explain the results presented in graph or text.	Homework 1	Graph the VO2 values (found in Table 2) for subjects who drank tap water and for subjects who drank super oxygenated water.	25.0%
	Final exam	Which group of animals in Figure 3 has the best chance of living to 36 months?	85.5%
		Which is more effective at increasing lifespan in rats, voluntary exercise or reduced food intake?	86.3%
7. Make inferences or conclusions using statistics and hypothesis testing and justify them with data from the article.	Homework 1	What was the heart rate in stage 1 and stage 3 for subjects who drank super oxygenated water?	38.3%
		How did the authors determine if the oxygen consumption between subjects who consumed tap water and subjects who drank super oxygenated water was different?	22.5%
	Final exam	Which is more effective at increasing lifespan in rats, voluntary exercise or reduced food intake?	86.3%
8. Evaluate or critique the results or conclusion and justify the position.	Homework 1	Could drinking super oxygenated water improve your workouts?	15.0%
	Final exam	Which is more effective at increasing lifespan in rats, voluntary exercise or reduced food intake?	41.3%
9. Build personal knowledge and understanding from journal articles.	Homework 1	Not evaluated on Homework 1.
	Final exam	What is the respiratory quotient?	66.5%

Summative learning was evaluated on the final exam. A week before the final exam students were given the peer-reviewed journal article “Effect of voluntary exercise on longevity of rats” (14) to read and on the day of the exam they were provided with a copy of the article and 12 short-essay questions similar in style and construction to the homework questions. The mean final exam score for Year 2 was 83.5% (±20.3%) and 32.4% of the students scored 90% or higher on the exam. This score was significantly higher than the mean score on final exam questions over lecture material (Table 7) (p < 0.001). Results of the final exam indicated that students improved in their ability to read and understand the scientific literature. Students showed significant learning gains for seven of the nine science literacy objectives (Fig. 1) and more than half of the class achieved competency in six of the learning objectives (Table 9).

FIGURE 1

Comparison of class performance on HWK1 and Final Exam in Y2. Homologous questions on HWK1 and Final Exam were compared. The difference in the percentage of students who answered correctly was calculated for each literacy learning objective (LO) and plotted. *p < 0.05, HWK 1 vs. Final Exam.

A comparison of HWK1 and the final exam showed substantial gains in science literacy. For example, at the beginning of the course most of the students had difficulty interpreting or explaining results presented in a graph or text (LO6). When asked in HWK1 to “Graph the VO2 values found in Table 2 for subjects who drank tap water and for subjects who drank super oxygenated water,” many students mislabeled the graph or misrepresented the data on the graph (75%) and some did not answer the question at all (15.8%). At the end of the course however, 86.3% of the class successfully interpreted graphed data to answer the final exam question, “Which is more effective at increasing lifespan in rats, voluntary exercise or reduced food intake?”

Students also made significant gains in identifying and critiquing the controls of a study (LO5). Results showed that at the beginning of the course some students were able to identify controls (23.2%), but very few could explain their relevance or critique them (14.8%). By the end of the course the number of students who could both identify and critique controls increased by 415% (Table 9).

Results of HWK1 clearly showed that at the beginning of the course few students understood how science is practiced or recognized the importance of previous experimental studies in shaping a hypothesis or experimental protocol (LO2) (Table 9). Most student responses to the HWK1 question “Describe why the authors did this study,” lacked any reference to prior experimental studies. However, at the end of the course the percentage of students who answered the homologous question “What was the purpose of this experiment?” by referring to prior experiments increased significantly (Table 9).

Students’ knowledge of statistics and their understanding of how they were used for hypothesis testing (LO7) were assessed with the HWK1 question, “How did the authors determine if the oxygen consumption between subjects who consumed tap water and subjects who drank super-oxygenated water was different?” Results showed that 77.5% of the students did not refer to statistics or hypothesis testing in their answers. In contrast, 86.3% of the class correctly answered a homologous question “Which is more effective at increasing lifespan in rats, voluntary exercise or reduced food intake?” on the final exam.

Finally, student skills at justifying a statement or conclusion with evidence (LO8) increased significantly during the course. On HWK1 students were asked, “Could drinking super oxygenated water improve your workouts?” and were expected to justify their conclusion with evidence from the journal article. Very few supported their ideas with evidence but by the end of the course there was a 275% increase in this number (Table 9).

DISCUSSION

A survey of the literature reveals that many strategies have been used to teach students to read the scientific literature and some studies have shown that this practice can enhance student learning (6, 7, 9, 10, 15, 17, 20, 23, 25, 31). In this report we attempt to extend these practices by presenting a strategy that uses the literature to teach science literacy to novice students in a way that will serve as a foundation for developing higher-order literacy skills throughout the curriculum.

This manuscript presents a two-year study in which Year 1 served as a preliminary investigation that guided the protocol for Year 2. In the first year of the study learning objectives were identified by the student deficiencies displayed on the TIPS II test and HWK1 assessment. Results from these evaluations were used to develop project learning objectives and a curriculum. Summative results from Y2 showed that students made significant gains in science literacy. Student ability to understand experimental design, interpret results from graphs, and use statistics to make conclusions improved.

Two evaluations were administered in order to develop a comprehensive understanding of students’ baseline science literacy skills. The first evaluation, TIPS II, was an objective assessment that focused on science processing skills. This tool provided information on how students applied their literacy skills in a classroom environment on an objective multiple-choice exam. The second evaluation tool, HWK1, evaluated student literacy skills in a situation similar to that encountered by scientists as they practiced science. This assessment evaluated how the student, acting as an independent learner, practiced literacy skills to read an article, interpret data, write a conclusion, and gain knowledge from the scientific literature.

The goal of this project was to teach students to apply their knowledge and understanding of the science process to research questions and the scientific literature so that they could interpret and learn from them. Results from both assessments showed that while students knew the terms and concepts associated with the scientific process they did not completely understand them or were unable to apply their knowledge in a practical way to evaluate or critique research questions and the scientific literature. To address this issue, a set of nine science literacy learning objectives was developed and used to frame a curriculum that would teach students science literacy. Many of the learning objectives (LO1, 2, 3, 4, 5, 6) were based in Bloom’s lower cognitive skill levels, remember, understand, apply, and analyze. However, several others (LO5, 7, 8, 9) were founded in the higher cognitive levels (2).

Summative results from Y2 showed that student learning gains were different for each set of objectives. Student performance on lower-order literacy skills was high and class learning gains moderate. These learning objectives were based on concepts like controls, simple statistics, graphs, and the scientific method. Many of these subjects are part of pre-college curricula and some students had been exposed to them before. Thus this achievement may have resulted from building upon prior knowledge. In contrast, class performance on higher-order literacy skills of evaluation and synthesis was more modest. While these skills are also part of the pre-college curriculum, they are not practiced to the extent and level of sophistication used in this study or by scientists. This more modest foundation in higher-order science thinking may explain why student learning gains were small for these objectives. Learning gains for the two learning objectives LO1 and LO2 were also modest. These objectives dealt with extracting information from an article and understanding how science is practiced. One of the major problems with these objectives was that students did not understand the purpose of prior studies in shaping an experiment. It is not clear why students were not more successful at this but perhaps more practice and exposure to the literature might help. Finally, this study found that even though many students achieved these competencies, a fraction of the class remained deficient in one or more skills, mostly higher-order literacy skills.

Selection of journal articles was central to the effectiveness of homework assignments. First, most of the articles selected for this study presented the highest standards of science. They were well written, well organized, presented high quality experimental protocols, and easily aligned with literacy goals (26). Occasionally articles with clearly identifiable weaknesses in the scientific method were chosen so that students could understand why this was problematic. Second the relationship between the data and the conclusions in the paper needed to be simple and direct (26) so that the information in the article was accessible to students. Finally, it was important for the articles to have had a practical or social significance that would engage students in a way that would promote true learning.

Studies show that science literacy and science processing skills are fundamental to practicing science (1), and faculty acknowledge that learning these skills is challenging for students (8). The results of this study show that this learning can be facilitated by curricula with explicit objectives and assignments that are carefully aligned with the objectives. Further, we propose that building scientific literacy is an ongoing process which begins in first-year science courses with a set of fundamental learning objectives that serve the progressive development of literacy skills throughout the undergraduate curriculum.