Expert-Novice Comparison Reveals Pedagogical Implications for Students' Analysis of Primary Literature.

Student engagement in the analysis of primary scientific literature increases critical thinking, scientific literacy, data evaluation, and science process skills. However, little is known about the process by which expertise in reading scientific articles develops. For this reason, we decided to compare how faculty experts and student novices engage with a research article. We performed think-aloud interviews of biology faculty and undergraduates as they read through a scientific article. We analyzed these interviews using qualitative methods. We grounded data interpretation in cognitive load theory and the ICAP (interactive, constructive, active, and passive) framework. Our results revealed that faculty have more complex schemas than students and that they reduce cognitive load through two main mechanisms: summarizing and note-taking. Faculty also engage with articles at a higher cognitive level, described as constructive by the ICAP framework, when compared with students. More complex schemas, effectively lowering cognitive load, and deeper engagement with the text may help explain why faculty encounter fewer comprehension difficulties than students in our study. Finally, faculty analyze and evaluate data more often than students when reading the text. Findings include a discussion of successful pedagogical approaches for instructors wishing to enhance undergraduates' comprehension and analysis of research articles.

INTRODUCTION

When students read primary literature, they have opportunities to encounter new material and vocabulary and analyze data and arguments. The process of reading and analyzing scientific articles results in gains in critical thinking and the ability to evaluate data, as well as gains in the understanding of how science is done (Hoskins ; Snow, 2010; Krontiris-Litowitz, 2013; Segura-Totten and Dalman, 2013). Another learning gain associated with reading primary literature is an increase in scientific literacy (Choe and Drennan, 2001; Hoskins , 2011; Gottesman and Hoskins, 2013; Krontiris-Litowitz, 2013; Round and Campbell, 2013; Abdullah ). Scientific literacy is defined as the understanding of the methods that produce scientific knowledge as well as those skills involved in analyzing and interpreting scientific data (Gormally ). Scientific literacy skills have been touted as necessary for the success of undergraduates in the sciences (American Association for the Advancement of Science [AAAS], 2011), and there is agreement that the development of these skills should be a priority in undergraduate biology education (National Research Council [NRC], 2003).

Given these results and calls for greater integration of primary literature reading in the undergraduate biology curriculum, faculty and researchers have developed and studied strategies for helping students learn to read research articles (e.g., Janick-Buckner, 1997; Kozeracki ; Hoskins ; Krontiris-Litowitz, 2013; Round and Campbell, 2013; Segura-Totten and Dalman, 2013; Sato ; Marsh ; Shorbagi and Ashok, 2016). Approaches to reading primary literature range widely. For instance, some approaches involve a high level of scaffolding that includes supplying a glossary, explicitly describing methods, and providing tips on how to read a research article (AAAS, 2017; Wenk and Tronsky, 2011). Other approaches walk students through the scientific process (e.g., Janick-Buckner, 1997; Hoskins ; Sato ). Most approaches include components that prompt students to analyze and critique data and involve giving particular assignments to help students prepare for discussion about an article (Janick-Buckner, 1997; Hoskins ; Krontiris-Litowitz, 2013; Round and Campbell, 2013; Segura-Totten and Dalman, 2013). In our experience, preassignments are important for students to fully engage with the reading (Segura-Totten and Dalman, 2013).

Research on how high school, college, and master’s students read research articles has contributed to the understanding of how students view scientific texts. For example, a comparison of the sections of articles that faculty, senior and first-year college students highlight found that faculty had a high level of agreement on important sections of the text, while this agreement decreased for college seniors and even more for college freshmen (Gallo and Rinaldo, 2012). This suggests that, as scientists progress in their careers, they are better able to distinguish important information in scientific articles. Marsh and colleagues (2015) also looked at student markings on articles and found that upper-level students, who had a higher level of comprehension of the text, highlighted papers more often and highlighted more sections of papers than introductory-level students. A survey of scientists at different career stages may help explain why novices and experts mark articles differently—Hubbard and Dunbar (2017) found that undergraduates had difficulty with the methods and results sections of primary literature. Further, undergraduates placed less value on methods and experimental results and their interpretation than individuals at later career stages (Hubbard and Dunbar, 2017). Similarly, when master’s students were asked at the beginning of a course about their perceived difficulties with reading primary literature, they reported techniques and experimental data as challenges (Lie ).

Research on the analysis of scientific texts has also shed light on factors that aid in student understanding. Several studies have found that college students’ ability to correctly evaluate the trustworthiness of a scientific source correlates with their comprehension of a topic (Bråten ; Wiley ). Self-explanation of concepts, monitoring comprehension of the text, and annotations of the text are also correlated with high learning gains in college and high school students (reviewed in Greenleaf ; Goldman ). The format of an article can also have an effect on comprehension: high school students who read texts adapted to facilitate student understanding had a better comprehension of an article, while those who read the versions that most closely resemble primary literature thought more critically about the topic (Norris ).

Expert–novice comparisons can provide useful insights into how to aid novices in developing more expert-like behaviors and thinking (NRC, 2012). For instance, research on expertise has shown that experts are different from novices in the following ways. One main difference between experts and novices is the vast domain knowledge of experts. Experts can organize the large amount of domain knowledge they possess to create patterns that facilitate skill application (Newell and Simon, 1972; Glaser and Chi, 2014; Feltovich ). How experts and novices approach problem-solving also differs: while novices will typically rush into solving a problem, experts first create mental models about the problem before attempting to solve it (Glaser and Chi, 2014; Feltovich ). Experts also consider a problem at a deeper level than novices. For example, studies of how experts and novices approach ill-structured problems in social science suggest that experts use their larger schemas involving the topic of the problem to decompose it into major factors that can then be addressed to find a solution (Voss and Post, 2014). Experts also have stronger monitoring skills than novices, which allows them to determine more often when they make an error (Chi ; reviewed in Glaser and Chi, 2014; Feltovich ). Experts’ monitoring skills are tied to their greater domain knowledge (Simon and Simon, 1978; Chi, 1987; Glaser and Chi, 2014).

Many of the differences in how novices and experts approach tasks are linked to how individuals organize and retrieve information from memory. For this reason, research on cognitive load theory is also useful for thinking about why experts and novices might differ in how they go about a cognitively demanding task like reading primary literature. Complex readings like primary research articles have a high level of difficulty, or intrinsic cognitive load, which places a high demand on a reader’s short-term memory. This is especially important to consider because individuals can hold a limited amount of information in their short-term memory (Miller, 1956; Sweller ). However, new information can be organized into schemas and stored in long-term memory, thus circumventing the limit on short-term memory (Paas ; Sweller ). Items that are stored in long-term memory can be recalled during a complex task without increasing cognitive load (Sweller ). Thus, experts’ larger access to schemas stored in long-term memory frees up space in short-term memory (Chase and Ericsson, 1982; Glaser and Chi, 2014). Coupled with experts’ more complex schemas is the fact that they have more practice in domain-specific skills. These two factors allow experts to retrieve information from memory faster than novices (Posner, 2014). Another way in which cognitive load can be managed is by bypassing the short-term memory limit through the chunking of bits of information into larger units, a process termed “recoding” (Miller, 1956; Paas ). Studies contrasting master chess players with experts and novices found that master players were better able to remember domain-specific knowledge through recoding of chessboard configurations (Chase and Simon, 1973; reviewed in Posner, 2014; Feltovich ). Another insight from studies of experts in diverse fields like chess, music, sports, and science is the importance of building expert-level skills through years of practice and study (reviewed in Posner, 2014; Sweller ).

Although the level of intrinsic cognitive load of a text cannot be modified, instructors can find ways of presenting the material in an easily digestible way, thus lowering the extrinsic cognitive load associated with the reading (Sweller ). According to cognitive load theory, if instructors convey material in a way that lowers the extrinsic cognitive load, learners can dedicate more of their short-term memory to the creation of new schemas (Sweller ). However, as learners progress, their schemas become more complex, and supports designed for novices can actually be counterproductive to the learning of more advanced students (Kalyuga , 2012).

The evidence on expertise suggests that novices develop domain-specific skills over time through practice and changes in cognition. For this reason, it is important to describe the cognitive engagement of students in domain-specific tasks as they progress toward expertise. The ICAP (interactive, constructive, active, and passive) framework is particularly suited for this because it defines the levels of student cognitive engagement during tasks through specific observable behaviors (Chi, 2009; Chi and Wylie, 2014). The ICAP hypothesis divides active learning into four hierarchical modes: 1) passive, or receiving; 2) active, or manipulating; 3) constructive, or generating; and 4) interactive, or dialoguing (Chi and Wylie, 2014). For example, while reading a research article, a student might show the following levels of engagement: 1) passive, if he or she reads text passages without interacting with the reading in any other way; 2) active, if he or she performs mechanical actions like underlining seemingly important sections of the text; 3) constructive, if he or she generates novel outputs like notes in his or her own words that synthesize sections of text; and 4) interactive, if he or she debates the findings of an article with other students. According to ICAP, greater engagement during a certain task leads to higher learning and more successful integration of new information within existing schemas (Chi and Menekse, 2015; Wiggins ; reviewed in Chi and Wylie, 2014).

In this study, we sought to explore, compare, and contrast how faculty members and undergraduate students engage with a scientific article. We accomplished this by conducting think-aloud interviews of six biology faculty and 11 undergraduate students while they read a research article (Kuusela and Paul, 2000; Meijer ). We analyzed the interview data inductively using standard qualitative content analysis procedures. We then grounded the interpretation of our data in cognitive load theory and the ICAP framework. We found that faculty had more complex schemas than students and that they reduced cognitive load through two main mechanisms: summarizing and note-taking. Faculty also engaged with the article at a higher cognitive level, described as constructive by the ICAP hypothesis, when compared with students. Finally, faculty used certain dimensions of scientific literacy skills more often than students when reading the text.

METHODS

Because our study contains 17 participants (11 students and six faculty), it is considered a small-N study. As noted by Gouvea (2017), insights from small-N studies can provide an in-depth look into how students learn science. Here, we describe our research methods, context, and participant demographics in detail to assist readers with determining how the study results apply to readers’ context.

Context

This study was conducted within the biology department at a 4-year, master’s-granting university situated in the Appalachian region of the southeastern United States. The student population is ∼20,000 and spans five campuses. The research conducted in this study occurred on one campus, which is the only residential undergraduate campus. Approval to conduct this study (expedited status, application 2014116) was granted by the University of North Georgia Institutional Research Board.

Participants

Participants were 11 students who were taking courses in the biology department and six faculty members in the biology department. To identify student participants, students who were enrolled in biology department coursework were emailed about the research study and invited to participate. Participation in the study was not connected to any coursework and was completely voluntary. Students who volunteered were provided with additional study information and the informed consent form. To identify faculty participants, M.S.-T. extended an open call to her colleagues in person in the biology department. They were presented with additional study information and the informed consent form.

Student participants completed a demographic survey (Supplemental Appendix 1) that also included questions about their experiences with reading scientific articles. Because we determined that comparing student novices among themselves was beyond the scope of this study, we did not use this portion of the survey. The ethnic composition of the convenience sample included one Asian, one Latinx, and 15 Caucasians. Eight of the participants were female, eight were male, and one participant did not identify a gender. Of the 11 students, six were upper-division students and five were lower-division students. Nine of the students majored in biology, and two were business majors, one who was pursuing a biology minor and one who had taken the introductory biology course sequence. The average of the self-reported grade point average (GPA) was 3.09, and all participants reported GPAs that reflected good academic standing. The demographics of the student participants were similar to the demographics of biology department student population (departmental student demographics: 56% female, 44% male; ∼10% Latinx), except for African-American students (none in the study vs. 4.6% in the department).

Of the six faculty participants, one held a master’s degree and five held PhDs. One participant had been a faculty member for 1 year, and the rest had 4–19 years of experience as faculty members. None of the participants conducted research directly related to the field of the chosen article. Five had expertise in molecular biology, and the sixth was an ecologist. All faculty participants had expertise in research involving animals, and two had experience in behavioral studies with mice, which the study involved. The individual who had been faculty for only 1 year had extensive experience with behavioral, molecular, and neurobiology research using rodents, so she could be considered an expert in the general area of the article chosen.

Think-Aloud Interviews

We wanted to choose a primary research article that would not be too complicated for first- and second-year student participants so as not to impede their thought processes. To select the article, we polled faculty in the biology department for examples of scientific articles that introductory students had read in class. From the articles nominated by faculty, we selected “Fatal Attraction in Rats infected with Toxoplasma gondii” (Berdoy ), because it was relatively straightforward and contained only two figures, reducing the time commitment for participants. See Supplemental Appendix 2 for the version of the article with line numbers that we used in the study.

We conducted think-aloud interviews with participants as they read the paper. Think-aloud interviews are a metacognitive exercise deriving from the information-processing tradition (Newell and Simon, 1972; Ericsson and Simon, 1993) in which participants perform a task and, while doing so, verbalize their internal thoughts (Boren and Ramey, 2000; Kuusela and Paul, 2000). The purpose of the think-aloud protocol is to ascertain an individual’s cognitive processes and provide insights into participants’ reasoning, self-talk, and feelings.

We asked participants to verbalize their thoughts as they read, and we reminded them to think aloud if they forgot to do so. As they read, we prompted participants with discussion questions (see Supplemental Appendix 3) that we designed to enable us to observe the participants’ ability to evaluate, analyze, and interpret data and to comprehend the material in the text. Approximately 80% of the codes came from the reading of the text, while ∼20% of the codes came from responses to the questions. The interviews lasted about an hour on average and were recorded for transcription.

Analysis

The data consisted of transcripts of more than 880 minutes of the audio portion of video recordings, which were analyzed using the constant comparative method (Glasser and Strauss, 1967). We employed a purely inductive coding process because we envisioned that pursuing this study in an exploratory manner could yield novel insights to the body of research on the analysis of primary literature. To initially form codes, we individually read the transcripts in small batches to categorize the text. After forming the initial list of codes, we met monthly to discuss our work and what we understood each code to mean. From the discussions, categories, or themes, emerged that were used to sort the initial codes. Using the collaboratively developed code list, we related categories of information to one another and to the problem being investigated, and finally, we analyzed the transcripts and revised the categories and codes until we were able to produce no new codes or categories. To reach a consensus as to what code best described the data being coded, we tried to avoid instances of double coding to establish consensus validation, but at times it was unavoidable when both codes applied. While we used NVivo (QSR International) software for its organizational components and ease of determining code frequency, we coded manually within the software. Through the analysis process, we identified three categories, which were further refined as themes, and 27 codes that captured all of the strategies being used or being demonstrated by participants (the most prevalent codes are found in Table 1, and a list of all codes is shown in Supplemental Appendix 4).

TABLE 1.

Themes encountered during qualitative analysisa

Subtheme (+/- indicates correct/incorrect)	Working definition	Example
Theme 1: Thinking tools
Rereading text one or more times“Rereading”	The participant commented that s/he reread a portion of the text.	I’m gonna go back to the last sentence.—Student
Summarizing or recapping“Summarizing”	The participant summarized a portion of the text.	So cats, they can reproduce out of cats, but they can infect all mammals. So everybody gets sick, but only cats can allow them to complete the life cycle.—Faculty
Using a reference point/prior knowledge“Prior knowledge”	The participant exhibited prior knowledge or used a reference point in the text while thinking aloud.	The wild animal, um, it’s really hard to collect and have any kind of consistency with wild animals because they come from so many different unknown social backgrounds. And when you’re studying behavior, that’s a really important thing to consider.—Faculty
Underlining a key piece of information“Underlining”	The participant underlined a portion of the text.	And whenever I’m reading papers I like to underline like the summary sentences.—Student
Taking notes	The participant wrote down notes.	So, I’m gonna write on the side, uhh, let me see, parasites … found … are transmitted through food … transmitted … through … food … exhibit … uhh, manipulation hypotheses.—Student
Relying on definition of term provided in article“Relying on definition provided”	In the event that a term was described in the text, a participant indicated that he or she either understood it or noticed it.	Oh, so that’s what they mean by laboratory–wild hybrids.—Student

aThe themes arising from qualitative analysis are shown in bold. The most prevalent subthemes encountered during the analysis are arranged below the themes. Shortened versions of the subtheme names used in the text are shown in quotation marks.

Although our goal with this study was to describe and qualitatively compare how faculty as experts and undergraduate students as novices read primary literature, we thought it would be useful to calculate how often particular strategies were used by each group. Thus, we used the number of times each code was observed to calculate the average usage of a code by group and the percentage of participants who used a specific subtheme (Table 2).

TABLE 2.

Think-aloud theme frequenciesa

	Faculty			Students
Themes/subthemes	No. out of 6	Percent	Average instance ± SEM	No. out of 11	Percent	Average instance ± SEM
Thinking tools
Rereading text one or more times	6	100	33 ± 8.7	11	100	10 ± 3.1
Summarizing or recapping	6	100	14.7 ± 4.8	11	100	4.6 ± 1.1
Using a reference point/prior knowledge	6	100	7.5 ± 1.9	6	55	2 ± 0.86
Underlining a key piece of information	4	67	8 ± 3.4	7	64	6.9 ± 3.2
Taking notes	4	67	5.2 ± 2.7	5	45	2 ± 0.89
Relying on definition provided	0	0	0	5	45	0.45 ± 0.16
Science literacy and process skills
Understanding research design +	6	100	10.9 ± 2.1	11	100	3.1 ± 0.73
Evaluating a scientific argument	6	100	9.2 ± 2.1	2	18	0.55 ± 0.37
Analysis +	6	100	13.6 ± 0.71	8	73	5.6 ± 0.97
Understanding research design −	3	50	0.50 ± 0.22	9	82	1.7 ± 0.38
Participant comprehension difficulties
Due to unknown vocabulary/jargon	4	67	1.50 ± 0.73	8	72	3.40 ± 1.2
Due to lack of knowledge/incorrect knowledge	4	67	0.67 ± 0.21	6	55	0.73 ± 0.24
Participant becomes distracted focusing on a small detail	2	33	0.33 ± 0.21	2	19	0.27 ± 0.19
Due to wording/sentence structure	1	17	0.17 ± 0.17	6	55	0.73 ± 0.31

aSubthemes are listed below themes in order of prevalence for faculty. The number and percentage of participants who demonstrated a subtheme as well as average instance are shown. N = 6 (faculty); N = 11 (students); SEM = standard error of the mean; + means that it was done correctly, and - means that it was done incorrectly.

RESULTS

Our analysis revealed three themes in how experts and novices read primary literature: thinking tools, scientific literacy and process skills, and comprehension difficulties (Table 1). Here, we present the themes and associated subthemes that were most prevalent in our data set (a complete list of subthemes can be found in Supplemental Appendix 4).

Key Theme 1—Thinking Tools

Students and faculty employed a variety of thinking tools to understand what they read, including rereading the text one or more times, summarizing and underlining sections of text, activating their prior knowledge, and taking notes. Thinking tools made up the majority of the codes we encountered for faculty and students. Rereading was the tool most often employed, and all faculty and student participants reread text during the interview (Table 2). However, faculty reread the text on average three times as often as students (average instance in Table 2). When rereading, participants paused after reading a section or sentence, went back to the beginning, and read the section or sentence again. All participants also summarized portions of the text during the interview, but faculty did it on average three times more often than students (Table 2). For an instance to be coded as summarizing, a participant had to paraphrase aloud what the authors wrote in the text. Often, participants summarized several sentences into a statement. For instance, one faculty participant summarized part of the experimental setup for the article, highlighting seemingly important areas, “Okay, they did it with a video camera, they had, they were illuminated and the rats had been habituated, that’s really important. Alright, so they had habituated the rats to the, but they don’t show any of that data.”

A second faculty participant summarized the life cycle of T. gondii after reading the introduction of the article:

But it looks like while they can infect everybody, the life cycle can’t be completed by everybody. Cats, for this species, are what are needed. . . . So essentially, the progeny have to come out of cats. They won’t come out of anybody else despite infections occurring in other animals. . . . So cats, they can reproduce out of cats, but they can infect all mammals. So everybody gets sick, but only cats can allow them to complete the life cycle.

All faculty activated prior knowledge when reading through the article, compared with 55% of students (Table 2). On average, faculty also activated prior knowledge more often than students (Table 2). As exemplified below, some of the prior knowledge of faculty participants pertained to their experience as scientists:

The wild animal, um, it’s really hard to collect and have any kind of consistency with wild animals because they come from so many different unknown social backgrounds. And when you’re studying behavior, that’s a really important thing to consider.

On the other hand, students’ prior knowledge consisted mostly of facts, some of which they commented on having learned in biology courses, like “Well, different things can be parasitic, it just means that it takes nutrients from another.” Students also relied on the definition provided for terms in the article (45% of students; Table 2), while faculty did not use this tool while reading.

Similar numbers of faculty and student participants underlined parts of the text (67% of faculty and 64% of students; Table 2). Participants used this tool for different reasons, such as to remind themselves of important information or to note words or phrases that were unfamiliar. At other times, they underlined sections of the text without indicating a specific reason. On the other hand, a larger percent of faculty took notes when compared with students (67 vs. 45%, respectively; Table 2). In addition, there were noteworthy differences between faculty and student notes. Faculty mostly wrote about their evaluation or analysis of a passage, as shown in these examples from two individuals:

So they’re saying, at least in this figure legend [points at figure legend with finger on page], they’re making the same assumption that cat urine equals high predation risk [writes “cat urine = high predation risk” below figure legend].

Uh, um, also making a note so the next question I’m supposed to answer at the end of the results is predict how the results would change if the author had used lab rats instead of wild hybrids and they address that in the methods saying that they make this hybrid so they show these innate behavior[s] or they’re arguing at least they’re showing these innate behavior responses.

In the first example above, the faculty participant is analyzing the data in the graph and the figure legend text to make a connection between avoidance of the cat urine smell and the y-axis of the figure, which is shown as cumulative preference. This analysis leads the participant to determine that cat urine represents high predation risk. In the second example, the faculty participant sounds skeptical of the authors’ claim that the hybrid rats show innate behavior, which suggests that the participant is evaluating this claim before accepting it.

In contrast, most of students’ notes were written as reminders of what the text stated, such as, “So, I’m gonna write on the side, uhh, let me see, parasites … found … are transmitted through food … transmitted … through … food … exhibit … uhh, manipulation hypotheses.”

Because of the qualitative differences we observed between faculty and student tool usage, we further classified thinking tools into active or constructive according to the ICAP framework (Chi and Wylie, 2014). Passive engagement with the text involves reading without taking other actions, while interactive engagement involves dialogue between two or more individuals, so our codes reflect only active or constructive behaviors. Rereading, underlining, and relying on the definition provided for a term are active behaviors, because the reader is interacting with the text but is not producing new information or inferences as a result. Summarizing, using prior knowledge, and taking notes can be used actively or constructively, depending on whether the participant derives new information or inferences while using the tool. For faculty, 26% of instances of activating prior knowledge were active, and 74% were constructive, with faculty either explaining what the text meant to them using words not mentioned in the text, or generating new knowledge (Chi and Wylie, 2014; Table 3). On the other hand, 38% of instances of prior knowledge activation in students were active and 62% were constructive (Table 3). Seventy-five percent of notes taken by faculty were constructive and 25% were active, while 21% of student notes were constructive and 79% were active (Table 3). Constructive notes contained new ideas, the synthesis of several ideas in the article, or the evaluation of information presented, while active notes were verbatim from the text. Finally, 73% of the time faculty generated summaries that were constructive, containing new inferences and explanations of the material (also called “self-explaining”; Chi and Wylie, 2014; Table 3). Students generated constructive summaries 60% of the time, while 40% were “copy-and-delete” summaries, which are verbatim summations of the text (Chi and Wylie, 2014). Overall, for activation of prior knowledge, note-taking, and summarizing, 74% of faculty instances were constructive and 26% were active compared with 52% constructive and 48% active for students.

TABLE 3.

Frequency of active and constructive usage of thinking toolsa

	Faculty		Students
Tool	Constructive	Active	Constructive	Active
Prior knowledge	74	26	62	38
Note taking	75	25	21	79
Summarizing	73	27	60	40

aEach instance of tool usage was classified as either active or constructive, as defined by Chi and Wylie (2014). The percent of the total number of times a tool was used is shown.

In summary, faculty and students used a variety of thinking tools as they read, but they used these tools differently. Faculty summarized portions of the text, activated their prior knowledge, and took notes more often than students. Furthermore, faculty notes often included the results of analysis and evaluation of information, while student notes reflected material taken directly from the text. Finally, faculty used thinking tools in a constructive manner more often when compared with students.

Key Theme 2—Science Literacy and Process Skills

Faculty and students displayed science literacy and process skills as they read, including analyzing information, evaluating arguments, and making sense of the research design.

All faculty and students displayed an understanding of the research design at some point while reading the article (Table 2). However, on average, faculty displayed this ability almost four times as often as students (average instance in Table 2). This faculty participant demonstrates understanding of how the study was set up. In addition, the passage illustrates how the participant incorporated the research design described in the article into his existing schema of factorial design and quantification of rat behavior:

I thought, we’ll get to the t test later then won’t we to compare the two corners. So they did it the way I would have done it, which is a factorial design. [Tilts head to read Figure 1.] And you’ve got infected versus noninfected and you’ve got the four corners. And it is a repeated measure in that case. [Nods head.] Sure, because each rat is going to invest[igate], could potentially go into all four corners and if they don’t go into a corner, they get, they just get a zero.

Students were unable to understand elements of the research design more often than faculty (82% of students and 50% of faculty; Table 2). For instance, this student demonstrated lack of understanding of experimental design in response to the prompt “Predict how the results would change if the authors had used laboratory rats instead of laboratory–wild rat hybrids.” Specifically, the student failed to put the selection of laboratory rats in the context of the rest of the experimental setup described:

So they probably would respond a lot differently and most likely die from the parasite. Um, secondly you wouldn’t really be able to compare them to an environmental, like factor, I guess like an environmental situation if they grew up all their life not living in the wild. Um, so, they might, they would just all, like the parasite would probably affect them a lot differently.

All faculty evaluated a scientific argument while reading the article, compared with only 18% of students (Table 2). Moreover, on average, faculty demonstrated the skill of evaluation 17 times more often than students (Table 2). For example, in response to the question “Given the information presented in the Introduction, do you think that T. gondii will interfere with the rat’s innate reaction to potential predation risk by cats?,” this faculty participant is skeptical of the authors’ claims given the data they presented:

No. Uh, and, and to explain my answer, there was not enough concrete behavioral evidence to support it. They make statements about studies without really providing any of the evidence that is in those papers. So I don’t have enough to go on to actually make that call. In fact I’m a little, little bit, I’m a little suspicious of the whole, of the whole thing. I think that was obvious when I was talking about the lab rats that they used.

Faculty and students engaged in three different types of analysis that are encompassed by the analysis subtheme: analyzing the relationship between different pieces of information, the analysis of graphical data, and the interpretation of statistical analyses (Table 1). All faculty and 83% of students engaged in analysis while reading the article (Table 2). On average, faculty correctly analyzed information twice as often as students (average instance in Table 2). This student correctly interpreted a figure in response to one of the questions embedded in the article:

Okay, let’s see there is a significant difference with the noninfected rats and the infected rats when it came to the smell of the cat. And it looks like that was their activity. So the infected rats had a higher activity level and seemed to go out more where the smell was than the uninfected who probably stayed in their little pen things. And let’s see the other ones were pretty close to each other, so it doesn’t look like a big difference.

In summary, faculty used scientific literacy skills more often than students when analyzing the article. Specifically, faculty were more capable of making sense of the study design, they more frequently evaluated the claims presented, and they analyzed information more often than students.

Key Theme 3—Comprehension Difficulties

Comprehension difficulties arose for students and faculty as they read through the research article. These included jargon, lack of knowledge, distraction, and sentence structure. Not surprisingly, faculty had a lower level of comprehension difficulties (2% of the total code instances), while students voiced five times more comprehension difficulties than faculty (10% of the total code instances).

In this study, we defined “jargon” as a technical word or phrase that is unknown to the participant. If participants noted that they did not know a particular word and did not look the word up or otherwise try to understand its meaning, that occurrence was placed in this subtheme. For both faculty and students, jargon comprised the largest subtheme within comprehension difficulties, with 67% of faculty and 72% of students encountering jargon at least once while reading the article (Table 2). Additionally, students noted coming across jargon twice as often as faculty on average (average instance in Table 2). Students frequently made statements such as “never heard that word before” or “I don’t know that word either” while reading and thinking aloud.

The second most prevalent reason why faculty had trouble comprehending the text was due to lack of knowledge (67%; Table 2). Although the percentage of students who reported lack of knowledge was lower than that of faculty (55%; Table 2), both groups voiced lack of knowledge at a similar rate (average of 0.67 ± 0.21 for faculty and 0.73 ± 0.24 for students; Table 2). In this usage, participants expressed that they did not know something, or they speculated about the meaning of it. Lack of knowledge is more extensive than just the meaning of a word (i.e., jargon). For example, in the following passage, the faculty participant does not know how T. gondii is transmitted to humans:

Which also, I don’t understand why the parasite would be in humans, even though it said that it affected all mammals, but if the parasite had to lay its oocysts … that does—I don’t understand because there’s no predator of humans so I don’t know, unless humans contracted them from meat that they were eating. That would be the only thing that would make sense.

The third most prevalent reason for faculty having difficulty in comprehending the text was distraction caused by focusing on a detail in the text; this affected 33% of faculty compared with 18% of students (Table 2). A distraction would manifest in a manner that would prevent a participant from progressing in the reading, ultimately delaying their reading comprehension. For example, this faculty participant was hindered by the phrase “The rising line for uninfected cats”:

[Participant rereads] “The rising line for uninfected rats.” [Pauses, continues paraphrasing] indicates a prolonged avoidance of cat-scented areas. [Continues reading, starting with the legend of Figure 2.]

[Participant finishes line 17 of Figure 2 legend and draws line adjacent to x-axis, separating third and fourth sortie.]

[Participant rereads] “The rising line for uninfected rats.” [Circles “The rising line for uninfected rats” on Figure 2 legend.] “The rising line of uninfected rats.” Something is wrong there, either with me or with this paper. “The rising line for uninfected rats.” Okay, I don’t think that can be right.

The least prevalent reason for faculty not comprehending the text was “Due to wording/sentence structure” (17% of faculty; Table 2). In contrast, 55% of students did not understand the text because of wording and sentence structure (Table 2). Although published papers are written by experts and reviewed by editors before publication, a reader may still have difficulty with the wording or the sentence structure of the text. When this occurs, it could delay or even prevent comprehension. For example, this student expresses frustration with the dense construction of a sentence: “[Rereads sentence that starts in line 3] It’s like, a lot of words, haha, that I don’t, I don’t know, don’t feel like they go together.”

DISCUSSION

The inclusion of primary literature analysis in the classroom has grown in significance over the past two decades, as evidenced by the number of published approaches to reading the literature (e.g., Janick-Buckner, 1997; Kozeracki ; Hoskins ; Krontiris-Litowitz, 2013; Round and Campbell, 2013; Segura-Totten and Dalman, 2013; Sato ; Marsh ; Shorbagi and Ashok, 2016). To better understand how expertise in reading primary literature develops, we compared how faculty and students went about reading a research article during think-aloud interviews. Our study reveals processes and techniques that biology undergraduates and faculty employ as they read through primary literature. Our research also shows that faculty and students engage with research articles dissimilarly, suggesting areas of student development that could improve their analysis of scientific papers.

Interpreting How Experts and Novices Read Primary Literature Using Cognitive Load Theory and the ICAP Framework

Cognitive Load Theory.

Perhaps not unexpectedly, our study revealed that faculty experts have fewer comprehension difficulties than student novices when they read a research article. Three reasons grounded in cognitive load theory may help explain why faculty demonstrated fewer comprehension difficulties: 1) experts have more complex and extensive schemas related to scientific techniques, knowledge, and process (as evidenced by the higher number of instances of faculty comments and their complexity within the subthemes of understanding of research design and prior knowledge), 2) experts encounter unfamiliar technical words less often than students (perhaps because of their more extensive schemas), and 3) experts reduce their cognitive load by recoding information.

Faculty experts’ more complex schemas for technical terms, techniques, and scientific process allow them to retrieve prior knowledge relevant to the reading more often than students, thus placing fewer demands on short-term memory. This may lead experts to have a higher cognitive-processing capacity (as has been shown for problem-solving in mathematics; Sweller, 1988) and, thus, a deeper understanding of the textual material. Faculty may also use their complex schemas to make more connections between the information presented in the article and their prior knowledge, thus facilitating analysis and evaluation of data.

To reduce cognitive load while reading the article, faculty used recoding through summarizing and note-taking. Faculty summarized passages from the text and took notes on average about three times as often as students (Table 2). Recoding by summarizing has also been described as self-explanation of material when it contains new information generated by the individual (Chi ; Chi and Wylie, 2014). This finding is particularly interesting to us, because self-explanation improves students’ understanding of a text as well as their course performance, particularly for students with low levels of initial knowledge (Chi ; McNamara, 2017). Also, summaries that are more complete and connect the material to prior knowledge are linked with better student performance (Bednall and James Kehoe, 2011). Because faculty notes in many cases contained the analysis or evaluation of a passage, these notes could also be considered the simplifying, or recoding, of a large amount of information. On the other hand, student notes were often reminders of what the text stated without significant analysis or evaluation. Thus, while it seems that students were still attempting to reduce cognitive load through note-taking, they did not do it as frequently as faculty, and they did not take notes of recoded information. Interestingly, a study of high school graduates who were asked to take notes while reading a text showed that participants who took notes that summarized the text and those who took more notes better comprehended the text when compared with students who took fewer notes and to those whose notes contained verbatim information from the text (Slotte and Lonka, 1999).

On the basis of our findings, we recommend reducing the extrinsic cognitive load of a research article to facilitate student understanding. One way to do this is by including resources to supplement student prior knowledge. For example, the use of a glossary could help to reduce student issues related to jargon. Several published approaches for the analysis of the primary literature feature ways of supplementing student knowledge on techniques (Janick-Buckner, 1997; Jacques-Fricke ; Segura-Totten and Dalman, 2013), and one of these resulted in students who were better able to select the correct technique to answer a particular scientific question, an important component of research design (Jacques-Fricke ). The annotated research articles created by the American Association for the Advancement of Science (AAAS, 2017; McCartney ) and the approach described in Abdullah both target student issues with unknown terms and techniques and may serve as effective platforms for literature discussions. Additionally, the process of annotating leads to a more readable version of the research article, which in turn may lead to better student understanding (McCartney ).

Instructors could also decrease the extrinsic cognitive load of a research article through focused instruction. Master’s students who were asked about the perceived challenges associated with reading research articles reported issues associated with jargon and lack of knowledge of scientific methods (Abdullah ). After focused instruction on reading scientific articles, including discussion of terms and methods found in the texts, the less experienced student participants (who may align better with the undergraduates in our study) reported fewer issues with scientific terms and information in scientific articles (Lie ). They also reported fewer problems understanding the writing style of scientific articles, an issue that surfaced for students in our study (Table 2, “Participant Comprehension Difficulties: Due to Wording/Sentence Structure”). A third way to lower cognitive load is by having students summarize and take notes as they read an article. We will discuss these two techniques in the context of the ICAP hypothesis in the next section.

ICAP Hypothesis.

The ICAP framework describes how an individual’s knowledge is changed by different types of engagement during a cognitive task: 1) new information is stored in isolation (passive engagement); 2) new information leads to the activation of prior knowledge and is integrated into existing schemas (active); and 3) after integration, additional new knowledge is inferred based on the activated and integrated knowledge (constructive). We will not address the interactive mode of ICAP, because it requires constructive dialogue between two or more individuals, something that was not captured in our think-aloud interviews. ICAP predicts that active engagement during a task allows students to fill in gaps in their schemas so they may more easily retrieve this information when they encounter similar cognitive tasks (Chi and Wylie, 2014). Constructive engagement during a task leads to deeper learning than active engagement, because the knowledge obtained can be transferred and applied in different contexts (Chi and Wylie, 2014). Analysis of published studies and the results of studies designed to test the ICAP framework show that a higher level of engagement leads to higher learning gains and student understanding (Chi and Wylie, 2014; Chi and Menekse, 2015; Wiggins ). In our study, faculty engaged with the article at a constructive level more often than students (Table 3), better inferred new knowledge, and created outputs more often (e.g., rich notes that include data analysis and evaluation). Thus, the higher level of cognitive engagement of faculty while reading the research article may help explain why this group encountered fewer comprehension difficulties than students in our study.

We hypothesize that instructors can enhance student comprehension of scientific articles by encouraging them to engage with the material in constructive ways. For example, we predict that prompting students to summarize information constructively while they read an article will lead to better comprehension of the text. Summarizing allows readers to connect different pieces of the text to form a cohesive idea, rather than evaluating separate pieces of information in isolation (Dunlosky ). This, in turn, may be helpful in distilling the big picture of a scientific study. We recommend that instructors train students in how to summarize information and support them in doing so, for example, by showing them how to draft summaries, creating assignments that involve students in practicing this skill or by using approaches to reading primary literature that engage students in summarizing (e.g., Round and Campbell, 2013; Sato ). Another way for students to engage constructively while reading a research article is by creating notes that capture data analysis and evaluation. Because the cognitive engagement of faculty and students differed the most during note-taking (Table 3), we foresee that this will be an area where students will require a lot of support and practice. Two ways of teaching students how to take richer notes are to use an approach to reading primary literature that contains a note-taking component (Hoskins ; Round and Campbell, 2013) and to model expert behavior in class.

Development of Expertise in Scientific Literacy

In our study, faculty experts analyzed and evaluated scientific information more often than student novices. It is very possible that experts in our study acquired these skills through years of reading scientific articles. Previous studies show that the analysis of research articles increases dimensions of scientific literacy and knowledge of the science process in undergraduate and graduate students (Choe and Drennan, 2001; Hoskins , 2011; Gottesman and Hoskins, 2013; Krontiris-Litowitz, 2013; Round and Campbell, 2013; Abdullah ). In fact, thousands of hours of practice are required to attain expertise in a certain domain (Chase and Simon, 1973; reviewed in Posner, 2014; Ericsson, 2018). Alternatively, students’ epistemological beliefs on the nature of scientific knowledge may lead them to view information in scientific articles as immutable facts (Schommer, 1990). This may in turn prevent them from attempting to analyze or evaluate the text. It would be interesting to determine whether changes in students’ epistemological beliefs go hand in hand with increases in their ability to analyze and evaluate scientific sources and whether students with more expert-like epistemological beliefs are better able to analyze and evaluate scientific data. It is worth noting that the two explanations we posit are not mutually exclusive and that the development of student epistemological beliefs may be another facet of the progression from novice to expert.

Students in our study also had more difficulty than faculty in understanding research design. Instructors could help students make sense of research designs by dedicating one or more class sessions to the nature of scientific research design in the context of the articles that students will be discussing. Alternatively, lecture activities could highlight the research design of classical experiments or of those related to the content of the lecture. The textbook described by Barsoum and colleagues (2013) provides a good model to follow for the incorporation of experimental design into lectures. Instructors could also employ approaches that prompt students to consider the research design of an experiment (Janick-Buckner, 1997; Hoskins ; Krontiris-Litowitz, 2013; Sato ). Although it is unlikely that undergraduates will achieve mastery of research design in one semester, exposing students to the approaches detailed above over the course of several semesters, especially early in their college careers, may accelerate their ability to understand this important facet of scientific literacy (Coil ). Instructors may choose to further develop student scientific literacy skills through a first-year course that trains students to read primary literature and exposes them to the nature of science or, alternatively, throughout students’ college careers by adding activities to courses that involve data analysis and evaluation and expose students to research design.

Limitations of Our Study

While our study involved a small group of faculty and students, the demographic composition of our participants is representative of the overall biology department population at our institution. It will be interesting to repeat our study in other types of institutions and with other individuals to determine whether there are variations in the way students read and analyze primary literature that we did not observe in our student population.

It is important to note that, while faculty participants can be considered experts in the analysis of research articles, they are not experts in the topic of the research study for the article they read. Thus, the skills we observed in this study denote those that are transferable across domains of knowledge, at least within biology. As has been shown for expert–expert comparisons in other processes (e.g., Roth and Bowen, 2003), we may find other dimensions or a different organization for the construction of knowledge if we examine faculty as they analyze articles in their subjects of expertise. A comparison of faculty who are experts in the content area of a research article with students who are not familiar with the content area might yield further insight into how experts and novices differ in their analysis of primary literature.

Future Directions

The results of our study suggest different approaches that would help support students’ analysis of research articles. It would be interesting to test whether including elements in literature discussions that lower the extrinsic cognitive load of the text, such as glossaries and information on experimental design, leads to better student comprehension. Looking at student comprehension of a text in its original format compared with an annotated version, like those available through the Science in the Classroom initiative (AAAS, 2017), could shed light on whether scaffolds to lower extrinsic cognitive load aid in student understanding. This study focused on the commonalities rather than differences in how undergraduates approach reading the literature by looking at the student participants as a group instead of comparing participants at different points of the college experience. However, it would be interesting to investigate how student methods for reading articles vary, especially as they progress through their college studies. Determining how students’ approaches to reading research articles change during their college career through longitudinal think-aloud interviews of a cohort of students could allow us to better describe the transition from novice to expert in this process.

Click here for additional data file.