Letter to the editor.

To the Editor,

Thank you for the opportunity to respond to “The CREATE Method Does Not Result in Greater Gains in Critical Thinking than a More Traditional Method of Analyzing the Primary Literature” by M. Segura-Totten and N. Dalman (JMBE 14: 166–175, 2013). With regard to the comparison of the CREATE strategy and a “more traditional” approach, our comments focus on the nature of the “traditional” teaching experienced by the non-CREATE group, the degree to which the CREATE strategy was actually used, and the critical thinking assessments.

The study compares a modified version of CREATE with “traditional” teaching with primary literature. The “traditional” teaching described in this study does not match our experiences. We view “traditional” to mean that an instructor assigns complete journal articles to students, with the intent that students will read (with no guidance) in preparation for discussion. Subsequent class “discussions” typically involve either the faculty member lecturing about the paper or individual students presenting it, lecture-style. In this format, discussion usually is limited as only a small subset of students engage and most of the dialogue occurs between the instructor and the presenting students. We experienced such teaching ourselves, both as undergraduates and graduate students. Indeed, in our early years of college teaching, we used such approaches with primary literature. Faculty trained in our recent summer workshops also report that this is a typical format for teaching with primary literature. Our lack of success with this traditional approach was part of what motivated us to develop CREATE.

In contrast to this view, the “traditional class” described by Segura-Totten and Dalman is much more directed and models many features of the CREATE strategy. Some aspects typical of CREATE teaching, for example initially distributing the papers without their titles and abstracts (to us a “non-traditional” approach we have not seen used in other contexts), were used by the authors in both the “CREATE” and “traditional” student cohorts. Pre-class preparation in the “traditional” group involved written homework addressing some questions that mirror the pre-class preparation with CREATE tools. The significant overlap in the two teaching approaches used in this study makes comparisons difficult.

We consider the version of CREATE used in the study to be a limited version of the strategy we have described (1–3). In our version, students read a series of papers from one lab group or from different groups pursuing the same scientific issue; thus students see the evolution of a research trajectory over time. Students carry out multiple experimental design/grant panel sessions during their CREATE semester. This repeated practice builds their prowess at both designing and intelligently criticizing experiments. Our CREATE students “build their own textbooks” by compiling notebooks/portfolios that include their annotated articles, concept maps, cartoons, notes from in-class activities, and additional information they have gathered on their own to support their understanding. The portfolios are resources for open book assessments we give during the semester. Much class time focuses on analyzing “how the experiment was done,” with student cartoons demystifying methodology. Students use data templates to pull together their insights from pre-class cartooning and annotation and draw conclusions about data from individual experiments, ideally interpreting the data as if they had performed the studies themselves. Our CREATE students also carry out email surveys of paper authors as an inroad to understanding the motivations of working researchers; authors’ candid and thoughtful responses have strongly influenced students’ sense of both scientists and of the research process (see Tables 1, S1 in reference 2). Many of these features of CREATE were lacking or substantially modified in the version of CREATE used in the Segura-Totten and Dalman paper. In addition, it is not clear how much discussion time was devoted to “how” experiments were done or “why” experiments were done in particular ways (such discussions get into roles of controls, experimental design and the like, and are standard features of our CREATE courses). In our CREATE classrooms, lecture is rare or absent, and discussions do not move directly from defining hypotheses to discussing results.

Finally, the critical thinking assessments reported in this paper are very different from those we have used to address the same issue. We have used two assessments, a set of questions derived from the Field Tested Learning Assessments (www.flaguide.org); and the Tennessee Tech Critical Thinking Assessment Test (CAT) (5) to document significant pre-course/post-course gains in CREATE students. Neither test is distributed in advance, and both require reasoning and analysis in topic areas separate from those of our CREATE classes; thus we think it likely that each test measures transferable thinking skills. In the Segura-Totten and Dalman paper, critical thinking was assessed in two ways. One involved analyses of journal articles, portions of which were distributed a week in advance of the in-class assessment. A second assessment focused on student responses to individual exam questions designed at particular Bloom levels. Whether these assays and ours address the same aspects of critical thinking is an open question.

We look forward to studies that explore how CREATE compares with other methods of teaching and learning from primary literature. We suggest that the use of CREATE portfolios, open-book testing, repeated experimental design/grant panel activities, and email surveys of authors be included in studies of the CREATE strategy. We see these features, coupled with intensive discussion of how experiments were carried out and what the reported data mean—complemented by sustained focus on the development of a project over time—, as contributing significantly to the range of cognitive and affective gains, including critical thinking gains, we have documented in CREATE students (1, 2, 4).

Sincerely,