A Validated Novel Tool for Capturing Faculty-Student Joint Behaviors with the COPUS Instrument.

The Classroom Observation Protocol Undergraduate STEM (COPUS) tool was developed to quantify the time instructors and students engage in various activities within STEM courses. We offer a matrix of joint instructor-student behaviors rather than a pie chart of individual behaviors as an alternative perspective on the presentation of the results from the COPUS instrument. We suggest that the presentation of the results using this matrix tool allows for finer-scale insights into the learning environment in a classroom. Using this matrix tool, we identified four profiles of instructor-student behavior in undergraduate STEM classes at our regional comprehensive Master's institution: lecture, lecture plus (lecture with students posing questions to the instructor), standing clickers/IF-ATs (immediate feedback assessment technique-instructor poses questions using some form of immediate response method but does not move around groups), and roving groups (requiring instructor to move between groups). Prior to using the COPUS instrument we placed each of the observed faculty along the active learning spectrum. Our matrix tool was validated by alignment of the matrix tool profiles with these a priori designations. We offer suggestions regarding how this matrix tool can be best used to inform faculty professional development to move instructors along the active learning spectrum. ©2019 Author(s). Published by the American Society for Microbiology.

INTRODUCTION

There is a particularly strong need for active learning in STEM higher education (1–4). Research suggests that active learning facilitates student engagement in the educational process. In fact, the body of evidence supporting active learning has increased enough that a meta-analysis published in 2014 suggests that having the traditional “lecture” classroom as a “control” for active learning is no longer valid because there is repeated strong evidence for the efficacy of active learning. Instead, it claims, researchers should move to studying which active learning interventions have the greatest impact on specific populations in specific contexts (2). Others have pointed out that defining “lecture” can be difficult (5), thus making active learning also difficult to define. This difficulty assumes that there is a dichotomy between active learning and lecture, rather than a spectrum. We assume an active learning spectrum and describe active learning as activities that involve students in the learning process (6) and, when focusing on in-class settings, any activity that does not involve students listening while instructors lecture (7). General types of active learning strategies, including collaborative and cooperative learning, with specific examples are presented in Felder and Brent (7) and Prince (6).

Even while evidence suggests that active learning benefits students, we lack baseline data regarding the amount of active learning occurring in higher education. An interview in Scientific American with Carl Wieman, Nobel Prize winning physicist and former Associate Director of the White House Office of Science and Technology Policy, underscored this issue: “How prevalent is active learning among universities? I’ve polled the leaders of all the major universities across the county, and I’ve found that no institution actually collects any data on what teaching methods are being used in the classroom, which tells you something about the level of importance attached to this” (https://blogs.scientificamerican.com/budding-scientist/stop-lecturing-me-in-college-science/).

As evidence for the efficacy of active learning mounts (2, 6), the need for faculty development in the ways to best use evidence-based teaching increases. The Teaching Practices Inventory (8) is an excellent self-reflective tool for instructors to gauge “what” strategies they are using, but, as instructors adopt evidence-based strategies, they may have difficulty accurately judging how much time they spend using these strategies (9, 10). This disconnect leads to difficulty in targeting faculty professional development (11). The level of disconnect may increase as an instructor starts to use a teaching strategy that stretches across several days, such as Team-Based Learning (12). We know that the allocation of activities in instructional time is important (13), and if instructors are not aware of the time they are spending using particular strategies, it could affect student learning.

Many assessment tools and inventories are available for collecting data on instructors’ activities in the classroom. Some have the simultaneous advantage and disadvantage of being administered by very well-trained observers (14–16). While the data emerging from these instruments are of high quality, training the observers requires a large time investment. These instruments also seek to evaluate the quality of instruction, a potentially subjective assessment. There are low-barrier assessments emerging that are evidence-based, require little to no observer training, and eliminate assessment of quality, such as PORTAAL (Practical Observation Rubric to Assess Active Learning) (17) and COPUS (Classroom Observation Protocol for Undergraduate STEM) (4). PORTAAL is an excellent instrument for understanding the type of active learning occurring in a classroom that embeds evidence-based best practices into the rubric. This rubric seems very helpful for understanding the faculty perspective in the classroom but lacks the connection of student behaviors to instructor actions (17). Alternatively, COPUS observers record which of 25 distinct instructor and student behaviors occur in each two minutes of class time in order to get a complete picture of the activities. The benefits of COPUS include a fine-scale glimpse into the joint behaviors of faculty and students in the classroom along with the low training barrier and the ease of use relative to other observation protocols (18). Data from COPUS were originally analyzed using two pie charts representing what students and instructors are doing, respectively. Some issues around this analysis, such as the difficulties of identifying instructional styles and comparing the pie charts across courses, were addressed by combining the Reformed Teaching Observation Protocol (19) and COPUS (18).

The original motivation for our study was to use COPUS to develop a baseline active learning spectrum in STEM courses at our own institution (a regional Master’s comprehensive) per Wieman’s aforementioned quote. In other words, “What does the baseline active learning profile look like in STEM at our institution using the COPUS instrument?” In the process of analyzing our data, we developed a novel way of presenting results from the COPUS instrument in the form of a joint-behavior matrix of students and faculty behaviors. We present our methods for the study and the development of the alternative presentation of the results and discuss how well our initial assessment of where faculty would fall on the active learning spectrum aligned with the joint matrix results. We conclude with recommendations for faculty professional development activities that might move faculty along the active learning spectrum.

METHODS

Situational factors

James Madison University (JMU) is a Master’s level institution with a Carnegie Classification as a Community Engaged University. As of fall 2016, the faculty–student ratio was 1:16, with 19,400 undergraduates. In fall 2015, there were approximately 3,500 students with a STEM discipline as their first major, and courses in STEM disciplines represented 35,850 credit hours of which 61% were for the purposes of fulfilling general education requirements. The 2015–2016 academic year saw 757 STEM degrees conferred. The campus has a faculty development center that seeks to provide professional development experiences for faculty in teaching, scholarship, and career planning. There are four full-time and ten part-time faculty developers in this center, with both authors having been in the part-time developer group.

Selection and training of observers

We invited eight STEM and four non-STEM faculty who had previously participated in significant faculty development experiences to be trained as COPUS observers. All 12 were trained in a 1.5-hour workshop as per the COPUS protocol (4). In the workshop, observers were trained in the codes (Fig. 1 of Smith et al. [4]) and developed a consensus understanding of each using videos provided in the COPUS protocol (www.cwsei.ubc.ca/resources/COPUS.htm). We added the codes IG and IQ (Table 1) because the Immediate Feedback Assessment Technique (IF-AT) tool (www.epsteineducation.com/home/about/) is freely available to JMU instructors and provides a distinctly different experience to that of clickers. The IF-AT tool provides students with immediate feedback on whether their selected choice was correct; while this can be done when using clickers, in our experience there is typically a delay before the “correct” answers are given to students using clickers. However, projection of aggregated clicker responses on a screen allows students to see the level of disagreement/agreement within the whole class, something the IF-AT tool is unable to do without some form of follow-up activity. These new codes were observed in only one class.

TABLE 1

Instructor and student behavior codes.

1. Students Are Doing
L	Listening to instructor/taking notes, etc.
Ind	Individual thinking/problem solving. Only mark when an instructor explicitly asks students to think about a clicker question or another question/problem on their own
CG	Discussing clicker question in groups of two or more students
IGa	Discussing IF-AT question in groups of two or more students
WG	Working in groups on worksheet activity
OG	Other assigned group activity, such as responding to instructor question
AnQ	Answering a question posed by the instructor with rest of class listening
SQ	Asking a question
WC	Engaged in whole class discussion by offering explanations, opinion, judgment, etc. to whole class, often facilitated by instructor
Prd	Making a prediction about the outcome of demo or experiment
SP	Making a presentation
T/Q	Taking a test or quiz
W	Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied, etc.)
O	Other – explain in comments
2. Instructor Is Doing
Lec	Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.)
RtW	Real-time writing on board, doc. projector, etc. (often checked off along with Lec)
FUp	Providing follow-up/feedback on clicker or IF-AT question or activity to entire class
PQ	Posing non-clicker question to students (non-rhetorical)
CQ	Asking a clicker question (mark the entire time the instructor is using a clicker question, not just when first asked)
IQa	Asking an IF-AT question (mark the entire time the instructor is using a IF-AT question, not just when first asked)
AnQ	Listening to and answering student questions with entire class listening
MG	Moving through class guiding ongoing student work during active learning task
1o1	Engaged in one-on-one extended discussion with one or a few individuals, not paying attention to the rest of the class (can be along with MG or AnQ)
D/V	Showing or conducting a demo, experiment, simulation, video, or animation
Adm	Involved in administration (assign homework, return tests, etc.)
W	Waiting when there is an opportunity for an instructor to be interacting with or observing/listening to student or group activities and the instructor is not doing
O	Other – explain in comments

All except IG and IQ are reproduced from Figure 1 of Smith et al. (4).

IF-AT = immediate feedback assessment tool.

Selection of disciplines and faculty to be observed

The researchers considered all instructors in STEM disciplines at JMU who were teaching undergraduate classes in STEM. We invited specific faculty who we believed collectively represented the spectrum of active learning (sage-on-stage to guide-on-the-side) to be observed in order for our baseline active learning spectrum to represent the full range of behaviors present in STEM at JMU. Prior to any COPUS profiling, we classified the faculty participants into one of the following teaching styles: “active,” “mid-range,” or “lecture” based on our knowledge of the faculty member’s teaching through prior conversations, observations, or consultations. Our definition of the active group required use of evidence-based active learning in their courses, “mid-range” consisted of faculty who engaged students within their lecture using some approach, e.g., an interactive lecture, and the lecture group was for faculty who were believed to lecture without the use of techniques to engage students. Of the 22 faculty we observed, 6 were in each of the lecture and mid-range groups, and 10 were in the active group.

Courses observed and inter-rater reliability

Each of the 22 classes was observed live by a pair of observers, with those pairs selected based on their availability to observe a particular class. Class sizes were representative of STEM class sizes at JMU; twelve classes had between 12 and 36 students, there were two classes of approximately 60, and eight classes of over 100 students. Of the 22 classes, eight were in mathematics or statistics, five in chemistry, three in biology, two were in computer science, and engineering, physics, geology, and integrated science were each represented by a single class. We do not attempt to distinguish the disciplines nor the class sizes based on a sample of this size.

Each observer pair attended their focal class and conducted the assessment during class times that ranged from 50 to 75 minutes. The pairs used an egg-timer to time the two-minute intervals. We measured inter-rater reliability using Cohen’s Kappa. The average Cohen’s Kappa between pairs of ratings for the 22 classes was 0.77. Forty-one per cent of the Kappa values were above 0.81, indicating almost perfect agreement, and the remaining were above 0.6, indicating substantial agreement. Observers’ notes were used to understand coding differences between a pair of observers when these occurred, and in one instance an observer informed us of a misunderstanding of a code.

Behavior summaries

Our original analysis produced pie charts (per Smith et al. [4, 11]) but we had difficulty fully interpreting the activities occurring in the class when the faculty and student behaviors were separated. While it is possible to assume the joint faculty-student behaviors from the pie charts, we desired an approach that would remove the need for assumptions. (We define joint behaviors as the interactions of instructors and students with each other, e.g., students could be listening [L] while the instructor is lecturing [Lec], or conducting a demo [D/V], moving through groups [MG].)

A matrix provides a clearer picture of the types of interactions and how students are being engaged. In both example profiles in Figure 1, the instructors and students are individually engaged in the same three activities one third of the time; however, the second profile shows nine distinct interaction types versus three in the first. For example, consider two of our observations summarized in Figure 2. The pie charts suggest that students in profile 1 are listening (L) for a larger percentage of the two-minute intervals than those in profile 2 (58.1% vs. 53.4%). However, the matrix more clearly indicates that while an equal percentage of these listening intervals correspond with instructors lecturing and students passively listening in the two profiles (22.5% vs. 21.1%) (one definition of inactive learning [7]), students in profile 1 are engaged in listening during instructor-posed questions (PQ 14.4%) versus the instructor answering student-posed questions in profile 2 (AnQ 14%). The presented profiles are not meant as examples of active vs. passive learning but rather two possibilities for instructor-student behaviors while students are listening. Shading on the matrix views corresponds to the size of the observed percentages. Since more than two codes can be recorded in each time interval, the pie charts do not display the row and column totals of our matrix views.

FIGURE 1

Hypothetical profiles of instructor and student joint behaviors that illustrate the benefit of a matrix or grid summary. The row and column sums are the same in both profiles and would thus be indistinguishable on pie charts of instructor or student (but not both) behaviors.

FIGURE 2

Pie charts depicting two profiles of student (top) and instructor (middle) behaviors, and the corresponding matrix/grid views of the joint behaviors of these students and their instructors. Stated values in the pie charts are the percentage of all two-minute time intervals both raters agreed the student and/or instructor codes took place, and values stated in the matrix are the percentage of all pairs of student and instructor codes that both raters agreed were observed in the two-minute time intervals. See Table 1 for definitions.

Generation of profiles

A matrix was generated for every course observed. We analyzed these matrices using informal pattern recognition. Since our sample size was limited, traditional cluster analysis is not a viable statistical option for generating profiles from the data (per Lund et al. [18]). Due to the way we purposefully selected the instructors to observe, we assumed a priori that we would have “lecture profiles” and “active profiles,” the two ends of the spectrum on which we selected instructors. We did not have a sense of the number of distinct profiles to look for beyond this. One author then looked for unique and prominent combinations of joint behaviors for students and instructors. For example, moving through groups in the classroom seemed to be a unique behavior of faculty that identified a mode of active learning. The resulting four combinations were then adjudicated by both authors with descriptors of the profiles determined. Two observations that had a variation of the described profiles were further discussed and placed in a profile. The joint behaviors that define each profile are detailed in the results along with comments on these two observations.

Institutional review board

This study was approved by James Madison University’s Institutional Review Board, protocol #16-0037, and has complied with all relevant federal guidelines and institutional policies.

RESULTS

Observed active-learning profiles

We observed four profiles of joint behaviors (Fig. 3) as a result of our informal pattern seeking based on prevalent joint behaviors: lecture is a group of profiles characterized by students listening (L) and answering questions (AnQ) and instructors lecturing (Lec), but with no students asking questions (SQ) nor the instructor answering questions (AnQ or FUp). Students in lecture plus courses listen (L), answer questions (AnQ), and pose questions (SQ), while their instructors lecture (Lec), write on board (RtW), or pose questions (PQ). Instructors may or may not answer questions (AnQ). The distinguishing characteristics between the lecture and lecture plus groups are students answering and posing questions and instructors writing on the board or posing questions. The standing clicker/IF-AT group uses clickers (CG) or IF-ATs (IG) with perhaps other forms of group work. However the instructor is not recorded as moving through the class (MG). Roving groups are characterized by students listening (L), doing some form of group work (WG or OG) and either answering or posing questions (AnQ or SQ). Instructors (including peer instructors or graduate students) in roving groups must be observed moving around the class (MG). As mentioned in the methods, two observations required discussion between the authors to assign to profiles due to having variations on the required behaviors. An observation placed in the lecture group has the student posing questions code (SQ) but lacks the required students answering questions code (AnQ). It was placed in lecture profile due to the prevalence of students listening; we decided that the presence of student AnQ codes was a sign of engagement and more critical to the lecture plus profile than to the lecture profile. An observation placed in standing clickers/IF-AT had moving through the groups code (MG) observed in a single two-minute interval. Due to the very low occurrence of this code, we placed this observation in standing clickers/IF-AT rather than roving groups.

FIGURE 3

Matrix views of the four joint student-instructor behavior profiles observed in this study. Values stated in the main grid are the percentage of all pairs of student and instructor codes that both raters agreed were observed in the two-minute time intervals. Shaded codes in the first row (and first column) indicate student (and instructor) codes used in generating the profile; italics indicate there must be an absence of the code. Values in the last row and column (labeled Individual) are the percentage of all pairs of codes observed that contained that code, i.e., row and column sums.

The supplemental materials provide the matrix for each of the 22 observations. Of the four profiles, lecture and lecture plus are the most similar in terms of required behaviors (their absence or presence). To further elucidate the distinctiveness of these two, the matrices include (in last row and column) the percentage of observed pairs of student and instructor codes that included each particular individual code. With the exception of the previously discussed lecture observation, RtW codes are absent from the lecture profile but are present in all but one lecture plus profile; the exceptional lecture plus profile is an instructor who had no available space to write. Between 2.5% and 22.8% of the pairs of codes observed in lecture plus observations involved students asking questions, the absence of this code in lecture (again with the exceptional observation) was notable as this represents an important student behavior.

STEM courses observed at JMU show a prevalence of active profiles. Roving groups (n=7) and lecture plus (n=8) were the dominant profiles, while lecture (n=3) and standing clickers/IF-AT (n=4) were less frequently observed (Fig. 3). This may be an artifact of our non-random sampling of participants rather than an indication of the true nature of STEM instruction at JMU.

Adjudication of predictions and profiles: initial validation

After we identified the four profiles from the matrices, we looked at the agreement between the authors’ a priori beliefs of each instructor’s practices and the profile to which they were assigned. Of the 22 faculty who were observed in this study, we initially (prior to COPUS observation) predicted that six would be at the lecture end of the scale, ten would be at the active end and six would be in the middle of the active-lecture scale. Our COPUS profiling placed eight of the ten initially active group into roving groups or standing clicker/IF-ATs, four of the six initially lecture faculty into lecture or lecture plus, and five of the six initially middle-group into something other than the lecture group. We consider this good alignment between our a priori predictions and the observed behaviors.

DISCUSSION

We have developed a novel approach to viewing the data that are generated by the COPUS tool. This approach allowed us to identify profiles of student and instructor behaviors in STEM courses at JMU. These profiles can be used by instructors and faculty developers to introduce active learning strategies in a progressive manner. We interpret these results in light of our directed sampling, in which we recruited instructors from multiple STEM disciplines and across the active-learning spectrum at a Master’s granting institution. We identified four major profiles ranging from primarily lecture to highly active students and instructors, with the dominant groups sampled being roving groups and lecture plus.

Joint behaviors in the context of the literature

The presentation of the observations in a joint matrix of instructor and student behaviors provides a fine-scale view of the co-occurring behaviors of instructors and students in the classroom. Felder and Brent’s (7) definition of active learning within the classroom setting would be consistent with any of the instructor-student behaviors outside of Lec-L (or RtW-L depending on the discipline), but their looser definition is perhaps less useful in delineating active versus passive at institutions such as ours where Lec-L occurs less often. The roving groups profile would also seem the most consistent with the collaborative and cooperative learning strategies discussed in Prince (6). We suggest that the T/Q, Adm, W, and O codes be considered neither passive nor active. Our lecture profile most resembles Lund et al.’s (18) lecture (with slides) and lecture (at board) profiles, with differences in the RtW prevalence; our lecture plus profile most resembles their Socratic profile; and our standing clicker/IF-ATs and roving groups profiles are most consistent with their peer instruction and collaborative learning profiles.

Faculty professional development

Instructors and faculty professional developers can use the tool presented here to establish an understanding of an instructor’s current behaviors and to help identify strategies that would increase the variety and duration of student activity in the course. In a recent study, Biology 101 students who engaged in a greater variety and extent of COPUS behaviors showed significantly improved exam scores, a higher odds ratio of passing exams, and higher retention of content than students in a different section of the same course in which students were engaged in fewer COPUS behaviors (20). Having a spectrum of profiles may help in designing experiences that move instructors incrementally toward active learning. We consider a few suggestions for moving faculty from profile to profile.

Lecture to lecture plus

The main difference between lecture and lecture plus is that the latter requires that students ask questions (SQ). Faculty development activities could focus on helping a faculty member to identify behaviors that be prevent students from asking questions, such as not providing opportunities for questions or not developing a sense of psychological safety in the classroom. One concrete and accessible way to allow for student questions while providing students with the safety of anonymity is to incorporate an exit poll in which the students write questions on an index card before leaving the room, and the instructor fields some of these questions in the next class. This exercise will start the students on the path to feeling comfortable about asking questions, give the instructor insight into the types of questions students will ask (helping with their preparation for future iterations of the course), and give the instructor who may be uncomfortable answering on the spot the opportunity to construct responses.

Lecture to standing clickers/IF-AT

The main difference between lecture and standing clickers/IF-AT is the use of clickers or IF-ATs (some sort of immediate feedback mechanism), but there is no moving among groups (MG). Faculty development could be focused in the area of using either of these technologies, although with the advent of multiple apps for collecting real-time responses, there does not need to be a “university supported” technology. One of the criticisms that often arises with the use of clickers is in the area of inclusivity, the assumption that everyone can answer the questions using a smart phone implicitly assumes that all students have a smart phone. We suggest using an app that uses SMS texts to alleviate this criticism and offering alternative forms of interaction for students without a phone such as students simultaneously holding up the number of fingers (1, 2, 3, 4, or 5) that correspond to what each thinks is the correct response. A second recommendation would be to use the IF-AT forms as a type of exit quiz/poll as a low-stakes way to use the technology and empower the instructor to use them in additional contexts.

Lecture plus to roving groups

The roving groups profile requires that an instructor move among groups in the classroom, implicitly assuming that the instructor is using group work as an instructional strategy. Group work takes many forms, from an ad hoc think-pair-share group to an intentional team strategy as in Team-Based Learning (12). The easiest path from lecture plus to roving groups would be for the instructor to pose questions and then move around the room listening to student discussion. Encouraging the use of more complex questions would allow the instructor time to move around the room but also encourage students to work together to find solutions. One of the most common criticisms from instructors regarding having students work in groups or teams is that the physical environment or structure of the classroom is prohibiting, e.g., stadium seating may only allow movement around the room’s perimeter. Faculty developers could counter this criticism by discussing the research on instructional immediacy (reducing the physical and psychological distance between instructor and groups of students), which has been shown to have multiple positive impacts (21, 22). Brainstorming solutions to the problems presented by the physical environment, such as allowing students to work in teams in hallways or alcoves if available, may empower a faculty member to allow team work. Of course, moving through groups allows instructors to dispel misconceptions before a summative assessment.

Limitations of the study

There are several limitations to the current study. First, each class was observed on a single day. The closest comparison to our study, Lund et al. (18), observed multiple class periods for the same class/instructor. However, each class period was treated as a single unit rather than aggregated to build a picture of behaviors across multiple class periods. Thus, while their sample size is larger, their analysis was on the same unit.

One of the limitations of single-day observations is the chance of observing on an atypical day. This could happen for logistic reasons—e.g., immediately post-test or snow day—or due to instructors employing a pedagogy that uses a particular sequence of activities across multiple days, such as lecture during the first class of the week and group work during the second, or sequences consistent with Team-Based Learning (12). Further, observing any one class will either inflate or deflate measures of active learning. The authors are aware that some of the instructors observed in this study use such pedagogies. In fact, the incorrect predictions identified for three faculty—placed into different profiles by the matrix than the authors predicted—was a function of observing the instructor on an “alternative” day. The authors suggest that for institutions where there is variety in day-to-day activities, it may be best to observe multiple class periods that represent the typical sequence of activities in the pedagogy used. The scheduling burden could be eased by capturing these classes using video with observers coding the videos at their convenience (per Lund et al. [18]).

Benefits to observers

Observers have anecdotally reported that they learned new classroom management and activity tips from observing the classes, such as appreciation of the teaching styles in other disciplines, a broader understanding of what “active” is, and how to manage large classes. Similar comments are reported at our institution by facilitators of Small Group Instructional Diagnostics (SGIDs), although no formal research has yet been conducted.

CONCLUSIONS

The COPUS instrument is a straightforward, non-evaluative, easy-to-use instrument that allows faculty to discover their location on the path to active learning. The presentation of joint behaviors in a matrix more clearly facilitates the understanding of instructors’ teaching styles by elucidating the range of behaviors that students engage in depending on what the instructor is doing. Faculty professional developers could use this matrix tool to suggest approaches that meet instructors at their current level of comfort and move them further along the active learning spectrum.

Click here for additional data file.