Adapting Under Pressure: A Case Study in Scaling Faculty Development for Emergency Remote Teaching.

This case study examines the adaptation of an existing online, asynchronous faculty development resource at the University of New Mexico to support the unanticipated need for all instructors to teach remotely starting in spring 2020, due to the COVID-19 pandemic. The course-entitled Evidence-Based Practices for Teaching Online (EBPTO)-was previously utilized to support instructor transitions to distance education by applying constructivist principles to the development of evidence-based online teaching practices. The course was adapted to address institutional and instructor needs as a result of the pandemic, including increasing facilitation resources. The largest EBPTO cohort, with 117 participants, began in June 2020. Data were collected through a reflective journal administered at the mid-point and an end-of-course survey. Analysis of the reflective journal provided insight into participants' learning experience in terms of key "takeaways," LMS tools that they had the opportunity to practice, and "lingering questions" that they had. The top 3 takeaways were the usefulness of course mapping, the usefulness of backwards design, and the deepening familiarity with LMS tools. Results from the end-of-course survey showed positive feedback from participants regarding perceived achievement of the course learning objectives, even after scaling the course to accommodate the large number of instructors moving to remote instruction.

Problem Statement

In recent decades, universities have created instructor development programs to manage and support the steady adoption of instructional technologies, as classes move from traditional formats to hybrid, remote, and online modalities (Belt & Lowenthal, 2020). Instructor development to support this transition is often offered in the form of an online course, a format that makes it logistically feasible to lengthen the instructor development process while also increasing enrollment (Chen et al., 2017; Lowenthal, 2008). The value of longer development processes has been confirmed by Kukulska-Hulme (2012) and Mourlam (2017).

While these studies suggest the appropriateness of an online format for the sizeable numbers of instructors adopting new technologies, they do not specifically address urgent and broad adoption, such as the sudden need to teach all courses remotely during the COVID-19 pandemic. This case study examines the application of an existing online instructor development resource to the challenges posed by sudden and dramatic increases in remote teaching. How can resources be scaled to address the needs of substantial cohorts of instructors who need development in instruction-related domains while still maintaining the sense of community that instructors so often value in a professional development experience? Can it be done without losing the social learning benefits that we hope to model and encourage instructors to adopt?

Case purpose

Many universities worldwide were faced with a dramatic shift in operations in early 2020, including the need to close physical classrooms and switch to remote modalities starting in the middle of a term and continuing in subsequent terms over the following year. This study focuses on the adaptation of a resource designed for the normal flow of instructor adoption of remote technologies to the challenges of exigent circumstances during a sudden shift to remote learning. Given the steady pre-pandemic move to remote modalities, this case also carries some relevance for normal trends as they occurred pre-pandemic and as they may resume post-pandemic. Despite the many drawbacks of emergency remote teaching during 2020 (Hodges et al., 2020), distance education continues to show growth in adoption, and student experiences during the COVID-19 health crisis may have increased student interest in both distance education and hybrid courses (McKenzie, 2021).

While it has specific relevance to the adaptation of professional development programs when there is some sort of institutional or community shift, this case will also make clear how long-term investments in infrastructure supporting online education allow an institution to provide better teaching and learning experiences in times of both normal growth as well as rapid adoption. Having a sustainable, scalable solution in place requires investment and continued commitment by an institution, and it yields dividends in the overall improvement of online student learning in “normal” times as well as in the midst of crisis.

Significance

This case makes two contributions to institutions looking for models to support online learning. First, it demonstrates that a strong culture and infrastructure around quality online teaching enables effective and rapid responses in times of increased growth or need. Second, it illustrates how careful design can allow a scaled resource to maintain the benefits of social learning. The University of New Mexico is a Hispanic-Serving (HSI) Research Intensive (R1) institution. Similar to many peer institutions, UNM is challenged by resource scarcity that impacts the number of instructor lines available for a department and the responsibilities those instructors bear as part of their normal teaching, research, and service. As a result, the authors are keenly aware of how challenging it can be to allocate long term investments toward instructor development.

While many instructors are open to taking EBPTO because of their perceived lack of technical understanding, the course specifically situates technology as a tool to enhance learning. Instructors will often comment on how the course helped improve their online and face-to-face teaching. EBPTO applies a constructivist approach to assist instructors in meeting the course’s learning objectives. Active learning (Bruner, 1960; Perkins, 1992) plays a key role in both the UNM Online Course Standards Rubric and the design of EBPTO, which allows instructors to learn from the kinds of experiences we hope they will create for their students.

Inspired by constructivist views and theories of distance education, the creators of EBPTO aimed to design an online course for instructors that would provide an example of an authentic learning community. This was accomplished by providing support for participants as they engaged with the LMS environment in a student role, and by using learning activities that manifest as real-world course development tasks, such as the creation of a partial course map that participants can subsequently use in the design of their online course. The learning environment is inherently multidisciplinary and is not constructed to teach a subject such as geometry or philosophy, but to represent the real-world application of designing a sound distance education course or a course that must be delivered remotely. The course is, in a sense, a metacognitive learning experience.

Building a learning community is an implicit goal of EBPTO. It is not stated as one of the course learning objectives, but collaborative learning and interaction are encouraged through the use of discussion forums. The learning process, the negotiation of meaning and validation of knowledge, often takes place in this learning community via the discussion forums. EBPTO levels the playing field for participants of different academic rank, due to the nature of asynchronous communication. The course takes seriously critiques of how the discussion tool can be abused in online courses: “To have discussion for discussion’s sake is not good instructional design. The discussions within an online course must be orchestrated to enable the learner to meet the learning outcomes, and build knowledge and insights” (Shearer, 2013, p. 257).

Instead, the course models an approach in line with Vygotsky’s (1997) theory on social and cultural aspects of learning. He suggests that “every higher mental function was external because it was social before it became an internal, strictly mental function” (p. 105). Therefore, participants should not only work toward their own progress in course activities, but also use their time and effort to help their peers and serve the larger community with their project ideas, with guidance from the course facilitators. Participant support becomes a key strategy to encourage the growth of the learning community and the transformation of participants’ views of distance education. This is modeled not only by the co-facilitators of EBPTO, but also the instructional designers who each guide a pre-determined number of instructors during the course.

Delimitations

The sheer volume of participant submissions in this largest of the EBPTO cohorts necessitated selectivity in the materials examined in this study. Two sources were selected—journal entries, and the end-of-course survey—because they provide the greatest amount of information on participant learning experiences, learning outcomes, and experiences participants had in the midst of the pandemic. Journal entries were posted by participants at the course mid-point. The journal prompt asked them to identify three things: (1) key takeaways from the course, (2) LMS tools they have practiced using, and (3) any lingering questions they may have. The survey at the end of the course is perhaps the richest source of information about the perceived takeaways of participants, whether they feel course objectives were met, and how the course functioned for them in the midst of rising pandemic-related responsibilities.

The intent of the authors is not to generalize findings to individuals, sites, or places outside of those under study. The value of this case study lies in the specific description and themes developed in the context of EBPTO. Particularity rather than generalizability is the distinctive characteristic of qualitative research (Creswell, 2009). Thus, the intent of the authors is to provide detailed descriptions so that anyone interested in transferability can transfer—to a certain degree—the results of this study to similar contexts or settings.

Limitations

This case study relies on the qualitative examination of surveys and journal entries. Limitations related to this type of data according to Creswell (2009) may include the following: (1) Not all people are equally articulate and perceptive, (2) Documents may be protected information unavailable to public or private access, (3) Data require the researcher to search out information in hard-to-find places, (4) Data require transcribing or optically scanning for computer entry, (5) Materials may be incomplete, and (6) The documents may not be authentic or accurate. The authors found limitations 1, 4, and 5 applicable to their analysis process and every effort was made to control for them.

While the examination of a variety of documents allows researchers to triangulate information, this case study could have used follow up interviews with EBPTO participants or an analysis of course deliverables to determine how well instructors understood the concepts they identified as most valuable. Learning analytics often provide interesting supplemental information such as participant activity reports about time spent in an online course, number of posts in a discussion forum, or assessment scores but they were not considered due to the nature and substance of the approach. This information could have also provided insight connecting participants’ connection in the course, the impacts of their social learning, and their subsequent evaluation of the experience.

While the authors recognize that most learning theories and associated applications are effective for some learning tasks, they also acknowledge—as Instructional Designers—a reliance on the seminal work of Bruner (1960) and Perkins (1992) on Constructivism and Active Learning, respectively, as the “best” approaches to the design of online courses with a visible enthusiasm for high quality instruction based on cultural inclusivity, student initiative and self-direction, and integration of tools that foster interaction and collaboration.

Literature Review

Instructor development in Educational Technology

Developing educators to teach with technology is the focus of a significant amount of research and theoretical modeling. Developers of the technology, pedagogy, and content knowledge (TPACK) integration model of the teacher knowledge base, Koehler and Mishra, describe teaching as “an ill-structured discipline, requiring teachers to apply complex knowledge structures across different cases and contexts” (2009, p. 61). They also describe emerging instructional technologies as “protean, unstable, and opaque, present[ing] new challenges to teachers who are struggling to use more technology in their teaching” (2009, p. 61). When teaching with technology, faculty combine knowledge in a domain with which they are comfortable (content) with knowledge in two domains that are ill-structured and unstable (pedagogy and technology).

The complexity of teaching with technology helps explain findings from several studies. Faculty are not using instructional technologies to the degree they could (Bates & Poole, 2003; Kukulska-Hulme, 2012). In a recent literature review, Belt & Lowenthal (2020) analyzed 45 studies on teaching with technology published between 2013 and 2018. Polly et al. (2021) also reviewed the literature as foundations for their faculty development studies. Their findings identify time as the major issue for faculty. Existing commitments to teaching, research, and service leave little space to address the ever-increasing role of instructional technologies (Belt & Lowenthal, 2020; Polly et al., 2021). Due to their high levels of education, some faculty may be unaccustomed to the feeling of being less informed in a knowledge domain relevant to their profession. They may feel more uneasy regarding technological aspects of teaching than other aspects (Belt & Lowenthal, 2020). In terms of training formats, faculty tend to find short, “one shot” development opportunities “unhelpful and frustrating” (Mourlam, 2017, p. 22; Kukulska-Hulme 2012). Online courses, such as UNM’s EBPTO, have made it possible to lengthen the faculty development process while also increasing participant enrollment (Chen et al., 2017; Lowenthal, 2008). Participation in such courses can be limited when voluntary, but may be substantially boosted when stipends are provided (Belt & Lowenthal, 2020; Lowenthal et al., 2013).

It is not surprising to find that these general studies of instructor development for teaching with technology have been supplemented recently by research on emergency remote teaching (ERT) during the pandemic. A number of studies of ERT at the university level have already been published, including four on instructor experiences while teaching emergency remote classes (Al-Freih, 2021; Chierichetti & Backer, 2021; Ferri et al., 2020; Walsh et al., 2021), two on the development of webconferencing skills and their use for remote instruction (Brereton, 2020; Moorhouse & Kohnke, 2021), and a case study of a learning community started by faculty to support their own transition (Song et al., 2020). There is also a study offering professional development framework for ERT (Al-Naabi et al., 2021), another on the course development process (Hodges et al., 2021), a set of guidelines for student assessment during ERT (Rahim, 2020), and a survey-based study addressing the ERT experiences of students (Petillion & McNeil, 2020). Another study examines the views of instructional designers on their changing roles during remote teaching (Xie, Gulinna, & Rice, 2021).

While there are two studies examining the professional development experiences of instructors during ERT (Redstone & Luo, 2021; Toquero & Talidong, 2020), we found just one focusing on adjustments to their professional development supports (Evmenova et al., 2021). It describes a university’s existing course for transitioning faculty to teach online – a six-week instructor-led Online Teaching Initiative (OTI) – and two stages of revisions to the course to meet the unexpected needs of ERT. The first revision was a “stopgap measure” that involved turning the instructor-led course into an online self-paced course (p. 30). The second involved adding instructor-led modules back to the course, making it partially self-paced and partially instructor-led. The designers took this second step in part based on their experiences with faculty and in part based on previous participant feedback from the original OTI course. Neither this study nor the others we found examined both the adjustment of faculty supports due to ERT and the experiences of faculty during and after the training. The present case study meets a need to investigate adjustments to development resources and the resulting experiences of faculty.

Methodology

Case Study

The design of this investigation follows research principles that stem from basic qualitative studies, which focus on meaning, understanding, and process as suggested by Merriam (2009), thus a purposeful sample of information is necessary to provide for data collection (e.g., surveys, journal entries). Typically, this kind of investigation is inductive, leading to richly descriptive findings generally presented as themes. More specifically, the design of this investigation is that of a case study focused on a rare or critical event, so its results should have transferability to similar contexts and settings. Qualitative data analysis and interpretation is an ongoing process that involves a constant reflection about the data, asking analytic questions, and writing notes throughout the study. Furthermore, data analysis is conducted concurrently with gathering data, making interpretations, and writing reports (Creswell, 2009).

Subject of analysis

The primary development tool for instructors adopting online modalities at UNM is the six-week Evidence-Based Practices for Teaching Online (EBPTO) course. It was created in 2016 to support instructors planning to teach in UNM’s Accelerated Online Programs (AOPs), which are fully online degree completion programs. Awareness of the course gradually increased, and it was opened to other interested instructors at UNM, including contingent faculty, instructors from UNM’s four branch campuses, and graduate students. Updates were made periodically to improve the course, and it went through a substantial redesign in summer 2019, with revised learning outcomes, improved alignment, more engaging learning activities, and expanded research on teaching and learning. Since its inception, EBPTO has been offered in the fall, spring, and summer terms, often several times each term, making it available year-round. Two additional sessions of EBPTO were added as a response to the need for instructors to design and build sound learning solutions for both online courses and remote teaching modalities during and after Spring 2020.

With the arrival of COVID-19, there was an immediate and unexpected need for all UNM instructors to complete their courses remotely and to continue teaching remotely or fully online in the subsequent summer, fall, and spring terms. EBPTO registrations rose from normal levels of 11 and 15 in the two sessions that started in January and March, 2020, to a peak of 117 in June, 2020. It is worth noting that the initial registration in June, 2020 was 145, but not all participants continued with the course after Module 1. Previous to June 2020, approximately 365 instructors at UNM had completed EBPTO.

In all of 2020, a total of 286 instructors from different colleges, myriad disciplines, and academic ranks registered for EBPTO. To mention just a few examples, instructors were from departments such as Mechanical Engineering, Sociology, English, Music, Chicana Chicano Studies; Individual, Family, and Community Education, Theater, Foreign Languages and Literature, Film and Digital Arts, etc. In short, all UNM colleges were represented in EBPTO. The availability of this course laid the foundation for an orderly response to an incredibly high demand in instructional design services at UNM due to the COVID-19 pandemic.

The course objectives for EBPTO are that upon successful completion, participants are able to: (1) apply evidence-based practices for online learning, (2) prepare to build and/or facilitate an online course following the UNM Online Course Standards Rubric, and (3) practice using UNM Learn tools to understand the student experience. Each session of the course has two facilitators from the UNM instructional design team (part of the UNM Center for Teaching and Learning), and each participant is assigned an instructional designer who guides them through the course process with key touch points aligned with reflective assignments, a course map, and course building activities. While this may seem like a high touch instructor development experience (and it is meant to feel this way for participants), the added workload for instructional designers is moderate because it is spread across a total of nine designers with a project manager and associate director taking on additional instructors when the need arises. Generally speaking, the course allows instructional designers to begin a relationship with participants they will continue to work with moving forward or to help or connect participants to support resources that they can access long-term. By building rapport early, instructional designers are often saved time in the long run because instructors have an early connection to sound resources and design practices.

An important piece of institutional information that supports EBPTO is UNM’s Online Course Standards Rubric. This document was commissioned by a Provost committee serving from 2005 to 2009 comprised of faculty and distance education experts. It was approved by the Faculty Senate Committee on Teaching Enhancement in 2013, and updated in 2019 to better align it with research on effective online teaching practices. A council of faculty and distance education experts meets twice each semester in continued stewardship of the rubric and institutional processes that rely on the rubric. The rubric provides a framework for online course design based on expectations for quality online courses in line with federal regulations for online courses as well as national standards like those established by entities such as Quality Matters, SUNY Course Quality Review Rubric, and the Peralta College Online Equity Rubric. It serves to guide instructors in developing new online courses and in self-assessing existing online courses for improved design and delivery based on nationally recognized, evidence-based practices for teaching online. Expectations for quality online courses are divided into the following standards: (1) Course Overview and Introduction, (2) Instructional Elements, (3) Interaction and Collaboration, (4) Assessment and Feedback, and (5) Course Evaluation. The learning experiences in EBPTO are designed to illustrate practices defined by this rubric that are applicable to a 6-week long faculty development online course.

Within the Online Course Standards Rubric, the Course Overview and Introduction standard requires instructors to clearly outline the course purpose, structure, policies and performance expectations before the student begins the course, which are typically outlined in the syllabus. The Instructional Elements standard requires that learning objectives and aligned measurable outcomes are clearly defined and communicated, and that active learning is promoted through an interactive learning environment, utilizing different types of media, multiple tools, and different formats of materials.

According to the rubric, online course instructors are also required to orchestrate learning activities that are designed to promote both student-to-student and student-to-instructor interaction and engagement. These activities need to be explicitly aligned with learning objectives and outcomes in order to meet the Interaction and Collaboration standard. The Assessment and Feedback standard requires that assessments be aligned with specific learning objectives and general outcomes of a course, and that performance expectations are clearly defined, but most importantly, that expectations are presented to students upfront, and that the grading and feedback procedure is clearly defined in terms of the amount of time it will take the instructor to grade activities. Last but not least, the Course Evaluation standard indicates that course evaluations such as welcome, or mid-course, or end-of-course surveys are utilized by the instructor to provide opportunities for continuous improvement in future course delivery.

To model one way of meeting the rubric criteria, EBPTO begins with an orientation that includes welcome videos from the facilitators and from the Associate Director of the Center for Digital Learning. During the June 2020 sessions, the complement of facilitators was increased from two to four, to provide additional guidance and to handle the increased volume of facilitation tasks. During the orientation, participants are asked to introduce themselves, post any questions or concerns they may have, and complete a survey in which they share their perceptions of online learning. They are also asked to contemplate and share “enduring understandings” they would like their students to take away from their classes over the long term.

In Module 1, participants are shown the results of the perceptions of online learning survey they took in the orientation. They also view a presentation covering fundamental research on the impact of online instruction. They discuss both their perceptions and the research. The remainder of the course takes them through basic design information, such as learning objectives and the backward design process, followed by modules on collaboration and interaction, active learning, assessment of student work, and finally, the processes for ongoing evaluation and revision of online courses.

These materials are covered using videos, presentations, reference documents, and example documents. Participants interact with one another and with facilitators using text-based discussions, video discussions (one within the university LMS and one using FlipGrid), a reflective journal, and course messages. Course assignments include writing course objectives, designing an active learning activity, and completing a course map for the first four modules of an online course. The course ends with a capstone online teaching portfolio that compiles and illustrates an instructor’s work in the course.

The focus of this case study is the largest EBPTO cohort, a session that began in late June, 2020 and continued through July and early August. There was a total of 117 instructor participants actively enrolled in this session. The peak in demand that occurred with this session was fueled in a general way by the pandemic, and specifically by the unprecedented volume of remote classes that needed to be taught in Fall, 2020. The content of EBPTO remained largely the same and focused on the same core objectives, though there were adjustments to interactive course elements to account for the anticipated volume of discussion posts, assignment submissions, and other course activities.

Data Collection and Analysis

To inform broader perspectives about the research problem, the authors considered different raw data from the online course as a preliminary approach to validating the accuracy of the information. After preliminary work was completed, a purposeful sample of information was necessary, so the authors collected end-of-course survey results, and journal entries from the institutional Learning Management System, Blackboard Learn. Then data were anonymized and deidentified in a spreadsheet, which was securely stored in the university’s file hosting service OneDrive with restricted access to the research team.

To conduct the analysis, the following steps were taken according to Creswell (2009): (1) Organizing and preparing data for analysis, (2) Reading through all data, (3) Coding the Data, (4) Developing descriptions, (5) Interrelating descriptions, and (6) Interpreting the meaning of descriptions. Even though these steps or stages are interrelated, in practice they are iterative, i.e., they are not linear.

The research team included reliability procedures in the study such as collaboratively checking data to make sure that they didn’t contain mistakes made during collection or organization of data. Also, the team held regular documented meetings and shared analysis materials throughout the study. Furthermore, the team incorporated validity strategies to assess the accuracy of findings that included: (1) Triangulation through the use of different data sources of information, (2) Use of detailed descriptions to convey findings, (3) Self-reflection on the bias that the researchers bring to the study provided earlier under the Limitations section, (4) presentation of discrepant information that on occasion runs counter to the themes as explained later under the Implications section, and (5) Spending a prolonged time in the field i.e., each researcher has at least 7 years of professional experience directly related to topic of the study.

Results

Journal entries

Journal entries were posted by 93 participants at the course mid-point. The journal prompt asked them to identify the following: (1) Key takeaways from the course, (2) LMS tools they have practiced, and (3) Any lingering questions they may have. Participants identified creating a course map and applying backward design as useful practices to teach online. These two practices are complementary. Backward design, foregrounded by Wiggins and McTighe (2005), is an approach to course design based on the premise that learning experiences should be driven by large goals or outcomes. An instructor should determine what would be acceptable evidence of a student meeting these goals. Then, the instructor can build learning experiences that either allow a student to provide evidence that they’ve met the learning goal (summative assessments), or practice the skills and knowledge necessary to work toward that learning goal (formative assessments). Course mapping refers to an instructional design technique that is used to align complex course level learning objectives, targeted module-level learning objectives, required materials, and assessments in a design/planning document to show how learning is measured. The course map enables instructors to scaffold assessments (formative and summative) and learning activities to ensure that a student can successfully meet the course learning goals.

Participants also journaled about realizing the variety of tools within a LMS as well as readily available external tools that can be incorporated into online courses to enrich the learning experience. Also, they recognized the usefulness of an online course to create a sense of community for students in a couple of ways. For example, students can be divided into small groups, which allows them to engage in more complex conversations with their classmates. Another way is to set up a community forum where students can help each other in addition to institutional student support services. Bloom’s taxonomy of action verbs was also mentioned by several participants as a key resource to create measurable/observable learning outcomes and specific learning objectives that are more complex in nature, thus more applicable to real world scenarios.

Regarding LMS tools that participants had the opportunity to practice halfway through the course, participants mentioned the Forum tool 30 times. Media was mentioned 21 times, and the vast majority of participants who mentioned it explicitly referred to recording audio/video and deploying it, while a small fraction of participants who mentioned it referred to the use of pictures and/or photos. Participants also identified the creation and modification of a Module, mentioned 11 times, and Blackboard Learn’s Content Collection, mentioned 10 times. In a strict technical sense, modules and the content collection are not often considered tools and were not presented as such in the course content. Creating and modifying a module is an essential feature that all LMSs have, while the Content Collection is basically a repository for all the files an instructor uploads to a course. However, it’s testament to instructors’ understanding of the importance of clear organization and course alignment that they approached these two functions as design features. The Announcements tool, which is an effective communication channel to send a message to all students enrolled in a course was mentioned 9 times. The Tests tool was mentioned 6 times. Participants identified Group Discussions as another tool and they mentioned it 5 times, however it is worth noting that its actual name in Blackboard Learn is the Groups tool, which serves multiple purposes in a course, one of them being the possibility to assign one discussion forum to different groups of students, which is exactly what participants referred to, i.e., dividing a large number of students into smaller discussion groups. The Journal tool which allows students to submit confidential reflections on their learning experience or on sensitive topics covered in a course was mentioned 4 times, followed by the creation and modification of assessments in the Assignment tool, mentioned 3 times, and lastly the creation and modification of Rubrics, mentioned 1 time.

With regards to any lingering questions that participants had mid-point, several themes associated with how-to questions emerged from their journal entries. Participants asked facilitators about possible ways to implement evidence-based practices into their courses. Participants also inquired about course mapping, logistics of the course itself, basic features of a module, and the overall usability of the LMS. Also, they asked about different ways in which video can be used by both students and the instructor, for example to carry out video discussions, submit assignments in video format, and to record video lectures. Participants also showed a need to know how to manage small groups of students. They had doubts that ranged from the difference between emergency remote teaching (Hodges et al., 2020) and distance education courses, to how-to questions related to tracking media views, setting up a community forum, using the journal tool, applying Bloom’s taxonomy of verbs, to concerns about the possibility of cognitive overload for students. There were also those who stated that their questions had already been resolved by either facilitators or the instructional designer assigned to guide them.

Figure 1 shows the most relevant themes that emerged from journal entries when participants were asked to identify 3 key takeaways. Twenty-one participants identified creating a course map as a useful practice; 16 participants identified Backward Design as a useful practice; 15 participants realized the variety of LMS and external tools/apps was more than they initially thought; 12 participants recognized the usefulness of an online course to create a sense of community for students; and 8 participants mentioned Bloom’s Taxonomy of verbs as a key resource to create measurable/observable learning outcomes and specific learning objectives.

Fig. 1

Key takeaways from the course

Figure 2 shows the number of times that participants mentioned different tools when asked which 2 tools they had practiced halfway through the course. In order of frequency, the following is the number of times that tools were mentioned: Forum 29, Media 21, Module 11, Content Collection 10, Announcements 9, Tests 6, Group Discussion 5, Journal 4, Assignment 3, and Rubrics 1.

Fig. 2

LMS tools that participants practiced

Figure 3 shows the most relevant themes that emerged from journal entries when participants were asked about any lingering questions that they had mid-point. Eighteen participants asked facilitators about possible ways to implement evidence-based practices into their courses; 12 participants asked about different ways in which video can be used by both students and the instructor; 11 participants asked about managing small groups of students; 8 participants had questions that were classified as “other;” 7 participants asked about course mapping; 6 participants stated that their questions had already been resolved by facilitators; 6 participants asked about logistics of the course; 6 participants asked about basic features of a module; and 5 participants asked about the usability of the LMS.

Fig. 3

Lingering questions

End of Course Survey

The end-of-course survey for EBPTO was administered via a form linked within both the final wrap-up module and a final course announcement sent by email to all participants. It consisted of 8 questions, one of which is a matrix response made up of three sub-questions. Of the 117 participants who had access to the survey during the June 2020 EBPTO, 51 responded, resulting in a response rate of 44%. For the purposes of this case study, we will be reviewing the quantitative results of the first question (all three sub questions) and question five. We’ll also examine the qualitative results of questions two and four to review participant perceptions of the effectiveness of EBPTO.

The first question asked for a rating (on a three-point scale) of the level of achievement of the three EBPTO learning objectives. The objectives are closely aligned with the goal of utilizing EBPTO to support the surge of new remote instructors during the pandemic, focusing on whether participants feel prepared to: (1) apply evidence-based practices for online learning, (2) build and/or facilitate online courses following the UNM rubric, and (3) understand the student experience in UNM Learn. Of the 51 respondents, 45 gave the highest rating (very effective) on achievement of the first learning objective, 41 gave the highest rating on the second objective, and 37 gave it on the third objective.

Figure 4 shows that for question one, sub question (A) How effective was this course in helping you achieve the course learning objectives? (Apply evidence-based practices for online learning), forty-five of the fifty-one respondents rated the course as very effective. For question one, sub question (B) How effective was this course in helping you achieve the course learning objectives? (Prepare to build and/or facilitate an online course following the UNM Online Course Standards Rubric.), forty-one of the fifty-one respondents rated the course as very effective. For question one, sub question (C) How effective was this course in helping you achieve the course learning objectives? (Practice using UNM Learn tools to understand the student experience.), thirty-seven of the fifty-one respondents rated the course as very effective.

Fig. 4

Responses to end-of-course survey

Question five is also closely aligned with the second learning objective and the goal of assisting new instructors during the pandemic. It asked participants to rate their level of preparedness to teach online as a result of taking EBPTO on a five-point scale as shown in Fig. 5, where one equals not prepared at all and five equals very prepared. Of the 51 respondents, 44 gave a rating of either four or five.

Fig. 5

Count of question: how prepared do you feel to teach online as a result of taking this course?

The rest of the survey was largely qualitative, asking open-ended questions that can help deepen our understanding of these perspectives. To help us better understand how the course supported instructors in meeting the overall learning objectives we analyzed the responses based on their relationship to the three learning objectives. In particular, questions two and four reinforced the positive response to the course learning outcomes even when instructors were not being asked about those learning outcomes directly. Question two asked instructors what part(s) of the course were most useful to them. When analyzed, 49 instructor responses aligned with the course learning objectives, and some responses aligned with more than one learning objective. Question two asked participants to name the most useful part of the EBPTO course. This was an open-ended question with no text character limit. An analysis was conducted of the responses to investigate how they aligned with each of the three course objectives. Each response was reviewed and coded indicating to which objective(s) it related. Some comments fell outside of the bounds of the three course objectives and those were coded as Other. Some responses related to multiple objectives and several other themes appeared in the analysis of the Other category.

Once the responses were all analyzed, then the columns were transposed and a count was taken to quantify these results. As shown in Fig. 6, twenty-five of fifty-one responses (49%) related to the first objective (How effective was EBPTO at helping you to apply evidence-based practices for teaching online?). Twenty-three out of fifty-one responses (45%) related to the second objective (How effective was EBPTO at helping you to prepare to build/facilitate an online course using the Online Course Standards Rubric as a guide?). And, eleven out of fifty-one responses (22%) related to the third objective (How effective was EBPTO at helping you to practice using UNM Learn tools to gain a student experience?).

Fig. 6

What was the most useful part (or parts) of this course?

There were twenty-four responses that fell into the Other category. Upon closer inspection, these responses aligned with three common themes: peer-learning, working with or support from an instructional designer, and exposure/practice with new tech tools as shown in Fig. 7. Of the twenty-four responses, thirteen of them related to exposure/practice with new tools, nine of them related to working with an instructional designer, and nine of them related to peer-learning.

Fig. 7

Additional themes that emerged from Question 2 of the End-of-Course survey (What was the most useful part (or parts) of this course?)

Question four asks instructors what they will do differently in their courses as a result of taking this course. Eighty-one responses related to either objective one or two. This aligns closely with the responses to question number two regarding the relationship to the course objectives, indicating that the most useful parts of the course were also the most applicable to their teaching/needs. For question four, the same process was followed as above in reviewing the open-ended comments as they pertained to each course learning objective. The counts shown in Fig. 8 indicate that of the fifty-one responses, thirty-nine (76%) related to the first objective, forty-two (82%) related to the second, and nine (18%) related to the third. Additionally, there were eight comments that fell into the ‘Other’ category, seven of which were related to new tools and one of which was related to working with an instructional designer.

Fig. 8

As a result of taking this course, what will you do different in your courses (either face-to-face or online)?

There were other themes that appeared in the comments, but in general comments seemed to reinforce that our course design and objectives, even when scaled for the pandemic response, were effective. The themes that fell outside of the scope of the course objectives are not unusual in terms of typical feedback in the end-of-course survey, but highlighted that these benefits of the course (working with an instructional designer, peer-learning, and discovery of new technology/tools) were also present when the course was scaled.

Implications

The data from the End-of-Course survey and mid-course journal entries show, on the whole, a positive response from participants in terms of the efficacy of the course in guiding participants to achieve the course learning objectives, and a diversity of views on online learning and educational technology that reflect the myriad of academic disciplines represented in EBPTO. It also shows that the existing course was able to be scaled to assist a large number of instructors moving instruction online in a short amount of time while maintaining the integrity of the course learning outcomes.

Data also suggest that an explicit and clear distinction between emergency remote teaching, distance education and hybrid courses, and self-paced eLearning products is important for educational institutions in general for policy, theory, and subsequent research.

Recommendations

The effectiveness of EBPTO, even at scale, has been greatly appreciated by UNM instructors and administrators. And, having a scalable model in place was beneficial to our instructional design team during the Summer of 2020 because we did not have to build from scratch to meet the dramatically increased need while also adapting to remote work and support ourselves. We recommend that other institutions adopt or create a course standards rubric for online courses and have a scalable, asynchronous delivery model in place for faculty development that supports instructors in making the transition to distance learning. The set of standards should be agreed upon either by a faculty body or in coordination with a group of experienced online educators, alongside online compliance experts, and teaching and learning support staff such as instructional designers.

We understand that not every institution has a large team of instructional designers that can share the workload of guiding faculty through a professional development course, but an institution can identify a team of experienced online instructors who could fulfill this role. When enrollments in the faculty development course are normalized (10–20 participants), experienced online faculty can rotate facilitation responsibilities. As a result, when an emergency takes place, a larger number of instructors or teaching and learning staff would be prepared to share the facilitation responsibilities to support the institution. We strongly discourage having one “expert” person responsible for this kind of work, even at a small institution. A larger group can make a significant impact by sharing the workload while not substantively altering their daily responsibilities, whereas a single person would need to stop all other responsibilities or provide less interaction with participants.

Future Research

The authors are interested in doing a follow-up survey with participants in the 2020 EBPTO to see what, if anything, they’ve continued to incorporate from the course after their emergency remote teaching experience. They are also interested in conducting a long-term analysis of the survey results from normal and high-enrollment EBPTOs to further ensure the model’s scalability. A follow-up study to examine social construction of knowledge in the courses’ online discussion forums would also lend credence to the authors’ argument that the highly interactive aspects of EBPTO contribute significantly to its effectiveness and should not be replaced by self-paced modules when the course is required to scale.

Conclusions

UNM’s Online Course Standards Rubric and the corresponding EBPTO course were not built with a crisis or emergency remote teaching in mind. They were, like so much faculty and instructional development, built to cultivate positive teaching and learning experiences for students. However, having these two collaboratively built and facilitated tools in place significantly impacted UNM’s ability to respond to the need for many courses to quickly pivot to either a remote teaching or online course format in 2020. Even though the number of participants were nearly quintuple the average enrollment, they were able to meet the learning outcomes and benefit from a highly interactive social constructivist learning environment. The authors hope other institutions can make use of this case to advocate for necessary resources, plan for online learning quality, and make adjustments to their current models to make them more flexible in the face of dramatic changes.