TriHy: teaching an MA TESOL class face-to-face, synchronously online, and asynchronously online.

This study investigated whether the success of students in a Master of Arts in Teaching English to Speakers of Other Languages (MA TESOL) assessment course was comparable regardless of their chosen mode of attendance (face-to-face, synchronously online, asynchronously online) in this "Triple Hybrid" (or "TriHy") class. In an interactive, convergent, mixed-methods design, a pragmatic, participant-focused framework guided the study. Data collection extended to pre-, while-, and post-surveys of the participants; tracking of mode of communication with the instructor; as well as proxies for students' success in the course, including the rate of course completion, weekly class attendance, completion of weekly assignments, grades on low-stakes individual assignments, grades on a high-stakes individual assignment, and a final course grade. The findings of the quantitative and qualitative analyses revealed that overall there was no statistically significant difference in the learning outcomes among the modalities even though one of the groups' pre-test scores did differ from the others' significantly. Although the students' success in the course did not differ, their perception of the factors that contributed to their success did. The findings suggest that with considerable institutional support, substantial investment of time and commitment from the instructor, and meaningful choices from the students, the quality of instruction even in a language-teacher-preparation course focused on skill building does not need to be compromised. Supplementary Information: The online version contains supplementary material available at 10.1007/s43545-022-00434-4.

Introduction and motivation for the study

For years before the SARS-CoV-2, or COVID-19, coronavirus pandemic, students wishing to study Teaching English to Speakers of Other Languages (TESOL) at the university in the Midwestern United States where one of the co-authors of this article teaches would frequently express a desire for courses, even entire degrees, to be available online. Some would do so for personal-professional convenience, others in response to the shifting geo-political-cultural climate. While some institutions worldwide have eagerly embraced online language-teacher education (OLTE), numerous others, such as this study’s site, have been hesitant to do so. OLTE has frequently been assumed to be inferior to in-person language-teacher education, primarily for fear of substandard learning outcomes.

This perception of program inferiority must be juxtaposed against the enrollment reality of the vast majority of language-teacher education programs. The once homogeneous student body, consisting of full-time on-campus students, had even in the years leading up to the pandemic fragmented into, on the one hand, students favoring in-person education and, on the other, those seeking online programs. Some language-teacher education programs have taken their entire degrees online, satisfying some of their students while losing others. Other programs have remained committed to in-person training, drawing on a decreasing pool of students. Another solution might be for universities to offer separate classes for the in-person and online cohorts if their student numbers warranted such a step. Dwindling enrollments in recent years, however, must spur language-teacher education programs to seek out innovative solutions or risk closing their doors at a time of a teacher shortage (Editorial Board 2021; Pawlewicz 2021; Prang 2022; Singer 2021).

This study was conceived in 2019 to take a small yet significant step toward allaying the fears of the opponents of OLTE on the grounds of inferior learning outcomes. The study was designed to investigate the learning outcomes of preservice of English as a Second Language (ESL) teachers at the master’s level. The MA cohort was selected for this proof of concept due to the lack of evidence for the efficacy of this model in the target setting and because this group of students had the greatest likelihood of success with this innovative approach. (Since the success of this initial attempt, the approach has been implemented more broadly: with undergraduate students as well and in a variety of TESOL classes.) The current study was conducted in a graduate language assessment course among three groups of students, those attending face-to-face (F2F), synchronously online (SO), and asynchronously online (ASO). The approach employed in this study will henceforth be called the Triple Hybrid (TriHy).

Literature review

Although in the scholarly literature studies investigating the efficacy of online education abound (Beatty 2019; DETA Center n.d.), there are only a small handful of publications that are relevant to the main research question that this study seeks to answer. Even among the relevant ones (as discussed below), some focus on information transfer rather than skill building, the latter being integral to language-teacher preparation in TESOL, such as at the site of the current study.

The following review of the literature serves two purposes. Primarily, in order to establish the gap that the current study fills, those publications are examined that are relevant to student success in foreign- or second language-teacher education in hybrid and online contexts. Secondarily, the design and interpretation of the current study have been informed by a broader circle of publications, which is also discussed. A clarification of key terms and an examination of how online education has been viewed in recent years precede the review of the aforementioned two broad areas of the literature.

Key terms

The most prevalent forms of computer-mediated communication are synchronous online (SO), when students are logged on in real time, and asynchronous (ASO) online, when they communicate online but are not logged on at the same time (Ko and Rossen 2017; Murray 2013; Nejad et al. 2021). A rapidly growing form of education is the hybrid-flexible (HyFlex) model championed by Beatty (2006). HyFlex is a hybrid of on-campus and online modes of delivery where students have the flexibility to participate, at their convenience, in any given class session F2F and some form of online (SO, ASO, or both).

Perceptions of online education

In light of the fact that online education continues to be viewed with suspicion in some settings, such as the current study’s site, a brief examination of perceptions of online education is warranted. According to Nunan, the first fully online master’s TESOL program was founded in the mid-1990s at Anaheim University (England 2012). Since then their numbers have exceeded forty (England 2012) and are continuing to grow. Online education in 2010, more broadly than TESOL, was already experiencing a growth rate ten times that of higher education in general, with 39% of all US college students (6.1 million) taking at least one online class (Allen and Seaman 2011). Fifty-nine percent of college presidents as compared to 29% of the American public viewed it favorably when compared to face-to-face education (Parker and Lenhart 2011). While lately overall enrollment in colleges and universities in the United States has been uneven, ebbing or flowing depending on various factors, the number of students involved in distance education has been increasing steadily (Seaman et al. 2018). This number since the pandemic has grown exponentially and distance education has taken center stage (Taparia 2020). Because the transition to online learning at the start of the pandemic was tumultuous, a survey of 1287 students revealed that 75.5% were dissatisfied with the quality of online education (OneClass 2020). As a return to in-person instruction becomes possible, faculty and administrators consider the way forward, in particular when a discipline is not easily conducive to online instruction (Zahneis 2022). Investigating the success of students in online settings is a key step in this direction.

Student success

The quality of online education is measured by the Online Language Consortium’s quality framework: the “Five Pillars of Online Education” describe the components comprising successful online learning: learning effectiveness, access, scale (achieved through cost-effectiveness and institutional commitment), faculty satisfaction, and student satisfaction (OLC n.d.). The foremost indicator of educational quality is learning effectiveness, which is the main focus of our inquiry. Numerous studies have investigated the learning outcomes of online (some SO, some ASO) and F2F students in various settings. Fishman et al. (2013), for instance, have found that both online and in-person groups made significant gains, in their investigation of the learning outcomes in the professional development of science teachers.

Rovai and Jordan (2004) compared sixty-eight K-121 teachers seeking an MA in Education through F2F, ASO, and blended (a hybrid of F2F and ASO) modalities. In this comparison of different courses, curricula, and instructors, the authors found that the ASO group reported the lowest levels of both connectedness and learning, the blended group outperformed the others in connectedness, while the F2F group excelled in learning outcomes. Technology and educational practices have advanced considerably in the decades since this publication. It would be rather uncommon nowadays for a F2F class not to include an online component as well, such as a discussion board or a video recording. Therefore, what the authors call blended is what is nowadays thought of as F2F. Furthermore, some of the critiques of the ASO class in the study reveal that it was entirely text-based. This hampered the ASO group’s ability to learn. Since the three classes were run independently of the others, the various modalities were not integrated with each other. Connectedness developed only in those classes where the students met F2F. As a result, the ASO students lacked both the opportunity to feel integrated and to learn according to their learning styles (beyond visual, also auditory and interactional). This study underscores the importance of intentional community building and that instruction needs to cater to various learning styles.

In a study of nonnative-English-speaking students, no significant differences in learning outcomes were yielded in terms of the frequency of negotiations of meaning (Moradi and Farvardin 2019). Although the study was conducted on English learners rather than their teachers, it is still relevant to the present context because language learning (as well as teaching) builds skills through frequent pair and small-group interaction, over a period of time as well as require formative feedback from the instructor. In this study, F2F and SO computer-mediated students’ interactions, in particular negotiation of meaning in dyads, were investigated. Although the frequency of negotiations in the two modalities were comparable, their quality did differ. The F2F dyads produced considerably more negotiation moves in less time. This is hardly surprising since the F2F dyads were able to speak to each other directly while the online groups typed their answers. Therefore, although this study is of interest because two modes of modality were compared in a foreign language class with no significant difference between the modalities in one regard, the study design is rather curious because one of the modalities was oral-aural while the other written. A more parallel design would have required the F2F students to communicate in writing as well.

Persistence

In critiquing the efficacy of online education, Xu and Jaggars point out that most studies ignore the potential impact of online delivery on course withdrawal, which depending on the institution can be over 30% (2013). In their study at a community college, the authors found that online course persistence was 3.6% lower, as compared to F2F. Among those who completed the course, the online grade was 0.19 points lower than among F2F students. Although we were able to find numerous studies concerning students withdrawing from online courses, we do concur with Xu and Jaggars that students’ persistence in online courses at community colleges are under-studied, where due to the competing demands on students’ time, this is of particular concern. Recognizing that the students in the current study lead busy lives, they were offered all three educational modalities to choose from.

Students’ persistence in other online educational settings, such as universities, is also frequently problematic. In addition to the students’ busy schedules, attrition can be caused by isolation as well. According to Nunan, students learning in isolation feel that they are “facing the world of learning alone” (England 2012, p. xi). Isolation may diminish motivation, which in an online course may go unnoticed by other students and the instructor (Hughes 2007; Moss and Pittaway 2013). Rovai and Jordan (2004) found that in their study comparing the motivation of F2F, ASO, and a blend of the two modalities, it was the F2F group that had the highest motivation in the class. Harker and Koutsantoni’s (2005) study compared students in a F2F (with blended online component) and online modalities. Although the students’ performance and satisfaction in the two groups were comparable, the persistence in the F2F group was significantly higher. The authors speculated that the online students lacked a sense of community, which the F2F students developed by virtue of being together in the same physical classroom. Similarly, in investigating F2F and ASO students, Johnson (2019) found that the former reported higher levels of motivation and expectations for grades. Motivation (as well as self-efficacy, self-directedness, independence, organization, interest in the subject matter, family support, and computer savvy) has been found to be an important precursor to success for online students (Irizarry 2002).

Finally, Liu and Rodriguez (2019) evaluated the impact of the HyFlex model on the learning outcomes of undergraduate business students who attended F2F, SO, and ASO over five consecutive semesters. The learning outcome by modality was not statistically significantly different. Unfortunately, Liu and Rodriguez were unable to ascertain that SO students did actually attend, through which modalities the students who dropped had attended, and how the grades of the disappeared students would have influenced the authors’ conclusions had they been included in the results. Despite these limitations and the fact that the course does not appear to have been focused on skill building the way language-teacher education does, this study is of interest because unlike other similar studies, it encompassed three modalities.

In order to prevent the aforementioned concerns over attrition, the researchers in the present study intentionally built communities of practice across the three modalities. In particular, they took great care that the ASO students feel integrated into the class, that they too would feel that classmates cared about them. This was achieved through the group project, where at least two modalities were present, and through assignments where students took turns serving as the lead on a task for the rest of the class (such as leading a discussion on the discussion board or in-class through a video recording).

Further aspects of students’ experiences

The present study’s design has been informed by a broad range of findings in the literature. These include students’ preferences for and against modalities, confusion and frustration in particular in ASO.

Modalities

It is important to understand students’ interest in the various modalities at their disposal. A focus on online education alone reveals the distinct advantages of this modality. It reaches students who would otherwise be inaccessible. Examples include students who cannot easily visit campus such as those with low-incidence impairments (Mills et al. 2009) and older, nontraditional students with work and family responsibilities (Johnson 2019). Distance education also facilitates the creation of a global community of learners to exchange ideas (Malczyk 2019). Similarly, Murray and Christison found that between ASO and SO, students were partial to the ASO configuration for ease of navigating time zones and family obligations while instructors preferred SO because it fosters interaction and the development of communities of practice (2017).

Multiple studies report on preferences for and against modalities. Nunan’s early study of MA TESOL students compared F2F and ASO (2002). The ASO students, who were ten years older than their F2F counterparts, appreciated the convenience of being able to remain in their home countries while pursuing their education but they, too, would have preferred a F2F education under different circumstances.

In studying language-teachers-in-training, Murray and Christison (2017) compared what attracts students to online classes according to the instructors versus the students themselves. The top five reasons are identical: from first to fifth reasons according to the faculty being flexibility, schedule conflict due to work and class, schedule conflict due to other reasons, studying at own pace, lack of F2F class option. Even the rankings were identical except that the students ranked the schedule conflict for other reasons fifth as opposed to the faculty’s third.

Confusion

A common concern in online courses includes confusion and frustration over lack of feedback from the professor (Carr 2000; Hara and Kling 2001; Rovai and Jordan 2004; Sikora and Carrol 2002). Unlike in a F2F setting, where the instructor can respond to a student’s quizzical look by offering clarification, in online education this is more challenging. SO students may find it more difficult to enter a conversation naturally due to lagging internet or a flurry of F2F discussion or the instructor not noticing their raised hand, etc. Leijon and Lundgren found that teachers often struggled with the design of interaction in the different physical and virtual spaces inhabited by the various modalities (2019). Thus the current study was designed with a combined online- and room-oriented approach. The instructor-researcher alternated in eliciting questions and contributions from the F2F and SO modalities. ASO students’ ideas were highlighted as part of the lesson as well.

Receiving timely clarification and feedback is even more of a concern for ASO students. A considerable amount of time may pass between when they may become confused or frustrated and when they receive clarification from the instructor or classmates. To minimize the potential for confusion and frustration among students, the instructor-researcher in the present study scheduled a synchronous appointment with all ASO students in the first week of the semester, devoted a considerable amount of time to answering questions promptly through various avenues, and encouraged the students to rely on their classmates as resource.

Online discussion

Online discussion boards have been found to be beneficial if executed well (Armstrong and Manson 2010; Beaudoin 2002; Davies and Graff 2005), for instance by the instructor modeling quality online interaction (Bergstresser 2013) and by setting up well-scaffolded outcome-oriented tasks (Verenikina et al. 2017). However, discussion boards have also been criticized for lacking true discussion (Abhyankar 2019), especially if the instructor’s evaluation hampers community building. Informed by the literature, our study was designed in such a way that the discussion board served as the primary sphere where the three modalities were integrated.

When examining faculty and student perceptions of building key elements of an online community, Vesely et al. (2007) concluded that students long for a strong instructor presence online, even if it is ASO. To this effect, the instructor-researcher of the current study engaged with the students’ contributions to the discussion. However, these were graded on effort rather than on quality to prevent a potential chilling effect on student contributions.

Summary of literature review

While discussion on student-learning outcomes in online higher education can be found in publications (Beatty 2006; DETA Center n.d.), the body of literature in TESOL or more broadly in language-teacher education is practically nonexistent. The publications that do exist, do not compare the success of TESOL students in all three modalities of attendance. Programs fearful of compromising their students’ learning outcomes by making online learning available need directly relevant data demonstrating that this does not need to be the case. Programs believing that their only option is to either compromise their students’ learning outcomes through online education or lose their programs due to dwindling in-person enrollment need to be convinced that on the one hand, this is a false dichotomy and on the other, that students’ success is comparable in the two types of remote learning (SO and ASO) when directly compared to face-to-face learning. This is the gap that the current study set out to fill.

Purpose, research questions, and objective

The purpose of the current study was to investigate whether, in a 15-week teacher-training course on second/foreign language assessment, the success of the students was comparable regardless of the modality of attendance. The construct of success in this context was measured by

the rate of course completion,

weekly class attendance,

completion of weekly assignments,

grades on low-stakes individual assignments,

grades on a high-stakes individual assignment, and

a final course grade.

In addition, the study attempted to capture various aspects of the students’ experiences, as suggested by the literature, including their preferred teaching modalities, motivation for attendance through their chosen modality, levels of clarity-confusion, perceived access to sources of support, aspects of practicality of course attendance, and student satisfaction.

The study thus attempted to answer the following primary (1) and secondary (2, 3, 4) research questions.

Is there a difference in the students’ success in the language assessment course depending on the modality of attendance?

Is there a difference in the students’ perception of the factors that are expected to contribute to their success based on the literature review?

Is there a difference in the students’ prediction/perception of their success?

Which sources of information and support do the students rely on in pursuit of success in the class?

The primary objective of the study was thus to investigate the success of MA TESOL students attending a language assessment teacher-training course through three educational modalities (face to face, synchronously online, and asynchronously online) by measuring markers of success—the rate of course completion, class attendance, completion of assignments, grades on individual low- and high-stakes assignments, and final course grade. A secondary objective was to identify key aspects of the students’ experiences as related to choice of modality (i.e., modality preference, motivation for chosen modality, level of clarity, perceived access to support, practicality of attendance, and satisfaction).

Methodology

Research design

We followed an interactive, convergent, mixed-methods design, in which we collected quantitative and qualitative data concurrently, analyzed them, merged the results, and interpreted the results so that each could inform the other. We collected the data using the same variables, constructs, and concepts (as described in Creswell and Creswell 2018). We were guided by a pragmatic, participant-focused framework, in that we made choices that may have weakened the study design to accommodate the participants’ preferences.

Procedure

The study was conducted from January through April of 2020. Students registered for the same section but prior to the first week were permitted to choose whether they would attend the class face to face (once a week in the evening), synchronously online (joining the face-to-face evening class), or asynchronously online. Once they selected a modality, they did not have the flexibility to make any changes. The instructor-researcher chose not to randomize student-participants to modalities because of the students’ strong preferences for or against certain modalities for reasons that will become apparent later in this article. From January through March students attended according to their chosen modality. By reason of force majeure, March 11, 2020 was the last in-person class meeting. From the next day until the end of the semester in April, the formerly face-to-face group joined the synchronous online group. Table 1 in Online Appendix E2 depicts a breakdown of participants and mode of class attendance before and after March 11.

Students were surveyed at three different times in the semester: at the start, at midterm, and at the end (see Online Appendix A). The initial and midterm surveys were administered prior to the coronavirus quarantine. The final survey was then revised to include a reflection on the semester prior to the quarantine and another one thereafter (see Online Appendix B). Students’ communication with the instructor was also tracked by modality (face to face, email, phone, text, and video conference), categorized as related to content or housekeeping, and analyzed for themes. After final course grades had been submitted, the instructor-researcher deidentified the data, shared it with the statistician-member of the research team and only then did the analysis commence.

Planned integration of the TriHy modality groups

The instructor intentionally fostered a class community. Rather than keeping the three modality groups separate from each other, she integrated them as much as possible from the design stage. To this end, each student had to write discussion questions for the class and lead the discussion if they were present in person or synchronously. The asynchronous students would take the lead on Canvas. They would publish their posts first and the remaining students would respond to their posts. A further attempt at integrating the groups happened through the culminating group project (designing a language assessment for a group of ESL students, administering it, analyzing the results, and writing it up in a research paper). Students formed heterogeneous modality groups: all the groups had representatives from at least two of the three modalities. Finally, the integration of the TriHy students was further facilitated by technology, which is explained in the next section.

Educational technology use

For the purposes of the study, the class utilized a variety of educational technology. The main platform was the Canvas learning management system (LMS), through which all students could access the textbook, weekly PowerPoint (PPT), supplemental reading and viewing material, weekly quizzes, the discussion board, instructions for the group project, announcements, and an email box for internal communication (Instructure n.d.). Prior to the pandemic-induced quarantine, a Blue Yeti Nano premium USB microphone, and a Logitech USB webcam were connected in the classroom to the instructor’s Dell XPS laptop. Panopto lecture capture, integrated into the LMS, was used to video record the lecture. The Panopto recording was launched directly from Canvas and captured the instructor’s laptop screen and the projector screen in the classroom. Meanwhile the PPT was displayed both on the instructor’s laptop and on the projector screen. All face-to-face and synchronous students joined the class on the instructor’s institutionally licensed Zoom video meeting platform. Their faces were visible on the instructor’s main screen as well. The in-class students’ individual devices (laptops, tablets, smartphones) were muted during the main class activities so as to prevent reverberation, as the instructor’s Yeti microphone and laptop were connected to the classroom’s projection system. For maximum integration of the in-class and online students, during pair and small-group activities, all students were randomized into breakout rooms on Zoom. Most often each group consisted of a mix of face-to-face and synchronous students. After March 11, once all students were either synchronous or asynchronous online, instead of Panopto, classes were recorded on Zoom with the microphone and camera built into the laptop. Class recordings—both Panopto and Zoom—were made available to all students through Canvas immediately after class.

Population and sample

The setting of this study was a graduate language assessment class in Teaching English to Speakers of Other Languages (TESOL) at a public university in the Midwestern United States. Study participants comprised the students in the course. Eighteen of the twenty students consented to the study. Table 2 depicts the demographic information of the participants by their pre-March 11 modality group.

Instrumentation

Students’ success was measured using a variety of instruments. At the start of the semester and at midterm, the original course survey was administered (Online Appendix A). During the last week of the semester, the revised course survey was administered (Online Appendix B). It was revised to capture any changes in students’ experiences resulting from the pandemic-induced lockdown. Online Appendix C depicts the anonymous survey where students could leave any comments about the class. Nobody took advantage of this mode of communication. In Online Appendix D, communication between instructor and students about the content of the class or housekeeping matters is covered. The former category contained questions, comments, and arguments about language assessment. The latter included communication about deadlines, division of labor within groups, homework submission, requests for clarification of assignments, etc.

Research questions versus instrumentation

Table 3 depicts a summary of data sources gathered to answer the research questions.

Results and analyses

Research question 1

Is there a difference in the students’ success in the language assessment course depending on the modality of attendance?

Result

No. There was a sig. difference on the pre-test (Quiz 1) and on the percentage change from the pre-test to the final exam, but not on any criteria evaluating the learning outcomes of the class.

Analysis method: quantitative

Rate of course completion: 100% for all three groups.

Weekly class attendance: 100% for all three groups. F2F and SO students’ attendance was tracked by their presence during class time. ASO students’ attendance was tracked by their presence on Canvas.

Grades on low-stakes individual assignments—These assignments were considered low stakes because they did not contribute much toward the final grade. There were no sig. diff. among the scores by group.

Grades on high-stakes individual assignments—The only high-stakes individual assignment was the cumulative final exam. Table 6 depicts the median final-exam scores by pre- and post-March 11 modality groupings. For comparison purposes, the pre-test’s (Quiz 1) scores are also displayed and percentage change calculated from the pre-test to the final exam. There is no sig. diff. by modality in the final-exam scores. However, the SOa group started the class with sig. less knowledge than the ASO and F2F groups. Looking at the percentage change, the SOa group had a sig. higher gain than did the ASO group.

Final course grade—There was no sig. diff. in the participants’ final course scores. Final grades are shown in Table 8. In summary, sig. diff. were found for changes in scores (pre-test to the final exam), not for differences at the final exam (see Table 7).

Semester-long group project (both individual and group parts)—From the semester-long group project, no conclusions could be drawn about differences in learning outcomes based on the participants’ modality of class attendance due to the deliberate mixing across groups.

Qualitative

Alongside quantitative data, qualitative data were also gathered. Sources included the individual Canvas discussion-board posts (throughout the semester) as well as the semester-long group project, the individual reflection part of the group project, and the short-answer questions on the final exam.

Upon entering the course, the vast majority of the students recognized basic assessment terms such as formative and summative and agreed that they disliked assessment. They knew enough about the course topic to hold uncomplicated yet definite views about assessment, as illustrated in the following: “I use formative assessment daily, and I feel strongly—very—that all state standardized tests should be taken to a ‘galaxy far, far away.’ The standardized testing system is not conducive to EL [English learner] students and sets them up for failure” (participant 13, SOa); “As far as assessment goes, I feel that it does have its place, but it is not my favorite part of being a teacher. At present, I view it as a necessary evil, as somewhat of a burden” (participant 8, F2F). Only one of the participants entered the course with a somewhat sophisticated understanding of assessment issues.

Thus both qualitative and quantitative data suggest a broad range in the participants’ incoming knowledge of the concepts, which did not divide along modality lines. The only discernible pattern in the participants’ understanding of the course content at the start of the semester appeared to be that several of the K-12 teachers in the program were negatively predisposed to assessment.

By the end of the semester—revealed by the weekly discussions, the final exam, and the individual and group portions of the semester project—the participants appeared to have gained a more layered and deeper understanding of assessment concepts. Their understanding went beyond a recognition of basic terms and into a deep understanding of numerous concepts; a creation of a test blueprint; the crafting of a test with selected-response, constructed-response, and authentic item types; setting a test form; piloting the test; administering the test; running basic statistical calculations; interpreting the results; and improving their assessment based on their findings. Course participants were able to argue that without assessment, one could not tell if their learners were learning.

The following comment is a representative demonstration of their learning. Participant 1 (F2F) wrote the following:In summary, course participants may not have become enthusiasts of high-stakes standardized tests and their impact, but they developed a far more nuanced understanding of assessment issues.

I can see your situation is a little tricky. One thing you might consider is to give students pre and after writing tests, grade them based on the same rubric, and then conduct [a] statistical analysis to see if there is a significant difference in scores between the two tests. If there is, it could tell you that students have got progress in writing skills; otherwise you may consider reteaching lessons.

Research question 2

Is there a difference in the students’ perception of the factors that are expected to contribute to their success based on the literature review?

Yes, there is, depending on the survey question. Overall, before March 11, the F2F group agreed sig. more than the SO group that their modality of attendance allowed them to retain information, to process information, to be well informed, to be part of the class community, and to have access to the instructor. Prior to March 11, the ASO group agreed sig. more that their modality of attendance afforded them cost-effectiveness, time-efficiency, and convenience. After March 11, the ASO group scored sig. higher than the F2F group for cost-effectiveness, but lower for being better informed, being part of the class community, access to the instructor, and access to classmates (see Table 10). Several sig. changes were observed in the responses of F2F participants after March 11: attending online due to the pandemic increased the convenience of attendance but decreased their ability to retain and process information, be informed, feel part of the class community, and have access to classmates and the instructor (see Table 11).

Analysis method

Students were surveyed for their opinions at four time points (‘pre’, ‘mid’, ‘post part 1’ and ‘post part 2’).3 At each time point, there were 14 questions (2 and 3 ‘a’ through ‘m’). In survey question 2, participants selected from among categorical variables (F2F, ASO, SO) (see Table 9). In survey question 3, all questions were answered on a Likert scale (1–5, with 5 being the most favorable rating). For the first analysis, median scores were compared across groups within time points (e.g., question ‘pre3a’ across the pre- and post-March 11 modality groups) (see Table 10). For the second analysis, median change scores were compared across groups (e.g., the change in responses to question ‘pre3a’ to ‘Post_Part13a’, across the pre- and post-March 11 modality groups) (see Table 11). Pre-March 11, nine students attended F2F, five attended SOa, and four attended ASO. Post-March 11, fourteen students attended SOb (‘synchronous’) and four attended ASO (‘asynchronous’).

Research question 3

Is there a difference in the students’ prediction/perception of their success?

On questions 3d, 3L, and 3m, there was no sig. diff. among the median ratings on the pre-, mid-, post-1, and post-2 surveys.

Survey question 3k: Attending this class in the way that I’m attending will allow me to be as successful in this class as I can be … 1 / 2 / 3 / 4 / 5.

On the post-course survey-1, of the pre-March 11 modality groups, the F2F median ratings were sig. higher than the ASO ratings. Of the post-March 11 modality groups, the SOb median ratings were sig. higher than the ASO ratings.

Until March 11, the F2F group rated their ability to meet deadlines very highly.

There were sig. diff. in median change scores for Post_Part1_to_Post_Part2_3k. The median change for the F2F group was sig. lower from that for the SOa group (the F2F had a decline, while the median change for the SOa group was 0).

Survey question 3L: Attending this class in the way that I’m attending will result in the best scores on assessments … 1 / 2 / 3 / 4 / 5

There were sig. diff. in median change scores for Post_Part1_to_Post_Part2_3L among the three groups, but there was not a sig. diff. between any pairs of groups (see ‘pairwise tests’ in the next paragraph).

Changes for the post-March 11 modality groups: The two groups were the synchronous and asynchronous. When comparing the changes in median scores from the first to the second part of the post-class survey, there were sig. diff. changes for the following survey questions:

Attending this class in the way that I am currently attending since COVID-19

3g ‘allows me to be the best informed’: the ASO group’s median rating did not change; the SO group’s median rating declined.

3h’allows me to be most part of the class community’: the ASO group’s median rating increased; the SO group’s median rating declined.

3i ‘gives me the best access to the instructor’: the ASO group’s median rating did not change; the SO group’s median rating declined’

3j ‘gives me the best access to my classmates’: the ASO group’s median rating increased; the SO group’s median rating declined.

Analysis method

Software used: SPSS (version 26) was used for tables and normal plots. Python (3.7.4), running in Jupyter Notebook (6.0.3) was used for testing between group differences. The between group differences were analyzed using the package’s statsmodels (0.10.1), scipy (1.4.1) and scikit-posthoc (0.6.6). The median, rather than mean, scores were calculated because of the small sample sizes and frequently skewed data with outliers. In such cases, the median becomes more representative of the sample. The group differences were tested initially with the Kruskal-Wallace test, using unadjusted alpha = 0.05. For the pre-March 11 (pre-COVID) modality groups, pairwise tests were done for the three groups using the Dunn test with a family-wise type alpha = 0.05 if the overall test was significant. For the post-March 11 (post-COVID) modality groups no such tests were performed, since there were only two groups. In addition, histograms, boxplots, and bar charts (for the mean) were done by modality group.

Research question 4

Which sources of information and support do the students rely on in pursuit of success in the class?

The participants relied on numerous sources of information and support (see Table 12): All students took advantage of the textbook and the PowerPoint lecture slides. All F2F and SO participants had perfect attendance and all but one of them also reported having relied on their classmates and additional readings. All four of the ASO participants (as well as one SO participant) relied on the video recording of the class. Other sources of information included searching on the internet and communicating with the instructor.

Communications rates with the instructor were further tracked throughout the semester. Analysis of the results reveals no sig. diff. in the communication rates, but one sig. diff. for the change in rates. The SO group had a sig. higher percentage change in communication rate from pre-test to final exam, compared to the ASO group (see Table 13, ‘Communication Rates’).

On post-survey2 in question 5, participants were asked to select from a list of sources of information and support which they availed themselves of during the semester. An “other” option was also provided. In addition, communication with the professor was tracked. Any time a participant communicated with the instructor during the semester about the course or content, the instructor made a note of the name of the student (later deidentified), the modality of communication, and the content of the communication (for the blank form, reference Online Appendix D). The modalities of communication included F2F after class, SO after class, email, Canvas email, separately scheduled video conference, telephone call, text message, and F2F office visit. The types of communication were subdivided into “content” and “housekeeping,” as explained above. Originally, all parties in the communication were tracked. Eventually during the analysis, only the spokespeople were counted and bystanders and copied students were excluded. One communicative event was defined as a question, comment, or complaint initiated by a student to which the instructor (and possibly other students) responded. This method was chosen for ease of categorization and analysis. The authors acknowledge that the way the communicative events were defined and counted may have resulted in in an under- or overcount. Had bystanders been included, it would have added a large amount of weight to the F2F and SO groups since they always had bystanders. Furthermore, it is worth noting that because the number of sessions before and after March 11 was different, the change in communication rates was calculated by taking the number of communications and dividing it by the number of sessions.

Additional findings

Participants had the chance to leave open-ended comments at the end of each of the pre-, mid-, and post-surveys. Table 14 depicts a summary of their responses: most participants who left comments in the surveys at the beginning, middle, and end of the course emphasized their motivation for attendance through their specific educational modality. Of the opinions expressed, fourteen were in support of F2F attendance: all of them due to learning-style preferences. One representative comment follows:Most of those who preferred F2F attendance were, indeed, taking the class F2F but some were taking it SO or ASO due to personal, family, work, or other reasons. In addition to those participants who were forced online for practical reasons, one participant clearly learns better online, as illustrated by the following comment:

It would have been more convenient, cost- and time-efficient for me to take this course online, as I would not have to commute in bad weather, but I chose the face-to-face option because I believe that I benefit more from this type of instruction and interaction with the professor and peers. (Participant 11, F2F, pre-survey)

Asynchronously online allow[s] me to pause and go back to a lecture, allowing me to learn better. The idea of being able to view the lecture on a time table that works for my life, is a HUGE DRAW. Most of us are not 18 year[s] old live[ing] on-campus: We manage children, aging parents, a full time job, and a home (and if you’re like me single parenthood). This format allows me to further my education.

Discussion

Learning outcomes, modalities, persistence, information sources, and teaching

The study investigated whether the MA TESOL students’ success was comparable regardless of their chosen mode of TriHy class attendance. From the interactive, convergent, mixed-methods design, where quantitative and qualitative data were gathered concurrently (Creswell and Creswell 2018), we found that the students’ success in the course did not differ by modality of attendance. These findings are discrepant from the findings of Rovai and Jordan (2004) and Inglis et al. (2011) but align with the findings of Fishman et al. (2013), Liu and Rodriguez (2019), Moradi and Farvardin (2019), and Siczek and Stanchevici (2019). Our course proved to be an equalizer, as the post-test scores no longer differed by modality. Although pre-test scores revealed a sig. diff. in the incoming knowledge of the students’ of course content, where one of the modality groups had a sig. lower score than the other two groups, by the end of the semester, all the students were able to display a nuanced understanding of the complexities of second-language assessment. This is an important finding at a time when many in academia are contemplating a forced return to F2F instruction due to the perceived inferiority of online education (Donaldson and Long 2022; Gluckman 2022).

Preferences for and against modalities

The aforementioned findings, however, must be nuanced by subsequent findings: students in the various modality groups, differed sig. in their perception of the factors that contributed to their success. Pre-pandemic, the F2F group credited their chosen attendance modality for their ability to be successful: specifically, their ability to learn effectively (retain and process information), meet deadlines, be well informed, belong to the class community, and have access to the classmates and professor, as well as earn the highest scores. F2F students did not rate their attendance as convenient, cost-effective, and time-efficient, as some had to drive an hour to campus after a day of full-time work. Nevertheless, it was worth the students’ effort because F2F attendance was most compatible with their learning style. During the pandemic, the formerly F2F group’s evaluation of all of the above evaluation criteria dropped sig. They perceived the effect of being forced into a non-preferred modality negatively and stated that if they were to take this class again, they would prefer to do so F2F.

The F2F students were strong proponents of the F2F option while many of the online students felt equally strongly about the need for distance learning. Students who attended online throughout the semester emphasized that without the SO and ASO options, they would not have been able to pursue their studies due to work, family, and other obligations. They cited convenience and efficiency as the primary reasons for their choice of modality. The only exception was an ASO student whose learning style aligned with the ASO modality. This student appreciated being able to process the information at their own pace by pausing the recorded lectures at will.

In summary, students’ preferences for an attendance modality varied greatly. What mattered to the students and what enhanced their success in the course was the appropriate match between their preferences and the modality of attendance.

Persistence

Persistence is an important marker of student success. In this course everyone had perfect attendance and completed the course. These findings are inconsistent with those of Carr (2000), Sikora and Carrol (2002), Hara and Kling (2001), Rovai and Jordan (2004), who experienced attrition. Our students’ persistence can be attributed to their small number, the intentional course design, and supportive teaching. The instructor strove to consider the needs of each group of students in selecting instructional technology, considered the TriHy class architecture when teaching, intentionally elicited contributions from the SO group throughout each class period, conferenced with students individually in preparation for their quiz-design and discussion-design assignments, and connected the students through assignments. The design of the discussion board aimed to integrate the TriHy modalities. It served as a space for knowledge exchange and community building (Steadman 2020). Even though the instructor did have a strong online presence, as recommended by Vesely et al. (2007), she did not grade the posts for quality so as to avoid the chilling effect on contributions, as reported in the literature (Steadman 2020). We also attribute the students’ persistence in the course to close interpersonal contact. After the March-11 pandemic-induced lockdown, several of the SOb (formerly F2F) students might have dropped out had it not been for the psychological support provided by their cohort colleagues and instructor. The maintenance of online students’ motivation is a well-researched phenomenon (Hughes 2007; Irizarry 2002; Johnson 2019; Moss and Pittaway 2013). What is noteworthy, however, is that—unlike Means and Neiler (2020)—in the present study we experienced no dip in the motivation of the SOa and ASO students from pre-pandemic to post-pandemic. Rather it was the motivation of the formerly F2F students that plummeted and only post-March 11. While distinct preferences for and against modalities have been reported in the literature (Inglis et al. 2011; Malczyk 2019; Murray and Christison 2017; Nunan 2002), we are able to offer a unique perspective in that the same group of students experienced the same course through both a preferred and dispreferred modalities during the study.

Sources of information and communication types

Albeit no sig. diff. were observed by modality group, interesting patterns did emerge. In addition to the predictable sources of information (textbook, PPT, recording of lecture, classmates), the vast majority of the students also took advantage of the opportunity to contact the instructor freely with both housekeeping and content-related questions and concerns. There were more exchanges about housekeeping matters (such as deadlines and homework) overall by the class, the ASO group, and the SO group. The only group that communicated more about content than about housekeeping was the F2F group and even they did so only while they were attending in person. Once they went online, they too communicated more about housekeeping matters. This insight makes a novel contribution to the scholarly literature, as we are not aware of any other research findings pointing to the content of communication of a group of students before and during the pandemic. What can only be hypothesized though is the reason for the prevalence of communication about course content by the F2F group prior to March 11 in contrast to the other groups and other times. The make-up of the F2F group consisting primarily of similar demographic (non-US-K-12-bound) and learning style (favoring F2F) may have been reasons. Another reason may have been that the atmosphere of the live class was conducive to discussion and exploration of content, which continued after class. Seeing the instructor in person may have made the instructor more approachable as well. Queuing up at the teacher’s podium and approaching the instructor in person can be done with more privacy than on a Zoom call with others listening in. One could view this finding from a different perspective as well: maybe the F2F group pre-March 11 did not so much engage with content more as with housekeeping less. Perhaps housekeeping matters—deadlines, assignments, expectations, etc.—were clearer in person than online after March 11. The other groups interacted with the instructor only through technology and thus always had a steady stream of housekeeping concerns on their minds. Any or all of these may have been reasons for the observed pattern of communication with the instructor though certainty cannot be offered at this time.

From a teaching perspective

Hall and Knox (2009) aptly point out that the additional time and effort required to teach an online course often goes unnoticed by the administration and our study confirms this claim: figuring out the technology, the hardware, the software, the layout of the room, where the instructor stands and looks, the order in which the instructor engages with the students of various modalities, interaction with and among the students. Requirements of the instructor included considerable advance planning, additional effort during the class, after the class exporting and processing the recording, and finally uploading the recording to Canvas. And this is uncompensated time and effort. The instructor needed to develop the technical expertise to execute this. It entailed several meetings with the e-learning unit on-campus, recruiting them to come to the class, solving reverberation issues. There was a lot of additional training on technology prior to the semester and during the first few sessions so as to capitalize on the advantages offered by the TriHy model rather than recreating the F2F interactions online (Skulmowski and Rey 2020). In our study the implementation of the TriHy model with a graduate class in the middle of their course of study was planned with considerable forethought. Others who had to move their instruction online suddenly due to the pandemic were less fortunate (see Fox et al. 2020, for post-secondary in the US; Kim and Asbury 2020, for K-12 in the UK; Majoundar 2020, for K-12 in the US; Skulmowski and Rey 2020, for post-secondary in Germany). This debate, unfortunately, is being unfairly categorized in terms of whether or not online teaching works. The reality is more subtle: if an institution is willing to invest a lot of resources, provide extensive faculty support and keep workloads manageable, and give students choices about their preferred form of learning, then it is possible to make the complex system that we are calling TriHy teaching into an effective class.

Limitations, recommendations, conclusion

This study has the following limitations. The sample size (18 participants out of a class of 20) was small. Spurious effects due to grouping were kept to a minimum, but the students were not randomized into attendance modalities. The results of statistical calculations were confounded by demographics because most K-12 teachers self-selected into one of the online groups while most non-US-K-12-bound teachers self-selected into the F2F group. Although this was a limitation, it was also a strength. It was a pedagogically sound decision, and the social interaction hopefully trimmed the causal effects down to the bare effects of the modality. Missing survey data occurred due to the instructor-researcher’s desire to avoid instructor bias. To this effect, one author did not examine the completed surveys until after grades were posted following the end of the semester; thus, any missing data were not detected in time to draw the individual students’ attention to them. There were two missing responses out of 936 total; they were ignored in all analyses. Nevertheless, we are publishing the results of this study with the hope that others will also come forward and contribute to the conversation. Pooling comparable studies and file-drawer results would enable us to perform a meta-analysis.

Conclusions, recommendations

The pandemic merely accelerated the need for a change in language-teacher education and has underscored the need for reimagined instructional models. In our study we took a step in this direction with the modality that we have named Triple Hybrid, or TriHy. The Triple Hybrid model is highly consistent with responding to the concerns identified in the introduction: a fragmentation of students seeking language-teacher education into those who continue to prefer face-to-face attendance, those who prefer to learn synchronously online, and those who prefer the asynchronous modality. It is not necessary for programs to cater to just one of these groups and risk losing students—future language teachers—at a time of severe teacher shortages, when they can meet all the three groups’ needs in one cohort. The encouraging findings of the current—albeit small—study have demonstrated that the student-learning outcomes do not need to be compromised in an attempt to meet the students’ changing needs.

TriHy gives students the flexibility to attend according to their preference but also the stability for the instructor to plan ahead. This way the professor has slightly more control of the way the lesson transpires. Instructional quality can be achieved with the proper integration of technology into language-teacher education and intentional community building among the three groups of students.

We also recommend that flexible models such as TriHy or HyFlex be extended to language education. Recent publications on ESL and EFL students are encouraging about the effectiveness of hybrid foreign- and second-language instruction (Moradi and Farvardin 2019; Ross and DiSalvo 2020; Siczek and Stanchevici 2019).

Changing enrollment patterns and the post-pandemic ‘new normal’ necessitate that higher educational institutions attempt to meet students’ needs by offering flexible attendance modalities and instructors’ needs through institutional support.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 48 kb)