A silver lining for pandemic-weary libraries: How blended and flipped instructional programs have improved upon pre-pandemic norms.

The COVID-19 pandemic presented instructors and learners with novel challenges related to the delivery and consumption of instructional content. Within academic libraries, these changes have resulted in an expanded reliance on asynchronous learning content delivered through a variety of instructional approaches. The study described herein was designed to evaluate the effectiveness of a blended and flipped program of information literacy (IL) instruction. Deploying a pre/post-test methodology within a quasi-experimental study of student learning outcomes, this research compares achievement across two cohorts of students receiving IL instruction as part of a first-year writing program (FYWP) requirement. While one cohort participated in a multi-shot, in-person, synchronous instructional program during the pre-pandemic period, another cohort participated in a blended and flipped instructional program delivered under pandemic-related restrictions. Comparative analysis revealed net positive achievement outcomes for both the pre-pandemic and pandemic cohorts, as well as interesting points of contrast between the two groups. In particular, students in the pandemic cohort arrived to their first year with less research experience than their pre-pandemic peers-signaling a gap in knowledge that pandemic-era library instruction must overcome. Fortunately, the blended and flipped instructional program experienced by the pandemic cohort proved successful in helping those students overcome their research experience gap, while delivering additional benefits that help confirm previous research in this area of practice.

It is difficult to identify any facet of daily life the COVID-19 pandemic1 did not affect. The changes precipitated by the pandemic, in and of themselves, were received with a host of negative emotions. Life can be complex and difficult, and routine is perhaps the most prolific tool used to manage life's uncertainty. Unfortunately, the pandemic, and humanity's response, upended all routine. Humans could no longer find comfort in familiar spaces such as libraries. Instead, novel solutions to novel problems were needed—and familiar solutions no longer held much weight.

While disconcerting, the unfamiliar environment of pandemic afforded opportunities for growth and progress. In response to the pandemic,2 academic libraries have expanded their use of non-traditional instructional models that utilize some level of virtual content delivery. For libraries with existing instructional assessment programs ready to pivot, these changes presented an opportunity to better understand the effectiveness of various instructional models.

At its core, this study was designed to evaluate the effectiveness of a blended learning model of information literacy (IL) instruction. Deploying a pre/post-test methodology within a quasi-experimental study of student learning outcomes, this research compares achievement across two cohorts of students receiving IL instruction as part of a first-year writing program (FYWP) requirement. While findings show net positive achievement gains for both the pre-pandemic and pandemic cohorts, comparative analysis revealed several points of contrast between these two groups. In particular, students in the pandemic cohort were coming into instruction with less research experience than their pre-pandemic peers—signaling a gap in knowledge that may be a result of pandemic-related learning loss during their senior year of high school.

Literature review

Online learning (or e-learning) is not a particularly new concept, emerging during the late 1970s and early 1980s as the modern answer to mail-based correspondence programs of yesteryear (Harasim, 2000). Then, with the birth and expansion of the World Wide Web during the late 1980s and throughout the 1990s, online education was poised to enter the mainstream (Allen & Seaman, 2011; Harasim, 2000). However, outside of for-profit institutions like the University of Phoenix, which launched their 100 % online programs in 1989, this mode of instruction in higher education remained relatively little used until well into the second decade of the 21st century (Allen & Seaman, 2015). Even then, a vast majority of enrollment growth in this sector is attributable to students with majority in-person course loads3 (Allen & Seaman, 2015). Of course, this changed during the pandemic, which forced many institutions to implement an unplanned move to 100 % remote learning for their students.

The shift to online learning during the pandemic was prolific and expansive, as evidenced by the host of recent research publications (e.g., Allam et al., 2020; Al-Baadani & Abbas, 2020; Ezell, 2021; Ives, 2021; Kurlanska, 2022, etc.). What's more, the abrupt nature of this transition, coupled with pandemic-induced stress brought on by sudden life changes, presented novel challenges for both instructors and students alike (Bacher-Hicks & Goodman, 2021; Browning et al., 2021; Coyne et al., 2020). While instructors were forced to learn/adopt new instructional technologies, students were faced with a completely new learning environment that lacked many of the social elements to which they had become accustomed (Tan et al., 2021).

Learning loss, as a result of instructional changes and interruptions during the pandemic, has been of particular concern to educators (Boulay & McChesney, 2021; Engzell et al., 2021). Citing previous research on student learning loss during summer months, many anticipate pandemic-related factors will lead to significant setbacks for students (Engzell et al., 2021; Harmey & Moss, 2021). Such factors include: lack of classroom-bolstered engagement (Zaccoletti et al., 2020), limitations for time-on-task (Andrew et al., 2020; Grätz & Lipps, 2021), as well as difficulties navigating instructional technologies for both instructors and students (Engzell et al., 2021; Schleicher, 2020). In response, educators have turned to the established literature on topics such as student success and effective instructional practice in the virtual environment.

Pedagogy and online learning

Educators have noted a variety of challenges while navigating the paradigmatic shift toward online learning (Guidera, 2003; Harasim, 2000; Raes et al., 2020). Most notably, this includes challenges associated with both the effective deployment of technology in support of instructional goals, as well as best pedagogical practices within the e-learning environment (Guidera, 2003). In both cases, educator goals most often center on the promotion and maintenance of learner engagement/motivation, autonomy, and general academic success (Ameloot et al., 2022; de Brito Lima et al., 2021; Günes & Alagözlü, 2021). Of course, as indicators of instructional quality, these elements are not unique to the e-learning environment—they are simply indicators of effective instructional practice.

The research of Chickering and Gamson (1991) and Chickering et al. (1987), outlining seven principles4 of effective instructional practice, has proven incredibly influential within education research and practice. Examining these principles within the context of e-learning, Guidera (2003) noted that “online instruction is less effective at interactions both between faculty and students…as well as between students themselves…” (p. 164). In response, many educators have sought to bolster engagement and interactivity through flipped, hybrid, or blended learning approaches (Balakrishnan et al., 2021; de Brito Lima et al., 2021; Ng, 2018).

When discussing flipped classrooms, hybrid learning, and blended learning, confusion can arise due to the fact that these terms are often used interchangeably by educators (Balakrishnan et al., 2021). Nevertheless, there are import distinctions worth noting. The flipped classroom is a method of educational program delivery whereby new course content is introduced to students through homework assignments, with subsequent time in the classroom used for student-centered, active learning activities that put said content into practice (Balakrishnan et al., 2021; Findlay-Thompson & Mombourquette, 2014; Strayer, 2012). At its core the flipped classroom aims to increase supervised time-on-task, creating a more structured approach to the development of competencies.

Blended learning focuses on the thoughtful deployment of multiple instructional modes, specifically face-to-face and computer-mediated instruction (Garrison & Vaughan, 2008; Obyrne et al., as cited by Balakrishnan et al., 2021). As noted by Stein and Graham (2020), the computer-mediated content delivered within a blended learning program can be either synchronous or asynchronous. Further, Garrison and Vaughan (2008) point out, “[t]he basic principle is that face-to-face oral communication and online written communication are optimally integrated such that the strengths of each are blended into a unique learning experience congruent with the context and intended educational purpose” (p. 5).

Although blended and hybrid learning have been used interchangeably within research literature, some researchers draw a clear distinction between these two concepts. Specifically, hybrid learning has been defined as the synchronous delivery of course content to two groups of students—one that attends class in person, while another attends class virtually (Raes et al., 2020; Sanpanich, 2021). Providing additional clarity, Lakhal et al. (2017) highlight a variety of reasons why a non-traditional instructional approach might be taken. When said approach is employed toward some pedagogical end, the term “blended” is most appropriate. Alternatively, when the approach is adopted in service to student choice, the term “hybrid” is most appropriate. Speaking to this point, Raes et al. (2020) note, the broader movement of higher education toward hybrid course offerings aims to make education “less dependent on location and time [to] improve flexibility within the learning trajectory” (p. 143).

Assessing student learning outcomes

No matter the mode of delivery or tools used, student success is the central metric of import when judging pedagogical viability of any instructional approach. As noted by Millea et al. (2018), this metric is defined by a variety of other measures that speak to success at either the individual and/or institutional levels of analysis. For example, while course or assignment grades act as measures of success at the individual learner level, metrics like academic persistence or time-to-graduation can be used to define student success at the institutional level (Kim et al., 2010; Kuh et al., 2010; Millea et al., 2018; Walker & Whitver, 2020). What's more, researchers have identified “a host of personal, social, environmental, and institutional factors that can impact student success” (Walker & Whitver, 2020, p. 2).

As one might expect, the library's impact on student success has been assessed using both institutional and individual-level metrics. At the institutional level, several studies have investigated the library's effect on both grade point average (GPA) and student retention. For example, Gaha et al. (2018) found that graduating students, who were enrolled in classes that attended at least one library instruction session, had higher GPAs than those students who were not enrolled in such classes. Similarly, the research of both Wright (2021), as well as that of Rowe et al. (2021), found that students who attended library instruction were more likely to have a higher GPA and be retained by their university.

While institutional-level metrics such as GPA and retention provide readily understandable measures of student success, they are less than ideal for measuring student learning. As noted by Suskie (2009), these types of metrics act as indirect proxy measures of student learning that are less convincing than direct measures. For this reason, library researchers have employed pre/post-test assessment programs to study the impact of library instruction through a direct measure of student learning.5 What's more, these programs typically focus on assessing student learning in relation to those threshold concepts represented within the Association of College and Research Libraries (ACRL) Framework for Information Literacy for Higher Education.6

Several studies have noted statistically significant improvements from pre to post-test for students enrolled in librarian-taught IL instruction. However, the impact and importance of these findings vary in relation to the methods employed. Although pre/post-tests can support direct measures of student learning, they can also be used to gather indirect measures7 —as is the case with Grigg and Dale (2017) as well as Kennedy and Gruber (2020). On the other hand, research like that of Price et al. (2011) uses the pre/post-test methodology to measure student learning directly via questions that test students' IL competencies.

Walker and Whitver (2020) took this direct measure approach one step farther by coupling a similar set of IL competency questions with other question sets aimed at collecting data on a variety of peripheral factors8 tied to student success. In the case of both Price et al. (2011) and Walker and Whitver (2020), statistically significant improvement in IL competency from pre to post-test was found. Additionally, Walker and Whitver (2020) found that several peripheral factors 9 affected student success relative to pre/post-test scores. In particular, a student's previous research experience had a positive effect on pre-test score, while librarian teaching effectiveness and a course instructor's reinforcement of IL concepts outside of library instruction had a positive effect on post-test score.

Student success across modes

Blended instructional programs, especially those utilizing a flipped classroom component, are of particular interest to the current study. Previous research has identified ways in which student and faculty perceptions of quality and effectiveness can differ across traditional classroom-based instruction and virtual instructional models (Tan et al., 2021). For example, researchers have noted significant differences in students' motivation, learning achievement, as well as relatedness toward instructors and peers across various instructional modes.

It has been frequently noted that relatedness between students as peers, as well as between students and their instructors is important to persistence and the achievement of desired learning outcomes (Bower et al., 2015; Chickering & Gamson, 1991; Kuh et al., 2010; Raes et al., 2020). While some research has shown relatedness is hampered for students learning in the virtual realm (Raes et al., 2020), others have noted the design and thoughtful deployment of virtual instruction can temper this shortcoming (Strayer, 2012; Wang et al., 2021). For example, employing a blended rather than exclusively virtual approach has often proven an effective strategy as has the deployment of flipped classroom instructional elements.

Blended synchronous learning, as well as flipped approaches, have been shown to promote active learning (Bower et al., 2015; Kakarougkas & Abdellatif, 2022), which is a key component of effective instructional practice (Chickering & Gamson, 1991; Kuh et al., 2010). Günes and Alagözlü (2021) found that autonomy, motivation, and academic success were all higher in the blended learning environment than in an exclusively asynchronous virtual environment. Further, these findings are in agreement with those of Raes et al. (2020), which suggest that students engaged in asynchronous virtual course content are less motivated and less effective in achieving positive learning outcomes when compared with students learning within a blended instructional program.

Highly integral to motivation, self-efficacy is an important pre-cursor to student engagement, achievement, and success (Bandura, 1986; Schunk & Mullen, 2012). Self-efficacy has also been linked, specifically, with success in the realm of online learning (Shen et al., 2013). Bandura (1986) defines self-efficacy as one's belief in their own ability to successfully perform some task or attain some goal. “Students with high self-efficacy tend to set challenging goals, work diligently, persist in the face of failure, and recover their sense of self-efficacy after setbacks. As a consequence, they develop higher levels of competence” (Schunk & Mullen, 2012, p. 324). The flipped classroom model, in particular, has been shown to promote motivation and self-efficacy in undergraduate learners (Dixon & Wendt, 2021; Thai et al., 2020).

Study background

This study took place at a large research university10 with an enrollment of nearly 38,000 students, 21 % of which are first-year undergraduate students. These first-year students are the population of focus for this study. In particular, this research tracks IL competency development amongst students enrolled in the second half of a two-course sequence that defines the university's First-Year Writing Program (FYWP) (i.e., English 102, or “EN102”). The Library and the FYWP have a well-established relationship that includes close collaboration between instructional librarians and course instructors, as well as a program of supplementary, course-integrated library instruction.

Before the pandemic, course instructors would choose from a menu of instructional options provided for first-year students by the library. Depending on those selections, students would then attend anywhere from one to four instructional sessions—each focusing on a different set of Framework concepts.11 Instructional librarians commit to working with course sections for a full semester, providing support through IL instruction, integrated course content, and point-of-need research support. In 2018 the library and FYWP collaboratively designed and launched a set of IL learning modules that would function as a native component of the online EN102 courses, which are taught asynchronously. After two semesters of use, these modules were revised relaunched during the fall semester of 2019. Importantly, these modules were originally designed exclusively for distance learners to become familiar with the same Framework concepts being taught during the library's synchronous instructional sessions.

While the modules were made available to instructors teaching synchronous course sections of EN102, they were not a programmatic component of the library's instruction program until the campus entered a pandemic-related lockdown during March of 2020. As safety guidelines persisted into the following academic year, the asynchronous modules became an important component of the library's instructional content delivery. During this period, librarians engaged in higher levels of pre-planning with course instructors before each semester. They continued to meet with course sections synchronous via web-based teleconferencing software. These meetings occurred during the first week of class, providing a platform for librarians to introduce themselves, discuss various aspects of the library's provision of content and services, as well as to draw attention to the library's web-based learning modules (made available through the course section's online course shell).

Employing a flipped approach, students were assigned the library's two asynchronous learning modules within the first three weeks of classes. At that point, a librarian instructor would then lead a synchronous web-based session, during which students revisited the skills covered in the modules through librarian-led discussion. Students were asked to prepare research-related questions for the librarian to answer during these sessions, which provided a means of delving deeper into those concepts introduced via the modules (e.g., developing research questions, constructing search strings, and evaluating sources/content). These sessions offered librarians the opportunity to pivot from the general concepts covered in the modules to the specific research needs of the students (Table 1).

Table 1

Instructional profile.

Cohort	Synchronous sessions	Asynchronous modules	Flipped elements	Details for flipped elements
A	2 to 4 (in-person)	0	No	N/A
B	2 (web-based)	2	Yes	Modules completed before second synchronous session with librarian

Data and methods

This research takes the form of a quasi-experimental, comparative study and most notably builds upon the research of Walker and Whitver (2020). Students enrolled in the FYWP EN102 course, who also participated in the library's instructional program, are the population of interest for this study. Data analyzed herein were collected from two cohorts of study participants using a pre/post-test design, which was administered through the web-based Qualtrics survey platform.

The pre-test was administered during Week 1 of the semester, before students attended their first IL instructional session. Post-tests were administered during Week 15 of the semester, after students had attended all IL instructional sessions and completed their writing coursework. It is worth noting that the pre/post-test regimen was incorporated into the standard program syllabus for the FYWP during the spring semester of 2022, after the data gathering phase for this research had concluded. Before this, students were not required to complete the pre/post-test.

Data gathering efforts associated with Cohort A focused on students who participated in IL instruction during an eighteen-month period preceding the pandemic shutdown in late March 2020. During that time, 1576 pre-tests and 857 post-tests were completed. After eliminating unusable cases within the data,12 a final sample comprised of 813 students was achieved. This represents 9 % of all students enrolled in EN102 during the period in which Cohort A was assembled. Here it should also be noted that members of Cohort A did not access any of the asynchronous instructional modules used to teach members of Cohort B.

Data gathering efforts associated with Cohort B focused on students who participated in IL instruction during the 2021 academic year (i.e., August 2020 through May 2021). For this group, a total of 453 pre-tests and 236 post-test were completed. As was the case with Cohort A, pre/post-test completion imbalance and other data irregularities led to data withdrawal that resulted in a final sample of 207 students for Cohort B. This represents 5 % of students enrolled in EN102 during that period. In the case of both cohorts, the samples taken were smaller than desired. However, both samples provide for statistically viable findings that are generalizable to the wider population with a margin of error13 of ±3 % for Cohort A and ± 6.6 % for Cohort B.

Research questions

The primary focus of the pre/post-test is assessing learning outcomes for students enrolled in first-year writing. Within that vein, this research proposed the following research questions14 (RQs) at the outset of this study:

Will students who complete a blended and flipped IL instructional program demonstrate an improved understanding of those ACRL Framework concepts on which they are tested?

Scores from pre to post-test will show no statistically significant improvement.

Students who complete a blended and flipped instructional program will demonstrate improved understanding of ACRL Framework concepts, as evidenced by statistically significant improvement in scores from pre to post-test.

Is there any discernable difference in achievement when comparing the performance of students who participated in a synchronous instructional program versus those who participated in a blended and flipped instructional program?

Score improvement from pre to post-test for students who participated in blended and flipped instruction shows no statistically significant difference from score improvement for participants who participate in synchronous instruction.

Students who participate in a blended and flipped instructional program will see greater gains in test score from pre to post-test.

To contextualize the learning elements of this study, the researchers also collected a variety of data on peripheral factors, of a personal or environmental nature, that previous research has tied to student performance. These data were collected through the pre/post-tests and provide measures of: (a) each learner's previous research experience (i.e., experience writing research papers and experience using libraries), (b) the importance of supplementary instructional support for IL concepts outside the library classroom (i.e., professor support for IL concepts prior to, as well as after, IL session attendance), (c) student perceptions of instructional quality, as well as (d) student perceptions of their own research efficacy. 15 The following research questions were used to guide this study's exploration of these variables (Table 2):

Table 2

Research question overview.

RQ	Topical overview
1	Comparing pre and post-tests for students who participated in blended/flipped instruction program
2	Comparing student outcomes between traditional synchronous instruction and blended/flipped program
3	Comparing previous research experience between pre-pandemic and pandemic cohorts
4	Comparing supplementary support for IL skills acquisition between pre-pandemic and pandemic cohorts
5	Comparing perceptions of instruction quality across classroom-based and blended/flipped instructional programs
6	Comparing levels of perceived research efficacy between pre-pandemic and pandemic cohorts

Will students within the pandemic cohort indicate having less experience with research than students within the pre-pandemic cohort?

No statistically significant difference exists between participants' previous research experience when comparing pre-pandemic and pandemic cohorts.

Study participants in the pre-pandemic cohort bring higher levels of previous research experience into the instructional environment.

Are there differences in the amount of supplemental reinforcement of IL concepts outside of the library classroom when comparing the experiences of students in the two cohorts?

No statistically significant difference exists between the amount of supplemental reinforcement of IL concepts outside of the library classroom when comparing the experiences of students in the pre-pandemic and pandemic cohorts.

Students in the pre-pandemic cohort will experience higher levels of supplemental reinforcement of IL concepts outside of the library classroom.

Are there differences in the perceived quality of instructional content when comparing feedback from students who participated in traditional classroom-based instruction versus blended and flipped instruction?

Participant evaluation of the quality and presentation of instructional content will show no statistically significant difference when comparing across pre-pandemic and pandemic cohorts.

Students who participate in a blended and flipped IL instructional program will score the quality and presentation of instructional content more highly than students who participated in synchronous-only instruction.

Are there differences in students' perceived research efficacy when comparing the two cohorts?

No statistically significant difference exists between students' perceived research efficacy when comparing across the two cohorts.

Students who participate in the blended and flipped instructional program will indicate higher levels of research efficacy.

Pre-test/post-test design

The pre/post-test used to gather data throughout this study was originally conceived in 2017, with refinements made in 2018 and 2020. The 2018 pre/post-test was administered to Cohort A, while the 2020 version was administered to Cohort B. While changes made in 2018 were aimed at the expansion and refinement of the tests' primary and secondary question blocks, changes from the 2018 to 2020 versions16 of the pre/post-test involved relatively minor alterations in wording and the addition of four questions on both the pre and post-test (eight questions in total). These eight questions were designed to gather basic information about specific facets of the pandemic environment experienced by students, which were important to internal assessment concerns.

The core of pre/post-test is a block of 15 questions designed to evaluate students' familiarity with various IL/Framework concepts. This includes eight scenario-based questions and seven more traditional questions. While scenario-based questions require respondents to critically engage the IL concepts at play within a real-world research scenario,17 the more traditional questions require respondents to demonstrate a general understanding of specific research concepts (i.e., less critical thinking, more general knowledge). This core block of questions is identical for both the pre-test and post-test, providing a measure of each respondent's familiarity with IL concepts. This measure of basic information literacy acts as the variable of central interest for this study.

Secondary question blocks in the pre/post-test allowed the researchers to collect a variety of other data related to general respondent identification, demographics, previous research experience, an evaluation of the quality of the IL instruction received, perceived research efficacy, and any supplemental support for IL concepts provided by course instructors (i.e., outside of library instructional sessions). Data gathered for general respondent identification were used to match pre-tests to post-tests at the respondent level, then anonymized to protect the privacy of respondents. The remaining secondary question blocks collected data associated with several independent and control variables within the study. The identification of respondents, and associated demographics, are not of central importance to this research and will not be further discussed. Additional details regarding the makeup and scoring of these question blocks can be found in Table 3 .

Table 3

Variable coding.

Variable	Variable type	Composite	Number of questions	Coding details
Sessions attended	Scale	No	N/A	Score of 1 to 4
Pre-instruction
Previous research experience	Ordinal	Yes	4	Score of −12 (no experience) to 12 (much experience)
Professor support (pre-instruction)	Ordinal	Yes	2	Score of −6 (no support) to 6 (much support)
Pre-test score	Scale	Yes	15	Score of 0 to 15
Post-instruction
Post-test score	Scale	Yes	15	Score of 0 to 15
Instructional quality	Ordinal	Yes	2	Score of −6 (not engaged) to 6 (very engaged)
Research efficacy	Ordinal	Yes	1	Score of −3 (low efficacy) to 3 (high efficacy)
Professor support (post-instruction)	Ordinal	Yes	5	Score of −15 (no support) to 15 (much support)
Score improvement	Scale	No	0	Difference in score from pre to post-test (derived during analysis)

Three scoring/coding approaches were used to transform pre/post-test responses into numerical data suitable for statistical analysis. The primary question block, comprised of questions related to IL concepts, included seven multiple-choice questions with a single correct answer. These questions utilized a binary scoring method whereby a correct answer received a score of one (1) and an incorrect answer was scored with a zero (0). The remaining eight questions within this block were scored along a range of correctness, where scoring was determined by the number of possible answers. For example, while the most correct answer would receive a score of one (1), the second or third-most correct answer would be score with a half (0.5) or quarter (0.25) point. To ensure the validity of this scoring method, a panel of librarians were asked to provide independent scoring regimen. These regimen were then normed through group discussion to arrive at a single scoring method.

The pre-test's secondary question blocks include two question matrices, together comprised of six sub-questions. Four of these questions measure a respondent's previous research experience and supplementary instructional support for IL concepts outside the library classroom (i.e., professor support for IL concepts prior to the student's consumption of a library-designed instructional intervention). The remaining two questions measure a respondent's perceived research efficacy.

For the post-test, secondary question blocks are used to collect data on three peripheral factors believed to be related to student success. Both perceived research efficacy, as well as instructional quality, are measured using two-question matrices. An additional five-question matrix measures professor support (post-test), which aligns with instructional support for IL concepts during course-based interactions that occur after students' initial attendance of a librarian-led IL session.

As outlined in Table 3, all matrix-based questions take the form of a seven-point Likert scale, scored from negative three (−3) to three (3) with consecutive steps separated by a single point. Single variable measures are derived from question matrices using the sum18 of all scores within each question grouping (e.g., scores for the eight questions used to measure previous research experience are summed to derive a single score for that variable).

Analysis & findings

Due to the cohort-based design and research questions driving this study, analytical findings for each group are presented separately when appropriate. The nine variables of interest listed in Table 3 are analyzed using both descriptive and inferential statistical methods. Importantly, all but two of these variables are derived from questions using a Likert-type scale, which should be expected to produce data that are not normally distributed (Bishop & Herron, 2015; Siebert & Siebert, 2018). Additionally, as shown in Table 4 , a Shapiro-Wilk test of normality reveals non-normal distributions for both the pre-test and post-test score variables. Therefore, nonparametric19 statistical methods are employed to analyze these data.

Table 4

Tests of normality.

		Shapiro-Wilk
		Statistic	df	Sig.
Cohort A	Pre-test score	0.946	813	0.00
	Post-test score	0.890	813	0.00
Cohort B	Pre-test score	0.986	207	0.046
	Post-test score	0.975	207	0.001

Descriptive analysis

Descriptive statistical analysis alone cannot definitively answer the research questions driving this study, but it does help to important points of divergence between Cohort A (Table 5 ) and Cohort B (Table 6 ). While the median and mode for sessions attended was identical for both cohorts, a wider range of values was noted within Cohort A. This was expected, as the entirety of Cohort B participated in two synchronous instructional sessions, while members of Cohort A attended between one and six sessions. Here it's worth noting that students who attended six sessions are outliers. As indicated by the mode for this measure, it was most common for students to attend two sessions, though there were a significant numbers of students who attended three or four sessions.

Table 5

Descriptive analysis (Cohort A).

n = 813	Min	Max	Mode	Median	IQR
Sessions attended	1.00	6.00	2.00	2.00	1.00
Pre-instruction
Previous research experience	−12.00	12.00	6.00	5.00	6.00
Professor support (pre-test)	−6.00	6.00	6.00	4.00	3.00
Pre-test score	0.30	15.00	12.30	10.50	3.50
Post-instruction
Quality of instruction	−6.00	6.00	6.00	4.00	3.00
Perceived research efficacy	−6.00	6.00	4.00	3.00	3.50
Professor support (post-test)	−15.00	15.00	15.00	11.00	6.00
Post-test score	0.30	15.00	12.30	11.40	3.18
Score improvement	−6.80	11.70	0.00	0.50	2.10

Table 6

Descriptive analysis (Cohort B).

n = 207	Min	Max	Mode	Median	IQR
Sessions attended	1.00	2.00	2.00	2.00	0.00
Pre-instruction
Previous research experience	−10.00	12.00	4.00	4.00	5.00
Professor support (pre-test)	−5.00	6.00	6.00	4.00	3.00
Pre-test score	4.00	14.50	9.00	10.00	2.20
Post-instruction
Quality of instruction	−6.00	6.00	6.00	5.00	2.00
Perceived research efficacy	−6.00	6.00	6.00	4.00	3.00
Professor support (post-test)	−15.00	15.00	15.00	12.00	5.00
Post-test score	5.50	14.50	12.00	11.30	2.50
Score improvement	−2.00	4.50	0.00	1.00	2.00

With regard to peripheral factors, members of Cohort A indicate higher levels of previous research experience (RQ3) than members of Cohort B. While the data gathered through this study cannot provide for a definitive reason for such a discrepancy, pandemic-driven changes in both the high school and collegiate instructional environments/curricula seem a plausible, if not probable, cause. With both cohorts receiving relatively equivalent professor support (pre-test) (RQ4), the difference in previous research experience may help to explain the higher overall pre-test scores produced by Cohort A. However, additional inferential analysis is needed to confirm this.

Cohort B reported higher levels of perceived research efficacy (RQ6) and quality of instruction (RQ5). Members of Cohort B also indicated higher levels of professor support (post-test). Importantly, both cohorts showed improvement in scores from pre to post-test (RQ1). Cohort B showed greater overall score improvement (RQ2).

Inferential analysis

While simple descriptive statistical analysis can help to flesh out any superficial differences between this study's two cohorts, a more rigorous inferential analysis is required to determine if such differences are statistically significant. However, before proceeding it is important to confirm the reliability of the pre/post-test instruments being used. A simple correlation-based analysis of test-retest reliability is the preferred method when administering a single testing tool across multiple points in time (as is the case with this study). As shown in Table 7 , this study's testing instrument shows acceptable levels of reliability as indicated by a correlation coefficient of 0.72320 and 0.772 for Cohort A and Cohort B, respectively.

Table 7

Test-retest reliability.

			Post-test score
Cohort A	Pre-test score	Spearman Correlation	0.723⁎⁎
		Sig. (2-tailed)	0.000
		N	813
Cohort B	Pre-test score	Spearman Correlation	0.772⁎⁎
		Sig. (2-tailed)	0.000
		N	207

Correlation is significant at the 0.01 level (2-tailed).

As previously discussed, both of this study's participant cohorts displayed improvement in scores from pre to post-test (see Table 5, Table 6). This indicates a general improvement in participants' demonstrated mastery of those IL concepts covered during instructional sessions. Using the Wilcoxon signed-rank test, these within-group findings were confirmed to be statistically significant (see Table 8 ).

Table 8

Wilcoxon signed-ranks test.

		N	Mean rank	Sum of ranks
Post-test score - pre-test score (Cohort A)	Negative ranks	234a	282.88	66,194.50
	Positive ranks	478b	392.54	187,633.50
	Ties	101c
	Total	813
Test statistics
Z		−11.062d
Asymp. Sig. (2-tailed)		0.000

Post-test score < pre-test score.

Post-test score > pre-test score.

Post-test score = pre-test score.

Based on negative ranks.

Regarding potential divergence in achievement/performance between the two cohorts, pre/post-test scores are of central importance. The researcher wants to know if members of one cohort scored higher, on average, than members of the other cohort. Of course, it is also important to consider the context and validity of such comparisons. For example, it is important to determine if both groups start in relatively the same place, knowledge-wise. The type and amount of instructional exposure provides additional context to the analysis. With that said, differences in the type of instructional exposure constitute a central point of concern for both RQ1 and RQ2.

The basic descriptive statistics (see Table 5, Table 6) show Cohort A attending a higher median number of instructional sessions than Cohort B. Cohort A also displayed higher median pre-test scores and lower median post-test scores—10.5 and 11.4, respectively, compared with median scores of 10 and 11.3 for Cohort B. In addition, median score improvement from pre-test to post-test is 0.5 for Cohort A and 1.0 for Cohort B. As shown in Table 9 , subjecting these findings to more rigorous inferential testing reveals statistically significant divergence for the IL sessions and pre-test score variables. No statistically significant difference exists between post-test scores for the two groups.

Table 9

Mann-Whitney test (sessions and tests).

		N	Mean rank	Sum of ranks	Test statistic	Asymp. Sig. (2-tailed)
Sessions attended	Cohort A	813	552.00	448,772	50,410	0.000⁎⁎
	Cohort B	207	347.53	71,938
	Total	1020
Pre-test score	Cohort A	813	519.78	422,584	75,426.5	0.021⁎
	Cohort B	207	474.04	98,126
	Total	1020
Post-test score	Cohort A	813	515.36	418,990.5	80,191.5	0.296
	Cohort B	207	491.40	101,719.5
	Total	1020

Correlation is significant at the 0.01 level (2-tailed).

Correlation is significant at the 0.05 level (2-tailed).

Peripheral factors

Inferential analysis reveals statistically significant divergence between the two cohorts for three of the five peripheral factors of interest. As shown in Table 10 , Cohort A came into IL sessions with higher levels of previous research experience than members of Cohort B. In addition, Cohort B rated the quality of instruction more highly than Cohort A. Cohort B also indicated higher levels of research confidence via the perceived research efficacy variable. While descriptive analysis revealed Cohort B experienced slightly higher levels of professor support for IL concepts, both before and after attending library instruction, that difference was not statistically significant.

Table 10

Mann-Whitney test (peripheral factors).

		N	Mean rank	Sum of ranks	Test statistic	Asymp. Sig. (2-tailed)
Previous research experience	Cohort A	813	529.81	430,732.0	68,450	0.000⁎⁎
	Cohort B	207	434.68	89,978.0
	Total	1020
Professor support (pre-test)	Cohort A	813	503.38	409,247	78,356	0.120
	Cohort B	207	538.47	111,463
	Total	1020
Professor support (post-test)	Cohort A	813	502.34	408,399	77,508	0.074
	Cohort B	206	542.57	112,311
	Total	1019
Quality of instruction	Cohort A	813	497.36	404,357.5	73,466.5	0.004⁎⁎
	Cohort B	206	562.09	116,352.5
	Total	1019
Perceived research efficacy	Cohort A	813	495.01	402,447	71,556	0.002⁎⁎
	Cohort B	206	566.95	116,224
	Total	1019

Correlation is significant at the 0.01 level (2-tailed).

Discussion

All but one21 of the research questions driving this study related to phenomena previously explored within the established literature on student learning. What's more, the alternative hypotheses posed in response to these questions were based on previous findings in the literature, and not on the researcher's intuition. However, the fact is that the pandemic represented a significant X factor, with potential to affect this study's participants and outcomes in unexpected ways. So while in most cases the findings shared herein correspond with stated expectations, the researcher harbored significant doubts as to the final outcomes that would be seen. So, in a way, some of this study's findings were surprising, even when expected.

Student achievement

With regard to the achievement of learning outcomes, this research found that Cohort B (i.e., the pandemic cohort) did demonstrate significant improvement in their understanding of IL concepts after participating in a blended and flipped library instructional program. These findings align with previous research by Bower et al. (2015), Raes et al. (2020), and Walker and Whitver (2020). Therefore, the null hypothesis is rejected, and alternate hypothesis is affirmed, for RQ1.

Though both cohorts experienced significant improvement in scores from pre to post-test, the scores of Cohort B saw greater median improvement. This group experienced a median score improvement from pre to post-test of 1.0 out of 15 points, which represents a 7 % improvement. This is a significantly better than was experienced by Cohort A, which saw a median score improvement of only 0.5 points (i.e., a 3 % improvement). This finding allows for a rejection of the null hypothesis for RQ2. Nevertheless, uncertainty persists as to the underlying mechanism at play.

Previous research has shown blended and flipped instructional programs to be more effective than traditional approaches (e.g., Bower et al., 2015; Dixon & Wendt, 2021; Kakarougkas & Abdellatif, 2022). However, pandemic-related factors leave the door open to other possible explanations for this study's findings. For example, it is important to consider that student's in Cohort B came into their first year of university with significantly less research experience than their peers in Cohort A (see Table 5, Table 6, Table 10). This finding allows for a rejection of the null hypothesis for RQ3 22 and helps to explain why Cohort B recorded significantly lower pre-test scores than Cohort A. With that said, neither cohort experienced significantly higher levels of professor support for IL competencies—either before or after attending librarian-led IL instruction.23 Additionally, neither cohort scored significantly better on the post-test. Taken as a whole, these findings indicate the significant difference in score improvement is likely the result of lower pre-test scores for Cohort B, rather than better overall learning outcomes achievement.24

If the blended/flipped approach to IL instruction is more effective, there exists potential opportunities for IL instruction during subsequent disruptions or after pandemic restrictions have ended. For example, in this study both cohorts ended their first-year IL journey in about the same place—a median post-test score of 11.3–11.4 points out of 15. Importantly, in the case of Cohort B, this was accomplished with fewer sessions taught by librarians. In addition to providing support for the idea that the blended/flipped approach is at least as effective, if not more so, than traditional IL instructional approaches, these findings offer support for less obvious pedagogical and operational benefits. Consider the benefit of freeing up roughly 50 % of instructional librarians' current classroom-based investment in a single program objective (i.e., support for first-year writing). This time savings could be re-allocated to liaison outreach, or put toward the creation of additional asynchronous content that could support an expansion blended/flipped IL instruction across the disciplines. What's more, an improved instructional approach might allow librarians to incorporate a wider/deeper dive into Framework concepts that was simply not possible when employing an exclusively synchronous classroom-based approach.

Peripheral factors influencing achievement

Setting aside learning outcomes achievement, the blended and flipped instructional program offered to Cohort B did deliver positive outcomes on other important fronts. Speaking to RQ5 and RQ6, members of Cohort B judged the quality of library instruction more highly than members of Cohort A. Cohort B also indicated higher levels of perceived research efficacy, following their completion of the blended and flipped library instructional program. Importantly, both of these findings align with previous research (e.g., Dixon & Wendt, 2021; Thai et al., 2020), which has shown student perceptions of self-efficacy and satisfaction are improved within both the blended and flipped instructional environments.

Study limitations

This study has several limitations worthy of note, the most obvious being the pandemic conditions under which it was conducted. The data gathered during this study were undoubtedly affected by altered institutional operations, as well as a host of environmental factors that cannot be completely quantified or mitigated with the data available or methods employed. For example, it is believed that the sample size of Cohort B, though statistically viable, would have been larger under normal circumstances. This likely affected the visibility of more nuanced patterns in the data—hampering deeper comparative understandings across cohorts. Additionally, as was previously mentioned, design elements of this study related to RQ2 produced ambiguous findings in regards to the comparative analysis of achievement across the two cohorts under study.

Conclusion

Although one might expect an IL program initiated in response to pandemic restrictions to fall short of meeting standards set during pre-pandemic cycles, this study's findings seem to point in another direction. Specifically, it may be the case that the blended and flipped approach taken toward teaching IL competencies to Cohort B was the best move for helping these students overcome a research experience gap created during their senior year of high school. While this study's findings align with previous research, additional study is needed to confirm the comparative benefits of a blended and flipped instructional program within the library context.

This study's findings, at the very least, confirm the pedagogical viability and value of a blended and flipped approach to teaching IL competencies to first-year college students. Despite pandemic-related impediments, namely a gap in previous research experience, students who participated in the blended and flipped instructional program ended their first-year IL training on equivalent footing with their pre-pandemic peers in Cohort A. These students also indicated that the blended and flipped instructional program was of high quality, and moved on from the program with significantly higher levels of research self-efficacy than members of Cohort A. As previous research has shown, this is likely to result in bolstered resiliency in the face of academic challenges.

Several viable paths exist for impactful future research in this area. First, it will be important to confirm the viability of the blended and flipped instructional approach within the realm of librarian-led IL instruction. Ideally, such a study would employ simultaneous experimental cohorts that would help to control for any environmental factors that might affect student experiences or outcomes. In addition, this study broached two topics that are worth of expansion within future studies—students' self-efficacy in relation to research practice, as well as their perception of instructional quality within the context of non-traditional instructional approaches.

CRediT authorship contribution statement

As the sole author, Kevin W. Walker is responsible for all facets of this manuscript's preparation.

In this session, students will learn how to use the library's search engine Scout, and how to find databases using the database page. This session will not cover controlled vocabulary or keywords (covered in Option 2), or advanced search strategies in Scout or in individual databases (covered in Option 4).

This class will cover:• How to use Scout's basic search box • How to use simple limiters in Scout to narrow a search • How to access various items such as PDF and HTML files, books on the shelf • How to request an item from ILL • How to use the descriptive information on the databases page to choose a database • How to sort databases by source type or by subject

Learning outcomes:1. Students will locate the Scout Search interface in order to begin performing searches to find sources. 2. Students will review the descriptions on the database page in order to choose a subject specific database related to their individual research.

In this session, students will learn how to analyze a topic, use reference resources, and develop a search strategy for finding sources for their writing assignments.

This class will cover:• How to narrow a topic and brainstorm topic ideas • How to identify key concepts and controlled vocabulary • How to use reference tools • How to select search terms and develop a search strategy

Learning outcomes:1. Students will break a topic down into smaller components in order engage in the iterative process of narrowing a research question. 2. Students will select terms related to their key concepts in order to develop a search strategy.

In this session, students will learn to assess a source for relevancy and authority to decide whether it meets the needs of their writing assignment. Students will engage in activities and conversations throughout this session to help them understand the nature of popular and academic publishing.

This class will cover:• How to research an author • How to identify publication process • How to determine the intent of sources.

Learning outcomes:1. Students will recognize the difference between popular and scholarly sources in order to appraise a source's value within their own research. 2. Students will assess a source's original intent, editorial and publication process, and the expertise of its author in order to determine its credibility.

In this session, students will be taught to execute advanced searches in selected databases and resources. Databases will be selected when the session is scheduled, and advanced options will be chosen according to the needs of the class topic and assignment.

This class will cover:• Advanced search options for specific resources such as LexisNexis Academic, the Library's Catalog, JSTOR. • How to find a specific publication using a Journal Title Search • How to find a known item such as an article that is referenced in an index record or a works cited page • How to identify controlled vocabulary for targeted, field-specific searching

Learning outcomes:1. Students will use controlled vocabulary and field-specific search options and limiters in order to retrieve topically relevant sources 2. Students will combine search terms with boolean operators in order to effectively interpret their research question into an effective search query