Changes to summative skills-based assessments within the Big Ten Academic Alliance Performance-Based Assessment Collaborative (BTAA-PBAC) due to COVID-19.

Background: In Spring 2020 many academic institutions transitioned to remote learning in response to the developing COVID-19 pandemic. These changes affected skills-based training, as schools of pharmacy were forced to transition traditionally in-person assessments to a remote setting. The purpose of this article is to describe the experience of pharmacy skills lab coordinators when transitioning summative skills-based assessments (SSBA).
Methods: A web-based survey instrument administered through QualtricsXM was sent to all institutions in the Big Ten Academic Alliance-Performance Based Assessment Collaborative. Only one member from each institution completed the survey on behalf of the institution. The survey consisted of four sections: changes made to skills evaluated; changes made to the delivery of those evaluations; challenges to and strategies used by the skills lab program when switching to remote learning; and recommendations for incorporating remote learning within future SSBAs. Survey respondents were invited to participate in an optional unstructured interview regarding survey answers.
Results: Nine of ten invited institutions responded to the survey. Of the nine respondents, three participated in the post-survey interview. Overall, 79.5% (93/117) of skills planned to be assessed were assessed with or without modification, with 8.5% (10/117) of skills canceled and 10.3% (12/117) of skills assessments postponed. The most common challenges mentioned were the lack of preparation time, inability to assess certain skills virtually, and student barriers. The most common recommendations made were to prioritize lab components and incorporate flexibility in planning and scheduling. Discussion: The results indicate that most skills were still assessed during the Spring 2020 semester. Though the transition to remote learning was challenging and unique for each institution, common strategies and recommendations identified here provide opportunities for academics to analyze and prioritize learning objectives and to rethink how to develop and deliver SSBAs as remote assessments.

INTRODUCTION

After the first case of novel coronavirus disease 2019 (COVID‐19) was reported in the United States on January 20, 2020, no one could have predicted the profound and perhaps irrevocable disruption COVID‐19 would have on health care education. Educators were forced to be creative and seek new ways to educate students. Skills‐based education and assessment pose a unique set of challenges during this time of restricted in‐person activities. Skills‐based assessment is part of the health care curriculum in medical education and part of licensure qualifications. The COVID‐19 pandemic has caused several changes to medical and pharmacy examination schedules. For example, Canada uses an Objective Structured Clinical Examination (OSCE) as one requirement for licensure to practice pharmacy. Their May 2020 examination was canceled and rescheduled for November 2020. Although the U.S. does not require a clinical skills exam for licensure, creating competent pharmacy graduates is a priority of the Accreditation Council for Pharmacy Education (ACPE), and their 2016 Standards emphasize the importance of mastery of pharmacy skills. This places skills‐based education in an important role in facilitating the mastery of these skills and in developing practice‐ready pharmacy graduates that are prepared to directly contribute to patient care. Therefore, the COVID‐19 pandemic and subsequent disruption in teaching methods are likely to require significant adaptations to delivering skills‐based education within pharmacy curricula.

Information on the use of virtual skills assessments in pharmacy is not widely available. However, prior to the COVID‐19 pandemic, medical schools with distance campuses developed virtual OSCEs. These assessments are equivalent to in‐person OSCEs without bringing students onto campus. The inability to perform a physical examination and internet connectivity issues are the most common barriers to online medical OSCE assessment. , , In the era of COVID‐19, many medical schools have had to contemplate innovative solutions to conduct OSCEs. Schools that maintained face‐to‐face OSCEs implemented strict social distancing guidelines and infection control protocols such as temperature screenings, hand sanitization, and unidirectional flow of people in addition to limiting the number of students, faculty, and simulated patients (SP) in the building at a time. Other medical schools modified their OSCEs by conducting them virtually through video conferencing software such as Zoom (Zoom Video Communications, San Jose, CA). The breakout rooms feature in Zoom was utilized for SP encounters, while a faculty member observed and aided the SP on completing formal checklists and providing feedback to the student. , Both OSCE adaptations demonstrate feasible alternatives to traditional summative skills‐based assessments (SSBAs) in the pharmacy curriculum during this uncertain time.

Pharmacy curricula adaptations to address COVID‐19 concerns have been described on the association, campus, and global levels. , , , Lecture‐based courses are described as easier to convert to an online teaching modality within pharmacy education, as a significant amount of research has been conducted surrounding pedagogical best practices for online content delivery. However, specific details surrounding transitioning skills‐based education to an online modality are lacking. Examples of transitions to online modules for the introduction of concepts required for skills demonstration include in‐person demonstration and practice of these skills. , Recent findings have also suggested that student pharmacists prefer in‐person evaluation over video evaluation where instructors viewed the activity live using audio‐video recording or live‐streaming tools. Students often cited that the distance between their evaluator hindered their ability to establish a connection or professional rapport with their evaluator. Further, practice of skills is often required to achieve near mastery of a skill, which for physical assessment and other in‐person skills, would likely be difficult to achieve in a virtual environment. To address this lack of information in the skills‐based laboratory setting, members of the Big Ten Academic Alliance‐Performance Based Assessment Collaborative (BTAA‐PBAC) developed this study. The purpose of this article is to provide the collective experience from peer schools of pharmacy of changes made to SSBAs during a pandemic. To fully describe the experience of the participants, we sought to obtain the following information: changes made to the list of skills evaluated in summative assessments; changes made to the delivery of those skills evaluated; prominent challenges to the skills lab program when switching to remote learning and how those challenges were addressed; and recommendations and tips for incorporating remote learning within future SSBAs.

METHODS

A web‐based survey instrument was developed and administered using the online tool, QualtricsXM (Qualtrics, Provo, UT). Prior to distribution, the survey was pre‐tested by members of the investigative team. The survey was divided into four main sections centered around the four study objectives. The list of skills in the survey was based on common skills assessments previously identified by the BTAA‐PBAC and included the following: communication with patients; communication with health care providers; documentation; drug‐related problem (DRP) identification, prioritization, and resolution; physical assessment; collection of patient information; injection technique; sterile prescription processing; non‐sterile prescription processing; and professionalism. Skills‐based assessment components and requirements can vary between institutions and within a semester at one particular institution. To limit the number of questions and provide a consistent base from which to answer questions regarding changes due to the COVID‐19 pandemic, the scope of the survey was limited to asking about summative assessments that occurred in Spring 2020. These are defined as assessments that occur at the end of an instructional unit, evaluate the student on the previous instructional unit, and produce a score that is included in the student's overall grade. The survey consisted of multiple choice, closed‐ended, multi‐answer, and open‐ended questions. The survey used display logic and skip logic and provided a maximum of 103 questions and a minimum of 16 questions.

Additionally, respondents were prompted to select the type of change(s) made for each skill. The options were change in activity (eg, method of delivery, use of distractors, or use of standardized or simulated patients), change in evaluator (eg, role of the evaluator, communication with the evaluator, or information shared with the evaluator), change in rubric/evaluation (eg, rubric or analytical checklists, or significance of score), and change in student factors (eg, time provided for student preparation, information provided to students beforehand, or individual vs group work). Respondents were then asked to describe the specific changes for each skill. Lastly, the survey included items addressing changes to components of skills labs that were not necessarily tied to summative assessments. This included the frequency of communication, methods of communication, new tools used, changes in how current tools were used, and plans for continuing any changes made to lab structure due to the COVID‐19 pandemic.

A total of ten BTAA‐PBAC member public colleges and schools of pharmacy were invited via email to anonymously complete the survey in August 2020. A reminder email was sent 1 week later. A link to the survey was sent to all members of BTAA‐PBAC. Only one member from each institution was asked to represent their institution and complete the survey. BTAA‐PBAC members were encouraged to forward the survey information to or consult with a more appropriate individual (or group of individuals) within their institutions if they felt unable to accurately complete the survey. Research team members were encouraged not to answer the survey themselves, if possible.

Results from the survey were reported in aggregate and not linked to an identifiable institution. To avoid duplicate surveys and maintain anonymity, respondents were redirected to a separate survey to provide the names of their institutions. Respondents were provided an opportunity to participate in an optional post‐survey interview where they could share additional information and insights on their transition to remote learning for their skills lab program that was not included in the survey. The interview was conducted via phone or teleconference within 2 weeks of survey completion.

Descriptive statistics were used to evaluate the survey data. Free‐text responses were assessed qualitatively to determine key themes and recommendations related to summative assessment changes in the COVID‐19 pandemic.

The study was deemed exempt by the institutional review board at Purdue University and at the eight other schools that were part of the research team.

RESULTS

Survey responses were received from individuals at nine out of ten institutions (90%). The individual's information was not collected but was assumed to be an appropriate representative of the skills lab group. Of the nine respondents, three participated in the post‐survey interview. The online survey was divided into four main sections. The first section of the survey asked respondents to identify which skills were planned to be assessed in SSBAs and of those planned, which had to be postponed or canceled. These changes can be seen in Figure 1 and are stratified by professional year. Overall, 79.5% of skills planned to be assessed were assessed with or without modification (93/117), 8.5% of skills planned to be assessed were canceled (10/117), 10.3% of skills planned to be assessed were postponed (12/117), and 1.7% of skills planned to be assessed are unknown if changes were made (2/117).

FIGURE 1

Summative assessment skills assessed by professional year across institutions. Institutions listed are all public institutions with a college or school of pharmacy that participates in the Big Ten Academic Alliance Performance Based Assessment Collaborative. Collect = Collection of Patient Information; Comm Pt = Communication with Patient; Comm HCP = Communication with Healthcare Provider; DRP = drug‐related problem; Document = Documentation; Injection = Injection Technique; Non‐sterile = Non‐sterile compounding; P1 = first‐year pharmacy student; P2 = second‐year pharmacy student; P3 = third‐year pharmacy student; Physical Assess = Physical Assessment; Professional = Professionalism; Sterile = Sterile Compounding

The following general reasons were cited for canceling or postponing assessments: institution went virtual before the assessment; limited resources and time from faculty developing content because it was the first time delivering the course in the new curriculum; the skills not assessed were difficult to assess in an unplanned remote environment fully; and skills required special equipment, direct patient contact, and/or direct observation by a teaching assistant or an instructor. Respondents also provided some specific reasons that were tied to a particular component of the assessment. For example, reasons specific to elements involving standardized patients included SPs not being adequately trained for a virtual interview and inadequate time to train them, difficulty coordinating all parties involved due to scheduling and technology difficulties, and SPs not having access to campus resources. Reasons specific to injection and physical assessment included that skills to be assessed were not conducive to a virtual format because students were not allowed on campus.

The second section of the survey focused on the number and type of modifications that were made to the delivery of completed SSBAs, and is presented in Table 1.

TABLE 1

Changes to delivery of skills evaluated in summative assessment by institution

	Summative assessment skills
	Communication with patients		Communication with health care providers		Documentation		DRP identification, prioritization, and resolution		Physical assessment		Collection of patient information		Injection technique		Sterile prescription processing		Non‐sterile prescription processing		Professionalism
Institution	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change	Planned	Change
1	✓	U			✓	U							✓	U
2	✓	A, E, R	✓	A	✓	A, E, S	✓	A, S	✓	A, S	✓	A, R, S	✓	A	✓	N	✓	A, R
3	✓	A, E, R	✓	A			✓	A			✓	A, E	✓	N			✓	A, R	✓	N
4	✓	E, S	✓	N	✓	A, E, R, S	✓	A, E, R, S	✓	N	✓	A, E, R, S					✓	N	✓	E, S
5	✓	A, E			✓	N	✓	N	✓	N	✓	A, E	✓	N					✓	N
6	✓	A, E, R, S	✓	A, S	✓	A, S	✓	S			✓	N			✓	S	✓	A, S
7	✓	A, R, S	✓	A, S	✓	N	✓	N			✓	S							✓	N
8	✓	A			✓	N	✓	S	✓	U	✓	A			✓	U	✓	U
9	✓	A	✓	A	✓	S	✓	S			✓	A, S	✓	U	✓	A, S	✓	A, S	✓	A
Summary	87.5% (7/8) institutions made at least 1 change		83.3% (5/6) institutions made at least 1 change		50% (4/8) institutions made at least 1 change		75% (6/8) institutions made at least 1 change		25% (1/4) institutions made at least 1 change		87.5% (7/8) institutions made at least 1 change		20% (1/5) institutions made at least 1 change		50% (2/4) institutions made at least 1 change		66.7% (4/6) institutions made at least 1 change		40% (2/5) institutions made at least 1 change

Note: Institutions listed are all public institutions with a college or school of pharmacy that participates in the Big Ten Academic Alliance Performance Based Assessment Collaborative. A = change in activity; DRP = drug‐related problem; E = change in evaluator; R = change in rubric/evaluation; S = change in student preparation or role; U = unknown.

The majority of SSBAs that were completed required at least one modification to one of the four components; activity, evaluator, rubric, and student. There were 78 changes made overall in the nine institutions answering the survey (activity n = 32; evaluator n = 12; rubric/evaluation n = 10; student n = 24). Of note, an institution could report multiple changes for a single skill and this data was not stratified by professional year in the pharmacy curriculum. If a modification was reported, respondents were asked to provide specific information via free text response describing the changes that were made. These modifications are presented in Table 2.

TABLE 2

Examples of specific modifications made

Activity a	Evaluator b	Rubric c	Student d
• Videoconferencing technology utilized for synchronous telehealth patient encounters • Students submitted a video recording of them performing patient counseling • Instead of using standardized patients, utilize other people to play role of patient (eg, roommate, family member, upperclassmen students, TAs) • Students watched video of someone performing skill (eg, taking blood pressure or counseling) and evaluate • Activities transitioned to be completed asynchronously • Incorporation of online quizzes	• Standardized patients no longer utilized • Evaluator filled role of patient and evaluator • For recorded student submissions, evaluators watched recording to assess • Evaluator communicated with student through videoconferencing technology (eg, Zoom, WebEx)	• Specific rubrics were updated based on activity changes • Some grading changed to pass/fail • Students assessed on ability to identify errors or missing components from watching a video of someone performing skill (eg, taking blood pressure, patient counseling)	• Time for student preparation extended • Time for summative assessment stations/activities extended • Asynchronous access to materials and submission of materials

Abbreviation: TAs, teaching assistants.

Modification to activity encompasses changes in method of delivery, activity complexity, use of distractors, or use of standardized or simulated patients.

Modification to evaluator encompasses changes in the role of the evaluator, communication with the evaluator, or information shared with the evaluator.

Modification to rubric includes changes in items included in rubric or analytical checklists, or significance of score.

Modification to student includes changes in time provided for student preparation, information provided to student beforehand, or individual vs group work.

The third section surveyed challenges to the skills labs when switching to remote learning and how those challenges were addressed. Results of this section are presented in Table 3. The top three challenges cited were lack of preparation time, inability to assess certain skills virtually, and student barriers. The most common themes regarding strategies that made the transition to online learning easier included: administration support, teleconferencing and livestream technology, frequent communication, and internal collaboration. Raw responses are available in Appendix A.

TABLE 3

Summary and count of challenges, strategies, and recommendations from participating institutions

What was the most challenging aspect of the COVID‐19 pandemic and its effect on summative assessments within professional skills laboratories?
Themes of challenges	Number a
Lack of preparation time	3
Inability to assess certain skills virtually	3
Student barriers	3
Technology changes	2
Increased student stress	2
Increased instructor stress	2
Prioritizing assessments for in‐person vs remote assessment	2
Decreased interactions with students	1
Lack of support personnel	1
Logistics for instructor	1
Academic integrity of assessments in remote setting	1

Abbreviation: COVID‐19, coronavirus disease 2019.

Number refers to the number of survey responses that included the listed challenge, strategy, or recommendation.

The fourth section surveyed recommendations for incorporating remote learning within future SSBAs as outlined in Table 3. Common themes of recommendations for future SSBAs in skills labs included: prioritization (learning goals, learning objectives, skills), flexibility in planning and scheduling, having a back‐up plan, and frequent communication with students and instructors. Raw responses are available in Appendix A.

With respect to results pertaining to changes to components of skills labs that were not part of the SSBAs, seven institutions indicated that changes were made to policies within the skills lab. These changes included leniency on deadlines and student tardiness. Additionally, some institutions adjusted the overall grading of the course (eg, changing from grades to pass/fail and provided the option for “incomplete” status). Before moving to remote learning, six institutions were communicating weekly with students, one was communicating every 2‐3 days, and two were communicating on an as‐needed basis. Once the institutions transitioned to remote learning, that changed to four communicating on a weekly basis, and five communicating every 2‐3 days. The method of communication with students also changed due to the COVID‐19 pandemic. Before moving to remote learning, communication was done via work email (6/9), through the learning management system (LMS) (9/9), and in‐person office hours (6/9). After the transition to remote learning, the methods of communication changed to work email (7/9), LMS (9/9), teleconference office hours (7/9), and phone calls (2/9). Two of the three institutions that did not previously provide in‐person office hours started providing teleconference office hours. Lastly, respondents were provided the opportunity to describe any new tools used, tools they would have liked to have during the transition, change in use of the current tools available to them, and changes they plan to keep once restrictions due to the COVID‐19 pandemic are removed. This information is provided in Table 4.

TABLE 4

Summary of items regarding usage of tools

New tools used	Desired tools	Changes to current tools	Changes to keep	Reasons for keeping changes
• Student video submission • Livestream lectures • Teleconference platforms • Online shared drive • Academic integrity tools • Simulated patients	• More recordings • More robust LMS quiz options • Standardized patients • Academic integrity software • Scheduling software • Zoom	• Increased amount of content online • Utilized LMS more robustly and more often • Used teleconference platforms for assessments • Increased number of videos and recordings utilized for teaching and assessment • Utilized teleconference platforms to collaborate	• Methods by which skills are demonstrated by the students • Method by which activities are evaluated • Video submissions • Use of Blackboard Collaborate • Frequency of communication with students • Method of communication with faculty • Peer evaluation • Use of discussion boards • Teleconference office hours	• The changes were found to be useful and the components will be retained • Because the cases and rubrics differ from traditional in‐class processes • Telehealth is important to assess • Video submission allows for asynchronous evaluation and is more convenient • Video submission allows for increased student practice • Reduction in paper grading forms

Abbreviation: LMS, learning management system.

DISCUSSION

Colleges and schools of pharmacy have dealt with multiple issues since the start of the COVID‐19 pandemic, including remote delivery of content and examinations, supporting student wellness, and disruptions in experiential education. , , , The delivery of skills‐based assessments traditionally performed in‐person in a lab‐based setting was also forced to adapt. To our knowledge, this is the first study to specifically evaluate the initial changes made in skills‐based pharmacy education in response to the COVID‐19 pandemic.

Despite the restrictions with face‐to‐face instruction and social distancing, our survey of nine peer pharmacy schools found the majority of skills planned to be assessed were still assessed in the spring 2020 semester. Only a small percentage of skills that were planned to be assessed were canceled or postponed. Among all of the skills assessed, injection technique was the only skill that could not be completed during the spring 2020 semester across all institutions planning to assess this skill. This is not a surprising finding, given the hands‐on nature of this particular skill. Skills that were planned to be assessed during the first professional year of the program that could not be assessed were all able to be postponed to a later point in the curriculum. In contrast, skills that were planned to be assessed during the third professional year of the program that could not be assessed at that time had to be canceled. This finding makes sense as third professional year student pharmacists typically begin Advanced Pharmacy Practice Experiences (APPEs) shortly after completion of the spring semester, leaving little opportunity to assess the skill prior to progressing to APPEs.

Several modifications needed to be made to skills‐based assessments and across institutions in order to carry out the SSBAs that could not be rescheduled. Given the rapid shift from an in‐person environment to a virtual environment, many of the modifications that were made took advantage of available technologies including videoconferencing for synchronous assessments and uploading of recorded videos for asynchronous assessments. Because our survey allowed respondents to elaborate specifically on how assessments were modified, such as allowing students to submit recorded videos in place of in‐person skills demonstrations, the examples listed can be used by other institutions as the COVID‐19 pandemic continues to progress and virtual assessments may continue to be needed in future semesters. Many of the modifications made are similar to adjustments made in medical education. , , A consistent theme among the modifications made is the time related to the activity itself. Students were allowed more time to prepare, access material, or complete assignments. Evaluators and students were able to complete activities asynchronously, outside of the usual timeline. This indicates that remote assessments could improve flexibility in scheduling and completion for both faculty and students.

Due to the unexpected nature of the need for changes, the participating schools experienced several challenges. The most common challenges highlighted that participating schools struggled with the short time frame to adjust to remote learning and choosing appropriate skills to assess virtually vs postponing or canceling the assessment. Another challenge was navigating through students' barriers associated with adjusting to new assessment formats. These findings indicate that future changes to skills assessments require significant planning time to overcome challenges brought on by the pandemic and a virtual learning environment. Student barriers such as time zone differences, technology limitations, and stress should also be considered when deciding future remote assessments.

To address the challenges listed, participating schools cited several strategies they employed. The most common strategy was receiving support from administration. For example, administration provided guidance to transitioning online and prioritized student wellness when choosing skills to assess. Two major strategies for the development of future SSBAs were to prioritize components of the skills assessments and improve flexibility in planning and scheduling. Schools suggested prioritizing learning objectives and determining what skills need to be assessed in‐person vs remotely. This allows for pre‐planning in the case of future alterations, improving flexibility. These strategies and recommendations may help guide other institutions in developing a flexible summative skills assessment plan. Additionally, the participation of administration in addressing the challenges posed by the transition to remote learning allowed the lab coordinators to focus on prioritizing lab components and make needed logistical changes.

Of note, participants indicated they would keep some of the changes made due to the pandemic restrictions. For example, the use of video submission by students, frequency of communication with students, and use of peer evaluation were new changes to some institutions which were described as useful and allowed for increased student practice. This indicates that some benefits can be derived from the abrupt change in skills assessment structure, albeit in a challenging manner. Participants also provided recommendations for future incorporation of remote assessment into skills assessments. The recommendations provided emphasize the need for collaboration and communication with all parties involved in skills assessments, including students, administration, peer institutions, other faculty, and standardized patients. Additionally, these recommendations are unsurprising, as they are similar to how the skills lab coordinators addressed the most common challenges.

Strengths of this study include the high response rate and specific examples of modifications utilized by various institutions to ensure student proficiency before matriculating into their program. These examples demonstrate what can be done on short notice and serve as a place for starting to incorporate more remote learning/assessments that will likely continue beyond the pandemic. They serve as examples of how to quickly pivot summative skills‐based assessments to the virtual format. This pivot has been mirrored in clinical practice, with the telehealth visit becoming more common. Several of the strategies used, such as moving to telehealth encounters instead of face‐to‐face interviews will likely stay past the pandemic. This type of skill is proving to be essential for new pharmacists to be able to navigate as telehealth becomes more incorporated into health care. Students now have the opportunity to broaden their skill sets during these assessments, while faculty quickly adapt assessment methods to follow clinical practice.

Limitations of this study include a small sample size and a specific focus on SSBAs in a group of peer institutions. The sample size in this study is limited to academic members of the BTAA‐PBAC. Given the variation of how different parts of the country handled the COVID‐19 pandemic locally, institutions outside the Big Ten may have faced different challenges, thus needing other solutions. However, as unpredictable as the pandemic has been in various parts of the country at times, these strategies may apply more broadly at various time points for other institutions. Also, this study focused on SSBAs. Different strategies and challenges may need to be considered for various other types of assessment. Additionally, because of the timing of the survey, our study was unable to collect information regarding the specific plans for assessing skills that needed to be postponed. Schools and colleges of pharmacies should collaborate through professional organizations and other mechanisms to share ideas and disseminate information about how best to assess skills like injection technique that do not lend themselves to a virtual environment. With a COVID‐19 vaccine approved, it is now especially critical that students are adequately prepared and competent in performing this skill as pharmacists and student learners will be on the frontline for this important public health undertaking. Finally, since the initiation of this study, all institutions are continually learning and adjusting the curriculum to meet the needs of students and the learning objectives of our programs. It is likely that more lessons have been learned and more modifications have been added that have not been included in our survey data during 2020‐2021. We believe the information is still applicable and that publishing the point that we started is valuable as we reflect upon how COVID‐19 has impacted our teaching.

Future directions of this study include distributing these findings to other institutions, as well as possibly using this survey to identify broader trends across the country in how summative skills‐based assessments have been affected by the pandemic. The COVID‐19 pandemic has presented significant opportunities for academics to analyze and prioritize learning objectives and to rethink how to develop and deliver SSBAs. It has opened up opportunities to incorporate a wider array of technologies and telehealth skills. The changes, modifications, and pivots utilized during this unprecedented time may continue to be incorporated after the pandemic at those institutions that have seen positive effects as a result.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

TABLE A1

Institution‐specific responses to open‐ended items regarding challenges and strategies

What was the most challenging aspect of the COVID‐19 pandemic and its effect on summative assessments within professional skills laboratories?	What were the strategies that your institution implemented during the COVID‐19 pandemic that made the transition to remote‐learning easier?
• No time to prepare or familiarize with technology • Assessments could not mirror the format in which students learned the skills, student stress due to pandemic, and high‐stakes nature • Instructor stress due to pandemic and workload • Loss of regular supportive interactions with students • Inability to teach/evaluate certain hands‐on activities such as physical assessment, injection technique, compounding	• Extended spring break by a week • Priority from top leadership on holistic student wellness and determining minimum competency as opposed to getting lost in the weeds of online pedagogy (applies to Spring 2020 only)
• It was most difficult for assessments that could not be converted to an online assessment (such as immunizations)	• IT was assigned to required courses to help with the transition • All faculty had to do an addendum to their syllabus; the administration assisted with wording
• Making the remainder of the semester remote while converting grading rubrics and planning on how to evaluate students in the short period of time. If our Spring Break was not already scheduled, we would not have had a week to plan remote learning. We were fortunate to have at least 1 week planning vs 1 day • Although some faculty members were proficient in using remote technology, others were not prepared or never used the new programs	• Providing and investing in digital resources while providing examples of remote learning with best practices
• Short time frame to make huge changes created a lot of stress • Additionally, I did not have very many people to help make this transition	• Zoom • Frequent communication • Detailed instructions for students
• One of the most challenging aspects of the transition was determining the best structure of the lab to allow for all students to have equal access to review and complete the lab. Not all students had good internet connection for streaming lectures, and some students were in different time zones that prohibited synchronous activities	• Frequent communication between lab coordinators facilitated and standardized the methods of providing lab materials to students • The College of Pharmacy Dean's office also provided frequent communication to instructors and guidance for transitioning to remote learning. The information and guidance from administration was helpful in guiding our decisions • Once the class moved to remote learning, the assignments and lab activities did not necessarily have to be completed at the originally scheduled [time]. It was helpful to think about structuring lab in such a way that fewer days were used for synchronous activities. For example, P3s were evaluated in their summative assessments over 2 days, instead of five separate days. This helped streamline scheduling students and faculty
• Instructor standpoint—logistics (how to make everything work) • Student standpoint—adding stress to an already stressful assessment/situation	• Communication within the teaching team and collaborating with others (lessons learned, etc.) • Proactively establishing plans well in advance of the university closing
• Determining if skills could be assessed remotely or if needed to be postponed	• IT support during each class and online simulationWebinars how to utilize different online teaching resources • Access to Blackboard Collaborate and Zoom
• Overall, the team spent a significant amount of time determining the most critical objectives to assess as previously planned and converting these assessments to an online platform. It was difficult to maintain confidence in the academic integrity of our assessments. In addition, while we assessed their knowledge of compounding skills, we were unable to evaluate their performance of these. All of this was compounded by a variety of circumstances our students faced at home	• Collaborating with the skills lab team, as well as the BTAA‐PBAC, were instrumental in the overall success of our rapid transition

Abbreviations: BTAA‐PBAC, Big Ten Academic Alliance‐Performance Based Assessment Collaborative; COVID‐19, coronavirus disease 2019; IT, information technology; P3, third‐year pharmacy student.

TABLE A2

Institution‐specific responses to open‐ended items regarding recommendations

Some institutions are planning to have some restrictions continue into the next academic year. What are the strategies or changes you think are most important to share with others as they make plans for future summative assessments within professional skills laboratories?
• Have a backup plan • Peer evaluation can be a useful tool • New ways to ensure academic integrity are needed • Prioritizing learning goals/eliminating excess • Re‐evaluating efficient use of personnel
• It is important to reexamine what are the critical skills, when those skills need to be met/what progression is necessary at this time, and to be flexible (in order to move things around as needed)
• Plan to have equipment usually available in person (eg, blood pressure cuffs, blood glucose monitors, demo inhalers, etc.) available to students in some capacity via mail or pickup • Also, identifying what are essential skills that need to be evaluated (in person or remotely) and potentially incorporate them into future semesters, activities, and experiences
• Be flexible and have a back‐up plan in place before you start the semester • Telehealth skills are key and should be incorporated into the curriculum • You can never explain instructions too much
• It is important to be flexible in planning such that the lab material can more easily be switched to entirely remote learning. Do this by working closely with your lab instructors and guest presenters • Provide frequent communication to students and faculty about lab updates and changes • Optimize use of electronically‐based assessments, instead of paper assessments. This will increase the flexibility in providing the assessments as an asynchronous activity, and it will decrease contact between students • Think about using your technology differently than you do. Think about different ways you can use the same tool to complete different activities and assessments in lab
• Connecting with SP pool and administration to see what is possible to complete virtually • Purposeful selection of tools to use in the classroom based on prior experience
• Determine what skills are necessary for students to demonstrate and if those skills can be assessed remotely or in person
• Communicate regularly and often • Have back‐up plans and multiple options available for students to schedule any synchronous assessments

Abbreviation: SP, standardized patient.