Virtual Evaluation of Clinical Competence in Nurse Practitioner Students.

All in-person nurse practitioner (NP) student clinical activities were suspended in the spring of 2020 due to COVID-19. This posed a unique summer semester challenge, as students needed to complete an objective structured clinical examination (OSCE). The traditional face-to-face OSCE was revised into a virtual format using a teleconferencing platform and included the following learning objectives: obtain the medical history, describe a focused physical examination, formulate differential diagnoses, and develop an appropriate plan of care. Together with the Health Education Center, students met with a standardized patient (SP) virtually. All elements of the virtual OSCE were the same as the traditional OSCE, except students were required to demonstrate their clinical assessment skills by verbalizing to faculty what they would examine if the visit were in-person. When finished, all participants were invited to complete a survey about their experience. Survey findings revealed that most faculty and students considered the interactive virtual OSCE an extremely effective tool for assessing communication and history taking skills, differential diagnosis, and management of patients. All SPs felt comfortable communicating with the students and felt that the virtual OSCE was a very effective way to assess their interpersonal skills of students. The virtual OSCE also served as an opportunity to incorporate telehealth competencies into a simulation experience for students. This innovative distance learning activity facilitated effective virtual evaluation of clinical competence in NP students and all stakeholders expressed satisfaction with the experience. Most faculty and students strongly agreed that they wanted to continue using the virtual OSCE platform. © International Association of Medical Science Educators 2021.

Background

Nurse practitioners (NPs) play a vital role in our healthcare system by improving access to high-quality healthcare services. As the demand for NPs increases, so does the need for online programs which can expand their reach to include distance students. Although many NP programs utilize an online format for their didactic content, most programs still require students to travel to campus periodically for structured clinical examinations. The National Task Force on Quality Nurse Practitioner Education (NTF) has developed a set of criteria that are used to evaluate NP programs and mandate that student clinical competence and performance must be evaluated through observation [1].

In spring of 2020, the NP programs at the University of Texas Medical Branch at Galveston (UTMB), like many others nationwide, were significantly impacted when in-person student clinical activities were suspended due to COVID-19. This posed a unique summer semester challenge, as students needed to complete their objective structured clinical examinations (OSCEs). With the pandemic imposing travel restrictions and mandating social distancing, an alternative method of clinical evaluation was needed. After reviewing the literature and collaborating with the UTMB Health Education Center (HEC), faculty in the primary care NP programs implemented a virtual OSCE clinical simulation during Summer 2020. It was determined that the planned simulation would not only alleviate the need for travel and allow for social distancing but would incorporate telehealth competencies which would provide an important clinical immersion experience for students. These competencies included telehealth etiquette and professionalism, proficiency in history taking, and generating differential diagnoses while using videoconferencing technology [2]. The National Organization of Nurse Practitioner Faculties (NONPF) supports and encourages NP programs to incorporate telehealth into their curriculum, and there is a growing body of evidence supporting successful implementation of virtual/telehealth assessments into health and medical education [2-7].

Activity

The Model for Improvement [8] and the Association for Standardized Patient Educators (ASPE) Standards of Best Practice served as the framework for this project [9]. The goal of the project was to implement virtual OSCEs using highly trained standardized patients (SPs) and facilitate faculty evaluation of NP student clinical learning objectives. The virtual method needed to be feasible and easily implemented in nursing education. It also needed to be effective, which would be demonstrated by faculty being able to gather enough information from the platform to complete the same summative evaluation used for the in-person OSCE. Finally, it needed to be efficient, as measured by time spent virtually versus the in-person method.

Prior to implementation, project approval was received from the UTMB Institutional Review Board (IRB). The number of participants in the project was determined by convenience sampling. All students enrolled in clinical courses during the summer semester in the primary care NP programs at UTMB were part of the project as they were required to participate in OSCEs as part of their curriculum. In total, 14 faculty, 95 primary care NP students (in their third clinical semester), and 18 SPs participated in the project. Participant and respondent numbers are illustrated in Table 1. Ages of all participants ranged from 20 to 70 years of age. The project was implemented during the summer semester in two separate courses on two different dates in one month.

Table 1

Participants and survey completion

Participants	Received surveyTotal (n)	Completed surveyn (%)
Faculty	14	12 (85.7)
Students (FNP + AGPCNP)	95 (82 + 13)	76 (80.0)
Standardized patients	18	17 (94.4)

FNP family nurse practitioner, AGPCNP Adult-Gerontological Primary Care Nurse Practitioner

The traditional in-person OSCE was revised into a virtual format using the video-conferencing platform Zoom® [10]. Learning objectives included the following: obtain the medical history, describe a focused physical examination, formulate differential diagnoses, and develop an appropriate plan of care. Students and faculty received separate training on the use of Zoom® prior to the scheduled OSCE. Facilitated by the HEC, students met virtually with an SP to complete a medical interview while faculty observed and evaluated student performance. The virtual OSCE contained all the elements of the traditional assessment, except that students were instructed to verbalize to faculty what they would examine if the visit were in person. Faculty notified the student of any abnormal findings, and the student then had a few minutes to develop differential diagnoses and a plan for the working diagnosis. Finally, the student presented the patient’s pertinent history to the faculty and the diagnoses, including the working diagnosis, and the plan of care. Debriefing occurred after the student presentation. The SPs also provided written feedback for students regarding their interpersonal skills. Upon completion of the activity, both quantitative and qualitative data measuring the efficacy of the virtual OSCE were collected from participating students, faculty, and SPs using self-administered surveys accessed via the online survey tool SurveyMonkey® [11].

Separate surveys were developed for each participant type. Faculty were asked to rank the efficacy of the virtual OSCE in evaluating key clinical competencies (history taking, clinical assessment, differential diagnosis, patient education, and management and communication) using Likert scale survey items. Responses were on a 5-point scale, and participants were asked to rank efficacy as either “extremely effective,” “very effective,” “moderately effective,” “somewhat effective,” or “not at all effective.” Faculty were also asked to rank efficacy of student debriefing. Student participants were asked to rank the efficacy of the virtual OSCE in allowing them to demonstrate the aforementioned key clinical competencies using the same 5-point response scale. Standardized patients (SPs) were queried about how effective the virtual OSCE was in allowing assessment of student’s interpersonal skills, again using the same 5-point efficacy scale. Faculty and student participants were also surveyed using a 6-point response scale “strongly agree,” “agree,” “neither agree nor disagree,” “disagree,” or “strongly disagree” to collect data regarding whether the virtual OSCE saved them time and effort. Students used the same 6-point scale to indicate whether the activity was useful and valuable to learning and enhanced performance, and if they would be more comfortable conducing virtual patient visits in the future. Dichotomous questions (yes/no) were used in all participant surveys to identify ease of platform use, whether technical difficulties were experienced and whether participants would like continued use of the virtual OSCE process. For all survey items, an open comment option was included to collect qualitative data. It should be noted that open comment response rate varied by survey item and participants.

Analysis of survey responses was provided by SurveyMonkey® as aggregate data. Quantitative data was further analyzed using descriptive statistics (frequency and percentage). Qualitative feedback data from all surveys were reviewed and closely examined using thematic analysis. Identification of common topics, keyword connectors, and ideas was used to identify the key themes as well as the percent of respondents that provided qualitative data is summarized in Table 2.

Table 2

Qualitative survey data summarized by theme

	Survey question	Major Themes	% Participants contributing to themes
Faculty survey	Save time and effort	• Saved commuting time • Required extra effort timing	25% (n = 3)
	Efficacy in evaluating clinical assessment skills	• Need more time to verbalize exam • Cannot assess exam techniques virtually	66.7% (n = 8)
	Continued use of virtual OSCE	• Need at least one or two face-to-face OSCE during program to assess PE skills • Virtual OSCEs provide needed telehealth experiences	41.7% (n = 5)
Student survey	Efficacy in demonstrating clinical assessment (physical exam) skills	• Two-minute time frame too short • Variability in faculty expectations • Verbalizing a physical skill was difficult	30.3% (n = 23)
	How virtual OSCE influenced performance	• Created anxiety anticipating technical issues (internet etc.) • Nervous because new process/new platform • Lack of familiarity with performing telehealth visits • More comfortable because in home environment	47.4% (n = 36)
	Comfort with virtual patient visits (telehealth) in future	• First telehealth visit for many • More comfortable but would like more exposure to telehealth experiences in the program	7.9% (n = 6)
	Continued use of virtual OSCE	• Good alternative during COVID • In person OSCE preferred method for demonstrating PE skills	26.3% (n = 20)
	Save time and effort	• Saved drive/travel time/time off work • Saved travel expenses • Reduced travel related stressors	30.3% (n = 23)
Standardized patient (SP) survey	Efficacy in assessing student interpersonal skills	• No different than in person • Able to take notes out of view	41.2% (n = 7)

Results

Faculty Results

One hundred percent (n = 12) of faculty reported the Zoom platform was easy to use, with only 25% (n = 3) reporting technical problems or connection issues with the platform.

Seventy-five percent (n = 9) of faculty rated the virtual OSCE as an “extremely effective” method to evaluate student communication skills, with an additional 16.7% (n = 2) reporting it as “very effective” (Fig. 1). Fifty-eight percent (n = 7) of faculty found the virtual OSCE to be an “extremely effective” method in evaluating students’ history taking skills, and an additional 42% (n = 5) found it “very effective.” The majority of faculty (41.7%, n = 5) felt the virtual OSCE was only “moderately effective” in evaluating students’ clinical assessment skills, and a third (33.3%, n = 4) felt that it was only “somewhat effective.” For both evaluations of student differential diagnoses and patient evaluation and management, 75% (n = 9) of faculty reported the virtual OSCE as an “extremely effective” method of evaluation. Fifty percent (n = 6) of faculty found the method “extremely effective” for student debriefing, and approximately 42% (41.7%, n = 5) found it “very effective.”

Fig. 1

Virtual OSCE efficacy rating for key clinical competencies

The majority of faculty (75%, n = 9) “strongly agreed” that the virtual OSCE allowed them to save both time and effort, with about 17% (16.7%, n = 2) of faculty selecting “neither agree nor disagree.” Elimination of faculty travel time was cited as an advantage (Table 2). One hundred percent (n = 12) of faculty were in favor of continuing the virtual OSCE process.

Student Results

Ninety-six percent (96.1%, n = 73) of students reported instructions and guidance for virtual OSCE were clear and understandable. Ninety-seven percent (97.4%, n = 74) of students found Zoom easy to use with only about 11% (10.5%, n = 8) citing any technical difficulties.

Sixty-six percent (65.8%, n = 50) felt the virtual OSCE was “extremely effective” in allowing them to demonstrate history taking skills (Fig. 1). Only twenty-nine percent (n = 22) of students found it an “extremely effective,” 31.58% (n = 24) “very effective,” and 30.3% (n = 23) “moderately effective” method to demonstrate clinical assessment skills. The majority (61.8%, n = 47) found it an “extremely effective” method to demonstrate their ability to develop differential diagnoses. Sixty-six percent (65.8%, n = 50) found it an “extremely effective” method to demonstrate patient education and management and communication skills and to receive learning feedback. More than half (57.9%, n = 44) of students reported they felt that using a virtual format “influenced” their performance.

Eighty-three percent (82.9%, n = 63) of students would like to continue to use the virtual OSCE process, and most students (61.8%) “strongly agreed” it allowed them to save time and effort. Sixty percent (60.5%, n = 46) “strongly agreed” that they would be more comfortable performing telehealth visits in the future. Student qualitative survey responses are summarized by theme in Table 2.

Standardized Patient Results

One hundred percent (n = 17) of SPs felt comfortable interacting with students via the Zoom platform. Seventy-one percent (70.6%, n = 12) of SPs felt the virtual OSCE was a “very effective” method to evaluate students’ interpersonal skills, and almost 20% (17.6%, n = 3) reported it as “extremely effective” (Fig. 1). All SPs (100%, n = 17) reported that they would agree to participate in a virtual OSCE again in the future (Table 2).

Discussion

This project demonstrated that the virtual OSCE was a comparable and acceptable method for evaluation of primary care NP student clinical competence and can be used to fulfill clinical learning objectives. Students, faculty, and SPs all expressed satisfaction with the virtual method of evaluation. The virtual OSCE also provided a valuable telehealth simulation experience for students and has given faculty a solid foundation for developing and integrating further telehealth simulations into the curriculum. This integration of telehealth clinical experience aligns with the NP Core Competency of Technology and Information Literacy as delineated by NONPF [12].

Some limitations were observed during project completion. Since the case scenarios were not true telehealth visits, the students were required to “demonstrate” their clinical assessment skills by verbalizing to faculty what they would examine if the visit were being performed in-person. The faculty noted that verbalization of the physical exam required more time than allotted and assessment of physical examination technique is difficult in a virtual setting. These findings could be suggestive of the need for face-to-face evaluation of student physical examination skills at some point in the program. Despite this limitation, both students and faculty expressed interest in having more virtual/telehealth simulation experiences in the future.

The SP perspective was valuable for assessing students’ interpersonal skills in a virtual setting, and it was affirming that they found no difference between the virtual OSCE and the in-person method. It was also reassuring that the SPs felt connected to the students during the virtual OSCE. These findings are important as evidence has shown health outcomes, and patient compliance improves when the patient feels connected and listened to by the care provider [13-15].

Future implications of this project include further exploring the development of telehealth learning objectives, competencies, and skills for NPs and other healthcare professionals through the development of true telehealth case scenarios and their inclusion in curricula. This innovative distance learning activity was successful and facilitated meaningful learning and evaluation during a global pandemic. Using technology for remote assessments via telehealth and the utilization of technology in medical education is trending globally and will continue beyond pandemic times.