Acceptability and Feasibility of Virtual Behavior Analysis Supervision.

The COVID-19 pandemic necessitated a rapid transition to virtual service delivery and supervision. This preliminary study examined acceptability and feasibility of virtual supervision for 94 BCBA/BCaBA trainees during COVID-19, including variables that affected perceived satisfaction, effectiveness, and supervision preference for this sample. Results indicate a decrease in accrual of direct client hours during the pandemic, with a third of participants reporting a decrease in individual supervision. In general, participants were satisfied with virtual individual and group supervision as indicated by high satisfaction domain scores and individual item means, with minimal overall change in satisfaction. Participants indicated preference for in-person or hybrid supervision and considered in-person most effective. In general, participants reported that virtual supervision was feasible and supervisors used best-practice strategies. We discuss variables that affected satisfaction (e.g., length of supervisory relationship), preference (e.g., age, services provided), and perceived effectiveness (e.g., time supervisor was a BCBA). We provide practical implications and recommendations for virtual supervision. © Association for Behavior Analysis International 2021.

Supervision is a critical component of the training process toward becoming a board certified behavior analyst (BCBA) or board certified assistant behavior analyst (BCaBA). According to the Behavior Analyst Certification Board (BACB), the goal of supervision is to build and maintain the professional competence of the trainee and to ensure that the trainee’s clients are receiving the highest quality care. Within the supervisory experience, trainees are encouraged to gain practical experience in the varying responsibilities of a BCBA. These responsibilities include conducting behavioral assessments, implementing and monitoring skill acquisition and behavior-reduction programs, writing treatment plans, and overseeing others in the implementation of behavior plans (BACB, 2020a).

Trainees in the field of applied behavior analysis must accrue hours toward their BCBA or BCaBA, with the number of hours increasing because the task list for the BACB, Fifth Edition, is set to go into effect in January 2022 (BACB, 2020b). Trainees must accrue a minimum of 20 to a maximum of 130 supervised hr per month. Supervised hours include hours directly engaged with a client delivering specific “therapeutic or instructional procedures” (i.e., restricted hours; BACB, 2020a, p. 12), and hours related to behavior analytic duties such as conducting assessments, analyzing data, and writing reports (i.e., unrestricted hours), with a limit to the percentage of total hours that can be restricted hours (BACB, 2020b). A minimum of 5% of hours must be supervised by an approved BCBA supervisor. An integral part of supervision is the supervisor directly observing the trainee with a client each month (i.e., client contact; BACB, 2020b). The literature has highlighted best-practice recommendations for supervision of aspiring BCBAs (e.g., Garza et al., 2018; Sellers et al., 2016; Valentino et al., 2016). For further information, we recommend the reader reference a special issue in Behavior Analysis in Practice by LeBlanc and Luiselli (2016) that highlights essential elements of supervision (e.g., building and maintaining supervisory relationships, ethical principles of supervision, conflict resolution). Such guidelines have been extended to encompass virtual supervision. In a recent publication, Britton and Cicoria (2019) provide guidance to BCBAs regarding providing virtual supervision for trainees. Authors describe contexts for growing consideration of virtual supervision, including students in online programs seeking supervisors that may not be available in their local communities and trainees in rural or underrepresented areas. Further, authors describe challenges to virtual supervision including ease of interaction, demonstration of competency of key behavior analytic skills, and maintenance of confidentiality. Further empirical studies are warranted on trainee’s experiences with supervision, especially with regard to virtual supervision.

The COVID-19 pandemic has greatly affected the lives of individuals around the world. The nature of the virus has necessitated individuals, communities, and institutions to assess the need for in-person interactions and move to a virtual modality to reduce the spread of the virus. This move to virtual work was initially intended to both conserve personal protective equipment for healthcare providers and to reduce the overall spread of the virus to “flatten the curve” of COVID-19 cases (Caravella et al., 2020). However, the move to virtual modalities has had a significant impact on how individuals, communities, and institutions function.

Although the pandemic has affected the functionality of many occupations, the effects are particularly significant for healthcare professions that require in-person interactions with clients to deliver treatment services. In addition to disrupting clinical services, the COVID-19 pandemic also affected trainees’ opportunities for continued education and supervision (Caravella et al., 2020). The COVID-19 pandemic in the United States led to the closure of institutions of higher education, restriction of practicum and internship placements, and a shift to virtual learning in March 2020 (Kim, 2020). This sudden move to virtual modalities conceivably affected the nature of supervision for trainees. To mitigate the potential disruption in services and supervision, many supervisors had to rapidly adapt to virtual platforms in order to maintain supervisory requirements.

The effects of COVID-19 likely affected trainees obtaining fieldwork hours for the BCBA and BCaBA, across settings. Schools are one area of practice for a subset of trainees in which the effect of the pandemic was specifically documented. Due to the sudden closure of schools at the start of the pandemic, some trainees working in school settings temporarily halted accruing hours until adaptations could be made. Although such adaptations were made (e.g., virtual platforms, in-home services), trainees may have experienced a reduction in fieldwork hours, raising concerns about a quality supervisory experience (Fronapfel & Demchak, 2020). The difficulty of meeting the supervision requirements during the COVID-19 pandemic resulted in the BACB temporarily waiving the “observation with a client” requirement (BACB, 2020c). This waiver was instated to ensure that trainees who did not have the ability to work with their clients due to COVID-19 restrictions could still accrue hours towards their BCBA and BCaBA.

Although virtual supervision has recently been adapted in the field of behavior analysis, it has a long history of use across fields such as psychotherapy (Caravella et al., 2020), counseling (Nadan et al., 2020), education (Kim, 2020), speech therapy (Dudding & Justice, 2004), social work (Panos et al., 2002), and medical services (Wearne et al., 2015). Overall, studies have shown that virtual supervision has maintained adequate quality of supervision and positively affected the development of trainee competence in multiple domains of counseling and psychotherapy (Manring et al., 2011; Nadan et al., 2020). In a particularly noteworthy study, Caravella et al. (2020) explored the use of virtual supervision in response to the COVID-19 pandemic in the New York University Langone Health Consultant Liaison Psychiatry Service. The hospital service created a virtual rotation for psychiatric trainees that maintained supervision and provided exposure to all requisite aspects of the trainee learning experience. Virtual supervision allowed supervisors to sustain supervision of psychiatric trainees and to provide opportunities for training when in-person supervision was not feasible.

The overall shortage of BCBAs (Rios et al., 2018), the high percentage of BCBAs overextended with supervision responsibilities (Sellers et al., 2019), and the growth of trainees seeking supervision outside their immediate area (Britton & Cicoria, 2019) may have influenced the introduction of virtual supervision in the field of applied behavior analysis. To understand the barriers to supervision within behavior analysis, Sellers et al. (2019) surveyed current BCBA supervisors and found that supervisors reported that their schedules did not allow them to adequately prepare for supervisory sessions, discussions, or meetings with their trainees. With a high level of burnout in the field of behavior analysis (Plantiveau et al., 2018), COVID-19 conceivably intensified the marked stress placed on BCBAs to provide ethical and effective supervision. Fronapfel and Demchak (2020) indicated that virtual supervision technologies allow trainees to practice and demonstrate various behavior analytic skills with their supervisor through either an individual or group videoconference. These findings highlight the potential of virtual supervision for avoiding barriers to effective supervision in behavior analysis, both during the COVID-19 pandemic and beyond.

There are many potential advantages of supervisors incorporating virtual supervision into their supervisory relationship. Virtual supervision allows for increased flexibility in scheduling (Amodeo & Taylor, 2004), decreased distance and travel costs for supervisors (Panos et al., 2002), access to supervision in rural areas (Wood et al., 2005), and decreased reactivity of clients (Israel et al., 2009). Decreasing reactivity (i.e., behavior change due to an extraneous variable) is especially pertinent in behavior analysis with children who engage in challenging behaviors maintained by access to attention (Farley, 2019). In addition, researchers have found that virtual supervision positively affects the supervisory experience (Nadan et al., 2020) and increases the learning experience of trainees (Manring et al., 2011).

However, there are also potential disadvantages to using virtual supervision that have been documented in other fields. For example, when a supervisor is supervising a live session using virtual platforms, the session flow may be disrupted due to lengthy communication processes or missed feedback (Nadan et al., 2020) and technical difficulties (Chen et al., 2020). Further, Wilczenski and Coomey (2006) suggest that virtual modalities can create an impersonal environment where trainees may engage in inappropriate behaviors without recognizing the personal impact, and the use of virtual supervision could make it difficult for the supervisor to identify some inappropriate or unethical trainee behaviors that would otherwise be evident during in-person observation. For example, a trainee delivering virtual services may be in a location that does not maintain client confidentiality, may simultaneously engage in personal activities on their device that are not visible to the supervisor, or may discuss client behaviors within earshot of the client and family. A trainee delivering in-person services may engage in inappropriate body language (e.g., eye rolling) toward the client or in response to feedback that is not visible on camera, may use incorrect physical positioning during prompting procedures that the supervisor cannot detect, or may discuss client behaviors while the client and family are present but not visible to the supervisor on camera. Finally, virtual supervision raises concerns of maintaining confidentiality and security of client information during virtual communication (Wilczenski & Coomey, 2006).

The current study aimed to conduct a preliminary examination of the acceptability and feasibility of virtual supervision for BCBA/BCaBA trainees during the COVID-19 pandemic, including potential variables that contribute to perceived satisfaction, effectiveness, and supervision preference. The goal of this study was to obtain preliminary data on trainees’ experiences with virtual supervision and to document virtual behavior analytic supervisory practices with our sample that could lead to further research evaluations. We also explored the practical implications of virtual supervision and recommendations for virtual supervisory practices to maintain quality supervision in the field of applied behavior analysis.

Method

Recruitment

We recruited individuals in the United States who received virtual supervision toward their BCBA or BCaBA certification between March 1, 2020 and August 31, 2020. Participants were recruited through online platforms (e.g., Facebook groups, Reddit), listservs, and email invitations sent to the coordinators of all programs in the United States with an Association for Behavior Analysis International verified course sequence. Although we do not have data on the number of trainees who received the invitation to participate, our recruitment efforts aimed to capture a broad range of trainees across the United States. A total of 136 responses were collected: 6 (4.41%) reported that they did not receive virtual supervision, 1 (0.74%) did not reside in the United States, 28 (20.59%) did not complete at least 60% of the online survey, and 7 (5.15%) did not complete all survey questions in at least one of the satisfaction and change domains, yielding a final sample size of 94 participants (69.12%; see Fig. 1). We excluded participants who did not complete satisfaction and/or change domains as these were our primary dependent variables.

Fig. 1

Consort Chart for Recruited Participants

Response Definitions

We defined the period from March 1, 2020 through the end of data collection on August 31, 2020 as “during COVID-19” in our survey measures. March 2020 was selected because (1) a national emergency was declared in the United States on March 13, (2) individual states instated lockdown measures and other restrictions, and (3) confirmed cases reached all 50 states. For the remainder of the manuscript, we will use the terminology “during COVID-19” for ease of discussion. We refer to any activities deemed supervision by the BACB and delivered through virtual means as virtual supervision. We defined virtual as any communication or observation between supervisor and trainee in which both parties were not physically present in the same location (e.g., video conference, phone call, instant message, text message, email). Virtual supervision may include any of the following activities where the supervisor is virtual: (1) client contacts (i.e., telehealth or in-person service delivery); (2) individual supervision meetings; (3) group supervision meetings; and (4) asynchronous communication (e.g., written feedback via email).

Measures

Survey Development

Due to the widespread necessity of virtual supervision during the COVID-19 pandemic, this online survey was developed by the authors prior to conducting this study to measure the feasibility and acceptability of virtual supervision, with input from both BCBA/BCBA-D supervisors and aspiring BCBA/BCaBA trainees. Prior to this study, the survey was piloted by trainees and additional revisions were made based on their feedback. The online survey was administered using the Qualtrics® survey platform (2021). Questions included yes/no responses, multiple choice responses, Likert scale ratings, and open-ended responses. The survey took a median of 11.10 min to complete (range: 4.47–230.43 min).

Questions were grouped by conceptual similarity and yielded total scores in four domains. Cronbach’s alpha was used as a general measure of the item interrelatedness (Tavakol & Dennick, 2011). Domains included: (1) satisfaction with virtual individual supervision (10 items, α = .93, interitem correlations ranged from .39 to .86); (2) change in individual supervision (11 items, α = .90, interitem correlations ranged from .18 to .82); (3) satisfaction with virtual group supervision (7 items, α = .94, interitem correlations ranged from .67 to .92); and (4) change in group supervision (7 items, α = .94, interitem correlations ranged from .60 to .88). Skewness and kurtosis values (i.e., to characterize the shape and distribution of the data and identify outliers) were within normal limits (+/-2) of a univariate distribution (George & Mallery, 2016) for satisfaction scores. In particular, we analyzed these variables to determine whether our data were normally distributed (i.e., symmetrical with a similar number of low and high values and most values around the middle) to inform the statistical analyses conducted. Within the domains of change in individual supervision and change in group supervision, kurtosis values were outside of normal limits of a univariate distribution. When analyzing individual participant data of the 94 included participants for change in individual supervision, we detected four outliers in participants’ data with overly high change scores (> 2 SD above M; i.e., > 44.85). As such, we removed these participants’ data (4.26%) from all analyses involving change with individual supervision. Likewise, within the domain of change in group supervision, we detected eight outliers with overly high change scores (> 1.5 SD above M; i.e., > 29.97) and excluded these participants’ data (8.51%) from all analyses involving change with group supervision. We chose to remove these outliers as the few overly high reports of positive change that were greater than 1.5 standard deviations above the mean may have been influenced by participants’ overall positive perceptions of their supervisor or supervision placement itself leading to selecting “greatly improved” for all variable options or selecting down the row of response options without attending to individual items. Further, we intended for our preliminary results to generalize to the typical experiences of behavior analytic trainees.

Procedure

Participant consent was obtained in Qualtrics prior to accessing the survey and personal identifiers were not collected as part of the survey to provide anonymity. The survey began on July 31, 2020 and ended on August 31, 2020. Participants who provided their contact information in a separate survey upon completion were entered into a random drawing for a $100 digital gift card.

Data Analysis

We calculated individual supervision and group supervision satisfaction scores by applying numerical values to each Likert scale question in the corresponding domain and summing the item values for each participant. For example, each participant responded to 10 questions about satisfaction with individual supervision on a 5-point Likert scale ranging from greatly dissatisfied (1) to greatly satisfied (5). If a participant responded to each of the 10 questions with a rating of 5, we summed together each of the item scores (5 * 10) to obtain a domain score of 50. Likewise, we calculated individual supervision and group supervision change scores by applying numerical values to each Likert scale question in the corresponding domain and creating a total sum score for each participant.

We conducted statistical analyses to determine the statistical significance of selected variables on satisfaction, change, preference, and perceived effectiveness. A resulting p-value less than .05 was deemed statistically significant. Whereas our sample is representative of BCBAs across the United States (e.g., geographic location, gender, race/ethnicity, area of professional emphasis; BACB, 2020d), the small number of participants selecting specific survey options limit the conclusions that can be drawn from statistical analyses. We propose these results as a preliminary investigation into the status of virtual supervision during COVID-19 and as a methodological framework for future investigation. Simple linear regression was conducted for two continuous variables (e.g., age and satisfaction). Chi-square tests were conducted for categorical independent variables (e.g., type of service provided) and categorical dependent variables (e.g., preferred method of supervision). Post-hoc chi-square tests were conducted if main effects were significant. Multinomial logistic regression was conducted for a continuous independent variable (i.e., age) on categorical dependent variables (e.g., effectiveness of supervision). One-way analysis of variance (ANOVA) was conducted to analyze differences between polytomous (i.e., more than two options) categorical independent variables (e.g., amount of time BCBA was supervisor) on continuous dependent variables (e.g., satisfaction). Post-hoc Tukey tests were conducted if a significant main effect was detected. Linear regression was conducted for a dichotomous (i.e., two options) categorical independent variable (i.e., type of service) and continuous dependent variables (e.g., satisfaction score). Categories with two or fewer responses were removed from analyses.

Open-ended responses to preferred method of supervision and method of supervision that was considered most effective were thematically analyzed to determine common themes identified across participants using the constant comparative method (CCM) of qualitative data analysis, first described by Glaser and Strauss (1967) and refined by Strauss (1987). See Olson et al. (2016) for a discussion of the evolution of CCM methodology and a description of the 10-step processes adapted for the purposes of this analysis. In particular, the researchers allowed themes to emerge from participant responses. Researchers extracted the written responses and sorted them into categories based on participants’ selections for modality preference and modality perceived as most effective (e.g., in-person, virtual, hybrid). In chronological order of responses, two research assistants trained in qualitative data analysis independently coded the extracted written responses (i.e., not associated with any other participant data) into as many categories of analysis as possible. Categories were added as they emerged, and data were fit to existing categories. As subsequent text was coded, each incident of a category was compared with previous incidents of that category across participants. Subthemes were listed under categories to create an accurate operational definition of each category. When incidences could potentially be coded into multiple categories, specific rules and exclusionary criteria were developed. Categories were informed by the language of the participants. Researchers collaborated by comparing categories across data collectors to arrive at a final categorization of the data. Both researchers recoded data using unified categorizations. A point of saturation was reached when no new themes emerged from additional participants. We determined salience of themes by summing the frequency of mention of that theme across all participants. To determine interobserver agreement (IOA), referred to as intercoder reliability by Olson et al., we randomly selected 34.49% of participant text responses, across preference categories, and an independent data collector trained in qualitative data analysis scored these data with the categories we provided. We defined an agreement as both observers scoring a text response with the same category or categories. We calculated IOA using the formula: [agreements / (agreements + disagreements)] *100. Interobserver agreement was 92.5%. Disagreements included not scoring individual responses into more than one relevant category.

Results

Participant Demographics

Responses were collected from 136 participants. Ninety-four participants who met all inclusion criteria were included in the final sample (see Fig. 1 for each exclusion criteria met), with outliers removed for individual analyses, where relevant. The sample included participants from 24 states with the largest representation from New Jersey (17.02%), Pennsylvania (15.96%), and California (7.45%). Participants were between 21 and 55 years of age (M = 31 years) and predominantly female (93.62%). A majority of participants identified as white (75.33%), followed by Asian (9.57%), Hispanic/Latinx (8.51%), Black/African American (4.26%), and other (2.13%). Most participants reported being enrolled in a master’s program (82.98%) and accruing supervised hours from 1–3 months through 4 or more years (Mode = 7–12 months [39.36%]). Participants provided multiple types of services across a variety of settings during the 3 months before COVID-19. See Table 1 for a breakdown of location of services, type of services, and ages of clients served, across participants. Most participants reported that their supervisor was female (91.49%) and had been a BCBA for 1–3 years (22.34%) or 3–5 years (27.66%).

Table 1

Services Provided Prior to COVID-19

	n	%
Location of Services
Home	42	44.68
School	42	44.68
Clinic	39	41.49
Residential	15	15.96
Community	12	12.77
Other	6	6.38
Services Provided
Behavior Reduction	87	92.55
Daily Living Skills	74	78.72
Academic Skills	51	54.26
Early Intervention	46	48.94
Staff Training	42	44.68
Caregiver Training	37	39.36
Vocational Skills	28	29.79
Feeding	18	19.15
Promoting Health-Related Behaviors	13	13.83
Organizational Behavior Management	5	5.32
Treating Addiction	0	0
Other	0	0
Age Range of Clients
0–5 years	60	63.83
6–10 years	56	59.57
11–14 years	35	37.23
15–17 years	25	26.60
18–24 years	23	24.47
25–34 years	11	11.70
35–44 years	9	9.57
45–55 years	7	7.45
Over 55 years	4	4.26

State of Affairs

During the 3 months prior to COVID-19, the majority of our participants (91.49%) reported providing exclusively in-person services, with small percentages providing virtual services (5.32%) and a hybrid of in-person and virtual services (6.38%). During COVID-19, the majority of participants in this sample reported that they, for some period, provided virtual services (70.21%), followed by in-person services (37.23%), and a hybrid of in-person and virtual services (25.53%). Two participants reported that they did not provide any services due to COVID-19, but still received virtual supervision.

The majority of our participants reported that the amount of direct client hours changed while they provided virtual services (83.33%) and in-person services (85.71%). See Table 2 for information on length of service delivery, number of clients served, and how client hours changed during COVID-19.

Table 2

Services Provided during COVID-19

	Virtual Services (n = 66)			In-Person Services (n = 35)
		n	%	n	%
Length Services Provided	< 2 weeks	3	4.55	10	28.57
	2–4 weeks	9	13.64	7	20.00
	4–6 weeks	10	15.15	5	14.29
	6–8 weeks	3	4.55	2	5.71
	8–10 weeks	5	7.58	3	8.57
	> 10 weeks	36	54.55	8	22.86
Number of Clients Served	1	15	22.73	10	28.57
	2	13	19.70	7	20.00
	3	2	3.03	5	14.29
	4	11	16.67	2	5.71
	5	2	3.03	3	8.57
	6+	23	34.85	8	22.86
How Hours Changed	Greatly decreased	28	42.42	11	31.43
	Decreased	21	31.82	14	40.00
	Stayed the Same	11	16.67	5	14.29
	Increased	5	7.60	3	8.57
	Greatly Increased	1	1.5	2	5.71

During COVID-19, over half of participants (59.57%) in this sample reported that there was a period that they did not meet the BACB’s “observation with a client” requirement due to the pandemic. Of these individuals, only 60.53% reported that they completed the BACB’s Compassionate Exception Attestation for Experience/Fieldwork (BACB, 2020c) to waive the “observation with a client” requirement with the majority doing so for only 1 month (43.48%) or 2 months (34.78%), with two participants doing so for 3 months and one each for 4, 5, and 6 months.

By the end of August 2020, only 30.85% of participants in this sample reported that they had resumed in-person supervision. Of the 29 participants who had resumed in-person supervision, about half were providing in-person services (51.72%), followed by a hybrid of in-person and virtual services (34.48%). A smaller percentage of these participants (13.79%) were providing virtual services while receiving in-person supervision. The majority of our participants (62.07%) who had resumed in-person supervision indicated that the format had changed from the supervision they received prior to COVID-19.

Description of Virtual Supervision

Prior to COVID-19, 57.45% of participants in this sample reported that they received exclusively in-person supervision, 20.21% virtual, and 20.21% hybrid, with one reporting other. See Table 3 for amount of individual supervision hours per week and month during the 3 months prior to COVID-19. Approximately half of participants (55.32%) in this sample indicated that the amount of individual supervision they received changed during the virtual model, with the majority reporting that supervision decreased (42.31%) or greatly deceased (23.08%). It is interesting that of the participants who reported a change in amount of individual supervision, 17.31% indicated that individual supervision increased and 17.31% indicated that it greatly increased.

Table 3

Amount of Individual and Group Supervision Prior to COVID-19

	Time per Week	n	%	Time per Month	n	%
Individual	< 30 min	1	1.06	<1 hr	0	0
	30–45 min	12	12.77	1–2 hr	15	15.96
	45–60 min	19	20.21	2–3 hr	11	11.70
	60–90 min	32	34.04	3–4 hr	19	20.21
	90–120 min	17	18.09	4–5 hr	17	18.09
	More than 2 hr	10	10.64	5–6 hr	14	14.89
				> 6 hr	18	19.15
Group	< 30 min	19	27.54	<1 hr	4	5.80
	30–45 min	13	18.84	1–2 hr	21	30.43
	45–60 min	17	24.64	2–3 hr	12	17.39
	60–90 min	12	17.39	3–4 hr	14	20.29
	90–120 min	5	7.25	4–5 hr	9	13.04
	More than 2 hr	2	2.90	5–6 hr	4	5.80
				> 6 hr	4	5.80

A majority of our participants (73.40%) reported receiving group supervision prior to COVID-19, with 74.47% receiving group supervision during COVID-19. See Table 3 for amount of group supervision hours per week and month during the 3 months prior to COVID-19. During COVID-19, participants in this sample reported receiving group supervision for a mode of less than 30 min per week (35.71%) and a mode of 1–2 hr per month (25.71%).

Trainees reported the strategies used by their supervisor during virtual supervision. The most frequently endorsed strategies in our sample included answering questions (91.49%), discussion (85.11%), direct observation (77.66%), reviewing protocols and procedures (74.47%), and in-the-moment feedback (67.02%). See Table 4 for percentages of all strategies reported. The majority of our participants (64.89%) reported that their supervisor’s supervision strategies remained the same during the virtual model, 27.66% reported that they changed, and 6.38% reported that they greatly changed.

Table 4

Strategies Used during Virtual Supervision

Technique	n	%
Answering Questions	86	91.49
Discussion	80	85.11
Direct Observations	73	77.66
Reviewing Protocols and Procedures	70	74.47
In the Moment Feedback	63	67.02
Delayed Feedback	55	58.51
Modeling	43	45.74
Review Recorded Session	31	32.98
Role Playing	25	26.60
Others	2	2.13

Note. Other was reported as reviewing research articles, reviewing written reports, and providing feedback on feedback to others.

The majority of our participants reported using Zoom as a virtual platform for supervision (78.72%), followed by Google Hangouts (27.66%), phone call (23.40%), FaceTime (12.77%), and other options (11.70%), which included Microsoft Teams, WebEx, Skype, and GoToMeeting. Responses were variable with regard to frequency of technological difficulties during virtual supervision: (1) never (13.83%), (2) rarely (43.62%), (3) sometimes (34.04%), and (4) often (8.51%).

Satisfaction with Virtual Supervision

When reporting on overall satisfaction with their supervision experience before COVID-19, the majority of participants in this sample indicated that they were satisfied (43.62%) or greatly satisfied (42.55%). During the virtual model, although most participants still reported that they were satisfied (35.11%) or greatly satisfied (35.11%), these numbers decreased. See Figure 2.

Fig. 2

Percentage of Self-Reported Overall Satisfaction with Supervision before and during COVID-19

Prior to COVID-19, participants in this sample largely reported that they were satisfied (29.79%) or greatly satisfied (54.26%) with individual supervision. During virtual individual supervision, most of our participants still reported that they were either satisfied (34.04%) or greatly satisfied (36.17%; see Fig. 3). In general, these ratings corresponded with high domain scores in satisfaction with individual virtual supervision. The mean satisfaction score across our participants was 42.23 out of 50 (range: 22–50), with a median score of 44. See Table 5 for means for each individual satisfaction item and Fig. 4 for individual supervision satisfaction score distribution.

Fig. 3

Percentage of Self-Reported Satisfaction with Individual Supervision before and during COVID-19

Table 5

Mean Satisfaction and Change for Individual Supervision

Satisfaction Categories	Satisfaction Score Mean (Range)	Change Score Mean (Range)
Supervisor’s availability	4.28 (1–5)	3.11 (1–5)
Supervisor’s responsiveness	4.33 (2–5)	3.20 (1–5)
Supervisor’s preparedness	4.22 (1–5)	3.04 (2–5)
Supervisor’s engagement during client contacts	4.15 (1–5)	3.00 (1–5)
Supervisor’s engagement during one-on-one meetings	4.46 (1–5)	3.07 (1–5)
Effectiveness of supervision at teaching necessary skills	4.09 (1–5)	2.93 (1–5)
Clarity of supervision	4.18 (1–5)	2.97 (1–5)
Supervisor’s perceived stress level	N/A	3.04 (1–5)
Supervisor’s perceived mood	N/A	3.00 (1–5)
Your overall comfort with individual supervision	4.21 (1–5)	3.03 (1–5)
Your contribution during individual supervision	4.22 (1–5)	3.14 (1–5)
Your preparedness to implement procedures	4.10 (2–5)	N/A

Note: Satisfaction score = perceived satisfaction with virtual individual supervision. Change Score = how perceived satisfaction changed in virtual model. Numerical Values: 1 = greatly dissatisfied, 2 = dissatisfied, 3 = neither satisfied nor dissatisfied, 4 = satisfied, 5 = greatly satisfied. Questions related to stress and mood were omitted from the satisfaction survey due to desire to avoid discussion of satisfaction with these items. Preparedness to implement procedures was omitted from the change survey due to a routing error in the survey domain.

Fig. 4

Distribution of Individual and Group Satisfaction Domain Scores. Note. Scale ranges from 0 = greatly dissatisfied to 1 = greatly dissatisfied. Individual satisfaction domain scores could range from 10 (1 on all 10 items) to 50 (5 on all 10 items). Group satisfaction domain scores could range from 7 (1 on all 7 items) to 35 (5 on all 7 items)

Regarding group supervision, domain scores in virtual group supervision indicated overall high satisfaction with group supervision received in this sample. The mean satisfaction score across our participants was 28.94 out of 35 (range: 14–35), with a median score of 28. See Table 6 for means for each group satisfaction item and Fig. 4 for group supervision satisfaction score distribution.

Table 6

Mean Satisfaction and Change for Group Supervision

Satisfaction Categories	Satisfaction ScoreMean (Range)	Change ScoreMean (Range)
Supervisor’s preparedness	5.17 (2–5)	3.16 (2–5)
Supervisor's engagement	5.26 (2–5)	3.13 (2–5)
Effectiveness at teaching necessary skills	5.11 (2–5)	3.09 (1–5)
Clarity of supervision	5.16 (2–5)	3.09 (2–5)
Your overall comfort with group supervision	5.11 (2–5)	3.13 (2–5)
Your contribution during group supervision	5.10 (2–5)	3.06 (1–5)
Your preparedness to implement protocols	5.04 (2–5)	3.07 (1–5)

Note: Satisfaction score = perceived satisfaction with virtual group supervision. Change Score = how perceived satisfaction changed in virtual model. Numerical Values: 1 = greatly dissatisfied, 2 = dissatisfied, 3 = neither satisfied nor dissatisfied, 4 = satisfied, 5 = greatly satisfied.

There was a statistically significant effect of length of supervisory relationship on satisfaction with group supervision [F(5, 65) = 2.65, p = 0.031]. Post-hoc comparisons using Tukey tests indicated that supervisory relationship for 1–3 months (M = 31.13, SD = 4.09) was significantly different than 13–15 months [M = 23.44, SD = 4.84). In particular, these participants who had been supervised by their BCBA longer, reported lower satisfaction scores. In our sample, there was a statistically significant relationship between age and satisfaction with individual supervision [F(1, 92) = 6.69, p = 0.011], with the likelihood of higher satisfaction scores increasing with age. No other variables (i.e., location, type of services, amount of time accruing hours, how long supervisor was a BCBA) had a statistically significant effect on satisfaction with virtual individual or group supervision.

The majority of our participants reported minimal change in satisfaction with virtual individual and group supervision, as indicated by change scores near 33 for individual supervision (i.e., rating of 3 on all 11 items) and 21 for group supervision (i.e., rating of 3 on all 7 items), where 3 represents a score of “no change.” The distributions of change scores for individual and group supervision are presented in Fig. 5.

Fig. 5m m

Distribution of Individual and Group Change Domain Scores. Note. Scale ranges from -1 = greatly decreased to 1 = greatly improved. A score of 0 indicates no change. Individual supervision change scores could range from 11 (1 on all 11 items) to 55 (5 on all 11 items). Group supervision change scores could range from 7 (1 on all 7 items) to 35 (5 on all 7 items).

Preference for Supervision Modality

The majority of our participants reported that the modality of supervision they preferred was in-person (51.06%), followed by a hybrid of in-person and virtual (21.28%), virtual (7.45%), and 7.45% indicated that they had no preference. We conducted multinomial logistic regression to model the relationship between age and preferred method of supervision (in-person, virtual, hybrid, and no preference). Addition of age to a model that contained only the intercept significantly improved the fit between model and data, X (3, N = 82) = 10.50, Nagelkerke R = .136, p = .015. The low Nagelkerke R value indicated that age accounted for a very small amount of the overall variability observed in the model. More closely examining the data demonstrates that all participants who selected virtual supervision fell at or above the median age and 85.71% fell above the mean in our sample.

In our sample, there was a statistically significant effect of early intervention programming on supervision preference, X (3, N = 94) = 8.39, p = 0.04. This result indicated a difference between the supervision preference of those providing early intervention services (47.6%) and those not providing early intervention services (52.4%). However, post-hoc analysis of adjusted residuals using a Bonferroni adjusted p-value of .006 (i.e., .05 divided by 8 tests) indicated no significant relationships between preferences. No other statistically significant differences were detected for types of services provided. Statistically significant differences were not found for the effect of any other variables (i.e., location of services, age range of clients, amount of time accruing supervision hours) on preferred supervision modality.

Perceived Effectiveness of Supervision Modality

Similar to their responses on the most preferred supervision modality, our participants endorsed in-person supervision as most effective (50%), followed by a hybrid of in-person and virtual (23.40%), no preference (12.77%), and virtual (2.13%). How long their supervisor had been a BCBA had a statistically significant effect on perceived effectiveness, X (15, N = 66) = 27.65, p = 0.04. It is interesting that those participants whose BCBA had been a supervisor for less than 1 year were the only group to not perceive in-person supervision to be the most effective supervision modality. These participants were also the only group to select virtual as the method they perceived as most effective.

Discussion

This study provides a snapshot of the experiences of trainees pursuing their BCBA and BCaBA in the United States during the COVID-19 pandemic. We were interested in trainees who received virtual supervision at some point during the period from March 1, 2020 through August 31, 2020. As such, these results are not representative of those who continued to receive exclusively in-person supervision during COVID-19. Although we received a relatively low response to our survey despite rigorous recruitment efforts (e.g., repeated messages, recruitment across multiple platforms, contact with every program with a verified course sequence) and casting a wide recruitment net, we believe that our results present important data for the behavior analytic community and set the stage for future research on this important topic. First, we do not have data on the number of trainees in the United States who received virtual supervision as a result of the COVID-19 pandemic to identify the population we aimed to sample. Second, we received responses from participants across the United States (i.e. 24 states), with those states with larger numbers of BCBAs well represented. Our sample demographics also followed a similar pattern to that of BCBA certificants (BACB, 2020d). Third, we gathered data on a challenging period for many practitioners and trainees with the 6-month period at the beginning of the pandemic. We did not extend our recruitment beyond August 2020 because we aimed to capture a similar period of trainee experiences. The start of a new semester where trainees may obtain new fieldwork placements and a time period in the United States when the delivery modality of many sites may have shifted to return to in-person service delivery may have affected the homogeneity of our sample had we continued data collection efforts.

Results of this study indicated that trainees in our sample reported a decrease in their accrual of direct client hours to some extent during the pandemic, with nearly all our participants reporting a decrease in direct client hours both while providing virtual services and/or in-person services. These results are unsurprising as client hours may have been affected by a myriad of factors. For instance, clients and their caregivers may have chosen to not participate in in-person sessions due to personal circumstances (e.g., high risk status) or social distancing practices; whereas providers may have decreased trainees’ hours to enforce social distancing guidelines (e.g., number of staff present, number of clients present). Telehealth sessions may have been impractical (e.g., individuals who engage in high rates of severe destructive behavior) or infeasible (e.g., lack of access to technology, increase in child-care responsibilities) for some clients and their caregivers. Further, many providers in the field of applied behavior analysis work with young children and/or children with intellectual and developmental disabilities who may not readily participate in virtual sessions. As such, telehealth sessions may be shorter in duration than in-person sessions and necessitate a shift in focus (e.g., training parents to implement protocols rather than working directly with clients). The decrease in client hours has long-term implications for the length of time trainees may need to continue accruing hours prior to taking their BCBA or BCaBA exam. Future research should continue to investigate this potential impact on trainees.

Whereas nearly all of our participants reported providing in-person services prior to the pandemic, almost two thirds provided virtual services at some point during the pandemic, with a quarter providing a hybrid of virtual and in-person services. Therefore, most participants in our sample experienced providing virtual services in some capacity during this period without prior experience with virtual service delivery. Likewise, most of our participants received in-person supervision prior to the pandemic and then made the transition to virtual supervision during the pandemic. It is interesting that 20.21% of our participants indicated that they received virtual supervision prior to COVID-19 and 20.21% received a hybrid of in-person and virtual supervision. These results indicated that supervisors may have already been incorporating virtual supervision into their practice prior to the pandemic. At the conclusion of the study period, more than half of our participants were still receiving virtual supervision, suggesting that virtual supervision practices may continue for the foreseeable future. Future research should evaluate trends in the use of virtual supervision methods in behavior analytic supervision.

One of the primary objectives of this preliminary study was to evaluate the acceptability of virtual supervision, as indicated by trainees’ perceived satisfaction with virtual supervision. We quantified satisfaction by creating a sum score of perceived satisfaction with multiple aspects of supervision (e.g., supervisor’s availability, trainee’s contribution) for both individual and group supervision. We also asked participants to rate their perceived satisfaction on a Likert scale. Results demonstrated that in general participants in this sample were satisfied with all aspects of virtual individual and virtual group supervision as indicated by high satisfaction domain scores and individual item means. These preliminary results are promising as they suggest that the rapid change and restructuring of supervision practice in response to the COVID-19 pandemic was successful at maintaining quality supervision within our sample.

With regard to satisfaction, our results indicated a statistically significant difference between length of supervisory relationship and satisfaction with group supervision. It is surprising that participants in our sample reported lower satisfaction scores when they had been supervised by their BCBA for a longer length of time. This finding could potentially be explained by our participants having a long history of exposure to certain supervision practices, such that a change in supervision strategies or style in a virtual format may have caused perceived dissatisfaction.

Further, we sought to measure how satisfaction with supervision may have changed during the virtual model. Results from our sample indicated minimal overall perceived change in satisfaction across various aspects of supervision, as indicated by mean change scores of nearly 3.0 (i.e., stayed the same) across all items. Two items (i.e., effectiveness of supervision at teaching necessary skills and clarity of supervision) yielded mean scores slightly below 3.0 indicating that perceived satisfaction with these aspects of supervision may have decreased during the virtual model. As the goal of supervision is for trainees to acquire necessary skills to deliver quality services, a decrease in satisfaction in this area is concerning. Clarity of supervision would seem to be an area that may have been more greatly affected by virtual supervision due to technological challenges, inability to directly model skills, and challenges to observing nonvocal cues of understanding (e.g., confused expressions). Future research should consider directly measuring trainees’ competency in target skills (Sellers et al., 2016) following virtual supervision.

Another objective of this study was to evaluate the perceived feasibility of virtual supervision. In this study, the majority of participants were receiving both virtual individual and group supervision. Most participants in this sample who did not meet the minimum client contact hours only needed to complete the waiver form for 1 or 2 months. At the time of data collection, most of our participants had not returned to in-person supervision, indicating that virtual supervision was occurring and was facilitating direct client observations. In addition, our participants reported that their supervisors used a variety of strategies over the virtual platform that aligned with recommendations for behavior analytic supervision (Sellers et al., 2016; Sellers et al., 2019). These results are promising as supervisors needed to quickly adapt their supervision to a novel format in a short timeframe. Although conclusions are drawn from indirect assessment, results suggested that quality supervision was still provided for participants in this sample. Feasibility may be tied to readily available technology (e.g., cell phones, tablets, laptops) that most individuals have access to in some capacity. A small subset of our participants reported experiencing technological difficulties during virtual supervision; however, this variable did not have a statistically significant impact on satisfaction. Approximately one third of participants in this sample indicated that the amount of individual supervision they received decreased during the virtual model. Although this reduction in supervision may indicate a feasibility limitation of virtual supervision, it may also be the result of fewer direct client hours and reduced length of telehealth sessions.

Despite reported satisfaction with virtual supervision, trainees in our sample indicated an overall preference for in-person or hybrid supervision. The majority of our participants also indicated that in-person or hybrid supervision was most effective, with only 2.13% considering virtual supervision to be most effective. With regard to preference, our results indicated that age was a significant predictor of preference, with older participants more likely to select virtual or a hybrid model of supervision as most preferred than younger participants. Age was not a significant predictor of satisfaction with virtual supervision in our sample, suggesting that despite differences in preference, our trainees across the age range still benefited from virtual supervision.

It is interesting that our results indicated a statistically significant difference between amount of time a supervisor was a BCBA and trainees’ perception of the most effective service delivery modality with trainees in this sample more likely to endorse virtual supervision as most effective when their supervisor was a BCBA for less than 1 year. Although these results may be due to other variables not captured in the survey question, new supervisors may have invested more time learning to navigate the virtual platform (e.g., researching virtual supervision strategies) or may be more aware of technological advancements that can be used in virtual supervision.

When analyzing qualitative responses for trainees’ preference for and perceived effectiveness of supervision modality in this sample, themes that emerged for in-person supervision, in order of salience, were ease of implementing specific supervision strategies (e.g., modeling, role playing), ease of providing in-person feedback, better rapport and engagement, lack of technological challenges (e.g., video quality, muting and unmuting), ease of data collection and graphing (e.g., interobserver agreement, procedural fidelity), and greater supervisor preparedness. Themes that emerged for virtual supervision included greater use of resources (e.g., flexibility and availability), time and financial savings, perception of the environment (e.g., fewer distractions and greater comfort), and enhanced capabilities of a virtual platform (e.g., recording, screen sharing). Those preferring a hybrid model reported themes of greater flexibility and balance and better use of resources. Participants in this sample suggested preference for in-person client observations with virtual individual meetings or in-person individual meetings held when specific challenges arise.

Results of this preliminary study suggested that BCBAs can provide virtual supervision that is acceptable and feasible. Gaining quality supervision hours toward a BCBA or BCaBA is no easy feat and virtual supervision may alleviate some of the burden on supervisors to provide effective supervision. The preliminary results of this study have important implications for the future of behavior analytic supervision and future research should build upon these methods to continue to measure trainees’ experiences with virtual supervision. We provide some practical recommendations on effective virtual supervision that align with the BACB Professional and Ethical Compliance Code (BACB, 2014).

Future Implications of Treatment

Virtual supervision may provide benefits to BCBAs and trainees alike. First, virtual supervision affords both supervisors and trainees greater scheduling flexibility and reduces travel time. Decreasing the time for a BCBA to travel to in-person sessions and supervision meetings or between locations (e.g., different client homes, different schools) allows the BCBA to use their work time more efficiently for client-related activities. Some individuals may travel long distances to sessions (e.g., rural areas) or spend considerable time traveling (e.g., congested urban areas). In addition to an increase in time efficiency, virtual supervision has the potential to decrease the stress on BCBAs. In terms of individuals, virtual supervision may reduce the stress of travel itself (e.g., traffic, accidents, finding parking), focus on arriving at a session on time or departing to travel to the next session, or travel in inclement weather. Such stressful variables may negatively affect supervisor engagement during in-person sessions and meetings.

Further, virtual supervision may allow some BCBAs to take on additional clients or trainees, thus reducing long waitlists for behavioral services and providing quality supervision to trainees. Although there is no specific rule about caseload size, we can infer from Section 2.0 (“Behavior Analysts Responsibility to Clients”) and 5.0 (“Behavior Analysts as Supervisors”) of the BACB Professional and Ethical Compliance Code that a supervisor should only take on clients and trainees within the bounds of their competence and time constraints. With regard to clients, BCBAs have a responsibility to make decisions in the best interest of the client, including only accepting clients when they have the “available resources” (Section 2.01) to best serve that client. Without adequate time to devote to each client on a caseload, BCBAs should not accept additional clients. Regarding supervision, the code specifically mentions supervisory volume, stating that BCBAs should “take on only a volume of supervisory activity that is commensurate with their ability to be effective” (Section 5.02). Effective supervision requires not only a time commitment from the supervisor to meet the trainee’s requirements for certification, but to also provide adequate ongoing feedback to improve trainee performance and to evaluate their own supervision activities. Practitioners should exercise caution to not overburden themselves and abide by the BACB ethical code when taking on clients and trainees.

Another consideration is that virtual supervision could allow BCBAs to be more purposeful in the specific times that they observe sessions. For example, a BCBA might provide a window of time where they are “on-call” and prepared to observe a session. The BCBA may log into sessions when requested by the trainee rather than arbitrarily selecting a set observation time to fulfill supervision requirements. For instance, if a behavior reduction client were engaging in severe destructive behavior, a trainee could contact the BCBA to log on to observe and provide in-the-moment feedback. Likewise, if a trainee is implementing a new protocol or has questions about a specific program, the BCBA could observe and provide feedback during these times. When a BCBA is onsite, they may observe a session with no target behavior, observe downtime activities (e.g., snack time, bathroom trip, breaks), or not observe what is most critical (e.g., errors implementing a new program, challenging behavior during mealtime). Virtual supervision could produce more streamlined observation, support, and feedback to optimize treatment outcomes.

Virtual observation of client sessions may allow for discreet observation of the client and trainee. If observing during a telehealth session, supervisors can turn off their video and communicate directly with trainees using the chat feature, allowing for more discreet observation. If supervisors are virtually observing an in-person session, the video device can be discreetly placed such that clients are less aware they are being observed. This covert observation has the potential to decrease reactivity of the client, because they may be unaware they are being observed and to enhance data recording measures such as IOA and procedural fidelity.

During downtime in in-person client observations (e.g., break, reinforcer delivery), therapists and supervisors may chat about topics unrelated to the client. Some clients may choose independent leisure activities that do not require therapist attention, making such conversations more likely to occur. With telehealth service delivery, trainees may be more engaged with the client to facilitate virtual activities on screen and the presence of caregivers may deter side conversations. When virtually supervising in-person sessions, the supervisor can use downtime to provide feedback and even virtually engage with the client over the screen rather than engaging in conversation with the therapist. As a result, rapport building with trainees may be more purposeful during individual and group supervision meetings.

Another potential benefit of virtual supervision is that sessions can be discreetly recorded and reviewed for internal purposes (i.e., trainee and client progress, IOA data). During telehealth sessions, major HIPAA-compliant platforms (e.g., Zoom, WebEx) allow for session recording. Prior to using virtual platforms, BCBAs should first review best-practice recommendations for virtual communication and delivery of services (e.g., Cavalari et al., 2015; Rios et al., 2018). However, a HIPAA business associate agreement (BAA) that details each party’s responsibilities with the recording is necessary prior to internal use to minimize potential security concerns (Britton & Cicoria, 2019). Session recordings can be valuable to the internal system for many reasons. Recordings can enhance feedback capabilities during virtual supervision. During virtual observation of in-person sessions, supervisors could feasibly record session to later review with the trainee. Recordings also allow the supervisor to evaluate trainee and client progress asynchronously at a time that is convenient for them. Recorded sessions also benefit the trainee as they can review the accuracy of their data collection or to have independent naïve data collectors take reliable IOA data. Again, virtual supervision can decrease reactivity of the client by not having another person in the instructional setting taking data. In addition, supervisors can use video recordings to collect treatment fidelity data at another time. Finally, with appropriate consent, such videos can be used to train new staff members on protocols and procedures prior to their implementation.

Future Implications of Training

Given the potential benefits of virtual supervision and the current continuation of virtual supervision during the COVID-19 pandemic, virtual supervision is likely to continue for the foreseeable future and may see greater use in practice postpandemic. As such, behavior analytic training programs should begin to consider preparing trainees for delivering virtual supervision. This training may include didactic training on topics such as adapting supervision techniques to a virtual platform, establishing rapport virtually, and providing feedback virtually. Training programs may incorporate opportunities to observe trainees engaging in these skills prior to obtaining their BCBA or BCaBA. Likewise, training programs may consider increasing training on telehealth service delivery. For example, training may include teaching students how to structure telehealth sessions, interact with clients and caregivers virtually, conduct virtual parent training, and collect data virtually.

Further, to increase trainee satisfaction with virtual supervision, supervisors should attempt to integrate more “face-to-face” virtual communication (i.e., video conferencing instead of a phone call) into their supervisory practices. This could increase the level of engagement and rapport between the supervisor and the trainee while also providing the opportunity for the supervisor to model specific skills and clarify trainee questions as they come up. Implementing more “face-to-face” virtual communication into supervisory practices, instead of primarily using asynchronous modalities, could better integrate the benefits of in-person supervision with the ease of virtual supervision to increase trainee satisfaction. To maximize supervisor and trainee time, supervisors may consider virtual resources that can be accessed asynchronously such as developing a video model library of specific skills for trainees to view at their leisure. Finally, supervisors may consider planning activities to promote engagement and assess critical skills during supervision such as opportunities for trainees to train their peers on specific content during a group supervision session.

Other Considerations

Although there are potential positive outcomes of virtual supervision and trainees in our sample were generally satisfied with virtual supervision, we must proceed with caution before drastically changing the nature of behavior analytic supervision. The majority of trainees in our sample indicated that they preferred in-person supervision and considered in-person supervision to be more effective. Trainees brought up astute points regarding benefits of in-person supervision (e.g., modeling with client, assistance with managing behaviors) that may be difficult to replicate virtually.

Virtual supervision may increase the burden on BCBAs to be “on-call” for all situations, contributing to unpredictable schedules and difficulty setting boundaries of availability. The behavior analytic community is already faced with high levels of burnout (Plantiveau et al., 2018) and compassion fatigue is found to be positively correlated with burnout in the field of applied behavior analysis (Simpson, 2020). Stemming from the psychology literature, compassion fatigue is a type of caregiver fatigue that results from being an empathetic practitioner while disregarding one’s own self-care needs (Figley, 2002). With this effect in mind, BCBA supervisors should practice self-care, set boundaries of availability, and not take on more responsibilities than they can handle.

The focus of this study was on trainees’ experiences and perceptions with virtual supervision. Future research should examine satisfaction and feasibility from the BCBA supervisors’ perspective. Virtual service delivery has some potential benefits to clients and families (e.g., reduced travel time, not bringing siblings to sessions, ability to observe behaviors in natural environment, greater caregiver involvement in sessions) that should be examined in future research.

Limitations

This study has several limitations that warrant mention. In particular, this study collected self-report data on perceived effectiveness and perceived satisfaction with virtual supervision. Although trainees’ perceptions and experiences are important to document, we cannot draw conclusions about the actual effectiveness of supervision formats without direct observation. Further, although the survey asked about overall experiences with virtual supervision during COVID-19, trainees’ responses may have been influenced by specific salient events (e.g., recent negative feedback, recent cancelled appointment). In addition, despite asking respondents to describe their overall experience with virtual supervision during a 6-month period, the data were collected during the summer months when direct hours may have decreased in certain settings (e.g., schools). Although our aggregate results suggested an overall high perceived satisfaction with virtual supervision in this sample, there were still a few participants who rated dissatisfaction across all items. This negative rating could be due to their negative experience with all aspects of virtual supervision or might be skewed by animosity toward the participant’s supervisor resulting in emotional responding across all items. We could have observed positive behavior contrast if the participants’ behavior (e.g., expressing their discontent with supervisory practices) was punished or put on extinction by their supervisors. And we could have seen increases in that behavior by allowing an anonymous platform that served as a type of automatic reinforcement (e.g., expressing discontent on survey items makes one “feel” better), which can, in turn, lead to increased negative responding.

In addition, we reported on the perceived change in stress and mood of supervisors during the virtual model in an attempt to capture how personal circumstances of the pandemic (e.g., additional caregiving responsibilities, anxious behaviors related to health) may have affected supervision. Stress and mood are subjective and complex concepts that may result in different observable and measurable behaviors. Not all individuals engage in the same stress- or mood-related behaviors or perceive these behaviors in the same way. Without objective definitions of observable stress and mood-related behaviors, responses to our survey measure were subjective. For example, a trainee might consider stress to be displayed as vocal statements (e.g., "I'm stressed," or "I have so much to do"), whereas another trainee might consider stress to be displayed as shorter, snappier emails, and another trainee might consider stress to be physical features of the supervisor (e.g., bags under eyes, disheveled appearance). In addition, we only asked about how these variables changed during the virtual model and do not have baseline levels to compare. For instance, one supervisor may have appeared extremely stressed before COVID-19 resulting in the trainee reporting that stress did not change during the virtual model. On the other hand, another supervisor may have not appeared stressed before COVID-19 resulting in a trainee reporting that stress increased during the virtual model. We cannot compare which supervisor engaged in more stress-related behaviors during COVID-19. Finally, supervisors may not visibly display behaviors indicative of stress or decreased mood during virtual supervision; however, they may still be experiencing these variables. Our goal was to determine how trainees perceived these variables to affect supervision; however, future research may attempt to operationally define and observe these behaviors or measure supervisors’ reports of stress and mood.

This study focused on trainees’ experiences with virtual supervision. Future research may consider examining trainees’ personal experiences with the COVID-19 pandemic (e.g., personal illness or illness of family member, balancing of personal responsibilities, fear of attending in-person sessions) to identify resources to best support trainees. The focus of this study was on trainees’ experiences; however, future research should explore supervisors’ experiences with virtual supervision to obtain a more comprehensive picture of the impact of the COVID-19 pandemic on supervision.