Feasibility of Virtual Focus Groups in Program Impact Evaluation.

Focus groups are often used for qualitative investigations. We adapted a published focus group method for evaluating impact of an organizational intervention for virtual delivery using video conferencing. The method entailed convening small groups of three to five participants for a 2-hour facilitated workshop. We delivered the virtual workshops, adding qualitative evaluation with researchers and participants, to assess the effectiveness of the protocol. We address the questions of how to structure the data collection procedures; whether virtual delivery permits cross participant interactions about a studied intervention; and how easy and comfortable the experience was for participants. Participants were university faculty members who were the focus of an institutional diversity program. The results indicated that the virtually delivered focus group workshop could be successfully implemented with strong fidelity to the original protocol to achieve the workshop goals. The workshops generated rich data about the impacts of the institutional program as well as other events and conditions in the working environment that were relevant to consider along with the observed program outcomes. A well-planned virtual focus group protocol is a valuable tool to engage intervention stakeholders for research and evaluation from a distance. Video conferencing is especially useful during the current COVID-19 pandemic, but also whenever geography separates researchers and evaluators from program stakeholders. Careful planning of privacy measures for a secure online environment and procedures for structured facilitation of group dialogue are critical for success, as in any focus group. This article addresses a gap in the literature on feasibility and methodology for using video conference technology to conduct qualitative data collection with groups.

Introduction

The COVID-19 pandemic and the subsequent lock-down and rules for social distancing have imposed severe restrictions on in-person research. In many parts of the world, responses to the pandemic have forced many non-essential workers to convert to digital means of communication such as video conferencing and external access to company information systems. While these systems have enabled many to work from home, researchers who conduct qualitative interviews have found it challenging to conduct proper data collection, especially for focus groups (also known as group interviews), when they are not able to meet in person with respondents. The concerns go beyond simple logistics to methodologic concerns: interviews are an intimate setting where the researcher both listens for the verbal responses and reads nonverbal cues, such as body language, to interpret and rephrase questions. To what degree is the researcher able to respond to verbal and non-verbal skills in a virtual environment? What, if any, are the impacts on data quality? Many researchers are grappling with these questions while attempting to continue their research remotely.

Prior to COVID-19 some researchers had already experimented with web-based media for interviews (Archibald et al., 2019; Hewson, 2008; Horrell et al., 2015). Most descriptions of online video interviews for research are focused on the platform features and user instructions rather than the interactive process (Lobe et al., 2020). While these reports are insightful, they are not detailed methodologic descriptions of how to develop, plan, execute, and evaluate virtual interviews. In fact, virtual focus groups, where a group of simultaneous interviewees are each sitting alone, appear not to have been treated in the literature at all until very recently.

The context for this study is a university program to improve work climate for female faculty in Science, Technology, Engineering and Mathematics (STEM) fields, which was underway before the COVID-19 pandemic began. The Effect Modifier Assessment (EMA) method (Edwards & Winkel, 2018) was chosen as one evaluation method because it is designed to assess contextual factors which might affect the same outcomes addressed by the intervention and thus influence the evaluation results. The core data collection instrument in the EMA method is a focus group. As this study team was about to plan the focus groups, the COVID-19 pandemic of 2020 imposed remote work and social distancing requirements. Consequently, we were forced to adapt the method to a virtual format.

This paper presents a method for conducting virtual focus groups and the respondents’ evaluation of the method. We address the questions of how to structure the virtual data collection procedures; whether virtual delivery can generate the needed qualitative data and permit interactions between participants in the focus group; and the subjective experience for participants.

Online (virtual) Interviews

Researchers have used telephones for many years to conduct interviews that are not face-to-face. Telephone interviews do not provide visual cues, which are contextual and nonverbal data; the loss of these may compromise analysis (Novick, 2008). While the challenges of telephone interviews are often noted, they do offer advantages such as an increased geographical reach, low cost, and flexibility (Drabble et al., 2016; Holt, 2010).

The internet opened more opportunities for researchers to connect with their desired respondents. Faster internet connections, devices with cameras and supporting software have made it possible to conduct virtual interviews that allow the researcher and interviewee to see each other in real time. In theory, online video tools for real-time interaction offer similar possibilities as face-to-face interviews, such as observing verbal and nonverbal cues. At the same time, online tools offer unique possibilities and challenges that must be taken into consideration (Archibald et al., 2019). Software platforms such as Skype, Zoom, and Microsoft Teams offer face-to-face communication, but the functionality and use differ and may influence the interview experience. Some of these platforms offer recording and automated transcription, which are useful to researchers. Respondents with multiple types of devices (cell phone, tablet, computer) can participate on their preferred device where they wish. Interviews may also be broken into segments rather than conducted as a single long interview (Weller, 2015).

Online video interviews present potential disadvantages such as loss of signal, leading to dropped calls and segments where audio is missing (Seitz, 2016). Although video platforms allow the researcher to see the respondent, they might not provide sufficient ability to read nonverbal cues and body language (Seitz, 2016). Respondents often sit close to their devices allowing the researcher to see only the face.

Evaluation of quality of the collected data differs slightly among researchers, with Weller (2015) noting that the quality of the researcher-participant relationship may affect data quality. This is in part supported by Rowe et al. (2014) who concluded that interviewer skill and sensitivity can compensate for lack of observation of the respondent’s entire body in a study of schizophrenia. A study comparing Skype video calling with in-person qualitative interviews (Krouwel et al., 2019) found that in-person interviews were only marginally better, in terms of the amount of information shared by respondents. The range of topics discussed, however, were the same for the two modes of interview, leading the researchers to conclude that video call interviews can be justified. Jenner and Myers (2019) found Skype interviews yielded data that were equal to or better than in person interviews with no loss in rapport or intimacy.

Focus group interviews, like semi-structured interviews, are a method of data collection that uses interaction among participants as a data source, while the researcher facilitates group discussion (McLafferty, 2004). In individual interviews, it is the researcher who must receive and decode non-verbal cues, whereas in focus groups, the other respondents must do so as well. Focus group interviews have been conducted in online settings although only a few papers have been identified. Gaiser (1997) used focus groups to interview participants on internet mailing lists (Listserv) where the researcher poses questions in an email to all participants. Participants can then reply to all and answer the question as well as comment on each other’s answers. The author found it possible to conduct focus group interviews, but the offline nature of mailing lists required that participants self-manage, as a moderator was not present 24 hours a day. A recent study compared in-person and on-line written focus groups conducted on the discussion platform Reddit as a controlled brain storming experiment (Richard et al., 2021). In-person focus groups produce more words, but the number of unique ideas was equal which is supported by Namey et al. (2019). Tuttas (2015) studied traveling nurses using web conference technology for focus groups and found the technology useful. Participants were able to interact and establish an authentic sense of agreement (Tuttas, 2015) when a respondent acknowledged a description with nonverbal utterances.

Matthews et al. (2018) studied radiation therapy professionals in Australia and evaluated participant responses to the online experience. While attrition was high as also reported by Tuttas (2015) the online focus groups provided an opportunity for fluid discussion. Discussion moderators were able to facilitate conversation as in traditional focus groups and participants felt they were able to contribute ideas allowing rich data to be collected (Matthews et al., 2018). No additional tools or platforms were used to facilitate interaction. Gray et al. (2020) evaluated Zoom as a platform for semi-structured interviews and did a pilot survey of evaluating a parenting program which included questions about the Zoom interview experience. Respondents were comfortable sharing personal experiences and the online nature removed logistical factors like distance and cost while adding flexibility. Lobe et al. (2020) reviewed eight videoconferencing services but also offer advice on how to interview online. Online focus groups should be kept at a low number of participants ideally three to five and one-on-one precessions are advised to ensure smooth operation.

Daniels et al. (2019) reflect on their first experience conducting eight online focus groups and have specific recommendations to avoid upsetting group dynamics such as ensuring respondents participate from start till end. Participants should preferably have experience with the technology and test-calls are suggested. The environment from where the participants connect may also pose challenges as outsiders may enter the room and disrupt, which may in turn cause privacy concerns.

The Effect Modifier Assessment Method for Impact Evaluation

A recent study of facilitating and inhibiting factors in workplace change processes (Winkel et al., 2015) pointed out that contextual factors played a significant role in the success of job improvement efforts such as “Lean” implementation. In an effort to measure and assess the impact of these contextual factors during a defined intervention period, Edwards and Winkel (2018) developed the Effect Modifier Assessment (EMA) method. Its original purpose was to identify and assess contextual factors in ergonomic job improvement efforts. However, the method is equally suited to other types of intervention or organizational change projects (Edwards et al., 2020).

The EMA group interview is designed to identify and evaluate events during the intervention period, both those occurring because of and those independent of the intervention. The original setting for the EMA method is a physical room where respondents and researcher can work undisturbed, and a record of events can be displayed on a wall or similar surface. Respondents answer three questions with a common structure about events and changes that have taken place during the investigated period. The questions serve as a funnel, starting with all events within the institution or unit and ending with those specifically affecting the focus of the study. For example, in a study of work environment changes, the final question would be: “Think back in time. What important changes/events related to work environment have occurred during the intervention time period?”

The researcher facilitates the workshop and respondents answer a single question at a time. For each question in turn, the facilitator asks the respondent to write the events and changes that they remember on individual self-stick note cards (event-notes), without discussion. This way respondents are not influenced by what others are saying and or limiting their contributions to match those of others. The researcher collects the event-notes and interviews the respondents about each event, while placing the note on a timeline (Figure 1). Other participants are invited to comment on the event and share their experience. This provides an overview of the events during the investigated period.

Figure 1.

Chronological timeline of participant-reported events on a whiteboard.

After having answered all three questions and discussed all events, each event is scored by participants for (1) whether or not it was part of the intervention being evaluated; and (2) its impact on the goals or outcomes that the intervention sought to achieve. This allows the subsequent analysis to identify “effect modifiers” or confounders, i.e. events that had either positive or negative impact on the studied outcome. These features were determined important enough that the protocol for virtual workshops needed to support and replicate them successfully in the current study.

Method

Research Setting

The study took place in a large public university in eastern Massachusetts, USA, which was carrying out an institutional change initiative to promote diversity and gender equity among faculty in STEM disciplines. The program, Making WAVES (Women Academics Valued and Engaged in STEM), has three components, of which the most visible one on campus, overall, is a campaign to reduce micro-aggressions through targeted training and general awareness (Haynes-Baratz et al., 2021; University of Massachusetts Lowell, n.d.). The two other components seek to improve networking and mentoring opportunities, and to achieve equity in service workload and departmental accountability overall. Prior to data collection, this study was reviewed and determined to be program evaluation by the Institutional Review Board of the University of Massachusetts Lowell (#16-131-PUN).

Recruitment and Sampling

Full-time tenured, tenure-track, and teaching faculty were recruited for the study from the college of engineering and the college of natural sciences. These two colleges together comprise 233 STEM faculty members (University of Massachusetts Lowell, 2017).

Recruitment took place in March and May 2020. An e-mail was sent by the deans of each college to every eligible faculty member, inviting their participation in a 2-hour focus group to discuss the working environment at the university. The email provided a link to an electronic enrollment survey for volunteers to indicate consent and to supply their email address, preferred focus group dates, gender, academic department, job title, and length of employment at the university. Volunteers were arranged into three groups to include diverse representation of job tenure, academic discipline, and rank in each workshop.

Adaptation of the EMA Protocol for Online Delivery

The research team adapted the EMA workshop protocol for virtual delivery using a three-phase approach: phase 1—planning, phase 2—pilot testing, and phase 3—implementation (Figure 2) as described below. During each phase, procedures for materials, technology, and the group facilitation protocol were carefully adapted, tested, and refined to accomplish the two key activities of the workshop: focus group for event identification, and scoring to determine the event categorization as intervention vs modifier and level of impact.

Figure 2.

Video conference adaptation procedure for EMA focus groups.

Phase 1—Planning the participant interface.

Zoom video conference software (Zoom Video Communications, Inc., 2020) was selected to interface with focus group participants. This platform was familiar to faculty members because of its use during the spring of 2020, when the university shifted to remote teaching due to the COVID-19 pandemic. A cloud-based spreadsheet software (Google Sheets) replaced the physical timeline board to capture the participants’ responses during the workshop. The spreadsheet was formatted in 6-month increments, allowing easy plotting of events for all to see with the screen-sharing feature. Separate, identical timeline spreadsheets were prepared to process responses to each of three questions during the focus group of the workshop. Three separate timelines were used to avoid visual clutter when processing responses to each question.

To mimic the published EMA procedure of getting written event-notes from participants, researchers planned to collect participants’ responses one by one, using the chat window of the video conference platform. Participants were asked to submit event responses as brief two-to five-word phrases through the chat window, verbally providing the approximate date of the event. Each person was then interviewed about their response and other participants were invited to comment. Each phrase was placed by the researcher into a spreadsheet cell at the appropriate place along the timeline (Figure 3). For each event, participants were asked if anyone else had the same event-note; if so, they were enumerated in the same cell. During this time, the timeline was visible to all participants on the screen.

Figure 3.

Chronological timeline of participant-reported events on cloud-based spreadsheet, with two examples of “event-notes.”

During the second part of the workshop, the researchers asked the respondents to score each event listed on the timeline board. Participants scored the direction and strength of impact to STEM faculty generally, and to female STEM faculty specifically, as specified in Edwards and Winkel (2018). They also noted whether each event was or was not part of the intervention.

Two facilitators interacted live with participants during the EMA workshop session. Roles were assigned for a lead facilitator to pose questions and process responses, and a co-facilitator to type participants’ responses in real time into the timeline spreadsheet. Facilitators practiced typing responses and scores into the sheet, adjusting the font and sheet size for ease of readability. A monitor role was assigned to a third, off-camera researcher to monitor the chat box, keep track of time, and record observations about the facilitation process and the non-verbal cues of participants.

Phase 2—Pilot testing protocol with research team.

A pilot test of the virtual workshop protocol was conducted with the facilitators, moderator, and three volunteers as participants. The protocol involved reviewing informed consent, focus group ground rules, and procedures for on-screen privacy protections such applying a pseudonym to the on-screen display.

Phase 3—Implementing the EMA protocol with participants.

The refined EMA workshop protocol was implemented virtually with a single group of faculty participants in April 2020. Following the session, researchers held a debriefing discussion to identify aspects of the protocol that went according to plan and which aspects could be improved. Notes were taken and saved for later analysis. Based on debriefing notes, minor revisions were made to a “Facilitator Guide” which provided a focus group script (see Supplemental Material) and the presentation slides were further refined to improve clarity and to streamline the workshop facilitation. Two more workshops were held in June 2020. Researchers again held debriefing discussions and recorded notes each time about what worked well, what could be improved, and any other relevant observations.

Process Evaluation With Participants

Researchers invited feedback from participants following each session to assess their impressions of the virtual workshop experience. (This study element is not part of the EMA protocol but was added to serve the goal of evaluating the virtual protocol.) These process data were collected verbally and by email at the end of workshop one, and by email only following workshops two and three. The feedback questions consisted of five open-ended questions (Table 1). The participant feedback data were compiled into a single Word file, which was subsequently imported to NVivo for coding and analysis.

Table 1.

Virtual Workshop Process Evaluation Questions.

Evaluation Questions Asked of Participants Following Each Workshop
• How comfortable were you participating in a focus group online with respect to privacy and security?
• How easy or difficult did you find the discussion with respect to interacting with the facilitator and other participants?
• Which specific aspects about the facilitation, if any, made it easy or difficult to respond to the questions and share your ideas?
• If you had an opportunity to choose now between online and in-person focus group, which would you choose and why?
• Do you have suggestions about how to improve the participant experience?

Data Analysis

Research Team Reflections

Researchers’ debriefing discussion notes from the Phase 3 workshops implemented with participants were analyzed according to thematic domains such as procedures or facilitation techniques that worked well or needed improvement, suggested improvements, sources of struggle, and observations about the participants’ responsiveness at different stages of the workshop. Key themes were discussed among the four researchers present during the workshops to elaborate on lessons learned and possible areas for future improvement.

Participant Feedback

Narrative data from participant feedback were coded in NVivo software (NVivo, 2019) using an open coding approach, then categorized into themes (Savin-Baden & Major, 2013). The NVivo references for each theme were reviewed and discussed together by three members of the research team, and discrepant perspectives were reconciled to develop consensus for the final themes.

Results

Participants

The research team conducted one virtual practice session, and three virtual EMA workshops each with four to five participants. Workshops were conducted between April and June 2020. A total of 14 participants were grouped about evenly among the three workshops. Of the 14 participants who completed the EMA workshop sessions, one-half were female. They had worked an average of 8 years at the university and a minority (29%) were tenured (Table 2). Participants generated more than 100 event-notes over the course of the three EMA focus groups, of which about one-fourth were identified as related to the intervention. Overall, respondents seemed to feel comfortable reporting both positive and negative impacts of the changes and events they discussed.

Table 2.

Participant Demographics in Three EMA Workshops.

EMA Workshop	Years Employed at University Mean Years (Range)	Tenured N (%)	Female N (%)	Total N (%)
1	2 (1–5)	0 (0%)	2 (50%)	4 (29%)
2	12 (2–25)	3 (60%)	1 (20%)	5 (36%)
3	9 (2–19)	l (20%)	4 (80%)	5 (36%)
Total (Mean)	8 (1–25)	4 (29%)	7 (50%)	14

Research Team Reflections on Virtual Protocol Adaptations

Following the Phase 2 pilot test of the workshop protocol, the research team identified the need to present focus group instructions visually for better participant comprehension. For example, presentation slides were created to review initial instructions on screen and to present participants with the question prompts for the focus group interview and the instructions for the subsequent scoring procedure. The pilot test made it apparent that participants needed written information in advance regarding the video conference logistics, materials needed (paper and pencil), and privacy protection procedures. The “Facilitator Guide” script was revised to include a technical checklist when opening the video conference session, cues to display the presentation slides, refinements to question solicitation, and the mechanics of logging responses in the cloud-based spreadsheet. Table 3 outlines the virtual adaptations made to the original EMA workshop protocol (Edwards & Winkel, 2018) and the rationale based on the experiences of the pilot test and research workshops, which included adaptations to workshop location, equipment, supplies, and communication adaptations; facilitation techniques; and data capture.

Table 3.

Virtual Adaptations of the EMA Focus Group Protocol.

EMA In-Person	EMA Virtual	Reason for Change
Workshop logistical adaptations
Location
Physical Meeting Space	Virtual meeting space using Zoom software	The global pandemic forced social distancing, requiring a move to virtual platform.
Materials
Timeline drawn on a whiteboard, question prompts on a flip chart. Supplies: markers, large Post-it note cards	Timeline created in a cloud-based spreadsheet, question prompts on slides. Equipment: computer, speaker, microphone	An on-screen spreadsheet allowed researchers to plot responses along a timeline in an organized fashion in real time while sharing screen.
Communication
Discussed informed consent in the session.	Sent advance email about informed consent, supplies.	The pilot test indicated the need to send advance information about logistics, supplies needed (paper and pencil), and privacy protection procedures.
Facilitation adaptations
Number of Facilitators
One facilitator processed pre-written notecard responses; one observer records notes on group interactions.	Two co-facilitators process responses (see below); one observer monitored chat and time, took notes on group interactions.	Allowed one co-facilitator to focus on participant interactions while the other focused on accurately typing responses in the cloud-based spreadsheet.
Instruction delivery
Question prompts and scoring instructions given verbally or on a flip chart.	Question prompts and scoring instructions given verbally and presented using a slide show.	Slides made it easier for participants to recall and comprehend the instructions. Participants in the pilot phase had difficulty retaining instructions when delivered verbally only.
Engagement method
Facilitator invited responses in no particular order.	Facilitator invited responses in the order participants appeared on screen.	Streamlined the processing of responses to progress efficiently and minimize participants talking over one another.
Data capture adaptations
Focus group responses
Collected short written responses on large Post-it notecards, numbering each one, then placing on white board timeline.	Collected two- to five-word responses through the chat feature, then typed them, numbered, in the cloud-based spreadsheet.	Mimicked EMA protocol of processing events one at a time, but responses had to be very short to fit within a spreadsheet format.
Note cards with repeated responses were numbered then placed atop initial response card.	Repeated responses were assigned numbers, which were typed into the spreadsheet cell with the initial response.	Allowed visualization and quantification of all events on the timeline in an orderly manner.
Scoring responses
Scores were added to the large Post-it notes.	Scores were typed next to responses in cells of the cloud-based spreadsheet.	Allowed visualization of the scores for participant discussion.

Research Team Reflections on Virtual Facilitation Process

The variations in group size, although small, permitted the research team to compare ease of facilitation, which seemed to favor a maximum of four participants for better pacing of facilitation and better visibility to all participants (and between participants) on the computer monitor while screen sharing. Having a fifth participant did not affect the results, however, as the facilitator followed a sequence of engaging participants for their responses in the order of appearance on the screen. In addition, the 2-hour workshop schedule worked well for one group, which generated 21 event-notes, but did not seem sufficient for the two other groups, which generated far more event-notes (57 and 37) and dialogue about the events. Those two groups experienced time pressure for the participants and the facilitators, creating an environment where discussion between participants was sometimes sacrificed to keep the workshop on schedule. One workshop exceeded the time limit by 20 minutes.

Most of the planned protocols worked well for eliciting responses during the focus group and scoring segments of the workshop. One exception was the use of the chat window to collect responses to the question prompts. Despite attempting to guide the participants one by one to submit their responses using the chat box, participants often spontaneously and naturally verbalized their responses for the group. In other instances, participants were confused about when to submit their responses, and the facilitators had to process multiple chats arriving at one time. In each successive workshop, the level of instruction around the use of the chat box began to feel unnatural and even potentially stifling to the dialogue.

One-half of the participants completed the post-workshop feedback survey. Most expressed positive feedback regarding the overall experience with the virtual EMA workshops. A majority of the negative or neutral commentary involved sharing ease (i.e., facilitation and structured feedback), as discussed below.

All respondents but one expressed either equal or higher preference for a virtual session compared with in-person. Respondents who preferred the virtual session cited more privacy and convenience, health and safety (given the context of the ongoing pandemic), and the ability to talk without interruption through this medium (e.g., “easier for introverts to participate”). One participant cited an advantage for usability, stating that it was “easier to see the screen […] when you’re on Zoom because with a share screen function, it’s just on my monitor.”

Privacy was one aspect of the virtual environment experience for which participants provided specific feedback and recommendations. One participant remarked, “I was able to see everyone’s real name before they changed it and they were able to see mine.” This participant recommended the use of a “waiting room” feature in order to have time and assistance as needed to add a pseudonym to their onscreen display upon entering the virtual workshop room. Another participant expressed some discomfort about being assigned to a workshop with a fellow department member, as their perceptions and experiences could be disputed within the session. Two participants left their cameras off, participating only with audio. However, most responses endorsed feeling comfortable overall with the virtual focus group.

Participants generally reported interactions with the facilitator and other participants as “simple,” “easy,” and “respectful”. However, some participants also noted slight difficulty responding to prompts because of the challenges of interpersonal communication in the virtual environment. For instance, one participant said, “It’s much more difficult to ‘read’ if someone else wants to answer, especially if not all […] video images are being displayed […], it’s much easier for folks to talk over one another and takes longer to sort out who’s going to talk first.” Similarly, another participant said they preferred when the facilitator called upon participants in order to know when to contribute.

Participants commented on their ability to interact with the content of the focus group discussion. When it came to interacting with the timeline board (Google Sheet) on the screen, participants suggested the ‘raise hand’ software feature could speed communication when multiple participants wanted to endorse same event following a question prompt. Although some appreciated the ability to view the full, shared timeline spreadsheet on a monitor, others said they would have preferred alternative visual displays for the timeline, such as a shared whiteboard or a document camera.

A minor theme mentioned by participants was time management. One faculty mentioned unevenness of talk time and suggested that researchers should consider timed responses or limiting the number of workshop participants to four people to allow adequate discussion time.

Discussion

In this study, a virtual focus group method was successfully delivered using video conferencing to achieve the data collection goals. The on-line protocol was feasible to carry out, evaluated positively by the participants, and judged by the investigators to be successful in generating a rich set of data regarding the investigated intervention. The virtual focus groups generated eight events per person on average, similar to the typical quantity produced when the EMA is delivered in-person (K. Edwards, personal communication, December 3, 2020). The video conference screen-sharing feature enabled participants to easily see each other along with the information presented to them in real time (e.g., the question prompts and the cloud-based visual timeline). The structured facilitation protocol enabled orderly soliciting and processing of participant responses with strong fidelity to the original method.

Careful planning and practice of facilitation roles was critical to successful delivery of the workshop protocol in a virtual environment. Using two co-facilitators was effective for allowing one researcher to pose questions and manage participant interactions while another scribed the responses. Ensuring that both facilitators had access to the cloud-based presentation and spreadsheet files created seamless transitions as they switched roles midway through the workshop. Assigning a third researcher as an off-camera observer was useful for collecting research notes about the facilitation process and participant facial expressions and body language. The observer also supported facilitators with controlling audio recording, time keeping, and communicating privately with co-facilitators.

Even with a well-planned facilitation protocol, facilitators and participants noted minor difficulties with managing talk time and the flow of discussion among participants. In a virtual environment, focus group facilitators lack the ability to apply traditional non-verbal techniques to redirect conversation, such as eye contact, physical gestures, or physically pointing to an agreed upon ground rule. Facilitators were hesitant to interrupt a participant out of concern for being perceived as rude, which could make participants less willing to interact and provide open, honest responses. Some participants suggested that more structure might facilitate easier processing of question responses. Facilitators in this study had success with calling on participants individually to make it clearer for participants when it was their turn to respond to the question prompt. However, it is important to balance these dialogue supports with periods where participants can interact with each other without interference. As in any focus group, but especially in the virtual environment, the situational skill and sensitivity of the researcher to assess social cues is important for sensing how to get the best data possible.

Participants in this study generally reported the virtual workshop experience to be easy and comfortable, although some noted the challenges of perceiving non-verbal cues to know when it was their turn to speak. All but one person said they preferred the virtual session equally or better to an in-person focus group. These results were similar to those reported by Krouwel et al. (2019) and Archibald et al. (2019).

The research team identified some areas for future improvement, including privacy protections and adjusting the workshop length by 30 minutes to relax the pace of discussion. The potential concern, of course, is whether recruitment would be more difficult for a longer workshop. Privacy protections can be strengthened in a virtual focus group by ensuring that participants enter the video conference with pseudonyms already on their screen display. This may be achieved with brief, individual, pre-session meetings the day before with participants to assist them with setting their desired on-screen names (Lobe et al., 2020).

Indeed, privacy appears to be perceived differently by the research team and respondents. The original concern was that the online medium introduced privacy issues as video and voice data were recorded and potentially could be leaked and abused. This sentiment may have been a concern of two respondents, who participated with their video cameras turned off. Another respondent however, discussed privacy questions in relation to anonymity (i.e., being able to see each other’s real names). Hence, some respondents may perceive online focus groups to be anonymous despite being from same university and seeing their faces. The perception of privacy in online focus groups warrants further investigation.

A strength of the current study was the experience of conducting multiple workshops, which allowed the research team to make iterative improvements with each session. Another strength was implementing the EMA protocol in the context of an on-going program with a highly engaged institutional leadership team. Applying the EMA in this context allowed a strong participant recruitment effort and triangulation of results with the program stakeholders during data analysis, consistent with the EMA published protocol.

One limitation of this study is that there was no concurrent control group. While in principle it would have been desirable, it was not possible because the pandemic shutdown necessitated the conversion to virtual delivery. A further limitation for generalizing the results is that the participants represent a specific occupational sector (university STEM faculty) who were already experienced with using video conferencing for teaching. Prior training and practice would likely be needed for participants less experienced with video conferencing technology, although the amount of training remains to be determined. It was noteworthy that a few minutes were needed before formally starting the workshop, to assist participants to set their on-screen pseudonyms, despite their experience with the same platform. Attempting the virtual EMA in other intervention settings would be a useful focus of a future study.

Forced by COVID-19 pandemic social distance requirements, this study contributes to a growing body of literature regarding the use of online platforms for qualitative data collection. Online platforms in greater reach, reduced travel cost and easier planning and when people cannot meet it is a necessity. This study confirms that focus groups can be conducted online and that respondents engage in open and fluid discussions as also noted by Matthews et al. (2018) and Gray et al. (2020). This study further demonstrates that it is possible to facilitate online focus groups using artifacts such as virtual tables to mimic use of post-its and whiteboards in in-person focus groups. Online platforms allow respondents to choose the venue from where they connect and consequently researchers lose control of the physical space. This, however, does not appear to influence the quality of focus group interviews.

Conclusions

In this study, video conferencing was successfully used to deliver virtual focus groups as part of a program impact evaluation activity. The ability to conduct research using video conference software was especially useful during the pandemic period but will also be a valuable tool for any instance when geography separates evaluators from program stakeholders. Video conferencing software is a convenient, acceptable platform for research. As with any focus group, but especially with the limitations imposed by a virtual environment, privacy measures and a structured environment to facilitate dialogue are critical.

The virtual focus groups were able to generate information that was judged by the participants, researchers, and intervention stakeholders to be relevant for program impact evaluation. The virtual workshops were able to identify both intervention and modifier events, allowing for a detailed analysis and discussion of the efficacy of the studied intervention. Participant feedback confirmed the investigators’ assessment that the virtual protocol was successfully designed and carried out to achieve the original goal.