Planning, implementing, and evaluating an online group-model-building workshop during the COVID-19 pandemic: celebrating successes and learning from shortcomings.

Introduction

Public health researchers have increasingly turned to system dynamics (SD) modeling to explicate the latent dynamically complex causal mechanisms that shape and perpetuate health disparities (Apostolopoulos et al., 2020; Diez‐Roux, 2011; Hammond et al., 2017). As a participatory approach to the development of SD models, system dynamics group model building (SD GMB) has the potential to increase stakeholders’ understanding of the systems nature of public health problems and can maximize stakeholder buy‐in (Ballard et al., 2020; Mui et al., 2019). SD GMB workshops are commonly facilitated in a face‐to‐face format, where facilitators may advantageously engage stakeholders in opening icebreakers or culturally specific activities which often help to build rapport among facilitators and participants (Gerritsen et al., 2020) and can employ tools such as flipcharts, whiteboards, and printed materials to engage stakeholders in convergent and divergent tasks (Hovmand et al., 2015).

Along with the innumerable disruptions ushered in by COVID‐19 pandemic, face‐to‐face SD GMB workshops were generally rendered impractical (Wilkerson et al., 2020). For example, stakeholders became harder to reach because of public health mandates and general fear of COVID‐19 transmission (Süsser et al., 2021). Accordingly, researchers shifted to online facilitation of SD GMB workshops (Wilkerson et al., 2020). Although potential benefits are conferred by online workshop formats, such as reduced facilitation costs and greater accessibility for participants, unique and novel challenges also emerged, such as stakeholders’ interrupted engagement while working from home and unreliable Internet connections (Rahman and Arif, 2021; Rushina, 2020; Wilkerson et al., 2020). Further, online facilitation differs considerably from face‐to‐face facilitation in terms of facilitation techniques, engagement approaches, length of time needed for discussion, and type of tools used (Thorpe, 2016; Thomas and Thorpe, 2019; Wilkerson et al., 2020).

Extant literature that provides guidance on the design and implementation of online SD GMB workshops is limited, although two recent articles have addressed the emerging importance of this issue (Wilkerson et al., 2020; Yearworth and White, 2017; Zimmermann et al., 2021). Along with the valuable insights shared by these authors, calls for further research aimed at addressing critical issues in conducting effective online SD GMB workshops were set forth, such as determining optimal group sizes for this methodology (Wilkerson et al., 2020), elucidating optimal workshop designs and facilitation strategies (Wilkerson et al., 2020; Zimmermann et al., 2021), and evaluating the effectiveness of various workshop designs and formats (Wilkerson et al., 2020).

This article is methodological in nature and aims to contribute to enhancing the effectiveness of SD GMB through novel applications of tools and procedures. The purpose of this note is to describe insights gained from planning, implementing, and evaluating a recent online SD GMB workshop during the COVID‐19 pandemic. Specifically, this note will (i) describe the use and effectiveness of software tools for divergent and convergent tasks that has not been previously reported, (ii) describe theory‐driven facilitation and engagement strategies that can be applied and evaluated for effectiveness in future work, and (iii) advance the literature with its focus on systematically evaluating online SD GMB workshop design and effectiveness.

Background

Research team

The research team included a researcher with expertise in maternal health disparities and participatory research (KKB), a researcher with expertise in complex systems science and participatory modeling (MKL), a researcher with expertise in system dynamics modeling (SFF), and two graduate research assistants (AH, MO).

Public health issue and setting

Non‐Hispanic Black women (hereafter referred to as Black women) are more than two times as likely to die within 1 year of childbirth compared to non‐Hispanic White women. While preventable maternal mortality itself is caused by a combination of clinical, social, and health‐care factors, the racial disparity is fundamentally driven by structural gendered racism—the totality of ways in which society fosters racial and gender discrimination through mutually reinforcing social, economic, and health‐care systems and policies (Bailey et al., 2017; Davis, 2019; Crear‐Perry et al., 2021; Saluja and Bryant, 2021; Taylor, 2020). The project setting was the Dallas/Fort Worth (DFW) metroplex which is the most populous metropolitan area in both Texas and the southern United States. In Texas, the DFW metroplex has the third highest maternal mortality rate for all women and the second highest maternal mortality rate for Black women (Texas Department of State Health Services, 2018). The aim of this workshop was to understand the dynamic complexity underlying the disparate maternal mortality rates among Black women in DFW.

Participant recruitment

Eligible stakeholders included (i) mothers who experienced life‐threatening maternal health complications in the last 2 years and engaged in local maternal health advocacy in the community and (ii) individuals with expertise in maternal health research, clinical practice, or policy. Personalized emails and social media messages were used to identify 15 stakeholders with diverse and informed perspectives on the issue of Black maternal mortality. Although 12 stakeholders accepted the invitation to participate in the SD GMB workshop, nine stakeholders ultimately participated in the workshop: two maternal health researchers, a maternal health policy director, a midwife, an obstetrician/gynecologist, a full‐spectrum doulai, two women's health program directors, and two mothers, one of which was the founder of a peripartum cardiomyopathy prevention organization. All participants completed an online registration form that included the option to select a preferred workshop format (e.g. a one‐day workshop or a two‐day workshop). Most participants selected preference for a one‐day workshop. All participants reported they had no prior experience with SD GMB workshops. Participants received $200 each as compensation.

COVID‐19 impact

The research team originally expected to conduct a face‐to‐face SD GMB workshop during 2020. The SD GMB was postponed until spring 2021, in the hope that the COVID‐19 pandemic would wane in severity. However, by January 2021 it became clear that the COVID‐19 pandemic would persist for at least another year. Accordingly, the research team conducted the SD GMB workshop in an online format in April 2021.

Description

Workshop planning

Workshop planning consisted of four major components: (i) determining the workshop format, agenda, and roles; (ii) developing a plan for managing online group dynamics and engagement; (iii) choosing an online meeting platform and software tools; and (iv) adapting workshop scripts for an online environment.

Determining the workshop format, agenda, and roles

The research team planned for a one‐day online SD GMB workshop based on participant preferences. The primary roles for the workshop included one cofacilitator, one cofacilitator/modeler, one notetaker who was responsible for taking screenshots and notes, and one assistant who was responsible for checking in and providing general support to participants. Strategies were used to reduce potential virtual fatigue, including building multiple breaks into the agenda, providing links to brief physical exercise videos in the agenda, and playing upbeat music during breaks.

Developing a plan for managing online group dynamics and engagement

The research team applied the Theory of Interpersonal Relations (TIR) in anticipation of potential challenges related to managing group dynamics and engagement in the online format. TIR suggests that individuals have three basic needs when they convene in meeting settings: inclusion, control, and openness (Schutz, 1958). First, to promote inclusion, the facilitators shared a list of participant names prior to the workshop, opened the workshop with introductions and icebreakers, and asked participants to always identify themselves when speaking. Also, participants were encouraged to use the chat function to ask questions if they were not able to ask verbally, and one assistant sent private chat messages to check in with participants who seemed to disengage from the discussions. Second, to establish sense of control, facilitators discussed “housekeeping” items at the start of the workshop—using the “raise hand” function before speaking, staying on mute while others are speaking, and sending private messages to the research team for questions or concerns—to establish group norms. Finally, to foster an open environment for sharing ideas and opinions, the facilitators introduced group norms that encouraged all participants to listen deeply, be mindful of their assumptions, speak honestly, and maintain confidentiality.

Choosing an online meeting platform and software tools

The research team selected Zoom as the meeting platform for the SD GMB workshop, based on the need to meet accessibility standards for attendees with disabilities, video and chat recording, transcription, and interactive engagement functions (e.g. “raise hand”). The research team needed tools that could support divergent and convergent activities, be used for multiple activities (to minimize the number of transitions), and be easy for participants to access and use. Google Jamboard met all these criteria. It featured one main whiteboard on which all participants could access from a single hyperlink. The main whiteboard included multiple pages that participants could use as individual whiteboards for divergent activities. The facilitators were able to access the main whiteboard and individual whiteboards to view participants’ work and provide support as needed. Google Jamboard also featured a drawing tool that allowed participants to draw trend lines on behavior‐over‐time graphs.

Adapting workshop scripts for an online environment

The workshop activities were based on scripts from Scriptapedia (Hovmand et al., 2015). The research team adapted two scripts—the Graph Over Time script and the Initiating and Elaborating a Causal Loop Diagram script (Hovmand et al., 2015)—by replacing in‐person materials with online tools and extending the amount of time allocated for the activity. For example, according to the original Graph Over Time script, research teams should prepare sheets of paper with x‐y axes (graph paper), markers, and tape so that participants work can be displayed on walls, and 45 minutes should be budgeted for this activity during the session. However, using Google Jamboard, the research team prepared whiteboards for each participant which included an x‐y graph template and fillable textboxes to use for assigning variable names, and the duration of the activity was extended to 80 minutes to allow time for the research team to show participants how to use Jamboard, for participants to familiarize themselves with the tool, and to provide participants with enough time to create their charts and share them with one another in the larger group. Similarly, in adapting the Initiating and Elaborating a Causal Loop Diagram script, the co‐facilitator/modeler used Vensim to create the causal loop diagram and Zoom to share the screen with participants. The research team also extended the time budgeted for this activity from the recommended 20 minutes to 80 minutes to provide more time to explain the activity and in anticipation that it would take longer for participants to get comfortable with the process before more meaningful insights would emerge.

Workshop implementation

On the day before the workshop, the research team met in‐person to pilot planned activities and test the online meeting platform and software tools. On the day of the workshop, the cofacilitators (KKB, MKL) set up two computers in a research office. The first computer was used to host the Zoom meeting and present PowerPoint slides. The second computer was used for workshop activities that involved the use of online tools such as Google Jamboard and Vensim. This setup allowed the co‐facilitators to support one another, quickly identify and address technical problems, and adapt facilitation as needed. The remainder of the research team were online and provided session support remotely. Table 1 presents an overview of the workshop agenda, time allotted for each agenda item, and the tools used for each activity, and the eight workshop agenda items are discussed below.

Table 1

Detailed Overview of Workshop Agenda, Activities, and Tools

Time	Activity	Tool	Script or Exercise	Steps
8:00 am‐8:30 am	Introductions to each other, this project, and systems thinking as a tool	Zoom		Participants filled out premeeting surveys Introductions by the research team and participants An introductory overview of the project Stakeholders were introduced to the use of systems thinking as a tool and the language of system dynamics
8:30 am‐11:00 am	Conceptualizing Black maternal mortality a system	Zoom Google Jamboard	Five Rs Framework (United States Agency International Development, 2016)	For each of the Five R's (results, roles, relationships, resources, and rules), the first step was a divergent activity, followed by a convergent activity. Divergent activity: Using their own paper, stakeholders individually listed all the variables they could think of for each R domain Convergent activity: Stakeholders called out variables that were not yet identified by other participants and the variables were added to individual virtual sticky notes on the whiteboard. The facilitator clustered the virtual sticky notes into themes in real time.
11:00 am‐12:00 pm	Discussing important dynamics in Black maternal mortality	Zoom Google Jamboard	Graphs Over Time (Hovmand et al., 2015)	Divergent activity: Stakeholders drew behavior‐over‐time graphs for key variables that showed time on the x‐axis and change in the variable of interest on the y‐axis, with notations that defined those key changes over time that shaped trends Convergent activity: Stakeholders discussed their behavior‐over‐time graphs with the group
12:00 pm‐12:30 pm	Lunch
12:30 pm–1:50 pm	Developing a model boundary chart	Zoom Google Jamboard		Convergent activity: The group revisited the Five Rs, and stakeholders added additional R variables as needed. Convergent activity: Using multiple whiteboards, the stakeholders identified which R variables to keep, save for later or exclude. Stakeholder used marker dots to vote on each whiteboard.
1:50 pm–3:30 pm	Developing a shared hypothesis of the salient determinants of Black maternal mortality (building a “causal loop diagram”)	Zoom Vensim	Initiating and Elaborating a Causal Loop Diagram (Hovmand et al., 2015)	Convergent activity: Using previously solicited information and stakeholders’ input, facilitator diagramed salient determinants of problem as a causal loop diagram (CLD). The facilitator created the CLD in Vensim on a shared screen in Zoom so that stakeholders could review it, provide feedback, and suggest additions in real‐time.
3:30 pm‐4:40 pm	Brainstorming targets for action	Zoom Google Jamboard	Action Ideas (Hovmand et al., 2015)	Facilitator introduced idea of “levels of leverage” to stakeholders, to trigger more action ideas targeting higher levels of leverage Divergent activity: Stakeholders reflected on the diagrams the group created and brainstormed actions, policies, and interventions that could improve outcomes
4:40 pm‐5:00 pm	Closing session and feedback surveys	Zoom		Stakeholders filled out brief feedback surveys Stakeholders were thanked for their participation

Workshop opening

At the start of the workshop, the facilitators opened with an acknowledgment of the Black women and families who have been affected by maternal mortality. The facilitators then led an icebreaker activity that allowed each participant and research team member to introduce themselves with guided questions. Following introductions, the facilitators provided participants with a set of group agreements to review, discuss, and modify. Once the participants reached consensus on these agreements, the facilitators introduced the project and its purpose, with specific focus on persistent racial disparities in maternal mortality and the concepts and promise of systems thinking and system dynamics to address the problem.

Workshop Activity #1: Conceptualizing Black maternal health as a system

The facilitators used Google Jamboard to facilitate a Five Rs activity (United States Agency International Development, 2016) designed to allow participants to identify variables that are relevant to Black maternal mortality. In the context of reducing Black maternal mortality, the activity was used to elicit what success looks like (“Results”), who affects or is affected by changes in the outcome (“Roles”), the important relationships between the roles that affect, or are affected by, the results (“Relationships”), the formal and informal rules that affect the results (“Rules”), and the resources that are available to improve the results (“Resources”). First, participants were asked to write down the five most important “Results” variables related to Black maternal mortality. Next, participants took turns reading their variables in a round‐robin fashion. For each variable, the facilitators asked if other participants had similar variables. The facilitators encouraged participants to skip any redundant variables when it was their turn to read a variable. As participants read their variables, one of the facilitators wrote them on a virtual sticky note in Google Jamboard for everyone to see, clustering them into emerging themes in real time. The facilitators completed the same steps for each of the remaining R categories. Figure 1 shows the Jamboard whiteboard used for the Five Rs activity.

Fig. 1

Image of Google Jamboard Whiteboard for the Five Rs Activity [Color figure can be viewed at wileyonlinelibrary.com]

Workshop Activity #2: Discussing important dynamics in Black maternal mortality

This activity consisted of building behavior‐over‐time‐graphs (BOTGs) in Google Jamboard, with the goal of engaging participants in framing the problem, eliciting additional variables, and gathering input to decide the reference modes for the modeling process (Hovmand et al., 2015). First, one facilitator (MKL) introduced concepts and terms related to mental models, model boundaries, model boundary charts, and time horizons. Next, the second facilitator (KKB) showed participants an example of a BOTG that was created in Google Jamboard, followed by a step‐by‐step walk‐through of how to create and annotate their BOTG using the template provided. The facilitators asked participants to identify a variable that they deemed as most important to Black maternal mortality and to use the chosen variable in their BOTG. The participants used their individual whiteboards in Google Jamboard to develop their BOTGs while the facilitators monitored their progress. After completing their BOTGs, participants took turns explaining it to the larger group, during which facilitators engaged participants in a discussion about the similarities, differences, and overarching insights from the BOTGs. Figure 2 shows an example of a participant's BOTG in Jamboard.

Fig. 2

Image of a two example BOTGs developed by a participant in Google Jamboard [Color figure can be viewed at wileyonlinelibrary.com]

Workshop Activity #3: Developing a model boundary chart

The facilitators presented the variables from the Five Rs activity and asked participants to indicate whether new variables needed to be added to the list. Next, using Google Jamboard, the facilitators presented participants with the variables within each of the Five R categories. Participants were asked to identify which variables to “keep” (i.e. endogenous), “save for later” (i.e. exogenous), or “disregard” (i.e. excluded) for subsequent workshop activities. Using a shared Google Jamboard whiteboard, the participants used digital dot markers to vote on the variables to be kept. The variables with the most votes were considered endogenous.

Workshop Activity #4: Developing a shared hypothesis of the salient determinants of Black maternal mortality

This activity focused on developing an initial causal loop diagram (CLD), using the adapted Initiating and Elaborating a Causal Loop Diagram script (Hovmand et al., 2015). Because of the large number of potentially important reference modes to the problem, we focused on one reference mode—Black maternal mortality—that the CLD was intended to explain, and this was determined by a vote on participants’ BOTGs. This allowed the research team to effectively engage participants within the time and experiential constraints of the workshop and develop a meaningful initial CLD. The facilitators used Zoom's screenshare feature so that participants could see the emerging causal loop diagram in Vensim and actively provide insights and feedback that were discussed and incorporated in real time if there was consensus among the group. Figure 3 is a screenshot of the facilitator/modeler (MKL) cocreating the causal loop diagram with workshop participants.

Fig. 3

Image of facilitator and participants coconstructing a causal loop diagram using Vensim and Zoom [Color figure can be viewed at wileyonlinelibrary.com]

Workshop Activity #5: Brainstorming targets for action

For the final major workshop activity, the Action Ideas script (Hovmand et al., 2015) was used to brainstorm potential preventive actions to mitigate Black maternal mortality. Google Jamboard was used to display a 2×2 matrix reflecting level of potential impact (“low impact” to “high impact”) and level of difficulty (“easy” to “hard”) for each action. First, the facilitators asked participants to reflect on the discussion and insights from the workshop and write down their top three action ideas. Next, participants took turns sharing one of their action ideas in a round‐robin fashion. For each action idea, the facilitators asked participants to indicate the level of potential impact and level of difficulty for their action idea. The facilitators also asked if other participants had similar action ideas that needed to be added. Participants were encouraged to skip any redundant action ideas when it was their turn to share. As participants shared their action ideas, one of the facilitators added it to a virtual sticky note and placed it in the matrix for everyone to see. Figure 4 demonstrates the action ideas that were identified by the participants within the 2×2 matrix.

Fig. 4

Image of Brainstorming for Action Targets activity in Google Jamboard [Color figure can be viewed at wileyonlinelibrary.com]

Workshop Closing

At the end of the workshop, the facilitators summarized the key activities and the key insights gained from the CLD and engaged participants in a brief discussion about the future directions of this work. Facilitators also asked participants to indicate their interest in participating in follow‐up discussions, asked them to complete a postworkshop evaluation survey, and thanked them for their time.

Member‐Checking Meetings

The initial CLD that emerged from the workshop required a series of clarifying revisions by the research team. One month after the workshop, the research team held a series of three member‐checking meetings to verify the accuracy of the revised CLD. During the first meeting, the research team presented clarifying questions, and participants discussed concerns about some of the feedback mechanisms or language in the CLD that were inaccurate. Two additional meetings were conducted to review further edits and verify accuracy. After the final meeting, the research team recorded a video explaining each major feedback mechanism in the CLD and asked participants to confirm the accuracy of the final changes. All participants provided their agreement with the final CLD via email.

Workshop evaluation results

A pre‐ and postworkshop survey was used to evaluate workshop effectiveness. The survey design was informed by the Kirkpatrick evaluation framework that recommends four levels of assessment: Level 1: reaction (did participants enjoy the workshop?); Level 2: learning (what did participants learn from the workshop?); level 3: behavior (what skills or behaviors have participants gained?); and level 4: result (did the workshop influence organizational performance?) (Kirkpatrick and Kirkpatrick, 2016). Because of the short term nature of this SD GMB workshop, the survey addressed only the first three levels of assessment. All nine participants completed the presurvey, and seven participants completed the postsurvey.

Reaction

On the postsurvey, participants were asked to rate the quality of the overall workshop and each workshop activity. All seven respondents reported that the facilitators explained concepts in an easy‐to‐understand manner, that the workshop activities were engaging, and that the online tools used during the workshop were effective. All seven respondents rated the overall quality of the workshop as good/excellent. Moreover, participants were asked to rate each of the five core workshop activities. Six of the seven respondents (85 percent) rated the BOTG activities as good/excellent, and all seven respondents rated the other four workshop activities as good/excellent.

All seven respondents also reported that they were able to contribute their personal experience or expertise during the workshop, and six of the seven respondents (85 percent) reported interest in participating in future workshops. Finally, respondents were asked to describe what they enjoyed most about the workshop and what improvements could be made for future workshops, and these insights are demonstrated in Table 2.

Table 2

Participant Feedback on Workshop Strengths and Areas of Improvement

What did you enjoy most about this workshop?

What improvements could we make for future workshops?

Hearing from specialists in fields outside of my own I loved dissecting the 5Rs and the behavior‐over‐time graphs The 5Rs and the causal loop diagram How safe the space felt to share without judgment Joining in with other organization members The diversity in participation

Having half‐day sessions would be better than one full‐day session from an engagement perspective. In‐person would be ideal to help with reading facial expressions and body language when discussing concepts that could make participants unsure of themselves. In‐person [meeting] would be great Divide the days More music and/or having a yoga instructor to help facilitate stretching or moving in between sessions Not so long I am a person who cannot stand silence, how can the silent partners be encouraged to speak? They have much to offer.

Learning

On the postsurvey, all seven respondents reported that their participation in this workshop increased their insight into key factors related to racial disparities in maternal mortality and that the workshop increased their insight into possible systems‐level interventions that could reduce racial disparities in maternal mortality. The pre‐ and postworkshop surveys asked participants to rate their level of knowledge of systems science/systems thinking, definition and use of behavior‐over‐time‐graphs, definition and use of causal loop diagrams, and application of systems science to intervention development. Based on the pre‐post data, respondents demonstrated self‐reported knowledge increases across each of the workshop topic areas (Table 3).

Table 3

Percent of participants reporting

	Preworkshop n (%)	Postworkshop n (%)
Self‐reported ratings of knowledge by topic area
Systems Science or Systems Thinking	2 (28%)	7 (100%)
Definition and Use of BOTGs	0 (0%)	5 (71%)
Definition and Use of CLDs	0 (0%)	4 (57%)
Application of Systems Science to Intervention Development	3 (43%)	6 (86%)
Self‐reported behavioral outcomes
I plan to apply the information about systems thinking from this session into my own work.	N/A	7 (100%)
I plan to apply information about possible intervention strategies from this workshop to my own work.	N/A	7 (100%)

Behavior. On the postsurvey, all seven respondents reported an intent to apply the information about systems thinking from this session to their own work in the community and reported an intent to apply information about possible intervention strategies from this workshop to their own work in the community (Table 3).

Lessons learned and future directions

Nine lessons and corresponding recommendations for future projects emerged during the online SD GMB workshop.

Recognize that an online format does not guarantee increased participation

The workshop included three stakeholders who were experts in maternal health policies, but there was no representation from local and state policymakers who indicated that they were unable to dedicate a full day to participate in the workshop. The challenge of recruiting policy stakeholders to participate was not surprising, but the research team hoped that the online format might increase their chances of participation. Because policy stakeholders’ buy‐in is so crucial to implementation and sustainability efforts, the research team suggests the implementation of targeted strategies (e.g. relationship building, framing SD GMB as a tool to address constituency needs) for increasing policy stakeholder involvement in future online SD GMB workshops (Otten et al., 2015).

Ensure that online meeting platform and software tool(s) align with workshop needs and activities

Zoom and Google Jamboard were effective tools for the completion of the planned workshop activities. Zoom was likely effective because of participants’ existing familiarity with its features. While other researchers have reported using Miro (Wilkerson et al., 2020), Google Jamboard allowed participants to easily access and navigate its whiteboard format because of its simple and intuitive design. Although these tools were effective for this online SD GMB workshop, researchers should avoid a “one‐size‐fits‐all” approach to platform and tool selection and focus on aligning these choices with their online workshop needs and participant backgrounds.

Consider stakeholder preferences in the design of online SD GMB workshops

Gathering participant preferences prior to the workshop was helpful for choosing online tools that met accessibility standards. However, despite indicating their preference for the one‐day format, participants reported dissatisfaction with the one‐day format. Further, some participants reported a preference for face‐to‐face format, despite threat of the COVID‐19 pandemic. This unexpected finding underlined the importance of social interaction (verbal and nonverbal) and relationship building in the SD GMB process and raised questions about the potential for other approaches. For example, future work might consider hybrid workshops that consist of an initial face‐to‐face session followed by online sessions. A hyflex approach whereby some participants are face‐to‐face and some participants are remote may be worth investigating although this is not recommended elsewhere (Wilkerson et al., 2020).

Aim for “depth over quantity” in the design of online SD GMB workshops

The issue of time constraint presents a challenge for any SD GMB workshop, regardless of modality (Laurenti et al., 2014; Lembani et al., 2018; Pugel and Walters, 2017). The research team deliberately planned five core workshop activities that would result in in‐depth findings rather than many activities that provided rushed, surface‐level insights. Drawing from their online teaching experiences, the research team recognized that workshop activities in the online environment would likely take twice as long as face‐to‐face workshops to allow facilitators to adequately introduce, explain, and demonstrate tasks associated with the key activities to participants. In line with prior recommendations (Wilkerson et al., 2020), the research team also planned for additional time that would be needed for technology glitches. Future online SD GMB workshops would likely benefit from similar time considerations.

Apply theory‐driven approaches for online SD GMB workshop facilitation and engagement

Researchers engaging in online SD GMB workshops should consider applying TIR and other online learning principles to aid in managing online group dynamics (e.g. establishing group norms, identifying speakers by name) (Schutz, 1958; Thorpe, 2016). Researchers should also allow multiple modes of participant engagement (e.g. the Zoom chat feature) in the online environment. This approach worked well for the workshop because some of the participants were caregivers who wanted to stay engaged via the chat but needed to turn their cameras off to tend to caregiving responsibilities. Another recommendation is to encourage participants to keep their cameras on during the first 2 hours of the workshop, and then making the use of their camera optional thereafter to avoid Zoom fatigue. However, the research team kept their cameras on during the entire workshop. Based on the online learning literature, when facilitators keep their cameras on it supports better engagement and learning (Martin, 2019); therefore, we recommend that research teams in future online SD GMB workshops ensure that their cameras remain on.

Plan to provide technical support for online SD GMB workshop participants

One member of the research team was responsible for providing technical assistance to participants via chat. However, the research team realized that they lacked a formal process for technical support. For example, there were instances where it was difficult to provide chat‐based support to participants who needed more specific help with Google Jamboard. Therefore, one recommendation is to assign team members to specific technical support roles for participants. A second recommendation is to use the meeting platform to create a virtual breakout room, in which participants can receive individualized technical support and troubleshooting.

Consider various facilitation models for online SD GMB workshops

For this SD GMB workshop, all community stakeholders participated virtually, and two facilitators led the workshop from a research office while other members of the research team attended virtually. This structure proved to be effective in balancing the risks of convening community stakeholders in a face‐to‐face format with potential research insights that could be achieved with increased in‐person interaction. It is expected that this model of facilitation could also be effective in a post COVID‐19 era. For example, there is a benefit for researcher team members in geographic areas with unreliable residential Internet services to convene in a location that has more reliable Internet access (e.g. university campus, library). This approach might also be useful for research team members who are working together for the first time to pilot an SD GMB workshop. Given the limited information about online SD GMB, the varying models of facilitation, and their relative efficacies, researchers are encouraged to experiment with both facilitation models to evaluate their comparative efficacy in future SD GMB workshops.

Incorporate evaluation to document learning, guide‐quality improvement, and advance the SD GMB literature base

Workshop evaluation pre‐post surveys were crucial for documenting improvements in stakeholder knowledge and skills and gauging stakeholder satisfaction with workshop design. Moreover, the evaluation of stakeholder satisfaction was useful for identifying areas of improvement in workshop design and implementation. Prior studies have used various evaluation methods such as pre‐post surveys, content analysis of workshop conversations, and participant interviews (Rouwette et al., 2002; Scott et al., 2016). Future SD GMB projects should incorporate evaluation to gather insights into workshop design and implementation considerations to expand the corresponding literature and build a community of learning.

Conclusions

In this note, the authors shared key insights and recommendations from their experience planning, implementing, and evaluating an online SD GMB workshop. Conducting online SD GMB workshops, especially during times of social disruption, comes with unique challenges and opportunities related to workshop design, facilitation and engagement, and the selection of tools that support SD GMB activities. The insights from this note are consistent with growing literature indicating that online SD GMB is worthwhile and warrants further investigation. Further, this note shared several insights that were consistent with prior articles. For example, online SD GMB workshops should incorporate frequent, short breaks, allocate enough time for technical issues, consist of effective, well‐planned facilitation, and ensure a balance between participant and facilitator manipulation of elements in the online environment (Wilkerson et al., 2020; Zimmermann et al., 2021). Also, the current project's group size of nine stakeholders was sufficient which is consistent with the group size recommendation of five to 10 stakeholders in prior articles (Wilkerson et al., 2020; Zimmermann et al., 2021). However, there were several unique contributions in this note to the growing online SD GMB literature, including: (i) the utility of a novel software tool for online group‐model‐building activities (Google Jamboard, used in conjunction with previously introduced tools); (ii) the application of theory‐driven facilitation and engagement strategies that can be tested for effectiveness in future projects; (iii) critical nuances involved in effectively incorporating participant input (e.g. accessibility needs, desired length) into workshop design; (iv) the effectiveness of a model that consists of facilitators convening in‐person to conduct an online SD GMB workshop; and (v) the importance of systematically evaluating online SD GMB workshops to guide continuous quality improvement.