Virtual Symposium Designed as an Innovative Approach to Disseminate Leadership Development Content.

The American Association of Colleges of Pharmacy (AACP) Leadership Development Special Interest Group (LD SIG) embarked upon a one-week virtual symposium (VS) across three years. The purpose of the VS was to disseminate leadership content, case studies, and various topics of interest utilizing technology. This electronic format provided a new communication approach to facilitate content and discussion of leadership materials amongst faculty colleagues nationally. The VS allowed participants to gain knowledge and skills to address leadership development in a timely, flexible manner that can be challenging during in-person professional meetings and conferences due to conflicting programming and travel limitations. Evaluators across each of the VS (100%) indicated that they either agreed or strongly agreed with the scaled evaluative statements that the video presentations met their educational needs. © Individual authors.

INTRODUCTION

The American Association of Colleges of Pharmacy (AACP) Leadership Development Special Interest Group (LD SIG) seeks to provide leadership development information in a simple, timely, and interactive manner. Rather than solely providing this information in face-to-face professional conferences, which are not as widely accessible for viewing as virtual information sharing forums can be, it was determined that a Virtual Symposium (VS) would be an innovative way to expand educational access to the LD SIG’s leadership development information. The purpose of a virtual conference is to create a stimulating and supportive learning environment while reducing the in-person availability, allowing participants to engage at various times.[1] The purpose of the VS was to expand accessibility to leadership development information for LD SIG members. Virtual conferences allow multiple participants to attend from various locations that would not have otherwise been able.[1,2]

While development of the VS improves the overall accessibility of information, social interactions are more limited than for in- person conference attendees, and barriers caused by technology can also impact the attendees’ experience. Communication barriers that take place during virtual learning activities can prohibit participation.[3] This can be especially true for diverse communities participating in the VS.[3] While research is limited on the benefits of effective learning in virtual conference environments, the practical benefits to access in virtual conferences may include reduced expenses for transportation, lodging, and incidentals, as well as time lost during travel.[1,2] In order to address these needs, the LD SIG developed a VS for members.

In June 2018, the LD SIG launched an inaugural VS, which has continued over three years, and is expected to continue for many years to come. The number of presenters has grown substantially each year, along with the engagement of the participants. Each year of the VS identifies a unique topic and area of interest for the LD SIG members. In 2018, presenters were selected by the LD SIG Programming Committee for the topic “Leadership Frameworks: One Size Doesn’t Fit All”. Frameworks included concepts taught within classes or theories connected to courses. In 2019, the VS included twelve presentations related to the topic of “Creating Excitement for Student Leadership Development.” The 2020 “Virtual Poster Symposium” doubled the 2019 presentation numbers, with twenty-four presentations addressing several sub-topics including: curricular initiatives, co-curricular initiatives, and faculty/preceptor initiatives.

The purpose of the inaugural VS was to prepare educators to address ACPE Accreditation Standard 4, specifically Key Elements 4.2 (leadership) and 4.4 (professionalism) by exploring novel approaches to leadership framework implementation and assessments that meet the leadership development needs of students. Specific objectives of the second LD SIG Virtual Symposia included: 1) Evaluate various leadership approaches and assessments that educators have adopted at their home institutions that excite students about the importance of their own leadership development. 2) Discuss opportunities to improve the leadership curriculum at the current level of leadership content in the curriculum. 3) Design a plan for integrating a leadership strategy. The objectives of the Virtual Poster Symposium were consistent with the project objectives described in each of the posters.

METHODS

Members of the LD SIG were invited to submit proposals for the 2018 and 2019 VS, which were evaluated by the LD SIG programming committee. The LD SIG identified leadership topic posters presented at the 2019 annual meeting and invited presenters to encore present at the 2020 VS. Upon acceptance of the submission, presenters completed their video presentations and submitted them. VS presenters were asked to limit their presentations to a 5-10 minute duration.

The LD SIG VS provides case study vignettes and leadership- focused presentations to AACP members at a variety of stages in leadership development (beginner, intermediate, and advanced). While the third VS focused on topics previously presented in poster format on a variety of leadership topics, the first two years of the VS focused primarily on case study vignette presentations focused on a topic within the realm of leadership development. For case vignettes, presenters were instructed to use the following template for presentations: 1) introduction of leadership framework(s) used, 2) steps followed in implementation, 3) outcomes observed, and 4) lessons learned. Virtual Poster presentations follow the usual format of poster presentations, including 1) objectives, 2) methods, 3) results, 4) conclusions.

Digital Platform

Selection of a digital medium to share information was the key logistical challenge of the design phase of the VS, but has also proven to be the most important enabling factor for the event. The LD SIG chose to utilize a virtual professional networking platform system called AACP Connect®, which is AACP’s platform designed for AACP Sections, SIG Communities, and Councils to facilitate communication through discussion boards. While other video-sharing digital mediums like YouTube may allow for greater flexibility in video content, this decision to utilize AACP Connect® was multifactorial. AACP Connect® enables sharing of video content, offers the opportunity for members to view and respond to the videos, allows both presenters and attendees to post comments or questions in a virtual discussion board that is visible to all members of the LD SIG and archives presentations for future viewing (Figure 1). Additionally, AACP Connect® was already a familiar platform for LD SIG members to communicate through at the time of the inception of the VS. Finally, digital metrics are tracked and readily available on AACP Connect®, which allows the LD SIG to assess engagement outcomes and incorporate those data points into the assessment of the VS. Therefore, it was the obvious digital platform choice for the LD SIG VS.

Figure 1.

Virtual Professional Networking Platform used to host the Virtual Symposium (AACP Connect®)

Implementation Process

The steps taken each year to deliver the VS each year began with a “call for presenters” distributed to the leadership development members to submit the case study vignettes or poster presentations. Unless the content being proposed had already undergone peer review, which was the case for the Virtual Poster Symposium presentations, but was not true for the case vignettes (e.g. 2018 and 2019 VS), presenters were required to submit an abstract to be peer-reviewed prior to being officially selected as a VS presenter. The LD SIG programming committee reviewed each abstract using the Presenter Rubric, selected and notified presenters with a deadline for video completion (Figure 2). All submitted videos were uploaded into AACP Connect® and made available to LD SIG members for asynchronous viewing. Similar to a round- table discussion, attendees were encouraged to post questions, thoughts, or feedback in the AACP Connect® Discussion Board.

Figure 2.

Timeline structure of the virtual symposium

During the week of the VS, presenters were asked to provide intermittent virtual presence to respond to any questions asked and to address any comments posted. Upon completion of the VS week, those members who participated and reviewed the videos were sent an evaluation to complete, as well as a “Certificate of Completion,” if desired. Upon completion of each VS, videos were permanently archived in AACP Connect® to allow for perpetual viewing of the content by LD SIG members (Figure 4). Because of the large number of video submissions in 2020, the videos were divided up by leadership sub-topic, with 4-6 videos being posted each day for all five days of the VS. The timeline for VS activities is summarized in Figure 3 and Table 1.

Figure 3.

VS Presenter Rubric for Case Vignettes

Table 1.

Virtual Symposium timetables

Timeline for preparation before VS:

Timeline for activities during and after VS:

Program Engagement

Since the VS was conducted through the electronic professional networking platform, AACP Connect®, attendee engagement metrics were available to query. User engagement data including attendee downloads and views were available for every VS video. Detailed, individualized attendee data were also available for everyone participating in the VS. In a similar manner across all three years of the VS, user engagement data were retrieved during the specific 14-day timeframe spanning between 7 am on the Monday morning of the VS start date through the week of the VS and all the way through the week after the VS. Similarly, AACP Connect® LD SIG discussion board posts, replies, recommends, followers, and unique contributors were also evaluated across the same timeline each year of the VS. In 2020, an additional data point was collected from presenters quantifying the number of received and sent messages related to the VS that were also channeled through presenters’ personal email inboxes. Also, views of 2020 best poster award reviews were completed outside AACP Connect, so these views were manually added to the engagement numbers obtained from the electronic professional networking platform dataset.

Program Assessment

Each year after the Virtual Symposium, attendees were asked to complete an electronic evaluation indicating which of the sessions they attended, rate their level of agreement with a few statements related to the ability of the VS video presentations to meet educational needs, provide timely and relevant information, meet the stated presentation objectives, and to serve as an effective learning tool. Attendees were also asked to provide any highlights or take away points from the Virtual Symposium, suggest ideas for improvement, and indicate if they would like to see more Virtual Symposium offerings in the future.

Statistical Analysis

Given the non-normal distribution of the data for all evaluation items during all years of the VS, the Kruskal-Wallis test was conducted to assess for differences in mean ranks of attendee agreement scores between the three different years of the VS (2018-2020). In order to assess for between year differences that may have driven the significant differences in mean ranks for these two items, a pairwise two-sided multiple comparison analysis was performed using the Dwass, Steel, Critchlow- Fligner Method. Statistical analysis was conducted in SAS for Windows, version 9.4 (SAS Institute, Inc, Cary, NC).

RESULTS

Engagement Metrics

The highest number of total content views occurred in the inaugural VS in 2018, compared to all other years. However, the number of unique attendees viewing VS content increased when comparing 2020 to 2018 attendees, indicating an expanding reach of the program to an increasing number of LD SIG members over time. New engagement metrics monitored during the 2020 Symposium included a survey distributed to presenters, with eighteen out of twenty-four presenters responding (75% response rate), indicating that the total number of emails or AACP Direct Messages (not Discussion Board posts) received by VS presenters from VS Attendees during the 2020 VS was 28 (mean = 1.56, S.D. = 1.42). Thirty- three additional communications (emails or AACP Direct Messages (not Discussion Board posts) that were sent to VS attendees by VS presenters during the 2020 VS (mean = 1.83, S.D. = 2.83). Discussion Board data also showed positive trends over time, with increases in number of Discussion Board posts, replies, and contributors in 2018, 2019, and 2020.

Assessment Metrics

In 2018, ten faculty presented in the inaugural VS. Ten program evaluations were also completed and submitted after the VS. This program evaluation included four scaled evaluative statements and three open-comment questions (Table 2). One hundred percent of respondents rated each of the scaled questions ‘strongly agree’ or ‘agree.’ Ninety percent noted they would like to see more VS offerings from the LD SIG. Results were similar in 2019, with twelve faculty presenters sharing ways to breathe new life into teaching to excite students about the importance of leadership development, and how they have overcome barriers. Nine program evaluations were submitted in 2019, and once again, all respondents either agreed or strongly agreed to the four scaled evaluative statements, indicating a positive response to the VS content. Twenty-four posters were presented during the 2020 VS. The purpose of the 2020 VS was to showcase posters that members may not have viewed the previous year during the face-to-face professional meeting. Eleven individuals completed the VS program evaluation in 2020. As in prior years, all evaluators of the 2020 VS (100%) indicated that they either agreed or strongly agreed with the scaled evaluative statements that the video presentations met their educational needs (36% Strongly Agreed), were relevant, timely, and up-to-date (55% Strongly Agreed), met the objectives described in the poster abstracts (45% Strongly Agreed), and were an effective learning tool (36% Strongly Agreed) (Table 3). The highest mean agreement scores for each of the four scaled evaluative statements occurred during the inaugural year of the VS, in 2018. Thereafter, mean agreement scores for each of the evaluation items numerically declined very slightly, but these numeric changes were only statistically significant for two items. For these two items, “The video presentations met the stated objectives” and “The video presentations were an effective learning tool,” the distributions of 2018, 2019, 2020 attendee agreement score were not similar for each year, as assessed by visual inspection of a boxplot. For the item assessing whether the presentation met the objectives, the distributions of attendee agreement score mean ranks were statistically significantly different between groups (Kruskal Wallis Chi-Square Test Statistic =7.5 (degrees of freedom =2), p =0.0239).

Table 2.

Virtual Engagement Metrics: Virtual Symposium Evaluation Responses 2018-2020

Virtual Engagement Metrics	2018	2019	2020
Video Presentation Data
Number of presentations	10	12	24
Total Content views by attendees	180	76	152*
Total number of unique attendees	53	42	66
Average views per presentation	18	5.8	6.5
Discussion Board Data
# Original Discussion Posts	13	16	27
–# Replies to Discussion Posts	9	23	22
# Unique Discussion Contributors	2	2	5

*includes poster evaluator data

Table 3.

Program Assessment Metrics

Program Assessment Metrics	2018 (n=10)	2019 (n=9)	2020 (n=11)	p-values
The video presentations met my educational needs	4.78	4.56	4.36	0.1906
The video presentations were relevant, timely, and up-to-date	4.90	4.78	4.55	0.1845
The video presentations met the stated learning objectives	5.00	4.56	4.45	0.0193*
The video presentations were an effective learning tool	4.90	4.67	4.36	0.0412*

*p<0.05 indicates statistically significant differences

Likert Scale: 1=strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree

For the item assessing the effectiveness of the video presentations as learning tools, the distributions of attendee agreement score mean ranks were also statistically significantly different between groups (Kruskal Wallis Chi-Square Test Statistic = 6.3 (degrees of freedom =2), p = .0412). In order to assess for between-year differences that may have driven the significant differences in mean ranks for these two items, a pairwise two-sided multiple comparison analysis was performed using the Dwass, Steel, Critchlow-Fligner Method. Attendee agreement levels were significantly lower for both of these items in 2020 compared to 2018 (p=0.0192 and p=0.0363, respectively). No significant differences between 2018 and 2019 or 2019 and 2020 responses (for this item evaluating the ability of the presentation to meet the objectives) were observed.

Conversely, the two other scaled evaluative statements did not decline significantly over time. For the two statements, “The video presentations met my educational needs” and “The video presentations were relevant, timely, and up-to-date,” a visual boxplot inspection did reveal dissimilar distributions of agreement scores across all years. Although the agreement score mean ranks decreased numerically from 2018 to 2019 and 2019 to 2020, the differences were not statistically significant, χ[2](2) = 3.38, p = .18.

The VS program evaluation also included attendees’ comments that the key highlights of the VS in 2018 included the relevant content and several noted takeaways of ideas to incorporate into their curriculum/co-curriculum. Attendee highlights and take-aways from 2019 included: discovery of new leadership books, active learning techniques for leadership development, and leadership development opportunities within experiential education. Attendees also requested hyperlinks to presenters’ institutional biographies and suggested a future Virtual Symposium theme of “Assessment of Leadership.” Constructive feedback from both 2019 and 2020 suggested that the videos be released in a much more longitudinal manner, such as 1 or a few each week, rather than multiple videos in one day or across one week so that it would not be such a significant time commitment all in one week. Another point of feedback from an attendee was related to the difficulty of following discussion board conversations because of the way they are structured within the professional networking platform used. For 2020, some highlights and take-aways included ideas for development of a longitudinal leadership course and well as increased opportunities for professional networking related to leadership development.

Discussion

Disseminating case studies through a VS is a practical approach to providing faculty with the knowledge and tools to address leadership development in a timely, flexible manner. This format provided a new communication approach to facilitate content and discussion of leadership materials amongst faculty colleagues nationally. The inaugural VS of 2018 arose out of necessity. There was a demand for leadership development information-sharing from within the SIG and a lack of an appropriate medium to host those discussions. After the LD SIG’s programming application was denied in Spring of 2018, the LD SIG’s programming committee quickly implemented a plan to solicit video presentations under the theme of “Leadership Frameworks: One Size Doesn’t Fit All” and host these presentations on a digital platform.

LD SIG’s decision to utilize AACP Connect® as the digital platform of the VS meant growth was tied to the platform. As the platform functionality grew, the VS could similarly expand both types and numbers of presentations that could be feasible. AACP Connect® allows for presentations posted in several video formats, directly uploaded from a local source or file-sharing sites. Additionally, the integration of drafting and scheduling posts provided the ability to upload content asynchronously while specifying designated posting times. The 2020 VS agenda, which included twice the content previously offered, spread out over five days, was directly related to the improved functionality of AACP Connect®.

The VS format changed slightly in the third year of programming. The decision to include poster presentations in the 2020 VS resulted from a realization that, even during in- person professional meetings and conferences, information dissemination can be difficult. These challenges can include simultaneous offerings of competing programming, difficulty in quickly and efficiently identifying a group of posters to view related to a single theme such as leadership. As well as, inadequate time to view all poster presentations of interest during a single poster session. Given the wealth of information that is only available for a very small window of time in a live poster session, providing virtual encore presentations in the VS enables a much broader and more efficient dissemination of this otherwise underutilized wealth of information and expertise related to leadership development in pharmacy education. This medium allows for a greater level of information sharing, not beholden to the confined time of the specified poster presentation session. This allows what was once a one-time networking opportunity to become archived leading to an ongoing conversation.

The inaugural VS demonstrated strong engagement in 2018 but has continued to exhibit positive growth in engagement metrics in subsequent years. For example, there was an increasing number of presentations over time, which approximately doubled by the third year when compared to years one or two. Similarly, there was a general increase in the number of unique attendees, with an increase between year three (2020) when compared to year one (2018). These metrics indicate increasing interest in VS involvement, both from the presenter and the attendee perspectives. These engagement metrics indicate increasing engagement of the LD SIG over time through the VS.

The slight decrease in total content views after the inaugural year is likely related to the nature in which attendees would have participated in the Virtual Symposium poster format.4,5 With a more a la carte engagement for the wide variety of topics provided in the poster Symposium, attendees may have just viewed videos in their particular interest area. This is compared to the thematic approach to the VS taken in 2018 and 2019, which would have led the attendees to view multiple related presentations when engaging with the VS content, rather than just a single video. Additionally, while the average number of views per presentation decreased slightly over time, this is likely due to the increasing number of presentations over time. The fact that 2018 and 2019 presentations were topical, compared to 2020 when the data presented were scholarly, could have also impacted engagement metrics.

Although there were slight numerical declines in program assessment metrics following the inaugural VS year, globally, the program metrics were very positive. Attendees either agreed or strongly agreed with all of the scaled evaluative statements included in the program evaluation across all years of the VS. While there were slight numeric decreases in the mean program evaluation metrics, the changes were minimal.

Additionally, while there were statistically significant declines in attendee agreement related to the achievement of learning objectives and video presentations, the practical significance of these declines is questionable. This information can still inform aspects of longitudinal program improvement, such as not allowing poster objectives to be the same as the VS learning objectives. VS poster presentations may achieve different learning objectives than those described on a poster for a project.

Requesting that VS poster presenters use a more specifically guided presentation format may be considered in future offerings, instead of allowing presenters to utilize a variety of formats. It may be advantageous to require future VS poster presenters to utilize a slide-set or record themselves providing an oral presentation rather than voice-over an isolated PowerPoint slide of a poster, which may appear less personal and engaging to an attendee. Also, continuing to monitor various VS structures and metrics over time may reveal if there is a difference in efficaciousness to attendees between the two program formats.

The non-significant change in attendee agreement with the video content meeting attendees’ educational needs and the timeliness and relevance of the information highlights that the content shared is appropriate but suggests tweaks to the format of the presentations should be further monitored and considered going forward. The qualitative feedback provided by attendees requesting that only 1 or a few presentations be released each week, rather than multiple videos in one day or across one week is another consideration for quality assurance of future VS. One challenge of VS appears to be difficulty for attendees to allocate significant amounts of focused time to reviewing content provided in a VS, compared to the protected time available at a face-to-face professional conference. VS content must be delivered across a sufficient time period to allow for it to reviewed gradually as the attendee’s schedule allows, amidst other ongoing professional activities.

The VS drew positive attention from the AACP organization. Members of the inaugural VS were asked to share the process and results during an AACP webinar in 2019. Also due to the COVID-19 pandemic, a consultation with the VS organizers took place as the AACP Annual Meeting transitioned to a virtual format. Overall, it opened up the opportunity for the LD SIG to become increasingly virtual, starting a podcast (LDPEcast.com) shortly after the initial VS with an overall plan to continue to promote a community of practice within the LD SIG.[6] The podcast featured presenters from the VS five times since it began in 2018.

Limitations

The use of AACP Connect® as the digital platform of the VS required the LD SIG to modify the format of the VS to account for the specific functionality of the platform. AACP Connect® discussion boards did not allow for scheduling discussion board posts in advance of the VS until year three of the VS. The technical logistics demand required to post each presentation in a timely manner was challenging. Additionally, while the video-sharing components of AACP Connect® have improved, they are not the primary function of the platform, and they do lag behind other video-sharing mediums or virtual conference platforms. The video-sharing limitations led to two instances over the three years of the VS with technical difficulties.

AACP Connect® users also receive content alerts in different ways, which may affect attendees’ interactions with the VS. Many members receive email alerts in digest form with content bundled together and sent daily or weekly rather than a real- time alert system. There is a significant delay with this model for those members who do not use real-time alerts. Additionally, the concept of a VS is new and unique to AACP, which has likely contributed to reaching an overall small number of the over 1000 members of the LD SIG. Additional functions of the AACP discussion pages such as the ability to “follow” or “like” a post have yet to be utilized. The LD SIG plans to explore these features in future offerings to allow presenters and interested parties to keep continual updates on the conversations relating to a topic without changing their preferred email alert system. Additionally, more assessment metrics need to be collected over time to improve the robustness of the data and their application to continuous quality improvement for the VS.

CONCLUSION

The implementation of the VS was a valuable addition to the LD SIG. The unique format allows for greater information- sharing and member engagement across the SIG, without being beholden to the time, transportation, and financial cost of an in-person event. The specific agenda has changed annually, however participants report agreement that the presentations met their educational needs and were relevant, timely, and up-to-date. Participants also reported agreement that the presentations met the stated learning objectives and were an effective learning tool, however a statistically declining rate suggests strategic areas for process improvement in future iterations of the Virtual Symposium.

4 months	3 months	2 months	1 month	1-4 weeks	1 week
Select theme (if desired) and date	Deadline for abstract or participation interest submission	Notify presenters of selection	Deadline for video submissions	Promote VS to membership	Provide reminder to membership of virtual event
Call for presenters	Peer-review submissions (if required) and make selections	Provide presenters with technical guidance for video submissions (acceptable file types, sizes, etc.)

Day 1-5 (During VS)	Day 7-10 (After VS)	Day 10-14 (After VS)
Post videos	Email VS program evaluation to attendees	Review VS program evaluations
Stimulate virtual engagement (posts in the discussion board, presenters answer member questions and address member comments, etc.)	Email engagement survey to presenters (document VS-related engagement/email communications occurring outside of AACP Connect	Provide Certificate of Completion to attendees
		Archive VS presentations in AACP Connect® Library for perpetual viewing