Satisfaction With a Virtual Learning Collaborative Aimed at Implementing Treat-to-Target in Rheumatoid Arthritis.

OBJECTIVE: Limited information is available concerning experiences of participants in a virtual learning collaborative (LC), and little qualitative data or participant feedback on how this format can be improved. One prior in-person LC in rheumatology successfully improved adherence with treat-to-target (TTT) for RA. We conducted a virtual LC on TTT and herein report on participant satisfaction.
METHODS: We conducted a virtual LC with 18 rheumatology practices from across the United States during 2020 to 2021. The LC included a virtual kickoff meeting and monthly videoconferences, accompanied by data submission and feedback. At the conclusion of the LC, we surveyed the 45 LC participants concerning individual experience and satisfaction.
RESULTS: All sites and 78% of participants responded to the surveys. The LC included small and large practices, 14 academic and 4 nonacademic, and respondents ranged in their roles: 24 physicians, 5 nurses or nurse practitioners, 3 administrators, and 3 other roles. Overall, 94% of respondents indicated they were either somewhat or very satisfied with the LC, and 94% said they would recommend a similar LC to a colleague. Aspects of the LC described as "very useful" included a kickoff meeting, intersite discussion, and monthly speakers; however, digital tools such as the Web site and meeting recordings were not found useful.
CONCLUSIONS: Virtual LCs are feasible, and participants reported strong satisfaction. Virtual LCs were highly valued by rheumatologists, trainees, and their practice staffs. Potential topics were identified for future LCs that could improve the quality of care delivered to rheumatology patients.

Learning collaboratives (LCs) are an effective format for quality improvement projects across multiple practices. They address specific learning areas and offer feedback on quality measures through evidence-based measures, teaching, and discussion.[1] Learning collaboratives have been used in health care for several decades, focusing on various topics. The LC has been well described and includes clinicians and nonclinicians working together under the guidance of faculty with expertise in both the clinical content and quality improvement.[2]

Although the typical LC involves both in-person and virtual elements, several examples of LCs took advantage of a pure virtual format during the COVID-19 pandemic. These LCs covered a range of topics including best practices in tracheostomy, palliative care in oncology practices, and human papilloma virus vaccination adherence. Each collaborative reported high levels of success using a virtual format.[3-5] The LC on best practices in tracheostomy reported high levels of improvement in clinical assessments and found the virtual LC to be an effective teaching format. The LC tried to capture overall participant satisfaction but did not report anything outside of satisfaction with the collaborative generally on a Likert scale.[3] Similar improvements were seen in the LC on human papilloma virus vaccine adherence with noted improvements in vaccination initiation. All participants rated the project as “successful,” but no other information concerning satisfaction was reported.[5]

It is difficult to assess whether virtual LCs can facilitate participant discussion, faculty connection, and overall satisfaction compared with hybrid or in-person equivalents. The recent LC on palliative care in oncology reported high levels of effectiveness and was also able to address more interpersonal aspects of the collaborative. Participants reported that they found the coaching and experts in the LC valuable and noted relatively high levels of satisfaction with the virtual coaching. Overall, participants also found the virtual format useful.[4]

We describe satisfaction with an LC that was conducted in an exclusively virtual format during the pandemic. The current LC was based on a prior LC that included interpersonal meetings, focused on treat-to-target (TTT) in rheumatoid arthritis (RA).[6] The prior LC was a hybrid format and was well received. As qualitative information on the experiences of participants in virtual collaboratives is limited, we hoped to understand participant preferences and experiences in virtual collaboratives and capture participant satisfaction with this format across a range of practices and participants.

METHODS

Learning Collaborative

We conducted a virtual LC between October 2020 and April 2021 with 18 rheumatology practices from around the United States. The LC included 45 participants, some rheumatologists, nurse practitioners, physician's assistant, clinic staff, and others. The LC focused on implementing TTT, a widely recommended treatment paradigm in RA.[7] The collaborative spanned 7 months and featured a virtual kickoff meeting followed by 6-monthly webinars featuring speakers, data visualization, and discussions. Originally, the kickoff meeting was scheduled as an in-person event in Spring of 2020, but it was postponed because of the COVID-19 pandemic. The kickoff meeting was rescheduled to the fall and was transitioned to a fully virtual meeting using a video-conferencing service. The virtual kickoff meeting involved 5 to 6 hours with both lectures and time for questions. Participants were asked to perform some tasks during the LC, helping them to focus on plans for local interventions to improve TTT implementation. Sites received a small honorarium for their participation in the kickoff meetings and monthly webinars.

The monthly webinar meetings revisited some of the themes of the kickoff meeting. There were 2 webinar sections each month to give participants more flexibility. Speakers presented on “Plan-Do-Study-Act” (PDSA) planning, measuring disease activity in RA patients, TTT in virtual care, shared decision-making, and medications in TTT in RA. Participants were asked to submit monthly chart reviews and PDSAs, which detailed specific TTT improvement plans and goals from month to month. The chart reviews included data from 20 to 25 visits for RA over the prior month. Information was collected on the type of visit, disease activity measures, disease activity targets, as well as information about changes in treatment and evidence of shared decision-making (see Supplemental Fig. 1, http://links.lww.com/RHU/A461). Each LC featured a brief introduction, followed by discussion of PDSAs from sites. After PDSA discussion, our monthly speaker would give a brief 20-minute talk followed by discussion and questions. Each webinar was concluded by reviewing data from the monthly chart reviews submitted by each site.

We calculated adherence to TTT for each site using the monthly chart reviews. We defined TTT adherence based on whether disease activity measure and disease activity target were present in the note, whether a change was made in treatment if the patient was not at target, and whether shared decision-making played a role in the treatment change/ongoing disease activity goals. Chart reviews were completed via self-assessment or by another participant at the same site, and participants were able to select which charts they chose to review. Data from the chart reviews were shown and discussed. The collaborative meetings also were recorded so participants could watch asynchronously at a Web site that also contained pertinent collaborative resources.

Surveys

After the final monthly webinar, we distributed satisfaction surveys through an e-mail link to participants to assess their experience in the LC. Participants were given 2 weeks to complete the survey. We conducted 2 surveys: one focused on rheumatology practice characteristics, and the other on participant opinions regarding the LC.

Rheumatology Practice Survey

The rheumatology practice leads were surveyed regarding site characteristics and patient demographics. This survey consisted of 12 questions that included practice type (solo, group, group-multispecialty, etc) and practice setting (academic, private practice, etc). Other questions included number of participants at site, roles of participants at site, size of RA practice, and other questions about the nature of the site's participation in the collaborative (see Supplemental Fig. 2, http://links.lww.com/RHU/A462).

Participant Survey

The participant survey focused on the experience of individuals in the collaborative and featured 15 questions on overall satisfaction, perceived usefulness of different aspects of the collaborative, as well as participation and general experience during the LC. This survey was sent to all individuals who attended at least one kickoff meeting or webinar during the collaborative (see Supplemental Fig. 3, http://links.lww.com/RHU/A463).

Analysis

Results from the site and individual surveys were collected and described separately. Responses from the site surveys were collected to portray demographic information and site characteristics. The data from the individual surveys were pooled and described across all collaborative respondents. Survey results were used to describe the characteristics of the participating practices, whereas individual survey results were evaluated to assess participant experience and satisfaction qualitatively.

RESULTS

All 18 rheumatology practices responded to the practice survey, including 35 of 45 participants (78%). Characteristics of the rheumatology practices are shown in Table 1. Multiple practice types were included in the LC: 12 academic rheumatology practices, 2 academically affiliated practices, and 4 nonacademic practices. The number of RA patients estimated to be seen regularly at participating practices ranged from <300 to >1500. The number of LC participants per rheumatology practice ranged from 1 to 8.

TABLE 1

Site Characteristics Based on Site Surveys (n = 18)

	Category	n (%)
Practice type	Solo	1 (5.6%)
	Group rheumatology	9 (50%)
	Group multispecialty	6 (33.3%)
	Other	2 (11.1%)
Practice setting	Academic medical center	12 (67%)
	Private practice with academic affiliation	2 (11%)
	Private practice nonacademic affiliation	3 (17%)
	Community safety-net hospital	1 (5.6%)
Practice size (RA patients)	1–300 patients	5 (28%)
	301–600 patients	1 (5.6%)
	601–900 patients	3 (17%)
	901–1200 patients	4 (22%)
	1201–1500 patients	1 (5.6%)
	>1500 patients	2 (11%)
	Unknown	2 (11%)
Percentage of visits that were virtual at each site across collaborative	0%–5%	3 (17%)
	6%–10%	5 (28%)
	11%–30%	4 (22%)
	31%–60%	3 (17%)
	61%–80%	2 (11%)
	>80%	1 (5.6%)
No. individuals involved in learning collaborative at practice	1–2	9 (50%)
	3–5	8 (45%)
	6+	1 (5.6%)
Frequency of learning collaborative planning meetings and PDSA discussions	Never	2 (11%)
	Weekly	1 (5.6%)
	Every other week	4 (22%)
	Monthly	9 (50%)
	Other	2 (11%)
	1–2	8 (45%)
No. PDSAs submitted across 6 months of learning collaborative
	3–4	7 (39%)
	5–6	3 (17%)

Among the sites, 72% reported meeting every other week or monthly to plan small tests of change (PDSA cycles) during the LC; 66% of practices completed the monthly medical record reviews through self-assessment, whereas another 33% had another clinician or administrator review and submit data. As noted in Table 2, 63% of respondents indicated attendance at 5 or 6 of the monthly webinars; 97% of respondents indicated they were somewhat or very active in monthly PDSA planning. Approximately 77% of clinicians submitted chart reviews for 5 or 6 months of the LC.

TABLE 2

Role in and Satisfaction With Learning Collaborative Based on Individual Surveys (n = 35)

		Category	n (%)
Respondent’s characteristics and role in learning collaborative	Respondents role at site	Rheumatologist	24 (69%)
		Nurse practitioner	4 (11%)
		Physician assistant	1 (2.9%)
		Other	6 (17%)
	No. Monthly Webinars Attended	0	2 (5.7%)
		1–2	7 (20%)
		3–4	4 (11%)
		5–6	22 (63%)
	No. months with chart review submissions (clinicians only)	0–1	1 (2.9%)
		2–3	1 (2.9%)
		4–5	6 (17%)
		6	23 (66%)
		N/A (not clinician)	4 (11%)
	Level of participation in monthly PDSA planning	Very active	21 (60%)
		Somewhat active	13 (37%)
		Not active	1 (2.9%)
Respondent’s general satisfaction with learning collaborative	Overall satisfaction with collaborative	Very satisfied	26 (74%)
		Somewhat satisfied	7 (20%)
		Neutral	2 (5.7%)
		Somewhat dissatisfied/dissatisfied	0 (0%)
	How likely are you to recommend a similar collaborative to a colleague?	Very likely	23 (66%)
		Likely	10 (29%)
		Neutral	2 (5.7%)
		Not likely/very unlikely	0 (0%)

Table 2 indicates that 94% of respondents reported that they were either “somewhat” or “very satisfied” with the LC, and 94% said they would recommend a similar LC to a colleague. Regarding LC format, 37% of respondents indicated they would prefer a virtual LC, and 51% indicated they would prefer a hybrid model. In addition, 57% of respondents preferred the 6-month collaborative timeline, whereas 29% would have preferred a shorter collaborative and 14% a longer collaborative. In addition, 83% of respondents indicated they would prefer the same monthly webinar schedule with 17%, indicating they would prefer a less frequent meeting schedule.

Figure 1 illustrates participant satisfaction with several different aspects of the LC. Participants found the following very useful: intersite discussion, the collaborative faculty and speakers, monthly data submission, and feedback on the monthly data. Participants found certain aspects of the LC less useful, including the Web site or the meeting recordings. Respondents also noted high levels of interest (at or over 75%) in future LCs on a range of rheumatology-specific topics, including laboratory monitoring for drugs and diseases, reproductive health counseling, cardiovascular risk factor management, and improving management of glucocorticoid-induced osteoporosis.

FIGURE 1

Perceived usefulness of different pieces of the TRACTION learning collaborative (n = 35).

DISCUSSION

Our findings confirm that successful LCs can be implemented in a pure virtual model in 2020 to 2021, during the COVID-19 pandemic.[3-5] Our study extends these results with information concerning participant experience and satisfaction during these virtual collaboratives. Because virtual collaboratives are more accessible and less costly, there may be an interest in continuing the use of virtual LCs even after no longer required because of pandemic restrictions.

Our recent LC focused on implementation of TTT for RA among rheumatology practices in the United States. The LC produced robust improvement in adherence with TTT as noted in Figure 2. This LC included 45 participants from 18 rheumatology practices across the United States. Participants reported high levels of satisfaction across the LC. Specific feedback concerning different components of the LC may provide guidance in designing future LCs.

FIGURE 2

Trend in mean adherence with TTT over the 6 months of the learning collaborative. Error bars represent standard deviation (used with permission).

Participants indicated that the interpersonal aspects of the LC were very motivating and that it was an important advantage to have connections with a group, especially during the COVID-19 pandemic. Although a relatively large group, many different participants were able to voice their opinion, contribute to discussion, ask questions, and engage with monthly speakers across the 6 months of the LC. We found participants and speakers were able to discuss nuanced aspects of TTT and engage readily each month with new speaker topics and discussions. It is possible that quarantine and difficulties at work during the pandemic might have elevated satisfaction scores relative to what might be seen in pre– or post–COVID-19 times, but this is difficult to assess. We also received feedback that the LC time commitment was sometimes difficult to manage with COVID-19–related staffing changes and limited in-person team engagement at sites. Other comments included that the time commitment for the LC could be difficult to manage alongside other commitments, but having the option to participate in real-time was very helpful. These interpersonal aspects were highly valued during hybrid and in-person collaboratives and seem to be sustained during virtual LCs.[8]

Feedback from individuals on the format of the LC also was informative concerning designed future collaboratives. Although participants expressed high satisfaction with a virtual collaborative, only 37% indicated they would prefer another full virtual collaborative over an in-person or hybrid model. Given the state of the pandemic during the virtual LC, we were not able to incorporate any in-person components. In an ideal world, we feel that a hybrid model could help with participant engagement and might allow for more interpersonal interaction at the start of the LC. Participants in the LC generally preferred the monthly meeting schedule and the duration of the LC, but there was more variation in the preference for the duration of the LC. Depending on the amount of content, the level of engagement required from month to month, and participant preferences, future LCs could be easily redesigned to meet the goals of the collaborative and its participants.

Participants found resources such as the Web site and meeting recordings less useful than the meeting presentations, interaction with the faculty, or discussions with other participants. Accessing the Web site and/or meeting components involves an additional time commitment for the participants and played a more ancillary role in the LC curriculum. We also received feedback on the duration of the LC and the frequency of webinars during the collaborative. We did receive feedback on the structure of the collaborative, and 17% of the collaborative would have preferred less frequent meetings and around 30% would have liked the collaborative to be less than 6 months. Eighty-three percent of respondents preferred the monthly schedule, with close to 60% of respondents preferring a 6-month collaborative. We did not collect any information on participant preference for the duration of each webinar session. This can be easily adjusted to meeting collaborative needs because a 5-to-6-hour kickoff meeting and hour-long monthly webinars are not necessarily sustainable for other participants.

Our study included several limitations. The LC included a relatively small sample size of 18 practices from 10 different states. Many of these practices were academically affiliated; it is possible that nonacademic practices may have different attitudes regarding the format of LCs. We were also fortunate that many individuals had significant experience with virtual meetings before joining the LC. Although we received responses from all 18 sites, only 78% of participants responded to the individual survey. Nonrespondents may have different attitudes and preferences.

Although we cannot speak to the relative effectiveness of a pure virtual LC compared with a hybrid or in-person LC, participants enjoyed their experience and were very satisfied with the virtual format. Individuals from across the United States were able come together monthly in a low-cost, efficient, and accessible medium. We identified several possible topics for future learning collaboratives related to laboratory monitoring for drugs and diseases, reproductive health counseling, and improving management of glucocorticoid-induced osteoporosis. We received positive feedback that participants hope this can extend to more institutions and more interdisciplinary topics in the future. Although our LC focused on TTT in RA in a group of clinicians with a range of TTT experience, future collaboratives might spend more effort discussing nuanced cases. Additionally, LCs could cover a wider range of topics to vary content and keep participants engaged from month to month. Professional societies (eg, American College of Rheumatology or European Alliance of Associations for Rheumatology) could organize these at a national or international level, but these could also be adopted by hospital systems or multisite health care systems. The advantages of a virtual collaborative, including substantially lower costs and a possible larger number of participants from a broad geography, may outweigh some of the disadvantages noted previously. These virtual or hybrid LCs are relatively malleable and can shift in duration, frequency of meetings, and content to improve participant experience.

In summary, we found that participants in a virtual LC regarding outpatient rheumatology practice were highly satisfied with this format. The virtual format was found to achieve strong interpersonal connections across sites and with LC faculty. The virtual kickoff plus focused 1-hour monthly sessions reduces costs and travel commitment compared with an in-person or hybrid LC. In addition, we found that participants could attend different webinar sessions each month with relative ease and adjust their participation in the LC depending on their outside commitments. Speakers and faculty can contribute from around the world, and learning and collaboration opportunities can be facilitated between geographically disparate sites. The LC on TTT in RA is a good example of an effective virtual LC that achieved high levels of satisfaction and participation and may serve as a model for cross-site quality improvement interventions in health care.

A virtual LC in rheumatology is feasible, and participants reported being highly satisfied.

Participants gave specific feedback regarding the structure of the LC, allowing future LCs to be modified based on reports from participants.

Participants expressed strong interest in future virtual LCs and rated potential topics in rheumatology.