Results and Lessons Learned when Implementing Virtual Health Resource Centers to Increase Virtual Care Adoption During the COVID-19 Pandemic.

Implementation efforts to increase adoption of health technologies (e.g., telehealth, mobile health, electronic health records, patient portals) have commonly focused on increasing the adoption of specific health technologies in specific service lines. To facilitate adoption of multiple health technologies across a hospital setting, four Virtual Health Resource Centers (VHRCs) were established to provide clinical adoption support to healthcare staff and patients in four hospitals in a large healthcare system. This study spanned a 3-year period, with the first half including pre-implementation efforts, and the second half involved in implementation efforts. In order to compare sites to the national population, a binomial regression was used which allowed for adjustment of relevant covariates (e.g., differences in number of enrollees, level of complexity of facility). The pre-implementation phase and the initial year-and-a-half of the implementation phase resulted in an increase in internal facilitators' knowledge and skills of virtual care technologies, an increase in facilitator and site capacity, and high levels of adherence to implementation strategies were maintained across sites. Virtual care utilization increased across all sites and across the healthcare system during the implementation phase; however, a comparison to the increase in national level virtual care utilization metrics yielded no meaningful difference. While many implementation strategies aim to increase the adoption of a particular health technology product (e.g., a particular app or remote monitoring use case), the establishment of VHRCs may increase efficiencies in delivery of virtual care training and consultation to healthcare staff and patients, which may increase capacity and decrease barriers to adoption. However, due to the impact of the COVID-19 pandemic on the need for rapid adoption of technology and decreased in person care and services, it is not yet known the longer term impact that the establishment of VHRCs may have on the sustained adoption of health technologies. © This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2021.

Introduction

Virtual care health technologies such as synchronous and asynchronous telehealth, mobile applications, text-messaging applications, wearable/implantable biosensors, and patient-generated health data deployed in the setting of an integrated electronic health record (EHR) with and advanced analytic capabilities create the potential for huge benefits in the delivery and coordination of clinical care (Brown et al., 2014; Hermes et al., 2019; Hilty et al., 2019; Keyworth et al., 2018; Rathbone & Prescott, 2017). Barriers to implementation of virtual care include systemic barriers at the organizational level (governance models, lack of guidance regarding clinical integration), environmental (low levels of digital health literacy, lack of understanding of benefits, lack of training, learning new workflows), lack of infrastructure (poor wi-fi, lack of logistics for supplying equipment and access, poor integration of EHR systems), logistical barriers (scheduling, staffing), lack of resources (space, equipment, funding, time), challenges with collaboration (bureaucratic delays, communication), and policy issues (Armstrong et al., 2018; Hilty et al., 2018; Maheu et al., 2018; Muir et al., 2020; Torous et al., 2018; Zhou et al., 2019).

Implementation science, including the analysis of barriers and facilitators of adoption, provides a conceptual framework to systematically identify and evaluate strategies in order to identify the most effective and efficient implementation strategies (Kirchner et al., 2020; Morris et al., 2011; Nilsen, 2015). Facilitators of virtual care adoption include clinical staff (dedicated clinicians, experience with innovation, motivation, training), other staff (champions, coordinators, leadership support, external facilitators), and having an implementation strategy (tailored approach, pilot sites, standardized procedures) (Armstrong, 2019; Armstrong et al., 2020; Gould et al., 2019; Muir et al., 2020).

With an estimated 9.1 million Veteran enrollees, over 400,000 full time health care professionals and support staff at over 1200 health care facilities, supported by an annual budget of over $68 billion, the Veterans Health Administration (VHA) is the largest integrated health care system in the United States, and one of the largest in the world. While the complexities of delivering care across such a large and diverse system creates logistical challenges, the scope and integrated nature of the system also provides opportunities to create efficiencies of scale by standardizing operations, automating basic clinical activities, improving access to critical resources, and empowering both Veterans and healthcare staff. Health technologies enhance care coordination across VHA and promote Veterans’ ownership of and engagement with their own care. In accordance with VHA’s 2018–2024 strategic plan, the VHA aims to increase implementation and dissemination efforts to more fully integrate best practices in the use of virtual care technologies a to support the delivery of Veteran care.

While the utilization of virtual care health technologies across VHA has significantly increased over the past decade, evidence for their integration into the daily workflows of clinical teams across the spectrum of care, as well as their adoption by patients, leaves much opportunity for advancement. A limitation in the advancement of implementation studies in the area of health technologies is that many have focused on the increased adoption of an individual technology product or functionality (e.g.,, clinical video telehealth or a single mobile health app) into a specific clinical workflow (e.g., primary care), within a single clinic or hospital (Armstrong et al., 2020; Lindsay et al., 2015; Quanbeck et al., 2018; Yakovchenko et al., 2019). This approach, while simplifying analyses, creates challenges in the generalization of lessons learned to a system-level and focus on an individual technology may not the most efficient method of increasing overall virtual care utilization. This study aimed to leverage implementation science to increase the adoption of multiple health technologies through the establishment of Virtual Health Resource Centers (VHRCs) at four sites in a large healthcare system.

Methods

The aim of this mixed methods study was to test three multi-faceted approaches to increase health technology adoption, including a) Evaluating and building facilitator and site capacity at four sites; b) measurement of site adherence to 73 discrete implementation strategies; and c) establishment and evaluation of VHRCs and evaluation of their impact on health technology utilization metrics. Strategies were selected based on their perceived ability to decrease barriers and increase the adoption of health technologies in a large healthcare system. Implementation efforts focused on interdisciplinary VHA staff adoption of virtual care tools. The effectiveness-implementation hybrid design evaluated the use of facilitation specialists and compared within-site effects and between-sites effects on several usage metrics (Landes et al., 2020).

Participants

Participants included VHA staff on the implementation team directly involved in facilitating implementation of virtual care at study site locations. The interdisciplinary implementation team included twelve staff members in fiscal year (FY) 2019: two physicians, four nurses, one psychologist, one dietician, one human factors engineer, one program manager, and one healthcare specialist. The implementation team increased in FY20 to eighteen staff members, and included the additions of four healthcare specialists, one social worker, and one nurse.

Implementation team member roles were identified based on published recommendations (Ritchie et al., 2020; Rathbone & Prescott, 2017) and included external facilitators, experts in general implementation strategies and tools with expertise or credible knowledge about the clinical innovation and its evidence base; internal clinical leadership, individuals and groups of stakeholders who can impact the implementation of the clinical innovation; site internal facilitators, who were familiar with site-level organizational structures, procedures, and culture along with the national-level clinical processes within the healthcare network.

Procedures

The current study included activities completed over a 3-year period, from April 2018 to March 2021, and included a one-and-a-half-year pre-implementation phase (April 2018–September 2019) and the first year-and-a-half of the implementation phase (October 2019-March 2021). Implementation models used included “Promoting Action on Research Implementation in Health Services” (i-PARIHS) framework (Harvey & Kitson, 2016) and selection of seventy-three implementation strategies from the Expert Recommendations for Implementing Change (ERIC; Powell et al., 2015). The i-PARIHS framework was used as an overarching framework while discrete implementation strategies and activities were tracked throughout the pre-implementation and implementation phases using the compilation of implementation strategies from the Expert Recommendations for Implementing Change (ERIC) project (Powell et al., 2015).

The steps completed in the pre-implementation phase included the following: needs assessment and gaps analysis for program, funding to support program, program mission and goals (April-June 2018), program evaluation plan, site selection, site ‘memoranda of understanding’ signed by VA central office and site leadership (July–September 2018), staff hiring for sites (October 2018-March 2019), Pre-implementation efforts included a needs assessment and gap analysis of current virtual care implementation efforts across the VHA, accessing funding, establishment of program mission, goals and evaluation plan, site selection, and staff hiring.

Site identification was based on several factors including clinician, patient mix, and value; location, size, and type of VHA facility; existing relationships and availability of an on-site clinical leader; resource availability and collaborative opportunities near the site. Once selected, a site profile process was conducted including a needs and gaps analysis, and facility leadership approval. The needs assessment and gaps analysis were used to identify opportunities having the greatest potential to increase the adoption and utilization of virtual care technologies at the respective site. Preliminary personnel and site assessments were conducted and identified key baseline site performance metrics, patient population characteristics, and barriers and facilitators to adoption of virtual care technologies. Table 3 includes a comparison of key site operational metrics and highlights the diversity of the sites in terms of facility complexity, number of facilities included, number of patients, and rurality. Sites identified included medical centers and surrounding facilities in St Cloud, Minnesota (Site A), San Diego, California (Site B), New Orleans, Louisiana (Site C), and Tampa, Florida (Site D). Sites identified were diverse, providing optimum opportunity to evaluate current implementation strategies across a variety of settings.

Table 3

Changes in national healthcare system and study sites in operational metrics and health technology utilization metrics over initial year and a half of implementation phase

			National							Site A (St. Cloud, MN)						Site B (San Diego, CA)					Site C (New Orleans, LA)						Site D (Tampa, FL)
Operational metrics
Time point*			1*			2*		% difference		1		2		% difference		1		2	% difference		1		2		% difference		1		2		% difference
Facility complexity categorization			n/a			n/a		n/a		3		3		n/a		1A		1A	n/a		1B		1B		n/a		1A		1A		n/a
# of facilities included			1255			1299		n/a		4		4		n/a		7		7	n/a		9		9		n/a		15		15		n/a
# enrollees			9,100,000 +			9,093,691		2.2%		38,944		32,046		(− 17.7%)		86,140		72,931	(− 15.3%)		46,743		43,478		(− 7.0%)		100,329		95,836		(− 4.5%)
Operating beds			20,354			35,681		75.3%		388		389		0.3%		272		272	n/a		150		168		12.0%		499		499		n/a
Outpatient visits			81,305,962			45,965,521		(− 43.5%)		440,059		216,676		(− 50.8%)		993,457		522,244	(− 47.3%)		694,783		367,013		(− 94.7%)		1,437,900		863,301		(− 40.0%
Unique inpatient admissions			487,600			237,075		(− 51.4)		2491		576		(− 76.9%)		6642		3133	(− 52.8%)		2927		1783		(− 39.1%)		10,753		4368		(− 59.4%)
% Rural/highly rural			29.3%			32.5%		10.9%		75.3%		76.0%		0.9%		5.1%		4.7%	(− 7.8%)		19.4%		18.4%		(− 5.2%)		8.4%		9.5%		13.1%
Health technology utilization metrics
Synchronous video telehealth	Primary care providers using VA Video Connect	63.7%		93.6%	46.9%		87.8%		98.8%		12.5%		61.9%		96.8%		56.4%			53.7%		99.1%		84.5%		94.7%		99.1%		4.6%
	Mental health providers using VA Video Connect	62.4%		97.0%	55.4%		60.7%		91.6%		50.9%		58.4%		97.6%		67.1%			55.2%		93.2%		68.8%		90.1%		95.3%		5.8%
Online patient health portal	My HealtheVet (% premium accounts)	41.1%		46.2%	12.4%		47.1%		51.3%		8.9%		48.5%		55.2%		13.8%			45.1%		45.8%		1.6%		48.6%		53.2%		9.5%
Automated text message platform	Annie (# (%) unique veterans)	5247		30,342	478.3%		472		729		54.4%		86		589		584.9%			19		248		1205.3%		42		582		1285.7%
Mobile health apps	PTSD Coach (# lifetime downloads)	473,283		770,282	62.8%		No user authentication is required for these mobile health apps, so utilization metrics are not available by any specific site or individual
	Mindfulness Coach	376,242		836,386	122.3%
	CBT-i Coach	399,163		714,243	78.9%
	Insomnia Coach	0**		72,563	n/a
	COVID Coach	0**		199,310	n/a

Timepoint 1: end of pre-implementation phase = September 2019 (end of quarter 4, fiscal year (FY) 2019); timepoint 2: one-and-a-half years after implementation phase began = end of quarter 2, FY2021

Following leadership engagement and signing of memoranda of understanding, preliminary site needs and readiness assessments were conducted to identify baseline site performance metrics, patient population, barriers. and facilitators. A site capacity survey was administered to site internal facilitators measuring: motivations/interest in virtual care; current level of knowledge of mobile health core competencies; current level of knowledge and experience using VHA products; perception of site capacity; perception of site leadership support; perception of site provider support; perception of potential impact of virtual care at site; and barriers, limitations, need for support; and knowledge of VHA and military health systems and patient populations (Table 2).

Table 2

Change in facilitator and site capacity across first year-and-a-half of implementation phase

Facilitator or site focused	Knowledge area	Item	Sub-item	Across sites			Site A (St. Cloud, MN)
				Pre (N = 11)	Post (N = 8)	6-month Post (N = 7)	Pre (N = 2)	Post (N = 2)	6-month Post (N = 2)
Questions to facilitator regarding their own knowledge and readiness	Knowledge base of target audience (% rated high or very high level of knowledge)	Rate your current level of knowledge on VA health care: (response options: very high, high, moderate, low, very low)	Veteran Health Needs	72.7%	87.5%	100.0%	50.0%	50.0%	100.0%
			VHA System	63.6%	87.5%	100.0%	50.0%	100.0%	100.0%
	Knowledge base of virtual care core competencies (% rated high or very high)	Rate how strongly you agree/disagree with the following statements regarding adoption of virtual care tools and program in clinical practice at your facility (response options: very high, high, moderate, low, very low)	Clinical Integration of Virtual Care	72.7%	100.0%	100.0%	50.0%	100.0%	100.0%
			Evidence Base for Virtual Care	63.6%	100.0%	100.0%	50.0%	100.0%	100.0%
			Security and Privacy of Virtual Care	63.6%	100.0%	100.0%	50.0%	100.0%	100.0%
			Ethical Issues with Virtual Care	72.7%	75.0%	100.0%	50.0%	50.0%	100.0%
			Cultural Considerations with Virtual Care	63.6%	75.0%	100.0%	50.0%	50.0%	100.0%
	Readiness to use virtual care (% in action or maintenance stage)	Please select the item below that best describes where you are in terms of integration of virtual health technologies into clinical care	I do not intend on integrating virtual health technologies into my clinical practice (precontemplation)	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
			I am intending to begin integration virtual health technologies into my clinical practice in the next 6 months (contemplation)	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
			I am planning on integrating virtual health technologies into my clinical practice next month (preparation)	9.1%	0.0%	0.0%	0.0%	0.0%	0.0%
			I started integrating virtual health technologies into my clinical practice within the past 6 months (action)	18.2%	37.5%	0.0%	50.0%	0.0%	0.0%
			I started integrating virtual health technologies into my clinical practice more 6 months ago (sustainment)	72.7%	62.5%	100.0%	50.0%	100.0%	100.0%
	Product knowledge (% that had either used the product, prescribed, or recommended the product in clinical care, or trained others on how to use the product)	Average rate of product knowledge across suite of 56 virtual care products and programs (% that had either used the product, prescribed, or recommended the product in clinical care, or trained others on how to use the product)		21.8%	34.7%	42.4%	21.4%	34.1%	41.5%
	Product knowledge on key products (% that had either used the product, prescribed, or recommended the product in clinical care, or trained others on how to use the product)	Annie (VHA's automated health text messaging web platform)		36.7%	62.5%	85.7%	50.0%	100.0%	50.0%
		Mindfulness Coach (iOS and Android app)		36.7%	50.0%	57.1%	0.0%	50.0%	50.0%
		My HealtheVet (VHA's patient web portal)		63.6%	75.0%	100.0%	50.0%	50.0%	100.0%
		PTSD Coach (iOS and Android app)		36.7%	50.0%	71.4%	50.0%	50.0%	50.0%
		Rx Refill (iOS and Android app)		27.3%	37.5%	42.9%	0.0%	0.0%	0.0%
		VA Video Connect (live video telehealth)		72.7%	87.5%	100.0%	50.0%	50.0%	100.0%
Questions to facilitator regarding perceptions of site capacity	Quality of facility climate and support regarding virtual care implementation (% agree or strongly agree)	Rate how strongly you agree/disagree with the following statements regarding adoption of virtual care adoption in clinical practice at your facility (response options: strongly agree, agree, neither agree nor disagree, disagree, strongly disagree)	Virtual care can be integrated into care at your site	81.8%	100.0%	100.0%	50.0%	100.0%	100.0%
			Virtual care is supported by scientific evidence	72.7%	100.0%	100.0%	50.0%	100.0%	100.0%
			Virtual care implementation is supported by your facility leadership	72.7%	75.0%	85.7%	50.0%	50.0%	100.0%
	Site barriers to implementation (% rated barrier as moderate, high, or very high)	Time constraints/too busy		100.0%	87.5%	85.7%	100.0%	100.0%	100.0%
		Don’t know virtual care tools enough		81.8%	62.5%	57.1%	100.0%	50.0%	50.0%
		Need more training		72.7%	62.5%	57.1%	50.0%	50.0%	50.0%
		Unclear policies regarding use		72.7%	50.0%	42.9%	50.0%	50.0%	0.0%
		Concerns about security and privacy		72.7%	62.5%	57.1%	50.0%	50.0%	50.0%
		Need more evidence that proves effectiveness of virtual care		63.6%	37.5%	28.6%	50.0%	50.0%	50.0%
		Don’t know how to choose which virtual care tool to use		63.6%	50.0%	42.9%	50.0%	50.0%	50.0%
		Virtual care will interfere with the patient/client relationship		45.5%	37.5%	28.6%	50.0%	50.0%	0.0%
		No compensation for use of virtual care with patients		36.7%	37.5%	42.9%	50.0%	50.0%	50.0%
		Lack of connectivity in the clinic		36.7%	25.0%	28.6%	50.0%	50.0%	50.0%
		Employer restrictions/lack of employer support		27.3%	25.0%	14.3%	50.0%	0.0%	0.0%
		Cost or resources associated with use		18.2%	25.0%	14.3%	50.0%	50.0%	50.0%
		Don’t think that virtual care will work		18.2%	12.5%	14.3%	0.0%	0.0%	0.0%

Capacity building assessments used were adapted from the following standardized surveys: Organizational Readiness to Change Assessment Tool (ORCA; Helfrich et al., 2009), Checklist to Assess Organizational Readiness for Evidence-Informed Practice (CARI; Barwick, 2011), readiness for change (Prochaska et al., 1994, p. 68). These assessments were modified in alignment with recommendations provided by the i-PARIHS framework. Modifications consisted of additional items specific to the aim of this study. The additional items measured: motivations/interest in virtual care, current level of knowledge of mobile health core competencies, current level of knowledge of VHA, military health system and patient health needs, current level of knowledge and experience using VHA products, quality of organizational climate and support regarding virtual care, perception of potential impact of connected health at site, barriers, limitations, need for support. Items are included in Table 2.

Between July and September 2019, the lead external facilitator held one-on-one meetings with each site’s internal facilitators in order to develop individual capacity building plans for to inform the development of the broader capacity building strategy across sites. Processes included the administration of a preliminary assessment to each site internal facilitator, and results were used to assess knowledge and skill gaps, site level barriers to implementation, and quality of facility climate and support regarding virtual care implementation. During this component of process, implementation and dissemination plans were developed and deployed to facilitate the implementation process. Individual implementation plan meetings with sites included the following: (1) explanation of site capacity building process, (2) discussion of identified needs and possible solutions, (3) discussion regarding identified strengths, (4) discussion regarding identified opportunities, (5) discussion regarding bridging identified gaps. Implementation plans for each site included: (1) training preference/goals, (2) topics/prioritized (individual/group/resources), (3) types of promotional/training resources available and ideas for additional ones develop, (4) agreement on goals and development of site road map to achieve goals.

After individual and site capacity building plans were created based on the preliminary results, individual and site training plans were developed and delivered to increase all internal facilitators’ base level of knowledge and skills across areas measured. Results of the initial needs assessment, and gap analysis, as well as the results of site assessment and capacity building results, implementation strategies were selected across all sites and an implementation plan developed to provide an overall roadmap for the development of VHRCs at each site, which were launched between March 2020 and July 2020 and began the implementation phase at each site. Post-site capacity building assessments were delivered at 3-months after the preliminary assessment and provided information regarding progress toward knowledge and skills gained, as well as further training needed. Continued training and site capacity building efforts were included based on internal facilitators’ responses. A 6-month post assessment was also delivered in order to assess progress toward goals.

Implementation strategies were tracked across all sites, and key implementation strategies (training, marketing, encounters (e.g., consultations) at the VHRC) were further tracked. Some primary activities included delivery and evaluation of training and marketing events to increase the awareness of virtual care tools and programs. Training and marketing events were also assessed based on the type of event, with lower impact events aimed at increasing awareness evaluated by reach (number of attendees and number of events) and qualitative feedback provided by attendees, and higher impact delivery methods evaluated with course evaluations and follow-up assessments (when possible) included to measure awareness, knowledge and skills. In developing the training and marketing plan, the implementation team remained cognizant of VHA organizational structures, ensured trainings were tailored to local sites, and used a phased approach to allow for the refinement of procedures on systemic and site levels.

Evaluation

Evaluation was guided by the mixed-methods RE-AIM Qualitative Evaluation for Systematic Translation (RE-AIM QuEST) framework. Measure impact on virtual care utilization was calculated as percent increase from baseline (prior to the implementation phase in end of fourth quarter in FY19) to the end of the first year of implementation phase (end of fourth quarter in FY20). Rates of key telehealth metrics across five key domains were assessed (telehealth use, patients using telemental health, clinical video telehealth to offsite, primary care providers using VA Video Connect, and Mental health providers using VA Video Connect). The rate of utilization was also analyzed for products (VA Video Connect, My HealthVet, Annie, PTSD Coach, Mindfulness Coach, CBT-i Coach, and Insomnia Coach, and COVID Coach). VA Video Connect, My HealthVet, and Annie require secure login credentials to use, so site metrics were able to be calculated. For PTSD Coach, Mindfulness Coach, CBT-i Coach, and Insomnia Coach, and COVID Coach (and other VA mobile apps that do not require login credentials) site utilization metrics are not available.

The scope of the virtual care technologies included the suite of VHA mobile health applications (e.g., PTSD Coach, Mindfulness Coach, Annie, My VA Images, Rx Refill), the suite of products and programs offered in VHA telehealth (e.g., home telehealth, remote patient monitoring, VA Video Connect), and the VHA’s web-based patient portal (My HealthVet), which includes secure (encrypted) messaging capability between VHA staff and Veterans, access to healthcare records, prescription refills, and VHA medical records.

Results

Discrete implementation strategies utilized across sites are detailed in Table 1, and show activities completed and resources developed for each discrete strategy. Results show percentages of implementation strategies used across the national program team and across sites were consistent. The national program team used 87.7% (64/73) of available strategies, Site A and D used 79.5% (58/73) of strategies, and Sites B and C used 80.8% (59/73) of strategies,. The only strategy that was not used across sites was strategy 72 (site visits). In-person visits were completed for sites B and C, however, due to restrictions related to the COVID-19 pandemic in-person site visits were put on hold for sites A and D. Five of the seventy-three strategies were only used at the national program level to support efforts at sites and across the VHA enterprise, including the following strategies: 1) access new funding, 2) alter incentive/allowance structures, 3) alter patient/consumer fees, 22) create or change credentialling and/or licensure standards, and 34) fund and contract for the clinical innovation.

Table 1

Discrete implementation strategies used, activities completed, and resources developed

Strategy #	Strategy	Activities completed
1	Access new funding	Funding provided through VA’s Central Office of Connected Care budget
2	Alter incentive/allowance structures	Primary care leadership and program performance-based performance goals
3	Alter patient/consumer fees	Co-pay was eliminated for patients using VA Video Connect for a telehealth visit as a national initiative
4*	Assess for readiness and identify barriers and facilitators	Site readiness assessed at all barriers/facilitators identified; site implementation plans developed
5*	Audit and provide feedback	Created clinical performance and compliance data reports and deliver leadership briefings
6*	Build a coalition	Learning collaboratives established; Leadership meetings held; Collaboration with implementation stakeholders; Identified champions
7*	Capture and share local knowledge	Evaluated implementation strategies, document results in implementation reports
8*	Centralize technical assistance	Establishment of Virtual Health Resource Centers (VHRCs) providing one-stop support for virtual care technologies; establishment of toll-free number (1–844-813–4361) for centralized access
9	Change accreditation or membership requirements	n/a
10	Change liability laws	n/a
11	Change physical structure and equipment	Acquired space in facility and equipment for staff for VHRC
12	Change record systems	Created and implemented EHR templates and consults for facilities
13*	Change service sites	Change delivery of clinical services to be provided ‘anywhere to anywhere’
14*	Conduct cyclical small tests of change	Implementation pilots incorporating virtual tools and services into a specific clinic
15*	Conduct educational meetings	Weekly education strategy and planning meetings; community of practice
16*	Conduct educational outreach visits	Developed and distributed implementation toolkits training and promotional materials to sites
17*	Conduct local consensus discussions	Established interdisciplinary learning collaboratives; Established steering committee
18*	Conduct local needs assessment	Site level needs assessments and gap analysis
19*	Conduct ongoing training	Delivery of training and marketing events
20*	Create a learning collaborative	Established earning collaboratives
21	Create new clinical teams	Evaluated existing clinical workflows and guided improvements to increase efficiency and effectiveness in using health technologies; team-based approach
22	Create or change credentialing and/or licensure standards	VA expanded access to care by allowing healthcare providers to provide care through telehealth across state lines
23*	Develop a formal implementation blueprint	Developed implementation plans based on recommendations in VA Facilitation and Implementation Guide (Ritchie et al., 2017)
24*	Develop academic partnerships	Partnered with local universities to deliver marketing and training opportunities; Mentorship of interns
25*	Develop an implementation glossary	Developed at national level, deployed at each site
26*	Develop and implement tools for quality monitoring	Created implementation and quality monitoring tools; developed at national level, deployed at each site
27*	Develop and organize quality monitoring systems	Developed at national level, deployed at each site; quarterly reports regarding progress toward goals
28	Develop disincentives	n/a
29*	Develop educational materials	Targeted training to increase awareness and knowledge of health technologies
30*	Develop resource sharing agreements	Centers of Excellence were established in partnership with facilities to increase access to resources related to virtual care implementation
31*	Distribute educational materials	Targeted training to increase product knowledge (Annie, My VA Images, Patient Viewer, Mental Health Checkup, VA Video Connect, Self-Health VA Apps (PTSD Coach, Mindfulness Coach, etc.); weekly group meetings; weekly individual and site meetings
32*	Facilitate relay of clinical data to providers	Established learning collaboratives; created leadership reports
33*	Facilitation	Weekly national and site meetings
34	Fund and contract for the clinical innovation	All products implemented were all developed and funded by VHA
35*	Identify and prepare champions	Identified and trained local clinical champions; trained internal facilitators on knowledge and skills in facilitation; Recruited members to learning collaboratives; Provided ongoing training and collaboration with these individuals; Workflow analysis
36*	Identify early adopters	Identified early adopters at local sites and conducted field tests to understand their experience with the health technology
37*	Increase demand	Provided regularly scheduled presentations to service leadership and staff; Marketing and education provided to Veterans and VHA staff
38*	Inform local opinion leaders	Identified and collaborated with local opinion leaders; provided quarterly reports on progress toward goals
39*	Intervene with patients/consumers to enhance uptake and adherence	Established ongoing information/education sessions; expanded expand virtual outreach options; developed ongoing training and marketing efforts
40*	Involve executive boards	Reports to site executive leadership; Provided leadership briefings
41*	Involve patients/consumers and family members	Outreach to Veterans to recruit peer-trainers for connected devices pilot; Established Virtual Health Resource Centers providing services to Veterans and family members; Field testing with Veterans
42	Make billing easier	n/a
43*	Make training dynamic	Developed training materials based on Adult Learning Theory with the aim to increase engagement with learners; Evaluate training and marketing materials and delivery for satisfaction
44*	Mandate change	Communicated to stakeholders existing mandates and alignment of strategic goals: VHA strategic goals include meeting needs of Veterans, enhancing Veteran experience, modernizing systems, and improving patient experience; Performance goals related to health technology use
45*	Model and simulate change	Facilitation training specific to integration of health technologies in clinical care
46*	Obtain and use patients/consumers and family feedback	Established VHRCs to meet the need of VHA staff and Veterans in integration of health technologies providing individualized consultation services; data reports on results
47*	Obtain formal commitments	Signed Memoranda of Understanding (MOU) at each site
48*	Organize clinician implementation team meetings	Implementation project plans and reports that include results, lessons learned, improved clinical workflows, and road maps
49	Place innovation on fee for service lists/formularies	n/a
50*	Prepare patients/consumers to be active participants	Established VHRCs to meet the need of VHA staff and Veterans in integration of health technologies providing individualized consultation services
51*	Promote adaptability	Assessed needs for service lines regarding barriers for use of health technologies, and implementation plans, to include marketing and training, specific to the site and service line needs
52*	Promote network weaving	Identified staff willing to collaborate to promote utilization of virtual health; Established national Connected Care Community of Practice engaging VHA staff
53*	Provide clinical supervision	Leveraged train the trainer model to provide mentorship and guidance to champions
54*	Provide local technical assistance	Developed VHRCs providing technical assistance to Veterans and VHA staff; Internal facilitators partnered with local personnel to increase reach
55*	Provide ongoing consultation	Developed VHRCs providing consultation to Veterans and VHA staff
56*	Purposely reexamine the implementation	Provided outcome monitoring (aka control plan) in each implementation plan; post training follow-up completed to encourage adoption of virtual tools and detect barriers to utilization
57*	Recruit, designate, and train for leadership	Established process to educate and train local champions
58*	Remind clinicians	Assessed and modified clinical workflows to increase efficiency and effectiveness
59	Revise professional roles	n/a
60*	Shadow other experts	Identified subject matter experts on various health technologies and effective implementation strategies for team members to shadow and improve knowledge and skills
61*	Stage implementation scale up	Completed eight implementation studies to assess implementation strategies for combinations of health technologies; Established VHRC Implementation Consult Service to provide guidance and resources to additional VHA sites wanting to build VHRC
62	Start a dissemination organization	n/a
63*	Tailor strategies	Met with service lines and multiple disciplines to identify barriers to implementation; Incorporated feedback into implementation planning; Completed site specific assessment
64*	Use advisory boards and workgroups	Connected Care Steering Committee consisting of all service established to discuss implementation strategies and expansion
65*	Use an implementation advisor	Implementation team completed formal training in facilitation and implementation; Implementation advisor review and guidance regarding implementation plans
66	Use capitated payments	n/a
67*	Use data experts	Applied data analytics and informatics project planning and monitoring of implementation efforts; Worked with local group practice managers to develop local data reports based on electronic health records entered for virtual programs
68*	Use data warehousing techniques	Utilized healthcare system data warehouse to create reports to track site progress toward goals
69*	Use mass media	Collaborated with Public Affairs Office (national and at facilities) to promote virtual programs across platforms (e.g., GovDelivery email, social media, blog posts, videos, radio, etc.)
70	Use other payment schemes	n/a
71*	Use train-the-trainer strategies	Staff education and training using a train-the-trainer model
72	Site visits	In-person site visits were completed for two sites. Due to COVID-19 pandemic, in-person site visits were put on hold
73*	Work with educational institutions	Partnered with academic institutions and provide mentorship for three graduate student interns

*Strategies that were used at the national program level and at all sites. (Compilation of implementation strategies from the Expert Recommendations for Implementing Change (ERIC); Powell et al., 2015)

Eight of the seventy-three strategies tracked were not used by any site. These included strategies: 9) change accreditations or membership requirements, 10) change liability laws, 28) develop disincentives, 42) make billing easier, 49) place innovation on fee for service lists/formularies, 62) start a dissemination organization, 66) use capitated payments, and 70) use other payment schemes. These were identified in the pre-implementation phase strategies that either were irrelevant in context of the VHA (e.g., those involving change in billing or payment structure), those that were not necessary to create as they are already an existing resource in the VHA (e.g., start a dissemination organization), and those that were not identified as a priority during the initial year of implementation (e.g., work with educational institutions), but may be integrated as the program matures in subsequent implementation years. Training, marketing and consultations (e.g., encounters delivered through the VHRCs) delivered as specific implementation strategies used and involved internal and external facilitators. Activities completed and resources developed for each strategy used is included in Table 1.

Eleven internal facilitators across sites responded to a preliminary site capacity assessment and met with the external facilitator to build individual and site capacity building implementation plan. A total of eight internal facilitators participated in the post-assessment, which was assessed approximately three months after the preliminary assessment (M = 86.4 days, SD = 7.8). A total of seven internal facilitators responded to the assessment delivered six months (M = 187 days, SD = 12.7) after the post-assessment. Results are included in Table 2 and show that across all sites there was an increase over time on the knowledge base on the target audience, knowledge base of virtual care core competencies, quality of facility climate and support regarding virtual care implementation, readiness to use virtual care tools, and product knowledge.

When comparisons are made across sites, Sites A and B had lower baseline knowledge (based on preliminary assessment) in the areas of knowledge of target audience, and a lower level of baseline readiness to use virtual care compared to Sites C and D. Internal facilitators’ motivation and interest in virtual care was high across sites, but level of knowledge in the following areas was inconsistent across sites: knowledge of the VHA healthcare system and Veteran healthcare needs, knowledge of core competencies in the integration of virtual care, and knowledge of individual virtual care products, knowledge of implementation science and facilitation processes. However, results show that knowledge in these areas increased across all sites at the 3 and 6-month intervals. Additionally, results of the preliminary assessment indicated variation in the quality of organization climate and support across sites, and in the perception of potential impact of virtual care at site, barriers, limitations, and need for support.

Of the knowledge areas assessed specific product knowledge was the weakest area across sites at the preliminary assessment time period (ranged from 21.8 to 40.3% respondents having either used the product, prescribed or recommended the product in clinical care, or trained others on how to use the product). Site capacity building efforts aimed to increase product knowledge across a suite of fifty-six virtual care products commonly used in the VHA. Results showed that training efforts served to nearly double (40.3–43.8%) product knowledge across at the 3 and 6-month time periods.

During the implementation phase a total of 442 education/training events were delivered to a total of 21,653 attendees. This included 17,203 attendees across 307 webinars, 1598 attendees across 82 presentations, 1241 attendees across 10 communities of practice, 1143 attendees across 16 train-the-trainer events, and 468 attendees across 27 office hours. During the same time a total of 51 marketing/promotional events were delivered to a total of 3538 attendees. This included 1498 attendees across 8 resource fairs and 2040 attendees across 43 outreach events. Training effectiveness and satisfaction was measured for several training efforts. While marketing and promotional events (e.g.,, outreach events, resource fairs, conference booths) were not evaluated beyond number of events and attendees, education and training events were evaluated. A Connected Care Community of Practice (CoP) series (one-hour webinars aimed at increasing knowledge and awareness of virtual care technologies)) training VHA staff on the use of virtual care tools was delivered across ten events to a total of 793 attendees. Results showed that 98.6% of CoP attendees reported that they were “satisfied or very satisfied with the training”, 98.9% of attendees responding that they agreed or strongly agreed that they would “…use the information and skills learned in my work,” and 97.9% would “…share information learned with others.” A series of fourteen training sessions (live webinar, one-hour each) “VA Connected Care for Telehealth Teams” was delivered to a total of 1475 telehealth leads and facilitators across the VHA. The majority (97.2%) of attendees agreed or strongly agreed that they “learned new knowledge and/or skills from this learning activity”, 98.1% of attendees agreed or strongly agreed that “content of this training was relevant to the needs of my healthcare team”, and 98.6% agreed or strongly agreed that they would ‘ be able to apply the knowledge and/or skills learned from this activity to improve my performance as a member of my healthcare team.’ A series of fourteen training sessions were developed and delivered to 2768 clinicians across twenty training events aimed at increasing knowledge and skills on the use of the VHA’s automated text-message platform (Annie). Results showed an increase in use of Annie from 6% before training to 63% 1-month after training; increased knowledge about Annie (17–22% before training, 41–56% after training, and 63–89% 1-month after training); increased satisfaction with Annie (average of 40% satisfied with Annie before training, 75% after training, and an average of 83% 1-month after training).

In addition to training and marketing efforts, a total of 9182 total encounters (e.g., individual consultations with Veterans, family/caregivers, or VHA staff on the use of virtual care) were provided services. The breakdown of service recipients through the VHRCs included 92.8% Veterans, 7.1% VHA staff, 0.1% family members or caregivers of Veterans. Encounter methods included 47.5% via telephone, 13.8% via in-person meeting, 37.2% via live video, and 1.5% listed as ‘other’. Services provided through the VHRCs included those seeking support on a single product or service, or a combination of products and services. The results of the VHRCs encounter analysis showed 48.9% encounters in support of My HealthVet, 26.2% in support of telehealth, 24.9% encounters in support of mobile health apps or web-based tools or in support of devices (smartphones, tablets, activity trackers, glucometers, pulse oximeters, etc.).

Changes in virtual care utilization across first year-and-a-half of implementation phase are detailed in Table 3. National and site utilization metrics were compared to determine changes over the implementation period. Results were inconclusive with none of Site A’s results meeting or exceeding national levels, all health technology utilization metrics exceeded at Site B and some at Site C, and Site D exceeding the national level of increase in only one area. A binomial regression was used and allowed for adjustment of relevant covariates (e.g., differences in number of enrollees, level of complexity of facility), and showed that the comparison of the health technology utilization metrics at the national level, compared to sites with a VHRC did not provide statistically significant evidence of meaningful difference (95% CI, p-value = 0.23). Operational metrics, which are typically relatively stable over time, decreased during the implementation phase, with a decrease om the number of enrollees decreasing of 10.2% across all study sites, a decrease in outpatient visits by 44.8%, a decrease in inpatient admissions by 56.8%, and a slight increase in the percentage of patients living in rural or highly rural locations.

Discussion

The current study described the year-and-a-half pre-implementation phase, and the first year-and-a-half of the implementation phase of a study aimed at increasing virtual care adoption at four sites across a large healthcare system by implementing VHRCs. Discrete implementation strategies used across sites were tracked, key areas of knowledge and readiness were assessed, and site-specific implementation plans were developed. Results demonstrated that facilitator knowledge and readiness across key domains increased, perceived barriers to the integration of health technology across sites decreased, and that discrete implementation strategies can be applied with similar adherence rates across varied sites. Key areas of knowledge and readiness were assessed, and a site-specific implementation plan was effectively delivered. Results demonstrated increased knowledge and readiness of facilitators, improved quality of climate and support at study sites, and decreased site barriers. This strategy served as a basic guide and helped to streamline action at individual sites during system wide implementation, and increased the efficiency of facilitators to expand the reach of resource delivered at sites. Thus, systematically targeting key implementation strategies with the aim of increasing site capacity was an effective process. However, the impact of the COVID-19 pandemic on the launch of the in-person VHRCs may have affected the analysis of this, and other factors may have been at hand. While the need to pivot to virtual care modalities during COVID-19 drastically increased the need for virtual care (Connolly et al., 2021; Ferguson et al., 2021), it required a shift away from in person to the delivery of services to virtual platforms preventing many in-person interactions (Veterans Health Administration, 2021; U.S. Department of Veterans Affairs, 2021).

While increased engagement with virtual care increased utilization metrics as well healthcare staff and patient familiarity with the use of these tools, the increase in use at the pace we experienced could never have anticipated. Additionally, due to social distancing protocols and stay at home orders (Center for Disease Control, 2020), the vision of a VHRC as an ‘on site, in-person’ resource located within the hospital was not possible. Operational metrics nationally, and across all sites had decreases in number of enrollees, number of outpatient visits, and number of inpatient admissions. With increased demand for virtual care our team shifted quickly to meet the demand to provide training on virtual care tools to patients and healthcare staff. This, at least anecdotally, provided support for the on-site telehealth and My HealthVet teams to better meet training needs. While health technology utilization increased across sites, sites with increases that outpaced the national increases were minimal. While efforts were made to demonstrate that implementation strategies that have shown effectiveness at a smaller scale (single technology, limited target audience) can be generalized to better meet the needs staff and patients, given the small sample size and lack of control sites beyond comparison to the national, we were unable to make this conclusion at this time. Because the current study describes the results of the initial year-and-a-half of results during the implementation phase of this study, it may be that over time differences can be detected across sites to better determine specific strategies that increase virtual care adoption.

Further, it may have been due to the necessity for virtual care as driven by the COVID-19 pandemic that likely drove healthcare staff and patients to adopt virtual health care tools at a higher rate than pre-pandemic. However, the pandemic created a huge need for virtual care delivery, which drove adoption of many of these tools, which also made the evaluation of impact of efforts to increase adoption of virtual care during this time challenging. Another impact of COVID-19 was the surge in needs for acute care distracted that drained resources from organizations trying to implement new practices and technologies. In addition to the impact of COVID-19 on the ability to measure the impact of the establishment of VHRCs, it is also important to note is that virtual care across the VHA includes many teams involved with training and marketing activities. While increases in utilization at the study sites and comparison with sites not included in the study provide an indication of the impact of the implementation efforts described, increases in utilization cannot solely be credited to these efforts alone.

Future efforts include continuing to build capacity at established VHRCs and developing a VHRC implementation consult service to guide other facilities interesting in building a VHRC. While more research is needed on the value of site, regional, or nationwide VHRCs there has been interest in several additional VHA sites in developing a VHRC at their site, demonstrating an interest in the field of replication of this model. Future efforts will aim to improve evaluation and reporting processes to better assess the impact of VHRCs. For example, a lesson learned during the implementation phase was that there was a need to collect customer feedback regarding satisfaction with services provided through the VHRC. As a result, items were developed and a process was implemented in January 2021 which will provide an additional source of feedback to support continuous improvement process. Additionally, centrally located online platforms for entering data on VHRC activities across sites have been established, as well as data dashboards allowing VHRC sites to view data reports showing progress toward established target goals.

Having an impact on an organizational or program level requires investment in training that influences the culture to realize the benefits of these health technologies, and an overarching plan of action that aligns technology and utilization strategies across the organization/program. While implementation studies aimed at evaluating the integration of a single innovation into a single clinic or hospital make it easier to evaluate impact, the results create challenges making it difficult to generalize to other innovations. In addition, the impact of single site, single product implementation does not reflect the reality of health technology integration across a healthcare system, which is varied, complex, and needs to allow flexibility for staff and patients to choose which health technology tool or tools that best promote patient-centric care.