A Longitudinal Mixed-Methods Examination of Positive Health Check: Implementation Results From a Type 1 Effectiveness-Implementation Hybrid Trial.

BACKGROUND: Positive Health Check is an evidence-based video doctor intervention developed for improving the medication adherence, retention in care, and viral load suppression of people with HIV receiving clinical care.
SETTING: Four HIV primary care clinics within the United States.
METHODS: As part of a type 1 hybrid trial, a mixed-methods approach was used to longitudinally assess the following 3 key implementation constructs over a 23-month period: innovation-values fit (ie, the extent to which staff perceive innovation use will foster the fulfillment of their values), organizational readiness for change (ie, the extent to which organizational members are psychologically and behaviorally prepared to implement organizational change), and implementation climate (ie, the extent to which implementation is expected, supported, and rewarded). Quantitative mixed-effects regression analyses were conducted to assess changes over time in these constructs. Qualitative analyses were integrated to help provide validation and understanding.
RESULTS: Innovation-values fit and organizational readiness for change were found to be high and relatively stable. However, significant curvilinear change over time was found for implementation climate. Based on the qualitative data, implementation climate declined toward the end of implementation because of decreased engagement from clinic champions and differences in priorities between research and clinic staff.
CONCLUSIONS: The Positive Health Check intervention was found to fit within HIV primary care service settings, but there were some logistical challenges that needed to be addressed. Additionally, even within the context of an effectiveness trial, significant and nonlinear change in implementation climate should be expected over time.

INTRODUCTION

After the results of several studies supporting treatment as prevention,[1-4] HIV treatment has become a key HIV prevention priority. Nonetheless, the number of new HIV diagnoses in the United States has remained relatively stable at about 38,000 new diagnoses annually. Contributing to the number of new HIV infections each year is the number of people with HIV (PWH) who are not virally suppressed. Viral suppression rates have improved over time, yet about 420,000 PWH aged 13 years or older are not virally suppressed.[5] Suboptimal adherence to the prescribed HIV treatment regimen is a key factor in why viral suppression rates are not higher.[6] Research has focused on developing effective interventions to improve HIV medication adherence, including interventions that are computer based.[7,8] Web-based HIV adherence promotion interventions may be more cost-effective and sustainable than person-delivered interventions.

In 2015, Claborn et al conducted a systematic review of computer-based HIV adherence promotion interventions.[9] Of the 10 studies identified, only 3 were fully powered randomized controlled trials (RCTs).[10-12] Concluding there was not yet sufficient evidence to support the efficacy of computer-delivered HIV adherence interventions, the authors noted more RCTs were needed and “Future studies should be designed with regard to evaluation of implementation and sustainability of the intervention within the clinic setting.” In 2018, Kemp and Velloza reviewed research conducted since 2015 that focused on implementing electronic health interventions designed to improve outcomes along the HIV care continuum.[13] Of the 17 studies identified, more than half (n = 9; 53%) included a focus on improving HIV medication adherence,[14-22] only 2 were RCTs, and the most of the studies (n = 15; 88%) focused on acceptability and/or feasibility.[14,23-26] Consequently, in addition to recommending future research to examine other implementation outcomes, Kemp and Velloza highlighted the need for more research to help identify predictors of implementation.

The utility of implementation research for maximizing the prevention and treatment of HIV has been highlighted several times during the past decade.[27-34] Indeed, in their discussion of challenges in the optimal implementation of HIV prevention, diagnosis, treatment, and care, Eisinger et al[35] concluded that “knowledge gained through implementation science will be critical…to bring HIV prevention and treatment interventions to scale, and thus, achieve the goal of ending the HIV epidemic domestically and globally.”

The present study sought to build on the extant research on implementation effectiveness[36-41] defined as the consistency and quality of implementation over time. We examine several constructs hypothesized by the theory of implementation effectiveness[36-39] to be determinants of implementation effectiveness as part of an effectiveness-implementation hybrid trial focused on Positive Health Check (PHC). PHC is a web-based video doctor intervention designed to be delivered to PWH while in the clinic. PHC provides tailored content about medication initiation and adherence, sexual risk reduction, and other behaviors to decrease HIV transmission risk. The purpose of the effectiveness-implementation hybrid trial was to test if PHC supports viral suppression and retains PWH in care. Analyses showed that PHC was particularly effective for a priori defined subgroups. Males were more likely to achieve viral suppression, and the youngest and oldest participants were more likely to be retained in care.[42]

The design of the PHC effectiveness-implementation hybrid trial has been previously described.[43] As prompted by Kemp and Velloza, one aim of the PHC pragmatic trial was to explore beyond feasibility and acceptability to what extent other important implementation constructs could be identified as predictors of implementation effectiveness.[13] The theory of implementation effectiveness[36-39] posits that innovation-values fit, organizational readiness for change, and implementation climate are 3 key determinants of implementation effectiveness. These implementation constructs are defined as follows: (1) innovation-values fit (ie, the extent to which staff perceive that innovation use will foster the fulfillment of their values)[36-39]; (2) organizational readiness for change (ie, the extent to which organizational members are psychologically and behaviorally prepared to implement organizational change); and (3) implementation climate (ie, the extent to which organizational members perceive the use of a specific innovation to be expected, supported, and rewarded within their organization).

Given the limited empirical research examining these determinant constructs[39-41] and the need for psychometrically sound measures to assess these constructs,[44-46] we chose to use a mixed-methods longitudinal analysis for this study to use qualitative data from interviews with PHC implementers to help confirm and explain quantitative findings regarding the stability or change over time around these 3 constructs. Our general null hypothesis was there would not be any significant changes over time in these 3 contrasts. Consistent with the philosophy that change is the only constant in life our general alternative hypothesis was that there would be significant changes over time for each of these posited determinants of implementation effectiveness. We then used novel joint display tables to present the statistical outcomes and supporting qualitative data side-by-side. This approach to integrating qualitative and quantitative data allowed us to comprehensively describe our findings from the implementation component of the PHC hybrid trial.

METHODS

Study Design

For this hybrid trial's implementation aim, a longitudinal mixed-methods design was used in which quantitative (surveys) and qualitative (interviews) assessments were conducted regarding PHC implementation.

Surveys and interviews overlapped with the PHC implementation period across all 4 clinics and were conducted at the beginning of the study and then at every 3 months over a 23-month period, providing 8 data collection time points. These assessments were conducted from February 2018 to December 2019. The design uses data from the qualitative assessments to provide context for the interpretation of the quantitative findings.

Intervention

PHC is an interactive, highly tailored intervention informed by multiple health behavior theories, including motivational interviewing,[47] the Information-Behavioral-Motivation Model,[48,49] and the Transtheoretical Model.[49] The PHC intervention consists of 7 core components: (1) participant-reported tailoring questions. This included 4 demographic questions, delivered by a video nurse used to tailor and route a participant through the intervention, and 17 questions delivered by a video doctor, interspersed throughout the intervention to provide tailored information in 6 domains, including treatment readiness, medication adherence, RIC, sexual risk reduction, mother-to-child transmission, and injection drug use; (2) tailored content delivered in the 6 domains; (3) behavior change “tips” provided across the 6 domains; (4) 4 video doctor options [varying by race (Black, White) and sex (female and male)]; (5) library that autogenerated a list of tailored questions based on participant preferences that could be used during their clinical encounter; (6) patient handout, which could be printed on site, showing the behavior change tips and questions for the provider; and (7) an “Extra Info” microsite at the end of the intervention with additional resources and information, such as sexually transmitted infections, condom use, mental health, and transgender health. A clinic web application also helped clinic staff keep track of user progress, assisted with personal identification creation, and produced summary descriptive reports about overall use of PHC.

Implementation Strategy

The implementation strategy for the trial was informed by a previous 1-month pilot implementation. The strategy involved preimplementation preparation, such as clinic workflow assessments, technology assessments, staff training, and understanding the patient population. Implementation strategies supporting the PHC launch involved physical walkthroughs of the clinic environment and workflow integration, Wi-Fi testing, educational sessions for clinic staff, study staff training, development of an implementation binder describing all aspects required for successful implementation, and a soft launch. During implementation, each clinical site received tailored support, lessons learned were shared across sites, FAQs were revised in real time, implementation technical assistance was provided as needed, PHC implementation was monitored via the clinic web application, and the Centers for Disease Control and Prevention maintained a Help Line users could call with technical questions.[50]

Setting

PHC implementation and data collection occurred simultaneously across 4 demographically diverse US-based HIV care clinics, 1 in the south-central region, 2 in the southeast region, and 1 in the northeast region. The clinics are described as follows:

Clinic A Southeast Region Clinic is a primary care practice and medical specialty practice with an average of 20 HIV patients per day.

Clinic B South-Central Region Clinic is an ambulatory clinic, a community health center, a primary care practice, and a nonprofit clinic with an average of 15 HIV patient visits per day.

Clinic C Northeast Region Clinic is an ambulatory care and academic medical center with an average of 40 HIV patient visits per day.

Clinic D Southeast Region Clinic is a primary care practice and specialty care practice with an average of 110 HIV patient visits per day.

Participants

Guided by our teams' research focused on the rates and impacts of staff turnover,[51,52] our current research focused on clinic roles/positions (eg, project coordinator, data manger, outreach coordinator, research coordinator, or clinic champion). This is important given that clinics generally have much less control on the extent to which individuals are retained than they do on the extent to which their clinic roles/positions have 1 or more actively employed staff.

Data Sources and Data Collection Procedures

At each of the 8 time points (T1–T8), starting immediately before implementation of PHC with patients, and every 3 months thereafter, participating staff members across all 4 clinics completed a 15-minute online survey assessing 3 key implementation constructs: perceived fit, organizational readiness, and implementation climate. At the start of each survey, staff agreed to participate through an online consent form. After the online survey, we conducted individual interviews with staff who completed the survey at that time point. At the start of each qualitative interview, we obtained verbal consent from the clinic staff member. Each clinic's Institutional Review Board and the RTI International Institutional Review Board approved the study protocol.

Quantitative Measures

Innovation-values fit was assessed via 25 items developed for this project. More specifically, using a 5-point scale (0 = not at all to 4 = highest extent possible). Each staff member answered 5 fit questions related to implementing PHC (ie, fit your clinic workflow; fit your clinic values; fit your clinic treatment philosophy; was accepted by staff within your clinic; was well-matched to your clinic environment). These fit questions were asked in regard to 5 PHC study components: (1) PHC Patient Onboarding (ie, staff assisting users with logging-in), (2) PHC Delivery through iPads or android tablets, (3) PHC Handout Printing and Delivery, (4) Clinic Web Application for tracking patients' PHC use, and (5) PHC Patient Outreach. The average coefficient alpha (α) across the 8 time points was 0.95 (SD = 0.02). Organizational readiness was assessed with 12 items adapted from a measure developed by Shea et al.[53] Reponses were given on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Across the 8 time points, α was 0.92 (SD = 0.03). Example items include “Positive Health Check staff implementing PHC want to implement this intervention” or “Positive Health Check staff implementing PHC are motivated to implement this intervention.” Finally, implementation climate was assessed using 6 items adapted from a measure developed by Jacobs et al.[54] Reponses were recorded on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Although not as high as for the other 2 measures, the α was acceptable at 0.76 (SD = 0.11). Example items include “Positive Health Check staff implementing PHC were expected to help the clinic meet its goals for implementing Positive Health Check” or “Positive Health Check staff implementing PHC got the support from clinic management they need to use Positive Health Check with eligible patients.” Scale scores for innovation-values fit, organizational readiness, and implementation climate were constructed as the mean of the individual items corresponding to each construct computed at the respondent level.

Qualitative Measures

Using a semistructured interview guide, we sought to further understand the stakeholders' perceptions of their respective clinic's implementation climate, organizational readiness, and perceived fit of PHC as assessed by the online survey. After reviewing the clinic site's survey responses, individual staff were asked a series of open-ended questions, for example, “Thinking about the process overall, what are some activities related to clinic workflow that are facilitating the implementation of Positive Health Check?” and “How, if at all, are project staff supported to make sure PHC is implemented as it needs to be?” Interviews typically lasted 30–45 minutes. Two study staff conducted each interview by phone, one to lead the interview and one to take notes. All interviews were digitally recorded and transcribed for analysis.

Quantitative Data Analysis

Examination of the implementation context over time often entails having a relatively few analytic units,[55] in this case HIV primary care clinics. We managed the constraints of the small sample size by using longitudinal mixed-effects regression models of respondent level scale scores. Clinic differences were modeled using random intercept terms for clinic, and repeated measures within individual respondent were modeled using random slopes. The fixed effects for linear and quadratic slopes assess change over time, and their associated t-tests were used to test whether there was significant change over time within the implementation context for PHC. Each outcome was modeled separately with time and time squared as predictors, and the data collection time points were scaled from 0 to 7 so that the mean of the random intercepts can be interpreted as the predicted value of the outcome at the first time point. These predictors, respectively, tested whether there was change in the outcome overall and whether change was curvilinear (eg, the outcome improved, then deteriorated). Model fitting was implemented using the PersonAlytics[56] R package,[57] which uses the GAMLSS framework and R package for model fitting.[58]

Qualitative Data Analysis

We used a framework analysis method[59] for the qualitative coding and analysis approach. Once interviews were transcribed and entered into NVivo, a multifunctional software system for coding and analysis, 3 project team members independently coded a subset of transcripts. Discrepancies in initial coding were resolved by discussion between project team members. The coders then independently conducted a final coding of the remaining data, with 1 staff person coding each transcript. Intercoder agreement was quantified for each code using Cohen Kappa.[60] The range of Kappa across codes was 0.75–1.00. Finally, thematic analysis was conducted.

Mixed-Methods Integration

Upon completion of the quantitative and qualitative data analyses, we integrated the data using joint display tables to examine the changes in implementation context over time (Tables 1–4). These tables present the 2 forms of data to display how the commonalities and differences across dimensions of implementation context vary over time and also account for stakeholder type (eg, PHC outreach coordinator) and characteristics of each clinical site (eg, academic medical center).

TABLE 1.

Overall Trajectories for Innovation-Values Fit, Organizational Readiness for Change, and Implementation Climate

	Cross-clinic Trajectories	Beta	PHC Staff Experiences
Innovation-values fit		Intercept: 4.18*Time: −0.25Time2: 0.03	Three of the 4 clinics reported few major ongoingbarriers with integrating PHC into the clinic workflow.Communication and collaboration with front office staff and the PHC staff's physical location within the clinic emerged as key factors for integrating PHC into the clinic workflow.Addressing internet connectivity issues and printer access at 2 clinics and space concerns at 1 clinic align with the slight increase in perceived fit at T4 (Table 2).
Organizational readiness for change		Intercept: 4.42*Time: 0.11Time2: −0.02	Three of the 4 clinics noted a slight decrease inorganizational readiness at T7 because of clinic providers misunderstanding PHC requirements (clinic B), a nondirect chain of communication clinic C, and staff turnover clinic D.
Implementation climate		Intercept: 3.95*Time: 0.55*Time2: −0.09*	Clinics had not started implementing the interventionat T1. The initial trend upward may be because of ramp-up time.The most common leadership method used to support PHC was advocating for the study in clinical site meetings.All 4 clinics noted receiving support from site health care providers.Implementation climate dipped in the end for many reasons, including decreased engagement from champions and difference in priorities between research and clinic staff.

Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.

P < 0.05 indicates a significant change over time.

TABLE 4.

Implementation Climate, by Clinic

	Trajectories	PHC Staff Experiences	Illustrative Quotes
Clinic A	Intercept: 4.24*Time: 0.38*Time2: −0.06	PHC research staff felt supported by and had strongcommunication with clinic staff.Clinic leadership was highly engaged throughout and regularly sent email updates to clinic staff and discussed PHC during meetings.Early on and toward the end of the study, some clinical staff got upset when PHC caused delays in the clinic workflow. No clear theme emerged explaining a decline in climate at the end.	“Oh yeah. We definitely are all updates are sent to usduring staff meetings in the clinic. We are kept in the loop.” (T3, Project Coordinator)“Well, anytime you make a change to something, folks who are established in a particular clinic can feel like this is one more thing we got to do and we try not to make that a burden for them but for us to handle it.” (T5, Project Coordinator)
Clinic B	Intercept: 3.92*Time: 0.38*Time2: −0.06*	PHC was supported by the clinic and staffexpectations were consistently communicated.The most common leadership method used to support PHC was advocating for the study in clinical site staff meetings.There were changes at T4, including hiring a new clinical champion and data manager and moving to a new building, which temporarily decreased champion engagement.Toward the end, troubleshooting and problem solving was expected to be done independently by the research team with only ad hoc support from the clinical champion.	“I think it's been pretty unique at this clinic becausewe had so many changes in the past 3 months with […], our previous PI leaving and […], our data person, being absent for a while and the clinic moving, that it’s definitely leadership has been a little bit more absent than I think we'd like.”(T4, Outreach Coordinator)“On our monthly clinician meetings or provider meetings we always do some sort of update or reminder to the folks that, whether they're working on re-engagement or recruiting, we let the providers know, you know, that it’s happening.” (T6, Clinical Champion)
Clinic C	Intercept: 4.11*Time: 0.42*Time2: −0.05*	There was a lot of support within the clinic for PHC.The clinic had 2 clinical champions but not until T2 did PHC staff note feeling supported.Leadership engagement and support were mainly provided through routine or ad hoc meetings.Clinical staff showed their support by being flexible in their schedules and helping to identify patients.	“Leadership is very involved and that I can calleither one of the PIs on this study and ask them a question right now and they would help me. And if they couldn't help me they would find a way to help me. (…) They're always really available for us.”(T2, Data Manager)“We support each other […] we always talk to each other, the case managers, the outreach people, the social worker, nutritionist. All is integrated here in the clinic, and mostly we work together.”(T7, Project Coordinator)
Clinic D	Intercept: 4.13*Time: 0.38*Time2: −0.05*	The research team felt supported by clinicleadership. Although some were more involved than others, the clinical champion was engaged throughout.Early on, and at T6 and T7, staff noted a disconnect between PHC staff and clinic staff because of poor communication and resistance from clinical staff to adjust the clinic's workflow to support the project.Implementation climate improved when the clinical champion helped troubleshoot problems for PHC staff.	“We were still figuring out who should be doingwhat and at what time or at what stage of the patients' visit. But I think now everything is working pretty well and everyone more or less knows what they need to be doing.” (T2, Project Coordinator)“Yeah, I help to troubleshoot all the problems in terms of implementation, recruitment, I work closely with my team to do that.” (T5, Clinical Champion)

Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these two effects describes the upside-down “u” shape of the trajectory over time.

P < 0.05 indicates a significant change over time.

RESULTS

A range of staff across the 4 clinics participated in each round of surveys and interviews. Staff across clinics also shared roles. Consequently, a clear delineation of which staff member was taking on which PHC role in each round was not feasible. We interviewed site clinic champions, and staff responsible for implementing and monitoring the intervention, including site coordinators responsible for onboarding and introducing PHC to participants, data managers, and outreach workers. We had participation from 3 to 5 staff at each site for each time point, resulting in a total of 126 completed interviews and surveys. Staff did not receive individual incentives for participation. Clinics received funding for their overall involvement in the PHC study which covered the salary for involved staff. Although some clinic staff were assigned to work on the study full time, others spilt time between the study and other clinic-based tasks.

Outcome Data

Results are summarized in Tables 1–4. In Table 1, quantitative results are presented in the “Trajectories” column and results of the mixed-effects models are shown in the “Beta” column; they are also superimposed in the “Trajectories” column of Tables 2–4, with the labels “Time” and “Time2” (time squared). Qualitative results are summarized in the “PHC Staff Experiences” columns of Tables 1–4 and the “Illustrative Quotes” columns of Tables 2–4.

TABLE 2.

Innovation-Values Fit Trajectories, by Clinic

	Trajectories	PHC Staff Experiences	Illustrative Quotes
Clinic A	Intercept: 3.89*Time: 0.00Time2: −0.01	There were very few clinical workflow barriers to integrating PHC.Consistent communication and collaboration between PHC staff and clinic staff facilitated intervention delivery.The clinic's physical space enabled easy intervention and handout delivery within the clinic's existing workflow.The clinic web application was useful for monitoring patient progress, but it would time out too quickly.There were small but ongoing concerns around PHC delaying clinic schedules, particularly when patients arrive late.	“No (workflow barriers to report), our clinic isawesome and their workflow, they've adjusted so well to us and want to accommodate us. So that's not a problem ever.”(T3, Study Coordinator)“There's a designated spot for the patients who utilize the device. We have a phlebotomy room where we're able to see the patients privately deliver the intervention privately. Having those features of the clinic makes the implementation or the delivery of the intervention much easier.” (T5, Outreach Coordinator)
Clinic B	Intercept: 3.20*Time: −0.16Time2: 0.03	Finding space for PHC delivery was a significantchallenge until the clinic moved into a new building at T4.New clinical care hires at T3 exacerbated space issues.Early frustration with the printer location was alleviated when the clinic was relocated to a new building with technology that enabled staff to print handouts at any printer within the clinic.Patient availability for PHC was influenced by the number of appointments with other departments (eg, case managers, behavioral health).Toward the end of the study, decreased staffing on PHC became a minor capacity issue.	“As it’s been, it's been fine except for when we had tomove the printer around.” (T2, Project/Outreach Coordinator)“We've recently had an uptake in hires, especially at the marine building. The pods that we used to have open all the time, now they're starting to be taken […]”(T3, Study Coordinator)“We're in a newer building. […] The clinic is designed where there's lots of private rooms where they can be very close to either the lab or the doctor's office. So, I don't see any barriers.”(T7, Clinic Champion)
Clinic C	Intercept: 3.94*Time: −0.25Time2: 0.03	The physical location of the PHC staff within theclinic supported the integration of PHC into the workflow.Ongoing issues included limited clinic space, patient no-shows, and late arrival times that prevented patients from completing the tool—especially at T5 and T6.At T2, there were issues with the clinic's Wi-Fi and with printing patient handouts (which worsened with poor Wi-Fi). At T4, staff began using a clinic network printer, easing the handout printing process.At T5, staff noted that research was secondary to patient care and that patients occasionally were not interested in PHC because it did not provide new information.	“We did have a problem with printing the handoutsfor a while because the hotspot or whatever that we were given didn't really work. […] We started a new process […] where we get instead of having to print the handout on that specific printer and everything, we get it in our email. […] This is really an amazing change because now we get all the handouts.” (T4, Research Supervisor)“It depends on, patients don't always come when they're supposed to, and I think that has been the challenge. […] If they come right on time and the physician is ready to take the patient, they're not going to wait because then they will get backed up, and that impacts the clinic load.”T4, Clinic Champion
Clinic D	Intercept: 3.90*Time: 0.04Time2: −0.02	Overall, there was ample clinic space to implementPHC.PHC staff consistently noted challenges around communicating with and engaging clinic staff, particularly front desk staff. This was attributed in part to high clinic staff turnover.The location of the PHC research team outside of the physical clinic space posed a challenge. The relocation of the research office near T3 further contributed to this challenge.At T2, staff discussed concerns around Wi-Fi connectivity, which was addressed at T3 with the purchase of a Wi-Fi hotspot.With the exception of the printer occasionally being slow, there was no issue with handout delivery.	“Communication between research and the front desk,or providers. Although we've improved, there have been barriers where communication maybe hasn't been processed all the way through. Like we'll tell the provider, and the provider maybe forgets to tell the patient, or the front desk forgets to tell the patient.” (T5, Outreach Coordinator)“Although we are in the same building, we're not located in the same exact room. We're a little bit further away, and out of the natural flow of things, from like check-in to nursing station, to labs and everything. We're completely removed from that. So that's been a huge challenge.” (T6, Outreach Coordinator)

Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.

P < 0.05 indicates a significant change over time.

Results of the mixed-effects models are summarized in statistical tests for each predictor labeled as “beta” in Tables 2–4, which show the clinic average of the scale scores (see Quantitative Measures) for each outcome. The beta value for the intercept gives the predicted average baseline value for each outcome, and they are all significantly different from zero. The beta values for time and time squared are the rates of change over time and over time squared, respectively. When the beta values for time were not significantly different from 0, there is no change over time on average. A significant positive beta value indicates improvement over time on average, and a significant negative beta indicates deterioration over time. When the beta values for time squared were significantly different from 0 and were negative, this indicates that improvements in the outcome improved and then attenuated over time. All beta values for time squared that were significantly different from zero were negative (if positive values had occurred, this would indicate deterioration, then improvement).

Innovation-Values Fit

On average, innovation-values fit was 3.77 (SD = 0.79) at T1 and 3.62 (SD = 0.63) at the final time point, with no significant change over time (Table 1). As reported in Table 2, only one of the 4 clinics reported having a noteworthy issue regarding innovation-values fit. More specifically, staff from clinic B reported barriers to finding space for delivering PHC, which were exacerbated at T3 because of several new clinical care hires. By T4, space issues had been resolved as a result of moving into a new building with more available space. For clinic C, innovation-values fit was found to be the lowest at T5 and T6, with staff reporting that some patients were not interested in PHC because it did not provide new information.

Organizational Readiness for Implementing Change

On average, organizational readiness was 4.39 (SD = 0.69) at T1 and 4.67 (SD = 0.41) at T8, the final time point, with no significant change over time across all clinics (Table 1). As also reported in Table 3, clinic C staff noted at T1 that PHC had not fully been introduced to physicians, which may explain the relatively low organizational readiness rating at T1. However, organizational readiness significantly increased at T2 and remained relatively stable for the remainder of the project.

TABLE 3.

Organizational Readiness for Change Trajectories, by Clinic

	Trajectories	PHC Staff Experiences	Illustrative Quotes
Clinic A	Intercept: 4.43*Time: 0.04Time2: −0.01	Early engagement from the clinical champion set thestage for organization buy-in at the start and throughout the PHC implementation process.Continual communication between PHC staff and clinic staff enabled the organization to smoothly implement PHC.	“Over the past several months our clinic staffmeeting, which is not a research meeting but just a meeting of all the clinic providers and staff, I've been getting people excited and enthusiastic and trying to head off any concerns we may have as we roll into this.” (T1, Clinical Champion)“I think I speak for the entire project staff that we feel very supported by our leadership, by our clinic team […] and the clinic staff is as supportive as they can be without causing too much additional work for them because they're overworked and understaffed. […] But they're incredibly supportive.”(T5, Outreach Coordinator)
Clinic B	Intercept: 4.57*Time: −0.04Time2: 0.00	“Road bumps” were noted at T2 because PHCresearch staff were grouped within the IT department, whose workflow is not integrated into that of the clinic or research department.At T7, staff noted a lack of confidence navigating some PHC implementation challenges, specifically those related to co-enrolment in other studies in the clinic, and therefore contacted the clinical champion for help.	“I think for the position we're in, it's beingimplemented really smoothly, but I think a lot of the road bumps, or the bumps along the way, there's questions we've had to ask that would have been easier answered if we were in a different department.” (T2, Project/Outreach Coordinator)“We've been withdrawing some folks because they were getting to co-enrolled in another research program and we definitely had to go to (…) our PI, and some leadership and ask them to kind of step in and take over on that front because we didn't think we could do it.”(T7, Study Coordinator)
Clinic C	Intercept: 4.40*Time: 0.23Time2: −0.03	Organizational readiness was high except at T1 andT7. This may have been because of frustrations around communication.At T1, PHC staff noted that physicians had not yet been fully introduced to PHC, which might have been a barrier to their fully supporting the intervention.At T7, PHC staff noted that the chain of communication for PHC staff is not direct. However, meetings helped clarify roles and expectations within the organization.	“I think we've mentioned it at clinic meetings, andit'll be on our agenda for our next physician meeting, […] We'll be able to introduce it in a little bit more detail to the physicians, to let them know the study is going to be opening and that the types of patients that are going to be eligible for it.” (T7, Clinical Champion)“We'll set time at the end of the week to huddle, particularly when all the other changes, all of our staffing changes were happening. We were meeting a bit more frequently to go over what the expectations are, to understand what people's roles were, and to make sure everything was really clear.” (T7, Research Supervisor)
Clinic D	Intercept: 4.48*Time: 0.13Time2: −0.02	Overall, staff felt that leadership displayed opencommunication and problem-solving abilities.At T7, staff felt less confident in their ability to successfully execute PHC because of organizational challenges, such as high clinical staff turnover.	“PHC is definitely one of the main topics [discussedduring the monthly clinic meeting] most of the time because it's the one study that requires the most coordination between providers and research. I think it's always being discussed, better ways of maintaining the study.”(T2, Study Coordinator)“You know, we end up doing a lot of heavy lifting by the PHC staff rather than involve the clinic staff, because we have a lot of staff turnover and is so true. So, (Data Manager) eg, and (Out reach Coordinator) now has been more involved to get the patient to come back for follow up.” (T6, Outreach Coordinator)

Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.

P < 0.05 indicates a significant change over time.

Organizational readiness was very high and very stable in clinic A, where early engagement from the clinical champion was reported as having set the stage for organizational buy-in and maintained buy-in throughout the project. Although not significant, both clinic C and clinic D reported decreased organizational readiness at T7. At this time point, clinic C staff noted that the chain of communication for PHC staff was not direct, whereas clinic D staff noted feeling less confident in their readiness to implement PHC because of recent staff turnover.

Implementation Climate

There was significant curvilinear change over time regarding implementation climate. On average, implementation climate was 3.82 (SD = 0.88) at T1 and 4.08 (SD = 0.87) at T8, the final time point, but peaked at 4.77 (SD = 0.32) at T5 and decreased at T6 through T8 (Table 1). Table 4 shows that in clinic A, no clear themes emerged that helped explain the declines in implementation climate reported at T7 and T8, last 2 time points. In clinic B, however, decreased implementation climate seems to have been due in part to staffing changes and decreased support from the clinical champion toward the end of implementation, who was expecting staff to be able to address issues with only ad hoc support.

The implementation climate in clinic C remained relatively high over time. In clinic D, the decrease in implementation climate rebounded after the clinical champion helped troubleshoot problems.

DISCUSSION

As part of a type 1 effectiveness-implementation hybrid trial, we used a mixed-methods approach to examine the context of implementing PHC over a 23-month period within 4 HIV primary care clinics. Overall, we found innovation-values fit and organizational readiness for change to be quite high throughout the project. There were some logistical challenges, however, in implementing the intervention, and we found a significant curvilinear pattern of change over time in implementation climate.

The analysis of the quantitative survey data show that, overall, innovation-values fit remained stable. Staff interviews suggested that most of the barriers were around WIFI glitches and finding space. Any barriers to fitting PHC into clinic workflow were well managed by clinic staff. This is encouraging for the successful implementation of PHC in other settings and contrasts with previous research reporting that clinic workflow and other aspects of the clinic environment can be barriers to implementing STD/HIV interventions in busy clinic settings.[61,62]

With regard to organizational readiness for change, the quantitative data also show relatively stable levels over time across the 4 clinic environments. The qualitative data reveal many facilitators for organizational readiness, including the clear communication of staff roles and responsibilities for implementing PHC, and many barriers such as in one clinic, neglecting to confirm that all physicians were introduced to the PHC intervention.

Although adequate levels of innovation-values fit and organizational readiness may be prerequisites for effective implementation, they are not likely to be sufficient. Indeed, implementation effectiveness theory posits implementation climate—the extent to which implementation by the intended users is expected, supported, and rewarded—as the most proximal determinant of implementation effectiveness.[36,38] In alignment with the theory of implementation effectiveness, we found that both our quantitative and qualitative data support significant changes over time in implementation climate.

Importantly, we observed significant variation in implementation climate for each of the 4 HIV primary care clinics. The qualitative data provided more contexts to facilitate a partial explanation for this pattern of results. For example, during interviews, stakeholders reported that key issues between research staff and healthcare providers included communication, coordination, and staffing. Thus, we hypothesize that a key factor that may have helped improve implementation climate, especially the support dimension of implementation climate, was the increasing involvement of a strong clinic champion, which has been noted in previous literature as an important implementation strategy.[63] In implementation climate declines observed, we hypothesize that these declines may be common to research projects that have a predefined end and may not be observed if the implementation of the intervention was something the organization was planning to sustain or even scale-up over time.

As previously mentioned, much of the implementation research on HIV interventions has focused on feasibility and acceptability.[23,32-35] The current study moves beyond those constructs to explore other important implementation constructs, including innovation-values fit, organizational readiness for change, and implementation climate.[36-38] However, the extent to which research has empirically examined these constructs has been limited because of the lack of psychometrically sound measures.[44,64] In addition to being one of the first studies to use the organizational readiness measure developed by Shea et al,[53] the current study is the first known study to have collected this measure at multiple time points throughout the implementation process. The current study is also the first we are aware of to have longitudinally collected measures of innovation-values fit at so many time points during the implementation process. In the absence of the qualitative interviews, the limited change over time found may suggest that these 2 measures lacked sensitivity to change. Fortunately, however, the qualitative interviews substantiated the relatively high and stable levels of innovation-values fit and organizational readiness for implementing change.

Limitations

A potential limitation to this assessment of PHC implementation is the inability to adjust measures by clinic-specific respondent. Staff roles across each clinic shifted and the number surveys/interviews for each round ranged as a result of limited staff availability. Consequently, it was not feasible to make clear delineations for which staff member assumed which PHC role. Future studies examining clinic-level climate and culture issues should consider adjusting for respondent composition in the regression models.

Another possible limitation is that the data collection ended (December 2019) before the PHC implementation process ultimately ended (March 2020), so we cannot describe how these constructs varied near the end of implementation. In addition, because we had only 4 clinics in the study, we did not have sufficient power to examine the relationship between these implementation constructs and implementation effectiveness (ie, the consistency and quality of implementation over time). Future studies engaging larger numbers of clinics should include analyses that link these constructs with implementation effectiveness.

CONCLUSIONS

Technology-based interventions may be one of the most promising innovations to increase HIV viral suppression rates and decrease the number of new HIV infections.[65] However, given the consistent lag between research and practice, there is increasing recognition of the need to integrate effectiveness research and implementation research.[66,67] Our innovative type 1 hybrid trial design demonstrated that within the context of a research project, PHC could be successfully implemented within 4 HIV clinics. We believe that the combined factors of perceived fit, organizational readiness, and implementation climate facilitated implementation of PHC over time. Indeed, the qualitative data show that as participating HIV primary care clinics intermittently addressed space, workflow, and technical issues, they also consistently reported PHC as being a good fit within their clinic. Additionally, each HIV primary care clinic was able to achieve and then sustain adequate levels of readiness to implement the PHC intervention. In contrast, the extent to which PHC implementation was expected, supported, and rewarded was found to vary significantly over time in each of the HIV primary care clinics. Examining these factors as determinants of implementation effectiveness is planned as part of our future work.

Finally, as this study demonstrates, it may be important for implementation strategies to increase the extent and consistency with which an intervention is expected, supported, and rewarded to result in improved implementation outcomes.