Interventionist training and intervention fidelity monitoring and maintenance for CONNECT, a nurse-led primary palliative care in oncology trial.

CONTEXT: Intervention fidelity is a critical component of behavioral research that has received inadequate attention in palliative care studies. With increasing focus on the need for palliative care models that can be widely disseminated and delivered by non-specialists, rigorous yet pragmatic strategies for training interventionists and maintaining intervention fidelity are needed.
OBJECTIVES: (1) Describe components of a plan for interventionist training and monitoring and maintaining intervention fidelity as part of a primary palliative care trial (CONNECT) and (2) present data about perceived training effectiveness and delivery of key intervention content.
METHODS: Post-training evaluations, visit checklists, and visit audio-recordings.
RESULTS: Data were collected from June, 2016 through April, 2017. We include procedures for (1) identification, training and certification of oncology nurses as CONNECT interventionists; (2) monitoring intervention delivery; and (3) maintaining intervention quality. All nurses (N = 14) felt prepared to deliver key competencies after a 3-day in-person training. As assessed via visit checklists, interventionists delivered an average of 94% (SD 13%) of key content for first intervention visits and 85% (SD 14%) for subsequent visits. As assessed via audio-recordings, interventionists delivered an average of 85% (SD 8%) of key content for initial visits and 85% (SD 12%) for subsequent visits.
CONCLUSION: We present a 3-part strategy for training interventionists and monitoring and maintaining intervention delivery in a primary palliative care trial. Training was effective in having nurses feel prepared to deliver primary palliative care skills. As assessed via nursing checklists and visit audio-recordings, intervention fidelity was high.

Introduction

Given significant shortages of palliative care specialists, strategies to improve provision of ‘primary’ palliative care by non-specialists are needed [1,2]. CONNECT (Care management by Oncology Nurses to address supportive care needs) is an oncology nurse-led care management intervention designed to strengthen provision of ‘primary’ palliative care within oncology practices (3). CONNECT visits take place at the same time as a patient's regularly scheduled oncology clinic appointment, occur monthly for at least 3 months, and may be conducted via telephone if a patient is unable to attend in person. A pilot trial demonstrated excellent feasibility, acceptability, and perceived effectiveness [3], and a multi-site cluster randomized efficacy trial is underway (Clinical Trials.gov NCT02712229) [4].

Intervention fidelity is an important element of clinical trial design and conduct [[5], [6], [7], [8]] that has received variable attention in previous palliative care intervention trials. For example, a trial of specialty palliative care for patients with advanced cancer reported the frequency but not the content of palliative care visits [9], a trial of early specialty palliative care in advanced lung cancer included descriptive analysis of clinician visit documentation [10], and a trial of structured telephone-based palliative care included review of audio-recorded sessions [11,12]. Notably, these trials used interventionists with specialty palliative care training. With increasing focus on the need for evidence-based ‘primary’ palliative care models, rigorous yet pragmatic strategies for (1) identifying and training interventionists and (2) monitoring and maintaining intervention fidelity are needed. A recent systematic review concluded that, to date, implementation fidelity in palliative care is under-recognized [13].

In this manuscript, we describe the key components and implementation of a plan to train interventionists and monitor and maintain intervention fidelity as part of the CONNECT trial. We include assessment tools and present data about perceived training effectiveness and delivery of key intervention content as a blueprint for future ‘primary’ palliative care intervention research.

Methods

Design

CONNECT (Care management by Oncology Nurses to address supportive care needs) is an intervention in which existing oncology nurses are identified and trained to provide primary palliative care within oncology practices. The protocol for a cluster-randomized trial evaluating the impact of CONNECT versus usual care on outcomes among patients with advanced cancer and their caregivers has been published previously [4]. No prior analysis has focused on interventionist training or intervention fidelity monitoring and maintenance.

We designed the intervention fidelity and monitoring and maintenance plan (IFMP) for CONNECT with the goals of: (1) ensuring high quality and consistent delivery of a ‘primary’ palliative care intervention; (2) reducing drift in protocol adherence over time; (3) identifying potential problems in intervention delivery that may require immediate remediation or modifications in future studies; (4) ensuring the ability to draw accurate conclusions about treatment efficacy from our results; (5) ensuring that investigators can replicate the intervention in future studies and providing guidelines for implementation. Below we describe each component of the IFMP. Data were collected from June, 2016 through April, 2017.

Identifying training, and certifying oncology nurses as CONNECT interventionists

The CONNECT intervention is delivered by RN-level oncology nurses at each intervention site. We chose to have existing oncology nurses deliver the intervention, rather than research or palliative care nurses, because our goal was to develop and evaluate an intervention that could be widely disseminated (RN-level oncology nurses are available at all oncology practices) and to leverage existing relationships between oncologists and their staff. We convened a Nurse Advisory Board, led by a Nurse Project Manager (PM) with palliative care expertise, to assist with identifying and training nurses to deliver the intervention. Participating sites nominated candidates who had strong relationships with clinicians and staff, excellent communication and interpersonal skills, an interest in learning new skills, and a commitment to palliative oncology care [4]. Preferred candidates had a minimum of 5 years of oncology nursing experience and Oncology Nursing Society (ONS) certification. When possible, we selected and trained two oncology nurses from each of the 8 participating sites to allow for potential staff turn-over. Oncology nurses selected as CONNECT interventionists underwent a standardized 3-day training led by the Principal Investigator (PI), the Nurse PM, and two faculty members with expertise in communication training and nursing education. Staffing reassignments allowed nurses to have protected time for training. A structured curriculum focused on (1) symptom assessment and management, (2) emotional support for the patient and caregiver, (3) advance care planning, and (4) care coordination. Simulated patients were used for role-playing, allowing nurses to practice and receive feedback on key communication skills.

Following each training, nurses completed a formal self-evaluation (see appendix), rating the extent to which they felt prepared to perform key competencies. In the event that a nurse did not report feeling ‘well’ or ‘very well’ prepared after the standardized training session, supplemental targeted training was provided in that area.

To ensure that each nurse was well-prepared to deliver the intervention, we conducted an individual, in-person certification visit several weeks after the training. During these visits, the nurse conducted an observed visit with a simulated-patient, which was rated for content and quality using standardized criteria (see appendix). Upon demonstrating each intervention visit component with a total quality rating of two (out of a possible three) or higher, the nurse was certified as a CONNECT interventionist and eligible to deliver the intervention.

Monitoring intervention delivery

After every CONNECT encounter, the CONNECT interventionist completed a protocol summary and session-specific checklist. The protocol summary checklist includes the encounter date, visit number, mode (in-person vs telephone), and a field to comment on the reason for any protocol deviation. Session-specific checklists include the visit components and a field note form for the interventionist to document any unusual circumstances surrounding each intervention session (see appendix). These checklists were completed on the same day as the encounter to ensure accurate and real-time monitoring of study procedures.

CONNECT interventionists also audio-recorded patient visits occurring both in-person or by phone, except in cases where audio-recording was deemed to be too sensitive or disruptive. In these cases, visit field notes and/or direct observation were substituted. Interventionists used study tablets for audio-recordings, which were directly uploaded to secure study files for review by research staff. To ensure that any issues with intervention fidelity were addressed in a timely fashion, the first two visits by each CONNECT interventionist were reviewed by the Nurse PM. Subsequently, for each CONNECT interventionist, a randomly-selected subset of audio-recordings (20% of anticipated visits for the entire trial) were reviewed and rated for content and quality. Visit Evaluation Forms and Standardized Visit Quality Rating criteria were used to review and rate audio-recorded visits (see appendix). Our threshold for intervention content, based on audio-recorded encounters, was the presence (yes/no) of ≥80% of key components of the intervention.

Visit audio-recordings were audited by research staff and student assistants. In order to maximize reliability, all raters were trained by experienced staff using a three-step process: (1) listening to audio-recordings and rating together; (2) listening to audio-recordings together and rating separately; and (3) listening to audio-recordings and rating separately. Training was considered complete when raters achieved agreement (within 10%) for the visit content score.

Maintenance of intervention quality

The Nurse PM conducted a weekly telephone supervision session with each CONNECT interventionist. During these sessions, session-specific checklists and visit audio-recording evaluations were reviewed together. The Nurse PM shared visit evaluation scores with each CONNECT interventionist, identifying well-performed skills and opportunities for improvement. All scores falling below content thresholds were reviewed jointly to identify solutions. The Nurse PM also sent a weekly e-mail to all CONNECT interventionists with helpful tips and reminders about key competencies and conducted a monthly site visit to meet with each nurse in-person and review study procedures. Finally, a booster training session was held with CONNECT interventionists every six months to maintain intervention skills.

If problems with protocol adherence or intervention delivery were identified (e.g., failure to conduct CONNECT visits or delivery of <80% of key intervention components in audio-recorded visits), the Nurse PM and PI worked with the Nurse Advisory Board to implement a remediation plan. Remediation plans included meeting individually with the CONNECT interventionist to identify barriers and discuss strategies to improve adherence, observing the nurse's work flow processes to provide recommended strategies for improved protocol adherence, meeting with clinic staff and leadership to discuss opportunities for improvement, and providing targeted training in specific skills, followed by re-evaluation. The CONNECT Nurse PM also evaluated all subsequent visits until 2 consecutive visits were performed above the adherence threshold. If no acceptable solution was found after the remediation plans had been implemented, a different oncology nurse would be identified and trained to serve as the interventionist.

Results

Identification, training, and certification of CONNECT interventionists

To date (August, 2017), four three-day CONNECT training sessions have been held and 14 nurses have completed training. All nurses reported feeling well, or very well, prepared in key primary palliative care skill after training; no supplemental targeted training was required. Nurses reported that CONNECT training had the greatest impact on their preparedness to administer the Edmonton Symptom Assessment Scale (ESAS), explain their role as a CONNECT nurse, set a visit agenda, complete a shared care plan, and help a patient to complete an advance directive. Table 1 summarizes results from post-training evaluations.

Table 1

Self-reported preparation in key primary palliative care skills before and after CONNECT training (N = 14 participants).

Question	Average Score Before Training (SD) (1 = not well prepared; 5 = very well prepared)Mean (SD)	Average Score After Training (SD) (1 = not well prepared; 5 = very well prepared)Mean (SD)	Average Score Change from Before to After Training
Establish rapport with a patient	3.2 (1.0)	4.9 (0.4)	+ 1.6
Explain role as CONNECT RN	1.9 (0.9)	4.9 (0.4)	+ 2.9
Set Visit Agenda	1.9 (1.1)	4.8 (0.6)	+ 2.9
Assess patient views about his or her illness	3.1 (1.1)	4.7 (0.5)	+ 1.6
Assess how a patient is coping	3.1 (1.0)	4.6 (0.5)	+ 1.6
Provide emotional support	3.4 (0.6)	4.9 (0.3)	+ 1.5
Administer the Edmonton Symptom Assessment Scale (ESAS)	1.8 (0.8)	4.9 (0.3)	+ 3.1
Administer the NCNN distress thermometer	2.2 (1.2)	4.9 (0.3)	+ 2.7
Identify and assess symptom needs	3.6 (0.8)	4.9 (0.4)	+ 1.2
Help patients to focus on symptom goals	3.1 (0.8)	4.9 (0.4)	+ 1.7
Consider barriers to symptom management	3.1 (0.8)	4.8 (0.4)	+ 1.7
Use the “ask-tell-ask” approach to address a patient's symptoms	2.4 (0.9)	4.7 (0.5)	+ 2.3
Complete a shared care plan with a patient	1.8 (0.7)	4.6 (0.6)	+ 2.9
Use evidence-based symptom pathways to address common symptoms	2.8 (1.0)	4.6 (0.6)	+ 1.9
Help a patient to identify a surrogate decision maker	2.0 (0.9)	4.5 (0.6)	+ 2.5
Elicit a patient's readiness to think about the future	1.9 (0.7)	4.5 (0.6)	+ 2.6
Elicit how a patient would like their surrogate decision maker to approach decisions	1.7 (0.8)	4.4 (0.6)	+ 2.6
Elicit a patient's hopes for the end of life	1.9 (0.7)	4.4 (0.7)	+ 2.5
Elicit a patient's fears for the end of life	2.1 (1.0)	4.4 (0.6)	+ 2.3
Help a patient to talk with their family about the future	2.2 (0.9)	4.5 (0.6)	+ 2.3
Help a patient to ask their oncologist questions	3.5 (0.9)	4.9 (0.3)	+ 1.4
Help a patient to complete advance directive	1.8 (0.8)	4.8 (0.4)	+ 3.0
Discuss a patient's symptoms with the oncologist	3.9 (0.7)	4.9 (0.3)	+ 1.0
Discuss a patient's preferences and goals with the oncologist	3.5 (0.8)	4.9 (0.4)	+ 1.4

In addition, nurses rated the CONNECT training as ‘very important’ to the development of their own clinical skills (mean 4.9/5, SD 0.3) and unanimously rated their commitment to changing specific clinical behaviors as very high (mean 5/5, SD 0). Specific goals that nurses identified included overcoming fear of discussing end of life issues, displaying empathy, engaging with patients, addressing emotional needs, and working with advance directives. Nurses found the CONNECT training length to be ‘just right’ and strongly agreed that they would recommend this training for others (mean 4.9/5, SD 0.3). One nurse commented that training provided a ‘critical part of oncology nursing that has sadly been missing.’ Others noted that ‘repetitive role playing was a huge help’ and that the training provided ‘a skill set I believe will stick with me for quite a long time.’

In-person certification visits with each nurse were conducted an average of 2 weeks after the 3-day training. In these visits, all nurses demonstrated delivery of intervention components with adequate quality and were certified to begin delivering the intervention. The average content rating for certification visit was 91% (range 84%–100%), while the average quality rating was 78% (range 67%–94%).

Monitoring intervention delivery

Thirteen nurses had completed CONNECT intervention visits at the time of this analysis. Each of the nurses completed an average of 10 visits (range 2–31). CONNECT nurses completed 100% of protocol summary and session-specific checklists for the first 131 CONNECT visits (63 first visits and 68 subsequent visits). Most nurses completed the checklists using study tablets for direct database entry immediately following the CONNECT encounter. Some nurses chose to bring paper versions of these forms as a reminder of key content and completed them in real time.

Ninety-nine percent of visits were conducted in-person. Fifty-six visits (43%) included the patient only and 75 visits (57%) included the patient and a caregiver. Nurses reported an average length of 46 min (SD 25) for first visits and 42 min (SD 18) for subsequent visits.

Nurses reported delivering most visit components for first visits (mean 94%, SD 13) and subsequent visits (mean 85%, SD 14). For 62% of first visits and 33% of subsequent visits, CONNECT nurses reported delivering all visit components. Table 2 summarizes results from interaction checklists.

Table 2

Percentage of 131 visits (through 5/2/17) that included each intervention component, as reported on CONNECT RN Interaction Checklists.

Intervention Component	% of Visits That Included
	Visit 1	Subsequent Visit
Establish Rapport	98	N/A
Explain Your Role	98	N/A
Set Visit Agenda	98	92
Assess Views About Illness	98	98
Assess Coping	98	98
Provide Emotional Support	98	94
Administer ESAS	98	99
Administer Distress	98	98
Identify Symptom Needs	96	98
Address Symptom Needs	98	99
Discuss Preferences & Goalsa	90	94
Identify Medical Decision Maker	N/A	71
Communication and Decision-Making Preferences	N/A	59
Hopes and Concerns about the Future	N/A	85
Talking with Family	N/A	68
Asking the doctor Questions	N/A	56
Ask About Medical Decision Makers	90	N/A
Complete Shared Care Plans	98	98
Provide Shared Care Plans to Patient/Family	87	89
Advance Directives	79	58
Teach Back	93	88
Review Next Steps	90	79
Total Average Content Score	94	85

Indicates that at least one of the subcategory italicized items required at each visit.

Interventionists audio-recorded 97% (N = 127) of the first 131 patient visits. Four raters (including 2 medical students, a pre-medical student, and a social work student) were successfully trained to review and rate visit audio-recordings by the PI and Nurse PM. Adequate inter-rater reliability was achieved after reviewing an average of 13 audio-recordings.

Table 3 summarizes the results of visit evaluation forms completed for the 45 audio-recorded visits (N = 19 first visits and N = 26 subsequent visits) evaluated to-date, excluding the first two visits conducted by each CONNECT nurse. Each of the 12 eligible nurses had an average of 3 audio-recorded visits evaluated (range 1–9). For first visits, nurses performed an average of 85% (SD 8) of intervention components, with an average total quality rating of 71% (SD 11). For subsequent visits, nurses performed an average of 85% (SD 12) of the intervention components, with an average total quality rating of 75% (SD 7). CONNECT nurses consistently administered the ESAS and distress scales (100% of visits) and explained their role as the CONNECT nurse (95% of first visits). The least performed intervention component was the teach-back (asking patients to repeat back the plan to verify understanding), which CONNECT nurses performed during 32% of first visits and 42% of subsequent visits.

Table 3

Percentage of 45 visits (audited through 5/2/17) that included each intervention component, as identified via auditing visit audio recordings.

Intervention Component	% of Visits That Included
	Visit 1 (N = 19)	Subsequent Visit (N = 26)
Establish Rapport	89	N/A
Explain CONNECT RN Role	95	N/A
Set Agenda for Visit	68	65
Assess Views About Illness	74	81
Administer ESAS	47	62
Administer Distress Scale	100	100
Identify Symptom Needs	100	92
Address Symptom Needs	100	96
Discuss Preferences & Goalsa	N/A	N/A
Ask About Decision Maker	95	50
Discuss Hopes & Concerns About the Future	N/A	77
Discuss Communication and Decision-Making Preferences	N/A	27
Encourage Talking with Family	N/A	27
Ask About Questions for Doctor	N/A	46
Help with Advanced Directive	N/A	19
Complete Shared Care Plan – Symptoms	95	92
Complete Shared Care Plan – Preferences & Goals	89	88
Teach Back	32	42
Discuss Next Steps	95	88
Overall	85	85

Indicates that at least one of the subcategory italicized items required at each visit.

Maintenance of intervention quality

To date, four first CONNECT visits and nine subsequent visits have been below key content thresholds, warranting a discussion with the Nurse PM. Two interventionists have demonstrated more persistent quality deficits or drift in interventionist skills, necessitating supplemental individualized training and review of intervention components. After individualized remediation, both nurses subsequently delivered consecutive visits with adequate (>80%) content scores.

Discussion

As part of a cluster-randomized trial to evaluate the efficacy of a nurse-led primary palliative care intervention, we developed and implemented a rigorous plan for intervention fidelity monitoring and maintenance. In describing the key components of this plan and sharing our assessment tools, we provide a framework for other investigators seeking to monitor delivery of palliative care interventions. Several key findings emerged from our fidelity data.

First, training was effective in having nurses feel prepared to deliver key communication and supportive care skills. This preparation was evident in post-training evaluations and was demonstrated in certification visits, in which all nurses demonstrated intervention delivery with adequate quality. While our certification rate was 100%, nurses reported that certification visits were helpful in providing an additional opportunity to practice and receive feedback in their own oncology clinics before conducting their first CONNECT visit with a patient. Training two nurses per site also proved to be an effective approach. To date, two nurses have moved on to positions elsewhere, but these sites were able to remain active in the study with a single CONNECT interventionist seeing patients until a second nurse could be identified and trained at a later date.

Second, using both self-report checklists and audio recordings is a useful and novel strategy for monitoring and maintaining intervention fidelity. Checklists are simple and effective tools for improving delivery of medical care [14]. Traditionally used in surgical settings, checklists have more recently been developed for palliative care communication interventions [[15], [16], [17]]. However, self-reported checklists may not accurately capture intervention content. In our study, audio-recording intervention visits proved feasible and provided ‘gold standard’ data about intervention delivery [18,19]. Discrepancies between audio-recorded and self-reported content informed opportunities for individual and group feedback. For example, nurses reported conducting a “teach-back” far more frequently on self-report checklists than this element was identified on review of audio-recordings, illuminating a common misconception among nurses about what constitutes a ‘teach-back’ (ie, asking the patient to repeat the plan, rather than simply asking if the patient has any questions). This misconception was the subject of follow-up training and feedback. For administering the ESAS, this was frequently reported by nurses but not captured on review of audio-recorded visits if it was done before the audio-recording was started and not uploaded into the database. This discrepancy provided an opportunity to remind nurses about the importance of uploading the ESAS and referencing this scale during the visit itself.

Finally, nurses who demonstrated deficits in intervention delivery were able to improve after remediation training. We used specific visits that had been audio-recorded to collectively brainstorm ways to incorporate key skills that were missing and reframed remediation training as a positive, non-punitive opportunity for additional feedback. CONNECT interventionists responded positively to the support and additional training and both nurses who underwent remediation have remained active in the study.

Our approach has limitations. First, the 3-day training, while comparable to other palliative care interventions [20], is intensive and may not be feasible at all clinic sites. Future work is needed to determine whether similar fidelity can be achieved with a shortened in-person training. Furthermore, the study includes 14 nurses from multiple clinics within a single cancer center network. Findings may not generalize to other nurses at other cancer center sites. Similarly, the 45 audited visits were not evenly distributed among the interventionists due to the randomization method and time at which data was collected. This may represent a bias in our data. Our intervention fidelity monitoring and maintenance plan involves frequent telephone contact and monthly site visits from our Nurse PM. Future work is needed to determine whether similar results could be achieved with less intensive oversight. Finally, this study is being conducted at oncology practices in Western PA within a single cancer center network. Experiences may differ at other sites.

In conclusion, a 3-part strategy for training interventionists and monitoring and maintaining intervention fidelity is an effective way of ensuring quality ‘primary’ palliative care intervention delivery and will facilitate accurate conclusions about intervention efficacy.

Sources of funding

This project was supported by a grant from the National Cancer Institute (R01 CA 197103 01A1) and a training grant from the National Institute of Mental Health (T32 MH 19986-19). This project used UPCI clinical facilities that are supported in part by award P30CA047904.

Conflicts of interest

None.