Protocol for a type 2 hybrid effectiveness-implementation study expanding, implementing and evaluating electronic health record-integrated patient-reported symptom monitoring in a multisite cancer centre.

INTRODUCTION: Cancer symptom monitoring and management interventions can address concerns that may otherwise go undertreated. However, such programmes and their evaluations remain largely limited to trials versus healthcare systemwide applications. We previously developed and piloted an electronic patient-reported symptom and need assessment ('cPRO' for cancer patient-reported outcomes) within the electronic health record (EHR). This study will expand cPRO implementation to medical oncology clinics across a large healthcare system. We will conduct a formal evaluation via a stepped wedge trial with a type 2 hybrid effectiveness-implementation design. METHODS AND ANALYSIS: Aim 1 comprises a mixed method evaluation of cPRO implementation. Adult outpatients will complete cPRO assessments (pain, fatigue, physical function, depression, anxiety and supportive care needs) before medical oncology visits. Results are available in the EHR; severe symptoms and endorsed needs trigger clinician notifications. We will track implementation strategies using the Longitudinal Implementation Strategy Tracking System. Aim 2 will evaluate cPRO's impact on patient and system outcomes over 12 months via (a) a quality improvement study (n=4000 cases) and (b) a human subjects substudy (n=1000 patients). Aim 2a will evaluate EHR-documented healthcare usage and patient satisfaction. In aim 2b, participating patients will complete patient-reported healthcare utilisation and quality, symptoms and health-related quality of life measures at baseline, 6 and 12 months. We will analyse data using generalised linear mixed models and estimate individual trajectories of patient-reported symptom scores at baseline, 6 and 12 months. Using growth mixture modelling, we will characterise the overall trajectories of each symptom. Aim 3 will identify cPRO implementation facilitators and barriers via mixed methods research gathering feedback from stakeholders. Patients (n=50) will participate in focus groups or interviews. Clinicians and administrators (n=40) will complete surveys to evaluate implementation. We will graphically depict longitudinal implementation survey results and code qualitative data using directed content analysis. ETHICS AND DISSEMINATION: This study was approved by the Northwestern University Institutional Review Board (STU00207807). Findings will be disseminated via local and conference presentations and peer-reviewed journals. TRIAL REGISTRATION NUMBER: NCT04014751; ClinicalTrials.gov. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study uses a type 2 hybrid effectiveness-intervention design, allowing for simultaneous systemwide implementation of a quality improvement initiative and an effectiveness evaluation.

This study is implementing a patient-reported symptom monitoring system that is integrated within the electronic health record and clinical workflows in order to minimise staff burden and promote sustainability.

This study evaluates implementation and impact of an electronic symptom-monitoring programme in one healthcare system, which may limit the generalisability of our findings to comparable high-volume, well-resourced academic health systems.

The stepped wedge study design, while practical and highly acceptable to the healthcare system, limits the ability to mask conditions, as implementers and patients are aware of the change to screening, and a delayed intervention effect in any cluster could reduce power.

Introduction

Context

Advances in screening and early detection, and more successful treatment options, have led to an unprecedented number of people surviving cancer. There are currently almost 17 million cancer survivors in the USA, and that number is expected to exceed 22 million by 2030.1 Cancer is now characterised as a chronic, manageable condition, requiring specific and targeted comprehensive efforts to address long-term challenges and late effects of treatment. Despite advances in early detection and treatment success that extends longevity, survival benefit is often offset by debilitating cancer-related and treatment-related symptoms and psychosocial sequelae that compromise health-related quality of life (HRQoL).2

A growing body of literature has documented the needs of oncology patients, providing evidence that psychological and physical concerns are both prevalent and persistent.3 About 32% of patients with cancer have been shown to meet criteria for mental health conditions.2 4 In a meta-analysis of 70 studies with over 10 000 oncology patients in ambulatory settings, 16.3%, 10.3% and 19.4% met Diagnostic and Statistical Manual of Mental Disorders (DSM 5) criteria for major depression, adjustment and anxiety disorders, respectively; 38.2% met criteria for any psychological diagnosis.5 6 Physical symptoms such as fatigue, pain and poor physical function are among the most common and debilitating reported in oncology settings.7–9 On treatment completion, physical needs (eg, pain and nutrition) are among the top unmet needs.10 Other concerns such as practical needs (eg, transportation, childcare, stress management) are also common.11

In recognition of these challenges, key leadership organisations have prioritised the need to address and embed symptom screening with a referral process in ambulatory cancer care.12–14 This includes standards to better identify, monitor and manage patients’ health needs, including referral to supportive oncology care.15–19 However, work evaluating clinical management and intervention programmes that address the unique needs of oncology patients remain limited and poorly integrated within most institutions.20–23

Related Research

Patient-reported symptom screening has been found to be feasible and efficacious in ambulatory oncology. In a randomised trial of 286 patients with cancer, ongoing monitoring of HRQoL prior to clinical encounters, relative to no monitoring, was associated with better HRQoL over time and improved patient–physician communication.24 Our team has documented high patient compliance (92%) in a technology-based monitoring system developed for patients with lung cancer starting chemotherapy that assessed relevant patient-reported outcomes (PROs) such as fatigue, dyspnoea, cough, weight loss, anorexia, pain, insomnia, change in mental status and psychological distress.25 The majority (69%) of patients felt the questionnaire helped them focus on issues to be discussed with their physicians and, similarly, physicians indicated the reports from the monitoring system helped track and compare symptom burden over time. Recently, a web-based programme that allowed patients to report symptoms to their clinicians was associated with improved HRQoL and longer survival within a randomised single-centre trial.26–28 Patients in the self-reporting arm (relative to usual care) reported greater HRQoL at 6 months postbaseline assessment and had a 5-year absolute survival benefit of 5 months. While promising, most of the work evaluating the efficacy of systematically capturing and addressing PROs via the electronic health record (EHR) remains limited to controlled trials, with limited generalisability to health systemwide application.29 In general, most previous studies evaluating the efficacy of symptom monitoring: (a) are limited in their scalability, generalisability and implementation; (b) implemented measures limited in regards to sensitivity and specificity; (c) did not evaluate the impact of symptom monitoring on clinic workflows or system-level outcomes or (d) did not evaluate or address implementation of the programme as a standard of care.

Preliminary Work

To answer the need for comprehensive symptom assessment that leverages health information technology to reach patients in a feasible manner, we developed and piloted an electronic PRO assessment specific to cancer (‘cPRO’ for cancer patient-reported outcomes) within Northwestern Medicine’s primary EHR, specifically, Epic Systems medical record software (Epic, Verona, Wisconsin).

cPRO development

The cPRO system was custom designed to electronically administer validated PROs from the Patient-reported Outcomes Measurement Information System (PROMIS)30 31 that assess key health outcomes in patients with cancer (depression, anxiety, fatigue, pain interference and physical function) across the trajectory of care and a checklist to identify practical and supportive care needs (eg, financial and transportation concerns; nutritional support). PROMIS measures are administered as computer adaptive tests (CATs), allowing for assessment efficiency and precision.32 Assessment invitations are automated and launched 72 hours prior to scheduled medical oncology appointments (limited to once a month) and completed by patients via the EHR patient portal (Epic MyChart) prior to their visits.33 Results are scored and immediately available in the EHR to inform clinical communications and decision-making. Severe symptoms trigger notifications to clinicians (via the Epic inbox) who can then communicate with patients and make necessary referrals and care decisions in real time (figure 1). Alerts are addressed by clinicians via MyChart, telephone or in-person contacts.

Figure 1

EHR-integrated symptom and needs assessment (cPRO) system. cPRO, cancer patient-reported outcomes; EHR, Electronic health-record.

cPRO pilot studies

Two clinical quality improvement initiatives were conducted to assess the feasibility of implementing the cPRO system as a standard of oncology care at Northwestern Medicine. In the first, 636 women receiving gynaecologic oncology outpatient care received invitations and completed at least one symptom and need assessment through their Epic MyChart portal.34 In the second, 6825 adult oncology outpatients received invitations to complete an earlier version of the cPRO assessment through their Epic MyChart portal; 3526 (51%) completed at least one assessment.33 Assessments were launched 72 hours prior to scheduled medical oncology appointments (limited to once a month). Patients whose symptoms scored in the ‘severe’ range or endorsed supportive care needs (psychosocial and nutrition-related concerns) triggered alerts via Epic in-box messaging to the oncology care team for intervention according to clinic workflow. Together, these pilot studies demonstrated a successful integration of PRO and need assessment administration, scoring and reporting within an EHR system, implemented within a specialised oncology clinic and then more broadly across medical oncology clinics at one geographic site. EHR integration enabled standardised routine assessment and real-time reporting of patient-reported symptoms and needs within ambulatory cancer care, towards the goal of improving care quality and efficiency.

Current Work

The current study seeks to expand on our previous cPRO assessment implementation by expanding to medical oncology patients at all regions of a healthcare system and to patients who do not regularly use the EHR patient portal, by conducting in-clinic assessments using eHealth tools (eg, iPads). We will also conduct a formal evaluation of the programme’s implementation and effectiveness.

Study Aims

Our approach to the expansion, implementation and evaluation of a patient-reported symptom and supportive care need monitoring programme specific to cancer (cPRO) across all adult ambulatory cancer care clinics in a large urban healthcare system (Northwestern Medicine; serving >8000 new oncology patients yearly) is informed by the Framework for Spread35 and the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework.36 Specific aims include:

Aim 1

Expansion and Implementation Evaluation: the Framework for Spread will guide the implementation process to expand cPRO to reach patients at all Northwestern Medicine-affiliated ambulatory cancer clinics and allow for both at-home and in-clinic symptom assessment prior to medical visits. A mixed methods evaluation of implementation success will adhere to RE-AIM36 and its extension to enhance health equity and sustainment.37 The consolidated framework for implementation research (CFIR)38 will be used to assess and characterise implementation determinants at multiple levels of the system.

Aim 2

Evaluation of Effectiveness: we will evaluate the effectiveness of systemwide cPRO implementation on outcomes at the system and patient levels over 12 months via a quality improvement study (estimated n=4000 cases) and a human subjects substudy (n=1000 patients), respectively.

Aim 3

Identify Implementation Facilitators and Barriers: we will identify implementation facilitators and barriers to systemwide expansion of cPRO via qualitative research, gathering feedback from clinicians, administrators and patients participating in the symptom monitoring programme expansion.

Methods and analysis

Overall study design

This 4-year study (anticipated September 2018 to May 2022) uses a cluster randomised stepped wedge trial with a type two hybrid effectiveness-implementation design to test the expansion of cPRO across oncology care practices in a large healthcare system (figure 2).

Figure 2

A 4 year, type 2 hybrid effectiveness-implementation design to test the expansion and implementation of cPRO across oncology care practices in a large healthcare system. cPRO, cancer patient-reported outcomes.

The first aim (aim 1) will focus on the expansion of the symptom monitoring programme (cPRO) using health information technology (configuration and enhancement of technical workflows for the symptom assessment to allow for at-home and in-clinic assessment). Work will focus on the execution of the plan for implementation spread across all network medical oncology clinics using the Framework for Spread, which provides key strategic considerations and goals for implementing a systemwide change that will be evaluated using EHR and stakeholder survey data aligning with the domains of RE-AIM.

The second aim of the study will centre on two evaluations of the effects of implementation: a quality improvement protocol (aim 2a) to compare the impact of cPRO use on EHR-documented healthcare usage and patient satisfaction at the system level, using a stepped wedge design, in which clusters of study sites will sequentially be assigned to cross from serving as a control setting (preimplementation) to implementing cPRO; and a human subjects substudy (aim 2b) with patients who will complete the symptom screener and a battery of measures at baseline, 6 and 12 months to evaluate the impact of cPRO on patient-reported healthcare utilisation, quality, symptoms and HRQoL.

In aim 3, qualitative methods will be used to identify facilitators and barriers to systemwide expansion and adoption of cPRO. Patients will be invited to participate in focus groups or individual semistructured interviews designed to solicit feedback on experiences with cPRO. Key stakeholders (clinicians, administrators) will be invited to complete surveys designed to evaluate key measures of successful implementation (eg, acceptability, appropriateness and perceived sustainability of the intervention).

Study setting

Research will occur at outpatient oncology settings across multiple hospitals that are part of a single healthcare system, Northwestern Medicine. Existing regional clusters (Central, North and West) within Northwestern Medicine serve as the clusters for the stepped wedge trial. The central region includes a single, large, urban-based medical centre; the North and West regions are each comprised of smaller hospitals (two and four, respectively) in suburban communities. All regions include specialty clinics for the diagnosis and management of cancer.

Study population

For the implementation component of this study (aim 1: cPRO administration within clinical care), the study population includes any adult outpatient receiving or clinician (physician, nurse, social worker, dietician) administering, cancer care at a participating medical oncology clinic as well as clinic administrative staff. For the evaluation component of this study, inclusion criteria vary by aim and participant population. For aims 2a and 2b and aim 3, eligible patients must have a confirmed cancer diagnosis and have received oncology services within the past 12 months. Additional criteria for aim 2b, patient eligibility includes recent completion of a cPRO assessment and authorisation for access to the patients’ disease and treatment information in the EHR. For aim 3, participants must have received at least four invitations to complete cPRO. An additional criterion regarding actual number of completed screeners defines focus group assignment: patients who have had one or more cPRO clinical alerts are assigned to participate in an individual interview. For aim 3, healthcare clinicians and administrators must be working at a site participating in the cPRO implementation.

Sample selection

For aim 2a, Enterprise Data Warehouse (EDW) queries of the EHR system will be performed on all cases with a completed symptom assessment within the year prior to the go-live date for implementation across each region. The rationale is to have sufficient data for comparison with our implementation period for both the number of patients as well as across a calendar year to examine the presence of seasonal trends.

For aim 2b and aim 3, patients will be recruited via e-mail invitation from among those receiving invitations to complete the cPRO assessment. For aim 3, clinicians and administrators will be recruited from the pool participating in the cPRO implementation, at the point of their in-person cPRO training and/or via e-mail invitation. Recruitment for both patients and healthcare system stakeholders will happen across regions to ensure representation from each geographic site.

Study measures related to effectiveness and implementation outcomes

The cPRO assessment consists of PROMIS CATs30 31 39 of (1) Depression (PROMIS Item Bank V.1.0—Depression); (2) Anxiety (PROMIS Item Bank V.1.0—Anxiety); (3) Fatigue (PROMIS Item Bank—Fatigue V.1.0); (4) Pain Interference (PROMIS Item Bank V.1.1—Pain Interference) and (5) Physical Function (PROMIS Item Bank V.1.1—Physical Function), along with supportive care checklist items (covering psychosocial and nutritional needs). Cancer centre patients are asked to complete a screener before each oncology visit (but no more than once a month). Data related to cPRO completion, scores and alerts will be used in all study aims, primarily to evaluate the effects of the symptom monitoring system on severity of patient-reported symptoms related to cancer and cancer treatment.

For aim 2a, we will obtain data collected independently from a hospital-based Press Ganey Patient Experience survey40 and a Medallia customer experience questionnaire41 to assess patient satisfaction with their care experience. Patients are provided the opportunity to complete the survey after a care experience (appointment or hospitalisation). Patient-level healthcare utilisation data (eg, clinical notes, emergency room visits and hospitalisations) will be extracted from the EHR to help evaluate effectiveness of the intervention in terms of resource utilisation. For aim 2b, participants will complete questionnaires to assess intervention outcomes related to HRQoL, healthcare quality, symptom experience, financial toxicity, healthcare utilisation (including healthcare utilisation outside of the Northwestern Medicine (NM) health system, shared decision-making and health literacy (see table 1 for Patient-reported Effectiveness Outcomes and Measures).

Table 1

Patient-reported effectiveness outcomes and measures (aim 2b)

Outcome	Measure	Items	Measure details	Assessment
Health-related quality of life: Effects of the symptom monitoring system on patient-reported quality of life related to cancer	Functional Assessment of Cancer Therapy—General—7 Item Version (FACT-G7)65	7 items	The FACT-G-7 is a brief validated measure of patient-reported priority symptoms in cancer; The FACT-G7 has demonstrated internal consistency reliability, convergence and known-groups validity and is highly correlated with the parent measure (FACT-G) total score.65	Baseline, 6 and 12 months
Healthcare quality: Impact of the symptom monitoring system on patient experiences with their cancer care team	Select items from the Consumer Assessment of Healthcare Providers and Systems CAHPS Cancer Care Survey66 67	12 items (if endorsed up to 3 more)	CAHPS is a survey system designed to capture patient experiences with their cancer care team; a rigorously developed, well-tested, reliable and valid survey of patient experiences with their cancer care.67	Baseline, 6 and 12 months
Healthcare utilisation:Impact of the symptom monitoring system on healthcare services used by patients	Custom measure designed to measure assess healthcare utilisation outside of the Northwestern system	3 items		Baseline, 6 and 12 months
Symptom burden: Effects of the symptom monitoring system on patient-reported adverse events related to cancer	Select items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)68	6 items (if endorsed, up to 4 more)	PRO-CTCAE is a compendium of PRO items uniquely targeted to symptomatic treatment-related toxicity assessment in oncology care; Published data substantiates content and construct validity, reliability and responsiveness.69 70	Baseline, 6 and 12 months
Financial toxicity of cancer care	Summary item from FACIT Measure of Financial Toxicity (FACIT-COST)71	1 item	The last (overall summary) item of an 11-item questionnaire that measures personal financial burden of care.	Baseline, 6 and 12 months
Reading ability (component of health literacy)	Single Item Literacy Screener (SILS)72	1 item	A simple assessment of a person’s ability to read and understand printed health material; The SILS ‘performs moderately well at ruling out limited reading ability in adults.’72	Baseline
Shared decision-making	CollaboRATE survey73	3 items	A brief patient survey designed to assess the perceived extent of shared decision-making in a given clinical encounter; The measure has demonstrated discriminative and concurrent validity, interrater reliability and sensitivity to change.73	Baseline, 6 and 12 months

Patients will be asked to complete the battery assessment at baseline, 6 and 12 months via an electronic survey administered using the Research Electronic Data Capture (REDCap) platform.42 REDCap43 is a secure and flexible web application that is available both online and offline, supports longitudinal data collection and allows for data exports to common data analysis packages.

For aim 3, carefully designed focus group and individual interview guides and self-administered surveys, informed by the CFIR Interview Guide,38 will be developed and used to facilitate data collection from clinicians, administrators and patients. To assess the related barriers and facilitators to using cPRO, patients will be invited to provide targeted feedback about their experience with cPRO (ease of navigation and completion, comprehension of purpose and goals, general experience with care team and communication related to the symptom and needs assessment) at one point in time across two waves of qualitative data collection (short-term or long-term postimplementation). Most will be invited to participate in a focus group; patients who had one or more ‘alerts’ will be individually interviewed. For aim 3, surveys with clinicians and administrators, we will assess targeted implementation barriers and facilitators and implementation process domains from the Framework for Spread, including salient constructs of implementation leadership support, implementation climate and sustainability (see table 2 for clinician-reported and administrator-reported Implementation Outcomes and Measures). Surveys will be administered electronically at three time points: baseline (preimplementation) and then 3-month and 7 month postimplementation.

Table 2

Implementation outcomes and measures (aim 3)

Outcome variable(s)	Measure	Items	Measure details	Assessment
Organisational culture (Do clinicians, researchers and staff believe that implementing cancer patient-reported outcome (cPRO) is appropriate and beneficial for the patients, the practice, and themselves?)	Organisational Change Recipients’ Beliefs Scale (OCRBS)74	Five items from the‘Appropriateness’ subscale	The OCRBS has good to excellent reported internal consistency reliability (eg, α=0.89–0.95 reported across several studies)74 and includes item indicators such as ‘the change we implemented was correct for our organisation’.	Baseline (at the point of regional intervention), and 3 and 7 months postimplementation
Leadership support (Does staff feel supported by Northwestern Medicine (NM) leadership to implement cPRO in their practice? andIs the leadership proactive, supportive, knowledgeable and perseverant?	Implementation Leadership Scale (ILS) to assess the degree to which a leader exhibits specific supportive behaviors75	Three items from the ‘Supportive Leadership’ subscale	The ILS has excellent reported internal consistency reliability (eg, α=0.95; Aarons et al) and includes item indicators such as ‘supports employee efforts to use evidence-based practice’.	Baseline (at the point of regional intervention), and 3 and 7 months postimplementation
Acceptability, appropriateness and feasibility(Do physicians, staff and leaders find cPRO acceptable, appropriate and feasible for their practice?)	The Normalisation MeAsure Development questionnaire (NoMAD)76	23 items	The NoMAD is anticipated to have acceptable internal consistency reliability (ie, α≥0.70)77 and includes item indicators such as ‘I can see the potential value of cPRO for my work’.	Baseline (at the point of regional intervention), and 3 and 7 months postimplementation
Training experience related to cPRO (Do physicians, staff and leaders find cPRO training experience effective)	CBH Post-Training Survey78	6 items	The Training Survey is anticipated to have acceptable internal consistency reliability (ie, α≥0.70) and includes item indicators such as (‘the training prepared me for my role in cPRO’).	Post-training and 3 months postimplementation
Sustainability	Clinical Sustainability Assessment Tool (CSAT)-Short Form79 80	21 items	The CSAT includes items related to seven domains perceived by stakeholders to determine sustained implementation. It has shown to be reliable, usable and valid in a pilot study (n=126).	7 months postimplementation

Interventions to be measured

In addition to the cPRO intervention, this trial will measure the impact of a multicomponent implementation strategy. Implementation strategies are the methods and approaches used to support adoption and delivery of healthcare interventions in practice.44 Our approach comprises a number of discrete strategies that target multiple levels of the delivery system, including oncologists, implementation leaders/operational managers, workflows and internal monitoring of use. These include developing stakeholder interrelationships, training and educating stakeholders, engaging consumers, using evaluative and iterative strategies, and changing infrastructure. While the majority of these discrete strategies were prospectively proposed, we will carefully and comprehensively track strategy use over time using the Longitudinal Implementation Strategy Tracking System (LISTS).45 LISTS involves a time-line follow back procedure, in which members of the research and implementation teams meet at least quarterly throughout the project period to report on all dimensions of the Proctor, Powell and McMillen46 reporting standards for implementation strategies being used in the trial. This repeated evaluation of strategies allows for specifying when strategies are modified from their original planned usage (eg, discontinued, changed) using dimensions from the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS)47 and when new strategies are added, either as planned or in response to emergent barriers or effects of other strategies in use. Reporting also specifies with which cluster of the study the strategy is used/modified. This evaluation will aid in the interpretation of the trial results and inform replication efforts in other healthcare systems.

Sample size calculation

For aim 2 a, we plan to include all EHR ‘cases’ with completed symptom screener data during the study period of postimplementation, estimated at 4000 based on data from our pilot initiative. With 4000 cases, we will have the power to compare patient groups by specific trigger alerts (eg, PROMIS Anxiety) versus those who do not receive that alert. All cases will be analysed within the context of the proposed stepped wedge design.48 For aim 2b, we plan to enrol 1000 patients. These participants will be stratified by region, with site-specific accrual goals based on region size (oncology patient population). The primary outcome measures for this aim will be EHR and self-reported healthcare utilisation and patient satisfaction (see table 1 for a complete list of patient-reported outcome measures). We hypothesise that patients with triggered alerts will use more services and will be more satisfied with their care. Assuming 80% power, alpha (two tailed) of 5%, two equal-sized groups and a small effect (0.20), our proposed sample (n=1000) exceeds the required sample size of 785.

For aim 3, we plan to invite all clinicians and administrators involved in the cPRO implementation to complete the survey and expect to enrol a minimum of 40 across cancer clinics. Also, for aim 3, we plan to enrol at least 50 patients recruited from among those completing the symptom assessment. Focus groups will include roughly 6–10 participants, which is considered an appropriate size to maximise group dynamics and information sharing.49

Data analysis plan

Aim 2: evaluate the impact of cPRO

Aims 2 a/2b—descriptive statistics, reliability and CAT analyses. For all PRO measures, we will compute descriptive statistics at the item level (mean, SD, range, skewness, kurtosis, per cent of responses at floor/ceiling), flagging items exhibiting problems, for example, little or no variability. At the scale level, we will compute interitem and item-total correlations, checking for items with low construct validity, for example, item-total correlations <0.40. We will estimate Cronbach’s alpha to determine internal consistency reliability, using alpha ≥0.70 as a standard for making group-level comparisons. For CATs, we will examine how many participants reach the set SE stopping criterion for highly reliable scores (ie, SE <0.30 on the theta metric; reliability ≥0.91). If a substantial number of scores are unreliable (eg, 20% at reliability <0.70), we will investigate whether the issue is score level-specific, that is, associated with very low or very high (extreme) domain scores, which would represent a domain-targeting issue.

Aim 2 a—analysis of cPRO implementation impact. To evaluate the impact of cPRO on EHR-documented healthcare usage and patient satisfaction, we will use a stepped wedge48 (or dynamic waitlist) design,50 in which clusters of study sites will be assigned to cross over from serving as a control to implementing cPRO in a prospective, staggered manner. A priori, we created three clusters corresponding to the three NM regions; order of stepped cluster enrolment will be determined randomly. Our preliminary data analysis will include exploratory data inspection to identify general relationships as well as potential problems with asymmetry, extreme observations or outliers. We will examine differences in patient characteristics (eg, cancer type, cancer stage, demographic variables) to determine the comparability of intervention and control patients. Analysis details. We will use generalised linear mixed models (GLMM) with an appropriate link function specific to a dependent variable’s distribution (eg, an identity link for continuous outcomes; a logit link for binary outcomes). These models will include random effects to account for clusters, while simultaneously examining the intervention’s fixed effects. We will also include in our models covariates of specific interest (eg, age, race/ethnicity, gender, cancer stage/type); however, covariates will not be assessed for inferential purposes. We will run GLMM models to compare intervention to control groups on the primary outcomes of EHR-documented health service usage and Press Ganey/Medallia patient satisfaction, both of which will be treated as continuous variables.

Aim 2b: ePRO longitudinal analyses to explore trajectories of ePRO scores

Analysis 1—individual change trajectories. We will estimate individual trajectories of PROMIS Depression, Anxiety, Fatigue, Pain Interference and Physical Function scores as collected at three time points (baseline, 6, 12 months). We will graphically display the individual trajectories and determine the amount of variability associated with their intercepts (start parameter) and slopes (change parameter). Analysis 2—latent growth curves. Using growth mixture modelling (GMM), we will characterise the overall trajectory of each PROMIS domain score set by estimating its start and change parameters. We will then investigate the potential model impacts of gender, region and cPRO alert status (triggered alert vs did not) by including them as trajectory model explanatory variables in order to answer the following questions: Do gender and/or region statistically significantly (p<0.05) explain start or change parameter variability? Does cPRO alert status statistically significantly explain change parameter variability? In secondary analyses, we will investigate potential trajectory impacts of major clinical status variables such as cancer site. Analysis 3—latent classes. We will estimate potential subpopulations with distinct trajectories per set of PROMIS domain scores. For each identified subpopulation or ‘class’, we will characterise membership by (a) demographic, clinical and regional status and (b) healthcare utilisation and satisfaction with healthcare. Using multifactor analysis of variance, we will test whether characterisation variables statistically significantly (p<0.05) explain identified class membership differences. We hypothesise that (1) cPRO alert status will impact change parameters; (2) geographic region will not impact start or change parameters and (3) there will be sufficient heterogeneity in our studied sample to identify and define two or more latent patient classes per PROMIS domain. Analysis details. GMM assumes the population of interest consists of heterogeneous subpopulations with varying growth parameters.51 A GMM framework allows for unobserved heterogeneity in a sample and different individuals belonging to distinct subpopulations52; it enables identifying and characterising such subpopulations, known or unknown.53–55 Latent growth curve change parameter(s) and model assessment strategies include: (a) assessment of growth curve form, (b) determination of the statistical significance of the mean and variance of linear and non-linear slope terms, (c) inspection of overall model fit improvement when adding non-linear slope parameters, using χ2-based fit statistics (eg, the log likelihood ratio) and their associated p values and the root mean square error of approximation fit statistic.56 A bootstrap likelihood ratio test (BLRT) has been evaluated and appears promising for determining the number of latent classes to model.57 We will, therefore, use the BLRT, as well as the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), and the sample size-adjusted BIC (ABIC), with decreasing values of AIC, BIC and ABIC, indicating improved model performance; we will emphasise ABIC findings.58 We will also use the entropy measure as a summary of latent class classification success (the degree to which obtained latent classes can be distinguished). Its values range from 0 to 1, with values closer to 1 indicating greater classification clarity.59 We will use Mplus (V.8.6) to conduct GMM analyses.60

Aim 3: identify facilitators and barriers to systemwide cPRO implementation

Aim 3—implementation battery measure descriptive statistics and reliability analyses. As with our PRO measures (aims 2 a/2b), for all implementation battery measures, we will compute descriptive statistics at the item and scale level to identify poorly functioning items (see above for details). We will also estimate internal consistency reliability (Cronbach’s alpha); alpha ≥0.70 will serve as a minimum standard for making group-level comparisons. For measures with acceptable internal consistency reliability, we will graphically depict baseline, 3-month and 7-month status for (a) organisational culture, (b) leadership support, (c) use of research in practice, (d) acceptability, appropriateness and feasibility and (e) Collaborative Behavioral Health (CBH) training experience. Although our aim 3 sample size will not support conducting inferential statistical comparisons, graphical depictions of longitudinal status will allow us to visualise overall trends as well as trends within region and clinic. For each graph constructed, we will plot mean and median values per assessment time point as well as individual values per survey participant. We will summarise and share (in aggregate form) group-based descriptive statistics of the healthcare stakeholder survey measures with respondents. Using this sequential mixed methods (quantitative and qualitative) approach for implementation research,61 62 we will next use the implementation survey results themselves (combined with EHR data on patient engagement) to develop focus group guides to obtain additional information on barriers and facilitators of specific aspects of implementation.

Aim 3—focus group data analysis. We will code focus group data using a directed content analysis approach63 that involves use of existing implementation research theory, prior results and empirical frameworks to develop a manualised coding scheme and hypothesise the relationship between codes. Coders will meet to discuss initial thoughts, insights and observations for the development of preliminary coding categories. Through a systematic process, analysts will next begin investigating coded focus group data and refine emergent themes (eg, collapsing redundant themes, removing irrelevant themes); they will develop a coding dictionary for the remaining analyses. They will summarise coded data and identify the most important themes in terms of prevalence and/or impact. Twenty per cent of the focus group data will be double coded in order to calculate coder agreement reliability.64 Disagreements in coding will be resolved via expert consensus.

Patient and public involvement

No patients or public were directly involved in the design or implementation of this study protocol. However, this work builds on prior initiatives informed by the priorities, experiences and feedback of patients and the medical community at large regarding many aspects of the symptom monitoring process, including value and functionality. During the study, we plan to collect feedback formally via qualitative interviews and surveying and informally via ongoing feedback from patients and clinicians actively using cPRO to guide the implementation process and facilitate system adoption. Finally, at the point of dissemination, we plan to include relevant study findings in ongoing communication with current and future patients (eg, via brochures and website information and in personal communication from clinicians) to help define value.

Ethics and dissemination

Ethical and safety considerations

This study has undergone rigorous scientific evaluation via the Agency for Healthcare Research and Quality peer-review process, and the protocol has been approved by the Social and Behavioral Research Panel of Northwestern University’s Institutional Review Board (IRB; study number STU00207807). All study sites fall under a single IRB reliance agreement. All component human subjects’ research has been deemed ‘low risk’, and participation requires signed informed consent of an IRB-approved consent form. The study will not advertently recruit special patient populations, including adults with cognitive impairments, pregnant women, prisoners or other detained individuals.

Dissemination

This protocol details the rationale, design, materials and methods of an innovative quality improvement initiative that permits implementation and expansion of a patient-reported e-symptom screening system in a multisite cancer centre and a formal mixed-methods evaluation of the process and initial results. The outcomes of this study will not only help define the success of enhanced symptom monitoring in improving the quality of patient care and experiences in our study population but may also contribute knowledge to the field of implementation science to accelerate innovation and research. Data will be curated and findings from this research will be disseminated via professional scientific meetings and in peer-reviewed journals to the extent possible. Anonymised data from this study will be shared in a publicly accessible way with the research community at large to advance science and health, including patient-centred care.