Feasibility, usability and acceptance of weekly electronic patient-reported outcomes among patients receiving pelvic CT- or online MR-guided radiotherapy - A prospective pilot study.

INTRODUCTION: The potential of patient symptoms being monitored longitudinally in radiotherapy (RT) is still unexploited. When novel technologies like online adaptive MR-guided radiotherapy (MRgRT) are evaluated, weekly electronic patient-reported outcomes (ePROs) may add knowledge about the symptom trajectory. This study aimed at evaluating feasibility, usability and acceptance of weekly ePRO among patients receiving pelvic radiotherapy.
MATERIALS AND METHODS: In a mixed-methods convergent design, a prospective pilot study enrolled patients referred to pelvic radiotherapy with curative intent. Patients used their own device at home to self-report PRO weekly during and four weeks following radiotherapy and week 8, 12, and 24 (paper-questionnaire as an alternative). Feasibility was extracted from the ePRO software. The Patient Feedback Form and patient interviews were used to explore usability and patient acceptance. Patients were informed that clinicians had no access to PRO responses.
RESULTS: In total, 40 patients were included; 32 patients with prostate cancer and 8 with cervical cancer (consent rate 87%), median age 68 (36-76). The majority did digital reporting (93%). 85% of patients responded to ≥80% of the weekly questionnaires with 91% average adherence to weekly completion (60% for follow-up), although lower for patients ≥age 70. Time spent on ePRO (97%) and frequency of reporting (92%) was considered appropriate. Interviews (n = 14) revealed the application was usable and the patients requested real-time feedback from the clinicians.
CONCLUSION: Recruitment for ePRO during radiotherapy was feasible and adherence to weekly self-reporting high. The digital application was usable and weekly frequency and time spent acceptable. Real-time feedback from the clinicians is requested by the patients.

Introduction

Symptoms may go undetected for patients with cancer treated with radiotherapy, as digital monitoring of patient symptoms is not an integral part of radiation oncology. Many patients are affected by this as radiotherapy contributes to the cure or palliative care of >50% of patients diagnosed with cancer [1], [2]. Even though modern radiotherapy techniques and technologies have reduced the severity of treatment-related toxicity, symptomatic adverse events (AEs) still have a substantial impact on the everyday lives of the patients [2]. They receive their treatment in an outpatient setting with limited time for the clinicians to assess the severity of their acute symptoms and initiate supportive care.

Having patients report their symptoms during treatment has made it possible to detect symptoms earlier and intervene earlier during chemotherapy [3]. Patient-Reported outcome (PRO) engages patients in providing measures of their health status directly without clinician interpretation [4]. When clinicians monitor and use PRO responses it may improve patient-clinician dialogue and patient satisfaction and enhance a focused symptom recognition and assessment [5], [6]. This, as chemotherapy-related symptoms tend to be under-reported by clinicians compared to patient reporting [7], [8].

Improved outcomes have been established when real-time symptom monitoring is used among adult patients with cancer in systemic treatment [9], [5], [10], [11]. Real-time monitoring of PRO allows for timely patient-centered care [5], [12].

Unlike chemotherapy, recording of radiotherapy toxicity is still inconsistent [13], [14]. Studies with patients in radiotherapy found, that patients reported symptoms earlier and more frequently than physicians and a higher rate of patient reported clinically meaningful symptoms was found compared to clinician reporting [15], [16]. In addition to being used in clinical care, PROs are recommended in comparative effectiveness research [17]. A clinical benefit of novel technical innovations in radiation oncology is expected, however, systematic prospective evaluation of clinical effectiveness is scarce [18]. PRO data completes the picture by enabling the provider with real-world evidence of treatment safety directly from the patients [19].

The magnetic resonance-guided linear accelerator, the MR-linac, is an innovative technology providing online magnetic resonance-guided radiotherapy (online MRgRT) combining real-time soft-tissue imaging with radiotherapy [20], [21]. In 2018, the first high field MR-linac was approved for clinical use [22], [23], [24]. A systematic evaluation of this new technology was initiated [21]. To systematically include assessment of PRO in a prospective, longitudinal evaluation of online MRgRT it requires that the relevant symptoms for the specific patient population is identified, using valid PROs, and collecting data digitally when possible [6], [25].

A key challenge when electronic PROs (ePROs) are incorporated in cancer treatment is that implementation process considerations are often not addressed [6]. Previous studies found that the use of mobile apps for symptom reporting during pelvic radiotherapy has been reported acceptable by patients [26], [27]. However, the purpose of incorporating PRO in the specific clinical setting for a specific patient group must be considered carefully. To reduce the risk of PRO not bringing meaningful change to the patient feasibility, usability and patient acceptance of self-reporting must be explored for direct insight into the perceived value for the patients in the specific setting [6], [12], [28].

A few studies have investigated daily PRO in radiotherapy for intensive symptom management [29], [30]. However, a 1-week recall has been found to correspond well to daily reporting reducing the burden for patients in daily contact with the radiotherapy staff [31]. To our knowledge, only a few studies have investigated the feasibility of incorporating weekly ePRO in the course of radiotherapy [26], [32], [33]. None of these studies had the same patient population with pelvic cancer. In one of the studies, patients without an email address were excluded [26]. Two other studies offered patients an alternative option to web-based reporting at home; an automated telephone system [33] or patients being approached with a computer in the clinic waiting area [32]. The median ages in these three studies were 56, 59 and 66 years, respectively. Oncology trials with PROs as primary or secondary endpoint rarely includes a population with median age ≥70 [34]. Therefore, there is a need for investigating an integration of weekly ePRO into the clinical workflow of radiotherapy with a simple setup being feasible for all patients including patients age 70 or above. The current study is part of the PRO-MR-RT study evaluating the trajectory of patient symptoms to online MR-guided radiotherapy (MRgRT). This pilot study aims at investigating feasibility, usability, and patient acceptance of weekly ePRO among patients with pelvic cancer treated with radiotherapy with a curative intent to ensure sustainability in the integration of ePRO in radiation oncology.

Material and methods

Study design and participants

The study was designed as a prospective single-center observational pilot study. A mixed-methods convergent design was applied where the data collection of the survey data and interview data occurred simultaneously in the same period of time (October 2019–November 2020) [35]. All patients referred to pelvic radiotherapy with a curative intent at Department of Oncology in Odense in the study period were eligible for inclusion. The patients were to be aged 18 or above, able to give informed consent and able to read, understand and complete questionnaires in Danish. Patients were excluded if they were taking part in other clinical trials involving substantial completion of questionnaires during their course of radiotherapy. All eligible patients (n = 53) were approached and informed in the department by the primary investigator PKM.

A systematically developed item set with 18 acute symptomatic AEs was used [36]. Data were collected at baseline and weekly during radiotherapy (for 4–8 weeks according to diagnosis and treatment plan) and four weeks following to capture acute toxicity. Follow-up reports were to be collected at week 8, 12 and 24 (Fig. 1). Patient-initiated free-text reporting of symptoms was available at all times. The patients were informed that their responses were not available for the clinicians in the pilot study. A Patient-Reported Experience Measure (PREM) was also included having the patients fill out the Patient Feedback Form on paper four weeks following treatment (±1 week) supplemented by patient interviews.

Fig. 1

Data collection in the PRO-MR-RT pilot study.

Health-related quality of life (HRQoL) was collected according to recommendations for prospective evaluation of online MR-guided radiotherapy [37] using EQ-5D-5L (EuroQol-5 dimensions) [38] and the EORTC QLQ-C30 (European Organisation for Research and Treatment of Cancer QLQ-C30) [39] (Fig. 1). These data are not presented in this publication.

Online platform for patient reporting

The patients had to use their own device and internet access at home to report. If the patients did not have a device or technological abilities for electronic reporting, they were offered paper questionnaires. The patient app and website My Hospital was selected as ePRO application. My Hospital is an app or website for patients at hospitals in the Region of Southern Denmark developed by MedWare. MedWare has no influence on the study or publication of data. The app was already used in the department and the design of the app was therefore pre-defined. The app allows entered patient data to be transferred directly to the clinicians at the hospital in the Electronic Health Record and it contributes with written and visual information about e.g. appointments and treatment. At the time of enrollment, a demonstration and a written guideline on ePRO were provided to the patient. Two push-messages were set up for those using the app to remind the patient to respond the questionnaire. In addition, the patients were offered text messages if they found it hard to remember or used a computer.

Variables

Demographic data on age, marital status, comorbidity and Eastern Cooperative Group/World Health Organization Performance Status (ECOG/WHO PS) [40] were extracted from the electronic health record as well as clinical data on primary diagnosis, concomitant treatment and prescribed radiotherapy dose and fractionation. In a baseline questionnaire, patients responded to questions about educational length, employment status and how frequent they used technological devices.

Outcome measures

The feasibility of integrating electronic acute PRO in the pelvic radiotherapy course was measured with data from My Hospital software complemented by notes on technical difficulties (Fig. 2) [41].

Fig. 2

Outcome measures and data collection methods in PRO-MR-RT pilot study.

To investigate usability and patient acceptance of ePRO, the Patient Feedback Form was used. The form was adapted by Snyder et al. [42] from Basch et al. [43] to measure patient satisfaction with online self-reporting of toxicity. The form consists of 13 items and has been translated, culturally adapted and validated for measuring patient satisfaction with ePROs in a Danish cancer population [44]. In addition, usability and acceptance was also investigated with qualitative semi-structured patient interviews (Fig. 2). The quantitative and qualitative data were analyzed separately and the findings were compared and synthesized.

Patient interviews

Patients were informed about the interview at enrollment. A convenience sampling method was applied interviewing patients in the order they attended their 4-week follow-up continuing recruitment until data saturation was reached [45]. When caregivers accompanied the patient they were invited to join the interview.

The main investigator (PKM) carried out interviews and audio recording. For a wider analytical space, the transcription, data coding and analysis of data was carried out by two research assistants (ZVN, MFB) supervised by PKM. The research assistants were not involved in the clinical work of the department and did not have any contact with the participants.

As previous research has pointed out relevant themes for investigating patient acceptance of ePRO, these themes were selected in advance for the interview guide and the framework of the coding (Fig. 2). The strategy used for data analysis of the interviews was a systematic text condensation in four steps [46]. A deductive approach was applied given the themes were identified in advance [47]. Data not possible to characterize under one of the predicted themes was given a new code to be open for additional themes derived from the data.

Statistical analyses

Descriptive statistics were performed to describe the sociodemographic and clinical characteristics. The consent rate was defined as the proportion of informed patients giving consent. The attrition rate was calculated as the proportion of participants withdrawing or dying from the intervention leaving no data on outcomes available. The retention rate was the number of individuals who remained in the study and responded to the questionnaire in week 24. Patient adherence was the proportion of patients completing self-reports for each time point adjusted for withdrawals and death and the adherence rate as the proportion of participants replying to ≥80% of the weekly PRO questionnaires [15]. Adherence to weekly completion was analyzed according to gender, age (≥70 years), marital status, WHO PS and educational level using the Fishers Exact test. Frequencies were calculated for the categorical data in the analysis of the Patient Feedback Form. A pilot study sample size of 40 patients was established based on the sample sizes from other pilot studies testing PRO integration in clinical cancer therapy [48], [49]. Statistics were performed using STATA IC 15.

Ethical approval

Oral and written informed consent was obtained from all study participants. Approval was obtained from the Danish Data Protection Agency (18/51369). According to Danish Law, no approval was needed from the Health Research Ethics for Southern Denmark (20182000-172).

Results

Between October 2019 and May 2020 41 patients consented to participate; 32 patients with prostate cancer, eight with cervical cancer and one with bladder cancer. Being the only patient with bladder cancer, this patient was excluded from all analyses (Fig. 3). The median age was 68 (range 36–76). Most patients (93%) were comfortable using their own device for electronic reporting, thus three patients reported on paper (Table 1).

Fig. 3

Flow-chart of the Danish PRO-MR-RT pilot study.

Table 1

Characteristics of the study population in the Danish PRO-MR-RT pilot study (n = 40).

Characteristics	All, n (%)		Prostate cancer, n (%)		Cervical cancer, n (%)
Gender
Men	32	(80%)	32	(100%)
Women	8	(20%)			8	(100%)
Age, median (range)	68	(36–76)	69	(54–76)	67	(36–75)
<70 years	21	(52.5%)	16	(50%)	5	(62.5%)
≥70 years	19	(47.5%)	16	(50%)	3	(37.5%)
Cohabitation status
Cohabiting	32	(80%)	27	(84%)	5	(62%)
Living alone	8	(20%)	5	(16%)	3	(38%)
Highest attained education
Basic or high school	6	(15%)	5	(15.6%)	1	(12.5%)
Vocational training	13	(32.5%)	11	(34.4%)	2	(25%)
Short-cycle higher education	4	(10%)	2	(6.3%)	2	(25%)
Medium-cycle higher education	6	(15%)	4	(12.5%)	2	(25%)
Long cycle higher education	5	(12.5%)	5	(15.6%)	0	(0 %)
Not applicable	6	(15%)	5	(15.6%)	1	(12.5%)
Currently working, yes	11	(28%)	8	(25%)	3	(38%)
WHO, performance status
0	30	(75%)	25	(78.1%)	5	(62.5%)
1	5	(12.5%)	3	(9.4%)	2	(25%)
2	1	(2.5%)	1	(3.1%)	0	(0%)
Not applicable	4	(10%)	3	(9.4%)	1	(12.5%)
Treatment data, RT dose/fx
78 Gy/39 fx	17	(42.5%)	17	(53%)	0	(0%)
62 Gy/21 fx	1	(2.5%)	1	(3%)	0	(0%)
60 Gy/20 fx	14	(35%)	14	(44%)	0	(0%)
55 Gy/25 fx	2	(5%)	0	(0%)	2	(25%)
50 Gy/25 fx	2	(5%)	0	(0%)	2	(25%)
45 Gy/25 fx	3	(7.5%)	0	(0%)	3	(38%)
46 Gy/26 fx	1	(2.5%)	0	(0%)	1	(12%)
Online MR-guided radiotherapy, yes	13	(33%)	13	(41%)	0	(0%)
Concomitant systemic treatment, yes	28	(74%)	24	(80%)	4	(50%)
Technological abilities
Web-based reporting, yes	37	(93%)	29	(91%)	8	(100%)
Device at home, yes	40	(100%)	32	(100%)	8	(100%)
Frequency of device usage prior to RT
Several times a day	22	(55%)	18	(56%)	4	(50%)
Daily	17	(43%)	13	(41%)	4	(50%)
Weekly or less	1	(2%)	1	(3%)	0	(0%)

Feasibility

The majority of patients informed about the study consented to participate (consent rate 87%). Patients declining were mostly men with high-risk prostate cancer (83%) with a median age of 73. Not being able to report electronically was not the reason for them declining although 83% had no device for reporting. Three patients left the study; two dropped out during treatment and one died after follow-up week 4 (attrition rate 7.5%).

Overall, 448 of the 554 questionnaires distributed at 12–16 time points were completed (completion rate 81%). Reasons for missing responses were not collected systematically. However, patients explained they sometimes forgot, were too tired, or had too many appointments that day. The average patient adherence to weekly completions was 90.8% but the average adherence to follow-up weeks 8, 12 and 24 was 60.3% for patients still alive and enrolled in the study (Fig. 4). The adherence rate of patients responding to ≥ 80% of the weekly PRO item set questionnaires was 85%. Overall 65% of the patients responded to all 12–16 questionnaires according to study protocol. Nine patients received additional text messages to remember responding.

Fig. 4

Adherence to PRO completion at pre-specified time-points in the PRO-MR-RT pilot study (n = 40).

Adherence of responding to ≥80% of the weekly questionnaires appeared significantly poorer in the group ≥70 years compared to patients <70 years (79% vs. 90%, p = 0.041). No statistically significant differences in weekly completions was found according to gender (p = 0.549), marital status (0.876), WHO performance status (p = 0.717) or educational level (p = 0.683). Approximately half of the patients remaining at the last time point of data collection completed the questionnaire week 24 (retention rate 47.5%).

Technical difficulties

Five patients contacted the investigator for technical support in the pilot study (13%) with problems finding the questionnaire in the app and difficulties responding (technical error on the day).

Self-initiated reporting

Eight patients with prostate cancer (25%) and three with cervical cancer (38%) took advantage of the possibility to report symptoms outside the fixed time points (mean age 63 (range 38–76)). Each patient reported at 1–3 time points and self-reports covered 24 symptoms (1–5 symptoms/day 1–79 days after first treatment). Of these, 15 symptoms (62.5%) were included in the weekly questionnaire. No symptoms were reported by more than one patient and some used it only to write ‘no new symptoms’.

Usability and patient acceptance

37 patients (97% of patients still enrolled) completed the Patient Feedback Form. The patients found the frequency (95%) and time spent (97%) was sufficient and the questionnaire easy to understand (95%) and complete (100%). As there was no clinician feedback on the responses, the majority found ePRO did not improve discussion with clinicians (54%) nor was the information used (83%), communication (78%), or care improved (75%). Despite of this, all but one would like to continue responding (Table 2).

Table 2

Evaluation of PRO-MR-RT weekly ePRO in a Danish pelvic radiotherapy setting (n = 37).

	Response (%)
	Too short	Just right	Too long
1. Time it took to complete		97%	3%
	Not often enough	Just right	Too often
2. Number of times completing		95%	5%
	Strongly agree or agree	Disagree or strongly disagree
3. Easy to complete	100%
4. Completing was useful	100%
5. Easy to understand	95%	5%
6. Easier to remember symptoms and side effects	78%	22%
7. Improved discussions with clinician	46%	54%
8. Clinicians used information for my care**	17%	83%
9.The quality of care improved because of the questionnaire*	25%	75%
10. Communication with clinician improved	22%	78%
11. Made me more in control of care*	64%	36%
12. Recommend to other patients	97%	3%
13. Would like to continue responding	97%	3%

1 missing.

2 missing.

No patients declined to participate in the interview and after 14 patients data saturation was reached as diversity sampling was assessed appropriately. Mean age of informants was 64 years (37–74), three women and 11 men. The caregivers were present in nine of the interviews. They contributed with information about usability and acceptance of weekly reporting and how much the patient needed technical assistance.

For the analysis, a total of 215.53 min of interview was available. The mean duration of the interviews was 15.4 min (range 7–27 min).

Theme 1: My Hospital application

Once the participants had entered the ePRO application, they found it easy to use. Only half of the participants experienced receiving push messages reminding them to respond since some reported on a computer. Overall, the fixed weekday made it easy to remember. Some of the patients requested some kind of feedback whether the severity of side effects they reported was normal, how to act on it and what to expect. All but one participant said that they had no need for advice or feedback from the application, as they preferred discussing their health with clinicians in the department.

Theme 2: The pelvic item set

All participants reported the length of the questionnaire being adequate using 3–20 min on self-reporting every week and almost half added other symptoms in the free-text response option. Many described the content being relevant and did not find it burdensome to respond weekly to questions about symptoms they did not experience themselves.

“…from what I read about it..which side-effects you could get. Then it fit very well into that “ (Male, 66 years)

Theme 3: Initial user acceptance

Providing weekly reports on their health did not cause insecurity in the patients. On the contrary, patients described it as a positive experience and for some patients a feeling of being lucky not to have all the symptoms listed in the questionnaire.

Theme 4: Patient-clinician communication

The participants all talked about having good communication with the clinicians about their symptoms. However, the majority requested some kind of feedback on their PRO responses for it to be meaningful.

“Well, I think I took it for granted that if I replied that I had major problems with my stomach or something, well then someone would grab me and say “hey, we just have to look at that”. I took it for granted. Of course, there needs to be some feedback. Otherwise, it does not matter.” (Male, 63 years)

Theme 5: Caregiver involvement

In the beginning, some of the participants had their caregivers helping them with the technique, however, the majority handled the electronic reporting themselves. Weekly reporting made them discuss their symptoms at home with their caregivers.

Discussion

This pilot study is one of the first studies to investigate weekly PRO reporting from home during radiotherapy in a population with a sizable proportion age 70 or above including patients treated with online MR-guided radiotherapy. The study aimed at and found that it is feasible to integrate weekly ePROs, that the patients find it usable and accept electronic reporting at home. In addition, the study reports that for the patients it matters to have real-time feedback on their weekly responses from the clinicians.

Electronic reporting from home via app or web site was feasible and conducted by all but three patients. We tried to accede patients not using technology by having the possibility of paper questionnaires. Other studies chose to include other solutions for PRO responding or only included those with a smartphone or email [26], [30], [32]. However, the six patients declining, having a higher median age, lacked the resources to enter a study completing questionnaires at all, thus non-participation was not caused by a lack of technological skills.

Reasons for missing data in this pilot study is essential in the planning of the following prospective longitudinal PRO study. First, this pilot study depended on the patients using My Hospital on their own device at home. Adherence to weekly PRO completions in the app was high though no clinician feedback was provided. One reason might be that the app was already well implemented in the radiotherapy department and introduced to all patients. The average adherence to weekly PRO completion was similar to previous findings where the median age was 2–12 years below median age of this study [26], [32], [33]. This, although almost half of the patients in our study were age 70 or older and appeared to have worse compliance to weekly completion of questionnaires than the patients below 70 did. This is supported by previous findings that younger patients tend to use ePRO data capture more [26], [32], [33], [50].

Decreased response rates during follow-up was expected as compliance previously has been found to be higher during active treatment than after the course of treatment [32], [33]. A previous study found the same initial response rate six months post-treatment but collected additional responses that constituted one-third of the total responses via central coordinator backup calls [33]. We chose not to use backup calls for this study as it is time-consuming and we wanted a setup that subsequently would be feasible in clinical practice. Real-time feedback and further retention strategies may, however, enhance adherence in a patient group like this with patients above the age of 70 during treatment and follow-up [51].

Overall, the ePRO application was easy to use for the patients. The patients agreed on the frequency on fixed weekdays and time spent was appropriate. The need for self-initiated reporting outside the fixed time-points was limited, confused the patients and most symptoms was contained in the weekly item set. Thus, the initial user acceptance was positive and some even found it a help to remember symptoms and side effects like previous findings with ePRO in cancer care did [52].

As expected, the majority did not find their quality of care or communication with clinicians enhanced by questionnaire completion like other studies established [10]. To be meaningful and to have the reassurance of the symptom severity being normal, the patients and caregivers in this study found it essential to have real-time feedback. A minority of oncology practitioners have integrated PRO with clinician feedback even though previous studies found that the communication and quality of care could be improved when the patients felt their information was used by the clinicians [10], [52], [53]. In some ePRO solutions today, advice is provided to the patient via the app or website [54]. The patients and caregivers in this study, however, agreed that the feedback should be in the dialogue with the clinicians in the radiotherapy department and not via the application. Unlike the chemotherapy setting having longer periods without clinical visits, where it makes sense that alerts are triggered to the care team, the daily contact between patient and clinicians during radiotherapy makes it easy to make ePROs an integral part of care [55]. It is possible and relevant to monitor severe or worsened symptoms the day after ePRO completion and use the disease- and treatment-specific PROs as a communication tool to potentially intervene earlier and improve the physical well-being of the patient [9], [53].

This pilot study has some strengths worth mentioning. First, this study used mixed-methods to capture both feasibility, usability and patient acceptance. Furthermore, longitudinal weekly PRO reporting was successfully demonstrated in a clinical radiotherapy setting without the clinicians having extra tasks as the patients completed their PRO responses at home on their own device. In addition, caregiver experiences were included in the interviews. This is essential, as caregiver support is important for patient engagement in digital health interventions [56]. Finally, a sizable percentage of patients above 70 years consented to participate making it possible to explore if adherence was related to age.

One potential limitation of the study is the limited number of patients with cervical cancer included. Further recruitment was not possible in the study period; however, the total intended sample size was still reached. Secondly, the deductive approach used for the analysis of the interview data predetermined the structure of the coding framework with the risk of bias. However, the fact that the transcription and coding were conducted by two research assistants who worked with the data without any prior involvement in the interview minimizes the risk of bias as they could suggest other relevant topics appearing during coding. A third limitation is data on reasons for missing responses not being systematically collected. It would have been interesting to explore the barriers for completion during follow-up. Consequently, it is important to look at potential retention strategies and explore this further in future studies.

In conclusion, this pilot study confirmed that it is feasible to integrate weekly ePRO in the course of radiotherapy, thus the adherence to weekly self-reporting was high in a population with a sizable proportion of patients above the age of 70. The digital application and the questionnaire was usable and the frequency and time spent on weekly reporting acceptable for the patients, however, real-time feedback from the clinicians is requested by patients.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.