The administration of patient-reported outcome questionnaires in cancer trials: Interviews with trial coordinators regarding their roles, experiences, challenges and training.

AIMS: To explore cancer trial coordinators' roles and challenges in administering patient-reported outcome (PRO) questionnaires, and establish what PRO-specific training and guidance they received and needed.
METHODS: Eligible cancer trial coordinators experienced with PRO assessment from approved Australian sites participated in an audio-recorded, semi-structured interview (transcribed verbatim). Recruitment continued until data saturation. Transcripts underwent content analysis.
RESULTS: Twenty coordinators participated (professional training: nursing (n = 12), science/research (n = 4), both (n = 4)). PRO administration formed a minor component of most (85%) coordinators' roles. PRO administration challenges included managing 'English second language' participants, participants' companions who attempted to complete questionnaires, burdensome questionnaires, and balancing their duty of care against trial requirements. Coordinators reported inconsistencies in PRO administration, which appeared to arise as a result of confusion and inconsistent or contradictory PRO training. Inconsistencies concerned whether/when they explained the purpose of PRO assessment, which participants they approached to complete PROs, and whether they used PRO trial data to inform care.Coordinators received PRO training from various sources; most commonly study-specific start-up meetings (45%) or from colleagues (30%). Two received no PRO-specific training. Despite the challenges reported, many (55%) felt they did not need further PRO training.
CONCLUSION: Trial coordinators receive inconsistent PRO-specific training and are often unclear how to prioritise different aspects of data quality when faced with everyday challenges, leading to inconsistent methods, missing data, poor quality data, and even bias. Agreement on how coordinators should prioritise the requirements of PRO studies is a necessary pre-requisite for the development of much-needed, consensus-based PRO administration guidelines.

Introduction

Patient-reported outcomes (PROs) provide information about the impact of disease and treatment on quality of life and symptoms from the patients' perspective [1]. Within a cancer clinical trial, PRO evidence may be interpreted in the context of survival and other outcome evidence to inform evaluations of comparative treatment effectiveness, which ultimately may inform shared-decision making and health policy [2].

In cancer clinical trials, PRO questionnaires are usually administered to the patient or trial participant (for the participant to self-complete) by a nurse or research team member known variously as a ‘trial coordinator’, ‘clinical research coordinator’, ‘site coordinator’, or ‘research nurse’ [3]. For the purpose of this paper, we refer to the role as ‘trial coordinator’; reflecting the individual/s appointed at each trial recruiting centre, or ‘site’, who are responsible for ‘PRO administration’: preparing and providing instructions for participant self-completion of questionnaires, responding to participant queries, collecting completed questionnaires, and sometimes for entering questionnaire data; among other trial coordination and data collection duties.

Trial data collection methods related to physical examinations, imaging, laboratory tests, and PRO administration must be standardised to minimise the risk of bias resulting from inter-observer variability [4]. Standardisation of PRO administration methods is also an important strategy to minimise the risk of missing PRO data and subsequent generalisability issues [5]. Key aspects of PRO assessment may easily be standardised, for example, the choice of PRO questionnaire with which to compare treatment group outcomes, and the follow-up time points at which PRO questionnaires are administered. However the extent to which PRO administration procedures are standardised remains unclear. Reviews of PRO aspects of trial protocols suggest that although the basic aspects of PRO assessment are addressed frequently in trial protocols, guidance related to PRO administration procedures is often lacking [6], [7]. It is possible that other forms of guidance, for example, standard operating procedures or staff training have been used to standardise PRO administration methods, however to our knowledge, no previous studies have explicitly examined this.

A recent UK study found trial staff working in various clinical trial populations were dissatisfied with the minimal PRO-specific training they had received, particularly with reference to handling concerning PRO responses or participants who become emotional when completing questionnaires. Only four of these participants worked in oncology, and the sample was heterogeneous in terms of specific trial-related duties [8]. Therefore the extent to which the issues reported in the UK study exist for cancer trial coordinators is unclear.

The aims of our study were to: understand the various roles of Australian cancer trial coordinators responsible for PRO assessment, determine their challenges regarding PRO administration, establish what training and guidance is typically offered to trial coordinators, and determine the self-perceived PRO-specific training needs of trial coordinators.

Methods

Ethics

Human Research Ethics Committee (HREC) approval was provided by The University of Sydney (2014/383), Royal Prince Alfred Hospital (X14-0282), and Royal Brisbane and Women's Hospital (HREC/15/QRBW/475).

Participants

Trial coordinators based at approved Australian sites who were responsible for administering PRO questionnaires in cancer trials and who had at least 6 months experience in the role were eligible to participate. In response to an email invitation sent to trial coordinators at each approved site, volunteers provided written informed consent and were contacted by phone to confirm eligibility and to schedule an interview. The email invitation clearly outlined the aims and goals of the research. None of the participants had previous or existing working relationships with the interviewer. Participants were aware that the study team was comprised of specialist PRO researchers, as the roles and position titles of investigators were described on study information materials. Recruitment of consecutive, interested trial coordinators continued until data saturation was achieved. Participants did not receive any incentives for participation.

Interview methods

Interviews were semi-structured, allowing key issues to be explored in further detail or for clarification [9]. A topic guide was prepared comprised of three sections: (1) trial coordinator roles and responsibilities, including time spent on PRO assessment and other non-PRO responsibilities; (2) general procedures for each key stage and aspect of PRO administration (e.g. consent, discussing PROs with trial participants, challenges with PRO assessment, and forwarding PRO data to the sponsor/central trial office); and (3) training, including what professional and PRO-specific training they had received, their perceived PRO-specific training needs, and preferred formats of guidance. The topic guide was discussed as a team regularly and allowed novel ideas raised in earlier interviews to be discussed in subsequent interviews. Interviews were conducted over the phone or face-to-face (if location was feasible), as per participant preference. Interviews were conducted one-on-one by a trained and experienced interviewer (RMB), as part of her doctoral research. All interviews were audio-recorded, de-identified and transcribed verbatim by an objective, external, professional agency – therefore we did not require participants to comment on interview transcripts. No repeat interviews were conducted.

Analysis

Interview transcripts underwent content analysis; a method enabling identification, organisation, and interpretation of patterns within the data [10], [11]. Content analysis was appropriate for this study because we sought a content-sensitive method to synthesise the experiences, processes, challenges and needs of trial coordinator participants discussed during the interviews, and to quantify the findings when meaningful to do so [10]. We sought to present findings in a descriptive manner, to increase understanding of the trial coordinator role in the context of PRO data collection [10]. We acknowledge that the interview questions were formulated and the data interpreted through a ‘PRO methodological researcher’ lens, informed by our previous research findings, in order to highlight necessary future training topics, methodological and practical areas in need of improvement, and possible strategies to address these challenges. For transparency, we have presented quotes to support our interpretation. We have also highlighted where certain themes or practices were based on only a small number of interviews for transparency and discuss the broader role of trial coordinators for context.

A coding framework was developed based on an iterative process, using inductive (bottom-up or “data-driven”) and theoretical (top-down or “theory-driven”) methods [10], [11], [12], the latter based on past methodological work [5], [8], [13]. Coding was managed using Dedoose software [14]. RMB reviewed and coded all transcripts in depth and DK checked the coding framework and application of codes for 20% of the interviews. Based on team discussions, the codes were organised into categories for presentation. All authors agreed on the final code structure.

Results

Sample characteristics

Interviews were completed between July 2014 and April 2016. Twenty participants were interviewed from five Australian hospitals, two of which were private centres. The mean interview length was 47 min. Participant characteristics are presented in Table 1.

Table 1

Characteristics of the 20 trial coordinator participants.

Characteristic		n (%)
Sex	Female	19 (95)
Professional training	Nursing	12 (60)
	Science/Researcha	4 (20)
	Both nursing and science/research	4 (20)
Cancer trial context	Medical oncology	10 (50)
	Haematology	5 (25)
	Radiotherapy	4 (20)
	Endocrinology	1 (5)
		Mean (range)
Participant age		44 years (28–64 years)
Years' experience	Since completion of professional training	19 years (5–34 years)
	In trial coordinator role	9 years (1–20 years)

e.g. Bachelor of Applied Science or Bachelor of Health Informatics.

Findings

Roles and skills of trial coordinators

General roles

Coordinators described multiple responsibilities associated with trial coordination; commonly including: managing governance issues for multiple trials, consenting participants, reporting adverse events, completing case report forms, ensuring clinicians complete required paperwork correctly, reviewing prospective trial protocols, organising meetings with other trial coordinators, organising patient appointments for data collection (scans, blood tests), managing study budgets and contracts, liaising with sponsors, and preparing low risk ethics applications. Trial coordinators who were responsible for other staff additionally had human resource responsibilities. In one participating clinic, but not at others, trial coordinators (who were all trained as nurses) played an advocacy role, whereby they would attend all the participants' appointments with the oncology clinician, assist and prompt the participants to ask questions at those appointments.

PRO-specific roles

All explained that time spent on PROs was variable, depending on specific trial assessment schedules, length of questionnaires, and the number and types of trial participants enrolled. Seventeen trial coordinators (85%) felt that PRO-specific tasks formed only a minor part of their role. The remaining three coordinators explained that trials they were currently responsible for required more frequent PRO assessment.

“… one of my latest studies, quite a lot of time is spent. I spend probably about an hour with the patient” Participant 07.

Organisational skills and strategies

All coordinators described a need for organisational skills and processes, in relation to PRO assessment. Most had developed independent systems to ensure PRO questionnaires and other assessments were administered according to schedule.

“I have a spreadsheet … a giant list of what's due when and it flags up and I put it in my calendar as well and in my diary handwritten and on the computer … having seven studies and all at multiple time points … quite a lot of things need to be done along with the questionnaires, so being organised is pretty crucial. There's no way you can keep that in the top of your head.” Participant 14.

“I'll look through their schedule and see what time point they're in, review and see if they're due for a quality of life. I'll print it out and get it ready for the patient, (write) their de-identified details or study number …. I tend to leave the date blank because if the patient has a delay or doesn't turn up it just saves me from having to reprint, …and then put it with all the rest of their paperwork that I'll need for that visit” Participant 13.

Empathy for and rapport with participants

Coordinators also described the importance of having compassion and empathy for participants. They noted that developing rapport with participants and having open communication assisted them to achieve high PRO completion rates.

“I think just being clear about what's being asked for and, I guess engaging the patient in that process, being positive, being supportive or being available if they've got any queries, I think that helps. Usually we build up a pretty good relationship with the patients, …so by the time we get to completing the PROs then we've already got a reasonably well established rapport with the patient …” Participant 10.

“The relationship is very important in terms of compliance, I would have to say. It makes a big difference …. I think you just need to try not to be too clinical in that first visit, recognise them as like yourself going through that experience, or your mother or, you know, your brother, you know anybody. They're a person and they've got interests like yourself so, I always find that if you can get sort of like a common grounding or some sort of common interest. It doesn't take a long time to develop a relationship. But if you're there from go, and you're very clinical and you're only talking about cancer and, you know, well you just don't develop that relationship I suppose.” Participant 04.

Adapting to participants' needs

Coordinators also explained the importance of being adaptable and working around the study participants' needs to obtain the required PRO data.

“I think it's just what you know as a nurse. So the ability to be able to identify issues with patients and assist them when that's required. Troubleshooting, I guess …. if you think they're going to have trouble with it, you can look at other methods of assisting them. … they might be struggling with the wording and you could help them with that. It could be timing … If they're not feeling particularly well, reschedule another time” Participant 06.

Inconsistencies in PRO data collection

Discussions regarding PROs at the time of consent

Coordinators typically were involved in the trial consent process, along with clinician investigators; however the extent to which coordinators discussed PRO assessment during consent varied considerably. Most coordinators would inform participants about the need to complete PRO questionnaires throughout the trial, although not all coordinators would go into detail, as they would prioritise the details of the investigative treatment.

“Before you even consent a patient you like explain the … requirements of the study. So they're well aware … they enter the study and then one of the requirements is they do a quality of life questionnaire” Participant 04.

“At the beginning of the consent stage … if there are questionnaires involved in part of the study, um, how frequently the questionnaires are administered, how involved they are, how long we would expect patient to take to answer those questionnaires and um, like, depending on how much information there is at the time and how bombarded the patient feels, and probably we'd show the patients an example of the questionnaires as well.” Participant 10.

“Before they consent we kind of just talk about what's involved with the study, so, you know, that they have bone marrow, or, you know, blood tests and, and we'd also mention that they have, um, questionnaires involved which they, um, would fill out at, you know, certain periods of time, which is basically to see how they're doing on the study, so, I mean it's kind of generalised, we don't go really in deep about questionnaires …. they're more concerned about invasive procedures or if this drug's going to work for them” Participant 20.

However some trial coordinators noted they do not mention PRO assessment at consent stage.

“I would probably leave that up to the PIC [Participant Informed Consent] provided to the patient. We probably wouldn't discuss it in detail with the patient, ‘cause they usually have more questions about the treatment side of things and when, how often they'd be coming in and that sort of thing. Um, it's really only after they've consented that we would provide information specifically about the process.” Participant 11.

Explanations about the purpose of PRO assessment also varied in detail and content. Some explanations were detailed:

“So you say, ‘it important that we have this type of information, for instance, you know this study drug may work, but there might be another one that has the same results, but this one may make you considerably a lot sicker, like vomiting, nausea. So we need to have that reflected in how you're feeling and noted in your questionnaires, so this is why it's important to do these sorts of things'. So give them sort of an understanding of why we're doing the questionnaires, otherwise they're thinking ‘oh god, not another questionnaire’” Participant 6.

Others clearly lacked confidence or knowledge about the purpose of PRO assessment, which was evident by their hesitance in providing explanations to participants. One even believed coordinators were not allowed to speak to participants until the questionnaire had been completed.

“I just hand them the piece of paper” Participant 01.

“'usually I just find them in the wait room … and just, first thing, just hand them the questionnaires, a pen, and a clipboard or the site pad and just say—um, ‘cause it's really, really hard not to ask how someone is, ‘cause it's just the polite thing to do.”

“Sometimes I might just act busy and say, “I just need to, could you just, ah, fill out this questionnaire first and I'll be right back”. That's my trick . Um, ‘cause otherwise I feel very rude” Participant 15.

Timing of PRO assessments

All interviewed trial coordinators understood that ideally the questionnaire should be administered before other trial procedures, however the time at which coordinators reported administering questionnaires varied for both clinic- and patient-related reasons.

“The clinics will ring you and say that (the patient has) arrived. (Depending on) how many patients you have on the day, if you're in another clinic seeing another patient, they might get in to see the doctor before you get back over there to see them.” Participant 09.

“It's easier for them to have other procedures like a blood test done first because that's at a different area - closer to where they enter the hospital … you don't want to make a patient walk hundreds of metres back and forth just for the reason of having this PRO done before a blood test.” Participant 16.

“Say if they're coming for a whole day of chemo, we usually wait till they're actually settled in (treatment) and everything's a bit calmer, ‘cause they're a bit stressed as they come in, not stressed but, you know, there's a lot happening … when they're settled in the seat and calm, that's a good opportunity to give them the quality of life.” Participant 12.

One coordinator explained that she worked collaboratively with the clinical team to ensure PROs were completed prior to appointments.

“I'll get the clinic to call me when the patient arrives so I'll go and see them first …. The doctors know I'm there to see the patient with them anyway … So they'll say to me, ‘are you ready to see this patient now?’ … If I'm still doing things, I'll (ask) ‘can you just see someone else first?’.” Participant 10.

One coordinator posted questionnaires to trial participants ahead of clinic visits for participants to complete at home. This PRO administration method differed to what was explained in the trial protocol. The coordinator felt her process was more streamlined and that it assisted in reminding participants about their appointments.

“I do that of my own accord because as well as getting the questionnaire back, it also reminds the patient that they've got a visit coming up” Participant 17.

Checking questionnaires for, and following up, missing data

Another area where administrative inconsistencies were particularly evident related to checking completed questionnaires for missing items. Some coordinators did not check completed questionnaires at all, believing they were confidential based on past training. Among those who checked completed questionnaires, the time at which questionnaires were checked seemed to vary:

“Routinely I will always look at it afterwards, I suppose that's experience - having had them come back before and them not being completed - you're more wary the next time to make sure that they're completing them … If (the patient is) still there I can go back and say, “Oh, you forgot this page”.” Participant 03.

“… when you're recording it is when you sort of pick it up … It can be days, it can be a week (later)” Participant 09.

Some acknowledged that procedures were trial-specific.

“… it would depend on the trial because sometimes the quality of life forms are confidential and the patient can seal them up in the envelope after so you don't generally look at them.” Participant 03.

Whether coordinators would follow up missed PRO assessments varied by the reason assessments were missed. If participant appointments were rescheduled, coordinators would generally administer PROs at the revised date. However if the trial coordinator forgot to administer the PRO, generally they would not follow up.

“If I just forget, which is easy enough to do, I simply document that I forgot to administer it to the patient and usually just it's noted and I make sure to do it at the next time point … none of the studies I've ever done have said, “… just send it out to the patient and the patient can fill it out and send it back”, ‘cause that sort of negates the purpose of doing it before the review, because the answers will be completely different.” Participant 13.

Others explained they would strive to achieve 100% completion rates at all costs. One coordinator's strategy was to inform participants of the value of PRO assessments, and of trial participants' important and active role in research:

“… they're actually not a subject in the study, they're a participant … their input is crucial … I sort of say, “Well it'd be sad not to have this information after you've been on the study for two and a half years because it's making it less robust. If we don't receive this information, (it's) less likely to be used and I'd hate for you to have wasted your time over a few years.” I wouldn't put it quite a bluntly as that but, you know, you sort of get the picture that it’s a whole big picture and all these pieces of the puzzle are important … If they know how important the information is, they're more likely to contribute … they're not just a number in a box in a spreadsheet, they're actively part and parcel of a study ‘till its completion.” Participant 14.

Use of PRO trial data to inform clinical care

Coordinators described inconsistent practices regarding whether questionnaires were checked for responses that may be concerning or require clinical action (e.g. high anxiety scores, high pain scores), commonly referred to as PRO Alerts [13]. Four (20%) coordinators had worked on trials where formal PRO alert protocols were in place, meaning that coordinators were asked to check completed PRO questionnaires for certain item responses exceeding a set threshold and were required to follow an established action plan in response. Another coordinator noted that the central coordinating office determined whether patient responses necessitated a response and would contact the trial coordinator upon reviewing the uploaded PRO data to ensure the treating clinician was informed.

Five coordinators (all nurses) explained they would check completed PRO questionnaires of their own volition,

“… I'll discuss that with the patient and say, “You know I really think the doctor needs to know about this”.” Participant 19.

“I'll flick through and just follow up anything that's really obvious, like that they've got, they didn't sleep or that they've had really bad nausea” Participant 11.

Challenges faced by trial coordinators regarding the administration of PROs

Concerning PRO data

Managing concerning data, with or without guidance, was a major challenge for coordinators. One coordinator who had worked on a study with formal PRO alerts in place noted that initially she felt insufficiently trained and ill-equipped to implement the procedures. However discussions and troubleshooting with the central trial office led to protocol amendments to clarify the procedures and to involve appropriately trained health professionals to address any arising PRO alerts.

“Their instructions in their trial booklet said that the study coordinators were supposed to then try and delve deeper with the patient as to ascertain whether this person was truly depressed … I said, “Well, that's beyond our clinical scope”, you know, we don't have training for this. I said I would take that question up with the investigators here, because we wanted to participate in this study, but I certainly wasn't going to be counselling somebody on anxiety and depression. Um, so our solution was that if a patient answered anything other than “Not at all” to the two questions that were at the end, and the “Better off dead” one was one of them, that I would have to refer the outcome to the clinician … and make a note into our electronic medical record system … This is how it has been managed … our department policy that we came up with, which is—study coordinators, that's not something that the study can ask for you to do, and it should be clinically, you know, paid attention to. So we always have to wear our clinical hat, even though I'm not a radiation therapist, I'm not a nurse, I'm not a clinician, um, we still have a clinical role.” Participant 02.

Other coordinators expressed uncertainty as to whether it was part of their duty of care to check completed questionnaires for this type of information, or whether checking would violate the trial participant's privacy, as they had not received clear guidance on this point.

“I think some (coordinators) think, “Well, should I tell the doctor if I'm not really supposed to be looking at (the completed questionnaire), it's supposed to be completely confidential”, but, um, sometimes it is important that these are brought to the attention of the treating physician.” Participant 17.

Using PRO data in clinical care

Many trial coordinators also felt that PRO data should be used to manage trial participant care, as not doing so represented a waste of valuable information and caused burden to trial participants as they would then need to repeat their concerns verbally.

“When it's on a paper form we can use that as a tool to help guide the patient with their review appointment with the doctor … When we're using the electronic diaries we don't have that option of seeing their responses and so we can't use that as a tool to guide, or help the patient to discuss how they feel with a doctor. ” Participant 16.

“When I was a grad nurse, many, many years ago, um, a medical student came to see one of my patients … and was in with the patient for about an hour for an assessment and then they left. And I went in to see my patient and he had really bad chest pain … he was all clammy, and I said to him, “Why didn't you buzz me? You should've let me know”. His ECG was really bad. And he said, “I let the medical student know”, and so this medical student had picked it up as part of their assessment and then just left … And this patient just assumed that because they'd provided that information they didn't have to be discerning about how they provided it all, where it went, do you know what I mean? They'd let somebody know and that that was important enough for someone to act on. So I've always remembered that experience and been mindful of patients sharing information …. you might be the only person that they've let that know … they shouldn't have to keep telling people in lots of different ways if something's going on … and also they're also managing a lot of information, so they may not remember again until that night when they get home and go, “Oh, I forgot to tell the doctor about that”, but it's been represented on the PRO anyway.” Participant 11.

Needs of the trial vs the patient

The challenge of balancing the needs of the trial versus the needs of trial participants arose frequently, highlighting the need for some flexibility in procedures. Although coordinators did their best to adhere to specified procedures, they acknowledged that the needs of the participant would often necessitate flexibility:

“We're talking about oncology patients here. You've got to give them some leeway … They will often do (the questionnaire) if they can, but if they're really unwell and deteriorating, you're not going to push that.” Participant 06.

“I tell them, “Look, if it ends up being too much you're welcome to drop out of the study, you can't be forced to stay in it.” … But then we always sort of say, “Oh if you don't want to come in, would you mind still filling in the questionnaires?”, or something like that. … if (patients) were given the opportunity maybe to drop out for 6 months and say, “Look, could we maybe pick this up again in 6 months see how you feel or 2 months' time?”, something like that, they might be feeling better about things and be happy to recontribute later … if it's a long study and, you know, it's asking for a huge commitment, you've got to try and be flexible. That can help with the patient I think.” Participant 14.

“you're not going to pass or try to get someone to complete a questionnaire if they're crying or, you know, not in a physical state to complete a questionnaire, but, I don't really judge, because you don't know what's going through that person's mind, they just might be looking dull, but happy to complete the questionnaire, so I don't judge that.

Unless it's something very obvious, and, you know, don't go there … if a person is, say, looking physically distressed and has expressed that they are physically distressed and usually in that case, they are taken in to see the doctor immediately. You're not going to stop them and say, “Hey, can you please complete this quality of life now?” No.” Participant 05.

“'there's times that I've had to decide not to give the quality of life to a patient … and I'd let the monitor know. The patient may have progressed and they're distressed or there's some psychological sort of stuff going on there that the doctor's aware of too.It's only happened a couple of times … I'll make a notation that the quality of life wasn't attended on this visit … The patient's always your priority” Participant 04.

Others admitted to not offering questionnaires to patients if they felt trial participants were too sick, yet the participant was still willing to complete the questionnaire.

“Occasionally I've had such a patient, “Look, we're not going to do the questions today”, and a patient has surprised me and said, “No, that's ok, I'll do them” Participant 13.

Participant and trial coordinator burden

Trial coordinators also described challenges related to PRO study design and procedures. Coordinators noted that long, repetitive questionnaires were burdensome to both trial participants and themselves. Participants would often question the need for, and in some cases refuse to complete, questionnaires due to their excessive burden.

“I do find it annoying that there are some studies that ask for them every week … especially if it's not two or three pages. Some of them can be, like seven or eight pages long and when you've got really sick people, it's completely, I mean, they just in the end refuse.” Participant 17.

““Is this really of any use?”, “Do I have to complete this questionnaire?”, “What does this really mean?”, “Why do I get this question three times?”, you know, all those questions that I've had participants ask me” Participant 02.

“Simple, straight-to-the-point, don't make it too hard, patients are going through enough.” Participant 04.

“Why is it too long?”, “Why do they have to answer it every time? It's the same questions every time.”” Participant 08.

Other procedural aspects that burden trial coordinators included instructions for copying questionnaires or returning questionnaires to the central office.

“Then they (sponsor) give them to you double-sided. … if you want to send it out, send it out single-sided because people only look at one side of the form. Unless it's a booklet. And then if it's a booklet, it's a pain in the neck to photocopy. If you can just put that stack of forms in the top of the sheet feeder, hit the copy button, it's so much simpler than raising the lid and closing the lid, and raising the lid … [laughs] you get my drift?” Participant 02.

“The more standardised the approach the easier it is.… I think we've got 10 recruiting studies, if they all have the same (procedures) it's so much easier to get it all right, rather than this one does this, and this one does that, this one wants you to photocopy and send, this one says scan and email is fine” Participant 02.

Challenges with question wording and interpretation

Coordinators noted challenges with the questionnaires themselves. They noted that questions frequently skipped by patients often reflected a poorly worded or unclear question. Often, trial coordinators are not given any standardised guidance regarding how to assist participant queries about question wording, which coordinators felt directly impacted the data collected.

“I think sometimes the questionnaires are designed by people who don't have a lot of patient contact. Sometimes you need to highlight things. Like for example, you know, this is, “We want you to complete this as you how you've been feeling in the last 7 days”” Participant 17.

“I usually say …”question number such-and-such has been missed, you haven't given a response, is that because you weren't sure how to answer it, or you didn't want to answer that question?”, and they go, “Oh geez, I didn't see that one”, or, “Nausea, what does that mean?”” Participant 02.

“If they ask a question it'll generally be about their interpretation of the wording, or, you know, if they've got it right? So I just say, “Whatever your interpretation of the wording is, that's the right answer” “ Participant 05.

Trial coordinators were also challenged by questionnaires describing general issues which may not be related to cancer, as these questions often confused trial participants.

“they may say “It's asked if my legs feel heavy, well, that's got nothing to do with my cancer, so how do I answer that?”, and I go, “Well, the question asks if you have heaviness in your limbs. Do you feel like you have heaviness in your limbs? You have to answer it; it doesn't say do you have heaviness in your limbs because of your cancer. So you've just got to answer them as the questions are posed.” Participant 02.

“(Patients) get a bit worried ‘cause they're thinking, “Well, I'm not getting pain from this cancer … I'm getting pain ‘cause I have a bad back anyway” Participant 20.

Participants comments on questionnaires

Two coordinators noted that participants often write comments on questionnaires, and were unclear how to handle these comments based on training and guidance provided to them. One of these coordinators explained how she handled a particularly challenging scenario:

“ (the questionnaire was) asking kind of general questions about your health and (the patient) wrote comments all over it, “How is this supposed to make me feel better?”, “Why is this important?” … she hasn't really answered any of the questions. … [at the next visit, after this coordinator explained the purpose of PRO assessment] she continued on trial and filled the rest out.” Participant 07.

Provision of assistance to participants

Trial coordinators also described challenges in providing assistance to participants. The nature of assistance often needed varied. Interviewees reported that some participants experience difficulty understanding response scales, whereas others simply require assistance reading questions due to problems with their vision.

“I might have to explain to some people you need to circle a number that best suits you now, um, or they need clarification on, some of them are in the last week and some of them are within the last 24 hours. I think when it comes to health care people like to think about way beyond those dates and that doesn't stick, so, clarifying those two points” Participant 13.

One coordinator believed reading questions to participants was more transparent and objective than having patients independently self-complete, as this process would assure her that participants understood questions as they were intended.

“..they give you their answers straight away and you write it down … They haven't misinterpreted the way you've said it.(Otherwise) they might just tick all the same boxes and might not have read it properly” Participant 06.

Coordinators described challenges with e-PRO technology.

“sometimes they don't work and sometimes you think they worked and then they didn't work.” Participant 09.

“… particularly again my older patient population they're just not familiar enough with the technology, and (ask), “How am I meant to do this? Which button do I press next?” Um, that's (happening) less and less, but it does just occasionally happen and I just sort of sit with them … I try to give as minimal assistance as I can so I don't influence them at all.” Participant 13.

Language and translation

Another common challenge for coordinators was assessing PROs of trial participants whose first language was not English. Coordinators' strategies for handling this challenge varied considerably, and included attempting to explain the meaning of words in English, excluding non-English speakers, using validated questionnaire translations, and using interpreters.

“They speak English and they can understand English, but when they're reading something and some of the questions, the way they're worded, it trips them up a little bit, they don't quite understand what that word means, and so I'll explain … what that word is, in the context of the question.” Participant 13.

“If they're not understanding, you can word it in another way, …you know, (rather than) saying ‘we want to know your pain scale’, say ‘we want to understand, or did you have pain? This one being more hurt, this one being less hurt’ you know. ‘where is your pain?’ so yeah, and then they sort of understand.” Participant 06.

“We've got one Chinese lady who doesn't come with anybody and doesn't understand anything about it, and we don't have translated quality of life for her, so she just doesn't do it …. I mean, we have enough trouble getting her, understanding you know, just basic stuff.” Participant 09.

“when you meet them the first time for screening or stuff, so you identify whether they'd be comfortable doing it in English or, like, Mandarin or Cantonese so, yeah, that's what I try to identify before we start, so I can organise it or I can have, I can ask for a translated questionnaire from the sponsor.” Participant 08.

“that probably wouldn't worry me personally because of my nursing background, we use interpreters and would get interpreter service” Participant 09.

One coordinator explicitly sought guidance on this issue for a specific trial and was not given an answer.

“I raised that question that, what if our patients are not native English speakers? How do we deal about that? And they don't want us to use some paper questionnaires, they want it specifically as electronic diary. Yeah, they didn't give me a direct answer on that as well.” Participant 08.

Participants' partners or carers often pose a challenge for coordinators if they attempt to complete questionnaires on behalf of the trial participants (as a proxy reporter; i.e. “reports by someone who is not the patient responding as if that person were the patient” p25 [1]) in trials where self-report was required. Coordinators noted that often carers will try to influence participants' responses, believing they know better:

“The non-English speaking is another whole category … that's a problem as far as there's quite a lot of relatives that come in and do it with them … which I think's fine … except of course when you get the wife that does it for the patient and the wife says, “But I know what he feels like and I'll do it”, and they tick it all, and that happens quite a lot. Um, and if it's not the wife it's the son or somebody or other who speaks better English.” Participant 09.

“I had a really persistent patient's wife … I had to actually take (the questionnaire) off her and say, “Actually there's no wrong answer on the questionnaire, and you (the patient) just need to fill it out the best YOU can, um, on your own” … And ah, she said that she just wanted to make sure that he was doing it right … That's why she needed to answer it, and then, “So actually there is no right or wrong you just need to, um, fill in what you can”. And ah, so she kind of, she sat away but she's very, very keen to get in and just do it for him.” Participant 15.

“ I find, um, male patients especially older male patients always seek their wife's help … It's very hard to try and tell them that they need to do it themselves ‘cause it seems as though their wives do all of the talking … and the site pads that can be quite hard with elderly patients, they usually get their wives to do that too. ” Participant 15.

How the PRO data will be used

Two coordinators explained that feeling as though PRO data collected in previous trials had not been adequately analysed or utilised presented a challenge for them.

“But if they had looked more clearly at the QOL forms in this particular study that I'm thinking of, and I'm not going to say what it is, if they had utilised this tool, which we had collected at all these time points over the 24 months, they would have seen that yeah, their response rate to their survival with their disease progression and their free survival was exactly the same, but they could have used the QOL forms … that showed that their nausea was less, their vomiting was less and their actual QOL was a lot better on this particular drug. I think it would have been marketable. … Because everything else from a haematological point of view … bloods and chemistry, it was all the same … but the way that this particular drug caused a lot less nausea and vomiting, that's got to have improved QOL. Now they could really have marketed this (drug) with that.” Participant 06.

“… the sponsors stopped doing it, say after 2 years, although the study continued, they (stopped requesting) the quality of life outcome … The study started at 2000 … and that drug is marketed now … and the patients were asking, “Why aren't they collecting this information?” They should be collecting data all the way through.” Participant 19.

Back-up personnel

Another key challenge described by three coordinators was managing PRO assessments for colleagues who were absent, or having colleagues fill in for them whilst they were absent. Whilst some sites had procedures in place to train more than one coordinator to administer each trial, others did not, resulting in fill-in trial coordinators struggling to meet the needs of these trials.

“We have at least two people that can back-up on the studies and that's to cover for, like, if I was sick and then the other person was on annual leave then hopefully the third person would be here …. we prepare all the study packs. So if I had three patients due in while I was away, for instance, I'll just make sure that was all done and we handover so, we allocate the person that's going to look after the patient on that day, um, and then we handover to them prior. We tell the patient as well, you know, that this person's going to be seeing you on that day, um, so it's all handed over officially and the staff member that's taking over for me knows and the sponsor for the study knows that this person's the contact person while I'm away, so, it's all, it's all worked out in advance.” Participant 20.

“… I helped out on the study, it wasn't actually my particular study, um, we just didn't fill those (questionnaires) in at those time points.…There was probably some (guidance) in the site file but like I said, I was help—I was holiday relief for that study so I, no, I didn't get any (training) … (I was looking after the trial for) 4 weeks … I just conferred with other colleagues here and left it for her to sort when she got back, basically.” Participant 18.

PRO-specific training and guidance

PRO training received by coordinators

Trial coordinators received different levels of training about PROs. Two reported not having ever received PRO training, 2 attended PRO-specific training courses whilst the remaining 16 noted that PROs were addressed at non-specialist training events, such as study start-up meetings (n = 9), or training at trials group meetings (n = 3), as summarised in Table 2. Often trial-specific training was received through industry-sponsored trials, and less often through investigator-initiated trials, which typically had smaller budgets. The most valuable aspects of training reported by coordinators were communications skills (n = 8) and PRO administration or practical aspects (n = 7), which helped improve their day-to-day practices and address common challenges, such as patient carers completing questionnaires for patients:

Table 2

Training received and perceived training needs of 20 trial coordinators.

	Nursing background n = 12	Research Trained n = 4	Both n = 4
Training received - source
PRO-specific training course	1	1	0
PRO training covered in other coursea
Study-specific training	5	3	1
Training from colleague	3	1	2
CCTG training	0	2	1
Communication training	1	2	0
Good clinical practice training	2	0	0
Degree	1	0	0
No PRO training	0	2	0
Most helpful aspect of PRO training receiveda
Communication with patients	5	2	1
Practical/PRO administration	4	2	1
Theory (purpose of PRO assessment)	2	1	1
Database (e.g. data entry)	0	0	1
None expressed	1	0	0
Perceived training needs
Theory (purpose of PRO assessment, application of PRO results)	2	0	2
Concerning PRO responses (e.g. high self-reported anxiety)	1	1	0
Providing assistance to trial participants	0	1	0
Communication	0	0	1
English second language participants	1	0	0
None	8	2	1
Preferred format of PRO-specific guidancea
Face-to-face or over the phone	2	1	2
Online modules/downloadable presentations	4	2	0
Written guidance (e.g. protocol, coordinator manual, summary sheet, etc)	9	3	1
Start-up meetings	2	1	0
No preference	1	0	1

Trial coordinators may have noted more than one preference or category.

“Historically I used to say, “Oh, they've been naughty”, you know, just joking, “Oh you can't, you can't get involved” you know, “the (responses) have to be just from her”, … we did some role playing work with the communication (training) and the person who was playing the part of the carer felt disregarded when we said those sort of words to them, so that's when we worked out a different strategy - to say, you know, “what you're saying is very important, but, um, at the moment just for the purpose of this form, Mr. X has to fill it out all by himself but, um, I really want to make sure that you mention it to the doctor, because that's very important feedback”, you know, that sort of thing.” Participant 12.

Seeking guidance when unclear on procedures

Trial coordinators explained that observing or troubleshooting problems with colleagues was a less formal training approach, but helpful for them in addressing everyday issues.

“We'll talk to our other sites that are doing that same trial and say, you know, “Having difficulty with such-and-such, have you got any work arounds or how are you managing that?” Participant 19.

“If I really get stuck I'll go and speak to my manager … or the lead investigator and say, “Right, look, I'm having a bit of trouble with this … and talk it through with them because, you know, they may have had a bit more experience with this, depending on what their interests are.” Participant 13.

Others noted they would contact the sponsor directly if they were unsure how to handle a certain issue or situation regarding PRO assessment. One coordinator suggested that the coordinating centre should occasionally check in with trial coordinators to determine whether procedures were clear.

“… I've been meeting with (Project Manager) every so often—just to smooth things out before we start … It's always, “What do we do about this? How can we do that?” ” Participant 08.

“I would not hesitate to ring the monitor and have it clarified …. I guess that it takes up a bit more time. If everything's clearly written in your protocol then you don't have to do that.” Participant 04.

Self-reported training needs of trial coordinators

Despite the large number of challenges reported by trial coordinators, over half (n = 11, 55%) felt they did not need further PRO-specific training or guidance (Table 2). The desired training topics spontaneously raised by coordinators most frequently included the theory and purpose of PRO assessment (n = 4, 20%), and handling concerning PRO data (n = 2, 2%). Regarding the latter:

“Unless we're given training, um, you know, it would be hard for us to work out, you know, where that trigger point would be .. I guess it's all about comparing to other responses they've given … to see if there's been a big change or something. I certainly don't feel like that's within our (capabilities) …” Participant 12.

One coordinator suggested that training for PRO administration needed to change focus:

“I think that the training that we receive is all about the paper work and the boxes and making sure that there's an answer there, um, and not teaching people how to assist the patients into giving those answers for themselves.” Participant 19.

Coordinators explained it was often difficult to attend full-day or off-site training sessions as they would need someone to cover their trial coordination responsibilities during their absence or they were too busy to take leave. Preferred formats of training and guidance differed, with 13 (65%) preferring written guidance, 5 (25%) preferring face-to-face training, 6 (30%) preferring online modules, and 3 (15%) preferring trial-specific training at the study start up meeting, as shown in Table 2.

Discussion

Relative to their other trial responsibilities, trial coordinators generally considered PRO administration a minor but important component of their broader role. Most coordinators felt confident to administer questionnaires, yet described a range of PRO-specific challenges and inconsistencies in day-to-day practices which can adversely affect the quality and value of PRO data. Coordinators had received contradictory guidance regarding PRO administration and expressed confusion about how to respond to certain challenges, particularly in situations where they perceived a discord between their duty of care and the prescribed trial procedures. Although most (90%) coordinators had received some PRO administration training, there was a lack of consistency in the topics covered, recommended procedures, training sources, and in the training of back-up staff.

Coordinators described differences in the types of participants they approached to complete PROs, whether they offered participants assistance, whether they pursued missed PRO assessments, and how they managed low-English speaking participants. These issues have been associated with missing PRO data [5] or selection bias [4] in previous studies. Some coordinators deviated from scheduled timings of PRO assessment, which may lead to uninformative or poor quality data if the timing of administration fails to capture the impact of treatment (e.g. toxicities, palliative benefit), or if the participant receives information about their treatment outcomes before PRO completion, causing the participant to interpret their outcomes and respond differently. Inconsistencies in how, when, and whether coordinators explained the purpose of PRO assessments were also reported, which could cause trial participants to incorrectly assume their PRO responses inform their care, or to question the importance of completing questionnaires. This raises ethical concerns as it demonstrates that some participants are not fully informed about the nature of their research participation.

Similar to previous research [8], some coordinators reported using PRO responses to inform clinical care, even in the absence of guidance. This may be problematic if off-protocol interventions (e.g. psychosocial referrals) are administered based on PRO responses (e.g. high patient-reported anxiety) for some trial participants and not others, without being formally recorded, where these co-interventions impact the patient outcomes assessed as part of the trial (i.e. co-intervention bias [8], [15]). Other coordinators reported uncertainty as to whether they should act on concerning PRO responses. Some felt that PRO assessment may offer the only opportunity for participants to disclose their health concerns, therefore they felt some responsibility to review and act on PRO data.

Generally, coordinators considered their roles as a ‘researcher’ and ‘caregiver’ to be complementary or synergistic, for example many coordinators felt that developing good rapport with participants and taking time to explain PRO assessment led to higher PRO completion rates. However in a finding similar to that reported by Kyte [8] coordinators often felt they were juggling between providing best standard of care for the participant, their highest priority, (which involves responding to any concerns arising from PRO assessment, being flexible, and providing assistance when needed) and their research responsibilities (which involves standardising methodology).

It was apparent that coordinators were often unclear how to prioritise PRO-specific requirements of the trial. Is the need for complete data more important than the need to allow participants to independently self-complete questionnaires? Is it acceptable to explain the meaning of certain words in questionnaires if the participant clearly does not understand, at the risk of potentially influencing their response? When does encouragement to participate cross the line into coercion? Should coordinators act on concerning PRO responses if responses are not meant to inform patient care? These problems are potentially exacerbated by conflicting information offered (or not offered) in PRO training and guidance, and different practices across trials. Such inconsistencies in guidance have caused confusion among trial coordinators as to what constitutes best practice and how they should go about their work. The findings that: (1) the most common form of PRO training received was study-specific; and (2) more coordinators received PRO training from colleagues than at dedicated PRO sessions, represents the likely sources of confusion, highlights the importance of developing standardised PRO administration training for implementation across trials, and explains the variation in practices and training needs.

Before consistent PRO administration practices can be developed, there is a need for PRO research methodologists and stakeholders to achieve consensus on a hierarchy of priorities in PRO data quality. Clearly the PRO administration process is not straightforward. Participants differ in terms of literacy levels, functional abilities, severity of symptoms and toxicities, and their need for reassurance or assistance from trial coordinators. Coordinators highlighted that the need to adapt to trial participants' needs often means that trial guidance is not precisely followed. Therefore rather than simply prescribing a list of procedures, future training must equip coordinators with the knowledge and confidence required to logically prioritise how to respond to common administrative challenges. For example, based on a plethora of evidence demonstrating the problems associated with missing PRO data [16], [17], [18], we argue that complete data is of a higher priority than ensuring that participants independently self-complete questionnaires. Accordingly if faced with a situation where a participant requires assistance (e.g. someone to read the questions and record responses), or requires an explanation of a particular question in order to complete the questionnaire, we argue that this should be offered in favour of avoiding missing PRO data.

To complement such training, those developing the trial protocol must pre-empt these challenges and design trials to accommodate participants' needs, in line with the forthcoming Standardised Protocol Items for Randomised Trials (SPIRIT)-PRO Extension [19]. Trial investigators should discuss which language groups are commonly represented at recruiting sites and determine whether validated language translations are available. Many of the most commonly used cancer PRO questionnaires offer validated translations [20], [21], [22]. This practice would avoid any selection bias from exclusion of non-English speaking participants or from variation in access to interpreters. Furthermore, trial investigators should minimise trial participant burden associated with lengthy or poorly developed questionnaires. Coordinator burden could also be reduced by avoiding cumbersome data entry procedures.

A potential barrier to the training approaches described above is the finding that 55% coordinators felt they did not need further PRO training, despite reporting challenges and engaging in suboptimal and unstandardized practices. Coordinators who feel they do not need further training are unlikely to attend PRO training courses unless mandated. The training needs evident from these interviews are vaster than coordinators' self-reported training needs. Our findings suggest coordinators would generally benefit from training in PRO data quality, communication skills, the importance and purpose of PRO assessment, responding to participants' PRO-related queries, strategies for avoiding missing PRO data, and the importance of adhering to standardised PRO administration methods. Coordinators who reported PRO training needs recommended very short courses to accommodate busy schedules, with supporting written guidance which they could refer back to. Trial sponsors have a responsibility in line with good clinical practice guidelines to ensure coordinators are adequately trained to perform their duties [23]. Back-up coordinators should routinely be trained to stand in for absent trial coordinators to avoid missing PRO data. Potentially information gained through these interviews can be used to deliver training in innovative ways, for example by targeting and training trial coordinator “leaders” who are approached for informal “training” sessions, or by delivering more detailed PRO training presentations at trial start-up meetings. Further research is needed to determine the most feasible and preferred format of future, consensus-based PRO guidance, as preferences were inconsistent in this study.

Strengths and limitations

To our knowledge, this is the first study exploring the realities of PRO assessment for cancer trial coordinators and how PRO assessment fits in the context of their broader role. Coordinators represented five Australian cancer trial recruiting sites and had varying levels of experience with PROs. The participant representation in our study was similar in terms of sex, professional training and overall duties as reported in a previous survey of Australian trial coordinators [3]. We acknowledge that the issues raised in these interviews may not reflect the issues experienced by all cancer trial coordinators, however there were key similarities to Kyte's findings from 26 UK trial staff [8]. We did not explicitly ask whether the coordinators were describing international or local trials, which is a limitation of this study. However all of these coordinators work at sites where international trials are conducted.

Conclusions

Trial coordinators are at the coalface of PRO data collection and their role is of upmost importance in delivering high-quality and valuable PRO data. Ensuring that coordinators understand the impact of their PRO administration practices is key to successful implementation, and this can only be achieved through further education. Our findings highlight the importance of providing clear, consensus-based PRO-specific guidance and training to trial coordinators and to empower coordinators with the knowledge needed to respond to various PRO-specific challenges based on a clear hierarchy of priorities for PRO data quality. Further collaborative research co-designed by trial coordinators, trialists, and PRO specialists is needed to determine how best to implement such training, particularly for experienced trial coordinators.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. RMB is supported by Sydney Catalyst, courtesy of the Cancer Institute New South Wales. MK is supported by the Australian Government through Cancer Australia.