Protocol for the ORaClES study: an online randomised controlled trial to improve clinical estimates of survival using a training resource for medical students.

INTRODUCTION: Clinicians often struggle to recognise when palliative care patients are imminently dying (last 72 hours of life). A previous study identified the factors that expert palliative care doctors (with demonstrated prognostic skills) had used, to form a judgement about which patients were imminently dying. This protocol describes a study to evaluate whether an online training resource showing how experts weighted the importance of various symptoms and signs can teach medical students to formulate survival estimates for palliative care patients that are more similar to the experts' estimates. METHODS AND ANALYSIS: This online double-blind randomised controlled trial will recruit at least 128 students in the penultimate or final year of medical school in the UK. Participants are asked to review three series of vignettes describing patients referred to palliative care and provide an estimate about the probability (0%-100%) that each patient will die within 72 hours. After the first series, students randomised to the intervention arm are given access to an online training resource. All participants are asked to complete a second series of vignettes. After 2 weeks, all participants are asked to complete a third series. The primary outcome will be the probability of death estimates (0%-100%) provided by students in the intervention and control arms for the second series of vignettes. Secondary outcomes include the maintenance effect at 2-week follow-up, weighting of individual symptoms and signs, and level of expertise (discrimination and consistency). ETHICS AND DISSEMINATION: Approval has been obtained from the UCL Research Ethics Committee (8675/002) and local approvals will be obtained as appropriate. Results will be published in peer-reviewed journals using an open access format and presented at academic conferences. We will also publicise our findings on the Marie Curie website. TRIAL REGISTRATION NUMBER: NCT03360812; Pre-results. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

This is an evaluation of a novel training resource for improving prognostic skills in recognising palliative care patients who are imminently dying.

A multicentre randomised controlled trial design has been used, with (partial) blinding of participants and researchers and including a follow-up to test for any maintenance effects.

This study will provide evidence about whether an online training resource can influence how medical students make prognostic decisions in an experimental setting, using a prognostic task that may lack some ecological validity since it relies on an online rather than a face-to-face assessment of palliative care patients. Therefore, further testing of the intervention in routine medical education and assessment of the accuracy of clinicians’ performance in real-world prognostic tasks will be required.

Introduction

When living with an incurable disease that shortens life expectancy, many patients and their relatives wish to receive information on what the future might bring, including a time frame of the expected length of survival.1 2 Several reports on care near the end of life have highlighted that recognition of the dying phase in palliative care patients is inaccurate and that this can have a significant negative impact on patient care at the end of life.3–5 Being aware that death is imminent can help patients, families and professionals to engage in discussions about goals of care and make decisions about appropriate care and treatment, including hospice admission and starting end-of-life care plans.6 7 In addition, better prognostic awareness can shift patients’ preferences from aggressive life-prolonging treatments towards comfort-oriented care.7–10 It can also help patients and families to make plans for the time remaining, and discuss practical issues such as estate management and funeral planning.7 For families, information about imminent death can help to make decisions about how to look after their loved one, for example, deciding whether the patient can remain at home or deciding whether to stay overnight or invite other relatives to visit.7

While some prognostic models are available, in daily clinical practice it is usually the responsibility of a clinician to formulate a survival estimate.11 12 Making accurate survival predictions is notoriously difficult, estimates are often overoptimistic and prognostic skills do not necessarily develop over time.13–15 Many doctors try to avoid prognostication and feel insufficiently prepared to perform this clinical task.16 The European Association for Palliative Care has recommended that training could improve the accuracy of clinicians’ survival estimates, but there was little evidence to support this recommendation.17 Virtually no education or training resources are available to specifically improve prognostic skills and a better understanding of how clinicians formulate their predictions is crucial to develop such resources.

This study will describe and evaluate an intervention for predicting imminent death (ie, death within 72 hours). Given the current lack of training resources to improve prognostic performance, this study will, as a proof of principle, focus on medical students who have limited clinical experience so that any effects of the intervention are more likely to be detected.

The intervention

The development of the training resource was informed by social judgement theory, which assumes that judgements (prognostic decisions) result from the integration of different types of information, known as ‘cues’.18 Judgement analysis attempts to capture an expert’s ‘judgement policy’ using a multiple regression procedure to calculate the relative weights that the experts attach to different cues. In a previous study, we have used judgement analysis19 to identify the clinical cues (eg, breathing pattern and the presence of respiratory secretions) that expert clinicians use to formulate a prognosis of imminent death.20 The training resource will provide students with task information on how to use the most important cues when making prognostic decisions. Outside of the palliative care context this approach has been used successfully to train a variety of other student populations about how to make decisions more aligned to those of experts.21–24

Experts can be identified in several ways.25 In our previous study, expert palliative care doctors were selected based on the validity of their judgements.26 Palliative care doctors were asked to complete an online prognostic test consisting of a series of vignettes based on real cases and their prognostic estimates were compared against actual survival. The top 20% of performers were defined as ‘experts’ and were invited to complete a second series of fictional vignettes. We will evaluate students’ prognostic performance by comparing students’ estimates against the estimates provided by the experts for the same series of fictional vignettes. In addition, to gain a fuller picture of the expertise as demonstrated by the students, we will assess the extent to which they are able to discriminate between patients with different severities of symptoms/signs and to consistently make similar prognostic decisions for patients with similar symptoms/signs.25 It is important to note however that high levels of discrimination and consistency do not guarantee accuracy,27 therefore these results will be evaluated in conjunction with the comparison of students’ estimates against the experts’ estimates.

In addition to assessing whether students are able to follow the expert judgement policy, a follow-up assessment will be included where students are not given access to the training resource to assess whether they have learnt the policy. A relatively short interval of 2 weeks was chosen to minimise the risk of attrition, based on our experience with a previous study evaluating a similar training resource that showed 10% attrition at the 2-week follow-up, even though participants received a financial reward on completion.22

The training resource will be offered to study participants in an online format, which will enable easy access at the students’ convenience, regardless of geographical location.28–30 If the training material is found to be successful, the online format will enable widespread dissemination and facilitate easy updating and students will be able to re-access the information as and when required.28 31 Studies have indicated that e-learning in medical education, as a supplement to traditional ways of teaching, is perceived as acceptable and evaluated as useful by students.31 32

Objectives

The aim of this trial is to evaluate whether an online training resource can teach medical students to model the prognostic decisions of expert palliative care doctors about which palliative care patients are likely to die within 72 hours. This study will include:

Assess if the probability of death estimates formulated by medical students become more similar to experts’ estimates after completing an online training resource (primary objective).

Determine if any effect of the online training resource is maintained after 2 weeks.

Evaluate if the online training resource changes the weighting of individual symptoms/signs, and whether the students’ judgement policies become more similar to the experts’ judgement policies.

Assess if the online training resource improves the expertise of the medical students, in terms of the ability to discriminate between patients and be consistent in decisions.

Trial design

This is an online multicentre double-blind randomised controlled trial involving an intervention arm that will receive an online training resource for prognostication and a control arm. Medical students will be randomised to these two parallel arms using a 1:1 allocation ratio. Since this will be the first trial of an intervention to improve prognostic skills, the study is designed as a proof of principle study, evaluating whether the training resource can influence how medical students make prognostic decisions in an experimental setting.

Methods and analysis

This protocol follows guidance from the SPIRIT 2013 statement33 and the completed SPIRIT checklist is available as an online supplementary file.

Study setting

This study will be conducted online using a purpose-built study website hosted by University College London (UCL). The study will recruit students from up to 33 medical schools in the UK, approved by the Medical Schools Council.34 As we are offering a financial incentive for participating, it is important to control the potential total number of participants. We will approach individual medical schools as needed until the sample size has been achieved.

Eligibility criteria

Eligible participants should (1) be over 18 years of age; (2) be enrolled on a registered medical course within the UK; (3) be in the penultimate or final year of the course; (4) have sufficient English language proficiency and (5) be willing and able to provide consent as indicated by taking part in the online study assessments. Students in the penultimate and final year are felt to have sufficient knowledge to understand the terminology in the vignettes and will have had at least 1 year of clinical experience as part of their training. Recruitment strategies will be targeted at students in the penultimate and final year of participating medical schools to minimise the risk of non-eligible students taking part. Participants will be asked to confirm their eligibility.

Intervention

The intervention is a newly developed online training resource to improve the recognition of imminent death in palliative care patients. The content was based on a previous study in which we used judgement analysis19 to identify the clinical cues that expert palliative care doctors use to formulate a prognosis of imminent death in terminally ill inpatients.20 Experts were presented with 50 vignettes describing hypothetical palliative care patients (see figure 1 for a sample vignette). There were seven symptoms and signs (‘cues’) available in each vignette: (1) Palliative Performance Scale score35; (2) Richmond-Agitation Sedation Scale score36; (3) rate of decline in general condition; (4) breathing pattern; (5) respiratory secretions; (6) urine output and (7) peripheral cyanosis. The first four cues were the most heavily weighted in the decision-making process of the experts.26

Figure 1

Sample vignette.

Study participants will be presented with the same vignettes and cues that had been presented to the expert palliative care doctors in the previous study. The online training resource will educate the participants on how to use the cue information when formulating a prognosis of imminent death, providing a description of the four most important cues and, where possible, graphical information for ease of understanding. The intervention will be implemented via the study website and should take approximately 15 min to complete.

Data collection procedure and outcomes

The data collection procedure is shown in the study flow diagram in figure 2. After obtaining informed consent, participants will be asked a number of questions to obtain a description of the sample and enable subgroup analyses. This includes demographic questions (age, gender, ethnicity), course detail (place of study, year of study) and palliative care experience (training, placements, experience, confidence). Participants are asked for their name and university email address (to be entered twice for validation). This will allow them to log out and return to the same place at a more convenient time, which is hoped to reduce attrition. It will also allow the research team to check whether participants are affiliated with the universities the study is recruiting from, to populate the certificate of participation, and to send out reminders and gift vouchers.

Figure 2

Study flow diagram.

Participants will then be randomised to either the intervention arm or control arm. Next, participants are given instructions, and are reminded to complete the study individually, in a quiet location, free from distraction at a time and place of their choosing. Following this, they will be able to complete a practice vignette to familiarise themselves with the online environment.

All participants will then be asked to review a first series of 40 vignettes. Each vignette will present a description of a patient (the stem), which is identical for each vignette, with seven cues that describe differing severities of symptoms or signs that vary between vignettes (see figure 1). Participants will be asked to provide a percentage estimate of the probability that the patient will die within 72 hours (0% means no chance of death and 100% means certain death). This series of 40 vignettes includes 30 vignettes presented in random order for each participant, followed by 10 repeated vignettes, also in random order. These repeated vignettes are included to assess participants’ level of expertise, as measured by the discrimination and consistency of probability of death estimates.37 The order in which the seven cues are presented are also randomised per participant, to prevent order effects.

Participants in the intervention arm will then receive the online training resource, while participants assigned to the control group will not receive this additional information but will be informed that they are approximately half way through the study. All participants will be asked to provide probability of death estimates for a further series of 26 vignettes (including six repeated vignettes), in the same format as the first series of vignettes. The participants in the intervention arm will be able to access the online training resource during this second series of vignettes should they wish to do so. It is estimated that it will take up to 45 min to complete the first and second series of vignettes.

Two weeks after completing the second series of vignettes, participants will be asked to repeat the second series of 26 vignettes, although they will not be informed that the cases are the same as those that they have previously completed. Again, the vignettes will be presented in random order to minimise the risk of participants remembering vignettes or the estimates they provided previously. Participants in the intervention arm will not be given access to the online training resource on this occasion. This will enable us to determine if the effect of the intervention has lasted over time. It is estimated that this assessment will take up to 15 min to complete.

All participants can log out from the study website and return at a later time, at any point through the trial. Participants will be sent a reminder email when the third series of vignettes is due and if they start but do not finish the study. Participants will have a 4-week time window to complete the first and second series of vignettes, and another 4-week time window to complete the third series of vignettes. The web-based system will track the time students spend on completing the vignettes and the online training resource, if applicable. To improve data quality, drop-down lists are used where possible and participants will not be able to move on to the next page if essential information is missing or if information has been entered in an incorrect format. The online environment will be piloted by the study team.

Primary outcome

The primary outcome will be the continuous probability of death estimates (0%–100%) provided from the students for the second series of vignettes.

The maintenance effect will be measured by using the probability of death estimates as described for the primary outcome measure at the 2-week follow-up time point.

Cue weighting of the individual students will be compared against that of the experts. When students provide a probability of death estimate they weigh information or ‘cues’ from the vignette as part of the process. By asking students to make a number of decisions on a series of vignettes in which cue values are varied, it is possible to model the weights assigned to the various cues.

The level of expertise will be assessed with the Cochran-Weiss-Shanteau (CWS) index of expertise.25 The CWS index captures the degree of expertise demonstrated in a set of responses and consists of the ratio of discrimination to inconsistency.37 This will help us to understand if the participants become better at discriminating between patients after the intervention, and if their prognostic decisions become more consistent.

Sample size

A sample of 128 subjects (64 subjects in each group) is required to detect a medium effect size (Cohen’s d=0.5) between the intervention and control groups, assuming a common SD, 80% power and using a two-sample t-test at the 5% significance level. A medium effect size of 0.5 is described as an effect that is likely to be visible.38 Larger effect sizes were achieved in previous evaluation studies of similar online training resources by one of the members of our study team (PH).22–24 We estimate that it will be necessary to recruit approximately 183 subjects in order to obtain a final sample size of 128 participants with complete data sets for analysis. The anticipated 30% drop-out rate (ie, participants who start but do not complete the task) has been estimated on the basis of previous similar studies by our own group. Recruitment will start from April 2018 and we anticipate to complete recruitment by the end of December 2018.

Recruitment

Recruitment strategies will differ slightly between medical schools to comply with local ethical/governance requirements. These methods may include: (1) the palliative care lead at each participating medical school introducing the study to students; (2) the course leader or administrator at each participating medical school distributing an email to all penultimate and final year students and (3) advertising the study using newsletters, virtual notice boards and student associations. The study recruitment materials will show the link to the study website and the study email address for students to contact the study team.

Randomisation and blinding procedures

Participant randomisation will be undertaken automatically through the web-based system using a pre-generated randomisation list with a block size of 10. This list will be generated by a member of the study team who will not be involved in recruitment (CT), using computer-generated random numbers.

Researchers and participants will be blinded as far as possible. For data monitoring purposes, researchers will be able to check how many participants are randomised to each arm, but will be blinded to which arm will be the intervention arm. All other data that could reveal the allocation (eg, time taken to complete the intervention) will be concealed from the researchers who will be monitoring the data (NW and LO) and the statistician who will conduct the analysis (FR). The allocation will remain concealed until the statistician has completed the analysis. Participants will be partially blinded to the nature of the intervention and the randomised controlled design of the study to minimise attrition in the control group. Rather than telling participants that half will receive the intervention and half will not, we will inform students that they will receive an online training resource in one of two different formats.

Inducements for participation

As a gesture for the time taken to complete the study and to promote participant retention, participants will be offered a total of £30 gift vouchers for completing the study. Participants will be offered a £10 gift voucher for completing the first stage of the study and a £20 gift voucher for completing the 2-week follow-up. The study will be open for new participants until 64 complete cases are available in each group. When recruitment closes, students who have started the study will be able to complete the remaining assessments and will be eligible to receive the vouchers. Participants who complete the research will receive a certificate of completion that will add towards their academic portfolio. Taking part in the study will give each participant an opportunity to develop some of the clinical skills required to recognise dying palliative care patients. If the resource is found to be effective, then this could benefit future medical students and palliative care patients.

Patient and public involvement

We involved two medical students in the design of the study, to make sure the research design is appropriate for this population and the study documents are easy to understand. Two fourth-year medical students reviewed the recruitment documents and piloted the website, keeping track of how much time was needed to complete the study. Both students provided valuable comments, resulting in several changes in the recruitment email, advertisement material and participant information sheet. These changes mainly involved emphasising certain aspects of the study to make it more appealing to medical students. One of these students will also be involved in distributing recruitment emails.

Statistical methods

Demographic characteristics, course detail and palliative care experience will be summarised by treatment assigned and overall. As a result of the randomisation process, we expect the groups to be balanced. Categorical data will be presented as numbers and percentages. Continuous data will either be described with mean and SD, or median and IQR, pending the distribution. We will produce a Consolidated Standards of Reporting Trials flow diagram of all participants (http://www.consort-statement.org/).

In this proof of principle study, we are seeking to evaluate whether an online training resource can influence how medical students make prognostic decisions in an experimental setting. Therefore, we will employ a per-protocol analysis where those participants who do not complete all vignettes or violate the protocol (eg, putting the same answer for every vignette) will be removed from the analysis. To assess if the online training resource affected the probability of death estimates provided, we will calculate the degree of agreement between the study participants’ probability of death estimates for the first and second series of vignettes and the probability of death estimates obtained from the experts in our previous study. Suitable regression models will be fitted to estimate the effect of the intervention, comparing agreement with the experts for those who received it and those who did not. In addition to this, we will also visualise the degree of agreement in both groups for the first and seconds series of vignettes, using the Bland-Altman method.39

The maintenance of the study effect will be measured by repeating the primary outcome analysis with the estimates from the 2-week follow-up time point. To assess whether the intervention altered the judgement policy of the participants, we will examine participants’ cue weights for the three series of vignettes and correlate them with the expert’s cue weights presented in the training information. The CWS performance index and the subcomponents discrimination and consistency will be compared between the intervention and control arms for the three series of vignettes.25 37

Data monitoring and adverse events

Throughout the trial, the research team will review recruitment figures. Researchers will check the responses given by the participants, and the time taken by each participant to complete the vignettes, to assess for compliance with the protocol. Participants may be excluded from the analysis if their response record strongly suggests that they did not comply with the study protocol (eg, all items answered with the same response or too speedily). The Trial Management Group (PS, PH, LO, NW, CT, SY, FR and HG) will be responsible for overseeing the trial and will meet regularly (at least four times per year) to review recruitment figures.

This is a very low risk study. Students will have had at least 1 year of clinical experience as part of their training, and are informed that the vignettes are hypothetical. In the participant information sheet students are encouraged to contact the student support services at the medical school they attend if they do experience psychological distress. This study will not have a Data Monitoring Committee.

Ethics and dissemination

In case any protocol amendments are required, these will be reviewed by the Trial Management Group before submission to the relevant committees, and the trial registry will be informed where necessary.

Consent

Participants will be given a brief summary of what the study involves in the recruitment email. On the website, there will be a welcome page with information about the study and the participant information sheet is available to download. However, as described above, participants will be partially blinded to the nature of the intervention and the randomised controlled design of the study. The participants will be reminded that they are free to withdraw at any time. Informed consent will be obtained via two checkboxes before starting the study assessments: (1) to participate and (2) to acknowledge the results will be used in future publications, research and educational packages. Those who do not consent will not be able to continue to the next page.

On completion of the second and third series of vignettes, a debrief page will be shown to remind the participant what the results will be used for. The contact details for the study team will also be displayed should they have any concerns or issues they wish to follow-up. This debrief will not include any more detailed explanation of the two groups students were randomised to or the active intervention, as we feel this would not be appropriate. Students will participate at a time convenient to them, and we would not want students who have completed the study to disclose this information to others who have yet to participate.

Data management

Our study received ethics approval before the introduction of the General Data Protection Regulation (GDPR) on May 25th, 2018. The documents that were approved state that all data will be handled in accordance with the UK Data Protection Act 1998. No formal amendment was required following the introduction of the GDPR, but we added a transparency message to the study website to make participants aware of how we will use their information. Participants will be asked for their names and university email address as a personal identifier as well as being assigned a unique participant ID. During the trial, all data will be kept securely on a web-based database, which is encrypted and password protected. The database will be accessible to approved members of the research team only as access to the Intranet will be restricted to their IP addresses only. Recruitment strategies may include the study being introduced to students by their local palliative care lead. This person will not have access to the study database and will therefore not be aware which students participated and which did not, nor will he or she have access to the prognostic performance data of individual students.

Once all data have been reviewed and the gift vouchers have been distributed, the names and email addresses will be deleted from the web-based database in a secure manner and only the participant ID will be referenced. The final trial database will be downloaded from the website by the research team for statistical analysis and UCL will act as the data controller of such data for the study.

Dissemination policy

Study results will be published in peer-reviewed, indexed journals using an open access format, and the results will be presented at academic conferences. Authorship eligibility will be in accordance with The International Committee of Medical Journal Editors. We will also publicise our findings on the Marie Curie website. If the online training resource is proven effective, it will be made freely available after the trial. Data (suitably anonymised) may be shared with other research groups if a reasonable request is submitted to and agreed by the CI.