Evaluating the impact of national education in pediatric palliative care: the Quality of Care Collaborative Australia.

PURPOSE: The Quality of Care Collaborative Australia (QuoCCA) provided pediatric palliative care education across Australia with the aim of improving the quality of services. The education was delivered through a collaboration of six tertiary pediatric palliative care services, through funding for Nurse Educators, Medical Fellows, a National Allied Health Educator, and national project staff.
METHODS: Pre- and post-education surveys were completed by participants immediately following the education, and confidence and knowledge were measured along nine domains related to the care of the child and family, including managing a new referral, symptom management, medications, preparing the family, and using local agencies.
RESULTS: Education was provided to over 5,500 health and human service professionals in 337 education sessions across Australia between May 2015 and June 2017. Paired pre- and post-surveys were completed by 969 participants and showed a significant improvement in all the domains measured. Those with no experience in caring for children receiving palliative care showed greater improvement following QuoCCA education compared to those with experience, although the latter had higher scores both before and after education. Similarly, those with no previous education showed greater improvement, but those with previous education showed higher scores overall. Participants in full-day and half-day sessions showed greater improvement than those in short day sessions. Thus, the dosage of education in the length of the sessions and prior attendance impacted knowledge and confidence. Topics requested by the participants were analyzed. Educator learnings were that education was more effective when tailored to the needs of the audience, was interactive, and included story-telling, case studies, and parent experiences.
CONCLUSION: These results encouraged the continuation of the provision of education to novice and experienced professionals who care for children with a life-limiting condition, leading to higher levels of confidence and knowledge. The learnings from this evaluation will be transferred into the second round of funding for the national QuoCCA education project. The next stage will focus on developing simulation and interactive training, accessible training modules, and videos on a national website.

Introduction

Based on data from the UK,1 it is estimated that 32 per 10,000 children in Australia are living with a life-limiting condition. Most specialist pediatric palliative care services (PPCS) are based in capital cities in Australia and provide outreach via telehealth.2,3 In Queensland (QLD) alone, 150 patients were referred to the tertiary PPCS based in Brisbane over a 2-year period from July 2009, 58% of which were from regional or rural areas.4

The needs of families and communities supporting children with life-limiting conditions are complex and often ongoing, and health professionals and other agencies require specialized knowledge and skills to provide effective care. As there are no dedicated PPCS outside of tertiary hospitals in Australia, children and their families rely on a network of existing local health professionals for care. This can include pediatric services (eg, child health nurse), palliative care services (adult and community), and other public and private community services. The tertiary PPCS are available for consultation and advice. The 1:60 ratio of child-to-adult deaths in Australia indicates that health care professionals who routinely care for adults with palliative and end of life care needs require extra support to provide care for children due to the infrequency in presentation to their services.4 This is particularly the case for health professionals in regional and remote areas. Consideration also needs to be given to the significant impact that the death of a child has on the family, community, and health professionals involved.

To address the educational needs of these health professionals, a national collaboration titled the Quality of Care Collaborative Australia (QuoCCA) was funded under the Australian Government Department of Health, National Palliative Care Projects, to provide national education in pediatric palliative care (PPC) from May 2015 to the end of June 2017 initially.5

This education was coordinated through Nurse Educators, Medical Fellows, a national Allied Health Educator and national project staff based in the tertiary PPCS in the following hospitals: Queensland Children’s Hospital (Brisbane, QLD), Sydney Children’s Hospital (Randwick, New South Wales [NSW]), John Hunter Children’s Hospital (Newcastle, NSW), Royal Children’s Hospital (Melbourne, Victoria [VIC]), Women’s and Children’s Health Network (Adelaide, South Australia [SA]), and Perth Children’s Hospital (Perth, Western Australia [WA]). These teams also provided education to the Northern Territory (NT), Tasmania (TAS), and Australian Capital Territory (ACT), thus covering all of Australia. The activity was governed via a National Steering Committee and managed through a lead entity at Children’s Health Queensland (CHQ). A learning needs analysis was conducted to guide the content of the education.6

The goal of QuoCCA was to achieve service quality improvement in national PPC in acute and community settings, through education, research, and evaluation within a quality framework, building on the foundation of the work of the national collaborative. The objectives were to have:

Improvement in skill, knowledge, and confidence of national health professionals at all levels involved in the delivery of PPC enabling them to provide best practice care for children and families.

Improvement in timely and appropriate access to, and quality of, PPC in national acute and community settings.

Improved processes for care planning in local communities.

Increased community and health professional awareness of death and dying as a normal part of life-limiting conditions in childhood and the need for family centered and developmentally and culturally appropriate services.

The reach included health professionals involved in acute and community care and those working in regional, rural, and remote locations throughout Australia through a program of education and a suite of resources.

Existing clinical education standards recommend evaluating four levels: 1) process evaluation regarding participant satisfaction; 2) impact evaluation regarding acquisition of knowledge, skills, or confidence; 3) outcome evaluation regarding changes in practice; and 4) service quality improvement.7,8 Based on this, a project logic was developed that detailed the outputs, impacts, and longer term outcomes required of the project, with the associated evaluation tools.

This article examines the impact of QuoCCA education through the use of pre- and post-education surveys completed by health professionals and other participants during the first funding round of the project, with education being provided between May 2015 and August 2017. The evaluation plan for QuoCCA education also included interviews with health professionals, educators, and families and an outcome survey examining changes in practice and long-term improvement in knowledge and confidence, which will be reported on in future papers.

Methods

The QuoCCA funded educators (medical, nursing, and allied health) were experienced PPC providers who had various degrees of experience in the delivery of PPC education. In addition, other medical, nursing, and allied health staff from tertiary services worked alongside QuoCCA educators, bringing their expertise to the sessions. This reflected the multi-disciplinary approach to PPC. QuoCCA provided education above a measured baseline of education, which was already being provided by the tertiary PPCS in each state.

The educators worked together to construct standard curricula, through regular meetings to discuss the education content, quality, and effectiveness; sharing resources and presentations through a national secure file transfer system; and the educators traveling to other states to participate in education sessions.

The content of education was flexible for each audience to maximize the relevance against current needs. Educators had a collection of baseline presentations from which they chose the most appropriate for the audience.

An introduction to PPC agenda would normally include what is PPC, what/who is the PPCS, myths and dilemmas, pain and symptom management (including nonpharmacological strategies), communication, contemporary grief and bereavement concepts, spirituality, case studies, end of life care, and self and team care.

More advanced topics were delivered as relevant to the audience and/or the patient(s) in the area and included advance care planning, cultural/spiritual concerns, transformational care and bereavement, disease-specific information, age-specific information (eg, neonates, adolescents, and young adults), end of life care, allied health specific information, ethics, and practical considerations (eg, disability, equipment, and respite).

The majority of education sessions conducted by QuoCCA were evaluated with pre- and post-education surveys with questions measuring the confidence or knowledge related to nine dependent variables (Supplementary materials). These variables (and shortened names) were as follows: the management of new referrals for PPC patients (referrals), symptom management along five domains (“pain”, “nausea”, “dyspnea”, “seizures”, and “fear or anxiety” [fear]), confidence in helping families prepare for the death of their child (preparation), knowledge of resources such as agencies that can assist in providing care (resources), and confidence giving medications (medication).

The four confidence questions had a Likert scale of 1–5 from 1= not at all confident to 2= slightly confident, 3= moderately confident, 4= very confident, and 5= extremely confident, or N/A = not applicable. The knowledge questions had a Likert scale of 1–5 from 1= no knowledge to 2= slight knowledge, 3= moderate knowledge, 4= high level of knowledge, and 5= extreme level of knowledge, or N/A = not applicable.

The survey also collected information on eight independent variables, which were used as collected or re-coded as follows:

Education type – this was categorized as scheduled, pop-up, or incidental. Scheduled education was pre-planned general PPC education in any location. Pop-up education was related to the care of a specific patient, supporting local services away from a tertiary center to care for the child and family. In this education type, a small inter-professional team of health professionals traveled to a center to deliver education, responding in a timely way to patient/family needs.9 Incidental education was provided as a part of other meetings or activities in a tertiary center (eg, handover, multi-disciplinary, and morbidity and mortality meetings).

Experience – previous involvement in the care of a child with a life-limiting condition (yes/no).

Previous education – previous attendance at education days conducted by the tertiary PPCS in their state (yes/no).

Remoteness – this was based on the location of the education as determined by the Australian Statistical Geography Standard remoteness structure10 and accessed through Doctor Connect.11

State – this was initially coded as the state in which the education was conducted. However, as there were a smaller number of surveys from the smaller states without a QuoCCA educator, this was re-coded as the state who conducted the education. ACT was recoded to QLD, NT was recoded to SA, and TAS was recoded to VIC, representing the states that mostly provided QuoCCA educators to conduct this education. Each of the five states with QuoCCA educators conducted education in their own state, although on occasion they were accompanied by QuoCCA educators from other states.

Length – the length of education session was re-coded as short = sessions of 2 hours or less, half-day = 3–4 hours and full-day = more than 4 hours.

Financial year – this was recoded to combine 2014/2015 and 2015/2016 due to the occurrence of limited sessions in the former; 2016/2017 stood alone.

Occupation – nurse, doctor, social worker, occupational therapist, physiotherapist, pharmacist, psychologist, music therapist, and others (specified).

Following education sessions, QuoCCA educators completed a report on the session through an online survey, including date, location, educators, attendees, and open text comments on the following:

What is your feedback on how this session went?

What lessons did you learn from this education session?

Qualitative data related to the topics and the effectiveness of education were analyzed using an inductive thematic approach.12 Educators were also asked for feedback at the end of the project.

Ethics

The protocol for the evaluation of this newly developed national collaborative in education was approved by the Children’s Health Queensland Human Research and Ethics Committee (HREC/16/QRCH/60). Participants were provided with information about the project and advised that participation was not mandatory and all questionnaires completed were anonymous.

Statistical analyses

The recommended sample size for a population size of 5,773, with a confidence level of 95% and a margin of error of 5%, is 361. We had paired surveys completed by 969 participants.

Descriptive statistics of the dependent variables was reported as median and inter-quartile range (IQR) for non-normally distributed continuous data. Normality was assessed using a Shapiro–Wilk test, which indicated that none of the variables were normally distributed (P<0.001). Frequencies (raw counts and percentages) were presented for categorical data. Proportions of categorical data were compared using a Chi-squared test, eg, occupation, previous education, and experience.

Pre- and post-education surveys were paired by participant. A nonparametric Wilcoxon signed ranks’ test was used to assess the difference between pre- and post-education scores for each of the dependent variables (referral, pain, nausea, dyspnea, seizure, fear, resources, preparation, and medication).

Change scores in paired education surveys were calculated for each dependent variable by subtracting the post-score from the pre-score when information was collected at both time points for individual participants. These change scores were analyzed against the independent variables to determine if they influenced improvement following education. A Mann–Whitney U test was used for independent variables with two categories (experience, previous education, and financial year). A Kruskal–Wallis test was used (along with Dunn’s post hoc test with multiple testing adjustment when the overall result was significant) for independent variables with more than two categories (education type, state, length, occupation, and remoteness).

In addition, the proportion of participants who showed an improvement in scores between pre- and post-education surveys (a change score of 1 or more), were compared with categories of experience and education using a chi-squared test.

Finally, a binomial logistic regression was performed to investigate factors that could potentially predict improvement following education. This involved re-coding the change score for each dependent variable into a binomial variable indicating improvement (a difference between pre- and post-scores of 1 or more) or no improvement (a difference of less than 1). The analysis was performed on the categorical independent variables using the following as the reference points – having experience caring for children with life-limiting conditions, having had previous education, financial year of 2014/2015 or 2015/2016 (vs 2016/2017), occupation of nursing (vs allied health and other), and major city (vs regional/remote). The odds ratio (OR) and the P-value were included where they were significant.

All statistical analyses were performed using SPPS Version 24, and P-values<0.05 were considered statistically significant.

Results

What were the outputs of the education?

From May 2015 to June 2017, there were 337 education sessions conducted by the QuoCCA Project, which included 183 scheduled, 107 pop-up, and 47 incidental sessions, totaling 767 hours and delivered to 5,773 attendees. Education was delivered in all states and territories of Australia, and the remoteness of the locations is shown in Table 1.

Table 1

Characteristics of QuoCCA education sessions by remoteness

Remoteness	Sessions (% pop-up)	Participants	Hours	Average hours/session	Average participants/session
Major city	203 (25)	3,684	404	2.0	18
Inner regional	77 (38)	1,168	224	2.9	15
Outer regional	40 (55)	605	93	2.3	15
Remote	11 (18)	208	25	2.3	19
Very remote	5 (60)	88	20	4	18
Webinar	1 (0)	20	1	1	20
Total	337 (32)	5,773	767	2.3	17

Abbreviation: QuoCCA, Quality of Care Collaborative Australia.

The mean number of participants in scheduled education sessions was 20.5 (maximum 274), 10.3 for pop-up sessions (maximum 38), and 12.7 for incidental sessions (maximum 50). There was generally a greater percentage of pop-up education sessions in the more regional and remote areas and a greater number of recorded participants per session in remote areas (Table 1).

A range of health professionals received education including medical (n=808), nursing (n=3,280), allied health (n=562), and others (n=617) such as administration staff, aboriginal health workers, community development workers, chaplains and pastoral carers, teachers, funeral directors, and paramedics (note that 506 participants did not report their occupation). Health professional participants included students, managers, educators, and research staff. Allied health professions represented were occupational therapy, speech pathology, dietetics, music therapy, pharmacy, physiotherapy, psychology, radiation therapy, and social work. Dentists also attended the education.

Examining the attendance of the different occupations at QuoCCA education, the pop-up sessions had a greater proportion of medical staff compared to other education types (14% medical, 74% nursing, and 11% allied health) (P=0.02). In scheduled education sessions, there was no significant difference in the proportion of different occupations who attended full-day, half-day, or short sessions (P=0.29).

By occupation, a greater percentage of nurses had received previous education and had experience in caring for a child with a life-limiting condition. A greater percentage of medical and allied health staff had no previous education (P=0.002) or experience (P=0.04).

Did participants report a change in confidence/knowledge following education?

Twenty-six percent (n=1,474) of participants submitted either a pre-education survey or post-education survey, and 17% (n=969) submitted both pre- and post-surveys. Only these paired survey responses were analyzed. Not all education sessions had surveys collected. Of the 1,100 participants at pop-up sessions, 9% of the participants completed both pre- and post-surveys. This is compared with 22% of the 3,747 scheduled education participants and 5% of the 547 incidental participants completing both surveys.

Table 2 shows the median and IQR of the pre- and post-education scores, as well as the change score, for all nine dependent variables. For all measures, there was a significant increase in knowledge or confidence following education (P<0.001).

Table 2

Result of univariate analyses on the pre- and post-education scores

Variables	Pre	Post	Wilcoxon signed ranks p-value	Change score pre to post
	Median (IQR)	Median (IQR)	Difference pre to post	Median (IQR)
Referral	2 (2,3)	3 (3,4)	<0.001	1 (1,2)
Pain	3 (3,3)	3 (3,4)	<0.001	1 (1,1)
Nausea	3 (3,3)	3 (3,4)	<0.001	1 (1,1)
Dyspnea	2 (2,3)	3 (3,4)	<0.001	1 (1,1)
Seizures	2 (2,3)	3 (3,4)	<0.001	1 (1,1)
Fear	3 (3,3)	3 (3,4)	<0.001	1 (1,1)
Resources	2 (2,3)	3 (3,4)	<0.001	1 (1,2)
Preparation	2 (2,3)	3 (3,4)	<0.001	1 (1,1.5)
Medication	3 (3,4)	3 (3,4)	<0.001	0 (0,1)

Table 3 shows the number of paired surveys that showed a negative change score between pre- and post-education, a positive change score (an improvement in confidence or knowledge), and no difference. The questions related to the management of referrals, preparation for the death of a child, and resources showed the largest percentage of participants with an improvement in knowledge or confidence. Only 2%–6% of participants showed a decline in change score following education.

Table 3

The number of participants showing a decline, no change or improved scores from pre-education to post-education surveys for each variable measured

Measures	Decline in score Post < pre	Same score Post = pre	Improved score Post > pre	Percentage of improved scores
Referral	17	200	693	76.2
Pain	33	328	503	58.2
Nausea	45	356	448	52.8
Dyspnea	32	313	496	59.0
Seizures	35	369	433	51.7
Fear	24	293	558	63.8
Resources	20	209	685	74.9
Preparation	17	208	692	75.5
Medication	48	374	346	45.1

Which independent variables had an impact on the education change score?

Change scores (the difference between post- and pre-education survey scores) for all measures were analyzed against the categories of each independent variable (Table 4).

Table 4

Result of univariate analyses (P-values) on the impact of independent variables on change scores (difference between pre- and post-education scores) for all measures

Independent variables	Referral	Pain	Nausea	Dyspnea	Seizure	Fear	Resources	Preparation	Medication		Statistic
Experience	<0.001***	<0.001***	<0.001***	0.013*	<0.001***	0.139	0.019*	<0.001***		0.069		Mann–Whitney
Education	<0.001***	0.010**	0.019*	0.014*	0.001***	<0.001***	<0.001***	<0.001***		0.027*		Mann–Whitney
Year	0.726	0.180	0.873	0.521	0.698	0.681	0.001***	0.624		0.871		Mann–Whitney
Type	0.464	0.546	0.042*	0.223	0.114	0.921	0.275	0.151		0.577		Kruskal–Wallis
Occupation	0.492	0.111	0.713	0.236	0.977	0.760	0.146	0.984		0.408		Kruskal–Wallis
Length	0.023*	<0.001***	0.002**	0.094	0.966	0.010**	0.235	0.115		0.084		Kruskal–Wallis
Remote	0.295	0.820	0.962	0.711	0.598	0.633	0.001***	0.251		0.729		Kruskal–Wallis
State	0.004**	0.001***	0.045*	<0.001***	<0.001***	0.029*	<0.001***	0.007**		0.059		Kruskal–Wallis

Notes: P-values from the statistical tests that examined how the independent variables impacted on the difference between pre-education and post-education measures.

P<0.05.

P<0.01.

P<0.001.

Participants with no experience in caring for a child with a life-limiting condition had significantly higher change scores for all measures except fear and medication (Table 4, Mann–Whitney U test, P<0.05). In addition, when comparing each of the measures for those participants with experience and those without, a significantly greater percentage of participants without experience showed an improvement in post-education scores except for dyspnea, fear, resources, and medication (Chi-squared, P<0.05; Table 5).

Table 5

Percentage of participants who reported improved change scores following education (increase of one or more in scores) and median values (IQR) for pre- and post-education scores by experience in caring for children with life-limiting conditions

	Experience			No experience			Mann–Whitney P for pre-scores	Mann–Whitney P for post-scores
Variables	Percentage of improved participants	Median pre (IQR)	Median post (IQR)	Percentage of improved participants	Median pre (IQR)	Median post (IQR)	Experience, yes vs no	Experience, yes vs no
Referral	74.3	2 (2,3)	3 (3,4)	88.4	1 (1,2)	3 (3,3)	<0.001	<0.001
Pain	56.2	3 (2,3)	3 (3,4)	70.5	2 (1,3)	3 (3,3)	<0.001	<0.001
Nausea	50.1	3 (2,3)	3 (3,4)	68.6	2 (1,3)	3 (2,3)	<0.001	0.001
Dyspnea	57.9	3 (2,3)	3 (3,4)	66.7	2 (1,2.25)	3 (2,3)	<0.001	<0.001
Seizures	48.7	3 (2,3)	3 (3,4)	70.9	2 (1,2)	3 (2,3)	<0.001	<0.001
Fear	63.7	3 (2,3)	3 (3,4)	64.8	2 (1,3)	3 (3,3.75)	<0.001	0.008
Resources	74.9	2 (2,3)	4 (3,4)	76.6	2 (1,3)	3 (3,4)	<0.001	0.053
Preparation	73.5	2 (2,3)	3 (3,4)	87.6	2 (1,2)	3 (2,3)	<0.001	<0.001
Medication	44.1	3 (2,4)	3 (3,4)	52.4	2 (1,3)	3 (2,3)	<0.001	<0.001

Although those with no experience had relatively higher change scores, so showed greater improvement, their actual median scores were lower than those with experience (Table 5). The participants with experience scored themselves significantly higher before the education session (Mann–Whitney for prescores, P<0.001; Table 5) and then reported higher post-education knowledge or confidence across the measures of the education (except for resources) (Mann–Whitney, P<0.01; Table 5).

Participants with no previous education in PPC had significantly higher change scores for all measures (Table 4, Mann–Whitney U test, P<0.05). In addition, a significantly higher percentage of participants who had no previous education showed an improvement in post-education scores for all measures except medication (Chi-squared, P<0.05; Table 6).

Table 6

Percentage of participants with and without previous education who reported improved change scores following education (% 1+) and median values (IQR) for pre- and post-education scores

	Previous education			No previous education			Mann–Whitney P-value for pre-scores	Mann–Whitney P-value for post-scores
Variables	% 1+	Median pre -scores	Median post -scores	% 1+	Median pre -scores	Median post -scores	Previous education, yes vs no	Previous education, yes vs no
Referral	60.1	3 (2,3)	4 (3,4)	74.4	2 (1,3)	3 (3,4)	<0.001	0.002
Pain	43.6	3 (2,3)	3 (3,4)	54	3 (2,3)	3 (3,4)	0.011	0.566
Nausea	38.2	3 (2,4)	3 (3,4)	48	3 (2,3)	3 (3,4)	0.004	0.152
Dyspnea	44.1	3 (2,3)	3 (3,4)	53.1	2 (2,3)	3 (3,4)	<0.001	0.134
Seizures	33.5	3 (2,4)	3 (3,4)	47.3	2 (2,3)	3 (3,4)	<0.001	0.050
Fear	48.2	3 (2,3)	3 (3,4)	59.9	2 (2,3)	3 (3,4)	<0.001	0.041
Resources	56.5	3 (2,4)	4 (3,4)	73.8	2 (2,3)	3 (3,4)	<0.001	0.019
Preparation	57.1	3 (2,3)	3 (3,4)	74.7	2 (1,3)	3 (3,4)	<0.001	0.001
Medication	30.0	3 (3,4)	4 (3,4)	37.1	3 (2,3)	3 (3,4)	<0.001	<0.001

Those with no previous education showed a greater improvement in their change scores, although their median pre- and post-scores were lower than those with previous education (Table 6). Participants with previous education scored themselves significantly higher before the education session (Mann–Whitney for prescores, P<0.05; Table 6) and then generally reported higher knowledge or confidence following education across the measures of the education (except for pain, nausea, and dyspnea) (Mann–Whitney, P<0.05; Table 6).

There was no significant difference (P>0.05) in change scores in different financial years except for knowledge regarding resources, where 2016/2017 education sessions showed a greater improvement between pre- and post-scores (P=0.001; Table 4).

The type of education provided no significant impact on the change score, except for confidence in managing nausea (P=0.042), where pop-up education showed a higher mean rank for the difference between pre- to post-education scores (Table 4). There was also no significant difference in the percentage of participants that showed improved scores by education type (P>0.05).

There was no significant impact of occupation on the change scores for any measure (P>0.05; Table 4).

The length of the education session had an impact on the change scores for management of referrals (P=0.023; Table 4), pain (P<0.001), nausea (P=0.002), and fear (P=0.01). There were higher rankings of change scores in full-day education for pain, nausea, and fear and half-day education sessions for management of referrals. Short education sessions showed significantly lower change score rankings for referrals, pain, nausea, and fear.

Table 7 shows the percentage of participants that improved by the length of education session, which was significant for all variables except dyspnea and seizures (P<0.05). In all the significant variables, short education sessions showed a smaller proportion of participants whose scores had improved, followed by half-day education and then full-day education (Table 7). In total, for all variables, 55% showed improved scores following short education, 63% for half-day and 66% for full-day.

Table 7

Percentage of participants who showed an improvement in knowledge or confidence by length of the education session

Measures	Full- day	Half-day	Short	Total	Chi- squared P -value
Referral	78.5	77.8	70.3	76.2	0.046
Pain	64.6	58.1	45.5	58.2	<0.001
Nausea	56.8	54.5	43.5	52.8	0.004
Dyspnea	61.9	59.9	52.4	59.0	0.057
Seizures	52.3	51.0	51.1	51.7	0.94
Fear	68.1	63.7	54.1	63.8	0.002
Resources	77.9	75.0	69.0	74.9	0.029
Preparation	79.3	73.8	68.8	75.5	0.006
Medication	48.1	46.4	38.3	45.1	<0.001

The only measure significantly impacted by remoteness was the knowledge of resources (P=0.006). Very remote education sessions showed higher change scores related to resources, followed by major city sessions.

The state that conducted the education had a significant impact on the pre- and post-education scores for all measures except for medication (Kruskal–Wallis test, P<0.05). SA had the highest ranking of pre- to post-education scores for referrals, pain, seizures, fear, resources, and preparation. QLD had the higher ranking for dyspnea.

Can we predict an improvement following education?

The results of the binomial logistic regression are shown in Table 8. Experience, previous education, length and type of education session, and state conducting the education were significant for most variables, and financial year of the education was significant for knowledge about resources (Table 8).

Table 8

Binomial logistic regression of improvement (yes, no) in confidence/knowledge between pre- and post-education surveys, showing the model’s P-value, odds ratio and 95% confidence interval.

Variables	Reference	Comparison	Referral	Pain		Nausea	Dyspnea	Seizure	Fear	Resources	Preparation
Experience Previous education	Model P-value		<0.001		0.003	<0.001		<0.001			<0.001
	Yes	No	2.66(1.49, 4.74)		1.86(1.23, 2.82)	2.18(1.44, 3.31)		2.57(1.68, 3.93)			2.55(1.48, 4.41)
	Model P-value		<0.001		0.015	0.03	0.028		0.001	<0.001	<0.001
	Yes	No	2.05(1.42, 2.97)		1.55(1.09, 2.21)	1.48(1.04, 2.13)	1.49(1.04, 2.14)		1.77(1.25, 2.52)	2.44(1.70, 3.49)	2.45(1.71, 3.52)
Length	Model P-value		0.050		<0.001	0.004			0.002	0.03	0.007
	Full	Short	0.65(0.46, 0.92)		0.46(0.33, 0.63)	0.59(0.43, 0.81)			0.55(0.40, 0.77)	0.63(0.45, 0.89)	0.58(0.41, 0.81)
	Half	Short			0.60(0.40,0.91)	0.64(0.43, 0.97)
	Short	Full	1.54(1.09, 2.18)		2.18(1.59, 3.00)	1.71(1.24, 2.35)			1.71(1.30, 2.51)	1.59(1.13, 2.24)	1.74(1.23, 2.45)
	Short	Half			1.66(1.10, 2.50)	1.56(1.04, 2.35)
Year	Model P-value									0.001
	2014/2016	2016/2017								1.69(1.25, 2.29)
Remoteness	Model P-value					0.03
	City	Regional/remote				1.35(1.03, 1.77)

Notes: Where the model was significant (P<0.05), the model P-value and OR (95% CI) are listed. No comparisons for medication, education type, or occupation (nursing vs other) were significant.

Participants with no experience in caring for children with a life-limiting condition were 1.9–2.7 times more likely to improve in post-education scores as compared with those with such experience across the dimensions of referral, pain, nausea, seizures, and preparation for death. Note that those with experience started with a higher pre-education score and also had a higher score following education (P<0.05; Table 5).

Participants with no previous education in PPC were 1.5–2.5 times more likely to improve in post-education scores as compared with those with previous education across the dimensions of referrals, pain, nausea, dyspnea, seizures, fear, resources, and preparation for death (that is all dimensions measured except medication). Those with previous education started with a higher pre-education score and had a higher score following education (P<0.05; Table 6).

Participants in the financial year 2016/2017 were 1.7 times more likely to improve in the knowledge of resources, compared to the earlier 2014/2016 participants.

Regional and remote participants were 1.4 times more likely to improve than city participants in confidence with the management of nausea and seizures.

Regarding the length of the education session, full-day education participants were 1.5–2.2 times more likely to improve compared to short day education participants along the dimensions of referral, pain, nausea, fear, resources, and preparation (conversely, participants in short education sessions were 0.5–0.7 times likely to improve compared to full-day education). Half-day education participants were 1.6–1.7 times more likely to improve compared to short education sessions along the dimensions of pain and nausea (conversely, short education sessions were 0.6 times more likely to improve compared to half-day education).

What education topics did participants request?

Table 9 shows the education topics that were requested by the participants in the surveys and the numbers of participants that requested them. There were six major themes in the topics requested related to clinical care (with sub themes of family, treatment, symptoms, processes, and others), bereavement, health care team/services, communication, staff well-being, and education.

Table 9

Topics that participants hoped to hear (pre-education) and would like to be covered in future (post-education survey)

Areas	Combined pre- and postsurvey topics	n	Areas	Combined pre- and postsurvey topics	n
Clinical care		333	Health care team/services		103
Family	Emotional management	24		Multi-disciplinary team roles	7
	Adolescents	4		Allied health	21
	Family support	29		Nursing	1
	Siblings	4		Team preparation	1
	Preparing for death	1		Neonatal	1
	Spiritual	6		Intensive Care Unit	5
	Age appropriate support	2		Ambulance	1
	Counseling	1		Links to pediatric ward	1
	Relaxation	1		Community care	5
	External support services for family	5		General practitioner	1
	Family support network	4		Local services	23
Treatment	Medication	44		Dental	1
	Syringe drivers/pumps/lines	16		National Disability Insurance Scheme	2
	Practical information	2		School’s role	1
	Hydration	1		After hours	2
	Equipment	3		Carer respite	2
	Resources	1		Link to tertiary PPCS	8
	Feeding	3		Transition to region	1
	Nasogastric tube	1		App	1
	Non-pharm aids	2		Debriefing	1
	Tackle box	1		Handover	1
	Functioning	1		Communication	2
Symptoms	Symptom management	29		Telehealth	1
	Dyspnea	2		Employment pathways	1
	Sleep	1		Resources	12
	Nausea	1	Communication		101
	Secretions	1		General communication	26
	Seizures	1		Child/adolescent/siblings	12
	Anxiety	3		First meeting with family	6
	Pain	17		End of life	12
Processes	Advance care planning	7		Bad news	6
	Referral	16		Decision making	3
	Admission	1		Resources	1
	Assessment	1		Conflict	7
	Treatment	1		Cultural differences	22
	End of life	25		Patient function	1
	Sudden deaths	3		Other families	1
	Documentation	2		What not to say	4
	Stages	2	Staff well-being		31
	Discharge	2		Staff support/coping	14
	Crisis	1		Self care	10
	After death	10		Debriefing	4
	Organ donation	2		Professional boundaries	1
	Care at home	6		Grief/bereavement	2
Others	Euthanasia	2	Education		70
	Ethics	13		More education/annual	13
	Legal information	2		Parents/child stories	8
	Pathways	2		Mythbusters	1
	Pediatrics	8		Copies of PowerPoint slides	2
	Diseases	14		Role plays	9
	Neonatal	1		Scenarios	2
	Physiology	1		Annual case review	1
Bereavement		41		More interaction	1
	Grief/bereavement	30		Specific to professions	2
	Memory making	7		Case studies	24
	Acute grief reactions	1		Videos and simulations	2
	Siblings	2		Online resources	4
	Counseling	1		Frequently asked questions	1

Abbreviation: PPCS, Pediatric Palliative Care Service.

The participants commented about the modes of education that they found most effective. These included case studies, parent and child perspectives, simulations, mythbusters and FAQ, videos, online learning resources, and copies of the PowerPoint slides.

Two participants suggested education targeted to various professions, particularly allied health professions. Presentations from parents and patient stories were very effective. Thirteen participants wanted more frequent, such as annual, education in the subject area.

What was the educators’ feedback on the effectiveness of education?

There were 303 comments provided by the QuoCCA educators in their reports on education sessions and what lessons they learnt, 50 of which were related to the effectiveness of education. Educators found they should tailor the presentation to the audience both in content and style. Learning needs of the group were ascertained up front, by gathering audience concerns and questions or by having a discussion with a local champion prior to the session.

Good outcomes were evident when participants shared their experiences of palliative care, including “pair and share” items, when audience examples were sought, and the educator deferred to the knowledge in the room.

One educator had a pattern for half-day or longer workshops, where a case study would flow into a role play, involving group work. An overview of PPC and the tertiary service would include story telling. Another educator stated that audience participation and knowledge sharing in the session had a vital role to play in the effectiveness of education. This also took the pressure off the educator needing to be everybody’s expert. For short sessions of an hour or less, educators would consider having an informal discussion rather than a slide presentation, which encouraged audience participation and directly addressed their needs.

Discussion

QuoCCA education was delivered to a range of health professionals and other service providers in each state and territory around Australia. The variety of participants reflected the response required in the community to care for a child receiving palliative care and their family, including community services, hospital services, schools, pastoral carers, primary health carers and funeral directors.

The QuoCCA national collaboration provided opportunities for educators based in tertiary PPCS from different states to develop a common curriculum, share information, deliver education sessions together as a team, and discuss their experiences of effective education. This included the use of innovative and interactive education methods such as case-based discussion, role play, simulation, and the use of video and artwork. In this way, the QuoCCA collaboration contributed to the quality of education delivered to health professionals to improve learning outcomes and consequently improve the quality of PPC services throughout Australia. This was also the experience in centers working together to improve care quality and reduce variation in the care of children with hypoplastic left heart syndrome.13

The evaluation scores showed a significant improvement in knowledge or confidence in all the measures following education. A large percentage of participants had improved education scores for confidence in managing new referrals, preparing the family for the death of their child and knowledge of resources such as agencies that can assist. These broad dependent variables were less task orientated than those related to symptom management and giving medication, which were case driven and more challenging to deliver in a workshop setting.

We were able to predict which participants were more likely to show an improvement in their scores following education. They were those who had no experience for caring for a child with a life-limiting condition, those who had no previous education in PPC, and those who attended full-day or half-day education rather than short education sessions. In one or two measures, those who received education in the latter half of the project, and those from regional or remote areas showed a greater improvement.

Our results demonstrate that novice and experienced professionals build confidence and knowledge from participation in ongoing professional education. In fact, the higher dosage of education in terms of the length of the session and previous attendance at education resulted in increased knowledge and confidence. This result was also shown in other studies, where prior preparation for education and postgraduate qualifications in the area improved the impact of education.14–16 In Taiwan, the frequency of contact with palliative care services and further education were predictors of knowledge and attitude to a palliative care consultative service.17 Importantly, they found that access to a palliative care team for consultation regarding patients had a positive impact on the care provided. In another study, there was a threshold of palliative care education of 40 hours, beyond which physicians were less likely to recommend major operative interventions, implying that there was a dosage of education that was necessary to impact practice.18 Future evaluation in pediatric palliative care education could analyze the amount of education received by individuals, and the impact that has on their practice.

Ferguson et al19 stated that most education programs in PPC included symptom management and pain control. Our QuoCCA participants requested education along the themes of clinical care, staff well-being available services, communication, interprofessional roles, and bereavement. Within these themes, the specific education topics most requested were management of medications, bereavement of families and staff, family support, symptom management, communication, end of life management, emotional management, utilization of local services, and cultural differences. These topics aligned with the 12 most highly rated learning needs found by the QuoCCA project according to weighted average, which were end of life management, communication skills, symptom management, bereavement care, emotional supports, grief and loss, and self-care.6

Traditionally, education programs in PPC have taught knowledge and skills through lecture and presentation style education.19 A review of the impact of education on clinician practice and patient outcomes found interactive techniques to be most effective, while didactic presentations and printed information had little or no benefit.20 Interactive education that used blended innovative methods such as multimedia, case studies, role play, small group discussion, and simulations is known to be effective.21–28 Improved learning outcomes occurred with reflective questions, prior planning of education, quality improvement activities incorporated into the education,29 as well as the use of a human patient simulator30 and simulations.31 The American Nurses Credentialing Center’s Commission on Accreditation32 also reported that the transfer of new knowledge and skills was facilitated through simulation, interactive computer programs, and web-based applications. The experience of the QuoCCA educators was that the most effective education was more interactive and involved greater participation and reflection. QuoCCA education focused on promoting the palliative care approach. Such concepts can only be transferred to health professionals through education that facilitates reflection and learning from the experience of families, children, and other health professionals.

Participants in the longer QuoCCA education sessions (full-day or half-day) showed higher likelihood of improvement in knowledge or confidence. A review of continuing medical education also found a positive correlation between education effectiveness and the length of interventions.21

The QuoCCA education sessions benefited from a small number of participants per session21,23 and the provision of additional structured resources for self-directed learning following the education session via a QuoCCA website (https://www.caresearch.com.au/quocca/).25

Limitations

The study methodology incorporated surveys in which participants self-reported their level of knowledge or confidence. However, participant’s self-awareness of their individual knowledge or skills, and various influences on their confidence level, may impact on the effectiveness of these surveys.33–36 For example, an experimental design with a control group found that extraneous variables may have interfered with measures of nursing quality.37

The surveys examined generic areas of the application of PPC education that would improve the management of new referrals, symptom management, medications, preparing the family, and using local agencies. Effectively the analysis examined the increase in knowledge and confidence in as much as it fulfilled the gaps as perceived by the audience, enabling them to provide higher quality PPC in these areas. With any education, the perception of increase in knowledge or confidence as assessed by the participant will be a reflection of how the education has met their expectations and brought them forward in their knowledge and confidence in delivering the target service.

Survey collection was not consistent for all education sessions. Not all the education sessions had pre- and/or post-evaluation performed, and for practical reasons, the shorter and pop-up education sessions were less likely to have surveys collected. This may have skewed the results for the short education sessions. There may also be some reporter bias in the participants who were more likely to complete a survey.

Between 2% and 6% had decreased scores in the various measures following education and reasons for this could vary. It is possible that after the education, the participant realized they did not know as much as they initially thought. Another possibility is that they simply did not take into account the score they put in the pre-survey when they answered the post-survey. Alternatively, they could have been dissatisfied with the education and decided to indicate that in their scores.

It is recommended that the goals of evaluation should be clearly linked to the required outcomes of the teaching.38 However, the goals of each QuoCCA education event were determined by the learning objectives of participants at each session. Furthermore, pop-up education was specific to a patient’s and family’s needs, their supporting services, and the incidental education specific to its context. Thus, the curriculum provided was not standard across educators or states but tailored to the audience and the needs of the service.

The same pre- and post-education surveys were used across the national project and were not tailored to the education session. However, the actual content of the education sessions varied. For example, not all symptom management and medication measures were covered in all sessions and this was reflected in lower improvement scores for some of these measures. The surveys for the second round of QuoCCA funding (2017–2020) will be combined to a single post-education survey with more generic measures relevant to PPC and include an option to tick when the session did not cover that topic.

Generally, the educational impact of QuoCCA sessions may have been influenced by disparate education models and methods and characteristics of the audience receiving that education. The national collaborative will continue to provide benefit through ongoing harmonization of the curriculum and teaching methodology throughout Australia.

Conclusion

QuoCCA provided education to over 5,770 participants during the 2.2 years of education delivery in this initial project. The evaluation of education sessions found significant improvements in scores following education in all the measures collected. Longer education sessions resulted in greater improvement in confidence or knowledge. Participants who did not have experience in caring for children with a life-limiting condition and those with no previous education in PPC showed higher improvement in scores from pre- to post-surveys. Although their overall improvement was less, experienced staff had higher scores in both their pre- and post-education surveys. Therefore, these results encourage the continuation of the provision of education to both novice and experienced professionals and others who care for children with a life-limiting condition, moving both to higher confidence and knowledge.

The learnings from this evaluation will be transferred into the second round of funding for this national education project, QuoCCA 2. The next stage will continue to deliver education and focus on quality improvement methodologies, simulation and interactive training, accessible harmonized training modules, and videos on the national website (https://www.caresearch.com.au/quocca/, hosted by CareSearch). The project will develop and share a body of knowledge to raise the awareness of the palliative care approach and the benefits of timely referral to local services that provide good quality PPC.

Participant – name, organization, postcode, and occupational group.

Experience – percentage of week involved providing palliative care directly to patients/families, number of children cared for in the past year who have died, and previous involvement in the care of a child with a life-limiting condition.

Previous education – previous education days attended led by the Pediatric Palliative Care Service in that state.

While preparing for this education session, do you feel that you are confident in managing the new referral of a child with palliative care needs to your service?

While preparing for this education session, do you feel that you are confident in the management of symptoms that a child with a serious life-limiting condition may experience? – pain, nausea, dyspnea, seizures, fear or anxiety

While preparing for this education session, how confident do you feel in helping a family prepare for the death of their child?

While preparing for this education session how confident do you feel giving medications, including setting up a subcutaneous delivery system?

While preparing for this education session do you feel that you have knowledge of resources such as agencies that can assist you in providing palliative care in your area?

Supplementary materials

Pre- and post-education survey content

Education assessment

The post-education survey repeated these questions preceded by the phrase “since attending the education session” or “after completing the education session…”.

Education topics

The pre- and post-survey forms included open text responses about the topics that participants hoped to hear and topics that they would like to be covered in the future. The questions were as follows.

Pre-survey – are there any other topics you would like to see covered in this session?

Post-survey – are there any other topics which you would like to see covered in sessions in the future?