Knowledge acquisition and retention after a high flow training programme in Peru: a quasi-experimental single group pre-post design.

OBJECTIVE: Respiratory infections remain the leading infectious cause of death in children under 5 and disproportionately affect children in resource-limited settings. Implementing non-invasive respiratory support can reduce respiratory-related mortality. However, maintaining competency after deployment can be difficult. Our objective was to evaluate the effectiveness of a comprehensive multidisciplinary high-flow training programme in a Peruvian paediatric intensive care unit (PICU).
DESIGN: Quasi-experimental single group pre-post intervention study design.
SETTING: Quaternary care PICU in a resource-constrained setting in Lima, Peru. PARTICIPANTS: Attending physicians, fellows, paediatric residents, registered nurses, respiratory therapists and medical technicians working in the PICU were invited to participate.
INTERVENTIONS: Concurrent with initial high-flow deployment, we implemented a training programme consisting of lectures, case-based discussion and demonstrations with baseline, 3-month and 12-month training sessions. Pre-training and post-training assessment surveys were distributed surrounding all training sessions. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was achieving minimum competency (median score of 80%) on the high flow training assessment tool. Secondary outcomes included knowledge acquisition (differences in pre-baseline and post-baseline training assessments), short-term retention (differences in post-baseline and pre-3-month refresher training assessments) and long-term retention (differences in post-3-month refresher and pre-12-month refresher training assessments).
RESULTS: Eighty participants (50% nurses, 15% ICU physicians and 34% other providers) completed the baseline assessment. Participants showed improvement in overall score and all subtopics except the clinical application of knowledge after baseline training (p<0.001). Participants failed to retain minimum competency at 3-month and 12-month follow-up assessments (70% (IQR: 57-74) and 70% (IQR: 65-74), respectively). After repeat training sessions, overall knowledge continued to improve, exceeding baseline performance (78% (IQR: 70-87), 83% (IQR: 74-87) and 87% (IQR: 83-91) at baseline, 3 and 12 months, respectively).
CONCLUSION: This study suggests the need for repeat training sessions to achieve and maintain competency after the implementation of new technology. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Our study is the first to evaluate the impact of repeat training sessions on short-term and long-term retention in a resource-constrained setting.

We targeted a multidisciplinary approach to high flow training as part of a larger deployment project.

We did not specifically exclude participants who had not received prior training as part of our real-world approach.

Because training assessments were completed without unique identifiers, we were unable to track individual progress over time but rather evaluated overall knowledge acquisition and retention on a group level.

High provider attrition in refresher training sessions limited statistical power and introduced potential selection bias.

Introduction

Respiratory infections remain the leading infectious cause of death in children under 5, and children in resource-limited settings (RLS) are disproportionately affected.1 Increased availability of non-invasive respiratory support, such as continuous positive airway pressure (CPAP) and nasal high flow (high flow), in RLS has likely contributed to the 20% decrease in mortality from lower respiratory tract infections in this age group.1 Outcomes following the deployment of respiratory technology have shown variable success,2 and concerns related to safety and sustainability in these settings have arisen.3 4

One important factor for the successful implementation and sustainability of respiratory technology in RLS is the presence of a high-quality training programme.5 Current data suggest that knowledge and skills retention after initial training programmes decline over time,6–11 highlighting the need for ongoing refresher training in order to sustain competency. As part of the initial deployment of high flow in the paediatric intensive care unit (PICU) at Instituto Nacional de Salud del Niño (INSN) in 2016, we developed a comprehensive multidisciplinary high flow training programme that included initial and refresher training sessions for all PICU staff. The objective of this study was to evaluate the effectiveness of our high flow training programme by determining participant knowledge acquisition and retention over time. We hypothesised that training programme participants would achieve competency and demonstrate short-term and long-term knowledge retention through scheduled refresher training and clinical experience.

Methods

Design

We performed a quasi-experimental single group pre–post interventional study, introducing a high flow training programme with scheduled refresher training sessions at three discrete time points: baseline, 3 months and 12 months.

Setting and participants

INSN is the largest quaternary care children’s hospital in Peru. Its basic demographics and PICU characteristics have been previously described.5 12 Prior to this study, high flow was not available at INSN. Available respiratory support included invasive mechanical ventilation and non-invasive positive pressure ventilation via CPAP. CPAP was infrequently used due to patient discomfort, difficulties maintaining an adequate seal and skin breakdown. The research team designed a pilot implementation project of high flow within the PICU based on recommendations from INSN PICU providers. Prior to high flow deployment, PICU staff received high flow training according to our comprehensive multidisciplinary training programme.

Subjects were eligible for participation in this study if they were working clinically in the PICU at the time of a high flow training session. We did not exclude participants if they had not attended previous training sessions. Participants included PICU attending physicians, PICU fellows, paediatric residents on their PICU rotation, registered nurses working exclusively in the PICU, respiratory therapists and medical technicians.

Intervention

A multidisciplinary high flow training programme was implemented that included sessions prior to initial high flow deployment and at 3-month and 12-month follow-up intervals. Baseline training consisted of didactic lecture and hands-on demonstration of the high flow system. The didactic lecture was adapted from a high flow training programme used at authors LEE and KRN’s institution, using the evidence-based literature. Baseline training was performed two times per day for four consecutive days during clinical hours to maximise PICU staff participation. Each day, PICU nurses arranged coverage so that half could attend the morning session and the other half could attend the afternoon session. Based on participant feedback from initial sessions, follow-up refresher training incorporated case-based discussion following the didactic lecture. Refresher trainings at 3 and 12 months were performed daily for four consecutive days during clinical hours. No nursing coverage was arranged, so refresher training took place within the clinical space so that more nurses could attend. All training sessions lasted approximately 90 min. The didactic lecture, high flow system set-up video and protocol-specific visual aides were available on the hospital computer for just-in-time training throughout the study period.

Authors LEE and KRN developed assessment surveys to reflect learning objectives from the didactic lecture (online supplementary appendix 1). Surveys were piloted with a convenience sample of seven physicians in paediatric critical care, paediatric emergency medicine and general paediatrics at the two authors’ institutions, all of whom used high flow in their clinical practice. The assessment tool was translated into Spanish and reviewed by authors RB, GM and JT for grammar, clarity and cultural appropriateness. No formal reliability or validity assessment was performed prior to deployment. Questions evaluated participant understanding of the following subtopics: high flow indications and contraindications, protocol-specific details, clinical signs of high flow failure, potential adverse events and clinical application of knowledge. For questions with multiple correct answers, participants received one point for identifying each correct answer and one point for identifying each incorrect answer. There were 23 total points possible: 7 points for indications/contraindications, 2 points for protocol-specific details, 7 points for signs of high-flow failure, 5 points for potential adverse events and 2 points for clinical application of knowledge (online supplementary appendix 2).

Data collection

Pre-training and post-training assessment surveys were distributed immediately prior to and following each training session (online supplementary appendices 1 and 2). Protocol-specific visual aids with the high flow protocol were available to participants during post-training assessment surveys and have been published elsewhere.12

Outcomes

The primary outcome of interest was achieving minimum competency on the high flow training assessment tool. We defined minimum competency as achieving a median score of 80%. Secondary outcomes included knowledge acquisition and short-term and long-term knowledge retention. Knowledge acquisition was defined as improvement in median score from baseline pre-assessment to post-assessment. Short-term knowledge retention was defined as no difference in median score between post-baseline and pre-3-month refresher assessment. Long-term knowledge retention was defined as no difference between post-3-month refresher and pre-12-month refresher assessment.

Statistical analysis

The basic demographics of survey respondents were summarised using counts (and proportions). Participant performance was summarised using median (IQR) for overall score and for subtopics outlined above. We evaluated knowledge acquisition and short-term and long-term retention by comparing pre-training and post-training scores using the Wilcoxon rank sum test (a non-parametric test alternative to a t-test), as the data were not all normally distributed according to the Shapiro-Wilk normality test. Analyses were conducted using STATA statistical software (V.14.2).

Given that paediatric residents rotate in the PICU for 1–2 months duration and would not be available for assessment of knowledge retention, we performed post hoc sensitivity analyses to determine whether their exclusion from knowledge retention analyses influenced results. Due to differences in short-term and long-term retention analyses, we report retention results that exclude paediatric residents.

Patient and public involvement statement

No patients were involved in this study.

Results

Eighty participants completed the baseline pre-training assessment: 40 registered nurses (50%), 15 medical technicians (19%), 12 PICU-trained physicians (15%), 10 paediatric residents (13%) and 2 respiratory therapists (3%). Sixty-six percent of participants had >10 years’ experience as a healthcare provider and 44% had >10 years’ ICU experience. There was >50% drop off in refresher training participation, with only 27 individuals completing the 3-month pre-refresher training assessment and 36 completing the 12-month pre-refresher training assessment (table 1).

Table 1

Participant characteristics for each training session

	Baseline training		3-month refresher training		12-month refresher training
	Pre	Post	Pre	Post	Pre	Post
n	80	76	27	32	36	27
Provider role, n (%)*
ICU-trained physician (fellow or attending)	12 (15)	12 (16)	4 (15)	4 (13)	2 (6)	3 (11)
Paediatric resident	10 (13)	8 (11)	8 (30)	11 (34)	3 (8)	2 (7)
Registered nurse	40 (50)	40 (53)	14 (52)	14 (44)	27 (75)	19 (70)
Respiratory therapist	2 (3)	1 (1)	0	0	0	0
Medical technician	15 (19)	14 (18)	1 (4)	1 (3)	4 (11)	3 (11)
Years of ICU experience, n (%)
>10 years	35 (44)	30 (39)	13 (48)	15 (47)	21 (58)	13 (48)
Years as healthcare provider, n (%)
>10 years	53 (66)	45 (59)	16 (59)	18 (56)	27 (75)	15 (56)
High-flow usage, n (%)
Never	52 (65)	66 (87)	19 (70)	23 (72)	7 (19)	9 (33)
1–5 times	10 (13)	5 (7)	7 (26)	9 (28)	20 (56)	13 (48)
>5 times	13 (16)	3 (4)	1 (4)	0	7 (19)	5 (19)

*Missing provider role data: baseline pre (n=1), baseline post (n=1), 3-month post (n=2).

ICU, intensive care unit.

Participants showed statistically significant improvement in overall score and all subtopics except the clinical application of knowledge after baseline training (p<0.001) (figure 1). After baseline training, participants failed to achieve minimum competency in overall score (78% (70–87)), but achieved minimum competency in all subtopic areas except the clinical application of knowledge (table 2). Participant overall performance exceeded baseline performance after 3-month and 12-month refresher training sessions, (78% (70–87), 83% (74–87) and 87% (83–91) at baseline, 3 and 12 months, respectively), demonstrating incremental improvement and achieving minimum competency after 3-month and 12-month refresher training sessions. Clinical application of knowledge performance was low throughout the study period regardless of training. Scores from all training sessions are reported in tables 2 and 3.

Figure 1

Provider performance on pre-assessment and post-assessments during the study period. Baseline training occurred at time 0 with refresher training sessions 3 and 12 months after baseline training. Open circles illustrate pre-training assessment, and closed squares demonstrate post-training assessment. Error bars represent SD. Minimum competency was defined by a minimum score of 80% (horizontal dashed line). All providers, including paediatric residents, are included in this figure. Differences in pre-baseline and post-baseline assessments determined knowledge acquisition. Acquisition was attained overall and for all subtopics except clinical application of knowledge (p<0.001). Participants did not achieve short-term (from post-baseline to pre-3-month refresher training assessments) or long-term (from post-3-month refresher to pre-12-month refresher training assessments) retention in the overall scores (p<0.05). Long-term retention was not achieved in any subtopic except high flow indications/contraindications (p=0.09). All comparisons were performed using the Wilcoxon rank-sum test.

Table 2

Knowledge acquisition following baseline and refresher training sessions

	Median score of knowledge with training (median % (IQR))						Baseline acquisition	3-month acquisition	12-month acquisition
	Pre-baseline training	Post-baseline training	Pre-3-month refresher training	Post-3-month refresher training	Pre-12-month refresher training	Post-12-month refresher training	p value*	p value*	p value*
	A	B	C	D	E	F	A vs B	C vs D	E vs F
n	80	76	27	32	36	27
Overall	61 (52–65)	78 (70–87)	74 (61–78)	83 (74–87)	70 (63–76)	87 (83–91)	<0.001	0.003	<0.001
Indications/contraindications	86 (71–86)	86 (86–100)	86 (71–86)	86 (86–100)	86 (71–86)	86 (86–100)	<0.001	0.178	0.017
Protocol-specific details	0 (0–0.5)	100 (50–100)	50 (0–50)	100 (50–100)	0 (0–50)	100 (50–100)	<0.001	<0.001	<0.001
Signs of high flow failure	57 (43–86)	86 (57–100)	86 (71–100)	100 (71–100)	71 (57–100)	100 (86–100)	<0.001	0.153	0.003
Potential adverse events	60 (40–60)	80 (60–100)	60 (40–80)	80 (60–100)	70 (60–80)	80 (80–100)	<0.001	0.002	0.006
Clinical application of knowledge	50 (0–50)	50 (0–50)	50 (0–100)	50 (0–50)	50 (0–50)	50 (50–100)	0.504	0.272	0.010

*Wilcoxon rank-sum test.

Overall short-term and long-term retention suffered throughout the study period, with participants failing to retain minimum competency at 3-month and 12-month follow-up assessments (74% (61–78) and 70% (63–76), respectively). This poor retention correlated with infrequent high flow use during the study period (table 1). In the short- and long-terms, participants retained minimum competency only in the high flow indications/contraindications subtopic. In the long-term, participants also retained competency in adverse events (figure 1, table 3).

Table 3

Short-term and long-term knowledge retention following baseline and refresher training sessions excluding paediatric residents

	Median score of knowledge with training(median % (IQR))				Short-term retention	Long-term retention
	Post-baseline training	Pre-3-month refresher training	Post-3-month refresher training	Pre-12-month refresher training	p value*	p value
	B	C	D	E	B vs C	D vs E
n	67	19	19	33
Overall	78 (70–87)	70 (57–74)	83 (74–87)	70 (65–74)	0.017	0.001
Indications/contraindications	86 (86–100)	86 (71–86)	86 (86–100)	86 (71–86)	0.180	0.097
Protocol-specific details	100 (50–100)	50 (0–50)	100 (50–100)	0 (0–50)	<0.001	<0.001
Signs of high-flow failure	86 (57–100)	71 (57–100)	100 (71–100)	71 (57–100)	0.447	0.126
Potential adverse events	80 (60–100)	60 (40–80)	80 (60–100)	80 (60–80)	0.001	0.427
Clinical application of knowledge	50 (0–50)	50 (0–100)	50 (50–50)	50 (0–50)	0.088	0.992

*Wilcoxon rank-sum test.

Discussion

In this study, we found that participants in our high flow training programme demonstrated knowledge acquisition after baseline training and achieved minimum competency in all subtopic areas except the clinical application of knowledge after refresher training sessions. Short-term and long-term knowledge retention was suboptimal, with failure to retain minimum competency at 3 and 12 months. Despite the loss of knowledge, refresher training sessions helped participants achieve competency, with overall scores exceeding those after baseline training. Our findings suggest the importance of repeat training sessions in order to achieve and maintain competency after the implementation of new technology.

Other studies have reported challenges with knowledge retention after training programmes at intervals ranging from 3 to 12 months after initial training.6–11 All reported varying degrees of knowledge loss over time, but none performed interim refresher training but rather relied on clinical experience to reinforce concepts between assessment’s time points. Two studies assessing the effect of cardiopulmonary resuscitation (CPR) refresher training reported improved participant confidence13 and time to obtain skill success.14 Our study provides evidence for an additional benefit of refresher training—to continue to build on participants’ existing knowledge base, particularly in settings where clinical use is low.

Despite this success, this study had several limitations, identifying areas for improvement of our training programme. First, we observed substantial attrition in attendance at refresher training sessions, which had fewer than 50% the number of participants as baseline training. Attrition in provider participation limited statistical power in our study and may have introduced selection bias. With only 27–36 participants completing post-enrolment assessment, the study likely lacked sufficient power to detect anything but large effects in knowledge acquisition and retention. Despite this limitation, most of the pairwise comparisons for knowledge acquisition were statistically significant, suggesting the lack of power may not have been a major issue. However, insufficient power may have affected retention analyses by decreasing our ability to detect differences and therefore concluding that the lack of difference was consistent with knowledge retention.

There are several potential explanations for this drop off in participation. Although efforts were made to maximise provider participation by scheduling training sessions around the nurses’ clinical schedules, due to resource constraints, we were not able to offer the same number of refresher training sessions as we did with the initial training. Furthermore, the combination of a lack of protected time for non-clinical activities and the non-mandatory nature of training likely influenced attendance. In addition, the interdisciplinary nature of our training programme may have made it less desirable for some providers, given variable learning expectations among different provider groups. Other multidisciplinary training programmes have observed differential results according to provider role,15 suggesting that providing separate training programmes tailored to the needs of different provider groups may improve attendance by increasing relevance of content and impact performance on competency assessments.

An additional limitation was the high variability of performance in different subtopic areas, which may be due to the assessment tool itself. The questions ranged from direct questions with a single correct answer to application questions in which multiple answers were correct. We attempted to account for the varying difficulty by giving marks for both correct and incorrect responses to give more weight to complex questions and reward participants for knowing when high flow would not be appropriate to use. However, we acknowledge that this scoring system may have skewed results. Participants in the pilot testing did not express concerns about or struggle with the different questions types; however, all pilot participants were physicians. It is possible that the format of the questionnaire could have been challenging for training participants to understand and may have impacted performance. Including other provider roles in the pilot would have been helpful. In addition, each subtopic had a range of potential points available, from 2 to 7. Therefore, maintaining competency in each subtopic (at least 80%) would be difficult if possible scores are 0%, 50% or 100%. That said, our data suggest that some subtopics were easier to learn and retain than others.

Another limitation was the low frequency of high flow use during the study period, which could have influenced participants’ ability to remember practical details of high flow use, independent of the training programme. Previous studies suggested that high volume (or lack thereof) contribute to retained competency following a training programme,16 17 emphasising the importance of evaluating a healthcare setting’s patient volume prior to introducing new technology. Prior to this training programme, we did not anticipate low high flow use. The high proportion of children with chronic comorbid conditions and/or other exclusion criteria for the pilot implementation research study decreased the volume of candidates for high flow during the study period. Given the low frequency of clinical use of high flow, just-in-time training was available on the local computer, but its usage by staff was not assessed. Altogether, these findings underscore the difficulties in retaining information in the setting of the low frequency of use. This could represent an opportunity for the introduction of simulation to reinforce practical aspects of care delivery.

Finally, despite our efforts to collect unique identifiers for study participants in order to track performance over time, this portion of the questionnaire was infrequently completed by participants. This made it impossible to perform a paired analysis. Given that some individuals likely completed assessments at multiple time points, the populations were not completely independent, which is an assumption of the Wilcoxon rank-sum test. A more appropriate test would have been the Wilcoxon signed-rank test, which we would have performed if we had successfully tracked individuals at all time points throughout the training programme and paired the data.

One concerning finding was participants’ poor retention in identifying potential adverse events associated with high flow use. These included life-threatening complications, such as pneumothorax, which must be promptly recognised to prevent catastrophic outcomes. These results prompted us to reconsider the goals of our training programme. Although disseminating general knowledge about high flow is an essential component of a training programme, most of these details could be provided in a variety of educational handouts or modules at the bedside. However, helping providers better recognise life-threatening complications has the potential to reduce preventable harm and adverse clinical outcomes. Emphasising these serious adverse events in the take-away points of the training module and/or including a case-based example of a child with a pneumothorax could help raise awareness of these life-threatening complications and improve retention over time.

Given the limited power of our study, particularly in subtopic areas, future directions should include efforts to improve provider participation in refresher training sessions to better understand knowledge retention in this setting. Furthermore, future studies evaluating training programmes should consider adding a qualitative aim to better understand participant attrition using individual interviews or focus groups with those who chose to participate and others who did not.

In conclusion, the poor retention of high flow knowledge after initial training underscores the need for frequent refresher training and robust monitoring and evaluation after implementation of new technology. Although applicable to all clinical settings, RLS face their own unique challenges, such as limitations in protected time for training, technological expertise and opportunities for clinical application, which add to known difficulty with knowledge retention. However, even in these settings, appropriately tailored, frequent refresher training programmes are effective ways to achieve and maintain competency following the deployment of new technology. By engaging local collaborators and continuously adapting training programmes to local needs, the sustainability of new technology, such as high flow, in RLS can be achieved.