Participant retention in trauma intensive care unit (ICU) follow-up studies: a post-hoc analysis of a previous scoping review.

BACKGROUND: The study aimed to synthesize participant retention-related data for longitudinal follow-up studies of survivors from trauma intensive care units (ICUs).
METHODS: Within a published scoping review evaluating ICU patient outcomes after hospital discharge, two screeners independently searched for trauma ICU survivorship studies.
RESULTS: There were 11 trauma ICU follow-up studies, all of which were cohort studies. Twelve months (range: 1-60 months) was the most frequent follow-up time point for assessment (63% of studies). Retention rates ranged from 54% to 94% across time points and could not be calculated for two studies (18%). Pooled retention rates at 3, 6, and 12 months were 75%, 81%, and 81%, respectively. Mean patient age (OR 0.85 per 1-year increase, 95% CI 0.73 to 0.99, p=0.036), percent of men (OR 1.07, 95% CI 1.04 to 1.10, p=0.002), and publication year (OR 0.89 per 1-year increase, 95% CI 0.82 to 0.95, p=0.007) were associated with retention rates. Early (3-month) versus later (6-month, 12-month) follow-up time point was not associated with retention rates. DISCUSSION: Pooled retention rates were >75%, at 3-month, 6-month, and 12-month time points, with wide variability across studies and time points. There was little consistency with reporting participant retention methodology and related data. More detailed reporting guidelines, with better author adherence, will help improve reporting of participant retention data. Utilization of existing research resources may help improve participant retention. LEVEL OF EVIDENCE: Level III: meta-analyses (post-hoc analyses) of a prior scoping review. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Advancement in trauma care has improved survival among critically injured patients, who often experience prolonged admissions in an intensive care unit (ICU).1 Survivors often have reduced health-related quality of life.2–4 Hence, there is an increasing number of studies evaluating patient outcomes after hospital discharge, including survivors from trauma5 6 and trauma ICUs.7 This approach is critical to understanding the full reintegration of injured patients into society, as promoted by the National Academies of Sciences, Engineering, and Medicine report titled “A National Trauma Care System: Integrating Military and Civilian Trauma Systems to Achieve Zero Preventable Deaths After Injury”8 9 and the National Quality Forum’s Population-Based Trauma Outcomes report.10

Retaining study participants in longitudinal follow-up studies can be challenging, but is integral for study validity and statistical power.11 There is growing interest in understanding and implementing the most effective participant retention strategies12–14; however, to our knowledge, there has been no synthesis of participant retention-related data across studies evaluating postdischarge outcomes of trauma ICU survivors. Thus, the objective of this article was to synthesize retention rates and strategies from studies evaluating postdischarge outcomes of trauma ICU survivors.

Methods

This article follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist.15 This systematic review’s protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews; CRD42018087835).

Search strategy and study selection

The database from a prior comprehensive scoping review of 425 ICU survivorship articles that included at least one posthospital outcome measure was searched for publications on trauma ICU studies for inclusion in this analysis.7 The detailed search strategy and methods for this scoping review are reported elsewhere.7 In summary, the scoping review retrieved 20 189 citations from searching five online publication databases (PubMed, EMBASE, PsycINFO, Cumulative Index of Nursing and Allied Health Literature, and the Cochrane Controlled Trials Registry) during the designated search period (1970–2013). No language restrictions were applied in the scoping review.

From among the 425 articles reported in the scoping review, two trained researchers (HR and RN) independently screened full articles for studies that included trauma ICU patients. The researchers were not blinded to author/journal details. Studies were excluded if (1) non-trauma patients were included in the study or (2) there was only a single follow-up time point at which both consent and follow-up data collection occurred (ie, no prospective follow-up happened after consent).

Data abstraction

Duplicate data abstraction was performed by pairs of researchers. Conflicts were resolved by consensus, in consultation with a senior researcher (VDD or DMN). The following data were collected: participant retention rates and related data at each follow-up time point; reasons for loss to follow-up; use of a participant flow diagram; modes of data collection (eg, in person, phone, mail); reporting of mortality during follow-up; blinding of assessors (if interventional study); accounting for loss to follow-up in sample size/power calculation; study exclusion criteria related to barriers to follow-up (eg, homelessness); any discrepancy in reporting participant retention-related data; and description of participant retention strategies. Authors were contacted for additional data when necessary.

Risk of bias

There were no randomized controlled trials included in this systematic review. For observational studies, risk of bias was assessed using a modified Newcastle-Ottawa Scale,16 excluding three criteria not applicable to this systematic review given its focus on participant retention rather than a specific clinical end point: (1) demonstration that the outcome was not present at enrollment, (2) assessment of the outcome and (3) follow-up long enough for the outcome to occur.

Statistical analysis

Pooled average participant retention rates were calculated in this analysis. Among eligible studies, the following follow-up time points were reported: 3, 6, 12, 24, 36, and 60 months. Data from follow-up times >12 or <3 months could not be pooled due to only a single study evaluating the time point. For studies reporting participant age as median and IQR, for purposes of this analysis, mean and SD for participant age were estimated using the methods proposed by Wan et al.17 One study18 published mean and SD age separately for four treatment groups; these data, along with the sample size in each group, were used to calculate an overall mean and SD age for the study. One study supplied CI for age instead of SD, and the CI was converted to SD. Participant retention data reported separately for treatment groups or patient subgroups at the same time point within a study were tested for a statistically significant difference using Fisher’s exact test and grouped if not significant. For studies where retention rates were 100%, the Haldane-Anscombe correction was used to correct the CI.19 20

Participant retention rates were calculated in two ways. For the primary approach, the retention rate was calculated by dividing the number of participants who had a study assessment (numerator) by the total number of participants alive and eligible for follow-up at that same time point (denominator). The secondary definition also excluded those who withdrew from the study in calculating the denominator. Retention rates were not calculated if all requisite data were not reported or if mortality was combined with loss to follow-up data.

A linear random intercept regression model (logit transformation) was used to pool retention rates across all eligible studies and time points, where each study was represented by a value of the random intercept. This regression model was extended to determine if the pooled average retention rate was associated with two patient demographic characteristics reported in all studies (average age and percent of men) and with study publication year. A separate extended regression model was constructed for each of the patient and study characteristics.

Statistical heterogeneity among included studies was evaluated using the I2 statistics (with >50% deemed to be substantial heterogeneity).21 The I2 statistics were calculated for each time point when there were more than >2 studies reporting data.22 SAS V.9.4 was used to conduct all analyses.

Results

Of the 425 publications included in the original scoping review, 16 publications, reporting on 11 unique studies (figure 1), met the eligibility criteria for this analysis (ie, focused on trauma ICU survivors). All 11 studies were cohort studies.18 23–36 37 The time points for follow-up ranged from 1 to 60 months, with the most frequent time point being 12 months, occurring in seven (63%)25–27 29 31 33 36 37 studies (table 1). Five (45%) studies were exclusively conducted in Europe and four (36%) in the USA (table 1). In each of the 11 eligible studies, a majority of the participants were male, with a range of 57% to 83%. Across studies, the mean age ranged from 27 to 44 years old.

Figure 1

Flow diagram of identification of eligible studies on trauma ICU survivors. ICU, intensive care unit.

Table 1

Study cohort characteristics

Publications for each unique study	Start and end of study	Type of study	Countries	Sample size*	Age† (years), mean (SD)	Male (%)	Modes of contact ever used in follow-up?	Time points for follow-up (months)	RetentionratePrimary definition‡(%)	Retention rateSecondary definition§ (%)
Aitken et al23	June 2008– August 2012	Cohort	Australia	112105	40 (20)	83	Phone, mail	16	7672	8384
Christensen et al18	August 2005–December 2008	Cohort	USA and 26countries	472	38 (14)	77	Not reported	3	72	74
Richards et al24 25 37	July 2006–July 2008	Cohort	USA	160	43 (17)	57	In person	12	67	68
Orwelius et al26	August 2000–June 2006	Cohort	Sweden	10883	44 (18)	68	Mail	1224	7954	7969
Schnyder et al27 28	January 1996–June 1998	Cohort	Switzerland	120	38 (13)	74	Not reported	12	88	88
Tøien et al29 30	June 2005–December 2007	Cohort	Norway	148147	40 (16)	70	Mail	312	7979	8080
Hepp et al31 32	January 1996–June 2003	Cohort	Switzerland	12110690	38 (13)	74	Not reported	61236	888874	88100100
Davydow et al33	July 2001–September 2003	Cohort	USA	1781	41 (32)	71	Phone	12	83	83
Frutiger et al34	1980–1988	Cohort	Switzerland	177	36 (17)	81	In person, other	60	94	94
Holbrook et al35	January 1990–June 1990	Cohort	USA	–	30 (13)	74	Phone	3	NA¶
Mackenzie et al36	July 1982–March 1984	Cohort	USA	–	27 (7)	78	In person, phone	612	NANA

*Sample size was calculated as the number of ICU survivors eligible for follow-up at hospital discharge. For studies that did not provide this information we used the sample size at start of the study, and for studies that did not provide either we used the sample size after informed consent was obtained. In cases where there were articles published while the study was still ongoing, we used the article with the largest sample size.

†Age was expressed as mean (SD) or median (IQR).

‡Primary participant retention rates were calculated as the number of participants assessed at each follow-up time point divided by the number presumed alive and eligible for assessment at that time point (this included the participants who withdrew and withdrawn just prior to the time point).

§Secondary participant retention rates were calculated after a sensitivity analysis where we calculated the number of participants assessed at each follow-up time point divided by the number alive and eligible for assessment at the time point (this did not include the participants who died, withdrew, and withdrawn just prior to the time point).

¶Retention rates could not be calculated because mortality was combined with loss to follow-up.

ICU, intensive care unit; NA, not applicable;

Risk of bias assessment

Of the 11 cohort studies, 6 had adequate reporting of follow-up (online supplemental table S1). Nine studies had a positive rating for comparability of cohorts. Lastly, two studies compared their trauma ICU cohort with a non-trauma cohort.

Reporting of retention data

Eight (72%) studies23–25 27 29 31 33 35–37 reported exclusion criteria related to ability to follow up participants after hospital discharge, with the most common exclusion criterion being language proficiency in five (62%) studies23 25 27 29 31 (table 2). No study reported accounting for loss to follow-up to calculate sample size or statistical power. Nine (82%) studies18 23–27 29 31 33 34 37 reported loss to follow-up and mortality data separately, with two (18%) publications35 36 combining them in their study reporting. A flow diagram for patient follow-up was included in five (45%) studies, with only four (36%)26 27 31 34 reporting reasons for lost to follow-up at each time point.

Table 2

Participant retention-related data in longitudinal studies of trauma ICU survivors

Participant retention-related issue	Studies reporting, n (%) (N=11)
Study exclusion criteria included barrier(s) to follow-up (eg, homelessness)	8 (72)
Sample size or power calculation	0 (0)
Reported use of strategies to improve participant retention	4 (36)
Mortality reported during follow-up	8 (72)
Reported lost to follow-up rates combined with mortality	2 (18)
Included flow diagram with retention rate for each follow-up time point	5 (45)
Reported reasons for lost to follow-up at each follow-up time point	4 (36)

ICU, intensive care unit.

Participant retention

In the nine studies with adequate reporting, retention rates could be calculated for all time points (range: 54%–94%). Pooled average retention rates (95% CI) at 3-month, 6-month, and 12-month follow-up time points were 75% (22% to 97%; 2 studies; n=632), 81% (1% to 100%; 2 studies; n=244), and 81% (72% to 88%; 6 studies; n=2441; I2=86%), respectively (figure 2).

Figure 2

Pooled average retention rates in trauma ICU survivor follow-up studies. Retention rates were calculated as the number of participants assessed at each follow-up time point divided by the number presumed alive at that time point (this included the participants who withdrew and withdrawn just prior to the time point). Diamonds in the graph are the pooled average retention rates, whereas bars represent 95% CI. Linear random effects regression model was used to pool retention rates across all eligible studies and time points. ICU, intensive care unit.

Retention rates of the earliest (3 months) time point were not statistically different from the later time points at 6 months (p=0.653) or 12 months (p=0.278). For every 1-year increase in average participant age in the eligible studies, the odds of retention were lower by 15% (OR 0.85, 95% CI 0.73 to 0.99, p=0.036). For every 1% increase in the proportion of male participants in the eligible studies, the odds of retention were higher by 7% (OR 1.07, 95% CI 1.04 to 1.10, p=0.002). Finally, publication year was also significantly associated with retention rate; with every 1-year increase (ie, 1-year more recent publication), the odds of retention were lower by 11% (OR 0.89, 95% CI 0.82 to 0.95, p=0.007). These results did not qualitatively change when evaluating the participant retention rates using the secondary definition, as previously described in the Methods section.

Discussion

In this analysis, we report and synthesize participant retention-related data for 11 longitudinal studies reporting on functional outcomes of adult trauma ICU survivors. Two (18%) of the studies did not report adequate data for calculating retention rates. Among the remaining nine studies, retention rates ranged from 54% to 94% across follow-up time points, with pooled retention rates at 3, 6, and 12 months of 75%, 81%, and 81%, respectively.

The pooled retention rates from this analysis (75%–81%, during the first year of follow-up) were similar to posthospital follow-up rates from 21 studies of acute respiratory failure survivors (82%–89% during the first 2 years of follow-up).38 These findings were also similar to a broader range of predominantly non-trauma/non-critical illness healthcare-related follow-up studies, as reported in a systematic review of 82 studies that reported retention strategies and rates (median 85%, IQR 79%–92%).12 13

Timing of follow-up (3 months vs. 6 months or 12 months) was not associated with a difference in retention rates, but mean age, proportion of male participants, and publication year of studies were significantly associated with retention rates. Retention rates were higher with a greater proportion of male participants, and lower with older participants and with more recent study publication.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines were published in 2007.39 STROBE recommends reporting the number of “potentially eligible” participants and “confirmed eligible” participants, reporting reasons for non-participation, and including a participant flow diagram. Participant retention rates were reported in 100% of studies published after 2007, but only 60% studies before 2007. In addition, the two studies35 36 for which retention rates could not be calculated were published before the STROBE guidelines. Hence, perhaps STROBE guidelines have helped improve reporting in studies of trauma ICU survivors.

No study reported a sample size or statistical power calculation, and <50% of studies reported other important research methodology components (eg, participant retention strategies used, participant flow diagram). Reporting on loss to follow up-data varied widely. We were unable to calculate retention rates in two (18%) studies because loss to follow-up was combined with mortality. These findings highlight the potential value of updating the STROBE guidelines to require more detailed reporting. Interestingly, the majority of the studies that reported retention strategies had high retention rates.

To reduce selection bias in follow-up studies, there is growing interest in participant retention and related methodology, as evidenced by an increase in publications and resources focused on improving participant retention.13 The National Institutes of Health/National Heart, Lung, and Blood Institute funded a national research infrastructure project (R24HL111895), with one aim specifically focused on improving participant retention via creation and dissemination of practical retention tools and resources to aid investigators (www.improvelto.com/cohort-retention-tools/).

For example, this project supported completion of a systematic review on participant retention strategies.12 13 From the studies included in that systematic review, the project compiled 618 participant retention strategies, across 12 different themes, which are available as a free searchable online database (www.improvelto.com/sysrevstrategies). Moreover, best practices for participant retention in healthcare-related studies have been published,14 along with four empirical analyses relating to participant retention.40–43 Such publications are important in ensuring evidence-based advancement of methods for participant retention.

Furthermore, one of the publications from this project provides empirical evidence to debunk the myth that intensive retention efforts are bothersome to participants.41 Ultimately, this national infrastructure project has shared >30 downloadable tools, including customizable telephone scripts and letters, as well as templates relevant to participant follow-up, such as a detailed participant contact information form. With increasing interest in posthospital outcomes of trauma patients,5 10 44 improving participant retention in studies evaluating long-term outcomes is critical to help reduce bias and better inform the care of critically injured patients.

Strengths and limitations

To our knowledge, this is the first evaluation of participant retention methodology in studies of adult trauma ICU survivors. There are potential limitations to be acknowledged. First, there are a relatively small number of studies, and studies published after 2013 could not be included since that was the end date of the database of studies from the prior scoping review on which this analysis was based. Second, there is heterogeneity in the studies that were pooled; hence, caution is advised in interpreting the pooled average retention rates, with recognition that there is some variability across studies and time points. Third, other factors that may be relevant to retention of post-ICU patients, such as discharge location, were not collected in this synthesis and should be considered in future studies. Lastly, since the focus of this analysis was adult trauma ICU survivors, these results may not generalize to other populations of critically ill patients.

Conclusion

In this evaluation of 11 studies of trauma ICU survivors, the pooled participant retention rate was >75% across 3-month, 6-month, and-12 month follow-up assessments. However, retention rates across individual studies were highly variable (54%–94%) and there was inconsistent reporting of retention-related methodological data. Although guidelines (Consolidated Standards of Reporting Trials and STROBE) recommend reporting participant retention data, more detailed guidance on such reporting, along with strict adherence by researchers, may help further advance research aiming to understand the postdischarge outcomes of trauma ICU survivors. Moreover, use of existing participant retention resources, including new NIH-funded free resources (see www.improveLTO.com), may help researchers mitigate loss to follow-up and its associated potential for low statistical power and bias.