Comparison of three meta-analytic methods using data from digital interventions on type 2 diabetes.

AIMS: Pooling the effect sizes of randomized controlled trials (RCTs) from continuous outcomes, such as glycated hemoglobin level (HbA1c), is an important method in evidence syntheses. However, due to challenges related to baseline imbalances and pre/post correlations, simple analysis of change scores (SACS) and simple analysis of final values (SAFV) meta-analyses result in under- or overestimation of effect estimates. This study was aimed to compare pooled effect sizes estimated by Analysis of Covariance (ANCOVA), SACS, and SAFV meta-analyses, using the example of RCTs of digital interventions with HbA1c as the main outcome.
MATERIALS AND METHODS: Three databases were systematically searched for RCTs published from 1993 through June 2017. Two reviewers independently assessed titles and abstracts using predefined eligibility criteria, assessed study quality, and extracted data, with disagreements resolved by arbitration from a third reviewer.
RESULTS: ANCOVA, SACS, and SAFV resulted in pooled HbA1c mean differences of -0.39% (95% CI: [-0.51, -0.26]), -0.39% (95% CI: [-0.51, -0.26]), and -0.34% (95% CI: [-0.48-0.19]), respectively. Removing studies with both high baseline imbalance (≥±0.2%) and pre/post correlation of ≥±0.6 resulted in a mean difference of -0.39% (95% CI: [-0.53, -0.26]), -0.40% (95% CI: [-0.54, -0.26]), and -0.33% (95% CI: [-0.48, -0.18]) with ANCOVA, SACS, and SAFV meta-analyses, respectively. Substantial heterogeneity was noted. Egger's test for funnel plot symmetry did not indicate evidence of publication bias for all methods.
CONCLUSION: By all meta-analytic methods, digital interventions appear effective in reducing HbA1c in type 2 diabetes. The effort to adjust for baseline imbalance and pre/post correlation using ANCOVA relies on the level of detail reported from individual studies. Reporting detailed summary data and, ideally, access to individual patient data of intervention trials are essential.

Background

The number of published research doubles every 9 years,1 and its growth particularly in medicine and health care is exponential.2 In 2010, 11 systematic reviews and 75 trials were reported to be published every day. In this fast growing era of medical research publishing, being up-to-date in the latest medical and health care evidence is important but not easy.2

Medical or health care studies often deal with similar or related research questions at different locations, with different populations, at different time points. These studies are many in number, and their results are often diverse or sometimes contradictory.3 Importantly, medical and health care decisions require up-to-date and consolidated evidence. Meta-analysis offers a strategy to collect evidence available from individual studies and quantify the effect of interventions, prevalence of diseases, or risk factors associated with diseases.4

Rigorous meta-analyses are fundamental for estimating the true effects of interventions which guide clinical and public health practice.3,5,6 Most medical journals encourage or call for aggregation of evidence using meta-analyses and the number of meta-analyses in the medical research has exponentially grown from what it was in the 1990s.3,6 Meta-analysis has also become a very important aspect of diabetes research. A simple PubMed search on diabetes and meta-analysis resulted in about 10,000 meta-analyses articles published so far (https://www.ncbi.nlm.nih.gov/pubmed/?term=diabetes+and+(meta-analys*+or+metaanalys*)).7

A well-conducted meta-analysis is a powerful tool for informing medicine and health care decisions.3,6,8 However, there are many challenges that meta-analyses authors face, for instance, clinical, methodological and statistical heterogeneity, publication bias, language barriers, outcome definitions, and statistical challenges.6,8,9 Meta-analyses of continuous outcomes are recognized to be more challenging than those with binary outcomes.10,11

In meta-analyses of continuous outcomes, mean difference (MD), standardized mean difference, and ratio of means are used as effect size measures. The choice of the two most commonly used effect size measures, ie, MD and standardized mean difference, of continuous outcomes is mainly determined by the scale of measurement. If the scale of measurement is similar as in glycated hemoglobin level (HbA1c), the MD can be used to aggregate effect sizes across studies. To pool the effect sizes of continuous outcomes, it is important to consider whether the baseline MD of the outcome data between the intervention and control groups is adequately balanced.12,13 In general, baseline imbalance can result from chance especially in small trials, or selection bias due to inadequate allocation concealment or poor randomization.14 Therefore, in meta-analysis, it is important to consider accounting for baseline imbalance and pre/post correlation.13,15–17 However, the meta-analyses available so far have by and large not taken into account a specific methodological challenge posed by baseline imbalances between groups.

None of the previous meta-analyses on digital interventions on type 2 diabetes that we have identified through scoping were adjusted for baseline imbalances and pre/post correlations. Meta-analyses of randomized control trials reporting continuous outcomes, such as HbA1c, with high baseline imbalances require adjustments using an Analyses of Covariance (ANCOVA effect size estimator, both at a pooled and individual study levels.13,15–17 ANCOVA produces a relatively more precise effect size estimate than simple analysis of change scores (SACS) and simple analysis of final values (SAFV).13,15,16

While there is methodological guidance available to employ ANCOVA effect size, the unavailability of summary data from randomized controlled trials (RCTs) or absence of individual participant data (IPD), as well as the complexity of the ANCOVA methodology, has limited its application to synthesize continuous outcomes in the medical literature.13,15–17 Although it is important to recognize that determining precise effect sizes is essential to understand the true effect of interventions to guide clinical and public health practice, to our knowledge, no study has applied ANCOVA to determine changes in HbA1c level effect sizes of digital interventions on type 2 diabetes. Therefore, this study was aimed to compute and compare changes in HbA1c effect sizes of digital interventions on type 2 diabetes using ANCOVA, SACS, and SAFV meta-analyses.

Materials and methods

This meta-analysis uses data from our recently completed systematic review and meta-analyses on digital interventions among poorly controlled type 2 diabetes mellitus (T2DM) patients.18 The detailed description of the protocol and the systematic review methods followed can be accessed elsewhere.18,19 Briefly, we searched three databases (MEDLINE via PubMed, ISI Web of Science via Thomson Reuters, and PsycINFO via OvidSP) for English language RCTs employing digital interventions among persons with poorly controlled type 2 diabetes, and published until the end of June 2017. Details of search strategy are available elsewhere.18,19 In this study, technology-based interventions delivered via mHealth, web-based applications, Personal Digital Assistant, tablet, computer, or other forms of eHealth applications were considered as digital interventions.18,20

Two authors (MK and MP) independently conducted title, abstract, and full-text screening using Covidence. Methodological quality assessment was conducted using the Cochrane Risk of Bias Assessment tool for RCTs.21 Two reviewers (MK and MP) independently conducted the risk of bias assessment by using Covidence22 and discussed quality ratings until consensus was reached. The risk of bias tool consists of seven domains which were rated as low, high, or unclear risk of bias. To rate “other sources of bias,” the recommendation by Fu et al13 was used, which focuses on the question whether the baseline distribution of participant characteristics and outcome data of both control and intervention groups are sufficiently described and balanced. Beside this, quality rating should include whether there was baseline imbalance and high rate of attrition. Therefore, quality ratings for this domain were downgraded if important baseline prognostic factors and outcome variables were not balanced in the included studies.13

Missing crude or pooled SD values were computed from the reported standard errors, confidence intervals, or from exact reported P-values using functions in Microsoft Excel. If an exact P-value was reported, depending on the statistical test used, we calculated z- or t-scores using the function norm-sinv(1 – P-value/2) or tinv(P-value, degree of freedom).13,23 Whenever SD could not be calculated from the reported data, we contacted the corresponding and the last authors. If no response was obtained, SD values were imputed using arithmetic means.13 Using this method, we computed the SD values for the follow-up mean HbA1c values from a study by Wakefield et al.24 One study25 reported median HbA1c with its range, but mean and SD values were not available from the authors. After contacting the corresponding author was not successful, we estimated the mean and SD values from the reported median and range using Hozo’s formula.26

Pre/post correlations were not reported in the majority of the studies. However, it is necessary to account for them in meta-analyzing the effect sizes of continuous outcomes.13,15 Our approach accounting for this issue was based on recommendations of previous methodological studies. If a study reported baseline and follow-up standard deviations (SDb and SDf), and standard deviation values for change scores (SDd), we computed pre/post correlation using .13 Whenever, baseline and follow-up SD were not known, correlation was estimated from the pooled SD (SDp) and change score SD (SDd) values using .16

Data syntheses and analysis

As highlighted in the previous sections, effect sizes of continuous outcomes with an inherently similar scale of measurement can be computed using three methods: SAFV, SACS, or using the ANCOVA effect size estimator.13,15–17 In comparison to MD computed through using either SAFV or SACS, the ANCOVA effect size estimator provides a more precise and unbiased effect size estimates.13,15–17 The ANCOVA effect size estimator reduces the bias that arises from baseline imbalance across the included studies and accounts for pre-/post-test correlation.13,15,17,27 In this systematic review, the absolute value of the mean baseline HbA1c differences between intervention and control groups in the included studies ranged from 0% to 0.64%, with only two RCTs having perfectly balanced MDs.28,29 Given no publication bias, if the two treatment groups are balanced, meta-analysis of baseline score differences between control and intervention groups produces a combined effect size estimate of close to zero.13 Therefore, we conducted a meta-analysis of the baseline MDs across the included studies. Further, random-effects meta-analysis of the baseline MDs resulted in a pooled HbA1c difference of 0.14% (95%CI: [−0.31, 0.59]). In addition, the computed pre/post correlation values ranged from −0.06 in a study by Ralston et al30 to 0.74 in a study by Torbjønsen et al.31 Hence, to adjust for the observed baseline imbalance and pre-/post-test correlation, ANCOVA effect size estimator was preferred to pool the mean HbA1c difference. If reported, ANCOVA effect size estimates were extracted directly from studies. Whether studies reported effect sizes using SACS, SAFV, or ANCOVA was also documented.

Assuming Xint and Xctrl are the baseline mean values of intervention and control groups while Yint and Yctrl are the follow-up mean values, the MD from SAFVs was computed as follows: SAFS = Yint − Yctrl, while MD from SACS was computed using SACS = (Yint − Yctrl) − (Xint − Xctrl). Moreover, an ANCOVA effect size estimate was modeled using ANCOVA = (Yint − Yctrl) − β (Xint - Xctrl), where β is a regression coefficient calculated by using . SDy and SDx are the pooled SD values of the treatment and control groups.13,15 The variances of the final values, change scores, and ANCOVA effect size estimates were computed using equations by Jo McKenzie et al15 and Riley et al.17 Whenever there was no possibility to compute the ANCOVA effect size estimator, the reported change scores or final values were pooled with the ANCOVA effect size estimates, following the strategy documented in the existing methodology literature.13,15,17 If a study reported both SACS and final values, the estimate with a smaller effect size was combined with the ANCOVA effect size estimates. Practically, studies having zero or negligible baseline MD, equivalent or close to equivalent MD values were obtained using any of the three methods. Therefore, for studies having no or a negligible baseline difference, adding any of the three estimates in the meta-analysis yielded comparable pooled estimates.13,15

Meta-analyses

All meta-analyses were performed using Stata version 13. Studies that were judged as homogenous in terms of participants, type of interventions, and type and scale of measurement of the outcome were subsequently combined to determine the overall pooled effectiveness of digital interventions for reducing HbA1c levels.

For each study, we calculated three effect size measures using the “black-belt” ANCOVA, SACS, and SAFV approaches to compare individual and pooled effect size differences.

For all meta-analyses, observed statistical heterogeneity across studies was assessed with Cochrane’s chi-squared test. The degree of heterogeneity was quantified using the I2 statistic. In addition to statistical heterogeneity, the diversity of studies with respect to clinical and methodological aspects was assessed to choose from random- or fixed-effects meta-analysis. Hence, random-effects meta-analysis was used for all meta-analyses.32 Sensitivity analyses were performed by 1) dropping studies with baseline imbalance (baseline mean difference ≥±0.2, or ≥±0.3) and 2) removing studies having high pre/post correlations (>±0.7 and >±0.6).

For all the three meta-analytic methods, differences in the publication bias were compared using visual inspection of the funnel plots and statistically using Egger’s test with a P-value <0.1 indicating publication bias.33 The number of missing studies in the funnel plots was estimated using the “trim and fill” imputation method to determine the changes in effect size estimate across the three methods.34

Results

Study selection and characteristics

In total, 1,669 abstracts and titles were retrieved from the database search. Twenty-two studies fulfilled the inclusion criteria.24,25,28–31,35–50 Twenty-three arms of 21 RCTs were included in the quantitative syntheses. Two studies reported results of three-armed RCTs.31,41 All of the 21 control groups of the 21 RCTs received standard or usual care. The details of the study selection procedure and the PRISMA flowchart can be accessed in our previously reported meta-analysis.18

Studies included in our review were published between 2009 and 2017. A majority (n=9) of the studies were conducted in the United States. In the 23 intervention arms of the 21 RCTs, 3,787 patients were included and followed for an average of 7.3 months (SD=3.05). Average retention rate at a study end point was 89.4% (SD=10.0, range =75% to100%), whereas attrition rate was 10.6% (SD=10.0; Table 1). One study was judged to have low risk of bias on all dimensions of the Cochrane risk of bias assessment tool.50 Four studies were considered to have a high risk of bias on three domains.36,38,43,45

Table 1

Characteristics of the included studies

Study	Location	Intervention	Intervention end points (in months)	Study population	Baseline HbA1c (%)	Included (N)	Intention to treat analysis
							N control group	N intervention group	Analyzed control	Analyzed intervention	Loss to follow-up	Loss to follow- up %	Retention %
	Mobile phone-delivered text message interventions
Agboola et al, 201635	USA	Text to move (text message)	6	Spanish- or English-speaking low-income and ethnic minorities, T2DM patients	>7.0	126	62	64	62	64	0	0.00	100.0
Arora et al, 201425	USA	Two daily text messages for 6 months. Education/motivation–one text per day, medication reminders–three per week, healthier living challenge–two per week. Trivia: Unidirectional text message	6	English- or Spanish-speaking Latino and black T2DM patients	>7.5	128	64	64	45	47	36	28.1	71.9
Capozza et al, 201536	USA	Text message (Care4Life program) for education and motivation, medication adherence, glucose control, weight, and exercise	3 and 6	No specific population, adult patients with T2DM	>7.5	156	Not reported	Not reported	35	58	11	7.0	93
Fortmann et al, 201739	Canada	Dulce Digital: An mHealth SMS-Based Intervention	3 and 6	Underserved Hispanics with poor glycemic control, T2DM patients	≥7.5	126	63	63	60	53	13	10.3	89.7
	PDA, tablet, computer, and/or smartphone delivered web-based interventions
Cho et al, 201128	South Korea	Internet diabetes management	3	No specific population, T2DM patients, South Koreans	>7.0	71	35	36	32	32	7	9.9	90.1
Egede et al, 201729	USA	Telehealth and clinical decision support system	3 and 6	Type 2 diabetes, ≥18 years, T2DM patients	≥8.0	113	59	54	44	41	28	24.8	75.2
Holmen et al, 201441 (Usual Care vs FTA-HC)	Norway	Few Touch Application (diabetes diary app with health counseling (FTA-HC)	12	No specific population, adult patients with T2DM	>7.0	100	50	50	41	40	18	18.0	82
Holmen et al, 201441 (Usual Care vs FTA)	Norway	Few Touch Application (diabetes diary app without health counseling (FTA-HC)	12	No specific population, adult patients with T2DM	>7.0	101	50	51	41	39	19	19.0	81
Kim et al, 201643	China	Internet-based glucose monitoring system	3 and 6	Male and female outpatients with T2DM patients	7.0 to 10.0	182	90	92	90	92	0	0.0	100.0
Kleinman et al, 201744	India	Smart phone app for patients and smart phone app and a web-based portal for providers	3	No specific population, T2DM patients for >6 months	7.5 to 12.5	90	46	44	33	35	22	24.4	75.6
Ralston et al, 200930	USA	Web-based care management	12	No specific population, adult patients with T2DM	>7.0	83	41	42	35	39	9	10.8	89.2
Tang et al, 201346	USA	Online disease management system	6 and 12	No specific population, adult patients with T2DM	>7.5	415	213	202	193	186	36	8.67	91.33
Tildesley et al, 201147	Canada	Internet-based glucose monitoring system (IBGMS)	3, 6, and 12	No specific population, T2DM patients	>7.0	46	23	23	23	23	0.0	0.0	100.0
Trobjohnsen 201469 (Usual Care vs FTA-HC)	Norway	Few Touch Application (diabetes diary app with health counseling (FTA-HC)	4	No specific population, adult patients with T2DM	>7.0	100	50	50	43	44	13	13.0	87
Trobjohnsen 2014 (Usual Care69 vs FTA)	Norway	Few Touch Applications (diabetes diary app without health counseling (FTA-HC)	4	No specific population, adult patients with T2DM	>7.0	101	50	51	42	44	12	12.0	88
Wang et al, 201748	China	Monitoring via computer/web/mobile phone connected to glucometer via cable	3 and 6	No specific population, T2DM patients confirmed for over 1 year	7 to 10.0	212	106	106	106	106	0	0.0	100.0
Welch et al, 201549	USA	Internet-based integrated diabetes management system	6	Latino, T2DM patients	>7.5	399	200	199	181	172	46	11.5	88.5
Wild et al, 201650	Scotland	Monitoring through computer-/web-based/mobile phone connected to glucometer via modem	9	No specific population, T2DM aged >17 years	>7.5	321	161	160	139	146	36	11.2	88.8
Telehealth (communication with provider via telephone or video)
Dario et al, 201738	Italy	Videoconferencing	12	No specific population, T2DM patients	>7.0	246	78	168	77	166	3	1.2	98.8
Hansen et al, 201740	Denmark	Videoconferencing	8	Danish speaking T2DM patients	>7.5	165	82	83	71	68	26	15.8	84.2
Khanna et al, 201442	USA	Automated telephone support with dialogic telephone card	3	Spanish-speaking patients with T2DM	>7.5	75	37	38	37	38	0.0	0.0	100.0
Liou et al, 201445	Taiwan	Web-based and videoconferencing	6	No specific population, adult T2DM patients	>7.0	95	41	54	41	54	0	0.0	100.0
Wakefield et al, 201424	USA	Tele-monitoring	3 and 6	No specific population, subjects with established T2DM	>8.0	108	55	53	43	40	25	23.1	76.9
			Mean			180.3	85.5	94.8	75.8	84.6	16.45	9.2	89.8
			Standard deviation			123.8	63.0	63.2	60.7	61.4	15.6	9.8	11.6

Abbreviations: FTA-HC, Few Touch Application (diabetes diary app with health counseling); HbA1c, glycated hemoglobin level; IBGMS, Internet-based glucose monitoring system; T2DM, type 2 diabetes mellitus.

The mean HbA1c baseline difference of the studies included in the quantitative syntheses ranged from −0.2%24,41 to 0.64%.35 Only two RCTs29,37 had a perfect baseline balance with a mean HbA1c difference of 0.0%. The pooled baseline difference was 0.14% (95%CI: [−0.31, 0.59]).

Differences in individual and pooled effect sizes

Multivariate test of means showed that the MDs estimated using the three methods across the studies were not statistically different (Hotelling T2=4.65, Hotelling, F(2, 21)=2.22, Prob>F=0.134). Visual inspection of a box plot constructed using the individual studies’ MD values obtained using the three methods also did not indicate substantial differences (Figure 1).

Figure 1

Box plots of ANCOVA, change scores, and final values MDs.

Abbreviations: ANCOVA, analysis of covariance; MD, mean difference.

The pooled mean HbA1c difference calculated using the ANCOVA approach yielded a statistically significant pooled HbA1c reduction of −0.39% (95%CI: [−0.51, −0.26]) favoring the intervention group, with considerable heterogeneity statistic (I2=80.8%; Figure 2).

Figure 2

Random-effects meta-analysis of ANCOVA adjusted MDs.

Abbreviation: ANCOVA, analysis of covariance; FTA-HC, Few Touch Application (diabetes diary app with health counseling); MD, mean difference.

MD aggregated using meta-analysis of change scores and final values also yielded statistically significant effect estimates ie, −0.39% (95%CI: [−0.51, −0.26]; Figure 3) and −0.34% (95%CI: [−0.48, −0.19]; Figure 4), respectively. The heterogeneity I2 statistics for change scores and final values meta-analyses were 32.3% and 64.5%, respectively. All of the above results are from the random effects meta-analysis. Considering the I2 statistics for change scores, we conducted a fixed-effects meta-analysis. The pooled MD was −0.37% (95%CI: [−0.468, −0.268]; Figure 5).

Figure 3

Random-effects meta-analysis of change scores.

Abbreviations: FTA-HC, Few Touch Application (diabetes diary app with health counseling); HbA1c, glycated hemoglobin level; MD, mean difference.

Figure 4

Random-effects meta-analysis of final values.

Abbreviations: FTA-HC, Few Touch Application (diabetes diary app with health counseling); HbA1c, glycated hemoglobin level; MD, mean difference.

Figure 5

Fixed-effects meta-analysis of change scores.

Abbreviations: HbA1c, glycated hemoglobin level; MD, mean difference.

Publication bias

Visual inspection of the funnel plots that were obtained using effect sizes of ANCOVA and SACS shows symmetry at the top of the plot, and there were studies missing at the bottom of the funnel plot indicating publication bias. However, a relatively symmetric funnel plot was obtained from effect sizes computed using SAFV (Figure 6).

Figure 6

Funnel plots for assessing publication bias.

Abbreviation: MD, mean difference.

Egger’s test for funnel plot symmetry obtained from the three methods suggested that there is not enough evidence for small-study effects (Table 2).

Table 2

Egger’s test for assessing publication bias

	Std_Eff	Coeff.	SE	t	P>|t|	CI
ANCOVA	Slope	−0.207	0.093	−2.23	0.037	(–0.400, −0.014)
	Bias	−1.13	0.815	−1.38	0.182	(−2.82,0.569)
SACS	Slope	−0.322	0.137	−2.35	0.028	(–0.606, −0.038)
	Bias	−0.215	0.559	0.38	0.704	(−1.38, 0.948)
SAFV	Slope	−0.237	0.153	−1.54	0.138	(−0.556, 0.083)
	Bias	−0.430	0.812	−0.53	0.602	(−2.12, 1.26)

Abbreviations: ANCOVA, analysis of covariance; Coeff, coefficient; SACS, simple analysis of change score; SAFV, simple analysis of final values; Std_Eff, standard effect; SE, standard error.

Performing the “trim and fill” test using effect sizes of ANCOVA and SAFV did not result in changes suggesting that the influence of publication bias was negligible. However, performing the “trim and fill” test using effect sizes obtained from SACS resulted in imputation of one study and the pooled HbA1c difference was changed into −0.40% (95%CI: −0.53,−0.26) using a random-effects meta-analysis.

Sensitivity analyses

We performed a sensitivity analysis by removing five studies with high baseline imbalance (≥±0.3) from the meta-analysis. The ANCOVA approach resulted in an MD of −0.41% (95%CI: [−0.54, −0.28]), while SAFV and SACS showed an MD of −0.37% (95%CI: [−0.52, −0.22]) and −0.35% (95%CI: [−0.45, −0.242]), respectively (see Supplementary material 1). Expectedly, the differences in the aggregated MDs across the three methods became less prominent when all studies having a baseline MD of ≥0.2 were removed from the meta-analyses. The ANCOVA approach resulted in an MD of −0.43% (95%CI: [−0.597, −0.27]), while SAFV and SACS show an MD of −0.40% (95%CI: [−0.59, −0.216]) and −0.39% (95%CI: [−0.55, −0.23]), respectively (see Supplementary material 1).

Additional sensitivity analyses were performed to check the effect of pre/post correlation on the effect size estimates obtained by the three methods. ANCOVA and SACS resulted in similar estimates but higher than SAFV effect size estimates after dropping studies with high pre/post correlation from the meta-analyses. Hence, ANCOVA and SACS resulted in a pooled effect size estimate of −0.40% (95%CI: [−0.53, −0.27]) and −0.40% (95%CI: [−0.54, −0.26]), respectively, while SAFV resulted in −0.34% (95%CI: [−0.49, −0.19]) after dropping one study with pre/post correlation value >0.7 (see Supplementary material 2). Similarly, dropping five studies with pre/post correlation value >±0.6 shows ANCOVA and SACS yielded a pooled effect size estimate of −0.47% (95%CI: [−0.64, −0.30]), −0.47% (95% CI: [−0.62, −0.32]) respectively, while SAFV provided −0.37% (95%CI: [−0.54, −0.20]; see Supplementary material 2). Meta-analyses performed by removing studies with both high baseline imbalance (≥±0.2%) and pre/post correlation of >±0.6 resulted in a pooled MD of −0.39% (95%CI: [−0.53, −0.26]), −0.40% (95%CI: [−0.54, −0.26]), and −0.33% (95%CI: [−0.48, −0.18]) using ANCOVA, SACS, and SAFV meta-analyses, respectively (see Supplementary material 2).

Discussion

This systematic review is the first to compare the effectiveness of digital interventions on changes in HbA1c levels by computing effect size estimates using SACS, SAFV, and ANCOVA adjusted MDs. This is also the first review to account for baseline imbalance and pre/post correlations using available robust statistical methods.

The pooled estimates obtained using the SACS, SAFV, and ANCOVA suggest clinically significant effects of digital interventions meaning reduced HbA1c levels of persons with poorly controlled T2DM. These findings reinforce the previously reported evidence regarding the beneficial effects of digital interventions.51–53 Digital interventions facilitate diabetes self-management by supporting patients with diabetes to keep track of their blood glucose, physical activity, nutrition, and other clinical and behavioral outcomes related with diabetes.54–65

Random-effects meta-analysis using ANCOVA and change scores provided identical effect size estimates. However, fixed-effects meta-analysis using change scores and comparing the results with ANCOVA and final values meta-analyses shows a slight difference in the pooled effect estimates acquired using the three methods. For all meta-analyses, the direction of the effect estimate remained unchanged. Sensitivity analyses performed by removing studies with higher baseline imbalance resulted in relatively similar pooled effect estimates across the three methods. This supports the recommendations in previous methodological literature pointing to the importance of adjusting for baseline MD using the ANCOVA approach in meta-analyses of trials with baseline imbalance.13,15,17 However, practical application of ANCOVA to synthesize effect sizes of continuous outcomes is complex due to the unavailability of summary data from RCTs or absence of IPD. Nevertheless, with available methodological guidance, it is possible to calculate ANCOVA adjusted MDs given that study authors report summary data for all intervention and control groups, such as baseline and follow-up mean values, as well as corresponding SDs, mean and SD values for changes over time, and finally sample sizes.13 Because the results of systematic reviews rely on the summary findings of individual studies, future RCTs, particularly those with baseline imbalance, need to be reported with extensive detail if they are to be included in more advanced meta-analyses. Beside this, publishing IPD with the results of the interventions will enhance transparency of the results at the primary study level and simplify evidence syntheses subsequently.

In this review, the heterogeneity I2 statistics computed using the three methods shows differences across the choice of meta-analytic methods. A lower heterogeneity estimate was obtained using SACS. Eyeball test on the forest plots of the three methods shows the confidence intervals of one study (a study by Khanna et al)42 deviates from the general pattern of the other studies on the plots. The deviation by this study gets smaller in the forest plot for SACS meta-analysis. This explains the reason why SACS has the lowest heterogeneity compared with ANCOVA and SAFV. Previously, Fu and Holmer described that there is no clear pattern of heterogeneity estimates among the three methods.12 Whether to use random-effects or fixed-effects meta-analyses can be statistically guided by the results of the I2 statistics.32 Which meta-analytic method produces decreased or increased heterogeneity statistics or whether there is a particular pattern across the three methods requires further research.

The results of publication bias assessment via an inspection of funnel plots were not consistent across the three methods. A relatively more symmetric funnel plot was constructed using the estimates obtained from the SAFV. However, funnel plots displayed for ANCOVA and SACS indicated the presence of symmetry at the top and studies missing at the lower half of the plot indicating a publication bias with regard to our sample of included studies. However, Egger’s test for all the three methods suggested that there was no evidence suggesting publication bias. Further publication bias analyses using “trim and fill” method did not impute any missing study for ANCOVA and SAFV. However, one missing study was imputed for the SACS. Literature regarding comparison of publication bias across the three meta-analytic methods is currently lacking.

Performing sensitivity analyses is important to check the robustness of estimates obtained from trials with baseline imbalance.13,66 Following this previously stated recommendation, we performed sensitivity analyses by removing studies with high baseline imbalance values. These analyses show that relatively comparable pooled estimates were obtained using ANCOVA, SACS, and SAFV meta-analyses. These results, in line with existing studies, suggest the importance of accounting for baseline imbalance by aggregating continuous outcome measures.12,13,27,67

ANCOVA and SACS yielded similar pooled estimates after removing studies with high pre/post correlation values from the meta-analyses. Similar to our result, existing evidence shows that when the value of pre/post correlation gets closer to 1.0, ANCOVA and SACS tend to produce similar effect size estimates.12,13,27 Inspecting individual study effect sizes obtained using the three methods also shows that, ANCOVA and SAFV tend to produce similar effect size estimates as correlation values approach zero.68

Limitations

The study has limitations. First, our search was limited to three databases only: MEDLINE, ISI Web of Science, and PsycINFO. We tried to check whether this had an impact on our search output. There was no noticeable difference compared with our preliminary search in additional search databases, such as EMBASE and CINAHL. Hence, we decided to focus on the three included databases, especially considering the workload to request and impute missing data which was crucial to answer our research questions of interest. Second, we did not consider unindexed databases and gray literature. Third, most of the studies did not report ANCOVA effect sizes. Our ANCOVA effect size calculation mainly relies on imputation from the reported data using robust statistical methods, but uncertainties remain.

Conclusion

All three meta-analytic methods show a significant effect of digital interventions on changing HbA1c levels. Analysis on the effect sizes computed for each study using the three methods did not differ significantly. However, some differences were noted among the pooled effect sizes applying different statistical methods by accounting for baseline imbalances of the outcome. Authors of future systematic reviews and meta-analyses should consider using ANCOVA to estimate effect sizes, at least for interventions with baseline imbalance. However, we recognize the statistical challenge of computing ANCOVA effect sizes, if the necessary data are not reported for individual studies. Hence, future RCTs, particularly those with baseline imbalance, should report ANCOVA effect sizes. In addition, publishing IPD along with the changes in the outcomes as a result of intervention participation is helpful to simplify robust evidence syntheses.

Data sharing statement

The data collected for this study can be received from the corresponding author.