Introduction, comparison, and validation of Meta-Essentials: A free and simple tool for meta-analysis.

We present a new tool for meta-analysis, Meta-Essentials, which is free of charge and easy to use. In this paper, we introduce the tool and compare its features to other tools for meta-analysis. We also provide detailed information on the validation of the tool. Although free of charge and simple, Meta-Essentials automatically calculates effect sizes from a wide range of statistics and can be used for a wide range of meta-analysis applications, including subgroup analysis, moderator analysis, and publication bias analyses. The confidence interval of the overall effect is automatically based on the Knapp-Hartung adjustment of the DerSimonian-Laird estimator. However, more advanced meta-analysis methods such as meta-analytical structural equation modelling and meta-regression with multiple covariates are not available. In summary, Meta-Essentials may prove a valuable resource for meta-analysts, including researchers, teachers, and students.

INTRODUCTION

The term meta‐analysis refers to a range of methods to provide an overview of effects for the relationship between an independent and a dependent variable.1, 2 In this paper, we present a new tool for meta‐analysis: Meta‐Essentials, which functions as a set of spreadsheet workbooks. The tool can be downloaded from the accompanying website (www.meta‐essentials.com), which also provides an elaborate (online) user manual,3 a guide on how to interpret the results of meta‐analysis,4 and answers to frequently asked questions. Meta‐Essentials is suitable for meta‐analysis of a wide range of effect sizes as it automatically calculates effect sizes from commonly reported statistics. The basic results of meta‐analysis are presented using a forest plot and accompanying statistics, including confidence and prediction intervals (see Figure 1 for an example). The tool also supports additional analyses including subgroup analysis, moderator analysis, and various publication bias analyses.

Figure 1

Forest plot in Meta‐Essentials [Colour figure can be viewed at wileyonlinelibrary.com]

There are many existing tools to aid researchers in conducting a meta‐analysis. Each of the tools is suitable for a specific purpose and limited in other areas. Most prominently, some programs are not freely available (eg, CMA and MIX Pro) and others require syntax for conducting meta‐analysis (eg, packages for R, commands for Stata, and syntaxes for SPSS). These 2 aspects limit the tools' suitability for some users. Although there are other software tools that are available free of charge and do not require programming skills (eg, OpenMeta[Analyst] and RevMan), we found that they have some limitations of their own, which we will discuss in detail later.

In summary, we think Meta‐Essentials is particularly useful as a tool that is available free of charge, does not require programming skills, is relatively comprehensive as it handles many effect sizes and standard meta‐analysis methods, and is adaptable and extendable to their preferences. On the other hand, users may find Meta‐Essentials of limited use for more advanced meta‐analysis methods, such as meta‐analytical structural equation modeling and meta‐regression with multiple covariates, and for more accurate estimators of between‐study variance (eg, restricted maximum likelihood and Paule‐Mandel).

Meta‐Essentials itself is available free of charge and open source (licensed under Creative Commons BY NC SA, http://creativecommons.org/licenses/by‐nc‐sa/4.0/). Meta‐Essentials works with Microsoft Excel, which requires a license, but it can also be used with the freely available WPS office 2016 Free (https://www.wps.com/office‐free) or Microsoft Excel Online (https://office.live.com/start/Excel.aspx).

In this paper, we will describe the features and limitations of Meta‐Essentials in detail. We first introduce the design of the tool as a set of workbooks (Section 2). Next, we compare its features against other known meta‐analysis tools (Section 3). Furthermore, we describe how the tool was validated (Section 4) and finally discuss the usefulness and applicability of Meta‐Essentials (Section 5). A worked example of a meta‐analysis in the tool is provided in Appendix A.

INTRODUCING META‐ESSENTIALS

Meta‐Essentials is a set of 7 workbooks each designed to serve a special purpose. The structure of all workbooks is similar. Each workbook consists of 6 sheets. The input sheet is for inserting data. Next, there are 4 output sheets: one for the main meta‐analysis (forest plot), one for subgroup analysis, one for moderator analysis, and one for several publication bias analyses. All the calculations and procedures between the user‐provided inputs and the tool‐generated outputs are separately available in the calculation tab.

Each workbook is designed for different types of effect sizes, ie, a set of workbooks, rather than a single workbook, for 2 main reasons. First, different types of research designs can be used to investigate a relationship. Each research design leads to a different type of effect size, and there are many different effect size measures.5 For example, let us consider the following research question: What is the effect of acetaminophen (X) on headache severity (Y)? One researcher may conduct an experiment by providing one group with acetaminophen and one group with a placebo and measure headache severity in both groups. The difference between headache severity in the treatment and control groups is one answer to the research question. However, another researcher may conduct an observational study by surveying a population of patients on the amount of acetaminophen intake and the severity of the headaches they experience subsequently. The correlation between intake of acetaminophen and headache severity provides another answer to the research question, even though no strong causal inferences can be drawn from this observational study. The 2 research designs (of the d‐family and r‐family, respectively) lead to different types of effect sizes because they present different types of answers.5 Second, studies with the same research design often present their results using different statistics, which makes effect size calculations from input data more complex. As we aimed to design a simple tool for meta‐analysis, we developed several workbooks to serve a different effect size type and to enable easy effect size calculation from a wide range of inputs. Therefore, users of Meta‐Essentials cannot “mix and match” continuous, binary, and correlational data in one meta‐analysis, in contrast to, for example, CMA.

The workbooks, other than the generic Workbook 1, are organized in 2 families: the d‐family and the r‐family,5 see Table 1. The d‐family (Workbooks 2, 3, and 4) applies when effect sizes indicate group differences, as in experimental designs. Workbook 2 is designed to meta‐analyze studies that compare groups on dichotomous outcomes or binary data. Effect sizes for these types of data are odds ratios, risk ratios, and risk differences. Workbooks 3 and 4 are designed to meta‐analyze studies that compare groups on continuous outcomes. Effect sizes for these types of data are standardized mean differences: Cohen's d and Hedges' g. Workbook 3 applies when the treatment and control groups are independent, ie, different people across the treatment and control groups. Workbook 4 applies when groups are dependent, as in paired (pre‐post) experimental designs, ie, the same people before and after their treatment. Separate workbooks for these types are required due to differences in the calculation of the effect size. Note that raw (unstandardized) mean differences are not automatically calculated in Workbooks 3 and 4; users can use Workbook 1 for those applications, provided the outcomes are measured on the same scale.

Table 1

The seven Meta‐Essentials workbooks

	File name	Type of effect	Effect size measure	Example
Generic	1 Effect size data.xlsx	Any, as long as directly comparable	Mean Difference (for example)
d‐family	2 Differences between independent groups ‐ binary data.xlsx	Difference between two independent groups with binary outcome	Odds ratio, risk ratio, or risk difference	Counts of patients that survived or died cancer after an experimental versus control treatment.
	3 Differences between independent groups ‐ continuous data.xlsx	Difference between two independent groups with continuous outcome	Standardized mean difference: Cohen's d or Hedges' g	The difference between the performance of sports teams that received intensive training and those that did not receive intensive training
	4 Differences between dependent groups ‐ continuous data.xlsx	Difference between two dependent groups with continuous outcome	Standardized mean difference: Cohen's d or Hedges' g	The difference between the performance of sports teams before and after receiving intensive training
r‐family	5 Correlational data.xlsx	Correlation between two variables	(Zero‐order) correlation coefficient	The relationship between age and income
	6 Partial correlational data.xlsx	Relation between two variables, controlled for other variable(s) in both predictor and outcome	Partial correlation coefficient	The relationship between age and income, controlled for socio‐economic status, assuming socio‐economic status is related to both age and income
	7 Semi‐partial correlational data.xlsx	Relation between two variables, controlled for other variable(s) in outcome	Semi‐partial correlation coefficient	The relationship between age and income, controlled for education, assuming education is related to income, but not age

The r‐family (Workbooks 5, 6, and 7) applies when effect sizes indicate association between variables. If both independent and dependent variables are continuous, a measure of association is the Pearson product moment correlation coefficient, but other types exist as well.5 Workbook 5 is designed to meta‐analyze correlation coefficients, Workbook 6 is for partial correlations, and Workbook 7 is for semipartial correlations. The latter 2 types of correlation coefficients are applied when zero‐order correlations are not reported in the primary articles, and data are instead provided in the form of regression models and tables6, 7. Since regression coefficients are sensitive to the inclusion of (different) control variables between studies, it is preferable to conduct meta‐analysis on (semi)partial correlation coefficients.6 In Workbook 5, Fisher's r‐to‐z transformation (and back) is automatically applied8; in Workbook 6, this is provided as an option, but more research is required to validate this transformation for partial correlations.

Researchers should select the workbook that is most appropriate for their data, based on Table 1. The user can insert data on the input tab, and the workbooks automatically calculate the appropriate effect sizes (when necessary). Researchers can also add information on study‐level characteristics in the respective columns that will subsequently be used in subgroup or moderator (meta‐regression) analysis. Appendix A provides a worked example of a meta‐analysis in Meta‐Essentials.

STRUCTURED COMPARISON OF META‐ANALYSIS TOOLS

In this section, we compare the features of Meta‐Essentials to other available software tools, to examine the contribution of the tool and describe its limitations. Since the publication of previous reviews of meta‐analysis tools,9, 10 several tools have been updated and new tools developed. In this comparison, we review features similar to Bax et al.9 and Schmid et al.10 We retrieved the required information from these 2 previous reviews, documentation accompanying each tool (websites, books, articles, user guides, etc) and by performing meta‐analyses with each tool.

Meta‐analysis tools

To determine which tools besides Meta‐Essentials to include in the comparison, we used 2 criteria. First, we included tools that scholars have been using for research and exclude tools that primarily designed for educational purposes, such as MIX Lite with only built‐in data sets. Second, we included tools that scholars from multiple disciplines have been using frequently and recently and exclude therefore, for instance, MetAnalysis, MetaWin, PhyloMeta, WEasyMA, and macros for SAS. We thus include the following tools (in alphabetical order): CMA,1 commands for Stata,11 MIX Pro,12 OpenMeta[Analyst],13 Review Manager (RevMan),14 packages for R (meta15 and metafor16), and syntaxes for SPSS.17, 18

Comparison

We assessed the basic characteristics, supporting material, input, method settings, and output of each tool. Each of these aspects is important to examine the usefulness and applicability of tools for meta‐analysis. Appendix B provides a detailed overview of the features of the software for meta‐analysis included in our comparison.

Basic characteristics

A clear difference between the various tools is whether they are stand‐alone tools or whether an additional tool is required to use the meta‐analysis software. Stand‐alone tools can be commercial (CMA) or freeware (OpenMeta[Analyst] and RevMan). Tools developed on top of other software programs are also available: plugins for Microsoft Excel (MIX Pro), packages for R (meta and metafor), syntaxes for IBM SPSS Statistics,17, 18 and commands for Stata.11 These tools themselves are available for free but operate on commercial statistical software (except packages for R, which are completely free of charge). Meta‐Essentials can be used with the freely available WPS Office Free or Excel Online, or the commercial Microsoft Excel. Table 2 provides an overview of the tools based on whether they are free or commercial and on whether they have a graphical user interface or rely on syntax.

Table 2

A categorization of software for meta‐analysis

	Freeware	Freeware on Commercial Platform	Commercial
Graphical user interface	OpenMeta13   RevMan19   WPS Office/Excel Online: Meta‐Essentials (this paper)	Excel: Meta‐Essentials (this paper)	CMA41   MIX Pro12
Syntax	R: meta15   R: metafor16	Stata11   SPSS17, 18

All tools run on Microsoft Windows, although OpenMeta[Analyst] is not available for 32‐bit versions of Microsoft Windows. Most tools, except CMA and MIX Pro, also run on Mac OS.

CMA and MIX Pro can be run on Mac OS using a Windows emulator.

Supporting material

General information about the tools can be found in books or articles. Most programs also offer more specific and technical documentation, such as tutorials, help, formulae, and FAQs, online.

Input

All programs can conduct meta‐analysis using precalculated effect sizes and their standard errors, ie, “generic” effect sizes. In addition, some programs are able to calculate effect sizes based on a range of input data. MIX Pro, OpenMeta[Analyst], and RevMan include this feature for effect sizes of the d family but offer only limited support for calculating effect sizes of the r family, as they lack the commonly applied Fisher r‐to‐z transformation and effect size calculations for (semi)partial correlations. The syntaxes for SPSS can only process precalculated effect sizes with their standard errors.

CMA has the unique feature of “mixing and matching” effect sizes from different effect size families. However, CMA's developers readily acknowledge (Borenstein et al.1 p. 45) that one needs to make certain assumptions for these conversions that are not always appropriate.

Method settings

Next, we investigated how the tool is operated, possibly adapted, and which methods for estimating the weights of individual studies are available. Tools that are controlled using syntax require some programming skills. Conversely, tools with a graphical user interface require no programming skills; see Table 2. Some of these graphical user interface tools (specifically, CMA, MIX Pro, and RevMan) have relatively limited possibilities of adapting or extending procedures and (graphical) output. Meta‐Essentials is fully adaptable by anyone with modest Microsoft Excel knowledge, and OpenMeta[Analyst] can also be adapted but this requires programming skills (source code publicly available on GitHub). Tools based on general statistical software can inherently be extended and adapted using the full capabilities of the statistical software.

Regarding the featured methods for estimating between‐study variance, all tools provide the DerSimonian‐Laird method of moments estimator.20 However, other estimators of between‐study variance achieve more satisfactory performance across a range of situations.21, 22, 23 Based on previous simulation studies and empirical investigations, Veroniki et al23 recommend the Paule‐Mandel estimator,24, 25 supported by meta(for), MIX Pro, and OpenMeta[Analyst], and the restricted maximum likelihood estimator,24, 25 supported by CMA, commands for Stata, metafor, OpenMeta[Analyst], and the syntax for SPSS by Wilson. Meta‐Essentials only provides the DerSimonian‐Laird estimator because other estimators involve multiple iterations, which Microsoft Excel does not support unless these are programmed using macros, which we wanted to avoid for transparency and security reasons.

For dichotomous data (ie, results presented in 2 × 2 tables) 3 common methods of weighting effect sizes exist (Inverse Variance, Mantel‐Haenszel, and Peto). Most tools offer all 3 weighting methods, except MIX Pro (which does not offer the Peto method) and the syntaxes for SPSS (which only offer the inverse variance method). A second choice when meta‐analyzing dichotomous data is the choice of effect size to conduct the meta‐analysis on. Deeks26 and Fleiss and Berlin27 show the mathematical properties of the odds ratios to be preferable for meta‐analysis, compared to risk ratios or risk differences. However, the latter effect sizes can be more easily interpreted by both academics and practitioners26, 28, 29 and researchers often confuse the odds ratio with the risk ratio.30 Therefore, some authors suggest conducting meta‐analyses in odds ratios and subsequently transforming the outcomes into effect size measures that can be easier understood.1, 27, 31 Implementing such a method requires the transformation of the combined effect size in odds ratio into the risk ratio or risk difference, using, eg, the substitution method.30, 32 Subsequently, the confidence and prediction intervals need to be transformed. This can be done, assuming that a statistical test of the overall effect would produce the same result, regardless of the effect size measure used in the meta‐analysis. This procedure has not been extensively validated and should therefore be used cautiously, especially when baseline risk in individual studies is high, and when odds ratios are large.33 It has been included in Meta‐Essentials (the exact formulas are described by van Rhee and Suurmond34) but not in any of the other packages.

Output

By default, most meta‐analysis tools provide a confidence interval (CI) of the overall effect based on a normal distribution. However, this distribution is not always accurate because it disregards the uncertainty of the heterogeneity estimator (τ ), which leads to too narrow CIs especially when sample sizes (N) are small or the number of studies (k) is small.35 Therefore, some tools allow the user to choose the Student's t distribution for CIs (CMA and MIX Pro). The nominal coverage of CIs can be further improved by using the Knapp‐Hartung adjustment (KNHA) (also known as weighted variance or Hartung‐Knapp‐Sidik‐Jonkman method,35, 36). It provides better coverage of CIs than the normal distribution, quantile approximation, or Student's t distribution.35 The weighted variance method, using the KNHA with a Student's t distribution to estimate the CI of the overall effect, is available in OpenMeta[Analyst], in meta and metafor, in Stata, in the regression module of CMA 3.0, and the default in Meta‐Essentials.

Forest plots that show the dispersion of effect sizes and accompanying prediction intervals that express this dispersion are key to state‐of‐the‐art meta‐analysis.4, 37, 38 All tools, except the macros for SPSS, provide a forest plot with a few easy steps. However, prediction intervals are not supported by all tools. The prediction interval offers “a convenient format for expressing the full uncertainty around inferences, since both magnitude and consistency of effects may be considered.”39 , p. 139 If we assume that all studies provide estimates of different true effects, we must also assume that no single overall effect size can express these different true effects' best.39 Therefore, the prediction interval accurately embraces the notion of heterogeneity and the dispersion of true effects.38 Meta‐Essentials provides the prediction interval by default and automatically includes it in the forest plot (see the green line in Figure 1). Prediction intervals are not available in CMA, MIX Pro, and syntaxes for SPSS.

CMA provides a separate Excel workbook on its website to calculate prediction intervals based on CMA output.40

All tools offer subgroup analysis, which allow a user to run separate meta‐analyses on subsets of the included studies. All tools, except RevMan, also feature meta‐regression, although Meta‐Essentials and MIX PRO only offer it for a single covariate.

Publication bias analyses help researchers to estimate the threat of unpublished or undiscovered research reports for the validity of a meta‐analysis. A basic funnel plot is available in most programs except in OpenMeta[Analyst]. More (sensitivity) tests and plots are available in all programs except in OpenMeta[Analyst], RevMan, and syntaxes for SPSS. In Meta‐Essentials, packages for R, syntaxes for SPSS, and commands for Stata, additional plots and tables can be generated based on user specifications.

VALIDATION

We extensively validated Meta‐Essentials by comparing the results of a meta‐analysis with CMA (v. 2.041), the metafor package for R (metafor v.1.9‐816; R v.3.2.542), and MIX Pro (v. 2.0.1.443). To validate the formulas and results from Meta‐Essentials, we compared the results of equivalent analysis across these programs based on 5 data sets: generic effect sizes, binary data, group differences between independent and dependent groups, and correlation coefficients. The data sets contain fictitious but realistic data from 12 to 18 “studies,” and Appendix C provides an example of such a data set for correlation coefficients. The other data sets are similar if not equal to the default entries in the input tabs provided in the distribution of Meta‐Essentials. We ran a meta‐analysis on each of these data sets using the 4 programs and compared the results to the extent possible. In all cases, weights (both fixed and random), heterogeneity (DerSimonian‐Laird), overall effect size, CI (t distribution; KNHA ), prediction interval, subgroup analysis, and meta‐regression (one covariate) were exactly equal (to at least 6 decimals).

For validation purposes, we examined the results in metafor using the Knapp‐Hartung adjustment (KNHA) using a Student's t distribution. In MIX Pro and CMA, results were different because of the employed standard normal distribution, but recalculation using the KNHA shows equivalent results.

Only in metafor and Meta‐Essentials, not available in MIX Pro and CMA.

Publication bias analyses (fixed effect) led to small differences among the programs, also between MIX Pro, CMA, and metafor. Funnel plots appear the same, except in MIX Pro, where CIs are plotted around zero and not around the combined effect size. Trim and fill methods are equal in CMA and in Meta‐Essentials but sometimes slightly different in MIX Pro and metafor due to the numbers of iterations. Egger's regression test is exactly equal for all programs. Begg and Mazumdar's rank correlation test is exactly equal for MIX Pro, CMA, and Meta‐Essentials, but metafor automatically corrects Tau for both ties and continuity that leads to small differences. Standardized residuals and their histograms and the Gailbraith (radial) plot are exactly equal in Meta‐Essentials and metafor but are not available in CMA. MIX Pro instead plots a standard normal distribution by default and does not calculate the width of bins for standardized residuals histograms. Normal quantile plots are not the same between the tools: MIX Pro does not plot all the data points; CMA does not provide a normal quantile plot; and Meta‐Essentials calculates normal quantiles based on (rank‐1/3)/(k + 1/3), which is considered better than (rank‐0.5)/k as incorporated in metafor.44 The l'Abbe plot, applicable to binary data only, appears to be the same in MIX Pro, metafor, and Meta‐Essentials but is not available in CMA. Rosenthal's Failsafe N (CMA, metafor, Meta‐Essentials) and Orwin's Failsafe N (CMA, Meta‐Essentials) are also equal.

We could not directly validate the effect size calculations for (semi)partial correlations, as these are not available in any of the other tools. However, we checked these effect size calculations in a spreadsheet obtained through personal communication with Aloë (based on the formulas by Aloë6 and Aloë and Becker7).

We further validated the tool by conducting an actual (nonfictitious) meta‐analysis on the effect of communication (face‐to‐face vs virtual) on team performance, which was run as a data set in all 4 programs. Results revealed no other differences between tools than those previously described. Finally, numerous meta‐analyses have been conducted with the tool and no problems have been reported to us, some of which have been published45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66.

An updated list is maintained at http://www.erim.eur.nl/research‐support/meta‐essentials/references‐to‐meta‐essentials.

DISCUSSION

In this paper, we have introduced the Meta‐Essentials workbooks for meta‐analysis in Microsoft Excel. In the previous sections, we compared the features of this software to other tools for meta‐analysis and provided more information on the validation of the program. In this final section of the paper, we discuss our conclusions on the usefulness and applicability of Meta‐Essentials as a tool for meta‐analysis.

First, Meta‐Essentials is a comprehensive tool for meta‐analysis, in the sense that many features have been incorporated that are also available in other tools or that have been suggested as methods for meta‐analysis. Some of these features are subject to debate or are not appropriate in some contexts. For example, researchers disagree as to whether and which publication bias analyses can accurately detect (or even remedy) the threat of unpublished studies with small effect sizes (see Rothstein et al67). In Meta‐Essentials, these publication bias analyses can be conducted and can even be run using a random effects model, which is often not appropriate.68, 69 Furthermore, the use of a substitution method between odds ratios and risk ratios, as discussed in Section 3.2.4, has not been extensively validated (yet) and is not appropriate when baseline risk or odds ratios are high.33 It is the user's responsibility to ensure that the settings and parameters of statistical software are appropriate in their context.

Second, Meta‐Essentials operates as a “black‐box” by default, meaning that users do not observe the procedures or formulas in the main output tabs. Nonetheless, the procedures and formulas are openly available in the “calculation” tab. We recommend unexperienced users not to make changes to the formulas or procedures. However, as the tool is available as open source, advanced users and experienced meta‐analysts can adapt the formulas and build added functionality to the tool.

Third, we recommend the tool for use in both research and teaching. For research, Meta‐Essentials is an excellent choice for users who are not familiar with general statistical software and programming language, those looking for a free, yet comprehensive meta‐analysis tool, and users that want to “quickly” explore the literature on their topic of interest. Meta‐Essentials has indeed been used for recently published meta‐analyses (see Section 4). Additionally, Meta‐Essentials can be used as an educational instrument to teach students in “new statistics” and meta‐analytical thinking (as suggested by Cumming and Calin‐Jageman70). We have also used the tool in an undergraduate course on research methods, where student teams conducted small‐scale meta‐analyses of about 5 to 10 studies. We found that students quickly learn the purpose and usefulness of meta‐analysis, as others have also reported,71 and that a free and simple tool for meta‐analysis supports this learning process.

Fourth, we readily admit that Meta‐Essentials is not the best tool currently available on the market for all users and/or purposes. Users already familiar with Stata or R can easily use such general purpose statistical software.11, 72 RevMan and OpenMeta[Analyst] are 2 alternative free meta‐analysis tools that can be used without programming skills. Specific limitations of Meta‐Essentials are that it lacks capabilities for more advanced analyses, such as general linear models, network meta‐analysis, meta‐analytical structural equation modeling, hierarchical subgroup analyses, and meta‐regression with multiple covariates, most of which can easily be conducted using a variety of packages in R or commands in Stata. Additionally, Meta‐Essentials uses the DerSimonian‐Laird estimator of between‐study variance for the random effects models, which has been shown to be suboptimal in some situations. Other tools provide other between‐study variance estimators to choose from.

In conclusion, we present Meta‐Essentials as a new tool for meta‐analysis. It is a set of workbooks for Microsoft Excel that is available free of charge and does not require programming skills. It is comprehensive because it can handle many effect size types and meta‐analysis methods and is adaptable and extendable to user preferences. However, some more advanced meta‐analysis methods are not available. Therefore, it provides sufficient capabilities for conducting meta‐analysis for many users, including researchers, teachers, and students.

Table A1

The example data set

Study Name	Correlation	Number of Subjects	Continent	Data Collection/Pub Year (Mean‐Centered)
Tessarolo74	0.25	154	Europe	−2
Parker et al75	0.35	116	North‐America	−1
Lin76	0.23	111	Asia	0
Koufteros et al77	0.23	191	North‐America	1
Perols et al78	0.09	116	Europe	4
Yan and Dooley79	−0.02	214	North‐America	1
Lau et al80	0.29	251	Asia	1
Yan and Kull73: China	0.04	210	Asia	1
Yan and Kull73: United States	0.02	206	North‐America	1
Brulot81	0.17	137	Europe	−2
Yan82	−0.04	425	North‐America	1
Laseter and Ramdas83	0.11	50	North‐America	−10

Table B1

Features of software for meta‐analysis

		Meta‐Essentials	CMA	Commands for Stata Discussed by Palmer and Sterne11	meta and metafor package for R
Basic characteristics	Version	1.1	3.3.070	Versions available from the statistical software components archive on April 20, 2017. eg, metan 3.04, metareg 2.6.1, and metafunnel 1.0.2. Stata 14.1	meta 4.8–0 metafor 1.9–9. R 3.3.3
	Website	www.meta‐essentials.om	www.meta‐analysis.om	http://www.stata‐press.com/books/meta‐analysis‐in‐stata/	meta: http://meta‐analysis‐with‐r.org metafor: http://www.metafor‐project.org/
	Freeware/commercial	Freeware	Commercial	Freeware	Freeware
	Prerequisite software	Microsoft Excel (commercial)	None	Stata (commercial)	R (freeware)
	Operating systems	Windows, Mac OS	Windows	Windows, Mac OS, Linux	Windows, Mac OS, Linux
Supporting material	Documentation	User manual, website, this paper	Borenstein et al,1 website, and tutorials	Palmer and Sterne,11 website, and help function in Stata	meta: Schwarzer et al,74 website, and help function in R metafor: Viechtbauer,16 website, and help function in R
Input	Effect size calculation	d and r families	d and r families	d and r families	d and r families
Method settings	User interface	Graphical user interface	Graphical user interface	Syntax	Syntax
	Adaptability	Full	Limited	Full	Full
	Between‐study variance estimators	DL	DL, (RE)	DL, EB, HE, (RE), SJ	meta: DL, PM metafor: DL, EB, HE, HS, PM, (RE)ML, SJ
	Weighting methods	IV, MH, Peto	IV, MH, Peto	IV, MH, Peto	IV, MH, Peto
Output	Confidence and prediction interval	Both	CI only	Both available, CI default	Both available, CI default
	Confidence interval distributions	KNHA Student's t	Normal or Student's t (KNHA Student's t in meta‐regression)	Normal or KNHA Student's t	Normal or KNHA Student's t
	Automated forest plot	Yes	Yes	Yes	Yes
	Subgroup‐analyses	Yes	Yes	Yes	Yes
	Meta‐regression	Yes, single covariate	Yes, multiple covariates	Yes, multiple covariates	Yes, multiple covariates
	Funnel plot and trim‐and‐fill	Yes	Yes	Yes	Yes
	Failsafe‐N	Yes	Yes	Yes	meta: No metafor: Yes

Abbreviations: Between‐study variance estimators include the following: DL, DerSimonian‐Laird; (RE)ML, (Restricted) Maximum Likelihood; PM, Paule‐Mandel; HE, Hedges; SJ, Sidik‐Jonkman; and EB, Empirical Bayes. Models for calculating the weights of individual studies include the following: IV, inverse variance; MH, Mantel‐Haenszel; and Peto. CIs can be based on the standard normal distribution, the Student's t distribution, or the Student's t distribution with KNHA; see Section 3.2.5.

Table C1

Example of a fictitious data set for validation purposes

#	ID	Correlation	N	Subgroup	Moderator
1	aaaa	0.976	100	AA	15
2	bbbb	0.947	130	AA	16
3	cccc	0.956	80	AA	13
4	dddd	0.967	300	AA	18
5	eeee	0.050	95	BB	20
6	ffff	−0.537	90	BB	14
7	gggg	0.964	120	AA	19
8	hhhh	0.947	130	AA	13
9	iiii	0.380	80	BB	19
10	jjjj	0.970	240	AA	22
11	kkkk	−0.380	90	BB	17
12	llll	−0.462	100	BB	18