Error Analysis and Propagation in Metabolomics Data Analysis.

Error analysis plays a fundamental role in describing the uncertainty in experimental results. It has several fundamental uses in metabolomics including experimental design, quality control of experiments, the selection of appropriate statistical methods, and the determination of uncertainty in results. Furthermore, the importance of error analysis has grown with the increasing number, complexity, and heterogeneity of measurements characteristic of 'omics research. The increase in data complexity is particularly problematic for metabolomics, which has more heterogeneity than other omics technologies due to the much wider range of molecular entities detected and measured. This review introduces the fundamental concepts of error analysis as they apply to a wide range of metabolomics experimental designs and it discusses current methodologies for determining the propagation of uncertainty in appropriate metabolomics data analysis. These methodologies include analytical derivation and approximation techniques, Monte Carlo error analysis, and error analysis in metabolic inverse problems. Current limitations of each methodology with respect to metabolomics data analysis are also discussed.

Introduction

Error analysis is the detection, identification, and quantification of different types of uncertainty present in measurements and the propagation of this uncertainty through mathematical calculations and procedures. This definition associates the term error more with precision and less with mistake or accuracy. As such, error analysis plays a fundamental role in describing the degree of confidence in results derived from experiments across a variety of disciplines. Naturally, the importance of error analysis has grown with the increase in the number and heterogeneity of measurements obtained from newer, high-throughput, omics-level technologies.

The increase in heterogeneity is particularly problematic for metabolomics, which has more heterogeneity than other omics technologies due to the much wider range of molecular entities detected and measured: thousands of distinct metabolites versus single classes of repetitive linear polymers like DNA, RNA, and proteins. Also within the context of metabolomics, error analysis has several fundamental uses: i) the improvement of experimental design, ii) the quality control of experiments, iii) the selection of appropriate statistical methods, and iv) the determination of uncertainty in results. This review will introduce the fundamental concepts of error analysis as they apply to a wide range of metabolomics experimental designs and will discuss current methodologies for determining the propagation of uncertainty in appropriate metabolomics data analysis, with increasing level of statistical detail as the review progresses.

Basic statistical terminology

Owing to the confusion and misconceptions about statistical definitions within published metabolomics literature, I begin by concisely defining the main statistical terminology and concepts used throughout the rest of this review, but this paper is no substitute for more in-depth reading and study [1-3]. As shown in Table 1, the estimate of an expected value is usually represented as the mean or average of repeated measured values () of a given measured variable x. The median, defined as the middle value in a sorted list of repeated measured values, is another common estimate of an expected value, often preferred when the distribution of the measured variable is significantly skewed (non-symmetric). The variance represents the spread of these repeated measured values around this mean. The standard deviation σx is just the square root of the variance. Another useful term for describing uncertainty is the standard error (also known as the standard deviation of the mean), which is a probabilistic description of how close the mean is to the expected value.

Table 1

Common statistical terms and their mathematical definitions.

Term	Equation
Mean (estimate of the expected value)	x¯=∑iNx_iN
Variance	σx2=∑iN(x_i-x¯)^2N-1
Standard Error	SE_xorσ_x¯=σ_xN
Covariance	σxy2=∑iN(x_i-x¯)(y_i-y¯)N-1
a(Pearson's) Correlation	r_xy=∑iN(x_i-x¯)(y_i-y¯)(N-1)σ_xσ_y

Pearson's correlation coefficient [4]

A related term is a confidence interval, which identifies a range that includes the expected value at some level of confidence (typically 95% or 99%). A multidimensional generalization of a confidence interval is a confidence region, typically approximated with an elliptical shape. Both confidence intervals and confidence regions are especially useful descriptions of uncertainty when the probability distribution of the measurement(s) is unknown and clearly non-normal. However, the term confidence interval should not be confused with a tolerance interval which is more analogous to standard deviation and describes a range that includes a certain proportion of the population. Next, covariance describes how two measured variables vary together. And correlation rxy describes the dependence between two measured variables. Both of these terms are useful in describing the relationship between measured variables.

Finally, the term statistical power is the probability that a statistical test will properly reject the null hypothesis and not make a false negative decision (Type II error). A null hypothesis is a falsifiable assertion. A statistical test is a method for making decisions from data by deciding whether to reject a null hypothesis at a certain level of significance. Statistical methods can be divided into parametric methods that assume an underlying probability distribution for the data and non-parametric methods that make no such assumptions and treats data from a categorical or ordinal (having an order or ranking) perspective. A p-value is the probability that a true null hypothesis would have an observed value at a certain extreme or worse. And a Type I error is the improper rejection of a null hypothesis, also known as a false positive.

Types of variance and error

The major divisions of variance in bioanalytical experiments are biological versus analytical variance, which categorize the source of the variance (Figure 1). Biological variance arises from the spread of measured values observed from multiple biological samples due to differences in individuals. Analytical variance arises from the spread of measured values observed from multiple measurements made from the same biological sample, including all technical steps from sample acquisition to primary analytical data collection. More often, the biological variance is significantly larger than the analytical variance. The major divisions of error are systematic versus nonsystematic (random) error, which describe the type of error. Systematic error is a type of uncertainty not revealed by repeated measurements and represents biases in measurements that must be tested for separately in order to address or correct. While systematic error does not appreciably affect variance, it can affect covariances and correlations between measured variables. Nonsystematic error, also known as error variance, is the experimental uncertainty revealed by repeated measurements and can be reliably estimated by statistical methods.

Figure 1

The major divisions of variance, error, and bias in bioanalytical experiments. The expanding cone represents the effects of major types of variance and error on the spread (red line) and distribution of measured values. The major divisions of variance by source are biological versus analytical variance. The major divisions of error are systematic versus nonsystematic error, with a third related entity, systematic variance, representing the variance between groups of related samples in the sample set. The major divisions of bias are biological, analytical, and interpretive, based upon when their effects manifest in the bioanalytical experiment and data analysis.

A third kind of variance is systematic variance, which represents the variance between groups of related samples in the sample set. Depending on the specific experiment and the applied statistical analysis, specific systematic variances can be part of the detectable signal or part of the uncertainty in the measurements due to confounding factors. In other words, one scientist's uncertainty is another scientist's usable systematic variance. For example, in matched case-control measurements, the intergroup variance is the actual desired signal to detect. In other experiments, differences in group composition between case-control measurements like subject sex (male/female) can be an additional source of confounding systematic variance. In the special case when the sample set is unintentionally uniform (i.e. homogeneous), the group effect on the measurements will be systematic error and will not contribute to systematic variance. The classification of systematic error is further complicated by interpretive errors that often have the appearance of systematic error. The classic example is the error introduced by a “standard” measurement used in the correction of other measurements. Also, these divisions in variance and error are not mutually exclusive (Figure 1). Both biological and analytical variance can be divided into systematic variance and nonsystematic error components.

Typical biases in metabolomics

Most forms of error and confounding variance in measured variables originate from some type of bias. This term bias refers to any factor that distorts the design, execution, analysis, and interpretation of a measurement [5] and is an ever-present problem for metabolomics experiments [6, 7]. A bias typically causes: i) a systematic error that distorts the measured values but does not change the variance; ii) a systematic variance arising from a confounding factor that is either unknown or inadequately addressed; or iii) an interpretive error due to an inadequate or improper statistical method of analysis. One can also categorize these biases as biological, analytical, and interpretive according to whether their effects manifest in the biological experiment, the analytical measurements, or subsequent data analysis (Figure 1).

Of the biological biases, selection bias has historically been one of the most problematic for case-control experiments [5], which are very common in metabolomics observational studies. There are very many types of selection bias [5], but usually they represent an unbalanced selection of subjects that differ genetically or epigenetically. Also, temporal selection biases (when samples are taken) are especially hard to control for, given the natural cycles that exist in organisms. Biological conditions bias is another significant type of biological bias where some experimental condition endured by the subjects is not properly controlled for or considered.

Of the analytical biases, sample preparation bias usually presents the largest set of problems for the experimentalist. Any deviation in how and how long a sample is extracted, quenched, and stored can greatly impact the measured analytes [8, 9]. Standards bias represents the effects that different standards have on deriving some absolute or even relative quantification from a measurement. Sample complexity bias represents the effects on measurement due to physical interaction between a mixture of analytes present in a sample. Analytical conditions bias is due to some change in analytical conditions either during or between measurements.

Various interpretive biases and errors can also cause serious problems. However, many of the biases are simply born out of ignorance or laziness. Researchers tend to use the methods that they know and are easy to use, whether or not such methods are appropriate for the analysis. As scientists, it is a tendency that we should resist. Of the methodological biases, statistical assumptions about metabolomics data are most rampant. The two worst assumptions are that: i) nonsystematic error is completely Gaussian distributed and ii) measured variables are uncorrelated. The first assumption should never be made with analytical techniques like mass spectrometry, which by their very nature should include Poisson distributed error [10]. For NMR data, the nonsystematic error is often complicated and may include Lorentzian distributed observables, for which the variance is not analytically defined, but can be numerically estimated [11]. The second assumption of independence is unfounded given the correlation of metabolites in cellular metabolic networks. Another common interpretative bias is a lack of statistical power due to poor design and insufficient subject numbers. This situation is often confounded by a lack of correction for multiple testing. Also, assignment errors like the misassignment of a subject to a specific group (i.e. class assignment error or class noise) or the misidentification of an analyte can lead to serious misinterpretations. Another common interpretive bias is the use of preconceptions in interpretation, which can lead to a confirmation bias. For example, use of unverified metabolic models (i.e. a model of part or all of the metabolism of one or more specific organisms) is a common confirmation bias in metabolic data analysis. Also, the careless use of priors and limitation to expected metabolic models can lead to confirmation bias within a Bayesian statistical formalism [12].

Dealing with biases

Constant vigilance is required to mitigate the many biases (Figure 1) that may add both significant systematic error and confounding systematic variance to a metabolomics dataset [6, 7, 13]. While there is no simple checklist of biases to look for and procedures to follow, there are several straight-forward strategies for dealing with many of these biases. The first is to use reasonably consistent experimental designs that exclude: i) partial consistencies for specific groups of samples, which may lead to a systematic variance from biological or analytical biases, and ii) trivial consistencies that may limit the generalization of results, due to systematic errors from biological or analytical biases. Building on the first strategy, the second strategy is to use effective experimental designs like matched-pair case-control experiments to limit the effects of confounding factors, especially from large biological biases. Also, the samples should be balanced between case-control groups for possible confounding factors (sex, age, related biological condition), in order to prevent systematic variance. The optimal design is equally balanced sampling (i.e. blocking) between the most likely confounding factors within case-control groups. Blocking allows the use of more sophisticated statistical methods [13]: i.e. ANOVA instead of a t-test or Welch ANOVA [14] instead of a Welch's t test [15].

The third strategy is to directly test how well a set of measured values for a given measured variable fits an expected/assumed analytical nonsystematic error distribution. Various implemented algorithms exist for testing how well a set of measured values fits different common distributions. For example, the nortest R package [16] includes several normality (normal distribution) tests including the Shapiro-Wilk test [17] and the Anderson-Darling test [18].

The fourth strategy is to validate results with temporally-separated datasets (i.e. analytical cross-validation) as a method for detecting the presence of biases. However, failure to detect bias by this method does not guarantee a bias-free approach.

The fifth strategy is to use blinded metabolomics experiments to reduce bias [6]. The double-blind randomized control trial is considered the gold standard for reducing researcher-introduced performance bias that can affect both biological and analytical conditions [19]. However, even this very effective method has known masking biases due to the psychological effects of the trial itself, especially when human subjects are involved [20]. Also, many observational studies cannot be double-blinded. Despite this, the blinding of analytical and/or statistical researchers (i.e. analytical and/or interpretive single-blinding) can reduce performance biases that affect both the analytical conditions and statistical interpretation [21].

The sixth strategy is to use analytical controls to correct for or to prevent analytical biases. However, application of this strategy is often specific to the analytical technique and instrumentation. With each sample in its own tube or vial, periodic controls or time-stamped near-random controls (random except to prevent neighbor effects) can be used to track analytical conditions during measurement [9]. For samples handled on plates, Latin square or 2D near-random patterns can be used [9]. An easier alternative representing a combination of strategies five and six involves the use of blind controls to detect and correct for several analytical biases including performance bias [22]. For full thoroughness, often a series of controls composed of complex mixtures of representative or chemically similar metabolites must be used to determine systematic error arising from sample extraction methods [23] and even from interactions between the metabolites themselves with respect to the specific analytical techniques employed. The latter is particularly problematic for electrospray mass spectrometry.

The seventh and final strategy is to fully document: i) the biological and analytical experimental procedures that produced a given metabolomics dataset; ii) the statistical procedures used in the analysis of the dataset; iii) a detailed list of all known or potential biases and assumptions, along with results of any analysis and testing of these bias and assumptions; and iv) results with adequate measures of uncertainty and confidence or at least a good explanation for why uncertainty and confidence measures are not provided. The two most common explanations are that not enough replicates were collected for error analysis or methods to determine the propagation of uncertainty do not exist for a specific statistical method or procedure. Such documentation enables thorough evaluation and peer-review by others and it facilitates future meta-analyses, especially when the datasets and documentation are deposited into public repositories like MetaboLights [24]. Minimum reporting standards for metabolomics experiments already exist for biological (plant specific) experimental procedures [25], analytical experimental procedures [26], and statistical procedures for analysis [27]. However, it would be better to augment these standards, especially for reporting known and potential sources of bias by borrowing from well-documented clinical standards [6] like STARD [28] and CONSORT [29, 30].

The overall purpose of these seven strategies is to identify which possible biases (Figure 1) may affect the proper interpretation of a metabolic experiment and then to limit their effect. However, in the end, a judgement call must be made on whether these strategies were adequate for the proper interpretation of the obtained metabolomics dataset(s).

Methodology

Standard error analysis methods

One central question that error analysis must address is whether nonsystematic error, variances, or covariances from either an analytical or biological source will prevent the desired detection and interpretation of biological systematic variance in a given metabolomics dataset. From this perspective, any nonsystematic error, variance, and covariance that could interfere with biological interpretation represent uncertainty that should be determined/estimated and properly addressed. To be most effective, error analysis needs to be part of the experimental design before the experiment even begins. Decisions must be made on the number of replicates at each stage of the experimental protocol in order to have the necessary dataset for thorough error analysis [31]. In some instances, it will be practically impossible to obtain the necessary replicates for a thorough error analysis and other approaches for estimating sources of variance and error will be necessary. Also, issues of statistical power must be considered for the statistical methods employed [34]. However, even when issues of statistical power are satisfied, typically when statistical power ≥ 0.8 at α=0.05 with “reasonable” statistical assumptions, it is advisable to test these “reasonable” assumptions by increasing analytical replicates for a subset of the samples. If these assumptions do not hold true or there is a lack of statistical power due to large analytical variance, then additional steps may be taken to address these issues, including: i) increasing the number of analytical replicates to deal with analytical nonsystematic error; ii) correcting for factors that cause analytical systematic variance; iii) switching to appropriate statistical methods that do not rely on these failed assumption(s); and/or iv) incorporating estimates of analytical variance and covariance into more sophisticated statistical methods.

So standard error analysis can be broken down into two major steps: i) error estimation and probability distribution testing and ii) error (uncertainty) propagation analysis. The first step involves the distribution testing, calculation/estimation, modeling, and comparison of nonsystematic error, variance, and covariance arising from biological and analytical sources. Testing for a normal distribution (normality test) is by far the most important probability distribution test. For normality testing, 8 to 10 replicates are considered the minimum needed with the Shapiro-Wilks test [35]. However, 20 to 30 replicates are typically desired for significant power, even though the Shapiro-Wilks test is considered the most statistically powerful normality test [17, 36]. When a measured variable fails normality tests, then other non-parametric or more fault-tolerant statistical methods should be employed to compensate for the lack of normality. For example, the non-parametric Wilcoxon-Mann-Whitney test is preferred to a t-test when the data are significantly non-normal [37-39]; but neither test works well if the data are highly skewed [40]. Sometimes, it is advantageous to interpret continuous measured variables categorically as discreet ranges and use binomial or multinomial statistical tests.

While the use of 3 analytical replicates is the minimum needed for quality control that includes the ability to detect outliers [9], 13 replicates (12 + 1) are considered the minimum for calculating variances with ∼half-width confidence intervals at least at the 90% confidence level when approximately normally distributed; 30 replicates are required to calculate variances with ∼half-width confidence intervals at the 99% confidence level [41]. In addition, these analytical replicates will naturally lower the standard error of the measured variable; however, there is a practical limit due to the availability of analytical instrumentation or of usable sample. Also, some analytical techniques have rather complex nonsystematic error structures that may pose additional difficulties [10, 42]. In these instances, a bootstrap procedure can be used to calculate confidence intervals [43, 44]. If the appropriate replicates across an experimental protocol are obtainable, then advanced mixed effect modeling methods can be employed to tease apart different sources of variance, including from random sources [32, 33, 45]. However, inadequate number of replicates requires the use of other approaches for estimating variance and its sources: i.e., estimation of variance indirectly from similar experiments, assuming biological variance is any variance not directly attributable to an analytical source, and/or modeling variance from time series measurements [46].

Calculating or estimating covariance is typically problematic for metabolomics datasets due to the small number of replicates n versus the number of measured variables p (n<p-1)/2 correlations are being estimated [41]. Despite these problems, there are ways to deal with the n<