Evaluation of instrument error and method agreement.

Safely operating life support equipment and evaluating new technology both require some basic understanding of measurement theory. Measurement errors fall into two main categories: systematic errors (predictable problems usually due to calibration) and random errors (unpredictable). These two types of errors can be quantified by experiments involving repeated measurements of standards or "true" values. Systematic error (called bias) is usually expressed as the mean difference between measured and true values. Random error, called imprecision, can be expressed as the standard deviation of measured values. Total error can be expressed as an error interval, being the sum of bias and some multiple of imprecision. An error interval is a prediction about the error of some proportion of future measurements (e.g., 95%) at some level of confidence (e.g., 99%) based on the variability of the sample data and the sample size. Specifically, a tolerance interval gives an estimate of the true value of some variable given repeated measurements with an assumed valid measurement system. An inaccuracy interval predicts the validity of a measurement system with an estimate of the difference between measured true values (given that a standard or true value is available for measurement). An agreement interval evaluates whether or not one measurement system (e.g., a known valid system) can be used in place of another (e.g., a new unknown system). Statistical analyses such as correlation and linear regression are commonly seen in the literature, but not usually appropriate for evaluation of new equipment. Instrument performance evaluation studies should start out with a decision about the level of allowable error. Next, experiments are designed to obtain repeated measurements of known quantities (inaccuracy studies) or of unknown quantities by two different measurement systems (i.e., agreement studies). The first step in data analysis is to generate scatter plots of the raw data for review of validity (e.g., outliers). The next step is to make sure the data adhere to the assumption of normality. The third step is to calculate basic descriptive statistics, such as the mean and standard deviation. Finally, the data should be presented in graphic form with the differences plotted against the reference values and including numerical values for the calculated error intervals. The key idea to remember is that device evaluation and method agreement studies are based on the desire to know how much trust we should place in single measurements that may be used to make life support decisions.