Automated data analytics workflow for stability experiments based on regression analysis.

INTRODUCTION: We define a designated data analytics workflow for the evaluation of stability experiments, which takes all data situations into account. This complements the evaluation described by the CLSI EP25 [1] guideline by including a targeted exception handling algorithm and thus allows one to automatically evaluate stability data based on linear regression analysis. DESCRIPTION: The evaluation of stability experiments based on regression analysis requires the calculation of the confidence interval of the regression line. The stability time is estimated at the intersection of the confidence interval with the acceptance criterion. This approach results in solving a quadratic equation, with factors that depend on the estimated intercept, slope, the measurement variability and the chosen timepoints. When defining an automated data analytics workflow for this problem, the different cases for the solutions of the quadratic equation must be considered. For some data situations there might be no solution at all, other data situations result in a negative and a positive solution and finally there might be even two positive solutions. All these cases have to be considered for the choice of the right solution to become the estimated stability time. The CLSI EP25 [1] guideline on stability evaluation of in vitro diagnostic reagents addresses this problem only superficially and might even lead to incorrect results for some specific data scenarios.
RESULTS: We evaluate all possible data scenarios and provide examples for each. Based on the gained theoretical insights, we define a designated data analytics workflow and visualize it with a flowchart. By following this flowchart one can implement an automated analysis workflow, targeting all data scenarios with the appropriate exception handling. DISCUSSION: We deduce that the description for obtaining stability times according to CLSI EP25 is not fully adequate, as it addresses only best-case scenarios. However, for automated data analytics workflows all possible data situations have to be considered. With the here presented workflow one can program automated data analytics pipelines, which ensure that the right stability time is obtained, in case it exists. In addition all exceptions, where no stability times are present, are addressed in the right way and it provides hints as to the failure reason.

Background

The calculation of stability times is a mandatory component of diagnostic assay development. Stability must be proven for reagent material, for the analyte in calibrator and control samples and for the analyte in native samples taking into account respective pre-analytical handling or post-analytical storage conditions of the samples. Herein we discuss the analytical approach for stability studies, where one side of a 95% confidence interval of a linear regression line intersects the predefined acceptance criterion as proposed, for example, by the ICH Q1A(R2) guideline on stability testing of drug substances and products [2] and the CLSI EP25 guideline on the stability of in vitro diagnostic reagents [1].

For illustration purposes, we take the example data introduced in CLSI EP25 guideline [1] Appendix A. The example employs data from a shelf-life study for a new ferritin IVD reagent with study duration of 13 months. Test samples include 5 calibrator samples, 2 QC samples and a native pooled sample, measured on 10 testing days, between baseline and day 402. In Fig. 1 we show the data of the native pooled sample, with the estimated regression line and the one-sided upper and lower 95% confidence limits. As the slope of the regression line is positive, the estimated stability time for this sample is given by the intersection of the upper one-sided 95% confidence limit with the acceptance criterion of .

Fig. 1

Regression analysis of the native sample pool sample of the CLSI EP25 data example. The black line shows the estimated regression line, the dashed lines show the one-sided upper and lower confidence interval. The acceptance criterion is at and is indicated as a black, horizontal line, since the estimated slope is positive. The estimated stability time is given at the intersection of the upper confidence interval with the acceptance criterion (grey arrow).

Although both mentioned guidelines provide formulas for the evaluation of stability times based on the mentioned approach, these formulae lack relevant details and do not address worst-case scenarios. The stability data of the native pooled sample has low variability around the regression line, compared with the acceptance criterion and there is a clearly increasing trend. However, less clear data situations might occur, especially when implementing this approach in statistical software tools. We observe that all scenarios need to be considered and clearly defined in order to avoid incorrect results for stability times.

An automated data analytics workflow for the evaluation of stability data must report the correct data-derived stability time if it exists, or, otherwise, report that it is not possible to estimate a stability time for these data. Such a workflow should also report the specific reason why no estimated stability time could be derived and indicate the next steps for the user. In terms of reproducibility and error avoidance it is unacceptable that the user has to visually check graphs, manually verify the correctness of calculated stability times or identify the root cause for the reason why no stability time could be estimated by the algorithm. Visual inspections are, of course, cumbersome and subjective.

Therefore, we present a data analytics flowchart for estimation of the stability time based on the intersection of the one-sided 95% confidence interval of a linear regression line and predefined acceptance criterion. In doing so, we follow the same logic which underlies the stability estimation calculations implemented in the statistical software SigmaPlot [3][4]. However, we did not find any clear description of the flowchart, which would allow us to reproduce the SigmaPlot approach.

In the following, we give a short overview of the statistical background of linear regression, the confidence interval approach and the quadratic equation solving. For better understanding will follow up on the example data from the CLSI EP25 guideline, given in Appendix A of this guideline together with some artifical data, which present more extreme scenarios. Although these scenarios might occur infrequently, it is important to handle them properly in an automated data analytics workflow.

Applied Methods

Let denote the result of a measurement of an analyte for sample at time point with replicate .

Denote by the mean of the replicates of sample i at time point t, that is

Let be the time of measurement, with the first time of measurement, called also baseline measurement, and with the last time of measurement, with , with . The linear regression analysis for each sample can either be based on the measured concentration values or on the percentage deviation from the different time points to the baseline measurement. The choice here depends on the concentration of the sample and the absolute value of the measurements. For samples with low analyte concentration an analysis based on measured concentrations might be better, whereas the analysis based on recoveries for higher concentrated samples might be more comparable. We denote the percentage deviation of the mean of replicates at time point t from the mean of the replicates at with

We present our results for percentage deviation values. However, they can be easily replaced by measured concentrations. For each sample i, a linear regression between the result and the time is assumed, that iswhere denotes the intercept, denotes the slope and denotes the random error. It is assumed that the random errors are identically and independently normally distributed with mean 0 and variance .

Linear regression modelling

For this bivariate linear regression model, estimators of the coefficients and are given bywithand

The intercept is estimated by

Predicted values can be calculated aswhere denotes any time point not necessarily used in the design.

The difference between a percent deviation at time t and its predicted value is called the residual and the estimator for the standard deviation of the residuals is given by

The squared standard deviation of the residuals is called the variance of the residuals and is denoted by .

For stability analysis, a one-sided % confidence interval for a predicted value has to be calculated. In most cases the analysis of stability studies is based on or , however the calculations of stability times can be performed for all values with . Without loss of generality we regard the case of . The calculation of the confidence interval depends on the direction of the slope:where is the critical value of Student’s central t-distribution usually with as significance level and with degrees of freedom andas total sum of squares for the variable X.

If the estimated slope is positive, then the confidence interval limit is given by

If the estimated slope is negative, then the confidence interval is given by

From (10), (11) we see that a confidence interval can only be estimated for the given stability design if holds. This implies that at least three observations are present, and that these observations do not fall on a single line. Such a “perfect fit” situation occurs in experiments where the resolution of the measurements is lower than the ongoing degradation. We discuss the calculation steps for this case in more detail below.

The acceptance criterion is defined as the maximum limit of acceptance of the one-sided confidence interval of the regression line. If the regression analysis is based on absolute concentration values, an absolute criterion will be applied. If, however, the percentage deviation from the baseline mean is used as data to calculate the regression line, a relative criterion will be used. The relative criterion is defined as percentage deviation from and denoted by . In case of a negative slope the acceptance criterion is negative and in case of a positive slope the acceptance criterion is positive.

When an absolute criterion is given, but the analysis shall base on relative deviations, the absolute criterion has to be converted to a relative criterion: Denote the absolute criterion by . The corresponding relative criterion is then given by .

For the estimation of the stability time , we first consider the case that holds. Based on the confidence interval, the stability time is calculated as intersection between the acceptance criterion and the one-sided 95% confidence interval.

Solving or for X leads to the following quadratic equation [1]:where the coefficients of the quadratic equation arewith

It is well known that solving the quadratic equation in (13) results in two roots which are given by [1]:

In order to obtain numerically stable solutions, we change the equations (18), (19) into equivalent formulas, dependent on the sign of . This protects against catastrophic cancellation when doing subtraction [5].

In its mathematical essence, the problem of estimating stability times reduces to solving a quadratic equation and obtaining at least one real-valued, positive solution. The solutions for the quadratic equation in (13) are presented in [1], [6]. However, none of the aforementioned publications provides guidance on how to verify if the mathematically valid solution for (13) also corresponds to the solution of the underlying substantial problem which is finding the correct stability time. This becomes evident if both roots (18), (19) have positive solutions or if the discriminant, i.e. the formula under the square root in (18), (19), is negative so that there is no solution for (13) at all.

In addition, we have to ensure that the estimated intercept is lower than the acceptance criterion in case of a positive slope and higher than the acceptance criterion in case of a negative slope. If this is not the case, it means that already at baseline the stability criterion is not met. From a mathematical point of view, the aforementioned case may lead to (13) having one (or two) positive solution(s) which, however, would describe the intersection of the acceptance criterion with the opposite confidence interval, i.e. the upper confidence interval for negative slopes and the lower confidence interval for positive slopes.

For the definition of an adequate data analysis workflow and a software implementation which provides the user with the correct stability time, the understanding of the two factors and is crucial.

Comparing the statistical t-test for the slope (see eg. [1]) with the definition of factor , it can be derived that if and only if the p-value of the statistical t-test for the slope is . This means that the slope is statistically different from 0. Note that this is a major criterion within [1], for the estimation of the stability time. However, this criterion does not guarantee the estimation of a correct stability time.

For we can derive the following important property: If holds, then or holds. The coefficient can be interpreted as the difference between the specification and the upper confidence interval limit at baseline. If holds, then the upper confidence interval limit is below the specification for a positive slope and above the specification for a negative slope.

Finally, it is important to mention, that the discriminant of Eq. (13) can only become negative if and , in case that the absolute value of the intercept is lower than the absolute value of the acceptance criterion.

Within the Results section we show how the interplay between the two factors and impact the solutions of the stability time calculation.

Note that both solutions (20), (21) can result in stability times, which are greater than the maximal tested time range. The CLSI EP25 guidelines forbids extrapolation further than the maximal tested range, whereas other guidelines [2] allow for extrapolation, with additional measures of care. Certainly, one has to be very careful when extrapolating beyond the tested range, as it is not known whether the degradation model stays the same. Extrapolation always assumes that the change pattern will continue to apply beyond the tested time. However, it might make sense in some situations to allow for some extrapolation, i.e. beyond the maximal time. Therefore, we do not include a extrapolation/no extrapolation rule within our examples or within the data analytics workflow. If there is a defined extrapolation rule, it should be added to the workflow in the last calculation step.

Perfect linear fit

For the sake of completeness, we also consider the case of a perfect linear fit implying that all data points fall on one perfect line such that . This situation typically arises when working with data whose resolution (in terms of the number of reported digits) is so low that all data points of the measurement series have the same value. It can be seen from (10), (11) that such a case does not allow for the calculation of a confidence interval. Hence, there is only an intersection of the acceptance criterion with the regression line. Furthermore, the following cases have to be distinguished:

If the estimated slope is zero so that holds, then the stability time is set to T, that is . This case is only possible for a given sample stability design if for all timepoints t and all observations per timepoint.

If the estimated slope is non-zero, one has to check whether the absolute value of the intercept is lower than the absolute value of the acceptance criterion or not. If this is not the case, a positive stability time does not exist, since the acceptance criterion is not met even at baseline. However, if the absolute value of the intercept is lower than the absolute value of the acceptance criterion, a positive stability time can be estimated by

Results

For better understanding and visualization of the aforementioned considerations on the calculation of stability times, we provide a calculation example including interpretation, based on the example data of the.

CLSI EP25 guideline [1] Appendix A. In addition, we present data on artificial samples to point out the limitations of the analysis approach provided in CLSI EP25 and to visualize the data scenarios which must be covered by a reliable workflow for the calculation of stability times.

In order to show the interplay between the slope of the regression line and the standard deviation of the residuals, we calculated the factors and along with the discriminant of the quadratic equation for different combinations of slopes and residual standard deviations. In doing so, we used the distribution of the timepoints according to the data example data presented in CLSI EP25 and set to and the intercept to 0.

Fig. 2 shows the results of the above calculations dividing them into five different color-coded areas: The lightest grey area describes combinations of the residual variance and slope where the discriminant is negative. In the bottom left grey and black area , whereas for the upper three light grey areas it holds that . In the black and top right grey areas it holds that , whereas for the left grey areas . The white and grey lines indicate where . With exception of the white line, the area below the black curve describes combinations of slope and residual variance , for which the estimated stability time exists.

Fig. 2

Plot of the residual variation on the absolute value of regression coefficients stratified by combinations of the coefficients , and . The area below the black curve with exception of the white line describes combinations of the regression coefficient and , where solutions for estimating the stability time exist. For combinations in the ND named region the discriminat of the quadratic equation is negative. For all other regions the discriminant is positive.

Data scenarios with solutions for the stability time

In Fig. 2, the black area ( and ) reflects situations where the recovery of the measured samples, as compared to baseline, shifts over time so that the related lower (for negative slope) or upper (for positive slope) one-sided confidence interval intersects with the acceptance criterion at exactly one positive time point. Note that in this case the correct stability time results from Eq. (20), while the solution given by Eq. (21) corresponds to the intersection of the acceptance criterion with the opposite and thus undesired confidence interval (i.e., lower (upper) confidence interval for positive (negative) slope). CLSI EP25 lacks the above case discrimination and might thus lead to incorrect stability times.

We see that the data of the native sample pool presented in the CLSI example falls within this area. For this sample the estimated regression line together with the acceptance criterion and confidence intervals is shown in Fig. 3A. The dashed vertical line displays the estimated stability time at 393 days, whereas the dotted line exemplifies the intersection of the lower confidence interval with the acceptance criterion, which is at day 555.

Fig. 3

Regression analysis of the native sample pool, the L2 calibrator sample from the CLSI EP25 shelf-life experiment and an artifical sample. The black line shows the estimated regression line, the dashed lines show the one-sided upper and lower confidence interval. The acceptance criterion is at and is indicated as black, horizontal line, since the estimated slope is positive for all samples. For all three situations positive stability times exist, given at the intersection of the upper confidence interval with the acceptance criterion.

For the bottom left grey area, we obtain a positive stability time from (20) and a negative one for (21). In this case, the relevant confidence interval intersects with the acceptance criterion, once in the positive range and once in the negative range. Here, and holds, which means that the slope is not statistically different from zero, but at baseline the confidence interval is within the acceptance criterion. For a positive slope the confidence interval intersects with the acceptance criterion only once in the positive range and the lower confidence interval never intersects with the acceptance criterion.

If the definition of the stability time does not require the statistical significance of the slope, (20) gives the calculated stability time. However, there are different opinions, whether the statistical significance of the slope should be taken into account or not. [7] argues for the statistical significance for the estimation of shelf life in drug or reagent stability studies. We value this argument, especially if the substance is subjected to conditions that cause degradation. However, for sample stability studies where the stability of the analyte should be shown, we do not necessarily see the need for statistical significance. Nevertheless, we will include in the definition of the data analysis workflow both possibilities.

Fig. 3B shows the estimated regression line, the one-sided upper and lower confidence intervals and the acceptance criterion for calibrator sample L2. The dashed line displays the estimated stability time at 583 days given by (20). In the case that extrapolation beyond the maximal measured time is not allowed, this value would be trimmed to 402 days. The second solution as given by (21) is negative and describes the intersection of the upper confidence interval with the acceptance criterion, which is in the negative range. The lower confidence interval never intersects with the upper acceptance criterion.

In Fig. 2, the top left light grey area below the black curve shows combinations of slope and residual variation where the right confidence interval intersects with the acceptance criterion twice in the positive range. Here, (21) gives the first intersection, (20) the second intersection. Similar to SigmaPlot [3], we define this second intersection as obtained stability time. Note that CLSI EP25 does not mention this data situation at all. According to the guideline in this situation the stability time would be set to the maximum time of the stability experiment, as the slope of the regression line is not statistically different from 0. This situation is visualized by the artificial sample No. 1 in Fig. 3C. The estimated slope is positive, but not statistically significant, and at baseline the upper confidence interval is broader than the acceptance criterion. As the upper confidence interval is a convex function, it becomes more narrow as it progresses towards the regression line and intersects the acceptance criterion once at 24 days and the second time at 320 days. We define the second intersection at 320 days as the obtained stability time.

Data scenarios with no solutions for the stability time

For the areas above the black curve in Fig. 2, we do not obtain adequate stability times. This also holds for the white separation line between the black and bottom left grey area. In these cases, further evaluations of (a) the observed measurement error within the experiment, (b) the defined acceptance criterion and (c) the measurements at baseline have to be made.

The light grey, parabolic area in Fig. 2, defines where the discriminant is negative and there exist no solution of the Eq. (13). This is the case if none of the confidence intervals intersect with the acceptance criterion. Fig. 4A shows this case at the example of the regression analysis for the second artifical sample. Here, the measurement error of the experiment needs to be checked carefully. If, however, the measurement error is within the expected range of the measurement procedure, one can show that the design and measurement procedure is not good enough for the defined acceptance criterion.

Fig. 4

Regression analysis of three artifical samples based on the experimental data presented in CLSI EP25. The black line shows the estimated regression line, the dashed lines show the one-sided upper and lower confidence interval. The acceptance criterion is at and is indicated as black, horizontal line, since the estimated slope is positive for all samples. For all three situations no stability time exists, since there is no intersection of the upper confidence interval with the acceptance criterion.

Within Fig. 2, the top right grey area defines combinations of slope and residual variation, resulting in and , which means that the estimated slope value is statistically significant, but at baseline the confidence interval of interest is broader than the acceptance criterion. In this case, none of the solutions for (13) gives a stability time. (20) results in negative values and the positive value obtained by (21) describes the intersection of the opposite confidence interval with the acceptance criterion. Fig. 4B shows this case at the example of the regression analysis for artifical sample No. 3. The slope of the sample is already quite steep and also shows a higher residual variation, such that already at baseline the upper confidence interval is out of the acceptance criterion. Only the lower confidence interval intersects with the acceptance criterion at a positive time point, on day 80. However, this is not the correct value for the stability time. Note that this case is not at all considered in CLSI EP25, although the slope is statistically significant. Samples which fall within the top right grey area have a relatively steep slope, compared to the acceptance criterion and a higher measurement error. In addition to checking the measurement error within the experiment, it is important to reconsider the validity of the baseline measurements.

The reconsideration of the validity of the baseline measurements is especially important for results where the estimated intercept is already above/below the acceptance criterion, as shown by the regression analysis of the artificial sample No. 4 in Fig. 4. Also in this case, there does not exist a stability time, as the regression analysis tells us that requirements are not even met at baseline.

For the sake of completeness, we briefly discuss the problematic case of the white separation line between the black and bottom left grey area. For combinations of slope and residual standard deviation falling exactly on this line, we have , which is equivalent to a p-value for the significance test of the slope of exactly . In this case, the lower and upper confidence interval asymptotically approach the acceptance criterion, but none intersect with it. Therefore, the stability time does not technically exist in this case, which we consider extremely unlikely.

Exception Handling

For the implementation of an automated data analytics workflow, all process steps have to be defined narrowly, based on the considerations of the theoretical calculations. Before starting the analysis the following exception handling needs to be implemented:

The number of time points is at least three, that is .

There are no negative times .

There are no negative results , since we consider concentrations that are always positive.

In addition, it might be useful to warn the user in the case of the following data inputs:

There are results below limit of quantitation (LoQ) or above the upper limit of the measuring interval.

There are missing data, in particular data are missing for any time point and the data are unbalanced with respect to the desired number of replicates across all time points.

The design is different for samples within the same experiment, in particular concerning the number of replicates and the number of time points and the location of the times .

Data Analytics Workflow Flowchart

Fig. 5, Fig. 6 shows a flowchart that describes the conditions where it is possible to estimate the stability time . This flowchart is the backbone of a software implementation for estimating the stability time . Following this chart, a straightforward implementation of a data analytics workflow for all possible data scenarios is achievable. In order to describe the area below the black curve, as shown in Fig. 2, the following notations are introduced:

Fig. 5

Flowchart for calculating the stability time .

Fig. 6

Three sampling schemes for the CLSI stability example, with 10 sampling time points, between 0 and 402 days.

The workflow starts with the evaluation with regard to whether the residual standard deviation is positive and different from zero. If this is not the case, all measured values fall on a line and the workflow enters the decision tree for the perfect linear fit. If the residual standard deviation is different from zero the next level checks, whether the estimated intercept of the regression line is larger than the acceptance criterion. If this is the case, there is no solution for the stability time. One should especially check the validity of the baseline measurements and the overall measurement variability. If the intercept is smaller than the acceptance criterion, the workflow continues to check whether . If yes, we check whether . If this is also fulfilled, the obtained combination of slope and residual standard deviation lies in the black area and the stability time is calculated based on (20). If , the combination falls within the top right grey area and there is no value for the stability time, potentially due to a high measurement variability.

In case of , we first have to distinguish whether the combination of slope and residual standard deviation is above or below the black curve. This is done by applying the inequalities for slope and residual standard deviation with the notations of and I. If the combination is above the black line, no solution for the stability time exists because the measurement variability is too high. In case the combination is below the black curve, there are two possible solutions for the stability times, depending on whether we ask for the statistical significance of the slope or not. In the first case, the entire time range becomes the stability time, in the second, the solution is again given by (20), in case that is not equal to 0.

Discussion

We carefully examined different data situations for estimation of the stability time according to the analysis approach of CLSI EP25 [1] guideline. This approach defines the stability time as the intersection of a respective confidence interval of the regression line with the predefined acceptance criterion. However, this approach results in solving a quadratic equation, which mathematically has either no solution or two solutions. Although guidelines and publications show these results, they do not clarify which of the two solutions of this equation should be used and for which data situation. However, a fully automated data analytics workflow can only be put in place, if this is done in a rigorous manner.

We show that the existence of an intersection of the respective confidence interval with the acceptance criterion mainly depends on the combinations of the coefficients and of the quadratic equation and the relation between the intercept and the acceptance criterion. For different data situations, one has to define the solution to obtain the correct stability time. In addition, we give for each scenario a clear example to exemplify the different intersection situations of the confidence interval with the acceptance criterion. Also extreme data scenarios have to be considered for the definition of a fully automated data analytics workflow. One important scenario is that all data points fall on a line, i.e. the residual variation is 0. This might happen if the resolution of the measurements is lower then the change of the analyte over time. In addition to the step-by-step data analytics workflow definition, we give guidance for further evaluations in case no solutions for the stability time exist.

The gained insights of the different data scenarios can be used to deduce comparison measures for different sampling schemes and their suitability for the defined acceptance criterion. The acceptance criterion for stability should be suitable for a result to be used in a medical decision and ideally based on total allowable error requirements. Total allowable error requirements should be based on models defined by the 2014 European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) consensus conference [8] including clinical outcome studies, a selected proportion of biological variation, or, when information derived from the first two models are lacking, state-of-the-art of the measurement performance. If the acceptance criterion is based on total error requirements, an appropriate sampling scheme must be defined for stability time estimation. Comparison of different schemes is already helpful in the planning phase of a stability study. They allow one to deduce the maximal allowed measurement variability for the given sampling scheme and acceptance criterion within the planning phase. This maximum allowed measurement variability can then be compared to the actual measurement variability of the measurement system to check whether the planned stability study can be performed in this manner. More details are provided in the supplemental material.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: All authors are employees at Roche Diagnostics GmbH, Germany.