AMGET, an R-Based Postprocessing Tool for ADAPT 5.

ADAPT 5 is a powerful modeling software for population pharmacokinetic and pharmacodynamic systems analysis, but provides limited built-in functionality for creating pre- and post-analysis diagnostic plots. ADAPT 5 Model Evaluation Graphical Toolkit (AMGET), an external package written in the open source R programming language, was developed specifically to support efficient postprocessing of ADAPT 5 runs, as well as NONMEM and S-ADAPT runs. Using interactive navigational menus, users of AMGET are able to rapidly create informative diagnostic plots enriched by the display of numerical and graphical elements with a high degree of customization using a simple settings spreadsheet. This article describes each feature of the AMGET package and illustrates how it allows users to utilize the powerful numerical routines of the ADAPT 5 package in a more efficient manner through the use of a simulated dataset and a simple pharmacokinetic model optimized using the maximum likelihood expectation maximization (MLEM) algorithm of ADAPT 5.CPT: Pharmacometrics & Systems Pharmacology (2013) 2, e61; doi:10.1038/psp.2013.36; published online 31 July 2013.

Characterization of the pharmacokinetics (PK) and pharmacodynamics (PD) of a drug product through population-based approaches and mathematical modeling of the relationships between exposure, safety, and efficacy is essential for the approval process at regulatory agencies, such as the United States Food and Drug Administration[1] or the European Medicines Agency.[2] Additionally, modeling and simulation techniques leveraging PK and PD data are often utilized in postapproval procedures for which safety and efficacy markers are being explored in special populations or situations.[3,4] These activities are supported by the analysis of complex data, typically collected in large phase II/III clinical trials enrolling sometimes several thousands of patients, and require specialized software and significant computing power.

The pioneering work in population PK/PD analysis was first explored in the NONMEM computer program.[5] With the growth of the community of pharmacometricians and the evolution of data modeling requirements, other estimation algorithms and population-based programs have been developed, including ADAPT,[6] Monolix,[7] and S-ADAPT,[8] among others. All these tools are now widely used in academia and industry to optimize the use of nonlinear mixed effects models.

The diagnostic tables and plots created from the output of these modeling programs are essential in the evaluation of the model quality and the communication of the modeling results. Whenever available, the tools built into these programs are often inflexible and inefficient, making the integration of created displays into presentations, reports, and manuscripts difficult or unpractical. Postprocessing programs are dedicated to the manipulation of modeling software outputs and perform convenient formatting and summarization tasks and/or creation of graphics. They typically provide greater flexibility and faster processing time for analysis and presentation of population-based modeling results. Several packages, including Xpose,[9] PKgraph[10] and NONMEMory[11] for NONMEM, or S-ADAPT-TRAN[12] and SADAPTPlot[13] for S-ADAPT have been published and released for public use free of charge and open source in most cases. As far as the authors know, currently no dedicated tool for leveraging the output of ADAPT version 5 for diagnostic purposes has been created.

We describe here the functionality of ADAPT 5 Model Evaluation Graphical Toolkit (AMGET), an R-based package, which uses base R graphic functions to generate customizable displays from ADAPT 5 output datasets. This flexible tool can be used with individual and population-based estimation methods found in ADAPT 5 and requires little knowledge of the R programming language, with the exception of standards for specifying graphical settings such as colors, symbols, and line types.[14] Each category of plot created with AMGET is illustrated using an example of population PK data analysis designed to present how the detection of model misspecifications can be facilitated by inspection of diagnostic plots. The PK data used in this example were generated by Monte-Carlo simulation of a linear, two-compartment model with first-order absorption and elimination in 50 virtual subjects using ADAPT 5. An under-parameterized one-compartment model with first-order absorption and elimination was then intentionally fitted to the simulated data with the ADAPT 5 maximum likelihood expectation maximization (MLEM) algorithm. We assume here that the reader is familiar with the data requirements, workflow, and operation of ADAPT 5, which are described in detail in the ADAPT 5 manual.[6]

Results

Dependent variable vs. time profiles

The first category of plots generated by AMGET is a plot of the dependent variable plotted vs. time. The purpose of this function is to provide preliminary insight on the PK and PD profile shapes. This includes observing the density of sampling, the influence of the dosing history on the PK and PD profiles, nonlinearity, potential interindividual variability (IIV) and the number of points below the limit of quantitation.

This feature is different from other AMGET functions as it does not use ADAPT output but creates the plots of interest using the input data file (ASCII-based *.dat format or comma-separated *.csv format). We have developed AMGET to be compatible with ADAPT 5 population datasets, with folders containing a collection of individual ADAPT 5 datasets, or population datasets formatted according to NONMEM or S-ADAPT standards (variable headers and *.csv extension required). The selection of the data file to be read and the format to be utilized is defined by the user through the navigational menu.

When NONMEM or S-ADAPT datasets are used, the program will search the headers for typical names such as: ID (Subject Identifier), DV (Dependent Variable), MDV (Missing Dependent Variable), and TIME (Elapsed Time). If a required variable cannot be found, a list will display all the column names available and the user will be asked to manually specify the column of interest. A typical example of this situation would be the use of the column name CONC instead of DV with S-ADAPT files.

Once the formatted data is read by AMGET, three types of graphics are generated: population scatter plots and line plots (also called “spaghetti” plots) for each dose level detected in the dataset (top panel in ), as well as individual plots (bottom panel in ). These graphics can be created in linear and semi-logarithmic scale. If defined, the limit of quantitation is displayed on every graphical output as a horizontal line coupled with its numerical value and units. The number of points below this quantitation limit is given and compared with the total number of observations. Additional information is provided on individual plots, including the display of the minimal value of the concentration (Cmin), the time of Cmin (Tmin), the maximal value of the concentration (Cmax), the time of Cmax (Tmax), as well as a visual display of the time and amount of doses, represented as vertical lines for bolus and horizontal arrows for infusions, the tips of the arrows marking the beginning and end of the infusion.

Applied to our simulated data, AMGET functions quickly to create highly informative concentration vs. time profiles (). The PK profiles all appear to follow a biphasic distribution with first-order absorption on the semi-logarithmic scale spaghetti plots. The plots also show limited variability in the data with Cmax and Tmax at about 0.7 mg/l and 4 h. Finally, 22 observations out of 500 are reported as below the “theorical” quantification limit of 0.05 mg/l.

Goodness-of-fit

Goodness-of-fit (GOF) plots are the primary diagnostic graphics used to evaluate model adequacy. Within AMGET, seven different GOF plots can be created: observations vs. individual and population model predictions on both linear and logarithmic scales, standardized residual vs. time, and standardized residual vs. individual and population model predictions.

The informational content of the GOF plots are enhanced by the display of several elements, including reference lines, loess curves (local regression), the Akaike information criterion, Bayesian information criterion, and objective function value or (−2 log-likelihood). The last three numerical values are directly read from the ADAPT 5 *.run file. In addition, the coefficient of determination (R2) is calculated by AMGET and displayed on the observation vs. prediction plots. Additional information is provided in the title space of each plot and include the model name, the ADAPT algorithm, the number of iterations necessary for model convergence, and the model description read from the *.run file (Supplementary Data online).

The AMGET GOF function was applied to the under-parameterized one-compartment model as shown in (graphics in logarithmic scale are not presented). Visual inspection of the precision plot (observation vs. individual and population model prediction) shows acceptable predictions (top panels in ). However, the residual plots (bottom panels in ) exhibited unbalanced distributions of weighted residual vs. time or individual model prediction, most notably a systemic under-prediction of the data for the 24 h time point, suggesting model misspecification.

Post hoc fits

Post hoc fits are overlay plots of predicted (i.e., based on the posterior mean parameter estimates for each individual) and observed dependent variable vs. time, allowing users to assess whether the model fits match the observations for each individual. In contrast to most modeling programs which only provide predictions at the discrete time points of observation, ADAPT 5 returns for each subject a richly sampled model fits containing a thousand observations (stored in *PLT.csv files). Taking advantage of this data, AMGET will overlay the rich individual and population prediction time course on top of the discrete observations for each individual. Plots will also include the individual estimate of model parameters, as well as the objective function value or −2 log-likelihood and R2.

Post hoc fit plots were generated for each of the 50 individuals of our example dataset ( only shows the plot for one typical subject). The inspection of the plots shows that the latest time points at 12 and 24 h were often poorly predicted.

Parameter distribution profiles

In ADAPT 5, the choice is given to model IIV in model parameters using either a normal or a log-normal distribution. However, the program provides no easy means to evaluate if the chosen parameter distribution was appropriate. Observation of the parameter distribution can sometimes reveal binomial distribution indicating the presence of subpopulations. AMGET creates plots of individual Bayesian estimate distribution for each model parameter: a fitted density of estimate distribution (top panel in ) and a histogram of estimate frequencies (bottom panel in ). Efforts have been made to enhance the informational content of the plots by displaying the estimated parameter mean, median and range statistics. In our example, the choice was made to use a log-normal distribution for all parameters. The parameter distribution function of AMGET was then applied to evaluate if this type of distribution was a proper assumption for our data. The visual inspection of the plots gives information of the variability around each parameter: the shape of the density curves appear to resemble a normal distribution, which may have been a better choice for this model.

Visual predictive check

Visual predictive check (VPC) plots provide an effective means to assess the predictive performance of a model. VPC plots do not directly rely on the output of the model estimation run but on a Monte-Carlo simulation run given the model structure, final parameter estimates, and analysis dataset. The VPC plots are created such as to allow the comparison of the distribution of the simulated data (typically a chosen set of percentiles) to that of the observations taken from the analysis dataset. The procedure to run those simulations, calculate the percentiles and create the graphics can be complex and time-consuming; as a result, VPC may be under-utilized by ADAPT 5 users.

AMGET is not intended to run the simulations, but will make use of the ADAPT 5 simulation output table to calculate percentiles on the simulated data, as well as the median for the observations, and create the graphics. The envelope created by the upper and lower percentiles of the simulation can be represented either by two lines or by a shaded area. VPC plots will be created in linear and semi-logarithmic scale for each of the different outputs available in the data.

A VPC plot was generated as an assessment of our under-parameterized model (). In this example, a 1000-subject Monte-Carlo simulation was performed with ADAPT 5 based on the model structure and final MLEM estimates. The VPC function of AMGET used the *POP.csv file containing the simulated data and the data file *.dat of the MLEM estimation step containing the observations. The lower percentile was set using the setting sheet at 5% and the upper percentile at 95%, creating a 90% prediction interval envelope. The VPC confirms that the one-compartment model offered a reasonable description, the central tendency and the variability of the simulated data being highly representative of the observations. However, model misspecification can clearly be observed by the systematic under-prediction at 24 h postdose time point.

Discussion

ADAPT 5 is a free and powerful modeling software for PK/PD and many other modeling applications. These attributes make it one of the most widely used tool for modeling PK and PD systems both in academic and industrial settings. Built-in graphic options are available, however, they remain quite limited and generate *.eps post-script files, which cannot easily be manipulated. As an alternative, users may attempt to use the output files generated after each run, and independently create the desired plots. When repeated over all the runs tested in any particular project, a manual approach proves to be redundant and time-consuming, whereas any automated processing requires the extended knowledge of a scripting language to enable full use of the data generated by ADAPT 5.

The development of postprocessing tools has demonstrated a significant benefit for other modeling programs such as NONMEM[9,10,11] or S-ADAPT[12,13] by allowing the modeler to spend more time evaluating the modeling results and less time creating the graphics. The optimization of population PK and/or PD models can be a time-consuming process, largely dependent on the number of observations and parameters, the processing power available, and the software algorithm used. On the other hand, the time required to create diagnostic plots for each model run is highly dependent on the skill of the user. Automated graphing tools, such as the ones provided in AMGET, save the user a considerable amount of time by streamlining very repetitive tasks, and also decrease the probability of errors by manipulating the data automatically and consistently rather than manually.

An extensive literature search failed to reveal any pre- or post-processing package for the creation of diagnostic graphics based upon the use of ADAPT 5 outputs. The AMGET package was therefore developed to offer a broad selection of highly informative diagnostic graphics, featuring dependent variable vs. time plots, goodness-of-fit plots, post hoc fits, posterior mean estimates distribution of model parameters and VPC plots. The primary objective of AMGET was to improve end-user productivity by readily generating diagnostic graphics. The secondary motivation was to enhance the information typically provided to the end-user by inserting additional elements in the diagnostic plots (e.g., loess lines).

AMGET has not been designed to run ADAPT 5 independently, but is a complementary tool that utilizes the information contained in the output files. Information obtained in the model runs is subsequently summarized and displayed, thus combining both graphical and pertinent numerical model evaluation criteria on the same display. Moreover, efforts have been made to improve the user experience through the development of a navigational menu, allowing any user unfamiliar with the R programming language to take full advantage of the program features through a simple and intuitive interface. This allows both the inexperienced trainee and senior researchers an equivalent experience, helping to lower the bar of entry for students interested in PK/PD modeling. Additionally, advanced users can write their own scripts calling the functions distributed in the AMGET package. The use of a settings file linked to the program gives the user enhanced freedom to customize graphics. This flexibility allows the graphics to be used for routine model evaluation, and for publications or reports. Information, including analysis date, time, and model description are embedded into the graphical output files to better organize the analysis project.

All graphics of a given category are saved in *.pdf files (Portable Document Format), because such files can contain multiple graphic pages, making browsing through plots more convenient. This format also has the advantage to support vector graphics, combining high quality pictures with small file size; *.pdf files can also be easily converted into other graphical formats. Finally, to facilitate the retrieval of modeling results, all *.pdf files created based on a given model file are stored into a single subfolder named according to the model file name.

AMGET 1.0 is available online for free download on the Comprehensive R Archive Network (CRAN) website at http://cran.r-project.org. Updates will be regularly released in a constant effort of improvement, to make new features, improvements to existing functions, and bug fixes available to the community. Ideas for future development of AMGET include the insertion of the area under the curve in the post hoc fits plots or the stratification of the VPC plots by a given variable. Regarding the latter feature, as ADAPT 5 simulation table (*POP.csv) do not contain any other variables than the elapsed time and model predictions, it is consequently more difficult for AMGET to create stratified VPC. However, this may be improved with a future release. Furthermore, a side-project aims at integrating AMGET within a graphical user interface, improving the user experience by bridging the selection of settings options with side-by-side graphic output, thus allowing for more customization options.

Methods

Software. AMGET is coded using the R programming language,[14] which has been employed with several other PK/PD postprocessing tools including Xpose,[9] S-ADAPT-TRAN[12] or SADAPTPlot.[13] R is a cross-platform, free, and open source programming environment offering powerful data management, statistical computing and graphical production capabilities, which makes it particularly suited to process nonlinear mixed effect program outputs. AMGET has been developed under R version 2.15.1; while it has been tested on some earlier versions without any apparent issues, the authors recommend to keep R updated to the latest available version. AMGET also runs under the current version of R (R version 3).

ADAPT 5, which is distributed at http://bmsr.usc.edu/software/adapt and produces the output processed by AMGET, is available free of charge for academic or commercial use, in contrast to most PK/PD modeling software packages. Accordingly, AMGET is distributed under the General Public License ≥ v2.0 at no cost on the CRAN website (http://www.cran.r-project.org).

Description of dataset and model for simulation and re-estimation. A population dataset of 50 subjects was generated using Monte-Carlo simulation in ADAPT 5. The structural model applied for the simulation (dose: 20 mg) was a linear two-compartment with first-order absorption and elimination. Typical values and variability of model parameters are provided in . Log-normal distributions were used to describe IIV in model parameters, an additive plus proportional error model, referred to as combined error model, was used to describe residual variability. The additive part of the combined residual error model accounts for the quantification limit of the assay, whereas the proportional part represents a coefficient-of-variation-like term around the concentration values. Concentrations were simulated for each subject at 5, 15, 30 min, 1, 2, 3, 4, 6, 12, and 24 h following a single dose of 20 mg administered into a depot compartment.

The model optimization step was performed using 50 MLEM iterations, IIV was described by log-normal models, and a combined model was used for the residual error. All parameters were estimated except for the additive component of the error model, which was fixed to a “theoretical” limit of quantification. As said previously, the data shown in this example was simulated based on a two-compartment model. Therefore, the ‘true model structure' was known beforehand and this exercise only aimed at illustrating the different graphic functions of the AMGET package. Applying a one-compartment model to this data resulted in model misspecification, which can be detected in diagnostic plots such as those created by AMGET.

Initial configuration of AMGET. When AMGET is run for the first time after it has been installed, a prompt window will ask the user where to copy configuration files. Once the directory is defined by the user, the users manual for AMGET and a setting file will be placed inside this folder. The path to this directory will be saved into a file within the AMGET library folder for future utilization of the package. The users manual will provide details and examples about how to use AMGET. The settings file will contain all settings used by AMGET to customize the diagnostic plots, as described in the next section. Default values are provided during the initial configuration, but users will be able to modify those values according to their own requirements. The configuration steps are part of the software installation and, once completed, do not need to be repeated for subsequent use of AMGET.

Settings file. Initial development of AMGET elected to gather all user-specified settings for AMGET in a *.csv table. This file is stored into a specific directory defined during the initial configuration step. Although it cannot be moved, the settings file can be edited or copied. This approach allows the user to save and re-use specific collections of settings for different runs and projects, and we believe that this approach is more efficient than requesting the definition of settings for every run in a step-by-step manner.

The settings file is organized in six sections, each marked with a specific tag composed of the ‘#' sign followed by a three-letter abbreviation (). These tags are recognized by AMGET as the start of the settings associated with a particular function.

The first section, identified with the #DIR tag, is used for general settings. Within this first section, the user can at any time change the directory where the graphics will be saved (i.e., output directory). This functionality can be used to save the graphics into multiple output directories each associated with a different project. Additionally, either the user can elect to use the R console or a prompt window to select the working directory (WD), the directory where the ADAPT 5 model to be analyzed was run. The other five sections are each associated with one of the current features of AMGET and are identified by the following tags: #IDV for the dependent variable vs. time profiles, #GOF for the GOF plots, #PHF for the post hoc fit plots, #PRM for the parameter estimate distribution plots, and #VPC for the VPC plots. Within each category section, it is possible to define the names and units of the X and Y axes, the colors, shape and width of the different lines or points, the organization of the plot window (e.g., multiple plots per page), the size of the text, the size and position of the legend, the names of the different outputs (e.g., parent drug or metabolite), and finally to disable the titles. These options must be defined using the standards of base R graphics, which are described in great detail in external books[14] or the help pages provided within R. Each category also has its specific options such as the definition of the units and value of the quantification limit for the dependent variable vs. time plots, the loess smoothing factor (span) for the GOF plots, the number of histogram breaks for the parameter estimate distribution plots, or the values of the lower and upper percentiles to be used for the VPC simulation envelope.

Working directory definition and plot creation in interactive mode. Every time the AMGET is started, it will ask the user to define the WD. This folder is expected to contain the model, the result file and output tables. The WD can be defined by pasting the path in the R console or by using a prompt window. The choice of method is defined in the settings file and allows AMGET to run in batch mode or through the interactive menu in the R console. As mentioned above, the content of the settings file will also define where the generated plots will be stored (i.e., the output directory).

If the user elects to use AMGET through the interactive menu, and if the specified WD is found, the main menu will be displayed and will allow the user to choose among the five main features of AMGET and two navigation options: (i) time vs. dependent variable profiles, (ii) goodness-of-fit plots, (iii) post hoc fits, (iv) model parameter distributions, (v) visual predictive check, (vi) Change WD, and (vii) Exit AMGET. The WD can then be changed at any time by selecting the option #6 in the main menu. For user convenience, the WD currently in use is displayed on top of the main menu.

If any of the options #1 to #5 are selected and if multiple ADAPT 5 runs are located in the WD, AMGET will ask the user to choose which run will be process for the creation of the selected category of plots. Upon the selection of the run, AMGET will generate the selected plots (.pdf) and tables (.csv) and store them in a subfolder of the output directory. To maintain consistency with the selected run and facilitate the retrieval of information, this subfolder will be named according to the name of the selected model run file.

Author contributions

B.G., L.W., and A.F. designed the research. B.G. performed the research. B.G. analyzed the data. B.G., L.W., A.F., and S.A.-O. wrote the manuscript.

Conflict of interest

The authors declare no conflict of interest.

Study Highlights

Table 1

Typical values and variability of model parameters for the simulated data