What "Impact" Do NLME Publications Have Outside Our Community?

The number of publications applying nonlinear mixed-effect (NLME) modeling has increased yearly since its first appearance in 1979. Here, we evaluated articles that have used NLME modeling, were published in journals that attract a broader audience, and we discussed the standard of presentation of these to stimulate target audience-specific improvements for increased impact in the future.

Motivation

Pharmacometrics aims to describe the quantitative aspects of disease and pharmacology dynamics, to understand drug‐patient‐disease interaction by connecting various fields, such as physiology, pharmacology, clinical pharmacy, mathematical modeling, statistics, systems biology, pharmacokinetics/‐dynamics in a coherent framework to generate knowledge, and improve drug development and patient outcomes (http://www.pharmetrx.de). The mathematical nature of the models used is not easily translatable to scientists outside the pharmacometric community, we, therefore, need to endeavor to improve communication within multidisciplinary teams and convey results to clinicians, editors, other modelers, reviewers, regulators, pharmacists, healthcare decision makers, and statisticians in a translational manner for having impact on scientific advancement and drug‐based decisions.

The aims of this study were to identify (i) publications that have applied nonlinear mixed‐effect (NLME) modeling since its first appearance in 1979, (ii) identify and evaluate the ones published in high impact journals, and (iii) to evaluate to which degree these were published in line with reporting standards for NLME analyses, , , which are in line with the recently updated US Food and Drug Administration (FDA) guideline (https://www.fda.gov/media/128793/download).

Literature search and framework

NLME publications were identified from three databases (PubMed, Web of Science, and Embase) in August 2018 and in January 2019, and restricted to articles published in English between 1979 and December 2018. Search terms used were: “nonlinear mixed effect modeling” OR “nonlinear mixed effect modelling” OR nonmem OR monolix OR pharmacometric*. Journal impact factors (IFs) were identified via Web of Science or journal webpages.

For the purpose of this review, journals with an IF ≥ 6.7 were defined as “high impact.” The cutoff was based on the IF of the journal Clinical Pharmacology and Therapeutics (CPT; 6.655; December 2018), which had the highest IF of the 20 journals that published the most NLME articles since 1979 based on the complete search. The consistency of IFs over the last 9 years was investigated and confirmed CPT’s IF ≥ 6.7 (Figure ).

Only original articles in high IF journals were read in full by two authors and examined against the criteria framework. Reviews, commentaries, and articles that did not use NLME methodology in their original work were excluded. For the analysis of the articles, the main text and Supplementary Material, if referred to in the main text, were reviewed. A framework (online supplement) was developed to establish a set of standard presentation criteria (n = 44; Table ) based on three guidelines. , ,

Outcomes

Of over 12,000 identified articles, 4,837 articles remained after duplications were removed. An increasing number of NLME articles per year (Figure a) was published in 633 unique journals. The majority of articles (50.6%) were published in 11 individual journals (Figure b). The IF of indexed journals ranged from 0.1–26.3 (Journal of Clinical Oncology). The median IF of 3.08 (interquartile range (IQR) 2.2–4.3)] across all 4,837 articles seemed to be consistent across the last ~ 40 years (Figure c). The journal CPT: Pharmacometrics & Systems Pharmacology, most commonly publishing NLME articles as of December 2018, had no IF at the time of the search. Most articles (85.3%) were published in journals (489; 77.3%), which had an IF < 6.7 (excluding journals without IF). Of the articles identified, 11.5% (557) were published in journals that were not indexed in Web of Science (Figure ), representing 15.0% (95) of the identified journals.

Figure 1

Overview of data from all nonlinear mixed‐effects articles (NLME). Display of the results for all articles (n = 4,837) identified: (a) the number of publications per year and (b) treemap of all journals (n = 633) with NLME articles sorted in descending order with the size of each rectangle being proportional to the number of articles published in that particular journal relative to the total number of nonlinear mixed‐effects articles (journal abbreviations displayed are explained in Table ). The left side of the treemap displays the 11 journals, where the majority of articles were published. (c) NLME article’s Impact Factor (IF; based on the IF of the journal that they were published in) across years. The purple line represents the median associated IF across all articles published since 1979. (d) Percentage of all NLME articles per year published in journals with an IF ≥ 6.7.

After applying the exclusion criteria, 100 articles (2.1% of 4,837) published in journals with IF ≥ 6.7 remained (IF range 6.8–26.3; median IF = 10.2; IQR 7.2–11.6). Seventy‐four percent of these 100 articles were published in journals whose IF ranged from 6.8–11.4. Six articles were published in journals with an IF of 26.3 (Figure a). Articles in high IF journals most commonly had a first author from academia (66.0%), followed by hospital (27.0%), and industry (7.0%; Figure b) from 18 different countries (Figure c). The most common therapeutic area was oncology (55.0%), followed by infectious diseases (10.0%), and pain (6.0%; Figure ). Of the 100 articles, 21.0% were performed in animals, 77.0% in humans, one in vitro, and one in silico. Half the studies (49.5%) presented a population pharmacokinetics and pharmacodynamics analysis, whereas 41.2% presented a population pharmacokinetic analysis and 9.3% a population pharmacodynamic analysis. Median number of authors per article was 8.5 (range 2–40; IQR 6–12]. Median total number of citations per article was 35 (range 12–64; IQR 26–42). On average, 13.4% of citations were referring to other NLME articles, but four articles were not citing any NLME article.

Figure 2

Overview of data from articles published in journals with an impact factor (IF) ≥ 6.7 (n = 100). Display of the results for articles (n = 100) reviewed in detail after applying exclusion criteria: (a) the number of articles distribution by IF, (b) the affiliation of first author, and (c) the number of publications per country identified from the first author.

Across the reviewed articles, between 34.1% and 90.5% (median of 66.3%) of the standard publication criteria were applied within each article. Thirteen articles used < 50.0% of criteria. One article applied 90.5% of the presentation standards and 22 articles fulfilled > 75.0% of criteria. Criteria for the introduction section were met largely. Study objectives were stated explicitly in 92.0% of articles. Standards of presenting methods and results varied greatly. Most articles provided general methods, such as study details, software used (97.0%), and type of study population (98.8%). But presentation was less consistent regarding the reporting of assumptions made (64%) or calculation of derived covariates (18.6%). The description of the model‐building process was often insufficient (e.g., 82.0% of articles explained the model building strategy, but only 64.6% of articles informed on covariate selection). Most articles reported results for the number of subjects (90.8%) and patient demographics (82.9%). However, other results, such as the number of observations, was presented only in 42.4% of articles, excluded data (33.3%) and outliers (1.0%) were rarely mentioned explicitly. The results of the model‐building process were presented in the majority of articles: 74 articles gave a full description of the final model, 50 articles presented equations or figures, 26 articles presented both, and 24 articles showed neither. When displayed, parameter estimates were presented in the text (12.0%) or in tables (81.0%). Model evaluation was reported in 75.0% of articles. Goodness of fit plots or other evaluation plots were featured in 71.0% of all articles. One‐third of articles stated the use of a specific evaluation method, but then did not show this in the results section. In the discussion, the achievements of the study results (99.0%) were presented in almost all articles, but only 51.0% of the articles addressed the limitations and 62.0% explained how their results could be used. Most articles (91%) provided a statement that answered the questions posed by their objectives.

Inclusion of Supplementary Information was found for 40 of 100 articles and increased in recent years (none before 2007). The model code was published in the appendix of 5 articles only and further model methods and results were displayed in 35.0% of the 40 articles. All criteria and results are shown in Tables and S4.

There was no trend noted among the IF, the year of publication, and a more diligent application of publication standard or improved understanding. This may reveal that these criteria, whereas crucial for understanding, reproducibility and the modeling audience are not relevant to identify “impact.” Future steps should identify relevant criteria for the “high impact” target audience to better understand and meet their expectations and interest.

Perspective and future directions

This perspective provides an overview on the current status of NLME publications. A fourfold increased output has been noted in the last 10 years. However, the proportion of articles published in high IF journals remains low and was consistently under 4% for the last 12 years (Figure 1 d). Various reasons might lead to the fact that output is quantitatively increasing, but not quality/impact. We are aware that using IF as a standard to measure the quality of a journal is not without controversy, as it is often misleadingly associated with the quality of an article. Articles in higher IF journals had a median of 4.4 citations per year. A smaller percent (19%) of articles had a higher median number of citations/year compared with journal’s IF compared with other fields of research.

A trend was recognized toward teams from academia and hospital publishing in higher IF journals and 79.0% of articles being published by author teams from only five countries. We suggest that potentially growth outside these countries could increase diversity and lead to wider application. Directed education initiative have increased output over time (see Australia and New Zealand’s position 6 and 7, respectively, in Table ). Initiatives in other parts of the world could change this further, potentially. , ,

The standards for presenting research findings from NLME studies varies widely and may lead to a lack of reusability, translatability, and acceptance within the wider scientific community. Study methods and results were often presented inaccessible to nonpharmacometricians (tested by a pharmacist and student volunteer). This might lead to valuable research findings potentially being disregarded by a wider audience.

Standardizing the presentation of NLME articles to ease understanding and to enhance the accessibility to a broader audience has been previously stipulated , to increase communication of research results and consequently strengthen the influence of pharmacometrics. However, we noted that fulfilling the majority of criteria does not mean that it can be well understood. In fact, we thought that some of the articles that met over 75% of criteria were written convoluted or were missing relevant information to be fully understood, which shows that we tend to use terminology that are misunderstood by our partners in the multidisciplinary teams. We found that often the description of modeling methods was prioritized over presenting translational aspects of the findings. Potentially, some work should be separated into technically interesting aspects published in therapeutic area specific journals and a translational paper for the broader audience. The community needs to find ways to appeal to different gatekeepers, such as editors, reviewers, and journal statisticians/clinicians. We hope to have stimulated some inspirations to further broaden the translation of NLME articles and increase quality in the presentation standards, and generate thoughts on achieving higher proportion of articles in higher IF journals. It was outside of the scope of this paper to capture explicitly which criteria were most important to capture the attention of a high IF journal and readers outside the pharmacometrics area and future work should explore how to identify stakeholder’s reasons for not finding our work attractive.

Funding

Stefanie Hennig was partly supported by the Alexander von Humboldt Foundation, Germany, during this project.

Conflict of Interest

The authors declared no competing interests for this work.

Author Contributions

S.H. study concept. S.H. and J.F. data collection and literature review and analysis. S.H., J.F., and C.K. interpretation of results, manuscript writing, and review.

Figure S1. Overview of Impact Factor changes across 9 years.

Click here for additional data file.

Figure S2. Number of articles published in each journal sorted by Impact Factor (IF) category (high: IF ≥ 6.7, low: IF < 6.7, no IF).

Click here for additional data file.

Figure S3. Shows the distribution of therapeutic areas stated in the 100 original research article published in a journal with an Impact Factor ≥6.7.

Click here for additional data file.

Table S1. Criteria framework established to evaluate each original research article published in a journal with an Impact Factor ≥6.7.

Click here for additional data file.

Table S2. List of additional information collected from each original research article published in a journal with an Impact Factor ≥6.7.

Click here for additional data file.

Table S3. List of the results of the analysis for articles published in a journal with an Impact Factor ≥6.7.

Click here for additional data file.

Table S4. List of additional information collected from each original research article published in a journal with an Impact Factor ≥6.7.

Click here for additional data file.

Table S5. List of journal abbreviations that can be visually seen (n = 49) in Figures 1B and S2.

Click here for additional data file.

Supplementary Material S1.

Click here for additional data file.