Modeling the dissemination and uptake of clinical trials results.

A select set of highly cited publications from the National Institutes of Health (NIH) HIV/AIDS Clinical Trials Networks was used to illustrate the integration of time interval and citation data, modeling the progression, dissemination, and uptake of primary research findings. Following a process marker approach, the pace of initial utilization of this research was measured as the time from trial conceptualization, development and implementation, through results dissemination and uptake. Compared to earlier studies of clinical research, findings suggest that select HIV/AIDS trial results are disseminated and utilized relatively rapidly. Time-based modeling of publication results as they meet specific citation milestones enabled the observation of points at which study results were present in the literature summarizing the evidence in the field. Evaluating the pace of clinical research, results dissemination, and knowledge uptake in synthesized literature can help establish realistic expectations for the time course of clinical trials research and their relative impact toward influencing clinical practice.

Introduction

Systematically determining the impact of research is a complex endeavor and no single indicator can be used to evaluate success. In general, evaluating the impact of scientific research involves describing the influence of research outputs in terms of the value and benefit to those for whom the research is targeted. Most models of translational research frame the progression of knowledge traversing multiple phases along the research to health impact continuum (Trochim et al. 2011; Rajan et al. 2012). One segment of this continuum is the advancement of knowledge from clinical trials research to clinical practice. Broadly speaking, new clinical research knowledge is generated through systematic study, and the results are published in the peer-reviewed literature. Published results are subsequently included in the synthesized literature base, including meta-analyses, systematic reviews, and guidelines, which summarize the state of knowledge in a particular specialty. Next, this synthesized knowledge is broadly disseminated and utilized in the context of practice-based research, where its effectiveness is assessed, and when incorporated as a standard of care, may ultimately lead to improved health outcomes. While a phase model approach is often used to describe parts of the translational research continuum to a variety of researchers, practitioners, and policymakers, there is little consensus on the exact points of reference within and across models. It has been argued that a phased orientation is insufficiently precise for measurement in evaluation, and instead quantifiable markers along the pathway from research to practice need to be operationalized (Kane and Trochim 2011; Trochim et al. 2011).

The path clinical research findings take on their way toward influencing clinical care and supporting the prevention of diseases has been described as complex and varied, with multiple points and segments to consider (Lewison 2002). Some have tried to assess temporally the movement along the translational pathway, resulting in a wide range of time estimates based upon the segment under investigation. Others have expanded upon this by disaggregating the research process, including in their assessment the relative contribution at each stage. Despite these approaches, there is a lack of consensus regarding definitions, key stages, and measures, thus resulting in discontinuity across stages and yielding an incomplete picture. Consequently, the state of knowledge on the description and quantification of time lags in the health research translation process is of limited use to those responsible for improving the timeliness of the translation of health research (Morris, Wooding and Grant 2011).

Historically, publication counts and citations have been the most widely used metrics for the evaluation of individuals, programs, and systems (Cozzens 1997; Wells and Whitworth 2007). The relative influence of scientific research on the knowledge base is traditionally measured by the number and recognition of primary outputs in the serial peer-reviewed research literature (Grant, Green and Mason 2003; Greenseid and Lawrenz 2011). In operationalizing clinical research utility, the inclusion of published research (i.e., primary output) in the synthesized literature base (i.e., secondary output) can be considered an intermediate outcome preceding a change in clinical practice. Thus, a temporal approach to the use of bibliometric information to distinguish some order in the uptake in the scientific literature is warranted (Grant et al. 2000). Indeed, in order to objectively assess the contributions of clinical research outputs to the synthesized literature of a particular field, it is essential to know what specific work is being cited, where it is being cited, and in what context.

In this article, we illustrate an improved model for describing and evaluating the progress and uptake of clinical research in the peer-reviewed literature. The appearance of these primary research outputs in significant secondary research outputs (reviews, meta-analyses, and guidelines) can help to distinguish when a primary research output takes the next step along the translational research continuum. Core to our approach is the integration of bibliometric citation data as process markers along the translational research timeline. Within one segment of the research to practice continuum, we operationalize and assess the progress of research outputs, examine publication citation patterns, and represent dissemination outcomes over time in order to observe the rate at which clinical trials are completed and their results are acknowledged in the peer-reviewed literature.

Context for study

The National Institutes of Health (NIH) HIV/AIDS Clinical Trials Networks, within the National Institutes of Allergy and Infectious Diseases (NIAID) and the Division of Acquired Immunodeficiency Syndrome (DAIDS), are one example of a large-scale scientific endeavor engaged in clinical research to support the development of therapies, vaccines, and other prevention strategies for HIV/AIDS. The six NIH HIV/AIDS Clinical Trials Networks, while deeply rooted in pathogenesis research, are nevertheless practically oriented and their missions are devoted to improving clinical practice and health outcomes. Determining the success of large-scale scientific research programs requires acknowledging that the primary goal is to increase knowledge within and across relative disciplines, and is based on a core assumption that researchers with important information actively seek to publish their findings in open, international journals (Cozzens 1997; Van Raan 2005). Our own prior studies evaluating the NIH HIV/AIDS Clinical Trials Networks’ scientific outputs indicated that the networks, as a system of clinical research, produced scientific results recognized as highly impactful (Rosas et al. 2011). Although the networks’ research was well-regarded, we also observed that even at the highest percentile, there was substantial variability among publications in terms of how much recognition they accrued in citations. Furthermore, previous research of HIV/AIDS clinical trials results showed differences in how quickly trials were completed and subsequently published, affecting the availability of evidence to the broader scientific community (Ioannidis 1998). Research also shows a strong adherence to the recommendations found in practice guidelines by clinicians for the treatment of HIV/AIDS patients (Suarez-Lozano et al. 2009; Petersen et al. 2011), emphasizing the importance of the use of secondary research outputs in the field. These prior studies suggest the need to operationalize patterns of recognition over time as a way to more precisely assess the movement from clinical trial results to changes in clinical practice. Thus, the scientific outputs within this clinical trials discipline provide fertile ground for exploring the feasibility and utility of incorporating bibliometric data over time to assess the uptake in secondary outputs.

The data compilation and analyses described here are primarily focused on articulating the length of time (i.e., duration) needed to reach specific markers along a translational segment of the NIH HIV/AIDS Clinical Trials Networks research process. The time (in months) was then assessed to determine when the sample of protocols and their respective outputs reached these milestones.

Method

Sample

In an effort to objectively develop a representational model of the timing and uptake of scientific outputs across a clinical research pathway, we selected 22 publications of primary studies conducted by the NIH HIV/AIDS Clinical Trials Networks. The 22 publications were the results of ‘flagship studies’ of the networks and were selected on the basis of their scientific priority as primary interventional clinical trials. The 22 publications of these primary studies were identified as a subset of an initial group of 419 articles produced between 1 January 2006 and 31 December 2008 (Rosas et al. 2011). As the primary research outputs of highly recognized major studies across the six NIH HIV/AIDS Clinical Trials Networks, these 22 publications were found to be within the top 10% of the most cited publications of the same age, type, and within the same field. For each of the 22 primary study publications, we evaluated the time from study submission for scientific and regulatory approval to publication, and the subsequent time to citation and dissemination in reviews, meta-analyses, and guidelines, using several markers to note progress in dissemination.

Procedure

In general, data required for this modeling exercise were collected and organized through several steps and data aggregation processes. First, working retrospectively we identified the specific clinical research protocols and their descriptive information (i.e., protocol name, number, network, and principal investigator) for each of the 22 highly cited publications. We then queried the Division of AIDS Enterprise System (DAIDS-ES) for information on the date the study protocol was submitted to NIH/NIAID for scientific and regulatory approval. The DAIDS-ES is a management information system modeled around a protocol lifecycle paradigm that features standardized protocol status, milestone, and event definitions (Kagan et al. 2011). This system is designed to harmonize protocol data elements reported by the network-specific data management centers. Next, we identified the date of publication for each of the 22 primary studies publications. Since the date marking the end of the study was not routinely available across protocols, the date of publication was used as the point marking the completion of the study and the start of dissemination.

Several publication data sources were utilized, specifically the Web of Science (WoS), Journal Citation Reports (JCR), and PubMed databases, each of which offer information associated with the peer-reviewed literature. We collected a set of citation metrics for the 22 highly cited publications from the WoS and journal-level metrics from the JCR. These contained the overall number of citations, including self-citations, as well as the publication types for those citing the 22 publications. An important source of information needed for modeling the pace and uptake of the NIH HIV/AIDS Clinical Trials Networks’ scientific outputs was the dates for citation in the broader literature, including the dates of 1st citation in secondary outputs up to a specific point in time. In this case, 28 February 2011 marked the endpoint of our data collection of citations. Each citation has a date (i.e., month and year) by virtue of the citing publication date. We used the PubMed database to identify advanced classification data and specific publication types for those that cited the 22 publications, and to distinguish which of these citing publications were meta-analyses, reviews, or guidelines. PubMed does this by utilizing the MeSH (Medical Subject Headings) vocabulary standardized by the National Library of Medicine (NLM), which is useful in organizing the citing literature, particularly specialty publications such as meta-analyses, reviews, and practice guidelines. Finally, we operationalized ‘date markers’ in the citation data to track the progress of dissemination by selecting endpoints that are easily recognized and intuitive. In this case, each citation marker as a milestone essentially represents a doubling in the number of citations so that the dissemination interval is roughly the time (in months) it takes to observe a minimum of a 100% increase in citations.

Using the dates associated with a specific number of citations or the point when a publication was first cited in a secondary output, and then analysing the intervals between markers, we were able to represent individual timelines for each protocol and the subsequent published results. We examined the patterns between specific intervals individually and collectively, as well as across the entire timeline. We also sought to identify and include variables (e.g., study duration and influence of the article based on the journal, etc.) that may be related to the time course. In an effort to further distinguish the speed at which this set of 22 highly cited publications reached a series of citation milestones, we used the journal-level metric Article Influence Score (AIS) to organize the publications in terms of their predicted influence. This index is a measure of the average influence of a journal’s articles over the first 5 years after publication. It is the ratio of a journal’s citation influence to the size of the journal’s article contribution over a period of 5 years, and it represents a measure of the per article contribution to the total influence of the journal (Bergstrom, West and Wiseman 2008). Because the AIS includes a longer citation window, a lower credit for citations in long reference lists, and a consideration of the entire citation network, it may provide a more accurate predictor of the expected pace of uptake over multiple points in time than other well-known metrics like a journal’s impact factor.

Results

Overall, the 22 primary studies in this sample were cited 1,429 times by other publications between 1 January 2006 and 28 February 2011. Citations were found across 327 different journals, and citing authors were from 1,677 institutions representing 67 different countries. The proportion of self-cites was 14.97% (n = 214), which is slightly less than typically found in the medical literature (Gamiet al. 2004; Kavacic and Misak 2004). Figure 1 represents the distribution of all citations across publication types, including general research articles and research syntheses (i.e., reviews, meta-analyses, and guidelines). Relatively speaking, these 22 primary studies publications received a substantial number of citations in the 5-year period of time, averaging nearly 65 citations per publication. The majority of citations (58%) were found in general research articles, 11% were found in other article types (e.g., commentaries, editorials, and letters), and 7% were found in proceedings papers. Although all publication types serve different functions in dissemination and uptake, the number of citations found within synthesized research outputs, specifically reviews, meta-analyses, and guidelines, was especially notable given their role in summarizing the evidence in the field. This group accounted for approximately 24% of all citations. Their presence was important, as it indicated that despite the relatively young age of the primary studies publications, they were already being noted in the secondary outputs of the synthesized research.

Figure 1.

Distribution of all 1,429 citations from 2006–10 for the 22 primary studies publications.

As shown in Table 1, the 22 primary studies with publications above the 10% highly cited threshold averaged about 6 years (71.9 months) from the time the study concept was approved until the trial was completed and the results were published. For those publications where a submission date and an acceptance date were available (n = 15), it took an average of 3.37 months for manuscripts to be reviewed and accepted. From the point of publication, half of the 22 primary studies publications (n = 11) were included in research reviews within 7 months (median = 6.4) and all were included within 2 years (max = 20.3 months). Moreover, half of the publications (n = 11) were cited in clinical guidelines within 2 years of publication (20.8 months). On average, these 11 articles were cited in a clinical guideline within 1 year (mean = 6.91), with several articles cited in a clinical practice guideline within 1 month of initial publication. All papers were cited at least once within the first year (max = 8.1 months) and half were cited within the first 6 months (median = 5.0 months). Cumulatively from the point of publication, approximately two-thirds (64%) had reached the 20th citation milestone within 3 years.

Table 1.

Milestone timing (in months) for the 22 primary studies publications

Segment	N	Mean	SD	Med	Min	Max
Study approval, conduct, publication	22	71.90	19.75	74.14	35.65	121.25
Publication to 1st citation	22	4.05	2.75	5.00	0.00	8.10
1st citation to 5th citation	22	6.58	4.82	4.80	1.20	20.10
5th citation to 10th citation	20	7.19	6.06	5.10	0.70	21.30
10th citation to 20th citation	17	9.71	5.46	8.50	1.50	19.30
20th citation to 50th citation	7	10.71	7.55	8.90	0.10	23.30
50th citation to 100th citation	4	10.50	6.04	10.75	3.00	17.50
100th citation to 200th citation	1	6.20	–	6.20	6.20	6.20
200th citation to 400th citation	1	18.10	–	18.10	18.10	18.10
Time from publication to 1st review citation	22	6.56	4.72	6.40	0.00	20.30
Total time until 1st review citation	22	78.47	21.62	78.32	39.02	131.35
Time from publication to 1st guideline citation	11	6.91	7.77	2.90	0.00	20.80
Total time until 1st guideline citation	11	74.10	16.72	76.97	54.14	99.18
Time from publication to 1st meta-analysis citation	5	17.44	17.35	11.00	1.50	45.30
Total time until 1st meta-analysis citation	5	85.90	15.39	89.23	60.32	99.44

We computed separate Pearson’s Product-Moment correlation coefficients to assess the relationship between the protocol submission to publication process, and the time it took for publications to reach the 1st, 5th, 10th, and 20th citation milestones. All correlations were very low (−0.07, 0.26, 0.00, and 0.13, respectively) and none were significant. Similarly, low non-significant correlations indicated that the length of time for protocol submission to publication of results was not related to the time it took for the publication to be cited in research reviews (r = 0.29) or clinical guidelines (r = −0.16).

In representing the variation in the pace of the 22 primary studies publications, the time it took each study and their respective publications to reach each milestone is shown in Fig. 2. This graphical representation illustrates the time (in months) of each study from the point of submission for scientific and regulatory approval to the point of publication (shown in gray), and then charts the course of dissemination in the peer-reviewed literature for the various citation milestones (shown in various colors). Each colored bar represents the length of time (in months) between the separate citation markers. A taller colored bar indicates a longer period between citation milestones, and conversely, a shorter colored bar indicates a briefer period. No colored bar in the progression of citation milestones indicates the publication met the citation milestone in less than 1 month. Also included in this graphical array are specific markers, each noting when the primary study publications were first cited in synthesized research outputs, if at all. A marker located above a colored bar indicates that the publication was cited in the respective synthesized research output but has not yet reached the next citation milestone.

Figure 2.

Publication dissemination landscape for the 22 primary studies publications.

From left to right, the publications are ordered from the oldest publication date (i.e., published early in 2006) to the relatively most recent publication date (i.e., published late in 2008). As expected, many older publications had already reached the 100th and 200th citation marker at the time of our analysis, with several of the new publications still hovering between the 5th and 20th citation marker. Clearly, the more time a publication has to accrue citations, the more markers that specific publication would have reached as it is more broadly recognized by other researchers. However, a few of the relatively newer publications (#15, #20, #22) reached the 100th citation marker within 36 months from the point of publication, signaling a rapid uptake in the literature. One publication in particular (#9 in Fig. 2) reached the 400th citation marker and was cited in a research review, clinical guideline, and meta-analysis in less than 7 years from the point when the primary study was first submitted for approval and less than 4 years since its publication.

Overall, the higher the AIS for each publication, the greater the anticipated influence based on the journal status and the other published articles’ contributions in the journal. We examined the AIS and timing by separating the 22 primary studies publications into two groups: those with AIS below or at the group median (median = 2.02; range = 0.53 – 17.9) for the set of papers and those with AIS above the median (i.e., creating two groups of influentially higher and lower publications). We then calculated the time it took each group to reach the 1st, 5th, 10th, and 20th citation milestones. We also modeled the cumulative time across the four citation milestones (i.e., 1st, 5th, 10th, and 20th) for each group of publications. The results of this comparison are represented in Fig. 3.

Figure 3.

Differences in pace of citation for high–low AIS groups of the 22 primary studies publications.

A significant difference in the mean number of months was found at the 1st (F[1,20] = 42.76, P < 001), 5th (F[1,20] = 6.41, P < 05), 10th (F[1,20] = 22.34, P < 001), and 20th (F[1,18] = 15.77, P = 001) citation milestones. Cumulatively, the time from publication to the 20th citation milestone was quite long, with the group of publications with an AIS value below or at the median taking about three times longer to reach this marker than those with an AIS value above the median. Thus, despite the fact that all of the publications were considered above the highly cited threshold by virtue of the number of citations they received, we found significant variation in terms of the pace in which the publications reached specific markers based on a measure of ‘influence’. Interestingly enough, there were no differences in the time it took to be included in synthesized research outputs for the primary publications based on whether they were below or at the median or above the median AIS.

Discussion

Consistent with previous work emphasizing the use of citation analyses as a means for studying the pattern of flow of published material within a field (Bergstrom et al. 2008), we demonstrate in this example that not only can citation data be used to assess the dispersion of published work, but also the rate of speed at which it moves by incorporating dates into the modeling exercise. Our approach, as illustrated in this article, was to model the uptake in the serial peer-reviewed literature over time for select studies and their respective published results in order to assess the pace of knowledge generation and dissemination within one segment of the translational research continuum (i.e., the clinical trial). More specifically, we examined the feasibility of integrating both time interval and citation data, operationalizing the progression of a select set of clinical trials results from the NIH HIV/AIDS Clinical Trials Networks. Such a time based, process marker approach may prove useful in evaluating the dissemination and uptake of primary research outputs and gauging the pace of the initial utilization by denoting the presence of these outputs in the next translational step.

Several challenges to the assessment of knowledge transfer from clinical trials research have been noted. These include the potentially slow accumulation of the benefits from research knowledge, the unpredictability of accrual patterns, the difficulty in tracing added value, and the contingent nature of such gains. Furthermore, the criteria and indicators used to evaluate the impact of research are often dependent on the mission of a specific research group or institution (Beacham, Kalucy and McIntyre 2005). We suggest in our illustration that by delineating when a publication reaches a particular milestone, the interval from the previous milestone can provide important information about the pace of uptake. Instead of the focus being placed exclusively on the volume of citations as a means of indicating uptake and utilization of research outputs, this example focuses on the rate of uptake as a more dynamic indication of dissemination and utilization (Cummings et al. 2011).

Using a very small sample of studies and their respective publications, we were able to effectively demonstrate the combination of different sources of information to reveal more about uptake trends than what is typically documented. However, until critical metrics are fully defined and operationalized, any attempt to compare within and across settings cannot be interpreted accurately. For evaluators of biomedical research systems, simple metrics such as citations are an obvious and useful starting point. But the limitations of these metrics often present more questions than answers about what is meant or even what should be done in response (Kane and Trochim 2011). To be able to quantify time lags in an effort to improve the timeliness of research translation, it is imperative to obtain a clear picture of the movement along different parts of the translational research pathway.

Overall, we found fairly rapid uptake in the 22 publications of flagship studies from the NIH HIV/AIDS Clinical Trials Networks, with publications cited very early in the synthesized literature compared to what has been found in other studies. All publications in this sample were included in a review within 24 months of publication, and half were included in a clinical guideline within 12 months of publication. In contrast, analyses examining uptake in secondary research outputs have found substantial time lags for the inclusion of significant health research publications across a broad spectrum of disease areas. For example, the median time for inclusion of health research publications in clinical practice guidelines and meta-analyses have been lengthy, at 8.0 years (Grant et al. 2000) and 11.5 years (Trochim et al. 2009), respectively. We did not find a relationship between the length of time to complete a study and the time to meet the first four citations markers, as well as the time to appear in secondary research outputs, suggesting that the length of study completion does not negatively affect immediate dissemination and uptake.

Although the time-based study of various components of the clinical research processes is emerging and becoming more common (Dilts et al. 2009; Kagan, Rosas and Trochim 2010), unique to this illustration is the temporal application of citation data to indicate the progression of study findings in the peer-reviewed literature. Time-based modeling of publication results as they meet specific citation milestones enables us to easily observe the movement from one point of the translational research pathway to the next. What is not clear based on the results of this modeling exercise, however, is how the pace of this set of 22 highly cited HIV/AIDS publications from flagship studies can be compared to other sets of publications (i.e., non-HIV/AIDS publications). Thus, there is a need to establish some normative sense of what is expected more broadly in terms of the rate of uptake so that reasonable judgments can be made about performance overall. It may be that the rapid uptake in the peer-reviewed literature and synthesized research is due to the fact that HIV/AIDS clinical research is quite specialized. Given the ubiquity of digital media, questions also arise as to how sources outside of the peer review literature contribute to rapid dissemination and uptake, especially for highly visible research. The high visibility of these clinical trials raises the possibility that the results may have been widely known through outlets other than the scientific literature. Thus, depending on the timing of the clinical guideline publication, it is entirely feasible for the results to be cited by a guideline within 1 month. The expedited uptake may not be a reflection of the quality or influences of the studies, but rather a characteristic of the field, and therefore field-specific patterns of uptake may exist.

In this example, we demonstrated the ability to assess the pace of uptake by establishing clear process markers, despite the inclusion of a small set of studies and their publications. This approach may have utility in evaluations of scientific research systems where larger numbers of protocols and their primary outputs are available. In addition to the specification of markers for assessing the timing and pace of the movement along the translational research continuum, the identification of predictors related to the pace may prove to be important. In our example, the AIS of the journal showed some predictive value in understanding the pace of influentially higher and lower groups of publications. However, it is clear that more in-depth analyses with larger data sets need to be conducted in order to identify a range of meaningful predictors. While the relationship of the AIS to the movement of the publications through specific citation markers was observed, the AIS was not related to the length of time for publications to be cited in research reviews, meta-analyses, or guidelines. Thus, while the AIS may be a robust indicator of the pace of uptake of scientific outputs in the broader scientific community, it offers little information about the uptake in synthesized research, which is often viewed as critical for facilitating a change in clinical health practice. In future analyses, more specificity in delineating multiple markers within the study development, conduct, and publication segment is also needed. Greater continuity in data across studies will provide a more refined look at smaller intervals, and will enable the examination of how delays incurred during the study affect timing during dissemination.

Given the unique data compilation used in this example, we recognize the inherent challenges of standardizing this approach for routine use within evaluations of scientific research systems. However, continued replication and application of the approach modeled here, whether small- or large-scale, will further enhance evaluations tracing scientific research outputs as they move along the translational research continuum. The development of comprehensive, reproducible ways to evaluate the pace of clinical research, results dissemination, and knowledge uptake in clinical guidelines can aid in establishing realistic expectations for the time course of clinical trials research and their relative impact toward influencing health care practice. Measuring the progress of clinical research outputs in ways that are adaptable to different research disciplines and processes will also help to improve the evaluation of the overall impact of scientific research.