Recent trends in biomedical informatics: a study based on JAMIA articles.

In a growing interdisciplinary field like biomedical informatics, information dissemination and citation trends are changing rapidly due to many factors. To understand these factors better, we analyzed the evolution of the number of articles per major biomedical informatics topic, download/online view frequencies, and citation patterns (using Web of Science) for articles published from 2009 to 2012 in JAMIA. The number of articles published in JAMIA increased significantly from 2009 to 2012, and there were some topic differences in the last 4 years. Medical Record Systems, Algorithms, and Methods are topic categories that are growing fast in several publications. We observed a significant correlation between download frequencies and the number of citations per month since publication for a given article. Earlier free availability of articles to non-subscribers was associated with a higher number of downloads and showed a trend towards a higher number of citations. This trend will need to be verified as more data accumulate in coming years.

Introduction

As the field of biomedical informatics continues to evolve and grow into subspecialties, it is important to track the literature generated in the field and to understand which topics attract the most attention by readers and by authors who use and cite particular articles in their own work. Universities are increasingly making use of bibliometrics to evaluate academic progress, and some already require that candidates include the total number of citations to their articles, the impact factor for each publication, the author's h-index (or equivalent), in addition to an account of the author's contribution to each article.1 2 Some institutions provide guidance as to which journals should be considered first tiered, second tiered, etc.

Following the pattern of academic institutions that attempt to measure productivity with bibliometrics, some funding agencies have also established guidelines for progress reports in high visibility programs. The measures include quantity and quality of publications, with a proxy for the latter represented in the form of journal impact factors. To respond to this increasing demand for quantification of academic production, web services to track citations and h-indices, such as the Institute for Scientific Information's (ISI) Web of Knowledge,3 Scopus,4 and Google Scholar5 have been created. Some journals also provide the number of article views and/or downloads per month (DPM). Even though, as pointed out by several authors, these measures are imperfect,6–9 they may provide useful clues to define trends in biomedical informatics and to start taking into account how citation in subspecialized fields varies. Although the number of downloads and citations is not the only way to measure information dissemination and scientific impact, they are relatively easy to track.

It is well known that the citation rates of journals from different disciplines vary significantly due to many factors such as the average number of authors, average number of citations in an article, size of the community, and so on.10–12 However, little is known about how citation rates vary within biomedical informatics topics. JAMIA is a generalist journal in biomedical informatics. To understand the publications that readers consider interesting and cite-worthy, we expanded our initial analysis of citation rates by topic and free access status done in late 2011 (which covered only articles published from 2009 to 2010),13 to articles published in 2011 and 2012. The distribution of citations for JAMIA peaks between 3 and 4 years, so our analyses for 2009–11 are reasonably accurate, whereas those for 2012 are still very preliminary. We present here an analysis that shows the yearly rates of publication and average citations for JAMIA articles according to selected MeSH topics in the past 4 years (2009–12) to understand whether different subfields of biomedical informatics have different citation distributions and how they are changing over time. In this analysis, we take into account the fact that the number of articles with free immediate access when published online or in print has increased rapidly in this period due to the option for authors to select an open access model. We report the associations among downloads, number of citations, and free access status.

Materials and methods

In July 2013 we collected the number of yearly citations to JAMIA articles published between 2009 and 2012 from the ISI Web of Science database. We removed non-peer reviewed materials such as highlights, editorials, letters, and American Medical Informatics Association (AMIA) messages, which left us with a total of 564 articles to analyze. These articles described research and applications, brief communications, perspectives, reviews, and case reports. Note that JAMIA had slightly different article category names and several article subcategories before 2011 (for example, model formulation, synthesis of research).

We recorded dates for the online and in-print publication in a regular issue of the journal, or in a special online issue when applicable. We recorded whether the article was open (that is, freely available immediately at online release, before inclusion in a specific journal issue), free (that is, freely available when it was included in a print or online journal issue), or embargoed (that is, freely available to non-subscribers only after 1 year of embargo from the date of publication in an issue of the journal). We also recorded the number of downloads for abstracts, full-text in html, and full-text in PDF from the JAMIA web site (http://www.jamia.org).

We utilized the same methodology first described by Kim et al13 to calculate the number of citations per month (CPM) from the date at which the article became available online, and calculated DPM using the number of full-text online views plus the number of PDF downloads. For articles in which no date of online publication was available, the date of publication in a journal issue was used. We calculated p values for comparisons that were sufficiently powered. With an effect size (Δ) of 0.8 in CPM, the required sample size was 25 (for each group) for a desired statistical power (1-β) of 0.8 at a significance level (α) of 0.05 using a two-sample t-test.

The nine most frequent topic categories, which were also employed in our 2011 article,13 were used to classify the articles. Two authors independently selected the most relevant topic for an article, by retrieving MeSH terms from PubMed, when available, and reading the titles and abstracts. The κ interrater agreement was 0.8916. Another author served as an arbitrator when there was a conflict, selecting a topic from the two proposed topics (there were 75 disagreements). Supplementary appendix A (available online only) shows the types of disagreements. This final selection was then used for the analysis.

Results

The topics in JAMIA are constantly changing. The visual trends for the nine most frequent MeSH topics are plotted in figure 1 to show the changes in the past 4 years. The figure shows the total number of articles in each of the categories (right vertical axis), and the breakdown of the average number of CPM (left vertical axis). A topic for which the block of bars is displayed in the group 201X in the X-axis shows the average number of CPM since publication for articles published each year between 2009 and 201X, with one bar representing each year, as indicated in the legend.

Figure 1

Breakdown of number of accepted articles (line plot, right axis), average number of citations for articles published each year from 2009 to 2012 (bar plot, left axis), grouped by year in which the citation appeared (X-axis). (A) and (B)–(J) show the breakdown of citations for all articles and for articles in nine different categories, respectively. Note that the scale for the right axis, which represents the number of published articles, varies depending on the topic, while the left axis is fixed. ACPA, average number of citations per article; NLP, natural language processing.

Medical Record Systems was the most frequently assigned topic (124/573), while Prevention and Control represented the smallest group (24/573). Medical Informatics included articles related to biomedical and health informatics education, as well as some policy and other articles that could not be well classified into the other categories. As shown in figure 1, Medical Record Systems, Algorithms, and Methods are the fast growing categories in terms of accepted articles, and they also show an increasing trend in citations. Natural Language Processing, Medical Informatics, Information Storage and Retrieval, and Methods have a consistent pattern of increased citations. Topics related to computer science, such as Algorithms and User–Computer Interaction tend to have relatively fewer citations than those related to healthcare.

The correlations between DPM and CPM are significant in all years—that is, 2009 (p<0.01), 2010 (p<0.01), 2011 (p<0.01) and 2012 (p=0.06)—at a 0.1 threshold. These results indicate a consistent trend between downloads and citations in general. The CPM versus DPM plots are illustrated in figure 2, in which we also show the linear regression model for each year.

Figure 2

Downloads per month (DPM) versus citations per month for all articles in the past 4 years. DPM include full-text online views and PDF downloads.

Figure 3 shows the boxplots displaying CPM for open access (freely available on publication online), free (free access on publication in a journal issue), and embargoed articles (for 2009 there were only embargoed articles, and for 2010 there was a small number of free access articles but no open access option).

Figure 3

Boxplots for citations per month displaying citation trends from 2009 to 2012. (A) Boxplots of open access plus free articles and embargoed articles, and p values for comparisons that are powered to detect significant differences (β=0.2). (B) Comparisons among open access (that is, free on online publication), free (that is, free on publication in a journal issue), and embargoed articles for 2011 and 2012. Note that 2009 did not have open access or free articles, and 2010 did not have an open access option. Analyses for 2012 are preliminary given the relatively short time from publication to the time all data were collected in mid-2013.

Figure 3 also shows a number of comparisons based on the free availability of papers from 2010 to 2012. In figure 3A, we combined open and free access articles into one category (O+F), then compared to embargoed articles. O+F articles received significantly higher CPM in 2011 when tested with a two-tailed t-test (p=0.0036). This trend continued in the preliminary analysis for 2012, and the difference was statistically significant (p=0.0410). The free (typically either editor's choice or AMIA-endorsed) articles received higher CPM than the embargoed articles in the year 2011 (analysis not sufficiently powered given only 21 free articles) (see figure 3B). The open access articles in 2012 also had a higher CPM than the embargoed articles in the same year, but the difference was not statistically significant (p=0.3600).

As the embargoed articles are made freely available after 1 year of publication, there could be a stronger effect in citations accrued in the short term (that is, in the same year of publication and the subsequent year). We thus compared the mean number of citations (for two consecutive years including the year of publication) for freely available and embargoed articles published in 2010 and 2011 (see figure 4). For 2012, there was incomplete information as the total number of citations for 2013 was not yet available. Results show that this difference was significant for 2011 (p=0.0034), the only year for which the test had sufficient power.

Figure 4

Citations (in two consecutive years including year of publication) for articles published in 2010 and 2011. The difference between freely available (free in 2010 or open access plus free in 2011) and embargoed articles was evaluated using a two-sample t-test. The test was not sufficiently powered for comparisons of 2010 articles.

Table 1 lists the most cited 2009–12 papers for two consecutive years starting with the one in which they were published in a journal issue and gives an idea of topics and types of articles that attracted most short-term citations. Table 2 lists the most downloaded papers for the entire period.

Table 1

Most cited articles when considering two consecutive years including the year of publication in a journal issue

Rank*	Title	Citations	Type
Most cited 2009 articles in 2009–10
1	A randomized trial comparing telemedicine case management with usual care in older, ethnically diverse, medically underserved patients with diabetes mellitus: 5-year results of the ideatel study14	12	Research paper
2	Tiering drug–drug interaction alerts by severity increases compliance rates15	12	Viewpoint paper
3	What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior?16	11	Review paper
4	Active computerized pharmacovigilance using natural language processing, statistics, and electronic health records: a feasibility study17	10	Research paper
5	Does computerized provider order entry reduce prescribing errors for hospital inpatients? a systematic review18	9	Review paper
6	Recognizing obesity and comorbidities in sparse data19	9	Viewpoint paper
7	Core content for the subspecialty of clinical informatics20	8	AMIA board white paper
8	Community-wide implementation of health information technology: the Massachusetts ehealth collaborative experience21	8	Case report
9	Program requirements for fellowship education in the subspecialty of clinical informatics22	7	AMIA board white paper
10	Electronic support for public health: validated case finding and reporting for notifiable diseases using electronic medical data23	7	Viewpoint paper
Most cited 2010 articles in 2010–11
1	Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications24	17	Application of information technology
1	Extracting medication information from clinical text25	17	Viewpoint paper (FREE)
3	An overview of MetaMap: historical perspective and recent advances26	17	Synthesis of research
4	MedEx: a medication information extraction system for clinical narratives27	17	Application of information technology
5	Leveraging informatics for genetic studies: use of the electronic medical record to enable a genome-wide association study of peripheral arterial disease28	15	Research paper
6	Health information technology: fallacies and sober realities29	14	Viewpoint paper
7	Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)30	14	Model formulation (FREE)
7	Evaluating re-identification risks with respect to the HIPAA privacy rule31	14	Research paper
9	The impact of computerized provider order entry on medication errors in a multispecialty group practice32	12	Research paper (FREE)
10	Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC)33	9	Application of information technology
Most cited 2011 articles in 2011–12
1	2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text34	23	Perspective
2	Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings35	20	Review (FREE)
3	Factors motivating and affecting health information exchange usage36	15	Research and applications
4	Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting37	15	Perspective
5	Translational bioinformatics: linking knowledge across biological and clinical realms38	13	Perspective (OPEN)
6	Data from clinical notes: a perspective on the tension between structure and flexible documentation39	13	Perspective
7	Anaphoric relations in the clinical narrative: corpus creation40	11	Research and applications
8	Social disparities in internet patient portal use in diabetes: evidence that the digital divide extends beyond access41	11	Case report (FREE)
9	Ability of pharmacy clinical decision-support software to alert users about clinically important drug–drug interactions42	10	Research and applications
10	Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support43	10	Research and applications
Most cited 2012 articles in 2012 to June 2013
1	Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study44	15	Research and applications
2	iDASH: integrating data for analysis, anonymization, and sharing45	10	Brief communication
3	A novel signal detection algorithm for identifying hidden drug–drug interactions in adverse event reports46	10	Research and applications (FREE)
4	The National Center for Biomedical Ontology47	9	Brief communication
5	The dangerous decade48	7	Perspective
6	The financial impact of health information exchange on emergency department care49	7	Research and applications (OPEN)
7	High-priority drug–drug interactions for use in electronic health records50	7	Research and applications
8	A translational engine at the national scale: informatics for integrating biology and the bedside51	6	Brief communication
9	A systematic review of the psychological literature on interruption and its patient safety implications52	6	Review
10	Portability of an algorithm to identify rheumatoid arthritis in electronic health records53	6	Research and applications (FREE)

*Ties are broken by date of publication, with most recent articles ranking higher.

Table 2

Most downloads* from 2009 to 2012

Rank	Title	PDF downloads plus full-text online views	Article type (availability)	Year published in a journal issue
1	Transmitting and processing electronic prescriptions: experiences of physician practices and pharmacies54	10218	Research and applications (OPEN)	2012
2	Definition of biomedical informatics and specification of core competencies for graduate education in the discipline55	9166	AMIA Board white paper (FREE)	2012
3	The impact of computerized provider order entry on medication errors in a multispecialty group practice32	8985	Research paper (FREE)	2010
4	The financial impact of health information exchange on emergency department care49	7397	Research and applications (OPEN)	2012
5	The SMART Platform: early experience enabling substitutable applications for electronic health records56	7177	Research and applications (OPEN)	2012
6	The impact of the electronic medical record on structure, process, and outcomes within primary care: a systematic review of the evidence57	6660	Review (FREE)	2011
7	Shifts in the architecture of the Nationwide Health Information Network58	6621	Perspective (OPEN)	2012
8	The inadvertent disclosure of personal health information through peer-to-peer file sharing programs59	6400	Research paper	2010
9	Health information technology: fallacies and sober realities29	5734	Viewpoint paper	2010
10	Health-information exchange: why are we doing it, and what are we doing?60	5591	Perspective (FREE)	2011

*Note that the total number of downloads is calculated as the sum of the full text access and PDF downloads, and does not include abstract access.

Discussion

JAMIA has published several special focus issues in the past 2 years, and the scope of the journal was expanded to include translational bioinformatics,38 61–74 economic evaluations of health information technology,49 75–80 and brief communications of large national initiatives for sharing of tools and data.45 47 51 81–86 It is too early to know whether these topics will result in higher, the same, or lower numbers of citations than other topics. Some topics appear to have an increasing number of citations in the past few years, and the distribution of articles across the topics also appears to be changing.

There are several limitations in the topic analysis. First, only one topic was allowed per article. We could have chosen two or more per article. Fixing the number of topics per article was considered to be important so an article that was assigned several topics would not have a greater influence in the citation count than an article that was assigned fewer topics. By assigning just the main topic to an article, the arbitration was also much simpler, but may also have introduced bias.

The patterns of citations for different topics vary considerably. For a small but highly varied field such as biomedical informatics, in which some researchers are doing research more closely aligned with computer science than medicine, some form of weighting should be developed to account for the fact that the average number of citations to a computer science topic is smaller than the average number of citations to a health science topic.87

The analyses displayed for 2012 are very preliminary, as the number of CPM is still expected to increase. Citations were collected mid-year in 2013, and are significantly lower than the total number of citations expected in 2013. The number of downloads may serve as a proxy for upcoming citations in these years. However, this proxy has limitations. Although correlations between DPM and CPM are all statistically significant, it is easy to see in figure 2 that some highly downloaded articles have relatively few citations and vice versa. It is possible that some articles appeal to many readers, but only some of these appeal both to readers and authors. That some articles are highly cited but have relatively few downloads is harder to explain, but it is possible that authors are citing articles after only reading their abstracts online, or that they are using a paper copy of the journal. More research is needed to determine patterns of articles for which DPM and CPM are not correlated.

An important trend towards an increased number of online full-text views and downloads was observed. Free downloads and online views were not always associated with increased citations, but may be important for the dissemination of biomedical informatics to journal non-subscribers. Some influential articles may not result in a large number of citations, but may have a significant impact on the dissemination of knowledge into clinical practice and public policy.88 It is possible that this is happening with some JAMIA articles. The number of downloads is very high for recent free or open access articles. However, it is too early to assess the impact of these articles in clinical practice and health policy. In addition, the assessment of download frequencies can be biased (for example, it is possible to generate a large number of downloads to any particular article by developing automated programs to access those articles) and hence it is difficult to use download frequency in isolation as a measure of academic productivity and impact. In addition, we do not report here on PubMed Central download frequencies, which are likely to constitute a large portion of downloads.89 It is interesting to note that freely available 2012 articles dominate the list of most downloaded articles (that is, five were from 2012, and eight were freely available on publication in a journal issue). We will follow the trend of citations for those articles in the upcoming years to see whether they result in an overall higher number of citations than articles that were not freely available on publication. As, according to Davis,90 social stratification may result in most citations originating from authors who are based in institutions that have access to subscription journals, it is possible that citation rates will not be associated with free access status.

The analyses presented here show trends in the field that emphasize association but not causation. They have important limitations. There are unavoidable biases in terms of the analyses involving free articles. These are typically AMIA communications endorsed by its board of directors, and usually reporting on the results of workshops, or consensus statements, and JAMIA editor's choice articles, which represent embargoed articles that the editor considered important to be freely disseminated in the short term. Therefore, it could be the case that this designation was the main driver for higher exposure and higher citation rates. In addition, many of these articles had corresponding freely available webinars that attracted thousands of viewers,91 and some had news media or social media coverage.92 Similarly, some authors may have elected the open access option depending on the importance they placed on short-term dissemination. This could result in ‘self-selection’ of the authors’ best work, which might have been more highly cited anyway. In a retrospective study like this, it is impossible to eliminate these potential biases, but they do not invalidate the observation that freely accessible articles generate a larger number of online full-text views and downloads, and that they have a tendency to be more cited.

Conclusion

The rates of publication and citation vary within biomedical informatics topics. Our analyses showed that, for the JAMIA articles studied, the average monthly number of online full-text views plus article downloads (DPM) was highly correlated with the average number of CPM, and hence it may potentially serve as a short-term surrogate for the latter. We also showed an overall tendency for a higher number of citations to JAMIA articles that are freely available at the time of publication. Citation differences between free and embargoed articles were significantly higher for the former in 2011 (p=0.0096).