'Initial Clinical Experience' articles are poorly cited and negatively affect the impact factor of the publishing journal: a review.

OBJECTIVES: The phrase 'Initial Clinical Experience' in a manuscript's title implies that the described technique offers promise of future clinical relevance. We assessed, using rates of subsequent citations, the actual academic relevance of such articles in comparison to articles not containing the phrase.
DESIGN: We searched ISI database for articles that included the studied phrase in their titles between 1975 and 2009 and grouped the results by the related medical specialty. We excluded articles from journals with no available impact factor. For each identified article, we extracted number of included patients, citations/year, the average impact factor of the publishing journal over the last five years and the proportion of articles published in the same journal that garnered zero subsequent citations.
SETTING: Retrospective review of a scientific database. PARTICIPANTS: None. MAIN OUTCOME MEASURES: Citation rate.
RESULTS: Among a total of 982,127 articles published in 186 journals representing eight major publishing medical specialties, 531 (0.05%) were Initial Clinical Experience articles. Thirty percent of Initial Clinical Experience articles were never cited compared with 7% of the overall article volume (p < 0.0001). Citations/year for Initial Clinical Experience articles were significantly lower than the median impact factor (p < 0.0001). There was no correlation between citations and number of patients described in the Initial Clinical Experience articles (p = 0.61).
CONCLUSIONS: Initial Clinical Experience articles are cited less frequently than the average, especially in Cardiovascular, Radiology and Ophthalmology journals.

Introduction

Authors of clinical studies frequently include modifiers to titles of articles, especially those focused on relatively new methodologies. These modifiers include words such as ‘preliminary’, ‘pilot’, ‘early’ or ‘initial’. It is not entirely clear whether authors are trying to impart the sheen of novelty to their work or are simply trying to lower the expectations of reviewers and editors, so that the study can be given the ‘benefit of the doubt’ given its (apparent) newness.

Editors typically strive to maximize the academic relevance of their respective journals. One of the most widely applied metrics of such relevance is the rate of citation of papers in a given journal. Although many measures of relative rates of citation are available, journal impact factor represents the most widely applied metric. Impact factor simply counts the number of current year citations to the source items published in that journal during the previous two years.[1]

Publication of nascent technologies or methodologies introduces both potential risk and reward for journal editors focused on maximizing high impact factors. If work that is described by the authors as ‘early’ subsequently enjoys widespread application, then publication of that early work likely will lead to high rates of citation and thus improves a journal's impact factor. However, if such early work fails to stimulate future studies, the journal's impact factor will suffer. As a continuation of a previous effort of studying ‘preliminary’ and ‘pilot’ articles,[2] the purpose of the current study was to characterize the subsequent academic relevance of papers with titles containing the phrase ‘Initial Clinical Experience’.

Methods

Data selection

In October 2011, we collected our study cohort from ISI Web of Knowledge database.[3] We searched between 1975, the earliest date provided by ISI, and 2009, to allow a minimum of two years for articles to be cited. We used the search phrase ‘Initial Clinical Experience’ in the ‘Title’ search bar to identify articles of interest and, accordingly, the publishing journal. Articles from journals that had no reported impact factor on ISI were excluded from our studied population. The ISI database provides not only the number of citations of the studied article and the impact factor of the publishing journal in the last five years, but also categorizes articles by medical specialty. Articles were categorized into one of the following groups: Anesthesiology, Cardiovascular system, Neurosciences, Oncology, Ophthalmology, Radiology, Surgery, Urology and other specialties with less frequent Initial Clinical Experience publications (i.e. Gastroenterology, Hepatology, Dermatology). We then extracted the number of citations of each included article, the publishing year, the five most recent impact factors of the publishing journals and the number of patients of each study. The five most recent values of each journal's impact factor were averaged. We also examined all articles that were published by the included journals over the same study period (1975–2009) and calculated the percentage that were never cited in each group.

Given the cumulative nature of number of citations, older articles are more likely to have more citations than newer articles. To normalize older and newer articles, we calculated the ‘citations per year’ for each article as the number of citations garnered divided by the number of years since publication.

Statistical analysis

All statistical analyses were performed using the software packages Microsoft Excel 97 (Microsoft Corporation, Redmond, WA, USA) and JMP (SAS Institute, Cary, NC, USA). Statistical significance was assigned to P values of <0.05. Continuous variables were presented as median with interquartile ranges due to deviations from normal distributions as calculated using the Shapiro–Wilk test. Correlation between number of patients and citations per year was determined using linear regression. Two-tailed Fisher's exact test was used to analyse differences in the never-cited proportion among specialties. Wilcoxon rank-sum test was used to compare median citations per year with the average impact factor and to compare the median number of patients among specialties.

Results

Our search identified 581 articles from 201 journals that contained the phrase Initial Clinical Experience in their titles. In total, 50 (9%) of these articles from 15 journals were excluded as the journals did not report an impact factor. As such, 531 articles from 186 different journals were included in our study.

Table 1 summarizes the demographics of the studied articles and their publishing journals. Cardiovascular (33%, 174/531) and Radiology (21%, 112/531) represented the most frequently publishing specialties of initial clinical experience articles. Median number of included patients in Initial Clinical Experience articles was found to be higher in Radiology, Surgery and Cardiovascular journals without significant differences among specialties (p = 0.25). There was no correlation between the median number of patients in each specialty and the subsequent citation rate (p = 0.61).

Table 1

Journal and article demographics

Journal specialty	Number of journals	Total number of articles	Initial Clinical Experience articles
			Number	Median number of patients (IQR)*	Median publication year
All	186	982,127	531	19 (9–39)	2001
Anesthesiology	7	36,906	12 (2.3%)	12 (7–51)	1996
Cardiovascular	37	175,163	174 (32.8%)	20 (9–47)	2000
Neurosciences	19	90,699	40 (7.5%)	15 (9–46)	2004
Oncology	15	85,131	36 (6.8%)	10 (6–28)	2005
Ophthalmology	7	55,204	19 (3.6%)	17 (13–43)	1998
Other	57	318,398	60 (11.3%)	14 (7–23)	1998
Radiology	20	100,037	112 (21.1%)	25 (13–40)	2000
Surgery	16	60,107	28 (5.3%)	25 (11–39)	2003
Urology	8	60,482	50 (9.4%)	18 (6–55)	2006

*Interquartile range

Table 2 summarizes the citation rates of Initial Clinical Experience articles compared with the overall published volume. Overall, 159 (30%) of 531 Initial Clinical Experience articles were never cited. The proportion of never-cited articles was significantly different among journal specialties (p = 0.0420). Journals with the highest proportion of never-cited articles were specialized in Ophthalmology (47%, 9/19), Urology (40%, 20/50) and Cardiovascular (36%, 62/174). A significantly higher proportion of Initial Clinical Experience articles were never cited (30%, 159/531) compared with all articles (7%, 68,559/982,127) (p < 0.0001).

Table 2

Comparison of citation rates between Initial Clinical Experience articles and overall journal publications

Journal specialty	Articles with zero citations			Median citations per year
	All articles	Initial Clinical Experience articles	p value	All articles (journal impact factor) (IQR)*	Initial Clinical Experience articles† (IQR)*	p value
All	68,539 (7%)	159 (30%)	<0.0001	3.0 (1.8–4.5)	1.5 (0.5–3.9)	<0.0001
Anesthesiology	2327 (6%)	2 (17%)	0.17	2.5 (0.8–3.2)	1.2 (0.3–2.7)	0.20
Cardiovascular	14,250 (8%)	62 (36%)	<0.0001	2.9 (1.4–4.2)	1.3 (0.3–3.0)	<0.0001
Neurosciences	4608 (5%)	11 (28%)	<0.0001	2.7 (1.9–3.0)	2.4 (1.0–5.5)	0.56
Oncology	2350 (3%)	11 (31%)	<0.0001	4.4 (3.6–4.4)	2.3 (0.8–7.5)	0.21
Ophthalmology	1475 (3%)	9 (47%)	<0.0001	2.9 (2.1–3.2)	1.7 (0.8–4.0)	0.17
Other	31,363 (10%)	13 (22%)	0.0072	1.9 (0.9–2.6)	1.0 (0.4–2.5)	0.0372
Radiology	5043 (5%)	27 (24%)	<0.0001	3.0 (2.1–5.7)	2.2 (0.8–6.2)	0.053
Surgery	3584 (6%)	4 (14%)	0.08	1.7 (1.2–2.6)	1.3 (0.2–4.0)	0.46
Urology	3539 (6%)	20 (40%)	<0.0001	2.2 (2.1–4.0)	1.6 (0.4–4.8)	0.24

*Interquartile range

†Excluding zero citation articles

Of the Initial Clinical Experience articles that were cited, the median citations per year was not significantly different among journal specialties (p = 0.127). Neuroscience, Oncology and Radiology articles had the highest median citations per year. Cardiovascular and Other articles were found to have significantly fewer citations per year than the average impact factor of their publishing journals (p < 0.0001, p = 0.0372 respectively). Overall, Initial Clinical Experience articles had significantly lower rates of citations per year in comparison to the average impact factor of the publishing journals (p < 0.0001).

Discussion

Our findings suggest that articles with the phrase Initial Clinical Experience in their title are less likely to be cited than other articles published in the same journals. This indicates that the volume of subsequent dependent research work does not reach the expected degree that is promised from the title. Thus, the lower average of citations would likely have a negative effect on the publishing journal's impact factor. Our results are comparable to an earlier study showing that only 27% of ‘preliminary’ or ‘pilot’ articles were subsequently followed by a more definitive publication.[2]

Several studies have examined the impact of an article's title on subsequent citations. Howard et al.[4] suggested that methodological and review articles tend to be cited more frequently than other article categories in Drug and Alcohol journals. Montori et al.[5] and Bhandari et al.[6] separately further specified that systematic reviews are more cited than narrative ones. Jacques et al.[7] concluded that number of citations was positively correlated with the length of the title and the presence of specific words like ‘colon’ or an acronym. They, however, found that reference to a specific country makes the article poorly cited. Our study of Initial Clinical Experience articles expands upon the effect of an article's category on its chances of success.

The current study has several limitations. Due to conducting an automated search, we may have missed related articles that do not literally mention our search phrase. We also calculated the average impact factor of each journal over the last five years even if the article was published earlier. Finally, we were forced to exclude (9%) of initially identified Initial Clinical Experience articles that were published in journals that did not list impact factors.

Conclusions

Articles with the phrase Initial Clinical Experience in their titles are cited less frequently than average articles published in the same journals, especially in Cardiovascular, Radiology, and Ophthalmology journals. These articles likely negatively affect the impact factor of the publishing journal.

DECLARATIONS

Competing interests

None declared

Funding

None declared

Ethical approval

Not applicable

Guarantor

IR

Contributorship

Authors

Acknowledgements

None

Reviewer

Farrokh Habibzadeh