Factors predictive of an academic otolaryngologist's scholarly impact.

OBJECTIVES: Describe the h index as a bibliometric that can be utilized to objectively evaluate scholarly impact. Identify which otolaryngology subspecialties are the most scholarly. Describe if NIH funding to one's choice of medical school, residency, or fellowship has any impact on one's scholarly output. Determine other factors predictive of an academic otolaryngologist's productivity. STUDY
DESIGN: Analysis of bibliometric data of academic otolaryngologists.
METHODS: Active grants from the National Institutes of Health (NIH) to otolaryngology departments were ascertained via the NIH Research Portfolio Online Reporting Tools Expenditures and Reports database. Faculty listings from these departments were gleaned from departmental websites. H index was calculated using the Scopus database.
RESULTS: Forty-seven otolaryngology programs were actively receiving NIH funding. There were 838 faculty members from those departments who had a mean h index of 9.61. Otology (h index 12.50) and head and neck (h index 11.96) were significantly (P < 0.0001) more scholarly than the rest of subspecialists. H index was significantly correlative (P < 0.0001) with degree of NIH funding at a given institution. H index was not significantly higher for those that attended medical school (P < 0.18), residency (P < 0.16), and fellowship (P < 0.16) at institutions with NIH funding to otolaryngology departments.
CONCLUSIONS: H index is a bibliometric that can be used to assess scholarly impact. Otology and head and neck are the most scholarly subspecialists within otolaryngology. NIH funding to an individual's medical school, residency, or fellowship of origin is not correlative with one's scholarly impact, but current institutional affiliation and choice of subspecialty are.

Introduction

Many factors influence the scholarly productivity of an academic otolaryngologist. Even at the earliest stages of training, some prospective physicians may choose a training institution based on prestige of the institution, cost, focus of expertise, location, and proximity to family or friends. Some prospective physicians may consider that the amount of external funding channeled to the institution can be a positive influence on a future career in academic medicine.

Extramural funding is frequently utilized to evaluate the research productivity of a given medical institution. The U.S. News and World Report annually publishes a ranking of acclaimed medical schools and their associated departments, and uses the amount of funding from the National Institutes of Health (NIH) to determine a significant portion of the ranking system. Many unofficial ranking systems also incorporate NIH funding totals as a metric to gauge research productivity of both hospitals and individual departments. Although using NIH funding as a metric for research productivity is an objective assessment, some have argued that it does not truly represent scholarly impact.

Metrics traditionally used to gauge an individual’s scholarly impact include one’s number of publications and total number of citations. However, both of these metrics have limitations. If an author has published many articles that have little relevance to his medical discipline, he may be unduly acclaimed as an avid scholar if the number of publications is used as the gauge. Conversely, if an author has spent considerable effort in producing few, yet impactful publications that have contributed significantly to his scientific discipline, he would be unfairly unheralded if the number of publications were used as the metric. If total number of citations by other writers is used as a metric, an author may be well regarded if he is part of a single paper that has been widely cited in the literature, yet still lack a certain quantity of publications that exhibit a consistent scholarly contribution to the field. Furthermore, an author may be cited frequently in the literature for a negative reason (such as a series of disastrous surgical complications), thereby increasing his total number of citations and inflating his presumed scholarly impact to the field.

The h index was described in 2005 by Dr. Jorge Hirsch at the University of California–San Diego as a tool to capture both the quantity and the quality of an individual’s research productivity. He defined the h index to be the number of papers with citation number ≥ h. For example, if an author had 25 publications with 10 publications having at least 10 citations in the literature, then his h index would be 10. As soon as his 11th publication was cited for the 11th time then his h index would increase to 11.

We wondered if an otolaryngologist’s h index would correlate well with the sum total of NIH funding granted to his affiliated otolaryngology department. We hypothesized that a department with high amounts of NIH funding would be associated with faculty members with a higher mean aggregate h index. Furthermore, we wondered if NIH funding granted to an otolaryngologist’s previous choice of training institutions would have any implications on his/her future scholarly output. We also wondered which otolaryngology subspecialties were the most scholarly.

Methods

The NIH’s Research Portfolio Online Reporting Tools Expenditures and Reports (RePORTER) database was queried to identify all United States otolaryngology departments with active NIH grants. Sum totals of NIH dollars were calculated for each department.

Faculty lists from the above-described departments were then generated based on individual departmental websites. The following data were collected about each faculty member: academic rank, subspecialty, and associated medical school, residency, and fellowship when listed. Chairmen were counted in their own category (instead of with the professor subgroup) and the division head was counted as a “chairman” if otolaryngology was a division of general surgery. Individuals who were nonacademic “clinical” or “adjunct” faculty were not included for analysis. NIH dollars tied to an individual’s previous training institutions were also listed, if applicable. This consisted of current NIH dollars affiliated with their previous institutions; not necessarily funding that was present when that individual trained.

The h index was calculated for each faculty member using the calculator on the Scopus Database (www.scopus.com). Statistical analyses were performed using a one-way student’s t-test and analysis of variance (ANOVA) with p = 0.05 used as a threshold for significance.

Results

Forty-seven otolaryngology departments were identified that received NIH funding. Of these, there were 838 total faculty members identified who had an aggregate mean h index of 9.61. The mean NIH total of NIH grants to a given department was $2,721,759 and the median was $1,044,025. The minimum sum of grants was $108,000 (University of Virginia) and the maximum was $13,420,181 (Johns Hopkins University).

As seen in Fig. 1, otology (h index 12.5) and head and neck (h index 11.96) had higher h indices than all other subspecialties (p < 0.0001) but there was no statistically significant difference between otology and head and neck themselves (p = 0.60).

Figure 1

Mean H index per otolaryngology subspecialty.

As noted in Fig. 2, there was a significant ANOVA, (P < 0.0001) stepwise increase in the h index with increasing academic rank. Assistant professors (n = 375) had an h index of 4.57 (SEM .23), associate professors (n = 206) had an h index of 9.85 (SEM .43), professors (n = 210) had an h index of 16.09 (SEM .66), and chairmen (n = 47) had an h index of 20.04 (SEM 1.55).

Figure 2

H index per academic rank.

When stratified according into quartiles of NIH funding, it was noted that those faculty members that belonged to higher-funded institutions had significantly higher (ANOVA, P < 0.0001) h indices, as seen in Fig. 3.

Figure 3

H index of individuals in NIH Funded otolaryngology departments.

As seen in Fig. 4, faculty was stratified according to whether or not the otolaryngology department affiliated with their medical school received NIH funding. Three hundred and sixty-three attended a medical school without current NIH funding to the otolaryngology department and had a mean h index of 9.32. Four hundred and seventy-five attended a medical school with departmental NIH funds and had a mean h index of 9.84. There was no significant difference (single tail student t-test, P < 0.18).

Figure 4

H index stratified according to presence of NIH funding to otolaryngology departments affiliated with each phase of one’s training.

A similar stratification was performed according to otolaryngology NIH funds to one’s residency program. Two hundred and fifty-five faculty members attended a residency program without NIH funding and had a mean h index of 9.18 while 583 attended a residency with NIH funds and had a mean h index of 9.80 (single tail student t-test, P < 0.16).

Six hundred and fifty-eight faculty members reported completing a fellowship in a subspecialty of otolaryngology. Of these, One hundred and seventy-seven attended a fellowship without NIH funding and had a mean h index of 10.59. Four hundred and eighty-one completed a fellowship in a department with NIH funding and had a mean h index of 9.80. Again, there was no significant difference (student t-test, P < 0.16).

Discussion

Many metrics have been used to gauge one’s scientific scholarly impact. Total number of publications, total number of citations, and citations per publication have historically been used, but each of these has its drawbacks, as previously discussed. The h index is a bibliometric reported by Hirsch in 2005 which seeks to overcome the aforementioned weaknesses by capturing both the quantity and quality of an author’s work. The advantages of the h index are that it is not skewed by any one well-cited paper, and it is not skewed by a high number of poorly cited papers. It has even been shown to have predictive properties of future scholastic achievement.

Critics of the h index posit that it does not reflect why a paper was cited (possibly for negative reasons), it does not account for the significance of an author’s contribution to a publication when there are multiple authors, it does not account for the “Matthew effect” that a highly renowned author will be disproportionately cited more frequently than those of lesser fame, and those in the non-English speaking world are at a disadvantage.

Self-citation has also been discussed as a negative drawback of the h index., Theoretically, if an individual knew he was on the cusp of increasing his own h index by one more citation to a certain publication, he may be motivated to self-cite that publication. This tactic may be particularly effective for young authors with low h indices. Engqvist and Frommen studied this dilemma in 40 scientists and found only a minimal impact on the h index after excluding self-citation. It was concluded that over time the h index was fairly robust and immune to self-citation.

Because the Scopus database does not index articles before 1995, the h index for older authors may be unfairly low. The h index can also be calculated from other online tools such as the Thomson Institute of Scientific Information Web of Science database, and Google Scholar. Although Google Scholar indexes articles before 1995, there has been a high degree of correlation noted with Scopus.

The h index has gradually been introduced into several medical disciplines including neurosurgery, radiology, urology, anesthesiology, and gastroenterology. Work within the otolaryngology literature has shown that among otolaryngology principal investigators with NIH grants, those with an MD degree had a higher h index than either an MD/PhD or a PhD degree. A recent report showed that mean h indices for chairmen are lower in otolaryngology (15.8) compared to chairmen in general surgery (27.8), internal medicine (24.6), neurosurgery (20.3), orthopedics (19.4), but higher than chairmen within radiology (15.2) and anesthesiology (12.3). It should be noted that, in contrast to this publication, our direct calculation of otolaryngology chairs’ H-indices revealed the mean to be somewhat higher (20.04).

A number of reasons have been proposed as to why there is so much variability between h indices across different specialties of medicine. Hirsch noted that the average h index across fields can vary from factors such as the number of scientists in a field, the average number of publications per scientist in the field, the average number of publications per scientist in a field, and the applicability of the field to other fields. Because there are more general surgeons overall than otolaryngologists, for example, there may be more opportunity for a general surgeon to have his work cited as there are an increased number of publications in that field.

It is not surprising that the h index increases with academic rank as displayed in the present study. Certainly there are several components that are considered when a faculty member is considered for academic promotion. One’s clinical practice volume, research productivity, and the ability to teach residents and medical students may all factor into the decision of academic advancement. The h index may be used as an objective measure when evaluating a faculty member’s scholarly impact in his/her field.

The present study indicates that there is a correlation between highly NIH-funded otolaryngology departments and a higher h index of the faculty that constitute that department. It is likely that this relationship is not causal, but rather due to those institutions having a preference to offer academic appointments to those individuals that are inherently more research-driven. Furthermore, those individuals that are inherently more research-driven may seek out employment opportunities in departments where extramural funding is already established.

It appears from our data that there is no relationship between an individual’s future scholarly activity and the NIH funding granted to the otolaryngology departments of that individual’s medical school, residency, and fellowship. One major assumption to make this conclusion is that there is relatively little change in the funding granted to those otolaryngology departments from the time the individual trained to the current time. We felt that although there may have been minor increases or decreases in NIH funding granted to a given institution over time, highly academic institutions over the last 30 years generally continue to be highly academic today.

Although one would assume that individuals with high h indexes are the products of more robust NIH funding, 30% of 70 Nobel laureates who won recognition in medicine, physics, and chemistry between 2000 and 2008 produced Nobel-level work even when completely unfunded.

This study indicates that both otology and head and neck had higher h indices and by extension were more scholarly than other subspecialties. However, one point to consider is that other subspecialties (e.g., rhinology, laryngology) are relatively newer and individuals with fellowship training within a newer subspecialty would not have had as much time pass by for their research to be cited, which would naturally decrease their h index.

Conclusions

The h index is a bibliometric that can be used to assess scholarly impact. Otology and head and neck are the most scholarly subspecialties within otolaryngology. The h index has a high correlation with increasing academic rank and can be used to evaluate academic promotion. NIH funding to an individual’s choice of medical school, residency, or fellowship was not correlative with one’s future scholarly impact.

Funding

None.

Financial disclosures

None.

Declaration of Competing Interest

None.