Do Departments in a College of Medicine Differ in Academic Productivity As Assessed by the H-Indices of Tenured Faculty Members?

BACKGROUND: It is uncertain how different academic medical departments differ in academic productivity as assessed by commonly used bibliometric measures, eg, the h-index (the maximum value of h such that an author has published h papers that have each been cited at least h times). AIM: This project examined whether departments in the University of Iowa's Carver College of Medicine differed in h-indices of tenured faculty members.
METHODS: Based on 2020 data obtained from the College (and other University sources), the author compiled three data sets of Scopus h-indices of tenured faculty members identified by department, varying in size due to slightly different inclusion criteria (N's=334, 341, and 354). Analyses compared h-indices between ranks and among departments.
RESULTS: In the basic data set (N=334), h-indices of the 230 (69%) full and 104 (31%) associate professors differed based on a t-test, means (standard deviations)=37 (17) and 20 (7), respectively, p<0.0001. For both full and associate professors separately, departments differed in h-indices based on analyses of variance, p=0.04 and p=0.02, respectively. In the expanded data sets, departmental differences were significant for full and associate professors (with N=341) and full professors (with N=354).
CONCLUSION: Departments differed in academic productivity of tenured faculty members as assessed by h-indices. This was not a powerful, monolithic effect, ie, relative departmental standings for full and associate professors were not consistent, and departmental differences for associate professors were nonsignificant in the largest (N=354) data set. Multiple factors probably contributed to departmental differences and should be further investigated.

INTRODUCTION

This project examined whether departments in the Roy J. and Lucille A. Carver College of Medicine (CCOM) of the University of Iowa differed in academic productivity of tenured faculty members as assessed by a commonly used bibliometric measure, Hirsch’s h-index (1). Hirsch noted that some quantitative measure of academic productivity is useful for evaluation and comparison for making decisions such as hiring and promotion of faculty members and award of grants. He proposed the h-index, ie, the maximum value of h such that an author has published h papers that have each been cited at least h times.

Hirsch suggested that h values of individuals would be expected to vary in different research fields (1). He reported that h-indices in the biological sciences tended to be higher than in physics, particularly with respect to the most highly cited researchers (1). His article seemed to imply that biological sciences and physics were distinct research fields across which h-indices should not be compared, but that it would be reasonable, with some qualifications, to compare h-indices within biological sciences as a whole or within physics as a whole. It might be inferred from this position that h-indices could reasonably be compared across all the departments of a medical school, because all could be construed as focusing on subfields of biological sciences. However, any a priori judgment about this, without evidence, seems debatable.

The main previous comparison of h-indices across diverse fields of medicine and academic ranks is a review and meta-analysis (2). This review demonstrated that h-indices increased with academic rank. The authors also indicated that there were unique distributions of h-indices among medical specialties. Confidence in the latter conclusion, however, should be guarded as it was based on comparisons across numerous separate studies of different fields that used very different methodologies and were done in different years. Another study reported h-index data across diverse fields of medicine and academic ranks, but did not directly compare fields (3). Other studies have compared smaller numbers of medical fields or subfields, but not as broad a spectrum (4-14).

The authors of this review and meta-analysis stated that they initially searched for h-index data concerning all types of clinical and non-clinical researchers, but due to limits on relevant publications, had to restrict their analysis to medical fields, and only to the limited number of fields for which studies that met their criteria were available (2). For the current project, the CCOM made available for analysis a 2020 h-index data set that it had compiled of CCOM faculty members in all departments. This afforded an opportunity to compare h-indices of both clinical and non-clinical tenured faculty members in all departments both among departments and between ranks, based on h-indices that were calculated simultaneously for everyone based on a uniform methodology. The purpose of this project was to make such comparisons.

Based primarily on the aforementioned review and meta-analysis (2), the author hypothesized that there would be differences in h-indices between tenured full and associate professors, and that both full and associate professors would show differences among departments.

AIM

This project examined whether departments in the CCOM differed in h-indices of tenured faculty members.

METHODS

H-index data set

On August 13, 2018, the CCOM initiated a streamlined process for tracking publications by requesting faculty members to establish an Open Researcher and Contributor Identifier (ORCiD) that was linked with their publications in the Scopus abstract and citation database and linked to the University of Iowa Libraries by September 1, 2018. Although mandatory, compliance was not complete. CCOM information technology staff arranged for daily downloading of Scopus data, including h-indices (1) as calculated by Scopus’ methodology (15). The CCOM Office of Research staff periodically reviewed the resulting data set, with greater, but not exclusive, emphasis on faculty members with higher h-indices. The CCOM Office of Research made the most recently reviewed data set, dated February 3, 2020, available for the present analysis. The CCOM Associate Dean for Research, who managed this process, considered the quality of the most recent h-index data set to be very good. The main problem detected in earlier reviews was that some faculty members had multiple Scopus profiles (created by Scopus’ algorithms) that contained different sets of publications, which resulted in inaccurate h-indices being calculated by Scopus. He indicated that CCOM staff had made a big effort to identify faculty members with multiple Scopus profiles and encourage these faculty members to unify them, and that this process had been substantially successful, so that there was only a handful remaining with discrepancies such as multiple Scopus profiles. Another type of discrepancy requiring attention was that some faculty members, particularly those with common names, had to be distinguished from other individuals with identical names.

Faculty Information

The h-index data set did not include faculty members’ departments, tracks, or ranks. Another CCOM administrator provided a list of CCOM faculty as of May 7, 2020, that contained such information. I merged this list with the h-index data set and identified discrepancies. The analyses were limited to tenured faculty members, who showed a smaller percentage of discrepancies than non-tenured faculty members. To resolve discrepancies, I made comparisons to a publicly available list of CCOM faculty as of October, 2019 (16), the general University of Iowa directory (17), and other information on the University of Iowa web site. I omitted one full professor who was not listed in either the May, 2020, nor the October, 2019, faculty lists, possibly because she received no salary from the University of Iowa. Two full professors had h-indices of zero in the h-index data set. I determined that these values were erroneous and corrected them.

The basic data set that I analyzed consisted of tenured faculty members with records in both the h-index data set and the May, 2020, faculty list (N=334). Departments with fewer than five tenured faculty members were combined into a category labeled “Other” for analysis. These departments were Dermatology (N=2), Family Medicine (N=3), and Physician Assistant Studies (N=1). There were no tenured associate professors in these departments.

Some additional tenured faculty members (N=9) were in the May, 2020, faculty list, but not the h-index data set. It was not feasible to calculate Scopus h-indices for two of them due to multiple Scopus profiles for them and other individuals with identical names. I determined h-indices for the others (N=7), who appeared to have been omitted from the h-index data set by mistake, and added them to the basic data set to form an expanded data set (N=341). I added additional faculty members (N=13) who were in the h-index data set, but not the May, 2020, faculty list, and who were in the October, 2019, faculty list, to form a larger expanded data set (N=354). They appeared likely to have been omitted from the May, 2020, faculty list because they transitioned from tenured to emeritus or adjunct appointments between October, 2019, and May, 2020 (N=7), in some cases due to moving to other institutions or retiring; or because their salary support came from appointments as Howard Hughes Medical Institute investigators (N=2) or areas of the University of Iowa outside CCOM (N=5).

Institutional Review Board Determination

The University of Iowa Institutional Review Board determined on May 19, 2020, that this project did not meet the regulatory definition of human subjects research and did not require Institutional Review Board review.

Statistical Analyses

I compared h-indices of full professors and associate professors with a t-test for two independent samples using the Satterthwaite approximation. I compared h-indices for full professors and associate professors separately among departments by one-way analyses of variance, with pairwise comparisons among departments done using Tukey’s Studentized Range Test. I calculated the Pearson correlation coefficient of departmental mean h-indices for full professors versus associate professors across departments. I used a significance level of α=0.05 with two-tailed tests and did statistical analyses using SAS 9.4 (SAS Institute Inc., Cary, NC).

RESULTS

The basic data set (N=334), which consisted of tenured faculty members with records in both the h-index data set and the May, 2020, faculty list, included 104 (31%) associate professors and 230 (69%) full professors. The difference between h-indices of full professors and associate professors was statistically significant, means (standard deviations)=37 (17) and 20 (7), respectively, difference=18 (95% confidence interval, 15–20), t(331.4)=13.18, p<0.0001, Cohen’s d=1.19.

Table 1 shows the rankings of departments by mean h-indices for full professors (left) and associate professors (right) separately. The departments are listed in descending order of mean h-indices for full professors. The differences among departments in h-indices were statistically significant for both full professors and associate professors, F(21, 208)=1.67, p=0.04, η2=0.14, and F(20, 83)=1.92, p=0.02, η2=0.32, respectively. Tukey’s Studentized Range Test showed that the significant pairwise differences in h-indices between departments were Psychiatry > Surgery for full professors and Neurosurgery > Orthopedics & Rehabilitation and Neurology > Orthopedics & Rehabilitation for associate professors. The Pearson correlation coefficient of departmental mean h-indices for full professors versus associate professors, calculated across departments, was not statistically significant, r=0.08, p=0.74, i.e., a department’s mean h-index for full professors did not predict its mean h-index for associate professors.

Table 1.

H-index values for tenured faculty members of the Roy J. and Lucille A. Carver College of Medicine, University of Iowa, in both the h-index data set and the May, 2020, faculty list (N=334). Abbreviation: SD=standard deviation. See the Methods section for an explanation of the category labeled “Other.” For associate professors, the SD is missing for Obstetrics & Gynecology and Urology because each had only N=1. The ranks are based on the means before rounding. Departments with identical ranks had identical means. The departments are ordered by the ranks for full professors. Pairs of means within a column marked by either b or c differed significantly from one another by Tukey’s Studentized Range Test.

Department	Full Professors				Associate Professors
	N	Rank	Mean	SD	N	Rank	Mean	SD
Radiation Oncology	5	1	52	17	2	19	12	1
Psychiatry	15	2	50b	26	9	10	21	8
Neurology	7	3	45	25	3	2	27b	13
Otolaryngology-Head & Neck Surgery	7	4	43	20	2	19	12	1
Physical Therapy & Rehabilitation Science	2	5	42	17	3	3	24	3
Radiology	10	6	41	17	4	9	21	5
Anatomy & Cell Biology	3	7	41	28	9	11	20	8
Ophthalmology & Visual Sciences	16	8	39	16	4	15	18	5
Internal Medicine	53	9	39	15	22	13	20	7
Anesthesia	4	10	38	12	7	12	20	9
Neuroscience & Pharmacology	6	11	38	15	5	5	23	3
Microbiology & Immunology	10	12	37	17	4	6	22	4
Neurosurgery	5	13	36	8	2	1	29c	1
Orthopedics & Rehabilitation	7	14	35	9	3	21	6b,c	4
Pediatrics	22	15	34	22	7	17	13	5
Pathology	9	16	33	14	6	4	24	7
Molecular Physiology & Biophysics	10	17	33	9	4	8	22	4
Biochemistry	12	18	33	8	4	14	19	7
Obstetrics & Gynecology	4	19	29	11	1	18	13	-
Urology	5	20	28	14	1	6	22	-
Surgery	12	21	24b	10	2	16	15	4
Other	6	22	22	12	0	-	-	-

When I added three more full professors and four more associate professors who were in the May, 2020, faculty list, but not the h-index data set, for whom h-indices could be determined (N=341), the differences among departments remained statistically significant for both full professors and associate professors separately, F(21, 211)=1.69, p=0.03, η2=0.14, and F(20, 87)=2.03, p=0.01, η2=0.32 (data not shown).

Table 2 shows the rankings of departments by mean h-indices for full professors (left) and associate professors (right) separately when, additionally, I added 10 more full professors and three more associate professors who were in the h-index data set, but not the May, 2020, faculty list (and who were in the October, 2019, faculty list) (N=354). For consistency between the two tables, the departments are listed in the same order as in Table 1. The differences among departments remained significant for full professors, but were no longer statistically significant for associate professors, F(21, 221)=1.63, p=0.04, η2=0.13, and F(20, 90)=1.49, p=0.10, η2=0.25, respectively. Tukey’s Studentized Range Test showed that the significant pairwise differences in h-indices between departments were Psychiatry > Surgery and Psychiatry > Other for full professors. There were no significant pairwise differences for associate professors. The Pearson correlation coefficient of departmental mean h-indices for full professors versus associate professors, calculated across departments, was not statistically significant, r=0.16, p=0.48.

Table 2.

H-index values for tenured faculty members of the Roy J. and Lucille A. Carver College of Medicine, University of Iowa, in both the h-index data set and the May, 2020, faculty list, plus 20 additional faculty members (N=354. Abbreviation: SD=standard deviation). This table includes 20 faculty members in addition to those included in Table 1, making a total N=354; see the Methods section for an explanation. Also, see the Methods section for an explanation of the category labeled “Other.” For associate professors, the SD is missing for Obstetrics & Gynecology because it had only N=1. The ranks are based on the means before rounding. Departments with identical ranks had identical means. The departments are ordered by the ranks for full professors in Table 1. Pairs of means within a column marked by either b or c differed significantly from one another by Tukey’s Studentized Range Test. No pairwise differences between means were significant for associate professors.

Department	Full Professors				Associate Professors
	N	Rank	Mean	SD	N	Rank	Mean	SD
Radiation Oncology	5	2	52	17	2	19	12	1
Psychiatry	17	1	52b,c	25	9	10	21	8
Neurology	8	3	45	24	4	2	27	10
Otolaryngology-Head & Neck Surgery	7	4	43	20	2	19	12	1
Physical Therapy & Rehabilitation Science	2	5	42	17	3	3	24	3
Radiology	11	6	41	16	4	9	21	5
Anatomy & Cell Biology	3	7	41	28	9	11	20	8
Ophthalmology & Visual Sciences	16	11	39	16	4	15	18	5
Internal Medicine	57	10	39	17	24	8	21	11
Anesthesia	5	9	40	11	8	12	20	8
Neuroscience & Pharmacology	6	12	38	15	5	5	23	3
Microbiology & Immunology	11	13	36	16	4	7	22	4
Neurosurgery	5	14	36	8	2	1	29	1
Orthopedics & Rehabilitation	7	16	35	9	3	21	6	4
Pediatrics	22	17	34	22	8	17	14	5
Pathology	10	15	35	14	6	4	24	7
Molecular Physiology & Biophysics	11	8	41	28	5	6	22	4
Biochemistry	12	18	33	8	4	13	19	7
Obstetrics & Gynecology	4	19	29	11	1	18	13	-
Urology	5	20	28	14	2	13	19	5
Surgery	12	21	24b	10	2	16	15	4
Other	7	22	23c	11	0	-	-	-

DISCUSSION

Associations of H-Indices with Departments and Ranks

The findings generally supported the hypotheses, ie, in the basic data set, with N=334, mean h-indices were higher for tenured full than associate professors and differed among departments for both full and associate professors. In the expanded data sets, differences among departments were significant for both full and associate professors with N=341 and full professors, but not associate professors, with N=354. However, relative departmental standings for full and associate professors were not consistent, r=0.08 with N=334 and r=0.16 with N=354. In pairwise comparisons of departments, only a few comparisons of the highest-ranked and lowest-ranked departments showed significant differences. Thus, while departments differed in mean h-indices overall, this was not a powerful, monolithic effect.

Related Prior Studies

The review and meta-analysis of h-indices across various medical specialties mentioned previously found differences among ranks (2), as did the current project. This finding accords with findings in other studies (3, 6) and is unsurprising, inasmuch as h-indices by definition cannot decrease over time, but always increase or remain stable, and most full professors were associate professors before they were promoted. The review and meta-analysis of h-indices across various medical specialties also stated that there were unique distributions of h-indices among specialties (2). As mentioned previously, this statement was based on comparisons across separate studies of different specialties that used different methodologies and were done in different years, whereas the current project involved h-indices that were calculated simultaneously for all tenured faculty members (both clinical and non-clinical) in all departments with uniform methodology. The relative departmental standings in the present project show little agreement with the corresponding standings in this review and meta-analysis for either full or associate professors. The reasons for this disagreement are unclear, but it also suggests that there is not a powerful, monolithic effect of departments.

Other studies, as mentioned previously, have either reported h-index data across diverse fields of medicine and academic ranks, but without directly comparing fields (3); or have compared smaller numbers (three to nine) of medical (especially surgical) fields or subfields, but not as broad a spectrum of fields (4-14). No studies have compared departmental h-indices for all tenured faculty members in an entire college of medicine, as was done in the current project.

There have also been comparisons of fields outside biological science, eg, a study of five social science fields reported a difference among fields in mean h-indices (18). Another study noted the wide variation in average number of citations per paper among 21 fields distinguished in the citation database of the Institute for Scientific Information (subsequently acquired by Clarivate) (19). Based on these citation averages, a mathematical model for adjusting raw h-indices to make them comparable between fields was proposed, but not validated by comparisons with actual h-indices for different fields.

Limitations

The current project had significant limitations. This project was limited to a single college of medicine. It is unknown whether the findings would generalize to other colleges of medicine. Merging departments into the “Other” category only for departments with fewer than five tenured faculty members left some departments with very small numbers of tenured full or associate professors, resulting in a strong influence of particular individuals on these departmental means. The CCOM’s data set of faculty members’ h-indices was derived from Scopus. Although Scopus is among the most comprehensive scientific citation databases, the h-indices calculated by its algorithms may be underestimates, because they are calculated based on the Scopus database only and with certain limits regarding coverage (15). Independent of the current project, a broader limitation is that the h-index itself has various weaknesses as a quantitative measure of academic productivity (20).

The CCOM’s Honors Program

One impetus to the current project was a news release by the CCOM that it was honoring CCOM faculty members who had attained h-indices of 50 or higher (21). This honors program seemed to implicitly assume that h-indices of faculty members in all departments could reasonably be compared. Perusal of the list of honorees suggested that some departments seemed overrepresented relative to others.

Reading between the lines of most honors or rankings, there often seems to be an implication that there was a level playing field with respect to attaining the designated threshold and that individuals who attained it were superior to others. Often, there can be a potential counter-narrative that the playing field is not level and that factors other than meritorious individual attributes influenced attainment of the designated threshold. With respect to the CCOM’s honors program (21), assuming a level playing field implies that individuals who made the list are better researchers than those who didn’t, perhaps due to greater creativity, intelligence, energy, achievement orientation, or other meritorious individual attributes. A counter-narrative might be that there are causes or correlates of high h-indices that are unrelated to meritorious individual attributes, whether at the individual level, the department level, or otherwise.

Potential Confounding Factors

With respect to the current project, such causes or correlates of high h-indices other than meritorious individual attributes could be considered potential confounding factors. This project examined associations of departments and ranks with h-indices, but did not attempt to identify potential confounding factors, including causes or correlates of individual or departmental differences. The finding that relative departmental standings were not consistent for full and associate professors suggests other causes or correlates distinct from departments per se had a substantial impact. The limitation of the current project to a single college of medicine entailed a limited sample size, which provided insufficient power to examine other causes or correlates.

There are many possible causes or correlates of higher or lower h-indices. Possibly relevant characteristics differing among individuals (and potentially confounded with their departmental affiliations) include demographic characteristics that are presumably unrelated to meritorious individual attributes, such as gender, race, ethnicity, and age; as well as other characteristics such as years of experience, degrees, fellowship training, grant funding, and topicality of selected research topics (3, 6, 8, 10, 22-26). Of these characteristics, the most frequently studied within medical specialties have probably been years of experience and gender (3, 6, 10, 22, 23). Unsurprisingly, h-indices generally increase with years of experience (3, 6). A frequent, although not universal, finding with respect to gender has been that women have lower h-indices than men overall across diverse medical specialties; this may be attributable, at least partly, to underrepresentation of women at more senior levels (3). Characteristics such as gender, race, and ethnicity might affect h-indices of individuals through invidious effects of interpersonal, systemic, and/or societal sexism, racism, and ethnocentricity; or through other influences such as greater child care obligations of female, relative to male, faculty members.

It seems more plausible to conceptualize departments as a proximate rather than an ultimate cause of individual differences in h-indices. In the current project, departments served to some degree as an imperfect, but unambiguous and convenient, operationalization of the ambiguous notion of a research field. A department or research field might serve as a cause or correlate of individual h-indices in various ways. One structural characteristic that might be important is whether the department or research field focuses on a relatively smaller or larger number of specific diseases or other research topics, and/or whether these topics are focused on by more or fewer types of medical specialists and basic scientists. Relatively more concentrated focus might lead to more researchers citing one another’s publications, resulting in relatively higher h-indices. It is possible that the top-ranked departments for full professors in Table 1, Radiation Oncology and Psychiatry, might have relatively more concentrated foci than some other departments. Other potential influences on h-indices at the level of research fields include the typical duration or amount of research experience required during residency, fellowship, or other graduate training; average number of references per article; average number of articles per researcher; number of researchers in the field; and typical size of collaborations (1, 2). Other potential influences of departments on h-indices that could include variations among specific departments, as opposed to general research fields, include the emphasis of departments or colleges on research; academic versus non-academic programs; availability of intradepartmental, intrainstitutional and external funding for research; numbers of faculty members; fellowship accreditation status; and numbers of residents, fellows, and graduate and undergraduate students, if any (2, 27-30).

What Is a Research Field?

Writers who state or imply that it is questionable to compare h-indices across certain research fields generally do not provide a clear definition of a research field (1, 2, 18, 19). Without such a definition, the statement or implication seems somewhat vacuous. To say that one should not compare h-indices of two fields if they differ in mean h-indices seems like circular reasoning, but it is hard to see how to provide an a priori definition of a research field relevant to valid comparison of h-indices. Scientific research may be conceived as loosely organized in some complex graph-like structure of fields with a partially hierarchical, tree-like arrangement. Broad fields such as biological sciences, physics, and chemistry are analogous to the boughs, and progressively smaller and smaller subdivisions of these topics into subfields are analogous to progressively smaller branches and twigs. Perhaps the boughs should not be compared with respect to h-indices, but at what point in the branching structure are such comparisons permissible? At the twig level, it seems reasonable to assume that h-indices could validly be compared among researchers focusing on two related diseases that are roughly equal in prevalence, seriousness, and topicality. But identification of any clear-cut dividing point between fields or subfields that should or should not be compared with respect to h-indices seems elusive. It might be more feasible to permit making comparisons of h-indices between research fields and, if statistically reliable differences in means are found, try later to identify at a more granular level the causes or correlates of these differences. In some cases, the differences might actually be attributable to differences between fields in meritorious individual attributes of the researchers involved (think of comparing two fields, one you regard highly and another you disdain). In many cases, with sufficient effort, it may be possible to identify other causes or correlates of the difference between fields in mean h-indices.

CONCLUSION

Departments differed in academic productivity of tenured faculty members as assessed by h-indices. This was not a powerful, monolithic effect, ie, relative departmental standings for full and associate professors were not consistent, and departmental differences for associate professors were nonsignificant in the largest (N=354) data set. The current project was akin to a “proof of concept” within a single college of medicine. Multiple factors probably contributed to the observed departmental differences in h-indices. Future investigations should probe the causes or correlates of departmental differences in larger studies involving multiple colleges of medicine or other multidisciplinary samples.