The relative citation ratio and the h-index among academic ophthalmologists: A retrospective cross-sectional analysis.

BACKGROUND: Limited research has evaluated the relative citation ratio (RCR), a novel measure of research productivity. Accordingly, there remains a minimal understanding of its practical value relative to established metrics such as the h-index. Here, we examined correlations between the mean and weighted RCR scores and the h-index and explore the influence of academic rank, career duration, PhD acquisition, and fellowship training on these metrics.
METHODS: Data regarding the academic rank (e.g. assistant professor, associate professor, professor, or "other"), career duration, degrees, fellowship training, and research yield were collected for 1018 academic ophthalmologists practicing in the southern United States of America. The iCite and Scopus databases were utilized to quantify research yield via calculations of mean and weighted RCR, and h-index, respectively.
RESULTS: Significant correlations were observed between the h-index and the mean (ρ = 0.62, P < 0.001) and weighted RCR (ρ = 0.84, P < 0.001). Advanced academic rank was associated with increased indices values. In a subset of ophthalmologists excluding members of the "other" category, career duration was moderately correlated with h-index (ρ = 0.45, P < 0.001), and weakly correlated with mean (ρ = 0.14, P < 0.001) and weighted (ρ = 0.26, P < 0.001) RCR. PhD and fellowship acquisition were associated with increased research yield.
CONCLUSION: The findings suggest that the RCR is an effective measure of research yield, while resolving deficiencies present in the h-index. Further research remains to characterize the RCR's value relative to other established markers of research productivity.

Introduction

In recent years, substantial research has focused on quantifying academic productivity among clinician-scientists due to its importance towards faculty promotion [[1], [2], [3]]. Efforts have been made to summarize an individual's scientific yield into a singular numerical figure. Multiple measures, elementary and complex, have been developed for this endeavor, but the most universally employed assessment remains the Hirsch index (h-index). The h-index, developed in 2005 to effectively aggregate the number of publications and citations, is defined as follows: “A scientist has index h if h of his or her N papers have at least h citations each and the other (N – h) papers have ≤ h citations each” [4]. Otherwise stated, if an individual has 20 publications, each with at least 20 citations, then the author's h-index is 20. If an author has 12 publications, each with at least 12 citations, then that researcher's h-index is 12.

While the h-index possesses significant utility [5], it suffers from multiple limitations, which have become the subject of criticism directed towards the measure. To begin with, the h-index remains under scrutiny due to its simplicity – it fails to accurately assess the impact of an author's corpus of literature [6]. For example, a scientist who has authored 10 publications with 100 citations each is assigned the same score as a scientist who has 100 publications with 10 citations each. In effect, the h-index may be unfairly benefitting older researchers with large numbers of low-impact publications, thereby disadvantaging younger authors with fewer, but perhaps more impactful publications [7]. Discussions of quality aside, this measure is unable to make inter-field comparisons. As a non-field-normalized metric, the h-index does not account for the possibility of research in larger fields accruing a higher number of citations than those specializing in more niche fields. Consequently, those publishing literature in the field of internal medicine may have a greater h-index than those publishing literature in the field of ophthalmology, irrespective of the impact of their research [8,9].

With a consideration of these weaknesses, the National Institutes of Health (NIH) developed the relative citation ratio. This metric theoretically compensates for the limitations of the h-index by providing an article-level score derived from dividing the total number of citations a publication has received by the average annual number of citations for NIH-funded publications in that paper's field, which is determined by the article's co-citation network [10]. This unique standardization feature allows for more accurate comparisons between multiple different specialties by accounting for the magnitude of research conducted within a given field [11]. Furthermore, the mean RCR, which is the average of the RCR scores for each of an author's publications, potentially eliminates temporal discrepancies between authors at different stages of their careers. In essence, the mean RCR provides an assessment of an author's “impact”. Contrastingly, the weighted RCR, which is the summation of RCR scores for all of an author's publications, enables comparisons when the quantity of publications is the parameter of interest. This would specifically evaluate a scientist's “productivity”. Thus, the RCR addresses the limitations of the h-index by standardizing career duration and inter-field differences, allowing for meaningful comparisons of impact and research productivity among all published scientists.

Although previous investigations have examined the RCR within multiple medical specialties [[12], [13], [14], [15]], no studies have compared it to the h-index. Such an assessment should prove valuable to understanding the RCR's utility. Despite its relative novelty, the RCR's capacity to address key limitations of the h-index, while concurrently providing equivalent evaluation of a scientist's publication history, would set this measure apart from its predecessor and project its value into the future. The present study correlates mean and weighted RCR with the h-index for a cohort of Southern academic ophthalmologists; further differences in trends are assessed for these metrics to determine the practical applicability of the RCR.

Methods

Inclusion and exclusion criteria

In this retrospective cross-sectional analysis, practicing ophthalmologists currently employed at Accreditation Council for Graduate Medical Education (ACGME)-accredited ophthalmology programs within the Southern United States (U.S.), as defined by the U.S. Census Bureau (https://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdf), were identified and analyzed as the sample cohort for this study (n = 1018). Relevant nonmilitary departments were ascertained from the ACGME program listing (https://apps.acgme.org/ads/Public/Programs/Search), but were omitted if faculty were not listed; as such, 40 programs were included. Information regarding names, degrees, academic ranking, fellowship acquisition, and career duration were acquired using departmental websites, the American Association of Ophthalmology (AAO) membership database, and other official sources (e.g., physician directories, private practice information pages, etc.). Individuals were excluded if their training was exclusively as optometrists, PhDs, neurologists, or pathologists.

Bibliometric data collection

RCR scores for PubMed-indexed articles from 1980 to 2021 are accessible through the NIH iCite website (https://icite.od.nih.gov/analysis). As described previously, the RCR is standardized to the NIH benchmark for a particular field, such that an RCR of 1.0 is the median for NIH-funded publications. Author-level metrics, such as mean and weighted RCR, are further calculated as the average and sum, respectively, of all RCRs for a scientist's corpus of publications. Using the iCite database, academic ophthalmologists were individually searched, and non-research articles (e.g., opinion pieces, presentation abstracts, etc.) authored by these individuals were excluded.

Although multiple methods of obtaining a researcher's h-index exist, data were obtained using Scopus. This database has previously been demonstrated to correlate with other sources of citation data including Google Scholar and Web of Science (WOS) [16], suggesting its suitability for the present study. All data were collected in January and February 2021. These data were partially reported in a previous study we conducted [15].

Ethical considerations

These data are publicly available online and therefore Institutional Review Board approval was not required. This work has been reported in line with STROCCS guidelines [17]. This study is registered with the ResearchRegistry and the unique identifying number is: researchregistry7317 [18].

Statistical analysis

Mean RCR, weighted RCR, and h-index were calculated for the included academic ophthalmologists. For the purposes of this study, physicians were categorized into assistant professor, associate professor, professor, and “other.” “Other” was defined as academic faculty with qualifiers in their title such as “adjunct” or “clinical” or with titles such as “instructor” or “lecturer.” Where assessment of correlation was necessary i.e., for between-measure comparisons and comparisons with career duration (calculated by subtracting residency start year from 2021), Spearman's correlation coefficients were computed. The Mann-Whitney U test and Kruskal-Wallis test were utilized to determine significant differences for two and three groups, respectively. Because the data were skewed, median and interquartile range are presented for all analyses. Statistical analysis was performed using GraphPad Prism 8.4.2 (San Diego, CA), with a threshold for significance set at P < 0.05.

Results

To determine the degree of association among the three metrics, between-measure correlations were assessed. Mean RCR (hereafter, RCR) and the h-index correlated moderately (ρ = 0.62, P < 0.001). The correlation between weighted RCR and the h-index was more robust (ρ = 0.84, P < 0.001). These relationships are illustrated in Fig. 1.

Fig. 1

Correlation between h-index and A) mean and B) weighted relative citation ratio (RCR) scores for all southern United States academic ophthalmologists (mean RCR, ρ = 0.62, P < 0.001; weighted RCR, ρ = 0.84, P < 0.001).

Academic rank

Our cohort was comprised of 302 assistant professors (29.7%), 147 associate professors (14.4%), 197 professors (19.4%), and 372 individuals with another title (“other”, 36.5%). As was expected from analyses in prior studies [15], RCR, weighted RCR, and h-index scores correlated positively with increased academic rank.

For individuals in the “other” category, median RCR was 0.45 (IQR 0.00–1.31), median weighted RCR was 0.96 (IQR 0.00–7.46), and the median h-index score was calculated to be 2.00 (IQR 0.00–6.00). Assistant professors had a median RCR of 0.78 (IQR 0.16–1.50), a median weighted RCR of 3.74 (IQR 0.31–13.95), and a median h-index score of 10.00 (IQR 4.00–15.00). For associate professors, the median RCR was 1.23 (IQR 0.65–1.81), the median weighted RCR was 19.86 (IQR 4.11–43.23), and the median h-index was 10.00 (IQR 4.00–15.00). Professors had the greatest values for these measures, with a median RCR of 1.53 (IQR 0.97–2.21), a median weighted RCR of 59.42 (IQR 13.88–175.6), and a median h-index of 25.00 (IQR 12.50–36.00), respectively.

The distribution of these scores is highlighted in Fig. 2. Differences between academic ranks were statistically significant for RCR, weighted RCR, and h-index (P < 0.001).

Fig. 2

Box plot distributions of A) h-index, B) mean relative citation ratio (RCR), and C) weighted RCR scores for all southern United States academic ophthalmologists stratified by academic rank. All significance assessed using the Kruskal-Wallis test. Assist.: Assistant Professors; Assoc.: Associate Professor; Prof.: Professor. ***P < 0.001.

Career duration

Previously published literature has suggested that career duration, as estimated by residency start year, is significantly associated with RCR and weighted RCR exclusively following omission of ophthalmologists in the “other” subgroup [15]. Because the latter subset may conceal between-group differences, further analyses excluded this population (herein, n = 641). Of the three metrics, h-index had the greatest correlation with career duration (ρ = 0.45, P < 0.001), followed by weighted RCR (ρ = 0.26, P < 0.001), and RCR (ρ = 0.14, P = 0.001), as shown in Fig. 3.

Fig. 3

Correlation between career duration and A) h-index, B) mean relative citation ratio (RCR), and C) weighted RCR scores for a subset of southern United States academic ophthalmologists (h-index, ρ = .45, P < 0.001; mean RCR, ρ = .26, P < 0.001; weighted RCR, ρ = .14, P < 0.001). Data is excluded for ophthalmologists in the “other” subgroup (individuals without a title explicitly stated as assistant professor, associate professor, or professor).

Fellowship acquisition

Fellowship acquisition has been associated with increased academic productivity in prior investigations [19]. As such, these trends were explored for the included measures. Physicians with fellowship training (n = 839) had an RCR of 1.18 (IQR 0.63–1.82), weighted RCR of 15.68 (IQR 3.05–51.00), and h-index of 8.00 (IQR 3.00–18.00). This was significantly greater (P < 0.001) relative to physicians without equivalent training, who had a median RCR of 0.29 (IQR 0.00–1.22), weighted RCR of 0.62 (IQR 0.00–2.96), and h-index of 1.50 (IQR 0.00–4.00), as described in Fig. 4.

Fig. 4

Box plot distributions of A) h-index, B) mean relative citation ratio (RCR), and C) weighted RCR scores for a subset of southern United States academic ophthalmologists stratified by PhD acquisition. Data is excluded for ophthalmologists in the “other” subgroup (individuals without a title explicitly stated as assistant professor, associate professor, or professor). All significance assessed using the Mann-Whitney U test. ***P < 0.001.

PhD acquisition

Research has established that attainment of a PhD degree is associated with significantly increased RCR and weighted RCR among academic ophthalmologists [15]. The results of this study mirror these findings. Physicians with a PhD (n = 82) had an RCR of 1.72 (IQR 1.09–2.34), a weighted RCR of 31.82 (IQR 13.94–65.17), and an h-index of 15.00 (IQR 7.50–25.00). Comparatively, physicians without this terminal degree had an RCR of 1.06 (IQR 0.49–1.72), a weighted RCR of 9.44 (IQR 1.56–43.14), and an h-index of 7.00 (IQR 2.00–15.50). These differences were significant (P < 0.001) and are illustrated in Fig. 5.

Fig. 5

Box plot distributions of A) h-index, B) mean relative citation ratio (RCR), and C) weighted RCR scores for a subset of southern United States academic ophthalmologists stratified by fellowship training. Data is excluded for ophthalmologists in the “other” subgroup (individuals without a title explicitly stated as assistant professor, associate professor, or professor). All significance assessed using the Mann-Whitney U test. ***P < 0.001.

Discussion

With the inception of numerous novel measures of research productivity, including the h-index, bibliometric analyses have become increasingly prevalent within the field of medicine [3,[20], [21], [22]]. However, these measures possess critical limitations, some of which the NIH has attempted to resolve through the creation of the RCR. Previous studies have assessed the RCR's utility within medical specialties [[12], [13], [14]], but none have broadly compared the metric against the h-index.

The present study provides an analysis of these measures within a cohort of Southern U.S. academic ophthalmologists, with the intent to examine differences between the h-index and RCR. Here, the h-index correlated moderately with RCR and strongly with weighted RCR. A practical application revealed clear differences among academic ranks for all studied metrics. After exclusion of members of the “other” subgroup, there was a robust association between PhD and fellowship acquisition for both research measures. Interestingly, although h-index moderately corresponded with career duration, weaker relationships for RCR and weighted RCR were observed. Overall, these data are consistent with previously observed relationships in similar subgroups where the h-index was employed [[23], [24], [25], [26]], thereby indicating the usefulness of the RCR and weighted RCR for comparisons of research impact and productivity. Furthermore, as these metrics possessed weaker relationships with career duration than the h-index, they should prove to be more beneficial for comparisons of researchers with varying amounts of experience.

The substantial correlation between the h-index and weighted RCR was unsurprising considering both evaluate lifetime research output. In contrast, there was a less robust association between h-index and RCR, as the latter exclusively assesses research impact without consideration to the number of authored publications. The h-index has been extensively studied in ophthalmology and other specialties and, as such, is a proven measure of research productivity [19,20,22,27]. The findings of this study validate the RCR metrics as they closely correlated to the well-established h-index. However, we argue that the RCR possesses greater utility because it enables standardized comparisons of impact that do not disadvantage younger scientists [7,[11], [12], [13]], a trend that will be examined later in this discussion. Furthermore, considering these metrics are field-normalized, unlike the h-index, comparisons between specialties are not intrinsically biased by size [8,9].

A functional application of the RCR and weighted RCR to comparisons of academic rank confirmed that it was useful for determination of research yield. As observed in a prior investigation we conducted [15], among academic ophthalmologists, professors had the highest scores, whereas faculty in the “other” category had the lowest scores. Other studies have reported similar relationships between the h-index and academic rank [19,22], results that are replicated here. It is important to note that this is a purely correlative analysis and therefore causality cannot be ascertained, nor can we comment on the dependence of academic advancement on metrics of scholarly impact. Investigations comparing the research yields of those who have and have not successfully achieved academic promotion would be more informative for this purpose. Nonetheless, the RCR may be considered as an additional, accurate modality of evaluating the academic achievement of faculty.

One of the recognized limitations of the h-index is its inability to facilitate comparisons for researchers at different stages in their careers [12,13]. As expected, then, a moderate positive correlation between the h-index and career duration was noted. In contrast, career duration was weakly associated with RCR and weighted RCR. Because the mean RCR is the average of all RCR scores received by an author's publications, it effectively eliminates the influence of publication number. The findings reported here confirm that the design of the RCR is appropriate for this intent. Interestingly, although weighted RCR was formulated to compare lifetime productivity, its correlation with career duration was similarly weak. This discrepancy perhaps emphasizes that the h-index unfairly disadvantages younger researchers to a more significant extent than the weighted RCR. Because established researchers are likelier to have a larger quantity of lower impact publications, it is possible that the h-index rewards these articles more appreciably than the weighted RCR. However, this reasoning is speculative and future studies should aim to explore these findings.

An increasingly larger proportion of ophthalmology residents are pursuing fellowships, as determined by data from the National Residency Matching Program and Ophthalmology Fellowship Match Program [28,29]. Such training offers increased exposure to a specific ophthalmological subspecialty, thereby facilitating development of related clinical and surgical skills. Here, it was observed that fellowship acquisition was positively associated with increased research yield, as assessed by RCR, weighted RCR, and h-index. Previous investigations have similarly reported increased h-index among fellowship-trained ophthalmologists [19,26], a finding that was replicated and extended to include RCR metrics in the present study. These results are not surprising as fellowship acquisition may reflect an increased inclination to pursue research or lead to a greater availability of research opportunities, particularly within the field of sub-specialization.

Completion of a PhD degree has been established to correlate with increased RCR and weighted RCR within ophthalmology [15]. but there is limited research associating PhD acquisition and the h-index. A previous investigation by Svider et al. stratified the h-indices of NIH-funded primary investigators in ophthalmology departments by terminal degree, but did not extend this evaluation to non-NIH-funded faculty, thereby limiting the generalizability of findings [23]. Regardless, acquisition of a PhD degree requires extensive multiyear training and focused, formal research. These features are extremely valued by grant reviewers and improve the efficiency and success of time physicians dedicate towards research [30]. As such, it is unsurprising that authors with a PhD are more frequently cited by colleagues within the same field of study. Indeed, studies in other medical specialties have similarly identified a positive correlation between PhD acquisition and research productivity [12,14,24,25,31]. Taken together, these findings suggest that all included measures are accurate predictors of research yield within a specific field.

Strengths and limitations

Strengths of this study include its large sample size and the comprehensiveness of variables assessed (e.g., academic ranking, degree, fellowship acquisition, career duration) that have been previously described in the literature to influence research yield.

While this study provides evidence of the RCR's effectiveness in measuring research productivity, key limitations exist. The geographic restriction of the sample population – including only academic ophthalmologists in the Southern United States – may deflate the RCR metrics compared to a national sample. Indeed, the findings of Thiessen et al. suggest that the mean h-indices are significantly different for ophthalmologists practicing in different geographical regions of the country [19]. However, the results of this study mirror those of previous investigations in which the h-index was utilized [32]. As such, we argue that the trends apparent in this sample should be replicated in a national sample and thus, the potential concern of generalizability is less prominent.

Moreover, despite the advantages of the RCR, several limitations exist with respect to its application. As with most measures of productivity, it fails to differentiate authorship contribution, which may obscure the genuine output of a scientist. Furthermore, although the RCR possesses the ability to facilitate comparisons between multiple specialties, the implications of this standardization remain unknown. As medicine becomes increasingly interdisciplinary, the RCR's capacity to accurately assess research yield may diminish. Exploration of this trend may be the focus of future investigations.

In the present study, two distinct databases were employed for the collection of research metric data, a methodological necessity that likely influenced the subsequent analyses. The iCite database is incapable of distinguishing among multiple authors of the same name, requiring exclusion of authors that generate an inordinately large number of results or inclusion of a middle initial for searches, where possible. Furthermore, the iCite website does not query articles published before 1980, thereby underestimating the productivity of researchers who may have conducted research prior to that year, especially when compared to the Scopus database. Importantly, the latter provides institutional data and creates a bibliometric profile that collates articles for each author, thereby minimizing potential errors of data collection.

Conclusion

This investigation represents the first attempt to provide a theoretical and practical comparison of the RCR and the h-index in a medical specialty. Overall, the findings suggest that the RCR is an effective measure of research yield, overcoming some of the limitations observed in the h-index. Nonetheless, an incomplete understanding of the metric remains. Although the findings of this study provide evidence towards its utility as an evaluation of productivity in the context of academic promotion, further research is required to characterize the RCR relative to other measures.

Ethical approval

Not required.

Sources of funding

None.

Author contribution

Parth A. Patel contributed to the study conception and design. Material preparation, data collection and analysis were performed by Parth A. Patel, Rhea Gopali, Anvith Reddy, and Kajol K. Patel. The first draft of the manuscript was written by Parth A. Patel and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Registration of research studies

Name of the registry: ResearchRegistry

Unique Identifying number or registration ID: researchregistry7317

Hyperlink to your specific registration (must be publicly accessible and will be checked): https://www.researchregistry.com/browse-the-registry#home/registrationdetails/617e975d712665001e89dc3e/

Guarantor

Parth A. Patel, Kajol K. Patel.

Provenance and peer review

Not commissioned, externally peer-reviewed.

Declaration of competing interest

None.