The Clinical Research Bias Index (CRBI): A novel journal ranking method applied to child health respiratory studies.

Background and Aims: Journal impact factor has historically been taken as a proxy for quality. However, this is open to significant manipulation and bias. There is currently not widely adopted, robust journal and paper ranking metric which is focused solely on risk of bias.
Methods: Risk of bias data was extracted from all Cochrane database systematic reviews in Child Health, Lungs, and Airways for the years 2017-2019. A novel paper quality score, the Clinical Research Bias Index (CRBI), was applied. Individual paper data were pooled for each journal. A comparison was made to journal impact factors, individual paper citations, reads, and altmetric scores.
Results: 927 papers were analyzed for risk of bias. 119 (12·8%) scored a CRBI of 100%, with a mean score of 70%. A journal's overall CRBI risk of bias score was poorly correlated with impact factor (r 0.25). Citations (r 0.02), and reads (r 0.01) of individual papers showed very little association with the paper's risk of bias. Likewise, reads were not correlated with citations (r 0.03). H-index and Altmetric scores were similarly poorly correlated with CRBI.
Conclusion: The novel research quality tool CRBI demonstrates the poor correlation between journal impact factor, citations, and risk of bias. Journal and paper ranking metrics should ensure that they are fit for purpose, and enable the dissemination of high-quality research for the benefit of patients. We propose the CRBI as a potential solution which is resistant to manipulation and will reward the creation and publication of bias-free research.

INTRODUCTION

Clinicians and scientists seek to improve their knowledge and practice through reading high‐quality research. Due to a vast number of journals in circulation, currently around 30,000, it is unsurprising that this task becomes a daunting one. When coupled with ever‐improving access to medical literature, it is vital that clinicians have tools to discriminate between low and high‐quality evidence. There are various systems of ranking journals, including the most pragmatic of all: the reputation of the journal. Journal ranking systems however have two common significant shortfalls: they rank with a metric that does not equate to quality, and the ranking systems are vulnerable to manipulation.

Most journal ranking systems, such as the Impact Factor (IF), rely upon citation metrics. It would appear reasonable to presume that future authors of publications will preferentially reference the highest quality work. However, IF essentially rewards popularity of a journal, article, or author, and is closely followed by readers, authors and editors. Unfortunately, IF can be artificially elevated through several techniques including self‐citation, the use of citation clubs and citation amplification, and preferentially publishing articles which are certain to by highly referenced (for instance a practice guideline), to increase IF. Noncitation based tools which employ social networks, such as the Altmetric Attention Score are also vulnerable to manipulation, but in other ways, such as self promotion.

Evaluating journals based on a rigorous scrutiny of research quality would be intrinsically more useful than citation metrics. Randomized controlled trials (RCTs) when conducted well produce the highest quality evidence. Because the design of randomized trials has been carefully studied, methods for evaluating trial design have been developed. These methods are frequently employed when evaluating trials in systematic reviews, and include the Jadad Score, and the risk of bias table in Cochrane systematic reviews. Unfortunately, evaluating these scores for the totality of the clinical trial literature in a systematic manner has been regarded as unfeasible.

The Cochrane Collaboration is the world's largest repository of high‐quality, peer‐reviewed, systematic reviews and meta‐analyses of the clinical literature. A key step of any Cochrane review is the conduct of an assessment of risk of bias, which is inherently linked to trial quality. We developed an approach to using this data to develop a new journal ranking method, the Clinical Research Bias Index (CRBI). As a proof of concept, we applied this to the field of respiratory research in child health.

METHODS

The risk of bias tables from Cochrane systematic reviews was the primary data source used. Each Cochrane systematic review uses a traffic light system of red (high risk of bias), yellow (indeterminate risk of bias) and green (low risk of bias) to summarize the reviewers assessment of biases defined in that review within a risk of bias table. The reviews can specify their own category of biases, often within an overall approach recommended by the Cochrane group within which the review sits, and which are ultimately usually based on the Cochrane Handbook approach.

We converted the Cochrane traffic light system to a numerical scale, with a score allocated of one, two, and three for red, yellow, and green respectively. If a study did not account for a particular bias, they scored zero for that bias. Each study within a review was scored, and those scores were then applied to an average score for the journal publishing the original research.

A standardization for each topic was developed to ensure that unfair comparisons between different study categories were eliminated. For instance, it is impossible to effectively blind many surgical studies, and it would be unjust to penalize all studies in such fields. Within each systematic review, the highest score achieved by any of the studies in that bias category was designated as the highest possible score, and used as the reference value. See Table 1 for example risk of bias table scoring. Any study achieving the maximum possible score for that topic was given 100% in that field. Lower scores were reduced to 50% and 0% (if all three scores were in use), and to 0% if only two scores were in use.

Table 1

An example risk of bias table, with the scoring system in use

Note that for the allocation concealment category, in this systematic review the “best” bias score was unclear risk of bias (yellow) for all the trials in the review. Therefore yellow scored 100%. If a future trial was incorporated into the review with a better bias score of low risk of bias, this would result in the score for the current yellow studies to decrease to 50%. This approach means that the Clinical Research Bias Index (CRBI) score is dynamic and promotes investigators to design trials with ever‐decreasing bias.

As a proof of concept, data were extracted from all Cochrane systematic reviews in the Child Health, Lungs, and Airways category of the database, for the years 2017–2019. Data were extracted from each systematic review's risk of bias table. A CRBI assessment was applied to each of the studies, with results being recorded in Microsoft Excel (Microsoft Inc). The journal in which each paper was recorded, as well as that journal's ranking metrics from the annual Web of Science Journal Citation Report. The number of reads (to assess the “popularity” of a paper) and citations each paper had amassed on Researchgate.com was recorded.

Data were collected by one author (MV) and a random sample of 10% of records were independently checked by a second author (APP). To assess the reproducibility of the Cochrane Risk of Bias assessment by different Cochrane reviewers, where a study was in two different Cochrane reviews, we compared the Risk of Bias Assessment between reviews.

We undertook quantile regression to explore the relationship between the CRBI and impact factor, Researchgate citations and reads and using a tau of 0.5 (i.e., the regression assesses the conditional median rather than the conditional mean, and is more robust to outliers). Data analysis was undertaken with R (version 3.6.2) and the quantreg package version 5.5. The complete data set is available from https://github.com/andrewprayle/NORQUI_respiratory_dataset and can be accessed in Supporting Information file VPD supplement.

RESULTS

We found 62 Cochrane reviews that met the inclusion criteria; these referenced 927 papers, and we were able to calculate a CRBI score for each of these. Within these Cochrane reviews, we identified 40 categories of sources of bias that had been assessed by the Cochrane Reviewers, (Data S1). The most common categories were Random Sequence Generation, Allocation Concealment, Blinding of Participants and Personnel, Blinding of Outcome Assessment, Incomplete Outcome Data, Selective Reporting Bias, and Outcome Bias.

Of these 927 papers, 119 scored a CRBI of 100% and none of the papers scored a CRBI of 0% (for consort diagram see Data S1). The mean CRBI was 70% and the median was 70.9%. CRBI was not normally distributed (Figure 1).

Figure 1

A Frequency distribution of CRBI for pediatric respiratory medicine. Mean CRBI score was 70%, standard deviation was 20.8%. CRBI., Clinical Research Bias Index.

Journals with the highest impact factors were not the journals with the least amount of bias as measured by CRBI (Figure 2). There was medium to moderate correlation between IF and CRBI (r of 0.25). There were two outliers that heavily influenced this correlation (The Lancet and The New England Journal of Medicine; both with high impact factor and high CRBI). When these were removed from analysis, the r was 0.42. The highest ranked journal in this field by CRBI was the Lancet Respiratory Medicine (94.3%).

Figure 2

Journal Impact Factor against the CRBI percentage for the top 20 CRBI scored journals included within this project (each point represents the mean CRBI for the journal). We show data for top 20 journals as in our search of Cochrane studies, each of these journals had published at least five journal articles which could be scored. The fitted line is a quantile regression of the conditional median. CRBI, Clinical Research Bias Index.

We compared ResearchGate citations and CRBI, and found little correlation (0.14; Figure 3A). Similarly, ResearchGate reads were not well correlated with CRBI (Rr of 0.01; Figure 3B), and in turn number of reads on Researchgate was not well correlated with citations (Rr of 0.14; Figure 3C).

Figure 3

CRBI scores at the study level (each point represents a single research study). (A) ResearchGate Citations (as a proxy of the article's influence in the literature) compared to the CRBI. (B) ResearchGate Reads (as a proxy for how well read an article is) against the CRBI. (C) Logarithmic graph displaying ResearchGate Reads versu Citations for each of the 927 studies. Lines represent a quantile regression of the conditional median.

To understand if the CRBI was reproducible, we compared risk of assessments of studies that appeared in more than one Cochrane Review (i.e., those where a study had been assessed twice in two different Cochrane reviews). There were 31 studies that were in more than one Cochrane review, and they had complete agreement (Kappa score 1.0).

DISCUSSION

We present the CRBI assessment tool; a tool to rank journals, studies, and researchers by way of repurposing the risk of bias assessment (undertaken independently by the Cochrane Collaboration as part of their work undertaking systematic reviews) as an impartial assessment of the quality of clinical research.

Using this tool, we found that there is medium to moderate correlation between the quality of evidence (high‐quality studies with a low risk of bias) contained in a journal, and the current journal ranking tools utilized, such as Impact Factor. We suggest that the most popular journals (based upon commonly employed citation metric tools) do not always contain the highest quality evidence. Likewise, the citation numbers of individual papers did not correlate with the CRBI assessment of quality. Finally, using data from ResearchGate (a social media site for academics) we found that readers did not preferentially read the highest quality papers.

The highest scoring journals on CRBI are in the range of 80%–95%. Despite containing excellent research, these journals are often less visible than some of those with a low CRBI rating. In our results, we found two journals separated by less than 1% on CRBI assessment, whose IFs were 39 points apart. It is possible that due to this, clinicians are missing out on the opportunity to read high‐quality papers in lower Impact Factor rated journals, limiting their reading to high Impact Factor journals. Ultimately this will have a detrimental influence on patient care. CRBI aims to drive the field of medical literature towards increased quality, justly rewarding high‐quality medical research. In addition to being useful for journals to independently assess their research quality and compare themselves to the field, the CRBI can also be calculated for individual researchers or research teams, allowing them to, for example) demonstrate to funders that their work is of sound methodology with a low risk of bias.

Other workers in the field have developed scores and methodologies to allow the evaluation of clinical and comparing this to journal metrics such as the automated method reported by Vinkers, which also used data from the Cochrane Database of Systematic Review. However, this study did not specifically aim to develop an index that can be used to rank journals, but rather studied trends in overall trial methodological quality over time.

A core property of any tool to measure journal quality should be that it is resistant to manipulation. Our use of the Cochrane Risk of Bias tool means that a third party independently assesses the risk of bias for each study. This assessment is independent of the authors or editors of a journal, and is peer‐reviewed as part of the Cochrane process. We found excellent internal consistency of bias assessment in our analysis. In our data, there were 31 primary studies assessed in more than one Cochrane review each. On each occasion the studies were scored identically, by different reviewers, reassuring us that the risk of bias assessment undertaken by Cochrane Review authors is reproducible.

The only method of achieving a high score on CRBI is via publication of high‐quality, unbiased research, as assessed by the Cochrane group, compared to the current available literature. The current methods of citation manipulation are ineffective in altering the CRBI. CRBI is also resistant to self‐promotion by social media, or by paying for open access.

CRBI allows for the fact that in some situations it is impossible to avoid all bias. For example, in a hypothetical trial of early versus late tracheostomy on intensive care, blinding would be effectively impossible. For this reason, the CRBI score for a paper in a review is normalized against the highest score for all papers in that review. This approach allows for journals to publish studies in areas that are challenging to, for example, blind participants, without concern that their journal CRBI will decrease.

This approach also motivates researchers to produce the most “bias‐free” research possible. Consider a series of surgical trials, all of which are not blinded in a Cochrane review. These will all be scored the maximum possible. However, the first study published which is blinded will attain the first “green” assessment in a bias category, will reduce the scores of all previous historical studies which have so far only achieved a “yellow” categorization. Journals would be motivated to publish such ground‐breaking studies, to increase their overall CRBI score.

CRBI has limitations. Firstly, it is limited in its scope to evaluating the risk of bias of a clinical trial. Many other factors, some of which are encapsulated in other scores, are important for a detailed overall assessment of clinical research quality. Secondly, it is reliant on the Cochrane database to create an independent assessment of bias. There is a lag time between publication of a paper, however, this is similar to the delays inherent in being cited by another researcher. Thirdly, a study area that has hitherto been poorly researched, having very few studies available for a systematic review, may score disproportionately highly as the comparison to the “best possible” study will be limited. Indeed, in a Cochrane review where there is only one applicable study, that study will score 100%. This may however motivate researchers and journals to increase their efforts in this field. As not every research field is studied by Cochrane, there is a risk that a body of work (perhaps in more “niche” areas) will not contribute to a journal's CRBI score. A final limitation pertains to the use of the site ResearchGate, from which paper reads and citations have been taken. There are a number of competing citation counting services, and each is dependent on the ability of their systems to find citations. For consistency we used the same citation counting tool throughout.

CONCLUSION

We propose a research ranking method entitled the CRBI for journal, researcher, and paper ranking, which has quality of research as its sole metric. It is resistant to manipulation, and will reward the creation and publication of bias‐free research. CRBI quality assessment is very weakly correlated with journal impact factor, individual paper citations, or individual paper reads. This study used data from a manual extraction of the relevant primary data from the Cochrane Database of Systematic Reviews; the next step in this study is to automate this study for the entire Cochrane Library.

AUTHOR CONTRIBUTIONS

Manishaa Vairavan: Data curation; formal analysis; investigation; writing—original draft; writing—review & editing. Andrew Prayle: Data curation; formal analysis; investigation; software; supervision; writing—review & editing. Patrick Davies: Conceptualization; formal analysis; project administration; supervision; writing—review & editing.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

TRANSPARENCY STATEMENT

The corresponding author confirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Supplementary information.

Click here for additional data file.

Supplementary information.

Click here for additional data file.