Cross-sectional analysis of bibliometrics and altmetrics: comparing the impact of qualitative and quantitative articles in the British Medical Journal.

OBJECTIVES: In comparison to quantitative research, the impact of qualitative articles in the medical literature has been questioned by the BMJ; to explore this, we compared the impact of quantitative and qualitative articles published in BMJ.
DESIGN: Cross-sectional survey.
SETTING: Articles published in the BMJ between 2007 and 2017. MAIN OUTCOME MEASURES: Bibliometric and altmetric measures of research impact were collected using Web of Science, Google Scholar, Scopus, Plum Analytics and ProQuest Altmetric. Bibliometric measures consisted of citation numbers, field weighted citation impact and citation percentile. Altmetric measures consisted of article usage, captures, mentions, readers, altmetric attention score and score percentile. Scores were compared using the Wilcoxon Rank-sum test.
RESULTS: We screened a total of 7777 articles and identified 42 qualitative articles. Each qualitative article was matched to 3 quantitative articles published during the same year (126 quantitative articles). Citation numbers were not statistically different between the two research types; the median number of citations (google scholar) per quantitative article was 62 (IQR 38-111) versus 58 (IQR 36-85) per qualitative article (p=0.47). Using Plum Analytics, qualitative articles were found to have a significantly higher usage, with a median of 984 (IQR 581-1351) versus 379 (IQR 177-763) for quantitative (p<0.001). The Altmetric Attention Score was higher for quantitative articles at 16 (IQR 7-37) versus qualitative articles at 9 (IQR 5-23, p=0.05), as was the Altmetric Score percentile 93 (IQR 87-96) versus 88 (IQR 76-95; p=0.02).
CONCLUSION: Qualitative and quantitative articles published in the BMJ between 2007 and 2017 both have a high impact. No article type was consistently superior in terms of bibliometric or altmetric measures, suggesting that type of article is not the major driver of impact. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Impact of qualitative and quantitative research evaluated using two valid impact measurement tools.

Evaluation of impact included both the academic and social impact of research.

Use of five different data management sources to collect a variety of impact measures.

Impact was assessed via bibliometrics and altmetrics; however, there are many other ways to evaluate impact.

Comparison of impact was limited to articles published in the BMJ.

Introduction

The number of qualitative studies published in medical journals has increased over the last 20 years1–3; however, the overall number of qualitative articles published to date in these journals remains low.4–6 Qualitative research allows researchers to explore the meaning given by participants to their experiences and understand the context that informs those experiences.7 Strauss and Corbin define qualitative research as ‘any type of research that produces findings not arrived at by statistical procedures or other means of quantification’.8 Because qualitative researchers reject quantification, and therefore generalisability, and rely on subjective accounts, some view qualitative research findings as limited in their usefulness and scientific credibility.9 The Strength of Recommendation Taxonomy classifies qualitative research as the lowest level of evidence, similar to case reports, expert opinion, and anecdotal findings.10 These negative views may explain the low publication rate of qualitative research in medical journals.

In 2016, the BMJ developed a policy of rejecting qualitative articles on the grounds that such studies were ‘low priority’, ‘unlikely to be highly cited’, ‘lacking practical value’, or ‘not of interest to readers’.11 12 In response, several authors wrote letters to the editors reinforcing the importance of qualitative research publications in the BMJ.13–19 Following discussions, the BMJ editors stated that qualitative articles would be considered for publication, although the editors stated that this research methodology was less accessed and cited than other quantitative designs, and hence less helpful for doctors in making decisions.20

The impact of a research article can be measured through a variety of means. Traditionally, bibliometrics measured the academic impact of articles by reporting journal impact factors and citations numbers.6 21 22 Bibliometrics specifically capture research productivity and impact through citation analysis: papers per researcher, citations per paper, journal of publication and place of production. Bibliometrics focus on the academic impact of articles and do not assess the influence of articles beyond the research community.23–25 In 2001, the editor of the BMJ underlined weaknesses with current measures of impact such as bibliometrics, highlighting a lack of assessment of social impact of research.26 Altmetrics were created to complement bibliometrics and provide immediate social impact of scholarly publications. Altmetrics collect measures from a variety of sources including reference management tools, Wikipedia, and popular social media applications such as Facebook, Twitter, news outlets and blogs. Using various algorithms, altmetric platforms transform the gathered data into altmetric scores which denote the social impact of research articles. Altmetrics broaden impact evaluation beyond researchers by including a diverse audiences such as policy-makers, clinicians, researchers, educators and the general public.27–30

To investigate the 2016 BMJ editors’ statements claiming lower impact of qualitative articles, we performed a bibliometric and altmetric analysis to compare the academic and social impact of qualitative and quantitative research articles published in the BMJ over the recent period of eleven years. The BMJ was specifically selected as it publishes the most qualitative research among the high impact medical journals.1 3

Methods

Overview

Bibliometric and altmetric measures included citation numbers, article usage, abstract views, article captures, article mentions online and social media usage. Google Scholar, Web of Science, Scopus, Plum Analytics (https://plumanalytics.com/) and ProQuest Altmetric (www.altmetrics.com) were reviewed to obtain data (tables 1–3).

It was not appropriate to involve patients or the public in the study.

Selection and categorisation of articles

All articles published in the BMJ between 1 January 2007 and 31 December 2017 were screened. One author specifically screened the title and methods section of all articles to determine the article type. The research team was consulted if uncertainty arose during the screening process.

Articles between 2007 and 2017 were included as this marks the beginning of online platforms such as Twitter and provides 2 years for bibliometric measures to be generated. Previous authors have suggested that 2 years post publication are needed to allow for reliable bibliometric indicators.31

Only research articles presenting primary sources of data were included in our analysis. Research articles included quantitative studies (observational studies, economic evaluations and clinical trials) and qualitative articles. Article types such as systematic reviews, meta-analysis, research methods, editorials, rapid responses, personal views and opinions were excluded. Systematic reviews and meta-analysis were excluded as they present secondary data, that is, synthesis of published primary data from qualitative and/quantitative sources.

The research articles identified were classified as qualitative or quantitative studies through title and abstract review (and full text as necessary). Qualitative publications included various methodologies such as ethnography, grounded theory and case study and used methods such as interviews, focus groups, observation and discourse analysis. In addition, qualitative publications included articles using (1) purely qualitative (see above list), and (2) mixed methods methodology with at least one component being qualitative. Qualitative research was defined as ‘any type of research that produces findings not arrived at by statistical procedures’ which involved ‘an interpretive, naturalistic approach to its subject matter’.32 Quantitative publications included observational studies (cross-sectional studies, case-control studies, cohort studies), economic evaluations and clinical trials.

Qualitative research articles were matched 1:3 without replacement to quantitative articles published during the same year using an excel random number generator (Microsoft Excel V.14.6.2; Microsoft). Specifically, each qualitative article published in a given year was given a unique identifier, which was then matched to three unique quantitative articles from that same year. We matched each qualitative article to three quantitative articles in the aim of balancing covariates, decreasing variance, while controlling for year of publication.33–35

Source of data

Bibliometrics

Bibliometric measures collected consisted of citation numbers, field-weighted citation impact and citation percentile. We determined these measures from Web of Science, Google Scholar, Scopus and ProQuest Altmetric for all articles in April 2019. We selected multiple sources for bibliometric measures as each platform collects citation numbers in a slightly different manner (table 1).

Table 1

Bibliometric data reported

	Source	Definition
Citation number	Web of Science, Scopus, Google Scholar, ProQuest Altmetric	Number of citations since publication of article.
Field-weighted citation impact	Scopus	Metric indicating how the number of citations received by a publication compare with the average number of citations received by similar Scopus publications.A field-weighted citation impact greater than 1.00 indicates that the publication has been cited more than is expected.
Citation percentile	Scopus	Indicates the position of the article based on citation numbers as compared with the citation distribution within its field.

Altmetrics

Altmetric measures were obtained from Plum Analytics (table 2, https://plumanalytics.com/) and ProQuest Altmetric (table 3, https://www.altmetric.com) for all articles in April 2019 (tables 2 and 3).

Table 2

Plum Analytics measures definitions36

Categories	Explanation	Example
Usage	Signal that individuals are reading the articles and using the research.	Abstract Views	Number of times the abstract has been viewed
		Full Text Views	Number of times the full text has been viewed
		Clicks	Number of clicks of a URL
		Downloads	Number of times the artefact has been downloaded
		Holdings	Number of libraries that hold the artefact
Captures	Measure indicating that the individual wants to return to the work.Captures can be linked to future citations.	Bookmarks	Number of times an artefact has been bookmarked
		Favourites	Number of times the artefact has been marked as a favourite
		Readers	Number of people who have added the artefact to their library
		Exports/Saves	Number of times an artefact citation has been exported to bibliographic toolNumber of times an artefact’s citation has been saved/emailed/printed
Mentions	Mentions indicate that people are engaging with the research.	Blog Mentions	Number of blog posts written about the artefact
		Comments	Number of comments made about an artefact
		News Mentions	Number of news articles written about the artefact
Social media	Social Media can help measure the interest for an article.	Likes	Number of times an artefact has been liked
		Shares, Likes and Comments	Number of times a link was shared, liked or commented on
		Tweets	Number of tweets and retweets that mention the artefact

Table 3

ProQuest Altmetric measures definition37

Categories	Explanation37
Altmetric Attention Score	The Altmetric Attention Score provides a weighted count of the quantity of attention that a paper has received online and is derived from an automated algorithm.49 The score is weighted based on three main factors: (1) volume (how many times the article is mentioned), (2) sources (where the mentions come from), and (3) authors (authors of the mentions).50 This score helps to identify the level of online activity surrounding an article from a variety of sources (eg, News, blogs, Twitter, Facebook, Wikipedia).
Altmetric Score Percentile	The altmetric score is contextualised into altmetric score percentile to facilitate comparison.
Mentions	Mentioned by reports sources such as news outlets, blogs, Facebook, tweets, and so on.
Readers	Readers from various sources including Mendeley (free reference manager). Mendeley readership is the number of unique Mendeley users who have added copies of an article to their personal library.51

We selected Plum Analytics as it provides a detailed breakdown of five categories (usage, captures, mentions, social media and citations) allowing for comprehensive assessment of impact. We selected ProQuest Altmetric as this platform calculates an overall altmetric score, a useful tool to compare individual articles.

Plum Analytics generates PlumX Metrics which collects article level metrics: (1) usage, (2) captures, (3) mentions and (4) social media.36 ProQuest Altmetric measures included the (1) Altmetric attention score, (2) Altmetric score percentile, (3) mentions and (4) Mendeley (free reference manager) readers.37

Statistical analysis

Bibliometric and altmetric measures were assessed using medians with IQR as the data were not normally distributed (Kolmogorov-Smirnov test, p<0.05). The Wilcoxon Rank-sum test was performed to compare individual altmetric and bibliometric measures for qualitative versus quantitative articles. P value<0.05 was considered statistically significant. If percentile scores were available (ie, Scopus citation percentile and Altmetric score percentile), measures above 75th were considered high. A sensitivity analysis was performed to evaluate the effect of excluding mixed methods articles and considering only purely qualitative research article types. All statistical analyses were performed with the SAS Studio University Edition, V.3.8 (SAS Institute).

Results

A total of 7777 articles were screened and we identified 42 qualitative articles published in the BMJ between 2007 and 2017 (figure 1). These were matched to 126 quantitative articles. Notably, of the qualitative articles 41 of 42 were published from 2007 to 2011, while only 1 article was published after this time period, in 2013. Most qualitative articles were purely qualitative (83%), rather than mixed method. Quantitative articles included randomised trials (47%), observational (48%) and decision analysis/economic evaluation (5%) studies.

Figure 1

Flow diagram.

Based on bibliometric measures, citation numbers were not statistically different between the two research types (table 4). Using Web of Science data, we found a median of 62 citations (IQR 38–111) per quantitative article as compared with 58 (IQR 36–85) for qualitative articles (p=0.47). This was similar for Scopus, Google Scholar and ProQuest Altmetrics. For the field weighted citation impact, no statistical differences were noted for the two research types; quantitative articles had a median field weighted impact at 7.06 (IQR 4.58–12) as compared with 5.96 (IQR 4.33–10.26) for qualitative (p=0.16). Both qualitative and quantitative articles were frequently cited with a citation percentile above 91, potentially indicating high impact.

Table 4

Summary statistics for bibliometrics

	Quantitativen=126	Qualitativen=42	P value
Citation number, median (IQR)
Web of Science	62 (38–111)	58 (36–85)	0.47
Scopus	78 (45–131)	74 (48–105)	0.52
Google Scholar	121.5 (68–203)	134 (85–185)	0.98
ProQuest Altmetric	88 (52–151)	73 (44–113)	0.16
Scopus Field Weighted Citation Impact, median (IQR)	7.06 (4.58–12)	5.96 (4.33–10.26)	0.16
Scopus Citation Percentile, median (IQR)	93 (84–97)	91 (86–95)	0.21

Citation numbers are different between Scopus, Web of Science, Google Scholar and ProQuest Altmetric as each platform collects citation information differently. Web of Science covers the oldest citations, from 1900 to present.52 Scopus covers citations starting in 1966, but it indexes a larger number of journals including a greater number of international and open access journals as compared with Web of Science. Google Scholar and ProQuest Altmetric do not reveal information on their method of citation collection.44 ProQuest Altmetric uses a novel research insight platform called Dimensions which captures references beyond classic publication-based citations.53 The platform developers state that Dimensions is not comparable to Web of Science, Google Scholar or Scopus, though no further information is provided.

For altmetrics measures, using Plum Analytics, qualitative research was found to have a significantly higher usage, with a median of 984 (IQR 581–1351) clicks, downloads or views as compared with 379 (IQR 177–763) for quantitative research (p<0.001, table 5). Likewise, qualitative articles had significantly higher captures with a median of 191 bookmarks, favourites, or readers (IQR 98–292) as compared with 88.5 (IQR 35.5–191) for quantitative research (p<0.001, table 5). The two research types had similar mentions and social media impact on Scopus (p=0.13 and p=0.43, respectively).

Table 5

Summary statistics for Altmetrics

	Quantitative n=126	Qualitative n=42	P value
ProQuest Altmetric
Altmetric Attention Score, median (IQR)	16 (7–37)	9 (5–23)	0.054
Altmetric Score Percentile, median (IQR)	93 (87–97)	88 (76–95)	0.022
Mentions, median (IQR)
News Outlet	2 (1–5.5)	1 (1–3)	0.54
Blog	1 (1–2)	1 (1–2)	0.41
Policy Source	1 (1–2)	1 (1–1)	0.09
Tweeters	6.5 (2–22)	6 (3–24)	0.72
Twitter Followers	19 533 (2888.5–86 571.5)	29 595 (2,809 – 123,449)	0.56
Facebook	2 (1–4)	1.5 (1–2)	0.39
Wikipedia	1 (1–2)	1 (1–1)	0.23
Google+user	1 (1–5)	1 (1–1)	0.24
Readers, median (IQR)	90 (56–149)	110 (80–131)	0.28
Plum Analytics
Usage, median (IQR)	379 (177–763)	984 (581–1351)	<0.001
Abstract Views	326 (146–613)	948 (500–1231)	<0.001
Link-Outs	8.5 (3–25)	23 (11–35)	<0.001
Full Text Views	3 (1–23)	1 (1–2)	0.08
Clicks	22 (5–91)	35 (5–40)	0.58
Captures, median (IQR)	88.5 (35.5–191)	191 (98–292)	<0.001
Readers	76 (9–144)	114 (64–142)	0.043
Exports and Saves	29.5 (12–59.5)	70 (38–130)	<0.001
Mentions, median (IQR)	1 (1–4)	1 (1–1.5)	0.13
Social Media, median (IQR)	10.5 (2–33.5)	5 (2–24)	0.43

Bold values are statistically significant values.

Using ProQuest Altmetric, the Altmetric Attention Score was higher for quantitative research at 16 (IQR 7–37) as compared with qualitative research at 95–23 (p=0.05, table 5). This translated to a significantly higher Altmetric Score percentile with quantitative research scoring a median Altmetric Score percentile of 93 (IQR 87–96) compared with 88 (IQR 76–95) for qualitative (p=0.02). Of note, both percentiles were considered high (above 75th percentile). Despite differences in the overall Altmetric Attention Score, the individual factors included in this weighted score were no different between the two types of research. In fact, new outlets, blogs, policy sources, tweets, Facebook posts, Wikipedia mentions were not statistically different between both types of research (table 5). Both qualitative and quantitative articles were highly used online, with frequent reports on social media outlets.

The exclusion of mixed methods articles with purely qualitative research articles did not alter our findings (data not shown).

Discussion

This bibliometric and altmetric analysis compared the academic and social impact of qualitative and quantitative article published in the BMJ from 2007 to 2017 and did not find a dominant article type using these metrics. Bibliometrics as evaluated by citation analysis were similar between both article types. Altmetric measures differed depending on the method of assessment, with usage and captures significantly higher in qualitative articles and Altmetric Score Percentile higher in quantitative articles. With regards to ProQuest Almetrics, both research types had high percentile scores (>88th Altmetric score percentile) indicating high impact for both article types. Interestingly, although our analysis spanned 11 years, no qualitative articles were published in the BMJ after 2013.

Our analysis highlights that research articles published in the BMJ had high impact regardless of whether the article was quantitative or qualitative, likely due to selective publication process of this medical journal. Schroter performed an audit of publications in the BMJ from 2003 to 2012 and reported that while over 3000 research articles were submitted each year to the BMJ, only 2.1%–7.7% were accepted for publication.38 Our findings indicate that articles published in the BMJ score in the highest percentiles with regards to citations and altmetric scores, reinforcing the notion that research articles published in this journal are widely used regardless of their methodology. We found no indication that qualitative articles published in BMJ had had less impact than quantitative articles.

In contrast to the 2016 editorial, our analysis did not find that qualitative articles published in the BMJ were ‘unlikely to be highly cited’, ‘lacking practical value’, and ‘not of interest to readers’.11 12 First, using a variety of citation trackers, we found high citations numbers for both qualitative and quantitative articles, with the median score for both times of articles above the 90th citation percentile. To evaluate reader interest and practical value, we used altmetrics as measures of the social impact of articles. Qualitative and quantitative articles had similar mentions online on news outlets, blogs, twitter, Facebook and Wikipedia. On twitter, both article types influenced a large audience with more than 19 000 individuals reached by the articles. These findings suggest high social impact of both qualitative and quantitative articles published in the BMJ.

Previous authors have compared the impact of qualitative and quantitative research using citation analysis. In 2013, Mori and Nakayama reviewed the academic impact of qualitative studies in healthcare.6 Using citation numbers, the authors concluded that qualitative studies were cited less frequently as compared with systematic reviews and randomised trials, but were similarly cited compared with the median of the impact factor of the journal. Importantly, the authors did not compare qualitative studies to quantitative observational studies. In 2011, Weiner et al compared citation numbers for qualitative articles published in nine major health services and management journals to a random sample of quantitative articles and found similar citation frequencies between the two, suggesting that qualitative research articles contribute comparably to the field’s scientific knowledge base.4 Our analysis reinforces the findings that qualitative and quantitative articles have similar academic impact, and brings forward additional information on the social impact of qualitative research in order to provide a broader analysis of article impact.

In a variety of fields, individuals have evaluated impact using bibliometrics and altmetrics to determine research interests of the members of the public and of the scientific community.39–43 In 2018, Powell et al compared the altmetric score and bibliometric data of the 100 most cited articles in surgery and found, similar to our study, that bibliometric and altmetric analyses provide important but different perspectives on article impact.43 Altmetrics offered the unique advantage of timely assessment of articles generating discussions online and positively correlated with citation numbers.43 In 2018, Banshal et al compared the top 100 Altmetric papers and the top 100 cited papers and found minimal overlap between these.42 The authors suggested that it was unlikely that one assessment tool predicts the other. The authors concluded that altmetric score provided additional rather than duplicate information to citation scores. In 2019, Azer and Azer found no correlation between the number of citations and the altmetric scores for top-cited articles in medical professionalism; though, for articles published after 2007, the authors found a significant correlation between number of citations and altmetric scores.44 In contrast, in 2020, Luc et al found that tweeting significantly increased citations numbers over time.45 The relationship between altmetric and bibliometric measures thus remains controversial.46 47 Because of the debatable correlation between altmetrics and article citations, some authors had cautioned the use of altmetrics as a measure of impact.27 48 Rather, we suggest that altmetrics should be viewed as complementary to existing tools to measure impact.

Our study strengths include the use of two valid impact measurement tools to evaluate both the academic and social impact of research articles published by the BMJ. In addition, we use five different data management sources (Web of Science, Google Scholar, Scopus, Plum Analytics and ProQuest Altmetric) to conclude that both qualitative and quantitative articles published in this journal have a high impact. Our study has some limitations. We used bibliometrics and altmetrics to assess impact; however, many of the measures generated by the websites do not have reference values to help adjudicate what is a high or low score (other than percentile scores). Furthermore, the comparison of impact between qualitative and quantitative research focused only on articles published in the BMJ, and therefore these findings may not be generalisable.

In conclusion, using bibliometric and altmetric measures, our study has found both qualitative and quantitative research published in the BMJ have a similar impact on the public as well as the academic community. Based on our findings, there is no evidence to support claims that qualitative studies published in the BMJ are of lower impact than quantitative articles.