Top-cited articles in medical professionalism: a bibliometric analysis versus altmetric scores.

INTRODUCTION: Citation counts of articles have been used to measure scientific outcomes and assess suitability for grant applications. However, citation counts are not without limitations. With the rise of social media, altmetric scores may provide an alternative assessment tool.
OBJECTIVES: The aims of the study were to assess the characteristics of highly cited articles in medical professionalism and their altmetric scores.
METHODS: The Web of Science was searched for top-cited articles in medical professionalism, and the characteristics of each article were identified. The altmetric database was searched to identify report for each identified article. A model to assess the relationship between the number of citations and each of the key characteristics as well as altmetric scores was developed.
RESULTS: No correlations were found between the number of citations and number of years since publication (p=0.192), number of institutes (p=0.081), number of authors (p=0.270), females in authorship (p=0.150) or number of grants (p=0.384). The altmetric scores varied from 0 to 155, total=806, median=5.0, (IQR=20). Twitter (54%) and Mendeley (62%) were the most popular altmetric resources. No correlation was found between the number of citations and the altmetric scores (p=0.661). However, a correlation was found for articles published in 2007 and after (n=17, p=0.023). To further assess these variables, a model was developed using multivariate analysis; did not show significant differences across subgroups. The topics covered were learning and teaching professionalism, curriculum issues, professional and unprofessional behaviour.
CONCLUSIONS: Altmetric scores of articles were significantly correlated with citations counts for articles published in 2007 and after. Highly cited articles were produced mainly by the USA, Canada and the UK. The study reflects the emerging role of social media in research dissemination. Future studies should investigate the specific features of highly cited articles and factors reinforcing distribution of research data among scholars and non-scholars. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Four searches were conducted in the web of Science database and the altmetric tracks.

The analysis explored a range of bibliometric parameters.

The study was limited to top-cited articles in the English language.

Introduction

Citation counts have been used by universities and funding bodies to measure scientific outcomes, make decisions about professional promotion and assess suitability for grant applications.1 2 In this context, it was claimed that the higher number of citations received, the higher quality of work and the more likely that other researchers cite the work.3 While these claims may not necessarily be true, there is a substantial body of evidence that the number of citations correlates with other research achievements including research awards, honours, nomination for Nobel laureateship,3 4 prestigious research positions5 and academic ranking.6 7 However, there are factors other than scientific quality that may affect the decision to cite.8 For example, there is evidence that early interest in a research publication reflected by online access within a week of publication predicts citations up to 15 years later.9 Also, scientific citations favour positive results and authors tend to cite primarily works by authors with whom they know and personally acquainted.10 11

With these limitations in mind, there is a continuous search for alternatives or metres that can complement the citation counts. Currently, there is a rising interest in the altmetric scores. Contrary to traditional citation-based analysis, the altmetrics reflect the widespread attention to published scientific articles and the rise of social media for dissemination and discussion of scientific information. Therefore, it is possible to quantify discussion of an article on blogs, news media or other social media platforms.12

Considering these two tools, it was decided to assess highly cited articles on medical professionalism.13 14 The top-cited articles were selected because an earlier study revealed a number of attributes of articles on medical professionalism.14 The use of the altmetric scores in this study in particular is thought to be useful since articles on professionalism are usually shared on social media.

Therefore, the present study aims at the following: first, identify the most-cited articles in medical professionalism and evaluate their characteristics, and study any correlations between the number of citations and each of their biliometric characteristics. Second, assess the impact of such articles on social media by calculating the altmetric scores and conducting an exploratory analysis examining the altmetric findings compared with citation analysis. The conduction of multivariate analysis model may provide additional insight into such evaluation. The findings from this study may enable researchers to identify common features of articles behind the progress of medical professionalism and key topics discussed over the last two to three decades. The study may provide more insight into any relationships between citation analysis and the altmetric scores. The identified list of publications may be useful to medical educators and those teaching medical professionalism or doing masters or research in these areas.

Methods

Study design

To achieve the objectives of this study, it was decided to search the Web of Science database of Clarivate Analytics for highly cited articles and track the citation records of publications identified. Although Scopus and Google Scholar databases also provide citation tracking, it was decided to limit the search to the Web of Science. This is because the Web of Science is regularly updated and its 2016-Journal Citation Reports reported over 59 million citations in its Science Edition and 7 million from its Social Science Edition. In the area of medical education, medical ethics and bioethics, general medicine and surgery, the Web of Science has included 16, 49, 457 and 180 peer-reviewed journals, respectively. Google Scholar was not included in the search because it is difficult to search, and it is not possible to identify the number of citations for each year across the last two to three decades, and the citations in Google Scholar usually include textbooks, monographs, conference proceedings, as well as non-peer-reviewed work. The Scopus database was not included in our search because its records only go back to 1966.

To achieve the first aim, we planned to identify the highly cited articles in medical professionalism and their characteristics using three mechanisms: (1) searching the Web of Science using keywords, (2) searching medical education, ethics, general medicine and surgery journals in the Web of Science, (3) searching the webpage of journals and (4) searching for related resources mentioned in the list of references of articles identified. For the second aim, the altmetric bookmarklet application was used to obtain the altmetric scores and construct exploratory analysis examining the role of social media and the different resources contributing to altmetrics. At the end, we compared these findings with those obtained from the citation analysis.15–17 A description of the steps used in the search is discussed below.

Searching the web of science database using keywords

Searching the Web of Science database was carried out in the 5 April 2017 by two researchers (SAA is a professor of medical education with a 20-year experience in research in the field of medical education and professionalism, and SA a surgical registrar and researcher). The search words used were the following: ‘Medical professionalism’, ‘Patient safety’, ‘Professional behavior’, ‘Unprofessional behavior’, ‘Role modeling’, ‘Accountability’, ‘Faculty training in professionalism’, ‘Altruism’, ‘Physician code’, ‘Physician charter’, ‘Medical ethics’, ‘Integrity’, ‘Consent’, ‘Defining medical professionalism’, ‘Empathy’, ‘Compassionate doctor’, ‘Professional conduct’, ‘Collaborative doctor’, ‘Self-assessment’, ‘Professional development’, ‘Resilient doctor’, ‘Social justice’, ‘Patient autonomy’, ‘Patient Welfare’, ‘Professional responsibility’, ‘Managing conflict’, Patient confidentiality’, ‘Quality of care’, ‘Social contract’, ‘Team work and professionalism’, ‘Personal development’, ‘Public professionalism’, ‘Interpersonal professionalism’ and ‘Intrapersonal professionalism’. These keywords were identified from the terminology and themes used in defining medical professionalism in six resources including.18–23 We also looked at conference proceedings in the field and websites of organisations and agencies responsible for accreditation of medical education worldwide including: The World Federation for Medical Education, the UK’s General Medical Council, the Association of American Medical Colleges, the Australian Medical Council, the Liaison Committee on Medical Education, and the Quality Assurance of Basic Medical Education, and documents such as: Tomorrow’s doctors, 2003; The New Doctor, 2004; and General Medical Practice, 2001.

For each search word, the results were arranged using a link on the Web of Science database system ‘sort-by’—‘Time Cited- highest to lowest’. The results showed the articles organised in a descending order with the articles most frequently cited on the top. The findings from each search word were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching journals in the Web of Science

The second search involved searching all journals in the field of medical education, ethics, general medicine and surgery included in the Web of Science database. These journals are known to publish articles on medical professionalism. They were selected on the basis of the outcomes of the Web of Science search and the references cited by the articles identified. The aims of this second search were to maximise the yield of the search and detect any articles that were possibly missed during the first search. This search was conducted under the same conditions of the first search, by the two researchers, on the same day, and by using the same keywords used in the first search. The journals in medical education that were searched included Academic Medicine, Medical Education, Medical Teacher, BMC Medical Education, Advances in Health Sciences Education Theory, and Practices, Teaching and Learning in Medicine and the Journal of Continuing Education in the Health Professions. The journals searched in general medicine and surgery were the New England Journal of Medicine, the Lancet, the British Medical Journal, the Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Internal Medicine, Archives of Internal Medicine, Canadian Medical Association Journal, PLOS Medicine, Annals of Surgery, Archives of Surgery, British Journal of Surgery, Perspectives in Biology and Medicine, Mayo Clinic Proceedings and the Australian Medical Journal. The journals in bioethics that were searched included the American Journal of Bioethics, Journal of Medical Ethics and BMC Medical Ethics. The findings from journals were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching the webpage of journals

To maximise the yield of our search and to ensure that no paper was missed through searching the Web of Science, we conducted a third search using the webpage of the journals mentioned above. We examined the titles of articles listed in each issue of these journals during 2011 and prior years. This search was particularly important as for example, Teaching and Learning in Medicine first appeared in Web of Science in 1996 but the journal was published since 1989. Therefore, any relevant articles from this journal or others prior to 1996 would be included.

Inclusion and exclusion criteria

The inclusion criteria were: (1) papers focusing on medical professionalism in the English language and (2) articles, reviews, research papers, reports, editorials on any aspect related to medical professionalism in the English language. The exclusion criteria were: (1) articles on medical professionalism in languages other than English, and (2) articles that focused on education/curriculum or clinical practices and medical professionalism was not the main focus. Articles with identical absolute number of citations were ranked on the basis of the average citation per year (the number of citations obtained divided by the number of years since published).24 A copy of all papers included in the list was obtained and read by the evaluators.

It is interesting to note that none of the articles excluded on the basis of language were qualified for inclusion in the list because they had less citation numbers than those of the article marked number 50 in the list.

Assessing the articles

For each of the identified articles (online supplementary appendix 1), a full text was obtained and a copy was given to each researcher. The following information was collected: (1) the authors’ names and their affiliations, and the number of females contributing to authorship (2) the number of institutes involved and the city and country of the origin of the publication, (3) the total number of citations obtained up to the day of searching the database, and the number of yearly citations since publication, (4) the year of publication and the calculated number of years since publication and (5) grants/funding bodies stated in the publication and (6) the 2016-JIF of the journal that published the work.

We also aimed at grouping the identified top-cited articles into categories. We have not used the categories provided by the Web of Science ‘study type’ because we noted that the Web of Science system does not differentiate between ‘original research’ or ‘articles’ and classified both as ‘articles’. For consistency and the purpose of this study, we grouped the articles into four categories—article, review, editorial material and research. A definition of each category is given in the glossary. Using these definitions, two researchers independently allocated each article under a category. For articles that were difficult to classify or not fitting into the same category, a meeting was held to discuss these articles and a final decision was made.

The topics covered in identified articles were created by each researcher independently by generating key words reflecting the main idea covered in an article and using these words to phrase a short statement that could help in grouping more than one article under one topic. The topics were then discussed in a meeting to harmonise the grouping into a logical, simple and practical approach. Articles covering more than one topic were classified on the basis of the aim of the study, the title and the main outcomes.

Identification of author’s gender

Regarding the data collected for each article, it is important to mention here that the identification of the gender of each author was a challenging task particularly when a journal uses abbreviations of the first and second name rather than the full name, which was the case in three articles. The approach used in order to identify the females in the top-cited articles included (1) searching the Google database to find the university website, personal website of the author, LinkedIn webpage and ResearchGate account. This approach was particularly useful for authors who could have moved to other universities, (2) searching the university websites not only provided the full names but also provided identification photos of these authors, and in many times a list of their publication records, as well as areas of research/teaching interests (3) Searching the Google Scholar database to identify their accounts, where we can find other publications under their names, the full name or an identification photo showing them. Usually, authors of highly cited articles have other publications related to the same topic, or work with the same coauthors, which could also help in identifying them and tracing them and (4) In two difficult cases we emailed the corresponding author of these articles for help.

Altmetric system

The altmetric system comprises, but not limited to, policy documents, news, blogs, tweeters, online reference managers (eg, Mendeley, CitULike), postpublication peer reviews (eg, Publons), Social media platforms (eg, Facebook, Google+, Pinterest), citations on Wikipedia, sites running Stack Exchanges (Q&A) and reviews on Faculty 1000 (F1000) and YouTube. Therefore, altmetric scores may reflect interest of the public as well as clinicians and researchers in a publication and the scores may provide information about the geographical and demographic details of those involved in such online/social media discussions.25

The altmetric programme processes raw data collected from the above-mentioned resources and the data are weighted according to a system created by altmetrics to reflect the relative contribution of each source to the total altmetric score. News, Blogs, Wikipedia and policy documents have a relatively higher weighting values.26 While Mendeley and CiteULike are shown in the report, they do not contribute to the total score.

Searching the altmetric system

The search of the altmetric system was conducted on the same day (5 April 2017). The scores were identified using the Altmetric bookmarklet provided by the company.27 In summary, the articles were searched on PubMed database (the PMID or DOI are essential for triggering the altmetric bookmarklet to function). By clicking on the LinkOut link, we identified the publisher webpage hosting the original article and by clicking the altmetric bookmarklet application, we can check the attention records for the article. The altmetric attention score and donut help in identifying the relative quantity and the type of attention received by a published article. The meaning of the colours included in altmetric donut is explained in this link.28

Statistical analysis

All analyses were conducted using SPSS Software (IBM SPSS Statistics Premium V.22.0 for Mac OS-SPSS) and the results were reported at total, mean, median, IQR and percentage. Pearson’s correlation coefficient (r) was calculated to determine if the high citation numbers obtained were related to parameters characteristic of articles. Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis model comprising the effect of number of authors and other parameters. The inter-rater agreement between evaluators was calculated using the Fleiss kappa scale.29

Patient and public involvement

Patients and the public were not involved in this study.

Results

Top-cited papers identified

Online supplementary appendix 1 summarises the 50 most-cited articles in medical professionalism identified by searching the Web of Science database,30–79 out of a total of 3500 articles identified on professionalism. The articles are listed in a descending order from 1 to 50 with the highest absolute citation number is ranked 1 and the article with the lowest citation ranked 50 as per the day of the search. Articles with the same number of citations were ranked on the basis of average citation per year (eg, the articles ranked 34 and 35 had the same citation number 97, they were allocated to a ranking order based on the calculated citation per year, 13.86 and 7.46, respectively). Other articles that had the same citation number and were ranked on the basis of their calculated citation per year were articles ranked 36 and 37; 43 and 44; as well as 46 and 47.

Table 1 summarises the year of publication and article category. The articles were published over 17 years (from 1994 to 2011). During the period from 1994 to 1999, only seven articles (14%) were published. However, the number increased significantly from 2000 to 2005 making a total of 24 (48%) articles. The number in the years from 2006 to 2011 dropped to 19 (38%). No correlation was found between the citation counts of these papers and the number of years since publication (Pearson correlation (r)=0.188, p=0.192).

Table 1

The most-cited papers in medical professionalism, summarised by year of publication and category

Article category	Year of publication: no of articles (reference)
	1994–1996	1997–1999	2000–2002	2003–2005	2006–2008	2009–2011	Total (%)
Article		259 74	435 46 60 67	840 47 49 51 53 61 64 76	333 65 68	269 75	19 (38)
Review	178	232 50	230 36	455 56 62 70		238 52	11 (22)
Editorial material		144	157	131	158		4 (08)
Research		142		434 37 66 73	543 45 48 54 71	639 41 63 72 77 79	16 (32)
Total (%)	1 (2)	6 (12)	7 (14)	17 (34)	9 (18)	10 (20)	50 (100)

The distribution of the medical professionalism topics covered in these articles is summarised in table 2. The inter-rater agreement between assessors was in the range 0.758–0.846.

Table 2

The most-cited articles in medical professionalism summarised by category and topic

Topics in medical professionalism	Category: no of articles (references)
	Articles	Review	Editorial material	Research	Total (%)
Defining and measuring medical professionalism	235 47	330 56 62			5 (10)
Role modelling, mentoring and professional clinical practice	140	232 78			3 (6)
Physician charter and professionalism			131		1 (2)
Response to conflict, social responses and social environment	333 46 61	155			4 (8)
Professional and unprofessional behaviour/disciplinary actions	368 74 75	136		334 37 71	7 (14)
Empathy and moral development		138		345 48 73	4 (8)
Professional conduct of medical students				241 63	2 (4)
Learning/teaching professionalism and curriculum issues	549 59 60 65 69 76	150	158	142	8 (16)
Online social networking and professionalism	169			339 43 72	4 (8)
Quality improvement and evidence-based practices	151	152		254 79	4 (8)
Self-assessment	153			177	2 (4)
Public roles and medical professionalism	164		244 57		3 (6)
Faculty development in medical professionalism				166	1 (2)
Medical ethics and end of life care	167	170			2 (4)
Total (%)	19 (38)	11 (22)	4 (8)	16 (32)	50 (100)

The articles were published in the following journals: Academic Medicine (n=19, 38%), the Journal of the American Medical Association (n=9, 18%), Journal of General Internal Medicine (n=4, 8%), Annals of Internal Medicine (n=4, 8%), the New England Journal of Medicine (n=3, 6%) and Medical Education (n=3, 6%). It is interesting to note that 24 (48%) articles were published in eight general medicine journal and the remaining were published in four medical education journals and one journal specialised in bioethics. Most journals have high journal impact factors and are on the top of their field (table 3). This finding reflects the significance of medical professionalism in undergraduate and postgraduate training. The first author of the top-cited articles was from the USA (n=37, 74%), Canada (n=8, 16%), the UK (n=2, 4%), Germany (n=1, 2%), Israel (n=1, 2%) and New Zealand (n=1, 2%).

Table 3

The journals that published the top-cited articles in medical professionalism and the journal impact factor

Journal	2016-Journal Impact Factor	No of papers published (references)
Journal of the American Medical Association	44.405	930 32 33 39 41 42 64 67 75
New England Journal of Medicine	72.406	334 44 57
Academic Medicine	5.255	1935–38 40 45–47 50 52 56 59 60 63 65 68 70 74 77
Journal of General Internal Medicine	3.701	443 48 61 69
Medical Education	4.005	349 66 72
Health Affairs	4.980	151
Advances in Health Sciences Education Theory and Practices	1.852	153
Annals of Internal Medicine	17.135	431 54 71 78
British Medical Journal	20.785	155
Medical Teacher	2.502	258 62
Canadian Medical Association Journal	6.784	173
American Journal of Bioethics	6.434	176
Archives of Internal Medicine	17.333	179

Table 4 summarises the 26 authors who have contributed to two or more articles in the list. Of these, five authors were the first authors of two or more papers, 10 were coauthors of two or more papers, and the remaining 11 were the first authors and coauthors of two or more articles. Top authors were Papadakis, M (n=4; first author of all four papers) and Blank, L (n=4; first author of one paper and coauthor of three). Other top authors are shown in table 4.

Table 4

Authors and coauthors of two or more articles

Author’s name	No (reference)		Author’s name	No (reference)
	First author	Coauthor		First author	Coauthor
Epstein	230 32	–	Greysen	169	139
Blank	131	333 56 71	Kind	–	239 69
Kimball	–	231 33	Mann	–	240 77
Brennan	133	164	Dyrbye	141	148
Rothman	157	133	Thomas	148	141
Blumenthal	–	233 54	Sloan	–	241 48
Papadakis	434 37 71 74	–	Shanafelt	–	241 48
Teherani	–	234 37	Holmboe	–	271 77
Veloski	156	134	Coulehan	246 47	–
Hodgson	–	234 37	Cruess and Cruss	250 58	166
Swick	235 42	–	Eva	153	177
Chretien	139	169	Wear	260 76	–
Levinson	–	252 75
Gruen	164	154

The leadership of universities and institutes that have contributed to the creation of these publications are shown in online supplementary appendix 1.

Characteristics of the top-cited articles

These articles were created by 252 authors, median 4, minimum 1, maximum 19, IQR 4 and the females in authorship were 102, median 2, minimum 0, maximum 11, IQR 2. The institutes involved were 168, median 2, minimum 1, maximum 17, IQR 3; the countries involved were 67, median 1, minimum 1, maximum 9, IQR 0; and the grants/funds received were 35, median 0, minimum 0, maximum 8, IQR 1. Significant correlations were found between the number of citations and the 2016-JIF (Pearson correlation (r)=0.318; p=0.024), and the number of countries (r=0.453; p=0.001). No significant correlations were found between the number of citations and the number of years since publication (r=0.188, p=0.192), the number of authors (r=0.159; p=0.270), the number of females in authorship (r=0.343; p=0.150), the number of institutes involved (r=249; P=0.081) or the number of grants received (r=−0.126; p=0.384).

The altmetric scores

The altmetric scores and reports were found for 70% of articles. The total scores were 806, median 5, minimum 0.0, maximum 155, IQR 20. No correlation was found between the number of citations and the total altmetric scores (r=0.064; p=0.661). A significant correlation was found between number of citations and altmetric scores for articles published in 2007 and after (n=17, r=0.547, p=0.023). No correlation was found for articles published in 2006 or earlier. Only 38% of the articles had readers on CiteULike (mean 1.6, 95% CI 0.4 to 2.7, median 0, minimum 0, maximum 19, IQR 1; while 62% were read Mendeley (mean 72.7, 95% CI 45.2 to 100.3, median 39.5, minimum 0, maximum 499, IQR 120. The coverage of journal articles by Twitter was 54% (mean 7.8, 95% CI 2.7 to 13.0, median 1.5, minimum 0, maximum 117, IQR 10) followed by blogs 38% (mean 1.2, 95% CI 0.6 to 1.8, median 0, minimum 0, maximum 7, IQR 1, then policy sources 24% (mean 0.38, 95% CI 0.15 to 0.61, median 0.0, minimum 0.0, maximum 3, IQR 0, then Facebook 20% (mean 0.3, 95% CI 0.07 to 0.61, median 0, minimum 0, maximum 6, IQR 0). The Wikipedia was the lowest resource. No significant correlation was found between the number of citations and altmetric scores (r=0.064; p=0.661). The geographical breakdown showed that USA had the highest share, followed by UK. Other countries identified for some articles were Canada, Mexico, Spain, Australia, Spain, Chile, Netherlands, Portugal, Japan, Columbia, Italy, France and Brazil. We looked at journals’ webpages of the top-cited articles and those hosting the altmetric metre; we did not find significant correlation between hosting the altmetric metre and recorded altmetric scores.

Multivariate analysis

Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis comprising the effect of number of authors, number of institutes, number of countries, number of females in authorship, number of grants obtained on the citation scores and altmetric scores (table 5).

Table 5

Assessing the impact of publication variables on citation scores and altmetric scores using multivariate analysis

Category (n)	Citation scores		Altmetric scores		P value
	Mean±SD	95% CI	Mean±SD	95% CI
Authors: four or more (6)	228.5±186.4	86.5 to 370.5	2.5±4.8	−41.2 to 46.2*	0.762
Authors: 2–3 (16)	161.7±211.2	74.7 to 248.6	9.9±16.3	16.8 to 36.6	0.802
One author (28)	166.5±144.3	100.8 to 232.3	33.6±69.1	13.4 to 53.9	0.803
Institutes: 3 or more (18)	200.3±219.2	118.7 to 281.9	6.0±7.4	19.3 to 31.3	0.563
Two institutes (12)	131.2±46.6	31.2 to 231.1	34.2±97.6	3.2 to 65.2	0.272
One institute (20)	172.0±170.4	94.6 to 249.5	29.8±38.5	5.8 to 53.8	0.541
Countries: 2 or more (40)	169.8±158.9	115.1 to 224.6	21.8±55.7	4.6 to 39.1	0.411
One country (10)	182.7±221.0	73.1 to 292.3	24.1±47.3	10.4 to 58.6	0.808
Female authors: 2 or more (10)	263.6±281.9	157.1 to 370.1	6.4±8.0	−27.6 to 40.4	0.151
One female author (14)	166.8±187.7	76.8 to 256.8	12.4±20.6	−16.4 to 41.1*	0.286
No female author (26)	140.4±75.6	74.3 to 206.4	33.8±71.5	12.7 to 54.8*	0.334
Grants: 2 or more (30)	197.5±213.1	134.6 to 260.4	10.7±16.4	8.4 to 29.9	0.451
One grant (15)	36.7±60.4	47.7 to 225.7	34.7±86.9	7.4 to 61.9	0.132
No grants (5)	128.8±53.7	25.3 to 282.9	55.0–63.3	7.7 to 102.3	0.394

None of the categories studied caused significant differences on the citation scores or the altmetric scores.

*The analysis involved subgroups, smaller sample size, and because the smaller altmetric scores for these subgroups, the CI was negative.

Discussion

The aims of this study were to identify the highly cited papers in medical professionalism and compare their characteristics and citation analysis with the altmetric scores. Currently, there is a great interest to examine if there is a relationship between altmetric indicators and citation counts. The question remains can we measure the impact of scientific publications by measuring their social density effects?

The study has identified key topics related to medical professionalism including: learning/teaching professionalism and curriculum issues, professional and unprofessional behaviour/disciplinary actions, defining and measuring medical professionalism, response to conflict, social responses and social environment, empathy and moral development, online social networking and professionalism, quality improvement and evidence-based practices, role modelling, mentoring and professional clinical practice, and public roles and medical professionalism. While these topics highlight major issues related to medical professionals, topics related to transition of first-year students from being laypersons to being members of the medical profession, how medical schools change assessment to focus much more on a student’s attitudes and personal development as a professional, not just on his or her knowledge of medicine, as well as strategies to introduce new teaching/learning approaches that facilitate the integration of medical professionalism across the years in the medical curriculum and demonstration of professional behaviour in day-to-day practices may be lacking.80

The study revealed the characteristics of the 50 most-cited articles; the following points are worth discussion:

First, the study demonstrated that there is no significant correlation between the citation counts and the number of authors or the number of female authors. The number of authors and females in authorship varied from 1 to 19 and from 0 to 11, respectively. The two questions that can be raised in this regard; are we expecting an increase in number of citations as the number of authors increases? And is the gender of authors a factor affecting citation counts? Several studies indicated that a number of authors or the gender of authors are not among the factors affecting the citation received by a publication; factors such as having a higher level of evidence may be more likely to affect citation counts.81

Although the proportion of women in authorship of original research in the USA in general has significantly increased in the last four decades and more women are enrolling Master’s and PhD degrees,82 women still compose a minority of the authors of original research and there are some differences by subfield.83 84 Recently, an increased satisfaction about the proportion of women faculty, especially full professors in academic medicine, has been reported, suggesting an improvement in the balance at least in this subfield.85

Second, the study showed significant correlation between the citation counts and the number of countries involved, but not the number of institutes. However, further analysis using multivariate analysis model did not show significant relationships at different subgroups. While Figg et al 86 reported that there is a correlation between the number of authors and the number of times an article is cited by other researchers, the work of García-Aroca et al 87 showed that publishing in English in certain journals and collaborating with certain authors and institutes increase the visibility of the manuscripts published on the subject. Therefore, it is the quality of collaboration rather than the absolute number of these parameters. Recently, Tanner-Smith and Polanin showed that studies conducted by more established authors (have higher h-indices) and reported in more prestigious journal outlets are more likely to be cited by other scholars, even after controlling for various proxies of study quality.88

Third, the study showed no significant correlation between the number of citations and the number of grants received. This finding is not surprising. Recently, it was shown that too many of the US authors of most influential papers in science do not receive National Health (NH) funding.89 Another group of researchers found no association between grant percentile ranking and grant outcome as assessed by number of top-10% articles per dollar million spent.90 Interestingly, the work of Gök et al 91 showed that funding on its own is not a measure of citation impact but is principally related to funding variety and negatively related with funding intensity. Also, there was an inverse relationship between the relative frequency of funding and citation impact.

Fourth, the lack of significant correlation between the number of citations and the number of years since publication may indicate that the higher citations are not due to ageing of articles but possibly due to the new knowledge discussed and the evidence presented by authors to answer challenging questions. In fact, the majority of these articles (36, 72%) were published in 2003–2011 and the oldest article in the list was published in 1994.

The USA, Canada and the UK contributed most to these articles. The leadership of universities from these countries in medical professionalism is no surprise. Altmetrics have a number of functions including: first, a record of the degree to which people, public and academics/clinicians, engaged with a scholarly publication. Second, a measure of the dissemination of a scholarly work including the geographical and demographic details of those involved in such discussions on social media channels. Third, possibly an indirect measure of influence and impact of scholarly work.

The idea of ‘altmetrics’ or social web metrics was first proposed by Priem and Hemminger92 and is based on the hypothesis that the analysis of scientific outputs and discussions in social media tools can be used as an alternative to citation bibliometrics created by Garfield.93 The hypothesis may bring new insight into the understanding of scientific impact and the type of relationship between alternative metrics and citation scores. However, currently, there is evidence that the use of social media in promoting and discussing research is low in the research community. It has been reported that 15%–25% of scientific publications have some altmetric activities and these activities are observed mostly in recent publications in social sciences, humanities, medical and life sciences.94 With these limitations in mind, it is clear that altmetrics open new directions in understanding scientific impact of a publication not just through peer-review and citation indices, but through assessing other aspects of impact at society, education and public domains. While Powell et al found a correlation between number of citations and altmetric scores for articles published after 2000,95 we found significant correlation for articles published in 2007 and after. No correlation was found for articles published in 2006 or earlier. Our findings and those of Powell et al indicate the presence of such correlation for articles published after the year 2000.

This study has several strengths, firstly: the search was conducted by two researchers independently using four approaches with the aim to maximise the outcomes of the search and not to miss a publication. Second: the study examined the citation numbers, related bibliometric parameters and altmetric scores, the relationships between these variables and their possible effect on citation counts and altmetric scores were evaluated using correlation studies and multivariate analysis. Third: the study covered top-cited articles on medical professionalism over the last two decades. However, this study is not without limitations, First, we limited the search to Web of Science database, and we have not searched other databases such as Google Scholar or Scopus for reasons mentioned under methods. Also, we have searched highly cited journals in the area of medical education, ethics, bioethics, general medicine and surgery to compensate for using one database; and second, we limited the study to articles published in the English language. However, further evaluation revealed that articles published in languages other than English were not qualified for inclusion and their citation counts were below the article listed number 50.

Conclusions

Using a multivariate analysis model and correlation studies showed that several bibliometric factors neither correlated with citation scores nor altmetric scores. These variables included, years since publication, the number of authors, the number of female authors, the number of institutes and the number of grants received. The number of females in authorship (40% of total number of authors) highlights the progressive role of females in medical education and the area of medical professionalism. It may be premature to make conclusive remarks about the significance of altmetric scores. However, the finding of correlations between the number of citations and altmetric scores of articles published in 2007 and after provides an additional parameter to the value of altmetric scores.