Manuscript review continuing medical education: a retrospective investigation of the learning outcomes from this peer reviewer benefit.

OBJECTIVES: This study investigates the learning outcomes for peer reviewers participating in a manuscript review continuing medical education (CME) process. CME from serving as a peer reviewer is one of the many benefits of serving as a reviewer.
DESIGN: This is a descriptive study retrospectively analysing learning outcomes self-reported by peer reviewers from 2013 to 2017 using a CME assessment framework. SETTING, PARTICIPANTS AND PRIMARY OUTCOME MEASURES: Participant data are from 1985 peer reviewers who completed 2413 manuscript reviews over 32 medical journals from 2013 to 2017 and completed the CME process after their prepublication manuscript review. 417 reviewer responses were practice behaviour change(s) that were studied in depth using an assessment framework on changes in knowledge, competence and performance.
RESULTS: The results show positive learning outcomes reported by reviewers at the knowledge, competence and performance behaviour levels as a result of reviewing manuscripts. Higher levels of learning outcomes are more frequently achieved when reviewers consult multiple sources when conducting reviews. Reviewer demographics, such as gender or years of experience, did not have a significant association to learning outcomes.
CONCLUSIONS: Manuscript Review CME is an effective way that learning within the peer reviewer process can occur and helps reviewers gain knowledge, improve competence and make changes to their professional practice at all stages of their careers. Journal publishers should emphasise and support reviewers through offering CME to reviewers and encourage consultation of multiple sources when conducting reviews, which is an added benefit and resource to help professionals continue their development. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of this study is assessing learning outcomes from peer review.

Manuscript review continuing medical education is a process overlaid to peer reviewing and learning outcomes can be determined and analysed using an assessment framework.

The voluntary nature of participation is a limitation as it indicates a degree of selection bias.

Self-reported outcomes are a limitation and more objective observations should be sought in future studies.

Introduction

Peer review of medical journal manuscripts is a long-standing process in which manuscript content and scientific validity are evaluated by outside sources with expertise in the area under review prior to acceptance for publication.1–3 While peer review is a process that aims to aid authors and publishers by ensuring scientific validity, it also carries benefits for reviewers as well. One of those benefits is that medical journals are granting continuing medical education (CME) credits to manuscript reviewers for reviewing, which is a part of continuing professional development.4 Various factors motivate reviewers to engage in peer review apart from receiving CME credits, such as professional responsibility, scholarly activity and remaining current in their field.5 The literature on peer review in higher education points to various potential benefits for manuscript reviewers. A summary of these is as follows:

Recognition: Invitation to serve as a reviewer may be viewed as an honour and participation as one underscores the reviewer’s reputation as a subject matter expert.6–10

Professional responsibility: It is seen as a service to one’s profession and an opportunity to give back particularly if one has been a beneficiary of peer review.5 6 10

Become better scholar/improved scientific capability: Serving as a reviewer enables acquiring knowledge and skills that contribute to becoming a better scholar and scientist.7 9–14

Access to new information in the field of interest: Peer review exposes reviewers to new information and discoveries in their fields of interest, thus helping them stay abreast of new developments in their fields.4–6 8 11 13 15–17

Contribution to scholarship: It offers reviewers the opportunity to contribute to scholarship and influence the information that gets disseminated in their fields.4 10 15 16

Professional advancement and development: Reviewing can contribute to the reviewer’s professional growth; and participation in peer review can be looked on favourably in review for promotion or tenure.6 8 9 11

Opportunities for collaboration: Peer review can result in contacts with potential collaborators or collaborative opportunities.8 12

Building on these many positive benefits for manuscript reviewers is the opportunity to earn CME credit through completing a process of reporting learning outcomes from conducting the manuscript review. CME credit typically pertains to postgraduate learning interventions designed by education providers to meet educational needs that narrow gaps in practice, implemented independent of commercial influence and evaluation about changes in outcomes is conducted and analysed. CME activities have demonstrated positive outcomes on physician performance and patient health outcomes.18 Despite the research into CME effectiveness thus far, little is known about outcomes related to manuscript review CME, a distinct type of CME.

Since 2005 the Accreditation Council for CME (ACCME) has collected data about providers offering manuscript review as a CME activity. A manuscript review CME activity is a process that an education provider offers to reviewers through which they obtain CME credit for reflecting on their individual learning from participating as a reviewer in the prepublication stage. The credit is valuable for reviewers to use for maintaining professional licensure and documentation of professional development. CME is valuable for providers to use for rewarding reviewers with a benefit, advertising of CME availability to recruit future reviewers and analysing outcomes to help guide future reviewer training. In 2018, the ACCME reported that 139 manuscript review CME certified activities were offered among ACCME accredited providers and that 62 206 physicians and 8593 non-physicians participated in the CME process.19 There has not been an empirical investigation of learning outcomes from peer-reviewing in the CME domain, which is warranted to study the effects this form of learning has for reviewers.

This study investigated the perceived learning outcomes achieved by reviewers from the manuscript review CME process. In particular, we investigated the types of outcomes achieved such a knowledge gained, improvement to competence and performance changes.

Design/methods

The Cleveland Clinic Institutional Review Board approved this study as exempt as a minimal risk study that used data collected for routine practice.

Patient and public involvement

The development of the research question was framed by the goal of investigating how healthcare professionals learn and improve their practice through CME, which ultimately leads to improvements in patient care. No patients or the public were involved in the design of this study.

Setting and participants

The Cleveland Clinic Center for CME has been providing CME credits to reviewers on behalf of numerous peer-reviewed medical journals since 2012. The Center offers up to 3.0 CME credits for each review. The steps in the process are as follows:

Review completion: Each reviewer submits his review of a manuscript through the editorial management platform.

Editorial determination of CME credit eligibility: The journal’s editorial team determines if the review is eligible for CME credit based on timely submission and substantive review of the manuscript.

Reviewer completes a CME questionnaire: A confirmation email is sent to the eligible reviewer with a link to claim CME credit. The link leads to a set of questions about the reviewer’s experience and to report on his/her perceived learning gained from conducting the review. It is not mandatory to answer every question. After completing the questionnaire, the CME process concludes with the reviewer claiming up to 3.0 credits and receiving a CME certificate for the review.

The Center reviews the completed manuscript reviews for CME credit on a quarterly basis to monitor trends and share the results with stakeholders.

Included in this study are data from 1985 reviewers who completed 2413 reviews from 2013 to 2017 and completed the CME process after their prepublication manuscript review. Reviews were from 32 medical journals that ranged in focus from general practitioner to medical specialty audiences, as well as of wide-ranging interest to all healthcare professionals. A subset of the reviewer responses (n=417) reported practice behaviour change(s) resulting from conducting the peer review. These responses were analysed in-depth using a CME outcomes framework (discussed later) to determine the corresponding level of learning outcomes attained as a result of conducting the review, which may not have been at the practice behaviour change level and was instead related to a change in knowledge or competence.20

Questionnaire

The participating reviewers responded to an online questionnaire of closed-ended and open-ended questions after completing each manuscript review (table 1). Reviewers were asked to select the sources they had consulted in completing the review, if the manuscript review resulted in acquisition of knowledge and change in their practice behaviours. Those who attributed their practice behaviour change to the manuscript review were asked to respond to the following open-ended statement: If so, please list specific behaviours that you may change.

Table 1

Manuscript review CME questionnaire template

Please identify sources consulted in completing review (check all that apply)	As a result of completing this manuscript review I gained knowledge useful in my clinical practice:	As a result of completing this manuscript review my practice behaviour will change*	How would you rate using the manuscript review CME process?
Existing personal knowledge base (specialist in the field)	N/A—I am an expert reviewer	Reinforced my practice behaviour	Excellent
Referenced articles in manuscript	Completely agree	Completely agree	Good
Medline (PubMed)	Agree	Agree	Satisfactory
Textbooks	Somewhat agree	Somewhat agree	Poor
Professional colleague(s)	Disagree	Disagree
Other	Completely disagree	Completely disagree

*As a follow-up to affirmative answers to this question reviewers were asked to respond to the following open-ended statement: If so, please list specific behaviours that you may change.

CME, continuing medical education.

The questions about gaining knowledge and practice behaviour changes are asked uniformly across all types of educational activities offered by Cleveland Clinic’s CME programme, including live courses and online educational activities. The CME programme offers approximately 1700 activities annually that reach over 175 000 participants. The evaluation questions were designed to monitor and compare learner and activity outcomes across the whole CME programme. The question about sources consulted was identified by CME department leadership for the manuscript review CME process to gather data on the review process. The questionnaire has been in use since the manuscript review process launched in 2012 and has not changed. It was implemented via a custom-developed SQL server database with a web-based application that is run on a Linux server.

Conceptual framework

We conducted a more in-depth analysis of the reported behaviour changes to determine the level of learning outcomes achieved. We relied on an outcomes framework for assessment and planning of CME developed by Donald Moore to analyse and synthesise findings.20 It consists of seven levels of outcomes. Level 1 pertains to tracking the number of participants in a CME or educational activity. Level 2 measures the participants’ level of satisfaction with a CME activity. Level 3 measures learning from a CME activity and consists of acquisition of declarative, procedural or conditional knowledge. Level 4 assesses competence, the extent to which participants can demonstrate how to do what the CME activity taught them to do. Level 5 assesses performance, the degree to which participants actually do what the CME activity expected them to do in practice. Level 6 measures the degree to which the health of patients improves as a result of participants’ changes in practice behaviours attributed to a CME activity. Finally, level 7 gauges changes in the health status of a community of patients resulting from participants’ practice behaviour changes.

Procedure

For the participants that provided responses to the open-ended question regarding the specific behaviours they would change as a result of their reviews, we coded their response using Moore’s CME framework to see if they pertained to changes in knowledge (level 3) or competence (level 4) outcomes, or were at the performance level outcome (level 5). Levels 3, 4 and 5 pertain to individual learning outcomes. The higher outcomes at Level 6 or 7 pertain to patient or community health status, which are outside the scope for this individual learning activity. Two medical educators (SK and SM) with social science backgrounds first independently reviewed and classified the 417 responses into one of the levels on Moore’s framework. The educators then discussed all the classifications to reach consensus on their classifications for the reported behaviour changes.

Statistical analysis

We used descriptive statistics (eg, frequencies) to examine reviewers’ response to the close-ended questions about their manuscript review experience. χ2 tests were used to compare review outcomes across levels of reviewer characteristics such as gender, geographic region, years of experience postresidency and medical school. The Cramer’s V statistic was used as a measure of the effect size (ES). Cohen’s (1977) criteria (ie, 0.10=small ES, 0.30=medium ES and 0.50=large ES) were used to judge the practical significance of the ES indices. The above analyses were conducted on SPSS V.24 (IBM Corporation) with p<0.05 set as statistically significant for all hypothesis tests.

Results/findings

A total of 2413 reviews were completed by 1985 unique reviewers from 2013 to 2017 and went through the CME process postreview. There were 32 instances (1.3%) where reviewers opted not to respond to the questions on the questionnaire.

The results show positive learning outcomes reported from reviewing manuscripts (table 2). For the majority of manuscript reviews (79%), reviewers reported gaining knowledge useful for their clinical practice. This represented reviewers who completely agreed, agreed and somewhat agreed to achieving this outcome. Sixteen per cent reported that they were expert reviewers hence the outcome was not applicable to them whereas 6% did not result in useful knowledge gain. Twenty-three per cent and 45% of the reviews reinforced existing practice behaviours and predicted change in behaviours, respectively, while approximately one-third (32%) of the reviews did not have any effect on practice behaviour changes.

Table 2

Summary of total reviews for CME

As a result of completing this manuscript review I gained knowledge useful in my clinical practice?			As a result of completing this manuscript review my practice behaviour will change:
	n*	%		n*	%
N/A—I am an expert reviewer	375	16	Reinforced my practice behaviour	544	23
Completely agree	586	25	Completely agree	177	7
Agree	889	37	Agree	356	15
Somewhat agree	405	17	Somewhat agree	534	23
Disagree	89	4	Disagree	675	29
Completely disagree	37	2	Completely disagree	76	3
	2381			2362

Unique reviews 2413, total number of reviewers 1985, # of journals 32.

*The total of responses (n) for each question does not add up to 2413 because some reviewers did not answer those volitional questions; thus we excluded the missing data from the calculations.

CME, continuing medical education.

In 417 of the reviews, specific practice behaviour changes were noted in the open-ended responses by reviewers (see table 3). A review of the data led to the exclusion of 34 responses that were comments unrelated to learning outcomes, such as pertaining to the quality of the paper reviewed or noted as being an invalid response. A few examples of these invalid responses are: ‘manuscript was of low quality so not relevant’, ‘not sure I agree with the authors conclusions’, ‘NA—I was an expert reviewer’ and ‘unclear how practice should or should not change at present’. The remaining 383 responses represented categories of learning outcomes on Moore’s framework such as knowledge (44%), competence (44%) and performance (12%) (see table 3). Some examples of changes include:

Table 3

Learning outcomes levels of reported behaviour changes and cross-tabulation by number of sources consulted

# of sources	Knowledge	% of total	Competence	% of total	Performance	% of total
1	34	9	19	5	5	1
2	58	15	49	13	18	5
3	59	15	84	22	18	5
4	15	4	12	3	5	1
5	4	1	3	1	0	0
	170	44	167	44	46	12

Total number of reviewers that reported specific behaviour changes in the open-ended response field: 417, valid reponses: 383, invalid responses: 34.*

*Invalid Responses pertained to comments about the quality of the paper or the lack of relevance to practice, which did not indicate an identifiable behaviour change learning outcome.

Knowledge level: Awareness of new treatments, studies, and methods.

Competence level: Consideration of medical history patterns in diagnostics, more effectively implementing procedures using modifications and best practices, ability to more effectively treat and manage.

Performance level: implementing new screening tools in practice, using treatments for acute conditions, avoiding unnecessary tests and making use of nutrition assessments to assist oncology patients.

The majority of reviewers who reported gains in knowledge, competence or performance had consulted two or more sources. They include 136 out of 170 respondents indicating acquisition of knowledge, 148 out of 167 respondents reporting changes in competence and 41 out of 46 reporting changes in performance (table 3). This suggests that higher levels of learning could result from consulting multiple sources prompted from the review process. Reviewers learning occurred not only from reading the manuscripts under review but also from consulting other sources to help conduct the review. While 35% of the reviewers relied on existing knowledge, the majority of them consulted external sources. Sources consulted ranged from articles referenced in the manuscript (29%), articles identified in their Medline/PubMed searches (23%), textbooks (6%), colleagues (4%) and other sources (2%) apart from those listed (table 4). Most of the reviewers consulted multiple sources.

Table 4

Frequency of sources consulted when practice change reported

Source	n	%
Existing knowledge	364	35
Referenced articles	301	29
Medline/PubMed searched articles	239	23
Textbooks	63	6
Colleagues	43	4
Other	16	2

Reviewers’ gender and years of experience postresidency were not significantly related to the learning outcomes (p>0.05), whereas the reviewers’ continent and years of experience postmedical school were statistically significant (p<0.05). See table 5 for specific details. ESs were small (ranged from 0.05 to 0.20) for all contingency tables, suggesting weak associations between learning outcomes and reviewers’ demographic characteristics.

Table 5

Relationship between characteristics of reviewers with type of review outcome of the Cleveland Clinic Center for Continuing Medical Education CME credits, 2013–2017

Reviewer characteristics	Number (%)	Type and frequency of review outcomes					P value	Effect size†
		K	C	P	NA	χ²*
Gender (n=417)						1.12	0.77	0.05
Male	118 (28)	50	48	10	10
Female	299 (72)	120	119	36	24
Continent (n=390)						31.45	0.03	0.16
Africa	4 (1)	1	1	0	2
Asia	13 (3)	5	7	0	1
Europe	196 (50)	86	75	26	9
North America	163 (42)	60	67	16	20
Oceania	10 (3)	6	2	0	2
South America	4 (1)	0	3	1	0
Years of experience (postresidency) (n=180)						22.08	0.11	0.20
Less than 10 years	30 (17)	11	16	2	1
Less than 20 years	45 (25)	20	20	4	1
Less than 30 years	37 (21)	15	16	2	4
Less than 40 years	53 (29)	26	15	8	4
Less than 50 years	13 (7)	2	6	2	3
Less than 60 years	2 (1)	1	0	0	1
Years of experience (postmedical school) (n=290)						27.49	0.03	0.18
Less than 10 years	14 (5)	7	4	1	2
Less than 20 years	95 (33)	38	46	7	4
Less than 30 years	64 (22)	23	28	7	6
Less than 40 years	70 (24)	37	24	7	2
Less than 50 years	40 (14)	15	11	5	9
Less than 60 years	7 (2)	3	1	1	2

*χ2 values were used to assess differences in outcomes across levels of reviewer characteristics such as gender, continents they were from, their years of experience postresidency and medical school.

†The Cramer’s V statistic was used as a measure of the effect size. This statistic has a lower limit of 0 and an upper limit of 1 where standards for interpreting Cramer’s V (0.10=small, 0.30=medium and 0.50=large) help determine the magnitude of difference across variable levels.

C, competence; CME, continuing medical education; K, knowledge; NA, not applicable; P, performance.

Discussion

While serving as a peer reviewer often validates existing knowledge, the CME learning outcomes indicate that reviewers can not only gain new knowledge but also have the opportunity to improve their competence and change behaviours in practice. This study shows that learning most likely occurs from reading the manuscript itself and also through consultation of other materials, such as referenced articles and related publications on the topics addressed in the article under review that were identified in the reviewers’ independent searches on Medline/PubMed. Higher levels of learning outcomes are achieved when multiple sources are consulted in the review process.

There are many potential benefits to serving as a peer reviewer, also including learning as a CME activity. Those that desire to learn during peer review should maximise their opportunity by consulting literature cited in the article, but also by independently searching for related literature.

The findings show another benefit resulting from the peer review process. Even though there is some debate about the effectiveness of peer review in protecting the scientific process for disseminating findings, of which reviewers play a central role, reviewers can advance their own knowledge, competence and performance improvements by service as a reviewer.21 Manuscript review as a CME activity should encourage potential reviewers’ participation in peer reviewing. This could support journals’ recruitment and retention of reviewers. Additionally, observing that higher levels of learning outcomes are related to reviewers’ consultation of multiple sources should be factored into publishers training of reviewers to not only ensure quality reviews but also to maximise reviewers’ learning.

There are some limitations to this study. First, there is a degree of selection bias. It is a voluntary process for reviewers to claim CME credits and report learning outcomes, which indicates selection bias as participants desiring CME are more likely to report their learning and earn credit complete the process. Second, the study included self-reported outcomes, thereby falling short of objective measures that might demonstrate evidence of actual behaviour changes made. Thus, future investigations should focus on objective review of outcomes, instead of self-reporting, to ascertain demonstrated competence or actual performance changes attained. Future studies into factors impeding or contributing to positive learning outcomes would be helpful so education providers or journals can consider facilitating achievement of desired learning outcomes. Additionally, the number of reported outcomes is proportionately a small percentage and limited to one education provider’s experience, thus more studies across multiple providers could offer additional insights into learning outcomes.

Conclusions

This study indicates that manuscript review CME is an effective way that learning within the peer review process can occur and help reviewers gain knowledge, improve competence and make changes to their professional practice at all stages of their careers. As long as peer review continues to be a central part of the dissemination of scientific findings, journal publishers should emphasise and support reviewers through offering and encouraging CME to reviewers. Peer-reviewing offers many benefits to the reviewer, and the evidence that the CME process can be one of those benefits should serve as another motivation to become a peer reviewer.