Recommendations for reviewing a manuscript for the GMS Zeitschrift für Medizinische Ausbildung.

The aim of this paper, written by the committee of educational research methodology of the "Society for Medical Education" of the German-speaking countries, will give recommendations for the review process of scientific papers in medical education. The recommendations are based on the results of a workshop in 2007 and on a survey among reviewers of the journal GMS Z Med Ausbild. It reflects on international standards and research in medical education in Germany. The paper describes reviewer's function concerning the journal of GMS Z Med Ausbild and specifies criteria for the review process with regard to the editors and reviewers; it also gives proposals for a feedback to the author. The catalogue of criteria for the reviewers is pictured in a checklist. The present recommendations shall help to increase the quality of the review process and to improve the national and international acceptance of the journal GMS Z Med Ausbild. Additionally, transparency of the review processes will support authors to submit a scientific article of high quality.

Introduction

For five years, membership of the German Society for Medical Education (GMA) has been increasing annually by 20%; currently, the society counts 780 members. Amongst others, the members are physicians, dentists, veterinarians, psychologists, biologists, pedagogues and students. The number of visits to the GMA online newspaper GMS Z Med Ausbild increased from 300,000 in 2007 to 430,000 in 2009. To meet the members’ interests, subject matters are published from the field of education and further training in medicine, medicine didactics, adult education, quality management of the training, pedagogical psychology (as far as it is relevant for the training of physicians), educational policy and university education” (http://www.egms.de/en/journals/zma/authors.htm). “Every person involved in university education as well as the interested public” is stated as the target group (http://www.egms.de/en/journals/zma/authors.htm).

The readers and their interests are represented by the editor. In order to support the editor in decision making and to increase the quality of the submitted manuscripts [1], [2], overviews, original works, projects and comments are reviewed by committed people from the field of medical education. The aim of this article is to describe the tasks of a reviewer and to give recommendations for reviewing for the GMS Z Med Ausbild. As a position paper of the committee, this article refers to international standards especially taking into account the situation of medical education research in Germany. This article was produced in close cooperation with reviewers of the GMS Z Med Ausbild.

Literature overview

High-quality studies facilitate reviews by experts and editors and offer high benefits for the readers further down the line [3]. The submitted manuscripts often lack in critical literature overviews, conceptual frameworks, statement on study designs, definitions of the control groups or notes on ethical aspects [2]. In 1997 and 1998, Bordage [4] counted the grounds of acceptance and refusal of submitted research manuscripts during the Research in Medical Education Conference. The most commonly stated reasons of refusal were insufficient statistics (11.2%), over-interpretation of the results (8.7%), inadequate instruments (7.3%), too small or distorted control samples (5.6%), difficult to read texts (3.9%), insufficient data descriptions (3.9%), problem descriptions (3.4%), literature overviews (3.1%), data presentation (2.7%) and tables/figures (2.5%). Reasons for acceptance of manuscripts were: important subject matter (20.2%), well written (18.3%), good study design (10.3%), current literature overview (6.7%), control sample big enough (4.4%), practical implications (4.4%), limitations in the interpretation (4.4%), well described problem (3.6%) and innovative approach of the analysis (3.6%).

Quality could be increased by adhering to guidelines or checklists when writing manuscripts [3]. Check lists have the advantage that they help reviewers read the articles more quickly, review them critically and make the approach taken more easily accessible. Rather, due to their simple structure, the reviewers can identify possible deficiencies in the presentation more easily. Even authors receive active help from checklists during the planning phase [5].

Review checklists are offered by several medical education magazines, for example Academic Medicine, Medical Education, Applied Behavioral Science, New England Journal of Medicine, The Lancet and Annals of Internal Medicine [6]. In addition to checklists, Medical Education also offers a mentoring programme for reviewers [7].

Description of the national situation

Due to the welcome increase in GMA memberships, the new possibilities of designing medical undergraduate studies within the Medical Licensure Act for Physicians 2002 and the extension of professional degrees in the medical education research in the German-speaking area, a certain “spirit of optimism” is now prevailing in medical education research.

Within in the past years, the number of original works submitted to the GMS Z Med Ausbild and the share of refused manuscripts increased continuously (see table 1 (Tab. 1)).

Table 1

Publications and refusals by the GMS Z Med Ausbild (2005 – 2009)

At the annual GMA conference in 2007, the committee of educational research methodology ran a workshop on “How to review a manuscript on education research in medical science”. The aim of this workshop was to develop joint recommendations for reviewing manuscripts, especially for medical education research. The result of the workshops was that reviewers recognised a demand for standardised criteria for reviewing manuscripts in medical education research. In order to obtain a more detailed idea of the wishes and needs of all GMS Z Med Ausbild reviewers, subsequent to the workshop we interviewed the target group.

In 2008, all of the 119 GMS Z Med Ausbild reviewers were provided with a questionnaire (see Appendix 1 ). This questionnaire contained general questions about the review activity (how long they had been reviewing, number of reviewed manuscripts in the past two years, experience with manuscripts from educational research), questions about the quality of the reviewed manuscripts (5-level rating scale from poor to high), questions on defects of reviewed manuscripts (5-level rating scale from small to grave), questions on manuscript requirements (5-level rating scale from low to high) and open questions on potentials for improvement and the wishes for improving the review process.

24 of 119 interviewed reviewers sent an answer. The main activity of the reviewers consisted in teaching, followed by clinical work and research. Only a few reviewers indicated pre-clinical studies as their main field of activity. The reviewers stated different periods of experience with reviewing manuscripts, only 25% less than 3 years of experience. In the past two years, half of them reviewed 5 to 12 manuscripts.

The descriptive statistics of the questionnaire are listed in the following tables. Due to the ordinal level data, Figures 1 (Fig. 1) and 2 (Fig. 2) show medians, and, where applicable, percentiles. The 25% percentile indicates up to which value were the “worst” 25% of the answers. The median (50% percentile) indicates the mean answers and the 75% percentile shows the lowest limit of the 25% best answers.

Figure 1

Experience with the quality of manuscripts from the education research area

Figure 2

Experiences with the frequency of defects in manuscripts and the requirements for a good manuscript in education research

For better understanding of the following figures and tables, the questionnaire is also included in Appendix 1 .

As to the open questions, similar statements were summarized and the statements were counted. Table 2 (Tab. 2) shows the results.

Table 2

Potential for improvement

The GMS Z Med Ausbild uses a review form called Manuscript Operating System (MOPS) which is available on the portal of the German Medical Science. It contains questions and used the rating scales yes/no/unsure/not applicable. The questions refer to content and form. In addition, it contains text boxes for general and specific comments by the author. This includes the general scientific review as well as specific comments which refer to certain pages and lines of the submitted manuscript. It is followed by a field called “Explanation for the editorial office”. In the final review, the reviewer can accept the manuscript in the present form, accept it in a revised form, refuse it with the possibility of re-submission or refuse it. The general and specific comments are the feedback for the author. The author will not have access to the review criteria and the rating.

Evaluation/Comments

The results of the survey of 24 of 119 reviewers of the GMS Z Med Ausbild surveyed show that the quality of peer-reviewed manuscripts on average will be assessed as mediocre, with experience/project reports and original works showing a particularly large spread. Especially common were deficiencies in the statistical analysis and study design, as reported in other journals [4]. Particularly important criteria for the reviewers were objectivity, study design and conclusions. The surveyed experts surveyed said that in the submitted manuscripts in particular the presentation of new and useful things has to be improved especially in the discussion. Furthermore, the desire for specific instructions for authors and the respect of the same droplet. The reviewers surveyed said that in the submitted manuscripts, the presentation of the discussion of new and useful things must be improved in particular. Furthermore, there is a desire for concrete guides for authors and adherence to such. The criteria underlying the assessment process should be clarified and made transparent for the authors. It would be desirable if the manuscripts could be pre-reviewed more intensively and if sufficient time for processing would be available.

In the evaluation of survey results it must be borne in mind that the survey was conducted in 2008 and only 20% of all reviewers answered questionnaires. Due to the low response rate, selection artefacts cannot be excluded. Overall, specific experience in reviewing manuscripts in medical education was rather low. The aim of the survey was, however, to shed more light on substantive aspects of the assessment and to formulate more suggestions for improving the reviewing process. In this respect the more qualitative comments of the reviewers who responded particularly could be used to complement and expand the results of the workshop.

The following requests for a position paper could be derived from the results of the workshop, the survey and critical reflection of the existing MOPS form: Definition of the duties of a GMS Z Med Ausbild reviewer, recommendations for the reviewing process, revision of the assessment scheme for reviewers and review of the author guidelines. The revision of the guidelines for authors is not part of this position paper.

Recommendations

Reviewer Duties

The duties of a reviewer include supporting the decisions of the editor, feedback for the author, the promotion of scientific standards and the professional scientific community and the strengthening of the national and international reputation of the journal (see also [8]). The reviewer should not only be familiar with the current literature in their area of expertise but also be confident in using scientific methods. They should dedicate about 3 hours to produce the review [9], [10].

1. Supporting the Decisions of the Editor

The reviewer helps the editor in the selection of manuscripts and suggests changes that support the authors in presenting their message to the reader in a precise and concise fashion. The editor weighs the opinions of the various reviewers, evaluates the importance and takes potential bias into account. The reviewer should therefore carefully evaluate the manuscript in question, determine its scientific quality and fit it into current literature [9].

In supporting the decision of the editor, observing certain criteria has proven useful [4], [11], [12]. These are consistent with the requirements for peer review processes in the German-speaking countries (see Figure 1 (Fig. 1)). They are divided into primary and secondary quality factors. Non-compliance with main quality criteria has to lead to immediate rejection of the article, as they constitute deficiencies that can not be corrected:

Article not important (Publication expendable, not timely, irrelevant)

Study design is inadequate (for example cross-sectional design to prove change)

Method, incl. statistics, is inadequate (for example samples too small, “distorted” sample)

In case of non-compliance with secondary quality factors, correction is needed:

Inadequate or incomplete presentation of the problem

Method inadequately described

Statistics: incomplete, poorly described

Inconsistent and inaccurate representation of the data

Data representation is not extensive enough

Over-interpretation of the results

No added value produced out for the reader (take-home message)

Text is difficult to understand or follow

Incorrect figures or tables

Inadequate, incomplete, inaccurate or outdated literature cited

The criteria should always be applied depending on the type of publication (http://www.egms.de/en/journals/zma/authors.htm).

2. Feedback for the Author

The reviewer should not only assess the manuscript in question, but should also include proposals for better presentation of the data in their review [9]. The reviewer should not write their review as a summary judgement for the editor but to give helpful hints to the author [13], [9]. In writing a review, the reviewer should ask themselves two basic questions: Have I learned something new? How can I help the author to revise the article so that I can understand their article better [7]? An important feature of the assessment process is therefore to give constructive feedback to the author about their manuscript. This feedback should identify weaknesses and strengths of the work to enable the author to maintain good aspects and to work on aspects that require improvement. Observing the following feedback rules may prove to be useful:

A summary should be at the beginning mentioning also positive aspects. This indicates appreciation and may help to reveal potential misunderstandings early on.

Giving feedback means to report back information. The reviewer should describe their own perception and to explain how something has been understood. Suitable phrases begin with “I”. The wording should be clear and unambiguous.

Positive and problematic aspects should be mentioned and it is advisable to start with the positive.

Feedback will be better accepted if it is phrased constructively and in a concrete fashion. Specific examples should be given while generalisations, in-depth analyses and interpretations, moral or general comments should be avoided.

Feedback should be open, honest and not hurtful and contain constructive suggestions for improvement.

We are aware that the assessment process is very time consuming. However, it is desirable to create a comprehensive review even if it is quite obvious early on that the work being examined is not suitable for publication (for example non-compliance with of the above mentioned primary criteria).

3. Promotion of Scientific Standards and the Professional Scientific Community

Research and science serves to gain knowledge. Research projects or even the decision not to carry out further research in a particular area are based largely on previously published data. Accordingly, the publication of research results is a responsible task. In addition, it must be remembered that the exchange experience and (untested) ideas can enrich the scholarly discussion and possibly serve as a basis for valuable research. Editors and reviewers have a role in the reviewing process that goes beyond mere quality control. By setting minimum standards, the publishability of research results, design and conduct of scientific activities in the field of medical education are further improved.

4. Strengthening of the National and International Reputation of the Journal

The GMS Z Med Ausbild journal to date is the only German-language journal with its sole focus on the didactics of medicine. This position is to be consolidated and expanded. Many - some high-quality - studies with a focus on the didactics of medicine from the German-speaking countries are being published in international journals or by specialised organs and thus escape the attention of GMA members. Making their publication in the GMS Z Med Ausbild can be improved by increasing the international reputation of the journal. This requires a review process which conforms with international standards, which enables consistently high quality of published studies. A key role of a German expert body for medical education is, after all, to strengthen the acceptance of this new discipline in German-speaking countries amongst other medical professional associations and to strengthen the role of medical education in general.

Recommendations for the Reviewing Process

1. Pre-review by the Editor

The articles are pre-reviewed by the editor according to the publication urgency, formal criteria and spelling. The editor also checks the self-specification of the type of publication and if necessary, changes the format (for example, an original work may turn into a project). They select the reviewers according to the content of the manuscript and the specialisms of the reviewer. In principle, two reviewers should be consulted. Should the assessment of both reviewers be diametrically opposed, the views of a third party should be consulted. Three weeks should be set as a guide period for the review process.

2. Writing Reviews

An accurate review contains a comprehensive assessment, which lays out how the reviewer has understood the manuscript. Furthermore, the summary, introduction, materials and methods, results, discussion, references, tables/illustrations including legends are critically evaluated in detail and suggestions for improvement made. A final conclusion follows with regard to the publishability of the manuscript. This final section will be kept confidential [9].

A proposal for a new checklist for the online review system of the GMS was developed by the authors of this paper and is attached (see Appendix 2 ). The focus of the checklist aims at the review of original papers and project reports. When reviewing other formats, it may be necessary to fall back on the item “not applicable”.

The checklist briefly covers all relevant aspects of the assessment mentioned above and has a five-point rating scale. It enables an easy way of creating a structured review based on verifiable criteria. Every individual aspect can be commented upon with free text and it thus offers an opportunity for detailed, constructive feedback. It concludes with a decision on the publication urgency. The evaluation criteria to be applied to an academic medical publication have been taken into account in developing the checklist. The checklist also includes feedback rules. It makes it possible to give feedback both to the author and the editor. The feedback to the editor may also be relevant in existing cases of a conflict of interests.

The full, completed review form (excluding the feedback to the editor) is returned to the author. This increases the transparency of the review process.

Conclusion

These recommendations are intended to support the reviewing process in the GMS Z Med Ausbild journal and to improve both the quality of reviewing as well as the quality of future submissions. The opinions and experiences of members of the committee of educational research methodology, a group of experts of the GMS Z Med Ausbild and international literature have contributed to the development of these recommendations. The improvement of the scientific quality in medical education is the future task of all members of the Society for Medical Education.

Contributions

K. Schuettpelz-Brauns conducted the workshop together with C. Kiessling and S. Schubert and wrote the introduction. A. Herrler and K. Schuettpelz-Brauns designed the questionnaire for the reviewers, conducted the survey, analysed the responses and interpreted them. A. Herrler and B. Huenges described the national situation. C. Stosch, J. Matthes and M. Himmelbauer developed the recommendations. C. Kiessling wrote the section on author’s feedback. C. Bachmann and S. Huwendiek created the new version of the checklist for reviewers of the Z Med Ausbild. K. Schuettpelz-Brauns, C. Bachmann and C. Kiessling revised the original manuscript. All authors were involved in correcting the final version of this manuscript.

Thanks

We would like to thank Sebastian Schubert and Jörg Marienhagen for reviewing the first manuscript and their constructive suggestions which led to a comprehensive review. We also thank André Schuettpelz for creating the box plots in Matlab.

Competing interests

The authors declare that they have no competing interests.