Do studies published in two leading reproduction journals between 2011 and 2020 demonstrate that they followed WHO5 recommendations for basic semen analysis?

STUDY QUESTION: Do publications that involve the interpretation of the results of a basic semen analysis, published in Human Reproduction and Fertility & Sterility between 2011 and 2020, give sufficient evidence in their methodology to demonstrate that they followed the technical methods recommended in the fifth edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen (WHO5)? SUMMARY ANSWER: Evidence of methodological agreement of studies with the WHO5 recommendations was low, despite 70% of papers stating that they followed WHO5 recommendations. WHAT IS KNOWN ALREADY: A basic semen analysis is currently an integral part of infertility investigations of the male, but method standardization in laboratories remains an issue. The different editions of the WHO manual for the basic semen analysis (WHO1-6) have attempted to address this by providing increasingly rigorous methodological protocols to reduce experimental error. However, to what extent these methods are followed by studies that involve the interpretation of the results of basic semen analysis remains unknown. STUDY DESIGN, SIZE, DURATION: A survey of the technical methods used to perform a basic semen analysis was conducted on studies published in two leading reproduction journals (Human Reproduction and Fertility & Sterility) between 2011 and 2020. PARTICIPANTS/MATERIALS, SETTING,
METHODS: The literature search was performed on the electronic databases PUBMED and MEDLINE Ovid between January 2021 and March 2021. The MeSH terms included in the search were 'sperm concentration' OR 'sperm motility' OR 'sperm morphology' OR 'sperm vitality' OR 'male fertility' AND 'human spermatozoa' NOT 'animals'. A total of 122 studies were available for analysis. MAIN RESULTS AND THE ROLE OF CHANCE: In total, 70% of the studies cited WHO5 in their methods section. Of the remaining studies, 10% cited the fourth edition of the WHO laboratory manual (WHO4), 7% cited both WHO4 and WHO5, 1% cited the third edition of the WHO laboratory manual (WHO3), and 12% did not cite the WHO at all. Overall methodological agreement with WHO5 recommendations was poor, with the main reason for this lack of agreement being that the research studies did not disclose specific details of the technical methods and equipment used. LIMITATIONS, REASONS FOR CAUTION: In the case of studies that did not disclose any specific technical methods that they used, we did not attempt to contact these authors and so were unable to confirm the agreement between their technical methods and WHO5 recommendations. WIDER IMPLICATIONS OF THE
FINDINGS: Our findings suggest there is an urgent need to develop strategies to address standardization in reporting the results of a semen analysis for publication. This is particularly timely given the recent publication of WHO6 and ISO standard 23162 for the basic examination of human semen. STUDY FUNDING/COMPETING INTEREST(S): There was no funding for this project. C.L.R.B., as an employee of the University of Dundee, serves on the Scientific Advisory board of ExSeed Health (from October 2021, financial compensation to the University of Dundee) and is a scientific consultant for Exscientia (from September 2021, financial compensation to the University of Dundee). C.L.R.B. has previously received a fee from Cooper Surgical for lectures on scientific research methods outside the submitted work (2020) and Ferring for a lecture on male reproductive health (2021). C.L.R.B. is Editor for RBMO. TRIAL REGISTRATION NUMBER: N/A.

Introduction

To date, there are six editions of the World Health Organization (WHO) andrology laboratory manual, which provide evidence-based guidance on how to perform a standardized and robust semen analysis. These methods are recommended to be used by laboratories who perform a basic semen analysis as part of clinical and/or research work. In general, the literature reports a global lack of conformance to the WHO recommendations and poor compliance with the standardized protocols for semen analysis (Björndahl ). To date, published data have focused on the conformance of laboratories who perform a semen analysis in a clinical setting and the subsequent clinical implications of a non-standardized semen analysis (Riddell ; Punjabi ; Nieschlag ; Zuvela and Matson, 2020). Currently, there is no quantification of the level of adherence of research studies, which report the results of a basic semen analysis, according to WHO recommendations. This is an important point to consider as the results of many of these studies are used to formulate policies, patient information, WHO reference ranges and recommendations for male reproductive health.

During the last decade, several authors have published concerns about the reproducibility, transparency and reliability of published results in scientific research (Ioannidis, 2005; Prinz ). These concerns have also been raised in editorials in leading scientific journals, such as Nature and Proceedings of the National Academy of Sciences (Baker, 2016; Fanelli, 2018), and covered in the wider public media (Economist, 2013; Harris, 2017). To address these concerns, various recommendations for authors and editors to follow during study design and writing have been developed. Björndahl published a checklist for researchers in Andrology, entitled ‘How to count sperm properly’, which was intended to be taken into account when authors are designing studies involving a basic semen analysis, with the completed checklist being made available in the supplementary data section. Nosek proposed in their recommendations that authors should make their methods available in a trusted repository to increase reproducibility in social and behavioural sciences, and 754 journals adopted the TOP (Transparency and Openness Promotion) guidelines, including Nature and Science journals. Other journals created their own recommendations for authors, such as the STAR (Structure, Transparent, Accessible Reporting) method in Cell Press journals, in which they ask if the methods are provided in full, explaining why and how the procedures were conducted and analysed, and if proper citations were given (https://www.cell.com/star-methods; Marcus, 2016). More recently, the Reproducibility Project: Cancer Biology (RP: CB) was undertaken to independently replicate key published findings in cancer biology, and The Materials Design Analysis Reporting (MDAR) framework was launched to focus on the transparent reporting of methods and data in life sciences publications (Macleod ). Relevant to the overall scope of this article, the RP: CB project found the availability of key methodological information to be the main reason why selected experiments were not able to be repeated. Taken together, these recommendations not only demonstrate that behind every experimental result there should be a detailed methodology, but also they stress the importance of methods standardization, transparency and reproducibility in allowing us to fully interpret experimental results, gauge their robustness, and therefore draw valid conclusions from them.

The aim of this study is to evaluate whether papers published in two leading reproduction journals (Human Reproduction and Fertility & Sterility), between 2011 and 2020, demonstrate that they followed technical methods published in the WHO5.

Materials and methods

Study design

Research papers reporting results of a basic semen analysis on human semen were retrieved from two leading journals in human reproductive biomedicine, Human Reproduction and Fertility & Sterility, between 2011 and 2020. These journals were chosen as they have the highest impact factors in reproduction and therefore hold a sentinel place in informing research and subsequent recommendations in human reproduction. Analysis focused on initial semen handling and the following semen characteristics: volume, sperm concentration, sperm motility and sperm morphology and sperm vitality. Studies that used computer-aided semen analysis (CASA) for any of the semen characteristics were excluded as the WHO does not currently recommend CASA for routine semen analysis (World Health Organization, 2022).

Literature search

The literature search was performed using PUBMED and MEDLINE Ovid. The MeSH terms included in the search were ’sperm concentration’ OR ‘sperm motility’ OR ‘sperm morphology’ OR ‘sperm vitality’ OR ‘male fertility’ AND ‘human spermatozoa’ NOT ‘animals’. Additionally, Scopus was used to identify and retrieve studies that cited the article by Björndahl , entitled ‘‘How to count sperm properly’: checklist for acceptability of studies based on human semen analysis’.

Study selection and analysis

Firstly, the titles and abstracts were assessed through screening (n = 2384), then the exclusion criteria (reviews, meta-analysis, surveys, opinions, factors related to female infertility, such as premature ovarian insufficiency, tubal patency and polycystic ovary syndrome, animal models and articles not written in English) were applied (n = 843). Only papers published in the two leading journals in human reproductive biomedicine, Fertility & Sterility and Human Reproduction, between 2011 and 2020, were included in this study. A total of 122 eligible studies were selected and compared for analysis in this study (Fig. 1).

Figure 1.

Flow chart of study selection according to PRISMA checklist.

Results

Overall, 70% of the studies cited WHO5. Of the remaining studies, 10% cited the WHO4 manual, 7% cited both WHO4 and WHO5, 1% cited the WHO3 and 12% did not cite the WHO at all. The majority of studies did not fully describe WHO-recommended technical methods for each of the semen characteristics (Fig. 2). For sperm concentration, motility, morphology and vitality, the majority of studies did not disclose technical information for any of the steps recommended by WHO5 (Tables I–IV). Similar findings were seen for initial semen handling and sperm volume, with the exception of WHO5 recommendation that patients maintain 2–7 days of sexual abstinence before collecting a sample for investigation (Table V). Of the papers published since 2016 that also cited the Björndahl checklist, the majority reported technical methods not recommended by WHO5 for sperm concentration, morphology and vitality, whereas the majority reported WHO5 recommendations for motility and volume assessments (Fig. 3). There was no obvious trend in the number of papers citing WHO5 annually between 2011 and 2020 (Fig. 4).

Figure 2.

Overall reporting of the WHO5 recommended technical method for each semen characteristic among the 122 studies. All 122 included studies were published in Human Reproduction and Fertility & Sterility between 2011 and 2020. WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Table I

Percentage of studies that report the different steps of the WHO5 technical method for measurement of sperm concentration.

	Yes (%)	No (%)	Not disclosed (%)
Semen aliquot to be diluted for sperm concentration assessment was taken with a positive displacement pipette (i.e. a ‘PCR pipette’) using a recommended diluent.	8	0	92
Only standard dilutions were used (1:10, 1:20, 1:50).	8	0	92
Sperm concentration was assessed using haemocytometers with improved Neubauer ruling.	14	14	72
Haemocytometers were allowed to rest for 10–15 min in a humid chamber to allow sedimentation of the suspended spermatozoa onto the counting grid before counting.	6	0	94
Sperm counting was done using phase contrast microscope optics (200–400×).	10	0	90
Comparisons were made between duplicate count, and counts re-done when the difference exceeded the acceptance limits.	5	0	95
Typically at least 200 spermatozoa were counted in each of the duplicate assessments.	12	1	87

Yes means that the study stated the step in their methods section; no means that the study stated an incorrect version of that step in their methods section; not disclosed means that they did not mention the step in the article. Only studies that performed the parameter were included in the analysis (n = 114).

WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Table V

Percentage of studies that report the different steps of the WHO5 technical method for initial semen handling and measurement of semen volume.

	Yes (%)	No (%)	Not disclosed (%)
Patients were instructed to maintain 2–7 days of sexual abstinence before collecting a sample for investigation.	61	0	39
For specimens not collected at the laboratory, patients were instructed to avoid cooling or heating of the semen sample during transport to the laboratory.	7	0	93
Samples were kept at 37.0°C before initiation of and during the analysis in case of sperm motility assessment.	11	1	88
For samples collected adjacent to the laboratory, analysis was initiated after completion of liquefaction and within 30 min after ejaculation. If this was not done—and more importantly when some of the samples are collected in the laboratory and others are collected at home—it should be checked that this did not influence the data (and, if yes, that this effect must be included as a confounding factor in the statistical analysis).	11	0	89
Liquefaction was first checked within 30 min after ejaculation.	26	10	64
Sperm volume was determined either by weighing or using a wide-bore volumetric pipette.	10	2	88
Viscosity was measured using either a wide-bore pipette or a glass rod.	6	0	94
All staff members who performed the analyses have been trained in basic semen analysis (ESHRE Basic Semen Analysis Course—or equivalent—and further in-house training) and participate regularly in internal quality control.	8	1	91

Yes means that the study stated the step in their methods section; no means that the study stated an incorrect version of that step in their methods section; not disclosed means that they did not mention the step in the article.

WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Figure 3.

Overall reporting of the WHO5 recommended technical methods for each semen characteristic in studies that also cite the WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Figure 4.

Number of WHO5 citations per year of publishing among the 122 studies. All 122 included studies were published in Human Reproduction and Fertility & Sterility between 2011 and 2020. WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen. n = 14 studies did not mention any WHO manual.

Discussion

A basic semen analysis is currently the cornerstone of male fertility investigation. Since its first publication, the WHO laboratory manual has continued to provide up-to-date evidence-based recommendations to enable andrology laboratories—both research and clinical—to perform a standardized and robust semen analysis. In this study, we assessed the technical methods of studies published in Human Reproduction and Fertility & Sterility, between 2011 and 2020, for evidence of conformance to technical methods detailed in WHO5. In general, we found most papers to provide insufficient evidence to demonstrate that they followed these recommendations, with some papers using equipment and protocols not recommended by the manual.

Designing a study using standardized technical methods and writing a detailed methods section in associated research outputs, is fundamental to science as it builds confidence and assurance in research findings, enabling scientific consensus to emerge and ultimately science to progress. This is essential when designing a study that involves performing and interpreting the results of basic semen analysis, as a human semen sample is inherently heterogeneous, leading to uncertainty in the result of the different semen characteristics (Tomlinson, 2016). Indeed, the different editions of the WHO manual have addressed this by providing increasingly rigorous methodological protocols to reduce experimental error, and by limiting the number of recommended techniques for the different semen characteristics, with WHO6 recommending one methodological protocol for each semen characteristic. Furthermore, the ESHRE Basic Semen Analysis Course has demonstrated long-term effectiveness in raising the theoretical knowledge and practical capabilities of laboratory personnel in semen analysis (Björndahl ). Although we acknowledge the effectiveness of these initiatives it is clear from our results that more can be done. Lessons can certainly be learned from tools provided by other journals that aim to ensure transparent reporting of technical methods to increase data reproducibility. For instance, life sciences journals published as a part of Cell Press journals have replaced the traditional formatting of the methods section with STAR Methods (Structured, Transparent and Accessible Reporting). Although the semen analysis checklist provided by Björndahl recommends a similar format, it is not a mandatory requirement for publication in any of the reproduction-focused journals and therefore one way to enhance the transparency of semen analysis method reporting could be to adopt a similar approach to that of the Cell Press journals. Our findings herein support the potential benefits of reproduction journals to adopt a checklist for semen analysis results as we found that the majority of papers that cited the Björndahl checklist reported WHO5 recommended methods for sperm motility and semen volume assessments.

There are many other ways that we can encourage authors to be more transparent with their technical methods (see Vasconcelos ). Several foundations (i.e. Netherlands Organization for Scientific Research) have announced funding specifically for replication studies, and many journals have adopted the TOP guidelines, including Nature and Science. Peterson and Panofsky (2021) suggested that reforms should be made in each specific scientific field and in collaboration with scientific societies to increase adherence and reduce the burden of bureaucracy, in order to pave the way for more robust scientific research. Therefore, we encourage our professional societies (i.e. ESHRE, American Society for Reproductive Medicine, European Academy of Andrology and The International Society of Andrology) to consider adopting these, or similar ideas, as this will help foster confidence in existing published data. Another way to increase standardization would be through international collaborative networks and the creation of video tutorials and/or training materials to walk through the protocols for each semen characteristic (Vasconcelos ). These could be embedded in social media platforms to normalize semen analysis and make information more accessible and the methods easier to follow.

The data presented here suggest there is an urgent need to address standardization in reporting semen analysis for publication. New tools and/or the adoption of methods used in other disciplines are required. Investigators, authors, referees and editors need collectively to address these challenges. Undoubtedly, recent publication of the sixth edition of the WHO manual with accompanying ISO standards (ISO, 2021) will help transparency but, based on the experience so far, mandatory systems will be required to ensure high-quality scientific publications.

Table II

Percentage of studies that report the different steps of the WHO5 technical method for measurement of sperm motility.

	Yes (%)	No (%)	Not disclosed (%)
Motility assessments were performed at 37.0°C ± 0.5°C.	9	0	91
Motility assessments were done using phase contrast microscope optics (200–400×).	11	1	88
Sperm motility was assessed using the three-category scheme (progressive, non-progressive and immotile).	21	17	62
Motility assessments were done in duplicate and compared; counts were re-done on new preparations when the difference between duplicates exceeded the acceptance limits.	11	0	89
The wet preparation was made with a drop of _____ ml and a ____ × ____ mm coverslip to give a depth of _____ mm (must be at least 10 mm, but not too deep so as to allow spermatozoa to move freely in and out of focus; typically ca. 20 mm).	9	0	91
At least 200 spermatozoa were assessed in each duplicate motility count.	16	2	82
At least five microscope fields of view were examined in each duplicate count.	10	0	90

Yes means that the study stated the step in their methods section; no means that the study stated an incorrect version of that step in their methods section; not disclosed means that they did not mention the step in the article. Only studies that performed the parameter were included in the analysis (n = 117).

WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Table III

Percentage of studies that report the different steps of the WHO5 technical method for measurement of sperm morphology.

	Yes (%)	No (%)	Not disclosed (%)
Tygerberg Strict Criteria were used for the evaluation of human sperm morphology. Note: Another classification could be used for scientific studies with specific aims if the classification is described or referenced. Depending on the aim of the study, the evaluation of particular abnormal forms might be useful.	18	10	72
Abnormalities are recorded for all four regions of the spermatozoon (head, neck/midpiece, tail, and cytoplasmic residue) and the Teratozoospermia Index or ‘TZI’ was calculated (Barratt et al., 2011).	7	3	90
If the laboratory claims to use Tygerberg Strict Criteria for the evaluation of human sperm morphology, then the laboratory must participate in an external quality assurance scheme to verify that its assessments comply with these criteria.	3	3	94
The Papanicolaou staining method adapted for the assessment of human sperm morphology was used. For specific aims, other staining methods could be used, but must then be declared and explained.	12	13	75
At least 200 spermatozoa were assessed in each ejaculate.	11	2	87
Morphology assessments done in duplicates using independent technicians.	5	0	95
Assessments were done under high magnification (×1000–1250) and bright field microscope optics (Köhler illumination).	8	2	90

Yes means that the study stated the step in their methods section; no means that the study stated an incorrect version of that step in their methods section; not disclosed means that they did not mention the step in the article. Only studies that performed the parameter were included in the analysis (n = 109).

WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.

Table IV

Percentage of studies that report the different steps of the WHO5 technical method for measurement of sperm vitality.

	Yes (%)	No (%)	Not disclosed (%)
A validated supravital staining, appropriate to the type of microscope optics utilized, was used to assess sperm vitality.	7	0	93
At least 200 spermatozoa were evaluated in each sample.	11	0	89
Assessments were done under high magnification (×1000–1250) and bright field microscope optics (Köhler illumination).	6	0	94

Yes means that the study stated the step in their methods section; no means that the study stated an incorrect version of that step in their methods section; not disclosed means that they did not mention the step in the article. Only studies that performed the parameter were included in the analysis (n = 101).

WHO5: the 5th edition of the World Health Organization (WHO) laboratory manual, entitled WHO Laboratory Manual for the Examination and Processing of Human Semen.