Royal College of Obstetricians and Gynaecologists guidelines: how evidence-based are they?

Evidence-based medicine aims to translate scientific research into good medical practice. The Royal College of Obstetricians and Gynaecologists publishes recommendations and guidelines to guide clinicians in decision-making. In this study, the evidence base underlying the 'Green-top Guidelines' has been analysed in order to establish the quality of research underlying recommendations. During this descriptive study of 1,682 individual recommendations, the authors found that only 9-12% of the guidelines were based on the best quality (Grade A) evidence. The authors believe that this type of analysis serves to provide greater clarity for clinicians and patients using guidelines and recommendations in the field of obstetrics and gynaecology to make collaborative clinical decisions.

Introduction

The purpose of evidence-based medicine is to apply the best scientific evidence to clinical decision-making. In order to guide doctors and health professionals, the Royal Colleges publish guidelines and recommendations for good clinical practice within their respective specialties.

However, guidelines have limitations. Critics (Wright 2007) say that they are unsuited to address flexibility and choice expected by patients. Browman (2010) argues that it is tempting to develop rigid protocols but patient treatment should be individualised. However, in the area of acute obstetrics and gynaecology, where management of life-threatening conditions is unsuitable for randomised controlled trials, guidelines are heavily relied upon, to ensure consistent quality of care. Many are based on recommendations which have little or no scientific background and rely more on clinical or expert opinion, which is particularly susceptible to bias (Detsky 1924). Wright (2007) argues that while the evidence-based medicine movement was intended to add evidential basis to guideline recommendations, it only resulted in a moderate improvement. Wright is also sceptical of the system of grading recommendations according to evidence (Grade A, Grade B, etc.), as some readers of guidelines may not consult the type of evidence underlying each grade. This has led analysts of evidence-based medicine to claim that this is a misnomer and it is actually better referred to as ‘eminence-based medicine’ (Charles et al. 2011).

There have been no previous analyses of the overall evidence underlying the Royal College of Obstetricians and Gynaecologists (RCOG) guidelines. However, a study by Wright in 2011 found that only one-third of the recommendations by the American College of Obstetricians and Gynecologists were based on good scientific evidence (Wright et al. 2011). In similar studies, this was the case in 14% of the recommendations by the Infectious Diseases Society of America (Lee and Vielemeyer 2011), and 45% of cardiovascular risk recommendations from guidelines across the USA, Canada and Europe (McAlister et al. 2007).

The objective of this study was to analyse the evidence used in guidelines published by the RCOG and to identify areas where evidence is high-quality, adequate or lacking. It is important to be aware of the standard of evidence which underpins the advice which doctors access for the diagnosis and management of their patients.

Materials and methods

The Royal College of Obstetricians and Gynaecologists produces a large number of guidelines for the management and diagnosis of a wide variety of diseases and conditions. The College produces different categories of advice: ‘Consent Advice’, ‘Good Practice’, ‘Joint Guidelines’, ‘National Evidence Based Guidelines’ and ‘Project Reports’, among others. For the purpose of this analysis, we used the ‘Green-top Guidelines’. Each Green-top Guideline addresses a specific topic, giving a brief introduction to the subject in question and detailing how the supporting evidence was obtained. For the guidelines included, see Table I. The guidelines themselves include a series of recommendations, each graded by the overall quality of supporting evidence, the details of which are included in Table II. In addition, the studies and trials cited in support of the recommendations made were further classified using a numerical system, which has not been included for simplification. The method of grading of recommendations and classification of evidence was changed in December 2007. For this reason, guidelines created before that date and guidelines created after that date were analysed separately to avoid the introduction of any bias or subjectivity into classification.

Table I.

The Green-top Guidelines analysed.

	Guideline number	Date published
Amniocentesis and Chorionic Villus Sampling	8	Jun 2010
Antenatal Corticosteroids to Reduce Neonatal Morbidity	7	Oct 2010
Antepartum Haemorrhage	68	Dec 2011
Birth After Previous Caesarean Birth	45	Feb 2007
Blood Transfusions in Obstetrics	47	Dec 2007
Breast Cancer and Pregnancy	12	Apr 2011
Breech Presentation, Management	20b	Dec 2006
Cervical Cerclage	60	May 2011
Chickenpox in Pregnancy	1	Sep 2007
Chronic Pelvic Pain, Initial Management	41	Apr 2005
Early Pregnancy Loss, Management	25	Oct 2006
Endometriosis, Investigation and Management	24	Oct 2006
External Cephalic Version (ECV) and Reducing the Incidence of Breech Presentation	20a	Dec 2006
Female Genital Mutilation Management	53	May 2009
Genital Herpes in Pregnancy, Management	30	Sep 2007
Gestational Trophoblastic Disease	38	Feb 2010
Group B Streptococcal Disease, Early Onset	36	Nov 2003
HIV in Pregnancy, Management	39	Jun 2010
Hormone Replacement Therapy and Venous Thromboembolism	19	May 2011
Hysteroscopy, Best Practice in Outpatient	59	Apr 2011
Laparoscopic Injuries	49	May 2008
Late Intrauterine Fetal Death and Stillbirth	55	Nov 2010
Malaria in Pregnancy Diagnosis and Treatment	54b	Apr 2010
Malaria in Pregnancy Prevention	54a	Apr 2010
Maternal Collapse in Pregnancy and the Puerperium	56	Feb 2011
Monochorionic Twin Pregnancy, Management	51	Dec 2008
Obstetric Cholestasis	43	Apr 2011
Operative Vaginal Delivery	26	Feb 2011
Ovarian Cysts in Postmenopausal Women	34	Oct 2003
Ovarian Hyperstimulation Syndrome, Management	5	Sep 2006
Ovarian Masses in Premenopausal Women: Management of Suspected	62	Dec 2011
Pelvic Inflammatory Disease	32	Nov 2008
Placenta Praevia and Placenta Praevia Accreta: Diagnosis and Management	27	Jan 2011
Polycystic Ovary Syndrome, Long-Term Consequences	33	Dec 2007
Postpartum Haemorrhage, Prevention and Management	52	May 2009
Premenstrual Syndrome, Management	48	Dec 2007
Preterm Labour, Tocolytic Drugs	1b	Feb 2011
Preterm Prelabour Rupture of Membranes	44	Nov 2006
Recurrent Miscarriage, Investigation and Treatment of Couples	17	Apr 2011
Reduced Fetal Movements	57	Feb 2011
Rh Prophylaxis, Anti-D Immunoglobulin	22	Mar 2011
Shoulder dystocia	42	Mar 2012
Sickle Cell Disease in Pregnancy, Management	61	Aug 2011
Third- and Fourth-degree Perineal Tears, Management	29	Mar 2007
Thromboembolic Disease in Pregnancy and the Puerperium: Acute Management	37b	Feb 2007
Thrombosis and Embolism during Pregnancy and the Puerperium, Reducing the Risk	37a	Nov 2009
Tubal Pregnancy, Management	21	May 2004
Umbilical Cord Prolapse	50	Apr 2008
Vaginal Vault Prolapse	46	Oct 2007
Venous Thromboembolism and Hormonal Contraception	40	Sep 2010
Venous Thromboembolism and Hormone Replacement Therapy	19	May 2011
Vulval Skin Disorders, Management	58	Feb 2011

Table II.

The classification scheme for the recommendations by the Royal College of Obstetricians and Gynaecologists.

	Classification scheme for recommendations by the Royal College of Obstetricians and Gynaecologists
Pre-December 2007
A	At least one randomised controlled trial as part of a body of literature of overall good quality and consistency addressing the specific recommendation.
B	Requires the availability of well controlled clinical studies but no randomised clinical trials on the topic of recommendations.
C	Requires evidence obtained from expert committee reports or opinions and/or clinical experiences or respected authorities. Indicates an absence of directly applicable clinical studies of good quality.
D	Recommended best practice based on the clinical experience of the guideline development group.
Post-December 2007
A	At least one meta-analysis, systematic review or randomised controlled trial rated as 1++ directly applicable to the target population and demonstrating overall consistency of results. A systematic review of randomised controlled trials or a body of evidence consisting principally of studies rated as 1 + directly applicable to the target population and demonstrating overall consistency of results.
B	A body of evidence including studies rated as 2++ directly applicable to the target population, and demonstrating overall consistency of results. Extrapolated evidence from studies rated as 1++ or 1+.
C	A body of evidence including studies rated as 2 + directly applicable to the target population and demonstrating overall consistency of results. Extrapolated evidence from studies rated as 2++.
D	Evidence level 3 or 4. Extrapolated evidence from studies rated as 2+.
E	Recommended best practice based on the clinical experience of the guideline development group.

All 52 ‘Green-top Guidelines’ were obtained from the Royal College of Obstetricians and Gynaecologists’ website on 1 April 2012. One guideline (Green-top 31, ‘Small-for-Gestational-Age Fetus, Investigation and Management’, published on 1 November 2002) was not included in the analysis, as its evidence levels were classified using a different system from the others. Two guidelines have been archived by the RCOG (Green-tops 10A and 35), with advice to refer to the appropriate National Institute of Clinical Excellence (NICE) guidelines for assistance, and therefore could not be obtained or included in analysis.

RCOG Green-top guidelines published before December 2007 are classified according to one of four levels of evidence: A (literature of overall good quality and evidence); B (well controlled clinical studies); C (evidence from expert committee reports or opinions) and √ (recommended best practice based on the clinical experience of the guideline development group). Those issued from December 2007 were classified using a five-tiered system: A (meta-analysis, systematic review or a good quality randomised controlled trial); B (high quality systematic reviews of case–control or cohort studies); C (high quality case–control or cohort studies with a low risk of confounding); D (non- analytical studies, such as case reports/expert opinion) and √ (recommended best practice based on the clinical experience of the guideline group).

We gathered the grades of each recommendation within each guideline, and then stratified each guideline into those primarily focussed on obstetrics and those primarily focussed on gynaecology. We then sub-divided the latter into the following: General Gynaecology, Oncology, Fertility and Endoscopy. In addition to this, we sorted the guidelines based on whether they referred to Evaluation, Diagnosis and Treatment, or Mode of Delivery (Table III). This was carried out independently by three reviewers; categorisations were then discussed and any discrepancies were resolved by consensus (majority opinion of all the reviewers). Descriptive statistics were then used to report the findings of this analysis. The Research Management and Governance Department at the Whittington Hospital, UK, agreed that the project did not require NHS ethics approval.

Table III.

The classification scheme for the stratification of guidelines into topics.

	Classification scheme for the stratification of guidelines into topics
Guideline specialty grouping
Obstetrics
Gynaecology	General Gynaecology
	Oncology
	Fertility
	Endoscopy
Guideline classification scheme
Diagnosis and treatment	Recommendations for the diagnostic evaluation in patients without a known abnormal condition, and for the specific interventions for the diagnosed condition.
Evaluation	Recommendation for evaluation, monitoring, and follow-up of patients with a known abnormal condition.
Mode of delivery	Recommendations specifically tailored to directing for a particular mode of delivery (obstetrics guidelines only).

Results

A total of 52 guidelines that offered 1,682 individual recommendations were studied (Table I); 32 of the guidelines were obstetric (61.5%) and 20 were gynaecological (38.5%). Within these guidelines, 1,160 (69%) of the individual recommendations were obstetric and 522 (31%) were gynaecological. Of the recommendations published before December 2007, 52 (12%) referenced level A evidence; 94 (22%) level B evidence; 126 (29%) level C evidence; and 163 (37%) were based on recommended best practice (Figure 1a). Regarding guidelines published from December 2007 onwards, 114 (9%) were based on level A evidence; 145 (12%) level B; 210 (17%) level C; 276 (22%) level D and 502 (40%) were recommended best practice (Figure 1b). This suggests that, contrary to the aims of evidence-based medicine, guidelines published more recently are actually more likely to be based on clinical experience alone. However, since the new classification system places more stringent measures on evidence that can qualify as level A, the two sub-sets cannot be directly compared.

Figure 1.

Classification of evidence levels underlying RCOG guidelines published (a) before and (b) after December 2007.

Within the two classification systems (that used prior to December 2007 and that used after), the distribution of quality of evidence was divided into that supporting obstetric guidelines and that supporting gynaecological guidelines. Among the obstetric recommendations published under the old system, 22 (8%) provided level A evidence; 48 (18%) level B; 89 (33%) level C and 111 (41%) were recommended best practice (Figure 2a). Of those published after December 2007, using the new five-tiered system, 69 (8%) provided level A evidence; 93 (10%) level B; 147 (16%) level C; 229 (26%) level D and 352 (40%) were recommended best practice (Figure 2b). This suggests there has been no significant change in the quality of sources of evidence used by the RCOG to write obstetric guidelines over the past few years, and that the greater reliance of more recently published guidelines by the RCOG on clinical experience alone is due to the newer gynaecological rather than obstetric guidelines.

Figure 2.

Classification of evidence underlying Obstetrics guidelines published (a) before and (b) after December 2007.

Among the gynaecological recommendations published before December 2007, 30 (18%) were based on level A evidence; 46 (28%) level B; 37 (22%) level C and 52 (32%) were recommended best practice (Figure 3a). In those published post-December 2007, 45 (13%) provided level A evidence; 52 (14%) level B; 63 (18%) level C; 47 (13%) level D and 150 (42%) were recommended best practice (Figure 3b). This demonstrates that across all guidelines (regardless of publication date), those concerned with gynaecology had a higher quality of underlying evidence than obstetric guidelines, with greater numbers based on the findings of meta-analyses and well-conducted randomised controlled trials.

Figure 3.

Classification of evidence underlying Gynaecology guidelines published (a) before and (b) after December 2007.

The recommendations were then further stratified by our classification scheme. Within obstetrics, 68.8% of recommendations addressed diagnosis and treatment; 12.5% evaluation and 18.8% mode of delivery. Within gynaecology, 85% of recommendations addressed diagnosis and treatment and 15% evaluation. Once again, those published under each of the two classification systems had to be evaluated separately. Of the obstetric guidelines published under the old system, 5.6% of guidelines concerning evaluation were based on level A treatment, compared with 6.4% of diagnosis and treatment, and 12.5% of mode of delivery recommendations (Figure 4a). Of the newer obstetric guidelines, 6.4% of evaluation, 8.1% of diagnosis and treatment and 7.4% of mode of delivery recommendations were based on level A evidence (Figure 4b).

Figure 4.

Distribution of quality of evidence underlying obstetric recommendations stratified by the type of guideline, published (a) before and (b) after December 2007.

Of the older gynaecological guidelines, 14.3% of evaluative and 18.8% of diagnostic and treatment recommendations were based on level A evidence (Figure 5a), compared with 0% of evaluative and 15.1% of diagnostic and treatment recommendations published under the new classification system (Figure 5b).

Figure 5.

Distribution of quality of evidence underlying gynaecology recommendations stratified by the type of guideline, published (a) before and (b) after December 2007.

The 20 gynaecology guidelines were further categorised by subspecialty; the majority (70%) addressed general gynaecology issues; 20% were concerned with fertility; 5% with oncology and 5% with endoscopy. The results showed that 10.8% of fertility guidelines published under the old system and 20.8% of those published more recently, were based on level A evidence, compared with 24.2% and 7.3% of general gynaecology guidelines published before and after December 2007, respectively. There was only one guideline related to gynaecological oncology, for which none of the evidence was level A. Conversely, the only endoscopy guideline was based on evidence, of which 42.3% was level A (Figure 6).

Figure 6.

Quality of evidence underlying gynaecological subspecialty guidelines, published (a) before and (b) after December 2007.

Overall, the results suggest that the majority of supportive evidence behind the RCOG guidelines is based on clinical expertise or studies defined as low quality, and that this does not appear to be changing with new guidelines. Gynaecological guidelines seem to generally have a higher quality evidence base than obstetric recommendations.

Discussion

The findings of this guideline analysis suggest that on average across the categories, fewer than 20% of recommendations by the RCOG were based on high quality evidence, with a large proportion based on ‘recommended best practice’ and expert opinion. We go on to question why this is so and what this means in an era of evidence-based practice.

The ‘Green-top Guidelines’ reviewed in this study are written and developed by the RCOG to provide systematic recommendations in the hope of assisting clinicians and patients in clinical decision-making. Guideline topics are selected through a process coordinated by the Guidelines and Audit Committee, made up of clinicians, with input from the RCOG Consumers’ Forum. Guidelines are developed by collating evidence and making value judgements by consensus. They must then meet the Appraisal of Guidelines for Research and Evaluation in Europe (AGREE) criteria and are subsequently peer reviewed in a formal, open process. The system used by the RCOG to grade the evidence quality is based on the scoring system developed by the ‘GRADE Working Group’ (Balsham et al. 2011). This is intended to introduce further rigour into the classification of evidence quality underlying guidelines. However, there is also concern that such a simplistic representation of evidence (stating quality as A, B or C) might not communicate an accurate enough interpretation of quality (Charles et al. 2011). Recommendations are not intended to be used exclusively to decide upon a course of management or treatment; instead the clinician’s judgement should also give equal consideration to individual patient needs and the variation of recourses between institutions (RCOG 2012).

The recent Institute of Medicine report in the USA on the development of guidelines and their worth in modern practice highlights the fact that guidelines can be flawed in their formation; such flaws may be due to a lack of transparency in how they are created and rated, and an absence of rigorous external review. They suggest that authors of guideline recommendations should provide a summary of the quality of the evidence they have used (Institute of Medicine, no date).

Research in obstetrics and gynaecology, as in other fields (Lee and Vielemeyer 2011; McAlister et al. 2007) is clearly lacking in randomised controlled trials. This can be put down to several factors. The acute nature of the specialty means there are ethical and practical difficulties that make trial design and implementation impossible (Lee and Vielemeyer 2011). Some have postulated that practice based on such ‘real-life’ observational evidence might be of more value in obstetrics than randomised control trials (Vintzileos 2009), however, this evidence would be prone to error and bias. In addition, only a minority of obstetricians conduct a literature search when presented with a clinical dilemma, and personal experience and views of experts still have a far-reaching influence over obstetric practice (Olatunbosun et al. 1998).

It is also important to acknowledge the uniqueness of obstetric practice in relation to its partner, midwifery. The ancient tradition and long history of midwifery has its own body of expert opinion, based on historical and research experience, and The Royal College of Midwives has its own set of online guidelines, which does not use the GRADE system.

Within the subspecialties of obstetrics and gynaecology, there is a difference in quality of supportive evidence; endoscopy, a relatively new intervention in this field, would require evidence to prove its value and would have the financial backing needed to provide this – the sole endoscopy guideline was based on 42.3% level A evidence. However, some Grade C recommendations for established practice can be very useful in a ‘common sense’ way. For example, that low risk postpartum women do not need thromboprophylaxis (RCOG 2009). Even fundamental corner-stones of clinical practice, such as partograms, are not based on strong evidence. A review of the use of partograms on birth outcomes conducted by the Cochrane Collaboration concluded that they could not recommend the routine use of the partogram as part of standard labour management (Lavender et al. 2008). Zhang (2002) also found that the average labour of nulliparous women varies markedly from the Friedman curve during the first stage of labour, and as such, it is possible that interventions for slow progression may come too early due to overly strict guidelines (Zhang et al. 2002). Yet the partogram remains an important foundation of intrapartum care in the UK. Even when there is supposedly good quality evidence underpinning guidelines, this is sometimes flawed – a case in point is the issue of caesarean delivery of breech presentations. The publication of the Term Breech Trial (Hannah 2000), a multicentre randomised trial, implied that it was safer to perform a caesarean section at birth for all breech infants, and as such, it was incorporated into guidelines in the USA, UK and Canada. However, a post hoc inquiry into this trial’s methodology has revealed inconsistencies and errors in data collection and analysis, which make the study’s findings unreliable (Kotaska 2011).

We accept that there are several limitations to this study. The categorisation system developed was designed to assist us in providing meaningful conclusions. However, the introduction of such rigid parameters may be detrimental to the accuracy of the analysis. Naturally this system is dependent on the reviewer, and other researchers may differ in their approach to classification. We have modelled our analysis on the study published by Wright et al. (2011) to facilitate meaningful comparison between the two papers.

This analysis has revealed that the majority of guideline recommendations were not underpinned by high quality evidence. The same sort of discourse is occurring in the basic sciences, as in the New Scientist article: ‘Is medical science built on shaky foundations?’, which raises the point that more than half of biomedical findings cannot be reproduced (Iorns, 2012). One could ponder the impact of these conclusions that, in both the UK and USA, the majority of practice guidelines are not founded on high quality evidence, and this is reflected in clinical practice, in science and medicine across the board. We should not dismiss the limitations of the RCOG or ACOG recommendations, but acknowledge them in order to neutralise any assumptions made by clinician or patient that recommendations are based on high quality evidence.

All medical colleges could offer this form of descriptive analytic summary of the quality of evidence their guidelines are based on. This would provide greater clarity for clinicians as to the core scientific foundations of their clinical practice, as well as serving to aid transparency of speciality knowledge in the clinician–patient partnership of decision-making.