Incorporation of randomized controlled trials into organizational guidelines for obstetricians and gynecologists.

BACKGROUND: The American College of Obstetricians and Gynecologists publishes practice bulletins and committee opinions to serve as clinical guidelines for physicians. The objective of this study was to quantify the frequency that randomized controlled trials become incorporated into the American College of Obstetricians and Gynecologists documents (either practice bulletins or committee opinions).
METHODS: Original research articles published in The American Journal of Obstetrics and Gynecology, The Journal of the American Medical Association, The New England Journal of Medicine, and Obstetrics and Gynecology between 2009 and 2014 were examined and randomized controlled trials (RCT) in obstetrics and gynecology were identified. Adjusted odds ratio (aOR) with 95% confidence intervals (CI) were calculated to examine the factors associated with a citable RCT being referenced versus not in ACOG documents.
RESULTS: Of the 306 randomized controlled trials identified 248 (81.0%) met the inclusion criteria, with 128 (51.6%) of eligible RCT being cited The factors which increased the likelihood of a RCT being referenced, versus not being, were: if device or surgery was the intervention (aOR 3.60; 95% CI 1.85-7.00) and if the sample size of the trial was 500-999 (aOR 3.70 (1.39-9.82). The following factors were not associated with whether the RCT was or was not referenced in the ACOG documents: topic was obstetric or gynecologic, the trial was conducted in the US or abroad, multi- or single center, year of publication and the journal.
CONCLUSION: Since about half of the citable randomized controlled trials published in obstetrics and gynecology are incorporated into ACOG practice bulletins and committee opinions a greater transparency is warranted as to why RCTs are or are not referenced.

Introduction

To improve the quality of care and implementation of best practices, the American College of Obstetricians and Gynecologists (ACOG) publishes clinical guidelines for physicians [1]. Practice bulletins and committee opinions are the most utilized guidelines by obstetricians and gynecologists in the United States and have been shown to influence patient care [2], [3] Within these guidelines, ACOG produces recommendations with various levels of confidence: Level A (good or consistent evidence), Level B (limited or inconsistent evidence) or Level C (consensus or opinion). Less than one third of the recommendations published in ACOG practice bulletins are level A recommendations [4]. Of the level A recommendations less than 30% were supported by evidence from randomized controlled trials (RCTs) [5].

One explanation for the infrequent citations of RCTs is that there is comparatively low amount of research funding in women’s health, and the resulting comparative lack of high-quality trials when compared to other specialties [6], [7]. Another possibility is that there is a gap in ACOG’s ability to incorporate RCTs into the ACOG guidelines. The objective of this study was to examine the frequency with which published RCTs in obstetrics and gynecology were cited in ACOG practice bulletins or committee opinions.

Methods

This retrospective review was considered exempt by the Institutional Review Board at the McGovern School of Medicine. For the years of 2009–2014, four prominent publications (New England Journal of Medicine, Journal of the American Medical Association, American Journal of Obstetrics and Gynecology, Obstetrics and Gynecology) were queried. Abstracts for all publications were analyzed. All original research in obstetrics and gynecology which reported randomized controlled trial methodology were abstracted for analysis (Supplement A). Trial design was abstracted by two reviewers (SW, GO) and trial results by two (RG, GO). Trials were stratified into obstetrics or gynecological in focus (RG, GO). Trials that did not examine clinical care were excluded. STROBE guidelines for reporting observational studies were followed [8].

In 2020, the ACOG website was queried for all practice bulletins and committee opinions available to ACOG members. Individual trials were cross referenced to practice bulletins and committee opinions topics based on keywords. Practice bulletins and committee opinions were “hand reviewed” and inspected for citation of a randomized controlled trial by reviewing document references. The ACOG website search function was then utilized with study title and first author name to ensure all citations of that study in bulletins or opinions were captured. This was completed independently by at least two reviewers for each study (RG, GO, SW). Disagreements between two reviewers were resolved in discussion with the third reviewer. For example, in Labrie et al. [9] Surgery versus Physiotherapy for Stress Urinary Incontinence, keywords of “physiotherapy” and “stress urinary incontinence” were used. Practice bulletin 155 “Urinary Incontinence in Women” was identified and the study was found in its references. It was therefore considered incorporated in the ACOG guidelines. “Labrie” and “Surgery versus Physiotherapy for Stress Urinary Incontinence” were then put in the ACOG website search function to capture any other citations of the study in bulletins or opinions.

Due to the possibility of trials being incorporated in systematic reviews that would be cited, Cochrane reviews that were cited in the practice bulletins and committee opinions were further inspected to see if the trials were included in the analysis. Utilizing the search function on the Cochrane website (www.cochrane.org) study titles were searched in the cited reviews. If the study was in the Cochrane review that was cited in the ACOG educational publication, then RCT was considered cited in the guideline. This abstraction and analysis of the Cochrane reviews was accomplished by two reviewers (RG, GO).

Comparisons between categorical variables were made using chi-square and Fisher’s exact tests and results are reported using P-values. Logistic regression was used to analyze the odds of citation for each predictor. Multivariate logistic regression was used to control for any variables that were significant in univariate analysis at a P-value of < 0.05. Results of the logistic regression are expressed as odds ratios (OR) with 95% confidence intervals (95% CI). All statistics were performed using Stata/MP 16.1 (StataCorp, LLC).

Results

Over the six-year study period (2009–2014), in the four journals there were 4872 publications and among them there were 306 (5.9%) RCTs on obstetrical or gynecological topics. After excluding studies that were not pertinent to the subject of a practice bulletin or committee opinion (n = 41), or whose subject matter had not been updated since publication of the study (n = 17), 248 eligible trials remained, with 140 trials being on gynecological topics and 108 on obstetrical (Fig. 1).

Fig. 1

Flow diagram of randomized controlled trials in obstetrics and gynecology included in the analysis.

Eligible studies that were cited in practice bulletins or committee opinions were significantly more likely to incorporate a device or surgical intervention when compared to non-cited studies (aOR 3.60, 95% CI 1.85–7.00). They were also more likely to have a sample size between 500 and 999 (aOR 3.70, 95% CI 1.39–9.82). Subject, study location, funding source, journal, year of publication and the presence of positive findings were not significantly different among trials that were referenced versus were not (Table 1).

Table 1

Factors Associated with Citation in Eligible Practice Bulletin or Committee Opinion.

	Cited (n = 128)	Not Cited (n = 120)	Odds Ratio (95% CI)	Adjusted Odds Ratio (95% CI)
Subject
Gynecology (n = 140)	75 (54)	65 (46)	1.2 (0.7–2.0)	0.9 (0.6–1.6)
Obstetrics (n = 108)	53 (49)	55 (51)	0.8 (0.5–1.4)	1.0 (0.6–1.8)
U.S. Study (n = 134)	70 (52)	64 (48)	1.0 (0.6–1.7)	1.2 (0.7–2.0)
Multicenter (n = 113)	66 (58)	47 (42)	1.7 (1.0–2.7)	1.5 (0.9–2.6)
Federal Funding (n = 52)	28 (54)	24 (46)	1.12 (0.6–2.1)	1.1 (0.6–2.2)
Industry Funding (n = 47)	23 (49)	24 (51)	0.9 (0.5–1.7)	0.9 (0.4–1.7)
Intervention
Drug (n = 119)	55 (46)	64 (54)	0.7 (0.4–1.1)	0.7 (0.4–1.1)
Device/Surgery (n = 56)	41 (73)	15 (27)	3.3 (1.7–6.4)	3.6 (1.9–7.0)
Other (n = 73)	32 (44)	41 (56)	0.8 (0.6–1.1)	0.6 (0.3–1.0)
Positive Finding (n = 141)	70 (50)	71 (50)	0.9 (0.7–1.2)	0.8 (0.5–1.4)
Year Published
2009 (n = 32)	20 (63)	12 (38)	1.7 (0.8–3.6)	2.0 (0.9–4.4)
2010 (n = 51)	24 (47)	27 (53)	0.8 (0.4–1.5)	0.9 (0.5–1.8)
2011 (n = 36)	21 (58)	15 (42)	1.4 (0.7–2.8)	1.1 (0.5–2.4)
2012 (n = 51)	26 (51)	25 (49)	0.9 (0.5–1.8)	0.9 (0.5–1.7)
2013 (n = 44)	21 (48)	23 (52)	0.8 (0.4–1.6)	0.8 (0.4–1.5)
2014 (n = 36)	16 (47)	18 (53)	0.8 (0.4–1.7)	0.8 (0.4–1.7)
Sample Size
0–99 (n = 56)	29 (53)	26 (47)	1.1 (0.6–1.9)	0.8 (0.4–1.6)
100–199 (n = 81)	36 (44)	45 (56)	0.7 (0.4–1.1)	0.7 (0.4–1.2)
200–499 (n = 57)	30 (54)	26 (46)	1.1 (0.6–2.0)	1.1 (0.6–2.0)
500–999 (n = 24)	18 (75)	6 (25)	3.1 (1.2–8.1)	3.7 (1.4–9.8)
1000 and above (n = 32)	15 (47)	17 (53)	0.8 (0.4–1.7)	0.9 (0.5–2.1)
Journal
AJOG (n = 70)	32 (46)	38 (54)	0.7 (0.4–1.2)	0.7 (0.4–1.3)
Obstet Gynecol (n = 138)	72 (53)	64 (47)	1.1 (0.7–1.9)	1.1 (0.7–1.9)
JAMA (n = 17)	7 (41)	10 (59)	0.6 (0.2–1.7)	0.6 (0.2–1.8)
NEJM (n = 25)	17 (68)	8 (32)	2.1 (0.9–5.2)	1.9 (0.8–5.0)

Data are n (%) or OR (95% CI)

Adjusted for sample size and intervention

A similar proportion of eligible trials were referenced in gynecological and obstetrical PB (P = .58), and lead to either level A, B, C recommendations or did not lead to any recommendations (P = .83). There were significant differences in the characteristics of eligible obstetrical and gynecological studies (Table 2). Gynecological RCTs were more likely to be based in the United States, be industry funded and have a positive finding (p < .01). Obstetrical trials were more likely to have a drug-based intervention (p < .04). The only difference in journal citations was that the American Journal of Obstetrics and Gynecology was more likely to cite obstetrical RCTs (p < .03; Table 2). While 75 of 140 (53.6%) of gynecological RCTs were referenced in the two ACOG documents, 53 of 109 (49.1%) obstetrical RCTs were cited (p = .48).

Table 2

Characteristics of Eligible Publications by Subject (among “eligible” studies).

	Gynecology (n = 140)	Obstetrics (n = 108)	P-value
Cited in either ACOG Documents	75 (54)	53 (49)	0.48
Practice Bulletin	49 (51)	36 (47)	0.58
Committee Opinion	30 (32)	19 (36)	0.66
Cited in both	8 (16)	3 (14)	0.77
U.S. Study	86 (61)	48 (44)	< 0.01
Multicenter	63 (45)	50 (46)	0.84
Federal Funding	30 (21)	22 (20)	0.84
Industry Funding	40 (29)	7 (7)	< 0.01
Intervention
Drug	59 (42)	60 (56)	0.04
Device/Surgery	44 (31)	12 (11	<0.01
Other	37 (26)	36 (33)	0.24
Positive Finding	90 (64)	51 (47)	< 0.01
Cited in Bulletin or Opinion	75 (54)	53 (49)	0.48
Journal
AJOG	32 (23)	38 (35)	0.03
Obstet Gynecol	82 (59)	54 (50)	0.18
JAMA	12 (9)	5 (5)	0.22
NEJM	14 (10)	11 (10)	0.96

Data are n (%), Percentages may not equal 100% as some studies are eligible for citation in a Practice Bulletin but not a Committee Opinion (or vice versa) due to release of the ACOG guidelines after the publication of the trial.

Discussion

The key finding of our review article is that just over 50% of the randomized controlled trials published between 2009 and 2014 in four leading journals (NEJM, JAMA, Am J Obstet Gynecol and Obstet and Gynecol) are referenced in ACOG practice bulletins or committee opinions by early 2020. Approximately 25% of RCTs in Gynecology or Obstetrics that were referenced in practice bulletins lead to level A recommendations; for both subjects, the majority of the studies referenced in bulletins did not lead to level A, B, or C recommendations. Two characteristics of the trials—intervention being a device or surgical procedure and sample size being 500–999—improved the likelihood that the trial would be cited in either practice bulletins or committee opinions. Several factors (e.g. funding source, positive versus negative findings, and which of the 4 journals the study was published) were not associated with whether the RCT was cited within 5–10 years after publication.

Delay in referencing RCTs has clinical implications. Mugo et al. [10], for example, randomized over 1700 HIV-serodiscordant heterosexual couples to antiretroviral preexposure prophylaxis (PrEP). The investigators concluded that periconception PrEP was not associated with infertility, pregnancy loss, or congenital anomaly. Three years after the publication of the studies in JAMA, ACOG reaffirmed a committee opinion on the topic and did not reference the publication from JAMA [11]. Jones et al. [12], published a double-blind, double-dummy, flexible-dosing, randomized, controlled study in which 175 pregnant people who either received buprenorphine or methadone. The trial concluded that compared to those managed with morphine, neonates whose mothers were treated with buprenorphine had significantly less usage of morphine, shorter hospital stay, and a shorter duration of treatment for the neonatal abstinence syndrome. In spite the positive findings published in NEJM in 2010, ACOG’s committee opinion on the topic, published in 2017, did not reference the trial [13]. It is possible that there are reasons for the omitting trials like this, but the basis may not be apparent to most clinicians. Additional concerns for not referencing almost half of the trials are that RCTs are appreciably more expensive and time consuming than observational studies, but are considered more reliable for testing interventions [14], [15].

Our findings are consistent with other publications on the topic. The increased citation of trials involving surgical interventions or devices maybe be the result of previous research noting that surgical RCTs are more likely to be of higher quality then medical trials [16].Review articles examining the references of national guidelines on the same topic have noted that there is discordance and lack of transparency on what is versus what is not cited in the ACOG practice bulletins [17], [18], [19]. On the topic of intrapartum fetal heart rate monitoring, for example, among the guidelines published by United States (ACOG), Canada (Society of Obstetricians and Gynaecologists of Canada), and Australia and New Zealand (Royal Australian and New Zealand College of Obstetricians and Gynaecologists), only one reference was common to all 3 documents. Thus, it seems that not only the publications by ACOG but also from other countries have an indiscernible manner of selecting referencing. The potential downstream consequences of lack of transparency in developing these documents are skeptic disregard for the recommendations, knowledge gaps, or poor compliance with them [27], [28], [29].

The strengths of our review are that we examined all RCT in four leading journals. Previous investigators who examine ACOG guidelines have often limited themselves to either just obstetric topic or to practice bulletins [17], [20], [21]. In the current analysis we examined trials in obstetrics as well as gynecology, and we examined committee opinions along with practice bulletins. The time interval we allowed for a trial to be referenced was a minimum of five years.

There are limitations to our review articles. We focused our attention to studies published in four journals. It is possible that trials reported in other journals are cited more frequently than studies in the journals examined in this study. We speculate it is unlikely due to the high impact factor of the selected journals. Regardless, it is notable that almost 50% of references in obstetric practice bulletins were from these journals and 25% of references on gynecological topics [3]. It is also feasible that in future these trials will be referenced in these documents. ACOG does publish other documents besides practice bulletins and committee opinions in which the RCTs examined here could have been incorporated. Furthermore, while Cochrane reviews were included in the analysis, decision and cost-effective analyses cited were not examined for incorporation of otherwise uncited trials. Exclusion of RCTs in ACOG documents does not imply that their inclusion would influence the recommendations or practice. Though in other fields there is an association between industry sponsored trials and its influence on treatment effect size estimates, early stoppage of trial, positive results, [22], [23], [24], [25], [26] and potential biases, such a link in obstetrics and gynecology is currently unacknowledged. Since almost 20% of the studies referenced in the practice bulletin were industry sponsored, future investigation should explore the potential bias among trials funded by industry versus other sources. Lastly, our statistical analysis was limited by the sample size of some subsets, and therefore the degree of association that would show statistical significance was higher than in subsets with a larger sample size.

In conclusion, our review suggests that 5–10 years after the publication of RCTs in some of the most impactful journals, about half are referenced in ACOG documents which influence clinical practice. The majority of the RCTs referenced in practice bulletins do not lead to level A, B or C recommendations, irrespective of whether the trial is in gynecological or obstetrical topic. Further study into how to better incorporate RCTs into guidelines is indicated.

Funding

There was no funding received for conducting this project.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.