Association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews: systematic review.

OBJECTIVE: To investigate the association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews.
DESIGN: Systematic review. ELIGIBILITY CRITERIA: Studies that compared the association between conflicts of interest and favourable recommendations of drugs or devices (eg, recommending a drug) in clinical guidelines, advisory committee reports, opinion pieces (eg, editorials), or narrative reviews. DATA SOURCES: PubMed, Embase, Cochrane Methodology Register (from inception to February 2020), reference lists, Web of Science, and grey literature. DATA EXTRACTION AND ANALYSIS: Two authors independently extracted data and assessed the methodological quality of the studies. Pooled relative risks and 95% confidence intervals were estimated using random effects models (relative risk >1 indicates that documents with conflicts of interest more often had favourable recommendations than documents with no conflicts of interest). Financial and non-financial conflicts of interest were analysed separately, and the four types of documents were analysed separately (preplanned) and combined (post hoc).
RESULTS: 21 studies that analysed 106 clinical guidelines, 1809 advisory committee reports, 340 opinion pieces, and 497 narrative reviews were included. Unpublished data were received for 11 studies (eight full datasets and three summary datasets). 15 studies showed risk of confounding because the compared documents could differ in factors other than conflicts of interest (eg, different drugs used for different populations). The relative risk for associations between financial conflicts of interest and favourable recommendations for clinical guidelines was 1.26 (95% confidence interval 0.93 to 1.69; four studies of 86 clinical guidelines), for advisory committee reports was 1.20 (0.99 to 1.45; four studies of 629 advisory committee reports), for opinion pieces was 2.62 (0.91 to 7.55; four studies of 284 opinion pieces), and for narrative reviews was 1.20 (0.97 to 1.49; four studies of 457 narrative reviews). An analysis of all four types of documents combined supported these findings (1.26, 1.09 to 1.44). In one study that investigated specialty interests, the association between including radiologists as authors of guidelines and recommending routine breast cancer was: relative risk 2.10, 95% confidence interval 0.92 to 4.77; 12 clinical guidelines).
CONCLUSIONS: We interpret our findings to indicate that financial conflicts of interest are associated with favourable recommendations of drugs and devices in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews. Limitations of this review were risk of confounding in the included studies and the statistical imprecision of individual analyses of each document type. It is not certain whether non-financial conflicts of interest influence recommendations. SYSTEMATIC REVIEW REGISTRATION: Cochrane Methodology Review Protocol MR000040. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Diagnostic and treatment recommendations in clinical guidelines or advisory committee reports have an important impact on patient care. Similarly, recommendations in opinion pieces, such as editorials, and narrative reviews written by key opinion leaders could influence clinical practice. But making recommendations requires judgment, and a concern is whether conflicts of interest might influence such recommendations.

Recommendations are often written by authors with financial conflicts of interest related to the drug or device industry.1 2 For example, in a study of 45 clinical guidelines, 53% of authors had financial conflicts of interest.3 Researchers have also studied non-financial conflicts of interest such as specialty and academic interests, although which interests and relationships constitute a non-financial conflict of interest and whether the term is appropriate is debatable.4

Numerous studies have investigated the impact of financial conflicts of interest on the interpretation of study results. One Cochrane methodology review reported an association between industry funding and favourable conclusions in primary research studies, mainly clinical trials,5 and similar results were reported in another Cochrane methodology review on financial conflicts of interest in systematic reviews.6

In the current systematic review we investigated to what degree financial and non-financial conflicts of interest are associated with favourable recommendations (eg, recommending a drug) in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews.

Methods

The details of the methods have been published in a Cochrane methodology review protocol.7 Here we describe the core methods.

Eligibility criteria

Studies considered eligible for review were published and unpublished studies in any language and of any design that assessed the association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, or narrative reviews of drug or device interventions. We defined advisory committee reports as transcripts or reports from meetings held in committees to advise an organisation on a drug or device intervention, such as records from the Food and Drug Administration advisory committee on oncological drugs. Opinion pieces were defined as commentaries, editorials, and letters. Narrative reviews (non-systematic reviews) were defined as literature reviews without a systematic search of the literature and without clear eligibility criteria (see supplementary appendix 1).

For financial conflicts of interest, we included studies regardless of the type of financial conflict—that is, financial conflicts of interest related to both industry funding of documents and authors’ company ties. For non-financial conflicts of interest, we included studies on intellectual, academic, professional, or specialty interests, and on personal or professional relationships.8

Studies were excluded if they concerned: financial conflicts of interest not related to the drug or device industry (eg, tobacco or nutrition industry) as the impact from conflicts of interest might differ between industries; beliefs (eg, religious), personal experiences (eg, experiencing the medical condition), or membership of certain groups (sex or ethnicity), even if the original authors defined this as non-financial conflicts of interest; both financial and non-financial conflicts of interest at the level of an institution (eg, employment at a university that collaborates with industry); and conflicts of interest related to reports from scientific grant committees.

Search strategy and study inclusion

PubMed, Embase, and the Cochrane Methodology Register (from inception to February 2020) were searched for studies and protocols. The search strategy we developed for PubMed was adapted for the other databases (see supplementary appendix 2). To identify additional studies and protocols, we searched reference lists of the included studies, Web of Science (from inception to March 2020) for studies that cited any of the included studies, and PubMed (from inception to March 2020) for publications by the first and last author of the included studies.

We also searched proceedings from peer review congresses,9 Cochrane colloquiums,10 and Evidence Live11 for conference abstracts published up to February 2020. PROSPERO (from inception to February 2020) was searched for registered systematic reviews, and the ProQuest database (from inception to February 2020) for dissertations and theses. Finally, we searched Google Scholar (from inception to March 2020).

One review author (CHN) screened titles and abstracts for obvious exclusions. Two review authors (CHN and AWJ or AL) independently assessed potentially eligible studies based on the full text. Disagreements were resolved by discussion, with arbitration by a third review author (AL or AH) when needed.

Outcomes and data extraction

Our primary outcome was favourable recommendations, defined as such by the authors of the included studies.

Two review authors (CHN and either AWJ, ML, or AL) independently extracted data from included studies. Disagreements were resolved by discussion, with arbitration by a third review author (AH or AL) when needed.

We extracted data on basic study characteristics and on the association between conflicts of interest and favourable recommendations. Extracted data on conflicts of interest were based on the definitions used by the authors of the included studies. Information was also extracted on funding and authors’ conflicts of interest for the included studies. Supplementary appendix 3 provides details of our data extraction.

Unpublished data

We contacted the authors of the included studies to obtain unpublished data, clarify problems in our assessment of methodological quality, or receive copies of unpublished protocols (supplementary appendix 4).

Assessment of methodological quality in included studies

As tools for assessing methodological quality in these types of studies have not been published, we developed our own criteria based on those used in previous Cochrane methodology reviews on financial conflicts of interest in primary research studies and systematic reviews.5 6

Two review authors (CHN and either AWJ, ML, or AL) independently assessed methodological quality in included studies. Disagreements were resolved by discussion, with arbitration by a third review author (AL or AH) when needed. We used the following criteria:

• Whether the methods for including documents were adequate (adequate methodological quality might, for example, include reporting of clear inclusion criteria, with two or more assessors independently selecting documents).

• Whether the methods for coding conflicts of interest were adequate (adequate methodological quality might, for example, include coding by two or more assessors based on multiple information sources).

• Whether the methods for coding recommendations were adequate (adequate methodological quality might, for example, include coding by two or more assessors blinded to conflicts of interest information).

• Whether the methods for dealing with confounding were adequate. The documents included in a study might differ on key aspects—for example, in a sample of clinical guidelines, the guidelines might differ in types of patients and conditions, interventions, the quality of the underlying evidence, and the quality of the guidelines, which could potentially confound the association between conflicts of interest and favourable recommendations. Therefore, adequate methodological quality could, for example, include documents with and without conflicts of interest discussing the same treatment used in similar groups of patients.

We coded a study as having overall adequate methodological quality if all criteria were assessed as adequate; otherwise, we coded it as having inadequate methodological quality.

Data synthesis

Data management of individual studies

In our primary analyses, we used similar coding of conflicts of interest and recommendations to the included studies. If an ordinal scale was used to grade recommendations, for example highly positive, positive, neutral, negative, and highly negative, we recoded recommendations into two categories: favourable versus neutral or unfavourable.

If a study included different types of documents (such as both clinical guidelines and research papers), we included the study in our pooled analyses only if we had separate data for the types of documents relevant for our review.

In our analyses on clinical guidelines, we included one study that investigated 13 guidelines that each included recommendations on 24 different drugs.12 To allow for this type of panel data, we used Poisson generalised estimating equations to calculate effect estimates, which we could include in our pooled analyses.13

In our analyses on advisory committee reports, we included studies with two types of analysis units: committee members and their individual votes (individual level) and advisory committee reports and the overall voting outcome (meeting level). In our primary analysis, we analysed data at meeting level, as this level of analysis was most comparable with recommendations in the other types of documents (eg, clinical guidelines).

In some cases, the same document was included in two separate studies. When we had access to unpublished data, it was possible to remove the duplicate documents, and we chose to remove it from the study with the latest publication date. We included two studies that investigated the same FDA advisory committee reports14 15 and removed duplicates from one of the studies.15 In our analyses on opinion pieces, we included two studies that investigated editorials published in some of the same oncology journals in overlapping periods16 17 and removed duplicates from one of the studies.16

Primary analyses

Owing to expected clinical and methodological heterogeneity between the included studies, we used inverse variance random effects models to estimate relative risks with 95% confidence intervals. We compared recommendations between documents with and without conflicts of interest and ensured uniform directionality, so a relative risk value of more than 1 indicated that documents with conflicts of interest more often had favourable recommendations than documents without conflicts of interest. We analysed financial and non-financial conflicts of interests separately, and clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews separately. We dealt with statistical heterogeneity using the I2 statistic and prediction intervals (supplementary appendix 5).

Using the methods for calculating a number needed to treat, we calculated a number needed to read for each document type (supplementary appendix 6).18 The number needed to read was defined as the expected number of documents with conflicts of interest needed to be read rather than documents without conflicts of interest for one additional document having a favourable recommendation. As it is difficult to describe the 95% confidence interval for number needed to read when the confidence interval of the relative risk crosses the boundary of no effect,19 we report the 95% confidence interval of the number needed to read in supplementary appendix 6.

Secondary analyses

We analysed advisory committee reports at individual level (ie, individual votes).

In a post hoc analysis, we combined all four types of documents (clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews) in one analysis of financial conflicts of interest.

Subgroup and sensitivity analyses

Various subgroup analyses were undertaken, including stratification by different types of financial conflicts of interest, such as funding, honorariums, and gifts, and stratification by different degrees of financial conflicts of interest (≥50% vs <50% of the authors or committee members with financial conflicts of interest). In addition, we undertook various sensitivity analyses in relation to how information on conflicts of interest and recommendations were coded, using fixed effect models, and by excluding studies with authors who had conflicts of interest (supplementary appendices 7 and 8).

Analyses were conducted in either RevMan 5.4 or Stata 15.

Assessment of certainty of the evidence

Based on experience, using formal systems such as GRADE (Grading of Recommendations Assessment, Development and Evaluation) for assessing the certainty of evidence from methodological studies is challenging. We therefore focused on interpreting our results in the context of the statistical precision of our estimates (width of confidence intervals) and risk of confounding. Supplementary appendix 9 shows the GRADE assessments using a similar approach to both observational intervention studies and prognostic studies.20 21

Patient and public involvement

We decided to undertake our study without patient or public involvement. Although our protocol was available in the public domain,7 we received no comments on it.

Results

Of 9973 records identified in the searches, 21 studies that analysed 106 clinical guidelines, 1809 advisory committee reports, 340 opinion pieces, and 497 narrative reviews were included in the review (fig 1).2 12 14 15 16 17 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 No unpublished studies or protocols for planned studies were identified.

Fig 1

Flow chart of study inclusion

Table 1 presents the characteristics of the included studies. The 21 studies were published between 1998 and 2019. Eight studies analysed clinical guidelines (median 9 (range 2-50) guidelines), seven analysed advisory committee reports (376 (79-416) reports), six analysed opinion pieces (44 (8-131) opinion pieces), and five analysed narrative reviews (84 (7-213) narrative reviews). Sixteen studies investigated drugs, three investigated devices, and two investigated both drugs and devices.

Table 1

Characteristics of included studies

Studies	Type and No of included documents	Definition of conflicts of interest	Definition or classification of favourable recommendations
Studies investigating financial conflicts of interest
Aakra 201222	18 clinical guidelines on self-monitoring of blood glucose	Guideline funded by industry	Weakly or strongly in favour of self-monitoring (using 4 point scale)
George 201423	2 clinical guidelines on treatment of primary immune thrombocytopenia	Guideline funded by or author financial ties to companies that manufacture products in guideline	Strong recommendation for thrombopoietin receptor agonists
Norris 201312	13 clinical guidelines on glycaemic control in type 2 diabetes	At least one author with financial ties to companies that manufacture drugs included in guideline	Drug recommended in guidance portion of guideline
Schott 201324	2 clinical guidelines on efalizumab for treatment of psoriasis	At least one author with financial ties to drug companies	Efalizumab judged more favourable
Tibau 201525	50 clinical guidelines on anticancer drugs*	At least one author with financial ties to companies with economic, commercial, or competing interest in guideline recommendation	Specific drugs recommended in guideline abstract
Ackerley 200915	98 committee reports and 1191 committee members from FDA drug, radiology, device, and biologic advisory committees†	At least one committee member with financial ties to the product manufacturer or competitor	Voted in favour of product
Cooper 201926	416 committee reports and 1483 committee members from FDA drug advisory committees	Committee member with financial ties to any drug company	Voted in favour of drug
Lurie 200614	76 committee reports and 886 committee members from FDA drug advisory committees‡	At least one committee member with financial ties to drug manufacturer or competitor	Voted in favour of drug
Pham-Kanter 201427	379 committee reports and 15 739 committee members from FDA drug advisory committees	Committee member with financial ties to drug manufacturer or competitor	Voted in favour of drug
Tibau 201628	79 committee reports from FDA oncological drug advisory committees§	At least one committee member with financial ties to drug manufacturer or competitor	Voted in favour of drug
Xu 20172	385 committee reports from FDA drug advisory committees	At least one committee member with financial ties to drug manufacturer or competitor	Voted in favour of drug
Zhang 201929	376 committee reports from FDA drug advisory committees	At least one committee members with financial ties to drug manufacturer or competitor	Voted in favour of drug
Bariani 201316	131 editorials commenting on phase III oncology clinical trials¶	At least one author with financial ties to drug company	Positive or highly positive interpretation of trial (using 5 point scale)
Lerner 201217	54 editorials commenting on phase III oncology clinical trials	At least one author with financial ties to for profit organisation	Favourable interpretation of trial (using 3 point scale)
Dunn 201630	213 narrative reviews of neuraminidase inhibitors for influenza	At least one author with financial ties to manufacturer of neuraminidase inhibitor of interest	Concluded safety and efficacy of ≥1 neuraminidase inhibitors
Hartog 201231	153 narrative reviews on hydroxyethyl starch for various conditions	At least one author with financial ties to manufacturer of any commercially available intravenous fluid	Recommended hydroxyethyl starch over other fluids
Downing 201432	4 clinical guidelines; 23 editorials and commentaries; 40 reviews (mainly narrative) commenting on randomised trial of fenofibrate (ACCORD-Lipid trial)**	At least one author with financial ties to manufacturer of fenofibrate or any other drug company with commercial interests in fenofibrate	Recommended fibrates
Hayes 201933	8 opinion pieces; 7 narrative reviews commenting on randomised trial on tumour treating fields	At least one author with financial ties to manufacturer of tumour treating fields	Supported tumour treating fields without caveats
Stelfox 199834	33 letters; 32 reviews (mainly systematic); 5 original research studies on safety of calcium channel antagonists	Individual authors with financial ties to drug companies	Supported calcium channel antagonists (using 3 point scale)
Wang 201035	5 clinical guidelines; 91 letters, editorials, and commentaries; 84 narrative reviews on cardiovascular risk of rosiglitazone	Industry funding of document or at least one author with financial ties to manufacturers of antihyperglycaemic drugs	Recommended rosiglitazone
Studies investigating both financial and non-financial conflicts of interest
Norris 201236	12 clinical guidelines on screening mammography	Percentages of authors disclosing any financial conflicts of interest. At least one radiologist in author team	Recommended routine screening

FDA=Food and Drug Administration.

91 clinical guidelines included in study (not all had data available in a format for inclusion in analysis).

611 advisory committee reports included in study and 221 duplicates also included in Lurie 200614 removed (not all had data available in a format for inclusion in analysis).

221 advisory committee reports included in study (not all had data available in a format for inclusion in analysis).

82 advisory committee reports included in study (not all had data available in a format for inclusion in analysis).

131 opinion pieces included in analysis after removing 19 duplicates also included in Lerner 2012.17

5 clinical guidelines, 24 editorials and commentaries, and 70 reviews included in study (not all had data available in a format for inclusion in analysis.

Twenty studies investigated financial conflicts of interest only and one study investigated both financial conflicts of interest and specialty affiliations among guideline authors (non-financial conflicts of interest). None of the included studies reported industry funding, but six did not report funding information. Seven of the included studies that investigated documents with and without financial conflicts of interest were conducted by authors who themselves had financial conflicts of interest.

Unpublished data were received for 11 studies; full datasets (n=8)14 15 16 17 29 30 31 35 and additional summary data (n=3).25 27 28 No published protocols were found, and only two studies provided unpublished protocols.14 32 No discrepancies were found between outcomes in these protocols and study publications. Nine of 21 author teams replied that no protocol existed for their study, and two author teams supplied reports that we did not consider to be protocols (supplementary appendix 4).

Methodological quality in included studies

In total, 20 studies were assessed as having overall inadequate methodological quality and one study as having adequate methodological quality (fig 2). Around half of the included studies had adequate methodological quality in the document inclusion process (n=10), and most had adequate methodological quality in the coding of conflicts of interest (n=15) and recommendations (n=17). Six studies were assessed as adequate for dealing with confounding and 15 as inadequate for dealing with confounding, because they included documents of different topics, such as various cancer drugs for different indications, or included documents on the same drug used for different populations, such as diabetes drugs used in adults, children, or pregnant women.

Fig 2

Methodological quality in included studies

Financial conflicts of interest: differences in recommendations

Clinical guidelines

Eight studies investigated a total of 106 clinical guidelines.12 22 23 24 25 32 35 36 Data from four of these studies (86 clinical guidelines) could be included in the pooled primary analysis.12 22 25 35 The relative risk for the association between financial conflicts of interest and favourable recommendations in clinical guidelines was 1.26 (95% confidence interval 0.93 to 1.69, I2=0%; fig 3). The number needed to read for clinical guidelines was 9.1 (supplementary appendix 6). The remaining four studies had similar results to those of the pooled analysis (supplementary appendix 6).23 24 32 36

Fig 3

Meta-analysis of association between financial conflicts of interest and favourable recommendations for each type of document and documents combined. IV=inverse variance; COI=conflicts of interest

Advisory committee reports

Seven studies investigated a total of 1809 advisory committee reports.2 14 15 26 27 28 29 Data from five studies could be included in our primary or secondary pooled analyses.14 15 27 28 29 In the primary analysis, including four studies of 629 advisory committee reports, the relative risk for the association between advisory committee reports with any member who had financial conflicts of interest and voting in favour of approving a drug or device was 1.20 (0.99 to 1.45, I2=24%; fig 3). The number needed to read for advisory committee reports was 7.7 (supplementary appendix 6). In the secondary analysis, including three studies of 17 816 votes, the relative risk for the association between financial conflicts of interest of individual advisory committee members and voting in favour of approving a drug or device was 1.14 (1.07 to 1.21, I2=35%; fig 4). The remaining two studies investigated voting behaviour among advisory committee members; one of these studies had similar results to our pooled analysis (supplementary appendix 6).2

Fig 4

Meta-analysis of association between financial conflicts of interest and favourable votes of committee members. IV=inverse variance; COI=conflicts of interest

Opinion pieces

Six studies investigated a total of 340 opinion pieces.16 17 32 33 34 35 Data from four of these studies (284 opinion pieces) could be included in our pooled primary analysis.16 17 33 35 The relative risk for the association between financial conflicts of interest and favourable recommendations in opinion pieces was 2.62 (0.91 to 7.55, I2=78%; fig 3). The number needed to read for opinion pieces was 2.3 (supplementary appendix 6). The remaining two studies had similar results to our pooled analysis (supplementary appendix 6).32 34

Narrative reviews

Five studies investigated a total of 497 narrative reviews.30 31 32 33 35 Data from four of these studies (457 narrative reviews) could be included in our pooled primary analysis.30 31 33 35 The relative risk for the association between financial conflicts of interest and favourable recommendations in narrative reviews was 1.20 (0.97 to 1.49, I2=39%; fig 3). The number needed to read for narrative reviews was 8.3 (supplementary appendix 6). The remaining study had similar results to our pooled analysis (supplementary appendix 6).32

All document types

In a post hoc analysis, when all types of documents were combined, the relative risk for an association between financial conflicts of interest and favourable recommendations was 1.26 (1.09 to 1.44, I2=38%; fig 3). The number needed to read was 7.1 (supplementary appendix 6).

Non-financial conflicts of interest: differences in recommendations

One study investigated specialty interests and included 12 clinical guidelines on mammography screening.36 The focus was on whether the guideline author team included a radiologist. The relative risk for an association between having radiologists on the guideline panel and recommending routine screening for breast cancer was 2.10 (0.92 to 4.77). The number needed to read was 2.1 (supplementary appendix 6).

Subgroup and sensitivity analyses

No differences were found in effect estimates by the type of financial conflicts of interest or the degree of financial conflicts of interest for any document type (supplementary appendix 7).

Sensitivity analyses were robust in 20 of 23 analyses of financial conflicts of interest. In three analyses the association between financial conflicts of interest and favourable recommendations became stronger (supplementary appendix 8).

Assessment of certainty of the evidence

The evidence on financial conflicts of interest in all four types of documents and non-financial conflicts of interest in clinical guidelines should be interpreted with caution as most of the studies (15 out of 21) dealt inadequately with confounding and all effect estimates in our primary analyses lacked statistical precision. Using the GRADE approaches for intervention and prognostic studies resulted in low to very low certainty of the evidence depending on the type of document and the GRADE system used (supplementary appendix 9).

Discussion

In this systematic review we found an association between financial conflicts of interest and favourable recommendations of drugs and devices in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews. The four primary analyses resulted in effect estimates of a fairly similar magnitude and consistent direction, but each with varying degrees of statistical precision. The post hoc analysis in which all document types were combined confirmed these findings, and statistical precision was increased. Our findings on the impact of non-financial conflicts of interest on recommendations were limited to evidence from a single study of breast cancer screening guidelines and the involvement of radiologist authors, with statistically imprecise results. It is therefore uncertain whether specialty interests or other types of non-financial conflicts of interest have an effect on recommendations.

Strengths and limitations of this study

A major strength of our study is the inclusion of unpublished data from 11 of 21 studies. We retrieved eight full datasets and unpublished summary data for three additional studies, which ensured high data quality and comprehensive analyses thereby increasing statistical precision and minimising reporting bias. Furthermore, we searched grey literature for published and unpublished protocols. We only obtained two protocols,14 32 and a comparison of outcomes in the protocols with outcomes in the study publications showed no evidence of selective outcome reporting.

Six of 21 included studies were, however, reported in a format that did not allow inclusion in meta-analysis. Four of these studies reported similar results to our meta-analysis. Two of the four studies combined different types of documents without stratifying results, with estimates (relative risk 1.69, 95% confidence interval 1.07 to 2.67, and 13.91, 1.99 to 96.97) in line with our primary analysis.32 34 The other two of the four studies sampled a single pair of clinical guidelines with and without financial conflicts of interest, and in both cases only guidelines with conflicts were favourable.23 24 The last two of the six studies (29% of all documents)2 26 sampled FDA committee reports from the same period as the studies included in our meta-analysis, implying a considerable risk of documents overlapping between the studies. The two studies reported no results for our primary analysis; if we had had access to the raw data we would likely have excluded a considerable proportion of the documents to avoid double counting. Thus, we find it unlikely that our result would have been qualitatively different had the six studies reported results in a format suitable for meta-analysis.37

Nevertheless, this review has some limitations. Firstly, the different types of documents were described using various terms in the included studies, and, despite using a comprehensive search strategy, we might have missed relevant studies. Furthermore, only four studies were included in each of our four primary analyses. Therefore, our effect estimates have some degree of statistical imprecision and none of our primary analyses were statistically significant at the conventional 5% level. The sizes of the effect estimates were, however, similar for clinical guidelines, advisory committee reports, and narrative reviews, and slightly higher for opinion pieces, and when we combined all document types in a post hoc analysis, including 13 studies, the statistical precision was increased and we found a statistically significant association with moderate heterogeneity.

Secondly, our criteria for assessment of the methodological quality of the studies for adequately dealing with confounding might be viewed as strict, and others might interpret the methodological quality of studies differently. Nevertheless, most of the studies were at risk of confounding because compared documents might differ in other factors than conflicts of interest (eg, documents on different drugs used for different patient groups). Although confounding could have influenced our estimates, the association between conflicts of interest and recommendations was fairly consistent across document types, despite some studies including comparable documents, such as clinical guidelines on efalizumab for the treatment of psoriasis,24 and others including different documents, such as advisory committee reports on a wide range of different drugs.27 Moreover, recommendations in guidelines and narrative reviews could have been influenced by conflicts of interest in the underlying evidence. For example, in certain clinical specialties such as oncology,38 conflicts of interest are common, which could have impacted the conclusions of clinical trials and systematic reviews5 6 and thereby indirectly affected guideline recommendations and potentially resulted in effect modification. Furthermore, how conflicts of interest in the primary clinical trials and systematic reviews underpinning a guideline are interpreted could be associated with the guideline authors’ conflicts of interest.

Thirdly, the number of authors with financial conflicts of interest might influence recommendations in a document. Our subgroup analyses of documents where a majority of the authors had financial conflicts of interest compared with those with a minority of authors found no difference in effect. However, the analyses were simplistic and based on few data, resulting in statistically imprecise results. Another important factor is the role of authors with financial conflicts of interest. For example, the chair of a guideline committee or the lead author of a narrative review could have a greater influence on recommendations than an author with a less prominent role. However, none of the included studies reported data that allowed such a comparison.

Fourthly, 11 of the 21 included studies relied solely on disclosed information in the included documents for coding conflicts of interest. This could have led to an underestimation of our effect estimates, as conflicts of interest are often underreported in various publication types, including clinical guidelines.3

Finally, the interpretation of our results can be debated. No published guidance is specifically tailored for summarising and interpreting evidence from methodological studies. One approach could be to use the GRADE system,20 but it is questionable whether using GRADE for observational intervention studies or prognostic studies is best suited for methodological studies, since the methodology of studies or the presence of conflicts of interest cannot be randomised. In our supplementary appendix 9, we reported assessments using both strategies and obtained low to very low certainty of evidence depending on the type of document and approach. Using the GRADE approach for intervention studies resulted in a more conservative interpretation of the certainty of the evidence.

Comparison with other studies or reviews

Other systematic reviews of financial conflicts of interest in different types of studies produced similar findings to those of our review. A recent Cochrane methodology review focusing on primary research studies, mainly trials, reported that industry funded studies more often had favourable conclusions than non-industry funded studies (relative risk 1.34, 95% confidence interval 1.19 to 1.51).5 Similarly, another recent Cochrane methodology review reported that systematic reviews with industry funding or by authors with financial conflicts of interest more often had favourable conclusions than systematic reviews without financial conflicts of interest (relative risk 1.98, 95% confidence interval 1.26 to 3.11).6

Financial conflicts of interest have also been investigated in relation to other industries than the drug and device industry. A systematic review reported that industry funded nutrition studies and reviews more often had favourable conclusions than non-industry funded nutrition studies and reviews (relative risk 1.31, 95% confidence interval 0.99 to 1.72).39

Meaning of the study

For our analyses, we included studies of four types of documents that are common and involved the authors’ interpretation of external evidence (involving methods less stringent than in a systematic review). Although we had anticipated potential differences between the document types, we found a fairly consistent association between financial conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews. One reason could be that authors with conflicts of interest are more prone to confirm prior beliefs by selectively citing and interpreting the literature.40 This could also explain the somewhat stronger association found in opinion pieces, which to some degree allow authors more room for interpretation than narrative reviews, which undergo peer review, and clinical guidelines, which are increasingly done using standardised methods. On an absolute scale, the association between conflicts of interest and recommendations was particularly strong for opinion pieces and specialty interest in clinical guidelines with numbers needed to read of only 2.3 and 2.1, respectively, although the estimates had considerable statistical imprecision.

Our findings support conflicts of interest policies from major organisations that issue guidelines, such as the US Preventive Services Task Force, World Health Organization, and National Institute for Health and Care Excellence.41 42 43 These policies aim to minimise the number and role of guideline authors with conflicts of interest. Similarly, some high impact journals manage conflicts of interest beyond disclosure—for example, The New England Journal of Medicine prohibits narrative reviews and editorials by authors with major financial conflicts of interest (>$10 000; >£7715; >€8540), and The Lancet prohibits commentaries, seminars, reviews, and series by authors with relevant stock ownership, employment, or company board membership.44 45 Other journals should consider introducing such polices to minimise the influence of conflicts of interest on journal content.

In line with this, in 2008 the FDA introduced more stringent criteria on the types of conflicts of interest allowed by committee members.15 This might explain why a study29 that exclusively sampled committee reports from 2008 and onwards, found a weaker association between financial conflicts of interest and recommendations in advisory committee reports than the three other studies included in the pooled analysis.14 15 28

Unanswered questions and future research

Ideally, future studies should try to minimise the risk of confounding by, for example, using a matched study design.46 However, identifying editorials commenting on the same study, or guidelines addressing the same question and developed using similar methods, might be a challenge. Furthermore, future research could focus on investigating whether specific types of financial conflicts of interest (eg, advisory board membership) or conflicts of interest related to specific companies (eg, drug manufacturer) have a greater impact than others. The included studies used various definitions of financial conflicts of interest and recommendations, and therefore use of a standardised terminology would be helpful.

Investigating the impact of non-financial conflicts of interest is challenging as no uniform definition exists. Nonetheless, a multitude of factors can be viewed as non-financial conflicts of interest, such as specialty interests, intellectual interests, personal beliefs, and personal relationships.47 48 Labelling personal beliefs and theoretical schools of thoughts as conflicts of interest is problematic as no researcher is completely free from interest or from intellectual preconceptions.4 49 50 Furthermore, the distinction between financial and non-financial conflicts of interest is not always clear. For example, in the included study on mammography screening guidelines36 it can be debated whether being a radiologist should be considered a purely non-financial conflicts of interest, as radiologists often have direct financial income (in the form of salary) from breast cancer screening. Future studies could focus on investigating the impact of the various types of non-financial conflicts of interest on favourable recommendations and on the impact of managing such interests using guideline panels with a broad range of skill sets, rather than mainly content area experts.

Conclusions

We interpret our findings to indicate that financial conflicts of interest are associated with favourable recommendations of drugs and devices in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews. Although the magnitude of effect is fairly consistent across document types, most studies had a risk of confounding and our individual analyses of each document type had some degrees of statistical imprecision. It is uncertain whether non-financial conflicts of interest influence recommendations.

Clinical guidelines, opinion pieces, and narrative reviews are often written by authors with conflicts of interest related to the drug or device industry; similarly, members of advisory committees, such as regulatory drug advisory committees, often have conflicts of interest

Previous studies found that financial conflicts of interest are associated with favourable conclusions in primary research studies and systematic reviews

It is not known to what degree conflicts of interest affect recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews

The findings of this review indicate an association between financial conflicts of interest and favourable recommendations of drugs and devices in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews

The included studies were, however, at risk of confounding, and some degrees of statistical imprecision was found in individual analyses by document type

It is uncertain whether non-financial conflicts of interest influence recommendations