Correlation analysis of financial conflicts of interest and favourability of results or conclusions in addiction medicine systematic reviews and meta-analysis.

OBJECTIVE: To quantify conflicts of interest, assess the accuracy of authors self-reporting them, and examine the association between conflicts of interest and favourability of results and discussions in addiction medicine systematic reviews.
DESIGN: A search was performed on Medline (Ovid) from January 2016 to 25 April 2020 to locate systematic reviews and meta-analyses focused on treatments of addiction disorders using a systematic search strategy. Data were extracted from each systematic review, including conflict of interest statements, authorship characteristics and the favourability of the results/conclusion sections. A search algorithm was used to identify any undisclosed conflicts of interest on the Open Payments Database (Dollars for Docs), Dollars for Profs, Google Patents/United States Patent and Trade Office, and prior conflict of interest statements in other published works from these authors.
RESULTS: The search identified 127 systematic reviews, representing 665 unique authors. Of the 127 studies, 81 reported no authors with conflicts of interest, 28 with 1 or more conflict, and 18 had no conflict of interest statement. Additional non-disclosed conflicts of interest were found for 34 authors. There were 69 reviews that had at least one author with a conflict of interest. Of the 69 reviews, 14 (20.3%) reported favourable results and 26 (37.7%) reported favourable discussion/conclusions with no statistically significant association. A subanalysis was performed on publications with only US authors (51) with 35 (68.9%) having at least 1 conflict of interest. US authored studies that had a conflict of interest favoured the results (p = <0.001) and discussion/conclusion (p = 0.018) more often.
CONCLUSION: Although multiple undisclosed financial conflicts of interest were found, there was no correlation with the favourability of the results or discussion/conclusions across all addiction medicine systematic reviews. Further research needs to be done on US-based publications and encourage disclosure systems worldwide to provide more accurate reporting. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We included systematic reviews and meta-analysis in addiction medicine published between January 2016 to 25 April 2020.

Articles were initially screened by abstract using Rayyan in a double-blind fashion and then by full text to ensure they met inclusion criteria. Study characteristics and conflicts of interest (COI) statement information were extracted from each systematic review.

A step-by-step systematic search algorithm was used to identify undisclosed COI through the Open Payments Database, Dollars for Profs, Google Patents/United States Patent and Trade Office, and PubMed for other studies conducted by the authors in our sample. No statistically significant correlation was found between systematic reviews that had at least one author with a disclosed or undisclosed COI and the favourability of the results and conclusion.

Financial COI is a prominent focus in research currently and continued studies should evaluate how they continue to change or address them in the future.

Introduction

In 2018, 20.3 million people were classified as having substance dependence or abuse.1 Between 1999 and 2018, more than 700 000 Americans died from overdose.2 The National Institute on Drug Abuse estimates that tobacco, alcohol and illicit drug misuse results in roughly US$740 billion spent on crime, unemployment and healthcare.3 Despite the large number of prevention and treatment programmes implemented over the last 35 years and the billions of dollars spent to fund them,4 we are now faced with a significant health crisis. The high prevalence of substance abuse, with the increased mortality and morbidity associated with addiction prompts the need for rigorous research to guide treatment plans.5

Physicians make treatment decisions using evidence-based clinical practice guidelines; oftentimes, these guidelines include systematic reviews as supporting evidence for treatment recommendations. The American Society of Addiction Medicine’s 2020 National Practice Guideline for the Use of Medications in the Treatment of Addiction Involving Opioid Use6 used 35 systematic reviews in the updated guideline and provides recommendations for the use of pharmacological treatments including methadone, buprenorphine and naltrexone. The American Psychiatric Association also has a recently updated guideline for alcohol use disorder citing 15 systematic reviews used in the rationale for treatment options.7

Given the influence of systematic reviews on clinical judgement and treatment regimens, they must be well conducted and well reported. Careful attempts should be made to mitigate the effects of bias on systematic review outcomes. Two forms of bias—industry sponsorship and conflicted authors—have both been shown to result in bias affecting the results in numerous publications.8–10

Further exacerbating this problem of financial bias is the inaccurate reporting of conflicts of interest (COI). Andreatos et al11 found more than 87% of general payments to authors of clinical guidelines were inaccurately reported. A specific analysis of three top psychiatry practice guidelines reported that 90% of authors had a financial tie to the drug manufacturer and none of them correctly reported a COI.12 Previously published literature has revealed the pervasiveness for conflicted authors in psychiatric and other medical specialty trials with associated positive outcomes.13–15 With the negative effects that COI have on publications outcomes, further research must be done to limit conflicts and increase accurate reporting when present.16

The Sunshine Act promoted greater transparency of US physician disclosures such as honoraria, travel expenses and ownership.17 The Open Payments Database (Dollars for Docs) contains information regarding the financial relationships between manufacturers of devices/pharmaceuticals and US-based physicians. Researchers have previously used and continue to use Open Payments as a tool for cross-referencing US-based physician authors and their financial disclosure statements.18–20 Databases such as ProPublica’s Dollars for Profs provide a resource for searching the reported disclosures of PhDs who are employed through public universities. Given that bias of competing interest must be accounted for, this study aims to assess the accuracy of disclosure practices among authors of systematic reviews investigating treatments of addiction medicine and to investigate the associations between COI and industry funding and the nature of the results and discussions in the systematic reviews.

Methods

Transparency, reproducibility and reporting

We have provided study materials and protocol on Open Science Framework to increase the transparency and reproducibility of our results.21 While drafting this paper, we referred to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)22 and Murad and Wang’s guidelines for metaepidemiological studies.23

Search strategy

Medline (Ovid) was searched from January 2016 to 25 April 2020 to locate systematic reviews and meta-analyses focused on treatments of addiction disorders using the search strategy provided in the online protocol.21 The search results were then uploaded to a systematic review screening platform, Rayyan (https://rayyan.qcri.org/).

Screening

Two investigators (SD and SS) screened abstracts and titles for all search returns in a masked, duplicate manner. Full-text articles were evaluated following title and abstract screening to determine final inclusion. Disagreements were discussed until a consensus was reached. Additional authors were available for third-party arbitration.

Eligibility criteria

We used the PRISMA-P definition of a systematic review/meta-analysis, which states that a systematic review is ‘a review of a clearly formulated question that uses systematic and explicit methods to identify, select, critically appraise relevant research, and collect/analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies. Meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies’.24

Included studies were a systematic review or meta-analysis designed to address interventions for drug, alcohol or tobacco. Furthermore, to qualify for inclusion, systematic reviews must have been published between September of 2016 and the date which the search was conducted (25 April 2020). We chose the prespecified date range from September 2016 forward to allow 36 months from the time of the Open Payments Database which appeared online in September 2013. The date range (January 2016 to 25 April 2020) was selected according to the International Committee of Medical Journal Editors’ recommendation that any financial interests be disclosed up to 36 months prior to the time of journal submission.25 We chose the prespecified date range to allow 36 months from the time the search of Medline was conducted as the Open Payments Database began publishing data from August 2013.

Only systematic reviews published in English and reviews which synthesise studies of human data were included. The following study types were excluded from our study: observational studies (case control, cohort, surveys), clinical trials, narrative reviews, systematic reviews not related to (1) drug, alcohol and tobacco addiction prevention, (2) stabilisation following excessive use of a substance, (3) relapse prevention or (4) recovery maintenance, duplicates, withdrawn or retracted studies, non-human studies, systematic reviews without abstracts, letters to the editor and any remaining study which does not meet the inclusion criteria.

Training

All investigators were required to complete online training modules, which provided an overview of the study design, objectives of the study, study materials and examples of data extraction from systematic reviews. The training was recorded and is available online for reference.21

Data extraction

Two authors performed data extraction independently in a masked, duplicate fashion. Data extraction was performed in June/July of 2021 to provide sufficient time for the Open Payments Database or other databases to update information for their prior year. Investigators extracted the following data items from each SR: (1) PubMed identification number and/or DOI; (2) journal name; (3) date of publication; (4) name of author(s); (5) affiliation(s) for the first and last author; (6) author funding source; (7) complete COI statement; (8) whether the SR or meta-analysis addressed risk of bias (RoB); (11) the verbatim RoB statement; (12) whether author(s) were also an author on one or more of the primary studies included in the review (yes/no); (13) total number of self-cited primary studies; (14) primary outcome; and (15) whether narrative results and conclusions favoured the treatment or comparison group (eg, placebo, standard of care, control). We used the term ‘conclusion‘’ to represent a combination of the discussion and conclusion section of included reviews. Author funding sources for the systematic review were categorised as follows: industry, government, private non-profit, mixed, other, not funded, or not disclosed. Each possible COI was reviewed to ensure that it was relevant to the topic being studied. Irrelevant COI were not counted for the purpose of this study. COI were all considered equally weighted as the primary endpoint was to see if there was a correlation between authors with any conflict and the favourability of the SR results or discussion/conclusion towards the treatment group.

Favourability of narrative results and conclusions

Narrative results and conclusions were designated as favourable’, unfavourable’ or ‘mixed/inconclusive’. To evaluate the favourability of results and conclusions, we defined a favourable result or conclusion as one where the authors of the systematic review directly stated or implied in the results or conclusion section that the experimental group was determined to be definitively or probably superior to the control group or placebo. An unfavourable result or conclusion was defined as one where the authors of the systematic review directly stated or implied that the experimental group was not superior to the control group or placebo.

When appraising the results section, ‘favourable’ was assigned to SRs with only positive results. ‘Unfavourable’ was assigned when negative results were exclusively reported. ‘Mixed/inconclusive’ was assigned to narrative results sections that included both positive and negative results with no clear interpretation of the results. When appraising the conclusion sections, ‘favourable’ was assigned to when authors stated or implied favourability towards the target intervention. ‘Unfavourable’ was assigned when authors stated or implied favourability towards the comparison or control group. When neither ‘favourable’ nor ‘unfavourable’ applied to the conclusion, ‘mixed/inconclusive’ was assigned (ie, reporting negative population outcome but positive subgroup analysis).

Identification of undisclosed COI

Searches for undisclosed COI were undertaken using the algorithm provided in figure 1. This stepwise search was based on the methodology provided by Mandrioli et al,8 with modifications. These modifications included the incorporation of three additional databases—the Open Payments database (Dollars for docs), Dollars for Profs, the United States Patent and Trademark Office (USPTO). Dollars for Profs was included as it catalogues self-reported financial payments received by professors. To ensure consistency between investigators, authors created standardised search strings for PubMed, USPTO Database, and Google Patents using the Python programming language (Python Software Foundation, https://www.python.org/). If we were unable to verify a patent belonged to the author, we considered the search inconclusive and continued our process. In accordance with ICMJE standards of COI disclosure, PubMed searches were limited to 36 months prior to the publication of the original SR to determine if previously published studies included additional COI not disclosed in the SR from our sample. If this search yielded more than 20 publications, each investigator individually assigned random numbers to the resulting publications. The COI statement of the first 20 studies numerically was then examined. Each investigator individually generated random numbers to include a wider search of publications and opportunities for authors to disclose a COI. This process was performed until an undisclosed COI was discovered, at which time the author was then counted as having an undisclosed COI. This stop procedure is identical to that used by Mandrioli et al8

Figure 1

Search pattern to identify undisclosed financial conflicts of interest ().

RoB evaluations

To evaluate the risk of funding bias, we applied the Cochrane Collaboration’s criteria for assessment, and the following four items from Mandrioli et al8: (1) whether explicit and ‘well defined’ criteria that could be replicated by others were used to select studies for inclusion/exclusion; (2) whether an adequate study inclusion method, with two or more assessors selecting studies, was used; (3) whether search strategies were comprehensive; and (4) whether methodological differences that may introduce bias were controlled for. Each item was designated as yes, no, or unclear. We considered the overall RoB to be low if at least three of the aforementioned criteria were sufficiently met. Otherwise, the RoB was considered to be high. Authors SD and SS performed an independent and masked evaluation of RoB items. Discrepancies were discussed between investigators until a consensus was reached. DT and MV were available for third-party adjudication.

Statistical analysis

Results were quantified using descriptive statistics, and relationships were evaluated by Fisher’s exact tests, when possible. Stata V.16.1 (StataCorp, LLC) was used for all analyses. Because of the correlational nature of the research design, a power analysis was not performed.

Patient and public involvement

Patients and the public were not involved in the development of the research design or question addressed in this study. This study evaluated systematic reviews, meta-analyses and the authors of such publications. No patients or health information was used in this study.

Results

Sample characteristics

A total of 1331 manuscripts published between January 2016 and 25 April 2020 were identified using the search string listed in online materials. Of the initial sampling, 2 researchers reviewed each through rayyan.com and determined that 321 met the inclusion criteria. Of the 321 initially included publications, 194 were excluded after a full-text review. The reasons for exclusion included 62 being outside the date range, 43 not being a systematic review, 27 being a published poster/abstract, 59 did not address the 4 treatment areas of addiction evaluated here, and 3 were inaccessible even after interlibrary loan request. A final number of 127 publications were evaluated for authors with financial COI (figure 2).

Figure 2

Stepwise progression of search strategy to identify SR/MA and authors in addiction medicine.

The journals with the most publications analysed include Addiction (30), Drug and Alcohol Dependance (18), Addictive Behaviors (14), Journal of Substance Abuse Treatment (14), and Nicotine & Tobacco Research (12). The interventions used in each publication includes pharmacological (64), behavioural therapy/psychosocial treatments (53), prevention of addiction (8), and procedures (2).

COI statements within publications

Of the 127 systematic reviews or meta-analyses identified, 28 contained a statement reporting 1 more COI, 81 reported no authors with COI, and 18 provided no COI statement. Public funding was the most commonly reported with 66 of the 127 publications compared with university (4), public and university (3), and private/industry (2). Furthermore, 33 declined receiving any funding and 19 did not have a statement addressing funding (table 1). A total of 69 of the systematic reviews were found to have a least 1 author with a COI. Of the 127 publications, 104 (81.9%) of those were found to have a high RoB including 62 which were found to have a COI.

Table 1

Characteristics of included systematic reviews and meta-analysis

Characteristic	Form response	N (%)
Journal in which systematic reviews were published(n=127)	Addiction	30 (23.6)
	Drug and Alcohol Dependance	18 (14.2)
	Addictive Behaviors	14 (11.0)
	Journal of Substance Abuse Treatment	14 (11.0)
	Nicotine and Tobacco Research	12 (9.4)
	Alcohol and Alcoholism	6 (4.7)
	Other*	33 (26.0)
Conflict of interest (COI) statement(n=127)	All authors report no COI	81 (63.8)
	No COI statement present	18 (14.1)
	One or more authors report a COI	28 (22.0)
Intervention type(n=127)	Pharmacologic	64 (50.4)
	Procedure	2 (1.6)
	Behavioural therapy/psychosocial treatments	53 (41.7)
	Prevention	8 (6.3)
Affiliation of first author(n=127)	Public academic institution	92 (72.4)
	Private academic institution	15 (11.8)
	Government	14 (11.0)
	Public academic institution, government	1 (0.8)
	Non-profit institution	4 (3.1)
	Private-for-profit	1 (0.8)
Affiliation of last author(n=127)	Public academic institution	94 (74.0)
	Private academic institution	15 (11.8)
	Government	13 (10.2)
	Public academic institution, government	1 (0.8)
	Non-profit institution	3 (2.4)
	Private-for-profit	1 (0.8)
Author source of funding(n=127)	No funding received	33 (26.0)
	No statement listed	19 (15.0)
	Private/industry	2 (1.6)
	Public	66 (52.0)
	University	4 (3.1)
	Public and university	3 (2.4)
Self-citation of primary studies(n=127)	No, did not include self-cited primary studies	109 (85.8)
	Yes, included one or more self-cited primary studies	18 (14.2)

Author-specific COI

Of the 127 systematic reviews analysed, 655 total authors were identified. The most common countries of origin included the USA (276), UK (116), Canada (69) and Australia (61). Publications with COI statements listed 103 of the 655 authors as having a COI. By searching the Open Payments database, 21 authors had profiles, 15 reported receiving financial payments and 10 authors receiving funding did not report it as specified by ICMJE standards. Additional undeclared COI were identified on Dollars for Profs (1), registered patents (3) and PubMed searches of other authored publications (20) (table 2).

Table 2

Characteristics of systematic review authors (n=655)

Accuracy of author COI disclosure statement(n=655)	Reported conflict of interest (COI)	103 (81.1)
	Undisclosed FCOI found on open payments database	10 (7.9)
	Undisclosed FCOI found on docs for profs	1 (0.8)
	Undisclosed FCOI found by patents	3 (2.4)
	Undisclosed FCOI found on PubMed	71 (55.9)
	Additional FCOI besides what is already declared	20 (15.7)
Country of affiliation for authors conducting the systematic review (n=655)	USA	276 (42.1)
	UK	116 (17.7)
	Canada	69 (10.5)
	Australia	61 (9.3)
	India	17 (2.6)
	Netherlands	16 (2.4)
	Germany	15 (2.3)
	China	13 (2.0)
	Ireland	11 (1.7)
	Malaysia	11 (1.7)
	Switzerland	9 (1.4)
	France	7 (1.1)
	Belgium	6 (0.9)
	Spain	6 (0.9)
	Other	22 (3.4)

Favourability of results or discussion/conclusion related to financial COI

Of the 127 systematic reviews, a total of 69 (54.3%) had at least 1 author with a relevant COI that was initially reported or found through the search algorithm. The systematic reviews with financial COI reported favourable results in 14 (20.3%) studies and favourable discussion/conclusions in 26 (37.7%). There was no statistically significant correlation between a systematic review having at least one or more COI and the favourability of results (p=0.138, Fisher’s exact) or the favourability of the discussion/conclusion (p=−0.611, Fisher’s exact) (table 3).

Table 3

Frequency of favourability of results and conclusions if there is a pertinent conflict of interest

Review outcome	No financial conflict of interest	Financial conflict of interest	Fisher’s exact
Favourability of results
Results Favour Treatment Group	20	14	p=0.138
Results are Mixed/Inconclusive	25	41
Results Favour Placebo or Control Group	13	14
Favourability of Discussion/Conclusions
Discussion Favours Treatment Group	27	26	p=0.822
Discussion is Mixed/Inconclusive	20	27
Discussion Favours Placebo or Control Group	11	16

A subanalysis was performed on the total number of conflicted authors per publication and the favourability of results (p=0.50) and discussion/conclusion (p=0.77). An additional subanalysis was performed on publications with only United States (US) authors (51) with 35 (68.9%) having at least 1 COI. US authored studies that had a COI favoured the results (p≤0.001) and discussion/conclusion (p=0.018) more often.

Discussion

The primary endpoint of this study was between systematic reviews with one or more authors having a COI and the nature of the results or conclusions. For this complete sample of addiction systematic reviews, there was no statistically significant correlation found. A subanalysis was performed on publications with only US authors. The analysis found a positive correlation between studies with at least one conflicted author showing favourability towards results and discussion/conclusions. Continued research into COI and the effects they have on study outcomes is important as multiple publications have found that authors that receive funding from pharmaceutical companies are more favourable with the reporting of results and recommendations than research performed independently.8 26 27

Multiple authors in the systematic reviews were found to inaccurately report or did not report a financial COI at all. Of the 655 authors, 105 (16%) had an undisclosed conflict, which represents nearly 1-in-6 authors. We presume that the true number of authors with undisclosed COI is underestimated since only US physician–researchers have a legal responsibility to list financial support on The Open Payments website. Thus, non-US authors may have undisclosed conflicts that were not findable through our searches. This finding concerns us, as a large and consistent body of evidence indicates that self-disclosure is inaccurate. For example, Wayant et al reported that approximately one-third of oncologist authors of pivotal cancer therapy trials (ie, establishing the basis for drug approval) did not disclose financial conflicts with the industry sponsor. We believe that transparency and third-party reporting structures are necessary to successfully mitigate this issue. It is therefore critical to think about alternative reporting mechanisms to improve public trust in science and for readers of research studies to be able to critically evaluate the likelihood of financial bias on decision-making, results and discussions.

Another concerning finding is that authors who referenced their own papers in the systematic review were more likely to have an undisclosed COI. Self-citations increase important research metrics, such as the h-index (for some calculations) and the number of citations received by the author. Thus, there may be possibilities where authors may selectively favour their own studies for inclusion in systematic reviews. There are potentially countless reasons for self-citation that could include increasing one’s academic profile or increasing the impact of previous research. We acknowledge that determining which characteristics might contribute to these relationships between undisclosed COI and self-citations is outside the scope of our current investigation. Additionally, authors of systematic reviews may be experts in their field or perform research on a narrow topic. These authors may be appropriate when performing a systematic review but should be forward about the inclusion of their own research and address any other potential bias that may stem from it. Future research that expands on this finding is warranted and encouraged.

It is important to improve reporting and limit possible opportunities in the future. The author guidelines section of the top 5 psychiatry journals based on Google Scholar metrics was performed. These journals included Biological Psychiatry, JAMA psychiatry, Molecular Psychiatry, American Journal of Psychiatry and The Lancet Psychiatry all require an accurate statement for individual authors on a publication. The requirements for these statements are very specific but there is no mention of verifying the information reported. We recommend that journals implement a screening protocol to search the Open Payments database at the very least for possible undisclosed COI. Regarding database selection to uncover undisclosed conflicts, PubMed produced the greatest yield. The Open Payments Database is desirable because the data contained within it are not self-disclosed; however, only healthcare workers are currently listed. Many authors of systematic reviews are not healthcare workers; instead, they are methodologists, epidemiologists, scientists, research assistants or students. In these cases, Open Payments will not provide a significant yield. In an effort to include non-physician scientists in our search, we used Dollars for Profs, which was created by ProPublica from National Institutes of Health (NIH) COI records. Again, this database is limited to author self-disclosure. It yielded little return and may not be worth considering in future investigations. Likewise, our patent searches generated very few returns. Searching patent databases such as ‘Google patents’ for discrepancies in disclosure statements has been previously verified as a valid tool for locating undisclosed patents.28 The use of the U.S. NIH’s National Library of Medicine (NLM) for examining consistency in authors’ disclosure statements between separate publications has previously been validated as a source for identifying discrepancies.29

Strength and limitations

This study was subject to both strengths and limitations. Regarding its strengths, our study was performed in duplicate across screening and data extraction phases by two of the authors who were masked throughout. Performing the study in duplicate limits errors in data extraction and errors in study selection. This process is considered the gold standard methodology of the Cochrane Collaboration.30 We performed this study according to a previously developed and published protocol, and any deviations to our protocol were described in subsequent protocol updates. Regarding its limitations, we may have not included relevant systematic reviews or our searches may not have retrieved all relevant systematic reviews. Furthermore, there is always the possibility that the authors who performed data extraction exercised some degree of subjectivity, especially related to whether a systematic review conclusion favoured the intervention or not. Sample size in our study is also a limitation. International authors with COI may be under-reported as there is no legal obligation outside of the USA to report such payments. This under-reporting may alter findings by increasing the number of systematic reviews with conflict authors. The correlation found for strictly US-based authors is difficult to correlate if it is because US authors are more conflicted due to the lack of international reporting. Because of the correlational design of this study, our results should not be generalised to other authors or systematic reviews in other fields. Rather, our results should be viewed descriptively. Studies across other specialties are needed so a meta-analysis can be performed to provide a more informed understanding of whether authors with COIs are more likely to report results and conclusions favouring the intervention.

Conclusion

Our study found that there was no relationship between authors with COI and the favourability of the systematic review discussion/conclusion. A subanalysis of authors from the USA found that conflicted publications were more likely to favour the treatment group in results and discussions. We did identify 105 authors with undisclosed financial COI.