Discrepancy between financial disclosures of authors of clinical practice guidelines and reports by industry.

There is a substantial effort to increase the accuracy of conflicts of interest (COI) reporting, and reduce the influence of COI between physicians and industry, especially as it relates to clinical practice guidelines.We used the newly implemented Open Payments dataset to evaluate the accuracy of COI disclosures of authors of clinical practice guidelines that were either newly published or revised within 2014 and were included in the National Guideline Clearinghouse (NGC) website (maintained by the U.S. Department of Health and Human Services). Authors were considered as having inaccurate COI disclosure if they had not reported all companies from which they had received funds >$5000 in the 12 months preceding the guideline's publication.We identified 223 guidelines that were either newly published (109/223; 48.9%) or revised (114/223; 51.1%) within 2014 and were included in the NGC website. Among the 1329 guideline authors with available Open Payments data, 523 received >$5000 from at least 1 healthcare-associated entity. However, only 56 out of the 523 authors (10.7%) were found to have accurate COI disclosure. The percentage of authors with accurate COI disclosure in revised guidelines was significantly lower than in newly published guidelines (6.8% vs 14.3%; P < 0.01) and was also found to differ between specialties. Furthermore, authors were less likely to inaccurately disclose "research payments" (37/49, 75.5%) compared to "general payments" (488/559, 87.3%, P = 0.02) as well as "other/associated research funding" (430/506, 85.0%, P = 0.08). No statistically significant association was detected between funding amount and disclosure accuracy.The majority of guideline authors lacked significant COIs, but among authors that received significant funds from at least 1 healthcare-associated entity the frequency of accurate disclosure was low. These findings indicate that the current process of disclosing COIs may be suboptimal and a proactive approach should be adopted in order to minimize COI reporting discrepancies. Furthermore, every effort should be undertaken to ensure the completeness and accuracy of the data recorded in the Open Payments database.

Introduction

Clinical practice guidelines are extremely influential and constitute indispensable guides to clinical care.[ The standardizing influence of practice guidelines is particularly important in the US, as the health care system does not follow the single-payer model and scientific accuracy, moral integrity, and transparency in their composition is particularly important. A factor that could potentially compromise the integrity of guidelines is the presence of financial conflicts of interest (COI), due to relationships between the authors and industry.[ Although such relationships are common and necessary for the discovery of new treatment and diagnostic modalities, they can potentially introduce real or perceived bias to many areas of research and clinical practice.[ Reflecting this valid concern, the Institute of Medicine has defined COI as “circumstances that create a risk that professional judgments or actions regarding a primary interest will be unduly influenced by a secondary interest.”[ Primary interests are defined as research integrity, medical education quality, and patient welfare.[ Secondary interests include financial gain, professional achievement, etc. Federal regulations set by the Institute of Medicine, as well as additional rules determined by local Institutional Review Boards are designed to identify and curtail the influence of potential COIs.[

The Physician Payment Sunshine Act is part of the Patient Protection and Affordable Care Act and was signed into law in 2010.[ This program (described in Section 2) aims to provide the public with detailed data regarding physician funding. However, the appropriate interpretation of these data or their potential utility in promoting accurate COI reporting remain to be investigated.[ In this report, we sought to evaluate the accuracy of COI disclosure among authors of clinical practice guidelines that were newly published or revised within 2014 and included in the National Guideline Clearinghouse (NGC). We proceeded to estimate the number of authors with inaccurate COI disclosure, as well as the potential discrepancies in the frequency of such inaccurate disclosures by guideline type and amount of funding.

Methods

Data sources

NGC (http://www.guideline.gov/index.aspx), an initiative by the Agency for Healthcare Research and Quality that is part of the US Department of Health and Human Services (HHS). We chose the NGC database, because it contains a list of high quality clinical guidelines that are updated periodically by NGC, as such, the guidelines included in this website may be expected to be more influential in “shaping” clinical practice. Furthermore, NGC provides “structured, standardized summaries containing information derived from guidelines using the NGC Template of Guideline Attributes” (http://www.guideline.gov/about/), thus facilitating the rapid identification of clinically relevant information. In addition, the included guidelines are publicly available, providing physicians and other health professionals with free access to guideline-related documents. The NGC website is updated weekly; the updated list of guidelines is formulated, in part, by considering whether guidelines were reviewed or revised within the previous 5 years. The NGC inclusion criteria were updated in June 2014; however, this change did not affect the requirement for appropriate COI disclosure. Detailed information about the revised as well as the original inclusion criteria can be found at https://www.guideline.gov/help-and-about/summaries/inclusion-criteria.

The Open Payments program was used as a reference in our study (https://www.cms.gov/OpenPayments/index.html). Open Payments is a federally run program that annually collects information on the financial relationships of healthcare providers and healthcare facilities with the healthcare industry. This program operates under the auspices of the Centers for Medicare & Medicaid Services (CMS), which is part of the HHS and collects information from healthcare companies about payments satisfying the aforementioned criteria; this information is then made publicly available. Reported payments “and other transfers of value can be for many purposes, like research, consulting, travel, and gifts” (https://www.cms.gov/About-CMS/About-CMS.html).

Data collection

We searched the NGC database to identify the guidelines that were newly published or revised within 2014 by US-based organizations. After first identifying all eligible guidelines, we proceeded to compile a list of all guideline contributors. All listed contributors were defined as authors for the purposes of this study. For each author, we reviewed the amount of funding received from healthcare companies through the Open Payments database. We then identified the authors who were reported as recipients of a significant amount of funding (>$5000 from a single entity, see detailed definition below) in the 12 months preceding the corresponding guideline's publication. Subsequently, we searched the COI section published with each guideline and matched the results of the COI report with data extracted from the Open Payments database. Approval by an ethics board was unnecessary, as only publicly available data were used.

We also collected data on the type of funds that were received by the included authors and categorized them as follows: general payments (“payments that are not associated with research study”), research payments (“payments that are associated with research study” that includes payments or transfers of value that are applied to research), and other/associated research funding (“funding for research project or study in which the physician is named as principal investigator” on a research transaction, but is not the primary recipient of the transaction). All definitions follow the funding classification scheme provided in the CMS website (https://openpaymentsdata.cms.gov/search) and were verified through personal communication with the CMS Open Payments Team. The relationship of authors with industry was further categorized based on the amount of funding received as follows: funding of $5000 to $10,000; $10,001 to $100,000; $100,001 to $200,000; and >$200,000. This classification scheme was intended to facilitate the investigation of potential differences in COI reporting by the amount of funding. As no similar study with Open Payments data had been conducted before, the employed cut-off points were determined de novo.

Definitions

Significant funding: the definition follows the guidelines established by the HHS/National Institutes of Health (NIH) 2011 revised regulations. Specifically, a financial interest was deemed to be significant “if the value of any remuneration received from a single entity in the 12 months preceding the disclosure, when aggregated, exceeds $5000, or when the investigator (or the investigator's spouse or dependent children) holds any equity interest (eg, stock, stock option, or other ownership interest)” (http://grants.nih.gov/grants/policy/coi/coi_faqs.htm#3192).

Accurate COI disclosure: Authors were classified as having accurate COI disclosure if they had correctly reported all payments >$5000, received from a single entity in the 12 months preceding the guideline's publication. Individual payments meeting the NIH definition of significant funding were classified as accurately disclosed, only if they were correctly reported in the COI section of the guideline. Individual payments not meeting the NIH definition of significant funding were not considered.

Inaccurate COI disclosure: Authors were classified as having inaccurate COI disclosure if they had failed to correctly report all payments >$5000, that were received from a single entity in the 12 months preceding the publication of the guideline. Failure to accurately disclose even a single payment that met the NIH definition of significant funding (see above) resulted in the author in question being classified as having inaccurate COI disclosure. As for individual payments meeting the NIH definition of significant funding, lack of disclosure, as well as any reporting discrepancy in either funding amount or source resulted in the payment's classification as inaccurately disclosed. Individual payments not meeting the NIH definition of significant funding were not considered.

Analyses

For statistical analysis, categorical variables were expressed as frequencies (%) and compared using the chi-squared test. A P value <0.05 was considered statistically significant. Statistical analysis was performed by use of the Stata v13 software package (Stata Corporation, College Station, TX).

Results

As previously stated, the NGC website is updated regularly. Data were last collected during the week of October 17, 2016. At the time of our search we identified 239 guidelines that were published/revised in 2014 by US-based organizations. Of these, 16 guidelines were excluded as they lacked an attached authors’ list or COI disclosure forms; as such, 223 guidelines were considered eligible for further analysis. Among the eligible guidelines, 109/223 (48.9%) were newly published and the remaining 114/223 (51.1%) were revised within 2014. These 223 guidelines, included hematology/oncology guidelines (43/223, 19.3%), followed by guidelines in general medicine (41/223, 18.4%) and surgery (23/223, 10.3%). A complete list of guidelines per specialty can be found in Table 1. In total, the aforementioned guidelines had 3594 authors; of these, only 1329 had available Open Payments data during the period in question (1329/3594, 37.0%). This percentage was not consistent across all specialties. Specifically, 13 specialties had a significantly lower percentage of authors with available Open Payments data, as demonstrated in Supplemental Table 1. Out of the 1329 authors with available Open Payments data, a total of 523 were reported as recipients of >$5000 from at least 1 healthcare-associated entity (523/1329, 39.4%). Among the 523 identified authors, the greatest number were authors of hematology/oncology guidelines (120/523, 22.9%), followed by authors of general medicine (64/523, 12.2%) and cardiology guidelines (57/523, 10.9%). Detailed information on authors and author disclosures per specialty is provided in Table 2.

Table 1

Guidelines categorized by specialty.

Table 2

Total number of authors, number of authors who had significant financial relationship with the healthcare industry and number of authors with accurate COI disclosure.

Interestingly, only 56 out of the 523 authors with significant funding (10.7%) had accurate COI disclosure on the published guidelines; of the remaining 467 authors with inaccurate COI disclosure, 56 failed to report significant funding from 3 or more sources. Authors of general medicine (P = 0.02), orthopedics/trauma (P = 0.01), pulmonology (P = 0.02), gastroenterology (P = 0.02), and radiology (P = 0.03) guidelines had significantly less accurate COI disclosures compared to other specialties (Table 2). Furthermore, the percentage of authors with accurate COI disclosure in newly published guidelines was significantly higher than the corresponding percentage in revised guidelines (14.3% vs 6.8%; P < 0.01 (Table 3).

Table 3

Results of the comparison in COI disclosure trends: A, between newly published versus revised guidelines; B, between “research payments”, “general payments” and “other/associated research funding”; and C, based on the amount of funds.

In regards to individual financial relationships, it was noted that the included authors were significantly less likely to inaccurately report “research payments” (37/49, 75.5% of cases) compared to “general payments” (488/559, 87.3% of cases, P = 0.02); a similar trend was noted for “other/associated research funding”, although it did not reach statistical significance (430/506, 85% of cases, P = 0.08) (Table 3). Detailed information on the included guidelines, such as the number of authors who received funding, as well as the number of authors with accurate COI disclosure is provided in Supplemental digital content.

The potential for disparities in accurate payment disclosure, according to the amount of funding, was also assessed. Importantly, no statistically significant association was detected between the percentage of inaccurately disclosed payments and the monetary sums involved. Specifically, for payments in the $5000 to 10,000 range, the percentage of inaccurately disclosed payments was 86.4%. Similarly, the corresponding percentages for payments in the $10,001 to 100,000, $100,001 to $200,000, and >$200,000 ranges were 85.4%, 86.1%, and 85.2%, respectively (Table 3).

Discussion

Clinical practice guidelines play a central role in clinical care and are among the most widely utilized and cited clinical references. Given their significance, clinical practice guidelines need to be as scientifically accurate and devoid of perceived and actual COI influences as possible. As such, the federal initiative to render the amount of funding provided by healthcare companies to physicians and healthcare entities publicly available has been considered a milestone toward greater transparency in both research and clinical practice.[ However, the practical applications and potential limitations of this initiative have yet to be examined. Consequently, in the present work, we sought to utilize the newly established Open Payments database to evaluate the accuracy of COI disclosures among authors of clinical practice guidelines published in the NGC database within 2014. To the best of our knowledge, the present study is the first to employ these data to investigate potential inaccuracies in COI reporting.

Importantly, our results suggest that the Open Payments database can be improved. For example, approximately two-thirds of authors (63.0%) had no recorded payment data during the period in question and it is not clear whether these authors truly received no payments from industry or were simply excluded from the database. The provision of a complete list of all healthcare providers eligible to be included in the database (essentially a national registry of providers) alongside existing payment records could help address this concern. As for the 1329 authors with available payment data, the majority lacked significant COIs to disclose (60.6%). However, among authors that received significant funding, only 10.7% of the authors had accurately disclosed potential COIs.

It is noteworthy that the percentage of accurate COI disclosure was consistently higher among authors of newly published guidelines, compared to authors of revised guidelines. Additional analysis confirmed that individual payments made to authors of newly published guidelines were more likely to be accurately disclosed (Table 3). This discrepancy may be the result of a failure to update COI disclosure forms concurrently with the revision of the guideline itself. Therefore, it may be advisable to place greater emphasis on updating the COI disclosure sections of guidelines following revisions. This can be accomplished by treating every revision or update as a new submission, thus necessitating the completion of new COI disclosure forms.

Furthermore, a discrepancy was detected in the percentage of accurate disclosures according to payment type; specifically, inaccurate disclosures were significantly less frequent in the reporting of research payments, compared to general payments. A similar trend was detected when comparing the percentage of inaccurate disclosures between research payments and associated research funding, although it did not reach statistical significance. As research payments are, by their definition, directly made to the investigator in relation with research activities, it may be expected that recalling and accurately reporting them would be comparatively easy. On the other hand, “general payments” constitute an umbrella term, including everything from royalties and consulting fees to traveling and accommodation expenses, journal and conference subscriptions, food and beverages, etc. Similarly, associated research funding is not directly paid to the author in question, but to his employer/institution. It is understandable that accurate accounting and reporting of so many different payments is challenging and confusing. Specific measures should be undertaken to address the aforementioned challenges. To this end, the provision of dedicated administrative support on the part of the author's institution during the process of COI disclosure may greatly facilitate research payment accounting and, consequently, accurate disclosure of potential COIs.

After analyzing disclosures by specialty, we found that 5 specialties had significantly lower percentages of accurate disclosure. However, this finding should be interpreted with caution, as data reporting across different specialties may lack consistency. To further clarify this point, we performed a subanalysis to evaluate the percentage of authors with available data and we found that it varied across different specialties (Supplemental Table 1). In fact, as many as 13 specialties had a significantly lower percentage of authors with available data, compared with the others. Although this finding may simply result from differences in research funding patterns, it also raises concerns regarding the consistency of data reporting across different specialties. Additional studies, relying on future Open Payments data should revisit this issue, as any potential reporting inconsistency could severely undermine the practical applications of this important initiative.

More importantly, our findings strongly indicate that the likelihood of disclosing a payment accurately is independent of the sums involved. As such, the presence of significant financial ties between an author and a company did not appear to increase the frequency of accurate disclosure. This finding is in line with a recent study that examined articles authored by recipients of >$1 million from 5 orthopedic device manufacturing companies in 1 year; similarly, it was found that more than half of the authors did not disclose their ties with these companies.[ However, it is equally noteworthy that the accuracy of disclosure did not seem to vary inversely with the sums involved, as would be expected if inaccurate disclosure was motivated by considerations incompatible with professional integrity. Taken together, our findings suggest that failure to disclose a payment accurately is, in the vast majority of cases, a result of inadvertent error, rather than conscious bias.

A number of limitations, stemming from the Open Payments database in general and our study design in particular, should be taken into consideration before interpreting our results. First, our decision to employ the HHS/NIH 2011 revised definitions as a uniform benchmark of significant funding that necessitates disclosure was done for consistency, but it is arbitrary, since each journal and/or organization follows different reporting standards. As such, it may be that a number of authors labeled as having inaccurate COI disclosure were merely adhering to the policies of their respective sponsoring organization or journal. This variability in reporting is confusing and results in a lack of consistency that should be addressed. To this end, the NIH guidelines constitute a reasonable uniform benchmark for the purposes of determining significant COIs, given their wide availability and the prestige of the issuing institution (http://grants.nih.gov/grants/policy/coi/coi_faqs.htm#3192). It should also be noted that our funding classification was determined arbitrarily, as no similar study of Open Payments data had been conducted previously. As such, our findings will need to be validated by future studies. Moreover, our study was limited to a 1-year period. Although a significant number of guidelines and authors was included, thus allowing the formulation of some notable conclusions, it would certainly be interesting to investigate how COI disclosures and the quality of the Open Payments dataset develop over time, as more data become available. An additional limitation is that the accuracy and completeness of the data recorded in the Open Payments database is safeguarded only by the right of listed physicians to challenge payment records. However, the extent to which this right is exercised is currently unclear. In fact, our analysis suggests, that given the relative novelty of this initiative, a number of issues in data collection and management have yet to be addressed. For example, we noted that only slightly more than one third of the total number of guideline authors had available data on the Open Payments database during our study period; what is even more concerning is that this percentage was significantly different across various specialties. The extent and magnitude of our findings hints at potential deficiencies in the data collection methods employed in the creation of the Open Payments database and should be revisited in future studies.

Conclusion

Although it is greatly encouraging that the majority (60.6%) of authors of clinical practice guidelines included in our study lacked significant COIs, it should also be noted that accurate disclosure was relatively rare among authors with significant COIs (10.7%). Although it is possible that our analysis may overstate the extent of the problem, it clearly indicates the need for heightened vigilance in the management and public reporting of potential COIs. A proactive approach should be adopted in order to minimize discrepancies in COI reporting. A specific emphasis should be placed on ensuring that the COI sections of guidelines are updated along with the guidelines themselves, while institutional administrative support might ensure accurate accounting and reporting of received payments. Furthermore, given the potential applications of the Open Payments database in promoting transparency in research and clinical practice, every effort should be made to ensure that the data on the website are as complete and accurate as possible. Finally, the adoption of uniform COI definitions and disclosure standards among journals and organizations that participate in guideline formulation would greatly facilitate any future assessment of COI disclosure accuracy. In conclusion, financial relationships between academia and industry help medical discovery but can also result in potential COIs that could introduce bias, or perceived bias, in clinical practice. The vigilant implementation of clear, uniform standards, and review policies aimed at regulating physician relationships with industry can be helpful in avoiding these pitfalls.