Searching for unpublished data for Cochrane reviews: cross sectional study.

OBJECTIVE: To describe the experiences of authors of Cochrane reviews in searching for, getting access to, and using unpublished data.
DESIGN: Cross sectional study.
SETTING: Cochrane reviews. PARTICIPANTS: 2184 corresponding authors of Cochrane reviews as of May 2012. MAIN OUTCOME MEASURE: Frequencies of responses to open ended and closed questions in an online survey.
RESULTS: Of 5915 authors contacted by email, 2184 replied (36.9% response rate). Of those, 1656 (75.8%) had searched for unpublished data. In 913 cases (55.1% of 1656), new data were obtained and we received details about these data for 794 data sources. The most common data source was "trialists/investigators," accounting for 73.9% (n=587) of the 794 data sources. Most of the data were used in the review (82.0%, 651/794) and in 53.4% (424/794) of cases data were provided in less than a month. Summary data were most common, provided by 50.8% (403/794) of the data sources, whereas 20.5% (163/794) provided individual patient data. In only 6.3% (50/794) of cases were data reported to have been obtained from the manufacturers, and this group waited longer and had to make more contacts to get the data. The data from manufacturers were less likely to be for individual patients and less likely to be used in the review. Data from regulatory agencies accounted for 3.0% (24/794) of the obtained data.
CONCLUSIONS: Most authors of Cochrane reviews who searched for unpublished data received useful information, primarily from trialists. Our response rate was low and the authors who did not respond were probably less likely to have searched for unpublished data. Manufacturers and regulatory agencies were uncommon sources of unpublished data.

Introduction

Selective reporting of trials is common.1 Thus despite the existence of hundreds of thousands of published randomised trials and thousands of updated Cochrane reviews, the true benefits and harms of many interventions are still unknown.

Recent studies have reported successes in obtaining details, including results, of unpublished clinical trials from licensing authorities and health technology agencies.2 3 4 These sources have the potential to reduce reporting biases in reviews of drug interventions. The inclusion of unpublished or inadequately reported data in meta-analyses generally leads to more reliable effect estimates.5 However, only a little over 10% of the Cochrane reviews from 2000-06 included unpublished trials.6

Unpublished data include complete trials that have never been published as well as specific outcomes that are not reported in published trials. For this study we considered data published even if published only in conference abstracts, research reports, and dissertations.

The Cochrane handbook suggests searching for unpublished data from the following sources: local experts, pharmaceutical companies, national and international trial registers (for example, clinicaltrials.gov), company trial registers, subject specific trial registers, and trial results registers.7 Regulatory agencies are not mentioned in the handbook and no guidance is given on how to obtain data or protocols from such agencies. It is also unclear how the different sources should be prioritised—that is, which sources are most likely to supply useful data.

Many review authors have obtained unpublished trial protocols, reports, additional summary data, or individual patient data from a variety of sources. We provided an overview of the experiences of Cochrane review authors in searching for, getting access to, and using unpublished information from trials.

Methods

We conducted an online survey of corresponding authors of Cochrane reviews and protocols. The survey contained closed and open ended questions.

We gathered information on how trial characteristics and data were obtained, types of data (for example, whole trials, missing outcomes, and additional analyses), difficulties encountered, and how the data were used. Our previous experience suggested that unpublished data are obtained in a wide variety of—and sometimes unexpected—ways, indicating that open ended and qualitative questions would provide useful information that we could not collect using only structured questions.

Sample selection

We retrieved a list of all corresponding authors of Cochrane reviews and protocols through the Cochrane Collaboration Information Management System (Archie). This information was imported into an online survey application (SurveyMonkey) and we invited all authors by email to participate. If the invitees did not respond within 10 days, we sent a reminder. A second reminder was sent after 20 days, and a final one after 30 days. Respondents who only partially filled in the survey also received a reminder. We collected data from the 21 May to 8 August 2012.

Statistical analysis

We reported frequencies of responses for each question response choice. Partial responses were also included. During data collection, but before we analysed the data, we hypothesised that drug manufacturers might differ from the other data sources. We used a χ2 test to compare the proportions of the recorded characteristics between manufacturers and non-manufacturers. We dichotomised scales with more than two categories.

Questionnaire

The survey was tested by 10 pilot testers. Their comments were implemented in the final version.

The respondents were asked to answer the questions in relation to a review in which they had been directly involved. If the respondents had been involved in several reviews, they were encouraged to choose one that included searching for unpublished data and that had resulted in experiences that could possibly benefit other review authors. Respondents who did not search for or obtain unpublished data were asked to give a reason. Respondents who did search for and obtain data were asked to provide a citation for their work and to state their primary source of unpublished data. They could choose between manufacturers, regulatory agencies, investigators, commercial and non-commercial trial registers, funders, ethics committees, and others. For their primary source they were asked to provide a name, year of query, number of attempts at getting the data, delay until the data were obtained, method of communication, reasons for thinking that data might be available, details on the data obtained, and whether the data were used in their review. Respondents could also provide information on secondary sources of unpublished data. Finally, they were asked whether they investigated a drug intervention, what the biggest difficulties were in obtaining unpublished data, and if they had any additional comments. The survey contained 82 questions but took less than five minutes to complete, as not all questions were relevant for each specific case (see supplementary file).

Results

We sent the questionnaire to 5915 corresponding authors of Cochrane reviews and protocols; 2184 replied (response rate 36.9%), 1889 of whom completed all questions in the survey (figure). Most of the dropout occurred when the respondents were asked to provide a citation to the work they authored (n=194).

Flowchart

Percentages can add to more than 100%, as several of the response options were not mutually exclusive. Of the 2184 respondents, 528 (24.2%) did not search for unpublished data. The reasons, “not expecting success,” “not expecting reliable data,” and “too time consuming,” each accounted for around 20% of the replies (table 1). The most common reason given was “other” (n=265, 52.4%) of which the majority specified that the review was still in an early phase and that the search had yet to be performed (n=177). Box 1 lists other reasons, which can be categorised into the following groups: only wanted to include published data and therefore deliberately chose not to search for unpublished data (11 authors), found published data and therefore did not think it was necessary to search for unpublished data (n=5), did not know how to search for unpublished data (n=26), thought searching for unpublished data was the responsibility of the trial search coordinator in the Cochrane review group in question (n=5), tried to search but failed (n=12), and simply stated it was not relevant (n=14).

Table 1

 Responses by Cochrane authors to questions in survey

Reasons	No (%)
Why didn’t you try to get access to unpublished data?:	n=506*
Other or not relevant	265 (52.4)
Did not expect success	116 (22.9)
Too time consuming	108 (21.3)
Did not expect data to be reliable	102 (20.2)
Did not know it could be important	42 (8.3)
What were the main reasons why you did not obtain unpublished data?:	n=717*
Never received a response	475 (66.2)
Information requested was not available	282 (39.3)
Other	155 (21.6)
Respondent said it was too much trouble	54 (7.5)
How did you approach the source of data?:	n=794*†
Email	666 (83.9)
Telephone/in person	116 (14.6)
Letter/fax	114 (14.4)
Other	101 (12.7)
How many times did you make contact?:	n=794†
1-3	597 (75.2)
4-6	117 (14.7)
7-9	29 (3.7)
≥10	51 (6.4)
How long did it take before you got the data?‡:	n=794†
<1 week	101 (12.7)
1 week to <1 month	323 (40.7)
1 month to <6 months	298 (37.5)
≥6 months	72 (9.1)
How did you know the potential data source might have data?:	n=794*†
They conducted trials	485 (61.1)
Other	156 (19.6)
Colleagues	128 (16.1)
Through websites	90 (11.3)
I had no idea	86 (10.8)
Earlier published attempts at accessing data (for example, data from EMA or FDA)	34 (4.3)
Cochrane handbook	33 (4.2)
Court proceedings	7 (0.9)
What were the main challenges in incorporating the unpublished data in your review?:	n=666*
Time	273 (41.0)
There were no difficulties	248 (37.2)
Poor organisation of the material obtained	139 (20.9)
Other	109 (16.4)
Poor readability of the material obtained	65 (9.8)
Sheer mass of obtained material	45 (6.8)
Expenses	24 (3.6)
Did you use the obtained data in your review?”:	n=794†
Yes	651 (82.0)
Other	64 (8.1)
No, data not usable	50 (6.3)
No, trial quality not sufficient	13 (1.6)
No, trial excluded owing to other reasons	16 (2.0)

EMA=European Medicines Agency; FDA=Food and Drug Administration.

*Percentages may total >100 because responses are not mutually exclusive.

†Relates to number of data sources described by total of 676 respondents. Some respondents contacted more than one data source.

‡From first contact to when data were delivered.

One of our inclusion criteria is that the data must be published

We are still in the process of data extraction of papers, if we do get very little publications we might think about unpublished data

Was not aware there was any unpublished data in my topic area

Did not know it was possible to get

Haven’t tried yet. Assumed this was done as part of the searching process done by the Cochrane group searcher

Where I have previously asked for unpublished data, authorship has been requested

Among the 1656 authors who did search for unpublished data, 730 (44.1%) never obtained any, 913 obtained data, and 13 did not reply to this question. The most common reason for not obtaining data was never receiving a response (66.2% of 717; an additional 13 did not specify a reason, table 1). The second most common reason was that the contacted person did not have the data (39.3% of 717). By analysing the comments in the “other” category, we found additional common reasons for not obtaining data: no unpublished studies were found, investigators were reluctant to release data until the study was published, commercial confidentiality, promised data but never delivered, and author’s contact information could not be found. In some instances, authors only wanted to deliver data if they became coauthors of the review. See box 2 for more examples.

Got a response to say they would look for data, but then no further response

Drug company responded that data were confidential

Drug company stated it could not be used for research, only for formulary decision making

Said they were preparing for future publication

Respondent said they did not think the information I requested was relevant/helpful to the review question (this was a drug sponsored trial for which I requested subgroup data)

Respondent said it was unnecessary for my clinical question

Big pharma said they didn’t regard the question of sufficient clinical value to warrant disclosing the data

It was too long since the original studies were published. Some authors were uncontactable and we had answers from authors who had thrown away the data we needed

A total of 676 respondents gave details on 794 sources that provided data. The most common data source was trialists, accounting for 73.9% of the 794 (table 2). Only 6.3% of the data came from manufacturers, 3.0% from regulatory agencies, and 6.3% from non-commercial trial registers. The “other” category accounted for 8.3% (n=66), where the most common sources were dissertations and conferences (which these authors regarded as unpublished, contrary to our definition of published data). Journal editors, Cochrane review groups, the World Health Organization, librarians, consumer support groups, and Google searches also contributed. The respondents did not contact any sources not already listed in the Cochrane handbook.6 The most common regulatory agencies that provided data were the Food and Drug Administration (n=11) and the European Medicines Agency (n=4).

Table 2

 Source of data from which respondents obtained unpublished data

Sources of data	No (%)
Trialists/investigators	587 (73.9)
Other	66 (8.3)
Manufacturers	50 (6.3)
Non-commercial trial register (for example, clinicaltrials.gov)	50 (6.3)
Drug and device regulatory agencies	24 (3.0)
Company owned trial register	7 (0.9)
Funders	7 (0.9)
Research ethics committees/institutional review boards	3 (0.4)
No of sources*	794 (100.0)

*676 respondents gave details on 794 sources.

Source details

The most common way to approach sources of information was by email (table 1). Using websites was the most common approach specified by respondents that chose the “other” category. In 75.2% of the 794 cases, 1-3 contacts were enough, but in 6.4% of the cases (n=51) more than 10 contacts were necessary to get the data (table 1).

Unpublished data were provided in less than a month in 53.4% (n=424) of the cases (table 1), but in 9.1% (n=72) of the cases, the authors had to wait for more than six months. The most common reason why authors contacted a specific source was that they knew a trial had been conducted (61.1% of the 794, table 1). The idea to contact a specific data source only came from the Cochrane handbook in 4.2% of the 794 cases. Authors quite often specified that they learnt about unpublished data at conferences, either through personal contacts or abstracts. Trial registers were also used to identify unpublished studies or missing outcome data. Published papers with poor reporting could also be used to find missing outcome data by contacting the authors of the papers. Other sources were whistleblowers, peer reviewers who drew attention to unpublished trials, and meta-analyses of unpublished trials, sometimes done by the manufacturers—for example, pooling premarket studies to increase power. One respondent routinely contacted all manufacturers of a drug and another respondent always approached corresponding authors to confirm the validity of data extraction and to query unpublished data.

In 44.3% of the cases where data were obtained (295/666), the authors investigated a drug intervention. The time involved in searching for unpublished data was the most challenging element (41.0% of 666, table 1). Poor organisation and readability of the data was challenging for 20.9% and 9.8% of the 666 respondents who obtained data, respectively. Thirty seven per cent had no problems and 16.4% specified challenges not covered by the standard answers. The most common reasons were that the authors did not receive data or did not receive a reply (see examples in box 3).

Data for one trial were provided in an old database format that was very difficult to access and navigate

Delineating what was useable from what was not, especially as we were not replying on study design as a filter—this made it a nightmare

We just weren’t sure what we had been sent was right there were discrepancies between published report and data provided. When I asked the author for clarification they did not respond

In the one case that I received IPD [individual patient data], I didn’t use this because the amount of data was overwhelming and would have taken too much of my time to decipher

Data obtained

The most common outcome data authors obtained were unpublished summary data from already published trials: this was supplied by 50.8% of the 794 data sources that provided data (table 3). Missing data on outcomes (28.5%) and individual patients (20.5%) was also common. Data on harms were rare (8.4%). A total of 17.5% of the 794 cases had obtained “other” data, which were mostly data on methodological quality (randomisation, blinding, etc). Respondents also acquired subgroup analyses, theses, information about ongoing trials, and reports of protocol modifications that had not been reported. However, some data were partial, redacted, and subject to confidentiality agreements. Most used the acquired data in their review (82.0% of 794, table 1). The most common reason for not using the data was that they were in an unusable form (6.3%). Eight per cent chose “other” and the majority specified they had not used the data because their review was ongoing.

Table 3

 Common sources of outcome data

Sources of outcome data	No (%)*
Missing data:
Summary data (mean, standard deviation, sample size, etc)	403 (50.8)
Missing outcomes (for example, quality of life)	226 (28.5)
Individual patient data/raw data	163 (20.5)
Alternate analysis (for example, intention to treat)	96 (12.1)
Data on harms	67 (8.4)
Clinical study reports (regulatory authorities, full report)	45 (5.7)
Unpublished trials:
Outcomes in summary format only	135 (17.0)
Study report without individual patient data	115 (14.5)
Individual patient data/raw data	63 (7.9)
Protocols	95 (12.0)
Contact information for trialists	77 (9.7)
Correspondence, approval letters, reviewer comments (for example, from regulatory agency)	5 (0.6)
Other (please specify)	139 (17.5)

*Percentage totals >100 because responses are not mutually exclusive.

In around a third (267 of 794) of the cases the authors got information from previously unpublished trials, and in around two thirds (562 of 794, not mutually exclusive) they got additional data from already published trials. It could be suspected that the strategy for accessing unpublished trials compared with unpublished data differed. We performed a post hoc subgroup analysis and found no difference between the number of contacts needed before the author received data.

Drug and device manufacturers

The authors who obtained data from drug and device manufacturers were more likely to have to contact them 10 or more times compared with authors obtaining data from other sources (24% v 5%, P<0.001, table 4). They also more frequently waited for more than one month (74% v 45%, P<0.001), and more frequently the contact was in person or by telephone (36% v 13%, P<0.001). Manufacturers less frequently supplied individual patient data than other sources (12% v 26%, P=0.02). Data from non-manufacturers were more often used and it was more common that the respondents reported that there were no difficulties compared with manufacturers. However, these differences were not significant (P=0.07 in both instances).

Table 4

 Subgroup analysis comparing manufacturer compared with non-manufacturer data sources. Values are numbers (percentages) unless stated otherwise

Variables	Manufacturer (n=50)	Non-manufacturer (n=744)	P value*
Investigated a drug	42 (84)	254 (34)	<0.001
10 or more contacts needed†	12 (24)	39 (5)	<0.001
Waited a month or more†	37 (74)	333 (45)	<0.001
Contacted by telephone or verbally	18 (36)	98 (13)	<0.001
Used the data	36 (72)	515 (83)	0.07
Got individual patient data	6 (12)	196 (26)	0.02
No difficulties encountered	10 (20)	238 (32)	0.07

Each respondent could describe several sources (manufacturers, trialists, etc).

*χ2 test.

†Data dichotomised from original four categories.

Discussion

A large proportion (around three quarters) of Cochrane review authors searched for unpublished data. A large fraction of those who did not search for unpublished data did so because their work was still ongoing, but another large fraction abstained from searching because they did not expect success. Searching for unpublished data from already published trials is problematic because authors may be difficult to locate and rarely respond.8 Around 20% of authors refrained from searching unpublished data because they did not expect them to be reliable.

In our survey, 55.1% of those who searched for data obtained them and most (82.0%) used these data in their review. This suggests that the methodological rigor (or quality) of the data are adequate even though some of the data were of a nature where risk of bias assessment was pointless (additional point estimates, standard deviations, etc). Other studies have also evaluated the reliability of published and unpublished trials without finding differences.9 10

The respondents’ last common concern was that searching for unpublished data was time consuming. This is not necessarily the case. One study8 found that when the source was contacted by email, the reply arrived within a median of one day. In our survey, more than half of the authors had received their data within a month. But even though it might be time consuming, completely omitting searching for unpublished or inadequately reported data is a risky strategy, as such data will generally be less positive than published data.1 2 4 5 11 Several respondents refrained from searching unpublished data because they found published studies, but this strategy cannot be recommended as, on average, it leads to biased reviews.

Some respondents thought that the trial search coordinator in Cochrane review groups searched for unpublished data, which may not be the case. Trial registers should always be consulted and this could be done by the search team. However, querying authors for missing outcomes or missing data and additional studies can only be done by the authors of the review, who have in-depth knowledge of the literature. Lastly, some respondents abstained from searching because of previous demands for authorship. This is, hopefully, rarely the case and should not discourage authors from searching for unpublished trials.

Almost half of the respondents who sought unpublished data obtained none. The most common reasons were that they never received a reply or were told that no data were available. Another 54 were told it was too much trouble to deliver the data. In a few cases, confidentiality and lack of interest in helping were the obstacles. We have experienced a drug company that only wanted to deliver unpublished data to a Cochrane review if they saw the draft manuscript. This was obviously unacceptable as the delivery of data should not depend on what the drug company or any other data source thinks about the preliminary results.

It was surprising that only 6.3% of authors got data from drug and device manufacturers. When our respondents tried to obtain data from manufactures they experienced longer waits, received fewer individual patient data, needed to make more requests, encountered more difficulties, and were less likely to be able to use the data. Owing to the low response rate in this study these associations should be interpreted with caution. It is nevertheless of concern that only 6.3% of authors received data from manufacturers as a large proportion of research funded by drug manufacturers remains unpublished.12 The respondents who were successful more often contacted manufacturers by telephone or verbally (36%) than they did the non-manufacturers (13%). From respondents’ comments we learnt that authors often knew that manufacturers had data because one of their own authors had been involved in the trials. On at least two occasions, respondents were told by drug manufacturers that their clinical question was not sufficiently relevant for the data to be released (see box 2). It has been well documented that manufacturers often refuse to share data.13

We had expected that research ethics committees and funders would rarely be a source of information, but it was unexpected that company owned trial registers and non-commercial trial registers in particular were also rarely a source of information (0.9% and 6.3%, respectively). The company owned trial registers might not contain relevant information, and non-commercial trial registers should be used more.

The authors often became aware of unpublished data through colleagues and websites. Only 4.2% got the idea from the Cochrane handbook to ask a specific source for data, despite the handbook containing a detailed section about searching for unpublished data.

Regulatory agencies are uncommon sources of data even though the FDA website has contained a lot of valuable data for decades, and even though the EMA opened up its archives in 2010.14 Among the respondents who searched for data in 2011 and 2012 only 5% got data from regulatory agencies compared with 3% for our entire population.

Among the 24 authors who obtained data from regulatory agencies, only seven got full reports and only one unique review incorporated the data in the review. Some authors might not be aware of the amount of accessible data at regulatory agencies. We therefore suggest that the Cochrane handbook should mention regulatory agencies as a source of unpublished data and provide specific guidance on how to search the websites of the FDA and EMA as they are difficult to access.

Almost 21% of authors got individual patient data, primarily from trialists. Authors should be encouraged to request this type of data, and it can probably be done without compromising the response rate.15

How to obtain data

The respondents in our survey most commonly sent emails to corresponding authors, agencies, and companies, and this has also given the best response rates previously.15 A combined approach with both email and letter might be even better.15 Asking specific compared with open ended questions improves response rates.15 This was also the experience of several of our respondents.

Guidance on how vigorously authors should search for unpublished data is needed. Our results suggest that authors should routinely ask trialists for more data when conducting a review. Searching a trial register (clinicaltrials.gov or similar) is also a good idea and is not time consuming. For data on drugs and devices, we suggest that the authors contact the regulatory agencies first, as it is time consuming and generally disappointing to go to the manufacturers.

Limitations of this study

The response rate in our study was low. A substantial part of our emails may have reached inactive email boxes or been caught by spam filters. We sent our survey to busy authors, and previous research has shown that time is a barrier and some people routinely bin surveys.16 Our respondents were probably more likely to search for unpublished data than the authors who did not reply. Our sample might therefore not be representative of all authors of Cochrane reviews.

Conclusion

Most authors who searched for unpublished data received useful data, primarily from trialists. Manufacturers and regulatory agencies were seldom sources of unpublished data.

Unpublished data are less positive than published data

Omitting unpublished data in meta-analyses can bias the results

Authors of Cochrane reviews often search for unpublished data (75.8% in our sample) and around half of the authors who did, succeeded

Drug and device manufacturers infrequently provide data

Drug regulatory agencies should be used more