Poor description of non-pharmacological interventions: analysis of consecutive sample of randomised trials.

OBJECTIVES: To evaluate the completeness of descriptions of non-pharmacological interventions in randomised trials, identify which elements are most frequently missing, and assess whether authors can provide missing details.
DESIGN: Analysis of consecutive sample of randomised trials of non-pharmacological interventions. DATA SOURCES AND STUDY SELECTION: All reports of randomised trials of non-pharmacological interventions published in 2009 in six leading general medical journals; 133 trial reports, with 137 interventions, met the inclusion criteria. DATA COLLECTION: Using an eight item checklist, two raters assessed the primary full trial report, plus any reference materials, appendices, or websites. Questions about missing details were emailed to corresponding authors, and relevant items were then reassessed.
RESULTS: Of 137 interventions, only 53 (39%) were adequately described; this was increased to 81 (59%) by using 63 responses from 88 contacted authors. The most frequently missing item was the "intervention materials" (47% complete), but it also improved the most after author response (92% complete). Whereas some authors (27/70) provided materials or further information, other authors (21/70) could not; their reasons included copyright or intellectual property concerns, not having the materials or intervention details, or being unaware of their importance. Although 46 (34%) trial interventions had further information or materials readily available on a website, many were not mentioned in the report, were not freely accessible, or the URL was no longer functioning.
CONCLUSIONS: Missing essential information about interventions is a frequent, yet remediable, contributor to the worldwide waste in research funding. If trial reports do not have a sufficient description of interventions, other researchers cannot build on the findings, and clinicians and patients cannot reliably implement useful interventions. Improvement will require action by funders, researchers, and publishers, aided by long term repositories of materials linked to publications.

Introduction

Secret remedies—branded drugs whose ingredients were kept secret—were once common, until successful campaigns in the United States and United Kingdom in the early 20th century required labels to include all ingredients.1 This policy allowed independent evaluation of treatments and provided clinicians and consumers with the means to understand what they were considering using.

However, a treatment consists of more than the list of ingredients: the dose, frequency, monitoring, titration, mode of delivery, and duration of use can all influence efficacy and safety. Such details are often not well described in trials. For example, a recent analysis found that only 11% of 262 trials of cancer chemotherapy provided complete details of the trial treatments.2 The most common missing elements were dose adjustment and pre-medications, but 16% omitted even the route of administration of the drug. The completeness of descriptions of treatments may be worse for non-pharmacological interventions: one analysis found that 67% of drug treatment descriptions were adequate compared with only 29% of non-drug treatments. However, this analysis also included systematic reviews, which were less informative on average than were reports of individual trials.3

Unlike the bogus “secret remedies” of the 19th century, the current incomplete descriptions of interventions are likely to arise from poor communication and lack of awareness of the matter among authors and lack of attention by reviewers and editors. However, the consequences are similar: other researchers cannot replicate and build on research findings, and clinicians and patients cannot reliably adopt interventions shown to be useful.

To better understand what elements are most frequently missing from descriptions, and whether this is remediable, we aimed to assess the completeness of descriptions of non-pharmacological interventions in reports of trials published in major general medical journals, using a checklist to evaluate the completeness, and to assess whether missing details can be obtained by contacting authors of trial reports.

Methods

Search strategy and selection of reports of trials

We included all reports of randomised controlled trials of non-pharmacological interventions published in 2009 in one of the six leading general medical journals (based on ISI Web of Knowledge impact factor for 2010): New England Journal of Medicine, JAMA, Lancet, Annals of Internal Medicine, PLOS Medicine, and BMJ. We defined non-pharmacological interventions as interventions including surgery, technical procedures, devices, rehabilitation, psychotherapy, behavioural interventions, and complementary and alternative medicine.4

We excluded reports if they evaluated a pharmacological intervention, a screening programme, or a diagnostic technique (for example, endoscopy) or if they reported secondary analyses, such as an economic analysis of a non-pharmacological intervention trial for which the main results had been published elsewhere or before 2009. Reports were eligible if a non-pharmacological intervention had been compared with a pharmacological intervention or when a complex intervention included a pharmacological component if the major focus was a non-pharmacological component. When reports contained evaluations of more than one non-pharmacological intervention (such as by using three groups or a factorial design), we rated the descriptions of each intervention separately.

An experienced medical librarian searched PubMed in April 2011, using the restrictions of year (2009), publication type (“randomized controlled trial”), and journal title (the six chosen journals). Two authors (TH and PG) screened the 358 titles and abstracts retrieved, identified reports that might meet the inclusion criteria, and retrieved the complete and unabridged reports (n=138). Disagreements about unclear eligibility were resolved by discussion.

Rating of intervention descriptions

Before rating the reports in our study sample, we rated a sample of reports that met our eligibility criteria but had been published in 2008, discussed responses, and continued rating reports until high agreement was reached. Two authors (TH and PG) independently used a checklist to rate the description of the interventions in each eligible report. The checklist is based on the work of Davidson and colleagues and the CONSORT extension statement for non-pharmacological interventions and has been used, in a slightly modified form, in a previous study.4 5 6 The checklist contains eight items (table 1). Seven of the items are rated as “Yes” (indicating that the element of the intervention had been clearly described) or “No” (not reported or not clearly described). Each item in the checklist had an explanatory statement that guided the raters in the scope and interpretation of the item. For “No” responses, we recorded detailed comments about what was missing. The eighth item was an overall question: “Is the description of the intervention complete?” Reports could score a “Yes” on this item only if all other items had been rated as “Yes.” We also routinely and systematically assessed whether the intervention details might have been described further in other sources (such as websites, online appendices, or reference materials). If so, we used these additional sources when completing the checklist items. Disagreements in ratings were resolved by discussion, with reference to the explanatory statement for the relevant item/s, between the two raters.

Table 1

 Items in checklist used to assess reports of randomised trials of non-pharmacological interventions

Checklist item	Elaboration of item
Setting	Is it clear where the intervention was delivered?
Recipient	Is it clear who received the intervention, and do you know all that you need to know about the participants?
Provider	Is it clear who delivered the intervention?
Procedure	Is the procedure (including the sequencing of the technique) of the intervention sufficiently clear to allow replication?
Materials	Are the physical or informational materials used adequately described (and available)?
Intensity	Is the dose/length of individual sessions of the intervention clear?
Schedule	Is the schedule (interval, frequency, duration, or timing) of the intervention clear?
Missing (overall)	Is the description of the intervention complete?

For all reports with any checklist items rated as having an incomplete description, the raters generated a list of questions about element/s of the intervention that were missing or unclear. We emailed these questions to the corresponding author. If no response was received within four weeks, we sent up to two reminder emails at four weekly intervals. When responses were received from authors, the raters collaboratively re-rated the relevant items as either “Yes—clear after author reply” or “No—not clear after author reply.”

Data analysis

We entered initial and follow-up ratings of reports and questions emailed to authors into a customised database. We used Excel to analyse data descriptively.

Results

Of the 138 reports for which we were able to obtain full texts, we excluded five (three economic evaluations and two long term follow-up results) and four reports contained evaluations of two non-pharmacological interventions (see supplementary figure). Our final sample thus contained published reports of 137 interventions from 133 trials. Table 2 shows the journals in which the reports had been published and the categories of intervention that had been evaluated.7

Table 2

 Publication source of trials (n=133) and categories of interventions (n=137) evaluated

Source/category	No (%)
Journal
BMJ	39 (29)
JAMA	29 (22)
New England Journal of Medicine	23 (17)
Lancet	22 (17)
Annals of Internal Medicine	15 (11)
PLOS Medicine	5 (4)
Categories of interventions evaluated
Education and training	23 (17)
Device	23 (17)
Surgery or perioperative intervention	19 (14)
Complex intervention	17 (12)
Diet	16 (12)
Exercise or physical therapy	15 (11)
Service delivery	9 (7)
Other (such as sand as playground surface)	8 (6)
Psychosocial intervention	6 (4)
Complementary and alternative therapy	1 (0.7)

Completeness of intervention descriptions

Figure 1 shows, for each of the checklist items, the percentage of interventions that were clearly described in primary reports and after reply from authors of trial. Overall, 53 (39%) of the interventions were adequately described in primary reports, and this increased to 81 (59%) after contact with authors. The checklist item about intervention materials scored most poorly in primary reports (complete in 64 (47%) of interventions) but was also the item that showed the most improvement (complete in 126 (92%)) after reply from trial authors.

Fig 1 Percentage of interventions rated as adequately described, in primary report and after author reply, for each checklist item

We sent questions to authors of reports about 88 of the interventions (from 84 trials), with a mean of 2.7 (SD 1.5, range 1-6) questions per author. We received no response from 25 authors after two reminders. On the basis of the responses from the 63 authors who replied, the overall rating changed from “No” to “Yes” for 29 of the interventions and remained as “No—description not complete” for 34 interventions (for example, requested materials were not sent or the response did not clearly answer the question). Table 3 provides verbatim examples of incomplete reporting of descriptions of interventions from the primary reports.

Table 3

 Examples of poor reporting of intervention elements in primary reports (key phrases underlined)

Checklist item	Verbatim examples of poor reporting*	Reason for initial rating as “Not reported or not clearly described”
Setting	“We conducted a randomised, sham-controlled study involving 24 patients with stroke (11 men and 13 women). The median age was 62 years (range, 53 to 71), and the median time since stroke was 14 months (range, 7 to 21).”	No details given about setting of intervention—for example, outpatient setting, community setting, or in participants’ homes (author clarified in an email)
Provider	“The exercise training consisted of 36 sessions of supervised aerobic exercise training (ie, walking, treadmill, or stationary cycling) at 60% to 70% of heart rate reserve 3 times per week followed by prescribed home-based training at the same intensity 5 times per week.”	No details provided about who supervised training and their role in supervising training
Procedure	“Patients in the intervention group followed a standardised exercise protocol tailored to individual achievement and were supervised by a physical therapist. The programme consisted of a general warm up on a bicycle ergometer followed by static and dynamic muscular exercises for the quadriceps, adductor, and gluteal muscles. The programme also included balance exercises and flexibility exercises for major thigh muscles.”	Details of the procedure, including “standardised exercise protocol” are not clear (author provided further details in an email)
	“Behavioral counseling was integrated into the group and individual sessions to promote adherence to the assigned diets.”	Details of the behavioural counselling not clear (author provided further details in an email)
	“Based on previous research related to maternal dissatisfaction with peer support, the peer volunteers were requested to make a minimum of four contacts and then to interact as deemed necessary.”	Procedure of intervention not clear
Materials	“Patients randomised to the intervention joined a manual based, self directed, physical rehabilitation programme developed by physiotherapists and introduced by a study nurse.”	Procedure of intervention not clear, and details about accessing manual not provided in report (author provided manual and details of procedure in an email)
	“A 90-minute, semiscripted group session that was led by the genetic counsellor. . .”	Neither script nor details about how to access it was provided in report
	“We offered two half day training seminars for 20 health professionals in each locality: one on group facilitation skills led by an external consultant and one on trial conduct, protocol, and data collection. We provided a written training pack and a password protected website with access to all training materials.”	No details about accessing training materials were provided in paper (authors provided materials after request via email)
Intensity	“The intervention was delivered by 1 nurse during bimonthly telephone calls.”	Duration of telephone calls (planned or actual) not reported (author clarified in an email)
	We offered participants assigned to moderate-intensity disease management up to 2 telephone-based counselling sessions every 6 months (Ellerbeck)	Length of counselling session not known
Schedule	“Each encounter included a core group of modules . . . plus additional modules activated at specific intervals.”	“Specific intervals” are not provided (author clarified details in an email)
	“Nutritionists and dietitians gave dietary advice to participants in both groups in monthly sessions in the first year and bimonthly sessions thereafter.”	Not clear how long bimonthly sessions continued for (follow-up in trial was 4 years)

*Details of sources of examples available on request.

Process of obtaining complete description of intervention—an example

Figure 2 illustrates the key steps in the study methods and the process of obtaining a complete description of an intervention for one of the included reports. After the initial rating of this report,8 four items—setting, provider, procedure, and materials—were assessed as incompletely described, so the overall rating item (“Is the overall description complete?”) was “No.” We emailed the corresponding author with five questions (shown in fig 2). On the basis of the responses, each of these items and the overall rating were changed to “Yes.” By contacting the author, we also learnt that even though the trial used a DVD that was not available, an iPhone app (in Dutch only) containing the exercises and exercise schedule was now available, as was a YouTube clip of the exercises. However, neither of these resources was mentioned in the 2009 report.

Fig 2 Illustration of process of obtaining complete description of an intervention (from: Effect of unsupervised home based proprioceptive training on recurrences of ankle sprain: randomised controlled trial8)

Materials used in interventions

Most of the questions that were sent to authors were about materials used in the intervention. We emailed 70 authors with at least one question about materials and received responses from 27 authors that enabled the rating of the materials item to be changed to “Yes.” Typically this was because authors sent copies of the materials used (for example, written materials used with study participants, such as patient education materials or staff training materials) or provided information about how to access intervention materials or further details (such as a website or journal article that was not referenced in the primary report). Of the other 21 authors who responded, the rating for the materials item, and the overall rating, remained as “No.” The box shows typical reasons for this.

Reasons given for study intervention materials being unavailable

Category of reason (number of authors providing a response in this category) and illustrative quotes from authors:

“Due to legal copyright restrictions at my university I am unable to send”

“Not publicly available because we based them on materials provided by our local government”

“Not publicly available—only to our trainers”

“Not yet—they will be made publicly available within two years”

“No it is not. Attached is a table of contents”

“The training materials from the trial are not online—we had no real reason to do that”

“People originally in the position have moved on”

“I am unable to find . . . my old computer files”

“I’m afraid I no longer have access to those materials”

“I do not have it”

“I am not able to answer most of your questions. I was not involved with running the trial, only analysing and reporting on the QOL results after the data was collected”

“I can’t provide these”

“You will have to read the literature”

“No, is in Dutch”

“The [materials] are tailored, thus it is difficult to disseminate. We could send an example”

“URL doesn’t exist anymore”

“We had been making it previously available, but need to update it, so are no longer”

Further intervention information contained on websites

For one third (n=46) of the interventions, a relevant website existed that contained further information about the intervention or at least some of the materials used in the intervention. Only 35 (76%) of these websites were mentioned in the report and, of these, just over half (19; 54%) were freely accessible. Many websites were behind a paywall, and for others the website address provided in the report no longer existed. Several interventions had relevant websites, but we learnt of these only by contacting the author or through internet search engines. Figure 3 summarises these websites according to how we found out about the website’s existence (for example, mentioned in the original report, email from author), whether any website links provided worked, and whether the site content was accessible free of charge.

Fig 3 Access to and source of websites that contained additional intervention information

Discussion

More than half (61%) of the interventions assessed in this study were not described in sufficient detail in the published primary report to enable replication of the intervention in practice. This problem is partly remediable: a third of the incompletely described interventions could be completed by contacting study authors for further information. Obtaining this additional information took some effort: compilation of omissions, up to three emails, and subsequent piecing together of information from disparate sources. Clinicians wishing to use an intervention in practice are unlikely to invest this amount of time in obtaining the necessary details and materials. Even for trials in which the intervention was not effective, complete descriptions of interventions are important for other groups of research users. For example, this includes other researchers (who may wish to build on the trial or modify the intervention in some way) and systematic reviewers (who need details of intervention to assist with assessing and understanding heterogeneity).

Most interventions included some materials, such as written materials for educating patients or materials for training staff, without which the interventions cannot be used in practice. Despite their importance, just over half (53%) of the reports neither sufficiently described these materials nor gave details about how to obtain copies. Consequently, questions about materials were our most common question to authors. Comments from some authors suggested a lack of awareness of the importance of making intervention materials available. Other authors were reluctant to make materials available publicly (often owing to real or perceived concerns about copyright or intellectual property). Sometimes, corresponding authors no longer had copies of the materials or were uncertain about intervention details.

Several previous studies have documented deficiencies in the descriptions of interventions: an analysis of patient education interventions found that only 17% were replicable, with information about the content of sessions most commonly missing9; none of 11 trials of music therapy provided all necessary details10; and an analysis of 158 surgical studies found that only 41% provided some details of the surgical procedure.11 However, only one previous study seems to have included writing to authors to obtain additional information; a checklist was not used, and only a small sample of non-pharmacological interventions was included.3

Recommendations

The omission of essential information about interventions is a substantial, yet remediable, contributor to the enormous worldwide waste in research funding that occurs because research is unpublished or unusable.12 Reducing this waste will require action by funders, researchers, and publishers at multiple stages, including pre-submission, editorial review, and publication. However, missing details in the reports of intervention studies is part of a wider problem of non-replicable interventions in trials and systematic reviews. This has been the topic of a consensus meeting,13 which made several recommendations. Two of these are relevant to the problems identified in this research.

A key recommendation was that: “The reporting standards for interventions in trials (CONSORT, etc) and systematic reviews (PRISMA) should be improved and standardised (specific checklists).” The CONSORT extension statement for the reporting of randomised trials of non-pharmacologic treatment contains four intervention related items: precise details of the intervention, description of the intervention components (and where applicable, description of tailoring procedures), details of how interventions were standardised, and details of how adherence to the protocol by care providers was assessed.4 However, this CONSORT extension does not mention intervention materials, which we found to be the most poorly reported, and remediable, element. The mere existence of an appropriately detailed checklist is unlikely to be sufficient to improve quality of reporting. Few journals request the use of extension statements, and, even when endorsed by journals, subsequent adherence to them by published reports does not necessarily occur.14 Journal editors and reviewers have a responsibility to be aware of the importance of complete reporting of non-pharmacological interventions and to implement policies and processes to ensure that this occurs.

A second recommendation was that: “A stable ‘intervention bank’ should be established (e.g. videos, manuals, and fidelity tools linked to trial registration number) to overcome the problem of word restrictions in journals, etc.” Although hosting intervention materials on websites may seem to be a logical way of making them available, problems such as maintaining operational sites were already apparent in this sample of quite recent reports. One striking finding was that a third of the trials had a website with additional intervention information/materials, but a quarter of these were not mentioned and many were not freely accessible or the URL no longer worked. This suggests that responsibility for maintaining websites that contain intervention details should not lie solely with journals, authors, or their institutions, but instead with organisations that have greater stability and longevity. Similar to drug treatments in pharmacopoeias, an equivalent compilation for non-drug interventions is needed.15 16

Strengths and limitations of study

The strengths of this study include the wide range of non-pharmacological interventions, duplicate rating, and the verification of missing details and materials with authors. However, 28% of authors did not respond, limiting the completeness of this verification. A further limitation is that we selected reports only from leading general medical journals, and findings may not be generalisable. If the quality of reporting of interventions is similar to the quality of reporting of methodological features in randomised trials, which is better in journals with a higher impact factor,17 then our study may have underestimated the size of the problem. Our study may have also found an even larger problem of poor reporting if we had assessed whether the reports had evaluated or described the fidelity of the intervention, which has been identified as an additional element that should be included in descriptions of interventions.5

Further work needed

Further work is needed in several areas. Firstly, we need to better understand why authors do not provide more complete descriptions of interventions. Secondly, we need to develop the guidelines and tools to assist authors and editors in providing complete descriptions of interventions. Several of the extensions to the CONSORT statement have been motivated by a need to obtain better details of interventions, but these differing extensions need harmonisation to clarify what is generic and what is specific to different types of interventions. As many trials now publish a protocol, providing additional intervention detail in the published protocol may be one way of overcoming word limit restrictions in the trial’s primary paper. The recently published reporting guideline for trial protocols, the SPIRIT 2013 Statement,18 contains four items (11a-d) about the reporting of the intervention. Further work is needed to ensure that the primary paper makes explicit mention of all related documents (such as protocols, online supplementary material, and websites) so that readers can easily obtain a complete description of the intervention.

Although reporting standards are necessary, they are unlikely to be sufficient. Development and enhancement of tools to assist with describing interventions would be useful. One such tool is PatPlot,19 which provides a graphic depiction of the elements and sequencing of complex interventions. However, given the documented substantial deficiencies in reporting of interventions and the demonstration that it is partly remediable, authors and editors should take action now to reduce this waste in research.

Incomplete descriptions of interventions render the intervention uninterpretable and unusable by clinicians, patients, and researchers

The completeness of descriptions of interventions may be worse for non-pharmacological interventions than for pharmacological interventions

Missing information is a common problem (occurring in more than 50%) in reports of non-pharmacological interventions

Details of intervention materials and procedures were the most commonly missing elements

The problem can be partially remediated by contacting authors of trial reports for missing details