Pragmatic trials of pain therapies: a systematic review of methods.

ABSTRACT: Pragmatic randomised clinical trials aim to directly inform clinical or health policy decision making. Here, we systematically review methods and design of pragmatic trials of pain therapies to examine methods, identify common challenges, and areas for improvement. Seven databases were searched for pragmatic randomised controlled clinical trials that assessed pain treatment in a clinical population of adults reporting pain. All screening steps and data extractions were performed twice. Data were synthesised descriptively, and correlation analyses between prespecified trial features and PRECIS-2 (PRagmatic-Explanatory Continuum Indicator Summary 2) ratings and attrition were performed. Protocol registration: PROSPERO-ID CRD42020178954. Of 57 included trials, only 21% assessed pharmacological interventions, the remainder physical, surgical, psychological, or self-management pain therapies. Three-quarters of the trials were comparative effectiveness designs, often conducted in multiple centres (median: 5; Q1/3: 1, 9.25) and with a median sample size of 234 patients at randomization (Q1/3: 135.5; 363.5). Although most trials recruited patients with chronic pain, reporting of pain duration was poor and not well described. Reporting was comprehensive for most general items, while often deficient for specific pragmatic aspects. Average ratings for pragmatism were highest for treatment adherence flexibility and clinical relevance of outcome measures. They were lowest for patient recruitment methods and extent of follow-up measurements and appointments. Current practice in pragmatic trials of pain treatments can be improved in areas such as patient recruitment and reporting of methods, analysis, and interpretation of data. These improvements will facilitate translatability to other real-world settings-the purpose of pragmatic trials.

1. Introduction

Increasingly, alternatives to the classical placebo-controlled randomised clinical trial (RCT) are proposed. The main criticism of traditional RCTs concerns the lack of generalisability of research findings due to key aspects of the trial design,[98] including exhaustive exclusion criteria (comorbidity, polypharmacy, psychiatric illness, and substance use disorder),[4,98,100,107,121] trial populations differing from the general patient population,[72,79] and unrealistic treatment compliance.[24,60,73] Maybe most importantly, what matters to a patient may not have been assessed in an RCT: To be relevant for clinical decision making, statistical changes in outcome measures need to be reflected in clinically noticeable and personally valuable changes in symptoms, quality of life, or disease risk.[25,31,112,120] Even so, the time horizon of a patient's decision is rarely encapsulated by common RCT follow-up periods that are usually 6 months or less.[32] Despite this lack of generalizability, RCTs still form the basis of most health policies, medicines regulatory approval, and treatment guidelines[43,45,57,89] because they allow for controlling most factors apart from the intervention as well as realistically possible.

Pragmatic trial designs have been proposed as a possible remedy to bridge the gap between highly controlled RCTs and clinical practice. The concept of “pragmatism” refers to the research aim of directly informing a healthcare or health policy decision, especially in situations where there is a choice between 2 or more options.[17,40,56,103,115] Importantly, “pragmatism” in clinical trials is best viewed as a continuum, the 2 poles being explanatory (efficacy) RCTs and pragmatic (effectiveness) trials[68,115]: many RCTs entail pragmatic elements to increase their external validity, whereas some “pragmatic” trials use methods such as placebo control and blinding.[42]

By concept, pragmatic trials are large in scale, embedded into ongoing clinical practice, and frequently investigate complex interventions. Although patients are still randomly assigned to treatment groups, they are rarely blinded to their allocation. In addition, the treatment protocol is deemed flexible, eg, allowing clinicians to adjust drug therapy to individual patients. Outcome measures are believed to reflect what is important in clinical practice, focusing on disability and function, risk-benefit analyses, or even cost-effectiveness, rather than average pain scores.[17,42,78,96,124] At the extreme end of the explanatory–pragmatic spectrum, pragmatic trials assess outcome data sampled routinely in clinical practice and alter routine care minimally or not at all. To enable clinicians to judge how relevant a study's findings are to a particular clinical scenario, many have called to improve reporting of features associated with the external validity of trials (such as details of the study population, provider expertise, treatment centre volumes, and intervention standardisation).[11-13,33,68,99] Tools are available to guide pragmatic trial design[68,114] and an extension of the CONsolidated Standards of Reporting Trials statement (CONSORT) exists for pragmatic trials.[130]

1.1. Aims and objectives

Pragmatic approaches to trial design have been promoted with the goal of increasing the relevance of clinical trials to real-world decision making and policy implementation. To understand the current specifics of trial design, conduct, and reporting in the field of pragmatic trials of pain treatments, the objectives of this review were as follows:

Survey the number of randomised controlled trials that are declared to be “pragmatic” or “comparative effectiveness” and that are investigating any therapy aimed at pain reduction in an adult human population experiencing clinical pain.

Identify which therapeutic interventions have been assessed in such trials.

Evaluate the prevalence of individual design features relating to the concept of “pragmatism” among the included studies.

Determine areas for future debate and research within the field of pragmatic trials of pain treatments.

Notably, the aim of this review was not to gauge trials' risk of internal bias or review the effectiveness of treatments.

2. Methods

2.1. Protocol registration

A protocol formulated in accordance with the 2015 statement of Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)[76,86] and detailing both the review methods and analysis plan was preregistered with the International Prospective Register of Systematic Reviews (PROSPERO) before commencing data extraction[10] ([55], registration ID: CRD42020178954). Ethical approval was not required.

2.2. Eligibility criteria

We reviewed any RCT,[126] declared by the study authors to be “pragmatic,” “practical,” or “comparative effectiveness research.” To be included, studies had to investigate people with clinical, ie, nonexperimental pain (including procedure-related pain, irrespective of their age, sex, underlying pathology, or the severity and duration of their pain). All interventions aimed at reducing pain in a clinical population or at affecting an outcome measure relevant to the treatment or management of people in pain were eligible, irrespective of treatment setting or delivery format. Trials were included where pain or pain-related measures formed part of the primary analysis or where the primary aim was to assess end points directly relevant to the treatment and management of patients in pain and not administrative processes or diagnosis. No geographic restrictions were applied. Included trials had to have a control or comparison group, but the type of comparator was irrelevant for study selection. Within-patient controls were not eligible. Retrospective and observational studies were excluded, as were studies drawing exclusively from registry data. We excluded feasibility or pilot studies to capture the challenges of conducting full-scale pragmatic trials, and a minimum of 40 participants per study arm was required. Primary outcome reports had to be published in peer-reviewed sources between January 2018 and March 2020. This time frame was chosen for several reasons: the rapidly evolving nature of the field,[128] the aim to capture the status quo to inform future methods development, and also because the last milestone article for the design of pragmatic trials, the PRagmatic–Explanatory Continuum Indicator Summary (PRECIS) 2 tool,[68] was published in 2015 and we deemed 3 years to be the minimum amount of time for this recommendation framework to be reflected by the published reports of pragmatic trials. Studies published in the languages English, German, Spanish, Italian, French, and Mandarin were eligible, and others if translations could be obtained. Studies were excluded if no full text could be retrieved, neither online nor through the corresponding author.

2.3. Information sources

The following databases were searched from January 1, 2018, to March 1, 2020: MEDLINE, Embase, and PsychINFO (through Ovid interface); the Cochrane Central Register of Controlled Trials; NIH Clinicaltrials.gov; CINAHL (nursing and allied health, through EBSCO); and the Physiotherapy Evidence Database (pedro.org.au). As pragmatic trials were expected to be relatively large and costly, we did not anticipate publications in the gray literature and no such sources were searched. Reference lists of included studies were reviewed for additional eligible studies. Systematic reviews or meta-analyses were used as sources of further primary studies. We consulted trial registries or contacted authors electronically to identify the trial status when protocols were retrieved. Similarly, authors were contacted if full reports of potentially eligible trials could not be obtained. For any included study, protocols were consulted for additional information during data extraction.

2.4. Search

Medical Subject Headings (MeSH) or equivalent and text-word terms were used and are provided in full as supplement (available at http://links.lww.com/PAIN/B374): pain OR painful conditions (ie, specific disease names) AND (pragmatic trials OR practical trials OR comparative effectiveness). Limit: human studies, 2018 to current. The search strategy was developed in an iterative manner and under consultation of published literature, designated experts who are part of the research team (pain researchers, trial designers, and therapists), as well as experts in systematic review methodology and database searching. The full search string is provided as supplementary material (available at http://links.lww.com/PAIN/B374).

2.5. Study selection

Before screening, search results were imported into EndNote (X9) and duplicates removed. For subsequent screening, the studies were exported from EndNote into Covidence, an online platform for systematic reviews (Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org), and another automated deduplication was performed. Eligibility screening was performed in duplicate, ie, screened twice by independent reviewers (D.H.-S., B.A.K., E.M., J.D.-R., M.F., J.C., and J.P.). Disagreements were resolved through discussion or, if not possible, by a third party (D.H.-S. or R.H.D.). In a first step, screening was performed based on study title and abstract. For studies conforming with the eligibility criteria at this stage and not meeting any of the exclusion criteria, full-text publications were accessed and again screened in duplicate.

2.6. Data collection process

Like the screening process, the data extraction required a minimum of 2 independent reviewers. Discrepancies were resolved through discussion and involved a third party if necessary. If available, trial protocols were examined for methods not reported in the trial reports. Missing data that could not be retrieved in this way were recorded as not reported. Where reports were judged to be ambiguous, authors were contacted for clarification. The data extraction form was created iteratively and piloted before data extraction.

2.7. Data items

The domains of data extraction were source details, funding, trial methods, outcome measures, analysis methods, discussion and contextualisation of information, and reporting. The full extraction table is available as supplementary file (available at http://links.lww.com/PAIN/B374). The extraction of study methods had a triple focus. First, key aspects of pragmatic trial design, conduct, and analysis were extracted. This included eligibility criteria, treatment provision, and statistical methods. Second, to assess how trialists handled the tension between external and internal trial validity, methods deemed to affect internal trial validity, such as randomization procedures, allocation concealment, and blinding of participants and personnel, were extracted.[110] By extracting information on placebo control groups, blinding, and number of trial settings, currently debated areas of pragmatic trials' design were addressed.[28,129] Furthermore, potential shortcomings in randomization were assessed by means of heterogeneity testing between trial groups, performed on baseline age data.[21] Third, to examine how researchers dealt with the specific challenges of pragmatic trials, information on the discussion and methodological treatment of potential heterogeneity between study arms, differences between multiple study centres, differences in therapist expertise, setting resources, treatment flexibility and fidelity, lack of blinding, prolonged follow-up periods, differential attrition, and study cost (ie, funding information) was extracted. Given that many included studies were expected to be comparative effectiveness trials, they were extracted whether these were designed as superiority or as noninferiority (equivalence) trials.[41]

Complementary to the above-mentioned methods, descriptive data relating to the 9 core domains of pragmatic trial design were sampled, as defined by the PRECIS-2 tool.[68] As part of the data extraction process, trials were rated for each of these domains. The PRECIS instrument has been used both during the design phase of trials,[58] including in pain research,[65] and to retrospectively rate RCTs on a pragmatic–explanatory spectrum.[14,48,62,70,92,105,125] In the latter application, however, some authors have commented on difficulties with interrater reliability and missing or unreported data.[28,80,125,129] For this reason, the rating of PRECIS-2 domains was trialed extensively within the team drawing on seminal publications and their explanations,[42,62,68,114,128,131] published annotations, and examples (https://www.precis-2.org/Trials) as well as the experience of other researchers performing reviews with this tool.[105] Rating occurred in duplicate, and interrater reliability was assessed. Disagreements of more than one (of 5) points were resolved through discussion or expert consultation. Otherwise, the average rating was used. Where domains were not applicable or information was insufficient to perform ratings, domains were left blank, reflecting the current state of the debate in this field.[28,129] PRECIS-2 ratings require comparing a given trial intervention with “usual care.” In studies where “usual care” was not described in detail, reviewers had to draw on their own knowledge of the current practice standard. For ambiguous cases, it had been planned to consult national guidelines or clinicians to inform reviewers' conceptions of respective “usual care” but this was not deemed necessary. Additional data extractions were conducted based on discussions with the review's steering group, including information regarding the content of treatment as usual and details on concomitant pain treatments, risk-benefit, and cost-effectiveness analyses.

Apart from methodological features and PRECIS-2 ratings, recommended reporting items for pragmatic trials were identified, as proposed by the CONSORT statement extension for pragmatic trials.[130]

2.8. Risk of bias in individual studies

Effect sizes of clinical outcome measures were not extracted nor were potential causes for heterogeneity formally examined (apart from heterogeneity arising from randomization, see below). As the purpose of this review was not to judge clinical or comparative effectiveness, a formal risk of bias assessment was not performed.

2.9. Data synthesis

This report was formulated in accordance with the PRISMA statement.[76] Some subheadings had to be adapted to the purpose of this systematic review of trial methods.

The main results of this review are qualitative and presented using descriptive statistics (means, standard deviations, and percentages of total sample) and appropriate graphs. In addition, it was assessed whether certain trial methods were more prevalent under certain circumstances, using appropriate statistical correlation methods and following a prespecified analysis plan.[55] Analysing the data in the above way may help inform the design of future trials by highlighting areas for potential conflict and opportunity in trial design.

2.10. Risk of bias across studies

Risk of bias across studies was addressed by sampling data on reporting quality, which mainly affected the readers' ability to judge the generalizability of results. By extracting and analysing average and dispersion measures of study participants' age, a heterogeneity meta-analysis between groups sought to identify potential shortcomings in randomization procedures.[21,52]

2.11. Additional analyses

Sensitivity analyses examined the above correlations without preidentified covariates. The only deviation from the analysis plan[55] was the addition of 2 subgroup analyses investigating whether PRECIS-2 ratings differed between trials of pharmacological and nonpharmacological pain therapies as well as between trials of acute vs chronic pain.

3. Results

3.1. Study selection

The search resulted in 769 records after duplicate removal. After excluding 527 records based on titles and abstracts and a further 185 based on assessing the full text for eligibility, 57 individual trials were included in the final sample (Fig. 1). Meta-analysis in the sense of descriptive statistics and several correlation analyses was performed on the entire sample of included studies.

Figure 1.

Preferred Reporting Items for Systematic Review and Meta-Analysis flow diagram showing the identification, screening, and selection process of records for this systematic review. Reasons for exclusion at the full-text eligibility screening phase are provided.

3.2. Study characteristics: descriptive statistics

An overview table of included studies is provided as supplementary material (available at http://links.lww.com/PAIN/B374), specifying each trial's patient population, experimental and comparison interventions, primary outcome measures, and time points of primary and longest follow-up. Authors of 9 trials described their study as “comparative effectiveness” but not “pragmatic,” but these trials did not differ significantly from declared pragmatic trials in sample size (t(55) = 0.13, P = 0.9) nor overall PRECIS-2 score (see below).

Pharmacological treatments for pain were the most studied index treatment (21%), followed by cognitive–behavioural and other psychotherapy approaches (16%), surgery (12%), acupuncture or acupressure (11%), manual therapies (also 11%), physiotherapy (7%), and others (Table 1). All trials investigated programmes of complex interventions, such as rehabilitation, manual therapy, cognitive–behavioural therapies, various forms of patient management, surgery or drug regimens, or treatment programmes of several modalities.

Table 1

Treatment modalities and demographics.

	n of trials	%	Trial references
Component therapeutic modalities
Pharmacological therapy	12	21.05	5,20,24,37,45,66,97,99,106,110,113,129
Cognitive–behavioural and other psychotherapy	9	15.79	8, 17, 68, 85–87, 95, 121, 125
Surgery	7	12.28	6,7,51,69,73,89,96
Acupuncture/acupressure	6	10.53	10,21,76,83,84,93
Manual therapy	6	10.53	31,36,50,52,90,104
Physiotherapy	4	7.02	9,103,119,124
Multidisciplinary care (nondrug)	3	5.26	2,3,27,35
General practice (nondrug)	2	3.51	19,30
Rehabilitation	2	3.51	65,77
Body–mind therapies	2	3.51	61,120
Education	1	1.75	108
Automated symptom and treatment side-effect monitoring	1	1.75	2
Virtual reality	1	1.75	111
Dentistry	1	1.75	40
Pain disorder/descriptor
Back or neck pain	19	33.33	19,24,27,30,31,36,45,50,52,66,76,83,90,96,103,104,119,124,125
Peripheral joint pain	10	17.54	7,35,51,73,77,87,89,95,97,121
Arthritis (RA or OA)	8	14.04	6,8,35,37,66,76,87,113
Pain (not further specified)	6	10.53	9,17,68,85,86,111
Postmedical intervention pain	5	8.77	84,96,108,124,129
Abdominal and other visceral pain	4	7.02	10,21,69,110
Neuropathic pain	3	5.26	2,20,106
Headaches	3	5.26	5,61,99
Leg pain	2	3.51	93,103
Postinjury pain	2	3.51	27,65
Tooth pain	1	1.75	40
Diffuse chronic pain (CFS, FM, and CRPS)	1	1.75	120
Musculoskeletal pain (not further specified)	1	1.75	3
Pain duration
Acute	7	12.28	27,30,45,65,84,106,110
Subacute	2	3.51	37,83
Chronic	31	54.39	All others
Mixed	1	1.75	50
Not reported	16	28.07	2,5,9,10,19,31,35,36,40,69,73,85,89,108,111,121
Type of setting
Primary	25	43.86	Not provided, see below for detail
Secondary	20	35.09
Tertiary	17	29.82
Community	5	8.77
Setting specification
Public hospital	23	40.35	All others, unclear in: 83,96
Private hospital	6	10.53	73,90,93,96,99,119
Patient home, phone, text messaging, mail, or online (entirely or predominantly)	6	10.53	17,68,87,104,108,124
Private practice	8	14.04	9,31,36,52,76,77,99,113
Military medical practice	3	5.26	24,50,66
Research institute	1	1.75	104
Rehabilitation centre	1	1.75	8
Emergency Deptartment	1	1.75	45
University teaching clinic	1	1.75	21

All therapies studied across included trials as well as pain disorders, and the average duration of pain is presented. Note: Some pain descriptors have been applied twice, eg, knee arthritis has been classified as “peripheral joint pain” and “arthritis.” Furthermore, some samples included patients with musculoskeletal pain, which may have included “back or neck pain” and “peripheral joint pain.” Depending on individual trial reporting, the average duration of the pain-related diagnosis or the time since the onset of pain was used. Acute pain was defined as pain lasting < 4 weeks, subacute as 4 weeks to 3 months, and chronic as 3 months.

CFS, chronic fatigue syndrome; CRPS, chronic regional pain syndrome; FM, fibromyalgia; OA, osteoarthritis; RA, rheumatoid arthritis.

Concomitant pain treatments were disallowed in 6 trials (11% of applicable cases, N = 55), either by means of eligibility criteria for participants or after enrolment, and 2 further trials discouraged patients from seeking treatment outside the trial. However, 10 trials did not report whether or not concomitant therapies were permissible.

Where allowed, concomitant treatments were unrestricted in 30 trials (68% of applicable cases) and 6 trials permitted any concomitant treatment other than those akin or similar to the study interventions. Some trials excluded individual unrelated interventions, such as injections and surgery,[88] injections alone,[87] physiotherapy,[16] tricyclic antidepressants,[5] or pain medication (not further specified).[59,102] Only one drug trial made specific allowances for medications and dosages, including some of the same class as study interventions and opioids.[35] Another trial changed the regimen from allowing nonsteroidal anti-inflammatory drugs only in the first 6 weeks to applying no restrictions thereafter.[19] Thirty trials (68% of applicable cases) reported detail on concomitant pain-reducing treatment actually received, 4 of which, however, only partly or for specific treatments deemed to relate to the trial intervention (eg, physiotherapy in a treatment-as-usual control group of a physiotherapy trial) but not for others (eg, pain medications[122]).

Although pain associated with the musculoskeletal system was most commonly studied (more than 60% of trials), diffuse chronic pain conditions such as fibromyalgia were only studied by a single trial[118] (Table 1).

In more than half the included trials, the patient population consisted of patients with chronic pain (Table 1). Although this is typically defined as pain lasting for at least 3 months, patients in most trials had been experiencing pain for several years (supplementary table, available at http://links.lww.com/PAIN/B374). In 12% of trials, pain was studied in an acute context, such as pain after injury or associated with medical interventions. Reporting of the sample's exact or even approximate pain duration was, however, poor, with 28% of studies not providing any indication.

The median sample size at the point of randomization was 234 (Q1/3: 135.5; 363.5) with the largest trial featuring a total of 1702 participants[18] and the smallest trial 80.[71]

The median number of trial centres or settings was 5 (Q1/3: 1; 9.25), with one open-label comparative effectiveness trial of drugs for gout flare-ups taking place across 100 general practice clinics across England[95] and 12 (21%) single-centre studies. Seven studies did not report the number of participating treatment centres.

Although 37 studies (65%) did not report the number of providers in the treatment group, the median number of reported therapy providers was 11 (Q1/3: 6.25; 35.5), with Adams et al.[2] having 400 general practitioners take part in their trial.

Only 7 trials (12%) were fully industry funded,[26,29,35,94,97,104,127] 44 trials (77%) had public funders, and 5 (9%) were funded by mixed sources.[50,59,67,83,85] Funding sources were not reported in one trial.[91] Most trials were conducted in the United States (35%), followed by the United Kingdom (14%), Australia (7%), and Norway (7%). Four studies (7%) were conducted in East Asia and one in South America.[38] Only one study was conducted across multiple countries.[94] Protocols were registered for all but one trial.[91] For the purpose of this systematic review, accessing protocols for additional data extraction was deemed necessary in 34 cases (60%).

3.3. General trial methods

All but one of the included trials were parallel group RCTs, with Berdal et al. using a stepped-wedge design[8] and another trial including a crossover option after 12 weeks.[97] The number and nature of groups differed between trials. Most trials (45; 79%) used a two-group design, 10 trials (18%) had 3 groups, and 2 trials (4%) had 5 groups[111,118] (supplementary table, available at http://links.lww.com/PAIN/B374). More than 3-quarters of the trials in the sample were comparative effectiveness (comparative effectiveness research) trials, comparing multiple specific interventions or using treatment as usual as the comparator. Placebo control groups were used in 9% trials and 7% a no-treatment control group. One trial each used one of the following alternative comparators: Waitlist controls, advice only, wait and see, and no-treatment without informing patients of a trial being performed (Table 2). Reporting of the content of treatment-as-usual controls is shown in Table 3.

Table 2

Methods of trial design.

	n of trials	% of sample	Trial references
Comparator
Another active specific therapy (comparative effectiveness)*	29	50.88	All others
Treatment/care as usual*	14	24.56	3, 8, 10, 17, 30, 35, 50, 83–85, 87, 95, 108, 124
Placebo or sham intervention	5	8.77	5,7,45,84,129
No-treatment group (explicitly assigned, ie, patient know they would not get any treatment)	4	7.02	7,9,93,110
Treatment/care as usual plus something else (advice, education, etc.)*	2	3.51	2,93
Waitlist control	1	1.75	125
Advice only	1	1.75	121
Wait and see (not waitlist but monitoring)	1	1.75	77
No-treatment group (but unaware of trial)	1	1.75	19
Recruitment method
Targeted recruitment (such as identification through records)	25	43.86	All others
Convenience sampling	16	28.07	5,6,8,9,20,27,30,40,45,52,65,73,76,89,95,110
Not reported	9	15.79	24,37,61,93,111,113,119,124,129
Mixed (convenience and targeted)	8	14.04	35,50,51,69,83,86,96,97
Method of randomization
Individually randomized	27	47.37
Of which simple randomization	15	26.32	All others
Of which blocked randomization	12	21.05	17,21,27,31,36,37,45,68,73,90,104,119
Stratified by site	15	26.32	5,7,20,24,50,51,61,65,69,76,77,83,96,106,129
Other stratification	9	15.79	66,84,86,89,93,95,103,113,120
Cluster randomised	6	10.53	2,3,8,30,35,121

Used comparators and recruitment and randomization methods are presented. Notes: Multiple comparator groups were possible. The difference between a waitlist control group and a no-treatment control group is that patients expect treatment at a later point or know that they have been assigned to not receiving any treatment, respectively. Categories marked * are deemed part of comparative effectiveness research (CER). Convenience sampling is the recruitment of patients who attend the trial-delivering service anyway, although targeted strategies seek to specifically contact populations of potentially eligible participants. The category of “blocked randomization” includes various ways of blocking, including a single fixed block size, regularly varying sizes, and randomly permuted block sizes. Blocking was occasionally stratified by site. Stratification was usually by trial centres (sites). “Other stratification” includes stratification by sex, diagnosis, or treating surgeon. Cluster randomization refers to trials where the unit of randomization was not patients but, for example, clinics or individual providers.

Table 3

Selected items of the 2010 update of the Consolidated Standards of Reporting Trials (CONSORT) statement (Schulz et al., 2010)1 and all items of the extension for the reporting of pragmatic trials, as published in 2008 by Zwarenstein et al.2

Item (number, section and CONSORT document (1 or 2))	Description of reporting item	Results: number of studies that complied with respective reporting items (n, %)
2: Background2 modified	Describe the health or health service problem that the intervention is intended to address and other interventions that may commonly be aimed at this problem modified.	55 (96.49%)Not complied: 6110
		33 (57.89%)
3: Participants2	Eligibility criteria should be explicitly framed to show the degree to which they include typical participants or, where applicable, typical providers (eg, nurses), institutions (eg, hospitals), communities (or localities, eg, towns), and settings of care (eg, different healthcare financing systems).	39 (68.42%)
5: Interventions1 modified	Precise details of the interventions intended for the intervention group each group and how and when they were actually administered.	57 (100%)
	If the “treatment-as-usual” or “usual care” was used as comparator, provide additional information as to the nature of the intervention(s) available as part of this.	applicable in n = 1712 (70.59%)
	If the “treatment-as-usual” or “usual care” was used as comparator, collect and report data on care received by patients in this group.	(applicable in n = 17)10 (58.82%)
4: Interventions2	Describe extra resources added to (or resources removed from) usual settings to implement intervention.	28 (49.12%)
	Describe the health or health service problem that the intervention is intended to address.	55 (96.49%)
	Indicate if efforts were made to standardize the intervention or if the intervention and its delivery were allowed to vary between participants, practitioners, or study sites.	Not applicable as intervention automated: 1; reported: 46 (82.14% of 56); of those standardized: 35; not standardized: 9.
	Describe the comparator in similar detail to the intervention.	43 (75.44%)
6: Outcomes2	Explain why the chosen outcomes and, when relevant, the length of follow-up are considered important to those who will use the results of the trial.	29 (50.88%)
7a: Sample size1	Report how the sample size was determined.	55 (96.49%)
7: Sample size2	If calculated using the smallest difference considered important by the target decision maker audience (the minimally important difference), then report where this difference was obtained.	Not extracted
8b: Randomization1	Report the type of randomization and details of any restriction (such as blocking and block size).	57 (100%)
10: Allocation concealment implementation	Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions?	43 (75.44%)
11a: Blinding/masking1 modified	Whether participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment.		Not applicable in one case as patients unaware of participating in a trial.[105] Reported in 54 (96.43%) of 56; not reported in Refs. 65,82
			Not applicable in one case as intervention independent of providers. Reported in 55 (98.2%) of 56 relevant trials.
			Reported in 50 cases (87.72%).
11: Blinding/masking2	If blinding was not performed, or was not possible, explain why.		31 (72%) of relevant studies reported reasons (n = 43).
13a: Participant flow1 modified	Flow of participants through each stage (a diagram is strongly recommended)—specifically, for each group, report the numbers of participants randomly assigned.	57 (100%)
	receiving intended treatment	Not extracted
	completing the study protocol	57 (100%)
	analysed for the primary outcome;	57 (100%)
	Describe deviations from planned study protocol, together with reasons.	21 (36.84%) complied. Of those: 10 reported following protocol; deviations with reasons reported in 11; and deviated without providing reasons: 4.
13: Participant flow2	The number of participants or units approached to take part in the trial, the number which were eligible, and reasons for nonparticipation should be reported.	44 (77.19%)
16: Numbers analyzed1 modified	Whether the analysis was by “intention-to-treat”; for each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups	48 (84.21%) reported the primary analysis as “intention-to-treat”; not discernible in one instance[18]; all other did not explicitly report a primary intention-to-treat analysis.
		Of those 48 studies that called their primary analysis “intention-to-treat,” 10 trials (20.83%) excluded participants from the primary analysis who did not provide follow-up data or where data were missing.
19: Harms1	All important harms or unintended effects in each group.	Whether or not significant harms or unintended effects occurred was reported in 47 studies (82.46%).
		Harms did occur in 22 of those studies, and in 9 of those cases there was a significant difference between groups.[23,62,70,80,83,93,94,96,126]
21: Generalizability2 modified	Describe key aspects of the setting that determined the trial results.	21 (36.84%)
	Discuss possible differences in other settings where clinical traditions, health service organisation, staffing, or resources may vary from those of the trial.	19 (33.33%)
25: Funding1	Sources of funding and other support (such as supply of drugs) and role of funders.	40 (70.18%)

Where indicated by italic or bold formatting, items were modified to match the review question or split into several items to be extractable as individual data points. Bar charts indicate the percentage of studies complying with respective reporting items (green) and not complying although applicable (red). If items were not applicable to the entire sample of 57 studies, the applicable number is stated in the respective row.

Participants were reported to be blinded in 13 trials (24% of the trials reporting on participant blinding, n = 54),[2,5,7,8,18,23,29,38,44,82,119,123,127] providers in 4 (7%, n = 55),[2,5,44,127] and outcome assessors in 45 trials (90%, n = 50). Only 5 studies reported unblinded assessment (Table 3).

Fifty-four of 57 trials (95%) reported that patients gave informed consent, with none of the trials stating clearly what this information and consent process entailed. In 2 trials,[30,106] consent for patients was waived. In a third trial,[2] the unit of randomization was physicians for whom consent was not required; patients, however, did provide informed consent. Finally, a single article[104] did not report whether or not patients provided informed consent.

3.4. Outcome analysis and interpretation

Trials were designed as superiority trials in 52 instances (91%), 4 (7%) were noninferiority or equivalence trials,[38,63,71,74] and a single study[29] did not report whether or not the trial was designed to show a difference between groups. “Unsuccessful” superiority trials cannot claim equivalence between interventions[47]; nonetheless, equivalence or comparative effectiveness was reported in 9 of the 24 (38%) superiority trials where no significant difference between groups had been demonstrated.[6,7,23,50,59,101,104,111,118] Despite the recommendation to include a third (placebo) control group in noninferiority trials to account for the trial-specific possibility of no demonstrable effect beyond placebo in the control group,[41] none of the 4 noninferiority trials included a placebo control group.

Of 15 trials with multiple outcome measures defined as primary outcomes, 9 (60%) did not address the issue of multiplicity in their analysis.

3.5. Adherence to reporting guidelines

Adherence to relevant reporting guidelines is presented in Table 3.

3.6. Pragmatic trial methods

3.6.1. Average PRECIS-2 ratings

The PRECIS-2 instrument has 9 domains that are each rated from 1, indicating a very explanatory design, to 5, indicating a pragmatic approach to a design feature.[68]

Interrater reliability was moderate for overall PRECIS-2 ratings (intraclass correlation coefficient [ICC] 0.73; 95% confidence interval 0.68-0.78, P < 0.001), having calculated the ICC using a two-way mixed-effects model for absolute agreement.[51,61,90] Disagreements had to be resolved through discussion in 22.4% of all instances as initial disagreements exceeded one point on the PRECIS-2 scale. As per protocol, disagreements of a single point were averaged automatically. Assessing interrater reliability for individual PRECIS-2 domains, we found moderate (ICC of 0.5-0.75) or good (0.75-0.9) agreement for all domains but domain 1 (participant eligibility), for which initial agreement was poor (ICC < 0.5) (see Supplement table, available at http://links.lww.com/PAIN/B374).[90]

In our sample of studies, the average rating across all domains was 3.8 (SD 0.62). Only 19 of 513 overall items were deemed impossible to rate because of required information not being reported (3.7%).

There was no significant difference in overall PRECIS-2 ratings between declared pragmatic trials and those which authors described as “comparative effectiveness” trials and not “pragmatic” (t(55) = 1.72, P = 0.092). The only individual domain where there was a significant difference was “organization” (t(49) = 2.13, P = 0.039), with trials not declared pragmatic showing less pragmatic features. T-tests for all other domains had P values of > 0.175.

Ratings for individual domains and factors influencing these ratings are presented in more detail below, rating statistics are illustrated in Figure 2 and presented in a supplementary table (available at http://links.lww.com/PAIN/B374).

Figure 2.

Average PRECIS-2 scores per domain for all included trials. SDs and n not indicated (see supplementary table, available at http://links.lww.com/PAIN/B374). Less pragmatic design choices result in “dents” in the wheel diagram while higher average ratings per domain cause the line to be closer towards the rim of the wheel.

3.6.2. Eligibility

Assessment of this domain was possible in all instances (M 3.97; SD 1.1; n = 57), although in only 68% of studies the reporting of eligibility criteria was “explicitly framed to show the degree to which they included typical participants and/or, where applicable, typical providers (eg, nurses), institutions (eg, hospitals), communities (or localities eg, towns), and settings of care (eg, different healthcare financing systems).”[130]

The main reasons for low ratings in this domain were the exclusion of patients with comorbidities common for the specific trial population, which was the case in more than a quarter of all trials (15 studies; 26%). Similarly, common medications were a reason for noneligibility in 7 studies (12%). Once enrolled, patients were advised not to seek additional care outside the trial in 9 studies (16%)[19,23,35,49,88,93,102,111,127]; the report was unclear on that point in 19 instances.

Eligibility criteria for providers, specifically a minimum number of years in practice, could be confirmed in 6 trials (11%). This information was not reported in 41 studies (72%) and not relevant for assessment in further 3 trials (eg, where the treatment was automated). A minimum amount of experience with the trial intervention (other than trial-specific training) was required in at least 6 trials (11%).

Entry criteria for trial centres existed in at least 8 cases (14%), not in 4 cases (7%), and were not reported on in 42 trials (74%). Three further studies took place entirely in the patients' home or another community setting and were thus not relevant for this assessment.

3.6.3. Recruitment

For patient recruitment, convenience sampling was deemed most in line with the principles of pragmatic trials.[68] Instead, however, almost half the trials (25 cases, 44%) resorted to targeted recruitment methods, such as patient identification through records or targeted adverts. A mixed approach was used by 14%, and recruitment methods were not reported on in another 16% of studies. This reliance on more laborious recruitment strategies is reflected by the fact that PRECIS-2 ratings were lowest on average for this domain (M 3.03; SD 1.6; n = 47); it also points to patient recruitment as a major challenge in pragmatic trials, especially when sought to be performed “pragmatically,” ie, in tune with every day practice. In further support of this argument, of 56 studies that reported a target recruitment number, 15 (27%) did not reach their aim.

3.6.4. Setting

The PRECIS-2 domain “setting” asks how different the setting of the trial and the usual care settings are. Reporting in line with the CONSORT extension item 21 would help readers to assess generalisability of results: “Key aspects of the setting which determined the trial results”[130] were only reported in 21 studies (37%), and a discussion of “possible differences in other settings where clinical traditions, health service organisation, staffing, or resources may vary from those of the trial”[130] happened in 19 reports (34%). Although explicit considerations of generalisability were often lacking, extrapolation allowed reviewers to rate this PRECIS-2 domain in all but one case, averaging at 3.79 points (SD 1.4; n = 56).

3.6.5. Organization

The PRECIS-2 domain “organization” compares the provider expertise, resources, and the organisation of care delivery in the intervention arm of the trial to those available in usual care.[68] Information on these trial features may help readers to assess the generalisability of trial results to another setting. Nonetheless, 37 trial reports did not indicate the level of experience of those delivering the intervention (69%; n = 54; not relevant in 3 automated intervention trials). Twenty-nine reports (51%; n = 57) did not state whether or not resources were altered compared with usual care settings to implement the intervention. Where this item could be assessed (n = 28), half the studies (50%) did alter resources for the purpose of the trial and half did not. Trial-specific training in an intervention constitutes such an alteration of resources and was part of at least 23 trials (40%). PRECIS-2 ratings for this domain were lower than those for most other domains, specifically a mean of 3.5 (SD 1.3; n = 51). Poor reporting meant that this domain could not be rated in 6 cases.

3.6.6. Flexibility (delivery and adherence)

Aspects of the intervention delivery were standardised in 35 trials (61%), with 11 reports not indicating (19%). Of those trials in which the delivery was reported to be standardised, about one-third reported monitoring of the fidelity with which treatments were provided, eg, by record checking or video taping of treatment sessions (n = 11, 31%). Where applicable, these features contributed to low PRECIS-2 ratings for this domain, and the sample's average was 3.51 (SD 1.2; n = 56) for flexibility in treatment delivery.

For the patients' adherence to treatments and interventions, however, an average of 4.34 points was obtained (SD 1.0; n = 56), meaning that patients were flexible in how they had to follow intervention plans; adherence was encouraged little more than what would be expected in usual practice, and nonadherence rarely meant exclusion from the trial analysis. In fact, postrandomization exclusion criteria, such as minimum compliance, absence of adverse events, or other outlier criteria, were only present in a small percentage of trials (n = 5; 9%).

3.6.7. Follow-up

The primary time point at which outcomes were measured was a median of 26 weeks postrandomization (Q1/3: 8; 52 weeks; n = 36) ranging from a single day[104] to 5 years.[6] However, there were many studies that assessed outcomes over a period,[5,9,30,34,97,102,106] at flexible time points,[50,81,108] studies that defined several time points as primary,[75,111] and several that did not specify a primary point of follow-up.[3,8,26,59,66,83,109,122] A better indicator for how long self-declared pragmatic trials are, may thus be the longest time point of follow-up, for which the median was about one year after randomization (median 50 weeks, Q1/3; 23.25; 52 weeks; n = 56; not reported in one case[50]). The shortest trial assessed peak chest pain during a stenting procedure for acute myocardial infarction, ie, lasted for no longer than a few hours after the intervention.[108] On the other extreme, Beard et al.[6] are comparing the clinical and cost-effectiveness of total vs partial knee replacements for up to 10 years after the event (10-year results not yet published).

Over such potentially long follow-up periods, attrition of study participants can be expected. We found that intervention groups lost an average of 14.9% (SD 12.9, n = 57) of participants until the point of primary follow-up and control groups lost 14.8% (SD 12.78), ranging from no attrition at all to a trial of invasive uterine fibroid surgery where 63% of participants in the intervention group did not complete the trial as per protocol.[67]

Across our sample of included studies, there was a significant difference in attrition between groups (t(56) = 7.16, P < 0.001) and a third of studies reported differential attrition (19; 33%, n = 57). Where there was differential attrition, it was almost as often into the direction of the control group (9 cases[5,16,81,84,97,101,117,127]) as it was into the direction of the intervention group (11 cases[7,9,20,67,81,82,85,87,94,111,118,123]), with intervention groups losing an average of 14.9% (SD 12.9) and control groups losing an average of 14.8% (SD 12.8) of participants until the point of primary follow-up (t(112) = 0.032, P = 0.97), possibly accounting for the fact that both groups were of active interventions in most cases.

Where patients were lost, reasons for drop-out were reported in 35 articles (65%, n = 54).

The PRECIS-2 ratings for the domain “follow-up” is concerned less with the length of the follow-up period or differential attrition, but rather the frequency and duration of follow-up appointments as well as the intensity of clinical assessments compared with usual care.[68] Based on this, the average rating was 3.24 points (SD 1.3; n = 57), meaning that follow-up was often more elaborate than what would be expected from normal practice.

3.6.8. Outcomes

The choice of outcomes in pragmatic trials should reflect what “matters” to the patient, choosing direct symptom reports or function-related measures over laboratory tests, surrogate markers, expert assessments, or other external judgements.[68] In our sample of trials, subjective pain ratings, certain condition-related questionnaires, or pain-related functional assessments were the obvious choice.[31] Indeed, on average, trials had the highest rating for this domain (M 4.46; SD 1.0; n = 57). Forty-nine trials obtained a primary outcome through such patient reports (86%; n = 57), 5 trials used a laboratory or other remote physiological assessment such as radiographs (9%),[35,38,87,91,111] 3 used a physical or personal assessment (5%),[16,91,101] and one trial each (2%) used data obtained from health records[106] or an objective incident in the medical management of patients, namely the reoccurrence of a medical intervention.[67] As secondary outcomes, objective measures were far more common, being reported in 30 trials (54%) and including measures of healthcare utilization, physical tests, laboratory markers, and absence from work. Only half the reports, however, complied with the CONSORT item to justify the chosen outcome and length of follow-up (29 cases, 51%) (Table 3, item 6). Whether or not significant harms or unintended effects occurred was reported in 47 studies (82%) (Table 3, item 19). Harms did occur in 22 of those studies, and in 9 of those trials, there was a significant difference between groups.[23,63,71,81,84,94,95,97,127]

Not affecting PRECIS-2 ratings, but arguably relevant for clinical decision making, are outcome measures and analyses that directly juxtapose treatment risks and benefits.[36,37] None of the included studies used such composite metrics. Risk-benefit considerations were, however, implicit in 3 trials[6,64,111] assessing high-risk interventions or comparing a high-risk vs a low-risk intervention (eg, opioid and nonopioid medications). These trials provided extensive data on adverse events. In most other trials, the studied interventions held very little apparent risk to the patients' safety, arguably making risk-benefit analyses less pertinent.

Cost-effectiveness analyses were performed as part of 12 studies (21%) and considered in another 8 (14%; either declared in protocol but not reported or considered as part of trial rationale). Downstream healthcare utilization was reported in 2 trials (4%), allowing for some economic considerations. Again, in some instances one of the tested interventions was so apparently less costly that cost-benefit analyses did not seem warranted if comparative effectiveness or superiority had been shown.[74]

3.6.9. Primary analysis

The highest PRECIS-2 rating for this domain is obtained by trials that perform a true intention-to-treat (ITT) analysis for their primary outcome assessment,[68] meaning that all patients randomized are analysed as if treated, irrespective of actual treatment compliance or a failure to attend follow-up assessments (essentially resulting in missing data). “Pragmatism” in the primary analysis was high, averaging at 4.3 points (SD 1.3, n = 57). The distinction between a true ITT analysis and a modified ITT[1,53,77] is made clear by the following data: Although 48 studies reported to have performed an ITT as primary analysis (84%; not discernible in one instance[18]), 10 of those (21%) excluded participants who did not provide follow-up data or had missing data.

3.6.10. Multicentre trials

At least 45 trials (79%) of our sample were multicentre studies (with 7 studies not reporting the number of participating treatment centres). Despite the possibility of differences between trial centres, eg, in case load, resources, and attending patient population, only 2 studies[49,87] reported having assessed such differences between centres. Those authors also discussed how such differences may have affected the trial results, and thus contextualising their findings and enabling the reader to better judge generalisability. Randomization was stratified by site in 15 trials, another method to account for potential differences between study centres.

At least 21% of trials were single-centre trials, which have been highlighted as potentially unpragmatic in recent debates because of supposed low generalizability.[28]

3.7. Second-level analyses

3.7.1. Baseline heterogeneity

When testing for differences in mean age between intervention and control groups as an indicator of baseline heterogeneity by means of a paired samples t test, no significant difference was detected (t(52) = 1.79, P = 0.079), suggesting that randomization was not systematically biased in this sample of studies.

3.7.2. Preliminary analyses

3.7.2.1. Testing for potential confounders

There were significant correlations between the overall trial size (total sample size at randomization) and a number of other variables of interest: These included overall PRECIS-2 scores, driven by highly significant correlations with the domains “setting,” “organization,” and “analysis.” Larger trials were also less likely to show differential attrition, irrespective of in which group most drop-outs occurred (treatment vs control group) (Table 4). Sample size was thus included as a covariate of no interest in subsequent analyses of PRECIS-2 scores and attrition.

Table 4

Correlation analysis between domain-specific PRECIS-2 ratings and total sample size at randomization.

	PRECIS-2 domain	Eligibility	Recruitment	Setting	Organization	Flexibility (delivery)	Flexibility (adherence)	Follow-up	Outcome	Analysis	Total PRECIS-2 score	Different. attrition
Total sample size at BL	Correl. Coeff.	0.250	0.029	0.451†	0.281*	0.031	0.193	0.190	0.161	0.273*	0.408†	−0.360†
	P	0.061	0.846	0.000	0.046	0.822	0.155	0.157	0.232	0.040	0.002	0.006
	n	57	47	56	51	56	56	57	57	57	57	57

Also showing the correlation analysis between sample size and differential attrition (last column) as measured by the difference in drop-outs, irrespective of the “direction” of attrition, ie, in which group more patients were lost to follow-up.

Spearman rank order correlation used as data not conforming with normality assumption P < 0.05.

Spearman rank order correlation used as data not conforming with normality assumption P < 0.01; 2-tailed.

BL, baseline.

3.7.3. Correlation analyses

A range of planned correlation analyses were performed to identify potential associations among different trial features and with ratings of pragmatism, randomization method, and blinding status of participants (Table 5).

Table 5

Correlation analyses among trial methods with ratings of trial pragmatism (PRECIS-2 scores), randomization methods, and analysis method.

	DV	Results	Sensitivity analysis
PRECIS-2 average	Sample size
	Number of trial centres*	0.190, P = 0.191, (df = 47)
	Funding source*	−0.048, P = 727, (df = 54)
	Index therapy*	−0.005, P = 0.97, (df = 54)	0.089, P = 0.512, n = 57
	Index pain disorder*	−0.285†, P = 0.033, (df = 54)	−0.213, P = 0.112, n = 57
	Analysis method*	0.198, P = 0.16, (df = 50)	0.284†, P = 0.039, n = 53
Randomization method	PRECIS-2 average*	−0.197, P = 0.145, (df = 54)	−0.137, P = 0.31, n = 57
	Baseline heterogeneity	This analysis has not been conducted as there were no trials with significant between-group age differences at baseline.
	Sample size	0.21, P = 0.122, n = 57
	Analysis method	0.0, P = 0.99, n = 53
	Funding source	−0.072, P = 0.594, n = 57
Blinding of participants	PRECIS-2 average*	0.214, P = 0.124 (df = 51)	0.135, P = 0.331, n = 54
	Sample size	−0.081, P = 0.562, (n = 54)
	Analysis method	0.2, P = 0.16, (n = 51)
	Funding source	0.111, P = 0.424, (n = 54)

Statistical tests were part of correlation analyses where covariates were controlled for, and Spearman rho where this was not indicated.

Sample size was used as covariate of no interest. Sensitivity analyses assess the same correlation without controlling for preidentified confounding variables.

Significant at P < 0.05 level (2-tailed).

DV, dependent variable.

Specifically, we asked whether overall PRECIS-2 scores were associated with the number of trial centres, the source of trial funding (public, industry, or mixed), the primary therapy investigated, the pain condition of participants (index pain disorder), and the used analysis method (distinguishing a true ITT, modified ITT, and no ITT). Controlling for sample size, only the variable “index pain disorder” was associated with PRECIS-2 ratings (r(54) = −0.285, P = 0.033). Post hoc analyses to see whether specific diagnoses drove this correlation were not possible due to small case numbers in some of the categories.

Whether or not study participants were reported to be blinded to group allocation did not correlate with average PRECIS-2 ratings, the funding source, the size of the trial, or the used analysis method.

3.7.4. Attrition

The size of the trial (total n) did not correlate with the percentual attrition, neither in the intervention group nor the control group (r = −0.154 and −0.103, n = 57, P = 0.254 and 0.445, respectively). When ignoring the direction of the attrition, however (ie, whether more drop-outs occurred in the intervention or the control group), the testing showed that larger trials had less percentual attrition than smaller trials (r = −0.360, n = 57, P = 0.006) (Table 4). This latter analysis seems more suitable as the distinction between intervention and control group is somewhat arbitrary in comparative effectiveness trials, where the comparator was often another active intervention.

3.7.5. Subgroup analyses

Given the apparent division of our sample into drug and nondrug trials as well as populations of patients with acute and chronic pain, we examined whether total PRECIS-2 ratings differed between these groups of trials. Only trials providing a clear indication of the patient sample's duration of pain and fitting into the categories of acute (<4 weeks) or chronic (>3 months) were included into this analysis (n = 38).

One-way analysis of variance revealed no difference in average PRECIS-2 scores between trials of pharmacological and nonpharmacological therapies (F(1, 55) = 0.27, P = 0.6), with ratings averaging 3.7 (±0.7) and 3.8 (±0.6) of a maximum score of 5, respectively. Domain-specific ratings only differed for the flexibility with which treatments were delivered, with drug studies allowing significantly less flexibility (2.8 ± 1.2 vs 3.7 ± 1.6; F(1,54) = 5.14, P = 0.027).

A significant difference in overall PRECIS-2 scores existed between acute and chronic pain trials (F(1, 36) = 5.14, P = 0.03), with higher scores in acute trials (4.3 ± 0.7 vs 3.7 ± 0.6). Exclusion of an outlier, a chronic pain trial with the lowest overall rating,[91] did not alter the statistical significance of this result (F(1,35) = 5.32, P = 0.027). On evaluation of individual PRECIS-2 domains, only the domain “recruitment” differed significantly between groups (F(1,28) = 5.88, P = 0.02), with chronic pain trials investing more into patient recruitment than acute trials.

4. Discussion

This systematic review of methods describes the current status in the field of declared pragmatic trials in clinical pain therapy research. Such trials typically include several hundred participants, multiple trial centres, and have average follow-up periods of one year. Pragmatic trials in pain research compare 2 or more treatments with one another or with “care as usual.” Treatments are often applied flexibly, and adherence is rarely monitored. Pragmatic trials of pain treatments use outcome measures that are deemed relevant for clinical decision making and, in the main, analyse all patients irrespective of treatment compliance or provision of follow-up data. Pragmatic trials in pain research mainly recruit patients living with persistent pain, often musculoskeletal such as back pain or peripheral joint pain, but a small number of pragmatic trials are also conducted in in-patient settings and perioperatively.

Included trials predominantly investigated complex nonpharmacological interventions rather than drugs. Many manual, rehabilitation, or cognitive–behavioural interventions are already established in routine practice so that equipoise is between 2 or more alternative (or complementary) treatment options rather than between a new treatment and a placebo. Another driver for pragmatic comparative effectiveness research for nondrug therapies is that these treatments are not subject to drug regulators who require early efficacy and safety signals for market approval. Instead, the nondrug therapy research is produced for clinicians and clinical treatment guidelines, where evidence from comparative effectiveness trials may be acceptable. It is unclear, however, why therapies for centralised pain disorders such as fibromyalgia as well as common complaints such as headaches and neuropathic pain were studied so rarely in pragmatic trials for the past 2 years.

This review provides readers with an overview of what is currently called a “pragmatic trial of pain treatments,” enabling them to compare any given trial to this comprehensive description. We did not include a comparison group, eg, from a randomly selected sample of pain trials or based on existing reviews. Not only were there feasibility constraints but we also did not want to bias our findings by the selection of comparison data from noncomparable populations. For example, if we had chosen a systematic review of treatments for neuropathic pain as comparator,[39] we would unsurprisingly find large differences to our sample because the neuropathic pain review only studied pharmacological interventions. We are not aware of any reviews of pain treatments that are not restricted to specific populations or interventions.

Apart from describing the “typical” pragmatic trial in pain research, this systematic review identified several areas for improvement centred around trial reporting, design, and interpretation.

If the pragmatic aim of “informing real-world decision-making”[115,129] is to be reached, readers require more information about the environment in which the trial was conducted, including a better description of trial centres, their resources, and the typical patient population and diagnoses. This seems particularly important in single-centre trials, making up 21% of our sample, around which there is debate as to whether they can be considered pragmatic at all because of the arguably limited generalisability of results.[28,129] Multicentre trials, on the other hand, provide the opportunity to assess for differences between study centres and how these factors may have influenced trial results. This was performed in 2 reports only. Additional information about the characteristics of trial centres could facilitate readers' assessment of the applicability of trial results to their particular setting, even when considering single-centre trials. Relatedly, but unlikely specific to pragmatic trials, there is a need to better describe the population of patients: Too many trials do not indicate the average duration of pain in their sample and many omitted descriptions of the nature or location of pain reported by patients. Similarly, provider characteristics, such as professional qualifications and practical experience with the intervention under investigation, need to be reported. More broadly, trialists cannot assume that readers are aware of the particularities of the healthcare system or socioeconomic and cultural context in which the trial has been conducted. What constitutes “care as usual” or how a comparator therapy is implemented may differ widely and is rarely reported in detail. The same is true for concomitant pain treatments, with a fifth of the assessed trials not even indicating whether these were permitted. Detailed information on comparator groups and out-of-study interventions is, however, fundamental to interpreting and understanding the results of any clinical trial and likely more variable in pragmatic trials. Authors are in a unique position to highlight likely similarities with and differences to other potential settings. Another reporting issue is the justification of used trial methods.[80] For example, why and how did those designing the trial choose certain outcomes and the duration of follow-up periods? Appropriate outcome measures in pain research have been discussed extensively and in an influential publication in 2005.[31] Possibly, these outcomes have become common practice, making an extensive justification of their choice seem arbitrary. The appropriate length of follow-up periods, on the other hand, is not as well researched.[32] Authors should thus indicate whether the follow-up periods were chosen for clinical reasons, due to patient preferences, or for reasons of trial feasibility, such as funding and drop-out risk.

Our sample of 57 trials obtained an average rating of 3.8 (±0.6) on the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) 2 instrument.[68] Although comparisons with other research fields are difficult, it is noteworthy that this overall score is very similar to ratings of 23 self-declared pragmatic cardiovascular trials that averaged at 3.83 (±0.78).[105]

Domain-specific PRECIS-2 ratings showed that there are a few areas in pain research where “pragmatic” trial design and conduct are particularly challenging: the relatively large number of patients required often conflicts with the aim to recruit patients in ways comparable to normal practice. Instead of convenience sampling, trialists frequently implement targeted recruitment strategies, such as identification through records and the selective contacting of potentially eligible patients. How much this interferes with the generalisability of trial results remains to be determined. Interestingly, recruitment was more elaborate when patients with chronic pain were sampled, demonstrating that challenges and opportunities for pragmatic trials depend on each trial's circumstances and objectives.[125] In the field of pain research, differences seem to exist between trials with patients with acute and chronic pain as well as between drug and nondrug trials.

Relatedly, challenges to and opportunities for the implementation of a pragmatic attitude to trial design can be domain specific. In general, more pragmatism seems easier to implement in the area of follow-up assessments by means of reducing the frequency and extent of outcome assessments. Nonetheless, follow-up assessments in this sample often exceeded what would be expected in normal practice, mirroring findings from a small retrospective analysis of weight loss trials.[48] It could be tempting for trialists to implement more complex and more numerous tests, simply because the opportunity arises. Although understandable from a research perspective, extensive outcome testing adds to patient burden and research costs.[15,31,40,116] The extent to which this interferes with patient recruitment and retention is an important question for pragmatic trials and worthy of investigation.[27]

Standardisation of treatment delivery was common (61% of the overall sample), and protocol fidelity monitoring occurred in a third of those trials. Ratings for this domain were significantly lower in trials of pharmacological than that for nonpharmacological treatments, reflecting findings from Koppenaal et al.[62] who reviewed a set of lifestyle intervention trials and a group of beta-blocker RCTs, few of which, however, declared pragmatic trials. From general practice to complementary and manual therapies, treatments are rarely delivered in an inflexible way. Instead, they are adapted to the patient's needs and preferences, subject to provider expertise and inclinations, as well as influenced by available resources.[22,54,113] To account for these factors and reconcile them with the need to describe what happened during a trial, instead of artificially restricting the variability in treatment delivery, qualitative research methods may be more appropriate to assess and communicate generalisability. Conversely, such added variability would increase the need for larger samples. Interestingly, treatment adherence was rarely controlled (also compare[70]).

The real or perceived need to control what happens during a trial may also have contributed to the organisation of participating trial centres being more complex and likely more sophisticated than what would be seen in normal practice. Comparably low ratings for this domain were given in a review of RCTs in patients with diabetes.[70] Again, this points to a possible risk to trial design: Should researchers resort to treatment centres and providers who they know can comply with the various requirements of a trial or do they trust “normal” practitioners to do the same? Although the first option is assumed to further the successful recruitment and completion of a trial, it also compromises generalisability, and vice versa for option 2. As an encouraging example that large trials can be conducted in non-research facilities, Eklund et al.[34] conducted a trial with 40 chiropracters treating more than 300 participants in their private clinics across Sweden.

Another area where the ability of a trial to inform real-world decision making is potentially hampered by the trial's design is the implementation of placebo control groups and, relatedly, blinding of participants and providers. As Dal-Ré et al.[28] point out, these aspects are not part of normal clincial practice, and the authors argue that any trial using them is inherently explanatory. In our review, 5 trials (9%) used a placebo control group. Participant blinding was performed in 13 trials, representing a quarter of all trials that reported on participant blinding, and providers were blinded to group allocation in 4 trials. In the debate on whether these design features preclude labelling a trial “pragmatic,” Zwarenstein et al.[129] respond that, eg, in scenarios where patient or provider subjectivity needs to be excluded as a source of apparent effectiveness, such studies can still inform real-world decision making, the main intention behind pragmatism in trial design. The pain field with its predominatly subjective outcome measures offers illustrative examples of this reasoning, such as Bayer et al.,[5] a self-declared pragmatic trial comparing an off-label beta-blocker vs placebo in the prevention of vestibular migraine, or the CSAW trial of Beard et al.,[7] which was the first placebo-controlled trial for subacromial decompression surgery, demonstrating no benefit of real surgery over the surgical placebo (exploratory arthroscopy). Following Dal-Ré's reasoning, however, by using a third, no-treatment arm and clearly demonstrating a marked placebo effect of both interventions, the CSAW trial had a strong explanatory component that was not reflected in its PRECIS-2 score of 4.33. On the other hand, the results of this trial are clearly relevant to clinical decision making given that decompression surgery is (still) common practice. It seems therefore that a pragmatic intention is compatible with elements of mechanistic, explanatory studies but that these instances should be clearly highlighted alongside PRECIS-2 ratings to understand the reasoning behind the trial design (also see Ref. 82).

Many of the above consideration point to difficulties when applying the PRECIS-2 instrument to trials' design. When understood as an “incentive” during the planning of a trial, higher ratings in each domain may conflict with internal validity requirements of a trial and the developers rightly point out that high ratings are not an end in themselves.[68,131] Despite being a scale, PRECIS-2 may have contributed to a false dichotomy. Often, trial methods are discussed as either pragmatic or explanatory.[28] Rather than the design, however, it is the trial's objectives that make it pragmatic or explanatory and trial methods simply follow the need to answer pragmatic research questions in a methodologically sound manner.[42,103] Furthermore, when used retrospectively, the comparison of trials from different fields may be challenging, with, eg, provider training and fidelity monitoring being much more pertinent issues in complex intervention trials than in pharmacological studies. For the present purpose, however, discrepancies in such ratings allowed for a nuanced discussion of the potential reasons, again highlighting that PRECIS-2 ratings require context.

For future methodological work on pragmatic trials for pain therapies, it is worthwhile to contextualize the interrater reliability of our PRECIS-2 ratings. In general, our overall moderate agreement compares favourably to the 2017 PRECIS-2 validation study of Loudon et al.[69] that found good interrater reliability for 3 domains and moderate reliability for the remaining 6,[90] but we achieved much smaller confidence intervals in our study (Supplementary table 3, available at http://links.lww.com/PAIN/B374). Interestingly, rating a sample of 15 trial protocols from a variety of fields, the test raters in Loudon's study had most difficulty agreeing on ratings for the domains recruitment and intervention adherence, whereas in our study the rating of domain 1 (participant eligibility) was most ambiguous, possibly underlining the need for authors of pragmatic trials to more clearly report if and how their trial population generalizes to the target population of the intervention in routine practice. In our study, domains 2 (recruitment) and 4 (organization) had the most missing data due to insufficient information in protocols and trial reports (48 and 47 complete ratings, respectively, less after reconciliation); for domain 4 (organization), Loudon et al. also had the highest percentage of missing data, attesting to suboptimal reporting of this information in many trials. Our approach of detailed preparation and training of those researchers who performed the PRECIS-2 ratings, plus the averaging of discrepancies of a single point, led to moderate agreements and a very feasible reconciliation process where only a fifth of items required a mostly brief discussion and usually without involvement of a third party. However, the fact that the initial interrater reliability was nonetheless only moderate, plus the fact that about 4% of domains could not be rated even after discussion, testifies to the inherent challenges of retrospective PRECIS-2 ratings and the need to improve trial reporting to facilitate such assessments in the future.[28]

Apart from reporting and design considerations for pragmatic trials, this review raises concerns regarding the analysis and interpretation of trial results. As most pragmatic trials are comparative effectiveness studies and mostly designed to show a difference between group means (superiority trials), authors need to be explicit about the clinical significance of differences, if detected, and cannot claim “equivalence” if the trial failed to show a significant difference. The latter occurred in over a third of 24 nonsignificant superiority trials in this sample, much higher than the 10% found in a review of 76 reports of pain therapy trials with nonsignificant primary analyses.[47] If designed as noninferiority or equivalence trials, trial designers need to establish assay sensitivity, ideally by including a third, no-treatment or placebo control group.[41] Although only 4 noninferiority trials were included in this sample, none of them complied with this recommendation, again making it difficult to interpret the results. Finally, what authors understand as “intention-to-treat analysis” differs, with 20% of self-declared ITT analyses excluding participants who did not provide follow-up data or where data were missing. The use of such modified ITT analysis and incorrect labelling has direct implications for the interpretation and meta-analysis of results[1,53,77] and mirrors the findings of a 2014 review of phase II and III trials of pain treatments.[46]

5. Limitations

We excluded 14 studies because the authors did not use the terms “comparative effectiveness,” “pragmatic,” or “practical” in reference to their own study. We pointed out in our protocol that “This review will only capture trials which have been declared as “pragmatic,” “practical,” or “comparative effectiveness” by their authors, ie, publications which contain this or related terms in the title or abstract.” We acknowledged that “relying on author self-report might result in the inclusion of studies which score low on current tools for the evaluation of pragmatic aspects of trial design (specifically PRECIS-2) as well as the omission of trials not explicitly declared “pragmatic” but in fact conforming with many criteria of pragmatic trials.” A future sensitivity analysis may wish to examine how including these trials would have affected the results of this systematic review.

Furthermore, the decision to include trials that were declared comparative effectiveness trials but not necessarily declared pragmatic trials may have resulted in the inclusion of trials that were not explicitly designed as pragmatic trials. An area where this may have had an effect is compliance with pragmatic trial reporting guidelines. In addition, authors may label their study pragmatic without considering that this should mean a trial with the potential to directly inform clinical decision making.[42,103,129] Indeed, the lowest rated study in our sample by Qi et al. may be such a case of conceptual misapplication.[91] Finally, there likely are declared pragmatic trials that are not randomized and these were outside the scope of this review.

6. Conclusion

In summary, this systematic review provided a comprehensive snapshot of the current practice in the pragmatic design of comparative effectiveness and other pragmatic trials of pain treatments. Such trials typically include several hundred participants, numerous sites, are publicly funded, and assess complex interventions for the treatment and management of pain, predominantly chronic pain. These trials have long follow-up periods, use clinically relevant outcome measures, and resemble usual care in the extent to which patients are required to adhere to treatments. The resources used for patient recruitment and the intensity of follow-up often pre-empted higher ratings of “pragmatism.” The included trials comply well with basic reporting guidelines but the assessment of generalisability is frequently hampered by poor reporting of design features relevant to pragmatic trials. Overall, the challenges and opportunities for pragmatic trial design are likely largely dependent on an individual trial's objectives and circumstances. There are no recommendations for trial designers regarding how to navigate these challenges, on balancing internal and external validity, and on harnessing the potential for pragmatic trials to provide highly clinically relevant insights in a trustworthy manner.

This review ascertained the prevalence of self-declared pragmatic, practical, or comparative effectiveness trials for the treatment and management of patients with pain and will inform future development of and guidance on trial methods designed to enhance real-world application of trial findings.

Conflict of interest statement

D. Hohenschurz-Schmidt reports personal fees from the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT), during and for the conduct of the study. B.A. Kleykamp has received in the past 36 months support from the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) public–private partnership with the U.S. Food and Drug Administration and compensation for medical writing from Palladian Associates, Hayes Inc/TractManager, and PBS Next Avenue. In addition, between 2014 and 2018 during her previous employment at the consulting firm, PinneyAssociates, she provided consulting advice to pharmaceutical companies, the e-cigarette company NJOY, and the tobacco company RAI Services Company on noncombustible tobacco products including e-cigarettes; M. Ferguson reports other funds from ACTTION, during the conduct of the study; J. Vollert reports personal fees from Casquar, personal fees from Vertex Pharmaceuticals, outside the submitted work; S.T. Evans reports personal fees from Takeda/Millennium, personal fees from Pfizer, personal fees from Roche, personal fees from Novartis, personal fees from ACTTION, personal fees from Genentech, personal fees from Amgen, personal fees from American Statistical Association, personal fees from FDA, personal fees from Osaka University, personal fees from Nationa Cerebral and Cardiovascular Center of Japan, personal fees from NIH, personal fees from Society for Clinical Trials, personal fees from Statistical Communications in Infectious Diseases (DeGruyter), personal fees from AstraZeneca, personal fees from Teva, personal fees from Austrian Breast & Colorectal Cancer Study Group (ABCSG)/Breast International Group (BIG) and the Alliance Foundation Trials (AFT), personal fees from Taylor and Francis, personal fees from Vir, personal fees from Shire, personal fees from Alexion, personal fees from Gilead, personal fees from Clinical Trials Transformation Initiative, personal fees from Tracon, personal fees from Deming Conference, personal fees from Antimicrobial Resistance and Stewardship Conference, personal fees from Advantagene, personal fees from Cardinal Health, personal fees from Microbiotix, personal fees from Stryker, personal fees from Atricure, personal fees from BENEFIT, personal fees from Roivant, personal fees from Neovasc, personal fees from Nobel Pharma, personal fees from Horizon, personal fees from Roche, personal fees from Rakuten, personal fees from Duke University, personal fees from U. of PENN, personal fees from Takeda, personal fees from Nuvelution, personal fees from AbbVie, personal fees from Clover, personal fees from FHI Clinical, personal fees from Lung Biotech, personal fees from SAB Biopharm, personal fees from CIOMS, personal fees from SVB LEERINK, grants from NIH, outside the submitted work; R.H. Dworkin reports grants from U.S. Food and Drug Administration, during the conduct of the study; personal fees from serving on advisory boards or consulting on clinical trial methods from Abide, Acadia, Adynxx, Analgesic Solutions, Aptinyx, Aquinox, Asahi Kasei, Astellas, AstraZeneca, Biogen, Biohaven, Boston Scientific, Braeburn, Cardialen, Celgene, Centrexion, Chromocell, Clexio, Collegium, Concert, Confo, Decibel, Dong-A, Editas, Eli Lilly, Ethismos (equity), Eupraxia, Glenmark, Grace, Hope, Immune, Lotus, Mainstay, Merck, Neumentum, Neurana, NeuroBo, Novaremed, Novartis, Olatec, Pfizer, Phosphagenics, Quark, Reckitt Benckiser, Regenacy (also equity), Relmada, Sanifit, Scilex, Semnur, SIMR Bio, SK Life Sciences, Sollis, SPRIM, Teva, Theranexus, Trevena, Vertex, and Vizuri, outside the submitted work; D.C. Turk reports personal fees from Pfizer, personal fees from Lilly, personal fees from GSK, outside the submitted work; and is the Editor-in-Chief of the Clinical Journal of Pain. A.S.C. Rice reports personal fees from IMMPACT, during the conduct of the study; personal fees from Imperial College Consultants, from Spinifex, outside the submitted work; In addition, A.S.C. Rice has a patent WO 2005/079771 pending and a patent EP13702262.0/WO2013 110945 pending. The remaining authors have no conflicts of interest to declare.

Supplemental digital content associated with this article can be found online at http://links.lww.com/PAIN/B374.