Systematic review of the cost-effectiveness of implementing guidelines on low back pain management in primary care: is transferability to other countries possible?

OBJECTIVE: The primary aim is to identify, summarise and quality assess the available literature on the cost-effectiveness of implementing low back pain guidelines in primary care. The secondary aim is to assess the transferability of the results to determine whether the identified studies can be included in a comparison with a Danish implementation study to establish which strategy procures most value for money.
DESIGN: Systematic review. DATA SOURCES: The search was conducted in Embase, PubMed, Cochrane Library, NHS Economic Evaluation Database, Scopus, CINAHL and EconLit. No restrictions were made concerning language, year of publication or publication type. The bibliographies of the included studies were searched for any studies not captured in the literature search. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: To be included, a study must be: (1) based on a randomised controlled trial comparing implementation strategies, (2) the guideline must concern treatment of low back pain in primary care and (3) the economic evaluation should contain primary data on cost and cost-effectiveness.
RESULTS: The title and abstract were assessed for 308 studies; of these, three studies were found eligible for inclusion. The Consensus Health Economic Criteria (CHEC) list showed that the 3 studies were of moderate methodological quality while application of Welte's model showed that cost results from two studies could, with adjustments, be transferred to a Danish setting. It was questionable whether the associated effects could be transferred.
CONCLUSIONS: Despite the resemblance of the implementation strategies, the 3 studies report conflicting results on cost-effectiveness. This review showed that transferring the results from the identified studies is not straightforward and underlines the importance of transparent reporting. Future research should focus on transferability of effects, for example, development of a supplement to Welte's model. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

Policymakers could potentially make wrong decisions if they are not aware of the difficulties in transferring costs and effects between countries.

To the best of our knowledge, this is the first review to assess transferability concurrently with methodological quality.

Only studies focusing on both the costs and effects of strategies for implementing low back pain guidelines in general practice were included in this review.

Only limited research concerns the cost-effectiveness of implementing low back pain guidelines in primary care.

Introduction

A comparison of all relevant alternatives should be carried out when assessing the cost-effectiveness of new a intervention.1 This may be challenging if not all alternatives have been evaluated in the decision-makers' jurisdiction and, as it is very time-consuming and expensive to conduct research, the utilisation of foreign data may be of great value. Utilisation is, however, conditional on transferability of the foreign data from the study country to the decision country.2

In connection with the dissemination of a clinical guideline on the management of low back pain (LBP) in primary care, a cluster randomised controlled trial was carried out in Denmark.3 4 The objective of this trial was to assess the effectiveness and cost-effectiveness of an extended implementation strategy to enhance the general practitioners' adherence to the guidelines compared with usual dissemination.5 6 It would significantly add to this study if the results of the economic evaluation could be contrasted with results from other studies assessing different strategies towards implementing LBP guidelines in primary care. This could provide insight into which strategy procures most value for money.

The primary aim of this systematic review is to identify, summarise and assess the quality of economic evaluations concerning strategies used in the implementation of guidelines for management of patients with LBP in primary care. The secondary aim is to assess the transferability of the results by application of the model of Welte,7 which is a decision chart for assessing the transferability of results from economic evaluations between countries. By use of the model of Welte,7 the transferability appraisal aims to inform whether the existing studies can be included in a comparison with the Danish guideline implementation study,6 followed by a discussion of the applicability of Welte's model.

Methods

The reporting of this systematic review follows, where applicable, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).8 9 See online supplementary file 1 for the PRISMA 2009 checklist.

For studies to be included in the systematic review, the following eligibility criteria must be met: (1) the study must be based on a randomised controlled clinical trial comparing implementation strategies, (2) the guideline must concern treatment of patients with LBP in primary care, and (3) the economic evaluation should contain primary data on cost and cost-effectiveness. To identify all relevant studies, a literature search was conducted in the following databases: Embase, PubMed, Cochrane Library, NHS Economic Evaluation Database, Scopus, CINAHL and EconLit. No restrictions were made concerning language, year of publication or publication type. The keywords ‘guideline implementation’, ‘low back pain’, ‘primary health care’ and ‘cost and cost-analysis’ were searched for in combination by use of the Boolean operator ‘AND’. As far as possible, thesaurus terms were applied and adjusted to the individual databases and used in combination with free text search. The last search was run on 13 February 2016. See online supplementary file 2 for a complete search history.

Study identification

After removal of duplicates, two authors (CEJ and AR) screened the title and abstracts of the recovered studies applying a five-point scale: (1) the study does not concern LBP, (2) the study does not concern LBP guideline implementation in primary care, (3) the study is not based on a randomised controlled clinical trial, (4) the study is not an economic evaluation, or (5) the study is eligible for full-text screening. Studies assigned a value from one to four were excluded, while studies assigned the value five were full-text screened and included if eligible. Any uncertainty about whether or not to include a study in the systematic review was resolved by consensus among the authors. Following this, the bibliographies of the included studies were searched for any studies not captured in the systematic literature search.

Data extraction

Information about study country, study design, type of economic evaluation, population characteristics, interventions, perspective, time horizon, effectiveness measures, incremental cost-effectiveness ratio (ICER) and sensitivity analyses was extracted from the included studies. Data extraction was carried out by CEJ and KDP.

Assessment of quality

The Consensus Health Economic Criteria (CHEC) list, developed for use in systematic reviews involving economic evaluations, was applied to assess the methodological quality of the included studies.10 The CHEC list is composed of 19 questions, where each question is assigned either a ‘yes’, indicating that the item was either covered adequately or reported in an appropriate way, or ‘no’, indicating that the item was not met.11 Studies receiving a ‘yes’ score higher than 75% were defined as high-quality evaluations, while studies with a score higher than 50% and up to 75% were defined as moderate quality evaluations, and studies with a score of up to 50% were defined as low-quality evaluations.

As recommended, two reviewers (CEJ and KDP) independently assessed the included studies using the CHEC list.10 11 Any disagreements were resolved by consensus.

Assessment of transferability

To assess the transferability of the included studies from the study country to a Danish setting, the decision chart for assessing and improving the transferability of economic evaluation results between countries developed by Welte et al7 was applied.

Welte's model is a transferability decision chart method that includes both general and specific knockout criteria to assess if a study can be transferred to the decision country.7 Initially, the study must meet three general knockout criteria. The first general knockout criterion requires that the evaluated alternative is compatible with the one that may be used in the decision country. The second general knockout criterion entails that the comparator must likewise be compatible with one that is relevant in the decision country. Finally, the third general knockout criterion requires that the study must be of acceptable methodological quality.7 In this systematic review, results based on the CHEC list were used to assess the third general knockout criteria.

For studies passing the three general knockout criteria, an additional 14 specific knock out criteria are applied afterwards.7 For each of the 14 specific knockout criteria, the estimated relevance of each criterion, the correspondence between the study country and the decision country, and the expected effect on the cost-effectiveness ratio (CER) must be determined. Estimated relevance and correspondence is graded on a five-level scale from very high to very low. According to Welte et al,7 a very high or high correspondence between the study country and the decision country, regardless of estimated relevance of the criterion, is needed for an unbiased estimate of the CER. CEJ carried out the transferability assessment, which was discussed afterwards and adjusted in collaboration with all authors.

Results

Through the systematic literature search, a total of 459 studies were identified. Duplicates (n=151) were removed and the title and abstract were assessed for the remaining 308 studies. Of the assessed studies, 136 (44%) were excluded as they did not deal with LBP. An additional 149 studies (48%) did not concern specific strategies aimed at implementing LBP guidelines in primary care. Furthermore, 13 studies (4%) were excluded, as these were not based on randomised clinical trials, and seven studies (2%), as these were not economic evaluations. In total, 305 studies (99%) were excluded based on title and abstract, leaving no more than three studies (1%) eligible for full-text screening and inclusion. No additional studies were identified through a search of the bibliographies of the included studies. Figure 1 is a depiction of this process.

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of study selection.9 LBP, low back pain; RCT, randomised controlled trial.

Study description

The included studies12–14 compared two or more different strategies for implementing guidelines on LBP management in primary care (table 1).

Table 1

Characteristics of the included studies

Study	Mortimer et al12	Becker et al13	Hoeijenbos et al14
Country	Australia	Germany	The Netherlands
Economic evaluation	Cost-effectiveness	Cost-effectiveness	Cost-utility
Study design	Cluster randomised clinical trial	Cluster randomised clinical trial	Cluster randomised clinical trial*
Study population	NA†	LBP, age >20, understand GermanExclusion criteria: pregnancy, isolated thoracic or cervical pain	LBP, understand DutchExclusion criteria: pregnancy*
Participants	NA†92 general practices	1322 patients126 general practices	483 patients113 physiotherapists
Intervention	Multifaceted and theory-based implementation strategy (IMPLEMENT) vs dissemination.	Physician education vs disseminationPhysician education+motivational counselling vs dissemination.	Active implementation strategy vs dissemination.
Perspective	Health sector	Societal	Societal
Time horizon	12 month follow-up	12 month follow-up	12 month follow-up
Effectiveness measures	Number of X-ray referrals†‡ Adherence to guideline in simulated practices	Hannover Functional Ability Questionnaire‡ Physical activity Days in pain Days of sick leave Quality of life	Quality of life
ICER	IMPLEMENT dominates standard dissemination.§	Both intervention groups dominate standard dissemination.§	Not calculated. No significant differences between active strategy and dissemination were found in either costs or effects.
Sensitivity analysis	PSA	PSA	None
Author conclusion	Substantial additional upfront investment, which may not result in better outcomes sufficient to render active implementation cost-effective.	Both interventions show superiority by trend.	Active strategy appears not to be cost-effective.

*Derived from two articles by Bekkering et al.15 16

†Failure of patient recruitment necessitated a departure from the originally proposed analyses.

‡Primary outcome.

§Valid for the primary outcome.

ICER, incremental cost-effectiveness ratio; LBP, low back pain; NA, not applicable; PSA, probabilistic sensitivity analysis.

In the study by Mortimer et al,12 a cost-effectiveness analysis was carried out alongside a cluster randomised controlled trial. To increase Australian general practitioners’ adherence to the LBP guidelines, a multifaceted and theory-based implementation strategy (IMPLEMENT) was developed. The hypothesis was that the IMPLEMENT intervention, when taking into account reductions in health service use, would be cost-effective compared with standard dissemination. The intervention consisted of two facilitated interactive workshops concentrating on two key messages from the guidelines. The first workshop focused on reducing X-ray referrals, as these are rarely beneficial, and how to handle patients with X-ray seeking behaviour. The second workshop centred on the importance of patients remaining active to reduce pain and disability. For the primary outcome, the number of X-ray referrals, the IMPLEMENT intervention dominated standard dissemination when excluding development costs. However, from a healthcare sector perspective, >95% probability of cost-effectiveness could not be achieved.

The study by Becker et al13 is likewise a cost-effectiveness analysis carried out alongside a cluster randomised controlled trial. In the clinical trial, two different implementation strategies were compared with the usual strategy of dissemination. In the first intervention group ‘Guideline Implementation’ (GI), the implementation strategy consisted of physician education, where the general practitioners were trained in using the LBP guideline. In addition to physician education, the implementation strategy in the second intervention group ‘Guideline Implementation and Motivational Counselling’ (GI+MC) consisted of motivational counselling by practice nurses, who had a 20-hour training course in how to motivate patients with LBP for physical activity. In the German setting, both interventions appeared less costly, when excluding implementation costs, and more effective for the primary outcome functional capability. From a societal perspective, the probability of GI being cost-effective compared with dissemination was 97% at a threshold value of €67 per point on the functional capability scale, while there was a 99% probability at €99 when comparing GI+MC with dissemination.

Hoeijenbos et al14 undertook a cost-utility analysis conducted alongside a clinical trial. The aim of this economic evaluation was to evaluate the cost-effectiveness of an active implementation strategy aimed at physiotherapists working in primary care compared with dissemination. The active implementation strategy featured a training session, where the LBP guideline was explained as well as discussed with the physiotherapists, followed by special skills practice. For the following 4 weeks, the physiotherapists had to practise using the guideline after which a second session was conducted, where experiences and problems, if any, were discussed and the physiotherapists received feedback on their current management of patients with LBP. Since no significant difference in either costs or effects was found between the intervention and the control groups, no assessment of the cost-effectiveness of the active implementation strategy was made.

Application of the CHEC list indicated that all three studies were of moderate methodological quality (table 2).

Table 2

Quality assessment of the included studies using the Consensus Health Economic Criteria (CHEC) list

Item	Mortimer et al12	Becker et al13	Hoeijenbos et al14
1. Is the study population clearly described?	Yes	Yes	No
2. Are competing alternatives clearly described?	Yes	Yes	Yes
3. Is a well-defined research question posed in answerable form?	Yes	Yes	No
4. Is the economic study design appropriate to the stated objective?	Yes	Yes	Yes
5. Is the chosen time horizon appropriate to include relevant costs and consequences?	Yes	Yes	Yes
6. Is the actual perspective chosen appropriate?	No	Yes	Yes
7. Are all important and relevant costs for each alternative identified?	Yes	Yes*	Yes
8. Are all costs measured appropriately in physical units?	Yes	Yes*	Yes
9. Are costs valued appropriately?	Yes	No*	No
10. Are all important and relevant outcomes for each alternative identified?	No	No	No
11. Are all outcomes measured appropriately?	No	No	No
12. Are outcomes valued appropriately?	No	Yes	Yes
13. Is an incremental analysis of costs and outcomes of alternatives performed?	Yes	Yes	No
14. Are all future costs and outcomes discounted appropriately?	No	Yes	Yes
15. Are all important variables, whose values are uncertain, appropriately subjected to sensitivity analysis?	No	No	No
16. Do the conclusions follow from the data reported?	No	No	Yes
17. Does the study discuss the generalizability of the results to other settings and patient/client groups?	Yes	Yes	Yes
18. Does the article indicate that there is no potential conflict of interest of study researcher(s) and funder(s)?	Yes	Yes	No
19. Are ethical and distributional issues discussed appropriately?	Yes	No	No
Total (%)	63	68	53

*Derived from by Becker et al.18

Both Mortimer et al12 and Becker et al13 clearly described the characteristics of the study population (item 1). Mortimer et al12 failed to include patients in the clinical trial, necessitating a departure from the planned analyses. This entailed that the general practitioners became the study participants and as a reference is made to the analysis of the effectiveness,17 this item is categorised as fulfilled. Hoeijenbos et al,14 however, received a ‘no’ on this item, as no eligibility criteria were described for the study population and no reference was made for additional information. Moreover, Hoeijenbos et al14 received a ‘no’ on item 3 concerning a well-defined research question, because, in addition to clearly identifying the alternatives compared, the research question should also state the population.11 With regard to identifying, measuring, and valuing relevant costs in accordance with perspective (item 7–9), only Mortimer et al12 successfully met all three criteria. Item 16, which is centred on whether the conclusion follows from the data reported, was only covered by one study,14 while only Mortimer et al12 addressed any ethical or distributional implications (item 19).

The first step in assessing the transferability of the included studies to a Danish setting was to apply the three general knockout criteria. A review by Koes et al19 has shown that guidelines on managing LBP in primary care are quite similar over a range of countries, including Australia, Germany and the Netherlands. Hence, all studies12–14 pass the first two general knockout criteria. Application of the third general knockout criterion is, however, not as straightforward.

Although the study by Mortimer et al12 was evaluated as being of moderate quality using the CHEC list, it had to deviate from the planned analyses20 and use an intermediate outcome. This is associated with several limitations and results should be interpreted with caution and, furthermore, precludes comparison with the other studies included. On these grounds, the study is considered not to pass the third general knockout criterion.

Becker et al13 likewise have some methodological challenges influencing the transferability assessment. This is primarily related to the lack of results presented, including consumption of resources, valuation and effects. Only incremental cost and effect are presented. Furthermore, the base case ICER does not include implementation costs and is based on complete case analysis, possibly introducing selection bias. Adjusting for baseline difference in costs and clustering of data is done in a sensitivity analysis and presented only in an incremental cost-effectiveness plane showing an estimated ICER close to zero. Since the authors may provide further information or a simple adjustment based on purchasing power parities could be carried out, it is relevant to proceed with the 14 specific knockout criteria (table 3).

Table 3

Transferability of Becker et al13 to a Danish setting

Transferability factor	Estimated relevance	Estimated correspondence between the study country and the decision country	Estimation of CER of decision country based on CER of study country
Perspective	Very high	Very high	Unbiased
Discount rate	NA*	NA	Unbiased
Medical cost approach	Medium	Very high	Unbiased
Productivity cost approach	High	Very high	Unbiased
Absolute and relative prices in healthcare	High	Unclear†	Too high or too low
Practice variation	Medium	High	Unbiased
Technology availability	Low	High	Unbiased
Disease incidence/prevalence	Low	High	Unbiased
Case mix	Low	Very high	Unbiased
Life expectancy	NA	NA	Unbiased
Health status preferences	High	Low	Too high or too low
Acceptance, compliance, incentives to patients	Low	High	Unbiased
Productivity and work-loss time	Very high	Medium	Too high or too low
Disease spread	NA	NA	Unbiased

*Time horizon of 12 months.

†Level of detail presented is not sufficient to estimate correspondence.18

CER, cost-effectiveness ratio; NA, not applicable.

The study by Hoeijenbos et al14 differs from other studies, as the implementation strategy is aimed at physiotherapists working in primary care. Since the Danish guideline on management of LBP in primary care is also aimed at physiotherapists,3 it is relevant to evaluate whether it would be more cost-effective to increase their adherence to the guideline as neither general practitioners nor physiotherapists have high adherence.21 Application of the 14 specific knockout criteria are shown in table 4. Lack of accordance in the approach to productivity cost estimation between the Netherlands and Denmark increases the risk of a biased CER. The transferability of lost productivity might, according to Welte et al,7 be improved by adjustment of the resource valuation.

Table 4

Transferability of Hoeijenbos et al14 to a Danish setting

Transferability factor	Estimated relevance	Estimated correspondence between the study country and the decision country	Estimation of CER of the study country compared with the decision country is:
Perspective	Very high	Very high	Unbiased
Discount rate	NA*	NA	Unbiased
Medical cost approach	Medium	Very high	Unbiased
Productivity cost approach	Very high	Low	Too low
Absolute and relative prices in healthcare	High	High	Unbiased
Practice variation	Medium	High	Unbiased
Technology availability	Low	High	Unbiased
Disease incidence/prevalence	Low	High	Unbiased
Case mix	Low	Very high	Unbiased
Life expectancy	NA	NA	Unbiased
Health status preferences	Medium	High	Unbiased
Acceptance, compliance, incentives to patients	Low	High	Unbiased
Productivity and work-loss time	Very high	Medium	Too high or too low
Disease spread	NA	NA	Unbiased

*Time horizon of 12 months.

CER, cost-effectiveness ratio; NA, not applicable.

On the basis of the transferability assessment, both the costs from Becker et al and Hoijenbos et al may with adjustments be transferable, with caution, to a Danish setting. However, the model of Welte does not pay much attention to effectiveness, which raises issues in the assessment of the transferability of these two studies. In Becker et al's study, only incremental effect at follow-up was reported, while in Hoijenbos et al's study no final follow-up effect was reported. This renders adjustment of foreign utilities impossible.

Discussion

The systematic review identified no more than three studies that evaluated the cost-effectiveness of different strategies to implement guidelines on management of LBP in primary care.12–14 Application of the CHEC list showed that the three studies were of moderate methodological quality while use of Welte's model showed that cost results from two studies13 14 could, with adjustments, be transferable to a Danish setting, whereas it was questionable whether the effectiveness results could be transferred as well. In a review, Koes et al19 compared international guidelines on managing LBP and showed that recommendations are generally similar, which underlines the relevance of the present comparison of strategies for implementing LBP guidelines. Despite the resemblance of the implementation strategies, the studies nonetheless reported conflicting conclusions of whether the additional upfront cost of the implementation strategies was counterbalanced by improvements in clinical practice or patient outcomes. Neither Mortimer et al nor Hoeijenbos et al found substantial evidence of cost-effectiveness, while Becker et al reported that the intervention was cost-effective compared with the usual implementation strategy. However, to establish which implementation strategy truly procures most value for money, it is important to transfer both costs and effects to the decision country to ensure a valid comparison. This review underlines the importance of transparent reporting to increase transferability and thus facilitate valid comparisons between competing alternatives.

This systematic review, which is the first to look into the cost-effectiveness of strategies used in the implementation of LBP guidelines in primary care, has shown that only a limited amount of research exists within this field. In addition to evidence synthesis, a transferability appraisal was simultaneously carried out to assess whether the results of the economic evaluations could be transferred to a Danish setting. To ensure that all relevant studies were identified, free text search was used in combination with thesaurus terms. Applying a broad search strategy ensured that no relevant studies were overlooked; however, this also resulted in identification of a large number of irrelevant studies. A limitation of this review is that studies only carrying out an analysis of the effectiveness of the intervention were not included. Furthermore, one author carried out the assessment of transferability. It would have been advantageous if two or more independent assessments had been carried out. Moreover, since there are six other checklists of transferability in addition to Welte's model,22 it is possible that application of any of these would have procured another result. Welte's model was chosen as it has more distinctive criteria for evaluation of transferability; however, there are some concerns in the application of this model.

Welte's model is portrayed as transparent and user-friendly; nonetheless, it is based on subjective judgements with low repeatability/reliability, especially with regard to the 14 specific knockout criteria, where there is no clear distinction of whether or not a study is transferable if it has passed the three general knockout criteria. The great attention to the different aspects of cost estimation and valuation is advantageous. Furthermore, it is specified for each of the 14 criteria how transferability might be improved by adjustment of resource utilisation or valuation; this is, however, under the assumption that all subgroups of costs can be transferred between countries, which might not be the case. For the criterion ‘Productivity and work-loss time’, impact on CER was deemed to be too high or too low as choice of method for assessing lost productivity, human capital versus friction method, greatly affects the cost estimation. Since Welte et al7 state that adjustment of both utilisation and valuation might increase transferability, medium correspondence between countries was chosen due to national differences in choice of method for estimating lost productivity. Another study, however, found that due to large differences between countries, transferring lost productivity is not advisable.23

A common measure of effect is necessary when comparing or pooling different economic evaluations, but effectiveness is not paid much attention in the model of Welte, where only one of the 14 specific criteria addresses this subject. The criterion assumes a preference for a generic measure of effect, for example, EuroQol health status values, but does not discuss how transferability might be improved. A number of studies have shown that transferring utilities between countries might produce biased CERs24–29 why the application of a model that adjusts foreign mean utilities to reflect national preferences of health could increase transferability.30 Moreover, there may be cases where a disease-specific measure of effect would be preferred, but this is not reflected in Welte's model. Within the field of LBP research, a set of core outcomes31–33 has been identified to increase comparability across studies, and perhaps one of these outcomes is more readily transferable between countries.

A study by Knies et al34 further supports more focus on the importance of transferability of the effects, as they found that application of Welte's model yielded better results for cost prediction than for effect prediction. However, no assessment of the homogeneity of the studies was carried out, meaning that both the cost and the effect prediction might have procured another result having only included studies that could be pooled.

Applying a systematic checklist ensures a consistent assessment across studies. However, as Welte et al emphasise that the model is a first attempt, it could be relevant to update this model based on the aforementioned as well as positive features of the other transferability checklists.2 35–39 Furthermore, in future research it could be interesting to apply the different checklists to the same studies to compare and contrast transferability results.

Conclusion

A comparison of all relevant strategies for implementing guidelines on LBP management in primary care could provide insight into which strategy procures most value for money. This systematic review, however, showed that transferring the results from the identified studies is not straightforward and underlines the importance of transparent reporting. Moreover, increased focus on transferability of effect is desirable, for example, as a supplement to the model of Welte.