Systematic assessment of the quality of osteoporosis guidelines.

BACKGROUND: Numerous agencies have developed clinical practice guidelines for the management of postmenopausal osteoporosis. The study objective was to conduct a systematic assessment of the quality of osteoporosis guidelines produced since 1998.
METHODS: Guidelines were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, bibliographies of retrieved guidelines, and through consultation with content experts. Each guideline was then assessed by three independent appraisers using the 'Appraisal Instrument for Clinical Guidelines' (version 1) by Cluzeau.
RESULTS: We identified 26 unique guidelines from 1998-2001 and 21 met our inclusion criteria. Of the 21 guidelines reviewed, 8 were developed by medical societies, 6 by national groups, 6 by government agencies, and 1 by an international group. Twelve of the guidelines were published, 7 were organizational reports, and 2 were accessible only from the web. Half or more of the 20 items assessing the rigor of guideline development were met by 15% (median quality score 23%, range 5-80%, (95% CI 16.5, 34.7)), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, range 17-83%, (95% CI 50.8, 65.5)), and none met half or more of the items assessing guideline application (median score 0%, range 0-47%, (95% CI -0.5 to 12.6)). Eight guidelines described the method used to assess the strength of evidence, and in 6 there was an explicit link between recommendations and the supporting evidence. Ten guidelines were judged not suitable for use in practice, 10 were acceptable with modification, and one was acceptable for use without modification.
CONCLUSION: The methodological quality of current osteoporosis guidelines is low, although their scores for clinical content were higher. Virtually no guidelines covered dissemination issues. Few guidelines were judged as acceptable for use in their current format.

Background

Clinical practice guidelines are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" [1]. In many respects this is an ideal time to develop guidelines in the management of postmenopausal osteoporosis, since large randomized controlled trials of various therapies with clinically important outcome measures such as fractures have been conducted [2]. In the past few years, numerous agencies and organizations have developed clinical practice guidelines (CPGs) for the management of post-menopausal osteoporosis. For the primary care physician, it is difficult to assimilate all the information contained in the numerous guidelines and decide which guideline to use in clinical practice. Recent research demonstrated that the application of two different guidelines [3] and [4] to a cohort of women who had sustained a fracture resulted in quite different estimates of the proportion of women who should be treated (87.9 versus 58.9%) [5].

CPGs may have the potential to improve the use of research findings and ultimately the care of patients, but only if the guidelines are associated with rigorous development. Special consideration should be given to the implementation of guidelines, including the needs of the users. A recent needs assessment conducted by Jaglal et al. revealed that family physicians wanted guidelines or information aids about osteoporosis that have significant input from family physicians [6]. Additionally, with looming health care shortages, CPGs are increasingly used as the basis for the rationing of resources. It is therefore important that the guidelines are developed with methodological rigour, then appropriately disseminated and implemented [7].

The objective of this study was to identify and conduct a systematic assessment of the most recent postmenopausal osteoporosis guidelines produced from 1998–2001 that made recommendations on therapy. This process involved the systematic evaluation of the quality of the guidelines using a validated quality assessment tool [7].

Methods

Search strategy

Guidelines pertaining to postmenopausal osteoporosis were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, references of retrieved guidelines and through consultation with content experts. We restricted the search to English language items. Search terms included the following medical headings and text words: osteoporosis practice guidelines, osteoporosis, consensus, consensus statements, and osteoporosis standards. We also searched known websites listing guidelines: such as: , , .

Inclusion criteria

We included those documents that had clinical recommendations, that used a group process to formulate recommendations and that included a bibliography of relevant references.

Data extraction

For each guideline, the following descriptive information was extracted: the year of development, type of developer, publication status (peer-reviewed or not published) stated industry sponsorship, and clinical recommendations of pharmaceutical and non-pharmaceutical interventions.

The standardized quality instrument used to assess the guidelines was the 'Appraisal Instrument for Clinical Guidelines' (version 1, 1999) by Cluzeau and colleagues [7]. This instrument is currently widely used in the United Kingdom and Canada, and has been shown to have acceptable reliability and criterion validity [7]. This instrument uses the Yes, No and Not Sure format, consists of 37 items and is divided into 3 dimensions which assess the adequacy of documentation (Table 1). Dimension 1 (20 items) focuses on the rigour of guideline development and reflects attributes necessary to enhance guideline validity and reliability. It inquires about the responsibility and endorsement of the guidelines, composition of the development group, identification and interpretation of the evidence, the link between the evidence and recommendations, and plans to update the guideline. Dimension 2 (12 items) focuses on the context and content and addresses attributes of reliability, applicability, flexibility and clarity. The 12 items appraise the objectives of the guidelines, the presentation and format, the estimated outcomes-benefits, costs and harms. Dimension 3 (5 items) addresses the dissemination, implementation, and monitoring of the guidelines.

Table 1

Proportion of Guidelines Meeting Quality Criteria

Quality Criteria*		% CPGs where 2 or more appraisers said "yes"
Items included in Dimension 1: Rigour of Method Development
1	Is the agency responsible for the development and/or endorsement of the guidelines clearly identified?	95.2
2	Was funding or other support received for developing the guidelines?	38.1
3	If funding or other support was received, is there evidence that the potential biases of the funding body(ies) were taken into account?	14.3
4	Is there a description of the individuals (e.g. professionals, interest groups – including patients) who were involved in the guidelines development group?	85.7
5	If so, did the group contain representatives of all key disciplines?	23.8
6	Is there a description of the sources of information used to select the evidence on which the recommendations are based?	5.0
7	If so, are the sources of information adequate?	5.0
8	Is there a description of the method(s) used to interpret and assess the strength of the evidence?	38.1
9	If so, is(are) the method(s) for rating the evidence satisfactory?	23.5
10	Is there a description of the methods used to formulate the recommendations?	28.5
11	If so, are the methods satisfactory?	9.6
12	Is there an indication of how the views of interested parties not on the panel (or the consensus conference participants) were taken into account?	19.6
13	Is there an explicit link between the major recommendations and the level of supporting evidence?	28.5
14	Did the guidelines receive an independent external review prior to their publication/release?	28.6
15	If so, is explicit information given about methods and how comments were addressed?	0
16	Were the guidelines piloted?	0
17	If the guidelines were piloted, is explicit information given about the methods used and the results adopted?	0
18	Is there a mention of a date for reviewing or updating the guideline?	19.1
19	Is the body responsible for the reviewing and updating clearly identified?	19.0
20	Overall, have the potential biases of guideline development been adequately dealt with, i.e. have they attempted to minimize the introduction of bias adequately?	14.3
Items included in Dimension 2: Context and Content
21	Are the reasons for developing the guidelines clearly stated?	90.5
22	Are the objectives of the guidelines clearly defined?	61.9
23	Is there a satisfactory description of the patients to which the guidelines are meant to apply?	81.0
24	Is there a description of the circumstances (clinical or non-clinical) in which exceptions might be made in using the guidelines?	28.5
25	Is there an explicit statement of how patient preferences should be taken into account in applying the guidelines?	14.3
26	Do the guidelines describe the condition to be detected, treated, or prevented in unambiguous terms?	95.3
27	Are there different possible options for management of the condition clearly stated in the guidelines?	90.4
28	Are the recommendations clearly presented?	85.7
29	Is there an adequate description of the health benefits that are likely to be gained from the recommended management?	76.2
30	Is there an adequate description of the potential harms or risks that may occur as a result of the recommended management?	61.9
31	Is there an estimate of the costs or expenditures likely to incur from the recommended management?	33.4
32	Are the recommendations supported by the estimated benefits, harms and costs of the intervention?	9.6
Items included in Dimension 3: Application
33	Does the guideline document suggest possible methods for dissemination and implementation?	9.6
34	Does the guideline document specify criteria for monitoring adherence to the guidelines?	9.6
35	Does the guideline document identify clear standards or targets for adherence to the guidelines?	4.8
36	Does the guideline document define measurable outcomes (e.g. health, process, economic, outcomes) that can be monitored?	0
37	Does the guideline document identify key elements which need to be considered by local guideline groups?	0

*Items are presented in the same order as they appear in the appraisal instrument

We included an evaluation of the global assessment of guidelines according to a similar strategy used in another study [7]. Appraisers were asked whether they would "strongly recommend this guideline for use in practice without modifications", "recommend this guideline for use in practice on the condition of some alterations", or "not recommend this guideline". We also included a global quality rating using an 11-point scale, with 0 indicating the lowest possible quality and 10 representing the highest possible quality [8]. Where the information pertaining to background development was lacking, we searched listed websites and attempted to contact authors for further details.

Three appraisers using the Cluzeau instrument independently assessed each guideline. The three appraisers consisted of a rheumatologist/epidemiologist (AC), physiotherapist/methodologist (LW) and one of either: geriatrician/epidemiologist (MM-S-H), geriatrician (AB) and clinical biochemist (DO). Using the assessments, we calculated the frequency with which the guidelines adhered to each of the 37 appraisal items. Adherence was defined by agreement by at least two of the three appraisers.

Next, using the values assigned by each appraiser for each item, we calculated a summary score for each dimension: Dimension 1 – Rigour of development, Dimension 2-Context and content and Dimension 3-Application. A "yes" response by an appraiser was assigned a value of 1 and all other responses 0.

Statistical analysis

The dimension quality score was derived by calculating the mean of the appraisers' scores. Each dimension was expressed as a score out of 100% (95% CI) so that scores could be compared across the three dimensions.

Appraiser agreement was evaluated by calculating the percentage of guidelines for which the three appraisers scored the quality of each dimension within 20 percentage points of each other and by calculating intraclass correlation coefficients (ICC) for the three dimensions. Reliability was assessed by internal consistency (Cronbach's alpha) of each dimension by calculating the correlation between all items within a dimension to test to what extent they measured the same underlying concept. Pearson's correlation coefficients were used to evaluate the correlation between appraisers' dimension scores and global quality scores. Differences in mean quality were analyzed using the Mann-Whitney test. All analyses were completed using SPSS/PC Windows version 10.0.

Results

We identified 26 unique guidelines from 1998–2001: 21 were retrieved and on assessment met our inclusion criteria [3,4,9-27]. Five were excluded for the following reasons: three did not include clinical recommendations or were review articles [28-30] and two were not retrievable [31,32]. Consultation with content experts identified seven guidelines, results of a Medline search identified four further guidelines; search of the World Wide web yielded three guidelines, and known guideline websites yielded six potential guidelines and review of references a further seven guidelines. Eight of the 21 guidelines reported receiving outside funding, and 6/8 of the guidelines acknowledged funding from industry.

Twelve guidelines were published, seven were organizational reports, and two were accessible only from the web. Guideline developers included medical societies (n = 8), national groups (n = 6), international groups (n = 1), and government agencies (n = 6). Table 1 presents the percentage of the guidelines that satisfied each of the 37 quality criteria. Figure 1 illustrates the distribution of the guidelines by dimension quality score. Twenty of the 21 guidelines did not include patients/consumers on the development panel. Only 11 of the 21 included family physicians.

Figure 1

Frequency Distribution of Dimension Scores

Half or more of the 20 items assessing the rigor of guideline development were met by 15% of the guidelines (median quality score 23%, (95% CI 16.5, 34.7, range 5–80%), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, (95% CI 50.8, 65.5), range 17–83%), and none met half or more of the items assessing guideline application (median score 0%, (95% CI -0.5, 12.6) range 0–47%).

Table 2 presents the mean dimension quality scores for the 21 osteoporosis CPGs. Eight guidelines described the method used to interpret and assess the strength of evidence. Six of the 21 guidelines provided an explicit link between major recommendations and supporting evidence.

Table 2

Mean Quality Scores of Osteoporosis Practice Guidelines (CPG), Maximum score = 100 for each Dimension (D)

			D1	D2	D3
Guideline Developer (Ref)	CPG Title	Acceptable for use	Rigor of Development	Context and Content	Implementation
American College Obstetrics Gynecology [15]	Osteoporosis	No	5	50	0
American Family Physician [26]	Prevention of osteoporosis and fractures	No	5	56	0
Cdn Consensus Menopause and Osteoporosis [19]	Osteoporosis	No	8	58	0
Hong Kong PG [10]	Clinical management guidelines for osteoporosis in Hong Kong	No	12	39	0
ASCP American Society of Consultant Pharmacists [16]	Diagnosis and Treatment of OP in long term care Facilities	Yes	12	64	0
Israeli Centre for Technology Assessment [25]	Consensus statement on prevention and treatment of osteoporosis	No	12	17	0
Finland [22]	Osteoporosis: Diagnostics and pharmacotherapy	No	12	33	0
Foundation for Osteoporosis Research and Education [11]	Guidelines of care on osteoporosis for the primary care physician	Yes	13	53	0
International Committee for Osteoporosis Clinical Guidelines [23]	Diagnosis and management of OP in postmenopausal women: clinical Guidelines	No	15	47	0
Brigham & Womens Hospital [14]	Osteoporosis: guide to prevention, Diagnosis and Treatment	Yes	15	78	0
Osteoporosis Coalition of New Jersey [12]	Recommended practice guidelines for the diagnosis and treatment of osteoporosis	Yes	23	61	7
American Medical Directors Association [13]	Osteoporosis: clinical practice guideline	Yes	23	75	0
Society for Obstetrician and Gynecologists of Canada [17]	An Evidence-based review of the management of osteoporosis	No	27	44	0
National Osteoporosis Foundation [3]	Osteoporosis; evidence for prevention, diagnosis and treatment and cost-effectiveness	Yes	27	83	0
American Association of Clinical Endocrinologists [9]	Guidelines for clinical practice for prevention and management of postmenopausal osteoporosis	Yes	32	64	0
NIH Consensus Statement [20]	Osteoporosis Prevention, Diagnosis and Therapy	No	32	67	0
CREST [18] Clinical Resource Efficiency Support Team	Guidance on the prevention and treatment of osteoporosis	Yes	33	58	0
London Department of Health [21]		No	35	53	40
South African Medical Journal [27]	Osteoporosis Clinical Guideline	Yes	58	72	0
Ontario Program of Optimal Therapeutics [24]	Ontario Guidelines for the prevention and treatment of osteoporosis	Yes	61	67	0
Royal College of Physicians [4]	Osteoporosis: clinical guidelines for prevention and treatment	Yes	80	83	53

The mean global quality score was 5.23 (95% CI 4.52, 5.94; median 5.33, standard deviation 1.56), suggesting that the appraisers felt the guidelines to be of medium quality.

The mean global assessment rating demonstrated that 11 of the 21 guidelines were recommended for use in practice (1 without change, 10 with modifications) and 10 guidelines were not recommended for use in practice.

Assessment of the reliability of the guideline appraisal instrument showed that D1 and D3 had good internal consistency, and D2 was borderline. The Cronbach's alpha was 0.87 for rigour of guideline development, 0.64 for context and content, and 0.71 for application.

For agreement between appraisers, the intraclass correlation coefficients were 0.91 for D1, 0.63 for D2 and 0.93 for D3. The appraisers' scores were within 20 percentage points for 95%, 67% and 100% of the guidelines for D1, 2, and 3, respectively. The Pearson's correlation coefficients between appraisers' dimension scores and their global quality scores (0–10) were 0.71 for D1, 0.60 for D2 and 0.46 for D3, with all coefficients being significant. The Pearson's correlation coefficients between appraisers' dimensions scores and their overall global assessment (recommend or not) were 0.59, 0.61 and 0.49 for D1, D2 and D3 respectively, with all coefficients being significant.

Eighteen of 21 guidelines included both pharmaceutical and non-pharmaceutical options (fall prevention, exercise or hip protectors). Ten of the guidelines included recommendations on fall prevention, and nine included recommendations on hip protectors.

There was no significant difference in the mean quality assessment when analyzed according to industry funded versus non-funded guidelines, type of publication (published, web or report), or the type of organization. There was no significant difference in quality according to year of publication, although there appeared to be a trend to improved quality from 1998 to 2001. Only 1/8 of the guidelines that acknowledged funding gave evidence that the potential biases were taken into account.

Discussion

The methodological quality with respect to the rigor of development and implementation of current osteoporosis guidelines is low, although their scores with respect to clinical content are higher. Virtually no guidelines covered the application/dissemination issues. In particular, they did not define outcomes that could be monitored and failed to identify key elements that could be monitored by local guideline groups. Only one guideline was judged acceptable for use in its current format [4]. Clinical practice guideline developers are usually clinicians, or methodologists who have little or no training in the behavioural and organization sciences. Tremendous time and effort are devoted to the development of evidence-based guidelines and the dissemination and implementation are often considered to be a separate component which has only recently thought to be the jurisdiction of the CPG developers.

Our findings are similar to other published reviews of guidelines [8,33]. We also chose to limit our study to one clinical area and to focus on recent guidelines from 1998–2001, recognizing that guidelines currently being developed may follow a more rigorous methodology. Our results indicate that it is important to search both the Internet and Medline as some guidelines were only posted on the web and that retrieving all relevant guidelines can be a difficult task that requires a comprehensive search strategy.

Another potential limitation is that very few of the guidelines encouraged patient input into the development of CPGs. Potential benefits of including patients in the guideline development process include incorporation of priority setting from the patient perspective, improved adherence with recommendations, and reinforcement of the shared decision-making process.

Twenty-nine percent of the guidelines acknowledged involvement or support from industry. Choudry et al. conducted a survey of CPG authors and found 87% had some form of interaction with the pharmaceutical industry, with 58% receiving financial support for research. Given the heavy involvement of industry in the osteoporosis field, one would anticipate that the majority of the guideline developers would have relationships with industry that predated the guideline creation process. Industry has a vested interest in supporting the recommendations of osteoporosis guidelines, which raises concerns about industry's potential to influence the guideline development process. Acknowledgement of the degree of support/involvement with industry in guideline development is important and Choudry et al. [34] recommended that there be a formal process for discussing these conflicts prior to CPG development.

In addition to the goal of promoting good clinical practice and enhancing appropriate management, another potential goal of developing clinical practice guidelines is to improve the use of health care resources. However, we found that over two-thirds of guidelines neglected to include costs. The majority of guideline recommendations are based on results from efficacy, not effectiveness studies and therefore are not always useful in clinical practice [35]. While the methodological quality of guidelines is a prerequisite to the development of good guidelines, it is not sufficient to ensure improvements in clinical practice. In addition, research methodologies on the impact of evidence-based guidelines in clinical practice would be valuable.

Educational material, including the dissemination of practice guidelines has been shown to have little or no effect on promoting behavioural change among health professionals [36]. In contrast, strategies that encourage the development of a local guideline initiative have been shown to result in an increased uptake and dissemination. Only one of the osteoporosis guidelines that we reviewed encouraged development of a local initiative [4].

Conclusions

It is clear from the results of our systematic assessment that the developers of future osteoporosis guidelines need to incorporate greater methodological rigor and patient involvement into the development process. It would also be beneficial if guideline developers could follow common standard of reporting similar to that used by CONSORT for RCTs [37].

The multitude of guidelines dealing with postmenopausal osteoporosis makes it difficult for individual clinicians to decide which guideline to follow. Perhaps greater collaboration and sharing of resources between developer groups would result in fewer guidelines, but with local input as appropriate. More of the resources and energy could then be directed towards the dissemination process.

Competing interests

None declared.

Authors' contributions

AC participated in design of the study, evaluation of guidelines, data analysis, and drafting the manuscript. LW participated in the design, coordination of the study, evaluation of guidelines and commented on the paper. IG participated in design of the study, statistical analysis and writing of manuscript. MM participated in the evaluation of guidelines and provided feedback on the manuscript. AB participated in the evaluation of guidelines and provided feedback on manuscript. DSO participated in evaluation phase and was involved in preparation of the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: