Evaluation of reporting quality in clinical practice guidelines for acute myeloid leukemia using the RIGHT checklist.

BACKGROUND: Acute myeloid leukemia (AML) is an aggressive hematologic malignancy. Clinical practice guidelines (CPGs) on the management of AML have great value in clinical practice. However, the reporting quality of CPGs for AML has not yet been evaluated. This is the first study aiming to evaluate the reporting quality of the most recent AML CPGs published worldwide using the Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist.
METHODS: We systematically searched PubMed, Chinese National Knowledge Infrastructure (CNKI), Wanfang, and Chinese Biomedical Literature (CBM) to extract CPGs for AML published between January 2016 and December 2020. Websites for guideline development organizations and medical associations were also searched. Two independent researchers assessed compliance of the guidelines to each of the 35 checklist items and summarized reporting rates for the 7 domains of the RIGHT checklist.
RESULTS: We identified 16 guidelines, of which 3 (18.8%) were written in Chinese and 13 (81.3%) were written in English. The average overall reporting rate of the 16 guidelines was 52.9%, and only 7 CPGs (43.8%) had a reporting rate >50%. The average reporting rates of the 7 domains (basic information; background; evidence; recommendations; review and quality assurance; funding, declaration, and management of interests; and other information) were 79.2%, 62.5%, 38.8%, 53.6%, 21.9%, 32.8%, and 43.8%, respectively. For the 35 checklist items, the average reporting rate was 52.9%, and only 16 items had a reporting rate >50%, of which 5 items were reported by all the guidelines. There was 1 item which was not reported by any of the guidelines.
CONCLUSIONS: The reporting quality of recently published AML guidelines remains poor. While the recommendations of CPGs have great value in clinical practice, the reporting quality of CPGs for AML still needs to be improved. 2021 Annals of Translational Medicine. All rights reserved.

Introduction

Acute myeloid leukemia (AML) is a highly heterogeneous molecular phenotype characterized by the malignant transformation of hematopoietic stem or progenitor cells (1). As the most common form of acute leukemia in adults, AML remains a devastating disease, with an annual incidence of approximately 16,000 new cases in China and a 5-year overall survival (OS) of only 10% (2). Effective treatment of AML is very challenging (3). For over 4 decades, therapeutic options for AML were limited to high-dose cytotoxic chemotherapy, followed by either allogeneic stem cell transplant or consolidation chemotherapy (4). An improved understanding of the genomic and molecular landscape of AML has resulted in better molecular characterization, leading to more accurate prognostic stratification and treatment decisions. Additionally, advances in treatment with newly approved drugs have resulted in the updating of clinical practice guidelines and paved the way for a new therapeutic era for AML (5).

Clinical practice guidelines (CPGs), which are based on systematic evaluations of evidence, provide recommendations on the management of diseases to guide, optimize, and establish norms for clinical practice (6). An important tool, CPGs standardize the behavior of clinicians to enhance the quality of medical care and also facilitate the allocation of medical resources and protect the rights and interests of patients. Most importantly, updates in CPGs regarding new drugs and therapies can expand the clinical benefit to a broader patient population through the dissemination of guidelines (7). An increasing number of academic organizations and institutions have formulated CPGs for the management of diseases to optimize medical care.

Reporting checklists can assist in developing CPGs and also be used to evaluate the reporting quality of guidelines (8). The Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist, which contains an elaboration statement with detailed information and examples, has been widely recognized as a standard for reporting criteria and is used to assess the reporting quality of CPGs for different diseases (9-12). However, the reporting quality of CPGs for AML has not been evaluated. We used the RIGHT checklist to evaluate the reporting quality of AML CPGs published between 2016 and 2020. The aim of this study is to support a more comprehensive, clear, and transparent reporting of CPGs in this field and to provide recommendations for guideline development in the future.

Methods

Search for AML CPGs

Two independent researchers performed systematic and detailed searches of Medline (via PubMed), Chinese National Knowledge Infrastructure (CNKI), Wanfang, Chinese Biomedical Literature (CBM), and other databases to obtain AML CPGs for the last 5 years [2016-2020], the PubMed search strategy was shown in Appendix 1. In addition, official websites of relevant organizations and medical associations were also searched, including the World Health Organization (WHO), European Society for Medical Oncology (ESMO), National Comprehensive Cancer Network (NCCN), Guidelines International Network (GIN), National Institute for Health and Care Excellence (NICE), American Society of Clinical Oncology (ASCO), and Chinese Society of Clinical Oncology (CSCO). The searches were conducted in January 2021.

Inclusion criteria

CPGs were extracted based on the following inclusion criteria: (I) they were related to AML, including screening, surveillance, diagnosis, treatment, or follow-up of AML; (II) they were published in English or Chinese; and (III) they were published publicly in a peer-reviewed journal or on a website.

Exclusion criteria

Documents were excluded if they were: (I) older versions of guidelines when newer versions were available and accessible; (II) translations and interpretations of guidelines; (III) expert consensus statements; (IV) repeatedly published guidelines, and (V) guidelines for which the full text was inaccessible.

Data extraction

All eligible studies were imported into EndNote X9 to eliminate duplicates. Based on the inclusion and exclusion criteria, the titles, abstracts, and full texts were screened by two researchers. Two additional researchers independently screened and cross-checked the guidelines again. Disagreements were resolved through consensus or consultation with another expert adjudicator.

The RIGHT checklist consists of 22 key items and employs a clear and detailed implementation process. Some items are further divided into several subitems, for a total of 35 items. These items are organized into 7 domains: basic information; background; evidence; recommendations; review and quality assurance; funding, declaration, and management of interests; and other information (Table S1). Four researchers independently assessed the reporting quality of the included guidelines one by one, with “Yes” indicating full or partial reporting of the necessary information and “No” indicating no reporting; if an item did not apply to a particular guideline, it was assigned “Not applicable” (NA) (Table S2). The decision was referred to another expert adjudicator when disagreement was encountered.

Statistical analysis

We calculated the number of checklist items that were reported and defined the reporting quality as (number of reported)/35. The overall reporting rate for the included guidelines, the reporting rate for each domain, and the reporting rate for each item were calculated. If the reporting rate was lower than 50%, the reporting quality of the guideline was regarded as low (13). The data were calculated and analyzed using SPSS 20.0 and Microsoft Office Excel 2019.

Results

Data overview

In total, 341 relevant documents were extracted from the databases. After reading the titles, abstracts, and full texts of the relevant documents, 16 guidelines that met the study criteria were identified (14-29) (). Of the 16 CPGs, 13 (81.3%) were written in English, and 3 (18.8%) were written in Chinese. Four (25%) of the 16 guidelines were from the United States, 3 (18.75%) were from China, 3 (18.75%) were from Europe, 2 (12.5%) were from Canada, 2 (12.5%) were from Japan, 1 (6.25%) was from Brazil, and 1 (6.25%) was from India. For 2 (12.5%) of the guidelines, the author type was individual, and the rest (87.5%) were institutional. One (6.25%) CPG was published in 2016, 5 (31.25%) in 2017, 2 (12.5%) in 2018, 2 (12.5%) in 2019, and 6 (37.5%) in 2020. The characteristics of the 16 CPGs for AML are summarized in .

Figure 1

Flowchart of systematic search.

Table 1

Characteristics of the guidelines included in the study

Number	Title	Year of publication	Reporting rate (%)	Developer	Country or region	Journal or website of publication
1	Treatment of acute promyelocytic leukemia in older patients: recommendations of an International Society of Geriatric Oncology (SIOG) task force (14)	2020	45.7	An International Society of Geriatric Oncology (SIOG)	United States	Journal of Geriatric Oncology
2	Acute myeloid leukemia in adult patients: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up (15)	2020	65.7	European Society for Medical Oncology (ESMO)	Europe	Annals of Oncology
3	JSH practical guidelines for hematological malignancies, 2018: I. Leukemia—2. Acute promyelocytic leukemia (APL) (16)	2020	37.1	The Japanese Society of Hematology	Japan	International Journal of Hematology
4	JSH Practical Guidelines for Hematological Malignancies, 2018: I. Leukemia-1. Acute myeloid leukemia (AML) (17)	2020	37.1	The Japanese Society of Hematology	Japan	International Journal of Hematology
5	American Society of Hematology 2020 guidelines for treating newly diagnosed acute myeloid leukemia in older adults (18)	2020	97.1	American Society of Hematology	United States	Blood Advances
6	NCCN Guidelines Version 4.2020 Acute Myeloid Leukemia (19)	2020	28.6	National Comprehensive Cancer Network (NCCN)	United States	NCCN
7	SIOP PODC adapted risk stratification and treatment guidelines: Recommendations for acute myeloid leukemia in resource-limited settings (20)	2019	65.7	Pediatric Oncology in Developing Countries (PODC)	India	Pediatric Blood & Cancer
8	Management of Acute Promyelocytic Leukemia: Updated Recommendations from an Expert Panel of the European Leukemia Net (21)	2019	57.1	An Expert Panel of the European Leukemia Net	Europe	Blood
9	Initial Diagnostic Work-Up of Acute Leukemia: ASCO Clinical Practice Guideline Endorsement of the College of American Pathologists and American Society of Hematology Guideline (22)	2018	91.4	The College of American Pathologists and American Society of Hematology	United States	Journal of Clinical Oncology
10	Diagnosis and management of AML in adults: 2017 ELN recommendations from an international expert panel (23)	2017	77.1	An International Expert Panel	Europe	Blood
11	Treatment of older patients with acute myeloid leukemia (AML): revised Canadian consensus guidelines (24)	2017	40.0	Expert Panel	Canada	American Journal of Blood Research
12	Brazilian Guidelines on HSCT in Acute Myeloid Leukemia (25)	2017	54.3	Brazilian Society of Bone Marrow Transplantation	Brazil	European Journal of Hematology
13	Management of relapsed and refractory childhood acute promyelocytic leukemia: recommendations from an international expert panel (26)	2016	42.9	An International Expert Panel	Canada	British Journal of Hematology
14	Guidelines for the diagnosis and treatment of acute promyelocytic leukemia in China (2018 edition) (27)	2018	37.1	Leukemia and Lymphoma Group, Hematology Branch of Chinese Medical Association	China	Chinese Journal of Hematology
15	Adult acute myeloid leukemia (non-acute promyelocytic leukemia) Chinese diagnosis and treatment guidelines (2017 edition) (28)	2017	34.3	Leukemia and Lymphoma Group, Hematology Branch of Chinese Medical Association	China	Chinese Journal of Hematology
16	Chinese Journal of Diagnosis and Treatment of Relapsed and Refractory Acute Myeloid Leukemia (2017 Edition) (29)	2017	34.3	Leukemia and Lymphoma Group, Hematology Branch of Chinese Medical Association	China	Chinese Journal of Hematology

Reporting quality of the guidelines

The mean reporting rate of the 16 CPGs for AML was 52.9% (28.6–97.1%). Only 7 out of the 16 CPGs (43.8%) had a reporting rate >50%, and 2 guidelines had a reporting rate above 90% (). Six guidelines reported <40% of the 35 items. The 13 guidelines in English were published in journals and could be retrieved through PubMed, while the 3 Chinese guidelines were accessible only in Chinese journals. The mean reporting rates of guidelines written in Chinese and in English were 35.2% (34.3–37.1%) and 56.9% (28.6–97.1%), respectively.

Reporting quality of the domains

The mean reporting rates in the 7 RIGHT domains of the 16 CPGs were: 79.2 (50–100%) for basic information; 62.5% (37.5–100%) for background; 38.8% (0–100%) for evidence; 53.6% (28.6–100%) for recommendations; 21.9% (0–100%) for review and quality assurance; 32.8% (0–75%) for funding, declaration, and management of interests; and 43.8% (0–100%) for other information ().

Figure 2

The mean reporting rate for each item of the RIGHT checklist.

Reporting quality of the items

The details of the reporting quality of each item are summarized in . Most of the guidelines identified their document as a guideline and described its focus (items 1a and 1c), with a reporting rate higher than 80%. Similarly, abbreviations and acronyms (item 3) and corresponding developers (item 4) were also sufficiently reported. Items 7 (target population), 9b (list all individuals involved in developing the guideline), 13a/b (provide clear recommendations and present separate recommendations for subgroups) and 20 (access) had reporting rates higher than 80%. Among them, items 1a, 3, 7b, 9b, and 13b were reported by all 16 CPGs (100%). Item 18b was not reported by any of the guidelines, and items 6 (aim of the guideline and specific objectives), 8 (end-users and settings), 10 (healthcare questions), 12 (assessment of the certainty of the body of evidence), 14a/b (rationale/explanation for recommendations), 15 (evidence to decision processes), 16 (external review), 17 (quality assurance), 18a (specific sources of funding), 19b (conflicts of interest), 21 (suggestions for further research), and 22 (limitations of the guideline) were reported by less than 50% of the guidelines.

Figure 3

The mean reporting rate for each domain of the RIGHT checklist.

Discussion

High-quality guidelines provide information on the latest treatment options using reliable evidence, accurate subgroup classification, and transparency, while low-quality guidelines are difficult to interpret and implement, ultimately leading to worse results. The RIGHT checklist was established to assist developers in creating reporting guidelines and help clinicians understand and implement the guidelines.

We used the RIGHT checklist to evaluate 16 AML CPGs of various types published in the past 5 years and found that the guidelines varied greatly in reporting quality. Of the 16 CPGs included in our study, only 7 guidelines reported >50% of the items in the RIGHT checklist, which suggested that most CPGs for AML were of low reporting quality (13).

Of the 16 guidelines, most were developed by European, American, and Asian organizations, and the overall reporting rate of guidelines in Europe and America was slightly higher than that in Asia. In addition, the completeness of the CPGs written in Chinese was lower than that in English. These results showed that an established norm for guideline developers from different geographical regions is particularly important for developing CPGs with high reporting quality.

Of the 7 domains, the basic information domain had the highest reporting rate at 79.2%. The most poorly reported domain was review and quality assurance, with a reporting rate of 21.9%, suggesting that CPG developers tended to include basic information but ignored the review and quality assurance domain. Although most of the items had high reporting rates in the basic information domain, the publication year was not clear for some guidelines, and a summary of the recommendations was not always included. These items are essential for readers who want to know how up-to-date CPGs are and for clinicians to be able to quickly extract the information they require and assess the scope of the guidelines.

Under the background domain, all 16 of the guidelines included subgroups given special consideration (item 7b), which is essential for individualized treatment. However, few guidelines described the intended users of the guideline (item 8a) or the settings in which the guideline was intended (item 8b). Additionally, the developers often did not report on the contributors in guideline development (9a). Reporting on contributors and their roles and responsibilities (e.g., steering group, guideline panel, external reviewer, systematic review team, methodologist, etc.) could increase the accuracy and reliability of the guidelines. The background of AML guidelines should be fully described in the future.

For the evidence domain, items 10 (healthcare questions), 11 (systematic reviews), and 12 (assessment of the certainty of the body of evidence) had a reporting rate lower than 60%. While guideline development is always based on systematic reviews, reporting these items are essential for improving the transparency and accuracy of the guidelines. For item 10, when a guideline states the basis for recommendations in PICO (population, intervention, comparator, and outcome) format and indicates how outcomes were selected and sorted, readers can easily identify useful information and make evidence-based decisions. Items 11 and 12 can greatly assist readers in understanding the evidence and evaluating the accuracy of CPGs. Moreover, these items are essential for peer review and permit any shortcomings in the guidelines to be identified and remedied.

Under the recommendations domain, most CPGs did not report on item 14 (rationale/explanation for recommendations), which suggested that the developers did not regard issues such as values, preferences, cost, and resource implications, among other factors, as important when developing guidelines. Although different management methods may be suitable for specific subgroups, some new drugs are not approved in every country, and even if approved, not every patient can afford expensive drugs. The guidelines that lacked item 14 did not accurately present recommendations and thus are not applicable for larger populations. This means that hematologists cannot easily use the guidelines in different clinical situations. In addition, item 15 (evidence to decision processes) was not fully reported, which indicated that the development of the guideline was not transparent or rigorous enough.

For the review and quality assurance domain, items 16 (external review) and 17 (quality assurance) had reporting rates lower than 30%. And the reporting rate of these items in other diseases such as gastric cancer is also low (30). The lack of such information may lead readers to doubt the quality of the guidelines.

For the funding, declaration, and management of interests’ domain, the RIGHT checklist showed that the reporting quality was low. Item 18b (describing the roles of funders in the different stages of guideline development and in the dissemination and implementation of the recommendations) was not reported by any of the guidelines, suggesting that developers did not pay enough attention to this domain. Experts of multiple interdisciplinary teams may receive grants or consulting fees from the company that developed the drugs included in guidelines. A lack of funding information may lead to the inference that the guideline recommendations may be influenced by multiple interested parties, reducing the credibility of the guidelines.

For the other information domain, some guidelines failed to describe the gaps in the evidence or provide suggestions for future research (or both). Most of the guidelines failed to describe any limitations in the guideline development process or indicate how these limitations might have affected the validity of the recommendations. Such information could provide a reference for readers to use in assessing the suitability of the recommendations and offer guidance for updates and research in the future.

Strengths and limitations

This is the first study to assess the reporting quality of guidelines for AML using the RIGHT checklist to assist developers in standardizing the reporting quality of future guidelines and to help hematologists understand and implement the guidelines. However, there were some limitations to this study. First, only guidelines written in Chinese and English were included, which might have caused selection bias. Second, only guidelines published in journals and websites were included, while guidelines published in books or government documents were not analyzed. In addition, while the RIGHT checklist is not used to assess methodology quality or the effectiveness of guideline recommendations, it can assist guideline developers with reporting and help readers to better understand and implement guidelines.

Conclusions

Our evaluation of AML guidelines using the RIGHT checklist suggested that reporting quality is poor and varies greatly among guidelines. In most of the guidelines we evaluated, the domains with low reporting quality were review and quality assurance; evidence; and funding, declaration, and management of interests. AML guideline developers should pay more attention to these items to improve the standardization of reporting, resulting in reporting that is clearer, more complete, and more transparent to better disseminate and implement advanced guidelines in the future.

The article’s supplementary files as