Reporting quality of practice guidelines on colorectal cancer: evaluation using the RIGHT reporting checklist.

BACKGROUND: Clinical practice guidelines are an essential tool for translating evidence into practice. Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist assists to guide the reporting in guidelines. We used RIGHT to assess the reporting completeness and quality of guidelines on colorectal cancer (CRC).
METHODS: We searched the electronic databases Medline (via PubMed), Chinese National Knowledge Infrastructure (CNKI), Wanfang and Chinese Biomedical Literature (CBM) from January 1st, 2018 to December 1st, 2020 for guidelines on CRC. Websites of guideline development organizations were also searched. Two investigators assessed the reporting quality of the included guidelines, and calculated the numbers of guidelines that were compliant with each RIGHT checklist item and the mean proportions of reported items for each of the seven RIGHT checklist domains.
RESULTS: Twenty-seven colorectal guidelines were included. The proportions of reported items in each RIGHT domain were 71.0% for Basic information, 66.2% for Background, 45.9% for Evidence, 68.8% for Recommendations, 24.1% for Review and quality assurance, 33.3% for Funding and declaration and management of interests, and 40.7% for Other information.
CONCLUSIONS: The reporting quality of colorectal guidelines was moderate. A systematic use of the RIGHT checklist during the development process could improve the reporting quality of guidelines in the future. 2021 Annals of Translational Medicine. All rights reserved.

Introduction

Colorectal cancer (cancer of the colon or rectum, or bowel cancer, CRC) is the third most common malignancy in terms of incidence, and the fourth leading cause of cancer-related death worldwide (1). About 1.85 million new cases of CRC occurred and 881,000 people died from CRC worldwide in 2018 (2). The incidence of CRC has increased in recent years as modern lifestyle factors known to be associated with the risk of CRC (such as certain dietary and lifestyle changes, obesity, sedentary behavior, and alcohol and tobacco use) have become more widespread, which represents a significant burden to the society (3). In 2016, the global burden of CRC was estimated at 17.2 million disability-adjusted life years. Notably, 97% of the disease burden resulted from years of life lost due to premature mortality (4). An increasing number of countries have thus become more concerned about CRC, developing a diversity of clinical practice guidelines for screening, diagnosis and treatment of CRC to optimize patient care. High-quality guidelines can effectively enhance decision-making by clinicians and patients by translating complex scientific research findings into concise recommendations, improving the quality and reducing the cost of medical services (5). However, the reporting quality of CRC guidelines has not yet been systematically and comprehensively evaluated.

The International Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist was developed and launched by a multidisciplinary international working group in 2016 (6). The RIGHT tool contains an explanation and elaboration statement with detailed information and examples, which the AGREE (Appraisal of Guidelines for Research and Evaluation) reporting tool, an early instrument for reporting quality assessment of guidelines, lacks (6). In practice, RIGHT checklist can assist guideline developers in reporting, journal editors and peer reviewers in decision making, and health care practitioners in understanding and implementing guidelines (3,6,7).

Therefore, we used the RIGHT checklist to evaluate the reporting quality of guidelines on screening, detection, diagnosis, treatment and management of CRC published between 2018 and 2020. Our aim was to identify reporting deficiencies reporting in guidelines in this field, and provide suggestions for the development of guidelines in the future. In addition to guidelines published in English, we included guidelines from China as an example of guidelines with a focus on country-specific recommendations, and compared the reporting quality of guidelines from China with that of the rest of the world.

Methods

Literature search

We searched the databases Medline (via PubMed), Chinese National Knowledge Infrastructure (CNKI), Wanfang and Chinese Biomedical Literature (CBM) from January 1st, 2018 to December 1st, 2020 to identify guidelines on CRC. We also searched the websites of the following guideline development and oncology organizations: the World Health Organization (WHO), National Institute for Health and Care Excellence (NICE), Guidelines International Network (GIN), National Comprehensive Cancer Network (NCCN), Scottish Intercollegiate Guidelines Network (SIGN), and Chinese Society of Clinical Oncology (CSCO). We used the following search strategy: (“Guideline” OR “Recommendation”) AND [(“colorectal*” OR “colon*” OR “rect*” OR “intestine*”) AND (“Cancer*” OR “carcinoma*” OR “neoplasm*” OR “adenoma*” OR “adenocarcinom*” OR “tumour*” OR “tumor*” OR “polyp*” OR “malignan*”)], adjusted to the format of each database. The search strategies for different databases are presented in detail in the Supplementary Appendix 1.

Eligibility criteria and study selection

We included all evidence-based guidelines on CRC published between January 1, 2018 to December 1, 2020 in English or Chinese in either peer-reviewed journals or on publicly available websites. We excluded translations, summaries and interpretations of guidelines, as well as older versions of guidelines if an updated edition was available.

Two investigators (Jing Han and Xuan Wu) independently screened first the retrieved titles and abstracts, and then the full texts of the articles deemed potentially eligible, and selected eligible studies according to the predefined criteria. Disagreements were resolved by consensus or by consultation with a senior investigator (Meng Tao).

Data collection

One investigator extracted the following information from the included guidelines: title, publication year, country or region of development, developer, grading system, and the journal or website where the guideline was published. If some of these data could not be retrieved, we contacted the authors of the articles to ask for the missing information. Quality assessment was performed independently by two trained investigators (Jing Han and Xuan Wu), who reviewed the quality of each eligible guideline. Conflicting opinions or disagreements were solved by consulting a third investigator (Meng Tao).

RIGHT checklist

The RIGHT tool provides evaluators a clear, transparent and comprehensive checklist with accompanying explanations and detailed instructions. It contains 35 items grouped into seven domains: basic information, background, evidence, recommendations, review and quality assurance, funding and declaration and management of interests, and other information. Before collecting the data, the researchers attended a training course on the use of the RIGHT checklist to ensure that the evaluation criteria were applied consistently. Most items were evaluated on a dichotomous scale (“Fully Reported” or “Not Reported”). “Fully Reported” means that relevant information was fully reported, and “not reported” that the relevant information was completely missing. If an item did not apply to the guideline, we assigned it as “Not applicable”. For items with multiple contents [for example, “If the guideline developers used existing systematic reviews, reference these and describe how those reviews were identified and assessed (provide the search strategies and the selection criteria, and describe how the risk of bias was evaluated) and whether they were updated”], we added a third possible answer, “Partially Reported”, meaning that a part of the content was reported.

Data analysis

We calculated the percentages of guidelines that were compliant with each of the 35 RIGHT items, as well as the overall reporting rates (the sum of all items assigned as either “Fully Reported” or “Partially Reported” divided by the total number of items) and the mean reporting rates for each of the RIGHT checklist domains (reporting rate of items within each domain) for all guidelines. We presented the mean reporting rates over all guidelines, and for guidelines stratified by continent. We calculated the mean differences (MD) with 95% confidence intervals (CI) comparing the overall and domain reporting rates between guidelines from China and the rest of the world, as well as between guidelines published in peer-reviewed journals and on websites only. T tests were used to test statistical significance. The analyses were conducted with SPSS 24.0 and Review Manager 5.3.

Results

Our search identified a total of 1,015 records, of which 932 remained after removing duplicates. After screening the titles and abstracts, 870 articles were excluded. Finally, 27 guidelines were deemed eligible in the full-text review and included in the analysis ().

Figure 1

Flowchart of the literature search.

Characteristics of the included guidelines

Nine (33.3%) guidelines were developed in the United States (USA) and published mainly by American Cancer Society (ACS), American Society of Clinical Oncology (ASCO) and National Comprehensive Cancer Network (NCCN), 6 (22.2%) by European multinational organizations or collaborations, and four (14.8%) in China. The remaining guidelines were from France (n=2), Spain (n=2), United Kingdom (n=1), Japan (n=1), Italy (n=1) and Canada (n=1) developed by their independent domestic research institutions. Six (22.2%) guidelines were published only on the website of the developer. Twenty-two guidelines (81.5%) performed grading of evidence quality and recommendation strength, of which eleven guidelines used the GRADE system ().

Table 1

Characteristics of the included guidelines

First author	Year	Country/region	Topic	Version	Developer	Rating or Grading System	Publication platform	No. of recommendations
Leddin D (8)	2018	Canada	Screening	Original	The Canadian Association of Gastroenterology	GRADE	Journal	19
Wolf AMD (9)	2018	USA	Screening	Update	American Cancer Society	GRADE	Journal	12
Xu J (10)	2018	China	Diagnosis & treatment	Original	SGSBSCMA; SCSBSCMA; CCPCCACA; CSCRSCCCSCMDA; MRCABCMDA; CCPCCMDA; CCECCSCO; CLMPCCIEPAMHC	Customized	Journal	—
Bisschops R (11)	2019	Europe	Detection	Update	ESGE	GRADE	Journal	15
Lieu C (12)	2019	USA	Treatment	Original	ASCO	GRADE	Journal	3
Lopes G (13)	2019	USA	Detection	Original	ASCO	GRADE	Journal	24
Hassan C (14)	2019	Europe	Management	Original	ESGE and ESDO	GRADE	Journal	7
Phelip JM (15)	2019	France	Diagnosis & treatment	Original	SNFGE, FFCD, GERCOR, UNICANCER, SFCD, SFED, SFRO, SFR	GRADE	Journal	43
Gómez-España MA (16)	2019	Spain	Diagnosis & treatment	Original	SEOM	Customize	Journal	8
Costas-Chavarri A (17)	2019	USA	Treatment	Original	ASCO	AGREE II	Journal	47
Yao H (18)	2019	China	Treatment	Original	CSGCMASBLESGCMASB	–	Journal	–
NCCN (19)	2020	USA	Diagnosis & treatment	Update	NCCN	Customized	Website	–
NCCN (20)	2020	USA	Screening	Update	NCCN	Customized	Website	–
CSCO (21)	2020	China	Diagnosis & treatment	Update	CSCO	Customized	Website	–
Vecchione L (22)	2020	Europe	Management & treatment	Original	ESMO	Customized	Journal	60
Calvo FA (23)	2020	Europe	Treatment	Original	ESTRO/ACROP	–	Journal	–
Calvo FA (24)	2020	Europe	Treatment	Original	ESTRO/ACROP	–	Journal	–
NCCN (25)	2020	USA	Management	Update	NCCN	Customized	Website	–
Hashiguchi Y (26)	2020	Japan	Treatment	Original	JSCCR	GRADE	Journal	42
Argilés G (27)	2020	Europe	Diagnosis & treatment	Original	ESMO	America-United States Public Health Service Grading System	Journal	37
Salvatore L (28)	2020	Italy	Management	Original	IMOA	GRADE	Journal	18
NHCPRC (29)	2020	China	Diagnosis & treatment	Original	NHCPRC	–	Journal	–
NCCN (30)	2020	USA	Diagnosis & treatment	Update	NCCN	Customized	Website	–
Guillén-Ponce C (31)	2020	Spain	Diagnosis & management	Original	SEOM	GRADE	Journal	17
Aparicio T (32)	2020	France	Treatment	Original	SoFOG	–	Journal	–
Chiorean EG (33)	2020	USA	Treatment	Original	ASCO	AGREE II	Journal	58
NICE (34)	2020	United Kingdom	Diagnosis & treatment	Update	NICE	GRADE	Website	–

Customized: specific rating or grading system developed by guidelines developers themselves, different from the universally used system such as GRADE. ESGE, European Society of Gastrointestinal Endoscopy; ASCO, American Society of Clinical Oncology; GRADE, Grading of Recommendations Assessment, Development and Evaluation; ESDO, European Society of Digestive Oncology; SEOM, Sociedad Española de Oncología Médica; NCCN, National Comprehensive Cancer Network; SoFOG, French Society of Geriatric Oncology; NICE, National Institute for Health and Care Excellence; GRADE, Grade of Recommendations Assessment Development and Evaluation; NHCPRC, National Health Commission of the People's Republic of China; IMOA, Italian Medical Oncology Association; ESMO, European Society for Medical Oncology; CSCO, Chinese Society of Clinical Oncology; JSCCR, Japanese Society for Cancer of the Colon and Rectum; SGSBSCMA, Section of Gastrointestinal Surgery, Branch of Surgery, Chinese Medical Association; SCSBSCMA, Section of Colorectal Surgery, Branch of Surgery, Chinese Medical Association; CCPCCACA, Colorectal Cancer Professional Committee, Chinese Anti-Cancer Association; CSCRSCCCSCMDA, Chinese Society of Colon & Rectal Surgeons, Chinese College of Surgeons, Chinese Medical Doctor Association; MRCABCMDA, Metastasis Research Committee, Anorectal Branch of Chinese Medical Doctor Association; CCPCCMDA, Colorectal Cancer Professional Committee, Chinese Medical Doctor Association; CCECCSCO, Colorectal Cancer Expert Committee, Chinese Society of Clinical Oncology; CLMPCCIEPAMHC, Colorectal liver Metastases Professional Committee, China International Exchange and Promotive Association for Medical and Health Care; CSGCMASBLESGCMASB, Colorectal Surgery Group, Chinese Medical Association Surgery Branch; Laparoscopy and Endoscopic Surgery Group, Chinese Medical Association Surgery Branch; ESTRO/ACROP, European Society for Radiotherapy and Oncology-Advisory Committee on Radiation Oncology Practice; SNFGE, est un travail collaboratif sous égide de la Société Nationale Française de Gastroentérologie; FFCD, de la Fédération Francophone de Cancérologie Digestive; GERCOR, du Groupe Coopérateur multidisciplinaire en Oncologie; UNICANCER, de la Fédération Nationale des Centres de Lutte Contre le Cancer; SFCD, de la Société Française de Chirurgie Digestive; SFED, de la Société Française d’Endoscopie Digestive; SFRO, de la Société Française de Radiothérapie Oncologique; ACHBT, de l'Association de Chirurgie Hépato-Bilio-Pancréatique et Transplantation; SFR, et de la Société Française de Radiologie.

Overall reporting quality of included guidelines

The overall reporting rates of the RIGHT items in the 27 guidelines ranged from 22.9% to 85.7%, with a mean of 56.3%. Ten guidelines (37.0%) had a score above 60%, 15 guidelines (55.6%) between 30% and 60%, and two guidelines (7.4%) lower than 30% ().

Table 2

Compliance to each item of RIGHT in the included guidelines(6)

No.	Item	Fully reported, n (%)	Partially reported, n (%)	Not reported, n (%)	Not applicable, n (%)
Basic information
1a	Identify the report as a guideline, that is, with “guideline(s)” or “recommendation(s)” in the title	26 (96.3)	0 (0.0)	1 (3.7)	0 (0.0)
1b	Describe the year of publication of the guideline	13 (48.1)	0 (0.0)	14 (51.9)	0 (0.0)
1c	Describe the focus of the guideline, such as screening, diagnosis, treatment, management, prevention or others	18 (66.7)	0 (0.0)	9 (33.3)	0 (0.0)
2	Provide a summary of the recommendations contained in the guideline	12 (44.4)	0 (0.0)	15 (55.6)	0 (0.0)
3	Define new or key terms, and provide a list of abbreviations and acronyms if applicable	26 (96.3)	1 (3.7)	0 (0.0)	0 (0.0)
4	Identify at least one corresponding developer or author who can be contacted about the guideline	19 (70.4)	0 (0.0)	8 (29.6)	0 (0.0)
Background
5	Describe the basic epidemiology of the problem, such as the prevalence/incidence, morbidity, mortality, and burden (including financial) resulting from the problem	23 (85.2)	0 (0.0)	4 (14.8)	0 (0.0)
6	Describe the aim(s) of the guideline and specific objectives, such as improvements in health indicators (e.g., mortality and disease prevalence), quality of life, or cost savings	16 (59.3)	0 (0.0)	11 (40.7)	0 (0.0)
7a	Describe the primary population(s) that is addressed by the recommendation(s) in the guideline	27 (100.0)	0 (0.0)	0 (0.0)	0 (0.0)
7b	Describe any subgroups that are given special consideration in the guideline	23 (85.2)	0 (0.0)	4 (14.8)	0 (0.0)
8a	Describe the intended primary users of the guideline (such as primary care providers, clinical specialists, public health practitioners, program managers, and policy-makers) and other potential users of the guideline	15 (55.6)	0 (0.0)	12 (44.4)	0 (0.0)
8b	Describe the setting(s) for which the guideline is intended, such as primary care, low- and middle-income countries, or in-patient facilities	4 (14.8)	0 (0.0)	23 (85.2)	0 (0.0)
9a	Describe how all contributors to the guideline development were selected and their roles and responsibilities (e.g., steering group, guideline panel, external reviewer, systematic review team, and methodologists)	11 (40.7)	0 (0.0)	16 (59.3)	0 (0.0)
9b	List all individuals involved in developing the guideline, including their title, role(s) and institutional affiliation(s)	24 (88.9)	0 (0.0)	3 (11.1)	0 (0.0)
Evidence
10a	State the key questions that were the basis for the recommendations in PICO (population, intervention, comparator, and outcome) or other format as appropriate	10 (37.0)	0 (0.0)	17 (63.0)	0 (0.0)
10b	Indicate how the outcomes were selected and sorted	3 (11.1)	0 (0.0)	24 (88.9)	0 (0.0)
11a	Indicate whether the guideline is based on new systematic reviews done specifically for this guideline or whether existing systematic reviews were used	21 (77.8)	0 (0.0)	6 (22.2)	0 (0.0)
11b	If the guideline developers used existing systematic reviews, reference these and describe how those reviews were identified and assessed (provide the search strategies and the selection criteria, and describe how the risk of bias was evaluated) and whether they were updated	4 (14.8)	4 (14.8)	13 (48.1)	6 (22.2)
12	Describe the approach used to assess the certainty of the body of evidence	20 (74.1)	0 (0.0)	7 (25.9)	0 (0.0)
Recommendations
13a	Provide clear, precise, and actionable recommendations	27 (100.0)	0 (0.0)	0 (0.0)	0 (0.0)
13b	Present separate recommendations for important subgroups if the evidence suggests that there are important differences in factors influencing recommendations, particularly the balance of benefits and harms across subgroups	23 (85.2)	0 (0.0)	0 (0.0)	4 (14.8)
13c	Indicate the strength of recommendations and the certainty of the supporting evidence	19 (70.4)	4 (14.8)	0 (0.0)	4 (14.8)
14a	Describe whether values and preferences of the target population(s) were considered in the formulation of each recommendation. If yes, describe the approaches and methods used to elicit or identify these values and preferences. If values and preferences were not considered, provide an explanation	10 (37.0)	0 (0.0)	17 (63.0)	0 (0.0)
14b	Describe whether cost and resource implications were considered in the formulation of recommendations. If yes, describe the specific approaches and methods used (such as cost-effectiveness analysis) and summarize the results. If resource issues were not considered, provide an explanation	19 (70.4)	0 (0.0)	8 (29.6)	0 (0.0)
14c	Describe other factors taken into consideration when formulating the recommendations, such as equity, feasibility and acceptability	16 (59.3)	0 (0.0)	11 (40.7)	0 (0.0)
15	Describe the processes and approaches used by the guideline development group to make decisions, particularly the formulation of recommendations (such as how consensus was defined and achieved and whether voting was used)	12 (44.4)	0 (0.0)	15 (55.6)	0 (0.0)
Review and quality assurance
16	Indicate whether the draft guideline underwent independent review and, if so, how this was executed and the comments considered and addressed	10 (37.0)	0 (0.0)	17 (63.0)	0 (0.0)
17	Indicate whether the guideline was subjected to a quality assurance process. If yes, describe the process	3 (11.1)	0 (0.0)	24 (88.9)	0 (0.0)
Funding and declaration and management of interests
18a	Describe the specific sources of funding for all stages of guideline development	0 (0.0)	16 (59.3)	11 (40.7)	0 (0.0)
18b	Describe the role of funder(s) in the different stages of guideline development and in the dissemination and implementation of the recommendations	0 (0.0)	0 (0.0)	16 (59.3)	11 (40.7)
19a	Describe what types of conflicts (financial and non-financial) were relevant to guideline development	14 (51.9)	0 (0.0)	13 (48.1)	0 (0.0)
19b	Describe how conflicts of interest were evaluated and managed and how users of the guideline can access the declarations	6 (22.2)	0 (0.0)	21 (77.8)	0 (0.0)
Other information
20	Describe where the guideline, its appendices, and other related documents can be accessed	18 (66.7)	0 (0.0)	9 (33.3)	0 (0.0)
21	Describe the gaps in the evidence and/or provide suggestions for future research	11 (40.7)	0 (0.0)	16 (59.3)	0 (0.0)
22	Describe any limitations in the guideline development process (such as the development groups were not multidisciplinary or patients’ values and preferences were not sought), and indicate how these limitations might have affected the validity of the recommendations	4 (14.8)	0 (0.0)	23 (85.2)	0 (0.0)

The details of RIGHT checklist can be found at http://www.right-statement.org/right-statement/checklist.

Reporting quality of each domain

The mean reporting quality in the “Basic information“ domain was highest (71.0%), and the reporting quality in the domain “Review and quality assurance” the lowest (24.1%) among the seven RIGHT domains. The mean reporting proportions in the remaining domains were 68.8% (Recommendations), 66.2% (Background), 45.9% (Evidence), 40.7% (Other information), and 33.3% (Funding and declaration and management of interests) ().

Reporting quality of each item

Items 7a [Describe the primary population(s) that is addressed by the recommendation(s) in the guideline] and 13a (Provide clear, precise, and actionable recommendations) of the RIGHT checklist was reported in all guidelines. No guideline reported item 18b [Describe the role of funder(s) in the different stages of guideline development and in the dissemination and implementation of the recommendations]. Sixteen sub-items (1b, 2, 8b, 9a, 10a, 10b, 11b, 14a, 15, 16, 17, 18b, 19b, 21, 22) were reported by less than half of the guidelines ().

Figure 2

Percentage distribution of the scores for each item of the RIGHT checklist (n=27). The descriptions of each item can be found in . RIGHT, Reporting Items for Practice Guidelines in Healthcare.

The reporting rates of guidelines developed in different continents varied across the seven domains of the RIGHT checklist. For “Basic information”, the guidelines developed in Asia had the highest reporting proportion (83.3%). For the other domains of RIGHT checklist, North America guidelines had the highest reporting rate (). The guidelines from rest of the world had slightly higher reporting quality than those from China (MD =0.26, 95% CI: −0.04 to 0.57, P=0.082); the difference was significant in the domains “Evidence” (P<0.00001), “Recommendations” (P<0.00001) and “Other information” (P=0.0002) (). The reporting quality of guidelines did also not differ significantly between those published in peer-reviewed journals and those published on websites only (MD =0.09, 95% CI: −0.03 to 0.20, P=0.136). Guidelines published in journals had a higher reporting rate in the “Basic information” domain than guidelines published on websites only ().

Figure 3

Comparison of reporting quality in each domain of the RIGHT checklist between guidelines developed in different continents. RIGHT, Reporting Items for Practice Guidelines in Healthcare.

Figure 4

Comparison of reporting quality in each domain of the RIGHT checklist between guidelines developed in China and rest of the world. SD, standard deviation; CI, confidence interval; RIGHT, Reporting Items for Practice Guidelines in Healthcare.

Figure 5

Comparison of reporting quality in each domain of the RIGHT checklist between guidelines published in journals and on websites only. SD, standard deviation; CI, confidence interval; RIGHT, Reporting Items for Practice Guidelines in Healthcare.

Discussion

We reviewed 27 guidelines on CRC to evaluate their reporting quality, and found that the mean compliance to the items of the RIGHT checklist was 56.3%. The reporting rates of items related to the review and quality assurance and funding and declaration and management of interest were lower than of items related to other aspects. The reporting quality of guidelines on CRC is thus suboptimal and needs improvement.

More than half of the guidelines did not provide a summary of the recommendations. Guidelines developers may not fully understand the importance of providing a summary: previous study found that the low reporting rate in the “basic information” domain resulted from the insufficient importance attached by guideline developers (35). Another possible reason is that most journals do not have a specific submission category for guidelines, so when a guideline is published in a journal it needs to fit the standard article format which usually does not contain sections such as a summary of the recommendations. A complete executive summary is however essential as it presents the key points such as the target population and recommendations of the guideline in a concise format, assisting the readers find the key information directly and quickly. Otherwise, guideline users need to read through the full document to find the recommendations relevant to the problem of concern. Most CRC guidelines also did not report the target country or setting that their guideline. Because of the incomplete reporting healthcare practitioners or clinicians may find it difficult to know if and when the guideline is applicable.

Although most guidelines reported the approach and to assess the certainty of the evidence and strength of the recommendations, we identified some guidelines that did not report any method to assess this information. In the era of evidence-based medicine, the development of guidelines and the formation of recommendations should follow standardized procedures. Clear and transparent assessment of the strength of recommendations and the quality of evidence can help clinical practitioners and clinicians to make explicit decisions for specific subpopulations or conditions.

Our comparison between guidelines from China and elsewhere found that guidelines from China had a lower reporting quality in reporting the evidence, review and quality assurance, funding and declaration and management of interests than guidelines from the rest of the world. It is possible that guideline from China developers do not attach enough attention in describing the development process. This finding is supported by another study showing that the adherence of guidelines from China to the RIGHT checklist was poor, below 30% (36). Many users rely on the guidelines developed by a panel of experts in the field who have rich experience in clinical practice but little knowledge about standard guideline procedures in China (37). Thus, many guidelines lack transparency in searching, evaluating and composing evidences to have a poor quality. However, a scientific and standardized process can benefit further acceptance and dissemination of the guidelines, so Chinese developers should pay more attention on presenting the details of the development thoroughly to make up for the present deficiency in the future. Moreover, Asians are different in heredity, body constitution and other aspects with the west, so Asian specific guidelines need to be developed standardly according to RIGHT checklist to provide better guidance to the public.

We found that guidelines published on websites had the trend to report the evidence and recommendations better than those published in journals. In other domains, guidelines in journals tended to be better reported than those published on websites only. Many aspects in these domains such as funding and declaration and management of interests are standard requirements of journal article submissions. Two thirds of the website guidelines we included were NCCN guidelines which are well recognized in the field of oncology, but they nevertheless showed low reporting rates on these areas. Although NCCN guidelines play a significant role in the diagnosis and treatment of CRC to clinicians throughout the world, the reporting rates are inadequate in some domains, and improvements in reporting should be considered.

What is worth mentioning is that the reporting rate of funding, declaration and management of interest seems poor, the second lowest of RIGHT checklist domains in our study. Actually, this problem has existed many years all over the world. When guideline recommendations are controversial, and that’s often, suspicion quickly turns to financial conflicts of interest. The perception of conflicts can call the reliability of a recommendation into question, and even more so if there was no disclosure (38). So developers should attach enough significance to the transparency of disclosure of financial interest. Specifically, clinical guidelines should describe the funding sources and roles of funders for all stages clearly and how conflicts of interest were evaluated and managed and how users of the guideline can access the declarations. Only with adequate financial conflict management, can guidelines become acceptable to a far wider circle.

Multiple tools, including AGREE, have been developed to assess the quality and reliability of guidelines and recommendations (39-41). The RIGHT checklist was developed to primarily give guidance during the actual guideline development process and support clear and transparent reporting, but it also completes the existing tools in assessing reporting quality. RIGHT is an evolving document that is being continuously revised and extended for different types of guidelines, in the future also possibly using the users’ feedback (6). As we found the reporting in many aspects incompliant with the RIGHT checklist items, we anticipate that promoting the consistent use of RIGHT during the guideline development process could considerably improve the quality of guidelines in the future.

Strengths and limitations

To the best of our knowledge, this is the first study to apply the RIGHT checklist to assess the reporting quality for CRC guidelines. We performed critical quality control in the evaluation process, by two investigators conducting the assessment independently followed by a thorough and systematic discussion on any unclear or conflicting opinions. However, the present study has some limitations. We only include guidelines published in English or Chinese, meaning that our findings are not necessarily representative of the situation of CRC guidelines with a more national or local focus.

Questions to be further discussed and considered

Question 1: What impact do you think the low reporting quality of clinical practice guidelines on colorectal cancer will have on clinicians and clinical practices?

Expert opinion: Dr. Shaheenah Dawood

Personally I don't think it will have a major impact on actual care of patients in clinical practice since most clinicians use high quality guidelines such as the NCCN and ESMO. However it can cause confusion when multiple low quality guidelines are published. It thus becomes the responsibility of journals publishing to ensure that submitted guidelines adhere rigorous quality parameters and it becomes also important for authors to be transparent about reporting quality guidelines followed.

Expert opinion: Dr. Colin W. Steele

Clinical guidelines should address the evidence for them, reflected by grades of evidence with clear descriptions of how this was assessed within the guideline. These guidelines should be developed by experts to ensure the most robust and contemporaneous evidence is included. Low reporting quality of certain guidelines may unfavorably promote practices that do not have the strongest evidence base alongside those that do, leading to poorer quality patient care and irregular practices.

Expert opinion: Dr. Ker-Kan Tan

I think 2 issues. Some oncologists and surgeons may have pre-conceived ideas that clinical practice guidelines are inaccurate and biased. And hence, they would continue to do what they have been doing without the backing of the latest evidence. This would then translate to direct implications on patients’ oncological outcomes as well as counselling of the associated risks and complications of each treatment options.

Question 2: What do you think the most important aspects needed for developing high-quality clinical practice guidelines on colorectal cancer are?

Need to group together established experts in the field.

Need to include upcoming experts in the field who are not biased and the contribute appropriately.

Experts should include those who are geographic specific to the guidelines as well as those from an international perspective.

Adhering to the right check list is important.

Surveying the impression of the guidelines within the community it is targeting prior to publication may also be something think about to get feed back from community oncologist.

Clinical practice guidelines should be developed by an international collaborative. A thorough systematic literature search should be performed and data extracted from high quality meta-analysis and randomized controlled trial sources. The guidelines must cover all aspects of the clinical experience and remain contemporaneous. Guidance should be easily accessed and summarized with easy to follow instruction.

To develop high quality CPG in this current field of literature is difficult because of the plethora of open access publications citing their own evidence, whilst also the ever-presence of publication biases and this is complicated by the appointments of certain “famous” clinicians to review the evidence and therefore to come up with the CPGs.

Firstly, the wide-ranging of predatory journals and widely available open access publications often confuse patients, clinicians and academics alike on what are the most robust evidence.

Secondly, only publications with statistically significant results are published due to publication biases. Rarely are well-conducted and large scale surgical trials performed, and even if they do, issues of sample size, heterogeneity and surgical expertise are huge confounders.

Lastly, we often only appoint esteemed clinicians to review the evidence to come up with the CPG who may have fixated ideas. CPGs should involve academics looking at modelling, health economics other than just clinicians in that particular field, say, colorectal cancer.

Question 3: How do you think conflicts of interest in the guidelines should be handled?

This is a very important question as conflict of interest can introduce biases on multiple levels. First authors should be bound to reveal all conflict of interest no matter how insignificant. Failure to declare should carry a hefty penalty. Second journals should investigate thoroughly conflicts that could potentially introduce biases e.g., having stocks in a company that the author is a PI on or having a spouse working in a company for which the author is leading and reporting on clinical trials or authors that have a large chunk of their publications and research focused on a specific drug etc. Here not is only is transparency important but it is vital that such authors not be the lead authors of such guidelines as this can be interpreted as bias. Not only is declaration of conflict interest important but the amount of renumeration obtained should also be declared privately to the journals to decide the level of conflict that could be detrimental.

Conflicts of interest should be stated upfront in an open and clear fashion. Conflicts of interest should be removed from the guidance by having a panel of experts from different areas of colorectal cancer management sitting together to discuss any controversies and come to a consensus as to how to proceed. Those with conflict over certain areas of the guidelines should be excluded from discussions around these parts to try and minimize inherent bias. There should be sufficient depth of this grouping to prevent biased guidelines being proposed. By presenting the quality and depth of evidence for a certain approach any conflict of interest should be overcome.

Conflict of interest is often understated and is subconscious. Although upfront declaration is mandatory, that does not mean that subsequent decisions would not have conflict of interest. Clarity of intent of the guidelines associated with the methodology would be critical to minimise individual’s influence.

Conclusions

Our critical assessment of CRC guidelines found that they were of moderate reporting quality. The reporting of evidence basis, review and quality assurance, and funding and declaration and management of interests were particularly worrying. In the future, guideline developers should pay particular attention on reporting these aspects of the guidelines. The RIGHT checklist is a useful tool to assist guideline developers to report their guidelines clearly and transparently.

The article’s supplementary files as