Do clinical practice guidelines consider evidence about diagnostic test consequences on patient-relevant outcomes? A critical document analysis.

RATIONALE, AIMS AND
OBJECTIVES: Supporting evidence for diagnostic test recommendations in clinical practice guidelines (CPGs) should not only include diagnostic accuracy, but also downstream consequences of the test result on patient-relevant outcomes. The aim of this study is to assess the extent to which evidence-based CPGs about diagnostic tests cover all relevant test-treatment pathway components.
METHODS: We performed a systematic document analysis and quality assessment of publicly accessible CPGs about three common diagnostic tests: C-reactive protein, colonoscopy and fractional exhaled nitric oxide. Evaluation of the impact of the full test-treatment pathway (diagnostic accuracy, burden of the test, natural course of target condition, treatment effectiveness, and link between test result and administration of treatment) on patient relevant outcomes was considered best practice for developing medical test recommendations.
RESULTS: We retrieved 15 recommendations in 15 CPGs. The methodological quality of the CPGs varied from poor to excellent. Ten recommendations considered diagnostic accuracy. Four of these were funded on a systematic review and rating of the certainty in the evidence. None of the CPGs evaluated all steps of the test-treatment pathway. Burden of the test was considered in three CPGs, but without systematically reviewing the evidence. Natural course was considered in two CPGs, without a systematic review of the evidence. In three recommendations, treatment effectiveness was considered, supported with a systematic review and rating of the certainty in the evidence in one CPG. The link between test result and treatment administration was not considered in any CPG.
CONCLUSIONS: The included CPGs hardly seem to consider evidence about test consequences on patient-relevant outcomes. This might be explained by reporting issues and challenging methodology. Future research is needed to investigate how to facilitate guideline developers in explicit reliable consideration of all steps of a test-treatment pathway when developing diagnostic test recommendations.

INTRODUCTION

Clinicians use medical tests to confirm or exclude a clinical diagnosis (e.g., polymerase chain reaction‐test to diagnose COVID‐19), to test the likelihood of a certain clinical diagnosis (e.g., prostate‐specific antigen‐test to screen for risk on prostate cancer) or for follow‐up of patients to monitor recovery (e.g., rehabilitation checklists). Test results guide (treatment) decisions. The clinical value of a medical test depends on various elements: the patient population characteristics (e.g., prevalence of the disease), test characteristics (e.g., sensitivity and specificity) and its downstream consequences (e.g., benefits and harms of treatment) on patient‐important outcomes.

Clinical practice guidelines (CPGs) provide recommendations to support professionals and patients in clinical decision‐making, with the ultimate goal of improving or maintaining patients' health. In the development of CPGs, the benefits and harms of the interventions of interest are systematically assessed with regard to patient‐relevant outcomes. The grading of recommendations, assessment, development and evaluation (GRADE) approach is designed to facilitate this process.

Diagnostic CPGs provide recommendations about the use of a certain test (or test strategy). Supporting evidence for these recommendations consists of studies about diagnostic accuracy. However, acceptable test characteristics (sensitivity and specificity) are not enough to improve patients' health. CPG developers should also consider downstream consequences (e.g., burden of the test and the proportion of patients with a certain test result who receive the recommended treatment) on patient‐relevant outcomes (e.g., mortality, morbidity and quality of life) (see Figure 1). ,

Figure 1

Test‐treatment pathway (adapted from Harris et al, 2001)

The interpretation of evidence about the value of therapeutic interventions is complex, and there is room for improvement. This applies even more to evidence about diagnostic tests and its translation into CPG recommendations. , , There have been few randomized controlled trials on the value of test‐treatment pathways for patient‐relevant outcomes. Evaluating the value of diagnostic tests on patient‐relevant outcomes in CPGs is thus complex since it requires integration of various pieces of evidence for the different links in a chain (see Figure 1).

In the GRADE approach for diagnostic tests and test strategies, the first step is to formulate the clinical question, including definition of patient‐important outcomes and description of the aim of the test (add‐on, replacement or triage). The next step is to assess diagnostic accuracy and downstream consequences of testing. These include the burden of the test, clinical management, natural course of the target condition (to estimate the outcomes of patients with a false negative test result), and the link between test result and management (proportion of patients with a certain test result who receive the recommended treatment). Ideally, each evidence component is based on a systematic review of the literature and the certainty in the evidence for each component is determined separately. Finally, the evidence components are integrated and the overall certainty in the evidence is assessed. , To move from evidence to recommendation, guideline developers use the GRADE evidence‐to‐decision framework.

The aim of this study is to assess the extent to which evidence‐based CPGs about diagnostic tests cover all relevant test‐treatment pathway components.

Specific objectives are to assess the types of supporting evidence used for CPG recommendations about diagnostic tests, and to explore determinants of best practices. In the context of CPG development about the value of a diagnostic test, we formulated the following research questions:Answers to these questions elucidate gaps in the implementation of good CPG development methods when developing recommendations about diagnostic tests and test strategies and can help guideline methodologists in developing strategies to facilitate this process.

Which types of evidence (diagnostic accuracy, burden of the test, natural course, treatment effectiveness, link between test result and administration of treatment) are used to support the recommendations?

Which factors (e.g., composition of the guideline panel, use of the GRADE approach, methodological quality according to AGREE II's domain methodology) contribute to completeness of the evidence?

To what extent can differences between CPG recommendations be explained by including different types of evidence?

METHODS

Design

In order to assess the types of supporting evidence used for CPG recommendations about diagnostic tests, and to identify factors related to the extent of the supporting evidence, we performed a systematic document analysis of recent versions of publicly accessible CPGs concerning three diagnostic topics.

Topics

We chose tests that are frequently used to diagnose three common diseases. We considered tests with different characteristics (primary vs. secondary care, negligible vs. reasonable risk of serious burden of the test, low vs. high costs) to identify possible factors related to differences in methodological approach in the development of the CPGs. We therefore selected the following tests:

C‐reactive protein (CRP) test to increase the likelihood of pneumonia (annual incidence estimated at 0.5%–1.1%) in primary care patients with cough (excluding diagnostic procedures in patients suspected of having a COVID‐19 infection),

Colonoscopy to detect colon cancer (annual incidence 1 148 515 new cases) in secondary care patients suspected of having (primary) colon cancer (excluding screening and tests in patients at risk of hereditary types of colon cancer),

Fractional exhaled nitric oxide (FeNO) to diagnose (severe) asthma (prevalence 3.6%) in children and adults in primary and secondary care (excluding monitoring of asthma).

Search and selection of relevant CPGs

Current, publicly accessible, recent (publication date 2016–2020) CPGs were eligible if they included recommendations about the tests mentioned above, were CPGs at a national or international level, and were published in English, German or Dutch.

To identify relevant CPGs, one author (Mariska K Tuut) performed the search and selected the CPGs. The selection was checked for accuracy by a second author (Jako S Burgers). In February 2021, we searched the international guideline library from guidelines international network (GIN, [https://guidelines.ebmportal.com/], including around 3000 CPGs, mostly developed by organizational GIN members), databases from organizational GIN members active in CPG development (n = 103), the TRIP database (Turning Research Into Practice [https://www.tripdatabase.com/], containing around 10,000 English‐language CPGs) and Medline (see Appendix 1 for full search details).

Identification of recommendations

We analysed the content of the selected CPGs to identify relevant recommendations, including supporting evidence available online (e.g., tables with study characteristics, evidence documents, GRADE Evidence Profiles), as well as information about the methods of CPG development of the developing organization (e.g., methodology manuals).

Data extraction

In the preparatory phase of this study, we piloted data extraction on two recommendations with four authors (Mariska K Tuut, Miranda W Langendam, Jako S Burgers and Trudy van der Weijden) to refine the data extraction form and define the variables for which we needed data extraction in duplicate. One author (Mariska K Tuu) extracted the initial characteristics of each recommendation and CPG (CPG title (including English translation if relevant), initial developing organization, country, publication year, recommendation text (including English translation if relevant)).

Detailed information about each recommendation and CPG was extracted by one author (Mariska K Tuu) and critically reviewed by another author (Miranda W Langendam, Jako S Burgers or Trudy van der Weijden) using a predefined and piloted data extraction form (see Appendix 2 for the data extraction form and the categorisation of the variables). The form consisted of questions about scope and target audience of the CPG and composition of the CPG panel, involvement of methodologist(s), methodological quality of the CPG (using AGREE II, domain methodology, items 7–12), , patient involvement (using AGREE II item 5), , the types and extent of supporting evidence for the recommendation (consideration and inclusion of systematic evaluation with assessment of the certainty in the evidence about diagnostic accuracy, burden of the test, natural course, treatment effectiveness and link between test result and administration of treatment), grading of the recommendation and use of the GRADE approach, direction of the recommendation, and characteristics of the test and target condition. Disagreements between the reviewers were discussed until consensus was reached.

Analysis

We tabulated basic and detailed characteristics of the included recommendations and CPGs. Systematic evaluation (with a systematic review of the literature and assessment of the certainty in the evidence) of the impact of the full test‐treatment pathway (diagnostic accuracy, burden of the test, natural course of target condition, treatment effectiveness, and link between test result and administration of treatment) on patient relevant outcomes was considered best practice for developing medical test recommendations.

We planned to analyse a possible relation between differences in evidence base and methodological factors (e.g., composition of the CPG panel, involvement of patients and methodologists, development approach). However, because the data about the evidence base were quite homogenous we were not able to perform these analyses.

RESULTS

Full details of the search and selection process are described in Appendix 1. In short, the search identified 15 unique relevant recommendations in 15 CPGs: four about CRP related to the diagnosis pneumonia in primary care, , , , five about colonoscopy in secondary care patients suspected of having colon cancer, , , , , and six about the use of FeNO to diagnose (severe) asthma. , , , , The search and selection process is illustrated in Figure 2.

Figure 2

Search and selection of relevant CPGs

In Table 1, we present the included CPGs with information about the developing organization, the country of publication and the publication year. All guidelines originated from high‐income countries.

Table 1

Basic characteristics of the included CPGs

Organization	Year	Country	Title (original language)	English‐translated title in case of non‐English language CPG
CRP
Deutschen Gesellschaft für Pneumologie und Beatmungsmedizin (DGPB) 20	2016	Germany	Behandlung von erwachsenen Patienten mit ambulant erworbener Pneumonie und Prävention	Prevention and management of adult patients with community acquired pneumonia
American College of Chest Physicians (ACP) 21	2019	United States	Adult Outpatients with acute cough due to suspected pneumonia or influenza
Ministry of Public Health, Qatar (MoPH) 19	2019	Qatar	The diagnosis and management of community acquired pneumonia
Deutschen Gesellschaft für Pädiatrische Infektiologie (DGPI) 22	2017	Germany	Management der ambulant erworbenen Pneumonie bei Kindern und Jugendlichen (pädiatrische ambulant erworbene Pneumonie, pCAP)	Management of community acquired pneumonia in children and adolescents
Colonoscopy
European Society for Medical Oncology (ESMO) 25	2020	Europe	Localized colon cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow‐up
Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e.V. (AWMF) 23	2019	Germany	Kolorektales Karzinom	Colorectal cancer
Association of Coloproctology of Great Britain and Ireland (ACPGBI) 26	2017	Great Britain and Ireland	Guidelines for the Management of Cancer of the Colon, Rectum and Anus (2017) ‐ Diagnosis, Investigations and Screening
Federatie Medisch Specialisten (FMS) 24	2019	The Netherlands	Colorectaal carcinoom	Colorectal cancer
Nederlands Huisartsen Genootschap (NHG) 27	2017	The Netherlands	Rectaal bloedverlies	Rectal bleeding
FeNO
Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e.V. (AWMF) 28	2020	Germany	Nationale VersorgungsLeitlinie Asthma	National Guideline on asthma
Ministry of Public Health (MoPH_A) 31	2019	Qatar	The diagnosis and management of asthma in adults
Ministry of Public Health (MoPH_C) 32	2019	Qatar	The diagnosis and management of asthma in children
National Asthma Education and Prevention Program (NAEPP) 29	2020	USA	Managing Asthma in Adolescents and Adults
National Institute for Health and Care Excellence (NICE) 30	2020	UK	Asthma: diagnosis, monitoring and chronic asthma management
Scottish Intercollegiate Guidelines Network (SIGN) 33	2019	UK	British guideline on the management of asthma

(ACPGBI)

Quality of the guidelines and use of the GRADE approach

Table 2 presents detailed information about the composition of the CPG panel, the methodological quality of the included CPGs, the direction and grading of the recommendation and the reported and actual use of the GRADE approach. Nine out of 15 CPGs included a methodologist in the development process, in the CPG panel and/or at bureau level. , , , , , , , In all CPGs about FeNO a methodologist was involved, and in none of the CPGs about CRP. Patient involvement and inclusion of patient perspective varied a lot between the CPGs. AGREE II methodology domain scores varied from 8 to 42 (possible range from worst to best: 6–42), with the highest scores for the CPGs about FeNO. Thirteen of the included recommendations were in favour of the test of interest, only one recommendation about CRP, and one recommendation about FeNO, advised against the use of the test. Eleven recommendations were graded, which included all recommendations about CRP, , , , two out of five recommendations about colonoscopy, , and five out of six recommendations about FeNO. , , , , Seven CPGs reported to have used the GRADE approach , , , , , ; in four of these elements of the GRADE approach (such as a GRADE evidence profile) were recognized. , , , No clear differences between the topics were identified in the (reported) use of the GRADE approach.

Table 2

Detailed characteristics of the CPG quality, the recommendation and the (reported) use of GRADE

Topic	Guideline	CPG panel		Methodology (AGREE II scores)							Recommendation		GRADE
		Methodologist involvement	Patient involvement (AGREE II score)	Systematic evidence search methods	Clear criteria for evidence selection	Clear description of the strengths and limitations of the body of evidence	Clear description methods for formulating recommendations	Health benefits, side effects and risks have been considered	Explicit link between recommendation and supporting evidence	SUM score methodology domain†	Direction	Graded	Reported use of GRADE approach	Elements of GRADE approach in CPG
CRP	DGPB, 2016 20	−	2	4	2	3	6	3	5	23	+	+	+	−
	ACP, 2019 21	−	5	6	5	2	6	2	6	27	+	+	+	−
	MoPH, 2019 19	−	1	2	1	2	1	1	1	8	+	+	−	−
	DGPI, 2017 22	−	2	3	1	1	7	5	2	19	‐	+	−	−
Colonoscopy	ESMO, 2020 25	−	1	1	1	1	3	1	2	9	+	+	−	−
	AWMF, 2019 23	+	3	7	7	5	7	3	6	35	+	−	−	−
	ACPGBI, 2017 26	−	1	2	1	3	2	1	4	13	+	+	−	−
	FMS, 2019 24	+	5	2	1	2	4	6	7	22	+	−	+	+
	NHG, 2017 27	+	1	5	1	1	5	6	7	25	+	−	−	−
FeNO	AWMF, 2020 28	+	7	7	7	7	7	6	7	41	−	+	+	+
	MoPH_A, 2019 31	+	3	4	4	2	2	2	2	16	+	−	−	−
	MoPH_C, 2019 31	+	3	3	3	2	2	2	2	14	+	+	−	−
	NAEPP, 2020 29	+	7	7	7	7	7	7	7	42	+	+	+	+
	NICE, 202030‡	+	7	7	7	7	6	7	7	41	+	+	+	+
	SIGN, 2019 32	+	7	7	3	3	4	5	7	29	+	+	+	−

Note: +, yes; +/−, unclear; −, no; †possible range: 6–42; ‡This CPG contains two separate recommendations concerning the use of FeNO in the diagnosis of childhood respectively adult onset asthma; the scores are identical.

Supporting evidence for the recommendations

Detailed information about the supporting evidence for the included recommendations is presented in Table 3. Ten CPGs out of 15 considered diagnostic accuracy, , , , , , , , , , of which four underpinned these considerations with a systematic review of the literature and a judgement of the certainty in the evidence. , , , Burden of the test was considered in three CPGs, , , and two CPGs considered the natural course of the disease, , all without systematically reviewing the literature. Three CPGs considered treatment effectiveness, , , of which one performed a systematic review of the literature with judgement of the certainty in the evidence. Not any CPG considered the link between the test result and administration of treatment. As a consequence, there were no CPGs that considered all test consequences of the test‐treatment pathway.

Table 3

Detailed information about the supporting evidence for the recommendations

Topic	Guideline	Diagnostic accuracy			Burden of the test			Natural course			Treatment effectiveness			Link between test result and administration of treatment
		Considered	Considered with systematic review of the literature	Considered with systematic review of the literature and certainty in the evidence	Considered	Considered with systematic review of the literature	Considered with systematic review of the literature and certainty in the evidence	Considered	Considered with systematic review of the literature	Considered with systematic review of the literature and certainty in the evidence	Considered	Considered with systematic review of the literature	Considered with systematic review of the literature and certainty in the evidence	Considered
CRP	DGPB, 2016 20	+	−	−	−	−	−	−	−	−	−	−	−	−
	ACP, 2019 21	+	+	+	−	−	−	−	−	−	−	−	−	−
	MoPH, 2019 19	−	−	−	−	−	−	+	−	−	+	−	−	−
	DGPI, 2017 22	+	−	−	−	−	−	−	−	−	−	−	−	−
Colonoscopy	ESMO, 2020 25	−	−	−	−	−	−	−	−	−	+	−	−	−
	AWMF, 2019 23	−	−	−	−	−	−	−	−	−	−	−	−	−
	ACPGBI, 2017 26	+	−	−	−	−	−	−	−	−	−	−	−	−
	FMS, 2019 24	+	−	−	+	−	−	−	−	−	−	−	−	−
	NHG, 2017 27	+	−	−	+	−	−	−	−	−	−	−	−	−
FeNO	AWMF, 2020 28	+	+	+	−	−	−	−	−	−	+	+	+	−
	MoPH_A, 2019 31	−	−	−		−	−	−	−	−	−	−	−	−
	MoPH_C, 2019 31	−	−	−	−	−	−	−	−	−	−	−	−	−
	NAEPP, 2020 29	+	+	+	+	−	−	−	−	−	−	−	−	−
	NICE, 202030†	+	+	+	−	−	−	−	−	−	−	−	−	−
	SIGN, 2019 32	+	−	−	−	−	−	+	−	−	−	−	−	−

Note: +, yes; +/−, unclear; −, no; †This CPG contains two separate recommendations concerning the use of FeNO in the diagnosis of childhood respectively adult onset asthma; the scores are identical.

Since no CPG systematically evaluated all steps of the test‐treatment pathway, we were not able to identify a best practice, nor could we study possible relationships between clarifying factors and supporting evidence for a recommendation.

DISCUSSION

Our document analysis on a sample of 15 CPGs about CRP, colonoscopy and FeNO diagnostic tests revealed that none of these CPGs reported evidence on all components of the test‐treatment pathway. Consideration of any test consequences on patient‐relevant outcomes was described in only six CPGs (three CPGs considered burden of the test, two considered natural course of the disease of interest, and four considered treatment effectiveness). Systematic review of the literature, including a judgement of the certainty in the supporting evidence was only reported for four recommendations and covered diagnostic accuracy in all four cases and treatment effectiveness in one case.

The importance of systematically evaluating test consequences for the purpose of developing CPGs has been recognized. , , For instance, one could imagine that a certain diagnostic test might have limited value when it has no treatment consequences (e.g., no treatment available). Or when comparing two tests with the same diagnostic accuracy to ascertain the same disease, one could recognize that differences in test burden may play an important role.

This study suggests that implementation of the systematic evaluation of the value of a test is lagging behind. This also applies to CPGs that claim to use the GRADE approach. There seems to be a gap between following a methodologically robust approach and developing CPGs in practice.

Two issues may explain that gap. First, guideline developers may have considered the downstream consequences of a diagnostic test but did not explicitly report these. It may not be strictly necessary to systematically evaluate all evidence components. However, we still recommend transparent documentation of choices made in the guideline development process. A guideline user should be able to read which elements of a test‐treatment pathway were considered and how, and which were not considered and why.

Second, performing systematic literature reviews of the complete test‐treatment pathway–including assessment of the certainty in the evidence of test accuracy and downstream consequences–is complex and time‐consuming. The use of the GRADE approach for the evaluation of diagnostic tests and test strategies is considered challenging. , Strategies to facilitate the use of this approach, such as training of CPG panel members, may improve the application. Unfortunately, we could not determine factors that contribute to successful use of the GRADE approach, because we could not identify a “best practice.”

A lack of transparency in combination with the use of state‐of‐the‐art methods was also described by Arevalo‐Rodriguez and colleagues, who studied the methods and reports of 191 rapid reviews of medical tests. In the majority of those reviews, the study selection method was not reported. Although almost 20% of the reviews claimed to have applied the GRADE approach, few actually reported the data extraction and quality appraisal methods.

This finding is consistent with a recent report on the application of GRADE in US guidelines. Although guideline developers indicated that they used the GRADE approach, only 10% of the included CPGs reported on all eight criteria for assessing the certainty in the evidence (e.g., indirectness and dose‐response gradient), and around half of these included an evidence profile or summary of findings table.

Gopalakrishna et al. studied barriers in the development of recommendations about medical tests in a qualitative study among European CPG developers. They also reported challenges in the development of recommendations about medical tests, for example, in the definition of key questions, the types of evidence and outcomes included in the CPG, and synthesizing and appraising the evidence. Awareness and education were reported as the most important ways to solve these challenges.

Our study emphasizes the need for more knowledge and expertise among CPG developers when evaluating diagnostic tests. Currently available competency‐based frameworks for CPG developers do not include a special focus on diagnostic test evaluation. , This also applies for current training programs of CPG panel members, for example, INGUIDE. Facilitating the implementation of GRADE for diagnosis by defining competencies and training needs may improve the quality of CPGs about diagnostic tests.

Strengths and limitations

This study evaluated the supporting evidence of recommendations in CPGs on three medical tests. The selection of only three topics is a limitation of this study. However, we chose three diagnostic tests with divergent characteristics (e.g., invasiveness, possible burden of the test, disease of interest, costs) allowing comparison of many CPGs. The homogenous results in all three clusters of CPGs strengthens the external validity of our findings. Additionally, we found large variance in methodological quality of the included CPGs. However, high‐scoring CPGs on the AGREE II domain methodology did not reflect a better or more transparent underpinning of the recommendations than lower scoring CPGs.

Due to the document analysis design, we could not retrieve information about the dynamics in the CPG panels that could explain their decisions and reasons for lack of transparency in the CPG documents. We did not contact the CPG developers, since in our opinion CPG users should be able to find the considerations of the panel beyond the recommendations in the published documents of the CPG.

Implications for practice

We suggest that developers of CPGs about diagnostic tests clearly describe which elements of a test‐treatment pathway were or were not considered and why. In addition, CPG developers should indicate the presence or absence of systematic reviews of the evidence, including determination of the certainty in that evidence, for all evaluated parts of the test‐treatment pathway, which is also usual in recommendations about therapy. Facilitating the implementation of GRADE for diagnosis will be useful to improve the clinical content of CPGs.

Implications for research

This study highlighted the lack of (transparency about) supporting evidence for diagnostic test recommendations in CPGs. A next step could be to study why CPG developers do not report all elements of the test‐treatment pathway, including a review of the evidence and its quality. Furthermore, it is worthwhile to research how to facilitate CPG developers in explicitly and reliably considering all relevant steps of a test‐treatment pathway when developing diagnostic test recommendations.

CONCLUSION

Diagnostic test recommendations in the included CPGs are mainly based on evidence and considerations on diagnostic accuracy. Other steps of the test‐treatment strategy, such as burden of the test, natural course of the disease of interest, effectiveness of treatment of the disease of interest and the link between the test result and the administration of treatment should receive more attention in CPGs in order to consider evidence about test consequences on patient‐relevant outcomes.

CONFLICT OF INTEREST

Mariska Tuut and Miranda Langendam are members of the GRADE Working Group. This research did not receive any specific grant from funding agencies in the public, commercial, or not‐for‐profit sectors.

AUTHORS' CONTRIBUTIONS

Mariska K Tuut conceptualized the study, including methodology, extracted, analysed, and interpreted the data, and wrote the original draft of the manuscript. Jako S Burgers, Trudy van der Weijden and Miranda W Langendam were involved in the conceptualization of the study, including methodology, performed data checks, reviewed and edited the manuscript and had a supervising role. All authors read and approved the final manuscript.

Appendix 1: Description of the search process to reveal the CPGs and recommendations of interest

Click here for additional data file.

Appendix 2: Data extraction form

Click here for additional data file.