Characteristics and conflicting recommendations of clinical practice guidelines for COVID-19 management in children: A scoping review.

BACKGROUND: Clinical practice guidelines (CPGs) are statements that should be rigorously developed to guide clinicians' decision-making. However, given the scarce evidence for certain vulnerable groups like children, CPGs' recommendations formulation could be challenging.
METHODS: We conducted a scoping review of CPGs for COVID-19 management in children. Documents were included if they claimed to be a "clinical practice guideline", published between January and October 2021, and described the process followed to issue their recommendations. We assessed the quality using the "Appraisal of Guidelines for Research and Evaluation II" (AGREE-II) and described how the recommendations were reached.
RESULTS: We found five CPGs that fulfilled our inclusion criteria. The median score on the overall AGREE-II evaluation was 61% (range: 49%-72%), and the score on the third domain referred to the rigor of methodological development was 52% (range: 25%-88%). Recommendations for remdesivir, tocilizumab, and intravenous immunoglobulin were heterogeneous across CPGs (in favor, against, no recommendation), as well as the methodologies used to present the evidence, perform the benefits/harms balance, and issue the recommendation.
CONCLUSIONS: Heterogeneous recommendations and justifications across CPGs were found in the three assessed topics. Future CPGs should describe in detail their evidence-to-decision process to issue reliable and transparent recommendations.

American College of Rheumatology

Spanish Pediatric Association (Asociación Española de Pediatría)

Appraisal of Guidelines for Research and Evaluation II

Coronavirus disease 2019

Clinical Practice Guideline

Evidence Profile

Evidence-to-Decision

Grading of Recommendations, Assessment, Development, and Evaluations

Intravenous Immunoglobulin

Multisystem Inflammatory Syndrome in Children

Australian National Health and Medical Research Council

National Institutes of Health

Pediatric Infectious Diseases Group of Switzerland

Pediatric inflammatory multisystem syndrome temporally associated with SARS-CoV-2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Severe Acute Respiratory Syndrome Coronavirus 2

Summary of Findings

Swiss Society of Intensive Care Medicine

United States of America

Introduction

The evidence-based clinical practice guidelines (CPGs) are “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [1]. These documents should be rigorously developed to give adequate guidance, and should be transparently reported to maximize their trustworthiness [2]. Flawed guidelines could eventually harm patients, confuse and frustrate practitioners, and compromise the health systems' resources [3].

As of April 22, 2022, about 505 million of cases of the coronavirus disease 2019 (COVID-19) had been confirmed worldwide, causing more than 6 million deaths [4]. Several treatments have been studied with great speed, from the hype of antibiotics like azithromycin, antimalarials like chloroquine/hydroxychloroquine, and antiparasitics like ivermectin to the well-stablished role of corticosteroids for severe cases and anticoagulants for hospitalized patients [5]. Antivirals like the orally administered molnupiravir and antibody‐based treatments such as convalescent plasma and the combination of monoclonal antibodies (tocilizumab, anakinra, etc.) with intravenous immunoglobulin (IVIG) have also received wide attention by the scientific community [6]. Despite this, there are still important gaps in the management of certain vulnerable groups, such as children [7]. Although most pediatric cases were asymptomatic and mild, with a better prognosis than adults [8,9], severe disease does occur, as well as important complications with a major mortality risk such as the multisystem inflammatory syndrome (MIS-C) [[10], [11], [12]].

CPGs that assess the management of children with COVID-19 face a challenging scenario since randomized controlled trials have been almost exclusively performed in adults, and it is unclear how to extrapolate their result to the pediatric population (with a lower mortality rate and a different disease presentation) [13]. A systematic review that assessed the pediatric CPGs for COVID-19 until August 2020 [14] found that recommendations varied widely across CPGs, but did not delve into the evidence-to-decision process, which is needed to understand how CPG developing groups are managing to reach recommendations given the lack of evidence.

Thus, this study aimed to describe the characteristics of the current CPGs for COVID-19 management in children and assess their evidence-to-decision process.

Methods

This is a scoping review of CPGs for COVID-19 management in children. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for scoping reviews (PRISMA-ScR) to secure adequate reporting of the study [15].

Eligibility criteria

We included all documents that fulfilled the following inclusion criteria:

The term “clinical practice guideline” or its variants (guideline, orientation, guide, among others) were mentioned in the title or abstract

Were published or totally/partially updated from January 2021 to October 2021

Issued recommendations for managing pediatric patients with suspected or confirmed SARS-CoV-2 infection.

Described (at least briefly) the process carried out to issue their recommendations.

Lastly, previous versions of updated guidelines were excluded.

Search strategy and study selection

We performed a comprehensive search in five sources (Trip Database, PubMed, The international database of GRADE guidelines, Google Scholar, and Google). Our search strategy included terms related to COVID-19 and guidelines/practice guidelines ( Supplementary Table 1 ).

Searches were performed on November 1st, 2021. Two independent researchers (AQL and LCR) evaluated if the retrieved documents met the eligibility criteria for inclusion. A consensus was reached after consulting with another researcher (ATR) when there were discrepancies.

Data extraction

Two researchers (AQL and LCR) independently extracted the variables of interest of the included CPGs in a Microsoft Excel sheet. We included general characteristics, methodological characteristics, and the recommendations/justification issued for the following topics: remdesivir use in children with COVID-19, tocilizumab use in children with COVID-19, and IVIG/corticosteroids in the cases of MIS-C. When there were discrepancies in data extraction, a consensus was reached after debating them among all the researchers.

To standardize the terminology used in the guidelines, any denomination of Pediatric inflammatory multisystem syndrome temporally associated with SARS-CoV-2 (PIMS-TS) was replaced by MIS-C in the present study.

Quality appraisal

To assess the quality of CPGs, we used the Appraisal of Guidelines Research and Evaluation II (AGREE-II), which has 23 items distributed in six domains (scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence). Each guideline was rated by two researchers (JMA and GAG). When a difference in two or more points in each item was found, the item was discussed to get to a consensus. Otherwise, we used the mean of the two scores for each item. Lastly, we followed the AGREE-II Instrument guideline to calculate the scores for each domain [16].

We considered that when a CPG had a total score ≥70% it had an acceptable quality. This cut-off point was taken from previous studies that evaluated the quality CPGs with this instrument [17].

Results

Selection

We assessed 1568 records, from which five CPGs met the inclusion criteria (Fig. 1 ). These CPGs were developed by the American College of Rheumatology (ACR) [18], Spanish Pediatric Association (AEP) [19], National Institutes of Health (NIH) [20], Swiss Society of Intensive Care Medicine and the Pediatric Infectious Diseases Group of Switzerland (SSICM and PIGS) [21], and the Australian National Health and Medical Research Council (NHMRC) [22].

Fig. 1

Flowchart of clinical practice guideline selection.

CPGs characteristics

The five CPGs were published in institutions from five countries: the United States of America (USA), the USA and Canada, Spain, Switzerland, and Australia. CPGs were published from March to October 2021. 2/5 CPGs assessed specifically MIS-C management. Regarding the method to reach recommendations and grade their strength, 2/5 CPGs claimed to have used the GRADE methodology for both steps, whereas the rest claimed to have based their recommendations on an expert consensus (2/5) or their own criteria (1/5). Also, 4/5 CPGs claimed to have conducted systematic reviews of the evidence ( Table 1 ).

Table 1

Characteristics of clinical practice guidelines that assessed COVID-19 management in children (2021).

Author	Country	Topics addressed	Date of publication or update	Methodology used to reach recommendations	Methodology used for grading the strength of recommendations	Claimed to have performed systematic reviews of the evidence	Showed the benefits and harms of the interventions	Used structured frameworks for decision-making
ACR	USA and Canada	MIS-C (diagnosis and management)	Apr-2021	Expert consensus was built through a modified Delphi process (anonymous voting and webinar discussion)	ACR (Expert consensus)	Yes	Yes, only narratively	No
AEP	Spain	Epidemiology, diagnosis, treatment, and prevention of COVID-19	Mar-2021	GRADE	GRADE	Yes	Yes, either using SoF tables or narratively, depending on the question	No
NIH	USA	Management of COVID-19	Apr-2021	NIH methodology (blind vote)	NIH criteria	Yes	Yes, only narratively	No
SSICM and PIGS	Switzerland	MIS-C (diagnosis and management)	May-2021	Expert consensus was built through a modified Delphi process	It appears to be an expert consensus	No (performed a “focused literature review”)	No	No
NHMRC	Australia	Prevention, diagnosis, and management of COVID-19	Oct-2021	GRADE	GRADE	Yes	Yes, either using SoF tables or narratively, depending on the question	No

ACR: American College of Rheumatology.

AEP: Spanish Pediatric Association.

NIH: National Institutes of Health.

SSICM and PIGS: Swiss Society of Intensive Care Medicine and the Pediatric Infectious Diseases Group of Switzerland.

NHMRC: Australian National Health and Medical Research Council.

CPGs quality

The scores on the overall AGREE-II evaluation ranged from 49% to 72%, and the scores on the third domain of AGREE-II about the rigor of methodological development ranged between 25% and 88%. Only 1/5 CPG obtained scores ≥70% for both the third domain and the overall scores ( Table 2 ).

Table 2

Quality of the included clinical practice guidelines, using the AGREE-II tool (domain 3: rigor of development).

Items	ACR	AEP	NIH	SSICM and PIGS	NHMRC
1. Systematic methods were used to search for evidence.	33%	100%	17%	0%	92%
2. The criteria for selecting the evidence are clearly described.	50%	100%	42%	0%	83%
3. The strengths and limitations of the body of evidence are clearly described.	0%	100%	33%	0%	33%
4 The methods for formulating the recommendations are clearly described.	100%	100%	50%	92%	100%
5. The health benefits, side effects, and risks have been considered in formulating the recommendations.	50%	100%	50%	0%	67%
6. There is an explicit link between the recommendations and the supporting evidence.	83%	100%	67%	58%	100%
7. The guideline has been externally reviewed by experts prior to its publication.	0%	0%	0%	0%	100%
8. A procedure for updating the guideline is provided.	100%	100%	92%	50%	100%
Overall	52%	88%	44%	25%	84%

How CPGs reached their recommendations

We assessed the recommendations issued by the CPGs for three topics: remdesivir, tocilizumab, and IVIG for MIS-C. Recommendations were heterogeneous across CPGs, as well as the methodologies used to present the evidence, perform the benefits/harms balance, and issue the recommendation ( Table 3 ).

Table 3

Selected recommendations and their justification.

CPG	How was evidence shown	Evidence shown	How was evidence interpreted to reach the recommendation	Recommendation
Remdesivir for children with COVID-19 (no MIS-C)
NIH	Narratively	The CPG panel reviewed 5 RCTs performed in adults, but the evidence description focuses on the ACTT-1 trial.	Authors mention that the ACTT-1 is a double-blinded trial, which found that remdesivir was associated with improved time to recovery in participants who required minimal supplemental oxygen.	In favor:For hospitalized children aged ≥16 years with COVID-19 who have an emergent or increasing need for supplemental oxygen, or those ≥12 years who also have risk factors for severe disease.
AEP	SoF table	Meta-analyses of 3 RCTs performed in adults.	The authors claimed that the evidence suggested that remdesivir may reduce mortality but not the risk of mechanical ventilation. However, the evidence was uncertain, there was a lack of cost-effectiveness evaluations and a high probability that clinicians, and informed patients would not accept this medication. Thus, the authors decided not to state a recommendation.	No recommendation.
NHMCR	SoF table	Meta-analyses of 8 RCTs performed in adults.	Although the results suggested that remdesivir may have benefits in all-cause mortality and risk of invasive mechanical ventilation, authors point out that it is unclear how these possible benefits extrapolate to the pediatric population (with lower case-fatality rate and different disease presentation).	Against:Use of remdesivir for children or adolescents with COVID-19 outside a trial setting should not be routinely considered.
Tocilizumab for children with COVID-19 (no MIS-C)
NIH	Narratively	3 RCTs performed in adults	The authors discuss that the RCTs results were contradictory.	No recommendation.
AEP	SoF table	Meta-analyses of 3 RCTs performed in adults	The SoF table did not report significant benefits for mortality and worsening (death, mechanical or non-invasive ventilation).	Against:Do not use in pediatric patients with COVID-19.
NHMCR	SoF table	Meta-analyses of 8 RCTs performed in adults.	In hospitalized adults who require supplemental oxygen, tocilizumab may decrease the need for invasive mechanical ventilation and death. Although there is uncertainty regarding the benefits and harms of tocilizumab use in children, its previous experience on other diseases in children led to a positive recommendation.	In favor:Consider using tocilizumab to treat COVID-19 in children and adolescents who require supplemental oxygen, particularly where there is evidence of systemic inflammation.
Intravenous immunoglobulin (IVIG) for children with MIS-C
NIH	Narratively	1 cohort compared patients using IVIG with and without corticosteroids.a	It does not issue a recommendation due to a lack of evidence.	No recommendation.
AEP	Narratively	1 cohort compared patients using IVIG with and without corticosteroids.a	Authors refer that indirect evidence from other acute inflammatory diseases (such as Kawasaki) suggests a possible benefit.	In favor:Consider using IVIG + methylprednisolone in children with MIS-C.
NHMCR	SoF table	3 cohorts compared patients using IVIG with and without corticosteroids.a	Authors refer that indirect evidence from other acute inflammatory diseases (such as Kawasaki) suggests a possible benefit.	In favor:Consider using IVIG + methylprednisolone in children and adolescents with MIS-C.
ACR (only MIS-C)	Narratively	7 cohorts compared patients using IVIG with and without corticosteroids.a	The authors refer that indirect evidence from other acute inflammatory diseases (such as Kawasaki and fulminant myocarditis) suggests a possible benefit.	In favor:IVIG should be given to MIS-C patients hospitalized or fulfill Kawasaki Disease criteria.
SSICM and PIGS (only MIS-C)	Narratively	2 cohorts compared patients using IVIG with and without corticosteroids.a	The authors refer that indirect evidence from other acute inflammatory diseases (such as Kawasaki) suggests a possible benefit.	In favor:Recommend IVIG use in patients with MIS-C shock. Also, consider its use in non-shocked patients with MIS-C undefined inflammatory presentation.

MIS-C: Multisystem inflammatory syndrome in children.

RCT: randomized controlled trial.

Since these studies compared patients using IVIG with and without corticosteroids, they did not assess the effect of IVIG but of corticosteroids.

Three CPGs (NIH, AEP, and NHMCR) assessed the use of remdesivir for children with COVID-19 (no MIS-C). All these CPGs found randomized controlled trials (RCTs) performed in adults. NIH laid out the evidence narratively, focusing on one RCT, and issued a recommendation favoring remdesivir use in particular situations. AEP summarized 3 RCTs in a summary of findings (SoF) table but claimed that the evidence was uncertain and thus did not issue any recommendation. NHMCR summarized 8 RCTs in the SoF table, finding a possible benefit for adults, but given that it is unclear how these benefits extrapolate to the pediatric population, it issued a recommendation against its use.

Three CPGs (NIH, AEP, and NHMCR) assessed the use of tocilizumab for children with COVID-19 (no MIS-C). All these CPGs found RCTs performed in adults. NIH laid out the evidence (3 RCTs) narratively, finding contradictory results and therefore deciding not to issue any recommendation. AEP summarized 3 RCTs in a SoF table, concluding that no significant benefits were found, and therefore recommended against tocilizumab use. NHMCR summarized 8 RCTs in the SoF table, finding a possible benefit, and even though the studies were performed in adults (indirect evidence), issued a recommendation in favor of tocilizumab use.

All five CPGs assessed the use of IVIG for children with MIS-C. NIH did not find any direct evidence and decided not to issue a recommendation. AEP also did not find any direct evidence but, given the indirect evidence from other acute inflammatory diseases, issued a recommendation in favor of IVIG use. NHMCR, ACR, and SSICM/PIGS cited studies (narratively or using SoF tables) that compared patients using IVIG with and without corticosteroids (thus, these studies assessed the effect of corticosteroids). However, given the indirect evidence from other acute inflammatory diseases, it issued a recommendation in favor of IVIG use.

Discussion

Characteristics and quality

Our study found five CPGs that assessed the management of children with COVID-19, which justified their recommendations and were published between January and October 2021. Those CPGs had a median AGREE-II score of 61% (range: 49%–72%). A previous review that included 20 CPGs for managing children with COVID-19 published until August 2020 [14] also found a poor quality of the GPGs: an overall AGREE-II score of 37% (range: 22%–62%).

A decision regarding the use of certain intervention should value the current evidence regarding its benefits and harms. Many methodologies have been used to do so, although in recent years, the GRADE methodology has managed to position itself as a reference in the area. The GRADE methodology, which intends to transparently assess the certainty of the evidence and guide evidence-based decision-making [23], has been adopted by over 110 organizations worldwide [24], and is being used in several COVID-19 CPGs, such as the ones of the World Health Organization [25] and the Infectious Diseases Society of America [26].

The GRADE methodology states that, to assess the effects of certain intervention, the evaluation of the evidence should not be focused on the results of each study, but on the results of the meta-analyses for each outcome [27]. For this purpose, recommends the use of SoF or Evidence Profile (EP) tables that summarize the effect of the intervention for each outcome and how certain the calculated estimates are. This allows performing a benefits/harms balance to inform an evidence-based recommendation [27].

For remdesivir and tocilizumab recommendations, 2/3 CPGs (AEP and NHMCR) performed meta-analyses and displayed SoF tables, while NIH did not. In fact, NIH narratively focused on one of the 5 RCTs for remdesivir (to issue a recommendation in favor), and narratively pointed that the 3 RCTs assessed for tocilizumab had contradictory results (to decide not to issue a recommendation). This presentation of the results (narratively per each study, without performing meta-analyses and SoF tables) could make it difficult for CPG panel members and the readers to make a clear benefits/harms balance [28].

Although AEP and NHMCR CPGs displayed SoF tables for their remdesivir and tocilizumab recommendations, it is not clear how these guidelines faced the problem of indirectness (having indirect evidence from RCTs performed in adults to issue a recommendation in children). For example, although NHMCR CPG found evidence that suggested potential benefits of remdesivir and tocilizumab in adults, it issued recommendations against remdesivir (due to the indirectness) and in favor of tocilizumab (based on the fact that it has previously been used in children with cytokine release syndrome and systemic and polyarticular juvenile idiopathic arthritis). The last information was not specified in the SoF table (to level up the certainty of the evidence or be added as indirect evidence of benefit), so it is not clear how it impacted the judgment of the magnitude of the benefit [29].

All five CPGs assessed the use of IVIG in MIS-C. NIH did not issue a recommendation due to the lack of evidence, while the other four CPGs issued recommendations in favor of its use (with certain differences). AEP did not show evidence, while NHMCR, ACR, and SSICM/PIGS showed cohorts that compared patients using IVIG with and without corticosteroids (thus, did not assess the effect of IVIG but of corticosteroids). Little or no explanation was given regarding including these studies in the decision-making. Instead, authors of these four guidelines referred to indirect evidence from other acute inflammatory diseases (such as Kawasaki disease, macrophage activation syndrome, fulminant myocarditis, and other vasculitis), which suggested a possible benefit, mostly in adults [14,[30], [31], [32], [33]]. However, no balance of benefits/harms of this indirect evidence was shown (narratively or using SoF/EP tables).

We found that recommendations regarding three topics (remdesivir, tocilizumab, and IVIG for MIS-C) were heterogeneous across guidelines, as observed in a previous review of CPGs for managing children with COVID-19 [14]. Certainly, different well-designed evidence-based CPGs may reach different recommendations regarding the same clinical question due to differences in the perspective, the collected evidence (new studies are continually being published), the values of outcomes, or other factors considered by the CPGs panels [34]. However, among the assessed CPGs, even those that summarize their meta-analyses using SoF tables do not clearly report the magnitude of the benefits and harms considered by the CPG panel and do not clearly state the contribution of other factors (such as costs, feasibility, acceptability, or equity), or the use of structured frameworks for decision-making, such as GRADE's Evidence-to-Decision (EtD) framework [29,35].

Recommendations

CPGs are supposed to guide clinicians’ decision-making. However, we found that few guidelines try to describe the process carried out to issue their recommendations. Of these, some do not perform meta-analyses or SoF tables to help the benefits/harms balance, nor structured frameworks to systematically assess the decision-making.

Since this might erode the confidence of CPG readers, lead to misleading recommendations [28], and difficult the clinicians’ decision making [36], CPGs developers should use systematic and transparent methodologies (such as GRADE) not only to show the summarized estimates and its certainty, but to manage indirectness, consider other factors, and finally reach a decision.

Also, although the development of high-quality guidelines is a priority, training clinicians to have certain core competencies in evidence-based medicine is necessary to understand and critically appraise the CPGs recommendations and ultimately improve the quality of healthcare [14]. This includes the understanding of SoF tables and decision-making frameworks [37].

Limitations and strengths

This study has some limitations. We intended only to assess the CPGs that issued recommendations and justified them, so our results could not be extrapolated to all CPGs for COVID-19 management in children. Also, we only assessed the information available for each CPG (since we tried to emulate putting ourselves in the place of the readers of these recommendations for decision-making). Lastly, we only deeply evaluated three recommendations in which we considered that there was still no consensus.

However, to our knowledge, this is the first study that has evaluated CPGs for COVID-19 management in children focusing on the variability of the recommendations and the evidence that supports them. Our results fuel a necessary debate about how challenging recommendations like these are being formulated in the face of scarce evidence and where we can go to make better decisions [3,38].

Conclusions

We found 5 CPGs for COVID-19 management in children that were published in 2021 and have explained their recommendations. Of these, only 1/5 CPG reached a score ≥70% in the overall assessment of the AGREE-II instrument, and 2/5 CPGs reached a score ≥70% in the rigor of development domain. In addition, heterogeneous recommendations and justifications across CPGs were found in the three assessed topics (remdesivir, tocilizumab, and IVIG in MIS-C). Some CPGs did not clearly present the benefits/harms, and no CPG seems to have used a structured decision-making framework.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Alvaro Quincho-Lopez: Conceptualization, Data curation, Roles/Writing – original draft, Writing – review & editing.

Lesly Chávez-Rimache: Conceptualization, Data curation, Roles/Writing – original draft, Writing – review & editing.

José Montes-Alvis: Conceptualization, Data curation, Writing – review & editing.

Alvaro Taype-Rondán: Conceptualization, Data curation, Writing – review & editing.

Giancarlo Alvarado-Gamarra: Conceptualization, Data curation, Writing – review & editing.

Declaration of competing interest

The authors report no conflict of interest regarding this study.