Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review.

Background: The development of evidence-based clinical practice guidelines (EB-CPGs) has increasing global growth; however, the certainty of impact on patients and health systems, as well as the magnitude of the impact, is not apparent. The objective of this systematic review was to assess the effectiveness of the application of EB-CPGs for the improvement of the quality of health care in three dimensions: structure, process and results in the patient for the management of cardiovascular disease.
Methods: We followed the methods described by the Cochrane Handbook and present a descriptive analysis because of the high heterogeneity found across the included studies. We searched the Cochrane Central Register of Controlled Trials, MEDLINE and EMBASE databases, as well as the grey literature, between 1990 and June 2016. No language restrictions were applied. Only randomised clinical trials (RCTs) were selected. Three authors independently carried out the data extraction, using a modified version of the Cochrane Effective Practice and Organization of Care form.
Results: Of the total of 84 interventions included in the nine RCTs evaluated, three (4%) were related to health care structure, 54 (64%) to the health care delivery process and 27 (32%) to patient outcomes. Regarding the impact of using the EB-CPGs, in 55 interventions (65%), there were no significant differences between control and experimental groups. In four interventions (5%), the result favoured the control group, and the result favoured the intervention group on 25 of the interventions (30%). Conclusions: This systematic review showed that EB-CPGs could be useful to improve the process and structure of health care and, to a lesser extent, to improve the patients' outcomes. After analysing many studies, we could have one more hypothesis for further research, which could shed more light upon those undiscovered variables that might interfere with the use of the EB-CPGs. Registration: PROSPERO CRD42013003589. Copyright:

Introduction

This review refers to the changes in the quality of healthcare services that are direct consequences of the systematic use of evidence-based clinical practice guidelines (EB-CPGs). “Evidence-based” means that the recommendations are created using rigorous, unbiased and transparent methods of collation and appraisal, alsousing scientific findings of the highest quality and value to assist in providing optimal clinical care to the patient. ( Guyatt, 1992; Sackett ). EB-GPCs are evidence summaries and include systematically developed recommendations to assist physicians and patients in the process of making decisions ( Alonso-Coello ; Glasziou ). EBCPGs are one of the most available and widely used tools for translating the knowledge generated by scientific research to practice. They should be based on the best scientific evidence available in order to improve the quality of patient care ( Alonso-Coello ; Grimshaw & Russell, 1993). Experts, institutions, and organizations worldwide have developed EBCPGs in all areas of medicine as a key tool to improve the quality of care Donabedian, 1988; Glasziou ; Lugtenberg ; Moore states that before evaluating health care, we must first decide how to define “quality” and whether it depends only on the actions of physicians or if it also depends on contributions from the patients and the healthcare system.

Defining quality is challenging since it is not easy to characterise coherently and objectively. Health must be analysed from a holistic point of view, and guideline developers must determine the ideal amount of influence health should receive from individual preferences and social components. We must also understand the relationship between structural characteristics and healthcare processes, as well as their results in health services ( Donabedian, 1988; Moore ).

Since the 1990s, increasing numbers of EB-CPGs are being developed. However, it is unknown whether these high-quality recommendations have a beneficial impact on patient health. Despite the high number of recently published EB-CPGs, there are few studies on their effectiveness in improving clinical outcomes, the process and structure quality throughout the healthcare system.

After an exhaustive search, only two systematic reviews (SR) we found on this topic ( Lugtenberg ; Worrall ). Worrall only analysed patient outcomes missing two of the proposed Donadebian Model. Lugtemberg 2009 used the full Donabedian (1988) model only including studies from The Netherlands.

This review sought to assess whether the quality of health care improves in patients with the cardiovascular disease when using EB-CPGs vs standard professional medical practice. The primary aim was to assess the impact of the EB-CPGs for the management of cardiovascular diseases on healthcare quality, in terms of patient outcomes, management process, and healthcare structure.

Methods

We designed a methodology aimed to find and analyse studies measuring the impact of EB-CPGs on the improvement of quality in health care services in the three areas proposed by Donabedian ( Donabedian, 1988; Moore ): structure, process, and patient outcomes. From the very beginning of the process when the authors wrote this review protocol, it was clear that these items would not be easy to measure and, as the search and data extraction moved forward, it became harder to synthesise the information delivered by the different studies included. The main obstacle was the inconsistency observed in the different outcome measures used by the studies, which included continuous as well as dichotomous values for different clinical conditions and interventions. The intervention definition read as any planned action taken to modify the clinical practice and use of the clinical guidelines for influencing in the clinical practice. The authors followed the methodological recommendations described in the Cochrane Handbook ( Higgins & Green, 2012). This review is registered with PROSPERO (ID: CRD42013003589).

Study searches

We did a systematic search using the following electronic databases for primary studies (randomised controlled trials (RCTs)): Cochrane Central Register of Controlled Trials (CENTRAL). The Cochrane Library, including the Cochrane EPOC (Cochrane Effective Practice and Organization of Care) specialised database; MEDLINE; EMBASE; CINAHL; PsycINFO; LILACS; Health Technology Assessment Databases and Web of Science, Science Citation Index, and Social Sciences Citation Index.

The review authors combined search strategy for indexed terms and developed free text terms. We included searches of grey literature in different sources, such as reports of the world and regional conferences, academic theses and scientific reports not published in indexed journals. We searched for studies published between January 1990 and June 2016, without any language restriction. An advanced search strategy is available as Extended data, Appendix 1 ( Ramirez ).

Study selection

We analysed the studies found through the search strategy, and two authors independently selected the articles according to the following inclusion criteria: RCT measuring the change in health care when using EB-CPGs on cardiovascular disease. The study should measure the change in any of the three health care dimensions (structure, process, and patient outcomes).

Data extraction

Three authors independently performed data extraction using a modified shorter version of the Cochrane Collaboration EPOC Data Collection Checklist translated into Spanish. Apart from preparing the Spanish version, we eliminated several items not applicable to this review, given that we only included RCTs. We used a standardised digital form for data extraction and analysis. We used a standardised digital form for data extraction and analysis. We used Review Manager software (RevMan 5.3) for the data analysis.

We assessed the risk of bias (quality) according to the Cochrane Handbook ( Higgins & Green, 2012). We found very high variability, so the study results were introduced as a narrative in the Results.

Because of the variability between the measurements of the effect of the impact of EB-CPGs on the change of quality in the studies included in this review, it was not possible or appropriate to perform a meta-analysis; therefore, it was not possible to measure the statistical heterogeneity.

Results

Study identification and selection

When the first version of the protocol of this Systematic Review was initiated, consider the inclusion of several clinical topics in a single systematic review. We decided to create a series of three systematic reviews on different clinical topics: Cardiovascular Health, Breast Cancer and Child and Mother Health. The initial searches included all those topics with the same inclusion criteria described. After removing duplicates, the search produced 4279 potential studies. After screening by title and abstract, 4051 were excluded. After the full-text evaluation, 96 studies of all clinical subjects were selected, extended data, Appendix 2 ( Ramirez ).

Study characteristics

For the analysis, we organised the studies according to the topic or pathology being the core subject of the EB-CPG, and for this report, we only selected RCTs on cardiovascular diseases, because this theme accounts for the higher number of original articles. In total, we selected nine RCTs ( Figure 1). Characteristics of the included studies are available as Extended data, Appendix 2 ( Ramirez ).

Figure 1.

Flow chart of the studies (PRISMA, Moher 2009).

We included nine RCTs analysing cardiovascular diseases ( Beaulieu ; Berner ; Ellis ; Guadagnoli ; Hand ; Jäntti ; Kiessling & Henriksson, 2002; Tierney ; Tsuyuki ). All trials were carried out between 2000 and 2015, five in the United States of America, two in Canada, one in Finland and one in Sweden. Eight articles addressed outpatient care and three inpatient care. The EB-CPGs included in the selected trial looked at the following clinical problems: management of stable angina pectoris (over 65 years), unstable angina, dyslipidaemia, acute infarction, heart failure and blood pressure. Besides, the trials included perioperative cardiac evaluations of patients with non-cardiac surgery, cardiopulmonary resuscitation and secondary prevention in patients having coronary artery disease.

A number of the 4,279 studies found in the initial search were not randomised clinical trials, as the authors described in their titles or abstracts. When assessing the full-text articles, we found that many were cluster trials and observational studies with a “before and after” design.

Quality of evidence assessment

As for the quality of the evidence, we observed the presence of a high or unclear risk of bias for allocation concealment (selection bias), blinding of participants and personnel (performance bias), and blinding of outcome assessment (detection bias), which can be explained by the nature of the interventions studied. We found several types of systematic errors: random sequence generation (selection bias), incomplete outcome data (attrition bias) and selective reporting (reporting bias). We found that the interventions measured yielded outcomes assessed with moderate to low evidence certainty according to the GRADE classification.

Risk of bias assessment

Analysing the risk of bias in the nine included RCTs studies found the random sequence generation (selection bias) assessment had a low risk of bias (between 50–75%) for allocation concealment (selection bias) 33% low, 33% unknown and 33% high risk of bias. The blinding of participants and personnel (performance bias) obtained almost 50% high risk of bias. The blinding of outcome assessments (detection bias) obtained a 50% low and 50% uncertain risk of bias. Incomplete outcome data (attrition bias) obtained between a 50 and 75% low risk of bias. For the selective reporting (reporting bias), there was a 100% low risk of bias. Finally, other types of bias had a low risk of between 50 and 75% ( Figure 2).

Figure 2.

Overall bias risk chart of all included studies.

The studies with the lowest and highest risk of bias ( Kiessling & Henriksson, 2002 and Jäntti , respectively) were chosen to be evaluated with the GRADE methodology and to build a summary of findings table. The aim of doing this GRADE table was obtaining the rank of possible grades of the evidence certainty found in the 84 interventions from the nine studies included. The results of these studies produced moderate-to-low evidence certainty, according to the GRADE classification (see Table 1 and Table 2).

Table 1.

Evaluation of the Certainty of the Evidence for the Jäntti 2007 study on cardiopulmonary resuscitation, according to the GRADE classification.

Are the recommendations presented in the ERC 2005 CPG better than those presented in the ERC 2000 CPG more adequate to improve the outcomes for cardiopulmonary resuscitation?
Patient or population: Manikins simulation of cardiopulmonary arrest Intervention: CPG ERC 2005 Control: CPG ERC 2000 Bibliography: Jäntti H1, Kuisma M, Uusaro A.. The effects of changes to the ERC resuscitation guidelines on no flow time and cardiopulmonary resuscitation quality: a randomised controlled study on manikins. Resuscitation November 2007;2(75):338–44. [PubMed: 17628319]
Outcomes	Average duration (seconds ± SE)		Relative effect (p)	№ of participants (studies)	Certainty of the evidence (GRADE)	Relevance
	Average with ERC 2000	Average with ERC 2005
Total time without flow (s)	393 ± 19	190 ± 23	p < 0.001	34 Control Group: 17 Intervention Group: 17 (1 RCT (Randomized Controlled Trial)	⨁⨁⨁◯ MODERATE [1, 2]	Critical
Delay to start CPR (s)	8 ± 6	8 ± 4	p = 0.949	34 Control Group: 17 Intervention Group: 17 (1 RCT (Randomized Controlled Trial)	⨁⨁◯◯ LOW [1, 3]	Critical
ERC: European Resuscitation Council
SE: Standard deviation
p: significance
GRADE Working Group grades of evidence
High certainty: We are very confident that the true effect lies close to that of the estimate of the effect
Moderate certainty: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different
Low certainty: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect
Very low certainty: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

Explanations

1. For the generation of the random sequence (selection bias) the risk is High: sealed, numbered and opaque envelopes were used for the randomisation of the cases to be treated. It is not clear why people with less work experience or no academic degree were assigned to the ERC 2000. For the allocation concealment (selection bias) the risk is High: It is likely that by the nature of the study the allocation concealment of the selection could not be made. For blinding of participants and staff (performance bias) the risk is High: It is likely that the nature of the study could not prevent participants from knowing which group they belonged to (which CPG they were using). For blinding of the outcome evaluation (detection bias) the risk is Low: A computer automatically collected it. For incomplete results data (attrition bias) the risk is Low: There was no loss of follow-up since it was a single session. For the particular report (notification bias) the risk is Low: A computer automatically collected it. In other risks of bias, the risk of bias is High: The description of the study design was not clear; therefore, we assumed that the study has a high risk of bias.

2. P < 0.001

3. P 0.949. Downgraded -1 for imprecision.

Table 2.

Evaluation of the Certainty of the Evidence for the secondary prevention of coronary artery disease according to the GRADE classification: ( Kiessling & Henriksson, 2002) study.

Is the active implementation more effective compared to passive implementation as it was used in the control group to improve secondary prevention of coronary artery disease?
Patient or population: person with coronary artery disease. Intervention: Active implementation (general practitioners participated in learning dialogues using the CPG on secondary prevention of coronary artery disease, method of recurrent cases in their primary care centers) Control: Passive implementation (the CPG was mailed to general practitioners and presented at a conference) Bibliography: Kiessling A1, Henriksson P. Efficacy of case method learning in general practice for secondary prevention in patients with coronary artery disease: randomised controlled study. BMJ October 2002;7369(325):877-80. [DOI: 12386042]
Outcomes	Average Change of active versus passive implementation of the CPG (%) (95% CI)		Relative effect (p)	№ of participants (studies)	Certainty of the evidence (GRADE)	Relevance
	Control Group	Intervention Group
Difference in percentage of LDL change at 2 years [Mean (mmol / L)]	0.7% (CI -4.1 a 5.9)	-9.3 % (CI -15.8 a -2.9)	p < 0.05	176 Control Group: 88 Intervention Group: 88 (1 RCT (Randomized Controlled Trial)	⨁⨁⨁◯ MODERATE [1, 2]	IMPORTANT
Difference in percent change in total cholesterol at 2 years [Mean (mmol / L)]	1.8% (CI -2.2 a 5.9)	-6.0 % (CI -10.4 a -1.5)	p > 0.05	176 Control Group: 88 Intervention Group: 88 (1 RCT (Randomized Controlled Trial)	⨁⨁◯◯ LOW [1, 3]	IMPORTANT
CI: Confidence interval
p: significance
GRADE Working Group grades of evidence
High certainty:We are very confident that the true effect lies close to that of the estimate of the effect
Moderate certainty:We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different
Low certainty:Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect
Very low certainty:We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

Explanations

1. For the generation of the random sequence (selection bias) the risk of bias is Uncertain: Not described. For allocation concealment (selection bias) the risk is low: general practitioners and patients did not know in which research group they were assigned. For blinding of participants and staff (performance bias) the risk is low: General practitioners were not aware of being involved in the study and a blinded nurse on which group each patient belonged to, was the one who handled the paperwork, protocols of the investigation and had no contact with general practitioners. For blinding of the outcome assessment (detection bias), the risk is Low: The research codes and databases were not disclosed until the authors completed the statistical analysis. For incomplete results data (attrition bias), the risk is Low: The study used the intention-to-treat analysis and indicated that the follow-up for two years was 86%. For the particular report (notification bias) the risk is Low: The research codes and databases were not disclosed until the authors completed the statistical analysis. For Other biases the risk is Low: None known

2. p < 0.05

3. p > 0.05. Downgraded -1 for imprecision.

Assessment of study outcomes

The authors grouped the outcomes in simple relative and absolute numbers. There was not a global estimate of the measurements of the effects included in the studies because of the considerable variability of measuring units, as well as the clinical heterogeneity found among the studies included. The measurement of the outcomes reported in the studies was dichotomous, continuous or nominal; the majority were dichotomous and were used to measure the mastery of the process, e.g. the number of patients receiving an adequate treatment following a recommendation versus those who did not.

Of the total of 84 interventions included in the nine RCTs evaluated, three (4%) corresponded to the health care structure dimension, 54 interventions (64%) to the dimension health care delivery and 27 interventions (32%) to the dimension of patient outcomes. Regarding the impact of EB-CPG use, we found that in 55 interventions (65%) there was no significant difference between the control and experimental groups.

In four interventions (5%) the outcome favoured the control group (comparison of the measure of compliance of the recommendations (the average adjusted by the patient characteristics, the setting, and the number of measures applied per patient)). The measure of effects (odds ratio (OR)) regarding the conditions acute myocardial infarction and heart failure, separated by care provided by a cardiologists or primary care physicians, were as follows: acute myocardial infarction, cared by a cardiologist OR 0.81 (95% CI: 0.79 to 0.83), and cared by a primary care physician OR 0.73 (95% CI: 0.71 a 0.76); heart failure, cared by a cardiologist OR 0.88 (95% CI: 0.86 to 0.90), and care by a primary care physician OR 0.79 (95% CI: 0.76 to 0.81).

The result favoured the intervention group for 25 interventions (30%). Some of the recommendations were: use of antiplatelet medication during the 24-hour hospital stay, a study of the left ventricle ejection fraction, total cholesterol and LDL measurements, cardiopulmonary resuscitation, and degree of compliance with EB-CPGs recommendations.

Discussion

Methodological efforts have been made to develop trustworthy EB-CPGs. However, it still seems that some of these EB-CPGs are far from the reality of the clinical practice ( Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, 2011).

It is essential to emphasise the main findings from the outcome analysis. It is surprising that most of the studies did differentiate between the control and experimental groups regarding the improvements with the use of EB-CPGs. The effects of recommendations of the interventions included in the nine RCTs on the areas of health care structure (4%) and patient outcomes were the least studied (32%). This fact could lead us to assume that researchers have given more importance to the evaluation of using (or not using) the recommendations in the area of the process, instead of their direct impact on the patient health.

The changes observed in the patient progression were generally modest. In the six studies evaluating patient outcomes (27 interventions), four of them reported a positive result for seven measurements (26%) of the interventions. For example, the study by Ellis measured the difference in the average change of total cholesterol (mg/dl) and reported a decrease of 7.4 mg/dl in favour of the intervention. It is essential to mention that these measures were mostly surrogate variables. Only one study reported variables for clinical results ( Tierney ), with 13 interventions, but found no meaningful differences in the results between the groups.

Most of the studies (eight) reported results in the process area and showed favourable results for the intervention in six studies with 16 measurements. This fact reflects the reluctance of health care providers to use EB-CPGs and means that only 30% of these interventions followed the recommendations given by the EB-CPGs.

Our findings in this review coincide with some of those captured by Gagliardi and Kastner most of the studies included in this review used multiple strategies to implement the EB-CPGs (electronic notifications in a digital file, cell phone applications, letters, phone call memos, printed material), and the authors do not tailor to every recommendation (the implementation strategy is usually the same for every recommendation within a EB-CPG). Cosby (2006) formulates very clearly that although clinicians have the new evidence at hand, using it means a change in the behaviour and habits of the clinical management of patients that is complex to achieve unless appropriate strategies are applied. A more specific approach, based on the results of the analysis of the obstacles hindering the adoption of every recommendation separately, might improve the use and effect of the recommendations in practice.

Conclusions

There is an imbalance between the number of EB-CPGs developed and the number of high-quality studies evaluating their effectiveness. After analysing many studies, we can have one more hypothesis for further research for more light upon those undiscovered variables that might interfere with the use of the EB-CPGs. Therefore, more studies of good quality are still needed.

The variation in the effects of the recommendations of the EB-CPGs suggests that it would be useful to focus on the analysis of the adherence limitations, as well as on designing implementation strategies by adapting every recommendation, instead of considering the EB-CPGs as a whole. Further research is still needed to determine which factors related to the EB-CPGs and their specific recommendations are essential to predict the use of EB-CPGs, and thus achieve better patient outcomes.

Implications for practice

The initial objective of this review was to strengthen the development programs for EB-CPG by evaluating their effects on the quality of health care and to give reliable evidence to sustain the decision-making process related to the construction of EB-CPGs in medium- and low-income countries. Even the research evidence is not strong enough to support the EB-CPGs as a tool to improve the practice for a better quality of care, the results of this review need to be interpreted with caution. Definitely. It seems the standard application used so far must be reviewed and must incorporate new psychosocial strategies oriented toward driving change in clinical practice and the doctor-patient relationship We need reaching the right hands at the right time. It is necessary to emphasise that the standard CPG implementation used so far must be reviewed, and must incorporate new psychosocial strategies oriented toward driving change in clinical practice and the doctor-patient relationships.

For an adequate implementation of a EB-CPG, it is necessary to take into account the possible costs, risks, and benefits that will be necessary to know the expected results precisely. The health systems that build EB-CPGs must maximise their efforts to get health care personnel to follow the recommendations of the EB-CPGs and make an effort to evaluate their impact.

Implications for research

This is the first of series of three systematic reviews on the effects of Evidence-Based Clinical Practice Guidelines (EB-CPGs) on health care quality improvements. The second RD will cover Breast Cancer and the third child and maternal health. We created an interactive https://lnkd.in/dwTxaD8 LIVE GUIDELINES program following the CDC https://lnkd.in/d5r4z6m

Since research in this field is so new, and the results of this study were not conclusive, more research is needed to evaluate the change that EB-CPGs could make to the quality of health care, emphasising the less studied areas, such as the structure of health care services and patient outcomes.

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Extended data

Open Science Framework: Effects of Evidence-Based Clinical Practice Guidelines in cardiovascular health care quality improvements- A Systematic Review. https://doi.org/10.17605/OSF.IO/9A5FM ( Ramirez ).

This project contains the following extended data:

Appendix 1 Advanced Search Strategy

Appendix 2 List of studies selected after screening and assessing the full text

Appendix 3 Characteristics of included studies English version

Some electronic data extraction forms

Reporting guidelines

Open Science Framework: PRISMA checklist for “Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review”. https://doi.org/10.17605/OSF.IO/9A5FM ( Ramirez ).

It is OK, the new version of the article clearly improve the original one and include the reviewers suggestions.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thank you for the invitation to review this well-conducted systematic review.

The study under review is very interesting and necessary for the Clinical Practice Guidelines (CPGs) community and for the implementation science and healthcare quality communities.

I would like to congratulate the authors for this difficult challenge with some minor comments to contribute to improve the content and message of this necessary study

ABSTRACT: Conclusion

"There are probably still undiscovered variables that interfere with the use of the CPGs".

That could not be a conclusion from the study, given that the aim of the work was not to study factors and conditions that facilitates the implementation of CPG neither the conditions to change professional behavior. No data on these issues are reported, then no conclusion could be provided.

A coherent conclusion could be the need to improve and systematize this kind of implementation studies.

"After an exhaustive search, only two systematic reviews (SR) we found on this topic,"

It is needed to provide both references

RESULTS: pag 3, 4 and figure 1. Study identification and selection.

"Of these, we selected 96 studies after assessing the full texts".

It is needed, in order to strengthen the transparency and reproducibility of the work, to display in an extended data appendix, the full reference list of rejected originals and reasons to be rejected (96 found minus 9 selected = 87 rejected). It should be clarified what is meant in the figure 1 tab "# 87 of studies to include in other analyzes".

Pag 4, left column, para 3, rows 4-6:

"When assessing the full-text articles, we found that many were cluster trials"

There is no clear reasons to reject RCT clusters studies, when the intervention could be randomized to cluster offices, and the outcomes effect measured at individual patient level. It is a well-accepted methodology in implementation science, given that can control for bias and non-measured variables. (Please see 2 references provided: Eccles and Romero). [1] , [2]

Pag 5, Table 1, Rows 2-4 and Column 5:

"Certainty of the evidence (GRADE)"

It is difficult to understand why the certainty of evidence for the outcomes "time without flow" and "delay to start CPR" are classified differently when, as displayed in the explanations, the bias and systematic errors are the same, and there is not a loss of patients to follow up. It is true that both effects measure score different statistical significance, but with data provided the readers feel with the same certainty (either moderate or low) that one effect is significantly different from the control group, and for the other outcomes (delay) there are not differences with the control group.

Pag 6, Table 2, Rows 2-4 and Column 5:

"Certainty of the evidence (GRADE)"

The same reasoning for the difficulty to understand why are classified differently the certainty of evidence for the outcomes "LDL cholesterol" and "Total cholesterol".

Discussion:   Pag 7, para 2, rows 7-13

"This fact could lead us to assume that researchers have given more importance to the evaluation of using (or not using) the recommendations in the area of the process, instead of their direct impact on the patient health".

Another explanation could be to emphasize that it use to be easier to measure process of care than patient health outcomes. These health outcomes take a longer period of observations than to measure how professionals adopt CPG recommendations. That can explain the "preferences" of researchers.

Conclusions

The same considerations exposed in the abstract regarding the sentences  "There are probably still undiscovered variables that interfere with the use of the CPGs".

That could not be a conclusion from the study, given that was not the aim of the work to study factors and conditions that facilitates the implementation of CPG neither the conditions to change professional behavior. No data on these issues are reported, them no conclusion could be provided.

A coherent conclusion could be the needs to improve and systematize these kinds of implementation studies.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Dear Ignacio Marín León

Thank you for your Review Report

This response was concomitant between Dr. Ramirez and Dr. Tristan, part of the authors of this systematic review.

APPROVED WITH RESERVATIONS

Thank you for the invitation to review this well-conducted systematic review.

The study under review is very interesting and necessary for the Clinical Practice Guidelines (CPGs) community and for the implementation of science and healthcare quality communities.

I would like to congratulate the authors for this difficult challenge with some minor comments to contribute to improving the content and message of this necessary study

#1 ABSTRACT: Conclusion Answer: We change the text for a better understanding of our meaning. After analyzing many studies, we can have one more hypothesis for further research for more light upon those undiscovered variables that might interfere with the use of the CPGs.

"There are probably still undiscovered variables that interfere with the use of the CPGs".

That could not be a conclusion from the study, given that the aim of the work was not to study factors and conditions that facilitates the implementation of CPG neither the conditions to change professional behavior. No data on these issues are reported, then no conclusion could be provided.

A coherent conclusion could be the need to improve and systematize this kind of implementation studies.

pag 3, para 4, left column Answer: We agree and change for: After an exhaustive search, only two systematic reviews (SR) we found on this topic (Worrall, 1997; Lugtenberg, 2009). Worral (1997) only analysed patient outcomes missing two of the proposed Donadebian Model. Lugtemberg 2009 used the full Donadebian (1988)  model only including studies from The Netherlands.

"After an exhaustive search, only two systematic reviews (SR) we found on this topic,"

It is needed to provide both references

#3 RESULTS:  pag 3, 4 and figure 1. Study identification and selection. Answer: We agree . The Appendix 2: List of studies selected after screening and assessing the full text is added at OSF Home, the link is: https://osf.io/9a5fm/?view_only=0ec7b68ed1004bf6b1aa74a19ed1912d

"Of these, we selected 96 studies after assessing the full texts".

It is needed, in order to strengthen the transparency and reproducibility of the work, to display in an extended data appendix, the full reference list of rejected originals and reasons to be rejected (96 found minus 9 selected = 87 rejected). It should be clarified what is meant in the figure 1 tab "# 87 of studies to include in other analyses".

We did not reject the 87 publications, not analyzed. This SR is the first of series that has three systematic reviews. This two new ones will include the topics breast cancer, type 2 diabetes and others.

Pag 4, left column, para 3, rows 4-6:  Answer: The authors decide when preparing the protocol to include only RCT.

"When assessing the full-text articles, we found that many were cluster trials"

There is no clear reasons to reject RCT clusters studies, when the intervention could be randomized to cluster offices, and the outcomes effect measured at individual patient level. It is a well-accepted methodology in implementation science, given that can control for bias and non-measured variables. (Please see 2 references provided: Eccles and Romero). ,

#5 Pag 5, Table 1, Rows 2-4 and Column 5:  It is difficult to understand why the certainty of evidence for the outcomes "time without flow" and "delay to start CPR" are classified differently when, as displayed in the explanations, the bias and systematic errors are the same, and there is not a loss of patients to follow up. It is true that both effects measure score different statistical significance, but with data provided the readers feel with the same certainty (either moderate or low) that one effect is significantly different from the control group, and for the other outcomes (delay) there are not differences with the control group.

"Certainty of the evidence (GRADE)"

Answer: Please check the note explanations both have different statistical significance due to this fact the certainty of evidence is downgraded for “Delay to start CPR (s)”.

#6 Pag 6, Table 2, Rows 2-4 and Column 5:  The same reasoning for the difficulty to understand why are classified differently the certainty of evidence for the outcomes "LDL cholesterol" and "Total cholesterol".

"Certainty of the evidence (GRADE)"

Answer: Please check the note explanations both have different statistical significance due to this fact the certainty of evidence is downgraded for “Difference in percent change in total cholesterol at 2 years [Mean (mmol / L)]”.

#7 Discussion:  Pag 7, para 2, rows 7-13  Answer: You are right.

"This fact could lead us to assume that researchers have given more importance to the evaluation of using (or not using) the recommendations in the area of the process, instead of their direct impact on the patient health".

Another explanation could be to emphasize that it use to be easier to measure process of care than patient health outcomes. These health outcomes take a longer period of observations than to measure how professionals adopt CPG recommendations. That can explain the "preferences" of researchers.

#10 Conclusions That could not be a conclusion from the study, given that was not the aim of the work to study factors and conditions that facilitates the implementation of CPG neither the conditions to change professional behavior. No data on these issues are reported, them no conclusion could be provided.

The same considerations exposed in the abstract regarding the sentences  "There are probably still undiscovered variables that interfere with the use of the CPGs".

A coherent conclusion could be the needs to improve and systematize these kinds of implementation studies.

Answer: We change the text for a better understanding of our meaning. After analyzing many studies, we can have one more hypothesis for further research for more light upon those undiscovered variables that might interfere with the use of the CPGs.

Thank you for the invitation to review this well-conducted systematic review.

The study under review is very interesting for the evidence-based clinical practice guidelines (CPGs)- and for the healthcare quality and safety- research communities.

Moreover, I would like to congratulate the authors for this work with some minor comments with the aim of improving the content and message of this distinguished study.

Abstract:

I recommend replacing the phrase 'results in patients' with ' as it is more commonly used in the field of quality and safety in healthcare. Additionally, the authors have used it for the rest of the MS.

Introduction:

I would recommend adding as nowadays clinicians are getting increasingly interested in the fields of evidence-based medicine (EBM) and HCQ either from a research perspective or through engagement in their respective healthcare organizational accreditation. The HCQ definition and description of EB-CPGs are already in the PROSPERO protocol, please include them here as well.

Additional relevant studies to cite here:

(i) about the relation between EBM and HCQ:

Glasziou P, Ogrinc G, Goodman S. Can evidence-based medicine and clinical quality improvement learn from each other?.  BMJ Qual Saf. 2011;20 Suppl 1(Suppl_1):i13–i17. doi:10.1136/bmjqs.2010.046524 [1]

(ii) an earlier similar key review:

Grimshaw J, Russell I: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.  The Lancet. 1993;  342 (8883): 1317-1322 [2]

page 3, left column, para 4:

'..After an exhaustive.....only two systematic reviews ....., but ' did not....'

Please add citations for these two references (SRs).

'...and international perspective together.' What do you mean by an 'international perspective'; of what; (i.e. of CPGs, of healthcare quality, or of cardiovascular healthcare?), please clarify or re-write.

page 3, right column, para 4:

Regarding the data extraction modified and translated from the EPOC checklist, was it validated and if not, why the authors felt they did not need to? Additionally, please provide the readers who would be interested in the replication of this research with this tool as an online supplementary here.

It was indeed mentioned with a link in the PROSPERO protocol ( https://fs20.formsite.com/mtristan/form41/index.html, attached on Appendix 3) but the link did not work, giving an error!

Moreover, I checked the (Extended data) links in the OSF Home but could not retrieve it.

Please delete the duplicated sentence; 'We used a standardized digital ..........and analysis'.

page 3, right column, para 7:

The #87 studies that were planned to be included in another analysis were not mentioned here since the final included were 9 rather than 96 studies. Please elaborate on this in this location.

Figure 1. PRISMA flow chart:

#87 of studies to include in other '

Tables 1 and 2.: the GRADE quality assessments and the explanations for the certainty of evidence were very well conducted and written.

Table 1: please, if feasible, replace the question in the GRADE SoF table; '.....the results for cardiopulmonary resuscitation?' with '.....the  for cardiopulmonary resuscitation?'

page 7, left column para 1:

Once more, please replace 'patient results' with '

page 7, left column, para4:

Discussion:

'..However, it still seems that some of CPGs are...'

page 7, right column, para 2:

As part of discussing 'multiple strategies to implement the CPGs', I would recommend adding insights and citing these relevant keynote articles that discuss multi-faceted CPGI strategies and their importance:

(i) Grol R, Wensing M. Improving patient care: the implementation of change in health care (in Dutch: Implementatie: effective verbetering van de patiëntenzorg). Amsterdam: Reed Business Education; 2013.

(ii) Kastner M, Bhattacharyya O, Hayden L, et al. Guideline uptake is influenced by six implementability domains for creating and communicating guidelines: a realist review. J Clin Epidemiol. 2015;68(5):498–509. doi: 10.1016/j.jclinepi.2014.12.013. [3]

(iii) Gagliardi A, Brouwers M, Bhattacharyya O, et al. A framework of the desirable features of guideline implementation tools (GItools): Delphi survey and assessment of GItools. Implement Sci. 2014;9:98. doi: 10.1186/s13012-014-0098-8 [4]

(iv) Gagliardi A, Brouwers M, Palda V, Lemieux-Charles L, Grimshaw J. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6:26. doi: 10.1186/1748-5908-6-26. [5]

page 7, right column, para 4:

'...... as well as on designing implementation strategies adapt every recommendation...'

Do you mean "designing implementation strategies AND adapt every recommendation....if so add a comma ',' " or "designing implementation strategies (by) adapting every recommendation".

Additionally, I suggest to add:

' instead of considering  the CPGs as a whole'.

page 7, right column, para 5:

Implications for practice The initial objective of this review was to strengthen the development programs for

The results of this review must need to be interpreted with caution.

It reads better either (the results of this review interpreted with caution.) or (the results of this review interpreted with caution.).

'It is necessary to emphasise that the standard implementation used so...'

page 7, right column, para 6:

'For an adequate implementation of a CPG, it is necessary to take into account the possible costs, benefits

Finally, I would like to congratulate the authors once more on this work and invite all interested researchers to conduct similar research using this robust methodology to study the impact of EB-CPG implementation on all different healthcare specialties in addition to their effects on the interdisciplinary workflows and transitions of care between different specialties and services in healthcare.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Dear Yasser Sami Abdel Dayem Amer

Thank you for your Review Report

This response was concomitant between Dr. Ramirez and Dr. Tristan, part of the authors of this systematic review.

Thank you for the invitation to review this well-conducted systematic review.

The study under review is very interesting for the evidence-based clinical practice guidelines (CPGs)- and for the healthcare quality and safety- research communities.

Moreover, I would like to congratulate the authors for this work with some minor comments with the aim of improving the content and message of this distinguished study.

Abstract: Answer: We agree and we will make the change.

I recommend replacing the phrase 'results in patients' with  ' as it is more commonly used in the field of quality and safety in healthcare. Additionally, the authors have used it for the rest of the MS.

Introduction:

I would recommend adding   as nowadays clinicians are getting increasingly interested in the fields of evidence-based medicine (EBM) and HCQ either from a research perspective or through engagement in their respective healthcare organizational accreditation. The HCQ definition and description of EB-CPGs are already in the PROSPERO protocol, please include them here as well. Additional relevant studies to cite here:(i) about the relation between EBM and HCQ:Glasziou P, Ogrinc G, Goodman S. Can evidence-based medicine and clinical quality improvement learn from each other?.  BMJ Qual Saf. 2011;20 Suppl 1(Suppl_1):i13–i17. doi:10.1136/bmjqs.2010.046524 (ii) an earlier similar key review:Grimshaw J, Russell I: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.  The Lancet. 1993;  342 (8883): 1317-1322 Answer: We agree and we will make the change CPGs for EB-GPGs and added the recommended references the are very useful.

page 3, left column, para 4: Answer: We agree, and we will make the change.

'..After an exhaustive.....only two systematic reviews  ....., but ' did not....'Please add citations for these two references (SRs).

Answer: We agree, and we will make the change for: after an exhaustive search, only two systematic reviews (SR) we found on this topic (Worrall 1997, Lugtenberg 2009). Worral (1997) only analysed patient outcomes missing two of the proposed Donadebian Model. Lugtemberg 2009 used the full Donadebian (1988)  model only including studies from The Netherlands.

'...and international perspective together.' What do you mean by an 'international perspective'; of what; (i.e. of CPGs, of healthcare quality, or of cardiovascular healthcare?), please clarify or re-write.

page 3, right column, para 4: Answer: We agree, and we will make the change. The Some electronic data extraction forms the link is not available now it is a temporary link during the active time of the survey period in the form site server.  Now some results of the on line survey are  available  in this link  https://osf.io/9a5fm/?view_only=0ec7b68ed1004bf6b1aa74a19ed1912d as  Some electronic data extraction forms/Short version of EPOC Check list.docx. The link in the OSF Home is working.

Regarding the data extraction modified and translated from the EPOC checklist, was it validated and if not, why the authors felt they did not need to? Additionally, please provide the readers who would be interested in the replication of this research with this tool as an online supplementary here.It was indeed mentioned with a link in the PROSPERO protocol ( https://fs20.formsite.com/mtristan/form41/index.html, attached on Appendix 3) but the link did not work, giving an error!Moreover, I checked the (Extended data) links in the OSF Home but could not retrieve it.

Please delete the duplicated sentence; 'We used a standardized digital ..........and analysis'. Answer: We agree, and we will make the change

page 3, right column, para 7: Answer: We agree, and we will make the change. And we add the following:

The #87 studies that were planned to be included in another analysis were not mentioned here since the final included were 9 rather than 96 studies. Please elaborate on this in this location.

When the first version of the protocol of this Systematic Review was initiated, consider the inclusion of several clinical topics in a single systematic review. We decided to create a series of three systematic reviews on different clinical topics: Cardiovascular Health, Breast Cancer and Child and Mother Health. The initial searches included all those topics with the same inclusion criteria described. After removing duplicates, the search produced 4279 potential studies. After screening by title and abstract, 4051 were excluded. After the full-text evaluation, 96 studies of all clinical subjects were selected. For this SR, only nine studies covering the cardiovascular issue were included.

This is the first of series of three systematic reviews on the effects of Evidence-Based Clinical Practice  Guidelines (EB-CPGs) on health care quality improvements. The second RD  will cover Breast Cancer and the third child and maternal health.  We created an interactive https://lnkd.in/dwTxaD8 LIVE GUIDELINES program following the CDC https://lnkd.in/d5r4z6m

Answer: We agree, and we will make the change

Figure 1. PRISMA flow chart: #87 of studies to include in other  '

Tables 1 and 2.: the GRADE quality assessments and the explanations for the certainty of evidence were very well conducted and written. Answer: We agree, and we will make the change

Table 1: please, if feasible, replace the question in the GRADE SoF table; '.....the results for cardiopulmonary resuscitation?' with '.....the  for cardiopulmonary resuscitation?'

page 7, left column para 1: Answer: We agree, and we will make the change

Once more, please replace 'patient results' with '

page 7, left column, para4:Discussion: Answer: We agree, and we will make the change

'..However, it still seems that some of   CPGs are...'

page 7, right column, para 2: (ii) Kastner M, Bhattacharyya O, Hayden L, et al. Guideline uptake is influenced by six implementability domains for creating and communicating guidelines: a realist review. J Clin Epidemiol. 2015;68(5):498–509. doi: 10.1016/j.jclinepi.2014.12.013.

As part of discussing 'multiple strategies to implement the CPGs', I would recommend adding insights and citing these relevant keynote articles that discuss multi-faceted CPGI strategies and their importance:(i) Grol R, Wensing M. Improving patient care: the implementation of change in health care (in Dutch: Implementatie: effective verbetering van de patiëntenzorg). Amsterdam: Reed Business Education; 2013.

(iii) Gagliardi A, Brouwers M, Bhattacharyya O, et al. A framework of the desirable features of guideline implementation tools (GItools): Delphi survey and assessment of GItools. Implement Sci. 2014;9:98. doi: 10.1186/s13012-014-0098-8

(iv) Gagliardi A, Brouwers M, Palda V, Lemieux-Charles L, Grimshaw J. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6:26. doi: 10.1186/1748-5908-6-26.

Answer: Thank you very much for the recommended references. We agree that they will give strength to the implementation aspect, one of the most sensitive components for the use of CPG.

In relation to the reference  Gagliardi 2011, We will not take it in the article because the Research Implementation Guide and Application Network (GiRAnet), is not active and in its link http://www.cebhc.co.za/guideline- implementability-research -y-application-network-giranet-2 /, one link does not work and the other is restricted access.

We had access to another article by the same author of 2012, which we share ideas that we have already implemented and this study is in protocol: Gagliardi, A. R., Brouwers, M. C., & Bhattacharyya, O. K. (2012). The guideline implementability research and application network (GIRAnet): an international collaborative to support knowledge exchange: study protocol. Implementation Science, 7, 26. https://doi.org/10.1186/1748-5908-7-26

The new paragraph reads as follows:

Our findings in this review coincide with some of those captured by Gagliardi (2012) and Kastner (2015) most of the studies included in this review used multiple strategies to implement the EB-CPGs (electronic notifications in a digital file, cell phone applications, letters, phone call memos, printed material), and the authors do not tailor to every recommendation (the implementation strategy is usually the same for every recommendation within a EB-CPG) Cosby (2006) formulates very clearly that although clinicians have the new evidence at hand, using it means a change in the behaviour and habits of the clinical management of patients that is complex to achieve unless appropriate strategies are applied. A more specific approach, based on the results of the analysis of the obstacles hindering the adoption of every recommendation separately, might improve the use and effect of the recommendations in practice.

page 7, right column, para 4: Answer: We partially agree, and we will make the change. We will not mention or suggest of pilot program implementation because most of them majority yield very little or nothing.

'...... as well as on designing implementation strategies adapt every recommendation...'Do you mean "designing implementation strategies AND adapt every recommendation....if so add a comma ',' " or "designing implementation strategies (by) adapting every recommendation". Additionally, I suggest to add: ' instead of considering   the CPGs as a whole'.

page 7, right column, para 5: Answer: We agree, and we will make the change

Implications for practice The initial objective of this review was to strengthen the development programs for

Answer: We agree, and we will make the change

The results of this review must need to be interpreted with caution. It reads better either (the results of this review   interpreted with caution.) or (the results of this review    interpreted with caution.).

'It is necessary to emphasise that the standard   implementation used so...' Answer: We agree, and we will make the change

page 7, right column, para 6: Answer: We agree, and we will make the change

'For an adequate implementation of a CPG, it is necessary to take into account the possible costs,   benefits