Awareness, value and use of the Australian living guidelines for the clinical care of people with COVID-19: an impact evaluation.

BACKGROUND AND
OBJECTIVE: The Australian National COVID-19 Clinical Evidence Taskforce is developing living, evidence-based, national guidelines for treatment of people with COVID-19. These living guidelines are updated each week. We undertook an impact evaluation to understand the extent to which health professionals providing treatment to people with COVID 19 were aware of, valued and used the guidelines, and the factors that enabled or hampered this.
METHODS: A mixed methods approach was used for the evaluation. Surveys were conducted to collect both quantitative and qualitative data and were supplemented with qualitative interviews. Australian healthcare practitioners potentially providing care to individuals with suspected or confirmed COVID-19 were invited to participate. Data were collected on guideline awareness, relevance, ease of use, trustworthiness, value, importance of updating, use, and strengths and opportunities for improvement.
RESULTS: A total of 287 people completed the surveys and 10 interviews were conducted during November 2020. Awareness of the work of the Taskforce was high and the vast majority of respondents reported that the guidelines were very or extremely relevant, easy to use, trustworthy and valuable. More than 50% of respondents had used the guidelines to support their own clinical decision-making; and 30% were aware of other examples of the guidelines being used. Qualitative data revealed that amongst an overwhelming morass of evidence and opinions during the COVID-19 pandemic, the guidelines have been a reliable, united source of evidence-based advice; participants felt the guidelines built confidence and provided reassurance in clinical decision-making. Opportunities to improve awareness and accessibility to the guidelines were also explored.
CONCLUSIONS: As of June 2021, the guidelines have been published and updated more than 40 times, include more than 140 recommendations and are being used to inform clinical decisions. The findings of this impact evaluation will be used to improve processes and outputs of the Taskforce and guidelines project, and to inform future living guideline projects.

Rapid living GRADE-based guidelines are both feasible and acceptable.

High level of trust in the National COVID-19 evidence based clinical guidelines.

Guidelines facilitated change in practice, informed the development of clinical care pathways, and underpinning education activities.

Introduction

Living guidelines are a new approach to developing and maintaining rigorous evidence-based guidelines in which any new, relevant evidence is rapidly incorporated, ensuring that recommendations are continually up to date with the latest research [1]. Living guideline methods are particularly useful in clinical areas in which research and practice are rapidly developing, of which COVID-19 is a clear example.

A small but increasing number of living guideline projects are underway, including in stroke, diabetes, arthritis, maternal health and the recent living WHO guideline on drugs for COVID-19 [2], [3], [4], [5], however little is currently known about the characteristics of the development or dissemination of these guidelines that make them more or less likely to succeed in having impact.

The Australian National COVID-19 Clinical Evidence Taskforce is a consortium of 32 Australian health professional organizations and collaboratives representing the full range of health professionals providing care to Australians with COVID-19, cofunded by Australian national and state governments and philanthropic organizations. The Taskforce develops and maintains living guidelines for the care of people with suspected or confirmed COVID-19. The guidelines are updated each week to reflect new evidence, and the structure and methods used by the Taskforce have been described previously [6]. In brief, the guidelines use the rigorous Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach [7] and are designed to meet Australian National Health and Medical Research Council (NHMRC) standards [8]. Each week, the team identify and review new evidence; convene multiple multidisciplinary guideline panels; revise existing recommendations; and make new recommendations. The guideline recommendations are published and freely accessible via MAGIC, an online guideline platform, are disseminated widely through mainstream and social media, and are promoted widely by Taskforce member organizations [9]. The website of the National COVID-19 Clinical Evidence Taskforce (https://covid19evidence.net.au) allows users to access the living guidelines and associated clinical flowcharts.

A process evaluation was conducted to explore the activity and experience of participants in the first five months of the Taskforce. The aim of the process evaluation was two-fold: to enable the Taskforce to improve process and outputs as the guideline development was underway; and to identify factors that might be useful to inform the design and development of future living guidelines [10].

Once the process evaluation was completed, and the methods and process of the living guidelines had become established, an impact evaluation was conducted to understand the extent to which clinicians were aware of, valued and used the guidelines; and the factors that underpinned this awareness, value and use. This manuscript presents the findings of the impact evaluation of the Australian living guidelines for the clinical care of people with COVID-19.

Methods

An impact evaluation protocol was developed by T.T., refined in discussion with the Taskforce Executive team (B.T., J.E., J.V., R.T., S.N.) and approved by the Taskforce Steering Committee (S.G.) (See Appendix). We originally intended to interview policy makers however this was not feasible due time constraints. An expanded evaluation will commence late 2021 to capture impacts of the guidelines from policy makers. Ethics approval was provided by Monash University Human Research Ethics Committee (Project ID: 26506). A mixed-methods approach using both quantitative and qualitative data was used. Surveys were conducted to collect both quantitative and qualitative data and were supplemented by one-to-one interviews with selected participants.

Impact evaluation matrix

End-user	Undifferentiated	Health practitioners		Health policymakers
Data type	Quantitative	Quantitative	Qualitative	Quantitative	Qualitative
Awareness measures	Website trafficTraditional and social media mentions	Membership in TaskforceAccess to flowcharts/ guideline recommendations		Membership in JLGApproaches from policymakers for evidence
		Survey assessing awareness and accessibility of guidelines and flowcharts
Acceptance/ adoption measures	Approval of guidelines by NHMRC	Adoption of recommendations by health organizations & professional groups		Adoption of recommendations by jurisdictional groups	Interviews exploring value of the Taskforce as a unified voice on COVID-19 treatment
		Survey assessing credibility and relevance of guidelines and flowcharts
Action measures		Prescribing data and/or registry data relevant to gamechanger recommendations	Interviews to capture stories of practice changes/ confirmation of current practices resulting from guidelines	Revisions to Position Statements etc., for example, re drug indications, availability, use.	Interviews about influence of outcomes of Taskforce on policy-making
		Survey assessing adoption and uptake of guidelines and flowcharts
Additional value of living approach		Survey assessing benefits & challenges of living approach			Interviews exploring benefits & challenges of living approach
Living evidence systems/capacity development	Tech development to support living evidence/ guidelines	Completion of GRADE training

Awareness, value and use surveys

Two rounds of surveys were undertaken to investigate the awareness, value and use of the guidelines. The first survey was distributed on Friday July 31, 2020, and remained open until Monday August 17, 2020. The second survey was distributed on Friday November 19, 2020, and remained open until December 9, 2020. Australian healthcare practitioners potentially providing care to individuals with suspected or confirmed COVID-19 were invited to complete these surveys. Primarily these were individual members of the 32 Taskforce member organizations. Participants were invited to complete the survey via the regular communication methods of the Taskforce and its member organizations, including but not limited to email lists, newsletters, and social media. Participation was voluntary, and completion of the online survey was considered implied consent.

Data were collected using an online survey tool (Qualtrix). The survey was the same in each round. It is possible that some participants completed the survey twice. The authors were unconcerned if participants answered both surveys as all information provided was relevant to the aims of the evaluation and representative of the state of the guidelines at the respective timepoint(s). Data were collected on awareness, relevance, ease of use, trustworthiness, value, importance of updating, use, and strengths and opportunities for improvement. Both quantitative data (Likert scales, yes/no) and free text data were collected. Quantitative data were analyzed using simple descriptive statistics. Qualitative data from the online surveys were combined with data collected through the interviews. Participants in the survey could also provide contact details if they wished to participate in an interview.

Interviews

Interview participants were invited via the regular communication methods of the Taskforce and its member organizations, including but not limited to email lists, newsletters, and social media. Potential interviewees known to the investigators were also emailed directly. Participation was voluntary and agreeing to conduct the interview was considered consent. All those who volunteered to participate were interviewed. Interview questions were based on a predetermined interview schedule, with questions varied to suit the interviewee's roles and experience. The interviews explored participants’ experience of the living guidelines including how they heard about the guidelines, how they used the guidelines, strengths and weaknesses, areas for improvement (such as refining or expanding), and their judgment of the overall impact of the guidelines. Interviews were conducted via online meeting software or by phone, and were audio-recorded, deidentified and transcribed verbatim. Detailed field notes were also taken. Interviews were conducted by an experienced qualitative researcher (T.T.) who was not previously known to any of the interviewees.

Deidentified data were thematically analyzed using an approach that was both inductive and deductive. Transcripts were read and reread for familiarization. An initial set of codes were developed by T.M., and verified by T.T. Some codes were identified a priori, using the interview schedule while others emerged inductively from the data. Coded extracts were collated into emerging themes which were reviewed and refined through discussion within the study team. T.M. undertook the primary data analysis. T.T. reviewed and collaborated on the conceptual development and refining of themes.

Results

Awareness, value and use surveys – quantitative findings

A total of 287 people completed the awareness, values and use surveys. The surveys were distributed across two different time points with the intention of maximizing the response rate. The first survey received 148 responses, and the second survey received 139 responses. While recommendations had been updated and new recommendations added to the guideline, no major changes had been made to the living guidelines development methods or the distribution strategy between survey rounds. An initial comparison between the results of the first and second round of the surveys revealed a high level of congruence between both the quantitative and qualitative data. As such, the data were merged and the results are presented here together.

Characteristics of respondents

The sample included a broad range of respondents by professional role, area of clinical expertise, practice location, state and territory. At the time of this evaluation Victoria had seen the highest number of COVID-19 cases in Australia, followed by NSW which is reflected in the demographic spread of the participants (Table 1).

Table 1

Survey respondents’ characteristics

Characteristic	Number	Percent of total
Professional role(s) (129 responses)
Allied health	3	2%
Medical	44	34%
Nursing	72	56%
Other	11	8%
Area(s) of clinical practice (130 responses)a
Emergency	19	15%
General	33	25%
Infectious diseases	24	18%
Intensive or critical care	22	17%
Pediatrics	5	4%
Pregnancy and childbirth	5	4%
Respiratory	4	3%
Other	45	35%
Main practice location (129 responses)
Metropolitan	76	59%
Regional/rural/remote	53	41%
Practice typea (129 responses)
Primary care	27	21%
Hospital	87	67%
Other	24	18%
States/territories (125 responses)
NSW	35	28%
NT	2	1.6%
QLD	21	16.8%
SA	6	4.8%
VIC	44	35.2%
WA	17	13.6%

Multiple selections responsible, so percentages don't sum to 100%.

Awareness

Awareness of the work of the Taskforce was high: 76% of respondents had heard of the guidelines prior to the survey; 62% had read the guidelines prior to the survey; and 53% had read or downloaded the flow charts prior to the survey. Respondents had heard about the guidelines from a wide range of sources, including Taskforce member organizations, colleagues, employers, Facebook, hospital libraries, internet searches, Primary Health Networks, and Twitter.

Respondents suggested many strategies to increase awareness, including increased dissemination through the member organizations, and encouraging the participants in the Guideline Panels to share on social media. Other suggestions included working closely with opinion-leaders; distributing through clinical networks, professional groups, health services, communities of practice, Australian Health Practitioner Regulation Agency (AHPRA), Australian Medical Association, and health unions; working with CEOs, senior executives and health librarians of health districts and health services; ensuring links to the guidelines are included on state and national government health websites; increasing traditional, academic and social media engagement, including podcasts; incorporating the guidelines into other decision-support tools such as “Medical Director,” “Therapeutic Guidelines,” and state guidelines; linking with clinical educators and training providers. Having a stable web address for the most recent version of the guidelines, improving the ease of download, improving findability in web searches, and developing a web/mobile app were also raised as important for building awareness.

Relevance

The majority of respondents reported that the guidelines were moderately (17%), very (41%) or extremely (30%) relevant to their health care practice. For most respondents who reported the guidelines were not at all or slightly (13%) or moderately relevant to their practice, this was because either they hadn't read the guidelines, or they had not seen people with COVID-19. Suggestions to improve relevance included more information on pregnancy and childbirth, and aged care, and consideration of transmission issues, infection control and the use of personal protective equipment. These topics were out of scope at the time of the evaluation.

Ease of use

Almost all respondents reported that the guideline recommendations were moderately (31%), very (47%) or extremely (14%) easy to use. The suggestions for improving ease of use focused on improving access, and improving presentation, navigation and layout within the MAGIC platform to better emphasize key information, integrating the guidelines with related guidelines in their jurisdiction, and simplifying the flow charts to make them easy for users to adapt.

Trustworthiness

Levels of trust in the guidelines were very high, with respondents reporting they were moderately (20%), very (52%) or extremely (21%) trustworthy. Issues impacting trustworthiness included a lack of primary research to include in the guideline and perceptions of inconsistency with other guidelines. Respondents emphasized that trustworthiness would be enhanced when the evidence underpinning the guidelines becomes more robust. Respondents noted that trust could be improved by increasing the visibility of care providers using the guidelines and having the guidelines endorsed by the state Chief Health Officers.

Use

Levels of use of the guideline were very high: more than 50% of respondents had used the guidelines to support their own clinical decision-making; and 30% were aware of other examples of the guidelines being used. Respondents described having used the guidelines in a wide variety of ways, including to: inform treatment decisions for COVID-19 patients; develop local treatment guidelines and COVID-19 response strategies; compare past treatment decisions made to recommendations in the guidelines; compare with recommendations in other guidelines; seek new reliable evidence on unconventional or novel treatments; develop and deliver clinical education, and share with colleagues; explain and discuss with patients. Several respondents (30%) also reported that they were aware of others using the guidelines for purposes such as to develop or check clinic, unit, hospital, health service and state-wide guidelines, treatment pathways, protocols and policies; and to develop education programs.

Strengths and opportunities for improvement

Frequent updating was seen as being vital to the usefulness of the guidelines, with over 90% of respondents reporting that it was very or extremely important. Respondents described the strengths of the guidelines are that they are “simple, clear, easy to follow” and “evidence based, regularly updated, clinician led.” Overall, the majority of respondents thought that the guidelines were very (51%) or extremely (28%) valuable. Further data about the strengths and opportunities for improvement are presented under the qualitative evaluation.

Qualitative findings from surveys and interviews

In the following presentation of the results, data from the interviews were combined with qualitative data collected through the online survey. Nine interviews were conducted with 10 interviewees during November 2020. Interview participants included medical specialists (Pharmacologist; Sleep physician; ICU consultant; ICU Registrar; Nurse educator/remote area nurse; Critical Care Liaison Nurse Practitioner; Obstetrician; Public Health Nurses) who were involved in the care or guidance of care for people with COVID-19.

Uncertain times: overwhelming diversity of evidence and opinion

Collectively, participants described the “early days of COVID” as being particularly stressful and overwhelming. They explained that the amount of emerging information from varied sources, along with the quantity of very diverse opinions regarding all aspects of COVID as producing a high level of uncertainty for themselves and their colleagues. Several participants were departmental heads and/or key decision makers involved in hospital care. They described feeling particularly stressed as they attempted to navigate the evidence to identify trustworthy, reliable sources to guide their clinical and policy decisions. They expressed concern that much of the emerging information at that time wasn't peer reviewed and experienced difficulties aligning views among staff members regarding important clinical decisions.

“And there was a lot of talk, and a lot of conflicting views about what we ought to be doing… this is so hard because we all think something different. How are we actually going to come up with anything?”

“A lot of the early information before it was accumulated into guidelines came from different overseas experiences and so recommendations were changing back and forth. Information was coming through bit by bit and a lot was less peer reviewed literature.”

Participants discussed their sense that there was a lack of common ground. They described feeling as though there was a wide variation of opinion and strategy regarding issues surrounding infection control, treatment and patient management.

“The biggest frustration that I encountered with managing COVID, was the range of opinions, beliefs, and views from all sorts of people, including my learned intensive care colleagues but particularly also from those who have very little insight into respiratory disease, pandemic management, epidemiology, intensive care treatments, PPE and [occupational health and safety].”

Several participants described the quantity of evidence as becoming a bigger and bigger problem over time as they struggled to “keep up” while also attempting to discern what was most accurate, evidence based and trustworthy. They described the burden of this load on top of their rapidly expanding clinical load and the stress that accompanied this. The need for a single point of reference and guidance was becoming increasingly clear.

“And so when COVID first came out we were discussing all the different evidence available and all the different guidelines. In the very beginning there was minimal available and then there was a sudden rush of everything and so much information coming out. We were wondering what the true source was and what was the most accurate information.”

“Media and social media were affecting our ability to interpret objective data and there was fear that came into people. Which is an example of why you need to have a learned group of people come out and put forward published guidelines. Come out and say, actually, there is a big group of us and this is the position that we think we should be taking.”

Reliable, consistent source of evidence-based advice

The overall standpoint expressed by participants was that the guidelines served as a reliable, trustworthy reference point in a time when everything was both rapidly changing and highly uncertain. They described the relief they felt when the guidelines were released and widely endorsed by contributing organizations and reputable sources.

“Really, it's the one voice, the one clear voice.”

“We find them really useful because it's a one-stop shop for the latest evidence of where we're up to.”

“When the living guidelines came on board, it was just fantastic to have someone distilling all the evidence that was coming in and writing some high-level national guidelines that we could use as our source of truth, because there was just so much disparate information. It was really lovely to have them assimilated and presented as this is the best practice currently with what we know.”

Several factors contributed to participants’ perceptions of the guidelines as being the most reliable source for evidence-based advice, including accessibility of the guidelines, the reliable “living” process of rapidly identifying and integrating new evidence, the wide consultation methods used in developing the guidelines, and the transparent processes the guidelines employed.

Accessible, comprehensive and well organized

The group described accessibility as a major facilitator in their use of the guidelines as their preferred source for evidence-based advice. Participants recounted the different ways through which they accessed the guidelines, including keeping the homepage of MAGIC open as a browser on their phone for frequent review, relying on the weekly communiqué to alert them to updates, or accessing the guidelines via MagicApp or the Taskforce website as needed. Convenient access, clear organization and structure were emphasized as key facilitators to their use of the guidelines as a reference point.

“I found it really helpful particularly back in March, April when there were more unknowns about how it was going to play out. As the data was coming in I thought [the guideline] was really well compiled. It didn't look like it was just a rushed job. It looked like it was well considered, well organized, well formatted and easy to read.”

“What I love is how searchable it is. It doesn't take a long time to familiarize yourself with the guidelines to find what you need.”

Reliable living process

Some of the participants specifically described the methods utilized by the Taskforce as key to their use of the guidelines as their preferred reference point for evidence-based advice. The daily identification and weekly integration of new evidence made the process both rigorous and highly reliable. Participants described the benefit of having the updated recommendations highlighted on both the website and in the weekly communiqué. Users reported they can quickly identify which recommendations are most relevant to them and if there have been any changes requiring changes to clinical practice.

“The advantage of the guideline was that it kept updating us with new information as it came out. The results of trials and things that came out, I felt, were fairly rapidly incorporated into the guidelines.”

Wide consultation, expert clinical involvement

The wide consultation methods utilized through the guideline development process was repeatedly highlighted by participants as increasing the rigor of the process and thus, reinforcing their trust in the guidelines. Participants emphasized the value of having their senior colleagues, clinicians and trusted national organizations involved in the development of the guidelines as paramount to the value/trust they placed in the guidelines. They described this as critical to increasing awareness, implementation and translation of the recommendations.

“Most of the organizations I'm part of or familiar with are involved with the Taskforce. Also there's a significant number of practicing clinicians, people working on the ground who are involved with the Taskforce as well. Obviously we want to listen to what they have to say because they are the people that have actually put this into practice with real COVID-19 cases.”

“I think that's what they really like, that it's clinicians telling other clinicians what to do. It's not academics or researchers saying this is what you should be doing.”

The importance of transparency - a clear evidence base

Several participants emphasized the use of GRADE methodology as ensuring transparency of the guidelines and being key to understanding exactly why recommendations were being made for or against a particular treatment or management modality. The clarity and rigor provided by GRADE processes, particularly in describing which level of evidence was available for each recommendation facilitated enhanced trust in the guidelines. Where there was limited evidence, consensus statements were described as being extremely useful to provide guidance that was endorsed by a group of esteemed, trusted colleagues.

“It's the fact that it's transparent as to what it's based on. So when the advice comes out that we should or should not use dexamethasone, it's made clear as to why or why not. Or why aren't we using hydroxychloroquine? It's clear why the advice is no, don't use it. Whereas the opinion makers just tend to promote their opinion and don't tell you why.”

“So I think people find [consensus based recommendations] very persuasive, if we say something is consensus recommendation that it's coming from people who are treating people with coronavirus, this isn't just someone in a room somewhere saying, oh yeah, I've seen this paper this is what we ought to do.”

Trust, confidence and reassurance

The theme of trust ran deeply throughout the interviews. Participants emphasized the value of the guidelines in providing ‘stable’ reassurance that they were delivering the most current, evidence-based and peer reviewed care. They described feeling reassured knowing that the guidelines would be readily accessible should they need to access them. In a situation that commenced with an overwhelming lack of clarity and was dominated by feelings of uncertainty and being overwhelmed, participants’ trust in the guidelines, and the rigorous methods the guidelines followed, translated into feelings of confidence and certainty.

“I would say [the guidelines] had a stabilizing influence because whenever we've had people worried about –“is this right thing to do,” you can always point them to the guidelines and say, well this is evidence-based, wide consensus in the industry based on the best science and the best health advice we've got. So you can trust that this is the best we've got.”

“I found it much easier when I said to all my staff, “listen guys, we're just going to do what the guidelines have been saying. You know, if you've got any questions before coming to me, read the guidelines and read the department of health website. If you're still uncertain, come and talk to me.” And it got everyone on the same page, it made conversations a lot easier and it made people recognize that actually we can do this.”

The guidelines were also described as a “security blanket,” not necessarily immediately required in settings of low caseload, however reassuring to have in the case of an outbreak.

“But I have heard from a lot of people and certainly the way I feel is that if things change within [our location] and within our hospital system, I don't necessarily need to be on top of all the evidence because I know where to go if I need extra help on any of the areas of how to manage these patients in hospital, because it's all there. And I think even things like the CPR flow charts whilst I have looked at them, I haven't focused on every single detail because it's not something that I need to at the moment. But I know it's there as a kind of a safety security blanket. I know how to get to it if I need it.”

Complex ways through which the guidelines have had an impact

In describing they ways they used the guidelines, serving as a reference point was most common. Participants explained their use of the guidelines to inform their own clinical practice, and also, from a policy and procedure point of view, where participants cross checked the recommendations of their organization against the taskforce guidelines, aligning them as required.

“So every now and then just to ensure that I'm up to date I flick over to the living evidence to make sure that nothing new is coming out.”

“For me, it was nice to have this reference point to go back to and say, has the position on this medication or this treatment, or this therapy changed in the light of any new evidence? No, we are still saying clinical trial only, or yes, this is recommended based on RCT data.”

“So as the guidelines came on, we'd check them against the pathway and identify any points of difference. Then we would escalate that to the medical controller at the public health unit and say, this is what the national guidelines are saying. I can't think of a single occasion where we didn't adopt what you'd written in the living guidelines.”

Evolution of the guideline - adapting to the changing landscape of COVID within Australia

In discussing the future directions of the guidelines or suggested improvements, participants again emphasized the security they felt in knowing that the guidelines were current and readily accessible. Participants were satisfied with the guidelines and felt that they provided a comprehensive, useful and relevant repository of evidence and guidance.

Participants were largely aware of the plans for further expansion of the guidelines. While they recognized the vital importance of keeping currency with regards to COVID-19 disease modifying treatments, participants noted that evolution of the guidelines, and potentially the model of maintaining/updating these guidelines was important. Participants expressed that the planned addition of guidance surrounding infection control, recovery and long COVID would be beneficial.

“And of course now we have evolved a bit more from focusing only on treatments, now we're focusing more on the primary care aspects. And we're now getting into some of the natural history. I am starting to see some of those kinds of patients, patients who've had COVID-19 who have got ongoing issues, even though they're not infectious anymore. So I think the guidelines have definitely evolved and we're now moving into other areas, but that still doesn't take away from that initial focus, which is very much treatment focused.”

One of the participants suggested expanding the guidelines to provide guidance on patient education and resources to support knowledge translation. This participant noted the need for emotional support for patients and staff who are impacted or affected by COVID. This idea is connected to the sense of stress and uncertainty outlined in earlier themes.

“We've got information for health practitioners this is what you need to do but for people that are isolated and having to do stuff they're not comfortable with, one of the real challenges is how do you explain this to people. How do you explain these things to the general public who are anxious or sometimes aggressive, particularly people who have been ordered to have a swab and they don't want one because they think it doesn't apply to them? How do you have those conversations? So whether it's tips and tricks or some resources to help people to have those difficult conversations or some links for support for those staff that are feeling distressed because of those conversations. Obviously the task force wouldn't actually do any of that stuff, but just to provide links, if people are going to the site for the one-stop shop, being able to say, okay, well this is really upsetting me, I'm really struggling now, what do I do?”

Suggestions for improving awareness and accessibility of the guideline

Several participants across the interviews and surveys expressed that more should be done to increase awareness of the guidelines, both on a national and an international level, with some providing specific examples of situations where people were unaware of the guidelines.

“When I give talks, there are still people that haven't heard of it. That worries me that it hasn't permeated through the healthcare system in the way that it should have, in particularly people that are in hospitals.”

Suggestion for improving the accessibility of the guidelines were proffered including developing an app to host and access the guideline, and making the guideline in a printer friendly format.

“Maybe if it was advertised as an app or had some usability features like that maybe that would mean a few more people able to use it click through it.”

Better integration with the guidance provided by state-level health departments was highlighted by several participants as vital to increase usability and trust in the guidelines. The need for consistent national guidance, rather than guidance which varied between states was also emphasized. Participants suggested that comparisons of the Taskforce guidance with international and state guidelines should be provided and a rationale to justify the different recommendation made in the Taskforce guidelines.

“In Victoria we need to follow state guidelines, so they were helpful but not very useful for us (survey response).”

“We have to follow local guidelines so couldn't follow these directly (survey response).”

Discussion

In mid-April 2020, the first version of the living, evidence-based Australian guidelines for the clinical care of people with COVID-19 was published including 10 recommendations. As of June 2021, the guidelines have been published and updated more 40 times and include more than 140 recommendations. At the time of the evaluation Victoria had seen the highest number of COVID-19 cases in Australia.

The participants in this study described a high level of trust in the National COVID-19 evidence based clinical guidelines. They repeatedly emphasized the relief they felt that a trusted and well-supported group were producing well informed, evidence-based clinical guidelines. In the absence of strong evidence in the earlier days of the pandemic, they emphasized the value of consensus statements being developed by experienced colleagues and organization representatives. The transparent and reliable process utilized by the Taskforce in developing recommendations was crucial in respondents viewing the guidelines as a reliable reference point for evidence-based advice. The complex ways through which the guidelines have an impact included facilitating change in practice of individual clinicians, informing the development of clinical care pathways at state and local levels, and underpinning education activities for clinicians.

The volume of research output in COVID-19 has been exponential, presenting numerous complexities to the health care system and clinicians. Over 63,000 studies have been published or registered to date, a number that increases by approximately 1000 studies per week [11]. At a time of great uncertainty, the rapid development of national clinical practice guidelines has relied on consistent and trustworthy interpretation of this rapidly emerging evidence from around the world.

The guidelines have provided one of the first, large-scale test cases for a living approach to GRADE-based guideline development. The success of the Australian guidelines in terms of their trustworthiness and use, highlight the value of living evidence synthesis approaches for novel diseases where the evidence base is rapidly evolving. The update schedule is significantly more frequent that other living guidelines projects [2,3,5]. The World Health Organization's guideline for COVID-19 drugs and the BMJ Rapid Recommendation on Remdesivir are other examples of living guidelines methods, though these guidelines are narrower in scope and updated less frequently [5]. Unlike the guidelines developed by the Taskforce, The National Institute of Health's COVID-19 Treatment Guidelines did not use GRADE “because the urgency of the pandemic required that the guideline be launched expeditiously, and there was insufficient time to train the full Panel on how to apply the GRADE framework [12].”

Limitations of this evaluation include the small number of interview participants, however the convergence of themes across the interviews and the survey results along with the diverse roles of the respondents increase our confidence in the results. Rich data were provided through the mixed-methods design. The qualitative findings were reflective of the quantitative data and provided valuable depth. The researchers in the current evaluation participated in the development of the guidelines which could have limited objectivity, however this had the benefit of increasing the depth of the understanding of the context. Ideally, the evaluation period would have been extended and results compared across time points as the living guideline methods used by the Taskforce are further refined however this was beyond the scope of this project. It was also beyond the scope of this evaluation to address the effects of the guidelines on prescribing practices or health outcomes for patients. Work by the Observation Data Working Group is underway to address these important outcomes. Finally, the findings of this study, and the feasibility and acceptability of a living guideline model such as the one used by the Taskforce may be limited in their generalizability in countries and locations that have limited access to internet.

The Australian Guidelines for the Clinical Care of People with COVID-19 are an important example of intensive application of living guideline methods. The findings of this study clearly highlight the feasibility and acceptability of rapid living GRADE-based guidelines, and begin to explore how living approaches might influence the impact of guidelines. Funding to support the maintenance of living guidelines, endorsement from State and National government and integration with existing guidelines is important to the utilization and sustainability of the living guidelines.

The findings of this study provide useful insights and will be used to guide future work in this area. Specifically, the findings will be used to improve processes and outputs of the Taskforce, and to inform future living guideline projects. Work to further refine the living guideline model continues.

Authors’ contributions

T.T. and S.G. developed the methods, which all of the authors revised on behalf of the National COVID-19 Clinical Evidence Taskforce. T.T. collected the data, which T.M. and T.T. analyzed. T.M. prepared the first draft of the manuscript and incorporated the feedback to produce the submitted version of the manuscript, which all of the authors approved.

Funding

The National COVID-19 Clinical Evidence Taskforce is funded by the Australian Government Department of Health, the Victorian Government Department of Health and Human Services, the Ian Potter Foundation and the Walter Cottman Endowment Fund, managed by Equity Trustees, and the Lord Mayors’ Charitable Foundation. The funders played no role in the development of the methods, in the writing of the report, or in the decision to submit the article for publication.