Combating information chaos: a case for collaborative clinical guidelines in a pandemic.

The speed and scale of new information during the COVID-19 pandemic required a new approach toward developing best practices and evidence-based clinical guidance. To address this need, we produced COVIDProtocols.org, a collaborative, evidence-based, digital platform for the development and dissemination of COVID-19 clinical guidelines that has been used by over 500,000 people from 196 countries. We use a Collaborative Writing Application (CWA) to facilitate an expedited expert review process and a web platform that deploys content directly from the CWA to minimize any delays. Over 200 contributors have volunteered to create open creative-commons content that spans over 30 specialties and medical disciplines. Multiple local and national governments, hospitals, and clinics have used the site as a key resource for their own clinical guideline development. COVIDprotocols.org represents a model for efficiently launching open-access clinical guidelines during crisis situations to share expertise and combat misinformation.

Main text

Introduction

The scale, speed, and unprecedented nature of the COVID-19 pandemic has posed challenges for evidence-based medicine (EBM). EBM systems generally have not been designed to parse evidence that is rapidly shifting and not yet peer reviewed. Digital health tools provide a unique opportunity to incorporate many of the principles and processes of EBM into rapid guideline development when traditional methods are not possible. Collaborative writing applications (CWAs) such as Wikis and Google Documents can shorten turnaround and enable fast-paced expert review, the product of which can then be made public on digital health platforms. Here, we describe COVIDProtocols.org, a model of open-access and continuously updated guidelines that was built using a CWA and has been used over 2 million times by 500,000 users in 196 countries. We discuss the challenges of emerging COVID data and misinformation, the limits of existing guidance structures, and the potential for digital health to solve some of these issues.

The COVID-19 pandemic is the most rapidly escalating novel disease crisis in the EBM era. It coincided with the advent of preprint archives, proliferating online journals, and an increase in social media information sharing. This confluence created a deluge of information that bypassed normal EBM practices such as peer review and expert consensus. At the beginning of the pandemic, frontline healthcare workers faced an influx of patients with limited and often poor-quality (case series, pre-print, and social media) information about diagnosis, treatment, and infection control practices for COVID-19. These sources often lacked peer review and data transparency. In the first week of March 2020 there were a mere 261 publications related to COVID-19. In the first 5–6 months of the pandemic, 35%–50% of articles were pre-prints and nearly all were retrospective or descriptive in nature.1, 2, 3 The early dearth of variable quality information quickly gave way to an overabundance: as of July 1, 2021, there are over 175,000 publications relating to COVID-19 indexed on PubMed. Journals have made strong efforts to expedite review and make access to COVID-19 content free to all users., But the rapid escalation of information has allowed misinformation about COVID-19 to emerge on both traditional and social media, pushing unproven and sometimes harmful options, such as hydroxychloroquine, and likely costing many lives.,

Under circumstances such as the COVID-19 pandemic, the role of expert guidance is critical: in the paucity-of-data scenario, expert guidance can help create a “best guess” at recommended practices by using a combination of experience with similar diseases, known and putative biological mechanisms, limited clinical data, and expertise in critically interpreting imperfect scientific literature. For example, early in the pandemic, The Society of Critical Care Medicine (SCCM) released initial guidelines based on previous experience with sepsis and influenza. In the overabundance-of-data scenario, expert moderation serves to arbitrate and prioritize information. However, it is a challenge to rapidly achieve expert consensus and to continually update that consensus. Further, clinicians and the public need ready access to expert content. In situations like the COVID-19 pandemic, digital collaboration tools, such as CWAs and shared databases, can be used to expedite expert guidance and facilitate frequent re-review as new data emerge. They can also make expert guidance available as easy-to-use tools that are more readily disseminated compared to traditional static guidelines.

Traditional guideline development

The National Academies of Science, Engineering, and Medicine (NASEM) established eight standards for guideline development, which include transparency, managing conflict of interest, establishing evidence foundations, and updating (www.nationalacademies.org/our-work/standards-for-developing-trustworthy-clinical-practice-guidelines). To achieve these standards, clinical guidance organizations typically rely on an invited panel of experts that review and discuss available evidence and then vote to reach final consensus. While many groups use their own internally designed protocols to facilitate this process, some use formalized methods such as the Nominal Group Technique (NGT, a face-to-face facilitated group process) or Delphi methods (serial questionnaires followed by rounds of voting without face-to-face interaction).

There are three main types of guideline-forming organizations: (1) governmental and intergovernmental, such as national ministries of health and the World Health Organization (WHO); (2) societal, released by professional societies such as the Infectious Disease Society of America (IDSA); and (3) institutional, produced by individual health care institutions.

The guidance produced by these organizations have their limitations. Many are relatively static and difficult to use at the point of care (e.g., not searchable or mobile optimized). Governmental and societal guidelines often lack operational-level detail due to their mandate to be widely applicable. Governmental guidelines are also often slow to be published. The US National Institutes of Health (NIH) released COVID-19 clinical guidance on April 21, 2020, 3 months after the CDC declared coronavirus a public health emergency and almost 6 weeks after the WHO declared it a pandemic. Societal guidelines tend to focus on their membership’s domain of expertise and practices and do not typically cover all aspects needed for the clinical care of a patient. Institutional guidelines have the most operational detail but are often not publicly shared and may not be generalizable. At the outset of the COVID-19 pandemic, several institutions, such as the University of Washington (covid-19.uwmedicine.org), admirably made protocols publicly available online or by informal email circulation, but these guidelines were generally in PDF form without search functions.

Digital health platforms and EBM

Digital health reference applications, defined here as websites and mobile applications designed to look up medical information, have several advantages for clinicians: point-of care access from different kinds of devices, capacity for more frequent updating than textbooks, and non-linear presentation of information that permits cross-linking, decision support tools, and interactive graphics.

Digital health references can be both a boon and a threat to EBM. The for-profit, subscription-based digital health references with expert moderation, a market dominated by UpToDate.com and DynaMed, have supplanted textbooks for practical clinician use: UpToDate reports they are used by 1.9 million clinicians (www.learn.uptodate.com/RTR_July_RB). These resources typically source, verify, and reference information and require declaration of conflict of interest in a manner similar to societal guidance organizations. Many high-quality free sites also provide evidence-based information to clinicians and patients. In contrast, many lower quality digital health tools are riddled with misinformation, and some can raise concerns regarding their processes for evidence adjudication, experts’ credentials, or conflicts of interest.

Collaborative writing applications

Collaborative writing applications (CWA) are online applications that allow for concurrent text editing by multiple authors. They offer a unique solution for rapid editing and review by multiple editors with clear authorship attribution. CWAs can be designed for multiple different purposes including general shared text editing (e.g., Google Docs), scientific paper collaboration (e.g., Manuscripts.io), programming language (e.g., version control systems), spreadsheets (e.g., Airtable), and webpages (e.g., wikis).

The use of wikis for “crowd-sourcing” web content has taken off in the last two decades. Contrary to popular belief, wikis do not require full open-sourcing: user participation in these platforms can either be open to the general public or limited to certain groups (i.e., medical professionals) or a combination of both, depending on the specific goals. In some areas, public moderation via CWAs has supplanted expert moderation; for example, Wikipedia, where anyone can make changes to all but the most controversial pages directly.

The use of wikis in medicine has so far been less successful, although the use of CWAs in the healthcare sector for non-website reasons is increasing. Clinical guidance wikis are still uncommon, likely in part because they suffer from opaque policies on how data are reviewed, curated and updated. One systematic review found over 88% of medical wikis had only half of their content revised yearly, and 24% of the sites were essentially unused. The majority of sites had no governing rules on how data were reviewed. A Cochrane review of medical wikis revealed that over 70% of users felt wiki-based information is either partially or completely unreliable.10, 11, 12 Significant barriers to successful adoption were users’ unfamiliarity with internet communication technology, lack of IT resources, and distrust in the information’s scientific quality. However, the use of wikis to develop or review clinical evidence-based guidelines is not without precedent: in 2012, a working group of oncologists in Australia developed a wiki to establish guidelines for the treatment of sarcoma. After a core working group conducted a systematic literature review and drafted treatment guidelines, outside providers and organizations were invited to comment via a wiki platform. The guidelines were published online after final review and are updated by the editorial group., In a second example, the CareTrack Kids project in Australia invited a broad range of professionals to review pediatric guidelines via a wiki edited by a clinical champion and a site administrator.,

Wikis tend to have several technical limitations that also limit their use in medical settings. Most wikis do not allow for extensive formatting options: images, graphics, charts, tabs, font format, highlighting, and other interactive options may be limited. Wikis also require contributors to learn how to use the software and usually necessitate password access. Many wiki hosting sites are blocked by hospital firewalls, and many are not mobile optimized.

COVIDProtocols.org experience

On March 20th, 2020, 9 days after the WHO declared the Coronavirus pandemic, a group of COVID-facing clinicians at Brigham and Women’s Hospital (BWH) made our work-in-progress protocols publicly available as a creative-commons copyright Google Doc published directly as a web page at COVIDprotocols.org. In less than 2 weeks, we turned this into a formal website in partnership with Upstatement, a volunteer professional web development firm. Our initial guidelines focused on tertiary care, but in November 2020, the BWH team partnered with UCSF Open Critical Care and with Partners In Health as part of a United States Agency for International Development (USAID)-funded initiative to produce content relevant to both tertiary care and resource-variable settings.

We explicitly designed our website to use Google Docs as a content management system (CMS) for the website to avoid some of the shortcomings of wikis described above, but still maintain the editing advantages of CWAs. This meant that the software for the site drew on content directly housed in Google Docs and did not require an intermediate step of repeated reformatting and pasting content into a different web CMS (e.g., Wordpress, Joomla, Drupal). This approach reduced delays between finalizing content and publishing. Using Google Docs allowed us to choose our own URL, build a professional interface, and add features not available with most wikis. The editor interface for Google Docs is very familiar to most people and does not require a significant learning curve, new software, or new password access.

The site received 1.38 million page views from over 417,000 users by August 2020, the majority of which occurred in the first month before the NIH and other guidance organizations released their content. To put this in perspective, the digital health industry leader, UpToDate, was searched 4.1 million times for COVID-19 in that same time frame. With permission, our protocols were incorporated and adapted into multiple hospital and governmental protocols around the world (for example Armenia and Malawi). We owe this success to three factors: (1) a transparent and credible review process, (2) speed and agility in editing and deployment, and (3) end-user experience.

Transparent and credible review

Editor controls and committee review. Our vetting process is based on a modified Delphi process (Figure 1A) with over 200 authors contributing to chapters edited by about 15 section editors. Senior reviewers and three senior editors then approve or reject the new content. The editor-in-chief cross checks content between disciplines for discrepancies, which prompts “in-person” review; an online multidisciplinary committee arbitrates the eventual recommendation. New content can be created or suggested from any author or editor but must go through the same approval process. Transparency in authorship is abetted by Google Doc’s automatic archiving/tracking function, so it is very simple to follow changes. Further, different levels of editing can be granted: editing, suggesting (changes are visibly tracked), or viewing. Notably, this process is volunteer driven and very labor intensive and would have been unsustainable had it not been for the enthusiasm of contributors and, 7 months after the project began, extramural grant funding to cover the costs of coordination.

Figure 1

COVIDprotocols.org models a continuously updated collaborative clinical guideline process

(A) Collaborative guidelines process; CWA, collaborative writing application (i.e., Google Docs).

(B) COVIDprotocols.org homepage and key features.

Transparent evidentiary basis. All authors are required to provide the evidence behind their recommendations. With the understanding that early pandemic guidelines would be largely based on weak evidence, we did not formalize a grading system for each recommendation. Instead, authors referenced the quality of the evidence followed by rationales given in the text and a link to the original evidence. Eventually, because in-text literature reviews became highly unwieldy, we developed a literature review database using the commercial collaboration platform Airtable.com that is linked in-text and allows users to pull summaries, evidence grading, and the original article by topic tag.

Institutional credibility. All senior editors are from institutions with known reputations within the field, and authors are listed publicly. We explicitly discussed whether to have content directly crowdsourced from the general public and decided against this based on user-feedback about the imperative of institutional credibility.

Speed and agility

Rapid editability and deployability. Having concurrent editing avoids versioning issues and delays in sharing drafts. Comment features allow for the discussion of points of confusion without frequent meetings. The familiarity of the Google Docs interface for authors and the use of Google Docs as a CMS reduces delays in editing and publishing the site, as discussed above.

Updated dates. Each subsection is dated with the most recent update. The design of COVIDprotocols.org allows continuous, modular updates rather than needing to wait until the next release of the entire guidelines.

End-user experience

Multiplatform. The site is mobile optimized for use on tablets and smartphones. It is also available as an android web application for those with limited internet. Half of our users are mobile or tablet users.

Drop-down menus, cross-linking, and searchability. The site is organized into layered chapters with cross-linkable subheaders (Figure 1B). The left-hand bar offers chapter titles and highest-level subheaders. The landing page is a search bar to encourage people to search and find all places where a relevant term may appear. Most users (98%) access the content through the drop-down menus. However, in user feedback surveys, the ability to search for a term across multiple chapters is regularly listed as one of the most useful features of the site. Cross-linking is incredibly important to the site structure, as it allows content to be covered/edited in a single place. However, using Google Docs as a CMS does limit the graphical tools and clinical decision-making supports that can be included.

Operational-level detail. Users requested operational-level step-by-step detail for many protocols. Subpages, linking, and “hover” text allow for the inclusion of specific details that would be unwieldy in a different format, without cluttering the main content. These step-by-step instructions provide detailed guidance to frontline staff that is typically missing from the more general concepts in governmental or societal guidelines.

Multidisciplinary content. The operationalization of COVID-related protocols involves nursing, pharmacy, respiratory therapy, and facilities management. While we are a clinical resource targeted largely at clinicians, 21% of our contributors are from these disciplines, and they are integral to the creation of the content that works in real-world conditions.

Multilingual content. Much of the content of the site is translated into Spanish and available on a parallel platform that maintains all the functionality of the English site, including search.

Discussion

Medical crises can generate immediate clinical needs and poorer quality information that challenge traditional EBM methods. Digital health tools can facilitate the development and dissemination of high-quality clinical protocols and the sharing of best practices among institutions. COVIDprotocols.org’s experience has shown that CWAs can help incorporate the principles of evidence-based medicine with the NASEM standards for guideline development when rapid evaluation and deployment of clinical evidence are needed. However, true open-source clinical guidance protocols are likely still a way off: our experience reinforces that of others who have used wikis for guidance development and affirms the need for layers of vetting to ensure trustworthiness to end-users. It is also clear that an enormous investment of dedicated time and effort is required to set up, manage, and maintain the reviewing system. Further, using CWAs as content management systems for websites produces some fundamental limitations to the variety of features that can be incorporated into the site; we were unable to incorporate certain graphics, decision support tools, and push notifications for users when changes were made. Our experience appears to depart from that of prior published wiki-based clinical guideline development in several ways: (1) we used Google Docs instead of wikis as the CMS to facilitate the learning curve for editors, (2) we used a professional development firm to design software that focused on end-user experience, and (3) we explicitly limited input to a small number of credible institutions. Based on our experience, we believe that future crises may benefit from deploying similar collaborative, digitally enabled, open-access guidelines to both address the need for rapid, trustworthy guides to care and combat the problem of information integrity.