A multistakeholder development process to prioritize and translate COVID-19 health recommendations for patients, caregivers and the public. A case study of the COVID-19 recommendation map.

BACKGROUND AND OBJECTIVES: To develop a digital communication tool to improve the implementation of up-to-date COVID-19 recommendations. Specifically, to improve patient, caregiver and public understanding of healthcare recommendations on prevention, diagnoses and treatment.
METHODS: Multi-stakeholder engagement design. In conjunction with the COVID-19 Recommendations and Gateway to Contextualization RecMap, we co-developed a stakeholder prioritization, drafting and editing process to enhance guideline communication and understanding.
RESULTS: This paper presents the multi-stakeholder development process with three distinct plain language recommendation formats: formal recommendation, good practice statement, and additional guidance. Our case study of COVID-19 plain language recommendations PLRs addresses both public health interventions (e.g., vaccination, face masks) and clinical interventions (e.g., home pulse oximetry).
CONCLUSION: This paper presents a novel approach to engaging stakeholders in improving the communication and understanding of published guidelines during the COVID-19 pandemic.

The uncertainties that have accompanied the COVID-19 pandemic have accelerated advancements in science and increased public demand for accessible, easy-to-read health advice.

We developed a digital tool to communicate, in plain language, up-to-date COVID-19 recommendations to enhance accessibility to COVID-19 health information.

We report on our structured, multi-stakeholder, plain language development process for formal recommendations, good practice statements and additional guidance.

We include discussion based on our plain language recommendation case studies: COVID-19 vaccines, face masks, physical distancing, school recommendations, and home pulse oximetry; recommendations.

Introduction

The COVID-19 pandemic has led to an unprecedented number of public health guidelines [1,2].

Television, electronic press, news websites, and social media interpret these guidelines and report this as health information [3]. Patients, caregivers, and members of the public have had to process this information and make informed decisions about their health, often without the traditional direct support of healthcare professionals.

How can I protect myself against COVID-19? How can I prevent transmission? Should I get vaccinated? Can I safely send my children to school?

The world health organization (WHO), Centers for Disease Control and Prevention, Public Health Agency of Canada and other guideline development organizations have published recommendations to guide healthcare professionals. The contextualization and plain language translation of these recommendations could enable the public to better understand their health and healthcare options as well as create a more meaningful patient-physician relationship. In fact, research has shown that patients and the public think that accessible recommendations could educate, advise, reassure, promote self-care and prepare patients for a visit to a healthcare professional [4].

A recommendation is an actionable statement about the choice between two or more management options (sometimes including no action) in a specific population or setting that result from methods and processes that are deemed trustworthy [[5], [6], [7]]. ‘Trustworthy’ guidelines provide recommendations that are supported with rigorous systematic reviews with an explicit development process, a deliberative evidence to decision approach and declaration and handling of competing interests [6,8]. The eCOVID-19 RecMap project has identified more than 4,000 COVID-19-relevant actionable statements [5,9]. These statements include recommendations, good practice statements, additional guidance, and implementation considerations, tools and tips [5]. Most statements have targeted health professionals as the end user. Of note, new guideline development methods promote the engagement of patients, the public and other stakeholders in the guideline development and implementation processes to include public perspectives [10]. However, the end product of the guideline development process, i.e., the actual recommendations, need to be made accessible and understandable to a range of public stakeholders.

We created a digital communication tool to be accessed in web browsers without needing to be downloaded. Successful digital initiatives need to be co-developed with ‘users’, and this means partnering and engaging citizens and addresses specific ‘user’ contexts, culture and behaviors within the ongoing tool development process [11].

Plain language summaries are defined as easy-to-read, balanced statements that are derived from scientific research or guidelines [5]. The Campbell and Cochrane Collaborations have developed and instituted plain language summaries to effectively summarize the results of systematic reviews. The Guidelines International Network (GIN) Public Toolkit has emerged as a leading guide for preparing patient summaries [12]. Our reported reformatted and digitalized plain language recommendations aim to give the public access to recent, reliable and educational healthcare information [13]. This paper reports on enhancements to the GRADE plain language recommendation template and case studies that use a multi-stakeholder plain language development process focusing on prioritized COVID-19 recommendations.

Methods

Our plain language recommendations (PLRs) are derived from selected, up-to-date, and critically appraised published guidelines from the COVID19 Recommendations and Gateway to Contextualization RecMap [9]. We developed an administration table (see Appendix 1) for each of our selected PLR topics including the original guideline citation, the guideline organization, a RecMap link to the underlying evidence, rationale and the original AGREE II appraised published guideline. Our methods built on the Guideline International Network public guideline development approach [14] and the UK DISCERN checklist for authors of consumer health information [15]. The eCOVID project partnered with the Cochrane Consumer Network. We refined our methods using a series of 32 COVID-19 related plain language recommendations addressing the following clinical and public health topics: for example, vaccinations, usage of face masks, physical distancing in long-term care, breastfeeding guidance, international travel, and home pulse oximetry.

Multi-stakeholder PLR prioritization and development methods

We developed a multi-stakeholder engagement method for the prioritization and plain language drafting of our COVID-19 PLRs. Stakeholders involved in the prioritization process included undergraduate science students, members of the public, clinicians, guideline methodologists, and academicians. Our team prioritized COVID-19 topics for plain language translation using three criteria: (1) availability of recommendations that included quality appraisal on the COVID19 RecMap (2) of interest to stakeholders as judged by the COVID-19 RecMap executive team, and (3) interest to citizen stakeholders as judged through multi-stakeholder implementation team discussion. We enhanced our GRADE plain language recommendation template following the UK DISCERN Checklist to guide the development of our plain language summaries [15]. This checklist provided a valid and reliable way to improve the quality of written consumer information about treatment choices. Section 1 covers reliability and certainty, Section 2 covers the quality of the information and Section 3 covers an overall rating.

We assigned selected guideline statements to trained PLR drafters (undergraduate health science university students and other volunteers with a health science background) who completed the first draft by applying the original GRADE PLR template [16] with instructions and following the plain language training manual. As part of the plain language translation process, drafters were instructed to utilize a machine learning programmed editor called “Hemingway Editor” [17] to detect complex sentences, unnecessary adverbs, and to obtain a grade level estimate for their writing sample. We also instructed drafters to use plain language dictionaries and glossaries, including the Michigan Plain Language Medical Dictionary and GET IT Glossary. During the drafting of the initial Plain Language Recommendations, we determined that writing at a grade 11 level provided easy-to-read accessible PLRs that did not lose their intended meaning. A PLR checker, graduate student level, then verified the work using the same process, and resolved any discrepancies with the drafter.

A clinical expert then verified the translation for nuance, balance, and medical accuracy. Finally, as part of our feedback loop, a trained public member (citizen editor) verified the translation for succinctness, readability, and understandability for a public audience.

This process was facilitated using a centralized PLR drafting platform (Google Sheet at docs.google.com) that could be accessed by all drafters, reviewers and editors. Our final step, verification and linking the digital PLR to the RecMap, involved using a trained and standardized staff person and supporting medical editor. These team members checked the draft for version fidelity relative to original recommendations, content and plain language details before uploading the content into a customized GRADEpro module (COVID-19 Recommendations and Gateway to Contextualization Platform, Evidence Prime). The PLR was then added to the eCOVID-19 RecMap alongside the original recommendation statement, guideline details, AGREE II score, and supplementary information including evidence to decision and summary of findings tables. See Figure 1 for the full multi stakeholder process flow diagram (See Fig. 1 below).

Fig. 1

COVID-19 plain language prioritisation and development process flow diagram.

Results

Our team (14 undergraduate students and volunteers, six clinicians, four guideline developers four members of the public, and three research assistants) completed 32 digital COVID-19 plain language recommendations for patients, caregivers and the public [See Fig. 2 and RecMap (https://covid19.recmap.org/recommendations?plainLanguageSummaryPresent=T)]. Using the UK DISCERN Checklist, we report on the reliability, certainty of the original recommendation, and the fidelity of the patient information for our case series GRADE PLRs. The original GRADE PLR is described in our companion paper [16].

Fig. 2

GRADE Plain Language Recommendation Template with Pfizer Bio-NTech Vaccine Example (see web link: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_vecolpani_gmail_com_0_816885f8-63a8-4a56-b9e7-d2b48dd4dde1_c40a86df-5431-4a96-80ce-388fe9b476d6).

During the ongoing development process, we received a CIHR grant to optimize and conduct three randomized controlled trials on our PLR tool. The team worked with the Cochrane Consumer Group, youth partners and advisors from Sick Kids and the pediatric parent advisory groups affiliated with the Alberta Research Centre for Health Evidence and Translating Evidence in Child Health to Enhance Outcomes research groups to optimize the digital layout of the tool.

Reliability of the information

Our multi-stakeholder process developed and assessed the GRADE plain language recommendations summaries. Within the RecMap we included a description of the aim, development process and relevance of our plain language recommendations: “PLR are easy-to- read summaries of published and quality-checked recommendations. PLR are balanced statements that include an explanation of the recommendation, what it means for patients and the public, and the guideline source. These PLR are derived from leading guideline development organizations and include a link to the underlying evidence and their rationale.”

Our multi-stakeholder development process identified a few language improvements (i.e., Talk with vs. Speak to) for the original PLR template. Working originally with two citizen partners (MS, SS), we recognized the importance of adding the source of the recommendation and the publication date of the original recommendation in the GRADE PLR. For example, WHO Pfizer BioNTech recommendation published 2021.

We linked each PLR to the parent eCOVID RecMap which provides background details on our guideline producing organizations (e.g., WHO, CDC, PHAC), a description of the evidence-based nature of the project and an overall research disclaimer for the eCOVID project [9].

Type of actionable health statement

Our multi-stakeholder team selected topics that included (formal) recommendations (e.g., WHO Pfizer BioNTech vaccine and WHO Home Pulse Oximetry) (Table 1). But our team also selected two other types of actionable health statements: good practice statements (e.g., PHAC mRNA vaccines) and additional guidance statements (e.g., CDC Breastfeeding). These latter two ‘types’ of actionable statements occurred where guideline panels identified an important issue, described the population and intervention options, but the published guideline lacked information needed for formal ratings of certainty of evidence.

Table 1

Actionable Statement Type, Definition and Example

Actionable statement type	Technical definition	Description	Our PLR examples
[Formal] Recomme ndation [5]	A formal recommendation is an actionable statement about the choice between two or more management or policy options (interventions) in a specific population and, if relevant, in a specific setting. Alternative option(s) (i.e., comparator(s)) should be specified in the recommendation if they are not self-evident. These statements are the results of a formal deliberation process, and contain an explicit and direct link to the bodies of evidence resulting from a systematic literature search and appraisal process.The statement should have a clear direction (for or against the options) and strength (e.g., strong or conditional) to support the options. Ideally, the strength of recommendation and the certainty of supporting evidence are explicitly stated for all factors that were considered when making the recommendation (e.g., intervention effects, test accuracy, values or cost, etc). A formal recommendation should be supported by a deliberative, structured and transparent development process. An explicit and direct link to the evidence is provided, preferably in the form of an evidence profile. In particular, it should be supported by systematic reviews or health technology assessments for the factors that determine its direction and strength.	A recommendation is a statement that can help people make a choice between two ormore options about their health care. Recommendations are created using an organized process that includes research evidence and other important considerations like values, feasibility and costs.A recommendation can be strong or conditional. When a recommendation is strong, most people will want to follow it. When a recommendation is conditional, the majority of people want to follow it, but they may want to talk with their health care professional first.	WHO Pfizer BioNTech vaccine recommend ation PLR [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_vecolpani_gmail_com_0_816885f8-63a8-4a56-b9e7-d2b48dd4dde1_c40a86df-5431-4a96-80ce-388fe9b476d6]WHO Home Pulse Oximetry recommendation PLR [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_nasirze_mcmaster_ca_0_8c9738dd-bb8b-4314-a893-204ab352e6db_324e4065-7b8a-476d-8bcd-7aec49299a8e]WHOLTCF Physical Distancing Recommendation PLR [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_fclement_ucalgary_ca_0_ad7e7af1-4043-4ebb-9a6f-1dc1da43d6db_31b08b00-62dd-4477-bb95-2cdc48ec8c72?_k=yyvrlr]WHO Face Masks (Unvaccinated) Recommendation PLR [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_charifazemouri_hotmail_com_0_641fb838-1b13-4c55-aeb7-9c1070765f3c_322194ad-ef5a-4e88-b1a1-143bf0258f75?_k=z8oyq0]
Good Practice Statement (GPS) [5,18]	A Good Practice Statement (GPS) is an actionable clear guidance statement. They describe the population and intervention options and, if appropriate, comparator components of the recommendation.Good practice statements are not appropriate for formal recommendation ratings of certainty of evidence or strength of the recommendation. Development of good practice statements should adhere to five principles and pass the following question for evaluation (updated from (10), Dewidar et al., manuscript in preparation):1. Is collecting and summarizing the evidence a poor use of a guideline panel's limited time and energy (opportunity cost is large)?2. Is the message necessary in regard to health care practice?3. After considering all relevant outcomes and potential downstream consequences, does implementing the GPS result in a large net positive consequence?4. Is there a well-documented clear and explicit rationale connecting the indirect evidence?5. Is the statement clear and actionable?	A good practice statement can help people make a choice about their health care. Good practice statements should be created using an organized process, but do not undergo a systematic approach to reviewing all the evidence.	PHAC mRNA Vaccine [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_matthm9_mcmaster_ca_0_b8c787c1-42eb-4e5c-bf81-ed294defb33c_c00e2ad1-2805-4ef2-9369-7a003f4af11c]
Additional Guidance	Additional guidance are ‘informal’ actionable statements that do not fulfill the taxonomy criteria for formal recommendations or Good Practice Statements (GPS) in terms of methods and/or process.They often lack a clear link to all the relevant evidence, or a systematic review, or a Health Technology Assessment (HTA) process. They also lack a clear description of the judicial process of moving from the evidence to the recommendation, including a lack of the process of how consensus was reached. These ‘informal’ actionable statements should be used with caution.	Additional guidance are statements that can help people make choices about their health care. These statements might come from research studies or evidence, but there is information missing on the clear link between the studies/evidence, and the judgment process about them. Because of this missing information, these statements are considered less credible. Additional guidance includes informally developed recommendations and implementation tools and tips as well as other statements (see informal recommendations).	CDC Breastfeeding PLR [URL: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_charifazemouri_hotmail_com_0_cb35fdff-fd61-45dd-9627-d327d6122fd4_d4392caa-f8ab-462a-802e-5df6d873a6de]

Quality of the information

We also distinguished the quality or certainty of the information. Each PLR began with a clear statement related to the recommendation ‘type’; for example, Recommendation, or GPS of Additional Guidance and generic statement (see Table 2 ). Within each PLR, we also described the intervention and the benefits, harms or risks of each intervention. Although suggested by the UK DISCERN Checklist, we were not able to explicitly describe the impact of the intervention on overall quality of life because this information was not available in the original guideline publications. We clearly described intervention choices when available. Our final section focused on “What does it mean for you?” This section included explicit suggestions on what you may wish to talk with your health care professional about. Some of the guidelines included detailed discussion points for shared decision making; for example, how to assess home pulse oximetry readings. Other guidelines did not always provide explicit discussion point information; for example, WHO Moderna vaccine.

Table 2

Plain Language Generic Statement relating to the certainty of the actionable statement

Formal recommendations	Good practice statements	Additional guidelines recommendations
A recommendation can be strong or conditional. When a recommendation is strong, most people will want to follow it. When a recommendation is conditional, the majority of people want to follow it, but they may want to talk with their health care professional first.	A good practice statement can help people make a choice about their health care. Good practice statements are created using a specific process, but without the step of systematic reviewing of all of the evidence.	Additional guidance are informal statements that might be able to help people make choices about their health and health care. These statements might come from studies, but there is information missing concerning the studies and development process.

Interactive summary of findings table

A “drop down” interactive summary of findings table was added to the PLR template for formal recommendations at a later stage allowing users to have the opportunity to explore the benefits and harms using numbers. For four PLRs, our team created simple plain language statements to summarize each outcome within the table. We determined that it was important to communicate both the magnitude of the effect and the certainty of the evidence. This aligned with GRADE guideline 26 which provides specific terminology for communicating both elements in GRADE tables [19]. For example, to describe an outcome with a large effect size and high certainty of evidence, terms such as “large decrease” and “very certain” were used, respectively. Emoticons and plus signs were also incorporated to enhance the understanding of the certainty in the evidence. Our CDC Pfizer BioNTech Vaccine example can be found using this web link: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_f_cruciani_deplazio_it_0_a2c6674c-5bd0-459c-b230-be5552544296_cd92e290-2465-4294-b017-0bcb96ed5a88.

Discussion

Plain language recommendations have been high demand during the COVID-19 pandemic [1]. This paperreports on a multi-stakeholder priority setting and translation process for digital COVID-19 plain language recommendations and the creation GRADE PLR templates for ‘good practice statements’ and ‘additional guidance’. Our multi-stakeholder team recognized that public health and practice recommendations can create uncertainty and a strain on citizens [20]. Therefore, our plain language recommendations attempted to balance between education and confidence (certainty of information) for patients [21], caregivers and the public. As part of our research, we are now optimizing the digital PLR tool in collaboration with consumer groups. We will conduct three randomized controlled trials to test the effectiveness of the new digital tool to improve adolescent, parent, and public understanding of COVID-19 recommendations. We have also begun translating the digital PLR into several languages, and research on this element is ongoing.

Our multi-stakeholder development process identified a few language improvements (i.e., ‘Talk with’ vs. ‘Speak to’) for the original PLR template. DISCERN [21]and our citizens suggested adding in the publication year with the recommendation to enhance reliability of the information. Conceptually, the most important innovation is the development of generic statements that distinguished between three recommendation types: one for ‘formal recommendations’ that was in the original PLR [15] one for ‘good practice statements’ and one for ‘additional guidance’. In the drafting process, we noted that the original guidelines often provided only limited, if any, information on what the recommendation would mean for a patient or the public. DISCERN [21] identifies this as a key element in consumer health information and so we thus identify this as an important area for improvement in guideline development standards.

Our administration table provided links to the most recent guideline and distinguished the recommendation ‘type’ for student drafters. To ensure accurate depictions of the existing guidelines, we found an annotated template was necessary to build consistency in our sections, provide direct links to extracted data and original publications, and to prevent the inclusion of new scientific evidence. An example of this was the WHO Oxford-AstraZeneca vaccine recommendation, where we were careful not to introduce new UK and Israel observation post-vaccination data. The training manual and annotated template provided the multi-stakeholder team with clear expectations and a shared platform for engagement. Our drafting process benefited from the linked pre-existing abstractions in the COVID-19. Recommendations and Gateway to Contextualization RecMap [9].

The eCOVID RecMap has identified over 4,000 actionable statements for providers, policymakers, and the public. This open access RecMap classified the ‘type’ of recommendation [5] and provided quality AGREE II appraisals [8]. Our multi-stakeholder team used the map to identify priority recommendations for patients and the public, while recognizing that these plain language recs may also be useful for providers and policymakers. We initially focused our attention on ‘formal’ recommendations, however, citizens on our team identified topics that involved other recommendation ‘types’: good practice statements and additional guidance. In order to include these novel actionable statements, and distinguish the certainty of information, we created two other PLR formats with appropriate interpretive generic plain language translations.

During our development process, we recognized that our online PLRs benefited from staying closely linked to the COVID-19 Recommendations and Gateway Contextualization RecMap. The online link provided the public with our PLR definition and process description, information in evidence to decision extractions, Appraisal of Guidelines for Research and Evaluation II (AGREE II) appraisals, plain language glossaries, and background details on the eCOVID-19 RecMap. We observed that publishing the PLR as stand-alone products would require additional work to ensure there is a clear reference to the original source guideline, version and publication dates, and information on how the PLRs were developed as per the DISCERN reporting guidelines [21]. We encourage guidelines developers to include GRADE PLR for their guidelines, including more information on “What does it mean for you?” We will also seek additional support for multilingual publication of our ongoing series of COVID-19 PLRs.

Strengths and weaknesses

The strengths of our development process included having the tool link directly to a digital guideline appraisal platform with sustained engagement of various stakeholders. Our digital PLR were amendable to timely updates and optimization; for example, our consumer groups have optimized tool design. The limitations of our approach included the variable amount of training needed to support public members and citizens and the importance of including staff checkers to verify the consistency of the health information. Another consumer challenge that presents in the final component of our digital tool, is the high degree of health and digital literacy required to navigate and interpret the interactive summary of findings table data.

Implications for public health and practice

Our series of multi-stakeholder translated COVID-19 PLRs demonstrate easy-to-read summaries of high quality published guidelines. Priority selection criteria and grounding the PLR in original guidelines set the stage for GRADE evidence to decision concepts. These concepts include judging the benefits and harms of an intervention, certainty of the evidence, and values and preferences’ relevance for the public [22]. Working directly with multi-stakeholders including students and members of the public may improve access along with our use of plain language medical glossaries. Using an annotated PLR template and an interdisciplinary team contributed to the consistency of our initial series of 32 COVID-19 PLRs.

Implications for research

The multi-stakeholder development has produced a series a digital PLR tool for the public. Our team has received CIHR funding to optimize and conduct three clinical trials with adolescents, parents and the general public. These trials will evaluate the effectiveness of the digital tool to improve understanding of COVID-19 recommendations compared to a generic summary of the technical guidelines. Subsequent research will be needed to evaluate if the tool can improve patient and public behavior and important outcomes. Finally, research on the impact of translating the tool into other languages is needed.

Conclusions

We used and enhanced the GRADE PLR template, and developed a multi-stakeholder process to create digital plain language COVID-19 recommendations that were reproducible and transferable to other recommendations. Plain language, easy-to-read summaries have long played an important role in presenting systematic review results to the public. Here we introduce our digital plain language COVID-19 recommendation tool poised to communicate COVID-19 guidelines to patients, caregivers, and the public. As guideline developers formalize their development processes, we anticipate more formal and good practice statements. Furthermore, we believe that our plain language translations could also enable rapid and consistent communication to providers, policymakers and other stakeholders.

CRediT authorship contribution statement

K.P., M.S., M.M., N.S., O.M., H.J.S. conceived and drafted the manuscript, A.S., R.H., Y.T., N.D., S.S. contributed to the early draft and P.L.R., O.M., M.S., K.B., T.C., J.L.M., E.A., B.D., A.I., L.R., V.W., M.K., M.X.R., I.D.F., T.L., A.Q., T.L., P.T., A.M. contributed to P.L.R. development process. All authors provided revisions and approved the submission of the manuscript.