Lessons Learned from Developing Plain Language Summaries of Research Studies.

BACKGROUND: Plain language translation may facilitate the public's ability to understand and use results of scientific research. Brief description of activity: This article describes the Patient-Centered Outcomes Research Institute's (PCORI) approach to and lessons learned from developing plain language summaries of PCORI-funded research for the lay public. IMPLEMENTATION: We developed and tested a standard template for the summaries, incorporating feedback in the template design from focus groups with members of the public. Between February 2017 and March 2020, we completed translation of 272 plain language summaries of PCORI-funded studies, covering topics including cardiovascular disease, obesity, cancer, mental health, asthma, HIV/AIDS, and comparative effectiveness research methods.
RESULTS: Templates use a question-and-answer format, with sections on the rationale, methods, results, limitations, and how the research will help inform decisions. In addition to feedback on template heading wording and order, focus group participants stressed the importance of establishing relevance and conveying credibility and limitations. LESSONS LEARNED: Lessons learned relate to supporting consistency across individual summaries, carefully prioritizing content to include in the summaries, and balancing plain language and reading level with precision. These lessons learned from template development and implementation may be useful to other organizations or publishers contemplating similar efforts to make scientific research results more accessible. [HLRP: Health Literacy Research and Practice. 2021;5(2):e155-e161.] Plain Language Summary: The Patient-Centered Outcomes Research Institute (PCORI) funds comparative effectiveness research. This research compares the benefits and harms of two or more health care choices. In this article, we describe lessons learned from PCORI's efforts to develop plain language summaries of results from the research it funds. These lessons may help other organizations that want to share research results in plain language.

Increasingly, scientists are encouraged to translate scientific concepts and research findings into plain language for the lay public and people without a scientific research background. This trend is particularly prominent in medical research, reflecting a growing focus on patient-centeredness and on the ethical responsibility to report findings to the public (European Commission, 2014; World Medical Association, 2013). The call for user-friendly presentations of research findings is also pragmatic: to put evidence into practice, health professionals as well as the public need to understand key messages of research findings (Buljan et al., 2018).

Evidence summaries can help communicate key messages effectively and efficiently to readers (Alderdice et al., 2016). However, scientists struggle to communicate research concepts and findings in accessible ways (Gagliardi et al., 2016; Kurtzman & Greene, 2016; Maguire & Clarke, 2014). A plain language summary of a research study is a concise synthesis of the study written in a way that the lay public would understand (Whiting et al., 2018). Best practices for summarizing research in plain language include reducing jargon and defining technical terms; using short sentences and unifying points within a paragraph; testing with intended audiences for intended comprehension; and asssessing reading level, usually via a metric that considers the number of syllables per word and number of words per sentence (Centers for Disease Control and Prevention, 2015; Centers for Medicare & Medicaid Services, 2012; Office of Disease Prevention and Health Promotion, 2016).

Biomedical journals increasingly have committed to presenting findings for lay audiences. For example, PLOS Medicine author summaries feature short statements for non-expert readers about why the study was done, what the researchers did and found, and what the findings mean (PLOS Medicine, 2019). Other journals capture and highlight specific information in accessible formats. The British Medical Journal, for example, includes a summary box providing a “thumbnail sketch” of what an article adds to the literature (BMJ, 2019). In addition, perhaps the most well-studied plain language summaries are from the Cochrane Collaboration's systematic reviews (Alderdice et al., 2016, Buljan et al., 2018; Maguire & Clarke, 2014; Whiting et al., 2018). These short summaries present findings from systematic reviews of 40 or more pages on a wide range of clinical topics. The format of Cochrane's plain language summary has evolved over time with the goals of improving clarity and usefulness for the public, for example by adding information about the benefits and harms of interventions and the quality or strength of evidence (Santesso et al., 2015).

Over the past several years, the Patient-Centered Outcomes Research Institute (PCORI) has undertaken a substantial new effort to translate research findings into plain language summaries for the lay public. PCORI's effort derives from its commitment to transparency—the presentation of all results, regardless of the outcome of the study—and from its charge to make findings available in a manner that is comprehensible and useful in informing health care decisions. Although some aspects of PCORI's plain language summaries are specific to the organization's mission and mandate, many elements of the summary translation and publication process may be useful to health communication professionals and other organizations or publishers contemplating similar efforts to make scientific research results more accessible. To inform these efforts, this article summarizes lessoned learned from PCORI's efforts to develop plain language summaries.

Overview of PCORI and its Public Reporting Efforts

Authorized by the United States Congress in 2010, PCORI is an independent, nongovernmental organization that funds research to help patients and those who care for them—along with clinicians, purchasers, and policy makers—make better-informed decisions about the health care choices they face. PCORI funds patient-centered comparative effectiveness research on a wide range of topics, requiring that researchers engage patients and other health care stakeholders throughout the research process to better enable studies to address their concerns (Patient-Centered Outcomes Research Institute, 2017).

PCORI's authorizing legislation requires that the institute make research findings available to the public quickly and in a way that is useful to patients, clinicians, and the public. In addition, these materials must discuss considerations specific to certain subpopulations, risk factors, and comorbidities; describe research limitations; and identify what future research may be needed, as appropriate (Patient Protection and Affordable Care Act, 2010).

To fulfill this statutory mandate, PCORI's Board of Governors adopted a process for releasing research findings. Within 90 days of PCORI's mandated peer review of study findings and formal acceptance of the final research report (Broitman et al., 2019), PCORI posts two summaries of the study's results to its website: one for patients and the public, with readability at a sixth to eighth grade level, and one for professionals. The summaries are kept short to promote readability and to ensure that they will not include so much detail as to interfere with publication opportunities. All summaries must be tested for comprehensibility with intended audiences. After 1 year, PCORI posts the entire final research report on its website.

Implementation

PCORI's translation efforts are supported by its PCOR Translation Center (hereafter, the Translation Center). Building on best practices for plain language, the Translation Center developed an approach for creating plain language summaries of all completed research studies, in satisfaction of PCORI's legislative requirements. This approach included: developing, testing, and finalizing a template for the plain language summaries; establishing a workflow that is efficient, incorporates all required expertise, and accords with all legislative requirements; and implementing the translation process and modifying based on lessons learned.

Template Development

A main initial challenge for translation of PCORI's research findings was to provide a consistent approach for presenting individual comparative effectiveness research studies that have different designs, focus on different health care topics, and employ different research methods. Applying best practices for plain language and building on previous research exploring the public's views on medical evidence (Carman et al., 2010, 2016), we developed templates that identified what information should be included in summaries and how it should be presented. To engage readers in the content, templates use a question-and-answer format, with sections on the rationale, methods, results, limitations, and how the research will help inform decisions. The template headings refer to high-level elements that are common across individual studies, using wording general enough to apply across content areas.

We tested the templates with their intended audiences—primarily members of the public (called consumers for the purposes of this article), but also clinicians and others who would be in a position to use plain language summaries in their work with patients and others. After receiving Institutional Review Board approval from the American Institutes for Research, we conducted six focus groups in English and Spanish with 52 consumers representing three geographic regions and a range of race, ethnicity, age, gender, and educational attainment (Table ). We also conducted 1-hour phone interviews with 10 doctors and 2 nurse practitioners, and 15 stakeholders representing 3 patient and consumer advocacy organizations, 4 professional societies, 2 health systems, and 2 payer and employer organizations. In addition, four members of the PCORI Board of Governors participated in the interviews.

Using a purposeful sampling approach, we used market research firms to recruit consumers and clinicians. For consumer focus groups, the firms recruited adults who represented diversity in gender, race, and ethnicity and who had obtained an educational level of some college or less. Consumers were excluded if they worked in a clinical setting or took part in a research study in the past 6 months. For clinician interviews, the firm recruited practicing clinicians from specialties relevant to PCORI research (i.e., primary care, oncology, cardiology, neurology, psychiatry). For stakeholder interviews, we worked with PCORI's Engagement Department to create a list of 29 PCORI stakeholders who had previously partnered with PCORI and had used, or were likely to use, the summaries posted on PCORI's website and conducted interviews with those who responded and agreed to participate.

Interviews and focus groups concentrated on what information participants expected or wanted to know about research findings from a specific single study, information sources for research, and participants' overall reactions to the proposed formats. During the interviews and focus groups, moderators and interviewers who had experience in qualitative research and cognitive testing used semi-structured interview protocols. Opening questions focused on what information participants expected or wanted to know about research findings from specific single study. Then, we presented an example summary using the template and asked participants about their overall reactions, feedback on the wording and order of the headings, and the usefulness of the information. We used two existing PCORI projects as examples to assess whether problems arose from the content of the research study or the template itself and changed the order in which the example projects were presented by group or interview. Participants were compensated for their time.

We audio-recorded and had all focus groups and interviews professionally transcribed. A certified Spanish language transcriptionist translated and transcribed recordings into English for analysis. We uploaded transcripts into the NVivo 11.0 qualitative research software (QSR International) for review and coding. A team of four analysts read and applied a priori codes based on the main topic areas of the interview protocol (e.g., expectations, reactions to headings, usefulness of information). One analyst coded and summarized findings for each audience: English-speaking consumers, Spanish-speaking consumers, clinicians, and stakeholders. Then, the lead researcher (M.M.) and team of analysts summarized findings across audiences and made recommendations for revising the templates. The four analysts met weekly to discuss reactions and emerging ideas and ensured codes were applied consistently.

A Technical Expert Panel also provided input on the templates. Its 15 members included patients, practicing clinicians, and experts in health communication, health literacy, and evidence review.

Developing a Workflow

Drawing on Lean Six Sigma quality and process improvement tools, we developed work processes to efficiently manage and organize the workflow for activities and reviews from the beginning to end of plain language translation. The processes account for the volume of production, the expertise required for each abstract, and the timing of all steps in the process, highlighting places where flexibility in timing can be accommodated. Over the course of the project, the translation process has been modified to improve efficiency while maintaining quality. Below, we describe the current process.

Writers develop the summary based on a draft report submitted to peer review with input from health communication and subject matter experts. Communication experts work closely with writers to identify key messages from the report and translate these into plain language. Subject matter experts assess scientific accuracy and ensure that the translations maintain fidelity to the report. Then, the summaries undergo cognitive testing with four to six individuals drawn from a pool of panelists who provide feedback on the overall clarity and quality of each section and reactions to specific content related to medical or methodology terms that may be unfamiliar, such as cardiac tamponade, transforming variables, or sleep impairment. Currently the panel includes about 175 individuals, including patients, caregivers, clinicians, researchers, and representatives of advocacy organizations, health systems, and payers. Panelists are compensated for their review. An analyst summarizes the findings from testing to highlight areas requiring revision.

Once PCORI's peer review team accepts the awardee's research report as final, the writer revises the summary based on any changes made in response to peer review comments, and also incorporates feedback from the testing. Then, PCORI's Program Officer and the Principal Investigator review for accuracy. Final summaries undergo copy-edit and approval before being posted on PCORI's website within the 90-day period.

Finally, the Translation Center produces an audio file of the English summary and a Spanish version of the summary available for download as a PDF file. Audio recording experts review the summaries and suggest edits to make the summaries more comprehensible when read out loud in English. Then a voice-over professional records the summaries. For Spanish translations, a certified translator translates the abstract from English to Spanish. A native speaker then reviews the translation for accuracy and plain language; the translator and native speaker then meet to resolve discrepancies.

Length of Time for Implementation

Between February 2017 and March 2020, we completed 272 plain language summaries of PCORI-funded studies, covering topics including cardiovascular disease, obesity, cancer, mental health, asthma, HIV/AIDS, and comparative effectiveness research methods.

Results

This section describes findings from audience testing that influenced template development as well as other lessons learned from implementation.

Final Template

Final template designs for plain language summaries reflect input from the technical expert panel and from testing. The final template consists of six sections that provide context for the research, report the results, and describe the study:

What was the research about?

What were the results?

Who was in the study?

What did the research team do?

What were the limits of the study?

How can people use the results?

Below, we summarize findings from audience testing that influenced decisions made for the content and organization of the final plain language template. Overall, reactions to the template were consistent across testing participant type.

Participants confirmed that the question-and-answer format was straightforward and easy to understand. The headings were “precise and to the point,” answering questions that people wanted to know. A clinician commented, “They're focusing on what's most important to patients.” When testing alternate wording of templates, most consumer participants indicated a preference for simple wording, for example preferring “What were the results” to “What did the research team learn.”

In contrast to the order of a traditional research abstract, the template emphasizes results before methods. Consumer participants preferred to read the results and conclusions drawn from the study right away—before deciding to read other information, like the study's methods.

Consumer participants wanted to be able to evaluate easily whether the study and its results were of interest and applied to them personally. In response to these preferences, we include in the summary title, whenever possible, details about the health condition, interventions, outcomes assessed, and relevant demographics of study participants. Summary text also includes race, ethnicity, age, gender, geographic location, and other participant demographic characteristics.

Consumer participants wanted to know if the funder or investigators had a vested interest in the outcomes. As one put it, it is important to know if the research studies are “independent research versus [the] Lipton Tea Company that conducted it to tell you that coffee was bad for you.” We highlight patients' and patient organizations' participation on the research team in the “what did the research team do” section to emphasize their interest in the study, and to allay concerns about motivations for study design. As one participant said, “I think you've got to play up that this wasn't some researcher in a white lab coat in some ivory tower who's decided this.” PCORI also discloses researchers' conflicts of interest on the summary webpage.

Consumers appreciated that the summary included a limitations section, describing it as “honest.” As one consumer summarized, “Transparency. It's a synonym of honesty. They don't just tell you about the good things [translated from Spanish].”

Implementation Lessons Learned

The experience gained through writing these summaries brought to light additional lessons learned that may be relevant for those producing plain language summaries.

To support consistency in preparing individual summaries and to assure that each summary meets the mandated requirements for PCORI summaries, such as length and grade level, the Translation Center developed guidance for writers and reviewers. This guidance included the content to include in each section of the template, how to present specific concepts, and how to address new issues as they emerge. The Translation Center also created and continues to update a common lexicon for terms related to health (e.g., specific health conditions), interventions (e.g., decision aids, community health workers), methods (e.g., randomization), and study-related situations or limitations (e.g., implications of different study designs, generalizability).

Given the word limits PCORI prescribes for its summaries, writers learned to prioritize what content to include while still telling a coherent story. Summaries usually report only on the study's comparative aim and prioritize results, details about the interventions, participant demographics, future research needs, and limitations. Information to provide context is also important but must be conveyed in one or two sentences.

Although best practices for plain language health materials suggest specifying an action for a reader to take, writers must be cautious about using any language that would recommend treatments or a course of action from a single study. Ultimately, clinical decisions need to be considered within the limitations of the study and the larger body of evidence around a topic, as well as patients' values and preferences.

The plain language summaries for completed projects have an average reading level of 8.1, with a range of 6.8 to 8.5, using the Flesch-Kincaid assessment tool in Microsoft Word. A challenge in writing plain language summaries is to avoid oversimplifying terms or concepts, which can lead to content that is misleading or inaccurate, or that obscures what happened during research. For this reason, summaries may include technical terms, even though they can increase reading level. A technical term may be acceptable, for example, if health care providers often use it and patients are familiar with it. In these cases, we define the technical term on first use. For example, chemotherapy is defined on first use but then used throughout a summary to distinguish it from other types of cancer treatment.

Implications

Publishers, health research funders, and researchers themselves have demonstrated an increasing commitment to making available plain language summaries of research studies that are readily understandable by patients, caregivers, and other stakeholders. Making research results accessible has the potential to help patients and the lay public understand the role of research in health care and improves transparency and accountability.

Although the goal is simple, the process of creating plain language summaries is not. Writing in plain language requires a specific skillset that scientists may or may not have. Further, scientists may be too close to their topic to write a concise summary (Denegri & Faure, 2013). Consistency of summaries across different studies or systematic reviews may also be a challenge. For example, none of the Cochrane plain language summaries, as written by the authors of the systematic reviews, adhered to the Cochrane's standards (Jelicic Kadic et al., 2016); they varied in structure, length, details included, and use of jargon. Employing independent writers, as PCORI does, may help improve readability and consistency of plain language summaries (Kirkpatrick et al., 2017).

Producing consistent plain language summaries requires a commitment of resources. PCORI's investment has included developing a standard approach for all funded research projects and includes resources for writing, reviewing, developing audio files, and translation into Spanish plus project management and quality control. Costs per summary have decreased over time as standard language for common terms and situations have been developed.

PCORI's commitment to plain language presentation of research findings stems from the high priority the organization places on transparency in service to—and as part of its responsibility to—patients and the public. An equivalent level of investment may not be possible for all organizations; publishers and research funders will need to weigh the benefits of readability and consistency with the expense (Denegri & Faure, 2013). To the extent that other organizations can leverage PCORI's investments, we may be able to improve the feasibility of plain language translation of research findings going forward.

A range of resources, including frameworks and descriptions of best practices, currently informs the translation of findings from health care research into language accessible to patients and others. However, further research could support improvements in this field. For example, research could investigate how different audiences read and use plain language summaries for single studies and whether alternate formats or displays, such as infographics, can effectively convey study information. Google analytics on PCORI's posted summaries—nearly 29,000 views in the final quarter of 2019—demonstrate interest; studies could also document the ways in which actual and intended audiences use these summaries.

Ultimately, creating a structure and process for research summary development requires balancing precision with simplicity, completeness with brevity, and what the research team knows to be true with what the audience finds important to know. PCORI's transparency efforts reflects the complexity of producing clear summaries and the evolving understanding by researchers, funders, and publishers of how best to meet public needs.

Table 1

Demographics of Consumer Focus Group Participants

Characteristic	Location (N = 52)
	NC (n = 18)	MD (n = 16)	CA (n = 18)a
	(%)
Race/ethnicity
White	10 (55)	11 (69)	0 (0)
Black or African American	4 (22)	5 (31)	0 (0)
Hispanic or Latino/Latina	1 (6)	0 (0)	18 (100)
Multiracial, Asian, Native American	3 (17)	0 (0)	0 (0)
Sex
Female	11 (61)	6 (37)	8 (44)
Male	7 (39)	10 (63)	10 (56)
Age (years)
Average	43	53	45
Range	22–68	30–63	27–61
Education
High school graduate or less	8 (44)	10 (63)	12 (67)
Technical/vocational/certification program	2 (11)	1 (6)	2 (11)
Some college or Associate degree	8 (44)	5 (31)	4 (22)

Note. CA = California; MD = Maryland; NC = North Carolina.

Spanish-language groups.