Incorporating the Concept of Relevance in Clinical Rehabilitation Research and Its Reviews May Improve Uptake by Stakeholders.

ABSTRACT: The "relevance" of research to stakeholders is an important factor in influencing the uptake of new knowledge into practice; however, this concept is neither well defined nor routinely incorporated in clinical rehabilitation research. Developing a uniform definition, measurement standards, stakeholder engagement strategies, and guiding frameworks that bolster relevance may help incorporate the concept as a key element in research planning and design. This article presents a conceptual argument for why relevance matters, proposes a working definition, and suggests strategies for operationalizing the construct in the context of clinical rehabilitation research. We place special emphasis on the importance of promoting relevance to patients, caregivers, and clinicians and provide preliminary frameworks and innovative study designs that can assist clinical rehabilitation researchers in doing so. We argue that researchers who include a direct statement regarding why and to whom a study is relevant and who incorporate considerations of relevance throughout all phases of study design produce more useful research for patients, caregivers, and clinicians, increasing its chance of uptake into practice. Consistent consideration of relevance, particularly to nonacademic audiences, during the conceptualization, study design, presentation, and dissemination of clinical rehabilitation research may promote the uptake of findings by patients, caregivers, and providers.

The concept of “relevance” to stakeholders is not frequently or systematically addressed in rehabilitation research, nor is it well defined in clinical science more generally. In common parlance, relevance denotes something that is useful to an inquiry, provides pertinent information, or matters to an audience. Some in the research community have expressed support for adding the dimension of relevance as a component in the design and evaluation of clinical research.[1-3] The argument is that there is an increased likelihood of adoption of research findings if studies make a concerted effort to consider who the end users are and whether their interests or needs are directly addressed in the research plan.[4,5] Incorporating considerations of relevance to end users beyond academics or other researchers, and doing so throughout all phases of study design and dissemination, may enhance the appeal, utility, and uptake of research literature.

The goal of this article is to open a dialogue with researchers about the meaning of relevance and strategies for enhancing it in the context of clinical rehabilitation research (CRR). Certainly, the need to optimize research relevance is not limited to CRR; relevance should be better explored, defined, operationalized, and communicated in other areas of clinical and translational science as well. Many outside of the scientific community may question the utility of research if its application is not immediately obvious or it does not have direct implications for patients. As such, it is important that researchers articulate the relevance of their work and engage in efforts to enhance relevance throughout all phases of study design. However, the topic of enhancing relevance in clinical science requires a more comprehensive literature review and a dialogue with a broad base of stakeholders that is beyond the scope of this article. Hence, we focus on ways in which relevance can be enhanced within our area of expertise: CRR.

Specifically, in this article, we aim to equip researchers with strategies that can be used to raise awareness of relevance as a concept in CRR and guide them in incorporating stakeholder input to enhance the relevance of research for academic as well as clinical and nonacademic audiences. For context, we start by briefly presenting exploratory findings from a high-level survey of the CRR literature. We then provide perspective on why end-user relevance matters and the benefits of including it as a research design consideration. Next, we offer strategies that CRR investigators can use to enhance the relevance of their work to nonacademic end users—namely, patients, caregivers, and clinicians—without diminishing its relevance for researchers. As part of this discussion, we propose a working definition of relevance for CRR, questions that researchers can ask themselves when conceptualizing relevance, and a preliminary framework that can guide the operationalization of relevance in CRR. We end with a brief overview of innovative study designs that illuminate the relevance of research products for nonacademic end users.

By creating conversation around the importance of enhancing relevance to nonacademic stakeholders and providing suggestions for its integration and measurement in CRR, we believe that this article will be relevant to clinical rehabilitation researchers who strive to increase the reach, impact, and adoption of research findings by patients, caregivers, and clinicians.

BACKGROUND

Our recommendation that rehabilitation researchers emphasize the concept of relevance stems from our initial motivation to understand whether and how CRR authors incorporate and address considerations of relevance in systematic reviews (SRs). We believe that investigators value end-user relevance and often embed its consideration in the selection of research questions and methods. However, readers of some SRs, which aim to synthesize and disseminate knowledge, may still find themselves asking the question, “so what?” This suggests that the relevance of some SRs may be lacking but that its consideration could promote the uptake of evidence.

Accordingly, we undertook an exploratory survey of SRs in CRR to determine: how relevance is defined, contextualized, or measured; whether authors identify the audiences for whom SRs are relevant; how relevance is considered throughout the research life cycle (i.e., conception, design, execution, delivery); and whether guidelines for incorporating or appraising relevance exist. Although the questions we ask about SRs could be asked of the underlying studies they review, focusing on the higher level allowed us to take advantage of syntheses SRs already performed, covering a wider range of CRR topics with a more manageable number of publications. Based on the findings, we planned to assess whether sufficient discussion of relevance existed to warrant our own SR that could formally evaluate the broader hierarchy of CRR literature.

We searched PubMed, CINAHL, MEDLINE, Embase, Cochrane Library, PEDro, OT-Seeker, Rehabdata, Cirrie, and Web of Science using key words “rehabilitati*”; “systematic review”; “research”; “relevan*”; “utilit*”; “usability”; “usable”; “uptake”; and “impact.” We looked for SRs of CRR published since 2000 that addressed relevance or a related concept (e.g., usefulness, meaningfulness, applicability, social importance). We also explored non-SRs and gray literature that addressed the concept generally. We reviewed more than 100 articles and their reference sections.

We found no overt discussion of relevance in SRs of CRR. When the term “relevant” was mentioned, it was consistently with respect to the identification of relevant articles to be included or excluded in an SR. The term was not used in the context of study relevance to stakeholders. Ultimately, we determined that the discussion of relevance in SRs of CRR was not sufficiently robust to conduct a formal SR of our own. Some authors acknowledged the lack of stakeholder relevance as a barrier to SR uptake and advocated for more attention to the issue.[2,4-18]

The lack of focus on stakeholder relevance in SRs of CRR suggests that (1) relevance is not articulated as a priority and (2) because it is not articulated, there is no evidence that it is a priority. It may be that the underlying studies that SRs review do not address relevance, pointing to a gap in CRR more generally. We also acknowledge that SRs and the studies that comprise them should not be faulted for not addressing relevance, given that no existing guidelines or requirements stipulate that they do so. Nonetheless, one way to inculcate end-user relevance might be to establish it as a required element for assessment in future SRs. The gap is concerning, given that a keen sense of alignment between chosen research questions/design and recognition of stakeholders’ needs facilitates adoption of findings into clinical practice.

Our findings, or lack thereof, lead us to initiate a conversation with the CRR community regarding the conceptual importance of end-user relevance in facilitating knowledge translation to nonacademic stakeholders. There is value in expanding the scope of SRs, and the CRR literature overall, to include stakeholder relevance as a key factor in study design and dissemination to increase the uptake of research findings. The distinct focus on patients’ long-term outcomes and independence puts the field of physical medicine and rehabilitation in a favorable position to advocate for the incorporation of stakeholder relevance criteria into SRs. We hope that this dialogue and the suggestions provided facilitate rehabilitation researchers in advancing the uptake and appeal of their research findings for patients, caregivers, and clinicians.

PERSPECTIVE: WHY DOES RELEVANCE MATTER?

Literature shows that patients, caregivers, and clinicians see the lack of perceived or actual relevance as a key drawback to using CRR literature.[2,19,20] For example, studies executed without stakeholder input may result in findings that are too narrow or that do not matter to patients. Evaluations of expensive, complex equipment that are relevant to academic medical centers may be inaccessible to clinicians in rural areas. Articles that lack streamlined summaries may seem too technical or obscure to busy caregivers or practitioners who cannot quickly discern key takeaways. By succinctly documenting, in plain language, why and to whom a study is relevant, CRR researchers can facilitate a reader’s assessment of whether findings apply to their needs, practices, and goals.

The Patient-Centered Outcomes Research Institute has moved the needle forward by prioritizing research that incorporates stakeholder engagement and patient-centered outcomes, advancing knowledge that is relevant and meaningful for patients, caregivers, and decision makers. However, the Patient-Centered Outcomes Research Institute framework does not address the spectrum of relevance throughout the research lifecycle. To enhance utility and uptake, research questions, methods, and findings must be replicable, acceptable, practical, cost-effective, timely, understandable, and accessible.[12]

OPERATIONALIZING RELEVANCE

Public health, communications science, and information science offer an understanding of the term relevance and propose theoretical frameworks for enhancing it in the context of their fields.[5,18,21-23] However, the lack of a generally accepted definition of relevance in CRR precludes the development of metrics that can be used to measure the concept. Without measurement tools to evaluate relevance, it is difficult to assess how well a given article meets the needs or interests of a specified audience. We do not propose that a metric of relevance be prioritized over other indicators of methodological rigor or evidence strength (e.g., reliability, validity) but believe that equal consideration for relevance would make CRR research more applicable.[10,24-26] The lack of a definition and measurement tool hampers our ability to select outcomes for evaluating an article’s impact on end users who are not researchers. Similar to recent efforts to establish a working definition for “rehabilitation” within the context of scientific research while ensuring applicability to stakeholders within and outside the field,[27] defining and measuring relevance are first steps to enhancing the quality, interpretation, dissemination, and applicability of CRR to patients, caregivers, and clinicians.

Defining Relevance

We propose a working definition of relevance that spans the research process from design to dissemination:

Relevant research addresses the needs, wants, interests, values, or resources of specified stakeholders, and is applicable, useful, or meaningful to the end-user’s activities or goals. In addition:

Relevant research applies methods, processes, or techniques that are replicable, acceptable, or practical in their extension to real-world settings with regard to feasibility, cost, time, resources, comfort, and ethics.

Relevant research is accessible and easily understandable to the intended beneficiaries of the research findings with regard to its delivery, language, and format.

Research may be relevant for one type of stakeholder, but not for another. Thus, relevant research specifies each kind of stakeholder it is intended to benefit, how that stakeholder has been engaged, and how the stakeholder will benefit.

Including a Relevance Statement in CRR

Building on our proposed definition of relevance in CRR, we recommend that authors include a dedicated statement that explains why and to whom their study is relevant. Specifying the stakeholder is key, because relevance is contingent on the audience at hand in that it is thought of in terms of “to whom or for whom.” This is especially important if there are multiple audiences for whom relevance may vary; authors should specify why the research benefits each one.

As a real-world example, Coquart et al.[28] measured self-reported levels of exertion in determining physical capacity, predicting physiological variables, and verifying exhaustion and exercise safety. Self-reported measures were considered “relevant” to patients because the personalized approach increased adherence to prescriptive activity[28]; it was simple to use, cost-effective, and adaptable to a variety of exercise regimens and settings.[28] Salbach et al.[13] used a similar approach. Including such relevance statements may better orient readers to CRR that is aimed at advancing their interests or care, expediting the dissemination of clinically meaningful interventions to the clinicians and patients whom they are intended to benefit.[3,13,29]

Measuring Relevance in Reporting Guidelines

By encouraging researchers to state why and to whom a study is relevant, relevance could then be measured through process-driven metrics, such as whether an article includes a relevance statement. Including a relevance component in the grading systems that SRs use to evaluate the quality of underlying studies is one possibility. For instance, the Grading of Recommendations Assessment, Development and Evaluation approach incorporates patient-relevant outcomes in its ranking of evidence.[30,31] This approach uses a data-driven stakeholder approach by requiring that the outcomes examined by a given study are critical, meaningful, and important to patients; that this criterion is defined from the outset of the review process; and that it is based on concrete data (e.g., patient preference surveys, stakeholder recommendations, patient panels).[7] Other guidelines (e.g., the Cochrane Library’s handbook for SRs of interventions[15,16,32]; the CONsolidated Standards of Reporting Trials for randomized trials[33]) begin to touch upon relevance but could go further by offering a component or metric for evaluating relevance in the appraisal of research. Process-driven “relevance” criteria could include synonymous terms such as usefulness, applicability, responsiveness, or meaningfulness. Wang et al.[26] suggest that along with validity, reliability, standardization, and comprehensiveness, a “relevance and responsiveness” criteria be used to appraise authors’ justification of selected outcome measurement tools. Including such metrics in CRR reporting guidelines would facilitate the measurement of relevance and provide authors with clearer standards for bolstering it as a key factor in CRR of high quality and utility.

STRATEGIES AND SUGGESTIONS FOR ENHANCING RELEVANCE THROUGHOUT THE RESEARCH LIFE CYCLE

Careful consideration of nonacademic end users throughout all phases of CRR, including the selection of research topics, methods, and dissemination, is likely to foster relevance for these stakeholders. From the outset, investigators should think carefully about the motivation for their research and consider designing studies that meet the needs of patients or clinicians as opposed to other academics or journal editors.[31] In CRR, this is particularly true for outcomes related to long-term health and quality of life, which present time and cost challenges but are valued by patients, caregivers, and clinicians.[2] Researchers should be mindful of the resources and time their methods require and avoid the use of expensive or inaccessible equipment.[19]

Authors should also describe how their research was informed by and responsive to stakeholders.[5,14,34,35] For instance, were listservs, surveys, focus groups, public deliberation, or key informant interviews used to collect end user’s preferences?[14,34,36] Are selected outcomes clinically meaningful, patient centered, or function related?[26] Engaged scholarship, a form of collaborative inquiry between academics and practitioners that leverages different perspectives to generate knowledge[4] can be used to promote relevance for clinicians. Providing justification is important, stakeholders might not agree with researchers about which outcomes are meaningful.[31,37] For example, fatigue, pain, or range of motion may be less important to patients than the impact of an intervention on their ability to return to work or function in daily life.[36] Stakeholder involvement promotes CRR that is relevant to practice and recovery, reflects end users’ values, and prioritizes methods that are easier to implement in real-world settings.

Authors should also consider the presentation of their methods and results when disseminating findings. If patients comprise the target audience, researchers should include high-level summaries and minimize the complexity of their writing.[20] Allowing for sufficient “white space,” using bullet points, and incorporating smaller tables constitute minor formatting adjustments that facilitate reader comprehension.[38] In contrast, research catered to clinicians should provide enough detail to make replication in various clinical settings feasible.[39,40] The importance of practicability has been expressed by practitioners who often have difficulty applying CRR because it is not applicable, specific enough, broad enough, or timely.[2,19] Discussion sections should clearly describe pathways to success and focus on the most effective interventions or on those for which populations respond most favorably.[14,38,41,42] The juxtaposition of needs based on the stakeholder at hand demonstrates the importance of specifying the audience. The relevance of a study in part depends on the accessibility of its structure, format, content, and delivery,[2,38] yet these elements are traditionally tailored toward other academic researchers.

Journals can also improve the translation and dissemination of CRR to augment relevance. Clinical ambassadors who work with the patient population of interest can be used to distill key findings and promote buy-in from nonacademics.[10] Journals might request that authors suggest clinical ambassadors during the manuscript submission process, similar to suggesting peer reviewers. Journals might also require summary boxes that highlight the research’s beneficiaries and why the article relates to them. For example, the American Journal for Health Promotion includes a “So What?” box that presents, in a few plain-language sentences, how the article relates to health promotion practitioners.[43,44] Adding a relevance statement to such boxes in journals of CRR would facilitate the reader in identifying research that is directed to them. Similarly, journals might require authors to submit patient- or clinician-oriented abstracts. The authorizing legislation for the Patient-Centered Outcomes Research Institute recognized the importance of this approach by requiring publication of plain-language abstracts for patients, caregivers, and practitioners for all Patient-Centered Outcomes Research Institute–funded studies. Extending this idea, 1- to 2-page companion pieces could summarize takeaways using lay language and omit information that is tangential to the needs of therapists, patients, or caregivers; such details could be provided through hyperlinks to supporting documents or the larger manuscript. A two-pronged publication approach would allow journals to cater different research products to different audiences’ needs.

TOOLS FOR ENHANCING RELEVANCE IN CRR

Frameworks for Conceptualizing and Operationalizing Relevance

To advance the conversation regarding relevance in CRR, we have devised two preliminary frameworks that integrate our suggestions above, guiding authors in their efforts to conceptualize and operationalize relevance throughout all phases of the research life cycle.

In Figure 1, we present a possible framework for conceptualizing the components of research that may contribute to relevance in CRR. This framework includes questions that CRR authors can ask themselves in their efforts to improve the relevance of their research throughout all phases of study design. Consideration of this framework during a study’s conceptualization, design, presentation, and recommendations for clinical practice would help clarify why and how investigators’ interventions and findings are relevant to nonacademic stakeholders.

FIGURE 1

Factors and questions to consider when conceptualizing relevance during the development of CRR.

In Figure 2, we offer a preliminary framework for operationalizing relevance in CRR, which demonstrates how researchers can integrate components of relevance during each phase of CRR. This is not meant to be a blueprint for a protocol, but rather, it considers components that could be used to select measurements, report results, and involve stakeholders during study development.

FIGURE 2

Ways in which relevance can be operationalized through patient-oriented metrics and stakeholder inclusion strategies.

Input for these frameworks was derived from (1) the exploratory survey of peer-reviewed and gray literature described earlier; (2) discussions between investigators and students working as part of the Model Systems Knowledge Translation Center, which conducts research to improve long-term functional, vocational, cognitive, and quality-of-life outcomes in individuals living with spinal cord injury, traumatic brain injury, and burn injury[45]; and (3) feedback received from qualitative interviews with patients, caregivers, and clinicians in the field of rehabilitation.[20] The concepts identified from these sources were transposed into the frameworks to equip researchers with strategies that promote relevance under a traditional research schema for generating studies (i.e., introduction, methods, results, discussion). We recommend that the proposed frameworks undergo iterative review, testing, and validation to better satisfy the needs of the CRR community.

Alternative and Enhanced Study Designs

Researchers who are interested in promoting relevance in their research products might consider conducting or disseminating their research through less conventional approaches. In Table 1, we summarize variants of SRs and alternative designs whose content and structure promote relevance as a critical research component, demonstrating how adapting research products can bolster relevance in CRR.

TABLE 1

Innovative study designs for enhancing research relevance to nonacademic audiences

Approach	Description
Meta-syntheses	This SR variant focuses on constructing greater meaning by interpreting and incorporating the experiences of patients, caregivers, and clinicians.[46] In practice, this method is usually qualitative in nature. These studies paint a better picture of user experience, quality, and trajectories of care, capturing outcomes that matter most to patients and caregivers.
Practice-based evidence	In juxtaposition to evidence-base practice, practice-based evidence is an approach that relies on large numbers of cohort studies (which tend to be conducted in real-world settings), a diverse array of patients, a wide variety of variables, and a focus on external validity. It is an example of participatory action research and emphasizes the inclusion of many stakeholders. In comparison with traditional SR grading systems, which disproportionately weight in favor of randomized controlled trials, this approach gives more weight to quasi-experimental and observational results supported consistently by multiple studies. The inclusion of more stakeholders, studies, patients, variables, and research questions makes it likely that findings will be relevant, credible, and actionable to a variety of beneficiaries, particularly practitioners.[5,42,47]
Realist reviews	These studies seek to understand how a series of interventions work, the key active ingredients that enable them to work, and the conditions under they are most likely to be effective. Realist reviews can be used to complement traditional meta-analyses by providing in-depth considerations of context, mechanisms, and outcomes. They yield explicit recommendations for policy and practice regarding strategies and settings in which interventions are most likely to be successful, generating results that are particularly relevant for policymakers and healthcare administrators.[8,48,49]
Effectiveness plus review and effectiveness plus parallel review	These two designs are intended to promote relevance by accompanying and complementing more traditional SRs. Effectiveness plus reviews are largely consistent with the conventional SR but include modifications to design, such as extracting additional data on how, why, costs, risks, and other contextual factors. These designs give greater weight to quasi-experimental and observational studies than do typical SRs, which tend to focus on randomized experiments. By integrating gray literature and anecdotal evidence, these reviews may help to contextualize findings for rehabilitation, wherein concrete evidence for the effectiveness of some therapies may be difficult to produce (e.g., acupuncture). Effectiveness plus parallel reviews provide all of this information, but are paired with a traditional SR, maintaining relevance to academic researchers more interested in a traditional assessment of methodological rigor or study design.
Overviews (umbrella reviews)	These “SRs of SRs” address broad research questions, exploring the effects of different interventions for the same population or problem, or the same intervention for different populations or problems, enhancing the likelihood that findings may be relevant to multiple audiences. While they hold promise, there are currently very few in CRR.[41]
Evidence summaries	Evidence summaries are one-page reviews that often use content experts or knowledge brokers to translate research findings into practice recommendations. They make use of straightforward, “take-home” messages that are targeted and tailored, leading to improved communication of the relevance and importance of a topic, the key results and their potential impact, and implications regarding the applicability of findings for decision-makers.[2,10,11,38,50] These concise documents allow for quick interpretation, an element valued by policymakers, healthcare managers, and clinicians.

CONCLUSIONS

Although the field of CRR has made progress in enhancing stakeholder engagement and patient-oriented outcomes, we believe that the concept of relevance remains underrepresented. We did not find a clear definition for the term “relevance” in the CRR literature, nor have we identified many studies that elaborate on what is meant by clinically relevant research. We did not find an existing measurement tool that measures relevance in CRR. There seems to be a gap in the CRR literature with regard to how relevance can be defined, operationalized, and promoted.

To begin addressing this problem, we have proposed a definition of relevance for CRR investigators’ consideration. We suggest that relevance can be bolstered by requiring authors to include relevance statements in their applications for funding and presentation of findings. In Figures 1 and 2, which were informed by patient, caregiver, and clinician interviews, we learned that these stakeholders found that CRR results rarely addressed their values or needs and were not easy to apply to their practices. Hence, we pose questions about relevance for CRR investigators to reflect on and provide a guiding framework that investigators can use to integrate relevance during the study design phase. In Table 1, we suggest innovative research designs that researchers looking to promote relevance for nonacademic audiences may consider.

To improve the uptake of CRR findings by decision makers, the professional performance of clinicians, and the usefulness of research findings to patients, it is critical that the CRR community discuss the importance of relevance. Relevance criteria should be defined, operationalized, and routinely incorporated into CRR.[2] It is our opinion that by identifying who is likely to benefit from CRR studies, soliciting input from those recipients during the design phase, using that input to inform and justify the selection of study methods and outcomes, and tailoring CRR content so that it is understandable, relatable, and accessible to them would generate CRR that is more relevant to prespecified beneficiaries. Ultimately, this would generate quality publications that are accessible and applicable to stakeholders and likely to promote knowledge diffusion and adoption of evidence into practice. Such steps represent critical, yet overlooked, components of research planning, and should be championed by physical medicine and rehabilitation in its efforts to enhance the accessibility, utility, and application of CRR products.