Moving to a user-driven research paradigm.

The traditional "bench-to-bedside" paradigm for clinical research has been successfully used for many decades. This model of knowledge generation has led to discoveries that have enhanced the quality and length of life. The combination of changes in research practice and in health care delivery, growing complexity in decision-making, increasing use of electronic health records (EHR), and growing resource constraints necessitate a shift to a user-driven research paradigm to generate new knowledge. This conceptual framework was created to clarify the perspective of the decision makers as well as the range of factors and the variability in thresholds used to make decisions. This framework may help researchers in creating actionable information to meet the needs of decision makers, which is needed for the transition to a user-driven research paradigm. Further, it is important to create an appropriate set of incentives to facilitate this transition to a user-driven research paradigm.

Background

The classic “bench-to-bedside” paradigm of clinical research has been used for the rapid adoption of many biomedical breakthroughs—such as insulin, penicillin, and the polio and rabies vaccines—in clinical practice and public health. The cumulative impact of many clinical and public health innovations has enhanced the quality and length of life and has changed the disease burden from acutely life-threatening, disease to chronic disease. The dramatic impact of combination antiretroviral therapy for HIV infection is a recent example of this change.1

Many past discoveries did not use the randomized control trial (RCT) study design to evaluate the benefit of interventions prior to their adoption in clinical practice; some studies had either no patient controls (e.g., penicillin) or no comparable controls (e.g., insulin).2,3 The evidence bar for new interventions has been raised over time. Three factors have contributed to this change: advances in research methods, smaller benefit of a new intervention compared with others of its class (such as drugs to lower cholesterol, glucose, or blood pressure), and the need to demonstrate an improvement in health outcomes.

The research practice has also evolved from a small group of scientists working on a shoestring budget to a large, team-based, multidisciplinary, resource-intensive approach. Funding for research has dramatically increased over the past 75 years; for example, the appropriations for the National Institutes of Health in 1938, 1950, and 2000 were $0.46 million, $52.7 million, and $17,840 million, respectively.4 However, the current federal budget constraints make it unlikely for this pace to continue.

A notable development is the ongoing rapid pace of adoption of electronic health records (EHR), which have the potential to improve efficiency and effectiveness of clinical care. They also have the potential to improve research efficiency, especially in shortening the time from data collection and analysis to making a decision. Recent investments in building an electronic clinical data infrastructure were designed to ensure it can be used for diverse purposes: improving patient outcomes, answering research questions, conducting public health surveillance, and quality improvement. Therefore, this infrastructure can meet the needs of patients, providers, researchers, and policymakers.5

A clinical researcher has traditionally focused on creating new knowledge, which helps in recognition by peers and in securing research funding. There is a paucity of incentives to ensure research questions are designed to meet the needs of patients and other health care decision makers. Similarly, the focus of decision makers of health care delivery organizations has been on issues related to care delivery and not on shaping the research agenda. These factors have resulted in large gaps in our knowledge of the impact of diagnostic and therapeutic interventions on patient outcomes in the real world.5

The recent advances in methods and the creation of new tools to link diverse electronic databases and analyze a large volume of information are a boon to researchers.6 However, in order for researchers to effectively influence clinical practice, it is vital for them to understand the perspective of health care decision-making and to focus their research accordingly. It is also important for decision makers to understand the research process and to help formulate questions that can be answered by research and still meet their needs, which requires their early engagement in the research process and promoting a culture of research within their organizations. This necessitates a use of incentives that align the interests of researchers and decision makers and a shift to a user-driven research paradigm. The Agency for Healthcare Research and Quality (AHRQ) and the Patient-Centered Outcomes Research Institute (PCORI) have taken early steps in this direction, but more needs to be done to make it a routine part of research practice.7,8

The conceptual framework described in this paper was created to clarify the perspective of the decision makers and the range of factors (several of which are not based on published evidence) and variability in thresholds used to make decisions. This framework is primarily based on my experience and observations during my work with two AHRQ programs: the U.S. Preventive Services Task Force and the Effective Health Care Program. The framework may help researchers in creating actionable information. It does not examine decision-making at the regulatory or law-making level; its focus is on decision makers of health care delivery organizations. Legal and regulatory requirements are one of several factors considered by these decision makers. Although the focus is on the policy level, there are similarities with decision-making at the individual level (either patient or provider). Issues that shape clinical policy overlap with, but are not identical to, those that shape payer policy. Additionally, the framework is focused on issues related to health care delivery and not public health programs. Public health decision-making is important and deserves a separate framework.

Conceptual Framework

Figure 1 depicts the flow of information in the design and conduct of research and the translation of research findings into practice in the traditional bench-to-bedside paradigm. The first box lists factors that influence the design and conduct of a research study. The second box lists factors used by health care decision makers; the upper part of the box lists factors that can be answered by research (and therefore available as published evidence), and the lower part of the box lists additional factors (not based on published evidence) taken into account when making a decision. The third set of boxes indicates various types of decision-making. The mix of factors considered and the threshold for making a decision depend on the type of decision-making.

Figure 1:

Flow of Information in the Bench-to-Bedside Paradigm

Research Study Design and Conduct

This box refers to factors that influence the design and conduct of a research study. The first step is to clarify the research question(s). When designing a study to evaluate a health care intervention, a researcher needs to specify the study population(s), the intervention(s) to be evaluated, the comparator(s) to the intervention, the outcome(s) to be evaluated (may include patient or health system outcomes), the timing of the intervention, and the setting (including the care delivery setting and its similarity to a real-world setting for a comparative effectiveness research study). PICOTS is an oft-used mnemonic for these factors.9 It is important to specify these factors regardless of which study design is ultimately selected.

When choosing a comparator to the intervention, it is important to select a realistic alternative. Numerous discoveries in the last few decades have resulted in many choices to treat diseases; it is rare for a placebo to be considered the only viable clinical alternative. The appropriate comparator for a new intervention is an existing realistic clinical alternative, not a placebo.

The researcher also needs to clarify the available resources to answer the study question. These include availability of qualified personnel, data infrastructure, and financial resources. The time-line to completion of the study and publication of study results is often a relevant consideration.

The investigator-initiated grant is a common mechanism to fund clinical and health services research. The investigator specifies the hypotheses and study design that need to pass peer review necessary to secure funding and to publish results, which in turn is necessary for future grant funding. This process traditionally has not engaged the end users of research, but there is no reason that it cannot (as discussed later in this paper). A heavy emphasis on molecular discovery and paucity of end-user engagement by investigators has led to a skewed focus of the clinical research enterprise. For example, a review of National Cancer Institute’s 2007 extramural grant portfolio in cancer genetics showed 827 grants (more than 80 percent) focused on biomedical discovery and only one grant with a population-based focus to reduce burden of disease.10

Decision-Making Factors

Policymakers in health care delivery have to consider several factors in making a decision. The evidence from a research study (or from a systematic review of existing research studies) is only one component. Research can clarify the benefits and harms of an intervention compared with alternatives. Descriptive epidemiology can clarify the natural course of a disease. The natural course of disease and the epidemiological causation criteria are particularly relevant when all evidence is derived from observational studies (see discussion of clinical guideline development later in this paper). Cost-effectiveness and other economic analyses may also help, although their use is limited by the quality of the model and the perspective used in framing the question. All these factors are listed in the upper part of the second box in Figure 1.

The lower part of the second box in Figure 1 lists factors that require local input and perspective, including knowledge of the relevant legal and regulatory requirements (that may change, especially for disruptive interventions); these are not obtained from published research. In order to apply the research findings within a health care delivery organization, the decision maker needs to know the local population of patients and clinicians and how their differences from those in a research study may alter the expected balance between benefits and harms. The feasibility of implementing an intervention is particularly relevant when it requires additional financial commitments, including trained personnel and purchase of new equipment. Additional training of professionals is often required; for example, there is a learning curve before becoming proficient in the use of new surgical procedures or in the use and interpretation of results of a new imaging device. There may be no conveniently available behavioral or genetic counselor to deliver the appropriate intervention. Feasibility also includes local clinical work flow and time considerations. A computer-based clinical decision support tool will not work in a paper-based clinic. A time-intensive patient questionnaire to be administered at the point of care won’t be used by a primary care clinician with limited time for a patient encounter.

Budget is an important component of operational constraints. The cost of hiring new personnel, purchasing expensive equipment, or construction of a building to deliver an intervention may not be feasible within the budget of a small organization. The local culture of the providers may influence the support for, or resistance to, implementing a new guideline. The delivery of an intervention must meet the relevant legal and regulatory requirements. Variability in state laws and local regulations may modify the delivery of a new intervention. Finally, a decision maker’s desire to be equitable may mean the intervention is offered to diverse patient subgroups, even if the intervention conferred a large benefit to some patient subgroups and a marginal benefit to others.

Type of Decision

The type of decision-making determines the appropriate range of factors considered in making a decision and the trade-offs (or the decision-making threshold) that drive the decision. It is useful to briefly discuss the consideration of relevant factors from a decision maker’s perspective.

Clinical Guideline Development

The primary consideration is the trade-off between benefits and harms to a patient, and the certainty in the magnitude of benefits and harms.11,12 A health technology assessment or a systematic review of the relevant evidence informs guideline development. The following factors are considered in this type of decision-making:

Prevention or treatment: It is important to clarify whether the intervention will be used as a preventive service in a population (i.e., intervening in an asymptomatic population to prevent disease) or for treatment of an individual who seeks relief from the symptoms of a disease. In adhering to the principle of “first do no harm”, the evidence bar for an intervention (i.e., the certainty in the magnitude of net benefit; see next item in this list) in a healthy population needs to be necessarily higher. Conversely, when there is a known high risk of harm of non-intervention (e.g., someone with metastatic cancer having failed conventional therapy), a decision-maker may tolerate higher risk of harm of the intervention for smaller chances of benefit (i.e., the threshold to make a decision is lower). Some interventions (e.g., open surgical procedures) are intrinsically more likely to cause physical harm to a patient compared with others (e.g., counseling interventions), therefore the magnitude of benefit needs to be larger when considering intrinsically harmful interventions.

Net benefit compared with a realistic comparator: The net benefit is the magnitude of the benefit minus the magnitude of the harm. A decision maker assesses whether a new intervention has a net benefit compared with an existing alternative used in clinical practice. The evidence from a RCT that compares a new intervention with a placebo is not helpful.

Hierarchy of evidence: There is a well-described hierarchy of study design to evaluate outcomes of interventions, and guideline developers use the entire range. In many instances the net benefit of an intervention is established by an RCT and information from other types of studies is used to fill in the margins. For example, the net benefit of colorectal cancer screening was obtained from an RCT; information on other aspects related to screening, such as modality and frequency of screening and age to start or stop screening, was obtained from observational and modeling studies.13,14 However, in some cases observational studies alone have conclusively established benefit of an intervention, such as cervical cancer screening in women and bariatric surgery (compared with pharmacological therapy) in patients with morbid obesity.15,16,17 When considering evidence from observational studies alone, the guideline developers rely on the study quality, the magnitude of effect size, a thorough understanding of the natural course of disease, and whether the evidence meets other relevant epidemiological criteria of causation such as temporality, biological plausibility, dose response, and consistency.

Timeline to make a decision: Guideline developers often are willing to wait for a systematic review of available published evidence that can answer relevant questions before making a decision. In contrast, the decisions are made in extremely short time frames in other situations. In these cases, the only feasible option is either to use an existing systematic review (when a relevant one is available) or to conduct an abbreviated review of published evidence. In the uncommon situation when both time and resources are available to commission a new study, the choice of the study design will be driven by a combination of factors (see Table 1 later in this paper).

Table 1:

Factors Influencing Choice of Study Design

Factor	Randomized Trial	Nonrandomized Study
Common disease	Yes	Yes
Rare disease	Depends	Yes
Many study variables	Depends	Yes
Anticipated small effect size	Yes	No
Anticipated large effect size	Yes	Yes
Need for high certainty	Yes	No
Hypothesis testing	Yes	Depends
Hypothesis generating	Depends	Yes
Effectiveness setting	Depends	Yes
Long-term outcomes	Depends	Yes
Short-term outcomes	Yes	Yes
Scarce resources	No	Depends
Rapidly changing technology	No	Yes
Evaluate policy or system-level interventions	Depends	Yes
Clarify natural course of disease	No	Yes
Disease or safety surveillance	No	Yes
Study new public health problem	No	Yes
Factors precluding randomization	No	Yes

Prior belief and willingness to act: Prior belief and the requirement to show proof of either actual harm or actual benefit leads to variability in decision-making. The prior belief of a clinician, in the face of an equivocal result from a diagnostic test, will determine the course of action (e.g., to proceed with another diagnostic test or a therapy or not). A similar issue leads some to be early adopters and others to be late adopters of a new technology. The early adopters focus on the potential benefit and want to know evidence of actual harm before stopping the use of a new technology. The late adopters prefer to have the benefits and harms clarified by evidence before deciding to adopt a new technology. These different perspectives can result in a controversy on whether or not to adopt a new technology such as a cancer screening test.18

Guideline Implementation

A guideline is typically made by a national organization and its implementation occurs in a local health care delivery organization. A large number of factors influence the implementation of a guideline. These include applicability of the guideline to the local population, technical feasibility, operational constraints, relative priority given other competing demand for resources, organizational culture (including resistance or support of personnel needed to implement the guideline), and compliance with local laws and regulations. Most of these factors do not rely on research findings.

Quality Measurement and Improvement

The focus here is to improve clinical practice (with the goal of improving patient outcomes) in accordance with the evidence. The decision-making is not focused on clarifying the evidence of benefits and harms of an intervention, since this was already done in formulating the appropriate guideline; rather, it focuses on design and evaluation of a program to improve quality of care. Local factors and available resources will have a large role in determining the design and rollout of the program. Clinical and organizational work flows are important considerations. Usability considerations are important, especially in the design of a new clinical decision support tool. There is also flexibility in learning by trial and error. For example, if a new implementation strategy had the desired impact on clinical practice in some clinical organizations but not in others, it is feasible to redesign the implementation strategy to meet the needs of an organization. Additionally, trust and credibility are important factors in the decision of an organization to join a network that will share performance on quality measures and other important information across organizations.

Coverage and Benefit Design

All factors considered by clinical guideline developers are also considered in this type of decision. Costs, cost-effectiveness, regulatory compliance, and equity are additional factors in decision-making.

Operational Efficiency

A system engineering approach can be used to improve operational efficiency of a healthcare delivery organization. The performance of different parts of a system, and interactions among these parts, needs to be clarified prior to the design and implementation of appropriate strategies to optimize system performance. Factors that influence decision-making include operational constraints, workflow, local culture, and the trade-offs among the competing needs and priorities.

A User-Driven Research Paradigm

A change in circumstances over the past century has necessitated a shift from the bench-to-bedside paradigm to a user-driven research paradigm (see Figure 2). There is increased complexity in health care delivery and decision-making at the policy level. There is an increased focus on interventions that directly improve patient and health system outcomes, many of which do not rely on biomedical discoveries. The challenges facing decision makers include increased availability of several therapies with varying efficacy, adverse-effect profiles, and costs; the need to make care delivery patient centered and to improve outcomes; and increasing costs without a commensurate improvement in outcomes. It is important to focus on research that helps decision makers meet these challenges.

Figure 2:

User-Driven Research Paradigm

A critical first step is to engage decision makers early in the research process to ascertain their needs. This input can be used to clarify the questions that can feasibly be addressed by a research study. It is also important to clarify the timeline to obtain the answers, the certainty needed in the conclusions, and the resources that are available to conduct the study. The study design will be driven by a combination of factors such as goals of the study, resources available, anticipated effect size, and desired level of certainty. Table 1 lists several relevant factors to be considered in deciding which study design is likely to meet the decision makers’ needs. It is not meant to be a comprehensive list of all factors to be considered in designing a study, which is an important issue but beyond the scope of this framework. Additionally, it is important to keep in mind that some types of decisions, such as guideline development or coverage, need a portfolio of research (as discussed earlier in the context of screening); one study will not answer all relevant scientific questions.

The availability of EHR-based data infrastructure and the rapid proliferation of mobile devices and applications can enable rapid and relatively inexpensive access of researchers to clinically rich, patient-specific information. It also has the potential to increase researchers’ interactions with clinicians and patients, which may help in focusing the research questions on the needs of end users. For example, mobile devices and electronic questionnaires can help the routine collection of patient-reported outcomes before and after a visit to a clinician. This information can help clinicians tailor care to the needs of a patient. Additionally, it can help a researcher answer questions without building a de novo prospective data collection infrastructure and also understand the needs of patients and of clinicians.

The availability of EHR, database linkages, and new methods and applications is necessary but not sufficient to change the focus of research. It is important to actively engage decision makers early in the research process; however, this cannot be the sole responsibility of researchers. The decision makers need to prioritize engagement with researchers and help to shape the research agenda. An appropriate set of incentives needs to be designed to ensure a meaningful dialogue between researchers and decision makers prior to submission of a grant proposal as well as to ensure appropriate recognition of the value of the research by the peer reviewers. Finally, it is encouraging to note several efforts—domestic (e.g., AHRQ and PCORI) and international (e.g., The James Lind Alliance) 19,20—that are collectively moving toward a user-driven research paradigm.