Credibility matters: mind the gap.

Introduction

Clinicians, policy makers, and patients need to be able to rely on high-quality scientific research to make informed decisions about health care options and policy. Frustration ensues on all levels when there is low confidence in the quality and integrity of available research on spine care. When research quality is low and reporting of it is poor, clinicians and patients may have confusion regarding best health care options. Policy makers may not reimburse for treatments or diagnostic modalities that are deemed not effective based on available evidence. At the most basic level, all parties want the same thing: to do what “works,” yet they all suffer when there is low confidence in evidence.

There is a credibility gap that spans all aspects of medical research, from study planning to study reporting to availability of data for verification to final study publication. Several studies provide empirical evidence on publication and related biases and how conclusions may differ based on what is and is not reported and how.1 2 3 One example of publication-related bias is seen in the recent controversy surrounding results from the Yale Open Data Access (YODA) studies4 5 as compared with original trial publications on bone morphogenetic protein. A primary conclusion from consideration of these reports was a call for timely and complete transparency of data reporting.6 7

Subsequently, media and scientific circles have reiterated strong calls to reduce study bias in study analysis and reporting.8 9 10 11

Outcome reporting bias is one type of publication-related bias and is an under-recognized problem.12 13 This occurs when there is selective reporting of some outcomes but not others, possibly depending on the nature and direction of the findings. In addition to ethical concerns regarding such selective reporting, the reported results can be misleading. One example of the impact of such selective reporting is an analysis of 283 Cochrane Reviews. Kirkham et al report that 34% of reviews contained at least one trial with high suspicion of outcome reporting bias for the primary outcome.12 Sensitivity analysis on these reviews revealed that the treatment effect was reduced by 20% or more in 23% of reviews. After adjustment for outcome reporting bias, 19% of meta-analyses with a statistically significant result became non-significant and 26% would have overestimated the treatment effect by 20% or more. This can impact policy making and clinical decision making and potentially result in harm to patients.

Transparency and attention to detail in research design, specification of outcomes, analysis, reporting, and dissemination are critical to “minding the gap” regardless of study design or level and type of funding. This article (and previous Science in Spine articles) describes some key components for such transparency related to conducting research with a focus on outcomes reporting.

Where Does It Start?

The credibility gap must be considered and addressed at all levels of study planning, reporting, and publication of any study, regardless of design. It starts with:

Fully formulating a focused and answerable study question as described in the previous Science in Spine.14

Creating specific study aims and testable hypotheses that are objectively stated a priori.

Using a structured approach to specify the study question and to guide research design and execution. The Patients, Intervention, Comparison, and Outcomes (PICO) table for treatment and diagnostic studies or a Patients, Prognostic factors, and Outcomes (PPO) table for prognostic studies is one method for providing the blueprint for conceptualizing, operationalizing, and reporting results from your study.

Using your PICO/PPO to stay on track. Use of the PICO/PPO topology or other organizing framework can enhance the quality of your study and the quality of reporting by decreasing ambiguity, clarifying objectives, as well as identifying and focusing on aspects of primary importance. All reported outcomes should be traced back to your PICO/PPO, study question, and specific aims.

The value of using the PICO format was highlighted in a study of 89 RCT reports.15 Rios et al created a score based on the PICO elements and then examined the extent to which the reports stated the PICO elements of a structured research question and correlation with an overall quality reporting score based on the Consolidated Standards of Reporting Trials (CONSORT) guidelines. The result: The PICO-related score was independently associated with the overall reporting quality score. The implication is that when care is taken to specify the question and to use a framework for study design and execution, reporting of study components related to quality is higher as is the perceived quality of the study. It also helps prevent biases such as outcome reporting bias. In the era of evidence-based practice, attention to the quality of study design, execution, and reporting is important to policy makers and others.

The Components and Applying the Concept

Table 1 provides an overview of PICO/PPO. In many instances, it is logical to add two components, one for Timing and one for Setting, modifying the acronym to PICOTS/PPOTS. The “S” may also be used to denote study design.

Table 1

Overview of PICOTS and PPOTS

	PICO		PPO
	Therapeutic	Diagnostic	Prognostic
Patients	What patient group?	What patient group?	Patients	What patient group?
Intervention	In what surgical treatment, procedure, or implants are you interested?	What diagnostic procedure?	Prognostic factors	What primary prognostic (risk) factor might influence outcome?
Comparison	What is the comparison (control) treatment?	Is there a gold or suitable reference standard?		What other factors might influence outcome?
Outcomes	In what outcomes are you interested? (e.g., pain)	Are you interested in validity (e.g., sensitivity/specificity), and/or reliability (e.g., inter/intra rater reproducibility)	Outcomes	In what outcome(s) are you interested? (e.g., nonunion)
Timing	What follow-up times are important	Does timing of the test influence the outcome	Timing	What timing of follow-up or assessment of outcome is important (e.g., peri-operative)
Setting (or study design)	In what setting(s) is treatment performed (e.g., in the emergency department)	Under what conditions or locations (setting) is the diagnostic test performed?	Setting (or study design)	What conditions or settings are important to consider?

Patients: A homogenous patient population is best. It is important to define the patient population in terms of all factors that relate to the condition of interest, patient demographic features (e.g., age, gender), behaviors (e.g., smoking), medical history, medications (e.g., steroids, NSAIDS, etc.) that may influence outcomes, general health factors, comorbidities, factors that may be associated with the treatment selection (e.g., location/severity of condition), and others that may be relevant to treatment selection or influence of outcomes. Are patients with previous surgical interventions to be included or excluded? Are specific pathologies to be excluded?

Intervention: This may be a newer or novel treatment that is to be compared with a more standard treatment (called the comparator).

Comparator: This is your “control” group and consists of those receiving the alternative, standard, or “other” treatment to which the intervention will be compared. All comparative studies will have a control/comparator group. Sometimes your question will not have a control group, such as in the case when you are interested in safety or handling characteristics of a new implant or procedure.

Outcome(s): What is the primary outcome of importance? Be specific and aim for the most important outcomes. Conceptual examples include patient reported outcomes such as pain, function, and quality-of-life as well as more clinical outcomes such as nonunion, major complications, repeat surgery, or death. It is best to use validated outcomes measures and measure clinically meaningful outcomes as well as harms. Future articles will discuss operationalizing and measuring your outcomes.

Resources for additional details about applying this to diagnostic and prognostic studies (PPOTS) can be found in the SMART Handbook for Spine Clinical Research,16 and Agency for Healthcare Research and Quality Methods publications.17 18

How Does This Help? Returning to Our Focus on Outcomes Reporting

Choosing to not evaluate or report on an outcome (particularly a harm) may reduce the credibility and applicability of your findings. The PICOTS is the start of your research game plan. Specifying and defining the outcomes to measure, how and when they will be measured a priori, and sticking to the plan assists you in avoiding ambiguity and misreported results. You are committing to measuring, analyzing, and reporting on those outcomes, including those related to harm, regardless of their statistical significance. This enhances the transparency and credibility of your study report and provides a sound basis for drawing objective conclusions.

Why Does It Matter?

Empirical evidence of outcome reporting bias (particularly related to treatment harms) over the past decade has led to a call for the registration of clinical trials and publication of protocols prior to trial completion to ensure transparency.3 12 19 20 Increasingly, there is a call for researchers to publish study protocols of non-randomized studies. There is also increased interest in comparing the extent to which published results from a study are consistent with the study aims and the prespecified protocol. Study credibility is at stake, even if yours is not an RCT.

Regardless of study design, using the PICOTS/PPOTS framework as part of your prespecified protocol helps you stay on track as you plan and execute your study and is an important initial step toward transparency. It can form the basis of a checklist for ensuring that you have followed the basic game plan. Keeping it in mind as you write up results (positive as well as negative findings) will assist you in avoiding selective reporting and other reporting bias. This in turn enhances the credibility of your study within and outside your field.

Consider how reporting of your results may impact future studies that may build on yours, and how your data may be used in synthesis of data across studies such as in meta-analysis. The accuracy and completeness of your report may impact the credibility of the overall body of evidence.

Summary

The purpose of the Science in Spine articles in EBSJ is to assist surgeons in understanding research, facilitate critical thinking about research beyond “statistical significance,” and to help enhance the quality of research that they report. Decisions by clinicians, patients, and policy makers rest on the quality and integrity of reported research. To avoid biased study reporting:

It is important to have a framework such as PICOTS/PPOTS for specific primary study features a priori.

It is important to report on all study results/outcomes regardless of statistical significance.

It is important to consider the potential for various types of reporting and publication bias when critically appraising studies and systematic reviews.

It is in the best interest of all to “mind the gap” and actively take steps to improve the value and reporting of research (regardless of study design or funding source) by following basic research steps to ensure quality.