Reading between the Lines: A Plastic Surgeon's Guide to Evaluating the Quality of Evidence in Research Publications.

An important component of practicing evidence-based medicine (EBM) in plastic surgery is staying current with published research, an increasingly difficult task given the rapid growth of the literature. This article reviews the methodological aspects specific to the aesthetic surgery field that should be considered when evaluating the quality of evidence in research publications in the context of the level of evidence (LOE) grading scales that are currently used by plastic surgery journals. Reporting the LOE in a research publication can help to highlight the quality of the research and the potential for bias so that the reader may prioritize information accordingly. However, LOE classifications are not an absolute index of the quality of evidence and do not preclude careful evaluation of the study's methods and results in the context of the authors' conclusions. As the application of evidence-based medicine expands in the plastic surgery community, surgeons must be mindful of how to appropriately interpret research findings and assess the utility of applying results to their practice.

INTRODUCTION

The evidence-based medicine (EBM) movement in plastic surgery strives to integrate the best research evidence with clinical expertise to enable rational treatment decisions and optimal patient outcomes.[1] An important component of practicing EBM is staying current with published research, an increasingly difficult task given the rapid growth of the literature.[1] To keep pace with the new research, busy clinicians may only have time to skim the abstract and conclusions of a publication without critically evaluating the methodology or the results of the study.[2] However, before applying research findings to their practice, clinicians must take responsibility for critically appraising the quality of the methods and results and assess how the authors’ conclusions are framed in the context of the presented evidence.

There are many important questions to ask when reading a research article. The following are 3 helpful questions to consider[3]:

Does the research address a worthwhile question? Is it the correct question to ask?

Are the study’s methods appropriate to answer the stated research question(s)?

Are the authors’ conclusions justified by the data and the methods?

However, questions about the appropriateness of the study’s methods and the justification of the authors’ conclusions may be difficult to address if one is not profoundly familiar with the specific area of research. This article reviews methodological aspects specific to aesthetic surgery studies that should be considered when evaluating the quality of evidence in research publications and discusses these factors in the context of level of evidence (LOE) grading scales that are currently used by plastic surgery journals.

Assessing Methodological Quality of Plastic Surgery Studies

In general, randomized controlled trials (RCTs) and systematic reviews are considered to be the ideal for obtaining high quality evidence.[4,5] However, a study cannot automatically be deemed “high-quality” based simply on these designations. There are many methodological factors to consider when evaluating the quality of evidence within a study.[4-6] For example, studies conducted without a priori power analyses to determine sufficient sample sizes can be problematic and predispose a study to false-negative statistical errors.[6] Improper study conduct and planning can also result in erroneous conclusions,[6] as can conclusions from studies with inherent biases, such as test/retest performance bias or recall bias.[7] Studies that do not report effect sizes cannot be evaluated for the true magnitude of differences between study cohorts, even if P values are reported.[8] The results of meta-analyses warrant critical evaluation based on the heterogeneity of the studies chosen for inclusion; studies with vastly different methodologies may not be suitable for a combined analysis.[9]

Cosmetic surgery is distinct from other medical fields because it is more commonly performed to correct an external defect for aesthetic and psychosocial reasons than it is to address an underlying pathology associated with an individual’s health.[10-12] Measuring the effectiveness of an intervention from the patient’s perspective is often more important than objective physical measurement in evaluating treatment success.[6] Consequently, evaluating the use of validated patient-reported outcome measures that assess patient satisfaction and quality of life is an important component of assessing the quality of evidence in a plastic surgery study.[13] Selecting the appropriate patient-reported outcome measure based on context of use and key concepts to be measured in a study is an important initial step in study design.[14] However, challenges still remain in that patients and surgeons may have discordant perceptions of outcomes after cosmetic surgery, thus making results difficult to interpret.[15]

One recent publication illustrates the importance of evaluating a study’s methods when interpreting the results. The study was undertaken to evaluate the risk of rare systemic harms associated with silicone breast implants by analyzing publicly available data from long-term postapproval studies of silicone breast implants.[16] Yet, careful reading of the methods and results sections reveals several methodological flaws that lower the quality of evidence in this study. For example, the preponderance of data supporting the authors’ conclusions of increased risks of systemic harms was derived from one study of 41,342 patients with a follow-up rate of 20%.[16] The extremely low follow-up rates resulted in a lack of outcomes data for the majority of the enrolled patients and thereby introduced substantial bias into the population.[17]

Level of Evidence Classification Scales Used by Plastic Surgery Journals

The use of hierarchical systems to classify a study’s LOE has become a foundation of the EBM movement.[18] The LOE rating scales, which stratify the quality of evidence from strongest to weakest on the basis of the study design and its susceptibility to bias, were first implemented by plastic surgery journals in 2011.[4,18] Over the past decade, a number of variations of LOE grading scales have been published[5,6,19] and some are currently used by plastic surgery journals.[4,5,19] Plastic and Reconstructive Surgery and the Aesthetic Surgery Journal use the grading recommendations published by the American Society of Plastic Surgeons (ASPS) to assign an LOE rating to eligible publications (Table 1).[4,18] JAMA Facial Plastic Surgery uses a modified version of the LOE ratings that were established by the Oxford Centre for Evidence-Based Medicine (OCEBM) (Fig. 1).[5,19] Both grading systems are specific to the type of research question being asked (ie, prognostic/risk, diagnostic, or therapeutic questions). In general, both assign the highest LOE to systematic reviews and high-quality cohort studies or RCTs and lower levels to case reports.[4,5]

Table 1.

American Society of Plastic Surgeons’ Levels of Evidence Grading Recommendations4

Fig. 1.

Levels of evidence established by the OCEBM.[5] Modified with permission from OCEBM Levels of Evidence Working Group. *Level may be graded down on the basis of study quality, imprecision, indirectness; because of inconsistency between studies; or because the absolute effect size is very small. Level may be graded up if there is a large or very large effect size. †A systematic review is generally better than an individual study.

Basic science, bench work, and cadaveric and animal studies are deemed nonratable by the ASPS and OCEBM scales because they are not considered to be directly applicable to patient treatment decisions.[4,19] The OCEBM scale allows for downgrading the LOE rating on the basis of study quality, imprecision, or indirectness; because of inconsistency between studies; or because the absolute effect size is very small.[5] The ASPS scale gives higher ratings to “high-quality” studies and lower ratings to “lesser quality” studies. However, neither scale clearly defines the factors that comprise a high-quality study.

Several important indicators of the methodological quality of plastic surgery studies (ie, power analyses, adjustments for confounders, follow-up, and scale reliability) are not explicitly considered in the ASPS and OCEBM LOE scales.[4-6] The Cosmetic Level of Evidence and Recommendation (CLEAR) scale (Table 2) incorporates several of these factors into a modified version of the ASPS LOE scale.[6] Although the CLEAR scale has not yet been adapted by any journals, it may be a helpful tool for plastic surgeons when assessing quality of evidence in the literature.

Table 2.

Cosmetic Level of Evidence and Recommendation Scale6

LOE in the Real World

Despite the implementation of LOE classification systems by plastic surgery journals, the percentage of level I/II studies remains low.[20,21] An analysis of studies published in Plastic and Reconstructive Surgery in 2013 found that only 3% of the studies were level I and 15% were level II.[20] The paucity of level I plastic surgery publications has been attributed to the difficulty of conducting RCTs to address clinical questions in surgery compared with other fields.[20,22] Although the numbers of RCTs reported in aesthetic surgery journals have increased several-fold since the 1980s,[23,24] many questions in the practice of plastic surgery remain unamenable to RCTs. One analysis of treatment evaluation questions in the surgical literature estimated that 61% of surgical questions are not suitable for randomized trials.[25] In addition, differences in operative technique from surgeon to surgeon are often unavoidable, making it difficult to control for some methodological aspects of an intervention.[6,23,24] Placebos, sham surgeries, and blinding are usually not feasible, and randomization is not well accepted by patients and surgeons.[12,23,24,26,27] Surgeons may feel it is unethical to randomize patients into different groups if one of the treatments is known to be inferior.[6,27] Study designs that introduce a control by involving different interventions on contralateral sides of the face or body may produce asymmetry.[6]

Historically, publications with lower LOE ratings have been relatively more valuable in plastic surgery than in many other specialties.[28] A review of the 50 most cited papers published in the 20 highest impact plastic surgery journals between 1945 and 2015 found that none of the top 50 publications were rated level I or II.[28] The majority (84%) of the top 50 cited papers was categorized as level IV or V evidence and the average LOE rating for the top 50 papers was 4.28.[28] Although there is a need for more level I/II studies in the literature, high-quality observational studies (ie, case series and retrospective cohort studies), which by default receive lower LOE ratings,[4] play an essential role in evidence-based plastic surgery.[29] Observational studies can provide valuable data for patient populations that would likely be excluded from RCTs, such as those with complex, multifaceted conditions and comorbidities.[29] Given that the study type and LOE rating do not necessarily reflect the overall importance of a study’s results, it is the reader’s responsibility to critically appraise the significance and accuracy of study conclusions in the context of the methodology, presented results, and other publications in the field.

CONCLUSIONS

Reporting the LOE in a research publication can help to highlight the quality of the research and the potential for bias, so that the reader may prioritize information accordingly. However, the LOE classification should not be considered an absolute index of the quality of evidence and does not preclude careful evaluation of the study’s methods and results in the context of the authors’ conclusions. As the plastic surgery community continues to adapt to the concept of EBM, surgeons need to be continually mindful of how to appropriately interpret research findings and assess the utility of applying the results to their practice.

ACKNOWLEDGMENTS

Writing and editorial assistance was provided to the authors by Lela Creutz, PhD, of Peloton Advantage, Parsippany, N.J., and was funded by Allergan plc, Dublin, Ireland.