AAPS Podium Presentations-Has the Level of Evidence Changed over the Past Decade?

An increase in the number and quality of randomized controlled trials (RCTs) and trends toward higher levels of evidence (LOE) in the plastic surgery literature has been reported; however, there has not been a specific focus on the LOE of presentations at scientific meetings. The purpose of this study was to ascertain trends in the LOE of studies presented at the annual meeting of the American Association of Plastic Surgeons.
METHODS: A hand search was conducted identifying all abstracts of podium presentations from 2009 to 2019. LOE, using American Society of Plastic Surgeons guidelines, were ascribed to each presentation, along with identification of any corresponding journal publications. RCTs were further analyzed using the 12-item Modified Consolidated Standard of Reporting Trials checklist.
RESULTS: Four hundred forty-one studies with a median LOE of 3 were included in the study. A non-significant improvement in the mean level of evidence was noted over time (P = 0.09) along with an increase in the number of level 2 studies (P = 0.589) and RCTs (P = 0.717). Level 1 studies were rare (0.91%) and of fair quality (median checklist score 8 out of 12). Seventy-two percent of abstracts resulted in publication, and the mean lag time to publication was 422 days.
CONCLUSIONS: A favorable trend is observed with respect to the mean LOE as well as the number of level 2 studies and RCTs over time. The importance of research meeting attendance to maintain up-to-date information is noteworthy, given the long lag time from presentation to publication.

INTRODUCTION

Evidence-based medicine (EBM) is defined as the “conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”[1] Incorporating EBM into plastic surgery research and clinical practice has become a major focus by leaders of the specialty over the past decade. Numerous articles have highlighted specialty-wide efforts toward this goal; the introduction of the “level of evidence pyramid” in Plastic and Reconstructive Surgery (PRS), which displays the level of evidence of clinical studies and which uses the American Society of Plastic Surgeons Levels of Evidence Rating Scale,[5] is a prominent example for the increasing importance being ascribed to EBM principles within plastic surgery.[2-6] Multiple studies have reported an increase of the level of evidence of plastic surgery research over time, likely, in part, due to these specialty-wide initiatives.[7,8] Furthermore, a greater number of randomized controlled trials (RCTs) in plastic surgery have been published over time; this is particularly important becaue RCTs are considered the gold standard for investigating the effects of an intervention.[9-11] Furthermore, there is some evidence that the quality of reporting of RCTs has improved as well.[12,13] We sought to determine whether these trends found in journal publications carried forward to studies presented at the American Association of Plastic Surgeons (AAPS) annual meeting, which represents one of the most competitive forums for plastic surgery research.

The primary aim of the present study was to analyze the level of evidence of podium presentations at the AAPS annual meeting over an 11-year period (2009–2019). A secondary aim was to critically review the quality of reporting of RCTs presented at the meeting.

METHODS

Accessing the AAPS website (https://meeting.aaps1921.org/Archives/), 2 independent reviewers (AC and RW) identified all archived abstracts of podium presentations at the AAPS Annual Meeting from 2009 to 2019. Only clinical studies were considered for analysis. Additionally, corresponding journal publications were identified, and the lag time from AAPS presentation to journal publication was calculated.

The level of evidence of all abstracts was determined using American Society of Plastic Surgeons guidelines.[5] All discrepancies were resolved by consensus (ie, discussion among all investigators of the study).

Next, all RCT presentations and publications during the study period were identified and analyzed. RCTs were identified using the well-established Cochrane criteria:[14]

Participation of living human beings in the study;

Analysis of an intervention related to health care;

Investigation of an intervention (experimental study);

Inclusion of a comparison intervention;

Assignment of participations to intervention groups by randomization.

A 12-item Modified Consolidated Standard of Reporting Trials checklist was used to determine the quality of reporting of RCTs.[15] Each checklist item was given a score of 1 versus 0, based on whether the item was reported versus not reported, respectively. Thus, the maximum possible total score was 12 if every item was reported. Scores of 4 or less, 5 to 8, and 9 or greater indicated “less than fair,” “fair,” and “good” quality of reporting, respectively.

Statistical analysis of trends in level of evidence by year and category as well as quality of reporting of randomized clinical trials was performed using Chi-squared and Fischer exact tests. Difference in lag time was calculated with ANOVA. P < 0.05 was considered statistically significant.

RESULTS

A total of 574 abstracts, of which 441 met inclusion criteria, were presented at AAPS meetings between 2009 and 2019. Of these presentations, 72% (N = 318) were subsequently published in the following scientific journals: PRS (N = 161; 36.5%), PRS Global Open (N = 24; 5.4%), Annals of Plastic Surgery (N = 22; 5.0%), Aesthetic Surgery Journal (N = 16; 3.6%), Journal of Plastic, Reconstructive & Aesthetic Surgery (JPRAS; N = 7; 1.6%), and other (N = 88; 20.0%).

The mean lag time from presentation to publication was 422 days (range: 582–2763, with negative values representing publication before presentation at the meeting). Lag time varied significantly depending on the journal (P < 0.001); PRS Global Open published articles within the shortest time period (mean, 190 days), whereas PRS and JPRAS had a mean lag time of 401 days and 941 days, respectively.

The majority of presentations represented clinical studies with an evidence level 3 (N = 202; 45.7%); level 1 studies constituted <1% of all presentations (N = 4; 0.9%) (Fig. 1). Of note, a non-significant improvement in mean level of evidence was noted over the study period from a mean level of 3.50 in 2009 to a mean level of 3.25 in 2019 (P = 0.090, Fig. 2). Although an increase in the number of published level 2 studies (P = 0.589) and RCTs (P = 0.717) was noted, these were not found to be statistically significant. In total, 2–3 level-2 studies were presented each year from 2009 to 2014, increasing to 6–8 from 2015 to 2019. One RCT was presented in 2009, 2010, and 2019, whereas 2–3 were presented each year from 2014 to 2016. No RCTs were presented at the remaining five annual meetings. No significant increase in the number of published level-1 studies was noted during the study period (P = 0.609) (Fig. 3). No more than 1 level-1 study was presented in 2009, 2010, 2014, and 2016, with no level-1 studies identified in the last 3 years.

Fig. 1.

Distribution of levels of evidence of all studies.

Fig. 2.

Mean level of evidence of all studies over time, indicating a non-significant improvement over time (P = 0.09).

Fig. 3.

Non-significant changes in the number of level 1 studies, level 2 studies, and RCTs over time.

Ten studies (2.3%) were identified as RCTs using Cochrane criteria; of these, 9 studies were published in plastic surgery journals. These 9 published studies were further analyzed using the 12-item Modified Consolidated Standard of Reporting Trials Checklist; the single unpublished RCT was not analyzed due to the lack of sufficient information. Four RCTs were classified as level 1 studies with a score of 9 or above, and 5 studies were classified as level 2 studies. In 2015, all 3 RCTs were classified as level 2. The median score of all RCTs was 8 (range, 5–11). No statistically significant trends in the median 12-item checklist score were noted over time (P = 0.245). The individual items of the 12-item checklist were analyzed in greater detail to ascertain areas of under-reporting (Fig. 4). Although all studies reported trial design and the majority reported statistical significance (N = 8; 88.9%), eligibility criteria (N = 7; 77.8%), definition of primary and secondary outcomes (N = 7; 77.8%), and sources of funding (N = 7; 77.8%), they were most frequently deficient in reporting allocation concealment (N = 2; 22.2%) and intention-to-treat analysis (N = 2; 22.2%).

Fig. 4.

Percentage of trials reporting each checklist item.

DISCUSSION

The creative nature of plastic surgery is reflected by the fact that advances in the specialty have commonly originated from the contributions of individual surgeons who have reported their innovations via “low-level evidence” studies, such as retrospective case reports and case series.[16,17] Although the importance of low-level evidence is beyond dispute, plastic surgeons have recognized that the rigorous, data-driven approaches that have been adopted in other fields of medicine are important for improving decision-making and, ultimately, ascertaining “the best course of care for a patient.”[6,18] As such, multiple efforts have been made in recent years to increase awareness of evidence-based plastic surgery as well as increase the overall level of evidence of studies in the plastic surgery literature. In particular, the Colorado Springs EBM Summit in 2010 convened leaders in plastic surgery with experts in EBM, representing a sea change in the emphasis on EBM in the specialty.[2-6] The Summit produced specialty-specific level of evidence pyramids for diagnostic, prognostic/risk, and therapeutic studies, and PRS has since reported the level of evidence for qualifying studies published in the journal.[5]

Despite these efforts, adoption of EBM principles into plastic surgery literature has been variable. Several studies have investigated the levels of evidence of clinical studies published in major plastic surgery journals, whereas others have focused on merely identifying the prevalence of RCTs in the literature.[7,8,10-13,19-22] Sinno et al compared all articles published in PRS, Annals of Plastic Surgery (Annals), Journal of Plastic, Reconstructive, and Aesthetic Surgery (JPRAS), and American Journal of Aesthetic Surgery (Aesthetic) in 2007. They found the average level of evidence of articles in these 4 prominent journals to be 3.2, with 2.2% of them representing level 1 evidence.[8] Multiple studies have reported positive trends in the number of RCTs published in the plastic surgery literature over time; however, although the number of RCTs has increased, the quality of reporting remains deficient.[10-13,20,22]

Although prior studies have focused on publications in major plastic surgery journals, few have investigated trends of clinical studies presented at plastic surgery meetings. Chuback et al analyzed presentations at 3 major plastic surgery meetings in 2010 and 2011, and found a low overall number of level 1 evidence studies (1.6%)[23]—an observation that mirrors the findings of this study. There is a lack of data on whether the level of evidence at meetings has improved in the last decade, particularly after the Colorado EBM summit in 2010. Therefore, we chose to focus on podium presentations at the AAPS meeting. The AAPS meeting was chosen because it represents all plastic surgery subspecialties and is one of the most competitive forums for plastic surgery research, as evidenced by rather low acceptance rates for podium presentations, namely ~13% and ~10% for 2019 and 2020, respectively.

Our analysis of podium presentations corroborates previous reports. The majority (45.7%) of presented clinical studies were level 3 evidence, with only 0.9% meeting criteria for level 1 evidence. However, the most promising finding of our study was that the mean level of evidence improved over time from 3.50 to 3.25, albeit without being statistically significant. It is noteworthy that our observations represent a continuation of the favorable trend reported by Loiselle et al, who reported on an improvement of the mean level of evidence from 4.42 to 4.16 from the year 1983 to 2003 in PRS.[7] Our study suggests continued improvement from 2009 to 2019. Additionally, both the number of level-2 studies and the number of RCTs displayed a favorable trend. Taken together, these results indicate that research presented at the AAPS annual meetings is trending toward improved levels of evidence, although the highest level of evidence remains a rarity. The lack of statistical significance of our findings could, however, also be interpreted in a different manner—that is, no real changes have occurred and more effort is indicated to improve the levels of evidence presented. Although the latter demand (ie, increased effort to improve the levels of evidence) is valid, it is also well known that changes occur rather slowly over time. Therefore, we are overall rather encouraged by the favorable, yet non-significant, trend observed.

Shah et al suggest that although increasing the level of evidence in plastic surgery research is important, RCTs are inherently challenging to perform in all surgical subspecialties, and that only 40% of surgical interventions could possibly be studied using RCTs.[18,24] Similarly, Burns et al emphasize that “readers not assume that level 1 evidence is always the best choice or appropriate for the research question.” Rather than stressing the need for a greater number of RCTs, both authors argue that poor-quality studies can lead to artifacts and inappropriate intervention, instead calling for higher quality of evidence at all levels in plastic surgery.[4,25]

Although it is challenging to assess the quality of reporting of all levels of evidence, quality metrics for reporting RCTs are well established.[15,26] To evaluate the quality of reporting of the RCTs presented at the AAPS meetings during the study period, we utilized the 12-item Modified Consolidated Standard of Reporting Trials checklist.[15,27] Different scales for assessing the quality of reporting of clinical trials exist and have been used to evaluate the plastic surgery literature; in 2 separate studies, Veiga et al utilized the Jadad scale to evaluate RCTs in plastic surgery journal articles from 1966 to 2003 as well as 2004 to 2008 and found an increase in quality of reporting between the 2 study periods.[20,28] However, the Jadad scale was not used in this study because it can be associated with poor interrater reliability and is not particularly suitable when applied to surgical trials, as it relies heavily on the criterion of double blinding.[12,29] Thus, we chose the 12-item Modified Consolidated Standard of Reporting Trials checklist; adherence to this checklist has been shown to improve quality of RCT reporting due to its ease of use and clear guidelines, as well as its lack of subjectivity compared with the Jadad scale.[27] The checklist was used by Becker et al to evaluate publications in PRS from 1990 to 2010.[12] The study found that the median quality of RCT reporting was 6 (“fair”) with a trend toward improved quality over time. Similarly, we found that the quality of RCT reporting in studies presented at the AAPS meeting was overall fair.

Our analysis of areas of deficiency demonstrated, in particular, 2 areas that were commonly underreported in RCTs, namely allocation concealment (22.2%) and intention-to-treat analysis (22.2%). These observations corroborate previous reports that identified these areas as being underreported in journal publications.[12] These areas indicate potential areas for improvement in future studies. Interestingly, a higher percentage of RCTs in our study compared with the study by Becker et al included sample size calculation (44.4% versus 10.1%)—one of the most important indicators of a good quality trial.[26] This is an encouraging observation because it gives further credit to clinical research in plastic surgery being conducted more rigorously.

A key limitation of this study is the relatively small number of abstracts that were analyzed. Given that only 9 RCTs underwent scoring with the checklist, it is difficult to draw conclusions on whether quality of RCT reporting is improving with time. A more general conclusion that can be drawn is that the median score in our study was 8, with a third of the studies achieving “good” reporting, suggesting that the overall quality of reporting at RCTs presented at the AAPS meeting during the study period is favorable compared with RCTs published in previous decades.[12] This may indicate that efforts to improve awareness of standardized reporting and quality measures in plastic surgery clinical research are coming to fruition. Another limitation is the fact that we restricted our analysis to the AAPS meeting only. One could argue that while all subspecialties are represented, the scientific focus of any given meeting may have affected the topics chosen for podium presentation, thus, potentially not reflecting the general trend in plastic surgery. However, as our study analyzed a period of 11 years, we believe that important trends of the specialty were likely captured. Finally, the data regarding lag time to publication can be influenced by a variety of factors, including journal-specific factors (such as the number of reviews required per submission) and author-specific factors (such as time of article submission).

A notable finding in this study was that although 72% of AAPS meeting podium presentations were eventually published in scientific journals, a significant delay was noted from presentation to publication across all journals (mean, 422 days). This observation underscores the importance of attending plastic surgery meetings as a mode of disseminating the most up-to-date research in the specialty. Interestingly, we identified that 33 studies had been published before presentation, although prior publication renders abstracts ineligible for meeting submission. Twenty-one of these studies were published within 1 year of presentation, likely indicating that the study article may have been submitted but not accepted to a journal during the time of abstract submission. Only 9 of these studies were published more than 1 year before presentation; further analysis showed that these were generally prospective in nature, and new information, such as an increase in number of enrolled subjects with updated study conclusions, had been added in the interim.[30]

In summary, although level 1 evidence remains rare in plastic surgery research, a favorable trend is observed with respect to the mean level of evidence as well as the number of level 2 studies and RCTs over time, although these trends are not statistically significant. Furthermore, all RCTs were characterized as having “fair” or “good” quality reporting, suggesting that quality of reporting continues to improve. We additionally highlight the importance of research meeting attendance to maintain up-to-date information, given the long lag time from presentation to publication. Taken together, research presented at the AAPS meeting reflects the sustained specialty-wide efforts to incorporate EBM into practice over the last decade and encourages continued progress.