The Plastic Surgery In-service Training Examination: An In-depth Reference Analysis.

The goal of this study was to characterize the references provided as supporting evidence of the Plastic Surgery In-service Training Examination (PSITE) syllabi, including those on the novel "core surgical principles" section.
METHODS: We analyzed the references from five consecutive PSITE Examination syllabi (2016-2020). We collected the following information from each question: question section, total number of references, and source of publication of each reference.
RESULTS: We analyzed 1250 questions and 3436 references. Plastic and Reconstruction Surgery was overall the most frequently referenced journal followed by Journal of Hand Surgery (American Volume) and Annals of Plastic Surgery. The most commonly referenced textbooks were Plastic Surgery (by Neligan), Green's Operative Hand Surgery, and Grabb and Smith's Plastic Surgery. Regarding the "core surgical principles" section, Plastic and Reconstruction Surgery remained the most frequently cited journal, followed by the Journal of the American Medical Association, New England Journal of Medicine, Annals, and Aesthetic Surgery Journal. "Core surgical principles" contained the largest number of unique journals (n = 209) among all test sections. Within the "core" section, Statistics in Medicine was the most frequently referenced textbook followed by Grabb and Smith's Plastic Surgery.
CONCLUSIONS: The main plastic surgery texts and literature were used to support approximately half of the answers within the "core surgical principles" section. The references within this section originated from the largest number of different journals, thus highlighting the breadth and variety of this content and the challenges in preparing for this section.

INTRODUCTION

The Plastic Surgery In-service Training Examination (PSITE) is administered to plastic surgery residents annually. This examination allows residents to assess their knowledge of core plastic surgery topics, gauge their knowledge against national peers, and serve as an indicator of areas needing concentrated educational efforts. Recent work has demonstrated that performance on the PSITE can predict success on the American Board of Plastic Surgery Written Examination,[1] thereby further increasing motivation to prepare for and perform well on this examination. The American Society of Plastic Surgeons’s In-service Examination Committee prepares the questions. The examination is divided into five sections: comprehensive, hand and lower extremity, craniomaxillofacial, cosmetic and breast, and core surgical principles. Each section contains 50 questions, totaling 250 questions per examination. Each question is accompanied by a list of references providing supporting evidence for the correct answer.

Silvestre et al[2] analyzed the supporting references from 10 consecutive examination syllabi from 2006 to 2015, reporting on the most frequently referenced sources, publication lag, and level of evidence,[3] among other variables. The authors evaluated the craniomaxillofacial,[4] cosmetic,[5] and lower extremity[6] sections, delineating question taxonomy, subject matter, and focus.[7] The most recent of these investigations reports up to the 2015 PSITE examination.

The “core surgical principles” section was added to the PSITE in 2016.[7] This section covers topics and principles of general surgery, anesthesia, dermatology, oculoplastic surgery, oral and maxillofacial surgery, and orthopedic surgery. To our knowledge, assessment of the “core surgical principles” references has not been performed previously. Given the broad variety of topics covered under the “core surgical principles,” such analysis would help plastic surgery residents better prepare for the PSITE.

In addition to analyzing the newly instituted “core principle section,” the secondary aims of this study were to determine (1) the most frequently referenced journal and textbooks, (2) the publication lag of journal and textbook references, and (3) the impact factors for referenced journal articles in five consecutive PSITE syllabi. It is our hope that such an analysis will allow for better preparation for the examination and more relevant focus of educational efforts on the part of residents, fellows, and program directors.

METHODS

We evaluated the references of five consecutive PSITE syllabi (2016–2020). A database was created for all the references used for each question of the examination in the 5 included years.

Each year, some questions are omitted from scoring due to question inaccuracies, poor performance, ambiguous wording, or nondiscriminatory nature of the question. However, we elected to include these questions to capture the entire spectrum of references provided by question writers. Institutional review board approval was not required for this study.

Two authors (RE and SJ) collected the following information from each question: year of the examination, question section, the total number of references per question, source of publication of each reference, and year of publication of each reference. Reference sources were divided into journal publication, textbooks, or other. The “other” category included any source that did not fit the journal or the book category, such as podcast, electronic journal, or on-line publication. We also collected the journal impact factor (JIF) for each journal reference based on the Journal Citation Reports 2018.[8] Articles published more than 20 years before the examination date and journals referenced fewer than four times throughout this 5-year period were excluded to eliminate outliers. Publication lag was calculated as the time between the source publication (book or journal) and the date of the examination. To evaluate the source of questions for each of the sections, we identified the top five journals and top three textbooks that supported evidence for all references and “unique” questions in each of the five sections. A journal was considered a source for a unique question if one of the references of that question was from the journal of interest.

STATISTICAL ANALYSIS

Mean and median were used to summarize continuous variables, while percentages and proportions were used to present categorical data. We analyzed the overall references identified independently and per unique question (eg, if a question has two references from the same journal, the overall number of references counted these two references twice, while analysis on the unique question level counted the journal of the two references once). To compare the JIF and publication lag among the sections and examination years, we used the Kruskal–Wallis one-way analysis of variance test. P < 0.05 was considered statistically significant. Statistical analysis was performed using JMP Pro 14 software (JMP, Pro 14, SAS Institute Inc, Cary, N.C.).

RESULTS

We analyzed a total of 1250 questions and 3436 references. The mean number of references per question was 2.7 (SD 0.9, range 1–9) (Fig. 1). “Hand and lower extremity” had the highest mean number of references per question (3 ± 0.9), whereas “core surgical principles” had the lowest (2.5 ± 0.9). The majority of references were comprised of journal citations (n = 3034, 88%), followed by textbooks (n = 309, 9%), and other references (n = 93, 2.7%) with some differences among the sections. Although other references constituted 10% of the references in “core surgical principles,” it made up only 0.1% of the “hand and lower extremity” references (P < 0.0001). The mean JIF for all references was 5.1 ± 13. There were significant differences in JIF between the sections (P < 0.0001) with references in “core surgical principles” having the highest mean JIF (9.6 ± 19.5), whereas references in “hand and lower extremity” had the lowest (mean JIF, 3.1 ± 4.8). Table 1 demonstrates the number of references, sources, and JIF per section and for all included references. Figure 2 depicts the source of references per year.

Fig. 1.

Number of references per question.

Table 1.

JIF and Reference Type by Section

Section	Total No. References [N(%)]	Total No. Journal References [N(%)]	Total No. Book References [N(%)]	Total No. Other References [N(%)]	No. References per Question [Mean (SD)]	Journal IF[Mean (SD)]
Comprehensive	725 (21)	657 (91)	58 (8)	10 (1)	2.9 ± 1	4.5 ± 11.5
Hand and lower extremity	756 (22)	672 (89)	83 (11)	1 (0.1)	3 ± 0.9	3.1 ± 4.8
Craniomaxillofacial	655 (19)	578 (88)	71 (11)	6 (1)	2.6 ± 0.8	4 ± 13.4
Breast and cosmetic	680 (20)	607 (89)	56 (8)	17 (3)	2.7 ± 0.9	5.2 ± 13.1
Core surgical principles	620 (18)	520 (84)	41 (7)	59 (10)	2.5 ± 0.9	9.6 ± 19.5
Total	3436	3034 (88)	309 (9)	93 (27)	2.7 ± 0.9	5.1 ± 13

Fig. 2.

Source of references by year.

The median journal publication lag for the all references was 6 years (IQR, 3–10). There were significant differences between the sections with “hand and lower extremity” having the longest publication lag [median, 7 years; IQR, 4–11, P = 0.0002]. The overall median book publication lag was 7 years (IQR, 4–11) with “comprehensive” having the longest lag [median, 10; IQR, 5–12; P = 0.018]. Table 2 summarizes publication lag by examination section. No difference in publication lag was identified between the examination years for both journal publications (P = 0.19) and textbooks (P = 0.56).

Table 2.

Publication Lag by Section

Section	Total No. References	Publication Lag Journals [Mean (SD) Years]	Publication Lag Journals [Median (IQR) Years]	Publication Lag Journals [Range Years]	Publication Lag Book [Mean (SD) Years]	Publication Lag Book [Median (IQR) Years]	Publication Lag Book [Range Years]
Comprehensive	725	8 (7)	6 (3–10)	1–52	11 (11)	10 (5–12)	2–83
Hand and lower extremity	756	10 (10)	7 (4–11)	0.2–1	9 (11)	7 (3–9)	1–101
Craniomaxillofacial	655	8 (9)	5 (3–10)	0.5–91	13 (24)	7 (5–11)	1–174
Breast and cosmetic	680	8 (7)	6 (3–10)	1–65	8 (5)	7 (5–10)	1–28
Core surgical principles	620	8 (7)	6 (3–10)	0.5–48	10 (6.6)	7 (5–13)	1–31
Total	3436	8 (8)	6 (3–10)	0.2–91	10 (14)	7 (5–11)	1–174

Plastic and Reconstruction Surgery (PRS) was overall the most frequently referenced journal, constituting 33% of all references and supporting 50% of unique questions. This was followed by Journal of Hand Surgery (American Volume) (J Hand Surg Am) (7% of all references and 11% of unique questions) and Annals of Plastic Surgery (Annals) (3% of all references and 7% of unique questions). The most commonly referenced textbooks were Plastic Surgery (by Neligan) (16% of all references), Green’s Operative Hand Surgery (12%), and Grabb and Smith’s Plastic Surgery (8%). Table 3 outlines the most frequently referenced journals and textbooks for each section. Specifically regarding the “core surgical principles” section, PRS remained the most frequently cited journal, representing 23% of references and supporting 33% of unique questions. This was followed by the Journal of the American Medical Association (JAMA), New England Journal of Medicine (NEJM), Annals, and Aesthetic Surgery Journal (ASJ), each comprising 2% of all reference and 4% of unique questions. “Core surgical principles” was the section with the largest number of unique journals (n = 209). Regarding textbooks within the “core surgical principles” section, Statistics in Medicine was the most frequently referenced (7% of all references), whereas Grabb and Smith’s Plastic Surgery, Guyton and Hall Textbook of Medical Physiology, Plastic Surgery (by Neligan), Principles and Practice of Pediatric Plastic Surgery, and Essential Medical Statistics represented 5% of references each.

Table 3.

Most Frequently Referenced Journals and Textbooks for Each Section

Section		Journal	No. References (% of Total References)	No. Unique Questions Supported (% of Questions)	Total No. Journals per Section	Book	No. References
Comprehensive	1	PRS	315 (48)	190 (76)	154	Plastic Surgery	23 (40)
	2	Annals	34 (5)	30 (12)		Grabb and Smith	6 (10)
	3	JPRAS	24 (4)	24 (10)		Reconstructive Surgery	6 (10)
	4	J Hand Surg Am	14 (2)	11 (4)
	5	J Recon Micr	13 (2)	11 (4)
Hand and lower extremity	1	J Hand Surg Am	198 (29)	126 (50)	137	Green Operative Hand Surgery	37 (45)
	2	PRS	124 (18)	86 (34)		Plastic Surgery	8 (10)
	3	Hand Clinics	44 (7)	39 (16)		Atlas of Human Anatomy	6 (7)
	4	Annals	19 (3)	19 (8)
	5	J Bone Joint Surg Am	17 (3)	16 (6)
Craniomaxillofacial	1	PRS	132(23)	94 (38)	193	Grabb and Smith	10 (14)
	2	J Craniofac Surg	49(8)	37 (15)		Cummings Otolaryngology: Head and Neck	7 (10)
	3	J Oral Maxillofac Surg	28 (5)	22 (9)		Plastic Surgery	6 (8)
	4	JPRAS	16 (3)	15 (6)
	5	Annals	16 (3)	13 (5)
Breast and cosmetic	1	PRS	300 (49)	174 (70)	133	Plastic Surgery	9 (16)
	2	ASJ	49 (8)	39 (16)		The Art of Aesthetic Surgery	5 (9)
	3	Aesthetic Plast Surg	20 (3)	18 (7)		Grabb and Smith	4 (7)
	4	JPRAS	18 (3)	17 (7)		Current Therapy in Plastic Surgery	(4) 7
	5	Annals	16 (3)	12 (5)
Core surgical principles	1	PRS	121 (23)	82 (33)	209	Statistics in Medicine	3 (7)
	2	JAMA	12 (2)	10 (4)		Grabb and Smith	2 (5)
	3	NEJM	11 (2)	10 (4)		Essential Medical Statistics	2 (5)
	4	Annals	10 (2)	10 (4)		Guyton and Hall Physiology	2 (5)
	5	ASJ	10 (2)	9 (4)		Plastic Surgery	2 (5)
						Principles and Practice of Pediatric Plastic Surgery	2 (5)
Overall	1	PRS	992 (33)	626 (50)	578	Plastic Surgery	48 (16)
	2	J Hand Surg Am	219 (7)	143 (11)		Green Operative Hand Surgery	37 (12)
	3	Annals	95 (3)	84 (7)		Grabb and Smith	25 (8)
	4	JPRAS	78 (3)	75 (6)
	5	ASJ	71 (2)	57 (5)

J Recon Micr, Journal of Reconstructive Microsurgery; J Bone Joint Surg Am, Journal of Bone and Joint Surgery (American Volume); J Craniofac Surg, Journal of Craniofacial Surgery; J Oral Maxillofac Surg, Journal of Oral and Maxillofacial Surgery; Aesthetic Plast Surg, Aesthetic Plastic Surgery.

DISCUSSION

In-service training examinations are an important component of surgical residencies. They provide a standardized assessment of resident knowledge, and performance has been correlated with future written board examination pass rates across multiple disciplines,[9-12] including the American Board of Plastic Surgery Written Examination.[1] As such, there have been a number of publications evaluating various interventions to improve in-service training examination performance.[12,13]As the timing of the written examination for the American Board of Plastic Surgery is now given before the end of a trainee’s final year of residency, review of previous PSITEs may serve as an important resource to identify surgical principles that are high yield for study. The PSITE content has been investigated in the past.[2-6,14] However, this analysis occurred before the institution of the “core principles section.”

The results of our study reiterated that importance of the main plastic surgery journals as study tools and primary sources of information. PRS was referenced to support 50% of all unique questions, followed by JHS, Annals, Journal of Plastic and Reconstructive Surgery (JPRAS), and ASJ. This is similar to Silvestre et al’s[2] previous report. However, our study provides a more comprehensive breakdown of the most referenced journals overall and by section. Again PRS was the most frequently cited journal, though the addition of JPRAS is new as this was not listed in the top three of any section in their study. These results also reinforce prior reports regarding the utility of textbooks. Both in this study and Silvestre et al’s,[2] Plastic Surgery (by Neligan), Green’s Operative Hand Surgery, and Grabb and Smith’s Plastic Surgery were the most frequently referenced texts overall.

With regard to the “core surgical principles” section, PRS, JAMA, NEJM, Annals, and ASJ were the most frequently cited, thus highlighting the utility of reviewing core plastic surgery journals in preparing for this section. However, the “core” section also referenced the most unique journals (n = 209), further highlighting the broad scope of this section and the resulting challenges in effectively preparing for it and creating a standardized curriculum to teach it. Core plastic surgery textbooks were also frequently referenced, with Plastic Surgery (by Neligan), Grabb and Smith’s Plastic Surgery, and Principles and Practice of Pediatric Plastic Surgery comprising 3 of the 5 most referenced texts. As highlighted by prior authors,[3] “tested material [on the PSITE] reflects the [American Society of Plastic Surgeon’s] vision for core curriculum training in plastic surgery.” The prominence of plastic surgery texts within the “core surgical principles” section, in place of more traditional general surgery texts, follows the continued gradual deviation of plastic surgery training away from its general surgery origins. Modern plastic surgeons, no longer a form of specialist general surgeon, now rely on a set of unique core surgical principles tailored to the demands of the profession. Accordingly, these salient principles are often highlighted within core plastic surgery texts and journals. Although overall preparation for the “core principles section” may be more challenging, this investigation suggests that reading the main plastic surgery texts and literature may be sufficient to identify the commonly test core surgical principles. Inclusion of readings focusing on basic statistics and pertinent physiology may be of additional benefit. Although it would be challenging to incorporate articles from other major journals outside the field of plastic surgery (such as JAMA and NEJM), as these are often outside the plastic surgeon’s scope of practice, it is prudent to identify pertinent landmark articles that may be published outside our conventional literature. PRS often dedicates a portion of each issue to highlight relevant landmark articles from other journals. These spotlights may provide guidance on which articles one should use to focus one’s studies or journal club discussions, and it would be interesting to see if there is any correlation between these selected articles and those used as references for “core surgical principles” questions. Core knowledge obtained on general surgery and related subspecialty rotations during the early year of training should highlight some of these key principles, as well. Yet with the disappearance of traditional general surgery texts from even the “core” component of contemporary in-training examinations, one must wonder whether trainees would be better served by limiting their time spent on general surgery services in favor of increased plastic surgery exposure.

The mean journal impact factor for all references was 5.1. The “core surgical principles” section had the highest mean JIF at 9.6, whereas the “hand and lower extremity” section had the lowest at 3.1. This represents an increase from that reported previously,[2] where the mean JIF for all sections was 2.3 with the hand section again lowest at 1.7. This could be explained by the increase in the number of plastic surgery journals and their JIF over the last 22 years, as shown in our previous study.[15] The addition of the “core surgical principles” section could also partially explain the higher JIF in our study, as the inclusion of general medical journals (such as NEJM and JAMA) would inflate the JIF, given their much higher JIF compared with that of plastic surgery journals.

Overall, our study reinforces the importance of the core plastic surgery journals and textbooks that are anecdotally emphasized in our training program. We believe that knowledge of the most commonly referenced resources used for in-service questions will help guide preparation, and a better understanding of the journals or textbooks from which references originate may provide insight for educators and program directors to help design effective curricula and/or reading lists. As suggested by Frojo et al,[16] theses analyses support the need for early initiation of critical literature analysis (ie, journal club) and a curriculum including both established principles and contemporary trends. Based upon these data, perhaps journal club articles should focus solely on publications from PRS and JHS. Curricula outlines should parallel Plastic Surgery (by Neligan), Green’s Operative Hand Surgery, and/or Grabb and Smith’s Plastic Surgery, while also including components dedicated to statistics, pertinent physiology, and other key surgical principles (ie, breast cancer staging).

The relative infancy of the “core surgical principles” section of the PSITE makes it difficult to interpret the correlation, if any, between a test taker’s success on this section and his/her future success on the American Board of Plastic Surgery Written Examination. Future studies, carried out once several residency classes have had the opportunity to complete the PSITE “core surgical principles” section and their written board examinations, may help shape future iterations of this section and improve its utility as a harbinger of written board performance. Interestingly, as currently structured, the American Board of Plastic Surgery Written Examination lacks a section completely dedicated to core surgical principles.[17] The 20% of the PSITE now allocated for the assessment of “core surgical principles” dwarfs the 5% of the written board examination dedicated to “plastic surgical aspects of specific related disciplines,” which contains questions on critical care, anesthesia, immunology, and pharmacology and seems the most logical extension of the “core surgical principles section.” Future studies may examine whether the current allocation of one of every five questions of the PSITE potentially disproportionately outpaces of the demand for core surgical knowledge placed on surgeons who sit for the written board examination.

This study is not without limitations. This assessment only encompasses the previous 5 years of examinations. The PSITE may undergo additional evolutions, so previous examinations may not be predictive of future questions. This study does not provide an in-depth analysis of the questions or the content for individual sections. Despite the weakness, we hope these findings may help guide study and curricula development to be more high-yield and efficient, and ultimately improve performance.

CONCLUSIONS

The “core surgical principles” section has comprised 20% of the PSITE examinations since 2016. We found that main plastic surgery texts and literature were used to support approximately half of the answers within this section. The references within the “core surgical principles” section originated from the largest number of different journals, thus highlighting the breadth and variety of the “core” content and the challenges in preparing for this section. Future studies correlating performance on the “core principles section” and success on the American Board of Plastic Surgery Written Examination would be beneficial to validate its content.