Non-cognitive factors predicting success in orthopedic surgery residency.

Admissions to orthopedic surgery is a highly competitive process. Traditionally measures such as United States Medical Licensing Examination (USMLE) Step 1, class rank, AOA status have been major determinants in the ranking process. However, these traditional objective measures show mixed correlation to clinical success in orthopedic surgery residency. There have been several studies on the cognitive factors and their correlation with success in residency. However, it is clear that residency requires more than objective cognition, emphasizing complex social interactions that are influenced by non-cognitive variables including personality, work ethic, etc. This review aims to summarize the current understanding of non-cognitive factors influencing performance in orthopaedic surgical residency.

Introduction

Matching into an Orthopedic Surgery Residency program can be a truly challenging task for the rising fourth year medical student. The process has continued to become more competitive, especially since the introduction of 80-hour work week regulations. [1] The influx of applications each year, presents to program directors, the difficult task of filtering through vast amounts of data surrounding hundreds of applicants. After interviewing candidates, program directors and selection committees must form a rank list from an already-qualified pool of applicants.

Bernstein et al. surveyed residency program directors nationwide and found the three most important factors in residency selection were rotation at their institution, United States Medical Licensing Examination (USMLE) Step 1 score, and class rank.[2] While these three factors weigh heavily on program directors when making a selection, it’s unclear whether they actually predict success for orthopedic surgery residents. Step 1 score has been shown to have a variable predictive value, at best, for resident performance on The American Board of Orthopedic Surgery part I & II licensing examinations.[3-6] According to the USMLE website, the goal of Step 1 is to assesses whether you understand and can apply important concepts of the sciences basic to the practice of medicine.[7] USMLE Step 1, taken after the second year of medical school traditionally, was not designed as a predictor for performance after medical school. But the exam has evolved into a surrogate measurement because it is one of the only objective measures available to programs, and can be compared across all potential candidates.

Success in residency remains illdefined. Many studies utilize examination performance including ABOS part I&II pass rates, Orthopedic In-Training Exam (OITE) scores to define success.[3,4] However, standardized exams clearly are limited in their ability to assess domains outside of intellectual capacity such professionalism, surgical proficiency, patient interactions. Other studies have sought to define success by achievement of chief status, [8] a status not ubiquitous to all programs, limiting applicability across the literature. The heterogeneous definitions of success suggest that further agreement is needed amongst the community of what defines a successful orthopedic surgeon. Predictors for success in residency can be divided into two broad categories: Cognitive and noncognitive. Measurable cognitive domains include variables such as USMLE Step 1, Step 2 scores, class rank/grades, and even Medical College Admission Test (MCAT) and Scholastic Assessment Test (SAT) scores. Non-cognitive factors refers to a broad spectrum of domains which fall outside of the traditional intellectual domains but still impact performance. Non-cognitive factors to predict success during residency training are vast, representing an unexplored frontier of candidate selection and include personality traits, extra-curricular activities, and a newer attribute termed grit (defined as steadfast passion and perseverance for long-term goals).[9] In turn, these traits can be subjectively and objectively assessed during the interview day or during an audition rotation.

A retrospective review at New York University found that no pre-residency factors predicted success in residency, which was defined based upon ABOS part I pass rates.[8] This failure to find direct correlation to performance in residency highlights the challenge of selecting successful candidates from an already-accomplished pool of applicants. While there exists a great deal of literature regarding cognitive factors (i.e. USMLE, OITE, Class Rank) affecting success in the orthopedic resident, there remains limited research into other factors that are predictive of success. This review aims to synthesize the current research into non-cognitive and psychometric variables that help predict performance in residency.

Materials and Methods

Utilizing PubMed, a review of the available literature was conducted utilizing orthopedic, residency, and success, as keywords. This search yielded 42 articles, of which eight were initially deemed applicable, further studies were added as discovered in relation to the original eight articles. The articles were selected for inclusion if they addressed outcomes in residency performance. Studies were excluded if they failed to identify an outcome measure for success, if they were editorial in nature, or if they did not identify factors that could be assessed prior to residency. The resulting studies were reviewed to summarize the available data on non-cognitive variables predicting success in orthopedic residency.

Personality as a predictor

Personality is what drives human connection, and human connection is at the core of practicing as a physician. As a result, it would seem reasonable that an individual’s personality would be a driving factor surrounding decision making about matriculation into an Orthopedic Residency position. The amount of work compression that occurs as a result of the 80-hour work week, means interpersonal conflicts can have disastrous consequences for the teams associated with Orthopedic House Staff. These teams include relationships between other residents, faculty, nurses, and ancillary staff in the clinic and operating room, and personality deficits could pose a major hindrance to a productive work environment. Unfortunately, limited studies exist analyzing the effect of personality on resident performance.

Orthopedic surgery literature has shown that interviewers are more likely to rank similar personalities higher.[10] While it may be surmised that this is common sense, the effect on resident performance has not been determined. Literature in other fields does indicate that individuals perform better in environments suited to their personality.[11] Rating applicants with similar personalities higher, though subconscious, may selfselect for residents that are a better fit. However, this practice can lead to potential in-breeding and a program with limited diversity. Data from an academic general surgical training program suggests that personalities are not adequately gleamed from the application and interview process, and that pre-interview personality assessments potentially impact rank-order decisions.[12] The same study found that clear world and self-views were correlated with successful residents. Logically it would make sense that an introspective resident would possess the fortitude to improve upon perceived deficiencies. Further, understanding the external world is necessary when practicing in a system full of complexities.

A recent prospective, cross sectional study conducted across twelve orthopedic surgery residency programs helped gain insight into the effect personality has on performance in training.[13] The study looked at the Five Factor Model for personality, which includes agreeableness, extroversion, neuroticism, conscientiousness, and openness. The Five Factor Model has consistently been shown to be representative of personality sub-groupings, stable over time, and predictive of performance in the workplace. [11,14,15] In addition, the study looked at inquisitiveness and learning approach as two additional traits. The findings indicated that agreeableness, neuroticism, and learning approach were most closely associated with resident performance (Table 1). Interestingly, the analysis showed that significant disparities existed between facultyrated importance of personality traits and the existence of those traits within their residents. This study is important because it is the first in orthopedic literature to correlate personality traits with resident performance. Given the relative stability of the five factor personality traits, this presents an opportunity for screening applicants who are a suitable fit for respective programs. Further, the disparity between resident personality traits and faculty-expressed importance of various traits suggests that the selection process does not adequately account for personality at this time.

Table 1.

Personality factors accounting for majority of performance variance in overall performance, listed in descending contribution to variance.

Personality factors most predictive of performance
Learning approach
Neuroticism
Agreeableness
Inquisitive
Ambition sociability
Conscientiousness

Extra-curricular involvement

A retrospective review from NYU revealed that clinical performance correlates with applicants’ charity involvement.[8] The same review found that applicants’ participation in collegiate sports correlates to them attaining a chief resident title.

Traditionally, orthopedic surgery has suffered from the stereotype of being a specialty dominated by former athletes.[16] There are attributes of varsity involvement that translate well into orthopedic surgery such as visuospatial skills, teamwork, leadership, etc. However, research validating that previous athletic involvement predicts performance in surgical subspecialty training remains limited. Previous literature supports that athletes are prone to intrinsic factors making them more successful, with male collegiate athletes earning more and graduating at a higher rate than their age matched peers.[17]

Volunteer work has always been a staple of the medical school and residency application cycle. According to the 2016 NRMP match data, orthopedic surgery applicants, both matched and unmatched, had, on average, 6.7 volunteer experiences. [18] But it may not matter: Volunteering did not surface as a topic in 26 questions queried by Bernstein et al. in his survey of nationwide program directors about what factors into their resident selection.[2] Further, there is no difference in number of volunteer experiences between matched and unmatched applicants indicating that it is not a significant factor currently in selection.

The Grit Factor

Grit was first defined by Duckworth et al. in 2007 as perseverance and passion for long-term goals.[9] Since its introduction, grit has been correlated with enhanced achievement across a variety of disciplines such as increased graduation rates at West Point, lower divorce rates, improved retention of sales employees and teachers.[9,19,20] In a prospective study across a diverse collection of medical specialties, grit was associated with decreased burnout, decreased emotional exhaustion, and increased psychiatric well-being at six months among residents.[21] Interestingly, orthopedic surgery residents achieved the highest grit scores, amongst house staff surveyed, but this data did not trend to significance.

A prospective study among ENT residents and faculty (consultants) in the UK showed that consultants had significantly higher levels of grit than residents. Further, the study showed that grit had an inverse relationship with burnout.[22] While it might be presumed that those who obtain longterm goals − i.e. becoming an attending − are grittier, the application of grit for predicting burnout offers utility to programs looking to maximize resident selection. Using grit to predict burnout is of particular importance with a recent study citing orthopedic resident burnout rate of 56%.[23]

No published study to date has studied the relationship between grit and performance in orthopedic surgery residency. However, the existing literature presents a compelling argument for why further research is needed. Grit may offer a quantifiable and non-cognitive method for identifying successful orthopedic applicants. It may also serve as a screening tool to identify current residents who have early signs of burnout and need professional development.

Limitations and future directions

This review reveals a gap in quality studies examining non-cognitive domains in orthopedic residency. Further research is needed to prospectively identify the effect on resident performance of domains such as personality traits, grit, extra-curricular involvement, hobbies.

One major limitation to research on performance in orthopedic surgical residency is the lack of universal definition for success. While the literature continues to grow regarding success predictors in orthopedic surgery residency, one problem remains: How do we define success? Some authors consider ABOS part I pass rates; others look at faculty evaluations. However, to date no unified metric exists to quantify resident performance. Work by the ABOS/CORD to implement a mandatory skills assessment program is ongoing and offers a potential universal platform to enhance our understanding of resident performance characteristics. Perhaps with greater and more unified data on resident performance the literature will identify further non-cognitive predictors of orthopedic surgery residency performance.

Discussion and Conclusions

Becoming a successful orthopedic surgeon involves more than just passing boards. Being a successful surgeon requires interpersonal skills to communicate and to develop a productive physician-patient relationship. The admission process to orthopedic surgery residency should be multi-factorial, as success is clearly derived from multiple domains. Current methodology focusing on class rank and standardized tests scores does not adequately assess for the intangible qualities necessary to succeed as an orthopedic surgeon.

While limited, the literature offers unique non-cognitive domains that can be used to select the best and the brightest. Factors such as grit − a quantifiable attribute − can be integrated into the screening process. Personality tests can help select residents with traits that predict better performance. Finally, the admission process can move beyond Step 1 scores and subjective interviews and into a process driven by psychometrics − resulting in resident selections based on evidence demonstrating their likely ability to succeed.