Clinical Competency Committees in Plastic Surgery Residency.

BACKGROUND: Clinical competency committees (CCCs) are now an Accreditation Council on Graduate Medical Education (ACGME) requirement for plastic surgery training programs. They serve to monitor resident progress and make formal recommendations to program directors on promotion, remediation, and dismissal, based on resident progress toward the curricular milestones.
METHODS: Here, we present an overview on building, conducting, and improving a CCC, reviewing the literature available regarding best practices regarding this novel assessment system, with attention to the particular requirements for plastic surgery training.
RESULTS: We present the results of the Duke University CAQCC as a case study in the efficacy of a well-executed group in terms of improved resident outcomes, particularly regarding In-service Examination scores as an objective measure.
CONCLUSIONS: Rather than simply serving as a necessary ACGME dictum, the CCC has the opportunity to demonstrably improve resident education. This article is valuable for department leaders, program directors, faculty, and residents toward understanding the purpose and design of their CCC.

HISTORY AND PURPOSE

In 2009, the Accreditation Council on Graduate Medical Education (ACGME) began restructuring accreditation to be based on resident achievement in clinical competencies.[1] This Next Accreditation System (NAS) was first trialed by seven specialties (internal medicine, emergency medicine, urology, orthopedic surgery, diagnostic radiology, pediatrics, and neurosurgery) with subsequent adoption by all residency programs. The Next Accreditation System included direction of the creation of a clinical competency committee (CCC) to review resident performance at least semiannually. Next Accreditation System charged the field with the creation of “milestones” for which resident performance is to be conveyed to the ACGME[2] and the field has responded by validating assessment tools for these milestones.[3] Plastic surgery, as a “phase II” program, was first charged with creation of CCCs in 2014.[4] The ACGME now recognizes all CCC requirements as core requirements and they are thus mandated for every program.

REQUIREMENTS

As per the ACGME International Foundational Program Requirements for Graduate Medical Education, the CCC must:

be appointed by the program director,

be composed of program faculty,

have a written description of its responsibilities, including those to the institution and program director, and

actively participate in reviewing all resident evaluations and making recommendations to the program director regarding resident progress, to include promotion, remediation, and dismissal.[5]

MEMBERSHIP AND STRUCTURE

The ACGME Common Program Requirements detail that the CCC must include three members of the program faculty, at least one of whom is a core faculty member, with additional members being faculty from the same or other programs “or other health professionals who have extensive contact and experience with the program’s residents.”[6] Andolsek et al recommend no more than 8–10 members in total. It is essential to have a diverse set of perspectives because this diversity maximizes the CCC’s effectiveness.[7] There is an allowance for chief residents to serve on committees, but this is not appropriate for plastic surgery training because those in their final year of training (often referred to as a chief resident in surgical programs) are not permitted to be on their CCC. This is more appropriate for programs (eg, internal medicine) where selected graduated residents return for a year of chief resident designation. Whether members are appointed for a set term or until retirement is at the discretion of the program director.

CCCs have a variable number of members, most importantly affected by the number of residents in the program. Remember that every residency in the country is required to have a CCC. The ACGME guidance on the subject categorizes small programs as having less than 15 learners, medium programs with 15–75, and large programs with more than 75. Given the current distribution of plastic surgery training (six clinical years of integrated residency or three clinical years of independent residency, with each program enrolling one to four new integrated residents and/or one to three independent residents each year), every plastic surgery training program would fall into the small or medium category, meaning a single CCC is sufficient. Larger CCCs may require subcommittees, but these are not recommended for plastic surgery training.

Chair

CCC chairs maintain a special position within a plastic surgery residency. They must be content experts on the milestone curriculum as well as be adept at maintaining a committee and conducting meetings. French et al[8] lay out guidelines, which state that a chair’s responsibilities are to:

be the milestones expert,

encourage a positive working environment and open communication,

ensure members know their roles, the milestones, and the review process,

keep meetings on task, and

delegate appropriate committee documentation and meeting minutes.

Program Director

Out of any particular CCC topic, the role of program director (PD) carries the most controversy. The CCC must be appointed by and report to the PD, though the PD can still be part of the committee. The advantages to having PDs as members are that they may have specific knowledge on residents being evaluated and that they have a vested interest in the CCC since they are ultimately responsible for residents’ milestone acquisition. Additionally, small programs may find it difficult to fulfill the minimum three-person faculty membership requirement without the PD. Disadvantages to including PDs are that they may restrict openness of the committee and fairness in decision-making.[8] The PD is even permitted to be the CCC chair, though this is specifically forbidden in anesthesiology. It is our opinion that while the CCC should serve at the discretion of the PD, the PD should not serve on the actual committee. This maintains the independence of the CCC and increases both the efficacy and legitimacy of the organization. Because the PD is responsible for generating the curriculum and overseeing resident learning in each of the milestones, the CCC serves as an impartial “watchdog” of the program by avoiding conflict of interest.

Program Coordinator

Program coordinators, whether or not they attend CCC meetings, serve an important role. They collect, organize, and present assessment data to the CCC.[9] Coordinators also may take minutes of CCC meetings and communicate each resident’s milestone status to the ACGME, but may not be a part of the committee.

RESPONSIBILITIES

Preparing for, Attending, and Documenting Meetings

In the general sense, preparation for any particular meeting begins when the CCC is formed or when a new member joins. Members must have an agreed-upon understanding of the milestones and how their program teaches and assesses these milestones. Before a specific evaluation meeting, the CCC chair should ensure, with the assistance of the coordinator, the availability of assessment data. At the chair’s discretion, residents may be pre-assigned to a specific committee member, or the entire committee may review all residents as a group. However, members must not have a predetermined verdict regarding any particular resident.[9] Committee meetings must review residents in a consistent, fair, organized matter. All CCC members should attend these evaluation meetings and programs should provide protected academic time for these faculty members to attend meetings and execute their additional responsibilities. The chair decides on the format of the meeting but must be cognizant of potential influence (particularly if the chair enjoys an elevated position within the department’s hierarchy) and should deliberately seek a diversity of thought. Our institution seeks to avoid this hierarchical bias by including members from various subspecialties within the field. CCCs benefit from a structured discussion model where each member’s information is shared such that the group reaches an optimal decision.[10] At the conclusion of each meeting, the results of the discussion, including specific recommendations on each residents’ learning plans, must be carefully and thoroughly documented and conveyed to the program director for resident evaluation and improvement, as well as a written summary of the findings presented to the resident for their records. The goal is to provide an honest and transparent functioning of the committee.

Evaluation of Residents

The chief responsibility of the CCC is to independently evaluate residents and recommend the appropriate actions to the PD. The CCC should use all available instruments to form its professional opinion on each resident in training. The American Society of Plastic Surgeons In-service Examination, for example, is taken annually by all plastic surgery residents across the nation. Each year, the ASPS prepares a “norm table” for each year of residency (both independent and integrated), where each resident’s raw score is converted to a percentile that is normalized against all plastic surgery residents at the same post-graduate year level. This is a valuable tool due to its standardization; every resident from every program is evaluated equally. These percentiles can be used to “spot check” an entire program as well as provide the CCC with the only equal tool at their disposal. The resident’s scores on various sections can also be applied to the content-specific milestones. The In-service Examination is not merely an arbitrary assessment. Indeed, scores on this examination predict whether a graduate will pass the American Board of Plastic Surgery written examination.[11] Low rates of Board passage can result in an entire program being placed on probation. A plastic surgery program is judged successful by the ACGME if either of the following is true:

The aggregate pass rate of graduates taking the American Board of Plastic Surgery board examination is greater than that of the bottom fifth percentile in that specialty; or

An estimated 80% of graduates pass the examination.[12]

Using the prior 70% ACGME cutoff (from the 2017 guidelines), Girotto et al calculated the optimal In-service percentile cutoffs at 22nd (integrated) and 36th (independent) to maximize the area under the curve of testing toward board passage.[11]

There are several other assessment tools to consider, and programs are free to innovate toward developing the proper tools. At the University of Pittsburgh, the plastic surgery residents individually rate themselves against the milestones, with the self-ratings often lower than the corresponding CCC scores.[13] Schumacher et al surveyed pediatric CCCs (the variation was wide) with programs utilizing a combination of rotation assessment forms, teaching evaluations by students, professionalism measures, anecdotal feedback from chiefs, compliance with procedure logs, conference attendance, composite milestone scores, multi-source feedback assessments, resident self-assessments, structured clinical observations, 360 assessment forms, evidence-based medicine evaluations, quality improvement evaluations, and nursing assessments.[14] McKinnon et al have reviewed the myriad of technical and nontechnical competency measurement tools specifically studied with plastic surgery residents.[15] At Johns Hopkins University, residents and faculty both rated residents on “operative entrustability” using a web-based tool completed after each surgery. Using these ratings, the CCC was able to significantly reduce the amount of time spent reviewing each resident.[16] None of these assessments are without bias. Indeed the “operative entrustability assessments” show a gender bias, with male residents overrating their performance (compared with the attending’s rating of the same resident), and female residents underrating their performance.[17] Resident evaluations must be conducted at least semi-annually. Based on the CCC’s findings, it recommends promotion, remediation, or dismissal.

Individualized Learning Plans

In addition to the semiannual evaluations described above, the other prescribed role of the CCC is to provide input to the PD to assist each resident in developing an individualized learning plan (ILP).[9] Contrary to popular belief, ACGME expects every resident to have an ILP, not just those who are not meeting expectations. This is a common shortfall. Hauer et al showed that most CCCs utilize a “problem identification” model in lieu of a “development” model, even though the latter would make milestone-based assessment more meaningful.[18] Residents, not CCCs or PDs, create ILPs. However, any remediation recommended by the CCC to the PD may be included in the ILP.

According to Li and Burke[19] and adapted in the CCC Guidebook,[9] ILPs involve the following:

Reflection on career goals and self-assessment of strengths/weaknesses,

Goal generation,

Development of strategies to achieve the goal,

Assessment of progress toward the goal, and

Goal revision or generation based on assessment.

The ILP is a learner-driven plan that should rely on CCC input.

THE DUKE EXPERIENCE: A CASE STUDY IN THE BENEFITS OF FORMING A CCC

At Duke University, our plastic surgery residency program has maintained a CCC since 2014, before the ACGME designated the CCC requirements as core requirements. As one of the earliest CCCs in plastic surgery, we present our history of innovation, operating philosophy, and demonstrable changes to the program.

Duke Philosophy

The Duke CCC seeks to be a transparent and impartial monitor of resident progress. Too often, such a committee’s work can be viewed as punitive. Yet we adopt a developmental model over one of problem-identification to maximize every learner’s progress, rather than simply solving problems. The entire conduct of the CCC is based on objectivity, with the goal of being supportive of residents’ learning and career goals. When residents matriculate into the program, they are provided documented ACGME milestones and performance expectations that they use for self-assessment. These are the same measures the CCC uses to evaluate residents using a shared decision model. By avoiding ambiguity, every member of the division is privy to the functioning of the committee and the standard for resident assessment. All actions taken by the CCC are written in nature—not as a formality but to ensure transparency for all stakeholders. Ultimately, the CCC must provide an impartial process for the resident while safeguarding the division and specialty as a whole. The CCC allows us to fulfill our promise to the public that we are training plastic surgeons who, upon graduation, will be able to operate with the requisite skill, conduct, and professionalism.

Membership and Structure

Our CCC has four members, all full-time core faculty, with one member serving as the chair. The faculty interacts with residents at various points throughout their training by virtue of their clinical specialties. Currently, the four faculty members operate in craniomaxillofacial trauma, pediatric/craniofacial surgery, hand surgery, and breast reconstruction. The CCC meets semi-annually for formal reviews, with “ad hoc” meetings on an as-needed basis. Each month, the CCC provides updates to the Division-wide all-faculty meeting. In addition to the formal recommendations to the PD on promotion, remediation, or dismissal, the CCC assists residents in the construction of ILPs.

In line with the historic data on board passage,[11] our CCC recommends a textbook reading plan from any of the various comprehensive plastic surgery textbooks to any resident who scores below the 30th percentile on the In-service Examination. This specific ILP is not meant to be punitive in nature, but more so an acknowledgment that a structured reading program will assist the resident in the intellectual maturation associated with life-long learning and application of acquired skills as a practitioner of the art of surgery, as opposed to the regurgitation of facts for the satisfactory completion of a medical school examination. As a component of this reading program, the resident completes a written summary of each assignment, similar to a “high-yield facts” document, which is available for use in preparing not only for the annual In-service assessments, but also for the future written and oral board examinations. The format for the creation of the document is left to the individual resident, incorporating educational platforms that best suit their specific style of learning. Our CCC has seen various forms of review materials, from simple flash-cards to complex computer-based programs, all of which are satisfactory if they meet the resident’s educational needs. The outline and timing of the completion of this project is reviewed and agreed to by the resident in writing before the start of the program. At the completion of each section, the resident meets with a faculty representative from the CCC to evaluate their completed study guide and undergo an oral review of the topic, simulating the format of the oral board examination. The purpose of this exercise is to verify intellectual integration of the subject matter as it relates to patient care, as well as “desensitizing” the resident to the stress of a future oral board examination. We view this process to be an integral component of our commitment to provide every educational opportunity, and more importantly faculty support, for our residents to successfully complete their training program. While no doubt challenging to accomplish in addition to their other clinical obligations, the residents have uniformly completed this responsibility, leading to a more robust application of knowledge into their clinical training. Additionally, the CCC has recommended other measures such as stress management, organizational skills programs, leadership seminars, and life coach counseling when appropriate.

Effect on Resident Outcomes

As a testament to the efficacy of a well-run CCC, Duke’s resident performance improved significantly in the years after it was enacted. Here, we present the truly objective data that can be compared across all programs: ASPS In-service Examination mean percentiles. The authors received IRB review and exemption from the Duke University Health System Institutional Review Board (Protocol ID: Pro00106378) to present these data points.

As shown in Figure 1, the mean In-service Examination percentiles for plastic surgery residents at Duke University Hospital observed a steady increase since the implementation of the CCC, which was implemented between the 2014 and 2015 examinations. Linear regression was used (GraphPad Prism, GraphPad Software, San Diego, Calif.) to calculate the slope (5.060), r2 (0.7866), and F-test nonzero slope P value (0.0078). Although multiple factors clearly contribute to the overall residency performance on this examination, we offer that the CCC had a significant influence on this trend.

Fig. 1.

In-service examination percentiles for Duke University’s Plastic Surgery Residency. The past 7 years of data were available, reported as mean percentiles. Between the 2014 and 2015 examinations, the CCC was implemented in its current format.

CONCLUSIONS

CCCs can be an extremely powerful tool to improve resident education, if best practices are observed. Plastic surgery training carries its unique set of challenges with competency-based assessment, which are addressed above. Readers are invited to review the formal ACGME guidelines.[9] The Duke model has been quite effective and we offer our program as a model which other institutions can use as a framework for the design of a CCC that meets the needs of their program and institution.