Comparison of clinical competency self-assessments among Advanced Education in General Dentistry (AEGD) residents before and during COVID-19 pandemic.

PURPOSE/
OBJECTIVES: Self-assessment of clinical competence is an important tool for effective learning and training for some educational programs. The New York University (NYU) Langone Hospital's Advanced Education in General Dentistry (AEGD) Program has had its residents complete self-assessment of clinical competency evaluations for many years. The evaluation is used to understand the residents' perception of their own clinical skill upon beginning the program and to determine the necessary resources to provide to the residents for them to meet program standards. The same evaluation is completed by the residents 6 months later to determine if they perceived advancement in their clinical performance while in the program. Dental education, along with other fields of education was disrupted by the coronavirus disease 2019 (COVID-19) pandemic. The purpose of this study was to examine the impact of COVID-19 on clinical competency self-assessments among the NYU Langone AEGD residents before and during the pandemic.
METHODS: In this cross-sectional study, data was collected from two AEGD cohorts representing 2019-2020 and 2020-2021 academic years; from July 2019 (n = 196) to January 2020 (n = 189) and July 2020 (n = 202) to January 2021 (n = 184). The self-assessment evaluations were administered via an online residency management platform on the first days of July and January of the academic year. The survey consisted of 48 questions on "clinical skills and performance" as established by CODA standards for postdoctoral general dentistry programs.
RESULTS: Survey response rate was 100% for both cohorts. When comparing results, the findings indicate the COVID-19 pandemic had interrupted clinical learning during dental school. However, training through the AEGD program led to improvements in perceived clinical competence by the residents in mid-program evaluation.
CONCLUSION: The self-assessment evaluation can be used as a tool to enhance training as part of the AEGD program's performance improvement plan.

INTRODUCTION

The practice of dentistry is mostly an independent endeavor and is considered a self‐analytic profession. , , Therefore, the ability to self‐critique accurately and in an accountable manner is of the utmost importance and a priority for learning and training in dental education. , Self‐assessments are not yet universally utilized in dental education and quality measurements are limited to assess interpretation, validity, and application of these evaluations, but are still of benefit to the program as a tool used to assess some level of the quality of education. Self‐assessments are a method deployed in pharmacy, nursing, and medicine. , , Educators in the pharmacy field have utilized self‐assessment methods extensively by students and faculty and emphasize that self‐assessment can enhance learning skills and develop competencies. , , Self‐assessments are leveraged in nursing as well. Hadid states self‐assessment facilitates students’ monitoring and evaluation of the quality of their thinking and behavior and allows them to identify strategies that improve understanding and skills. In work more specific to dental education, Emam et al. assert that self‐assessment is a “critical skill” that dentists must possess for developing competencies as oral healthcare providers. Habib and Sherfudhin found that dental students’ ability to self‐evaluate their own work may be an effective learning tool that enhances their performance.

In medical education, knowledge is not the final step but is considered as a basis to developing higher levels of skills including professionalism, cultural competence, clinical reasoning, and so forth categorized under the title of competencies. In 1993, Chambers introduced competency‐based education into dentistry. He defined competencies in dentistry as “combining appropriate supporting knowledge and professional attitudes and they are performed reliably in natural settings without assistance. ” Competency assessments in dental education according to the Commission on Dental Accreditation (CODA) are used to evaluate the stage of knowledge and skills of students and identify their strengths and weaknesses in order to advance further in the dental school. The New York University (NYU) Langone Hospitals Advanced Education in General Dentistry (AEGD) Program has had its residents’ complete self‐assessment of clinical competency evaluations for many years. The evaluation is used to understand the residents’ perception of their own clinical skill upon beginning the program and to determine the necessary resources to provide to the residents for them to meet program standards. The same evaluation is completed by the residents 6 months later to determine if they perceived advancement in their clinical performance while in the program.

Education worldwide, in every aspect, experienced disruption by the COVID‐19 pandemic during the academic period of 2020–2021. Dental education was no exception to this unfortunate and unforeseen crisis specifically, early in the pandemic because of shortage of personal protective equipment (PPE) and raised concerns on spreading the Severe acute respiratory syndrome coronavirus 2 virus through aerosol producing dental treatments. The pandemic interrupted the clinical education for students who were still working to complete their clinical requirements for graduation or taking the regional clinical board exams. With little notice, dental educators had to implement and execute online education programs to fill the gap in education during a period when in‐person learning was an impossibility or extremely limited. While their herculean efforts are applauded and appreciated, some aspects of hands‐on learning were lacking due to the nature of this type of pedagogy in dental education. Hattar et al. report that during this pandemic pre‐doctorate students in dentistry, while appreciating the online learning opportunity, did not consider it a “substitute for face‐to‐face clinical practice. ” The same cohort reported reservations regarding independent practice following graduation. The COVID‐19 pandemic crisis not only affected the quality and delivery of dental education but heightened stress levels for students and trainees as reported by Hung et al. They found the pandemic “significantly impacted dental education” and stated that students experienced increased levels of stress and felt their clinical education had suffered.

Within NYU Langone Hospitals Postdoctoral Residency Programs, a variety of formal evaluations are used to assess a resident's progress throughout the postdoctoral training year. In the AEGD program, a self‐assessment of clinical competencies has been designed to assess residents at the beginning of the program and in the seventh month of the training program. These evaluations allow the residents, who are predominantly recent dental school graduates, to indicate their perceived level of theoretical knowledge and applied clinical aptitude, strengths, and challenges, and allow the program to determine the resources to direct towards the residents to provide them the opportunity to succeed during their year of training. During the program orientation session, it is explained to the residents the purpose of the evaluation, instructions to complete the evaluation are provided, and they are informed as to how the results will be utilized by the program as a quality improvement tool for educational training. Residents also have access to the CODA standards for postdoctoral general dentistry that outlines clinical curriculum expectations. Results of the evaluations are shared with clinical training faculty to provide the residents with the necessary supervision, support, training, and experience to successfully progress through the program and continue to enhance their clinical skills. The objective of this project was to analyze the observed trends and changes in clinical competency self‐assessments among two AEGD cohort residents, before and during the pandemic.

Two objectives directed this quality improvement study. Objective one was to compare the first self‐assessments at the beginning of the program for the resident class of 2020–2021 with the previous cohort 2019–2020 to identify disruptions to the predoctoral education programs during the COVID‐19 pandemic. The second objective was to identify any differences in the second evaluation taken by the residents having completed 7 months of training in the program between 2020–2021 and 2019–2020 cohorts. The second evaluation demonstrates residents’ self‐perception of performance improvement, experience, and competency compared to the results generated upon the beginning of the program.

METHODS

Setting and participants

This was a cross‐sectional study that analyzed data collected from the residents at the NYU Langone AEGD program, from July 2019 to January 2020 and from July 2020 to January 2021. This project did not require Institutional Review Board (IRB) review due to the Quality Improvement nature of the study based on responses to the self‐certification form questions for the NYU School of Medicine IRB.

The AEGD Program of NYU Langone Hospitals is based in Brooklyn, NY, and has affiliations with approximately 100 clinical training sites throughout the US and the Caribbean that provide the training for up to 240 residents annually. This survey was administered to, and completed by, all residents of the program. The program utilizes a variety of formal evaluations to assess a resident's progress throughout the postdoctoral training year. One of the evaluations utilized is the self‐assessment of clinical competency which accounts for one's actions and attitudes, in particular, of one's performance at a job or learning task considered in relation to an objective standard and is used in a number of employment settings.

This evaluation includes 48 competency statements based on the CODA for AEGD program and verbiage and definitions adapted from Chambers and Gerrow's manual. A graduate dental education assessment committee directed by the AEGD program director (first author: Anna D'Emilio) developed and validated the 48 questions on “clinical skills and performance”. Self‐assessment evaluation surveys are deployed to residents via an online residency management platform on July first (start of the program) and January first (mid‐point of the program) of the academic year. Completion of the evaluation by the residents across all clinical training sites is mandatory, and answers to all questions are required.

Residents rated themselves among the selected clinical competencies by explicitly defined criteria as seen in Table 1.

TABLE 1

Ratings for self‐assessments with defined criteria

Rating	Definition Criteria
Does not meet expectations	The clinical performance of the resident is at a level that is below postdoctoral standards and expectations
Meets expectations	The clinical performance of the resident is at a level that is acceptable, and appropriate to postdoctoral standards and expectations
Exceeds expectations	The clinical performance of the resident is at a level that surpasses postdoctoral standards and expectations
No Experience	The resident has not had the opportunity to perform the clinical activity

Data analysis

Raw exports of the results of the self‐assessment evaluations for July 2019, July 2020, January 2020, and January 2021 were generated as excel files. Non‐US trained residents were excluded from data analysis as these individuals may not be recent dental school graduates and be confounding variables. They accounted for 5% of the resident cohort in this study.

For each export, instances of “Does not meet expectations” were tallied for each question to provide the number of respondents/responses for that question. For each evaluation period, the number of “Does not meet expectations” responses to each question were divided by the total number of responses for the corresponding evaluation/export to obtain a percentage of “Does not meet expectations” for each question.

In order to find the answers to the objectives developed for this project, totals, and percentages of “Does not meet expectations” were compared between the July 2019 and July 2020 exports, and between the January 2020 and January 2021 exports. This simple percentage difference calculation indicated some clear trends. In addition, for each question, the odds ratio (OR) of having a “Does not meet expectations” response in the recent assessment versus the previous assessment was calculated and associated p‐values were computed using Fisher's Exact test. OR > 1 was interpreted as meaning more likely to observe “Does not meet expectations” in the recent assessment, for example, July 2020 versus July 2019.

RESULTS

The number of AEGD residents from both cohorts 2019–2020 and 2020–2021 taking part in the clinical competency self‐assessments are depicted in Table 2. The response rates to surveys at the beginning of the program and the seventh month of the program in both cohorts were 100% (Table 2).

TABLE 2

Number of US‐trained residents in each of the Advanced Education in General Dentistry (AEGD) cohorts participating in the study according to the time of the survey

Cohort 1 (2019/2020)
	July 2019	January 2020
Number of respondents	196	189
Total number of US‐trained residents in the program	196	189
Response rate	100%	100%

Note: Some residents leave the program early, which accounts for the difference in numbers during the same academic year.

The first self‐assessment evaluations revealed that residents in the July 2020 assessment rated themselves as “Does not meet expectations” in six areas out of the 48 competency questions (p < 0.05). Manage advanced periodontal disease (n = 40), perform uncomplicated non‐surgical anterior endodontic therapy (n = 22), manage patients requiring complicated fixed restorations including implants (n = 46), treat patients with missing teeth requiring removable appliances (n = 28), communicate case design with laboratory technicians and evaluate the resultant prostheses (n = 34) and treat patients with missing teeth requiring uncomplicated fixed restorations (n = 13).

The self‐assessments in January 2021 depicted that only performing uncomplicated, molar endodontics did not meet the expectation by 6.5% (n = 12) of the residents from 2020–2021 cohort.

The “OR” results were similar to the “simple percentage” difference calculation. When comparing July 2020 to July 2019 assessments, there were higher frequencies of “does not meet expectation” responses in 2020 (OR > 1) and some differences were statistically significant (p < 0.05). In the 2021 January versus 2020 January comparison, the differences were very small and did not reach statistical significance, except for question number 23 (Diagnose and manage occlusal disorders and disorders of the temporomandibular joint). There was a higher frequency of “Does not meet expectations” responses by residents in January 2021 for question number 23. For most questions in January assessments, the frequencies of “Does not meet expectations” responses in both January 2020 and January 2021 were zero and therefore ORs were not meaningful, and not plotted.

Further data analysis of incoming residents completing the first self‐assessment evaluations in July 2019 and July 2020 revealed that significantly more residents in 2020 rated themselves as “Does not meet expectations” in six areas out of the 48 clinical competency questions compared to residents assessed in July 2019 as shown in Table 2. After 7 months in the program, residents in both 2019–2020 and 2020–2021 cohorts did not choose “Does not meet expectations” responses. However, performing uncomplicated, non‐surgical posterior including molar endodontics was chosen by 6.5% of the residents from 2020–2021 cohort as not meeting the expectation in the January evaluation. The low percentage of “Does not meet expectations” responses in the January assessments of 2019–2020 and 2020‐2021 cohorts demonstrate improvement in AEGD residents’ self‐assessed competencies as shown in Table 2.

In addition to tabulating “Does not meet expectations” results on these evaluations, the program reviewed competency areas where residents commonly self‐assessed as “Meets expectations” upon beginning the program in the following competency areas: diagnosis and treatment planning, non‐complex restorations, non‐surgical periodontics, and non‐complex hard and soft tissue surgery (exodontia) and conscious sedation.

Furthermore, regarding the residents’ perception of advancement from the beginning of the program to the seventh month of the program, common reasons provided by the residents in comment form were related to a higher volume of patients to treat with a variety of non‐complex and complex treatment needs under direct and indirect supervision by general (and specialist) faculty that resulted in gaining additional instruction, experience, speed and confidence with performing procedures.

DISCUSSION

In this study, through two consequent self‐assessment clinical competency surveys completed six months apart, interesting information was identified about the impact of the NYU Langone Hospitals’ AEGD residency program on self‐assessed clinical competencies. The major highlight of this quality improvement study was that the surveys were conducted before and during the COVID‐19 pandemic.

Typically, results from the first evaluation at the start of the AEGD training program in July 2020 indicated that although residents believed they have received the required predoctoral education and training, they may have felt less accomplished or developed the necessary skillset in certain areas of clinical practice and rated themselves as “Does not meet expectations” (Table 2). Common disciplines in which residents rated themselves as such at the start of their residency period include endodontics, surgical periodontics, removable and fixed prosthodontics (Table 3). Typically, residents rated themselves as “Meets” or “Exceeds expectations” in the following disciplines: diagnostics, non‐complex restorations, non‐surgical periodontics, non‐complex hard and soft tissue surgery, and conscious sedation. By the second evaluation period in January 2021 during which time residents have typically been in the AEGD program for 6 months, there were significant changes in those areas previously rated as “Does not meet expectations” (Table 3). The clinical experiences received by the residents at their clinical training sites did not change as monitored by the program's outcomes reporting platform for clinical procedures performed.

TABLE 3

Percentage of clinical performances and odds ratio for self‐assessments on not meeting expectation among Advanced Education in General Dentistry (AEGD) residents in two cohorts upon admittance to the program and 6 months into the program

Clinical performance question/time of survey	Cohort 1 (2019–2020)	Cohort 2 (2020–2021)	Odds ratio	p‐value
Manage advanced periodontal disease
July 2020 vs. July 2019	8.16%	19.80%	2.77	0.001
Manage patients requiring complicated fixed restorations including implants
July 2020 vs. July 2019	11.73%	22.77%	2.21	0.005
Perform uncomplicated non‐surgical anterior endodontic therapy
July 2020 vs. July 2019	3.06%	10.89%	3.86	0.003
Communicate case design with laboratory technicians and evaluate the resultant prostheses
July 2020 vs. July 2019	9.69%	16.83%	1.88	0.039
Treat patients with missing teeth requiring removable appliances
July 2020 vs. July 2019	7.14%	13.86%	2.09	0.034
Treat patients with missing teeth requiring uncomplicated fixed restorations
July 2020 vs. July 2019	2.04%	6.44%	3.29	0.045
Perform uncomplicated non­‐surgical posterior, including molar, endodontic therapy
July 2020 vs. July 2019	17.86%	19.31%	3.29	0.045
Manage advanced periodontal disease
January 2021 vs. January 2020	1.06%	1.09%	1.03	>0.99
Manage patients requiring complicated fixed restorations including implants
January 2021 vs. January 2020	1.59%	0.00%	0.00	0.24
Perform uncomplicated non‐surgical anterior endodontic therapy
January 2021 vs. January 2020	0.00%	0.54%	–	0.49
Communicate case design with laboratory technicians and evaluate the resultant prostheses.
January 2021 vs. January 2020	0.00%	1.09%	–	0.24
Treat patients with missing teeth requiring removable appliances
January 2021 vs. January 2020	0.53%	0.54%	1.03	>0.99
Treat patients with missing teeth requiring uncomplicated fixed restorations
January 2021 vs. January 2020	0.53%	0%	0.00	>0.99
Perform uncomplicated non­‐surgical posterior, including molar, endodontic therapy
January 2021 vs. January 2020	1.06%	6.52%	6.5	0.005

These findings may speak both to the perceived possible challenges recent dental graduates may encounter when joining the workforce directly after graduation and to the perceived benefits of a residency program in preparing dental residents for the next level of their professional pursuit.

Results indicate that residents believe they have received the experience during the time training in the program and are performing satisfactorily and at a postdoctoral level.

As a result of the COVID‐19 pandemic, dental students experienced disruptions of traditional predoctoral training and education. , Dental schools were closed beginning March 2020 in compliance with local and state government quarantine mandates. Clinical education and direct, elective patient care activities were suspended. In the interim, students continued their studies utilizing distance education platforms. Eventually, students were able to return to clinical campus however, direct patient care activities were attenuated to continue to observe local and state government mandates regarding heightened infection control practice and social distancing to protect the patients and providers. Thus, in 2020, many pre‐doctoral dental students completed the remainder of their clinical training utilizing simulation and other non‐patient‐based exercises. The AEGD program at NYU Langone Hospitals surmised that this unanticipated disruption and attenuation of a dental student's fourth and critical year of clinical training and experience would affect the program's cohort of residents graduating dental school during the late spring/early summer of 2020 and entering the program in July 2020.

AEGD Programs are educational programs designed to provide training beyond the level of pre‐doctoral education in oral health care, using applied basic and behavioral sciences. Use of the results of the resident self‐assessment of clinical competency, as one of the many program outcomes resources, allows the program to evaluate its clinical curriculum to ensure it provides the residents with the required clinical education armamentarium to successfully progress through the program and in preparation for the resident's future professional endeavors.

LIMITATION

The generalizability of the results to other AEGD programs in the US is considered a limitation since this is a quality improvement project and has been implemented on only two cohorts from the AEGD program at NYU Langone Hospitals. However, the survey developed and used in the project was based on the CODA accreditation standards and the well‐known manual developed by Chambers and Gerrow on competency assessments. ,

Another limitation to consider is the number of residents that left the program between July 2020 and January 2021. This was partially attributable to the tumult experienced during COVID‐19 and may be considered another negative effect of the pandemic on dental education. Also, the authors acknowledge that since the program requires the residents to complete the survey, this may introduce bias into the study.

Nevertheless, the survey results may serve as a representation of the experience of AEGD residents and dental education programs before and during the COVID‐19 pandemic and provide insight for educational leaders and residents on adapting through challenging periods.

CONCLUSION

The results indicate that AEGD residents had disruptions in their final year of clinical training in dental school during the pandemic. After seven months in the AEGD postdoctoral training program, self‐assessment competency findings showed positive improvements in perceived clinical competency. The performance improvement plan in the AEGD program includes determining residents’ baseline clinical skills and the resources required to assist them earlier in the program year towards accelerating to the expected level of competency that would include, for example, bench testing exercises, selection of appropriate clinical cases and an additional one on one supervision and mentoring by faculty. In conclusion, the study provides evidence for educational leaders and dental health professionals on the benefits of using self‐assessment surveys for further improving education in the AEGD program and enhancing the training of residents as part of the AEGD program's performance improvement plan.