The impact of COVID-19 pandemic on student performance and self-evaluation in preclinical operative dentistry.

INTRODUCTION: During the COVID-19 pandemic, dental schools were required to reformat their curricula to accommodate regulations mandated to protect the health of students and faculty. For students enrolled in the Operative Dentistry preclinical courses at the Harvard School of Dental Medicine (HSDM), this modified curriculum included frontloading the course with lectures delivered remotely, followed by in-person laboratory exercises of learned concepts. The aim of this article was to determine the impact that the modifications had on student performance and student self-evaluation capabilities.
MATERIALS AND METHODS: Thirty-eight students were introduced to this restructured course. Their performance in a final multiple-choice (MC) examination, four preclinical laboratory competency assessments (class II amalgam preparation and restoration, class III composite preparation and restoration) and their self-assessment of these preclinical competency assessments were then compared with the pre-COVID pandemic (P-CP) classes from years 2014 to 2019 (n = 216 students). Linear regressions were performed to determine differences in mean faculty scores, self-assessment scores, student-faculty score gaps (S-F gaps) and absolute S-F gaps seen between the class impacted by the pandemic and the P-CP classes.
RESULTS: The results demonstrated that students during the COVID-19 pandemic (D-CP) had a higher average faculty score in all four preclinical laboratory competency assessments and in the final MC examination. In addition, the S-F gap was smaller in this cohort as compared with the P-CP classes.
CONCLUSION: Despite the challenges of restructuring the preclinical curricula, D-CP students performed better than their P-CP predecessors in multiple facets of this Operative Dentistry course including self-assessment accuracy.

INTRODUCTION

As a result of the recent COVID‐19 pandemic, dental schools across the country have had to rapidly restructure and reformat their curriculum to ensure student safety and to meet state and national regulations on social distancing. , , ,  During the initial outbreak, universities transitioned all learning to remote avenues to provide continuity in education. Upon reopening of the school facilities, many universities followed the didactic lectures with preclinical exercises that divided the students into smaller subgroups to maximise social distancing. For the Operative Dentistry preclinical course taken by third‐year dental students at the Harvard School of Dental Medicine (HSDM), all didactic lectures were given remotely during the school's closure of physical facilities. Once protocols were established to reopen the university safely, this Operative Dentistry course began focusing on preclinical laboratory exercises.

Although these changes were necessary to ensure the safety of students and faculty, it also resulted in added challenges in dental education. Students reported during this time that they experienced an increased feeling of burnout, difficulties in understanding and retaining material, and reduced engagement and participation, which negatively affected their preclinical learning. , , Studies suggested providing more interactive virtual classes, utilising ungraded quizzes during lecture to increase engagement and combining different virtual tools and class formats to help combat some of these challenges. Despite recent research on the impact that the COVID‐19 pandemic has had on dental education, the effects of all of these changes on student performance and self‐evaluation accuracy have yet to be fully analysed.

In addition to preclinical and clinical skills, student self‐assessment abilities are paramount in fostering healthcare professionals who are lifelong, self‐directed learners. ,  Competencies in self‐assessment skills are prioritised and integrated in many academic programmes, and in dental education, these requirements are highlighted within the accreditation standards written by the Commission on Dental Accreditation (CODA).

Previous studies have found that students are often not effective in self‐assessment skills upon matriculation in professional programmes. The lowest‐performing students have a tendency to overestimate their performance, whilst the highest‐performing students, despite more accurate, often underestimate their performance. ,  Studies focusing on self‐assessment in dental education in particular have also disclosed similar results. These studies have determined a significant correlation between dental students' preclinical performance and their self‐assessment skills. , ,

Taking into consideration the need for social distancing, remote learning and other protocols implemented to keep students safe during the pandemic, this article focuses on how these changes have affected student performance and student self‐evaluation capabilities in an operative dentistry course. Due to the relative objectivity found in operative dentistry along with the consistency in faculty and in the course director for the course, student performance and faculty evaluations in the Operative Dentistry preclinical course at HSDM could be calibrated and compared between students prior to the pandemic and the students introduced to this new, modified curriculum.

The aim of this study was to assess the differences in student performance and student self‐evaluation abilities as a result of the new COVID‐19 pandemic protocols. Outcome data were executed by comparing the student didactic and preclinical performance and self‐evaluation skills for the P‐CP classes (2014–2019) to that of the D‐CP students (2020) who were introduced to the modified curriculum.

METHODS

Modifications of the curriculum

Thirty‐eight third‐year predoctoral students participated in the preclinical Operative Dentistry course at HSDM during the COVID‐19 pandemic. To optimise learning whilst the school's physical space was temporarily closed, didactic lectures were provided remotely through distance learning on Zoom (Zoom Video Communications). Once this lecture series was completed and limited laboratory access was granted by the University, students were able to proceed with the in‐person, hands‐on portion of the course. The students were divided into smaller groups to maintain social distancing and to abide by new limitation on occupant capacity. In addition, access to the preclinical laboratory outside of their mandatory sessions was limited.

Students were given opportunities to self‐evaluate their own work throughout the course and to familiarise themselves with the self‐assessment forms prior to the competency assessments at the end of the course. These assessment forms outlined the grading criteria that would be used by faculty to assess the students' work.

Upon completion of both the didactic and laboratory exercises, students were tested through a MC examination and preclinical competency assessments consisting of a class II amalgam preparation, a class II amalgam restoration, a class III composite preparation and a class III composite restoration. Assessment forms, identical to the ones used in practice, were provided to grade students on a scale from 1 to 4. These forms included detailed descriptions of the preparation and restoration parameters for each category. Students were graded on ten sections with a total possible score of 40 points for the amalgam and composite preparations, and on four sections totalling 16 points for the amalgam and 24 points for the composite restorations.

Two faculty members were chosen as graders for this study based off of their active involvement in preclinical operative courses over the past 7 years and their strong level of calibration that has been statistically confirmed in previous studies. , ,  These graders participated as faculty in the preclinical laboratory providing feedback throughout the course's duration and remained constant throughout the course of this study from 2014 to 2020. Faculty calibration exercises were also completed throughout the course and prior to administering the preclinical laboratory competency assessments each year. All assessments were graded blindly by removing all student identifiers from any work that was submitted. Two faculty members graded the preparations and restorations independently.

A final MC examination (120 questions in 3 h) in 2020 was administered remotely through the LockDown Browsers and a camera‐monitoring system (Respondus) to maintain social distancing and academic integrity. The conventional (hard copy) method with Scantron (Eagan) answer sheets was used for the P‐CP classes. Scores on the final MC examination and the preclinical laboratory competency assessments were compared with the student performance prior to the pandemic by using the data from this course from 2014 to 2019. Other than the separation of didactic lectures and preclinical laboratory exercises, the contents of the preclinical laboratory exercises were identical to those of all previous years prior to the pandemic. Lastly, the course directors evaluated the difficulty of the final MC examination each year and maintained it at similar levels to the best of their abilities. This study was completed as outlined and approved by the Harvard University's Institutional Review Board (IRB20‐1673, IRB20‐1131).

Performance outcomes

In addition to the final MC examination scores, the average faculty scores and self‐assessment scores were converted to percentages for each of the four preclinical laboratory competency assessments: class II amalgam preparation and restoration and class III composite preparation and restoration. The student‐faculty (S‐F) gap was then calculated by taking the difference between the self‐assessment and average faculty scores. Because a positive gap score (student over‐estimating their performance) could be cancelled by a negative gap score (student under‐estimating their performance), the absolute difference in the S‐F gap was also calculated to measure the accuracy of self‐assessment to actual performance.

Statistical analysis

In order to understand the students' population characteristics, descriptive analysis was performed. Gender distribution and mean scores with corresponding standard deviations (SD) were calculated for the admission scores, which included grade point averages (GPA), science GPA, scores on the Dental Admissions Test (DAT) and scores on the Perceptual Ability Test (PAT) portion of the DAT. The current study included reports of overall characteristics, as well as data stratified by P‐CP and D‐CP classes. Inter‐rater reliability (IRR) of the faculty members grading the preclinical competency assessments was also evaluated to provide absolute agreement and consistency of the preclinical exercises across students using the average‐measures intraclass correlations (ICC).

Linear regressions were performed to estimate differences in the mean faculty scores, self‐assessment scores, the S‐F gap and the absolute S‐F gap between the D‐CP class and the P‐CP classes for six outcomes: class II amalgam preparation and restoration, class III composite preparation and restoration, the average of all preclinical competency assessments combined and the final MC examination. Students' gender and admission scores were controlled in the multiple linear model to report the adjusted difference in the outcomes. Alpha level was set at .05 to reject the null hypothesis, and all statistical analyses were performed using Stata/MP Version 16.1 (StataCorp).

RESULTS

A total of 254 students were included in the analysis, representing seven classes with an average class size of 36.3 students (SD ± 1.3 students). The D‐CP class size consisted of 38 students, or 15 per cent of our total sample size. Female students represented 52.2 per cent of the study population within the study period, whilst males represented 47.8 per cent. The average student GPA was 3.87 ± 0.13, 3.85 ± 0.16 for the science GPA, 23.36 ± 1.68 for the DAT score and 21.81 ± 2.11 for the PAT score. Overall, P‐CP students were comparable to D‐CP students in terms of academic performance prior to matriculation into dental school (Table 1). IRR of the faculty, with absolute agreement and consistency for each exercise ranged between 0.73 and 0.80, indicating good agreement (Table 2). The highest coefficients were for the class II amalgam preparations (absolute agreement = 0.7 and consistency = 0.80), whilst the lowest coefficients were seen for the class II amalgam restorations (absolute agreement = 0.74 and consistency = 0.73).

TABLE 1

Characteristics of students at HSDM in the P‐CP classes (2014–2019) and the D‐CP class (2020)

Characteristics	Overall frequency (n)/Mean	Per cent/SD a	P‐CP b		D‐CP c
			Frequency (n)/Mean	Per cent/SD	Frequency (n)/Mean	Per cent/SD
Overall	254	100.0%	216	85.0%	38	15.0%
Gender
Female	128	52.2%	110	52.9%	18	48.7%
Male	117	47.8%	98	47.1%	19	51.4%
GPA d	3.87	0.13	3.86	0.14	3.91	0.07
Science GPA	3.85	0.16	3.84	0.17	3.89	0.09
DAT e score	23.36	1.68	23.29	1.72	23.76	1.40
PAT f score	21.81	2.11	21.86	2.17	21.57	1.76

Standard deviation.

Pre‐COVID pandemic.

During COVID pandemic.

Grade point average.

Dental admission test.

Perceptual ability test.

TABLE 2

Measures of inter‐rater reliability with absolute agreement and consistency for each exercise

Exercise	Absolute agreement a	Consistency b
Class II amalgam preparation	0.79	0.80
Class II amalgam restoration	0.73	0.74
Class III composite preparation	0.77	0.78
Class III composite restoration	0.73	0.76
All combined	0.81	0.84

The matching of scores between evaluators.

The amount of core difference between evaluators.

In general, the D‐CP students had a higher average faculty score in all preclinical laboratory competency assessments (Table 3). After adjusting for gender and admission scores, the D‐CP students had a statistically significant higher average faculty score compared with the P‐CP students for class II amalgam preparations by 5.92 percentage points (95% CI = 1.91, 9.92), 4.02 percentage points for class II amalgam restorations (95% CI = 0.34, 7.71) and 3.62 percentage points for all exercises combined (95% CI = 1.12, 6.12).

TABLE 3

Linear regression analysis of the preclinical laboratory competency assessments and final MC examination, comparing the score of P‐CP classes (2016–2020) to the D‐CP class (2021)

Exercise	P‐CP a	D‐CP b	Difference (D‐CP‐P‐CP)
	Mean c  ± SD d	Mean ± SD	Crude (95% CI e ) (N = 246)	Adjusted f (95% CI) (N = 239)
Class II amalgam preparation
Mean faculty score	72.8 ± 11.5	78.8 ± 9.6	5.9 (2.0, 9.8)*	5.9 (1.9, 9.9)*
Mean self‐assessment	84.4 ± 9.5	86.2 ± 8.7	1.8 (−1.5, 5.1)	1.7 (−1.7, 0.2)
Mean S‐F g Gap	11.6	7.5	−4.1 (−7.78, −0.4)*	−4.2 (−8.0, −0.4)*
Mean Absolute S‐F Gap	12.7	8.9	−3.8 (−7.0, −0.6)*	−3.9 (−7.2, −0.5)*
Class II amalgam restoration
Mean faculty score	82.2 ± 10.7	85.9 ± 6.7	3.8 (0.2, 7.3)*	4.0 (0.3, 7.7)*
Mean self‐assessment	87.6 ± 9.8	89.7 ± 7.4	2.2 (−1.1, 5.5)	1.6 (−1.8, 5.0)
Mean S‐F Gap	5.4	3.8	−1.6 (−5.9, 2.7)	−2.4 (−6.9, 2.1)
Mean Absolute S‐F Gap	10.7	8.1	−2.7 (−5.6, 0.3)	−3.0 (−3.0, 0.1)
Class III composite preparation
Mean faculty score	82.4 ± 9.5	84.6 ± 9.0	2.2 (−1.1, 5.5)	2.8 (−0.6, 6.2)
Mean self‐assessment	88.6 ± 8.3	91.2 ± 8.0	2.61 (−0.3, 5.5)	2.5 (−0.6, 5.5)
Mean S‐F Gap	6.1	6.5	0.38 (−2.7, 3.5)	−0.4 (−3.6, 2.9)
Mean Absolute S‐F Gap	8.7	7.7	−1.03 (−3.4, 1.3)	−1.6 (−4.1, 0.8)
Class III composite restoration
Mean faculty score	80.4 ± 10.4	81.4 ± 8.6	1.0 (−2.5, 4.5)	1.4 (−2.2, 5.0)
Mean self‐assessment	89.5 ± 7.9	90.5 ± 8.2	1.0 (−1.8, 3.7)	1.5 (−1.4, 4.3)
Mean S‐F Gap	9.1	9.1	0.0 (−3.7, 3.7)	0.1 (−3.7, 3.9)
Mean Absolute S‐F Gap	11.2	11.7	0.5 (−2.4, 3.4)	0.6 (−2.4, 3.5)
All exercises combined
Mean faculty score	79.4 ± 7.3	82.7 ± 5.8	3.3 (0.8, 5.7)*	3.6 (1.1, 6.1)*
Mean self‐assessment	87.5 ± 6.4	89.4 ± 6.4	1.9 (−0.3, 4.2)	1.9 (−0.4, 4.2)
Mean S‐F Gap	8.1	6.7	−1.4 (−4.0, 1.3)	−1.8 (−4.5, 1.0)
Mean Absolute S‐F Gap	9.1	7.9	−1.2 (−3.4, 1.0)	1.4 (−3.7, 0.9)
Final MC h examination	73.6 ± 7.2	78.3 ± 4.9	4.7 (2.4, 7.1)*	4.2 (1.8, 6.5)*

*p‐Value <.05.

Pre‐COVID pandemic.

During COVID pandemic.

Mean score out of 100%.

Standard deviation.

95% confidence interval.

Adjusted for gender, overall grade point average (GPA), GPA in science, Dental Admission Test score, Perceptual Ability Test score, interview score.

Student‐faculty score.

Multiple choice.

Crude (95% CI )

Adjusted (95% CI)

Self‐assessment scores were slightly higher in the D‐CP class for all laboratory exercises, but they were not statistically different when compared to the P‐CP class. Although the average self‐assessment scores were higher than the average faculty scores in both groups, the S‐F gap was smaller between the D‐CP cohort compared with the P‐CP classes. The difference in S‐F gap was smaller by 4.23 percentage points amongst D‐CP students compared with the P‐CP period (95% CI = −8.03, −0.42), and the mean absolute S‐F gap was smaller by 3.87 percentage points (95% CI = −7.20, −0.54) after adjusting for confounders. In addition, the D‐CP class scored a higher average in the final MC examination by 4.17 percentage points compared with the P‐CP students (95% CI = 1.80, 6.54).

DISCUSSION

The results of the current study show that despite a non‐traditional course curriculum that included a separation of didactic learning and laboratory exercises, remote online lectures, socially distanced laboratory exercises and limited access to the preclinical laboratory, students demonstrated higher faculty scores in all preclinical exercises and in the final MC examination. This held true even after adjusting for gender and methods of academic evaluation at the time of admission to dental school. Although previous studies have demonstrated a potential difference between these factors and student performance and self‐evaluation skills, , the average crude estimates did not differ significantly in this study from the adjusted values, which took into account these student characteristics. Although not all challenges as a result of the COVID‐19 pandemic were met with ideal solutions, this study provides evidence that this reformatted course may have offered students better structure, support and resources that fit their learning needs.

Unlike previous years in which the Operative Dentistry course ran for 9 weeks during which both lectures and laboratory exercises were offered concurrently, the D‐CP cohort were provided with 6 weeks of remote, online lectures followed by 8 weeks of laboratory sessions where the class was divided in half and assigned to either morning or afternoon laboratory sessions. One possible explanation for improved student performance in the D‐CP class is that the extended timeline of the course due to various interruptions caused by the pandemic allowed for quality reflection of concepts taught within this module. Metacognition, or the concept of “thinking about thinking,” requires both the knowledge and regulation of cognition. , Although the concept of metacognition is traditionally implemented within dental school curricula through student self‐evaluation, this restructured course may have given students important opportunities to plan, monitor and evaluate their learning as they adapted their mental models.

Another possible reason for the higher overall scores in D‐CP cohort is the introduction of new online materials. The foundation of the online material given to students was provided by each faculty who lectured throughout the course by way of original PowerPoint files. Faculty utilised the extra time they had during the initial physical closure of the university to improve and update their materials. The content catered to students whose application of lectured materials was limited due to the initial lack of access given to the preclinical laboratory. Not only were these materials posted to Canvas Learning Management System (Canvas LMS) prior to schedule lectures, but also all lectures were also recorded with links accessible through Canvas LMS. Thus, students were able to revisit concepts taught in lecture on their own time and at their own pace. Faculty members were readily available through virtual platforms such as email and Zoom throughout the entirety of the course to help answer any questions that arose during the students' learning processes. In addition, video contents and 3‐D images for visualisation were utilised as supplemental material. The separation of lecture and laboratory exercises resulted in students' prioritising quality online materials, like recorded didactic lectures, that allowed them to revisit topics and review relevant information prior to each laboratory session. These inadvertent spacing and interleaving tactics could also explain this year's improved student performance. Although students may perceive this approach to learning as counterintuitive, these desirable difficulties could have enhanced learning in the long term.

The logistics behind preclinical laboratory sessions may have also contributed towards better student performance in the D‐CP class. Although students were given limited time to practice outside of mandatory class sessions, they were also given higher student‐to‐faculty ratios in class sessions due to social distancing limitations. This increased faculty availability could have improved the quality of instruction and feedback given to help narrow the S‐F gap seen this year. It was the faculty's perception that students may have also been more engaged and efficient during these sessions knowing their access to the laboratory outside of mandatory class time was limited.

Additionally, given the sequence in which these preparations and restorations were taught, students were provided with more time to practice amalgam preparations and restorations, having learned these procedures first. This could have provided students with more practice in performance and self‐evaluation resulting in statistically higher faculty scores in amalgam class II preparations and restorations.

One main criticism of the methodologies of this study was that it was retrospective in nature. Although the faculty who assessed the preclinical competency assessments for the past 7 years and the nature of these assessments themselves were held constant, the final MC examination questions inherently varied slightly each year. However, the course directors made a conscious effort to create questions that maintained an overall similar level of difficulty of the examination.

This study also used faculty evaluations as a standard comparison for quantifying student performance. Given that these evaluations are subject to variability, they cannot represent the absolute assessment of student preparations and restorations. Therefore, for these reasons, faculty who participated in the evaluation of students in the current study were held constant year after year, and faculty were calibrated each year to reduce such variability. An inter‐rater reliability test was also performed to confirm precision between faculty and to strengthen the analysis of this study.

Lastly, another main limitation of this study is both the sample size of the D‐CP group and this size in relation to the P‐CP sample size. Due to the large variability in students' scores and the small population of students at HSDM, the 95% CIs of the estimates were relatively wide. As a result, this study could not detect a statistical difference in most of the preclinical laboratory exercises by failing to reject their respective null hypotheses.

CONCLUSION

Despite socially distanced preclinical laboratory exercises and concerns about student engagement and burnout with online lectures, students in the pandemic group performed better overall compared with their P‐CP predecessors when it came to preclinical laboratory assessments and final MC examination in the Operative Dentistry course. However, there were no significant differences in students' self‐evaluation skills between P‐CP and D‐CP except in the Class II amalgam preparation exercise.

Dental schools across the country and internationally are still faced with the challenges of meeting the needs of their students, whilst following government regulations set in place to keep the community safe. Further research is needed to improve upon distance learning, preclinical learning and clinical experiences during the pandemic and beyond. A future area of research specific to the findings of this article includes applying this modified curriculum in teaching other disciplines within dentistry to determine its applicability and translatability, especially during non‐COVID‐19 times. Additionally, further studies will be needed to evaluate more specifically which of the possible contributing factors outlined above are associated with the betterment of student performance.