Prediction of practical performance in preclinical laboratory courses - the return of wire bending for admission of dental students in Hamburg.

Although some recent studies concluded that dexterity is not a reliable predictor of performance in preclinical laboratory courses in dentistry, they could not disprove earlier findings which confirmed the worth of manual dexterity tests in dental admission. We developed a wire bending test (HAM-Man) which was administered during dental freshmen's first week in 2008, 2009, and 2010. The purpose of our study was to evaluate if the HAM-Man is a useful selection criterion additional to the high school grade point average (GPA) in dental admission. Regression analysis revealed that GPA only accounted for a maximum of 9% of students' performance in preclinical laboratory courses, in six out of eight models the explained variance was below 2%. The HAM-Man incrementally explained up to 20.5% of preclinical practical performance over GPA. In line with findings from earlier studies the HAM-Man test of manual dexterity showed satisfactory incremental validity. While GPA has a focus on cognitive abilities, the HAM-Man reflects learning of unfamiliar psychomotor skills, spatial relationships, and dental techniques needed in preclinical laboratory courses. The wire bending test HAM-Man is a valuable additional selection instrument for applicants of dental schools.

Introduction

Over the past 20 years many studies concluded that manual dexterity is not a reliable predictor of performance in preclinical laboratory courses in dentistry [1], [2], [3], [4], [5], [6], [7]. Dexterity was commonly measured with different manual ability tests, e.g. the O‘-Connor-Tweezer-test [4], [7], a block-carving-test [1], a set of subtests from different established dexterity tests [2] or even partially with a computer game [3]. The use of these instruments is surprising with regard to the findings of Weinstein and Kiyak [8] who concluded that dexterity tests for the prediction of performance should represent activities comparable to the tasks of preclinical laboratory courses. Weinstein and Kiyak developed the Dental Dexterity Test (DDT) that contains a device that simulates the human mouth. Subjects insert and then extract 32 pins with a tweezer with a time limit.

Moreover, the cited studies did not take into account the results of Kao et al. [9], who reported moderate relationships between a self developed wire-bending test and manual performance in the first and second dental years (0.337psychomotor skills.” (p. 672). These psychomotor abilities and skills are also needed for dental techniques in preclinical laboratory courses.

Currently wire bending tests are used as a constituent part of student selection procedures, e.g. at the Witten/Herdecke University in Germany [10] and at the University of Innsbruck in Autria [11]. Arnold et al. [10] reported findings for the practical test in their selection procedure and the first dental examination (r=0.2, p<.05), as well Beier et al. [11] did, but for the average grades after the first clinical year as criterion (r=-.373, p<.01). Unfortunately, the used wire bending tests are just one task among others in both practical tests and no task-specific correlations are reported in both research studies.

Selection of dental students in Hamburg

Since 2004 German universities are allowed to select 60% of their freshmen in dentistry using admission criteria additional to high school grade point average (GPA) as university quota [12]. The remaining study places are assigned by the GPA (20%, top GPA quota) and waiting period quota (20%) [13]. In each quota, GPA has to be considered for the selection process. However, GPA is focused on cognitive abilities in contrast to psychomotor skills [9], [14] and its relationship to practical performance in preclinical laboratory courses is very limited [9], [15], [16]. Therefore, we were looking for additional criteria to improve the selection process with respect to manual dexterity needed in preclinical laboratory courses. With reference to the wire-bending tests of Lienert [17] and Kao et al. [9] we developed a wire-bending test, the “Hamburg Assessment Test for Medicine - Manual Dexterity” (HAM-Man) [18] for the measurement of manual abilities required in preclinical laboratory courses.

As a first step towards a more widespread admission procedure we implemented the natural science test HAM-Nat [12], [19], [20], [21] for the admission of dental students in the university quota in 2009 [22]. The HAM-Nat results (59 points maximum) and GPA (60 points maximum) were summarized in a ranking list. The aim of the HAM-Nat is to lower the drop-out rate in the preclinical science courses. In 2010 we extended our dental selection procedure by inserting the wire bending test HAM-Man.

The purpose of our study is to evaluate if the HAM-Man is a useful selection criterion in addition to GPA in dental admission. We expect the HAM-Man to possess satisfying incremental validity with regard to students’ practical performance in preclinical laboratory courses.

Materials and Methods

Sample

The HAM-Man was administered during the dental freshmen’s first week in October in three successive years (see table 1 (Tab. 1)). The participation was voluntary. Twenty-three out of 78 participants from all admission quotas participated in our study in 2008 (admission by GPA), 69 out of 80 in 2009 (admission by GPA and HAM-Nat) and 54 out of 78 in 2010. Applicants of the latter sample were first ranked by a combination of their HAM-Nat scores and GPA (admission list 1) and then ranked by a combination of their HAM-Man scores and GPA (admission list 2). Applicants’ HAM-Man scores (59 points maximum) were only considered for admission, if their HAM-Nat score was among the best 70 results in 2010. In total 20 freshmen were admitted by the second admission list. All participants gave written informed consent.

Table 1

Descriptive statistics

In 2008, participants with top ten HAM-Man results received a fifty Euro voucher for a local bookstore. The objective was to increase study motivation and to establish a competitive atmosphere comparable to the real selection conditions. As the first cohort’s students (2008) stated that the reward did not influence their practical performance, the students in the second cohort (2009) did not receive a reward. However, as analyses showed deviating average HAM-Man results between the two first cohorts (see table 1 (Tab. 1)), the third cohort (2010) received a fifty Euro voucher again.

Predictor variables

GPA

The German GPA represents an average of high school grades from 1.0 (very good) to 4.0 (sufficient). It combines grades from written and oral final exams with grades achieved in the senior years in secondary school.

HAM-Man

Aim of the HAM-Man is to measure manual abilities needed in dentistry. The 45 minute HAM-Man test comprises three wire shapes (see Figure 1 (Fig. 1)), which have to be bent with commercially available standard pliers. We refrained from using professional dental pliers, as these are mostly unknown to applicants and their use requires training. Participants read the instruction sheet, which contains original size drawings of the wire shapes. They are free in their decision how to approach the HAM-Man. A test set includes six strong chromium-wires of 0.8 mm diameter and 150 mm length, so half of the available wires can be used for trial bending attempts. Two raters assess the three bent wires on a seven point Likert scale from 0 (very poor) to 6 (very good) for the criteria “flatness of wire” and “accuracy of fitting” and from 0 (very poor) to 3 (very good) for the criterion “quality of bending”. The latter criterion evaluates if there are any buckles, wire deformations or scratches on the wire due to multiple bending attempts. This criterion is weighted with 0.5, because there is no professional equipment available. The mean of both raters is accumulated to the total test score.

Figure 1

HAM-Man shapes

Raters for the HAM-Man were recruited from our faculty staff, namely a dental laboratory technician and an experienced dentist. In a rater training session, every rater evaluated several wire samples of each shape (see Figure 1 (Fig. 1)). The raters discussed the results referring to the three rating criteria until they were satisfied with the agreement in their judgments. Intraclass-correlations (ICCs) were used to determine the interrater-reliabilities. In detail, we computed two-way mixed ICC-models (consistency) including single measures of the two trained raters, which revealed agreements greater than or equal to 94% in all three cohorts. The internal consistency (Cronbach’s alpha) was greater than 0.88 for each wire shape in every cohort.

Performance variables

The dependent variables were operationalized as average (practical) examination performance in preclinical laboratory courses, which represent the practical share of preclinical study success. The first laboratory course (TPK) took place in the students’ first semester. Performance consisted of four examinations (“carving wax teeth”, “waxing”, “production of a metal crown”, and “production of a transitional prosthesis”). The second laboratory course (PHA I, second semester) included three tasks on a phantom head (“total Gysi prosthesis”, “glued pin insertion & shell temporary”, “tangential bridge”). In the last preclinical course (PHA II, fifth semester) students had to manufacture a “total prosthesis (Gerber)”, a “Michigan splint” and a “bridge preparations & shell temporary”. All examinations were rated on a grade scale from 1.0 (very good) to 6.0 (insufficient). It is required to pass TPK before entering PHA I, which again has to be completed successfully to enter PHA II. Results of courses until spring 2013 were included in the study. Due to tragic extra-curricular circumstances data of the PHA I in 2010 were not available for our analysis.

Statistical analysis

We calculated two regression models to identify the predictive power of GPA and HAM-Man. In the first model we included GPA as a stand-alone predictor of practical performance in preclinical laboratory courses. In the multiple regression models (method: enter) we added the HAM-Man as a second predictor in addition to GPA for the estimation of

the total amount of explained practical performance variance by both predictors and

the incremental contribution of the HAM-Man.

The data were analyzed separately for each cohort because we can not ensure that there has been no confounding influence on freshmen’s practical performance over the years, e.g. motivation.

Durbin-Watson statistic is used to detect presence of autocorrelation in the multiple regression models. Collinearity statistics identify high correlations between predicting variables in multiple regression models, what is of relevance to avoid errors for individual predictors. The Software package PASW Statistics 18.03 was used for all data calculations.

Results

Descriptive statistics

Due to dropout the number of participants in PHA I and PHA II are lower than in the TPK (see table 1 (Tab. 1)). Of particular note were the significant HAM-Man score differences between the cohorts (F=15.3; p<.001). The 2009 cohort showed significantly lower HAM-Man scores than the 2008 (p<.05) and the 2010 (p<.001) cohort, but there was no significant difference between the 2008 and the 2010 cohort. Therefore, we decided to perform separate analyses for each cohort.

The assumption of normal distribution had to be rejected for most predicting and control variables, but could be confirmed for all performance variables as the significance of the Shapiro-Wilk test indicated (see table 1 (Tab. 1)). However, Pearson (r) and regression coefficients (ß) are not affected by violations of normal distribution, but the reliability of their significance tests is doubtful [23], [24]. Every multiple regression model passed Durbin-Watson factor as well as collinearity statistics so we assumed reliable parameter estimations.

Correlation analysis

The relationship between the two predictors HAM-Man and GPA was very weak with insignificant correlations within the range 0.105Man and practical performance in the first two laboratory courses (TPK, PHA I). Furthermore, we detected moderate correlations with practical performance in PHA II(r<-0.3), which wer significant (p<.05) in one cohort. For GPA we only detected a significant correlation with TPK in cohort 2010 (r=0.242, p<.05) and an insignificant correlation stronger than r>0.3 to PHA II only in cohort 2008. Control variables age and gender did not give evidence for covariance according to HAM-Man as predicting variable, but age was strongly correlated to GPA (p<.01, see table 2 (Tab. 2)).

Table 2

Correlation analysis

Regression models

Regression analyses revealed little influence of GPA on practical laboratory performance in all three cohorts (see table 3 (Tab. 3)). In five out of eight models even less than one percent of performance variance in preclinical laboratory courses could be explained by GPA. The incremental variance explained by the HAM-Man was up to 20.5% (see table 3 (Tab. 3)). The practical laboratory performance variables were influenced more strongly by HAM-Man than by GPA, as indicated by the standardized regression coefficients in each multiple regression model, emphasizing the HAM-Man as a stronger predictor of practical laboratory performance in preclinical dentistry (see table 3 (Tab. 3)).

Table 3

Predicted technical course performance by HAM-Man and GPA

Discussion

Does our wire-bending test HAM-Man prove to be a suitable predictor of practical, dental performance in preclinical laboratory courses? In addition to GPA the HAM-Man results explained up to 20.5% of practical performance variance, which supports findings from Kao et al. [9]. This is a clear indication for its usefulness as one instrument in the dental student selection process. We also confirmed that GPA is a low quality predictor of practical performance in laboratory courses (see table 3 (Tab. 3)).

One restriction in many validation studies in dental education is the small sample size. We circumvented this problem by analyzing three cohorts. Due to diverging sample characteristics as e.g. the mode of student selection or test motivation we analyzed these cohorts independently. By comparing the results of the different samples a clear picture arises showing a much better prediction of success in practical courses by HAM-Man than by GPA.

Very low correlations between HAM-Man and GPA (see table 2 (Tab. 2)) indicated a good discriminant validity in line with conclusions of prior studies [9], [14]. GPA is more focused on cognitive abilities, whereas the HAM-Man intends to measure manual dexterity. The HAM-Man reflects learning of new or unfamiliar psychomotor skills, of spatial relationships, and of dental techniques in preclinical laboratory courses. Dental techniques require not only the assessment, but also handling of shapes, length, depth, and materials. Moreover, students have to distinguish between acceptable and unacceptable work results, detect and correct errors, and interpret clues correctly by themselves. The better an instrument represents these kinds of dental procedures and techniques, the better its prediction of manual performance in dentistry will be.

We came across some turbulence in our data. The deviant HAM-Man score of the second cohort (see table 1 (Tab. 1)) is very probably due to the lack of an incentive in 2009. Moreover, the non-parametric distribution of our variables did not fit the mathematical requirements for regression analysis, which leads to unreliable confidence intervals. We therefore refrained from interpreting the significance of our findings, and analyzed the three cohorts separately to make sure that we report a trend instead of a random effect. The weakly negative relations of GPA and laboratory grades (see table 2 (Tab. 2)) should not be interpreted as an indication that a lower GPA score leads to better practical performance in technical courses. Correlations could be influenced by diverging sample characteristics, range restriction in GPA due to admission criteria, sample size, or test motivation. The reported strong correlations between GPA and age were not surprising. The German admission procedure gives applicants with lower GPA scores a chance to study dentistry only if they wait several years for their admission (“Wartezeitquote”). The HAM-Man proved to be a useful admission criterion for dental school. In future studies we will examine the relation of dexterity and spatial abilities to identify further selection tools.

Conclusions

The wire-bending test HAM-Man is a valuable additional selection instrument for applicants of dental schools. It can be easily administered with low financial and organizational effort. In addition to the GPA, a wire-bending test would not only improve prediction of student’s practical performance in preclinical dentistry, but also chances of admission for motivated students with a lower GPA.

Acknowledgements

We thank Prof. U. Koch-Gromus, Prof. Bärbel Kahl-Nieke and Prof. Guido Heydecke for their general support of our research. Our special thanks go to Prof. Heike Korbmacher-Steiner, Dr. Sascha Pieger and PD Dr. Hartwig Seedorf for their helpful suggestions, data supply and discussions. Furthermore, we thank Alexander Vogelsang for his essential data merging work, Julia Weinberg for data handling and Dr. Eva Vahle Hinz and Lutz Knobloch for their endless ratings and helpful feedback. Our study was supported by the “Förderfonds Lehre” of the University Medical Center of Hamburg Eppendorf and the “Universitätskolleg Hamburg of the Federal Ministry of Education and Research” (BMBF).

Competing interests

The authors declare that they have no competing interests.