The effect of e-learning on the quality of orthodontic appliances.

PURPOSE: The effect of e-learning on practical skills in medicine has not yet been thoroughly investigated. Today's multimedia learning environment and access to e-books provide students with more knowledge than ever before. The aim of this study is to evaluate the effect of online demonstrations concerning the quality of orthodontic appliances manufactured by undergraduate dental students.
MATERIALS AND METHODS: The study design was a parallel-group randomized clinical trial. Fifty-four participants were randomly assigned to one of the three groups: 1) conventional lectures, 2) conventional lectures plus written online material, and 3) access to resources of groups one and two plus access to online video material. Three orthodontic appliances (Schwarz Plate, U-Bow Activator, and Fränkel Regulator) were manufactured during the course and scored by two independent raters blinded to the participants. A 15-point scale index was used to evaluate the outcome quality of the appliances.
RESULTS: In general, no significant differences were found between the groups. Concerning the appliances, the Schwarz Plate obtained the highest scores, whereas the Fränkel Regulator had the lowest scores; however, these results were independent of the groups. Females showed better outcome scores than males in groups two and three, but the difference was insignificant. Age of the participants also had no significant effect.
CONCLUSION: The offer that students could use additional time and course-independent e-learning resources did not increase the outcome quality of the orthodontic appliances. The advantages of e-learning observed in the theoretical fields of medicine were not achieved in the educational procedures for manual skills. Factors other than e-learning may have a higher impact on manual skills, and this should be investigated in further studies.

RCT Entities:   Population Interventions Outcomes

Introduction

Quality and innovation in university teaching are continuously evaluated. Currently, these aspects are dominated by the Internet because devices such as smartphones and tablet PCs have become part of the daily lives of students. This ever-increasing mechanization is also present at the university when students are taught over the Internet. It has been shown that from 2008 to 2012, the use of mobile devices for learning increased fivefold.1 The task of any academic institution is therefore to evaluate the benefits of such media tools with respect to universities and their students.

Numerous studies have been conducted on e-learning in the field of medical and dental training in which participants are asked about their attitudes toward this subject. Results of these studies show that the respondents enjoy taking new media course offerings,2–7 without missing the traditional style of teaching.8

The majority of studies focus on theoretical education. There is a positive trend that e-learning leads to an increase in theoretical knowledge,9–14 and only a few studies have shown no difference in terms of knowledge improvement15 or that e-learning alone leads to poorer results.16

A questionnaire on the topic of e-learning is a valuable tool but does not reflect the quality of the practical outcomes in medicine and dentistry. Only a few studies have assessed the transition from e-learning to practical skills. Investigations of life support training show similar results17 or moderate improvements18 in practical outcomes when e-learning is compared with traditional didactic teaching. In physiotherapy education, students with access to e-learning sources needed less time for musculoskeletal palpation but more time in obtaining ultrasound images.19 Dental students showed better results in carious lesion excavation when traditional lectures are accompanied with live demonstrations.20

The effect of e-learning on practical skills is yet not well known. There is a lack of data, specifically in disciplines where manual dexterity is required. Therefore, the present investigation aimed to evaluate the influence of online video material on students’ manual skills in manufacturing orthodontic appliances. The hypothesis is that live demonstration plus access to an online demonstration would be more effective than live demonstration alone.

Materials and methods

Ethical approval

The ethics commission of the Medical Faculty reviewed the research project (Ref No: 2014-524-f-N) and gave permission to commence the study. A copy of the Institutional Review Board letter was sent to the Editor-in Chief. All participating students freely gave their informed consent to participate in this trial.

Study design

This study used a semi-blind three-parallel-group randomized controlled trial. The primary outcome was the students’ practical skill measured by the quality of the appliances that were manufactured during a preclinical orthodontic course.

Participants

The inclusion criterion for study enrolment was that students were eligible to attend the first preclinical course in orthodontics at the Dental School of the University of Münster, Germany. This included all students in the sixth semester who had passed the Physikum (preliminary examination). The exclusion criteria were failing the course (excluded from analysis), repeating the course, and violation of the course guidelines (Figure 1). All course students gave their informed consent to participate in the study prior to the beginning of the trial.

Figure 1

Flow diagram of the assessment for eligibility.

Intervention

Regular attendance at the lectures was mandatory. Conventional lectures on orthodontic theoretical background and live demonstrations about appliance manufacturing were presented to all students. Lectures and live demonstrations were held in the lecture hall of the Dental School. Manufacturing of the appliances by the students took place in the student laboratory also located in the Dental School.

During the course, students were required to make three orthodontic appliances (Figure 2). The manufacturing of the appliances was demonstrated by an experienced dental technician. The live demonstrations were supported with cameras and monitors installed in the lecture hall. All procedures were video recorded by professional camera operators and prepared for viewing via the e-learning portal of the department. The video material was divided into sections and labeled for clarity (Table 1).

Figure 2

Appliances that students were required to make during the course. (A) Schwarz Plate (B) Karwetzky’s U- Bow Activator Type 1 (C) Fränkel’s Functional Regulator Type 3.

Table 1

Film processing

Appliance	Content	MB	Duration
Schwarz Plate	Construction of the brackets	405.5	16:36
	Construction of the labial bow; construction of the frontal feather	371.9	14:48
	Position of the stretching screw; overpass in dental plastic	472.2	17:29
Karwetzky’s U-Bow	Articulation of the models; construction of labial bows	443.1	11:18
Activator Type 1	Construction of the frontal feathers; position of the stretching screw	387	14:20
	Insertion of the U-Bows	220.2	07:20
Fränkel’s Functional	Introduce, prepare models; construction of bite	339.8	12:23
Regulator Type 3	Construction of the labial bow; construction of the wire elements in the lower jaw	368	13:15
	Construction of the connecting elements in the upper jaw	356.9	11:53
	Construction of the palatal connector in the upper jaw; construction of the frontal feather	354.3	12:05
	Stannic foil and palatal connector	227.2	07:00
	Overpass in dental plastic	339.4	12:17

Abbreviation: MB, megabyte.

At the beginning of the course, participants were randomly divided into three groups. Randomization was performed by the investigator SS-B using a dice roller. Group one received conventional lectures and live demonstrations during the course schedule. Group two attended the lectures and also had access to an online blog. The blog contained general information about the course, handouts of oral lectures, links to relevant literature and a step-by-step description of the appliances’ manufacturing process.

Group three had access to all resources of groups one and two plus access to the online video material. Students in group three were allowed to download the videos on their devices for offline use.

Evaluation

At the end of the course, all custom-built appliances made by the students were collected and pseudonymized by the investigator SS-B. Tutors (postgraduate students of the Department of Orthodontics), who were blinded to the groups and the students’ names, rated the appliances according to a predetermined point system. It was based on a 0–15 point scale in which 0 represents the lowest and 15 the highest achievable score (outcome measure). The evaluation protocols for the different appliances are shown in Tables 2–4.

Table 2

Grading system Schwarz Plate

Exercise	Points
Construction of the Adams bracket	2
Construction of the Bullet bracket	1
Construction of the Triangle bracket	1
Position of the stretching screw	1
Construction of the labial bow	1
Construction of the frontal feather	1
Overpass in dental plastic	2
Without sharp edges	2
Right expansion of the dental plastic	2
Good fix on the model	2
Overall	15

Table 3

Grading system Karwetzky’s U-Bow Activator Type 1

Exercise	Points
Construction of the labial bows	2
Construction of the frontal feathers	2
Position of the stretching screw	1
Overpass in dental plastic	2
Without sharp edges	2
Right expansion of the dental plastic	2
Good fix by the patient	4
Overall	15

Table 4

Grading System Fränkel’s Functional Regulator Type 3

Exercise	Points
Construction of the labial bow	1
Construction of the wire elements in the lower jaw	1
Construction of the connecting elements in the upper jaw	1
Construction of the frontal feather	1
Construction of the palatal connector in the upper jaw	1
Overpass in dental plastic	2
Without sharp edges	2
Right expansion of the dental plastic	2
Good fix on the model	4
Overall	15

Measurement error

Eighteen pseudonymized appliances were randomly selected and both tutors evaluated them independently according to the evaluation protocol.

Statistics

The statistical analysis of the data was performed using the statistical program GraphPad Prism 4 (GraphPad Software, San Diego, CA, USA). Variance analysis and a non- parametric t-test were utilized. To estimate inter-rater agreement, Cohen’s kappa values were calculated. To further differentiate Cohen’s kappa test, Spearman’s rank correlation coefficient was calculated. Means were compared using Dunnett’s test. A significance level with a 5% error probability level (P<0.05) was observed.

Results

The first orthodontic course at the Dental School has a fixed group size of 55 students. All students were included in the study and randomized into one of the three groups. The trial started with the first lecture of the course and ended on the closing date for the completion of the appliances. During the course, five students violated the course guidelines and were excluded from the analysis. There were no further dropouts. There are no significant group differences concerning age and sex (Table 5).

Table 5

Representation of group distribution

Group	Number	Sex	Age, M ± SD	E-learning
1	17	f=11; m=6	25.4±2.5	No access
2	16	f=9; m=7	25.6±4.7	Online written instructions
3	17	f=11; m=6	25.2±1.9	Full access

Abbreviations: f, female; m, male.

Cohen’s kappa for inter-rater agreement was 0.212, different from 0, which means that this is not a random match (P=0.029). According to Landis and Koch,21 this is a fair agreement. However, a more reliable measure for ordinal data is the Spearman’s rank correlation coefficient, which was 0.834 (P<0.001). Cohen’s kappa measures the agreement between the evaluations of two raters when both are rating the same object on categorical scale. The Spearman’s coefficient is a parameter-free measure of correlations, which means it measures how well an arbitrary monotonic function to describe the relationship between two variables, without making any assumptions about the probability distribution of the variables.22

Group comparisons

Concerning the different appliances made, there was no significant difference in the outcome score for all students (Table 6). The distribution according to the appliance type and group is shown in Figures 3–5.

Table 6

Mean outcome scores and standard deviations concerning groups and appliances

	Group 1	Group 2	Group 3	Sum
Schwarz Plate	7.8±2.8	8.8±2.4	7.1±3.0	7.8±2.7
Male (n=6)	9.5±1.9	7.8±2.3	5.8±2.3
Female (n=11)	6.8±2.8	9.6±2.4	7.8±3.3
U-Bow Activator	7.2±2.2	7.4±2.8	7.1±2.7	7.2±2.5
Male (n=7)	7.0±1.9	6.7±2.6	6.3±2.7
Female (n=9)	6.3±2.7	7.9±3.1	7.6±2.8
Fränkel Regulator	6.2±2.4	5.9±2.6	6.9±2.6	6.4±2.5
Male (n=6)	6.5±1.6	5.3±1.4	4.3±1.2
Female (n=11)	4.3±1.2	6.4±3.2	8.3±2.0
Average	7.1±2.5	7.3±2.7	7.0±2.7

Figure 3

Result-specific differences in the Schwarz Plate on a 15- point scale.

Figure 4

Result-specific differences in Karwetzky’s U-bow on a 15- point scale.

Figure 5

Result- specific differences in Fränkel’s functional regulator Type 3 on al 15- Point scale.

The Schwarz Plate received the highest score, whereas the Fränkel Regulator had the lowest score (Figure 6). Within the groups, there was a similar pattern without any significant difference.

Figure 6

Device-specific differences on a 15- point scale.

Note: **=P>0.01.

Abbreviations: SP, Schwarz Plate; UBA, Karwetzky’s U-Bow Activator; FR, Fränkel’s Functional Regulator Type 3.

Considering the different appliances, the results show that group two had slightly higher scores than the other groups, except for the Fränkel Regulator. However, the differences are not significant.

Concerning sex differences, females showed better outcome scores than males in groups two and three. The highest difference was found in the Fränkel group with 4.3±1.2 points for males and 8.3±2.0 points for females. Given the small group size, a statistical test was not applied.

To assess whether age has an effect on the outcome scores, the data were split according to participants’ age: ≤24 years (n=23) and ≥25 years (n=27). Although there are differences in the outcome scores because of the small numbers of the subgroups, the differences are not significant.

Discussion

In this study, we assessed the effect of e-learning resources on the outcome quality of orthodontic appliances manufactured by dental students in their first orthodontic course. Development of e-learning content is a time- and resource-consuming process aimed to improve students’ theoretical and manual skills in various fields of education. It is therefore mandatory to evaluate the benefit of new teaching methods. Particularly in disciplines where manual dexterity is required, we think that additional resources like e-learning content can improve students’ ability to convert what they learn during lectures into practical knowledge. Compared with theoretical knowledge, manual outcome quality in medical education is difficult to measure and there is a lack of data in the literature.

Strengths and limitations of the study

Chen et al23 discussed teaching studies and suggested that methods are valid only if a) students are assigned randomly, b) quality and performance of the demonstrators are identical, and c) the content of the lectures and the learning environments are the same. All of these conditions were ensured in this study.

However, one limitation of the study may be the restricted number of participants assessed. The first orthodontic course at the Dental School in general is limited to students in the sixth semester who have passed the Physikum (preliminary examination). The course size depends on the number of laboratory work places available. Allocating students from higher semesters would bias the study because of their gained knowledge and skills in orthodontics. For example, Foley24 showed that the outcome quality between semesters varies. On the other hand, restriction to beginners could be seen as strength of the study because they have the same level of knowledge. Johnson et al showed that when students have a common subject but come from other disciplines, this leads to different behaviors in terms of the commitment to the learning progress.25

Randomization of participants is also a strength, but the procedure used should be discussed. Dice tossing is not truly random but pseudorandom.26 However, in this study, the bias of pseudo-randomization should be neglected.

Grading systems that are based on point scales are used in orthodontics to assess malocclusions and treatment outcomes.27,28 There are no validated indices available to evaluate the quality of orthodontic appliances. However, the validated American Board of Orthodontics Objective Grading System has a comparable reliability to our score (Table 7). The 15-point scale has been used for years at the Dental School, and this score is equivalent to a passing grade in the overall school system. Rater and participating academic staff were familiar with the application of the score. We therefore assume that the score system did not bias the results.

Table 7

Comparisons of reliability measurement between the American Board of Orthodontics Objective Grading System and a 15-point scale system

Studies	Spearman	Kappa x (interpretation)
Song et al27	0.64	0.58 (moderate)
This study	0.83	0.21 (fair)

Students were explained the study design and the hypothesis of the study to be able to give their free informed consent to participate. This involves the risk of bias with regard to the Hawthorne effect. We assume that this type of bias was minimized by the fact that the evaluation of the appliances (score) had no effect of passing the course. Moreover, participants were told that they will be unaware of their particular score. Another discussion point would be whether the Hawthorne effect may have been responsible for the student performance in the courses. On the one hand, it was inevitable in this study to let the students ignorant as you could communicate with each other during the course. On the other side, it is also improbable that the knowledge has changed over the study outcome, as the students were told that their performance will have no benefit for them. Which in turn leads to the hypothesis that the students carefully carried out the production of the devices using online courses when no advantages in terms of knowledge queries or university classroom benefits were present. In the sixth dental semester at the University of Münster, students are required to undertake many tasks and manufacturing devices is only a small part of their training. The students were not informed that the manufactured devices had been evaluated, or that a knowledge test of the manufactured equipment was not performed. Thus, whether a knowledge check would lead to significant differences in this field remains unclear. The theory was that students would prefer to use only the easily accessible, quickly viewable online answers for their exams.

A further bias could be the knowledge exchange between students because it was not possible to separate participants during the course. This problem was encountered by explaining the study protocol and behavior during the course prior to the study.

A further strength of the study could be seen in the low dropout and the statistical power, although fixed group size was fixed. Age and sex of the groups did not differ significantly, and all participants have completed the same basic studies in dentistry and have technical courses of undergraduate successfully passed. Further, all participants were skilled in the use of the e-learning resources that were maintained by the university’s IT structure.

E-learning is more than desire among students. This aspect was evident in the surveys of students where over 50% of respondents took advantage of the e-learning offer29 and 95% responded that e-learning was helpful.30

Moreover, the economic benefit of e-learning has been shown. The University of Chicago conducted a study to determine whether online lectures about sleep medicine could replace the traditional classroom setting. After 1 year, the cost of the online course was paid off, while the cost of employing the teaching staff was incurred repeatedly.31

Cohen et al32 conducted a meta-analysis on the impact of visual-based instruction in education. The authors compared the effects of visual-based and conventional teaching in five areas: achievement, retention, aptitude-achievement correlation, student rating, and course completion. In the area achievement 37 of 65 studies, visual based was better than examination performance in a conventional class and a clear majority of studies favored visual based. In the area retention, the difference, based on six studies, was not statistically significant. By 10 aptitude-achievement studies no reported a significant difference. In the point student attitudes five of six studies favoring visual based. Only 10 studies examined the effect of visual based on course completion. The average withdrawal rate was 13.1% in visual based, and the average rate in the conventional classes was 13.2%. Given these results, it could be speculated that information technology development facilitates e-learning.

The impact of e-learning on theoretical knowledge gain has been well investigated.9–14

However, there are only a few studies that evaluate e-learning with respect to teaching practical skills in comparison with traditional lectures. Preston et al33 divided graduate-level physiotherapy students into two groups to evaluate whether e-learning leads to an improvement in the practical skills applied to patients when coupled with traditional teaching methods. Although the results showed no significant differences, the group with the additional e-learning module tended to achieve slightly better results.33

Hards et al17 divided 20 anesthesia residents into two groups to examine the management of maternal cardiac arrest by using high-fidelity simulation and comparing the subsequent performance following either didactic teaching or e-learning. The results showed no significant differences; however, they discovered that e-learning and didactic teaching offer comparable benefits.17

In a study by Perkins et al,18 3,732 health care professionals received an advanced life support. One group of participants completed a 1-day course with e-learning; the other group participated in a conventional 2-day course. In comparison with traditional advanced life support, the e-learning group had a lower pass rate for cardiac arrest on the simulation tests, equal scores on a knowledge test, and diminished costs.18

Rosa et al20 assessed the influence of e-learning on students’ ability to remove artificial decayed dentine by hand. Twenty-seven students were divided into two groups in order to compare the traditional cavity removal lecture with a traditional lecture that included a practical, live demonstration. An evaluation of the practical cavity removal skills between the two groups showed no significant difference, although the group that received the live demonstration enjoyed a temporary benefit. However, students who participated in the live demo were able to remove nearly twice as much carious dentine in the same time amount of time as the other group.20

Chen et al23 investigated two groups of 16 dental assistants who were educated at the orthodontic department of the University of London. A test was conducted to determine whether the dental assistants obtained a better understanding by watching a 15-minute video demonstration on practical bracket positioning instead of listening to a 30-minute lecture.

No significant difference was found between the two groups, although with respect to bracket angulation, the group that viewed the video demonstration tended to show better results. Chen et al concluded from this study that it is slightly more effective to show video demonstrations of practical movements.23

There is hardly any published study that addresses e-learning effects on the quality of manufacturing processes in education. The results of the few studies on the practical outcome quality suggest that e-learning has only minor effects. Further studies will be necessary to proof or disproof the benefits of e-learning on manual skills.

Conclusion

Our study suggests that providing students with time and location-independent e-learning resources additional to the classical lectures did not increase the outcome quality of the orthodontic appliances. The advantages of e-learning observed in the theoretical fields of medicine were not achieved in the educational procedures for manual skills. Factors other than e-learning may have a higher impact on the outcome quality. Further studies will be necessary to proof or disproof the benefits of e-learning on manual skills.