How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study.

The interpretation of medical images is an error-prone process that may yield severe consequences for patients. In dental medicine panoramic radiography (OPT) is a frequently used diagnostic procedure. OPTs typically contain multiple, diverse anomalies within one image making the diagnostic process very demanding, rendering students' development of visual expertise a complex task. Radiograph interpretation is typically taught through massed practice; however, it is not known how effective this approach is nor how it changes students' visual inspection of radiographs. Therefore, this study investigated how massed practice-an instructional method that entails massed learning of one type of material-affects processing of OPTs and the development of diagnostic performance. From 2017 to 2018, 47 dental students in their first clinical semester diagnosed 10 OPTs before and after their regular massed practice training, which is embedded in their curriculum. The OPTs contained between 3 to 26 to-be-identified anomalies. During massed practice they diagnosed 100 dental radiographs without receiving corrective feedback. The authors recorded students' eye movements and assessed the number of correctly identified and falsely marked low- and high prevalence anomalies before and after massed practice. Massed practice had a positive effect on detecting anomalies especially with low prevalence (p < .001). After massed practice students covered a larger proportion of the OPTs (p < .001), which was positively related to the detection of low-prevalence anomalies (p = .04). Students also focused longer, more frequently, and earlier on low-prevalence anomalies after massed practice (ps < .001). While massed practice improved visual expertise in dental students with limited prior knowledge, there is still substantial room for improvement. The results suggest integrating massed practice with more deliberate practice, where, for example, corrective feedback is provided, and support is adapted to students' needs.

Introduction

Taking radiographs is a standard diagnostic procedure in dentistry. In contrast to other medical disciplines, which rely on the expertise of certified radiologists, dentists perform and interpret radiographs themselves. As in other medical fields, interpretation of medical images is a highly error-prone process in dentistry with error rates between 19% and 41% even for experts [1]. These errors can have severe consequences for patients. Thus, it is crucial that dentistry students develop visual expertise—knowledge about how to search and detect anomalies—during their study [2]. A frequently used and rather traditional instructional method for teaching students how to read and interpret radiographs is massed practice. Here students are required to provide a full written description of their observations including the identified anomalies for each radiograph. This procedure is repeated for a high number of radiographs, which are selected to reflect the full range of potential anomalies that students could be exposed to. No other learning activities are interspersed and only limited corrective feedback, if any, is provided. The reason for the lack of feedback often lies in the fact that medical teachers do not have sufficient time and resources to review their students’ diagnostic competence for such a high number of radiographs. Whereas educational research has shown beneficial effects of massed practice for certain types of tasks [3, 4], evidence regarding its effectiveness for the development of visual expertise in medical and dentistry studies is scarce [5]. Moreover, it is yet unclear how students process radiographs, which might have important consequences for their ability to identify anomalies. Therefore, we studied the development of diagnostic competence and gaze behavior in dentistry students during an obligatory standard radiology massed practice course to determine its effects and derive possible implications for improving training.

Massed practice and the development of visual expertise in radiology

According to Nodine and Mello-Thoms “massed practice is the main change agent in achieving expertise” [6] (p. 868), with a strong relationship between the number of images read and diagnostic accuracy. Moreover, perceptual sensitivity with regard to recognizing low-contrast targets (e.g., lighter areas that may represent nodules) has been shown to improve with massed practice [7]. However, apart from the study of Sowden et al. [7] and anecdotal evidence, to our knowledge there is no evidence that massed practice (without or with corrective feedback) is an effective instructional method for the development of visual expertise.

Nodine and Mello-Thoms [6] delineate that visual expertise requires the development of domain-specific cognitive skills and decision strategies. Observers need to acquire knowledge about perceptual features of anomalies required for their identification. In addition, they need strategies that allow them to interpret conspicuous features by relating them to categories of anomalies. From a theoretical perspective, the mere massed exposure to radiographs without corrective feedback should mainly affect students’ ability to match conspicuous features in the images to mental schemata about anomalies (illness scripts, [8]). Massed practice therefore may increase students’ experience in finding conspicuous features, which should be evident not only in their ability to correctly identify anomalies (accuracy) but also in their gaze behavior as a fine-grained process-oriented measure of visual expertise. Eye tracking serves as a valuable research tool to study visual expertise development in the medical field [2, 5, 9]. According to this research, experts tend to fixate images for a shorter time and have more and earlier fixations on relevant areas containing conspicuous features compared to non-experts [10]. During a fixation the eyes remain relatively still on a certain location, which allows information intake or active processing of perceptual features [5, 11]. Importantly, while there is a wealth of studies addressing expert-novice differences in medical image interpretation [12-14], there are no studies to the authors’ knowledge on how visual expertise develops during unversity training of students.

Moroever, there is hardly any research (two studies see [15, 16]) on medical image interpretation regarding panoramic radiographs (orthopantomograms, OPTs), which are frequently used in dentistry, and which is our area of interest. OPTs typically contain multiple, diverse anomalies within one image [17]. Typical anomalies of the dentition are located in the central area of an OPT (Fig 1) around the teeth and adjacent alveolar bone, for example, root remnants, periodontal defects, and apical lesions. The peripheral area of an OPT (Fig 1) shows the temporomanibular joints, maxillary sinus, parts of the orbital cavities, and soft tissues of the neck including the hyoid bone, where typical anomalies contain cysts or tumors of the soft and hard tissue or calcifications of salivary glands, lymph nodes or the carotid artery. Anomalies located in the peripheral area are of rather low prevalence [18, 19] whereas high prevalence anomalies are predominantly located in the central area.

Fig 1

Example orthopantomogram (OPT).

The left panel shows an OPT with a highlight on the central area whereas the right panel highlights the peripheral area of an OPT.

Importantly, the task of diagnosing an OPT is very different from diagnosing, for example, mammograms or chest radiographs that have been used in previous studies on visual expertise in radiology, e.g., [14, 20–26]. For example, in a study by Donovan, Manning and Crawford [27] chest radiographs were presented to participants with the task to detect lung nodules only. In a mammography screening-diagnostic task, Nodine and colleagues [26] asked experts with different levels of experience to detect malignant lesions; Mammography radiographs typically contain only a modest number of lesions, different from OPTs that can entail a large number of very different anomalies. Consequently, it may be particularly challenging to develop visual expertise regarding OPTs for a number of reasons:

A high interindividual variability of the visual appearance of both normal and abnormal anatomy as well as the phenomena caused by artifacts and superimpositions makes it difficult to detect anomalies [28-31]. In addition, even panoramic radiographs of apparently “dentally healthy patients” can often show one or more dental or non-dental anomalies [32-36]. Therefore, diagnosing OPTs relies on hybrid search for multiple different targets requiring observers to know all characteristics of potential targets, and match those to actual visual characteristics of radiographs [37]. The occurrence of multiple targets is known to complicate visual search and make it less effective; the prevalence of targets also affects visual search processes [38]. Moreover, anomalies are also found in the peripheral areas of the jawbone or in the maxillary sinus or are part of the X-ray as superimpositions of soft and hard tissue in the vicinity of the oral cavity. These may include a number of secondary findings of general medical relevance (oncology, cardiovascular disease) that require referral to other specialists for further diagnosis. To conclude, there are several reasons why studies in other medical imaging domains have limited informative value for teaching and learning the task of diagnosing OPTs [39].

Against the backdrop of the rather weak empirical basis regarding the effectiveness of massed practice for the development of visual expertise, we investigated the following research questions:

Does massed practice improve diagnostic accuracy when reading OPTs and does its effectiveness depend on prevalence rates and hence the anomaly’s location in either the center or the periphery?

How does massed practice change students’ gaze behavior when reading OPTs as revealed by tracking students’ eyes during the inspection of OPTs? Based on visual expertise research, we expected that an increase in diagnostic performance due to massed practice should be accompanied by earlier, longer, and more frequent fixations on anomalies. Moreover, we expected students to cover a larger proportion of an OPT during visual search after massed practice.

In brief, our results show that as expected massed practice had a positive effect on detecting anomalies, especially for low-prevalence anomalies. Also, we provided evidence that massed practice leads to changes in gaze behavior. After massed practice students covered a larger proportion of the OPTs during visual inspection and their coverage positively predicted the detection of low-prevalence anomalies. Students also focused longer, more frequently, and earlier on low-prevalence anomalies after massed practice.

Materials and methods

Participants

Sixty-nine dental students participated in this study. They were tested three times during their first clinical semester, which is the second half of their third year: (i) prior to massed practice (pre-test), (ii) directly after massed practice (post-test), and (iii) at the end of the semester (13 weeks after pre-test). Because the analyses showed no differences in dependent variables between the second and third measurement, we decided to use only the pre- and post-test for the analyses. In cases for which the second measurement was missing, we replaced it with values from the third measurement (n = 5 students). Nevertheless, data from 14 students had to be excluded due to incompleteness, leaving 55 students (Mage = 24.05, SD = 2.56 years old; 61.8% female). Due to insufficient eye tracking quality and/or calibration accuracy, data from another eight students had to be excluded from the eye-tracking analyses. The remaining 47 students were Mage = 23.94, SD = 2.51 years old and 59.6% were female. The study was approved by the Ethical Review Board of the Leibniz-Institut für Wissensmedien Tübingen under number LEK 2017/016. Participation in the study was voluntary; all students provided written informed consent including that their data could be analyzed and published.

Materials and apparatus

Students were asked to mark anomalies in 10 OPTs that had been taken during routine diagnostic processes in the hospital and were of good image quality. Those 10 OPTs showed between three and 26 anomalies (k = 95 anomalies in total). No normal images (radiographs without pathological findings) were included. We used the same ten OPTs in the pre- and post-tests because research suggests that observers do not recognize previously seen radiographs, which suggests that the repetition of OPTs in the pre and post-test may only have little if any effect on diagnostic performance [25, 40, 41].

Two experts (a maxillofacial radiologist and a prosthodontist, each with over 13 years of clinical experience) selected and coded the OPTs independently and agreed on a solution scheme for coding the students’ responses.

Stimuli were presented on a computer screen (1920 x 1080 pixels) and at maximum screen brightness. Eye movements were recorded using a video-based remote eye tracking system by SensoMotoric InstrumentsTM (SMI 250REDTM; 250 Hz sampling rate). A 13-point calibration image was used to calibrate the system. We used the SMI BeGazeTM default velocity-based algorithm (eye movements with a speed lower than 40°/s were classified as fixations; eye movements with a speed above 40°/s as saccades) to detect events in the gaze data. The calibration accuracy was below 0.98° visual angle. The mean tracking ratio was 95.03% at the first and 94.92% at the second measurement. The light in the experimental room was kept constant throughout the experiment (range: 30 to 40 lx).

We aggregated gaze data by areas of interest (AOIs) in two different ways: (i) we drew AOIs around anomalies to compute fixation time, fixation count, and the time to first fixating anomalies, and (ii) we used gridded AOIs (14x11 and 15x11 grids depending on the size of the OPT) to compute the overall gaze coverage of the OPTs (see Fig 2).

Fig 2

Example orthopantomogram (OPT) with areas of interest (AOIs).

The left panel shows AOIs located around anomalies on an OPT. In addition to a bilateral shortening of the collum mandibulae, the colla and condyles present hypoplastic on both sides. Shortened roots of tooth 16, lacking the apical tips with periapical translucencies, possibly corresponding to a status post root resection. In region 26, 27 and 28 spheroid, sharply defined homogeneous opacification (projecting on the maxillary sinus floor) corresponding to a mucosal antral pseudocyst. Translucencies in the approximal areas of teeth 46 and 47 possibly indicating caries. (Dental notation according to the FDI-system). Gridded AOIs, as displayed in the right panel were used for the computation of the overall gaze coverage.

Instruments

Accuracy

We computed the accuracy as the sum of correctly identified anomalies separately for the central and peripheral parts, respectively (see Fig 1) of the OPTs and transformed theses scores into percentages for easier interpretation.

False positives

When students marked an area in an OPT as being abnormal but did not actually contain an anomaly, we coded those markings as false positives, with reference to either the central or peripheral area of the OPTs.

Eye tracking parameters

We used four measures related to students’ gaze behavior: (i) the mean fixation time on central and peripheral anomalies (in milliseconds), (ii) mean number of fixations on central and peripheral anomalies, (iii) mean time to first fixation on central and peripheral anomalies (in milliseconds), and (iv) the coverage of OPTs as the percentage of grids that were fixated at least once. The first fixation on each OPT was excluded from data analyses, because this fixation can be traced back to the fixation cross, that was presented just before each OPT. Data were averaged across stimuli.

Dental pathology test

A multiple-choice test with 20 items assessed knowledge about dental pathology (e.g., misaligned teeth, root resorptions, soft tissue issues). The items were self-developed with each five answer options (including ‘I cannot answer the question yet/I don’t know’) and one option being correct.

Experimental procedure

We collected the data in multiple group sessions. In the pre-test session, students were asked to perform a diagnostic task, which was to first look at the OPT (limited to 90 seconds) and then mark anomalies (without time constraint). In the marking phase students used a mouse-operated drawing tool to draw ellipses around conspicuous areas. Before each OPT, a fixation cross was displayed for two seconds. After students had diagnosed all 10 OPTs, they worked on the dental pathology test. In the remaining semester students attended weekly lectures on radiology. The lectures addressed radiation physics and biology, radiation exposure and protection, dosimetry, technical equipment, imaging procedures, quality control, legal directives, and technical exercises. In addition, OPTs are introduced including the clarification of anatomical structures, common anomalies, and artefacts from technical failures as well as common anomalies. Finally, students perform massed practice of reading dental radiographs. Within 24 hours spread across one week each student diagnosed 100 dental radiographs with a written report for each. As to expect due to prevalence 15–20% were without pathological findings. The students worked in teams of three without receiving any corrective feedback by the teacher. Thereafter, two out of these hundred radiographs are discussed in depth with each student together with an experienced radiologist (teacher). At the end of the massed practice week, students were invited to the post-test session. In the post-test we asked them to repeat the diagnostic task used in the pre-test. The diagnostic task was repeated at the end of the semester; in addition, the dental pathology test was administered again.

Statistical analyses

Repeated-measures analyses of variance (ANOVA) with two within-subjects factors time of measurement (ToM; pre/post massed practice) and anomaly location (AL; central/peripheral area of an OPT) were used to determine the effects of massed practice on accuracy, false positives, fixation time, number of fixations, and time to first fixation. We used Bonferroni-corrected post-hoc comparisons to disentangle significant interactions. For the gaze coverage of OPTs and the control variable dental pathology knowledge we computed repeated-measures ANOVAs only with the within-subjects factor ToM. Finally, a correlation analysis was conducted to test how changes in gaze behavior were related to changes in accuracy. For the ANOVAs effect sizes are reported in ηp2 to denote small (range from 0.01 to 0.05), medium (range from 0.06 to 0.13), or large effects (from 0.14 upwards), respectively. The effect size d is used to denote small (range from .20 to .40), medium (range from .50 to .70) and large effects (from .80 upwards) resulting from pairwise comparisons [42]. The alpha level was set to .05.

Results

Dental pathology knowledge

Students’ knowledge increased significantly from the beginning to the end of the semester, F(1,48) = 151.45, p < .001, ηp2 = .76 (Table 1). However, these knowledge gains were unrelated to diagnostic accuracy (r = -.07, p = .639) and gaze coverage after massed practice (r = .05, p = .748).

Table 1

Means and standard deviations for dental pathology knowledge as a function of the time of measurement.

ToM	Dental pathology knowledge in % correct (standard deviation)
Pre*	18.98 (3.58)
Post**	41.33 (12.28)

SD, standard deviation; ToM, time of measurement.

* Cronbach’s alpha 0.82.

**Cronbach’s alpha 0.51.

n = 49.

Diagnostic competence

Accuracy improved after massed practice, F(1,54) = 431.10, p < .001, ηp2 = .89, and differed between AL, F(1,54) = 72.28, p < .001, ηp2 = .57. These main effects were qualified by a significant interaction, F(1,54) = 49.86, p < .001, ηp2 = .48: Accuracy increased for both central and peripheral anomalies from pre- to the post-test (both ps < .001), but this increase was stronger for peripheral (d = 2.91) than for central anomalies (d = 1.52) (Table 2).

Table 2

Means and standard deviations for diagnostic performance as a function of anomaly location and time of measurement.

ToM and AL	Percentage of correctly detected anomalies♭ (standard	Number of false positives (standard deviation)
	deviation) [raw scores]	[log-transformed values♭]
Pre
Central anomalies	35.27 (10.30)	14.33 (11.74)
	[27.51 (8.04)]	[1.09 (0.29)]
Peripheral anomalies	12.41 (11.75)	1.04 (1.52)
	[2.11 (2.00)]	[0.22 (0.27)]
Post
Central anomalies	51.40 (10.97)	15.51 (12.03)
	[40.09 (8.56)]	[1.12 (0.30)]
Peripheral anomalies	48.34 (12.94)	6.11 (5.65)
	[8.22 (2.20)]	[0.71 (0.38)]

ToM, time of measurement; AL, anomaly location.

♭Used for the analyses.

n = 55.

Because the number of false positives was not normally distributed, we log-transformed the variables (Table 2). Results revealed main effects for ToM, F(1,54) = 45.46, p < .001, ηp2 = .46, and AL, F(1,54) = 251.31, p < .001, ηp2 = .82, as well as a marginally significant interaction, F(1,54) = 3.65, p = .062, ηp2 = .06. The number of false positives in the central areas of OPTs did not change (p = .524), whereas it increased significantly from before to after massed practice in the peripheral area (p < .001). Fig 3 shows students’ diagnostic competence reflected in the accuracy for finding anomalies and the number of falsely marked anomalies as a function of ToM and AL.

Fig 3

Students’ diagnostic performance as a function of ToM (pre and post massed practice) and AL (central/peripheral).

The left panel depicts the accuracy in detecting anomalies whereas the right panel shows the number of falsely marked anomalies.

Gaze behavior

Fixation time

There were main effects of ToM, F(1,46) = 18.14, p < .001, ηp2 = .28, and AL, F(1,46) = 102.48, p < .001, ηp2 = .69, as well as an interaction, F(1,46) = 93.24, p < .001, ηp2 = .67, for the fixation time on anomalies. Whereas fixations on central anomalies were shorter after massed practice (p < .001), fixation times for peripheral anomalies increased from pre- to post-test (p < .001) (Table 3).

Table 3

Means and standard deviations (in parentheses) for variables related to gaze behavior as a function of anomaly location and the time of measurement.

ToM and AL	Fixation time (ms)	Number of fixations	Time to first fixation (ms)	Coverage (percentage)
Pre				32.11 (4.70)
Central anomalies	2,421.77 (741.17)	4.05 (1.22)	23,310.76 (4,998.91)
Peripheral anomalies	2,436.31 (933.24)	5.55 (2.19)	23,960.42 (7,788.68)
Post				45.59 (4.93)
Central anomalies	1,662.70 (541.19)	2.89 (0.71)	28,462.83 (5,758.25)
Peripheral anomalies	4,139.66 (1,306.08)	8.81 (2.34)	17,140.60 (6,243.09)

ToM, time of measurement; AL, anomaly location.

n = 47.

Number of fixations

An analogous pattern holds true for the number of fixations (Table 3). There were main effects of ToM, F(1,46) = 14.91, p < .001, ηp2 = .25, and AL, F(1,46) = 249.01, p < .001, ηp2 = .84, as well as a significant interaction, F(1,46) = 85.68, p < .001, ηp2 = .65. From pre- to post-test the number of fixations significantly decreased for central anomalies (p < .001) but increased for peripheral anomalies (p < .001).

Time to first fixation

Results for the time to first fixating anomalies revealed a significant main effect only for AL, F(1,46) = 31.57, p < .001, ηp2 = .41; ToM: F < 1. Moreover, there was a significant interaction of ToM and AL, F(1,46) = 44.30, p < .001, ηp2 = .49. After massed practice central anomalies were fixated later (p < .001), whereas peripheral anomalies were fixated earlier (p < .001) compared to before massed practice (Table 3). Fig 4 depicts the number of fixations on anomalies and the time to first fixating anomalies as a function of ToM and AL.

Fig 4

Students’ gaze behavior as a function of ToM (pre and post massed practice) and AL (central/peripheral).

The left panel depicts the number of fixations on anomalies whereas the right panel shows the time to first fixating anomalies.

Gaze coverage

The coverage of OPTs increased significantly after massed practice, F(1,46) = 274.69, p < .001, ηp2 = .86 (see Table 3).

Correlations among change scores for accuracy and gaze behavior

We computed change scores for accuracy and gaze behavior measures by subtracting each pre-test value from the value achieved in the post-test. A correlation analysis showed that the increase in accuracy for central anomalies was not related to any changes in gaze behavior. The increase in the accuracy for peripheral anomalies was, however, positively correlated with an increase in fixation time (p = .037) and fixation count (p = .017) on peripheral anomalies and with an increase in gaze coverage (p = .035) (Table 4).

Table 4

Correlations among change score values for accuracy and gaze behavior measures.

AL	Change scores post-pre	Accuracy central anomalies	Accuracy peripheral anomalies
central	Fixation time	.19
	Fixation count	-.09
	Time to first fixation	.02
peripheral	Fixation time	-.19	.31*
	Fixation count	-.22	.35*
	Time to first fixation	.21	-.11
Independent of AL	Gaze coverage	-.27	.31*

AL, anomaly location.

* p < .05.

** p < .01. n = 47.

Discussion

We investigated the effectiveness of massed practice for learning how to read and interpret panoramic radiographs. To this end, we assessed diagnostic performance and gaze behavior before and after a regular massed practice university course. Results revealed that massed practice is an effective instructional method for improving students’ accuracy in detecting anomalies and training them to not only focus on central areas, but also frequently neglected peripheral areas. These improvements were positively related with more attention being paid to these areas, indicating that students modify their visual search strategies due to massed practice. In addition, the improved visual coverage after massed practice positively predicted the accuracy of detecting anomalies in the periphery [22]. These results are promising because they demonstrate the effectiveness of massed practice for learning how to perform the complex hybrid search task of diagnosing OPTs containing multiple, diverse anomalies. Moreover, our study showed that regular massed practice training with 100 radiographs already significantly changes students’ viewing behavior with more and longer fixations on relevant areas of an OPT [10].

Despite revealing improvements, our results also reveal limitations of massed practice as an instructional method. Students made more false positive markings in the periphery after massed practice. Massed practice trained students in finding and matching conspicuous features in the images to their mental schemata about anomalies. Since students adapted their visual search strategies and more fully covered OPTs, it seems that students’ mental schemata about the visual features of anomalies could be further improved by means of systematic and repeated training addressing the variability in the visual appearance of anomalies.

Moreover, students’ accuracy was at about 50% after massed practice, which leaves substantial room for improvement. From our cross-sectional data collection of all dentistry semesters we know that this accuracy level does not change during the further course of their studies. This is despite the fact that students gain further experience by treating their own patients and interpreting radiographs together with supervising experienced dentists [43]. Potential reasons for the stagnation of accuracy may be that this learning process is not systematically oriented towards diagnosing and interpreting radiographs and is strongly influenced by the patient cases available for treatment and the focus of the supervising dentist. Both findings suggest combining massed practice with more deliberate practice [44, 45]. According to the deliberate practice approach domain-specific expertise is the result of structured practice. It is characterized by the adaptation of contents to learners’ expertise level and repetition of contents. Individualized feedback is an important aspect of deliberate practice because it draws attention to those aspects of one's performance that need correction and further practice [46]. Moreover, research consistently shows that spaced practice leads to better learning outcomes compared to massed practice for varying types of tasks (e.g., verbal memory tasks, motor learning) [47-49]. Spaced practice means that the contents to be learned are repeated after a certain time interval and are tested after a further retention interval [47, 50]. Also in radiology teaching and surgical skill training, studies have shown that spaced procedures are beneficial compared to massed practice [51, 52]. Future research should therefore specifically compare massed and spaced practice for learning how to diagnose OPTs to identify potentials for further improvements of students’ accuracy. The present study did not aim at contrasting different training approaches because our focus was on the effects that current practice has on students’ skill development. Therefore, we implemented the present study in a within-subjects design within students’ regular training to achieve a high ecological validity. We acknowledge that this approach has its limitations due to a lack of a control group that would have been trained with a different approach.

Importantly, the current study refers only to the effects of massed practice regarding training of a single skill, where it has been suggested that training of that single skill should occur spaced in time. This recommendation is not to be confused with another instructional design principle, for which multiple labels have been used in the literature, that is, the contextual inference effect [53-55], the variability effect [56, 57] or interleaved practice [58], respectively. Here it is suggested that when training multiple skills, these should be trained in an interleaved way (abcabcabc etc.) to highlight the differences between them. While interleaved practice of multiple skills results in spaced learning of the same skills, it is different from the focus of the present study.

To summarize, the present study complements existing research on medical image interpretation [12–14, 59–61] in that it focuses on the effects of training of students rather than on expert-novice differences and it uses OPTs rather than, for instance, chest radiographs, which require different types of visual search processes. Our results indicate that traditional massed practice training is an effective instructional method to develop visual expertise for interpreting OPTs in dentistry students. Students do not only improve their diagnostic accuracy but also change their visual search behavior due to massed practice training. However, at the same time the effectiveness of massed practice is limited. Further improvements may be achieved by combining massed practice with more systematic training such as deliberate practice.

This is the data file used for the analyses reported in the present manuscript.

(SAV)

Click here for additional data file.

28 Jul 2020

PONE-D-20-05443

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

PLOS ONE

Dear Dr. Richter,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 11 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ezio Lanza, M.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere.

"At this point in time one other manuscript is under review that uses data from the same student population we used in the present study. However, they only used the measure pupil diameter, which is not included in our present manuscript. Therefore, there are no overlaps between the two manuscripts regarding results, data, or figures. But if the reviewers feel that they want to check the overlap themselves we are happy to share the other manuscript upon request."

Please clarify whether this publication was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript.

[Note: HTML markup is below. Please do not edit.]

5. Review Comments to the Author

Reviewer #1: The authors are to be commended for their innovative work, as they sought to “assess the development of diagnostic competence and gaze behavior in dentistry students during an obligatory standard radiology massed practice course”.

Nonetheless, as it is, the manuscript presents two major and important weaknesses

a) The design of the study does not include a control group that might allow us effectively assess and compare different learning outcomes. Other studies using a spaced learning/distributed practice approach appear to show that the use of spacing and testing promotes long-term or durable memory (Morin CE et al. Pediatr Radiol. 2019; 49: 990-999 and Versteeg M et al. Med Educ. 2020; 54: 205-216);

b) Moreover, despite the presented data, there is little to no discussion of the impact on the actual learning outcomes of the students who participated in the study. If one is to assess massed practice, one should also discuss and comment on alternative approaches such as distributed practice.

Unless the authors can, somehow, address these key aspects,

Reviewer #2: Interesting article with new information. Here are a few suggestions/recommendations

- Line 110: I'd suggest using "hyoid bone" instead of only "hyoid)

- Line 131: You mention a few studies (# 15-16 and 20 to 26) but do not provide explanation as to what these studies brought. If pertinent, maybe you could expand on at least a few of these

- Line 132: "oral sinus"... are you talking about the maxillary sinus?

- Line 135: "for why". I think the "for" is not needed here

- Line 146: you talk about "increases", plural. But I'm not sure to what multiple things you are referring to

- Participant section: I'd like to know what year were these students in?

- Methods section: you mention the 1st test, 2nd test, and then a 3rd test 13 weeks after the 1st. When was the 2nd test administered?

- Lines 185 and227/228: I don't think it's relevant to indicate your experts as the authors. Simply explaining their background should be enough.

- Figure 2: I'm not sure I see what's wrong with the TMJs in this case? It might be worth explaning in you caption (lines 204-206) what the experts saw on this image.

Reviewer #3: The article is simple but the idea is satisfactory.

About English language: The manuscript needs an English review in order to sound as native as possible, preferably changing some words that can be understood but are not routine words in academic writing. For example, in the first phrase of the introduction, the words: "Taking radiographs" sound weird. Also, in academic writing, passive voice is preferable, and the article is almost entirely wrote in active voice.

Also, punctuation needs review.

# Introduction:

Please, provide a reference to the phrase: "The reason for the lack of feedback often lies in the fact that medical teachers do not have sufficient time and resources to review their students’ diagnostic competence for such a high number of radiographs." - I don't think the introduction is the right place to author's opinion.

I suggest you to explain further the meaning of "massed practice" with proper references. Explain how the practice is performed in a summarized and systematic explanation. Maybe you can use a figure or a flow chart to illustrate and clearer this point.

In the phrase: "According to Nodine and Mello-Thoms [6]“massed practice is the main change

76 agent in achieving expertise” (p. 868)," please, check if the reference citation is correct.

Please, add references to the phrases: "As a consequence, it may be

122 particularly challenging to develop visual expertise regarding OPTs for a number of

123 reasons. A high interindividual variability of the visual appearance of both normal and

124 abnormal anatomy makes it difficult to detect anomalies. In addition, even OPTs of

125 apparently healthy patients can often show several anomalies"

# Discussion

I suggest you add more references in order to compare your findings. For instance:

Again, maybe it would be better to express your opinion in the discussion, for exemple. Remove or re-wright theses paragraphs.

Introduction, last two paragraphs: I don't think Introduction is the right place to express your opinion and what you expect from the study. You can express it in the discussion.

# Materials and Methods

Participants: you mentioned 55 students in the abstract, however, in the first paragraph of M&M you mentioned, after exclusion criteria applied, 47 students as your final sample. Please, correct it or clarify the numbers divergence.

#Discussion

I suggest you to improve your discussion with other references and comparisons with different studies, such as, for example:

* Performance evaluation of different observers in the interpretation of panoramic radiographs by the mandibular cortical index. DOI: 10.15448/1980-6523.2018.1.29202

*Observer performance in diagnosing osteoporosis by dental panoramic radiographs: results from the osteoporosis screening project in dentistry: https://doi.org/10.1016/j.bone.2008.03.014

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

1 Oct 2020

Dear Dr. Lanza, dear Reviewers,

we would like to thank the editor and the reviewers for the assessment of our manuscript and the feedback offered by the review. In the following, we will address the comments one by one. Reviewer comments are printed in regular font, responses in italics. Moreover, we have added sections from the manuscript whenever we made a change in it (in italics and quotation marks). We hope that this way we have addressed all issues raised by the reviewers and the editor in a satisfactory manner.

The authors

5. Review Comments to the Author

Reviewer #1:

The authors are to be commended for their innovative work, as they sought to “assess the development of diagnostic competence and gaze behavior in dentistry students during an obligatory standard radiology massed practice course”.

Thank you for your time to review our manuscript and providing constructive feedback.

Nonetheless, as it is, the manuscript presents two major and important weaknesses

a) The design of the study does not include a control group that might allow us effectively assess and compare different learning outcomes. Other studies using a spaced learning/distributed practice approach appear to show that the use of spacing and testing promotes long-term or durable memory (Morin CE et al. Pediatr Radiol. 2019; 49: 990-999 and Versteeg M et al. Med Educ. 2020; 54: 205-216);

Thank you for your comment. Indeed, we did not use a control group because we did not aim at contrasting different training approaches. Not only would this have been difficult to implement from an ethical and practical perspective, since we would have experimentally manipulated students’ real study conditions that may affect passing/failing the course. Moreover, our focus was more on the effects that current practice has on students’ skill development. Therefore, we felt that it was necessary to implement our study within students’ regular training to achieve a high ecological validity. Hence, from an experimental psychologist’s point of view, massed practice would have to serve as our experimental group, but it is unclear what the control group would be. A waiting-control group cannot be implemented for practical reasons. Hence, we favored a within-subjects design where the pretest represents the performance level of somebody who has received no training. However, we agree that future research should focus on introducing and evaluating learning approaches that from a psychology perspective can be expected to result in better performance such as deliberate practice and distributed/spaced learning. We added the potential use of spaced learning respectively distributed practice in the discussion section:

Line 481

“Moreover, research consistently shows that spaced practice leads to better learning outcomes compared to massed practice for varying types of tasks (e.g., verbal memory tasks, motor learning) [47-49]. Spaced practice means that the contents to be learned are repeated after a certain time interval and are tested after a further retention interval [47,50]. Also, in radiology teaching, studies have shown that spaced procedures are beneficial compared to massed practice [51,52].”

(added) References:

47. Cepeda NJ, Vul E, Rohrer D, Wixted JT, Pashler H. Spacing effects in learning: A temporal ridgeline of optimal retention: Research article. Psychol Sci. 2008;19: 1095–1102. doi:10.1111/j.1467-9280.2008.02209.x

48. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychol Bull. 2006;132: 354–380. doi:10.1037/0033-2909.132.3.354

49. Logan JM, Castel AD, Haber S, Viehman EJ. Metacognition and the spacing effect: The role of repetition, feedback, and instruction on judgments of learning for massed and spaced rehearsal. Metacognition Learn. 2012;7: 175–195. doi:10.1007/s11409-012-9090-3

50. Versteeg M, Hendriks RA, Thomas A, Ommering BWC, Steendijk P. Conceptualising spaced learning in health professions education: A scoping review. Med Educ. 2020;54: 205–216. doi:10.1111/medu.14025

51. Morin CE, Hostetter JM, Jeudy J, Kim WG, McCabe JA, Merrow AC, et al. Spaced radiology: encouraging durable memory using spaced testing in pediatric radiology. Pediatr Radiol. 2019;49: 990–999. doi:10.1007/s00247-019-04415-3

52. Rozenshtein A, Pearson GDN, Yan SX, Liu AZ, Toy D. Effect of massed versus interleaved teaching method on performance of students in radiology. J Am Coll Radiol. 2016;13: 979–984. doi:10.1016/j.jacr.2016.03.031

b) Moreover, despite the presented data, there is little to no discussion of the impact on the actual learning outcomes of the students who participated in the study. If one is to assess massed practice, one should also discuss and comment on alternative approaches such as distributed practice.

Thank you for pointing out the necessity of discussing the impact of massed practice on learning outcomes more thoroughly and discussing further alternative learning approaches other than deliberate practice. We also think these are important aspects, which is why we expanded the discussion section respectively as we explained in the previous comment. Nevertheless, it is important to note that despite the fact that there may be more effective approaches to teaching image interpretation skills from a psychology perspective, massed practice is the favored approach in medical education practice. Thus, our study serves to make a statement of what can be achieved using this approach, while at the same time discussing possible alternatives that seem recommendable from a psychology perspective. Hence, the purpose of the paper was not to identify the most optimal training approach, but rather to study the impact of current practice.

Unless the authors can, somehow, address these key aspects,

Unfortunately, this statement is incomplete.

Reviewer #2:

Interesting article with new information. Here are a few suggestions/recommendations

Thank you for this overall positive assessment of our manuscript and the constructive feedback.

- Line 110: I'd suggest using "hyoid bone" instead of only "hyoid)

Thank you. We changed the term.

- Line 131: You mention a few studies (# 15-16 and 20 to 26) but do not provide explanation as to what these studies brought. If pertinent, maybe you could expand on at least a few of these

Thank you for this comment. We agree that it may be useful for readers to get some more information about the studies that investigated diagnosing tasks for radiographs other than OPTs.

Line 143

“For example, in a study by Donovan, Manning and Crawford [27] chest radiographs were presented to participants with the task to detect lung nodules only. In a mammography screening-diagnostic task, Nodine and colleagues [26] asked experts with different levels of experience to detect malignant lesions; Mammography radiographs typically contain only a modest number of lesions, different from OPTs that can entail a large number of very different anomalies. Consequently, it may be particularly challenging to develop visual expertise regarding OPTs for a number of reasons. A high interindividual variability of the visual appearance of both normal and abnormal anatomy as well as the phenomena caused by artifacts and superimpositions makes it difficult to detect anomalies [28-31]. In addition, even panoramic radiographs of apparently “dentally healthy patients” can often show one or more dental or non-dental anomalies [32-36].

- Line 132: "oral sinus"... are you talking about the maxillary sinus?

Indeed, what we were referring to is the oral cavity. We changed the term accordingly (Line 161).

- Line 135: "for why". I think the "for" is not needed here

Changed.

- Line 146: you talk about "increases", plural. But I'm not sure to what multiple things you are referring to

It is an increase in diagnostic performance, which is singular as you noticed correctly. We changed it accordingly. Thank you.

- Participant section: I'd like to know what year were these students in?

Students in their first clinical semester were in the second half of their third year. We added this information in line 199.

- Methods section: you mention the 1st test, 2nd test, and then a 3rd test 13 weeks after the 1st. When was the 2nd test administered?

The second test was administered directly after massed practice (see line 200) and therefore depended on the date when students finished their massed practice training. They were trained in small groups consisting of 2 to 4 students, which is why massed practice training was distributed across the semester in order to train all 69 students.

- Lines 185 and227/228: I don't think it's relevant to indicate your experts as the authors. Simply explaining their background should be enough.

Thank you, we changed it.

- Figure 2: I'm not sure I see what's wrong with the TMJs in this case? It might be worth explaning in you caption (lines 204-206) what the experts saw on this image.

Thank you. We added a summary of findings in the OPT shown in Figure 2 in the respective caption.

Caption Figure 2:

The left panel shows AOIs located around anomalies on an OPT. In addition to a bilateral shortening of the collum mandibulae, the colla and condyles present hypoplastic on both sides. Shortened roots of tooth 16, lacking the apical tips with periapical translucencies, possibly corresponding to a status post root resection.

In region 26, 27 and 28 spheroid, sharply defined homogeneous opacification (projecting on the maxillary sinus floor) corresponding to a mucosal antral pseudocyst.

Translucencies in the approximal areas of teeth 46 and 47 possibly indicating caries.

(Dental notation according to the FDI-system). Gridded AOIs, as displayed in the right panel were used for the computation of the overall gaze coverage.

Reviewer #3:

The article is simple but the idea is satisfactory.

About English language: The manuscript needs an English review in order to sound as native as possible, preferably changing some words that can be understood but are not routine words in academic writing. For example, in the first phrase of the introduction, the words: "Taking radiographs" sound weird. Also, in academic writing, passive voice is preferable, and the article is almost entirely wrote in active voice.

Also, punctuation needs review.

Thank you for your suggestions. A native speaker proofread the manuscript to improve the language and punctuation. Regarding the use of active voice, we adhered to the guidelines of the American Psychological Association that recommend using “…the active rather than the passive voice” (APA Publication Manual, Sixth Edition, Section 3.18, p.77).

# Introduction:

Please, provide a reference to the phrase: "The reason for the lack of feedback often lies in the fact that medical teachers do not have sufficient time and resources to review their students’ diagnostic competence for such a high number of radiographs." - I don't think the introduction is the right place to author's opinion.

Thank you for this comment. We agree that we should be careful with such statements in the introduction. However, with massed practice we aim at investigating an educational setting that did not get much attention from researchers over the past decades. Therefore, already in the introduction we want to explain why this particular setting is used in dentistry education. In the described scenario it is not realistic to instruct four students, let them evaluate and consecutively give feedback on 400 single panoramic radiographs within a 24 hours curriculum. This is therefore not our opinion but rather a description of the current situation in dentistry education.

I suggest you to explain further the meaning of "massed practice" with proper references. Explain how the practice is performed in a summarized and systematic explanation. Maybe you can use a figure or a flow chart to illustrate and clearer this point.

Thank you. We agree that it is eminently important to explain the meaning of “massed practice” unequivocally. Therefore, we added an explanation (see below). We also believe that Nodine and Mello-Thoms and Sowden have done outstanding work in this context and to the best of our knowledge are the only ones to address this much used learning strategy in radiology. Apart from that, the massed practice procedure was only investigated for very different tasks in general educational psychology (Radosevich & Donovan, 1999; Kornell & Bjork, 2008). Therefore, we were unable to identify any further, even more appropriate references.

Line 62

A frequently used and rather traditional instructional method for teaching students how to read and interpret radiographs is massed practice. Here students are required to provide a full written description of their observations including the identified anomalies for each radiograph. This procedure is repeated for a high number of radiographs, which are selected to reflect the full range of potential anomalies that students could be exposed to. No other learning activities are interspersed and only limited corrective feedback, if any, is provided.

In the phrase: "According to Nodine and Mello-Thoms [6]“massed practice is the main change agent in achieving expertise” (p. 868)," please, check if the reference citation is correct.

Thank you for pointing out this issue. We carefully checked the Vancouver style, which is the basis for the PLOS ONE style and in addition we checked published manuscripts in PLOS ONE for the correct way to indicate a direct quote in the text. However, we found very different implementations ranging from citing just the source without page number after the quote to different ways of indicating source and page number before or after the quote. Since we think it is important to make clear where exactly the quote can be found in the source, we used a style, that is actually used in published PLOS One manuscripts and is as well suggested by the Mendeley PLOS ONE style sheet.

Line 90

According to Nodine and Mello-Thoms “massed practice is the main change agent in achieving expertise” [6] (p. 868), with…

Please, add references to the phrases: "As a consequence, it may be

particularly challenging to develop visual expertise regarding OPTs for a number of

reasons. A high interindividual variability of the visual appearance of both normal and

abnormal anatomy makes it difficult to detect anomalies. In addition, even OPTs of

apparently healthy patients can often show several anomalies"

Thank you. We added respective references:

Line 149

Consequently, it may be particularly challenging to develop visual expertise regarding OPTs for a number of reasons. A high interindividual variability of the visual appearance of both normal and abnormal anatomy as well as the phenomena caused by artifacts and superimpositions makes it difficult to detect anomalies [28-31]. In addition, even panoramic radiographs of apparently “dentally healthy patients” can often show one or more dental or non-dental anomalies [32-36].

(added) References:

28. Akkaya N, Kansu Ö, Kansu H, Çağirankaya LB, Arslan U. Comparing the accuracy of panoramic and intraoral radiography in the diagnosis of proximal caries. Dentomaxillofacial Radiol. 2006;35: 170–174. doi: 10.1259/dmfr/26750940

29. Molander B. Panoramic radiography in dental diagnostics. Swedish Dent J Suppl. 1996;119: 1–26.

29. Nardi C, Calistri L, Grazzini G, Desideri I, Lorini C, Occhipinti M, et al. Is panoramic radiography an accurate imaging technique for the detection of endodontically treated asymptomatic apical periodontitis? J Endod. 2018;44: 1500–1508. doi: 10.1016/j.joen.2018.07.003

31. Perschbacher S. Interpretation of panoramic radiographs. Aust Dent J. 2012;57: 40–45. doi:10.1111/j.1834-7819.2011.01655.x

32. Laganà G, Venza N, Borzabadi-Farahani A, Fabi F, Danesi C, Cozza P. Dental anomalies: Prevalence and associations between them in a large sample of non-orthodontic subjects, a cross-sectional study. BMC Oral Health. 2017;17: 1–7. doi:10.1186/s12903-017-0352-y

33. Hernándes G, Plaza SP, Cifuentes D, Villalobos LM, Ruiz LM. Incidental findings in pre‐orthodontic treatment radiographs. Int Dent J. 2018;68: 320–326. doi: 10.1111/idj.12389

34. Schroder AGD, de Araujo CM, Guariza-Filho O, Flores-Mir C, de Luca Canto G, Porporatti AL. Diagnostic accuracy of panoramic radiography in the detection of calcified carotid artery atheroma: a meta-analysis. Clin Oral Investig. 2019;23: 2021–2040. doi:10.1007/s00784-019-02880-6

35. Macdonald D, Yu W. Incidental findings in a consecutive series of digital panoramic radiographs. Imaging Sci Dent. 2020;50: 53–64. doi:10.5624/ISD.2020.50.1.53

36. Monteiro IA, Ibrahim C, Albuquerque R, Donaldson N, Salazar F, Monteiro L. Assessment of carotid calcifications on digital panoramic radiographs: Retrospective analysis and review of the literature. J Stomatol Oral Maxillofac Surg. 2018;119: 102–106. doi:10.1016/j.jormas.2017.11.009

# Materials and Methods

Participants: you mentioned 55 students in the abstract, however, in the first paragraph of M&M you mentioned, after exclusion criteria applied, 47 students as your final sample. Please, correct it or clarify the numbers divergence.

Thank you. We corrected the number of the participants in the abstract.

#Discussion

Again, maybe it would be better to express your opinion in the discussion, for exemple. Remove or re-wright theses paragraphs.

Sorry, we do not know which paragraphs you were referring to. This information is missing in the comments we received.

Introduction, last two paragraphs: I don't think Introduction is the right place to express your opinion and what you expect from the study. You can express it in the discussion.

In the last two paragraphs of the introduction we give an overview of the state of research and highlight the research gap. In the last sentence, we give a summary on what our study is about. We do not think that we expressed our opinion in the mentioned paragraphs (except the reason for missing or limited corrective feedback – see our answer above). Therefore, we do not see where we could change the introduction any further without leaving out relevant research and the argumentation regarding the relevance of our present study.

I suggest you to improve your discussion with other references and comparisons with different studies, such as, for example:

* Performance evaluation of different observers in the interpretation of panoramic radiographs by the mandibular cortical index. DOI: 10.15448/1980-6523.2018.1.29202

*Observer performance in diagnosing osteoporosis by dental panoramic radiographs: results from the osteoporosis screening project in dentistry: https://doi.org/10.1016/j.bone.2008.03.014

Thank you for this suggestion. We added the studies to the discussion section to highlight why our study complements existing research, since the two suggested studies are generally interested in expert-novice differences in diagnosing part of a panoramic radiograph rather than evaluating a common educational procedure in dentistry education referring to students’ skill develoment.

Line 487

To summarize, the present study complements existing research on medical image interpretation [12-14,53-55] in that it focuses on the effects of training of students rather than on expert-novice differences and it uses OPTs rather than, for instance, chest radiographs, which require different types of visual search processes.

(added) references:

54. Taguchi A, Asano A, Ohtsuka M, Nakamoto T, Suei Y, Tsuda M, et al. Observer performance in diagnosing osteoporosis by dental panoramic radiographs: Results from the osteoporosis screening project in dentistry (OSPD). Bone. 2008;43: 209–213. doi:10.1016/j.bone.2008.03.014

55. Munhoz L, Kim JH, Park M, Aoki EM, Abdala R, Arita ES. Performance evaluation of different observers in the interpretation of panoramic radiographs by the mandibular cortical index. Rev Odonto Cienc. 2019;33: 6–10. doi:10.15448/1980-6523.2018.1.29202

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

27 Oct 2020

PONE-D-20-05443R1

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

PLOS ONE

Dear Dr. Richter,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the second review process.

Please submit your revised manuscript by Dec 11 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ezio Lanza, M.D.

Academic Editor

PLOS ONE

Comments to the Author

Reviewer #1: Overall, the authors are to be commended for the efforts made to address the comments made by the reviewers.

Regarding my comment on the inexistence of a control group, the authors argued that “it is unclear what the control group would be”. As an example, I might mention a study conducted by Andersen et al, which managed to compare the impact of massed and distributed practice in the learning curves of virtual mastoidectomy (Andersen SA et al. JAMA Otolaryngol Head Neck Surg. 2015; 141: 913-8).

Hence, as much as I understand that it may not be suitable to change the study design at this stage, it is, nonetheless, important that the authors acknowledge the weaknesses of the current approach.

Furthermore, discussion-wise I suggest that the authors discuss the contextual interference hypothesis, which proposes that “when learning multiple skills, massing practice leads to better within-day acquisition, whereas random practice leads to better retention and transfer” (Savion-Lemieux T, Penhune VB.Exp Brain Res. 2010; 204: 271-81). In other words, the authors should discuss the potential variations of massed practice’s outcomes, depending on the task/learning objectives.

-------------------------------------------------------------------------------------

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Nov 2020

Dear Dr. Lanza, dear Reviewer,

we would like to thank the editor and the reviewer for the assessment of our manuscript and the feedback offered by the review. In the following, we will address the comments one by one. Reviewer comments are printed in regular font, responses in italics. Moreover, we have added sections from the manuscript whenever we made a change in it (in italics and quotation marks). We hope that this way we have addressed all issues raised by the reviewer in a satisfactory manner.

The authors

Comments from the Editor:

Dear Dr. Richter,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the second review process.

Please submit your revised manuscript by Dec 11 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ezio Lanza, M.D.

Academic Editor

PLOS ONE

5. Review Comments to the Author

Reviewer #1:

Reviewer #1: Overall, the authors are to be commended for the efforts made to address the comments made by the reviewers.

Thank you for this overall positive assessment of our revised manuscript and your constructive feedback.

Regarding my comment on the inexistence of a control group, the authors argued that “it is unclear what the control group would be”. As an example, I might mention a study conducted by Andersen et al, which managed to compare the impact of massed and distributed practice in the learning curves of virtual mastoidectomy (Andersen SA et al. JAMA Otolaryngol Head Neck Surg. 2015; 141: 913-8).

Hence, as much as I understand that it may not be suitable to change the study design at this stage, it is, nonetheless, important that the authors acknowledge the weaknesses of the current approach.

Thank you very much for this comment. We agree that it is necessary to acknowledge the weakness of our approach regarding the missing control group. We therefore addressed this aspect in the discussion section:

l.460

Also in radiology teaching and surgical skill training, studies have shown that spaced procedures are beneficial compared to massed practice [51,52]. Future research should therefore specifically compare massed and spaced practice for learning how to diagnose OPTs to identify potentials for further improvements of students’ accuracy. The present study did not aim at contrasting different training approaches because our focus was on the effects that current practice has on students’ skill development. Therefore, we implemented the present study in a within-subjects design within students’ regular training to achieve a high ecological validity. We acknowledge that this approach has its limitations due to a lack of a control group that would have been trained with a different approach.

(added) References:

52. Andersen SAW, Konge L, Cayé-Thomasen P, Sørensen MS. Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol - Head Neck Surg. 2015;141: 913–918. doi:10.1001/jamaoto.2015.1563

Furthermore, discussion-wise I suggest that the authors discuss the contextual interference hypothesis, which proposes that “when learning multiple skills, massing practice leads to better within-day acquisition, whereas random practice leads to better retention and transfer” (Savion-Lemieux T, Penhune VB.Exp Brain Res. 2010; 204: 271-81). In other words, the authors should discuss the potential variations of massed practice’s outcomes, depending on the task/learning objectives.

Thank you for this suggestion. However, it is important to note that spaced vs. massed training is different from what has been proposed in research on contextual interference and interleaved practice, respectively. Spaced learning refers to the recommendation that when training a single skill, there should be pauses between multiple training sessions rather than training the same skill in a massed way. Thus, spaced vs. massed practice solely refers to the timing of the training. Contextual interference / interleaved practice refers to the training of multiple skills or skill components. Here it is recommended that these different skills are not to be trained one by one in a blocked fashion; rather, their training should be interleaved (abcabc etc.). It is true that in the case of interleaved practice this will mean that training of a single skill will be spaced at the same time. Nevertheless, the mechanism for the contextual interference effect are fundamentally different from that underlying spaced learning. In particular, interleaved practice is beneficial because it allows contrasting different training experiences, thereby highlighting that different contexts require application of different skills. The beneficial effects of interleaved practice only occur in delayed tests, whereas during training interleaved practice typically yields worse training performance. The confound between interleaved practice and spaced learning has been discussed and investigated, for instance, by Taylor and Rohrer (2009), who showed that interleaved practice has an effect above and beyond spacing. Because our study focused on spacing rather than interleaved practice, we have decided to not elaborate the concept of contextual interference in order to not cause any confusion of these two instructional principles. Rather, to highlight the focus of our paper on spaced learning, we added a footnote as follows:

l.470

Importantly, the current study refers only to the effects of massed practice regarding training of a single skill, where it has been suggested that training of that single skill should occur spaced in time. This recommendation is not to be confused with another instructional design principle, for which multiple labels have been used in the literature, that is, the contextual inference effect [53-55], the variability effect [56,57] or interleaved practice [58], respectively. Here it is suggested that when training multiple skills, these should be trained in an interleaved way (abcabcabc etc.) to highlight the differences between them. While interleaved practice of multiple skills results in spaced learning of the same skills, it is different from the focus of the present study.

(added) References:

553. Battig WF. The flexibility of human memory. In: Lermak LS, Craik FIM, editors. Levels of processing in human memory. Hillsdale, NJ: Erlbaum; 1979. pp. 23–44.

54. Shea JB, Morgan RL. Contextual interference effects on acquisition and transfer of a complex motor task. J Exp Psychol Hum PLearning Mem. 1979;5: 179–187.

55. Brady F. A theoretical and empirical review of the contextual interference effect and the learning of motor skills. Quest. 1998;50: 266–293. doi:10.1080/00336297.1998.10484285

56. Paas FGWC, Van Merriënboer JJG. Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. J Educ Psychol. 1994;86: 122–133. doi:10.1037//0022-0663.86.1.122

57. Likourezos V, Kalyuga S, Sweller J. The Variability Effect: When Instructional Variability Is Advantageous. Educ Psychol Rev. 2019;31: 479–497. doi:10.1007/s10648-019-09462-8

58. Taylor K, Rohrer D. The effects of interleaved practice. Appl Cogn Psychol. 2009;24: 837– 848. doi:https://doi.org/10.1002/acp.1598

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

16 Nov 2020

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

PONE-D-20-05443R2

Dear Dr. Richter,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ezio Lanza, M.D.

Academic Editor

PLOS ONE

20 Nov 2020

PONE-D-20-05443R2

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

Dear Dr. Richter:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Ezio Lanza

Academic Editor

PLOS ONE