Inter-observer reliability in cone-beam computed tomography assessment of the retromolar canal: A practical plan to improve diagnostic imaging.

Purpose: This study aimed to investigate inter-observer reliability among observers with different levels of proficiency and the diagnostic imaging reliability of cone-beam computed tomography (CBCT) images of the retromolar canal. Materials and
Methods: CBCT images of 307 patients were assessed for the presence of retromolar canals (RMCs) by 3 observers independently. Diagnoses were made twice by each observer at intervals of more than 3 weeks. Inter-observer reliability was assessed using the kappa coefficient. One observer had no experience in diagnosis using CBCT images. Therefore, a specialist in diagnostic imaging explained the CBCT images for interpretation and practiced diagnostic imaging together with this observer, while the other observer interpreted the images independently. Thereafter, the observers re-evaluated the images.
Results: The interobserver kappa coefficients (including bilateral RMCs) calculated at the first reading were low, ranging from 0.21 to 0.61. Their values ranged from 0.95 (right side) to 1.00 (left side) after one-on-one practice with a diagnostic imaging specialist, while the values ranged from 0.65 (right side) to 0.66 (left side) without one-on-one practice.
Conclusion: Diagnostic accuracy was improved through diagnostic imaging practice. To improve the anatomical interpretation of images, it is important to practice diagnostic imaging with a specialist in diagnostic imaging. One-on-one instruction about diagnostic imaging was an effective method of training.

Introduction

Cone-beam computed tomography (CBCT) devices are increasingly commonly available. CBCT can depict detailed anatomical structures with high resolution. General practitioners who are not experts in diagnostic imaging have more opportunities to perform diagnostic imaging with CBCT in countries where there is no national professional qualification for dental maxillofacial radiologists. In this context, it is noteworthy that the inter-observer reliability between professional radiologists and dentists unfamiliar with diagnostic CBCT has not been studied to date.

It is important to understand the normal anatomical structures on CBCT images before performing diagnostic imaging. For instance, being able to detect a retromolar canal (RMC) is useful for avoiding unnecessary complications during surgical procedures involving the molars, such as extraction, osteotomy, and bone harvesting of mandibular third molars.

Von Arx et al.1 described the RMC as an anatomical structure that was generally neglected in anatomy textbooks and stated that the RMC was very rarely visualized on conventional radiographs. However, the RMC was deemed to be of clinical interest because it might convey additional or abnormal neurovascular structures to and from the posterior mandible and cheek. Löfgren2 was the first to describe a small foramen in the fossa retromolaris using the term “foramina retromolaria” and called the canal in the mandible that opened at this foramen the RMC. According to Kodera et al.,3 who reported the frequency of RMCs in analyses of dry skulls in Japan, arteries and nerve bundles might be present in a wide RMC that could be visualized on X-ray images.

Some authors have classified the RMC as 1 of 4 different types of bifid mandibular canals in the posterior body of the mandible.45 Nortje et al.4 classified the RMC into 4 categories on panoramic X-rays and included branches from the mandibular canal in this classification. According to their classification, the RMC is a supplemental canal arising from the mandibular canal and reaching the retromolar pad region. Previous reports contained considerable in the classification of the RMC. There have also been many studies on the frequency of the RMC, branching of the mandibular canal, and bifid mandibular canal based on CBCT.34567891011

CBCT has facilitated the radiographic identification of the presence and course of the RMC. However, many studies have not described the inter-observer reliability of detection of the retromolar foramen (RMF) and RMC on CBCT images.456711 In contrast, Nikkerdar et al.8 and Kikuta et al.9 found that the interobserver agreement, calculated using the kappa coefficient, was very high. In previous reports with a high rate of agreement, the observers were both clinically experienced and/or were oral surgeons who could visualize anatomical structures during surgery or professional dental maxillofacial radiologists.

Therefore, this study aimed to examine the inter-observer reliability among observers with different levels of proficiency and knowledge to assess the reliability of interpretation of CBCT images. The authors focused on identifying the presence or absence of the RMC in CBCT images as a normal anatomical structure rather than in a pathological diagnosis to assess the effectiveness of training among professional dental maxillofacial radiologists as recommended by the European Academy of Dental and Maxillofacial Radiology (EADMFR) guidelines.12 Additionally, this study aimed to evaluate the prevalence of RMCs in order to provide greater confidence in the accuracy of imaging diagnoses.

Materials and Methods

This retrospective study included 307 patients (131 men and 176 women) with a mean age of 27 years (range, 14-71 years) who had undergone CBCT bilaterally for dental implant surgery, impacted third molar surgery, or orthodontic treatment. All patients provided informed consent to participate in the study and written consent for the use of their images. The study was approved by the ethical review board of the authors’ affiliated university (no. 1601). The study was performed according to the Declaration of Helsinki.

CBCT was performed using an Asahi Alphard-VEGA scanner (Asahi Roentgen Ind. Co., Kyoto, Japan), with the following technical parameters: 80 kV, 5-8 mA, and an exposure time of 17 seconds. The field of view was set to be Φ 102×102 mm, and the voxel size was 0.2 mm. The axial images obtained from CBCT were transmitted in the Digital Imaging and Communications in Medicine format, and 2-dimensional images of the mandible were then reconstructed using the AZE Virtual Place Fujin (CANON Medical Systems Asia Pte. Ltd. Tochigi, Japan). Sagittal section (parallel to the dental arch) and cross-sectional (perpendicular to the dental arch) images were reconstructed from the projection data by radiological technologists.

The CBCT images of the 307 patients were assessed twice for the presence of RMC by 3 observers independently. All CBCT images were evaluated on the same computer with an Intel Core i5-4590, 3.30 GHz; 32″ NEC ME-K workstation with a resolution of 1280×1024 pixels, 4 GB memory, and the Windows 7 operating system. As the resolution of diagnostic monitors used by dentists in general practice was different from that of the monitors used in the Department of Radiology, Tsurumi University Dental Hospital, the same monitors used in the authors’ affiliated hospital were used for evaluation in this study.

The 3 observers were as follows. Observer A was an oral radiologist who had performed diagnostic imaging every day, had more than 25 years of clinical experience, and was a professional member of the Japanese Society for Oral and Maxillofacial Radiology. Observer B was a dentist and visited the authors’ university to study abroad, who performed diagnostic imaging on CBCT here for the first time. Observer C was an oral surgeon with 10 years of clinical experience who had not performed diagnostic imaging every day. All observers evaluated all images twice independently for the presence or absence of the RMC with an interval of more than 3 weeks. To determine the presence of the RMF and RMC, they first identified it on a sagittal section and then reconfirmed it on a cross-sectional image perpendicular to the canal on the image. As observer B was unfamiliar with diagnostic imaging, the RMC was defined as a branch arising from the main canal and opening at the RMF in the mandibular molar region and the ramus.

At first, observer A instructed observer B on how to use electronic medical records and adjust the contrast and magnification of the images and described a typical image of the RMC (Fig. 1). Observers A and B jointly diagnosed and practiced the interpretation of several images used in the study.

Fig. 1

A typical image of the posterior canal on cone-beam computed tomography. The retromolar canal is the branch arising from the main canal, opening at the retromolar foramen (white arrows).

However, discrepancies occurred between the observers in some diagnostic images. Thus, observer B performed diagnostic image interpretation independently, after which observer A explained the interpretation of the CBCT images and practiced image interpretation together with observer B, with calibration of the diagnostic criteria. Observer C double-checked the diagnostic imaging of daily clinical cases and practiced the diagnosis alone rather than with observer A.

Statistical analysis was performed using Ekuseru-Toukei 2015 (Social Survey Research Information Co., Ltd. Tokyo, Japan). The reliability of the data was analyzed using the kappa test. A P value <0.05 was considered to indicate statistical significance. The kappa coefficient13 was interpreted as indicating poor (0), slight (0.01-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), and almost perfect (0.81-1.0) agreement. This study considered the findings of observer A as the gold standard and evaluated the degree of agreement between observers A and B and between observers A and C.

Results

The 3 observers evaluated the presence of the RMC bilaterally on CBCT images of 307 subjects. The intra-observer kappa coefficient for observer A was 0.94 on the right side and 0.96 on the left side, indicating almost perfect agreement. For observer B, the coefficient was 0.40 on the right side and 0.72 on the left side, while that for observer C was 0.51 and 0.59 which indicated moderate and substantial intra-observer agreement, respectively (Table 1). Examples of an image about which the observers did not agree are shown in Figures 2 and 3.

Table 1

Intra-observer kappa coefficients

Fig. 2

Image of a case with no agreement between observers. A branch (white arrow) from the mandibular canal is seen, but it is not continuous with the retromolar foramen.

Fig. 3

The mylohyoid muscle nerve groove (white arrowhead) running along the lingual aspect of the mandible. The white arrow indicates the mandibular canal.

The kappa coefficients for inter-observer readings were low. The inter-observer kappa coefficients for observers A and B were 0.61 on the right side and 0.65 on the left side, indicating substantial agreement, and those for observers A and C were 0.21 on the right side and 0.26 on the left side, indicating fair agreement (Table 2). Observer B then practiced image interpretation one-on-one with observer A and then re-evaluated the images independently within 1 week. Observer C practiced diagnostic imaging on CBCT images by himself on different cases from those included in this study, and then re-evaluated the CBCT images in this study.

Table 2

Inter-observer kappa coefficients

The inter-observer kappa coefficients for observers A and B for the second round were 0.95 on the right side and 1.00 on the left side, indicating almost perfect agreement, while those for observers A and C were 0.65 on the right side and 0.66 on the left side, indicating substantial agreement (Table 2).

The frequency of RMCs, as determined from the assessment of the inter-observer reliability of observers A and B, was high (with almost perfect agreement as shown by the kappa coefficient). An RMC was observed in 48 of 307 right sides (15.6%) and 29 of 307 left sides (9.5%) (Fig. 4). Among the patients, 13 had bilateral RMCs. The frequency of RMCs was significantly higher on the right side than on the left side, as assessed using the chi-square test (P<0.05).

Fig. 4

Frequency of retromolar canals in 614 mandible sides. Thirteen patients had bilateral retromolar canals. There was a significant difference in the frequency of left and right retromolar canals using the chi-square test (P<0.05).

Discussion

This study investigated the inter-observer reliability of CBCT image interpretation for the presence of an RMC by readers with different levels and fields of expertise. The inter-observer kappa coefficient calculated for the first set of readings was 0.21-0.61 while the coefficient for the second set of readings was 0.65-1.00. Thus, diagnostic accuracy was improved through diagnostic imaging practice. To improve image anatomy interpretation, it is important to practice diagnostic imaging with a specialist in diagnostic imaging.

The EADMFR guidelines specify the following.12 CBCT images must undergo a thorough clinical evaluation (‘‘radiological report’’) of the entire image dataset. All those involved with CBCT must have received adequate theoretical and practical training for the purpose of radiological practices and relevant competence in radiation protection.

Continuing education and training after qualification are required, particularly when new CBCT equipment or techniques are adopted. Dentists responsible for CBCT facilities who have not previously received ‘‘adequate theoretical and practical training’’ should undergo a period of additional theoretical and practical training that has been validated by an academic institution (university or equivalent). Where national specialist qualifications in dental maxillofacial radiology exist, the design and delivery of CBCT training programs should involve a dental maxillofacial radiologist. For dental and maxillofacial CBCT images, the clinical evaluation (‘‘radiological report’’) should be made by an adequately trained general dental practitioner or by a specially trained dental maxillofacial radiologist.

As indicated above, when using CBCT in medical practice, it is important to have not only technical knowledge of the device, but also training in the interpretation of CBCT images. It is recommended that dentists using CBCT take part in a practical training course provided by an academic institution (university or equivalent), but in Japan, the course is often only 1 day long; thus, it is difficult to provide comprehensive training. In recent years, undergraduate students at dental schools have received lectures on the principles of CBCT and diagnostic imaging. However, they cannot completely understand the clinical practice and imaging diagnosis based on CBCT images.

The reason for the low inter-observer reliability of the initial diagnostic image interpretation in this study is thought to be that 1 observer was initially unfamiliar with diagnostic imaging on CBCT and because, in contrast to previous reports, the cross-sectional orientation of the CBCT image could not be adjusted freely by the observers. Previous studies were conducted using imaging software that allowed observers to adjust the cross-sectional orientation according to their needs.891011 In this study, the authors evaluated the reconstructed images created by the radiologists on the same image monitor used in other departments in the hospital, which included some cases in which it was difficult to evaluate the RMC. Therefore, practice in constructing a reconstructed image by oneself is also important for improving the reliability of diagnostic CBCT image interpretation.

There are few detailed reports on how to improve the accuracy of diagnostic imaging, and although the primary target of such studies is now magnetic resonance imaging, improvements in inter-observer variability may be achieved by using readers who have previously worked and trained together, as well as by allowing observers to become familiar with the diagnostic imaging modality.14 The findings of the present study suggest that dentists who are inexperienced or unfamiliar with this form of diagnostic imaging should perform diagnostic image interpretation alongside an experienced specialist and that it is necessary for dentists to become familiar with diagnostic imaging.

In the present study, observer B, who was instructed one-on-one, evaluated and discussed the same images within a short-term period and agreed with the experienced reader regarding the presence or absence of the relevant diagnostic criteria. Thus, the reliability of the diagnostic imaging of CBCT images was significantly improved. In contrast, observer C, who practiced alone, improved the reliability of diagnostic imaging, but the degree of improvement was lower than that for observer B. It is well-known that the length of experience increases the level of expertise in medical practice and diagnostic imaging, but it is also possible to increase one’s level of expertise by working with a specialist for a short period of time and receiving one-on-one instruction.

This study involved the diagnosis of anatomical structures. Because imaging diagnosis of diseases varies from case to case, it is important to be trained to perform diagnostic imaging together with an experienced specialist. The authors speculate that one-on-one instruction was a particularly effective method of training in diagnostic imaging. At first, a participant who is not familiar with diagnostic image interpretation performs the interpretation alone and is then instructed by the specialist. To diagnose the branching of the mandibular canal, including the RMC, the authors believe that the best method to approach the diagnosis is to examine the sagittal section image first and then the perpendicular section of that image.

Ossenberg15 analyzed the RMF in a large series of mandibles (N=2500) and suggested that the prevalence of the RMF in native North Americans was higher than that in people from other regions such as Africa, Europe, India, and Northeast Asia. Schejtman et al.16 reported that the different paths taken by the mandibular RMC could be considered useful to typify ethnological material. A relationship was proposed between different racial categories and the frequency of the RMF.34567891011141516171819 With consideration of racial differences, the prevalence of RMCs in this study was 12.5%, which is similar to the prevalence previously reported in the Japanese literature,357917 and the outcomes of observer B’s diagnostic imaging practice was reflected in these results.

There was a difference in the frequency of RMCs between the right and left sides, with RMCs more frequently observed on the right side. However, previous studies in Japan and Taiwan reported no differences between the right and left sides.6919 Conversely, a study from Turkey reported that the frequency of RMCs in the left mandible was higher than that in the right mandible,20 whereas Ossenberg15 reported RMCs to be more common on the right side than on the left side. Thus, evidence on the occurrence of RMCs relative to each side is inconclusive. However, the reliability of the intra- and inter-observer diagnostic imaging was higher on the left side, which could have been influenced by the lower occurrence of RMC on the left side. In other words, the results suggest that even if one is not familiar with diagnostic imaging, it is possible to make a correct diagnosis in the absence of anatomical structures.

In conclusion, in addition to describing the frequency of RMCs in this population, the authors demonstrated that in order to interpret anatomy on CBCT images correctly, it is important to practice diagnostic imaging with a specialist, which should include receiving instruction and discussions. Our study indicates that one-on-one instruction about diagnostic imaging was an effective method of training.