A novel reference model for dental scanning system evaluation: analysis of five intraoral scanners.

PURPOSE: The aim of this in vitro study was to investigate the accuracy (trueness and precision) of five intraoral scanners (IOS) using a novel reference model for standardized performance evaluation.
MATERIALS AND METHODS: Five IOSs (Medit i500, Omnicam, Primescan, Trios 3, Trios 4) were used to digitize the reference model, which represented a simplified full-arch situation with four abutment teeth. Each IOS was used five times by an experienced operator, resulting in 25 STL (Standard Tessellation Language) files. STL data were imported into 3D software (Final Surface®) and examined for inter- and intra-group analyses. Deviations in the parameter matching error were calculated. ANOVA F-test and Kruskal-Wallis test were applied for inter-group comparisons (α = .05); and the coefficient of variation (CV) was calculated for intra-group comparisons (in % ± SD).
RESULTS: Primescan (matching error value: 0.015), Trios 3 (0.016), and Trios 4 (0.018) revealed comparable results with significantly higher accuracy compared to Medit i500 (0.035) and Omnicam (0.028) (P < .001). For intra-group comparison, Trios 4 demonstrated the most homogenous results (CV 15.8%).
CONCLUSION: The novel reference model investigated in this study can be used to assess the performance of dental scanning technologies in the daily routine setting and in research settings.

INTRODUCTION

The trend for digitalization in dental medicine has led to advanced computer-aided design and computer-aided manufacturing technology (CAD-CAM), which is now widely used in the field of prosthodontics. New data acquisition devices, such as intraoral scanners (IOS), provide numerous benefits in daily clinical use, including selective repeatability and capture of relevant areas, chairside options, virtual follow-ups, and quick communication with dental technicians.12 For implant impressions, IOS technology has been preferred by dentists and patients over conventional impression technique.34 Compared to the analog protocol, the use of IOS simplifies the workflow and can reduce clinical treatment time, while increasing patient comfort.5678

The ability of an IOS system to capture high-quality impression can be determined by measuring its accuracy, comprising both trueness and precision.9 Trueness defines the comparison between a prototype STL (Standard Tessellation Language) dataset and a control STL dataset, whereas precision evaluates different datasets within the same intraoral scanner device.10 Therefore, the more precise an IOS is, the more predictable the measurements are; on the other hand, a high trueness describes a value, which is close or even equal to the reference model.11

Several studies have investigated the accuracy of different IOS systems using study-specific reference models.1213141516 Common approaches include the use of stone casts with three to six implant lab-analogues,17 or steel models,11 as the reference model. Recently published trials have worked with tooth-like morphologies with various preparation designs to analyze the accuracy of digital scanners.1314 However, as the technology used in digital IOS devices advances, it becomes more difficult to generate an adequate overview of scanner quality and to compare corresponding studies without a general standardized baseline. In addition, the lack of standardized calibrations for the routine laboratory and clinical procedures involved in the IOS and CAD-CAM workflow make it difficult to compare different IOS systems and technologies.

The aim of this in vitro investigation was to assess the trueness and precision of five IOS systems using a novel standardized reference model. The null hypothesis was that there is no difference in trueness and precision among the five IOS systems.

MATERIALS AND METHODS

This study used a novel reference model (design and production by Institut Straumann AG, Basel, Switzerland) that simulated a simplified full-arch situation with four abutment teeth with schematized crown preparations in the position of the canines and the first molars (Fig. 1). Each abutment tooth has a specific marking in the form of bevels at the tip of abutment (1. lower left side: 1 bevel; 2. upper left side: 2 bevels; 3. upper right side: 3 bevels; 4. lower right side: 4 bevels), which is intended to prevent data overlay by the IOS. In addition, four varying inlay preparation designs are located in the center of the test model. The specific design was calculated in a 2.5D model with the Autodesk Inventor® software (Autodesk; San Rafael, CA, USA) and exported as the “main STL-file” for all comparisons. The reference model was milled from polyether ether ketone (PEEK), a non-reflecting organic thermoplastic polymer,18 using a highly precise CNC (computerized numerical control) machine (Kern Micro SC; Kern Microtechnik, Eschenlohe, Germany).

Fig. 1

Reference model (design and production by Institut Straumann AG, Basel, Switzerland).

Five commercially available IOS systems were used: Medit i500 (MEDIT Corp., Seoul, South Korea), Omnicam (Dentsply Sirona, Charlotte, NC, USA), Primescan (Dentsply Sirona, Charlotte, NC, USA), Trios 3 (3Shape, Copenhagen, Denmark) and Trios 4 (3Shape, Copenhagen, Denmark). Scanner software was updated to the latest version for each IOS system at that given period (February 2021) (Table 1).

Table 1

Software versions of each IOS used in this trial (February 2021)

IOS	Software version
Medit i500	Medit Link 2.3.
Omnicam	Cerec SW Version 5.1.3.
Primescan	Cerec SW Version 5.1.3.
Trios 3	3Shape Dental System 20.2.0.
Trios 4	3Shape Dental System 20.2.0.

All scans were performed by one experienced operator (IKS) under the same day light conditions at room temperature (20℃). The scan protocol followed a standardized procedure. The reference model was scanned five times by each IOS after system-specific calibration. In between, breaks of five minutes were made to let the IOS cool down and to prevent operator fatigue. The scanning pathway started from the occlusal surface on the bottom right, following clockwise to the abutment teeth on the bottom left side, slowly moving back to buccal sites and finishing the scan from the oral surface (Fig. 2). The mentioned pathway was chosen for all IOS over the individual manufacturer’s recommendation to simplify and equate handling. According to a study by Mennito et al.15 in 2018, the scanning pattern has no effect over trueness and precision, except for Omnicam, where the shown pattern was superior to a snakelike movement, scanning occlusal surface first and rolling from buccal to lingual to buccal from posterior to anterior. All scans were checked for completeness.

Fig. 2

Scanning and recognition pathway: 1. occlusal surface; 2. vestibular surface; 3. oral surface.

Twenty-five STL-files (five scans from five IOS systems) were superimposed to the “main STL-file” of the novel reference model and were analyzed for accuracy. The STL-data sets were imported, evaluated, and superimposed on each other using the Final Surface® 3D-software (GFaI; Berlin, Germany). A best-fit algorithm of the 3D-analysis software was applied for accuracy testing of the superimposed STL-files.14 To visualize the distance protocol, a color mapping function of the 3D analysis software was applied (Fig. 3). Smaller artifacts around the edges of the virtual model situations, as identified polygons on the scans, were trimmed and cropped in the Meshmixer® software (Autodesk; Inc., San Rafael, CA, USA) before the data were imported into Final Surface® by one operator (IKS).

Fig. 3

Example of colored distances in range [-0.80 ; 0.00], [0.00 ; 0.80] of two superimposed digital scans. Yellow marked areas show almost no distance differences between the scans, whereas the green areas shows distances from 0.08 mm and above. The four abstract abutment teeth were also held as reference points for superposition.

Trueness analysis for inter-group comparisons of the five IOS systems was the primary outcome, while precision for intra-group comparisons of reproducibility within each IOS system (e.g.: Trios 3 STL-1 vs. Trios 3 STL-2; Trios 3 STL-1 vs. Trios 3 STL-3; etc.) was the secondary outcome. Matching error (software-specific calculations based on the best-fit algorithm) and mean distance were the parameters analyzed for the various STL-pairings: (1) STL IOS vs. raw STL-file; and (2) STL IOS vs. IOS’.

All statistical data were analyzed in software R (The R Project for Statistical Computing; Vienna, Austria) with a significance level set at α = .05. Variables of interest were analyzed with sample means for 80% quantiles including standard deviation. Too small or too big area errors were minimized for deviation analysis.14 ANOVA F-test and Kruskal-Wallis test were applied to analyze an overall inter-group comparison (trueness) among the five IOS systems. Multiple comparisons of means were analyzed with the Tukey contrasts (intra-comparison). Data were compared pairwise with adjusted P-values.

RESULTS

For matching error, the distribution was linear and analyzed with the parametric ANOVA f-test; while for mean distance, the values were logarithmically transformed to harmonize the skewed distribution, and examined with the non-parametric Kruskal-Wallis test. The coefficient of variation (CV) was calculated for the normally distributed parameter matching error (mean in %; standard error). For mean distance the interquartile range (IQR) calculation (IQR/median in %) was calculated.

The lowest matching error corresponding to highest trueness was found with Primescan (0.015), Trios 3 (0.016), and Trios 4 (0.018) (Table 2), while Omnicam and Medit i500 demonstrated significantly higher values (P = .028 and 0.037, respectively). Matching error for inter-group comparison was significant (P ≤ .001) in at least one group (reference model/IOS). Mean distance was not significantly different among the five IOS systems (P = .457).

Table 2

Inter-group comparisons: trueness for matching error and mean distance. Deviation of IOS STLs compared to the raw reference STL summarizing mean values of 80% quantiles including standard deviations

	Medit i500	Omnicam	Primescan	Trios 3	Trios 4	Stat. Test	P-value
Matching error (SD)	0.035 (0.011)	0.028 (0.006)	0.015 (0.005)	0.016 (0.005)	0.018 (0.003)	ANOVA-F-test	< .001
Mean distance (SD)	0.017 (0.014)	0.011 (0.012)	0.006 (0.006)	0.004 (0.002)	0.005 (0.003)	Kruskal-Wallis test	.457

IOS: intraoral scanner; SD: standard deviation; STL: Standard Tessellation Language.

Matching error is an unitless accuracy calculation: the smaller the value, the smaller the deviation from the reference model.

The intra-group comparison revealed a significantly different matching error when Primescan, Trios 3 and Trios 4 were compared to Medit i500 (P ≤ .001; P ≤ .001; P = .002, respectively), and between Primescan and Omnicam (P = .031), corresponding to precision. The relative standard deviation of the matching error was the lowest with Trios 4 (CV 15.8%), demonstrating the most homogenous results within the five scans (Table 3). For mean distance, Trios 3 showed the lowest relative standard deviation (rel. IQR 25%).

Table 3

Intra-group comparisons: coefficients of variation for normally distributed data matching error and interquartile range calculation for mean distance

IOS	Matching error CV (%)	Mean distance IQR (%)
Medit i500	30.0	75.0
Omnicam	20.3	40.0
Primescan	35.7	266.7
Trios 3	28.9	25.0
Trios 4	15.8	66.7

CV: coefficient of variation; IQR: interquartile range.

Results of the pairwise comparisons among the different IOS systems for matching error and mean distance are summarized in Table 4 and Table 5.

Table 4

Pairwise comparisons of IOS systems with Tukey contrasts for matching error

IOS	IOS’	Estimate	Adjusted P-values
Omnicam	Medit i500	-0.009	.245
Primescan	Medit i500	-0.022	< .001a
Trios 3	Medit i500	-0.020	< .001a
Trios 4	Medit i500	-0.018	.002a
Primescan	Omnicam	-0.013	.031a
Trios 3	Omnicam	-0.012	.064
Trios 4	Omnicam	-0.010	.177
Trios 3	Primescan	0.001	.997
Trios 4	Primescan	0.004	.896
Trios 4	Trios 3	0.002	.982

aSignificantly different P-values.

IOS: intraoral scanner.

Table 5

Pairwise comparisons of IOS systems with Tukey contrasts for mean distance.

IOS	IOS’	Estimate	Adjusted P-values
Omnicam	Medit i500	-0.719	.505
Primescan	Medit i500	-1.282	.063
Trios 3	Medit i500	-1.384	.039a
Trios 4	Medit i500	-1.245	.074
Primescan	Omnicam	-0.563	.715
Trios 3	Omnicam	-0.665	.578
Trios 4	Omnicam	-0.526	.761
Trios 3	Primescan	-0.102	.999
Trios 4	Primescan	0.036	1.000
Trios 4	Trios 3	0.139	.998

aSignificantly different P-values.

IOS: intraoral scanner.

DISCUSSION

The aim of this in vitro study was to evaluate the accuracy of five IOS systems with the parameters matching error and mean distance using a novel reference model. The results distinguished two groups of IOS, with Primescan, Trios 3 and Trios 4 revealing similar values for trueness and precision, while Medit i500 and Omnicam demonstrated larger values for matching error and mean distance. Therefore, the null hypothesis that there would be no difference in performance with regards to the primary and secondary outcomes was rejected.

The present results demonstrated higher accuracy for the newer IOS generations. The most stable performance for trueness and precision were found with Trios 4. Several studies have investigated scanning accuracy under different circumstances.141516171819202122 To the authors’ knowledge, only a few have used standardized models.1222 Dental scanning technology continues to advance rapidly with the introduction of updated and new IOS devices as well as face scanners, along with processing CAD-CAM software.18 There is no longer any doubt that digital impression techniques are attractive for dentists, as well as for the dental manufacturer and researcher.20212324 Therefore, introducing a reference model into daily clinics, laboratory work, and research could facilitate the measurement of trueness and precision of current and future scanners, independent of scanner technology and manufacturer. Establishing the standard deviation with a reference mean could be a calibration check for dental scanning systems and an automated software performance, widely used in daily clinics and laboratories.

Today, companies incorporate the use of standardized models into the development of new IOS systems. The optimal model design is one that would avoid recognition errors.12 The presented test model with the schematised specifications, proved to be efficient, since there were no recognition errors with any of the IOS. The current version of ISO 20896-1:2019, which provides guidance on test methods and objects to assess the accuracy of digitizing hand-held scanning devices for CAD-CAM systems, focusing on precision of manufactured test specimens, while no specific requirements for IOS itself are defined. Authors of a study from 2017 suggested that it is necessary to have complex and variable elements as a part of a standardized model in order to increase the recognition rate of the model.12 The design specifications of a reference model would need to expand to include full-arch prototypes, with tooth-like shape and translucency. The IOS 20896-1:2019 test objects serve as a good template, which can be expanded with several different crown and inlay preparation designs to generate an individual object that is highly recognizable for IOS, but also for the operator afterwards during evaluation. Whatever design is used, an in vitro standardized reference model can never fully replicate the situation encountered in clinical practice, due to the absence of saliva, impairments from limited mouth opening, the tongue, and the individual subject; consequently, the scanning of a reference model will always be simpler than an actual clinical situation.14 The characteristics of the mentioned model not only made it easier for the IOS to recognize the structures, but also helped in the later phase of evaluation to detect potential artifacts.

In the current study, the five IOS systems were chosen because of their commercial availability and popularity. All of them except for Omnicam are approved for full-arch restorations. The manufacturer of the Omnicam system does not recommend its use for full-arch scanning, and its application in this specific full-arch situation was beyond its capacity. Adding other IOS would have provided more depth to the comparison. In the current study, five scans were performed with each IOS, which is a generally accepted number also applied in previous studies.1718 A possible limitation of the present study was that analyses were restricted to matching error and mean distance. Regarding the scanning pathway, there were minor deviations from the manufacturer’s recommendation, which could be discussed as a limiting factor.

Future investigations could focus on scanning performance with the introduced reference model with higher case numbers of Primescan, Trios 3, or Trios 4 scanning systems and even transferring it lab-side to more powerful optical industrial scanning devices.

CONCLUSION

The novel reference model used in this study can facilitate the assessment of trueness and precision of IOS devices and could be considered for inclusion in an extended version of ISO 20896-1:2019. Using this reference model, the Trios 4 was shown to have the best combination of trueness and precision of the IOS devices tested.