OBJECTIVES: Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated. METHODS: Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed. RESULTS: The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory. CONCLUSIONS: We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.
OBJECTIVES: Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated. METHODS: Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed. RESULTS: The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory. CONCLUSIONS: We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.