The Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms-How We Do It.

There is a plethora of Artificial Intelligence (AI) tools that are being developed around the world aiming at either speeding up or improving the accuracy of radiologists. It is essential for radiologists to work with the developers of such algorithms to determine true clinical utility and risks associated with these algorithms. We present a framework, called an Algorithmic Audit, for working with the developers of such algorithms to test and improve the performance of the algorithms. The framework includes concepts of true independent validation on data that the algorithm has not seen before, curating datasets for such testing, deep examination of false positives and false negatives (to examine implications of such errors) and real-world deployment and testing of algorithms.