Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities.

A common study design for comparing the diagnostic performance of imaging modalities is to obtain modality-specific ratings from multiple readers of multiple cases whose true statuses are known. Typically, there is overlap between the modalities, readers, and/or cases for which special analytical methods are needed to perform statistical comparisons. We describe our new R software package MRMCaov, which is designed for multi-reader multi-case comparisons of two or more imaging modalities. The software allows for the comparison of reader performance metrics, such as area under the receiver operating characteristic curve (ROC AUC), with analysis of variance methods originally proposed by Obuchowski and Rockette (1995) and later unified and improved by Hillis and colleagues (2005, 2007, 2008, 2018). MRMCaov is an open-source package with an integrated command-line interface for performing multi-reader multi-case statistical analysis, plotting, and presenting results. Features of the package include (1) ROC curves estimated parametrically or non-parametrically; (2) reader-specific ROC curves and performance metrics; (3) user-definable performance metrics; (4) modality-specific estimates of mean performance along with confidence intervals and p-values for statistical comparisons; (5) support for factorial, nested, or partially paired study designs; (6) inference for random readers and cases, random readers and fixed cases, or fixed readers and random cases; (7) DeLong, jackknife, or unbiased covariance estimation; and (8) compatibility with Microsoft Windows, Mac OS, and Linux.