Strengths and weaknesses of state of the art fiber tractography pipelines--A comprehensive in-vivo and phantom evaluation study using Tractometer.

Many different tractography approaches and corresponding isolated evaluation attempts have been presented over the last years, but a comparative and quantitative evaluation of tractography algorithms still remains a challenge, particularly in-vivo. The recently presented evaluation framework Tractometer is the first attempt to approach this challenge in a quantitative, comparative, persistent and open-access way. Tractometer is currently based on the evaluation of several global connectivity and tract-overlap metrics on hardware phantom data. The work presented in this paper focuses on extending Tractometer with a metric that enables the assessment of the local consistency of tractograms with the underlying image data that is not only applicable to phantom dataset but allows the quantitative and purely data-driven evaluation of in-vivo tractography. We furthermore present an extensive reference-based evaluation study of 25,000 tractograms obtained on phantom and in-vivo datasets using the presented local metric as well as all the methods already established in Tractometer. The experiments showed that the presented local metric successfully reflects the behavior of in-vivo tractography under different conditions and that it is consistent with the results of previous studies. Additionally our experiments enabled a multitude of conclusions with implications for fiber tractography in general, including recommendations regarding optimal choice of a local modeling technique, tractography algorithm, and parameterization, confirming and complementing the results of earlier studies.