Automated assessment of joint synovitis activity from medical ultrasound and power doppler examinations using image processing and machine learning methods.

OBJECTIVES: Rheumatoid arthritis is the most common rheumatic disease with arthritis, and causes substantial functional disability in approximately 50% patients after 10 years. Accurate measurement of the disease activity is crucial to provide an adequate treatment and care to the patients. The aim of this study is focused on a computer aided diagnostic system that supports an assessment of synovitis severity.
MATERIAL AND METHODS: This paper focus on a computer aided diagnostic system that was developed within joint Polish-Norwegian research project related to the automated assessment of the severity of synovitis. Semiquantitative ultrasound with power Doppler is a reliable and widely used method of assessing synovitis. Synovitis is estimated by ultrasound examiner using the scoring system graded from 0 to 3. Activity score is estimated on the basis of the examiner's experience or standardized ultrasound atlases. The method needs trained medical personnel and the result can be affected by a human error.
RESULTS: The porotype of a computer-aided diagnostic system and algorithms essential for an analysis of ultrasonic images of finger joints are main scientific output of the MEDUSA project. Medusa Evaluation System prototype uses bone, skin, joint and synovitis area detectors for mutual structural model based evaluation of synovitis. Finally, several algorithms that support the semi-automatic or automatic detection of the bone region were prepared as well as a system that uses the statistical data processing approach in order to automatically localize the regions of interest.
CONCLUSIONS: Semiquantitative ultrasound with power Doppler is a reliable and widely used method of assessing synovitis. Activity score is estimated on the basis of the examiner's experience and the result can be affected by a human error. In this paper we presented the MEDUSA project which is focused on a computer aided diagnostic system that supports an assessment of synovitis severity.

Introduction

Chronic arthritis is a heterogeneous group of diseases characterized by long-lasting inflammation of joints. They can influence the patients’ general condition and involve other organs besides joints. The estimated prevalence rate of chronic arthritis is up to 1.5% of the population. Rheumatoid arthritis causes substantial functional disability in approximately 50% of patients after 10 years. Accurate measurement of the disease activity is crucial to provide adequate treatment and care to the patients. Semiquantitative ultrasound with power Doppler is a reliable and widely used method of assessing synovitis. Synovitis is estimated by an ultrasound examiner using a scoring system graded from 0 to 3. The activity score is estimated on the basis of the examiner’s experience or standardized ultrasound atlases. The method requires trained medical personnel and the result can be affected by human error.

The goal of this paper is to present and summarize of the research output of the MEDUSA project. The project was carried out by research teams from the Polish-Japanese Institute of Information Technology, Sogn og Fjordane University College, Helse Forde, Medical Technology and Equipment Institute and the Silesian University of Technology.

The project is focused on a computer aided diagnostic system that supports an assessment of synovitis severity. The main goals of the project are the creation of algorithms and methods for automated assessment of synovitis activity and the development of a prototype software system. As a research output of the MEDUSA project, novel image processing techniques were developed to detect multiple types of localized features which will provide a reference for measurements performed on ultrasound images. These measurements were integrated into a function approximating the human assessment. Machine learning methods were used to train the feature detectors and the assessment function on the ultrasound images of synovitis cases, annotated and scored by medical experts. As a final result of the MEDUSA project the software prototype system was clinically validated by comparing its results with the assessment carried out by medical personnel [1].

Methods and algorithms for automatic assessment of degree of synovitis by analysis of ultrasonic images of finger joints

The precise localization of finger bones in the ultrasound images is an important part of the synovitis detection algorithms. Within the research project a novel technique of bone surface detection based on the concept of a confidence map was proposed. The confidence map depicts the certainty of the information registered in ultrasound images, which is directly related to the likelihood of ultrasound transmission and allows for highly accurate localization of the bone surface [2]. The research was based on the texture approach, assuming that there exists a characteristic texture pattern, which enables automatic description of the changed tissue. In order to find it, several concepts were examined. Some of them used a matrix of features based on the global information, for example first order features, co-occurrence matrix, gray-tone difference and run-length matrix. Other applied masks work locally to generate a new image, e.g. local binary patterns and Laws’ texture energy.

The classic metrics used to evaluate the accuracy of image segmentation are: sensitivity, specificity, precision, fallout and the commonly used F-measure, which compares segmentation results with a manually labeled reference image to find the mismatching regions. The next implemented metric is Dice’s coefficient, which is not very different in form from the Jaccard index. This coefficient is defined as the ratio of twice the number of matching pixels to the sum of pixels of the two segmented results, the ground truth and the automatic segmentation result. Also, the Hausdorff distance was implemented as a reliable evaluation method. This measure is generally treated as a useful similarity measure between two images, although it is noise sensitive. Another metric – Pratt’s Figure of Merit corresponds to an empirical distance between the ground-truth boundaries and those obtained with the chosen automatic segmentation result [3].

The second developed technique was based on the bilateral filtering concept. However, instead of the direct comparison of pixels intensities, the costs of the connecting paths joining the pixels were taken as their similarity measure. The connection costs were computed using the fast double-scan algorithm, which establishes the cumulated deviations between the pixels in the processing mask. The experiments performed on a set of test images contaminated with multiplicative noise of known distribution proved the high efficiency of the denoising methods. Additional tests confirmed the usability of the developed enhancement schemes for the image segmentation purposes.

Another part of the performed research was focused on the methods of biomedical image colorization. The comparison of efficiency and accuracy of a wide range of colorization methods was performed and consequently presented as a book chapter [4]. Several state-of-the-art methods were implemented and evaluated. Within the project, a new approach to the colorization, based on the bilateral filtering schema, was also elaborated. The new colorization method is a semiautomatic approach, where a user (physician) indicates color hints in the form of so-called scribbles, which provide the information about the required color in a given region of the image. The main aim of the method is to assign an unknown chrominance value to each grayscale image pixel using the values from the nearest scribbles. It was achieved by the calculation of pixel membership for each inserted scribble using the bilateral-filtering based approach. Two adjustable parameters included in the algorithm allow for fine tuning of colorization outcomes.

After approval of local Ethics Committee patients recruitment had been started from outpatients’ clinic at Rheumatology department in Forde Norway. Both new diagnosed arthritis patients and patients with known arthritis were included

Results

Joint ultrasound is a standard procedure in new diagnosed arthritis patients and patients with known arthritis. It was previously decided that only metacarpophalangeal and proximal interphalangeal from fingers 2, 3, 4 and 5 were assessed. The collection of over 400 images from the 21 patients was created. Collected images were used in clinical testing of developed software.

The developed prototype of a computer-aided diagnostic (Fig. 1) provides centralized storage for medical data and their descriptions, enables exchange data between users, giving them possibility to consult specific cases. Local repository was and access has been given by remote storage limited only to verified/registered users. Users are able to filter the data according to given criteria and manage filters themselves. This stage supports browsing, viewing, processing and comparing data records. An important component of an integrated system is the Medusa Annotation Editor, which was embedded, extended and tested in a Virtual Desktop Interface (VDI) model. The resulting system was provided as a cloud-based application. Its advantages are: no need to download data and no risk of hardware incompatibility on the client site [5].

Fig. 1

MEDUSA – a computer-aided diagnostic system prototype.

Bone and skin detectors were used to evaluate all implemented classifiers, using optimal parameters found before, by an automated search system. The evaluation metrics used for experiments are precision, recall, fallout, and general accuracy obtained from classification over the testing set which contains data from outside the training set. For all experiments the learning and classification time was measured to compare computational requirements of the classifiers. The Random Forest classifier was selected as the final bone and skin detector configuration because it provides the best compromise between performance and computational requirements. The final feature selection was performed, starting with a basis of 10 image filters, which were used as features in a Random Forest classifier. The following filters were used: Gaussian smoothing (GS), first and second derivatives of GS in y-direction, Laplacian of GS as well as positive and negative threshold filters of the first and second derivatives and Laplacian of GS. The test results indicate that usually only two or three filters are enough to achieve good performance [6].

The Medusa Evaluation System prototype uses bone, skin, joint and synovitis area detectors for mutual structural model based evaluation of synovitis. Bone and skin detectors were compressed into a command line environment which is easy to set up and run on the new Linux platforms without special prerequisites. The images uploaded by a simple web interface (PNG files or DICOM files) are detected by the system and automatically processed [7].

Discussion

Conventional ultrasound examination requires appropriate equipment and sufficient experience the examiner, also assumption of the method is to evaluate the bone and other tissues structures in real time. Whereas repeatability would relate to the same equipment and the examiner [8]. Ultrasound in joints arthritis has proven advantages over conventional radiography in the detection of active synovitis [9]. Undoubtedly the introduction of ultrasound improved the detection of early inflammation and allows to assess the effectiveness of treatment. However, the human factor as well as the difficulties of interpretation prompt to improve this method.

Method using semi or all-automated tools for detecting synovitis may help in an objective, comparable and quantitative assessment of changes in RA.

Our observation confirmed that the developed prototype of a computer-aided diagnostics of the severity of synovitis demonstrates features and benefits related to medical informatics systems [7]. The modular architecture of the system supports intuitive data flow from sources to sinks to easily swap and extend modules. Data processors perform appropriate data manipulations such as filtering and transforming images or feature detection. Finally data sinks are placed where data flow results would be permanently stored allowing to browse such data and use it in further work. Such data flow allows for grouping of elements performing some more general steps and use them separately. It is also possible to save and restore the structure of the processed objects. Such structure can be also exchanged between platform users for discussion or support in USG data assessment. Although obtained results are worse them assessment given by experienced medical staff the final conclusion given by medical experts was “result are promising but further optimization and development is necessary” [1].

Conclusions

The MEDUSA project focuses on a computer-aided diagnostic system that allows the automated assessment of the severity of synovitis. Many aspects concerning ultrasound image processing were addressed in order to create software for the automatic annotation and classification of USG data. The software prototype enables the ultrasound images presenting finger joint synovitis inflammation with the manual annotation prepared by an expert physician to be collected. Finally, several algorithms that support the semi-automatic or automatic detection of the bone region were prepared as well as a system that uses the statistical data processing approach in order to automatically localize the regions of interest. Authors believe that MEDUSA research output will improve the quality and the repeatability of synovitis exams. That may help better control of the disease what is known to improve the prognosis.

The results of the research work were presented at the conferences entitled Computer Systems in Medicine and Health 2015 and 2016 that were created by the MEDUSA project. A list of research papers, that present more scientific details can be found on the project website: www.medusa.aei.polsl.pl