The state of the art of medical imaging technology: from creation to archive and back.

Medical imaging has learnt itself well into modern medicine and revolutionized medical industry in the last 30 years. Stemming from the discovery of X-ray by Nobel laureate Wilhelm Roentgen, radiology was born, leading to the creation of large quantities of digital images as opposed to film-based medium. While this rich supply of images provides immeasurable information that would otherwise not be possible to obtain, medical images pose great challenges in archiving them safe from corrupted, lost and misuse, retrievable from databases of huge sizes with varying forms of metadata, and reusable when new tools for data mining and new media for data storing become available. This paper provides a summative account on the creation of medical imaging tomography, the development of image archiving systems and the innovation from the existing acquired image data pools. The focus of this paper is on content-based image retrieval (CBIR), in particular, for 3D images, which is exemplified by our developed online e-learning system, MIRAGE, home to a repository of medical images with variety of domains and different dimensions. In terms of novelties, the facilities of CBIR for 3D images coupled with image annotation in a fully automatic fashion have been developed and implemented in the system, resonating with future versatile, flexible and sustainable medical image databases that can reap new innovations.

INTRODUCTION

Medical imaging has learnt itself well into modern medicine as a hallmark in the last 30 years and revolutionized health care delivery and medical industry, benefiting the world to a greatest extent, both personally and professionally. Stemmed from the discovery of X-ray by Nobel laureate Wilhelm Roentgen, radiology was born, so was the medical imaging industry. In 2010 alone, as many as 5 billion medical imaging studies were conducted worldwide [1], demonstrating the rich application of images.

Extending human vision into the very nature of disease, medical imaging is enabling a new and more powerful generation of diagnosis and intervention. For example, image-guided surgery proffers minimum invasive operations, while removing intended target completely by transforming vision into reality, not only saving lives that could not be helped otherwise but also bringing immeasurable value to countless more patients, survivors and their loved ones.

To this end, the acquired images not only serve as a barometer measuring the degrees of deterioration of a patient’s disease as well as the extent of recovering progress, but also contribute to a great deal to making a correct clinical decision. As a direct result, efficient and effective management systems are in amounting demand to ensure the data archived today will be still available tomorrow being safe and retrievable. For radiologic images, Picture Archiving and Communications Systems (PACS) are in place to manage them, whereas at the mean time many disparate databases are employed to archive domain-based data. Although plethora efforts have been made to update PACS into managing the other non-radiology images, it has been proven that the model of ‘one-size-fits-all’ is neither sustainable nor workable for today’s modern imaging facilities and image-based applications. For instance, the current state of the art radiology therapy offers highly personalized care that can be tailored to each patient’s unique characteristic factors that cannot be formulated by a unified model.

On the other hand, with this vast amount of rich collections of medical images growing from terabyte (=1012) to petabyte (=1015), innovations can be made from it, as put forward by Sir Muir Gray [2], then chief knowledge officer of the National Health Service (NHS) at the UK, the ‘application of what we know already will have a greater impact on health and disease than any drug or technology likely to be introduced in the next decade’ Image repositories or databases are a typical example collecting large amount of information waiting to be exploited. Hence effective and efficient exploitation of existing databases can provide advances to help researchers unlock the mysteries of insidious diseases and thereby can bring better treatments and cures within reach. Furthermore, melding these advances with the power of the cutting edge digital and information technology is fostering greater efficiency, quality, and value in health care.

This paper will give a summative account of the above issues and draw attention to retrieval of images of high dimensions (>2) and annotation of images in a full automatic manner, the novel development in this study. A brief account of various image modalities is given in the next Section that is then followed by the description of archiving techniques. After giving a detailed explanation of the implementation of our current repository of 2D and 3D images, MIRAGE, the potential renovations from the existing data are then addressed, preceding the conclusion.

THE INNOVATIVE CREATION OF MEDICAL IMAGES

Radiology Imaging

As it happened, medical imaging started with the discovery of X-radiation by Röntgen [3] in 1895 [4]. Composed of X-rays that are in a form of electromagnetic radiation, X-radiation has the ability of identifying bone structures. Consequently X-rays have been employed widely in medical imaging, in particular in the fields of dental care and chest study. To overcome the limitation that X-rays can only give 2D views, X-ray computerized tomography (CT) was subsequently invented in the 1970s by the use of computer processors to generate three-dimensional images on the inside of an object. In doing so, a series of 2D X-ray images are taken [5] from different angles in an attempt to produce detailed cross-sectional pictures of the areas inside the body. As a consequence, the resulting images can provide more information than regular X-rays, and thereforeallow doctors to look at individual slices within these 3-D images. This creation has led to the awards of Nobel Prize to the two creators, Cormack and Hounsfield, in 1979. Since then, CT has become an important tool in medical imaging to supplement X-rays and commonly referred as a CAT scan, which is applied into many fields includingchest, colon health (CT colonography), pulmonary embolism (CT angiography), abdominal aortic aneurysms (CT angiography) and spinal injuries.

Although CT has been routinely applied in the study of human brain, it is not capable for the investigation of functions, giving rise to the development of positron emission tomography (PET) and single photon emission computed tomography (SPET). PET [6] is a type of nuclear medicine imaging technique that produces a three-dimensional image or a picture of functional processes in the body, i.e., how tissues and organs are functioning. By doing so, the system detects pairs of gamma rays emitted indirectly by a positron-emitting radionuclide (tracer) on a biologically active molecule, which is introduced into the body before the scanning. Images of tracer concentration in 3-dimensional or 4-dimensional space (i.e., including time) within the body are then reconstructed by computer analysis software. Because of this advantage, PET is often used to evaluate and study neurological diseases such as Alzheimer’s and multiple sclerosis, cancer, and heart disease. In modern scanners, this reconstruction is often accomplished with the aid of a CT X-ray scan performed on the patient during the same session in order to show the bone structure of an object.

On the other hand, the arrival of magnetic resonance tomography (MR)[7] had strengthened the earlier creations in a radiology department with its first application taking place in 1977 [8]. It appears to offer better contrast and detailed internal structures by showing the difference between normal and diseased soft tissues of the body than PET or CT and contributes significantly when imaging organs like brain, muscles, heart and cancerous objects. Unlike CT scans or traditional X-rays, MR does not utilize ionizing radiation. Instead it employs a powerful magnetic field to align the magnetization of some atoms in the body, and then relies on radio frequency fields to systematically alter the alignment of this magnetization.

Fig. () illustrates an example of 3D datasets of brain images for the same subject obtained by different modality of imaging techniques, including CT, MR and PET. In addition, in order to retain its positive features of each image technique, hybrid tomographyis emerging by consolidating two imaging techniques, for example, PET/CT and PET/MR can display both functional and structural information within one set of constructed images.

Ultrasonic Imaging

In addition to those radiological techniques discussed in Section 2.1, there are many other imaging scanners that have been widely used as diagnostic tools in modern medicine. Diagnostic ultrasound, also known as medical sonography or ultrasonography, has been in service to image the human body for more than 50 years. It utilizes high frequency sound waves that are greater than the upper limit of human hearing to create images on the inside of a body [9] by converting the returning sound echoes into a picture. Subsequently, this technology can also produce audible sounds of blood flow, allowing medical professionals to use both sound and visual data to assess a patient’s health. The well-known application fields of ultrasonic imaging include monitoring pregnancy, studying abnormalities in the heart and blood vessels, and investigating organs in the pelvis and abdomen.

Optical Imaging

Another addition to the advanced radiological imaging technology is the optical imaging techniques that are increasingly relied on to supply traditional patient information in order to arrive at an accurate diagnostic decision. The examples include images of skin lesions in dermatology, retinal pictures in ophthalmology, endoscopic images in gastroenterology and otolaryngology, and gross and microscopic specimens in pathology. The basic principle behind these techniques is the same as that of a camera to capture the light, hence coined the name of light imaging techniques. Fig. () shows an example of retinal image that is taken from the pupil of an eye (the author Gao’s eye), the only window that can see vessel distributions in vivo [10].

Additionally, endoscopic imaging allows medical doctors to peer inside the human body, examining the interior of a number of organs and helping to diagnose medical conditions much earlier and more effectively in the treatment process. Although modern medical device makers have done their best to minimize the scope and size, by its very nature any form of endoscopy is going to be invasive, i.e., somehow, a camera has to find its way to go inside of a patient. Fig. () illustrates an example of key-hole surgery performed on an abdomen with the technique of endoscope.

THE STATE OF THE ART OF ARCHIVING TECHNIQUES

With such large amount of image data created every day, image management faces a huge challenge, in many ways if not possible. By nature, medical images are highly heterogeneous with a variety of domains, including brain, abdominal, cardiac, etc.. Whereas upon each domain, images can be acquired using many imaging modalities, as addressed in Section 2. Furthermore, each image dataset can be presented in various formats (DICOM, TIFF, JPEG) and in different dimensions (2D (retinal), 3D (brain), 4D (e.g, a 3D heart over a period of time), and even 5D (e.g, a functional 4D at a specific anatomic location). Thus, it is neither practical nor work able to standardize these data into a ‘one size fits all’ model. Archiving these data therefore remains a challenging but rewarding task to provide clinicians with all the necessary data in a seamless fashion at the point of care. Significantly, the established archiving system has to be with scope, scale, and the necessary flexibility, to ensure the databases established today will be still in operation tomorrow in the rapidly progressing digital era.

Although at present, large amount of images can be accommodated easily and cheaply, thanks to the advances of computer hardware technology on storage media, it is the way in which images are indexed and thereby retrieved back in real time that appears arduous, for browsing through billions of images in order to find the right one is not an option. Image retrieval can be either conducted based on text-based labelling, e.g., patient’s ID, or textual annotation that can be extracted from an image manually or semi- or full-automatically, or upon its visual or embraced content. In the later case, the retrieval is performed by querying the collections using a sample image to search for the data with similar appearance or similar features, in other words, content-based image retrieval (CBIR).At present, in clinical applications, all the medical image systems are indexed and therefore retrieved using text-based features. Specifically, for radiology images, Picture Archiving and Communications Systems, or more commonly known as PACS, has been in place to supervise the workflow by distributing, managing and interpreting images.

PACS

The first generation of the software of PACS was developed in the mid-1980s in the USA and Europe [11] when modules of PACS were programmed and were implemented independently in the departments of different domains, e.g., paediatric, coronary, or neuro-radiology. PACS enable images, such as X-ray scans, to be stored electronically and viewed on video screens, so that doctors and other health professionals can access the information and compare it with previous images at the touch of a button. The arrival of PACS has since replaced large numbers of radiological film, the medium that had been around for over 100 years and had been almost the exclusive way for capturing, storing, and displaying radiographic pictures. PACS also remove all the costs associated with hard films and releases valuable space currently used for storage. Most importantly, PACS have the potential to transform patients’ experience of the care they receive across an array of networked hospitals. By this way, if a patient has relocated from one place to another, his/her medical records together with their medical images can be easily retrieved from the system as if from the same hospital.

In principle, PACS can take forms of any of image formats, such as TIFF, or lossy compression format JPEG, as well as home grown formats built into an image scanner. In practice, however, in order to communicate images with each other to ensure an image acquired from one radiology department can be readable by the others, a standard format is required to prevent the repetitive work on converters translating formats from one to another. This requirement has led to the development of DICOM, also known as (aka) Digital Imaging and Communications in Medicine, which was initiated in 1992 in the USA, while in liaison with other Standardization Organizations in both Europe and Japan [12]. DICOM currently has widely been utilized in medical imaging field, in an attempt to facilitate standard communications of medical images. This standard has been continuously extended to meet the demands of practical applications.

CBIR

As much as we enjoy using text to search a database of any types of data due to its simplicity and straightforwardness, other forms of retrieval would be welcome for interpretation of images into words is neither nature nor practical. Although PACS has gradually matured over the last 25 years, there are still abridged parts to be bridged even for radiologic images. For example as explained in [13], general radiology PACS don't provide the display functionality to shore up nuclear medicine studies. In addition, some PACS don't fully support DICOM for nuclear medicine in favor of easy storage and retrieval of nuclear medicine studies. Nevertheless, because of the maturity of PACS, other image specialties tend to join PACS, which has led to the deployment of mini-PACS as self-contained tailor-made image management system for visible light images, for example, mini-PACS for pathology [14], by which image scanners are in service to scan specimen slides, whereas mini-PACS archive and store the scanned images. This however in turn leads to an issue of IT infrastructure since the size of this technology generated can be in an excess of 1 TB per week in a typical institution with 10 pathologists and can well overwhelm local global IT facility infrastructures. Furthermore, the challenges of integration and interoperability have to be faced when it comes to join the main flow of whole hospital PACS, as these scanning devices need to accommodate the currently used and accepted standards, including DICOM for image format and HL-7 (http://www.hl7.org/about/index.cfm), for information systems connectivity, which appears far a field. With this in mind, an efficient and effective archiving system requires a novel architecture to ensure queries can span not only to data sources of distributed collections, but also without prior standardization of the original data, entailing each local-repository with scope, scale and flexibility. Therefore, mini, niche, and third-party databases that fit for purpose should be encouraged instead of opting for a unified PACS that fits for all.

In a PACS, the search for any image is normally performed by usingtext, such as patients’ ID numbers, the types of disease, etc., which is a very efficient way in search of correct records of a patient, especially useful when it comes to interface with Hospital Information Systems (HIS) or Radiology Information Systems (RIS) with known IDs. But, the establishment of image archive systems also serves the intension of research and education, leading to new discoveries and creations. To this end, searching image databases based on contents, i.e., the information that an image contains can certainly help in an effective way in finding relevant data, especially within a repository housing large amount of image volumes where browsing through the data is impossible. Therefore a plethora amount of research has been carried out to study content-based image retrieval for the past 25 years with many promising results [15-17], in particular, since the forum of image CLEF, the cross language image retrieval [18] was established in 2003.

Content-Based Image Retrieval (CBIR) for 2D Images

Since the emerging of internet in 1990, coupled with the advances of computer hardware, vast amount of textual and imagery data are available online, prompting the creation of text-based search engine, such as Google and Yahoo, in an attempt to filter the relevant data. For image data, however, thanks to their embedded information being in the pictures, CBIR has been researched both horizontally and vertically. As the trend continues, the progress in the last century (1994-2000) has been very well documented at [19], whereas the state of the art in the last decade (2001-2008) was reviewed by [20] with future directions. In general, a CBIR system extracts features of images in terms of their global visual information, such as colour, texture, and shape, and then represents these features using mathematical vectors that in turn are employed to index each image. When a query image is submitted, the system only needs to extract these features from the query and perform the comparison with the feature database that has been stored in advance. In this way, the retrieval process of an image can be as fast as that in a text-based system since the similarity calculation is based on numerical data.

In terms of CBIR in the medical domain, the research started in 2002 with many established systems, mainly with an intention of research and illustration [21, 22]. The more recent development on 2D images has been reviewed at [23]. CBIR might not be applied in any medical domain at present; but it is the nature of utilizing images to search images which shows its potentials.

CBIR for 3D Images

With the advances of modern imaging techniques, more and more medical images are in three or higher dimensions, allowing a coherent and collective view. However most current databases still archive and index them in 2D form based on the fact that many of these images are constituted of 2D slices, especially in CBIR systems. As a result, a number of limitations have arisen with the most significant one being that the information extracted from a single 2D slice cannot be representative as slices are getting thinner, or in other words, resolutions are getting higher.

In terms of non-medical images, retrieving 3D images mainly focuses on shapes, to meet the demand of internet retrieval. For this reason, the feature of invariance of transformations (including rotation, scaling and translation) is required to ensure the same object can be retrieved when the view angle is changed [24, 25], i.e., invariant of view angles. In contrast, retrieval of medical images of 3D pays more attention to the inside of an organ rather than to the overall shape, as in the case of brain images. Therefore for medical images, the approaches employed are quite different. Table lists most of the current 3D CBIR systems for medical images, which are further detailed below.

In the system of QBISM, 3D functional brain images are queried and visualized [26], by which intensity-based volume data are stored for spatial references, whereas brain atlas Talairach [35] is employed to concert a region-based retrieval. The key to the system is the application of volumetric data type (i.e., Region or Volume as expressed as ) to represent image data.

In other cases, the retrieval task of 3D images can work well based on the feature extraction [27] from 2D slices, depending on the application fields of the created database.

On the other hand, semantics based retrieval may cover images of all dimensions as evidenced by [30]. The strength of this work therefore lies in the approaches employed for categorisation of images that bear semantically well-defined sets. In other words, the images should be represented by well-defined combinations of features with varying levels (low, high, global, regional, etc.). This task itself however in most cases poses greater challenges than semantic representation itself. Nevertheless, semantics based retrieval of medical images remains one of the current trends. Likewise, ontology-based [28] and knowledge-based approaches [29] can shorten the semantic gap to a certain extent between low level features and high level semantics, which in turn requires skilful expertise, i.e., knowledge, to interpret images and convert contents into textual descriptions.

For subject-based images that bear centralised characterises, local features can play an important role in indexing and retrieving images. For example, the system of FICNDS [32] employs physiological kinetic features for retrieving images. Similarly, Batty and Gao [33, 34] have employed binding potential (BP) values to index functional PET images. Although effective, this method is very discipline-defined and heavily relies on the additional supply of extra information. In FICNDS, to define a tracer kinetic model, plasma time activity (PTA) curves should be obtained from a series of blood samples, which are not easily available for most of the images in a database. Although PTA can still be modeled by the application of control regions as applied by Gao et al, a sequence of images acquired over a period of time, say 90 minutes, are still in need, which again is not readily available in most of image repositories. Additionally, in essence, the establishment of kinetic models stems from the data of 2D slices, which may lose information in between slices. For a system warehousing images of variety of domains, more general approach appears to be in demand in order to be sustainable.

Texture-based approach for retrieving of 3D+ cardiac images has been applied by Glatard [31] with the employment of 2D Gabor filter. While working on 4D (3D + time) heart images, their adoption of Gabor is again a strict 2D form based on regions coupled with an extra parameter dedicated to myocardium feature.

Although a texture-based approach can be applied in a 2D form due to the nature that a 3D dataset is composed of a stack of 2D slices, using a slice-by-slice 2D approach can miss the information that interlaces within the volumetric data. Thus, in terms of a 3D form of texture, this spatial structural information should be extracted from a cube instead of a square. Towards this end, while working on images of 3D brain, Gao [24] and Qian [25] have extended the approach of Local Binary Pattern (LBP) [36] into 3D form to extracting texture information that is subsequently utilized for indexing them in their developed system MIRAGE (http://image.mdx.ac.uk/vin/demo.php), which is described in details in the next Section. The other three popular texture based approaches, including Grey Level Co-occurrence Matrices (GLCM), Wavelet Transforms (WT) and Gabor Transforms (GT), are also implemented in the system, again with appropriate extensions to 3D forms.

MIRAGE - AN ONLINE LEARNING SYSTEM ON MEDICAL INFORMATICS

Fig. () depicts a few interfaces of a CBIR system, MIRAGE. With the server located at Middlesex University in the UK, the system at present accommodates over 100,000 2D and 3D images and facilitates domain-based(top-right), atlas-based(bottom-left), and content-based retrieval for both 2D and 3D images (bottom-right).

The System

Fig. () illustrates the flowchart of the architecture of MIRAGE. To address the problems that current text-based image retrieval systems suffer, MIRAGE integrated the methods of both CBIR for 2D and 3D collections and automatic image annotation to label the images with its keywords, leading to a high level of semantic search. It is therefore consists of three modules as shown in Fig. (), with components of image annotation, 2D image retrieval and 3D image retrieval respectively.

Built on the open source GNU Image Finding Tool (GIFT, http://www.gnu.org/software/gift/), the online database is based on the Query-by-Example (QBE) paradigm coupled with a facility of user-relevance feedback whereby retrieved images most closely resemble a query image in appearance (i.e., the content that an image is carrying).

For 2D images, two algorithms have been implemented for indexing image collections, which are IDF (Inverse Document Frequency) and Separate Normalization. The IDF is a classical method and is based on counting the number of documents in the collection being searched, which contain (or are indexed by) the terms in question [37], and has been applied in text retrieval systems, giving rise to the efficiency when employed in an image system.

Additionally, feature normalization is required to compensate the scale disparity between the feature components that are defined in different domains. On the client side, a web page based interface is given. Whilst the client-server communication is achieved using the XML-based Multimedia Retrieval Mark up Language (MRML). All client-server communication, including queries from the client or results returned by the server, is realized through message passing. Consequently, the client can be implemented in any programming language. The current MIRAGE client is implemented using PHP (Personal Home Programming) language to generate dynamic web pages for the client web browser.

Archiving 3D Brain Images in MIRAGE

To develop a CBIR system, two phases are in need including deposit and retrieval respectively. At the phase of deposit of the data, the collected data firstly undergo a pre-processing stage to normalize them to ensure that all the images share the same resolution before the indexing stage.

For 3D brain images, four texture based methods are implemented as pointed in Fig. (, at bottom-right graph), including 3D Local Binary Pattern (LBP), 3D Grey Level Co-occurrence Matrices (GLCM), 3D Wavelet Transforms (WT) and 3D Gabor Transforms (GT) that are detailed in [24, 25]. Fig. () displays the framework of CBIR for 3D images.

As illustrated in Fig. (), the collection of 3D brain images firstly underwent a pre-processing stage to normalize them into the same resolution before the indexing stage. After spatial normalization of volumetric brain data into a standard template, the data are then divided into 64 non-overlapping equally sized blocks, from which, 3D texture features are extracted to create a feature database. On the query side, a pre-processing stage is introduced to detect the potential VOI of lesions after spatial normalization from a query image. Subsequently, the extraction of 3D texture features from a query only takes place from these potential sub-blocks that, in the retrieval stage, are in turn compared with the corresponding features in the feature database to obtain retrieval results.

Specifically, the formulae of 3D forms for the four texture-based approaches are an extension of their 2D versions. For example, in terms of 3D LBP, a 3D dynamic texture is recognized by concatenating three histograms obtained from the 2D LBP on three orthogonal planes, i.e., XY, XZ, and YZ planes, while applied to our normalized brain images and is expressed as

where

And LBP (f (x, y)) is LBP operator in 2D form, and is derived from a general definition of texture in a local circular (p, r) neighborhood. In other words, for each pixel in an image f (x,y), a binary code C is produced by thresholding its value with the value of a centre pixel.

On the other hand, in 3D form, GLCM are defined as two dimensional matrices of the joint probability of occurrence of a pair of grey values separated by a displacement d = (dx, dy, dz), which can be calculated using Eq. (3).

where d,θ represent the offsets of distance and angle in order to achieve rotation invariant and subsequently determine the values of dx, dy, and dz. A normalized co-occurrence matrix P d,ϑ(i, j) is usually obtained by

where L is the total number of grey level of a 3D image.

In addition, a 3D WT provides a spatial and frequency representation of a volumetric image and is formulated as Eq. (5) for a 3D image of f (x,y,z).

where is the daughter wavelet generated by translation on xy and z-axes by b, b and b respectively, dilation by a, and rotation by the angle of the mother wavelet ψ (x,y,z). refers a rotate factors in 3 dimensional space.

With respect to Gabor Transforms, Eq. (6) has been applied in MIRGAE as a 3D Gabot filter.

where is a 3D Gaussian function, together with radial centre frequency F and orientation parameters(θ and φ), determining a Gabor filter in three dimensions. Given a 3D volumetric textured image f (x,y,z), its 3D Gabor transform GT is then defined by

Fig. () demonstrates an example of retrievedusing different texture approaches.

Similar to the situation in 2D forms, the judgement of each approach is subject to the applications of the retrieval task as to which of the measures of size, location or shape plays more important role than the others.

Image Annotation

Another feature that the MIRAGE has is its ability of annotating imagesin a fully automaticfashion, in ahope to achieve ahigh level search ofsemantics while organizing and categorizing images of interests. Automatic image annotation is the process by which a computer system automatically assigns metadata that arecaptioning or keywords to describe a digital image. At present, the Bag-of-visual-Words (BoW) [38] paradigm becomes very popular and has been successfully applied in image categorization. By transforming images into a set of symbolic ‘visual vocabulary’, images are represented using the statistics of the appearance of each word as feature vectors, upon which the learning of an image classification rule could be achieved by a classifier. This idea has been adopted in the MIRAGE system coupled with SIFT sparse coding approach [39], which is conducted in the following four steps as indicated in Fig. ().

Step 1 -- the visual features are extracted from local patches of each image in the training dataset, leading to the construction of a visual dictionary of codebook;

Step 2 -- to quantize the visual features of the image dataset into discrete symbolic‘visual words’;

Step 3 -- an image is represented as a unique distribution (e.g. a histogram) over the generated dictionary of words; and

Step 4 -- image representations of the training dataset obtained in Step 3 are applied to train the classifiers using supervised machine learning methods, thereby the trained classifier automatically allocates new images into corresponding categories and hence labels them.

Different from traditional BoW paradigm, sparse coding is employed in the MIRAGE instead of vector quantization (VQ) to extract the SIFT descriptors of local image patches. Furthermore, instead of using histograms, multiple scales of max pooling are utilized as an image representation by the application of simple linear support vector machines (SVMs). This is in the consideration that when compared with the SVMs using nonlinear kernels of histogram intersection, linear SVMs can dramatically reduce the training complexity while maintaining a good performance.

Evaluation Resultsof theMIRAGE

Results on CBIR

In terms of retrieved results with reference to lesion locations, LBP approach appears to perform the best amongst the four texture-based methods. Table gives the retrieval results on the evaluation of LBP, where by the first row is the labeling number of the location of a VOI assigned by the authors for the convenience of calculations, e.g., ‘1’ refers to the abnormal part in the front top left-most part of the brain. The second row is the total number of images containing VOIs in such positions in the database, whilst the number of correctly detected images by the LBP is given on the third row. Therefore the overall performance of the LBP with respect to VOI locations is measured as a ratio of detected positive VOIs to the total positive VOIs, yielding 91.3% (=168/184).

In terms of precision, the retrieved accuracy is 78% based on ten query images with the ground truth being diagnostic information on the locations and sizes of tumors, demonstrating very promising results.

It is understandable that retrieving images in 3D form might not be performed in real time due to its lengthy processing time occurred on whole volume of data rather than a single 2D slice, one of the setbacks in the development of CBIR systems for higher dimensions. Table 3 lists the average querying time, accounting for the total time that has spent on both feature extraction and retrieval. The second column is the averaged querying time without a pre-processing stage of VOI selection, whilst the third column is with VOI selection. All methods run in Matlab R2009a with a CPU of an Intel P8600 1.58GHz and 3.45GByte RAM.

Table 3

Retrieval Time and Precision of 3D CBIR

Mean Retrieval Time			Mean Average Precision
Methods	Without VOI Selection	With VOI Selection	Without VOI Selection	With VOI Selection
3D GLCM	43.37s	10.96s	0.677	0.690
3D WT	4.46s	1.22s	0.731	0.749
3D GT	38.79m	10.77m	0.714	0.691
3D LBP	0.74s	0.21s	0.774	0.786

As can be seen in Table , the retrieval time with VOI selections offers 4 times faster than the one without. In particular, the retrieval time for 3D GT takes a much longer time than the other methods spending 38 minutes, due to the employment of 144 times of 3D convolutions for each block, whereas the query times for the other methods are in the space of few seconds. The table is also in favor of 3D LBP approach with sub-second retrieval times and the highest precision rate of 78%.

Results on Image Annotation

To assess the effectiveness of image classification, a confusion matrix is firstly created as explained at [40]. In addition, in order to train a codebook for image annotation, a training dataset is firstly selected containing 1000 images that are randomly chosen from our medical image repository. Then 200,000 random patches are collected from these images. Subsequently, from each patch, SIFT descriptors are extracted, yielding a feature database that has the size of 200,000*128 elements, which are finally applied to train the codebook with the size of 1024*128 in terms of feature vectors.

With respect to ground truth for image annotation, six domain names are defined at the highest level, which are brain, lung (x-ray), microscopy, abdomen, ultrasound and graph. Each category is allocated 100 images as ground truth with each half (50) as being training and testing sets respectively. The classification results for the six categories are visualized in a confusion matrix given in Table .

The values in the last column of Table are the Accuracy Rate (AR) values for each class, whereas the values in the last row are the Error Rate(ER) for each class. The Average Accuracy Rate (AAR) for all classes is 99% (297/300), and Average Error Rate (AER) is 1% (3/300), demonstrating the approach of annotation being of efficiency though not conclusive. In addition, an online questionnaire has been created to help to further improve the system. So far, the feedbacks have not shown any untoward result with over 70% response in ‘strong agreement’ with ‘retrieved results’, ‘ease to use’ and ‘useful’ in e-learning processing.

DISCUSSION - IMAGE INFORMATICS

With such wonderful creation of imaging techniques and comprehensive archiving systems fostering generated huge amount of image data, it is anticipated that more valuable information can be exploited, leading to more creations.

A database serves two ways, which is from creation to archive and then from archive to creation. Therefore it is in the expectation that the collection of these huge volumes of medical images can help researchers to unlock the mysteries of insidious diseases and thus can bring better treatments and cures within reach (e.g., personalized radiation therapy). These innovations can be explored in one of the following trends that are shown as following examples to elicit more creations.

PACS-Based Informatics

Based on PACS, computer aided diagnosis are in demand as described in [41]. Towards this, medical imaging informatics infrastructure (MIII) has emerged focusing on existing PACS resources and their images as well as related data for organized horizontal and longitudinal clinical, research, and education applications which would otherwise not be able to perform due to insufficient information [42].

PACS-based imaging informatics is the next research and development frontier in utilizing PACS-related technologies and data for various image-assisted applications. These include image-assisted radiology and surgery, minimally invasive surgery, robotic and remote surgery, maxillofacial imaging, cardiovascular imaging, image-assisted detection (CAD) and diagnosis (CADx), image matching, and imaged-based learning and teaching.

Image-Guided Surgery

Image-guided surgery is another burgeoning field by using minimum invasive ways in assisting surgeons manipulating, monitoring, and/or guiding operations with the advantages of being higher precision of targeting, persistence of longer duration, and the ability of pre-operative planning drawing from patients’ images. As of 2001, there are around 270,000 cases of image-guided robotic operations conducted per year Worldwide [43]. And the trend follows.

During an image-assisted or guided surgery, the images can be acquired in real time to provide an augmented view to the surgeons as exemplified in Fig. (), whereby a tiny camera has to be inserted into the intended position inside the body. Another image-assisted surgery or more commonly known as image-guided surgery is less invasive, especially when it is employed to perform on a brain where every tissue is too delicate to be traumatized in order ascertain a person’s normal life. In this case, images are acquired pre-operative by using MR or CT, whereby the position to open a hole on the skull, the route to insert a probe and the extent of a lesion to be incised can be planned in advance, leading to a success clinical outcome. Fig. () exhibits an interface of path planning where each line refers to the probe route viewed at a different angel to be inserted in for resection of the tumor (red in the bottom-right graph).

The biggest advantage of this type of minimum invasive operation is the maximum safety and minimum trauma to a patient. In most cases, as soon as an operation is completed taking about 20 minutes, patients can be back to normal, resulting in less suffering in terms of time, money, and psychological issues with none or less scars incurred.

Image-Assisted Diagnosis

With respect to optical imaging techniques, one application field is to help clinicians in making a correct clinical decision. At [10], an accurate measurement system, MIDAS, has been developed in determining vascular network properties such as power losses, shear stresses, etc., leading to the diagnosis of hypertension diseases as demonstrated in Fig. (), whereas the left graph is the interface that assists clinicians in measuring vessel geometric in terms of angle, diameter (right-top) and vessel length (bottom-right) in-between bifurcations.

Image-Based Learning and Teaching

A picture is worth a thousand of words, which is particularly true when it comes to learning and education. In particular, in the medical domain, the study of the anatomy of internal organs of the human body relies heavily on graphs and images. Therefore, with the advances of modern imaging techniques, image-based learning and teaching have been flourished. In the case of MIRAGE [24, 25], as pointed out in Fig. (), it has been developed to dedicate to teaching and learning, specifically for MSc students who are on the programme of Biomedical Modeling and Informatics running at Middlesex University at the UK. The system has guided students significantly, especially when they are in need to know more about anatomy of one particular organ.

CONCLUSION

While the wonderful creation of medical imaging techniques has benefited both patients and health professionals tremendously, it has great potentials to contribute more in finding better treatments in the future. To this end, the archive of those generated medical images of huge quantities in a sustainable way becomes paramount, especially which can be further exploited when a new generation of data analysis tools or new information become available. At present, it had been proved that the model of ‘one size fits all’ is neither sustainable nor workable, leading to the exploitation of the development of each individual database that fits for purpose and equips with facilities that can be both functioning independently and searchable when integrated with the other image databases. With this in mind, the infrastructure of the future archive systems should be lightweight, incremental and in a measured way. To this end, MIRAGE repository has been developed facilitating domain-based, atlas-based and content-based retrieval with an aim of e-learning and e-teaching, which has made huge impact to both students and lecturers. Significantly, 3D images are included together with 3D visualization over the platform of internet while delivering the learning materials. Another advantage of MIRAGE is that it employs open source software that is accessible and will contribute to open source collection of software, leading to a wider access of medical image repositories on the search of better education for students in the short-term and better solutions and better treatments for patients in the long-term.

Table 1

3D CBIR Systems of Medical Images

Name/Feature	Imaging Modality	Domain	Reference
QBISM / intensity-based	MRI/PET	Brain	Arya [26]
Pre-defined-semantic-based	CT	Brain	Liu [27]
MIMS / ontology-based	All	All	Chbeir [28]
Knowledge-based	All	All	Chu [29]
ILive - modality-based	All	All organs	Mojsilovic [30]
2D Texture-based	MR	Heart	Glatard [31]
FICBDS / Physiological information -based	Functional PET	Brain	Cai [32]
3D PET / lesion-based	PET	Brain	Batty [33, 34]
MIRAGE / 3D texture-based	MR	Brain	Gao [24], Qian [25]

Table 2

VOI Detection Rate

VOI Location	1	2	3	4	5	6	7	8	Total
Number of images	24	46	18	38	24	12	14	8	184
Correctly detected images	24	42	16	34	24	8	12	8	168
Correct Detection Rate (%)									91.3

Table 4

Confusion Matrix for the Six Medical Image Categories, where B, L, M, A, U, G Represent Categories Of Brain, Lung, Microscopy, Abdomen, Ultrasound, and Graph

		Classification Results						AR (%)
		B	L	M	A	U	G
Ground Truth	Brain	48	0	2	0	0	0	96
	Lung (X-Ray)	0	50	0	0	0	0	100
	Microscopy	0	0	49	0	1	0	98
	Abdomen	0	0	0	50	0	0	100
	Ultrasound	0	0	0	0	50	0	100
	Graph	0	0	0	0	0	50	100
ER(%)		0	0	3.92	0	1.96	0