Deep neural networks outperform human expert's capacity in characterizing bioleaching bacterial biofilm composition.

BACKGROUND: Deep neural networks have been successfully applied to diverse fields of computer vision. However, they only outperform human capacities in a few cases.
METHODS: The ability of deep neural networks versus human experts to classify microscopy images was tested on biofilm colonization patterns formed on sulfide minerals composed of up to three different bioleaching bacterial species attached to chalcopyrite sample particles.
RESULTS: A low number of microscopy images per category (<600) was sufficient for highly efficient computational analysis of the biofilm's bacterial composition. The use of deep neural networks reached an accuracy of classification of ∼90% compared to ∼50% for human experts.
CONCLUSIONS: Deep neural networks outperform human experts' capacity in characterizing bacterial biofilm composition involved in the degradation of chalcopyrite. This approach provides an alternative to standard, time-consuming biochemical methods.

Introduction

“Biomining” is an industrial process that employs microorganisms for the recovery of valuable metals from sulfidic ores [1], [2]. Dissolution of metal sulfides, such as the copper mineral chalcopyrite, is catalyzed by microbial oxidation of ferrous iron that provides ferric ions for the chemical oxidation of metal sulfides. This regenerates ferrous ions and a cycle between chemical and biological reactions occurs. In addition, sulfur-oxidizing acidophiles assist the mineral degradation process by producing sulfuric acid from inorganic sulfur compounds. Biomining is less harmful to the environment than conventional metal recovery processes [3] and therefore, it is important to further optimize this method.

Biofilms are communities of microorganisms embedded in a self-generated matrix of extracellular polymeric substances (EPS). This microbial lifestyle confers several advantages compared to free-living planktonic cells, such as water retention, protection against stresses, providing nutritional requirements, etc. [4]. Biofilm-forming microorganisms are crucial in commercial heap biomining operations, in which they partly determine the initial metal sulfide dissolution rate [1], [5], [6]. Acidophilic microbes have differing abilities to generate energy from the conversion of the mineral components under moderately thermophilic temperatures. Acidithiobacillus caldus is an obligate chemolithoautotrophic sulfur oxidizer that thrives at pH 2.5 [7], [8]. Leptospirillum ferriphilum is a ferrous iron oxidizing autotroph that is often the dominant iron-oxidizer in biomining environments at extremely low pH (1.3–1.6) and high redox potential conditions [9]. Sulfobacillus thermosulfidooxidans is a mixotroph that primarily oxidizes iron but is also capable of oxidizing sulfur compounds at higher pH conditions compared to other acidophiles [10], [11]. The interplay of species in mixed community biofilms affects metal dissolution rates and therefore, biofilms consisting of acidophilic microbial consortia are important to understand and optimize during metal dissolution.

Epifluorescence microscopy (EFM) can be used in biomining applications to study biofilm structure and spatial distribution of cells on mineral sulfides [12], [13], [14]. In combination with nucleic acid dyes that label bacterial species, it enables the detection of specific groups of microorganisms attached to metal sulfide surfaces [15], [16], [17]. This allows to evaluate the extent of bacterial colonization on chalcopyrite mineral grains and to visualize the biofilm morphology [18], [17].

Machine learning is a field of computer science that enables computers to process and learn from data without being explicitly programmed. Convolutional neural networks (CNNs) are a category of deep neural networks that are able to make predictions in areas such as image recognition and classification. A CNN consists of an input and an output layer, as well as multiple hidden layers that carry out the following tasks: (i) Convolutional layers emulate the response of an individual neuron to visual stimuli and applies a convolution operation to the input, passing the result to the next layer. (ii) Pooling layers combine the outputs of neuron clusters at one layer into a single neuron in the next layer. (iii) Fully connected layers connect every neuron in one layer to every neuron in another layer (Fig. 1). Altogether, this workflow allows fitting multiple parameters of a CNN upon training with a set of images and then the trained CNN is used to classify new images and infer the category they belong to. These neural networks have proven successful in many different real-life case studies and applications, including identifying faces [19], detecting objects [20], and assisting self-driving cars [21]. However, while deep learning is commonly applied to diverse fields of computer vision and easing biological image processing in different fields of biology [22], it only rarely outperforms humans in image classification [23], [24].

Fig. 1

CNN workflow showing how an input image is analyzed by a CNN where image features are detected in the convolutional layer followed by processing of maximum pooling and finally resulting into classification (output layer) of the different microbial species in the biofilm.

In this study, we used CNNs trained with a low number of EFM images representing biofilms of different bacterial compositions, and compared the performance of CNNs versus human experts in correctly classifying new images.

Materials and methods

Microbial species cultivation

EFM pictures were taken of biofilm grown strains of A. caldus DSM 8584 [25], L. ferriphilum DSM 14647 [26], and S. thermosulfidooxidans DSM 9293 [27]. Bacteria were grown in sterile Mackintosh basal salt (MAC) medium [28] with soluble electron donors for inoculation of chalcopyrite cultures. For L. ferriphilum, 4 g/L iron(II)-ions were provided as FeSO4·7H2O. Precipitation of ferric salts was prevented by addition of sulfuric acid to maintain the pH in the range of 1.6–1.8. A. caldus and S. thermosulfidooxidans were pre-cultured using 0.9 g/L potassium tetrathionate (K2S4O6). The medium for S. thermosulfidooxidans was supplemented with 0.02% yeast extract (YE) and 0.1 g/L iron(II)-ions [29]. Cells were harvested by centrifugation at 11,270g for 10 min, washed in sterile medium, and inoculated at an initial cell density of 107 cells/mL to chalcopyrite cultures in 300-mL-Erlenmeyer flasks with 150 mL MAC medium and 2% (wt/vol) chalcopyrite grains (50–100 μm grain size). Equal proportions of cells of each species were used in mixed cultures. All strains were cultivated on a rotary shaker at 37 °C and 150 rpm.

Microscopy sample preparation

About 25 mg of mineral grain particle samples were withdrawn from mineral cultures using a flame-sterilized spatula and incubated in 1 mL sterile MAC medium (pH 1.8) with 4% formaldehyde at room temperature for 1 hour for fixation of mineral-attached cells. This was followed by two washing steps with water and subsequently with 1 mL phosphate-buffered saline (PBS). Samples were stored at −20 °C in 50% ethanol in PBS. Mineral particles were incubated for 10 min in 200 μL of an aqueous solution of 0.01% 4′,6-diamidine-2′-phenylindole dihydrochloride (DAPI) in 2% formaldehyde. Before and after staining of attached cells, mineral grains were washed with 1 mL PBS. Mineral particles were mounted on 10-well diagnostic glass slides (10-well, 6.7 mm; Thermo Scientific) using a glycerol-based mounting medium (CitiFluor AF2) and 22- by 50-mm cover glasses [30].

High-throughput epifluorescence microscopy

Microscopy images were obtained using the EFM platform AxioImager M2m (Zeiss) equipped with a motorized microscopy stage (IM SCAN 130 × 85 – DC 1 mm, Märzhäuser Wetzlar) and a AxioCam MRm camera. This setup was used to generate sets of images for different acidophile microbial mixtures. Each image represented the bacteria stained on mineral particles in an area of 450 × 335 μm (see Fig. 2 and images in the Supplementary Material section “Test For Humans”). Automated acquisition allowed to image between 180 and 504 images per category composed of a different combination of bacterial species that were then used to train deep neural networks [30]. For all categories, data augmentation was used by simple random image duplication to obtain the same number of images (504) for each category used for deep neural networks training.

Fig. 2

Example of EFM images representing the different biofilm categories. The leaching mixtures were composed of A. caldus (A), L. ferriphilum (L), and S. thermosulfidooxidans (S) that were used as pure or mixed cultures, resulting in the following categories: A, L, S, AS, LS, and ASL.

Deep learning application

TensorFlow [31], [32], developed by Google, is one of the most recent deep learning frameworks and provides state-of-the-art implementations to build CNNs. Briefly, the software is written in C++ and offers interfaces to Python. A suite of visualization tools, called TensorBoard, is included within TensorFlow that allows visualization of networks in a web browser and to monitor the training progress. The CNN training was carried out with TensorFlow v1.6 in <10 h on an Intel Core i5 2.0 GHz computer (with 4 CPU cores) but the procedure can be shortened by parallelizing the network training on GPUs (code and images are available in Supplementary Material section “TensorFlow code and microscopy images”). In order to account for variation in accuracy of the algorithm and the effect of over-fitting, resampling was carried out where the same procedure was applied to a different division of the images into training and testing sets (see Supplementary Material section “Deep Learning Resampling”).

Deep learning versus human expert performance

TensorFlow-based deep neural networks were trained on less than 600 microscopy images (after data augmentation) per experimental category (each category represented by a different microbial composition of the leaching mixture, each grown on the same mineral substrate and under same conditions). The leaching mixtures were composed of A. caldus (A), L. ferriphilum (L), and S. thermosulfidooxidans (S) that were used as pure or mixed cultures, resulting in the following categories: A, L, S, AS, LS, and ASL. The category AL (see examples in Supplementary Material section “Example of AL Images”) was not processed because of the insufficient images at disposition for CNN training compared to the other categories and therefore was omitted in the present results. Deep neural networks were subsequently trained on the different categories and their ability to correctly assign new images was tested on a subset of 100 images per category that were not included in the training set. The training performance progress is shown in Supplementary Material section “Deep Learning Performance vs. Amount of Training Data”. Further, the same approach was applied to human subjects in order to compare the performance of neural networks versus humans in their ability to distinguish biofilms of different bacterial composition, and classify them according to their category based on visual features. Human subjects (n = 20; all experienced in work related to microbial bioleaching) were questioned once by a custom-built double-blind test (see Supplementary Material section “Test For Humans”). The test included a training set of images (70 images per category), followed by a test section where subjects were asked to classify 10 unknown images belonging to the same categories as the training set, which corresponded to the same image dataset used for the deep neural network training. The “AS” category was not selected by the random procedure generating the double-blind test for human experts and therefore, it was omitted in the present results (the accuracy of deep learning for identifying “AS” was 81%).

Results and discussion

Deep neural networks can identify characteristic biofilm patterns

Deep neural networks applied to our bioleaching experiments correctly deduced the species or combination of species in a biofilm of unknown composition. Moreover, the neural networks achieved an accuracy of 92.8% compared to 51.5% by the human experts consensus (Fig. 3; expected accuracy by random guessing was approximately 17%, and the best human expert performance was of 80% (see Supplementary Material section “Human Expert's Performance”), deep neural networks performance with samples devoid of bacteria was 16.67%, stdev = 2.87%, CV = 0.17% (see Supplementary Material section “Negative Control”)). For example, the deep neural network correctly classified 97 images out of 100 belonging to the category “A”, while misclassifying three of them to either “SL” or “S” (Fig. 3). Surprisingly, the “SAL” category was poorly classified by humans (13% accurate), but was the second-best guessed category by neural networks (95% accurate). In comparison, the best performance of deep neural networks for image analysis in another biological area was approximately 72% correct for classification of skin cancers compared to approximately 66% for expert dermatologists [23]. This does not imply superiority of the method presented in this study, but might be related to the nature of the analyzed images.

Fig. 3

Deep neural networks (A) versus human experts’ (B) ability in predicting the species composition of bacterial biofilms. The matrices indicate the share of images correctly deduced in the diagonal line (shaded grey) and categories the misclassified images were assigned are shown in the horizontal plane.

While it is difficult to determine the image features used by deep neural networks that lead to a correct decision [22], the following features were mentioned by human experts as decisive factors for classification (see Fig. 2 and images in the Supplementary Material section “Test For Humans”): (i) Cellular shape, in particular for the distinction between “S” having longer-shaped cells compared to “A” and “L” plus “A” was identified based on a slightly smaller size than “L”. (ii) Abundance of “L” cells imaged compared to the other categories. (iii) Increased “A” cell clusters in the colonization pattern compared to “L” and “S” being more sparsely distributed. (iv) Poor attachment of “A” cells compared to the other species. (v) The brightness of images representing “L”. The reasons listed above partially account for the outcome of human experts’ best performances, in particular concerning the classification of “S”. However, it does not explain the major source of confusion between the categories “SAL” and “SL” among human experts that was not encountered in the deep learning evaluation.

Potential of CNNs for the characterization of biofilm colonization pattern images

CNNs were trained on microscopy images of different bacterial compositions and their ability to correctly classify new images was tested and compared to human expert performance. This application demonstrated that deep learning enabled image classification based on recognition of attachment patterns, which are cryptic to human perception. Moreover, our results were achieved with a small training set (between 180 and 504 images per category, on 504 images per category after data augmentation) compared to applications of equivalent performance (e.g., 129,000 images in Ref. [23], 2 millions in Ref. [33]). Finally, the proposed methodology is not limited to a specific experimental setup and therefore represents a method of choice for microscopy imaging-based quantification in environmental microbiology beyond the field of biomining.

Limitations

One limitation of this study was represented by the low number of human testing subjects that were not stratified into different categories (e.g., experts, non-experts, etc.). Therefore, a reduced number of subjects familiar to the field (termed “experts”, whose expertise relied on their working experience related to microscopy imaging of bacterial biofilms and bioleaching research for at least eight months) were included, assuming that they would perform better than subjects external to the field and that inclusion of the non-experts would not improve the information. A further limitation is the unclear character of the determining features that the CNNs used to classify images. However, this common issue relates to all applications of deep neural networks processing a large amount of data [22]. Finally, the bioleaching bacteria of our experimental setup form simpler biofilms in comparison to other environments [4] and the presented method might have limited performances when applied to other environmental samples with a higher species diversity than typical, low-diversity bioleaching environments.

Conclusions

Deep neural networks were trained on a reduced set of microscopy images of microbial biofilms composed of different bacterial bioleaching species colonizing chalcopyrite particles. The performance of deep neural networks in correctly classifying new images was compared to human experts’ ability in performing the same task based on the same training set. Deep neural networks showed superior performance compared to human experts and were able to predict the presence of microbial species or combination thereof in a biofilm of unknown composition. This allowed to measure important features of biofilm development under different experimental conditions by imaging only. Therefore, this provides an efficient alternative to standard and time-consuming biochemical methods, such as qPCR, which may be biased by nonhomogeneous cell lysis during mineral sulfide samples preparation [30]. Biofilms are important in biomining due to ferric attack on the mineral and high concentrations of ferric accumulate in the EPS. Therefore, studies of biofilm can help optimize the bioleaching process. This methodology opens the way to efficient evaluation of high-throughput microscopy imaging in the field of mineral biofilm leaching, and is applicable beyond the presented experimental setup.

Conflict of interest

The authors declare that they have no conflict of interest.