A Novel Method for the Intelligent Recognition of Lattice Fringes in Coal HRTEM Images Based on Semantic Segmentation and Fuzzy Superpixels.

High-resolution transmission electron microscopy (HRTEM) can directly obtain the lattice fringes and structure parameters of coal. Aiming at present problems in extracting lattice fringes in HRTEM images, such as unlocated fringe regions, single-threshold segmentation, unclassified fuzzy superpixels, and tedious fringe pruning, an intelligent recognition method based on semantic segmentation, deep neural networks, fuzzy superpixels, and other algorithms is proposed. For unlocated fringe regions, the fringe regions are automatically localized with semantic segmentation. The whole semantic segmentation network adopts DeepLab V3+ based on ResNet to reduce unnecessary operations brought by non-fringe regions. For single-threshold segmentation of the image, the image is chunked before anything else. The genetic-optimized watershed algorithm is applied to divide the fringe base maps and non-fringe ones in order to avoid the distortion caused by different lights and shades of the image. For the fuzzy superpixels between the fringes and non-fringes, a similarity category judgment method based on neighboring pixels is proposed to solve the problem of unclassified fuzzy superpixels and to enrich and perfect the information of the lattice fringe base map. Eventually, as for lattice fringe overlap caused by coals piling together, a similarity judgment method based on the fringes' characteristics is proposed to remove the bur portion of the lattice fringes and improve the pruning rate. Combining the above theories, a visualization tool based on MATLAB App Designer is designed, and the above four steps can be completed by this app to accurately display the results of coal aromatic lattice fringe identification in HRTEM images. Comparison with the lattice fringes drawn by leading experts shows that the fringes interpreted by this method are reliable. This method facilitates the extraction of lattice fringes in HRTEM, which lays the foundation for the labeling of HRTEM images in a variety of deep learning algorithms and facilitates the direct observation of coal structures by researchers.

Introduction

The complexity and nonhomogeneity of coal make it exceptionally difficult to research its structure. The utilization of both coal and natural gas in coal is closely related to the coal structure. In fact, the structure of coal has been studied by testing techniques such as X-ray diffraction,[1,2] nuclear magnetic resonance,[3,4] infrared spectroscopy,[5,6] high-resolution transmission electron microscopy (HRTEM),[7−9] and atomic force microscopy (AFM).[10] Especially, HRTEM has garnered a lot of attention from coal researchers because it can directly examine the coal microcrystalline structure,[11−14] study lattice fringes, and extract structure parameters in coal,[15−19] which may then be utilized to build coal structure models.[20] Deep processing and interpretation of high-resolution transmission electron microscopy images is an important part of obtaining lattice fringes. At present, there are three main methods for the interpretation of lattice fringes: direct manual interpretation of fringes by the human eye, which is of low accuracy and time-consuming;[14] traditional extraction,[21] which is currently the dominant method of interpretation and means that the HRTEM images are processed using relevant software to obtain binarized images and then manually interpreted or quantitatively calculated at the pixel level; and intelligent extraction,[22] where a computer does all the interpreting work.

For manual extraction, the main manifestations are that in the process of manual extraction, a large number of lattice fringes need to be processed and labeled; the whole process is tedious and consumes lots of time and labor.[14]

For traditional extraction, negative images, region selection, contrast enhancement, and some morphological operations are available to obtain parameters such as fringe length and orientation to construct macromolecular models.[9,23−26] In the process of traditional extraction fringe extraction, a high-resolution HRTEM image has different brightness. Because the single-threshold method does not completely save the lattice fringe information, the computer cannot accurately locate the region of lattice fringes each time. In addition, the problem of unclassified fuzzy superpixel blocks between the lattice fringe base map and the non-lattice fringe base map has not been resolved. The final lattice fringe base map loses a lot of information.

For intelligent extraction, intelligent extraction is currently less applied. MASK R-CNN neural networks can be used for interpretation work with good results.[22] Neural network training requires an accurate fringe base map, but the method of obtaining the fringe base map does not solve the problem of unclassified images with varying brightness and darkness and blurred superpixels. Therefore, training the neural network by manually drawing fringes on these distorted lattice fringe base maps only results in imperfect fringes.

In view of the above problems, this paper integrates neural networks, image processing, and other theories into lattice fringe extraction. A new method is proposed to solve the problems of information distortion by the single-threshold method, non-automated lattice fringe region determination, and unclassified fuzzy superpixels. The results are compared with the lattice fringes drawn by experts and prove that this method is feasible. It will make the base map information of the lattice fringe richer and more complete, and it will enable scientific researchers to interpret fringes more accurately and easily.

Methodologies

Fourteen samples from coal seams 2, 3, and 8 in Xishan, Shanxi Province, China, and six samples from coal seam 16 in Yimin, Inner Mongolia Autonomous Region, China, were selected. The coal samples were lightly crushed in an agate mortar for 10 min followed by grinding the coal samples to less than 200 mesh, dispersing them in ethanol, sonicating for 20 min, and then dropping them into a standard TEM copper mesh. The “high-resolution” mode with a magnification of up to 50 million times is used to image the contours of the lattice fringes. Finally, HRTEM images were taken by a 200 keV transmission electron microscope (JEOL, JEM-2100F).

The method of extracting HRTEM lattice fringes based on semantic segmentation and fuzzy superpixels proposed in this paper is mainly divided into four steps: semantic segmentation to locate fringe regions, determining fuzzy superpixels, classifying fuzzy superpixels, and automatically pruning fringes. The entire extraction flow chart is shown in Figure .

Figure 1

Flow chart of the extraction.

Semantic Segmentation

To extract the coal aromatic lattice fringes from the HRTEM images, the first step is to locate the fringe region accurately. The current main method is to actively distinguish the image after thresholding, which has many defects. Semantic segmentation can solve the difficulty of positioning the fringe region perfectly. Semantic segmentation is a deep learning algorithm that associates labels or categories with each pixel of an image, and it is used to identify the set of pixels that constitute a distinguishable category.[27,28] Therefore, it is of great necessity to apply semantic segmentation to precisely localize fringe regions. The background and fringe regions of the HRTEM image are shown in Figure . The human eye can clearly distinguish between the background and fringe regions.

Figure 2

Schematic diagram of the background and fringes in the HRTEM image. (a) Original HRTEM image; (b) background region; (c) fringe region.

Researching a large number of the pictures of background regions and fringe regions, it turns out that there is a huge difference in the grayscale distribution. We select a part of the 256 × 256 picture as an example (Figure ). The background region is more concentrated in the grayscale distribution, while the fringe region is relatively scattered. The human eye can easily identify the approximate location of the background and fringe regions, but the thresholding is difficult to achieve. Based on the above features and combined with the means of semantic segmentation in artificial intelligence, a large number of images are trained to achieve the purpose of separating the background region and the fringe region from each other, and then the fringe region is processed accordingly. If you skip the positioning step of the fringe area, directly label the lattice fringes in the HRTEM image, and use semantic segmentation to train them, the trained network model can theoretically input the HRTEM image to get the output of the lattice fringes. However, the process of label definition must take a lot of time to complete manually, and the portability of the result is poor because of the great difference between different coal sample images. However, in this semantic segmentation, using the architecture of DeepLab V3+ based on the ResNet network, the image is labeled through the MATLAB Image Label App, and after the label, the mask image is exported. The size of the input image is 224 × 224; thus, for a complete HRTEM image, it can be first chunked into multiple 224 × 224 sized images for testing and then merged into one complete image afterward. The training device hardware comprises dual NVIDIA GeForce RTX 2080Ti graphics cards and 64 GB RAM. The Deeplab V3+ model adopts an encoder–decoder structure; the main part of the encoder is a pretrained residual network Resnet18, which is used to extract image features, the encoder uses Atrous Spatial Pyramid Pooling (ASPP) to introduce multiscale information, and the decoder further merges the low-level features with the high-level features to improve the accuracy of the segmentation boundary. Encoders and decoders all use cavity-separable convolution.[29,30] The whole part is shown in Figure .

Figure 3

Schematic diagram of features of different parts in an HRTEM image. (a) Background; (b) fringe region; (c) difference in gray distribution.

Figure 4

Schematic diagram of the DeepLab V3+ structure.

Block Threshold Segmentation

The HRTEM image is a high-resolution image, and the brightness of different parts of the image varies greatly. If the single-threshold method is used to determine the fuzzy superpixel range of one image, much useful information will be lost. In response to this problem, combined with the idea of block division, the fringe region is divided into blocks by the size of 64 × 64. After that, a threshold segmentation is performed on each small block to obtain a fringe base map and a non-fringe base map in order to make the fringed bottom map as accurate as possible.

To let the computer automatically fix the threshold required by the watershed algorithm, the annealing algorithm is generally used. However, in order to reduce the computation amount and accelerate the optimization, the genetic algorithm can be used for optimization. The so-called “genetic algorithm” is an algorithm that simulates the mechanism of biological evolution in nature. It follows the law of the survival of the fittest so as to find the optimal solution in the simulation of natural evolution and increase the speed of the algorithm operation. The three fundamental genetic operators are selection, crossover, and mutation.[31−33] Selecting the best individuals from the population and eliminating the inferior individuals is called selection. Crossover plays a vital role in biological evolution, called gene recombination. Mutation is gene mutation that occurs during biological existence. According to the three basic operators, there is a direction to find the optimal solution under the given conditions. When the optimum individual reaches fitness, the problem is solved. The flow chart of the genetic algorithm is shown in Figure .

Figure 5

Flow chart of the genetic algorithm.

The genetic algorithm is capable of global optimization and generality, and it can be combined with other algorithms. The threshold parameter in each block is coded by the genetic algorithm to obtain the initial population, cross, and then mutate, which leads to the optimal threshold parameters. In this way, the uncertainty and subjectivity of each artificially selected threshold can be avoided and the turning rate can be accelerated. Since the human eye selects the duller part when distinguishing lattice fringes, the threshold value in each block should be less than the mean value of the part. Therefore, the initial population should be one gray unit less than the mean gray level instead of being generated randomly in order to speed up the optimization rate.

After the best individual is obtained, it is known that the part of the pixel value less than the threshold of the best individual is the fringe base map, the part of the pixel value greater than the mean value is transformed to 255 as the non-fringe base map, and the others’ category is to be determined. Compared to the time of determining the threshold for the single block, the annealing algorithm is 0.069671 s, while the optimized time of the genetic algorithm is 0.02713 s, which explicates that the genetic algorithm accelerates the speed of finding fuzzy superpixels.

Fuzzy Superpixels

The concept of superpixels is an image segmentation technology proposed and developed by Ren and Malik[34] in 2003. They refer to irregular pixel blocks with certain visual significance composed of neighboring pixels with similar textures, colors, luminance, and other characteristics. When the labels of some pixel blocks cannot be accurately determined, they are defined as fuzzy superpixels.[35] The fringe region is divided into two parts by the watershed algorithm: the fringe base map and the non-fringe base map, both of which are superpixel blocks. The remaining part to be determined is the fuzzy superpixel part. The correct classification of fuzzy superpixels directly affects the accuracy of the fringe base map.

As shown in Figure a, a region in the original image is selected. Figure c is the image after the watershed algorithm is performed on Figure b. There are two superpixel blocks in Figure c, and the black superpixel block is the lattice fringe base map. In order to make the classification more accurate, the fuzzy superpixel parts of the image should be identified. Since the color of the fringes is relatively dim and the human visual system cannot distinguish the gray scale from 0 to 255 as clearly as the computer, the mean value is used as the boundary in an image; those above the mean value are defined as relatively whiter, and those below the mean value are defined as relatively darker. The pixel classification in the range from the segmentation threshold to the mean value is not accurate. What may be a fringe base map or a non-fringe base map is called fuzzy superpixels. The blue part in Figure d is the fuzzy superpixels. By comparing Figure c,d, it can be found that the bottom map of the lattice fringe in the upper left part of Figure c may be decoded into several separated fringe segments. However, if the fuzzy superpixels are considered, it is clear that the fringe bottom map will not be completely separated for sure. Therefore, it is crucial to solve the problem of unclassified fuzzy superpixels.

Figure 6

Schematic diagram of the fuzzy superpixels. (a) Original HRTEM image; (b) image of the selected region; (c) superpixels; (d) fuzzy superpixels.

At present, determining the classification of the fuzzy superpixel in the fringe regions remains a problem to be solved. The most common methods for determining the classification are neural networks, regression, and others. If the BP neural network is used to predict the fuzzy superpixels in the fringe regions, the input of the network is the grayscale values of the non-fringe and fringe parts of each block, and the corresponding outputs are 0 and 1. The neural network is trained for each block, the corresponding fuzzy superpixels are put in the trained network, and the probability of fuzzy superpixels in each block can be predicted. A probability of less than 0.5 is equal to 0, which corresponds to the non-fringe part; a probability of more than 0.5 is equal to 1, which corresponds to the fringe part. Such an approach requires neural network predictions for each block, which is computationally intensive and time-consuming. At the same time, the prediction is based only on the gray value of a single pixel, completely ignoring the gray value around the pixel, which lacks comprehensive consideration. For example, there are four pixel points with grayscale values of 91 and 139, namely, A, B, C, and D. By means of the BP neural network, the probabilities of A and B are the same, and so are C and D. This shows, regardless of the other pixels in the 8-neighbor, that the center pixels are predicted only with respect to their own grayscale size and not with respect to the surrounding pixel distribution. Obviously, that is extremely unreasonable. Similarly, the regression prediction method is used to make the data satisfy a certain functional model, which also does not take into account the grayscale values around the pixel.

Therefore, it is not practical to completely ignore the situation around the fuzzy superpixels and merely use one mathematical model to predict the classification of the fuzzy superpixels. This paper proposes a similarity category judgment method based on neighboring pixels. Since the gradient around the fuzzy superpixel decreases in an increasing trend, a neighborhood size of 3 × 3 is selected. As shown in Figure , with Pixel x5 as the center, all pixels in the 8-neighbor are taken as a whole x, and the blue regions are fuzzy superpixels. We select the pixel with a gray value of 109 in the fuzzy superpixels as the judgment point (marked as pixel α). We then compare the similarity between pixel α and each of the remaining 8-neighbor, namely the pixels with gray values of 46, 89, 123, 119, 140, 121, and 63 (fuzzy superpixels with a value of 91 are excluded). The greater the similarity is, the more similar they are. The whole process uses the Spearman correlation coefficient judgment formula.[36] The Spearman formula uses monotonic equations to evaluate the correlation of two statistical variables. If there are no duplicate values in the data and when the two variables are completely monotonically correlated, the Spearman correlation coefficient is +1 or −1, as shown in eq .where x is the original data and y is the comparison data.

Figure 7

Schematic diagram of pixel comparison. (a) 8-Neighbor; (b) superpixels; (c) the central pixel; (d) the compared pixels.

Calculating all similarities between the centered pixel and each of the remaining 8-neighbor not only takes into account the surrounding pixels but also facilitates the calculation (compared with BP neural network and regression). When the human eye observes a single pixel, it observes the surrounding pixels as well. Therefore, the gray scale of the surrounding pixels will affect the judgment of fuzzy superpixels. For the human eye to observe certain fuzzy superpixel β, its gray value is H. The threshold of the part (where β is) is X, and the mean value of this part is M. The pixel value that is not more than X must be the fringe base map. The pixel value that is no less than M cannot be the base map. The number of pixels in the β-centered 8-neighbor whose pixel value is no more than threshold X is n1, and that with their pixel value being no less than M is n2. If n1 > n2, it means that it belongs to the fringe base map. Since the 8-neighbors have eight pixels excluding the center, each pixel has a probability of 0.125. Thus, the probability of β belonging to the fringe base map is 0.125 × n1. If n1 < n2, the probability of β belonging to the non-fringe base map is 0.125 × n2. If n1 = n2, we calculate the sum of the gradient of β and the superpixels in the fringe base map and the superpixels in one fringe base map in the eight neighborhoods. Wherever the gradient sum is lower is the type of superpixel the module β is closer to. As shown in eq ,

In conclusion, it will be more accurate to comprehensively consider the classification of pixels from two aspects of similarity and the human eye’s characteristics. Suppose that the weights of the two factors are ω1 and ω2, respectively, where ω1 + ω2 = 1, h = a × h1ω1 + h2ω2. The value of a is ±1. If n1 > n2 and the gray level of the contrasted pixel is less than X, a = 1; if n1 > n2 and the gray level of the contrasted pixel is more than M, a = −1. Likewise, if n1 < n2 and the gray level of the contrasted pixel is less than X, a = −1; if n1 < n2 and the gray level of the contrasted pixel is more than M, a = 1.

Fringe Pruning

After solving the issue of fuzzy superpixels, the image will be skeletonized to obtain the lattice fringe image. However, due to the stacking between coal seams, some lattice fringes are intersected. In the previous interpretation methods, this problem was solved by manual painting, which was time-consuming and labor-intensive. A method of automatic processing of fringe branching is proposed in order to make the whole step more intelligent. The preliminarily obtained lattice fringes are labeled and sorted for each connected region so that each fringe has a corresponding label, which is finally shown in Figure a. Then, the bur of the image is removed. Due to the pixels of the boundary points in the image, it is not possible to judge their 8-neighbor. Therefore, the image can be enlarged, and the periphery of the image can be padded with zeroes as shown in Figure b.

Figure 8

Schematic diagram of the enlargement of the label image. (a) Label image; (b) expanded label image.

As for each element in the traversal images, if the pixel value in the image is more than 0, the convolution operation is performed on it. The convolution kernel is

The convolution formula is as follows:where, h(i, j) is the convolution kernel and f(x, y) is the gray value.

The step length S of the convolution operation is 1. As shown in Figure , all the elements whose operation result is more than 3 times of their own pixel value are counted. At the same time, if the value of the convolution operation is the largest in its 8-neighbor, it is labeled as the intersection point; all the convolution results equal to 2 times of their own are marked as boundary points.

Figure 9

Schematic diagram of convolution.

Excluding the intersection points, what remain are broken fringes. Since the shortest lattice fringe is 0.25 nm and a pixel in the HRTEM image is generally 0.03 nm, considering intersection, the connected region formed by three or less than three pixels is the bur, and all the connected regions are traversed to remove burs. After removing the burs, all the lattice fringes after segmentation are obtained, and the similarity of these fringes is calculated one by one according to the new label order. The grayscale average, grayscale variance, grayscale third-order moment, median, and modal number of the fringes are selected as the eigenvalues for judgment, and the Spearman correlation coefficient judgment formula is used. We then compare the similarity of each fringe. If the eigenvalues of each original connected region are similar to the lattice fringes without intersection points, it means that they belong to the same category. Otherwise, they do not belong to the same kind. If the categories are the same, we add the intersection points, and vice versa.

Results and Discussion

Results of the Fringe Area Localization

The original 280 high-resolution pictures were cropped and rotated to obtain 16,800 224 × 224 pictures. Of these, 11,760 were used as the training sets, 2520 were used as the cross-training sets, and 2520 were used as the test sets. Semantic segmentation was trained on those three kinds of sets. The final accuracy rate reached 97.3%. The training results are shown in Figure . It can be seen that the abovementioned problems of thresholding have been solved to a large extent, reducing the presence of a large number of noise points.

Figure 10

Schematic diagram of training results. (a) Sample image 1; (b) test result of sample image 1; (c) sample image 2; (d) test result of sample image 2; (e) sample image 3; (f) test result of sample image 3.

We now take the grayscale thresholds of 60, 70, 80, 90, 100, and 110 as an example (Figure ). It can be seen from the thresholding that when the thresholds are 60 or 70, the background region is relatively removed more, but the main fringe region loses lots of useful pixels. When the threshold is 70 or 80, more fringe regions are retained, but the background parts have more and more noise. When the threshold value is 100 or 110, the background regions have much more noise, which makes the processing extremely inconvenient. It can be seen that the thresholding directly obtains the lattice fringe base map. Locating the lattice fringe regions requires human observation, and the computer cannot locate it autonomously. Meanwhile, semantic segmentation can locate them accurately, reducing some unnecessary operations in computer processing.

Figure 11

Schematic diagram of threshold segmentation. (a) Threshold 60; (b) threshold 70; (c) threshold 80; (d) threshold 90; (e) threshold 100; (f) threshold 110.

Results of the Block Threshold Segmentation

A single-threshold segmentation cannot solve the distortion of the fringe base map caused by different lights and shades in the image. The size of original image is 2240 × 2240 and can be divided into 1225 64 × 64 blocks. Figure a is an example of a section from the original image. It can be seen that the red part is brighter and the blue part is darker. With a lower threshold under which the base map of the dark part is shown accurately, the bright part loses most of the information as shown in Figure b. On the other hand, with a higher threshold under which the base map of the bright part is shown accurately, the dark part has too much noise, affecting the subsequent processing, as shown in Figure c. Figure d illustrates the result of block threshold segmentation, which shows that both the bright and dark parts can be accurately displayed and much information of the fringe base map is retained.

Figure 12

Results of block threshold segmentation. (a) Original HRTEM image; (b) lower threshold; (c) higher threshold; (d) block threshold.

Results of the Fuzzy Superpixel Classification

Taking Figure as an example, the pixel with a gray value of 109 is judged for similarity as shown in Table . It can be found that pixel α is most similar to the pixel with a gray value of 140. Therefore, the similarity category judgment method for neighboring pixels proposed in this paper not only takes into account manual extraction but also applies similarities for the purpose of improving objectivity and accuracy. We apply this method to Figure to identify the attribution of each pixel as accurately as possible, and the final effect is shown in Figure .

Table 1

Calculation of the Similarity Value of Each Pixel

contrast pixel grayscale	a × h1	h2	h
123	0.5	0.68260	0.64608
89	–0.5	0.67960	0.44368
46	–0.5	0.59870	0.37896
119	0.5	0.73850	0.69080
140	0.5	0.88440	0.80752
121	0.5	0.68050	0.64440
63	–0.5	0.79320	0.42990

Figure 13

Schematic diagram of blurred fuzzy pixel processing results. (a) Fuzzy superpixels; (b) lattice fringe base map; (c) removing burs; (d), merging the same fringes.

From Figure a,b, it can be found that the fuzzy superpixels have their own labels, and the lattice fringe base maps obtained by this method can maintain more complete information.

Results of Fringe Pruning

The lattice fringe base map is skeletonized as shown in Figure b. We then remove the burs of the lattice fringes and finally compare the separated fringes according to the similarity degree. If the similarity is high, the intersection points will be filled in; otherwise, the separation continues. Figure b can be divided into 59 parts after deburring, of which 26 parts belong to the fringe after breaking. The similarity calculation is performed on these, and the results are shown in Table . From Figure d, it can be seen that many burs are removed and intersections are added according to the similarity degree. Comparing the expertly interpreted lattice fringes in Figure c, it can be found that the fringes interpreted by the method of this study are accurate and reliable.

Figure 14

Schematic diagram of lattice fringes. (a) Original HRTEM image; (b) original lattice fringe; (c) expert drawing; (d) final lattice fringe.

Table 2

Calculations of Different Label Similarity Values

label	h	label	h
1	–0.09546	35	0.99304
3	0.99802	36	0.83548
4	0.98325	40	0.99547
9	0.99989	42	0.99976
14	0.99924	43	0.99972
19	0.99990	45	0.08508
22	0.99999	47	0.25132
24	0.99999	50	0.99906
25	0.97146	51	0.13861
27	0.99499	52	0.99973
28	0.99993	53	0.99851
30	0.98311	54	0.99061
33	0.99565	56	0.83532

An HRTEM image with a size of 2240 × 2240 takes several hours to a day to trim manually, but this method can trim it in about 20 min. After the final fringe is obtained, the angle of the lattice fringes is calculated according to the start and end positions of the fringe, and the direction of the lattice fringes is determined by the angle. A straight line can correspond to two angles, for example, y = x, which is both 45 and 225°. Thus, in the process of calculating the angle, what is needed is to calculate the angle between 0 and 180°. Finally, the angle distribution diagram can be obtained, which is convenient for researchers to observe the direction of the lattice fringes.

Development of the MATLAB App

All the above steps are developed by MATLAB and encapsulated with the MATLAB app. The early MATLAB human–computer interaction is realized through the MATLAB GUI, in which some visualization tools can be made to facilitate the usual algorithm calibration and verification. However, it still has many shortcomings, such as relatively low-end components, constant bugs, errors reported when opening the last saved interface in many cases, and difficulty in writing GUI function codes and their poor readability. At present, the above shortcomings can be completely avoided through MATLAB App Designer. First of all, there are many kinds of components. Component control is convenient. Interfaces built by them look appealing. Furthermore, because the MATLAB app is object-oriented, the code is simple to write and easy to read. Finally, the portability is strong. The finished app interface can also be encapsulated and packaged. It is convenient for use in the MATLAB software and online as well. This is why MATLAB App Designer is chosen to develop a visual interface.

The app mainly includes the following functions: loading images that need to be processed, like HRTEM images, lattice fringe images, superpixel images, fringe length distribution images, and fringe direction distribution images; analyzing fringe length and fringe direction; and allowing scientific researchers to process and judge more quickly and conveniently. All the above images, length distribution data, and angle distribution data can be saved in one folder for future use. One example of using the app is shown in Figure .

Figure 15

Schematic Diagram of the MATLAB App.

Conclusions

This study proposed an intelligent recognition method based on semantic segmentation, deep neural networks, fuzzy superpixels, and other algorithms. The original HRTEM image was processed to obtain an accurate lattice fringe base map, and then the pixels of this map were quantitatively calculated to finally obtain the fringes. Problems of previous handling are addressed in this study with targeted solutions:

For unlocated fringe regions, semantic segmentation was used to automatically identify fringe regions and ignore non-fringe regions, reducing noises generated during HRTEM image processing while saving a lot of time.

For single-threshold filtering, the image was chunked first to avoid the distortion caused by different lights and shades of the image. Then, the genetic-optimized watershed algorithm was applied to determine the optimal threshold for each block, weakening the influence of human subjectivity and binarization on the decoding process while preserving the information in the image as complete as possible.

For the fuzzy superpixels between fringes and non-fringes, a similarity category judgment method based on neighboring pixels was proposed to solve the problem of unclassified fuzzy superpixels and to enrich and perfect the information of the lattice fringe base map. Accurate fringe base maps can lay the foundation for labeling of HRTEM images in a wide range of deep learning.

For lattice fringe overlap caused by coals piling together, this paper proposed a similarity determination method based on the fringes’ features, which was used to quantify the relevant pixels of the fringe base map in order to remove burs rapidly and accurately. Comparison with lattice fringes drawn by leading experts demonstrates the feasibility of pruning the fringes.

The development of the MATLAB app can provide reliable technical support for users to obtain a large amount of lattice fringe information more conveniently. All the above four steps can be completed by the app. The app file is freely available to relevant researchers on GitHub. The website is https://github.com/YICHUANSUANFA/Research-of-HRTEM.

Future studies need to explore the causes of the spatial state distribution of coal lattice fringes. A coal HRTEM database containing various regions can also be established for researchers. The study of lattice fringes can facilitate the construction of coal macromolecule models in the field of coal molecular geochemistry. In addition, the novel method for intelligent recognition of lattice fringes is more beneficial to HRTEM research in polymer materials, carbon materials, and graphene fields.