Pressure vessel-oriented visual inspection method based on deep learning.

The detection of surface parameters of pressure vessel welds guarantees safe operation. To address the problems of low efficiency and poor accuracy of traditional manual inspection methods, a method for welding morphological parameters combined with vision and structured light is proposed in this study. First, a feature point extraction algorithm for weld parameters based on deep convolution was proposed. An accurate extraction method of weld image feature point coordinates was designed based on the combination of the loss function via seam undercut feature recognition and weld feature point extraction network structure. Second, a training data enhancement method based on the third-order non-uniform rational B-spline (NURBS) curve was proposed to reduce the amount of data collection for training. Finally, a pressure vessel measurement device was designed, and the feature point extraction performance of the deep network and common feature point extraction networks, DeepLabCut and HR-net, proposed in this study were compared to analyze the theoretical accuracy of the surface parameter measurement. The results indicated that the theoretical accuracy of the parameter measurements was within 0.065 mm.

Introduction

The butt welds of pressure vessels A and B are important factors that affect stress. The weld surface consists of four parameters: weld width, reinforcement, undercut, and misalignment, as specified in the relevant standards. These parameters directly reflect the stress concentration at the welding position, and their measurement is an important for evaluation the welding quality [1,2]. Currently, relatively mature parameter measurement methods are manually completed using a magnifying glass, weld inspection ruler, angle ruler, and other tools. However, these methods exhibit low accuracy, low efficiency, large workload, and high labor intensity. Given the good characteristics of machine vision inspection methods, there have been various degrees of research and applications in welding process inspection or welding seam surface parameters.

Based on the different imaging light sources, machine vision weld inspection methods can be classified into two branches: passive and active visual weld inspection. The passive vision welding seam detection method involves imaging in a natural light environment, using clustering, template matching, Hough transformation, and other algorithms to identify and locate the weld defect information (undercut, wrong edge, etc.) in the image. For example, Kumar used a BP neural network to identify and classify MIG butt welding surface defects according to the standard EN25817 [3]. Apostolos (2015) used the Sobel edge detection algorithm to obtain the undercut edge feature of the weld image and matched the undercut feature template with the feature extracted from the edge of the image, which was detected to realize weld undercut feature recognition [4]. Ding (2016) proposed the use of the circular Hough transformation to locate weld images after edge extraction for determining the location of weld defects [5]. Passive visual weld inspection method has been mainly used in the field of weld defect recognition and classification. However, owing to the poor reflection ability of the weld parameters, it is impossible to calculate the specific values of the weld size parameters in natural-light weld imaging.

Active vision imaging refers to imaging under an artificial light source (including ultraviolet light, infrared light, and laser.). The artificial light source applied to weld inspection is the most structured laser, which can be used to inspect and capture the laser line on the welded surface. The parameter information of the curved welded surface can be expressed as the feature points of the curve. [6,7]. The extraction of the feature points of the welding parameter measurement from the welding seam laser line is important for active vision welding seam measurements [8]. Current mainstream feature point determination methods include the feature point extraction method based on curve slope analysis [9], corner detection method curve inflection point [10], and curve processing and fitting [11]. The feature point extraction methods based on curve slope analysis are as follows: In 2017, Muhammad calculated the first-order derivative of each pixel in each laser curve in an image coordinate system [12]. If the derivative value exceeded the threshold value, it was determined to be a feature point. However, this method is only appropriate for extracting simple-shaped sheet joint weld feature points, and it is not applicable to the extraction of feature points for cylindrical longitudinal welds. Researchers from Tsinghua University have used the third-order Savitzky–Golay filter to calculate the second-derivative value of the laser fringe curve [13]. If the calculated value exceeded the threshold value, then it was determined as an undercut feature point. In [14], a structured light-vision sensor with a narrowband filter was developed. After the centerline extraction of the sensor output image, the weld width and undercut feature points are extracted by calculating the second-order derivative of the laser stripe centerline. However, the algorithm can only measure the parameter width and undercut. This method is better than the previous curve method, but it is sensitive to noise. The feature point extraction methods based on the corner detection method curve inflection points are as follows. In [15], Kovacevic used a corner detection algorithm to extract the feature points of the weld toe and reinforcement on both sides of the weld of U-shaped, V-shaped, square, and lap joints. However, this method does not consider the feature point extraction of undercut defects. In [16], Hessian eigenvectors were used to locate the position of the weld centerline inflection point as a feature point, where the parameter width and reinforcement can be calculated. Consequently, it cannot be used to measure the undercut and misalignment parameters. Feature point extraction methods, based on curve processing and fitting, include performing a continuous sym8 wavelet transformation on the laser curve in the image coordinate system and using the local maximum value of the sym8 wavelet coefficient as the feature point of the weld undercut [17]. However, a smoothly changing feature point (reinforcement, width) cannot be obtained using this method because it produces sharp protruding points in the curve (undercut defects, etc.). A curve fitting plus derivative method was proposed in [18], which uses the zero point of the second derivative of the fitted curve as the characteristic point of the apex of the V-shaped plate welded joint. In [19], a structured light vision sensor was used to determine feature points using NURBS curve segmentation and fitting. The influence of image noise was significantly reduced via fitting because the derivative of the fitted curve was used as opposed to direct determination of the slope of the curve. However, the fitting curve differed from the curve, which in turn caused feature point position errors. The representative products of active visual weld inspection include the weld measuring instrument used for various groove welds devised by Servo-Robot of Canada [20], which can measure the reinforcement, width, and undercut, and the seam tracking system developed by the British MetaVision Company [21], which can monitor the longitudinal and circumferential seam welding process of welding robots. It is evident that when compared to the passive visual weld inspection method, the active visual weld detection method exhibits a better reflection of the image characteristics of the weld parameters and lower inspection environment requirements. This has become the mainstream method for weld visual tracking, identification, and inspection. However, the applicability of the current feature-point extraction algorithm should be improved. It is not possible for any algorithm to simultaneously detect the four parameters of weld width, reinforcement, undercut depth, and misalignment.

In recent years, deep learning has advanced rapidly in machine vision research. The processing methods for extracting specified feature points from images are classified as pose estimation algorithms. The Deep-Pose network for image feature estimation was first proposed by Google in 2014 [22]. The front end of the network uses deep convolutional neural networks (CNN) to extract image feature information at multiple scales, and the back end uses multi-scale features of the convolutional layer. The output is connected to the fully connected layers (FC), and the coordinate extraction task of the feature points in the image coordinate system is completed. Owing to the limitation in CNN’s feature extraction performance at the time, the Deep-Pose accuracy was low. Megvii Technology (2018) proposed a cascaded pyramid network (CPN) [23], which was classified into Global-Net and Refine-Net. Global-Net uses a pyramid structure to extract CNN to extract image multiscale features, and Refine-Net structure regresses on image feature points. To accurately estimate the coordinate information of feature points, the complex and inefficient FC layer in Deep-Pose was abandoned, and a simple and efficient regression method of the deconvolution layer and mean pooling layer structure was used. To date, this is the best network for feature point extraction. A simple and effective bottom-up structure of DeepLabCut has been proposed in the literature [24] for pose estimation of animals, and the network exhibits good migration performance and can be applied to feature point extraction of other objects. In [25], a residual step network structure was proposed to fuse features of the same spatial dimension (intra-level features) to further optimize the key point locations. To solve the problem of severe degradation of feature map resolution in the calculation of traditional CNN structures, [26] proposed an HR-net that can maintain a high-resolution feature map output during the convolution calculation. Later, [27] designed a bottom-up human pose estimation structure based on HR-net. Reference [28] proposed a TokenPose network structure, which replaces the image convolution calculation with the transformer structure in the NLP research field. The network feature extraction AP is close to the literature result, but the network structure is lighter, and it is only suitable for human pose estimation. Therefore, this study attempts to combine a deep learning image feature extraction method with active visual weld inspection technology to provide a new research idea for the field of weld inspection.

The main contributions of this study are as follows:

We analyzed the reasons due to which the standard welding seam surface parameter measurement method cannot be used for actual pressure pipeline inspections, and we proposed a measurement index for welding seam surface parameters to account for the coexistence of numerous surface parameters.

2. A deep convolutional neural network-based image parameter feature point extraction approach can simultaneously extract all parameter feature points in a single image.

3. We proposed a method based on the third-order NURBS curve for enhancing the image data of the weld surface profile of a pressure vessel, which can significantly reduce the number of deep network data collection tasks.

The remainder of this paper is organized as follows. In the section on modeling and numerical analyses, the design details of a weld surface device based on a laser profile sensor are provided and an overview of the weld surface parameter calculation algorithms are outlined. Various experimental results are provided to confirm the validity of the proposed methods in the results section. Finally, conclusions are provided in the discussion section.

Modeling and numerical analyses

Weld surface parameters measuring device based on laser profile sensor

A laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer are shown in Fig 1 as a system for measuring the weld surface parameters based on a structured light model. A KEYENCE LJ-V7080 laser profile sensor with a 32-mm built-in camera and built-in laser with a wavelength of 405 nm were used in the experiment. The measuring ranges of the sensor in the X-and Y-axis directions were 20 mm and 46 mm, respectively. The sensor was installed directly above the cylinder for measurement, and its imaging distance was within this range. The one-line-shaped laser emitted by the sensor hit the surface of the pressure vessel to be measured. The camera inside the sensor captured images near the linear laser range in real time, and the point cloud data were output to the computer via an internal algorithm. The sensor was fixed on the Z-axis electric guide rail, and the line laser emission surface was parallel to the cross-section of the pressure vessel cylinder. The Z-axis electric slide rail was connected to the Y-axis manual slide rail via a bracket, allowing the sensor to move in both directions. The laser sensor and pressure pipe cylinder were kept tangential to the movement direction during the detection procedure.

Fig 1

Device for measuring weld surface parameters, which consists of a laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer.

The laser profile sensor-based welding seam surface profile parameter detection system described in this study is shown in Fig 2. The entire detection process is as follows: ① the laser profile sensor emits a laser that is perpendicular to the weld surface of the pressure vessel; ② point set generated by the sensor is preprocessed to generate the weld contour curve image ; ③ deep learning network, and the characteristic points of the weld parameters are output at coordinates (x, y); ④ according to the proposed numerical calculation index of the weld surface parameters, the numerical calculation of the parameters is completed.

Fig 2

Flowchart of the laser profile sensor-based welding seam surface profile parameter detection system.

In general, the weld contour curve image is generated from the weld contour point set , and this method requires that it can highlight and accurately reflect the surface contour characteristics of the weld. The common image-production method first creates an m×n×3 three-channel back matrix I. Then, to generate image , each contour point set was filled with a black background image in the form of monochrome pixels. Although the picture background (black background) and content (contour pixels) scales are extremely small, as low as 1/10000, this image-generating method can precisely depict the surface contour properties of the weld. The model can be over-fitted if the image is used for subsequent deep learning training. Therefore, after the black background image is created using the above method, the following formula is used to generate a contour curve image with a large ratio between the image background and content. Let Pix be the RGB value of the contour point color and 〈⋅〉 be the calculation method of rounding down. Then, can be expressed as follows:

Weld surface parameter measurement index with multiple defect parameters

The butt joint longitudinal and girth weld reinforcements, as well as three measurement surface parameters (width, undercut, and misalignment), were defined in accordance with the AWS A3.0 "Definition of Standard Welding Terms" [29], ISO 5817 "Welding Joints" [30], AWS D1.1 "Welding Specification for Steel Structures" [31], and ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standards [32]. A schematic of the four-parameter measurement requirements for butt welds is shown in Fig 3. The parameter corresponding to weld width is defined as the distance between the two weld toes according to the AWS A3.0 "Definition of Standard Welding Terms," which is the junction between the weld surface and base metal. The weld reinforcement hre is a parameter in which the weld metal exceeds the height of the fillet welding groove. The weld parameter undercut hun_cut denotes the size of grooves or depressions produced along with the base metal of the weld toe owing to improper selection of welding parameters or incorrect operation methods. Fig 3(A) shows the definitions of weld width lwidth, weld reinforcement hre, and weld undercut hun_cut in the standard. Weld misalignment is defined by the ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standard as the phenomenon of dislocation and unevenness due to the deformation of the welding deviation and other factors during welding. The parameter weld misalignment halign denotes the size and amplitude of the misalignment as shown in Fig 3(B).

Fig 3

Schematic diagram of parameter definition of butt weld 4.

(a) Definition of weld reinforcement, width, and undercut. (b) Definition of weld misalignment.

The existence of weld defects alone, which is a measurement index under the ideal weld condition, is an example of each weld appearance parameter defined and detailed in the standard state. An initially formed weld can exhibit multiple coexisting defects. In the case of weld defects (such as wrong under the influence of edge volume), the definition of the welding seam parameter measurement index in the standard example diagram is no longer applicable. Hence, the appearance parameters of welds at the cross-sectional position of the welds under the conditions of normal welds without defects, single-defect welds, and multi-defect welds are discussed in this study with reference to measurement diagrams of weld appearance parameters in relevant standards. Fig 4 shows a schematic of the weld parameter measurement indicators for various weld shapes.

Fig 4

Schematic diagrams of welding seam parameter measurement index under different welding seam shapes.

(a) Normal weld. (b) Welds with the wrong sides. (c) Welds with single undercut and misalignment. (d) Welds with double undercut and misalignment.

The parameter reinforcement feature point Pre was determined as the highest point of the welding area of the profile curve. The parameter width feature points and are determined as the intersection of the welding curve and the base metal curve, i.e., the welding toes on both sides, as shown in Fig 4(A) and 4(B). The misalignment feature points and coincide with the width feature points in the cases of no defect parameters and single misalignment defects. However, given that the weld toe on the undercut side disappears in the case of a defective undercut, the parameter width feature point is modified to the junction point of the undercut depression curve and welding curve. The misalignment feature point at this time is the intersection point between the undercut curve and base material curve as shown in Fig 4(C) and 4(D), respectively.

The parameter feature points in normal welds are extreme points or corner points, as shown in the above feature point selection example, and the traditional curve extreme and reciprocal analysis methods can complete the feature point extraction task. The characteristic points of the cross-section curve width are weak owing to undercut defects. It is difficult to simultaneously extract all of the characteristic points based on traditional curve analysis methods, and to date, no scholars have proposed a method for simultaneously extracting all of the appearance parameters. Consequently, in terms of image processing feature analysis, in this study, we employed deep learning image semantic segmentation methods to classify weld defects and extracted four parameter feature points of welds in laser curve images.

Design of image feature point extraction network based on CNN

The structural diagram of the coding-decoding-free image feature point extraction network (EDE-net), proposed for this study, is depicted in Fig 5. The input of this network is the preprocessed laser profile image of the weld, and the output of the network is the pixel position of the parameter feature point. The coding part of the network is composed of the CNN backbone. First, the CNN backbone output feature map outputs an n-dimensional feature map after upsampling at the branch of the decoding part. Then, the location of the feature point, roughly extracted by the network, can be expressed as (i = 1,…, n). On branch two of the decoding part, the output feature map with dimensions of 2n is sampled and processed, and feature point correction information is finely extracted. Finally, the position of the parameter feature point in the input image, namely , can be obtained by combining the information of the two output feature maps and .

Fig 5

Network structure diagram of EDE-net.

The EDE-net coding structure output feature map up-sampling methods include bilinear interpolation, deconvolution, and depooling methods. If the image up-sampling multiple is kupsample, then the input scale is ih×iw×3 image , the up-sampling output scale is ih*kupsample×iw*kupsample×3 image , and the pixel position (kp, kp) in the image pixel value is sampled and mapped to the pixel position (p, p) of the image pixel value . The up-sampled image after bilinear interpolation is determined as follows: where , , and 〈⋅〉 are round-down calculations. Based on the formula, it is evident that the bilinear interpolation upsampling method should traverse each pixel, which features a large amount of calculation and slower running speed.

The deconvolution upsampling mechanism is shown in Fig 6. The adjacent and surrounding pixels in Image I were interpolated and filled with pixels (usually the pixel value was 0). To obtain the inverse convolution output image , the convolution kernel and supplemented images were used for the convolution calculation. The pooling kernel was set according to the maximum and average pooling methods, and the de-pooling up-sampling mechanism was the same as the deconvolution up-sampling image supplement method. The up-sampling magnification of the deconvolution and de-pooling methods was lower, but the amount of calculation was less. In deconvolution, the convolution kernel can participate in the entire network training and update the weights, whereas the weights of the pooling kernel cannot be changed. Therefore, EDE-net uses the deconvolution calculation as the up-sampling method.

Fig 6

Deconvolution up-sampling mechanism diagram.

The EDE-net network branch is the task of rough extraction of feature point positions. Based on the concept of completely convolutional network semantic segmentation, the deep feature map output by the convolutional network was processed via a single layer with a stride of 2 and a 3 × 3 deconvolution kernel with a number of feature points n. The output dimension of the feature map was reduced to n, with each dimension corresponding to the approximate position information of each feature point in the input weld centerline image.

Each pixel value is the probability that the feature point is at that position. The theoretical output result, M, of the feature extraction module branch is shown in Fig 7. If the theoretical regression object is the only single-point position of the characteristic point in the laser centerline image of the weld, then the position outside the characteristic point in is a negative sample. If the positive and negative samples are seriously out of balance, overfitting can occur during network training. Therefore, the feature point distance threshold T was introduced, and the pixel distance from the feature point position was set as positive samples within T. This in turn solved the imbalance problem. The theoretical output feature map can be obtained as follows:

Fig 7

Theoretical output results of EDE-net branch 1.

The distances between the feature point and background within the threshold T can be treated as a binary classification task. Hence, the focal loss of the feature classification task can be utilized as the loss function as shown in the output feature map of the branch-network theory. The adjustment parameters α and γ are compatible with the feature classification task Focal-loss [33]. Therefore, the branch-loss function L is as follows: where matrix elements are all one. EDE-net branch 2 is a feature point position–correction task. The convolutional network’s deep feature map is processed through a single layer with a stride of 2, scale of 3×3, and dimension of 2n deconvolution kernels. This yields a 2n-dimensional feature map . The theoretical output results of EDE-net network branch 2 are shown in Fig 8. Based on the approximate location of feature points, the branch two-theory output feature map N adds a feature point location correction value. This implies that the position value of the corresponding element of 2 is the true value of , and 2 is the pixel difference between the theoretical feature points and X-Y-axis of the image coordinate system at the element’s position. Therefore, the theoretical outputs 2 and 2 branch 2 are defined as follows: where c denotes the ratio of the input image scale to the output feature map scale of upsampling. The feature point position correction task is a numerical regression task. Therefore, the Huber loss is used to establish the loss function:

Fig 8

Theoretical output results of EDE-net branch 2.

Then, the EDE-net network loss function can be obtained as L = L+L.

Image data enhancement method for pressure vessel weld surface profile based on third-order NURBS curve

A CNN requires a large number of datasets for training, and the data capacity of the training set is directly related to the CNN’s ability to extract feature points. The conventional approach of producing training sets involves acquiring contour images with an active vision imaging device and manually labeling the location coordinates of the feature points in each image. To avoid this, in the current study, we provide a surface parameter simulation method with the coexistence of multiple defects on the weld surface of the pressure vessel. This in turn allows variation of the types of weld parameters, number of feature points, and parameter values, and thereby, effectively reduces the number of training set collection tasks.

A simulation diagram of the normal weld curve is shown in Fig 9. The parent material area of the curve in the image coordinate system (Fmetal(x,y)) is expressed as follows: where Setwidth denotes the weld width parameter, Rstand denotes the diameter of the pressure vessel cylinder, Lpic denotes the imaging area length, and Wpic denotes the width.

Fig 9

Simulation of normal weld curve.

The simulation feature point position of the normal weld contour curve is presented in Table 1 when the simulation reinforcement is Setre.

Table 1

Simulation feature point position of normal weld contour curve.

Feature point type	Image coordinate system X coordinate	Image coordinate system Y coordinate
P re	L pic	F_metal(x,y)|_L_pic−Set_width+Set_width
Pwidthleft	Lpic−Setwidth	Fmetal(Lpic−Setwidth)
Pwidthright	Lpic+Setwidth	Fmetal(Lpic+Setwidth)

A non-uniform rational B-spline (NURBS) curve [34] simulation weld zone curve was constructed based on the aforementioned three points: Pre, , and . Assuming that the control point of the third-order NURBS curve is k = 3, = [x,y], w is the weight factor of the curve control point, and w, w2>0, the remaining w≥0, Fweld(u) can be expressed as follows:

Furthermore, information can be inversely calculated with three NURBS curve data points: Pre, , and . Let the node vectorbe when the curve is opened and the control point be

n = m+k-1 = 5 such that the curve passes through the first and last control points. The node vector should exhibit k+1 repeatability, and it is set using the accumulation chord length method. This implies that u0 = u1 + u2 = u3 = 0, u5 = u6 + u7 = u8 = 1, , and the setting of the weight factor w of the curve control point are as follows:

The beginning and end points of the curve control points were consistent with those of the data points throughout the reverse solution procedure. The connection points of the curved segment correspond to the nodes of the NURBS curve-defining domain. Therefore, if the data point corresponds to the node value u(i = 0,1,2), then the solution condition for is as follows:

To complete the control-point solution provided by the tangent vector boundary conditions, the following two equations must be included:

By combining the above equations, we can complete the coordinate position solution and construct curve Fweld(u).

Defect parameter undercut simulation

If the undercut width, depth, and offset are δun_cut, Setun_cut, and Δun_cut, respectively, then they indicate that the undercut feature point and width feature point are separated in the X-axis direction. Then, the value range of x in the curve of of the parent metal area on the right is modified to . The positions of the simulated feature points of the undercut weld profile curve with defect parameters are listed in Table 2.

Table 2

Location of the feature point in the simulation of the undercut weld profile curve with the defect parameter.

Feature point type	Image coordinate system X coordinate	Image coordinate system Y coordinate
P re	L pic	F_metal(x,y)|_L_pic−Set_width)+Set_width
Pwidthleft	Lpic−Setwidth	Fmetal(Lpic−Setwidth)
Pwidthright	Lpic+Setwidth	Fmetal(Lpic+Setwidth)
P un_cut	Lpic+Setwidth−δun_cut−Δun_cut	Fmetalright(L_pic+Set_width−δ_un_cut)−Set_un_cut
P mis_align	Lpic+Setwidth−δun_cut	Fmetalright(L_pic+Set_width−δ_un_cut)

To perform the single-defect undercut weld contour curve simulation, five-point coordinates Pre, , , Pun_cut, and Pmis_align can be used as the NURBS curve data points as shown in Fig 10.

Fig 10

Simulation of weld profile curve with single defect parameter undercut Setun_cut.

Defect parameter misalignment simulation

In this case, the width feature points on both sides of the weld coincide with the deviation feature point, and curve of the base material area on the left is consistent with the simulation of the normal weld. The curve of the base material area on the right by Setmis_align along the Y-axis of the image coordinate system and on the right side of the weld is as follows:

The single-defect misaligned weld profile curve simulation was performed using the coordinates Pre, , and . The three points were used as NURBS curve data points as shown in Fig 11.

Fig 11

Simulation of weld profile curve with single defect parameter misalignment Setalign.

Results

Experiment on EDE-net performance of different backbone networks

Test CNN backbone selection

The accuracy of the overall network feature point extraction is affected by the feature extraction performance of the CNN backbone network in the image feature point extraction network based on encoding-decoding. Among the common CNN networks, including AlexNet [35], VGG [36], Res-Net [37], and Inception [38], AlexNet and VGG networks are linear and branchless structures. When the network layers are deeper, they are more difficult to develop, and common problems, such as gradient disappearance and explosion, can occur. To solve the aforementioned problems, Res-Net introduces the residual unit residual in the convolutional layer and realizes identity mapping by constructing direct connections. The network does not degrade as the depth of the network convolutional layer increases owing to the continued stacking. The inception network proposed a structure to obtain close feature extraction capabilities with fewer network layers. The feature map was produced using several convolutions and pooling kernels, and the results were stacked to reduce the number of layers in the network. Resnet50, Resnet101, Resnet152, InceptionV4, and Inception-Res-net were used as the backbones of the CNN based on the encoding-decoding deep feature-point extraction network. Table 3 lists the tested CNN network information, where Top1 accuracy is the CNN structure in the image net image classification result. This can be used as the performance level of the network. The trainable parameters indicate the complexity of the network training owing to the involvement of many parameters, such as the standard of performance, higher accuracy, and better network performance.

Table 3

Tested CNN network information.

CNNs	Top1 accuracy	Trainable parameters	GFLOPs
Resnet50	75.2	23,508,032	11.7
Resnet101	76.4	42,500,160	19.2
Resnet152	76.8	58,143,808	26.7
InceptionV4	80.2	41,111,232	247.5
Inception Res-net	80.4	26,855,264	269.9

Selection of training sets

Based on the existence of the undercut defect parameter, the number of feature points of the weld parameters in images are 3, 5, and 7, namely, datasets D3, D5, and D7. To collect boiler butt type B and pressure pipeline type A welds with diameters of 1300 and 255 mm, respectively, a Keyence LJV-7080 sensor was used. The set of surface contour points comprised the training and test sets. Fig 12 shows the simulation-generated contour image effect and previously acquired contour maps.

Fig 12

Acquisition of contour images to conduct simulations to generate contour images.

(a) No defect parameter welds profile data set D3. (b) Weld profile data set D3 for parameter simulation of defects with a single misalignment error. (c) Weld profile data set D5 for parameter simulation of single undercut defect. (d) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is on the misalignment side.) (e) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is not on the misalignment side.) (f) Weld profile data set D7 with parameter simulation of undercut defects on both sides. (g) Weld profile data set D7 for simulation of the misalignment and undercut parameters on both sides (h) Actual collection data set D3. (j) Actual collection data set D5. (j) Actual collection data set D7.

To improve the data from the aforementioned photographs, an affine transformation was adopted. Let the image size be W×H, image rotation angle be β, and image scaling size be hscale. Then, the affine matrix is determined as warp as follows:

One hundred physical collection datasets D3, D5, D7, and 100 simulation-generated datasets D3, D5, and D7 were selected, with a random rotation angle β = 0–30° and random scaling size hscale = 0.5~0.8. Table 4 contains the training of hyperparameter information, and the network training loss function is used as the evaluation index to determine the ideal CNN structure.

Table 4

Training hyper-parameter information.

Steps	Learning rate
10000	0.005
43000	0.02
73000	0.002
103000	0.001

Fig 13(A)–13(C) show the trend charts of the network loss function extracted from different CNN structures based on the encoding-decoding depth feature points. In the figure, it can be observed that (a) the CNN network structure is more difficult to train without fine-tuning migration, resulting loss does not converge, and network model fails. As the number of training steps increased, the training difficulty of the CNN structure after fine-tuning dramatically decreased, and the ultimate convergence effect improved. (b) Network training is more effective when the layers of the same CNN structure are much deeper. (c) As the performance improves, the accuracy rate of the CNN structure increases. For instance, the Inception-Res-net network structure exhibited a Top1 accuracy of 80.4. The training loss function is reduced when compared to that of the ResNet series CNN.

Fig 13

Loss function trend chart of different CNN backbones based on the encoding–decoding depth feature point extraction network.

(a) Comparison of transfer learning and non-transfer learning. (b) Comparison of loss functions of different network backbones. (c) Final convergence of the network.

Experiment on the actual pressure vessel weld of measurement results

The experiment was initiated by selecting 150 physical collection datasets and 60 simulation datasets D3, D5, and D7 with a random rotation angle β = 0–30° and random scaling size hscale = 0.5–0.8. Among the datasets, 90 physical collection datasets and 60 physical collection training sets with 60simulation-generated training sets were used to train DeepLabCut, HR-net, and EDE-net (Inception Res-net), respectively. Finally, the remaining 60 physical collection datasets D3, D5, and D7 were used as the test set, and the accuracy of the parameter feature point extraction was used as the network performance evaluation index.

All the parameter feature points were regarded as feature points of the same nature if the Euclidean distance between the artificially marked parameter feature point and network output feature point was used as the evaluation index. The parameter reinforcement and undercut feature points are extreme-value natural points. The characteristic reflection is stronger when compared to the parameter width and natural point of the wrong edge inflection point. Given that the feature points of the weld parameters differ in marking and extraction difficulty, using the Euclidean distance as the evaluation index is no longer appropriate. Consequently, the object key-point similarity (OKS) weighted Euclidean distance was introduced as the characteristic evaluation index of the network output parameters. It is defined as follows: where denotes the presence of an undercut in the weld image, denotes the square of the Euclidean distance between the artificially marked parameter feature point and network output feature point, denotes the pixel area of the weld curve in the image, denotes the artificially marked feature point and real position deviation information of the feature point, and is replaced by the mean square error in the numerical calculation.

Currently, average precision (AP) is used to evaluate the same task in a deep network and performance evaluation index of different network structures. The feature points of the OKS and AP indicators contain AP evaluation methods. The correlation between the tasks is as follows: where T denotes the OKS threshold. The network output feature points deviate from the actual feature points. The accuracy of the feature point extraction can be improved further if the network output feature points are returned to the laser curve using the following equation. where (x−y) denotes the coordinate of the characteristic point of the welded seam output by the network, (x,y) denotes the coordinate of the characteristic point of welding on the laser line after correction, and (x−y) denotes the coordinate of any point on the contour line. The network results are presented in Table 5.

Table 5

AP results of feature points extracted from different CNNs.

Methods	CNNs	Training sets	AP0.5	AP0.7	mAP
Before correction	Deep Lab-Cut	90	0.68	0.46	0.42
		60+60	0.63	0.55	0.44
	HR-net	90	0.42	0.29	0.24
		60+60	0.59	0.43	0.33
	EDE-net (Inception Res-net)	90	0.51	0.51	0.45
		60+60	0.65	0.55	0.42
After correction	Deep Lab-Cut	90	0.80	0.64	0.55
		60+60	0.79	0.65	0.49
	HR-net	90	0.81	0.68	0.57
		60+60	0.79	0.68	0.55
	EDE-net (Inception Res-net)	90	0.86	0.78	0.62
		60+60	0.83	0.75	0.64

As shown in Table 5, EDE-net (Inception-Resnet) before and after correction of AP, AP0.5, AP07 is better than Deep Lab-Cut,and HR-net network, and these corrections can significantly improve the feature point. The extraction accuracies were similar for the two training sets. However, the training method for actual measurements and simulations can effectively reduce the data-collection workload.

Table 6 lists the absolute error and standard deviation data for all the parameter feature points. The measurement resolutions of the X-axis and Y-axis of the sensor were 0.005 and 0.001 mm, respectively. If we select a 99.73% confidence interval [μ-3σ, μ+3σ], then the confidence interval of the reinforcement feature point extraction error is [-10.958, 10.8629], and the theoretical measurement accuracy can be as high as 0.011 mm. Similarly, for other parameters, in which the measurement accuracies are mentioned, the confidence interval of the width feature point extraction error was [-9.182, 13.069], and the theoretical measurement accuracy was as high as 0.065 mm. The confidence interval of the extraction error of the undercut feature points was [-10.245, 8.885], and the theoretical measurement accuracy was as high as 0.011 mm. The confidence interval of the error of the feature point extraction of the amount of error was [-11.833, 11.833], and the theoretical measurement accuracy was as high as 0.012 mm.

Table 6

Feature point extraction error information.

Feature point type	Width feature points	Reinforcement feature points	Undercut feature points	Misalignment feature points
Absolute error of Dx	-0.963	-0.250	0.394	1.217
Standard deviation of Dx	1.774	1.962	0.396	2.264
Absolute error of Dy	0.881	-0.115	0.612	-0.068
Standard deviation of Dy	1.426	1.413	0.965	2.224

Discussion

The measurement indicators for the surface parameters of a single weld specified in the relevant verification standards for pressure vessels cannot be effectively used in the measurement of an actual weld surface profile where multiple defects coexist. In this study, the appearance characteristics of the weld surface parameters were measured in the form of image feature points, and algorithm design ideas of the regression image from the feature point coordinates were proposed using the excellent nonlinear mapping ability of CNN networks. An image feature point extraction network based on deep learning was designed to simultaneously extract all the parameter feature points. For the network training measurement, a method based on the 3rd NURBS curve simulation of a realistic weld surface profile was proposed to enhance the training data. Finally, an experimental device was designed to collect the surface data of A and B butt welds, and the deep learning network proposed in this study was compared with the DeepLabCut and HR-net methods under different training sets. The results show that the difference between the training output of the training set network after data enhancement and training set network output AP, which is completely measured, is low. However, the data enhancement method can effectively reduce the workload of sample collection, and the theoretical accuracy of parameter measurement can be realized within 0.065 mm.

16 Feb 2022

27 Mar 2022

Respones to Editor and Reviewer

Dear Editor and Reviewer:

Thank you for your letter and for the reviewers’ comments concerning our manuscript entitled “Pressure vessel-oriented visual inspection method based on deep learning” (ID: PONE-D-22-00457). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Revised portion are marked with “Track Changes” in the paper. The main corrections in the paper and the responds to the reviewer’s comments are as flowing:

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1: Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

Response 1: According to the editor’s suggestion, we edit our manuscript to meets PLOS ONE's style requirements, including those for file naming.

2: We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service.

Response 2: Thank you for your suggestions. I have used Editage to edit my manuscript for language usage, spelling, and grammar. The editing certificate is in the attachment.

3: Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

Response 3: According to the editor’s suggestion, I have uploaded my code in https://gitee.com/Meliao/CAD.

4: We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

Response 4: Thank you for the reminder, ‘Financial Disclosure’ modify to the following: This work was supported by the Science and Technology Plan Project of the State Administration for Market Regulation, grant number 2019MK143.

5: We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

Response 5: According to the editor’s suggestion, I have added Data Availability statement in my manuscript.

6: We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly.

Response 6: According to the editor’s suggestion, I have uploaded data to the website https://gitee.com/Meliao/CAD.

7: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references.

Response 7: According to the editor’s suggestion, I have updated my reference list.

Reviewer #1:

1: This study presented by the authors is well organized, the presentation of the article and the results are close to satisfactory. However, the reviewed literature studies are not up-to-date, the article should be reupload by examining the recent studies (especially 2020- 2021).

Response 1: Thank you for your suggestions. I have added the last 2 years of research in the introduction section.

Attachment

Submitted filename: Response to Editor and Reviewer Comments.docx

Click here for additional data file.

14 Apr 2022

Pressure vessel-oriented visual inspection method based on deep learning

PONE-D-22-00457R1

Dear Dr. Guixiong,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Antonio Riveiro Rodríguez, PhD

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I consider that the revisions have been carried out. this paper is acceptable for this valuable journal

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

20 Apr 2022

PONE-D-22-00457R1

Pressure vessel-oriented visual inspection method based on deep learning

Dear Dr. Guixiong:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Antonio Riveiro Rodríguez

Academic Editor

PLOS ONE