Differentiation of Cytopathic Effects (CPE) induced by influenza virus infection using deep Convolutional Neural Networks (CNN).

Cell culture remains as the golden standard for primary isolation of viruses in clinical specimens. In the current practice, researchers have to recognize the cytopathic effects (CPE) induced by virus infection and subsequently use virus-specific monoclonal antibody to confirm the presence of virus. Considering the broad applications of neural network in various fields, we aimed to utilize convolutional neural networks (CNN) to shorten the timing required for CPE identification and to improve the assay sensitivity. Based on the characteristics of influenza-induced CPE, a CNN model with larger sizes of filters and max-pooling kernels was constructed in the absence of transfer learning. A total of 601 images from mock-infected and influenza-infected MDCK cells were used to train the model. The performance of the model was tested by using extra 400 images and the percentage of correct recognition was 99.75%. To further examine the limit of our model in evaluating the changes of CPE overtime, additional 1190 images from a new experiment were used and the recognition rates at 16 hour (hr), 28 hr, and 40 hr post virus infection were 71.80%, 98.25%, and 87.46%, respectively. The specificity of our model, examined by images of MDCK cells infected by six other non-influenza viruses, was 100%. Hence, a simple CNN model was established to enhance the identification of influenza virus in clinical practice.

Introduction

Despite the availability of rapid tests and nucleic acid amplification assays for quick identification of virus infection, isolation of viruses by the cell culture system remains as one of the golden standards for identifying virus pathogens, especially for emerging virus species. Nevertheless, observation of cytopathic effects (CPE) induced by virus infection is relatively subjective, and requires subsequent reagents such as virus-specific monoclonal antibody to confirm the presence of virus [1,2]. Moreover, it will take longer for cytopathic effects to develop if the amounts of viruses in the inoculated specimens are insufficient or due to some virus strain-specific effects [2,3], which makes the observation of cytopathic effects quite labor-intensive. A better and more objective way to identify cytopathic effects is required.

Nowadays, numerous medical tasks have utilized neural networks to solve the problems or to get better solutions because many tasks we desire to solve were hardly solved by traditional stochastic methods [4-7]. In particular, convolutional neural network (CNN) is a remarkably appropriate model for image recognition and it can differentiate the differences of many classifications at the expert level [5,6,8]. The two special layers, convolutional layer and pooling layer are inspired from the visual system. The convolutional layer of CNN model extracts features from the previous layer and the extracted features are more complex in the later layer as visual mechanism [9,10]. Another important property of the CNN model is the independence of object location for image-recognition [11]. This characteristic is essential for recognition of cytopathic effects because it seems impossible to control the location of cytopathic effects upon virus infection. Therefore, we plan to utilize the properties of the CNN model to solve the challenges in specific identification of cytopathic effects.

In this study, a deep CNN model was developed to recognize the cytopathic effects induced by influenza virus infection of a routinely used cell line, Madin-Darby canine kidney (MDCK) cells. Influenza virus is known to cause annual outbreak of respiratory illness worldwide in people of all ages, and it is one of the most important respiratory viruses causing medically attended acute respiratory illness [12]. The MDCK cell line, first reported in 1966 [13], is the most routinely used cell line to inspect cytopathic effects induced by influenza viruses [1,2]. In this study, we used different input of influenza virus to infect MDCK cells and took photos of cell morphology changes at different time points in the hope to train the CNN model to recognize cytopathic effects induced by influenza virus infection. A total of 601 photos were collected with various time points and viral titers to train the CNN model, and another 400 photos were used to test whether this model could significantly differentiate the cytopathic effects from the normal cell morphology. Additionally, 1190 photos were obtained from a new experiment to test the limit of the model.

Results

Model training

Influenza-induced cytopathic effect is a continuous process, beginning with mild morphological changes within the MDCK cells, and then accumulating over time. Usually, the infection process is described by using the percentage of cytopathic effects observed overtimes, and different inputs of influenza viruses could induce different levels of cytopathic effects. The mock-infected MDCK cells are tightly adjacent to each other as shown in Fig 1A. In the initial development of cytopathic effects, the MDCK cells become smaller and rounded as shown in Fig 1B. At this stage, only 20% of the cells developed cytopathic effects. In the Fig 1C, 50% of the cells became rounded and some of them began to be detached from the culture flask due to cell death induced by severe cytopathic effects. At this stage, the influenza-induced cytopathic effects are obvious for a medical technologist to recognize under the microscope. When more cells were infected by influenza viruses as shown in Fig 1D, 80% of the cells had cytopathic effects or had been detached from the culture flask. Photographs were taken at different time points to collect various phases of cytopathic effects. The numbers of photographs collected consist of virus-infected cells with different virus input and were summarized in Table 1. The intervals between the groups of photographs were not regular because we assumed the model would be more accurate with a more diverse training data. Examples of different levels of influenza-induced cytopathic effects and mock-infected MDCK cells used in our dataset were shown in Fig 1. A total of 601 photos were utilized to train the Training 1 and amongst them, 154 were mock-infected control images.

Fig 1

Influenza-induced cytopathic effects (CPEs) in MDCK cells.

MDCK cells were mock-infected (a), or infected with influenza viruses, which resulting in 20% (b), 50% (c) and 80% (d) of cytopathic effects. Scale bars: 200 μm. Black arrows indicate influenza-induced cytopathic effects. White arrows indicate the area where all infected cells were completely detached and floating in the culture medium.

Table 1

Information table for the Training 1 training data.

	Condition		Sample numbers
	Dose (M.O.I.)	Time (hpi)
Negative samples	-	20	36
	-	24	48
	-	26	50
	-	40	20
Positive samples	0.5	20	46
	0.1	24	15
	0.5	24	34
	0.5	26	55
	1	26	55
	2	26	54
	0.1	27	7
	0.01	40	145
	0.1	40	36

Abbreviation: M.O.I. (multiplicity of infection); hpi (hours post infection)

The original images were colored, but were converted into grayscale pictures to be inputted into our model. Reason being, if a colored photo with a size 1024x1360 pixels were used, the dimension for input would require three channels, red, blue and green color which is three times as much as the dimensions needed with a grayscale image. Therefore, we converted all colored photos into grayscale photos to reduce the input dimensions. Furthermore, we considered the property of recognition where identification of cytopathic effects, theoretically, relied on the patterns of changes instead of color variance. Third, Bui et al. demonstrated that the use of grayscale pictures can generate better accuracy in differentiating figures in some neural network models than color ones. [14]. Therefore, grayscale photos were chosen as the input of the model, considering it may reduce the potential interference caused by colors and enhance the accuracy.

Model validation

After training the Training 1 with 1200 epochs, 99% of the training data (including the training set and validation set) can be recognized accurately as tabulated in Table 2. For examination of overfitting, we further took 400 images from an independent experiment, including 300 images of infected cells and 100 images of uninfected cells as shown in Fig 2A–2C and S1 Table. Our trained model identified 99.75% of the testing data. Moreover, all of the 300 positive pictures were identified correctly, with no error, resulting in a 100% negative predictive value. The accuracy for negative samples was also high with a percentage of 99 and the positive predictive value was 99.66% (Table 2).

Table 2

Comparison of Training 1 and Training 2 with 1200 epochs weights and the saved weights.

	Training 1		Training 2
Weight chosen	1200 epochs	saved weights	1200 epochs	saved weights
Training numbers	601	601	503	503
Training accuracy	0.9900	0.9600	0.9860	0.9662
Testing numbers	400	400	498	498
Testing accuracy	0.9975*	0.9925	0.9457	0.9216
PPV a	0.9966*	0.9900	0.9673	0.9278
NPV b	1**	1	0.8846	0.9

Abbreviation: Pos, positive samples; Neg, negative samples; PPV, positive predictive value; NPV, negative predictive value.

a PPV = numbers of true positives/(numbers of true positives+ numbers of false positives). The ideal value of the PPV for a perfect test will be 1, while the worst value will be zero.

b NPV = numbers of true negatives/(numbers of true negatives+ numbers of false negatives). The ideal value of the NPV for a perfect test will be 1, while the worst value will be zero

**: p-value <0.01

*: p-value <0.05

Fig 2

Influenza-induced cytopathic effects (CPEs) at different time points after infection.

Images of MDCK cells were taken at 25hpi (hours post infection) (a-c), 16hpi (d-f), 28hpi (g), and 40hpi (h). (a) and (d) represent MDCK cells which were mock-infected. (b)(e) and (c)(f) represent MDCK cells which were infected with influenza viruses at 0.05 M.O.I. and 0.5 M.O.I., respectively. (g) represents image of MDCK cells infected with 0.5 M.O.I. of influenza virus at 28hpi. (h) represents image of MDCK cells infected with 0.05 M.O.I. of influenza virus at 40hpi. Scale bars: 200 μm. Black arrows indicate influenza-induced cytopathic effects.

Specificity of the model

In the clinical practice, MDCK cells are regularly used to isolate influenza viruses. However, influenza viruses are not the only species that can infect MDCK cells. Other viruses, such as RSV and HSV, have been described to infect MDCK cells and induce distinct patterns of cytopathic effects [15]. Therefore, we would like to verify whether our model would miss-identify other virus-induced cytopathic effects as influenza infected samples. Additional experiment images of MDCK cells infected with adenovirus, coxsackievirus B3, HSV-1, HSV-2, parainfluenza virus, and RSV were included for analysis. Twenty images were taken for each virus infected cells at 40 hours post infection, considering 40 hours being the last time point of observation for influenza-infection in this study. HSV-1 formed cytopathic effects as reported by Meguro et al. [15] (Fig 3A). On the contrary, HSV-2 did not induce any cytopathic effects in MDCK cells as shown in Fig 3B. Adenovirus induced a relatively minor changes in cell morphology compared to mock-infected MDCK cells in Fig 3C. No cytopathic effect was observed in MDCK cells infected by other viruses (Fig 3D–3F). Next, the images collected from these non-influenza viruses infected MDCK cells were included for analysis to examine the specificity of our model. All of the images mentioned above were not recognized as influenza-infected MDCK cells. In addition, twenty images were taken for each virus infected cells at 16, 25, and 28 hpi and validated by the model. The results were consistent with those using the 40 hpi images (S3 Table). These tests further validate the specificity of our model towards influenza infection.

Fig 3

Infection of MDCK cells by non-influenza viruses.

MDCK cells were infected with herpes simplex virus type 1 (HSV-1) (a), herpes simplex virus type 2 (HSV-2) (b), adenovirus (c), coxsackie B3 virus (d), parainfluenza 3 virus (e), and respiratory syncytium virus (RSV)(f). The images were taken at 40 hpi (hours post infection). Scale bars: 200 μm.

Test for earlier and later images

According to our preliminary analysis on the testing set, this model had a considerably high accuracy when differentiating different virus input, 0.05 multiplicity of infection (M.O.I.) and 0.5 M.O.I., at 25 hours post infection (hpi) (Table 2). Next, we aimed to examine if our model could differentiate influenza-induced cytopathic effects at earlier time points after infection. We fed more experiment data taken at earlier time points, 16 hpi, into our trained model (Fig 2D–2F). We also collected images at 28 and 40 hpi, as shown in Fig 2G and 2H, to examine the recognition rate for late phase of cytopathic effects by our model. The numbers of the influenza experiment data collected at different time points were tabulated in Table 3. Our model had great accuracy identifying all negative samples at 16, 28, and 40 hpi with accuracy of 97.14%, 99.18%, and 99.02%, respectively (Table 4). The positive samples taken at 28 hpi were also recognized specifically with accuracy of 97.83%. However, the recognition rate of images taken at 40 hpi was slightly reduced to 83.08%, probably due to the detachment of virus-infected cells at later time points. The recognition rate of the earlier experiment data (16 hpi) was not as expected, with the accuracy of 88.88% and 15.92% for positive samples infected at 0.5 M.O.I. and 0.05 M.O.I., respectively. The reason for deficiency in the recognition of those pictures at 0.05 M.O.I., could be attributed to the fact that the morphology of MDCK cells infected with influenza in a shorter period was too similar to that of the mock-infected cells (Fig 2D–2F). However, no significant difference could be observed even by manual identification at this stage, either.

Table 3

Information table for the influenza experiment data set.

	Infection condition		Sample numbers
	Dose (M.O.I.)	Time (hpi)
Negative samples	-	16	140
	-	28	123
	-	40	103
Positive samples	0.05	16	113
	0.5	16	162
	0.05	28	154
	0.5	28	123
	0.05	40	150
	0.5	40	122

Abbreviation: M.O.I. (multiplicity of infection); hpi (hours post infection)

Table 4

Comparison of Training 1 and Training 2 with 1200 epochs weights and the saved weights on Experiment Data.

		Training 1		Training 2
Weight chosen		1200 epochs	saved weights	1200 epochs	saved weights
Accuracy of additional testing data	16 hpi Pos+Negc (415)d	0.718**	0.8024	0.6168	0.7638
	16 hpi Neg (140)	0.9714	0.9214	1	0.9285
	16 hpi Pos (275)	0.5890***	0.7418	0.4218	0.6800
	16 hpi 0.5M.O.I. (162)	0.8888***	0.9815	0.6790	0.9320
	16 hpi 0.05M.O.I. (113)	0.1592*	0.3982	0.0530	0.3185
	28 hpi Pos+Neg (400)	0.9825	0.9625	0.9625	0.98
	28 hpi Neg (123)	0.9918	0.9186	1	0.9349
	28 hpi Pos (277)	0.9783	0.9819	0.9458	1
	40 hpi Pos+Neg (375)	0.8746***	0.9120	0.7733	0.9733
	40 hpi Neg (103)	0.9902	0.9417	1	0.9708
	40 hpi Pos (272)	0.8308***	0.9007	0.6875	0.9742
Accuracy of other viruses data	HSV-1	1	0.95	1	0.95
	HSV-2	1	1	1	1
	RSV	1	1	1	0.35
	Parainfluenza virus	1	0.95	1	0.7
	Coxsackievirus B3	1	1	1	1
	Adenovirus	1	0.9	0.95	0.8
PPVa	16 hpi	0.9759	0.9488	1	0.9492
	28 hpi	0.9963	0.9645	1	0.9719
	40 hpi	0.9955	0.9760	1	0.9888
	16 hpi+28 hpi+40 hpi	0.9909	0.9639	1	0.972
	All testing data	0.9927	0.9714	0.9871	0.9569
NPVb	16 hpi	0.5461	0.645	0.4682	0.5963
	28 hpi	0.9531	0.9576	0.8913	1
	40 hpi	0.6891*	0.7822	0.5478	0.9345
	16 hpi+28 hpi+40 hpi	0.6857***	0.7669	0.5856	0.7840
	All testing data	0.7355***	0.8089	0.6370	0.8072

Abbreviation: Pos, positive samples; Neg, negative samples; PPV, positive predictive value; NPV, negative predictive value; HSV-1, herpes simplex virus type 1; HSV-2, herpes simplex virus type 2; RSV, respiratory syncytium virus

a PPV = numbers of true positives/(numbers of true positives+ numbers of false positives). The ideal value of the PPV for a perfect test will be 1, while the worst value will be zero.

b NPV = numbers of true negatives/(numbers of true negatives+ numbers of false negatives). The ideal value of the NPV for a perfect test will be 1, while the worst value will be zero

c Pos+Neg: positive samples and negative samples

d Numbers in the brackets represent the amount of photos

***: p-value <0.001

**: p-value <0.01

*: p-value <0.05

The positive and negative predictive values of these experiment data were also determined. The positive predictive values for the positive samples in the influenza experiment set were 97.59%, 99.63%, and 99.55% at 16, 28, and 40 hpi, respectively. The positive experiment images taken at 28 hpi had a better recognition rate and the negative predictive value was 95.31%. On the other hand, due to the lower sensitivity of initial and late stages of influenza-induced cytopathic effects, the negative predictive values were not satisfactory, where merely 54.61% for 16 hpi experiment data and 68.91% for 40 hpi experiment data. When the machine recognized the influenza experiment data, the positive and negative predictive values were 99.09% and 68.57%, respectively. Finally, the positive and negative predictive values of the machine for all of the influenza experiment data and testing data were 99.27% and 73.55%, individually as tabulated in Table 4.

Training the model by reassigned training data

Since the virus titers might vary in the clinical specimens, we reassigned the original training data and the testing data. All of 1001 pictures were randomly divided into two groups, the training and testing materials. To avoid uneven distribution of the images in the training data and testing data, we split each image set equally, and the amount of images in the reassigned training data and testing data were 503 and 498, respectively. The structure of the model and the training epochs were not changed. We named the previous training as “Training 1” and the retrained training as “Training 2”. The differences of training and testing data of these two trainings and the analysis results were summarized in Fig 4.

Fig 4

Flow charts for Training 1 A and Training 2 B.

(A) The flow chart of Training 1. The blue, red, and black frames indicate the data condition of influenza infection in the training process, testing process, and limitation set, respectively. The viruses in the yellow frame were used for specificity test. (B) The flow chart of Training 2. The blue frame indicates the data condition of influenza infection in the training and testing process. The viruses in the yellow frame were used for specificity test. The black frame indicates the data condition of influenza infection in the limitation set.

Examination of Training 2

The accuracy of the resigned training data of Training 2 was 98.6%. Next, the resigned testing data was used to examine whether Training 2 was overfitting the training data. The accuracy of the testing data was 94.57%, slightly lower than that of the training data. The influenza experiment data were also tested and the results were compared with Training 1 as tabulated in Table 4. The accuracy of the influenza experiment data for 16 hpi (61.68% v.s. 71.8%), 28 hpi (96.25% v.s. 98.25%), and 40 hpi (77.33% v.s. 87.46%) was all slightly lower than those of Training 1, as shown in Table 4. However, the specificity of the influenza experiment data by Training 2 was slightly better than those of Training 1 for 16 hpi (100% v.s. 97.14%), 28 hpi (100% v.s. 99.18%), and 40 hpi (100% v.s. 99.02%), respectively. Finally, we used the same images of MDCK cells infected with adenovirus, coxsackievirus B3, HSV-1, HSV-2, parainfluenza virus, and RSV to examine whether Training 2 can differentiate these non-influenza-induced CPEs from those by influenza viruses. Most of the classification results of Training 2 were perfect without any image being misjudged as influenza-infected cells, except one image from adenovirus-infected cells (5%). (Table 4).

Comparison of Training 1, Training 2, and visual examination

There are minor differences in the performance between Training 1 and Training 2, regarding the accuracy, specificity, positive predict values, and negative predict values. The chi-square analysis was performed to determine whether the difference is statistically significant. The testing accuracy and negative predict value of the testing data of Training 1 are significantly higher than those of Training 2 (Table 2).

Moreover, Training 1 showed a significant higher recognition rate on earlier recognition than those of Training 2, except the accuracy of 16 hpi negative images of influenza experiment set, which did not achieve statistically significance. Nevertheless, the recognition of the late stage by Training 1 was significantly higher than those of Training 2.

To examine our model can reduce the time of identifying influenza-infected MDCK cells, we compared our model with visual examination. As shown in Table 5, both Training 1 and Training 2 performed significantly higher accuracy than visual examination. The significantly higher value of negative predictive value of Training 1 and Training 2 implicated that our model can recognize influenza-induced CPE images earlier than human.

Table 5

Comparison of Training 1, Training 2 and visual examination.

	Accuracy	PPV a	NPV b	confusion matrix
				Actual	Predicted
					Pos	Neg
Training 1	0.9975***	0.9966*	1***	Pos	300	0
				Neg	1	99
Training 2	0.9457***	0.9673	0.8846***	Pos	356	15
				Neg	12	115
visual examination	0.8025	1*	0.5586	Pos	221	79
				Neg	0	100

Abbreviation: Pos, positive samples; Neg, negative samples; PPV, positive predictive value; NPV, negative predictive value.

a PPV = numbers of true positives/(numbers of true positives+ numbers of false positives). The ideal value of the PPV for a perfect test will be 1, while the worst value will be zero.

b NPV = numbers of true negatives/(numbers of true negatives+ numbers of false negatives). The ideal value of the NPV for a perfect test will be 1, while the worst value will be zero

***: p-value <0.001

*: p-value <0.05

Result of other weights

In the beginning stage of model training, we split 20% of training data as validation dataset. We saved the weights to examine if the accuracy of the validation data would be better than before. We also examined all of the testing data accuracy with those saved weights, but the amount of saved weights was numerous. Thus, we only tried the weights with validation set accuracies above 90%. The results of those weights from two Trainings were almost as great as the weights after 1200 epochs training, as shown in Tables 2 and 4. However, these results were slightly worse than the final weights after 1200 epoch training. Those saved weights were saved within 100 epoch. It probably implied that the model was convergent quickly in the beginning, but 1200 epochs were necessary.

Transfer learning model

Using transfer learning is one way to deal with small amount of training data. In our case, however, the results were not as good as we had expected. The architecture of our transfer learning model is taking whole convnet block of VGG19 [16], followed by four fully-connected layers. The accuracy of the transfer learning model is lower than the model without transfer learning as shown in S2 Table. We suspected that one reason is that the trainable parameters of the transfer learning model are actually much more than previous model.

Discussion

In this study, we used two different training sets to train a deep CNN model which can recognize influenza-induced cytopathic effects. Both of them showed great sensitivity and specificity in recognition of influenza-induced cytopathic effects, whereas the Training 1 displayed slightly higher performance. The only difference between two trainings was the composition of training data, with Training 1 trained by 601 images and the Training 2 by 503, yet with more diverse data. The increment in number of training data was probably contributed by the bias of training where more sample number would provide more information and fewer sample number with less information may be less accurate. Therefore, Training 1 displayed better performance on sensitivity, even on specificity. Secondly, due to the higher amount of positive testing samples, an improved overall accuracy of Training 1 was observed, although Training 1 also tends to misjudge the images of non-influenza-infected samples as influenza-induced cytopathic effects. Nevertheless, as compared to Training 2, Training 1 could statistically significantly recognize the influenza-induced cytopathic effects at earlier time points. Hence, based on the aim for earlier recognition, we concluded that Training 1 is superior and is recommended for future applications.

In this study, we aimed to develop a model which can timely recognize the morphological changes induced by virus infection, which generally takes several days by manual observation in clinical practice. In our analysis, the accuracy for images taken at 28 hpi was 98.25% and 96.25%, respectively, for Trainings 1 and 2. We further examined whether our models could detect cytopathic effects at earlier time points, i.e. 16 hpi. At higher virus inoculation (0.5 M.O.I.), both Trainings exhibited relatively comfortable recognition rates, 88.88% and 67.9%, respectively. At lower virus input (0.05 M.O.I.), the accuracy decreased to 15.92% and 5.3%, respectively. There was a slight gap in the recognition rate of the earlier experiment data. One of the reasons for the deficiency in the recognition of those pictures, especially 0.05 M.O.I., may be that the morphology of cells infected with influenza at a shorter infection period was too similar to those of the mock-infected cells (Fig 2A, 2E and 2F). However, at that stage, no significant difference could be even noticed by manual observation. In fact, in the clinical practice, some laboratories use R-mix to accelerate the process of virus identification. R-mix is a mixture of two cell lines for isolation of viruses. However, unlike manual observation, its results are determined by anti-virus specific antibody and immunofluorescent assay. In a recent study which compared the performance between R-mix and conventional culture, the rates of influenza A and B virus identification were 78% and 91%, respectively, by R-mix as compared to 25% and 39% using conventional culture [17]. Considering our accuracy for images taken at 28 hpi was 98.25% and 96.25%, respectively, for Trainings 1 and 2, our model is superior to R-mix, which has been shown to be better than the conventional culture.

The recognition rate of images taken at 40 hpi was slightly reduced to 87.46% and 77.33% for Trainings 1 and 2, respectively. The reduced recognition was likely contributed to the detachment of virus-infected cells at later time points, where more than 50% of cells developed cytopathic effects. However, the development of virus-induced cytopathic effects is a dynamic process and our model was designed as the screening machine for early recognition of influenza-infected samples. Therefore, the decreased accuracy of positive samples with late stage of influenza-induced cytopathic effects would not reduce the advantages of our model, since our model could identify the cytopathic effects in positive samples at earlier time points.

In this study, we used the PR8 virus strain to conduct the experiment. The PR8 virus strain is a laboratory reference strain for influenza study and the CPE induced by PR8 is representative to most influenza viruses. To examine that our model can recognize influenza CPE induced by different influenza virus strains, we collected about 120 image of MDCK cells infected by pandemic H1N1/09, H3N2, and influenza B at 16, 25, 28, and 40 hpi, respectively. Most results of recognition were as good as our testing data and experiment data, even better (S4 Table). The slight difference might be contributed by the different growth kinetics of influenza strains.

Compared to CNN models used in general, our models were relatively simple, consisting of only ten hidden layers. We did not choose a deeper model with the pretraining process because we believe the calculation would be more efficient with a non-complex model. Although we trained the model with a relative minute quantity of 1200 epochs, our model still showed remarkable accuracy in recognition of influenza-induced cytopathic effects. First, we only classified our data into two categories. For clinical diagnosis, only positive or negative readouts are required for final reports, rather than the numerical percentages where we believe a complex model would require more classification with more training epochs and more training data. Second, we utilized the original size of the photos, 1024x1360 pixels. Many reports described their model with a smaller size input [18,19], and some researches even adjusted the images to fit the pretraining model [5,6,8]. By downscaling the images, it may lead to loss of some features and in turns cause more difficulty in model training. The original size of the images is believed to provide more information to the model. To examine this possibility, we also tried to train the same model with a quarter size of the original pictures. Nevertheless, the training results were not prominent (S5 Table). It suggests that a smaller size of input does affect the complex of the models.

Considering the input size and patterns of influenza-induced cytopathic effects, we used larger sizes of convolutional filters and max-pooling kernels than common situations. Based on the filer size of the first convolutional layer we used, it is estimated that four to nine cells can be covered, which should provide sufficient information to recognize the features of cytopathic effects. The increased kernel sizes of max-pooling can help to describe the differences regarding the location of images, considering the patterns of influenza-induced cytopathic effects might include the empty holes in the cell layer. These strategies in combination could reduce the complexity of the model without transfer learning, while maintaining the great performances. We also compared our results with those from a model applying transfer learning. The accuracy of the model without transfer learning was better by at least 20%. Therefore, we established a simple model with small amounts of training data, yet having great performance.

Our model has the advantage in shortening the screening time for virus isolation. Indeed, it is expected that a better discrepancy could be achieved at the later time point when a higher percentage of cytopathic effects develop. Therefore, one of the potential pitfalls for our model would be higher false positive ratios at earlier time point. If we assume the percentage of positive samples in the clinical laboratory is 30%, we can roughly estimate the false positive ratio. For cell culture at 25 hours post infection, the sensitivity of Training 1 was 100% and the specificity was 99%, which lead to a false positive ratio of 2.28%. The amount of misjudgment was acceptable because a subsequent immunofluorescence assay will be performed in the clinical diagnosis procedure to confirm the positivity of the observation.

In summary, we constructed a ten-layer convolutional neural network to differentiate influenza-induced cytopathic effects from normal MDCK cells. The model shows great performance both in accuracy and specificity with two trainings, although the recognition of initial cytopathic effect at early stages is not as impressive and can be further strengthened. Besides influenza virus, many other viruses can induce variant patterns of cytopathic effects on specific cells. We will try to train other models with the same structure to recognize different kinds of cytopathic effects. We also believe that our model structure could be applied on other similar imaging approaches, such as images of fluorescence assay and immunohistochemistry.

Materials and methods

Cells and viruses

MDCK cells (ATCC CCL-34, USA) were cultured in Modified Eagle’s Medium (MEM; 61100–61, Gibco, USA) supplemented with 10% fetal bovine serum (FBS; 10082–147, Gibco, USA) and penicillin G sodium 100 units/mL, streptomycin sulfate 100 ug/mL and amphotericin B 250 ng/mL (antibiotic-antimycotic; 15240–062, Gibco, USA) at 37°C with 5% CO2. The viruses used in this study included influenza laboratory strain PR8 virus [A/Puerto Rico/8/34 (H1N1)], pandemic H1N1 virus [A/California/07/2009 (H1N1)], H3N2 clinical isolate [A/Hong Kong/4801/2014 (H3N2)-like], influenza B clinical isolate [Victoria lineage], adenovirus, coxsackievirus B3, parainfluenza virus, respiratory syncytial virus (RSV), herpes simplex virus type 1 (HSV-1) and herpes simplex virus type 2 (HSV-2). Influenza virus and parainfluenza virus were amplified in MDCK cells, and the infected cells were maintained in the MEM with 2ug/mL tosylphenyl alanyl chloromethyl ketone (TPCK) trypsin (T1426, Sigma-Aldrich, USA). Other viruses were amplified in HEp2 or MK2 cells, and the infected cells were maintained in the MEM with 2% fetal bovine serum. The harvested virus supernatant was titrated by the plaque assay and stored at -80°C for subsequent experiments.

Virus infection

MDCK cells (2x106 cells/well) were infected with influenza virus, adenovirus, HSV-1, HSV-2, coxsackievirus B3, parainfluenza virus, and RSV for one hour before the addition of the MEM supplemented with 2ug/mL TPCK trypsin. The multiplicity of infection (M.O.I.) of influenza virus used in this study was 0.01, 0.05, 0.1, 0.5, 1 and 2, respectively. The cytopathic effects were recorded by the Olympus IX71 microscope at 16 and 48 hours after infection.

Datasets and taxonomy

The dataset was split into four sets, training set, validation set, testing set, and experiment set (including influenza experiment set and other viruses set). The cytopathic effects were monitored at 16 hours post infection. Six independent influenza virus infection experiments were conducted and the photos taken in the last two experiments were used as the testing set and the influenza experiment set. All of the training data (including training set and validation set), testing data (testing set) and influenza experiment data (influenza experiment set) included photos of uninfected MDCK cells and influenza-infected MDCK cells. The training, validation, and influenza experiment datasets classify photos into two categories. Photos taken in the well of uninfected MDCK cells were labeled as negative samples; the others with confirmed cytopathic effects were marked as positive samples. The images of other viruses set were all taken at 40 hours post infection, and they were not further labeled.

Data collection

The images of cytopathic effects were recorded by the Olympus IX71 microscope. After the fields were selected, photos were taken continuously and refocused randomly. Because not all fields of influenza virus infected cells contained the cytopathic effects, we only took the fields with cytopathic effects as positive samples. The fields of uninfected cells were taken randomly as negative samples. For the independence between data sets, images of the training, testing set, and influenza experiment set were not taken from the same infection experiment.

To train the machine, we selected 601 photos, including 447 from influenza-infected MDCK cells and 154 from uninfected MDCK cells, respectively. After that, additional 400 pictures were used to test the model. Additionally, a total of 1190 images were taken as experiment set to evaluate the limit of the model, including 824 from influenza-infected MDCK cells and 366 from mock-infected MDCK cells. The image numbers for the other viruses set were 120. A total of six non-influenza viruses were included and twenty pictures from each virus infected cells were included for analysis. The output of other viruses set was negative. The detail would be revealed in the result section. If the non-influenza virus induced cytopathic effects on MDCK cells, these areas would be chosen for image taken. Otherwise, pictures were taken randomly. All of the original images were color with 1024 ×1360 pixels.

Model architecture

Our deep convolutional neural network model consists of nine hidden layers, as shown in Fig 5. The first six layers are convolutional network and the following three layers are fully-connected network. The output layer is fed to a softmax generating two categorization labels.

Fig 5

Architecture of the Convolutional Neural Network model.

There are three convolutional layers with max pooling and four fully-connected layers. The numbers beside the input indicate the size of the image. The numbers of the convolutions and pooling are the size of output from previous layer. The numbers in the cube indicate the size of convolutional filter and pooling kernel. The bold numbers are the number of the neurons in the fully-connected layers.

We use three convolutional layers, and each is followed by a max-pooling layer. Stride factor s in each convolutional layer is one. The max-pooling layer is as large as kernel size. Each input image is a grayscale image of size 1024 × 1360. The first convolutional layer has 20 kernels of size 7 × 7 and following a max-pooling size of 10 × 10. The second convolutional layer filers the input from the previous layer with 25 kernels of size 5 × 5. Then, the second kernel size of max-pooling is 6 × 6. The third convolutional layer has 30 kernel of size 3 × 3 and following is a max-pooling layer of size 6 × 6. Then, each fully-connected layers are 100 neurons. The reason we choose somewhat large size of both filter and max-pooling kernel than traditional approach is because our input image is large. In the 3-layer fully-connected network, each has 100 neurons. Finally, there are two outputs and we apply softmax function to get the final result. The code and data can be downloaded from https://github.com/b96404011/influenzaCPE_CNN.

Training

Our CNN model was trained with back propagation. The weights of the CNN model are initialized randomly. All layers of the neural networks used the same global learning rate of 0.0001 because our model scale is small, only ten layers. We used Adam with the preset parameter value, namely, β1 = 0.9, β2 = 0.999, epsilon = 0, and decay = 0. We used TensorFlow deep learning framework to train, validate, and test our CNN model.

Transfer learning

VGG19 [16] is a popular convolutional neural network model with 19 layers which is often used in transfer learning model. We took the whole convolutional part of VGG19 and connected with three fully-connected layers, each of 100 neurons. We upsample input images to 1024x1360x3 to fit the input of VGG19. The weights of the network from VGG19 are fixed (from the original VGG19). Only weights of newly added fully-connected layers are trainable. The other statuses are the same as our own model. We used Adam as the optimizer to train the neural network.

Statistical analysis

All statistical analyses were performed using SPSS software version 16.0 (SPSS Inc., Chicago, IL). Categorical variables were compared using χ2 or Fisher’s exact test. A p<0.05 was considered statistically significant.

Information Table for the Training 1 Initial Testing Data Set.

(DOC)

Click here for additional data file.

Transfer Learning Model Comparison of Training 1 and Training 2.

(DOC)

Click here for additional data file.

Comparison of Training 1 and Training 2 with 1200 epochs weights on other viruses infected images at earlier time point.

(DOC)

Click here for additional data file.

Comparison of Training 1 and Training 2 with 1200 epochs weights on other strains of influenza virus data.

(DOC)

Click here for additional data file.

CNN Model with quarter size images.

(DOC)

Click here for additional data file.

6 Jan 2020

Dear Dr Chang,

Thank you very much for submitting your manuscript 'Differentiation of Cytopathic Effects (CPE) Induced by Influenza Virus Infection Using Deep Convolutional Neural Networks (CNN)' for review by PLOS Computational Biology. Your manuscript has been fully evaluated by the PLOS Computational Biology editorial team and in this case also by independent peer reviewers. The reviewers appreciated the attention to an important problem, but raised some substantial concerns about the manuscript as it currently stands. While your manuscript cannot be accepted in its present form, we are willing to consider a revised version in which the issues raised by the reviewers have been adequately addressed. We cannot, of course, promise publication at that time.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Your revisions should address the specific points made by each reviewer. Please return the revised version within the next 60 days. If you anticipate any delay in its return, we ask that you let us know the expected resubmission date by email at ploscompbiol@plos.org. Revised manuscripts received beyond 60 days may require evaluation and peer review similar to that applied to newly submitted manuscripts.

In addition, when you are ready to resubmit, please be prepared to provide the following:

(1) A detailed list of your responses to the review comments and the changes you have made in the manuscript. We require a file of this nature before your manuscript is passed back to the editors.

(2) A copy of your manuscript with the changes highlighted (encouraged). We encourage authors, if possible to show clearly where changes have been made to their manuscript e.g. by highlighting text.

(3) A striking still image to accompany your article (optional). If the image is judged to be suitable by the editors, it may be featured on our website and might be chosen as the issue image for that month. These square, high-quality images should be accompanied by a short caption. Please note as well that there should be no copyright restrictions on the use of the image, so that it can be published under the Open-Access license and be subject only to appropriate attribution.

Before you resubmit your manuscript, please consult our Submission Checklist to ensure your manuscript is formatted correctly for PLOS Computational Biology: http://www.ploscompbiol.org/static/checklist.action. Some key points to remember are:

- Figures uploaded separately as TIFF or EPS files (if you wish, your figures may remain in your main manuscript file in addition).

- Supporting Information uploaded as separate files, titled Dataset, Figure, Table, Text, Protocol, Audio, or Video.

- Funding information in the 'Financial Disclosure' box in the online system.

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see here.

We are sorry that we cannot be more positive about your manuscript at this stage, but if you have any concerns or questions, please do not hesitate to contact us.

Sincerely,

Amber M Smith

Guest Editor

PLOS Computational Biology

Rob De Boer

Deputy Editor

PLOS Computational Biology

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors Chang et al. described a CNN that classifies samples into two categories (infected with flu vs not infected with flu) based on images of cells cultured with sample after various times. Two training/validation sets have been constructed and results were similar. The results appear to be robust and have clinical application.

Major issues:

The training data set (images, and their labels) are not provided.

It would be better if the code and trained model is also available.

When images of cells infected with other viruses are used as the input, what is the output? The data in Table 4 indicated that for RSV, training 2, 1200 epochs, accuracy is 1 but the best one is 0.35. What does the column "best one" indicate? Why is the best one is 0.35 and 1200 epoc is 1? Should the best one be better than 1200 epochs (most of the rows are but not always)?

Based on the images provide for the other virus-infected cells, some of them have apparent changes in cell morphology. If they are not classified as influenza damage, are they classified as no damage? (because the CNN only has two outputs).

The manuscript can use some better organization regarding the introduction of training1 vs. training2. Training 2 data appeared before they were introduced, I suspect this was a response to a prior review of some sort.

Minor issues

The manuscript could use some editing of the English Language.

For example:

Page 10: line 145: were differentiated correctly: Do the author mean identified?

Page 19: line 220: resigned training data? What is "resigned" referring to?

Reviewer #2: The manuscript by Want et al. describes a deep convolutional neural network (CNN) approach to detect influenza induced cytopathic effects from microscopic images of influenza infected MDCK cells. The aim is to develop a fast and accurate tool for rapid influenza diagnoses in clinics. Deep learning approaches have been proven to be extremely powerful for identifying objects of interests from images in the past decade. Thus, the approach to the problem was well chosen in the study. The authors showed that trained model can achieve a very high percentage of correct recognition (99.75%). Although the results look convincing, my enthusiasm about the study is dampened by a couple of major concerns and some minor concerns.

First of all, the images used to train the model are images of cells infected by a single virus strain (PR8). While the authors tested the model against a variety of other non-influenza viruses, I think it is critical/essential to test how the model works with other influenza strains. This is because we usually do not know the virus strain that a person is infected with as a priori. The model is useful only when it can correctly predict the presence of influenza virus irrespective of the strain a cell is infected with.

Second, it is not clear to me how this method is compared with other method in terms of accuracy and time needed for diagnoses. Although the authors discussed and compared the performance of their model with the performance of R-mix, the comparison does not seem to be fair, because the tests were done using different sets of data and images collected from different experiment with different protocols. Furthermore, the CNN model may be an automation method and a useful alternative to visual examination, for example by a medical technologist. However, it is not clear to me how much (quantitative) benefit one gains as compared to visual examination, e.g. how much more accurate the CNN model performs compared to visual examination and how much time it saves. Overall, the advantages and significance of this approach in terms of clinical use is not clearly discussed and highlighted. This really limit the enthusiasm about the proposed approach.

Third, in Fig 4, the authors used the model to test against non-influenza viruses and concluded that ’no cytopathic effect was observed in MDCK cells infected by other viruses’. It is not clear to me what analysis was exactly performed. Did the author use the model trained on influenza infected and mock MDCK cells to classify images of cells infected by non-influenza viruses; or the authors retrained the model including the images of cells infected by non-influenza viruses? In the images with other virus infections, is that indeed ‘there is no cytopathic effects’; or other viruses cause cytopathic effects that are different from influenza? I also notice that the images were all taken at 40phi (for other viruses). What about other time points?

Some minor points below:

Lines 186- 201, it is not clear to me how the accuracy of prediction of images taken at different time points were done. Did the authors retrain the model; or just used the model trained on the Training dataset described in the earlier text?

Line 194-195, Can the authors test why the accuracy drops at 40hpi? It is not sufficient to speculate the cause.

Line 198, typo ‘act’ -> ‘fact’?

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: No: The training data set (images and labels) are not provided.

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

8 Mar 2020

Submitted filename: Reviewer_Plos_com v2.docx

Click here for additional data file.

26 Mar 2020

Dear Dr. Chang,

Thank you very much for submitting your manuscript "Differentiation of Cytopathic Effects (CPE) Induced by Influenza Virus Infection Using Deep Convolutional Neural Networks (CNN)" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Amber M Smith

Guest Editor

PLOS Computational Biology

Rob De Boer

Deputy Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors need to incorporate the reply to reviewer's question in the revised manuscript. Most of the answers to Reviewer 1 were not reflected in the revision. The authors stated that they changed "best one" to "saved weights". But in table 2, they still used "best one". Also although the code is provided in github (much appreciated), the data is provided through a link. The link time out when I tried to download the data.

Reviewer #2: All my concerns are appropriately addressed. I appreciate the author's efforts to collected additional data (images) of cells infected by other strains. This added a lot of robustness to the model.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: No: As stated, the data were not available.

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Ruian Ke

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at .

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, PLOS recommends that you deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see

8 Apr 2020

Submitted filename: Reviewer_Plos_com R2.docx

Click here for additional data file.

16 Apr 2020

Dear Dr. Chang,

We are pleased to inform you that your manuscript 'Differentiation of Cytopathic Effects (CPE) Induced by Influenza Virus Infection Using Deep Convolutional Neural Networks (CNN)' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology.

Best regards,

Amber M Smith

Guest Editor

PLOS Computational Biology

Rob De Boer

Deputy Editor

PLOS Computational Biology

***********************************************************

30 Apr 2020

PCOMPBIOL-D-19-01629R2

Differentiation of Cytopathic Effects (CPE) Induced by Influenza Virus Infection Using Deep Convolutional Neural Networks (CNN)

Dear Dr Chang,

I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript.

Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work!

With kind regards,

Laura Mallard

PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom ploscompbiol@plos.org | Phone +44 (0) 1223-442824 | ploscompbiol.org | @PLOSCompBiol