Prediction of Patient's Adherence to the Post-Intubation Tracheal Stenosis Follow-up Plan in Iran: Application of two Data Mining Techniques.

BACKGROUND: Timely diagnosis of post-intubation tracheal stenosis (PITS), which is one of the most serious complications of endotracheal intubation, may change its natural history. To prevent PITS, patients who are discharged from the intensive care unit (ICU) with more than 24 hours of intubation should be actively followed-up for three months after extubation. This study aimed to evaluate the abilities of artificial neural network (ANN) and decision tree (DT) methods in predicting the patients' adherence to the follow-up plan and revealing the knowledge behind PITS screening system development requirements.
MATERIALS AND METHODS: In this cohort study, conducted in 14 ICUs during 12 months in ten cities of Iran, the data of 203 intubated ICU-discharged patients were collected. Ten influential factors were defined for adherences to the PITS follow-up (P<0.05). A feed-forward multilayer perceptron algorithm was applied using a training set (two-thirds of the entire data) to develop a model for predicting the patients' adherence to the follow-up plan three months after extubation. The same data were used to develop a C5.0 DT in MATLAB 2010a. The remaining one-third of data was used for model testing, based on the holdout method.
RESULTS: The accuracy, sensitivity, and specificity of the developed ANN classifier were 83.30%, 72.70%, and 89.50%, respectively. The accuracy of the DT model with five nodes, 13 branches, and nine leaves (producing nine rules for active follow-up) was 75.36%.
CONCLUSION: The developed classifier might aid care providers to identify possible cases of non-adherence to the follow-up and care plans. Overall, active follow-up of these patients may prevent the adverse consequences of PITS after ICU discharge. Copyright

INTRODUCTION

Although endotracheal intubation is a life-saving procedure, it may have significant long-term effects on the individual’s communication, swallowing, and breathing (1). Tracheal stenosis, as one of the most severe complications of endotracheal intubation, is still a challenging problem for surgeons. Various factors may contribute to tracheal stenosis, including prolonged intubation as the most common one (2). The leading cause of post-intubation tracheal stenosis (PITS) is tracheal wall ischemia due to direct pressure of the cuff and/or tip of the endotracheal tube over the mucosa, besides forceful intubation by inexperienced medical staff, especially in critically injured patients (3). Ischemia can trigger an inflammatory process, mucosal edema, granulation tissue formation, fibrosis, and finally cartilage destruction (3,4).

The incidence of tracheal stenosis following tracheostomy and laryngotracheal intubation ranges from 0.6% to 21% and 6% to 21%, respectively (5,6). However, different rates have been reported in different countries due to different local risk factors. These patients frequently undergo bronchoscopy and dilation in repeated admissions, even though the optimal treatment involves airway resection and reconstruction of stenosis (4,7). Several studies have reported that patients discharged from intensive care units (ICUs) may experience tracheal stenosis from several hours after intubation up to two years afterward (1). However, in the majority of cases (75%), tracheal stenosis appears three months after extubation (4). These patients (44%) may be misdiagnosed as asthmatic cases (8,9). Also, delay in diagnosis can decrease the probability of treatment with good outcomes (10). Therefore, early diagnosis is essential for effective treatment and prevention of further physical, psychological, and economic burdens on both patients and the healthcare system.

Non-compliance with the follow-up plan in ICU-discharged patients is a major cause of undiagnosed PITS, which may result in emergent tracheostomy and even death. Among ICU survivors, patients with a history of intubation should be followed-up for PITS six to 12 weeks after discharge (11). Although some factors, such as nutritional status, respiratory and cardiac consequences, neuropathy, and other physical problems, are considered in ICU-discharged patients by follow-up clinics, no follow-up plan has been established for PITS screening yet. Overall, providing the necessary follow-up services to all patients may require remarkable resources. Also, screening of high-risk cases that refuse follow-up and checkup can be cost benefited. To the best of our knowledge, there is no standard tool to identify patients with PITS who fail to visit the hospital for follow-ups three months after ICU discharge. An effective tool can be a computational model to predict high- and low-risk patients regarding compliance with the follow-up plan, based on their historical data.

Classification tasks have been extensively used in medical sciences (12–14). Advanced classification models play a major role in the extraction of medical knowledge, treatment planning, intelligent monitoring, and patient management (15,16). Overall, different machine learning methods have different predictive capabilities, based on particular outcomes and information. Artificial neural networks (ANNs) and decision trees (DT) are two frequently used machine learning methods for classification purposes. These learning methods have been successfully developed for medical purposes (17,18). Besides good performance, a decision tree can produce rules to explicitly reveal the hidden knowledge behind data (19). ANN also has an operational profile for accurate prediction (12). However, no single learning algorithm can uniformly outperform other algorithms over all datasets (12,17,20). Therefore, this study aimed to evaluate two classification algorithms for predicting adherence to routine follow-up plan three months after ICU discharge in intubated patients.

This study aimed to predict the adherence of ICU-discharged patients to the follow-up plan for PITS screening.

MATERIALS AND METHODS

Subjects and data collection

This cross-sectional multi-center study, as part of a national cohort study, was conducted in 14 ICUs in ten cities of Iran over ten months (from September 2014 until June 2015). The ICU-discharged patients were intubated during their hospital stay. Symptomatic and asymptomatic patients were followed-up for at least three months after extubation. ICU-admitted patients with more than 24 hours of endotracheal intubation were included in this study. To enhance the follow-up rate, an educational program was designed, including verbal and written educational materials upon discharge; the materials were also repeated in a telephone-based follow-up almost three months after discharge. Through written (at home) and oral (upon discharge) education, the patients were trained about tracheal stenosis, complications of delayed diagnosis, and follow-up date, place, and physician in charge.

In the follow-up sessions, asymptomatic patients underwent a pulmonary function test (PFT) to evaluate the flow-volume loop. All patients prone to tracheal stenosis (i.e., patients with abnormal PFT and symptomatic patients) underwent rigid bronchoscopy for a definite diagnosis. In the telephone-follow-ups, the patients were re-trained, and their knowledge about PITS and the follow-up process upon discharge was examined. By using a valid form and asking questions from the patients, the collected data were classified into 18 variables, according to Chi-square and Fisher’s exact tests for model development to predict the patients’ adherence to the follow-up plan. The level of statistical significance was less than 0.05. The institutional review board of NRITLD approved this study.

Applied algorithms

ANN model

A neural network is a set of connected input/output units, where each connection has a certain weight. It contains an input layer, an output layer, and hidden layers. The hidden layers have an arbitrary number of nodes, which makes it easier to regulate the weight of each node to satisfy the input-output relationship. A multi-layer neural network consists of many units (neurons), joined in a pattern of connections. The units in a net are usually segregated into three classes: 1) input units which receive information for processing; 2) output units which present the results of processing; and 3) hidden units as in-between units. Feed-forward ANNs allow signals to travel in only one direction, that is, from input to output (21). This technique is well-known for highly accurate classifications, where the output is a class that can be considered as a risky or non-risky case of follow-up for PITS screening.

DT model

A DT is a tree-like structure, which starts from root attributes and ends with leaf nodes. A decision tree has several branches with different attributes; the leaf node on each branch represents a class or a type of class distribution. This tree is generated according to the information gain measure (22). The DT algorithms describe the relationships between attributes, besides the relative importance of attributes; also, human-understandable rules can be extracted from the trees. Generally, both learning and classification steps of DT induction are fast (22). In the present study, the well-known C5.0 algorithm was used, which classifies instances by sorting them based on their feature values. Each node in a DT represents a feature in an instance to be classified, and each branch represents a value that the node can assume. Cases are classified starting at the root node and are sorted based on their feature values. The output of the system is a tree-like structure that creates rules with a high level of understandability (22).

Model development

In the first step, data cleaning was conducted for an available database, containing 203 instances, by removing the outliers. The discretization method, which converts a continuous variable (X=age) into a discrete variable with three categories (<25, 25–55, and >55 years), was used in this study. Data of the selected attributes were used for model training and development; the attributes and their meanings are described in Table 1. These attributes were selected by Chi-square and Fisher’s exact tests at P<0.05. Two classifiers, including DT and multilayer perceptron (MLP), were applied for the patient dataset. For model evaluation, criteria, including the model accuracy, sensitivity, and specificity, were assessed. To evaluate the model accuracy, a testing set was used, as it is much more valuable for appraising the model accuracy.

Table 1.

The analysis of the patients’ characteristics and follow-up visit using chi-square test

Variables		Successful follow-up		p-value
		Yes: n (%)	No: n (%)
Gender	Male	96(77.4%)	28(22.6%)	>0.9999*
	Female	60(76.9%)	18(23.1%)
	Under 25	48(39%)	12(15.4%)	<0.0001**
Age	25–55	64(52%)	43(55.1%)
	55 and more	11(8.9%)	23(29.5%)
	Employee	54(47.8%)	39(52%)
Occupation	Unemployed	37(32.7%)	26(34.7%)
	Retired	5(4.4%)	6(8%)	0.172*
	Student	17(15%)	4(5.3%)
Marital status	Single	55(50.5%)	18(25.7%)	0.001*
	Married	54(49.5%)	52(74.3%)
	Diploma	28(28.3%)	12(20%)	0.069**
Educational level	University	24(24.2%)	9(15%)
	Under diploma	40(40.4%)	28(46.7%)
	Literate	7(7.1%)	11(18.3%)
	Patient	36(29.5%)	16(20.5%)	0.006*
Interviewee	Parent	32(26.2%)	9(11.5%)
	Wife or husband	10(8.2%)	14(17.9%)
	Others	44(36.1%)	39(50%)
Reason for hospitalization	Car accident	43(35.2%)	17(21.8%)	0.002**
	Suicide	41(33.6%)	17(21.8%)
	Others	38(31.1%)	44(56.4%)
Written patient education	Yes	73(61.3%)	26(33.3%)	<0.0001*
	No	46(38.7%)	52(66.7%)
Oral patient education	Yes	83(68.6%)	35(46.7%)	0.003*
	No	38(31.4%)	40(53.3%)
Patient’s knowledge of Tracheal stenosis	Yes	49(41.5%)	20(26%)	0.032*
	No	69(58.5%)	57(74%)
Patient’s knowledge about warning symptoms	Yes	35(29.9%)	17(22.4%)	0.319*
	No	82(70.1%)	59(77.6%)
Symptoms before or at Interview	Yes	74(60.7%)	52(66.7%)	0.454*
	No	48(39.3%)	26(33.3%)
Information about management protocol	Yes	30(25.6%)	7(9.3%)	0.005*
	No	87(74.4%)	68(90.7%)
Information about the date of F/U	Yes	76(62.8%)	26(34.2%)	<0.0001*
	No	45(37.2%)	50(65.8%)
Information about the place of F/U	Yes	74(63.8%)	23(30.7%)	<0.0001*
	No	42(36.2%)	52(69.3%)
Information about his/her physician	Yes	48(41%)	11(14.7%)	<0.0001*
	No	69(59%)	64(85.3%)
Interview time	Early	39(31.5%)	33(42.3%)	0.130*
	Late	85(68.5%)	45(57.7%)

: Fisher’s Exact test

: Chi-Square test, significant test: p value< 0.05

We aimed to present a predictive model to determine whether an ICU-discharged patient with intubation would adhere to the follow-up plan to receive the necessary care over the next three months after discharge. For this purpose, two classification models, based on DT and ANN, were examined. To evaluate the model accuracy and avoid over fitting, the entire dataset was used for training (2/3 of dataset), as well as testing and validation (1/3 of dataset for both), based on the holdout method.

Model assessment

To evaluate the DT and MLP classifiers, the model accuracy, sensitivity (Pr (+|D), and specificity (Pr (-|∼D) were measured, where “D” denotes not attending the follow-up; “∼D” denotes attending the follow-up; “+” indicates risky follow-up cases according to the model prediction (suspected of not attending the follow-up); and “-” indicates cases that would attend the follow-up according to the model prediction. The overall accuracy of the model was determined based on the proportion of correctly classified cases to total cases. Also, the area under the receiver operating characteristic (ROC) curve was determined by plotting the true positive rate against the false-positive rate (23). The model accuracy, which was determined by applying the test dataset, was important for evaluating the model quality, as it was calculated by using independent data that were not used for developing the model. Also, to test the generalization of DT and MLP classifiers, the validation dataset was used, and the results indicated the model’s ability to identify new cases and classify them correctly.

RESULTS

Selected features

In this study, 203 extubated patients in 14 ICUs across the country, with the median age of 34 years (range: 13–88 years), were educated upon and after discharge. The patients’ characteristics, knowledge, and follow-up information were documented in the follow-up evaluation form, which had been previously prepared and validated (24). During telephone follow-ups, the patients were asked about their symptoms after discharge. The patients reported dyspnea (n=73, 36.7%), stridor (n=43, 21.6%), hoarseness (n=64, 32.2%), and cough (n=72, 36.4%). There was no significant difference between the patients’ symptoms and their participation in the follow-up (P>0.05). Table 1 presents the results of Chi-square and Fisher’s exact tests for the independent variables and successful follow-up. Variables which were significantly correlated with a successful follow-up were selected for model development.

Model development results

An ANN model was developed in this study (Figure 1). Table 2 and Figure 2 display the model accuracy, sensitivity, and specificity. Also, calculations are addressed in Section 2.2.4.

Figure 1.

Performance trend of train, test and validation sets in 21 epochs using the developed model by ANN, using hold out method

Table 2.

Model accuracy, sensitivity, and specificity by Artificial Neural Network Method

Model Type developed by ANN	Instances number	Accuracy (%)	Sensitivity (%)	Specificity (%)
Model fitness by training set	141	82.50	67.20	92.90
Model accuracy by testing set	31	83.30	72.70	89.50
Model validity by validation set	31	86.70	77.80	90.50
Total	203	83.30	69.20	92.0

Figure 2.

Receiver operating characteristic (ROC) curves of the model developed by testing, training, validation sets plotted sensitivities and (1- specificity) to predict successful follow-up cases

DT model

Using the Weka package, the C5.0 DT model was developed and validated. Table 3 shows the results regarding the accuracy, sensitivity, and specificity of the DT model. The extracted rules from the tree are shown in Figure 3. For each rule in branches, the accuracy is shown as a division of accurately classified cases by misclassified cases. The DT model with five nodes, 13 branches, and nine leaves produced nine rules (75.36% accuracy).

Table 3.

Model accuracy, sensitivity, and specificity of Decision Tree Results

Model Type developed by DT	Instances (N)	Accuracy (%)	Sensitivity (%)	Specificity (%)
Model fitness by training set	141	75.36	74.0	91.0
Model accuracy by testing set	31	74.36	69.0	90.0
Model validity by validation set	31	77.36	71.0	92.0
Total	203	75.69	71.30	91.0

Figure 3:

Decision tree root node, branches, and leaves for successful follow-up presenting nine main rules:

If place info is yes THEN follow-up result is yes (103.09/24.52)

If place info is NO and Doctor info is yes THEN follow-up result is yes (6.36/0.13)

If place info is No AND Doctor info is NO AND age is => 55 years THEN follow-up result is NO(18.4/3.4)

If place info = No AND Doctor info = NO AND Age=<25 THEN follow-up result= yes (24.34/7.9)

If place info= No AND Doctor info= NO AND Age=25–55 AND interview time = Early THEN follow-up result = No(13.94/2.9)

If place info =No AND Doctor info = No AND Age =25–55 AND interview time=Late AND interview patient THEN follow-up result= yes(14.11/4.9)

If place info =No AND Doctor info = No AND Age =25–55 AND interview time=Late AND interview parents THEN follow-up result= yes(3.54/1.0)

If place info =No AND Doctor info = No AND Age =25–55 AND interview time=Late AND interview spouse THEN follow-up result= No(4.05/0.05)

If place info =No AND Doctor info = No AND Age =25–55 AND interview time=Late AND interview Others THEN follow-up result= No(15.17/5.68)

DISCUSSION

According to the present results and experts’ opinions, the features of ten patients were selected as inputs for the model development process, as they were more effective than others for a successful follow-up (P<0.05). Generally, different machine learning methods with different advantages and disadvantages are available, and selection of the proper algorithm depends on the task at hand (12). In this study, algorithms, which were based on externally supplied instances, were used to test general hypotheses through developing a concise model of class label distribution according to the applied features. The resulting classifier was then used to assign class labels to test instances, where the values of the predictive features are known, whereas the value of the class label is unknown. In this experiment, two well-known methods that adequately perform this task were selected.

Neural networks tend to perform very well when dealing with multi-dimensional and continuous features, such as age of the patients. On the other hand, logic-based systems tend to perform better when dealing with discrete/categorical features, such as the patient’s knowledge of the place of follow-up visit or their doctor (Yes/No). Since our data included both continuous and categorical variables, and almost a large sample size was available, we selected the ANN and DT models as two separate algorithms.

Our findings of the constructed DT showed that younger patients (<25 years) tended to adhere to the follow-up plan more than older patients (>55 years). However, those who were aged between 25 and 55 years attended their follow-up visits, depending on the time of the interview. Early interviews did not positively affect the follow-up, and late interviews depended on the person being interviewed. For late interviews, the patients (25–55 years) and their parents had to come to the hospital on time; otherwise, no suitable result was obtained. Marriage was also an important factor, as late interviews with the partners of married patients did not result in satisfying outcomes compared to patients.

Moreover, patients who were admitted to the ICUs because of a car accident or suicide attempt were significantly more likely to complete their follow-ups. Also, the patient’s knowledge of the date, place, and name of the physician in charge of the visit was significantly correlated with adherence to the follow-up plan, based on the DT model. The extracted tree represented two main findings, that is, the most influential variables (tree nodes) to predict the dependent variable (patient’s adherence to the follow-up plan) and the non-linear relationship between the independent variables and the outcome variables (branches and leaves). As can be seen in Figure 3, the patient’s knowledge of the follow-up place, name of the physician in charge, patient’s age, interview time, and interviewee were important in adherence to follow-up.

Considering the heuristic nature of the empirical method, several DTs were constructed with different input-output arrangements, and the selected tree was the one with the least error and the most reasonable distinctiveness rules (23). Another aspect of the DT model was the relationship between the nodes based on branches, producing leaves that could be used to identify follow-up failure (23). For example, rules No. 3 and No. 4 suggest that elderly patients who do not have any information about the place of follow-up visit or the physician in charge, will not return for the follow-up, whereas younger patients (≤25 years) will return on their own. Additionally, the interview time was another decision-making criterion to screen failed cases of follow-up after discharge, as middle-aged patients, who were interviewed after the appointment, behaved differently based on the telephone interviews.

The ANN model outperformed the C5.0 DT model, probably due to the particular algorithm and architecture of ANN. This machine learning technique exploits a weighting approach for the input variables relative to their importance. At least three primary layers (input, hidden, and output) are included in the structure to produce the best model (25). On the other hand, in the DT structure, there is only one input node at the top of the tree, called the root node, whereas in the ANN configuration, there is a greater complexity, as there are numerous neurons and different entries equal to the number of independent variables, which should be weighted and then entered in the hidden layer for further calculations (25).

The learning process in ANN is continuous, unlike DT, as it adjusts and tests different weights during the learning process to satisfy all input and output relations. Considering the advantages and disadvantages of these two methods, it is suggested to develop a system with the advantages of both techniques. The resulting system can be more efficient than the current conventional ones, which involve either a face-to-face request from the patient to re-visit the clinic using educational materials or making phone calls to every individual patient (26,27). According to some studies, about 50% of interventions are ineffective in increasing the patient’s communication and response rate, regardless of the method used (28).

In 2002, Broomhead and Brett (29) reported the need for a clinical follow-up center to prevent all post-ICU discharge complications; however, they provided no guideline to put this idea into practice. For active follow-up of ICU-discharged patients, it is necessary to use ANN for case detection and DT for the detailed follow-up method. Since each method has its limitations (e.g., black-box modeling approach in ANN and lower accuracy in DT), we used both methods for more accurate results. The output of these two models may be used to develop a monitoring system that addresses risky cases of non-adherence to follow-up and follows them up electronically. This can be a part of a personalized management system that classifies intubated ICU-discharged patients into positive and negative groups returning for PITS screening. Further research may clarify this point, based on the specific characteristics of the patients.

This study had some limitations. Since we used a restricted number of records to develop the DT and ANN models, the level of sensitivity for diagnosing risky cases of non-adherence to follow-up visits was high; therefore, it is recommended to use a much richer dataset to improve the model measurements. On the other hand, this study had some strength. The data were collected in this multicenter study using two different modeling techniques to mitigate the defects of each method and integrate their advantages.

CONCLUSION

In the current study, we used the high accuracy of ANNs for classification and the ability of DTs for presenting the risk factors of PITS non-follow-up. For a successful follow-up, some factors, such as the patient’s age and communication between the caregivers and patients (and their parents), should be considered.