Ventilator-associated pneumonia in patients on prolonged mechanical ventilation: description, risk factors for mortality, and performance of the SOFA score.

OBJECTIVE: Ventilator-associated pneumonia (VAP) is a serious complication of mechanical ventilation (MV). However, data on VAP in patients on prolonged MV (PMV) are scarce. We aimed to describe the characteristics of VAP patients on PMV and to identify factors associated with mortality.
METHODS: This was a retrospective cohort study including VAP patients on PMV. We recorded baseline characteristics, as well as 30-day and 90-day mortality rates. Variables associated with mortality were determined by Kaplan-Meier survival analysis and Cox regression model.
RESULTS: We identified 80 episodes of VAP in 62 subjects on PMV. The medians for age, Charlson Comorbidity Index, SOFA score, and days on MV were, respectively, 69.5 years, 5, 4, and 56 days. Episodes of VAP occurred between days 21 and 50 of MV in 28 patients (45.2%) and, by day 90 of MV, in 48 patients (77.4%). The 30-day and 90-day mortality rates were 30.0% and 63.7%, respectively. There were associations of 30-day mortality with the SOFA score (hazard ratio [HR] = 1.30; 95% CI: 1.12-1.52; p < 0.001) and use of vasoactive agents (HR = 4.0; 95% CI: 1.2-12.9; p = 0.02), whereas 90-day mortality was associated with age (HR = 1.03; 95% CI: 1.00-1.05; p = 0.003), SOFA score (HR = 1.20; 95% CI: 1.07-1.34; p = 0.001), use of vasoactive agents (HR = 4.07; 95% CI: 1.93-8.55; p < 0.001), and COPD (HR = 3.35; 95% CI: 1.71-6.60; p < 0.001).
CONCLUSIONS: Mortality rates in VAP patients on PMV are considerably high. The onset of VAP can occur various days after MV initiation. The SOFA score is useful for predicting fatal outcomes. The factors associated with mortality could help guide therapeutic decisions and determine prognosis.

INTRODUCTION

In recent years, progress in medical care has led to a decrease in in-hospital mortality of critically ill patients. , The increase in survival in the acute stage of a critical illness has caused a progressive increase in the population of subjects with chronic critical illness (CCI). The prevalence of CCI is 7.6% of the number of admissions to ICUs in the United States and can be as high as 33% in patients presenting with acute respiratory failure. Although the definition of CCI is heterogeneous, the need for prolonged mechanical ventilation (PMV) is the most important feature to define it. , One of the latest and most widely used definitions of CCI describes it as the length of ICU stay of 8 or more days, associated with at least one of the following conditions: mechanical ventilation (MV) for 96 h or more, tracheostomy, sepsis, severe wounds, and multiple organ failure. The definition of PMV also varies among authors, although the most widely used one is the need for MV for 21 days or more, at least for 6 h/day. , Using this definition, the incidence of PMV in patients on MV admitted to an ICU ranges from 6.3% to 9.9%. ,

Infections in patients on PMV are one of the most common complications. In a multicenter study that included 1,419 patients on PMV, the top six complications recorded over a 1-year period were infections, the most frequent of which were urinary tract infection and ventilator-associated pneumonia (VAP), in 34.5% and 31.0% of the patients, respectively. The high risk of infections is due to multiple factors: prolonged use of invasive elements (tracheostomy tube, catheters, etc.), prolonged exposure to environments contaminated with virulent and resistant microorganisms, and immunological alterations due to comorbidities and the recent critical illness of the patient.

VAP is one of the most common hospital-acquired infections. Roughly, 10% of patients requiring MV develop VAP, with a mortality rate of 20-50%. , It is known that the clinical presentation and management of VAP differ in patients on PMV. The risk of VAP in patients on PMV is lower than that in those who are in the acute stage of MV, but it increases by 1-3% the duration of MV per day, suggesting that between 50% and 66% of tracheostomized patients will develop VAP. The impact of respiratory infections on patients on PMV is relevant and has been linked to increased mortality. However, data on VAP in patients on PMV are scarce. Not only do most of the studies about VAP include patients on acute MV, but differences in the definitions of PMV and CCI have revealed that the knowledge on the subject is sparse and almost incomparable.

The SOFA score is a widely used tool for predicting mortality in septic ICU patients. However, its utility in subjects with CCI has yet to be well established.

The objective of the present study was to describe the characteristics of VAP patients on PMV, to determine factors that are associated with 30-day and 90-day mortality, and to evaluate the performance of the SOFA score as a predictor of mortality in this population.

METHODS

Study design

We conducted a retrospective cohort study involving patients on PMV and diagnosed with VAP who were admitted to the Chronic Critical Patients Unit (CCPU) at the Sanatorio Güemes, located in the city of Buenos Aires, Argentina, between June of 2015 and October of 2019. Our institution is a 680-bed university-affiliated acute-care general hospital. The CCPU is a 24-bed unit developed for the treatment and rehabilitation of patients with CCI. Patients admitted to the CCPU are mainly referred from ICUs and stroke units. The overall mortality rate in the CCPU is 44.8%, and 179 patients are admitted per year on average. VAP prevention bundles include elevation of head of bed to 30-45 degrees, daily spontaneous breathing trials, oral decontamination, and endotracheal tube cuff pressure monitoring. We used the hospital-acquired infection surveillance registry to identify consecutive patients with a diagnosis of VAP who were admitted to the CCPU. Using electronic medical records, subjects with VAP who were on PMV were selected. The following baseline variables were collected: reason for admission to the institution, demographic data, comorbidities, and age-adjusted Charlson comorbidity index. Variables collected at diagnosis of VAP were number of days on MV, SOFA score, need for vasoactive agents, and PaO2/FiO2. Data for the 90-day follow-up period after the diagnosis of VAP were also collected, including microbiological isolate results, adequacy of empirical antibiotic treatment, 30-day mortality, and 90-day mortality. We used these variables to search for factors associated with 30-day and 90-day mortality. Given the retrospective nature of the study, no written informed consent was required. The study was approved by the institutional research department. All data were kept confidential, and the study was carried out in accordance with the Declaration of Helsinki.

Study population and definitions

Patients ≥ 18 years of age who were on PMV and had been in the CCPU for at least 48 h prior to the onset of VAP were included in the study, regardless of having had previous episodes of VAP. Subjects who developed VAP in other units or within 48 h after admission to the CCPU were excluded. In patients with more than one episode of VAP, a new episode was defined as the development of new clinical, radiological, and bacteriological findings after completion of the antibiotic treatment of the previous episode. In accordance with the National Association for Medical Direction of Respiratory Care, we defined PMV as MV for ≥ 21 days for at least 6 h/day. We used the definition of the National Hospital Infection Surveillance Program of Argentina (Programa de Vigilancia de Infecciones Hospitalarias de Argentina), which defines VAP on the basis of clinical and radiological criteria : presence of at least one of the major criteria (temperature > 38°C, and white blood cell count > 12,000 cells/mm3 or < 4,000 cell/mm3) and at least one of the three minor criteria (purulent sputum, decreased PaO2/FiO2, and new or persistent infiltrates on at least two chest X-rays). Microbiological diagnosis of VAP was confirmed by blood cultures (detection of growth of microorganisms, with no other obvious cause); by quantitative cultures of BAL fluid samples (≥ 104 CFU/mL); or by quantitative cultures of endotracheal aspirate samples (≥ 105 CFU/mL).

Organ function was assessed using the SOFA score, calculated within the first 24 h after the diagnosis of VAP. Comorbidities were assessed by age-adjusted Charlson comorbidity index. We defined VAP-related use of vasoactive drugs as the prescription of noradrenaline or dopamine at any time from the day of the diagnosis of VAP to the end of antibiotic treatment. The PaO2/FiO2 ratio recorded was the one obtained at the time closest to the onset of VAP. Microbiological cultures were ordered by attending physicians in accordance with institutional protocols and incubated in standard media. We performed quantitative cultures of the isolates in respiratory samples. Bacterial identification was carried out using conventional biochemical tests and an automated microbiology system (BD Phoenix; Becton Dickinson, Sparks, MD, USA). Antimicrobial susceptibility was determined by disk diffusion, and colistin susceptibility was determined by the automated system (BD Phoenix), in accordance with the Clinical and Laboratory Standards Institute recommendations.

Statistical analysis

Statistical analyses were performed with R Studio, version 3.5.2 (The R Foundation for Statistical Computing, Vienna, Austria). Continuous variables are expressed as means and standard deviations or medians and interquartile ranges, according to their distribution. Qualitative variables are shown as absolute and relative frequencies. We carried out univariate and multivariate analyses to identify factors associated with mortality. Qualitative variables were compared using the chi-square test or the Fisher’s exact test, whereas continuous variables with parametric and nonparametric distribution were compared using the Student’s t-test and the Mann-Whitney U test, respectively. Cox regression models were used in order to find predictors of 30-day and 90-day mortality. All of the variables showing p < 0.25 in the univariate analysis were included in the multivariate model. Nested models were selected using Akaike information criterion. Kaplan-Meier curves and log-rank tests were performed for variables associated with 90-day mortality. Hazard ratios and 95% CIs were used in order to identify variables associated with 30-day and 90-day mortality. All tests were two-sided, and statistical significance was set at p < 0.05.

RESULTS

Patient characteristics

During the study period, 80 episodes of VAP were identified in 62 patients on PMV, 13 of whom had more than 1 episode of VAP (2 episodes in 9 patients, 3 episodes in 3 patients, and 4 episodes in 1 patient). The characteristics of the patients and clinical presentation of VAP are shown in Table 1. There was a slight predominance in males, and 40 episodes (50%) occurred in patients aged > 70 years. The major reasons for hospital admission were community-acquired pneumonia, in 20 patients (25.0%); stroke, in 19 (23.7%); infections other than VAP, in 7 (8.7%); traumatic brain injury, in 7 (8.7%); and congestive heart failure, in 5 (6.2%).

Table 1

Patient characteristics.a

Characteristic	Group
	All patients	30-day survivor	30-day nonsurvivor	p	90-day survivor	90-day nonsurvivor	p
	(N = 80)	(n = 56)	(n = 24)		(n = 29)	(n = 51)
Male	52 (65)	37 (66)	15 (62)	0.95	20 (69)	32 (61)	0.75
Age	69.5 [58-80]	65.0 [56.5-80.0]	71.5 [65.0-80.5]	0.11	61 [38-80]	71 [63-80]	0.01
Cause of admission Medical Surgical Trauma	60 (75) 11 (14) 9 (11)	41 (73) 7 (13) 8 (14)	19 (79) 4 (17) 1 (4)	0.4	25 (86) 1 (3) 3 (10)	35 (68) 10 (20) 6 (12)	0.11
Charlson comorbidity index	5 [2-6]	4.0 [2.0-6.0]	5.5 [4.5-6.0]	0.041	4 [2-5]	5 [3-6]	0.054
Comorbidities Critical illness polyneuropathy Neurological injury Focal neurological deficit Persistent vegetative state Altered sensorium COPD Congestive heart failure Diabetes mellitus Malignant disease Chronic renal failure Obesity Immunosuppression	73 (91) 52 (65) 30 (37) 11 (14) 10 (12) 24 (30) 14 (17) 13 (16) 10 (12) 8 (10) 7 (9) 4 (5)	52 (93) 38 (68) 23 (41) 6 (11) 6 (11) 13 (23) 8 (14) 7 (12) 9 (16) 5 (9) 2 (4) 3 (5)	21 (87) 14 (58) 7 (29) 5 (21) 4 (17) 11 (46) 6 (25) 6 (25) 1 (4) 3 (12) 5 (21) 1 (4)	0.42 0.57 0.45 0.39 0.71 0.079 0.4 0.29 0.27 0.93 0.024 1	28 (97) 24 (83) 16 (55) 5 (17) 2 (7) 2 (7) 4 (14) 3 (10) 5 (17) 3 (10) 1 (3) 0 (0)	45 (88) 28 (55) 14 (27) 6 (12) 8 (16) 22 (43) 10 (20) 10 (20) 5 (10) 5 (10) 6 (12) 4 (8)	0.41 0.02 0.02 0.51 0.31 < 0.001 0.72 0.36 0.48 1 0.25 0.29
Days on MV from the onset of VAP	56.0 [40.0-88.0]	54.0 [40.5-88.5]	58.5 [42.5-94.0]	0.49	64 [45-93]	49 [35-81]	0.23
Days of hospital stay from the onset of VAP	62.5 [43-97]	62.5 [43.0-96.0]	66.5 [46.5-102.0]	0.4	63 [43-103]	62 [43-94]	0.59
SOFA score Patients with a SOFA score > 5	4 [3-7] 27 (34)	4 [3-5] 11 (20)	8 [4-10] 16 (67)	< 0.001 < 0.001	4 [3-5] 7 (24)	4 [3-8] 20 (39)	0.21 0.26
Vasoactive agents during the episode	32 (40)	15 (27)	17 (71)	< 0.001	8 (28)	24 (47)	0.14
PaO2/FiO2 Patients with PaO2/FiO2 < 200b GCS score Patients with GCS score ≤ 5 Sedative therapy	242 [187-323] 20 (31) 9 [7-10] 12 (15) 18 (22)	270 [200-340] 10 (23) 9.0 [7.5-10.0] 6 (10) 7 (12)	200 [157-246] 10 (48) 9.0 [6.0-10.5] 6 (25) 11 (46)	0.004 0.08 0.55 0.19 0.002	310 [194-354] 5 (24) 9 [8-10] 2 (7) 3 (10)	223 [187-270] 15 (34) 9 [6-11] 10 (20) 15 (29)	0.01 0.58 0.93 0.19 0.09
Microbiological isolates P. aeruginosa Enterobacteriaceae ABC Other Negative culture	38 (47) 20 (25) 6 (7) 1 (1) 15 (18)	29 (52) 14 (25) 2 (4) 0 (0) 11 (20)	9 (37) 6 (25) 4 (17) 1 (4) 4 (17)	0.35 1 0.062 1 7 (24)	13 (45) 9 (31) 1 (3) 0 (0) 8 (16)	25 (49) 11 (22) 5 (10) 1 (2) 7 (14)	0.89 0.50 0.41 1 0.53
Inadequate empirical antibiotic therapyc	6 (9)	5 (11)	1 (5)	0.7	2 (9)	4 (9)	1

MV: mechanical ventilation; VAP: ventilator-associated pneumonia; GCS: Glasgow Coma Scale; and ABC: Acinetobacter calcoaceticus-baumannii complex. aValues expressed in n (%) or median [IQR]. bn = 62. cEvaluated in subjects with identified isolates (n = 65).

Regarding MV, all of the patients were submitted to tracheostomy, and the median number of days on MV from the onset of VAP was 56 (range, 21-564 days). Pathogens were identified in 65 of the 80 episodes (81.2%), Pseudomonas aeruginosa accounting for almost half of the microbiological isolates, whereas no Staphylococcus aureus was isolated. There was only 1 episode in which two different isolates were identified (P. aeruginosa and Providencia stuartii). Antibiotic resistance patterns are shown in Table 2. The most commonly used empirical antibiotic treatment regimens used were meropenem+colistin; meropenem+colistin+vancomycin; and piperacillin/tazobactam+colistin, in 23 (29%), 22 (27%), and 13 (16%) of the overall number of VAP episodes, respectively. Beta-lactam antibiotics were prescribed as definite therapy in 65 (81%) of the episodes (combined with other antibiotics in 28), as were colistin as monotherapy in 11 and aerosolized colistin+parenteral antibiotics in 7 (all cases of P. aeruginosa infection).

Table 2

Antibiotic resistance patterns of the pathogens isolated.a

Pathogen	SAM	3GCs	FEP	CIP	TMP/SMX	TZP	IPM	MEM	AMK	CST	TGC
P. aeruginosa	100	46	46	59	100	51	57	59	35	0	100
Enterobacteriaceae	86	38	33	76	71	33	24	24	5	76	72
ABC	100	100	100	100	100	100	100	100	33	0	17

SAM: ampicillin/sulbactam; 3GCs: third generation cephalosporins; FEP: cefepime; CIP: ciprofloxacin; TMP/SMX: trimethoprim/sulfamethoxazole; TZP: piperacillin/tazobactam; IPM: imipenem; MEM: meropenem; AMK: amikacin; CST: colistin; TGC: tigecycline; and ABC: Acinetobacter calcoaceticus-baumannii complex. aValues expressed in %.

All but 1 patient completed 30-day follow-up in the unit. That patient was referred to a long-term acute care hospital (LTACH) once the antibiotic treatment was completed. The 30-day and 90-day mortality rates after the diagnosis of VAP were 30.0% and 63.7%, respectively.

Time of onset of VAP

Figure 1 shows the frequency of VAP episodes in relation to duration of MV in days. Between days 21 and 50 of MV, 36 of the 80 episodes of VAP (45.0%) occurred. By day 90, there had already been 62 episodes of VAP (77.5%).

Figure 1

Number of episodes of ventilator-associated pneumonia (VAP) per days of mechanical ventilation. The last episode of VAP occurred on day 564 of mechanical ventilation.

Factors associated with 30-day and 90-day mortality rates

The Cox regression model showed that the SOFA score and use of vasoactive agents were associated with 30-day mortality, whereas age, SOFA score, use of vasoactive agents, and COPD were associated with 90-day mortality (Table 3). In a post-hoc analysis, we explored the interaction between PaO2/FiO2 < 200 and SOFA score and between PaO2/FiO2 < 200 and use of vasoactive agents by adding an interaction term to the model. None of the interactions were significant (p = 0.15 and p = 0.09, respectively). Figure 2 shows Kaplan-Meier curves for the variables associated with 90-day mortality. Among the patients who received vasoactive agents, 22 (68%) had a SOFA score > 5, and 10 (32%) had a SOFA score ≤ 5 (p < 0.001).

Table 3

Cox regression model for variables associated with 30-day and 90-day mortality.

Variable	30-day mortality, HR (95% CI)	p	90-day mortality, HR (95% CI)	p
Age	1.02 (0.99-1.05)	0.13	1.03 (1.001-1.05)	0.003
SOFA score	1.3 (1.12-1.52)	< 0.001	1.2 (1.07-1.34)	0.001
Vasoactive agents	4.01 (1.24-12.95)	0.02	4.07 (1.93-8.55)	< 0.001
ABC	2.62 (0.8-8.57)	0.11	1.81 (0.55-5.31)	0.27
PaO2/FiO2 < 200	1.73 (0.64-4.7)	0.28	1.2 (0.62-2.32)	0.59
COPD	2.59 (0.93-7.18)	0.06	3.35 (1.71-6.6)	< 0.001

HR: hazard ratio; and ABC: Acinetobacter calcoaceticus-baumannii complex.

Figure 2

Kaplan-Meier curves for variables associated with 90-day mortality in patients with ventilator-associated pneumonia patients on prolonged mechanical ventilation. Age and SOFA score were dichotomized for the analysis.

DISCUSSION

We performed a retrospective analysis of VAP patients on PMV. We found that age, SOFA score, use of vasoactive agents, and COPD were associated with increased mortality. To our knowledge, this is the study with one of the largest cohorts of VAP patients on PMV and the first one to address predictors of mortality. The lack of information on this topic could be partly due to the variety of terms that have been coined to study this population and the different ways in which such terms have been defined over time. We decided to use PMV to include subjects with VAP, because, in a retrospective study, it is easier to identify patients by the length of MV, in accordance with the definition of the National Association for Medical Direction of Respiratory Care. However, using the definition of the Chronically Critically Ill Population Payment Recommendations, all of our patients could be considered patients with CCI.

In our cohort, we found an elderly population with a significant burden of comorbidities, mainly due to neurological impairment. This is in line with a study on VAP in an LTACH. In that study, 19 patients had 23 episodes of VAP, 69% of whom required MV due to a neurological cause. Neurological impairment probably explains the difficulty in weaning from MV and the need for PMV.

Regarding microbiological isolates, all of the episodes of VAP in our study were caused by gram-negative bacilli, almost half of them being P. aeruginosa, corroborating one study involving subjects with VAP in an LTACH, reports of the U.S. National Healthcare System Network, and a study on VAP in tracheostomized patients. P. aeruginosa is one of the leading causes of VAP worldwide. , Prior P. aeruginosa colonization of the airway is a crucial factor for developing VAP. It is possible that patients on PMV and submitted to tracheostomy have higher rates of colonization, partially explaining such observations. No S. aureus was isolated from our patients, which might corroborate the findings of a study that reported a lower proportion of VAP caused by S. aureus in late-onset VAP than in early-onset VAP.

The median number of days on MV prior to the onset of VAP was 56, which is considerably lower than that reported by Walkey et al. -median = 166 days (IQR, 66-450). Unlike an LTACH, our unit receives patients immediately after the acute stage of a critical illness, which explains the shorter time from MV to the onset of VAP. The 30-day mortality rate was 30%. The mortality of patients with VAP ranges from 20% to 50%, with an attributable mortality of 13%. However, those data come from cases of VAP during acute MV. Few studies have addressed VAP in patients on PMV. In a small study about VAP in tracheostomized patients, 3 of 12 died, but larger scale studies are needed to establish the mortality rate in this population. Furthermore, the mortality of patients with CCI is considerably high: in-hospital mortality rates range from 17% to 31%, whereas 6-month and 12-month mortality rates are 49% and 68%, respectively. , , Infections have been identified as one of the factors associated with mortality in patients with CCI. However, the mortality attributable to VAP in this population has yet to be determined.

We found factors associated with 30-day and 90-day mortality in our population. There are some scores to predict mortality in patients with CCI, such as the designated ProVent score, but they are not specific for VAP. Similar to the predictions based on the ProVent score, we found that age and the use of vasoactive agents were associated with fatal outcomes. Although the SOFA score includes the use of vasoactive agents as a factor, their use on any day of the VAP episode until the end of the antibiotic treatment were taken into consideration, and the SOFA score was calculated using the results obtained for the variables at the time closest to the diagnosis of VAP. We observed that almost two-thirds of the patients who received vasoactive agents had higher SOFA scores, so collinearity cannot be entirely ruled out. However, the effect of vasoactive agents on mortality was greater than that predicted on the basis of the SOFA score, and some patients who received vasoactive drugs had lower SOFA scores. Therefore, the exact influence of each variable has yet to be determined.

The SOFA score is a useful tool for predicting mortality in critically ill patients. Although this tool is used in different ways (differences between SOFA scores obtained on different days, highest score, mean score, etc.), the initial SOFA score has been proven to have value in predicting ICU mortality. In patients with VAP, the SOFA score has also been validated as a prognostic factor, , although those studies were carried out in acute care ICUs. Tseng et al. included 163 patients with VAP; however, only 42 (26%) of those were tracheostomized, and a subgroup analysis was not performed, which makes it difficult to extrapolate their results to the CCI population.

The relationship between COPD and VAP has been studied to some extent; both conditions interact in a number of levels: COPD patients on MV are at an increased risk of VAP. COPD is an independent factor associated with increased mortality, longer length of MV, longer length of ICU stay, and higher rates of infection with P. aeruginosa in patients with VAP. - Our results are in line with the results in those studies with patients on acute MV. - However, we found no association between COPD and 30-day mortality, even though there was a trend toward that (p = 0.06). This could be due to the small number of patients in our study, but it might also reflect a greater effect of COPD on 90-day mortality. It is probable that the longer the patient is followed from the onset of VAP, the greater will be the effect of chronic conditions on mortality. Similar considerations could be drawn regarding age. In patients with CCI and a high burden of comorbidities, chronic conditions and age weigh more consistently on long-term outcomes, matching overall mortality. Perhaps a shorter endpoint, such as 30-day mortality, is more suitable for this population in order to define the attributable mortality of nosocomial infections. Comparative, prospective studies are needed to shed light on this issue.

The present study has several limitations. First, it is a study carried out in a single center. The settings of care provision for patients on PMV may differ among other centers and have heterogeneous populations. In fact, our results might not be generalizable to patients at LTACHs, because several of those patients may not have complications or characteristics similar to those of patients with CCI. Second, the retrospective nature of the study may carry biases inherent to this type of design. Some variables and confounders may not have been taken into account given the difficulty of including them in a retrospective study. Although we tried to control for confounders using Cox regression analysis, we cannot fully rule out that other variables not included in the analysis, such as the time of antibiotic therapy initiation or the presence of coinfections, might have affected our results. Third, the definition used for VAP might have led to the inclusion of some patients with ventilator-associated tracheobronchitis, which is a limitation inherent to the definition. Fourth, the sample size was somewhat small, so external validity might have been compromised and the results might not be fully generalizable for some of the findings. This highlights the need for multicenter studies that address the particular aspects of VAP patients on PMV.

In conclusion, we found a high burden of comorbidities in our sample, mostly related to neurological conditions, as well as considerably high 30-day and 90-day mortality rates. We identified factors associated with fatal outcomes, which could help identify patients who might benefit from adequate, early empirical antibiotic treatment, as well as determine prognoses. These findings should be validated by studies with larger samples of patients.