A psychometric evaluation of the National Stressful Events Survey for PTSD-Short Scale (NSESSS-PTSD) among Korean psychiatric outpatients.

The National Stressful Events Survey for PTSD-Short Scale (NSESSS-PTSD) is a brief screening measure for DSM-5 PTSD that has not been evaluated for its psychometric properties in clinical population. We developed a Korean version of the original English scale through translation-back translation process and examined its reliability and validity among treatment-seeking adults at a psychiatric outpatient unit of a university-affiliated hospital in South Korea. The sample comprised adults diagnosed with PTSD (n = 100) and other psychiatric disorders (n = 134). The NSESSS-PTSD, the PTSD Checklist for DSM-5 (PCL-5), the Beck Depression Inventory-II (BDI-II), and the Beck Anxiety Inventory (BAI) were used to determine validity and reliability. The findings show modest test-retest reliability (r = .43), good internal consistency (Cronbach's α = .81), high convergent validity (r = .78) with PCL-5 and good concurrent validity with the BDI (r = .55) and BAI (r = .50), respectively. A cut-off score of 16 best predicted PTSD from other psychiatric disorders with specificity of .90 and sensitivity of .87. This study reveals sound psychometric properties of the Korean version of the NSESSS-PTSD and supports its use in the clinical population.

The latest edition of Diagnostic and Statistical Manual of Mental Disorder (DSM-5) has introduced a novel diagnostic category, the trauma- and stressor-related disorders for those who experienced negative life events and later showed distressful symptoms and difficulty in psychosocial functioning (American Psychiatric Association, 2013). Posttraumatic stress disorder (PTSD), previously classified among anxiety disorders, now have become a representative entity in this section. Major changes from previous DSM-IV PTSD criteria include adding occupational exposure to traumatic events (criteria A4), removal of subject reactions in definition of trauma, increased number of symptom clusters from three to four (B to E) and total symptoms from 17 to 20 (Pai et al., 2017).

To reflect these changes in the diagnostic criteria of posttraumatic stress disorder (PTSD) in DSM-5 (American Psychiatric Association, 2013), adults’ self-reported questionnaires from previous DSM-IV versions were upgraded. Furthermore, the psychometric properties, including the PTSD Checklist for DSM-5 (PCL-5) (Weathers et al., 2013), the PTSD Diagnostic Scale for DSM-5 (Foa et al., 2016), and the Primary Care PTSD Screen for DSM-5 (Prins et al., 2016) were tested.

Additionally, a new PTSD instrument called the National Stressful Events Survey for PTSD (NSES-PTSD) was developed in accordance with DSM-5 (Kilpatrick et al., 2011). The original scale comprised 20 items that encompassed each symptom of the DSM-5 criteria for PTSD and yielded stable construct validity, including a high Cronbach’s alpha of 0.94 (LeBeau et al., 2014). In the same paper, authors developed a reduced version, a 9-item subset of 20 PTSD symptoms that can be used to screen for PTSD or to measure symptom change over time. This brief scale, the National Stressful Events Survey PTSD Short Scale (NSESSS-PTSD) was developed by examining the reduction of internal consistency upon corresponding item deletions. The nine items included two intrusive symptoms (flashback and emotional reactivity), avoidance (avoidance of internal triggers), negative cognition or mood (distorted blame, negative emotions, and loss of interest), and three elements of arousal or reactivity (aggression, hypervigilance, and startled reactions).

The original NSESSS-PTSD was validated from 318 online respondents who met probable DSM-5 PTSD criteria: internal consistency was excellent (Cronbach α = .90) with the 20-item scale being 0.94 (LeBeau et al., 2014). Results from another independent sample of 66 undergraduate students confirmed its high internal reliability (Cronbach α = .91) and good convergent validity (r = .84) with the PTSD Checklist – civilian version (LeBeau et al., 2014). Other language versions of the NSESSS-PTSD that have undergone psychometric evaluations include the Turkish version for university students and inpatients with alcohol use disorder (Evren, Dalbudak, Aydemir, et al., 2016; Evren, Dalbudak, Umut, et al., 2016) as well as the Persian version for Iranian earthquake survivors (Rafiey et al., 2017).

However, there is a lack of psychometric data regarding the scale’s application to clinical settings, most importantly concerning screening properties for adults with DSM-5 PTSD. Under-diagnosis of PTSD in mental health outpatient setting has been reported as a comorbid condition to other principal diagnoses (Kiefer et al., 2020) or as a principal diagnosis (false-negatives) by psychiatrists in training (da Silva et al., 2019). In fact, PTSD was highly prevalent among psychiatric outpatients in these studies 17.4% and 20.5%, respectively. Psychiatric outpatients with unrecognised PTSD also showed negative outcomes such as lower improvement of symptoms, poor perceived health status, and less employment when they were re-evaluated after 3–4 years (Al-Saffar et al., 2002). In short, screening instruments with good psychometric properties for PTSD are needed for initial assessment in mental health facilities.

As such, we designed our study to include comprehensive psychometric properties. First, as the NSESSS-PTSD was not available in Korean, we did a standard translation and back-translation process to adapt the scale to the Korean language. Second, we sought to evaluate the scale’s psychometric properties including internal consistency, test-retest reliability, and construct validity among psychiatric outpatient populations. Based on prior findings on online responders with probable PTSD and patients with alcohol use disorders, we hypothesised that the NSESSS-PTSD would show good reliability and validity among mental health outpatients with DSM-5 PTSD.

Methods

1.1. Participants and procedures

Participants were treatment-seeking adults at the psychiatric outpatient unit of Hanyang University Guri Hospital, Gyeonggi Province, South Korea, who were initially given DSM-5 diagnosis of PTSD by attending psychiatrists and recruited by consecutive sampling from March 2017 to February 2020. When the process of written informed consent is completed, the questionnaires were given and one of a research psychiatrist (DK) and clinical psychologists (YM & HL) administered the Structured Clinical Interview for DSM-5 Disorders (First et al., 2016; Oh et al., 2017) in a blind condition to the results of the questionnaire. The inter-rater reliability among assessors was not examined. The completors of interview and questionnires received 30 US dollars as an imbursement for participation.

For comparison group, we used medical records and questionnaire data from consecutive sample of other psychiatric diagnoses but acute stress disorder. Each DSM-5 diagnosis in the comparison group was made by clinical interviews at the first appointment by attending psychiatrists. We did not confirm their diagnoses with the SCID or examine the inter-rater reliability of diagnoses.

The exclusion criteria were (1) intellectual disability and neurocognitive disorders, (2) severe medical conditions, and (3) comorbid psychotic illnesses. All participants provided written informed consent, and the study was approved by the hospital’s Research Ethics Committee (IRB No. 2015-05-013).

In the PTSD group, of 106 subjects responded to questionnaires, 6 (5.7%) were excluded due to incomplete or missing data; in the comparison group, 417 questionnaires were gathered, and 107 were disregarded due to incomplete or missing data (25.7%). As we did not investigate PTSD comorbidity with other diagnostic group, we excluded an additional 176 (42.2%) participants who scored higher than 33, a cut-off score for possible PTSD on the PTSD Checklist for DSM-5 (Bovin et al., 2016). Therefore, the final sample comprised 100 participants with PTSD and 134 with other psychiatric diagnoses.

To account for test-retest reliability, another convenient sample (n = 48) with PTSD at the same outpatient unit was administered with the NSESSS-PTSD between four-week interval. All of them were under active treatment, either psychotherapy alone (n = 24, 45.8%) or both psychotherapy and psychotropic medications (n = 26, 54.2%). Compared to the original sample, those tested for temporal stability had significantly younger age (median = 27.5 vs. 39.5, Mann–Whitney U = 1841.0, p = .022), longer period of time since the index trauma (median 96.0 vs. 6.5 months, Mann–Whitney U = 1138.5, p < .001), and more interpersonal trauma (58.3% vs. 41.0%, chi square = 3.92, p = .048). There were no significant differences in regard to sex and initial PTSD scores.

1.2. Measures

1.2.1. NSESSS-PTSD

The National Stressful Events Survey PTSD Short Scale (NSESSS-PTSD) is a nine-item abridged version of the original scale, which originally comprised 20 items that were designed to measure the 20 symptoms of DSM-5 PTSD (LeBeau et al., 2014). It the test, respondents were instructed to list the traumatic event they had experienced and rate how much they were bothered by each problem during the past week on a 5-point scale from 0 (not at all) to 4 (extremely). Thus, the total raw score ranged from 0 to 36, and this total score was to assess the severity of posttraumatic stress symptoms. Another use of the scale is the average total score, which is calculated by dividing the total score by the number of items. This average score easily and reliably categorised overall severity of PTSD as follows: none (0), mild (1), moderate (2), severe (3), and extreme (4) (Kilpatrick et al., 2013).

The NSESSS-PTSD has not been psychometrically evaluated in the clinical population, although it had been tested with college students to determine its internal consistency and validity compared to other PTSD instruments (LeBeau et al., 2014). Test-retest reliability, discriminant validity, cross-cultural adaptability, and establishing cut-off score have been suggested as future evaluation by the authors of the original version and those are addressed in our study.

1.2.2. PCL-5

The PTSD Checklist for DSM-5 (PCL-5) is a Likert-type self-report instrument that measures PTSD symptoms (Weathers et al., 2013). This scale has 20 items that address the DSM-5 diagnostic criteria for PTSD, and each item is rated on a Likert scale ranging from 0 (not at all) to 4 (extremely). The respondents were asked to rate each question how much they were bothered by the worst event they had experienced during the past month.

The original version proved excellent internal consistency (α = .96), high four-week test-retest reliability (r = .84), and good convergent validity with DSM-IV PCL (r = .84) in 468 treatment-seeking veterans (Bovin et al., 2016). The Korean version used in this study showed high internal consistency, high test-retest reliability, and good concurrent validity among elderly Vietnam veterans (Kim et al., 2017). Additionally, it resulted in high internal consistency, good convergent validity with other PTSD measures, and discriminant validity against anxiety and depression among earthquake survivors (Seo & Cho, 2021). The Cronbach’s alpha for this study was .92 for PTSD group.

1.2.3. BDI-II

The Beck Depression Inventory-II is a self-questionnaire designed to measure the degree of depressive symptoms present over the past two weeks. The scale comprises 21 items that ask respondents to choose similar states on a four-point Likert-type scale ranging from 0 to 3, which reflects intensity or frequency of the problem. The soundness of the psychometric properties of the BDI-II has been proven in many studies on psychiatric, clinical, non-clinical and medical populations (Wang & Gorenstein, 2013). The Korean version showed high internal consistency, excellent criterion validity with other depression measures, and good sensitivity and specificity among DSM-IV depressive disorders (Lim et al., 2011). The Cronbach α obtained in this study was .92 for PTSD group.

1.2.4. BAI

The Beck Anxiety Inventory is a 21-item instrument used to assess the severity of anxiety symptoms over the past month (Beck et al., 1988). It asks individuals to rate how affected they were by diverse symptoms of anxiety (e.g. ‘terrified or afraid’ and ‘fear of worst happening’) on a four-point scale ranging from 0 (not at all) to 3 (severely). The Korean version of the BAI has shown excellent internal consistency and discriminant validity for clinical anxiety disorders (Yook & Kim, 1997). The Cronbach’s alpha for this study was .94 for PTSD group.

1.3. Data analyses

The PTSD and other diagnostic groups were compared using the Mann–Whitney U test (due to the violation of normal distribution in continuous variables) and chi-square tests. The Pearson’s correlation coefficient was calculated to evaluate test-retest reliability, whereas Cronbach’s alpha was calculated to determine internal consistency. To test convergent and discriminant validity, we conducted Spearman’s correlation analyses between the NSESSS-PTSD and other measures. To obtain the best cut-off scores for identifying a participant with PTSD, a receiver operator characteristic (ROC) curve analysis was conducted with the NSESSS-PTSD assigned as the criterion variable (1 = PTSD, 0 = comparison group).

All statistical analyses were performed using SPSS 25 for Windows (IBM Co., Armonk, NY, USA) and the statistical significance was set to α = .05 bidirectionally.

Results

2.1. Participants characteristics

The participants with PTSD had experienced a variety of traumatic events; the most common kind of event was traffic accidents (n = 38, 38.0%), followed by childhood abuse (n = 19, 19.0%), physical assault (n = 12, 12.0%), traumatic loss (n = 10, 10.0%), accidents (n = 8, 8.0%), family violence (n = 8, 8.0%), and others (n = 5, 5.0%). Acute PTSD (i.e. less than three-month duration) involved 41.7% (40/96) of the sample.

Approximately two-thirds of the participants were women (62.0%) and had an annual income of less than 40,000 US dollars (67.0%). Slightly less than half of the participants were married (48.0%) and employed (47.0%), and most (86.0%) had high school education or higher. The mean the NSESSS-PTSD score was 24.5 (SD = 7.17, 95% CI = 23.1–25.9).

Among participants with other psychiatric diagnoses, panic disorder was the most common diagnosis (n = 37, 27.6%), followed by depressive disorder (n = 26. 19.4%), adjustment disorder (n = 19, 14.2%), insomnia (n = 17, 12.7%), and others (n = 35, 26.1%).

2.2. Comparison with the comparison group

As shown in Table 1, there were no statistically significant differences between the PTSD and comparison group regarding age, sex, marital status, employment, education and annual income (Table 1).

Table 1.

General characteristics of participants with PTSD (n = 100) and other psychiatric disorders (n = 134).

Variables	PTSD	others	statistics	p value
	N (%)	N (%)	χ2	p
SexMaleWomen	38 (38.0)62 (62.0)	64 (47.8)70 (52.2)	2.22	.136
Marital statusMarriedNever been marriedDivorced or widowed	48 (48.0)41 (41.0)11 (11.0)	77 (58.3)36 (27.3)19 (14.4)	4.87	.088
EmploymentEmployedUnemployedStudentshousekeepers	47 (47.0)27 (27.0)13 (13.0)13 (13.0)	63 (48.5)24 (18.5)12 (9.2)31 (23.8)	6.10	.107
EducationBelow high schoolHigh school graduateUniversity or more	13 (13.0)51 (51.0)36 (36.0)	22 (16.9)63 (48.5)45 (34.6)	.72	.699
Annual Income (US dollars)<2000020000–3999940000–59999>60000	38 (38.0)29 (29.0)21 (21.0)12 (15.4)	34 (27.4)37 (29.8)29 (23.4)24 (19.4)	3.95	.267
	Mean (SD)	Mean (SD)	U	p
Age	39.1(14.5)	44.6 (14.9)	8059.0	.008
NESESSS	24.5 (7.1)	7.6 (5.8)	649.0	< .001
PCL-5	50.9 (17.0)	33.6 (20.5)	635.5	< .001
BDI-II	33.5 (11.9)	18.0 (8.4)	1946.5	< .001
BAI	32.0 (13.8)	16.1 (11.7)	2517.5	< .001

BAI: Beck Anxiety Inventory; BDI: Beck Depression Inventory; NSESSS: National Stressful Events Survey PTSD Short Scale; PCL-5 PTSD checklist (PCL)-5; U: Mann-Whitney U.

Owing to missing data, the total sum of subjects may differ by variables.

2.3. Reliability

The Cronbach’s α for the NSESSS-PTSD was good (α = .81). Additionally, four-week test-retest reliability was revealed a significant correlation (mean = 25.3 (SD = 5.7) vs. 16.3 (SD = 9.4), r = .43, p = .002) across the time points considered.

2.4. Convergent and discriminant validity

The NSESSS-PTSD and PCL-5 demonstrated a high correlation (ρ = .78, p < .001) between the two scales and significant correlations were found between the NSESSS-PTSD and the BDI (ρ = .55, p < .001) and between the NSESSS-PTSD and the BAI (ρ = .50, p < .001). The NSESSS also showed significant higher scores when compared with those of other psychiatric diagnoses: the mean scores of 24.5 (SD = 7.1) vs. 7.6 (SD = 5.8), (median 159.3 vs. 86.3, Mann–Whitney U = 649.0, p < .001).

2.5. ROC analysis

The ROC curve analysis yielded an area under curve (AUC) value of .95 (95% CI: .92–.98), indicating excellent accuracy when it comes to predicting PTSD cases (Figure 1). A cut-off score of 16 corresponded to a sensitivity of .87, a specificity of .90, a kappa of .74, a positive predictive power of .90, and a negative predictive power of .86.

Figure 1.

The ROC curve of the NSESSS-PTSD scale scores against PTSD diagnosis. Note. NSESSS-PTSD: National Stressful Events Survey PTSD Short Scale.

Discussion

Our results suggest that the Korean-translated version of the NSESSS-PTSD has adequate to good psychometric properties, particularly its diagnostic utility against other diagnostic group, supporting its use for screening purposes in mental health outpatient setting. Our research hypothesis that the NSESSS-PTSD will provide favourable reliability and validity among clinical population has been confirmed.

In this study, the cut-off score of 16 for the identification of PTSD vs. other psychiatric disorders was identical to one Turkish study applying the NSESSS-PTSD to inpatients with alcohol use disorder (Evren, Dalbudak, Umut, et al., 2016) and lower than another Turkish study reporting 24 in the case of university students (Evren, Dalbudak, Aydemir, et al., 2016). Both studies determined cases with probable PTSD by PCL-C (DSM-IV based) scores while our study used a structured diagnostic interview (SCID) for PTSD cases and non-PTSD case definition by PCL-5 scores. The ROC curve demonstrated that the Korean version of NSESSS-PTSD had high predictive validity shown by excellent sensitivity, specificity, and the AUC.

Next, the Cronbach’s α of the NSESSS-PTSD was good (.81) in our study. This is comparable to the original English version, which initially reported a Cronbach’s α of .90 based on 318 adults from a large national sample with probable DSM-5 PTSD, as well as that of .91 from 66 college students who had experienced at least one traumatic event (LeBeau et al., 2014). Compared to versions in other languages, the Cronbach’s α in our study was lower than those reported for Iranian earthquake survivors (α = .88, n = 600) (Rafiey et al., 2017) and Turkish inpatients with alcohol use disorder (α = .88, n = 190) (Evren, Dalbudak, Umut, et al., 2016). However, higher than those of impoverished urban residents in Bangladesh (α = .70, n = 435) (Islam et al., 2021).

These differences between studies may depend on the population studied; in this case, the sample comprised treatment-seeking psychiatric outpatients with DSM-5 PTSD diagnoses from diverse traumatic events. It should be noted that values between .80 to .90 are ideal for a psychological instrument because a value higher than .9 may indicate that the scale has too many items (i.e. redundancy of the scale) (Cortina, 1993). Additionally, the NSESSS-PTSD is a brief test with relatively fewer items because longer tests tend to constitute more reliability (Tavakol & Dennick, 2011). Our study confirmed the internal consistency of the Korean version of the NSESSS-PTSD.

In our study, the correlation coefficient was lower (r = .43), which is considered to be modest for a psychometric questionnaire (Cicchetti, 1994). Only one study reported the retest reliability of the NSESSS-PTSD (Pearson r = .60) in Turkish inpatients with alcohol use disorder although the time interval between evaluations and any PTSD treatments given during the period were not mentioned. We have set the interval of four weeks for convenience of sampling between outpatient appointments, which is longer as intervals of one or two weeks are typical in health research (Polit, 2014).

Additionally, all participants for test-retest study received intensive treatments, i.e. psychotherapy and/or medication; changes in the time interval may reflect true changes in clinical status (Matheson, 2019). Thus, low correlation between treatment interval may in fact suggest usefulness of the NSESSS-PTSD in measuring symptom change over time.

Next, the NSESSS-PTSD had good convergent validity (ρ = .78) with other DSM-5 PTSD measure and moderate correlations with anxiety (ρ = .50) and depression (ρ = .55). Previous studies have reported correlation coefficients ranging from .76 to .84 using the previous DSM-IV version measure (PCL), thus showing similar results as ours. Nevertheless, the NSESSS-PTSD is a DSM-5 criteria-based instrument and should correspond to the DSM-5 PTSD measures as PCL-5 used in our study. Our findings confirmed the convergent validity of the NSESSS-PTSD.

Discriminant validity has been shown in part to have a lower, albeit moderate correlation with anxiety and depression, which often coexist and have overlaps with PTSD symptoms (Spinhoven et al., 2014). This is also in line with the re-positioning of the diagnostic entity of PTSD from anxiety disorder to trauma- and stressor-related disorders in the DSM-5. To further validate the discriminant validity, we additionally compared the scores of the NSESSS-PTSD between PTSD and other psychiatric disorders and found the significantly higher scores in PTSD.

This study has some limitations. First, the sample in our study was drawn from a single psychiatric centre and individuals with acute PTSD (41.7%) and single accident (46.0%) are overrepresented; thus external validity of our findings should be considered with caution. Second, comorbid psychiatric conditions were not investigated in both groups although those with probable PTSD (high self-reported symptoms over cut-off score) had been excluded in other diagnostic group. Possible concurrency of other psychiatric disorders in PTSD group may act as a confounding factor for discrimination of diagnoses. Third, we cannot rule out any influence of subtle cultural nuisance in translation and back-translation processes, although we tried to meet the principles of the cross-cultural adaptation of psychological instruments (Mason, 2005).

For future research, psychometric properties of the Korean version of the NSESSS-PTSD should be investigated in non-clinical population including diagnostic utility for screening. Conducting confirmatory factor analyses and expanding these finding to more heterogeneous trauma population are also needed.

To our knowledge, this is the first study to prove the reliability and validity of the NSESSS-PTSD in clinical population with DSM-5 PTSD. We have provided basic and encouraging psychometric data that show that the NSESSS-PTSD, which is both brief and accessible, can be used as a first-line evaluation tool for PTSD patients.