Psychometric Properties of the Chinese Version of the Primary Care Post-Traumatic Stress Disorder Screen-5 for Medical Staff Exposed to the COVID-19 Pandemic.

PURPOSE: The COVID-19 pandemic may increase the development of psychiatric disorders, such as posttraumatic stress disorder (PTSD) among medical staff. A brief validated screening tool is essential for the early diagnosis of PTSD. The purpose of the present study was to evaluate the validation of a Chinese version of the Primary Care-PTSD-5 (C-PC-PTSD-5) and determine an appropriate cutoff score with optimal sensitivity and specificity for medical staff in China during the COVID-19 pandemic. PARTICIPANTS AND METHODS: An online cross-sectional survey was conducted on medical staff (n = 1104) from 17 medical institutions in Shanghai. Questionnaires comprising general information, medical-related traumatic event experiences, the PTSD Checklist (PCL-5), and C-PC-PTSD-5 were distributed to participants using the online Questionnaire Star electronic system. Internal consistency, convergent validity, and test-retest reliability were calculated. Receiver operating characteristic (ROC) analysis was performed to determine diagnostic accuracy and the optimal cutoff score of the C-PC-PTSD-5 for medical staff.
RESULTS: We included 1062 valid questionnaires for the analysis. Data of 838 traumatic experiences were analyzed. Internal consistency of the C-PC-PTSD-5 was satisfied (Cronbach's α = 0.756). The total score of the C-PC-PTSD-5 showed good test-retest reliability (r = 0.746). We found a strong correlation between the C-PC-PTSD-5 score and PCL-5 total score (r = 0.669, p < 0.001), which indicated good convergent validity. The ROC analysis showed an area under the curve of 0.81 ± 0.016. A cutoff score of 2 provided optimal sensitivity and specificity for the C-PC-PTSD-5 (sensitivity = 0.632, specificity = 0.871, Youden index = 0.503, and overall efficiency = 0.768).
CONCLUSION: Our results indicated that the C-PC-PTSD-5 can be employed as a brief and efficient screening instrument for medical staff exposed to the COVID-19 pandemic. A score of 2 was identified as the optimal threshold for probable clinical PTSD symptoms.

Introduction

The outbreak of COVID-19 pandemic has resulted in various mental health problems among the public in China since December 2019. In the fight against the COVID-19 pandemic, medical healthcare workers in China have faced enormous pressure, which has included a high risk of infection, overworking, and social isolation.1 These stressors may trigger various mental disorders, such as anxiety and depressive disorders and posttraumatic stress disorder (PTSD),2 which in turn, could result in hazards that exceed the consequences of the COVID-19 epidemic itself.3 Because PTSD is a highly prevalent condition with a significant economic burden, early detection of symptoms in medical staff is essential.4

Efforts have been made to improve early detection and diagnosis accuracy of PTSD through the development of self-reported screening instruments. The PTSD Checklist (PCL) is one of the most widely applied tools for assessing PTSD in clinical and research settings.5 Recently, the PCL was updated to PCL-5, which contains 20 items that are rated on a five-point Likert-type scale. The items are based on the Diagnostic and Statistical Manual of Mental Disorder version 5 (DSM-5) criteria for PTSD.6 In addition to the original English version of the PCL-5, Swedish,7 Chinese,8 and German versions9 of the PCL-5 have also been developed.

The Primary Care Posttraumatic Stress Disorder Screen (PC-PTSD) is a four-item measure that is short and easy to administer,10 and despite its brevity, it has shown to be equivalent to other longer screening tools.11,12 In 2013, the PC-PTSD was revised to the PC-PTSD-5, according to the DSM-5 criteria for PTSD.13 The PC-PTSD-5 comprises five “Yes” or “No” items for re-experiencing, avoidance, hyperarousal, numbing, and alteration in mood and cognition, which correspond to the symptom clusters of the DSM-5 criteria for PTSD. One of the purposes of the PC-PTSD-5 is to screen individuals for PTSD and make a provisional PTSD diagnosis. The diagnostic accuracy of the PC-PTSD-5 was initially demonstrated in a veteran sample.14

The prevalence of probable PTSD in medical staff is approximately three times that of the general population, which is attributed to the long hours and work-related stressors.15,16 In particular, emergency department staff were found to exhibit more severe PTSD symptoms than general staff in psychiatric wards during previous pandemics.17 It is essential to explore the psychometric properties of the Chinese version of the PC-PTSD-5 (C-PC-PTSD-5) before it is implemented for the identification of probable medical-related PTSD among medical staff.

The objective of present study was to evaluate the validation of the C-PC-PTSD-5 and determine an appropriate cutoff score with optimal sensitivity and specificity for medical staff in China during the COVID-19 pandemic.

Materials and Methods

Participants

The survey was carried out from December 6, 2020, to December 24, 2020, in Shanghai approximately 1 year following the outbreak of COVID-19 in China. The study was conducted in accordance with the Declaration of Helsinki and was approved by the ethics committee of Ruijin Hospital/Lu Wan Branch, School of Medicine, Shanghai Jiaotong University. Fifteen hospitals and two centers for disease control (CDCs) across five districts of Shanghai were selected as research sites using the multi-stage stratified random cluster sampling method. With the help of the labor union of selected hospitals and CDCs, questionnaires were distributed to participants using the online Questionnaire Star electronic system (Changsha Ranxing IT Ltd. Corporation, ). All participants have signed an informed consent form on the first page of the questionnaire, and used WeChat or the webpage to complete the questionnaires. The inclusion criteria for subjects were those working in a hospital or a CDC, which included doctors, nurses, allied healthcare workers (pharmacists, physiotherapists, and occupational therapists), technicians, administrators, clerical staff, and maintenance workers. To ensure data quality, the same IP address could only be used once.

Screening Questionnaire

The study questionnaire was written in Chinese and comprised four main components: General information, medical-related traumatic experiences, the PCL-5, and the C-PC-PTSD-5.

General subject data collected included sex, age, marital status, type of medical institution (hospital or CDC), years working, average hours worked per week, and monthly salary.

The checklist for traumatic events in the questionnaire comprised nine typical medical-related traumatic experiences. A free-text “other” option was provided for participants to describe traumatic events experienced that were not included in the checklist. Participants indicated whether they had experienced any of the nine listed traumatic events and when they had occurred. Participants who had experienced at least one traumatic experience were asked to complete both the PCL-5 and PC-PTSD-5. If they had not experienced a traumatic experience, the survey was terminated and participants were excluded from the validation test of the C-PC-PTSD-5.

PCL-5

The PCL-5 is a 20-item self-report measure to assess the 20 DSM-5 symptoms of PTSD. The PCL-5 is applied as a Likert scale, and the severity of each symptom is divided into five levels from “Not at all” = 0, to “A little bit” = 1, “Moderately = 2”, “Quite a bit” = 3, and “Extremely” = 4. The maximum score is 80.

The PCL-5 is scored using two methods. First, a total score of between 31–33 indicates probable PTSD. Second, a provisional PTSD diagnosis can be made by treating each item rated as 2 or higher as a symptom endorsed and following the DSM-5 diagnostic rule, which requires at least: one B item (questions 1–5), one C item (questions 6–7), two D items (questions 8–14), or two E items (questions 15–20). In the present study, a cutoff score of 33 points was used as a diagnosis of PTSD.9

Chinese Version of the PC-PTSD-5

The C-PC-PTSD-5 was translated by a group of psychiatrists and psychologists. It was adapted using a two-stage process of translation and reverse translation. First, we invited a bilingual psychiatrist to translate the PC-PTSD-5 into Chinese, which was then translated back into English by a bilingual psychologist. Discrepancies were resolved and corrected until the translated version was considered adequate.

To analyze for test–retest reliability of the C-PC-PTSD-5, 30 medical health workers were invited to complete the online questionnaires before the cross-sectional survey, and those who reported medical-related traumatic experiences were required to be re-assessed 3 weeks after the initial assessment.

Data Collection

The results of the survey administered using the online Questionnaires Star were exported as an Microsoft Office Excel 2007 data table A total of 1104 questionnaires from 15 hospitals and two CDCs in Shanghai were collected, and 42 questionnaires were excluded because of incomplete data (18 questionnaires) or response times of less than 90 s (24 questionnaires). The valid response rate was 96.19% as 1062 valid questionnaires were included. In addition, 224 participants who answered “No” to all nine medical-related traumatic experiences were excluded for the validation test of the C-PC-PTSD-5, which resulted in a final sample size of 838 (Figure 1).

Figure 1

Flow chart the study.

Data Analysis

All statistical analyses were performed using IBM SPSS Statistics for Windows version 25.0 (Armonk, NY). Categorical variables are expressed as absolute values (percentages) and continuous variables are expressed as means ± standard deviations (SDs). Test–retest reliability of the C-PC-PTSD-5 was assessed using Spearman correlation and a paired t-test. Internal consistency of the C-PC-PTSD-5 total score was evaluated using Cronbach’s alpha. Sensitivity, specificity, Youden index, and overall efficiency were used to quantify the diagnostic utility of the C-PC-PTSD-5. A receiver operating characteristic (ROC) analysis was performed to determine the optimum cutoff value by comparing the patterns of sensitivity and specificity with those of the PCL-5 as the standard.

Results

Baseline Characteristics of Participants

Data from 838 valid questionnaires were used for the analysis. Baseline characteristics, PCL-5 scores, and PC-PTSD-5 scores of the 838 participants are summarized in Table 1. Participants had an average total score of 34.464 for the PCL-5, and 359 (33.8%) participants met or exceeded the recommended cutoff score of 33 for a standard PTSD diagnosis.

Table 1

Baseline Characteristics of Study Participants (N = 838)

Characteristic	N	%	PCL+	PCL-5 M (SD)	C-PC-PTSD-5 M (SD)
Types of medical institutions
Third class hospital	635	75.78	271	34.82(15.61)	1.28 (1.55)
Second class hospital	171	20.41	75	33.15(13.88)	1.13 (1.44)
CDC	32	3.82	13	34.91(17.93)	1.28 (1.65)
Sex
Female	356	42.48	177	32.14(13.03)	1.13(1.43)
Male	482	57.52	182	37.61(17.55)	1.41(1.64)
Age
20–25	299	35.68	119	34.41(16.04)	1.34(1.61)
26–30	360	42.96	156	34.29(14.55)	1.19(1.46)
31–35	76	9.07	34	34.07(15.52)	1.26(1.58)
36–40	36	4.30	16	34.78(16.55)	1.06(1.49)
>40	67	8.00	34	35.94(15.88)	1.24(1.55)
Education
Undergraduate	456	54.42	197	34.95(15.58)	1.27(1.55)
Graduate	243	29.00	104	34.33(15.68)	1.28(1.54)
Doctor	136	16.23	57	33.26(14.05)	1.15(1.46)
Other	3	0.36	1	26.00(8.66)	0.00(0.00)
Medical practitioner or not
Yes	671	80.07	289	33.58(13.90)	1.17(1.48)
No	167	19.93	70	35.81(17.66)	1.44(1.63)
Marital status
Single	612	73.03	261	34.71(15.53)	1.27(1.55)
Married	220	26.25	94	33.55(14.72)	1.19(1.49)
Other	6	0.72	4	43.17(18.42)	0.83(1.33)
Average hours work per week
<50h	187	22.32	67	31.20(12.29)	0.92(1.37)
50–60h	352	42.00	137	33.18(14.38)	1.12(1.42)
60–70h	168	20.05	85	36.74(15.99)	1.45(1.62)
70–80h	56	6.68	26	39.00(18.80)	1.55(1.54)
>80h	75	8.95	44	40.12(19.24)	1.97(1.87)
Monthly salary(¥)
<5000	227	27.09	107	36.89(16.54)	1.45(1.62)
5001–8000	418	49.88	177	33.54(14.44)	1.17(1.46)
8001–10,000	95	11.34	39	34.14(15.78)	1.23(1.56)
10,001–15,000	75	8.95	28	32.87(15.78)	1.15(1.62)
>15,000	23	2.74	8	33.91(14.50)	1.04(1.33)

Notes: PCL+ means PCL-5 scores were equal or more than 33; M is mean; SD means standard deviation.

Internal Consistency, Convergent Validity, and Test–Retest Reliability of the C-PC-PTSD-5

The C-PC-PTSD-5 showed good internal consistency (Cronbach’s α = 0.756, 95% confidence interval [CI]: 0.729, 0.781), p < 0.001) based on the criterion of 0.30 as an acceptable corrected item-total correlation. All five items of the C-PC-PTSD-5 performed adequately (0.688, 0.776, 0.613, 0.656, and 0.690, respectively; range, 0.613–0.776).

We found a strong correlation between the C-PC-PTSD-5 score and PCL-5 total score (r = 0.669, p < 0.001), which indicated good convergent validity.

For test–retest reliability, 24 participants were re-assessed using the C-PC-PTSD-5 2 weeks after the initial assessment. The total score of the C-PC-PTSD-5 showed good test–retest reliability (r = 0.746). The paired t-test revealed no significant difference in scores between the assessment times (Time 1: Total score [mean ± SD] = 1.333 ± 2.060; Time 2: Total score [mean ± SD] = 0.916 ± 1.782, t = −0.959, p = 0.358), which indicated good consistency across both assessment times.

Diagnostic Utility and ROC Analysis

Receiver operating characteristic (ROC) analyses were performed to calculate the sensitivity and specificity of the C-PC-PTSD-5 total score when compared with the PCL-5 score’s PTSD diagnosis cutoff value of 33 (Figure 2). The analysis was performed to establish the optimal cutoff score for the C-PC-PTSD-5 for medical staff who had experienced traumatic events and was determined by calculating the maximum area under the curve (AUC). As shown in Figure 2, the AUC of the C-PC-PTSD-5 was 0.81 ± 0.016, which indicated that the C-PC-PTSD-5 has a high degree of discriminatory power (95% CI: 0.779, 0.841).

Figure 2

ROC curves for C-PC-PTSD-5.

When examining the cutoff of a score of 2 for the C-PC-PTSD-5, diagnostic utility was considered more acceptable (sensitivity = 0.632, specificity = 0.871, Youden index = 0.503, and overall efficiency = 0.768) than that of a cutoff of 1 (sensitivity = 0.799, specificity = 0.689, Youden index = 0.488, and overall efficiency = 0.736) or 3 (sensitivity = 0.454, specificity = 0.958, Youden index = 0.412, and overall efficiency = 0.742; Table 2).

Table 2

Diagnostic Utility of Different Scores for the C-PC-PTSD-5

C-PC-PTSD-5 Score	Sensitivity	Specificity	Youden Index	Overall Efficiency
1	0.799	0.689	0.488	0.736
2	0.632	0.871	0.503	0.768
3	0.454	0.958	0.412	0.742
4	0.248	0.987	0.235	0.671
5	0.114	0.998	0.112	0.619

Discussion

This cross-sectional survey in a large sample of participants who had experienced traumatic events indicated that the C-PC-PTSD-5 (a Chinese version of the PC-PTSD-5 that is designed to reflect the new DSM-5 PTSD criteria) is a valid and reliable screening tool for probable PTSD for medical staff exposed to the COVID-19 pandemic. The diagnostic utility analysis of the C-PC-PTSD-5 identified a recommended cutoff score of 2, based on an optimal balance of sensitivity and specificity.

The current results need to be interpreted with consideration of several limitations. First, because clinical interviews in a large sample were not approved in light of the COVID-19 pandemic in China, an online electronic PCL-5 was employed as the standard in the present study, which likely decreased model accuracy. The PCL-5 is currently used globally and has been found to have highly reliable psychometric properties across various settings.9,18–20 In addition, the Chinese version of the PCL-5 has shown good validity and has been used to screen for probable PTSD in earthquake survivors,8,21 and has also been previously implemented as a control to test the psychometric properties of the PC-PTSD-5 in Chinese children.22 Further research comparing the PCL-5 with structured clinical interviews, such as the Clinician-Administered PTSD Scale for DSM-5, is needed. Second, most study participants were physicians, nurses, or technicians (N = 1062, 99.06%), and only 10 labor workers completed the questionnaires. This was due to unfamiliarity or reluctance to complete the electronic questionnaire on a mobile phone. Moreover, socioeconomic status and education level may have influenced our results. Third, 1 year following the COVID-19 outbreak, the spread of virus and infections had been controlled in the Shanghai region, and most participants had not been in direct contact with COVID-19-infected patients. Therefore, our findings may not generalize to medical staff in severely affected areas.

Various studies have reported an increased risk of psychopathologies and stress-related disorders among medical staff, as well as a high rate of PTSD symptomatology.23–26 In the present study, 359 (33.8%) healthcare workers met the PCL-5 standard for a PTSD diagnosis, which is higher than that observed in recent research.27 Specific sample characteristics may play a role in this discrepancy. Healthcare workers have a relatively high level of education, which has been shown to be a significant predictor of PTSD.28 In fact, the mental health of healthcare workers during the Severe Acute Respiratory Syndromes (SARS) and Middle East Respiratory Syndrome (MERS) epidemic was found to be poorer in the phase following the acute outbreak than in the initial phase.29 PTSD symptoms usually appear months or years after the traumatic experience, and it may increase along with the pandemic. The detection of probable PTSD symptoms in medical staff is essential, even several years after experiencing a traumatic event.

Although we did not compare the time taken by participants to complete the PCL-5 and C-PC-PTSD-5, it is clear that the C-PC-PTSD-5 can be completed more quickly than the PCL-5. Participants are required to respond with a “yes” or “no” to only five items compared with the 20 items of the PCL-5, which has five options for each item. We randomly interviewed several healthcare workers who were not involved in the survey and found that a brief questionnaire was more popular and suitable for large samples in medical institutions in light of the COVID-19 pandemic.

Our findings indicated that the C-PC-PTSD-5 has good psychometric properties and the contextually validated cutoff score provides an effective screening instrument for identifying medical staff with PTSD symptoms. A cutoff score of 2 showed an optimal balance of sensitivity and specificity, although a cutoff of 1 revealed the highest sensitivity (0.799). Sensitivity is more important than other diagnostic accuracy measures in primary care settings, especially in non-treatment-seeking medical staff exposed to the COVID-19 pandemic. Participants who score over 1 should be recommended for further assessment for PTSD using a clinical interview.

Conclusion

The present findings demonstrate that the C-PC-PTSD-5 possesses good psychometric properties and can be used as a reliable, valid, and time-saving tool to diagnose and assess PTSD. A score of 2 was identified as a threshold for probable medical-related PTSD symptoms. The Chinese C-PC-PTSD-5 will help identify medical staff with PTSD who may benefit from early intervention, which will facilitate the delivery of mental health care.