Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam.

The present study investigated the internal consistency reliability, construct validity, and item response characteristics of a newly developed Vietnamese version of the Kessler 6 (K6) scale among hospital nurses in Hanoi, Vietnam. The K6 was translated into the Vietnamese language following a standard procedure. A survey was conducted of nurses in a large general hospital in Hanoi, Vietnam, using a questionnaire including the Vietnamese K6, other scales (DASS21, health-related QOL, self-rated health, and psychosocial work environment), and questions about demographic variables. Internal consistency reliability (Cronbach's alpha coefficient) was calculated. A confirmatory factor analysis was conducted. Eleven hypotheses were tested (as Pearson's correlations with the K6) to assess the scale's construct validity. Item response theory (IRT) analysis was conducted to identify the item response characteristics. The Cronbach's alpha coefficient was 0.864. The explanatory and confirmatory factor analyses indicated a one-factor structure. Most hypotheses tested for construct validity were supported. IRT analysis indicated that response categories were located in order according to severity. K6 provided reliable information regarding higher levels of psychological distress. The findings suggest that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress among hospital nurses in Vietnam.

Introduction

The Kessler 6 (K6) scale is a six-item short measure of psychological distress and is also used as a screening instrument for common mental disorders in the community [1, 2]. The items were selected from a large item pool based on item response theory (IRT) analysis [1]. The interviewer-administered K6 showed a good screening performance for common mental disorders in the community, with the area under the curve (AUC) of receiver operating characteristic of 0.854, a sensitivity of 0.36 and a specificity of 0.96. The K6 was translated into 21 language families [3]. Because of its brevity, the scale has been used widely in mental health epidemiology. A cross-country validation study including data from 14 countries reported a substantial concordance between interviewer-administered K6 scores and diagnoses of common mental disorders (CMD) (the median AUC, 0.83) [4]. Country-specific validation studies have been published for the Arabic [5], English [6-8], Chinese [9], French [10], Japanese [11, 12], Korean [13], and Farsi (Persian) languages [14]. These studies in general reported good internal consistency reliability (0.69–0.92 in Cronbach’s alpha) [5, 7, 9, 10, 12–15] and factor-based [9, 10] and other construct validity (correlations with other mental health measures) [5, 7, 8, 10, 13] across various populations (university students, community residents, employees, people living with HIV, and psychiatric patients), for both self-report and interviewer-administered versions. However, contrary to the initial assumption of uni-dimensionality [1, 2], some studies reported a multifactorial factor structure for the K6 [9, 10]. In addition, while a scoring approach based on unweighted item scores (i.e., 0-1-2-3-4 for the five response categories) has been used in most studies, the authors of the K6 argued that IRT-based weighted scoring could be a better and easy-to-use alternative [1]. In fact, an IRT analysis combining data from 14 countries suggested the use of a weighted scoring approach (e.g., 0-0-0-1-1) [4]. An IRT study reported that the psychometric properties of the K6 may vary depending on ethnicity in the United States [16]. Psychometric properties of the K6 should be systematically investigated including confirmatory factor analysis and IRT analysis for specific country and ethnic populations [17].

There is a growing concern for mental health in low- and middle-income countries (LMICs) [18]. One of the proposed strategies to scale up mental health service in these countries is to integrate mental health services into workplaces [19]. In LMICs, healthcare professionals face increasing stress and burnout at work [20] because of a shortage of healthcare professionals [21, 22], population aging, and demands from service users (i.e., patients and families) [23-25]. This is particularly the case in South-East Asia [21]. Increasing stress at work has been reported among nurses in Vietnam [26, 27], Thailand [28], and Myanmar [29]. High levels of stress at work and poor mental health of healthcare professionals may diminish the quality of health care services in these countries [30]. It is important to manage work-related stress and to improve mental health among nurses in Southeast Asian countries.

The present study investigated psychometric properties, i.e., internal consistency reliability, construct validity, and item response characteristics of the K6 scale translated in Vietnamese in a sample of nurses working in a large general hospital in Hanoi, Vietnam. This is the first validation study of a Vietnamese version of the K6. While the study used a sample from a specific occupational group the findings may be useful in exploring the cross-cultural validity of the K6 scale in Vietnam. This is also the first study in which IRT analysis was conducted on data from the K6 in a LMIC since a previous cross-national study used a combined data set from high-income countries and LMICs to conduct an IRT analysis [4].

Methods

Participants

A questionnaire survey was conducted as the baseline survey for a stress management intervention study in a large general hospital in Hanoi, Vietnam [31], from August 21 to September 10, 2018. Out of all registered nurses (n = 1,269) in this hospital, a total of 1,258, excluding those who were about to retire (n = 11), were invited to complete a questionnaire and return it in an envelope to collection boxes. The returned questionnaires were collected by staff at Hanoi University of Public Health and entered into a database.

The aim and procedure of the study was reviewed and approved by the Research Ethics Committee of the Graduate School of Medicine/Faculty of Medicine, The University of Tokyo (no. 11991-(1)) and the Ethical Review Board for Biomedical Research, Hanoi University of Public Health (no 346/2018/YTCC-HD3).

Measures

K6 scale

The K6 scale is a 6-item self-report measure of psychological distress, with a 5-point response option (4 = All of the time, 3 = Most of the time, 2 = Some of the time, 1 = A little of the time, and 0 = None of the time) [1]. The possible range of the total score is 0 to 24, with higher scores indicating greater psychological distress. The translation of the K6 into the Vietnamese language was conducted following the standard procedure of translation and cultural adaptation of patient reported health outcomes [32]. After obtaining permission to translate it into Vietnamese from the lead author of the research team that developed the K6 (Prof Ronald C Kessler, Harvard Medical School, United States of America), the original English version was translated into Vietnamese independently by two translators who were fluent in English and were knowledgeable about mental health symptomatology. These translations were integrated into a preliminary translation. This translation was reviewed by collaborators at the Hanoi University of Public Health, and a brief cognitive interview was conducted of a group of nurses (n = 30) to ask their opinions on draft items to know if the wordings were meaningful and relevant, followed by necessary amendments. This amended version was back-translated into English by an independent translator, and reviewed by staff of the research team at Harvard Medical School, as well as by the authors. Further careful revisions were made to make sure some of the K6 items were appropriately translated: "hopeless" in item #2; "restless or fidgety" in item #3; "depressed" in item #4. The final draft was pre-tested in a pilot study of 150 nurses in the same hospital in June 2018. A psychometric analysis showed a problem in this draft: item #5 "…that everything was an effort?" did not correlate with other items. The item was further revised based on a discussion among the collaborators. The final version of the Vietnamese version of K6 was posted on the website of the Harvard Medical School: https://www.hcp.med.harvard.edu/ncs/k6_scales.php

Other scales for testing the construct validity

Depression, anxiety and stress. The 21-item version of the DASS scale, which measures symptoms of depression, anxiety, and stress in community settings [33, 34], is comprised of the three corresponding subscales, each with seven items. Items are scored on a 4-point scale ranging from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time). Each subscale score ranges from 0 to 21. Reliability and validity of the Vietnamese version of the DASS21 scale have previously been reported [35].

Health-related quality of life. The EQ-5D-5L is a widely used indicator of health-related quality of life (HR-QOL) [36]. The EQ-5D-5L consists of five items covering mobility, self-care, usual activities, pain/discomfort and anxiety/depression, each of which is rated on a five category scale from no problems (1) to extreme problems (5). Reliability and validity of the EQ-5D-5L are well established [36]. A Vietnamese version of DASS21 has been developed and tested, and has acceptable reliability and validity [37]. We used the standard value set for Vietnam to calculate participants’ HR-QOL scores [38].

Psychosocial work environment. The Job Content Questionnaire (JCQ) was used to assess the psychosocial work environment [39]. It includes four scales: a five-item psychological demand scale, a nine-item decision latitude scale, a four-item supervisor support scale, and a four-item coworker support scale. The items are scored on a 4-point Likert scale, ranging from 1 (strongly disagree) to 4 (strongly agree). The reliability and validity of the Vietnamese version of the JCQ has been found to be acceptable, while Cronbach’s alpha coefficients were lower for job demands and job control (0.446–0.499) than those for supervisor and coworker support (0.856–0.868) [40].

Self-rated health. Participants was asked to rate their health status on a 0–100 scale, in which 0 indicates the worst health status and 100 indicates the best health status. This was part of the EQ-5D-5L scale [36, 37], but not used in the calculation of the HR-QOL score.

Demographic variables

Demographic variables were included in the questionnaire. Respondents were asked to indicate their sex (male or female), and their birth year. Personal income per month was assessed by a single-item question, “What is your current monthly income (including salary and any other sources of income)?”, with a 3-point response scale: ≤ 5 million VND (220 USD) (1), 5–10 million VND (220–432 USD) (2), and ≥ 10 million (432 USD) (3). Respondents were also asked whether they were currently married, never married, or divorced/separated, and about the nature of their labor contract: fixed-term contract for less than one year, fixed-term contract for more than one year, unspecified-term contract, permanent contract, or “other.”

Statistical analysis

Distribution of the K6 scores was examined separately for men and women. The average and standard deviation (SD), maximum and minimum values of K6 scores were calculated. The proportions of participants who had a score of 13 or higher [2] were calculated for men and women. Internal consistency reliability was calculated by using the Cronbach’s alpha coefficient. Then a confirmatory factor analysis was conducted to assess the fit of the data to a one-factor structure. The goodness of fit index (GFI), adjusted goodness of fit index (AGFI), and comparative fit index (CFI) greater than 0.95 and the root mean square error of approximation (RMSEA) smaller than 0.06 to 0.08 were considered to be indicators of acceptable fit [41] For a comparison with previous studies, an exploratory factor analysis was conducted with maximum likelihood extraction, in order to assess the number of factors using a scree plot and proportion of variance explained by the first factor.

Construct validity was assessed by using Pearson’s correlations (rs), with the following 11 hypotheses: DASS21 scores of (1) depression, (2) anxiety, and (3) stress moderately or strongly and positively (rs>0.4) with K6 score [5, 10, 13]; (4) HR-QOL and (5) self-rated health status moderately and negatively (rs = -0.4 to -0.7) correlated with K6 scores [42]; (6) job demands weakly or moderately positively (r = 0.2 to 0.4), and (7) job control, (8) supervisor support, and (9) coworker support weakly or moderately negatively (rs = -0.2 to -0.4) correlated with K6 scores [43]; socioeconomic position, such as (10) higher education and (11) personal income weakly and negatively (rs = -0.1 to -0.2) correlated with K6 scores [44].

A screening performance of K6 for identifying a CMD (i.e., depressive disorders and anxiety disorders) was also examined. Cases with CMD were defined as those who had 34 or greater scores of the total DASS21, that was the best cut-off score found in a previous study in Vietnam [35]. An area under the curve (AUC) of a receiver operative curve (ROC) and the 95% CIs were calculated predicting these cases with CMD on the sore of K6. An optimum cut-off score was determined based on the best Youden index (a sum of sensitivity and specificity), and the sensitivity, specificity, and positive likelihood ratios were calculated with their standard errors [45].

The IRT analysis was conducted using the graded response model. Threshold parameters for response categories and slope parameter of each item were estimated. The test information curve and item information curves were presented in a graphical way.

Most of these analyses were conducted using IBM SPSS version 22. The confirmatory factor analysis and IRT were conducted using PROC CALIS and PROC IRT, respectively, from the SAS software package version 9.4.

Results

Respondent’s characteristics

A total of 949 (75.6% of the target population) respondents returned the questionnaires. One respondent had a missing response on one of the K6 items, and was excluded from the analysis. Participants were mostly women, married, and younger than 40 years old, with age range between 22 and 58 years (Table 1). Most were with monthly personal income of more than 5 million VND (equivalent to 220 USD) About half had completed college or higher education. About half had a permanent contract, while about one fourth were employed with a fixed-term contract for one year.

Table 1

Demographic characteristics of the study participants, hospital nurses in Vietnam (n = 948).

Variables	n	%
Sex
Men	143	15.1
Women	805	84.9
Age in years
18–29	348	36.7
30–39	428	45.1
40+	172	18.1
Education
Vocation school	442	46.6
Colleges	137	14.5
University undergraduate	348	36.7
Postgraduate	17	1.8
Unknown†	4	0.4
Marital status
Single	136	14.3
Married	793	83.6
Divorced/widowed	16	1.7
Unknown†	3	0.3
Personal income per month in VDN
≤ 5 million (220 USD)	91	9.6
5–10 million (220–432 USD)	563	59.4
≥ 10 million (432 USD)	284	30.0
Unknown†	10	1.1
Employment contract
Fixed-term, <1 year	219	23.1
Fixed term, > 1 year	29	3.1
No fixed-term	194	20.5
Permanent	504	53.2
Others	2	0.2

† Treated as missing in the analysis.

Distribution of the scale scores

The K6 scores showed a skewed distribution for both men and women (S1 Appendix). The average K6 scores of women and men did not significantly differ (Table 2). The proportion of respondents with a K6 score of 13 or greater was 2.4%, and did not significantly differ between men and women. Average item scores were higher for item #1 compared to those for the other items (Table 3). For items #4 and #5, no respondents endorsed the extreme category “all of the time”).

Table 2

Average, standard deviation (SD), median, minimum and maximum values of K6 scores in a sample of hospital nurses in Vietnam.

Sex*	N	Average	SD	Median	Minimum	Maximum
Men	143	4.14	3.39	3.4	0	17
Women	805	4.64	3.51	3.9	0	20
Total	948	4.57	3.50	3.8	0	20

* No significant difference in the average score (t-test, p = 0.114) between men and women.

Table 3

Average and standard deviation (SD) of item scores, item-total correlation, and Cronbach’ alpha coefficient when an item was deleted for K6 in a sample of hospital nurses in Vietnam (n = 948).

	Item scores
Items	Average	SD	Range	Item-total correlation (corrected)	Cronbach’ alpha coefficient when the item was deleted†
1. …nervous	1.66	0.83	0–4	0.543	0.864
2. …hopeless	0.61	0.78	0–4	0.718	0.830
3. …restless or fidgety	0.68	0.75	0–4	0.678	0.838
4. …so depressed that nothing could cheer you up	0.77	0.81	0–3	0.703	0.833
5. …that everything was an effort	0.53	0.71	0–3	0.720	0.831
6. …worthless	0.31	0.63	0–4	0.615	0.850

† Cronbach’s alpha coefficient for the total K6 score was 0.864.

Internal consistency reliability

The Cronbach’s alpha coefficient for the total K6 score was 0.864. Most items substantially contributed to the Cronbach’s alpha coefficient, while item #1 did not (Table 3).

Factor-based validity

The confirmatory factor analysis assuming a one factor structure indicated a moderate fit of the one factor model: GFI = 0.957; AGFI = 0.901; RMSEA = 0.115 (90% confidence intervals, 0.098 to 0.134); CFI = 0.954. Standardized coefficients (error variance) for items 1 to 6 were 0.582 (0.662), 0.764 (0.416), 0.726 (0.472), 0.774 (0.400), 0.794 (0.369), and 0.695 (0.517), respectively. In the explanatory factor analysis, initial eigenvalues (% of variance explained) for factors 1 to 6 were 3.619 (60.3), 0,750 (12.5), 0.472 (7.9), 0.449 (7.5), 0.399 (5.2), and 0.311, respectively, yielding one single factor.

Construct validity

DASS21 scores for depression, anxiety, and stress correlated moderately and positively with K6 score (rs = 0.513 to 0.544) (Table 4). HR-QOL and self-rated health correlated negatively with K6 score (rs = -0.399 and -0.355, respectively), while the correlation coefficients were slightly smaller than 0.4. Job demands weakly and positively correlated with K6 score (r = 0.292). Job control, supervisor support and coworker support weakly and negatively correlated with K6 score (rs = -0.147, -0.204, and -0.160, respectively). Personal income was only marginally significantly associated with K6 (r = -0.06, p = 0.066).

Table 4

Eleven hypotheses tested for the construct validity of K6 among hospital nurses in Vietnam: Pearson’s correlation coefficients (rs) with K6 scores.

Variables	n†	Average	SD	rs	p
DASS depression score (0–21)	932	3.0	2.9	0.513	<0.001
DASS anxiety score (0–21)	935	3.9	3.1	0.539	<0.001
DASS stress score (0–21)	935	5.6	3.6	0.544	<0.001
HR-QOL score (0–1)	948	0.9	0.1	-0.399	<0.001
Self-rated health (0–100)	946	85.7	11.6	-0.355	<0.001
JCQ job demands score (12–48)	933	31.5	4.4	0.279	<0.001
JCQ job control score (24–96)	931	81.1	6.4	-0.147	<0.001
JCQ supervisor support score (4–16)	943	12.0	1.9	-0.204	<0.001
JCQ coworker support score (4–16)	946	12.2	1.5	-0.160	<0.001
Education (1–4) ‡	944	1.9	1.0	-0.083	<0.001
Personal income (1–3) §	938	2.2	0.6	-0.060	0.066

† The number of participants varied due to missing responses on each variable.

‡ Coded as 1 = vocation school, 2 = colleges, 3 = university, 4 = postgraduate, excluding missing responses

§ Coded as 1 = ≤ 5 million (220 USD), 2 = 5–10 million (220–432 USD), 3 = ≥ 10 million (432 USD) per month, excluding missing responses.

Among the participants, 22 cases were identified as having CMD (i.e., 34 or greater scores of the total DASS21). The AUC predicting CMD on K6 score was 0.919 (95% CIs, 0.855–0.984). The best cut-off score for K6 was identified as 9+, with 129 (12.4%) of the participants having the scores equal to or greater than this cut-off. The sensitivity and specificity were 0.864 (SE, 0.073) and 0.876 (SE, 0.031), respectively, with the positive likelihood ratio of 6.96 (SE, 0.121).

Item-response theory analysis

For all the items, the threshold parameters almost linearly increased from mild to severe response categories (Table 5). For items #2–6, the threshold parameters for most response categories of each item were greater than 0. On the other hand, for item #1, the first two threshold parameters were less than 0. The slope parameters were similar for items #2–6 (1.512–1.770), while the parameter was smaller for item #1 (0.909). The test information curves indicated that the K6 provided more information when the trait was greater than 0 (i.e., when people had higher psychological distress than the average) (Fig 1). Item information curves indicated that most of the items provided good information where the trait was greater than 0 (Fig 2). Item #1 had markedly lower information across the latent trait continuum than the other items. However, item #1 provided better information than other items where the trait was less than -0.5.

Table 5

Response category characteristic curves for each item of K6 among hospital nurses in Vietnam.

Item	Parameter	Estimate	Standard error	Pr > |t|
k1	Threshold 1	-2.162	0.129	< .0001
	Threshold 2	-0.291	0.062	< .0001
	Threshold 3	1.564	0.101	< .0001
	Threshold 4	3.578	0.252	< .0001
	Slope	0.909	0.060	< .0001
k2	Threshold 1	0.164	0.048	0.0004
	Threshold 2	1.285	0.069	< .0001
	Threshold 3	2.420	0.128	< .0001
	Threshold 4	3.331	0.267	< .0001
	Slope	1.536	0.104	< .0001
k3	Threshold 1	-0.058	0.051	0.1256
	Threshold 2	1.320	0.075	< .0001
	Threshold 3	2.987	0.183	< .0001
	Threshold 4	3.564	0.285	< .0001
	Slope	1.320	0.087	< .0001
k4†	Threshold 1	-0.185	0.049	< .0001
	Threshold 2	1.096	0.064	< .0001
	Threshold 3	2.267	0.116	< .0001
	Slope	1.512	0.100	< .0001
k5†	Threshold 1	0.259	0.046	< .0001
	Threshold 2	1.384	0.071	< .0001
	Threshold 3	2.805	0.166	< .0001
	Slope	1.770	0.133	< .0001
k6	Threshold 1	0.853	0.058	< .0001
	Threshold 2	1.841	0.098	< .0001
	Threshold 3	2.712	0.162	< .0001
	Threshold 4	3.424	0.276	< .0001
	Slope	1.513	0.121	< .0001

† No respondent endorsed the highest response category for these items.

Fig 1

Test information curve of K6 among hospital nurses in Vietnam.

The vertical axis indicates the latent trait of psychological distress.

Fig 2

Item information curve for each item of K6 among hospital nurses in Vietnam.

The vertical axis indicates the latent trait of psychological distress.

Discussion

The Vietnamese self-report version of the K6 showed high level of internal consistency reliability. The uni-dimensional structure of the K6 was supported by the confirmatory factor analyses of the items. Most hypotheses tested for construct validity were supported. Item response analysis indicated that response categories were located in order according to severity. The K6 provided information for detecting higher psychological distress as a whole, but item #1 did not. The study suggests that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress.

The confirmatory factor analysis indicated a good fit for the one-factor structure, with acceptable levels of most goodness of fit indicators, while RMSEA was not within an ideal range (<0.6–0.8) but close. The findings of the confirmatory factor analysis suggest that the Vietnamese K6 has a one-factor structure. The first factor had an eigenvalue of 3.6 and explained 60.3% of the total variance in the explanatory factor analysis. The one-factor structure was consistently observed across 14 countries including LMICs (Brazil, China, Colombia, India, Mexico, Nigeria, Ukraine) [4] with similar eigenvalues for the first factor. The one-factor structure seems applicable to most countries, including LMICs. However previous studies in France and China reported a two-factor structure consisting of items related to depression and anxiety [9, 10, 15]. The factor structure of the K6 may still vary depending on language, ethnicity, and culture.

For construct validity, all of the 11 hypotheses tested were supported by the data. DASS21 scales for depression, anxiety, and stress all correlated moderately with the K6, which is concordant with previous findings of moderate to strong correlations with other depression scales [5, 10, 13]. Consistent with previous studies [42], HR-QOL and self-rated health also moderately correlated with K6 score. The correlations were slightly below 0.4. Job demands, job control, supervisor support, and coworker support correlated weakly (|rs| = 0.15–0.28) in expected directions [43]. The modest associations between job demands and job control with K6 scores may be attributable to lower reliability of these scales [40]. Educational attainment and personal income correlated very weakly (|rs|<0.1) with the K6. These findings are in line with previous studies on socioeconomic factors associated with depression [44]. The small correlation for educational attainment may be a result of the fact that all participants had the same occupation; the small correlation for personal income may be attributable to some participants having income from other family members. The present study suggests an adequate level of construct validity of the Vietnamese version of the K6.

A previous IRT analysis using data from 14 countries found the two most severe response categories were useful for discriminating degrees of psychological distress, and discussed the use of a weighted scoring approach (e.g., 0-0-0-1-1) [4]. However, in the present study, the threshold parameters almost linearly increased according to the severity of response categories. The present study rather supports the 0-1-2-3-4 scoring for the Vietnamese K6. The test information curve was better when the latent trait was greater than 0, suggesting that the Vietnamese version of the K6 measure high levels of psychological distress more accurately. This version of the K6 may be useful for detecting poor mental health among hospital nurses in Vietnam, although how K6 score relates to probability of presence of diagnosable psychiatric disorder, and what is the most appropriate cutoff score, in Vietnam is currently unknown.

Internal consistency reliability measured by Cronbach’s alpha coefficient was high enough in this sample to be comparable with previous reports [5, 9, 10, 12–15]. However, item #1 did not contribute much to the internal consistency reliability. The item information curves from the IRT analysis indicated that item #1 provided poorer information than the other items. This is partly because the item #1 that originally asks “nervousness” was translated by using a Vietnamese word “stress”, that is popular wording used frequently in daily life. The item #1 can be dropped from the Vietnamese version of the K6, still keeping the current level of internal consistency reliability and the scale information. However, it might be good to have an item with such lower difficulty in the beginning of the scale, to make it easy for respondents to start responding to the scale. Potential benefits of keeping and dropping this item should be investigated further, e.g., by conducting a randomized controlled trial comparing psychometrics of the two versions.

The distribution of scores of the Vietnamese version of the K6 was skewed to the right. The pattern is similar to an exponential distribution of K6 scores found in a study in the United States [46]. However, there was a peak at scores of 1–3 in this sample, not at the score of 0 as in the US sample [46]. This may be because the item response function of item #1 for those response categories had lower difficulty parameters. It is hard to compare average scores of K6 in this sample (4.57) with other reports, since some reports did not indicate the average score, and available average scores varied a lot (e.g., 3.6 in a community sample in Japan [12]; 5.9 in a community sample in the United States [1]; 10.0 among older people in Korea [13]; and 12.9 among Palestinian social workers [5]. The screening performance of K6 score predicting CMD defined by using DASS21 was high: the AUC was 0.919, with the sensitivity of 0.864 and the specificity of 0.876 for the best cut off score of 9+. The finding is similar to one from a previous study in Japan that the stratum-specific likelihood ratio became higher (>11) for the cut off scores of 9+ on K6 [47]. However, the screening performance and the best cut off score of the Vietnamese version of the K6 should be assessed by comparing people who have or have not been previously diagnosed with mental disorders.

A common use of the K6 (and K10) is in large-scale population mental health surveys. Despite some reservations about varying sensitivity of K6 for different disorders [48] it is a strong candidate for such use in Vietnam, where a population representative national mental health survey to estimate prevalence of common mental disorders has not yet been conducted. The absence of reliable estimates of population prevalence of mental disorders is a significant problem for mental health policy-making and advocacy. The use of the K6 in such national surveys would require studies to establish the criterion validity of the K6 in Vietnam.

The present study has several limitations. First, we aimed to validate the K6 scale to screen psychological distress among nurses, a special occupational group under high stress, in Vietnam. Nurses are usually more educated than general workers, and also familiar with questions of psychological symptoms included in K6. The present findings may not be generalized to the general population or working population in Vietnam. The reliability and validity of the present scale should be further tested in these populations. Second, as we noted earlier, we did not investigate the scale’s screening performance for clinically diagnosed mental disorders. Third, the study did not examine test-retest reliability or indicators for interpretability (e.g., minimal important difference or minimally important change) which might be useful in applying the K6 in practice. Finally, 150 nurses who responded to a pilot study were also invited to the main validation study; most of them probably actually participated. Because the pilot study was anonymous, it was impossible to conduct an analysis excluding those who participated in the two surveys. Repeated surveys have been reported to result in attenuated responses to anxiety symptoms of respondents [49], which may affect the finding of the present study. Despite these limitations, the study clarifies the psychometric properties of the newly developed Vietnamese self-report version of the K6, including item response characteristics first examined in a LMIC.

Conclusions

The present study found an acceptable level of internal consistency reliability, and supported a unidimensional factor structure and other construct validity of Vietnamese self-report version of the K6. Item response analysis indicated that the K6 provided information for detecting higher psychological distress as a whole. It is suggested that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress at least among nurses in Vietnam.

Distribution of K6 score among males and females in a sample of hospital nurses in Vietnam.

(TIF)

Click here for additional data file.

10 Mar 2020

PONE-D-19-34231

Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam

PLOS ONE

Dear Dr. Kawakami,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

===In order to provide a more complete information to our readers on the topic, we would like to emphasize the importance to cross referencing very recent material on the same topic published in "PLoS ONE ". Therefore, it would be highly appreciated if you would check the contents published in the last two years of "PLoS ONE" (https://journals.plos.org/plosone/) and add all material relevant to your article to the reference list.

===========================

We would appreciate receiving your revised manuscript by Apr 24 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Wen-Jun Tu

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

"NK reports grants from Infocom Corp, Fujitsu Ltd, Fujitsu Software Technologies, and TAK Ltd, personal fees from Occupational Health Foundation, Japan Dental Association, Sekisui Chemicals, Junpukai Health Care Center, Osaka Chamber of Commerce and Industry, outside the submitted work. The other authors declare that they have no competing interests."

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The report presents the findings from a study on the reliability and validity of the Kessler 6 scale among nurses in Vietnam. The study includes relatively large sample size for validation study and used appropriate statistical methods. However, there are some issues that need to be addressed.

1. The reference no. 2 "Kessler RC et al, 2003" was conducted with the aim of screening for Serious Mental Illness. Hence, the cut-off for estimated prevalence is high i.e. 13. It is not clear why the authors simply used the same cut-off for the detection of psychological distress. This cut-off will essentially underestimate the magnitude of mental health conditions in the population.

2. The scoring explanation given in lines 107 - 109, "5=all of the time"... contradicts with the information in the following sentence. Please make it clear how the responses were scored.

3. Lines 127 - 128, the pilot study was done in 150 nurses who also participated in the subsequent validation study. This is not methodologically correct. Those 150 nurses' response could be affected by their prior experience with the questionnaire.

4. If the aim is to use the K6 in community samples to screen for psychological distress, the validation conducted in health professionals is less informative. Nurses are more likely to be familiar with questions related to psychological symptoms than the general population.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Markos Tesfaye

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

21 Mar 2020

Response to Reviewers (PONE-D-19-34231)

To Dr Wen-Jun Tu, Academic Editor

Thank you very much for your positive evaluation and valuable comments/suggestions from a reviewer for our manuscript submitted to Plos One, titled ‘Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam’ (PONE-D-19-34231). We carefully studied the comments from you (the academic editor) and the reviewer and prepared a “Response to Reviewers” letter responding to these, together with a marked up and unmarked copies.

1. Following your recommendation, we add one article published in the last two years of "PLoS ONE", that is relevant to our study:

van Heyningen T, Honikman S, Tomlinson M, Field S, Myer L. Comparison of mental health screening tools for detecting antenatal depression and anxiety disorders in South African women. PLoS One. 2018;13(4):e0193697. doi: 10.1371/journal.pone.0193697.

2. We deposit our data at figshare.com: https://doi.org/10.6084/m9.figshare.12005511.v1

Please kindly update our Data Availability Statement.

3. Based on your recommendation, we studied “protocols.io” to register our laboratory protocols, However, we found that this is not very relevant to our study. Thus we do not register our protocol. We hope that this is acceptable.

To Dr. Markos Tesfaye, the reviewer #1

Reviewer #1: The report presents the findings from a study on the reliability and validity of the Kessler 6 scale among nurses in Vietnam. The study includes relatively large sample size for validation study and used appropriate statistical methods. However, there are some issues that need to be addressed.

RESPONSE: Thank you for your valuable time and effort for reviewing on our manuscript and giving us valuable comments. The below is our responses to the comments. Please also see revisions in the manuscript that are highlighted in yellow color in the marked-up copy in a separate file.

1. The reference no. 2 "Kessler RC et al, 2003" was conducted with the aim of screening for Serious Mental Illness. Hence, the cut-off for estimated prevalence is high i.e. 13. It is not clear why the authors simply used the same cut-off for the detection of psychological distress. This cut-off will essentially underestimate the magnitude of mental health conditions in the population.

RESPONSE: We agree that using the cut off score of 13+ proposed by Kessler et al., 2003 may underestimate psychological distress. We deleted the presentation of the proportions of respondents with K6 score of 13+ from Table 2, and related descriptions from the Method and Discussion. Instead, we add an additional analysis of screening performance of K6 predicting cases with common mental disorders defined by the cut-off scores of DASS21. The calculated best cut-off score of K6 was 9+, with 129 (12.4%) of the participants meeting the criteria. This would provide readers with more insights on the prevalence of psychological distress in this population. We added the related descriptions in the Methods, Results, and Discussion.

“A screening performance of K6 for identifying a CMD (i.e., depressive disorders and anxiety disorders) was also examined. Cases with CMD were defined as those who had 34 or greater scores of the total DASS21, that was the best cut-off score found in a previous study in Vietnam [34]. An area under the curve (AUC) of a receiver operative curve (ROC) and the 95% CIs were calculated predicting these cases with CMD on the sore of K6. An optimum cut-off score was determined based on the best Youden index (a sum of sensitivity and specificity), and the sensitivity, specificity, and positive likelihood ratios were calculated with their standard errors [44].” (page 8-9, Methods)

“Among the participants, 22 cases were identified as having CMD (i.e., 34 or greater scores of the total DASS21). The AUC predicting CMD on K6 score was 0.919 (95% CIs, 0.855-0.984). The best cut-off score for K6 was identified as 9+, with 129 (12.4%) of the participants having the scores equal to or greater than this cut-off. The sensitivity and specificity were 0.864 (SE, 0.073) and 0.876 (SE, 0.031), respectively, with the positive likelihood ratio of 6.96 (SE, 0.121).” (page 13, Results)

“The screening performance of K6 score predicting CMD defined by using DASS21 was high: the AUC was 0.919, with the sensitivity of 0.864 and the specificity of 0.876 for the best cut off score of 9+. The finding is similar to one from a previous study in Japan that the stratum-specific likelihood ratio became higher (>11) for the cut off scores of 9+ on K6 [46]. However, the screening performance and the best cut off score of the Vietnamese version of the K6 should be assessed by comparing people who have or have not been previously diagnosed with mental disorders.” (page 18, Discussion)

2. The scoring explanation given in lines 107 - 109, "5=all of the time"... contradicts with the information in the following sentence. Please make it clear how the responses were scored.

RESPONSE: We are very sorry that the description was wrong. We corrected this as follows sticking to the original scoring method.

“(4=All of the time, 3=Most of the time, 2=Some of the time, 1=A little of the time, and 0=None of the time)” (page 5, Methods – Measures)

3. Lines 127 - 128, the pilot study was done in 150 nurses who also participated in the subsequent validation study. This is not methodologically correct. Those 150 nurses' response could be affected by their prior experience with the questionnaire.

RESPONSE: This is a quite important point and a large limitation of our study. We added a description on this problem in the Discussion:

“Finally, 150 nurses who responded to a pilot study were also invited to the main validation study; most of them probably actually participated. Because the pilot study was anonymous, it was impossible to conduct an analysis excluding those who participated in the two surveys. Repeated surveys have been reported to result in attenuated responses to anxiety symptoms of respondents [48], which may affect the finding of the present study.” (page 18, Discussion)

4. If the aim is to use the K6 in community samples to screen for psychological distress, the validation conducted in health professionals is less informative. Nurses are more likely to be familiar with questions related to psychological symptoms than the general population.

RESPONSE: We primarily aimed to validate the K6 scale to screen psychological distress among nurses in Vietnam. However, it should be noted that the validation of the current Vietnamese version of K6 need to be done in other occupational groups or the community population if the scale is to be applied to these groups/populations. We expanded our limitation description to make this clear to readers.

“First, we aimed to validate the K6 scale to screen psychological distress among nurses, a special occupational group under high stress, in Vietnam. Nurses are usually more educated than general workers, and also familiar with questions of psychological symptoms included in K6. The present findings may not be generalized to the general population or working population in Vietnam. The reliability and validity of the present scale should be further tested in these populations.” (page 18, Discussion)

Submitted filename: PlosOne-Response to Reviewers.docx

Click here for additional data file.

6 Apr 2020

PONE-D-19-34231R1

Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam

PLOS ONE

Dear Dr. Kawakami,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

In order to provide a more complete information to our readers on the topic, we would like to emphasize the importance to cross referencing very recent material on the same topic published in "PLoS ONE ". Therefore, it would be highly appreciated if you would check the contents published in the last two years of "PLoS ONE" (https://journals.plos.org/plosone/) and add all material relevant to your article to the reference list.

==============================

We would appreciate receiving your revised manuscript by May 21 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Wen-Jun Tu

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The comments on the previous version have been satisfactorily addressed. It is good that the limitations are now elaborated in the discussion.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Markos Tesfaye

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

27 Apr 2020

To Dr Wen-Jun Tu, Academic Editor

Thank you very much for your positive evaluation of our manuscript submitted to Plos One, titled ‘Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam’ (PONE-D-19-34231R1). Following your recommendation, we add one more article published in "PLoS ONE", that is relevant to our study:

Choi SK, Boyle E, Burchell AN, Gardner S, Collins E, Grootendorst P, et al. Validation of six short and ultra-short screening instruments for depression for people living with HIV in Ontario: results from the Ontario HIV Treatment Network Cohort Study. PLoS One. 2015;10(11). doi: https://doi.org/10.1371/journal.pone.0142706.

We hope that the manuscript is now acceptable for the publication in Plos One.

Thank you.

Norito Kawakami, MD

On behalf of the authors

Submitted filename: PlosOne-Response to Reviewers.docx

Click here for additional data file.

29 Apr 2020

Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam

PONE-D-19-34231R2

Dear Dr. Kawakami,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Wen-Jun Tu

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

7 May 2020

PONE-D-19-34231R2

Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam

Dear Dr. Kawakami:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Wen-Jun Tu

Academic Editor

PLOS ONE