Comparison of the Diagnostic Accuracy and Validity of a Short Version of Teen Screen Questionnaire-Mental Health (TSQ -M-Short) for Use in Community.

BACKGROUND: A few self-administered questionnaires are available for assessing mental health among adolescents in primary-care settings. Brief measures are desirable for use in big-data, epidemiological studies.
OBJECTIVES: To evaluate a 7-item version, of the Teen Screen Questionnaire-Mental Health (TSQ-M), the TSQ-M-Short.
MATERIALS AND METHODS: In this prospective cross-sectional study of 140 adolescents, recruited from 6 rural or urban schools, the newly developed TSQ-M-Short as the measure for validation and General Health Questionnaire-12 item (GHQ-12) as the gold standard measure were administered by independent trained raters. Tests for diagnostic accuracy and validity were conducted.
RESULTS: A total TSQ-M-Short score of ≥ 6 had a sensitivity of 76%, specificity of 74%, positive likelihood ratio of 2.99, negative likelihood ratio of 0.33, positive predictive value of 6% and a negative predictive value of 82.1%. The area under curve (AUC) in the Receiver Operating Characteristic (ROC) for the TSQ-M-Short version was 0.84 (95% cumulative incidence (CI) = 0.76-0.89). The AUC for the TSQ-M-Short version was higher than the AUC for the original version, and the difference between the areas was 0.10 (95% CI = 0.02-0.19), which was statistically significant (z = 2.49; P = 0.01). The internal consistency of TSQ-M-Short, as measured by chronbach's α, was 0.34 (95% CI = 0.15-0.48). The construct validity demonstrated a 3-factor structure, which explained 55% of the variance.
CONCLUSION: The TSQ-M-Short has an overall diagnostic accuracy which is better than the original TSQ-M. Although the original version includes symptoms for more mental health disorders, providing a wider screen. This short version will prove useful in big-data studies.

INTRODUCTION

Despite the adolescent mental health problems being widely prevalent,[1] the diagnostic methods are quite disorganized in primary-care settings,[2] resulting in poor identification in the community.[3] Among different methods available to improve the diagnostic accuracy in mental health, using questionnaires are relatively inexpensive, can be used by health as well as non-health professionals,[4] and thus forms the central pillar for mental health research among adolescents in primary-care. However, brief questionnaires have been documented to reduce respondent burden,[5] improve the response rates[6] and eventually the quality of data collected.[7] Thus, short versions of existing measures are invaluable in conducting big-data studies in primary-care settings. Recently, we had validated the original 21-question version of Teen Screen Questionnaire-Mental Health.[8] In this study, we have developed and validated a self-administered, brief version (7-question) of the questionnaire named Teen Screen Questionnaire-Mental Health- Short (TSQ-M-Short), which identifies mental ill health over the previous 1-year for big-data epidemiological studies.

MATERIALS AND METHODS

This study was conducted as part of the Adolescent Health District Plan (AHDP) Project with the support of National Rural Health Mission (NRHM) Kerala state. Data was collected from 3 schools from Thiruvananthapuram city and 3 schools from the rural taluks of Thiruvananthapuram district (Chirayinkeezhu, Nedumangad and Neyyattinkara). Schools from both these settings were randomly selected and students, of both genders, from 9th to 12th standard were included in the study if they gave verbal assent to their participation. Prior to the data collection, written permission to conduct the study was obtained from the District Educational Office and the project was approved by the Institutional Review Board of the Child Development Centre, Thiruvanthapuram. The confidentiality of the data was protected with reversible anonymization and by limiting the availability of the data to only the primary investigator and who did the statistical analysis.

Measures

Teens Symptoms Questionnaire-Mental Health-7 (TSQ-M-7) is a 7-item version of Teens Symptoms Questionnaire-Mental Health (TSQ-M).[8] The seven items, of TSQ-M-Short, had the highest factor endorsement TSQ-M or clinical relevance and were culled to form this short version of the measure. Like in the original version, the endorsement pattern for the TSQ-M short version was also a 3-point likert scale of ‘Never’, ‘Sometimes’ and ‘Often’ to minimize endorsement related error in big data studies. This abridged version (TSQ-M-short) was the index measure for studying the diagnostic accuracy and comparing with the original version in this study. General Health Questionnaire-12 item[9] is a 12-item measure of current mental health among adults and adolescents. This gold standard, self-reported measure of psychological morbidity, detects psychiatric disorders in community settings and non-psychiatric settings. GHQ-12 cut-off of 2/3 (sensitivity = 87.4% and specificity = 79.2%) was used to define ‘caseness’ in this study as suggested for the Indian population.[10]

Data collection was done by qualified Clinical Child Developmental therapist trained in using the TSQ-M-Short and GHQ-12. The data was collected with face-to-face interviews with the adolescent by these trained auxiliary health professionals using TSQ-M-Short and GHQ-12, independently on the same day to minimize the rater bias as well as maximize the stability of the rating with time.

Statistical analysis

Sensitivity, specificity, likelihood ration and predictive values for various TSQ-M and TSQ-M-Short cut-off scores were calculated in order to determine the optimal screening threshold with Receiver Operating Characteristic (ROC) analyses, against the GHQ-12 cut-off of ≥2/3. We selected the optimal cut-off scores for screening that satisfied both sensitivity and specificity criteria (highest Youden index) for mental ill health using ROC analysis. To assess the diagnostic accuracy of each version of TSQ-M, we compared the Area Under the Curves (AUCs) of TSQ-M short version and TSQ-M original version by calculating the critical ratio ‘z’ (Hanley and McNeil, 1983). Multiple linear regression analysis with step-wise variable selection assessed the relative contributions of age, education and gender to TSQ-M short version and TSQ-M long version. The internal consistency of the measure was calculated using Chronbach's α and factor structure with Exploratory Factor analysis (principal component analysis with promax rotation). The items were excluded if they failed to load on any factor (loading <0.40) or had unacceptably high secondary loadings/cross loading (>0.40). All statistical analyses were performed using Statistical Package for the Social Sciences (SPSS) (version 19.0) and MedCalc (version 12.2.1).

RESULTS

Among the 140 participants, 41% and 58% were boys and girls, respectively. The mean (SD) age of the teenagers was 15.60 (3.48) years. Most of the participants were from a nuclear family (66%) and others were from extended (20%) and joint families (14%). The TSQ score in the study population ranged from 5 to 16.

A total TSQ score of ≥6 had a sensitivity of 76%, specificity of 74%, positive likelihood ratio of 2.99, negative likelihood ratio of 0.33, positive predictive value of 6% and a negative predictive value of 82.1% making it appropriate for screening in the primary-care settings. The AUC in the ROC for the TSQ-M-Short version was 0.84 (95% CI = 0.76 to 0.89). The AUC for the TSQ-M-Original version was 0.73 (95% CI = 0.65 to 0.80). The AUC for the TSQ-M-Short version was higher than the AUC for the original version, and the difference between the areas was 0.10 (95% CI = 0.02-0.19), which was statistically significant (z = 2.49; P = 0.01) [Figure 1].

Figure 1

The comparison of area under curve (AUC) for Teen Screen Questionnaire-Mental Health (TSQ-M) Short version and TSQ-M Original version against General Health Questionnaire-12 (GHQ-12) as the reference standard Difference between the AUCs = 0.10 (95% cumulative incidence (CI) = 0.02-0.19); z = 2.49; P = 0.01

The internal consistency, as measured by chronbach's α, was 0.34 (95% CI = 0.15-0.48). For further investigation of the construct validity, we explored the factor structure of the items in the TSQ-M as the Kaiser-Meyer-Olkin measure of sampling adequacy was 0.53 and Barlett's test of sphericity was significant (P = 0.001). We extracted those factors with an eigen value of 1, and thus a 3-factor structure was derived explaining 55% of the variance [Table 1].

Table 1

The factor structure of Teen Screen Questionnaire-Mental Health (TSQ-M-Short) version

DISCUSSION

This brief version of the scale (TSQ-M-Short) has an overall diagnostic accuracy (AUC = 0.84) that is statistically significantly better than the original version of the scale (AUC of 0.79) (Nair et al., 2014).[8] This improved diagnostic accuracy will enhance the identification of mental ill-health among adolescents in primary-care settings and in conducting big-data studies. Like the TSQ-M, Strength and Difficulties Questionnaire, Indian Adolescent Health Questionnaire and Patient Health Questionnaire the construct this brief version measures is mental ill health. However, the TSQ-M-Short encompasses a fewer number of Priority Mental Health Disorders of adolescence than its original version.

The diagnostic accuracy of TSQ-M-Short at the cut-off score of ≥6 has adequate sensitivity and specificity to be used as a screening instrument in the primary-care [Table 2]. The internal consistency of 0.34, which was lower then the original version of the measure and thus suggestive of multiple sub-constructs within the construct of mental ill-health as measured by TSQ-M-Short. This was theoretically anticipated because of inclusion of symptoms of mood and anxiety disorders, biological and impairment symptom clusters in the questionnaire. This was further proved by the 3-factor structure in the construct of mental ill-health by TSQ-M-Short. The strength of our study is that the validation methodology followed the guidelines as given COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) protocol[11] for validation and STAndards for the Reporting of Diagnostic accuracy studies (STARD) guidelines[12] for diagnostic accuracy respectively.

Table 2

The diagnostic accuracy of Teen Screen Questionnaire-Mental Health (TSQ-M) short version

CONCLUSION

In conclusion TSQ-M-Short is an easy to use measure with adequate psychometric properties for conducting big-data studies in the primary-care settings.