Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans.

To validate the Computerized Adaptive Test Suicide Scale (CAT-SS), Veterans completed measures at baseline (n = 305), and 6- (n = 249), and 12-months (n = 185), including the CAT-SS (median items 11, duration of administration 107 seconds) and the Columbia-Suicide Severity Rating Scale (C-SSRS). Logistic regression was used to relate CAT-SS scores (baseline) to C-SSRS assessed outcomes (active ideation with plan and intent; attempt; interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors; all outcomes combined). A mixed-effects logistic regression model was used to evaluate the relationship between the lagged CAT-SS scores and outcomes (6- and 12-months). The baseline CAT-SS demonstrated predictive accuracy for all outcomes at 6-months, and similar results were found for baseline and all outcomes at and through 12-months. Longitudinal analysis revealed for every 10-point change in the CAT-SS there was a 50-77% increase in the likelihood of suicide-related outcomes. The CAT-SS demonstrated added value when compared to current suicide risk prediction practices.

Introduction

In the United States, rates of suicide have been increasing among military and civilian cohorts [1, 2]. According to work by Ahmedani et al. [3], almost 30% of individuals who died by suicide had a healthcare visit in the week prior to their death. Recognizing the importance of risk screening within healthcare systems, in 2016 the Joint Commission released a Sentinel Event Alert, which recommended that universal suicide risk screening be implemented [4]. Ideally, such efforts would facilitate identification of those with occult risk (individuals who may disclose suicidal thoughts and behaviors only if they are directly asked) who may not be engaged in mental health treatment [5]. Nonetheless, options and evidence regarding tools which can be used to facilitate universal risk screening remain limited [4].

As screening for depression frequently occurs in primary care settings, often using the Patient Health Questionniare-9 (PHQ-9) [6], efforts to evaluate the utility of the PHQ-9 as a suicide risk screener have been undertaken. However, likely related to the measure only containing one item (item 9) specifically focused on suicidal ideation (“bothered by thoughts of being dead or of hurting yourself in some way”), as well as the reality that a sizable number of individuals’ risk for suicide is related to factors other than depression (e.g., chronic pain, anxiety), results have been mixed. In specific, data regarding psychometric properties (e.g., positive predictive value) have been less than ideal [6-8]. Moreover, results from most rapid screeners like the PHQ-9 item 9 [6] often do not provide the clinician with information regarding risk severity or magnitude [7].

In addition, many suicide risk screening measures (e.g., the Columbia Suicide Rating Scale (C-SSRS)-Screener) [9] include items solely focused on suicidal ideation and behavior; thereby limiting the ability to measure “the full spectrum of suicidal symptomatology” [7; pp. 1376]. Ideally, suicide risk screening approaches would incorporate personalized items associated with a range of risk factors. Tailoring screening measures while maintaining psychometric properties requires implementation of novel approaches such as computerized adaptive testing (CAT) based on unidimensional or multidimensional item response theory (M/IRT). Traditional mental health measures are based on classical test theory, where all respondents receive all items and which are equally weighted in terms of deriving the test score, which is the often the summation of the individual item scores, rated either dichotomously or as polytomous Likert scale items. In contrast, IRT-based CAT uses unidimensional or multidimensional item response theory (MIRT) to pre-calibrate a large “bank” of symptom-items, that are then adaptively selected to match the severity of the person’s disorder, which is adaptively estimated from the responses to prior items administered [7; pp. 1377]. As a result, different items are administered to different respondents, targeted to their level of severity on the underlying construct of interest (in our case suicide risk). For further information regarding MIRT-based CAT see Gibbons et al., 2008 [10] and Gibbons et al., 2016 [11].

Thus, Gibbons and colleagues developed and conducted an initial validation study on the Computerized Adaptive Test-Suicide Scale (CAT-SS) [7]. Using data from individuals receiving outpatient psychiatric treatment, the team was able to calibrate the CAT-SS, and demonstrate that the CAT-SS measured suicide risk severity using a mean of 10 items, in under two minutes. Moreover, initial validity was demonstrated comparing CAT-SS and C-SSRS structured clinical interview results among those seeking care in two non-Veterans Affairs emergency departments (University of Chicago and University of Massachusetts). Contrasting the CAT-SS high-risk group to the no-risk group a sensitivity of 1.0 and specificity of 0.92 were found for the C-SSRS active ideation category. Per the authors, additional prospective validation efforts, including prediction of future suicidal events, were warranted. Towards this end, members of this team conducted a longitudinal study among Veterans eligible for Veterans Health Administration (VHA) care to validate the CAT-SS self-report measure in terms of its ability to predict future suicide events based on repeated C-SSRS clinical interviews at 6-months and 12-months following the baseline CAT-SS assessment.

Methods

Participants

This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving human participants were approved by the Colorado Multiple Institutional Review Board (COMIRB). Participants (n = 305) were recruited from a mountain state metropolitan VA health care system between April 2017 and February 2019. Recruitment strategies included posting flyers at local facilities, contacting Veterans who had participated in previous research or who indicated interest in participating in research, and encouraging providers to tell patients about the study. Veterans were eligible if they were between the ages of 18 and 89 and able to provide written informed consent, which was obtained. The number of veterans who completed measures at each timepoint is as follows: baseline, n = 305; 6-month follow-up, n = 249; 12-month follow-up, n = 185.

Measures

Computerized Adaptive Test-Suicide Scale (CAT-SS) [7]* is an adaptive measure, comprised of 111-items, which dimensionally measures suicide risk severity on a 100-point scale with 5 points of precision. The scores are also thresholded to yield categories of low, moderate, and high risk.

The Columbia-Suicide Severity Rating Scale (C-SSRS) [9]* is a clinician-administered interview used to evaluate suicidal ideation (including intensity) and suicide-related behavior (e.g., preparatory, attempt).

Structured Clinical Interview for DSM-5 Disorders (SCID-5) Research Version [12] is a reliable and valid semi-structured interview used to diagnose Axis I psychiatric disorders in clinical and research settings. The SCID-5 was used to determine current presence of the following disorders: Bipolar I and II; Major Depressive; Alcohol Use; Substance Use, Generalized Anxiety; and, Sleep. The trauma/PTSD L Module of the Structured Clinical Interview for SCID-5 [12] was used to assess Criterion A events. If a Criterion A event and at least one current symptom was endorsed, the Clinician-Administered PTSD Scale for DSM-5 CAPS-5 was administered. The CAPS-5 is the gold standard for assessing PTSD, and was used to determine current PTSD diagnosis [13].

Rocky Mountain MIRECC Demographic Questionnaire was used to gather information on topics such as participant age, gender, race/ethnicity, education, period of military service, and combat exposure.

*Measures administered at baseline, and 6- and 12-month follow-up appointments.

Procedures

Data were collected at three timepoints (baseline, 6- and 12-month follow-up). After confirming eligibility, Veterans were invited to an in-person baseline study visit. Informed consent was obtained prior to administration of clinical interviews listed above, self-report measures (not included in this study), and the CAT-SS. Study team members were clinically trained to administer the measures and interview schedules were reviewed by licensed clinicians.

To facilitate retention, participants were re-contacted at approximately 6 months post the baseline study visit and offered an in-person or telephone visit. During this visit, the CAT-SS was re-administered. In addition, reminder letters to invite completion of the 12-month follow-up were sent 1–3 months prior to their 12-month window to promote retention. The final in-person study visit was conducted approximately 12 months following the baseline assessment, and the CAT-SS was again re-administered. Participants were compensated for all study visits. Two Veterans who had incomplete data at the 6-month visit and one at the 12month visit were removed from analyses. Reasons for attrition were not collected, however, Veterans were invited to complete the 12-month follow-up regardless of their completion of their 6-month follow-up. The final sample size for analysis was n = 265.

Statistical analyses

Logistic regression was used to relate the CAT-SS scores at baseline to the C-SSRS assessed outcomes (active ideation with plan and intent; attempt; interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors; all outcomes combined) at 6 months, and the CAT-SS scores at baseline and 6-months to the outcomes at 12-months, and all events between baseline and 12-months. From the logistic regression model, we generated a receiver operating characteristic (ROC) curve and computed the area under the ROC curve (AUC). We also examined the unique contribution of the CAT-SS in predicting suicide-related outcomes over and above what has traditionally been considered a robust predictor, a suicide attempt within the past year. To test this, logistic regression models with: (a) previous suicide attempt in the past year; (b) the CAT-SS; and, (c) previous suicide attempt in the past year and the CAT-SS were fitted to these data and the AUCs statistically compared.

To study longitudinal trends in C-SSRS assessed outcomes (active ideation with plan and intent; attempt; interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors; all outcomes combined), a mixed-effects logistic regression model was used to perform a longitudinal analysis of the relationship between the lagged CAT-SS scores and suicide-related outcomes at 6 and 12 months (i.e., CAT-SS score at baseline predicting suicide-related outcomes at month 6 and CAT-SS at month 6 predicting suicide events at 12 months). CAT-SS scores were divided by 10 so that the odds ratios were interpretable as the relationship between a 10-point change in CAT-SS (on a 100-point scale) and the likelihood of a suicide-related outcome. Separate analyses were conducted for each outcome, with and without adjustment of suicide attempt in the past year.

This study was powered to estimate an AUC of 0.8 with a 95% confidence interval of plus or minus 5%. Assuming an event rate of 10%, n = 250 subjects at the 6-month follow-up were required. A total of n = 247 subjects completed the CAT-SS at the 6-month follow-up.

Results

Demographic characteristics of the study sample at baseline are presented in Table 1. Mental health diagnoses (current) at baseline as determined by administration of the SCID-5 [12] included: Bipolar Disorder I and II (3.9%), Major Depressive Disorder (26.6%), Alcohol Use Disorder (8.9%), Substance Use Disorder (9.2%), Generalized Anxiety Disorders (3.6%), and Sleep Disorders (6.9%). Current PTSD was determined by responses to the CAPS-5 [13], with 28.5% of the sample meeting PTSD criteria (n = 87).

Table 1

Sample demographic characteristics and current mental health conditions.

	Baseline n = 305*	Participants with at Least 1 Follow-up Visit n = 265*
Age	47.2 ± 12.6	47.4 ± 12.6
	47 (22–77)	47 (22–77)
Male	247 (81.0%)	215 (81.1%)
Race	n = 305	n = 265
Caucasian/White	221 (72.5%)	192 (72.5%)
Black or African American	50 (16.4%)	42 (15.9%)
Native American/Alaskan Native	4 (1.3%)	4 (1.5%)
Asian or Pacific Islander	3 (1.0%)	3 (1.1%)
Multiracial/Other	27 (8.8%)	24 (9.1%)
Ethnicity	n = 305	n = 265
Hispanic or Latino/a	45 (14.8%)	37 (14.0%)
Education	n = 304	n = 264
High school education	47 (15.5%)	35 (13.2%)
Some college, no degree	83 (27.3%)	72 (27.3%)
Associate’s or Bachelor’s degree	112 (36.8%)	99 (37.5%)
Graduate degree	62 (20.4%)	58 (22.0%)
Marital Status	n = 305	n = 265
Married	131 (43%)	118 (44.5%)
Single	79 (25.9%)	73 (27.6%)
Cohabitating	15 (4.9%)	12 (4.5%)
Widowed/Divorced/Separated	80 (26.3%)	62 (23.4%)
Sexual Orientation	n = 305	n = 265
Gay/Lesbian/Queer	16 (5.2%)	13 (4.9%)
Heterosexual	280 (91.8%)	244 (92.1%)
Bisexual	9 (3%)	8 (3.0%)
Employment Status	n = 303	n = 263
Employed Full-Time	88 (29%)	79 (30.0%)
Employed Part-Time	31 (10.2%)	29 (11.0%)
Unemployed, not currently seeking employment	74 (24.4%)	60 (22.8%)
Unemployed, seeking employment	37 (12.2%)	32 (12.2%)
Retired	73 (24.1%)	63 (24.0%)
Branch of Military Service	n = 304	n = 265
Army	180 (59.2%)	156 (58.8%)
Air Force	46 (15.1%)	41 (15.5%)
Navy	34 (11.2%)	31 (11.7%)
Marines	32 (10.5%)	25 (9.4%)
Multiple Branches	12 (4%)	12 (4.5%)
	n = 305	n = 265
Number of Deployments	2.1 ± 3.5	2.2 ± 3.7
	1 (0–40)	1 (0–40)
Number of Deployments to Combat Zone	1.1 ± 1.8	1.2 ± 1.9
	1 (0–20)	1 (0–20)
Years of Military Service	9.4 ± 7.5	9.4 ± 7.5
	6.2 (0.5–39)	6.4 (0.5–39)
Current Mental Health Conditions	n = 305	n = 265
Bipolar Disorders	12 (3.9%)	11 (4.2%)
Major Depressive Disorder	81 (26.6%)	69 (26.0%)
Alcohol Use Disorder	27 (8.9%)	25 (9.4%)
Substance Use Disorder	28 (9.2%)	24 (9.1%)
Generalized Anxiety Disorder	11 (3.6%)	10 (3.8%)
Sleep Disorders	21 (6.9%)	20 (7.6%)
Post-Traumatic Stress Disorder	87 (28.5%)	71 (26.8%)

* n (%) or (Mean ± SD; Median (Range))

Administration of the CAT-SS resulted in a median administration time of 107 seconds with median administration of 11 items to meet a precision threshold less than 5.0 points on the 100 point scale. At baseline, using CAT-SS thresholds [7], 137 (51.6%) of the participants were categorized as being at low, 125 (47.3%) at moderate, and 3 (1.1%) at high risk. Per the baseline C-SSRS, 91 (29.8%) had lifetime active ideation with a plan and intent, and 97 (32.0%) had a lifetime attempt. Data from the C-SSRS across all three study visits (baseline, 6-month, 12-month) are presented in Table 2.

Table 2

Columbia-Suicide Severity Rating Scale outcomes* at baseline, and 6 and 12 month visits.

	Baseline Visit n = 305		Participants with at Least One Follow-up Visit n = 265		6 Month Visit n = 247	12 Month Visit n = 184
Outcome	Lifetime n (%)	Past 3 Months n (%)	Lifetime n (%)	Past 3 Months n (%)	Since Last Assessment n (%)	Since Last Assessment n (%)
Active ideation with plan and intent	91 (29.8%)	14 (4.6%)	76 (28.7%)	13 (4.9%)	5 (2.0%)	4 (2.2%)
Attempt	97 (31.8%)**	19 (6.2%)	82 (30.9%)***	16 (6.0%)	3 (1.2%)	4 (2.2%)
Interrupted, Aborted or self-interrupted attempt, or Preparatory acts or behaviors	107 (35.1%)	24 (7.9%)	91 (34.3%)	20 (7.5%)	8 (3.2%)	9 (4.9%)
Combination of the Above	142 (46.6%)	38 (12.5%)	122 (46.0%)	32 (12.1%)	11 (4.5%)	10 (5.4%)

*Outcomes are not mutually exclusive

**n = 303

***n = 263

As a continuous measure the CAT-SS was strongly associated with suicide-related outcomes (active ideation with plan and intent; attempt; interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors; and all outcomes combined) over a 12-month period, with the strength of the associations increasing with repeated longitudinal assessments (see Table 3). Analyses were also conducted to study the added predictive accuracy of the CAT-SS for future suicidal events, above and beyond the predictive accuracy of a suicide attempt within the past year. Findings suggested large increases in AUC for all 4 outcomes (active ideation with plan and intent chi-square = 15.80, df = 1, p<0.0001; attempt chi-square = 5.78, df = 1, p = 0.02; interrupted, aborted or self-interrupted attempt or preparatory acts or behaviors chi-square = 17.92, df = 1, p<0.0001; and, all outcomes combined chi-square = 9.86, df = 1, p<0.002). The ROC curves for active ideation with plan and intent, and attempt for past year suicide attempt, CAT-SS, and past year suicide attempt and CAT-SS are displayed in Figs 1 and 2.

Table 3

Areas under the curve by Columbia-Suicide Severity Rating Scale outcomes for the Computerized Adaptive Test Suicide Scale.

	AUC 95% (Confidence Intervals)
6 Months
Active ideation with plan and intent	0.81 (0.74, 0.87)
Attempt	0.65 (0.19, 0.99)
Interrupted, aborted or self-interrupted attempt, or Preparatory acts or behaviors	0.72 (0.52, 0.91)
Combination of the above	0.74 (0.60, 0.89)
6–12 Months
Active ideation with plan and intent	0.91 (0.77, 0.99)
Attempt	0.72 (0.46, 0.98)
Interrupted, aborted or self-interrupted attempt, or Preparatory acts or behaviors	0.82 (0.69, 0.96)
Combination of the above	0.82 (0.70, 0.94)
All 12 Months *
Active ideation with plan and intent	0.84 (0.77, 0.91)
Attempt	0.77 (0.58, 0.95)
Interrupted, aborted or self-interrupted attempt, or Preparatory acts or behaviors	0.83 (0.74, 0.91)
Combination of the above	0.81 (0.73, 0.89)

*Individual with an event at either 6- or 12-months

Fig 1

Receiver operating characteristic curve for attempt in the past year and the Computerized Adaptive Test Suicide Scale, and both predicting active ideation with plan and intent over 12 months.

Fig 2

Receiver operating characteristic curve for attempt in the past year and the Computerized Adaptive Test Suicide Scale, and both predicting actual attempt over 12 months.

Longitudinal analysis of these data revealed that for every 10-point change in the CAT-SS score there was between a 50 and 77% increase in the likelihood of a suicidal event across the 4 outcomes, all of which were statistically significant, or a 5-fold to almost 8-fold increase over the range of the scale. Moreover, adjusting for suicide attempt in the past year, revealed similar strong associations between the CAT-SS and suicidal event outcomes ranging from 36 to 73% or 4-fold to 7-fold increase across the range of the scale (see Table 4).

Table 4

Likelihood of a suicidal event per 10-point change in CAT-SS score.

Outcome	Odds Ratio (95% CI) Unadjusted	p-value	Odds Ratio (95% CI) Adjusted for Past Year Suicide Attempt	p-value
Active ideation with plan and intent	1.50 (1.02, 2.19)	0.04	1.36 (0.92, 2.03)	0.13
Attempt	1.77 (1.23, 2.54)	0.002	1.73 (1.14, 2.63)	0.01
Interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors	1.74 (1.25, 2.41)	0.0009	1.59 (1.14, 2.21)	0.006
Combination of the above	1.63 (1.18, 2.25)	0.003	1.55 (1.12, 2.15)	0.009

Discussion

To address the pressing public health problem of suicide, efforts must be aimed at validating measures that can be used to evaluate suicide risk in both primary and specialty care medical settings. Ideally, such measures would be rapidly administered (e.g., self-report) via an electronic platform, and personalized to individual patients. Among Veterans seeking care at a VAMC, the CAT-SS assessed suicide risk severity with a median of 11 items in under two minutes (107 seconds); thereby highlighting feasibility of administration similar to that identified among the initial validation cohort (11 items and 110 seconds) [7].

Moreover, results revealed that CAT-SS scores were strongly associated with future suicide-related outcomes over the 12-month study period. Although results, in terms of such associations, were similar at 6- and 12-months, the strength of associations increased with repeated CAT-SS assessment. These findings highlight the utility of the CAT-SS for both initial identification and continued monitoring of risk. Longitudinal analysis also revealed that for every 10-point change in the CAT-SS score there was between a 50 and 77% increase in the likelihood of a suicidal event across the 4 outcomes, all of which were statistically significant, or a 5-fold to almost 8-fold increase over the range of the scale.

Previous research has shown that history of suicide attempt is one of the most significant risk factors for suicide [14]. Similarly, when clinicians were asked about factors which they considered “most important” in assessing suicide risk, they weighed the presence of suicide-related behaviors (e.g., preparatory behavior) as well as a history of attempts more heavily than other factors [15]. In fact, prior history of suicide attempt is strongly recommended as one of the risk factors that should be assessed as part of a comprehensive suicide risk evaluation in the Departments of Veterans Affairs and Defense Clinical Practice Guideline for the Assessment and Management of Suicidal Behavior [14]. Thus, a critical marker of validity for any suicide risk measure is the degree to which it can predict future suicidal events when compared with other empirically robust variables, such as suicide attempt history. That is, the measure should increase the ability to predict future suicidal behavior, above and beyond known epidemiologic risk factors (e.g., history of a suicide attempt). In this study, CAT-SS scores outperformed history of suicide attempt in the past year as a predictor of future suicide-related thoughts and behaviors. As highlighted above, statistically significant increases in AUC were found in models that that added CAT-SS results to a model that only included a history of suicide attempt; thereby illustrating the added value of the CAT-SS over traditional predictive models based on past suicidal behavior only.

Recently, the CAT-SS has shown to be unbiased in a sample of 1,073 sexual and gender minority youth, mean age 20.3 years (SD = 3.2) [16], and to predict future suicidal events (ideation; plan; ideation, plan or attempt). Similar to our study, the CAT-SS improved predictive accuracy over traditional self-reports of ideation from an AUC of 0.70, 95% CI (0.64, 0.76) to AUC = 0.85, 95% CI (0.79, 0.90); suicide plan from AUC of 0.65, 95% CI (0.56, 0.73) to AUC = 0.84, 95% CI (0.77, 0.92); and, ideation, plan, or attempt from AUC = 0.71, 95% CI (0.65, 0.77) to AUC = 0.83, 95% CI (0.78, 0.88), all of which were statistically significant improvements in fit. The full model that included demographic characteristics, previous suicidal events, and the CAT-SS at baseline predicted suicidal ideation (AUC = 0.86, 95% CI (0.82, 0.91)), suicide plan (AUC = 0.86, 95% CI (0.80, 0.92)), and ideation, plan, or attempt (AUC = 0.84, 95% CI (0.79, 0.89)) at 6 month follow-up. Berona et al. [17] conducted a separate analysis of these data and showed predictive validity of the baseline CAT-SS in predicting time to suicide attempt during 6 months (HR = 1.34, 95% CI (1.03, 1.74)) overall and HR = 1.51, 95% CI (1.06, 2.15) for the transition from suicidal ideation to suicide attempt for each 10 point increment on the CAT-SS. These findings are remarkably similar to the findings of our study in a very different sample and age group, demonstrating the generalizability and robustness of our results.

These findings have important clinical implications for suicide risk screening across healthcare settings. The VHA has developed and implemented an enterprise-wide evidence-informed approach to suicide risk screening and evaluation, VA Suicide Risk Identification process (VA RISK ID) [5]. Currently, universal screening is being implemented using the C-SSRS Screener. However, findings from this study provide compelling evidence regarding both the efficiency and the long-term predictive validity of the CAT-SS in a medically diverse patient population. Further research is warranted to evaluate whether the CAT-SS could be feasibly implemented as part of universal screening efforts like the VA RISK ID [5], and whether CAT-SS dimensional scores could facilitate more accurate identification of suicide risk levels, while reducing patient and provider burden. Doing so, would be expected to provide additional time to facilitate personalized suicide risk-stratified care management.

As noted above, measures were administered as part of a research protocol, additional work is required to evaluate where the CAT-SS could be implemented in clinical settings. Efforts aimed at exploring this are warranted. Nonetheless, findings from this study suggest that if implemented in the electronic medical record, the CAT-SS would be expected to rapidly facilitate precise and personalized screening and assessment of suicide risk severity.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

23 Aug 2021

PONE-D-21-22855

Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans

PLOS ONE

Dear Dr. Brenner,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Oct 07 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see:  http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at  https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Sarah A. Arias

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section

3. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

4. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

5. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for an opportunity to review this interesting and well written paper reporting on Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans. The study and the mss is a valid contribution to the current knowledge. Several point and queries for authors' consideration:

1. Introduction: Can the authors clarify "results from most rapid screeners like the PHQ-9 (6) identify non-negligible suicide risk versus risk severity or magnitude (7)" and “the full spectrum of suicidal symptomatology”? Can the authors provide more information regarding the 'computerized adaptive testing (CAT)' method?

2. Methods/participants: Can the authors provide more detailed information on the study recruitment procedure? Re 'n = 305; 6-month follow-up, n = 249; 12-month follow-up, n =185' - can the authors report on reasons for drop-out from the study?

3. Statistical analyses: Can authors clarify which instrument(s) allowed to measure 'active ideation with plan and intent; attempt; interrupted, aborted or self-interrupted attempt, or preparatory acts or behaviors'?

4. Discussion: Can authors clarify 'Thus, a critical marker of validity for any suicide risk measure is the degree to which it can predict future suicidal events when compared with other empirically robust variables, such as suicide attempt history'?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

2 Nov 2021

November 2, 2021

Members of the Editorial Board:

We are pleased to resubmit the following manuscript entitled: Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans. Requests edits/actions are noted below (italics) and followed by responses.

1. It is important that you include a cover letter with your manuscript. Please ensure that this letter is addressed specifically to PLoS ONE.

A cover letter that includes the necessary elements has been submitted.

2. It appears that your ORCiD iD has not been validated in your Editorial Manager account and we are unable to proceed until that step is complete.

My ORCHID iD is : 0000-0002-2629-214X

I have attempted on a number of occasions to “fetch” my iD so that it can be entered into the system and received the following warning:

I have also sent an email to the editorial office regarding this matter.

On 10/17/21, 5:09 PM, "Brenner, Lisa" wrote:

Have resubmitted this manuscript - am getting error messages re: my orchid ID. Stating that it is associated with another account. Assistance would be super.

3. Please include your full ethics statement in the 'Methods' section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

The following has been added to the manuscript.

This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving human participants were approved by the Colorado Multiple Institutional Review Board (COMIRB).

and

Veterans were eligible if they were between the ages of 18 and 89 and able to provide written informed consent, which was obtained.

4. We note your updated Data Availability Statement: "Data cannot be shared publicly because of VA policies regarding data and security."

We have updated our Data Availability Statement.

The full dataset cannot be made publicly available due to privacy concerns and restrictions imposed by the Colorado Multiple Institutional Review Board. All relevant de-identified data are included in the manuscript. For investigators with appropriate authorizations within the Department of Veterans Affairs, requests for data access can be made to VHAECHMIRECCAdmin@va.gov.

Many thanks for your assistance.

Lisa

Lisa A. Brenner, Ph.D.

Director, VHA Rocky Mountain MIRECC

Vice Chair of Research, Physical Medicine and Rehabilitation (PM&R)

Professor of PM&R, Psychiatry, & Neurology

University of Colorado, Anschutz Medical Campus

Lisa.2.Brenner@cuanschutz.edu

303-724-4588

Submitted filename: Response to Reviewers 11 2 21.docx

Click here for additional data file.

14 Dec 2021

Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans

PONE-D-21-22855R1

Dear Dr. Brenner,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Sarah A. Arias

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you, the authors have addressed my queries and suggestions. More details as per above tick boxes.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

20 Dec 2021

PONE-D-21-22855R1

Validation of a Computerized Adaptive Test Suicide Scale (CAT-SS) among United States Military Veterans

Dear Dr. Brenner:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Sarah A. Arias

Academic Editor

PLOS ONE