Diagnostic Accuracy of the Primary Care Screener for Affective Disorder (PC-SAD) in Primary Care.

BACKGROUND: Depression goes often unrecognised and untreated in non-psychiatric medical settings. Screening has recently gained acceptance as a first step towards improving depression recognition and management. The Primary Care Screener for Affective Disorders (PC-SAD) is a self-administered questionnaire to screen for Major Depressive Disorder (MDD) and Dysthymic Disorder (Dys) which has a sophisticated scoring algorithm that confers several advantages. This study tested its performance against a 'gold standard' diagnostic interview in primary care.
METHODS: A total of 416 adults attending 13 urban general internal medicine primary care practices completed the PC-SAD. Of 409 who returned a valid PC-SAD, all those scoring positive (N=151) and a random sample (N=106) of those scoring negative were selected for a 3-month telephone follow-up assessment including the administration of the Structured Clinical Interview for DSM-IV-TR Axis I Disorders (SCID-I) by a psychiatrist who was masked to PC-SAD results.
RESULTS: Most selected patients (N=212) took part in the follow-up assessment. After adjustment for partial verification bias the sensitivity, specificity, positive and negative predictive value for MDD were 90%, 83%, 51%, and 98%. For Dys, the corresponding figures were 78%, 79%, 8%, and 88%.
CONCLUSIONS: While some study limitations suggest caution in interpreting our results, this study corroborated the diagnostic validity of the PC-SAD, although the low PPV may limit its usefulness with regard to Dys. Given its good psychometric properties and the short average administration time, the PC-SAD might be the screening instrument of choice in settings where the technology for computer automated scoring is available.

INTRODUCTION

Depressive disorders are relatively common in the general population [1] and are the fourth leading cause of worldwide disease burden, as they account for 12% of all total years lived with disability [2]. The cost of depression has been estimated at $83.1 billion in the USA [3] and £9 billion in England [4], most of which was attributable to time off work. In primary care [5], as well as in other non-psychiatric medical settings [6], their prevalence is high. In the PREDICT study carried out in six European countries, the 12-month cumulative incidence of DSM–IV major depression in screened GP attendees was 7.7% [7].

Given that effective treatments are available, the importance of their early detection and treatment cannot be overstated and indeed several authors have called for the better identification and treatment of depression [8]. Primary care physicians are often the first, and not rarely the only, health professionals that are contacted by depressed patients [9]. However, the accuracy of depression recognition by primary care practitioners and other non-psychiatrist physicians is not satisfactory [10, 11]. Hence, improving diagnostic efficiency for depressive disorders in primary care and other non-psychiatric medical settings is an important public health objective.

In recent years, screening for depression in non-psychiatric medical settings has gained acceptance as a first step towards improving depression recognition and management [12], and several screening tools have been developed. However, not all depression screeners are suitable for routine use [13, 14]. Lengthy self-administered questionnaires as well as clinician-administered instruments are impractical, while the instruments that do not provide clinicians with a list of the symptoms of depression that are present in each case make physician follow-up more difficult and time-consuming. Instruments such as the 9-item Patient Health Questionnaire (PHQ-9), a short questionnaire that is worded closely after the DSM-IV summary criterion for each MDD symptom, constitute a better approach to the problem. However, even the PHQ-9 has a limitation in the complexity of various items. Given that the instrument uses a single item to inquire about both polarities of vegetative symptoms, it includes several multiple concept questions that require the patient to integrate various emotions and behaviours into a summary answer, which may put less-educated subjects into some trouble.

The Primary Care Screener for Affective Disorders (PC-SAD) [14] is a 37-item self-administered questionnaire designed to screen for Major Depressive Disorder (MDD) and Dysthymic Disorder (Dys) in primary care. It makes use of technological advances [15] and has a sophisticated scoring algorithm that confers several advantages, such as the possibility to yield valid results even if many items are left unanswered. Its validity was first tested against other established screening questionnaires in health plan members, primary care outpatients, and psychiatric patients [14], and then was tested against a standardised psychiatric interview in dermatological inpatients [16]. Also, valid and reliable continuous measures of depression severity derived from the PC-SAD are available [17, 18]. However, no study has yet validated the PC-SAD against a standardised diagnostic interview in a primary care setting. This study aimed at filling this research gap by testing the performance of the PC-SAD against a ‘gold standard’ diagnostic interview in a sample of primary care patients.

METHODS

Patients and Procedure

A convenience patient sample consisting of adults attending 13 urban general internal medicine primary care practices took part in this diagnostic validity study. Participants were recruited by fliers posted in the waiting room while attending their regular appointments. They were offered to participate in a program for early detection of depression called ‘SET-DEP’ (Screening and Enhanced Treatment for Depression) and they provided written informed consent to participate in the study according to procedures approved by the Italian National Institute of Health ethical committee.

Fig. () shows the flow of patients through the study. A total of 416 primary care patients aged 18-65 years were included in the study and completed the PC-SAD in the waiting room. Only 7 PC-SAD questionnaires were unusable because of missing answers. Of the 409 participants with a valid PC-SAD, 121 were positive for MDD, either with (N=61) or without (N=60) positivity also for Dys, and 30 for Dys alone. The remaining 258 participants were negative for both MDD and Dys.

All patients with a positive PC-SAD result and a random sample of 41% (N=106) of those scoring negative on the PC-SAD were selected for a follow-up mental health assessment to be performed by telephone by a psychiatrist three months after PC-SAD completion, which included the Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition (SCID-I) [18]. Two psychiatrists performed the follow-up assessments. Of 151 patients scoring positive on the PC-SAD, 132 (81% females, mean age 47.4±11.5] were administered the SCID-I, while 8 could not be contacted despite repeated attempts and 11 refused the follow-up interview. Of the 19 patients with a positive PC-SAD who did not undergo the follow-up assessment, 11 were positive for MDD, either with (N=6) or without (N=5) positivity also for Dys, and 8 for Dys alone. There were no differences in gender, age, education, or marital status between the patients who underwent the follow-up assessment and those who did not. Of the 106 patients with a negative PC-SAD result sampled for the follow-up assessment, 80 (69% females, mean age 45.0±11.3) were administered the SCID-I, while 22 could not be contacted despite multiple attempts and 4 refused the follow-up interview. There were no differences in demographic variables between the patients who were administered the follow-up interview and those who did not, except for a higher mean age in patients who did not take part in the follow-up assessment.

Instruments

The SCID-I [19] is a clinician-administered detailed semi-structured interview that covers most Axis I disorders and is regarded as the ‘gold standard’ for making DSM-IV diagnoses. Several studies showed the superior validity of the SCID-I over standard clinical interviews at intake episode [20, 21].

The PC-SAD [14] is a short, self-administered questionnaire, constructed in accordance with DSM-IV criteria. The PC-SAD makes it possible to screen not only for MDD but also for Dysthymic Disorder (Dys). It consists of a 3-item pre-screener (two MDD questions and one Dys question, which are part of the screener score but reduce respondent burden by terminating the questionnaire if all are negative), a 26-item MDD section, and an 8-item Dys section. The PC-SAD breaks down each of the nine DSM-IV symptoms of MDD into several simple items, each consisting of single concept questions, and it integrates the answers mathematically. The responses linked to a given symptom are numbered and summed, and if the sum is above a certain threshold, the patient is said to have the symptom. The questionnaire is scored using an automated system, and the scoring algorithm is built in a way that the presence of each symptom can be determined independently from the presence of missing answers to one or more items related to the symptom, provided that at least one of the items related to the symptom has been answered. Hence, the questionnaire can give valid results even if many items are left unanswered, provided that at least one item for each DSM-IV symptom is answered. Similarly for the Dys section, which is the only validated self-administered screener available that is specific for Dys.

Performance of the Screening Test

Using the SCID-I as the reference standard, we calculated the sensitivity, specificity, predictive values, and likelihood ratios of the PC-SAD for MDD and Dys. Sensitivity is defined as the proportion of individuals with a disease who have a positive test result, whereas specificity is the proportion of individuals without the disease who have a negative test result. The positive predictive value (PPV) is the percentage of patients with a positive test result who have the disease, while the negative predictive value (NPV) is the percentage of patients with a negative test result who are free from the disease. The likelihood ratio for a positive result represents the odds that a patient with the disease has a positive test result as compared with a patient without the disease. The likelihood ratio for a negative result represents the odds that a patient without the disease has a negative test result as compared with a patient without the disease. Differently from predictive values, sensitivity and specificity are largely independent of the setting in which a screening test is used as they are quite stable with changes in prevalence of the disease in the population, while likelihood ratios may be even more stable with changes in prevalence [22].

Adjustment for Partial Verification Bias

In studies where only a portion of patients who underwent the screening test are administered the reference standard, the estimates of sensitivity and specificity may be biased. The term partial verification bias is commonly used to describe the quantitative effect of having different fractions of patients with a positive and negative screening result receiving the reference standard. Usually, this kind of bias leads to overestimate sensitivity and underestimate specificity [23].

To adjust for partial verification bias in patients with a PC-SAD positive for depression, we relied on the missing at random assumption, i.e., that the decision to ascertain the presence of depression does not depend on any information related to true disease status other than the observed PC-SAD result. This assumption is reasonable as the patients with a negative PC-SAD who were administered the SCID-I were randomly selected from the full sample of test negatives, and only a small proportion (12.6%) of patients with a positive PC-SAD did not receive the SCID-I.

Under this assumption, to adjust for partial verification bias the observed test results in each test stratum are weighted by the probability that patients in that stratum were administered the SCID-I [24]. As positivity to MDD or Dys may be a measured cofactor that influenced test verification, the patients with a PC-SAD negative for MDD who did not receive the SCID-I were further stratified according to their positivity for Dys or not, and the patients with a PC-SAD negative for Dys who did not receive the SCID-I were further stratified according to their positivity for MDD or not. Then, the adjustment was made weighted according to cofactor-defined strata to derive unbiased estimates of sensitivity and specificity.

RESULTS

The results of the follow-up assessment, performed on 132 patients scoring positive and 80 scoring negative on the PC-SAD, are described in detail in Table .

Of 110 patients scoring positive for MDD, 56 received a SCID-I diagnosis of MDD, and 35 of another mental disorder, while only 19 had no diagnosable mental disorder. Of 22 patients scoring negative for MDD but positive for Dys, 5 had MDD as determined by the SCID-I. All 80 patients scoring negative for both MDD and Dys were free from MDD as determined by the SCID-I.

Of 77 patients scoring positive for Dys, 6 received a SCID-I diagnosis of Dys, 44 had another depressive disorder, 6 an anxiety disorder, and only 21 had no diagnosable mental disorder, of whom one had symptoms of grief in response to a recent loss. Of 55 patients scoring negative for Dys but positive for MDD, 2 had Dys as determined by the SCID-I. All 80 patients scoring negative for both Dys and MDD did not receive a SCID-I diagnosis of Dys.

Most patients with a negative PC-SAD (N=63, 80%) did not receive any DSM-IV diagnosis, while the remaining 17 were diagnosed with a mental disorder, in most cases (N=15) an anxiety disorder.

The performance of the PC-SAD after adjustment for partial verification bias is presented in detail in Table .

DISCUSSION

Consistently with previous findings on dermatological patients [16], the PC-SAD displayed excellent sensitivity and good specificity for the diagnosis of MDD as determined by the SCID-I. Also, both the likelihood ratio positive and likelihood ratio negative were better than the median values reported for other instruments [25]. The NPV was very high and the PPV was also quite high, as compared with the usual scenario where most patients screening positive do not have MDD [26]. It should be noted that the PPV is partly dependent on the prevalence of the condition in the population being tested, and that the prevalence of depressive disorders in our sample was relatively high, likely due to the fact that patients self-selected for participation in the study. Under a scenario of 8% prevalence, consistent with epidemiological studies performed in primary care settings [7, 27], the PPV would be 31%. On the other hand, in general medical settings where all patients are ill and a substantial proportion have a chronic physical health problem the prevalence of depression is estimated to be as high as 24% [28], and under such a scenario the PPV would be 62%.

The MDD/Dys ratio as based on PC-SAD positive screens for MDD and Dys was similar to that found in the Work and Health study [29]. With regard to SCID-based diagnoses, consistently with the finding of a much higher prevalence of MDD as compared with Dys (3:1 ratio) in the Italian general population [30], Dys was much less common than MDD in our patient sample. With respect to Dys, sensitivity and specificity were fairly satisfactory, as were likelihood ratios and the NPV. The PPV was particularly low and clearly not satisfactory, although, as also observed in our previous study [16], many false positives, despite not having a diagnosis of Dys, were actually affected by another depressive condition, so that a positive result for Dys retains clinical value as it suggests the possible presence of a depressive disorder. A possible explanation for the low PPV for Dys is that the wording of the Italian version is such that patients may too easily indicate the presence of symptoms. Although a future study testing the English version of the PC-SAD against the SCID-I would be necessary to rule out this explanation, critical examination of the relevant Italian items suggests that wording is unlikely to account for the finding of a low PPV for Dys. More likely, this finding reflects difficulties inherent in the diagnosis of Dys itself. In fact, with any self-report instrument, no matter how sophisticated, it is extremely difficult to establish the co-occurrence of several depressive symptoms for most days over a very long period of time, and it is impossible to distinguish between the long-lasting depressive symptoms of chronic MDD or MDD in partial remission and those of MDD comorbid with Dys. Moving the PC-SAD cut-off for Dys may help reducing the number of false positives, but would hardly solve the problem given the difficulties involved in questionnaire-based diagnosis of Dys. Possibly, the PC-SAD section on Dys might be renamed as ‘chronic depressive symptoms’ so as to retain its clinical worthiness without formal diagnostic implications, given also that the nosology of chronic depression is complex and has long been a matter of controversy [31, 32]. Future studies are needed to fully elucidate this issue.

This study has some limitations. First, not all patients whose PC-SAD result was negative were administered the reference standard due to cost considerations. However, given the high sampling fraction and the use of random sampling, there is reason to believe that the missing at random assumption was satisfied, and that the adjustment for partial verification yielded unbiased estimates of sensitivity and specificity. Second, patients volunteered for participation in the study, which likely led to an increased prevalence of depression as compared to the prevalence in all patients attending the primary care practices. While this likely accounts for the finding of a particularly high PPV for MDD, the estimates of sensitivity, specificity, and likelihood ratios are reliable as these test properties are relatively unaffected by changes in disease prevalence. Third, the patients were assessed by telephone rather than face-to-face interviews. However, several studies using standardized psychiatric interviews have obtained moderate to good diagnostic agreement between telephone and in-person administration [33-36]. Fourth, the time elapsed between PC-SAD completion and SCID-I administration may have led to missing some depression diagnoses. However, the SCID-I allows reliable retrospective psychiatric diagnoses and the 3-month follow-up interval was so short to make underdiagnosis of a recent depressive episode quite unlikely. At any rate, if some degree of underdiagnosis occurred, its impact on the results would have been conservative as it would have led to underestimate the performance of the PC-SAD.

While these limitations suggest some caution in interpreting our results, this study is the first to provide evidence of validity against a ‘gold standard’ for the PC-SAD in primary care patients, and corroborated the notion that the PC-SAD is a sensitive and specific screening tool for MDD and Dys, although the low PPV displayed for the latter disorder may limit its usefulness with regard to Dys.

While depression screening alone is insufficient to substantially improve patient outcomes and should be supplemented by physician case management or mental health specialist involvement in order to assure accurate diagnosis, effective treatment, and careful follow-up [12], the availability and use of simple, practical, reliable and valid depression screeners is a key first step in any depression screening and management program. Many questionnaires are available to screen for depression in primary care and other non-psychiatric medical settings [25, 28, 37-40]. The selection of a screener requires consideration of several issues, such as characteristics of the population being screened, psychometric properties of the screener, time required to complete and to score the measure, ease of use, and cost. Given its high sensitivity and specificity, the short average administration time owing to the 3-item pre-screener, and the possibility to screen not only for MDD but also for Dys if desired, the PC-SAD might be the instrument of choice in settings where the technology for computer automated scoring is available.

Table 1.

Sociodemographic and Clinical Characteristics of Patients Undergoing the Follow-Up Assessment

		Patients positive for MDD alone (N=55)			Patients positive for MDD and Dys (N=55)			Patients positive for Dys alone (N=22)			Patients with a negative PC-SAD (N=80)
		N	%	Mean ± SD	N	%	Mean ± SD	N	%	Mean ± SD	N	%	Mean ± SD
Gender
	Male	12	21.8		10	18.2		3	13.6		25	31.3
	Female	43	78.2		45	81.8		19	86.4		55	68.8
Age				47.0 ± 12.3			47.5 ± 12.0			48.1 ± 8.4			45.0 ± 11.3
Marital Status
	Unmarried	21	38.2		25	45.4		8	36.4		31	38.7
	Married	23	41.8		26	47.3		7	31.8		39	48.8
	Separated, divorced, or widowed	11	20.0		4	7.3		7	31.8		10	12.5
Education
	Primary school	1	1.8					2	9.1
	Junior high school	10	18.2		6	10.9		5	22.7		7	8.8
	Senior high school	26	47.3		31	56.4		7	31.8		35	43.8
	College	18	32.7		18	32.7		8	36.4		38	47.5
Primary psychiatric diagnosis
	MDD	26	47.3		29	52.7		5	22.7
	MDD and Dys	1	1.8
	Dys	1	1.8		5	9.1		1	4.5
	Depressive Disorder NOS	3	5.5		5	9.1		2	9.1		1	1.3
	Bipolar Disorder NOS, mixed				1	1.8
	Mood Disorder due to a GMC	1	1.8
	Alcohol-induced Mood Disorder	1	1.8
	Mood Disorder NOS							1	4.5
	Adjustment disorder with depressed mood	2	3.6		1	1.8
	Adjustment disorder with mixed anxiety and depressed mood	4	7.3		1	1.8		1	4.5		1	1.3
	Panic Disorder				1	1.8					6	7.5
	Generalised Anxiety Disorder	2	3.6		1	1.8					3	3.8
	Social Anxiety Disorder	1	1.8
	Generalised Anxiety Disorder due to GMC										1	1.3
	Anxiety Disorder NOS	4	7.3		1	1.8					5	6.3
	No DSM-IV disorder	9	16.4		10	18.2		11	50.0		63	78.8

MDD=Major Depressive Disorder; Dys= Dysthymic disorder; NOS=Not Otherwise Specified; GMC=General Medical Condition

Table 2.

Performance of the PC-SAD

	Major Depressive Disorder	Dysthymic Disorder
Sensitivity (95% CI)	89.8 (79.6-95.5)	77.8 (40.2-96.0)
Specificity (95% CI)	82.6 (78.1-86.4)	79.0 (74.6-82.8)
PPV (95% CI)	51.2 (42.0-60.4)	7.7 (3.4-15.7)
NPV (95% CI)	97.6 (94.8-98.9)	99.4 (97.5-99.9)
LR+ (95% CI)	5.2 (4.0-6.6)	3.7 (2.5-5.5)
LR- (95% CI)	0.12 (0.06-0.24)	0.28 (0.08-0.96)

CI = Confidence interval

PPV = Positive predictive value

NPV = Negative predictive value

LR+ = Likelihood ratio for a positive result

LR- = Likelihood ratio for a negative result