Concurrent Validity and Interrater Reliability of the "Clinical Schedules for Primary Care Psychiatry".

Background and Objectives  There is limited access to specialized mental health care in countries such as India with a wide treatment gap for psychiatric illnesses. Integrating mental health delivery with primary health-care services is vital. The clinical schedules for primary care psychiatry (CSP) was designed for training primary care doctors (PCDs) to identify and diagnose psychiatric illness in patients presenting to primary care settings. This study aims to study the validity and reliability of the CSP and its hypothesis is that the CSP would help PCDs to identify psychiatric caseness. Methods  The study was conducted at three primary health centers of Karnataka. Consented PCDs were briefly trained in the use of CSP and screened patients who were later interviewed by a psychiatrist using a semistructured interview and confirmed by International Statistical Classification of Diseases and Related Health Problems 10th edition (ICD-10) symptom checklist. The appropriate statistical analysis was performed. Results  A total of 180 patients were included. Agreement was found between diagnoses made by PCDs and psychiatrist for 142 (78. 9%) patients with a Cohen's kappsychiatry pa (K) = 0. 57. The sensitivity was 91. 1% and specificity was 68. 3%. The interrater reliability showed K = 0. 7. Conclusion  The CSP helps PCDs to make psychiatric diagnoses. It has a relatively high sensitivity with reasonably high specificity but mayneed clinical training.

Introduction

Psychiatric illness is a major contributor to the global burden of disease. It is an oft-recorded observation that the access to specialized mental health care in low- and middle-income countries including India is significantly restricted by numerous factors, such as limited numbers of trained professionals and their inequitable distribution, stigma and prejudice,lack of awareness, and limited resources and lacunae in policy making. 1

While the list of challenges is lengthy, the treatment gap for mental disorders is also wide, particularly for common mental disorders (CMDs) and substance use disorders (SUDs). As per the findings of the National Mental Health Survey of India 2016, the prevalence of psychiatric illnesses in India is 10. 6% with a treatment gap ranging from 60% for CMDs to 90% forSUDs. 2 3 4 The World Health Organization (WHO) has encountered recommended a strategy to tackle this through the integration for psychiatric care with primary health care. 5 6 7 Under the National Mental Health Program of India, it was proposed to provide psychiatric services by training primary care doctors (PCDs) in the early identification and treatment of psychiatric disorders at primary care level. 5 There are many attempts to continue in favor of implementing this strategy. 8

Earlier studies have found that CMDs including anxiety, depression, and somatization present in 17 to 46% of patients attending primary health centers. 9 However, PCDs may not have the requisite training to diagnose psychiatric disorders in these patients. The past studies have found that PCDs may fail to correctly diagnose and treat nearly 50 to 75% of patients initially presenting with psychiatric disorders. 8 10 11 12 Ideally, screening for psychiatric disorders by PCDs needs to be achieved in a time-limited manner with reasonable accuracy, as against the psychiatric clinical interview performed by psychiatrists which is considered as a gold standard. 13 Inadequate knowledge about diagnostic criteria of CMDs, lack of awareness about the appropriate interview questions that need to be asked to identify these disorders, and limitations of time in a busy clinic lead to underdiagnosis by PCDs. 8

The WHO 14 has developed the Mental Health Gap Action Program (mhGAP) and evolved a tool called the mhGAP Intervention Guide for use in nonspecialized settings for the first- and second-line health-care professionals. It is very lengthy, complex, and not specific for PCDs. It includes nine priority- based conditions (mental, neurological, and SUDs) such as depression, psychosis, bipolar disorders, epilepsy, developmental and behavioural disorders in children and adolescents, dementia, alcohol and drug use disorders, selfharm/suicide and other significant emotional or medically unexplained complaints. These are rather high-prevalent psychiatric conditions at primary care with higher treatment gap. It is not on PCD friendly manual, especially for developing countries, like India, considering heavy patients load at primary care. Guidelines of child mental health conditions and dementia in mhGAp manual are impossible to implement at Indian primary health centers by PCDs. Despite higher prevalence at primary care, mhGAP manual grossly missed anxiety and somatization disorders, and management of tobacco addiction. Despite very low prevalence of psychosis, especially bipolar disorder at primary care, mhGAP laid heavy focus on management of these conditions. This manual does not allow safer, wider range of pharmacological agents for highly prevalent psychiatric conditions, for example, escitalopram is commonly prescribed antidepressants in India. Emphasis heavily laid on psychosocial interventions for these conditions, which PCDs unlikely to practice in India. It also requires special training for its application and requires support for its interventions. It also grossly lacks integrated, yet comprehensive screener (WHO, 2016). There are many disease-specific screening tools, such as Patient Health Questionnaire- 9 (PHQ-9), 15 Primary Care Screening Questionnaire for Depression (PSD), 16 and Global Mental Health Assessment Tool for Primary Care. 17 However, there are limitations, such as the absence of a single screening tool for most commonlyencountered conditions in primary care, lack of cultural adaptation for Indian settings, and considerations for use in primary care settings. 15 16 17 Hence, there is a need of integratedall-in-one tool for PCDs with incorporation of principle of general clinical practice.

A sole aim of designing “Clinical Schedules for Primary Care Psychiatry” (CSP) is an all-in-one integrated tool for PCDs to provide the first-line safe and effective pharmacotherapy to highly prevalent (may not be priority based) psychiatric disorders at primary care with rapid screening using culturally appropriate questionnaire. 18 CSP Version 2. 1 (Department of Psychiatry, National Institute of Mental Health and Neurosciences, Bengaluru, India) consists of screener, classification of psychiatric disorders adapted for use in primary care settings, diagnostic criteria, referral points, and management guidelines of only six high-prevalent psychiatric disorders at primary care where highly effective and safe prescription medications are available. These six psychiatric disorders are tobacco addiction, alcohol (harmful and addiction), psychotic, depressive, anxiety (panic and generalized anxiety disorders), and somatization disorders (acronyms as TAP DAS). The CSP screener consists of a questionnaire containing 21 culturally appropriate questions to screen patients of these six highly prevalent psychiatric disorders. In simple, CSP use three cluster- based transdiagnostic classification of psychiatric disorders adopted for the use of PCDs; CMDs cluster subdivided as predominantly depressive, anxiety, somatization, or mixed symptoms; psychotic disorders cluster is subdivided as acute (cover acute psychosis and mania), chronic (schizophrenia and schizoaffective disorders), or episodic (focus on bipolar disorder); and alcohol disorders simplified as harmful (frequent and infrequent type) and addiction with simplified diagnostic criteria for primary care use. Discussion about designing curriculum and screener questionnaire of this CSP is beyond the scope of this article and is discussed elsewhere. CSP is available on request for readers. The screener includes a hint or anchor column which leads to a broad diagnosis. This has been extensively used in digitally driven primary care psychiatry program from Telemedicine Centre of National Institute of Mental Health and Neurosciences (NIMHANS), Bengaluru, for the purpose of training PCDs to identify and manage psychiatric disorders. 19 CSP requires a very brief training about how to ask the screening questions and making a diagnosis using the hint/anchor column. Further, the diagnostic categories lead to management guidelines including medication, brief counseling advice, and follow-up advice along with strategies for referral. Thus, PCDs must be sensitized to detect “red flags” or warning signs of psychiatric illness particularly for identifying CMDs which have a treatment gap of 50 to 60%. 2 6 Those patients who are likely to have problems after ruling out those due to medical illness and SUDs need to be picked up. The questions must be suited to reflect the pragmatics of the reallife scenario of primary care practice in India. CSP screener has this inbuilt pragmatism of real-world scenario. The aim of this study was to validate the CSP and establish its psychmetric properties with the hypothesis being that the CSP would be able to help PCDs to establish psychiatric caseness (meaning to achieve a broader diagnosis) after brieftraining in its use. This would establish its practical applicabilityto enable PCDs to identify psychiatric diagnoses in primary care settings.

Methods

Study Center Location

The study was conducted at the outpatient services of three primary health care centers (PHCs) under the District Mental Health Program (DMHP) at Ramanagara district in Karnataka, India, after administrative approval from the National Health Mission, Government of Karnataka. The study was also approved by the Institute Ethics Committee at the NIMHANS at Bengaluru in Karnataka, India. This study was performed from October 2017 to June 2018.

Participants

Written informed consent from PCDs working at the above-mentioned PHCs was obtained to participate in this study. After providing consent, the PCDs were briefly trained about the CSP, particularly how to ask questions contained in the screener, how to use the hint/anchor column to lead to the diagnosis with the investigator explaining briefly about the diagnostic categories and management guidelines of the CSP. This entire discussion would take around 15 to 20 minutes/doctor.

Psychometric Properties

Validity and reliability for the CSP needed to be established. To establish an agreement, the diagnoses made by the PCDs using the CSP were compared with those made by a qualified psychiatrist using a semistructured psychiatric interview to make a diagnosis, confirmed by the International Statistical Classification of Diseases and Related Health Problems 10th edition (ICD-10) checklist which was used for concurrent validity. 20 21 This ICD-10 symptoms checklist is a semistructured diagnostic instrument used for clinicians’ assessment of F0 to F6 categories in the ICD-10 classification of mental and behavioral disorders and tested in the field trials of the ICD-10 diagnostic criteria for research. 22 To establish interrater reliability, a total of 20 patients were simultaneously and independently interviewed by two PCDs and diagnoses were compared.

Sample Size

A total sample of 210 patients were calculated (using the principle of 10 patients per item of the screening questionnaire). 23 However, a total of 180 patients ( n = 180) who provided written informed consent included in the study. Those under the age of 18 years were excluded. Eleven patients declined to participate due to logistical reasons, such as greater waiting time for a second interview by the psychiatrist

Sampling

As a part of DMHP Ramanagara, mental health outreach camps and “Super-Tuesday Manochaitanya” clinics would be conducted at PHCs. Subsequent to the camps, the psychiatrist would visit OPD of the nearby PHCs, where consecutive adult patients presenting to the outpatient clinic (around 15–20 patients at the time that the investigator would visit) of the PHCs who provided written informed consent were included for the study.

Procedure

Procedure The three PCDs who consented were trained independently about method to use CSP its screener, classification system and diagnostic criteria, and to make psychiatric diagnosis. Patients who consented for the study were administered CSP in addition to regular examination at the PHC. The PCDs would initially examine the patients and first note if the patients had symptoms which could be explained by medical illness alone. If yes, then they would not proceed with the CSP and continue treatment as usual. If, however, these symptoms were medically unexplained, they would need to check if they had lasted greater than a fortnight. If not, then the patient could be reassured and monitored at follow-up. If yes, then the PCDs would administer the questions contained in the CSP to determine if the patient likely had a psychiatric illness (psychiatric caseness). The screener was verbally translated into the vernacular language, that is, Kannada by PCDs at the time of administering it. Appropriate diagnosis was marked in the classification provided in CSP below the screener after checking the diagnostic criteria.

Soon after PCDs consultation, a psychiatrist (K. K. ) performed a semistructured psychiatric interview for the same patients and the diagnosis was confirmed with the ICD-10 checklist. This interview involved history-taking and mental state examination during which clinical characteristics of patients were noted for making the diagnosis. The psychiatrist was blind to the diagnosis made by PCDs. The patients who were diagnosed by psychiatrist to have CMD/SMD/ SUD were advised appropriate investigations, advised for follow-up, and started with appropriate treatment.

Statistical Analysis

Descriptive statistics were used for sociodemographic and relevant clinical variables. Sensitivity, specificity, and positive predictive value were calculated for the use of CSP to detect and diagnose psychiatric illness. Concurrent validity consisting of comparing the diagnoses made by PCDs using the CSP, and the diagnoses made by the investigator conducting a psychiatric interview to make a diagnosis confirmed by the ICD-10 checklist used as a gold standard. Cohen’s kappa (κ) was calculated for interrater reliability. The Statistical Package for the Social Sciences (SPSS version 16; SPSS Inc. , Chicago, United States) was used for the purpose of the analysis.

Results

Sociodemographic Characteristics

A total of 180 patients were included in the study. Of these, 116 (64. 4%) were female and 64 (35. 6%) were male. The mean age was 44. 5 ± 15. 5 years. All of them are residents of three PHCs.

Clinical Characteristics

Of the total patients, 17 (9. 4%) had a past history of psychiatric illness, with depression in nine (5%), anxiety disorders in two (1. 2%), dissociative disorder in one (0. 6%), and postpartum psychosis in one (0. 6%) patients. In terms of known comorbid medical illness, 63 patients (34. 4%) had hypertension (16; 9. 9%), diabetes mellitus (18; 11%), epilepsy (11; 6. 2%), alcoholic liver disease (two; 1. 1%), tuberculosis (two; 1. 1%), and human immunodeficiency virus–(one; 0. 6%).

Agreement and Concurrent Validity

There was agreement between a diagnosis of psychiatric illness made by the PCDs and the psychiatrist for 142 (78. 9%) patients ( Table 1 ). The Cohen’s κ for the concurrent validity was 0. 57, suggestive of moderate agreement. 24

Table 1

Diagnoses made by primary care doctors using clinical schedules for primary care psychiatry versus psychiatrist using interview

Diagnostic category	Diagnosis	CSP diagnosis by PCDs, n (%)	Psychiatrist diagnosis, n (%)
Abbreviations: CMDs, Common mental disorders; CSP, Clinical schedules for primary care psychiatry; PCDs, primary care doctors; SMDs, severe mental disorders; SUDs, substance use disorders.
SUDs	Alcohol harmful use	4 (2. 2)	2 (1. 1)
	Alcohol dependence	5 (2. 8)	6 (3. 3)
	Tobacco dependence	12 (6. 7)	11 (6)
	Panic disorder	6 (3. 3)	3 (2)
	Generalized anxiety disorder	10 (5. 6)	7 (3. 9)
CMDs	Depression	20 (11. 1)	26 (14. 5)
	Somatoform disorder	27 (15)	10 (5. 6)
	Mixed CMDs	(9. 4) 17	9 (5)
SMDs	Psychoses/bipolar disorder	3 (1. 7)	3 (1. 7)
Others	Intellectual disability/autism spectrum disorder	–	12 (6. 6)
Nil psychiatry		76 (42. 2)	101 (56. 1)

Sensitivity, Specificity, and Positive Predictive Value

The sensitivity of the CSP for making diagnoses comparable to those made by a psychiatrist was 91. 1% ( Fig. 1 ). The specificity of the CSP for making diagnoses comparable to those made by a psychiatrist was 68. 3%. The positive predictive value of using the screener was 69. 2% ( Table 2 ).

Fig. 1

Flowchart of patients screened by clinical schedules for primary care psychiatry and then confirmation by a psychiatrist. CSP, Clinical schedules for primary care psychiatry.

Interrater Reliability

The interrater reliability for 20 patients showed κ = 0. 7, suggestive of good agreement.

Discussion

The CSP demonstrates a reasonably high sensitivity of 91% to detect psychiatric illness with a specificity of 68% and a positive predictive value of 69%. Thus, the CSP has high sensitivity with a lower specificity and positive-predictive value. It is important to consider the implications of the same. This would imply that a higher number of patients are likely to be diagnosed as having psychiatric illness and treated for the same, resulting in false positives. It may also lead to underinvestigation and underdiagnosis of a possible medical condition. One must also consider the implications in terms of stigmawhich is attached to mental illness in India, while labeling a patient as having a psychiatric illness. Patients may be treated with psychotropic drugs, albeit these are relatively safe. It is important to remain open-minded aboutthe diagnosis of those patients who do not appear to have a good response withthese drugs. In the real world, the specificity of the CSP can be improved with skill-based clinical training of the PCDs. Therefore, the pragmatic aspectof striking a balance between its sensitivity and specificity needs to be considered as equally important.

Furthermore, it is important to consider here the sensitivity and specificity of previously mentioned questionnaires which have been used in comparable settings. The PSD has a relatively high sensitivity and specificity of 96 and 87% at a cut-off of two or more questions out of a total of four. The authors found that relatively high specificity and positive-predictive value could be obtained by determining a threshold of two or more questions as cut-off. Changing this threshold to a higher number would lower sensitivity and lowering it would bring down the specificity. 16

The aforementioned PHQ-9 (depression), however, has a sensitivity of 91% and a specificity of 76% which is comparable to the CSP itself. The authors ofthe PHQ-9 have similarly discussed that the most favorable combination of sensitivity and specificity could be obtained using a cut-off score of ≥9, wherein a higher score would increase specificity at the cost of reducing sensitivity. 15

It must, however, be discussed again that the above-mentioned questionnaires are designed to screen for a particular condition, such as depression, whereas the CSP is designed for rapid screening for highly prevalent psychiatric conditions in a primary care setting. Therefore, a tool with higher sensitivity may be preferable in these settings. The problem of high-false positives in psychiatric illnesses has often been examined by earlier studies. In fact, a study which particularly looked at the problem of higher false-positive rates in screening for psychiatric illnesses in primary care declared that even those who were falsely positive on screening warranted greater clinical attention because these patients had higher functional impairment and higher rates of service uses than those who were true negatives. 25 Other studies have also recommended taking a closer look at the patients who screen falsely positive suggesting that other forms of psychopathology or a protean medical condition may be responsible. 26 Yet another study which looked at psychiatric diagnoses itself stated that the thresholds for making a diagnosis for psychiatric illness in screening should always be set, so that the benefits outweigh the risks and that patients who screen as false positives must be kept in follow-up and treatment as they might need further investigation and care. 27

Limitations

However, it is important to acknowledge the limitations of the study, most importantly, the fact the CSP was administered by a small number of motivated and interested PCDs. Comorbid psychiatric diagnoses are not made using the screener. Furthermore, it was verbally translated into Kannada at the time of administration to the patients, which may lead to some variation in how the questions were asked. CSP should be seen as a knowledge enhancer rather than a skillenhancer in clinical settings. Hence, the results of this article should alsobe seen as being a translation of the knowledge-enhancing effect of CSP in clinical settings rather than the skill-enhancing effect of a clinical training program (especially other programs, such as the direct skills transfer effect of an innovative on consultation training of PCDs which is now being implemented at our center).

Conclusion

In conclusion, the CSP is designed for use by PCDs and demonstrates a high sensitivity and fairly high specificity for making psychiatric diagnoses at primary care. It is important to keep in mind that it requires a pragmatic clinical training which has higher translational quotient for its administration as well as needs further refinement in its validation using a larger sample size. Pragmatically, its clinical effectiveness and acceptability remain to be examined.

Future Directions

The next step in its validation would be to acquire a larger sample size so that most disorders, including SUDs and SMDs, have an adequate representation. It may also be necessary to revise some of the questionnaires of the CSP to help improve its specificity and positive predictive value. Specificity of CSP can be improved by clinical skill-based training of PCDs (one such example is the on-consultation training is being conducted for PCDs at this center) which shall be tested later. 19 It would also be necessary to study the clinical usefulness and effectiveness of CSP with its acceptability by PCDs working in real-life clinical scenario with patients.

Table 2

Sensitivity, specificity, and positive predictive value of clinical schedules for primary care psychiatry

CSP diagnosis	Psychiatrist diagnosis		Total
	Disease	Not diseased
Abbreviations: CSP, clinical schedules for primary care psychiatry; PPV, positive predictive value.Note: sensitivity = 72/79 × 100 = 91. 1%, specificity = 69/101 × 100 =68. 3%, PPV of + test = 72/104 × 100 = 69. 2%.
Positive	72	32	104
Negative	7	69	76
Total	79	101	180