Development and validation of Vellore Assessment of Social Performance among clients with chronic mental illness.

BACKGROUND: Social skills deficits are hallmark symptoms of chronic mental illness. The absence of a culturally sensitive instrument to measure social skills in the Indian population demands the need to develop and standardize such instruments. AIM: The aim of this study was to develop and validate a context-specific, culturally relevant, and performance-based assessment scale for social performance.
MATERIALS AND METHODS: An expert committee of mental health professionals reviewed existing literature, identified standardized scales, examined items for cultural relevance, and identified possible issues for measurement. The items were categorized into 5 domains with a 7-point scale. The instrument was initially piloted on 10 participants, then among 101 consecutive clients with chronic mental illness between 18 and 60 years of age who provided written informed consent. They were assessed by two therapists to evaluate inter-rater reliability and test-retest reliability. They were also assessed on the Social Interaction and Communication Skills Checklist (SICSC) to evaluate convergent validity and on the 12-item General Health Questionnaire (GHQ-12) to assess divergent validity. Standard statistical tests were used to study its characteristics.
RESULTS: The scale had good inter-rater reliability (0.941; 95% confidence interval [CI]: 0.914, 0.960) and test-retest reliability (0.928; 95% CI: 0.810, 0.965). The correlation between total score of Vellore Assessment of Social Performance (VASP) and SICSC (Pearson's correlation coefficient = 0.696; P = 0.001) suggested moderate convergent validity. The correlation between total score of VASP and GHQ-12 (Pearson's correlation coefficient = -0.046; P = 0.648) implied good divergent validity.
CONCLUSION: VASP seems to be a promising scale to assess social performance in people with mental illness. Copyright:

INTRODUCTION

Social dysfunction is a hallmark characteristic of chronic mental illness that has important implications for the development, course, and outcome of the illness.[1] As services for individuals with mental illness have shifted from the hospital to the community; there has been a shift in the philosophy of service delivery from symptom control to optimum participation in life activities. These activities may include obtaining education, maintaining employment, and living independently, all of which require appropriate social skills. Social skills and social competence are believed to be protective factors in the vulnerability–stress model of schizophrenia, the strengthening of which compensates for the deleterious effects of psychopathology.[2] Assessment and intervention for social dysfunction is essential to empower people with mental illness for leading productive lives as part of mainstream society.

The constructs “social competence,” “social skills,” and “social functioning” are often used interchangeably.[3] Some theorists view social functioning from a hierarchical perspective with social skills and social competence representing different levels of social performance.[4] Thus, social skills are the constituent verbal, nonverbal, and paralinguistic behaviors, which when used appropriately in a three-step process as given below results in social competence.[5]

Social perception or receiving skills consist of the ability to interpret other's nonverbal and verbal behaviors and evaluate the appropriateness of these behaviors as compared to social norms, considering the environmental factors affecting the interaction

Social cognition or processing skills are made up of the ability to analyze the current interaction, compare it with previous experiences, and thus ultimately decide on the necessary course of action

Behavioral response or expressive skills are the observable verbal and nonverbal behaviors that are seen at the end of the interaction.

The behavioral model of social skills in schizophrenia postulates that effective social functioning requires not only the skills but also the cognitive and motivational ability to perform the appropriate response.[6] Thus, separation of what a person can do and what he actually does, described as the competence/performance divide, is imperative in chronic mental illness.[7] As confounding factors such as cognitive deficits, negative symptoms, and low insight can prevent people with chronic mental illness from effective social performance, the ability or capacity of a person to perform socially cannot be inferred from behavioral performance alone.[3] Furthermore, the presence of capacity does not automatically translate into real-world functioning, as external factors such as reduced social opportunities can impede an exhibition of these skills. Therefore, it is imperative that the measurement of social functioning considers these issues for reliable assessment.

Assessment instruments for social functioning mostly take three forms – self- or proxy-rated checklists, direct observation of performance using role-plays, and naturalistic observation of real-world functioning.[8] Information from self-report checklists can be less reliable due to poor insight and the presence of cognitive, negative, or depressive symptoms in chronic mental illness.[9] The assessment of real-world functioning can be impractical and inaccurate as they are dependent on numerous environmental factors and may not reveal actual capacity. Performance-based measures, most using role-plays, have been found to be more feasible, accurate, and predictive of real-world outcomes as compared to the other methods of assessment.[7]

Rationale

The Vellore Assessment of Social Performance (VASP) was formulated specifically for inpatient rehabilitation setups providing social skills training for people with chronic mental illness, to aid in baseline assessment and evaluation of improvement over the training period. It addresses the following issues related to the assessment of social functioning which have been detailed earlier.

Constructs and terms for social skills, social competence, and social functioning are used interchangeably in literature and in most assessments: The VASP differentiates between “social skills” and “social competence” as operationally defined earlier so that therapists can discriminate whether social dysfunction is related to capacity or performance-related deficits. This, in turn, will help to tailor social skills training to individual requirements, for persons with adequate social skills, but inadequate social functioning; additional factors such as cognitive or motivational deficits will have to be focused on during training

Dearth of performance-based measures standardized for the Indian population: The Maryland Assessment of Social Competence (MASC)[6] and its shorter adaptation, the Social Skills Performance Assessment (SSPA),[10] are widely used role-play tests for social competence. However, the role-play situations for these assessments include asking the boss for a promotion, asking for a second chance at a job training program, and requesting attention from a landlord regarding a problem, which might not be familiar for the Indian milieu. The VASP is scored during routine life interactions, ranging from unfamiliar life situations (going shopping), familiar life situations (talking to other patients in the ward), and test situations (role-plays during training sessions). This enables the therapist to determine the situation-specific exhibition of skills and thereafter to ensure generalization of skills from therapy setting to patient-specific contexts.

MATERIALS AND METHODS

Study setting

The study was conducted in the department of psychiatry of a 122-bed tertiary referral center treating adults and children with mental and behavioral disorders. Psychiatrists, occupational therapists, psychiatric nurses, clinical psychologists, and psychiatric social workers form the treatment team and employ a multidisciplinary approach in the care of patients with mental illness. The inpatient facility is equipped with a comprehensive occupational therapy program with focus on improving various domains of social and occupational functioning. This instrument was developed specifically to assess changes in social functioning of patients undergoing the six-session social skills training program at the unit.

Review of issues

The study design and scale construction was conceptualized by RS and KSJ, who formed an expert committee consisting of occupational therapists, clinical psychologists, nurses, and psychiatrists. The committee reviewed current literature, identified issues in the field, and provided input on items and scoring. It was decided to formulate a performance-based assessment based on an already established construct.

Construction of assessment scale

Review of instruments

A search of literature identified the following instruments assessing social skills: (1) the Social Functioning Questionnaire,[11] (2) the Assessment of Communication and Interaction Skills,[12] (3) the Social Functioning Scale,[13] (4) the Social Dysfunction Rating Scale,[14] (5) the Social Interaction and Communication Skills Checklist (SICSC),[15] (6) MASC,[6] and (7) SSPA.[10] The contents, items, scoring, and interpretation of these scales were reviewed.

Item collection

Each scale and their items were examined for relevance, adaptability, and clarity by a multidisciplinary panel of mental health professionals.

Item categorization

The construct of verbal and nonverbal social skills as constituents of social competence: a process consisting of social perception, social cognition, and behavioral response was utilized to categorize the items into various domains.[5] This is the construct currently used in the training module for the social skills training sessions. Hence, items were categorized into five domains: (i) nonverbal social skills, (ii) verbal social skills, (iii) receptive social competence skills, (iv) processing social competence skills, and (v) expressive social competence skills [Appendix 1].

Scoring

The protocol for the social skills training sessions conducted in the unit includes therapist-guided, graded role-plays done by patients within the closed group (e.g., attending an interview – test situation), in the therapy hall/ward (e.g., talking to other patients – familiar life situation), or outside the ward (e.g., supervised visits to shops – unfamiliar life situation). Thus, it was decided to incorporate these situations into the scoring. A seven-point incremental scoring system was formulated for the assessment with 0 for being not able to perform the skill, 1 and 2 for being able to perform in test situations, 3 and 4 for being able to perform in familiar life situations, and 5 and 6 for being able to perform the skill in unfamiliar life situations. Anchor points for scoring each item were also elaborated and added [Appendix 1].

Validation of the scale

Face validity, utility, and feasibility for use were assessed by piloting the scale among 10 patients, after which modifications were made in discussion with the expert panel. The sample size was also decided a priori as 100, based on calculation from pilot study data.

Study sample

One hundred and fifteen consecutive inpatients with diagnosis of schizophrenia or bipolar affective disorder attending the occupational therapy program at the department of psychiatry aged between 18 and 60 years of age, who gave written informed consent, were recruited for the study. Fourteen assessments could not be completed; hence, the final sample size was 101. Clients with a clinical diagnosis of moderate to profound intellectual disability, those with organic mental disorders, those with acute psychotic presentations, and those with <1-year duration of illness were excluded from the study.

Assessment tools used

The following instruments were employed:

SICSC:[15] This is a 20-item checklist with a scoring from 1 to 5, 1 implying inability to perform even with assistance and 5 implying independent performance. A total score of <29 implies below par social interaction skills, and a score of above 90 implies exceptional social interaction skills. For the purpose of this study, this scale was chosen to evaluate convergent validity as it had items representing areas of both social skills and competence

General Health Questionnaire (GHQ):[16] This is a measure of current mental health and has been extensively used in different settings and different cultures. The questionnaire was originally developed as a 60-item instrument, but at present, a range of shortened versions of the questionnaire including the GHQ-30, the GHQ-28, the GHQ-20, and the GHQ-12 is available. For the purpose of this study, the GHQ-12 was used to evaluate divergent validity

Brief Psychiatric Rating Scale (BPRS):[17] The BPRS is a widely used instrument for assessing symptoms of individuals who have psychiatric disorders. The BPRS consists of 18 items with scores ranging from 1 (not present) to 7 (extremely severe) and 0 for not assessed. The BPRS scores, which are rated by the primary treating psychiatrist and presented during weekly case conferences, were documented

VASP: The final version of the scale has 20 items under 5 domains, with a scoring ranging from 0 to 6 [Appendix 1].

Procedure

The details of the study were explained to all participants, and written informed consent was obtained. The VASP was scored by two investigators (TS and PA) independently and simultaneously for evaluating inter-rater reliability. A third investigator (NA) scored the participants on the SICSC and GHQ-12 for evaluating convergent and divergent validity. The VASP was scored again by one investigator (TS) after 2 days to evaluate test–retest reliability. The scores of the participants on the BPRS, which is routinely done weekly, were also documented.

Data analysis

Summary statistics, mean and standard deviation frequencies and percentages, were used for reporting demographic and clinical characteristics. The correlation between VASP, SICSC, GHQ, and BPRS was evaluated using the Pearson's correlation coefficient. Differences were considered significant at P < 0.05. Inter-rater and test–retest reliabilities were evaluated using the intraclass correlation coefficient with a 95% confidence interval. All the statistical analysis was performed using SPSS 18.0 (SPSS Inc., Chicago, Ill., USA).

RESULTS

One hundred and one participants were recruited for the study. The sociodemographic and clinical characteristics of the sample are shown in Table 1. The majority of the participants were male, young adults, single, with undergraduate education, middle socioeconomic status, and currently unemployed.

Table 1

Characteristics of the study population (n=101)

Characteristics	Mean (SD)	PCC (P)	n (%)
Age (years)	30.00 (8.24)
Education
Undergraduate			51 (50.2)
Postgraduate			20 (19.8)
Higher secondary			15 (14.9)
Up to high school			15 (14.9)
Duration of illness (years)	6.74 (4.94)
Sex
Male			55 (54.5)
Female			46 (45.5)
Diagnosis
Paranoid schizophrenia			90 (90.1)
Bipolar affective disorder			10 (9.9)
Marital status
Unmarried			61 (60.4)
Married			40 (39.6)
Employment
Unemployed			78 (77.2)
Employed			23 (22.8)
Socioeconomic status
Low			14 (13.9)
Middle			79 (78.2)
Upper			8 (7.9)
VASP total score (120)	76.63 (26.05)
VASP - nonverbal social skills (24)	15.80 (5.04)
VASP - verbal social skills (30)	19.18 (6.70)
VASP - receptive social competence skills (18)	13.48 (3.95)
VASP - processing social competence skills (18)	10.49 (4.67)
VASP - expressive social competence skills (30)	18.20 (7.85)
SICSC total score (100)	67.70 (16.91)
GHQ total score (36)	13.83 (8.04)
Correlation - VASP/SICSC		0.696* (0.001)
Correlation - VASP/GHQ		0.046 (0.648)
Correlation - VASP/BPRS		−0.221 (0.026)

*Correlation is significant at the 0.01 level (two-tailed). SICSC – Social Interaction and Communication Skills Checklist; VASP – Vellore Assessment of Social Performance; GHQ – General Health Questionnaire; BPRS – Brief Psychiatric Rating Scale; SD – Standard deviation; PCC – Pearson Correlation Coefficient

Validity

Convergent validity

The SICSC was used to measure convergent validity. The correlation between the total scores on the two scales was moderate (Pearson's correlation coefficient = 0.696; P = 0.001), suggesting that these scales seem to assess similar constructs. The individual domains of the VASP could not be correlated as the SICSC is not further divided into domains.

Divergent validity

The VASP scores were correlated with the GHQ-12, which is a measure of overall mental health. The correlation between the total score of VASP and the total score of GHQ was low (Pearson's correlation coefficient = 0.046; P = 0.648), suggesting divergent validity.

Reliability

Table 2 documents the inter-rater and test–retest reliabilities. All five domains as well as the total scores of VASP recorded high inter-rater and test–retest reliabilities.

Table 2

Reliability data for all domains of Vellore Assessment of Social Performance

Domain	ICC (95% CI)
	Test-retest reliability	Inter-rater reliability
Nonverbal social skills	0.841 (0.732-0.901)	0.846 (0.781-0.894)
Verbal social skills	0.885 (0.802-0.930)	0.894 (0.847-0.927)
Receptive social competence skills	0.884 (0.787-0.932)	0.936 (0.907-0.957)
Processing social competence skills	0.876 (0.771-0.928)	0.946 (0.921-0.963)
Expressive social competence skills	0.831 (0.746-0.887)	0.894 (0.846-0.927)
VASP total score	0.928 (0.810-0.965)	0.941 (0.914-0.960)

VASP – Vellore Assessment of Social Performance; CI – Confidence interval; ICC – Intraclass Correlation Coefficient

Correlation with psychopathology

The VASP scores, when correlated with the BPRS scores, were statistically significant (Pearson's correlation coefficient = −0.221; P = 0.026). The negative correlation between the VASP and BPRS implies that those with higher psychopathology had lower social performance.

DISCUSSION

Since most chronic mental illnesses begin in early adulthood or late teenage, the development of age-appropriate interpersonal skills is hindered. The cognitive deficits, psychotic symptoms, and negative symptoms of chronic mental illnesses also further impair social functioning. The stigma associated with mental illness further perpetuates social isolation, unemployment, and lack of community support, thus initiating a vicious cycle of psychopathology superimposed on deficits which, in turn, contributes to social dysfunction. Hence, social dysfunction assessment and intervention remains an integral part of psychosocial management in chronic mental illness.

Most available assessments examining social functioning are in the form of pen-and-paper questionnaires with majority relying on self-report, while some revert to using peers or family in proxy. While these have certain advantages, they nevertheless appear to be inferior to direct assessment of skills with performance-based measures. There is also a discrepancy regarding the definition of social functioning and subsequently a lack of connection between measures to a theoretical model.

The VASP addresses these two main drawbacks of currently available social functioning assessments. The differentiation of domains into verbal and nonverbal social skills and receptive, processing, and expressive social competence skills enables better identification of the extent of deficit in order to tailor intervention strategies. The scoring system which differentiates between performance in life situations and performance in a simulated setting allows for more accurate competence/performance distinction. Although the scoring system is time-consuming and cannot be done in a single setting, the performance-based nature of assessment in contexts which are individually relevant makes up for this shortcoming. Considering the fact that the purpose of social functioning assessment is to provide in-depth information which can, in turn, be used to generalize social skills through training, the VASP seems to meet this requirement.

Limitations

The limitations of the VASP include relatively longer time taken for assessment, around 45 min, and need for observation in life situations, which might not be practical in all health-care settings. Since the VASP was specifically formulated for a setting with an on-going social skills training program where participants were already exposed to supervised practice in life situations, it was feasible to be scored. It has only limited utility in busy outpatient settings and as a screening or diagnostic tool for social functioning.

There was only moderate convergent validity of the VASP with the SICSC; domain wise convergent validity could not be assessed as the SICSC does not differentiate social skills into domains. The lack of convergent validity could be considered acceptable as the VASP was purposely developed with a different theoretical construct and scoring system in comparison to available instruments. The predictive validity of the VASP in terms of being able to predict employment needs has also not been looked into and is a recommendation for further studies. Furthermore, only preliminary analysis of correlation between VASP and BPRS was done; future studies can consider assessing the sensitivity of the VASP to the severity of psychopathology.

CONCLUSION

The VASP was developed to aid in detailed baseline as well as follow-up assessment of social skills in the patients who avail of the psychosocial rehabilitation services of an inpatient mental health setup. Preliminary data suggest that VASP seems to be valid and reliable for assessing the social skills of those with chronic mental illness among the Indian population. The findings need to be replicated and revalidated across diverse populations in India.

Financial support and sponsorship

This study was funded by an internal grant from the Institutional Review and Ethics Board, Christian Medical College, Vellore (IRB Min. No.10899).

Conflicts of interest

There are no conflicts of interest.

Items

I a	Nonverbal social skills	0	1	2	3	4	5	6
1	Eye contact
2	Facial expression
3	Gestures
4	Proximity
I b	Verbal social skills
5	Coherence
6	Volume and tone of speech
7	Initiation of conversation
8	Termination of conversation
9	Asking for needs appropriately
II a	Receptive social competence skills
10	Follow directions
11	Understand questions
12	Take turns in conversation/activity
II b	Processing social competence skills
13	Understand others’ verbal and nonverbal behavior
14	Understand impact of own behavior on others
15	Understand social norms
II c	Expressive social competence skills
16	Answer questions
17	Use social routines – thank you, good, sorry
18	Offer help
19	Modify behavior according to feedback
20	Participate in a social situation