OPCRIT+: an electronic system for psychiatric diagnosis and data collection in clinical and research settings.

BACKGROUND: The increasingly large sample size requirements of modern adult mental health research suggests the need for a data collection and diagnostic application that can be used across a broad range of clinical and research populations. Aims To develop a data collection and diagnostic application that can be used across a broad range of clinical and research settings.
METHOD: We expanded and redeveloped the OPCRIT system into a broadly applicable diagnostic and data-collection package and carried out an interrater reliability study of this new tool.
RESULTS: OPCRIT+ performed well in an interrater reliability study with relatively inexperienced clinicians, giving a combined, weighted kappa of 0.70 for diagnostic reliability.
CONCLUSIONS: OPCRIT+ showed good overall interrater reliability scores for diagnoses. It is now incorporated in the electronic patient record of the Maudsley and associated hospitals. OPCRIT+ can be downloaded free of charge at http://sgdp.iop.kcl.ac.uk/opcritplus.

The goals of personalised medicine and stratified healthcare carry the potential to improve the outcomes of psychiatric disorders, which to date are often characterised by insufficient treatment response. Defining psychiatric symptoms and establishing diagnoses using a standard assessment tool across major psychiatric disorders is a desirable aim on the path towards translating research findings into clinical practice. Furthermore, a common data-set across many different research and clinical populations facilitates the generation of large data-sets essential to studies of complex disorders. OPCRIT+ is an expansion and redevelopment of the OPCRIT system described by McGuffin et al in 1991.[1] The original tool is an electronic checklist of psychopathlology items with algorithms for objective diagnosis of psychotic and affective disorders. It has been found to be both reliable and easy to apply in research settings but has not been applied in clinical settings.[2,3] We have redesigned OPCRIT for use in clinical settings and expanded the number of objectively rated items to cover a broader range of diagnostic categories. This is part of a larger drive to improve objectivity in diagnosis in clinical settings, and to provide a mechanism for the routine collection of a core clinical, research and audit data-set. In this paper we describe the development and structure of OPCRIT+ and present the results of an interrater reliability study. Our aim is first a proof of concept: that OPCRIT can be developed into a broad clinical assessment tool for adult mental illness. Second, we aimed to discover whether OPCRIT+ could achieve adequate interrater reliability of diagnosis among junior clinical staff, which is a requirement if such a system is to be implemented routinely.

Method

Development of OPCRIT+

As in the OPCRIT system, OPCRIT+ consists of a checklist of items in either electronic or paper format that allows flexible data entry, provides definitions of the items, and applies algorithms based on ICD-10,[4] DSM-IV[5] and other criteria to provide a diagnostic classification. Two identical versions of OPCRIT+ were designed. The first was written as a standalone program running under Windows XP (or later versions of Windows) and the second as a component of the South London and Maudsley NHS Foundation Trust electronic clinical record (ePJS), which has a web-based interface. Each version collected an identical data-set, used identical definitions of items and terms and identical algorithms to make diagnoses (Appendix 1).

Item design and arrangement

The items in OPCRIT+ were structured around the headings of a standard psychiatric history and mental state examination. We retained the original items from OPCRIT (http://sgdp.iop.kcl.ac.uk/opcrit/) but the wording of some items was changed to take account of new items, as well as changes in terminology since publication of the original OPCRIT paper.[1] New items were designed based on the objective criteria requirements of the expanded range of diagnoses from the research version of ICD-10 and from DSM-IV. A range of anxiety, substance misuse and personality disorder diagnoses commonly encountered in psychiatric practice were included. Further items salient to the assessment of psychosocial status, risk and prognosis were also added (see online supplement and online Table DS1).

An OPCRIT+ assessment includes a number of screening items. Depending on the answers to these screening items, additional items are activated for rating. This approach reflects the need to create a system that could be rated quickly and efficiently by a busy clinician, while at the same time avoiding a Procrustean approach to diagnosis (the practice of tailoring data to fit a preconceived structure).[6] Thus, for example, items regarding delusions are only activated if a prior screening item about delusions is rated positively. Free-text boxes are available within all sections, allowing notes specific to each case to be recorded.

Each item is associated with a help entry. Definitions, suggested questions and other resources within the help were standardised where practical, with reference to the original OPCRIT items, ICD-10/DSM-IV criteria and, using as a model, questions contained within the Schedules for Clinical Assessment in Neuropsychiatry (SCAN)[7] and the Structured Clinical Interview for DSM-IV (SCID).[8]

Algorithm design

The OPCRIT+ checklist retained all algorithms from OPCRIT (v4.0) and additional diagnoses as listed in Appendix 2. Additional algorithms were designed around the criteria for the diagnoses in ICD-10 and DSM-IV shown in Appendix 2. All algorithms were coded in the programming language C++ on Windows.

Database design

OPCRIT+ uses the mySQL database platform (v5.1). The mySQL platform is a free, open-source database structure that is widely used, flexible and secure. Item ratings were recorded numerically, with a rating of –8 for ‘not applicable’, –9 for ‘unknown’ and a ‘null’ value to represent an unrated item. Ratings are time stamped. OPCRIT+ takes advantage of the internet-enabled features of mySQL so that data can be securely entered and stored in real time by multiple users in multiple locations over a secure connection via the internet to a mySQL database server. Using this feature avoids storing potentially sensitive data on local workstations or researcher's laptops.

Usage

OPCRIT+, like its predecessor, was designed to allow flexibility in the source of ratings. That is, the constituent items can be scored following a standard clinical clerking, following a structured interview, based on a set of case notes, from a prepared case abstract/ vignette or from a combination of sources. Like its predecessor, the authors assume the rater has a basic knowledge of psychiatric assessment and eliciting psychopathology. An online web page was constructed with instructions for installation and use. Training vignettes were designed and are available on request from the website (http://sgdp.iop.kcl.ac.uk/opcritplus).

Interrater reliability

Junior doctors in psychiatry were recruited by an email advertisement sent to trainees at the South London and Maudsley NHS Foundation Trust. The recruited group of trainees (n = 20) at the time of the study had between 0 and 3 years of clinical psychiatric experience. Ten cases were selected from the clinical case records to represent the broad range of diagnoses that the OPCRIT+ checklist covers. Most of these were clinically complex, with multiple, comorbid diagnoses. Case abstracts were anonymised and prepared by J.R. Each case was described in approximately 500 words, divided by paragraphs but without headings. Each participant received 1 h of training on the use of the OPCRIT+ checklist by J.R., J.G. and C.G. Trainees were then asked to rate the cases using the checklist, on the basis of a ‘lifetime-ever occurrence of symptoms and signs’. Raters were asked to check each item manually, regardless of whether they were nested within a screening item, and reconsider their screening item rating in light of any positive rating of a nested item. This sought to mitigate against the complication of data non-independence when kappa values were calculated. J.R., C.G. and J.G. were available to give technical assistance and advice when it was requested, but did not give guidance on the rating of cases. The OPCRIT+ algorithms were applied to the collected data to generate a list of diagnoses for each rater.

Statistical analysis

We calculated a variant of Fleiss' kappa statistic that allows for multiple raters, multiple categories and no pre-existing assumption that raters are forced to assign a certain number of cases to each category.[9,10] Values of kappa can range from –1.0 to 1.0, with -1.0 indicating perfect disagreement below chance, 0.0 indicating agreement equal to chance, and 1.0 indicating perfect agreement above chance.[10] The raw diagnostic output from OPCRIT+ was sorted into ordered, diagnostic groups between which the weightings were equal, as detailed in Appendix 3. We statistically analysed the diagnostic data generated from the rated cases by calculating the extent to which raters agreed on diagnoses. More specifically, within each diagnostic spectrum (i.e. psychotic/affective, substance misuse, anxiety and personality disorder spectra) we selected from the ten cases any one that might reasonably have attracted a diagnosis along that spectrum along with one ‘null’ case, and calculated the interrater reliability with respect to those cases.

We further analysed the interrater data by calculating kappa values for all original OPCRIT items, all screening and mental state examination items and any other items that were used by the diagnostic algorithms. We calculated kappa scores from counts of rating scores for each option within each item (i.e. how many out of the 20 raters rated 1, 2, 3, etc.) for each case separately and then combined scores for all cases to give a combined kappa score for that item. Thus we calculated interrater reliability scores for a total of 158 items.

Results

Interrater reliability results are detailed in Table 1. The combined, weighted kappa for diagnostic reliability among all groups of diagnoses was 0.70. For psychotic and affective disorders a kappa of 0.63 was obtained. Kappas for substance use disorders, anxiety disorders and personality disorders were 0.76, 0.71 and 0.83 respectively.

Table 1

Fleiss' free-marginal kappa scores and % of overall agreement for each diagnostic spectra

Diagnostic spectra	Diagnostic groups, n	Cases, n	Overall agreement, %	κ
Affective/psychotic disorders	7	5	68	0.63
Anxiety disorders	2	3	80	0.71
Substance use disorders	3	10	84	0.76
Personality disorder	2	4	92	0.83

The mean kappa for interrater reliability among items was 0.80. When kappas for individual items were stratified according to whether items were independent (screening items) or dependent (contained under screening items) then kappa scores were 0.73 and 0.84 respectively. Items contained under screening items can be interpreted as partially dependent variables, resulting in the kappa score being artificially inflated, therefore detailed results are presented in online Table DS2 and online Fig. DS1.

Discussion

Main findings

OPCRIT+ can be downloaded and installed free of charge from http://sgdp.iop.kcl.ac.uk/opcritplus/. OPCRIT+ has been incorporated within the electronic case record of the South London and Maudsley NHS Foundation Trust London, UK. Data collected by OPCRIT+ in the electronic case record are subject to automatic anonymisation and categorisation via an electronic search engine called the Case Register Interactive Search (CRIS), recently developed by Stewart et al.[11]

The OPCRIT+ checklist is a development and expansion of the OPCRIT checklist, a well-established and well-validated tool in research settings.[2,3] We demonstrate in this study good interrater reliability scores for diagnosis and good overall reliability for individual items. Most items have kappa scores between 0.6 and 0.9 (online Fig. DS1) and all but three diagnostic items have kappa scores of over 0.6 (online Table DS2). The interrater reliability study was performed on the standalone version of OPCRIT+. We would not expect any differences in interrater reliability between the two versions of OPCRIT+ in this study as they collect identical data-sets and run identical algorithms. A unified data-set and applied operationalised criteria for diagnosis in psychiatry is potentially achievable across the majority of patient groups and research groups using this tool. Such a data-set will be invaluable for the continuing research into biomarkers and personalised interventions in psychiatric disorders.

Strengths and limitations

In our study, raters were medically qualified, but had only between 0 and 3 years postgraduate experience in clinical psychiatry. They received 1 h of training in the use of OPCRIT+. Despite these limitations they achieved good interrater reliability. OPCRIT+ is therefore a robust system and could potentially be used by research workers and members of professions allied to medicine with appropriate training. However, we would also point out that our interrater reliability study was carried out using prepared abstracts and may not fully represent the sort of reliability that might be seen in busy clinical environments. Clearly, reliability may be influenced by many factors, including the time constraints on raters, their experience and knowledge of the case they are rating, their experience of psychiatric diagnosis and eliciting psychopathology in general, and their experience and training in the use of the OPCRIT+ system itself.

Interrater reliability can also be artificially inflated with non-independent data.[12] OPCRIT+ contains items that are dependent on the ratings of preceding screening items. Thus the data for these items became partially dependent. We tried to minimise this effect by asking raters to manually rate all items, whether or not they had rated a screening item in a certain way, and we also avoided calculating kappa scores for items that were dependent on two or more screening items as such items are, by definition, rarely rated and thus are likely to have high dependency and misleadingly high kappa scores.

Although we have demonstrated reliability for the new items and diagnoses included in OPCRIT+, there remains the question of the validity of the classifications produced by the instrument. The new diagnostic items certainly have face validity and all are taken from established diagnostic criteria[4,5] but the issue of other forms of validity will require further studies.

The advantages and disadvantages of the use of objective criteria for diagnosis in mental health have been questioned and criticised extensively.[13] Nonetheless, no viable alternative has been developed, and the atheoretical objective criteria of the modern versions of the ICD and DSM have undeniably led to a more consistently applied approach to diagnosis.[14] Biomarkers may eventually refine the process but objective symptom assessment and diagnostic stratification is an essential prerequisite to this. The OPCRIT+ system places established sets of objective criteria as well as salient historical markers for aetiology, diagnosis and prognosis into an electronic diagnostic system for clinical environments with benefits for both researchers and clinicians alike.

Finally, we briefly turn to the difficulties of establishing such a standardised computer-based system in routine clinical psychiatric practice. We made considerable effort in the design process of OPCRIT+ to incorporate features that were attractive to rating clinicians (Appendix 1). We collected informal feedback from the participants in the interrater study and subsequently from clinicians at our associated hospital about their experience of using OPCRIT+ and the time taken for completion. Time for completion was influenced primarily by familiarity with the tool and with the case being rated, ranging from 10 to 45 min. Even if it is not seen as time consuming, arguments against such a system may be rooted in an objection to the spirit of the process as a whole; more specifically that the mechanical application of diagnostic criteria with a computer does not include an understanding of the individual case, and therefore has limited utility for conceptualising the individual patient (for a review and commentary in this area, see Bertelsen[15]). Yet, as we have pointed out above, objectivity in psychiatry has led to a more robust stratification of individuals into different diagnostic categories and this is a logical prerequisite to good-quality, replicable research with all the eventual clinical benefits thereof. Moreover, the application of objective criteria should not detract from the clinical skill of eliciting the psychopathology in the first place, or from the need to also form an aetiological theory of each individual case, which we would acknowledge wholeheartedly as being essential in the successful treatment of each individual. We would therefore argue that both approaches to information gathering are needed if individuals with mental disorders are to be successfully treated and clinical research is to progress. We have developed OPCRIT+ to provide a framework for the collection of data from both these spheres of approach, which we hope represents a significant advance in this area.

Funding

The authors acknowledge financial support from the National Institute for Health Research (NIHR) Biomedical Research Centre for Mental Health at the South London and Maudsley NHS Foundation Trust (SLaM) and the Institute of Psychiatry, King's College London. J.R. is funded by a research training fellowship grant from the Wellcome Trust (086635) and was previously funded by a preparatory clinician scientist fellowship from the NIHR Biomedical Research Centre for mental health at SLaM and the Institute of Psychiatry, King's College London.