Challenges in validating quality of care data in a schizophrenia registry: experience from the Danish National Indicator Project.

BACKGROUND: Improvement of quality of care for psychiatric patients is a key objective of health care systems worldwide. Consequently, there is an increasing interest in documenting quality of care; however, little is known about the validity of the available data on psychiatric care.
OBJECTIVE: To assess the validity of process of care data recorded in the Danish National Indicator Project (DNIP), a national population-based registry containing quality of care data of patients diagnosed with schizophrenia in Denmark.
METHODS: A random sample of 1% of patients with schizophrenia registered in the DNIP between 2004 and 2009 (111 inpatient and 85 outpatient) was identified for validation. Medical records for these patients, which were used as the gold standard, were retrieved and reviewed for information on the processes of care received. Agreement between the data in the DNIP and the medical records were assessed by computing sensitivity, specificity, and positive and negative predictive values.
RESULTS: The agreement between the recorded processes of care in the DNIP and in the medical records varied substantially across the individual process of care variables. However, a collection of the processes of care demonstrated a high agreement (80% or more) between data in the DNIP and the medical records, according to all examined aspects of data validity (sensitivity, specificity, and positive and negative predictive values). The medical records contained varying levels of missing information regarding the processes of care, from 1% for antipsychotic medication prescription to 54% for psychoeducation.
CONCLUSION: Current documentation practices in Danish psychiatric hospitals appear to be inconsistent and may preclude the use of psychiatric medical records as the gold standard when validating registry data.

Introduction

Psychiatric registries are frequently used for administration, quality of care improvement, and research. The advantages of using well-established registries include substantial time and cost savings, availability of large sample sizes, high levels of representation, and lower recall bias and nonresponse bias.1,2 The usefulness of registries is entirely dependent on the validity of the data, including the completeness and quality of the recorded data; however, little is known about the validity of psychiatric registries. Byrne et al3 identified and reviewed 14 studies that examined the validity of administrative registries used in psychiatric research. The methodological quality of the constituent studies varied widely, making it difficult to draw any conclusions. Consequently, further information on the validity of psychiatric registries is needed to ensure that available data are used in a meaningful way, to avoid misleading results.

Working systematically with improvements in psychiatric health care relies on access to empirical evidence of the care provided to patients in routine clinical settings. Consequently, there is increasing interest in data on the basic processes of psychiatric health care, including access, detection, appropriateness of treatment, safety, and continuity of care.4 Nonetheless, very little is known about the validity of the available data regarding the processes of psychiatric care.

The aim of this study was to examine the validity of electronically maintained records of the processes of care for individuals with schizophrenia in the Danish National Indicator Project (DNIP) – a national schizophrenia registry.

Methods

The Danish health care system provides tax-paid health care to the country’s 5.5 million residents, including free access to hospital care and general practitioners. Patients with schizophrenia who require psychiatric treatment are exclusively admitted to public psychiatric hospitals. Data from this entire population is recorded through a large number of administrative and medical registries, which are used for monitoring and regulating all central aspects of the public sector, including the health care system. All Danish citizens are assigned a unique, ten-digit civil registration number that is used in all registries,5 enabling unambiguous linkage among the sources of data.

The DNIP – schizophrenia

Participation in the DNIP is mandatory for all Danish psychiatric hospitals, relevant clinical departments, and units treating patients with schizophrenia. The objective of the DNIP is to document, monitor, and improve diagnosis and care provided by the Danish psychiatric health care system among patients with schizophrenia. The registry has been established through collaboration among the Ministry of Health, the National Board of Health, the Danish Regions, health care professional organizations, and scientific societies.6 Data collection was initiated in 2004 and the data collected include information on key processes of care among psychiatric inpatients and outpatients. These processes of care were identified by an expert panel consisting of physicians, psychologists, nurses, occupational therapists, and social workers,6 and are described in next column. Data for the DNIP are collected using a registration form with detailed written instructions. For inpatients, the registration form is completed based on documentation in the medical record at the time of discharge from the psychiatric ward. Data from outpatients are recorded once a year. In 2009, the DNIP registry contained records for 98% of all inpatients and 93% of all outpatients with schizophrenia in the Danish psychiatric health care system, when compared with the National Registry of Patients, which captures all admissions and discharges from Danish hospitals. 7 The DNIP defines incident patients with schizophrenia as individuals who have been diagnosed with schizophrenia 12 months before the date of hospital discharge.

Study population

We aimed to retrieve and review medical records from a random sample of adult (≥18 years) patients (including inpatients and outpatients) recorded in the DNIP between January 1, 2004, and March 31, 2009, to determine agreement between information in the medical records and electronically entered information. Schizophrenia was defined according to the International Classification of Diseases version 10 (ICD-10)7 F20.0 to F20.9. A computer-generated 1% random sample was drawn among inpatients (n = 10,757) and outpatients (n = 7800). A total of 125 inpatients and 100 outpatients were included, 32 of whom were incident patients. The sample size was not determined based on a formal analysis but, rather, as a pragmatic tradeoff between achieving a reasonable statistical precision of the estimated sensitivity, specificity, and positive and negative predictive values and having a feasible number of medical records to retrieve and review.

Processes of care

Table 1 lists the definitions of recorded processes of care contained within the DNIP. The main processes are: antipsychotic medical treatment, contact with relatives, and psychoeducation. In addition, inpatients’ data included scheduled psychiatric aftercare, professional support, and suicide risk assessment. Data on individuals receiving outpatient treatment include continued contact with outpatient clinics and professional contact person(s). Finally, data on the processes of diagnosing schizophrenia in the incident patients were collected, including assessment of psychopathology, assessment of psychopathology by a psychiatric specialist, assessment of psychopathology using interview forms, assessment of cognitive function by a psychologist, and assessment by a social worker.

Table 1

Definitions of the processes of care provided for patients with schizophrenia

Processes of care	Definition
Antipsychotic medical treatment	Prescription for antipsychotic medical treatment at psychiatric hospital discharge or status period as psychiatric outpatient14–17
Contact with relatives	Staff contact with the patient’s relatives during hospitalization or status period18–20
Psychoeducation	Overall indication of whether the patient received psychoeducation during hospitalization or status period21–23
Psychiatric aftercare	Planned professional support for inpatients after discharge in patient’s own homes, residential facilities, or care homes, as well as for housing24
Professional support	Scheduled visits for inpatients with psychiatric aftercare, including outpatient psychiatric treatment team, psychiatric clinic, or general practice after hospital discharge
Suicide risk assessment	In the week leading up to the patient’s discharge, a required clinician’s assessment of the patient’s risk of suicide, including an evaluation of depressive symptoms
Ongoing contact with outpatient	Only outpatients: indication of whether the staff has ongoing contact with the patient during outpatient treatment
Professional contact person	Only outpatients: indication of whether the patient has an assigned contact person during outpatient treatment
Only incident patients
Assessment of psychopathology	An overall record of whether the patient has been assessed for psychopathological characteristics
Assessment of psychopathology by specialist in psychiatry	Indication of whether the patient’s psychopathological assessment was performed by a specialist in psychiatry
Assessment of psychopathology by interview form	Indication of whether the patient received a diagnostic interview with an established interview instrument, such as the Schedules for Clinical Assessment in Neuropsychiatry (SCAN) or Present State Examination (PSE)25,26
Assessment of cognitive function	Indication of whether cognitive testing of the patient was performed by a psychologist
Assessment by social worker	Assessment of need for acute or longer-term support, such as help with changing housing, financial help to purchase medicine, educational guidance, rehabilitation, and application for disability benefits

Medical record review

For patients with multiple registrations in the DNIP, we randomly selected a hospitalization record among inpatients and a status record date among outpatients and located data from the equivalent dates in the medical records. We were unable to retrieve the medical paper records for 10% (n = 13) of the inpatients and 15% (n = 15) of the outpatients; therefore, analyses were conducted on 111 inpatients and 85 outpatients. The 13 inpatients had been hospitalized in ten different hospitals, and the 15 outpatients had contact with eleven different outpatient clinics. No systematic differences were found in relation to the distribution of processes of care recorded in the DNIP, sex, or age, when comparing patients with and without available medical records.

All medical records were reviewed by a single reviewer (CGP, who is a certified and clinically experienced nurse) using the DNIP data definitions. The reviews were performed systematically, in accordance with the DNIP definitions for individual processes of care. Problematic records were discussed with specialists in psychiatry and clinical epidemiology, and consensus was reached.

Data from the medical records were entered into a database, and their accuracy was confirmed by re-entering the data a second time.

Ethics

The study was approved by the Danish Data Protection Agency (J no 2008-41-2894) and the Danish National Board of Health (J no 7-604-04-2/80/AHA).

Statistical analysis

The sensitivity, specificity, and positive and negative predictive values of data recorded in the DNIP were computed using the reviewed medical records as the gold standard. Sensitivity of DNIP registry data was expressed as the proportion of patients who were registered with a verified and received process of care in the DNIP (numerator), divided by the total number of patients in our random sample who received a process of care, based on the medical records (denominator). Specificity was computed as the number of patients without a specified process of care according to the DNIP (numerator), divided by the total number of patients without a process of care, according to the medical records. The positive predictive value (PPV) of data recorded in the DNIP was computed as the number of patients with a verified received process of care as the numerator and the total number of patients registered with a received process of care in the DNIP as the denominator. Similarly, negative predictive values were computed as the number of patients without a process of care as the numerator and the total number of patients registered without a process of care in the DNIP as the denominator. All estimates are presented with a 95% confidence interval (CI). The data were analyzed using STATA software (v 11.1; StataCorp LP, College Station, TX).

Results

A total of 18,557 patients with schizophrenia were registered in the DNIP from January 1, 2004, until March 31, 2009. Of these patients, 58% (n = 10,757) were inpatients and 42% (n = 7800) were outpatients (Table 2).

Table 2

Patient characteristics and processes of care among patients with schizophrenia

	Totaln = 18,557	Inpatientsn = 10,757	Outpatientsn = 7800
	n (%)	n (%)	n (%)
Sex
Female	7445 (40)	4352 (40)	3093 (40)
Male	11,112 (60)	6405 (60)	4707 (60)
Age (years)
≥18–29	4032 (22)	2632 (24)	1400 (18)
30–39	4608 (25)	2761 (26)	1847 (24)
40–49	4562 (25)	2628 (24)	1934 (25)
50–60	3198 (17)	1700 (16)	1498 (19)
60+	2157 (11)	1036 (10)	1121 (14)
Antipsychotic medical treatment
Yes	15,836 (85)	9264 (86)	6572 (84)
No	1117 (6)	417 (4)	700 (9)
Unknown	1604 (9)	1076 (10)	528 (7)
Contact with relatives
Yes	7325 (39)	4581 (43)	2744 (36)
No	10,147 (55)	5438 (50)	4709 (60)
Unknown	1085 (6)	738 (7)	347 (4)
Psychoeducation
Yes	10,851 (58)	6338 (59)	4513 (58)
No	6753 (37)	3792 (35)	2961 (38)
Unknown	953 (5)	627 (6)	326 (4)
Psychiatric aftercarea
Yes	–	5325 (49)	–
No	–	1079 (10)	–
Unknown	–	4353 (41)	–
Professional supporta
Yes	–	1218 (11)	–
No	–	210 (2)	–
Unknown	–	9,329 (87)	–
Suicide risk assessmenta
Yes	–	4528 (42)	–
No	–	1058 (10)	–
Unknown	–	5171 (48)	–
Ongoing contact with outpatientb
Yes	–	–	3433 (44)
No	–	–	1367 (18)
Unknown	–	–	3000 (38)
Professional contact personb
Yes	–	–	3704 (47)
No	–	–	375 (6)
Unknown	–	–	3721 (47)

Notes:

Only inpatients;

Only outpatients.

Table 3 illustrates the proportion of individuals lacking registration of processes of care in medical records among the random sample selected for the current study. The completeness of medical information on antipsychotic medical treatment was high, as only 1% (n = 2) of the patients lacked this registration information. In contrast, the majority of medical records (54%) did not contain information on the use of psychoeducation (n = 106), and 50% of the cases lacked information regarding staff contact with patients’ relatives (n = 98). Among the inpatients, 43% of the records had no information on professional support after discharge (n = 48), and 40% lacked information on suicide risk assessment (n = 45). Among the diagnostic processes of care for incident patients, the highest proportions of medical records with missing data were 37.5% without information on the assessment of psychopathology by psychiatric specialists (n = 12) and 37.5% lacking information on the assessment of psychopathology by interview form (n = 12). The records with missing data were not restricted to a few departments; they represented a large number of institutions.

Table 3

Proportions of medical records that lacked information on individual processes of care among patients with schizophrenia

Processes of care	Medical records n (%)
All patients	n = 196
Antipsychotic medication	2 (1)
Contact with relatives	98 (50)
Psychoeducation	106 (54)
Only inpatients	n = 111
Psychiatric aftercare	11 (10)
Professional support	48 (43)
Suicide risk assessed	45 (41)
Only outpatients	n = 85
Ongoing contact with patients	0 (−)
Professional contact person	0 (−)
Only incident patients	n = 32
Assessment of psychopathology	2 (6)
Psychopathology by specialist	3 (9)
Psychopathology by interview form	12 (38)
Cognitive test	12 (38)
Social worker	6 (19)

Table 4 shows the overall sensitivity, specificity, and predictive values for each of the 13 processes of care in the DNIP compared with the information available in the medical records of the current sample. In general, agreement between the recorded processes of care in the DNIP and the medical records varied substantially across the individual processes of care. A few of the processes demonstrated high agreement (80% or more) between data in the DNIP and the medical records for all examined parameters (sensitivity, specificity, positive and negative predictive values). Sensitivity ranged from 11% to 96%, with four processes having a sensitivity of 80% or more: medical antipsychotic treatment had 96% sensitivity (95% CI: 92%, 98%); psychoeducation had 80% sensitivity (95% CI: 69%, 87%); evaluated psychopathology had 81% sensitivity (95% CI: 62%, 94%); and assessment of psychopathology by psychiatric specialist had 85% sensitivity (95% CI: 62%, 97%). Specificity ranged from 11% to 100%, with three processes having a specificity of 80% or more: contact with relatives had 82% specificity (95% CI: 48%, 98%); cognitive testing had 100% sensitivity (95% CI: -); and social worker involvement had 100% specificity (95% CI: -). The ranges of the positive and negative predictive values were 44%–100% and 5%–100%, respectively, with six and one processes, having a predictive value of 80% or more. This included medical antipsychotic medication with a PPV of 97% and 95% CI of 94%, 99%; psychiatric aftercare with a PPV of 89% and 95% CI of 78%, 95%; ongoing contact with patients with a PPV of 100% and 95% CI -; professional contact person with a PPV of 98% and 95% CI of 91%, 99%, evaluated psychopathology with a PPV of 85% and 95% of 65%, 96%; and assessment by a social worker with a PPV of 86% and 95% CI of 65%, 97%. The negative predictive value included psychopathology by specialist 100%.

Table 4

Sensitivity, specificity, and predictive values of data on processes of care registered in the DNIP, using medical records as gold standard

	Verified positive/total N	Sensitivity % (95% CI)	Verified negative/total N	Specificity % (95% CI)	Verified N/total N	PPV % (95% CI)	Total negative N/total N	NPV % (95% CI)
Processes of care
Antipsychotic medication	179/186	96 (92, 98)	5/8	17 (24, 91)	179/184	97 (94, 99)	5/11	45 (17, 77)
Contact with relatives	62/87	71 (60, 80)	9/11	82 (48, 98)	62/78	79 (69, 88)	9/109	8 (1, 15)
Psychoeducation	56/70	80 (69, 87)	6/62	10 (4, 20)	56/127	44 (35, 53)	6/62	9 (1, 20)
Psychiatric aftercarea	55/97	57 (46, 67)	1/3	33 (1, 91)	55/62	89 (78, 95)	1/10	9 (1, 45)
Professional supporta	10/58	17 (9–29)	0/5	–	10/14	71 (42, 92)	0/1	–
Suicide risk assesseda	39/66	56 (46, 71)	–d	–d	39/55	71 (57, 82)	–d	–d
Contact with patientsb	52/83	63 (51, 73)	1/2	50 (1, 99)	52/52	100 (−)	1/18	5 (1, 27)
Professional contact personb	61/84	73 (62, 82)	–d	–d	61/62	98 (91, 99)	–d	–d
Evaluated psychopathologyc	22/27	81 (62, 94)	0/3	–	22/26	85 (65, 96)	0/1	–
Psychopathology by specialistc	17/20	85 (62, 97)	1/9	11 (2, 48)	17/25	68 (46, 85)	1/1	100 (−)
Psychopathology by interview formc	2/18	11 (1–35)	0/2	–	2/2	100 (−)	0/1	–
Cognitive testc	6/19	32 (13, 57)	1/1	100 (−)	6/8	75 (35, 97)	1/17	5 (1,18)
Social worker assessmentc	19/25	76 (55, 90)	1/1	100 (−)	19/22	86 (65, 97)	1/4	25 (1,80)

Notes:

Only inpatients;

Only outpatients;

Only incident patients;

Patients did not have “no” recorded in the medical records.

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; CI, confidence intervals.

Discussion

In this validation study, we found varying levels of agreement between data on delivered processes of schizophrenia care recorded in a national schizophrenia registry and the original medical records. The variation was found in relation to sensitivity, specificity, and positive and negative predictive values. Documentation of the provided care was incomplete in a high proportion of the examined medical records, and the records were consequently a problematic gold standard in the validation process.

Our findings indicate a concerning pattern of documentation practices in the Danish psychiatric hospitals regarding patients with schizophrenia. A variety of factors may potentially have contributed to this problem, including a limited availability of electronic medical records systems in the psychiatric clinical wards during the study period, difficulty locating a substantial proportion of the paper versions of medical records, and a tradition of inadequate documentation of psychiatric care, compared with current legislative requirements. Several studies have found that electronic records are beneficial in clinical documentation practices, with the rationale that they contain more complete and readily available information than paper records.9 This convenience ultimately promotes a better quality of care and patient safety. In addition, integrated electronic reminder systems found in electronic medical records may improve the quality of care,10 although this is not a consistent finding,11 and the use of electronic checklists has been reported to improve the quality of patient care.12 The disadvantages of electronic medical records include the high financial costs, difficulty in technical training and support for the staff, and complexity of the software and its interface (which may be combined with a lack of customizability, reliability, and standardization). Furthermore, due to difficulty in accessing computers and concerns that the use of electronic patient records will reduce the time spent bedside with the patients, logistical problems also may play a role in the underuse of electronic medical records.13 However, as demonstrated in this study, the use of paper records, which is still the current practice in many clinical settings, may be associated with serious logistical problems regarding locating and retrieving the records from institutional or hospital archives. This problem may constitute a particular challenge with patients with schizophrenia, due to the frequent readmissions in this patient group. The insufficient documentation of the provided care in the medical records that we were able to locate and retrieve may reflect the combination of a health care system under pressure due to financial restrictions and an inappropriate tradition of not giving high priority to organized documentation.

The insufficient access to a valid gold standard of provided care represents a major challenge in attempts to validate the DNIP, as well as other clinical registries. In addition to striving toward the implementation of electronic patient records and improved documentation practices, alternative information sources could be considered for the purpose of validation. Sources other than medical records could potentially contain valid corroborating data, such as nursing records and laboratory information systems. Therefore, combining different information sources could potentially result in a more accurate picture of the provided care for validation purposes and beyond, rather than relying solely on patient medical records.

The strengths of our study include the use of a random sample of both inpatients and outpatients with schizophrenia, prospectively registered in a national, population-based registry. Therefore, the findings are likely to be representative of the data recorded in the registry. The use of the unique civil registration number system in Denmark enabled us to locate the majority of the relevant medical records and accurately link information between medical records and the DNIP. All records were reviewed by the same experienced individual. While the criteria used to make the judgments were developed in collaboration with experienced colleagues and according to the detailed data definitions of the DNIP, the validation decisions within the current study are consistent, as they rely on the judgment of a single person. The most important limitation of this study, and perhaps the most important finding, was the high proportion of medical records with missing information on several of the examined processes of care, which made it difficult to draw firm conclusions about the validity of the data recorded in the DNIP.

The tasks of conducting quality assessments, facilitating the coordination of care, and managing an individual patient’s treatment all rely on complete and up-to-date medical records. Inaccurate documentation in medical records complicates each of these necessary activities. Current medical record documentation practices in Danish psychiatric hospitals appear to be inconsistent and may preclude the use of psychiatric medical records as the gold standard when validating registry data.