The potential of electronic medical record systems to support quality improvement work and research in Norwegian general practice.

BACKGROUND: Electronic medical record (EMR) systems are used for many purposes including patient care, administration, research, quality improvement and reimbursement. This study aimed to test a data extraction tool (QTools) and to provide information to support the interpretation of EMR data.
METHODS: Comparison of aggregated practice data for selected EMR fields and interviews with practice staff. Practices received summaries of their data and aggregated data for other practices. Summaries were discussed in interviews.
RESULTS: Fourteen general practices in the Oslo area using the Winmed EMR participated. QTools ran successfully at all 14 practices. Nine practices agreed to interviews. Apart from age and sex, general patient information was poorly recorded. Face-to-face consultations account for 59% of contacts but differences in coding led to variations between practices. Psychiatric problems accounted for 13% of diagnoses, other diagnosis groups rarely accounted for more than 5%. Over 90% of diabetics and 75% of patients with heart disease were identified by diagnosis code alone.
CONCLUSION: Some variation seen in EMR data is due to differences in the way staff use their EMR. These data can support quality improvement work but this requires an awareness of how the EMR is actually used by practice staff.

Background

Electronic medical record (EMR) systems are used for many purposes including patient care, administration, research, quality improvement and reimbursement. These applications require a knowledge of the underlying quality of the data within the EMR so as to avoid misinterpretation. There have been several studies of the quality of EMR data [1-4] and these have shown that data quality is variable. Clinical data are, however, potentially important for quality improvement, epidemiological studies and health services research, as well as clinical care – provided they are of adequate quality.

EMR systems are used by over 90% of general practitioners in Norway [5] and there are essentially only three EMRs in use: Winmed, Profdoc and Infodoc. Several groups, including ours, have used data from these systems to monitor and evaluate the quality of primary care [4,6,7]. To support our work we have developed software called QTools [6], which can extract data from the EMR and export it to a new file for analysis in spreadsheet and statistical packages.

The aim of the current study was to test QTools and some aspects of the quality of EMR data, namely completeness and variation in recording between practices. Additionally, we wished to investigate the use of QTools for planning new research studies aimed at increasing adherence to best practice guidelines. At the time of the current study, the Norwegian Department of Health had initiated a project to improve the treatment of hypertension and our department was developing a set of clinical guidelines as part of this work [8]. One of the major recommendations of these guidelines would concern drug choice. QTools was used to gauge the use of various hypertension drugs in a sample of general practices and to assess whether a full research study aimed at changing prescribing behaviour was warranted.

Methods

All general practices in the Oslo area and using the Winmed EMR were invited to participate in the study during the spring of 2001. These practices were identified from the supplier's list of practices using its software. The choice of Oslo practices and Winmed were pragmatic since we intended to visit the practices and testing of QTools had come further with Winmed than with other EMRs in 2001. Practices consenting to the study were visited by the author who then ran the QTools software. Data were extracted for the period 1/11/1999 to 30/10/2000.

Each practice received summaries of their data and aggregated data for other practices. These summaries were similar to those presented here. Practices were also asked if they would be able to discuss their results with the author. The discussions were focused around the feedback practices received but were not otherwise structured. Data on the prescription of hypertension drugs were used by our research group and were not discussed with practice staff.

The study was approved by the Regional Ethics Committee, the Norwegian Board of Health and the Data Inspectorate.

The QTools data extraction system

None of the three main EMRs provide a simple, flexible way to extract data. To overcome this problem, a collaboration between the Directorate of Health and Social Affairs (SHdir), a third-party software developer (Mediata AS), EMR suppliers, researchers, health professionals and others has developed a software tool called QTools. This tool provides the user with, among other things, an extraction tool that will extract data from the EMR and export it to a new file for analysis in spreadsheet and statistical packages. The user can extract a wide range of fields within the EMR and use selection criteria (eg. age, diagnoses and prescriptions) to extract data on a subset of patients. In principle, every field in the EMR can be extracted, although most free text fields are currently not available for extraction. The fields that can currently be extracted or used as selection criteria are shown in Table 1. Two export formats are supported (ASCII and Microsoft Access) and three levels of anonymisation (patient completely anonymous, EMR patient number and unanonymised). The QTools extraction system resembles MIQUEST software used in the United Kingdom (see ), in that it is a module that sits beside the EMR.

Table 1

EMR fields that can be extracted or used as selection criteria by QTools.

Fields that QTools can extract
• Patient number	• Patient name*
• Date of birth	• Sex*
• Age*	• Post code*
• Civil status	• Height and weight
• Profession	• Smoking, alcohol etc history
• Event date (contact, prescription etc)	• Diagnosis (ICPC)*
• Drug name*	• Drug code (ATC)*
• Laboratory test name*	• Laboratory test result
• Blood pressure	• Doctor*
Selection criteria
• All fields in the above list marked with a * can be used as selection criteria
• Event period (from... until...)
Operators that can be used with selection criteria
• Intervals (eg. 10 – 40)	• Greater than/less than (>, <)
• Comma-separated lists (eg. 12, 6, 89)	• Wild cards (eg. R4*)
• Include/Exclude matches

QTools has two particular strengths: it is being developed for all three EMRs and it can be distributed as a 'runtime version' that will run predefined extractions with a minimum amount of effort from the general practitioner. The development of QTools was financed by the Norwegian Research Council as an academic and industry collaboration. QTools is therefore a tool that SHdir can use in research projects and which Mediata AS can choose to market whole or in parts. The author and SHdir have no commercial interest in QTools.

Analysis

The extracted data were combined under six headings:

• Blank fields

• Use of various types of patient contact

• Recorded frequency of eight diagnostic groups

• Standardised frequency of eight diagnostic groups

• Recording of two chronic diseases (diabetes and heart disease)

• Drug prescription for hypertension

Completion of selected fields and use of various contact types were presented as simple percentages. The mean of the practice means was calculated, together with the 95% confidence interval around this 'mean of the means'. The n used in the calculation of confidence interval was the number of participating practices, not the total number of field items or contacts; ie. the confidence intervals represent the precision of the estimated average proportion across practices.

To make comparison of diagnostic information easier, eight commonly occurring problems were selected. These were asthma, back problems, diabetes, elevated cholesterol, heart disease, hypertension, psychiatric conditions and respiratory problems. All Norwegian EMRs use the International Classification for Primary Care (ICPC) system and the eight diagnosis groups were defined using these codes (see Figure 1). These groups were defined by the Nordic Medico-Statistical Committee [9].

Figure 1

Standardised frequency for the eight diagnostic groups at two general practices Standardised frequency for the eight diagnostic groups at two general practices, practice E and practice K. The standard population has the value 100. The 'range shown for each diagnostic group is the 95% confidence interval.

The eight groups were also standardised for age and sex. Indirect standardisation was used, a technique that compares the number of instances of a given diagnosis group in the local population (ie. a single practice) with the number of cases that might be expected if the local population had the same disease rate in each specific age and sex group as a chosen standard population (ie. all practices) [10]. The result of standardisation was expressed as a percentage, with the standard population having a value of 100. If the confidence intervals around the standardised rate did not include the value 100 then the difference in rates between the practice and all practices combined was considered significant.

Patients receiving diagnosis codes for diabetes and heart disease were cross-checked against patients receiving drugs used almost exclusively in connection with these conditions. ICPC codes were used to select diagnoses and Anatomical Therapeutic Chemical (ATC) classification codes were used for drugs. The relevant ICPC code for diabetes is T90 and the relevant drugs were insulin and analogues (ATC=A10A*: *=wildcard), sulphonamides and urea derivatives (A10BB*), biguanides (A10BA*), alpha glucosidase inhibitors (A10BF*) and other oral blood glucose lowering drugs (A10BX). For heart disease the ICPC codes were K74 – K80 and the drugs were glyceryl trinitrates (C01D A02), isosorbide dinitrates (C01D A08) and isosorbide mononitrates (C01D A14). Results were expressed as the percentage of patients identified by ICPC code only.

Prescriptions of anti-hypertensive drugs to patients receiving a diagnosis of hypertension were obtained. Drugs were grouped according to their ATC codes, these groups being: thiazides (C03A, C03E), ACE-inhibitors (C09A, C09B), alpha-blockers (C02C), beta-blockers (C07), calcium channel blockers (C08) and angiotensin II-inhibitors (C09C, C09D). Combined preparations (eg. ACE-inhibitors and thiazides) were excluded. Results were expressed as the mean use of each drug type, where the mean is the mean of the practice means. A 95% confidence interval around this mean was also calculated.

Results

General

Forty-eight practices were invited to participate. Fourteen agreed to participate, two were not general practices and eight did not answer multiple telephone calls and faxes. This gave a consent rate of 30% (14/46). Nine practices agreed to discuss their results. QTools was successfully installed and run at all 14 practices, the only problem being with the file that specified the fields to extract for patient contacts. This problem was easily corrected but did lead to a loss of patient contact data for three practices (see 'Use of various types of patient contact'). The number of patients contained within the EMRs for the 14 practices was 120934, ranging from 930 patients to 38508 patients at the smallest and largest practices.

Blank fields

Table 2 shows the percentage of blanks for seven fields that can be expected to contain data for all patients or, in the case of lifestyle information, many patients. We did not want to include all patients who had visited their doctor in the period 1/11/1999 to 30/10/2000 for this extraction because the extracted file could be very large. We selected, therefore, 500 patients at random from each practice's EMR, giving 7000 patients in all (14 × 500). This was a pragmatic balance between extraction time, file size and an adequate amount of data.

Table 2

Percentage of selected fields which were blank in 7 000 patients' electronic medical record.

Field	Mean % blank	Lower 95% CI	Upper 95% CI
Sex	0	0	0
Age	0	0	0
Marital status	73	59	88
Height	100	99	100
Weight	99	97	100
Job	61	49	73
Lifestyle information (smoking, alcohol etc)	97	95	99

CI = Confidence interval

Sex and age are obligatory fields in the Winmed EMR. The remaining five fields were generally poorly completed.

Practice staff were unanimous in explaining why this is the case: staff have found better ways to record much of this information. Height and weight are generally recorded in a section of the EMR that allows measurements to be recorded over time, such as the laboratory tests section. Lifestyle information is often placed in the free text field of the main EMR entry for each patient contact. There was no clear explanation for why marital status and job information are poorly completed.

Use of various types of patient contact

Table 3 shows the use of various types of patient contact. The table is based on 100610 patient contacts (this is not the same as individual patients because some patients may have contacted their general practice more than once) at 11 of the 14 practices.

Table 3

Use of the nine contact types given in the Winmed electronic medical record system. The data are based on 100610 contacts.

Type of contact	Mean	Lower 95% CI	Upper 95% CI
Unknown	8	6	10
Telephone	24	16	33
Simple (eg. picking up a prescription)	7	2	11
Consultation	45	37	54
Consultation, immediate help	6	2	10
Consultation, checkup	8	1	15
Home visit	1	0	1
Home visit, immediate help	1	0	1
Work-related health checkup	0.2	0	0

CI = confidence interval

The three remaining practices were not included because of an error in the file used by QTools to select the fields to be extracted. The error was corrected but it was not possible to run the extractions again at these three practices. Face-to-face practice consultations are used most often, accounting for almost 59% of all contacts.

Practice staff considered the variation in the use of contact types to be largely due to different classification routines. One practice routinely recorded all consultations regarding a chronic illness as 'Consultation, check up'. This contact type was used for 34% of all contacts at this practice, compared to the average of around 7%. Patients coming to collect repeat prescriptions were also coded as having had this contact type by two practices, including the one already mentioned. Two practices had a general policy of seeing patients rather than giving telephone consultations. Telephone consultations attract a lower fee than a standard consultation (€ 4.70 versus €14.90) and this was cited by one of these two practices as the reason for its low use (7% against an average of 19%). Finally, practices with few practice nurses had fewer 'Simple consultations' (eg. blood tests) because the doctors did these and registered the contact as a normal consultation.

Recorded frequency of eight diagnostic groups

All occurrences of a diagnostic code within the one year period were extracted. This generated 220400 diagnosis codes for 31632 patients (Table 4). Psychiatric problems accounted for 13% of diagnoses on average with a range of 5% to 26%. The other diagnosis groups rarely or never accounted for more than 5% of total diagnoses.

Table 4

Variation in the use of diagnoses within eight diagnostic groups at the 14 general practices. The mean % of all diagnoses is the mean of the 14 individual practice means.

Field	ICPC codes for diagnosis grouping	Mean % of all diagnoses	Lower 95% CI	Upper 95% CI
Psychiatric conditions	P01-P29, P70-P99	13.2	10.2	16.2
Back problems	L02, L03, L84-L86	5.1	4.0	6.3
Hypertension	K86, K87	2.9	1.9	3.9
Heart disease	K74-K80	2.6	1.6	3.7
Diabetes	T90	2.0	1.3	2.7
Asthma	R96	1.7	1.1	2.3
Respiratory	R05, R09-R23, R70-R83, H71-H74	1.2	0.9	1.5
Elevated cholesterol	T93	0.5	0.2	0.8

CI = Confidence interval ICPC = International Classification for Primary Care

Practice staff felt that the respective frequencies of the eight diagnostic groups fairly reflected their case mix. All agreed that patients with psychiatric problems are the largest single group.

Standardised frequency of eight diagnostic groups

When the diagnosis data were standardised for age and sex, all practices showed a statistically significant difference from the standard value of 100 for at least one diagnostic group. The data shown in Figure 1 for Practice E and K in F are typical.

All practices mentioned that the way staff code diagnoses varies and that this would be reflected in the standardised diagnosis data. For example one doctor at Practice K thought that he probably used the diagnosis code for asthma (R96) incorrectly and suggested this as the reason for the much higher prevalence of asthma at this practice. Two practices mentioned that the code for elevated cholesterol (T93) is actually used more widely, leading to differences between practices depending on individual patterns of use. Most practices also said that it is common to record only the two or three most important diagnoses associated with each patient contact.

Recording of two chronic diseases (diabetes and heart disease)

Recording of diagnosis and drug information were obtained for 849 diabetics and 1380 patients with heart disease (Table 5). Generally, over 90% of diabetics and 75% of patients with heart disease were identified by diagnosis code alone.

Table 5

Combining diagnostic and drug information to identify patients with diabetes and heart disease.

Diabetes				Heart disease
Practice	No. ICPC /No. ATC	Overlap	% identified by ICPC	Practice	No. ICPC /No. ATC	Overlap	% identified by ICPC
A	143 / 99	89	94	A	147 / 79	46	82
B	20 / 15	12	87	B	6 / 7	2	55
C	4 / 2	2	100	C	7 / 1	0	88
D	82 / 57	52	94	D	176 / 103	70	84
E	23 / 16	13	89	E	23 / 7	4	89
F	25 / 16	15	96	F	57 / 33	27	91
G	111 / 77	74	97	G	161 / 110	59	76
H	30 / 11	11	100	H	19 / 16	9	73
I	31 / 14	13	97	I	32 / 11	7	89
J	141 / 84	73	93	J	214 / 147	77	75
K	61 / 37	35	97	K	108 / 60	43	86
L	91/ 51	50	99	L	101 / 72	50	82
M	33 / 23	23	100	M	35 / 17	7	78
N	12 / 10	8	86	N	19 / 20	7	59

ICPC = InternationalClassification for Primary Care (coding system for diagnoses) ATC = Anatomical Therapeutical Chemical (coding system for drugs)

Staff at one practice suggested that some patients receiving prescriptions for diabetes but who were not registered as diabetic may have polycystic ovaries syndrome. Staff at another practice suggested that diabetics coming from another practice may have received prescriptions in connection with their diabetes but not yet have had a consultation concerning their diabetes. The diagnosis code for diabetes may not, therefore, be in the patient's medical record.

Drug prescription for hypertension

There were 1588 patients diagnosed as hypertensive and these patients received 3580 prescriptions for anti-hypertensive drugs. Of these patients, 1230 (77%) were receiving treatment. The various types of prescribed drug are given in Table 6. Thiazide use was 4% of prescriptions (confidence interval 2% to 6%).

Table 6

Type of drug prescribed for hypertension.

Drug	Mean % use	Lower 95% CI	Upper 95% CI
ACE inhibitors	26	19	33
Alpha channel blockers	3	2	5
Angiotensin II antagonists	25	14	35
Beta-blockers	19	14	23
Calcium channel blockers	23	18	29
Thiazides	4	2	6

CI = Confidence interval

Discussion

The EMRs used in Norwegian primary care are a rich source of data covering many aspects of patient care. However, as this study and others [1,2] have found, one must be extremely careful when interpreting these data.

The current study had a number of weaknesses. It was small, considered only one of the three EMRs used in Norway and focused on a few key areas of the EMR. Only nine of the 14 general practices could participate in the interviews. There was a broad range of practice sizes and recruitment of practices did not specifically seek to obtain a balanced mix of sizes. This may affect the completeness of EMR data in that staff working in a large and busy practice may have less time to complete the EMR. Data were extracted for a one year period, which made the extractions fast at the expense of losing some detail. The drugs used for the cross-check (Table 5) can be used in connection with other illnesses, particularly those listed for heart disease. Combined with the single year of data this could have contributed to the less than 100% identification by diagnosis of these patients, though it is doubtful that this accounts for the full shortfall. Checking accuracy of data in the EMR is difficult, particularly in Norwegian primary care where there is no paper or other version of the medical record to allow a cross-check. Moreover, at the time of the study general practitioners in Norway did not have a fixed patient population: a patient was free to visit the first doctor who gave him an appointment. Information on chronic disease can, therefore, be difficult to interpret.

One can, however, draw some careful conclusions from the data presented here. Firstly, one should not assume that health professionals use an EMR as intended. Practice staff were unequivocal in saying that they develop their own methods of recording information. Although height and weight fields were poorly completed, this information is in the EMR but in a different field to the one provided by the EMR supplier. This affected the blank fields results but practice staff did not raise this as an issue for contact information, diagnosis and prescription.

Lack of time during patient contacts means that only the most important information is recorded and what is considered important varies from individual to individual. Reasons for such variation can include not prioritising some types of information, using the EMR's main text area instead of a special field, different practice routines and being selective when recording diagnoses. Such sources of variation need to be considered when EMR data are aggregated, or when EMR data are used by decision support and reminder tools. Some variation may be an artefact of recording but other variation may reflect genuine differences in practice and even imperfect EMR data can be used to inform discussion as to whether this variation is acceptable. EMR data could form the basis of an audit and feedback system that could be expected to have a small to moderate effect on practice [11]. Using routine EMR data may make even small effects worth pursuing because data collection carries little or no additional cost.

The frequencies of the diagnoses shown in Table 4 differ in some regards to those presented in a recent publication by the Nordic Medico Statistical Committee (NOMESCO) [12], despite both studies using the same diagnostic groupings. Although the frequency of back pain, hypertension, diabetes and heart disease were similar, psychiatric problems and respiratory problems were very different (asthma and elevated cholesterol did not appear in the NOMESCO publication). NOMESCO reported a frequency of 15% for respiratory problems, against 1% for this study; the figures for psychiatric problems are respectively 5% and 13%. The reason for this difference is not clear. The NOMESCO study was based on data from 19 rural practices rather than from urban practices as used in the current study. It is possible that the case-mix in rural and urban practices is different, particularly with regard to psychiatric disorders. The NOMESCO study also used a shorter data collection period of four weeks, which, depending on the season (not reported), could perhaps explain a spike in respiratory problems.

The low use of thiazides at the 14 practices provided evidence that current practice is not yet best practice [8,13] and that a research project with the objective of supporting the implementation of best practice guidelines for the treatment of hypertension is justified. A large, randomised, controlled trial is now underway in two areas of Norway [14]. One of the major outcome measures for this trial is the proportion of first time prescriptions for hypertension where thiazides are prescribed. QTools can, therefore, be a useful tool with which to collect preliminary data to plan future research studies. Moreover, the preliminary data can help to identify weaknesses in the EMR data, which can then be addressed in the research study. For example, an alternative method of obtaining information on smoking habits was developed for the new study because Table 2 shows that these data are not recorded in the EMR field provided for this purpose. The EMR could also be used as one means of evaluating the effect of quality improvement programs (such as that described in reference 14) on patient health and professional practice.

Conclusion

As Mitchell and Sullivan [15] stated in their recent review, we should accept that the computer is a useful tool and start trying to evaluate key outcomes for patients, practices and the health service as a whole. EMR data can support quality improvement work if used wisely; such work could also contribute to steadily improving data quality. We should not use data quality as an excuse for inaction: our data need not be perfect to be useful.

Competing interests

None.

Pre-publication history

The pre-publication history for this paper can be accessed here: