Using standardised patients to assess the quality of medical records: an application and evidence from rural China.

BACKGROUND: Medical records play a fundamental role in healthcare delivery, quality assessment and improvement. However, there is little objective evidence on the quality of medical records in low and middle-income countries.
OBJECTIVE: To provide an unbiased assessment of the quality of medical records for outpatient visits to rural facilities in China.
METHODS: A sample of 207 township health facilities across three provinces of China were enrolled. Unannounced standardised patients (SPs) presented to providers following standardised scripts. Three weeks later, investigators returned to collect medical records from each facility. Audio recordings of clinical interactions were then used to evaluate completeness and accuracy of available medical records.
RESULTS: Medical records were located for 210 out of 620 SP visits (33.8%). Of those located, more than 80% contained basic patient information and drug treatment when mentioned in visits, but only 57.6% recorded diagnoses. The most incompletely recorded category of information was patient symptoms (74.3% unrecorded), followed by non-drug treatments (65.2% unrecorded). Most of the recorded information was accurate, but accuracy fell below 80% for some items. The keeping of any medical records was positively correlated with the provider's income (β 0.05, 95% CI 0.01 to 0.09). Providers at hospitals with prescription review were less likely to record completely (β -0.87, 95% CI -1.68 to 0.06). Significant variation by disease type was also found in keeping of any medical record and completeness.
CONCLUSION: Despite the importance of medical records for health system functioning, many rural facilities have yet to implement systems for maintaining patient records, and records are often incomplete when they exist. Prescription review tied to performance evaluation should be implemented with caution as it may create disincentives for record keeping. Interventions to improve record keeping and management are needed. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Medical records are an important tool for clinical care delivery, quality assessment and improvement.1–3 Properly maintained documentation of clinical encounters facilitates better continuation of care across time and providers, provides objective evidence through which to evaluate and monitor clinical practice, and serves as the basis for healthcare payment and reimbursement systems.4 5 Medical records are also used for public health surveillance, providing data on notifıable diseases as well as trends in other diseases such as influenza and diabetes.6 Medical records therefore play a fundamental role in modern healthcare delivery, clinical decision-making and public health systems.7–9

This study aims to evaluate the quality of medical records in a representative sample of rural health facilities in China. Improving medical record keeping is a priority in China given current health system reforms that seek to strengthen primary care and promote better integration across tiers of the health system.10 Moreover, providers in China have begun to rapidly adopt greater use of data from medical records in clinical practice and medical research.11 Despite the importance of medical records to the goals of healthcare reform and the increased use of these data in medical decisions, however, rigorous evidence on the quality of medical records is lacking.

This study contributes to the broader literature on medical records by using an approach employing unannounced visits by standardised patients (SPs) in a representative survey of health facilities in a low-resource setting. SPs are actors recruited from local communities and extensively trained to present standardised disease cases to providers. SPs enable detailed assessment of medical record quality by providing a full account of provider–patient interactions that can then be compared with medical records. This offers distinct advantages over other approaches that rely on alternative records12 or patient recall13–15 for comparison. SPs have previously been used to evaluate medical records in veterans administration facilities in the USA,16 17 but to our knowledge has not been previously used in representative facility surveys in either high-resource or low-resource settings.

Methods to assess medical record quality in low-resource settings are in need as developing countries such as China seek to strengthen medical record systems. In China, recent reforms have established standardised guidelines to improve the quality of medical records.18–20 In 2013, China’s National Health Commission mandated that all hospitals establish management systems for supervising the quality of medical records.20 This study evaluates whether rural hospitals have met the requirements set by these policies.

Methods

Study design and setting

Our study was conducted in rural areas in three provinces located in Western, Central and Eastern China. The facilities included in the study were selected from one prefecture (the administrative level below the province and above the county) in each of the three provinces. The prefectures included in the study from each province were chosen for having a predominantly rural population in consultation with local authorities.

We focus on township health centres (THCs), the middle tier of the three-tier rural health system between village clinics and county hospitals. THCs are playing an increasing role in the provision of primary healthcare in rural China, as reforms have sought to alleviate overcrowding in county hospitals by encouraging patients to first visit lower-tier providers and many areas face shortages of village providers.10 21 22

Our sample was chosen to be representative of rural facilities in the three prefectures. We used the following procedure to randomly sample THCs: first, across the three prefectures, we randomly chose 21 of 24 total rural counties. Straight randomisation was then used to select 10 THCs from each county. Because even counties designated as ‘rural’ have an urban township housing the county seat, we excluded the health centre of the urban township. One county had only nine rural townships, yielding a sample of 209 of the total 311 THCs in the 21 sample counties.

Recent reforms in China have established standardised guidelines to improve the quality of medical records.18–20 In 2010, the Ministry of Health published a list of items required to be included in medical records.19 In 2013, the National Health Commission mandated that all hospitals establish management systems for supervising the quality of medical records.20 These reforms have been established in conjunction with wider reforms of the health system in China, which have in part sought to strengthen primary care and promote better integration across health system tiers.10 Accurate and complete medical records are critical to both of these objectives, yet the the current quality of medical records in China has been largely unexplored. This study therefore aims to address this gap in knowledge by providing an objective assessment of medical records in rural primary care facilities, the first point of contact with the health system for most rural patients.

Data collection

We conducted three separate waves of data collection in sampled facilities (figure 1). An initial facility and provider survey was conducted for township-level providers in June 2015. At this time, we obtained verbal consent from providers for visits from unannounced SPs and audio-recorded interactions using concealed recording devices. SP visits then took place approximately 5 weeks following the initial facility survey, in August 2015. During interactions with providers, SPs presented one of three disease cases: unstable angina, pulmonary tuberculosis (TB) or a child with viral gastroenteritis (presented to the provider by a parent). SPs were extensively trained to present these disease cases consistently and covertly to providers. After each visit, audio recordings from concealed recording devices were transcribed into text by the trained enumerators and corresponding SPs. The Methodological appendix section in the online supplementary file provides further information on the development of disease cases and the implementation of SP visits, and details are also available in our previous study evaluating the quality of care provided to SPs.23 24

Figure 1

STROBE flow chart. SPs were randomly assigned to facilities, and within each facility SPs visited the provider following the normal procedures for any walk-in patient. Given a choice of which provider to visit, SPs randomly chose a provider following a predetermined randomisation protocol. After the SP visits, all the medical records produced in the interaction between providers and SPs were collected by our team. Our results, therefore, are designed to assess the quality of medical records as they are produced in real practice among our sample providers. SP, standardised patient; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

Finally, we conducted a follow-up visit to clinics in early September 2015 where we collected all available medical records from the clinics in our sample. We recorded whether the medical record for each SP visit was located by our team and, if located, transcribed all information containing individual patient information, symptoms that SPs presented to providers, medical history, diagnosis and treatment. Enumerators were instructed to transcribe content from medical records word-for-word into the survey form.

Our research design is unique in that it allows us to evaluate the actual quality of medical records across a variety of providers. This provides two distinct advantages. First, we are able to assess the accuracy and completeness of medical records because we can compare them with a real-time account of the clinical interaction. Other methods relying on patient exit interviews, for example, assume that patients have complete and accurate recall.25 Second, because SPs presented disease cases in a standardised way, we are able to analyse how the quality of medical records is related to the characteristics of providers and facilities. This is not possible with methods relying on data from real patients as comparisons can be complicated by differences across providers in disease case and patient mix.26

Evaluation standards for medical records

To evaluate the medical records, we used as a framework a list of 21 items that should be recorded based on national standards.18 These are organised into six categories, including basic patient information, chief complaints, history of present illness, medical history, diagnosis and treatment (table 1). For each of the 21 items, we assessed completeness and accuracy. Completeness was evaluated by assessing whether information for each item that was mentioned during a given clinical interaction was recorded in medical records, without regard to whether the recorded information matched what was said. Accuracy was then evaluated by whether the recorded information matched that from audio recordings of clinical interactions exactly.

Table 1

Framework for evaluating medical records derived from 2010 national standards*

Sections	Items (21 in total)
1. Patient information	Patient’s name, gender, date of birth, nationality, marital status, occupation, work unit, address and drug allergy history.
2. Chief complaint	Main symptoms and duration.
3. History of present illness	Onset time.
	Main symptoms and accompanying symptoms.
	Negative signs for differential diagnosis.
	Treatment situation during prior visits to other hospitals.
4. Medical history	Personal, past or family history related to the disease.
5. Diagnosis	Normative diagnosis name.
6. Treatment	Treatment suggestions.
	Drug treatment, including name, usage, dosage and time.
	Referral or follow-up.
	Cautions.

*Standards taken from the Ministry of Health requirements. Retrieved from: http://www.nhc.gov.cn/yzygj/s3585u/201002/0517a82e35224ee0912a5d855a9d249f.shtml, 2017.

Statistical methods

We calculated the mean or proportion and 95% CI across all SP interactions for each of our primary outcomes: the existence of a record and indicators of completeness and accuracy. To assess correlates of record keeping, we used logistic regression for existence (a binary indicator of whether any medical records were kept), and ordinary least squares regressions for completeness (the number of completed items in each medical record) and accuracy (the number of accurate items in each medical record). For each outcome, we assess correlations with a fixed set of facility-level and provider-level characteristics hypothesised to be related to completeness and accuracy. A detailed description of each of these covariates can be found in online supplementary appendix table 1. All regressions include dummy variables for each county in the sample to hold constant all county-level factors. In all regressions, we cluster SEs at the facility level. Analysis was done using Stata V.14.0.

Results

THC and provider characteristics

A total of 207 facilities were included in our study after excluding two facilities where providers were absent at the time of SP visits. On average, the general population in the catchment area for each facility was 26 125 (IQR: 12 174–32 000). Each facility received an average of 18 706 (IQR: 8000–22 978) visits during the 2014 calendar year. The average number of full-time providers in each facility was 7.7 (IQR: 4–9). More than half of the facilities had a system for reviewing prescriptions (often done by staff in the facility pharmacy), and 70.2% (145 out of 207) included resulting measures of prescription quality as part of provider work evaluations. Within the sample of 207 facilities, a total of 383 providers were visited by SPs. Online supplementary appendix table 2 describes the characteristics of participating facilities and providers.

Existence of any medical record

Of a total of 610 SP visits, 33.8% (210 out of 610) had any associated medical record. This was despite 84.6% of facilities (176 out of 207) reporting that they had protocols for keeping patient records. At the THC level, no medical record was recovered (across all SP interactions in the facility) in 34.8% (72 out of 207) of THCs. Comparing provider and facility characteristics for interactions where records were recovered and where they were not (online supplementary appendix table 3), we find that THCs with more patient visits and more experienced medical providers were more likely to have medical records.

We also find that the quality of medical record keeping varied across the three sample prefectures (online supplementary appendix table 4). Existence (% with any record), completeness (the average number of completed items in each medical record) and accuracy (the average number of accurate items in each medical record), although universally low, were all statistically different across the prefectures. The lowest quality was found in the prefecture located in Shaanxi Province (Central China), while the highest was found in the prefecture in Sichuan Province (Western China).

Further analysing correlates in a multiple regression (figure 2 and online supplementary appendix table 5, column 1) shows that, when controlling for other observed characteristics, providers’ income was positively associated with existence of a medical record (β=0.05, 95% CI 0.01 to 0.09). Our regression analysis also shows significant variation in keeping of a medical record by disease type. Compared with angina, there was a 12 percentage point (β=0.12, 95% CI 0.05 to 0.19) better chance of there being a record when the SP presented a case of TB and a 10 percentage point lower probability when the disease presented was diarrhoea.

Figure 2

Correlates of the existence of medical records. ​The figure reports the average marginal effects and corresponding 95% CIs from a multivariate logistic regression predicting the existence of a medical record. In addition to the characteristics shown, the regression also controls for county fixed effects. Detailed regression results are shown in column 1 of online supplementary appendix table 5. TB, tuberculosis; THC, township health centre.

Completeness of medical records

Figure 3 shows the rates of completeness for all 21 individual items for interactions for which a record was located. Online supplementary appendix table 6 shows the rates of completeness conditional on relevant information being mentioned in the interaction based on audio recordings.

Figure 3

The completeness and accuracy among 210 existing medical records.

Providers frequently recorded patients’ basic information (name, gender, date of birth and address), but recorded information on patient symptoms and non-drug treatments less often. Patients’ main symptoms were mentioned in 100% of interactions (as per the standardised protocol for SPs), but were only recorded in 25.7% of located records. The duration of symptoms and relevant history were documented in less than 11% of interactions where it was mentioned and a record was located. Non-drug treatments (such as lifestyle suggestions, referrals or instructions for follow-up) were discussed in approximately three of four visits where records were located, but only recorded in 13.3%.

Conditional on there being a record, diagnoses and treatments were recorded more frequently than other types of information. Diagnoses were recorded in 57.6% of located records, and drug-related treatment information was recorded 100% of the time.

Multivariate regressions using the sample for which a record was located show that, of the facility and provider characteristics included, only whether a facility has a prescription review system in place or not is significantly correlated with medical record completeness (online supplementary appendix table 5, column 2). Holding all other included characteristics constant, including the number of items (of the 21 total) mentioned in the interaction, having a prescription review was associated with recording 0.87 fewer items (β=−0.87, 95% CI −1.68 to 0.06). This represents a reduction of 19% compared with the sample average of 4.5 items recorded. Regression analysis also shows that completeness also varies by disease presented conditional on keeping of a record. Significantly fewer items mentioned in the interaction were recorded for patients with angina compared with patients with diarrhoea and TB.

Accuracy of medical records

Figure 3 also shows the rates of accuracy for all 21 individual items for interactions for which a record was located. Online supplementary appendix table 6 shows the rates of accuracy conditional on relevant information being recorded. When information was recorded, accuracy was relatively high. Five items were recorded with full accuracy: negative signs for differential diagnosis, medical history, non-drug treatment suggestions, drug treatment and referrals. The accuracy rates for patients’ main symptoms and their duration were 81.7% and 77.6%, respectively. For items relating to the history of the present illness, the onset time of disease was less accurate (70.4%) than the main symptoms and accompanying symptoms (88.7%). The accuracy rate for the diagnosis given was 83.6%. As shown in figure 2, among the 210 existing medical records, almost all items were accurate when they were recorded, but 7.1% of providers recorded the diagnosis inaccurately, which was the largest inaccuracy rate of the items assessed.

Due to the fact that items, when recorded, tended to be recorded accurately, multivariate regressions predicting accurately recorded items were not significantly associated with any of the facility or provider characteristics examined once the number of complete items in the medical record was controlled for (online supplementary appendix table 5).

Discussion

This study used an approach employing SPs to rigorously evaluate the quality of medical records in China. We present three main findings. First, providers of THCs kept medical records in only 33.8% of visits. This is far lower than previous studies using other methods in large urban hospitals in China, which have found recording rates around 90%.27 Second, when kept, basic patient information and information on diagnoses and treatment were more often recorded than patients’ symptoms and history. This is consistent with previous studies in high-income and low-income countries.16 17 28–30 Third, we found that the quality of medical records is correlated with the characteristics of providers and facilities. Specifically, we find that the existence of facility prescription review protocols was negatively correlated with completeness.

Our results have a number of implications for policy and research. Most clearly, medical records in primary care facilities currently cannot be relied on for management of patient care or for public health reporting. Improving medical records may be vital to the success of China’s reforms focused on strengthening the primary care system. Our results also imply that medical records are not a reliable source of data for evaluating the quality of care. This echoes results from the USA suggesting the use of alternative tools, such as clinical vignettes, provides a better indication of clinician quality.31 32 Moreover, although this study did not aim to evaluate causal factors that influence medical record quality, our results suggest an important role of institutional incentives facing providers to accurately and completely record information from medical encounters. Clearly, there may be strong disincentives generally due to potential liability. Unless required by enforced regulations, providers may not wish to risk recording information that could later be used to show or imply negligence. Our finding that medical records were less complete and accurate when facilities had in place a mechanism for reviewing prescriptions is consistent with this. A prescription review system increases the liability of providers who may respond by not recording information necessary to evaluate prescriptions. Moreover, that records were most complete for diagnoses and drug-related treatment information suggests that incentives to record may be tied more to billing and drug sales than to other uses of medical records. More research is needed to identify the causal effects of these factors on medical record quality.

Limitations

Even though the SP approach provides objective and unbiased information on medical record quality, this study does face a number of important limitations. First, SPs were trained to present only three disease cases. We find that the existence of a medical record and completeness of the record varied across the diseases presented by the SPs. There are a number of potential explanations for this variability, such as the perceived severity of the condition or the potential for drug sales. Future research could explore how record keeping varies across a wider range of diseases to understand whether variation in this dimension has meaningful implications for practice or policy.

Second, although SPs underwent intensive training to standardise their presentation of the disease cases, there may nevertheless remain differences across SPs. This concern is mitigated, however, as regression models including and excluding fixed effects for individual SPs yield similar coefficients on variables of interest. Moreover, when SP fixed effects are included in regressions, they explain only a small fraction of the overall variation in the quality of medical records (5.1% for recording, 7.1% for completeness and 4.9% for accuracy).

Third, the aim of this study was to assess the quality of medical records in THCs (the middle tier of the rural health system). Given documented differences in quality across the three tiers of the rural health system,23 future studies could use a similar approach to evaluate medical record keeping in other facility types.

A final limitation is that we only evaluated the existence of medical records approximately 3 weeks after patient visits. Ten providers in our sample claimed to have written medical records that could not be located because of the time that had elapsed. More records may have been located if follow-up visits were conducted sooner after SP visits.

Conclusion

There are significant deficits in the quality of medical records in township hospitals in rural China. Despite recent policies to improve medical record keeping, many rural facilities have yet to implement systems for maintaining patient records, and existing records are often incomplete. We find suggestive evidence of disincentives for providers to completely record information from clinical interactions. Addressing these disincentives may be a key target for improving medical record keeping in China, where complete and accurate records may be vital to broader health system reforms. Given the importance of records for health system management, further research into interventions to improve the quality of medical records are needed.