Establishment of a Comprehensive Evaluation System on Medical Quality Based on Cross-examination of Departments within a Hospital.

INTRODUCTION

The management of medical quality is the core element of hospital management, and it is reflected in every part and aspect of medical behavior. Strengthening the management on medical quality is crucial in ensuring medical safety, promoting the ability of clinical diagnoses and treatments as well as the increasing the efficiency of medical service.[1] On this basis, the establishment and operation of a comprehensive evaluation system on medical quality is not only the foundation in ensuring the orderly operation of the medical behavior and medical safety but also an effective means in continuously improving the quality of medical management.[2] This paper aims to discuss the establishment of an operable and objective evaluation system as well as the corresponding assessment indicators on internal medical quality within the hospital based on cross-examination of departments through extensive research into various national and regional medical quality evaluation systems, and practice was carried out to achieve the purpose of the continuous improvement in medical quality as well as ensuring medical safety.

SEVERAL MAJOR MEDICAL QUALITY EVALUATION SYSTEMS WORLDWIDE

Performance assessment tool for quality improvement in hospital of WHO

In 2003, the European WHO office proposed the performance assessment tool for quality improvement in the hospital (PATH) to help medical institutions evaluating their performance on medical service, and the results of the evaluation can be adopted in the continuous improvement of medical quality.[3] The evaluation framework of PATH consists of 6 dimensions within 4 areas (clinical effectiveness, efficiency, the adaptability of medical workers, and response management), as well as 2 transverse aspects (medical safety and the consciousness of patient-centered). According to the applicability as well as the cost in data collection, PATH divides the evaluation indicators into two categories as follows: core indicators and optional indicators.[4] Core indicators include the use of antibiotic drugs, the mortality of major diseases and surgeries, the re-admission rate of major diseases and surgeries, the monitoring indicators of day surgeries and so on. Optional indicators include the mortality of major diseases and surgeries through risk adjustment, the re-admission rate of major diseases and surgeries through risk adjustment, the occurrence of pressure ulcer within stroke and fracture patients, as well as hospital-acquired infection rate and so on.[5]

Joint commission on accreditation of healthcare organizations

The concept of the joint commission on accreditation of healthcare organizations (JCI) was first proposed by the joint commission on accreditation of international healthcare institutions, and all the evaluation standards of JCI are designed through the perspectives of patients.[6] JCI evaluation altogether covers 368 indicators (including 200 core indicators and 168 peripheral indicators). Each indicator contains several assessment items that sum up 1033 in total, all of which target to the most important sectors and aspects of medical treatment and nursery care in hospital management.[7] Notably, the objective of JCI management is to provide patients with the medical services that meet their various needs and increase the utilization of medical resources to promote life quality. Moreover, the core value of the evaluation is the continuous improvement of medical quality based on the patient-centered concept.[8]

International quality indicator project

International quality indicator project was formulated by the American center for performance research specialized in evaluating the clinical quality and safety of medical institutions.[9] It is the most widely used hospital medical quality evaluation system that specifically focusing on the results of medical service worldwide. There are altogether 30 categories of indicators focusing on the medical quality of general hospitals among the system, covering all aspects of medical care for outpatients and inpatients. For example, the inpatient mortality indicators include inpatient mortality, neonatal mortality, perioperative mortality and so on. Unplanned return indicators include the rate of unplanned readmission, the rate of unplanned readmission to Intensive Care Unit (ICU), the rate of unplanned readmission to the surgical operation, and so on. Hospital infectious indicators include ICU infection rate, surgery-related infection rate, methicillin-resistant Staphylococcus aureus infection rate, multiple drug-resistant bacteria infection rate and so on. Patients’ safety indicators include the complications of falls, sedation and pain relieving, the incidence of pressure ulcers, the incidence of postoperative deep vein thrombosis and pulmonary embolisms, the incidence of outpatient falls, etc. Besides, the management of the ICU medical equipment usage, the use of antimicrobial agents in surgical procedures, the management of parturition process, the management of postoperative thrombosis preventions is also included in the evaluation system.[10]

Clinical Indicator Program

Clinical Indicator Program (CIP) was the unified national clinical service quality index system that was first developed and adopted in 1989 by the Australian Council on healthcare standards.[11] CIP was used in the evaluation of inpatients, outpatients and community health-care services, with a regular update of the system to ensure that the indicators are suitable and operable in clinical evaluation, and that are conducive in reflecting the problems of clinical procedures as well as improving the quality of medical service.[12] The classification of CIP system is based on the attribution of clinical departments, and hence that clinical indicators as well as indicators of medical auxiliary departments (such as radiology and pathology department) are all included in the system. Currently, 22 indicators in 353 clinical areas are included in CIP system.[13] Among them, procedure-related indicators include records-related indicators, time-related indicators as well as preventive indicators. While results-related indicators included readmission indicators, mortality indicators, hospital infectious indicators, patients’ safety indicators, efficiency-related indicators, and so on.

Hospital rating system of the United Kingdom

Hospital rating system of the United Kingdom was first introduced by UK department of health in 2001, and it had been playing an important role in promoting the quality of medical service in the UK ever since.[14] The rating system mainly assesses two sectors of hospital services, which are the management of medical quality and service efficiency in spite of the scale sizes and technical levels of the hospitals. The rating system mainly includes four categories of indexes: medical services, clinical errors, patients’ satisfaction, and staff performances. In specific, all the 21 evaluation indicators are divided into four categories: key indicators of medical service, indicators in related to patients, indicators in related to clinical procedures as well as indicators in related to clinical capacities and capabilities.[15] Among them, 9 evaluation indicators are defined as key indicators. When a hospital reaches all the standards of key indicators, it will be rated as three-star hospital; two-star will be rated if the hospital fails 1 or 2 standards. When a hospital fails 3 or more key indicators, it will be rated as one-star hospital or unqualified hospital.[16]

Hospital evaluation system of Japan

In 1995, Japan established the third party evaluation organization of medical institutions, and officially started the evaluation project in 1997. Differ from other assessment systems, the first procedure of the evaluation is document review. In this process, a fundamental understanding of the basic situation of the hospital is acquired through 5 types of the questionnaire by the evaluation organization, including questionnaire of the hospital, questionnaire of the headquarter departments, questionnaire of financial management, questionnaire of diagnostic ability, and questionnaire of discharged patients. Then, the hospital surveyed should propose the existing problems and potential solutions for the organization to assess the cognition of the existing problems. The final course of the evaluation process is the interview of the surveyed hospital. The interview is conducted by a group of expertise from the evaluation organization based on the review criteria, raw data and self-assessment results. Moreover, the contents of the interview include the tenets and organizational structure of the hospital; the satisfaction degree of local residents toward the hospital; the medical quality of diagnoses and treatments; the suitability and effectiveness of nursery services as well as the rationality of hospital management. A review meeting will be held by the evaluation group after the interview to discuss the results of the evaluation, and an official report will be formed and fedback to the hospital for continuous improvement.

THE CHARACTERISTICS AND LIMITATIONS OF MEDICAL QUALITY EVALUATION SYSTEM IN CHINA

The current evaluation system and key indicators of medical quality in China

The evaluation system that most widely applied in assessing the medical quality in China is the Chinese medical quality indicator system (CHQIS), and the evaluation indicators were generated according to the standards of tertiary general hospitals released by ministry of health in 2011, which emphasizes the public welfare of state-owned hospitals in the contents and indicators of the evaluation. CHQIS has generated the experiences of hospital evaluation of China in the past in combine with the evaluation standards of other countries and regions, such as JCI, the hospital evaluation systems of United States, Europe, and Japan. Meanwhile, the concept of plan-do-check-act was widely applied in the formulation of standards in fields of clinical management, nursery management, medical technology management, hospital infectious management, and so on. Moreover, the key monitoring indicators include: basic hospital operation indicators, medical quality and safety indicators of inpatients, management indicators of single diagnostic diseases, medical quality management indicators of ICU, indicators of rational drug application, indicators of hospital infection, etc.

Limitations of the medical quality evaluation system in China

The current medical quality evaluation system of China covers various aspects of medical management of public hospitals, such as medical services, patients’ safety, the continuous improvement of medical quality and safety, nursery management, routine statistics, and so on. However, compared with other countries, we found that there are still limitations in the evaluation system. First, according to the approach of the evaluation, the indicators are divided into process-oriented indicators and results-oriented indicators. Most indicators included in the evaluation system are results-oriented indicators, and process-oriented indicators are relatively rare. However, extensive experiences showed that process-oriented indicators are more related to the quality of clinical services and that are able to reflect the outcome of clinical services more explicitly. Thus, it is imperative for medical evaluation system to adopt more process-oriented indicators in the assessment. In regards to the content of the current evaluation system, indicators concerning to work efficiency and quantity are far more than indicators concerning to quality and safety in medical management compare with evaluation system of other countries and areas. Besides, the evaluation cycle of the system is relatively long, thus it is not operable for the assessment and monitoring of daily medical quality within the hospital. Therefore, it is imperative for medical administration department of medical institutions to develop an evaluation system with specific assessment methods and indicators that are operable and effective for hospitals to carry out the internal evaluation.

THE ESTABLISHMENT OF A COMPREHENSIVE EVALUATION SYSTEM OF MEDICAL QUALITY BASED ON CROSS-EXAMINATION OF DEPARTMENTS WITHIN THE HOSPITAL

Therefore, to carry out medical quality assessment more comprehensively and practically within medical institutions, in the light of the existing evaluation system and key indicators, in combination with the characteristics of the hospital, an internal medical quality evaluation system based on cross-examination department was developed and had already been put into implementation in a grade-A tertiary hospital in Beijing for a year. According to the result of the assessments, significant improvement in the medical quality of the hospital has been going on through the process.

The framework and indicators of the evaluation system

The cross-examination evaluation system integrates the key indicators and procedures of medical quality as well as medical safety. Furthermore, it is suitable for management departments of the hospital to carry out statistics, analysis, and inspections with strong operability. The evaluation framework altogether consists of three parts: routine indicator evaluation, on-site evaluation, and survey on patients’ satisfaction. The full score of the evaluation system is 100, the proportion for the three parts are 40%, 40%, and 20%. The results of evaluation are directly linked to the bonus salary of department staffs in each quarter.

Routine indicator evaluation

Most of the routine indicators are result-oriented, the evaluation is conducted mainly by medical statistics and management departments at the end of each quarter of the year and the scores are marked according to the performance of clinical departments. The indicators of routine evaluation mainly include indicators of in-patient and out-patient medical quality, indicators of nursery management as well as hospital infectious management, and the specifics are shown in Table 1.

Table 1

Routine evaluation indicators of clinical departments

Categories	Contents	Evaluation indicators
Outpatients management	Management of outpatient services	Punctuality of outpatient services
		Suspension of outpatient services
	Management of medical defects	Complaints and disputes of outpatient services
Inpatient management	DRGs management	CMI index of discharged patients
		Mortality rate of low risk groups
		Time efficiency index
		Cost efficiency index
	Management of diagnoses and treatments	Rational use of antibiotics
		Management of clinical pathway
		Punctuality of medical records documentation
		Punctuality of group consultation
		Punctuality of surgical operations
		Management of adverse medical events
	Implementation of key regulations	Management of doctors on duty
		Management of death cases discussions
		Management of intractable cases discussions
		Management of the continuous improvement
		Management of medical records
Management of hospital infection	Quality of hospital infection management	Hospital infectious rate
		Rate of missing report on hospital infection
Management of nursery quality	Quality of nursery management	Rate of qualified specialized nursery
		Rate of intactness of first-aid supplies
		Nursery defects
		Incidence rate of pressure sores

DRGs: Diagnosis-related groups; CMI: Case mix index.

On-site evaluation

Differ from routine evaluation, most on-site evaluation indicators are process-oriented. The hospital established the panel of evaluation that consisted of experts within different major areas, and the experts within the panel are distributed into different department for cross-evaluation. The major contents of on-site evaluation include the management of medical records quality, the management of surgical operation safety, the implementation of key regulations on medical quality, and so on. The evaluation of medical auxiliary departments such as pharmacy, laboratory, and pathology departments are performed by experts from the quality control centers of Beijing in specialized major assessments to ensure the objectivity and impartiality of the evaluation. The design of specific indicators of on-site evaluation is shown in Table 2.

Table 2

On-site evaluation indicators of clinical departments

Categories	Contents	Evaluation indicators
Outpatients management inpatient management	Analyses on medical quality	Implementation of first visit responsibility
		Quality of medical records management
		Assessments on diagnosis and treatment effects
		Quality of inspection application form
	Rational use of outpatient medications	Prescription quality
	Management of critical values	Reporting and handling of critical values
	Management of inpatient medical records quality	Terminal quality control of medical records
		Procedural quality control of medical records
	Management of surgical operation quality	Management of operation safety checks
		Management of operative site marker
		Management of operational records
	Rational use of inpatient medications	Rational use of special management pharmaceuticals
		Assessment on prescriptions
		Assessment on inpatient cases of high medical expenditures
	Other indicators	Quality of ward rounds for attending doctors
Management of hospital infection	Management of hospital infection and implementation of disinfection and isolation	Management of routine monitoring
		Management of ultraviolet disinfection
		Management of disinfectant concentration
		Management of medical wastes
		Implementation of isolation measures
	Management of hand sanitation	Compliance and accuracy of hand sanitation
		Allocation and dosage of hand sanitation
	Management of medical wastes	Weight of medical wastes
Management of nursery quality	Management of nursery quality	Management of specialized nursery quality
		Intactness of first-aid supplies

Survey on patients’ satisfaction

The survey on patients’ satisfaction was regularly conducted by the follow-up center of the hospital toward discharged patients. The contents of the survey include the patients’ satisfaction degree and suggestions toward the management of medical service, nursery service, wards environment as well as the management on auxiliary examination, and grades are scored according to the results of the survey.

THE PRACTICE OF MEDICAL QUALITY EVALUATION BASED ON THE CROSS-EXAMINATION SYSTEM OF THE HOSPITAL

The general situation and results of medical quality evaluation of 2016 of the hospital

To examine the effect of the cross-evaluation system toward the improvement in medical quality, a practice was operated in a Grade A tertiary hospital of Beijing during 2016. The evaluation was carried out among 41 clinical departments within the hospital by quarters of the year, and the results of evaluation were fed back to the department for continuous improvement of medical quality and safety. The results of evaluation show that the overall average scores among the hospital are improving through the repeatedly evaluation, as it is shown in Table 3. While the general fractional distribution indicated the average scores of auxiliary department are the highest, followed by nonsurgical departments, and the average score of surgical departments are the lowest among the three departmental categories. As we analyze into the possible reasons to the variance within different departments, there are multiple factors that might be linking to the results. First, the evaluation indicators of medical auxiliary departments are relatively specialized compared with indicators of clinical departments and that the practitioners are more acquainted with so that they are inclined to perform better in the assessments. Second, the workloads of medical auxiliary departments are relatively less than clinical departments, so that the staffs of these departments are able to focus more on the management of medical quality. Third, since the evaluation experts of medical auxiliary department are chosen from quality control centers of Beijing instead of the panel within the hospital, the understanding of evaluation criteria may varied greatly among different individual and resulted to the deviation in grading. These facts may all lead to the general situation of the general fractional distribution mentioned above.

Table 3

Overall evaluation results of a Grade A tertiary hospital in 2016

Period	Average score of surgical departments	Average score of nonsurgical departments	Average score of medical auxiliary departments	Average score of all departments
The second quarter of 2016	93.91	95.94	96.81	95.30
The third quarter of 2016	94.26	96.11	97.13	95.86
The fourth quarter of 2016	95.56	96.30	97.47	96.32
The first quarter of 2017	94.81	96.76	98.07	96.44

The achievement obtained through the cross-examination evaluation

Through a year of comprehensive cross-examination evaluation between different departments, the medical quality of the hospital was greatly improved within the sampled hospital. As statistics showed that the number of discharged patients increased by 14.4% compared with same period last year; the average length of hospital stay decreased by 4.7% compared within last year; the rate of bed utilization increased by 2.6%; and the amount of the total surgical operation increased by 18.8%; the rate of Grade A medical records increased by 0.5%; the intensity and rate of antibiotic use decreased by 1.8% and 4.5%, respectively. On the contrary, the incidence of adverse medical events and medical tangles decreased significantly. All the key indicators of medical quality and safety showed a significant improving trend, as it is shown in Table 4.

Table 4

General statistics of medical quality before and after the evaluation within the hospital

Items	The first quarter of 2016	The first quarter of 2017	Variance ratio (%)
Number of discharged patients	10,476	9159	14.4
Average length of hospital stay (d)	8.9	9.3	−4.7
Rate of bed utilization (%)	91.5	88.9	2.6
Amount of the total surgical operation	5718	4832	18.3
Rate of Grade A medical records (%)	98.9	99.4	0.5
Intensity of antibiotic use (%)	34.2	33.6	−1.8
Rate of antibiotic use (%)	44.7	42.7	−4.5
Incidence of adverse medical events (%)	21	19	−9.5

The characteristics, shortcomings of the system as well as the improvement plan in the next stage of evaluation

Compared with the existing evaluation system of medical quality that officially formulated, the cross-examination evaluation of medical quality system shares the advantages of aptness in operation, objectiveness in assessment, as well as the focus on both the process and results of every sector in medical management within the hospital. Medical administrative departments are able to identify the problem existed in clinical departments through the evaluation and continuously improvement can be made afterwards. In addition, since the results of the evaluation are directly linked with the quarterly salary bonus, the incentive effects are obvious for medical staffs. However, problems and shortcomings of the evaluation system were also exposed after a year of practice. As we mentioned above, some of the indicators failed to reflect the characteristics and specialties of different departments, and deviation were seen in indicator weight setting and the process of scoring due to the lack of homogenization in evaluation criteria of different experts. Therefore, constant revision of standards is also a crucial part for the continuous improvement management. Thus, in the next stage further refinement of the evaluation standards with specialty characteristics, as well as the increase in the index weight and contents of specialty evaluation (such as endoscopy center and hemodialysis room) to better reflect the medical quality of specialty characteristics will be add to the evaluation system for continuous improvement.

Limitations and the generalization of the research

According to the research, significant improvement in medical quality was going on in the hospital through the process, which could be provided as valuable experiences for other medical institutions to follow. However, since the theoretical research and practical operation was based on a single centered tertiary hospital, the generalization should be more carefully in other subordinate and specialized medical institutions in the setting of evaluation index and weight of scores based on local situation.

In conclusion, through the construction of the comprehensive evaluation system on medical quality based on cross-examination of departments within the hospital along with a year of practice, the medical quality of the hospital was greatly improved. However, problems and shortcoming still exist in the current evaluation system and indicators which call for the continuous improvement in the next Plan-Do-Check-Act (PDCA) round.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.