Reliability and Validity of the Appropriateness Evaluation Protocol for Public Hospitals in Korea.

OBJECTIVES: This study was conducted to assess the applicability of the Appropriateness Evaluation Protocol (AEP) for public hospitals in Korea.
METHODS: In May 2016, 1500 admission claims were collected from Korean public district hospitals using stratified random sampling. Of these claims, 560 admissions to 37 hospitals were retrieved for analysis. Medical records administrators determined the appropriateness of admission using the criteria detailed in the AEP, and a physician separately assessed the appropriateness of admission based on her clinical judgment. To examine the applicability of the AEP, the concordance of the decisions made between a pair of AEP reviewers and between an AEP reviewer and a physician reviewer was compared.
RESULTS: The results showed an almost perfect inter-rater agreement between the AEP reviewers and a moderate agreement between the AEP reviewers and the physician. The sensitivity and specificity of the AEP were calculated as 0.86 and 0.56, respectively.
CONCLUSIONS: Our findings suggest that the AEP could potentially be applied to Korean public hospitals as a reliable and valid instrument for assessing the appropriateness of admissions.

INTRODUCTION

The appropriateness of hospital admission is a major healthcare issue around the world, as it is closely related to the quality of healthcare services and healthcare costs. However, the standards for determining the appropriateness of hospitalization vary depending on the perspectives of different stakeholders, and objective criteria to assess hospitalization appropriateness are difficult to develop [1,2]. To address this problem, many researchers have created evaluation tools such as the InterQual criteria, the Appropriateness Evaluation Protocol (AEP), and the Managed Care Appropriateness Protocol [3-5]. Of these, the AEP, which was developed by Gertman and Restuccia [4] in 1981, is the most widely used. This protocol is known for its ease of use and efficiency compared to similar assessments; a study found that the average time taken to assess an admission using the AEP was less than 10 minutes. The original AEP consists of 18 criteria across 2 sections. The medical necessity of admission is determined by assessing the condition of the patient and the clinical services, if any, that the patient requires. An admission must fulfill at least 1 of the 18 criteria to be identified as medically necessary [4]. The AEP has been validated in the USA and elsewhere [4,6-9], and it has been applied in many countries because of the high-level of objectivity and adaptability of its criteria [6,10-16].

In Mexico, the AEP criteria have been applied to evaluate the reliability and validity of the AEP regarding the appropriateness of admissions and hospital stays in elderly patients. In that study, inter-reviewer agreement had a kappa coefficient of >0.70. Although sensitivity and positive predictive value (used to detect inappropriate admissions) were not calculated, specificity and negative predictive value (used to detect appropriate admissions) were found to be >94.0% and >98.0%, respectively [17].

In the UK, the American version of the Paediatric AEP was reviewed by a panel of pediatricians and general practitioners. In that study, the agreement between the raters using the Paediatric AEP for admission criteria was excellent (κ=0.848), whereas there was poor agreement (κ=0.345) between clinicians using subjective judgment [18].

Validation of the AEP in previous studies was mostly conducted in teaching hospitals using a small sample (e.g., 1 or 2 teaching hospitals). In Asia, only a few countries, such as China, Malaysia, and Hong Kong, have applied the AEP to evaluate the appropriateness of hospital admissions [7,19,20]. Several studies have assessed the use of the AEP criteria to evaluate the appropriateness of hospitalization in Korea. However, those studies focused on specific clinical departments or patients with certain diseases [21,22].

The public sector accounts for fewer than 10% of Korea’s medical institutions. Recently, calls have been made to increase the proportion of public medical institutions due to the public’s demand for the national health insurance system to provide expanded coverage for their basic health needs and to the need for government intervention in the case of market failure. In this respect, it is necessary to review the quality and performance of Korean public healthcare services. The objective of this study was to assess the appropriateness of hospital admissions using the AEP and to validate the applicability of the AEP for public hospitals.

METHODS

Source of the Data Under Study

We utilized hospital admission records collected by the Health Insurance Review and Assessment Service (HIRA), an affiliate of the Ministry of Health and Welfare in Korea, in our attempt to validate the AEP. To monitor the performance of public district hospitals, HIRA collected the records of all patients admitted in May 2016 to a public district hospital (n=39) and stored them as scanned files. Public district hospitals are located thr-oughout the country to provide medical and surgical care to local communities. They also offer public health services that are not provided by private hospitals [23].

As part of their hospital performance monitoring project, HIRA randomly sampled 1500 admission records from 38 public district hospitals, as 1 hospital did not have any admission records corresponding to the study period. From HIRA’s sample of 1500 admission records, we used a random sampling method to select 624 cases for a retrospective chart review. Because the AEP was designed to assess the appropriateness of the medical and surgical admissions of adults, we excluded the pediatric and psychiatric admissions cases present in this randomly selected sample from our final data set. We also excluded admission records that contained invalid or incomplete information. The original set of 624 cases contained 36 records of pediatric admission, 15 records of psychiatric admission, and 13 records with invalid or incomplete data. The final sample consisted of 560 admission records from 37 public district hospitals; furthermore, all admission records from 1 hospital that only had 3 admissions during the study period were excluded from our final dataset.

Validation of the Appropriateness Evaluation Protocol

We attempted to validate the AEP by comparing the appropriateness of hospital admission determined by the AEP with the appropriateness determined by an experienced physician. The physician’s decision, in other words, served as the gold standard.

We trained 2 professional medical record administrators to apply the original US-AEP to review the 560 admission records selected from the HIRA dataset and determine whether admission was appropriate. If there was a discrepancy between the 2 medical record administrators, an experienced nurse would use the same AEP criteria to make the final determination. The physician whose judgment served as the gold standard was a faculty member in the Department of Family Medicine at a major tertiary teaching hospital and had more than 10 years of clinical experience. She reviewed all 560 admission records and determined whether each admission was appropriate based on her clinical judgment. All of the reviews were carried out concurrently and independently.

As a test of the reliability of the AEP, we examined the degree of agreement between the decisions on admission appropriateness made by the 2 medical record administrators. The degree of agreement was measured by calculating the overall agreement and the specific agreement. The overall agreement referred to the proportion of decisions in which the 2 reviewers agreed. The specific agreement was separated into 2 components: specific appropriate agreement (referring to the proportion of agreement in cases where admission is judged to be appropriate by at least 1 reviewer) and specific inappropriate agreement (referring to the proportion of agreement in cases where admission is determined to be inappropriate by at least 1 reviewer). To take into account the probability of chance agreement, Cohen’s kappa statistic was calculated for overall agreement. The kappa statistic was interpreted using the guidelines set forth by Landis and Koch [24], in which kappa coefficients between 0.00-0.20 are regarded as slight agreement, between 0.21-0.40 as fair agreement, between 0.41-0.60 as moderate agreement, between 0.61-0.80 as substantial agreement, and between 0.81-1.00 as almost perfect agreement.

The validity of the AEP was tested by comparing the final decisions made by the AEP reviewers with the physician’s judgment regarding the appropriateness of admission. The overall agreement and specific agreement were calculated to measure the level of concordance between the AEP and the gold standard. The level of overall agreement was presented using Cohen’s kappa values. In addition, we calculated sensitivity, specificity, and positive and negative predictive values.

All statistical analyses were performed using SAS version 9.3 (SAS institute Inc., Cary, NC, USA).

Ethics Statement

The present study protocol was reviewed and approved by the Institutional Review Board of the College of Medicine, The Catholic University of Korea (approval No. MC19EESI0014). Informed consent was obtained from all subjects at the time of their enrollment in the study.

RESULTS

Of the 560 admission records reviewed in the study, 424 (75.7%) represented patients who were admitted to medical wards, 128 (22.9%) to surgical wards, and 8 (1.4%) to other wards. In terms of hospital type, 503 patients (89.8%) were admitted to general hospitals (i.e., hospitals that have over 100 beds and more than 7 departments – surgery, medicine, pediatrics, radiology, anesthesiology, etc.), and the remaining 57 patients (10.2%) were admitted to small hospitals. The distribution of patients in the study sample by department (medical, surgical, or other) is shown in Table 1.

Table 1.

Distribution of study subjects across departments and proportion of appropriate and inappropriate admissions by department

Department	Total	Admission
		Appropriate	Inappropriate
Medical	424 (75.7)	337 (79.5)	87 (20.5)
Surgical	128 (22.9)	77 (60.2)	51 (39.8)
Other	8 (1.4)	8 (100)	0 (0.0)
Total	560 (100)	422 (75.4)	138 (24.6)

Values are presented as number (%).

Of the 560 admissions, 138 were assessed to be inappropriate using the AEP criteria (24.6%). The proportion of inappropriate admissions was 20.5% and 39.8% for medical wards and surgical wards, respectively (Table 1).

The level of overall agreement between the 2 AEP reviewers was 96.8%, and the agreement values obtained were 97.9% and 93.4% for appropriate and inappropriate admissions, respectively. The Cohen’s kappa value for overall agreement was 0.91 (95% confidence interval [CI], 0.87 to 0.95). According to the Landis and Koch [24] guidelines, the kappa statistics show-ed almost perfect agreement between the 2 AEP reviewers (Table 2).

Table 2.

Inter-rater agreement between two AEP reviewers

Reliability measure	All departments, %
Agreement
Overall	96.8
Appropriate admissions	97.9
Inappropriate admissions	93.4
Cohen’s kappa (95% CI) for overall agreement	0.91 (0.87, 0.95)[*]

AEP, Appropriateness Evaluation Protocol; CI, confidence interval.

p<0.05.

The level of overall agreement between the AEP reviewers and the physician was 78.6%. Of the 560 total admissions, 360 admissions were assessed to be appropriate and 80 admissions were assessed to be inappropriate by both the AEP and the physician reviewers. The agreement values for appropriate and inappropriate admissions were 85.7% and 57.1%, respectively. Of the 418 admissions that were judged to be appropriate by a physician reviewer, 58 admissions (13.9%) were judged to be inappropriate using the AEP. Of the 142 physician-judged inappropriate admissions, 62 admissions (43.7%) were judged to be appropriate using the AEP. The Cohen’s kappa coefficient for the level of overall agreement between the AEP reviewers and the physician was 0.43 (95% CI, 0.34 to 0.51). According to the Landis and Koch [24] guidelines, a kappa coefficient of 0.42 indicates a moderate agreement on the appropriateness of admissions between reviewers using AEP criteria and the judgment of an expert. The sensitivity and specificity of the AEP were found to be 0.86 and 0.56, respectively. The positive and negative predictive values were 0.85 and 0.58, respectively (Table 3).

Table 3.

Validity measures of the AEP on the appropriateness of hospital admissions when compared with the judgment of a physician (used as the gold standard)

Validity measure	Validity across all departments, %
Agreement
Overall	78.6
Appropriated	85.7
Inappropriate	57.1
Cohen’s kappa (95% CI) for overall agreement	0.43 (0.34, 0.51)[*]
Sensitivity	86.1
Specificity	56.3
Predictive value
Positive	85.3
Negative	58.0

AEP, Appropriateness Evaluation Protocol; CI, confidence interval.

p<0.05.

DISCUSSION

In this study, 24.6% of the admissions to public district hospitals in Korea under consideration were identified to be inappropriate based on the AEP criteria. Previous studies from Italy, Portugal, Iran, China, Denmark, and France reported that the rate of inappropriate admissions ranged from 8.8% to 28.1% [7,10,13,25-27]. In comparison, the rate of inappropriate hospital admissions in Korea is high, suggesting the importance of finding a valid and reliable tool to assess admission appropriateness. With such a tool, researchers can investigate the causes of inappropriate hospital admissions to help policymakers and healthcare leaders combat the problem.

This study showed the AEP to have high reliability and high validity. In particular, the level of overall agreement between the 2 AEP reviewers was found to be almost perfect (96.8%), with a kappa value of 0.91. These findings were similar to the values reported in studies conducted in Europe, China, Iran, Italy, and Hong Kong [6-8,10,20]. Regarding validity, a moderate level of agreement was found between the AEP reviewers and the physician whose clinical judgment served as a gold standard. The degree of overall agreement (78.6% with a kappa value of 0.43) obtained was lower than that found in studies conducted in Iran and China [6,7]. Esmaili et al. [6] tested the reliability and validity of the AEP for determining the appropriateness of hospital admissions in one of the largest Iranian teaching hospitals. The level of overall agreement between the AEP reviewers and the physician reviewers was 92.0%, with a kappa value of 0.80. Liu et al. [7] validated the AEP in China using hospital records from 2 tertiary hospitals. Of 350 randomly selected admissions, the level of overall agreement between the AEP reviewers and the physician reviewers was 90.9%, with a kappa value of 0.68.

The relatively low-level of agreement observed in this study between the AEP reviewers and the physician may be explained by several factors. First, using the original US-AEP may have led to a relatively low-level of agreement between the AEP reviewers and the physician reviewer, who was a Korean physician. As an example of this disconnect, healthcare providers in acute care hospitals in Korea have a tendency to give intravenous medication or fluids to all admitted patients at the time of their arrival. If following the AEP, reviewers may deem an admission appropriate because it meets the criterion of “intravenous medication or fluid replacement.” A physician reviewer, in contrast, may assess the appropriateness of admission based on the overall condition of the patient, rather than on the basis of that criterion alone. Second, the sample examined in this study contained a high proportion of elderly patients. Hospitals in Korea often admit older individuals not on the basis of medical necessity but instead due to the absence of a caregiver. This phenomenon is known as social admission [28]. In the present study, social admissions that are deemed inappropriate based on AEP criteria may have been considered appropriate by the physician reviewer. Finally, unlike other studies that employed AEP reviewers with clinical backgrounds, the 2 AEP reviewers in this study were medical records administrators who had been trained to follow the AEP protocol to assess the appropriateness of admissions. Although their assessments had close to perfect inter-rater reliability, their lack of medical knowledge may have limited their ability to assess the admissions in a medically comprehensive way, thus explaining the moderate level of agreement with the physician reviewer. To the extent that this explanation is correct, the findings of this study suggest that healthcare professionals with clinical knowledge are more suited for using the AEP than individuals without clinical backgrounds.

Restricted access to hospital records allowed us to obtain admission records from public hospitals for only 1 month of the year. Given the fact that the study period was limited to May 2016, we cannot guarantee that no seasonal variation exists in hospital admission data. However, we believe that this is not a major problem, considering that our study selected the most medical records available, thereby ensuring that the data included patients with an adequately wide range of conditions.

In addition to the limitations associated with the 1 month admission data window, another limitation of this study was that there was only 1 physician reviewer. Clinicians often decide whether patients should be hospitalized based on the present illness, the patient’s overall condition, and patient information and objective figures. Korea’s hospital system is less burdened by patients than other countries, and cultural differences may make it difficult to define the AEP as an admission standard.

In this study, some admission records were excluded after random sampling. Pediatric admission data were excluded from the study because pediatric cases have different hospital admission criteria than adult cases. Psychiatric admission data were also excluded because in those cases, the patient’s condition or the severity of the disease were not documented in detail. Finally, cases with different hospitalization periods or with invalid or incomplete charts were removed from the data set. In total, 560 cases were reviewed. This exclusion of certain data from the study may have compromised randomization.

Since the analyses were done solely on data obtained from public district hospitals, the absence of private hospital admission data means that these results may not be fully representative of Korean hospitals in general. Thus, further research is needed that includes data from hospitals from both the public and private sectors. However, there are many considerations associated with the establishment of a system for collecting medical records, such as the confidentiality of personal information and privacy issues.

Another limitation of this study is the absence of data on the healthcare costs and the physician workload associated with inappropriate admissions. It is widely believed that inappropriate hospitalization may have a negative impact on medical costs and the healthcare workforce. Several studies have been published on the economic cost of inappropriate admissions [26,29,30]. Menand et al. [27] conducted a multicenter study including 8 hospitals in France and showed that the median cost of a hospital stay was higher for inappropriate admissions (€4399.2) than for appropriate admissions (€3606.5). Mould-Quevedo et al. [29] also found that the cost of inappropriate hospitalization per patient (US$2323.3) was about 1.6 times higher than the cost of appropriate hospitalization (US$1497.2) among elderly patients (over 60 years old) admitted to a general hospital in Mexico City. Nevertheless, due to constraints on data availability, the cost of inappropriate admissions and associated factors were not explored in this study. Future studies should aim to collect information on these factors associated with the financial implications of the AEP and to perform in-depth analysis of these data.

The main limitations of the current study were limited access to medical records from private hospitals and an absence of data on the healthcare costs and physician workload associated with inappropriate admissions. As topics of future research, determination of the potential implications of adopting the AEP in terms of admissions per bed and the workload of physicians and nursing staff would provide insights into the efficiency aspect of the AEP as an appropriateness evaluation tool. Additionally, examining the effectiveness of educating reviewers with non-clinical backgrounds on the clinical knowledge needed to use the AEP to is suggested for future research.

In conclusion, our findings show that the AEP is a reliable instrument for determining the appropriateness of admissions in public district hospitals in Korea. However, due to differences in clinical practice and hospital admission policies between Korea and the USA, the AEP should be modified for use in Korea.