Feasibility and Accuracy of a Nonmedical Research Person in Assimilation and Calculation of Acute Physiologic Assessment and Chronic Health Evaluation Scores in an Indian Intensive Care Unit.

BACKGROUND: The Physiologic Assessment and Chronic Health Evaluation (APACHE) score assimilation and calculation, as well as other demographic data collection, is inherent to research and nonresearch related needs of intensive care. There may be a role for well-trained nonmedical personnel to collect this vital material to enhance research and the standard of care in the Intensive Care Units (ICUs) in countries that are poorly funded and resourced in terms of medical personnel. AIMS: The aim of this study is to verify the interrater reliability of a trained nonmedical personnel and ICU trainee in the collection and calculation APACHE scores.
MATERIALS AND METHODS: In a prospective study, two raters who were blinded, one a trained nonmedical ward clerk and another an ICU trainee, assimilated data and calculated APACHE scores for 60 consecutive patients admitted to two tertiary mixed ICUs (with a total of 19 beds). Primary outcomes were to assess interrater and interclass correlation as well as the agreement of scores between the two raters.
RESULTS: There was an excellent correlation of APACHE scores (Kappa coefficient of 0.92) and Bland-Altman plot depicted overall good agreement with low bias between raters.
CONCLUSIONS: A well-trained and supervised nonmedical research person can assimilate and calculate APACHE II scores with good agreement with an ICU trainee. This may help in deriving data from medically understaffed ICUs in India, thus promoting much-needed research from such ICUs.

INTRODUCTION

Aim

The aim of this study is to verify the interrater reliability of a trained nonmedical personnel and the Intensive Care Unit (ICU) trainee in the collection and calculation the Acute Physiologic Assessment and Chronic Health Evaluation (APACHE) scores.

Background

The APACHE and other severity of illness scores are keys to numerous facets of Intensive Care practice. APACHE scores help stratify patient risk, help in comparison of baseline risk, especially for interventional studies.[12] From patient care perspectives, the APACHE scores have been used to predict mortality and plan interventions.[3] From an administrative point of view, it may help in the utilization of resources.[4]

Intensive care research from India is vitally important in improving the standard of care administered to critically ill patients.[5] Resource and workforce constraints[6] dictate that severity of illness score calculations could be helpful in prioritizing efforts and providing data important to improving the practice of Intensive Care. However, the very same constraints of workforce and funding have resulted in either the inability to collect crucial data from Intensive Care centers in India or utilizing nonmedical staff for data collection.

Although the utility of nonmedical staff in collection of APACHE scores has been studied and ratified by Kho et al. in western settings,[7] data from India are needed as it is possible that many ICU's utilize in this country use nonmedical personnel for APACHE score calculation and other demographic data assimilation.

In a prospective, blinded, observational study, we intended to verify the accuracy of a nonmedical clerk in the calculation of the APACHE score.

MATERIALS AND METHODS

Design

The prospective analytical study was to verify accuracy and utility of a nonmedical trained clerk to measure APACHE scores.

Setting

Two adult ICUs comprised a 13-bedded tertiary level ICU and 6-bedded tertiary level ICU. The case mix of these two ICUs together is approximately 70% of surgical cases and 30% of medical cases. A total of 60 consecutive patients admitted to both ICUs were enrolled in the study. Assessors would collect scores at different periods of the day and be blinded to each other results.

Personnel (raters)

Trained research clerk

Nonmedical ward clerk with no clinical or medical training. The clerk in this study completed a Bachelor of Computer Application (Bachelor degree from Madras University, Tamil Nadu) and Master of Social work (Annamalai University, Tamil Nadu). This person was experienced with demographic data collection in an ICU with an initial supervised period (3 months) of APACHE score calculation and tabulation. Subsequently, the clerk has been collecting the APACHE scores for 3 years with the monthly supervision of score collection and score tabulation.

Intensive Care Unit trainee

Intensive care trainee who has finished his or her postgraduation (MD) in Anesthesiology and is training in Intensive Care for the past 2 years.

Data collection

Both the research clerk and ICU trainee would collect APACHE II scores of 60 consecutive patients at different periods of the day to ensure originality of scoring. Acute physiological parameters of the APACHE score were to be collected from patient records (derived from nursing and doctor assessments, ICU flow sheet, and electronic database information). The Chronic Health score would also be derived from patient records and electronic database information wherever possible. When in doubt, the clerk would be allowed to verify findings with medical personnel located in the ICU on that day (who were not involved with the conduct of the study). Both raters would use the same validated APACHE II calculator located within computers in the ICU to derive their final score. Both raters were not to communicate scoring information and blinded to each other's APACHE score results. The results were to be submitted to another investigator who tabulated their results in a Microsoft Excel sheet for analysis.

The Institutional Review Board for research and ethics approved the study. Waiver of consent was given to conduct the study as there was a negligible patient risk.

Primary outcome

Interrater and interclass correlation as well as the agreement of APACHE scores between research clerk and trainee.

Statistical methods

The sample size was tested for an Interclass Correlation Coefficient (ICC) of 0.80, and 0.90 from the literature,[7] when the values obtained are tested against 0.5 of worst ICC with a power of 80% and error of 5%, we need a sample size of 28 to detect a 0.8 ICC and 11 to detect 0.9. The sample size was calculated in R software using ICC sample size package.

Data were summarized using mean (Standard Deviation [SD])/Median (range) for quantitative variables and frequency along with percentage for categorical variables. The rater agreement between continuous variables was analyzed using concordance correlation coefficient and ICC and presented with 95% confidence interval [CI]. The maximum allowable limit between the raters was presented with limits of agreement (LoA) and visualized with Bland–Altman plot. The agreement between the categorical variables was analyzed using kappa and presented with 95% CI. The kappa coefficient interpreted using Landis and Koch[8] (slight = 0.0 to 2.0, fair = 0.21 to 40, moderate = 0.41 to 0.60, substantial = 0.61 to 0.80, and almost perfect = 0.81 to 1.00.) All the data analysis were performed using STATA I/C 13.1 software (StataCorp, USA).

RESULTS

Patient cohort

A total of 60 patients were enrolled consecutively and assessed for APACHE score calculation by the Research clerk and ICU trainee. Table 1 summarizes demographic details of these patients. The mean age of this patient cohort was 46.2 years and included 31.7% of women patients with a moderate acuity of illness entailing a median ICU stay of 3 days and mortality of 13.3%. About half of all patients were postoperative (53.3%), 83.3% were emergent, most required mechanical ventilation (68.3%), and a good proportion (41.6%) required vasopressors or inotropes [Table 1].

Table 1

Patient demography as collected by the medical rater

Primary outcomes

Table 2 summarizes the correlation of score calculations by the two raters. There was an almost perfect correlation of the acute physiology domain of the APACHE score (Kappa Coefficient of 0.95) as well as the Glasgow Coma Scale (GCS) domain (ICC of 0.82). The Chronic Health score domain had a substantial correlation (ICC of 0.66).

Table 2

Interclass correlation and kappa coefficient scores

Overall, the APACHE II Kappa Coefficient was almost perfect (0.92).

The Bland–Altman plot depicts that bias was low among the raters [Figure 1 and Table 3]. Most of the values were within the LoA constructed. The points were close to the average difference, and the presence of scattering indicated random bias (i.e., the difference in the agreement is due to the subject variation rather the rater variation) concluding an overall good agreement.

Figure 1

Bland–Altman plot with levels of agreement of APACHE score recordings among raters

Table 3

Agreement between Acute Physiologic Assessment and Chronic Health Evaluation scores among the two raters

DISCUSSION

Collecting data related to Intensive care admission is vital to prognostication, resource allocation, and research activities. Indian ICUs belong to diverse health systems and have a unique case mix.[9] Health-care workers are often stretched to the limit,[10] and the crucial role of data collection is often missed or is taken up by nonmedical personnel in some centers. With nonmedical personnel collecting this data, interobserver variation in scores may occur.[11]

Kho et al. have validated the interrater reliability of nonmedical and medical personnel in the collection of the APACHE score in the west. It is plausible this could be carefully duplicated in India.

In a prospective, blinded, single-center study, we intended to verify the inter-rater reliability of a trained nonmedical research clerk with an ICU trainee. Our findings suggest that there is an excellent agreement in the calculation of APACHE scores between a nonmedical research clerk and ICU trainee in a mixed ICU population.

Among the individual components, the acute physiology component seemed to have the best correlation (Kappa Coefficient of 0.95 [0.92–0.97]). These parameters are a conglomeration of acute physiological variables as well as laboratory parameters. These seemed to be efficiently recorded from the nurse's flow sheet in the ICU and the computerized patient archiving and communication system. The GCS score, which is recorded on the nurse's flow sheet, had a good correlation as well. However, the chronic health component had an ICC of 0.66 (0.47–0.85), which depicts room for improvement. This component was derived mainly from patient notes and/or discussion with overseeing medical personnel. It is probable that this component needs more careful attention and documentation to achieve better correlation.

This finding may substantiate the use of nonmedical clerks in many centers in the developing world who are also reasonably educated, trained in the ICU, and periodically supervised such as the clerk in our study. This could facilitate the assimilation of data for purposes such as internal audits, prognostication, resource allocation, and larger research purposes. Furthermore, with the introduction of the Customized, Health in Intensive Care, Trainable research and Analysis Tool, an interactive cloud-based application launched under the auspices of the Indian Society of Critical Care Medicine in 2015 (http://www.isccm.org/chitra.aspx); there may be a mechanism of routine and efficient data collection so as to benefit the local ICU as well as gather information relevant to the Indian ICU setting.

Our study has certain drawbacks, mainly related to generalizability and reproducibility. This is because the nonmedical research clerk in our study was unique for a number of reasons – previous exposure to an Intensive Care environment as a ward clerk, reasonable education with familiarity with data collection, use of technology, as well as frequent supervision of data collection and work in the unit. These features may not be replicable in other units; however, similar individuals with appropriate supervision could potentially have the same results.

CONCLUSIONS

A well-trained and supervised nonmedical research person can assimilate and calculate APACHE II scores with good agreement with an ICU trainee. This may help in deriving data from medically understaffed ICUs in India, thus promoting much-needed research from such ICUs.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.