Cross-sectional study of surgical quality with a novel evidence-based tool for low-resource settings.

BACKGROUND: Adverse events from surgical care are a major cause of death and disability, particularly in low-and-middle-income countries. Metrics for quality of surgical care developed in high-income settings are resource-intensive and inappropriate in most lower resource settings. The purpose of this study was to apply and assess the feasibility of a new tool to measure surgical quality in resource-constrained settings.
METHODS: This is a cross-sectional study of surgical quality using a novel evidence-based tool for quality measurement in low-resource settings. The tool was adapted for use at a tertiary hospital in Amazonas, Brazil resulting in 14 metrics of quality of care. Nine metrics were collected prospectively during a 4-week period, while five were collected retrospectively from the hospital administrative data and operating room logbooks.
RESULTS: 183 surgeries were observed, 125 patient questionnaires were administered and patient charts for 1 year were reviewed. All metrics were successfully collected. The study site met the proposed targets for timely process (7 hours from admission to surgery) and effective outcome (3% readmission rate). Other indicators results were equitable structure (1.1 median patient income to catchment population) and equitable outcome (2.5% at risk of catastrophic expenditure), safe outcome (2.6% perioperative mortality rate) and effective structure (fully qualified surgeon present 98% of cases).
CONCLUSION: It is feasible to apply a novel surgical quality measurement tool in resource-limited settings. Prospective collection of all metrics integrated within existing hospital structures is recommended. Further applications of the tool will allow the metrics and targets to be refined and weighted to better guide surgical quality improvement measures. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Lack of access to safe and affordable surgery and anaesthesia is a major cause of death and disability in low-and-middle-income countries (LMIC).1 2 Currently, LMIC perform only 6.3% of the world’s surgical procedures and experience two-thirds of adverse healthcare events and 50% of postoperative mortality.1–3 The importance of improved access to high-quality surgical care worldwide has been recognised. The World Bank has set four World Development Indicators (WDI) and targets to improve access to safe and affordable surgical care. To achieve the target of 5000 surgical procedures per 100 000 people by 2030, 143 million additional surgical procedures are required annually.2 As efforts to scale up surgical volume continue, efforts to attain high levels of surgical quality of care are essential to avoid needless harm.

Quality of care is complex to characterise and there are two widely accepted definitions. First, that of Institute of Medicine (IOM) which defines high quality of care as being safe, effective, timely, patient-centred, efficient and equitable.4 Second, that of Donabedian which refers to high-quality care as having high-quality structures, processes and outcomes.5 6 Given these multiple facets, quality of care is challenging to measure but the ability to do so is imperative for improvement. A recent systematic review of research on surgical quality in LMIC shows that research has focused on single outcomes such as mortality and surgical site infections with minimal efforts to standardise metrics or to comprehensively evaluate multiple aspects of quality of care or quality of health systems.7 Current tools for measuring surgical quality have mostly been developed in high-income countries, often require long-term patient follow-up and advanced health information systems and are resource-intensive.8–11 These tools are often neither relevant nor feasible to implement in low-resource settings. To overcome these challenges, a quality tool was developed specifically for low-resource settings with a set of evidence-based indicators spanning all domains of high-quality care: safety, effectiveness, patient-centredness, timeliness, efficiency and equity while using Donabedian’s framework of structure, process and outcomes.12 Unlike other tools, the metrics are based on single inpatient encounters and use data collection tools that represent simple modifications to data registries already in use in most low-resource settings.12 The theoretical framework has been developed but this tool has not yet been implemented.

Brazil is an upper middle-income country comprised of five regions which are further subdivided into 26 states and a federal district. It has a national health system (Sistema Unico de Saúde (SUS)) which provides universal, free at point-of-care services, with 28% of the population having supplemental private insurance plans.13 At the national level, Brazil meets benchmark targets for the surgical WDI, but there are large regional inequalities consistent with the country’s health and economic disparities.14 15

To improve quality of surgical care in low-resource settings, we must first improve the ability to measure it. Thus, the aim of this study was to measure surgical quality and assess feasibility of a novel surgical quality tool for low resource settings in the state of Amazonas, Brazil.

Methods

This is a cross-sectional study in which prospective data on emergency and elective abdominal surgery cases and retrospective data on all surgical cases were collected at Hospital e Pronto-Socorro 28 de Agosto, a tertiary trauma hospital in Manaus, State of Amazonas, Brazil. Abdominal surgeries were selected for prospective observation as they affect a cross section of the population and consistently require timely surgical care. These data were used to calculate the 14 indicators of surgical quality. Prospective data were collected from 24 July to 21 August 2017 and retrospective data spanned from January 2017 to January 2018.

Instrument

The quality tool implemented, adapted from Citron et al,12 consists of 14 evidence-based metrics that integrate the three Donabedian framework dimensions: structure, process and outcomes, with the six IOM dimensions for high-quality care: safe, effective, patient-centred, timely, efficient and equitable16 (online supplementary appendix).

The tool was adapted to the local site with the following modifications. Caesarean section rate, an effective outcome measure, was replaced by readmission rate as the study site does not provide obstetric care and sufficient evidence exists as to the validity of this measure.17 18 Provider density, a measure of effective structure, was replaced by proportion of surgeries with a fully qualified surgeon present as the hospital providers are employed as a pool and it was not possible to establish the full-time equivalents. A fully qualified surgeon was defined as a physician who had completed surgical residency training. It was recognised that prospective collection of perioperative mortality rate (POMR) and Clavien–Dindo classification of complications would not be possible, therefore only non-risk-adjusted POMR was collected retrospectively from hospital records. The remaining indicators were employed as originally designed (table 1).

Table 1

Modified surgical care quality tool implemented in this study

		Institute Of Medicine measures
		Safe	Effective	Patient-centred	Timely	Efficient	Equitable
Donabedian framework	Structure	Morbidity and mortality conference	Attending surgeon present (proxy for provider density)	–	Travel time to hospital	–	Patient median income to catchment population
	Process	Safe surgery checklist use	Procedure density	Use of consent	Time from ED arrival to non-elective abdominal surgery	Daily OR usage	–
	Outcome	POMR	Readmission rates within 30 days (proxy for caesarean section rate)	Patient hospital satisfaction questionnaire	Follow-up plan	–	Catastrophic patient-reported expenditure

ED, emergency department; IOM, Institute of Medicine; OR, operating room; POMR, perioperative mortality rate.

Data Collection

Retrospective and prospective data collection was performed using the four data collection tools: hospital administrative data,12 modified operating room (OR) logbook,12 modified intraoperative checklist and patient discharge survey12 19 20 (online supplementary appendix). The data collection method, metric, definition, domain and proposed targets for each indicator are shown in table 2.

Table 2

Indicator definitions, data collection methods and proposed targets

Measure	Domain covered	Data collection	Definition	Target
Data source: Hospital administrative data
Morbidity and mortality conference	Safe structure	Retrospective	Morbidity and mortality meeting conducted, documented as conducted	Minimum of 9 per year
POMR	Safe outcome	Retrospective from patient charts	Total number of inhospital deaths of patients who underwent a surgical procedure during the same admission over the total number of surgical procedures performed in the same time period	1%–2%
Readmission rate within 30 days	Effective outcome	Retrospective from patient charts	Percentage of patients who had a readmission to the study site within 30 days of their surgery	<10%
Data source: OR logbook
Procedure density	Effective process	Retrospective	Number of surgical cases performed in the OR/population of catchment area 100 000	5000 cases/100 000 population
Daily OR usage	Efficient process	Retrospective	Total usage/total available time across all OR. Usage was estimated from entry to exit of patient to the OR plus 30 min of turnover time. Availability was defined as time between start of the first case and end of the last case of the day, 7 days a week	85%
Data source: Modified intraoperative checklist delivered by researcher
Safe surgery checklist use	Safe process	Prospective	Proportion of surgical cases performed in the OR as per full WHO safe surgery checklist. Specific use of pulse oximetry, patient name band and intraoperative antibiotics	100%
Use of consent	Patient-centred process	Prospective	Proportion of cases performed in the OR with signed informed consent, paperwork or documentation that informed consent was obtained or unable to be obtained due to patient condition	100%
Hours from ED arrival to non-elective abdominal surgery	Timely process	Prospective	Mean (time of incision − time of arrival in ED) for non-elective abdominal cases	<24 hours
Fully qualified surgeon present	Effective structure	Prospective	Per cent of surgeries performed with an attending surgeon present	100%
Data source: Patient discharge survey delivered by researcher
Patient hospital satisfaction questionnaire	Patient-centred outcome	Prospective	Mean total score on HCAHPS survey	‘Top box response rate’ 70% across all fields
Travel time to hospital	Timely structure	Prospective	Proportion of patients reporting having travelled <2 hours to reach the hospital	80%
Follow-up plan	Timely outcome	Prospective	Proportion of patients answering yes to ‘Have you received instructions on how you will receive continuing care?’	100%
Mean income compared with mean catchment income (or GDP/capita)	Equitable structure	Prospective	Mean income determined on patient exit interview/mean income of catchment population or GDP per capita	≤1
Proportion of patients experiencing catastrophic expenditure	Equitable outcome	Prospective	Proportion of patients where sum of total self-reported expenditure/self-reported income is >40%	0%

ED, emergency department; GDP, gross domestic product; HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; OR, operating room; POMR, perioperative mortality rate.

​Prospective data collection

Prospective data collection was performed using a modified intraoperative checklist and a patient survey at the time of discharge (online supplementary appendix). Data collection took place during a 4-week period from 24 July to 21 August 2017 with at least one data collector present 24 hours a day. Our study population was determined to be all the patients undergoing abdominal surgery during this period, as it was determined 1 month would be sufficient time to determine feasibility of the tool. All consecutive cases during the data collection period were included to avoid potential sources of bias in cases observed. The modified intraoperative checklist was used during all cases during the study period. The survey was administered in Portuguese on the day of discharge following any abdominal surgery which involved an inpatient stay. All data were recorded electronically using Kobo Toolbox.21

​Retrospective data

Retrospective data collection of hospital baseline administrative data and OR logbook data from January 2017 to January 2018 was performed to obtain five of the indicators: safe structure, safe outcome, effective process, effective outcome and efficient process (table 2).

​Hospital administrative data

Five variables were collected retrospectively. Data on the number of morbidity and mortality conferences and surgical volume data were collected for 1-year period while POMR data were collected for a 6-month period. Data on readmission rate within 30 days of surgical procedure and OR utilisation were estimated during a 4-week period. For variables requiring a value per 100 000 population the hospital catchment area was defined as one-fourth of the population of Amazonas, as the study hospital is one of the four hospitals which serves as a referral centre for the state and currently no consensus on defined catchment area exists in the region.

Catastrophic expenditure was estimated using patient’s reported income and direct and indirect expenditures. The state of Amazonas income was obtained as the total income from all sources from the Brazilian Institute of Geography and Statistics data for the year of 2017.22 Purchase power parity was used to convert Brazilian Reals to 2017 US$.23

Data analysis

Descriptive statistics were calculated for the indicators as appropriate. For discrete data we present ratios or percentages, and for continuous data mean and SD or median and IQR. If a data point was missing, that patient was excluded only for the quality indicator calculation for which the missing data point was needed. The indicator for patient-centred outcome, patient hospital satisfaction, was calculated as the proportion of patients giving the highest rated option for responses using the Hospital Consumer Assessment of Healthcare Providers and Systems scoring methodology.24

Subgroup analysis comparing emergency and elective cases was performed for three of the indicators: safe process (WHO safe surgery checklist use and its specific components of pulse oximetry, name band and antibiotics), patient-centred outcome (patient satisfaction) and timely outcome (follow-up plan at the time of discharge). A two-tailed t-test was conducted and an alpha of p<0.05 was used to determine statistical significance. All statistical analyses were performed using Microsoft Excel V.16.4 and Stata V.15.0 (StataCorp).

Patient and public involvement

Patients and the general public were not involved in the development of the research question.

Results

In the year 2017, a total of 7214 surgeries were performed across all specialties. During the 4-week prospective data collection period, 207 abdominal surgeries were performed and 183 were observed during this study. A total of 125 patients gave consent to complete the discharge survey. Demographic characteristics of patients are shown in table 3.

Table 3

Demographics of observed surgeries and of discharge survey respondents

Abdominal surgeries demographics			Discharge surveys demographics
Surgery type		N (%)	Surgery type		N (%)
	Emergency	146 (80%)		Emergency	99 (79%)
	Elective	37 (20%)		Elective	26 (21%)
	Total	183		Total	125
Procedure type		N (%)	Gender		N (%)
	Appendectomy	62 (34%)		Female	61 (49%)
	Exploratory Laparotomy	52 (28%)		Male	64 (51%)
	Cholecystectomy	39 (21%)	Age
	Hernia repair	11 (6%)		Mean (SD)	40 (16.8)
	Other	10 (5%)	Education		N (%)
	Repeat Exploratory Laparotomy	9 (5%)		8th grade or less	45 (36%)
				High school, no diploma	14 (11%)
				High school, graduate	47 (38%)
				University courses	19 (15%)
			Annual income		N (%)
				<12 000 BR (US$5833)	31 (26%)
				12 000–240 000 (US$5833–US$11 667)	50 (42%)
				24 000–60 000 (US$11 667–US$29 169)	23 (19%)
				>60 000 (US$29 169)	14 (12%)
			Survey section response rates		N (%)
				HCAHPS	125 (100%)
				Income and expenses	118 (94%)
				Delays of care	104 (83%)

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Indicators

Fourteen indicators from the modified surgical quality tool were successfully collected. The hospital met the targets for effective outcome and timely process domain. The rest of the indicators were not met. This included safe structure, process and outcome, effective structure and process, patient-centred process and outcome, timely structure and outcome and equitable structure and outcome (table 4).

Table 4

Surgical care quality tool targets and results

		IOM measures
		Safe	Effective	Patient-centred	Timely	Efficient	Equitable
Donabedian framework	Structure	Morbidity and mortality conference0(minimum 9 per year)	Attending surgeon present98%(100%)	–	Travel time to hospital31% <2 hours(80% <2 hours)	–	Patient median income to catchment population1.1(≤1)
	Process	Safe surgery checklist use0%(100%)	Procedure density710/100 100(5000/100 000)	Use of consent74%(100%)	Time from ED arrival to non-elective abdominal surgery7 hours(<24 hours)	Daily OR utilisation45%(85%)	–
	Outcome	POMR2.6%(1–2%)	Readmission rates within 30 days3%(<10%)	Patient hospital satisfaction questionnaire70% in 3/10 fields(70% across all fields)	Follow-up plan47%(100%)	–	Catastrophic patient-reported expenditure2.5%(0%)

Green highlights the indicators for which set targets were met. Red indicates the targets that were not met.

ED, Emergency Department; ED, emergency department; IOM, Institute of Medicine; OR, operating room; POMR, perioperative mortality rate.

Safe care indicators

There were no morbidity and mortality conferences held at the hospital. The safe surgery checklist in its entirety was not used in any of the surgeries observed. In terms of the specific components patient name band was not used in any of the surgeries, while pulse oximetry was used for 100% of patients. Perioperative antibiotics were administered in 69% (124/181) of cases, with higher administration in emergency cases at 79% (108/144) than in elective cases at 43% (16/37) (p<0.05). POMR was estimated at 2.6% (86/3249) over a 6-month period. In the original tool, risk adjustment data on functional status, American Society of Anesthesiologists’ physical status classification system or wound type are prospectively collected, however in this study POMR data were retrospectively collected and risk adjustment data were not available. The only exception was re-operating rate estimated to be 6.7% (13/195) for the 4-week period of prospective data collection.

Effective care indicators

Procedure density for 2017 was 710 per 100 000 catchment population. Of the 207 patients who underwent surgery during the prospective period, we were unable to track nine patients during the 30-day postoperative period and three patients were reported as dead. Of the remaining 195 patients, 3% (5/195) were readmitted within 30 days of their first surgery at the study site. Of the 183 abdominal surgeries observed, 98% (179/183) of all surgeries were performed with a fully qualified surgeon present, with the remaining ones performed by a surgical resident.

Patient-centred care indicators

Written consent was obtained in 74% (134/182) of surgeries, with 92% (34/37) of elective cases having consent documentation but only 69% (100/145) of emergency cases (p<0.05). For the patient hospital satisfaction questionnaire, on average only 3 of the 10 categories received a top box response rate of greater than 70% (table 5). However, in elective cases, six categories received a top box response rate greater than 70% compared with only three categories for emergency cases. Overall, hospital cleanliness was the worst scoring category, and communication with doctors and nurses the best scoring.

Table 5

HCAHPS questionnaire average per cent top box score results by category

	Overall (%) (n=125)	Elective (%) (n=26)	Emergency (%) (n=99)
Communication with nurses (Q1–3)	86	94	84
Communication with doctors (Q5–7)	85	96	82
Responsiveness hospital staff (Q4,11)	34	30	33
Pain management (Q13,14)	80	77	81
Communication about medicines (Q16,17)	57	35	61
Discharge information (Q19,20)	34	82	21
Cleanliness of hospital (Q8)	21	31	18
Quietness of hospital (Q9)	58	77	53
Hospital rating (Q21)	61	77	57
Recommend the hospital (Q22)	55	62	54

Highlighted are the targets met with scores >70%.

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Timely care indicators

Of the 104 responses related to timing of care in the discharge questionnaire, 84% (87/104) were emergency cases and 16% (17/104) were elective. In emergency cases, the median time for patients to seek care was 8 hours (IQR=20.8), ranging from immediately after perceiving symptoms to 2 months (figure 1). Overall, only 31% (32/104) of the patients took less than 2 hours to travel to the hospital, with the median travel time of 4.2 hours (IQR=23.0). When looking at emergency cases specifically, 28% (24/87) of patients reached the hospital within 2 hours, with median travel time of 4.4 hours (IQR=23.3). On arrival, of the emergency abdominal surgeries observed, 79% (110/139) of cases had their surgery within 24 hours of arrival to the hospital, with the median time from arrival to surgery of 7 hours (IQR=8.1). The three delays in accessing care, commonly defined in the literature,19 were mapped for patients undergoing emergency abdominal surgery (figure 1)

Figure 1

The three delays of care in the patient’s journey for emergency abdominal surgery.

Forty-seven per cent (59/125) of patients had a follow-up plan at discharge, with 81% (21/26) of the elective cases having a follow-up plan, but only 38% (38/99) of the emergency cases.

Efficient care indicators

Over the 4 weeks of prospective study observation, for all surgeries performed in the hospital, the mean daily OR utilisation was 45.1% (SD 6.35).

Equitable care indicators

The annual average patient-reported income was US$12 455 (25 621 BRL) (SD=8077). The average annual salary of Amazonas is US$11 189 (23 016 BRL) resulting in a 1.1 ratio for the patient average annual income to the catchment population. Of the 118 patients, only 2 (2.54%) reported expenses that exceeded 40% of their annual income, corresponding to catastrophic expenditure.

Discussion

This is the first study to comprehensively measure quality of surgical care using a novel tool specifically designed for resource-limited settings. Due to foreseen challenges, the tool required adaptation for use in this particular hospital by replacing two indicators and collecting five indicators retrospectively. As a result of the tool adaptation, all 14 indicators were successfully collected and provided the hospital with insight on specific strengths and particular areas for quality improvement.

Previous studies addressing quality of surgical care in LMICs have largely focused on individual measures, mostly mortality and morbidity, with few taking a comprehensive approach to quality.7 25–27 The tool employed in this study combines many of these individual measures and systematically captures all facets of surgical quality, including structures, processes and outcomes. In Brazil, prior studies have shown that the WHO safe surgery checklist is not routinely employed and that delay in receiving surgical care is a major factor in patient mortality.25 27 The findings of this study support minimal implementation of the full WHO safe surgery checklist but they characterised compliance with specific items of the checklist and found differences between emergency and elective procedures. Our findings on timely care indicators are consistent with the literature in Brazil but the novel quality tool goes beyond measuring access to care and facilitates characterisation of the three delays in accessing care, providing useful information on when interventions to increase timely care would be the most beneficial. Previous studies had shown poor satisfaction with SUS medical care in Brazil, particularly with surgical care.28 This is supported by our findings on the patient-centred outcome of satisfaction which showed 70% of patients were satisfied in only 3 of 10 categories.

In comparison with other tools available and commonly used in high-income settings, this novel tool required few resources, covered a single inpatient encounter and used existing data, which contributed to making data collection feasible. The authors opted to have direct external observers to complete the prospective data collection and administer discharge questionnaire. This was to assess the feasibility of using the surgical quality tool for data collection, without confounding our results with challenges inherent in implementation. Implementation of the tool into everyday data collection will be an important next step. Additionally, POMR, readmission data and return to the OR data were collected retrospectively which was time-consuming. For the tool to be applicable and sustainable, the processes carried out by the external observers would need to be integrated into everyday practice by the facility staff, mainly through the adoption of modified data registries. Currently, efforts for the indicator collection tools to be integrated into routine hospital practice are underway.

In addition to quality, the tool can be used to evaluate surgical care access as it incorporates the surgical WDI indicators. Our study indicated that only 32% of patients accessed a facility providing surgical care within 2-hour access compared with the modelled regional estimate of 97.2% of the population being within 2 hours of surgical care, conversely 97.5% of patients in our study had protection from catastrophic expenditures compared with the modelled regional average of 82%.14 The study by Massenburg et al used modelled estimates of access compared with the direct patient reports in this study, likely making this study more accurate. The estimated POMR of 2.6% was higher than the previous estimate of 1.12% for the North Region obtained from national databases and the target of <1%. This is likely due to the study site being a tertiary centre with more complex cases and a hospital that does not perform caesarean sections, a common surgical procedure that has a low mortality rate. The differences in estimates between our studies and previously reported data highlight the importance of hospital-based indicators to provide meaningful data that can be used to guide quality improvement. And yet, by employing this tool widely, regional and national, collection of the six surgical WDI can be achieved.

Limitations and future directions

Our study was limited by the absence of a predefined catchment population for the study hospital which was required to calculate two of the indicators, procedure density and patient median income to catchment population. We defined the catchment area as one-fourth of the population of Amazonas, based on the number of hospitals in the state, which might be an overestimate of the true catchment area of the hospital. However, local providers experience is that most surgical cases are brought to the study site, highlighting the uncertainty when estimating the catchment area. Given the specialist nature of the hospital, targets for procedure density were not applicable as a number of procedure types, for example caesarians or elective orthopaedic surgery are not undertaken at the site. Similarly, the equitable structure indicator requires the catchment population income, instead GDP per capita in the state was used as proxy. Predefined catchment areas or refining of the indicators could facilitate implementation of the surgical tool. The readmission rate and POMR indicators were collected only at the study facility and patients who sought care at a different facility were likely missed. Furthermore, despite the quality tool being designed for prospective collection, five of the indicators had to be collected retrospectively. The retrospective collection was laborious, and for quality measurement to be a continuous sustainable endeavour we highly recommend prospective data collection for all the indicators by integrating these metrics as part of the hospital information system.

This study demonstrated that it is feasible to adapt the surgical quality framework to the local context and comprehensively measure surgical quality, suggesting that the tool could be used in other low-resource settings. In addition to local quality improvement, the widespread use of this tool would allow for comparison between facilities within a region, and regions within the country in order to guide the efficient allocation of limited resources and support facilities to provide high-quality care. For ease of comparison, the indicators in the tool should be combined into a composite score with indicators weighted according to importance. Although all measures in the tool are required in high-quality surgical systems, some indicators such as patient mortality, morbidity satisfaction and catastrophic expenditures are indivisible from high-quality care, therefore they may require greater weighting than other factors such as a patient having follow-up plan at the time of discharge. In order to be able to assign meaningful weights to each item to create a composite score that measures all aspect of surgical care, and not a single indicator, a significantly larger body of data from the tool is required.

Conclusion

As access to surgical care is scaled up worldwide, robust tools to track surgical quality and guide quality improvement at the facility, regional and national levels will be essential to ensure universal access to high-quality surgical care. This study demonstrates that it is feasible to comprehensively measure surgical quality in low-resource settings. Integration of prospective systems of data collection into routine practice is required to make this data collection sustainable.