Quality indicators for responsible use of medicines: a systematic review.

OBJECTIVE: All healthcare systems require valid ways to evaluate service delivery. The objective of this study was to identify existing content validated quality indicators (QIs) for responsible use of medicines (RUM) and classify them using multiple frameworks to identify gaps in current quality measurements.
DESIGN: Systematic review without meta-analysis.
SETTING: All care settings. SEARCH STRATEGY: CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched up to April 2018. An internet search was also conducted. Articles were included if they described medication-related QIs developed using consensus methods. Government agency websites listing QIs for RUM were also included. ANALYSIS: Several multidimensional frameworks were selected to assess the scope of QI coverage. These included Donabedian's framework (structure, process and outcome), the Anatomical Therapeutic Chemical (ATC) classification system and a validated classification for causes of drug-related problems (c-DRPs; drug selection, drug form, dose selection, treatment duration, drug use process, logistics, monitoring, adverse drug reactions and others).
RESULTS: 2431 content validated QIs were identified from 131 articles and 5 websites. Using Donabedian's framework, the majority of QIs were process indicators. Based on the ATC code, the largest number of QIs pertained to medicines for nervous system (ATC code: N), followed by anti-infectives for systemic use (J) and cardiovascular system (C). The most common c-DRPs pertained to 'drug selection', followed by 'monitoring' and 'drug use process'.
CONCLUSIONS: This study was the first systematic review classifying QIs for RUM using multiple frameworks. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. Although many QIs were identified, this approach allowed for the identification of gaps in quality measurement of RUM. In order to more effectively evaluate the extent to which RUM has been achieved, further development of QIs may be required. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A comprehensive literature search was undertaken across seven databases and government agency websites without restriction of disease categories and care settings.

The classification of quality indicators (QIs) was based on multiple frameworks (eg, Donabedian’s framework, the Anatomical Therapeutic Chemical classification system and a validated classification for causes of drug-related problems) for maximum understanding and profiling of the included QIs.

Content validated QIs that were developed using consensus methods were only included, and therefore valid QIs might have been excluded during the screening process.

Although 5% of this review processes were verified by multiple authors to check for accuracy, most of the classification was undertaken by one author.

Introduction

Responsible use of medicines (RUM) is an essential element in achieving quality of care for patients and the community. According to the WHO, RUM implies that the activities, capabilities and existing resources of health system stakeholders are aligned to ensure patients receive the right medicines at the right time, use them appropriately and benefit from them.1 RUM, however, is not easily achievable, and if medicines are used inappropriately, negative consequences for both patients and/or the society may occur. It is reported that worldwide more than 50% of all medicines are prescribed, dispensed or sold inappropriately, while 50% of patients fail to take them correctly.2 In addition, it has been reported that one-third of preventable drug-related admissions are associated with medication non-adherence, 31% are related to prescribing problems and 22% are related to monitoring problems.3 The frequency of these medication errors varies depending on the specific medicine. For example, previous systematic reviews have found that preventable drug-related admissions to hospital accounted for 3.7% of all admissions, of which four groups of drugs, antiplatelets, diuretics, non-steroidal anti-inflammatory and anticoagulants accounted for more than 50% of the drug groups associated with those preventable drug-related hospitalisations.3 From the economic perspective, globally, the cost associated with medication errors has been estimated at US$42 billion annually or almost 1% of total global health expenditure.4 Given the health concerns and the economic burden associated with medication errors, the achievement of RUM underpinned by an evidence-based approach has become increasingly important worldwide.

One critical element for any healthcare system or organisation is how to measure and evaluate RUM. A widely used method to do this is the use of quality indicators (QIs).5 6 QIs are explicitly defined and measurable items referring to the structures, processes or outcomes of care are usually described with a denominator and a numerator.7 The denominator is the total number of cases in the intended population, and the numerator is the number of cases that fulfil a predetermined criterion, and the calculated QI score indicates the quality of care.8 QIs can be used to monitor the quality of care provided by healthcare professionals in a single institution, to promote quality improvement activities, to make comparisons over time between institutions or to support consumers to choose healthcare providers.5 For QIs to be useful, they must be developed with scientific rigour, and all quality dimensions of care must be measured to capture a comprehensive landscape of healthcare quality.5

To achieve RUM using QIs, it is first necessary to identify existing QIs for RUM, independent of disease categories and care settings. Additionally, in the light of the concept of RUM, multifaceted assessment is required to gain full understanding of the breadth of coverage by QIs. To our knowledge, however, previously conducted systematic reviews have been restricted to setting (eg, hospital),9 disease state (eg, HIV/AIDS),10 specific to a healthcare group (eg, nursing sensitive QIs) or indicator name (eg, clinical indicators)11 and have only been classified based on Donabedian’s framework or implicit frameworks such as quality dimensions defined by the Institute of Medicine.12 Hence, the main purpose of this systematic review was to identify existing content validated QIs for RUM independent of disease category and care settings, and then classify them using multiple frameworks in order to identify gaps in current quality measurements.

Methods

Data sources

This systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (see online supplementary table S1).13 Two approaches were used to identify relevant QIs.

First, CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched to identify relevant articles published up to 5 April 2018. No restriction on year of study was applied. Search strategies comprised keywords and, when available, controlled vocabulary such as Medical Subject Headings/EMTREE based on three main terms: ‘quality indicators’, ‘development’ and ‘consensus’. Since ‘quality indicators’ are referred to by wide variety of terms such as clinical indicators, or performance measures, the finalised search strategies were developed using an iterative development process during which citations identified by various search terms were screened for relevance. We chose ‘consensus’ as a main term because QIs are recommended to be developed using expert panels based on rigorous evidence in order to ensure high face validity and content validity.14 Exact search dates for each database with the search strategies are included in online supplementary table S2.

Second, using Google, an internet search was also conducted (search terms: quality indicators, clinical indicators, performance indicator or performance measures) to capture additional QIs listed in the websites of relevant organisations responsible for quality improvement. Potentially relevant organisation’s websites, found in the process of literature review,9 12 15–17 were also searched (see online supplementary table S3).

Study selection

Inclusion criteria

Articles were included if they fulfilled the following criteria: (A) the article was peer reviewed and published in English, (B) numerators and denominators were defined for the QIs, or they could be directly deduced from the descriptions of the QIs, (C) the publication contained at least one medication-related QI, (D) the development of QIs was one of the objectives and (E) QIs were developed using consensus methods in order to confirm content validity. Furthermore, relevant organisations’ QIs found from websites were included if the organisation was a government agency for ensuring quality in healthcare, and at least one QI for RUM was reported with a clear description, as detailed above (B).

Given the concept of QIs and RUM mentioned above, we regarded a measurement tool as a QI for RUM when the definition of the QI referred to a medication. In addition, if publications concerned the same project/QIs set, the descriptions of the QIs in the most recent publication were used for data extraction.

Exclusion criteria

Articles were excluded if the consensus results for QI development were unclear, if QI lists were obtainable only by purchase or if QIs were for monitoring the effectiveness of national policies.

This study selection process was performed using a purposed designed screening proforma (see online supplementary table S4). The retrieved articles were transferred into Endnote to remove duplicates, then initial screening of journal names, titles and abstracts was conducted to remove irrelevant articles.

Data extraction

One researcher (KF) extracted the following data from the full text of included articles or websites: publication year, country or other targeted location in which QIs were intended to be used, name of measurement tools, total number of QIs, the number of relevant QIs for RUM, scope of the QIs and definition of QIs (numerator and denominator, if available). A data extraction proforma was designed, pilot-tested on five included studies, then refined accordingly.

Analysis

Descriptive statistics were computed for the results of the present review based on counts and proportions where relevant. Since the components of RUM are multidimensional, multiple frameworks were used to understand the breadth of coverage by QIs. That is, we used four types of classification: (1) problem type; (2) Donabedian’s framework; (3) the Anatomical Therapeutic Chemical (ATC) classification system; and (4) causes of drug-related problems (c-DRPs) classification system.

Problem type

The first step of a structured QI development process is to identify the problem for which measurement is needed.18 Classifying QIs according to problem type can highlight prioritised problems for QI development. Therefore, QI sets described in each source were classified into the following six problem types proposed by Evans et al18:

Disease based: problems relevant to diseases, illnesses, conditions, injuries or procedures for which the quality of care needs to be measured.

Patient based: problems related to patient groups, such as vulnerable elders and paediatric patients.

Treatment modality based: problems relevant to service providing areas, such as intensive care units or palliative care settings.

Organisation based: problems relevant to organisational issues, such as whether organisations have effective structures in place at an organisational level to support quality and safety.

Generic problems: problems relevant to issues that are multidisciplinary in nature and relevant to any form of healthcare delivery in multiple physical settings, such as falls prevention, or pain management.

Profession based: problems unique to the different healthcare professions and include availability and competence of healthcare personnel.

If a QI set related to more than one problem type, they were classified accordingly (eg, an article about QIs for nursing practice in the operating room fell into treatment modality-based and profession-based problem).

Donabedian’s framework

QIs were classified according to the widely used Donabedian’s framework of structure (referred to the factors that designate the conditions under which care is provided, such as material or human resources), process (referred to the actions of healthcare professionals, such as prescribing or monitoring) or outcome (referred to the changes in individuals that can be attributed to care provided), irrespective of the category defined in the original source.19 Online supplementary table S5 lists examples of QIs classified into these three categories.

The ATC classification system

QIs were first classified into medicine class specific indicators or general medication indicators, depending on whether the definition of the QI described a specific class of medicines. For example, a QI ‘numerator: patients with acute myocardial infarction (AMI) received aspirin within 3 hours of hospital arrival/denominator: AMI patients without aspirin contraindications’20 was classified as a medicine class specific indicator, while a QI ‘numerator: number of patients aged 65 years and older whose current medications are documented and reconciled at admission/denominator: number of patients aged 65 years and older in sample’21 was classified as a general medication indicator. After this process, medicine class specific indicators were classified using the first and second levels of the ATC code.22 A single QI was sometimes allocated into more than one ATC code. For example, a QI, ‘percentage of patients using opioids with concomitant laxatives’,23 represented A06 (drugs for constipation) and N02 (analgesics).

c-DRPs classification system

Since minimising the factors that contribute to drug-related problems (ie, causes of DRPs) is closely linked to achieving RUM, the extracted QIs were classified using a comprehensive taxonomy of the causes of DRPs.24 This taxonomy divides c-DRPs into the following nine categories.

Drug selection, for example, whether appropriate drugs are selected by healthcare professionals.

Drug form, for example, whether appropriate drug forms are selected by healthcare professionals.

Dose selection, for example, whether appropriate drug dosages are selected by healthcare professionals.

Treatment duration, for example, whether drugs are being prescribed or dispensed for an appropriate duration by healthcare professionals.

Drug use process, for example, whether drugs are taken properly by patients.

Logistics, for example, whether necessary drugs are properly delivered to the patients.

Monitoring, for example, monitoring for the effect/adverse effects of drugs.

Adverse drug reactions, for example, the occurrence of adverse drug reactions.

Other.

Note that a single QI was sometimes allocated into more than one c-DRP category. Online supplementary table S6 illustrates how QIs were classified using the c-DRP taxonomy.

All processes were conducted independently by one author (KF), and 5% of these processes were verified by TFC and RJM. Any issues that arose during the process were resolved by discussion between the research team (KF, RJM and TFC). Meta-analysis was not applicable due to heterogeneity in interventions, methods and reported outcomes. We believed that it was not necessary to assess the quality of the content validated QIs included in our studies such as their feasibility, and reliability because problems affecting QIs (eg, feasibility of data collection, reliability of calculating QI scores and opportunities for gaming) vary depending on the healthcare infrastructure and healthcare remuneration system in each country.

Patient and public involvement

As this was a literature review, there was no patient and public involvement in this study.

Results

Study selection

Initially, a total of 39 430 articles were obtained. The sample included 17 822 duplicate records, which were removed. After the initial screening, 973 full texts were assessed for eligibility with 842 excluded based on the inclusion and exclusion criteria. Eventually 131 articles met all inclusion criteria and were included in our review. Additionally, through the internet search, five relevant websites were identified and included in our review (figure 1).

Figure 1

Study flow diagram. QI, quality indicator; RUM, responsible use of medicines.

Study characteristics

Of the 131 articles, 78 articles (60%) developed QIs for use in three countries: USA (n=36),25–60 Canada (n=26)61–86 and Netherlands (n=16).23 87–101 The remaining 53 articles developed QIs for use in 16 other countries20 21 102–145 and 4 other targeted locations (such as the Organisation for Economic Co-operation and Development (OECD) countries)146–152 (figure 2). Of the five relevant websites, three were Australian organisations,153–155 one was a UK organisation156 and the other was USA organisation.157 The three Australian and UK organisations developed QIs at the organisation level, while the American website, National Quality Measures Clearinghouse, sponsored by the Agency for Healthcare Research and Quality, stored QIs developed by various countries. Of 7750 QIs listed in the 131 articles and 5 websites, we identified 2431 QIs for RUM: 1947 QIs from journal articles and 484 QIs from the web.

Figure 2

The number of publications by country and other target location.

While there were 21 different ways of labelling the measurement tools, ‘quality Indicators’ (n=80, 59%) was the most commonly used term in our included articles and websites, followed by ‘quality measures’ (n=11, 8%), ‘quality of care indicators’ (n=8, 6%) and ‘indicators’ (n=7, 5%).

In terms of the problem type, 43% of QI sets pertained to disease-based problems (n=89, eg, knee osteoarthritis), then 27% for treatment modality-based problems (n=55, eg, primary care), 21% for patient-based problems (n=44, eg, geriatric care), 5% for profession based problems (n=11, eg, community pharmacists), 3% generic problems (n=6, eg, long-term prescribing) or 1% organisation-based practice (n=2, eg, centralised intake systems). The majority of QIs (n=2289, 94%) were process indicators, while structure (n=80) and outcome (n=62) indicators accounted for only 3% each (table 1).

Table 1

Characteristics of studies and quality indicator sets

Reference	Year	Country/other target location	QI name	Scope of QIs (setting, condition, target patient group, occupation)	Problem type*	QIs for RUM (%)	Donabedian’s framework†			QI type‡
							S	P	O	Me	Ge
Broccoli et al102	2018	Africa	Quality indicators	Emergency care	T	19/76 (25)	5	14	0	16	3
Tropea et al21	2011	Australia	Clinical indicators	Older hospitalised patients	P	4/19 (21)	0	4	0	1	3
ACSQHC153 §	2012	Australia	Practice-level indicators	Primary care	T	3/35 (9)	0	3	0	0	3
ACSQHC154 §	2014	Australia	National Quality Use of Medicines Indicators	Hospital care	T	35/37 (95)	2	33	0	25	10
Caughey et al103	2014	Australia	Medication-related indicators	Primary care	T	28/28 (100)	0	28	0	28	0
Victorian Government155 §	2015	Australia	Quality indicators	Residential aged care	P, T	1/5 (20)	0	1	0	0	1
Nag et al104	2016	Australia	Quality indicators	Prostate cancer	D	1/12 (8)	0	1	0	1	0
O’Connor et al105	2017	Australia	Quality indicators	Psychotropic prescribing for people with dementia in aged psychiatry inpatient units	D, P, T	6/6 (100)	0	6	0	6	0
Sibthorpe et al106	2017	Australia	Indicators	Otitis media in primary healthcare for Aboriginal and Torres Strait Islander children	D, P, T	2/12 (17)	0	2	0	2	0
Stordeur et al107	2012	Belgium	Quality indicators	Breast cancer	D	9/32 (28)	0	9	0	9	0
Grypdonck et al108	2014	Belgium	Quality indicators	Knee osteoarthritis	D	8/21 (38)	0	8	0	8	0
Stordeur et al109	2015	Belgium	Quality indicators	Oesophageal cancer and gastric cancer	D	4/29 (14)	0	4	0	4	0
De Schreye et al110	2017	Belgium	Quality indicators	End-of-life care in people with Alzheimer’s disease, cancer or chronic obstructive pulmonary disease	D, P	23/81 (28)	0	23	0	23	0
Leemans et al111	2017	Belgium	Quality indicators	Palliative care	P	4/31 (13)	0	2	2	4	0
de Carvalho et al112	2017	Brazil	Indicators	Adult intensive care	P, T	7/62 (11)	7	0	0	7	0
Mackinnon and Hepler61	2002	Canada	Clinical indicators	Geriatric care	P	52/52 (100)	0	52	0	52	0
Robertson and MacKinnon62	2002	Canada	Clinical indicators	Geriatric care	P	52/52 (100)	0	52	0	52	0
Burge et al63	2007	Canada	Quality indicators	Cardiovascular primary care	D, T	11/31 (35)	0	11	0	11	0
Kröger et al64	2007	Canada	Quality indicators	Older adults with cognitive impairment or dementia	D, P	21/72 (29)	0	21	0	9	12
Ko et al65	2008	Canada	Quality indicators	Percutaneous coronary intervention	D	7/26 (27)	2	5	0	7	0
MacKinnon et al66	2008	Canada	Clinical indicators	Type 2 diabetes	D	12/21 (57)	0	12	0	12	0
Nigam et al67	2008	Canada	Medication use safety indicators	Medication use at inpatient and outpatient settings	T	20/20 (100)	1	17	2	6	14
Tu et al68	2008	Canada	Quality indicators	Acute myocardial infarction	D	16/38 (42)	0	16	0	16	0
Dixon et al69	2009	Canada	Quality indicators	Patients undergoing hepatic resection for metastatic colorectal cancer	D	2/18 (11)	0	2	0	2	0
Teresato and Lougheed70	2010	Canada	Performance indicators	Primary care for asthma	D, T	5/20 (25)	0	5	0	5	0
Krzyzanowska et al71	2011	Canada	Quality indicators	Cancer care for women	D, P	5/31 (16)	0	5	0	5	0
Schull et al72	2011	Canada	Quality of care indicators	Emergency department care	T	16/48 (33)	0	16	0	16	0
Addington et al73	2012	Canada	Performance measures	Schizophrenia	D	6/36 (17)	0	6	0	6	0
Stang et al74	2013	Canada	Quality indicators	High acuity paediatric conditions	D, P	28/62 (45)	0	28	0	28	0
Darling et al75	2014	Canada	Quality indicators	Non-small cell lung cancer operations	D	1/17 (6)	0	1	0	1	0
Nguyen et al76	2014	Canada	Quality indicators	Inflammatory bowel disease	D	6/11 (55)	0	6	0	6	0
Santana and Stelfox77	2014	Canada	Quality indicators	Adult injury care	D, P	4/31 (13)	1	3	0	4	0
Barber et al78	2015	Canada	Key performance indicators	Centralised intake systems for patients with osteoarthritis and rheumatoid arthritis	D, O	2/28 (7)	0	2	0	2	0
Fernandes et al79	2015	Canada	Clinical pharmacy key performance indicators	Hospital pharmacists	Pr	8/8 (100)	0	8	0	0	8
Khare et al80	2016	Canada	Performance indicators	Breast, prostate, colorectal and lung cancer	D	17/78 (22)	0	12	5	17	0
McKelvie et al81	2016	Canada	Quality indicators	Heart failure	D	1/6 (17)	0	1	0	1	0
Khare et al82	2017	Canada	Quality indicators	Bladder cancer	D	22/60 (37)	0	19	3	22	0
Tu et al83	2017	Canada	Performance indicators	Primary prevention of cardiovascular disease in the ambulatory care	D, T	9/28 (32)	0	9	0	9	0
Tu et al84	2017	Canada	Quality indicators	Chronic kidney disease in primary care	D, T	6/17 (35)	0	6	0	6	0
Chartrand et al85	2018	Canada	Quality indicators	Oral anticoagulant management in community pharmacies	D, Pr	38/38 (100)	2	34	2	38	0
Mukerji et al86	2018	Canada	Quality indicators	Ambulatory diabetes care	D, T	18/35 (51)	0	18	0	16	2
Sun et al20	2011	China	Quality indicators	Acute myocardial infarction	D	10/23 (43)	0	10	0	10	0
Bao et al113	2015	China	Quality indicators	Breast cancer	D	11/31 (35)	0	11	0	11	0
Chen et al114	2016	China	Quality indicators	Neonatal intensive care units nursing	Pr, T	2/11 (18)	0	2	0	1	1
Wu et al115	2016	China	Quality indicators	Nursing practice in the operating room	Pr, T	3/23 (13)	1	1	1	2	1
Li et al116	2017	China	Indicators	Rational drug use for community-acquired pneumonia in children	D, P	44/44 (100)	0	44	0	42	2
Wang et al117	2017	China	Quality indicators	Non-small cell lung cancer care	D	10/21 (48)	0	10	0	10	0
Ju et al118	2018	China	Quality indicators	Emergency nursing care	Pr, T	5/16 (31)	0	5	0	3	2
Tang et al119	2018	China	Quality indicators	Home care	T	1/70 (1)	1	0	0	0	1
Saust et al120	2017	Denmark	Quality indicators	Diagnosis and antibiotic treatment of acute respiratory tract infections in general practice	D, T	19/31 (61)	0	19	0	19	0
Campbell et al146	2008	Europe	Quality indicators	Cardiovascular disease in primary care	D, T	10/44 (23)	0	10	0	8	2
Adriaenssens et al147	2011	Europe	Quality indicators	Outpatient antibiotic prescribing	G, P	21/21 (100)	0	21	0	21	0
Petersson et al148	2014	Europe	Healthcare quality indicators	Rheumatoid arthritis	D	1/14 (7)	0	1	0	1	0
Boulkedid et al121	2013	France	Quality indicators	Obstetrical care in maternity units	P, T	2/18 (11)	0	2	0	2	0
Follmann et al122	2014	Germany	Quality indicators	Melanoma	D	3/12 (25)	0	3	0	3	0
Hussein et al123	2017	Germany	Quality indicators	Systemic antibiotics in dentistry	D, Pr	12/12 (100)	0	12	0	12	0
Hermann et al149	2006	International	Quality indicators	Mental health	D	4/12 (33)	0	4	0	4	0
Barber et al150	2015	International	Quality indicators	Cardiovascular disease care in patients with rheumatoid arthritis	D	2/11 (18)	0	2	0	2	0
Wakai et al124	2013	Ireland	Key performance indicators	Performance of emergency department	T	15/97 (15)	1	14	0	14	1
Murphy et al125	2016	Ireland	Key performance indicators	Prehospital emergency care	T	19/101 (19)	0	18	1	19	0
Barry et al126	2016	Ireland and UK	Prescribing indicators	Prescribing for children in primary care	P, T	12/12 (100)	0	12	0	12	0
Fukuma et al127	2016	Japan	Quality indicators	Non-dialysis chronic kidney disease	D	4/11 (36)	0	4	0	4	0
Masaki et al128	2017	Japan	Quality indicators	Elderly end-of-life care in nursing	P, Pr	4/33 (12)	0	4	0	4	0
Ueda et al129	2017	Japan	Quality indicators	Low-risk labour care provided by midwives	P, Pr	2/23 (9)	0	2	0	1	1
Ntoburi et al151	2010	Low-income countries	Indicators	Paediatric inpatient care	P, T	56/112 (50)	26	30	0	55	1
Perez-Cuevas et al130	2012	Mexico	Quality of care indicators	Type 2 diabetes	D	4/18 (22)	0	4	0	4	0
Doubova et al131	2014	Mexico	Quality indicators	Antenatal care	P	2/14 (14)	0	2	0	2	0
Muijrers et al87	2004	Netherlands	Prescribing indicators	General practice	T	34/34 (100)	0	34	0	34	0
Mourad et al88	2007	Netherlands	Quality indicators	Subfertility care	D	8/39 (21)	0	8	0	8	0
Drašković et al89	2008	Netherlands	Quality indicators	Clinical practice at memory clinics	T	1/14 (7)	0	1	0	1	0
Martirosyan et al90	2008	Netherlands	Prescribing quality indicators	Type 2 diabetes	D	14/14 (100)	0	14	0	14	0
van der Ploeg et al91	2008	Netherlands	Quality indicators	General practice care for vulnerable elders	P, T	36/81 (44)	0	36	0	24	12
Perry et al92	2010	Netherlands	Quality indicators	Dementia	D	2/23 (9)	0	2	0	2	0
Stienen et al93	2011	Netherlands	Quality indicators	Paediatric constipation	D, P	3/7 (43)	0	3	0	3	0
Wierenga et al94	2011	Netherlands	Quality indicators	Inhospital pharmaceutical care for elderly patients	P, Pr, T	85/87 (98)	0	85	0	71	14
Luitjes et al95	2013	Netherlands	Quality indicators	Hypertensive diseases in pregnancy	D, P	5/14 (36)	0	5	0	5	0
van den Bosch et al96	2014	Netherlands	Quality indicators	Antimicrobial use in hospitalised adult patients with sepsis	D, P, T	5/5 (100)	0	5	0	5	0
van den Bosch et al97	2015	Netherlands	Quality indicators	Antibiotic use in hospitalised adults	D, P, T	11/11 (100)	2	9	0	11	0
Woiski et al98	2015	Netherlands	Quality indicators	Postpartum haemorrhage	D	7/22 (32)	1	6	0	7	0
Hommel et al99	2016	Netherlands	Quality indicators	Perioperative diabetes care	D	7/36 (19)	1	6	0	7	0
Smits et al100	2016	Netherlands	Prescribing quality indicators	Chronic kidney disease	D	16/16 (100)	0	16	0	16	0
Teichert et al23	2016	Netherlands	Quality indicators	Pharmaceutical care in community pharmacies	Pr, T	67/67 (100)	21	43	3	23	44
Smits et al101	2017	Netherlands	Prescribing quality indicators	Type 2 diabetes in primary care	D, T	20/20 (100)	0	20	0	20	0
Idänpään-Heikkilä et al152	2006	OECD countries	Quality indicators	Cardiac care	D	8/17 (47)	0	8	0	8	0
Petek et al132	2012	Slovenia	Quality indicators	Cardiovascular disease prevention for primary care	D, T	14/88 (16)	0	14	0	12	2
Minaya-Muñoz et al133	2013	Spain	Quality measures	Lateral epicondylalgia	D	2/12 (17)	0	2	0	2	0
Calvet et al134	2014	Spain	Quality indicators	Inflammatory bowel disease	D	14/56 (25)	0	14	0	14	0
Ruiz-Canela-Cáceres et al135	2015	Spain	Quality indicators	Childhood asthma in primary care	D, P, T	2/7 (29)	0	2	0	2	0
Soria-Aledo et al136	2016	Spain	Indicators	General surgery	T	2/13 (15)	0	2	0	2	0
Bianchi et al137	2013	Switzerland	Quality indicators	Colorectal cancer	D	7/27 (26)	0	6	1	7	0
Chung et al138	2008	Taiwan	Performance measures	Breast cancer	D	2/15 (13)	0	2	0	2	0
Chung et al139	2010	Taiwan	Core measures	Colorectal cancer	D	3/17 (18)	0	3	0	3	0
Cantrill et al140	1998	UK	Indicators	Long-term prescribing in general practice	G, T	9/9 (100)	0	9	0	1	8
Morris and Cantrill141	2003	UK	Quality indicators	Preventing drug-related morbidity in primary care	G, T	24/24 (100)	0	24	0	24	0
Steel et al142	2004	UK	Quality indicators	Healthcare of older adults in primary and secondary care	P	40/102 (39)	0	40	0	37	3
Tully et al143	2005	UK	Indicators	Long term prescribing in primary and secondary care	G, T	14/14 (100)	0	14	0	0	14
Gill et al144	2014	UK	Quality indicators	Child healthcare in general practice	P	10/35 (29)	0	10	0	9	1
Spencer et al145	2014	UK	Prescribing safety indicators	Safety of prescribing in general practice	G, T	56/56 (100)	0	56	0	56	0
NICE156 3 §	2016	UK	NICE indicators	General practice	T	33/125 (26)	0	30	3	33	0
Hadorn et al25	1996	USA	Review criteria	Heart failure	D	4/8 (50)	0	4	0	3	1
Asch et al26	2001	USA	Quality of care indicators	Women with hypertension	D, P	6/13 (46)	0	6	0	5	1
Mikuls et al27	2004	USA	Quality of care indicators	Gout	D	9/10 (90)	0	9	0	9	0
Saliba et al28	2004	USA	Quality indicators	Nursing home residents	T	54/114 (47)	1	53	0	54	0
Krumholz et al29	2006	USA	Performance measures	ST-elevation and non-ST-elevation myocardial infarction	D	8/11 (73)	0	8	0	8	0
McGory et al30	2006	USA	Quality indicators	Patients undergoing colorectal cancer surgery	D	20/92 (22)	0	20	0	19	1
Mularski et al31	2006	USA	Quality measures	Palliative care	T	1/18 (6)	0	1	0	1	0
Mangione-Smith et al32	2007	USA	Quality indicators	Paediatric outpatients	P, T	58/175 (33)	0	58	0	58	0
Smith et al33	2007	USA	Quality indicators	Home-based primary care	T	71/200 (36)	1	70	0	59	12
Wenger et al34	2007	USA	Quality indicators	Vulnerable elders	P	146/392 (37)	0	146	0	129	17
Estes et al35	2008	USA	Performance measures	Outpatient adults with non-valvular atrial fibrillation or atrial flutter	D, P, T	2/3 (67)	0	2	0	2	0
Bilimoria et al36	2009	USA	Quality indicators	Pancreatic cancer	D	3/43 (7)	1	2	0	3	0
Lorenz et al37	2009	USA	Quality measures	Supportive cancer care	D	36/92 (39)	0	36	0	35	1
McGory et al38	2009	USA	Quality indicators	Perioperative care for elderly surgical patients	P, T	17/91 (19)	0	17	0	13	4
Yazdany et al39	2009	USA	Quality indicators	Systemic lupus erythematosus	D	14/20 (70)	0	14	0	14	0
Cheng et al40	2010	USA	Quality indicators	Multiple sclerosis	D	19/76 (25)	0	19	0	17	2
Kanwal et al41	2010	USA	Quality indicators	Nonvariceal upper gastrointestinal haemorrhage	D	6/26 (23)	0	6	0	6	0
Kanwal et al42	2010	USA	Quality indicators	Cirrhosis	D	12/41 (29)	0	12	0	12	0
Schenck et al43	2010	USA	Quality measures	Hospice and palliative care	T	7/34 (21)	0	7	0	7	0
Khanna et al44	2011	USA	Quality indicators	Systemic sclerosis	D	10/32 (31)	0	10	0	10	0
SooHoo et al45	2011	USA	Quality of care indicators	Patients undergoing total hip or total knee replacement	D	8/68 (12)	0	8	0	5	3
Wang et al46	2011	USA	Quality of care indicators	Children with sickle cell disease	D, P	13/41 (32)	0	13	0	13	0
Anger et al47	2013	USA	Quality of care indicators	Women with urinary incontinence	D, P	7/27 (26)	0	7	0	7	0
Jackson et al48	2013	USA	Quality indicators	Colorectal cancer	D	11/34 (32)	0	11	0	11	0
Melmed et al49	2013	USA	Quality indicators	Inflammatory bowel disease	D	8/21 (38)	0	5	3	8	0
Wang et al50	2013	USA	Quality of care indicators	Infantile spasms	D, P	10/21 (48)	0	10	0	10	0
Yadlapati et al51	2015	USA	Quality measures	Gastro-oesophageal reflux disease	D	15/25 (60)	0	15	0	15	0
Faro et al52	2016	USA	Quality indicators	Follow-up care for individuals with positive screens for sickle cell disease and trait	D	2/9 (22)	0	2	0	2	0
Vila et al53	2016	USA	Quality measures	Adult cochlear implant centres	D, P, T	2/8 (25)	0	1	1	2	0
Yazdany et al54	2016	USA	Quality measures	Rheumatoid arthritis	D	1/4 (25)	0	1	0	1	0
Chowdhury et al55	2017	USA	Quality metrics	Adult congenital heart disease and paediatric cardiology care	D, P	5/27 (19)	0	4	1	5	0
Hepner et al56	2017	USA	Quality measures	Unhealthy alcohol use	D	3/25 (12)	0	3	0	3	0
Ingraham et al57	2017	USA	Quality indicators	Emergency general surgery care	T	3/25 (12)	2	1	0	2	1
Mangione-Smith et al58	2017	USA	Quality indicators	Hospital-based care for common paediatric respiratory illnesses	D, P, T	43/76 (57)	0	43	0	43	0
Odetola et al59	2017	USA	Quality measures	Inhospital care of paediatric sepsis syndrome	D, P, T	3/7 (43)	1	2	0	3	0
NQMC157 §	2018	USA	Quality measures	Various care	All	412/2525 (16)	0	378	34	376	36
Parast et al60	2018	USA	Quality measures	Hospital-based care for suicidal youth	D, P, T	4/4 (100)	0	4	0	2	2
Total						2431/7750 (31)	60	2289	62	2184	247

*Problem type: D, disease based; G, generic; O, organisation based; P, patient based; Pr, profession based; T, treatment modality based.

†Donabedian’s framework: S, structure; P, process; O, outcome.

‡QI type: Me, medicine class specific; Ge, general medication.

§Government agency website.

ACSQHC, The Australian Commission on Safety and Quality in Healthcare; NICE, National Institute for Health and Clinical Excellence; NQMC, National Quality Measures Clearinghouse; QI, quality indicator; RUM, responsible use of medicines.

Of 2431 QIs, 247 QIs (10%) were general medication indicators, and 2184 QIs (90%) were medicine class specific indicators. Some of the 2184 QIs represented more than one ATC code resulting in 2613 first level of ATC classifications. Of these, the most number of QIs covered medicines for nervous system (N, n=407, 16%), followed by the anti-infectives for systemic use (J, n=397, 15%), cardiovascular system (C, n=364, 14%) and blood and blood forming organs (B, n=345, 13%) (figure 3). Dermatological medicines (D) were covered by the least number of QIs (n=19, 0.7%) aside from antiparastic products, insecticides and repellents (P, n=7, 0.3%).

Figure 3

The number of QIs by first-level ATC code. ATC, Anatomical Therapeutic Chemical; QIs, quality indicators.

The distribution of the QIs across the second level of ATC code and c-DRPs classification system is presented in table 2. General medication indicators were only classified using c-DRPs category. Because some QIs represented more than one ATC code and/or c-DRPs category, the total number of the QIs contained within each cell of the matrix was 3666. Of these, when investigating the number of QIs in each c-DRPs category, the largest number of QIs for ‘drug selection’ pertained to antibacterials for systemic use (J01, 176 of 2117, 8%), followed by antithrombotic agents (B01, 172 of 2117, 8%). Antithrombotic agents (B01) also contributed the largest number of QIs for ‘dose selection’ (20 of 142, 14%) and the ‘drug use process’ (52 of 439, 12%) and ‘monitoring’ (52 of 574, 9%). Likewise, the most number of QIs for ‘treatment duration’ (13 of 85, 15%) pertained to psychoanaleptics (N06).

Table 2

Distribution of QIs for RUM by the ATC code (rows) and the c-DRPs category (columns)*

ATC code	1. Drug selection	2. Drug form	3. Dose selection	4. Treatment duration	5. Drug use process	6. Logistics	7. Monitoring	8. Adverse drug reaction	9. Other	Total, n (%)
A: alimentary tract and metabolism
A01: Stomatological preparations	7				5					12 (0.3)
A02: Drugs for acid related disorders	37		3	2			8		1	51 (1.4)
A03: Drugs for functional gastrointestinal disorders	35				3		4			42 (1.1)
A04: Antiemetics and antinauseants	23				4		5			32 (0.9)
A06: Drugs for constipation	20			1	3		2			26 (0.7)
A07: Antidiarrheals, intestinal antiinflammatory/antiinfective agents	14		1	2	1	1	2			21 (0.6)
A08: Antiobesity preparations, excl. diet products							1			1 (0)
A10: Drugs used in diabetes	40	1	4		19	3	19		2	88 (2.4)
A11: Vitamins	28		1		1	1	1			32 (0.9)
A12: Mineral supplements	24		1		1	5	3			34 (0.9)
B: Blood and blood forming organs
B01: Antithrombotic agents	172		20	8	52	6	52	1	9	320 (8.7)
B02: Antihemorrhagics	2					1				3 (0.1)
B03: Antianemic preparations	9		1		1	3	2		1	17 (0.5)
B05: Blood substitutes and perfusion solutions	33		1		9	2	4	1	5	55 (1.5)
C: Cardiovascular system
C01: Cardiac therapy	30	1	4		4	4	15			58 (1.6)
C02: Antihypertensives	50		5		1	2	10			68 (1.9)
C03: Diuretics	57		6		2	3	23		1	92 (2.5)
C04: Peripheral vasodilators	5				1					6 (0.2)
C05: Vasoprotectives	1									1 (0)
C07: Beta blocking agents	108		5	5	11		10			139 (3.8)
C08: Calcium channel blockers	48		6		5		10			69 (1.9)
C09: Agents acting on the renin-angiotensin system	128		7	2	7	1	34			179 (4.9)
C10: Lipid modifying agents	53		2	1	7		8			71 (1.9)
D: Dermatologicals
D01: Antifungals for dermatological use	2									2 (0.1)
D02: Emollients and protectives	3									3 (0.1)
D06: Antibiotics and chemotherapeutics for dermatological use	4									4 (0.1)
D07: Corticosteroids, dermatological preparations	4			1						5 (0.1)
D08: Antiseptics and disinfectants	1									1 (0)
D10: Anti-acne preparations	1						3			4 (0.1)
D11: Other dermatological preparations							1			1 (0)
G: Genito urinary system and sex hormones
G01: Gynecological antiinfectives and antiseptics	4		1	2			1			8 (0.2)
G02: Other gynecologicals	2									2 (0.1)
G03: Sex hormones and modulators of the genital system	29				9		8			46 (1.3)
G04: Urologicals	16						5		2	23 (0.6)
H: Systemic hormonal preparations, excl. sex hormones and insulins
H01: Pituitary and hypothalamic hormones and analogues	8		1		1	2	5			17 (0.5)
H02: Corticosteroids for systemic use	53		5	3	5		9		3	78 (2.1)
H03: Thyroid therapy	2				2	2	5			11 (0.3)
H05: Calcium homeostasis	14						3			17 (0.5)
J: Antiinfectives for systemic use
J01: Antibacterials for systemic use	176	7	9	12	38	8	22		8	280 (7.6)
J02: Antimycotics for systemic use	2		1							3 (0.1)
J04: Antimycobacterials	2					1				3 (0.1)
J05: Antivirals for systemic use	12	2		2	6	4	3			29 (0.8)
J07: Vaccines	94				10	1	5		6	116 (3.2)
L: Antineoplastic and immunomodulating agents
L01: Antineoplastic agents	94		3	5	35	3	45	3	11	199 (5.4)
L02: Endocrine therapy	15			1	4		4			24 (0.7)
L03: Immunostimulants	4						2			6 (0.2)
L04: Immunosuppressants	23		4	1	6	2	12			48 (1.3)
M: Musculoskeletal system
M01: Antiinflammatory and antirheumatic products	53		5	6	14	1	18	2	1	100 (2.7)
M02: Topical products for joint and muscular pain,	45		5	5	13	1	18	2	1	90 (2.5)
M03: Muscle relaxants	20			1	1		3			25 (0.7)
M04: Antigout preparations	10		3				3			16 (0.4)
M05: Drugs for treatment of bone diseases	27		1		1		3			32 (0.9)
N: Nervous system
N01: Anesthetics	12				3	2		1	1	19 (0.5)
N02: Analgesics	82	1	9	1	24	5	24	3	3	152 (4.1)
N03: Antiepileptics	19		2		8	5	5		2	41 (1.1)
N04: anti-Parkinson drugs	30		1		4		5			40 (1.1)
N05: Psycholeptics	72		6	8	12	5	28	2	3	136 (3.7)
N06: Psychoanaleptics	73		8	13	9	2	29		2	136 (3.7)
N07: Other nervous system drugs	11				12		2			25 (0.7)
P: Antiparasitic products, insecticides and repellents (p01: antiprotozoals)		2	1	2		1	3
R: Respiratory system
R01: Nasal preparations	5			2						7 (0.2)
R03: Drugs for obstructive airway diseases	69	3	3		37	4	13		2	131 (3.6)
R05: Cough and cold preparations	6						1			7 (0.2)
R06: Antihistamines for systemic use	31					2	3			36 (1)
S: Sensory organs
S01: Ophthalmologicals	23				5	3	2			33 (0.9)
S02: Otologicals	2									2 (0.1)
V: Various
V03: All other therapeutic products	16	1	1		1		3		1	23 (0.6)
V06: General nutrients	5	1	2	1	1					10 (0.3)
V08: Contrast media	1									1 (0)
Other: general medication indicators	14	2	3		40	73	68	4	44	248 (6.8)
Total, n (%)	2117 (57.7)	20 (0.5)	142 (3.9)	85 (2.3)	439 (12)	161 (4.4)	574 (15.7)	19 (0.5)	109 (3)	3666 (100)

*Basger BJ, Moles RJ, Chen TF. Development of an Aggregated System for Classifying Causes of Drug-Related Problems. Ann Pharmacother 2015;49:405–18.

ATC, Anatomical Therapeutic Chemical; c-DRPs, causes of drug-related problems; QIs, quality indicators.

With regard to the c-DRPs classification system, the most common c-DRPs pertained to ‘drug selection’ (n=2117, 58%), followed by ‘monitoring’ (n=574, 16%) and the ‘drug use process’ (n=439, 12%). The remaining six c-DRPs categories accounted for only 14% of the QIs. Interestingly, only QIs for analgesics (N02) covered all nine c-DRPs categories. In terms of general medication indicators, the largest number of QIs covered ‘Logistics’ (n=73, 29%) among the c-DRPs category, which mainly focus on medication reconciliation problems during transitions of care, such as hospital admission and discharge.

A complete list of 2431 QIs is available in online supplementary table S7.

Discussion

The RUM is important for almost every healthcare setting in every country across the globe. Knowledge of whether medicines are being used in an optimal manner therefore presents a significant international challenge. In this systematic review, we identified 2431 QIs evaluating RUM and classified them using multiple frameworks. The large number of QIs reflects the multidimensional components of RUM and the different perspectives of multidisciplinary stakeholders involved in the RUM. The QI list presented in this review can be used as a comprehensive database and reference for existing content validated QIs pertaining to RUM. All stakeholders involved in quality assurance for RUM, for example, healthcare professionals, researchers and decision makers, can select QIs from the multicategorised QI list for their own purpose. Since healthcare systems and medication guidelines may vary between countries when using the QIs at the local setting, it is important for users to critically review the QIs for their acceptability, feasibility of acquiring necessary data, reliability, sensitivity to change, work load and validity.8 14

The vast majority of the QIs for RUM identified were intended to be used in only a few high-income countries. Low-income and middle-income countries, however, are estimated to have similar rates of medication-related adverse events, and the impact has been reported to be about twice as much in terms of the number of years of healthy life lost.4 Since feasibility of data collection for calculating QI scores in low-income settings remain a concern,151 further efforts for improving the data collection method might need to be made. We found that even though the role of all measurement tools (ie, QIs) relevant to RUM have the goal of quality improvement, the terminology used to describe QIs varied significantly. About 20 name variations were found, which reflects the absence of a universally accepted definition for such tools. For example, Campbell et al8 distinguished QIs from performance indicators, arguing that QIs infer a judgement about the quality of care provided, while performance indicators are statistical devices for monitoring care provided to populations without any necessary inference about quality. However, we found that these terms, ‘quality’ and ‘performance’, were used interchangeably. Hence, further research for standardising the definition that distinguishes these measurement tools is warranted.

We also found a significant gap in terms of the problem type (eg, ‘disease-based problems’ (43%), ‘treatment modality-based problems’ (27%) and ‘profession-based problems’ (5%)). Since RUM is facilitated by collaboration in multidisciplinary teams, all healthcare professionals involved in medication treatment should take responsibility for quality assurance, regardless of diseases, care settings and professions. When using Donabedian’s framework, about 94% of the identified QIs related to processes of care. This could be because processes of care are easier to measure, and because process indicators can provide interpretable feedback about care provided.158 In contrast, there was a paucity of outcome indicators. This may be because multiple factors influence health outcomes, many of which are outside the control of individual healthcare professionals. In addition, the difficulty of obtaining sufficient information for assessing outcomes, requiring the linkage of multiple data sources, could be another reason of the limited number of outcome indicators. For outcome indicators to become more useful, multiple confounders such as patient demographic characteristics, and severity of illness, may need to be considered.159 Similarly, there was a low proportion of structural indicators. This may be because they are not sufficiently sensitive for monitoring ongoing performance and they have traditionally been used to monitor standards of healthcare facilities, not RUM.160 It is noteworthy that there is no set requirement for equal proportions of structural, process and outcome indicators in quality measurement. Instead, it is important to recognise the interconnectedness of these measures. For example, high structure indicator scores increase the likelihood of good process indicator scores, which in turn, may lead to higher outcome indicator scores.161 Further research is needed to investigate the associations between the identified QIs in each framework within healthcare settings.

We found large differences in the degree to which c-DRPs categories were covered by the identified QIs. Not surprisingly, ‘Drug selection’ accounted for more than half of the QIs, as choosing an inappropriate drug is the main cause of DRPs.3 162 Since focusing on limited c-DRPs categories may divert attention and resources away from other factors contributing to DRPs,163 164 users of QIs should be aware of what c-DRPs categories are not being measured. Like Donabedian’s framework, we do not expect that QIs should be evenly distributed across each of the c-DRPs categories or ATC groups. We do, however, expect that there will be greater QIs in areas of greatest need. These clinical areas may include common areas of practice suspected to be associated with inappropriate use of medicines and significant economic burden (eg, over use of antibiotics for upper respiratory tract infection and overuse of opioid analgesics). Use of QIs in these areas may fill the evidence–practice gaps and minimise subsequent DRPs.165 166

QIs for antithrombotic agents (B01) accounted for the larger proportion of QIs targeting ‘drug selection’, ‘dose selection’, ‘drug use process’ and ‘monitoring’ in c-DRPs categories. This may be explained by the fact that the majority of preventable drug-related admissions have been attributed to antiplatelets and anticoagulants, which have narrow therapeutic indices and high risk of overdose or toxicity,3 and also the fact that medication adherence to long-term antithrombotic therapy remains challenging.167 Likewise, QIs for psychoanaleptics (N06) accounted for the largest part of QIs targeting ‘treatment duration’. Since medication adherence is an ongoing challenge for consumers being treated for depression with antidepressant therapy, it seems appropriate that a relatively large number of QIs have been developed in these categories. In contrast, there were few QIs for some ATC groups, such as dermatological medicines. This has previously been reported in the literature for QIs as a whole, when comparing the scope of dermatology QIs to other medical specialty areas (eg, internal medicine, paediatrics or cardiology).168 This may be because dermatological medicines, especially topical agents, are relatively less harmful and less expensive. Since irrational topical dermatological medication can occur because of drug selection error and patients’ misunderstanding, prescribing, dispensing and administration errors,169 more QIs targeting the wide range of c-DRPs categories may need to be developed for ensuring RUM. Furthermore, when focusing on general medication indicators, QIs largely focused on ‘logistic’ issues such as medication reconciliation at transition points and unavailability of medicines in the c-DRPs category. This differed from medicine class specific QIs, which mainly focused on ‘drug selection’ issues. These differences underscore the importance of the combined use of general medication QIs and medicine class specific QIs for the comprehensive evaluation of RUM.

In terms of interpretation of direction of QI scores, we found different methods of scoring: those for evaluating whether necessary or appropriate care was provided and those for evaluating whether unnecessary or inappropriate care was provided. Therefore, care in the interpretation of QI scores is recommended as they have different interpretations based on positively or negatively worded indicators. We also found there were many similar QIs, with only minor differences in wording or definition. These slight differences may be attributed to feasibility of acquiring the data, differences in national guidelines, targeted populations or healthcare systems between locations or countries. However, these minor differences could adversely affect comparability of QI scores and could decrease motivation of healthcare professionals to participate in initiatives if they feel they are being asked the same indicator questions repeatedly. This may be overcome by undertaking a mapping exercise of the QIs identified in our review, with the potential of aggregating some of the QIs. QI is one of the measurement tools to evaluate quality of care at the healthcare facility or group level. QI scores do not directly represent quality of individual patient care but are used as ‘flags’ or ‘alerts’ to potential problems that require further analysis.170 In addition, actions required for quality improvement vary from the level of individual patients, healthcare providers, facilities or healthcare system. Therefore, a multidisciplinary, multilevel quality improvement initiative is needed for comprehensive quality assurance.

Strengths and limitations

Our review has some notable strengths. This is the first comprehensive review of QIs pertaining to RUM without restriction of disease categories and care settings. In order to do this, a comprehensive literature search was undertaken across multiple databases and websites. Moreover, the classification of QIs was based on multiple frameworks (eg, Donabedian and c-DRPs) for maximum understanding and profiling of the included QIs. The rich dataset of identified QIs can be used as a starting point for healthcare professionals, researchers, decision makers and others, for identifying and selecting existing QIs for the evaluation of RUM. We also identified significant gaps in current quality measurements in each framework, underscoring the need for further QI development in some areas. We do however acknowledge that our approach has some limitations. First, we only included QIs that were developed using consensus methods and excluded QIs if consensus results for QI development were unclear. Therefore, we might have excluded valid indicators during the screening process. Second, although 5% of this review processes were verified by multiple authors, our mapping exercise into the classification system may be viewed as subjective. Third, we identified QIs developed using consensus methods to ensure content validity; however, the methodological rigour of each study was not assessed. Therefore, the quality of the content validity of identified QIs was not reported.

Conclusions

Overall, by using multiple frameworks, we were able to identify and classify 2431 QIs covering different constructs of RUM. However, this review also pointed to some significant gaps in current quality measurements, making it difficult for healthcare systems to fully assess whether RUM has been achieved or not. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. All stakeholders involved in quality assurance for RUM can select QIs from the multicategorised QI list for their own purpose. In order to more effectively evaluate the extent to which RUM has been achieved, further development and validation of QIs may be required.