Proposal of quality indicators for cardiac rehabilitation after acute coronary syndrome in Japan: a modified Delphi method and practice test.

OBJECTIVES: Cardiac rehabilitation is underused and its quality in practice is unclear. A quality indicator is a measurable element of clinical practice performance. This study aimed to propose a set of quality indicators for cardiac rehabilitation following an acute coronary event in the Japanese population and conduct a small-size practice test to confirm feasibility and applicability of the indicators in real-world clinical practice. DESIGN AND
SETTING: This study used a modified Delphi technique (the RAND/UCLA appropriateness method), a consensus method which involves an evidence review, a face-to-face multidisciplinary panel meeting and repeated anonymous rating. Evidence to be reviewed included clinical practice guidelines available in English or Japanese and existing quality indicators. Performance of each indicator was assessed retrospectively using medical records at a university hospital in Japan. PARTICIPANTS: 10 professionals in cardiac rehabilitation for the consensus panel.
RESULTS: In the literature review, 23 clinical practice guidelines and 16 existing indicators were identified to generate potential indicators. Through the consensus-building process, a total of 30 indicators were assessed and finally 13 indicators were accepted. The practice test (n=39) revealed that 74% of patients underwent cardiac rehabilitation. Median performance of process measures was 93% (IQR 46-100%). 'Communication with the doctor who referred the patient to cardiac rehabilitation' and 'continuous participation in cardiac rehabilitation' had low performance (32% and 38%, respectively).
CONCLUSIONS: A modified Delphi technique identified a comprehensive set of quality indicators for cardiac rehabilitation. The single-site, small-size practice test confirmed that most of the proposed indicators were measurable in real-world clinical practice. However, some clinical processes which are not covered by national health insurance in Japan had low performance. Further studies will be needed to clarify and improve the quality of care in cardiac rehabilitation. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

This is the first study to assess the utility of a whole set of quality indicators for cardiac rehabilitation that were established with the RAND/UCLA appropriateness method.

A guideline-based modified Delphi technique was used and medical records were reviewed as a practice test before implementing indicators in real-world clinical practice.

The composition of our panel members may lead to a biased selection of indicators.

The single-site, small-size practice test may limit to generalise the performance of indicators.

Introduction

Acute coronary syndrome is a leading cause of death and a major social burden in the world.1 Developments in coronary interventions, surgical procedures, pharmacological treatments and lifestyle modifications (eg, smoking cessation) have improved prognosis, prevented recurrence and reduced mortality in patients who experience acute coronary events.2 3 In urban and/or suburban areas of Japan, however, the incidence of myocardial infarction and sudden cardiac death among men has increased in the past few decades, although the trend in rural areas is unclear.4–6

Previous studies have demonstrated the effectiveness of comprehensive cardiac rehabilitation (CR) on the survival and quality of life of patients with coronary heart disease.7 Yet CR referral rates vary according to characteristics of patients, specialty units and hospitals.8–14 Moreover, little is known about the quality of programmes in real-world clinical settings.

Quality indicators are measurable elements of practice performance for which there is evidence or consensus. Process indicators, in particular, are used to express the proportion of patients who receive appropriate care. They can highlight and reveal quality issues, which in turn enable the formulation of solutions for improvement.15–17 The use of quality indicators has increased in Japan, particularly in cancer care settings.18

Quality indicators for cardiovascular disease prevention and CR have been developed in European countries, the USA and Canada, but not in Asia including Japan.19–23 Additionally, a practice test prior to real usage is needed when developing quality indicators.24 25 However, previous practice tests examined only a part of the indicator set in the CR field.26 27

This study aims to (1) propose quality indicators for CR following an acute coronary event (eg, acute myocardial infarction and unstable angina), as these diseases are the most common indications for CR, (2) conduct a small-size, pilot practice test with the whole set of the proposed indicators to confirm the feasibility and applicability before using them in real-world clinical practice and (3) describe a detailed process of developing quality indicators for applicability to other clinical circumstances.

Methods

Indicator development

Overview of the development process

This study was conducted by the task force on evidence-based healthcare and clinical practice guidelines under the Ministry of Health, Welfare and Labour in Japan. The aim was to propose the common method to develop quality indicators.

A modified Delphi technique (the RAND Corporation (RAND)/University of California, Los Angeles (UCLA) appropriateness method), which has been widely used to develop healthcare quality indicators, was used.28 The method integrates an evidence review, a face-to-face multidisciplinary panel meeting and repeated anonymous rating for consensus building.

Systematic search of evidence

Kötter et al24 previously reported that methods for quality indicator development based on clinical guidelines are increasing and may help in the efficient gathering of evidence when considering indicators. On this basis, we searched existing clinical practice guidelines available in English or Japanese and quality indicators related to CR after an acute coronary event developed in countries other than Japan. Specifically, we searched electronic databases with the search terms ‘cardiovascular disease’, ‘cardiovascular system’, ‘cardiology’, ‘cardiac’, ‘cardiovascular’ and ‘rehabilitation’ in August 2011. The following databases were used: one quality indicator database (Agency for Healthcare Research and Quality (AHRQ) National Quality Measures Clearinghouse); nine guideline databases (AHRQ National Guideline Clearinghouse, National Institute for Health and Care Excellence (NICE) Find Guidance, National Health Service (NHS) Evidence, Minds, PEDro, Guidelines International Network, Australian Government National Health Medical Research Council, Canadian Medical Association and British Intercollegiate Guidelines Network) and three medical literature databases (MEDLINE, CINAHL and ICHUSHI). The Japanese Association of Cardiac Rehabilitation website and the Japanese Circulation Society website were also reviewed for Japanese guidelines on CR. The search was limited to the English or Japanese language. In 2011, we searched literature published from April 2006 to March 2011, because clinical guidelines are generally outdated after 5 years.29 30 Two appraisers (SO and NK) evaluated the quality of the selected guidelines with the AGREE II instrument, which is the established tool for this purpose.31 For each guideline, overall assessments using AGREE II by two appraisers were averaged.

In order to generate indicators to be evaluated in a subsequent consensus panel, clinical practices which were strongly recommended in the selected guidelines were extracted. They were merged with existing quality indicators and applied to a structured format of the modified American College of Cardiology (ACC)/American Heart Association (AHA) methodology.32 One researcher (SO) assembled the indicators and was supervised by two experts: TN, an epidemiologist experienced in the development of a variety of Japanese clinical practice guidelines and also the chair of the task force on evidence-based healthcare and clinical practice guidelines, and KU, a cardiologist familiar with CR and evidence-based medicine.

Multidisciplinary panel

Panel members responsible for consensus development were selected from registered instructors of CR licensed by the Japanese Association of Cardiac Rehabilitation. Licensure status requires individuals to have practised CR for at least 1 year, submitted 10 case reports and passed an authorised examination. To assemble the multidisciplinary group, we attempted to select more than one person from each profession related to CR (cardiologists, nurses, physical therapists, clinical laboratory technicians, registered dietitians, health fitness programmers and clinical psychologists). The sampling strategy was non-random selection aiming to seek participants who would be informative, with recommendations by KU and approval by SO and TN. KU was a member of the committee on Japanese guidelines for CR and appointed as president of the Japan Association of Cardiac Rehabilitation in 2014. Each panel member signed a consent form for the present study. The indicator development part of this study was approved by the Kyoto University Graduate School and Faculty of Medicine, Ethics Committee.

Consensus building

On the basis of the modified Delphi technique, panel members formed a consensus regarding indicators in three rounds. In round 1, members individually evaluated indicators using a nine-point scale postal questionnaire. In round 2, they reconvened for a 1-day face-to-face meeting to discuss, revise and individually evaluate potential indicators, and then suggest additional ones at a later meeting. In round 3, they discussed additional indicators by email instead of a face-to-face meeting and evaluated them with the same postal questionnaire as above.

Round 1

A set of documents that described the quality indicators was distributed to 10 panel members. Members were then given 2 weeks to individually rate each indicator with a nine-point scale questionnaire, adapted from the nine domains suggested by Spertus et al:32 evidence-based, interpretable, actionable, denominator, numerator, validity, reliability, feasibility and overall assessment. We planned more than one reminder with email or telephone for non-responders.

Round 2

A 1-day face-to-face panel meeting was held. Panel members anonymously shared their results from round 1 and discussed each indicator. When the panel members felt the necessity, they revised the elements of each potential indicator considering a member's suggestion. Any disagreements were resolved through discussion. Finally, the panel rated the revised indicators individually, using the same questionnaire from round 1. At the end of the meeting, we asked members to suggest additional indicators that might be important for measuring the quality of CR services in Japan that had not been examined by the panel prior to the meeting.

Round 3

Additional indicators were developed from suggestions during the panel meeting (round 2). A document describing the new indicators was sent to panel members electronically, asking for any modifications. The indicators were then revised according to the suggestions and sent to panel members by mail to rate using the same questionnaire from round 1.

Selection criteria

Quality indicators were adopted according to the following criteria: the median individual rating during round 2 or 3 was >7, and the number of panel members who gave a rating <3 was two or fewer.

Pilot practice test for feasibility and adaptability

Study patients

Selected patients were those who (1) experienced an acute myocardial infarction or unstable angina and (2) were admitted to Kyoto University Hospital, which provides acute-to-recovery care in Japan, between 1 January 2013 and 30 June 2013. On the basis of the Ethical Guidelines for Epidemiological Research established by the Ministry of Education, Culture, Sports, Science and Technology and Ministry of Health, Labour and Welfare in Japan, we prepared a protocol mentioning that the investigators disclose requisite information on the study and, consequently, the protocol for the practice test was approved by the university ethics committee.

Data collection

Data were collected from Kyoto University's electronic medical records in January 2014. Records regarding eating habits (QI-2), coronary risk factors (QI-3), psychological responses (QI-4), tobacco and alcohol (QI-5), exercise capacity (QI-6), definition of the programme end period (QI-8), and work and leisure (QI-10) were available in the implementation planning sheets for CR per national insurance regulations. Professionals assessed a suitable period with regard to QI-8 for every patient in view of their clinical situation and recorded it in the planning sheet as routine in the hospital. Records on education regarding the importance of adherence to prescription medication (QI-12) were available in pharmacist instruction reports, also per national insurance regulations. SO and NK assessed performance of the indicators based on each patient's records. Disagreements were resolved by discussion.

Data analysis

Patients with acute myocardial infarction and unstable angina were selected consecutively during the study period from Kyoto University Hospital via coronary catheterisation records for measurement with the indicators. Percentage scores were obtained for each indicator as follows: the number of times the indicator was met/the number of participants (excluding those with obvious reasons for not implementing the process as defined by the indicator)×100. Medians of indicator scores were also computed as an overall quality score of the programme.

Disclosure and update

The adopted indicators were disclosed and externally reviewed at a symposium during the annual meeting of the Japanese Association of Cardiac Rehabilitation, of which KU was the president in July 2014. Considering suggestions from external experts, an indicator regarding referral to CR was updated in October 2015 based on an online panel discussion and the nine-point scale assessment.

Results

Quality indicators

The literature review identified 38 quality indicators from the AHRQ National Quality Measures Clearinghouse, of which 16 were selected according to the criteria (figure 1). Searches of guideline databases and medical literature databases identified 894 articles (103 from the AHRQ National Guideline Clearinghouse, 121 from NICE Find Guidance, 487 from NHS Evidence, 2 from Minds, 11 from PEDro, 25 from the Guidelines International Network, 6 from the Australian Government National Health Medical Research Council, 9 from the Canadian Medical Association, 22 from the British Intercollegiate Guidelines Network, 74 from MEDLINE, 24 from CINAHL and 10 from ICHUSHI). Seven additional guidelines were identified from the Japanese academic society websites. In total, 23 guidelines met our eligibility criteria. The average of overall quality scores based on AGREE II was 5.0 out of a possible 7 (minimum to maximum 3.5–6.0). A total of 27 potential indicators created from the aforementioned 16 existing indicators and 23 guidelines were included for panel assessment.33–55

Figure 1

Flow chart of the literature search. *Agency for Healthcare Research and Quality National Quality Measures Clearinghouse. †The Japanese Association of Cardiac Rehabilitation website and the Japanese Circulation Society website.

Ten Japanese clinicians in CR were invited to the panel, including two cardiologists, two nurses, two physical therapists, one clinical laboratory technician, one registered dietitian, one health fitness programmer and one clinical psychologist (a member of the committee on Japanese guidelines for CR). Finally, all of them participated in the consensus development process.

The round 2 ratings accepted six indicators (figures 2 and 3). Although an indicator regarding referral to CR from outpatient settings had been discarded in this round, it was later accepted in the update period as a modified indicator (QI-1, table 1), the rating of which was a median of 8, and minimum to maximum of 4–9 (figure 3).

Figure 2

Flow chart of quality indicator selection.

Figure 3

Rating distribution of candidate indicators. The X-axis indicates individual indicator evaluation by the panel (1: disagree, 9: agree). The Y-axis indicates the number of panel members who scored the indicator. *This candidate indicator did not gain consensus and was discarded in round 2. †This was an updated indicator that includes referral from outpatient settings as well as inpatient settings.

Table 1

Quality indicators and percentage scores for cardiac rehabilitation of inpatients with ischaemic heart disease

Domains	Indicators	Numerator/denominator	Performance (%)
Domain 1: referral to cardiac rehabilitation	QI-1: referral to cardiac rehabilitation	29/39	74
	QI-2: assessment and education regarding eating habits	29/29	100
Domain 2: modification of coronary risk factors	QI-3: assessment and education regarding coronary risk factors	29/29	100
	QI-4: assessment of psychological responses	28/29	97
	QI-5: assessment and education regarding tobacco and alcohol	29/29	100
Domain 3: exercise therapy	QI-6: prescribed exercise based on assessment of exercise capacity	20/27	74
	QI-7: reassessment of exercise capacity	11/24	46
Domain 4: completion of the cardiac rehabilitation programme	QI-8: definition of the programme end period	29/29	100
	QI-9: continued participation in the rehabilitation programme	8/21	38
Domain 5: return to social activity	QI-10: assessment and education regarding work and leisure	29/29	100
Domain 6: involvement of multidisciplinary team	QI-11: holding a multidisciplinary conference	14/29	48
Domain 7: management of prescription medication	QI-12: education on the importance of adherence to prescription medication	26/29	90
Domain 8: communication with other healthcare providers	QI-13: communication with a doctor who referred the patient to cardiac rehabilitation	8/25	32

The panel had a debate about the arrangement of indicators regarding coronary risk factors. The members understood the importance of assessment and education for each risk factor. However, at the same time, they concerned measurement burden in clinical practice due to a lot of indicators. They considered that practitioners, generally in Japan, assess coronary risk factors comprehensively and arrange education programmes for individual patients. Finally, the indicators were grouped as follows: (1) chronic disease as coronary risk factors and the preventive behaviour, (2) consumption of non-essential grocery items which increases coronary risk, such as tobacco and alcohol, and (3) psychological responses.

Of the 20 primary indicators, the panel reconstructed four indicators. The indicator ‘assessment and education regarding coronary risk factors’ was developed by combining the following 13 primary indicators: ‘hypertension assessment’, ‘education for patients with hypertension’, ‘target blood pressure goal achievement’, ‘assessment and education regarding blood lipids’, ‘reassessment of blood lipids’, ‘assessment and education regarding physical activity habits’, ‘intervention for physical activity habits’, ‘reassessment of physical daily activity’, ‘assessment of body weight’, ‘intervention for body weight control’, ‘assessment of diabetes’, ‘intervention for diabetes’, and ‘target glycated haemoglobin goal achievement’. Some panel members noted that clinicians tend to deal with these issues together rather than separately. As such, they agreed that combining these indicators would make it easier to perform high-quality measurements. The panel agreed that you judge to be met the criteria when there is a record regarding assessment and education on at least one of the coronary risk factor aforementioned or a record that a patient have none of the risks.

The two primary indicators, ‘assessment of depression’ and ‘intervention for a patient suspected of having depression’, gave rise to the indicator ‘assessment of psychological responses’. Patients with coronary disease often have psychological issues such as depression, anxiety or insomnia.20 56 Furthermore, the panel was of the opinion that patients in Japan rarely receive appropriate assessments, aside from interventions they receive for CR.

The indicator ‘assessment and education regarding tobacco and alcohol’ was generated by combining the following three primary indicators: ‘assessment of tobacco use’, ‘support of smoking cessation’ and ‘education on alcohol’. The panel agreed that combining these items would improve measurement quality in clinical settings, as clinicians typically deal with these issues together. It also agreed that you judge to be met the criteria when there is a record regarding assessment of tobacco and alcohol use and education on them if it is necessary.

The primary indicator counselling to return to work was reworded as ‘assessment and education regarding work and leisure’. The panel agreed that it was important for a patient requiring rehabilitation to enjoy leisure activities, in addition to returning to work.

In addition to the indicators discussed above, the panel suggested some indicators, including ‘definition of programme end period’ and ‘continued participation in rehabilitation programme’. These indicators were generated after some panel members emphasised the importance of completing the CR programme. Given the multidisciplinary nature of the rehabilitation team, the generation of another indicator (holding a multidisciplinary conference) was suggested.

In the end, a total of 13 quality indicators were proposed. These indicators spanned the following domains: (1) referral to CR, (2) modification of coronary risk factors, (3) exercise prescription, (4) completion of the CR programme, (5) return to social activity, (6) involvement of a multidisciplinary team, (7) management of prescription medication and (8) communication with other healthcare providers (table 1).

Performance in the pilot practice test

Of the 39 assessed patients (22 with acute myocardial infarction and 17 with unstable angina), 29 (74%) participated in a CR programme (QI-1, table 1). The median performance, based on QI-2 to QI-13, was 93% (IQR 46–100). Indicators for which data are collected for national insurance claims (QI-2, 3, 4, 5, 6, 8, 10 and 12) showed high performance (median 100%; minimum to maximum 74–100), whereas others, including QI-7, 9, 11 and 13, showed low performance (minimum to maximum, 32–48%).

Discussion

In this study, we proposed 13 quality indicators for assessing CR in patients who suffered from acute coronary syndrome in Japan. A widely used consensus approach, the modified Delphi method, with a Japanese multidisciplinary panel identified seven crucial domains for the quality measurement. On the basis of results of the single-site, small-size practice test, we found that most of the indicators proposed were measurable in real-world clinical practice and the performance for which data are collected for national insurance claims was high, whereas performance of indicators for which data are recorded voluntarily was low.

The structure of our proposed indicators is consistent with those reported in previous studies. A few reports have been published that deal with quality measures for CR in North America. ACC/AHA/American Association of Cardiovascular and Pulmonary Rehabilitation (AACVPR) provided a performance measure set that has (1) referral to CR and (2) performance in the programme as major parts.20 21 The Canadian Cardiovascular Society also originally developed quality indicators on CR, and then prioritised a list of five indicators, including (1) inpatients referred to CR; (2) wait times from referral to CR enrolment; (3) patient self-management education; (4) increase in exercise capacity and (5) emergency response strategy.22 27 In this study, our set of indicators reflects the importance of referral to CR and a comprehensive programme. Therefore, we consider these to be essential elements of quality indicators for CR. Additionally, our study includes a measurement for the completion of CR, which is common with a measure set of the USA. It is because completion of the prescribed course of the programme is a key to promoting patients' lifelong behaviour change and physiological adaptations from regular exercise.20

During the process of developing quality indicators, pilot practice tests prior to implementation are important for determining which indicators become established components because they evaluate validity, reliability and feasibility.24 Some studies in other medical fields (eg, neck tumours, diabetes, pneumonia) have reported that, among accepted indicators in the development process, ∼10–30% were not measurable or inapplicable.57 58 In the field of CR in the USA, Thomas et al26 revealed that reliability of the abstraction for the measure regarding referral was good to excellent. In addition, with respect to Canadian indicators, Grace et al27 assessed feasibility and concluded that the indicator assessment was acceptable for the CR programme. Unlike the previous studies, our practice test was small-sized but included all indicators proposed. Although the generalisability is limited, we found a possibility that some processes for which data were not recorded per healthcare insurance regulations have low performance. Therefore, the result suggests that incentives such as insurance coverage can improve performance in clinical practice and recording the process. Since Grace et al27 also reported that the information-tracking process was challenging when measuring quality, which may be a common issue in the field of CR across countries. Further studies will be needed to make quality measurements more appropriate and efficient.

Although panel nomination is one of the key elements of quality indicator development, many studies lack transparency in the process.24 To increase transparency, our study established the following explicit criteria: (1) the panel members must have an authorised licence from an academic society, (2) at least one member is selected from every relevant profession and (3) the panel involves some members who have experiences of CR guideline development. Finally, our panel included the people concerned with the committee on the clinical practice guidelines for CR jointly developed by the relevant academic societies in Japan (ie, Shunichi Ishihara and Shinji Sato who are respectively a member and collaborator of the committee).59

In contrast to previous reports from the USA and Canada, our study did not directly stand on the initiative of any academic society. However, there were opportunities for authorised experts of CR to express his or her views into the present quality indicators.

The adaptability of each quality indicator must be reviewed before implementation because healthcare systems and social circumstances differ. This is a key element in evidence-based clinical decision-making.60 61 We consider, however, disclosing the explicit process of guideline-based quality indicator development, which is a time-efficient and resource-saving approach,24 will be helpful for people attempting to develop similar quality indicators in other regions or different social levels.

Conclusion

Using an explicit and integrated approach based on evidence and the consensus of a multidisciplinary panel, we proposed 13 specific indicators to measure the quality of CR for patients who experienced acute coronary events in Japan. The practice test was small-sized but helpful to confirm the measurability of all indicators proposed, and suggested that incentives such as insurance coverage can improve performance in clinical practice and recording the process. Further studies will be needed to clarify the reasons for this, as well as to improve the quality of care in CR.