Defining quality outcomes for complex-care patients transitioning across the continuum using a structured panel process.

BACKGROUND: No standardised set of quality measures associated with transitioning complex-care patients across the various healthcare settings and home exists. In this context, a structured panel process was used to define quality measures for care transitions involving complex-care patients across healthcare settings.
METHODS: A modified Delphi consensus technique based on the RAND method was used to develop measures of quality care transitions across the continuum of care. Specific stages included a literature review, individual rating of each measure by each of the panelists (n=11), a face-to-face consensus meeting, and final ranking by the panelists.
RESULTS: The literature review produced an initial set of 119 measures. To advance to rounds 1 and 2, an aggregate rating of >75% of the measure was required. This analysis yielded 30/119 measures in round 1 and 11/30 measures in round 2. The final round of scoring yielded the following top five measures: (1) readmission rates within 30 days, (2) primary care visit within 7 days postdischarge for high-risk patients, (3) medication reconciliation completed at admission and prior to discharge, (4) readmission rates within 72 h and (5) time from discharge to homecare nursing visit for high-risk patients.
CONCLUSIONS: The five measures identified through this research may be useful as indicators of overall care quality related to care transitions involving complex-care patients across different healthcare settings. Further research efforts are called for to explore the applicability and feasibility of using the quality measures to drive quality improvement across the healthcare system.

Introduction

Patients with complex needs associated with their multiple diseases, comorbidities and conditions, frequently require care in multiple settings, and are particularly vulnerable to poorly executed transitions in care.1 2 Complex-care patients are managing multiple chronic diseases, taking several medications, may have limited social supports or suffer from dementia, mental illness or addiction. This complex-care patient cohort is predicted to grow substantially as the population ages and chronic diseases continue to be a major expenditure of healthcare costs.3 Poor transitions often result in unnecessary adverse patient outcomes and additional healthcare spending.4

Care transitions refer to a set of actions designed to ensure the coordination and continuation of appropriate healthcare provision as a patient transfers between different levels of care in the same clinical setting or different locations.5 Current literature examining transitions in care has mainly focused on transfers within acute-care organisations and to the home setting.6–8 However, complex patients, at any age, may experience transitions across the continuum of health services from acute care to multiple healthcare settings, such as complex continuing care rehabilitation centres, long-term care, home, or possibly from acute care teaching hospitals to community-based hospitals.

The importance of gaining insight into transfers of complex-care patients across the lifespan and between settings is a critical and timely policy issue. At present, there is no standardised set of quality measures associated with transitioning complex-care patients across the various healthcare settings and discharged and/or readmitted to/from home.9 This is an important gap to address, as quality indicators are increasingly being employed to assess and improve quality of care in many healthcare settings.10 Therefore, the purpose of this study was to define quality measures for care transitions involving complex-care patients across the continuum of care through a structured panel process. Our focus was to develop indicators that would be useful for accountability purposes as well as ones that had the potential for broader quality improvement (QI) efforts.

Methods

Overview of modified Delphi panel process

A modified Delphi consensus technique based on the RAND method was used to develop measures of quality care transitions across the continuum of care.11 The panel process was built on a technique that was shown to be effective for developing quality indicator measures in healthcare.12–18 For this study, multiple stages of the modified Delphi technique (literature review, individual rating, face-to-face consensus meeting, and final ranking) were used (see figure 1). Key to this methodology is engaging in multiple rounds of feedback from the expert panel using both anonymous surveys and round table meeting techniques to allow for optimal, unbiased expression of opinions.

Figure 1

Illustration of modified-Delphi process.

Initial work: literature review and selection of initial set of measures

The first step in the Delphi technique is to engage in a structured review of the literature. The intent of this review was to produce a list of measures that had been reported in the literature that would then be presented to a panel of experts to review and rate. This literature search was undertaken in September 2011 using OVID Medline with the following key words with the number of abstracts reported: Quality Assurance, Health Care or Quality of Health Care; Health Transition (4 217 740); Patient Transfer, Patient Discharge, Continuity of Patient Care, Health Services for the Aged, Delivery of Health Care, (714 391) Home Care Services, Chronic Disease, Aged, Long Term Care (2 280 704); Primary Health Care, Delivery of Health Care, Patient Transfer, Continuity of Patient Care (723 017); Health Transition (650); Outcome and Process Assessment (Health Care) and Outcome Assessment (Health Care) (571 586). These search terms were used as they were identified by the research team and key stakeholders as key terms for care transitions.

Other databases beyond OVID Medline were not included as the initial search yielded a significant number of abstracts. The search was limited to those written in the English language (1675) and from 2000 to 2011, as any previous studies would have been included in literature syntheses during this time frame. Other limits included articles that were clinical conferences, lectures and legal cases, newspaper articles, commentaries and bibliographies/biographies (1658). The final limit was to include meta-analysis and reviews only (76). These 76 abstracts were reviewed by three members of the research team to determine which ones reported on outcome measures associated with care transitions. This review yielded 23 articles including two meta-analyses 19 20 and 21 literature reviews,21–41 This review was augmented with other literature known to the research team, or were references included in the reference lists of the literature review of primary studies.5 41–51

Additionally, a search of targeted journals and grey literature was conducted in early January 2012 that yielded 14 other documents that included evidence for transitions of care outcome measures.9 52–64 This second search was undertaken for several reasons. First, some of the documents reviewed referred to other references that were not picked up in the initial literature search.9 51–58 61–64 Second, shortly after the literature search was conducted, a report was released on avoidable hospitalisations.52 Third, the research team became aware of additional interventions studies published in early 2012.59 60 Of the literature that was identified, all the studies were included in the next phase of the project measure development. Our research team developed an initial survey of potential measures from all measures of quality of transitions of care for complex-care patients listed in the final set of documents reviewed. The list of measures (n=119) were categorised under the following headings: health service usage; patient outcome; patient satisfaction and experience; provider satisfaction. Once the initial set of measures was determined, the research team identified potential panel experts to be engaged in this project.

First-round ratings

Panel member selection. The research team developed criteria for panel member selection in partnership with key stakeholders (Canadian Patient Safety Institute, Accreditation Canada and Patients for Patient Safety). Criteria for selecting panel members included having an interest in care transitions, patient safety, QI, patient care experience and related research initiatives. Additionally, key stakeholders (listed above) were asked to provide names of experts and organisations to include as potential panel members. From this, an initial list of panel members was created by the research team which had expertise in care transitions and measure development. There was originally a panel of 16 who committed to being part of the research project, with the final panel of 11 participants representing a broad scope of expertise in care transitions. The withdrawal of the five participants was due to scheduling conflicts. The final panel consisted of clinicians (medicine, pharmacy and nursing) including a general internist with patient safety expertise, researchers with measure development expertise and one who had extensive research background in care transitions. Both a national and provincial level policy maker were part of the panel as were decision makers from a variety of sectors, including acute care, home care, long-term care, complex continuing care/rehabilitation care and primary care. Stakeholder organisations and patient advocacy group were also represented on the final panel.

First round review survey. In the first round of the Delphi approach, the panel was sent the questionnaire electronically 4 weeks prior to the face-to-face panel meeting and asked for input to identify measures worthy of consideration. A reminder email was sent to panelists 2 days before, and on the day of the deadline. We chose to include all relevant measures (n=119) in the survey (see online supplementary file for list). The survey package included an overview of the literature review process, the list of measures with definitions for calculations as described in the literature (where provided), and literature summaries of each of the articles and reports. Additionally, an overview of the rating scale was provided and outlined in table 1.

Table 1

Rating matrix domains Desirable Attributes of a Quality Measure—Definitions: The research team has selected the Quality Measures Attributes developed by the Agency for Healthcare Research and Quality (AHRQ) to guide the rating matrix of the Transitions of Care Outcomes/Measures. Table 1 provides a description of each of the desirable attributes of a Quality Measure. For further information of AHRQ's Desirable Attributes of a Quality Measure visit: http://www.qualitymeasures.ahrq.gov.

AHRQ Desirable Attributes of a Quality Measure—Definitions
Quality measure domain	Criteria description
Importance of the measure	Relevance to stakeholders—the topic area of the measure is of significant interest, and financially and strategically important to stakeholders (eg, patients, clinicians, purchasers, public health officers, policy makers)
	Health importance—the aspect of health that the measure addresses is important as defined by high prevalence or incidence, and/or a significant effect on the burden of illness (ie, effect on the mortality and morbidity of a population)
	Applicability to measuring the equitable distribution of healthcare (for healthcare delivery measures) or of health (for population health measures)—the measure can be stratified or analysed by subgroup to examine whether disparities in care or of health exist among a diverse population of patients
	Potential for improvement—there is evidence indicating a need for the measure because there is overall poor quality or variations in quality among organisations (for healthcare delivery measures) or overall poor quality of health or variations in quality of health among populations (for population health measures)
	Susceptibility to being influenced by the healthcare system—for healthcare delivery measures, the results of the measure relate to actions or interventions that are under the control of those providers whose performance is being measured, so that it is possible for them to improve that performance. For public health measures, the results should be susceptible to influence by the public health system
Scientific soundness: clinical logic	Explicitness of evidence—the evidence supporting the measure is explicitly stated
	Strength of evidence—the topic area of the measure is strongly supported by the evidence, that is,  indicated to be of great importance for improving quality of care (for healthcare delivery measures) or improving health (for population health measures)
Scientific soundness: measure properties	Reliability—the results of the measure are reproducible for a fixed set of conditions irrespective of who makes the measurement or when it is made; reliability testing is documented
	Validity—the measure truly measures what it purports to measure; validity testing is documented
	Allowance for patient/consumer factors as required—the measure allows for stratification or case-mix adjustment if appropriate
	Comprehensible—the results of the measure are understandable for the user who will be acting on the data
Feasibility	Explicit specification of numerator and denominator—a measure should have explicit and detailed specifications for the numerator and denominator; statements of the requirements for data collection are understandable and implementable
	Data availability—the data source needed to implement the measure is available and accessible within the timeframe for measurement. The costs of abstracting and collecting data are justified by the potential for improvement in care or health
To ensure measures developed through the panel are useful and relevant over time, you as a panellist are encouraged to consider the extent to which identified measures can be linked to quality of care processes currently, as well as your view as future areas that necessitate attention related to improving transitions of care for medically complex patients across the continuum.
In addition to the four AHRQ attributes of a quality measure, two additional criteria for evaluation will be used:
▸ Measuring this is useful in driving transitions of care quality improvement.
▸ Measuring this is useful for accountability purposes such as public reporting.
Instructions—You will be asked to rate each of the transitions of care outcomes/measures and/or associated contextual variable identified from the literature review using the following rating matrix (domains of quality measures) on a 9 point Likert Scale ranging from 1 strongly disagree to 9 strongly agree. For example, choosing strongly agree on the “Importance of Measure” reflects your agreement about this particular measure in relation to the criteria description in the AHRQ Desirable Attributes of a Quality Measure provided in Table 1.

AHRQ, Agency for Healthcare Research and Quality.

Specifically, panel members were asked in the survey to evaluate the proposed measures using the Quality Measures Attributes developed by the Agency for Healthcare Research and Quality (AHRQ).65 This framework groups measures according to four broad conceptual areas: (1) importance of the measure; (2) scientific soundness of the measure—clinical logic; (3) scientific soundness of the measure—measure properties; (4) feasibility of the measure. Given the focus of our study was to identify quality measures that may in the future be useful for accountability purposes and to make system improvements for complex-care patient transitions, we also included two additional criteria for evaluation including the usefulness of the measure: (1) to drive QI efforts and (2) for accountability purposes in public reporting (PR). We included the means for the two latter measures, in addition to the overall mean average of the six evaluation criteria, due to our interest in developing measures for accountability purposes and QI efforts. The panel used a nine-point Likert scale which considered scores of 1–3 as strong disagreement, 7–9 as strong agreement and 4–6 as uncertain agreement. An overall total score that averaged all six evaluation criteria was calculated alongside the score for the usefulness of the measure to drive QI efforts and the usefulness of the measure for accountability purposes in PR. Panel members were also provided with the opportunity to suggest additional measures prior to the in-person meeting. To ensure measures developed through the panel are useful and relevant over time, panelists were encouraged to consider the extent to which identified measures can be linked to currently employed quality of care processes as well as what they view as future areas that necessitate attention related to improving transitions of care for complex-care patients across the continuum. Consensus on these surveys was determined to be reached upon a 75% agreement (standard percentage in other studies)16 17 among panel members on the three scores (mean total score for all six rating criteria; mean score for usefulness to drive QI; and mean score for usefulness for PR).

Second-round panel meeting

For the second round, panel members met in person for a 1-day facilitated structured discussion of the results compiled from the first round of the questionnaire. At the conclusion of the discussion, members were asked to complete a second round of the questionnaire using the same evaluation criteria and rating scale. After completion of all discussion and questionnaires, panellists were also asked to anonymously comment upon their experience regarding the overall consensus process as a mechanism to ensure the process was fair and unbiased.

The final results were tabulated using the same consensus agreement score (≥75%). The panel was then contacted once more to provide a priority ranking (from 1–11) among the final selected measures. Again, individual responses remained anonymous to all other panel members. The mean frequency of the rankings of the set of measures was calculated to yield the final set of measures. Additionally, panellists were asked for their personal views of where the priority for measure usage and development should be regardless of current data availability and ease in feasibility.

Results

First round ratings

From the initial set of 119 measures, 30 measures received an aggregate rating of >75% (7/9 or more) across one or more of the total score of the six domains, useful to drive QI, and useful for the PR domain (see table 2).

Table 2

Results from prepanel questionnaire of measures achieving at least 75% consensus

Measure	Meantotal score	Meandrive QI	Meanpublic reporting
Health services usage measures
Readmission/readmission rates (72 h, 7, 30, 90 days)	7.76	7.73	7.6
ED visits/ED admission rates	7.43	7.26	7.29
Time from discharge to homecare nursing visit for high-risk patients	7.57	7.93	7.36
Primary care visit within 7 days postdischarge for high-risk patients	7.29	7.64	7.07
Medication reconciliation completed prior to discharge	7.69	8.08	7
Discharge summary (all medications and follow-up appointments) given to patient	7.7	8.14	7.31
Discharge summary (all medications and follow-up appointments) given to primary healthcare provider	7.91	8.29	7.71
Ambulatory sensitive conditions hospitalisation/ambulatory care visits	7	7.63	6.94
Healthcare connected for unattached patients	7.08	7	6.85
Hospital discharge process (discharge planning, plan of care, postdischarge support, hospital assessing needs and using discharge plans)	7.07	7.5	6.71
Discharge medication list sent to pharmacy	7.04	7.07	6.29
Continuity of the medical records upon hospital discharge to the outpatient setting	7.27	7.46	6.77
Clinical performance (screening, immunisation, chronic care after acute myocardial infarction, diabetes mellitus control, hypertension control)	7.15	7.15	6.62
Potentially avoidable hospital admissions	6.76	6.67	7.27
Multiple psychiatric readmissions	6.91	6.53	7.27
Early collaborative care/multidisciplinary early supported discharge	6.9	6.33	7.31
Patient education/self-management support	6.93	6.21	7.07
Patient level measures—clinical
Nosocomial infection	7.19	7.25	7.17
Mortality/death/cause of death/risk of death	7.24	7	6.85
Functional status/physical function/improvement	7.08	7.23	6.62
Medication errors	7.28	7.33	6.92
Critical incidents	7.39	7.5	6.92
Medication continuity/discrepancies	7.13	7.08	6.67
Pressure ulcers	6.96	6.66	7.08
Pain	6.97	6.58	7.17
Patient level measures—satisfaction and perceptions
Patient and caregiver satisfaction (follow-up during interventions and/or after)	7.52	7.69	7.54
Activities of daily living/social activities	7.13	7.08	6.67
Quality of care	6.92	6.58	7
Provider level measures—satisfaction and perceptions
Communication between shift staff/optimal choice of communication/information transfer	7.21	7.54	6.23
Communication and collaboration of physician consultations	6.94	5.85	7.31

ED, Emergency Department; QI, quality improvement.

Second-round ratings—panel

Analysis from the second round of panel member ratings revealed a set of 11 measures (see table 3). This included seven measures that received an aggregate rating of >75% (7/9 or more) in all three scores—total score of the six domains, useful to drive QI and useful for the PR domain; three measures received an aggregate rating of >75% (7/9 or more) total score of the six domains, and useful to drive QI score and between 6 and 6.99 on the useful for PR domain; and one measure received an aggregate rating of >75% (7/9 or more) in the useful to drive QI. Totally, 11 out of the original measures were selected to go to the final round of rating, with wording changes to four of the measures (medication reconciliation, time from discharge to homecare nursing visit for high-risk patients, mortality rate and discharge summary), and readmission rate stratified by three time frames—within 72 h, 30 days and 90 days.

Table 3

Results from postpanel questionnaire of measures achieving at least 75% consensus

Measure	Mean total score	Mean drive QI	Mean public reporting
Patient and caregiver satisfaction (follow-up during interventions and/or after)	8.05	8.09	8
Readmission rates within 30 days	7.86	8.18	7.55
Medication reconciliation completed at admission and prior to discharge	7.82	8.45	7.18
Discharge summary (all medications and follow-up appointments) given to primary healthcare provider	7.7	8.1	7.3
Readmission rates within 72 h	7.55	7.82	7.27
Readmission rates within 90 days	7.45	7.91	7
Time from discharge to homecare nursing visit for high-risk patients	7.41	7.91	6.91
Mortality/death/cause of death/risk of death within 72 h	7.41	7.55	7.27
Discharge Summary (all medications and follow-up appointments) given to patient	7.35	7.9	6.8
Primary care visit within 7 days postdischarge for high-risk patients	7.27	7.64	6.91
Multiple psychiatric readmissions in 30 days	6.95	7.18	6.73

QI, quality improvement.

Totally, 21 out of the original 30 measures were eliminated in this round. Analysis of the key discussion points at the panel meeting and hand written notes from the participants revealed that although these measures were important quality measures of care, they were not perceived collectively to be a quality measure of a care transition. For example, several patient clinical outcome measures (eg, nosocomial infections, pressure ulcers, pain, functional status and medical errors) were viewed as not being an outcome of an ineffective care transition. Other key issues identified with those measures that were eliminated included the perception that the measures were unclear or too broad to be an effective indicator of the transition (eg, potentially avoidable hospital admissions and hospital discharge process). Other key discussion points included a focus on patients and family perspectives for what constitutes quality care transitions for complex-care patients across the care continuum. The panel members recommended further work be undertaken to develop more system-level measures that would capture complex-care patient and family members’ satisfaction and experiences along the care continuum.

A postmeeting survey found unanimous agreement from the participants that their opinions were valued, the process was fair, and that no one altered their responses or opinions as a result of feeling intimidated.

Final review ratings

When asked to rank the most important final indicators with respect to overall score, QI and PR purposes, with the exception of readmission rates within 30 days as the number one priority and multiple psychiatric readmissions in 30 days as the last priority, there were mixed responses regarding where the priority focus should be (see table 4). Ranking the measures using the overall total score yielded the following top five measures are readmission rates within 30 days, primary care visit within 7 days postdischarge for high-risk patients, medication reconciliation completed at admission and prior to discharge, readmission rates within 72 h, and time from discharge to homecare nursing visit for high-risk patients.

Table 4

Final rankings

Measure	Ranking	Mean score
Readmission rates within 30 days	1	3
Primary care visit within 7 days postdischarge for high-risk* patients	2	3.6
Medication reconciliation completed at admission and prior to discharge	3	3.8
Readmission rates within 72 h	4	4.5
Time from discharge to home care nursing visit for high-risk patients	5	4.8
Discharge summary (all medications and follow-up appointments) given to primary healthcare provider	6	5.6
Discharge summary (all medications and follow-up appointments) given to patient	7	6
Patient and caregiver satisfaction (follow up during interventions and/or after)	8	7.5
Readmission rates within 90 days	9	8.6
Mortality/death/cause of death/risk of death within 72 h†	10	9
Multiple psychiatric readmissions in 30 days	11	9.6

*The panel defined high-risk patients as those who were at high risk for being readmitted to hospital due to the complexity of their condition/disease and ability to adhere to and manage their care postdischarge.

†The panel defined mortality related to the complexity of their condition/disease with 3 days (72 h) of discharge from hospital.

Discussion

Through a structured, modified Delphi panel process, a series of quality measures for care transitions involving complex-care patients across the continuum of care were delineated. Our inclusion of health services researchers, decision-making organisations, frontline clinicians, (physicians, nurses and pharmacists) and a patient representative increased the likelihood that the selected quality measures are scientifically sound and have broad applicability and feasibility.10 18 Our final list of quality indicators included outcomes from the three domains of focus including nine health services measures (readmission rates within 30 days, primary care visit within 7 days postdischarge for high-risk patients, medication reconciliation completed at admission and prior to discharge, readmission rates within 72 h, time from discharge to homecare nursing visit for high-risk patients, discharge summary all medications and follow-up appointments given to primary healthcare provider, discharge summary all medications and follow-up appointments given to patient, readmission rates within 90 days, multiple psychiatric readmissions in 30 days); one patient outcome measure (mortality/death/cause of death/risk of death within 72 h); and one patient satisfaction and perceptions measure (patient and caregiver satisfaction follow-up during interventions and/or after). These measures were all associated with posthospital discharge and warrant further attention in future research to determine what measures should be included and/or developed to capture measures that hospitals could use to assess the quality of care transitions with complex-care patients.

Implications for practice and policy

To our knowledge, there has been no previous research in the area of quality measures for care transitions involving complex-care patients across the continuum of care. As a next step, further validation of these measures is recommended with other key stakeholders. This work will require: (1) testing of whether these measures actually measure the quality of care (vs differences in case mix); (2) determining if there are sufficient numbers to enable stable estimates when measuring at the provider/hospital level; (3) determining the need for risk adjustment and (4) ascertaining the level of measurement for the indicators. There is potential for these measures to be used as indicators of overall care quality related to care transitions involving complex-care patients across the continuum of care.

There is an increasing focus on moving a patient out of the hospital system to alleviate wait times and in an effort to provide patients with more appropriate care in the right settings, such as home-based care and community-based programmes and services. However, in order to gain the confidence of patients and their caregivers in these movements across settings, it is essential that organisations be accountable for providing and monitoring high-quality transitions. Having measures such as these can allow organisations to ascertain how well they are doing, make improvements where necessary and be accountable to the patients who they are transitioning. We acknowledge that this study has been conducted in the Canadian healthcare system, therefore, generalisablity to other countries and contexts may not apply exactly. However, the Delphi technique described here can be replicated in other countries in order to understand whether these measures are consistent in other contexts.

Moreover, there is broad applicability in the majority of the final quality measures selected as they use data that is already commonly collected by hospitals and other healthcare organisations and could feasibly be collected and reported. Some specific points made by panel members during the discussion merit emphasis. For example, the recommendation to further develop measures that are reflective of complex-care patient and family members’ satisfaction and experiences along the care continuum warrants further attention. This finding is consistent with the view that patients and their families play an integral role in ensuring they receive safe care, as they are the one constant in care transitions processes.5 This work can draw from existing measures such as the Care Transitions Measure used to measure satisfaction with discharge from hospital to home.1 5 56

Strengths and limitations

Engaging key stakeholders in quality measure development is paramount. Important attributes of high-quality measures are their relevance to the selected problem and field of application, their feasibility and their reliability.10 It is important to note that the inclusion of a patient representative on the panel is important, as a recent systematic review revealed that the participation of patients in quality indicator development is extremely uncommon.10 Our final panel consisted of individuals (researchers, front-line clinicians and a patient) with broad professional representation as well as consultants from stakeholder groups (national accreditation body and provincial quality council). Additionally, our panel included individuals from different healthcare systems and sectors increasing the potential for generalisability of our findings.

The literature review that was included as a summary in the background information and anonymous survey used in our process ensured that all members were entering the discussion with balanced background and content knowledge. We also included open-ended questions in the presurvey to provide an opportunity for panel members to comment on the list of indicators and submit additional indicators. Additionally, an experienced health services researcher facilitated the discussion and maintained equality among all panel members. The anonymous end-of-day survey verified that all panelists were comfortable with the process.

Furthermore, panel discussion was also focused on finding important and relevant measures that could act as quality indicators in addition to the ones initially presented by the research team. This open-ended discussion was presented to the panel prior to the face-to-face meeting to provide opportunity for panelists to review and reflect on the measures presented and potential missing measures reflective of quality care transitions for complex-care patients across the care continuum. Upon review of the measures premeeting and postmeeting, it was evident that the face-to-face discussion was a vital component to our process as the measures and parameters were redefined through the modified Delphi process. This is an important finding that should guide future quality performance measures—research and development.

There are limitations associated with our modified Delphi panel process. First, the review of literature was not a systematic review, rather it was a targeted review that was designed by researchers with expertise in performance measures and care transitions. We drew on the expertise of other researchers, clinicians and researchers that were part of the panel to ensure that key measures were not omitted. Second, there was the potential for selection bias inherent in panel member selection. To minimise selection bias, the research team used a similar process used by one of the researchers17 18 and solicited suggestions from unbiased third party individuals, stakeholder groups as well as national patient safety organisations. However, our initial international panels were not able to participate. Thus, some important perspectives may not have been identified. Another measure to reduce the risk of selection bias was the use of predesigned forms containing rating and selection criteria during systematic consensus processes.10 Third, the complex nature around the validity of the measures and whether they are measuring quality of care and the feasibility of collecting data on the identified measures is another limitation. In our study, panelists were not asked to limit their selection to quality measures that could be applied immediately with commonly available data. The use of administrative data for any quality measures always has the limitation of clinical application, as it is only a part of a patient's clinical story. Fourth, our panel had only one patient representative, and although they were a leader of a national patient safety advocacy organisation, their views may not be completely representative of patients and caregivers.

Conclusion

Quality measures are important tools used for accountability and QI in healthcare systems. The modified Delphi consensus panel approach applied in this study enabled us to garner a consensus on measures in an important area—care transitions. Although a number of these measures are not new and are used for the evaluation of other healthcare areas, this study provides evidence and support that these measures may also be effective for evaluating the quality of care transitions. This is the first study focused on care transitions and performance measures, thus, more research and testing of these measures using a rigorous research design are needed to determine their actual usefulness and feasibility. Further work is required to create standardised operational definitions, for example, the patient and caregiver satisfaction follow-up during interventions and/or after measure.

Collectively, future research efforts will produce findings that can be used by healthcare administrators, clinicians and policy makers to monitor care transitions and to work together to improve the quality of transitions of care with complex-care patients. Moreover, individual institutions can evaluate interventions aimed at enhancing care transitions for complex-care patents’ efforts, and provide healthcare systems with a consistent means to compare programme effectiveness across institutions. A longer-term goal is that this work will inform the development of a reporting standard of measures that demonstrate quality outcomes associated with transitions of care with complex-care patients.