'Care for Outcomes': systematic development of a set of outcome indicators to improve patient-relevant outcomes for patients with lung cancer.

OBJECTIVES: Measuring quality of care is important, however many of the quality indicators used do not focus on outcome of treatment and aspects which are valuable for patients and physicians. The project 'Care for Outcomes' aims to establish a relevant set of outcome indicators for lung cancer.
SETTING: Network of seven large, non-university teaching hospitals in the Netherlands (Santeon).
METHODS: By reviewing the literature, a list of potential outcome indicators for patients with lung cancer was composed and subsequently prioritised by expert's opinion. Three external parties, with expertise on lung cancer, clinical management and public health, evaluated and reduced the list of indicators to a working set. Finally, the resulting selection of outcome indicators was tested for feasibility and discrimination in patient data, by collecting retrospective data and performing regression and survival analyses. PARTICIPANTS: Development of the indicator set in six Santeon hospitals. Retrospective cohort study in 5922 patients diagnosed with lung cancer (all types and stages).
RESULTS: Selected outcome indicators were divided into three levels of outcome (tiers). The first tier about survival and the process of recovery include mortality, survival, positive resection margins, rethoracotomy after resection and quality of life at baseline and after 3, 6 and 12 months. Tier 2 concerning the sustainability of the recovery include complications after resection and toxicity after chemotherapy and/or radiation. Tier 3 about sustainability of health revealed no measurable outcomes. The retrospective data collection showed differences between hospitals and variation in case mix.
CONCLUSION: A relevant set of outcome indicators for lung cancer was systematically developed. This set has the potential to compare quality of care between hospitals and inform patients with lung cancer about outcomes. The project is ongoing in the current Santeon Value-Based Health Care programme through quality and improvement cycles. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A literature review followed by expert review formed the basis for the indicator selection.

The established set of indicators seemed feasible to collect data from medical charts retrospectively with few missing data.

The set is sensitive to detect differences in outcomes between hospitals in order to fuel discussions towards improvements in delivery of care.

Data collection on quality of life through patient-reported outcome measures was challenging.

The indicator set is mainly based on expert’s opinion and is subject to change.

Introduction

Lung cancer is the most commonly diagnosed cancer and the leading cause of cancer death worldwide.1 In the Netherlands, more than 13 000 people are diagnosed with lung cancer annually.2 There are two main types of lung cancer: non-small cell lung cancer (NSCLC), which is the most common type (85%), and small cell lung cancer (SCLC). The majority of patients is diagnosed with an advanced stage lung cancer (stage III or IV),3 and therefore has a poor prognosis. For example, median overall survival is approximately 2 and 10 months, for untreated and systemically treated patients diagnosed with stage IIIB or IV lung cancer, respectively.4

Significant regional differences and between-hospital variation in treatment patterns and outcomes for patients with NSCLC are shown in the Netherlands.5 For example, substantial variation in time to treatment exists between patients and between hospitals for patients with advanced NSCLC.6 Von Meyenfeldt et al found a clinically relevant between-hospital variation in length of stay after lung cancer surgery.7

Besides variation in treatment practices, quality indicators, which are commonly used to evaluate healthcare and identify areas for improvement, also vary across hospitals. Healthcare institutions often register a wide set of indicators; however, many indicators focus on the process and structure of the treatment, rather than on its outcomes. For instance, the time required to perform a CT scan on a patient with a pulmonary mass is a commonly used process indicator. However, while such an indicator might signal any number of issues in the diagnostic process, it is not usually relevant to the eventual outcome of the patient. Registering these kinds of indicators burdens doctors with administrative tasks that do not necessarily have an impact on relevant health outcomes for patients. Similarly, if hospitals register different sets of indicators to measure their quality of care, this will limit supervisory institutions’ ability to gain insight into the quality of care.

Value-Based Health Care (VBHC) is increasingly being promoted as a strategy for improving quality of care. Based on the principles formulated by Porter and Porter et al,8 9 value is defined as patient-relevant health outcomes per dollar spent for a specific medical condition over the full cycle of care. The first step is to define outcomes that matter to patients and other stakeholders. Relevant measures of quality would reflect the survival and degree of recovery, process of recovery (eg, treatment related complications) and sustainability of recovery. Furthermore, these measures of quality should be collected in a standardised way. Although efforts to report outcomes of lung cancer treatment exist, for example, the Dutch Lung Cancer Audit for Surgery (DLCA-S),10 11 there is no national standardised approach to report outcomes of all aspects of lung cancer treatment (including systemic treatment and best supportive care). Additionally, patients’ reports of their health-related quality of life (QoL) are rarely measured routinely. Furthermore, true comparison would only be possible when patient groups are relatively homogeneous or when correction for case mix (combination of patient and disease characteristics) is reliably achieved.

This article discusses the project ‘Care for Outcomes’ (CfO),12 which aims to develop a relevant set of outcome indicators for lung cancer, including patient-reported outcome measures (PROMs) and case-mix factors to increase the usefulness of comparisons across treatment modalities and institutions.

Methods

Outcome indicator selection process

An indicator selection process was performed in three phases in order to determine a set of outcome indicators (figure 1). The initial broad selection is based on a review of the literature and prioritising by expert’s opinion (Phase I). Subsequently, external stakeholders evaluated and reduced the list of indicators to a working set (Phase II). Finally, the resulting selection of outcome indicators was tested for feasibility and discriminative aspects in patient data (Phase III).

Figure 1

Different steps in the development of outcome indicators in the project ‘Care for Outcomes’.

Phase I: literature review and classification of potential outcome indicators

First, a literature review was conducted for defining potential outcome indicators. The goal of this search was not to identify all available relevant literature, but to obtain a comprehensive overview of all possible and relevant outcome indicators that could act as a starting document for the discussion between different caregivers. The search included all types and stages of lung cancer. A ‘care delivery chain’ was formulated in order to organise potential indicators. This chain lists all activities in the process of care.8 By distinguishing individual activities, it is possible to identify which of them create value for the patient.

The potential outcome indicators as derived from the literature were divided into three tiers according to the outcome hierarchy by Porter (online supplemental appendix figure 1). The first tier includes survival and degree of recovery, tier 2 relates to the process of recovery and the third tier comprises the sustainability of recovery. Next, a team of pulmonologists, nurse specialists and policy-makers prioritised systematically the selected possible outcome indicators from the literature. Therefore, a Delphi process—an established method of achieving a consensus between experts—was used.13 Every indicator was evaluated according to: (a) the relevance to the patient; (b) the medical relevance, that is, the extent to which the indicator measures the quality of care and (c) the relevance for patient populations, for example, the more prevalent complications are considered to be more relevant. Missing indicators could be added on the list by team members. All indicators with an SD of the ranking of more than 1.5, or added by one of the members, were discussed. With this list, the top 2–3 outcome measures per subtier were selected.

Phase II: evaluation and reduction of indicators by national and international stakeholders

In the second phase of the selection, every indicator that passed the first phase was evaluated by three external parties. An international academic council consisting of experts in lung cancer evaluated the medical relevance of the selected indicators. A methodology academic advisory council with experts on clinical management, public health and decision sciences assessed the methodology of the project. And thirdly, an advisory board consisting of representatives of patient associations, health insurers and the government gave more general advice on the process, based on social relevance. This second phase in the selection process resulted in a working set of outcome indicators.

Phase III: testing the feasibility and discriminative aspect of the indicator set in clinical practice

The third and final phase of the indicator selection process involved the testing of the indicator set in practice, in a retrospective data analysis based on data of patients in the Santeon hospitals, a network of seven large, non-university teaching hospitals in the Netherlands. First, the distinctive power, the comparability and the reproducibility of the selected indicators was tested in patient data collected retrospectively for the years 2008–2011 in two Santeon hospitals, the St. Antonius Hospital in Nieuwegein, Utrecht, and Catharina Hospital in Eindhoven.14 After this first round of data collection, the quality and feasibility of data collection was scored. The outcomes of this analysis were shared with the three external parties (councils and advisory board), who re-evaluated the indicators based on the results of the first round. The final set of outcome indicators was selected for analysis based on the empirical results and the re-evaluation by external parties.

Second, the patient data collection was extended to four other Santeon hospitals, in which two rounds of data collection were performed (round 1: data for the years 2008–2012 and round 2: data for the years 2008–2014).12 15 These data were analysed regarding the differences in outcomes when taking case-mix variables into account.

Statistical analysis

Descriptive statistics were used for evaluating the data collected in Phase III of the project. Multivariable survival analysis was conducted to assess the discriminative potential of the selected case-mix variables. The statistical analysis was performed using IBM SPSS statistics V.21.0. Continuous data were expressed as mean±SD or median (range) when appropriate. Categorical data were analysed using χ2 and continuous data using Student’s t-test, rank tests and one-way Analysis Of Variance (ANOVA) when appropriate. A p value of 0.05 or smaller was considered statistically significant.

Comorbidity was defined by the Charlson Comorbidity Index (CCI), which is a sum score based on the associated weight of each of the 17 comorbidities, and an indicator of disease burden.16–18 Feasibility is measured by calculating the percentage of patients for whom it was possible to collect the outcome indicator (non-missing).19

Next, a multivariable Cox-regression analysis was performed to find independent predictors of long-term survival. All available characteristics were explored as potential prognostic factors (Eastern Cooperative Oncology Group Performance Status (ECOG PS), age, gender, CCI, coronary heart disease, diabetes, lung function percentage of forced expiratory volume in one second (FEV1%) and percentage of carbon monoxide transfer factor (DLCO%) and morphology of the tumour (NSCLC and SCLC)). Outcomes were presented in whisker plots. For each hospital, a Cox regression was performed to compare the concerning hospital with the other five hospitals. Patients still alive at date of follow-up were censored and given the end of follow-up date as imputed date of death. Logistic regression was used to find independent predictors for outcomes such as complications after resection. In this analysis, missing values were imputed by giving the value 0, which indicates that the patient did not have the characteristic.

Patient and public involvement statement

The goal of this study was to develop outcome indicators that are relevant for patients. All the participants were very aware of this goal and throughout the process the interests of the patients were therefore of special importance. The patients were not themselves involved in the design and conduct of the study. Nevertheless, patient contribution was established by including patient representatives in the advisory board. Dissemination of project results to patients is facilitated by infographics and a website.

Results

Phase I

After reviewing the literature, a first set of 80 potential outcome indicators was selected. Subsequently, the team of pulmonologists, nurse specialists and policy-makers reduced the first set to 25 potential outcome indicators based on relevance. Finally, Phase I yielded 2–3 outcome measures per subtier.

Phase II

The selection of Phase I was evaluated and further reduced by the three external parties in Phase II. This resulted in the working set of seven outcome indicators, which are shown in table 1, per tier according to the outcomes hierarchy of Porter. The selected indicators are specific to the treatment, such as the share of positive resection margins, which signals the quality of surgical treatment. The QoL questionnaires used were the European Organization for Research and Treatment for Cancer Quality of Life Questionnaire (EORTC QLQ) C30 and LC13, which are the most commonly used and validated questionnaires for these type of patients. Tier 3 revealed no measurable outcomes, because they did not meet the criteria of adding value to the patient, medical relevance and relevance to patient populations.

Table 1

Final set of outcome indicators

Outcomes hierarchy according to Porter		ICHOM Standard Set for Lung Cancer	Selected outcome indicators, CfO	Feasibility project, CfO	Measured in population, CfO (n)
Tier 1Survival and degree of recovery/health	Survival	Cause of death	n/a	—	—
		Overall survival	Overall survival after diagnosis	95%	5922
		Treatment-related mortality	Overall mortality 1 and 2 years after diagnosis	95%	5922
	Degree of recovery/health	ECOG/WHO performance status	n/a	—	—
		—	Treatment result after resection: resection margins	97%	1169
		—	Treatment result after resection: rethoracotomy	99%	1169
		QoL/PROMs	QoL/PROMs(t=0, 3, 6 and 12 months)	27%*	672†
Tier 2Process of recovery	A: time to recovery	Acute complications of treatment: surgical	Complications after resection	98%	1169
		Acute complications of treatment: radiation	Side effects after radiotherapy or chemotherapy‡	90%	2192
		Acute complications of treatment: systemic therapy
	B: disutility of care	—	n/a	—	—
Tier 3Sustainability of health	A: durability of recovery	—	n/a	—	—
	B: long-term effects	—	n/a	—	—

*Rough estimation because of prospective character of QoL measurements and a plausible failure to invite all diagnosed patients to participate in the QoL measurements due to challenges in daily practice.

†Measured in prospective patient cohort of patients diagnosed with lung cancer in 2014. This prospective population (n=672) differs from the total population diagnosed with lung cancer where the retrospective data collection is based on (i.e. n=5922 in period 2008–2012).

‡Side effects were divided into anaemia, bone marrow suppression, cardiovascular, empyema, oesophagus related, lung related (pneumonitis, cough), respiratory fistula, nausea, vomiting and renal. Results of the side effects can be found in the CfO result book edition 2014.12

CfO, Care for Outcomes; ECOG, Eastern Cooperative Oncology Group; ICHOM, International Consortium for Health Outcomes Measurement; n/a, indicators not applicable; PROMs, patient-reported outcome measures; QoL, quality of life.

Besides the selected outcome indicators from the CfO Project, indicators from the International Consortium for Health Outcomes Measurement (ICHOM) Standard Set for Lung Cancer are also shown in table 1, which adds to discussion about the possibilities for benchmarking outcome measures in lung cancer worldwide. Furthermore, descriptive results from Phase III (retrospective data collection) were shown regarding the number of patients involved for each outcome measure and the feasibility of the data (percentage non-missing data).

Phase III

In Phase III, data of 5922 patients diagnosed between 2008 and 2012 in the Santeon hospitals were analysed. Baseline characteristics and outcome data can be found in the CfO result book12 and online supplemental appendix table 1. The data show high feasibility scores around 90%–99% (table 1), except for PROMs data, which were measured in a different cohort because of the prospective nature. Additionally, the response rate on PROMs deteriorated quickly after baseline measurement among patients with stage IV lung cancer (figure 2).

Figure 2

Patient-reported outcome measures response rates after baseline for patients with lung cancer stages IA–IIIB and IV.

Figure 3 shows that the largest group of patients (61%, n=3612) had advanced lung cancer (stage IIIB or IV), with some differences between hospitals (range 56%–64%). These patients received palliative treatment (systemic treatment or best supportive care). Patients with stages I–IIIA lung cancer received curative treatment with resection, radiotherapy and/or chemotherapy.

Figure 3

Distribution of stage and treatment in patients with lung cancer from Care for Outcomes database. CWZ, Canisius Wilhelmina Ziekenhuis; MST, Medisch Spectrum Twente; OLVG, Onze Lieve Vrouwe Gasthuis.

To illustrate between-hospital comparisons and the relevance of the set in clinical practice, the results concerning survival after resection in patients with lung cancer stages I–IIIA are shown in figure 4. In this analysis, case-mix characteristics were taken into account. A multivariable regression analysis showed that stage, ECOG PS and age were the strongest predictors of survival after resection (figure 4). Those characteristics were comparable across hospitals, although ECOG shows some variation in the distribution between ECOG 0 or 1 (ECOG 0: range 29%–46%; ECOG 1: range 24%–44%; resulting in a range of 58%–74% of the patients with ECOG 0–1 across hospitals) and the percentage of unknown ECOG values (2%–21%). After adjustment for case-mix variables using a Cox proportional hazard model, all hospitals showed comparable rates of mortality for patients with lung cancer stages I–IIIA and resection.

Figure 4

Influence of the most important case-mix variables on survival after resection (HR and 95% CI).

For patients with lung cancer stages I–IIIA without resection, analysis showed that one hospital showed a significantly better survival compared with the other hospitals. This result was used for further analysis for ‘best practice’, because it could be the effect of differences in the administration of chemoradiation therapy in each hospital. A team of healthcare professionals looked at these data and the corresponding method of practice in each hospital in detail and compared those to find possible differences. The survival data showed a significant better survival for patients aged <75 years who were treated with concurrent chemoradiation, compared with sequential administration of chemotherapy and radiation. For patients aged 75 years and older, no significant effect on survival was found between concurrent and sequential chemoradiation.15 This finding led to further research questions, because concurrent chemoradiation is associated with more adverse side effects.

The final set of outcome indicators was used in both rounds of data collection (2008–2012 and 2008–2014) in the CfO Project. Adjustment of the set followed in the new Santeon programme ‘VBHC’, which is the continuation to the CfO Project. At present, the set of indicators is in a constant process of review and is being evaluated in every cycle (6 months) of the VBHC programme.

Discussion

The aim of the project ‘CfO’ was to obtain a compact set of indicators to evaluate the quality of care in lung cancer treatments across the whole value chain. The project involves input from several external parties to ensure that medical, methodological and ethical aspects of the selection process are addressed. The obtained set of outcome indicators for patients with lung cancer, which encompasses the whole process of care, is novel and unique to the Netherlands. The resulting set of indicators consists of only six variables from three different outcome hierarchy levels: survival (mortality and median survival after diagnosis); degree of recovery (treatment result after resection and QoL) and process of recovery (complications after resection and side effects after radiotherapy or chemotherapy). These indicators, tested for their feasibility and discriminative aspect to patient outcomes, differ from most quality indicators nationwide, which focus on the process and structure of care or on only a part of clinical care for patients with lung cancer. For example, a review by Numan et al20 identified five evidence-based quality indicators for stages I–III NSCLC, but only focused on preoperative and postoperative care.

This study showed that the introduced set of outcome indicators seemed feasible to collect data from medical charts retrospectively, with few missing data. Furthermore, the set is sensitive to detect differences in outcomes between hospitals. The administrative burdens of gaining insight into the quality of care can thus be reduced when using such a smaller set of indicators than is currently used in many hospitals.

Alongside the established set of outcome indicators, patient and disease characteristics (case-mix variables) showed to be important in collecting data for comparisons across treatment modalities and hospitals. One of the analyses in Phase III of this CfO Project showed that a patients’ age at diagnosis is significantly associated with survival in patients with stages I–IIIA lung cancer. Existing literature about age as predictor for survival in lung cancer is inconclusive, with differences over time and in stages of disease.3 21 22 These different findings could be the result of, for instance, the population sample of patients used in the analysis, relations with other predictive factors (eg, ECOG PS and CCI) or differences between hospitals in accompanying (symptom-related) treatments. These findings emphasise the importance of proper adjustment when comparing hospitals on outcome indicators. Additionally, when analysing patient outcomes and benchmarking between hospitals, it is important to look beyond significant differences and use differences in outcomes with substantial clinical impact in the ongoing debate about improving quality of care. Furthermore, it remains essential to provide lung oncologists and other caregivers their own unadjusted data, which could promote opportunities to improve care processes.

A strength of this study is the combination of a structured development and a hands-on validation of an outcome indicator set in a large unselected cohort of patients, sized around 12% of the Dutch population with lung cancer. Therefore, the cohort presented in this study can be found as representation of current national practice. Furthermore, the established set of indicators seemed feasible to collect data from medical charts retrospectively with few missing data (high feasibility rates). Finally, the set is sensitive to detect differences in outcomes between hospitals in order to fuel discussions towards improvements in delivery of care.

Our approach has some limitations that should be considered. First, although the established indicator set for lung cancer reflects literature reviews and feedback among a relevant team of experts, representatives of patients associations and other stakeholders, the suggested outcomes and measures remain expert’s opinion. Additionally, no outcomes measures were specified in the third tier about sustainability of health or recovery and long-term consequences of therapy, since the potential outcome indicators in tier 3 did not meet the criteria for relevance according to the experts. However, the development of the set was defined as a starting point to obtain a compact set of indicators towards routine collection and evaluation of patient-centred outcomes for patients with lung cancer. The outcome indicator set is subject to discussion in a constant process of review with possible adjustments as the project evolves.

Second, the selected indicator set was tested on retrospective data. Therefore, the outcome set might not perfectly elucidate the current quality of care. Additionally, the CfO Project was faced with differences between hospitals in registration during the process of retrospective data collection (eg, complications or ECOG PS). Studies on benchmarking between hospitals would benefit from a more systematic measurement across hospitals, which eventually may also benefit outcome comparisons in general.

Finally, prospective data collection on QoL through patient-reported outcome measures was challenging. With the small number of responding patients to the QoL measurements (approximately 27% of patients), the question remains if the QoL results are valid and can be used in quality improvement. The Santeon hospitals are now working on improving the methods on PROM measurement for two problems experienced in the collection of QoL: (1) not every patient diagnosed with lung cancer received a QoL questionnaire and (2) not every patient who fills in the baseline questionnaire persevered in completing the QoL questionnaires at the successive measurement points. For patients with stage IV lung cancer, an explanation for the deteriorating response rates could be the poor prognosis among this group of patients. Shorter intervals between the surveys might improve the insights for such groups of patients.

Currently, the outcome indicator set for lung cancer is in a constant process of review in the Santeon VBHC programme, within a 6-month evaluation cycle. In this process, a number of factors are important to assess whether an outcome indicator remains suitable for the outcome set, for example, the availability of data, the prevalence of the measured indicator (eg, if a complication is rare, it is not a suitable indicator) and whether an indicator has a discriminative aspect (eg, if outcomes of an indicator are the same in every hospital, it is not very informative).

Furthermore, the results of this project are used in the Santeon hospitals to inform patients about their treatment options, prognosis associated with specific treatments and possible side effects. To facilitate patient understanding, typical outcomes were displayed in infographics. There is also a special website to inform patients about the project and its results.23 Hopefully, these insights support shared decision-making between patients and doctors when choosing treatment plans.

The data achieved by this project were used for several other studies.4 24–26 Additionally, the results of this project are associated with policy recommendations that apply to other fields of medicine. Similar projects were started within the Santeon hospitals to identify outcome indicators for breast cancer, colon cancer and prostate cancer. These are all aimed at eventually improving the quality of healthcare. This is being carried out through the establishment of quality cycles, which is an ongoing project within the current Santeon VBHC projects.

The CfO Project nowadays connects to several international projects on lung cancer, for example, the UK National Lung Cancer Audit.27–31 Furthermore, the ICHOM establishes sets of outcome indicators per disease in a similar systematic manner.32 It also employs experts and representatives of patient associations. Nowadays, a new initiative in the Netherlands, the DLCA, uses prospective registration and this is expected to increase transparency in lung cancer care.10 The DCLA now uses indicators that have been developed as part of the present project.

Conclusion

A relevant set of outcome indicators for lung cancer care is systematically developed in this ‘CfO’ Project. The indicators are relevant to the patient as well as doctors and are sensitive to detect differences in outcomes between hospitals in order to fuel discussions towards improvements in delivery of care. The goal of the Santeon hospitals is to inform patients about this process and the aim for improving care continuously. Insight in treatment options and expected outcomes can support shared decision-making. The project is ongoing in the Santeon VBHC programme, with continuously measuring outcomes and implementing improvements. Use of the Santeon set of outcome indicators and the resulting recommendations could also enable other institutions to monitor, compare and potentially improve the quality of their lung cancer care.