Latent composite indicators for evaluating adherence to guidelines in patients with a colorectal cancer diagnosis.

Evidence-based guidelines for the correct management of cancer patients are developed on the idea that timely care can improve health prognoses and quality of life.The aim of this paper is to evaluate the adherence of clinical pathways to clinical guidelines provided at the hospital level, for colorectal cancer care.By using a retrospective observational study, we proposed a method for associating each patient to a healthcare provider and modeling adherence as a latent construct governed by a set of 10 influential indicators. These indicators measure the adherence to specific guidelines for diagnosis, surgical treatment, chemotherapy, and follow-up. The model used was that of the item response theory (IRT). When evaluating providers, the IRT allows for a comparison of indicators in terms of their discriminating ability and difficulty, and in terms of their adherence to guidelines. The IRT results were compared with non-latent methods: numerator-based weight and denominator-based weight.A strong degree of coherence of the indicators in measuring adherence, and a high level of overall agreement between latent and non-latent methods were noted. The IRT approach demonstrated similar providers' evaluations between endoscopy and histological assessment indicators. The greatest discriminating ability among providers could be attributed to all diagnostic exams, while the lowest was associated with follow-up endoscopies. The most difficult indicator to achieve was fecal occult blood test, while follow-up imaging was the easiest.In a decision-making framework, valuable indications can be derived from the use of IRT models rather than weighting methods. Using IRTs, we were able to highlight the principal indicators in terms of strength of discrimination, and to isolate those that merely duplicated information.

Introduction

Colorectal cancer (CRC) represents the third most commonly diagnosed cancer in both genders[ and is one of the leading causes of deaths worldwide. For the Italian population, it was ranked in the 10 most significant causes of death for the year 2017, and represents the second most common cause of death from cancer.[ Previous studies have shown that survival rates are influenced by pathways of care that adhere to evidence-based guidelines that have been reported for CRC and other cancers.[ However, a large amount of heterogeneity is present across, and within, countries in the definition of the optimal pathway of care,[ as different health care agencies provide different guidelines for caring for CRC patients. The National Institute for Health and Care Excellence (NICE, The United Kingdom) has implemented guidelines for CRC diagnosis, treatment, and follow-up since 2007, which is mainly focused on procedures that have been shown to improve either survival rates or quality of life.[ At the national level, the Italian Association of Medical Oncology provides clinical recommendations for the management of patients with CRC.[ To evaluate adherence to evidence-based guidelines, the current goal is to define sets of indicators, usually defined by teams of experts, which include measures of the quality of care delivered to cancer patients from diagnosis through follow-up.[ In Italy, a set of 26 indicators have been constructed from a literature review, with the aim of evaluating the adherence to the entire pathway of treatment for CRC patients, with measures relating to diagnosis, surgery, medical treatment, and follow-up.[

Although each indicator describes a different and valuable portion of the pathways of care, there is a compelling need to develop composite measures to evaluate the degree of adherence on a global level.[ Several methods have been proposed in the statistical and economic literature to synthesize the indicators. This distinction stems from the use of latent and non-latent methods to define composite scores. Non-latent methods,[ such as all-or-none scores, are a weighted average of the individual indicators, which can be performed with either numerator-based weights (NBW) or denominator-based weights (DBW).[ On the other hand, latent methods suppose the existence of an unmeasurable variable – the “adherence to guidelines” – which determines the distribution of the indicators. The probability of achieving each indicator can be modeled using the item response theory (IRT) method.[ These models have been widely implemented before, especially for Pay-for-Performance programs,[ with the intention of rewarding hospitals that have a superior level of performance. Other applications can be found in the evaluation of care delivered in cases of acute myocardial infarction, congestive heart failure and community-acquired pneumonia.[ However, it is not equally common to find studies using the IRT method in the field of oncology.

The aim of our study was to evaluate the levels of adherence of CR pathways to the clinical guidelines provided at the hospital level, by comparing latent and non-latent methods, and to assess how each indicator discriminates against providers across various levels of adherence.

Methods

A retrospective cohort study was performed, including all new diagnoses of the colon and rectal cancer, occurring from 2007 to 2012, excluding non-epithelial cancers (i.e., cases of lymphoma and sarcoma), and cases identified by death certificate only (DCO), in the population of the Milan municipality. This geographical area is covered and accredited by the International Agency for Research on Cancer. It is further included in the Cancer Incidence in Five Continents – Volume XI,[ which continuously collects data on all new invasive cancers and covers the entire province of Milan with 3,480,513 inhabitants.

The study was carried out in accordance with the principles established in the Declaration of Helsinki. The study was not reviewed by the Ethics Committee, because it is a retrospective observational study based on cases routinely collected by the cancer registry of the province of Milan. The Agency for Health Protection (ATS) of the province of Milan is authorized by the Guarantor of Privacy to use this health-related data for analysis purposes. Anonymization was ensured by means of an internal code that is used in every administrative database belonging to the ATS of Milan.

In a recent project, 26 indicators have been constructed through a review of the literature and the use of previously described guidelines.[ Each indicator measures adherence to a recommended procedure for CRC patients. The indicators were calculated using specific algorithms, starting from CRC cases identified by the population-based cancer registry of Milan (Italy),[ which is linked with the healthcare databases. The indicators cover the entire pathway of treatment for CRC patients, from diagnosis (eight indicators), through surgery (seven indicators), medical treatment (seven indicators), and follow-up (four indicators). From this set of indicators, we selected the 10 measures with the highest level of correlation, according to Cronbach alpha as an estimate of internal reliability.[ The measures included are described in Table 1, together with a brief definition of the individual characteristics required for eligibility. For example, patients with peritonitis or bowel obstruction close to the date of diagnosis (within 30 days before or after diagnosis) were deemed ineligible for the indicator that evaluates the execution of pre-diagnosis endoscopies (D2). Adherence to guidelines is also captured by the time frame in which each the indicator was evaluated. For example, for diagnosis indicators (D1–D5), a patient was considered to be compliant if he/she received the diagnostic procedure within 6 months before, and 1 month after, the diagnosis. As a second illustration, we know that guidelines require a histological determination for each CRC patient receiving surgical treatment. For this reason, the indicator D6 identified the proportion of patients undergoing surgery, with a histological assessment in the 3 months prior to the surgical intervention. Finally, for the follow-up indicators (D7–D10), a period of 1 year after surgery was permitted.

Table 1

Selected indicators.

Provider accountability

The assignment of a patient to a single provider – which is considered to be the principal subject responsible for the care received – is a very difficult step because individual pathways of care are not uniquely associated with the same hospital. Chen et al[ addressed this problem by applying the plurality provider algorithm, which assigns a patient to the provider who billed the greatest number of care visits in a given year. For CRC patients, the fundamental step in the care process is represented by primary surgery. For this reason, in order to evaluate adherence to guidelines at the provider level, we associated each patient with the hospital where the first major surgical treatment was performed. If no operation had been performed, we associated the patient to the care provider where the endoscopy, chemotherapy or carcinoembryonic antigen (CEA) dosage were performed, in this order of priority. Patients who did not undergo any of these procedures were associated with the hospital where they were first admitted with a CRC diagnosis (ICD9 codes: 153–154).[ It is clear that different allocation methods will dictate different distributions of adherence across providers. However, this method ensured that, in cases where no surgery had been performed, the patient was associated with the provider where the first step of care was administered. Given that a healthcare organization can be composed of more than one structure or provider, we decided to perform the analysis at the provider level. However, the results are displayed as the estimated average adherence across providers that belong to the same organization. Furthermore, considering that the aim of the study was to compare organizations within the local health authority (ATS) of Milan, we chose to plot adherence levels only for those organizations that belong to the area governed by said ATS.

Statistical methods

Indicators of adherence for selected procedures

For each indicator, and for each provider, we have defined n as the number of eligible patients for the i-procedure, which had been administered by the j-provider, and y as the number of patients assigned to the j-provider who underwent procedure i (even if it was not administered by the j provider). For example, according to the reference guidelines, each patient should have a histological characterization of the tumor before their surgery. Therefore, we considered a patient with a biopsy record in the 3 months prior to the date of the surgery as being in compliance with this recommendation, regardless of whether it was performed by the same provider or not. The indicator measuring the recommended procedure, i, hereafter referred to as indicator i, for the j-provider is calculated as the proportion .

Composite indicators: non-latent methods

The DBW method assigns a weight to each of the 10 individual indicators for the proportion of eligible patients in the population:

On the other hand, the NBW method assigns a weight to each of the individual indicators for the proportion of patients receiving the associated procedure:

Providers with high levels of DBW or NBW will be considered to be highly adherent to guidelines.

The choice between DBW and NBW should depend on the aim of the analysis, and there are criticisms of both approaches. The former gives more weight to those indicators with a higher proportion of eligibility, while the latter assigns more weight to the indicators that have a higher proportion of compliance. According to Babbie,[ items should be equally weighted unless there are compelling reasons for differentiated weighting. If indicators share the same number of eligible patients across providers, then the DBW method is found to be most in keeping with this requirement. However, these methods do not deal with the correlation that exists between indicators, and their treatment of missing values is insufficient. In addition, they have been widely used to investigate constructs in formative models, where adherence is intended to be caused by the indicators themselves.[

Composite indicators: latent methods

In order to synthesize the information resulting from the 10 selected indicators, we used the latent IRT method. This model has primarily been employed in the fields of psychology and education.[ It has been used to measure abilities, by administering questionnaires composed of several questions – called items – with binary, polytomous, or categorical answers.[ Latent models assume the existence of an unmeasurable trait (e.g., the propensity to adherence), which is somehow connected to the observed variables (e.g., the indicators). IRTs assume a reflective construct, where items are caused by the latent variable, such that they are assumed to be independent.

This model attempts to measure adherence to guidelines θ (for j = 1, …, N) using the binomial variable y, with parameters n and p, such that:

where p is the probability of a patient receiving procedure i administered by provider j. Here, we assume that θ is normally distributed, with a mean of 0 and a variance of 1, that is, higher values of θ will describe more adherent providers. The model in Eq. (3) is referred to as the two-parameter Normal-Ogive model,[ where the latent trait is connected to the probability of receiving procedure i via two parameters, α and β. The parameter α is referred to as the discrimination parameter and represents the strength of the relationship between an indicator and the latent level of adherence of the j provider. The parameter β is referred to as the difficulty parameter and represents the median of θ for the i indicator. It has been proven elsewhere that there is a correspondence between the difficulty parameter, β, and the indicator's proportion of correct responses.[

For each indicator i, we can define an item characteristic curve (ICC), which is roughly equivalent to the regression of y (on the probit scale) on the distribution of adherence levels θ. The shape of each ICC is governed by the discrimination α, that is, the higher the value, the more the indicator discriminates against the adherence to guidelines of the underlying provider. The difficulty parameter shifts each curve on the horizontal axis, that is, it represents the minimum level of adherence for which the probability of receiving the procedure is higher than 0.5. In order to describe all providers, ranging from the least to the most adherent, a good set of indicators should cover the entire range of θ.

Providers with no eligible patients for the i-indicator (and, consequently, zero patients who are undergoing procedure i) were considered missing for the adherence evaluation for that indicator. Missing indicators are not treated insufficiently in IRT models. In fact, the IRT method is the perfect tool for handling data with missing values, as missing responses do not contribute to the evaluation of adherence.

IRT models presuppose different hypotheses for identifiability, such as monotonicity on the ICC, local independence among indicators, and the uni-dimensionality of the latent trait. The hypothesis of uni-dimensionality was tested using Cronbach alpha as an estimate of internal reliability.[ The choice of the selected set of indicators was based on a minimum item-total correlation value of 0.3.[ Higher positive values indicate the appropriateness of the item for the latent construct, that is, adherence to guidelines. The adequacy of the sample was evaluated using Kaiser measure.[ Values between 0.8 and 0.9 are considered to be adequate, while values below 0.5 are considered unacceptable.

The estimated adherence level was plotted using funnel plots, where the number of patients assigned to the corresponding provider was represented on the x-axis. Finally, we compared provider adherence levels, which were estimated using latent and non-latent methods, by means of Spearman rank correlation. All the analyses were implemented using PROC NL MIXED SAS/STAT software, Version 9.4, SAS Institute Inc., Cary, NC (see Supplemental Material, which illustrates IRT implementation in SAS).

Results

Description of the data

The cohort included 10,552 subjects with CRC incident in the period 2007 to 2012. We excluded 33 cases of lymphoma/sarcoma and 135 cases of DCO. We further excluded 710 patients that we could not associate with any provider from the analysis, as they had not undergone any of the following procedures – surgical treatment, endoscopy, chemotherapy or CEA dosage– and were never admitted to a hospital with a CRC diagnosis. The resulting cohort consisted of 9674 subjects (92% of the CRC patients included). The characteristics of the CRC patients in the original and the restricted cohorts were similar. The 710 people not included in the final analysis were mostly female (60%) and between the ages of 80 and 90 years (30%).

In accordance with the provider accountability method, 79% of the cases were associated with the hospital where they underwent their first surgical treatment, 13.4% with the provider where they underwent endoscopy, 3.3% with where they received chemotherapy, 2% with where they underwent CEA evaluation, and 2.4% with the hospital where their first admission with a CRC diagnosis occurred. The algorithm identified 135 providers, which were grouped into 68 organizations. Of them, 42 were in the Milan area and 26 were outside of it. Among those 42 organizations, 16 were laboratories associated with CEA evaluation. We presented the estimated adherence results for the remaining 26 organizations.

Indicators of adherence for selected procedures

Table 2 displays descriptive statistics for each of the selected indicators. Cronbach alpha and the item-total correlation were calculated for the subset of 89 providers without missing values. Cronbach alpha was 0.82, indicating a high level of reliability among indicators. All of the indicators included in the final analysis had an item-total correlation that was greater than 0.3, indicating a high level of overall appropriateness. Kaiser measure of sampling adequacy, based on the 89 providers without missing values was 0.61.

Table 2

Descriptive statistics of the 10 selected indicators and item response theory model parameters of difficulty and discrimination for each indicator.

Composite indicators

Descriptive statistics for each composite indicator are shown in Table 3. For each provider, both the DBW and the NBW were between zero and one. However, the DBW displayed a wider range of adherence values. Non-latent composite indicators were calculated for the set of 89 providers without missing values in any of the selected indicators, while adherence levels by IRT were calculated for the entire set of providers. Table 3 shows the Spearman correlation of provider rankings, based on latent and non-latent methods, for the set of 89 providers without missing values. The DBW showed a high level of correlation with the IRT, while the correlation was lower between the IRT and the NBW.

Table 3

Descriptive statistics and Spearman correlation among latent and non-latent composites indicators.

Figures 1–3 show the scatter plots of the distribution of mean adherence, calculated using the three methods (DBW, NBW, and IRT, respectively) and based on the total number of patients assigned to each organization. The organizations were classified as very small (<50 assigned patients), small (between 50 and 150 assigned patients), medium (between 150 and 500 assigned patients), and large (more than 500 assigned patients).

Figure 1

Scatter plot of the mean adherence of the 26 (non-laboratory) organizations in the Milan area, based on the total number of patients assigned to each organization. These are divided into very small (<50 assigned patients), small (between 50 and 150 assigned patients), medium (between 150 and 500 assigned patients), and large (more than 500 assigned patients), according to DBW. DBW = denominator-based weights.

Figure 3

Scatter plot of the mean adherence of the 26 (non-laboratory) organizations in the Milan area, based on the total number of patients assigned to each organization. These are divided into very small (<50 assigned patients), small (between 50 and 150 assigned patients), medium (between 150 and 500 assigned patients), and large (more than 500 assigned patients), according to the IRT method. IRT = item response theory.

Scatter plot of the mean adherence of the 26 (non-laboratory) organizations in the Milan area, based on the total number of patients assigned to each organization. These are divided into very small (<50 assigned patients), small (between 50 and 150 assigned patients), medium (between 150 and 500 assigned patients), and large (more than 500 assigned patients), according to NBW. NBW = numerator-based weights.

The distribution of adherence across providers was practically identical between the two non-latent methods, with limited variation in terms of mean adherence. A slightly different distribution was found with IRT. However, for each of the four categories of assigned patients, the uppermost and lowermost adherent organizations were fairly consistent between the three methods, that is, the most and least adherent organizations were almost identical. However, non-latent methods treat missing values in an insufficient manner. In fact, very small organizations, such as seven and 25, were missing for DBW and NBW, while their adherence could be evaluated using the IRT method. Table 2 shows the difficulty and discrimination parameters for the 10 indicators. The most difficult indicator to achieve was the fecal occult blood test (FOBT, D1), with β1 = 2.91, followed by all follow-up exams (D10), with β10 = 2.36. The easiest to achieve was follow-up imaging (D9), with β9 = −2.69. This is consistent with the information given in Table 2: that there is a correspondence between the difficulty parameters and the median of each indicator. The indicator evaluating all diagnostic exams (D5) had the highest degree of discrimination (α5 = 0.62), while the follow-up endoscopy (D7) had the lowest (α5 = 0.22). Figure 4 shows the ICCs for the 10 selected indicators. It can be seen that, for Endoscopy (D2) and Histological assessment (D6), the shape describing the probability of receiving the procedure is almost identical. This is in spite of the fact that Histological assessment (D6) had a stronger discrimination parameter. In fact, these indicators have a high level of correlation by definition, given that histological assessment usually results in an endoscopic biopsy. Moreover, the ICC curves for follow-up endoscopy (D7) and all follow-up exams (D10) appeared to be quite similar in terms of their discrimination parameters. This is because endoscopy is the most difficult follow-up exam to achieve (β7 = 1.76 vs β8 = −2.32 and β9 = −2.69), and hence, the majority of people undergoing all follow-up exams (D10) had also received a follow-up endoscopy (D7).

Figure 4

ICC of the 10 selected indicators (D1–D10, as described in Table 1). ICC = item characteristic curve.

Discussion

Our study presents a new methodological approach for evaluating the adherence of care at the individual level, but also among hospitals. Indicators evaluating a single recommendation are valid estimates of the adherence of care and each of them should be evaluated on an individual basis. However, within a decision-making framework, we often need composite measures to evaluate the degree of adherence among providers on a global level.

The use of composite indicators in the field of oncology appeared quite recently in the literature, and this can be considered to be one of the first works in this field. Chien et al[ used a simpler version of IRT[ to evaluate the quality of care for CRC patients, using 13 indicators that evaluated both surgical treatment (including pre- and post-assessments) and the colonoscopy before and after surgery. They presupposed equal levels of discrimination between indicators, while we hypothesized that each of these may have a different level of discrimination against the latent level of adherence. Furthermore, they measured the quality of care at the individual level, while we compared adherence at the provider level.

In this study, we used data from a population-based cancer registry to examine differences in the adherence of care among providers, by using indicators based on validated algorithms, established on the basis of healthcare data.[

However, this method posed some difficult decisions/questions, for example, how to assign providers to patients who underwent procedures in multiple hospitals. We used the provider accountability method to address this issue. In our opinion, the methods available in the literature (such as the plurality provider algorithm)[ are not appropriate for evaluating the administration of care to CRC patients, because they allocate the patient to the provider who handled the greatest number of visits, without considering the impact of each procedure on survival. Instead, we chose to adopt a method that closely resembled the typical management of every cancer patient. Another limitation could be found in the decision to use collapsing data about providers. A possible improvement could be obtained by defining a multilevel model for patients’ responses to each indicator, considering subjects clustered on providers.

We proposed an allocation method that associated each patient to the provider where, if the surgery had not been performed, the first step of care was administered. We further evaluated overall adherence via two non-latent methods and one latent method. The results showed a high level of overall agreement between the methods, especially between IRT and DBW, as previously observed.[ However, the IRT model is particularly interesting within a decision-making framework. It allows us to distinguish between higher-quality providers by choosing only the procedures with the highest capacity to discriminate, not including indicators with overlapping distribution across providers. For example, endoscopy and histological assessment evaluated providers in a similar manner. Hence, only one of them can and should be chosen to evaluate adherence to guidelines. A greater capacity to discriminate among providers could be attributed to CEA and all of the diagnostic exam indicators, followed by endoscopy, imaging, and histological assessment. The indicators regarding FOBT, follow-up endoscopy and all follow-up exams are characterized by a low level of adherence (<40% in the various providers), while follow-up CEA and follow-up imaging are characterized by a level of adherence that is consistently higher than 50%. According to the IRT method, only seven (out of 26) organizations had a negative level of mean adherence to guidelines (two providers, the 7th and 25th, have missing information for non-latent methods).

The distribution of the adherence across providers was quite similar, that is, we found a correlation greater than, or equal to, 0.6 between the IRT and DWB methods and the IRT and NBW methods. Greater differences were found in the providers closest to the mean level of adherence. Providers with the highest and lowest adherence values remained relatively consistent, except for the most adherent providers according to the NBW, but not the DBW and the IRT. Thus, we suggest adding a composite score, with IRT models being the preferred method, to globally evaluate providers in terms of their delivery of care that adheres to guidelines for CRC patients.

Finally, the results of this study require confirmation through further application, in order to validate the use of IRT methods in measuring adherence to guidelines. Furthermore, it would be helpful to verify its potential application, not only for other cancer types (breast and lung cancer), but also for non-cancerous, chronic diseases (cardiovascular, neurological and respiratory diseases and diabetes).

Author contributions

Conceptualization: Rossella Murtas, Adriano Decarli, Maria Teresa Greco, Anita Andreano, Antonio Giampiero Russo.

Data curation: Rossella Murtas, Maria Teresa Greco.

Formal analysis: Rossella Murtas, Maria Teresa Greco.

Funding acquisition: Antonio Giampiero Russo.

Investigation: Rossella Murtas, Antonio Giampiero Russo.

Methodology: Rossella Murtas, Adriano Decarli, Maria Teresa Greco, Anita Andreano, Antonio Giampiero Russo.

Project administration: Adriano Decarli, Antonio Giampiero Russo.

Resources: Antonio Giampiero Russo.

Software: Rossella Murtas, Maria Teresa Greco, Anita Andreano, Antonio Giampiero Russo.

Supervision: Adriano Decarli, Maria Teresa Greco, Anita Andreano, Antonio Giampiero Russo.

Validation: Rossella Murtas, Adriano Decarli.

Visualization: Rossella Murtas.

Writing – original draft: Rossella Murtas, Antonio Giampiero Russo.

Writing – review & editing: Rossella Murtas, Adriano Decarli, Maria Teresa Greco, Anita Andreano, Antonio Giampiero Russo.

Rossella Murtas orcid: 0000-0002-8989-7605.