Identifying multiple myeloma patients using data from the French health insurance databases: Validation using a cancer registry.

This study aimed to assess the performance of several algorithms based on hospital diagnoses and the long-term diseases scheme to identify multiple myeloma patients.Potential multiple myeloma patients in 2010 to 2013 were identified using the presence of hospital records with at least 1 main diagnosis code for multiple myeloma (ICD-10 "C90"). Alternative algorithms also considered related and associated diagnoses, combination with long-term conditions, or at least 2 diagnoses. Incident patients were those with no previous "C90" codes in the past 24 or 12 months. The sensitivity, specificity, and positive and negative predictive values (PPVs and NPVs) were computed, using a French cancer registry for the corresponding area and period as the criterion standard.Long-term conditions data extracted concerned 11,559 patients (21,846 for hospital data). The registry contained 125 cases of multiple myeloma. Sensitivity was 70% when using only main hospital diagnoses (specificity 100%, PPV 79%), 76% when also considering related diagnoses (specificity 100%, PPV 74%), and 90% with associated diagnoses included (100% specificity, 64% PPV).In relation with their good performance, selected algorithms can be used to study the benefit and risk of drugs in treated multiple myeloma patients.

Introduction

Multiple myeloma (MM) is the second most common hematological malignancy in France. [ In the last 2 decades, transplantation approaches and new drug regimens based on immunomodulatory drugs or proteasome inhibitors have considerably improved the survival of these patients.[ These patients are now essentially treated as outpatients. Hospital-based observational studies are then no more sufficient to study real-life practices and patients’ outcomes (adherence, among others.).

In parallel, researchers have access to the large French health insurance databases, covering >98% of the French population. French health insurance databases are potentially a valuable source for studying multiple myeloma epidemiology, healthcare use, and clinical outcomes, as it is among the rare automated databases in which certain hospital-administered medications are identifiable on an individual level. Indeed, the SNIIRAM (“Système national d’information inter-régime de l’assurance maladie”) gathers ambulatory and hospital data. Its potential for research is also in relation with its national coverage and the availability of details on long-term conditions.

To implement epidemiological or pharmacoepidemiological studies on these patients, the validity of the coding is of primary importance.[ As algorithms’ performance could be in many ways database-specific, there was a need to implement this validation study in French health insurance databases. A lot of previous validations were made with the ICD-9 in databases in the United States and validation studies are lacking for European and Nordic databases, in which ICD-10 is more frequent.[ Although several studies have measured the validity of cancer cases ascertainment in France,[ none focused on hematological diseases. Then, the validity of identification of multiple myeloma cases through these databases has not been previously established. This study aimed to assess the performance of several algorithms based on hospital diagnoses (PMSI, “Programme de médicalisation des systèmes d’ information”) and diagnoses from the long-term diseases (LTDs) scheme.

Material and methods

Setting and design

We conducted a population-based and retrospective validation study of MM case ascertainment through health insurance records, using the Tarn cancer registry as the reference standard. For the Tarn Cancer registry, the period of interest was 2010 to 2013 (all incident cases diagnosed during 2010–2013). In health insurance records, date of diagnosis is not available, and an “observation period” (12 or 24 months here) without any diagnosis of interest is required to discriminate incident (“new”) from prevalent patients. When no diagnosis of interest is recorded for up to 12 or 24 months, and then a first diagnosis occurs after this period, the patient is considered as incident. Then, for hospital diagnoses and long-term conditions, a longer extraction period has to be used (2008–2013) to enable at least 24 months (2008–2010) or 12 months (2009–2010) of observation (and use of the first occurrence of MM diagnosis after this period as a proxy of diagnosis date).

The Tarn Cancer registry

The Tarn Cancer Registry collects cancer data related to inhabitants of the Tarn area (about 400,000), an administrative area located in the southwest of France. Case ascertainment in the registry is based on systematic data collection from different sources: long-term diseases according to the health insurance schemes, hospital data for all residents of the Tarn area (all hospital data for the Tarn, plus data from hospital and reference centers in surrounding regions outside Tarn area), oncology regional network, pathology laboratories, hematology and cytology laboratories, all relevant hospital departments in public hospital or private clinics, radiotherapy centers, office from specialized physicians, and electoral registers.[ Diagnoses are coded according to the International Classification of Diseases for Oncology, 3rd Edition (ICD-O-3).[ The registry contains demographic details and some clinical or testing results. It also includes date of diagnosis attributed according international guidelines.[

Data from the registry were obtained for patients with hematological malignancies (ICD-O-3 topography code C42). Confirmed cases of multiple myeloma were patients diagnosed in 2010 to 2013 with ICD-O code “9732/3” for multiple myeloma, “9731/3” (for plasmacytoma), and “9734/3” (extramedullary plasmacytoma) in the registry. Clinical data were extracted for descriptive purposes.

Data from hospitalization stays PMSI

Data from hospitalization stays (PMSI) for the corresponding area and period (2010–2013 plus 2008–2009 or 2009 only for the observation period) were also obtained. Hospital data are managed within a single case-mix database of the activity-based payment system, (“tarification à l’activité,” T2A). Data provided came from medical, surgical, and obstetrics care (PMSI MCO). PMSI provides data on all claims paid by the national health insurance system (covering >98% of the French population[) to public and private hospitals. Main, related, and associated diagnosis codes are coded according to the 10th version of the international classification of the diseases (ICD-10).[ The data extraction was realized for hospital episodes involving a main diagnosis for cancer (ICD-10 “C” or “D”) or chemotherapy (“Z51”).

Diagnoses from “long-term conditions” scheme

Data from long-term conditions (LTDs) (affections de longue durée - ALD) for all patients with cancer (ICD-10 “C” or “D”) were extracted for the period 2008 to 2013, to enable at least 24 months (2008–2010) or 12 months (2009–2010) for the observation period. LTD provision is dedicated to patients suffering from a chronic condition which requires long-term treatment or expensive drugs. Healthcare expenses in relation to these conditions are fully covered. The list is established by decree (30 conditions), and include for instance malignant tumors, diabetes, or long-term psychiatric conditions. Diseases are coded according to ICD-10.[ Entry in the LTD is obtained following a request by a physician (often the general practitioner) and is not systematically requested, in particular when the patient is already in the scheme for another disease. However, it is a common practice for researchers working on French healthcare databases to use LTD in combination with hospital diagnoses to improve sensitivity of disease identification or to measure comorbidities (Charlson score, among others).[

Data collected

Data from both data sources were obtained as nonanonymized data. Linkage between both sources was done on the basis of combinations of 5 potential matching variables: family name, birth name, first name, date of birth, sex, place of birth (“commune,” lowest administrative area in France). Twenty-four possible combinations were tested. Unmatched patients were considered as having no hospital or LTD records during the period.

Nonanonymized hospital and LTD data have the same origin and structure as those contained in the national and anonymized health insurance database widely used for research (SNIIRAM). Combining hospital and LTD data at the local level is intended to simulate the performance of further algorithms that would be based on the SNIIRAM only.

Confidentiality and ethics

Data from hospitalization stays and long-term conditions were only those previously extracted for internal use of the Tarn Cancer Registry. All data were treated confidentially. The Tarn Cancer Registry is registered at the CNIL, French national data privacy institute (99 80 15 [12/1998], 99 80 15 version 2 [10/2003]).

Statistical analyses

Descriptive statistics were used to characterize the study population. Potential multiple myeloma patients in 2010 to 2013 were identified in the French health insurance databases using hospital records (PMSI) or LTD data. Indeed, the algorithm should use hospital data, but diagnoses are organized into 3 categories of diagnosis, and impacted by coding practices (main and related diagnoses are diagnoses of the current hospitalization, whereas associated diagnosis could be related to older episodes). When designing an algorithm, we have to include either main, main + related, or all types of diagnosis. As we did not have strong a priori on how these combinations will perform, we decided to test it systematically, for the 2 observation periods and with and without LTD to identify the combination providing the best performance. The algorithms tested began with a very straightforward approach (at least 1 main diagnosis), and then tested additional combinations and then cumulative diagnosis. Owing to the organization of the LTD scheme (long periods of coverage with start and end date), searching cumulative records was not relevant for this source.

In total, 13 algorithms corresponding to 3 strategies were tested with 2 different durations for defining incident patients (option A: 24-month observation period; option B: 12 months): Strategy 1, algorithms based on hospital data only (either main, main + related, or main + related + associated diagnoses), Strategy 2, algorithms based on hospital data or long-term condition (at least 1 long-term condition or either main, main + related, or main + related + associated diagnoses); and Strategy 3, cumulative diagnosis conditions for hospital data only (a 2nd MM main, main + related, or main + related + associated diagnosis, ≥30 days after the 1st). This cumulative condition was introduced with a temporal condition, as diagnoses belonging to the same episode (including transfers) are likely to be affected by the same potential coding error. True-positive patients were those ascertained as multiple myeloma cases in the registry (ICD-O code “9732/3,” “9731/3,” or “9734/3” in 2010–2013), and correctly identified as MM cases when applying the algorithm to health insurance data. True negative were those with no ICD-O code for MM in the registry, and not identified as MM cases according to the algorithm. False-positives were those not registered as MM cases in the registry (no corresponding ICD-O codes), but incorrectly identified as MM cases when using the algorithm based on health insurance data. False-negative were those ascertained as MM cases in the registry, but not identified as MM cases according to the algorithm. The sensitivity, specificity, positive predictive values (PPVs), and negative predictive values (PPVs) of the algorithms were then computed, using the cancer registry as the criterion standard. Exact binomial 95% confidence intervals were computed for each parameter. Youden index (sensitivity + specificity − 1) was computed as an indicator of model performance. Receiver-operating characteristic curves are provided as supplementary content. Considering further use in pharmacoepidemiological research, specificity was prioritized over sensitivity to reduce the potential impact of misclassification on risk estimates.[ Statistical analyses were performed using SAS 9.4 (SAS Institute Inc, Cary, NC). Method of validation was reported in accordance with the modified Standards for Reporting of Diagnostic Accuracy criteria.[ Concordance between date of diagnosis in the registry and first “C90”multiple myeloma code in LTD or hospital database was assessed using median time between date of diagnosis in the registry and first multiple myeloma “C90” code in number of days.

Results

Patients

For the period 2010 to 2013, the registry contained 125 incident cases of multiple myeloma (including 7 cases coded as plasmacytoma). According to the characteristics presented Table 1, median age was 74 (interquartile range, 63–81) and 57% were male (n = 71). Half of the patients were classified as stage III or IIIa according to the Durie-Salmon system.

Table 1

Characteristics of multiple myeloma patients in the Tarn cancer registry (N = 125).

Long-term conditions data recorded for the corresponding area concerned 11,559 patients for the period 2008 to 2013. Hospital data were available for 21,846 inhabitants of the Tarn area (2008–2013). Data from the registry were obtained for 1069 patients. Computations were then made on the joint population of both data sources, (i.e., 22,083), as 1 patient could be in >1 data source). Among the 125 MM patients in the registry, 115 (92%) had at least 1 matching record in hospital data, and 68 (54%) had at least one match with LTD records.

Sensitivity, specificity, and predictive values are reported in Table 2 (option A: 24-month observation period without diagnosis to select incident patients) and Table 3 (option B: 12-month observation).

Table 2

Algorithms performance when using hospitalization or long-term conditions (option A: 24-month observation period∗).

Table 3

Algorithms performance when using hospitalization or long-term conditions (option B: 12-month observation period∗).

Algorithms performance using both data sources separately (strategy 1)

From 2010 to 2013, 112 patients were identified as incident cases using main diagnoses from PMSI data (128 when using main and related diagnoses, and 177 with associated diagnoses) when using a 24-month observation period (option A, Table 2). Sensitivity was 70.4% (62.4%–78.4%) when using only main hospital diagnoses (specificity 99.9%, PPV 78.6%), 76.0% (68.5%–83.5%) when considering also related diagnoses (specificity 99.9%, PPV 74.2%), and 90.4% (85.2%–95.6%) with associated diagnoses included (99.7% specificity, 63.8% PPV).

Using a 12-month observation period (option B, Table 3) gave very close results, with similar sensitivity (90.4%) and slight differences visible for the PPV (59.5% vs. 63.8%), but with overlapping confidence intervals.

Impact of long-term conditions (strategy 2)

LTD alone exhibited very poor performance, with sensitivity around 55% whatever the period of observation used. In the algorithm considering a 24-month period to define incident cases (option A, Table 2), sensitivity was increased by up to 10% when incorporating long-term conditions to main hospital diagnoses. However, the interest of long-term conditions was attenuated after integrating associated diagnoses (+6%), and disappeared in all algorithms integrating associated diagnoses (same value for sensitivity for algorithm with main, related or associated code [90%], with or without LTD, 12 or 24 months’ period).

Using a 12-month observation period did not impact the performance of the algorithm (option B, Table 3).

Impact of the number of diagnoses required (strategy 3)

When a second diagnosis was required >30 days after a first diagnosis (24-month observation period; option A, Table 2), sensitivity dropped dramatically to very low values (19%). Impact of specificity was not observable as it was already maximal (99%) for the algorithm with only one diagnosis required. Among the 88 true-positive patients identified using the first algorithm (at least 1 main hospital diagnosis, 24-month period), 62% (55/88) have only 1 hospital diagnosis and 38% (33/88) ≥1 main hospital diagnoses.

The performance of this strategy for a 12-month observation period (option B, Table 3) was similar.

Impact of period of observation (24 vs. 12-month observation period)

There was no decrease in sensitivity and very slight reductions in PPV (<3%) as the observation window increased from 12 to 24 months (Table 3 vs. Table 2). Varying the window between MM codes and exclusion criteria did not improve algorithm performance. The algorithm using a 12-month observation period (option B) and “at least 1 main, OR related, OR associated hospital MM code” (strategy 1) exhibited the same highest performance (Youden's index: 90.1) as compared to the same algorithm with a 24-month period (Tables 2 and 3).

Exploration of diagnoses in false incidents

Using the first algorithm (at least 1 main diagnosis, option A: 24 months), 24 patients were classified as false-positive cases (patients misclassified as having MM). Among these false-positives, 2 were identified in the registry, with ICD-O-3 diagnoses of plasmablastic lymphoma (“9735/3”) and refractory thrombocytopenia (“9992/3”). All other false-positive patients were patients with hospital data, but not retrieved in the registry. When looking at their other hospital diagnoses, no other code for distinct hematological malignancies was retrieved. Two patients had diagnoses for bone metastasis (ICD-10 “C79”).

For the false-negative patients for algorithms using only main diagnoses (respectively, main or related), all appeared to be incident MM cases that would be selected when using either or related or associated hospital code (respectively, associated codes). Finally, almost all false-negatives had no available hospital records, and only 1 ascertained MM case had another hospital diagnosis (D46.2: refractory anemia with excess of blasts).

Exploration of delays between first hospital record and date of diagnosis in the registry

When considering time between diagnosis in the registry and date of first MM diagnosis in hospital data (Table 4), correspondence was high, with a median time of 0 days (interquartile range -2; 21 for main, related or associated diagnoses and a 24-month period), and 94% (83/88) of patients with a main diagnosis between 30 days before and within 1 year after registry documented MM diagnosis (72% between 30 days before and 30 days after, 63/88).

Table 4

Concordance between date of diagnosis in the registry and first multiple myeloma “C90” code in healthcare database.

Discussion

Main findings

Algorithms tested exhibited very different performances, ranging from poor performance when using only main hospital diagnoses to very acceptable parameters when hospital data are used in combination with long-term conditions diagnoses. The optimal algorithm to identify MM patients (maximizing both Youden index and specificity) was “at least 1 main, OR related, OR associated hospital MM code,” with a 12-month observation period, which had a sensitivity of 90%, a specificity of 100%, and a PPV of 60%. The same algorithm with a 24-month observation period demonstrated similar performance, but the algorithm with the shorter period of observation should be preferred. In the same way, one of the algorithms for strategy 2 performed equally well (“at least 1 MM main, related or associated hospital MM code) OR at least 1 LTD MM code”), but would require to have LTD data available. Faced to these 2 algorithms with equal performances, we choose the one requiring the minimum data (i.e., with no participation of data from the LTD scheme).

Indeed, the study design simulated the performance of algorithms that would be based on the large French health insurance databases (SNIIRAM) in further research. Using an algorithm with a restricted period of observation (12 months as compared to 24 months) has potentially a great interest for increasing sample size and length of possible follow-up in the context of limited longitudinal data availability (data are available since 2006 in the SNIIRAM).

Strengths and limitations

Some limitations must be acknowledged. First, this study was conducted in a single area and may not perfectly reflect the performance of hospital and LTD codes at the national level. However, coding is quite similar in all private and public hospitals. However, several data suggest that clinical and coding practices in Tarn region are very close to that of the whole nation. According to national estimations, 51% of newly diagnosed MM (or proliferative disorders) were male, with 70% aged >65 years. [ In our study, 58% were male and 67% were older than 65 years. Sex and gender characteristics are then quite close to the national reference. Clinical features of the patients are not expected to be different in France, but might be acknowledged at the larger scale (i.e., at the European level) owing to different delay to diagnosis. As we only used specific MM codes and not symptoms, the performance of our algorithm is not likely to be affected by the clinical aspect. Concerning coding practices for health insurance data, there are national standards of coding PMSI data, for the choice of the principal diagnosis for instance. Of course, we could not rule out various coding habits between hospitals or regions because of different interpretation of coding rules. This study is then conducted with the assumption of similar coding practices in all hospitals. However, in practice, quality of coding is regularly audited in public and private hospitals for reimbursement purposes.

Another potential limitation is related to the assumption that all patients with available data from the registry corresponding area and period should be considered as potential cases. In other words, failure of linkage with administrative data meant that the patient had no LTD or hospital record for the period of interest. However, even if there is systematic attempt to obtain cancer hospital data from all inhabitants of the covered area (Tarn), including hospitals outside the area of the residence, there is a possibility that hospital episodes were not complete, leading to underestimate the performance of the algorithm (increase in false-negative). Differences were observed when matching records from the cancer registry and the 2 data sources (21,846 for hospital data vs. 11,559 for LTD). These variations were expected. Indeed, although coding of hospital episodes will be always performed, LTD status is not automatically assigned to a patient and has to be requested by a physician. Then, the greater number of hospital records compared to LTD records should not be interpreted as missing information, but simply reflect the organization of the healthcare system.

In addition, the relative lack of sensitivity of the algorithm was expected owing to the particular natural history of multiple myeloma. Indeed, patients are not systematically managed in hospital, nor treated after diagnosis, because of possible asymptomatic or smoldering disease. Those patients are less likely to go to hospital or to enter in a long-term condition scheme. Lack of sensitivity for identifying these patients could be problematic for epidemiological purposes, but is acceptable for healthcare use or pharmacoepidemiological research, as these patients are not healthcare users.

In this study, we focused on the need to accurately identify myeloma patients (specificity), which would undergo additional selection criteria to be included in pharmacoepidemiological study for instance. Algorithms integrating treated patients would certainly have been more sensitive[, but we did not have sufficient information in the registry to ascertain this. Finally, lack of sensitivity is likely to be controlled in additional selection process, and the high specificity of the algorithm is an important strength here.

The false-positives for non-MM patients having at least 1 MM diagnosis code may be related to testing for disease rather than confirmed disease or to coding errors, or perhaps may be because of the evolution of diagnoses over time. There would be a potential interest to require not only one but several diagnosis codes (“cumulative diagnoses” strategy) to overcome this issue. Then, as expected, the increase in the number of diagnosis codes further improved specificity, but sensitivity reached unacceptable levels (<60%). In practice, MM patients may not be hospitalized several times after initial diagnosis (62% of confirmed MM cases has only 1 hospital diagnosis in our study), thus limiting the relevance of algorithms requiring multiple diagnosis codes.

Another consideration in relation to hematological malignancies is the possibility of lack of recording in the registry, as shown for myeloid malignancy in the United States.[

Other French experiences

Data from the French health insurance database have already been used for epidemiological identification of cancer,[ including for instance breast,[ colorectal,[ prostate,[ thyroid,[ or central nervous system malignancies.[ A work has also been implemented in French hospital data to select cancer related hospitalization, and myeloma was listed among the diseases of interest.[

International experiences

Validation of case identification algorithms represents an important issue, as demonstrated by recent calls,[ and also by several initiatives from Mini Sentinel and OMOP (Observational Medical Outcomes Partnership) in United States or EU-ADR in Europe.[ An important series of systematic review on methods for validating a wide range of disease, including lymphoma for instance,[ has been published since 2012.[ Lessons learned and proposal for improvement have been formulated during these validation studies.[ However, literature concerning multiple myeloma is very poor, and only one resource could be identified.[ According to this study, on MarketScan databases linked to Medicare claims databases, at least 2 diagnoses provided a sensitivity of 95%, a specificity of 73%, and a positive predictive value (PPV) of 76%.

Conclusions

This study revealed that including simultaneously main, related, and associated hospital diagnoses increased the sensitivity of the algorithm without generating excess false positives. The optimal algorithm to identify MM patients was “at least 1 main, OR related, OR associated hospital MM code,” with a 12-month observation period, which had a sensitivity of 90%, a specificity of 100%, and a PPV of 60%. This algorithm can be used in further pharmacoepidemiological studies for investigating the benefits and risks of drugs used by multiple myeloma patients.