Validity of Methods to Identify Individuals With Lower Extremity Amputation Using Department of Veterans Affairs Electronic Medical Records.

Objectives: To determine the positive predictive value (PPV) of algorithms to identify patients with major (at the ankle or more proximal) lower extremity amputation (LEA) using Department of Veterans Affairs electronic medical records (EMR) and to evaluate whether PPV varies by sex, age, and race. Design: We conducted a validation study comparing EMR determined LEA status to self-reported LEA (criterion standard). Setting: Veterans who receive care at the Department of Veterans Affairs. Participants: We invited a national sample of patients (N=699) with at least 1 procedure or diagnosis code for major LEA to participate. We oversampled women, Black men, and men ≤40 years of age. Interventions: Not applicable. Main Outcome Measure: We calculated PPV estimates and false negative percentages for 7 algorithms using EMR LEA procedure and diagnosis codes relative to self-reported major LEA.
Results: A total of 466 veterans self-reported their LEA status (68%). PPVs for the 7 algorithms ranged from 89% to 100%. The algorithm that required a single diagnosis or procedure code had the lowest PPV (89%). The algorithm that required at least 1 procedure code had the highest PPV (100%) but also had the highest proportion of false negatives (66%). Algorithms that required at least 1 procedure code or 2 or more diagnosis codes 1 month to 1 year apart had high PPVs (98%-99%) but varied in terms of false negative percentages. PPV estimates were higher among men than women but did not differ meaningfully by age or race, after accounting for sex.
Conclusion: PPVs were higher if 1 procedure or at least 2 diagnosis codes were required; the difference between algorithms was marked by sex. Investigators should consider trade-offs between PPV and false negatives to identify patients with LEA using EMRs.

It is estimated that over 1 million people in the United States are living with a lower extremity amputation (LEA), including approximately 58,000 veterans receiving care at Department of Veterans Affairs (VA) facilities., People with amputation require long-term and specialized care to optimize their reintegration into society. The number of people living with an amputation is expected to double by 2050. As the number of people with amputations grows, we must be able to accurately identify this population to assess health outcomes, provide high-quality care, and dedicate the appropriate amount of resources.

Electronic medical records (EMRs) can be a wealth of information for researchers. These data allow researchers to conduct large studies at lower costs and study rare conditions and specific subpopulations. However, there are limitations to EMRs. Because coding is influenced by billing incentives, the ability of a coder to interpret a clinician's notes, and diagnostic accuracy, miscoding and omissions may be present. Given these limitations and the value of using EMRs for research, it is critical to validate algorithms to identify populations of interest.

Prior studies validating EMR codes against a criterion standard have shown wide variability in positive predictive values (PPVs), a measure of the ability of codes to accurately identify the intended condition or procedure.5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 Sources of variability in PPVs include the type of diagnosis, quality control measures, reimbursement model, as well as other factors. Conditions with the highest PPVs were procedures such as knee replacement and hip replacement and common conditions such as asthma., Algorithms for serious conditions, such as sickle cell disease, also resulted in high PPVs when part of the algorithm accounted for number of visits and hospital admission. Diagnoses and events that had a short list of potential codes, such as palliative care, also had high PPVs. Those with lower PPVs included events that could vary in severity, such as nonsteroidal anti-inflammatory drug–related upper gastrointestinal events, or conditions requiring multiple tests and/or repeated testing, such as hepatitis B, for which a diagnosis may require results from 3 separate tests and often a set of follow-up tests 6 months later., Additionally, for conditions with varying severity, such as early onset dementia and posttraumatic stress disorder, algorithms often yielded the highest PPVs for those with other specific comorbidities., Individuals with early onset dementia and posttraumatic stress disorder but without specific comorbidities were often found to be misclassified. For diagnoses and events with long lists of potential codes, such as diabetes, complex algorithms were required to achieve high PPVs.

Although validation studies have been conducted for other conditions, we are not aware of any research validating the use of EMR codes for LEA. To more efficiently and effectively study veterans with LEA, it is important to evaluate the PPV of different algorithms to determine which approach minimizes false positives (those who have codes for LEA but do not truly have an LEA) and maximizes true positives. Understanding how PPV may vary by sex, age, and race is similarly important. Much LEA research has included predominately men., Given the increasing number of female veterans and accordingly female veterans with LEA, it is important to examine how PPV may vary by sex. Furthermore, the distribution of amputation etiology differs with age, with traumatic amputation more prevalent in younger age groups and dysvascular amputation more prevalent in the elderly.,,, Thus, it is important to examine how PPV may vary by age. Research also tends to focus on populations that are majority white, despite studies indicating that Black individuals have higher amputation rates.,, To ensure that Black/African American individuals are not underrepresented in future research because of differences in coding, it will also be valuable to evaluate differences in PPV by race.

The potential misclassification of LEA impairs our ability to accurately identify the intended population and conduct research that draws meaningful conclusions. Furthermore, it limits our ability to perform surveillance of LEA and to accurately report the prevalence and incidence of amputation. Thus, the objectives of this study were to estimate the PPV of algorithms used to identify patients with an LEA using the VA EMR overall and by age, sex, and race. This work can be used to inform the choice of algorithm(s) for identifying patients with LEA using the VA EMR and the EMR of other health care systems.

Methods

Study design and study population

We conducted a validation study to determine the PPV of International Classification of Disease (ICD, diagnosis and procedure) and current procedural terminology (CPT, which include only procedures) codes to identify LEA. Diagnosis codes included initial encounter, subsequent encounter, history/status, and sequelae. Codes for conditions that may have led to LEA were not included. Ascertainment of “true” LEA status was by self-report. Subjects were eligible if they had at least 1 LEA diagnosis or procedure code (see Supplemental Appendix S1, available online only at http://www.archives-pmr.org/, for codes) in the VA's EMR (the Corporate Data Warehouse [CDW]) between October 1, 2005, and September 30, 2018, and were alive at the time the data were pulled. The CDW is a national database that aggregates VA clinical, administrative, and financial information from across all VA sites. For this study, LEA included having a diagnosis or procedure code for any major LEA (at the ankle or more proximal), including those related to peripheral artery disease, diabetes, and trauma. We selected 700 patients with at least 1 LEA code for inclusion in the study. We oversampled women, younger men (<40 years of age), and Black men to make our PPV estimates in these subgroups more precise (fig 1). The VA Puget Sound Institutional Review Board reviewed and approved this study.

Fig 1

Selection of sample, recruitment, and number who completed the survey.

Data collection

We determined “true” LEA status via a self-administered mailed survey. Self-report is a reliable criterion standard for diagnoses and procedures that are objective and easy to self-determine. We mailed everyone a letter inviting participation in the survey, an “information sheet” (a document like an informed consent form for studies with a waiver of documentation of written informed consent that describes the purpose of the study, risks and benefits, and how to opt out of further contact), and the survey. The survey asked participants to indicate yes/no to the question, “Do you have a lower-extremity amputation?” for both the left and right legs. A stamped, addressed return envelope was included in the mailing to facilitate participation. Two weeks after the first mailing, we called nonresponders. At the time of the call, we gave individuals the opportunity to complete the survey over the phone. We sent a second mailing to nonresponders 2 weeks after the first call. We made a final attempt at contacting the remaining nonresponders 2 weeks after the second mailing via phone call (see fig 1). Covariate data, including age, sex, race, and CPT and ICD codes, were extracted from the CDW. The study protocol is available from the corresponding author.

LEA algorithms

We evaluated 7 algorithms for identifying patients with LEA using the EMR (table 1). Algorithm 1 (at least 1 diagnosis or procedure code) was evaluated because it is a commonly used approach in amputation research. Algorithm 2, which counts only procedure codes, was chosen because procedure codes are typically used to identify incident amputations and because procedure codes tend be more accurate (fewer false positives) than diagnosis codes based on others’, and our own (unpublished) research. Algorithms 3-6 were generated based on the logic that we would have more confidence that a person had an LEA if they had more than 1 encounter with an LEA diagnosis code in the EMR on different dates. Requiring 2 or more diagnosis codes on different days is an approach recommended for identifying people with diabetes, for example. Algorithm 7 was generated by examining the LEA codes from the parent study, which involved qualitative interviews in which a surprisingly large number of women reported not having an amputation despite having LEA diagnosis and/or procedure codes. From this study we suspected that history codes (eg, V49.75, V49.76, Z89.519, or Z89.619) might be more likely to be erroneous, so they were excluded from the code set.

Table 1

PPV and FNP of algorithms used to identify lower extremity amputation in Veterans Health Administration electronic health records with a self-reported gold standard.

Number	Algorithm	PPV (%)	False Negative (%)*
1	Any code	89	0†
2	At least 1 procedure code	100	66
3	At least 1 procedure code or 2 or more diagnosis codes	98	2
4	At least 1 procedure code or 2 or more diagnosis codes on different days	98	2
5	At least 1 procedure code or 2 or more diagnosis codes at least 30 days apart	99	6
6	Two or more procedure codes or 2 or more diagnosis codes at least 1 year apart	99	28
7	Any code except a single “status” code*	98	2

Status codes are defined as ICD-10 codes beginning with Z89 or ICD-9 codes beginning with V49. This algorithm treats those with exactly 1 status code and no other code as not having LEA, whereas those with 2+ status codes or 1+ nonstatus code(s) are considered to have LEA.

The FNP for algorithm 1 is 0% by study design; see text for details.

Data analysis

We conducted statistical analyses using Stataa and R version 4.0.2.b Demographic variables (age, sex, race) and amputation-related information (number of diagnosis and procedure codes) were summarized, stratifying by amputation status. For each algorithm, we calculated the PPV and the false negative percentage (FNP). The PPV was calculated as the number of people who self-reported an LEA among those identified as having an LEA by the algorithm (true positives) divided by the total number of people identified as having LEA by the specified algorithm (predicted positive). The FNP was calculated as the number of patients identified as not having an LEA by the algorithm who self-reported an amputation (false negatives) divided by the total number of patients who self-reported an LEA. Because the sample only included those with at least 1 diagnosis or procedure code, the FNP for algorithm 1 was 0% by design. Confidence intervals were generated using the percentile method with 5000 bootstrap replicates. We present overall estimates and stratum-specific estimates. For the overall estimates, we reweighted the data back to the source population (patients with at least 1 LEA code) by calculating the stratum-specific PPV (eg, PPV for women, Black men≤40 years, Black men>40 years, and so on) and weighting that PPV by the relative size of the stratum. To determine the associations between sex, race, age, and probability of self-reported LEA, given a diagnosis or procedure code for LEA in one's medical record, we fit a logistic regression model with age, sex, and race as covariates. We inferred that there was an independent association between the factor and the probability of self-reported LEA if covariate-adjusted P values for the factors were <.05.

Results

A total of 32,998 individuals met inclusion criteria; 98% of individuals were male. Of the 700 individuals we contacted, 16 individuals were later determined to not be eligible (see fig 1). Fifteen of these individuals had a household member reporting them as deceased. One other individual was removed because no procedure or diagnosis code for lower limb amputation was present in the EMR upon re-review. Codes change because the VA EMR is updated regularly and overwriting of past codes can occur. Of the 684 remaining individuals, 466 completed the survey for a response proportion of 68% (table S1, available online only at http://www.archives-pmr.org/). Individuals aged 55-84 years (vs <40 years) were more likely to respond, as were women relative to men and those with a race other than Black vs Black veterans. The distributions for number of amputation diagnoses and procedure codes were similar among those who did and did not respond.

Among responders, those who self-reported having an amputation were more likely to be <40 years old, be male, be Black, and have a higher number of procedure and diagnosis codes (see table 1). Specifically, whereas 40% of those who self-reported having an LEA were women, 85% of those who self-reported not having an LEA were women. Additionally, among those who self-reported having an LEA, only 3% had a single diagnosis or procedure code, whereas 86% of those who did not have an LEA had a single code.

PPVs for the 7 algorithms ranged from 89% to 100% (table 2). Algorithm 1 (which required a single diagnosis or procedure code) had the lowest PPV (89%). Algorithm 2 (which required at least 1 procedure code; if a person had only diagnosis codes, they were not considered algorithm positive) had the highest PPV (100%) but also had the highest proportion of false negatives (66%). Algorithms 3-7 had similar high PPVs (98%-99%) but varied in terms of FNP. Algorithms 3, 4, and 7 had FNPs of 2%, Algorithm 5 (which required at least 1 procedure code or 2 or more diagnosis codes at least 30 days apart) had an FNP of 6%, and Algorithm 6 (which required 2 or more procedure codes or 2 or more diagnosis codes at least 1 year apart) had an FNP of 28%. When we tested the associations of sex, age, and race for likelihood of self-reporting an amputation, sex (odds ratio for men vs women=8.4, P value<.001), but neither age (odds ratio for ≥40 vs <40 years=0.6, P=.15) nor race (odds ratio for Black vs non-Black race=0.7, P=.15) were significantly associated with self-reporting an LEA.

Table 2

Characteristics of survey respondents by self-reported amputation status (n=466).

Characteristics	Self-Reported Lower Extremity Amputation
	Yes (n=338)		No (n=128)
	N	%	N	%
Age, in years
25-39	105	(31)	14	(11)
40-54	35	(10)	23	(18)
55-69	114	(34)	48	(38)
70-84	69	(20)	38	(30)
85+	15	(4)	5	(4)
Women	134	(40)	109	(85)
Black race	116	(34)	38	(30)
Number of diagnosis and procedure codes
1	10	(3)	110	(86)
2-3	24	(7)	13	(10)
4-19	110	(33)	3	(2)
20+	194	(57)	2	(2)
Number of diagnosis codes
0	2	(1)	1	(1)
1	10	(3)	110	(86)
2-3	24	(7)	12	(9)
4-19	110	(33)	4	(3)
20+	192	(57)	1	(1)
Number of procedure codes
0	238	(70)	126	(98)
1	21	(6)	0	(0)
2-3	55	(16)	1	(1)
4+	24	(7)	1	(1)

We also assessed the performance of the 7 different algorithms in sex, race, and age subgroups (tables 3 and 4 and table S2, available online only at http://www.archives-pmr.org/). Among men, all algorithms gave high PPV estimates, ranging from 92% to 100% (see table 3). The distribution of estimates was wider among women, ranging from 55% to 96%. For women, algorithm 2, which required at least 1 procedure code, had the highest PPV estimate (96%; 95% confidence interval, 89-100), but this algorithm also had 74% false negatives.

Table 3

PPV of algorithms used to identify lower extremity amputation in Veterans Health Administration electronic health records with a self-reported criterion standard stratified by sex.

Algorithm Number*	Sex	Self-Reported Amputation	Algorithm Positive†	PPV (%)	95% CI	False Negative (%)
1	Women	134	243	55	49-62	0‡
1	Men	204	223	92	88-95	0‡
2	Women	46	48	96	89-100	74
2	Men	54	54	100	—§	66
3	Women	130	146	89	84-94	3
3	Men	199	201	99	98-100	3
4	Women	129	141	92	87-96	4
4	Men	199	201	99	98-100	3
5	Women	122	130	94	90-98	9
5	Men	195	196	100	98-100	4
6	Women	114	122	93	87-97	31
6	Men	181	182	99	97-100	28
7	Women	130	161	81	75-87	3
7	Men	200	202	99	98-100	2

Abbreviation: CI, confidence interval.

See table 2 for algorithm descriptions.

Algorithm positive is defined as meeting the criteria for the specified algorithm.

The FNP for algorithm 1 is 0% by study design; see text for details.

Not reported because there was no sampling variability in the bootstrap estimates.

Table 4

PPV of algorithms used to identify lower extremity amputation in Veterans Health Administration electronic health records with a self-reported criterion standard stratified by age.

Algorithm Number*	Age in Years	Self-Reported Amputation	Algorithm Positive†	PPV (%)	95% CI	False Negative (%)
1	<40	108	122	89	83-94	0‡
1	≥40	230	344	67	62-72	0‡
2	<40	18	18	100	—§	83
2	≥40	82	84	98	94-100	64
3	<40	105	106	99	97-100	3
3	≥40	224	241	93	90-96	3
4	<40	105	106	99	97-100	3
4	≥40	223	236	95	92-97	2
5	<40	105	106	99	97-100	3
5	≥40	212	220	96	94-99	8
6	<40	76	76	100	–	30
6	≥40	164	169	97	94-99	29
7	<40	106	109	97	94-100	2
7	≥40	224	254	88	84-92	3

Abbreviation: CI, confidence interval.

See table 2 for algorithm descriptions.

Algorithm positive is defined as meeting the criteria for the specified algorithm.

The FNP for algorithm 1 is 0% by study design; see text for details.

Not reported because there was no sampling variability in the bootstrap estimates.

For all algorithms, differences across race in PPVs and FNPs were only 1-4 percentage points, suggesting that the algorithms performed equally well among those who were and were not Black (table S2). Conversely, PPVs were consistently higher among those <40 vs ≥40 years of age, but for all algorithms other than algorithms 1 and 7, those differences were small (2-6 percentage points, see table 4). For algorithm 1, there was a 22 percentage point difference (89% for <40 years and 67% for ≥40 years) and for algorithm 7, there was a 9 percentage point difference (97% for <40 years and 88% for ≥40 years).

Discussion

To our knowledge, this is the first assessment of the accuracy of EMR-derived LEA diagnosis and procedure codes. PPV estimates were higher among men than women but did not differ meaningfully by age or race, after accounting for sex. Requiring a single LEA diagnosis or procedure code when the population of interest is limited to men is likely acceptable; however, this approach will incorrectly identify many women because only 55% of women who had a single LEA diagnosis or procedure code self-reported an LEA. PPV estimates for women may be lower because of the lower prevalence of LEA in women compared to men, because prevalence affects PPV. It may also be affected by factors not examined in this study, such as amputation etiology or coding practices.

The evaluation of the algorithms demonstrated trade-offs between PPVs and FNP. The best approach for identifying patients with LEA from EMRs will depend on the goals of the research and the ramifications of (possibly differential) misclassification. In our sample, 78% of respondents did not have a procedure code, likely because the amputation was not performed in the VA and therefore the procedure code was not captured in the VA EMR. Thus, our findings suggest that requiring a procedure code will maximize PPV but at the expense of missing patients whose amputations were performed outside VA. Requiring at least 1 procedure code or 2 or more diagnosis codes or using any procedure or diagnosis code except a status code maximized PPV (98%) while only misclassifying 2% of patients. Thus, either approach may be a good choice, but more stringent approaches (for example, requiring 1 procedure or 2 or more diagnosis codes at least 30 days apart) may be preferable if the target population includes a substantial proportion of women, there is a high cost to erroneously identifying someone as having an LEA when they do not, and a low cost of excluding patients who do, in fact, have an LEA.

Study limitations

Because we did not identify a completely random sample (regardless of LEA codes), we were unable to calculate sensitivity, specificity, and negative predictive value. However, because LEA is relatively rare, the probability that an individual who does not have a code for LEA truly does not have an LEA (the negative predictive value) is likely to be high. Furthermore, we expect the FNP estimated in this study to be biased upward because it is theoretically possible that there are patients who would self-report LEA without any record of it included in the CDW at the time of the data pull. Nonetheless, we expect this number to be small and would be equivalent to adding a constant to the denominator of the FNP formula for each algorithm, which still allows for comparisons between algorithms. Estimating the PPV and FNP allowed us to identify which methods are most accurate for identifying patients with LEA. A second limitation was the potential for nonresponse bias. To reduce nonresponse, we contacted each individual up to 4 times and ultimately obtained information on 68% of those approached. Responders and nonresponders were similar in the number of LEA codes, but they did differ somewhat in terms of age, sex, and race. To further assess the effect that nonresponse may have had on our results, we calculated PPV estimates overall and for each subgroup for algorithm 1, assuming that all nonresponders had LEA and alternatively assuming that all nonresponders did not have LEA. Based on these simple imputations, PPV estimates remained higher in men than in women, indicating that nonresponse could not explain the differences by sex seen in our results. A third possible limitation is reliability of self-report. Although it seems unlikely that people would report not having an LEA when they in fact did, this is possible. Therefore, conducting a future study using detailed medical record review or physical exam as the criterion standard would be valuable.

Conclusions

In summary, we found that requiring 2 or more diagnosis codes or ignoring status codes improved the PPV and negligibly increased the FNP. PPVs were generally higher in men than in women. The differential misclassification by sex warrants investigation in a future study. It will also be valuable for future studies to determine whether these finding are generalizable to populations outside the VA.

Suppliers

a. Stata, StatCorp LP. b. R version 4.0.2; The R Foundation.

Disclosures

None.