Design of the New York City Macroscope: Innovations in Population Health Surveillance Using Electronic Health Records.

INTRODUCTION: Electronic health records (EHRs) have the potential to offer real-time, inexpensive standardized health data about chronic health conditions. Despite rapid expansion, EHR data evaluations for chronic disease surveillance have been limited. We present design and methods for the New York City (NYC) Macroscope, an EHR-based chronic disease surveillance system. This methods report is the first in a three part series describing the development and validation of the NYC Macroscope. This report describes in detail the infrastructure underlying the NYC Macroscope; indicator definitions; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. The second report describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.
METHODS: We designed the NYC Macroscope for comparison to a local "gold standard," the 2013-14 NYC Health and Nutrition Examination Survey, and the telephonic 2013 Community Health Survey. NYC Macroscope indicators covered prevalence, treatment, and control of diabetes, hypertension, and hyperlipidemia; and prevalence of influenza vaccination, obesity, depression and smoking. Indicators were stratified by age, sex, and neighborhood poverty, and weighted to the in-care NYC population and limited to primary care patients. Indicator queries were distributed to a virtual network of primary care practices; 392 practices and 716,076 adult patients were retained in the final sample.
FINDINGS: The NYC Macroscope covered 10% of primary care providers and 15% of all adult patients in NYC in 2013 (8-47% of patients by neighborhood). Data completeness varied by domain from 98% for blood pressure among patients with hypertension to 33% for depression screening. DISCUSSION: Design and validation efforts undertaken by NYC are described here to provide one potential blueprint for leveraging EHRs for population health monitoring. To replicate a model like NYC Macroscope, jurisdictions should establish buy-in; build informatics capacity; use standard, simple case defnitions; establish documentation quality thresholds; restrict to primary care providers; and weight the sample to a target population.

Introduction

In 2009, the Health and Information Technology for Economic and Clinical Health Act authorized more than $30 billion to stimulate the adoption and Meaningful Use of electronic health records (EHRs), triggering a dramatic rise in EHR use among office-based physicians from 17 percent in 2003 to 83 percent in 2014.1 To receive payment under Meaningful Use, Congress established mandatory utilization and quality reporting to the Centers for Medicaid and Medicare Services. By 2013, 62,928 U.S. providers reported blood pressure control among 17 million patients with hypertension.2 Meaningful Use also dramatically increased access to health information for local public health jurisdictions. Meaningful Use Stage 2 requirements included submission to a public health agency of electronic syndromic surveillance data, reportable laboratory results, and cancer registry data. As of October 2014, 90 percent of state and local public health agencies were able to accept at least one of these data types.3 With both data sharing and improved documentation incentivized by Meaningful Use, EHRs have growing potential to contribute real-time, rapid, and inexpensive standardized health data about chronic health conditions like diabetes and depression, as well as risk factors such as body mass index (BMI), blood pressure, and smoking status. The Centers for Disease Control and Prevention (CDC), state and local public health agencies, and academia have all began to recognize the potential of EHR data for monitoring the health of populations.2,4–6 However, guidelines on how to use these data streams have lagged behind rapid EHR adoption. Specifically, validation of EHR data for chronic disease surveillance has been limited.7,8 To our knowledge, EHR data have not been incorporated into large-scale chronic disease surveillance systems in the United States.

The Macroscope Electronic Health Record (EHR) Surveillance System

In 2012, with support from external funders and in partnership with the City University of New York School of Public Health (CUNY SPH), the New York City (NYC) Department of Health and Mental Hygiene (DOHMH) sought to test whether EHR data obtained from a convenience sample of more than 700 outpatient practices could be used to produce accurate estimates of population prevalence for NYC. This novel EHR-based surveillance system, the NYC Macroscope, was designed to measure health outcomes among the NYC adult population actively seeking medical care, defined as having visited a doctor in the reporting year of interest. Health outcomes included prevalence, treatment, and control of diabetes, hypertension, and hyperlipidemia; prevalence of smoking, obesity, and depression; and uptake of influenza vaccination. NYC Macroscope estimates were designed to be validated against the gold standard 2013–2014 NYC Health and Nutrition Examination Survey (NYC HANES) and the 2013 Community Health Survey (CHS).

This methods report is the first in a three part series describing the development and validation of the NYC Macroscope. This report describes in detail the infrastructure underlying the NYC Macroscope, indicator definitions, design decisions that were made to maximize data quality, characteristics of the population sampled, completeness of data collected, and lessons learned from doing this work. The second report describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension, and hyperlipidemia.

Methods

Reference Data Sources

Primary Reference Survey: 2013–2014 NYC HANES

The 2013–2014 NYC HANES was modeled after the national HANES and conducted by the CUNY School of Public HealthSPH and DOHMH. NYC HANES is a representative population-based survey that uses household-based sampling for in-person interviews, a brief physical examination, collection of specimens, and computerized self-interview. Full NYC HANES methodology is described by Thorpe et al.9 The final sample consisted of 1,524 NYC adults 20 years of age and older, of whom 1,135 reported having seen a provider for primary care services in the past year. Estimates were limited to this in-care population.

Supplemental Reference Survey: 2013 Community Health Survey

The CHS is an annual, representative, population- based, random-digit dialed telephone survey of adult New Yorkers, modeled on the Behavioral Risk Factor Surveillance System, and carried out since 2002 by DOHMH. Full CHS methodology is available online.10,11 The 2013 CHS had a sample size of 8,698, of whom 6,166 were 20 years of age and older and reported having seen a health care provider in the past year. Estimates were limited to this in-care population.

NYC Macroscope Data Source

EHR Data Network

The NYC Macroscope uses data from the Hub Population Health System (“the Hub”), a network of outpatient practices participating in the DOHMH’s Primary Care Information Project (PCIP). PCIP assists medical practices use EHRs and other forms of health information technology to increase the delivery of needed preventive care, track chronic disease, and improve disease management.12–15 As the Regional Extension Center for Health IT in NYC, PCIP collaborates with more than 20,050 providers and 1,600 practices using multiple EHR platforms. In 2011, PCIP partnered with the EHR vendor eClinicalWorks to create the Hub, a proprietary infrastructure allowing DOHMH to send questions to practices that have agreed to share data with PCIP.

One of the largest ambulatory care data networks in the country, the Hub collects data by sending SQL queries to connected practices, which automatically process them and send aggregate count answers back to a secure database overnight, without transmitting patient-identifiable data. The technical and operational details of the Hub are available from Buck et al.16 The Hub is used to collect data for quality improvement, program targeting, and public health surveillance, including the NYC Macroscope. Sample size varies over time, but as of 2014 when the NYC Macroscope was designed, 711 practices were connected to the Hub.

Indicators

To define NYC Macroscope indicators, DOHMH and the CUNY SPH convened a working group of epidemiologists and clinical experts. Seven domains were selected to reflect drivers of morbidity and mortality in NYC: hypertension, diabetes, cholesterol, obesity, smoking, depression, and influenza vaccination (see Table 1).17 Prior chart review indicated that mammograms and colonoscopies were infrequently captured in the EHR, so no cancer-related domains were included.18

Table 1.

Indicator Definitions in 2013–2014 NYC HANES, 2013 CHS, and 2013 NYC Macroscope

INDICATOR	TYPE	NYC HANES (n=1,135)	CHS (n=6,166)	NYC MACROSCOPE (n=716,076)	ALIGNED NATIONAL MEASURES
Smoking	Prevalence	Smoked 100 cigarettes in lifetime and currently smokes every day or some days	Smoked 100 cigarettes in lifetime and currently smokes every day or some days	Current smoker recorded in structured smoking section**	NQF 0028t, CMS138v1tt MUEPCOS1 13tttMUEPCOS2 5tttt
Obesity	Prevalence	BMI ≥30 from measured height and weight	BMI ≥30 from self-reported height and weight	BMI ≥30 from most recent height and weight* in vitals	NQF 0421, CMS69v3, MUEPCOS1 12, MUEPCOS2 4
Depression	Prevalence	Ever told had depression or had PHQ-9 score of 10–27		Ever had diagnosis of depression or ever had PHQ-9 score of 10–27	NQF 0418, CMS2v4
Influenza vaccination	Prevalence	Self-reported receipt of influenza vaccination*	Self-reported receipt of influenza vaccination*	CVX, CPT, or ICD-9 code indicating receipt of influenza vaccination*	NQF 0041, CMS147v4
Hypertension
	Prevalence of history/diagnosis	Ever told had hypertension	Ever told had hypertension	Ever had diagnosis of hypertension	MUEPCOS1 9
	Total prevalence: HANES gold standard	BP systolic ≥140 mmHg or diastolic ≥90 mmHg, or ever told had hypertension and is currently taking medication		Systolic BP ≥140 mmHg* or diastolic ≥90 mmHg* or ever diagnosed with hypertension with medication prescribed	MUEPCOS1 9MUEPCOS1 10MUEPCOS1 12MUEPCOS2 4
	Total prevalence: augmented	BP ≥140/90 mmHg or ever told had hypertension		Systolic BP ≥140 mmHg* or diastolic BP ≥90 mmHg*, or was ever diagnosed with hypertension or had medication prescribed	MUEPCOS1 9MUEPCOS1 10MUEPCOS1 12MUEPCOS2 4
	Treatment	Medication prescribed* among ever told had hypertension		Medication prescribed* among ever diagnosed hypertension	CMS68v4, MUEPCOS1 10
	Control	BP <140/90 mmHg among ever told had hypertension		Most recent BP <140/90* mmHg among ever diagnosed with hypertension	NQF 0018, CMS165v1
Hyperlipidemia
	Prevalence of history/diagnosis	Ever told had high cholesterol	Ever told had high cholesterol	Ever had diagnosis of high cholesterol	MUEPCOS1 9
	Total prevalence: HANES gold standard	Total cholesterol ≥240 mg/dl or ever told had high cholesterol and is currently taking medication		Most recent total cholesterol ≥240 mg/dl** or ever had diagnosis of high cholesterol with medication prescribed	MUEPCOS1 9MUEPCOS1 10MUEPCOS2 10
	Total prevalence: augmented	Total cholesterol ≥240 mg/dl or ever told had high cholesterol		Most recent total cholesterol ≥240 mg/dl** or ever had diagnosis of high cholesterol or medication prescribed	MUEPCOS1 9MUEPCOS1 10MUEPCOS2 10
	Treatment	Medication prescribed* among ever told had high cholesterol		Medication prescribed* among ever diagnosed with high cholesterol	CMS68v4, MUEPCOS1 10
	Control	Total cholesterol <240 mg/dl among ever told had high cholesterol		Most recent total cholesterol <240 mg/dl* among ever diagnosed with high cholesterol	MUEPCOS2 10
Diabetes
	Prevalence of history/diagnosis	Ever told had diabetes	Ever told had diabetes	Ever had diagnosis of diabetes	MUEPCOS1 9
	Total prevalence: augmented	A1c ≥6.5 or ever told had diabetes		Most recent A1c ≥6.5** or ever had diagnosis of diabetes or medication prescribed	MUEPCOS1 9MUEPCOS1 10MUEPCOS2 10
	Treatment	Currently taking medication among ever told had diabetes		Medication prescribed* among ever diagnosed diabetes	CMS68v4 MUEPCOS1 10
	Poor control	A1c >9 among ever told had diabetes		Most recent A1c >9** among ever diagnosed diabetes	NOF 0059, CMS122v3, MUEPCOS2 10

Notes: glycated hemoglobin (A1c); body mass index (BMI); blood pressure (BP); Patient Health Questionnaire (PHQ).

In past calendar year

In past 2 calendar years

National Quality Forum

Meaningful Use quality measure, from Centers for Medicaid and Medicare Services

Meaningful Use Eligible Provider Core Objective, Stage 1

Meaningful Use Eligible Provider Core Objective, Stage 2

Indicator definitions were crafted with three objectives: (1) reflect national EHR-based measures like Meaningful Use, which are aligned with regulatory compliance and reimbursement; (2) use standardized data elements present in most EHR systems; and (3) align NYC Macroscope indicators with NYC HANES and CHS definitions for comparability. Most indicators had a one calendar year “look-back” period; but infrequently collected measures-such as cholesterol and hemoglobin A1C tests-had a two calendar year look-back. Indicators were finalized before the 2013 publication of new hypertension and cholesterol treatment guidelines and advisories, and thus, these were not incorporated into definitions.19,20 Final indicator definitions were approved by internal DOHMH content experts and an external scientific advisory group.21 Final indicators included 42 comparisons; we present the 15 core indicators in Table 1.

To examine fit within subgroups, all indicators were stratified by three patient characteristics: (1) age (20–39, 40–59, 60–100 years); (2) sex (male, female); and (3) neighborhood poverty, defined as the percent of the population in the patient’s home ZIP code with an annual income below the federal poverty threshold (<10.0 percent, 10.0–19.9 percent, 20.0–29.9 percent, 30.0–100.0 percent). DOHMH created neighborhood poverty designations using 2008–12 American Community Survey ZIP code approximations.22 Neighborhood poverty was the only measure of socioeconomic status, as insurance status could not be reliably queried and patient income was not collected in the EHR. At the time of indicator design, patient race/ethnicity was poorly documented in the EHR and thus was not used.

Data Collection

For each indicator, SQL queries were drafted for each age, sex, and poverty stratum for both numerator and denominator. For example, the first query in the influenza series automatically requested the number of patients at each practice who had a visit in 2013, were 20–39 years of age, were of female sex, were in the lowest poverty group, and who had an influenza vaccine documented in the past calendar year via ICD-9 code, CVX code, or CPT code. In the query development and testing process, a total of 3,848 queries were written and run from 2013 to 2014, and 1,156 queries were run to create the final indicator set. To address random Hub practice nonresponse to queries on any given night, each query was scheduled two to three times for each practice, maximizing query returns. One large hospital outpatient department was unable to robustly return query results. For this system, a SQL programmer crafted a custom query script to generate the same aggregate results as queries run on the Hub, which the hospital ran in house and returned manually to DOHMH.

Primary Care Focus

The NYC Macroscope was designed to monitor the health status of patients seen by primary care providers. We restricted the NYC Macroscope to primary care providers to reduce the probability of double counting patients who visited more than one NYC Macroscope practice, to maximize the comparability of the patient sample to the average in-care patient in NYC, and to minimize the influence of provider specialty on prevalence estimates. We also found documentation to be more complete among primary care providers than in specialists in our sample. In excluding specialists, the NYC Macroscope could not capture patients who received their primary care in a specialty setting. A literature review on patients receiving primary care in both primary care and specialist settings yielded little. National surveys have found that about 60 percent of primary care services are delivered in the primary care setting, with an additional 9.8 percent delivered by internal medicine subspecialists and 3.6 percent delivered by obstetricians and gynecologists.23 The Medical Expenditure Panel Survey estimated that, in 2009, 75.9 percent of Americans had a primary care provider; the 2013 CHS estimated 80.7 percent of New Yorkers had a personal doctor.24 We accepted the loss of specialty care information because our aggregate query data precluded us from matching patients across providers.

Inclusion Criteria

Figure 1 presents the sample selection process used by the NYC Macroscope. Oueries were assigned to a total of 711 practices active on the Hub in 2014, of which 660 returned data for every query in the global denominator set, which was used to build all indicators. Each indicator required complete response to the full set of indicator-specific queries. Accordingly, each indicator had random variation in sample size. Sample results are described for the global denominator query set of the 24 demographic strata, including 1,317,438 patients with at least one visit in 2013 who were 20–100 years of age and were living in an NYC ZIP code.

Figure 1.

Inclusion Criteria and Sample Selection

The first inclusion criterion was primary care provider status, defined as internal medicine without a subspecialty, pediatrics, geriatrics, or family medicine. The impact of each of the documentation criteria is outlined in Figure 1. This removed 2,450 specialty providers, 189 practices, and 550,783 patients from the sample of 660 practices with NYC Macroscope denominator data. If a returning practice included specialists and primary care providers, data from specialists were not included. Only providers seeing ≥10 patients who were 20 years of age or older in 2013 were retained, to exclude inactive providers but include pediatricians seeing young adults.

The second set of inclusion criteria targeted data documentation quality. Because NYC Macroscope results are returned in aggregate, minimal data cleaning can be done. To improve data quality, we targeted providers who were consistently and uniformly documenting care in their systems. Retained NYC Macroscope providers had (1) documented an ICD-9 code diagnosis in either problem list or assessments for ≥80 percent of their patients in 2013, consistent with the threshold set by Meaningful Use Stage 1 Objective 9; (2) recorded either blood pressure or BMI for ≥50 percent, consistent with the threshold set by Meaningful Use Stage 1 Objective 12; and (3) prescribed a medication or documented a current medication for ≥20 percent.25 The medication threshold was set to capture adult prescribing patterns without excluding those pediatric or family medicine providers who were seeing young adults.26–29 The base sample for indicators that did not include EHR lab results was 392 practices, which included 987 primary care providers and 716,076 patients. These practices included 357 independent practices, 29 community health centers, 3 hospital outpatient departments, and 3 foster care agencies.

Indicators with an EHR lab result component used only practices with at least one electronic lab interface, which returned lab results as standardized data with unique Logical Observation Identifiers Names and Codes (LOINC). Using the Regenstrief list of the 2,000 LOINC lab codes most commonly used in ambulatory care, we identified and excluded providers with fewer than 10 patients with a common lab result in 2013.30 The base sample for lab result measures was 356 practices, 867 providers, and 703,508 patients.

Weighting and Estimate Generation

Provider-level aggregate data were pulled by the Hub data system, filtered using NYC Macroscope inclusion criteria, and converted to line-level data using Proc Freq in SAS software version 9.4 (SAS Institute Inc., Cary, NC). Records with missing outcomes were dropped (applicable to smoking and obesity only). To reduce the impact of patient and practice selection bias and to facilitate generalization to the NYC population in care, the line-level data were weighted for each indicator to the age, sex, and neighborhood poverty distributions of the NYC HANES and CHS populations in care. NYC HANES and CHS weight distributions were used to maximize the comparability of NYC Macroscope to each reference survey. We also tested the impact of weighting to the total versus the in-care population. Across 10 indicators, weighting to the total population had minimal impact (an average 0.23 difference in percentage points). Future NYC Macroscope estimates will be weighted to the CHS in-care population, but other jurisdictions could use census distributions.

Weighted NYC Macroscope population-based estimates were computed using SUDAAN 11.0 (Research Triangle Institute, Research Triangle Park, NC) using a sampling with replacement design, nested within practice to control for dependency among patients from the same practice. All estimates were age adjusted to the U.S. 2000 Standard Population.

Methods For Examining Completeness and Representativeness of Electronic Health Record-Based Surveillance Indicators

Representativeness of NYC Macroscope Sample

To describe the representativeness of the patient sample, we compared the unweighted NYC Macroscope age, sex, and neighborhood poverty distribution to the distribution of in-care adults across NYC, as measured by our larger reference survey CHS.31 We also mapped the neighborhood penetration of the NYC Macroscope among the in-care population, as estimated by CHS.32 Finally, using provider specialty information from SK&A Information Services (Irvine, CA), we identified primary care providers across the city and compared NYC Macroscope and non-NYC Macroscope providers by specialty type and size of practice. Significance of variation was assessed using the χ2 test. Results are presented in the Findings section of this paper.

Completeness of NYC Macroscope Data

We assessed data completeness overall and by substrata for NYC Macroscope patients. Complete data were defined by the presence of a required standardized data element, and free text data were not included. We calculated the rates of completeness for indicators where a missing data point could be identified: BMI, smoking screening, and depression screening among all patients; blood pressure among all patients and the subset with hypertension; total cholesterol lab result among those with hyperlipidemia; and A1c lab result among those with diabetes. Completeness results are also presented in the Findings section.

Findings

The 2013 NYC Macroscope total sample of 716,076 represented 15.3 percent of the 4.7 million adult New Yorkers estimated by CHS to have been in care in the previous year. The NYC Macroscope neighborhood coverage of NYC patients ranged from 8 to 47 percent, with >10 percent penetration in 30 of 34 neighborhoods (Figure 2). The NYC Macroscope had lowest patient penetration in predominantly white, affluent areas across the city (i.e., southern Staten Island, Upper East Side) and also in two lower income neighborhoods in Brooklyn. Sunset Park and Washington Heights, two predominantly Latino areas had the highest penetration (47.9 percent and 30.7 percent, respectively) due to a high concentration of practices participating in NYC Macroscope, potentially skewing the unmeasured racial breakdown of NYC patients. Areas officially identified by DOHMH as high-risk, high-need neighborhoods (designated as District Public Health Offices) had >15 percent patient penetration.

Figure 2.

NYC Macroscope Coverage of Adults in Care in NYC, 2013

Note: Macroscope penetration is calculated by dividing the adult patients at practices contributing to the NYC Macroscope living in each neighborhood by the total number of in-care adults estimated to live in that neighborhood in CHS 2013.

Practices contributing to the NYC Macroscope are represented by each yellow point.

Before weighting, the NYC Macroscope captured 5.9 percent more younger adults (20–39 years of age) and 6.1 percent fewer older adults (60–100 years of age) compared with the CHS citywide distribution of adults in care seen in Figure 3 (p<0.01). Sex breakdown in the NYC Macroscope is comparable to that of NYC adults in care, with <0.01 percent difference. The largest deviations appear in neighborhood poverty, with 12.2 percent more in-care adults living in neighborhoods with ≥20 percent poverty in NYC Macroscope compared with CHS estimates (p<0.01).

Figure 3.

Distribution of NYC Macroscope (Unweighted) Versus CHS Estimations of the Population in Care, 2013

From National Provider Identifiers, we identified 10,171 primary care providers in NYC, of whom 9.7 percent contributed data to the 2013 NYC Macroscope. In both NYC Macroscope and across NYC, most primary care providers practiced internal medicine or general practice (52.0 percent versus 49.2 percent, respectively) and geriatricians made up less than 4 percent. NYC Macroscope providers were significantly less likely than other NYC providers to be pediatricians (13.6 percent versus 26.6 percent) and more likely to practice family medicine (31.0 percent versus 21.3 percent). More NYC Macroscope providers worked in small sites of 1–5 providers compared with all NYC primary care providers (93.7 percent versus 87.9 percent).

Data Completeness in the NYC Macroscope

Completeness of data elements varied greatly by measure (Table 2). Significance is not reported due to large sample sizes (>100,000 in the smallest denominator). Depression screening using the Patient Health Questionnaire (PHQ-2 or PHQ-9) was infrequent and present in only 33.9 percent of NYC Macroscope records. Depression showed the widest variation in documentation by substrata, with more screening of younger patients, females, and people living in poor neighborhoods. Two thirds of NYC Macroscope patients had smoking status documented, with younger patients screened less frequently than older patients. BMI was measured for most NYC Macroscope patients (92.2 percent); only neighborhood poverty showed meaningful documentation variation. Annual blood pressure readings among patients with hypertension had the least missing data at 1.9 percent overall. Appropriate lab results from the previous two years were present for 73.4 percent of patients with diabetes and 76.7 percent of patients with hyperlipidemia. Validation findings for each measure are presented in the other reports in this series.

Table 2.

Completeness of 2013 NYC Macroscope Data

CONSTRUCT	OVERALL % COMPLETE	DIFFERENCE IN COMPLETION (PERCENTAGE POINTS)
		AGE: (20 TO 39) MINUS (60 To 100)	SEX: FEMALE MINUS MALE	POVERTY: (<10%) MINUS (>30%)
		Depression Screening	33.9	7.2	3.1	−14.6
Smoking Screening	67.9	5.8	−1.2	−2.3
BMI measured	92.2	−0.6	0.8	−3.5
Blood pressure in hypertension	98.1	−0.2	0.2	−0.7
A1c test in diabetes	73.4	0.6	0.3	−1.2
Total cholesterol in hyperlipidemia.	76.7	N/A	1.5	−1.5

Note: Findings regarding the validity of each measure are presented in two papers accompanying this publication.

Discussion

The NYC Macroscope was designed to evaluate the ability of a convenience sample of EHR data to produce meaningful population-surveillance estimates. It used data from 9.7 percent of primary care providers in the city, covered 15 percent of the NYC patient population, and had >10 percent penetration in most neighborhoods. However, significant population differences were seen in age and income, underscoring the importance of weighting EHR data to maximize generalizability. Missingness varied considerably by domain but had improved unilaterally since a Hub quality measure chart review in 2009,18 suggesting that documentation quality has improved in NYC in the context of Meaningful Use and Regional Extension Center support, a finding that we hope will be replicated elsewhere. Although our approach leveraged a unique public-private partnership to garner EHR data, our tactics to reduce bias, improve data quality, and align measures could be adapted by other jurisdictions to produce estimates from available ambulatory care data. In presenting our methods, using structured comparisons with data from two established reference surveys, we hope to make design decisions easier and quicker for the next developers.

Lessons Learned from the Design and Implementation of the NYC Macroscope

We learned through trial and error during the design and implementation phases of this project. We were faced with many limitations, including access to only aggregate data, lack of transparency around practice documentation and workflow habits, limited bandwidth of both programmer time and query scheduling on the system, a data collection mechanism that we did not own and could not change, limited access to socioeconomic variables of interest to measure bias, and volumes of data that challenged traditional statistical techniques. Some of these limitations are addressed in the next two reports in the series. Here we share the lessons learned from tackling our NYC Macroscope development challenges and suggest strategies that made our approach possible.

Building the Network: Buy-In Is Essential. Using EHRs for chronic disease surveillance inherently requires collaboration. Cultivating buy in at participating institutions is crucial. DOHMH has access to rich data streams partly because PCIP provides a host of services to practices, including training, monthly feedback via performance dashboards, Meaningful Use attestation support, technical assistance, EHR adoption support, and connection to new reimbursement programs and pilot models. Public health goals can only be met when provider concerns and goals are addressed. DOHMH benefited hugely from partnership with the eClinicalWorks vendor, which collaboratively developed the Hub and continues to maintain system infrastructure.16 PCIP initially subsidized eClinicalWorks licenses for safety net providers and gave them assistance in adopting the EHR platform, which increased buy-in from providers and gave eClinicalWorks a reason to collaborate on data collection.14 Mutually beneficial collaboration is one route to sustainable data sharing and may be even more essential when EHR data is collected without the help of the EHR software company.

Establishing Readiness: Build Informatics Capacity. DOHMH invested significant resources in developing both the Hub infrastructure and the NYC Macroscope surveillance system.16,21 True costs of development are unknown because the creation of the infrastructure was largely covered by eClinicalWorks. Initial query development required two programmers: one to write queries and one to review for accuracy. Ongoing NYC Macroscope operations require 15 hours a week from a skilled programmer or analyst capable of handling a large data volume. Other jurisdictions may be able to extend their capacity by cross-training staff or partnering with academic training programs.

Building the Metrics: Choose Simple, Standard Case Defnitions. Simple case definitions using standardized data elements like diagnosis, vitals, and labs may be more feasible than complex natural language processing (NLP) algorithms for jurisdictions with technical challenges and limited personnel. Our query system was not able to process NLP, and queries with too many elements “timed out” on some systems. Aligning simple case definitions with national measures makes communicating with stakeholders, policymakers, and providers easier while streamlining execution.

Improving Data Quality: Establish Documentation Quality Thresholds. Because of aggregate data limitations, we needed to maximize the quality of incoming data. We examined both our data and the literature when setting our documentation thresholds. The resulting estimates were less subject to misclassification and less vulnerable to selection bias using different practice cohorts. The NYC Macroscope convenience sample drew from unaffiliated outpatient practices across a large city, and practice data entry workflows varied. Although all PCIP practices had common opportunities for coordinated quality-improvement support like dashboards and technical assistance from the DOHMH Regional Extension Center, the heterogeneity of 660 unaffiliated practices is likely larger than for data collection mechanisms tapping large health systems with significant market penetration, as may be the case in smaller jurisdictions. However, in jurisdictions with similarly heterogeneous samples without the ability to identify documentation patterns, inclusion criteria requiring providers to have attested to Stage 1 or Stage 2 Meaningful Use may be effective whether data is received in aggregate or line-level format.

Reducing Sample Variation: Restrict to Primary Care Providers. We were able to reduce heterogeneity in data quality and documentation trends by limiting our sample to primary care providers. This was especially important because our aggregate data precluded post hoc data cleaning and examination. Other jurisdictions without the ability to match patients may also benefit from a provider restriction to limit double counting.

Enhancing Generalizability: Weight the Sample to the Target Population. As anticipated, our patient population differed significantly from all NYC adults in care. Weighting by as many factors as are available in the reference and EHR data source alleviates selection bias. Ultimately, we found that weighting to the total versus patient population had minimal impact. Flood et al. have applied a double weighting schema that addresses missing data.6 This may be fruitful to include in the future. A chart review of a subsample of EHRs from NYC HANES participants will examine differences between NYC Macroscope patient records and other patient records and will provide additional lessons regarding generalizability.33

The NYC Macroscope is the first municipal chronic-disease EHR-based surveillance system in the United States, but it will not be the last. When we began this work, Massachusetts was to our knowledge the only other jurisdiction to attempt to incorporate EHR data into chronic disease monitoring, in partnership with a major academic medical-research institution.34 The field has since advanced significantly, with projects in Denver, San Diego, Chicago, and Massachusetts.34–37 CDC launched the Healthy Weight Surveillance Initiative focused on using EHRs to explore obesity and its risk factors.38 Healthcare Information and Management Systems Society (HIMSS) and the National Association of County and City Health Officials (NACCHO) have partnered to create the Public Health & Health Information Exchange (HIE) Toolkit.39 The Robert Wood Johnson Foundation has funded the Center for Health and Information Technology to identify and promote health data sharing across sectors.40 This collective body of work is shaping new methods in chronic disease surveillance.

The rapid expansion of access to large and rich streams of electronic health care data will help advance EHR-based surveillance. The Standards and Interoperability Initiative by the Office of the National Coordinator on Health Information Technology (ONC) currently has pilots across four states that are exploring how query platforms similar to the Hub data system could be layered on HIEs in any jurisdiction to capture any case definition in standard data shared in a continuity of care document.41 Regional HIEs in places like Rochester, N.Y. and Tulsa, Okla. are reaching high levels of population penetration and are developing internal analytics capacity that could be leveraged for public health.42–44 As Meaningful Use continues to push EHR adoption, and reimbursement models like Accountable Care Organizations and Patient Centered Medical Homes incentivize fluid data exchange, EHR data will be accessible in more jurisdictions. Our forthcoming study examining sensitivity and specificity of our indicators across different EHR platforms by Meaningful Use attestation status will further clarify the generalizability of our findings to these new data collection modalities.53 Together with the results of our validation presented in this series, the design of the NYC Macroscope yields one blueprint for harnessing a convenience sample of ambulatory care EHR data for population health surveillance.