Medicaid medical directors quality improvement studies: a case study of evolving methods for a research network.

OBJECTIVE: To describe the evolution of methods and share lessons learned from conducting multi-state studies with Medicaid Medical Directors (MMD) using state administrative data. There was a great need for these studies, but also much to be learned about conducting network-based research and ensuring comparability of results.
METHODS: This was a network-level case study. The findings were drawn from the experience developing and executing network analyses with the MMDs, as well as from participant feedback on lessons learned. For the latter, nine interviews with MMD project leads, state data analysts, and outside researchers involved with the projects were conducted. Interviews were transcribed, coded and analyzed using NVivo 10.0 analytic software.
FINDINGS: MMD study methodology involved many steps: developing research questions, defining data specifications, organizing an aggregated data collection spreadsheet form, assuring quality through review, and analyzing and reporting state data at the national level. State analysts extracted the data from their state Medicaid administrative (claims) databases (and sometimes other datasets). Analysis at the national level aggregated state data overall, by demographics and other sub groups, and displayed descriptive statistics and cross-tabs.
CONCLUSIONS: Projects in the MMD multi-state network address high-priority clinical issues in Medicaid and impact quality of care through sharing of data and policies among states. Further, these studies contribute not only to high-quality, cost-effective health care for Medicaid beneficiaries, but also add to our knowledge of network-based research. Continuation of these studies requires funding for a permanent research infrastructure nationally, as well as at the state-level to strengthen capacity.

Introduction

The Medicaid Medical Directors (MMDs) formed a learning network in 2005 to address collectively topics of high importance for the Medicaid population. As clinical leaders in states as well as leaders in many state-level quality improvement efforts, it is significant that these MMDs came together to think collectively about problems important to all of them. Support initially came from the Agency for Healthcare Research and Quality (AHRQ) through a contract with AcademyHealth.

In 2007, the MMDs began conducting quality improvement studies as a network, looking at indicators of quality in the Medicaid population in the participating states. Three studies have been conducted to date. The first study examined antipsychotic medication use in children and adolescents;1 the second examined hospital readmissions;2 and the third, on early elective deliveries, is still ongoing. These studies gave Medicaid programs an assessment of the prevalence, burden, and cost of these problems. A companion commentary examines the benefits and importance of these studies to quality improvement in Medicaid.3 That paper and this one join an emerging commentary on collaborative or network-based research.4,5

The purpose of this paper is to describe the evolving methods for conducting multistate projects informed by the three MMD network studies, to present lessons learned, and to provide recommendations for future network-based projects. With the increasing number of research networks at both the state level and beyond – for example, the Medicaid Evidence Based Decisions Project (MED), as well as the Patient Powered Research Networks and the Clinical Data Research Networks funded by the Patient Centered Outcomes Research Institute – the topics probed in this paper are especially timely. This paper not only explores methods for collecting and analyzing data (administrative and claims data in this case), but also addresses methods in a broader sense, many of which are relevant to networks using electronic health data. This broader view of methods encompasses organizing and managing the research process, and making choices and compromises necessary for network-based projects designed to improve health care quality.

Background

This network of MMDs is unique in several respects, among them is its size and scope. Medicaid is the nation’s largest payer, with 62 million beneficiaries in 2009,6 and it will grow larger still with the implementation of the Affordable Care Act (ACA). The data available to this network is also unique: namely, state-level administrative claims data in the individual states, sometimes linked with other state-based data (e.g., vital records in the examination of early elective deliveries). These claims databases are comparable, but not identical, in the various states, and this diversity in organization and data specification adds to the complexity of conducting distributed data studies in the MMD network. Added to these features of the network is the fact that these databases are the province of the states, and for this reason state-level studies are needed to gain access to them. National data sets constructed from state-level claims exist, but currently are available only after a significant time lag. Thus, current, actionable data needs to come from the states themselves. A lack of readily available national data has contributed to a paucity of national studies of the Medicaid population. In contrast, the availability of a national data set in Medicare has led to numerous national studies of this population.

These features of the MMD network and the state-level data give rise to an enhanced level of urgency around conducting these studies. The relative scarcity of robust evidence base for Medicaid gives heightened importance to these studies. Furthermore, the sheer size of the network as a whole means that improvements in utilization and quality have the potential to generate large savings and improve quality of care for the nation’s most vulnerable individuals.

Further, as electronic medical records come into greater use and interoperability features enhanced, the MMD network will be able to use this and other cutting-edge electronic data methods for monitoring quality in Medicaid. However, although there is great need and great potential for these network-based studies as evidenced by the Patient-Centered Outcomes Research Institute’s (PCORI) recent investments, there are significant methodological challenges to be dealt with, many of which stem from the fact that the states themselves need to supply the data. For example, there are tensions between depth versus breadth in data collection.

To ensure broadest participation, data requests need to be straightforward so that even states with limited resources for analysis can respond to them readily. The streamlined data request means a loss of some level of robustness in the data collected from the states. Further, and importantly, due to the diversity in states’ administrative database organization and data specification, there are challenges in ensuring the collection of comparable and high-quality data from states.

There is a paucity of studies that set forth both benefits and challenges of network-based studies. This paper seeks to fill this gap by identifying some important lessons learned over the course of these three studies for how best to conduct network-based quality improvement projects in the MMD network as new topics arise and data sources – for example, electronic medical records (EMRs) – become accessible.

Methods

Methods for conducting this case study of network-based quality improvement studies included a review of reports and other documents describing the MMDs’ studies; nine in-depth phone interviews with MMD project leads, state data analysts, and outside researchers involved with the projects; and the authors’ own experiences working with the projects. Interviewees were selected because they were MMD leaders in the projects, state level analysts working with the leaders, or actively involved.

Interview guides were structured to elicit information on respondents’ experience with the MMD studies, their views on the methods for developing topics, specifying data, gathering data, and reporting results. They were also asked about the policy importance of these studies in their own state and nationally (if pertinent). Interviews were transcribed, independently coded by two researchers, and analyzed using NVivo 10.0 analytic software. The authors of this paper, none of whom is an MMD, participated in the organization and coordination of the MMD network projects (KG), and in the analysis and reporting of the studies, including preparing a paper for publication (TT, GF).

Findings

Overview of Protocols for Conducting Studies Led by MMDs

In all of the MMD network studies, the general methodology involved developing research questions, identifying variables needed to address the research questions, defining data specifications for these variables, and organizing these data in spreadsheet form. State analysts then pulled the variables from their state databases, according to the specifications, and inputted data in the spreadsheets with aggregated state-level information. Analysis at the national level involved aggregating the state data overall and by specified subgroups. It is important to emphasize that only aggregated data were sent forward from the states, not individual-level line data.

The following sections summarize a standard approach applied for all three studies to address the following:

How the topics were chosen;

How the research questions and data specifications were developed;

How state-level data were aggregated; and

How results were reported.

Table 1 summarizes these features of the overall strategy and methods for conducting each of the three studies.

Table 1.

Strategies and Methods Used for the Three MMD Projects

	Antipsychotic (AP) Medication Use Study and Follow-Up	Hospital Readmissions	Early Elective Deliveries
Participation (States; enrollees)	16 states; 12 million Medicaid children and adolescents; 9 states in follow-up	19 states; 1.8 million Medicaid admissions	22 states; almost 1 million Medicaid births accounted for on claims and birth certificates
Topic selection	MMD Champion noting the increased use of AP medications in children, which can be detrimental to health and costly.	MMD Champions noting the high cost of readmissions and the evidence that at least some of them are preventable.	MMD Champion showing the growing realization of level of adverse outcomes for elective inductions, especially early ones (<39 weeks).
Data specifications developed by	MMD leader, state analysts, and university researchers	AHRQ, MMD leaders, and state analyst	MMD leaders, CMS contractor, and AcademyHealth researchers
Data specifications based on	Psychotropic Medication Utilization Parameters for Texas Foster Children	HCUP analysis	Epidemiologist’s prior work
State-level databases used	Medicaid claims	Medicaid claims	Vital Statistics data and Medicaid claims
Data Analyzed by	University researchers and AcademyHealth (Follow-up)	AHRQ and AcademyHealth	AcademyHealth
Report	Full report, resource guide, fact sheet for each state	Full report, journal article, fact sheet for each state	Journal article(s) (in preparation), fact sheet for each state and data chartbook (both pending)

Note: MMD=Medicaid Medical Director; AHRQ= Agency for Healthcare Research and Quality; CMS= Centers for Medicare and Medicaid Services

Each of the projects the MMDs have conducted garnered more participation from states than the previous studies. For example, for the antipsychotic medication use study, 16 states participated in the initial study and 9 in a follow-up. For the readmissions study, 19 states participated. And, 22 states submitted data for the perinatal care study. (See Table 1.) There could be various reasons for increased participation, including a better understanding of the purpose and importance of multistate projects, better project organization and staff support, and stronger partnerships with stakeholders. However, as discussed below, enhanced state-level resources would enable even fuller participation by the states and more inclusive reports.

Topic Selection

MMDs were routinely asked about their priority topics, and there was general agreement about what topics were hot at the state level (e.g., usually those that resulted in high costs or poor quality of outcomes). However, the decision to conduct research on a particular topic from the several most important ones turned to a champion among the state MMDs who then took the lead in overseeing the study.

For the antipsychotic medication use in children project, the champion from Washington State identified this as an emerging issue for state Medicaid and foster care agencies. For the hospital readmissions project, the state MMDs in Pennsylvania and Colorado took the lead in overseeing the project. At that time, a readmissions measure designed specifically for the Medicaid population did not exist, and there were no studies on readmissions in the whole Medicaid population. For elective deliveries, the MMD in Ohio championed the topic, and then co-led the project with the MMD from Minnesota. Ohio had already targeted reducing elective deliveries as a state priority and had organized an in-state learning collaborative about this topic (The Ohio Perinatal Quality Collaborative).7 MMDs agreed that this fairly informal process of topic selection worked well for their purposes.

Developing Research Questions and Data Specifications

For all three projects, the development of the research questions and data specifications has been similar. The MMDs first formed a work group to oversee the project. This group, usually led by the champion(s), developed the questions and topics of interest. Developing the research questions entailed formulating the questions themselves, as well as specifying subgroups. For example, in the readmissions project, the work group specified that to answer their research questions, readmissions were to be examined by the following: major diagnostic category, age group, gender, race, and payer type.

In all three cases, state analysts and outside study researchers then worked with MMD leadership to refine the topics into specific researchable questions and to develop the data specifications. For the antipsychotic medication use project, the MMD in Washington took the lead in developing the specifications with a work group of other interested states and university researchers. For the read-missions project, a state analyst/economist in Pennsylvania took the lead after working closely with the AHRQ on defining read-missions. For the perinatal project, study researchers at AcademyHealth worked with analysts in Ohio and a research firm contracted by the Centers for Medicare and Medicaid Services (CMS).

In all cases, these projects built on key prior work and prior definitions used in other studies. In the antipsychotic medication use project, definitions for the measures analyzed were based on the Psychotropic Medication Utilization Parameters for Texas Foster Children.8 For the readmissions project, the co-leads worked closely with AHRQ to define readmissions and test the definition using Healthcare Cost and Utilization Project (HCUP) data before developing the specifications.9 And finally, for the perinatal project, a definition and code for “elective deliveries” developed by an epidemiologist and his team at the University of South Florida and the Florida Department of Health were used. The code specified which deliveries were medically indicated, using responses on the birth certificate.

Data specifications for all studies were reviewed by several states, refined, and reviewed again. Getting the variable definitions correct was crucial to the success of the project, and much care was taken at this stage. In some cases new definitions needed to be developed for key variables, such as what constituted a hospital readmission or early elective delivery. In all cases, in addition to defining the key variables, covariates were specified that often included demographic characteristics of the population and payment mechanisms (such as managed care or fee-for-service).

The MMDs were aware of the limited time and resources at the state level – their own time, as well as time required by state analysts to conduct the studies – and were acutely aware that complicated data requests would limit the number of states that were able to participate. Thus, MMD leadership consistently designed streamlined data requests that could be accommodated by most of the states, even though this approach meant the data set might lose some level of detail that other states would find helpful. They called the streamlined data-collection request the “bicycle model” in the hospital readmissions project to emphasize its simplicity, which contrasts with the more complicated “Mercedes model” for states that wanted to do more. They provided specifications for the bicycle model; asking merely for a 30-day readmissions rate by some basic demographics, major diagnostic categories, and cost. The specifications did not request a more sophisticated analysis (i.e., the Mercedes model) that was risk adjusted with multiple time frames. Still, despite simplification, developing data specifications was a detailed, difficult and time-consuming task. In all three cases, study researchers assisted to ease the burden on the MMDs, while still enabling them to oversee and direct the work.

After the specifications were developed, spreadsheets were created for states to fill in with data to ensure standard submission across states. The spreadsheets indicated that the states were to summarize outcome variables (e.g., use of antipsychotic medications, readmissions, or early elective deliveries, depending on the study) by desired covariates. For the study of hospital readmissions, for example, spreadsheets were set up to display readmission rates by diagnosis, age, and race/ethnicity.

For all of the studies, some level of pilot testing was conducted or at least attempted. The importance of pilot testing the data request before it was sent out to the states to complete was clearly understood by the project team. The hope was always to pilot the data collection specifications and forms with several states; however, some of the pilot testing was more successful for some than for others. Time and resource constraints sometimes prevented a comprehensive pilot from being done.

State-Level Data Aggregation and Technical Assistance

States used the specifications and the variable definitions provided to collect data and input data into the spreadsheets provided. To do this, state analysts needed to do the following:

Interpret the specifications;

Understand the questions;

Pose questions to the study researchers around ambiguities;

Write the code for constructed variables; and

Run the analyses.

The amount of time for this step during the elective deliveries project was estimated to be at least 20–30 hours by one of the experienced analysts involved in the project, even with the streamlined data request. Depending on the analyst’s familiarity with the topic and the structure of the data, some may have needed more or less time. (As will be discussed more fully, the level of effort can be more than some states are able to devote to these projects, which in turn leads to more limited participation.)

Effort was required on the part of study researchers at this stage as well. Because the states constructed their own data warehouses, and because variables could be specified in slightly different ways in each state, there was a need for considerable back-and-forth between MMD leads, researchers, and state data analysts. This step was crucial for the success of the project and for obtaining comparable data that followed the specifications from the various states.

In the first and second projects, assistance to states was limited to all-state calls and an exchange of emails and individual state calls with project leads. This was due partially to the lack of resources allocated to these projects for researcher support; MMDs and their analysts took on the technical assistance roles. This limited assistance likely contributed to the few data inaccuracies identified during the analysis. By the time the third project was underway, the critical importance of technical assistance was fully realized with financial support for a study researcher to dedicate half of her time for a period to working with the states.

Need for Quality Assurance of Data

The importance of intensive quality-assurance testing was fully recognized and implemented by the time the third study was carried out. This was done as part of the initial analysis after states submitted their data. As part of this process, the researcher aggregated the data; calculated a mean, median, mode, minimum and maximum for all variables; and flagged data points that did not seem reasonable and were far from the mean. States were then contacted in a timely manner and asked to double check numbers of particular variables and correct any errors.

It was critically important to do this quality assurance step as early as possible in the process. The more time that elapsed between when the data were submitted and when the analyses began, the less likely it was that data issues could be resolved quickly. Time was also required for the state analysts at this stage to discuss with outside researchers issues that arose, and, if necessary, rerun the data if problems were uncovered. State analysts were extremely responsive at this stage and were eager for their data to be as accurate and comparable as possible.

The need for quality assurance flowed, to a large extent, from the fact that the data organization and variable specification at the state level need to be comparable, but not exactly the same, from state to state. Strategies for maximizing comparability in distributed data like these have gained attention recently.10–12 The process we used adhered to these recommended strategies. Although this process does not absolutely guarantee accuracy, because the quality check is for whether a data point is out of range, it does maximize the likelihood of securing correct and comparable data.

How Results Were Reported

Researchers aggregated the state-level data, and then worked with MMD leadership and publishing committees of state staff to select important findings to share and include in reports. The reports included simple aggregations at the national level of state-level results, as noted above.

For the first study on antipsychotic medication use a lengthy report and resource guide1 was written by a publication committee of state colleagues and Rutgers University. This included not only the aggregated results of states’ data, but also detailed descriptions of states practices rated as emerging, promising, or mature.

One MMD noted that “it is important not to just crunch numbers, but to have policies and best practices.” The MMDs also decided that they want to ensure widespread dissemination of the results. Thus, for the readmissions project, a manuscript was written and submitted for publication. For the perinatal project, plans are in place to write a manuscript and provide a chartbook of the complete data results to post online. One MMD said, “it is really important to publish our…work…so that people will know that we are out there and…start talking about what they think of this and do a follow up analysis based on that data, especially because [after ACA] we are going to be covering so many people.”

Importantly, for all three studies, a brief fact sheet showing national-level results was prepared for broad distribution, and separate fact sheets were prepared for each state, for use within the state. The latter showed the state results compared to the multistate aggregate. States have been uniform in their belief in the importance of these summaries for each state indicating that “the [fact sheets] are very valuable to the Medicaid Director and the program.” All materials with states’ aggregated data are shared with the states participating in the projects for review before they are shared more widely. This ensures states are comfortable with the information being distributed.

Because in all cases, researchers were working with aggregated state data, and not individual-level data, no individual-level adjustments were possible. However, data were displayed in a bivariate fashion for at least some of these factors. For example, in the hospital readmissions study, readmission rates were displayed separately for age groupings, gender, and payer source.

Discussion

As a result of conducting these studies, the states and study researchers have learned valuable lessons about organizing and conducting these studies for building an infrastructure for future projects. Some of the main lessons include: importance of MMD leadership; research support infrastructure needed at national and state levels; value of involving stakeholders; strategies for maximizing the quality and consistency of state-level data; and timing and resource levels needed to conduct the studies. Key aspects of these lessons learned are described below.

Importance of MMD Leadership

One clear lesson from all of the studies involved the importance of the MMD leadership. The leadership in these studies must reside with the MMDs, because a given MMD can organize peer MMDs from other states, and understand the medical issues involved and the state-level policy context. One MMD explained, “[MMDs] should always lead, because they are on the ground” and “understand what is going on in the state.” Another added that, “[MMDs] are the ones who are interested in using this [data] to make policy decisions.”

The MMD leaders also have an important role in securing participation from other states as well as securing buy-in from outside stakeholders; they are uniquely positioned to describe the benefits authentically. One MMD explained it this way, “MMD leaders need to be in front [of these projects]…[not only] to give the guidance, but also to get buy-in. It’s a lot easier for [one of the MMDs] to go to [national associations] and do the promotion while [others] are behind the scenes doing the work and the development.”

In addition, the experience with these projects points to the need for co-leaders – to complement each other’s strengths, to have two people to have a dialogue about ideas, and to provide backup when the other is busy. For the two most recent studies, there were two MMD co-leaders.

The role of the state analysts is also critical. They understand how their data warehouses are constructed and understand statistics. They have also played a key role in developing specifications and piloting data collection for these projects. Their knowledge of the data and understanding of the analysis is an essential component of the projects’ success.

Value of Stakeholder Involvement

For two of the three projects, there was collaboration with related associations. These partnerships were invaluable as they allowed access to data that may not have been available to Medicaid previously, whether they be mental health or vital statistics. In addition, support from various stakeholders provided state agencies with the encouragement needed to at least consider participating in these projects.

For example, because the multiple collaborating organizations contacted their constituents (e.g., state vital records registrars) about the perinatal project, almost three-quarters of the states indicated whether or not they intended to participate. By comparison, in other studies the indication of willingness to participate came when they submitted data, and not before.

High Level of Research Infrastructure Required to Take Burden from States and Maximize Data Quality

While the MMDs need to retain leadership, these are time-consuming, technically difficult projects, and it is essential to design a research infrastructure that takes burden from the states. “This is not their day jobs” was a comment often heard referring to the MMDs who have taken on the leadership of the projects and are guiding the studies. While individual states and their analysts developed the specifications for the readmissions project, these MMDs did not believe that this level of effort was feasible in the future, given that states will be involved with implementing the ACA and responding to the needs of a new and likely chronically ill population.

One MMD said flatly, about the prospect of assuming a leadership role in a future project, “I could personally not do that…I just do not have the time…with the ACA and all of the other things we have to do in the next year and a half to two years.” This MMD added that he would highly recommend additional resources to help with this effort.

Thus, there was general agreement on the need to continue to build infrastructure to make participation easier for states, including having outside study researchers –take the lead in developing the variables and data specifications and to be available to answer questions from states as they were pulling and aggregating the data. MMDs said they should always be in the lead, but not actually involved in the [data] specification. Another MMD added, “specifications, I don’t think [MMDs] need to do it as long as they are very actively involved in the questions and overseeing....”

The research infrastructure also needs to be designed so that there is support for state analysts as they are pulling, analyzing, and aggregating the data. This is in part because state data warehouses have different organizations and can have different variable specifications. Even though the specifications were developed with participation from various states, there were always questions about exactly how a given state would need to interpret the variables. Piloting the specifications in a few states is an important step to help work out issues with the variable definitions or data specifications. To date, however, this was not always feasible in the time allocated for the projects, and piloting sometimes happened later than needed. However, even with piloting, each state has its own intricacies within data warehouses, policies, payment structures, and other state-specific differences that may affect the way in which data is pulled. One analyst pointed out that, “there are a lot of nuances in how data are handled [in states].”

Further, the research infrastructure must include support for generating high-quality and comparable data. Infrastructure can be improved upon by looking at similar distributed models and methods and using their best practices, as well as by building a sustainable governance structure. In addition, states need technical assistance and data quality checks in order to complete these projects. As noted above, the importance of quality checking in distributed databases has been highlighted in recent articles, and our quality checking framework followed those recommended guidelines.10–12

States noted that the state analytic capacity needs to be augmented to accommodate these and other projects. State analytic resources vary greatly; some states have several economists and analysts (sometimes at the Ph.D. level), while others have only a few. Limited staff has been one of the reasons that some states have not been able to participate in these projects. This is an important gap for a major program and one that greatly constrains states’ abilities to examine quality in a substantial way. One of the MMDs proposed to the federal government the addition of two new positions under the MMD in each state – a data analyst and a policy analyst – to assure that each state has a baseline level of analytic and policy support for improvements in clinical quality. This addition would make it easier for states to participate in these studies, but would not obviate the need for central research infrastructure as described in this section.

Need for Robust Strategies When Working with State-Level Data

The state-level data can be thought of as a distributed data set, in that the variables in the data sets are similar across states. However, these data sets differ from what is often thought of as distributed data, in that the organization can be different across the states and the variables are not identical.11

There is much discussion about pros and cons around using aggregated data from states versus individual-level data. The analyses possible with individual-level data are much more accurate and sophisticated. Analyses can, for example, adjust for a variety of differences in demographics as well as health status in the populations.

However, as important as adjustments might be, securing individual-level data from multiple states for the release of data in a timely fashion is generally thought to be impossible, or nearly so, because of approvals and data agreements being needed. In the end, for all projects using data like this in the foreseeable future, the request will be for aggregated data from participating states. MMDs will continue to learn from their experiences and other research networks to improve the project methods.

Furthermore, although there are downsides to using state-aggregated data, there are upsides (in addition to feasibility). The first is that state analysts, who know the structure of their own data, will be doing the analyses. State analysts understand how their warehouses are constructed, nuances of where eligibility files come from and how they come, how managed care claims or encounters come in, and a myriad other particulars of individual state data. A second advantage of states conducting their own analyses is that states will have the specifications, modified as necessary for their own variables, and will be able to continue monitoring the outcome. Some of the states have continued to monitor readmissions and antipsychotic medication use, and are likely to continue to monitor elective deliveries. A final advantage is that having states work with their own data increases the buy-in.

State MMDs described the benefits of states working with their own data. One MMD said, Another MMD added, While, a third MMD summed it up by saying,

I think the benefits for doing a distributive model is [that] you get everybody really interested in the program, you get everybody to understand their data and work across their systems, and you generate expertise that you would not necessarily do in a centralized model. The bad part about it is that you can get somewhat dirtier data…you can’t do…sophisticated analysis.

I think it works pretty well for states to pull together, to have states do their own [analyses]…just because [they know] how their data is organized.

Having the states pull data themselves adds a layer of complexity, but also adds a layer of deeper understanding about what is going on in each state that may allow states to actually manage their own programs better.

Timing and Resource Requirements of the Studies

After conducting three studies representing numerous states and Medicaid enrollees (see Table 1), valuable lessons have been learned about timing and resources needed. With respect to timing, it should be noted that the first two studies took three years to complete with limited outside help to build momentum around the topic, develop research questions and specifications that states agreed upon, complete the analysis internally, clean and aggregate states’ data, and ultimately report it in various forms. For both projects there were aspirations to answer more questions and dive deeper into the data with states that have the capability; however because of a lack of resources this was not always possible. As such, some states undertook more analysis on their own, whereas others did not have the funds to do this.

As noted earlier, lack of state-level funding to participate in important studies such as these is a major gap in a program as large and important as Medicaid. As also noted earlier, we have recommended that CMS provide funding at the state level for such endeavors. The last study needed to be conducted within a year because of the contract termination date. The one-year time frame, which began at the start of the federal fiscal year, meant the specifications needed to be developed and piloted in the fall and winter, and the data request was sent to the states during their budget season in the spring – the busiest time of the year for state analysts and leaders. As a consequence of the constricted and inadequate time frame, not all states who wanted to submit data were able to do so, and much of the analysis and reporting occurred after the project ended.

Based on our experience, we believe that 18 months is an ideal time frame for the studies. Ideally, development of the research questions and the data specifications would take place during the fall and winter, pilot testing in the spring, and the data request would go out to states during the summer, after their budgets for the given year are in place. Data would be due back by sometime in the fall, and the remainder of the 18 months would be used for quality checking, analysis and reporting. Other sequencing is possible, but it will be important that the data request be sent to the states during a peak work period.

Conclusions

These studies, taken together, form a blueprint for future multistate quality improvement work, for both their importance and organization. Recommendations in Figure 1 show the most important recommendations based on lessons from past projects to be considered during future distributed data projects, including those that use additional data sets and EMRs. Many of these recommendations will not only help make the projects more efficient, but also will increase states’ participation by increasing the projects’ value and decreasing the states’ workload. Continuation of these studies requires funding for a permanent research infrastructure at the national, as well as the state, levels to strengthen capacity.

Figure 1.

Recommendations for Future Multistate MMD Network Projects

Note: HCU=Healthcare Cost and Utilization Project; PCORI=Patient-Centered Outcomes Research Institute