Use of research evidence in state health policymaking: Menu labeling policy in California.

Addressing the translational gap between research evidence and state health policy requires an understanding of the current use of research evidence in the state policymaking process. In this study, we explore the use of research evidence to inform the legislative debate about restaurant nutrition labeling policy in California. In 2008, California was the first state to enact a mandatory menu calorie labeling policy in the U.S. Using a qualitative approach, we examine data sources and types of evidence used in legislative documents (n = 87) related to six menu labeling bills introduced in California's state legislature between 2003 and 2008. Federal- and state-level government agency reports were the most frequently cited sources of technical knowledge. Advocacy coalition members who were active participants involved in the policy debate were also cited as experts. Five of the six bills included evidence in related legislative documents. While documents included considerable evidence on the magnitude and severity of the obesity problem to justify policy enactment, there were a limited number of statements referring to policy effectiveness and only one statement identified attesting to implementation context and acceptability. Reference to evidence on related policy suggests policy precedence may also play an important role in policy decision making. There is a need to improve the dissemination of obesity policy effectiveness and implementation studies in a politically time sensitive manner to influence state health policy debates. Strategies are discussed to effectively integrate the use of research evidence in the state health policymaking process.

Introduction

Integrating research evidence into the policymaking process can lead to the adoption of effective and equitable health policy (Brownson et al., 2009, Dodson et al., 2015, Whitehead et al., 2004). Little is known about the type of research evidence used in state health policymaking (Dodson et al., 2015, Gollust et al., 2014, Jewell and Bero, 2008), including obesity health policy. In Minnesota, one study found only 41% of legislative materials related to 13 obesity-related bills mentioned research evidence (Gollust et al., 2014).

Using research evidence as part of the policymaking process is a formidable task due to differences in decision-making processes and priorities between science and policy. Unlike policymakers, scientists have different incentives and data sources, conduct research over long periods of time, and prioritize objectivity (Brownson et al., 2009, Brownson et al., 2006). Barriers in the political process may inhibit the use of scientific evidence in policymaking, such as institutional features, lack of evaluation skills, lack of relevant research, competing interests, time constraints, political priorities, information overload, and budgetary constraints, among other factors (Dodson et al., 2015, Gold, 2009, Jewell and Bero, 2008, Jou et al., 2018, Kingdon, 1995, Oliver et al., 2014). Examining the use of research evidence in a state policy debate can help determine how to effectively integrate research findings into the policymaking process.

In the 21st century, policymakers shifted their attention to policy tools to improve dietary behavior and physical activity with the goal of addressing the obesity epidemic (Kersh and Morone, 2005). California presents a unique opportunity to examine the use of evidence in policymaking since the state has been an innovator in obesity policy. In 2008, California became the first state to enact a mandatory calorie menu labeling law for chain restaurants with the objective of providing restaurant customers with calorie information at the point-of-sale to reduce caloric consumption. The law would affect an estimated 17,000 restaurants with ≥20 state locations (Stein, 2010) and mandated chain restaurants post calorie information for standard menu items on indoor menu boards and menus. At the time, local jurisdictions in California (Payán et al., 2017) and elsewhere (e.g., New York City) had only recently adopted their own menu labeling policies and few published peer-reviewed articles were available on the topic (Harnack and French, 2008, Nelson et al., 1996). This study explores the types of data sources and research evidence used to inform California’s menu labeling policy debate.

Methods

Data collection

We systematically searched for restaurant nutrition labeling bills introduced in California’s legislature between 1999 and 2008 and identified six menu labeling bills (n = 87 legislative bill documents). Legislative documents include information documenting a bill's history, status, amendments, and analyses, including written commentary. Legislative bill documents were collected in 2014 and the data was analyzed in 2018.

The six bills were Senate Bill (SB) 679, SB 1171, SB 120, SB 180, SB 1420, and Assembly Bill (AB) 2572. SB 679, introduced in 2003, proposed mandating restaurants with ≥10 franchises post a sign for customers indicating nutritional information for all food items was available upon request and furnish this information if requested. Other SBs included provisions for large chain restaurants (note, this definition was modified throughout) to include nutritional information (i.e., calories, saturated fat, transfat, sodium) in printed menus and calorie information on menu boards. In 2008, AB 2572 proposed mandating restaurants use at least one of several listed methods to provide nutritional information (e.g., menu, brochure, etc.). SB 1420 was ultimately successful and signed into law in 2008. A detailed summary of the systematic search for these bills and policy information about the bills is described elsewhere (Payán et al., 2017).

Data analysis

Documents were prepared and uploaded into NVivo, a qualitative data management software program (QSR International Pty Ltd, 2012). Coding focused on: 1) identifying references to technical knowledge and data sources and 2) categorizing these references into type 1, type 2, or type 3 evidence (Rychetnik et al., 2004). Technical knowledge includes both research and knowledge/information type evidence since both inform the policy process. Knowledge and information evidence differs from research evidence in that the former includes published documents and reports (e.g., policy evaluations and statistical analyses) which may not have used the scientific method (Bowen and Zwi, 2005). Type 1 evidence focuses on the magnitude, severity, and preventability of a public health issue. Type 2 evidence provides information on the effectiveness of an intervention to address an issue. Type 3 evidence characterizes the context under which interventions were implemented and their acceptability. Detailed definitions for these types of evidence are provided in Table 1.

Table 1

Types of Evidence and Sources of Technical Knowledge included in Legislative Bill Documents Related to California’s Menu Labeling Policy Debate (2003–2008).

Type of Evidence	Definition (Brownson et al., 1999, Rychetnik et al., 2004)	Examples of Coded Text	Source (year), Organization
Type 1	“Research that describes risk-disease relations, & identifies the magnitude, severity, & preventability of public health problems”	Two-thirds of all adults are overweight or obese, and 17.5% of children and adolescents ages 2–19 years are overweight	National Health and Nutrition Examination Survey (2001–2004), Centers for Disease Control and Prevention
		Statewide, approximately 28 of every 100 children are overweight and 40% are unfit	California Center for Public Health Advocacy (CCPHA) report (2005), CCPHA
		The economic costs attributed to obesity to California alone in 2001 were over $21 billion	Website (2008), California Department of Public Health
		in 2003, Americans spent about 46 percent of their food dollars at restaurants, compared with 26 percent in 1970	Center for Science in the Public Interest (CSPI) Report (2003), CSPI
		Estimates that total nationwide sales for restaurants last year were $537 billion. In California, the 2006 restaurant sales were estimated at $51.5 billion	– (2007), National Restaurant Association
		Current eating habits are of grave cancer [sic] because poor diet, obesity, and physical inactivity may be responsible for one out of every three cancer deaths, as many as smoking	–, American Cancer Society
		The study reports that obesity is linked to higher medical costs and very high rates of chronic illnesses, higher than living in poverty, and much higher than smoking or drinking	RAND Study (2002), RAND Corporation
Type 2	Research on the “relative effectiveness of specific interventions”	Three-quarters of American adults report using food labels on packaged foods, which are required by the federal Nutrition Labeling and Education Act of 1990. Approximately one-half (48 percent) of people report that the nutrition information on food labels has caused them to change their minds about buying a food product.	–1
		84% of Californians surveyed want this information to be provided on menus	Field Poll (2007), CCPHA
		According to the study, a modest reduction in calorie consumption would have a significant impact in the annual weight gain of the population. As an example, the study found that if 10% of the customers at these restaurants reduced their consumption by 100 calories, annual weight gain across the population would be reduced by 40%.	Health Impact Assessment (2008), Los Angeles County Department of Public Health
		Patrons at 13 major chains were asked whether they saw and used calorie information while in the restaurant in the period the previous ordinance was in effect (May through June 2007). Taking a weighted average and excluding the restaurant chain Subway (where 31.3 percent of customers reported seeing calorie information in the restaurant), only 3.1 percent of customers (1 in 32) – reported seeing calorie information.	Notice of intention of a proposal to repeal and reenact §81.50 of the New York City Health Code (2007), New York City Department of Health and Mental Hygiene
Type 3	“Information on the design & implementation of an intervention; the contextual circumstances in which the intervention was implemented; & information on how the intervention was received”	Taking a weighted average and excluding the restaurant chain Subway (where 31.3 percent of customers reported seeing calorie information in the restaurant), only 3.1 percent of customers (1 in 32) - reported seeing calorie information. The notice reported that 95 percent of survey participants at McDonald's did not notice the voluntary nutrition information even after they had purchased their food.	Notice of intention of a proposal to repeal and reenact §81.50 of the New York City Health Code (2007), New York City Department of Health and Mental Hygiene

This statement did not include a source in the legislative documents. The statement refers to food label use trends research findings in the FDA’s “Calories County: Report of the Working Group on Obesity” published in 2004. The source cited in the report is: Derby M, Levy A. Do Food Labels Work? Gauging The Effectiveness of Food Labels Pre-and Post-NLEA (Pre-publication draft). In P.N. Bloom & G.T. Gundlack (Eds.) Handbook of Marketing and Society. 2000; 372–398.

An initial codebook was developed and pilot tested to assess the coding approach. Two researchers used the codebook to code 15% of the data for comparison. Coding discrepancies were discussed and addressed before the primary author coded remaining data. Analytical summaries were iteratively reviewed and revised by the research team and exemplary quotes were selected for inclusion in this article.

Results

Sources of technical knowledge

Federal- and state-level government agencies were the most frequently cited data sources in California’s menu labeling debate. Federal agencies mentioned were the Centers for Disease Control and Prevention, Food and Drug Administration (FDA), U.S. Department of Agriculture, and U.S. Department of Health and Human Services. State and local public health agencies were also cited, including the California Department of Public Health, Los Angeles County Department of Public Health, and New York City Department of Health and Mental Hygiene. Several statements referred to specific reports (e.g., Surgeon General’s report). For example, the FDA’s 2004 Obesity Working Group was cited as making a recommendation to encourage restaurants to provide consumers with calorie and nutrition information.

Advocacy coalition members previously identified as participants actively involved in the state’s menu labeling policy debate (Payán et al., 2017) were also cited. The National Restaurant Association was listed as an authority on out-of-home food consumption trends and the California Center for Public Health Advocacy was cited as an expert on state childhood obesity. Other non-profit organizations included the American Cancer Society, American Diabetes Association, and the Center for Science in the Public Interest. One organization who was not an active participant in the state’s menu labeling policy debate, the RAND Corporation, was also referenced.

Types of evidence

Five of the six bills included evidence in related legislative documents—SB 180 was the only bill that did not reference any evidence. Legislative documents included a preponderance of type 1 evidence (103 coded statements) describing the magnitude and severity of the obesity problem. This evidence was largely comprised of epidemiological research on obesity and the related economic impact in the U.S. and in California. The most frequently mentioned type 1 evidence statement was included in four bills (SBs 679, 1171, 120, 1420) and presented national obesity rate and trend information: “According to the Centers for Disease Control and Prevention, two-thirds of American adults are overweight or obese, and the rates of obesity have tripled in children and teens since 1980” (SB 1420). Several statements were repeated in other bills, potentially serving as boilerplate policy text. For example, Section 1 of SB 1171 incorporated nearly the same evidence statements as Section 1 of SB 679 which had been introduced the year prior by the same legislator (Senator Deborah Ortiz-D).

Minimal evidence was included on policy effectiveness (15 type 2 evidence statements). One statement, mentioned in documents from five of the six bills, described the effectiveness of the Nutrition Labeling and Education Act (NLEA) of 1990—which gave the FDA the authority to require nutrition labeling on most food products regulated by the agency—as evidence of the potential effectiveness of a menu labeling policy and was included in the actual bills’ content. The citation refers to food label use trends research conducted by Derby and Levy cited in an Obesity Working Group report. Other type 2 evidence statements consisted of results from a 2007 Field Poll conducted in California, a health impact assessment (HIA) on the impact of menu labeling on weight gain in Los Angeles County (Kuo et al., 2009), and survey data on awareness of calorie labeling in New York City. The latter was also coded as type 3 evidence since it described the implementation context and reception of a local menu labeling ordinance. Of note, both the HIA and New York City statements were only referenced in SB 1420 analysis documents. SB 1420 was introduced in 2008 and successfully enacted into law, suggesting types 2 and 3 evidence may have been useful in advancing legislation. Table 1 provides examples of coded content for each type of evidence.

Discussion

This study shows type 1 evidence was more frequently referenced than other types of evidence as justification for a state menu labeling policy during the several iterations in which restaurant nutrition labeling bills were introduced in California’s legislature. A study on the use of research evidence in obesity-related bills proposed between 2007 and 2011 in Minnesota similarly found a high percentage (51.1%) described the magnitude of the obesity issue or obesity as a risk factor for disease (Jou et al., 2018), which would be categorized as type 1 evidence. It is plausible that type 1 evidence may be more accessible to policymakers or readily available than other types. Use of type 1 evidence may also reflect the trajectory of obesity research, which began with mounting scientific and economic evidence prior to intervention research (McKinnon et al., 2009). Another potential reason for the inclusion of type 1 evidence in legislative documents is that state legislators are interested in and seek evidence in the form of data and statistics to understand the severity of a problem (Dodson et al., 2015). Evidence of the effectiveness of similar policies was also cited (e.g., NLEA) suggesting policymakers consider existing evidence on related policy instruments when making decisions, potentially as a default when research is not readily available on a specific topic.

Most type 2 and 3 evidence statements identified in this study were included in analyses documents for SB 1420—the bill that was ultimately enacted into law in 2008. This finding suggests research on policy effectiveness, feasibility, and implementation may be more effective toward bolstering support and increasing the likelihood of success for a policy instrument than type 1 evidence. HIAs and systematic reviews have previously been identified as examples of particularly useful methods to inform evidence-based health policy (Brownson et al., 2006, Fielding and Briss, 2006). Integration of Los Angeles County’s HIA into the policy debate in 2008 (Kuo et al., 2009) demonstrates how to conduct timely type 2 research to inform state policy. Experiential evidence from other jurisdictions may also contribute as evidence in a policy debate (e.g., New York City).

During the debate period, limited research was available on the impact of calorie labeling information on food purchasing behavior. A systematic review on the effectiveness of restaurant labeling policy identified 16 related studies (VanEpps et al., 2016), but none were available during the debate period we analyzed. Another systematic review of studies on the effect of restaurant menu labeling on calories and nutrients (Cantu-Jungles et al., 2017) identified a 1996 study examining the purchasing behavior of consumers when a nutrition information pamphlet was given at the point-of-sale in a full service University restaurant. The authors found people did not make more nutritious choices when the information was available, however, there was a statistically significant difference in menu selection between those that read and did not read the information (Nelson et al., 1996). A third article reviewed studies on the effect of point-of-purchase calorie labeling on cafeteria or restaurant menu food selections and identified six that met the inclusion criteria, but found only small effects for five of the studies (Harnack and French, 2008). Although these studies were published before 2007, each had methodological limitations and did not measure actual behavior in a restaurant setting. Future studies are needed examining access to research and research training needs among state legislators and their staff.

Our findings also highlight government agencies’ role as vehicles of technical knowledge informing state health policy, which supports prior work identifying federal and state agencies as important sources of evidence in state health policymaking (Jou et al., 2018). A prior review found government agency reports play an important role in disseminating technical knowledge to political actors (Weible, 2008). Government agencies may be viewed by policymakers as trusted sources of information who provide relevant and useful evidence (Dodson et al., 2015, Sorian and Baugh, 2002). Recommended pathways to integrate research into agency reports includes training researchers to participate in user-commissioned studies and contribute to synthesis around policy problems/questions (Brownson et al., 2006, Gold, 2009, Jou et al., 2018). Increasing interaction between legislative staff and researchers is another method to integrate research findings into the policy pipeline since staff tend to have more time and capacity to receive the research evidence and report the information to a legislator (Brownson et al., 2006, Dodson et al., 2015, Sorian and Baugh, 2002). In one study, 19% of state legislators said they do not read journals at all compared to 10% of staff (Sorian and Baugh, 2002). Policy and research briefs, as well as other concise communication documents may be a more effective means of communicating to policymakers and integrating research into reports than peer-reviewed articles (Jou et al., 2018, McKinnon et al., 2009). These documents should be publically accessible to promote use among policymakers (Jewell and Bero, 2008) and others who are involved in the policymaking process. Key factors influencing whether a policymaker reads given information include: the timeliness of the information, its relevance to a jurisdiction, and the format (i.e., brief paragraphs/bullet summaries and use of charts/graphs are preferable) (Sorian and Baugh, 2002).

Advocacy coalition members involved as participants in the policy debate also contributed as experts. This finding is valuable as it suggests that technical knowledge can be strategically used by coalition participants to promote their policy position. Public health and industry coalition leaders often referred to themselves or other coalition participants as experts, suggesting that organizations embedded in the policy debate simultaneously engage in the production and consumption of policy-relevant evidence. Advocates may use evidence in tandem with persuasive arguments to support a specific policy position (Jou et al., 2018). This was the case in California’s menu labeling policy debate. A prior analysis of expressed beliefs and policy arguments used by coalitions involved in California’s menu labeling debate found the public health coalition consistently used an informed decision making argument in support of a menu labeling policy whereas the industry coalition presented economic and operational barriers to oppose a mandatory menu labeling policy (Payán et al., 2017). Additional research is needed to examine the quality and rigor of evidence introduced by advocacy coalition members in health policy debates.

Strengths and limitations

A study strength is that this is one of the few studies to examine the use of obesity research evidence in state policymaking. Further, it is the first to use evidence categories to code for types of evidence referenced in legislative documents. A limitation is the lack of interview or survey data with state policymakers which could have provided information on the perceived influence and role of research evidence in policy decisions.

Conclusion

The menu labeling policy debate in California involved reference to type 1 evidence on the obesity epidemic in the country and state as justification for policy but lacked integration of policy effectiveness and implementation studies. This may be due to a lack of type 2 and 3 research at the time or lack of awareness of existing studies by policymakers. Our findings suggest a need for researchers to increase the production of pragmatic and politically time-sensitive health policy effectiveness and implementation studies.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.