Making Open Science Work for Science and Society.

BACKGROUND: The open science movement is transforming scientific practice with the goal of enhancing the transparency, productivity, and reproducibility of research. Nevertheless, transparency is a complex concept, and efforts to promote some forms of transparency may do relatively little to advance other important forms of transparency.
OBJECTIVES: Drawing from the literature in history, philosophy, and sociology of science, we aim to distinguish between different forms of scientific transparency. Our goal is to identify strategies for achieving forms of transparency that are relevant not only to scientists but also to decision makers and members of the public. DISCUSSION: We draw a distinction between "scientifically relevant transparency" and "socially relevant transparency." Most of the prominent strategies associated with the open science movement (e.g., making data publicly available and registering studies) are designed primarily to promote scientifically relevant transparency. To achieve socially relevant transparency, which is particularly important in fields like environmental health, further steps are needed to provide scientific information in ways that are relevant to decision makers and members of the public.
CONCLUSIONS: Promoting socially relevant transparency will require a range of activities by many different individuals and institutions. We propose an array of strategies that can be pursued by scientists and other scholars, journals, universities, funders, government agencies, and members of the public. https://doi.org/10.1289/EHP4808.

Introduction

The open science movement has received a great deal of attention from governments, policy makers, and scientists around the globe (European Commission 2014; NAS 2018; Nosek et al. 2015; Royal Society 2012). According to the U.S. National Academy of Sciences, “Open science aims to ensure the free availability and usability of scholarly publications, the data that result from scholarly research, and the methodologies, including code or algorithms, that were used to generate those data” (NAS 2018). The goal of making scientific information accessible to others is often conceptualized to include not only other scientists, but also the general public. As the British Royal Society puts it in its report on open science: “A realistic means of making data open to the wider public needs to ensure that the data that are most relevant to the public are accessible, intelligible, assessable and usable for the likely purposes of non-specialists” (2012).

The open science movement encompasses a number of initiatives, ranging from efforts to more systematically publish scientific results (Chalmers et al. 2013), preregister studies (FDAAA 2007; Kupferschmidt 2018), promote open-access publishing (Else 2018), post papers to preprint websites (Bourne et al. 2017), make all study data publicly available (NAS 2018), make peer review more transparent (Lee and Moher 2017), encourage or mandate sharing of study materials and computer code (Nosek et al. 2015), report the progress of studies in real time so that other scientists can provide input (Foster and Deardorff 2017; NAS 2018), and promote successful communication between experts and decision makers so they can make effective use of scientific information (Holloway et al. 2018; Royal Society 2012).

These initiatives are based on the idea that increasing transparency is good for science and for society. We do not dispute that this presumption is generally true, but recent conceptual and empirical studies of scientific judgment and decision making suggest that transparency is a complex concept. Different forms of transparency can be more or less helpful, depending on the context in which they are implemented and the goals they are designed to promote (Hood 2007; Löfstedt and Way 2016). We distinguish between two major kinds of transparency: transparency designed to achieve goals of the scientific community vs. transparency designed to achieve goals of the broader society. We caution that there is a danger of focusing primarily on “scientifically relevant transparency” when engaging in initiatives to promote open science. To pursue “socially relevant transparency” as well, which is particularly relevant in the area of environmental health, creative strategies are needed for communicating clearly about science in ways that matter to decision makers.

Discussion

Background on Transparency

Efforts to promote open science should distinguish between various types of transparency designed to achieve an array of different goals (Heald 2006; Hood 2007). For example, some commentators distinguish between “direct” transparency, which makes information directly available or observable to the public at large, and “indirect” transparency, which makes information available only to technical experts or other specific individuals (Hood 2007). Another important distinction is between “general” transparency, which strives to make all information about a topic openly available, and “particularized” forms of transparency that focus only on specific sorts of information (Hood 2007). Other authors distinguish “retrospective” transparency, which occurs after a period of time has elapsed, from “real-time” transparency, which makes information immediately available (Heald 2006; Löfstedt and Way 2016). In addition, “event” transparency focuses on inputs and outputs, whereas “process” transparency focuses on procedures and operations (Heald 2006).

To determine which types of transparency should be pursued under the auspices of the open science movement, it is important to consider the full range of motivations or goals at play as well as the range of potential limitations of different types of transparency. One of the common justifications for open science is to promote the reproducibility of scientific results. In some scientific fields, scientists have expressed growing concerns about a “reproducibility crisis” (Begley and Ellis 2012; Munafò et al. 2017). Failures to reproduce scientific work can stem from a range of factors, including confounders that are difficult to control, variations in study populations, inadequate study power or sample size, the use of models that do not adequately represent their target systems, inadequate description of methods, poor data handling, improper statistical analyses, questionable interpretive judgments, and publication bias (Ioannidis 2005). Although efforts to promote transparency about scientific data, methods, and assumptions are not a panacea for solving all of these problems, transparency can help in addressing many of them (Nosek et al. 2015; Elliott and Resnik 2015). As the Royal Society (2012) states in its report on open science, “Not only is open science often effective in stimulating scientific discovery, it may also help to deter, detect and stamp out bad science. Openness facilitates a systemic integrity that is conducive to early identification of error, malpractice and fraud, and therefore deters them” (p. 8).

Nevertheless, the goals of open science go beyond just the effort to promote reproducibility. Open science can speed scientific innovation by making it easier for scientists to make use of existing data and build on each other’s work (Cheruvelil and Soranno 2018; Lowndes et al. 2017). It also promotes inclusivity in the scientific community by making data available to scientists working in low-income countries or teaching-intensive colleges who might otherwise have trouble accessing large data sets (Bezuidenhout et al. 2017; Soranno et al. 2014). Moreover, open science can help policymakers and concerned members of the public obtain scientific information that is relevant to their needs (NAS 2018).

Strategies designed to serve some goals of the open science movement may be inadequate for achieving other goals. For example, efforts to provide access to all the data underlying a study may be helpful for enabling other scientists to reanalyze the results and identify weaknesses that could undermine its reproducibility. However, providing all that data may do little to help citizens or policymakers understand the overall significance of the study or the crucial limitations or weaknesses associated with it. Bezuidenhout et al. (2017) emphasize that one must not only provide data but also address a wide array of other factors—“infrastructural, social, institutional, cultural, material and educational elements”—for data to be meaningful and usable. Additional strategies—such as effective two-way communication between the scientists who designed the study and those using the information—might be needed to clarify the information needed by decision makers.

Moreover, strategies that are helpful for achieving some goals may even be counterproductive for achieving other goals. For example, providing large quantities of data to the public and policymakers without synthesizing and interpreting it could lead to poor decisions due to confusion or misunderstanding (Löfstedt and Way 2016). As Susanna Kim Ripken has concluded, “Evidence suggests that when people are given too much information in a limited time, the information overload can result in confusion, cognitive strain, and poorer decision-making” (Ripken 2006). Regulatory bodies have referred to these concerns about the potential for misinterpreting information when attempting to justify their decisions not to make all the regulatory data provided to them available to the public (Goldacre 2012). Thus, transparency of scientific information needs to be adjusted so that the information is in a form that is usable for those receiving it.

A variety of other potential drawbacks are associated with transparency initiatives. For example, some commentators worry that requiring access to the data and materials underlying scientific studies could inhibit scientific innovation and collaboration in the private sector, where it is important to maintain proprietary control of information (Fenichel and Skelly 2015). Other scientists worry that providing immediate access to their data could allow other scientists with greater resources to “scoop” them (NAS 2018). In fields that deal with human studies, another concern is the potential to reveal private information about the study participants (NAS 2018; Resnik 2018). Finally, transparency initiatives tend to be resource intensive, requiring considerable time and energy from those providing the information as well as from those receiving it. For example, putting research data in a usable form for those outside the original research group may require extensive amounts of time and effort; to promote these activities, funding agencies may find it necessary to provide additional money specifically for this purpose. Thus, ill-conceived efforts at transparency could, ironically, inhibit the progress of science by draining scarce resources that could be used more effectively elsewhere.

We do not think any of these concerns provide compelling reasons to abandon open science initiatives. However, these concerns do show that it is necessary to pursue thoughtful strategies that maximize benefits and minimize drawbacks. As part of this effort, maintaining attention to the full range of goals associated with the open science movement is important so that transparency policies are not designed in ways that meet some goals but fail to address other goals or make achieving other goals more difficult than necessary.

Scientifically Relevant vs. Socially Relevant Transparency

In the remainder of this paper, we focus on two general kinds of transparency, which we call scientifically relevant transparency and socially relevant transparency. We focus on these two forms of transparency because the goals of the open science movement tend to be associated either with conducting better science or with making science more useful and usable by society at large. Focusing on these two kinds of transparency is also helpful because important differences exist between the sorts of transparency initiatives that are most likely to achieve the goals of the scientific community vs. those that are most likely to benefit society at large.

When using the term “scientifically relevant transparency,” we refer to efforts designed to assist scientists in achieving their goals, such as promoting new scientific discoveries and maintaining the reliability of scientific research. The activities typically associated with the open science movement are directed primarily toward advancing these goals. For example, making all the data underlying studies openly available, preregistering studies, describing methods in detail, and providing open access to materials and computer codes are all geared primarily toward helping other scientists to build on previous work and evaluate its reliability. Even open-access publishing, which is frequently cited as a way to make scientific information available to the public, is limited in its ability to promote socially relevant transparency insofar as the average citizen cannot easily evaluate a scientific journal article.

In contrast, socially relevant transparency is focused on providing information that enables decision makers and members of the public to make effective use of scientific research (NAS 2018). Socially relevant transparency is important for people who use scientific information to make choices related to practical matters, such as decisions and policies related to public health, the environment, medical care, the economy, and technology. Socially relevant transparency is particularly important in the field of environmental health research because policymakers draw on this area of science to make important regulatory decisions, and members of the public depend on it to decide how to protect themselves from health risks.

Members of the public have many of the same interests as scientists (e.g., promoting scientific discoveries and maintaining science’s reliability), but the public has additional interests as well. For example, decision makers who use scientific information to promote public and environmental health typically want to understand the major weaknesses or shortcomings associated with the information that is provided to them. Crucially, these decision makers are typically not scientists themselves, and even if they are scientists, they are often forced to use information that extends beyond their areas of expertise. Thus, their informational needs are typically not met merely by receiving all the data, methods, and materials associated with individual scientific studies. Instead, members of the public typically also need to have the available information interpreted and synthesized in ways that clarify the issues that matter to them. Ethicist Onora O’Neill (2006) has emphasized that information needs to be provided in a way that is accessible, intelligible, assessable, and usable to meet the needs of decision makers (see also Royal Society 2012).

Using language taken from the fields of history, philosophy, and sociology of science, one could say that socially relevant transparency involves clarifying for decision makers both the “take-home lessons” in an area of research and the important value judgments associated with those lessons (Elliott and Resnik 2014). Value judgments are choices associated with scientific research that are not constrained solely by logic and evidence. For example, important value judgments can arise in decisions pertaining to the selection of topics and questions, the design of studies, the choice and implementation of methodologies, the analysis and interpretation of data, the weighing of evidence, and the framing and communication of results (Douglas 2009; Elliott 2017; Longino 2002). Especially in fields like public and environmental health, these judgments can end up serving different social values (e.g., public health, economic development, or sustainability) depending on how they are made, even if scientists do not intentionally have specific values in mind when making the judgments (Elliott 2017; Fernández Pinto and Hicks 2019). To make wise use of scientific information, decision makers typically need to have major value judgments related to research clarified for them: What are the major weaknesses associated with the available evidence? What are the major reasons why some studies disagree with others? Do the available studies fail to address important questions? Are there other plausible ways of framing the problem under investigation?

The strategies that promote socially relevant transparency overlap to some extent with the strategies that promote scientifically relevant transparency, but important differences exist between the two types of approaches. For example, it is sometimes helpful for the public when scientists publish their work in open-access journals and make their data publicly available. However, for this information to be fully usable and assessable, decision makers typically need others (e.g., journalists or experts associated with nongovernmental organizations or government agencies) to help analyze and interpret this information so that important value judgments are made explicit. At the very least, decision makers need experts to clarify how new findings relate to the existing body of evidence, so they can contextualize the new findings appropriately and not place too much weight on results that are atypical.

Similarly, decision-makers and members of the public often find it helpful when scientists make all the data underlying their work available, but not because they can directly make sense of the data. For example, in the case of a toxicity study evaluating a pesticide, most people will simply want to know whether the study was designed, analyzed, and interpreted in ways that are likely to overestimate or underestimate the pesticide’s risks and to know how the results from that study compare with those from other relevant studies. The value of making the study data openly available is that such open availability can enable other experts to scrutinize the study in greater detail to determine whether any questionable judgments were made. Thus, strategies for achieving scientifically relevant transparency tend to be helpful but insufficient for achieving socially relevant transparency.

This distinction between the strategies needed for promoting scientifically relevant and socially relevant transparency becomes especially important when evaluating policies like the new U.S. Environmental Protection Agency (U.S. EPA) proposal to require that all studies used for regulatory decision making be based on publicly available data (EPA 2018). Many scientists worry that this proposal is an effort by regulated industries to co-opt the open science movement and exclude high-quality scientific studies from influencing regulatory decision making (because many of these studies incorporate medical data about research participants that need to remain private; Malakoff 2018). Similar calls for open access to academic researchers’ data have been used to harass scientists, slow down their work, and facilitate questionable reanalyses of their data (Elliott 2016; Michaels 2008; Oreskes and Conway 2010). Sometimes open science policies even create an asymmetry, such that academic scientists are forced to provide detailed information about their work, but industry scientists are able to keep the details of their work in-house or disclose it only to regulatory agencies (McGarity and Wagner 2008; Michaels 2008). Thus, open science policies could actually inhibit socially relevant transparency by assisting special interest groups in their efforts to distort research results and mislead members of the public (McGarity and Wagner 2008; Oreskes and Conway 2010).

Distinguishing between socially relevant and scientifically relevant transparency can help alleviate these concerns about the potential for the open science movement to be co-opted by industry in fields like public and environmental health. By recognizing that socially relevant transparency is not identical to scientifically relevant transparency, it is possible to recognize situations where efforts to promote one form of transparency could prove counterproductive to the other if appropriate care is not taken. For example, requiring public access to researchers’ data could harm socially relevant transparency if special interest groups spread misleading reanalyses of the data (Michaels 2008). Thus, to fulfill the spirit of the open science movement, just promoting the release of scientific data and publications is insufficient; additional steps must be taken to clarify the information and to block efforts by special interest groups to sow confusion (Holman and Elliott 2018).

When open science is conceptualized in terms of socially relevant transparency, it becomes clear that its goals overlap with those of science translation, science communication, and public engagement (e.g., Bucchi and Trench 2014; Jamieson et al. 2017). To achieve the Royal Society’s goals of making scientific information “accessible, intelligible, assessable, and usable for the likely purposes of nonspecialists” (Royal Society 2012), the open science movement clearly needs to draw on insights from other fields that focus on making information intelligible to wide audiences. It would be a mistake, however, for proponents of open science to dismiss socially relevant transparency as a topic that is already being addressed by others. Although other fields are focused on the general topic of making scientific information available and useful, the open science movement is focused specifically on promoting greater transparency about the underlying data and judgments that contribute to scientific conclusions. Thus, the concept of socially relevant transparency brings together the open science movement with broader efforts at science communication by highlighting the importance of making the underlying elements of scientific research accessible in a manner that is useful to decision makers and members of the public.

Strategies for Socially Relevant Transparency

The foregoing clarification of socially relevant transparency indicates that it cannot be accomplished by enacting individual policies directed solely at scientists, like publishing open-access journal articles or placing study data in repositories. Instead, striving for this form of transparency will require a system of strategies involving many different people and institutions, including scientists, other scholars, universities, funding sources, government agencies, and members of the public. Not only are these multiple groups needed to combat the efforts of special interest groups, but they are also needed to recognize and clarify the important value judgments that matter to various members of the public. It is unrealistic to think that individual scientists can recognize all the important value judgments they are making (Elliott 2018). Although they can typically recognize and acknowledge the most important limitations or methodological choices associated with their work, they are often unaware of subtler value judgments, such as choices related to weighing evidence or framing the findings. Moreover, they are unlikely to recognize the range of different value judgments that might matter to different stakeholders or decision makers. Thus, promoting socially relevant transparency will require complex systems and collaborations.

Several strategies may be used to promote socially relevant transparency (see Appendix). Some of these strategies are directed toward scientists. Most scientists already strive to clarify important value judgments (although they might not use that term) when communicating about their work. An additional step they can take is to describe and explain these judgments in accessible ways through websites like The Conversation (https://theconversation.com) or through science blogs that reach broad swaths of the public (Bik and Goldstein 2013). They can also take advantage of systematic review methodologies that can help them assess evidence responsibly and provide recommendations for decision makers. For example, the Navigation Guide system developed by the Program on Reproductive Health and the Environment at the University of California, San Francisco, provides a systematic method for evaluating the quality of evidence about environmental hazards and gives clinicians concrete recommendations for medical practice (Woodruff et al. 2011). Scientists can also collaborate directly with members of the public through community-based participatory research (CBPR) efforts. Research projects that link scientists with community members have proven very valuable in uncovering environmental health threats, empowering public groups with the knowledge they need to take action in response, and in helping researchers direct their work in ways that are most relevant to affected communities (see e.g., Corburn 2005; Ottinger and Cohen 2011; Suryanarayanan et al. 2018).

Scholars from other fields, especially from those that study scientific practices, can also help facilitate socially relevant transparency. For example, the Toolbox Dialogue Initiative (tdi.msu.edu) was developed as a strategy for using philosophical dialog to help researchers working on interdisciplinary projects identify implicit assumptions among team members (Eigenbrode et al. 2007). These assumptions could include the sorts of value judgments discussed above (e.g., choices about what questions to ask, how to weigh evidence, and how to frame results) as well as assumptions about the nature of science (e.g., what forms of objectivity are possible and whether value-neutral science is desirable or achievable). The Toolbox Dialogue Initiative uses a set of questions to help team members identify and discuss these assumptions, thereby making it easier for them to communicate these assumptions to those who use the information they generate. In addition, humanists and social scientists can collaborate with science teams to help them identify implicit value judgments, address them responsibly, and communicate about them effectively (Schienke et al. 2011). For example, the Socio-Technical Integration Research (STIR) project includes scholars from other fields in science teams to help them ask questions about the social dimensions of their work (Schuurbiers and Fisher 2009).

Journals can take steps to help authors clarify their key results and important value judgments in accessible ways. Many journals now require their authors to include conflict-of-interest disclosures, which can help decision makers recognize important ways in which the reported research could be informed by those interests. Additional strategies could include supplying bulleted lists of highlights alongside articles, as well as providing access to blogs or podcasts where scientists can discuss their work. An exciting innovation is the possibility of creating “model cards” to accompany complex models that are ordinarily not very transparent, like those used in machine learning (Mitchell et al. 2019). These model cards describe basic features of the models, how they are intended to be used, which training data were used for developing them, and important caveats or recommendations. As the environmental health sciences begin using artificial intelligence and machine learning tools more extensively (see e.g., Luechtefeld et al. 2018), efforts like these to communicate in accessible ways about the strengths and weaknesses of the tools could be extremely important.

Universities also have important roles to play in promoting socially relevant transparency. Many of the activities discussed here require extra efforts to communicate in accessible ways and engage in interdisciplinary research projects or community-based collaborations. Universities can help provide incentives for these activities by providing financial support or including them in criteria for reappointment, promotion, and tenure (RPT). In addition, universities can alleviate some of the concerns about special interests manipulating scientific information by providing robust support for their scientists when they experience harassment.

Science funders can help facilitate the extra effort involved in promoting socially relevant transparency. Most open science activities, including open-access publication and deposition of data in open-access repositories, require funding in order to be successful. Funders can provide additional support for activities like community-based participatory research or collaborations between scientists and scholars working in other fields (e.g., the STIR project; Schuurbiers and Fisher 2009). The National Institute of Environmental Health Sciences has provided support for a range of community-based research projects and engagement activities (Lichtveld et al. 2016). Funders can also provide grants to help develop and support innovative tools for enhancing engagement and communication, such as the Toolbox Dialogue Initiative and the Navigation Guide.

Government agencies are in an important position to support and to benefit from socially relevant transparency. Agencies like the U.S. EPA frequently develop scientific advisory bodies that can help clarify important value judgments and assess the state of the science when important decisions are at stake. In response to concerns about the influences of special interest groups, these agencies can create advisory bodies that are composed of scientists with minimal or balanced vested interests. They can also develop policies that promote greater public access to regulatory studies and to data produced both by publicly and privately funded researchers (McGarity and Wagner 2008; Michaels 2008). Access to privately funded research plans and data like the Tobacco Archives and the Monsanto Papers has proven very important in identifying important judgments and identifying many of the concerns about special-interest science discussed previously (McHenry 2018; Proctor 2012). In addition, government agencies (as well as scholarly societies and national scientific academies) can produce publicly accessible and understandable reports on important topics relevant to decision makers and members of the public. For example, many agencies have collaborated with the U.S. National Academies of Science, Engineering and Medicine to produce reports on topics like climate change, vaccines, evolution, and genetically modified organisms. Similarly, the U.S. EPA publishes Integrated Science Assessments to help guide decisions about air pollution. The Intergovernmental Panel on Climate Change is also well-known for its reports that strive to communicate about the state of climate change and associated uncertainties in ways that are meaningful to decision makers (Edenhofer and Kowarsch 2015).

Government agencies have also been involved in innovative efforts to help decision makers make more effective use of data and influence research projects to make them as socially relevant as possible. For example, the National Aeronautics and Space Administration (NASA) has partnered with the U.S. Agency for International Development to develop the SERVIR project, which helps translate NASA data into information that can help local governments, forecasters, and researchers evaluate environmental threats and respond to natural disasters (see e.g., Schumann et al. 2016). In addition, NASA has supported a Health and Air Quality Applied Sciences Team (HAQAST), which helps stakeholders make use of NASA data to answer stakeholders’ environmental health questions (Holloway et al. 2018). Many organizations are also starting to provide climate services. Much like weather services, climate services provide decision guides and advisories about potential impacts associated with climate change. The American Meteorological Society emphasizes that climate services tend to involve a wide range of institutions [e.g., universities, nongovernmental organizations (NGOs), governmental agencies, and the private sector], and they function best when the providers and users of information are able to interact (AMS 2015; see also Hewitt et al. 2017). Philosophers of science are currently exploring how climate services can best take the concerns of different stakeholders into account when handling important value judgments (Parker and Lusk 2019).

Members of the public can also take steps to help promote socially relevant transparency. Citizen science initiatives, in which members of the public engage in research projects either on their own or in collaboration with professional scientists, are exploding in popularity (Cavalier and Kennedy 2016; Poisson et al. in press; Roy et al. 2012; Silvertown 2009). These initiatives provide members of the public with opportunities to guide the value judgments associated with scientific research and steer it in ways that meet their needs and incorporate their concerns. Although some scholars have worried that citizen science could be tainted by advocacy or low-quality methodology, these concerns can be addressed through education, training, and collaboration with professional scientists (Elliott and Rosenberg 2019; Resnik et al. 2015). Members of the public who are concerned about specific environmental health issues can also support the work of civil society organizations devoted to those issues because these organizations can employ or work with experts with the expertise needed to help evaluate and communicate about the relevant science.

Conclusion

As the scientific community moves forward with efforts to promote open science, there is a risk of focusing primarily on strategies for promoting scientifically relevant transparency and failing to take the additional steps needed for promoting socially relevant transparency. A recent survey of scientists’ perceptions of open science indicated that they are focused primarily on activities associated with scientifically relevant transparency (Levin et al. 2016). In keeping with this finding, important open science initiatives, like the Transparency and Openness Promotion (TOP) guidelines, focus on strategies like preregistering studies and making data, materials, and computer codes openly available (Nosek et al. 2015). Even though government reports about the open science movement typically emphasize the importance of both forms of transparency, they also tend to focus on strategies that are aimed primarily toward promoting scientifically relevant transparency (see e.g., NAS 2018; Royal Society 2012). Fortunately, a wide range of innovative strategies are available to promote socially relevant transparency as well. Emphasizing these strategies in future discussions of open science is crucial.

Click here for additional data file.

Click here for additional data file.