Fostering Conversation about Synthetic Biology Between Publics and Scientists: A Comparison of Approaches and Outcomes.

Public engagement with science (PES) is an emerging outreach method that builds trust between scientists and public audiences by encouraging two-way conversations and mutual learning about science content and societal values. Building with Biology, a PES initiative focused on synthetic biology, distributed 182 kits with two types of products to informal science education institutions across the United States: 1) hands-on activities for public events, and 2) materials to run public dialogue programs, called forums. This article compares the interest levels, perceived value, and learning of public participants at these events and forums. Forum participants reported slightly higher levels of increased interest in future activities related to PES and synthetic biology; valued aspects of interpersonal interactions central to dialogue-based programming; and described learning about societal decision-making around synthetic biology. Event participants valued enjoyment and access to content and reported slightly larger learning gains. The current study may help program coordinators and educators thoughtfully select a PES product type that promotes outcomes aligned with their goals: events featuring hands-on activities may support greater understanding of scientific relevance, and forum programs might encourage learning and behavior that leads to deliberative processes.

INTRODUCTION

In recent years, science communication literature has described a growing divide between scientists and publics, which may increase distrust of science (1–4). In response, there has been a call to increase public confidence in science and scientists through outreach (5–7). However, research shows that traditional models of outreach often do a poor job of building trust between scientists and publics (3, 8–10). Instead, studies find that building trust relies on the public seeing science communicators as competent, having honorable intentions that avoid the perception of persuasion, and being willing to both educate and listen (8, 11).

In contrast to traditional outreach models like the deficit model, which focuses on one-way transfer of information, public engagement with science (PES) encourages two-way conversation and mutual learning between scientists and publics. It includes science, technology, engineering, and math (STEM) content as well as values and personal experiences pertinent to those fields (9, 12, 13). Outreach activities involving PES can take a number of forms: public events such as maker fairs or science festivals; public participation in science research activities; and dialogue and deliberation programs such as forums and science cafes (13–15). However, to be considered PES, activities must have mutual learning between scientists and publics as a central goal (13, 16).

In 2014, the Museum of Science, Boston, in conjunction with the American Association for the Advancement of Science (AAAS), BioBuilder, Synberc, Science Museum of Minnesota, and Sciencenter in Ithaca, received a National Science Foundation grant to create and distribute over 150 PES outreach kits to informal science education sites across the country (17). The goal of the project, called Building with Biology, was for scientists and publics to use kit activities to learn from one another through two-way conversations that explored research outcomes and methods, personal and societal values, and societal implications of synthetic biology.

Building with Biology created two kinds of products that allow scientists and informal science education institutions to explore PES methods: hands-on activities and forums. Each kit included six hands-on activities that could be adapted and used within a public event. Throughout this paper, the term “public event” will signify a site’s use of multiple hands-on activities at the same time. Each activity was designed to be facilitated by an educator, preferably a synthetic biologist. The interactions around these activities typically lasted five to ten minutes. During these interactions, the scientists and public participants—who were often family groups—had conversations about topics such as which kinds of synthetic biology-based foods they would be comfortable consuming, or how synthetic biology might be used to solve problems around food security, healthcare, and the environment.

The forums were designed as one- to two-hour dialogue programs during which scientists and publics discuss societal and ethical implications of a STEM topic. There were two forum topics: how appropriate is it to use CRISPR technologies to edit the genomes of living things, and should we release genetically modified mosquitoes to reduce the transmission of malaria (and if so, how). More information about the PES outreach activities in the kit, evaluation instruments, and data collection protocols are on the Building with Biology website (www.buildingwithbiology.org/kit-contents).

There were inherent differences in the public event and forum experiences. The amount of time that publics spent participating in the events was much more variable than for the forums. Public events were free-choice, so publics could participate in varying combinations of activities for as little or as long as they wanted, generally five minutes to two hours. Forums followed a set agenda at each site, lasting between one and two hours for all participants. Additionally, the interactions between publics and scientists differed. During forums, publics engaged in dialogue with scientists, creating plans about when and how to apply synthetic biology. At the public events, publics and scientists conversed about the societal and ethical implications of synthetic biology, but did not create plans for implementation.

This paper evaluates these two PES products to discover differences in outcomes for participants. The authors hypothesized that the two products would have different affordances in promoting learning, participant values, and increased interest. Understanding these affordances may be valuable for future PES event hosts who wish to select activities to achieve particular goals. While the context of this study was primarily informal science education, these two product types have also been successfully integrated into formal education settings.

METHODS

Evaluation sites

In the summer of 2016, Building with Biology public events and forums took place at informal science education institutions (ISIs) around the United States. To understand their impacts, an evaluation of public outcomes occurred at 65 sites. Thirty-three sites collected data at a public event, 22 collected data at a forum program, and 10 collected data at both a public event and forum. The total evaluation sample represented locations in 33 US states and Washington, DC, and included museums (n = 38 for public events and n = 18 for forums), colleges/universities (n = 2 for public events and n = 7 for forums), and other ISIs (n = 3 for public events and n = 9 for forums).

For the public events, sites were invited to apply to be part of the evaluation when they registered for a kit. More sites applied than could be accommodated. Therefore, evaluators purposefully chose sites to ensure diversity in institution type, size, and geography. Stipends were distributed to encourage use of the forum materials, and those sites that received a stipend were required to participate in the evaluation. Project leaders chose stipend recipients based on their location and the likelihood they would successfully implement a forum.

Data collection

Two survey instruments, one each for the public events and forums, were developed to obtain data from participants (see demographics in Table 1). These surveys included the following types of questions:

TABLE 1

Demographics of respondents at public events and forums.

	Public events (n = 1,390 visitors)a	Forums (n = 667 participants)
Gender
Male	25%	44%
Female	75%	56%
Other	<1%	<1%
Ageb	Public events (n = 1,436 visitors)	Forums (n = 659 participants)
0–3	5%	—
4–7	19%	—
8–12	19%	—
13–17	3%	—
18–24	5%	39%
25–34	14%	20%
35–44	19%	12%
45–64	13%	18%
65+	2%	11%

There were 682 public event surveys, and these surveys asked respondents to provide age and gender data about all members in the respondents’ group.

Public surveys asked respondents to provide the ages for all group members. Thus, even though all survey respondents were adults, there is data about children who attended the event with the respondents. For the forum surveys, age data was only collected from the adult participant who filled out the survey.

Open-ended questions asking respondents what they learned from and valued about their experiences

Likert-scale questions asking respondents to rate their level of interest

Retrospective pre/post questions about respondents’ level of knowledge before and after attending the public event or forum

Demographic information including age, gender, and scientific background

At public events, participants were offered a passport activity through which they could get stamps for participating in aspects of PES. Upon finishing their time at the event, groups were encouraged, through the passport, to complete a survey. Data collectors were trained to use a continuous random sampling method to invite one adult from the next available visitor group to fill out a survey at the end of their experience, whether or not they had used the passport activity. A total of 682 public event surveys were gathered across 43 events. At forums, data collectors were trained to use census sampling, inviting all adult participants to complete a survey at the conclusion of the program. Data collectors gathered 721 forum surveys across 32 sites (Table 1).

The gender imbalance in the public event sample— which includes more women than men—is typical of adult participation at ISIs, as these institutions tend to attract groups with children who are often accompanied by female caregivers. Sampling for the forum programs had a more balanced gender makeup, as would be expected for programs that target adults rather than family groups. Analysis by gender was not part of the original evaluation goals, and so the effect of gender was not assessed.

Qualitative analysis

Qualitative data analysis relied on pre-established coding schemes developed through the evaluation of previous Museum of Science forums, particularly by the National Informal Science Education Network (www.nisenet.org, now the National Informal STEM Education Network). Codebooks were adjusted slightly for applicability to synthetic biology content after testing during Building with Biology’s pilot year, but they were kept consistent for public events and forums to allow for comparison between the two. After coders were trained and the code definitions were refined, two evaluators independently coded all responses (percent agreement was 89% for the learning question and 91% for the value question). A third evaluator, who was an expert in these codes, having used them previously, assessed any disagreements and finalized the coding.

Statistical analysis

To identify significant differences between public event and forum respondents, evaluators used Pearson’s Chi Squared (χ2) and Mann-Whitney U tests. Nonparametric tests were selected because data were negatively skewed, and in some cases, subsample sizes were relatively small. Because of the skewed data, medians, rather than means, are provided. Mann-Whitney U tests compared forum and event participants’ responses for ordinal Likert-scale items about interest and learning. Pearson’s Chi Square (χ2) tests assessed potential differences between frequency counts of codes for the qualitative data. When conducting 2 × 2 χ2 tests, the conservative Fisher’s Exact p value was used due to low expected cell counts in some cases. Evaluators used an alpha level of 0.05 for all statistical tests.

RESULTS

Research question 1: How, if at all, did interest levels and values differ between forum and hands-on activity participants?

Both the forum and event surveys asked participants, “How much did this event increase your interest in the following?” The statements that followed included:

Checking out news stories (online, TV, and/or print) about synthetic biology

Learning about how synthetic biology is connected to my daily life

Talking to a scientist about the impacts of scientific research in my community

Sharing my views about synthetic biology with friends and family

Respondents selected whether their interest in these activities increased “not at all,” “a little,” “somewhat,” or “a great deal.” Responses were combined into a single index value for each participant, ranging from 0 (people who selected “not at all” for each item) to 12 (those who chose “a great deal” for each). Only respondents who answered all four questions were included in the analysis.

Forum respondents reported larger gains in interest than did event participants (U = 154118.00, n = 1,190, p < 0.001). However, the effect size of this difference was small (r = −0.11). For both forum and event respondents, index values ranged from 0 to 12, and the median for both groups was 8.00, representing an average response value of “somewhat” across the four questions. Comparing forum and event participants for the individual questions, forum respondents reported larger increases in interest for checking out news stories (U = 148906.00, n = 1,201, p < 0.001, r = −0.15), talking to a scientist (U = 159492.50, n = 1,198, p = 0.001, r = −0.10), and sharing their views (U = 163006.00, n = 1,197, p < 0.009, r = −0.08). The effect sizes of these differences were small, and there were no statistically significant differences for learning how synthetic biology is connected to daily life.

Supplementing the quantitative questions about reported changes in interest, an open-ended question asked, “What, if anything, did you value about your participation in this event?” To facilitate comparison, the same coding scheme was applied for both the forum and event surveys. Table 2 shows example quotations and the frequency of codes for forum and event respondents. The most common response for both forum and event participants was that they valued the opportunity to learn (26.1% overall; there was no statistically significant difference in the frequency of this code between the two groups). Beyond this, forum participants were more likely to value aspects related to dialogue, such as hearing diverse opinions (27.7% versus 2.3% for event respondents), discussing the topic (18.8% versus 4.0%), the opportunity to share opinions (13.3% versus 2.0%), and meeting other participants (5.5% versus 1.3%). In contrast, public event respondents were more likely to value aspects related to format and content. This included access to experts (17.6% versus 5.5% for forum respondents), the interactive/fun experience (17.9% versus 2.0%), the great experience for kids (17.9% versus 0.0%), and the topic of synthetic biology (7.3% versus 0.0%).

TABLE 2

Responses to “What, if anything, did you value about your participation in this event/forum?”

Code	Hands-On Activities (n = 301)	Forum (n = 346)	Example Quotation
The opportunity to learn	28.9% (87)	23.7% (82)	“The opportunity to learn and discuss in a laid back, respectful environment.”
Hearing diverse opinionsa	2.3% (7)	27.7% (96)	“I liked hearing the variety of opinions.”
Discussing the topicb	4.0% (12)	18.8% (65)	“The group discussion.”
The access to expertsc	17.6% (53)	5.5% (19)	“Well informed scientists and helpers to teach us about synthetic biology.”
The interactive/fun experienced	17.9% (54)	2.0% (7)	“Fun way to do ‘smart’ things.”
Great experience for kidse	17.9% (54)	0.0% (0)	“Kids enjoyed it.”
The opportunity to share my opinionsf	2.0% (6)	13.3% (46)	“I felt like my opinion mattered.”
The format of the event	8.0% (24)	5.5% (19)	“Great format, good mix of people.”
Meeting other participantsg	1.3% (4)	5.5% (19)	“Met some interesting people in different views.”
The topic of synthetic biologyh	7.3% (22)	0.0% (0)	“Understanding more of the advances in synthetic biology.”

χ2(1, n = 647) = 77.708, Fisher’s Exact p < 0.001, ϕ = 0.347

χ2(1, n = 647) = 33.626, Fisher’s Exact p < 0.001, ϕ = 0.228

χ2(1, n = 647) = 23.895, Fisher’s Exact p < 0.001, ϕ = −0.192

χ2(1, n = 647) = 47.758, Fisher’s Exact p < 0.001, ϕ = −0.272

χ2(1, n = 647) = 67.726, Fisher’s Exact p < 0.001, ϕ = −0.324

χ2(1, n = 647) = 27.816, Fisher’s Exact p < 0.001, ϕ = 0.207

χ2(1, n = 647) = 8.134, Fisher’s Exact p = 0.005, ϕ = 0.112

χ2(1, n = 647) = 26.179, Fisher’s Exact p < 0.001, ϕ = −0.201

Research question 2: In what ways, if at all, did reported learning differ between forum and public event participants?

The assessment of learning took two forms. The first was a retrospective pre/post question: “How much did you know about the following topics BEFORE this event/forum, and how much do you know AFTER the event/forum?” The topics included:

Facts about synthetic biology

Applications of synthetic biology

Societal aspects of synthetic biology

What other people think about synthetic biology

Respondents selected a value on a four-point Likert scale of “nothing,” “a little,” “some,” and “a lot.” Change scores were calculated by subtracting the pre-knowledge value from the post-knowledge value, resulting in values from −3 (respondents who selected “a lot” for pre-knowledge and “nothing” for post-knowledge) to +3 (respondents who selected “nothing” for pre-knowledge and “a lot” for post-knowledge). Due to the meaningful differences between the topics in terms of the level of PES they represent (with learning about facts and applications representing more traditional educational models and learning about others’ views or societal aspects being more classically PES), items were analyzed separately rather than as a combined construct.

Several differences emerged, as shown in Figure 1. Event participants reported slightly greater learning gains about facts (U = 158711.50, n = 1,168, p = 0.028, r = −0.06) and applications of synthetic biology (U = 154174.50, n = 1,161, p = 0.007, r = −0.08). Forum respondents indicated greater learning gains about what other people think about synthetic biology (U = 153452.50, n = 1,156, p = 0.011, r = −0.08). While these are statistically significant, they represent small changes in the context of the survey, and the median value for both groups on all four topics was 1.0, representing a 1-point improvement for forum and event respondents alike. The range of change values for each item was −2 to 3 for event respondents and −1 to 3 for forum respondents, except for facts about synthetic biology, for which forum respondents’ change values ranged from −3 to 3.

FIGURE 1

Differences in responses to, “How much did you know about the following topics BEFORE this event/forum, and how much do you know AFTER the event/forum?” Note: Values of 1% or less are not labeled on the chart. Scores of −2 and −3 have been combined for each learning topic, and their combined totals are represented in black.

a U = 158711.50, n = 1,168, p = 0.028, r = −0.06

b U = 154174.50, n = 1,161, p = 0.007, r = −0.08

c U = 153452.50, n = 1,156, p = 0.011, r = −0.08

The second approach to assessing participants’ learning was the open-ended question, “What, if anything, did you learn from participating in this event/forum?” These responses were coded using the same code list, promoting comparison. All respondents, from both events and forums, were most likely to describe learning general facts about science or technology (38.3%) or applications of science (23.1%) (Table 3). There were no statistically significant differences between forum and event participants for these two codes. Among the less common codes, there were several differences. Forum respondents were more likely to describe learning about aspects of decision-making related to synthetic biology: what others think about science (12.7% of forum respondents versus 1.2% of event respondents), the complexity of scientific issues (8.0% versus 1.7%), risks of science (7.0% versus 1.9%), societal aspects of science (6.65% versus 1.7%), and ways the public can be involved in science (2.5% versus 0.2%). Event participants were more likely than forum respondents to describe learning about the significance of science in society (6.5% for hands-on activity respondents compared with 1.8% of forum respondents), particular aspects of the PES activities (4.6% versus 2.1%), and future directions/advancements in science (4.4% versus 1.8%).

TABLE 3

Responses to “What, if anything, did you learn from participating in this event/forum?”

Code	Hands-On Activities (n = 413)	Forum (n = 487)	Example Quotation
Science/technology general facts	41.6% (172)	35.5% (173)	“I learned a lot of specific science information.”
Applications of science	23.5% (97)	22.8% (111)	“There are many practical applications of synthetic biology.”
What others think about sciencea	1.2% (5)	12.7% (62)	“People have extremely varying opinions on what to do.”
The complexity of scientific issuesb	1.7% (7)	8.0% (39)	“There are many different ways to look at each problem to find the right solution.”
The benefits of science	3.6% (15)	6.2% (30)	“Benefits to environment.”
The risks of sciencec	1.9% (8)	7.0% (34)	“[I] learned more about the technology and risks.”
Societal aspects of scienced	1.7% (7)	6.7% (32)	“Ethical issues that may arise, religious perspective.”
The significance of sciencee	6.5% (27)	1.8% (9)	“The importance of synthetic biology.”
Current research	2.7% (11)	4.7% (23)	“There are many new experiments looking to end the scare of viruses affecting our world today.”
The activitiesf	4.6% (19)	2.1% (10)	“How to be a superhero.”
Lots of information	3.6% (15)	2.7% (13)	“Too many to list.”
Future directions/advancements in scienceg	4.4% (18)	1.8% (9)	“What science is doing for the future.”
What I need to consider (self-reflection)	2.4% (10)	2.9% (14)	“I learned about my own viewpoints on emerging technologies by fleshing them out.”
Public involvementh	0.2% (1)	2.5% (12)	“The importance of community dialogue.”

χ2(1, n = 900) = 43.046, Fisher’s Exact p < 0.001, ϕ = 0.219

χ2(1, n = 900) = 18.425, Fisher’s Exact p < 0.001, ϕ = 0.143

χ2(1, n = 900) = 12.783, Fisher’s Exact p < 0.001, ϕ = 0.119

χ2(1, n = 900) = 12.816, Fisher’s Exact p < 0.001, ϕ = 0.119

χ2(1, n = 900) = 12.798, Fisher’s Exact p < 0.001, ϕ = −0.119

χ2(1, n = 900) = 4.649, Fisher’s Exact p = 0.037, ϕ = −0.072

χ2(1, n = 900) = 4.839, Fisher’s Exact p = 0.031, ϕ = −0.073

χ2(1, n = 900) = 7.750, Fisher’s Exact p = 0.004, ϕ = 0.093

DISCUSSION

Public engagement with science is a promising means of developing trust between scientists and publics by allowing mutual learning through conversation about values and viewpoints. This paper has explored the affordances of two types of PES programming in informal learning environments: 1) forum programs involving dialogue, and 2) public events featuring hands-on activities.

Implications for educational design

The report Many Experts, Many Audiences: Public Engagement with Science and Informal Science Education (13), describes three dimensions along which informal education products can contain PES elements: the content of the project, how public audiences participate, and how experts participate (Table 4). Within the Building with Biology project, both the forums and the hands-on activities for public events included similar PES content related to the impacts and implications of synthetic biology as well as societal values related to its applications. However, the two products differed in how publics and scientists were involved. Public event participants engaged in a process of sharing viewpoints and knowledge with scientists who facilitated the activities. In the forums, publics and scientists were co-participants in dialogue and group problem-solving. These differing ways of interacting with the Building with Biology products may explain the differing public outcomes.

TABLE 4

Three dimensions of public understanding and public engagement with science.

	Content focus of the project	Audience involvement in the project	Expert involvement in the project
	Understanding of the natural and human-made world	Learning from watching, listening, and viewing lectures, media, exhibits, etc.	Experts serve as advisors and provide input to the project
	The nature of the scientific/engineering process or enterprise	Asking questions of experts and interactive inquiry learning	Experts actively present their expertise to the public
	Societal and environmental impacts and implications of STEM	Consultation and sharing views and knowledge among participants and experts	Experts work to become skilled and informed communicators
	Personal, community, and societal values related to STEM applications	Deliberation with other participants and group problem solving	Experts welcome and value participant inputs and direction
	Institutional priority or public policy change related to STEM	Participants produce recommendations or reports	Experts act on participant input and direction

Adapted from McCallie et al. 2009 (13).

Both the public events and forums successfully supported public learning about synthetic biology, and all participants valued this learning opportunity. However, there were differences in outcomes for participants in the forums and events. Generally, forums produced outcomes related to public involvement in deliberative processes. Forum participants had slightly greater gains in interest around future actions related to synthetic biology and PES; reported learning about public involvement with science as well as the interplay between science and society; and reported valuing of the interpersonal communication aspects that are central to dialogue programming. For public events, outcomes were related to increased understandings of the relevance of synthetic biology in participants’ lives. Publics reported that they valued the events’ synthetic biology content, access to experts, and overall enjoyment for themselves and their children. Additionally, public event participants reported a greater understanding of the significance of synthetic biology to their lives and the scientific future.

These findings suggest that forum programming is a strong approach for communicators who wish to promote potential follow-up behaviors including civic engagement, continued dialogue, or other public involvement in science. Forums are also a good way to provide a social, community-building experience. If reaching public audiences with enjoyable, free-choice family experiences is a major goal, the data suggest that public events featuring hands-on activities might be a good option. Public events may also be preferable if a communicator’s major goal is to help the public understand the relevance or importance of a STEM topic.

Study limitations

One limitation of this study was inconsistency across sites. The project allowed sites the flexibility to adapt the kits to suit their needs. Data collection did not track how many activities a respondent experienced or how long they engaged with each one, but from similar past projects we know that participants had diverse experiences. The training of volunteers who facilitated the hands-on activities varied as well, with 68% of volunteers receiving an orientation ahead of the event. Forums were generally more consistent in terms of length and type of participant experience. However, some forum sites featured live presenters whereas others relied on videos. Additionally, the proportion of scientist-participants varied, scientist training varied (65% attended an orientation), and there were two forum topics that sites could choose from. Both events and forums had variable attendance and audience composition (e.g., group type and age).

Additional inconsistencies may have arisen from the data collection process. The evaluation team made a substantial effort to train data collectors at each site via videos, written protocols, and ongoing mentorship. However, there was no way of ensuring fidelity to the prescribed data collection approach. Additionally, linking the event survey to the passport may have biased the sample if the passport appealed to some audiences more than others.

Finally, there were potential limitations in the data collection instruments. It was not the intention of the project to create validated scales. Rather, surveys were based on questions that had been used in previous PES projects, and questions were pilot-tested at Building with Biology sites in 2015.

Implications for future research and evaluation

This study raises a number of questions for future inquiry. Interest, value, and learning were the focus of this study, but additional studies may wish to explore different outcomes, including whether PES might reduce polarized views or enhance trust between scientists and publics. Further investigation of interest—including both situational and individual factors—could be valuable in better understanding how PES contributes to future behavior. Another promising line of future research would be isolating differences in outcomes by controlling for the time participants spent with the hands-on activities, ensuring comparable “dosage” of PES between events and forums and other types of outreach. Content analysis of forum and event conversations could also be valuable. Finally, this study focused on public participants, but PES involves both publics and experts. Future studies could benefit from comparing outcomes for these multiple audiences.