The effects of risk-taking, exploitation, and exploration on creativity.

The purpose of this paper was to investigate the effects of risk-taking, exploitation, and exploration on creativity by taking a model-based computational approach to both divergent and convergent thinking as primary ingredients of creativity. We adopted a reinforcement learning framework of Q learning to provide a simple, rigorous account of behavior in the decision-making process and examined the determinants of divergent and convergent thinking. Our findings revealed that risk-taking has positive effects on divergent thinking, but not related to convergent thinking. In particular, divergent thinkers with a high learning capacity were more likely to engage in risk-taking when facing losses than when facing gains. This risk-taking behavior not only contributes to the rapid achievement of learning convergence, but is also associated with high performance in divergent thinking tasks. Conversely, both exploitation and exploration had no significant effects on creativity once these risk attitudes were considered. Moreover, while convergent thinking relied on personality characteristics, it was not associated with risk-taking, exploitation, or exploration.

1. Introduction

Although creativity has long fascinated scientists, especially in the fields of neuroscience and cognitive psychology, more theoretical and empirical work is required for a thorough understanding of the mechanism and determinants of creativity [1, 2]. The purpose of this paper was to investigate the determinants of creativity by taking a model-based computational approach to creativity. We adopted a reinforcement learning (RL) framework to provide a simple, rigorous account of behavior in the decision-making process. The RL framework is supported by considerable empirical evidence, including neural signals in various cortical and subcortical structures that behave as predicted [3-6]. The framework has been applied to studies of decision making and learning in various social contexts [7-13]. However, little attention has been paid to creative aspects of decision making.

This paper assumed that a Q learning model in the RL framework underlies creative thinking and attempted to relate its properties to creativity performance. More specifically, we estimated parameters of Q learning such as risk-taking and inverse temperature based on the results of the Iowa Gambling Task (IGT). We investigated empirically how risk-taking and inverse temperature (exploitation/exploration) account for performance in creative tasks by controlling for several related cognitive and psychological factors. Thus, the Q learning model was related to creative thinking through risk-taking, exploitation and exploration in this study.

Several studies have revealed that creative people are energized and excited by challenging and risky problems [14, 15], which can lead to the experience of ‘flow’ [16]. These findings suggest that creativity is closely related to risk-taking. Indeed, several recent studies have investigated the relationship between risk-taking and creativity [17-23], yet the findings have been inconclusive. The differences in results could be accounted for by diversity in the research measures, definitions of risk-taking, and cultural backgrounds of the participants [24]. Furthermore, these studies commonly adopted questionnaire-based risk measurements, although the questionnaires used varied amongst the studies. Conversely, in the present study, we measured risk attitudes via the estimation of underlying utility function. We were interested in examining how the results would differ with different risk measures.

In addition to risk-taking, we also examined the effects of exploitation and exploration on creativity. Exploitation refers to the optimization of current tasks under existing information and memory conditions, while exploration implies wider and sometimes random search and trials that do not coincide with the optimal solutions provided by exploitation (see [25], for the trade-off between exploitation and exploration in the RL framework). Creativity requires both exploration and exploitation. In exploration, a wider search for a greater range of information is undertaken. In some cases, previously acquired knowledge must be unlearned so as not to get stuck with current knowledge, constraints or implicit assumptions, making it harder for them to ‘think outside the box’. At the same time, creativity also relies on exploitation because the efficiency of search in a much narrower space should take full advantage of existing information. Thus, both exploration and exploitation appear to be advantageous in creative thinking, although the relative weight of each depends on the phase of the creative thinking process.

In the Q learning model, exploitation implies the selection of choices yielding the highest Q values, whereas exploration involves a preference for other non-optimal choices. These factors can be represented via an inverse temperature scale using the softmax function, as described in the Methods section. On the scale, a higher (lower) value implies a greater (lower) emphasis on exploitation (exploration). This paper exploited this measure to examine the effects on divergent and convergent thinking performance in creative tasks. In so doing, we aim to shed new light on the underlying mechanism of creative thinking.

In so doing, it is important to distinguish between divergent and convergent thinking. While measures for creativity have been widely adopted in the neurosciences as a unitary and coherent construct by equating creative thinking with divergent thinking [2, 26], convergent thinking could also account for creative outcomes because a new idea must be tested as a candidate for solution to a practical problem. For example, convergent thinking is sometimes instrumental in generating insight during problem solving [27]. Accordingly, a number of recent studies investigated the difference between the two types of thinking [23, 28]. Following these studies, this paper examined the effects of risk-taking, exploitation, and exploration on both divergent and convergent thinking.

2. Methods

2.1. Participants

A sample of 113 healthy undergraduate students at Kobe University (49 females, age range = 18–20 years, SD = 0.66) participated in the study. All participants were native Japanese with normal or correct-to-normal vision. The local Ethics Committee at the graduate school of business administration, Kobe University, approved this study, and all participants and their academic advisers signed an informed consent form before the experiment and were paid JPY 3,000 (approximately USD 28).

2.2. Q learning model

We adopted a variant of Q learning model [25] to account for decision making in the IGT [29]. Participants make a series of 100 choices from four decks of cards. Two of the decks are advantageous and two of them are disadvantageous. The two disadvantageous decks always give rise to relatively high gains ($100) but also to occasional large losses ($150) with a 50% chance, which result in an average loss of -$25 per trial. The two advantageous decks always generate lower gains each time ($50) but produce no losses ($0) with a 50% chance, resulting in an average gain of +$25 per trial. The goal is to maximize net scores across trials.

At each trial t, the action value Q(t) of the chosen option (deck) i is updated via the following rule: with where R(t) is the reward associated with option i at trial t, and α± indicates the learning rate. U(R(t)) takes the form of the prospect utility function proposed by Tversky and Kahneman [30] in which μ and ν measure the degrees of risk aversion and risk seeking, respectively. We adopted this utility function because one of our research interests is the effect of risk attitudes on creativity. Thus, it was assumed that participants would evaluate the reward in terms of their own risk attitudes, which resulted in the utility function specified in [3].

The reward prediction error δ(t) is computed by subtracting the current value estimate from the obtained reward R. Participants thus update the action value estimate by scaling the prediction error with the learning rate and then adding this to the estimated value at the previous trial. Learning rates close to 1 indicate that a person makes fast adaptations based on prediction errors, and learning rates closer to 0 indicate slow adaptation. In the default setting, the initial action values are set to zero so that Q(1) = 0 for i = 1,…,4.

For the unchosen option j (i≠j), the action value is updated as:

Assume the chosen action at trial t is denoted by a(t)∈{1,2,3,4}. The action value estimates of these four options are used to determine the probability to choose either option. This probability is computed via the following softmax decision rule: where P(a(t) = i) is the probability to choose the action a(t) = i at trial t. The parameter β is the inverse temperature, a parameter that indicates the sensitivity of a participant’s choice to the difference in action value estimates. As described below, the inverse temperature β measures the relative strength of exploitation vs. exploration.

The parameters of α and β in this model were estimated by optimizing the maximum a posteriori (MAP) objective function, that is, finding the posterior mode: where p(D|θ) is the likelihood of data D for subject s conditional on parameters θ = {α,β}, and p(θ) is the prior probability of θ. We assume each parameter is bounded and use constrained optimization to find the MAP estimates. More specifically, since α is bounded between 0 and 1 and β takes non-negative values, their priors were assumed to follow beta distributions for α, and gamma distributions for β.

Although this Q learning model was specified for the IGT, we believe this model could also account for crucial aspects of creative thinking. In creative thinking, some ideas or alternatives must be chosen to further scrutinize existing possibilities. For this purpose, the individual must evaluate each idea or alternative before choosing one option. This aspect of decision making in creative thinking could be regarded as exercising the above Q learning model. In particular, convergent thinking hinges on this type of decision making to arrive at the correct solution. Even in divergent thinking, idea exploration requires the preliminary evaluation of candidate search fields. Therefore, the Q learning model could be interpreted as a model of creative thinking as well as a model for describing behavior in the IGT.

2.3. Measures

This paper used divergent and convergent thinking scores as dependent variables in the regression analysis. This is because there is growing support for taking both divergent and convergent thinking into account in creativity [31]. Our primary interest lies in the effects of exploitation and exploration on these dependent variables. As control variables that might affect divergent and convergent thinking, we used working memory capacities and personality scales.

2.3.1. Divergent thinking

Divergent thinking is defined as the ability to produce new approaches and original ideas by forming unexpected combinations from available information, and by applying such abilities as semantic flexibility, and fluency of association, ideation, and transformation [32]. In the current study, divergent thinking ability was measured with the S-A creativity test [33], a timed laboratory test corresponding to the measures used in the Torrance Test of Creative Thinking. The test involves three types of tasks. In the first task, participants are instructed to generate alternative ways of using objects specified in the test, which is known as an ‘alternative use’ test (AUT). The second task requires the participant to imagine desirable (and not realized) functions of specified ordinary objects. In the third task, participants are instructed to imagine the consequences of “unimaginable things” happening. Each task requires participants to generate as many answers as possible (up to 10).

The S-A creativity test measures divergent thinking in terms of (a) fluency, (b) flexibility, (c) originality, and (d) elaboration. Fluency is measured by the number of relevant responses to the questions and is related to the ability to produce and consider many alternatives. Flexibility is the ability to produce responses from a broad perspective and is measured by the sum of the total number of category types that answers are assigned based on a criteria table or an almost equivalent judgment. Originality is the ability to produce ideas that differ from others and its scoring is based on the sum of the idea categories that are weighted based on a criteria table or an almost equivalent judgment. Elaboration is the ability to produce ideas in detail and is measured by the sum of answers that are weighted based on a criteria table or an almost equivalent judgment. This test also provides the total score for divergent thinking, which was mainly used in this paper. For more detail about the S-A creativity test, see [34].

2.3.2. Convergent thinking

Convergent thinking is defined as the ability to apply conventional and logical search, recognition, and decision-making strategies to stored information to produce an already known answer [35]. Thus, convergent thinking requires knowledge and is typically correlated with measures of crystallized intelligence. However, most creativity researchers have described convergent thinking as a process entailing evaluation of initial ideas and/or a sudden insight in arriving at the correct solution for problems with task constraints [35-37]. As a result, in the insight problem-solving literature, convergent thinking has been typically measured by the Remote Associates Test (RAT) [38], which entails task constraints that the correct solution must fit with each of the three words in the presented triad (e.g., cheese for ‘cottage, cream, blue’). As all the participants in this study were native Japanese, we adopted a Japanese version of RAT developed by [39] where words were represented by Chinese characters. We used 40 problems selected by [40] in our experiment. RAT (convergent thinking) scores were measured by the number of correct solutions for the 40 problems.

2.3.3. Inverse temperature

We used the inverse temperature β to represent levels of exploitation and exploration. A higher β value implies that participants selected the decks based on the action value Q calculated in (1)~(3), leading to exploitation. Conversely, as β approaches zero, the choice is more likely to have been made randomly because the weight of the Q value in the soft max decision rule in (3) significantly declines. This implies that participants undertake exploration. Thus, the inverse temperature β measures the relative importance of exploitation and exploration.

2.3.4. Risk attitudes

As describe above, risk attitudes can be measured by the parameters μ and ν in (3), which incorporate part of the prospect utility function in which an asymmetric form of risk aversion is specified. Risk aversion in cases with positive rewards and risk seeking (risk-taking) in cases with negative rewards are respectively measured by μ and ν, which reflects the idea that participants have different risk attitudes towards gains and losses. We were interested in examining these effects on creativity performance.

2.3.5. Working Memory Capacity (WMC)

WMC was measured using reading span, operation span, and matrix span tests, which are representative working memory tests [41]. Reading span and operation span tests evaluate the capacity of verbal WMC and logical WMC, respectively, which in turn correspond to the phonological loop, according to Baddeley’s model [42]. The matrix span test measures spatial WMC, corresponding to the visuo-spatial sketchpad in this model.

2.3.6. Big five scale of personality

Big Five Scales (BFS) of personality traits are widely used to describe personality differences, which consist of five factors, namely openness to experience (inventive/curious vs. consistent/cautious), conscientiousness (efficient/organized vs. easy-going/careless), extraversion (outgoing/energetic vs. solitary/reserved), agreeableness (friendly/compassionate vs. challenging/detached), and neuroticism (sensitive/nervous vs. secure/confident) [43-45]. These scales were measured by 60 questions in Japanese, developed by [46]. The scores were measured in descending order. For example, high scores in openness to experience imply lower openness to experience. The descriptive statistics for all these variables used in the empirical analyses in this paper are reported in Table 1.

Table 1

Descriptive statistics.

Descriptive statistics
	Mean	SD	1		2		3		4		5	6		7		8	9		10	11		12
1. Divergent Thinking	39.87	10.07	―
2. Convergent Thinking	14.49	3.94	-0.08		―
3. μ (risk aversion in gains)	0.24	0.34	-0.08		0.07		―
4. ν (risk-seeking in losses)	0.26	0.34	0.09		0.03		0.52	***	―
5. Inverse temperature	1.36	1.88	0.04		-0.02		0.59	***	0.66	***	―
6. Extraversion	3.61	0.95	-0.05		0.12		-0.01		0.13		0.07	―
7. Neuroticism	3.29	0.97	0.08		0.02		-0.07		0.00		-0.02	-0.40	***	―
8. Openness	3.89	0.83	-0.16	*	0.00		-0.05		0.04		-0.09	0.23	***	-0.21	**	―
9. Conscientiousness	4.19	0.69	0.00		0.20	*	0.04		0.14		0.08	0.08		0.10		0.04	―
10. Agreeableness	3.4	0.89	0.12		-0.08		-0.25	***	-0.18	**	-0.14	0.37	***	-0.24	***	0.14	0.29	***	―
11. Spatial WMC	23.81	13.38	0.13		0.16	*	-0.08		0.00		0.04	0.04		0.00		-0.10	0.01		0.02	―
12. Verbal WMC	25.73	12.75	0.00		0.07		0.06		0.05		0.05	0.05		-0.07		-0.02	0.01		-0.08	0.37	***	―
13. Logical WMC	28.18	11.65	0.05		-0.12		0.02		0.11		0.10	-0.02		0.01		0.07	-0.01		-0.02	0.22	**	0.24	***

*p < .10

**p < .05

***p < .0.01

2.4. Procedure

Participants completed the S-A, Japanese RAT, reading span, operation span, and matrix span tests, the IGT and BFS tests. The experiments were presented in two independent sessions: a S-A/RAT session and a WMC/IGT/BFS session. Approximately half of the participants performed the two sessions in the order of the S-A/RAT session and then the WMC/IGT/BFS session. The remaining participants performed the sessions in the opposite order. There was at least a 10-day interval between the two successive sessions.

During the S-A/RAT session, participants completed the S-A test and RAT, which took approximately 70 minutes. The S-A test was completed in accordance with the instructor’s manuals. In the RAT, participants were given a practice session using two examples. Following the practice session, they engaged in 40 problems. Each problem was presented for 40 seconds to the participants. After 20 problems, a one-minute break was taken. Problems were presented in a randomized order each time, to remove order effects. A break of at least 5 minutes was taken between the two tests.

In the WMC/IGT/BFS session, participants completed reading span, operation span, matrix span tests [41], IGT, and BFS tests, which took approximately 60 minutes. This session was arranged for groups with a maximum of 50 participants in the presence of the instructor. The tests were performed on a 17” CRT monitor with PsytoolKit [47, 48]. A break of at least 1 minute was taken between the three tests. The order of these tests was also randomly assigned in this session. In the following discussion, reading span, operation span, and matrix span test scores are denoted by verbal WMC, logical WMC, and spatial WMC, respectively.

3. Results

3.1. Learning convergence

First, we analyzed the data from each participant regarding learning convergence. Each participant completed 100 trials of the IGT, and some of them eventually learned to consistently select the best (low risk, low return) decks. When participants repeatedly chose the best decks at least four times consecutively until the end of the game, we classified these participants as those in which learning convergence took place. Based on this definition, we identified 60 participants who succeeded in achieving learning convergence. In these participants, the average number of trials that were completed before learning convergence took place was 70.9 (SD = 29.5). To avoid overestimation of exploitation for those participants, we excluded the data from trials that took place after the achievement of learning convergence from the subsequent analysis. Thus, in addition to the pooled sample, we classified two subsamples, successful and unsuccessful participants, in terms of learning convergence.

3.2. Determinants of divergent thinking

To examine the effects of risk attitudes, exploitation and exploration on divergent and convergent thinking in the Q learning model, we first estimated the parameters of the learning rate (α), inverse temperature (β), risk aversion index for gains (μ), and risk seeking index for losses (ν) through the MAP estimation described above. We then implemented regression analyses on the determinants of both divergent and convergent thinking scores. Because both divergent and convergent thinking scores take only nonnegative integer values, we used the Poisson regression method to maintain statistical consistency.

First, we examined the data with respect to divergent thinking. The results are reported in Table 2. The dependent variable was the divergent thinking score. Columns (1) and (2) in the table show the results for the pooled sample, columns (3) and (4) refer to the successful subsample, and columns (5) and (6) show the results for the unsuccessful subsample.

Table 2

Determinants of divergent and convergent thinking.

Determinants of divergent and convergent thinking
(SE in parentheses)
Variables	Divergent thinking						Convergent thinking
	Pooled		Success		No success		Pooled		Success		No success
	(1)		(2)		(3)		(4)		(5)		(6)
Constant Terms	43.46	***	65.89	***	20.98	*	7.89	*	13.74	*	5.23
	(6.19)		(9.44)		(9.24)		(3.66)		(5.42)		(5.45)
μ (risk aversion in gains)	-3.97	.	-5.73	.	-5.91	*	1.52		-1.46		-1.94
	(2.30)		(3.01)		(2.63)		(1.42)		(1.83)		(1.57)
ν (risk-seeking in losses)	7.09	**	11.10	**	-1.52		-0.59		-3.40		1.61
	(2.52)		(3.40)		(3.24)		(1.53)		(2.14)		(1.94)
Inverse temperature	-0.12		-0.29		1.24	*	-0.24		-0.15		0.56	.
	(0.46)		(0.51)		(0.52)		(0.28)		(0.30)		(0.31)
Exraversion	-1.00		-3.21	**	1.12		0.87	.	0.89		0.93
	(0.75)		(1.10)		(1.09)		(0.45)		(0.65)		(0.65)
Neuroticism	0.49		-1.26		2.10	*	0.14		-0.40		0.56
	(0.71)		(1.06)		(1.02)		(0.43)		(0.63)		(0.61)
Openness	-1.98	**	-5.65	***	-0.19		0.09		1.99	*	-0.93
	(0.76)		(1.26)		(1.05)		(0.46)		(0.78)		(0.62)
Conscientiousness	-1.26		-0.09		-0.43		1.44	*	1.27		1.77	*
	(0.94)		(1.55)		(1.26)		(0.56)		(0.95)		(0.75)
Agreeableness	2.49	**	3.90	**	1.61		-0.98	*	-2.36	**	-0.65
	(0.82)		(1.37)		(1.05)		(0.49)		(0.84)		(0.63)
Spatial WMC	0.09	.	-0.08		0.22	**	0.06	*	0.05		0.12	*
	(0.05)		(0.07)		(0.08)		(0.03)		(0.04)		(0.05)
Verbal WMC	-0.03		0.06		-0.05		0.01		-0.01		-0.02
	(0.05)		(0.08)		(0.07)		(0.03)		(0.05)		(0.04)
Logical WMC	0.03		-0.07		0.06		-0.05	.	-0.15	**	-0.03
	(0.05)		(0.08)		(0.08)		(0.03)		(0.05)		(0.05)
AIC	905.31		432.13		460.71		634.76		305.19		330.04

*p < .10

**p < .05

***p < .01

In the pooled sample, the risk aversion index μ exerted a negative effect whereas the risk seeking index ν had a positive effect on divergent thinking. This implies that participants with learning convergence behaved in a risk-seeking manner, regardless of the gains or losses. Conversely, the inverse temperature β had no effect on divergent thinking, which suggests that risk attitudes, rather than exploitation/exploration, accounted for high levels of divergent thinking.

These results were also retained for the successful subsample. Regarding the control variables, openness to experience, extroversion, and agreeableness were identified as determinants of divergent thinking. Except for spatial WMC, the results were similar to those in the pooled sample. Since personality variables were measured in descending order, both openness to experience and extraversion exerted positive effects on divergent thinking. Moreover, the negative effect of agreeableness suggests that highly divergent thinkers favor challenging, rather than accepting, existing ideas. These results seem consistent with the definition of divergent thinking.

Regarding unsuccessful participants, the risk aversion index μ was negatively related to divergent thinking, whereas inverse temperature, neuroticism, and spatial WMC had positive contributions. The effect of the risk aversion μ was the same as that in successful divergent thinkers, which implies that both successful and unsuccessful divergent thinkers showed risk seeking in the face of positive gains. However, unsuccessful divergent thinkers relied more heavily on the results of the Q values due to the positive effect of inverse temperature. Compared with successful divergent thinkers, this exploitation effect seemed to inhibit learning convergence, although these participants achieved high divergent thinking scores. The positive effect of neuroticism likely contributed to high divergent thinking scores because high sensitivity facilitated internal memory search during the S-A test.

As Akbari, Chermahini, & Hommel [49, 50] reported nonlinear relationships between creativity and some of its determinants, we also tested the nonlinear effects of risk attitudes and inverse temperature in columns (2), (4), and (6) in Table 2. The positive effects of ν squared were identified according to columns (2) and (4), and the opposite effects were found according to column (6). Thus, in the pooled and successful samples, increasing returns emerged with respect to ν, which indicates the strong effect of ν on divergent thinking. In unsuccessful divergent thinkers, however, this effect had an inverted U shape, which suggests a much weaker effect.

Despite these contrasting effects, the results indicated that both successful and unsuccessful divergent thinkers engaged in risk seeking in the face of losses. Overall, highly divergent thinkers were more likely to engage in risk seeking as opposed to risk aversion, and the effects of exploitation and exploration were mostly negligible.

3.3. Determinants of convergent thinking

Next, we examine the determinants of convergent thinking. The results are reported in Table 3. The dependent variable was the convergent thinking score. As in Table 2, columns (1) and (2) in the table show the results for the pooled sample, columns (3) and (4) refer to the successful subsample, and columns (5) and (6) show the results for the unsuccessful subsample.

Table 3

Determinants of divergent and convergent thinking.

Determinants of divergent and convergent thinking
(SE in parentheses)
Variables	Divergent thinking						Convergent thinking
	Pooled		Success		No success		Pooled		Success		No success
	(1)		(2)		(3)		(4)		(5)		(6)
Constant Terms	43.42	***	67.87	***	12.51		7.47	*	13.91	*	4.52
	(6.21)		(9.56)		(10.29)		(3.66)		(5.46)		(6.08)
μ (risk aversion in gains)	2.71		-3.47		-9.80		-0.54		-2.32		2.17
	(10.28)		(11.79)		(11.44)		(5.96)		(6.94)		(6.84)
μ2 (risk aversion in gains)	-5.94		-3.23		4.03		2.23		0.50		-4.25
	(10.47)		(11.54)		(11.41)		(6.22)		(6.96)		(6.84)
ν (risk-seeking in losses)	-13.07		-12.16		23.33	.	7.76		-1.81		2.92
	(10.59)		(12.33)		(13.14)		(6.45)		(7.46)		(7.90)
ν2 (risk-seeking in losses)	20.63	*	21.78	*	-25.29	.	-7.60		-1.75		-1.12
	(10.01)		(11.02)		(13.12)		(6.07)		(6.59)		(7.89)
β (Inverse temperature)	0.32		0.14		2.16		-1.24		-1.73	.	0.79
	(1.39)		(1.72)		(1.36)		(0.82)		(1.05)		(0.82)
β2 (Inverse temperature)	0.00		-0.01		-0.09		0.12		0.20		-0.02
	(0.18)		(0.22)		(0.15)		(0.11)		(0.13)		(0.09)
Exraversion	-1.04		-3.23	**	1.57		0.96	*	1.35	.	0.95
	(0.76)		(1.16)		(1.14)		(0.45)		(0.71)		(0.67)
Neuroticism	0.44		-1.49		2.35	*	0.25		-0.14		0.52
	(0.72)		(1.09)		(1.05)		(0.43)		(0.65)		(0.62)
Openness	-1.93	*	-5.30	***	0.02		0.03		2.00	*	-1.01
	(0.77)		(1.28)		(1.08)		(0.47)		(0.79)		(0.65)
Conscientiousness	-1.17		0.20		-0.40		1.41	*	1.30		1.63	*
	(0.94)		(1.54)		(1.32)		(0.56)		(0.95)		(0.78)
Agreeableness	2.31	**	3.52	*	1.63		-0.98	.	-2.55	**	-0.53
	(0.83)		(1.43)		(1.09)		(0.50)		(0.87)		(0.65)
Spatial WMC	0.08	.	-0.08		0.27	**	0.07	*	0.07		0.12	*
	(0.05)		(0.08)		(0.08)		(0.03)		(0.04)		(0.05)
Verbal WMC	-0.01		0.08		-0.09		0.00		-0.01		-0.03
	(0.05)		(0.08)		(0.07)		(0.03)		(0.05)		(0.04)
Logical WMC	0.04		-0.06		0.06		-0.06	.	-0.15	**	-0.03
	(0.06)		(0.08)		(0.08)		(0.03)		(0.05)		(0.05)
AIC	906.71		305.19		462.25		638.32		308.47		335.52

*p < .10

**p < .05

***p < .0.01

According to this table, neither successful nor unsuccessful divergent thinking were associated with risk attitudes in any columns. However, inverse temperature was weakly correlated with unsuccessful divergent thinking. Therefore, regarding convergent thinking measured by the RAT, risk attitudes and exploitation/exploration did not play significant roles in achieving higher performance.

Regarding control variables, openness to experience, agreeableness, and logical WMC were identified in the successful subsample in both columns (3) and (4). In contrast to divergent thinking, successful convergent thinking hinged on negative openness to experience. Moreover, agreeableness was now positively related to convergent thinking, which implies that accepting, rather than challenging, the result of a memory search led to higher convergent thinking scores. Regarding the negative effects of logical WMC on convergent thinking, it is possible that logic per se does not play a critical role in the RAT because searching for the correct Chinese character that completes three words simultaneously requires more trial and error learning than logical reasoning. The results appear to suggest that high logical WMC impedes this trial and error learning attempt in the RAT in favor of logical reasoning.

As for unsuccessful convergent thinkers, conscientiousness and spatial WMC were significant in both columns (5) and (6). Once again, because the RAT assumed correct solutions, several candidate solutions must be organized efficiently to come up with correct solutions. As for spatial WMC, our RAT experiment used hieroglyphic Chinese characters, suggesting that the spatial WMC is positively related to convergent thinking. The results in the pooled sample reflected those results in an eclectic way.

Although the importance of these control variables could be interpreted intuitively, given our research interests, we want to emphasize our findings that that risk attitudes and exploitation/exploration did not play significant roles in convergent thinking for both the successful and unsuccessful subsamples.

4. Discussion

In this paper, the underlying cognitive mechanism of creativity was modeled in the RL framework. Recent computational analyses based on the RL framework were applied to studies of decision making and learning in various social contexts [7-12], showing the close correspondence between the phasic dopaminergic firing and the reward prediction error [6]. Although Hills, Todd, & Goldstone [51] made the first attempt to relate exploitation and exploration to creativity tasks, and several studies have directly modeled the creativity process (see [52] for review), we propose an alternative computational model of creativity using the simple RL framework. One of the contributions of this paper was that we examined the effects of risk attitudes and exploitation/exploration on creativity performance using this RL framework.

From this perspective, the current study revealed the novel finding that divergent thinking was explicitly related to risk seeking, whereas convergent thinking was not associated with risk attitudes or exploitation/exploration. These results stand in sharp contrast to the findings reported by Shen, Hommel, Yuan, Chang, & Zhang [23], who found that while low risk-taking contributed to better convergent thinking, it was not significantly correlated with divergent thinking. One reason for this disparity could be the different measures of risk attitudes. Namely, Shen, Hommel, Yuan, Chang, & Zhang [23] used the risk-taking preference index, while in the current study, we measured risk attitudes by imposing a prospective utility function during the IGT. These different measures could highlight separate aspects of risk-taking. The differences between the risk-taking preference index and the risk attitudes derived from the utility function should be explored in future studies.

Although they were not identical, Tyagi, Hanoch, Hall, Runco, & Denham [22] employed similar risk-taking measures to Shen, Hommel, Yuan, Chang, & Zhang [23]. They found that creativity was associated with high risk-taking tendencies in the social domain, and that the likelihood of social risk taking was the strongest predictor of creative personality and ideation scores. These results are consistent with our present findings.

The contrasting results among different studies might reflect cultural differences. As Shen, Hommel, Yuan, Chang, & Zhang [23] noted, while divergent and convergent thinking were positively correlated in their study, the experiment conducted in the Netherlands revealed correlations that were close to zero and, if anything, negative (e.g., [49]). In our study, as revealed in Table 1, the correlation between divergent and convergent thinking was close to zero. Thus, different roles during risk-taking might reflect cultural differences between China and Japan.

Regarding convergent thinking, it was found that both risk attitudes and inverse temperature did not account for its performance at least explicitly. However, we should be cautious about interpreting this result because, as we have noted, the RAT required both divergent and convergent thinking to yield correct solutions. Moreover, the correct solution was sometimes obtained through insight. That is, some participants solved some questions with insight. The mixture of problem solving with and without insight makes it difficult to identify the relative contributions of exploitation and exploration and risk attitudes. It has been claimed that solutions to insight problems are unpredictable [53], difficult to report [54], and solved in a distinct manner [55]. In other words, insight generates new information that is often discrete and domain-specific and transcends informational boundaries, yet still provides some value. This might be the result of underlying risk-taking exploration activities. In contrast, problem solving without insight goes through a step-by-step, risk averse incremental process, which corresponds to exploitation. Consequently, it could be that the mixture of problem solving with and without insight in the RAT obscures significant effects of both exploitation and exploration.

Insight problem solving does demand distant or remote associations, such as spreading activation in semantic memory, which operates largely outside of conscious attentional control [27, 56–58]. This suggests that it requires substantially less working memory than problem solving without insight. In support of this, it has been shown that higher WMC is positively related to analytical problem solving, but unrelated to insight [59]. Similarly, a higher degree of working memory load negatively affects analytic problem solving but has no impact on insight problem solving [55].

Problem solving with and without insight are likely to both involve many of the same cognitive processes and neural mechanisms. Indeed, some studies have suggested that insight plays no role and that creativity is essentially identical to analytical problem solving [56, 60–63]. Murray and Byrne [64] suggested that people with high WMC perform better on insight problems than people with less WMC. De Dreu, Nijstad, Baas, Wolsink, and Roskes [65] also performed several studies that indicated that high WMC was related to higher performance in a range of different creative tasks. Because both insight and analytical problem solving were involved in the RAT, these studies were consistent with our results that the spatial WMC was positively related to higher convergent thinking scores for unsuccessful participants.

The reason that only the spatial WMC positively accounted for convergent thinking was that the RAT used hieroglyphic Chinese characters in our experiment. Participants were required to memorize the forms of Chinese characters while searching for solutions. Regarding the negative effect of logical WMC on convergent thinking, as noted above, it is possible that logic per se does not play a critical role in the RAT because it requires trial and error learning rather than logical reasoning. The current results appear to suggest that high logical WMC impedes this trial and error approach in the RAT in favor of logical reasoning.

Spatial WMC also positively accounted for divergent thinking, except for in successful participants. In the alternative use test that was part of the S-A test, participants were asked to write alternative uses of given items. In this test, participants had to keep track of images of given items. Those who were slower learners might have needed to keep the image in their memory for longer, requiring higher spatial WMC. However, quick learners would not have to keep the image in their memory for long, meaning less reliance on spatial WMC.

The different effects of risk-taking on divergent thinking between successful and unsuccessful participants also deserve some remarks here. While risk seeking in the face of gains was positively related to divergent thinking for both subsamples, risk-seeking in the face of losses accounted for divergent thinking only for quick learners (successful participants). This result suggests that quick learners were more sensitive to losses, leading to more risk-taking behaviors. Thus, greater risk-taking regarding losses might differentiate quick from slow learners.

It should also be noted that inverse temperature (exploitation/exploration) did not account for both divergent and convergent thinking. Although we did not report in this paper, regression analyses without risk parameters revealed that inverse temperature was strongly significant for divergent thinking. However, once the risk parameters were incorporated, the significance was lost. This implies that risk attitudes, rather than exploitation/exploration, accounted for divergent thinking in our model. Therefore, as related studies have suggested [22, 23], risk attitudes appear to play a critical role in creativity.

Thus, the current findings revealed that divergent thinking tends to favor risk-taking rather than exploitation and exploration. In this respect, related neuroscientific studies of creativity [2, 66, 67] have underlined the importance of the default mode network in divergent thinking [68-72]. However, while the identification of neural substrates underlying divergent thinking has certainly improved our understanding of creativity mechanisms, the specific cognitive model behind the divergent thinking process has not yet been rigorously articulated.

This paper contributes to linking risk-taking to divergent thinking in the framework of Q learning and empirically confirms their relationship. However, the challenge of articulating how risk-taking in divergent thinking is related to underlying neural substrates, particularly the default mode network, remains unresolved.

Finally, it should be pointed out that risk parameters, exploitation and exploration in this study were not measured during divergent/convergent thinking tasks. Therefore, the relationships exhibited in the current study between risk attitudes, exploitation/exploration, and divergent/convergent thinking should be considered as indirect evidence, and the results do not unequivocally confirm that divergent thinkers always implement risk-taking behaviors. Obviously, measuring risk attitudes, exploitation and exploration during creative tasks remains a challenge for future research.

5. Conclusion

This paper related risk attitudes, exploitation and exploration specified in a simple Q learning model to creativity measured by convergent and divergent thinking. In other words, we examined the determinants of divergent and convergent thinking in terms of risk attitudes, exploitation and exploration as cognitive processes. While many creativity studies focus on neural substrates of creativity using EEG and fMRI data, this study was different in its introduction of a computational approach to creativity research.

Findings revealed novel characteristics of risk-taking in divergent thinking. That is, efficient divergent thinkers tend to engage in risk-taking, rather than risk-averting behavior. In particular, quick learners are more likely to exhibit higher levels of risk-taking in the face of losses compared with gains. This risk-taking behavior not only contributes to fast learning in the IGT, but also facilitates high performance in terms of divergent thinking.

To our knowledge, this study is the first to explore the cognitive mechanisms underlying creativity in the framework of reinforcement learning. Future progress in this area will require several challenges to be addressed. In particular, neural substrates of both risk attitudes and exploitation/exploration, together with convergent and divergent thinking, should be identified with neuroscientific methods such as EEG and fMRI.

16 Mar 2020

PONE-D-20-02635

Effects of Exploitation vs. Exploration on Creativity in Q Learning

PLOS ONE

Dear Prof. Harada,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Though Reviewer 1 rejected PONE-D-20-02635, the reviewer provided many valuable and constructive comments. Considering both reviewers’ useful comments and the interesting topic of the manuscript, I would like to give you a chance to revise your manuscript during the special period. The revised manuscript will undergo the next round of review by the same reviewers.

We would appreciate receiving your revised manuscript by Apr 30 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Baogui Xin, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please change "female” or "male" to "woman” or "man" as appropriate, when used as a noun.

3. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

4. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Partly

Reviewer #3: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: No

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript addresses the question of how two measures of creativity (divergent vs. convergent) thinking relates to exploration – exploitation strategies in the version of Iowa Gambling Task. The author made a prediction that divergent thinking measured through the S-A creativity test is related to more exploratory behavior while convergent thinking should bias behavioral strategies towards more exploitation. However, the obtained results do not confirm this hypothesis. Individuals who scored more on divergent thinking show some relationship to both exploration and exploitation (which depends on the chosen Q-learning model) while convergent thinking does not show any relationship with exploration-exploitation trade-off in this particular task.The author then made some post-hoc interpretations trying to reconcile the obtained results. Despite the fact that I appreciate the effort of trying to investigate the link between such a complex phenomenon as creativity and learning, these results alone do not lead to any conclusion and do not provide any clear guidelines to future research and, hence, could not be published.

Reviewer #2: The approach underlying this study is very interesting and the general method is useful. However, some possibly crucial information is missing, which may have serious impact on the results and their interpretation. The paper also shows a lack of scholarship, as it is presented as the first to consider the difference between convergent and divergent thinking--thus ignoring numerous studies looking into the difference between these two types of creativity. All this makes it difficult to judge whether the paper makes a useful contribution to the literature.

MAJOR

1. A big deal is made of the difference between divergent and convergent thinking. Not only does this ignore the key point of Guilford, who had introduced this terminological difference, but it also overlooks quite a number of studies comparing divergent and convergent thinking. For convenience, I list just a number of papers from my own lab:

Prochazkova, L., Lippelt, D.P., Colzato, L.S., Kuchar, M., Sjoerds, Z., & Hommel, B. (2018). Exploring the effect of microdosing psychedelics on creativity in an open-label natural setting. Psychopharmacology, 235, 3401-3413.

Shen, W., Hommel, B., Yuan, Y., Chang, L. & Zhang, W. (2018). Risk-taking and creativity: Convergent, but not divergent thinking is better in low-risk takers. Creativity Research Journal, 30, 224-231.

Colzato, L.S., de Haan, A., & Hommel, B. (2015). Food for creativity: Tyrosine promotes deep thinking. Psychological Research, 79, 709-714.

Zmigrod, S., Colzato, L.S., & Hommel, B. (2015). Stimulating creativity: modulation of convergent and divergent thinking by tDCS. Creativity Research Journal, 27, 353-360.

Zmigrod, S., Zmigrod, L., & Hommel, B. (2015). Zooming into creativity: Individual differences in attentional global-local biases are linked to creative thinking. Frontiers in Psychology, 6:1647.

Lippelt, D.P., Hommel, B., & Colzato, L.S. (2014). Focused attention, open monitoring and loving kindness meditation: Effects on attention, conflict monitoring and creativity. Frontiers in Psychology, 5:1083.

Reedijk, S.A., Bolders, A., & Hommel, B. (2013). The impact of binaural beats on creativity. Frontiers in Human Neuroscience, 7:786.

Hommel, B., Colzato, L.S., Fischer, R., & Christoffels, I. (2011). Bilingualism and creativity: Benefits in convergent thinking come with losses in divergent thinking. Frontiers in Psychology, 2:273.

Akbari Chermahini, S., & Hommel, B. (2010). The (b)link between creativity and dopamine: Spontaneous eye blink rates predict and dissociate divergent and convergent thinking. Cognition, 115, 458-465.

Hommel, B. (2012). Convergent and divergent operations in cognitive search. In: P.M. Todd, T.T. Hills, & T.W. Robbins (eds.), Cognitive search: Evolution, algorithms, and the brain (pp. 221-235). Cambridge, MA: MIT Press.

2. The author also ignores existing attempts to specify the mechanisms underlying convergent and divergent thinking; e.g.,:

Zhang, W., Sjoerds, Z., & Hommel, B. (2020). Metacontrol of human creativity: The neurocognitive mechanisms of convergent and divergent thinking. Neuroimage, 210:116572.

Mekern, V., Hommel, B., & Sjoerds, Z. (2019). Computational models of creativity: A review of single- and multi-process recent approaches to demystify creative cognition. Current Opinion in Behavioral Sciences, 27, 47-54.

Mekern, V.N., Sjoerds, Z., & Hommel, B. (2019). How metacontrol biases and adaptivity impact performance in cognitive search tasks. Cognition, 182, 251-259.

Hommel, B., & Colzato, L.S. (2017). The social transmission of metacontrol policies: Mechanisms underlying the interpersonal transfer of persistence and flexibility. Neuroscience and Biobehavioral Reviews, 81, 43-58.

3. The first attempt to relate exploitation and exploration to creativity tasks was also ignored:

Hills, T. T., Todd, P. M., & Goldstone, R. L. (2008). Search in external and internal spaces. Psychological Science, 19(8), 802–808.

4. The parameter is used to indicate exploitation and exploration are likely to be affected, if not confounded with risk taking. Given that risk-taking has an impact on creativity, this is a serious problem: Shen, W., Hommel, B., Yuan, Y., Chang, L. & Zhang, W. (2018). Risk-taking and creativity: Convergent, but not divergent thinking is better in low-risk takers. Creativity Research Journal, 30, 224-231.

5. It remains unclear what the ratio between optimal and non-optimal choices is supposed to represent theoretically/mechanistically. Given that this is the only parameter that shows relatively systematic effects, more theoretical foundation is necessary.

6. Divergent thinking was assessed by means of a non-established, self-developed task that doesn't seem to be validated. Why was AUT not used instead?

7. It remained unclear how the findings shown in tables 2 and 3 were generated. It seems that all available predictors were entered into a linear regression analysis. This should be clarified. If my guess is correct, we are facing the following problems: a) the predictors are obviously highly interrelated, at least in several cases, which is an absolute no go for regression analyses; b) this analysis can only assess linear effects, even though nonlinear functions seem to play a much more decisive role: e.g., Akbari Chermahini, S., & Hommel, B. (2012). More creative through positive mood? Not everyone! Frontiers in Human Neuroscience, 6:319. Akbari Chermahini, S., & Hommel, B. (2010). The (b)link between creativity and dopamine: Spontaneous eye blink rates predict and dissociate divergent and convergent thinking. Cognition, 115, 458-465.

8. It remains unclear which outcome suggests that "divergent thinkers pursue both exploitation and exploration simultaneously". The fact that the ratio is involved does not speak to this issue at all, as the values are likely to represent different trials.

9. The two models do not replicate each other. Given that they are theoretical status is unclear, this means that be don't know what the outcome means.

10. In the discussion, there are repeatedly Chinese characters mentioned. I had no idea where they come from. In the description of the RAT, Chinese characters were not mentioned, and I found it difficult to imagine an RAT in which Chinese characters play a role.

signed, Bernhard Hommel

Reviewer #3: Review of Effect of exploitation vs. exploration on Creativity in Q-learning by Tsutomu Harada.

In this study, the author investigates the relationship between exploration and exploitation as defined in the reinforcement learning paradigm and convergent and divergent thinking as measured with different creativity tests. To this end, the author first analyzed the behaviors of 113 subjects performing the Iowa Gambling Task, by modelling the learning process with two different versions of the Q-Learning algorithm. From this analysis, measures of exploration and exploitation are derived. Secondly, participants performed different creativity tasks, some assessing more convergent thinking and other divergent thinking. The hypothesis tested in the study is that convergent thinking is linked to exploitation whereas divergent thinking is linked to exploration, but counterintuitively, the author found opposite results. Indeed, divergent thinking was more associated with exploration and divergent thinking with exploitation.

The question investigated in this study (i.e. investigating computationally the cognitive mechanisms underlying creativity) is interesting and a wide range of data has been collected to answer this question. However, essential results and critical quality checks are missing in the current version of the paper, particularly in the IGT / Q-Learning part on which most of my major concerns focus.

Major Concerns

#1 The exploration and exploitation measures are directly linked to the subjects' performance on the task which compounds one condition (repeated choices between the 4 same decks of card) across 100 trials. I would be interested in seeing this performance. Do the subjects learn to pick the best card(s) eventually and if so when (at which trial) are they starting to select the best option(s) consistently? This aspect is critical, particularly as the task is quite long, if subjects learn which is the best option after 20 trials, it is very likely that the choices made in the 80 remaining trials will be defined as exploitation, and that would just be the results of subjects learning to play the task and not a strategy that could be opposed to exploration. Depending on this result, the author could consider doing a similar analysis focusing on the learning phase (before reaching the plateau) that is maybe occurring during the first 30/50 trials? This may not be necessary but showing the performance of subjects is the only way to make sure of that.

#2 In the study, two different versions of Q-learning are used to explain subjects' behaviors in the IGT but we are not given any indications about the quality of their respective fit. Do these models explain well the data and is one of these better than the other? This is a crucial point as the exploration/exploitation measures used in the regression analyses are directly based on the models' predictions. In addition, if one of the model explains parsimoniously better the data than the other, I doubt that a regression analysis based on the "worst" model would be meaningful enough. Can the author give a reason for doing such an analysis?

#3 Following my previous concern, I would like to know the author's justifications about the candidate models selection. The two models used are rather different, the most sophisticated one including two new essential features (i.e. an asymmetry in learning rates as well as dynamic learning rates). We can imagine a wider model space which would also include models with only one of these features and compare their predictions. In any cases, it would be interesting to get further details about the selection of these particular features and models.

#4 Regarding the model fitting, emerging methods are used to ensure that the parameters optimization and models comparison procedures are reliable in a given task (See Palminteri et al., The Importance of Falsification in Computational Cognitive Modeling, 2017). The latter consist in testing the fitting procedure on synthetic data obtained from the models of interest. A parameter recovery analysis allows to check if the fitting procedure is able to retrieve the generative parameters from simulated data whereas a model recovery analysis allows to check if the models comparison procedure is able to recognize the generative model of the data, in a given task. Such analyses would strengthen the modelling part of the paper, which is crucial as the exploration/exploitation measures used in the regression analyses depend on it.

#5 Exploration in the paper is rather extremely defined as the number of choices of the option with the lowest Q-value. That makes, for instance, a choice of the second worst option, not an exploratory one. Can the author justify this narrow definition of an explorative choice? And are the results different if a broader definition is used (e.g. any choice that is not exploitative is explorative. Or is explorative a choice of an option associated to one of the two lowest Q-values)?

#6 In the reinforcement learning literature, the temperature (or inverse temperature here) can be interpreted as a parameter governing the exploration/exploitation tradeoff. I am surprised not to see any attempt at correlating this parameter with either measures of exploration / exploitation or directly with scores of convergent and divergent thinking. Has the author explored this aspect?

Minor Concerns

#1 Page 6 of the manuscript, it is written that the average loss concerning the two disadvantageous decks is -$50, is it not -$25 instead?

#2 Concerning the second model, the dynamic learning rates seem not to be bounded between 0 and 1, is it the case? If it is, I think it would be important to consider adding details about parameters bounds.

#3 Page 10, last sentence of the Asymmetric time-varying Q-Model paragraph, equations references seem to be wrong, they should be 4 and 5 instead of 6 and 5.

#4 Pages 18 and 22, the numbers of stars in the legend of the table are erroneous. One star for p<0.1 and for p<0.05.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Bernhard Hommel

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: review_PONE-D-20-02635_Skvortsova.docx

Click here for additional data file.

7 May 2020

We greatly appreciate very useful comments and suggestions by reviewers. We believe the current revised manuscript responded to most of the reviewers' comments and requests. For details, please see the attached "Response to Reviewers".

Submitted filename: Revision list.docx

Click here for additional data file.

22 Jun 2020

The Effects of Risk-taking, Exploitation, and Exploration on Creativity

PONE-D-20-02635R1

Dear Dr. Harada,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Baogui Xin, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Based on the reviewers comments and suggestions the authors have considerably revised the manuscript, changed analyses, hypothesis and interpretation of their findings.

In the modified manuscript, the authors align their reanalysis on the hypothesis and suggestions made by the Reviewer #2 suggesting that it is risk-seeking attitudes but not exploration-exploitation trade-off that positively influences diverging thinking in creativity. In light of this new hypothesis the authors have also revised their computational model by integrating utility function into the Q-learning model to characterize risk-seeking or risk-averse distortions of outcomes.

I believe that authors have thoroughly reconsidered their manuscript although I have concerns with obtained findings and would be cautious with the interpretations. Firstly, obtained results stands clearly as a post-hoc interpretation of the data and their exploratory nature should be acknowledge and discussed. Secondly, most of the obtained effects are relatively weak (with p-value < 0.1 or 0.05) raising questions about their replicability.

Furthermore, sometimes authors do interpret the non-significant results (page 20 of the revised manuscript: “In the pooled sample, the risk aversion index μ exerted a negative effect whereas the risk seeking index ν had a positive effect on divergent thinking. This implies that participants with learning convergence behaved in a risk-seeking manner, regardless of the gains or losses”. In the Table 2 the effect of parameter μ is negative but does not reach significance so this statement is misleading.

Overall, I still think that the paper is weakly convincing but considering the overall interest of the subject it might be published as an exploratory paper in the field bridging instrumental learning and creativity testing.

Minor comments:

- The authors completely eliminated any model comparison and do not justify with their data whether adding utility function in the Q-learning model (which comes at a cost of 2 additional free parameters) actually accounted for the data better than the simple Q-learning model without risk-taking component.

- Text on page 22 of the revised manuscript does not match data in the Table 3: “The dependent variable was the convergent thinking score. As in Table 2, columns (1) and (2) in the table show the results for the pooled sample, columns (3) and (4) refer to the successful subsample, and columns (5) and (6) show the results for the unsuccessful subsample”.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

15 Jul 2020

PONE-D-20-02635R1

The Effects of Risk-taking, Exploitation, and Exploration on Creativity

Dear Dr. Harada:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Baogui Xin

Academic Editor

PLOS ONE