Application of Quantum-Markov Open System Models to Human Cognition and Decision.

Markov processes, such as random walk models, have been successfully used by cognitive and neural scientists to model human choice behavior and decision time for over 50 years. Recently, quantum walk models have been introduced as an alternative way to model the dynamics of human choice and confidence across time. Empirical evidence points to the need for both types of processes, and open system models provide a way to incorporate them both into a single process. However, some of the constraints required by open system models present challenges for achieving this goal. The purpose of this article is to address these challenges and formulate open system models that have good potential to make important advancements in cognitive science.

1. Introduction

One of the most fundamental topics in the cognitive and neural sciences concerns the dynamic and stochastic processes that humans (as well as other animals) use to make choices and decisions. Consider the following example of what is called a signal detection task: Suppose a radiologist is examining an image of a breast and trying to decide whether or not a cancerous node is present. The process requires accumulating evidence across time (by looking at various parts of the image) until a sufficient amount of evidence has been accumulated to make a decision. However, the decision also depends on the consequences that can occur, which depend on the true state and final choice. For example, missing a cancerous node may allow a cancer to grow into a more dangerous medical problem and falsely deciding that cancer is present produces a great deal of unnecessary stress and additional medical testing. The decision process is fundamentally probabilistic in the sense that if the same radiologist is presented the same image on two different occasions (separated in time with other images in between), she might make different decisions. In addition, the decision on each occasion takes time and the time to make the decision varies across occasions too. Finally after a decision is made, the radiologist could be asked to report how confident she is about her decision. Therefore the basic measures that are collected in a signal detection task are the probability of making each choice and confidence rating, and the distribution of decision times for each choice.

For over 50 years these types of decisions have been successfully modeled by cognitive scientists (see, e.g., [1]) and more recently also by neuroscientists (see, e.g., [2]) using Markov processes, such as random walk (discrete) or diffusion (continuous) processes. The general idea is similar to a Bayesian sequential sampling decision rule [3]. Using the radiologist example, the decision maker starts with some initial state of evidence (e.g., log likelihood) for or against the cancer hypothesis, denoted . During each moment in time t, the decision samples evidence, denoted , which increases or decreases the current state, . This evidence continues to be accumulated in the direction of the mean evidence until its magnitude exceeds a threshold bound at which point in time, say T, the decision maker stops and decides that cancer present if or decides cancer is not present if . These Markov models of evidence accumulation provide very accurate predictions for empirical distributions of choice and decision times from human decision makers [1] as well as predicting neural activation trajectories from electrophysiological recordings of neurons in primates [2].

Despite the success of Markov processes for modeling these types of signal detection tasks, there is empirical evidence that this class of model may not provide a complete picture of human decision making. Recent research suggests that an alternative way to model evidence accumulation, based on quantum walks, is also needed [4,5,6,7]. (This article is focused on applications of quantum dynamics to human decision making. There are many other applications of quantum theory to decision making that do not involve dynamics, which are not reviewed here. See [8] for a review. Quantum theory has also been applied to strategic games, see [9] for an interesting example, but this work is outside the scope of this article.)

One line of empirical evidence supporting quantum models comes from interference effects of choice on later confidence. In one experiment [5], a signal detection task was used to compare the results from two different conditions: (1) A choice-confidence condition under which participants started observing an image at and made a binary decision regarding the image at time , and then continued to observe the image until they made a confidence rating at time , and (2) a confidence-alone condition under which participants again started observing an image at , but simply made a pre-planned button push (to control for responding while not requiring any decision about the image) at time , and then continued to observe the image until they made a confidence rating at time . The critical test concerns the distribution of confidence ratings observed at time (pooled across choices at time for the choice-confidence condition). A Markov random walk model predicts no difference between conditions because it satisfies the Chapman–Kolmogorov equation and the quantum walk predicts differences, because a wave function “collapse” occurs at time under the choice-confidence condition but not under the confidence-alone condition. The results of the experiment found significant interference effects, contrary to the Markov model and supporting the quantum model. A second follow-up study, again using confidence ratings, found further support for the quantum walk over the Markov random walk [6].

A second line of empirical evidence supporting quantum models comes from temporal oscillations in preference. In one experiment [7], a preference task was used to investigate how preferences evolve across deliberation time. Participants were presented a choice between two different gift coupons for restaurants, which varied according attributes including the quality of the restaurant, the distance to the restaurant, and the monetary amount of the gift card. They rated their strength of preference for one gift over the other across a series of time points. Markov random walk models used in cognitive science to model preferences (see, e.g., [10]) predict that mean preference strength should monotonically increase across time in the direction of the alternative with greater mean utility. In contrast, a quantum walk model predicts that preferences should oscillate while moving in the direction of the alternative with greater mean utility. The results of the experiment found significant oscillation effects, contrary to the Markov model and supporting the quantum model.

In addition to these lines of evidence, quantum dynamics have been used to account for violations of rational decision making [11], as well as several dynamic decision inconsistencies [12].

In sum, properties of both Markov and quantum walk models may be needed to capture the full probabilistic dynamics underlying human choice and decision making. More fundamentally, Markov and quantum models represent two different types of uncertainty in the decision process [13]. Markov models represent an epistemic type of uncertainty in which an outside observer is uncertain about the internal location of evidence of the decision maker. Quantum models represent an ontic type of uncertainty in which there is no preexisting location of evidence before a decision is made. Instead, at each moment, the decision maker is in a superposed state with several different levels of evidence having potential to be realized, so that the decision maker has internal uncertainty about the level of evidence. Open system models are ideally suited for combining these two different types of dynamics into a single unified process [11,14,15,16,17].

Open system models were developed to represent quantum systems that are described by a combination of a target system of interest coupled to a complex and uncontrolled environment [18]. The original system-plus-environment model relies on purely unitary dynamics that generate interactions between the system and environment. The open system dynamics is derived from unitary dynamics by marginalizing (partial tracing) over the environment to focus on the dynamics of the system alone. The dynamics of the resulting open system starts out in a quantum regime in a“coherent” superposition state however, the dynamics produced by the environment produce decoherence and eventually reduces the system to a classical probability mixture that evolves according to a classical Markov system.

Methods for constructing open system models for applications in physics have been very thoroughly developed. But how can this work in cognitive science? The purpose of this article is to provide some guidelines for applying open system models to cognitive science in a compelling and effective manner.

2. Results

Before jumping into the general open system model, it may be helpful to first review versions of Markov and quantum walk processes in isolation. Both quantum and Markov processes can be developed using any combination of discrete versus continuous state, and discrete versus continuous time assumptions. For example, a standard random walk is a discrete state and time Markov chain, and the diffusion model is a continuous state and time Markov process, and quantum models for position and momentum are continuous state and time processes, but the “coin” quantum walk (for review, see [19]) is a discrete time and state model. Furthermore, note that as the increments between states and time steps decrease in size, the discrete models converge to the continuous models (see, e.g., Ref. [20], for Markov processes and Ref. [21] for quantum processes). For ease of presentation, we will work with Markov and quantum walks that are discrete state and continuous time. We present the two different classes of models in a parallel manner to illustrate their common and distinctive features.

2.1. Comparison of Markov and Quantum Walk Models

Consider a model in which the decision maker represents their beliefs within a dimensional vector space. The 101 basis vectors (eigenstates), symbolized as , represent 101 increasing levels of evidence for one hypothesis over another. Using the radiologist example, the basis vector represents 0.00 likelihood that cancer is present (1.0 likelihood that there is no cancer), represents a likelihood favoring cancer, represents equal likelihood, represents a likelihood favoring the cancer, and represents a likelihood favoring cancer (0.00 likelihood of no cancer). This fine grain evidence scale provides a close approximation of the finite state model to a continuous state model. (Cognitive science models, see, e.g., [1], often use a continuous scale of evidence).

Using the evidence basis to form a belief state, each basis vector can be assigned a coordinate value. For a Markov model, each coordinate is a probability, , representing the probability that the system is located at that level of evidence. For a quantum model, each coordinate is a complex amplitude, representing the potential to observe that level of evidence. The 101 coordinate values form a column matrix denoted here as for a Markov model and for a quantum model.

Suppose is an arbitrary vector in the space. A Markov model uses an norm defined by to define length, and requires . In other words, the probabilities must sum to one. A quantum model uses an norm defined by and requires . In other words the squared magnitudes of the amplitudes must sum to one.

A measurement in this space is represented by a projector. With respect to the evidence basis, the projector is simply a diagonal matrix, denoted as for response R, with zeros everywhere except one’s located at levels of evidence that represent the response. For example, a response to report a confidence rating equal to 0.65 could be represented by a diagonal matrix with a one located at the row 66 (corresponding to the basis vector ) and zeros elsewhere. A response to decide cancer is present could be represented by a diagonal matrix with one’s located at rows 51 to 100 and zeros otherwise. For a Markov model, the probability of a response R is given by . For a quantum model, the probability of a response R is given by .

The state of a Markov model evolves across time according to the Kolmogorov forward equation (assuming time invariance), where K is the generator or intensity matrix. The solution to this differential equation is , where is the transition matrix for time t. The generator K must have positive off diagonal entries and columns that sum to zero in order to produce a transition matrix . The transition matrix must contain transition probabilities that sum to unity within each column to generate a new state containing probabilities that sum to unity.

The state of a quantum model evolves across time according to the Schrödinger equation (assuming time invariance), where H is the Hamiltonian matrix. The solution to this differential equation is , where is a unitary matrix for time t. The Hamiltonian matrix H must be Hermitian in order to produce a unitary operator . The unitary matrix must be an orthonormal matrix in order to generate a new state containing amplitudes with squared magnitudes that sum to unity.

For example, according to a Markov model, the probability of deciding cancer is present at time and then giving a confidence rating equal to 65 at time equals: and according to a quantum model, . Essentially, a Markov model operates with a transition matrix on probabilities and uses the norm, but a quantum model operates on amplitudes with a unitary operator and uses the norm.

2.2. Representation of States by Density Operators

We can reformulate a pure quantum process using states described by a density operator instead of a state vector. A state vector can be turned into a density matrix by forming the projector . The quantum evolution of the density matrix is then given by with solution . The advantage of this state representation is that it provides a more general formulation of the state by encompassing a probability mixture across pure states, . By linearity, this more general density matrix continues to follow the same quantum evolution equation . The density matrix thus contains two different types of uncertainty: Epistemic uncertainty in which an outside observer is uncertain about the state of the decision maker, and ontic uncertainty in which the decision maker is in a superposition state over evidence.

The diagonal entries of the density matrix contain the probabilities of observing the evidence levels. The probability of a response R is now computed by the trace . For example, . Now we turn to the more general open system process that contains both quantum and Markov components.

2.3. The Open System Model

An open system model operates on a density matrix operating within the vector space rather than a state vector. The open system model describes the evolution of the density matrix using the following master equation:

The master equation is a weighted sum of two components: The first component represents the contribution from the quantum evolution and the second component contains what are called the Lindblad operators that generate the Markov contribution. The weight determines the relative importance of each contribution. The coefficients form a matrix G, which is required to be positive semi definite to guarantee that the master equation generates a density matrix. The matrices are the Lindblad operators that are discussed below in Section 2.4, and . The trace of must equal zero so that the trace of the density continues to equal to one across time. This implies that when , the trace of the Lindblad component must be zero.

There are at least two different ways to solve Equation (1). One way is to directly solve the differential equation, perhaps using numerical methods. A second way, described by [11,22], is to vectorize the state by stacking each column of on top of each other to form a vector. (Note that is a linear operation.) Equation (1) is linear with respect to , which implies that we can rewrite Equation (1) as a linear differential equation in the form , with: which has the solution . To identify the operator , the following tensor identity is useful (see [23], p. 333): where are matrices, and is the matrix transpose (without conjugation). Then we can write using the following identities:

Collecting these terms together produces:

2.4. Application to Cognitive Science

The first main challenge that a cognitive scientist must face when trying to apply this open system model is to define the Lindblad operators . We recommend following [11,22], and define , where is a column vector with zeros everywhere except for a one located in the row corresponding to the basis vector , and is a row vector with zeros everywhere except for a one located in the column corresponding to . Then the operator represents the transition to from .

The second main challenge is to select the coefficients that form the matrix G. Using , these coefficients can be set equal to the transition probabilities of a Markov chain model. This method provides a direct connection to Markov models familiar to cognitive scientists. The coefficients can be set equal to the transition probabilities of a Markov chain model.

Obviously, if we set , then we obtain exactly the original quantum dynamics for the density matrix. To see how the second (Lindblad) component of Equation (1) is related to a Markov process, we assume .

Using Equation (1) with , first we examine the contributions to the diagonal element of the density matrix (the following analysis was provided by [11]):

Therefore we obtain the final result:

If we set , then , because the columns of the transition matrix must sum to one. Assuming , if we define as the diagonal of then:

Note that if then as required (where is a row vector containing all one’s).

Recall that the continuous time Markov process is based on a generator , and obeys the equation . Comparing this to the final form of the Lindblad equation, , we see that they are not quite the same.

If instead we set , then , and the Lindblad component becomes identical to the Markov process on the diagonal elements of . When , this is not a problem because the diagonals exactly follow the Markov process. But if , then this could become a problem because K is a negative definite (the columns of K sum to zero) and the open system involving both the quantum and Lindblad components are no longer guaranteed to maintain a density matrix across time.

One possible bridge between the two is obtained by setting for a very small value of . Using this assignment, the Lindblad component produces: . This could be used to approximate K but at the same time maintain a positive semi-definite G. However, this proposal runs into trouble when we examine the off diagonals.

Returning to Equation (1) with , next we examine the contributions to the off diagonal element of the density matrix (the following analysis was provided by the second author):

Therefore we obtain the final result:

If we set , then and producing exponential decay of the off diagonals. Alternatively, if we set , then and with no decay of the off diagonals. Finally, if we set then , which very rapidly reduces the off diagonals when is very small.

A different way to compare models is to examine the probability distributions over time produced by a Markov process versus the Lindblad process. For small , the Markov process can be approximated by the equation with solution . The Lindblad component obeys the equation with solution . This comparison shows that both models produce the same probability distributions, but over different time scales: for the Markov process and a slower time t for the Lindblad component.

3. Examples

A couple of examples are presented to illustrate the predictions from the open system model, Equation (1), with .

Consider a simple dimensional open system with two possible responses: An up (e.g., no) state represented by and a down (e.g., yes) state represented by . Suppose , and and we set Then Equation (1) reduces to:

This model starts out oscillating like a quantum process, but eventually converges to the equilibrium of a Markov process. Figure 1 shows the probability of the down state (e.g., yes response) as a function of time. The black curve represents the probabilities generated by the 2-dimensional process with . The equilibrium state is obtained as follows. First, implies that which implies and also (because is Hermitian) that Next implies so that and , which exactly matches the asymptotic result obtained when , which produces a pure Markov process (see Figure 1, red curve). Note also that the asymptotic off diagonal element . Thus the system converges to a coherent density matrix. Without the Lindblad contribution, convergence to an equilibrium state, independent of the initial state, would not be possible. If in the density matrix, then the density is generated by an eigen state of H, and the initial and final states remain the same.

Figure 1

Probability of responding ‘yes’ as a function of time for the 2 dimensional Markov, quantum, open systems using , and .

Alternatively, suppose , and , with sufficiently large to reach the equilibrium transition matrix and we set The green curve in Figure 1 shows the probability of the down state (e.g., the yes response) as a function of time for the 2-dimensional process with . Then Equation (1) reduces to:

In this case, the equilibrium state is obtained as follows. First, implies that . Note that . Then, implies that , and , and the latter falls below the asymptote of a pure Markov process. Finally, .

Now consider another example with a large () number of levels of evidence. For a pure Markov process, we use a generator that has the following tridiagonal form to produce a Markov random walk:

The mean drift rate for this Markov random walk is defined by and the diffusion rate is defined by . In this example, we used a positive mean drift rate that pushes the probability distribution to the right (high evidence levels). This generator uses reflecting bounds, which produces a unique invariant distribution [20].

For a pure quantum process, we use a Hamiltonian that has the following tridiagonal form to produce a quantum walk:

The diagonal contains the potential function, , which can be defined by a quadratic potential . In this example, we simply used a linearly increasing potential, , that pushes the distribution of squared amplitudes toward the right (high levels of evidence). Once the wave hits the reflecting bound, it bounces back producing oscillation and interference. This pure quantum process never reaches an invariant state [19].

Below we compare four different models (see https://jbusemey.pages.iu.edu/quantum/HilbertSpaceModelPrograms.htm for the Matlab program used to make these computations). A pure Markov model, a pure quantum model, an open system model with (the generator for the pure Markov process), and an open system model with . To match the time scales of the pure Markov process and the open system model with , we set the time scale of the latter to . The initial distribution, , was a discrete approximation to a Gaussian distribution centered at the middle of the evidence scale for a pure Markov process and the square root of this distribution was used for the initial state of the quantum and open system models.

First we examine the open system model when . This should reduce the two open system models to a pure Markov process. (For the open system with , we set to obtain a good approximation to the pure Markov process.) The left panel of Figure 2, shows the probability distribution over evidence levels produced at a moderately short time interval. Only two curves can be seen. One is the bell-shaped curve produced by the pure quantum model. The potential function moved the distribution of squared amplitudes from the initial state in the middle, 0.50, up toward the right with a mode around 0.70. The other curve actually includes three overlapping curves produced by the pure Markov model and the two open system models with . This simply shows that the open system does indeed reproduce the Markov model when the quantum component is eliminated.

Figure 2

Left panel: Probability distribution across levels of evidence when for the open system. Right panel: Mean evidence across time when for the open system.

Next we examine the open system model when . The right panel of Figure 2 shows the mean evidence across time produced by the pure quantum model, the pure Markov model, the open system model with , and the open system model with . As can be seen in the right panel, the pure quantum process oscillates between 0.5 and 0.75 indefinitely. The pure Markov process monotonically increases to an asymptote equal to 0.86. The open system model with starts out oscillating like the quantum model, but then converges to the same equilibrium, 0.86, of the Markov model. The the open system model with starts out oscillating like the quantum model, but then converges to a lower equilibrium 0.82 than the Markov model. The supplement provides the first order equations that need to be satisfied for equilibrium (see Supplementary Materials).

4. Summary and Concluding Comments

For over 50 years cognitive scientists have successfully modeled choice probability and distribution of choice times for human decision making using Markov random walk or diffusion processes. However, there is new empirical evidence that these models may not be sufficient and quantum walks may be needed to capture some behavior that can not easily be explained by Markov processes. This new evidence includes interference effects of choice on later confidence and temporal oscillations in preference. Thus both types of processes may be needed and a promising way to combine the two processes is by using open system models. An open system combines quantum and Markov components into a single dynamic process.

One might argue that a simpler way to combine Markov and quantum processes together is simply to form a weighted average of the two separate probability distributions produced by the two separate processes. This is quite different from an open system, which computes a single probability distribution from a single unified process containing both quantum and Markov components. We think the open system is preferred for two important reasons. One is that open systems provide the proper dynamics by starting out in a quantum oscillatory regime and later converge to a Markov regime that reaches an equilibrium. Simply averaging the two processes would create dynamics in which both processes are always operating and present. In particular, a weighted average would continue oscillating indefinitely and never converge to an equilibrium. A second reason is the interpretation of the systems. An open system describes the dynamics of two different types of uncertainty: Epistemic uncertainty in which an outside observer is uncertain about the state of the decision maker, and ontic uncertainty in which the decision maker is in a superposition state over evidence. A simple weighted average would imply that a person’s state is somehow switching from time to time from an epistemic type of uncertainty about the state to an ontic uncertainty about the state.

In this article we reviewed pure Markov random walks, quantum walks, and open systems. We also reviewed two different methods for computing the predictions of an open system: One is to numerically solve the differential equations, and the second is to vectorize the matrix system. We recommend that latter because it provides predictions that can be computed directly from a matrix exponential function.

We also covered important challenges toward applications of open systems to cognitive science. One is the choice of Lindblad operators that form the Lindblad or Markov component of the open system. We recommend following the suggestion by [11] to use the basic transition operators which describe the transitions to state i from state j, making the model similar to Markov chains that are familiar to cognitive scientists.

A second challenge is to define the Lindblad coefficients that form the matrix G. This turned out to be a bit more complicated. On the one hand, one could set , where K is the generator of a continuous time Markov process. This has the advantage of reducing as a special case directly to a continuous time Markov process when the full weight is applied to the Lindblad component. The disadvantage is that it is no longer guaranteed to produce a density matrix across time. The trace always sums to unity but the diagonals could go negative. In all the examples that we have examined, this has not been a major problem, but it could happen. On the other hand, one could set , where is the transition matrix produce by the generator K. This has the advantage of guaranteeing the system always generates a density matrix. However, it has the disadvantage of requiring one to estimate an additional parameter , and it does not reduce to a continuous time Markov process when the full weight is applied to the Lindblad operator. Instead it operates on a time scale inversely related to . It is too early to say which choice of G is best. We recommend trying both ways at this stage. That way one can try and check to see if this causes problems with the density and also try and check to see if the time scale becomes a problem. Furthermore, because the predictions from the two choices for G will not be the same, one can check to see which one actually accounts for the behavioral data best.

Finally one advantage of using the open system as opposed to only the Markov system or only the quantum system is that the fit of the model to data can be used to determine the weight w on each component. If the quantum system is not needed, this weight will reduce to one. However, in applications so far, substantial weight ( on the Lindblad) has been needed to account for the findings [7,11].